Let \(A,B\) be complex \(n\times n\) matrices satisfying \[ AB=BA^2, \] and suppose \(A\) has no eigenvalue of absolute value \(1\). Prove that \(A\) and \(B\) have a common nonzero eigenvector.
Solution
1. For every eigenvalue \(\lambda\) of \(A\),\[ B\bigl(E_\lambda(A)\bigr) \subseteq E_{\lambda^2}(A), \]where \(E_\mu(A)=\ker(A-\mu I)\) and \(E_\mu(A)=\{0\}\) when \(\mu\) isnot an eigenvalue.
If \(v\in E_\lambda(A)\), then \[ \begin{aligned} A(Bv) &= BA^2v\\ &= B(\lambda^2v)\\ &= \lambda^2Bv. \end{aligned} \] Thus \(Bv\in\ker(A-\lambda^2I)=E_{\lambda^2}(A)\).
2. If \(A\) has an eigenvalue of modulus greater than \(1\), then \(A\) and\(B\) have a common nonzero eigenvector.
Choose an eigenvalue \(\lambda\) of \(A\) having maximal absolute value. Under the present hypothesis, choose it with \(\abs{\lambda}>1\). Then \[ \abs{\lambda^2} = \abs{\lambda}^2 > \abs{\lambda}, \] so \(\lambda^2\) is not an eigenvalue of \(A\) by maximality. Step 1 therefore gives \[ B(E_\lambda(A))=\{0\}. \] Any nonzero \(v\in E_\lambda(A)\) then satisfies \[ Av=\lambda v, \qquad Bv=0, \] so \(v\) is a common eigenvector.
3. Suppose every eigenvalue of \(A\) has modulus less than \(1\) and\(0\) is not an eigenvalue. Then \(A\) and \(B\) have a common nonzeroeigenvector.
Choose an eigenvalue \(\lambda\) of \(A\) having minimal absolute value. Since \(0<\abs{\lambda}<1\), \[ 0 < \abs{\lambda^2} < \abs{\lambda}. \] Thus \(\lambda^2\) is not an eigenvalue of \(A\) by minimality. As in step 2, step 1 gives \(B(E_\lambda(A))=\{0\}\), and any nonzero \(v\in E_\lambda(A)\) is a common eigenvector of \(A\) and \(B\).
4. Suppose every eigenvalue of \(A\) has modulus less than \(1\) and\(0\) is an eigenvalue. Then \(A\) and \(B\) have a common nonzero eigenvector.
Step 1 with \(\lambda=0\) gives \[ B(E_0(A))\subseteq E_0(A). \] The space \(E_0(A)\) is nonzero. Since it is a finite-dimensional complex vector space, the restriction \[ B|_{E_0(A)} \] has an eigenvector \(v\neq0\). For some \(\mu\in\CC\), \[ Av=0, \qquad Bv=\mu v. \] Hence \(v\) is a common eigenvector.
5. \(A\) and \(B\) have a common nonzero eigenvector.
Because \(A\) is a complex matrix, it has an eigenvalue. By hypothesis no eigenvalue has modulus \(1\). If some eigenvalue has modulus greater than \(1\), step 2 applies. Otherwise all eigenvalues have modulus less than \(1\), and either step 3 or step 4 applies according as \(0\) is or is not an eigenvalue.
6. Q.E.D.
Step 5 is the required conclusion.