Let \(T\) be the linear operator on \(\RR^3\) which is represented in the standard ordered basis by the matrix \[ \begin{bmatrix} 2 & 0 & 0 \\ 1 & 2 & 0 \\ 0 & 0 & 3 \end{bmatrix}. \] Let \(W\) be the null space of \(T - 2I\). Prove that \(W\) has no complementary \(T\)-invariant subspace. (Hint: Let \(\beta = \epsilon_1\) and observe that \((T - 2I)\beta\) is in \(W\). Prove there is no \(\alpha\) in \(W\) with \((T - 2I)\beta = (T - 2I)\alpha\).)
Solution
Let \(e_1,e_2,e_3\) be the standard basis. From the given matrix, \[ (T-2I)(x,y,z)=(0,x,z). \] Hence \[ W=\ker(T-2I)=\{(0,y,0):y\in\mathbb R\} =\operatorname{span}\{e_2\}. \]
1. Any complementary invariant subspace would have dimension two.
If \(V=W\oplus U\), then \[ \dim U=3-\dim W=2. \] Assume, toward a contradiction, that such a complement \(U\) is also \(T\)-invariant.
2. The quotient by \(W\) forces \(U\) to contain a vector with nonzero \(e_1\)-component.
If every vector of \(U\) had first coordinate zero, then \(U\subseteq\operatorname{span}\{e_2,e_3\}\). Since \(U\cap W=\{0\}\) and \(W=\operatorname{span}\{e_2\}\), the projection of \(U\) to the quotient \(\operatorname{span}\{e_2,e_3\}/W\) would have dimension at most one. But \(U\) has dimension two and intersects \(W\) trivially, so its quotient image must have dimension two, a contradiction. Thus some \[ u=(a,b,c)\in U \] has \(a\ne0\).
3. Invariance then forces a nonzero vector of \(W\) into \(U\).
Because \(U\) is \(T\)-invariant, it is also invariant under \(T-2I\). Hence \[ (T-2I)u=(0,a,c)\in U. \] Also \(e_3\) is an eigenvector with eigenvalue \(3\), so the one-dimensional \(3\)-eigenspace \(\operatorname{span}\{e_3\}\) is \(T\)-invariant. Since a two-dimensional complement of \(W\) must map isomorphically to \(V/W\), its image contains the class of \(e_3\); subtracting a suitable vector of \(W\) shows that \(U\) contains some vector \((0,b',1)\). Applying \(T-3I\) to that vector gives \((0,-b',0)\in U\cap W\). If \(b'\ne0\) this already contradicts directness; if \(b'=0\), then \(e_3\in U\).
In the latter case, subtract \(c e_3\) from \((T-2I)u\) to obtain \[ (0,a,0)=ae_2\in U. \] Since \(a\ne0\), this is a nonzero element of \(U\cap W\), again a contradiction. Therefore no complementary \(T\)-invariant subspace exists [@HK71].