Exhibit a set \(S\) of real \(2\times2\) matrices with the following property: every invertible \(2\times2\) integer matrix whose characteristic roots all lie on the unit circle is similar over \(\RR\) to exactly one matrix in \(S\).
Solution
Let \(A\in GL_2(\ZZ)\) have both characteristic roots on the unit circle, and write \[ \chi_A(T)=T^2-tT+d, \qquad t=\operatorname{tr}A, \qquad d=\det A. \]
1. The only possible characteristic polynomials are\[ T^2-1, \quad (T-1)^2, \quad (T+1)^2, \quad T^2+T+1, \quad T^2+1, \quad T^2-T+1. \]
If \(\alpha,\beta\) are the two roots, then \[ \abs d=\abs{\alpha\beta}=1. \] Since \(d\) is a nonzero integer, \(d=\pm1\).
Suppose first that \(d=1\). Then \[ \abs t=\abs{\alpha+\beta}\leq2. \] Because \(t\in\ZZ\), one has \(t\in\{-2,-1,0,1,2\}\), yielding the five determinant-\(1\) polynomials in the displayed list.
If \(d=-1\), the two roots cannot be nonreal: nonreal roots of a real quadratic are conjugate and would have product \(1\). Thus both roots are real points of the unit circle. Their product is \(-1\), so they are \(1\) and \(-1\), giving \(\chi_A(T)=T^2-1\).
2. For \(t\in\{-1,0,1\}\), every matrix with characteristic polynomial \(T^2-tT+1\) is similar over \(\RR\) to\[ C_t\coloneqq \begin{pmatrix} 0&-1\\ 1&t \end{pmatrix}, \]and every matrix with characteristic polynomial \(T^2-1\) is similar to \(\operatorname{diag}(1,-1)\).
For \(t\in\{-1,0,1\}\) the discriminant \(t^2-4\) is negative, so \(A\) has no real eigenvector. Choose any nonzero \(v\in\RR^2\). Then \(v\) and \(Av\) are linearly independent and form a basis. Cayley–Hamilton gives \[ A^2v=tAv-v, \] so the matrix of \(A\) in the basis \((v,Av)\) is exactly \(C_t\).
If \(\chi_A(T)=T^2-1\), the two eigenvalues \(1\) and \(-1\) are distinct. Hence \(A\) is diagonalizable over \(\RR\) and is similar to \(\operatorname{diag}(1,-1)\).
3. A matrix with characteristic polynomial \((T-1)^2\) is similar to exactly one of\[ I_2, \qquad J_+=\begin{pmatrix}1&1\\0&1\end{pmatrix}, \]and a matrix with characteristic polynomial \((T+1)^2\) is similar to exactly one of\[ -I_2, \qquad J_-=\begin{pmatrix}-1&1\\0&-1\end{pmatrix}. \]
Suppose \(\chi_A(T)=(T-1)^2\). If \(A=I_2\), this is the scalar class. Otherwise \(N=A-I_2\) is nonzero and satisfies \(N^2=0\) by Cayley–Hamilton. Choose \(v\) with \(Nv\neq0\). Then \((Nv,v)\) is a basis and the matrix of \(N\) in this basis is \[ \begin{pmatrix}0&1\\0&0\end{pmatrix}. \] Thus \(A\) is similar to \(J_+\). The two possibilities are not similar because one is scalar and the other is not.
The same argument applied to \(N=A+I_2\) gives the two classes \(-I_2\) and \(J_-\) when \(\chi_A(T)=(T+1)^2\).
4. A set with the required property is\[ S=\boxed{\left\{ I_2, \begin{pmatrix}1&1\\0&1\end{pmatrix}, -I_2, \begin{pmatrix}-1&1\\0&-1\end{pmatrix}, \begin{pmatrix}1&0\\0&-1\end{pmatrix}, \begin{pmatrix}0&-1\\1&-1\end{pmatrix}, \begin{pmatrix}0&-1\\1&0\end{pmatrix}, \begin{pmatrix}0&-1\\1&1\end{pmatrix} \right\}}. \]
Steps <1>1–<1>3 show that every admissible \(A\) is similar to at least one matrix in \(S\).
Matrices in \(S\) having different characteristic polynomials cannot be similar. The only pairs in \(S\) with the same characteristic polynomial are \(I_2\) and \(J_+\), and \(-I_2\) and \(J_-\). Step 3 shows that the two matrices in each such pair are not similar. Hence every admissible matrix is similar to exactly one member of \(S\).
5. Q.E.D.
Step 4 exhibits and verifies the required set.