Suppose \(M\) belongs to \(O(n, C)\) and that \(M'\) is similar to \(M\). Does \(M'\) also belong to \(O(n, C)\)?
Solution
1. The answer is no: a matrix similar to a complex orthogonal matrix is not necessarily orthogonal.
statement of claim.
2. Counterexample in \(n = 2\):
2.1. Consider the matrix:\[ M = \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix}. \]Compute \(M^t M = \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix} \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} = I_2\).Thus \(M \in O(2, \mathbb{C})\).
definition of the complex orthogonal group \(O(n, \mathbb{C}) = \{A \in M_n(\mathbb{C}) : A^t A = I_n\}\).
2.2. Choose the invertible matrix \(P = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix} \in \operatorname{GL}_2(\mathbb{C})\), with inverse \(P^{-1} = \begin{pmatrix} 1 & -1 \\ 0 & 1 \end{pmatrix}\).
\(\det(P) = 1 \neq 0\).
2.3. Define \(M' = P M P^{-1}\), which is similar to \(M\) by construction:\[ M' = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix} \begin{pmatrix} 1 & -1 \\ 0 & 1 \end{pmatrix} = \begin{pmatrix} 1 & -1 \\ 0 & -1 \end{pmatrix} \begin{pmatrix} 1 & -1 \\ 0 & 1 \end{pmatrix} = \begin{pmatrix} 1 & -2 \\ 0 & -1 \end{pmatrix}. \]
matrix multiplication.
2.4. Compute \((M')^t M'\):\[ (M')^t M' = \begin{pmatrix} 1 & 0 \\ -2 & -1 \end{pmatrix} \begin{pmatrix} 1 & -2 \\ 0 & -1 \end{pmatrix} = \begin{pmatrix} 1 & -2 \\ -2 & 5 \end{pmatrix} \neq I_2. \]
matrix multiplication.
2.5. Since \((M')^t M' \neq I_2\), \(M' \notin O(2, \mathbb{C})\).
failure of the orthogonality condition.
3. Conclusion: \(M'\) does not necessarily belong to \(O(n, \mathbb{C})\).Q.E.D.
1 and 2.