Let \(A,B\) be real \(n\times n\) matrices. Suppose \[ A^{\mathsf T}=A, \qquad v^{\mathsf T}Av\ge0\quad(v\in\RR^n), \] and \[ AB+BA=0. \] Prove that \[ AB=BA=0, \] and give an example in which neither \(A\) nor \(B\) is zero.
Solution
1. There is an orthogonal matrix \(Q\) such that\[ Q^{\mathsf T}AQ =D =\operatorname{diag}(\lambda_1,\ldots,\lambda_n) \]with \(\lambda_i\geq0\) for every \(i\).
The matrix \(A\) is real symmetric, so the spectral theorem gives an orthogonal diagonalization. The hypothesis \[ v^{\mathsf T}Av\geq0 \] for all \(v\) says that \(A\) is positive semidefinite, hence every eigenvalue \(\lambda_i\) is nonnegative.
2. If \(C\coloneqq Q^{\mathsf T}BQ=(c_{ij})\), then \(c_{ij}=0\) whenever \(\lambda_i>0\) or \(\lambda_j>0\).
Conjugating the relation \(AB+BA=0\) by \(Q\) gives \[ DC+CD=0. \] Its \((i,j)\) entry is \[ (\lambda_i+\lambda_j)c_{ij}=0. \] Because \(\lambda_i,\lambda_j\geq0\), the sum \(\lambda_i+\lambda_j\) is positive whenever either eigenvalue is positive. Hence \(c_{ij}=0\) in that case.
3. \(AB=BA=0\).
By step 2, a possibly nonzero entry \(c_{ij}\) can occur only when \(\lambda_i=\lambda_j=0\). Therefore every entry of both \(DC\) and \(CD\) is zero, so \[ DC=CD=0. \] Conjugating back by \(Q\) yields \[ AB=BA=0. \]
4. An example with neither matrix zero is\[ \boxed{ A=\begin{pmatrix}1&0\\0&0\end{pmatrix}, \qquad B=\begin{pmatrix}0&0\\0&1\end{pmatrix} }. \]
The matrix \(A\) is symmetric and positive semidefinite, both \(A\) and \(B\) are nonzero, and direct multiplication gives \[ AB=BA=0. \] Thus \(AB+BA=0\) as required.
5. Q.E.D.
Step 3 proves the asserted annihilation, and step 4 supplies the requested example.