Let \(A,B\) be real \(n\times n\) matrices. Suppose \[ A^T=A, \qquad v^TAv\ge0\quad\text{for every }v\in\mathbb R^n, \] and \[ AB+BA=0. \] Show that \[ AB=BA=0. \] Give an example in which neither \(A\) nor \(B\) is zero.
Solution
Since \(A\) is real symmetric and positive semidefinite, choose an orthonormal basis in which \[ A=\diag(\lambda_1,\ldots,\lambda_n), \qquad \lambda_i\geq0. \] Write \(B=(b_{ij})\) in the same basis.
1. If \(\lambda_i>0\) or \(\lambda_j>0\), then\[ b_{ij}=0. \]
The \((i,j)\)-entry of \[ AB+BA=0 \] is \[ (\lambda_i+\lambda_j)b_{ij}=0. \] Because all eigenvalues are nonnegative, the assumption that at least one of \(\lambda_i,\lambda_j\) is positive implies \[ \lambda_i+\lambda_j>0. \] Therefore \(b_{ij}=0\).
2. One has\[ AB=0. \]
The \((i,j)\)-entry of \(AB\) is \[ \lambda_i b_{ij}. \] If \(\lambda_i=0\), this entry is zero. If \(\lambda_i>0\), step 1 gives \(b_{ij}=0\). Thus every entry of \(AB\) is zero.
3. One has\[ BA=0. \]
The \((i,j)\)-entry of \(BA\) is \[ b_{ij}\lambda_j. \] If \(\lambda_j=0\), this entry is zero. If \(\lambda_j>0\), step 1 gives \(b_{ij}=0\). Thus every entry of \(BA\) is zero.
4. Neither matrix need be zero.
For example, in dimension \(2\) take \[ A= \begin{pmatrix} 1&0\\ 0&0 \end{pmatrix}, \qquad B= \begin{pmatrix} 0&0\\ 0&1 \end{pmatrix}. \] Then \(A\) is symmetric positive semidefinite, both \(A\) and \(B\) are nonzero, and \[ AB=BA=0. \]
5. Q.E.D.
Steps 2 and 3 prove the required vanishing, and step 4 supplies the requested example.