NOTES:

• The columns of $A^T$ are the rows of $A$; $(A^T)_{ij} = A_{ji}$.
• Proof of $(Ax)^T = x^TA^T$, where $x$ is a column vector:
  $Ax = \left( \begin{matrix} \mathbf{a_1} \, \mathbf{a_2} \, \cdots \, \mathbf{a_n} \end{matrix} \right) \left( \begin{matrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{matrix} \right) = \left( x_1 \mathbf{a_1} + x_2 \mathbf{a_2} + \, \cdots + \, x_n \mathbf{a_n} \right) = \left( \begin{matrix} x_1 a_{11} + x_2 a_{12} + x_3 a_{13} + \cdots + x_n a_{1n} \\ x_1 a_{21} + x_2 a_{22} + x_3 a_{23} + \cdots + x_n a_{2n} \\ \vdots \\ x_1 a_{n1} + x_2 a_{n2} + x_3 a_{n3} + \cdots + x_n a_{nn} \end{matrix} \right)_{n*1} $

  Then $(Ax)^T$ should be $ \phantom{ \begin{matrix} 1 \\ 1 \end{matrix} } \left( \begin{matrix} x_1 a_{11} + x_2 a_{12} + x_3 a_{13} + \cdots + x_n a_{1n} \quad x_1 a_{21} + x_2 a_{22} + x_3 a_{23} + \cdots + x_n a_{2n} \quad \cdots \quad x_1 a_{n1} + x_2 a_{n2} + x_3 a_{n3} + \cdots + x_n a_{nn} \end{matrix} \right)_{1*n} $, which equals to $ \left( \begin{matrix} x_1 \, x_2 \, \cdots \, x_n \end{matrix} \right) \left( \begin{matrix} a_{11} \, a_{21} \, a_{31} \, \cdots \, a_{n1} \\ a_{12} \, a_{22} \, a_{32} \, \cdots \, a_{n2} \\ \vdots \\ a_{1n} \, a_{2n} \, a_{3n} \, \cdots \, a_{nn} \end{matrix} \right) = x^TA^T $

  Therefore, $(Ax)^T = x^TA^T$

  $Ax$ combines the columns of $A$ while $x^TA^T$ combines the rows of $A^T$.
• Proof of $(AB)^T = B^TA^T$:
  $AB = A \left( \begin{matrix} \mathbf{b_1} \; \mathbf{b_2} \; \cdots \; \mathbf{b_n} \end{matrix} \right) = \left( \begin{matrix} A\mathbf{b_1} \; A\mathbf{b_2} \; \cdots \; A \mathbf{b_n} \end{matrix} \right) $

  Then $(AB)^T$ should be $ \left( \begin{matrix} (A\mathbf{b_1})^T \\ (A\mathbf{b_2})^T \\ \vdots \\ (A \mathbf{b_n})^T \end{matrix} \right) $, which equals to $ \left( \begin{matrix} \mathbf{b_1}^T A^T \\ \mathbf{b_2}^T A^T \\ \vdots \\ \mathbf{b_n}^T A^T \end{matrix} \right) $, according to the argument we just proved above.

  And, $ \left( \begin{matrix} \mathbf{b_1}^T A^T \\ \mathbf{b_2}^T A^T \\ \vdots \\ \mathbf{b_n}^T A^T \end{matrix} \right) = \left( \begin{matrix} \mathbf{b_1}^T \\ \mathbf{b_2}^T \\ \vdots \\ \mathbf{b_n}^T \end{matrix} \right) A^T = B^TA^T $, according to Matrix Operations Chapter 2 (2.4)

  Therefore, we have proved $(AB)^T = B^TA^T$.
• $ (ABC)^T = C^TB^TA^T $; If $A = LDU$, then $A^T = U^TD^TL^T$.
• $(A+ B)^T = A^T + B^T$
• $ (A^{-1})^T = (A^T)^{-1} $; $A^T$ is invertible exactly when $A$ is invertible.
• $(Ax)^Ty = x^T(A^Ty)$: Inner product of $Ax$ with $y =$ Inner product of $x$ with $A^Ty$.
• Given a symmetric matrix $S$, the inverse of $S$ is also symmetric. $$ (S^{-1})^T = (S^T)^{-1} = S^{-1} $$
• Choose any matrix $A$, probably rectangular. $S = A^TA$ or $S' = AA^T$ is automatically a square symmetric matrix.
• The $LU$ elimination of a symmetric matrix $S$ is symmetric: $ S = LDL^T$.
• A permutation matrix $P$ has the rows of the identity matrix $I$ in any order. There are $n!$ permutation matrices of order $n$.
• $P^{-1}$ is also a permutation matrix. A single row exchange is its own inverse.
• $P^{-1}$ is always the same as $P^T$. When we multiply $PP^T$, the "1" in the first row of $P$ hits the "1" in the first column of $P^T$. It misses the ones in all the other columns. So $PP^T=I$.
• If $A$ is invertible, a permutation $P$ will put its rows in the right order to factor $PA=LU$ There must be a full set of pivots after row exchanges for $A$ to be invertible.