Well the "matrix of minors" (of a square matrix) would be the matrix where each element is the minor of the number which was in that position, which isn't the same as the adjoint yet.
For the adjoint, you'd have to take the cofactors (which is the minors + sign taken into account) and transpose that. So adjoint = transpose of cofactor matrix.