I'm not sure what you mean by "for a polynomial". The formula is T(A)= AT- A.
Perhaps you are thinking that AT is a power? Even if it were, that would still be alright- if you can multiply matrices you can certainly take a matrix to a power- and a product of matrices still represents a linear transformation.
However, AT is the standard notation for the 'transpose' of a matrix: basically you swap rows and columns. For a 2 by 2 matrix
[tex]\left[\begin{array}{cc}a & b \\ c & d\end{array}\right]^T= \left[\begin{array}{cc} a & c \\ b & d\end{array}\right][/tex]
so that
[tex]\left[\begin{array}{cc}a & b \\ c & d\end{array}\right]^T- \left[\begin{array}{cc}a & b \\ c & d\end{array}\right]= \left[\begin{array}{c c} a & c \\ b & d\end{array}\right]- \left[\begin{array}{cc}a & b \\ c & d\end{array}\right][/itex]<br />
[tex]= \left[\begin{array}{cc} 0 & c- b \\ b- c & 0\end{array}\right][/tex][/tex]