If the matrix is not square, then this is impossible. The null space of a matrix A consists of vectors x such that Ax = 0. If A is not square, and Ax is defined (i.e. you are allowed to multiply A and x) then ATx is not even defined. I'm not sure what you're asking though. In general, the null space of a matrix is not the same if it as the null space of its transpose. However, certainly if the matrix is symmetric then its kernel is the same as the kernel of its transpose, since the matrix is its own transpose.