It might be a good idea to move away from matrices and look at the general picture. If T is a linear tranformation from an "inner product space" V (a vector space with an inner product, <u, v>V, defined) to an inner product space U (with inner product <u,v>U) then the "adjoint" of T, Tt, is defined as the linear transformation from U back to V such that, for all u in U, v in V, <Tv,u>U= <v,Ttu>V. Note that both Tv and u are in u so that inner product must be the U product while v and Ttu are in V and so that inner product must be the V product.
In particular, U and V are finite dimesional vector spaces over the real numbers, so that T and Tt can be represented by matrices with real entries, Tt is just the "transpose"- swapping rows for columns. If U and V are finite dimensional vector spaces over the complex numbers, and T and Tt can be represented as matrices with comples entries, then, because of the requirement that <u,v>= complex conjugate of <v,u>, Tt must be also take the complex conjugate of entries in T.