This is true because a matrix is invertible if and only if its column vectors are all linearly independent (If the columns are linearly dependent on the other hand, the matrix will have determinant zero and thus not be invertible)
i.e. if A has column vectors [itex]v_1,...,v_n[/itex], then those vectors are linearly independent if and only if the equation [itex]a_1v_1 +...+a_nv_n\cdot =0[/itex] has only the trivial solution [itex]a_1=...=a_n=0[/itex] This is equivalent to saying [itex]Ax=0[/itex] has only the trivial solution. To see this, write the linear independence expression in matrix notation.
I get it when I say that the column vectors are linearly independent (as you point out) but when one reduce a matrix to its identity matrix one will get I if the rows are independent. How does this show that also the columns are independent?
Well do you remember how you find the inverse of a matrix? you reduce it alongside the identity, i.e. reduce [itex](A|I_n)[/itex], If you can arrive at the identity via row reduction, then A is invertible. And a matrix is invertible if and only if the columns are independent. The two statements go hand in hand, each implies the other. Alternatively, you know that the Identity matrix has linearly independent column vectors. It follows that if you can reduce a matrix by performing elementary row operations into the identity, then the original column vectors must be linearly independent.