mathmari said:
Why do we take this $h$ ? (Wondering)
How exactly have we shown that? (Wondering) Do we conclude to that because of the following?
(Wondering)
Why do we have that these are the only cosets? (Wondering)
Note that:
$\begin{pmatrix}1&x&y\\0&1&z\\0&0&1\end{pmatrix} = \begin{pmatrix}1&x&0\\0&1&z\\0&0&1\end{pmatrix}\begin{pmatrix}1&0&y\\0&1&0\\0&0&1\end{pmatrix}$
If we call the matrix on the LHS, $A$, and the matrices on the RHS $B$ and $Z$, we have:
$A = BZ$, where $Z \in Z(G)$.
This shows that $A \in BZ(G)$, and thus $AZ(G) = BZ(G)$.
Since every coset is of the form $AZ(G)$ for (some) $A \in G$, every coset is of the form $BZ(G)$ (for some $B$, which we can get from $A$). The "special form" matrices $B$ (derived from $A$) form what is called a "traversal" of the cosets, they are "representative representatives" (much like the cosets $k + n\Bbb Z$ for $k \in \{1,2,\dots,n-1\}$ are a traversal of $\Bbb Z/n\Bbb Z$).
When are two cosets the same? That is, when does $AZ(G) = A'Z(G)$? We require:
$AA'^{-1} \in Z(G)$, so:
$\begin{pmatrix}1&x&y\\0&1&z\\0&0&1\end{pmatrix}\begin{pmatrix}1&-x'&x'z' - y'\\0&1&-z'\\0&0&1\end{pmatrix} \in Z(G)$
If we do the matrix multiplication, we get:
$AA'^{-1} = \begin{pmatrix}1&x-x'&x'z'-y'-xz'+y\\0&1&z-z'\\0&0&1\end{pmatrix}$
For this to be in the center, $Z(G)$, all we require is that the $(1,2)$-entry and the $(2,3)$-entry be 0. This is equivalent to setting:
$x - x' = 0$
$z - z' = 0$
so we see that two cosets are the same if $x = x'$ and $z = z'$ (the $(1,3)$-entry is a mess, but we don't care, it's in $M$, which is all that matters).
On the other hand, if $x = x'$ and $z = z'$, running the argument in reverse shows $AZ(G) = A'Z(G)$.
So if the $(1,2)$ and $(2,3)$ entries match, we get the same coset. So we may as well pick a $(1,3)$ entry which is easy to work with. 0 is good.
Look hard at the product decomposition at the beginning of my post. It says:
$\begin{pmatrix}1&x&y\\0&1&z\\0&0&1\end{pmatrix}$
is the same as:
$\begin{pmatrix}1&x&0\\0&1&z\\0&0&1\end{pmatrix}$
"up to multiplication by an element of $Z(G)$".
This is really the same type of thing as saying, for an integer $a = qn + r$, that $r$ is the same as $a$ "up to adding an element of $n\Bbb Z$", that is:
$\begin{pmatrix}1&x&y\\0&1&z\\0&0&1\end{pmatrix}$ is congruent to $\begin{pmatrix}1&x&0\\0&1&z\\0&0&1\end{pmatrix}$
modulo the center of $G$.
In other words, we might as well regard $G/Z(G)$ as the set of matrices:
$\left\{\begin{pmatrix}1&x&y\\0&1&z\\0&0&1\end{pmatrix}: y = 0\right\}$
where we multiply "like usual" except for zeroing out the $(1,3)$-entry after we're done (because anything in $Z(G)$, including the matrix:
$\begin{pmatrix}1&0&xz'\\0&1&0\\0&0&1\end{pmatrix}$
"acts like the identity" in $G/Z(G)$-just like $kn$ "acts like the identity (zero)" in $\Bbb Z_n$).