Theorem 2.G (p. 62, chapter 2):If $G$ is a group, $H$ a subgroup of $G$, and $S$ is the set of all right cosets of $H$ in $G$, then there is a homomorphism $\theta$ of $G$ into $A(S)$, and the kernel of $\theta$ is the largest normal subgroup of $G$ which is contained in $H$.
(a few words about notation: Herstein uses $A(S)$ to stand for the group of all bijections on $S$...if $|S| = n$, then $A(S)$ is isomorphic to $S_n$. Herstein also writes his mappings on the RIGHT, as in $(x)\sigma$ instead of $\sigma(x)$, so that composition and multiplication are "in the same order", instead of reversed. For this reason, he uses right cosets and right-multiplication instead of the left cosets (and left-multiplication) one often sees used in other texts. He also denotes the index of $H$ in $G$ by $i(H)$ , instead of $[G
]$ and denotes $|G|$ by $o(G)$).
Proof: Let $G$ be a group, $H$ a subgroup of $G$. Let $S$ be the set whose elements are right cosets of $H$ in $G$. That is, $S = \{Hg: g \in G\}$. $S$ need not be a group itself, in fact, it would be a group only if $H$ were a normal subgroup of $G$. However, we can make our group $G$ act on $S$ in the following natural way: for $g \in G$ let $t_g:S \to S$ be defined by: $(Hx)t_g = Hxg$. Emulating the proof of Theorem 2.f we can easily prove:
(1) $t_g \in A(S)$ for every $g \in G$
(2) $t_{gh} = t_gt_h$.
Thus the mapping $\theta: G \to A(S)$ defined by $\theta(g) = t_g$ is a homomorphism of $G$ into $A(S)$. Can one always say that $\theta$ is an isomorphism? Suppose that $K$ is the kernel of $\theta$. If $g_0 \in K$, then $\theta(g_0) = t_{g_0}$ is the identity map on $S$, so that for every $X \in S, Xt_{g_0} = X$. Since every element of $S$ is a right coset of $H$ in $G$, we must have that $Hat_{g_0} = Ha$ for every $a \in G$, and using the definition of $t_{g_0}$, namely, $Hat_{g_0} = Hag_0$, we arrive at the identity $Hag_0 = Ha$ for every $a \in G$. On the other hand if $b \in G$ is such that $Hxb = Hx$ for every $x \in G$, retracing our argument we could show that $b \in K$. Thus $K = \{b \in G|Hxb = Hx$ all $x \in G\}$. We claim that from this characterization of $K,\ K$ must be the largest normal subgroup of $G$ which is contained in $H$. We first explain the use of the word largest; by this we mean if $N$ is a normal subgroup of $G$ which is contained in $H$, then $N$ must be contained in $K$. We wish to show this is the case. That $K$ is a normal subgroup of $G$ follows from the fact that it is the kernel of a homomorphism of $G$. Now we assert that $K \subset H$, for if $b \in K, Hab = Ha$ for every $a \in G$, so in particular, $Hb = Heb = He = H$, whence $b \in H$. Finally, if $N$ is a normal subgroup of $G$ which is contained in $H$, if $n \in N,\ a \in G$, then $ana^{-1} \in N \subset H$, so that $Hana^{-1} = H$; thus $Han = Ha$ for all $a \in G$. Therefore, $n \in K$ by our characterization of $K$.
**********
Remarks following the proof:
The case $H = (e)$ just yields Cayley's Theorem (Theorem 2.f). If $H$ should happen to have no normal subgroup of $G$, other than $(e)$ in it, then $\theta$ must be an isomorphism of $G$ into $A(S)$...(some text omitted)...
We examine these remarks a little more closely. Suppose that $G$ has a subgroup $H$ whose index $i(H)$ (that is, the number of right cosets of $H$ in $G$) satisfies $i(H)! < o(G)$. Let $S$ be the set of all right cosets of $H$ in $G$. The mapping, $\theta$, of Theorem 2.g cannot be an isomorphism, for if it were, $\theta(G)$ would have $o(G)$ elements and yet would be a subgroup of $A(S)$ which has $i(H)! < o(G)$ elements. Therefore, the kernel of $\theta$ must be larger than $(e)$; this kernel being the largest normal subgroup of $G$ which is contained in $H$, we can conclude that $H$ contains a nontrivial normal subgroup of $G$.
However, the above argument has implications even when $i(H)!$ is not less than $o(G)$. If $o(G)$ does not divide $i(H)!$ then by invoking Lagrange's theorem we know that $A(S)$ can have no subgroup of order $o(G)$, hence no subgroup isomorphic to $G$. However $A(S)$ does contain $\theta(G)$, whence $\theta(G)$ cannot be isomorphic to $G$, that is, $\theta$ cannot be an isomorphism. But then, as above, $H$ must contain a nontrivial normal subgroup of $G$. We summarize this as:
Lemma 2.21 If $G$ is a finite group, and $H \neq G$ is a subgroup of $G$ such that $o(G) \not\mid i(H)!$, then $H$ must contain a nontrivial normal subgroup of $G$. In particular, $G$ cannot be simple.
(Note to the moderating staff: although this is an excerpt from a copyrighted work, I believe this sample falls under the province of "Fair Use" for the purpose of "Scholarly research and exposition", and is not intended for commercial gain or to circumvent existing copyright laws).