First off, your use of the preposition "in" is making it very difficult for me to determine if, no offense, you really understand what you're doing. If you would, please use "is a member/element of", "is a subset of", "is a subgroup of", or the appropriate symbols to make your intended meaning clear so that I can more easily help you.
Also, I realize I'm being kinda hard on you here. Don't take it personally.
Justabeginner said:
If a, b in A and H is a subgroup of G (assumption of entire problem) then abH(ab)^-1 in G too. So, abHb^-1a^-1 in G. This means that a(bHb^-1)a^-1 in G,
That ##abH(ab)^{-1}## is a subset of ##G## is too trivial to mention. That ##abH(ab)^{-1}## is a subgroup of ##G## is not as trivial, but as far as I can tell, it's not really relevant to the problem.
Noting that ##(ab)H(ab)^{-1}=(ab)H(b^{-1}a^{-1})=a(bHb^{-1})a^{-1}## is important, and I think most reasonable people would accept that as true with no further explanation. You should, however, convince yourself why the "obvious" symbol-pushing actually works in this case, though.
and if bHb^-1 in G, then b in G.
Again ##bHb^{-1}## being a subset/subgroup of ##G## is not particularly interesting or relevant to this proof. Furthermore, you don't need to know that to conclude that ##b\in G##. ##b## is assumed to be an element of ##A## which is assumed to be a subset of ##G##. ##b## is trivially an element of ##G##.
Also, aa^-1 in G is equivalent to e in G
Ok ...
Yes, ##A## is a subset of ##G##. That's trivial. If by "in" you mean "is a subgroup of", I ask "why?" Why is ##A## a subgroup of ##G##, and why was the fact that ##aa^{-1}=e## apparently the key piece needed to finish the proof?
Here is a string of equalities, some of which you've come up with on you own, that I think is important to proving that ##A## is closed under the groups operation of ##G##; $$(ab)H(ab)^{-1}=(ab)H(b^{-1}a^{-1})=a(bHb^{-1})a^{-1}=aHa^{-1}=H$$
What I would like for you to do is answer, for yourself, the "why"s. Why is each equality true? Why is this important to showing that ##A## is closed under the groups operation of ##G##?
I will try to focus more on specific key ideas rather than generalized theorems to pinpoint the meaning of the question.
I think that's a fantastic idea. If I'm not mistaken, you're self-studying. In my opinion, there's no point to trying to learn this particular material on your own if you're not aiming for near total understanding.
Am I correct in saying that all you have to prove for A to be a subgroup is closure and inverses/identity?
The are many ways to establish that a subset of a group is a subgroup. The most common for "beginners" are the "one-" and "two-srep" methods. The ""two-step" method involves demonstrating closure of the subset under the group operation and inverses; i.e. showing given ##a,b\in A## that ##ab\in A## (closure under the group operation) and ##a^{-1}\in A## (closure under inverses). The "one-step" method works by showing given ##a,b\in A##, that ##ab^{-1}\in A##.