as i understand it, in inorganic chemistry they classify point groups (what we would ordinarily call symmetry groups of geometric figures) according to character tables.
so really, it's not "groups" per se, but group representation theory as symmetry operations (in R
2, R
3, usually). it's the back-door into group theory, and i imagine many chemistry students find it totally bewildering.
so it's "more" than just group theory, to have it all come "full circle" for you, you'll need to not only understand group theory, but a decent amount of linear algebra, as well (the idea behind all this, is to turn "abstract" groups, into "concrete things" we understand better, namely, linear transformations).
for example, the symmetry group of H
2O, has 4 elements:
E = the identity
C
2 = a rotation of 180 degrees about the z-axis
σ
v = reflection in the xz-plane
σ
v' = reflection in the yz-plane
but an algebraist would just say "the Klein 4-group (or Viergruppe)", perhaps as:
{e,a,b,ab} a
2 = b
2 = e, ab = ba
or perhaps as Z
2xZ
2 = {(0,0),(1,0),(0,1),(1,1)}.
perhaps a more familar way to look at this group is the set of 4 3x3 matrices:
\begin{bmatrix}1&0&0\\0&1&0\\0&0&1\end{bmatrix},\ \begin{bmatrix}-1&0&0\\0&-1&0\\0&0&1\end{bmatrix},\ \begin{bmatrix}-1&0&0\\0&1&0\\0&0&1\end{bmatrix},\ \begin{bmatrix}1&0&0\\0&-1&0\\0&0&1\end{bmatrix}
although a close inspection will reveal all the action is happening in the 2x2 matrix blocks up in the upper left corner, the z-coordinate is "just along for the ride" (this representation isn't irreducible).
the general idea is this: if you have a set X of "somethings", and you have a collection of "transformations" (functions) that do something to X, and leave it "unchanged" (i am being deliberately vague here, on purpose) we call the set of such transformations symmetries. the most general set of such functions even has a special name: S
X, or Sym(X), and is called the full symmetric group on X. this is sort of "the big daddy group" where you can find most of the "baby groups" (this is a silly way of saying something that has it's own name, Cayley's theorem).
so (all of the proper and correct axioms stated in other posts aside), a group is (at its heart) a collection of "reversible transformations of a set". in chemistry, the "set" in question is a collection of atoms (a molecule), and the transformations are geometric operations that give you a molecule "you can't tell from the original" (hence the name "symmetry").
a proper course in group theory, will spend a lot of time on studying small sets first (there is a lot of richness, even with as few as, say 4 elements, in the set X). you'll want to learn about:
cyclic groups
dihedral groups
the symmetric group Sn (this one is a doozy)
general linear groups <--this, especially, is important
special linear groups
orthogonal groups
special orthogonal groups
because these will be the ones most useful to you.
learning how character tables are created, and what they mean, is a bit beyond what you will find in most books on group theory, even good ones. after (and if) you learn a bit more group theory (i like the dogschool site linked to above, it's easy on a newbie), if you're feeling more ambitious, you can try:
http://www.win.tue.nl/~amc/ow/gpth/reader.pdf
but i must warn you, it's not for the faint of heart (it's a good read for physics students, too).
