In the standard model of particle physics, the gauge group is SU(3) cross SU(2) cross U(1) and the Poincare group is also relevant. These are all Lie groups, the general theory of which, you wouldn't see in an introductory class. The Poincare group is all the symmetries of Minkowski space-time, and the gauge group is internal symmetries of the fields.
I would almost think of guys like SU(2) as being more linear-algebraic things that just happen to be groups, which I think explains why a lot of graduate physics students that I have come across get by to some extent without knowing what a group is (not sure what these people were studying, exactly).
So-called "representations" of these groups can be important in particle physics. And when you study representation theory, the "group" properties of guys like SU(2) really comes into play. Basic representation theory is simpler in the case of finite groups. With Lie groups, like those unitary ones, you need big heavy machinery, like the Haar measure, whereas with finite groups, a lot of things you would use the Haar measure for become a simple matter of taking an average over the group. So, the point is, the stuff you learn about in a basic abstract algebra class provides good background knowledge, even if you don't use it directly.
I'm not a physicist, but it's my hunch that knowing more about group theory would be helpful if you want to have a deeper understanding and go beyond blind calculations.