Let G be a group. Let GL(C,n) be the group of invertible nxn matrices with entries in C. You know what all these things mean. Let p be a homomorphism (you konw what a homomorphism is) from G to GL(n,C). This is a representation of G. It is a way of making G act by matrices on some finite dimensional vector space. (We can do it for infinite dimensional spaces too but I'd rather not explain those. not because they are harder but because they come in two flavours and i wouldn't want to talk about the wrong one. I'm sure we'll see which is more useful to you later; probably you'll care about unitary representations if at all).So, to recap, to understand representation theory you need to understand
G a group
GL(n,C) the invertible nxn matrices (also a group)
p:G-->GL(n,C) a group homomorphism.
the map p is the representation (some times we refer to C^n as the representation and leave the p implicit)
which of those is confusing?
SO(2) is a group. There is a natural and obvious representation where we identify an element of SO(2) with itself inside GL(2,C)
We can also create a map p:SO(2)-->GL(4,C) by making the matix act on the pair of vecotrs (u,v) (Ie think of u and v as vectors in C^2) by p(g)(u,v)=(gu,gv)
the tiitle "representation theory" is a very big one. It encompasses a great many things in (predominantly) algebra.
A representation of some object (be it a group or whatever) is a way of realizing this thing as a set of matrices (though we possibly lose information, like in the trivial representation of any group)
as long as you come away with a simple idea that a rep of a group is a way of interpreting the group as matrices (ie making it a concrete set of transformations of some vector space) then you're more than halfway to understanding what a representation is.
The best answer is to buy this book:
or if that link is broken representations and characters of groups by James and Loebeck. that is the hard cover and is stupidly priced, try to find the soft cover or get your library to buy it.
i will try and answer more questions over the weekend if tyou can come up with a list of queries that are bothering yuo