The electric and magnetic fields are always normal to each other for an electromagnetic wave. In a source-free homogeneous region of space, the electromagnetic waves can only be transverse waves. This means that the electric and magnetic field vectors are both normal to the direction of propagation. So, if we specify the vector direction of the electric field and propagation, then we have implicitly specified the direction of the magnetic field. By convention, we generally choose the electric field as the direction of reference for polarization.
Electric and magnetic fields are both vector fields. Not only do they have a magnitude and phase associated with each point in space, but they also have a direction. This direction is related to the direction that a test charge would travel from the force induced by the fields. For example, the Lorentz force on a test charge due to the electric field is simply
[tex]\mathbf{F} = q\mathbf{E}[/tex]
So that is why there is a direction associated with the fields (the relationship with the magnetic field is a bit more complicated, the cross product of the particle's velocity and magnetic field).
So the polarization of the electromagnetic wave is a way of describing the orientation of the electric field vector component of the wave as the wave propagates through space and time.