Quote by janismac
I set up the equations for the conservation of momentum and conservation of kinetic energy.

How have you used the angle of impact parameter? There are (as you've just discovered) multiple solutions that conserve momentum and energy; they correspond to different angles into and out of the collision.