Well you can see that the magnetic field doesn't do work simply from the Lorentz force, the magnetic field requires a velocity to exert a force and then the force it exerts is always perpindicular to the direction of motion. That's a rather cheap way out though, in my opinion.
Keep in mind, however, that when you change refrence frames the magnetic and electric fields also change. If you have a frame in which the field is entirely magnetic you can then transform into a frame moving relative to the first in which the field is both magnetic and electric. Combine this effect with the fact that a magnetic field can instigate charge separation and thus create electric fields and, as El Hombre mentioned, you can very quickly get some amazingly complex situations in which there are multiple sources of magnetic and electric fields which are tugging every which way on a particle.
It's good that you question these statements, and there are likely some simple cases that you will be able to work out, such as the two wire example given, but there will also be situations that will simply be too complex to get a grip on. In the end I think it's best to be comfortable with the reasoning and mathematics behind the fields and their interactions with the particles, then you can trust the mathematics to tell you the truth.