Magnetic field could only affect moving charges?

A current carrying wire can experience a force given by Biot-Savart law.
A current carrying loop under magnetic field can experience torque which comes under magnetic moment.
$$\tau = M*B$$
But even then in both cases there is a movement of charge(current).

 

Yes.
Incase youre wondering, what we normally consider a "magnet" is a little complicated
Normally electrons orbit around the nucleus in random configurations, while a permanent ferromagnet has electrons all orbitting around the same axis.
Essentially the force we normally experience as magnetism is the interaction between the electrons, that are moving(even though the atoms appear stationary)