The aluminum may increase the dissipation. Eddy currents will be induced every time the magnetic field changes. This change will happen when the plunger moves in the tube, for example.
Think about the experiment with a magnet falling in a copper or aluminum tube. The steel core will be magnetized, so it is a magnet moving in a conducting tube.
The damping may be a desirable effect, after all. It depends on the application.
See this, for example:
Your tube has much thinner walls, I suppose. So the effect will be much weaker.
But some dissipation will be there so why not avoid it, if you have no special reason for aluminum?