Before the Rutherford experiments, it was very popular to believe that an atom consisted of dot-like electrons suspended in a spread out positively charged cloud like a plum pudding (this being known as the plum pudding model of the atom).
If that were the case, then the alpha particles would mostly pass through or get deflected by small angles, because the only hard objects to scatter off of would be the tiny (and relatively light) electrons, which were much lighter than the alpha particles.
Since some of the alpha particles were actually deflected at large angles, the model of the atom was revised to be a small positively charged nucleus surrounded by orbiting negatively charged electrons.
Interesting side question:
At the time that those experiments were done, did scientists understand alpha particles to be charged nuclei of helium, or were they just thought of as different particles all their own?