There are several observational facts that mean that electromagnetism is not enough.
To start with, electromagnetism is not strong enough to bind two quarks of opposite charge as tightly as a nucleus or a meson. Instead, you would get something of a size similar to an atom, which is about five orders of magnitude larger.
Second, electromagnetism could not produce a proton, which has a net positive charge. Instead of being an energetically favoured state, the proton would not constitute a bound state and we would not exist.
Third, electromagnetism cannot explain several other effects that has to do with quarks, such as the color factor appearing in meson production and similar things.