How do you know the field isn't actually there? Keep in mind that a particle is no less mathematical or conceptual than a field is. And if the field isn't there, what happens to the energy lost from radiating particles before the radiation is absorbed by another particle? Is that energy simply gone? If the field doesn't exist, then why do field theories have such success over purely particle theories?
A field allows us to answer questions like this by treating the field as an actual entity and not just clever math. (of course one could argue that all math is "clever math" if it enables us to accurately describe reality, whether or not that math is for particles or fields)