Finny said:
So it seems there is something, indeed, fundamentally distinct between electromagnetic and gravitational potential.
Thanks for your efforts to explain.
I haven't read this whole thread, but I'll point out one thing about electromagnetism and gravity that is obviously different and not too technical - at least not on the surface. Gravity affects everything, while electromagnetism only affects objects with electric charge.
Even this simple observation turns out to need quite a bit of clarification and nit-picking. For one thing, we're only talking classically. If we wanted to cover the quantum realm, we'd have to properly categorize things like the Casimir force, something I'm not going to attempt in detail as my focus is classical.
The next point is that while uncharged objects don't experience any classical electromagnetic forces and do experience gravitational forces, charged objects experience both. Furthermore, the charge itself can (and does) have gravitational effects. So the conceptual separation is one-way, uncharged objects have gravity and not electromagnetism, while charged objects have both.
The second point is this - when a force affects literally everything, we can regard the force geometrically in a very natural manner, because geometry affects everything. Attempts to treat electromagnetism geometrically do exist - they haven't worked out for the most part, and it's more difficult (for me, at least) to imagine how geometry can affect some objects and not others.
Now, the third point. Gravity in general relativity causes effects other than forces, effects which are widely regarded and described as being geometrical in nature. For some obvious examples, consider gravitational time dilation. Electromagnetic forces do not cause time dilation (except insofar as they may cause additoinal gravitational effects). Time dilation is caused by gravity, and not by electromagnetism. Time dilation does not have a "force-based" explanation, it's a different class of effect.
Other geometrical effects exist in general relativity. The circumference (or surface area) of the space around a massive object can not be expressed as 2*pi*r where r is the radial distance. What's typically done is to define a coordinate "r" that describes a sub-space with a circumferene of 2*pi*r, and a surface area of 4 pi r^2. Given this definition of the r-coordinate, we then observe that the distance between two points is not dr, but a more complex expression, given by the Schwarzschild metric. We can summarize these observations conveniently by saying that the spatial slices of the Schwarzschild geometry are "curved".
To go further into why we say gravity is curvature, we'd need to go into what curvature is. Given that the OP has mentioned the Weyl tensor, I assume they have at least some exposure to the idea of curvature, though from the tone of the question I'm guessing there is still some uncertainity on their part about just what curvature entails. Regardless, I'm not going to attempt much in the way of a serious explanation of just what curvature is. What I will do is point out that GR is fundamentally based on this concept, in that the Riemann tensor is the mathematical description of curvature. Furthermore, the Einstein's field equations relate a tensor derived from the Riemann curvature tensor to a second tensor that describes the distribution of matter (the stress-energy tensor, which describes the distrubtion and density of energy and momentum).
Why the stress-energy tensor is the "right' tensor to describe the distribution of energy-momentum is also something I'm going to gloss over.
Underlying the concept of the stress energy tensor is the notion of why we talk about space-time, rather than space and time. In Newtonian physics, we talked about space, and time, as if they were separate. The origins of why we talk about "space-time" rather than space-time occur in special relativity. I think it would be too much of a digression to try to say why in this point we do this - and this post is already too long. But if it's not an understanding shared by the OP, it might be worth asking about this question in a separate thread if there is some interest.