I do not think that I am avoiding the underlying geometry. I am trying to understand how the underlying geometry of the worldline of the ball (in the position of event A from the graph) is straight. Doesn't it seem at least counterintuitive to you that the position of the ball can change on the x' coordinate without causing a curve in its worldline? This is what I do not understand; it is all about the underlying geometry for me.The graph is not real. I am not even sure what would make you think that a graph is real.
I think that problem that you are having is that you are overly focused on coordinates. That will work reasonably for inertial topics, but as soon as you start trying to think about non-inertial topics you need to de-emphasize the coordinates and focus on the underlying geometry. If you are unwilling to do that then you may need to just stick with inertial topics only.
Also, I am curious about why you are unwilling to engage about the geometrical discussion. That is the most important part of this topic and you have seemed to ignore it completely. Why?
Do you understand the concept of a physical geometry, independent of any coordinates?
If I physically draw a line on a piece of paper, do you need coordinates to determine if it is straight?
If a builder built you a crooked wall and tried to show you a graph of the wall in some coordinates where the wall had a constant coordinate, would you then think that the wall was physically straight?