Gauss's law and spherical symmetry make it zero. But yes if you try to think in terms of the Coulomb's law (like the blue arrows you draw suggest) it is not so obvious.
The intuitive approach to explain it using Coulomb's law is that the part of the conductor that is closest to the point exerts greater electric field but it is smaller in length than the part of the conductor that is furthest apart and hence exerts smaller electric field. The two electric fields cancel out. It is something like $$L_1D_1-L_2D_2=0$$ where ##L_1##,##L_2## the lengths of the bigger and smaller part and ##D_1## and ##D_2## the influence of the bigger and smaller part. The bigger part has smaller influence and the smaller part has bigger influence. So it is ##L_1>L_2## but ##D_1<D_2##. All these are very intuitive though, if you want a formal treatment of this using coulomb's law
you have to do a hard integral to prove that it is zero at any point in the interior of the sphere.
Yes check
@Ibix link for a formal treatment. It is for gravitational field but I guess you get it that it is pretty much the same thing, the same inverse square law holds for electric and gravitation field, in one case we call it Newton's law and in the other case Coulomb's law.