Sorry I am out of practice with TeX
E = -grad(V) + A'
where A is the vector potential and prime denotes derivative with time. Look for a discussion of retarded potentials to see how A propagates with time, but suffice to say it does, and whatever changes happen to A at a particular point d distance away from me will in principle be felt after a duration t = c/d, where c is the speed of light.
Edit: If B changes at a single point, then that change will eventually propagate away, but your argument assumes that B can change at one isolated point only, when this is impossible under the assumption of continuous fields i.e. the differential form of Maxwell would not apply at that point of discontinuity. Granted, discontinuities can occur in the theory of Maxwell's equations n matter, but I am not qualified to give an answer that takes matter into account.