Since the field extends outward from the magnet, the question relates to the ability of the physical magnet, as it begins to accelerate, to instantaneously communicate that acceleration to its field lines at the conductor a distance s away: there must be a delay (Δt≥ c/s) before the conductor feels the change in magnetic field and the resultant induced electric field; and since there is that delay, there must be a similar delay in the magnet feeling the opposing magnetic field from the induced current in the conductor. That seems to be the question.
Feynman studied the interaction of fields of approaching electrons. The effect should be similar. He discovered that it did not matter whether you assumed that the field travelled in advance, was retarded or was instantaneous. The reaction force was always the same. You may wish to google the "Wheeler-Feynman absorber theory"