I think your reason may be based on an unfounded assumption; but there are plenty of problems...
You're supposing that my sudden appearance in a past time is a problem because there was a certain correct configuration of things at that time, and my sudden presence there does not correspond or fit to that time's proper configuration... the particular past time I was supposed to be going to can't be the one I arrive in because I'm not supposed to have been there. My presence will have changed it. I'm displacing some space that would have been other wise occupied, and otherwise influencing things with heat and pressure and chemicals that would have not been there otherwise, not even to mention all the mischief my actions might provoke...
But is there really that otherwise uncontaminated proper past time?
If I did manage to go back in time, then maybe that "otherwise uncontaminated proper past time" never existed... my appearance is and always has been a proper part of that time. And wouldn't my sudden disappearance from the present when I begin my trip raise all the same concerns about the "otherwise uncontaminated proper present time" as my appearance in the past?
Once you begin to talk about changes in time ( the "otherwise uncontaminated proper past time" was altered when I went back so as to make a "new" version of past time), then conceptually you have to add another meta-time under which all the possible changes in time happen... and then you have to add another meta-meta-time layer to order the variations in the meta-time layer below... the whole thing stacks up and diverges wildly when you insist on changes in possible times.
So that leaves us back with me showing up by surprise in the past... do we need backward causality to account for this?
The backwards causality problem is that we attribute causality to our past; the time traveler has a more complex version of this problem... before his travel back in time, he has a past full of causes and memories, but at the point where he arrives in the past, all those causes are now in his future, but may still be in his mind.
If the traveler keeps his memory, he know his future when he arrives in the past, yet when he lived that span before traveling he did not know his future... this is another contradiction that requires another layer upon layer of meta-time hierarchy that diverges.