Time dilation in special relativity is a function of velocity, not acceleration. It is true that if two observers cross paths twice at different points in spacetime, and one has moved inertially between the two meetings while the other accelerated at some point during the journey, than the one who accelerated will always have aged less.
deals with naked singularities
, maybe you're thinking of the chronology protection conjecture
If your grandfather didn't have your parents until after the date wormholes were taken on the relativistic trip and brought back together with a time difference between them, then you certainly could meet your grandfather before your parents were born. Even if this type of time travel were possible, it doesn't necessarily mean you could succeed in killing him, perhaps the Novikov self-consistency principle
would apply and it would be impossible to change the past. But grandfathers aside, this type of time travel is certainly the kind that would "violate causality" in the sense meant by physicists, where it is no longer true that we can assign a temporal order to events in such a way that causes always precede effects...for example, you could wait until the results of a lottery drawing were announced, then travel back one day before that and tell your younger self to buy a ticket with those numbers...even if there is no paradox created here, the cause of your younger self buying the ticket with those numbers lies in his own future, so this is a causality violation. And this form of time travel would definitely contradict the "chronology protection conjecture" (the most common idea about how chronology protection would be enforced in the case of wormholes involves virtual particles which build up in a feedback loop as you try to move the two ends close enough to allow for backwards time travel, either destroying the mouths before they can be moved close enough or causing a repulsive effect...see here
for a discussion, and for a more detailed discussion read the last chapter of Kip Thorne's book Black Holes and Time Warps
Again, you misunderstand what "causality" means, time travelers from Earth today going back to Earth 4 billion years ago would automatically be a causality violation, even if something like the Novikov self-consistency principle held sway and this trip did not lead to any "changes" to history.
Since we don't have a theory that combines general relativity with quantum mechanics it's not at all clear what such a theory would say about the GR solutions that seem to allow backwards time travel, like spacetimes involving wormholes. Probably the most popular hypothesis is that quantum gravity will rule out these loopholes and chronology protection will hold true. But if backwards time travel is possible in quantum gravity, there's no reason to think it would necessarily
lead to the possibility of changing the past, it might be that even though there are multiple possible histories as in the MWI, the Novikov principle applies to each individual history so time travel does not allow anyone to visit a different history than their own.