I think that there certain concepts that only exist in the human imagination, they don't exist in reality. One of them is the concept of infinity. Nothing is infinite. As we learn more and more almost everything that we ever thought was infinite has turned out not to be. When infinity creeps into math it is usually a sign of a mistake isn't it?
Time travel to the past would require infinity in various forms. I believe that for this reason alone it isn't even remotely possible.
It would require every possible (as opposed to the actual configuration that supposedly could be deduced at the end of time) configuration (position, energy, etc.) of every particle in the universe at every instant from the big bang to the end of time to exist simultaneously. The actual configuration is the way things are without time travel, the possible would encompass changes that would inevitably occur with travel to the past, it WOULD change the future.
In actuality there may not be that much difference between the 2 since the only thing that can change anything from it's normal path as governed by the laws of physics is life. Life introduces a random factor so anything involved with life can't be deduced by the laws of physics. You can be a physics god but the laws of physics will not explain the existence of a lunar lander on the moon or one particular crater on a certain comet.