A possible way to reconcile time travel, with the fact that we don't seem to have had any visitors from the future, would be to say that it can occur only in the future. In this view, one would say space-time in our past was fixed, because we have observed it, and seen that it is not warped enough, to allow travel into the past. On the other hand, the future is open. So we might be able to warp it enough, to allow time travel. But because we can warp space-time only in the future, we wouldn't be able to travel back to the present time, or earlier.
This picture would explain why we haven't been over run by tourists from the future.