Actually it's better if the portal allows you to get somewhere FTL (FTL from the point of view of light traveling the "long way", of course you still move slower than any light which travels through the wormhole with you). If the regions of spacetime the portal connects have what's called a "space-like separation", meaning that no signal traveling the regular way could get from one to the other without traveling FTL, then there's actually no danger of time travel here. For example, say there's a star 100 light years away and traveling through the portal on Earth in 2000 would cause you to step out at the location of that star in 1950 (with time being defined relative to the rest frame of the Earth and star). Sure you've gone back in time, but if you send a light signal back towards Earth it won't actually reach them until 2050, and if you step back through the portal in the other direction it takes you 50 years into the future, so you're back at Earth in 2000 (or a little later if you hung out at the star for a while).
On the other hand, suppose there's another star 20 light years away, but stepping through the portal on Earth in 2000 takes you to the star in 2040. In this case the separation is "time-like", meaning that stepping through the portal won't get you to the star faster than a light beam would. It might seem like there's no problem here, but the portal is two-way, meaning if you step back through the portal at the star in 2040, you'll end up at Earth in 2000, and in general if you step through the portal at the star in year Y you'll end up at Earth in Y-40. So now say in 2000 you get in a rocket which flies to the star at 0.8c, covering the 20 light years in 20/0.8 = 25 years. This means you'll arrive at the star in 2025, so if you step through the portal you'll now be on Earth in 2025-40=1985, in your own past!