Another way to think of it is that time is always in one direction, so there is no need to mention a direction for it. Or you could look at it mathematically, for time to be a vector it should hold under vector addition, but when your adding time "vectors" its really just the addition of individual times. So its just scalar addition and hence a scalar quantity.