One aspect will be the supported floating point formats. Once upon a time, GPUs only supported float (32-bit), then came support for double (64-bit), but with "much" less speed than float, at least compared to the same difference for a CPU.
Todays GPUs support FP32, FP16, FP8, INT8 efficiently, + FP64, and other common CPU integer types, albeit less efficiently. Already this part is slightly annoying and challenging, when scientific computation software developped on CPU gets ported to GPU, because those details suddenly get much more important for overall performance than they were on CPU.
TPUs don't even bother to support FP64, and are most efficient for strange floating point formats like bfloat16. They have a wide dynamic range, but limited precision in the sense of "classical" numerics theory. Actually, because the numerics theory I learned at university is unsuitable for them, I probably won't use them for scientific computing. I simply I don't have the time to build-up new intuitions for that type of limited precision provided by bfloat16.