It depends how far you want to go I guess. To understand the fundamentals, you won't need much more than basic algebra, like the others said. If you go into more advanced formulations though, you would need to understand the concept of 4-vectors and tensors, which are considerably more difficult.