Funny, I am in EXACTLY the same position. Graduated with a BS in CS, worked for a couple of years and now want to pursue a Physics PhD. My undergrad was similar to yours as well, taking a few physics classes and quite a bit of math.
What I decided to do is take a class or two each semester...