Choppy said:
You can't go wrong with business casual. For guys this would be some kind of dress shirt, khakis, and shoes that aren't runners. Tie optional.
Most faculty probably won't care too much either way.
I dunno, one professor commented to me that a prospective student "looked a bit too fancy" when he was wearing jeans, a dress shirt and niceish shoes.
But I agree that most academics won't care.
To give some more substantive advice: Presumably, if you're applying for grad school, you've interacted with academics in your chosen field before. Dress like that.
Dishsoap: For women, we're often more judged on what we wear. For physics, and if you're doing lab tours, sensible shoes, pants or a nice (mid length) skirt (I'd say pants were more expected in physics, but I also don't think we should be advising women not to wear skirts, so). A top that's nice but also professional. I'd not wear runners, presumably you have walking shoes that aren't runners? But if you don't, I'd go for runners rather than limping!
But really, if you're doing engineering or physics, I don't think this matters much at all. Don't wear a suit or a tie, don't wear a t-shirt that's full of holes. Anywhere in-between will probably be ok.
Note: This is also geographically variable.