The difference mostly has to do with mathematical techniques and general sophistication. For example intermediate E&M involves more advanced techniques like vector calculus and basic partial differential equations, graduate E&M requires complex analysis, Green's functions and advanced PDEs.
Sophistication is more difficult to explain but equally important, this could roughly be described as what the textbook author considers to be obvious and what is considered to be worthy of explanation.