Of course. You can certainly apply to grad school in nuclear engineering after a BS in physics. This is a fairly common path. You will have some catching up to do to learn nuclear reactor physics and engineering, but this is very possible.
It depends on what you want to specialize in. If reactor engineering, then it's going to be a hard sell to the committee. If it's detector design, as is all the rage at Berkeley and Michigan, for example, then it should be straightforward
I don't think it's a "tough sell". Graduate schools don't expect you to be experts in their field before you come in - they expect you to have learned the basics so you can succeed in their graduate level courses. As a representative test, I looked at UC Berkeley's admission requirements for their graduate program in nuclear engineering. Here is what it says:
"Admission to the graduate program in nuclear engineering is available to qualified individuals who have obtained a bachelor’s degree from a recognized institution in one of the fields of engineering or the physical sciences. For all programs, required preparation in undergraduate coursework includes mathematics through partial differential equations and advanced analysis, nuclear reactions, and thermodynamics."
If you have done well in you undergrad physics work, I suspect you will be just as likely to be admitted as someone who was a nuclear engineering undergrad. I know several people who followed the physics undergrad -> nuclear engineering grad career path.