Possible? Sure.
But not without a whole lot of work.
A lot depends on the details of your undergraduate degree, which courses you've taken as options, and how flexible the program is that you're applying to. But as a general rule, you need the equivalent of an undergraduate degree in physics. This means that you need to have completed senior undergraduate courses in core topics like classical mechanics, statistical mechanics and thermodynamics, electrodynamics, quantum mechanics, and mathematical methods for physicists. Ideally you would also have taken a senior level introductory course in the sub-field you want to specialize in, if it's available.
I don't know the details of an architecture degree at your school, but I suspect it's unlikely for graduates to have covered all or even a substantial portion of that material. What that means is that you would likely have to go back and fill in the necessary courses. Programs can offer some leeway for exceptional students, but in most cases there are only a fixed number of positions and the queue of applicants is full of exceptional students who all have the necessary coursework.