I'm a computer engineering major and I'm a senior at UCF. I might be able to give you some pointers.
First off, why do you want to do computer engineering? To design computers? Well, let's see what goes into that.
Computers engineering often has a lot to do with microcontrollers. Embedded systems often has you programming a microcontroller in a low level language such as C or assembly.
Computer architecture and organization classes might interest you, since it involves the design of computer systems and you can choose to focus on things like performance and throughput.
When you're designing digital circuits, you may end up using a programming language such as VERILOG to design them. It's actually called a hardware description language.
You will have to become a pretty good programmer in the four years of taking computer engineering. Those were the most "hardware" courses I could think of, and they still involve a little bit of programming. There are plenty of intense programming courses such as System Software, Operating Systems, Software Engineering, etc.
If you think you wouldn't mind taking a little bit of programming, but you really want to design hardware, I would point you towards electrical engineering. It's a big field, involving lots of math, physics, and electrical hardware (digital & analog). You would never be asked to program an operating system as an electrical engineer, but you will definitely have classes that involve designing electrical hardware.
The point is this: As an electrical engineer you can choose to specialize in computer systems, and you won't be forced into taking all the software classes that you don't like. It may take until your senior year to be able to pick the electives you want, but I think it'd be much better for you.
You will have to learn lots of things that are not directly related to computers (analog circuits, semiconductors, power systems), but they are indirectly related and they lean towards the side of more physics, less software.
It's all a spectrum, and you could probably get away with a CE degree, but in my opinion you wouldn't like it.