Physics isn't harder than chemistry. No science is intrinsically more difficult than any other, they're just difficult in different ways. So the difficulty depends on your own predisposition as well as interest and attitude. Physics involves more mathematics and mathematical thinking than chemistry, which requires more than biology. So if you view maths as a 'necessary evil' then maybe chemistry is better for you (all else being equal). If you like math and it comes easy to your, then physics is probably better. Again, that's if all else is equal. In the long run, aptitude for a subject won't compensate much for a lack of interest.
Job opportunities depends mostly on what your specialization is. As far as academia is concerned, there's not a lot of difference with either physics or chemistry. As far as industrial/private-sector is concerned, I'd say chemists generally have more opportunity to find related jobs in industry. But that also depends on where you live and what your local industries are, and again your specialization. A nuclear physicist obviously has more industry jobs than a theoretical chemist, who's still probably better off than a string theorist, etc.
Either way, if you want to do research in either field you'll need a PhD, unless you intend to work more at the engineering end of things, so you probably have a while to make up your mind if you're only picking a major now. I was once told that when it comes to e.g. organic chemistry that while you could do research in industry with an M.S., all the 'fun' problems required a PhD. I wouldn't make up my mind until I'd studied the stuff. A topic that might sound fun to you might turn out to be boring in its details and how it really is, and vice-versa. If you're genuinely undecided between physics and chemistry, you might want to take courses in stuff that can be applied in both fields to keep your options open, e.g. physical chemistry/chemical physics.
And in particular: mathematics. Insufficient math skills is a deal-breaker when it comes to continued study in physics or physical chemistry and it's still useful in other areas. In my experience it can make you more competitive, even, since chemists with physicist-level math skills are unusual. In other words: A biochemistry course counts for nothing if you want to go into physics, but a math course still counts for something if you want to go into biochemistry.