Your assesment of difficulty is "they both use trigonometry"? What about Calculus or Differential Equations? What about thermodynamics and Relativity? I'll bet they both require each of those also. The basic fact is that MOST of the course work for physics and astronomy overlap (in fact, in most colleges "Astronomy" is a subdepartment of "Physics"). But frankly, you choice of what to major in and, hopefully, to spend your life working in, should not be based on "which is easier"!
(As I wrote that I suddenly remembered that a major reason why I majored in Mathematics rather than Physics was that Physics majors were required to take Physical Chemistry and I hated chemistry!)