It's too broad and subjective of a topic to make any statement that's definitive on the matter.
However, I can see why a lot of people might say this. First year physics is generally a single introductory course. For many students a big part of it is review. Also, it's taken by a large number of students, so you generally have some anchors is the class to keep the grade distribution reasonable.
By second year, you start on what I would consider the more rigorous fundamentals. You take multiple physics classes that are now topic-specific, go through labs that are topic-specific, and you're largely in them with other physics majors - most of whom have chosen to be there because they're good at it and did very well in first year. I think it's also where the typical student begins to learn proficiencies with skills that aren't specifically taught, but the student is expected to know (examples include picking up a programming language, writing a report, finding various resources in the library, familiarizing oneself with an environment like MATLAB, using a spreadsheet for more than just addition and subtraction, etc.). It also seems to me like it's not uncommon for many students to rely on high school advanced placement courses as substitutes for first year courses. Thus, by the time they get to their actual second year, they're taking senior-level classes before they're ready.
There's also a professor-factor to consider too. Because lot's of students take the first year physics class, departments like to put a 'best foot forward' and place instructors into those positions that are good at what they do. (This isn't ALWAYS the case.) But by second year you're likely to have at least some instructors who are stronger at the research side of the profession than at the teaching side, which can be challenging.
On a social level there is also a little less hand-holding that goes on. In first year many schools will provide you with on-campus housing and even a meal plan that means your most difficult decision outside of class is to go with the burger and fries or the pizza. By second year, in my experience anyway, many students moved off campus. You no longer have a frosh rep (or orientation leader as I believe they're called today) to go to for guidance or to help when you don't get along with a room mate. You have to commute, and you have to shop for your own groceries. Lots of students also put on the 'freshman fifteen' which can diminish your energy level. And depending on the local laws, that can be the year where it becomes legal for you to drink - not that that ever affected anyone's studies. There can also be additional financial stress too. Lots of scholarships are money for a single year, or a one-time-payout based on high-school marks. At the start of the second year that's dried up.
So, yeah, I could see why someone might argue second year is the hardest.