One way to prove the order of the composition matters is to pick a particular point in the plane, and compare the results of doing it "both ways".
So let's start with the point $(x,y) = (\frac{\sqrt{3}}{2},\frac{1}{2})$.
Note that this is just the intersection of the line (well, "ray" actually) at an angle of 30 degrees with the $x$-axis, and the unit circle.
If we reflect this about the line $y = x$ first, it becomes the point $(\frac{1}{2},\frac{\sqrt{3}}{2})$. This is the intersection of the ray at an angle of 60 degree with the $x$-axis and the unit circle.
So, if we rotate next 60 degrees counter-clockwise, we should wind up with the intersection of the ray at an angle of 120 degrees with the $x$-axis, which is:
$(-\frac{1}{2},\frac{\sqrt{3}}{2})$.
Ok, now let's do it the "other way".
First we rotate by 60 degrees, which takes us to the intersection of the unit circle with the y-axis on its upper half (30 + 60 = 90), which gives us the point $(0,1)$.
Next, reflecting (which just changes the $x$ and $y$ coordinates) gives us the point $(1,0)$, which is different than what we got applying the maps in the other order.
In general, in "most" mathematics, when "multiplication" is actually a kind of composition, then:
$A \circ B$ means: FIRST do $B$, THEN do $A$:
$ABv = (A\circ B)v = A(Bv) = A(B(v))$:
$v \to Bv \to ABv$.
Now, let's use matrices, and see if we get the same answers as I got above.
Let $B$ be the reflection (since the problem says to do that first). You have correctly given this matrix as:
$B = \begin{bmatrix}0&1\\1&0 \end{bmatrix}$.
Therefore, we have:
$B(\frac{\sqrt{3}}{2},\frac{1}{2}) = \begin{bmatrix}0&1\\1&0 \end{bmatrix} \begin{bmatrix}\frac{\sqrt{3}}{2}\\ \frac{1}{2} \end{bmatrix} = \begin{bmatrix} \frac{1}{2}\\ \frac{\sqrt{3}}{2} \end{bmatrix}$
So far, so good.
You have also correctly given the rotation as:
$A = \begin{bmatrix}\frac{1}{2}&-\frac{\sqrt{3}}{2}\\ \frac{\sqrt{3}}{2}&\frac{1}{2} \end{bmatrix}$
So:
$AB(\frac{\sqrt{3}}{2},\frac{1}{2}) = A(\frac{1}{2},\frac{\sqrt{3}}{2}) = \begin{bmatrix}\frac{1}{2}&-\frac{\sqrt{3}}{2}\\ \frac{\sqrt{3}}{2}&\frac{1}{2} \end{bmatrix} \begin{bmatrix}\frac{1}{2}\\ \frac{\sqrt{3}}{2} \end{bmatrix} = \begin{bmatrix}-\frac{1}{2}\\ \frac{\sqrt{3}}{2} \end{bmatrix}$
Since this agrees with what I found above, it appears we are on the right track.
Now, the "all-in-one-fell-swoop" matrix for the composition, should be $AB$, which is:
$AB = \begin{bmatrix}\frac{1}{2}&-\frac{\sqrt{3}}{2}\\ \frac{\sqrt{3}}{2}&\frac{1}{2} \end{bmatrix} \begin{bmatrix}0&1\\1&0 \end{bmatrix} = \begin{bmatrix}-\frac{\sqrt{3}}{2}&\frac{1}{2}\\ \frac{1}{2}&\frac{\sqrt{3}}{2} \end{bmatrix}$
Let's verify as a sanity check that this takes our chosen point to the same end result as "doing the steps one at a time":
$AB(\frac{\sqrt{3}}{2},\frac{1}{2}) = \begin{bmatrix}-\frac{\sqrt{3}}{2}&\frac{1}{2}\\ \frac{1}{2}&\frac{\sqrt{3}}{2} \end{bmatrix} \begin{bmatrix}\frac{\sqrt{3}}{2}\\ \frac{1}{2} \end{bmatrix} = \begin{bmatrix}-\frac{1}{2}\\ \frac{\sqrt{3}}{2} \end{bmatrix}$
which is exactly what we got before.
As an aside, note that:
$BA = \begin{bmatrix}0&1\\1&0 \end{bmatrix} \begin{bmatrix}\frac{1}{2}&-\frac{\sqrt{3}}{2}\\ \frac{\sqrt{3}}{2}&\frac{1}{2} \end{bmatrix} = \begin{bmatrix}\frac{\sqrt{3}}{2}&\frac{1}{2}\\ \frac{1}{2}&-\frac{\sqrt{3}}{2} \end{bmatrix}$
which is NOT the same matrix as $AB$, so we should not expect it to have the same effect. Indeed, using our same "test point", we find that:
$BA(\frac{\sqrt{3}}{2},\frac{1}{2}) = \begin{bmatrix}\frac{\sqrt{3}}{2}&\frac{1}{2}\\ \frac{1}{2}&-\frac{\sqrt{3}}{2} \end{bmatrix} \begin{bmatrix}\frac{\sqrt{3}}{2}\\ \frac{1}{2} \end{bmatrix} = \begin{bmatrix}1\\0 \end{bmatrix}$
which again agrees with our "non-matrix" analysis at the beginning of this post.
Again, I want to stress that it is NOT customary to do rotations or reflections first, but rather, if you are instructed to apply $B$, then $A$, the matrix you want is going to be $AB$, not $BA$ as you might think from "reading left-to-right" (in other words, the composition of two functions $f\circ g$ is "order-reversing", we do $g$ first, not $f$).
