Quote by Kyouran
So here is my question: Can we actually speak of "the number of elements in a set"?

Yes.
You might consider comparing the notion of "set" to those of "multiset" (sometimes called "bag") and "tuple". Sets don't allow duplicates ({1} = {1, 1}) and don't consider order ({1, 2} = {2, 1}); multisets allow duplicates ({1} ≠ {1, 1}) and don't consider order ({1, 2} = {2, 1}); tuples allow duplicates ((1) ≠ (1, 1)) and consider order ((1, 2) ≠ (2, 1)).