It may be helpful to first think about Fourier series expansion of a square integrable function ##f## on a bounded interval ##(a,b)##, in terms of an eigenbasis ##\{\phi_n\}## of a (possibly unbounded) linear operator ##A## on ##L^2(0,1)##. In this case, the ##n##th Fourier coefficient gives the contribution of ##\phi_n## to this expansion, and the spectrum of ##f## would be the square-summable sequence ##\{(f,\phi_n)\}## of Fourier coefficients. (Here ##(\cdot,\cdot)## denotes the inner product in ##L^2(0,1)##.)
In this case it is natural to speak about "spectrum of ##f##", because of the reference to an eigenbasis of ##A##: If ##A## is a matrix, then its spectrum is - by definition - precisely the set of its eigenvalues. For the case of the Fourier transform on the (unbounded) real line, the eigenbasis is no longer discrete, sums become integrals, and the situation is more delicate than on bounded intervals, but the terminology has remained the same.
It is a good question, by the way.