Then we have, [itex]x * g_{y}(x) = y*g_{y}(x)[/itex], so it is indeed an eigenfunction of the x operator. And it happens to be a normal function. So why does he say it's 'nothing but Delta'?

you are completely right. The delta function is not an eigenfunction of the operator x in a mathematical sense; your function is. The problem of your function is that is has zero norm, so it does not fit into Born interpretation.

The delta function does not fit either, but textbooks use it due to wish to simplify life and treat every operator in the same way as the Hamiltonian, in particular to write abstract formulas

This Dirac method is not perfect and was criticized, because it suggests something that is not mathematically true. Also, it easily leads to mistaken interpretations of calculations.
The good part is, one can avoid this formalism, just by sticking to normalizable states, but it requires some effort (so it is not popular).

This is also not correct. It's correct that the generalized eigenvectors of essentially self-adjoint operators on Hilbert space are not part of the Hilbert space but of the dual of the smaller subspace, where this operator is defined. It's a distribution. All this is mathematically well developed. A mathematical rigorous formulation, which is very close to the physicists' handwaving approach is known under the name "rigged Hilbert space". You find a nice exposition of this methods in the textbook by Galindo and Pascual.

vanhees71 is completely right about the mathematics of generalized eigenfunctions. Of course, the delta distribution and other generalized eigenfunctions, like the exponential function on R, are useful in QT.

The problem with the Dirac symbol [itex]|x_0\rangle[/itex] I meant is that it misleads one to regard [itex]|x_0\rangle[/itex] or [itex]\delta(x-x_0)[/itex] as [itex]\psi[/itex] functions obeying the same rules and having the same probabilistic meaning as the normalized functions, (say, of the hydrogen Hamiltonian). But the role of the delta distribution or the other generalized eigenfunction, like [itex]e^{ikx}[/itex], is different; they do not play the role of the [itex]\psi[/itex] function themselves, but they can be used to express the [itex]\psi[/itex] function only when under the integral, like in

or
[tex]
\psi(x) = \int c(k) e^{ikx} dk/(2\pi)
[/tex]

I have found useful comments on the use and meaning of delta function in QT in Landau & Lifgarbagez, Sec. 6 (these authors do not use the Dirac notation) and in the paper 29-1 "On "Improper" functions in quantum mechanics" in Fadeev, Selected works, V.A. Fock, Chapman&Hall, 2004; perhaps they will be useful to you.

This is indeed a very important point that is not so well explained in many textbooks. In the bra-ket formalism you have to distinguish between vectors representing (pure) states and (generalized) eigenvectors of essentially self-adjoint operators which are belonging to a larger space, i.e., to the dual space of the operators' domain, where they are well defined. This domain is a dense subspace of Hilbert space.

The state vectors by definition always belong to the true Hilbert space and are normalizable to one, i.e., they fulfill [itex]\langle \psi | \rangle \psi=1.[/itex]

Generalized eigenvectors belonging to spectral values of the obsevable operators that are in the continuous part of the spectrum, can only be "normalized" to the [itex]\delta[/itex] distribution. For the position operator you have only a continuous spectrum which is the intire [itex]\mathbb{R}^3[/itex]. If you normalize the corresponding generalized eigenvectors according to
[tex]
\langle \vec{x} | \vec{x}' \rangle=\delta^{(3)}(\vec{x}-\vec{x}')[/tex]
then the wave function is given by
[tex]\psi(\vec{x})=\langle{\vec{x}}|{\psi}\rangle,[/tex]
and due to completeness we have
[tex]1=\langle \psi|\psi \rangle=\int_{\mathbb{R}^3} \mathrm{d}^3 \vec{x} \; \langle \psi|\vec{x} \rangle \langle \vec{x}|\psi \rangle=\int_{\mathbb{R}^3} \mathrm{d}^3 \vec{x} \; \psi^*(\vec{x}) \psi(\vec{x})=\int_{\mathbb{R}^3} \mathrm{d}^3 \vec{x} \; |\psi(\vec{x})|.[/tex]
It's very much the same for the momentum operator. The generalized momentum eigenstate is denoted by [itex]|\vec{p} \rangle[/itex] with [itex]\vec{p} \in \mathbb{R}^3[/itex], and one can also build the position representation of the corresponding distribution. If we also normalize the momentum eigenfunction to the [itex]\delta[/itex] distribution it reads
[tex]\langle \vec{x} | \vec{p} \rangle=\frac{1}{(2 \pi)^{3/2}} \exp(\mathrm{i} \vec{p} \cdot \vec{x}).[/tex]
From this you get the momentum-wave function from the position wave function by introducing a completeness relation in the following way
[tex]\tilde{\psi}(\vec{p})=\langle \vec{p}|\psi \rangle = \int_{\mathbb{R}^3} \mathrm{d}^3 \vec{x} \langle \vec{p}| \vec{x} \rangle \langle{\vec{x}}|{\psi} \rangle = \int_{\mathbb{R}^3} \mathrm{d}^3 \vec{x} \frac{1}{(2 \pi)^{3/2}} \exp(-\mathrm{i} \vec{p} \cdot \vec{x}) \psi(x).[/tex]
This means that the momentum-space wave function is the Fourier transform of the position-space function. That's a unitary transformation on the space [itex]L^2[/itex] (after completion from the domain of the position and momentum space operators to the whole Hilbert space).

Vanhees71, you explained it well, and if used by an experienced user, the |x> notation works.

In much worse position is a beginner, who naturally thinks that |x_0> is a kind of quantum state that can enter the expressions

[tex]
\langle x_0| x_0 \rangle,
[/tex]

or

[tex]
\langle x_0 |\hat x | x_0 \rangle .
[/tex]

But these do not have a meaning and for this reason I think it is inappropriate to use the same notation |x> for generalized eigenfunctions as for the normalized states [itex]|\psi\rangle[/itex].

The generalized functions are important, but I think one should introduce them via the language of distributions, free of bra-ket notation and free of Hilbert space, like in the book by Landau&Lifgarbagez.

But this is a matter of taste, so Bob900, if you are still thinking about this, I recommend to read the references we gave above and also Dirac's book Principles of Quantum Mechanics, where he introduced [itex] |x\rangle[/itex], and make up your own mind.