happyparticle said:
I thought because the operator was an integral and the inner product of 2 functions is an integral as well.
I didn't know ##
\psi (x) = \langle \left.x\right|\psi\rangle
##
If I understand griffith (p.90 third edition) this is like a dot product between 2 vectors?
You want to recognize that ##f(x)## and ##\lvert f \rangle##, while related, are not the same thing and therefore are not interchangeable. ##\lvert f \rangle## denotes a vector while ##f(x)## represents the vector in the position basis. In your initial attempt, you mixed these two types of objects up, which led to the confusing mess you started with.
Say you have two vectors ##\mathbf{a}## and ##\mathbf{b}## in an ##n##-dimensional vector space ##V##, and you want to calculate ##\mathbf{a}\cdot\mathbf{b}##. One way to do that it to first choose an orthonormal basis ##\{\mathbf{e}_1, \mathbf{e}_2, \dots, \mathbf{e}_n\}## and then represent ##\mathbf{a}## and ##\mathbf{b}## by the ##n##-tuples ##(a_1, a_2, \dots, a_n)## and ##(b_1, b_2, \dots, b_n)##. In other words, you have
\begin{align*}
\mathbf{a} = \sum_{i=1}^n a_i \mathbf{e}_i \\
\mathbf{b} = \sum_{i=1}^n b_i \mathbf{e}_i
\end{align*} Then the dot product would be given by
$$\mathbf{a}\cdot\mathbf{b} = \sum_{i=1}^n a_i b_i.$$
Compare that last expression to the one you learned for ##\langle f \vert g \rangle##.
$$\langle f \vert g \rangle = \int f^*(x)g(x)\,dx.$$ ##\langle f \rvert## and ##\lvert g \rangle## corresponds to ##\mathbf{a}## and ##\mathbf{b}##. The integration corresponds to summing over ##i##. And ##f(x)## and ##g(x)## correspond to the ##a_i##'s and ##b_i##'s.
Just as ##(a_1, a_2, \dots, a_n)## is merely a
representation of ##\mathbf{a}## with respect to the orthonormal basis, ##f(x)## is merely a representation of ##\lvert f \rangle## in the position basis. In other words,
$$\lvert f \rangle = \int \psi(x) \lvert x \rangle\,dx.$$ Compare that expression to the sum above where ##\mathbf{a}## was expanded in terms of the chosen basis. The integral corresponds to the summation; ##\psi(x)## to ##a_i##, and ##\lvert x \rangle## to ##\mathbf{e}_i##.
Finally, note that
$$\mathbf{e}_i \cdot \mathbf{a} = \mathbf{e}_i \cdot \sum_{j=1}^n a_j \mathbf{e}_j = \sum_{j=1}^n a_j (\mathbf{e}_i \cdot \mathbf{e}_j) = \sum_{j=1}^n a_j \delta_{ij} = a_i.$$ The analogous calculation In Dirac notation is
$$\langle x \vert f \rangle = \langle x \rvert \int f(x')\lvert x' \rangle\,dx' =
\int f(x')\langle x \vert x' \rangle\,dx' =
\int f(x')\delta(x-x')\,dx' = f(x)$$
I glossed over some technicalities since you're generally dealing with complex numbers in quantum mechanics, but the gist should be clear. Dirac notation might look a little strange at first, but once you get used to it, you'll find it's a much more convenient way of expressing the ideas you learned in linear algebra.