This should be easy enough, but I find myself unsure of what they want, due to their wording.
[tex]\frac{d^2 \Psi(x)}{dx^2} = \frac{2m}{\hbar^2}(V(x) - E)\Psi(x)[/tex]
[tex]k^2 = \frac{2m(V(x) - E)}{\hbar^2}[/tex]
[tex]\frac{d^2 \Psi(x)}{dx^2} - k^2\Psi(x) = 0[/tex]
[tex]\Psi(x) = Ae^{kx} + Be^{-kx}[/tex]
[tex]<\Psi^*|\Psi> = 1 = A^2 \int_a^b 2 + e^{2kx} + e^{-2kx} \,dx [/tex]
At a glance this cannot be normalized for a free particle, but this is no suprise as you usually do not normalize a free particle over all space. On the other hand, it is also clear that you cannot normalize this within a box of any length, as [tex]\Psi[/tex] cannot have a value of zero. That is my answer, but I believe this situation can have a normalizable equation if it is part of a more complicated set of potentials (right?).
So is my answer incomplete?

Hold on there. You can't solve that Schrödinger equation generally, V(x) could be any potential. k depends on x here, remember? It's not a constant, so you can't integrate as if it were.

You have to show that if the energy of a stationary state is lower than the minimum value of V, the solution cannot be normalized. The big hint is given: [tex]\frac{d^2 \Psi(x)}{dx^2} = \frac{2m}{\hbar^2}(V(x) - E)\Psi(x)[/tex]
If E<V_min, then psi and its second derivative always have the same sign. Can such a function be normalized? (You can always take psi to be real, so that this makes sense)

Whoops, you are right; I treated V as constant. That hint doesn't mean much to me; all it tells me is that the exponential will not be to an imaginary power. With a specific situation I think I could show that it can't be normalized; for instance I don't think I have any problem with part b (though maybe I should post what I wrote just to make sure).

Unfortunately I'm cut short by having to leave work and will start again tomorrow.

Do you know what the second derivative of a function tells you? It's the rate at which the slope is changing. So if a function and its second derivative have the same sign it means that if your function is positive somewhere, then its slope is increasing at that point. If your function is negative somewhere its slope is decreasing. Take an arbitrary wavefunction, what will happen at x-> infinity or x->-infinity ? Take some cases if you want. No explicit calculation is needed.

I thought I did, but it didn't do me much good. If it's alright, let me post my answer to part b and see if it is at all acceptable. In this case we have some infinite square well. We will take [tex]V[/tex] inside the well to be zero and study the case where the energy is negative. The well has dimensions of length l and begins at point a.
[tex]\frac{d^2 \Psi(x)}{dx^2} = \frac{2m}{\hbar^2}(0-(-E))\Psi(x)[/tex]
[tex]k^2 = \frac{2m(E)}{\hbar^2}[/tex]
[tex]\Psi(x) = Ae^{kx} + Be^{-kx}[/tex]
[tex]\Psi(a) = 0 = Ae^{ka} + Be^{-ka}[/tex]
[tex]\Psi(a+l) = 0 = Ae^{k(a+l)} + B e^{-k(a+l)}[/tex]
Because [tex]e^{kx}[/tex] is not periodic it seems obvious to me that it will never be zero at both a and (a + l). These equations imply a dual value for A.
Please let me know if I'm on the right track. Thanks for your time and patience.

Locrian, I think you've missed the point that Galileo was trying to make in post #4.

1. First of all, stop thinking in terms of exponentials or plane wave states. These are solutions to specific (constant) potentials.

2. Re-read post #4. Consider just positive values of [itex]\psi(x)[/itex] for starters. For its second derivative to be positive, its slope must be monotonically increasing. Do you see what this means ?

Agreed. I'm sure I missed yours, too. I just hope no one thinks it is from lack of trying.

If [tex]\psi[/tex] is always increasing then I can't normalize over all space because the answer is infinite (which is true of the case where E>V too). But since it is always increasing, the particle can't be bound, either, because it can't possibly meet the boundary conditions. That would seem to rule out everything, but it doesn't sound like a very good argument to me.

That's really all I've got. Thanks for your responses. Maybe I can find a related problem and see if I do any better.

[itex]\psi [/itex] doesn't have to be always increasing, but if E<V_min its slope is always increasing in the regions where [itex]\psi[/itex] is positive. The function is curving upwards. It's also called http://mathworld.wolfram.com/ConvexFunction.html".
Likewise, in the regions where [itex]\psi[/itex] is negative, the second derivative is negative so the slope is decreasing. Thus the function is curving downwards (it's concave on that region).

What cases can there be? Either the derivative of [itex]\psi[/itex] is always zero or there is some point where the derivative is either positive or negative. Convince yourself that in each of these cases the function cannot be normalizable using what I said above.