It depends on your application. If you are doing Fourier transforms, then what you said is true and to be stable the impulse responds h(t) must be finite.
Its different for Z-transforms.
Ill assume you are working with Fourier transforms.
Lets say you have a circuit, and you input an impulse. The output you would read is the impulse response, h(t).
A common mathematical thing you might do, is use a Fourier transform on h(t) to get the frequency response, H(jw), because we like having frequency response of systems.
If you have a h(t) that doesn't 'approach zero for sufficiently large time', when you compute the Fourier transform, you could possible get an answer that is infinite, or something not find able.
If h(t) was stable (finite), then finding the H(jw) would be possible mathematically, using a Fourier transform.
In simpler words, if your h(t) goes to infinite, then you will have problems integrating it. And we like integrating things, to get H(s), H(jw), and other things.
Imagine a ball hanging from a string. In the beginning its hanging completely straight and not moving at all.
This ball now represents your system/circuit.
If you gave this ball a hit with a baseball bat, it amounts to giving your system an impulse input. Obviously the ball will start swinging from side to side and then eventually settle down to it's beginning state again - straight and not moving. That means the impulse response of your system approaches zero it goes back to "position 0" again.
As system like that - a system that gets back to its starting position, is called a stable system. There are two other possibilities of what can happen - it can begin swinging from side to side forever (if the world was ideal). This is a non stable system. Or the swinging could increase (not really physically possible will a ball and a string, but whatever).
Now if we turn our attention to circuits for a moment.
If you have some output voltage that begins oscillating and the oscillations increase when you give it an impulse (voltage spike) at the input. Then the system is unstable. It is unstable because the voltage will rise and rise and rise until something in the system (or at the output) will break down.
I hope that helped.
PS. If the oscillations from an input response are simply oscillating at a certain voltage forever, then your system is "on the verge" of becoming unstable. In reality, such systems will have a high likelyhood of becomming unstable, and so we also call such systems unstable. We want the responses to settle down.
It's my understanding that you want a non-infinite integral because in the frequency domain, (s-domain or Laplace), an impulse function creates a decaying exponential function. Another note is that as the Impulse function moves further and further from the origin, the range of frequencies under which a significant area of the integral falls, shrinks.