micromass's explanation is the correct one for basic analysis. However, it is possible to make integration by parts make sense for very rough functions using more advanced ideas which go by the name of "distribution theory". WARNING: although distribution theory is perfectly rigorous mathematics, the explanation below is anything but, so is probably not useful to you except for talking to physicists.
There is a thing which physicists call a "delta function", and we mathematicians also denote by \delta but insist on reminding everyone that it isn't actually a function in the conventional sense. Informally (physicist time), imagine that \delta(x) is a function which is zero everywhere except at x = 0, where it is infinite. But it's "exactly the right size of infinity" so that when you multiply it by something else and integrate, it pulls out the value at that point: \int_{-\infty}^\infty \delta(x) f(x)\,dx = f(0).
In your setup, what happens that makes integration by parts work is that you realize we should put f'(x) = \delta(x - 1) - \delta(x - 4), g'(x) = \delta(x - 2) - \delta(x - 5): that is, the derivative of f is a positive "unit spike" at 1, a negative "unit spike" at 4, and zero everywhere else. Makes sense, right?
Then you wind up with \int_0^3 f(x) g'(x)\,dx = f(2) = 1, \int_0^3 f'(x) g(x) \,dx = g(1) = 0, and both sides of your equation are 1, just as you want.
