The NB4 function is looking at how the frequency of noise from a gearbox changes as a damaged tooth passes the sensor. I understand the concept, but I don't understand what the math is actually computing...

The Hilbert transform is an integral transform (much like Laplace and Fourier) as defined at mathworld here.Mathwolrd Link

The Mathematical definition there really doesn't give much insight into the application of the HT in the example you quote. If however you look at the table of Hilbert Transforms at the link you'll notice that sin(.) and cos(.) are transform pairs, this is basically the key to what the HT is doing in your example.

Essentially for narrow band signals (and possibly others - cant remember the full details) the Hilbert Transform is much the same as a 90 degree phase shift on every frequency component. This has significant application in the field of "envelope detection" as a tractable method of obtaining the approx instantaneous envelope of an oscillatory signal.

Imagine for example that you're looking at an amplitude modulated sine wave and you wish to process it in some way so as to preserve only the modulating function (that is the envelope) and throw away oscillations. If you can generate a quadrature signal of the same modulation (envelope) then you can easily generate the instantaneous envelope (A(t)) from :

A(t) cos^2(.) + A(t) sin^2(.) = A(t).

So in a nut shell that's what the Hilbert transform is doing in your quoted application. :)