Break down the heat input into the various parts that create heat:
- Bearings (rolling element or hydrodynamic)
- Gears
- Windage
- Other frictional losses?
- Oil pump losses
Rolling element bearings have a very low coefficient of friction, around 0.002 or less. Check with SKF or other bearing manufacturer. Hydrodynamic bearings are similarly very low. Use that frictional load at the point on the radius where the bearing is and use the radial load as the normal load to calculate the frictional load. Then determine heat input by multiplying by velocity to get power.
http://www.roymech.co.uk/Useful_Tables/Tribology/Bearing Friction.html
http://www.roymech.co.uk/Useful_Tables/Tribology/Plain_Bearing Friction.html
http://www.bearings.machinedesign.com/guiEdits/Content/BDE_6_1/bdemech6_3.aspx
http://www.skf.com/portal/skf/home/products?newlink=1_0_40&lang=en&maincatalogue=1
Gears can be looked at simply as a 'loss'. Gears are x% efficient at transmitting power, so that x% gets converted to heat that must be rejected.
http://www.roymech.co.uk/Useful_Tables/Drive/Gear_Efficiency.html
Windage probably isn't very significant, and is very difficult to analyze with any accuracy. Just make sure the rest of your calculations are conservative and you'll be fine.
If there are any additional frictional losses such as a sliding bearing (ex: linear or cylindrical bushing) then you can calculate losses by estimating frictional coefficient times normal load times velocity.
http://www.roymech.co.uk/Useful_Tables/Tribology/co_of_frict.htm
An oil pump is essentially a direct heat input. If you have oil being circulated into the box using a pump that requires x amount of power, you can simply add that amount of power directly to the gearbox. That power has to come back out somehow, which is why an oil cooler is often used.
Once you have all the power losses, determine the steady state heat transfer out of the box by determining how warm the box has to get in order for that heat to be rejected.