Some jet and rocket engines use their own fuel as coolant. Hydrocarbons have even been used to cool electronics and nuclear reactors. Would it be viable for a combustion engine to use its own fuel as coolant instead of water or another liquid?
Start a problem like this with some ball part calculations. I'll suggest some assumptions and a path to help you get started.
1) Assume a power, say 100 hp.
2) Assume typical efficiency of 0.45 lb of fuel per horsepower-hour for a small (automotive size) gasoline engine.
3) Assume 1/3 of the fuel goes to work (power from the crankshaft), 1/3 as heat out the exhaust, and 1/3 as heat out the water jacket.
4) Assume typical operating temperature of 190 deg F.
5) Assume a starting temperature (tank temperature) for the fuel.
6) Choose your fuel, and search the specific heat.
7) How much heat goes into the water jacket?
8) What is the fuel burn rate?
9) Given the starting temperature, ending temperature, and specific heat of the fuel, how much heat is absorbed?
100 horsepower * 0.45 pounds of gasoline per horsepower-hour = 45 pounds of gasoline per hour
45 pounds of gasoline per hour * 1/3 of energy content to the radiator = 15 pounds of gasoline energy content to the radiator per hour
Gasoline requires 0.53 British Thermal Units to raise 1 pound of fuel by 1 degree Fahrenheit (source). Gasoline contains 114,000 BTU of energy per gallon (source), with a gallon of gasoline weighing 5.91 pounds or more (source), giving an energy content of 19289.34 BTU per pound.
15 pounds of gasoline energy content to the radiator per hour * 19,289.34 BTU per gallon of gasoline = 289,340.1 BTU to the radiator per hour
72 degrees Fahrenheit seems like a good starting temperature for the fuel, which means the gasoline can't warm more than 118 degrees Fahrenheit before it starts warming the engine instead of cooling it.
I'm not sure how to continue from here. I'm not from an engineering background so I'm not sure of the equation to use going forwards. There's a formula for heating energy where I found the specific heat content for gasoline (here), but it seems overly simplistic for this calculation. It's quite literally not taking place in a vacuum and some of the heat should be lost to the atmosphere through the radiator. After all, engines more powerful than this theoretical one have been cooled purely by air, with no liquid at all.
Next step: You have 45 pounds per hour of gasoline flowing from the tank at 72 deg F, then heated to 190 deg F in the engine. The heat to do that is 45 lbs/hr X 118 deg F X 0.53 BTU/(lb-deg F) = 2810 BTU/hr. Check the units, you will see that they balance.
Result: The engine is sending 289,000 BTU/hr to the water jacket, and the fuel could theoretically absorb 2810 BTU/hr, or 1%.
Most piston aircraft engines are air cooled. A rough rule of thumb is that cooling drag (power required to force air over the engine) is about 10% of engine horsepower. Here is a good link to a person who studied the cooling drag on his homebuilt airplane: http://www.n91cz.net/Interesting_Technical_Reports/106-111_BuildingBasics.pdf. Good search terms for further information are aircraft cooling drag.