First we have to define "sustainable". One obvious limitation is the lifespan of the Sun, which has already been mentioned, so we cannot mean indefinitely sustainable, and the only really long term hope for mankind is to get off the Earth, but for now let's assume "sustainable" means for as long as the Sun lasts. See
http://en.wikipedia.org/wiki/Earth#Future
Secondly, by "sustainable population" do we mean just the human population or do we wish to sustain the diversity of life on Earth? If we do, then we should be aware that even the current human population of 7 to 8 billion is already threatening the survival of many other species on Earth.
Next, to obtain a ball park figure for sustainable population we should ignore all fossil fuels as a source of energy, as they cannot be replaced at any reasonable pace, so the base survival figure should be based on solar energy. The average number of Joules of sunlight per square meter of the Earth's surface is approximately 14.2 MJ/day. For the entire Earth's surface the potentially available energy (PE) is 14.2*5.1*10^14 = 7.24*10^15 MJ/day.
If we care about diversity of life on Earth then obviously we cannot cover the entire Earth with solar panels and some areas will have to be left natural. Even if we do not care about diversity, if the Earth is entirely covered by solar energy panels there will no room for plants or algae and so there will be no CO2 recycling or oxygen production. Additionally if we use a very large fraction of the Earth's surface for solar energy production there will be artificial climate change implications. Thus there is limit to the Earth's surface that can be used for solar energy production and bearing in mind that 2/3 of the surface is ocean, a figure for the fraction of surface area that is exploitable (EA) might be somewhere in the region of say 2% of the total surface area, so that gives a figure of PE*EA = 1.44*10^14 MJ/day.
Now we need to take into consideration the conversion efficiency (CE) rate that we can convert exploitable solar energy into usable energy. Photo voltaic cells have an average conversion efficiency of solar power into electricity of around 20%. Systems that covert solar energy directly into home or water heating can be more efficient, but we have to bear in mind the energy costs of constructing solar power systems. When we take into account the energy to manufacture solar power systems and deduct this from the energy captured from the devices, the effective efficiencies are much lower. In fact some calculations show that more energy is used in manufacturing some of these devices, than is ever recovered and they only exist because of government subsidies. Taking into account manufacturing energy costs we might guess that the conversion efficiency is somewhere in the region of 5% and that might be generous. This makes the total usable energy PE*EA*CE = 7.24*10^12 MJ/day.
Now that we have a rough total potentially usable solar energy figure, we can in principle estimate the maximum sustainable population, if we know the average energy consumption per person. An average person consumes 1800 calories of food per day. This equates to 0.0075 MJ per day which is almost negligible even if takes ten times that in raw solar energy to produce the food. I say it almost negligible, because In 2008, total worldwide energy consumption was 474×10^12 MJ/year. This equates to an average energy consumption (EC) per person worldwide of about 200 MJ/day. Compare this to an average consumption of 895 MJ/day/person in the USA. This is energy used for home heating, air conditioning, cooking, refrigeration, entertainment devices, hot water, lighting, manufacturing energy costs, waste and water treatment, and vehicle fuel and they make food energy requirement pale into insignificance. There are countries that consume more than USA, but if we assume in the future that the worst users cut back their consumption a bit and that third world countries aspire to a living standard similar to that of the USA and other first world countries, then EC= 800 MJ/day per person might be a conservative estimate of usage in the future.
Now we can do a very rough estimate of a solar sustainable world population (SP) using SP = PE*EA*CE/EC = 7.24*10^12/800 = approx 9 billion people.
Of course you are free to come up with your own future projections for EA, CE and EC (and I invite you do so), but you should be aware that even with the current population we are causing climate change, fighting over resources like oil, water, land and threatening the survival of various species, so bear in mind that there is more to sustainability, than just theoretical availability of energy.
Note: I have only considered solar energy which includes indirectly wind power. Wave power might be considered as lunar power and independent of solar power. If we extracted 100% off all available wave power, would this have any consequences on the rotation rate of the Moon or the Earth? It certainly would have an effect on sea life affecting migration routes etc. Fusion power might be another option, but the means to reliably produce usable power from fusion has yet to be demonstrated and perhaps more resources should be put into fusion research.