Economics are probably the biggest factor. Just looking around for some estimates it seems a large solar plant cost $100 million for 30 million kWh per year. For nuclear a new plant may be as high as $5 billion, but would provide 10 billion kWh per year. Coal is even cheaper, about $500 million for 4.5 billion kWh per year. This works out to 0.3, 2, and 9 Wh per $ respectively. It doesn't take into account operating costs, which for solar and nuclear are both rather low (and yes solar has operating costs), but even with it factored in I doubt it would change things much. There's a reason coal is the main source of power throughout the world, it's cheap.
As for other factors, reliability is a big one. Solar and wind are probably the main sources people think of when they think alternatives. Both suffer from the fact that at any moment power could go out, this requires backup power from some sort of traditional power plant. Most traditional power plants have a hard time spinning up quickly, and to be always available they must be continually ran, even if the power isn't being used. Natural gas and hydroelectric are the two major types that are able to quickly provide power if needed. Natural gas is expensive, and still a fossil fuel, and hydroelectric isn't suitable for many areas.
They also suffer from the low hanging fruit problem. Certain areas are prime candidates for solar and wind, while others are very poor choices. As the best areas are taken advantage of we can expect less and less suitable areas to be used. This leads to a large portion of the nameplate capacity not being used. In other words even though your solar plant could produce 100 gWh per year you are only seeing 20 gWh because the conditions are just so poor for solar.
I also think there is not a lot of capability to increase production of solar. However, I don't feel this is a real problem. If there were a major increase in demand greater production capability should be created for no other reason than it will be profitable to do it.
If we want to be able to use some sort of intermediate source of power we will need a way to store excess power and use it later. There are a number of ways to do this, but none are very economical, or efficient. Most don't scale up very well either. I think one of the ways utilities do it is to pump water with excess power, and then use it to run turbines during shortages (effectively an dam without a natural water source). Another way to help ease storage issues would be to transmit excess power long distances to where there are shortages. That has it's own problems though, like transmission losses, and what happens during widespread low wind or sun times. The only two sources of power that seem capable of meeting our current demands are coal and nuclear. Coal is dirt cheap, but filthy, and people have a negative association with nuclear. Thus it looks like we will be stuck with coal for the foreseeable future.
The two main areas that need improvement before solar and wind could be realistic options are costs, and storage of excess power. Efficiency isn't very good for either, but it's unlikely to improve much, so I think the main effort should be in reducing the cost of current efficiency levels and improving the storage of excess power.