While there are a great many challenges, this doesn't necessarily mean they will all be insurmountable, the greatest risks are likely to be in space and the effects of 0 gravity and high radiation exposure have been extensively studied. Despite the idea of using rotation as a form of artificial gravity, which potentially could solve a lot of problems, this remains untested. We also know a great deal about radiation shielding but there is an issue of getting it into space, again increasing the use of space based construction and resource collection could be the way forward.
Initially, there is no reason that some preparatory work could not be completed by unmanned drones, though they might also be subject to radiation effects. Many of the issues might be easier to deal with on planetary bodies like mars, though the problems would be different.
I think the real issue might end up being how well we have identified and understood the risks. I think that considering the already huge costs involved, many of the potential solutions might form quite a small percentage of economic considerations, there might also be economic advantages in developing exploitation of the asteroid belt with drones. It might make the most sense to reduce the focus on manned missions, the manned part being a major problem to use the time to develop the knowledge and technologies needed. I don't really see anything to indicate that these issues would prevent progress.
