Additional problems with neutrons hitting the walls include such things as material of the wall getting knocked off by the neutrons. That means you get atoms of the wall floating into the plasma, which strongly tends to slow down the reaction.
Then there are a variety of radiation protection issues. The neutrons must be shielded, which requires some meters of whatever you are going to use. And that shielding has to be an efficient neutron absorber that won't produce a lot of secondary radiation. Or that will absorb the secondary radiation itself. It probably should not be lead, for example, because lead has the annoying habit of producing (n,2n) reactions. That is, you start with one neutron and end up with two lower energy neutrons.
And anything that gets exposed to neutrons tends to become radioactive itself. In addition to affecting the structural characteristics of the material, you wind up with radioactive material that must be itself shielded. And possibly disposed of. And that frequently means you have hard choices as to material you might use in constructing components of the reactor. Just as one example, you probably have to keep cobalt out of the mix, since activated cobalt is particularly nasty. So many grades of steel are probably off your list because they can contain cobalt. But there are many other potential problem materials in a neutron flux.