Yes, I agree that there are many subjects where needing reference material is going to be a serious problem. I still don't believe that a lot of mathematics is in that category, though.
Much of biology, medicine, anthropology, etc. are knowledge-based sciences. Math and physics tend to be computation-based sciences. Obviously, if you don't have the basic knowledge, you can't even start the computation. However, speed of computation is very different than speed of knowledge recall.