Most would say the problem lies in observable
maths - scientific models that are framed as formal statements with measurable consequences (rather than unmeasurable ones).
Science involves both models and measurements in interaction. So for a TOE to be scientific, it has to be testable.
If a TOE is just a formally stated hypothesis that is ultimately untestable, then it becomes metaphysics.
So I would say it is looking like the maths will stretch beyond the eventual reach of science. Instead of mathless fact, it is more likely we will end up with factless maths!
Which is good news for the future career prospects of metaphysicians.