Your question is essentially the same as "Why can't we have perfect weather forecasts infinitely far into the future?"
You are also far from the first to ask this philosophical question. See
https://en.wikipedia.org/wiki/Pierre-Simon_Laplace#Laplace's_demon
True, QM refutes that view. But even without QM, we have
chaos theory. And even without chaos we have nonlinearities in physics.
I like to think of it like a digital time simulation. As long as our knowledge is imperfect, there will be some inaccuracies in the results of the first time step. Those become the initial conditions for the second step which inevitably magnifies the inaccuracies in the second step, and so on.
Unless there are feedback mechanisms that limit the size of the errors, errors and uncertainties grow without limit until the only tools we have to study them are stochastic; as if they were random. It is not that the process itself is random (like throwing dice), but rather that the results inevitably become randomized.
In many real world simulations (think of a flight simulator) there are many feedback loops that limit the inaccuracies.
The second law of thermodynamics is closely associated with the same kinds of thinking. Thermalized and randomized are two words with associations.
But to complete the loop. If we scratch deeper and deeper, then we do see that QM indeed lies at the bottom of everything, and that is the ultimate answer.