How generalizable is simulation/model research?

[Simfish]

This is often something that gets on my mind. This, anyways, is generalizable to climate models and other models, but I've figured that I'll post this here since a lot of astronomy research is based on simulations/models these days.

By "generalizable", I usually have this question in mind: Will people still be using the model's results 50 years in the future?

With math/analytical research, there is usually only one true solution to the problem. So you know that what you're doing is real and generalizable to anything that needs it. At least you know that your solution is consistent with everything else. At the same time, though, math describes what's possible. What's realizable, of course, only forms a small subset of what's possible.

When you do research by testing hypotheses, you learn about the real world. And since it *is* the real world, it has to be consistent with everything else in the real world. Since everything in the real world is connected with *something*, it's thus generalizable almost by definition.

Models differ based on the assumptions you make and the parameters you set. In a sense, they're a way to instantiate what math makes possible. Yet, you still have so many unknowns. And the time evolution of the model is such that chaos eventually becomes inevitable. Here, most of your simulations will have outcomes that are far different from what will happen in the real world. You could publish papers on the outcomes of all sorts of random models - but in the end - will people trust the model's predictions? You can tweak the parameters in a way that it is consistent with things that have already happened. And if it matches prediction on many things, then people might be more inclined to trust it.

Of course, a robust simulation will give you many opportunities to tweak the parameters once you have more information about the real world. But is model-based research the type of research that's more likely (than other types of research) to be consigned to the trashbin after 50 years? Sure, you will predict some true things with models. But in the end, someone else will probably build a better model, and that model may be very different from your model. That being said, many models are based on physical formulas.

 

[Chronos]

The problem with simulations is they omit unknown variables. This is a natural consequence of our incomplete knowledge of the universe. The value of simulations is they illuminate our ignorance. When simulations do not match observation, it speaks volumes. A simulation is a mathematical representation of theory. Observational divergence points either to a problem with theory, observational bias [error], or programming error in constructing the simulation. I can think of no way to ensure it is not a combination of all three potential sources of error. That is why we run them ad absurdium. Observational errors are, by nature, the most difficult to reconcile.