Experimental Results vs Predictions

  • Thread starter StevieTNZ
  • Start date
  • #1
1,532
464
So this has been on my mind:
When an experiment is performed and results gathered/calculated etc and checked against theoretical predictions:

* when you take into account random errors and
* when you take into account systematic errors,

and the result is within the one predicted, are you confirming the prediction? Or is there some probability to the result and really you're saying what I've measured probably is as predicted?
 

Answers and Replies

  • #2
406
15
yes, predictions should be a value +/- error, and as long as your measurement falls within that range, your prediction is confirmed.
If your results are outside that range, you should consider if you've overlooked some sources of error. If you don't find any more sources, you should start over and redesign the experiment.
 
  • #3
russ_watters
Mentor
20,234
6,797
No, that's not correct. Not only does it not absolutely confirm the theory, being inside your error margin doesn't even absolutely confirm the experimental result. The error bars only represent a certain confidence - 95% iirc.
 
  • #4
cmb
863
43
When an experiment is performed and results gathered/calculated etc and checked against theoretical predictions...and the result is within the one predicted, are you confirming the prediction?
What you are describing here is 'proof by association/co-incidence'. Unfortunately, it seems that many occasions of what are deeemed (should be!) 'science' get caught up in this kind of thinking and promotes 'experimenter bias' (where the experimenter sees what they were expecting, so further entrenches their ideas).

The 'strong' scientific way is to use your theory to generate null hypotheses, that is to say; your theory should be able to predict things that won't happen, if it were correct. You then look for where, or try to make, those things happen.

The data you get back can never prove any theory in itself, but it may serve to either reject a null hypothesis (in which case your theory remains, not disproven), or you fail to reject the null hypothesis, in which case your theory has failed.

The particular strand of science I worry over in particular is in medical science where the results can be more subjective than one might care for and where clinical protocols themselves tend to work by drawing conclusions by co-incidence, viz. people with more X in their diet have a higher frequency of Y illness. It is almost impossible to construct a meaningful null hypothesis in such a scenario if you have no comprehension of the mechanistic reasons that cause illness Y.
 
  • #5
Oltz
What you are describing here is 'proof by association/co-incidence'. Unfortunately, it seems that many occasions of what are deeemed (should be!) 'science' get caught up in this kind of thinking and promotes 'experimenter bias' (where the experimenter sees what they were expecting, so further entrenches their ideas).

The 'strong' scientific way is to use your theory to generate null hypotheses, that is to say; your theory should be able to predict things that won't happen, if it were correct. You then look for where, or try to make, those things happen.

The data you get back can never prove any theory in itself, but it may serve to either reject a null hypothesis (in which case your theory remains, not disproven), or you fail to reject the null hypothesis, in which case your theory has failed.

The particular strand of science I worry over in particular is in medical science where the results can be more subjective than one might care for and where clinical protocols themselves tend to work by drawing conclusions by co-incidence, viz. people with more X in their diet have a higher frequency of Y illness. It is almost impossible to construct a meaningful null hypothesis in such a scenario if you have no comprehension of the mechanistic reasons that cause illness Y.

Well said and any "scientist" who come forward and annouces they have proven this or that theroy simply was not doing science they were doing statitics and correlation. All a scientist can generally do is step forward and say I preformed these tests with these controls and failed to disprove the theroy of blank thus as of now it is still valid.

Proof is rare and needs to be concrete hence even evolution is still a theory like relativity.

All good science is skeptical, beliefs do not belong in the lab.
 
  • #6
2,425
7
The error bars only represent a certain confidence - 95% iirc.
We often show so-called "1-sigma" error bars which are only 68% confidence.

95% <-> "2-sigma"
99.7% <-> "3-sigma" -> qualifies for serious publication ("anomaly")
"5-sigma <-> 0.5 chances in a million for a statistical fluke <- common discovery level in physics

In general, I would say there is no big deal happening in this thread, there is no "strong vs weak" scientific method. It is very well understood (in the scientific communities) what the difference between "measurement vs prediction" is. When we make a prediction, we use probability. When we infer the parameters of a theory from measurements, or test the agreement of a theory with the measurements, we use statistics. This distinction is understood well enough so that university lectures have different titles.

When we use statistics to infer the most probable values for the parameters of a theory, or to test the agreement between the predictions and the measurements, we never even think of "proving the theory". There is not any meaning to "the probability that the theory is right given the measurement" because there is not even a well defined space for all possible theories, let alone a metric in such a putative space. Whichever agreement level between predictions and measurements, the theory can always be modified ever so slightly that the agreement is essentially unchanged.

I would like to share one of the most hilarious moments I have ever witnessed in the interview of a professional physicist for popular account, because it is relevant to this thread. It happens at 45 s in the video below
 
Last edited by a moderator:
  • #7
167
6
Proof is rare and needs to be concrete hence even evolution is still a theory like relativity.
This is your personal opinion, isn't it ?
 
  • #8
FlexGunship
Gold Member
369
8
See, the difference between theory and practice is that in theory practice and theory are the same but in practice they are not.
 
  • #9
Oltz
This is your personal opinion, isn't it ?
No...Evolution and Relativity are both theroies unless you are asking if proof is rare and concrete is my opinion?

To which I would say can you Prove that Proof is common and does not need to be concrete? :tongue:

One of the common issues with this topic is that global getting hotter thing that people seem to think is "proven" every other week. but we will not talk about that here
 
  • #10
1,532
464
What you are describing here is 'proof by association/co-incidence'. Unfortunately, it seems that many occasions of what are deeemed (should be!) 'science' get caught up in this kind of thinking and promotes 'experimenter bias' (where the experimenter sees what they were expecting, so further entrenches their ideas).

The 'strong' scientific way is to use your theory to generate null hypotheses, that is to say; your theory should be able to predict things that won't happen, if it were correct. You then look for where, or try to make, those things happen.

The data you get back can never prove any theory in itself, but it may serve to either reject a null hypothesis (in which case your theory remains, not disproven), or you fail to reject the null hypothesis, in which case your theory has failed.

The particular strand of science I worry over in particular is in medical science where the results can be more subjective than one might care for and where clinical protocols themselves tend to work by drawing conclusions by co-incidence, viz. people with more X in their diet have a higher frequency of Y illness. It is almost impossible to construct a meaningful null hypothesis in such a scenario if you have no comprehension of the mechanistic reasons that cause illness Y.
I guess my question is: how can a theory have failed if your measurement result is just a probablistic statement?
 
  • #11
918
16
If the value predicted by the theory falls within the error bars of the experiment, then the theory is confirmed.
 
  • #12
2,425
7
Proof is rare and needs to be concrete hence even evolution is still a theory like relativity.
Yes you are right. And so is gravity, a theory. And so is the "theory" that you will die if you jump off a 10km altitude plane. And by that I mean, some people are known to have survived. Still a theory good enough for me not to test it.


One of the common issues with this topic is that global getting hotter thing that people seem to think is "proven" every other week. but we will not talk about that here
There is not such issue. It is getting hotter. The issue is why.
 
  • #13
Oltz
There is not such issue. It is getting hotter. The issue is why.
I concurr but again that is a topic we are not allowed on here.

Gravity on the other hand is a Law. If you jump out of that plane you may not die but you will fall period no questions asked.

The theroy that everyone dies who jumps out af an airplane is easy to disprove with a parachute.

That theroy would then need to be revised
the new Hypothesis could be:

Everyone who jumps out of an airplane from 10km and shoots them self in the head while falling after slitting their wrists with out a parachute before hitting the ground dies?

That Theroy could one day be promoted to Law of the natural world but really who wants to disprove it?
 
  • #14
dlgoff
Science Advisor
Gold Member
3,903
1,861
I would like to share one of the most hilarious moments I have ever witnessed in the interview of a professional physicist for popular account, because it is relevant to this thread. It happens at 45 s in the video below
It’s hard to assign probabilities for something that only happens once.
I thinking this might go into my signature. :rofl:
 
Last edited by a moderator:
  • #15
2,425
7
Gravity on the other hand is a Law.
I was not very specific so I will try to improve.

F = G m_{1} m_{2} / r_{12}^2 is a law. The gravitational part of the lagrangian density being equal to the Ricci scalar is a more precise law. More explicitly, Newton wrote down a full blown theory of gravity. Einstein did too. We know Newton's theory limitations because Einstein's is more precise. Einstein's theory predict its own doom and we have not been able to demonstrate any other of its limitations experimentally.

If you jump out of that plane you may not die but you will fall period no questions asked.
There are a few, here and there, historical exceptions to this "law" or "theory".

The theroy that everyone dies who jumps out af an airplane is easy to disprove with a parachute.
That is a loophole to my initial formulation and aside the point. My point was mostly that a "law" is concept weaker than what scientists call a "theory". A law is a mere mathematical relation which is suggested by experimental data but does not have a full blown conceptual basis. A theory is a set of conceptual hypothesis, translated into mathematical laws, numerically developed to experimental predictions which have been tested and validated to some degree (degree of accuracy, of kinematical reach, etc).
 
  • #16
cmb
863
43
If the value predicted by the theory falls within the error bars of the experiment, then the theory is confirmed.
No. That is absolutely and totally wrong, and is precisely why it is necessary to do science by null hypotheses. This is very much the subject of the post and the trap that many self-delusionists in science have fallen into.

OK, a trivial example. All cows are black and white. I go look at a field and count a hundred cows. They are all black and white. My theory is confirmed.

NO. NO!

What you do is say 'My theory is "All cows are black and white.". To disprove this I need to find one cow that is not black and white. I look at one field of cows and do not find a non-black and white cow. My theory remains, not disproven.' [You can say no more than this.]

And it will remain until an observation of a not-B&W cow is made. Once a not-B&W cow is observed, then it is no longer a theory, it is nothing. A theory is something that you can disprove. You can only disprove it, you can never 'prove' it. However, there are levels of confidence so high that we regard them as 'proof' and accept them axiomatically.

So there is still some pedantery there to overcome, on the borders of scientific philosophy.
 
  • #17
russ_watters
Mentor
20,234
6,797
Proof is rare and needs to be concrete hence even evolution is still a theory like relativity.
Use of the word "still" is improper here. "Theory" is the highest level of proof with a title in science.
 
  • #18
918
16
No. That is absolutely and totally wrong, NO. NO!
Yes. I said confirmed, not proven. I repeat. If the value predicted by the theory falls within the error bars of the experiment, then the theory is confirmed.
 
  • #19
1,532
464
Again, how can a theory be refuted by experimental results?
For example:
Experimental result: 1.5kg +/- 0.2
Theory predicts: 1.1kg

But the experimental result has some probability to being right. The true value might be 1.1, but that doesn't show up in the experimental result
 
  • #20
918
16
Again, how can a theory be refuted by experimental results?
For example:
Experimental result: 1.5kg +/- 0.2
Theory predicts: 1.1kg

But the experimental result has some probability to being right. The true value might be 1.1, but that doesn't show up in the experimental result
In this case, the experiment falsifies the theory.
 
  • #21
1,532
464
But the value obtained from the experiment might not be the right value, so you can't really say the theory is falsified.
 
  • #22
918
16
But the value obtained from the experiment might not be the right value, so you can't really say the theory is falsified.
Yes you can. That's how science works. Theory must match experiment.
 
  • #23
cmb
863
43
Yes you can. That's how science works. Theory must match experiment.
No. Again, if the theory is putting bounds on the measurement then it can only do so in statistical terms. If the '+/-0.2' means one standard deviation, then 1.1 away from 1.5 is 2 standard deviations, which is 95%. This is not 100%. It never, ever is. But we may accept that level of uncertainty as disproving a hypothesis. (Remember, there is never any case of saying 'it matches the prediction therefore the theory is proven'.)

The term '+/-' means different things in different circumstances. For example, in radio-isotope dating, if you hear something reported to be 'dated to 1000AD +/- 100years', then that means there is a 68% chance it is dated 900 to 1100AD, because this is a field that uses 'one standard deviation' for its +/- term. Other fields use other terms. LAB 34, for example, the UKAS ascribed means of quoting uncertainties for accredited EMC testing in the UK, recommends uncertainties are quoted as 'k=2', meaning to be quoted to 2 standard deviations.
 
  • #24
918
16
The search for 100% certainty is philosophy, not science.
 
  • #25
1,532
464
There is a difference between saying the theory is falsified, and the theory is probably falsified. You can't say it's falsified based on a probabilistic statement. The theory you suddendly falsified could be correct.
 

Related Threads on Experimental Results vs Predictions

  • Last Post
3
Replies
52
Views
36K
  • Last Post
Replies
3
Views
2K
  • Last Post
2
Replies
26
Views
5K
  • Last Post
Replies
12
Views
2K
  • Last Post
Replies
1
Views
2K
  • Last Post
2
Replies
25
Views
4K
  • Last Post
Replies
10
Views
2K
  • Last Post
2
Replies
48
Views
6K
  • Last Post
Replies
16
Views
5K
  • Last Post
Replies
18
Views
4K
Top