Experimental Results vs Predictions

[StevieTNZ]

So this has been on my mind:
When an experiment is performed and results gathered/calculated etc and checked against theoretical predictions:

* when you take into account random errors and
* when you take into account systematic errors,

and the result is within the one predicted, are you confirming the prediction? Or is there some probability to the result and really you're saying what I've measured probably is as predicted?

 

[elegysix]

yes, predictions should be a value +/- error, and as long as your measurement falls within that range, your prediction is confirmed.
If your results are outside that range, you should consider if you've overlooked some sources of error. If you don't find any more sources, you should start over and redesign the experiment.

 

[russ_watters]

No, that's not correct. Not only does it not absolutely confirm the theory, being inside your error margin doesn't even absolutely confirm the experimental result. The error bars only represent a certain confidence - 95% iirc.

 

[cmb]

When an experiment is performed and results gathered/calculated etc and checked against theoretical predictions...and the result is within the one predicted, are you confirming the prediction?

What you are describing here is 'proof by association/co-incidence'. Unfortunately, it seems that many occasions of what are deeemed (should be!) 'science' get caught up in this kind of thinking and promotes 'experimenter bias' (where the experimenter sees what they were expecting, so further entrenches their ideas).

The 'strong' scientific way is to use your theory to generate null hypotheses, that is to say; your theory should be able to predict things that won't happen, if it were correct. You then look for where, or try to make, those things happen.

The data you get back can never prove any theory in itself, but it may serve to either reject a null hypothesis (in which case your theory remains, not disproven), or you fail to reject the null hypothesis, in which case your theory has failed.

The particular strand of science I worry over in particular is in medical science where the results can be more subjective than one might care for and where clinical protocols themselves tend to work by drawing conclusions by co-incidence, viz. people with more X in their diet have a higher frequency of Y illness. It is almost impossible to construct a meaningful null hypothesis in such a scenario if you have no comprehension of the mechanistic reasons that cause illness Y.

 

[Oltz]

What you are describing here is 'proof by association/co-incidence'. Unfortunately, it seems that many occasions of what are deeemed (should be!) 'science' get caught up in this kind of thinking and promotes 'experimenter bias' (where the experimenter sees what they were expecting, so further entrenches their ideas).

The 'strong' scientific way is to use your theory to generate null hypotheses, that is to say; your theory should be able to predict things that won't happen, if it were correct. You then look for where, or try to make, those things happen.

The data you get back can never prove any theory in itself, but it may serve to either reject a null hypothesis (in which case your theory remains, not disproven), or you fail to reject the null hypothesis, in which case your theory has failed.

The particular strand of science I worry over in particular is in medical science where the results can be more subjective than one might care for and where clinical protocols themselves tend to work by drawing conclusions by co-incidence, viz. people with more X in their diet have a higher frequency of Y illness. It is almost impossible to construct a meaningful null hypothesis in such a scenario if you have no comprehension of the mechanistic reasons that cause illness Y.

Well said and any "scientist" who come forward and annouces they have proven this or that theroy simply was not doing science they were doing statitics and correlation. All a scientist can generally do is step forward and say I preformed these tests with these controls and failed to disprove the theroy of blank thus as of now it is still valid.

Proof is rare and needs to be concrete hence even evolution is still a theory like relativity.

All good science is skeptical, beliefs do not belong in the lab.

 

[humanino]

The error bars only represent a certain confidence - 95% iirc.

We often show so-called "1-sigma" error bars which are only 68% confidence.

95% <-> "2-sigma"
99.7% <-> "3-sigma" -> qualifies for serious publication ("anomaly")
"5-sigma <-> 0.5 chances in a million for a statistical fluke <- common discovery level in physics

In general, I would say there is no big deal happening in this thread, there is no "strong vs weak" scientific method. It is very well understood (in the scientific communities) what the difference between "measurement vs prediction" is. When we make a prediction, we use probability. When we infer the parameters of a theory from measurements, or test the agreement of a theory with the measurements, we use statistics. This distinction is understood well enough so that university lectures have different titles.

When we use statistics to infer the most probable values for the parameters of a theory, or to test the agreement between the predictions and the measurements, we never even think of "proving the theory". There is not any meaning to "the probability that the theory is right given the measurement" because there is not even a well defined space for all possible theories, let alone a metric in such a putative space. Whichever agreement level between predictions and measurements, the theory can always be modified ever so slightly that the agreement is essentially unchanged.

I would like to share one of the most hilarious moments I have ever witnessed in the interview of a professional physicist for popular account, because it is relevant to this thread. It happens at 45 s in the video below

 

[thorium1010]

Proof is rare and needs to be concrete hence even evolution is still a theory like relativity.

This is your personal opinion, isn't it ?

 

[FlexGunship]

See, the difference between theory and practice is that in theory practice and theory are the same but in practice they are not.

 

[Oltz]

This is your personal opinion, isn't it ?

No...Evolution and Relativity are both theroies unless you are asking if proof is rare and concrete is my opinion?

To which I would say can you Prove that Proof is common and does not need to be concrete? :tongue:

One of the common issues with this topic is that global getting hotter thing that people seem to think is "proven" every other week. but we will not talk about that here

 

[StevieTNZ]

What you are describing here is 'proof by association/co-incidence'. Unfortunately, it seems that many occasions of what are deeemed (should be!) 'science' get caught up in this kind of thinking and promotes 'experimenter bias' (where the experimenter sees what they were expecting, so further entrenches their ideas).

The 'strong' scientific way is to use your theory to generate null hypotheses, that is to say; your theory should be able to predict things that won't happen, if it were correct. You then look for where, or try to make, those things happen.

The data you get back can never prove any theory in itself, but it may serve to either reject a null hypothesis (in which case your theory remains, not disproven), or you fail to reject the null hypothesis, in which case your theory has failed.

The particular strand of science I worry over in particular is in medical science where the results can be more subjective than one might care for and where clinical protocols themselves tend to work by drawing conclusions by co-incidence, viz. people with more X in their diet have a higher frequency of Y illness. It is almost impossible to construct a meaningful null hypothesis in such a scenario if you have no comprehension of the mechanistic reasons that cause illness Y.

I guess my question is: how can a theory have failed if your measurement result is just a probablistic statement?

 

[Jimmy Snyder]

If the value predicted by the theory falls within the error bars of the experiment, then the theory is confirmed.

 

[humanino]

Proof is rare and needs to be concrete hence even evolution is still a theory like relativity.

Yes you are right. And so is gravity, a theory. And so is the "theory" that you will die if you jump off a 10km altitude plane. And by that I mean, some people are known to have survived. Still a theory good enough for me not to test it.

One of the common issues with this topic is that global getting hotter thing that people seem to think is "proven" every other week. but we will not talk about that here

There is not such issue. It is getting hotter. The issue is why.

 

[Oltz]

There is not such issue. It is getting hotter. The issue is why.

I concurr but again that is a topic we are not allowed on here.

Gravity on the other hand is a Law. If you jump out of that plane you may not die but you will fall period no questions asked.

The theroy that everyone dies who jumps out af an airplane is easy to disprove with a parachute.

That theroy would then need to be revised
the new Hypothesis could be:

Everyone who jumps out of an airplane from 10km and shoots them self in the head while falling after slitting their wrists without a parachute before hitting the ground dies?

That Theroy could one day be promoted to Law of the natural world but really who wants to disprove it?

 

[dlgoff]

I would like to share one of the most hilarious moments I have ever witnessed in the interview of a professional physicist for popular account, because it is relevant to this thread. It happens at 45 s in the video below

It’s hard to assign probabilities for something that only happens once.

I thinking this might go into my signature. :rofl:

 

[humanino]

Gravity on the other hand is a Law.

I was not very specific so I will try to improve.

F = G m_{1} m_{2} / r_{12}^2 is a law. The gravitational part of the lagrangian density being equal to the Ricci scalar is a more precise law. More explicitly, Newton wrote down a full blown theory of gravity. Einstein did too. We know Newton's theory limitations because Einstein's is more precise. Einstein's theory predict its own doom and we have not been able to demonstrate any other of its limitations experimentally.

If you jump out of that plane you may not die but you will fall period no questions asked.

There are a few, here and there, historical exceptions to this "law" or "theory".

The theroy that everyone dies who jumps out af an airplane is easy to disprove with a parachute.

That is a loophole to my initial formulation and aside the point. My point was mostly that a "law" is concept weaker than what scientists call a "theory". A law is a mere mathematical relation which is suggested by experimental data but does not have a full blown conceptual basis. A theory is a set of conceptual hypothesis, translated into mathematical laws, numerically developed to experimental predictions which have been tested and validated to some degree (degree of accuracy, of kinematical reach, etc).

 

[cmb]

If the value predicted by the theory falls within the error bars of the experiment, then the theory is confirmed.

No. That is absolutely and totally wrong, and is precisely why it is necessary to do science by null hypotheses. This is very much the subject of the post and the trap that many self-delusionists in science have fallen into.

OK, a trivial example. All cows are black and white. I go look at a field and count a hundred cows. They are all black and white. My theory is confirmed.

NO. NO!

What you do is say 'My theory is "All cows are black and white.". To disprove this I need to find one cow that is not black and white. I look at one field of cows and do not find a non-black and white cow. My theory remains, not disproven.' [You can say no more than this.]

And it will remain until an observation of a not-B&W cow is made. Once a not-B&W cow is observed, then it is no longer a theory, it is nothing. A theory is something that you can disprove. You can only disprove it, you can never 'prove' it. However, there are levels of confidence so high that we regard them as 'proof' and accept them axiomatically.

So there is still some pedantery there to overcome, on the borders of scientific philosophy.

 

[russ_watters]

Proof is rare and needs to be concrete hence even evolution is still a theory like relativity.

Use of the word "still" is improper here. "Theory" is the highest level of proof with a title in science.

 

[Jimmy Snyder]

No. That is absolutely and totally wrong, NO. NO!

Yes. I said confirmed, not proven. I repeat. If the value predicted by the theory falls within the error bars of the experiment, then the theory is confirmed.

 

[StevieTNZ]

Again, how can a theory be refuted by experimental results?
For example:
Experimental result: 1.5kg +/- 0.2
Theory predicts: 1.1kg

But the experimental result has some probability to being right. The true value might be 1.1, but that doesn't show up in the experimental result

 

[Jimmy Snyder]

Again, how can a theory be refuted by experimental results?
For example:
Experimental result: 1.5kg +/- 0.2
Theory predicts: 1.1kg

But the experimental result has some probability to being right. The true value might be 1.1, but that doesn't show up in the experimental result

In this case, the experiment falsifies the theory.

 

[StevieTNZ]

But the value obtained from the experiment might not be the right value, so you can't really say the theory is falsified.

 

[Jimmy Snyder]

But the value obtained from the experiment might not be the right value, so you can't really say the theory is falsified.

Yes you can. That's how science works. Theory must match experiment.

 

[cmb]

Yes you can. That's how science works. Theory must match experiment.

No. Again, if the theory is putting bounds on the measurement then it can only do so in statistical terms. If the '+/-0.2' means one standard deviation, then 1.1 away from 1.5 is 2 standard deviations, which is 95%. This is not 100%. It never, ever is. But we may accept that level of uncertainty as disproving a hypothesis. (Remember, there is never any case of saying 'it matches the prediction therefore the theory is proven'.)

The term '+/-' means different things in different circumstances. For example, in radio-isotope dating, if you hear something reported to be 'dated to 1000AD +/- 100years', then that means there is a 68% chance it is dated 900 to 1100AD, because this is a field that uses 'one standard deviation' for its +/- term. Other fields use other terms. LAB 34, for example, the UKAS ascribed means of quoting uncertainties for accredited EMC testing in the UK, recommends uncertainties are quoted as 'k=2', meaning to be quoted to 2 standard deviations.

 

[Jimmy Snyder]

The search for 100% certainty is philosophy, not science.

 

[StevieTNZ]

There is a difference between saying the theory is falsified, and the theory is probably falsified. You can't say it's falsified based on a probabilistic statement. The theory you suddendly falsified could be correct.

 

[humanino]

The search for 100% certainty is philosophy, not science.

Most certainly yes. However, a two standard deviation is also very likely to occur every week if we allow it to become publication.

 

[Jimmy Snyder]

Most certainly yes. However, a two standard deviation is also very likely to occur every week if we allow it to become publication.

That doesn't change what I say by one iota. If you require 3 standard deviations before falsification, so be it. If 4, fine with me, I'm good with 5, 6, 7, and 8. Just not infinite that's all. Once you go past the limit, whatever that limit is, there's no going back. Just out of curiosity though, how many standard deviations are required for publication?

 

[StevieTNZ]

I would rather like to know that when an experiment is done to "confirm" a theory, that it is actually somewhat doing that, rather than leaving it up to chance. But that's me and my 'everything has to be certain' stance.

 

[Jimmy Snyder]

I would rather like to know that when an experiment is done to "confirm" a theory, that it is actually somewhat doing that, rather than leaving it up to chance. But that's me and my 'everything has to be certain' stance.

If scientists insisted on certainty we would never have gotten anywhere. That's pretty much the situation that philosophy finds itself in.

 

[cmb]

The search for 100% certainty is philosophy, not science.

That's the point you don't seem to be getting (accepting?) still.

Yes, you can be 100% certain of something when you have confirmed the null hypothesis. That's why you should approach science by the means of null hypotheses.

As per my scenario above: You go to the next field and see a red cow. You are now 100% sure the null hypothesis is confirmed.

So 100% is something you leave to show the null hypothesis, while the theorist who posited the theory simply hopes that no-one will find evidence of the null hypothesis because that way his precious pet theory won't have ever been falsified to 100% - he can still cling on to some slight hope that his new crazy idea might still be true, even if 99% of the evidence merely suggests otherwise.

 

[humanino]

That doesn't change what I say by one iota. If you require 3 standard deviations before falsification, so be it. If 4, fine with me, I'm good with 5, 6, 7, and 8. Just not infinite that's all. Once you go past the limit, whatever that limit is, there's no going back.

My comment was in the 1.5 +/- 0.2 vs 1.1 example. I am sorry I was just re-iterating the obvious.

Just out of curiosity though, how many standard deviations are required for publication?

5 sigma counts as a discovery (that's pretty well the paradigm in physics). It is more or less well accepted that above 2.5 sigma is the threshold of "anomaly" for publication.

 

[Jimmy Snyder]

That's why you should approach science by the means of null hypotheses.

Theory A predicts 2 and theory B predicts 1. I wish to run an experiment that will help to decide which is the better theory. What should my null hypothesis be? If the measured value was 1.1 then how many standard deviations will it take to decide on theory B? How many standard deviations will it take to be 100% certain that theory A is wrong? Suppose I use humanino's criterion and find that 2 is more than 5 sigma away from 1.1. Can I publish?

 

[DoggerDan]

Theory A predicts 2 and theory B predicts 1. I wish to run an experiment that will help to decide which is the better theory. What should my null hypothesis be? If the measured value was 1.1 then how many standard deviations will it take to decide on theory B? How many standard deviations will it take to be 100% certain that theory A is wrong? Suppose I use humanino's criterion and find that 2 is more than 5 sigma away from 1.1. Can I publish?

ISBN: 0-8120-1869-9. I somehow aced it in college, then ram-dumped it. This handbook kept me out of trouble over the last decade.

Well, it's a simplistic guide. Sometimes I actually had to dig back into my college texts.

 

[cmb]

Theory A predicts 2 and theory B predicts 1. I wish to run an experiment that will help to decide which is the better theory.

(Sorry, I didn't pick up on this first time.)

'Which theory is better...'? My opinion of that question; therein lies a fundamental misunderstanding of what a 'theory' is. Both theories are equally as good if not disproven by the confirmation of a null hypothesis. However, there may be a much bigger confidence in one than the other.
What did you mean by 'better'?

 

[Jimmy Snyder]

(Sorry, I didn't pick up on this first time.)

'Which theory is better...'? My opinion of that question; therein lies a fundamental misunderstanding of what a 'theory' is. Both theories are equally as good if not disproven by the confirmation of a null hypothesis. However, there may be a much bigger confidence in one than the other.
What did you mean by 'better'?

Is Newtonian mechanics equally as good as relativistic mechanics, or was Newtonian disproven by confirmation of a null hypothesis? If the latter, what was the null hypothesis?