Huge and complex experiments -- validity

[Dante Augusto]

How blindly theorits trust the data comming from huge and complex experiments such as the LHC CERN? Is it possible for one person to understand the whole experiment mechanics and still be able to come up with theoretical freamworks describing the data behaviour? Is it possible even to experimentalists to say they have all the aspects of the experiment under control, ranging from thousands of lines of code written by humans to sensors and detectors proper instalation? It seems to me a bit unconfortable to trust this chain of specialized groups of specialistis, from my perspective there is a huge space for little erros in this whole effort, but i wish I'm wrong. Any links with papers and trustable texts on this subject are welcome, since i couldn't find much about it searching the web on my own.

Forgive my ignorance, but this is an actual sincere doubt i have and i believe there is no better place then a science forum to be skeptical and ask questions. Also forgive my english erros, I'm from Brazil and portuguese is my native language.

 

[Borek]

You are perfectly right that there is plenty of space for mistakes - which is why nobody trusts the data blindly and why they are checked by different teams using different methodologies where possible.

Yes, it is not a perfect approach, but it is the best we can design. Besides, it has worked so far quite nicely.

 

[phyzguy]

Why do you "blindly" expect the huge and complex network connected to your cell phone to call the right person when you dial their number? Do you have any idea how complex the sequence of events is that connects your phone to another person? No one person can understand this whole network and all the things that go on in it. The answer is that there are constant cross checks built into make sure things go right and correct them when they fail.

 

[newjerseyrunner]

Any valid scientific experiment is repeatable. So confidence from one run may be low, but if you get the same result a billion times on multiple pieces of hardware and software and they all agree, your confidence grows.

Conplicated things are built on the shoulders of giants. For example, nobody had to figure out how to create the beam for LHC, we'd been doing that for decades. Sure, the specifics of LHC are unique to LHC, but the concept is exactly the same as Fermilabs or any other accelerator.

I'm sure they also used modules that are known to be good. LHC shares and hosts a lot of data. I am sure they didn't do it from scratch, I would assume that it's just a RAID array and apache.

 

[ZapperZ]

How blindly theorits trust the data comming from huge and complex experiments such as the LHC CERN? Is it possible for one person to understand the whole experiment mechanics and still be able to come up with theoretical freamworks describing the data behaviour? Is it possible even to experimentalists to say they have all the aspects of the experiment under control, ranging from thousands of lines of code written by humans to sensors and detectors proper instalation? It seems to me a bit unconfortable to trust this chain of specialized groups of specialistis, from my perspective there is a huge space for little erros in this whole effort, but i wish I'm wrong. Any links with papers and trustable texts on this subject are welcome, since i couldn't find much about it searching the web on my own.

Forgive my ignorance, but this is an actual sincere doubt i have and i believe there is no better place then a science forum to be skeptical and ask questions. Also forgive my english erros, I'm from Brazil and portuguese is my native language.

1. You will notice that when the Higgs was first reported, they were reported by two different collider-detector groups at the LHC: ATLAS and CMS. They both work independently from each other, and they are different types of detectors probing different characteristics.

2. What they reported has to be consistent with the "phase space" that have already been ruled out previously, mainly by the Tevatron. So the result simply can't come out of nowhere. If it does, then something isn't quite right or inconsistent, and this is where usually people will start checking against each other, or there might be new physics.

3. The process of checking and double checking and result being reproducible are common in all experimental physics, not just for the big ones. When someone claim to have discovered a new superconductor, for example, the community will verify the resistivity measurement, and then will demand a magnetic susceptibility measurement, which is more stringent. Then maybe the detection of the superconducting gap, etc...etc... In other words, there are several different parameters that are probed by different experimental techniques that ALL need to be in agreement before everyone will accept that conclusion.

Big, small, medium size, etc... they all have to show the same type of robustness in the results.

Zz.

 

[epenguin]

Why do you "blindly" expect the huge and complex network connected to your cell phone to call the right person when you dial their number? Do you have any idea how complex the sequence of events is that connects your phone to another person? No one person can understand this whole network and all the things that go on in it. The answer is that there are constant cross checks built into make sure things go right and correct them when they fail.

Not a good answer to the question. The process of your example may be very complex, but the verification that it has worked is easy and immediate.

 

[Ygggdrasil]

A good example of the challenges faced by large complex experiments, which also demonstrates the self-correcting nature of science, is the mistaken detection of superluminal neutrinos in 2011. Scientists at the OPERA experiment detected neutrinos that apparently seemed to be traveling at faster than the speed of light. The team published a pre-print of their results basically saying, we think these results are wrong, but we can't find anything wrong with our experiment, so can someone please double check this in case our fundamental understanding of physics is wrong.

Later, it was found that a loose cable and a faulty clock within the experiment were to blame for the anomalous readings. Subsequent measurements at OPERA and at independent sites confirmed that neutrinos do not travel faster than the speed of light.

Further reading: http://nautil.us/issue/24/error/the-data-that-threatened-to-break-physics

 

[mfb]

No one understands all components of the experiments in all details. It is not necessary. Every component has some group that does understand all the details, and those groups work together.

Let's consider the Higgs discovery as an example.

There is a group monitoring the muon detectors: checking that the temperature of every module is right, checking that everything works and so on. The experts in that group have to know what to do if some power supply fails - but they don't have to know anything about the Higgs. They make the muon detector status available to others in the collaboration.

There is a group responsible for the software finding muons in the collision data. They take the muon detector status into account. Their software produces data like "here was a muon with this energy and flight direction, there was another muon with that energy and flight direction". They don't have to know what to do if a power supply fails, they just have to know how to take it into account that some module didn't work. They also don't have to know anything about the Higgs.

There is a group checking the results of the former group: If the software finds a muon, how likely is it an actual muon and not something else? Which fraction of muons stays undetected? How precisely is the muon energy estimate?

There is a group looking for Higgs bosons decaying to four muons. They use the results of the former groups: the software finding muons and the information how often they are actually muons, how many muons are undetected, and the precision of the energy estimates. They don't have to know every detail about the detector itself.
In 2012, they got a possible detection of the Higgs boson: "We have X more events than expected around a mass of 125 GeV, with an uncertainty of Y".Independent of the muon groups, there are similar groups responsible for detecting photons.

There is a group looking for Higgs bosons decaying two photons. In 2012, they also got a possible detection of the Higgs boson.There is a group combining the two independent results. They check if the two partial results are compatible with two possible decays of a single particle, if the observed numbers of events fits to the expectations from the Higgs boson, and so on. They have to know the analyses well - but not in every detail. In 2012, they combined the two results and found "We have a significant result: There is a new particle at a mass of about 125 GeV".

They made this result available to the rest of the collaboration, and everyone could check the analysis. The muon groups verified that their results were used properly, the photon groups checked that their results were used properly, statistics experts checked the combination, and so on. Finally, when everyone was happy with everything, it was made public - and checked by people outside the collaboration.

All this was done independently both in ATLAS and CMS, with the same result from both collaborations.

The analysis was repeated with larger datasets later, and with a higher collision energy even later. With improved analysis methods, with different people working on it and so on. In addition, all the steps described above have multiple internal cross-checks on their own. The "Higgs to muon analysis" was not done once - every step was done at least twice, often with different methods, to verify that (a) there are no bugs in the code and (b) the methods used are reliable. Same for the "Higgs to photon analysis", the combination and all the other steps.This is an extremely simplified description of how the collaborations work. There are many more groups involved, but the main idea is the same: Have experts for everything, let them produce well-checked and well-defined results that can be used by other groups who don't have to know how all the details work.
The Higgs discovery was the work of more than 1000 people per experiment, all with their small contribution in the group they worked in, checking every step multiple times.Theorists are yet another step: they don't have to know all that. They don't have to know how to exchange a power supply, how to find muons in collision data, or anything like that. All they typically need is the publication: "Ah, there were so many Higgs bosons decaying to muons, and so many Higgs bosons decaying to photons, and the Higgs mass is 125 GeV" together with the experimental uncertainties on all those values.

LHC shares and hosts a lot of data. I am sure they didn't do it from scratch, I would assume that it's just a RAID array and apache.

Hundreds of Petabytes on a simple RAID, accessible and analyzable by thousands of users? I want to see that RAID.
The Grid has a lot of custom software.

 

[russ_watters]

A good example of the challenges faced by large complex experiments, which also demonstrates the self-correcting nature of science, is the mistaken detection of superluminal neutrinos in 2011. Scientists at the OPERA experiment detected neutrinos that apparently seemed to be traveling at faster than the speed of light. The team published a pre-print of their results basically saying, we think these results are wrong, but we can't find anything wrong with our experiment, so can someone please double check this in case our fundamental understanding of physics is wrong.

Later, it was found that a loose cable and a faulty clock within the experiment were to blame for the anomalous readings. Subsequent measurements at OPERA and at independent sites confirmed that neutrinos do not travel faster than the speed of light.

It is worth noting here that when an experiment produces an error, it is likely the error will be a deviation from the predictions, not an erroneous confirmation of the predictions. This makes error checking easier, since the errors tend to highlight themselves.

You can also use the concept to identify badly conducted science, such as in the case of NASA's "EM Drive" research. There is no theoretical basis for the effect being measured. This makes error checking difficult because you don't know what value a to expect a "positive" outcome to have. As a result, every non-zero result is being reported as a potential "positive" outcome, despite the results being vastly different in magnitude and small enough that experimental error can't be ruled out.