Fact checking - Scientific method

[Toxis]

Hi,

I have a somewhat strange request (hence did not knew which section to use for this post). I have no formal training in sciences (have a degree in philosophy, and no matter how much my peers disagreed with my opinion, that is not a science:), but I fell in love with science while studying philosophy. Watched all I could find on the internet, but that does not constitute as a science degree does it. I have written this article how just understanding scientific method transformed me from a science believer/parotter to a science enthusiast. And how understanding it, understanding the steps involved and the terminology might help non scientific people making informed decisions, or simply while reading various hype stories in the media. However before publishing it (no grandioze plans for that, plan to post to internal newsletter of my company) I want to fact check is my understanding of the scientific method based on reality :) As this is actual representation how I analyzed it in my head.

If you could read the following text (I will not bother you with my whole article, only the part where I speak about scientific method) and point out where I'm wrong, or not factually correct. Oh, and I'm not a native speaker, so please excuse my use of language and possible mistakes, I will edit this much more before posting.
Thanks in advance! Quick note - original text has hyperlinks to sources where I quote Feynman and definitions are hyperlinked to http://www.nap.edu/openbook.php?record_id=6024&page=2

First let me start with how scientific method actually works. When you have an ‘Idea’ of how something yet unknown or in question works, first line of defence is self censorship (not an actual term, I named this for my own better understanding). It’s a process where you try to disprove it yourself. Start by thinking about every possibility how this could not be true, try find logic holes in your idea, then try to think of any known Scientific Facts (an observation that has been repeatedly confirmed and for all practical purposes is accepted as "true." Truth in science, however, is never final, and what is accepted as a fact today may be modified or even discarded tomorrow) that would violate your idea.Then if your guess is correct, could it violate some known Facts. If yes what would the testable outcomes be. If you are unable to disprove this guess yourself, then you move to second phase (thus making your idea a ‘hypothesis’) - if this mechanism/guess is correct, what else is correct, how could I test this (create predictions).Write them down (my people, philosophers like Karl Popper, suggested that this step needs one additional step - falsifiability, one has to have to imagine an outcome of the experiment, that could prove your initial hypothesis wrong). Then you actually go (or convince someone else) and create experiments to gather the data. Then you do the experiment and log everything, and I mean everything! When experimental data comes in, if it does not align with your predictions - your hypothesis is wrong. Or as Richard Feynman put this “If it disagrees with experiment it is wrong. In that simple statement is the key to science. It does not make any difference how beautiful your guess is. It does not make any difference how smart you are, who made the guess, or what his name is – if it disagrees with experiment it is wrong. That is all there is to it”. If it is correct though, you are far from done, then you publish your hypothesis, explain in detail how you did the experiment, share all the results. Then the third line of defence kicks in, a well known phrase called ‘Peer Review’. And do not confuse it with peer review we do here - your paper is sent to anonymous colleagues of yours, who do their best, to find holes in your reasoning (you get scientific ‘brownie points’ for doing that.) and in your proposed experiment. Then if it goes through this process successfully, you are still not done. Then, your peers have to recreate your experiment and check, have the data they get, match the data you said it did (a fourth line of defence). And this is repeated many times to be more certain. In fact science can not prove anything right, they can say that we are certain this is correct, but this certainty is never 100% - a gold standard in particle physics for example is 5 sigma (a 99.9999% chance something to be correct). The process I described above is just an example, usually, these days there is much more collaboration that that - if you have a good hypothesis (best if it’s described not only in words, as they are too vague for science, but in maths. Maths is truly a universal language of science). You collaborate with your colleagues, or students in trying to create a best possible experiment. In physics for example, there are even different flavours of physicists, theorists, who basically run through the first steps of the process, then you need experimentalists, who do the experiments, and more often than not - engineers who help create experimental protocols and actual apparatuses.

 

[Quantum Defect]

This does a better job than a lot of things that I have seen trying to describe what scientists do (the scientific method).

I would add some bits.

The whole "hypothesis generation" part of the scientific method has been overblown, I think. There are large areas of science (exploratory science) where the formation of hypotheses, etc. is kind of fuzzy. People do some entirely new experiments, not knowing what they will see. Think about the first microscope, or a lot of work in astronomy. In these cases, the important early work is building up a database that people can then look at, scratch their heads and think about. In these cases, "the hypothesis" might be the laughable: "I will see something interesting when I look at this with my doo-hickey." This is a very important kind of science, but it often gets forgotten.

Replication is all but impossible in many experiments (too expensive, not enough samples, etc.) In addition, there are no real rewards for exact replication. This actually is a problem in the health sciences, and people are trying to figure out ways to incentivize this -- it is necessary, because it has been shown that there is a lot of junk science out there. In the really splashy cases (like the stem-cell work at RIKEN) you will have people try to reproduce the exact work, but in most areas, I do not think that this is the case.

My mathematics friends are horrified that when physical scientists get their work peer reviewed, the peers do not actually reproduce the result, but you got that bit correct.

 

[Toxis]

Ah, thanks, I'll try to incorporate these point too. The main reason I have tried clearly show concrete steps of progression in this case was first, that this was how I understood the logic of it myself, and using this analogy I talk about the homeopathy, and try to show that their basic claim do not go past the first stage (analysis based on what we know about science), and even the controversial paper published in nature by Benveniste, was unable to pass the third and fourth 'lines of defence' - peer review and produce the results they claimed they produced in the study they published. " Hence this theory failed and can only be called pseudo-science (a claim or belief, falsely presented as scientific, but does not adhere to a valid scientific method, cannot be reliably tested, or otherwise lacks scientific status)".

 

[phinds]

Your write-up looks very good to me and you are on very solid ground via your understanding of Feynman.

 

[Toxis]

Thanks for good words, I know what changes I'll do - I will change last part, that starts with "The process I described above is just an example," - and add what Quantum Defect pointed, that this is oversimplified version of the process, in real world it's much more complicated and messy.. well, something along those lines.

I have another silly question, that does not deserve it's separate post:) How wrong I am in the following line of reasoning.
From the moment I have heard about tesseracts, I was baffled by them, and kept asking everyone, wait, if you can calculate it, does this mean that they exist, and my confusion only increased when I found out string theory and it's postulates.. But then one day it dawned on me - n-dimensional geometry and string theory are scientific version of 'fan fiction'. But in order to write it, one not only has to be an expert in the 'canon', but has to posses uber 'language' skill, as this fan fiction is written not in normal language, but in maths. By fan fiction I do not want to disrespect the field, or people doing that, by that I mean that it's a logical (wrong word) extension of the field (if this works for 3 dimensions, what the answer would be if there were 4 dimension). And by the way, do physicists really see maths as language - that is if you see a long equation, can you get the meaning of it as if you saw this written in words, but without any double meanings.. in it's purest form? Or you see it as noobs (me:) do, the difference you can follow it's logic through and get the result, thus 'translating' it in your head.. Sorry, if this sounds completely stupid this topic always fascinates me:)

 

[phinds]

... by that I mean that it's a logical (wrong word) extension of the field (if this works for 3 dimensions, what the answer would be if there were 4 dimension).

No, that is not at all the rationale behind string theory. String theorists didn't WANT there to be multiple dimensions, and they did not start out thinking about showing how things work with multiple dimensions, they started out positing that everything is made up of strings and it's just that multiple dimensions is what comes out of the math if you posit that everything is made up of strings and follow the logic of that to see how it could work.

String theory has held great promise for something like 30 years now without ever producing anything concrete. It is mathematically elegant, it would be wonderful if it turned out to represent reality because it would solve some problems and expand our understanding of how the universe works, BUT ... so far it can't even be shown to represent reality so it just remains a mathematically consistent construct and no more.

 

[mfb]

Then you actually go (or convince someone else) and create experiments to gather the data. Then you do the experiment and log everything, and I mean everything! When experimental data comes in, if it does not align with your predictions - your hypothesis is wrong. [...] If it is correct though, you are far from done, then you publish your hypothesis, explain in detail how you did the experiment, share all the results.

You do not need a new experiment to publish a new hypothesis. A better explanation for previous experiments is very nice, even an alternative explanation that is consistent with old experiments but makes different predictions for new tests is interesting.

Replication is all but impossible in many experiments (too expensive, not enough samples, etc.)

What is a result worth that never gets checked? I prefer an experiment that got verified by someone else over two untested claims. Checking the one interesting result out of 100 just takes 1% more effort. At least in physics, replications are absolutely necessary - a result that cannot be checked in some other way will never get widely accepted.
There is a reward: you can improve the accuracy with your replication. If the effect is interesting, you can get the best measurement of an interesting effect - until someone else makes an even better study.

And by the way, do physicists really see maths as language - that is if you see a long equation, can you get the meaning of it as if you saw this written in words, but without any double meanings.. in it's purest form?

If the things used in the equation are clearly defined, yes. Those definitions are always necessary (and sometimes they are ambiguous :().

"$a^2+b^2=c^2$" alone is meaningless - what are a, b and c?
"In every right triangle with sidelengths a,b,c, where c is the hypothenuse, $a^2+b^2=c^2$." is clear.String theory itself did not lead to clear predictions yet, but the tools developed there found some applications in quantum chromodynamics (the theory of the strong interaction).

 

[Quantum Defect]

What is a result worth that never gets checked? I prefer an experiment that got verified by someone else over two untested claims. Checking the one interesting result out of 100 just takes 1% more effort. At least in physics, replications are absolutely necessary - a result that cannot be checked in some other way will never get widely accepted.
There is a reward: you can improve the accuracy with your replication. If the effect is interesting, you can get the best measurement of an interesting effect - until someone else makes an even better study.

Different fields have different norms and different reward structures. Also some experiments cannot be reproduced (observational work on once-in-a-lifetime events).

Certainly, an experiment that gets verified is more trust-worthy than one that is not. In many fields, if you are the first to publish something new, you get a nice paper. If someone else comes along and checks the result, and gets the same answer, this is not really publishable. In many fields, people get scooped when two groups are working on the same project simultaneously, and one group publishes first. Often the second group is left with something that cannot be published.

If the experiment is reproduced and there is disagreement, then this could be publishable, assuming the first measurement was wrong. If the second measurement is not clearly wrong, it could be published, and this will prompt others to try to sort out what is going on.

However, given the reward structure, there is a disincentive for exact replication -- the chances of you finding something new and publishable are less when looking at something that has been looked at previously.

Experiments do get checked, but usually this happens when someone is trying to move forward on something new, and they are "calibrating" on something that has already been done.

There is also the murky areas of un-published negative results. Who knows how many people-years would be saved if scientists could publish the non-results of experiments that did not work. I have been surprised by the lack of work in a particular area, and thought that I had found fertile ground for some new research, only to hear (after some struggle) that a half dozen other people had the same idea that I had, and tried and failed.

Science would undoubtedly be stronger if all results were checked and the results of null experiments were published, but the "infrastructure" is in place, and the reward structure in many fields is firmly established, that this is unlikely to take place in many fields.

 

[mfb]

Different fields have different norms and different reward structures. [...] If someone else comes along and checks the result, and gets the same answer, this is not really publishable. In many fields, people get scooped when two groups are working on the same project simultaneously, and one group publishes first. Often the second group is left with something that cannot be published.

Yes, some fields have questionable scientific methods.

Also some experiments cannot be reproduced (observational work on once-in-a-lifetime events).

Okay sure, but I think in most fields that is an exception.

However, given the reward structure, there is a disincentive for exact replication -- the chances of you finding something new and publishable are less when looking at something that has been looked at previously.

Depends on the field. In particle physics for example, replications are common - sure, usually with better accuracy, but even if the accuracy is a bit worse it is done to check the previous measurement. Negative results (no effect seen) get published in the same way as positive results - the success of the Standard Model means most searches are negative anyway.
Proper science is possible.

 

[DennisN]

Hi Toxis,
since you are interested in the scientific method, I'd like to share a couple of links that you may find useful/interesting:

• Introduction to the Scientific Method (University of Rochester)
• An Introduction to Science (Miami University)
  and two so-called "baloney detection kits" :):
  
• Carl Sagan's Baloney Detection Kit (from the book "The Demon-Haunted World: Science as a Candle in the Dark", I think)
  
• Michael Shermer's Baloney Detection Kit: Part I - Part II - Video

 

[Toxis]

Thanks everyone for your input, that gives me additional ideas.

If I could just go back to the string theory part - I watched everything I could find by Brian Greene, Steven Weinberg, Jim Gates, and others, and I'm fully aware of the fact, that this is pure speculation so far (awesome mathematics, but no reality check), I've listen to Lawrence Crauss rants about ST:), and you almost answered the question I always had, but just to double check - how was it conceived, was it 'what if we go way way beyond standard model of particles, and let's say there is something at the plank scale, let's say strings, and try to use best current models and maths to calculate that' .. the question I have is why strings? Why not tiny spheres, or triangles, is it a tribute to the crazy cultist called Pythagoras, or was there some rationale behind this, or did calculations showed, that at this scale only string-like object could work?And more more question about replicability, that generated so many different views - would I be correct(ish) in saying that replicability is always preferred in scientific method, but not always possible due to many factors: costs, rarity of the event, specifics of the scientific field, to name a few.?

 

[DennisN]

but just to double check - how was it conceived, was it 'what if we go way way beyond standard model of particles, and let's say there is something at the plank scale, let's say strings, and try to use best current models and maths to calculate that' .. the question I have is why strings? Why not tiny spheres, or triangles, is it a tribute to the crazy cultist called Pythagoras, or was there some rationale behind this, or did calculations showed, that at this scale only string-like object could work?

First I have to clearly state that I personally know very little about String Theory. Very little. Yet, AFAIK, yes, there definitely was rationale behind String Theory. You could start by having a look at these:

• http://en.wikipedia.org/wiki/History_of_string_theory

and this short clip:



I've also got another clip (a lecture) by Susskind in which he describes the beginnings of String Theory, but I've got to find it first, so I'll be back when I do.

Why not tiny spheres, or triangles

(my bolding)
Hehe! Actually there is an entirely different theory that's called "Causal Dynamic Triangulation", see e.g.:

http://en.wikipedia.org/wiki/Causal_dynamical_triangulation
and
http://www.scientificamerican.com/article/the-triangular-universe/

But please note that - if I understand the basics of it correctly - it's more a model of spacetime (see http://en.wikipedia.org/wiki/Causal_dynamical_triangulation#Introduction). I could not resist mentioning it since you mentioned triangles.

 

[DennisN]

I think it was this lecture I've seen (during the first part of it, Susskind describes the origin of String Theory):



If you feel bold you can watch the entire lecture :).

This clip may also be worth watching ("Superstrings" (Fermilab)):

 

[Toxis]

oh, what an awful feeling, when after trying my best to understand Susskind's lecture, and failing I had to go to the 'string theory explained to imbeciles' version of it.. ah, the truth hurts so much =)

Thanks, that helped

And regarding CDT - wow, I never knew this is a viable theory, I spent several weeks (well, night time is when i usually go into these thoughts:) thinking about somewhat similar question (just, you know, having no basic training it was a 'duude, and what if' version of thinking, granted - I tried looking at it from different sides and trying to disprove the ideas myself:) - my initial thought that got me on this path was - can the spacetime itself be quantised - in other words is space time made of something, if yes, what would that mean, and how could one test that (in theory), and what could interact with said quanta of spacetime. And to know that actual scientist thought about this - niice - will have to spend some time trying to understand how did they approached this and what was their line of reasoning. Thanks!

 

[DennisN]

oh, what an awful feeling, when after trying my best to understand Susskind's lecture, and failing I had to go to the 'string theory explained to imbeciles' version of it.. ah, the truth hurts so much =)

Don't beat yourself up about it, these ARE difficult things :D.

Thanks, that helped

Cool!

By the way, I stumbled upon a lecture which I saw yesterday, which was partly about the development of string theory. I thought I might post it, you may find it interesting:

Edward Witten delivers his lecture "String Theory and The Universe" to an audience at the Institute of Physics in London.