Fanelli D (2010) “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE 5(4): e10068. doi:10.1371/journal.pone.0010068
http://www.plosone.org/article/info:doi/10.1371/journal.pone.0010068
Six comments.
1. Someone will inevitably post the XKCD comic on this topic. I'll leave that task to someone else.
2. This proves the adage "Politics ain't rocket science. It's much harder."
3. On a more serious note, there are a lot of different ways to interpret this paper. One is that the soft sciences are indeed soft. They are much more subject to confirmation biases than are the hard sciences.
4. "This study analysed 2434 papers published in all disciplines and that declared to have tested a hypothesis." They did this by searching papers (titles only? titles+abstracts? full text?) for the phrase "test* the hypothes*". I don't recall seeing that phrase in the articles I read. I just looked at several 'test of physics' type papers as a test of the hypothesis "physicists don't use the phrase 'test* the hyothes*'". That phrase is just not there for the most part.
5. Some disciplines use the phraseology tested for in this paper, others don't. Some may use it preferentially to report positive results, others tend to use it more when reporting negative results. There is no accounting for this potential bias in the paper.
6. Data. Where's the data? They analyzed 2434 papers. Which ones, and how did they classify them?
Do you any anecdotes to offer? I feel like I've had some pretty robust physics professors, but I've read very little of their published work and I'm probably not capable of an objective critique. I have been considering a psychology PhD for sometime (I did my undergrad in physics; finishing an MS in computational neuroscience currently).
I heard a Nature podcast from an economist criticizing the way economists use mathematical models (compared to physicists). The nature of the criticism was basically that economists don't have much for controlled experiment, so elegance becomes a determining factor of what kind of math survives, which isn't scientifically sound.
I am not familiar with modern economic theory, so I have no opinion, just regurgitating the podcast hearsay.
Do you any anecdotes to offer? I feel like I've had some pretty robust physics professors, but I've read very little of their published work and I'm probably not capable of an objective critique. I have been considering a psychology PhD for sometime (I did my undergrad in physics; finishing an MS in computational neuroscience currently).
My concerns:
- If you are a physicisist, read the paper by Daryll Bem, published in one of the highest impact journals of psychology and try stay sane while wondering how such a paper could ever be published in the name of science. Bem claims to have found evidence that his students can feel the occurence of future events above chance, especially men detecting erotic events. He claims this ability may have evolved because there is an evolutionary advantage for males to know where future possibilities for reproduction may occur. (no I am not making this up).
The editors claim the paper has been rigurously reviewed and it has been accepted because there seemed nothing wrong with the methods used. The fact that these results and their interpretation would turn almost all scientific disciplines from physics to evolutionary theory upside down (tear down that LHC!) is apparently not important. The paper should have been rejected on theoretical grounds. This concern may be summarised as: there is NO theory evaluation in psychology, all so called theories are actually theories of construction (that save the phenomena) they are not theories of principles (in the sense that Einstein proposed).
The highest quality clinical evidence of Evidence Based Medicine and Psychology has rejected the traditional model of academic science and returned to the fundamental principles under which science is supposed to be practised. It is prioritising the evidence over the theory and really does not care about such matters as who is upset by the evidence.
The academic scientific model has been criticised as extremely open to subjective abuse for the medical profession to maintain a monopoly in health for the sake of power and profit. In evidence based medicine, nobody really understands how clinical psychology, physiotherapy or acupuncture works scientifically but the evidence says that they do. The reasoning behind preferring exercise to bed rest or avoiding therapist dependency is backed by hard evidence but is more common sense than scientific. And the studies based on this type of 'theory' are saving lives.
Is there any scientific explanation for the placebo effect?
So the very best in clinical medicine and psychology is harder. It is always the result of multidisciplinary teamwork and subject to multidisciplinary review because getting it right is a matter of life and death. Objectivity excludes group interests and how people feel about their life's work collapsing under the weight of the evidence. Objectivity should apply equal scepticism to old and accepted theories as old.
These are all fundamental failings of the fundamental scientific process and the ethical logic under which it is supposed to be practised. Modern clinical medicine and psychology is doing something about it.
However, you have chosen an example that goes somewhat beyond the evidence in its speculative argument and that would probably not have been published in clinical psychology. More rational explanations are non-verbal communication or simple determination and desire being involved in a self-fulfilling prophecy.
This is actually the worst of two worlds and a left over of the traditional academic scientific system of practice and is slowly being erased in psychology. It is a culture in which those who collect the data suddenly have the divine right of the person with funding to interpret it at a subjective whim. It is extremely common in academic science too. One ivory tower can contradict another's hard evidence as a result of peer rather than multidisciplinary review.
An example is that the idea of infinite missing links is not consistent with atomistic Mendelean genetics where a single gene can holistically change a creature. Geneticists are now backing Lamarck over Darwin but the evolutionists are entrenched. And many geneticists are in denial of the existence of epigenetics when their belief that DNA is the be all and end all of inheritance is threatened. I know. I have tried talking to them about it.
Just look at science's historical reputation in the resistance to changing direction in the face of the evidence. Clinical evidence is being followed in weeks regardless of the tradition in cultures where Evidence Based Medicine is practised culturally and in the best of psychology too. It is also these areas that are learning to deal with subjective evidence which requires an extra level of rigour if the grain of truth is to be found within the colouration of reporting. It deals with the vastly more complex situations of real life and interactive systems that the academic scientist has the luxury of ignoring.