How reliable are peer reviewed scientific articles?

[Wax]

How reliable are peer reviewed scientific articles? I'm asking this question because someone on http://slickdeals.net/forums/showpost.php?p=25802864&postcount=913" claims that you can pick who reviews your article so you're not really getting an objective point of view. This was in a global warming debate thread.

Speaking of the peer review, and anyone who has published will know about this...everytime you submit a paper, they ask you for a preferred list of reviewers and ask you who you would like to exclude. This is pretty much the process of every peer review journal! So for anyone to think it is a pure art has never published! I am not saying it is horrible, but it is hardly objective anymore.

Interestingly, I just submitted a paper last week to a journal with an impact factor over 5. I received an email saying I had to provide 4 reviewers. It has always been optional on most journals I have submitted to but this was the first time I had to pick my own reviewers...and it is perhaps the most well known journal in the biopharmaceutical industry. So that is peer review...choose your reviewers. One will be a friend of a co-author who asked him to review an article last year. Two others will work at the same company as another co-author (she took another position before the manuscript was written). I am going to ask my director for a fourth...more than likely...it will be someone we do a lot of business with and understands the science, but also a friend of my director. All four reviewers will be experts in their respective field, but they will also have an association with our company.

I really do laugh when people think peer review is truly some altruistic and objective method!

 

[Borek]

How reliable are peer reviewed scientific articles?

To paraphrase Churchill - peer review is the worst form of assessing scientific papers except all the others that have been tried.

 

[Chronos]

Good question, how reliable are character references by relatives? Peer review is as good as the peers reviewing. I think academic peer reviews are reliable. Private publishers are less reliable. A better gauge of paper quality is how many citations they receive in peer reviewed journals, IMO.

 

[f95toli]

How reliable are peer reviewed scientific articles? I'm asking this question because someone on http://slickdeals.net/forums/showpost.php?p=25802864&postcount=913" claims that you can pick who reviews your article so you're not really getting an objective point of view. This was in a global warming debate thread.

This is simply not correct. Many journals will ask you to suggest a few referees and the editor might then use one of them; that reviewer will then be used in addition to another referee which the editor will pick from his/her internal list (I don't think any journal picks BOTH referees from the list suggested by the authors). The reason for this system is simply that it is impossible for the editors to keep track of who is an expert in what field, by doing it this way at least one referee is (hopefully) truly an expert whereas the other (picked by the editor) will have some more "general" knowledge of the topic.

Also, far from all journals use this system. Many prestigious journals won't let you suggest anyone, and they tend to use more than two referees (sometimes as many as four). But this is of course only be viable for well-known and well-funded journals (which can afford to have many editors, one or more for each sub-field) that are also able to persuade a lot of people to referee for them.

Also, I am very rarely asked to review papers written by someone I know, and even when it IS someone I know it is usually just someone I met professionally at e.g a conference.
I would obviously never agree to review a paper written by a friend or someone I collaborate with.

 

[Wallace]

There is also a common misconception that the peer review process is designed to ensure a paper is 'right' or something like that. All peer review does is assessing whether any errors have been made, and that the reasoning is sound. Referees can (and frequently do) vehemently disagree with the the findings of a paper, but will not reject it unless there is a specific mistake or error in logic.

The real peer review comes after a paper is published (or these days wheen pre-prints are common even before it is published) when the whole field gets to read and comment on a paper. Peer review is sometimes put on a pedestal in a well meaning but not entirely helpful way. By holding 'peer review' too sacredly, it leaves open the debating tactics used by 'climate sceptics' that because there exists peer reviewed papers that argue against AGW then that means 'the science isn't settled' or some other non-sensical phrase.

 

[sylas]

I've recently been blogging on this topic, and have repeated some of the comments that are here in this thread. So if you will excuse an advertisement... My physicsforums blog currently has three relevant articles:

• https://www.physicsforums.com/blog.php?b=1473 This starts the ball rolling; and I consider it especially in the light of PF guidelines. I mention peer review as a distinctive feature of mainstream science, but note that it is neither necessary nor sufficient.
• https://www.physicsforums.com/blog.php?b=1476 . One interesting feature of things that want to be science but don't quite live up to the usual expectations is that people make up journals that look a bit like a peer reviewed journal. I list some examples.
• https://www.physicsforums.com/blog.php?b=1493 . This is the most relevant. I start off the blog article with the point made by Wallace:
  
  Peer review, even when it works well, is not an assurance that an idea is correct, or that it is free of errors, or that the hypothesis argued in a paper should now be recognized by default. It only means that the paper has been deemed worthy of consideration by other scientists.​
  But the blog considers a few of those unusual cases where absolute nonsense gets into a mainstream peer reviewed journal. Not merely bad papers -- but papers that leave you wondering what the reviewers were smoking, let along the authors. (https://www.physicsforums.com/blog.php?bt=2574 I also include Borek's Churchill paraphrase, along with a link to an article on Winston Churchill and peer review.)

I am very interested to see more examples in the comments of journals that try to emulate the appearance of a real peer reviewed science journal but merely end up being a grotesque parody, or articles that get into a conventional peer reviewed journal despite being not merely bad, but mind-numbingly awful. If anyone knows examples of either of these, in any field of science, please come and tell me in the comments!

Thanks -- sylas

 

[Wallace]

Well put Sylas. At some level you can't really define 'science' (many philisophers of science have tried, and nothing is watertight yet), so trying to define 'mainstream science' in any hard and fast way will also be difficult. What you're doing is exploring some of the grey areas. I'm sure there is some interesting insight to be gained by doing so.

The ultimate test of science I think is utility. Or put another way, Engineering. If you can Engineer something based on the science, then you can be pretty sure you're doing it right. I think this is why areas like Cosmology or Climate Science have some difficulties, because we can't engineer a new Universe to study and neither can we Engineer a new climate (although some would argue that is effectively what we're doing at the moment, like it or not!).

In the end there is nothing really concrete you can point to, so you have to rely on weight of opinion. That's where snake oil salesmen can do a better job at selling their opinion than scientists. AGW 'sceptism', Intelijent Dezine, various crazy cosmology theories etc. Mainstream science can't point to something and say "we must be right, beacuse look at what we built, and just look at that baby go!" in these cases.

I'm sure these kinds of questions keep a bunch of philosophers and socioligists of science in business! :)

 

[f95toli]

There is also a common misconception that the peer review process is designed to ensure a paper is 'right' or something like that. All peer review does is assessing whether any errors have been made, and that the reasoning is sound. Referees can (and frequently do) vehemently disagree with the the findings of a paper, but will not reject it unless there is a specific mistake or error in logic.

Indeed, and even outright errors are not that uncommon. A few years ago I was working on a topic where the experiments are notoriously difficult to perform and you had to be VERY careful about double-checking everything. I was part of a group (i was still a PhD student) that did some pioneering work using samples made from a particular material; the work was published in Science and received quite a lot of attention. What happened then was of course that OTHER groups then tried to do similar measurements on other materials (usually a material they were already working on) but since they had no experience with this type of measurement neither they nor the referees were aware of the checks one has to do. So the following year a whole string of papers were published and at least half of them were based on measurements that were incorrect or contained conclusions that were simply wrong (I met some of the authors at various conferences and it was quite clear that they did not know what they were doing, I did my best to explain to them what they were doing wrong, with varying degree of success).

However, the following year a new batch of papers were published; this time the measurements were done by people who new what they were doing (and in some cases by the same groups as before, but now they new what they were doing since they had some experience AND had received feedback on their first papers) and most of the papers were fine. That first batch of papers is now long forgotten and no one cites them.

The point is that you should never trust a single paper when it comes to very novel results, but science is self-correcting and papers that stand up to scrutiny over a long period of time
are usually OK, especially if the results have been independently reproduced by other groups.

 

[Wallace]

Yes, I've also had my results refuted using mistaken methods in a peer reviewed paper. Eventually the other group did informally agree that they got it wrong, but nothing was put into print. No one really cites either of our papers though :)

 

[27Thousand]

How reliable are peer reviewed scientific articles? I'm asking this question because someone on http://slickdeals.net/forums/showpost.php?p=25802864&postcount=913" claims that you can pick who reviews your article so you're not really getting an objective point of view. This was in a global warming debate thread.

Well, if you have a choice between trusting something written by experts in the field reviewed by many other experts in the field, versus relying on infomercials or politicians, what would you say is more trustworthy? Although the Scientific Method can't prove, what makes you feel more at ease, if the method is used and put in a peer-review journal or if it's not? Although nothing can be proven 100% the truth and nothing but the truth, what's more trustworthy?

I hear infomercials about various health practices and nutrients. I wonder if their claims have been backed up by randomized experimental-control studies. Since infomercials are not necessarily experts nor reviewed by other experts, well you get my point. Thus, I feel more at ease if I find it comes from a peer-review journal and I can look at the article.

As far as not being able to prove for sure if it's truth, I'm a big fan of the Philosophy of Science and am interested in many arguments the anti-realists in Science use. They say you can't prove electrons exist since we haven't seen them under an electron microscope. They say all we know is that electrons are really good instruments for making predictions of what will happen empirically, so thus that's why it's good for technology and passes research tests (similar to "fail to be disproved but not necessarily real"). So what I think applies to your thread, the scientific realists' best argument against scientific anti-realism, is although you can't prove you can say such and such is the best explanation for the evidence.