The Astronomy Peer Review Process

[VikingF]

SpaceTiger's Note: Taken from https://www.physicsforums.com/showthread.php?t=130680"

Sorry, this question is somewhat off-topic, but aren't the arxiv.org-papers peer-reviewed?

 

[SpaceTiger]

Sorry, this question is somewhat off-topic, but aren't the arxiv.org-papers peer-reviewed?

No, papers are often submitted to peer review at the same time as to arxiv. If this paper just appeared, it likely won't yet have gone through peer review. Also, a paper is not considered "correct" once it has gone through the peer review process. I get the impression that astronomers tend to think of peer reviewed papers as "worthy of consideration".

 

[VikingF]

Well, I am no astronomer though, but I also had the impression that a peer reviewed paper contained at least some quality material, since it was actually peer reviewed.

 

[SpaceTiger]

Well, I am no astronomer though, but I also had the impression that a peer reviewed paper contained at least some quality material, since it was actually peer reviewed.

Unfortunately, the results of the peer review process are erratic. Reviewing a paper takes a lot of time, so many scientists will just rubber stamp any papers that don't appear to be from crackpots. Even carefully reviewed papers will often contain subtle errors that invalidate the entire analysis.

The true peer review process, in my opinion, occurs at the global level. Exciting results will be debated, reproduced, and reused by astronomers around the world. Anything that can survive this kind of scrutiny becomes accepted in the mainstream and is used for subsequent research.

Nothing is off limits, though. Famous and seemingly dogmatic results are constantly being revisited as new data arrives or as we gain theoretical insight. If you're taking anything on faith (or philosophical prejudice), then you're not doing science.

 

[Chaos' lil bro Order]

Unfortunately, the results of the peer review process are erratic. Reviewing a paper takes a lot of time, so many scientists will just rubber stamp any papers that don't appear to be from crackpots. Even carefully reviewed papers will often contain subtle errors that invalidate the entire analysis.

The true peer review process, in my opinion, occurs at the global level. Exciting results will be debated, reproduced, and reused by astronomers around the world. Anything that can survive this kind of scrutiny becomes accepted in the mainstream and is used for subsequent research.

Nothing is off limits, though. Famous and seemingly dogmatic results are constantly being revisited as new data arrives or as we gain theoretical insight. If you're taking anything on faith (or philosophical prejudice), then you're not doing science.

This disturbs me to hear an expert like ST say this. For once a scientist admits that a lot of science is largely based on the rubber stamping of axioms and current trendy theories. I applaud you ST, you are a true explorer of the truth and a huge asset to humanity. If only everyone was objective as you were, we would be much closer to answering a lot more of our best questions.

You are a hero of mine, plain and simple.

 

[J77]

Unfortunately, the results of the peer review process are erratic. Reviewing a paper takes a lot of time, so many scientists will just rubber stamp any papers that don't appear to be from crackpots. Even carefully reviewed papers will often contain subtle errors that invalidate the entire analysis.

That's not true.

If the journal (I'm speaking from a physics point of view here) is of some quality then it will be reviewed by at least two referees. Only on the odd occasion do you get single sentence acceptance - and I've the feeling that that's because the reviewer knows of your work/you've spoken to them about your work.

There are a lot of surplus journals out there which people through rubbish at - particularly, ime from China. However, on the other hand, I see lots of departments only counting journals with impact factors above, say, 0.3 - which can only be a good thing.

You can't dismiss the peer review process.

The only thing that you can say, is not to take things on arxiv as a final polished product.

 

[SpaceTiger]

That's not true.

If the journal (I'm speaking from a physics point of view here) is of some quality then it will be reviewed by at least two referees. Only on the odd occasion do you get single sentence acceptance - and I've the feeling that that's because the reviewer knows of your work/you've spoken to them about your work.

That's quite a straw man. "Rubber stamp" does not refer to one-sentence acceptances alone (though this is extremely unacceptable, in my opinion, even if they know your work). It refers instead to any case in which the reviewer only reads through the paper one or two times and gives superficial comments. I know for a fact that this happens on a regular basis in the major astronomy journals (after all, I work with astronomers and have had papers of my own reviewd) and I also know that a lot of garbage has gotten through as a result. There's not a lot that the journals themselves can do about this, since they're basically trusting the scientists to give a full and honest review.

You can't dismiss the peer review process.

I'm not dismissing the peer review process, I'm just saying that it doesn't mark a paper as "correct", even by a longshot. It's still important to have the process in place (as is obvious from some of the junk the appears on arxiv), but in mainstream academia, I think it serves more to polish pre-existing research than to select correct/incorrect results.

The only thing that you can say, is not to take things on arxiv as a final polished product.

Arxiv is not peer reviewed, so I would go much further and say to treat arxiv papers with extreme skepticism, especially if they're written by unknowns.

 

[J77]

That's quite a straw man. "Rubber stamp" does not refer to one-sentence acceptances alone (though this is extremely unacceptable, in my opinion, even if they know your work). It refers instead to any case in which the reviewer only reads through the paper one or two times and gives superficial comments. I know for a fact that this happens on a regular basis in the major astronomy journals (after all, I work with astronomers and have had papers of my own reviewd) and I also know that a lot of garbage has gotten through as a result. There's not a lot that the journals themselves can do about this, since they're basically trusting the scientists to give a full and honest review.

A straw?

I still would pull you up on the many scientist bit of your OP.

Or perhaps you mean many astronomers just give it the rubber stamp

 

[SpaceTiger]

Or perhaps you mean many astronomers just give it the rubber stamp

I very much doubt that astronomers are alone in this regard, but my experience is mostly in the field of astronomy (and a little bit in physics), so I can only speak for certain about that.

 

[Hacky]

I write and review papers, grants in clinical medicine and basic biomedical research. The situation is the same as what you describe in astronomy. A good review (even if it pans your work) can be worth its weight in gold, but all too often it is obvious the reviewer(s) either did not carefully read your submission, did not take the time to make constructive comments, or did not know the field well enough to do the job properly.

 

[selfAdjoint]

I write and review papers, grants in clinical medicine and basic biomedical research. The situation is the same as what you describe in astronomy. A good review (even if it pans your work) can be worth its weight in gold, but all too often it is obvious the reviewer(s) either did not carefully read your submission, did not take the time to make constructive comments, or did not know the field well enough to do the job properly.

Hacky, isn't it the case that if someone is good, she gets an awful lot of papers to review, and if she gave every on of them the attention it deserves, she'd never have time for her own work? Or so researchers feel?

Are there any suggestions to improve the system? Perhaps a first layer of gypsy postdocs looking for gross errors and the papers that pass that scrutiny get vetted by experienced people?

 

[J77]

Hacky, isn't it the case that if someone is good, she gets an awful lot of papers to review, and if she gave every on of them the attention it deserves, she'd never have time for her own work? Or so researchers feel?

Editors send papers to people who are well-published in that field - they find this out through the reviewers publication history and forms they get you to fill in detailing what subjects you're available for.

(The author may also suggest a group of people they feel could review their paper.)

You can get an awful lot of requests for reviewing, but you can simply decline them, eg. I have a paper to review at the moment so declined one this morning.

And making time to write a decent review is what part of doing research is all about.

 

[Chronos]

On Arxiv, the paper's status is usually evident. Accepted for publication = peer reviewed, Submitted = unreviewed. One must have certain credentials just to be eligible to post papers there [currently by referral from existing eligible posters] so there is at least some semblance of credibility to most papers that appear there. But peer review is at best a superficial process. The authors, in most cases, are the foremost living authorities on the subject of their papers so they have no 'peers' in the strictest sense of the word. The best that you can hope for from the peer review process is a paper that does not suffer from logic bombs, factual inconsistencies, or unreasonable assumptions, methodology or conclusions. In other words, it is not broken. This is not, and should not be confused with saying it will hold water.