I Tell me how scientific journals scrutinize statistical work

[phoenix95]

I have been reading statistics for a while (I am a physics major but also a stat-enthusiast), and one of the topics that drew my attention was the misrepresentation, or to be precise, misinterpretation of the data. This came up while reading about Simpson's paradox and the likes. When I see journals that analyze data using statistical tools, I never actually see the data most of the time but I know that the reviewers have access to it. I wonder what the screening/peer-review process for the said studies might be? In other words, how do the reviewers know that the study is misleading?

For instance, take the UK cigarette study example given in this video (time-stamp: 2:21) And the researcher came to you to submit the paper. For a moment assume that he was a rookie grad-student who didn't bother noting the age of the participants. Now the data seems to conclude that smokers have a higher survival rate. But you, the reviewer don't know about the hidden variable that is the age group. So how do you pass the study as publish worthy? Or for the same matter, how do you know that any study is not misleading?

 

[Dale]

There is no one answer to this question. Different journals have different policies and attitudes. That is one reason that different journals have higher or lower credibility than other journals.

In some journals the reviewers are subject matter experts and not statistical experts. They can judge the statistics to the degree that subject matter experts in that field are capable of judging the statistics. If the community has well-defined statistical standards then typically the experts will be able to judge whether a manuscript adheres to those standards. However, if those standards are lacking then there is generally no protection, and if the issues are subtle this may go for decades without even being recognized by the community and then for decades more before the community adapts.

Other journals have a dedicated statistical reviewer. This is a person that is not a subject matter expert but specifically a statistics expert. They are never the only reviewer, but instead conduct a more limited but more in-depth review of the statistical methods in a given paper. They are often able to have a dramatic impact on the statistical quality of a journal, and if it is a leading journal in the field they can have an impact on the whole community as very often people will apply to the leading journal first, get rejected due to statistical issues, and fix those issues either in a submission to a different journal or in their next submission to the leading journal.

This is by no means a solved problem.

 

[berkeman]

I would think that medical journals would have one of the highest needs for good statistical analysis when reviewing article submissions. Perhaps @jim mcnamara can comment...

 

[jim mcnamara]

A rookie is not going to get something published in a journal like Cell, without a co-author who has a reputation to vouch for it. Researchgate pretty much is another matter entirely.

So:
@Dale is correct.

There is an assumption the OP may have that might need a tune up. In other words:
There is another factor here. Let's consider medical only -

Physicians view similar problems differently.
Reports/papers are often clinical observations. Reports of 'what I/we saw'. Therefore: They are opinions. You have heard of 'getting a second opinion' on a tough medical problem.

Meta-analysis reports of these clinical reports often requires a decision what observation A from Dr. Smith versus observation A from Dr Jones 'really means'. Or to exclude, lump, or whatever that second report completely. This goes on through many reports. Think 'hodge-podge'.

So getting exercised about stats on these things, IMO, is not always helpful to a degree. Not like Physics. We have to go with what we have. Or. Would you prefer to be seriously ill, and have a physician say:

'We have no consensus on the validity of research for your disease. Go home. Come back when we all agree.'??

Probably not.

And this obtains across the board from your local chiropractor to the Mayo Clinic. The best generally approved therapy is called 'standard of care for "X" condition'. Usually per the CDC or WHO, etc.
ex: statin therapy and life style changes for elevated LDL-C levels. Research on these is the most scrutinized and reviewed.

It is a little like a menu or a formulay. In the US it is what medical insurance will cover. Off-label (not standard) treatments and drugs have to be insurance approved on a one-by-one basis.