Tell me how scientific journals scrutinize statistical work

In summary, the process of reviewing and approving studies for publication varies among journals, with some having subject matter experts review the statistics and others having dedicated statistical reviewers. However, there is no one solution to ensuring the accuracy of research findings, and issues may go unnoticed for years before being addressed by the community. In the medical field, where opinions and observations play a significant role in research, the validity of studies may be more difficult to determine and may rely on consensus among multiple reports. Ultimately, the goal is to establish a standard of care for certain conditions, with research and clinical observations contributing to this standard.
  • #1
phoenix95
Gold Member
81
22
I have been reading statistics for a while (I am a physics major but also a stat-enthusiast), and one of the topics that drew my attention was the misrepresentation, or to be precise, misinterpretation of the data. This came up while reading about Simpson's paradox and the likes. When I see journals that analyze data using statistical tools, I never actually see the data most of the time but I know that the reviewers have access to it. I wonder what the screening/peer-review process for the said studies might be? In other words, how do the reviewers know that the study is misleading?

For instance, take the UK cigarette study example given in this video (time-stamp: 2:21) And the researcher came to you to submit the paper. For a moment assume that he was a rookie grad-student who didn't bother noting the age of the participants. Now the data seems to conclude that smokers have a higher survival rate. But you, the reviewer don't know about the hidden variable that is the age group. So how do you pass the study as publish worthy? Or for the same matter, how do you know that any study is not misleading?
 
Physics news on Phys.org
  • #2
There is no one answer to this question. Different journals have different policies and attitudes. That is one reason that different journals have higher or lower credibility than other journals.

In some journals the reviewers are subject matter experts and not statistical experts. They can judge the statistics to the degree that subject matter experts in that field are capable of judging the statistics. If the community has well-defined statistical standards then typically the experts will be able to judge whether a manuscript adheres to those standards. However, if those standards are lacking then there is generally no protection, and if the issues are subtle this may go for decades without even being recognized by the community and then for decades more before the community adapts.

Other journals have a dedicated statistical reviewer. This is a person that is not a subject matter expert but specifically a statistics expert. They are never the only reviewer, but instead conduct a more limited but more in-depth review of the statistical methods in a given paper. They are often able to have a dramatic impact on the statistical quality of a journal, and if it is a leading journal in the field they can have an impact on the whole community as very often people will apply to the leading journal first, get rejected due to statistical issues, and fix those issues either in a submission to a different journal or in their next submission to the leading journal.

This is by no means a solved problem.
 
  • Like
  • Informative
Likes phoenix95, jim mcnamara and berkeman
  • #3
I would think that medical journals would have one of the highest needs for good statistical analysis when reviewing article submissions. Perhaps @jim mcnamara can comment...
 
  • Like
Likes phoenix95
  • #4
A rookie is not going to get something published in a journal like Cell, without a co-author who has a reputation to vouch for it. Researchgate pretty much is another matter entirely.

So:
@Dale is correct.

There is an assumption the OP may have that might need a tune up. In other words:
There is another factor here. Let's consider medical only -

Physicians view similar problems differently.
Reports/papers are often clinical observations. Reports of 'what I/we saw'. Therefore: They are opinions. You have heard of 'getting a second opinion' on a tough medical problem.

Meta-analysis reports of these clinical reports often requires a decision what observation A from Dr. Smith versus observation A from Dr Jones 'really means'. Or to exclude, lump, or whatever that second report completely. This goes on through many reports. Think 'hodge-podge'.

So getting exercised about stats on these things, IMO, is not always helpful to a degree. Not like Physics. We have to go with what we have. Or. Would you prefer to be seriously ill, and have a physician say:

'We have no consensus on the validity of research for your disease. Go home. Come back when we all agree.'??

Probably not. :frown:

And this obtains across the board from your local chiropractor to the Mayo Clinic. The best generally approved therapy is called 'standard of care for "X" condition'. Usually per the CDC or WHO, etc.
ex: statin therapy and life style changes for elevated LDL-C levels. Research on these is the most scrutinized and reviewed.

It is a little like a menu or a formulay. In the US it is what medical insurance will cover. Off-label (not standard) treatments and drugs have to be insurance approved on a one-by-one basis.
 
  • Informative
  • Like
Likes phoenix95 and berkeman

Related to Tell me how scientific journals scrutinize statistical work

What is the purpose of scrutinizing statistical work in scientific journals?

The purpose of scrutinizing statistical work in scientific journals is to ensure that the results presented are accurate and reliable. This helps to maintain the integrity of scientific research and allows other researchers to replicate and build upon the findings.

How do scientific journals scrutinize statistical work?

Scientific journals scrutinize statistical work by using a peer-review process, where experts in the field evaluate the methodology, data analysis, and conclusions of the study. They also check for any errors or biases in the statistical analysis and ensure that the results are supported by the data.

What are some common statistical methods used in scientific research?

Some common statistical methods used in scientific research include hypothesis testing, regression analysis, and ANOVA (analysis of variance). These methods help to analyze and interpret data, and make conclusions about the relationships between variables.

Why is it important to use proper statistical methods in scientific research?

Using proper statistical methods in scientific research is crucial because it allows for accurate and reliable conclusions to be drawn from the data. It also helps to avoid false or misleading results, which can have significant consequences in fields such as medicine and public policy.

What are some potential issues that can arise in statistical analysis?

Some potential issues that can arise in statistical analysis include biased samples, confounding variables, and incorrect interpretation of results. It is important for researchers to carefully consider these factors and use appropriate statistical methods to address them in their analysis.

Similar threads

  • Sticky
  • Set Theory, Logic, Probability, Statistics
Replies
12
Views
4K
  • Biology and Medical
Replies
5
Views
6K
  • STEM Educators and Teaching
Replies
8
Views
2K
  • Engineering and Comp Sci Homework Help
Replies
4
Views
1K
  • General Discussion
Replies
12
Views
1K
Replies
19
Views
2K
  • STEM Academic Advising
Replies
9
Views
3K
Replies
2
Views
1K
  • Feedback and Announcements
Replies
6
Views
1K
  • General Discussion
Replies
15
Views
2K
Back
Top