Reproducibility in condensed matter

[pines-demon]

You might know of Sergey Frolov as the one that uncovered the Delft-Microsoft Majorana confirmation bias affair. Frolov has been advocating for more transparency in condensed matter physics, particularly with experiments.

He just realized a video on his recently created conference on reproducibility:



You gotta love how the Nature people are there to say that it is not their issue but that they are there to help...

Do you think we should have more conferences like these? Have you heard of any recent efforts from your institutions on this topic? Have you heard of any misbehaviours happening systemically?

Edit: some in the video estimate that science misconduct is present in more than 10% of papers.

 

[pines-demon]

The conference report is now out:

• https://arxiv.org/abs/2501.18631 (A. Akrap et al, Report on Reproducibility in Condensed Matter Physics, 2025)

The main take is given in page 4 or so:

Key conclusions of the report

1. Reproducible research offers benefits to researchers and the public. The benefits include validation of findings by fellow scientists (mitigating errors in interpretation, confirmation bias, and fraud), benchmarking of progress by subsequent investigators, follow-on analyses, synergistic effects, and long-term preservation of work and results.
2. Sharing data and code is critical. Increased sharing of primary research materials is necessary to reproduce studies. This practice should be implemented community-wide. Fortunately, the infrastructure for making data available exists in the form of free public repositories, and is adequate for the vast majority of experimental and numerical studies in CMP. Nonetheless, additional technique-specific guidance for which material to share and by what means would be valuable.
3. We need a culture of advocating for reproducibility. In the longer term, making our field robust and replicable will require changes to how research is organized, recorded, and presented.Individual members of the community can shift the status quo by improving their own practices, requesting changes to procedures at our institutions, and promoting changes to policy and its implementation. See the list of recommendations in this report for ideas.

Also the recommendations that follow:

Recommendations of the Report: In Brief

Best practices for condensed matter researchers:

• When reporting experiments, include the full ranges of experimental parameters studied.
• All authors of a manuscript should have access to all primary data and analysis files/software before they are asked to consent to publication.
• The manuscript, its appendices, and supplementary information should contain enough information for reproduction.
• Authors should release as much primary data as possible and practical with the paper.
• Strive to achieve FAIR Data Standard but, if not possible, share materials in the form you have them.
• A manuscript should include all reasonable presentations of the data.
• Authors should specify full data processing steps from original raw data to published figures.
• A manuscript should summarize all plausible interpretations of results.
• Information about the number of samples tested should be reported.
• Research groups should have a stated policy and timeline for releasing in-house developed code.
• Subcommunities of CMP should develop subfield-specific minimum reporting standards.
• Authors should promptly retract or correct their own prior work if they discover errors post publication.

Recommendations for Scientific Publishers:

• Refocus on scientific validity rather than subjective appeal criteria for publication.
• Require availability of complete data and scripts in citable repositories.
• Formulate clear statements of acceptance criteria for reproduction efforts.
• Create review requirements that are specific to supplementary material.
• Develop specific guidelines for assessing the reproducibility of submissions.
• Consider adopting an open review process.
• Draft and publicize more transparent retraction and correction policies and procedures. Recommendations for universities and research laboratories
• Consider not only publications, but also datasets and code as research outputs.
• Recognize reproducibility practices in promotion cases.
• De-emphasize publication indices in evaluating the productivity of individual scientists.
• Follow government-mandated procedures for investigating misconduct, and inquire into the full factual basis of each report.
• Develop a clear definition of unreliable science beyond the definition of misconduct.
• Actively protect whistleblowers.

Recommendations for government and scientific funding organizations

• Consider funding work that explicitly aims to replicate prior results.
• Require grant proposals to address reproducibility in their planning.
• Evaluate and reward the reproducibility of research.
• Develop policies requiring grant recipient organizations to report allegations of scientific misconduct to the funding agency.