Before I continue, I need to make a short digression to make crystal clear that my suggestions in principle have nothing whatsoever to do with nonsensical SJW measures, equal social diversity rates or unproven/counterproductive anti-sexual harassment proposals. Carrying on.
king vitamin said:
What criterion for "validity" are you using for this? And what consequences do you propose for theoreticians breaching your rules of validity?
My rough operationalization of validity is w.r.t. conclusions of the researcher based on expert appraisal of the quality and originality of the researcher's chosen methodology. The experts should consist of a 'jury' of, let us say about 5 to 8, independent active practitioners in that same subfield working at least 15 years, picked at random from the pool of all experts in that subfield.
I am definitely not the one to decide what validity is; this panel of experts in literally every (sub)field of physics would need to somewhat aristocratically decide every few years what constitutes quality in new research and periodically publish these recommendations as guidelines for the subfield: I would merely suggest their independent judgements to be ordinally ranked, e.g. as 'high quality', 'average quality', 'low quality', and explicitly explained how this reflects the expert opinion within the subfield and why.
The actual establishment of what constitues quality and originality is far more difficult to establish than it seems to be, since quality can vary over time both in and between subfields and originality is even more slippery; there are already highly advanced methodologies and measures invented to measure exactly such things, e.g. topological data analysis and dynamic network analysis directly comes to mind. Moreover, if some research is of a highly interdisciplinary nature it might need to be judged both by experts in both fields as well as by a combination of experts in both fields.
To make this more concrete I will give an example: what constituted high quality research methodology in optomechanics in 2010 doesn't necessarily constitute high quality research methodology in the same subfield in 2018, nor does it necessarily constitute high quality research methodology in another subfield of physics, e.g. in high temperature superconductivity. There are even curious differences, e.g. a novel mathematically advanced methodology invented in high energy physics may literally exist under another name in other older fields such as fluid dynamics and even be regarded as pedestrian within that subfield, since their own methodologies have strongly evolved since.
king vitamin said:
I have seen many theorists - some with Nobel prizes - discredited within academia for their contributions being low-caliber. As far as I can tell, the old guard is not sacrosanct.
I agree that this is a problem, which is exactly why I believe when judging validity of conclusions one needs to take into account both quality and originality. Moreover, it might be that anyone who has a Nobel Prize would need to get judged in an altogether different manner than non-Nobel laureates.
king vitamin said:
This it totally unfamiliar to my experience as a research physicist - in fact I can thing of counterexamples. Can you please give explicit example before such a damning accusation?
I have done research in multiple fields in science (physics, neuroscience, economics, data science, medicine, psychology). In all of them I have invariably seen many researchers and practitioners - both consciously and unconsciously - take shortcuts and cut corners at times, for a variety of reasons: lack of time, frustration with co-workers, not receiving payment for a particular aspect of work, hyping work purely to get funds, work towards achieving performace indices instead of actually trying to perform high quality work, choose successful lower risk strategies of known low utility even when there are more promising strategies available but which have a higher risk meaning a smaller chance of publishing in a high impact journal and therefore choose for their career instead of for bettering science, not publishing negative findings, not speaking up against results in fear of risking their position/careers etc.
My intention is not to judge any researchers, but instead to make them aware that, despite any intentions, they are human and that they are therefore susceptible to the same biases and behavioral traits as other humans. This also means that the directly perceived consequences and appreciation of their work by not only other researchers but also in general has an effect on how they do their work.
king vitamin said:
I'm just confused by this entire paragraph. All work in science is peer reviewed, period. What are you talking about? What "groups of experts in the same field" do you propose who aren't already doing all the peer reviewing? It's the same people!
There is an argument to be made that the peer review system as is a bit too opaque. This invariably leads to clique formation, i.e. counterproductive group behavior making the experts more homogeneous in thinking and behavior than is warranted or reflective of actual practice. The usually used performance indices such as citation indices cannot control for this either since they are far too simplified, focussing mostly on the productivity of individual researchers in terms of papers and short term progress of a subfield than on subfields and the longer term picture; moreover, the advanced measures required cannot be properly chosen or wielded by administrators or regulators if they do not possesses the necessary mathematical background for understanding these tools.
There is a large amount of empirical research demonstrating this and what exactly are negative consequences for a field if such things are left unchecked. Consequences of making such strategic mistakes in theoretical physics don't seem as dire as e.g. in engineering; such differences in the appreciation of consequences among practitioners literally leads to the very idea that therefore there may be a good case to be made that cutting certain corners can be justified for whatever reason, e.g. heuristic or aesthetic reasons as Hossenfelder points out. Every junior researcher invariably mimics both the good and bad of coworkers and more experienced researchers when learning how to do research and how to survive as a researcher in practice; before long they may have developed a strategy that they need to keep to in order to survive or even thrive in practice, independent of increasing the quality of their papers.
In the practice of theoretical physics, this would largely translate to a lack of innovation in theory production and an excess of mimicking behavior, larger than should be expected based on the makeup of both the population of research programmes and researchers. In the ideal case, higher quality would lead not merely to more funds, but to a higher wage as well; I believe the relative lowness of physics wages actually causes much of the 'cutting corners'/lack of innovation problems in physics at the microeconomic and psychological level for individuals.
king vitamin said:
I'm not going to lie - the more your line of though goes on, the more it resembles the Cultural Revolution rather than an actual scientist interested in the truth. (And I feel the need to say that I am a leftist who is not using the Cultural Revolution as a red scare tactic: I really do feel like this is an anti-intellectual attack.)
I fully understand your trepidation. I am not saying that this needs to be done per se, I am saying that there is a good case to be made that physicists should themselves start wanting to do this for all the benefits; I think researchers like Hossenfelder see this as well. The fact is that peer review systems in all professional endeavors have evolved with time, in order to adapt to the environmental changes in their field (amount of practictioners, amount of funds, amount of research programmes, breakthroughs etc).
The review system in the practice of physics, apart from the arxiv, on the other hand seems to have stagnantly remained constant, even directly challenging innovation, despite extreme changes in the landscape of actual practice: the amount of novel mathematical, statistical and computational tools and techniques available alone has already exploded to such a degree that it somewhat of a mystery that physicists, given the choice, tend to stick to relatively outdated or less potent known techniques.
It is somewhat puzzling to me that (theoretical) physics seems to be the only STEM discipline that seem to do this to such a large extent; it is almost as if the familiar techniques are already too much that their wants and needs were already oversaturated 50 years ago, and that the utility of most new techniques can not even be judged because the backlog of available techniques has become so large.
Another thing is that most physicists, myself included, will eventually start complaining that they just want to get to the physics instead of worrying about such matters; the problem is that no one else is properly equipped to do this job except for physicists. For contrast and example, in clinical medicine the exact same thing occurred; the physicians eventually realized this for themselves, and over a period of 50 years encorporated the updated peer review process into the actual practice of doing clinical medicine. Their review system is continuously monitored and guidelines updated periodically by large councils consisting of practicing and retired expert physicians.
king vitamin said:
More directly: who is it that you think should judge what constitutes good cosmology paper, if not the peer reviewers or the relevant journals? Who are these "independent groups of experts in the same field" (as though they are not refereeing journals already)?
If you could not tell from my previous paragraphs (and the context of our conversation), I'm worried that you want these fields to be judged by those who are not experts.
As I said before: a small group of independent practicing experts in the same subfield with about 15 years or more of practical experience; the best thing would be if each expert could be working in a competing research programme. In order to make it even more clear: every single practicing physicist would eventually reach expert status and therefore need to be able to do this on the fly.
It should be clear that what I am describing here does not yet exist in practice for physics. I think the best institutions who could set up such a programme are large instutions such as the APS and the biggest research journals. One thing is sure: in order to carry out such a research quality assurance management, far more physicists would need to be trained than are being trained today; I see this as a very good thing, not in the least because it creates a new class of jobs which cannot disappear.
A potential strong benefit is that physicists who become experts in doing this could be hired by institutions of other professions to review their respective research methodologies w.r.t. physics, especially in government and healthcare; the irony is that this actually happened more during the 20th century, but then stopped for a large variety of reasons. In any case, I can tell you that right now that most practitioners in most other fields know practically no physics and have never had any significant professional interaction with a physicist or mathematician; the negative consequences of this should be starkly obvious for both parties.
