Frame Dragger said:
There is too much information out there for anyone with a job and a life to NOT depend on peer-review, not to mention that I don't automatically trust a reviewed paper. I automatically DISTRUST ones that are not submitted, although I'm happy to read them given the time.
I guess I am quite privileged in that I received very critical literature training where we were encouraged to study popular texts and analyze and critique the information and claims-making in them. Once you have done this, it changes the way you look at peer-reviewed texts because you can see how many of the rhetorical and logical shortcomings of non-reviewed literature actually appears in reviewed lit as well. Then your mouth drops open realizing that readers of peer-reviewed lit, perhaps only some, are accepting information uncritically because of the status they attribute to the publication and peer-review generally.
Generally, a rigorous approach to reading critically should entail treating citations and peer-review as additional information applied to evaluation of the text. In that sense, peer-review and source-citation are added values to the extent that they make it easier to see how the writer came up with certain ideas or information and why they might say (or get away with saying) certain things and not others, i.e. because of review by certain peers and not others for example.
What people shouldn't do, which I have dealt with many times, is shoot into defense of the validity of a text or piece of information/knowledge based on no reason except reference to the quality/status of the source/writer/etc. The very fact that something is peer-reviewed doesn't make it more valid. It just means that the particular individuals who reviewed it allowed it to pass as publishable. It's not like editors publish the reasons why reviewers recommended a particular piece for publication.
The idea is that if it is bad it would get filtered out, but no one ever asks what passes for bad and good to whom, and for what reasons.
I come from a field in which everything MUST be peer-reviewed or people don't get well, become sicker, and/or die. Did that stop The Lancet from setting off a panic about vaccinations and a posssible link with ASDs? No, so clearly healthy skepticism is warrented. That said, do you have any idea how much material is produced in a given sub-field of a field in a science? Hint: More than you could "ultimately [evaluate] according to your own knowlede and authority".
More than anything, critical analytical reading is about treating texts as archeological artifacts. It's like digging up a pot and asking how the pot could have been made, what materials were used, etc. You may not be able to evaluate every aspect of the research behind a publication just based on the publication, but from the language used by the writer you can get a good idea of how they think and what the shortcomings and strengths of their research are.
You never accept information except tentatively and critically, and that is why source-citation and explicit reasoning are valuable aspects of a publication. I.e. they make your criticism and reliance on tentative truth easier to process. Ultimately the responsibility will be yours if you act on information in a text, whether it is peer-reviewed or not. You are right that more heads are better than less when it comes to subjecting information to (multiple) authorities. However, you should also pay attention to the fact that sometimes people become less critical in a peer-authority situation out of social politics, i.e. they don't want to deviate from norms and expectations of their peers. Likewise, just as in non-scientific culture, people sometimes resort to blatantly attacking some texts or people to inflate the status of the texts and people they accept as legitimate. People just do this because it's less risky than sticking your neck out to exercise truly independent judgement, which would be more rigorous but more likely to put you in conflict with peer-authority.
Furthermore, the notion expressed in the sentence I quoted is utterly contrary to the scientific method, not just modern practice. Naturally you use your mind, and tune it as best you can for the occasion, but if you're "The Authority" even very very bright folks run into terrible trouble. It is through constant review by as many people as possible (peer-review, say JAMA, establishes a standard which its readers ALSO DEMAND) including the readers of peer-reviewed work that allows for mistakes by even such august insitutions as The Lancet to retract an error, and admit their folly.
This is a hard concept, I think, for many people due to the way authority was treated in their academic training. No one is ever "The Authority," in the sense of being the decisive authority on anything in
democratic science. Authoritarian science is a different matter, but imo science is never supposed to be authoritarian or autocratic - in fact, "authoritarian science" is an oxymoron imo.
The whole great thing about science is that it emerged as the technique of checking knowledge through empiricism, testing, and experimentation. The complement of this
empirical criticism is the critique that takes place at the level of theory and methodology. Ultimately, when you accept some piece of information or knowledge, you do so tentatively with the acceptance that it could later turn out to be wrong. So when you accept someone else's authority, you do so based on your own authority. You can't blame the peer-reviewers or the author for what you do with their text. You can only blame them for being wrong.
Finally, if Kip Thorne wrote a piece of Popular Science... I wouldn't cite it. Is he the real deal? Sure! What would I do?... find a proper citation. If you disagree with the entire notion of citation, you're going to be MISERABLE here, and in science, medicine, debate, politics, law...
If you read it and it informs your thinking you SHOULD cite it and explain how it influenced your thinking and how you were critical of it. If you were reading someone else's paper and they had read something in Popular Science that informed their thinking, wouldn't you want to know how? The problem is that many people will automatically reject a writer/text where pop.sci is cited at all, which puts writers in the position of hiding their influences. This whole situation makes for bad science and it would all be resolved if readers could grow up and move beyond accepting or rejecting texts/writers based on status cues and simply take literature at face value and subject it to their own critical reasoning and authority in whatever form it takes.
...Because ideally you're right, but as none of us live forever we have to choose what to read.
That's a bad reason to choose what you read. A better reason would be to select literature based on your interest and a certain estimate of utility. Learn to read selectively and critically to evaluate whether a particular text is useful to the goal you are pursuing in the current project. Know what you're reading for before you begin your literature search.
