Book or Course on How to Analyze Scientific Studies?

[RJLiberator]

We all know that there is an abundance of poorly created studies and the media sensationalizes and cherry picks certain elements of these studies.

I am looking to better arm myself against such problems. Currently I'm working on the Khan Academy statistic course to give myself a better simple understanding of stat-terms and how to produce a good report.

What I am looking for here is good books, sites, or available courses to help me through this process. Help me understand peer-reviewed research vs. non-peer reviewed, help me see through the advertisement and be able to identify a good study vs. a poorly done one.

Kind of an odd request, but I figure it was worth a shot to see what people had available.

 

[atyy]

There is no fundamental difference between peer-reviewed and non-peer-reviewed research in terms of correctness. Peer review does not make wrong research right, nor does non-peer-review make right research wrong. However, given that there is so much rubbish out there, non-experts will depend on peer-review to help them decide whether they want to give it a second look.

 

[RJLiberator]

Indeed, I am with you there.

I was trying to come up with a quick example about the type of material I am looking to learn about.

 

[atyy]

Are articles like these helpful?

https://quomodocumque.wordpress.com/tag/sabermetrics/

http://www.science20.com/quantum_diaries_survivor/supernatural_coincidences_and_lookelsewhere_effect

http://resonaances.blogspot.com/2014/09/bicep-what-was-wrong-and-what-was-right.html

Edit: Oops, sorry, I edited a bit while you were replying.

 

[RJLiberator]

Precisely. This breaks down a sample study and explains what things went well and what things didn't go so well -- allowing the reader to truly understand the ramifications of the study.

If there's any, perhaps more general, material on this subject in a book/course form, that would be ideal.

Like a tool-kit to see through the junk.

 

[gleem]

I seriously doubt that there are any general reference material for assessing research paper except for maybe ubiquitous techniques used in analyzing data like statistical analysis of data. To completely assess a research one needs to be familiar with the specifics which vary considerable even in the same disciple as nuclear physics, where a gamma angular correlation expert may not have much to say about neutron beam excitation physics. Peer review is done by persons doing closely related type of work. You wouldn't have a condensed matter physicist review an astrophysics paper. It is best for a non expert to seek critiques of papers from a reliable source. Such a source should precisely identify him/herself and credentials.

 

[micromass]

This should be helpful https://www.amazon.com/dp/0393310728/?tag=pfamazon01-20

 

[RJLiberator]

Interesting. Looks like a good read on this topic. I'll give it a whirl.

 

[tom aaron]

I seriously doubt that there are any general reference material for assessing research paper except for maybe ubiquitous techniques used in analyzing data like statistical analysis of data. To completely assess a research one needs to be familiar with the specifics which vary considerable even in the same disciple as nuclear physics, where a gamma angular correlation expert may not have much to say about neutron beam excitation physics. Peer review is done by persons doing closely related type of work. You wouldn't have a condensed matter physicist review an astrophysics paper. It is best for a non expert to seek critiques of papers from a reliable source. Such a source should precisely identify him/herself and credentials.

So true.

In my discipline we all know each other. 5 or 6 in the world who have an understanding of the research. Nobody outside of this group would be able to peer review or comment on the legitimacy of the methodology used.

As you state, people do not do research in 'nuclear physics' but a specific subset of the field. The same in geology. In my research in biostratigraphy we are in a bubble that is impenetrable to the understanding of most geologists.

Anyways, this is why the reputation of a publication is so important. Is it accepted as part of the total knowledge of science? Can it be used as a reference to support further research?