A Rough Guide to Spotting Bad Science

Answers and Replies

  • #2
Bystander
Science Advisor
Homework Helper
Gold Member
5,174
1,182
Looks like something that'd be worth pinning here and there around the forum, barring copyright problems, of course.
 
  • #3
18,086
7,510
Looks like something that'd be worth pinning here and there around the forum, barring copyright problems, of course.
I agree, I was thinking the same thing.
 
  • #4
Garth
Science Advisor
Gold Member
3,574
105
Now let's see....

The Announcement of the BICEP2 B Mode polarisation signal in the CMB. First Direct Evidence of Cosmic Inflation

1. Sensationalised Headlines.

The announcement went global and was headline news here on the BBC Inflation: A compact guide to big science and British newspapers.

2. Misinterpreted Results Constraint on the primordial gravitational waves from the joint analysis of BICEP2 and Planck HFI 353 GHz dust polarization data
We make a joint analysis of BICEP2 and recently released Planck HFI 353 GHz dust polarization data, and find that there is no evidence for the primordial gravitational waves
5. Speculative Language Worse than using the speculative 'may', could', and 'might' is to use definite language when you are wrong, or perhaps, not yet right. From the CfA announcement:
Researchers from the BICEP2 collaboration today announced the first direct evidence for this cosmic inflation.....Finally, the data confirm a deep connection between quantum mechanics and general relativity.
We now know there was no evidence (yet discovered), no confirmation of gravitational waves from inflation, and therefore no confirmation (yet) of the deep connection between QM and GR,

6&7. Sample Size Too Small & Unrepresentative Samples BICEP2 looked at a relatively small area of sky, PLANCK looked at the whole sky and found dust, if you look at the the crude map of the PLANCK findings from the CNES site you will see the boxed area and that there was a quieter (dark blue=less dust signal) part of the sky closer to the South Celestial Pole. (Might BICEP2/3 repeat the experiment looking at this part of the sky?)

12. Journals and Citations That's the point - there weren't any! The Havard-Smithsonian Center for Astrophysics went public before they had published in a peer reviewed reputable journal.

My point? It is not only crackpots who are guilty of 'Bad Science'.

Garth
 
Last edited:
  • #5
Doug Huffman
Gold Member
804
110
I would add ad-hockery.
 
  • #6
67
165
Part of that guide could be applied to politicians
 
  • #7
Garth
Science Advisor
Gold Member
3,574
105
Part of that guide could be applied to politicians
O come on, there would be nobody left.....

Garth
 
  • #8
OmCheeto
Gold Member
2,114
2,501
Perhaps we should ( @RonL ) add a poster of how to ( @RonL ) spot good science. hmmmm?

Scientific.method.jpg
 
  • #9
RonL
Gold Member
1,096
215
Perhaps we should ( @RonL ) add a poster of how to ( @RonL ) spot good science. hmmmm?

View attachment 77350
I'm brushing up on how to spot bad science, watching "Warehouse 13" for the third time around on Netflix, just can't figure out why Joanne Kelly makes me feel like Kaw-Liga ?:oldconfused::oldlove:
 
  • #10
OmCheeto
Gold Member
2,114
2,501
I'm brushing up on how to spot bad science, watching "Warehouse 13" for the third time around on Netflix, just can't figure out why Joanne Kelly makes me feel like Kaw-Liga ?:oldconfused::oldlove:
That's somewhat humorous, as I've been brushing up on my science by watching the 1st season of the Twilight Zone. (1959 edition. good stuff! episode 7 sparked an idea on how to get to Mars and back.)

I do not know what "Warehouse 13" is, who Joanne Kelly is, nor what a, um, "Kaw-Liga" is supposed to be.

Anyways..... This was an experiment in how the new forum software works. Did you get 1 or 2 alerts?
 
  • #11
RonL
Gold Member
1,096
215
That's somewhat humorous, as I've been brushing up on my science by watching the 1st season of the Twilight Zone. (1959 edition. good stuff! episode 7 sparked an idea on how to get to Mars and back.)

I do not know what "Warehouse 13" is, who Joanne Kelly is, nor what a, um, "Kaw-Liga" is supposed to be.

Anyways..... This was an experiment in how the new forum software works. Did you get 1 or 2 alerts?
I got the two alerts, they took me to my profile, does that mean they were successful ?
Warehouse 13 is a fun spoofy Si-Fi TV series, Kaw-Liga was a wooden indian, Joanne Kelly is just HOT :oldlove:.
 
Last edited:
  • #12
dlgoff
Science Advisor
Gold Member
3,802
1,704
  • #13
67
165
I don't think this one applies. Although I am somewhat scientifically literate, scientists expert in various fields have vocabularies which I find incomprehensible.

In the following, I've bolded the phrases I am not familiar with:


So do I label Chet a kook, based on the fact that he's talking technobabble?
Or do I look at his credentials? He's a mentor, and they're usually reliable. So perhaps I should google his babble, and maybe learn something.

ps. I was the one who first dropped the term "isosceles", after having to google it. I blame it on the Scarecrow.

hmmm.... I know I can ignore certain members. I wonder if there is a way to ignore certain forums. As, once in awhile, I accidentally wander into threads where I absolutely do not belong.
You wandered into that thread last June and you still remember it? That in itself puts you into a category of intellect that leaves me thinking "How the H-ll does he do that." :D
 
  • #14
OmCheeto
Gold Member
2,114
2,501
You wandered into that thread last June and you still remember it? That in itself puts you into a category of intellect that leaves me thinking "How the H-ll does he do that." :D
It's a long running joke between Drakkith and myself. I'm pretty sure it started before that post.

I liken it to jumping into the deep end of the pool, and realizing that I'm just going to have everyone eventually laugh at me, trying to get out, not knowing how to swim, so I ask the nearest person; "Please get me out of here".
Which he did that day.

@Drakkith , do you remember the first time? I surely don't.
 
  • #15
RonL
Gold Member
1,096
215
It's a long running joke between Drakkith and myself. I'm pretty sure it started before that post.

I liken it to jumping into the deep end of the pool, and realizing that I'm just going to have everyone eventually laugh at me, trying to get out, not knowing how to swim, so I ask the nearest person; "Please get me out of here".
Which he did that day.

@Drakkith , do you remember the first time? I surely don't.
I'm so slow, :( I think I just figured it out....... If I do @Drakkith , and @OmCheeto , you both get an alert to look at this post, no matter what forum it might be in ? right :):rolleyes:

ps. If I make some comment and have the word here, how do I make that a link, that clicking on it opens some intended place or document ? :confused:
 
Last edited:
  • #16
OmCheeto
Gold Member
2,114
2,501
I got the two alerts
Bad.
, they took me to my profile, does that mean they were successful ?
No. That means there's a bug in the software. This is why I used to write all of my own.
Warehouse 13 is a fun spoofy Si-Fi TV series, Kaw-Liga was a wooden indian, Joanne Kelly is just HOT :oldlove:.
I googled those terms this morning. My responses: If you say so, yes, and yes. :)

I'm so slow, :( I think I just figured it out....... If I do @Drakkith , and @OmCheeto , you both get an alert to look at this post, no matter what forum it might be in ? right :):rolleyes:
That is correct.
 
  • #17
RonL
Gold Member
1,096
215
Just witnessed an example of how fast our mods are at clearing out bad science, :nb) @Doug Huffman, I didn't have time to like your reply:eek::D
Something about a fusion engine ?
 
  • #18
WWGD
Science Advisor
Gold Member
2019 Award
5,180
2,496
How about having, e.g., Basketball players vouching for tooth paste. Why would I prefer brand X just because Shak
is sponsoring it --tho he does have shiny teeth.

My prof. used to refer to non-credible results using small sample sizes as "the law of small numbers".

Maybe you can summarize good research when you see the researcher making an honest effort to falsify , i.e., find as many flaws as possible in his/her own research results. Reminds me of Metallica (decrying the opposite) : "fighting the truth, winning is all " (Eye of the Beholder)

EDIT : I mean the researcher, before espousing a theory, should try to disprove it in every possible way, find and address any possible flaw, and after all this fails, the theory stands. I think this is the method some followed in the middle ages; when a theory was presented, all the arguments addressing why the theory could be wrong would be presented and countered.
 
Last edited:
  • #19
34,059
9,932
12. Journals and Citations That's the point - there weren't any! The Havard-Smithsonian Center for Astrophysics went public before they had published in a peer reviewed reputable journal.
To be fair, this is common practice if the field is dominated by large collaborations. They have their internal review processes and get close to 100% approval rate from journals anyway.
Conference talks or similar presentations are the usual way to make important results public, everything else appears on arXiv long before a journal publishes it.


I disagree with point 5 in the first post. Good scientists won't use "we are sure that Y" unless they are really, really certain (an extremely rare case). They will prefer "X is evidence for Y", "X suggests Y", "X could mean Y" or "we observed X" (and leave the conclusions out).
 
  • #20
Drakkith
Staff Emeritus
Science Advisor
20,746
4,457
@Drakkith , do you remember the first time? I surely don't.
I think it involved a bottle of tequila, three waitresses from a nearby diner, a lopsided pool table, and a hot tub. And I'm pretty sure I remember lawn gnomes coming into the picture at some point in the night... didn't you leave the left half of your pants on one?
 
  • #21
jim hardy
Science Advisor
Gold Member
2019 Award
Dearly Missed
9,839
4,869
Add to the twelve points:

Politicians scrambling to tax it
Brokerage industry scrambling to trade futures in it
Insurance industry scrambling to make you buy a policy for it
 
  • #22
Garth
Science Advisor
Gold Member
3,574
105
To be fair, this is common practice if the field is dominated by large collaborations. They have their internal review processes and get close to 100% approval rate from journals anyway.
Conference talks or similar presentations are the usual way to make important results public, everything else appears on arXiv long before a journal publishes it.


I disagree with point 5 in the first post. Good scientists won't use "we are sure that Y" unless they are really, really certain (an extremely rare case). They will prefer "X is evidence for Y", "X suggests Y", "X could mean Y" or "we observed X" (and leave the conclusions out).
Hi mfb!

Yes of course, my point was they shouldn't have gone to the press with it, saying they had definitely found the signal of primordial gravitational waves that was a confirmation of inflation, without time for others to check and counter their conclusions. They had actually found dust.

Of course theoreticians like to believe their hypotheses are true and nearly all have been convinced of inflation for years, they think the gravitational wave signal is there, it must be, hidden in the dust signal.

Observers don't be believe they have found something until they have found it!

I like your comment about point 5.

You can be sure of observations with high statistical significance, the conclusions you draw from those observations may be more contentious......

Garth
 
Last edited:
  • #23
726
27
Part of that guide could be applied to politicians
I don't think so. The stuff that makes science "bad science" is the same that makes a good politician.
 
  • #24
18,086
7,510
The stuff that makes science "bad science" is the same that makes a good politician.
I think you mean to say "successful". A successful politician many times is not a good one.
 
  • #25
WWGD
Science Advisor
Gold Member
2019 Award
5,180
2,496
I don't know why people pick on politicians as if they were of a different species; they are drawn from the same population, same schools, institutions as the rest of society. I don't see them being neither more nor less honest than your average person. And the statement that they lie all the time is nonsense; one must choose when to lie, or one may then never be believed and thus be completely ineffective.
 

Related Threads on A Rough Guide to Spotting Bad Science

  • Last Post
Replies
3
Views
2K
  • Last Post
Replies
8
Views
2K
  • Last Post
Replies
2
Views
4K
  • Last Post
2
Replies
36
Views
3K
  • Last Post
Replies
23
Views
4K
Replies
51
Views
7K
  • Last Post
2
Replies
32
Views
2K
Replies
10
Views
2K
  • Last Post
Replies
15
Views
2K
  • Last Post
Replies
5
Views
5K
Top