A Rough Guide to Spotting Bad Science

[Greg Bernhardt]

 

[Bystander]

Looks like something that'd be worth pinning here and there around the forum, barring copyright problems, of course.

 

[Greg Bernhardt]

Looks like something that'd be worth pinning here and there around the forum, barring copyright problems, of course.

I agree, I was thinking the same thing.

 

[Garth]

Now let's see...

The Announcement of the BICEP2 B Mode polarisation signal in the CMB. http://www.cfa.harvard.edu/news/2014-05

1. Sensationalised Headlines.

The announcement went global and was headline news here on the BBC Inflation: A compact guide to big science and British newspapers.

2. Misinterpreted Results Constraint on the primordial gravitational waves from the joint analysis of BICEP2 and Planck HFI 353 GHz dust polarization data

We make a joint analysis of BICEP2 and recently released Planck HFI 353 GHz dust polarization data, and find that there is no evidence for the primordial gravitational waves

5. Speculative Language Worse than using the speculative 'may', could', and 'might' is to use definite language when you are wrong, or perhaps, not yet right. From the CfA announcement:

Researchers from the BICEP2 collaboration today announced the first direct evidence for this cosmic inflation...Finally, the data confirm a deep connection between quantum mechanics and general relativity.

We now know there was no evidence (yet discovered), no confirmation of gravitational waves from inflation, and therefore no confirmation (yet) of the deep connection between QM and GR,

6&7. Sample Size Too Small & Unrepresentative Samples BICEP2 looked at a relatively small area of sky, PLANCK looked at the whole sky and found dust, if you look at the the crude map of the PLANCK findings from the CNES site you will see the boxed area and that there was a quieter (dark blue=less dust signal) part of the sky closer to the South Celestial Pole. (Might BICEP2/3 repeat the experiment looking at this part of the sky?)

12. Journals and Citations That's the point - there weren't any! The Havard-Smithsonian Center for Astrophysics went public before they had published in a peer reviewed reputable journal.

My point? It is not only crackpots who are guilty of 'Bad Science'.

Garth
http://smsc.cnes.fr/PLANCK/GP_actualites.htm

 

[Doug Huffman]

I would add ad-hockery.

 

[edward]

Part of that guide could be applied to politicians

 

[Garth]

Part of that guide could be applied to politicians

O come on, there would be nobody left...

Garth

 

[OmCheeto]

Perhaps we should ( @RonL ) add a poster of how to ( @RonL ) spot good science. hmmmm?

 

[RonL]

Perhaps we should ( @RonL ) add a poster of how to ( @RonL ) spot good science. hmmmm?

View attachment 77350

I'm brushing up on how to spot bad science, watching "Warehouse 13" for the third time around on Netflix, just can't figure out why Joanne Kelly makes me feel like Kaw-Liga ?

 

[OmCheeto]

I'm brushing up on how to spot bad science, watching "Warehouse 13" for the third time around on Netflix, just can't figure out why Joanne Kelly makes me feel like Kaw-Liga ?

That's somewhat humorous, as I've been brushing up on my science by watching the 1st season of the Twilight Zone. (1959 edition. good stuff! episode 7 sparked an idea on how to get to Mars and back.)

I do not know what "Warehouse 13" is, who Joanne Kelly is, nor what a, um, "Kaw-Liga" is supposed to be.

Anyways... This was an experiment in how the new forum software works. Did you get 1 or 2 alerts?

 

[RonL]

That's somewhat humorous, as I've been brushing up on my science by watching the 1st season of the Twilight Zone. (1959 edition. good stuff! episode 7 sparked an idea on how to get to Mars and back.)

I do not know what "Warehouse 13" is, who Joanne Kelly is, nor what a, um, "Kaw-Liga" is supposed to be.

Anyways... This was an experiment in how the new forum software works. Did you get 1 or 2 alerts?

I got the two alerts, they took me to my profile, does that mean they were successful ?
Warehouse 13 is a fun spoofy Si-Fi TV series, Kaw-Liga was a wooden indian, Joanne Kelly is just HOT .

 

[dlgoff]

I agree, I was thinking the same thing.

How about in a Signature?

 

[edward]

I don't think this one applies. Although I am somewhat scientifically literate, scientists expert in various fields have vocabularies which I find incomprehensible.

In the following, I've bolded the phrases I am not familiar with:So do I label Chet a kook, based on the fact that he's talking technobabble?
Or do I look at his credentials? He's a mentor, and they're usually reliable. So perhaps I should google his babble, and maybe learn something.

ps. I was the one who first dropped the term "isosceles", after having to google it. I blame it on the Scarecrow.

hmmm... I know I can ignore certain members. I wonder if there is a way to ignore certain forums. As, once in awhile, I accidentally wander into threads where I absolutely do not belong.

You wandered into that thread last June and you still remember it? That in itself puts you into a category of intellect that leaves me thinking "How the H-ll does he do that." :D

 

[OmCheeto]

You wandered into that thread last June and you still remember it? That in itself puts you into a category of intellect that leaves me thinking "How the H-ll does he do that." :D

It's a long running joke between Drakkith and myself. I'm pretty sure it started before that post.

I liken it to jumping into the deep end of the pool, and realizing that I'm just going to have everyone eventually laugh at me, trying to get out, not knowing how to swim, so I ask the nearest person; "Please get me out of here".
Which he did that day.

@Drakkith , do you remember the first time? I surely don't.

 

[RonL]

It's a long running joke between Drakkith and myself. I'm pretty sure it started before that post.

I liken it to jumping into the deep end of the pool, and realizing that I'm just going to have everyone eventually laugh at me, trying to get out, not knowing how to swim, so I ask the nearest person; "Please get me out of here".
Which he did that day.

@Drakkith , do you remember the first time? I surely don't.

I'm so slow, :( I think I just figured it out... If I do @Drakkith , and @OmCheeto , you both get an alert to look at this post, no matter what forum it might be in ? right :)

ps. If I make some comment and have the word here, how do I make that a link, that clicking on it opens some intended place or document ?

 

[OmCheeto]

I got the two alerts

Bad.

, they took me to my profile, does that mean they were successful ?

No. That means there's a bug in the software. This is why I used to write all of my own.

Warehouse 13 is a fun spoofy Si-Fi TV series, Kaw-Liga was a wooden indian, Joanne Kelly is just HOT .

I googled those terms this morning. My responses: If you say so, yes, and yes. :)

I'm so slow, :( I think I just figured it out... If I do @Drakkith , and @OmCheeto , you both get an alert to look at this post, no matter what forum it might be in ? right :)

That is correct.

 

[RonL]

Just witnessed an example of how fast our mods are at clearing out bad science, @Doug Huffman, I didn't have time to like your reply:D
Something about a fusion engine ?

 

[WWGD]

How about having, e.g., Basketball players vouching for tooth paste. Why would I prefer brand X just because Shak
is sponsoring it --tho he does have shiny teeth.

My prof. used to refer to non-credible results using small sample sizes as "the law of small numbers".

Maybe you can summarize good research when you see the researcher making an honest effort to falsify , i.e., find as many flaws as possible in his/her own research results. Reminds me of Metallica (decrying the opposite) : "fighting the truth, winning is all " (Eye of the Beholder)

EDIT : I mean the researcher, before espousing a theory, should try to disprove it in every possible way, find and address any possible flaw, and after all this fails, the theory stands. I think this is the method some followed in the middle ages; when a theory was presented, all the arguments addressing why the theory could be wrong would be presented and countered.

 

[mfb]

12. Journals and Citations That's the point - there weren't any! The Havard-Smithsonian Center for Astrophysics went public before they had published in a peer reviewed reputable journal.

To be fair, this is common practice if the field is dominated by large collaborations. They have their internal review processes and get close to 100% approval rate from journals anyway.
Conference talks or similar presentations are the usual way to make important results public, everything else appears on arXiv long before a journal publishes it.I disagree with point 5 in the first post. Good scientists won't use "we are sure that Y" unless they are really, really certain (an extremely rare case). They will prefer "X is evidence for Y", "X suggests Y", "X could mean Y" or "we observed X" (and leave the conclusions out).

 

[Drakkith]

@Drakkith , do you remember the first time? I surely don't.

I think it involved a bottle of tequila, three waitresses from a nearby diner, a lopsided pool table, and a hot tub. And I'm pretty sure I remember lawn gnomes coming into the picture at some point in the night... didn't you leave the left half of your pants on one?

 

[jim hardy]

Add to the twelve points:

Politicians scrambling to tax it
Brokerage industry scrambling to trade futures in it
Insurance industry scrambling to make you buy a policy for it

 

[Garth]

To be fair, this is common practice if the field is dominated by large collaborations. They have their internal review processes and get close to 100% approval rate from journals anyway.
Conference talks or similar presentations are the usual way to make important results public, everything else appears on arXiv long before a journal publishes it.I disagree with point 5 in the first post. Good scientists won't use "we are sure that Y" unless they are really, really certain (an extremely rare case). They will prefer "X is evidence for Y", "X suggests Y", "X could mean Y" or "we observed X" (and leave the conclusions out).

Hi mfb!

Yes of course, my point was they shouldn't have gone to the press with it, saying they had definitely found the signal of primordial gravitational waves that was a confirmation of inflation, without time for others to check and counter their conclusions. They had actually found dust.

Of course theoreticians like to believe their hypotheses are true and nearly all have been convinced of inflation for years, they think the gravitational wave signal is there, it must be, hidden in the dust signal.

Observers don't be believe they have found something until they have found it!

I like your comment about point 5.

You can be sure of observations with high statistical significance, the conclusions you draw from those observations may be more contentious...

Garth

 

[Nikitin]

Part of that guide could be applied to politicians

I don't think so. The stuff that makes science "bad science" is the same that makes a good politician.

 

[Greg Bernhardt]

The stuff that makes science "bad science" is the same that makes a good politician.

I think you mean to say "successful". A successful politician many times is not a good one.

 

[WWGD]

I don't know why people pick on politicians as if they were of a different species; they are drawn from the same population, same schools, institutions as the rest of society. I don't see them being neither more nor less honest than your average person. And the statement that they lie all the time is nonsense; one must choose when to lie, or one may then never be believed and thus be completely ineffective.

 

[Nikitin]

I think you mean to say "successful". A successful politician many times is not a good one.

Yes I meant good as in skilled, successful. And IMO lying and manipulation is the main trait of a successful politician. Nowadays the politicians in charge are running their campaign, not their country.

 

[Dotini]

And IMO lying and manipulation is the main trait of a successful politician.

And the statement that they lie all the time is nonsense; one must choose when to lie, or one may then never be believed and thus be completely ineffective.

I'm interested in these and other observations on lies. My hypothesis is that sometimes a lie is preferable to an ugly truth. I think three kinds of lie can be justified: beautiful lies, useful lies, and necessary lies.

 

[OmCheeto]

I'm interested in these and other observations on lies. My hypothesis is that sometimes a lie is preferable to an ugly truth. I think three kinds of lie can be justified: beautiful lies, useful lies, and necessary lies.

Reminds me of an argument I had with my "Writing 101" instructor, some 30 years ago.

Naive Young Om; "I hate political rhetoric. They're just a bunch of liars. Rhetoric is stupid."
Instructor; "Rhetoric, is beautiful".

In spite of our disagreement, he gave me an "A".

Rhetoric: the art of effective or persuasive speaking

...

Brain Bam!

Sometimes, I don't, speak, right. But yet, I know, what I'm talking about.​

Whenever I hear a lie, even when beautifully spoken, it makes my brain short circuit. And hence, @$&((#$@%&(%(%#^%#...

hmmm...

But Jordan is particularly proud of “Why Can’t We Be Friends?” The song immediately made history when NASA beamed it to the Apollo-Soyuz Test Project in 1975. As Soviet cosmonauts and U.S. astronauts joined together in the cooperative effort, Jordan watched the drama unfold on television with his track playing up in the heavens. “My mind was blown when I heard it playing in space,” Jordan says, still thrilled at the memory. “I love space travel. I haven’t done it physically, but my music has.”

----------------------------
As usual, it's ok to delete this post, as I've saved my thoughts. This should be a new topic.