A GCSE Physics Question from the UK

[scooby7]

I have no idea if this is the right forum for this, but I'd genuinely like to hear what real scientists think of this question which is taken from a GCSE Physics test paper.

Frances claims she can read people's minds. She asks them to think of a number.
She says she can tell whether they are thinking of an even or odd number.

She tried this out on 8 of her friends. She repeated it 4 times with each one. These are the results

Friend / No of correct readings (out of 4)
A 1
B 4
C 3
D 2
E 1
F 3
G 3
H 2

She says it is only people with dark hair who are telepathic. They give out "tele-waves". Friends B, C, F and G have dark hair.

Is there a correlation between having dark hair and being telepathic YES/NO

 

[billiards]

Although by no means conclusive I'd say that according to that data there appears to be a correlation. Of course she would need to do it on a lot more friends before she could publish her results!

 

[scooby7]

according to that data there appears to be a correlation.

Interesting - and right at the heart of what "correlation" means.

If 6 people toss a head and 2 people toss a tail, and you then notice that the 6 who tossed heads happen to be wearing shorts, does that mean you can claim a correlation between wearing shorts and tossing heads?

If scientists start writing their hypothesis AFTER they have the test data, we're in for some very dodgy claims

Further, if a scientist was to claim that from a sample of 32 people, 13 had got cancer and the same 13 took baths whilst the others showered, so there was a correlation between taking a bath and getting cancer, I think we'd all be concerned

My dictionery (oxford) defines correlation as "a relationship in which one thing affects or depends on another" - clearly that isn't true

 

[billiards]

Well if you forget the ridiculousness of the context and look at the raw data alone you'd be silly to deny a correlation. In my field (geophysics and geology) you often have to make conclusions based on little data/poor observations. Of course you always have to be aware of the pitfalls of misleading data and have to test your conclusions wherever possible. I suppose its harder in a subject where there is not necessarily a 'right' answer.

 

[scooby7]

The trouble is you can't ignore the context, and particularly in an exam which is designed to educate kids about the potential and limitations of scientific discovery.

In my road all the houses with red doors have oil spots on the drive, so there's a clear correlation (as long as I ignore the fact that only houses with red doors own cars!)

My dictionery (oxford) defines correlation as "a relationship in which one thing affects or depends on another" - clearly that isn't true

 

[AlephZero]

One way of testing the hypothesis: assume the guesses are random and calculate the expected number of people scoring 0 to 4 guesses. Compare with the observations:

Score: Frequency: Expected frequency if results were random
0 0 0.5
1 2 2
2 2 3
3 3 2
4 1 0.5

You could do a standard statistical test (Chi-squared) but it's pretty clear what the result would be: there's no reason to believe the results were non-random.

"If scientists start writing their hypothesis AFTER they have the test data, we're in for some very dodgy claims"...

Of course scientists often invent a hypothesis after they have done an experiment. But they then go on to do MORE experiments to test if the hypothesis is true or not. In this case, having created the hypotheses about dark hair, she should then test other people - preferably without knowing what their hair colour is, e.g. by being in a different room from them.

 

[scooby7]

Of course scientists often invent a hypothesis after they have done an experiment. But they then go on to do MORE experiments to test if the hypothesis is true or not.

I guess that's true in the real world, although I'd rather say that scientists make observations from empirical data after which they formulate a hypothesis which they go on to test.

This example looks to be a case of creating the hypothesis to fit the existing data and that never happens (unless you're performing clinical trials and are on retainer )

 

[billiards]

"My dictionery (oxford) defines correlation as "a relationship in which one thing affects or depends on another" - clearly that isn't true."

In my opinion a good scientist will not shut off the possibility based on their body of knowledge, after all knowledge can be a fragile thing.

 

[scooby7]

In my opinion a good scientist will not shut off the possibility based on their body of knowledge, after all knowledge can be a fragile thing.

Agreed, but remember who the question was aimed at - 15 year olds who don't have the knowledge or experience to be able to make very sophisticated judgements about the "reasonability" of the data.

You or I might play the lottery for fun, but with no real expectation of winning, but a scary number of people when you explain there is a 1 in 14Million chance say "agh but somebody's got to win it!"

Try telling any 15 year old that you've just tossed 10 heads in a row, how likely is the next toss to be a head, and most will "reason" that it must be time for a tail by now!

 

[billiards]

Well I'd be interested to know what the 'right' answer is. If exactly the same data were presented but they changed the words to I dunno, something about springs and it turned out that all the springs that could bear the heaviest loads were also springs with the thickest coil. Then clearly the right answer would be "yes" there's a correlation, but that's because there's a model that can explain that. Just because we can't explain this paranormal stuff doesn't mean we should just write it off without giving further investigation (although I personally wouldn't spend a minute of my time studying it!).

 

[scooby7]

Well the "right" answer is Yes, but I can't agree with it, and it isn't because I'm dismissing any paranormal event, It's because it's irresponsible to start drawing any conclusions from such limited data, and frankly it's stupid to have a yes/no answer.

I think this discussion shows that you can justify either answer, and for me the A* answer would be

"Although from the data there appears to be a strong relationship between hair colour and the ability to read minds, the sample is far too small and a blind test involving a hundereds or even thousands of people would be necessary before any correlation could be claimed. Additionally, as the hair colour claim was only made after the data was studied and not before conducting the experiment, it is only an observation and further work needs to be done to find other (more rational) explinations for the relationship"

 

[scooby7]

I thought of a much better reason why there is no correlation.

By random chance you should score 50%
Those with dark hair scored 81%
The rest scored 37%

If you combine them all, it comes out at 59% which is within the margins of 50% given the small sample size.

If however you are going to claim telepathy is the justification for the "over achieving" of the dark haired friends, how can you explain the failure of the rest which should have come out at 50% even though no telepathy was involved. Both groups are equidistant from the mean so neither set of results has significance

 

[vanesch]

If p is the probability per trial to get it right, then the distribution for 4 trials is:

P(0) = (1-p)^4
P(1) = (1-p)^3 p 4
P(2) = (1-p)^2 p^2 6
P(3) = (1-p) p^3 4
P(4) = p^4

For the dark-haired people, the probability to get 4, 3, 3 and 3 is then:
P_sample = P(4).P(3)^3 = p^(4+3x3) (1-p)^3 64
This function peaks at p = 13/16, so the maximum-likelyhood estimator for p is 13/16. The ratio between f[13/16] (the "maximum likelyhood" estimator) and f[0.5] is 30. This means that the data are 30 times less probable with p = 1/2 than with p = 13/16.

Hence, the set of data is rather in favor of p not equal 1/2 for the set of dark-haired people, although there is still 3% chance for this to be so.

For the complementary set, 1, 2, 1, 2 we have P(1)^2 P(2)^2
P_sample = (1-p)^10 p^6 16x36

The maximum-likelyhood estimator here is p = 3/8.
The ratio of p=1/2 over p=3/8, this time, is simply 0.6. So the data are 60% as likely to come from p=1/2 as they are from p=3/8. In other words, p=1/2 is entirely plausible.

In other words, the claim seems to be true: the second sample is entirely compatible with p=1/2, while the first set of data has only 1/30 the chance to come from p=1/2, as compared to p = 13/16.

Telepathy works, with a confidence interval of 97%.

 

[AlephZero]

Telepathy works, with a confidence interval of 97%.

That's not a valid conclusion unless we know more about the design of the experiment. Most likely, the process was:

1. Do the experiment on 8 people chosen at random
2. Select the 4 people with the highest scores and the 4 with the lowest scores.
3. Finally, look for some difference between those two groups of 4, to "explain" the difference.

In this case, you can't conclude anything. The reason is that there are thousands of possible differences that you could find between two random groups of 4 people, so you are guaranteed to find some difference, if you keep looking for long enough.

All you have shown with your statistical test is that there is a difference in scores between the 4 people with the highest scores and the 4 people with the lowest scores. That's not a very surprising conclusion.

A better test would be to find the probability distribution of the highest 4 scores chosen from 8, and compare that with the highest 4 scores in the experiment. If there is a real effect, you would expect these particular four people to do better than the expected value for the best 4 from 8.

 

[scooby7]

That was EXACTLY the process AlephZero, so you can always find some hypothesis to fit the data. If it hadn't been hair colour it could have been eye colour or shoe size or who had toast for breakfast etc etc...

It's scary that this is being TAUGHT as GCSE Science!

 

[rsk]

Doubtless those who devise such courses and exams think that real science is dull and boring and telepathy might appeal more to kids.

The question might have intended simply to check whether the students understand that the small sample size makes the data at best unrelaiable and at worst useless - but then it's either a 'science' question or a statistics question - it has nothing to do with physics in particular.


(In the UK, at GCSE - ages 14-16 - many students are now obliged to study what's called 'co-ordinated science' - which counts as a double subject and covers phys, chem and bio - ie roughly 2/3 the content of each. Only when they reach post-16 eductation do those students get to choose which sciences they want to study.

Also, for those who are not in the UK, there has been much debate here recently about a new science curriculum which many believe is little more than the discussion of popular science issues - without much in the way of real science content.

http://news.bbc.co.uk/2/hi/uk_news/education/6038638.stm

Sorry for diverting the thread, but it's something that really annoys me...)

 

[kesh]

It's scary that this is being TAUGHT as GCSE Science!

no it's great that 16 year olds are being taught to recognise bad science. iirc gcse science is big on hypothesis testing and other basic principles of science

 

[vanesch]

That's not a valid conclusion unless we know more about the design of the experiment. Most likely, the process was:

1. Do the experiment on 8 people chosen at random
2. Select the 4 people with the highest scores and the 4 with the lowest scores.
3. Finally, look for some difference between those two groups of 4, to "explain" the difference.

That would be cheating. I thought it was: we claim that there is telepathy in the case of darkhaired people, so we take a sample of 8 people, 4 of which are dark-haired, and then do the test.
Of course, if other combinations are "people with red pull-overs", "people with green eyes", etc... then you can conclude anything from anything.
But I'm affraid that we are having prejudices here, because obviously telepathy is not to have had, and we try to find good reasons why the correlation is bogus. If the test was about the effectivity of a protective oil against sun burning, I wonder if one would be so critical. So in fact, what one is doing here is not a maximum-likelyhood estimator, but a Bayesian estimator with a big a priori for p=0.5.

So WE are doing bad science here: we want to enforce the conclusion that there is NO correlation, irrespective of the data, and then try to find an excuse of why the obvious correlation in the data must be explained away.

 

[kesh]

the question implies the dark haired hypothesis comes after the experiment and so this is bad science. the students will get marks if they spot this.

this is what i understand it to mean (and i have taught GCSE science in the UK)

 

[rsk]

Why is it bad science to come up with a hypothesis after looking at experimental or statistical data?

Surely the problem here is the sample size and the fact that other factors have not been considered?

 

[rsk]

This has reminded me of a past paper question which may have been either from GCSE or from a Standard Grade Science paper (Scottish equivalent to GCSE)

There is a table of data showing % of deaths caused by heart disease agaisnt the number of cigarettes smoked per day. The students are asked whether this shows that smoking increases the risk of heart disease.

Some students, of course, say yes, but the answer the examiners are looking for is that although the data suggests there might be a link, we can't say because we're not given any info on other possible factors eg diet, exercise, etc.

This question used a far larger, statistically relevant sample so is different from the telepathy one, but in neither case can I see why forming the hypothesis in the light of the data, after the expt, is the problem.

Wouldn't it be good science, for example, to look at data and recognise that it disproves a previous hypotesis and makes a new one necessary?

I'm genuinely curious about this and about which UK exam boards use this reasoning in GCSE papers.

 

[kesh]

Why is it bad science to come up with a hypothesis after looking at experimental or statistical data?

it's not if you go on to test that hypothesis with new data, but it is if you use the data you looked at to come with the hypothesis to prove the hypothesis.

 

[vanesch]

it's not if you go on to test that hypothesis with new data, but it is if you use the data you looked at to come with the hypothesis to prove the hypothesis.

True. But there is no indication that it is after looking at these data, that Frances decides about the dark-haired people. I would even say that we don't know if this isn't the test data on which to test this hypothesis which had already been deduced from a previous set. Given that the question is to see if there is a correlation between being dark-haired and being telepathic, the question would be moot if this hypothesis were not formulated before the data were taken, so, unless one wants to be obviously obtuse, one can take it that this is objective test data on which one has to test the hypothesis of the correlation between being dark-haired and being telepathic, given that there is nothing else in the description that explicitly indicates that the hypothesis was deduced from the data.

Taking the stance that there "must have been" some non-described feature about the data or the hypothesis which makes the conclusion come out differently, is what I mean, doing bad science. We would like it to come out p=0.5, the data at hand seem to suggest otherwise, so we go on thinking about reasons why this will probably be erroneous. And we come up with things which are not explicitly stated, like saying that the to-be-tested hypothesis was drawn after analysis of the data, in order to justify the negation of the obvious correlation.

As I said, if this were data about a neutral issue for which one didn't have any intuitive indication (such as the effectiveness of a newly develloped drug on the cure of a certain disease, which can, or can not, be effective), one wouldn't go at lengths in claiming that the data were biased, or that the hypothesis were set up after looking at the data.

Imagine that the question is the following:

Johnson and Johnson claim they develloped a new drug against lung cancer. Currently, one has 50% chance to die of lung cancer (with radiotherapy or with chemotherapy) - this is a well-known statistic, so no need to do a control group. As these are long and difficult tests, only a few tests have been done. They tested it on 8 different hospitals, with 4 patients each. Here are the number of survivors after 5 years, for hospital A, B,...

Hospital / No of survivors (out of 4)
A 1
B 4
C 3
D 2
E 1
F 3
G 3
H 2

They claim that only hospitals which use radio-therapy (instead of chemotherapy) give positive results. This might be due to a neutralising effect of the chemotherapy on the drug. Hospitals B, C, F and G use radiotherapy.

Is there a correlation between using the drug successfully and undergoing radiotherapy ? YES/NO

 

[kesh]

True. But there is no indication that it is after looking at these data, that Frances decides about the dark-haired people. I would even say that we don't know if this isn't the test data on which to test this hypothesis which had already been deduced from a previous set. Given that the question is to see if there is a correlation between being dark-haired and being telepathic, the question would be moot if this hypothesis were not formulated before the data were taken, so, unless one wants to be obviously obtuse, one can take it that this is objective test data on which one has to test the hypothesis of the correlation between being dark-haired and being telepathic, given that there is nothing else in the description that explicitly indicates that the hypothesis was deduced from the data.

i think the phrasing of the question suggests the "telepath" came up with the dark haired hypothesis after the experiment. it isn't explicit, it only suggests this. GCSE science questions about experiments are often phrased in chronological sequence. my guess (and I've never marked GCSE science, only taught it, and I'm not trained to, I'm a maths teacher) is that marks will be awarded for measuring correlation, but with extra credit for noting a possible problem with the scientific method, seeing as this very problem will have been taught

 

[vanesch]

i think the phrasing of the question suggests the "telepath" came up with the dark haired hypothesis after the experiment. it isn't explicit, it only suggests this. GCSE science questions about experiments are often phrased in chronological sequence.

I have to admit not even knowing what GCSE stands for ; I was just commenting in general about the problem: I think it is a very bad problem statement, unless the idea was to "tickle" the candidate and try to make him say something coherent (no matter what) on the subject - a kind of "dissertation question", as is typical in France: a rather short sentence upon which you're supposed to build up an entire reasoning. The aim is not to arrive at a specific answer, but rather to show that you can put up yourself a problem statement, and treat it subsequently in a coherent and logical way, with quite some lattitude in exactly what you try to treat and how you treat it, as long as it is pertinent, logically structured and on topic. Although it is usually given on a litterary, historic or philosophical problem rather than on a pure scientific matter - maybe this GCSE thing is the scientific equivalent, I don't know. In that case, it is indeed a good question, but then there is no "right" or "wrong" answer to it.

 

[rsk]

GCSE is the General Certificate of Secondary Education - it's the exams they sit in England Wales and N Ireland at the age of 16.