How to analyze strange claims?

[Ivan Seeking]

For you statisticians out there, I have long wondered about the possibility of evaluating the "truth" of personal claims by using statistical analysis. Considering the incredible success of demographics and other fields, might this also be possible. If you look at my other threads you will find that I make the argument at times that we cannot readily dismiss claims of the paranormal simply because we lack sufficient evidence. Could scientific evidence exist - one way or the other- in the statistical evaluation of a well designed questionare...for claims of ghost encounters for example?

 

[suffian]

Are you trying to say we could possibly use statistics to analyze strange claims and then gauge how likely they are to be true?

If that's the case, I personally think most people would agree that it would be possible to analyze 'claims' using statistics, but once you took into account complex subjects like human psychology, the results would be near nill that such strange claims are actually true anyways. In other words, it sounds like a daunting task for which we already know the answer.

Of course, sometimes people can be surprised.

 

[selfAdjoint]

One type of case is the story you sometimes see of someone who dreamed of an airplane crash, and sure enough, next day an airplane crashed. In a nation of almost 300 million people, somebody's going to be dreaming about airplane crashes every night! Is that the kind of thing you meant?

 

[Ivan Seeking]

I mean an estimate such as: 42% certainty that at least 3% of 10,000 claims are true...or odds of 1:3 that at least one true and accurate story is found. Something like this; not a comment on any particular case but a statistical evaluation of the complete results. The psychology is the part that I understand - from other surveys discussed somewhere along the line - can be defeated given a well designed test.

Also, I mean situations where honesty is the issue. In many situations, the honesty [or sanity] of the claimant is the most significant question. I also mean for people who have experienced something...not who claim to do something.

I would love to hash out the reasons for pursuing such issues but I suspect I should keep these arguments in the pseudo-science section; this is of course only until I win the argument. But I will give you all time to catch up with me first.

Take me on in the PS section anytime. I already have some posts there if you wish to review and beat-up on a couple of lines of reasoning.

For a related summary, please see the end of this discussion in the Theoretical Physics Forum:
Definitions and limits

 

[selfAdjoint]

Professor J.B. Rhine, who did ESP experiments, used to use that simple kind of statistics. He would report that some subject had called x cards in a row, unseen, and that was one chance in scabillions. But he never reported the unsuccessful calls, so you couldn't extimate the probability of runs. And then it turned out that his experimental setups, which were managed by an assistant, had holes in them that would maybe let the subject know what the cards were.

 

[Ivan Seeking]

I have read about the Rhine history a little. I wasn't aware of claims of false data, but I know that they have never produced convincing results. Still, this is a different animal that what I was thinking.

All that is really needed is a statistical method to determine the honesty of claims. Again, no specific claim being the point, rather a conclusion based on large numbers of reports - based on a statistical analysis of carefully engineered questions, how many of these claim are likely true? It seems to me that I have heard or read that well designed surveys include questions designed for just such purposes. Then some kind of mathematical analysis yields results with +- margins of error for accuracy, or in my example, truthfulness.

Does your answer imply that such evidence - be it for or against - could be obtained by using statistics if done honestly?

 

[Tyger]

The evaluation

of statistical certainty is already well developed and used all the time by the pros. For instance the link between cigarette smoking and cancer has a high degree of certainty even though the sample taken was a small portion of the population.