Criticising the scientific method

  • Thread starter hadeka
  • Start date
  • #1
32
0

Main Question or Discussion Point

Criticising the scientific method !!

Hi all

Quoting a part of prof. Michael Shermer's book: Why people believe weird things?, about some problems of scientific thinking, made me a bit confused.

"Let us suppose that an ichthyologist is exploring the life of the ocean. He casts a net into the water and brings up a fishy assortment. Surveying his catch, he proceeds in the usual manner of a scientist to systematise what it reveals. He arrives at two generalisations: No sea-creature is less than two inches long. (2) All sea-creatures have gills. These are both true of his catch, and he assumes tentatively that they will remain true however often he repeats it.

In applying this analogy, the catch stands for the body of knowledge which constitutes physical science, and the net for the sensory and intellectual equipment which we use in obtaining it. The casting of the net corresponds to observation; for knowledge which has not been or could not be obtained by observation is not admitted into physical science.

An onlooker may object that the first generalisation is wrong. "There are plenty of sea-creatures under two inches long, only your net is not adapted to catch them." The icthyologist dismisses this objection contemptuously. "Anything uncatchable by my net is ipso facto outside the scope of icthyological knowledge. In short, "what my net can't catch isn't fish." Or — to translate the analogy — "If you are not simply guessing, you are claiming a knowledge of the physical universe discovered in some other way than by the methods of physical science, and admittedly unverifiable by such methods. You are a metaphysician. Bah!
"

My question is:
Is this really explaining the scientific method ?!! If yes, so how we're relying on such method ?!!!!!!!!!!
What is the appropriate way to criticize these claims and ensure that the scientific method is the best knowledge method that does not have such holes ?!!


Thank you !


Hadeka
 
Last edited:

Answers and Replies

  • #2
D H
Staff Emeritus
Science Advisor
Insights Author
15,393
683


My question is: Is this really explaining the scientific method ?!!
The post you quoted (I don't deign to lend it support by quoting it yet again) employs two logical fallacies, minimum. He first set up a parody of the scientific method in his ichthyologist example, and then proceeded to argue against this parody. He then extended this false description of the scientific method to the physical sciences. This is a red herring argument.

Scientists know that their measuring devices and the measurements from them are incomplete and somewhat erroneous. That does not mean the scientific method itself is flawed. It merely means scientists should strive to improve their techniques (which they do, by means of research into new and improved measuring devices), double check their results (which they do, by means of peer review and independent replication of experiments), and be careful of overgeneralization (which they try to do).

The quoted argument does have one thing right: Science treats explanations which cannot be possibly observed as inadmissible to the body of science. Science is predicated on the existence of reality. For example, an argument that we are all in a computer simulation is untestable and unscientific.
 
  • #3
Bill_B
Gold Member
40
0


Is this really explaining the scientific method ?!! If yes, so how we're relying on such method ?!!
It might be an oversimplification, but I think it's generally accurate.

Of course it sounds silly to believe that there are no sea-creatures smaller than 2 inches long. But from the perspective of the analogy, it's a reasonable conclusion to draw.

The key is this statement -
In applying this analogy, the catch stands for the body of knowledge which constitutes physical science, and the net for the sensory and intellectual equipment which we use in obtaining it.
By definition, the body of knowledge is EVERYTHING we know about a topic. If everything we know about sea-life is in the net in front of us (meaning we've never seen a tadpole or a minnow in pond, etc.), then we don't know very much. And if all of the fish we see are longer than 2 inches, then as far as we know (from observation) that's true for all sea-life, until we observe otherwise. It's important to note that the 2inch fish assumption is tentative -

and he assumes tentatively that they will remain true however often he repeats it.
which means it could change at a later time. The Scientific Method is an iterative process. (Like washing your hair - lather, rinse, repeat)

After thinking about it a little bit, we can postulate that there might be smaller fish that fall through the holes in the net. We then create a new test (build a tighter net), and gather more data using the new test (catch more fish with the new net). If we catch something shorter than 2 inches, then we've just disproved (or modified) our original hypothesis. Rinse, repeat. Welcome to the scientific method.

The problem is when things are outside of what is currently testable. If a person says to us "I KNOW that there are smaller fish" without any evidence, and with no way to test and observe (maybe we don't have the technology to build a better net), then it can't be added to the body of knowledge. After all, with no observable evidence, what proof does he have? How can he know? Maybe he just made it up. We can't apply the scientific method to something that isn't testable.

Once something becomes testable (maybe we invent a new technology, or maybe someone simply thinks up a new test that works), then the scientific method can be applied, and the data could become part of the body of knowledge.

What is the appropriate way to criticize these claims and ensure that the scientific method is the best knowledge method that does not have such holes ?!
Honestly, I'm not sure what claims you want to criticize or what holes you see. The analogy that Shermer used could have been a little clearer, but I think it works. And if done correctly, the scientific method works, albeit slowly (lots of iterations).
 
  • #4
D H
Staff Emeritus
Science Advisor
Insights Author
15,393
683


It might be an oversimplification, but I think it's generally accurate.
The first is correct. The latter is not. Because the author could not argue with the scientific method as it is, he (or she) intentionally oversimplified and overgeneralized to the extent as to make a parody - a straw man - of the scientific method. While this overly simplified and overly generalized parody is subject to attack, what he is attacking is not the scientific method. He is attacking his own parody of the method. In short, the author constructed a straw man argument.
 
  • #5


By my understanding of the scientific method that story does not depict it at all. The "scientist" in that story looks at a bunch of information and makes some guesses about the implications of patterns he sees. That might qualify as scholarship but not as science.

A scientific approach would begin with the hypotheses about the size and gills of sea-dwelling creatures. Then there would be design and execution of experiments to rigorously test those hypotheses through direct, counterfactual, and other methods. The experiments would need to be thoroughly documented so that they could be accurately reproduced by other scientists. The initial findings and experimental method would be published and reviewed by other scientists.

If the method and hypotheses were deemed sound by the scientific community, at that point the hypothesis wouldn't be regarded as fact but simply some interesting initial findings. In subsequent years other scientists would reproduce the experiments, perhaps with some adjustments to the experimental method. If the subsequent experiments produced the same results there would be more and more confidence in the hypotheses.

But the hypotheses would never reach the status of "truth" like you might have in religion or philosophy. You never really have more than "confirmed findings" in science: an idea, the results of earlier tests of that idea, and instructions on how to reproduce those tests to examine the subject yourself.

(So I think it's pretty obvious that the two hypotheses from the story - that no sea-dwelling creature is less than two inches long and that all sea-dwelling creatures have gills - would never successfully pass through the rigor of the scientific method as I've described it. (Unless, as Bill_B suggests, the net is really only a metaphor for everything that's directly perceivable. But that would be dumb, as if Shermer were trying to explain science with the language of dream interpretation or something.))
⚛
 
Last edited:
  • #6
Bill_B
Gold Member
40
0


The first is correct. The latter is not. Because the author could not argue with the scientific method as it is, he (or she) intentionally oversimplified and overgeneralized to the extent as to make a parody - a straw man - of the scientific method. While this overly simplified and overly generalized parody is subject to attack, what he is attacking is not the scientific method. He is attacking his own parody of the method. In short, the author constructed a straw man argument.
I think we disagree on what the argument is about. I don't believe Shermer is trying to argue against the scientific method (or any strawman facsimile thereof) at all. The way I read it, he is talking about why scientists don't automatically believe any willy-nilly idea offered to them, but instead require testable evidence. I think my wife has a copy of the book - when she gets home I'll see if she knows where it is. Perhaps there's more context that would be helpful.


I also think that the reason the analogy exists is to attempt to make the explanation more accessible to an average audience (to which his books are usually targeted). I'll concede that the analogy may not be the best possible one, but I think you're reading too much into it (or perhaps taking it too literally).
 
  • #7
Bill_B
Gold Member
40
0


By my understanding of the scientific method that story does not depict it at all. The "scientist" in that story looks at a bunch of information and makes some guesses about the implications of patterns he sees. That might qualify as scholarship but not as science.
I agree that it doesn't qualify as the whole of science, but I think it is very much a part of the scientific method. You have to be able to analyze the data and draw conclusions, as a basis for your next hypothesis. If your hypothesis is disproved, you drew an incorrect (or an incomplete) conclusion.

A scientific approach would begin with the hypotheses about the size and gills of sea-dwelling creatures. Then there would be design and execution of experiments to rigorously test those hypotheses through direct, counterfactual, and other methods.
How do you begin with a hypothesis? Do you think up totally a new hypothesis without basing it on some preexisting body of knowledge? Don't you usually start with a question about a peculiarity or pattern in existing data?

The experiments would need to be thoroughly documented so that they could be accurately reproduced by other scientists.

-- snipped for brevity --

You never really have more than "confirmed findings" in science: an idea, the results of earlier tests of that idea, and instructions on how to reproduce those tests to examine the subject yourself.
Except as noted, I don't disagree with your description of the scientific method - it's a more realistic picture of the way science works in the real world. However, I don't think Shermer was trying to build an exhaustive model of the scientific method, but a layman's model that could be used to explain why scientists often dismiss non-testable phenomena as hokum.

(So I think it's pretty obvious that the two hypotheses from the story - that no sea-dwelling creature is less than two inches long and that all sea-dwelling creatures have gills - would never successfully pass through the rigor of the scientific method as I've described it. (Unless, as Bill_B suggests, the net is really only a metaphor for everything that's directly perceivable. But that would be dumb, as if Shermer were trying to explain science with the language of dream interpretation or something.))
I agree that if taken literally, it wouldn't stand up to rigor. But it's an analogy, it's not supposed to be taken literally.

And yes, the way I read it, the net is a metaphor for our observational tools and processes - microscopes, accelerometers, our eyes, etc., while everything in the net (the catch) is a metaphor for everything we know about science.

As far as I can tell, all Shermer is trying to say is that because of the scientific method, if something can't be tested, we can't say it is "known" (in the usual scientific meaning of the term).

And reading it again, I think he may also be trying to point out the problems that may arise if a scientist entirely dismisses a hypothesis that cannot be tested at the present time (but might be in the future). Unfortunately, I'd need more context to be sure. I'll see if my wife can find her copy of the book. Or perhaps the original poster can post more snippets.
 
  • #8
32
0


And reading it again, I think he may also be trying to point out the problems that may arise if a scientist entirely dismisses a hypothesis that cannot be tested at the present time (but might be in the future). Unfortunately, I'd need more context to be sure. I'll see if my wife can find her copy of the book. Or perhaps the original poster can post more snippets.
Well ... It is written here:
http://www.skywise711.com/Skeptic/WPBWT/index.html
Twenty-five Fallacies That Lead Us to Believe Weird Things.


It's number three: Equipment Constructs Results, which is under the section of: Problems of scientific thinking.


Thanks to all of you !

Looking for more participation to shed more light on the topic..

Many thanks



Hadeka.
 
  • #9


@Bill_B: Yeah, I think all of what you're saying is true there. Analysis is of course part of the scientific method - but it's pretty much part of every epistemological approach. In the excerpt above there's no distinction made between observation and analysis and it appears to me that science is presented as no different from deep thought. Within the excerpt it doesn't mention testing hypotheses or conclusions at all.

...okay, I've just noticed that I was probably confused by an unmatched quotation mark in that last paragraph. I hadn't realized that the scientist was the one calling the onlooker a metaphysician; initially I thought it was the other way around. So what you're saying makes more sense.

Still though, I think the answer to hadeka's question "Is this really explaining the scientific method ?!!" is that no, that excerpt is not explaining the scientific method. It's more explaining what the epistemological domain of science is (with a confusing and inexact analogy, in my opinion).
⚛
 
  • #10


In reading the original source it appears that Bill_B's inference of the purpose of the story is dead-on: the title of that section is "Equipment Constructs Results" and the story is preceded with the assertion that scientific understanding of the nature of intelligence has been screwed up by the inexact and limited nature of IQ tests.

So hadeka, I would maintain that the excerpt is not trying to explain the scientific method. It's making a different point entirely, that the conclusions of a scientific investigation can be flawed by insufficient experimental design that does not account for the limitations of the experimental equipment. (Which is basically what Bill_B said, I think.)
⚛
 
  • #11


Shermer Fallacy #1: Imply that there's a singular scientific method.

There's no such thing. Lots of scientific discoveries did not go through the prototypical scientific method.
 
  • #12
Bill_B
Gold Member
40
0


Which is basically what Bill_B said, I think.
I don't think that's what I said, but I wish it was :biggrin: since I think you hit the nail on the head
 
  • #13
CRGreathouse
Science Advisor
Homework Helper
2,820
0


"Anything uncatchable by my net is ipso facto outside the scope of icthyological knowledge. In short, "what my net can't catch isn't fish." Or — to translate the analogy — "If you are not simply guessing, you are claiming a knowledge of the physical universe discovered in some other way than by the methods of physical science, and admittedly unverifiable by such methods. You are a metaphysician. Bah!"
If the existence of sea creatures below 2" is supposed to represent things beyond the researcher's equipment, but within the realm of science, then the experiment is simply inadequate to determine the existence of small sea creatures and the claim 'such knowledge is metaphysical' is wrong. If it's supposed to represent things literally beyond the ability of science, then the claim 'no creatures under 2" exist' is not falsifiable and cannot be claimed under the scientific method.

Either way the analogy fails as a criticism of the scientific method. If it's a criticism of researchers drawing untoward conclusions, though, it could work.
 
  • #14
32
0


If the existence of sea creatures below 2" is supposed to represent things beyond the researcher's equipment, but within the realm of science, then the experiment is simply inadequate to determine the existence of small sea creatures and the claim 'such knowledge is metaphysical' is wrong. If it's supposed to represent things literally beyond the ability of science, then the claim 'no creatures under 2" exist' is not falsifiable and cannot be claimed under the scientific method.

Either way the analogy fails as a criticism of the scientific method. If it's a criticism of researchers drawing untoward conclusions, though, it could work.
I agree with this ... We cannot prove that there are no sea creatures smaller than 2 inches except by using experiments that prove that such creatures does not exist [ A net that can catch sea creatures that are less than 2 inches long will either prove or disprove that such creatures exist ]

I mean, if i want to prove that there are no sea creatures less than 2 inches long, i should use a net that can catch such creatures to either prove or disprove their existence.

Thank you :D
 
Last edited:
  • #15
Ivan Seeking
Staff Emeritus
Science Advisor
Gold Member
7,213
176


You cannot prove the general case that something does not exist.
 
  • #16
HallsofIvy
Science Advisor
Homework Helper
41,833
956


It might be an oversimplification, but I think it's generally accurate.

Of course it sounds silly to believe that there are no sea-creatures smaller than 2 inches long. But from the perspective of the analogy, it's a reasonable conclusion to draw.
Yes, but the crucial part of the analogy is when someone objects to the Ichthyologist that there are sea creatures under 2 inches long and he asserts that any thing he cannot catch in his net is not a sea creature. That is a Parody. As D H said, any good scientist- or anyone using the scientific method- would respond "That may well be true but it hasn't been proved. Do you have any suggestions for improving the net?"

The key is this statement -


By definition, the body of knowledge is EVERYTHING we know about a topic. If everything we know about sea-life is in the net in front of us (meaning we've never seen a tadpole or a minnow in pond, etc.), then we don't know very much. And if all of the fish we see are longer than 2 inches, then as far as we know (from observation) that's true for all sea-life, until we observe otherwise. It's important to note that the 2inch fish assumption is tentative -
Which, in the analogy it wasn't. That was where Schermer "cheated". He said, at first, that the concusions were "tentative" but later had the Ichtyologist say, '"Anything uncatchable by my net is ipso facto outside the scope of icthyological knowledge. In short, "what my net can't catch isn't fish."'



which means it could change at a later time. The Scientific Method is an iterative process. (Like washing your hair - lather, rinse, repeat)

After thinking about it a little bit, we can postulate that there might be smaller fish that fall through the holes in the net. We then create a new test (build a tighter net), and gather more data using the new test (catch more fish with the new net). If we catch something shorter than 2 inches, then we've just disproved (or modified) our original hypothesis. Rinse, repeat. Welcome to the scientific method.

The problem is when things are outside of what is currently testable. If a person says to us "I KNOW that there are smaller fish" without any evidence, and with no way to test and observe (maybe we don't have the technology to build a better net), then it can't be added to the body of knowledge. After all, with no observable evidence, what proof does he have? How can he know? Maybe he just made it up. We can't apply the scientific method to something that isn't testable.

Once something becomes testable (maybe we invent a new technology, or maybe someone simply thinks up a new test that works), then the scientific method can be applied, and the data could become part of the body of knowledge.



Honestly, I'm not sure what claims you want to criticize or what holes you see. The analogy that Shermer used could have been a little clearer, but I think it works. And if done correctly, the scientific method works, albeit slowly (lots of iterations).
 
  • #17
arildno
Science Advisor
Homework Helper
Gold Member
Dearly Missed
9,970
132


I would say, though, that NO ONE who did not have other means of obtaining evidence than the ichtyologist in question are to be given more authority than that ichtyologist.
For example, a blind desert dweller has even less intelligent to say about life in the ocean than the ichtyologist..
 
  • #18


I would say, though, that NO ONE who did not have other means of obtaining evidence than the ichtyologist in question are to be given more authority than that ichtyologist.
For example, a blind desert dweller has even less intelligent to say about life in the ocean than the ichtyologist..
That's not scientific either, though - that's giving the ichthyologist a priori authority simply because he's the one who obtained the evidence. Him collecting the evidence doesn't make his conclusions / guesses about the nature of sea life any more authoritative.
⚛
 
  • #19


Wow, this is getting a little too philosophical for me. If you want a physical explanation for why scientific predictions are often flawed, it is called entropy. If you don't have a complete set of data points, but only say, a and b, you can't make that good of a prediction about what will happen "between" a and b, because the statistics of the system are unknown. At least, that's the way I look at it.
 
  • #20


Wow, this is getting a little too philosophical for me. If you want a physical explanation for why scientific predictions are often flawed, it is called entropy. If you don't have a complete set of data points, but only say, a and b, you can't make that good of a prediction about what will happen "between" a and b, because the statistics of the system are unknown. At least, that's the way I look at it.
What are you talking about?
 
  • #21
arildno
Science Advisor
Homework Helper
Gold Member
Dearly Missed
9,970
132


That's not scientific either, though - that's giving the ichthyologist a priori authority simply because he's the one who obtained the evidence. Him collecting the evidence doesn't make his conclusions / guesses about the nature of sea life any more authoritative.
⚛
Relative to the one having NO evidence whatsoever, sure the ichtyologist has the authority.
 
  • #22


Relative to the one having NO evidence whatsoever, sure the ichtyologist has the authority.
No. Anyone who has access to the evidence he has collected could draw conclusions and might be more successful in deducing the significance of the evidence or extrapolating patterns from the evidence than the ichthyologist. The ichthyologist having been the person who collected the evidence does not by itself make him more authoritative or more correct.

Or, to put it a different way, the ichthyologist is only an authority on his own evidence and that evidence itself - on the fact that "On this day, at this time, I did X and Y happened." He isn't automatically an authority on X or Y or their relationship to each other or whether the same thing might or might not happen at a different day and time.
⚛
 
Last edited:
  • #23


No. Anyone who has access to the evidence he has collected could draw conclusions and might be more successful in deducing the significance of the evidence or extrapolating patterns from the evidence than the ichthyologist.
No. The ichthyologist has an expertise in his particular field of study, and is therefore able to draw more accurate conclusions on the data than a non-expert in his field of study. The only other people who are capable of drawing conclusions on this data set would be other ichthyologists.
 
Last edited:
  • #24


No. The ichthyologist has an expertise in his particular field of study, and is therefore able to draw more accurate conclusions on the data than a non-expert in his field of study. The only other people who are capable of drawing conclusions on this data set would be other ichthyologists.
Other ichthyologists are included in the group of people other than the guy collecting the evidence, of course, even if they haven't collected their own evidence on the same subject of study. But - the only other people capable of drawing conclusions based on the list of stuff caught in a net are ichthyologists?

I think that basically sets up scientists as priests in the temple of Science. If you believe that scientists have special access to the truth that is beyond the questioning of mere mortals, that's up to you. But I think that one of the basic principles of science and empiricism is that ideas stand on their own and that the virtue of a theory is that it testable and has been vetted through scientific rigor, not the reputation of the idea's author or his or her resume.

This example - the ichthyologist drawing a conclusion that any fisherman (or sea-shore-dweller) can see is patently false, while humorous, is not entirely unknown in the history of science. And conversely, someone who isn't an expert or specialist being the one to draw the scientifically important conclusions isn't uncommon either. I wonder how many people at the beginning of the last century would have said "A mere patent clerk barely out of school, who couldn't even get a job teaching much less as a professional researcher, isn't capable of drawing conclusions about the most profound questions of theoretical physics!" (Referring to the flagship case of this sort of thing, Einstein.)
⚛
 
Last edited:
  • #25


Other ichthyologists are included in the group of people other than the guy collecting the evidence, of course, even if they haven't collected their own evidence on the same subject of study. But - the only other people capable of drawing conclusions based on the list of stuff caught in a net are ichthyologists?

I think that basically sets up scientists as priests in the temple of Science. If you believe that scientists have special access to the truth that is beyond the questioning of mere mortals, that's up to you. But I think that one of the basic principles of science and empiricism is that ideas stand on their own and that the virtue of a theory is that it testable and has been vetted through scientific rigor, not the reputation of the idea's author or his or her resume.
I have never stated that the validity comes from the reputation of the author. What I am saying is that in this context it is not appealing to authority by deferring to the expert in this field of study. It is very likely that the expert in the field of study is going to have a better grasp on drawing conclusions from the data set than a layman would.

And conversely, someone who isn't an expert or specialist being the one to draw the scientifically important conclusions isn't uncommon either. I wonder how many people at the beginning of the last century would have said "A mere patent clerk barely out of school, who couldn't even get a job teaching much less as a professional researcher, isn't capable of drawing conclusions about the most profound questions of theoretical physics!" (Referring to the flagship case of this sort of thing, Einstein.)
⚛
I'm not sure what point you're trying to make with this? I don't think anyone is suggesting that an expertise is an innate ability. It is something that is earned. In Einstein's case, he earned it by having his theories withstand the scientific process.
 

Related Threads on Criticising the scientific method

  • Last Post
3
Replies
55
Views
10K
  • Last Post
Replies
19
Views
4K
  • Last Post
Replies
17
Views
3K
  • Last Post
Replies
24
Views
4K
  • Last Post
Replies
15
Views
781
  • Last Post
Replies
4
Views
3K
  • Last Post
Replies
14
Views
2K
  • Last Post
4
Replies
76
Views
9K
Replies
75
Views
5K
Replies
3
Views
882
Top