twofish-quant
- 6,821
- 20
ZapperZ said:So you think the "dirty details" from just a few people have more credibility, and sufficient to draw a valid conclusion?
It's not either or.
In cosmology and social sciences there are two types of studies, those that go deep and shallow, and those that go wide and narrow. In quantitative research, you ask the same question for a large number of people, whereas in qualitative research, you do deep interviews of a few people. It's really useful if you do both.
One problem is reporting bias. If you what to know if fruit loops causes cancer, and you put out a internet post that asking for interviewees that have eaten fruit loops and gotten cancer, then surprise, surprise, surprise, you'll find that amount your interviewees, it appears that there is a correlation between fruit loops and cancer. And if you have a set of interviews of people that have eaten fruit loops and gotten cancer, and then you do a statistical survey of fruit loops eaters, then I'd be more likely to trust the latter because of reporting bias.
However, it is simply not the case that statistics are *inherently* less subject to reporting bias. Salary and satisfaction surveys are *notorious* for having self- reporting bias, and in this particular situation, I would tend to trust deep interviews more than I would a survey unless I had a lot of confidence that the creators of the survey were very, very careful about controlling for reporting bias, which I do not have in this case. The thing about deep interviews is that if you go in knowing that you are likely to have a biased sample, then by asking the right questions, you can figure out how the sample is biased, and then try to piece together what is going on. Also, you get deep information about what is going on that you would not get with surveys.
One other reason that I tend to trust interviews rather than surveys in this particularly situation is that interviews are harder for amateur data gatherers to get wrong. Statistical surveys are deceptively easy to get very wrong, and the problem with surveys is that it's harder to tell if you've biased your sample.
Also, if you ask me if I what I think about the AIP numbers, they don't wildly differ from what I think I would get if I asked my friends, but...
If you asked most of my Ph.D. friends if they would do it all over again, they would likely say "Yes, I'd do it over again" however if you asked them "why?" the answer would be "if I didn't do my Ph.D., then there would have been no chance in hell that I would have gotten my green card." This would nicely explain some of the numbers. The US citizen column has higher satisfaction because they include people that finished their Ph.D. and got US citizenship, the non-citizen category includes people that still have hope of getting US citizenship, and anyone that has given up hope of US citizenship has left the country and is not part of the survey.
Of course the problem with this is that it if turns out that most Ph.D. holders would do it over again for a green card, this would be useless information if you are already a US citizen, and if that is what is going on, you'd never be able to find it out by those survey questions. If you think that this what is going on, you might be able to get the data with other survey questions.
Also, it helps if you do studies with *different* biases. For example, if you ask people on this forum about their Ph.D. experiences, you are going to be biased toward people with native level fluency of English and are comfortable using that language to talk about themselves.
And note, if you think statistics can't tell you anything about what's going to happen in the future, what makes you think anecdotal stories can? These individuals are more clairvoyant?
People can think, and when you interview people you watch them think. One thing that you can get with stories that you can't get with this particular survey is historical data. You can talk to someone and they can tell you that they think that X is going to happen because X happened in 1970.
Also a lot of the data that gives *dynamical* information is stuff that doesn't fit into a close-response survey. When you do a survey, you have an implicit model for what is going on, and the answers are restricted enough so that it's hard to see that there is something basically wrong with the model, whereas if you use deep interviewing, you are more likely to get information that challenges your model of what is going on.