Standardized Tests: Have we gone too far?

[Dale]

Statistical methods for dealing with those kinds of issues are well known. As long as the analysts are competent (not a given) I don't see any of those as being a real problem. Furthermore, available statistical methods are quite good at testing the data itself to determine if these issues are even problems for a given data set.

 

[votingmachine]

That is the big thing. Unfortunately you can't assign kids randomly to placebo groups that get no instruction and different treatment groups. Bugger that whole problem of human test subjects.

Good statisticians are important. I wish I was one, but I'm strictly low level. I agree that the analysts have to be competent, and they have to be alert to some crazy potential confounding variables. Home life, which was mentioned, would have to be looked at.

Do you think my "intuition" is wildly wrong or on the mark about the variation issue in student populations? I recall my brother relating the story of how his rural school tried very hard to convince him to find a way to keep his two very bright kids in their system. He was aware that they were aware of the tumble in their metrics from the transfer of two excellent students (and high test results). Likewise, there is a whole bunch of news around how schools try to transfer out any student that has poor test scores. If you have that pregnant teen ... you know what will happen.

I see now the following post was made already:
"There is another thing to consider that whatever metric is chosen skews the results in a certain way as people try to optimize their score for their performance appraisal this invalidating the metric." (Jedishrfu)

That is what I was getting at.

 

[Dale]

Unfortunately you can't assign kids randomly to placebo groups that get no instruction and different treatment groups.

But you don't need to if you use the right methods.

Do you think my "intuition" is wildly wrong or on the mark about the variation issue in student populations?

I think your intuition is wildly wrong about the importance of the issue. It is simply something that the statisticians need to account for in their methodology, not something that fundamentally precludes analysis.

 

[votingmachine]

I think your intuition is wildly wrong about the importance of the issue. It is simply something that the statisticians need to account for in their methodology, not something that fundamentally precludes analysis.

Cool. Although I guess what I am saying is not that you can't do the analysis. As you point out the analysis tells you if there is a significant difference. I was getting at whether the randomness matters so much that the data set has to be extraordinarily large to draw conclusions. A grade-school teacher might have 25 students and teach several subjects to them, every year. If it ends up that you need 10 years of data to get to statistical significance, then that would accord with my first intuition that variation ends up making too much difference. 10 years is 250 kids, and that might let you start to control for socioeconomic factors, and confounding variables.

I recognize the analysis is its own thing, and it tells you what it tells you. My intuition was that the annual data sets would tell you that there was not a lot to conclude about individual teachers. And then schools systems need to have the teacher factor removed, and control for socioeconomic factors again. And of course everyone will try to game the system for better metrics as they move along.