Standardized Tests: Have we gone too far?

[mathwonk]

atyy, would you say giving a negative answer (with the correct absolute value) to a physics question whose answer must be a positive quantity would justify a zero score?

The whole question of how to assess something or someone correctly is a very hard one in my experience and requires diligent attention. The personal interview method is one that is used by people I respect in some technical business situations. E.g. I know someone with responsibility for recruiting, hiring, and when necessary firing, for a global high tech company. He always does all three of these tasks in person face to face, even if he has to fly 5 or 10 thousand miles to do so. Certainly he never hires anyone based on performance on a standardized written test.

 

[rollingstein]

He always does all three of these tasks in person face to face, even if he has to fly 5 or 10 thousand miles to do so. Certainly he never hires anyone based on performance on a standardized written test.

Which makes a lot of sense. Standardized testing for the masses & for initial screening followed by more expensive, time consuming methods for the really important decisions.

 

[PWiz]

As an HS student, I can't agree more. I can feel the immense pressure of the need to perform well in these standardized tests. It's like everyone around me is emphasizing how much of a passport high scores are to a good college, but I can't help but feel that the focus is less on learning and more on blind analysis of performance under stress. Everyone is not built out of the same wood you know.

 

[atyy]

atyy, would you say giving a negative answer (with the correct absolute value) to a physics question whose answer must be a positive quantity would justify a zero score?

My personal view is that as long as the steps are correct, I would only take off the mark for the final answer in a classroom test situation. Generally I would grade according to the marking scheme that is determined before the test.

Another friend of mine took an economics class and ended up with a complex profit in a test. She left the answer intact and got partial credit. It clearly didn't mean anything about her deep undertstanding of economics, since even a complete idiot knows a profit cannot be complex.

The whole question of how to assess something or someone correctly is a very hard one in my experience and requires diligent attention. The personal interview method is one that is used by people I respect in some technical business situations. E.g. I know someone with responsibility for recruiting, hiring, and when necessary firing, for a global high tech company. He always does all three of these tasks in person face to face, even if he has to fly 5 or 10 thousand miles to do so. Certainly he never hires anyone based on performance on a standardized written test.

I think it is impossible to have a fail-safe algrorithm in real life! Personal interactions and luck are so essential.

But most of the time, I don't know if one is trying to things that require luck in the classroom. Thinking a bit about your example of 13 X 65, I do find it a bit perturbing too. However, what part of arithmetic is really "essential"? To do the calculation by hand, one only needs to memorize 3 X 6 and 3 X 5, or more generally working in base 10, I guess one only needs the multiplication table up 9 X 9. Naively, is there anything deep about 9 X 9 = 81, or would it be ok to know how to set up the calculation by hand, but use a multiplication table?

 

[mathwonk]

I want to repeat my point about one positive aspect to standardized tests, namely they allow anonymous students with no connections or social status, to make an argument that they belong up there with the privileged few. In the 1960's, SAT scores and the consequent merit scholarships they brought, sent hundreds of relatively poor boys from low socio economic families like mine, to schools like Harvard, where we met sons of famous wealthy people, and future national politicians and scions of business. And the test itself is relatively cheap and easy to prepare for, all you need is a $20 prep book with old tests in it to practice on. I.e. anyone can afford the test, anyone can afford the practice materials, and the benefit is a chance to compare yourself favorably with much more privileged students, and possibly get accepted at top schools with financial assistance. I.e. the very definition of a standardized test is one that tests everyone the same, and hence let's you compete against people you have never met and who go to schools you can't afford. This is essentially the only way a high school boy from Tennessee can show he is comparable in ability and potential to a prep school boy from Connecticut. This is a very useful tool for social advancement. But of course you have to prepare or you don't benefit.

 

[stevendaryl]

But most of the time, I don't know if one is trying to things that require luck in the classroom. Thinking a bit about your example of 13 X 65, I do find it a bit perturbing too. However, what part of arithmetic is really "essential"? To do the calculation by hand, one only needs to memorize 3 X 6 and 3 X 5, or more generally working in base 10, I guess one only needs the multiplication table up 9 X 9. Naively, is there anything deep about 9 X 9 = 81, or would it be ok to know how to set up the calculation by hand, but use a multiplication table?

To me, the critical thing about arithmetic is the relationship between addition and multiplication, understanding integer multiplication as repeated addition, and understanding multiplication of a pair of positive reals as the area of a rectangle. The specific times facts are not as important as understanding the meaning of place notation for decimals.

 

[Redbelly98]

Ha, I took the SAT last week, and will take the SAT Math 2 subject test and SAT physics subject test less than a few months from now. What annoys me the most is that the SAT claims to be an aptitude test, yet you can significantly improve your score by preparing for it.

Actually, they do not make that claim about the SAT anymore. SAT no longer stands for "scholastic aptitude test", it is simply the name of the test and does not stand for anything.

 

[HomogenousCow]

I don't know if any of you remember what it's like being 17 and in school, but the majority of people would not study AT ALL if not from exam pressure. The sad truth is that you need a short term incentive for teenagers to learn, a long term incentive such as a "good job" is simply too far off to significantly motivate the majority of people.

 

[stevendaryl]

I don't know if any of you remember what it's like being 17 and in school, but the majority of people would not study AT ALL if not from exam pressure. The sad truth is that you need a short term incentive for teenagers to learn, a long term incentive such as a "good job" is simply too far off to significantly motivate the majority of people.

I think that's absolutely right. If you aren't challenged to do something (such as pass a test) with knowledge, it tends to go in one ear and out the other. I don't remember the reference, but there was a study that showed that frequent quizes on what someone has learned tends to improve his ability to recall what he learned.

 

[HomogenousCow]

The simple truth is that most high school students do not care about the intrinsic value of whatever they are being taught (assuming there is any, I'm looking at you Samuel Beckett). There seems to be some grand fantasy that students are inquisitive angels, oppressed by "The Man" and his weapon of choice-the SAT.

 

[stevendaryl]

The simple truth is that most high school students do not care about the intrinsic value of whatever they are being taught (assuming there is any, I'm looking at you Samuel Beckett). There seems to be some grand fantasy that students are inquisitive angels, oppressed by "The Man" and his weapon of choice-the SAT.

I think that a certain amount of drive to understand does exist in most (all?) people. Outside of the realm of academics, people are driven to understand how to win video games, they are driven to understand why Dumbledore told Snape to kill him, they are driven to understand why this batch of cookies turned out worse than the last batch. People really do have a drive to understand. The problem (for school purposes) is that the average person, if led by his own curiosity, would get around to understanding how to do calculus some time like never. What most people are naturally curious about isn't what schools want to teach.

 

[Andy Resnick]

I certainly agree that things like home life affect a student's performance on tests, but why does that make the test results not statistically significant? <snip>

It goes to the most basic reason for tests: the purpose of a test is to measure "something" (more on this later). So you start with some average score and then try and figure out interventions that result in higher test scores. And this is the basic problem- trying to establish statistically significant results showing how some specific intervention (just in time teaching, flipped classroom, thank-pair-share, problem-based learning, etc.) results in improved test scores. That's done (at best) by having the identical instructor teach multiple sections, some of which are negative controls and some are with the intervention. That assumes that every section is made of interchangeable students, for example. And then the process has to be repeated to ensure reproducibility. And then the intervention has to be performed at different schools, using different teachers. In the end, the statistical error associated with the study results is typically as large as the effect.

Now, what does a specific test actually measure? Much ink has been spilled on this topic. For example, the strongest correlation with SAT scores is family income- not ''aptitude'.

http://blogs.wsj.com/economics/2014...me-inequality-how-wealthier-kids-rank-higher/

 

[stevendaryl]

It goes to the most basic reason for tests: the purpose of a test is to measure "something" (more on this later). So you start with some average score and then try and figure out interventions that result in higher test scores. And this is the basic problem- trying to establish statistically significant results showing how some specific intervention (just in time teaching, flipped classroom, thank-pair-share, problem-based learning, etc.) results in improved test scores. That's done (at best) by having the identical instructor teach multiple sections, some of which are negative controls and some are with the intervention. That assumes that every section is made of interchangeable students, for example. And then the process has to be repeated to ensure reproducibility. And then the intervention has to be performed at different schools, using different teachers. In the end, the statistical error associated with the study results is typically as large as the effect.

Now, what does a specific test actually measure? Much ink has been spilled on this topic. For example, the strongest correlation with SAT scores is family income- not ''aptitude'.

I'm not going to argue about whether there even is such a thing as "aptitude", much less whether SAT measures it. However, it seems to me that if someone is tested on vocabulary (for instance), you can find out whether that person knows what a word means. If someone is tested on solving algebraic equations in one variable, you can find out whether they know how to do that. It certainly may be the case that finding out that someone doesn't know how to solve an equation in one variable doesn't necessarily say what should be done about it.

I see your first paragraph as simply about the difficulties of figuring out effective teaching methods. That certainly is a hard problem, but it seems orthogonal to the issue of testing.

 

[stevendaryl]

I'm not going to argue about whether there even is such a thing as "aptitude", much less whether SAT measures it. However, it seems to me that if someone is tested on vocabulary (for instance), you can find out whether that person knows what a word means. If someone is tested on solving algebraic equations in one variable, you can find out whether they know how to do that. It certainly may be the case that finding out that someone doesn't know how to solve an equation in one variable doesn't necessarily say what should be done about it.

I see your first paragraph as simply about the difficulties of figuring out effective teaching methods. That certainly is a hard problem, but it seems orthogonal to the issue of testing.

Just to make it clear, I'm not in favor of tests as "measurements". I don't think that an overall numeric result, from 0 to 100 or 0 to 1500 (or whatever the range is for SATs) means much at all. But the fact that a student is able or unable to answer specific questions certainly is meaningful. I favor tests as diagnostics or assessment, not as measures of quality of the student.

 

[Andy Resnick]

<snip>I see your first paragraph as simply about the difficulties of figuring out effective teaching methods. That certainly is a hard problem, but it seems orthogonal to the issue of testing.

But you would hopefully agree that there should be a reason to test and that the test should evaluate how effectively students achieve some specified learning objective.

Learning involves much more than rote memorization. A good example of a (relatively) new standardized test is the Force Concept Inventory:

http://www.flaguide.org/tools/diagnostic/force_concept_inventory.php

This is the key innovation: "Each question offers only one correct Newtonian solution, with common-sense distractors (incorrect possible answers) that are based upon student's misconceptions about that topic, gained from interviews."

This exam is an attempt to directly measure student learning, not simple recall.

 

[rollingstein]

As an HS student, I can't agree more. I can feel the immense pressure of the need to perform well in these standardized tests. It's like everyone around me is emphasizing how much of a passport high scores are to a good college, but I can't help but feel that the focus is less on learning and more on blind analysis of performance under stress. Everyone is not built out of the same wood you know.

But this stress mimics real life situations pretty well. Most jobs you get will test your descision making skills under pressure.

So might as well embrace the testing stress & try to thrive & perform better under stress. And I can tell you that the skill is learned to a large extent. The more tests you give the less the stress will impact your scores adversely.

Yes, not everyone is built of the same wood, but that is partly what the tests are trying to discern.

 

[stevendaryl]

But you would hopefully agree that there should be a reason to test and that the test should evaluate how effectively students achieve some specified learning objective.

To the extent that there is a reason to teach something at all, there is a reason to see whether you've accomplished it. If you don't care whether a student learns arithmetic, why teach it?

Learning involves much more than rote memorization. A good example of a (relatively) new standardized test is the Force Concept Inventory:

http://www.flaguide.org/tools/diagnostic/force_concept_inventory.php

This is the key innovation: "Each question offers only one correct Newtonian solution, with common-sense distractors (incorrect possible answers) that are based upon student's misconceptions about that topic, gained from interviews."

This exam is an attempt to directly measure student learning, not simple recall.

Yeah, sure. Tests shouldn't test memorization. (Or at least shouldn't test ONLY memorization. It's possible that there is a benefit to committing some things to memory.)

 

[rollingstein]

What most people are naturally curious about isn't what schools want to teach.

Should schools be teaching what kids are curious about or what is more likely to be useful to them to earn a living or to contribute to the skillset that society demands from them?

I think you are perfectly right that kids tend to be naturally driven & curious about certain things. But the whole point behind schooling & discipline is to teach people stuff they may not enjoy doing on their own but they ought to know. And funnily enough there are a range of activities that are not enjoyable in their initial learning curve that subsequently do become enjoyable & it takes something like school to take you through that initial uggh drudgery.

No one every enjoyed learning multiplication tables. But having learned them we now find them pretty useful.

I think one of the under-appreciated functions of school is to get us to endure the boredom of things that we do not like to do.

 

[stevendaryl]

Should schools be teaching what kids are curious about or what is more likely to be useful to them to earn a living or to contribute to the skillset that society demands from them?

I'm not making a claim about that, I'm just pointing out that the fact that students don't naturally want to lean what is taught in schools does not mean that they aren't naturally curious and driven to understand things. Just that there is a mismatch between what they are curious about and what schools teach. That might be inevitable.

 

[rollingstein]

I'm not making a claim about that, I'm just pointing out that the fact that students don't naturally want to lean what is taught in schools does not mean that they aren't naturally curious and driven to understand things. Just that there is a mismatch between what they are curious about and what schools teach. That might be inevitable.

Ah ok. My bad then. I thought you were being critical of schools for teaching what they teach.

 

[Andy Resnick]

Just to make it clear, I'm not in favor of tests as "measurements". I don't think that an overall numeric result, from 0 to 100 or 0 to 1500 (or whatever the range is for SATs) means much at all. <snip>

But this is, in fact, precisely what the function of a standardized test is: to provide numerical comparisons across the student population.

The only test I am aware of that is not associated with a numerical score is the Rorschach test.

 

[JorisL]

But this is, in fact, precisely what the function of a standardized test is: to provide numerical comparisons across the student population.

The only test I am aware of that is not associated with a numerical score is the Rorschach test.

I accept your point. But part of the video and discussion in this topic was related not only to the volume/usefulness of these tests but the quality.
As far as I can tell a lot/some of these tests require signing an agreement stating that you will not discuss the questions.
Someone also mentioned the problem with quality from personal experience with plain wrong questions in there.

Another point I find interesting is the example in the video where a student was expected to get more than 100% (486 points was the goal with only 483 points available I believe)
How can such expectations be assessed? Clearly the method of measurement as a whole has some serious flaws (using the language of physics).

At this point the discussion can take several directions e.g. are (well-designed) standardized tests useful, How can we assure well-designed tests, ...

 

[rollingstein]

As far as I can tell a lot/some of these tests require signing an agreement stating that you will not discuss the questions.

Has anyone gotten sued for discussing questions from the SAT / ACT etc? Just curious. Otherwise I think we should just disregard those agreements as unenforceable BS boilerplate. I would like to see the legal precedent on this.

You can put what you want in an agreement but ultimately you've got to find a sympathetic court that will enforce it. I want to see a jury find a kid guilty for violating Pearson's test confidentiality.

Someone also mentioned the problem with quality from personal experience with plain wrong questions in there.

What test doesn't have some wrong questions on it?!

 

[JorisL]

Has anyone gotten sued for discussing questions from the SAT / ACT etc? Just curious. Otherwise I think we should just disregard those agreements as unenforceable BS boilerplate. I would like to see the legal precedent on this.

You can put what you want in an agreement but ultimately you've got to find a sympathetic court that will enforce it. I want to see a jury find a kid guilty for violating Pearson's test confidentiality.

Well do you see people discussing questions anywhere? The post I referenced is post #28 by Fredrik, check it out. They hide behind the agreement.
And it works, which kid thinks about this stuff? They probably just keep it going long enough for the other party to either get fed up or out of money.

Point is its rotten, and if they feel their position is threatened, I'm certain something will happen.
Money makes things happen, remember?

What test doesn't have some wrong questions on it?!

And which teacher doesn't agree when you explain in detail why it is wrong? More importantly how can you without risk of prosecution get a second opinion?

Even when there are vague parts of a question you either ask for clarification or get back to the teacher afterwards if this is somehow impossible.

Finally, these are standardised tests.
Isn't the written test you take when getting your drivers license standardised? Here it is, I never ever heard of an error in the tests even right after changes in the law (it is easier for errors to slip in one would think).
Point is if a lot of students take these tests isn't it absolutely necessary to check, double-check, ... the tests?
Also I'm sure they have a lot of questions used for many years if not decades. Shouldn't those be 100% correct?

 

[Fredrik]

I should perhaps clarify that my experience isn't with the tests made for school kids. It's with a couple of professional certification exams (made by one of the Pearson companies). What they can do if you violate the agreement is to kick you out of the certification program. Now you have spent at the very least a few hundred dollars (possibly many thousands, if you took classes or bought hardware to practice on), and you're no longer certified, even though you passed the exam. I don't know what they would do if a high school kid would violate the confidentiality agreement.

Even if you want to violate the confidentiality agreement, it's pretty difficult to do that, especially for the kind of tests I did. It's difficult to remember the questions exactly, and you have to hand in all the notes you've made at the end of the exam. You're not even allowed to erase them. You're also not allowed to look at a question again once the test is over, not even to provide feedback about possible issues with a question. And you don't really have time to try to memorize the questions. The exam I did was extremely difficult to complete on time. You basically had to cheat, or remember what you did last time you took the test.

What I found especially bizarre about my experience was that they were completely unwilling to discuss any specific points I had made. Instead of trying to refute my arguments, they just said that there's nothing wrong with the test, even though I know for sure that some of the questions were bad.

It certainly seems to me that (at least with these professional exams), the rules are in place to ensure that they don't have to make the tests good.

 

[stevendaryl]

But this is, in fact, precisely what the function of a standardized test is: to provide numerical comparisons across the student population.

The only test I am aware of that is not associated with a numerical score is the Rorschach test.

If we are talking about SAT or ACT, then I agree with you that the point is to get a numerical score for the purpose of comparison between students. But is the word "standardized test" limited to those sorts of measurements?

To give you a counter-example: You can go to a website such as: http://www.sheppardsoftware.com/African_Geography.htm to test your knowledge of the countries in the continent of Africa. Now, you might object that there is no reason to know the names, locations and capitals of the countries in Africa, but it's just a simple example of knowledge that can be tested through a standardized test. The point of such a test is certainly NOT to compare the student to other students. It is NOT to come up with a numerical score: 0 to 100 (what percentage of the countries in Africa can you name). The point of such a test is to see if you DO know the countries in Africa. If you get them all right, then you do. If you miss even one then you don't. You can retake the test as often as you like, until you get 100%. Then you know all the countries in Africa (well, at least until you forget them).

To me, the proper goal of a test is to assess how well a student understands a subject. That can be done, at least with some subjects, using standardized tests.

Now, the SAT has all these questions that are not actually about understanding a subject, but seem to be some kind of measurement of mental fitness. It's been a long time since I've taken it, but back in the day, there were questions along the lines of:

Here is a sequence of pictures. Based on the pattern, what is the next picture in the sequence?

There were questions along the lines of:

Mustard is to hot dog as pickles are to what?

These questions were sort of interesting to me, because it was a challenge to figure out what the test-creators had in mind. I was pretty good at that sort of thing, but I'm not convinced that there is a strong point in asking those types of questions.

 

[WWGD]

<Snip>:

Here is a sequence of pictures. Based on the pattern, what is the next picture in the sequence?

There were questions along the lines of:

Mustard is to hot dog as pickles are to what?

These questions were sort of interesting to me, because it was a challenge to figure out what the test-creators had in mind. I was pretty good at that sort of thing, but I'm not convinced that there is a strong point in asking those types of questions.

Ah, yes, what is _the_ next figure. If you do not fit into their narrow world/experiential view, you are wrong. Same with sequences of numbers, other than obvious ones like 1,2,3,4,... I am remembering the phrase " limit your imagination, keep you where they must".

 

[Czcibor]

Now, what does a specific test actually measure? Much ink has been spilled on this topic. For example, the strongest correlation with SAT scores is family income- not ''aptitude'.

http://blogs.wsj.com/economics/2014...me-inequality-how-wealthier-kids-rank-higher/

(The article that you linked just shows that kids of richer parents get better score, without analysing any other factors like aptitude or heredity of IQ. I'm not saying that you are wrong, I'm merely pointing that the source that you linked does not prove your point)

I should perhaps clarify that my experience isn't with the tests made for school kids. It's with a couple of professional certification exams (made by one of the Pearson companies). What they can do if you violate the agreement is to kick you out of the certification program. Now you have spent at the very least a few hundred dollars (possibly many thousands, if you took classes or bought hardware to practice on), and you're no longer certified, even though you passed the exam. I don't know what they would do if a high school kid would violate the confidentiality agreement.

Even if you want to violate the confidentiality agreement, it's pretty difficult to do that, especially for the kind of tests I did. It's difficult to remember the questions exactly, and you have to hand in all the notes you've made at the end of the exam. You're not even allowed to erase them. You're also not allowed to look at a question again once the test is over, not even to provide feedback about possible issues with a question. And you don't really have time to try to memorize the questions. The exam I did was extremely difficult to complete on time. You basically had to cheat, or remember what you did last time you took the test.

What I found especially bizarre about my experience was that they were completely unwilling to discuss any specific points I had made. Instead of trying to refute my arguments, they just said that there's nothing wrong with the test, even though I know for sure that some of the questions were bad.

It certainly seems to me that (at least with these professional exams), the rules are in place to ensure that they don't have to make the tests good.

They behave even better than infallible beings that my gov put into examination boards. Except that now such boards in my country are being challenged as unconstitutional.

 

[verty]

Now, what does a specific test actually measure? Much ink has been spilled on this topic. For example, the strongest correlation with SAT scores is family income- not ''aptitude'.

http://blogs.wsj.com/economics/2014...me-inequality-how-wealthier-kids-rank-higher/

The data in that article is (are?) interesting...

Certainly I would think writing would scale with aptitude or intelligence. If one ignores spelling or simple grammar mistakes, the quality of writing is going to scale with ability. And writing is surely something that can't be standardized, it needs someone to mark it. So it seems reasonable prima facie to be able to look at the writing scores and read off how ability trends with wealth.

And looking at the data, writing scores ramp to $100k, are flat to $200k, and ramp thereafter. It's reasonable to assume those >$200k households have children who were privately tutored or have parents who are doctors, they are the geniuses and score very well. But we see that most middle class households score the same. I think it's reasonable to suppose that the sub $100k households are predominantly in poorer neighborhood and have schools that aren't as good, or that there could be a language bias in the writing scores for poorer households. So there's no real evidence from the writing scores that ability trends with wealth.

Reading is the same: a ramp to $100k, flat to $200k, ramping thereafter. It's clear the reading questions are sufficiently elementary that the same is true, any language bias only shows up in poorer households.

Math however is a consistent ramp, rising to the right. We know that this isn't measuring ability because it differs from the writing and reading scores. And math performance in general is contingent on quality of teaching/schooling. So for me it says more about the quality of the schools than any kind of proportion between ability with wealth.

So I see no evidence in the data that there is a trend between ability and wealth. The claim that the SAT is more a measure of affluence than ability would seem to be on point.

I apologize, the math score is not a good measure of ability but the reading and writing scores seem to be pretty good; a decent score on the reading section is quite a reliable indicator of ability. Perhaps this is more toward the concept of emotional IQ.

 

[RaulTheUCSCSlug]

Interesting since one of the students in my friends class got stressed during class (during a proof on logic or something like that) and vomited. But we only take at most two midterms, and one final. I think the stress on a person depends on the student, rather than what the student is doing. What's the difference between standardized tests, and regular chapter tests like the ones I did in high school?

Also common core is fairly new. Of course the students that have to join the program abruptly will suffer, but students that will grow up with the common core system may do better in things like math and science. At least that is the idea.