Is this a horribly ambiguous A Level Physics question?

[ChrisXenon]

I can't think of any solution to this problem that I would not preface with HORRORS.

1 HORRORS no standardized tests. Every teacher invents his own standard.
2 HORRORS every standardized test grader decides for himself which answer is correct.
3 HORRORS the wordings of questions and answer choices is designed by committee.
4, 5, 6, ... there is no end to horrible choices.

Could it be that the underlying evil is the multiple choice question?

Sorry I don't understand your point.

 

[ChrisXenon]

This thread will shortly become unmanagable but here's the latest crank of the handle with OCR. My inline response to the question setter begin with >>

Thank you for your continued correspondence on mark schemes [...]

Note that the question is not about LEDs in general, but about a particular LED.

>> I understand but do not see the relevance of your distinction.

Candidates have been provided with a graph of the I-V characteristic of this LED and would be expected to
use it in their response. We are always careful to make sure that it is clear whether a question
refers to a specific object or a class of objects, as has been done here.

>> OK but again - I don't see the relevance here

The command “Describe and justify...” requires candidates to do two things for each potential
difference range. First they must describe the variation of resistance, and second they must
provide some evidence for that variation with the expectation that this comes from the
information given to them in the question, ie the graph.

>> You are describing your intended interpretation. I already knew it once I read your marking scheme.
But the significant fact is that I did not know it before - it was not evident from the question.
Someone on the physics forum has said OCR publishes a set of verbs and special meanings.
Is that true? Where would I find it? Honestly, I HATE that idea - it is all wrong. Use the existing language
effectively and unambiguously - don't redefine it in an appendix which no one will ever see then punish them for misunderstanding it!

The mark scheme and examiner’s report are clear that this sort of response is exactly what was given by candidates, and therefore that
the question was not found to be ambiguous.

>> If most students got it right, then I suppose most students gave the interpretation you intended.
But I am surprised if that is so, and it doesn't change the fact that the term "justify" could legitimately mean a multitude of things.
To satisfy me you'd need to explain why the questions CANNOT mean "explain this behaviour in terms of semiconductor physics".

The answer given in the main part of the scheme is generally a minimally acceptable answer,
with clarification in the guidance column. It is very likely that a more discursive response about
the general properties of semi-conductors would be awarded all four marks since it would
contain the necessary information. Since the given mark scheme would cover such a response,
there would not be any need to include any mention of it in the mark scheme.

>> You seem to be saying (I paraphrase with no intention to offend you):
"It is OK to ask an ambiguous question because in a lengthy written answer the candidate is likely to provide what is required inadvertently"
I assume I must have misunderstood you. Please clarify.

Regards
Chris

 

[anorlunda]

Sorry I don't understand your point.

My point is similar to what you said in a later post.

HOWEVER I agree that this is a DREADFUL way to conduct education. My student clearly did not and does not have any knowledge of these special verb usages, and so I imagine, neither do his teachers.

The language of physics is mathematics. This thread discusses an exam question posed verbally, with multiple choice answers given verbally. Then we run into problems because natural language is imprecise and subject to interpretation.

The solution would be the old fashioned type of exam where the student is required to derive his/her own equations and solve for the answer. The symbols used are the student's choice. We use math as much as possible in the question and the answer. It is the technical analog to "essay" type questions.

Of course that type of exam means many fewer questions are possible in the allotted time. It is also much more demanding and time consuming on the person doing the grading. It also introduces some grading subjectivity. (Should I allow 8 out of the max 10 points for this answer?) But it avoids exactly the dependency on word definitions you are complaining about.

I always thought that the drift towards multiple choice verbal exam questions, is motivated by the ease of machine grading. Multiple choice also allows more questions in the exam, e.g. 100 shallow questions instead of 10 in-depth questions.

 

[f95toli]

HOWEVER I agree that this is a DREADFUL way to conduct education. My student clearly did not and does not have any knowledge of these special verb usages, and so I imagine, neither do his teachers. Whatever the intended outcome, the actual outcome is shambolic in a way which could fail him. My student, by the way, is attending a very expensive public school (which in the UK means one you pay to go to, outside of taxes).

By now your student should have a pretty good idea of what is expected of him when answering a question; and that includes how to interpret the verbs.

It is important to note that when studying for an A level in a subject in the UK the students are NOT taking a "general" A-level course. Everything -including the syllabus- is set by a specific exam board. Hence, if the teacher has done his/her job right the students will from day one be asked questions of the same type, where the same language is used.

The way the system is set up it is certainly not about learning physics; it is about learning how to pass an exam from a specific exam board.

At my step-sons school they stopped teaching them new material back in November; for the past few months they have only been revising; it reality practicing how to answer questions in the exams (which start in June).

They also spent a couple of months preparing for the mocks in year 1. Hence, out of a two year course a little over half is spent on revising and preparing for the exams; NOT learning new material.

So yes, the system is broken (by design) but there isn't much you can do about it.

 

[ChrisXenon]

f95toli, my commments inline with yours:

By now your student should have a pretty good idea of what is expected of him when answering a question; and that includes how to interpret the verbs.

>> Well, by now I should be slimmer and more patient - but I ain't either. The reality is that he has had no training in the special use of English foisted upon us by OCR. And, as I've mentioned that arbitrary reality is also un-necessary and tiresome.

It is important to note that when studying for an A level in a subject in the UK the students are NOT taking a "general" A-level course. Everything -including the syllabus- is set by a specific exam board. Hence, if the teacher has done his/her job right the students will from day one be asked questions of the same type, where the same language is used.

>> Same commens as above. What should be and what is are not related here.

The way the system is set up it is certainly not about learning physics; it is about learning how to pass an exam from a specific exam board.

>> The two should be very closely allied or soemthing is very wrong. What's a physics A level FOR, for goodness sake? I suggest it should not be to prove someone can learn and apply some alternate Engloid language.
At my step-sons school they stopped teaching them new material back in November; for the past few months they have only been revising; it reality practicing how to answer questions in the exams (which start in June).
They also spent a couple of months preparing for the mocks in year 1. Hence, out of a two year course a little over half is spent on revising and preparing for the exams; NOT learning new material.

I don't see anything with revising or consolidating mateial or techniques. What I'm objectgivnig to is the use of ambiguous questions and non-standrard inter[retations of English.

So yes, the system is broken (by design) but there isn't much you can do about it.

>> I thinh there is a lot people can do when they try.

 

[pbuk]

I did not know there were board definitions for action verbs or any other part of Enligh language. I will need to obtain them an use them in my teaching, and I will have to be on my guard for defficiencies like this in the future.

Oops sorry, I meant to provide a link https://www.ocr.org.uk/Images/273311-command-verbs-definitions.pdf - but now I look again I see this is for OCR 'Cambridge Technicals' not A-levels so there may be some differences. There will be an equivalent somewhere on the OCR site - you will get access to more materials if you register as a teacher, and even more if you pay for the teachers' pack.

HOWEVER I agree that this is a DREADFUL way to conduct education. My student clearly did not and does not have any knowledge of these special verb usages, and so I imagine, neither do his teachers. Whatever the intended outcome, the actual outcome is shambolic in a way which could fail him. My student, by the way, is attending a very expensive public school (which in the UK means one you pay to go to, outside of taxes).

Fee-paying schools in the UK are usually very good at this aspect of getting students through exams, but almost all schools are having difficulty with recruitment for Physics in particular - this is and has been an increasing problem over the past decade.

Any system which finds the need to redefine the English language is, in my view, idiotic. What is wrong with using the existing language effectively? Why not say "justify by reference to the graph"? I can confidently say that there is a hell of a lot wrong with saying "Justify*" and then - in some place no student will ever see, have this:

* Note: Any and all words may not have their normal meaning. The OCR board reserve the right to re-define English to the utter bewilderment of humanity. No responsibility accepted. Your mileage may vary. My dad's a policeman.

I have to disagree with you there, I think that defining terms clearly enables the examiner to communicate what is required more effectively than would otherwise be the case. They are not out to trick you, and if they present you with a graph and and ask you to justify an answer then it is clear that they want you to justify your answer by reference to the graph.

The rated of acceleration question is here and the marking scheme is here

I see for 1(c)(i)2 either 'constant deceleration' or 'constant negative acceleration' is allowed so that is not at issue. However I agree that for 1(c)(i)1 'The object is accelerating at a constant rate' should be an acceptable answer, and the mark scheme implies that it is not. In situations like this, further insight can often be gained from the examiner's report (remembered to link it this time!). Here he says "The correct way to have used rate to score the mark would be to write 'increasing rate of change of velocity', which a few candidates did." Further insight into the examiner's point of view is in the comment to the first part of this question: "A noticeable incorrect answer was 'the rate of change of velocity per unit time';candidates need to be aware that in dynamics the term 'rate' includes per unit time." This examiner clearly has a bee in their bonnet about the use of the word 'rate', and like you I think he is taking it too far. However now you know this you know that you need to be careful when talking about rates in OCR A level Physics exams (assuming this situation persists - check more recent exams).

As someone mentioned further up the thread, top schools and crammers finish teaching the syllabus by the end of the autumn term (assuming a 2 year cycle - AS's are a very unwelcome distraction). The rest of the time before the exams is spent making sure through practice that students know how to read questions to find out what the examiner wants, and know how to turn their knowledge into an answer that gives it to him.

I still don't se ehow it can't mean "present a reasoned case in terms of semiconductor physics".

Because (i) there is a graph in front of you which you are expected to use and (ii) semiconductor physics is not part of the syllabus.

 

[Mark44]

Of course, appeaking to "common interpretations" is dodgy ground in the physics world where common parlance is woefully inadequate, but I continue to think that the stipulation to punish "rate of acceleration" is wrong.

Like you say, "common interpretations is dodgy ground in the physics world..." Acceleration is already a rate, so the phrase "rate of acceleration" muddies the waters, IMO. To me, "rate of acceleration" just seems sloppy.

And in fact, if rate DOES mean that - then what on Earth does "rate of change" actually mean? Or is it tautological?

"Rate of change," is more precisely phrased as "rate of change of one variable with respect to another." In terms of the OP, velocity is the time rate of change of position, and acceleration is the time rate of change of velocity. There is a physical term for the time rate of change of acceleration - jerk.
In more mathematical terms where position s is a function time, velocity $v = \frac{ds}{dt}$, and acceleration $a = \frac{dv}{dt} = \frac{d^2s}{dt^2}$, which I assume you already know. Before anyone jumps in, the equations represent motion in one dimension, so I'm ignoring the vector aspects.

 

[f95toli]

The two should be very closely allied or soemthing is very wrong. What's a physics A level FOR, for goodness sake?

I don't think there is a clear answer to this.

And no, I am not trying to be obtuse.

One of the things that frequently comes up when discussing the current system is that there are quite a few people (including members of government) who see A-levels, including the exams, as a way of "demonstrating potential". That is, the goal is not actually to learn the material but to show that you have the potential to learn once you are accepted to university. Hence, from this point of view A-level science is a bit like a game: the challenge is to win while -mostly- playing by the rules. However arbitrary.

This is the very same argument that is often used to justify why so many kids have to learn Latin in school: it is not about learning a language but about "training your brain".

I realize that none of this is -unfortunately- helping your current situation. But I think it is important to understand that your job as a tutor is NOT to teach your student physics but to help him pass the exam. The only way to do this is by learning the "rules" set by the exam board.
I think just about everyone in the academic community agrees that this is broken system; but this is ultimately a political question; there is nothing you as a tutor can do about it.

(some of my PhD students have made extra money by also tutoring A-level physics, so this is something that I've had many discussion about)

 

[sysprog]

I disagree, on etymological grounds, with the opposition to restriction of the use of the word 'acceleration' to non-negative values only. The prefix 'ac' means to; not from. The term 'acceleration' refers to increase in celerity. We have the prefix 'de' to allow denotation of decrease in celerity, i.e. 'deceleration'.

Similary, 'rate' being used consistently as non-negative allows us to use terms like 'rate of increase' and 'rate of decrease' without fear of ambiguity. It's cumbersome to call acceleration a 'negative rate of decrease in speed', even though it can be logically parsed.

The logics of the terms of natural language ideolects are not exclusively the province of scientists, and when there is no strong reason to depart from broader conventions, it is preferable that the experts of the various disciplines not feel compelled to alienate one another by non-standard usage.

 

[jedishrfu]

The underlying problem here is that everyday English is ambiguous and scientific usage is not or tries not to be so. These questions are often created by test developers not well versed in the subject who are likely adjusting an older version of the question to the current test to avoid repeated questions.

In some ways, it remind you of Family Feud where they ask for answers to some broad question and then compare your answers via "and the survey said..."

I know when I was in school, I would get confused by word problems especially percentage based ones confused in how to correctly apply the percentages or in computing them.

You'll probably need to create a lexicography of terms and what they most likely mean for this group of tests.

Also one needs to be careful not to over think the question. I know in my tutoring time, I would read more into a problem than what was present and get the student momentarily confused.

 

[ChrisXenon]

The underlying problem here is that everyday English is ambiguous and scientific usage is not or tries not to be so. These questions are often created by test developers not well versed in the subject who are likely adjusting an older version of the question to the current test to avoid repeated questions.

In some ways, it remind you of Family Feud where they ask for answers to some broad question and then compare your answers via "and the survey said..."

I know when I was in school, I would get confused by word problems especially percentage based ones confused in how to correctly apply the percentages or in computing them.

You'll probably need to create a lexicography of terms and what they most likely mean for this group of tests.

Also one needs to be careful not to over think the question. I know in my tutoring time, I would read more into a problem than what was present and get the student momentarily confused.

Thanks jedishrfu. I understand the concerns you raise, but for me the solution remains in linguistic precision - simply saying only and exactly what you mean.All other strategies ACCOMMODATE a problem - this one ELIMINATES the problem.

 

[sysprog]

Thanks jedishrfu. I understand the concerns you raise, but for me the solution remains in linguistic precision - simply saying only and exactly what you mean.All other strategies ACCOMMODATE a problem - this one ELIMINATES the problem.

I think that's often easier said than done, or not necessarily sufficient, e.g.:

officer to motorist: the speed limit here's 65; you were doing 85.​

motorist to officer: those two statements can't both be true.​

officer: you know exactly what I meant.​

motorist: I'm not clairvoyant. If I was doing 85, 65 can't really be the limit.​

o: ok, the legal speed limit is 65, you were doing 85, that means you violated the law.​

m: the sign didn't say legal speed limit, and neither did you at first.​

o: what did you think it meant?​

m: I thought it meant only and exactly what it said -- that nothing can go faster than 65 here.​

o: well then it would have said speed limit c instead of speed limit 65.​

m: there's no need for a local sign for a universal speed limit.​

o: ok, here's your ticket, you now know that those signs refer to legal speed limits.​

​

 

[pbuk]

@jedishrfu @sysprog @... - again, UK public exam boards now remove the majority of this uncertainty by publishing dictionaries of terms, syllabus requirements and even teaching materials. This may not be the case in other countries (and was not the case in the UK until ~20 years ago).

 

[f95toli]

The underlying problem here is that everyday English is ambiguous and scientific usage is not or tries not to be so. These questions are often created by test developers not well versed in the subject who are likely adjusting an older version of the question to the current test to avoid repeated questions.

That is -or at least should not- be the case here. The exam boards employ professional test developers who are at -least nominally- specialists in their subjects. They spend months developing each questions and the marking scheme (I actually know someone who does this for a living, albeit for math).
These are national tests here in the UK meaning any mistake (and they do happen...quite frequently) will literally end up in the national news.

Hence, things are -at least mostly- the way they are by design. That it is a bad design is a separate issue.

 

[jedishrfu]

I’ve not seen this happen often in the US. I’ve heard of students challenging the answers. In my time, one couldn’t even do that because nothing about the test was ever made public. Your best resource were test prep books where folks reverse engineered the problems at least that’s how we thought they got them.

 

[ChrisXenon]

I don't think there is a clear answer to this.

And no, I am not trying to be obtuse.

One of the things that frequently comes up when discussing the current system is that there are quite a few people (including members of government) who see A-levels, including the exams, as a way of "demonstrating potential". That is, the goal is not actually to learn the material but to show that you have the potential to learn once you are accepted to university. Hence, from this point of view A-level science is a bit like a game: the challenge is to win while -mostly- playing by the rules. However arbitrary.

This is the very same argument that is often used to justify why so many kids have to learn Latin in school: it is not about learning a language but about "training your brain".

I realize that none of this is -unfortunately- helping your current situation. But I think it is important to understand that your job as a tutor is NOT to teach your student physics but to help him pass the exam. The only way to do this is by learning the "rules" set by the exam board.
I think just about everyone in the academic community agrees that this is broken system; but this is ultimately a political question; there is nothing you as a tutor can do about it.

(some of my PhD students have made extra money by also tutoring A-level physics, so this is something that I've had many discussion about)

I agree that my personal and immediate goal is to return value ot my paying clients/students.
My hope is that

I think that's often easier said than done, or not necessarily sufficient, e.g.:

officer to motorist: the speed limit here's 65; you were doing 85.​

motorist to officer: those two statements can't both be true.​

officer: you know exactly what I meant.​

motorist: I'm not clairvoyant. If I was doing 85, 65 can't really be the limit.​

o: ok, the legal speed limit is 65, you were doing 85, that means you violated the law.​

m: the sign didn't say legal speed limit, and neither did you at first.​

o: what did you think it meant?​

m: I thought it meant only and exactly what it said -- that nothing can go faster than 65 here.​

o: well then it would have said speed limit c instead of speed limit 65.​

m: there's no need for a local sign for a universal speed limit.​

o: ok, here's your ticket, you now know that those signs refer to legal speed limits.​

​

Sysprog thanks for your input. Of course you're right but, it seems to me, not in a way that's relevant here. The issues before us are, fore example. the use of the word "justify" in the phrase "justify your answer". The dictionary definition of that word does not help us. The usage is ambiguous. The questions setter meant "by using the graph" and two of the readers (me and my student) thought he meant "in terms of semiconductor physics". The removal of that ambiguity is trivially eaasy. It should have been removed.

For me, that's the end of the story. Whilst it's true that language can and maybe has to be ambiguous sometimes, it doesn't have to be here, so I don't care.

 

[ChrisXenon]

I’ve not seen this happen often in the US. I’ve heard of students challenging the answers. In my time, one couldn’t even do that because nothing about the test was ever made public. Your best resource were test prep books where folks reverse engineered the problems at least that’s how we thought they got them.

I'm seeing this with another student, this itme doing City & Guilding level 2 English & Maths. My student keeps failing, and they consistently refuse to say how or where and I can't think of a good reason for that. The BAD reasons I can think of for that are (a) they can't be bothered and (b) they don't want to be accountable in case they're found incompetent.

 

[ChrisXenon]

That is -or at least should not- be the case here. The exam boards employ professional test developers who are at -least nominally- specialists in their subjects. They spend months developing each questions and the marking scheme (I actually know someone who does this for a living, albeit for math).
These are national tests here in the UK meaning any mistake (and they do happen...quite frequently) will literally end up in the national news.

Hence, things are -at least mostly- the way they are by design. That it is a bad design is a separate issue.

You say the test developers are "at least nominally [competent]". That's not good enough! They should be actually and demonstrably conspicuously competent, in my view.

Maybe things are the way they are through [bad] design and that may be a separate issue, but either way, it's an issue which is matters and seems tome to be entirely and easily avoidable.

 

[ChrisXenon]

@jedishrfu @sysprog @... - again, UK public exam boards now remove the majority of this uncertainty by publishing dictionaries of terms, syllabus requirements and even teaching materials. This may not be the case in other countries (and was not the case in the UK until ~20 years ago).

Can anyone explain why this isn't a better strategy?

1. Use the Oxford English Dictionary to find the meanings of words
2. Use language intelligently and unambiguiously.
3. Do not publish your own damned dictionary!

 

[Mark44]

Can anyone explain why this isn't a better strategy?

1. Use the Oxford English Dictionary to find the meanings of words
2. Use language intelligently and unambiguiously.
3. Do not publish your own damned dictionary!

The OED lists all definitions for a given word, so relying on it alone might not result in an unambiguous meaning.
A short list of key words and what they mean in the context of the exam can reduce ambiguity.

 

[jedishrfu]

One funny story happened to my wife while in her MBA program. The qualifying test had a question on accounting methods and she was asked:

What was the difference between cost accounting and some other accounting scheme for the described business?

She described them both in detail highlighting their differences and got it wrong.

Later during the oral, the prof asked her about that question and told her what he really wanted was the numerical difference to which she pointed on her paper at another spot and said here it is in part B. He quickly passed her after that.

One could chalk it up to a language barrier as she was not a native speaker of English but I figured it was more incumbent on the professor to have seen that his question was ambiguous and that after having seen her answer he should have realized that and should not have marked off for it.

 

[jedishrfu]

Even lawyers do this in contracts in order to avoid confusion unless of course they want to confuse.

 

[sysprog]

Even lawyers do this in contracts in order to avoid confusion unless of course they want to confuse.

Many state and federal laws in the US begin with a 'definitions' section.

I think that would be a good standard practice for important standardized tests.

Although providing precise definitions for such terms as 'energy', 'force', and 'velocity' would be like providing crib notes, the multiple choice tests could have a general definitions statement saying that definitions for terms not defined in the question or test glossary, that are used differently in different standard texts, and are used in the question, should be resolved in favor of the definition that allows exactly one of the answers to be correct. If one standard definition led to one answer, and another to a different answer, that would strongly indicate a fault in the test.

 

[haruspex]

the marking scheme forbade the term "rate of acceleration"

Because acceleration already is a rate

That does not make "rate of acceleration" wrong.
There is no scientifically defined meaning to the expression "rate of X". Rate of change of X is defined, in terms of X and time, but not rate of X. Rate of change of acceleration would be the third derivative.

The only accepted usage of "rate of X", where X is a quantity rather than a delta to a quantity, is vernacular, so should be evaluated in that context. We do not say "rate of speed", but we do say rate of pouring, rate of setting, rate of acceleration... In each case, the X implies a change in something and the "rate" is the rate of that change. In the specific case of acceleration, the implied change is in velocity, so rate of acceleration just means rate of change of velocity.

 

[Mark44]

That does not make "rate of acceleration" wrong.
There is no scientifically defined meaning to the expression "rate of X". Rate of change of X is defined, in terms of X and time,

You're sort of making my point. "Rate of acceleration" is very sloppy, so in an exam where the precise use of language is important, "rate of acceleration" is IMO wrong.
I said this in post #37:

"Rate of change," is more precisely phrased as "rate of change of one variable with respect to another."

If you omit the "with respect to another variable part," the meaning is only implied, and therefore not as clear as it could be or should be.

 

[sysprog]

We all know that misuse of language is not uncommon; I think that ordinary understanding corrects some of the sloppiness of ordinary language; I agree with those who have opined here to the effect that problem posers are among those who should strive for non-misuse, and non-sloppy use, of language.

"the suspect was traveling at a high rate of speed"
"what was he suspected of?"
"travelling at a high rate of speed"

How can such reporting and responding be 'justified'?

 

[ChrisXenon]

The OED lists all definitions for a given word, so relying on it alone might not result in an unambiguous meaning.
A short list of key words and what they mean in the context of the exam can reduce ambiguity.

One funny story happened to my wife while in her MBA program. The qualifying test had a question on accounting methods and she was asked:

What was the difference between cost accounting and some other accounting scheme for the described business?

She described them both in detail highlighting their differences and got it wrong.

Later during the oral, the prof asked her about that question and told her what he really wanted was the numerical difference to which she pointed on her paper at another spot and said here it is in part B. He quickly passed her after that.

One could chalk it up to a language barrier as she was not a native speaker of English but I figured it was more incumbent on the professor to have seen that his question was ambiguous and that after having seen her answer he should have realized that and should not have marked off for it.

In my view, this is a no-brainer. The question-setter is entirely at fault and - being in the position where he gets to set questions in exams - shuold have known far better.

 

[ChrisXenon]

Even lawyers do this in contracts in order to avoid confusion unless of course they want to confuse.

What lawyers do, in my view, is very far from what ought to be done by anyone.

 

[ChrisXenon]

This thread has been interesting and somehwt helpful for me, but I'm struggling to form an overall view of your opinions. It would be helpful to me, to be able to say N% of those who responded felt the questions/marking schemes examined were Perfect/Imperfect-but-serviceable/Unacceptably poor. I can't see a poll feature in the forum - does anyone know of a good way to get this summary here?

 

[jedishrfu]

Truthfully, a poll isn't going to help your students. Basically you need to adapt or extend your tutoring in a way that let's the student make a best guess at the meaning of the words in the problem so as to get the best possible outcome in the test. The test isn't going to change because of the poll.

Its a lot like a soccer game with two equally matched teams. Its the little mistakes that add up to create the final outcome and you as a coach can only do so much. In this case, it's your students against the test developers only the ref is on their side** and the student must make the best of it.

** note 1: their is used ambiguously here although we should know it refers to the test developers.

** note 2: by being on their side, I mean the test developers are the ones who provide the questions (ambiguous or not) and the answers right or wrong (we hope right) and the student's score is based on that.

** note 3: because of the small level of uncertainty, its hard to always get a perfect score and the coach who trains them well shouldn't be blamed.

Hang in there and teach your students well (I know it sounds like a Crosby Stills and Nash song lyric)

** note 4: Crosby Stills and Nash song