Calculating Signal to Noise Ratio for Multiple Choice Exams

[curiousasker]

I've been trying to figure out how to get the signal to noise ratio of a completely multiple choice type exam. I can't do it and it's driving me nuts, so finally I found this website and decided I'd ask. :D

Say you let your students take a multiple choice test and you are able to get all their test answer sheets. How would you go about finding the signal to noise ratio, where the signal is the correct answer?

 

[SW VandeCarr]

I've been trying to figure out how to get the signal to noise ratio of a completely multiple choice type exam. I can't do it and it's driving me nuts, so finally I found this website and decided I'd ask. :D

Say you let your students take a multiple choice test and you are able to get all their test answer sheets. How would you go about finding the signal to noise ratio, where the signal is the correct answer?

There are a number of ways to express the signal to noise ratio in science and engineering depending on the specifics. In your case it would seem straightforward. If a correct answer is a signal and a wrong answer is noise and there are no other possibilities then the raw signal to noise ratio is SNR = \frac{signal}{noise} = \frac{n (correct.answers)}{n(incorrect.answers)}. Sometimes the ratio is expressed logarithmically. It depends on what you want to do.

 

[curiousasker]

So it's simply the average of the test scores divided by the average of the part they got wrong? Isn't noise the variance of the signal, rather than the incorrect answers?

The way I understand signal to noise ratio is through the gauge r&r, but that requires students to retake the same test, and that both test takings be independent events. It's not possible to have that since their second take will definitely be affected by their first take. So I'm not quite catching how to get SNR without repeat and independent trials.

It puzzles me because if a 40 item test is perfected by everybody, the mean of the correct answers is 40 and the variance is zero, so the SNR is infinite. In the same way, if it was the incorrect answers, the noise would still be zero and the SNR would be infinite.

Also I know that SNR is expressed in decibels most of the time, but what would the SNR in this particular case be expressed as?

 

[SW VandeCarr]

So it's simply the average of the test scores divided by the average of the part they got wrong? Isn't noise the variance of the signal, rather than the incorrect answers?

The way I understand signal to noise ratio is through the gauge r&r, but that requires students to retake the same test, and that both test takings be independent events. It's not possible to have that since their second take will definitely be affected by their first take. So I'm not quite catching how to get SNR without repeat and independent trials.

It puzzles me because if a 40 item test is perfected by everybody, the mean of the correct answers is 40 and the variance is zero, so the SNR is infinite. In the same way, if it was the incorrect answers, the noise would still be zero and the SNR would be infinite.

Also I know that SNR is expressed in decibels most of the time, but what would the SNR in this particular case be expressed as?

Frankly I've never heard of the SNR being applied this way. For a voltage dependent signal a standard formula would be SNR = 20_{log 10} (V_s/V_n). Note it is still a simple ratio with a logarithmic conversion to suit the particular situation. You could also express the SNR in terms of the mean and standard deviation of a signal. In this case that would be SNR = \mu/ \sigma.

I can't tell you how to best express it for your purposes other than what I've already done because I don't know your objectives. And yes, the signal to noise ratio can be infinite but almost never is in situations where it is usually applied. Perhaps someone else will assist you.

 

[curiousasker]

Hi. Thanks for the quick response :)

What I'm trying to do is something like in this article here (under Discrete Stimuli). It's basically how to get how strong is the signal over the noise in an experimental set up? I don't quite understand the article so I'm looking for help (and I've only just stumbled upon it too).

Unfortunately, the test method is a discrete case, so finding a way to measure by the usual decibels or voltage is a little difficult for me.

 

[haruspex]

As far as I understand it, to determine the SNR you need to be able to compare the output when there's no signal with the output when there is a signal.
Moreover, I would have thought that in the context of multiple choice questions the signal was any answer the student actually believed and the noise was the random guesses; can't see how you'd distinguish those in general, but you might be able to put a limit on it. E.g. If it's a five-way choice each time and the student scores 20% it could well be all noise, whereas if the student scores 90% then it's unlikely to be much more than 13% noise.

 

[curiousasker]

Hi haruspex. I figured it out. I'll say it here since it might be interesting to you guys =)

It's signal detection theory that should be used in this case, because it can identify hits, false alarms, misses, and correct rejections (basically signal present-response present; signal absent-response present; signal present-response absent; and signal absent-response absent).

But since it's a multiple choice, they always have to answer (so it's always signal present). What I used here was a k-alternative forced choice analysis instead. What that does is basically determine how well people can distinguish the signal from k-1 noises (ie, wrong choices). It calculates the distance d' from the mean of the correct choice and the mean of the incorrect choices, which says something about their discrimination ability.

Fortunately, (d')^2 = SNR for this kind of test (when there are only two response categories = {correct,incorrect}).

So that's how I got the SNR for a multiple choice test. It even gives the answer in terms of a linear power ratio, not decibels, which is what I need for my purpose =)