MHB Chance performance on a yes/no task where no is correct 2/3 of the time?

[ltuff1]

Title pretty much summarizes it. I know I can use the binomial distribution to calculate the chance of equal to or fewer than a certain number correct on a yes/no task where there's a 0.5 probability of being right on any trial. But what if you have 27 Yes/No trials where Yes is the correct answer on only 9 of the trials? Responding Yes to every trial would yield 9/27 correct responses while responding No to every trial would result in 18/27 correct responses. So the expected correct when the subject's Y/N responding is 50/50 is more like 10.5 than 13.5 out of 27. I need to calculate the probability that a certain number total correct is below chance...ltuff1

 

[Jameson]

Title pretty much summarizes it. I know I can use the binomial distribution to calculate the chance of equal to or fewer than a certain number correct on a yes/no task where there's a 0.5 probability of being right on any trial. But what if you have 27 Yes/No trials where Yes is the correct answer on only 9 of the trials? Responding Yes to every trial would yield 9/27 correct responses while responding No to every trial would result in 18/27 correct responses. So the expected correct when the subject's Y/N responding is 50/50 is more like 10.5 than 13.5 out of 27. I need to calculate the probability that a certain number total correct is below chance...ltuff1

Hi ltuff1.

I think the last line is the key to your question. I've marked it in bold. Can you give a concrete example of what you mean by "certain number total correct is below chance"? Any number of correct/incorrect responses is possible, although they will have a certain distribution based on the actual true answers and the strategy of guessing. I think I'm just missing a little bit of info in what you want to simulate. I follow your example of guessing all YES or all NO, just not the last line.

 

[ltuff1]

Hi ltuff1.

I think the last line is the key to your question. I've marked it in bold. Can you give a concrete example of what you mean by "certain number total correct is below chance"? Any number of correct/incorrect responses is possible, although they will have a certain distribution based on the actual true answers and the strategy of guessing. I think I'm just missing a little bit of info in what you want to simulate. I follow your example of guessing all YES or all NO, just not the last line.

Okay, so say a subject responds to the 27 trials of this yes/no task and gets 10 correct (there is a correct or incorrect response to each yes/no trial, 9 being in the yes direction, 18 in the no direction - so say he says 'yes' correctly 4 times and 'no' correctly 6 times, choosing the wrong response on the remaining 17 trials (saying 'no' 5 times when 'yes' was correct and 'yes' 12 times when 'no' was correct)). Using the binomial distribution on a similar task where there are equal numbers of yes/no items it's easy to calculate the cumulative probability of responding correctly on 10 or fewer trials (if the probability of success on a single trial out of 27 is 0.5 then the cumulative probability of getting 10 out of 27 or fewer correct by chance alone is 0.1239). If the subject is knowledgeable about the task he should score above chance (which would normally be the more interesting result) but in this case I'm interested in whether scores are below chance levels. Does that help? LT

 

[Jameson]

Okay, so say a subject responds to the 27 trials of this yes/no task and gets 10 correct (there is a correct or incorrect response to each yes/no trial, 9 being in the yes direction, 18 in the no direction - so say he says 'yes' correctly 4 times and 'no' correctly 6 times, choosing the wrong response on the remaining 17 trials (saying 'no' 5 times when 'yes' was correct and 'yes' 12 times when 'no' was correct)). Using the binomial distribution on a similar task where there are equal numbers of yes/no items it's easy to calculate the cumulative probability of responding correctly on 10 or fewer trials (if the probability of success on a single trial out of 27 is 0.5 then the cumulative probability of getting 10 out of 27 or fewer correct by chance alone is 0.1239). If the subject is knowledgeable about the task he should score above chance (which would normally be the more interesting result) but in this case I'm interested in whether scores are below chance levels. Does that help? LT

Yes I think so, but let's see.

In your example the order in which the subject answers correctly/incorrectly doesn't matter. It also sounds like we can assume that every question has the same probability of guessing correctly due to chance, that is 0.5. If the previous two assumptions are true then we actually do just have a simple binomial distribution.

We don't need to assume that there are equal number of yes/no answers for the binomial distribution to apply, we need to assume that each question has the same probability of being guessed correctly by chance. It could be 26 yes's and 1 no, or all no's, etc. The distribution of the outcome is irrelevant, only the probability of correctly guessing.

Does that makes sense? Do you agree?

 

[ltuff1]

Yes I think so, but let's see.

In your example the order in which the subject answers correctly/incorrectly doesn't matter. It also sounds like we can assume that every question has the same probability of guessing correctly due to chance, that is 0.5. If the previous two assumptions are true then we actually do just have a simple binomial distribution.

We don't need to assume that there are equal number of yes/no answers for the binomial distribution to apply, we need to assume that each question has the same probability of being guessed correctly by chance. It could be 26 yes's and 1 no, or all no's, etc. The distribution of the outcome is irrelevant, only the probability of correctly guessing.

Does that makes sense? Do you agree?

You're probably right. My first take on this was certainly that it's simply 27 independent trials with the same 50:50 chance on each one. It just seems off that the subject can say 'no' 27 times and then get 2/3 of them right. In practice as well the subjects tend to respond 'no' when unsure. I guess maybe that's really my issue - the true probability that a subject from my population will respond yes or no actually mirrors the correct answer ratio of 2:1 for no:yes answers...

 

[Jameson]

You're probably right. My first take on this was certainly that it's simply 27 independent trials with the same 50:50 chance on each one. It just seems off that the subject can say 'no' 27 times and then get 2/3 of them right. In practice as well the subjects tend to respond 'no' when unsure. I guess maybe that's really my issue - the true probability that a subject from my population will respond yes or no actually mirrors the correct answer ratio of 2:1 for no:yes answers...

I agree with you that if someone answers all of one value, then that's probably a sign that they are guessing. Another would be alternating every guess yes/no/yes/no,etc. Neither of these though matter under the assumptions I listed.

If you take a look at the part in bold, I want to make sure we're on the same page. We aren't calculating the probability that a subject will respond yes or no. We are calculating the probability of a success. That means yes when answer is yes and no when answer is no. The actual guesses are meaningless because we assumed a 50% success rate at the beginning. That assumption is independent of the distribution of outcomes and is the main driver of this calculation. In other words, it's a big assumption. Mathematically it makes sense that someone would always have a 50/50 chance of guessing correctly but in reality humans have psychological nuances that prevent us from acting randomly.

It sounds like this is a real-world question rather than a question from a book, so an idea to expand this topic might be to drop the assumption that there is always a 50% chance of guessing correctly. Maybe you are more interested in assuming a fixed number of yes/no correct answers and a fixed number of yes/no guesses? Maybe you are interested in looking at how certain strategies perform like all yes, all no, alternating yes/no?

 

[ltuff1]

I agree with you that if someone answers all of one value, then that's probably a sign that they are guessing. Another would be alternating every guess yes/no/yes/no,etc. Neither of these though matter under the assumptions I listed.

...It sounds like this is a real-world question rather than a question from a book, so an idea to expand this topic might be to drop the assumption that there is always a 50% chance of guessing correctly. Maybe you are more interested in assuming a fixed number of yes/no correct answers and a fixed number of yes/no guesses? Maybe you are interested in looking at how certain strategies perform like all yes, all no, alternating yes/no?

It is certainly a real-world question. What I'm really trying to calculate is the likelihood that a subject is performing statistically below chance where chance should be the worst you could do if you knew nothing about the correct answers. On a typical forced choice task like this where there are equal numbers of correct yes/no responses, performing significantly below chance (as per binomial calculation) means that the subject very likely actually had to have some knowledge of the correct response in order to do so badly so consistently. I have normative data for the task that indicates anything less than about 90% correct is a poor performance but there are all kinds of reasons for poor performance (including just not being very good at the task). If performance is significantly below chance, however, it suggests that poor performance was intentional. Being able to calculate the likelihood of poor performance not being a chance finding makes it easier to develop an hypothesis about why it happened for that subject (i.e. genuine poor performance versus intentional poor performance). It'd be sort of like trying to determine if somebody has a trick coin that lands on heads twice as often as tails. LT