Chance performance on a yes/no task where no is correct 2/3 of the time?

In summary: In your example the order in which the subject answers correctly/incorrectly doesn't matter. It also sounds like we're thinking about a different situation where some trials have a higher chance of being correct (e.g. a subject knows the answer to every trial). In that case, the expected number of correct responses is 13.5 out of 27, calculated as follows: 9 + 18 = 27 - 4 = 23.5.
  • #1
ltuff1
4
0
Title pretty much summarizes it. I know I can use the binomial distribution to calculate the chance of equal to or fewer than a certain number correct on a yes/no task where there's a 0.5 probability of being right on any trial. But what if you have 27 Yes/No trials where Yes is the correct answer on only 9 of the trials? Responding Yes to every trial would yield 9/27 correct responses while responding No to every trial would result in 18/27 correct responses. So the expected correct when the subject's Y/N responding is 50/50 is more like 10.5 than 13.5 out of 27. I need to calculate the probability that a certain number total correct is below chance...ltuff1
 
Last edited:
Mathematics news on Phys.org
  • #2
ltuff1 said:
Title pretty much summarizes it. I know I can use the binomial distribution to calculate the chance of equal to or fewer than a certain number correct on a yes/no task where there's a 0.5 probability of being right on any trial. But what if you have 27 Yes/No trials where Yes is the correct answer on only 9 of the trials? Responding Yes to every trial would yield 9/27 correct responses while responding No to every trial would result in 18/27 correct responses. So the expected correct when the subject's Y/N responding is 50/50 is more like 10.5 than 13.5 out of 27. I need to calculate the probability that a certain number total correct is below chance...ltuff1

Hi ltuff1.

I think the last line is the key to your question. I've marked it in bold. Can you give a concrete example of what you mean by "certain number total correct is below chance"? Any number of correct/incorrect responses is possible, although they will have a certain distribution based on the actual true answers and the strategy of guessing. I think I'm just missing a little bit of info in what you want to simulate. I follow your example of guessing all YES or all NO, just not the last line.
 
  • #3
Jameson said:
Hi ltuff1.

I think the last line is the key to your question. I've marked it in bold. Can you give a concrete example of what you mean by "certain number total correct is below chance"? Any number of correct/incorrect responses is possible, although they will have a certain distribution based on the actual true answers and the strategy of guessing. I think I'm just missing a little bit of info in what you want to simulate. I follow your example of guessing all YES or all NO, just not the last line.

Okay, so say a subject responds to the 27 trials of this yes/no task and gets 10 correct (there is a correct or incorrect response to each yes/no trial, 9 being in the yes direction, 18 in the no direction - so say he says 'yes' correctly 4 times and 'no' correctly 6 times, choosing the wrong response on the remaining 17 trials (saying 'no' 5 times when 'yes' was correct and 'yes' 12 times when 'no' was correct)). Using the binomial distribution on a similar task where there are equal numbers of yes/no items it's easy to calculate the cumulative probability of responding correctly on 10 or fewer trials (if the probability of success on a single trial out of 27 is 0.5 then the cumulative probability of getting 10 out of 27 or fewer correct by chance alone is 0.1239). If the subject is knowledgeable about the task he should score above chance (which would normally be the more interesting result) but in this case I'm interested in whether scores are below chance levels. Does that help? LT
 
  • #4
ltuff1 said:
Okay, so say a subject responds to the 27 trials of this yes/no task and gets 10 correct (there is a correct or incorrect response to each yes/no trial, 9 being in the yes direction, 18 in the no direction - so say he says 'yes' correctly 4 times and 'no' correctly 6 times, choosing the wrong response on the remaining 17 trials (saying 'no' 5 times when 'yes' was correct and 'yes' 12 times when 'no' was correct)). Using the binomial distribution on a similar task where there are equal numbers of yes/no items it's easy to calculate the cumulative probability of responding correctly on 10 or fewer trials (if the probability of success on a single trial out of 27 is 0.5 then the cumulative probability of getting 10 out of 27 or fewer correct by chance alone is 0.1239). If the subject is knowledgeable about the task he should score above chance (which would normally be the more interesting result) but in this case I'm interested in whether scores are below chance levels. Does that help? LT

Yes I think so, but let's see.

In your example the order in which the subject answers correctly/incorrectly doesn't matter. It also sounds like we can assume that every question has the same probability of guessing correctly due to chance, that is 0.5. If the previous two assumptions are true then we actually do just have a simple binomial distribution.

We don't need to assume that there are equal number of yes/no answers for the binomial distribution to apply, we need to assume that each question has the same probability of being guessed correctly by chance. It could be 26 yes's and 1 no, or all no's, etc. The distribution of the outcome is irrelevant, only the probability of correctly guessing.

Does that makes sense? Do you agree?
 
  • #5
Jameson said:
Yes I think so, but let's see.

In your example the order in which the subject answers correctly/incorrectly doesn't matter. It also sounds like we can assume that every question has the same probability of guessing correctly due to chance, that is 0.5. If the previous two assumptions are true then we actually do just have a simple binomial distribution.

We don't need to assume that there are equal number of yes/no answers for the binomial distribution to apply, we need to assume that each question has the same probability of being guessed correctly by chance. It could be 26 yes's and 1 no, or all no's, etc. The distribution of the outcome is irrelevant, only the probability of correctly guessing.

Does that makes sense? Do you agree?

You're probably right. My first take on this was certainly that it's simply 27 independent trials with the same 50:50 chance on each one. It just seems off that the subject can say 'no' 27 times and then get 2/3 of them right. In practice as well the subjects tend to respond 'no' when unsure. I guess maybe that's really my issue - the true probability that a subject from my population will respond yes or no actually mirrors the correct answer ratio of 2:1 for no:yes answers...
 
  • #6
ltuff1 said:
You're probably right. My first take on this was certainly that it's simply 27 independent trials with the same 50:50 chance on each one. It just seems off that the subject can say 'no' 27 times and then get 2/3 of them right. In practice as well the subjects tend to respond 'no' when unsure. I guess maybe that's really my issue - the true probability that a subject from my population will respond yes or no actually mirrors the correct answer ratio of 2:1 for no:yes answers...

I agree with you that if someone answers all of one value, then that's probably a sign that they are guessing. Another would be alternating every guess yes/no/yes/no,etc. Neither of these though matter under the assumptions I listed.

If you take a look at the part in bold, I want to make sure we're on the same page. We aren't calculating the probability that a subject will respond yes or no. We are calculating the probability of a success. That means yes when answer is yes and no when answer is no. The actual guesses are meaningless because we assumed a 50% success rate at the beginning. That assumption is independent of the distribution of outcomes and is the main driver of this calculation. In other words, it's a big assumption. Mathematically it makes sense that someone would always have a 50/50 chance of guessing correctly but in reality humans have psychological nuances that prevent us from acting randomly.

It sounds like this is a real-world question rather than a question from a book, so an idea to expand this topic might be to drop the assumption that there is always a 50% chance of guessing correctly. Maybe you are more interested in assuming a fixed number of yes/no correct answers and a fixed number of yes/no guesses? Maybe you are interested in looking at how certain strategies perform like all yes, all no, alternating yes/no?
 
  • #7
Jameson said:
I agree with you that if someone answers all of one value, then that's probably a sign that they are guessing. Another would be alternating every guess yes/no/yes/no,etc. Neither of these though matter under the assumptions I listed.

...It sounds like this is a real-world question rather than a question from a book, so an idea to expand this topic might be to drop the assumption that there is always a 50% chance of guessing correctly. Maybe you are more interested in assuming a fixed number of yes/no correct answers and a fixed number of yes/no guesses? Maybe you are interested in looking at how certain strategies perform like all yes, all no, alternating yes/no?

It is certainly a real-world question. What I'm really trying to calculate is the likelihood that a subject is performing statistically below chance where chance should be the worst you could do if you knew nothing about the correct answers. On a typical forced choice task like this where there are equal numbers of correct yes/no responses, performing significantly below chance (as per binomial calculation) means that the subject very likely actually had to have some knowledge of the correct response in order to do so badly so consistently. I have normative data for the task that indicates anything less than about 90% correct is a poor performance but there are all kinds of reasons for poor performance (including just not being very good at the task). If performance is significantly below chance, however, it suggests that poor performance was intentional. Being able to calculate the likelihood of poor performance not being a chance finding makes it easier to develop an hypothesis about why it happened for that subject (i.e. genuine poor performance versus intentional poor performance). It'd be sort of like trying to determine if somebody has a trick coin that lands on heads twice as often as tails. LT
 

Related to Chance performance on a yes/no task where no is correct 2/3 of the time?

1. What is "chance performance" on a yes/no task?

"Chance performance" refers to the expected success rate of randomly guessing on a task with two possible outcomes (yes or no). It is typically represented as a percentage and varies depending on the probability of the correct answer.

2. How is "no" being correct 2/3 of the time possible?

This is possible because the task has a higher probability of "no" being the correct answer compared to "yes". In other words, out of every three trials, two are expected to be "no" and one is expected to be "yes".

3. Can someone achieve higher than chance performance on this task?

Yes, it is possible for someone to achieve higher than chance performance by using strategies or knowledge that can increase their chances of selecting the correct answer. However, if the task truly has a probability of "no" being correct 2/3 of the time, then over a large number of trials, chance performance will still be the most likely outcome.

4. Is chance performance the same as random performance?

Not necessarily. While both involve randomly guessing, chance performance is specifically related to the expected success rate based on the probability of the correct answer. Random performance, on the other hand, can vary and may not align with the expected chance performance.

5. How does chance performance impact the validity of a study or experiment?

Chance performance can affect the validity of a study or experiment if the task being tested relies on participants' ability to make accurate guesses. In this case, the results may not accurately reflect the effects of the independent variable being studied. To minimize the impact of chance performance, researchers may use control groups or statistical tests to compare results to expected chance performance rates.

Similar threads

  • Calculus and Beyond Homework Help
Replies
12
Views
1K
  • Set Theory, Logic, Probability, Statistics
5
Replies
147
Views
7K
  • Set Theory, Logic, Probability, Statistics
Replies
11
Views
914
  • General Math
Replies
8
Views
4K
  • Precalculus Mathematics Homework Help
Replies
12
Views
3K
  • Classical Physics
Replies
6
Views
2K
  • Other Physics Topics
Replies
9
Views
1K
  • Calculus and Beyond Homework Help
Replies
5
Views
2K
  • General Math
Replies
22
Views
4K
Replies
1
Views
2K
Back
Top