Effectiveness of a Boolean test

[geo101]

I’m working with a Boolean parameter that has been proposed to identify good data and I want to test if it is effective on a data set where we know what is good (accurate) and bad (inaccurate results).

The theory (sensu amplo!) behind the Boolean parameter, called “SC”, is that a value of “1” should indicate accurate data and a value of “0” should indicate inaccurate data. I have a data set of 13,174 results, where we independently know that 5400 are accurate and 7774 are inaccurate.

Of the accurate results, 4016 (~74%) have an SC value of 1. Of the inaccurate results, 4373 (~56%) have an SC value of 1. What I would like to do is apply a statistical test to assess the significance of the difference between these two proportions and determine if, at some significance level, SC is effective at identifying accurate results.

Any thoughts and advice would be most welcome

 

[Office_Shredder]

I would be willing to eat a sock if they weren't different statistically - that should count as some sort of significance test rigt there.

More seriously you want to do this T-test:
http://en.wikipedia.org/wiki/Welch's_t_test

 

[geo101]

Thanks for the reply.

How would the t-test be adapted to this situation? Given that the SC parameter is essentially a yes/no result, would a test based on a binomial distribution not be more appropriate?

I would be willing to eat a sock if they weren't different statistically - that should count as some sort of significance test rigt there.

While that certainly would be significant, I would have to see evidence of the sock devouring :tongue:

 

[geo101]

OK, what about this...

I have 13,174 results, of which 5400 are accurate. The probability of randomly picking an accurate result is P = 0.4099.
Using the parameter SC = 1, I select a subset of 8389 results (i.e., N_t = 8389).
Of these N_t results 4016 are accurate (i.e., N_s = 4016).
Using the binomial CDF, I can calculate the probability that my realized success rate occurred by chance, using:

$p_r=1- \sum\limits_{i=0}^{N_{s}}{ N_{t} \choose i } P^{i}(1-P)^{N_{t}-i}$

When I crunch the numbers, I get p_r ≈ 0. So I can say that, at better than the 5% significance level, selecting results with SC = 1, will increasing the likelihood of selecting accurate results.

Is this correct??

 

[blue_raver22]

I would recommend conducting a statistical analysis to determine the effectiveness of the Boolean test in identifying accurate data. One approach could be to use a chi-square test to compare the observed proportions of accurate and inaccurate results with an expected proportion of 50% for both groups. This would allow us to determine if there is a significant difference between the two groups and if the Boolean test is effective at distinguishing accurate from inaccurate data.

Additionally, I would suggest considering other factors that may affect the accuracy of the Boolean test, such as sample size and potential biases in the data set. It may also be helpful to compare the results of the Boolean test with other methods of identifying accurate data, such as manual review or other statistical tests.

Overall, it is important to approach this analysis with a critical and objective mindset, considering all potential factors and limitations, in order to accurately assess the effectiveness of the Boolean test.