Probability of success using two methods?

[lavoisier]

Hi everyone, a colleague at work asked me a question this morning, which I answered with the caveat that I wasn't really sure. I would like to hear your opinion, please.

His question was the following
I have two different methods A and B to obtain a given desired outcome, and it's known that the probability of success of A is 90%, and the one of B is 80%.
If I apply both methods, what is the probability that at least one of them succeeds?

My initial answer was:
P(A or B) = 1 - P(not(A) AND not(B)) = 1 - P(not(A))*P(not(B)) = 1 -0.1*0.2 = 98%

Then I thought, what if the events where A and B succeed (or fail) are not independent?
In that case I couldn't say: P(not(A) AND not(B)) = P(not(A))*P(not(B))
But instead:

P(A or B) = P(A) + P(B) - P(A AND B)

where:
P(A AND B) = P(A) * P(B|A) = P(B) * P(A|B)

I sent another email to my colleague with this observation. But I don't know if this is correct, either. And if it is, I don't know how we could determine the conditional probability, given that (he said) the two methods would be applied independently.

What do you think?
Have we got enough information to answer this question?
If not, what else do we need to know?

Thanks!
L

 

[PeroK]

Your method for independent events is correct: It's the complement of the case where they both fail.

if they are dependent, then you need to know a measure of their dependence. Worst case is that B always fails when A fails, which leaves you with 90%.

 

[lavoisier]

Thank you @PeroK !
What I am still having trouble understanding is how one can in general decide and/or determine when the two events are dependent.
Could you please make a 'real life' example where method B would always fail when A fails (keeping in mind that the two methods are applied in sequence but neither alters in any way the system they are applied to)?

I thought of an example, but I'm not sure it is applicable.
Imagine two wine tasters A and B that must determine if the wine you bought is really from France or not.
A will correctly tell French wine from non-French 90% of the time. B 80%.
Both A and B taste your wine, and you conclude that it's from France if at least one of them says it is.
What is the probability that you got the right conclusion?
Here I could imagine that the probability that B fails to tell correctly the provenance of the wine is not unrelated to the probability that A fails, because the process by which tasters make their judgment is probably similar.
But is this scenario related to the original problem?

 

[PeroK]

Imagine component A fails 10% of the time and B 20% of the time. Assume that a power cut causes a failure 5% of the time, and causes both A and B to fail. Otherwise, failures of A and B are independent.

Then the failures of A and B are dependent and you don't have the 98% you have for completely independent components.

Your example of wine tasting works as well, especially if you assume one is simply better than the other. If the good one is fooled, so is the other.

 

[FactChecker]

The statement that the two methods would be applied independently, is not exactly the same as saying that the results would be independent. You either need some convincing logic to say that a success of A in no way implies that B will be more (or less) likely to succeed, or you need to run some experiments to show that A and B are independent. If you run some experiments and the results are dependent, that will give you an estimate of the conditional probability that you need to calculate your answer. (It would also give you a direct estimate of P( A or B ).)

 

[lavoisier]

Thank you both for your replies!

OK, so if I understand correctly, unless it's logically quite improbable that the performances of A and B are dependent, only an experiment can tell if they are, and if so, to what extent.

Say A and B are two archers, A hitting the target on average 90% of the time, B 80%.
If both A and B shoot one arrow, the probability that either A or B hit the target is, I believe, 98%, as the success or failure of one probably doesn't affect the other.
To make sure it is so, we could have a table where the successes and failures of each archer in 100 trials are listed chronologically.

A B
S S
S F
S S
F S
...

How would you analyse these data to know if there is dependency? A non-parametric test perhaps? And if A and B don't interact at all between them, are in different places and don't even do this at the same time, does it matter how the data are sorted?

In the case of the two wine tasters, we could give each of them the same 100 wine samples to judge (that's a job many people would like, I guess), and here I would say there is a more definite link between the two, as each record corresponds to a specific wine sample.

W A B
1 S F
2 S S
3 F F
4 S S
...

As above, what kind of test would you run to know if the successes and failures of A and B are related?
Given the high % of successes of both A and B, a simple scatter plot would probably be dominated by the matching S S cases, wouldn't it?
Is there any test that sees through that and actually measures the dependency between the two?

And in this case, would it matter to include the type of wine? Because a 'success' is either 'correctly calling French an actually French wine' or 'correctly calling non-French an actually non-French wine'.
So in principle we could have for each taster a contingency table like:
A=F A=NF
W=F 40 7
W=NF 3 50

The 90% success rate of A would be (40+50)/(40+50+7+3), so the 'accuracy', but is that a sufficient metric, or does it matter to separate out the true positive rate (40/47) and false positive rate (3/53) of A, and same for B?

I'm mentioning contingency tables because of a related question that came up from another colleague in the past.
Here's the discussion that resulted, but the part about the dependency between the two tests was left a bit on the side:
https://www.physicsforums.com/threads/false-positive-rate-over-two-tests.850919/#post-5336880

That's the trouble I have with statistics and probability (which I like a lot, don't get me wrong, and I'm trying hard to get better at it) : every time I think I understood something, a new example comes up that makes me doubt everything I know, and I'm not sure what test or method to use. :O(

 

[lavoisier]

I asked our statistics expert at work.
She pointed out that the simplest approach, if validation data on the same set are available for both A and B, is to calculate the actual contingency table of the combined results (where the combination can be done as AND or OR, i.e. you decide that a wine is French if both A and B say it is, or if either of them says it is).

So for instance:
W,A,B,AandB,AorB
F,F,F,F=TP,F=TP
F,F,NF,NF=FN,F=TP
F,NF,NF,NF=FN,NF=FN
...
NF,NF,NF,NF=TN,NF=TN
NF,NF,F,NF=TN,F=FP
...

and see if the true positive rate and false positive rate are better than the ones when using one assay only.
Of course when A and B are exactly the same, the TPR and FPR don't change at all going from A or B alone to AandB or AorB.
Instead for instance any 'disagreement' between A and B on true positives and true negatives will decrease the TPR and increase the FPR of AandB.
I still need to figure out all the implications.
Very interesting, I think.

Oh, and BTW, the colleague who asked the initial question told me yesterday that in his case it was sure the two methods were independent.

 

[FactChecker]

if validation data on the same set are available for both A and B, is to calculate the actual contingency table of the combined results

Yes. I tried to make that point in the last statement of post #5.

Oh, and BTW, the colleague who asked the initial question told me yesterday that in his case it was sure the two methods were independent.

That sounds strange to me if they are both trying to achieve the same thing. It is saying that the odds of succeeding with A in no way changes the odds of succeeding with B. Are you sure that your colleague understands that we are not talking about the two methods being done independently, but rather what the results will be? As your statistics expert says, if you collect data that directly measures P( A of B ) then you will just get the estimate directly. Then you do not have to worry about whether or not they are independent.

 

[lavoisier]

OK thank you @FactChecker .

For your first point, I had understood the part 'you need to run some experiments to show that A and B are independent'. What I did not grasp immediately was how you would measure such (in)dependence and what effect it would have on the combined outcome. It just took someone to spell it out explicitly to me, as our expert did.
Having said that, I think the problem is a bit subtler than simple dependence/independence.
Two assays that give exactly the same results in all cases will be perfectly 'dependent', and running both will not offer any advantage on running one only.
However, when the two assays disagree, i.e. they are less 'dependent' or less 'correlated', looking at their combined result may either improve or worsen our prediction depending on what logic we use (AND or OR), on whether we use such logic to tell overall positives or negatives, etc.
I will think a bit more about it, maybe simulating some cases.

For your second point, we ended up talking about tests or assays that are applied to the same set of items, but in fact my colleague never said it was an assay, he said 'method'. When we talked again, he explained to me that he was referring to two complementary approaches to obtain a certain goal.
Example: starting from an object X that has some property you want to improve, you can modify X to Y (method A) or to Z (method B), and measure again the property. From historical data you know that 90% of the time the type of modification going from X to Y succeeds in improving the property, and 80% of the time for X to Z. If you try both modifications, the probability that the property has improved in either Y or Z is, if our reasoning is correct, 98%.
I don't know, to me this seems a case where there is independence. I may be wrong.

Of course knowing if the probability of success actually increases or not by making both Y and Z is no theoretical question, as there is a cost involved.
E.g. say that making Y costs 100; making Z costs 70; the reward of succeeding (with one object or two) is 1000. There is no penalty on failing.

So the expected value if you only make Y is: 0.9 * 1000 + 0.1 * 0 - 100 = 800
If you only make Z: 0.8 * 1000 + 0.2 * 0 - 70 = 730
If you make both: 0.98 * 1000 + 0.02 * 0 - 100 - 70 = 810

With these figures it looks like too much effort for little gain. To be fair, I'm being a bit stingy here, as succeeding with both Y and Z (rather than with only one) reduces a long-term risk (Y or Z may be found to be unsuitable for other reasons later on, so having both protects you from failure down the line), and that's a potential reward that would add to the figure 1000 that I mentioned.