How can the probability of winning an individual match be estimated?

[taylrl3]

Basically I have a Master’s degree in Astrophysics but I haven’t done any mathematics for a few years now and am getting a little rusty. I thought a good way of dusting off some of my probability skills would be to do an exercise. Here is the problem:

Suppose we want to model a game of pong based on 10 matches. The game is played between two players and is scored as first to three wins. An example dataset for this might be as follows:

ABBAA
BAAA
AABBA
BBB
BABB
AAA
AABA
BAAA
AABBB
AAA

I would like to estimate the probability of player A winning a match and am thinking about the best way in which to do so.

1. The first way could be to calculate the proportion of points won by player A in that match and then take the mean of those values across the ten matches as the estimate of the probability (calling the estimator p’). When this is calculated, p’ = 0.61.

2. Another way to define p’ would be as the total number of points won by player A in all of the 10 matches, divided by the total number of points played, however this gives an estimate of p as 0.6.

I am thinking about whether either of these is unbiased and whether there is any alternative which might be better for estimating the probability of A winning as I would prefer the estimator to be unbiased ideally. Also, would there be any benefit to a biased estimator?

I believe the second method of calculating the probability is the unbiased one and the first has a bias of 0.01 as the bias is defined as being the difference between an estimator and the expected value.

It would be great to hear anyones thoughts on this little problem and ways of doing things. I am also a little under confident when it comes to writing down the mathematical definitions of this problem at this point so any pointers on those lines would be great too.

Thanks in advance!
Taylrl

 

[clamtrox]

You can't really estimate this without having a model of how the points are correlated.

Assuming that match scores are uncorrelated (so that result of a match is not affected by previous results), then you can just calculate the average of matches won, and also estimate the standard deviation which gives you some estimate of the probability.

You can go further too, if you for example assume that every win is uncorrelated as well. Then you would calculate first the probability of A winning a single round, and from that calculate the probability of winning a match.

 

[HallsofIvy]

There are three cases to consider:
1) A wins three games in a row: AAA
You don't say what is the probability A wins a single game so I will take that to be p= 0.5. The probability that A will win three games in a row is p^3= (0.5)^3= 0.125.

2) A wins three games while B wins one. BAAA, ABAA, AABA. "AAAB" does not count because A would already have won three games.
The probability of A winning three games and B one, in any particular order, is p^3(1- p)= (0.5)^4= 0.0625. But there are 3 orders so the probability of A winning three games while B wins one is 3(0.0625)= 0.1875.

3) A wins three games while B wins two. I won't list all the possibilities. There are $\begin{pmatrix}5 \\ 3\end{pmatrix}= 10$ ways of listing 3 "A"s and 2 "B"s but some of those have three "A"s before the last B and would not count. There are $\begin{pmatrix}4 \\ 1\end{pmatrix}= 4$ ways to order three "A"s and one "B" (the four ways shown in (2) above) leaving a "B" for the last so the number of ways A can win three games and B two is 10- 4= 6. The probability of A winning three games and B two in anyone order is $p^3(1- p)^2= 0.5^5= 0.03125$. Multiplying that by 6 gives 0.1875.

The probability of A winning is 0.125+ 0.1875+ 0.1875= 0.5.

Well, that's a kick in the head isn't it! It's obviously true- given that A and B are equally likely to win anyone game, then they are equally likely to win the tournament. More generally, if A has probability p of winning a single game (so B has probability 1- p) then the probability A will win the tournament by winning three games before B does is $p^3+ 3p^3(1- p)+ 6p^3(1- p)^2= 10p^3- 15p^4+ 6p^5$.

 

[taylrl3]

Massive thank yous for your responses.

Firstly clamtrox:

You can't really estimate this without having a model of how the points are correlated.

I forgot to also mention that each point is independent of every other point.

Assuming that match scores are uncorrelated (so that result of a match is not affected by previous results), then you can just calculate the average of matches won, and also estimate the standard deviation which gives you some estimate of the probability.

Would this mean that the second way that I suggested that gives p=0.6 as I thought is the correct way (by correct I mean unbiased)? If so, I am wondering why it is any less effective as an estimator than the other way that I suggested? It would be great if I could understand mathematically where the bias of 0.01 comes from.

You can go further too, if you for example assume that every win is uncorrelated as well. Then you would calculate first the probability of A winning a single round, and from that calculate the probability of winning a match.

How would I go about doing this? I think this is what I have done in the first way I mentioned, but then how would I go on to calculate the probability of winning a round from this average of winning individual game?

HallsofIvy:

I fear I haven't been explicit enough in my explanation of the example and so apologies. What I am trying to do is look at different ways of calculating p, based on the data provided

Well, that's a kick in the head isn't it! It's obviously true- given that A and B are equally likely to win anyone game, then they are equally likely to win the tournament. More generally, if A has probability p of winning a single game (so B has probability 1- p) then the probability A will win the tournament by winning three games before B does is p3+3p3(1−p)+6p3(1−p)2=10p3−15p4+6p5.

That will be my fault for not explaining properly. I am interested in the more general situation you have given a formula for in the last sentence I think this is the formula I have been looking for but then (and maybe more interestingly) why is this any better than just taking the average of all the games as an estimator?

Massive thanks again!

 

[clamtrox]

Massive thank yous for your responses.
Would this mean that the second way that I suggested that gives p=0.6 as I thought is the correct way (by correct I mean unbiased)? If so, I am wondering why it is any less effective as an estimator than the other way that I suggested? It would be great if I could understand mathematically where the bias of 0.01 comes from.

It's correct. But it's not the best way to estimate the probability, as you only have 10 data points. If you are able to use the individual rounds, you have much more data, and therefore you can determine the probability more accurately.

How would I go about doing this? I think this is what I have done in the first way I mentioned, but then how would I go on to calculate the probability of winning a round from this average of winning individual game?

This is what HallsofIvy was showing you.

 

[taylrl3]

Thanks HallsofIvy. I have been through your explanation and it is a very nice and clear explanation of how to calculate the probability of winning a match, given the probability of winning a point.

Given the formula you gave at the end of your post, I have taken p = 0.6, calculated from the average number of points won from all the points in the example data I gave above (p = 24/40) and then plugged that into your formula to get an overall probability of winning a match at p = 0.68256.

What I would really like to be focusing on is how to estimate the probability of winning any particular point and the effectiveness of different methods. If p is the probability of A winning and (1-p) is the probability of B winning (as HallsofIvy mentioned before). The two ways I currently have of doing I so I mentioned before as;

1. The first way could be to calculate the proportion of points won by player A in that match and then take the mean of those values across the ten matches as the estimate of the probability (calling the estimator p’). When this is calculated, p’ = 0.61.

2. Another way to define p’ would be as the total number of points won by player A in all of the 10 matches, divided by the total number of points played, however this gives an estimate of p as 0.6.

Given these two methods, I am wondering;

1. Could anyone help me to mathematically define p and p'?
2. Which of these give the true value of the parameter being estimated and consequently are either biased? (We say that p’ is an unbiased estimator of p if and only if, for every value of p with 0<=p<=1, the expected value of p’ is p.)
3. What alternatives are there to these methods and why would one be preferable to any other?
4. Could a biased estimator be better than an unbiased one?

 

[haruspex]

What I would really like to be focusing on is how to estimate the probability of winning any particular point

Since each point is taken to be independent of other points, what games the points were scored in becomes irrelevant. Use method (1): take the proportion that A won of all points.
That is certainly the best way to estimate the probability that A wins any given point. But it does not necessarily lead to the best way to estimate the probability of winning a match for the purpose to which it will be put. Ideally, you should look at how you will use the answer and the costs of getting it wrong by how much in which direction.

 

[taylrl3]

Ok, I am going to use method (1) as the proportion of all points won. I was also a little confused initially and I only want to calculate the probability of A winning a point and not a match anyway so that is fine.

I would like to attempt to define p and p' mathematically but at the moment I am confused by the definitions I have been given. These are the exact wordings of the definitions form the book:

1. Someone suggests that for each match we calculate the proportion of points won by player A in that match, and then we take the mean of these values across the ten matches as our estimate of p (let’s call our estimator p’). With the example data set, the proportion of points won by player A in the first match is 0.6, in the second is 0.75, in the third is 0.6 etc., and p’ would then be calculated as the mean of these ten values, in this case 0.61.

2. Someone else suggests defining p’ as the total number of points won by player A in the ten matches divided by the total number of points played in the ten matches. With the example data set, this would give a result of 24/40 = 0.6 as our estimate of p.

What is confusing me is that in the first half of the first definition p is defined as the mean of the means. My attempt at writing this mathematically would be p = ∑(∑x/n)/m. What then is p'? I can't seem to see the relation between p and p'. It gets even more confusing in the latter part of the definition when it goes on to say that using that definition then p' would then be calculated as the mean of the means and give a value of p' = 0.61. It seems to me that the two definitions are somewhat jumbled up.

The same problem is repeated in the second definition where it defines p' as p'=∑x/n however then goes onto say that p = 24/40 = 0.6. So is it p' or p that is 0.6?

I am either being quite stupid or there is some kind of trick here. Is there anyone out there with experience of using these things who knows how p and p' should be defined?
 

 

[clamtrox]

I would like to estimate the probability of player A winning a match and am thinking about the best way in which to do so.

1. The first way could be to calculate the proportion of points won by player A in that match and then take the mean of those values across the ten matches as the estimate of the probability (calling the estimator p’). When this is calculated, p’ = 0.61.

2. Another way to define p’ would be as the total number of points won by player A in all of the 10 matches, divided by the total number of points played, however this gives an estimate of p as 0.6.

Method 1 gives equal weight to every game, whereas method 2 gives equal weight to every point. I don't see why you would want to give equal weight to every game, ie. giving more weight to points scored in fast games (with only 3 or 4 points).

 

[taylrl3]

I agree entirely. I am set on using the mean across all of the points as the best estimator.

Could you look back at my last post and possibly help me with defining p and p' mathematically? I have given the definitions in the book but they seem to be conflicting.

 

[clamtrox]

Can you explain your notation a little more? What are p and p'?

 

[taylrl3]

p is the probability of player A winning any particular point (so that the probability of player B winning any point is 1-p). We don’t know the value of p, but we would like to estimate it based on the results of the ten matches that these players have contested.

1. Someone suggests that for each match we calculate the proportion of points won by player A in that match, and then we take the mean of these values across the ten matches as our estimate of p (let’s call our estimator p’). With the example data set, the proportion of points won by player A in the first match is 0.6, in the second is 0.75, in the third is 0.6 etc., and p’ would then be calculated as the mean of these ten values, in this case 0.61.

2. Someone else suggests defining p’ as the total number of points won by player A in the ten matches divided by the total number of points played in the ten matches. With the example data set, this would give a result of 24/40 = 0.6 as our estimate of p.

 

[clamtrox]

1. Let i label the match, and j label the game inside a match, and i=1...N and j=1...k_i where k_i is the number of games in given match. Also let d_ij be the result of the game j in match i (1 for a win and 0 for a loss). Then $$p = \frac{1}{N} \sum_{i=1}^{N} \frac{1}{k_i} \sum_{j=1}^{k_i} d_{ij}$$ And like I said before, this scheme makes no sense to me.

2. $$M = \sum_i {k_i}$$ $$p = \frac{1}{M} \sum_{i=1}^{N} \sum_{j=1}^{k_i} d_{ij}$$ or alternatively written
$$p = \frac{1}{M} \sum_{i=1}^M d_i$$

 

[taylrl3]

That is beautifully put :-)

I am now thinking about alternative methods of estimation. Am I right however in thinking that anything that produces a probability differing from the minimal error estimator in method 2 (p=0.6) is biased and therefore less satisfactory?

Also, as this is a small sample size I have read that Bayesian Inference would be a good idea. I am trying to get a quick understanding of it (I fear this might be impossible) though if anyone has any guidance on how I would do a Bayesian Analysis on this data, then that would be really interesting.

 

[haruspex]

Since each point is taken to be independent of other points, what games the points were scored in becomes irrelevant. Use method (1): take the proportion that A won of all points.

Sorry - I meant method 2.

 

[taylrl3]

Thanks haruspex.

I am wondering what other methods there are of estimating the probability of A winning a match. I have been looking at Bayesian methods (though I am not sure this would be appropriate) and I believe the first step is to find a prior. For this I was considering using a distribution based on the mean taken using the average of all of the values, combines with the standard deviation, though I think this might be the posterior distribution. A little confused :-S

Does anyone know how I might do this or if there are any other good ways of estimating a value like this? (ideally unbiased).

 

[Office_Shredder]

Saying that player A wins 60% of the points doesn't tell you a lot by itself. He could literally win 3 points, then B wins 2 points, then A wins 3 points, then B wins 2 points etc. and A ends up winning 100% of the games. You have to use the independence of the points, and it requires a bit of math to find out how exactly the 60% translates into the probability of winning a series.

From the probability of winning a match, find the probability of winning a series.
From the probability of winning a point, find the probability of winning a match. This requires knowing the rules of pong, which I do not. Is it time based or a "first to X wins" game? If it's the latter then the same math to solve the 'match probability -> series probability' will work for this problem (and in both cases the answer is easy). If it's time based then the problem is harder, but not too much so, you just need to include the distribution of how fast each person scores points (and you will end up using some Poisson distributions).

 

[haruspex]

I have been looking at Bayesian methods (though I am not sure this would be appropriate) and I believe the first step is to find a prior. For this I was considering using a distribution based on the mean taken using the average of all of the values, combines with the standard deviation, though I think this might be the posterior distribution.

No, a prior is a prior - i.e. it's what you thought before you looked at the data.
E.g. suppose a point consists of throwing one dart each at a target, the nearer to the bull winning. You might suppose that (a) a given individual's throws form a 2D normal distribution centred on the bull, with the variance representing the skill level (inverted); and (b) in the population as a whole there is a normal distribution of skill level. Now, given two individuals, without knowing their skill levels, you could in principle compute a prior distribution for the probability p that A scores the next point. Now you let them play and collect the data. Using that data you can adjust the distribution of p according to Bayes' Theorem.
That's a lot of work, which is why, mostly, people just take a uniform prior distribution and hope they collect enough data that the chosen prior ceases to matter.

 

[taylrl3]

Ok. Given that a Bayesian approach would not be appropriate, might there be any other estimators I might use to calculate the probability of winning an individual point? I am ideally looking at unbiased estimators.

From the probability of winning a match, find the probability of winning a series.
From the probability of winning a point, find the probability of winning a match. This requires knowing the rules of pong, which I do not. Is it time based or a "first to X wins" game? If it's the latter then the same math to solve the 'match probability -> series probability' will work for this problem (and in both cases the answer is easy). If it's time based then the problem is harder, but not too much so, you just need to include the distribution of how fast each person scores points (and you will end up using some Poisson distributions).

There aren't really any series involved. Earlier in the thread there was an explanation of how to calculate the probability of winning a match from the probability of winning a point. I am only really interested in how to estimate the probability of winning a point however. Do you know any other good methods of estimating the probability of winning an individual match?

Saying that player A wins 60% of the points doesn't tell you a lot by itself.

I agree and am therefore looking for something a bit deeper.

 

[haruspex]

I am only really interested in how to estimate the probability of winning a point however. Do you know any other good methods of estimating the probability of winning an individual match?

If that statement is true, why that question?
Without the priors approach discussed, and assuming all points are independent, the fraction of points actually won is clearly the best estimator for the probability of winning a point in general.
A more refined study might look at how a given player responds to score within the game - does the player do better or worse when under pressure?
If you do care about calculating the probability of winning a match (and assuming all points are independent), basing it on the fraction of points won (and using formulas mentioned) is still better than basing it on fraction of games won. That's because only looking at games won throws away information. Ten games won by player A by something to nil will give a very different estimate than if the games were close.
If you want to take into account response to pressure then you need to look at the probability of winning a point as a function of the state of the game.