Expectation of how many winners

[Ken G]

Imagine a million different names are in a hat, yours among them. Some number N of names will be drawn, decided by people that you know too little about to decide a meaningful expectation on N. The drawing is done in secret, and the newspaper reports one winner each day, in no particular order. On the first day, the newspaper reports that you are a winner! Does this give you reason to suspect that (1) N should have been fairly large, (2) you still don't know how large to expect N to be, but your expectation is larger than before you knew you won, or (3) you have no reason to update your expectation at all.

 

[disregardthat]

So, for example, if N = 1000 (a value which is unknown to you), a thousand names are drawn randomly. Then the newspaper will report one randomly chosen name of these 1000 per day?

I don't really understand. Say if they drew all the names, and your name was announced first. Or if they drew one name, and you came up. The probability is exactly 1 in a million that you would be announced a winner the first day.

 

[jz92wjaz]

If 1000 names are picked, it sounds like they would report one winner a day for the next 1000 days.

 

[Ken G]

Yes, that's right, it would be 1 each day for 1000 days, chosen in a random order, and you came up on day 1.

 

[mfb]

I don't see the point. The first day, exactly one name will get published, and every name has the same probability. This is completely independent of N (apart from the trivial bound N!=0) or even the drawing method in total (assuming every step is fair).

 

[Ken G]

Your analysis is quite correct. Some might think that for them to win, it must mean that others will win also, even if they have no evidence yet that anyone else has won. If I said that the newspaper has notified all N winners in secret, and you are among them but not necessarily the first chosen to be notified, would you then have reason to think N is fairly large?

 

[mfb]

As you did not get noticed in secret, nothing changes.

P(your are in the N winners)*P(you are the one that gets announced publicly) is still 1/(1million) for all N.

 

[Ken G]

I'm not sure I understand, it seems to me your equation there is the correct one if a single winner is chosen at random from among the N, and is announced publicly. However, it seems to come in answer to the version where all the winners are notified in secret. Perhaps I should mention that you are also told that all the winners are told that they are being notified in secret, and they are absolutely not allowed to share this information.

 

[mfb]

Ah sorry, I was still assuming we were announced in the newspaper.
If that is not the case (and we get one of the secret notes), then sure, we will update for a larger N.

 

[alan2]

If I understand the question then no, being announced on the first day tells you nothing about how many were chosen as winners. This is what I understand: You have a population of M and choose m winners. Then you are the first winner announced. Does this tell you anything about how many winners, m, were drawn? If that's the question then the answer is no.

Sorry, I don't know how to use the equation editor but the number of ways to choose m from M is C(M,m). The number of ways that you can be chosen as one of the m is C(M-1,m-1). So the probability that you were chosen is C(M-1,m-1)/C(M,m) = m/M. The probability that you were the first announced, given that you were one of the m chosen is 1/m. So the probability of you being the first announced is (1/m)(m/M) = 1/M which is just the probability that you were the first person chosen out of M. Makes sense if you think about it.

 

[Ken G]

Right, those answers are all entirely correct. Interestingly, if after you are notified in secret (which increased your expectation on m, or N as I called it, since we agree that your expectation on m does not go up if you are the first notified or if you are notified at random, but it does go up if you know that all winners have been notified in secret and you are among them), the announcement is made that you have been randomly chosen among the winners to have your name released to the public, then you would have to return your expectation of m (or N) back to where it started, because now you are back to the original situation of being the randomly chosen winner for the first announcement.

So hard as I try, I cannot think of a single way that your personal status as a winner could ever cause you to have a higher expectation on m than anyone else, unless you were privy to secret knowledge. That makes sense, if all information is public, then all the people should have the same expectation on m, regardless of whether or not they are personally winners.

The reason this pretty logical situation might seem counterintuitive is that we tend to want to interpret our experiences as "generic", so we would look for interpretations of the available data such that we are one of many. So if we are told in secret we won, we figure there must have been many winners, and that might be reasonable logic. But if something later happens, such as our name is chosen to be released, that forces us to confront the fact that something very special happened to us, it can force us to retroactively let go of the idea that our experience was generic. We end up with no reason to think there were many winners just because we won, if something later happens that shares all the information we have with everyone else, and makes us seem not so "generic" after all, even after the fact.

An application of this could be to looking for successful treatments of diseases. If you receive an experimental treatment and get a good result, you tend to increase the likelihood that the treatment is good. But if something happens that picks you out, like you are approached as the first person they are asking to give a testimonial as to the success of the treatment, it should give reason to doubt the treatment is effective after all!

 

[alan2]

Maybe I missed something here. Were you asking a question or giving a quiz Ken?

 

[Ken G]

The question I asked was too easy, people saw right through it to the correct answer. So I was then justifying the purpose of the question, which was to try and find a situation where we would have an intuition that somehow our own personal experience could be regarded as generic, even when we had evidence that it was not. So although the puzzles are too easy for this forum, the lesson might still be of interest-- that we are justified in treating our own experiences as somewhat "normal," until such a time that we have evidence they were not, which is pretty much anything that picks us out in a special way. In a sense, everything that happens to us in a day is remarkably unlikely to happen, but it doesn't count as unlikely until something about it can be picked out as special in some way. So what counts as special? When can we say we are having normal experiences, and what has to happen before we recognize that something remarkable and unusual has occurred?

 

[disregardthat]

The question I asked was too easy, people saw right through it to the correct answer. So I was then justifying the purpose of the question, which was to try and find a situation where we would have an intuition that somehow our own personal experience could be regarded as generic, even when we had evidence that it was not. So although the puzzles are too easy for this forum, the lesson might still be of interest-- that we are justified in treating our own experiences as somewhat "normal," until such a time that we have evidence they were not, which is pretty much anything that picks us out in a special way. In a sense, everything that happens to us in a day is remarkably unlikely to happen, but it doesn't count as unlikely until something about it can be picked out as special in some way. So what counts as special? When can we say we are having normal experiences, and what has to happen before we recognize that something remarkable and unusual has occurred?

Well, I personally still don't understand the exact situation you had in mind. What was published in a newspaper, what was told in secret, in short, who were supposed to know what at what time? And what was done randomly, what was done deliberately? And did you change the premise after a while?

 

[Ken G]

Yes, several premises were considered. In the original, the player is told they are the first winner (which has the same effect as if they are told they were selected at random from among the winners to be the one person told). Someone with less probability smarts than those who answered might think "if I'm a winner, others must be also", but of course that's not true if you are a special winner-- your specialness is not mitigated by having other winners as well. So then I looked at what would be needed to truly mitigate your specialness, such as if all winners were notified in secret, which would restore a sense of "genericness" to the individual winners, and would lead to an increased sense of how many winners there were, though perhaps not as much of an increase as one might think. So yes, there was some moving goalposts there, but the point was to figure out what constitutes a situation where the winner just has to conclude they witnessed something extremely unlikely, versus what could let them continue to regard themselves as "generic." What I noticed is that the winner was indeed special whenever everyone else regarded them as special-- it never mattered what the winner thinks about themself, there is always a kind of bias in their own perspective that is best removed by taking an outside onlooker's perspective. That principle would also apply to things like judging the effectiveness of medical treatments, the objectivity that science so values seems like the key device in avoiding errors in probability expectations.

 

[jz92wjaz]

Some could also mistakenly lower their expectation of N, by assuming that N is likely to be small because they know they won and that there is a 1/N chance of being announced on the first day if you are a winner.

 

[Ken G]

Yes that's a good point, if they recognize it is unlikely that they'd be announced if N is large, and forget they are more likely to win in the first place if N is large, they might think N has to be small. So the way it could actually go down is, you are told in secret that you are among the winners, and no winners are allowed to announce this. Then your expectation on N has gone up as a result (though just how much depends on that it was before, so if you have no expectation before, you really don't know how to use this information anyway). But then you see a public notification that you are the first winner that is being publicly announced, selected at random for the honor, and you must then do just the 1/N reasoning you just mentioned-- your expectation on N must then return to whatever it was before you found out you won.

The key lesson in all this, for me, is that to have a right to reach a different expectation on N than someone else requires that you be privy to secret information. As soon as all the information you have is relegated to the status of public information, as is true after the above public announcement, then you must always reach the same conclusion as all those other people who just see you as a random name. There is never a "reasoning from the point of view of being special because you are you", there is only access to secret information not generally available, or no such access.

 

[caveman1917]

The answer to your question in the OP is "no". The answer to your changed question is "yes". The difference has nothing to do with secrecy but everything to do with the difference between being the first winner (of which only one is chosen independently of N) and being a winner (of which the number chosen depends on N), the latter thus giving information on N but not the former.

For a detailed explanation please refer to your previous almost-identical http://cosmoquest.org/forum/showthread.php?152938-A-probability-puzzle-most-people-will-get-wrong. Do you have any questions about the references, detailed calculations or explanations provided there?

 

[Ken G]

Yet of course that does have everything to do with secrecy, just think about it. Ask yourself this: in the first case, when you answered "no", could that information be public? If you haven't seen this yet, imagine the following scenario. caveman1917 is alerted in private that he is among the winners, so he increases his expectation on N (by an amount that actually depends on what your original expectation was), which is also why the correct answer to that question is indeed "yes", as we all know. Obviously, this is secret information, or the answer will change (if you are allowed to tell everyone, and yet you hear nothing from anyone else, what does that tell you?). So yes, it really has to be secret to increase your expectation on N. However, if the next day there is a newspaper headline that announces to all "caveman1917 is among the winners", then your information is no longer secret. Guess what happens to your expectation on N?

As has been pointed out above, it's not so clear what happens to your expectation on N, it depends on why the newspaper announced you. But for whatever you take as the reason the newspaper chose you (maybe you are the brother of the editor, or maybe you were randomly chosen to be announced first, these all make a difference), the key point is, it will now be common knowledge-- all your secret information is now gone. So what you can be sure of, even though you seem to deny it, is that your expectation on N must be the same as everyone else's, even people who are not even involved in the drawing at all. So I repeat: there is never any situation where "you being you" allows you to reach any different conclusions, there is only secret information, and public information, and that's all that matters for anyone's expectations on N.

 

[caveman1917]

caveman1917 is alerted in private that he is among the winners, so he increases his expectation on N (by an amount that actually depends on what your original expectation was), which is also why the correct answer to that question is indeed "yes", as we all know.

It's good to see that you've finally changed your mind on that. The main problem remains that you are attempting to understand probability theory by vague appeals to intuition. This does not work and is counterproductive. It does not work in mathematics in general, and it certainly does not work in probability theory. What is needed to understand this field is precision and mathematical analysis. In as much as any intuition is going to work at all here it should be intuition gained from experience of working through the math.

Obviously, this is secret information, or the answer will change (if you are allowed to tell everyone, and yet you hear nothing from anyone else, what does that tell you?). So yes, it really has to be secret to increase your expectation on N. However, if the next day there is a newspaper headline that announces to all "caveman1917 is among the winners", then your information is no longer secret. Guess what happens to your expectation on N?

Again, no, it has nothing to do with secret or public information. This can be easily seen by considering this step by step. I am notified I am a winner, I update for larger N. Suppose I now post on twitter "I have won!". The information isn't secret anymore, but does this mean that I should now update for lower N, just because I've made the information public? Of course the answer is no.

What you seem to be thinking of is the situation where everyone has made public their status of winner/non-winner. But that isn't an issue of probability, it's an issue of simply counting the number of winners.

So I repeat: there is never any situation where "you being you" allows you to reach any different conclusions, there is only secret information, and public information, and that's all that matters for anyone's expectations on N.

Repetition does not make it correct. Is such a statement made by the textbook you are using (which one are you using?), or do you have some other reference?

 

[Ken G]

It's good to see that you've finally changed your mind on that.

You apparently misread me, either now or then. I have not changed my mind at all, but you typically misunderstand me, so I would assume that is the problem here as well.

The main problem remains that you are attempting to understand probability theory by vague appeals to intuition.

Again, you merely reach false conclusions based in incorrect input data. The fact is, you still don't know what I'm saying, and I don't think I can explain it better than the crystal clear explanations that exist above. What is written above in this thread completely describes the situation here, and what I conclude from it has been fully borne out by those analyses: expectations on N are based on information, and that information can either be public or private, and it makes a big difference which. But at no point does it ever count as special or private information that "I am me, so my own experience carries special meaning for me." There is never a case where the expectation on N is any different for someone playing the game, as for someone not playing the game, unless the player has secret information that could be made public, and if it were made public, the same expectation for all would be recovered once again. Hence, there is never any situation where a player should feel frustrated that they cannot get anyone else to appreciate how special or important it is that they personally won. If they sense there is indeed a frustration there, along the lines of "I know a truth but I can't get anyone else to believe me," then it means they have entered into bad logic. You can find the applications of this conclusions anywhere you like.

 

[caveman1917]

You apparently misread me, either now or then. I have not changed my mind at all, but you typically misunderstand me, so I would assume that is the problem here as well.

Your BAUT thread on this asks the question where you are "notified as being a winner". There you said the answer is "no update on larger N", here you say the answer is "yes an update on larger N". If i misunderstand, then perhaps you could try to explain?

Again, you merely reach false conclusions based in incorrect input data. The fact is, you still don't know what I'm saying, and I don't think I can explain it better than the crystal clear explanations that exist above.

As i said, mathematics requires precise analysis. Perhaps if you were to provide such analysis it would make things even more crystal clear? In as much as your explanations above are crystal clear they are wrong, and in as much as they are not wrong they are too vague to make any determination.

What is written above in this thread completely describes the situation here, and what I conclude from it has been fully borne out by those analyses: expectations on N are based on information, and that information can either be public or private, and it makes a big difference which.

It makes no difference whatsoever. The only things that make a difference are the person's prior probability, conditional probabilities and information set. Whether the information in that set is "private" or "public" has no bearing on the result.

But at no point does it ever count as special or private information that "I am me, so my own experience carries special meaning for me." There is never a case where the expectation on N is any different for someone playing the game, as for someone not playing the game, unless the player has secret information that could be made public, and if it were made public, the same expectation for all would be recovered once again. Hence, there is never any situation where a player should feel frustrated that they cannot get anyone else to appreciate how special or important it is that they personally won. If they sense there is indeed a frustration there, along the lines of "I know a truth but I can't get anyone else to believe me," then it means they have entered into bad logic.

But your conclusion isn't just about the specific example of the game, you're making it a general conclusion. What about all the games that can be presented in which your conclusion is false?

You can find the applications of this conclusions anywhere you like.

Yet it would still be rather helpful if you could point to a textbook or other reference where you have found such conclusion being presented. You must have gotten it from somewhere, so would you mind sharing that?

 

[caveman1917]

Perhaps i should refer to the previous question as on BAUT:

Imagine you are 1 of a million people who put their names in a hat. A number N of names is chosen from the hat, but you have no idea what N is, except that N > 0. You are informed that your name has been selected! Is it true that you can conclude with good probability reasoning that, most likely, N is fairly large?

(where later in the thread you specified the question as really asking about whether you update for larger N than you started with)

So you say that the answer to that question is no? And the answer to the question here with the change that you are simply notified of being a winner (rather than the first to be announced) is yes?

 

[Ken G]

Your BAUT thread on this asks the question where you are "notified as being a winner". There you said the answer is "no update on larger N", here you say the answer is "yes an update on larger N". If i misunderstand, then perhaps you could try to explain?

No one here cares about a thread on some other forum. If they are wondering, the simple answer to your question is that what was resolved in that thread is that any extent to which you increase your expectation on N depends on what your expectation was going in, so if you had no expectation going in, there is nothing to update and no change in an expectation that doesn't exist. So it all depends on one's interpretation of what it means to have "no idea what N is". For some, that means you have no way to generate a meaningful expectation on N, a point you still seem to struggle with. For others, it means you must assume a flat distribution over all the possible N. There is no correct answer to that, you simply get out what you put in, but that's the source of your struggle.

What was also decided in that thread is that all that is rather trivial once it is understood, yet what is interesting is to notice how extremely important are the different unstated assumptions that people make, those are what actually determine the answer. Also, there is the conclusion I mentioned as actually relevant to this thread: that expectations change when secret information becomes public information, which can alter the interpretation of the secret information such that all parties reach the same conclusions about N. Nothing else that you said seems relevant, and does not require comment. This forum has little patient for endless arguments, so this will be my last post on the subject.

 

[caveman1917]

For some, that means you have no way to generate a meaningful expectation on N, a point you still seem to struggle with.

But in your OP here you literally state that you have "no meaningful expectation on N", yet when you are notified in secret your expectation goes up. How do you explain that?

For others, it means you must assume a flat distribution over all the possible N.

Are you claiming that the expected value of N when using a flat distribution is "meaningful"? By what definition of "meaningful" would that be true?

Also, there is the conclusion I mentioned as actually relevant to this thread: that expectations change when secret information becomes public information

Then how do you explain that it doesn't change when I would make the secret information that I am a winner public?

This forum has little patient for endless arguments

I would hope that to be true.

so this will be my last post on the subject.

But you've yet to give us a reference such that we can learn more about your "new probability theory". You've been going around several forums expounding your notions in that area, how are all the mathematicians educated using the "old probability theory" to re-educate themselves if no learning materials are presented?

 

[mfb]

But in your OP here you literally state that you have "no meaningful expectation on N", yet when you are notified in secret your expectation goes up. How do you explain that?

It is no problem to discuss update factors even if we did not decide on an initial probability distribution.

Then how do you explain that it doesn't change when I would make the secret information that I am a winner public?

It would change, if you assume that all winners would post that on twitter in the same way, and if you are the first one to post it.

I would hope that to be true.

It is true.

 

[caveman1917]

It is no problem to discuss update factors even if we did not decide on an initial probability distribution.

Why would it be, and how is that relevant to the claims being made? Do you agree with Ken that "the expectation doesn't go up because I can not choose a meaningful prior probability and therefor there is nothing to update" is a correct answer?

It would change, if you assume that all winners would post that on twitter in the same way, and if you are the first one to post it.

Which again has nothing to do with the information being "private" or "public", but everything to do with your information set containing either "I am a winner" or "I am the first winner". You can easily see this by slightly changing your example, where instead of posting it on twitter everybody is supposed to post it on a physical messageboard in some building. Suppose you, as a winner, go to that building and when you want to enter the security guard privately whispers in your ear: "You are the first one to come post it". Would you now not update on that information because it isn't public yet? Would you wait updating until you actually get to the messageboard and have posted your result publicly on it, even if you have no reason to think that the security guard is lying?

It makes no difference whatsoever whether the information that you have is private or public (it's not even clear how you would define those concepts anyway), the only reason you're getting that result is because your examples are stated in such a way that the information of "being a winner" is given privately and "being the first winner" is given publicly.

The private/public thing is a huge red herring. As are the "feelings of, among others, specialness" that are being presented as a basis for getting the answer.

It is true.

Is there a requirement to use mainstream mathematics on the forum?

 

[mfb]

Why would it be, and how is that relevant to the claims being made?

Relative probabilities are independent of the priors.

Do you agree with Ken that "the expectation doesn't go up because I can not choose a meaningful prior probability and therefor there is nothing to update" is a correct answer?

I do not agree, and I do not see where such a claim has been made.

It makes no difference whatsoever whether the information that you have is private or public (it's not even clear how you would define those concepts anyway), the only reason you're getting that result is because your examples are stated in such a way that the information of "being a winner" is given privately and "being the first winner" is given publicly.

Right.

Is there a requirement to use mainstream mathematics on the forum?

Sure (if you use mathematics).

 

[caveman1917]

Relative probabilities are independent of the priors.

Not exactly. Relative posterior probabilities depend on the relative prior probabilities. What you probably mean is that the factor with which the relative prior probability is updated to form the relative posterior probability is independent of the relative prior probability. You can see this by writing out Bayes' theorem for any two hypotheses and dividing the equations.

I do not agree, and I do not see where such a claim has been made.

->

No one here cares about a thread on some other forum. If they are wondering, the simple answer to your question is that what was resolved in that thread is that any extent to which you increase your expectation on N depends on what your expectation was going in, so if you had no expectation going in, there is nothing to update and no change in an expectation that doesn't exist.

The "having no meaningful expectation on N" answer is, in Ken's language, the statement above. That is, he claims, because a flat probability distribution over N has a "meaningful" expected value.

Sure (if you use mathematics).

What if you use vague philosophy that contradicts mainstream mathematics?

 

[mfb]

Not exactly. Relative posterior probabilities depend on the relative prior probabilities. What you probably mean is that the factor with which the relative prior probability is updated to form the relative posterior probability is independent of the relative prior probability. You can see this by writing out Bayes' theorem for any two hypotheses and dividing the equations.

That's what I mean, and that's what has been discussed all the time.

The "having no meaningful expectation on N" answer is, in Ken's language, the statement above. That is, he claims, because a flat probability distribution over N has a "meaningful" expected value.

Well, you cannot calculate an expectation value without a prior distribution, but you can correctly state that the expectation value will increase/decrease/whatever (depends on the setup) for all non-trivial prior distributions.

What if you use vague philosophy that contradicts mainstream mathematics?

Can we stop that meta-discussion?