Struggling with Bayesian Truth Serum formula

In summary, the conversation is about the Bayesian Truth Serum formula and its use in calculating scores. The formula is used to elicit truthful subjective data in situations where the objective truth is unknowable. It involves asking individuals to endorse their personal answer to a multiple-choice question and then predicting what others in the sample might answer. The formula consists of an information score and a prediction score. The conversation also includes a discussion about a fictional survey and the calculation of information scores for the answers red and blue. The correct information scores for red and blue should be -0.04722677 and 0.24622888, but the individual is struggling to get the correct values. They are asking for help in identifying where they went wrong in their calculation.
  • #1
Illmaticus
1
0
Hi there,

right now, I am struggling to successfully calculate scores with the Bayesian Truth formula and I hope this is the right place to find someone who can help me along
For everyone, who doesn’t know it, I will summarize it briefly : The Bayesian Truth Serum is an scoring method, that provides truth-telling incentives and elicits “truthful subjective data in situations where the objective truth is unknowable”. The basic idea is, that truthful answers maximize ones individual score.

When using this method, people are asked for dual reports:
  • First they are asked to endorse their personal answer to an m multiple-choice question. E.g. Which color do you like most - Red or Blue?
  • Second, they are asked to predict what the other individuals in the sample might answer: Form a prediction of the sample distribution of all endorsments: Red __% ; Blue __%

  • The respondents are defined as r ∈ {1,2,…}
  • xrk indicates whether respondent r has endorsed answer k ∈ {1,…,m} and xrk ∈ {0,1} meaning that xrk=0 for all answers except the one endorsed by r.
  • yr = (yr1,…yrm) indicates the prediction of the sample distribution) and yrm>0 and [itex]\sum\limits_{k}^{} y^{r}_{k}[/itex]. So in this example it could be (0,6;0,4) for someone who thinks that most people like red.

The formula consists of an information score + prediction score.

  • The information score is given as [itex]\sum\limits_{k}^{} x^{r}_k \cdot log \frac{x^{-}_k }{y^{-}_k} [/itex] . [itex]x^{-}_k[/itex] is the fraction endorsing answer k and [itex]y^{-}_k[/itex] is the geometric average of endorsment prediction yr = (yr-,…yrm).
  • The prediction score is given as [itex]a\cdot \sum\limits_{k}^{} x^{-}_{k} \cdot log \frac{y^{r}_k }{x^{-}_k }[/itex]

Back again to the question about the favorite Color red or blue: I have created a table for a fictional Survey in order to practice the calculation of those BTS-scores.

HTML:
<table border="1" style="background-color:#FFFFCC;border-collapse:collapse;border:1px solid #FFCC00;color:#000000;width:100%" cellpadding="3" cellspacing="3">
    <tr>
        <td>Individual Answerl</td>
        <td></td>
        <td></td>
        <td></td>
        <td>Predicted fraction</td>
        <td></td>
    </tr>
    <tr>
        <td>r</td>
        <td>red</td>
        <td>blue</td>
        <td>l</td>
        <td>redl</td>
        <td>blue</td>
    </tr>
    <tr>
        <td>1l</td>
        <td>1</td>
        <td>0</td>
        <td></td>
        <td>0,8</td>
        <td>0,2</td>
    </tr>
    <tr>
        <td>1l</td>
        <td>1</td>
        <td>0</td>
        <td></td>
        <td>0,8</td>
        <td>0,2</td>
    </tr>
    <tr>
        <td>1l</td>
        <td>1</td>
        <td>0</td>
        <td></td>
        <td>0,8</td>
        <td>0,2</td>
    </tr>
    <tr>
        <td>1l</td>
        <td>1</td>
        <td>0</td>
        <td></td>
        <td>0,8</td>
        <td>0,2</td>
    </tr>
    <tr>
        <td>1l</td>
        <td>1</td>
        <td>0</td>
        <td></td>
        <td>0,8</td>
        <td>0,2</td>
    </tr>
    <tr>
        <td>1l</td>
        <td>1</td>
        <td>0</td>
        <td></td>
        <td>0,8</td>
        <td>0,2</td>
    </tr>
    <tr>
        <td>1l</td>
        <td>1</td>
        <td>0</td>
        <td></td>
        <td>0,8</td>
        <td>0,2</td>
    </tr>
    <tr>
        <td>7</td>
        <td>0</td>
        <td>1</td>
        <td></td>
        <td>0,6</td>
        <td>0,4</td>
    </tr>
    <tr>
        <td>7</td>
        <td>0</td>
        <td>1</td>
        <td></td>
        <td>0,6</td>
        <td>0,4</td>
    </tr>
    <tr>
        <td>7</td>
        <td>0</td>
        <td>1</td>
        <td></td>
        <td>0,6</td>
        <td>0,4</td>
    </tr>
    <tr>
        <td>average</td>
        <td>0,7</td>
        <td>0,3</td>
        <td>geometric average</td>
        <td>0,73385</td>
        <td>0,24623</td>
    </tr>
</table>
I also have the Information scores for both answers red and blue given: information score for red should be -0,04722677. The info-score for blue should be 0,24622888.

Now that's where i am struggling. When I calculate it myself, i am getting Information score (red)= -0,0205103 (so for people stating that their favorite color is red) and Information score (blue) = 0,0857823, which is both not correct.
should be -0,04722677. The info-score for blue should be 0,24622888.

I calculated it using the formula [itex]1*log\frac{0,7}{0,73385} + 0*log\frac{0,3}{0,24623}[/itex]

What am I doing wrong?
 
Physics news on Phys.org
  • #2
Only the html source shows up in my browser. I gather there are 10 respondents. Seven of them pick "red" and all seven estimate the population distribution to be 0.8 for red and 0.2 for blue. Three of the respondents pick "blue" and all three estimate the population distribution to be 0.6 for red and 0.4 for blue.

The original paper about "Bayesian Truth Serum" appears to be http://www.google.com/url?sa=t&rct=...hnoy5yfj1AcSDhg&bvm=bv.93756505,d.aWw&cad=rja

Now, we just need some kind person to check you arithmetic.
 

Related to Struggling with Bayesian Truth Serum formula

What is Bayesian Truth Serum formula?

Bayesian Truth Serum formula is a statistical method used to elicit truthful responses from individuals in a survey or study. It uses Bayesian inference to determine the likelihood of a response being true based on prior information and the responses of others.

How does Bayesian Truth Serum formula work?

The formula works by asking individuals to provide a response to a question, along with a confidence level for their response. The responses are then compared to the responses of others, and the confidence levels are used to weigh the likelihood of each response being true. The final result is a more accurate estimation of the truth.

What are the benefits of using Bayesian Truth Serum formula?

Bayesian Truth Serum formula has several benefits, including the ability to elicit more truthful responses from individuals, even for sensitive or controversial topics. It also takes into account the uncertainty of responses and provides a more accurate estimation of the truth compared to traditional survey methods.

What are the limitations of Bayesian Truth Serum formula?

One limitation of Bayesian Truth Serum formula is that it relies on the assumption that individuals will provide honest responses. If an individual intentionally provides a false response, it can affect the accuracy of the overall results. Additionally, the formula may not work well for complex or multi-faceted questions.

How can Bayesian Truth Serum formula be applied in real-world situations?

Bayesian Truth Serum formula can be applied in various real-world situations, such as market research, political polling, and social science studies. It can also be used in situations where traditional survey methods may be biased or inaccurate, such as in online surveys or surveys with sensitive topics.

Similar threads

  • MATLAB, Maple, Mathematica, LaTeX
Replies
3
Views
465
  • General Math
4
Replies
125
Views
17K
  • Math Proof Training and Practice
2
Replies
62
Views
9K
  • Programming and Computer Science
Replies
1
Views
2K
  • Math Proof Training and Practice
2
Replies
38
Views
9K
  • Math Proof Training and Practice
2
Replies
63
Views
9K
  • Math Proof Training and Practice
6
Replies
175
Views
21K
  • Math Proof Training and Practice
2
Replies
67
Views
11K
  • Advanced Physics Homework Help
Replies
1
Views
2K
  • Calculus and Beyond Homework Help
Replies
1
Views
3K
Back
Top