Comparing A & B in Bernoulli Trials w/ X & P

[slipperypete]

Let $$X$$ be the set $$X=\left\{x_1,x_2,...,x_n\right\}$$, where each $$x_i$$ is a Bernoulli random variable and $$P$$ be the set $$P=\left\{p_1,p_2,...,p_n\right\}$$, where $$p_i$$ is the probability that $$x_i=1$$. Now, suppose there are two other sets, $$A=\left\{a_1,a_2,...,a_n\right\}$$ and $$B=\left\{b_1,b_2,...,b_n\right\}$$, where $$a_i,b_i$$ are two different estimates for $$p_i$$.

In other words, p is the "true" (unknown) probability that x will occur, and a and b are both attempts to estimate that probability.

I am trying to design an experiment that will determine which set, A or B, is "closer" to P.

If these Bernoulli random variables represented, say, different-sided dice, then an experiment would be pretty straightforward: conduct a Bernoulli process; i.e., repeatedly perform a Bernoulli trial.

However, in my case, each $$x_i$$ represents a real-world event, and can be simulated only once. A Bernoulli process is impossible, I am limited to a single Bernoulli trial. If $$X=\left\{x_1,x_2,x_3\right\}$$, then I don't think there would be a practical way to solve this problem at any level of significance. But $$n$$ here is actually quite large, so I feel like there should be some test which would allow me to show $$A>B$$ or $$A<B$$ or $$A\neq B$$ or $$A=B$$.

Any thoughts?

 

[EnumaElish]

You can apply a matched-pairs t test to the difference between average(A) and average(B).