Relative frequency and nonmeasurable sets

[stevendaryl]

There is a conceptual puzzle that I don't understand about nonmeasurable sets. Take the unit interval $[0, 1]$ and let $S$ be some subset. Now, generate (using a flat distribution) a sequence of reals in the interval:

$r_1, r_2, r_3, ...$

Then we can define the relative frequency up to $n$ as follows:

$f_n = 1/n \sum_j \Pi_S(r_j)$

where $\Pi_S(x)$ is the characteristic function for $S$: it returns 1 if $x \in S$ and 0 otherwise.

If $S$ is measurable, then with probability 1, $lim_{n \rightarrow \infty} f_n = \mu(S)$, where $\mu(S)$ is the measure of $S$. But if $S$ is not measurable, then what will be true about $f_n$? Will it simply not have a limit? Or since this is a random sequence, we can ask what is the probability that it will have a limit. Does that probability exist?

 

[Stephen Tashi]

If $S$ is measurable, then with probability 1, $lim_{n \rightarrow \infty} f_n = \mu(S)$, where $\mu(S)$ is the measure of $S$.

Why take a limit in order to create a paradox? Consider the probability distribution of the random variable $f_1$. For a set $S$ that is measurable by the uniform probability density distribution $\mu$ on [0,1] , we have $P( f_1 = 1) = \mu(S)$. If $S$ is not measurable then apparently $P(f_1=1)$ is undefined.

Until we define how $f_1$ is distributed we can't proceed to the problem of defining $lim_{n \rightarrow \infty} f_n$

 

[Stephen Tashi]

The dilemma's connected with sampling are interesting to discuss and difficult. As far as I can see there is no theoretical basis for talking about random samples in the measure theoretic approach to probability. There is no axiom in measure theory that says it is possible to take random samples. There are no formal definitions that allow us to discuss an event that has a probability and then "actually" happens or doesn't happen.

The current treatment of sampling is matter of applied mathematics. The theoretical basis for assigning probabilities to outcomes of sampling is an application of conditional probability, but if we look at the measure theoretic definition of conditional probability, it is defined abstractly as a ratio , not as a formalization of the concept than an event that has been assigned a probability changes its state from "potential" to "actual".

Is it possible to create a mathematical system that treats sampling in a rigorous manner? For example, how do we handle the dilemma that the probability of realizing any given random sample from a normal distribution is zero? Do we define sampling-with-finite-precision and then define taking a random sample from a continuous distribution as some sort of limit?

The practical person's outlook on unmeasureable sets is "If I can take independent random samples then I can come up with a practical estimate for the probability of an outcome being in a supposedly unmeasureable set". I don't have a good intuition about how such an attempt would fail!

Two thoughts:

1) Assume random sampling is done with a finite precision, so a random sample is drawn from a discrete distribution and "theoretical" random sampling from a continuous distribution is defined as some sort of limit as we let a family of discrete distributions approach (in some sense) a continuous distribution. Then perhaps the inability to assign a probability to an umeasureable set is a situation where such a limit fails to exist. For example, perhaps two different ways of letting the discrete family approach the same continuous distribution would imply two different probabilities for the unmeasureable set.

2) The objection to a Vitali set being measureable isn't that it can't be assigned a probability, but rather that we can't assign it a translation invariant probability. I don't know whether all unmeasureable sets have this feature.

 

[stevendaryl]

The objection to a Vitali set being measureable isn't that it can't be assigned a probability, but rather that we can't assign it a translation invariant probability. I don't know whether all unmeasureable sets have this feature.

If we assume the continuum hypothesis (yes, I know that there is no good reason to assume that), then that means that we can come up with a total ordering $\prec$ such that for every real $x$ in $[0,1]$, the set of all $y$ such that $y \prec x$ is countable (and the set of all $y$ such that $x \prec y$ is uncountable). Then letting $S$ be the set of all pairs $(x,y)$ such that $0 \leq x \leq 1$ and $0 \leq y \eq 1$ and $x \prec y$, $S$ can't be measurable in the normal product measure on $R^2$.

 

[Stephen Tashi]

It would be interesting to formulate a definition of "realizable" sets that would formalized the intuitive notion of sets that can actually be the result of random sampling. For a continuous real valued random variable, there are measurable sets (such as a single numerical value - a "point") that cannot be realized due to limitations on precision of measurements - or, more abstractly, limitations on the information that is provided by a sampling procedure.

We could skirt the issue of how and whether probable events become "actual" events by defining "realizable sets" as those that exist in the context of sampling from a discrete distribution. Perhaps that's a cowardly approach, but, at the moment, I don't see a good alternative.

Using that approach, to speak of "realizable sets" of a continuous distribution, we have to make some connection between discrete distributions and the continuous distribution. The first thought that comes to mind is to partition the range of the random variable up into disjoint intervals (each having nonzero probability) and consider the discrete distribution that assigns the probability of a given interval being realized as the probability measure assigned to that interval by the continuous distribution. The realizable sets would be the disjoint intervals and finite (or countable?) unions of such intervals. Intersections of realizable sets would not necessarily be realizable (e.g. two intervals intersecting at a point) so the realizable sets would not form a sigma algebra.

( Are there clever ways to use sampling that has a limited interval of precision to realize sets that are more interesting than unions of intervals? Can anything clever be done by transforming the random variable and realizing samples form the transformed distribution? )

The terminology "r is a realizable set of the continuous distribution F". would mean that there exists a partition of the range of F into disjoint intervals, each having non-zero probability, such that r can be expressed as a countable union of some of these intervals.

I've been ambiguous about whether and "interval" should be an open interval, closed interval, half-open interval, etc. because I think any type of interval can be allowed.

----
Another way of considering practical sampling is consider what is practical in terms of computer simulations. For example, if we have an algorithm that generates pseudo random numbers from uniform discrete distribution on 1,2,..N then we can approximate sampling from a uniform distribution on [0,1] by creating the sample in stages. First, pick one of N equal sub intervals of [0,1]. The subdivide that sub interval into N sub-sub intervals and pick one of those sub-sub intervals, etc.

It seems that this point of view also leads to defining realizable sets in terms of intervals. However, a simulation isn't limited by the interval-based precision of physical measuring instruments, so perhaps there is room for more imagination.