Expected value on subset of Pareto

[xag]

Hi -
I have a Pareto distribution X (xm=1, k known)*. High-value samples are filtered out and I want the expected value of the remaining. Namely: E[X|X<a].

*Wikipedia notations

Many thanks in advance.
Xavier

 

[Pere Callahan]

What is the definition of the conditional expectation you are referring to? How did you try to calculate it? Where did you get stuck?

 

[xag]

Hi Pere -

What I'm looking for is a mean value.
I found out that the samples of interest belong to a larger set that roughly follow a Pareto distribution, and that among this larger set, the samples of interest are the ones whose values are below a certain number, 'a'.

I can estimate the parameters of the Pareto once for all, and calculate 'a' for each run. But I can't access the samples for each run, so if there was a way to predict it with 'k' (the Pareto shape) and 'a', that would be great.

My first guess is that the samples of interest also follow a Pareto, but I've not shown that. And in this case, could we express its shape K and scale Xm? Then the mean I'm looking for would be K Xm/(K-1).


Xavier

 

[xag]

Ok. It seems I got all the way to the finish line and stopped just before :

I guess I just had to integrate the probability density function between 1 and a (it's defined between 1 and +oo).
It gives : (a ^(1 - k) - 1) * k / (1 - k)

Can anyone confirm?

 

[Pere Callahan]

I guess I just had to integrate the probability density function between 1 and a.
Can anyone confirm?

What you did gives you the probability of a sample with value less than a.

Try the following instead: First calculate the conditional distribution function$\mathbb{P}(X\leqx| X \leq a)$.
By definition this is equal to

$$\frac{\mathbb{P}(X\leq x \wedge X \leq a)}{\mathbb{P}(X\leq a)}$$

(You already calculated the denominator.)

You can also easily calculate the numerator. For x>a the numerator is $\mathbb{P}(X \leq a)$ which cancels with denominator giving one. For x>a you calculate it in the same way as you did for $\mathbb{P}(X \leq a)$ with a replaced by x. Then you take the derivative with respect to x to find the conditional density function $\rho_a(x)$ (suported on [1,a]), which you then use to calculate the conditional expectation as
$$\mathbb{E}\left[X|X\leq a\right]=\int_1^a{x\rho_a(x)dx}$$
A more direct though maybe less obvious formula would be
$$\mathbb{E}\left[X|X\leq a\right]=\frac{\int_1^a{x\rho(x)dx}}{\int_1^a{\rho(x)dx}}$$,
where $\rho(x)$ is the density function of your original Pareto distribution.
I suggest you try them both and check if they really are the same
-Pere

 

[xag]

Ok. It seems I got all the way to the finish line and stopped just before :

I guess I just had to integrate the probability density function between 1 and a (it's defined between 1 and +oo).
It gives : (a ^(1 - k) - 1) * k / (1 - k)

Can anyone confirm?

Forget this post: the thing I integrated was x * dF(x) and moreover the result I gave here is wrong.

 

[xag]

Try the following instead: First calculate the conditional distribution function$\mathbb{P}(X\leqx| X \leq a)$.
[...]
I suggest you try them both and check if they really are the same
-Pere

Thank you so much.

In both cases I find $$\mathbb{E}\left[X|X\leq a\right]= \frac{k}{k - 1} \frac{a ^ k - a}{a ^ k - 1}$$