Calculating the variance of integrated Poisson noise on a defined quantity

[fab13]

Sorry, I woud like to put it in Cosmology forum.

Regards

 

[fab13]

Isn't really there anyone that could help me about a good or natural weighting for my new observable ?

the primordial condition is that the ratio in this observable between numerator and denominator should be equal to the ratio of squared bias of 2 probes. I think that a natural form for this new observable is the ratio
between the sum of both angular power spectrum :

\begin{gathered}
O=\left(\frac{\mathcal{D}_{s p}}{\mathcal{D}_{p h}}\right) \\
\mathcal{D}_{s p}=\sum_{i=l}^{n} C_{\ell, s p}\left(\ell_{i}\right) \\
\mathcal{D}_{p h}=\sum_{i=l}^{n} C_{\ell, p h}\left(\ell_{i}\right)
\end{gathered}

It gives for instant the best result from a Figure of Merit point (FoM) of view.

Another form has been suggested but less efficient at FoM level :

$O=\dfrac{\sum_{\ell=1}^{N} \sum_{m=-\ell}^{\ell} a_{\ell m}^{2}}{\sum_{\ell=1}^{N} \sum_{m=-\ell}^{\ell}\left(a_{\ell m}^{\prime}\right)^{2}}=\dfrac{\sum_{\ell=1}^{N}(2\ell+1)\,C_{\ell,sp}}{\sum_{\ell=1}^{N}(2\ell+1)\,C_{\ell,ph}}$

The motivation of this second form was to say that for numerator and divider, we adda huge number of chi2 distribution, so we tend to get a gaussian PDF.

\begin{align}
Z&\equiv \sum_{\ell=\ell_{min}}^{\ell_{max}} \sum_{m=-\ell}^\ell a_{\ell,m}^2 \\[6pt]
&= \sum_{\ell=\ell_{min}}^{\ell_{max}} \sum_{m=-\ell}^\ell C_\ell \cdot \bigg( \frac{a_{\ell,m}}{\sqrt{C_\ell}} \bigg)^2 \\[6pt]
&\stackrel{d}{=}\sum_{\ell=\ell_{min}}^{\ell_{max}} \sum_{m=-\ell}^\ell C_\ell \cdot \chi^{2}(1) \\[6pt]
&= \sum_{\ell=\ell_{min}}^{\ell_{max}} C_\ell \sum_{m=-\ell}^\ell \chi^{2}(1) \\[6pt]
&= \sum_{\ell=\ell_{min}}^{\ell_{max}} C_\ell \cdot \chi^{2}(2 \ell + 1). \\[6pt]
\end{align} But actually, I have to compute if there is a gain on the standard deviation for this "almost-gaussian" compared to the sum of $C_\ell$ in the first form.

How to proceed for this computation ? If I manage to calculate analytically the standard deviation of the first and second form, I could compare them and prove basically that the first form of observable is the more appropriate.

Best regards

 

[kimbyd]

As before (Post 2):
$$\begin{equation}a_{\ell m}^S = x_{\ell m} + iy_{\ell m}\end{equation}$$

Careful. These values are not independent, as the transform of the $a_{\ell m}$ values must be real-valued. In practical terms this means that there are $2\ell + 1$ independent variables (not considering the covered sky fraction) per $\ell$.

You can manage this explicitly by working in a real-valued superposition of $a_{\ell m}$: $a_{\ell m} \pm a_{\ell -m}$.

 

[Twigg]

Isn't really there anyone that could help me about a good or natural weighting for my new observable ?

This thread has gotten decently long, and what you're asking has changed since the first post. No offense, but most folks have probably given up trying to understand. I don't mean to be critical. I'm just saying this as advice for you to get the most from the forums, since you're clearly putting in a lot of effort. Shorter, clearer posts will get much more responses (it's better if your first post is too somewhat too short and others have to ask clarification than if its too long from the start). Repeatedly posting the same question in a long thread doesn't help because it adds to the "wall of text" that new readers will see. Again, just advice, not being trying to be critical.

Regarding my last post, so do you have multiple data sets to look for underscatter or overscatter of your ratios and their error bars? If not, can you break your main data set into statistically independent (i.e. non-overlapping) chunks of data? Let the data tell you which weighting is more natural!

 

[Twigg]

If you have the time, can you also try a third observable? $$D = \sum_l (2l+1) C_l$$. If that gives a FoM equivalent to the second observable, that would at least confirm that the issue is the weighting on different spherical harmonics and that the issue is not correlations between m-channels.

 

[fab13]

@Twigg . Thanks for your advices.

1) Unfortunately, I have already tested in my previous post your "third observable" like this :

$O=\dfrac{\sum_{\ell=1}^{N} \sum_{m=-\ell}^{\ell} a_{\ell m}^{2}}{\sum_{\ell=1}^{N} \sum_{m=-\ell}^{\ell}\left(a_{\ell m}^{\prime}\right)^{2}}=\dfrac{\sum_{\ell=1}^{N}(2\ell+1)\,C_{\ell,sp}}{\sum_{\ell=1}^{N}(2\ell+1)\,C_{\ell,ph}}$

But it gives a lower FoM (1443) than for the first observable :

$\begin{gathered} O=\left(\frac{\mathcal{D}_{s p}}{\mathcal{D}_{p h}}\right) \\ \mathcal{D}_{s p}=\sum_{i=l}^{n} C_{\ell, s p}\left(\ell_{i}\right) \\ \mathcal{D}_{p h}=\sum_{i=l}^{n} C_{\ell, p h}\left(\ell_{i}\right) \end{gathered}$

with which I get FoM = 1556.

2) Jus an important point to be more explicit :

From the beginning, I try to use Inverse-variance weighted :

Given a sequence of independent observations $y_{i}$ with variances $\sigma_{i}^{2}$, the inversevariance weighted average is given by :
$\hat{y}=\frac{\sum_{i} y_{i} / \sigma_{i}^{2}}{\sum_{i} 1 / \sigma_{i}^{2}}\quad(1)$
The inverse-variance weighted average has the least variance among all weighted averages, which can be calculated as
$\operatorname{Var}(\hat{y})=\dfrac{1}{\sum_{i} 1 / \sigma_{i}^{2}}\quad(2)$
In our case, let us take the quantity $\hat{y}=C_{\ell}$ which is the average of $a_{\ell m}^{2}$ and so the variance of an $a_{\ell m}$ :
$C_{\ell}=\left\langle a_{l m}^{2}\right\rangle=\dfrac{1}{2 \ell+1} \sum_{m=-\ell}^{\ell} a_{\ell m}^{2}=\operatorname{Var}\left(a_{l m}\right)\quad(3)$
If we take all $\sigma_{i}^{2}$ equal to :
$\sigma_{i}^{2}=\frac{2}{f_{sky} N_{p}^{2}}\quad(4)$
Then we get an optimum variance regarding the Shot Noise. Let's demonstrate it with shot noise equal to the inverse of galaxies density : $N_{p}=N_{gal}$. For a given multipole $\ell$ given, we have :

$\operatorname{Var}(\text { Shot Noise optimal })(\ell)=\dfrac{1}{\sum_{m=-\ell}^{\ell} \dfrac{1}{\sigma_{i}^{2}}} =\dfrac{1}{(2 \ell+1) \dfrac{f_{s k y} N_{p}^{2}}{2}} =\dfrac{2}{(2 \ell+1)\left(f_{s k y} N_{p}^{2}\right)}\quad(5)$

QUESTION : the choice made in eq$(4)$ has been infered from the following relation expressing the standard deviation on a $C_\ell$ :

$\sigma_{C_\ell}(\ell)=\sqrt{\dfrac{2}{(2 \ell+1) f_{\mathrm{sky}}}}\left[C_\ell(\ell)+\dfrac{1}{N_{p}}\right]\quad(6)$

1) How can I justify the fact that I decide to assimilate $\hat{y}$ to a $C_\ell$ ?.

$C_\ell$ is a quantity appearing in spherical harmonics but this is already an average of $a_{\ell m}^2$ : is it pertinent from your point of view ?

Moreover, I don't make appear nowhere the formula eq$(1)$, I take it as it was right but I am not sure in my case with the average $C_\ell$.

2) Is the derivation of formula eq$(4)$ correct from eq$(6)$ ?

As you can see, I need clarifications.

 

[kimbyd]

@Twigg$C_\ell$ is a quantity appearing in spherical harmonics but this is already an average of $a_{\ell m}^2$ : is it pertinent from your point of view ?

That is extremely pertinent. Each $C_\ell$ represents $2\ell+1$ variables.