Add factor Δℓ or Δℓ^2 in the variance for integrated Shot noise

[fab13]

TL;DR Summary

I would like to find the right expression of the variance for Shot noise integrated over multipoles l in the calculation on a Cℓ. I don't know in my code if i have to include a Δℓ or Δℓ^2 as factor of the variance since Var(aX) = a^2 Var(X) or simply get rid of Δℓ and make it disappear.

I have the following expression for an error on a Cℓ :

$\sigma(C_{\ell,X})=\sqrt{\dfrac{2}{(2 \ell+1)\Delta\ell \,f_{\mathrm{sky}}}}\left[C_{\ell,X}+N_{X}(\ell)\right]$

where $X$ corresponds to spectroscopic/photometric shot noise and with $\Delta\ell$ is the bin width between 2 values of computed angular power spectrum $C_{\ell}$.

I wonder if I can get rid of of $\Delta\ell$ in the expression of variance of Shot Noise (here spectroscopic) which is summed over multipole betwen $\ell_{min}$ and $\ell_{max}$.

I have for the moment the variance of integrated spectroscopic Shot Noise :

$<N_{sp}^{2}>=\sum\limits_{\ell=l_{min}}^{\ell_{max}} \dfrac{2}{(2 \ell+1)\, f_{\mathrm{sky}}}\,\dfrac{1}{\bar{n}_{sp}^{\,\,2}}$

As you can see, I have eliminated the $\Delta\ell$ bin width since I consider that I can take $\Delta\ell=1$ for the integral :

$\text{Var}(N_{sp})= \int_{\ell=\ell_{min}}^{\ell_{max}} \dfrac{1}{\Delta\ell} \dfrac{2}{(2 \ell+1)\,\, f_{\mathrm{sky}}}\,\dfrac{1}{\bar{n}_{sp}^2}\text{d}\ell$

Indeed, I could approximate also this integral by rectangular method integration and take $\text{d}\ell=1$.

this way, we could write simply :

$\text{Var}(N_{sp})=\sum\limits_{l=l_{\min }}^{l_{\max }} \dfrac{2}{(2 \ell+1)\, f_{\mathrm{sky}}}\,\dfrac{1}{\bar{n}_{sp}^{\,\,2}}\quad(1)$

Do you agree with this reasoning and obtained expression $(1)$ ?

Or do you think I have to keep a $\Delta\ell^2$ factor (since $\text{Var}(\Delta\ell X) = \Delta\ell^2\,\text{Var}(X)$ and write :

$\text{Var}(N_{sp})=\sum\limits_{l=l_{\min }}^{l_{\max }} \dfrac{2}{(2 \ell+1)\, f_{\mathrm{sky}}}\,\dfrac{1}{\bar{n}_{sp}^{\,\,2}}\Delta\ell\,\quad(2)$

??

 

[kimbyd]

If you're applying the variance of each multipole individually without binning, yes you can get rid of that factor. If your variance is for a bin of multiple ell values, then you need to use it.

 

[fab13]

@kimbyd . Thanks for your quick answer.

Do you talk about the $\Delta\ell$ into the pre-factor :

$\sigma(C_{\ell,X})=\sqrt{\dfrac{2}{(2 \ell+1)\Delta\ell \,f_{\mathrm{sky}}}}\left[C_{\ell,X}+N_{X}(\ell)\right]$

?

For me, the main issue is about the factor $\Delta\ell$ or $\Delta\ell^2$ that I have to apply when I compute $Var(aX) = a^2 Var(X)$ with $a=\Delta\ell$ : this occurs when I integrate over the multipole the quantity :

$\Rightarrow \mathcal{D}_{sp}=\int_{\ell_{\min}}^{\ell_{\max }} C_{\ell,sp}(\ell) \,\mathrm{d}\ell = \Bigg(\dfrac{b_{sp}}{b_{ph}}\Bigg)^{2}\,\int_{\ell_{\min}}^{\ell_{\max }} \hat{C}_{\ell,ph}(\ell) \,\mathrm{d}\ell= \Bigg(\dfrac{b_{sp}}{b_{ph}}\Bigg)^{2}\,\mathcal{D}_{ph}$

and our estimator :

$\hat{O}=b_{s p}^{2}\left(\mathcal{D}_{DM}+B^{C}\right)+B_{s p}$

I say to myself that I should keep the pre-factor $\dfrac{2}{(2 \ell+1)\Delta\ell \,f_{\mathrm{sky}}}$ for the variance , considering $\text{Var}(B_{sp}) =\dfrac{2}{(2 \ell+1)\Delta\ell \,f_{\mathrm{sky}}} \dfrac{1}{\bar{n}^{2}}$.

What do you think about if I must apply the factor $\Delta\ell$ or $\Delta\ell^2$ to $\text{Var}(B_{sp})$ since I integrate $B_{sp}$ over multipole between $\ell_{min}$ and $\ell_{max}$ ?

Regards

 

[kimbyd]

I'm a little unsure as to what you're trying to do here. Why integrate the shot noise?

 

[fab13]

oh sorry, I forgot to say : I integrate the shot noise since I am working on the estimator where I integrate the angular power spectrum :

$\hat{O}=\hat{\mathcal{D}}_{sp,tot}=b_{sp}^{2}\left(\mathcal{D}_{DM}+B^{C}\right) +B_{s p}$

with $\mathcal{D}_{sp}=\int_{\ell_{\min}}^{\ell_{\max }} C_{\ell,sp}(\ell) \,\mathrm{d}\ell$ and

implicitily : $\int_{\ell_{\min}}^{\ell_{\max }} C_{\ell,sp}(\ell) \,\mathrm{d}\ell = b_{sp}^2\,(\mathcal{D}_{DM}+B^{C})$

($B^C$ is cosmic variance = intrinsic noise).

All is done to be coherent : If I integrate on the angular power spectrum in the estimator $\hat{O}$, I should integrate also on the Shot noise $B_{s p}$, do you agree ?

And like I have a fnite number of values for angular power spectrum ($\Delta\ell \simeq 80$ in my code), I wonder how to manage this integration of Shot Noise with this ($\Delta\ell \simeq 80$), that is to say to know if I make appear a $\Delta\ell$ in the rectangular integration method of Shot noise, which would cause the presence of a $\Delta\ell^2$ if I compute the variance, wouldn't it ?

Thanks for your support.

 

[kimbyd]

oh sorry, I forgot to say : I integrate the shot noise since I am working on the estimator where I integrate the angular power spectrum :

$\hat{O}=\hat{\mathcal{D}}_{sp,tot}=b_{sp}^{2}\left(\mathcal{D}_{DM}+B^{C}\right) +B_{s p}$

with $\mathcal{D}_{sp}=\int_{\ell_{\min}}^{\ell_{\max }} C_{\ell,sp}(\ell) \,\mathrm{d}\ell$ and

implicitily : $\int_{\ell_{\min}}^{\ell_{\max }} C_{\ell,sp}(\ell) \,\mathrm{d}\ell = b_{sp}^2\,(\mathcal{D}_{DM}+B^{C})$

($B^C$ is cosmic variance = intrinsic noise).

All is done to be coherent : If I integrate on the angular power spectrum in the estimator $\hat{O}$, I should integrate also on the Shot noise $B_{s p}$, do you agree ?

And like I have a fnite number of values for angular power spectrum ($\Delta\ell \simeq 80$ in my code), I wonder how to manage this integration of Shot Noise with this ($\Delta\ell \simeq 80$), that is to say to know if I make appear a $\Delta\ell$ in the rectangular integration method of Shot noise, which would cause the presence of a $\Delta\ell^2$ if I compute the variance, wouldn't it ?

Thanks for your support.

Where is this shot noise coming from?

Normally in CMB science there are two sources of error: observation error and cosmic variance. The observation error cannot simply be described, as it's a function of the complex process which goes into generating the power spectrum from the data.

 

[fab13]

@kimbyd

The following formula :

$\sigma(C_{\ell,X})=\sqrt{\dfrac{2}{(2 \ell+1)\Delta\ell \,f_{\mathrm{sky}}}}\left[C_{\ell,X}+N_{X}(\ell)\right]$

comes from the equation (137) on the following paper : https://arxiv.org/pdf/1910.09273.pdf

In the case of spectroscopic Shot Noise, I hope you will understand my issue about to multiply or not the Shot Noise $N_{X}\equiv B_{sp}$ part of my estimator $\hat{O}$ by the bin width $\Delta\ell$ (or by its squared) which is involved since we have a $N$ finite number of values $C_{\ell_i}$ and $\ell_i$ with $i=1,..,N$.

Regards

 

[kimbyd]

The spectroscopic shot noise is for the galaxy observations, which are typically made using the linear power spectrum P_k. Why are you using spherical harmonics here?

 

[fab13]

in the case that I am studying, we use the spectroscopic angualr power spectrum noted $C_{\ell,sp}$.
Given the fact that I sample on the same volume, we have a link with the photometric angular power spectrum : $C_{\ell,sp}=\Big(\dfrac{b_{sp}}{b_{ph}}\Big)^{2}\,C_{\ell,ph}$.

Hoping you will understand this context but the main issue is about this factor $\Delta\ell$ or $\Delta\ell^2$ to apply or not in the computation $B_{sp}$ of the estimator $\hat{O}=\hat{\mathcal{D}}_{sp,tot}=b_{sp}^{2}\left(\mathcal{D}_{DM}+B^{C}\right) +B_{sp}$

 

[kimbyd]

To get back to it, I'm not completely sure, and sadly I'm not up for taking the time to wrap my head fully around the problem right now. But you could get at the answer by asking about the statistics of the power spectrum of the shot noise.

I honestly wouldn't trust anything terribly simple here. Ultimately you're adding up a bunch of correlated random variables when doing the integral. You might be able to get away with assuming they're not correlated, which is I think the simple answer you're looking for here. But I'm not sure that's the right thing to do, even if done correctly.

It honestly shouldn't be hard to simulate a pure shot noise signal, take its spherical harmonic power spectrum, and then just perform the integral. That exercise will tell you right away which factor is closest to being correct.

The thing I'm worried about here is that the equations you're using are assuming CMB-like statistics, which are that of a Gaussian random field. If you're not looking at the CMB, those statistics are going to be very, very different. Even toy simulations for the noise will probably do a much better job than the equations here.

If you want something close to the ideal case, try a shot noise signal whose probability distribution is uniform across the entire sky (one random value from the same distribution for each pixel). That will most likely give you something really close to the analytical solution you're looking for. Then if you make the shot noise look more like the real thing, that will probably break down.