Planck 2015 CMB temperature individual pixel variance

Click For Summary

Discussion Overview

The discussion revolves around the estimation of temperature variance for individual pixels in the Planck 2015 CMB temperature maps, specifically using the SMICA pipeline maps. Participants explore methods to derive variance, the availability of variance data, and the challenges associated with estimating noise from the provided maps.

Discussion Character

  • Exploratory
  • Technical explanation
  • Debate/contested

Main Points Raised

  • Tracey seeks guidance on estimating temperature variance for individual pixels in the Planck 2015 CMB maps, noting the absence of variance data in the provided maps.
  • Another participant points out that temperature variance data is available in the Planck Legacy Archive and provides instructions on how to access it.
  • Tracey expresses concern that the variance data from individual frequency maps may not apply to the SMICA map and questions the possibility of combining variances from different maps.
  • A participant confirms that the variance is specific to individual frequency maps and not applicable to the aggregated SMICA map, while also noting uncertainty about the publication of per-pixel noise for CMB maps.
  • Tracey shares solutions that worked for her, including using II covariances from specific frequency maps and smoothing HMHD maps for noise estimation.
  • Discussion includes a note that the Planck Legacy Archive does not provide raw instrument data and highlights the complexity of data analysis involved in processing the raw data.
  • A participant lists several technical challenges encountered in the analysis of raw data, emphasizing the intricacies involved in making the Planck data useful.

Areas of Agreement / Disagreement

Participants generally agree on the availability of variance data in the Planck Legacy Archive, but there is uncertainty regarding the applicability of this data to the SMICA map. Multiple competing views exist regarding the methods for estimating temperature variance and noise.

Contextual Notes

Participants mention limitations related to the absence of detailed noise maps and the complexity of the data processing steps required to analyze the raw instrument data.

tjsotherone
Messages
3
Reaction score
0
Hi,

I'm hoping someone can point me in the right direction please.

I'm using the Planck 2015 CMB temperature (intensity) SMICA pipeline maps (Nside = 2048) and am trying to determine the temperature variance of each individual pixel. Variance and hit-count were provided with the 2013 CMB maps (there were 3 columns: intensity, hit-count and variance) but they aren't in the 2015 maps (there is only an intensity column).

I've tried to estimate the variance from the half-difference of the half-mission and half-ring maps but my results look dodgy. I'm also very aware of the caution the Planck collaboration urge using these maps to estimate noise, and this approach doesn't feel well physically or statistically motivated to me.

There must have been hundreds of people using the Planck 2015 maps and I would imagine many of them would need to estimate temperature variance. But I can't find a standard/official approach in the literature or online. In fact, wherever I can find mention of variance it is usually rather vague and certainly not detailed enough for me to replicate the calculations.

Does anyone know whether the Planck collaboration have released the temperature variances? And if not, do you know if there is there a recommended method to derive them?

Many thanks,

Tracey.
 
Space news on Phys.org
They are available in the Planck Legacy Archive:
https://pla.esac.esa.int/pla/

Select "maps" then "advanced search". If you select the "exclude light maps" option, it will only show maps that have per-pixel errors attached. You're looking for maps with names like "LFI_SkyMap_070_2048_R2.01_full.fits". If you click the magnifying glass next to the map, it shows you what the columns are. The temperature variance is the II_cov column.
 
Dear kimbyd,

Thank you so much for your reply. I hadn't used the advanced search so that's really useful!

My understanding is that those maps are from individual instruments and individual frequency channels. So the example you used presumably is from the low frequency instrument 70GHz channel. I assume (always dangerous) that the variance is only applicable for that specific map and not the one I'm using (COM_CMB_IQU-smica-field-Int_2048_R2.01_full.fits). Do you know whether that's the case please?

Maybe there is a way to combine all the individual variances into a single SMICA one but I haven't found it yet...

Many thanks,

Tracey.
 
tjsotherone said:
Dear kimbyd,

Thank you so much for your reply. I hadn't used the advanced search so that's really useful!

My understanding is that those maps are from individual instruments and individual frequency channels. So the example you used presumably is from the low frequency instrument 70GHz channel. I assume (always dangerous) that the variance is only applicable for that specific map and not the one I'm using (COM_CMB_IQU-smica-field-Int_2048_R2.01_full.fits). Do you know whether that's the case please?

Maybe there is a way to combine all the individual variances into a single SMICA one but I haven't found it yet...

Many thanks,

Tracey.
There are a number of different ones. The one I pointed out is a map that is aggregated over the entire mission for all detectors in the 70GHz channel.

And yes, the variance is only applicable for the specific frequency map, not for the SMICA map, which is an aggregation over all of the maps.

Sadly, I'm not completely sure where or if they published per-pixel noise for the CMB maps. The component map products are described in detail here:
https://wiki.cosmos.esa.int/planckpla2015/index.php/CMB_and_astrophysical_component_maps

I don't see any mention of a noise map, but they do mention that you could use the sum/difference between the half-ring maps to produce a noise map estimate (the half-ring sum is in column HRHS, while the difference is in HRHD). Note the caveats they mention above.
 
After much gnashing of teeth (and much help from the IRSA and PLA folks - thank you!) I have a couple of solutions that have worked for me (all I'm doing is weighting a few pixels). Here they are in case they help anyone else out:

(1) the II covariances in the 143 & 217 GHz maps give good enough estimates of noise for weighting.
(2) the HMHD maps do work if you smooth them (which I hadn't previously).
 
While it may seem obvious, the Planck Legacy Archive does not offer raw instrument data as noted here https://commons.lbl.gov/download/attachments/111706420/DMP_planck.pdf:
"Plan for Serving Data to the Collaboration and Community:
Data are released to the public with the major Planck science publication releases. Both ESA and
NASA have dedicated archives to distribute the data, along with complete descriptions of their
contents. There is no formal plan to release the raw data processing pipelines since their use
depends on very detailed instrument knowledge that will not survive the collaboration."
For discussion of the processing steps traken to convert HFI raw instument data into published outputs, this may be of interest: http://planck.caltech.edu/pub/2015results/Planck_2015_Results_VII_HFI_Data_Proc_TOI_Beams.pdf
 
Chronos said:
While it may seem obvious, the Planck Legacy Archive does not offer raw instrument data as noted here https://commons.lbl.gov/download/attachments/111706420/DMP_planck.pdf:
"Plan for Serving Data to the Collaboration and Community:
Data are released to the public with the major Planck science publication releases. Both ESA and
NASA have dedicated archives to distribute the data, along with complete descriptions of their
contents. There is no formal plan to release the raw data processing pipelines since their use
depends on very detailed instrument knowledge that will not survive the collaboration."
For discussion of the processing steps traken to convert HFI raw instument data into published outputs, this may be of interest: http://planck.caltech.edu/pub/2015results/Planck_2015_Results_VII_HFI_Data_Proc_TOI_Beams.pdf
Yeah, I don't think it would make sense for anybody else to examine the raw data. I was actually involved in some of that analysis a few years ago (on the LFI side), and the data analysis is quite complex. Some of the things you have to worry about:
1) The time constants of the detectors (once a detector passes a bright source, how long does it take for the detector to recover?).
2) The shapes of what is known as the beam (usually called the point spread function in optical astronomy, this is the shape of the blob that a point source appears to be when viewed by the detector). These are not known precisely until the satellite flies, so they have to be computed from observations of bright objects (e.g. planets). Each detector has a slightly different beam shape, and that shape changes depending upon which polarization signal you're looking at.
3) There was a 1Hz square wave that appeared in the LFI detectors that had to be removed.
4) The telescope will receive some amount of signal from locations far from the direction the detector is pointing. The satellite was designed to minimize this, but it still makes for a substantial contribution (this is known as the "far side lobes" of the beam.
5) The precise angles of the two polarization arms for each detector weren't known until the satellite flew, making it challenging to correctly estimate the polarization signal.
6) The refrigeration units slowly degraded over time, causing various temperature fluctuations which impacted the signal.

Anyway, if you're at all curious as to the work that was involved in making the Planck data useful, and some of the work that would have to be redone to make use of the raw data, you can take a look at these papers:
https://arxiv.org/abs/1507.08853 (LFI systematic uncertainties)
https://arxiv.org/abs/1605.02985 (HFI polarization systematics)
https://arxiv.org/abs/1502.01586 (HFI time-ordered data processing)
 
  • Like
Likes   Reactions: Imager