News Science vs. Politics: Tipping Points in Climate Change Communication

[Proton Soup]

from what i remember on the CRU thread, CRU had a bunch of data sets in printed form. at some point, persons at CRU had to manually enter all this data into a computer. but what CRU did not have apparently was computing resources to store (and maybe more accurately, process) all this data. so, they produced a roughened, courser, "homogenized" data set from the original data with which to work. at some point, the printed data was deemed either burdensome or unnecessary and destroyed. presumably, either the computing resources to handle large data sets was still not available, or perhaps rough estimates were simply deemed "good enough", that seems unclear. waiting on simulations to complete is certainly time-consuming, so it is at least one factor to consider.

as for what i don't remember reading, what is this mysterious homogenization algorithm? has anyone ever reproduced the homogenized data set? one would assume data is shared between colleagues and that the basic tenet of science (reproducibility) is adhered to. if not, then we have no validation, a broken chain of custody, and no assurance that results produced from the data (and homogenized data) are valid.

another thing that's curious to me: how many cubic feet of printed hardcopies of original data are we talking about here? an entire room, a broom closet, or a bit of space on one of the researcher's bookshelf?

 

[Sorry!]

The most trusted one, common sense. If more people are "convinced" of AGW, then there would be more pressure for scientists to provide data to support AGW to get funding. I am NOT saying that there is some super deep international fraud, but the circumstances are special enough to make it suspect.

With all the politically motivated nonscientific "eyes" on it, it is hard to consider climatology as trustworthy as the other sciences.

This does not source anything for your comment, specifically:

I think this would explain why some climatologists "hide" data, to get funding.

It doesn't seem logical for common sense to indicate 'climatologists are hiding data to get funding'. I agree with everything you've said except for this and I'd like you to specifically cite where you know data has been 'hidden'.

It's important to note that what I'm asking for as well isn't a paper that has not included specific data then later on has explained why they have not used this data. You can hardly say that is hiding anything.

EDIT: Here's a pdf on quality control:
http://envgen.nox.ac.uk/courses/Microarray_data_quality/dataquality_control.pdf

 

[Proton Soup]

In this it says:

Inhomogeneities are introduced into the station temperature series by such things as changes in the station site, changes in measurement time, or changes in instrumentation. The station data that are used to make HadCRUT have been adjusted to remove these inhomogeneities, but such adjustments are not exact—there are uncertainties associated with them.

For some stations both the adjusted and unadjusted time-series are archived at CRU and so the adjustments that have been made are known [Jones et al., 1985, Jones et al., 1986, Vincent & Gullet, 1999], but for most stations only a single series is archived, so any adjustments that might have been made (e.g. by National Met. services or individual scientists) are unknown.[/color]​

ergo, adjustments for most stations are unknown. thanks.

 

[sylas]

ergo, adjustments for most stations are unknown. thanks.

That's right. The paper goes on to calculate the associated uncertainties, and this is part of the standard error estimated in the final data set. I refer you to the paper for details.

If you want to try your hand at homogenization yourself, try using the USHCN network, where you can get hold of all kinds of intermediate files plus the meta-data.

Cheers -- sylas

 

[Evo]

It's here Sylas, I gave the information about this back on 12-01. Russ, Astronuc, DH, we all dicussed the fact the CRU said they didn't have the requested information and you kept going off on a tangent saying that they had it, but weren't allowed to give it out, which had nothing to do with what the rest of us were discussing.

Roger Pielke had requested the data from CRU and had been told that they no longer held the data.

Obviously, the ability to do good research depends upon good data with known provenance. At the time WMO Resolution 40 was widely hailed in the atmospheric sciences community as a major step forward in data sharing and availability in support of both operations and research.

Thus it is with some surprise to observe CRU going through bizarre contortions to avoid releasing its climate data to Steve McIntyre. They first told him that he couldn't have it because he was not an academic. I found this to be a petty reason for keeping data out of the hands of someone who clearly wants to examine it for scholarly purposes. So, wanting to test this theory I asked CRU for the data myself, being a "real" academic. I received a letter back from CRU stating that I couldn't have the data because "we do not hold the requested information."

I found that odd. How can they not hold the data when they are showing graphs of global temperatures on their webpage? However, it turns out that CRU has in response to requests for its data put up a new webpage with the following remarkable admission (emphasis added)

(From CRU's website)
We are not in a position to supply data for a particular country not covered by the example agreements referred to earlier, as we have never had sufficient resources to keep track of the exact source of each individual monthly value. Since the 1980s, we have merged the data we have received into existing series or begun new ones, so it is impossible to say if all stations within a particular country or if all of an individual record should be freely available. Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added (i.e. quality controlled and homogenized) data.

Say what?! CRU has lost track of the original data that it uses to create its global temperature record!? Can this be serious? So not only is it now impossible to replicate or reevaluate homogeneity adjustments made in the past -- which might be important to do as new information is learned about the spatial representativeness of siting, land use effects, and so on -- but it is now also impossible to create a new temperature index from scratch. CRU is basically saying, "trust us." So much for settling questions and resolving debates with empirical information (i.e., science).

http://rogerpielkejr.blogspot.com/2009/08/we-lost-original-data.html

 

[Integral]

Yep, Just like Sylas said? Now where's the problem?

 

[Evo]

Yep, Just like Sylas said? Now where's the problem?

Like Sylas said when? Sylas disputes that this ever happened. He disputes that CRU ever said that (that's from CRU's website, BTW). What on Earth are are you talking about Integral? Please explain.

 

[Evo]

I object to you telling me what I "know", and I continue to assert that I am doing my best to give you honest information in good faith. I continue to stand by what I have said in the thread, and am happy to explain it, but I would prefer you to assume my good faith in this. If you cannot assume my good faith, say so frankly.

We should be able to disagree and work towards whatever common ground is possible without this kind of projection about what the other person really thinks. It's disruptive of civil and robust debate. You do this much too often.

Not nearly as bad as your petty attempts at trying to discredit me. Let's agree to stop the personal attacks.

For my part, I do not claim that you "know" I am right. I think you are mistaken.

I've never said you were right, so I have no idea what you're talking about.

Evo, in [post=2502750]msg #66[/post] you said: "they claim to not have knowledge of precisely which points were udes". (I presume udes is a typo for used.)

A typo which was immediately corrected, if you look at my post. Are you really dragging the level of discussion that low Sylas? Was this typo of some great importance?

You don't give much context for that, but no matter. There's no context in which that makes any sense. It is false; they have made no such claim, and you have not shown any such claim. Indirect inferences are not claims, and appear to be founded on various misunderstandings of the data and the processing steps.

My post is quite clear Sylas, somehow, somewhere, a month ago, your car left the rest of the train and went down a different track. When people keep telling you that you're not talking about what they're talking about, you might want to stop and figure out why that is. We were discussing CRU claiming to have not kept their original data. Why you kept posting off topic saying basically "they have the data but can't give it out" had nothing to do with what the rest of us were discussing.

 

[Proton Soup]

Sorry!:

calculated results are not data.

homogenized "data" is not data.

 

[Integral]

Sorry!:

calculated results are not data.

homogenized "data" is not data.

Sure it is. It is the data that is used by the climate models.

 

[Evo]

(quote)Data sets must not be passed on to third parties under any circumstances. Any scientist requiring data which happens to have been supplied already to someone else, even within the same institute or programme of research, must first approach one of the NERC Data Centres, who have agreed to maintain records of data users for UKMO.

Once the project work using the data has been completed, copies of the datasets and software held by the end user should be deleted, unless permission has been obtained for them to be retained for some alternative use.(quote)

So yes Evo, where is the problem? I think you do not understand what is going on between skeptics request for information and the CRU at all.

You misread what you posted. It is saying that if the data is released to someone else to use, that the end user (that requested the data) delete their *copies* of the data, once their work is completed.

 

[Proton Soup]

Sure it is. It is the data that is used by the climate models.

So I wonder which data they use for their models and pretty charts... weird.

it's an intermediate calculation. and they've already admitted they can't reproduce the data requested.

 

[NeoDevin]

LOL. You misread what you posted. It is saying that if the data is released to someone else to use, that the end user (that requested the data) delete their *copies* of the data, once their work is completed.

The CRU got some of the raw data from various other agencies. Once they were finished with whatever particular work it was given for, some of the agreements stipulated that the raw data should not be retained by the CRU. Thus, the CRU is unable to provide this raw data, because they do not possesses it, because the agreements they made when acquiring the data stated that they weren't allowed to keep it (or in some cases, just that they are not allowed to pass it on to anyone else). If you really want the raw data, you need to approach the owners of that data, the same way that the CRU did.

The agreement referenced by sylas and Sorry! was between the CRU and the NERC. The CRU wanted to use the NERC data for their models, but they had to agree not to retain the data afterwards.

Are you intentionally trying to twist everything out of context?

 

[Evo]

If you really want the raw data, you need to approach the owners of that data, the same way that the CRU did.

Have you read CRU's webpage on "Availabilty"? I posted the link in a previous post. They state that they will share the data with other "academics" but not third parties. Read Pielke's blog. Look at what their respose was first to McIntyre, then what their response was to Pielke. Then address what CRU said.

 

[NeoDevin]

it's an intermediate calculation. and they've already admitted they can't reproduce the data requested.

Because the data requested is not theirs to reproduce.

If I want to do some work with someone else's data, then I write to them and request it. If someone wants to follow up and proceeds to request that data from me, I can't provide them the data, because (1) it's not mine to provide, and (2) my agreement may have stipulated that once my calculations are done, the data not be retained. (2) is often done if the group which owns the data intends to publish additional work with the data, they want control over who uses it for what purpose, so that they don't get scooped.

Put another way:

If I've done a bunch of work to produce some data, and I have a number of ideas for further projects to do with that data, then I want to maintain control of that data. If another researcher asks for the data to do something that I'm already intending to do, I would decline their request. If someone asks for the data to (a) reproduce my previous work or (b) do something that I have no intention of doing, then I would (probably) agree to give them the data. However, because I intend to get more publications from work done with the data, I wouldn't want them to be able to reuse it for any other projects they might come up with, or pass it on to other researchers to use for who knows what, because they might do some of the work that I was intending to do, and thus it would be detrimental to my career to provide unrestricted use of the data.

If you want a particular set of data, you must request it from the owner. In this case, that is not the CRU.

 

[NeoDevin]

Have you read CRU's webpage on "Availabilty"? I posted the link in a previous post. They state that they will share the data with other "academics" but not third parties.

Yes I've read it, assuming you're referring to http://www.cru.uea.ac.uk/cru/data/availability/" . If there's a page other than this, then I missed it. Let me know and I will read it as well. It specifically does not say 'they will share the data with other "academics" '.

Since the early 1980s, some NMSs, other organizations and individual scientists have given or sold us (see Hulme, 1994, for a summary of European data collection efforts) additional data for inclusion in the gridded datasets, often on the understanding that the data are only used for academic purposes with the full permission of the NMSs, organizations and scientists and the original station data are not passed onto third parties.

Here "third parties" clearly includes researchers outside of the CRU.

These data are not ours to provide without the full permission of the relevant NMSs, organizations and scientists. We point enquirers to the GHCN web site.

No caveats like "unless the enquirer is a researcher".

They even give one reason why some stations don't want their data released:

The problem is a generic issue and arises from the need of many NMSs to be or aim to be cost neutral (i.e. sell the data to recoup the costs of making observations and preparing the data).

Edit: If I've somehow missed a section of that page which says that they will share it with other researchers, please quote the relevant section and I will apologize.

 

[Evo]

Here "third parties" clearly includes researchers outside of the CRU.

Since the early 1980s, some NMSs, other organizations and individual scientists have given or sold us (see Hulme, 1994, for a summary of European data collection efforts) additional data for inclusion in the gridded datasets, often on the understanding that the data are only used for academic purposes with the full permission of the NMSs, organizations and scientists and the original station data are not passed onto third parties

I would have to disagree based on the letter that CRU sent Pielke. If the specific data he had requested was restricted due to some non-disclosure agreement, then why wouldn't they say this? Why did they instead state that they no longer had the data? Why did CRU claim that they didn't retain data because they didn't have the space?

At least you, NeoDevin, can make concise, to the point posts, I have to read so many posts each day, you have no idea how much I appreciate that. Since my time is not unlimited, I can't wade through verbose posts unless they are reported to me, which they too often are.

Until the CRU formally retracts their statement that they threw away the data, I will have to accept what they say. Anyone making up excuses for them, unless they have been appointed the new CRU spokesperson, is pointless. CRU has made their statement in the press.

Anyway a much more interesting point of discussion, is the coded data. I'll be posting that in a bit.