Coldcall said:
Sylas,
"Once again, with feeling. The raw data is from all over the world, and it is owned by all sorts of different national meteorological bodies with different jurisdictions and commercial interests in the data -- and in many case the organizations are contractually obligated to consider the commercial value of their own data."
Exactly why they should have kept the raw data drom all these disparate sources! They put together a dataset from a wide variety of sources and that process is of fundamental importance to methodology behind their models.
What you are suggesting is that its okay for them to throw away that raw data and make it that much more difficult to reproduce their work in order to validate process, methods involved in setting proxies.
So any scientist who wants to replicate their "experiment" now has to go all request all the same old raw data from all those organisations.
You sure you have the best interest of science in mind? Doesn't sound like it to me.
I think you still don't get it. They can't give you that data. It isn't theirs to give. When all the various data sources were collated, they had the start of what they could work with as a global data set; and it still includes all the individual stations -- including those additional stations for which they can't release the data.
It might be possible to trawl through and pull out the individual records that have restrictions on redistribution; but that's a massive amount of additional work; the data was never set up with the intent of doing that. And why would you bother? What you would end up with is pretty much the GHCN data -- which IS available -- plus a bit extra maybe. You couldn't use that to get a perfect audit of the CRU result; but you could use it to get an independent cross check -- and that has already been done anyway!
What the CRU is trying to do -- and they've been working towards this for a while now -- is get permissions from everyone involved to make the whole collated data available. That requires co-ordination with a whole pile of national bodies that work under different regulations and ideas for the value of their data.
What do you think I should do to show that I support the free exchange of data? And I certainly do. Shall I raise up and army and overthrow other nations and organizations that treat data as a commercial asset? Shall I demand the CRU release everything right now regardless of their existing legal obligations? WHAT?
In the meantime, the big bad CRU when it started out this work long ago went around and obtained formal permission to use a whole pile of data -- the more data the better, right? Now (for shame!) it turns out that you can't get hold of it yourself to audit every last detail of their calculations. It also turns out that some of the individual records were not archived after they were merged into the global set that people actually use.
Some folks seem to think that is terrible. I'm rather "meh" about it, honestly. The data isn't lost; it's still maintained by the original owners of it. It is just copies that were no longer needed. Sure, in the modern day and age it would have been better to keep everything; but that's actually pretty expensive and they probably never even imagined this ludicrous state of affairs now. But suppose they find it again, in closet that has been overlooked. What changes?
NOTHING. You don't actually use that data; you use the merged data. Do you want to audit the process of putting it together? OK; I don't see any great value in that, but there's no harm in it. Once you get over the hurdles of obtaining all the permissions you need! Which will be much much harder with these sets than with the single collated set. But suppose we get those permissions. Given all the other independent calculations around we already know it isn't going to make substantial differences even if any errors are found; but it's still a nice thing to do. But wait! Can you be sure the files in the closet are correct? Why not audit them against the original data? Where does it end? What what difference does it make?
----
Tell you what; you plainly are not convinced and think there's something deeply wrong with the work the CRU has been doing. So forget the CRU. Use the GISS datasets instead. In that case you DO have access to all the data and all the code. Is that going to satisfy you? Surprise me. Here's the ftp site which will take you though the code, the makefiles, the raw data, and the procedures required to put it all together into anomalies.
ftp://data.giss.nasa.gov/pub/gistemp/[/URL][/indent]
Here's another clue for you. This is not just a game. It isn't just some media savvy trick in response to the CRU hack. It's a different independent analysis -- and THAT is far more useful scientifically than an audit. Further more it has been there for some time.
I've used this myself; for some time now. I have not simply compiled their code; I prefer to write my own. (I tried it at first, but I didn't have a suitable fortran compiler available and anyway I wanted the flexibility to try out processes of my own.) I have repeated various parts of the calculations that are of interest to me. Not a complete audit, but a fair bit all the same. I've written a suite of programs of my own to trawl through the raw data and pull out records I want for whatever reason. I haven't done anything much with it for some months now; but at the time I did a repeat calculation -- completely independent with my own programs -- of a regional anomaly around the continental USA, in order to test out some ideas for myself about the alleged problems with certain USHCN stations.
The data is there. Now what do [u]you[/u] propose to do with it?
Cheers -- sylas