Contoversy about NASA's Asteroid Tracking

[BillTre]

Nathan Myhrvold has raised questions about the accuracy of data NASA uses to track asteroids that may impact the earth.
Myhrvold is a former Microsoft chief technologist and a guy with many diverse interests.
Specifically he has questioned the size information and how it was derived.
This concerns the Neowise project which was derived from the Wide-field Infrared Survey Explorer (WISE) project.
It also has implications for the proposed Neocam (short for Near-Earth Object Camera) satellite.
This has apparently been an argument for about 2 years.
He has now published a peer reviewed paper on it (in Icarus) and might soon publish another.

NY Times story here.

 

[DrClaude]

The original article can be found here (it is open access):

An empirical examination of WISE/NEOWISE asteroid analysis and results

 

[f95toli]

Nature has also reported on this
https://www.nature.com/articles/d41586-018-05446-w

 

[f95toli]

Some quick googling also found this article on Retraction Watch...

https://retractionwatch.com/2018/06...lic-records-request-for-nasa-code/#more-66117

Note that this is -obviously- only one side of the story; but I can't help but wonder if there isn't something funny going on? Not so much about the science but in the way NASA has handled this.

 

[sophiecentaur]

the way NASA has handled this.

Nasa needs loads of public money and people are more prepared to fund organisations that are seen as 'right' and able to defend us from flying debris. Nasa is very political (at the top at least) and they will argue black is white if it would mean getting or not getting funding.

 

[Vanadium 50]

On the article: it's not very good. The problem being addressed here is that when you take a picture of an asteroid, you know how bright it is, and what you want to know is how big it is. The WISE/NeoWISE analysis group has a model which takes you from one to the other, and Myhrvold thinks he has a better one. The way a normal scientific paper would argue is that it would take a sample of asteroids where we know the brightness and the size - the latter from occultations or radar - and show that the proponent's model does better than the WISE/NeoWISE analysis group's model on those asteroids where we know the answer.

This paper, on the other hand, is more about perceived errors that the WISE/NeoWISE analysis groups might or might not have been made, rather than whether their model works or not. After all, if the model works, the model works. I don't think anyone other than Myhrvold worries that Asteroid A was in Category X in one instance of this analysis and Category Y in another. It's no better and no worse than deciding something is "light red" and later calling it "pink". He also makes some truly bizarre claims like a model has to get every single result exactly right, even when we know there are measurement uncertainties.

I can't say if Myhrvold is right or not, but I can say he doesn't make his case very well.

Furthermore, he has made suggestions - not in the paper - that the WISE/NeoWISE analysis group might be engaging in scientific misconduct in order to attract more funding. He then complains that they are not treating him as a scientific peer. Is anyone surprised by this?

On NASA behavior: first, the scientific collaboration is not part of NASA. A few members work for NASA, some work for universities under NASA funding, and some work for universities without any NASA funding. So pestering NASA is unlikely to get the results Myhrvold wants, because NASA is not the actor he's looking for. They only own the instrument. Next, an FOIA request is for an agency record. An agency record is a very specific thing, and communications between non-NASA employees at universities is not one, even if the conversation is about a NASA mission.

 

[Ken G]

Excellent point. Popular press tends to imagine that every press release sanctioned by NASA comes from NASA, and that NASA had some influence over the scientific result. That is usually not true, science is conducted independently of NASA, even when NASA puts its name on the picture because it was taken with one of its satellites. It is true that human nature is not to bite the hand that feeds you, and scientists must propose to NASA for time and money, but nevertheless the scientists must take responsibility for, and base their reputation on, the veracity of their own results. This situation is quite a bit different for people working for Microsoft, for example, so the latter camp are inclined to apply the model of their own experience even when it does not necessarily apply well.

 

[stefan r]

What was the argument against Neocam? If we cannot trust the asteroid data from WISE then I would think that argues in favor of launching a dedicated camera.

The Times article says they were arguing about the size of the asteroids. It seams to me that the direction that asteroids move is most important for planet defense. It is probably best to leave town or get in a bunker. "pffft its only a 10 kiloton blast. I don't move from couch for anything less than 2 megaton". An explosion leveling homes in a 1 mile radius or 5 mile radius will not matter if you can evacuate the day before. Once people get through the traffic jamb they can easily continue driving. Telling people to evacuate and then sending them the toward the impact site would be a fail. A more detailed and precise calculation of size can be determined later. If NASA announces that an object is going to hit there will by thousands of telescopes looking at that object. The error in impact location is going to be much higher than the blast radius.

 

[sophiecentaur]

The error in impact location is going to be much higher than the blast radius.

On top of the calculated error, you would have to at least double the radius for it to be a politically acceptable risk for Mr Average. That could mean clearing out a total city of its population. Except in war time we have never seen anything like it.
Pretty scary. Hope it doesn't happen till my great great grandchildren's time when we could possibly deal with it all.

 

[mfb]

"pffft its only a 10 kiloton blast. I don't move from couch for anything less than 2 megaton".

There is a threshold where evacuations cause more damage than an impact itself. You don't want to evacuate below that threshold.

The size and albedo estimates are also important to predict the future orbit.

 

[CWatters]

One of things Myhrvold says he wants is for JPL researchers to publish the techniques and software they used in their analysis. Is that a reasonable request? I don't know the answer but similar requests have been made by people wishing to scrutinise climate change data and they claim their requests have also been resisted. On the one hand scrutiny is important but on the other hand so is public confidence in scientific results that can have enormous implications for our future. There are claims some people only want access to scientific data so they can hunt for and publicise trivial errors to cast unwarranted doubt on the results in an attempt to change public opinion or government policy. Where does the balance lie? Should access be restricted? Should it all be published?

 

[pat65]

Yes, the ability of NASA to determine the size of an incoming meteoroid is limited. There is not only error on size but meteoroid count. I have looked into significant meteorite impact frequency and I have found that kiloton to megaton impacts occur at a higher frequency then has previously been published. The discrepancy here is (among other factors) in confirmed vs. unconfirmed impacts.

Confirmed:
1908, Tunguska, Russia: 10-15 megatons
1932, Spain: 190 kilotons
1963, Southern Ocean, South Africa: 176-356 kilotons
2013, Chelyabinsk, Russia: 440 kilotons

Unconfirmed:
1930, Brazil, Curuçá River: likely >= 1 megaton
1935, British Guyana (Guiana): likely >= 1 megaton
1976, New Guinea, Irian Jaya: likely >= 1 megaton

If you take this data at face value then there is one multi-kiloton to multi-megaton impact every 13.75 years. I realize that I am somewhat off topic here but I was quite alarmed when I looked into this and found the frequency of impacts large enough to do quite a bit of damage to be a fair bit more then what NASA had been indicating.

I wonder how much instrument and technique error plays into the estimates of potentially dangerous impacts here on Earth?

Edit: This post has been significantly modified from its original form as it was just too off topic.

 

[stefan r]

...There are claims some people only want access to scientific data so they can hunt for and publicise trivial errors to cast unwarranted doubt on the results in an attempt to change public opinion or government policy. ...

If you do not tell me the margin of error I do not believe your results. Talking to welders and carpenters I ran into this issue. When welders tell you to "cut it exact" they mean inside the width of a soap stone. Carpenters carry wide pencils. If you take out a microscope and point out the cell structure wood a carpenter will not refund your money. It is still a 90 degree angle when the corners are bevel, camfer, or fillet. That does not change the nature of a right angle. If you want a piece manufactured without a camfer you should probably specify that on the blueprint.

If someone claims that they have never made a mistake I do not trust them. I can be highly confident that they are either being dishonest with me or being dishonest with themselves. Some people will be convinced by anything that confirms what they want to believe. When it is obvious that someone fed you cherry picked data most people recognize that they cannot trust that source in the future. It is important that we know NASA is not cherry picking their own data.

 

[anorlunda]

unwarranted doubt

Who you trust with the power to decide warranted versus unwarranted is very much a matter of political philosophy.

 

[f95toli]

One of things Myhrvold says he wants is for JPL researchers to publish the techniques and software they used
in their analysis. Is that a reasonable request?

The techniques? Certainly, it would even be a requirement if you wanted to publish in certain journals (e.g. Nature).
Publishing the software? That depends, you should certainly be able to show which algorithm you used and a "reasonable" request for the software should at least be considered. Many "software heavy" research projects (and this would count as one of those) do publish their software these days. and if the development of the software was paid for by the project it is often a requirement.
None of the things he is asking for are -as far as I can tell- "unreasonable" and if the research was funded by .e.g. the EU H2020 program there would even be a requirement to to make most of it publicly available anyway. The fact that there are multiple organizations involved would not matter since any request would be to the collaboration as a whole and how the partners handles data etc would be covered by a signed collaboration agreements (as well as the Data Management Plan, DMP)

Things are obviously different in the US but the fact that his FOI requests have been first denied and then granted on a couple of occasions would suggest that someone is being silly. I very much doubt that there are any issues with the science itself and I don't believe there is a conspiracy, it is -in my experience- more likely that someone in the management chain has e.g. decided that there is some IP that they want to protect and/or they simply do not understand the rules.

 

[Vanadium 50]

Things are obviously different in the US but the fact that his FOI requests have been first denied and then granted on a couple of occasions would suggest that someone is being silly.

I would argue that it is Myhrvold. NASA does not own - and likely does not even have - the information he is asking for. His lawyers should know this. Furthermore, the data is public. The code may not be, but the algorithm surely is. Given that, why ask for something he doesn't really need from someone who doesn't have it?

 

[CWatters]

Isn't he claiming it's not possible to reproduce their results based on the info they have made available?

 

[Vanadium 50]

Myhrvold is arguing he can not reproduce the chain of reasoning that led to the model without access to the intermediate steps. But the model stands or falls on its own.

His plan seems to be to point out that at some step in the process, a functional form was chosen based on some physical argument, but when the actual fitting is done, the parameters have vales that differ from measurements or even non-physical values, and will let out a rousing "A ha!". Problem is this happens all the time when fitting complicated models, which is why people compare the model to the data and not the model's pedigree to the data.

As a simplified example, suppose I have a calorimeter with three samples. In principle, E = E1+E2+E3, but in fact there are three different calibrations, so E = aE1 + bE2 + cE3. It may well be that one of those coefficients is negative, even though its physically impossible. I've had this happen; it's no big deal. No "A ha!" required.

 

[stefan r]

... three samples. In principle, E = E1+E2+E3, but in fact there are three different calibrations, so E = aE1 + bE2 + cE3...

If your error is 10% in a, and 10% in b, and 10% in c then your reported error for a+b+c should not also be 10%.

 

[Vanadium 50]

Never said it was.