Having a difficult time with binary IEEE .dat file

[mesa]

I have been attempting to open up files on NASA’s GRS and neutron data,

http://pds-geosciences.wustl.edu/missions/messenger/grs_edr.html

but have been running into some roadblocks. The data is all stored in .dat IEEE binary format (which IEEE format they do not specify in their documentation as far as I can see), and the conversion software they have available hasn’t been updated since 2013.

Attempting to install their conversion software on Ubuntu 18.04 and Windows 10 did not work at all. I had an old laptop laying around that still had Ubuntu 14.04, and it at least let me open the program, but not do anything else.

I had one of my computational scientists work on just writing code to crack this binary format, and he concluded it was likely going to be too difficult to be 'successful' (a bit out of his field), and he wondered why anyone would format the files like this in the first place.

Any help would be appreciated!

 

[Mark44]

but have been running into some roadblocks. The data is all stored in .dat IEEE binary format (which IEEE format they do not specify in their documentation as far as I can see), and the conversion software they have available hasn’t been updated since 2013.

From poking around in the files in the link you gave, it doesn't seem all that difficult to extract the data. A modestly capable programmer in any programming language should be able to cobble together a program to read the data, and display it in some desired format.

I looked in the february directory shown in the link you provided. Here are the first three files listed:
5/7/2015 9:33 AM 92160 grs_adc2015032zzz.dat
5/7/2015 9:33 AM 1950 grs_adc2015032zzz.lbl
6/8/2018 5:50 PM 19961 grs_adc2015032zzz.xml

The binary data is in the file with the .dat file extension.
The file with extension .lbl gives some general information about the .dat file

PDS_VERSION_ID                 = "PDS3"

/*** FILE FORMAT ***/
FILE_RECORDS                   = 1440
RECORD_TYPE                    = FIXED_LENGTH
RECORD_BYTES                   = 64

/*** GENERAL DATA DESCRIPTION PARAMETERS ***/
PRODUCT_ID                     = "GRS_ADC2015032ZZZ_DAT"
PRODUCT_VERSION_ID             = "V1"
PRODUCT_CREATION_TIME          = 2015-02-02T21:33:30
PRODUCT_TYPE                   = "ANCILLARY"
STANDARD_DATA_PRODUCT_ID       = "GRS_FPGA"
SOFTWARE_NAME                  = "PIPE-GRS2EDR"
SOFTWARE_VERSION_ID            = "1.0"
INSTRUMENT_HOST_NAME           = "MESSENGER"
INSTRUMENT_NAME                = "GAMMA RAY SPECTROMETER"
INSTRUMENT_ID                  = "GRS"
DATA_SET_ID                    = "MESS-E/V/H-GRNS-2-GRS-RAWDATA-V1.0"
MISSION_PHASE_NAME             = "MERCURY ORBIT YEAR 4"
TARGET_NAME                    = "MERCURY"
START_TIME                     = 2015-02-01T00:00:38.000
STOP_TIME                      = 2015-02-01T23:59:38.000
SPACECRAFT_CLOCK_START_COUNT   = "2/65072834"
SPACECRAFT_CLOCK_STOP_COUNT    = "2/65159174"
^TABLE                         = "GRS_ADC2015032ZZZ.DAT"

/* FPGA ADC DIAGNOSTIC INFORMATION, TABLE DEFINITION */
OBJECT                         = TABLE
COLUMNS                        = 31
INTERCHANGE_FORMAT             = BINARY
ROWS                           = 1440
ROW_BYTES                      = 64

DESCRIPTION                    = "
   This table contains one set of FPGA ADC diagnostic information gathered by
   the MESSENGER Gamma-Ray Spectrometer (GRS). A set is defined as all data
   with MET timestamps corresponding to a given Earth day.
   The complete column definitions are contained in an external file found
   in the LABEL directory of the archive volume. Additional details are
   contained in the EDR SIS document.
"
^STRUCTURE                     = "GRS_FPGA.FMT"
END_OBJECT                     = TABLE
END

The file with extension .xml is an XML file that gives specific information about all of the fields in the file, including their byte offset within the file, the length of the field, the type of data, and what kind of information is in the file.
Here's a small snippet from the xml file:

<description> The 2-byte integer contains the following bit fields: Bits - Name - Description 1 - ch1_latchup - ADC Channel 1 latchup. =0 no latchup. =1 latchup 2 - ch0_latchup - ADC Channel 0 latchup. =0 no latchup. =1 latchup 3 - pwr_ch1_inverted_adc - Channel 1 inverted power. =0 not inverted, =1 inverted 4 - pwr_ch0_inverted_adc - Channel 0 inverted power. =0 not inverted, =1 inverted 8 - fpga_reset_adc - Always reads 0. Write a 1 to reset FPGA 9 - adc_ch2_clock - ADC Channel 2 clock. =0 off, =1 on 10 - adc_ch1_clock - ADC Channel 1 clock. =0 off, =1 on 11 - adc_ch0_clock - ADC Channel 0 clock. =0 off, =1 on 12 - ch1_2_adc_power - Channel 1,2 ADC power. =0 power on, =1 power off 13 - ch1_reset_latchup - Channel 1,2 reset latchup. =0 normal, =1 reset latchup 14 - ch0_adc_power - Channel 0 ADC Power, =0 power on, =1 power off 15 - ch0_reset_latchup - Channel 0 reset latchup, =0 normal, =1 reset latchup 16 - power_sleep_adc - =0 normal operation, =1 sleep mode </description>

I had one of my computational scientists work on just writing code to crack this binary format, and he concluded it was likely going to be too difficult to be 'successful' (a bit out of his field), and he wondered why anyone would format the files like this in the first place.

It seems to me that all the information needed to "crack this binary format" is in plain view. Perhaps you should look for a computational scientist who is at least as compentent as a first-year computer science student, to write a program in C, C++, Fortran, Python, or whatever.

 

[mesa]

Nice to see Physicsforums hasn't lost its charm, lol.

Thanks for the info!

 

[Mark44]

Nice to see Physicsforums hasn't lost its charm, lol.

My sense is that whoever you talked to was probably competent to pull the data out, but was just blowing you off with a lame excuse that it would be "too difficult," or other technobabble. I suspect the real reason was that he/she didn't want to invest any time looking into the problem.

 

[mesa]

It seems to me that all the information needed to "crack this binary format" is in plain view. Perhaps you should look for a computational scientist who is at least as compentent as a first-year computer science student, to write a program in C, C++, Fortran, Python, or whatever.

My sense is that whoever you talked to was probably competent to pull the data out, but was just blowing you off with a lame excuse that it would be "too difficult," or other technobabble. I suspect the real reason was that he/she didn't want to invest any time looking into the problem.

He is so fired, although he did get further than Mark44 did ;)

 

[Mark44]

He is so fired, although he did get further than Mark44 did ;)

But then, I am not getting paid to write code to display the data in those files.

 

[sysprog]

Nice to see Physicsforums hasn't lost its charm, lol.

Thanks for the info!

There's a wealth of information available at https://pds.nasa.gov

 

[mesa]

But then, I am not getting paid to write code to display the data in those files.

Good point, pm sent.