OpenGGCM Wiki | Main / SCData

Processed s/c data from CDAWEB and other sources

NOTE: this probably still works but is no longer supported. There is a script on the Scripts page that does the same thing.

There is a web interface for processed s/c data for OpenGGCM input and data comparisons. The data are all formatted into a standard ascii format, where each record holds:

  YY MO DA HO MI SE value UNIX_time
  with:
  YY: year, MO: month, DA: day, HO: hour, SE: second, value: data value, and UNIX_time: UNIX epoch time (seconds from 1970 Jan 1, 0000 UT)

Each variable comes in it's own file. Most sets begin in 2000, or whenever data became available, and most sets run through the end of 2018. DMSP data run from 2010 to 2014 inclusive, but we hope more will be added. The data are accessible through a web interface in averaged format. Available averages are 15, 60, 300, 900, and 3600 seconds. DMSP data also as 2 second averages. Missing data are filled with -1.0e30 or can be chosen to be left empty. The variable names should be self explanatory. However, we also provide the cdf headers where data come from cdfs, so full description meta data are available.

TODO: add DMSP drift (not in CDFs), add newer DMSP (waiting), add DISCOVR SW/IMF, add LANL (available?), add some more MMS variables, add E/PA-spectra, add interpolate option, add overview plots, add option for composite values (pdyn, beta, magnitudes, etc.).

Web access:

  USAGE:
  http://squirrel.sr.unh.edu/~jraeder/cgi-bin/get-data.cgi?var=help                 This page
  http://squirrel.sr.unh.edu/~jraeder/cgi-bin/get-data.cgi?var=vlist                list of available variables, processed and averaged
  http://squirrel.sr.unh.edu/~jraeder/cgi-bin/get-data.cgi?var=VARNAME&t1=STARTTIME&t2=ENDDIME&ave=AVE&gaps=GAPS
     where VARNAME is a variable from the list, t1,t2 are start/end times in the format year:month:day:hour:minute:second
     AVE is the averaging interval (15 60 300 900 3600) in seconds, for DMSP also "2"
     GAPS open: data left out, filled: gaps are filled with -1.0e31

Examples:

  curl  'http://squirrel.sr.unh.edu/~jraeder/cgi-bin/get-data.cgi?var=ace_Np&ave=300&t1=2012:03:03:00:00:0.0&t2=2012:03:26:00:00:0.0' > data.ace_Np

will fetch 300 seconds averaged ACE proton densities for the given interval. Although one can use the URL in a web browser, there is no formatting, so the browser will display just one long string. If there are no data available, nothing will be returned. The 3600 (hourly) averages are quite useful to see what data are available within, say a year.

Access to raw data

All availabe raw data that were dumped from CDFs or other sources are concatenated and save gzipped here: http://squirrel.sr.unh.edu/~jraeder/data.allcat/ Note that these are sometimes very large files!

Access to meta data

For each variable derived from SPDF cdf files, the meta data (ascii dump) can be found here: http://squirrel.sr.unh.edu/~jraeder/data.meta/

The file http://squirrel.sr.unh.edu/~jraeder/data.meta/CDF_map shows the mapping between cdf file name, the specific variable name within the cdf, the time variable in the cdf, and the suffix given in our data base.

GOES data are from https://satdat.ngdc.noaa.gov/sem/goes/data/full/, were also meta data information can be found. GOES data are reformatted from 0.5 second .csv files.

OMNI data are from ftp://spdf.gsfc.nasa.gov/pub/data/omni/high_res_omni/

Some of the orbit data are from the s/c cdfs. However, some PIs choose not to include ephemeris data in the cdfs. Definitive orbit data can he had at sscweb.nasa.gov, however, they have a very clumsy interface. I used their SOAP access from http://sscweb.gsfc.nasa.gov/WS/ssc/2/SatelliteSituationCenterService?WSDL but got complaints that I was overloading their servers. But now that I have 2 decades of data updates should be OK.

Overview plots

http://squirrel.sr.unh.edu/~jraeder/data.plots/ currently has yearly plots of 1h averages. These plots are useful to see what data are available.