Command line interface to Madrigal

Doc home

Madrigal home

Command line interface to Madrigal

Madrigal contains a number of command line applications that allow access to Madrigal data. Some of these require local access to the files on the Madrigal server, some use the remote API and so can be run from anywhere. This section describes the isprint application in detail, which is relevant both to direct users of the isprint application, and to users of the many API isprint methods that use the same arguments as the isprint command line application.

Command line applications:

Local applications (run only on a madrigal server)

• isprint (arguments are used by many isprint API methods)
• printCedarRecords
• summarizeCedarFile
• translateCedarFile

Remote applications (can be run from anywhere - included in either python or Matlab remote API downloads)

• globalIsprint.py (search over multiple Madrigal experiments - python version)
• globalIsprint.m (search over multiple Madrigal experiments - Matlab version)
• madrigalPColor.py - plot PColor plot of data from a given Madrigal file
• madrigaScatter.py - plot scatter plot of data from a given Madrigal file

Isprint

The isprint application:

• Works with any Cedar file format.
• Treats measured and derived parameters in the same way.
• Accepts any parameter (or two parameters added, subtracted, multiplied, or divided) as a filter, with any number of allowed ranges.
• Keeps track of 1D versus 2D data, and only prints one line of data if only 1D parameters are requested.

To use isprint, the user needs to specify four things:

1. The full filename
2. The parameters desired to be displayed (if any)
3. Any filters to limit the amount of data shown (if any)
4. Any non-default formatting options

The full filename

file= path to file (this argument is required)
Example: file=/opt/madrigal/experiments/1998/mlh/20jan98/mlh980120g.001

The parameters desired to be displayed

Any filters to limit the amount of data shown

Time range

date1=mm/dd/yyyy - starting date to be examined. If time1 not given, defaults to 0 UT.
Example: date1=01/20/1998

time1=hh:mm:ss - starting UT time to be examined. If date1 given, is applied to date1. If not, applies on the first day of the experiment.
Example: time1=13:30:00

date2=mm/dd/yyyy - ending date to be examined. If time2 not given, defaults to 0 UT.
Example: date2=01/21/1998

time2=hh:mm:ss - ending UT time to be examined - If date2 not given, refers to date1. If date1 and date2 not given, refers to 1st day.
Example: time2=15:45:00

In the follow arguments ranges are used. If any range value is not given, it may be used to indicate no lower or upper limit (but the comma is always required). Ranges are inclusive of the end points.

Geodetic altitude

z=lower alt limit1, upper alt limit1 [or lower alt limit2 , upper alt limit2 ...] (km)
Example 1: z=100,500 This would limit the geodetic altitude to 100 to 500 km.
Example 2: z=100,200or300,400 This would limit the geodetic altitude to 100 to 200 km or 300 to 400 km.
Example 3: z=,200or300,400 Since the lower limit of the first range is missing, this would limit the geodetic altitude to anything below 200 km or from 300 to 400 km.

Azimuth (from -180 to 180)

az=lower az limit1, upper az limit1 [or lower az limit2 , upper az limit2 ...] (from -180 to 180 degrees)
Example 1: az=100,120 This would limit the azimuth to 100 to 120 degrees.
Example 2: az=-180,-90or90,180 This would limit the azimuth to between -180 and -90 degrees or to between 90 and 180 degrees. Note this allows a filter to go through 180 degrees.

Elevation (from 0 to 90)

el=lower el limit1, upper el limit1 [or lower el limit2 , upper el limit2 ...] (from 0 to 90)
Example 1: el=0,45 This would limit the elevation from 0 to 45 degrees.

Pulse length (in seconds)

plen=lower pl limit1, upper pl limit1 [or lower pl limit2 , upper pl limit2 ...] (pulse len in sec)
Example 1: plen=,5e-4 This would limit the pulse length to 5e-4 seconds or less.

Free form filters

filter=[mnemonic] or [mnemonic1,[+-*/]mnemonic2] , lower limit1 , upper limit1 [or lower limit2 , upper limit2 ...] (any number of filters may be added)
Example: filter=ti,500,1000or2000,3000 Limits the data to points where Ti is between 500 and 1000 degrees or between 2000 and 3000 degrees. Note that the units are always those of the Cedar standard.
Example: filter=gdalt,-,sdwht,0, This filter implies "gdalt - sdwht" must be greater than 0.0. Since sdwht is shadow height (the distance above any point on the earth where the sun is first visible), this filter implies that only data in direct sunlight will be displayed.
Example: filter=ti,/,Dti,100, Limits the data to points where the ratio Ti/dTi is more than 100.

Format options

header=t or f (defaults to header=t, show headers at the beginning of each record)
Example: header=f

badval=bad value string (defaults to "missing")
Example: badval=n/a

assumed=assumed value string (defaults to "assumed")
Example: assumed=-32766

knownbad=known bad value string (defaults to "knownbad")
Example: knownbad=WARNING-BADVALUE

mxchar=maximum characters per line (defaults to no maximum)
Example: mxchar=80

Example: isprint file=/opt/madrigal/experiments/1998/mlh/20jan98/mil980120g.003 date1=01/20/1998 time1=15:00:00 date2=01/20/1998 time2=16:00:00 z=200,300or500,600 badval=noData filter=gdalt,-,sdwht,0, filter=ti,500,1000 uth gdalt gdlat glon ti te

This example would show data from mil980120g.003 between 15 and 16 UT on 01/20/1998 where altitude is either between 200 and 300 km or between 500 and 600 km, and where gdalt-sdwht is greater than 0 (point is in sunlight), and where ti is between 500 and 1000. "noData" would be printed if data was not available.

printCedarRecords

Usage: printCedarRecords filename firstRecord lastRecord

summarizeCedarFile

Usage: summarize CedarFilefilename

translateCedarFile

• 0 - Madrigal
• 1 - Blocked Binary
• 2 - Cbf
• 3 - Unblocked Binary
• 4 - Ascii

The format of the input file is detected automatically.

Usage: translateCedarFile inputFile outputFile outputFileFormat

globalIsprint.py

This script runs a global search through Madrigal data from a given URL. It is installed when you install the Remote python API. Because it is a remote program, it can be run from any machine on the internet, using any operating system. It is a python script.

Usage:

 globalIsprint --url=<Madrigal url> --parms=<Madrigal parms> --output=<output file> \
       --user_fullname=<user fullname> --user_fullname=<user fullname> \
       --user_fullname=<user fullname> [options]

 where:

 --url=<Madrigal url> - url to homepage of site to be searched
       (ie, http://www.haystack.mit.edu/madrigal/)
       This is required.

 --parms=<Madrigal parms> - a comma delimited string listing the desired Madrigal parameters
       in mnemonic form.  (Example: gdalt,dte,te).  Data will be returned
       in the same order as given in this string. See
       http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata and
       choose "Parameter code table" for all possible parameters

 --output=<output file name> - the file name to store the resulting data.

 --user_fullname=<user fullname> - the full user name (probably in quotes unless your name is Sting or Madonna)

 --user_email=<user email>

 --user_affiliation=<user affiliation> - user affiliation.  Use quotes if it contains spaces.

 and options are:

 --startDate=<MM/DD/YYY> - start date to filter experiments before.  Defaults to allow all experiments.

 --endDate=<MM/DD/YYY> - end date to filter experiments after.  Defaults to allow all experiments.

 --inst=<instrument list> - comma separated list of instrument codes or names.  See Madrigal documentation
       using http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata and
       choose "Instrument Table"
       for this list.  Defaults to allow all instruments. If names are given, the
       argument must be enclosed in double quotes.  An asterick will perform matching as
       in glob.  Examples::
   
       --inst=10,30
   
       --inst="Jicamarca IS Radar,Arecibo*"

 --expName  - filter experiments by the experiment name.  Give all or part of the experiment name. Matching
                     is case insensitive and fnmatch characters * and ? are allowed.  Default is no filtering by 
                     experiment name.

 --fileDesc - filter files by their file description string. Give all or part of the file description string. Matching
                     is case insensitive and fnmatch characters * and ? are allowed.  Default is no filtering by 
                     file description.
    
 --kindat=<kind of data list> - comma separated list of kind of data codes.  See Madrigal documentation
       for this list.  Defaults to allow all kinds of data.  If names are given, the
       argument must be enclosed in double quotes.  An asterick will perform matching as
       in glob. Examples::
   
       --kindat=3001,13201
   
       --kindat="INSCAL Basic Derived Parameters,*efwind*,2001"
    
 --filter=<[mnemonic] or [mnemonic1,[+-*/]mnemonic2]>,<lower limit1>,<upper limit1>[or<lower limit2>,<upper limit2>...]
       a filter using any measured or derived Madrigal parameter, or two Madrigal parameters either added,
       subtracted, multiplied or divided.  Each filter has one or more allowed ranges.  The filter accepts
       data that is in any allowed range.  If the Madrigal parameter value is missing, the filter will always
       reject that data.  Multiple filter arguments are allowed on the command line.  To skip either a lower
       limit or an upper limit, leave it blank.  Examples::
    
      filter=ti,500,1000  (Accept when 500 <= Ti <= 1000)
    
      filter=gdalt,-,sdwht,0,  (Accept when gdalt > shadowheight - that is, point in direct sunlight)
    
      filter=gdalt,200,300or1000,1200 (Accept when 200 <= gdalt <= 300 OR 1000 <= gdalt <= 1200)
    
 --seasonalStartDate=<MM/DD> - seasonal start date to filter experiments before.  Use this to select only part of the
       year to collect data.  Defaults to Jan 1.  Example:  --seasonalStartDate=07/01 would only allow
       experiments after July 1st from each year.
    
 
       --seasonalEndDate=<MM/DD> - seasonal end date to filter experiments after.  Use this to select only part of the
       year to collect data.  Defaults to Dec 31.  Example:  --seasonalEndDate=10/31 would only allow
       experiments before Oct 31 of each year.
    
 --showFiles - if given, show file names.  Default is to not show file names.
    
 --showSummary - if given, summarize all arguments at the beginning.  Default is to not show summary.
    
 --includeNonDefault - if given, include all files, including history.  Default is to search only default files.
    
 --missing=<missing string> (defaults to "missing")
    
 --assumed=<assumed string> (defaults to "assumed")
    
 --knownbad=<knownbad string> (defaults to "knownbad")
    
 --verbose - if given, print each file processed info to stdout.  Default is to run silently.
    
   Example:
   
        globalIsprint.py --url=http://www.haystack.mit.edu/madrigal/ --parms='uth,gdalt,ti' --output=test.dat 
	    --startDate=01/19/1998 --endDate=01/21/1998 --inst="Millstone*" --verbose --user_fullname="Bill Rideout" 
	    --user_email=brideout@haystack.mit.edu  --user_affiliation=MIT
    
    
globalIsprint.m
    
This script runs a global search through Madrigal data from a given URL. It is installed when you install the Remote Matlab API. Because it is a remote program, it can be run from any machine on the internet, using any operating system. It is a Matlab script. Because it uses the Matlab method urlread, which has an upper limit to the amount of data it can read in one call, it is not as robust as globalIsprint.py. It is recommended that globalIsprint.py be used instead if at all possible. 
    
function [] = globalIsprint(url, ...
             parms, ...
             output, ...
             user_fullname, ...
             user_email, ...
             user_affiliation, ...
             startTime, ...
             endTime, ...
             inst)
% globalIsprint is a script to search through the entire Madrigal database
% for appropriate data to print in ascii to a file
%
%    Inputs:
%
%        url - url to homepage of site to be searched (Example: 
%              'http://www.haystack.mit.edu/madrigal/'
%
%        parms - a comma delimited string listing the desired Madrigal 
%            parameters in mnemonic form.  
%            (Example: 'year,month,day,hour,min,sec,gdalt,dte,te').  
%            Ascii space-separated ata will be returned in the same  
%            order as given in this string. See 
%            http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata
%            "Parameter code table" for all possible parameters.
%
%        output - the local file name to store the resulting ascii data.
%                 (Example: '/tmp/isprint.txt')
%
%        user_fullname - the full user name (Example: 'Bill Rideout')
%
%        user email -  Example: 'brideout@haystack.mit.edu'
%
%        user_affiliation - Example: 'MIT'
%
%        startTime - a Matlab time to begin search at. Example:
%                    datenum('20-Jan-1998 00:00:00') Time in UT
%
%        endTime - a Matlab time to end search at. Example:
%                  datenum('21-Jan-1998 23:59:59') Time in UT
%
%        inst - instrument code (integer).  See 
%            http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata
%            "Instrument Table" for this list. Examples: 30 for Millstone
%            Hill Incoherent Scatter Radar, 80 for Sondrestrom Incoherent 
%            Scatter Radar
%
%    Optional inputs
%
%        filters - is the optional filters requested in exactly the form given in isprint
%         command line (example = 'filter=gdalt,,500 filter=ti,500,1000')
%         See:  http://www.haystack.mit.edu/madrigal/ug_commandLine.html for details
%
%        kindats - is an optional array of kindat (kinds of data) codes to accept.
%           The default is an empty array, which will accept all kindats.
%  
%        expName - a case insensitive regular expression that matches the experiment
%           name.  Default is zero-length string, which matches all experiment names.
%
%        fileDesc - a case insensitive regular expression that matches the file description.
%           Default is zero-length string, which matches all file descriptions.
%    Returns: Nothing.
%
%    Affects: Writes results to output file
%
%        
%
%  Example: globalIsprint('http://www.haystack.mit.edu/madrigal/', ...
%                         'year,month,day,hour,min,sec,gdalt,dte,te', ...
%                         '/tmp/isprint.txt', ...
%                         'Bill Rideout', ...
%                         'brideout@haystack.mit.edu', ...
%                         'MIT', ...
%                         datenum('20-Jan-1998 00:00:00'), ...
%                         datenum('21-Jan-1998 23:59:59'), ...
%                         30);
%
  
madrigalPColor.py
    
madrigalPColor.py is a command-line application  that creates PColor plots from Madrigal. For example, it could be used to plot electron density versus altitude and time for a given Madrigal experiment file. 
    
This application can be run from anywhere, and is included in the python Madrigal remote API download. Use of this module requires matplotlib (matplotlib.sourceforge.net).
    
Usage:
    
python madrigalPColor.py  --url=<url> --file=<file> --parm=<parm>
       --output=<output> --name=<name> --email=<email> --affiliation=<affiliation>
     [--filter=<filter> --title=<title> --startHour=startHour --endHour=endHour
    --minAlt=minimum_altitude --maxAlt=maximum_altitude --minParm=minimum_parm_value 
    --maxParm=maximum_parameter_value]
    
See isprint for details of how filters work.
    
Example:
    
python madrigalPColor.py  --url=http://madrigal.haystack.mit.edu/madrigal \
       --file=/opt/madrigal/experiments/1998/mlh/20jan98/mlh980120g.002 --parm=nel \
       --output=/tmp/mlh_20jan98.png --name="Bill Rideout" --email=brideout@haystack.mit.edu \
       --affiliation=MIT --filter="filter=elm,80,90 filter=gdalt,,500"
    
madrigalScatter.py
madrigalScatter.py is a command-line application  that creates scatter plots from Madrigal. For example, it could be used to Kp versus  time  for a given Madrigal experiment file.
This application can be run from anywhere, and is included in the python Madrigal remote API download. Use of this module requires matplotlib (matplotlib.sourceforge.net).
Usage:
python madrigalScatter.py  --url=<url> --file=<file> --parm=<parm>
   --output=<output> --name=<name> --email=<email> --affiliation=<affiliation>
   [--filter=<filter> --title=<title> --startHour=startHour --endHour=endHour
    --yMin=y_minimum --yMax=y_maximum]
See isprint for details of how filters work.
Example:
python madrigalScatter.py  --url=http://madrigal.haystack.mit.edu/madrigal \
   --file=/opt/madrigal/experiments/1998/mlh/20jan98/mlh980120g.002 --parm=systmp \
   --output=/tmp/mlh_20jan98.png --name="Bill Rideout" --email=brideout@haystack.mit.edu \
   --affiliation=MIT --filter="filter=elm,80,90"




  

    

    

    

    
Command line interface to Madrigal
    
Doc home 
    
Madrigal home
  


Previous:
Plot data from instruments 
  Up:
Madrigal user's guide 
  Next:
Remote access programming tutorial toc