Previous: Creating Madrigal data files   Up: Madrigal admin guide   Next: Other admin tasks

Creating and modifying Madrigal experiments

Adding data to Madrigal involves two steps. The first step is creating Madrigal data files, as discussed in the previous section. The next is installing them on the Madrigal database, with the appropriate metadata and any additional plots or other information you want to supply.

There are a number of scripts supplied with Madrigal to simplify the task of creating or modifying experiments on Madrigal. Use these scripts if you want to:

After changing a Madrigal experiment using one or more of the scripts above, the script madroot/bin/updateMaster must be run for the changes to take effect. That's because these scripts change the metadata files in the local directory, and updateMaster combines all the local metadata files into the ones in madroot/metadata, which the used by the user interface to Madrigal.

This page also describes how you can add auxiliary information to a given Madrigal experiment.

Create an experiment using a pre-existing Madrigal file

If you have already created a Madrigal file for a given experiment, and want to add that new experiment to Madrigal, use the script createExpWithFile.py, located in madroot/bin. Usage:

createExpWithFile.py is a script used to create a new Madrigal experiment
   based on an already existing file.  Information such as the duration of the
   experiment is obtained by analyzing the file.
Required arguments:
 --madFilename - full path to the complete Madrigal file. Basename will
   be maintained.
 --expTitle - experiment title. Use quotes if title contains spaces.
 
   --permission - 0 for public, 1 for private (restricted to certain IP range)
   (both the experiment and the file will set)
 --fileDesc - file decription
Optional arguments:
 --instCode - instrument code. If this argument missing, instrument code is
   taken from file, but error is thrown if more than one kinst found.
 --category - 1=default, 2=variant, 3=history, or 4=realtime.   If this argument is missing,
   1 (default) used.
 --dirName  - directory name to use for experiment.  If not given, the directory
   name will be the default name DDmmmYY[optChar].  Cannot contain "/"
 --optChar  - optional character to be added to experiment directory if no dirName
   given.  If dirName argument given, this argument ignored.  optChar
   is used if the default directory name DDmmmYY is used for
   more than one experiment created for a given instrument on a given day.
   For example, if --optChar=h for a MLH experiment on September 12, 2005,
   then the experiment directory created would be experiments/2005/mlh/12sep05h.
 

Example: If you have already created the Madrigal file mlh060120g.001 in the /tmp directory, and the experiment name is Calibration, the file should be public, and the description is "Final", you would enter:

/opt/madrigal/bin/createExpWithFile.py --madFilename=/tmp/mlh060120g.001 \
    --expTitle="Calibration" --permission=0 --fileDesc="Final"

Create an experiment when no Madrigal file yet exists

If you want to create an experiment for a Madrigal data yet to be created (such as when you want to create the Madrigal file in real-time), use the script createRTExp.py, located in madroot/bin. Usage:

createRTExpWithFile.py is a script used to create a new Madrigal experiment
   that will contain real-time files.  These real-time files are assumed not to exist
   yet.
Required arguments::
 --startDate - experiment start date in form YYYY-MM-DD
 --inst - instrument code or 3-letter Madrigal mnenonic
 --expTitle - experiment title. Use quotes if title contains spaces.
     
   --rtFiles - comma-separated list of realtime file basenames to be created
 --kindats - comma-separated list of ints or single int of kindats for each realtime file.
   The length and order must be the same as rtFiles.  If only one
   given, it is assumed that all rtFiles have the same kindat.
 --fileDescs - comma-separated list of file descriptions. If the file description contains spaces,
   quotes must be used.
Optional argument::
 --numDays - number of days the experiment is estimated to run - if not given, and no start
   and end times specified, defaults to one day.  Error raised if endDate and endTime
   also specified.
 --startTime - start time in form HH:MM:DD.  Defaults to 00:00:00
 --endDate - end day in form YYYY-MM-DD. endTime must also be specified
 --endTime - end time in form HH:MM:DD.  endDate must also be specified
     
   --permissions - comma-separated list of 0 for public, 1 for private (restricted to certain IP range).
   If only one given, it is assumed it applied to all.  If this argument is not
   given, it defaults to 0 (public)
 --dirName  - directory name to use for experiment.  If not given, the directory
   name will be the default name DDmmmYY[optChar].  Cannot contain "/"
 --optChar  - optional character to be added to experiment directory if no dirName
   given.  If dirName argument given, this argument ignored.  optChar
   is used if the default directory name DDmmmYY is used for
   more than one experiment created for a given instrument on a given day.
   For example, if --optChar=h for a MLH experiment on September 12, 2005,
   then the experiment directory created would be experiments/2005/mlh/12sep05h.
 --security - overall experiment access.  0 for public, 1 for private, -1 for ignore.
 Defaults to public (0)

Example: If you plan to create two real-time Madrigal files called mlh021001a.000 and mlh021001b.000 for the Millstone Hill Radar for an experiment planned to run 2 days, with both files having the kindat 3410, you would enter:

/opt/madrigal/bin/createRTExp.py --startDate=2002-10-01 --numDays=2 --inst=mlh --expTitle="test experiment" \
    --rtFiles=mlh021001a.000,mlh021001b.000 --kindats=3410 \
    --fileDescs="preliminary - single pulse,preliminary - alternating code"

Modify the status of a given experiment

All of the experiment attributes can be changed by the script changeExpStatus.py, located in madroot/bin. The most common reason you'd want to run this script is to change an experiment's security, with the options being 0 (public), 1 (limited by IP address), and -1 (ignored, or hidden from everyone). To completely remove an experiment, simply delete the directory. However, setting security to -1 allows you to bring the experiment back at some later time by running changeExpStatus.py again.

A number of other attributes can also be modified, but most are set automatically, and should not need modification.

Usage:

changeExpStatus.py is a script used to change the status of an
 existing Madrigal experiment.  The following attributes can be changed:
 expUrl
   experiment name
   siteID
   start date
   start time
   end date
   end time
   instrument code
   security (public, private. ignore)
Required argument: 
 --expDir - full path to experiment directory. Example:
   "/opt/madrigal/experiments/1998/mlh/20jan98"
Optional arguments - set these to change an experiment attribute:
     
   --expUrl - must be in form <cgi base>/madtoc/YYYY/<3 letter lower case inst code>/<expDir>
   example: http://www.haystack.mit.edu/cgi-bin/madtoc/1997/mlh/03dec97g
 --expName - experiment name.  Quotes required if contains spaces.  Example: "World Day"
 --siteID - Madrigal siteID of where data will be stored.  Error raised if not the siteID
   of the local Madrigal site. Example: 4
 --startDate - new start date of experiment (UT).  In form YYYY-MM-DD.  Example: 1998-01-20
 --startTime - new start time of experiment (UT).  In form HH:MM:DD.  Example: 12:30:00
 --endDate - new end date of experiment (UT).  In form YYYY-MM-DD.  Example: 1998-01-21
 --endTime - new end time of experiment (UT).  In form HH:MM:DD.  Example: 23:30:00
 --inst - new instrument code.  Example: 30
 --security - new security code.  Allowed values are 0 for public, 1 for private (limited IP range access)
                -1 for ignore, 2 for archived experiment, 3 for private (limited IP range access) archived
                experiment.

Example: to change to experiment /opt/madrigal/experiments/2006/mlh/20jan to be private, you would run:

/opt/madrigal/bin/changeExpStatus.py --expDir=/opt/madrigal/experiments/2006/mlh/20jan06 --security=1

Add a new file to an experiment

If you want to add a new file to an existing Madrigal experiment, use addFileToExp.py, located in madroot/bin. You can use this script to add a completely new file, or to update an existing one. It you update an existing file and want to change the status of the older file to history or variant, use the script changeFileStatus.py.

Usage:

addFileToExp.py is a script used to add a new Madrigal file to an
existing experiment.  Information such as the duration of the
experiment is updated by analyzing the file.

Required arguments:

    --madFilename - full path to the complete Madrigal file. Basename will
                    be maintained.

    --expDir - full path to experiment directory. Example:
               "/opt/madrigal/experiments/1998/mlh/20jan98"
    
    --permission - 0 for public, 1 for private (restricted to certain IP range)

    --fileDesc - file decription

Optional arguments:

    --category - 1=default, 2=variant, 3=history, or 4=realtime. If this argument is missing,
                 1 (default) used.

Example: If you want to add the new file /tmp/mlh060120g.002 to an existing experiment, you would enter:

/opt/madrigal/bin/createExpWithFile.py --madFilename=/tmp/mlh060120g.002 \
     --expDir=/opt/madrigal/experiments/2006/mlh/20jan06 --permission=0 \
     --fileDesc="alternative analysis"  --category=1

Modify an existing file in an experiment

If you want to modify an existing file in a Madrigal experiment, use updateFileInExp.py, located in madroot/bin. Note that if the modification in the file is significant, it is preferable to make the old file a history file using changeFileStatus.py, and to add a new file with a different name using addFileToExp.py.

Usage:

updateFileInExp.py is a script used to update an existing Madrigal file in an
   existing experiment. Information such as the duration of the
   experiment is updated by analyzing the file. This script is use to replace
   an existing Madrigal file. Use addFileToExp.py to add a new file, and
 changeFileStatus.py to change any file attribute.
Required arguments:
 --madFilename - full path to the new version of the Madrigal file. Basename will
   be maintained.
 --expDir - full path to experiment directory. Example:
 "/opt/madrigal/experiments/1998/mlh/20jan98"

Example: To modify the existing file /opt/madrigal/experiments/2002/01oct02/mlh021001a.000 with the file /tmp/mlh021001a.000, you would enter:

/opt/madrigal/bin/updateFileInExp.py --madFilename=/tmp/mlh021001a.000 \
 --expDir=/opt/madrigal/experiments/2002/01oct02 

Change the status of a file in an experiment

If you want to change the status of any existing file in an experiment, such as to make a default file into a history file, use changeFileStatus.py, located in madroot/bin. Usage:

changeFileStatus.py is a script used to change the status of an
existing Madrigal file.  The file permission, the file description,
or the file category can be changed.

Required arguments:

    --filename - basename of existing Madrigal file. 

    --expDir - full path to experiment directory. Example:
               "/opt/madrigal/experiments/1998/mlh/20jan98"

Optional arguments - set these to change a file attribute:
    
    --permission - 0 for public, 1 for private (restricted to certain IP range)

    --fileDesc - file decription

    --category - 1=default, 2=variant, 3=history, or 4=realtime

Example: If you want to change the status of /opt/madrigal/experiments/2002/01oct02/mlh021001a.000 to be a history file, you would enter:

/opt/madrigal/bin/changeFileStatus.py --filename=mlh021001a.000 \
    --expDir=/opt/madrigal/experiments/2002/01oct02 \
    --category=3

Remove a file from an experiment

To completely remove a file from an existing experiment, rather than simply make it a history file, use removeFileFromExp.py, located in madroot/bin. Usage:

removeFileFromExp.py is a script used to remove an existing Madrigal file from an
existing experiment. Information such as the duration of the
 experiment is updated by analyzing the remaining files. 
Required arguments:
 --filename - basename of the Madrigal file to be removed.
 --expDir - full path to experiment directory. Example:
 "/opt/madrigal/experiments/1998/mlh/20jan98"
 

Example: To remove the file /opt/madrigal/experiments/2002/01oct02/mlh021001a.000, you would enter:

/opt/madrigal/bin/removeFileFromExp.py --filename=mlh021001a.000 \
     --expDir=/opt/madrigal/experiments/2002/01oct02 

Adding auxiliary plots and information to a Madrigal experiment

Since Madrigal is a web-based application, you can also display auxiliary plots and information about your experiment as web pages. If you add additional web documents according to the rules below, these documents will also show up through the standard Madrigal interface.

As an example of plots associated with individual records, lets say you have a Madrigal experiment in the directory /opt/madrigal/experiments/2002/01oct02, with a file named mlh021001a.001. If this file had 10 records, you could then create 10 plot files called plot001.png through plot010.png. You would then create the subdirectories plots/mlh021001a.001/ under the main directory /opt/madrigal/experiments/2002/01oct02/, and put those 10 plot files there.


Previous: Creating Madrigal data files   Up: Madrigal admin guide   Next: Other admin tasks