-->
Previous: Remote access using Matlab   Up: Remote access programming tutorial toc   Next: Remote access using IDL

Remote access using python

This page describes the remote Python API, and gives some examples of using this API. These examples have been tested on both Windows and Linux, and require only access to the internet and python 2.3 to run. It is available for download here.

The remote Python API is organized in the same way as the Madrigal data model, from Instrument at the highest level, down to the level of data values. Readers who are not familiar with the Madrigal data model should read the material in that section before proceeding with this tutorial.

The basic object in the remote Python API is the MadrigalData, found in the madrigalWeb module. To initialize MadrigalData requires only the url of the home page on any Madrigal 2.3 (or above) site as an argument. Calling the methods of this object will return all possible information from one Madrigal site. The other objects in madrigalWeb are simply there to hold returned information - for example, the MadrigalExperiment object holds information about one experiment.

MadrigalData has the following methods:

See the Madrigal Python API reference guide for complete documentation.

Two applications written with the remote Python API follow. The first is a simple regression test that is run to test web services when Madrigal is installed. The second is a script that downloads realtime data from any desired Madrigal site.


Simple regression test

This simple script calls the following MadrigalData methods:

This example also shows how to get data from a different Madrigal site than the one you start with.

To use this regression test, cd to the examples directory in the installation directory, and type:

python exampleMadrigalWebServices.py


"""exampleMadrigalWebServices.py runs an example of the Madrigal Web Services interface
   for a given Madrigal server.

   usage:

   python exampleMadrigalWebServices.py 



"""

# $Id: rt_python.html 3691 2011-10-19 01:26:21Z brideout $

import madrigalWeb.madrigalWeb

# constants
user_fullname = 'Bill Rideout - automated test'
user_email = 'brideout@haystack.mit.edu'
user_affiliation = 'MIT Haystack'

madrigalUrl = 'http://madrigal.haystack.mit.edu/madrigal'


testData = madrigalWeb.madrigalWeb.MadrigalData(madrigalUrl)



print 'Example of call to getAllInstruments'
instList = testData.getAllInstruments()
# print out Millstone
for inst in instList:
    if inst.code == 30:
        print (str(inst) + '\n')
        

print 'Example of call to getExperiments'
expList = testData.getExperiments(30, 1998,1,19,0,0,0,1998,1,22,0,0,0)
for exp in expList:
    # should be only one
    print (str(exp) + '\n')


print 'Example of call to getExperimentFiles'
fileList = testData.getExperimentFiles(expList[0].id)
for thisFile in fileList:
    if thisFile.category == 1:
        print (str(thisFile.name) + '\n')
        thisFilename = thisFile.name
        break
    
print 'Example of downloadFile - simple and hdf5 formats:'
result = testData.downloadFile(thisFilename, "/tmp/test.txt", "simple")
result = testData.downloadFile(thisFilename, "/tmp/test.hdf5", "simple")

print 'Example of simplePrint - only first 1000 characters printed'
result = testData.simplePrint(thisFilename, user_fullname, user_email, user_affiliation)
print result[:1000]
print

print 'Example of call to getExperimentFileParameters - only first 10 printed'
fileParms = testData.getExperimentFileParameters(thisFilename)
for i in range(10):
    print fileParms[i]
print


print 'Example of call to isprint (prints data)'
print(testData.isprint(thisFilename,
                       'gdalt,ti',
                       'filter=gdalt,500,600 filter=ti,1900,2000',
                       user_fullname, user_email, user_affiliation))


print 'Example of call to madCalculator (gets derived data at any time)'
result = testData.madCalculator(1999,2,15,12,30,0,45,55,5,-170,-150,10,200,200,0,'sdwht,kp')
for line in result:
    for value in line:
        print ('%8.2e ' % (value))
    print('\n')

print 'Example of searching all Madrigal sites for an experiment - here we search for PFISR data'
expList = testData.getExperiments(61,2008,4,1,0,0,0,2008,4,30,0,0,0,local=0)
print expList[0]

print 'Since this experiment is not local (note the experiment id = -1), we need to create a new MadrigalData object to get it'
testData2 = madrigalWeb.madrigalWeb.MadrigalData(expList[0].madrigalUrl)

print 'Now repeat the same calls as above to get PFISR data from the SRI site'
expList2 = testData2.getExperiments(61,2008,4,1,0,0,0,2008,4,30,0,0,0,local=1)
print 'This is a PFISR experiment'
print expList2[0]

Script to download realtime data from Madrigal

The following is a demonstration script that shows how real-time data can be imported from any Madrigal site that is updated on a real-time basis.

In this example, data is imported from http://www.haystack.mit.edu/madrigal from "Millstone Hill IS Radar". The following Madrigal parameters are retrieved:

year,month,day,hour,min,sec,gdlat,glon,gdalt,az,el,vo,dvo

for all records from the past 15 minutes.

Although the particular Madrigal site (http://www.haystack.mit.edu/madrigal), the instrument ("Millstone Hill IS Radar"), the parameters, and the times are hard-coded in this example, they could be easily be modified to be arguments.

To avoid missing data, we choose one parameter to be the filter parameter: vo. By filtering on this parameter, any "missing" values are filtered out.

To run this script requires the python Madrigal API be installed, which can be downloaded from http://www.haystack.edu/madrigal/madDownload.html.




import os,sys,os.path
import string
import time


import madrigalWeb.madrigalWeb


#constants
madrigalUrl = 'http://www.haystack.mit.edu/madrigal'
instrument = 'Millstone Hill IS Radar'

user_fullname = 'Put your name here!!!'
user_email = 'your@email.here'
user_affiliation = 'Put your affiliation here!!!'


# each line of data contains the following parameters
params = 'year,month,day,hour,min,sec,gdlat,glon,gdalt,azm,elm,vo,dvo'
filterParm = 'vo'
timeDelay = 15

# create the main object to get all needed info from Madrigal
madrigalObj = madrigalWeb.madrigalWeb.MadrigalData(madrigalUrl)


# these next few lines convert instrument name to code
code = None
instList = madrigalObj.getAllInstruments()
for inst in instList:
    if inst.name.lower() == instrument.lower():
        code = inst.code
        break

if code == None:
    raise ValueError, 'Unknown instrument %s' % (instrument)


# next, get a list of real time experiments in the last timeDelay minutes
startTime = time.gmtime(time.time() - timeDelay*60.0)
endTime = time.gmtime(time.time())


try:
    expList = madrigalObj.getExperiments(code, startTime[0],
                                     startTime[1],
                                     startTime[2],
                                     startTime[3],
                                     startTime[4],
                                     startTime[5],
                                     endTime[0],
                                     endTime[1],
                                     endTime[2],
                                     endTime[3],
                                     endTime[4],
                                     endTime[5])

except:
    raise ValueError, 'No realtime experiments found'


# assume there's only one realtime experiment, and get the file names
fileList = madrigalObj.getExperimentFiles(expList[0].id)

if len(fileList) == 0:
    raise ValueError, 'No realtime experiment files found'


# get data from each of the files
startDateStr = time.strftime('%m/%d/%Y', startTime)
startDateStr = ' date1=' + startDateStr
startTimeStr = time.strftime('%H:%M:%S', startTime)
startTimeStr = ' time1=' + startTimeStr
endDateStr = time.strftime('%m/%d/%Y', endTime)
endDateStr = ' date2=' + endDateStr
endTimeStr = time.strftime('%H:%M:%S', endTime)
endTimeStr = ' time2=' + endTimeStr

filterString = 'filter=%s,-1E30,1E30' % (filterParm) + startDateStr + startTimeStr + endDateStr + endTimeStr
for dataFile in fileList:
    result = madrigalObj.isprint(dataFile.name, params, filterString,
                                 user_fullname, user_email, user_affiliation)
    # make sure it succeeded
    if result.find('No records were selected') != -1:
        continue
    if result.find('****') != -1:
        continue
    print result
Previous: Remote access using Matlab   Up: Remote access programming tutorial toc   Next: Remote access using IDL