Data storage, processing and visualisation for the ATCA T. MurphyA,C,D, P. LambB, C. OwenC, and M. MarquardingC ASchool of Physics, University of Sydney,NSW,2006, Australia BCSIRO ICT Centre, Canberra, ACT, 2601, Australia CAustralia Telescope National Facility, Epping, NSW,1710, Australia DEmail: [email protected] Abstract: We present three Virtual Observatory tools developed at the ATNF for the storage, pro- cessingandvisualisationofATCAdata. ThesearetheAustraliaTelescopeOnlineArchive,aprototype datareductionpipeline,andtheRemoteVisualisation System. Thesetoolsweredevelopedinthecon- 6 text of the Virtual Observatory and were intended to be both useful for astronomers and technology 0 demonstrators. Wediscuss thedesign and implementation of these tools, as well as issues that should 0 be considered when developing similar systems for futuretelescopes. 2 Keywords: astronomical data bases: miscellaneous — methods: data analysis n a J 1 Introduction 2 The ATCA 6 1 TheAustraliaTelescopeCompactArray(ATCA)isan Theso-calleddataexplosioninastronomypromisesex- east-westearth-rotationsynthesisinterferometer,with 1 v citing new scientific developments, but brings with it six 22 m antennas on a 6 km baseline. It has been in 4 manytechnicalchallenges,incollecting,storing,trans- operation at Narrabri since 1990. The telescope can 5 porting, processing and visualising data. Virtual Ob- observe at 6 bands with wavelengths 20 cm, 12 cm, 6 3 servatories(VO)havedevelopedtomeetsomeofthese cm,3cm,1cmand3mm. Eachantennaobservestwo 1 technical challenges. Falling under the broad area of frequencies simultaneously. There are six bandwidths 0 e-science (which incorporates other scientific domains available on Frequency 1 (128, 64, 32, 16, 8 and 4 6 facingsimilar challenges, suchasgeneticsandparticle MHz) and two bandwidths on Frequency 2 (128 and 0 physics) theaim of VirtualObservatory research is to 64 MHz). The telescope produces ∼ 0.5 GB of raw / providethetools necessary for dealing with this data. dataperday,andthisislikelytoincreasesignificantly h p TheAustralianVirtualObservatory(Aus-VO)1was with future telescope upgrades. - started in 2003 with the aim of both contributing to In the rest of this section we outline the exist- o the international VO effort, and developing tools of ing systems for archiving, processing and visualising r ATCA data. use to Australian astronomers. Australia has many t s areas of expertise (for example radio astronomy) and a it makes sense to focus our efforts on providing mod- 2.1 Data archiving : v ern tools for working in these areas. In this context i theATNFdecidedtodeveloparangeoftoolsforstor- SincethecommencementofoperationoftheATCAin X ing,processingandvisualisingdatafromtheAustralia June1990,acompleterecordofalldataobservedfrom r Telescope Compact Array. The aim was that these the telescope has been maintained offline at the tele- a tools would be useful to astronomers now, and at the scopesite,mostlyrecordedonCD.Inconjunctionwith sametimeletusexplorethetechnologythatwouldbe this,theATNFmaintainedarecordoftheprojectpro- necessary fordevelopingsoftware forfuturetelescopes posals for observations onthetelescope –the Projects such as theSquareKilometre Array (SKA). database—andashortformoftheobservationparam- etersforeachdaysobserving–thePositions database. This paper is based on a talk given at the ASA After a proprietary period of 18 months, in which the Annual Meeting in Sydney in July 2005. After giv- observing team has sole access to the data obtained ing some background about the ATCA and the Vir- inanobservation, thedataismadepubliclyavailable. tual Observatory, we discuss three tools developed at AstronomerscansearchforobservationsontheATNF the ATNF over the last few years. Firstly the Aus- webpage, submit the details of the observation data tralia Telescope Online Archive which contains all of requiredviae-mailandhaveaCDcontainingthedata thedatacollectedsofarbytheATCA;secondlyapro- prepared for them at nominal cost. totype data reduction pipeline for ATCA data; and finally the Remote Visualisation System for viewing 2.2 Data processing large datasets. ATCA Data processing (reduction) is generally per- formed with one of the standard radio data reduction 1http://www.aus-vo.org packages; most commonly Miriad (Sault et al. 1995), 1 2 Publications of theAstronomical Society of Australia but also AIPS2 and AIPS++3. After loading, editing • Data Curation IG and calibrating the data, the resulting product is an • Theory IG intensity map referred to as the dirty image. At this stage a deconvolution algorithm, usually a variant of whichfocusontherequirementsofparticularapplica- clean(H¨ogbom1974),isrequiredtoproducethefinal tion domains. image. TheaimoftheAustralianVirtualObservatory(Aus- At each stage in the process there are a range of VO)istoprovidedistributed,uniforminterfacestothe parameters that can beset tocontrol thetypeof pro- data archives of Australia’s major observatories and cessing performed. Both general parameters (such as the archives of simulation data. Aus-VO is a collab- calibration strategy, clean method and type of data oration between many Australian institutions, includ- editing)aswellasfine-grainedparameters(suchascal- ingtheUniversitiesofMelbourne,Sydney,NewSouth ibration solution interval, numberof clean iterations WalesandQueensland,MonashUniversity,Swinburne and median filter size) need to be modified to obtain UniversityofTechnology,theAustralianNationalUni- the best results. Hence data processing is typically a versityandMountStromloObservatory,theVictorian highly interactiveprocess. Partnership for AdvancedComputing, theATNFand There is an existing system operating at the tele- theAAO. scopeCAONISdesignedforon-the-flyimagingofATCA There are a range of VO projects underway in data. However the design of this system makes it dif- Australia, including the development of data archives ficult to port to current Linuxsystems. and software for HIPASS (Meyer et al. 2004), RAVE (Siebert 2004), 2QZ (Croom et al. 2004) and SUMSS (Bock et al. 1999). The initial focus of most of these 2.3 Data visualisation projectshasbeentomakedatafromAustralianprojects widely available within the international community, The final images from the ATCA are usually visu- in a VO compliant format. In addition there are sev- alisedusingtoolssuchasMiriadorkvis(Gooch1996). eral projects investigating novel methods for astro- These are well established tools that cover many of nomical data mining and data analysis, for example thevisualisation requirementsofATCAobservers. As Rohdeet al.(2005)applymachinelearningtechniques mentioned in the previous section, the CAONIS system to catalogue crossmatching. The Melbourne Univer- which runs at Narrabri also allows basic visualisation sitygrouphasalsobeensettingupinfrastructuresuch of images. asaregistryforAustralianwebservicesanddataarchives. 3 The Virtual Observatory 4 The Australia Telescope TheumbrellaorganisationforVirtualObservatorywork Online Archive istheInternationalVirtualObservatoryAlliance(IVOA). TheIVOAwasformed inJune2002 withamission to In June 2003, a joint project between the ATNF and the CSIRO ICT Centre was commenced to make the facilitatetheinternationalcoordinationand ATNF archive data available online as the Australia collaborationnecessaryforthedevelopment Telescope Online Archive (ATOA). This was planned and deployment of the tools, systems and as anew dataresource for astronomers, as well as the organizational structures necessary to en- foundationforthedevelopmentofonlinedataprocess- able the international utilization of astro- ing systems to make the raw data more accessible to nomical archives as an integrated and in- non-expert users (see Section 5). The construction of teroperating virtual observatory. the ATOA required the copying of the offline archive TheIVOAisacollaborationbetweenover15mem- (atthetime,∼2700CDs,totalling∼1.7TB)fromthe bercountriesincludingAustralia. Thefocussofarhas telescopesitetoCanberrawheretheonlinearchivewas been developing the standards required for interoper- to be developed, creating a meta-database describing abilitybetweensoftwaredevelopedanddataproduced the data, and making a web front-end to search and inallareasofastronomy. Anothersignificantaimisto download thedata. develop the infrastructure required (networks and or- Thedatabaseconsistsoftwoparts. Thefirstisthe ganisations)forthelargescalestorageanddistribution raw data from the telescope (RPFITS files) which is of astronomical data. stored as normal files on the host system. In addition The IVOA working groups address a range of is- thereisarelationaldatabasewhichstoresallthemeta- suessuchasgridandwebservices,datamodellingand data (discussed in Section 4.1). The ‘vital statistics’ standards for the data access. There are also four in- of the ATOA are shown in Table 1. The current rate terest groups of growth of the archive is ∼ 0.5 Gb/day. However • Applications IG this is likely to increase significantly in the future as new instruments come online. To maintain an grow- • Astronomy Grid IG ingarchive(ratherthanastaticone)itisnecessaryto 2Astronomical Image Processing System (AIPS), ensure the RPFITS files are stored in a readily acces- sible way (currently on a RAID system) that is easily http://www.cv.nrao.edu/aips. 3Astronomical Image Processing System (AIPS++), distributed over a number of drives. Also, that the http://aips2.nrao.edu. database itself is easy to update in a robust manner. www.publish.csiro.au/journals/pasa 3 The ATOA was made publicly available in December 4.2 A data model for the ATCA 2004andcanbeaccessedfromhttp://atoa.atnf.csiro.au. A data model is a comprehensive scheme describing how data is to be represented, for manipulation by humansor computerprograms. Data models are crit- Table 1: ATOA Statistics ical for planning how data will be organised within a Projects 2261 databaseastheydescribealltherelationshipsbetween Files 57147 thedifferent entities. Sources 128111 AsectionofourdatamodelfortheATCAisshown Metadata size ∼4 Gb inFigure1. WenowbrieflyexplaintheUML(Unified RPFITS data size ∼2 Tb ModellingLanguage)notationusedinthedatamodel. Each boxcontainsan entity(e.g. Scan) that hasbeen Growth rate ∼0.5 Gb/day identified in the metadata. Each entity has attributes (e.g. restFreq), each of which are of a specified data type (e.g. float). Associated entities are connected to each other with lines, which also specify the cardi- nality of the relationship. For example 4.1 Metadata Scan SPW 1..* 0..* Metadata is simply data which describes other data, for example the project code or the name of the pri- should bereadas“A scan has 0 or more spectral mary calibrator. The meta-database for the ATOA windows. A spectral windows has 1 or more consists of three main parts; thecontentsof theorigi- scans.” nalATNFonlineProjectsdatabase,metadatadescrib- The development of a data model that covers the ingtheobservation thatisextracteddirectly from the wholeofastronomyisanongoingprojectwithinthein- rawdatafilesproducedbythetelescopessoftware,and ternational VO community. We havecontributed this metadatainferredfromalloftheavailabledatasources data model to the IVOA Data Model WG as an ex- to assist in the automation of reducing the telescopes ample of a data model for radio astronomy. For more raw data to images. The types of metadata used in information on this topic, see the IVOA Data Mod- theATOA are summarised in Table 2. elling website4. The ATOA archive database structure is created directlyfromthedefinitionsintheATOAdatamodel. Table 2: Metadata in the ATOA Partsofthedatamodelcontaininformationforspecific databaseimplementationssothatalloftheimplementation- Metadata source Examples specific parts of the database creation are handled in Projects database proposal; observer name this process. The data model in Figure 1 corresponds country; institution tothepartofthedatamodelthatdescribesthemeta- Positions database∗ source names & positions data contained directly in the archive RPFITS files. Thedatamodelfor theinferreddataisavailablefrom observing band; receivers theATOA web pages5. RPFITS files scans; polarisations array configuration Inferred calibrator names & roles 4.3 Implementation calibrator–targetmatches The ATOA web interface was implemented as a Java ∗ The positions database is included in our data (ver. 1.4.2)6applicationandishostedusingtheApache model, and some of the metadata is used to Tomcat(ver. 5.0.28)7webcontainer. Relationaldatabase reconstruct the observation metadata. However, servicesareprovidedbyanOracle9iinstancerunning it is not actually loaded into the ATOA. on thesame machine. Aweb based interface was cho- sensoastomaximiseinteroperabilityandprovideeasy access to users. For example, RPFITS files may sim- plybedownloadedbyconstructingasuitableURLfor the ATOA file server. This allows files on the server Most of the metadata available in the ATNF Po- tobedownloadedbyaWebbrowser,bycommand-line sitions database is also available from the metadata programsthatallowuserstofetchthedatareferredto in the raw data files, and is finer-grained, since the by a URL, or by application programs using libraries Positionsdataisadailysummary,whilethefilemeta- that allow a URL to be opened in a similar way to a data is available for each telescope pointing. The in- file on a local file system. ferred metadata in the ATOA is ‘value added’ infor- mation that is automatically determined from the ex- 4http://www.ivoa.net/twiki/bin/view/IVOA/IvoaDataModel isting metadata, for example the calibration role of 5http://www.atnf.csiro.au/computing/web/atoa/implementation.html each source (primary calibrator, secondary calibrator, 6http://java.sun.com target, etc). This is discussed furtherin Section 5.2. 7http://tomcat.apache.org 4 Publications of theAstronomical Society of Australia The user interface centres around two main web pipeline which can process single pointing continuum pages: the query page which allows users to specify data from the ATOA and is available for testing at criteria for selecting RPFITS files from the archive, http://atoa.atnf.csiro.au/test. and aresults page which providesthemeans for users toinspectthemetadataofmatchingfilesanddownload 5.1 Metadata for automatic particular files if desired. The results page initially processing presentstheuserwithabroad,globalviewofthequery resultsintabularformlistingdetailssuchasfilename, The metadata in the Project and Position databases, filesize,principalinvestigatorandarrayconfiguration. whileprovidinginformationaboutwhichastronomical Theusermayalsointeractively‘drill-down’foramore sources have been recorded in an observing session, detailed view of any file in the list. RPFITS files can does not (in general) provide any information about be downloaded individually or in batches. therolethattheobserverintendedthesourcetoplayin AsmentionedinSection2.1ATCAdatahasapro- theobservation(eg. primarycalibrator,targetsource). prietary period of 18 months, in which it is only ac- This would be relatively easy to record in a new sys- cessible to members of the project team. Authorised tem, but as we are dealing with existing data we had accessissupportedfordatawithintheproprietaryac- to infer the roles of sources. cessperiod. Ifauserwishestoaccessproprietarydata Another problem for automatic processing is the theymustfirstgothroughamanuallyverifiedauthen- grouping of data into valid ‘observations’. An expert ticationprocessafterwhichapasswordisissuedtothe would typically choose an appropriate subset of files Principal Investigator for that project. In the future from the archive to image. However, a non-radio as- we plan to replace this authentication and authorisa- tronomermaychooseansubsetthatcontainsfilesthat tion method with a streamlined system that links the newATNFproposalsystem,OPAL8,anditsauthenti- shouldbeimagedseparately,orfilesthatcontaindata that should be ignored entirely. Although it is impos- cationdatabasetotheATOA.Userswillthenbegiven sible to deal with all cases, our aim was to have the accesstoproprietarydatabyusingtheirOPALcreden- pipelinegrouptogethertheselecteddatainsuchaway tials basedon theprojectstheyareassociated with in that an image could bemadein at least 80% of cases. theOPAL system A wide range of observation types can be recognised TheATOAwebserveranddatabasearehostedon andcharacterisedusingthemeta-databasebutarenot aDellPowerEdge750 runningDebianLinux3.0. The yet processed by the prototype pipeline (e.g. millime- hosthasa2.8GHzPentium4processor, 2GBofRAM tre and spectral line observations). andisattachedtoa3terabyteAppleXserveRAIDfor In the following section we discuss how we assign archive storage. the source roles within an observation, and the algo- rithmweusedtomatchtargetsourceswiththeappro- 5 A data processing pipeline priate calibrators. framework 5.2 Determining source roles The data products in radio astronomy are often less While matching target sources with their calibrators accessible to the non-expert than those in other do- wouldbestraightforwardforanastronomeritisachal- mainssuchasopticalastronomy. Itrequiresareason- lenge for an automatic system. In a typical (simple) ably highlevel ofdomain expertisetoprocess theraw observing session the primary calibrator is recorded data and produce an image. Obviously for carrying for a short period at the start or end of the observing out detailed scientific analysis it would be necessary session;andalternatingpointingsaremadetothesec- to develop this expertise, or collaborate with a radio ondary,and thesource of interest, or target. However astronomer. Howeverinaneraofmultiwavelengthas- there is a great variety of different ways that the ob- tronomy, astronomers expect to download and com- servercanchoosetostructuretheirobservations. Ifan pare data from a variety of telescopes, at a variety of observationcontainsmorethanonetarget,thetargets wavelengths. mayshare,orhavedistinct,secondarycalibrators, de- Withthisinmindwehavedevelopedanautomatic pending on their separation in thesky. There may be pipeline for people who want to quickly inspect the several secondary calibrators for each target, and the data in the ATOA, to see if it was suitable for fur- same source may be used for primary and secondary ther processing. One of the aims of this project was calibration. Inaddition,someobserversusesecondary to test the viability of ‘driving’ the pipeline using the calibratorsthatarenotinthelistofrecommendedcal- metadata discussed in Section 4.1. In other words ibrators, and that list has itself changed overtime. the pipeline should make decisions about what kind Inordertoclassifythesourcesinanobservingses- of processing to do — both on a general (e.g. contin- sion the following metadata is used uum/spectral line) and specific level (e.g. number of clean iterations). • The locations and names of sources extracted Inthissectionwediscussthedevelopmentofextra from the raw telescope data metadatarequiredtodrivingthepipeline,inparticular • Thetimesanddurationsofthesourcepointings thecalibrationprocess. Wethenoutlineourprototype • The names and locations of the four primary 8http://opal.atnf.csiro.au/ calibrators commonly used at the ATCA www.publish.csiro.au/journals/pasa 5 • A recent ATCA catalogue of recommended sec- minutes. We used the OpenPBS Batch Queuing Sys- ondary calibrators tem (ver. 2.3)13 for queue management, but unfortu- • Names of sources extracted from project titles nately it has no mechanism for reporting job comple- tion to another program. After processing for a web • Apre-assembledlistofpossiblecalibratorsources service completes, the batch job doing the processing Once the source roles have been determined, the sends a completion message to the program invoked proximityin theskyand theproximity in observation by the web service that controls the execution of the timeofthetargetsandtheirsecondarycalibrators are processing for theservice. However, at this point, the used to match targets with their respective calibra- batch processing system has not yet transferred the job’s output data back to the pipeline server. The tor(s). Foreachtargetpointing,aweightiscalculated for each secondary calibration pointing made within control program then polls the PBS batch queue at five second intervals to ensure that the batch job has two hours of observation of thetarget pointing: completed. wt,s=ΣPΣSe−(3a/a2max)2e−(3∆t/∆2tmax)2 filesTfrhoemratwheddataatafrpormoctehsesinAgT,OthAe,laolgltfihleesi,natenrdmtehdeiartee- sulting images are stored temporarily on the pipeline ∆tt,s<∆tmax server. The first web service call made by a pipeline where S is the set of candidate secondary calibrators, client reserves a private location for storage, and re- aistheangularseparationbetweenthetargetandthe quests a lifetime for the storage. The pipeline server secondary calibrator, amax is the maximum desirable has a configurable maximum lifetime, and the stored separation between the target and the secondary cal- data will be deleted after this time expires. Only ibrator (and is a function of the observing frequency clients who have the name of the storage area (a ran- band). ∆t is the separation of the time midpoints of domly generated string) can access it. There is no the target and calibrator pointings and ∆tmax is the quota on the storage use of any individual temporary maximum desirable time separation (two hours). The storagearea. However,aquotamaybeimposedonthe summation is over all pointings at a target (ΣP) and total amount of storage available to all active storage all secondary calibrators within two hours of a target areas. pointing(ΣS). Thewt,sareusedtoselectsuitablesec- ondary calibrators for the respective targets from the calibrators whosewt,s weightsdominateforaparticu- ATOA ATOA lar target. metadata raw data This procedure constructs the metadata required forcontinuumimagingatcentimetrewavelengths. The ATOA query server ATOA file server algorithm works well in general, but there are some problematic cases, for example where the target is a source from thesecondary calibrator catalogue. 5.3 Implementation Pipeline server The underlying processing of the ATCA data is car- riedoutusingtheGlishscriptinglanguageinAIPS++. Session state The ATOA imaging Web Services interface was con- structed using the Apache Axis tools (ver. 1.1)9, and interfacestotheprocessingscriptsthroughaPerl(ver. 5.4.8)10 script that deals with the control of the exe- Remote Visualisation Pipeline client cution of the Glish scripts. System The pipeline client is written using Python (ver. 2.3)11,andtheSOAPpywebservicestools(ver. 0.11.3)12. There were some minor, but difficult to find, prob- Figure 2: System architecture. This schematic lemsininteroperation betweentheSOAPpytoolsand shows the relationship between the three tools we ApacheAxis;thedata structuresused in theweb ser- have developed. vices calls are possibly more complicated than had been previously used between the two web services implementations. Documentation in both was not as TheATOAandpipelinewebservicesreturnaURL informative or complete as it might havebeen. forthegeneratedimagestotheend-user’ssystem. This Thepipelinewebservicescanbeconfiguredtorun allows the URL to be passed on to the Remote Visu- directly on the server host, or be directed to run on alisation System(seeSection6)imageviewingsystem othermachinesthroughabatchqueuingsystem,since so thattheimage can beviewed onlinewhile it isstill some stages in the pipeline can run for several CPU residentonthepipelineserver. Figure2showtheover- 9http://ws.apache.org/axis all system architecture, in particular how the ATOA, 10http://www.perl.com pipeline and RVSinteract. 11http://python.org 12http://pywebsvcs.sourceforge.net 13http://www.openpbs.org 6 Publications of theAstronomical Society of Australia 6 The Remote Visualisation large datasets. For example a 1.5 Gb data cube from System theGalacticAll-SkySurvey(GASS)(McClure-Griffiths et al. 2005) takes about one minute to load. Compare this with downloading the full cube from say the U.S. to TheRemoteVisualisationSystem(RVS)wasdesigned Australiawhichcouldtake∼1−2hours. Formorein- to enable visualisation of and interaction with large formationanddirectaccesstoRVS,seehttp://www.atnf.csiro.au/vo/r astronomical images in the context of the VO.As op- posed to other VO image displays, such as CDS Al- adin (Ferniqueet al. 2004), RVS does not require the 7 Discussion usertodownload thedatatotheclient machine. Fur- thermoreitprovidesrenderingofimagecubes,suchas TheATOAhasbeenpublicsinceDecember2004. We spectral-linecubescreatedfromATCAdata. TheRVS hope that it will encourage the reuse of ATCA data serveracceptsFITSimages-whichcanbecompressed forprojectsotherthanthoseitwasoriginallyintended - through local file:// URLsand remote http or ftp for. The framework used for the ATOA could easily access. Thedatashouldbeco-located withoratleast beextendedtoincludedatafromothertelescopesand be available to the server on high bandwidth connec- can be updatedas additional metadata is required. tion,whileitplacesnosuchrequirementsontheclient. The most significant improvement of the ATOA Only minimal data transfer to the client is necessary over existing online archives (such as NRAO15 and and this is independent of the size of the source data MAST16) is the data delivery mechanism. Most ex- set. The server-side architecture is distributed to en- isting archives do not support on demand delivery of able workload sharing and extensibility. RVS makes data over the web, instead requiring the user to sub- use of several software components: CORBA to make mitaformrequestingfilesthatthenhavetobetrans- it distributed, AIPS++ as the image rendering com- feredtoapubliclyaccessibleftpsiteortoothermedia ponent and Java for the web services and client. The (such as CD) for physical delivery. In the ATOA, the architecture of theRVS system is shown in Figure 3. batch downloading of multiple files is handled by a streaming TAR or ZIP archiving algorithm that per- formsdynamicarchivingasfilesarestreamedoverthe web, requiring no additional disk space on the server for theseoperations. IndevelopingtheATCAdatamodelandconsider- ing the type of metadata required for automatic pro- cessing we identified several new metadata types that would be useful to store in the RPFITS files. As a result thefollowing fieldshavebeenadded totheRP- FITS files and will be available in all future ATCA data: • four calibrator codes C (standard phasecalibrator) F (primary flux calibrator) B (bandpass calibrator) Figure 3: RVS system architecture. P (pointingcalibrator) • Pointing offsets RVSisexposedthroughawebserviceinterfaceus- • Weatherdata: added rain gauge and phase rms ing the standard Web Service Description Language and difference (WSDLver. 1.1)14. Thiscan easily beintegratedinto • Attenuatorsettings at start of scan custom applications. Several client applications make use of the web service interface; the RVSViewer - a • Subreflectorposition traditionalimageviewer,athumbnailservice-provid- • Correlator configuration ing preview images and a session viewer. The session • Scan type viewer connects to an existing RVSViewer via a key. Multiple instances can be runat thesame time, mak- • Coordinate type ing it apossible to useit as aconferencing tool where • Linemode people can observe and interact with the data. The ATOA pipeline re-uses the existing RVSViewer client • CACAL counter by passing it thefile location of theoutput image. Thesewillhelpbothautomaticprocessingsystemsand RVSisnotspecifictotheATOAorprototypepipeline astronomersassessthedataqualityintheobservations andthereareplanstouseitforallATCAarchives. It theyareinterestedin. Afulle-logbook systemwill be hasbeensuccessfully testedonimagesanddatacubes used in the future as currently the logs are all stored from various surveys and has good performance on 15http://archive.nrao.edu/archive/e2earchive.jsp 14http://www.w3.org/TR/wsdl 16http://archive.stsci.edu/ www.publish.csiro.au/journals/pasa 7 on paperat thetelescope and henceare not easily ac- H¨ogbom, J. A. 1974, A&AS,15, 417 cessible to ATOAusers. We have developed a prototype pipeline for pro- Meyer, M. J., et al. 2004, MNRAS,350, 1195 cessing of raw data for single-pointing continuum im- McClure-Griffiths, N. M., et al. 2006, ApJ, in press, ages. This is attached to the ATOA to provide an astro-ph/0510304 improvedservicefor usersoftheATOA.Atthisstage the image quality is suitable for previewing the data Rohde, D.J., et al. 2005, MNRAS,360, 69 in archive to see if it is of interest. Further manual processingwould thenberequiredtoobtainimagesof Sault, R. J., Teuben, P. J., & Wright, M. C. H. 1995, scientific quality. in ASP Conf. Ser. 77: Astronomical Data Analysis A significant challenge in developing the ATOA Software and Systems IV,433 andtheprototypepipelinewereintegratingpre-existing software with modern software tools. For example, Siebert, A. 2004, in SF2A-2004: Semaine de the Glish scripting language has no web service li- l’AstrophysiqueFrancaise, 567–+ braries and so an extra layer had to be developed be- tween the data processing level and the web services. If re-implementing from scratch, a language such as Python would be a better alternative for developing thepipeline. In developing these tools we have started to ex- plore the techniques necessary for astronomical soft- ware development in the VO era. This is essential for future telescopes and surveys that Australia will pro- duce. Making access to existing Australian data as easy as possible will maximise its use in the interna- tional community. Acknowledgments The authors would like to acknowledge the software development done on the RVS project, primarily by Anil Chandra and also by Praveena Tokachichu. The ATNF side of the prototype pipeline and ATOA de- velopment was managed by Neil Killeen and Jessica Chapman. Vince McIntyre contributed extensively to allthreeprojects, inparticularinsettingupthehard- ware required. A number of ATNF staff put in significant effort to get the ATOA set up, in particular Robin Wark, Bob Sault and Mark Wieringa. Warwick Wilson and MarkWieringaimplementedthechangestoaddextra metadata to the RPFITS files. FromtheICTCentre,RobertPowermadetheini- tial data model designs, the ATOA data loader soft- ware and ATOA query front end. Geoff Squire and Bella Robinson made significant contributions to the prototypepipeline. References Bock,D.C.-J.,Large,M.I.,&Sadler,E.M.1999,AJ, 117, 1578 Clark, B. G. 1980, A&A,89, 377 Croom, S. M., et al. 2004, MNRAS,349, 1397 Fernique, P., et al. 2004, in Toward an International Virtual Observatory,271–+ Gooch, R.1996, in ASPConf. Ser.101: Astronomical Data Analysis Software and Systems V,80 8 Publications of theAstronomical Society of Australia Figure 1: A portion of the data model for the ATCA. For an explanation of the no- tation see Section 4.2. The complete data model is available from the ATOA website: http://www.atnf.csiro.au/computing/web/atoa/implementation.html.