Astronomical Data Analysis Software and Systems XVII A.10 ASP Conference Series, Vol. XXX, 2008 J. Lewis, R. Argyle, P. Bunclarck, D. Evans, and E. Gonzales-Solares, eds. A new model for archiving synoptic data in the VISTA Data Flow System. 9 Nicholas Cross, Ross Collins, Nigel Hambly, Mike Read and Eckhard 0 Sutorius 0 Scottish Universities Physics Alliance, Institute for Astronomy, 2 University of Edinburgh, Blackford Hill, Edinburgh, EH9 3HJ, U.K. n a J Abstract. The VISTA Data Flow System comprises nightly pipeline and 3 archiving of near infrared data from UKIRT-WFCAM and VISTA. This in- 2 cludes multi-epoch data which canbe used to find moving andvariable objects. We have developed a new model for archiving these data which gives the user ] anextremelyflexible andreliabledatasetthatis easyto querythroughanSQL M interface. We have introduced several new database tables into our schema for I deep/synoptic datasets. We have also developed a set of curation procedures, . h which give additional quality control and automation. We discuss the methods p used and show some example data. Our design is particularly effective on cor- - related data-sets, where the observations in different filters are synchronised. It o is scalableto largeVISTA datasetswhichwill be observedinthe nextfew years r t and to future surveys such as Pan-STARRS and LSST. s a [ 1. Introduction 1 v Thestudyof photometrically or astrometrically varyingsources has led to many 9 important discoveries in astronomy, e.g. stellar masses from eclipsing binaries; 2 distance scales from stellar parallaxes and pulsating stars; the theory of gravity 7 from planetary motions and the physics of accretion disks from observations of 3 . cataclysmic variables. 1 With new wide field imagers on survey telescopes, large surveys of variable 0 9 objects have becomepossible. Over thenextfewyears allskyvariability surveys 0 willstartwithPan-STARRS(Kaiser2007)andLSST(Walker 2003). Nearinfra- : red technology has also improved and the UK Infra-red Telescope Wide Field v i Camera (UKIRT-WFCAM) and the Visible and Infra-red Survey Telescope for X Astronomy (VISTA; Emerson et al. 2004) are the first near infrared cameras r capable of undertaking large synoptic surveys. Here we discuss the design and a implementation of a dynamical relational database science archive for archiving synoptic data observed by UKIRT-WFCAM - WFCAM Science Archive (WSA; Hambly et al. 2008) - and by VISTA - VISTA Science Archive (VSA). 2. Database Design The basic design of the WSA and VSA is described in detail in Hambly et al. (2008). This work is an extension to incorporate new tables related to using multi-epoch dataandisanimprovementtoourinitialdesign(Crossetal. 2006). Figure 1 shows the new tables and their relationship to existing tables in two entity-relation models (ERMs). The left-hand ERM is for single or uncorrelated 1 2 Cross et al. VarFrameSetInfo Variability SourceXDetectionBestMatch * Mag−rms fit parameters −− bp paerhsostoptrAoteomprm emeerto,rtmritciicoe v nava naar,rinraimadbb siplii,ltaimytyr eaq qdluluaiaaaxnnnt,itmtiiteiaesds:: −##**** B esfm smlfeeaxooosutqgruNdt lN treeoiucfualrbemDamcjIehmiDsc tuetS nIiDeniqc eusaec sho iunrtceerm.ediate multiframe * expected mag limit Variability SourceXSynopticSourceBestMatch #F−−V itfSEasratx rrtmpaoFet eerdcSavateaetmad tpI Dimeanr Sasagyme nlteiomItpneittrifsco tables −−− S es oacso rlsuuflo oeorsrrcnbcs emt−jiee.anocXl estfsDoitg rswhe ccbititueehorniucancrtt sie1oio0n"n, obfut not −−C aplashtsorsotoimfmiceaettrtiriocicn v vaarriaiabbiliiltiyty q quuaanntittiiteiess − B feosrt eoabcjehc ut niniq euaec sho iunrtceerm.ediate multiframe S− s omtauacsrkctse.er source list from deeper D− aq resuattawreon c+mtitt iecieoatsrlniicb r+a tpehdo tometric S− s omtauacsrkctse erm seorugrecde alicsrt ofrsosm fi ldteeresp − D aq resuattawreon c+mtitt iecieoatsrlniicb r+a tpehdo tometric − Se maycenhrgo eeppdot ipccahSssobuanrcdse at − intermediate + deep stacks − intermediate + deep stacks MergeLog MultiframeDetector MergeLog MultiframeDetector SynopticMergeLog − f rmaemregsinegt. information for each − metadata for extensions − f rmaemregsinegt. information for each − metadata for extensions −−− ~ mEe 1mpa emocerghcringeh fui rnfitasieglmt es−inemr sfs1oa e prhltlme .oirnua tertei.porovnac fhlo:.r Figure1. The left hand plot shows the entity-relation model for single pass- bandoruncorrelatedmulti-epochdataandtherighthandplotshowstheentity- relation model for correlated multi-epoch data. multi-wavelength observations and the right-hand side is for correlated multi- wavelength observations. There are five new tables in the new schema. Two of them only appear in the correlated multi-epoch ERM and the other three appear in both ERMs. The new tables are: • The SynopticMergeLog table links the different bandpass frames that are taken within a short time of each other in a correlated multi-epoch survey. Itis much like theMergeLog, buthas theadditional primarykey attribute of meanMjdObs. • The SynopticSource table is the merged source catalogue created by mergingthedetectioncataloguesfromtheframesinSynopticMergeLog. This has similar attributes to the Source table, but with a primary key similar to the Detection table. This table makes it easier to get colour information at each epoch and makes the BestMatch table smaller. • The BestMatch table links the Source table - the unique list of merged bandpass sources from deep stacks - to each observation: either in Syn- opticSource for a correlated bandpass data set or in Detection for an uncorrelated or single passband data set. • The Variability table contains the statistical analysis of the multi-epoch data for each source in the BestMatch table. This includes both astro- metric and photometric analysis and classifications. • The VarFrameSetInfo table contains the fits for data across a whole frameSet (as defined in MergeLog). This is useful for understanding the limits of the data in each frame set. The model for the astrometric fit is also recorded here. These tables can be used together to find and categorise objects. The Variability table can be used to find objects which have interesting statistical properties. Linkingthis to the Source table and to neighbour tables of external surveys can select objects with very specific morphologies, variable properties, and colours. Only with very specificselections can a scientist hopeto finduseful Archiving Synoptic Data in VDFS. 3 9 targets for follow up in a database of 10 objects. The VarFrameSetInfo ta- ble can help define the noise properties of each frameset in each selection. The BestMatch can then be used with either the Detection or the Synoptic- Source to display the light curve of an object. The attributes in each of these 1 tables are described in detail in the WSA Schema Browser 3. Archive Curation Once new data has been ingested and quality controlled, curation proceeds as described in Hambly et al. (2008) and Collins et al. (2009): • production of deep stacks and catalogues. • deep stacks in different filters merged into Source table. • intermediate stacks are recalibrated against deep stacks • creation of SynopticSource and SynopticMergeLog • neighbour tables created • creation of BestMatch • creation of Variability and VarFrameSetInfo These steps are controlled at each stage by the properties in four curation tables: RequiredStack, RequiredFilters,RequiredNeighbours and Re- quiredSynoptic. The first three of these are discussed in Collins et al. (2009). RequiredSynoptic specifies whether a survey has correlated passbands and the correlation timescale for the SynopticMergeLog. Thenewsteps thathave beenaddedin aretherecalibration ofintermediate stacks that typically improves the zeropoints by 0.005 mag and the synoptic table creation. The recalibration is done by comparing bright stars in each intermediate stacks with ones in the deep stack. Frames which show big changes in zeropoint are deprecated at this stage. The matching of intermediate stacks to each unique source is the most important step. The SourceXDetection neighbour table is the starting point for the creation of the BestMatch table. The neighbour table links the source to all detections within a set radius. The BestMatch links sources to the nearest match within a smaller radius. If there is no match, and the position lies within an observed frame, then a default row is inserted. Flags are used to alert users to duplicate matches or non-detections close to the edge of a frame. The BestMatch table is a link between sources and observations, and allows users to track objects that disappear (an eclipse, or fading after an outburst). The statistics in the Variability table are based on good matches in the BestMatch table. The details of all these steps are in Cross et al. (2009). At each stage there are tight constraints on speed, since surveys such as the VISTA 2 Variables in Via Lactea (VVV) survey - a synoptic survey of the galactic plane and bulge - will have ∼ 109 sources, each with 100 observations. 1http://surveys.roe.ac.uk/wsa/www/wsa browser.html 2http://www2.astro.puc.cl/VVV/index.php/ 4 Cross et al. 10 -2.2 <K>=-999999488.0 <min(RMS)>=0.0044 -1.1 0 1.1 2.2 -2.2 <H>=10.568 1 -1.1 0 1.1 2.2 -2.2 <J>=12.31 -1.1 0.1 0 1.1 2.2 -2.2 <Y>=14.02 -1.1 0 0.01 1.1 2.2 -2.2 <Z>=15.499 -1.1 0 1.1 0.001 2.2 12 14 16 18 20 0 173 346 519 692 865 K / mag time (mjdObs-53510) / days Figure 2. The left hand plot shows the K-band magnitude-RMS plot for a subset of the DXS data. The green crosses are stars and the black dots are galaxies. The blue squares are objects classified as variable. The red curves are fits to the main locus of the magnitude-RMS plot, which give the noise properties of the data. The right hand plot shows light-curves in all 5 broad- band filters for a single star in a correlated data set (CAL). Green and red crosses aregoodandflaggeddata respectivelyandblack lines aremissingdata. 4. Examples We show some example data from two large data sets, the UKIDSS (Dye et al. 2006) Deep Extragalactic Survey (DXS) and the UKIRT Standard Star calibrationdata(CAL).Fig2showsanexamplemagnitude-RMSplot(usefulfor determining which objects are variables) and a set of correlated light-curves for a single star. This star was selected in the archive for its variable characteristics from the WSA. These plots can be produced using simple SQL queries that are 3 described in the WSA SQL cookbook . References Collins,R.S.et al. 2009,inASP ConfSer. XXX, ADASS XVIII, ed. D. A.Bohlender, D. Durand & P. Dowler (San Fransisco: ASP),[A.09] Cross, N. J. G. et al. 2009, in preparation Cross, N. J. G. et al. 2007, in ASP Conf. Ser. 376, ADASS XVI, ed. R. A. Shaw, F. Hill, & D. J. Bell (San Francisco: ASP), p54 Dye, S. et al. 2006, MNRAS, 372, 1227 Emerson J. P. Sutherland W.J., et al. 2004,ESO Messenger, 117, 27 Hambly, N. C. et al. 2008,MNRAS, 384, 637 Kaiser N. 2007, ”Proceedings of the Advanced Maui Optical and Space Surveillance Conference”, ed. S. Ryan, The Maui Economic Development Board, p9 Walker A. 2003,MmSAI, 74, 999 3http://surveys.roe.ac.uk/wsa/sqlcookbook.html#Multi-epoch