Cadre de Gestion des Données Data Management Framework 2015 Pan Am and Parapan Am Legacy Observational Data - Product User Guide- June 7, 2016 Version 2.0 DMF Team Pan Am Dataset User Guide June 7, 2016 Pan Am Dataset User Guide Page 2 of 59 DMF Team Pan Am Dataset User Guide Table of Contents 1. DOCUMENT INFORMATION ................................................................................................................................. 4 1.1 HISTORY .................................................................................................................................................................... 4 2. INTRODUCTION ........................................................................................................................................................ 5 2.1 OVERVIEW ................................................................................................................................................................. 5 2.2 PURPOSE AND SCOPE ................................................................................................................................................. 6 2.3 INTENDED AUDIENCE ................................................................................................................................................. 6 3. DATA STANDARDIZATION .................................................................................................................................... 7 3.1 UNITS ......................................................................................................................................................................... 7 3.2 CODES ........................................................................................................................................................................ 7 4. FILE FORMAT AND STRUCTURE ........................................................................................................................ 9 4.1 STRUCTURAL ORGANIZATION OF THE FILE ................................................................................................................ 9 4.2 DATE-TIME .............................................................................................................................................................. 10 4.3 QUALITY ASSESSMENT FLAG ................................................................................................................................... 11 5. DATASETS ................................................................................................................................................................ 13 5.1 OVERVIEW ............................................................................................................................................................... 13 5.2 DATA ELEMENT TERMINOLOGY ............................................................................................................................... 13 5.2.1 Wind ............................................................................................................................................................... 13 5.2.2 Precipitation and Rainfall ............................................................................................................................. 14 5.2.3 Station elevation ............................................................................................................................................ 15 5.3 PAN AM – MINUTELY MSC, S&T & PARTNER SURFACE WEATHER NETWORK ........................................................ 15 5.4 PAN AM – MINUTELY MSC ULTRAVIOLET RADIATION NETWORK - UV ................................................................... 21 5.5 PANAM – MSC MOORED BUOY (WATCHKEEPER™ MESSAGES) ............................................................................. 22 5.6 PAN AM – MINUTELY NAV CANADA AWOS ...................................................................................................... 24 6. APPENDICIES........................................................................................................................................................... 27 6.1 GLOSSARY ............................................................................................................................................................... 27 6.2 UNITS OF MEASURE ................................................................................................................................................. 29 6.3 UNIT CONVERSIONS ................................................................................................................................................. 32 6.4 STANDARD CODE TABLES ........................................................................................................................................ 34 6.4.1 buoy_type ....................................................................................................................................................... 34 6.4.2 present_weather ............................................................................................................................................. 35 6.4.3 total_cloud_amount ....................................................................................................................................... 53 6.4.4 station_type .................................................................................................................................................... 54 6.4.5 report_type ..................................................................................................................................................... 55 6.4.6 rapid_pressure_change ................................................................................................................................. 55 6.4.7 tendency_characteristic ................................................................................................................................. 56 6.5 PAN AM MESONET STATION METADATA ................................................................................................................. 57 6.6 PANAM MESONET MAP ........................................................................................................................................... 58 June 7, 2016 Pan Am Dataset User Guide Page 3 of 59 DMF Team Pan Am Dataset User Guide 1. Document Information 1.1 History Author Date Ver. Remarks Tahreem Ali February 10, 2016 1.0d Initial draft Tahreem Ali, Dale June 7, 2016 2.0 Final version Boudreau, Joan Klaassen June 7, 2016 Pan Am Dataset User Guide Page 4 of 59 DMF Team Pan Am Dataset User Guide 2. Introduction 2.1 Overview The creation of a legacy product of surface weather observations from the 2015 Pan Am and Parapan Am Games’ (hereafter referred to as Pan Am) Mesonet has been requested as part of the Pan Am Games Project deliverables. In response, the Data Management Framework (DMF) team at the Meteorological Service of Canada (MSC) is providing external clients with a concise, easy to decode product containing Pan Am surface and buoy weather data (and other observation, forecast and warning data products not included in this specific user’s guide). This product is generated by applications within the Data Management System (DMS). The DMS is a collection of real-time data acquisition, decoding, standardization, quality assessment (Qa) and product generation components for observation, forecast, and warning data. Within the DMS observational data has been stored elementally in the DMS Data Warehouse (DW), which follows a flat table format. The DW stored all Pan Am Mesonet data that was collected over the period of May 12 – August 31, 2015. These data were then extracted and output to comma- delimited text files (CSV). Given the size of the files, this component of the Pan Am legacy data product consists of a total of 20 files, with each file containing all station observations for a period of one week. Each of the 20 files starts on a Sunday and ends before the following Sunday. All data in these datasets are essentially ‘raw’, having undergone only minimal automated Quality Control (QC) (manual QC of the ATMOS Mesonet data will be posted to the Science Showcase section of the Legacy Archive, when available). The user is referred to an additional Instrumentation Metadata XLS file posted with the Datasets that describes specific data issues that the users must be aware of before using the datasets. During the production of these files, the following tasks were carried out:  Incoming DMS element packages were assigned an abbreviated label  Unit conversion from incoming units to standard units may have been performed  If the incoming element was a numeric code or a text value from a list of controlled vocabulary (so in effect a code), then code substitution was performed to map to a DMS standard code  A quality assessment (Qa) summary column (using incoming ‘native’ and DMS real-time quality assessments whenever available) was attached to each of the data elements to form a column pair The CSV files are intended for clients to obtain Pan Am data in an easy to read, user friendly format. Although this format easily loads into a Microsoft Excel spreadsheet, for clients needing to interact with data over long intervals or to compare observations from multiple stations geospatially, the use of tools such as programmatic parsers, graphical displays and GIS is advisable. There are several different streams of the observational component of the Pan Am legacy files, each containing data from the following networks (a short reference name is to the right of the arrow): Environment Canada’s (EC)1 new Mesonet surface weather stations reported each minute (i.e. ‘minutely’) and marine buoy reported every 10 minutes. This higher temporal resolution is in contrast 1 As of November 2015, Environment Canada has become Environment and Climate Change Canada (ECCC). June 7, 2016 Pan Am Dataset User Guide Page 5 of 59 DMF Team Pan Am Dataset User Guide to the normal hourly reporting from the existing surface and marine weather networks. The new Mesonet stations included:  MSC Reference Climate surface weather stations  RCS-minutely (3 stations; CA-RCS network)  MSC Compact weather stations  COMPACT-minutely (40 stations; CA-COMPACT network)  EC’s Science and Technology Branch (S&T) ATMOS stations  ATMOS-minutely (10 stations; CA-ATMOS network)  MSC Ultraviolet stations  UV-minutely (4 stations; UV network)  MSC WatchKeeper™ moored buoy  WATCHKEEPER (1 buoy; WATCHKEEPER network)  NAV CANADA (NC) AWOS minutely weather stations  NC-AWOS minutely (6 stations; NC AWOS network) To associate the station identifier with a particular network or station type, please refer to the station list in Appendix 6.5. 2.2 Purpose and Scope The main purpose of the CSV files is to provide easy access to the Legacy Pan Am dataset for internal and external clients who prefer to not interact with archived XML files stored within the DMS. If clients want to view the full decoded XML, each observation will be accessible via its own URI within the DMS, or possibly a file system containing the XML files referenced using a filename (e.g. the Canadian Meteorological Centre’s (CMC) DataDepot available through dd.weather.gc.ca). The observational elements included in the majority of the existing networks are from these basic observation groups:  pressure  wind  temperature  humidity  precipitation  radiation In addition to the elemental data reported from the Mesonet, the CSV file may also contain optional quality assessment (Qa) information in the form of a summary quality flag beside each element, whenever available. This flag's value is computed by considering any incoming Qa performed on the element at the source (so-called “NativeQC”), any real-time automatic Qa conducted in-house by the DMS, or a combination of the two. The absence of a Qa summary flag on an element in the file indicates the quality may not have been assessed, so is unknown. 2.3 Intended Audience Any clients interested in performing weather or climate analyses during the period surrounding the 2015 Pan Am and Parapan Am Games in Southern Ontario will find the CSV file product easy to ingest and interact with. June 7, 2016 Pan Am Dataset User Guide Page 6 of 59 DMF Team Pan Am Dataset User Guide 3. Data Standardization 3.1 Units All DMS elements have a standard element assigned by the DMS based on the data class (e.g. wind velocity, temperature, pressure, etc.), although some variations exist for certain elements within a class. For example, most pressure-related elements will have a DMS standard unit of hPa, but in the case of altimeter, the standard unit is ‘in Hg’ (i.e. inches of mercury) since it is used by a specific client community and in practically all cases measured and used in that unit. Examples of typical standard units for some classes are:  precipitation amount = mm  wind speed = km/h  pressure = hPa  visibility = km  height = m  temperature = °C The conversion to DMS standard units is only done at the last possible moment, typically when data leaves the DMS via product generators or during the population of custom datamarts and display tools where client requirements need to be satisfied. The DMS standard units will meet the majority of client needs, but inevitably some clients will have different preferences and will need to do some conversions. To assist in this, Appendix 6.3 has a list of unit conversions so clients can see what was used to convert incoming units to DMS standard units for a given element, or to apply client-side conversions. The dataset tables in Section 5 show the incoming uom (unit of measure) and the standard units they were converted to. 3.2 Codes A “standard” code table is also associated with any incoming code tables for a given element. The master list of code tables that the DMS maintains has cross referenced similar code tables for a given entity to a DMS standard table, which is in effect a superset of all the similar code tables for that entity. This allows for products or clients to use one standard code value for an element to map to their preferred codes, expressions or interpretation rather than having to map from many different tables for a given element across multiple networks. For example, present weather is reported by many aviation networks, but most use different code tables or even text strings (note, in the DMS text strings that are controlled vocabulary are also treated as if they were codes). June 7, 2016 Pan Am Dataset User Guide Page 7 of 59 DMF Team Pan Am Dataset User Guide Example 1. Present weather arriving to the DMS is mapped to a code value in a DMS standard code table. Below is an example: Network Observation Incoming Incoming Incoming DMS code- DMS code-type DMS code code-src code-type value src value Minutely Snow detected ca precipitation_type 4 std_code_src present_weather 374 MSC and S&T surface weather networks (CA Minutely) To see the meaning of the standard code table values for coded elements, please refer to Appendix 6.4. With this information clients can map the standard code values to their preferred expression or code using a single mapping table, rather than one for each network. June 7, 2016 Pan Am Dataset User Guide Page 8 of 59 DMF Team Pan Am Dataset User Guide 4. File Format and Structure 4.1 Structural Organization of the File The format of the product is a CSV file which can be viewed in either Microsoft Excel or any form of text editor, e.g. Notepad. Below is a sample of the raw file in a text editor: Data Description: The first row of the file contains the headers that sequentially identify the data elements in the rows below. When imported into a spreadsheet these would be the column headers. The first few fields are metadata elements about the observation. For instance, one could find the time of observation, the reporting station identifier (e.g. MSC ID, TC ID, etc.), the station's latitude, longitude and elevation, etc. Note that some datasets may not have data for all these metadata elements. Below is an example of the metadata in columnar form: June 7, 2016 Pan Am Dataset User Guide Page 9 of 59 DMF Team Pan Am Dataset User Guide NULL & MSNG: A null in the spreadsheet indicates that an element was never received or derived. A value of missing (MSNG) means that the element was received or a derived element was created, but the value was not available. For observed elements this could be because the value could not be measured or was illegal (e.g. over-ranged). For derived elements it means that the derivation could not proceed due to the unavailability of usable input elements (e.g. too many missing observations or the ones that were available failed quality assessment). Observational Data Elements: From row 2 onwards in the file are the actual data associated with each header field in row 1. The first set of elements in these rows contain data associated with station information, while the remainder to the right of this set contain actual observational data. The observational data are to be treated as pairs; the element’s value followed by its Qa flag (if available) E.g. air temperature followed by the air temperature Qa flag, wind speed followed by the wind speed Qa flag, etc. If Qa information is available for the element, then the corresponding Qa column is populated, otherwise it is left empty. Below is an example where the first two elements do not have a Qa flag, whereas the third element does: See Section 4.3 for details on the creation of the Qa flag and the meaning of the code value. 4.2 Date-Time The date-time of the observation is in UTC (otherwise known as Greenwich Mean Time, or GMT) and is present in the column “OBS_DATETIME_UTC”. The format of the date and time might differ depending on the program being used to view it. When viewing the file in a text editor such as Notepad, the format is as follows: 2015/08/01 18:01 Year/ month/ day time (hour:minute, 24 hour clock) However, the same data in Excel will be formatted to: 01/08/2015 18:01 Day/ month/ year time (hour:minute, 24 hour clock) Clicking on the cell in Excel will result in the format: 01/08/2015 6:01:00 PM June 7, 2016 Pan Am Dataset User Guide Page 10 of 59