A new globally-complete monthly historical gridded mean sea level pressure data set (HadSLP2): 1850-2004. Rob Allan and Tara Ansell, Hadley Centre, Met Office, UK Journal of Climate (accepted) Corresponding author address: R. J. Allan, Hadley Centre, Met Office, FitzRoy Rd, Exeter, Devon EX1 3PB, United Kingdom. email: [email protected] - 1 - Abstract We present an upgraded version of the Hadley Centre’s monthly historical mean sea level pressure (MSLP) data set (HadSLP2). HadSLP2 covers the period 1850 to date, and is based on numerous terrestrial and marine data compilations. Each terrestrial pressure series used in HadSLP2 under went a series of quality control tests and erroneous or suspect values were corrected where possible or removed. Marine observations from the International Comprehensive Ocean Atmosphere Data Set were quality controlled (assessed against climatology and near neighbours) and then gridded. The final gridded form of HadSLP2 was created by blending together the processed terrestrial and gridded marine MSLP data. MSLP fields were made spatially- complete using Reduced-Space Optimal Interpolation (RSOI). Grid point error estimates were also produced. HadSLP2 was found to have generally stronger subtropical anticyclones and higher latitude features across the Northern Hemisphere than an earlier product (HadSLP1). During the austral winter, however, it appears that the pressures in the southern Atlantic and Indian Ocean mid- latitude regions are too high; this is seen in comparisons with both HadSLP1 and with ERA 40. Over regions of high altitude, HadSLP2 and ERA-40 showed consistent differences suggestive of potential biases in the reanalysis model, though the region over the Himalayas in HadSLP2 is biased compared to HadSLP1 and improvements are required in this region. Consistent differences were also observed in regions of sparse data, particularly over the higher latitudes of the Southern Ocean and in the south eastern Pacific. Unlike the earlier HadSLP1 product, error estimates are available with HadSLP2 to guide the user in these regions of low confidence. An evaluation of major phenomena in the climate system using HadSLP2 provided further validation of the data set. Important climatic features/indices such as the North Atlantic Oscillation, Arctic Oscillation, North Pacific Index, Southern Oscillation Index, Trans Polar Index, Antarctic Oscillation, Antarctic Circumpolar Wave, East Asian Summer Monsoon Index and the Siberian High Index have all been resolved in HadSLP2, with extensions back to the mid-19th century. - 2 - 1. Introduction The earliest charts and maps of monthly mean sea level pressure (MSLP) over the globe were pioneered by the likes of Buchan (1867,1869,1889), Hildebrandsson (1897) and Teisserenc de Bort (1883,1889). These entirely hand drawn map products were subsequently built on into the twentieth century by other scientists, culminating in the work of Lamb and Johnson (1966) who produced global MSLP charts for the months of January and July back to 17501. In the age of the computer and sophisticated objective analysis techniques, several efforts have been made to develop high quality historical monthly mean sea level pressure (MSLP) data sets covering the Northern and Southern hemispheres and extending to global dimensions (e.g. Trenberth and Paolino, 1980; Jackson, 1986; Jones, 1991; Barnett and Jones, 1992; Jones et al., 1999b; Kaplan et al., 2000; Luterbacher et al., 2002; Smith and Reynolds, 2004; Ansell et al., 2006). Other than reanalysis products (Kalnay et al., 1996; Kistler et al., 2001; Uppala et al., 2005), the major efforts to develop globally-complete MSLP products blending historical terrestrial and marine MSLP data have been made by the Hadley Centre in the UK (Allan et al., 1996; Basnett and Parker, 19972) This paper details the development and evaluation of a new version of the Hadley Centre’s globally-complete monthly historical MSLP product (HadSLP2) on a 5o latitude by longitude grid covering the period from 1850 to 2004, with a near-real-time update version, HadSLP2r, also available. HadSLP2 is the most recent version of the Hadley Centre’s historical globally-complete gridded MSLP data products: GMSLP2 and HadSLP1 (Allan et al., 1996; Basnett and Parker, 1997). Its construction involved a major digitisation of hard copy and scanned surface pressure data from historical sources from all over the globe (see Appendix 1 and References). This material was then used to extend, fill gaps in, and produce additional station time series that could be added to existing collations of electronic terrestrial (land and island) surface pressure records. Finally, these terrestrial records were all reduced to MSLP and blended with marine (ship-based) MSLP data from the International Comprehensive Ocean Atmosphere Data Set (ICOADS) (Worley et al., 2005), which combines the Met Office's Marine Data Bank with the previous version of 1 April and October charts were never published, but copies are held by the Hadley Centre, Met Office, UK 2 Principally version 2 of the Global Mean Sea Level Pressure (GMSLP2) gridded monthly data set and HadSLP1 (an updated version of GMSLP2). - 3 - COADS (Slutz et al., 1985; Woodruff et al., 1993, 1998). These blended, quality controlled and gridded fields were made spatially-complete by using Reduced-Space Optimal Interpolation (RSOI) (Kaplan et al., 1997, 2000). Grid point error estimates and numbers of observations fields have also been produced. In this regard, HadSLP2 is superior to HadSLP1 and will ultimately be available as interpolated (HadSLP2), uninterpolated (HadSLP2.0), and near-real-time (HadSLP2r) products from http://www.hadobs.org. 2. Data development & sources Atmospheric pressure data from historical terrestrial and marine sources were collected, collated, digitised, quality controlled and blended together to form the HadSLP2 data set. This undertaking involved a concerted search of data sources held by the UK Met Office Library and Archives, the use of scanned records from various WWW sites (see Appendix 2), and requests to individual meteorological services around the world for specific station series. The prime sources for global monthly terrestrial (land, island and weather ship) data were the long duration records and/or ongoing climatic data compilations of the US Signal Office (Washington, War Department, 1870; U.S. Signal Office, 1871 – 1889), US International Observations (Washington, Signal Office, 1875-1881, 1881-1883, 1883-1885, 1884-1888), Hildebrandsson (1897), Lockyer (1908, 1909), Reseau Mondial (Air Ministry, Meteorological Office, 1910-1934), World Weather Records (Clayton, 1927, 1934, 1947; U.S. Weather Bureau, 1959; U.S. Environmental Science Services Administration, 1965-1968; NOAA, National Climatic Data Center, 1979-1985, 1987-1994, 1995-1999; 2005; WeatherDisc Associates, 1994), Monthly Climatic Data for the World (Washington, Weather Bureau, 1948-1967; Washington, Environmental Science Services Administration, 1968-1970; Washington, NOAA, EDS: 1971- 2004), CLIMAT (World Meteorological Organisation, 1995), the Global Historical Climate Network (GHCN) versions 1 and 2 (Vose et al., 1992; Wuertz per. com., 2002), the GCOS Surface Network (GSN) (http://lwf.ncdc.noaa.gov/oa/climate/gsn/gsnmap.html) and Young (1993). These were augmented by more regional, and various country and colonial records, data from European - 4 - Union (EU) funded projects (eg. monthly averages of daily data from IMPROVE (Camuffo & Jones, 2002), ADVICE (Jones et al., 1999b) and monthly averages of daily data from EMULATE (Ansell et al., 2006): see Appendix 2), and various publications by meteorological services throughout the world (see References and Appendix 1 for specific details). In addition, a number of individual station pressure records were provided through various contacts in meteorological services or research institutions world-wide (see Acknowledgements). As a consequence of this major effort, the number of terrestrial stations used in the construction of HadSLP2 has increased from 718 in HadSLP1 to 2228 in the new version (see Figure 1 for station distribution through time). Of these 2228 stations, 615 have series longer than 100 years, though 275 have less than 20 years of observations. Not surprisingly many are in Europe. Existing HadSLP1 stations were extended to 2004 using CLIMAT records, where available. Particular efforts have concentrated on improving coverage over Antarctica (region of strong trends) and over particularly sparse regions of Africa, South America, Russia and Asia. Marine observations from the International Comprehensive Ocean Atmosphere Data Set (ICOADS, Worley et al. (2005)) were also used in the construction of HadSLP2. ICOADS is a recent blending of the previous version COADS (Woodruff et al., 1993) with the Met Office’s Marine Data Bank, and also includes several million newly digitised observations (e.g. US Maury collection and the Japanese Kobe Collection), significantly improving coverage in the 1850-1860s and around the First World War years (Figure 1). 3. Methodology In order to create globally-complete gridded terrestrial and marine based MSLP fields, a number of steps were required. In section 3a) we describe the quality control procedure adopted for the terrestrial observations, in 3b) our quality control and gridding strategy for the marine observations is outlined, in 3c) we describe how these quality controlled terrestrial observations and gridded marine fields are blended. To create globally-complete fields we employ RSOI, described in section 3d. - 5 - a). Quality control – Terrestrial data Work on developing long, high quality MSLP stations series can be very manually intensive and time consuming (e.g. Madras (Chennai), Allan et al., 2002; Nagasaki, Konnen et al., 2003; Quebec, Slonosky, 2003). Unlike the studies cited above, which have focused on specific individual series, the number of stations requiring quality control in HadSLP2 necessitated a more automated quality control procedure being set up. While the automation procedure cannot compare to individual intense scrutiny, it has enabled us to include a very large number of series. With this procedure, each station record underwent a series of quality control checks, after initially being corrected for attached temperature and standard gravity (where required), converted to standard units of hPa, and reduced to MSLP. - Firstly, a check for internal consistency was performed. Each station series was compared to its monthly mean and standard deviations, calculated over the most recent and/or reliable period, in order to remove gross outliers caused by errors in station heights and misprints in data records. Anomalous values that were greater than 4 times the standard deviation were removed. - A large number of our station series come from multiple sources, with considerable overlapping years. We therefore, secondly, blended sources to create a single MSLP series for each station. When combining the sources, preference was given to those deemed to be more reliable. i.e. had required least quality control hitherto. - Thirdly, near neighbour checks were performed. Applying a similar technique to multiple qualitative comparisons and adjustments (MCA), described in Slonosky et al. (1999), each series was compared to its four nearest neighbours of similar length (to the north, south, east, west), and then flagged and adjusted only if a discontinuity was detected against 3 or more neighbours. This method however relies on having reliable neighbour series of complementary length. Unfortunately these were not always available, and so in these cases the station series was also compared to the nearest grid point value in HadSLP1 (this - 6 - check was only available for the period 1871-1998). We note that if all 4 neighbours also contained a discontinuity, no problem would be flagged. - Fourthly, we check for break points in data series by applying a Kolomogorov-Smirnov (KS) test (Press et al., 1992). Thorne et al. (2005) employed a KS test for homogenising radiosonde observations. This technique works by assessing the probability that two populations arise from the same distribution. A seasonal mean difference series is calculated (station ‘target’ series minus average neighbour series) and the KS test is applied to a time series with a 15 season window on either side of the current point. If a break point is flagged, corrections are then applied. The adjustment is calculated by taking the difference between the neighbour and the target series; this adjustment value is added to the target anomaly values before the break point, to make the series consistent with current data. - Fifthly, a manual adjustment of break points was considered in cases where suitable neighbour series were not available for the KS test. If metadata information and the series itself indicated obvious break points, manual adjustments were applied. The adjustments were calculated by taking the difference between the mean of the break point period with the mean of a reliable period in the same series, similar to MCA (Slonosky et al., 1999). Quality control procedures highlighted a number of issues of concern to long-term global pressure data set development. In many circumstances station series could only be completed by using all available sources, and often no major active repository (e.g. World Weather Records, Monthly Climatic Data for the World, CLIMAT, GSN or GHCN2) held the full station record even up to recent times. In addition, errors and deficiencies in pressure series were detected frequently in all of the major compilations from which data were being drawn. For instance, the quality control applied to MSLP series in the GHCN2 data set was found to have removed a substantial number of real data values which it took to be too extreme. Yet, even with apparently over zealous quality control checks, GHCN2 was still found to have retained a number of what were very - 7 - obvious erroneous data values and also station time series with distinct changes in data variance over time. These problems were detected during near neighbour checks, and erroneous errors have been corrected where possible. Some data variance problems were resolved by the replacement of affected series by versions from other sources, but others remain. This variance issue will be addressed in the construction of the international pressure data bank (Appendix 3) and will feed into subsequent versions of HadSLP2. Finally, efforts to focus on climatic data series from non- urbanised sites for the detection of anthropogenic climate change have seen the major active data compilations drop many long-term urban records, making updating of pressure data from such locales more difficult. This is likely to have even greater impact on efforts to develop near real- time pressure data compilations. b). Quality control and gridding – Marine data Marine observations from ICOADS were quality controlled and gridded using the marine data system (MDS) version 2, developed at the Hadley Centre. MDS has been used to grid sea surface temperature (SST) and surface air temperature observations (see Rayner et al., 2006 Section 2c for a full description). The quality control procedure involves a climatology check, using 5-day (pentad) fields3, and a near neighbour ‘buddy check’. Unlike the buddy check described in Rayner et al. (2006), which utilised neighbouring observations both forwards and backwards in time, the buddy check used here checks only against spatial buddies, not temporal. This is appropriate for MSLP given its rapid variations. MSLP observations passing these tests were then corrected as appropriate. These corrections included a diurnal cycle correction, using the gridded phase and amplitude fields of Dai and Wang (1999). A correction was also applied for an anomalously low (negative) MSLP bias in the US Maury collection. Both corrections were made using procedures described in Ansell et al. (2006). Previously undetected duplicates in the ICOADS database were also removed. 3 The pentad climatology was derived from monthly HadSLP1 fields, interpolated to pentad resolution using a cubic spline fit. - 8 - Next, data for each pentad were gridded onto a 1° latitude by longitude grid taking the winsorised mean (i.e. trimming the values that exceed a certain threshold: Barnett and Lewis 1994 ). This served to reduce the influence of any outliers that remained after the quality control procedure (Afifi and Azen, 1979). Monthly averages were then formed and the number of pressure observations in each grid box recorded. The measurement and sampling error for each month and grid box was also calculated as part of the MDS gridding procedure (see Rayner et al., 2006, section 3b). The sampling error is associated with not having enough observations to represent the ‘true’ grid box MSLP value; it is also known as the representivity error. The MDS gridding technique differs to that used previously for HadSLP1 and EMSLP (Ansell et al., 2006), in that it no longer contains a smoothing and infilling technique. While this reduces the coverage somewhat, the over smoothing is removed and the subsequent ‘number of observation’ fields are now more meaningful. The reduction in coverage is compensated by the increase in observations in the ICOADS database. c). Blending terrestrial and marine MSLP The final HadSLP2 product was constructed by blending together the quality controlled terrestrial with the gridded marine fields. For each month and in each year from 1850 to 2004, and in each 1° x 1° grid box, the marine grid box value and all terrestrial MSLP observations (if present) were collated. Residuals were formed by subtracting a monthly background field from each terrestrial observation and marine grid box value and then the median value (both land and marine) was selected. This gave greater weight to land observations in coastal regions. All the 1° x 1° median values were then averaged to 5° x 5° grid point values, taking account of their spatial distribution. Absolute pressures were formed by adding back the background field. The background field used here was based on HadSLP1. Prior to 1871, when HadSLP1 begins, we have used a monthly climatology (30 year average from 1871-1900). Post 1998, NCEP-NCAR reanalyses were used as the background field. - 9 - The blended land and marine fields were then visually quality controlled; suspect grid box values were deleted or smoothed as appropriate. The coverage prior to reconstruction is shown in Figure 1 for a number of decades. The blended data set, with spatially incomplete fields, is known as HadSLP2.0. It is available on a 5o latitude by longitude grid, covering the period 1850-2004; number of observations and measurement and sampling error gridded fields are also available. d). Reconstruction The blended and gridded fields were made spatially-complete by using RSOI (Kaplan et al., 1997, 2000). Ansell et al. (2006) applied this technique over the European-North Atlantic region with success; we adopt a similar methodology, working here with monthly fields. Complete MSLP anomaly fields were reconstructed using the leading 34 Empirical Orthogonal Function (EOF) modes and the measurement and sampling error field. For this error field we followed Ansell et al. (2006), in using the 1961-1990 root mean square of 30 combined marine and land fields of measurement and sampling error for given calendar months. For the marine observations the measurement and sampling error for each month was calculated as part of the MDS gridding procedure (see above). In addition we took account of the errors inherent in the ship observations. A value of 0.25 hPa for geographically random one sigma bias was estimated from the differences between synoptic charts and operational model analyses and added vectorially to the sampling error (Ansell et al., 2006). Over land, estimated errors were based on the altitude of the station. Following Ansell et al. (2006), an estimate of h/1500 was used as the bias associated with the reduction to mean sea level, where h is the altitude of the station (in meters). Again 0.25 hPa was added (vectorially) to the elevation-related bias, to reflect the random bias error. In grid cells with both land and marine data, the errors ascribed were a combination of these land and marine components. EOFs were calculated over the 1948-2004 epoch, the most recent and reliably observed period, also overlapping with the NCEP-NCAR reanalysis product. The fields used to calculate the EOFs were a Poisson blending (Reynolds, 1988) of the observed anomalies with NCEP-NCAR - 10 -