The influence of observation errors on analysis error and forecast skill investigated with an observing system simulation experiment 1,3 1,3 2,3 N.C. Priv´e, R. M. Errico, and K.-S. Tai e l c i t Corresponding author:Nikki C. Priv´e, Code 610.1 NASA/GSFC, Greenbelt, MD 20771, USA. r ([email protected]) A 1Goddard Earth Sciences Technology and Research Center, Morgan State University, d Baltimore, Maryland, USA. e 2Science Systems and Applications, Inc., Greenbelt, Maryland, USA. t p 3Global Modeling and Assimilation Office, Goddard Space Flight Center, Greenbelt, e Maryland, USA. c c A Thisarticlehasbeenacceptedforpublicationandundergonefullpeerreviewbuthasnotbeenthrough the copyediting, typesetting, pagination and proofreading process, which may lead to differences between this version and the Version of Record. Please cite this article as doi: 10.1002/jgrd.50452 (cid:2)c2013 American Geophysical Union. All Rights Reserved. Abstract. The National Aeronautics and Space Administration Global Modeling and Assimilation Office (NASA/GMAO) observing system sim- ulation experiment (OSSE) framework is used to explore the response of anal- ysis error and forecast skill to observation quality. In an OSSE, synthetic ob- servations may be created that have much smaller error than real observa- e tions, and precisely quantified error may be applied to these synthetic ob- l servations. Three experiments are performed in which synthetic observations c with magnitudes of applied observation error that vary from zero to twice i the estimated realistic error are ingested into the Goddard Earth Observ- t r ing System Model (GEOS-5) with Gridpoint Statistical Interpolation (GSI) A data assimilation for a one-month period representing July. The analysis in- crement and observation innovation are strongly impacted by observation error, with much larger variances for increased observation error. The anal- d ysis quality is degraded by increased observation error, but the change in root- e mean-square error of the analysis state is small relative to the total analy- t sis error. Surprisingly, in the 120 hour forecast, increased observation error p only yields a slight decline in forecast skill in the extratropics and no dis- cernible degradation of forecast skill in the tropics. e c c A (cid:2)c2013 American Geophysical Union. All Rights Reserved. 1. Introduction There are multiple sources of error in numerical weather analysis and prediction includ- ing model error, observation instrument and representativeness error, errors introduced by the data assimilation process itself, and physical-dynamical error growth. Because the e true state of the atmosphere remains unknown, it is not possible to directly assess these l errors or their impact on analysis quality or forecast skill. Many efforts have been made c to investigate the impact of initial condition errors on forecast skill, such as with idealized i identical or fraternal twin experiments (e.g. Tribbia and Baumhefner [2004]), but these t studies have not considered errors in the context of data assimilation systems. r A Previous studies (e.g. Tyndall et al. [2010], Irvine et al. [2011]) have examined the role of observation error in data assimilation, primarily in the form of the weighting of obser- vational data versus the background. Changing the specified observation error variance d or background error variance in a data assimilation system (DAS) alters how closely the e analysisfielddrawstotheobservationscomparedtothebackground. Thisstudyinsteadis focused primarily on how the observation errors themselves impact qualities of the model t analysis and forecast fields. p There are many unanswered quantitative and qualitative questions about how obser- e vation error impacts the errors of analysis and subsequent forecasts given that the DAS c is designed as an error filter and smoother (Daley [1991]). Modern DAS are based on elegant mathematical theory, as oulined in the Appendix, that unfortunately offers only c limited insight into answers to these questions because of the many unsupported assump- A tions generally implied for their computationally efficent application. Answers are also (cid:2)c2013 American Geophysical Union. All Rights Reserved. not forthcoming when using real observations since in that context the true state being analyzed is not sufficiently well known. In contrast, an Observing System Simulation Experiment (OSSE) alleviates many of these difficulties since relevant errors can be di- rectly calculated from the accurately known truth provided (Errico et al. [2013]). As long as the OSSE is a faithful simulation of reality, it can provide valuable insight into these e questions. l An OSSE suitable for this problem has been developed at the National Aeronautics and c Space Administration (NASA) Global Modeling and Assimilation Office (GMAO; Errico i et al. [2013], Priv´e et al. [2013]). It provides a tool for investigating how errors in sources t r of information or algorithms impact the analysis, background, and forecast errors. In A addition, the observation errors in an OSSE can be directly manipulated to explore the impactofobservationerrorontheanalysisqualityandforecastskill. Inthiswork, aseries of experiments with varied observation error are performed using the GMAO OSSE to d explore the influence of observation error in an operational numerical weather forecasting e system. t The motivating factors for this study include both the design of OSSEs and the effects p of observation error when assimilating real observations. The development of realistic observation errors for synthetic observations in OSSEs has been a challenging problem for e decades. Here,theimportanceofaccuratelyrepresentingobservationerrorsisinvestigated c bytestingtherespsonseoftheOSSEframeworktoarangeofobservationerrormagnitudes c from minimization of observation errors to gross overestimation of observation errors. A variety of metrics are employed, including explicit measures of analysis error. The A importance of proper weighting of error covariance matrices is also explored. (cid:2)c2013 American Geophysical Union. All Rights Reserved. Details of the GMAO OSSE framework and the experimental setup are given in Sec- tion 2. The influence of observation error on increment and error statistics of the data assimilation products is described in Section 3. Likewise the effect of observation error on forecast skill is presented in Section 4 and on observation impact metrics calculated with an adjoint model in Section 5. Finally, the results will be discussed in Section 6. e 2. Setup l c The GMAO OSSE framework is used for all experiments. This system is described i in detail by Errico et al. [2013]; a brief synopsis will be given here. An OSSE consists t of several components: a long, free model integration called the Nature Run (NR) that r represents the ‘truth’; a set of synthetic observations produced from the Nature Run A fields for all data types currently assimilated to create initial conditions for numerical weather prediction; an observation error algorithm to add otherwise missing instrument d and representativeness errors to observations; and a data assimilation system employing e a second forecast model for ingesting the synthetic observations. The NR used for the GMAO OSSE was generated by the European Centre for Medium- t Range Weather Forecasts (ECMWF) using the c31r1 version of their operational forecast- p ingmodel. Themodelwasfreelyrunfrom01May2005to31May2006atT511resolution e with 91 vertical levels and 3-hourly output. Prescribed boundary conditions included the c sea surface temperature and sea ice content observed during the NR period; all other fields were generated by the ECMWF model. The NR has been evaluated to ensure that c the model characteristics are suitable for use in OSSEs (Reale et al. [2007], McCarty et al. A [2012]). (cid:2)c2013 American Geophysical Union. All Rights Reserved. Synthetic observations were created at the GMAO for both conventional and radiance data types. Conventional data were computed by interpolating the NR fields according to the temporal and spatial locations of archived observations from corresponding dates during 2005-2006. Radiance observations were similarly generated using the Commu- nity Radiative Transfer Model version 1.2 (CRTM, Han et al. [2006]) with a simplified e treatment of the clouds based on cloud fractions from the NR. l A set of baseline observation errors were calibrated to match some assimilation statis- c tics of real data ingested into the same versions of GSI and GEOS-5. Uncorrelated errors i were added to all observation types and an additional component of correlated errors was t r added to some types. Vertically correlated errors were added to conventional sounding A data types, horizontally correlated errors were added to AMSU, HIRS, and MSU observa- tions, channel correlated errors were added to AIRS, and both vertically and horizontally correlated errors were added to satellite wind observations. No correlation of errors was d appliedbetweendifferentdatatypes, andnoobservationerrorbiaswasadded. Theobser- e vation errors were callibrated so that covariances of observation innovations and variances t of analysis increments in the OSSE matched corresponding statistics computed for the p DAS applied to real observations (Errico et al. [2013]). As a result of this tuning, the added errors may contain compensations due to mismatches between the OSSE and real e observation results of actual background error covariances. c In addition to explicitly added errors, the synthetic observations contain a small but c unspecified quantity of implicit representativeness error. This error arises from differences between interpolations used to create the synthetic observations applied on the NR and A DAS model grids. Errors are also introduced to the radiance observations through dif- (cid:2)c2013 American Geophysical Union. All Rights Reserved. ferences between treatments of cloud in the radiative transfer schemes applied to the NR and DAS gridded fields. ThenumericalweatherpredictionmodelusedfortheOSSEexperimentsistheGoddard Earth Observing System Model, Version 5 (GEOS-5) with Gridpoint Statistical Interpo- lation (GSI) data assimilation system (Kleist et al. [2009], Rienecker et al. [2008]). The e modelresolutionis0.5◦ latitudeand0.625◦ longitudewith72verticallevels. Thebehavior l of the OSSE forecasts has been validated in comparison to reality by Priv´e et al. [2013], c where it was found that the forecast skill of the OSSE is slightly better than for real data, i but the relative impact of different data types is well represented. t r For these experiments, the OSSE is cycled from 15 June 2005 to 05 August 2006, with A 120 hour forecasts launched daily at 0000 UTC. The first two weeks are discarded as a spin-up period, and results are calculated only for the month of July. Three experimental cases are tested: a Control case using the baseline set of synthetic observations with d calibrated observation errors described by Errico et al. [2013]; a Perfect case in which no e errorsareaddedtothesyntheticobservations; andacaseinwhichobservationerrorswith t standard deviation twice the magnitude as the Control case are added to the synthetic p observations, called the Double case. The explicitly added errors in the Double case are perfectly correlated to the errors in the Control case, with twice the magnitude. Table 1 e displays the attributes of all of the experimental cases included in this study. These three c cases can be compared to show the progression of the effects of observation errors as the c errors are increased from near zero to large values. For Perfect, Control, and Double cases, the background and observation error covari- A ances assumed by the GSI are not altered from the operational values. This preserves the (cid:2)c2013 American Geophysical Union. All Rights Reserved. GSI Kalman gain matrix and thus the weightings between observations and background. For none of these three OSSE experiments is this Kalman gain truly optimal since the assumed error covariances are not the actual ones. Even for assimilation of real observa- tions, the specified background error covariance likely differs from the actual covariances for some components and the specified observation error ignores significant correlations e known to exist for some observation types and instead grossly inflates the assumed er- l ror variances to partly compensate for this neglect. For the Perfect and Double cases, c the departures from optimality may be greater, but even in these cases more optimality i wouldrequireuseofaretunedassumedbackgrounderrorcovariance. Suchretuningwould t r partly offset use of a more appropriate assumed observation error variance. For any of the A experiments, assumption of truly accurate error covariances would produce the optimal analysis; i.e., analysis with minimum expected error variance given the observation and background errors. Results from these experiments therefore provide an upper bound on d what the corresponding optimal error variances would be. e An additional experiment is performed using the added observation errors from the t Double case, but with the standard deviations of observation errors used by the GSI p increased by a factor of two, denoted as the ‘Double GSI Adjusted’ case. While this also does not result in an identical match between the true observation error covariances and e the GSI error covariances, some underestimation of observation error covariances by the c GSI in the Double case should be relieved in this case. A case with greatly reduced GSI c error using the synthetic observations with no explicitly added error is not performed due to concerns that the data assimilation algorithm would become ill-conditioned. A (cid:2)c2013 American Geophysical Union. All Rights Reserved. For validation of certain analysis and forecast statistics, a parallel case is run using archived real data from the same time period instead of the synthetic observations. This caseisdesignatedasRealandisrunusingthesameGEOS-5andGSIversionandsettings as deployed in the OSSE. The analog of the Real case in the OSSE environment is the Control case, as the explicitly added observation errors in the Control case have been e calibrated to specifically match the observation innovations and analysis increments in l the Real case. A ‘Real Plus Error’ case is performed analogously to the Double case, c wherein errors of the real observations are increased by explicitly adding errors with the i same covariances used in the Control case to the real data. In this case, the observation t r error covariances are not expected to be identical to those used in the Double case, but A theimpactsofsignificantlyincreasingtheobservationerrormaybecheckedtoensurethat the OSSE results are not unrealistic. The background error covariances used by the GSI are taken to be the operational d 2011 GSI/GEOS-5 covariances for all experiments. Due to improvements in the observing e network between 2005 and 2011, these background error covariances may underestimate t the true background errors when working with the 2005 observational dataset. In addi- p tion, the true background error covariances may differ between experimental cases due to ingestion of different qualities of observation errors. e c 3. Analysis Quality The observation innovation, d , measures the differences between observations and the i c background state, A d = yo−H [xf(t ))] (1) i i i i (cid:2)c2013 American Geophysical Union. All Rights Reserved. where t is the time, yo is the observation vector, xf is the forecast model state vector, i i and H is an observation operator in standard notation [Ide et al., 1997]. Observation innovation statistics are expected to be strongly affected by the magnitude of observation errors, as yo is directly affected by observation error and xf(t ) is indirectly affected by i i observation error that has been ingested in earlier cycles of the DAS. e The analysis increment, or analysis minus background (xa(t )−xf(t )), measures the i i l amountof‘work’donebythedataassimilationsystemingeneratingananalysisstatefrom c the initial background state. The root-mean-square-error (RMSE) of such a difference is i calculated as an areal and temporal mean t (cid:2) (cid:3)(cid:5) (cid:6) (cid:6) Ar RMSEI = (cid:3)(cid:3)(cid:4) Ni=1 λλwe Nφφsn(cid:6)(λxλwea((cid:6)tφφis)n−Re2xcfo(stiφ))d2φRde2λcosφdφdλ (2) where x is the analysis field and x is the background field for N analysis states, R is a f e the radius of the earth, φ is the latitude between φ and φ and λ is the longitude between s n d λ and λ . w e e Figure 1 shows a sampling of global variances of observation innovation for the Perfect, Control, and Double experimental cases for rawinsonde (RAOB) temperature and wind, t p GOES infrared (IR) cloud drift winds, and AMSU-A brightness temperatures. The vari- ance of observation innovations for the Control case is intermediate to that seen for the e Perfect and Double cases. c Ifthetrueerrorcovariancesofthebackground,B,werethesameforthethreetestcases, andiftheexplicitlyaddedobservationerrorsareuncorrelatedwiththebackgrounderrors, c then the difference in variances of observation innovation between each pair of cases is A simplythedifferenceinthevariancesoftheobservationerrorsthemselves. Asthestandard deviation of the observation error in the Double case is twice the standard deviation of (cid:2)c2013 American Geophysical Union. All Rights Reserved.