Tellus000,000–000(0000) Printed5June2013 (TellusLATEXstylefilev2.2) The role of model and initial condition error in numerical weather forecasting investigated with an observing system simulation experiment By Nikki C. Priv´e1,2(cid:2) and Ronald M. Errico1,2 1MorganStateUniversity,Baltimore,Maryland, USA;2NationalAeronautics and SpaceAdministration, Global Modeling and AssimilationOffice,Greenbelt,Maryland,USA; (Manuscriptsubmitteddate) ABSTRACT Aseriesofexperimentsthatexploretherolesofmodelandinitialconditioner- rorinnumericalweatherpredictionareperformedusinganobservingsystem simulation experiment (OSSE) framework developed at the National Aero- nautics and Space Administration Global Modeling and Assimilation Office (NASA/GMAO). The use ofanOSSEallowsthe analysisandforecasterrors tobe explicitlycalculated,anddifferent hypotheticalobservingnetworkscan betestedwithease.Intheseexperiments,bothafullglobalOSSEframework and an ‘identical twin’ OSSE setup are utilized to compare the behavior of the dataassimilationsystem and evolutionof forecastskill with and without model error.The initial condition erroris manipulated by varying the distri- butionandqualityoftheobservingnetworkandthemagnitudeofobservation errors. Theresultsshowthatmodelerrorhasastrongimpactonboththequality oftheanalysisfieldandtheevolutionofforecastskill,includingbothsystem- atic and unsystematic model error components. With a realistic observing network, the analysis state retains a significant quantity of error due to sys- tematicmodelerror.Iferrorsoftheanalysisstateareminimized,modelerror acts to rapidly degrade forecast skill during the first 24-48 hours of forward integration. In the presence of model error, the impact of observation errors onforecastskillissmall,butintheabsenceofmodelerror,observationerrors cause a substantial degradationof the skill of medium range forecasts. (cid:2)c 0000Tellus 2 PRIVE´ ETAL. 1 Introduction Forecast skill in numerical weather prediction is affected by two types of error: initial condition error and model error. The magnitudes of model error and initial condition error havechangedoverthedecadesasforecastmodels,dataassimilation,andtheglobalobserving network have become more sophisticated (Simmons and Hollingsworth (2002), Compo et al. (2011)). Quantifying the relative impacts of these errors is of interest to determine where resources should best be expended in order to effect the greatest possible improvements in forecast skill. Initial condition error is influenced by many factors, including the quality of the ob- servations and the observational network and the handling of observation and background information by the data assimilation system (DAS). Some sources of initial condition error can be at least partially mitigated by techniques such as biascorrection to remove persistent observation error and proper weighting of the background and observation error variances in the DAS. However, data voids and formulation deficiencies in the data assimilation al- gorithms are more difficult to rectify, and in some circumstances the very methods used to attempt to improve the analysis quality may instead result in a degradation. For example, biascorrectionmayattributepersistentdifferencesbetweenobservationsandthebackground to observation biases when model bias is actually the root cause. Early theoretical exploration of the roles of model and initial condition error involved simple representations of error growth. Leith (1978) assumed exponential growth of initial conditionerrorandlineargrowthofmodelerrorforshorttermforecasts,whileLorenz(1982) and Dalcher and Kalnay (1987)included an additional quadratic growth term to account for saturation of initial condition error for longer forecasts. Simmons and Hollingsworth (2002) foundreasonablygoodagreement between thetheoreticalandassumed errorgrowthofoper- ational forecasts at the European Centre for Medium-Range Weather Forecasts (ECMWF) when systematic forecast errors were also taken into account. However, these comparisons could only be made after the first day of forecast integration because the true error during the early forecast period could not be satisfactorily estimated. While the growth of initial condition error can be estimated through a variety of means, the growth of model error is more difficult to determine. Comparison of ensemble forecasts (cid:2) Correspondingauthor. e-mail:[email protected] (cid:2)c 0000Tellus,000,000–000 MODELERRORINANOSSE 3 (Buizza, 2010) or ‘perfect model’ tests in which the differences between forecasts initialized on sequential days are examined (Lorenz, 1982) can be used to estimate the growth of initial condition errors. Statistics of model error are then estimated as residuals. Techniques such as restricted statistical correction (Schubert and Chang, 1996), and model drift (Orrell et al., 2001) have been used to attempt to quantify model error, but these methods have limitations. Observing System Simulation Experiments (OSSEs) are pure simulation exercises com- monly used toexamine the potential implementation of futureobserving networks in numer- icalweather forecasting. InanOSSE,therealworld isreplaced with alongmodelsimulation that captures the phenomena of interest. This simulation is referred to as the Nature Run (NR).Synthetic observationsaregeneratedbyspatio-temporalinterpolationoftheNRfields for both the current and future networks of observing systems. These synthetic observations are then ingested into the DAS. The OSSE should be rigorously tested and calibrated to ensure that the behavior of the system is sufficiently similar to real-world behavior to give results pertinent to the latter. In addition to evaluation of observing systems, an OSSE can be a powerful tool for investigating the behavior of data assimilation systems. Unlike the real world, in an OSSE, the ‘true’ state of the atmosphere is completely known. This allows the errors in model forecasts to be explicitly calculated, instead of the indirect methods required when working with real data. This is particularly advantageous during the analysis and early forecast periods, as the analysis and forecast errors are very difficult to quantify for real data at these times. An OSSE can also be used to determine how well the data assimilation process acts to improve the analysis state compared with the background. A global OSSE has been developed at the National Aeronautics and Space Administra- tion Global Modeling and Assimilation Office (NASA/GMAO; Errico et al. (2013), Priv´e et al. (2013b)) for use with investigations of DAS and forecast model performance. This OSSE includes both a well-calibrated configuration that emulates the real-world model per- formance, and in an ‘identical twin’ configuration, in which the NR and the forecast model are the same and there is no model error. The synthetic observation network may also be manipulated both in terms of the magnitude of the observation error and in terms of the frequency and location of observations. Thus, the GMAO OSSE may be used to investigate the roles of model and initial condition error in a more sophisticated framework than earlier idealized studies. (cid:2)c 0000Tellus,000,000–000 4 PRIVE´ ETAL. The intent of this work is to examine the relative effects of model error and initial condition error through a series of five experiments. In three cases, the OSSE is run in a configuration where model error is included. The initial condition error in these three cases is varied by manipulating both the observation error and the configuration of the observing network. In the two additional cases, the identical twin configuration is used to conduct perfect model tests with varying levels of initial condition error. The configuration of the GMAO OSSE will be described in Section 2. Results of the experiments are examined in terms of analysis errors in Section 3, behavior of the DAS in Section 4, and forecast errors in Section 5. Discussion and conclusions are presented in Section 6. 2 Method The GMAO OSSE framework includes code for the generation of synthetic observations for data types used in operational weather forecasting. Observation errors are added to the synthetic observations such that the variances of observation innovation and analysis increment in the OSSE are similar to those that occur when cycling the DAS with real observations. The forecast model and data assimilation system used for all cases are the Global Earth Observing System version 5.7.1 (GEOS-5, (Rienecker et al., 2008)) and the Gridpoint Statistical Interpolation (GSI, (Kleist et al., 2009)) data assimilation system, respectively. Two Nature Runs were used for the experiments described herein. The baseline NR is a 13-month integration of the version c31r1 European Centre for Medium-Range Weather Forecastsoperationalforecastmodel,runatT511horizontalresolutionwith91verticalsigma levels and 3-hourly output from 01 May 2005 to 31 May 2006. This integration was forced only with sea surface temperature and sea ice fields taken from 2005-6 archived datasets. No additional data was ingested into the NR. AsecondNRwasgeneratedusingashort,freerunoftheGEOS-5forecastmodelinorder to perform two experiments using an ‘identical twin’ setup with no model error. The initial state is taken as the operational analysis from 14 June 2011, and the model is integrated without observation ingestion until 11 August 2011. Synthetic observations were generated from both NRs using archived data as a basis for the time and location of observations. For the ECMWF NR, the observational suite was (cid:2)c 0000Tellus,000,000–000 MODELERRORINANOSSE 5 based on the real observations from June-August 2005, while for the identical twin NR, the observational suite was based on real observations from June-August 2011. The difference in the observing networks for the identical twin and ECMWF NR cases was not intentional, but merely a result of the generation of the identical twin cases at a considerably later time than the ECMWF NR cases, when the 2011 synthetic observations were newly available. The most significant differences between these two datasets are the inclusion of Quikscat, MSU, HIRS-2, and NOAA-15 for the ECMWF NR, and the inclusion of ASCAT, IASI, MHS, and the metop-a, NOAA-18, and NOAA-19 instruments in the identical twin cases. These differences are not expected to have a significant impact on the reported results of this study. Observations were first generated by interpolating from the ECMWF NR and then test- ing ingestion into the GEOS-5/GSI. These initial tests were used to calibrate the added observation errors. For the identical twin dataset, a new dataset was then generated using interpolation ofthe GEOS-5 NR. The errors added to this new dataset were generated using the same statistics as calibrated for the former. Since no realistic analog of the identical twin experiments exist (ie., no perfectly realistic model exists), calibration of the identical twin cases against real data is not possible. Therefore, the added observation errors are not re-calibrated for the identical twin cases. No additional observation biases were added to the observations although there are small intrinsic biases in the radiance observations due to the handling of clouds and surface emissivity (Errico et al., 2013). Five experiments were performed: threeusing theECMWF NRandtwo usingthe identi- caltwinGEOS-5NR.Anoverviewoftheseexperiments isgiven inTable1.FortheECMWF NR experiments, cycling began on 15 June 2005 and continued until 5 August 2005, with one forecast generated each day at 0000 UTC, for a total of 29 forecasts from 2 July to 30 July. In the first experiment, the ECMWF NR OSSE setup was employed, with cali- brated synthetic observations and observation errors that mimic the operational data suite from 2005, denoted the Control case. In the second experiment (NE), the same synthetic observations asin the Control were ingested with no explicitly added synthetic errors. These two experiments were repeated in the identical twin framework, one case featuring synthetic observations with no added observation errors (Twin NE) and the other case having added observation errors with the same magnitudes and correlations as in the Control experiment (Twin Control). In the third ECMWF NR case (DENSE), a global network of rawinsonde sounding (cid:2)c 0000Tellus,000,000–000 6 PRIVE´ ETAL. observations was generated, with one observation located at every other latitude, longitude, and vertical level of the NR grid at each cycling time, and with no added observation errors. No other data types were ingested. The rawinsonde observations were assumed to be taken instantaneously throughout the column fromthe surface to 1.5hPa. The soundings were not extendedtothetopoftheNRduetostronglyincompatibledifferencesinthedynamicsofthe upper atmosphere that result in numerical instability of the GEOS-5 forecasts when forced withobservationsfromtheECMWFNR.TheGSI-assumederrorcovariancesforrawinsonde typeswere decreased byafactorof10inordertomorestrongly drawthebackground toward the observations in this experiment. Some small but unspecified quantity of implicit representativeness error is present in the synthetic observations independent of the explicitly added observation errors. This arises because simulated radiance observations include cloud effects not accounted for by the as- similation system since the DAS instead attempts to remove cloud-contaminated radiance data through quality control. Small but still significant errors that are undetected by the qualitycontrolmaythusremain.Alsosurfacepropertiesusedtodeterminesurfaceemissivity for observation simulations are not the same asused in the DAS. In the case of the ECMWF NR, simulated observations are determined by spatial interpolations on the NR grid that differs from the DAS grid. Thus, although the spatial interpolation techniques used for the simulations and DAS are the same, the results generally differ. For this reason, the implicit error in the Twin cases is smaller than that for the ECMWF NR cases, because identical grids are used for the Twin NR and DAS. In the Control, NE, Twin Control, and Twin NE cases, the background and observation errorcovariancesassumedbytheDASarenotalteredfromthecovariancesusedoperationally in July 2011. For the Twin cases in particular, there is a significant mismatch between the actual and assumed background error covariances. A smaller mismatch between the actual and assumed background error covariances is expected for the ECMWF NR cases due to the change in the observational network between 2005 and 2011. Likewise, for the NE and Twin NE case, the actual observation error covariances are expected to be much smaller than the assumed covariances. Some of the ramifications of mismatched assumed and actual error covariances include the possible degradation of the analysis field in comparison to the background field, as discussed by Eyre and Hilton (2013) and Priv´e et al. (2013a). The ECMWF NR and the GEOS-5 model use hybrid η vertical coordinates, although the ECMWF NR has 91 levels while the GEOS-5 model uses 72 levels. In the upper at- (cid:2)c 0000Tellus,000,000–000 MODELERRORINANOSSE 7 mosphere (above 150 hPa in the GEOS-5 and above 80 hPa in the ECMWF), the η-levels follow pressure surfaces, while in the lower troposphere the η levels are dependent on the surface pressure in the fashion of σ-levels, with a blending in between (Untch et al. (1999), Rienecker et al. (2008)). Throughout this manuscript, the η levels will be referred to by the corresponding pressure that would occur if the surface pressure were 1000 hPa. For the identical twin cases, verification of forecasts and analyses can be made directly on the native grid of the GEOS-5. However, for the ECMWF NR cases, the NR fields must be interpolated onto a compatible grid for comparison with the GEOS-5 output fields. For ease of validation, the ECMWF NR fields are interpolated onto the same grid as that used by the GEOS-5. Details of the interpolation method are given in Errico and Priv´e (2013). 3 Analysis Error The analysis areal mean root-time mean-square error (RMSE) verified against the cor- responding NR fields for July is shown in Figure 1 for temperature, humidity, and wind in the tropics and the extratropics of both hemispheres. The qualitative form of the analysis error in the extratropics is similar in both hemispheres, but the behavior in the tropics is somewhat different from that in the extratropics. The OSSE Control case has the greatest analysis error for all variables and all regions, as would be expected since that case has the mostsources ofanalysis error(modelerror,observationerror,andsub-optimal observational network). For the NE case where explicit observation errors are not added, there is a slight reduction in the analysis error compared with the Control, with greatest reduction seen for wind fields in the extratropics. This indicates that the observation errors have a relatively small contribution to the total error in comparison to other sources of error. The smallest analysis RMSE were found in the DENSE case, except in the stratosphere and lowest levels of the troposphere where the TWIN NE case had the least analysis error. The DENSE case has less variation in analysis error with height compared with the Control case, the most striking example being the wind error in the tropics. The rawinsonde network in the DENSE case has consistent frequency and distribution of sampling throughout the troposphere and lower stratosphere, while the realistic observing network used in the other cases has very different distribution of observation sampling at different height levels. The analysisRMSEintheDENSEcaseis50-60%smallerthanintheControlintheextratropics, with 60-80% reduction in error in the tropics. The DENSE case estimates the limit of (cid:2)c 0000Tellus,000,000–000 8 PRIVE´ ETAL. improvement ofthe background state possible using the DAS and forecast model in question if the observing network were nearly ideal. The Twin NE case has error close to that of the DENSE case for temperature and wind in the extratropics, but significantly greater error than the DENSE case in the tropics and globally for humidity. The larger errors seen in the tropics and for humidity are believed to be due to convection, which behaves mathematically nonlinearly and discontinuously and has a short timescale of error growth. A greater increase in analysis error is observed when observation errors are included in the Twin cases in comparison to the difference between the NE and Control cases. The analysis error increases by 20-25% in the extratropics and 10% in the tropics for temperature, and by 40% in the extratropics and 20% in the tropics for winds from the Twin NE to the Twin Control case. The time mean analysis error field gives an indication of regions that experience a per- sistent source of error, which could stem from observation bias or systematic error of the forwardmodelorthedataassimilationsystem.IntheOSSE,observationbiasshouldbemin- imal as no explicit bias was added to the synthetic observations, so any time mean analysis error is likely to stem from model error or data assimilation processes. Time mean analysis error due to systematic model error would result from retention of model error in poorly observed areas, bias correction that incorrectly assumes observation bias in the presence of modelerror,orweighting ofbackground errorthatdoesnotaccountforthesystematic error. Systematic errors in the data assimilation process may also result from improper balance assumptions. Figure 2 shows the monthly mean analysis error for temperature and zonal wind at 500 hPa and 250 hPa respectively. The DENSE case has very little time mean analysis error, indicating that the extensive observational network successfully removes any systematic sources of error. The NE and Control cases both show significant regions of cold biased analysis temperature, especially in the deep tropics. In these two cases, systematic model error of the GEOS-5 in comparison to the ECMWF NR is suspected to account for much of this temperature bias. The zonal wind time mean analysis error features strong easterly biasesintheeastern Pacific andAtlantic equatorialbasins, andwesterly biasinthenorthern Indian Ocean. The easterly biases are due to a known issue with the cross-correlation of background errors of wind and temperature (and therefore to radiance data) by this version of the GSI/GEOS-5. The westerly bias over the Indian Ocean may be due to model error in representation of the upper tropospheric Asian monsoon circulation. (cid:2)c 0000Tellus,000,000–000 MODELERRORINANOSSE 9 TheTwincasesalsoshowacoldbiasinthedeeptropics,butofamuchsmallermagnitude thanintheECMWF NRcases. Since thereisnomodelerrorintheTwin cases, thelowtrop- ical temperatures in the analysis field may be due to destabilization of the vertical column during data assimilation, possibly resulting in excessive convection and mid-tropospheric cooling during the initial forecast period as the forward model physics attempts to adjust the unbalanced initial state. 4 Analysis Increments The analysis increment, or analysis minus background, is a measure of the work per- formed by the data assimilation system in modifying the background field to produce an analysis field x . The analysis increment can be expressed as a x −x = K[y −H(x )] (1) a b o b where thebackground statex isadjusted by theingestionofobservationsy using theoper- b o ationoperatorH andtheKalmangainK.Thetimemeanoftheanalysisincrement indicates the amount of persistent modification of the background. The left column of Figure 3 shows the zonal mean time mean analysis increment for temperature for the five experimental cases, and Figure 4 illustrates the same fields for zonal wind. The analysis increment gives a measure of work done by the observations, but does not indicate whether this work is beneficial, harmful, or neutral to the analysis quality. The difference between the absolute value of analysis error and absolute value of background error, denoted here as (|A| − |B|), gives an indication of whether the analysis increment is performing useful work or not, with negative values indicating an improvement of the analysis compared to the background. The zonal mean time mean distribution of (|A|−|B|) fortemperatureisshownontherightcolumnsofFigures3and4.Formostregions,(|A|−|B|) is negative, with the analysis having less error than the background. The largest time mean positive analysis increments for temperature are found in the tropics in areas of deep convection. Large negative time mean (|A| − |B|) is seen in the tropics, indicating that the time mean analysis increments are acting to remove errors from the background. These time mean analysis increments correspond to the regions of time mean analysis error seen in Figure 2. The Twin cases have smaller increments than the ECMWF NR cases (note the different contour intervals). (cid:2)c 0000Tellus,000,000–000 10 PRIVE´ ETAL. Figure 5 shows the zonal mean temporal variances of the analysis increment for the five experimental cases. While the time mean analysis increment primarily illustrates the removal of systematic background error by the observations, the variance of the analysis increment illustrates the role of non-systematic errors. Variance of the analysis increment is influenced both by the observation error directly (as in Eq. (1)) and through the growth of ingested observation errors from previous cycles. The Control case has larger analysis increment variance than the NE case as a result of these two factors. While the Control case has larger time mean and variance of analysis increments than the NE case, the |A|−|B| field shows that the useful work done by the observations in both cases is nearly the same. In a stable data assimilation system, the work done by ingestion of observations should be equal to the growth of errors between cycle times. This error growth is a function of the chaotic nature of the model dynamics and physics, the model error, and the initial analysis error; the first two are identical in the two experiments. As seen in Figure 1, there is also little difference in the analysis error between the NE and Control cases for temperature, so it is not surprising that the error growth rate in the two cases should be nearly the same. The NE and Control cases feature regions of wind field quality degradation by the as- similation process on the equator in the middle and upper troposphere, due to improper balancing of radiance observations (a known issue with this version of GEOS-5/GSI). This degradation is not observed in the DENSE case, where only rawinsonde observations with paired temperature and wind observations are ingested. The Twin NE case also shows some degradation of the analyzed wind field at the equator, although this is predominantly in the lower troposphere. There is little difference between the Twin NE and Twin Control for either time mean analysis increment or |A| − |B| for temperature, but a significant difference in |A| − |B| for zonal wind. While the Twin NE case shows improvement of the background state due to the DAS, the Twin Control case shows degradation of the background state in the mid and lower troposphere due to the presence of observation errors. The Twin Control case has largervariancesofanalysisincrement thantheTwin NEcase, althoughthevariancesofboth Twin cases are an order of magnitude smaller than for the ECMWF NR cases. The increase in error variance relative to the Twin NE case is much greater than the relative change for the ECMWF NR cases. In the Twin cases, the observation errors and their growth are (cid:2)c 0000Tellus,000,000–000