4.2 AN OBJECTIVE VERIFICATION OF THE NORTH AMERICAN MESOSCALE MODEL FOR KENNEDY SPACE CENTER AND CAPE CANAVERAL AIR FORCE STATION William H. Bauman III * NASA Applied Meteorology Unit / ENSCO, Inc. / Cape Canaveral Air Force Station, Florida 1. INTRODUCTION Table 1. Towers, launch adivities and sensor heights at The 45th Weather Squadron (45 WS) Launch KSC and CCAFS used in the objective analysis to verify Weather Officers (LWO's) use the 12-km the MesoNAM forecasts. resolution North American Mesoscale (NAM) model (MesoNAM) text and graphical product forecasts extensively to support launch weather operations. However, the actual performance of the model at Kennedy Space Center (KSC) and Cape Canaveral Air Force Station (CCAFS) has not been measured objectively. In order to have tangible evidence of model performance, the 45 WS tasked the Applied Meteorology Unit (AMU; Bauman et ai, 2004) to conduct a detailed statistical analysis of model output compared to observed values. The model products are provided to the 45 WS by ACTA, Inc. and include hourly forecasts from 0 to 84 hours based on model initialization times of 00, 06, 12 and 18 UTC. The objective analysis compared the MesoNAM forecast winds, temperature (T) and dew pOint (Td ), as well as the changes in these Therefore, the period of record (POR) for this data parameters over time, to the observed values from set starts with the first cool season month of the sensors in the KSC/CCAFS wind tower October 2006. The KSC/CCAFS wind tower data network shown in Table 1. These objective were acquired for the period October 2006 to April statistics give the forecasters knowledge of the 2009 from the AMU archive, and the AMU wind model's strengths and weaknesses, which will tower quality control (QC) software was used to result in improved forecasts for operations. remove erroneous observations from the dataset. 2. BACKGROUND The statistical analysis software S-PLUS® (Insightful Corporation 2007) was used to process The 45 WS requested the data sets be the wind tower data. Scripts were written in S stratified by year, warm season (May-September), PLUS to import and modify the QC'd wind tower cool season (October-April), month and model observation files to remove unneeded time periods initialization time. They also requested verification and sensor heights from the dataset for each the model forecasts for the current operational tower. The locations of the towers used for the version of the MesoNAM. This paper will address verification are shown on the map of KSC/CCAFS the following statistics requested by the 45 WS: in Figure 1. • Bias (mean difference), Since the tower data were reported every • Standard deviation of Bias, and 5 minutes and the MesoNAM forecasts were hourly, the 45 WS requested the AMU calculate • Root Mean Square Error (RMSE). the mean value for each observed parameter from 3. WIND TOWER DATA the tower data at the top of every hour using the observations from 30 minutes prior and 30 The current version of the operational minutes after the hour. The S-PLUS scripts were MesoNAM became available in August 2006. written to reformat the tower data and calculate the mean values in this manner. ·Corresponding author address: William H. Bauman III, ENSCO, Inc., 1980 North Atlantic Ave, Suite 830, Cocoa Beach, FL 32931; e-mail: [email protected] 1 • • • nnn~~I_. • LaunchlLancllng Tower. • OtMrT. ..... • ....a NAII Grid Po. ... Figure 1. Map of KSC/CCAFS showing the locations of the wind towers used to verify MesoNAM forecasts (red pentagons labeled with tower number and the supported launch activity), the locations of the MesoNAM model grid points (green circles with black dot) and CCAFS weather station (magenta square labeled KXMR). 4. MESONAM FORECAST PRODUCTS An inventory of the MesoNAM files in the POR revealed 128 missing files, or model runs, out of a The AMU requested and obtained the possible 3772 files for the 943 days. Some days archived MesoNAM forecasts from ACTA, Inc. The were missing less than four model runs while current operational version of the MesoNAM is the others were missing all four model runs. This 12-km Weather Research and Forecasting (WRF) resulted in a total of 910 days containing at least model. Based on the seasonal stratifications one model run. requested by the 45 WS and model availability, the MesoNAM forecasts were evaluated beginning 5. FILE FORMATTING with the October 2006 data, the first cool season Microsoft Visual Basic scripts were written to month in the data set. The POR included three import the MesoNAM files into Microsoft Excel cool seasons: 2006-2007, 2007-2008 and 2008- spreadsheets and reformatted to match the wind 2009; and two warm seasons: 2007 and 2008. tower observation spreadsheets. This included The MesoNAM forecast files were provided to converting the temperature and dew point from the AMU as space-delimited text files. Each file Celsius to Fahrenheit and moving rows and was based on a single model initialization time and columns in the MesoNAM spreadsheets to match was valid at a single point extracted from the the wind tower spreadsheets. Visual Basic scripts model forecasts. This point was identified as were written to create an Excel workbook for each "KXMR" in each file and represents the location of of the 910 days with at least one model run. Each the CCAFS weather station, which is located near workbook included up to four worksheets, one for the center of CCAFS and is identified in Figure 1 each available model run, containing combined by the magenta square. The closest model grid wind tower observations and MesoNAM data for point, which represents the point data used by the each sensor on every tower. This resulted in a 45 WS, is located 5.8 km southwest of KXMR over total of 24,570 workbooks. the Banana River. It is shown in Figure 1 as a green circle with a black dot and labele:d "NAM". 2 6. VERIFICATION STATISTICS 6.1. Temperature and Dew Point Example Verification statistics were calculated once the Figure 2 shows a graph of the model bias of T files were properly formatted and stratified. First and Td from a 1200 UTe model initialization at the model bias was calculated for each model Tower 0020 at a sensor height of 6 ft for the month forecast against every observation. The means of January in the POR. Preliminary results indicate and standard deviations of the model bias for all a periodic fluctuation was present in the model stratifications (e.g., one month) as well as the root bias of T as can be seen in Figure 2. This result is mean square error (RMSE) were calculated using consistent among the first three towers and all the following equations: sensor levels analyzed as of the writing of this paper. The periodic fluctuation was observed in all In four model runs per day. In January, the model BiaSMean = ~ C/;. - °i) had a positive T bias and a negative T d bias. The i=l biases also decreased with forecast hour, with T Where: coming closer to 0, but T d becoming more = negative. n number of available model forecasts in any given stratification, Model T Mel Ttl BI. 12 UTe Inltlellzlltlon, Tower 0020,6 ft,.JMu.-y f= MesoNAM forecast of T, Td, wind speed 3.0,------------- or wind direction, and = o observed T, T d, wind speed or wind _ 1.0 4--~~-_\_-~_\_-~f_+_- !I- direction from each tower/sensor height. .. 0.0 -t-----.,--....,--,--.--.--,.--:;.--.--.\--,.-..p-,-+~ 1~i -1.0 +;-,,;.6~ ~. .1.:2:. ::.1..8.. c::.2:.A.. ....3=.0:. ...3.:6.. :c~-4-=2-. ..:4..:8:. ...5..4.- '---'-':...:.::..:.--'-=-~ - T("F) ~ \.J\ 1\, ---Td("F) ~ -~o ~-~~V~~'--~-'\-~-\------ JICX _X)2 ~ -3.0 4----------1~\ ~\_"/\~ -r\ -, STDEVBias = n "'.0 4----------.#----..3ri~..;_ -s,o Where: ..1....-._ __________ Foreceat v.lld nme from Modellnltlellzlltlon = n number of available model forecasts in Figure 2. Graph of T and Td model bias from a any given stratification, 1200 UTe model initialization at Tower 0020 and = sensor height of 6 ft for January. The blue line is T x model bias of each forecast, and the red dashed line is T d. = i mean bias of each forecast period in any Figure 3 shows the model bias of T and T d from a given stratification. 1200 UTe model initialization at Tower 0020 at a sensor height of 6 ft for the month of May in the POR. The T bias was more negative than January RMSE = -vMSE and displayed a similar periodic fluctuation. The 1nI model Td bias was also negative in May. = - MSE n (VI't. - 0t' ) 2 Model Tn T.. BI. i=l 12 UTt Inltlellzlltlon, Tower 0020,6 ft, M-V 1.0 ,..------------- Where: = n number of available model forecasts in tL any given stratification, t... ~ -1.0 -lP------+--=----a--h-----:+-~_4___I-- f= MesoNAM forecast of T, Td, wind speed ~ - T("F) or wind direction, and El -~o ~~-~~~~~~~~-+ ---Td("F) ~ = o observed T, T d, wind speed or wind -3.0 -t-----------------'''"-- direction from each tower/sensor height. -4.0.L.------------- For.utt v.lld nme from ModellnltleUzlltlon Figure 3. As in Figure 2 but for May. 3 The model standard deviation and RMSE of T Model Wind Speed Blu and Td are shown in Figure 4 and 5, respectively, 12 UTe Initlellzlltlon, Tower 0020,54 ft,.-.u.y for a 1200 UTC model initialization at Tower 0020 6.0,------------------ at a sensor height of 6 ft for the month of January. Both graphs indicate the model performance ~4.0 tt---~'----+--I----\--JL----­ degraded with time through the 84-hr forecast. g ~O ~---------------- This preliminary result of model degradation with 1 1 time was also consistent among the three towers ~O +----------------- evaluated against the model thus far. 1.0 +----------------- Model StMdard Deviation ofT lind Tel BI. 12 UTC Inltlellzlltlon, Tower 0020,6 ft,"'"'*'1 o 6 U a 2A 30 36 G 48 54 60 66 72 n 84 8.0 ,-------------- Forec.t valid Time from Modellnltlellzldlon 7.0 +-------------J.~ E Figure 6. Graph of model wind speed bias from a 6.0 +-----------..~--=­ f 5.0 +----..,.-------:-~,......-,pA~~~-­ 1200 UTe model initialization at Tower 0020 and 1;:,i 4.0 ~~~'t-r~~_I___'V_~\-I---- sensor height of 54 ft for January. i 3.0 -f1i--+-l~---------------­ - T("I') E ---Td('F) Model Suncwd DevIation of Wind Speed Blu ~ ~o +------------- 12 UTe Inltlellzldlon, Tower 0020,54 ft, ~ 1.0 +-------------- 5.0 ... 0.0 +o- .,-6- ,--1,2- --1-8- -r2-A- ---3.0- --3.6- --4.-2- r-4r8- -154,. .-6-0- -,.6-6- .-7-2-- -7.-8- --8r4 44..50 1'\ r\ 1"\ 7 Y'J\j'l"'"" i L \ fr'./ ""'" V Forec.t valid Time from Modellnltlellzldlon 3.5 7' """"" ~ 3.0 II Figure 4. As in Figure 2 but for standard deviation of - ~5 "a bias. c: t M 1.5 1.0 Model Tn Tel RMSE 0.5 12 UTC Inltlellzatlon, Tower 0020, 6 ft, Jenu.-y 0.0 9.0,-------------- o 6 U a 2A 30 36 G 48 54 60 66 72 n 84 &0 +-------------7'~' Forecelt valid Time from Modellnltlllllzidlon ii:' 7.0 +----------::-----:..-->N'--- ~I 6.0 +--------=--#r-~~:..w--..I---=­ Figure 7. Same as in Figure 6 but for standard ~o ~r+~-+~~~~~~~-­ deviation. ~ 4.0 +-1~l-".;hJi&f_--'\,.,I\--J---JIf__---.::I!~--- - T("I') ~ ~o ~-~~-~------­ ---Td("I') Model Wind Speed RMSE ~ ~o +-------------- 12 UTe Initlallzlltlon, Tower 0020, 54 ft, Jenurt 1.0 +------------- 8.0,------------------ 0.0 +-.,--,--,------r-----.---.---.--r-.,--,,..----,.--.-----.----r o 6 12 11 2A 30 36 42 48 54 60 66 72 78 84 7.0 +-----:----------:::----~~-- _I6 .0 ~~~r_r_~~~_r~~~_1~--- Forec.t "-lId Time from Modellnltlellzldlon 1/1 ~O tt-------~--------- Figure 5. As in Figure 2 but for RMSE. - 4.01t----------------- 1 " 3.0 t_---------------- 6.2. Wind Speed and Direction Example ct ~O t_---------------- t----------------- Figures 6, 7 and 8 show the graphs of wind 1.0 speed bias, standard deviation of bias and RMSE, 0.0 +---,---,---.-----.------r---,----r---,,--,---,----,---.--.----,- o 6 U a 2A 30 36 G 48 54 60 66 72 n 84 respectively from a 1200 UTC model initialization Forecelt "-lId Time from Modellnltlellzldlon at Tower 0020 at a sensor height of 54 ft for the month of January in the POR. These results Figure 8. Same as in Figure 6 but for RMSE. indicate the MesoNAM forecasted wind speed 4-5 Figures 9, 10 and 11 show the wind direction kt too high throughout the entire 84 hour model graphs of bias, standard deviation of bias and forecast period for January. Preliminary results RMSE, respectively from a 1200 UTe model indicate the MesoNAM also forecasted wind speed initialization at Tower 0020 at a sensor height of too high at the other three towers evaluated thus 54 ft for the month of January in the POR. The far. Unlike T and Td, the wind speed forecast error MesoNAM bias of wind direction was more did not increase Significantly throughout the 84-hr negative, or to the right, of the observed winds. forecast period but remained fairly constant. The standard deviation shows the bias was highly variable between 30 and 60°. 4 45°, 90° and 180° with the sector directions Model Wind Direction BI. 12 UTe InItI.!lutlon, Tower 0020, 54 ft, *"'-Y oriented to maximize discrimination between lO ~----------------------------- onshore and offshore flow. 5 +---------------------~------- Model Wind Direction RMSE I O +-~+_~~~~~~~~~~~~ 12 UTe Inltlellzalon, Tower 0020,54 ft, J..w.ry ! 70 ~------------------- S I -5 ~ ~---------------------------- 1$ e. so +--+-t--------A-~-----::+c+\o:_-+'Wor-1~__\_1_ • -lO ~~_ff--~_Hr+~_+_+~--~~~ c5 40 +-I------\---1t--'--------".....\~--~~------­ -15 +-f.-------Hl~--!..---I.+_I_------*_--- S 30 4L-----...L.Jf----------------------- 1$ -~ ~----Fo-r-ec-u-t v-.l-ld- nm-e- f-ro-m- M-o-de-lln-lt-lll-li-za-tio-n- --- a• ~ +------------------------------- 10 r----------------------------- Figure 9. Graph of model wind direction bias from a 1200 UTC model initialization at Tower 0020 and Forecut Vllild nme from Modellnltllllizatlon sensor height of 54 ft for January. There was also a slight degradation in the wind Figure 11. As in Figure 9 but for RMSE. direction forecast through the 84-hrforecast period In order to present the data to the LWO's in a for this tower as shown in the standard deviation manageable and user-friendly manner, a graphical of the bias and RMSE in Figures 10 and 11. user interface may be developed. A total of 9,240 graphs would be generated based on the statistics Model St.nct.rd Oevlalon of Wind Direction BI. 12 UTe Inltlllllzlltlon, Tower 0020, 54 ft,~ being calculated for the monthly and seasonal stratifications for all sensors on all towers. The ~ ~---------------------------- number of graphs could exceed 36,000 if the I ~ ~---------------------------- so ~-+-t--------A-~-----:-+~-+'bor-1~-+1_ sample size is significant enough to stratify the !4O +-I------+----IIf-----,f---C--~f1L-----K.-------­ data by sector. . S 30 ~--..L..lf_------------------ 8. REFERENCES 1$ a ~ ~------------------- Bauman, W. H., W. P. Roeder, R. A. Lafosse, D. 10 ~---------------------------- W. Sharp, and F. J. Merceret, 2004: The Applied Meteorology Unit - Operational Contributions to Spaceport Canaveral. Forecut Vllild nme from Modellnltlellzatlon Preprints, 11th Conference on Aviation, Range, and Aerospace Meteorology, Amer. Figure 10. As in Figure 9 but for standard deviation Meteor. Soc., Hyannis, MA, 4-8 October 2004, of bias. 24 pp. 7. FUTURE WORK Insightful Corporation, 2007: S-PLUS® 8 for In addition to the statistics presented in this Windows® User's Guide, Insightful Corp., paper, the 45 WS has requested conducting Seattle, WA, 584 pp. = = hypothesis tests for bias 0, RMSE 0 and if the = composited bias and RMSE O. Also, if justified by the sample size, the data will be stratified by NOTICE Mention of a copyrighted, trademarked, or proprietary product, service, or document does not constitute endorsement thereof by the author, ENSCO, Inc., the AMU, the National Aeronautics and Space Administration, or the United States Government. Any such mention is solely for the purpose of fully informing the reader of the resources used to conduct the work reported herein. 5