Promoting Airmen with the Potential to Lead A Study of the Air Force Master Sergeant Promotion System Kirsten M. Keller, Sean Robson, Kevin O’Neill, Paul D. Emslie, Lane F. Burgette, Lisa M. Harrington, Dennis Curran CORPORATION ELECTRONIC COPIES OF RAND RESEARCH ARE PROVIDED FOR PERSONAL USE; POSTING TO A NONRAND WEBSITE IS PROHIBITED. THIS PUBLICATION IS AVAILABLE FOR LINKING OR FREE DOWNLOAD AT www.rand.org PROJECT AIR FORCE Promoting Airmen with the Potential to Lead A Study of the Air Force Master Sergeant Promotion System Kirsten M. Keller, Sean Robson, Kevin O’Neill, Paul D. Emslie, Lane F. Burgette, Lisa M. Harrington, Dennis Curran Prepared for the United States Air Force Approved for public release; distribution unlimited For more information on this publication, visit www.rand.org/t/RR581 Library of Congress Control Number: 2014954285 ISBN: 978-0-8330-8585-6 Published by the RAND Corporation, Santa Monica, Calif. © Copyright 2014 RAND Corporation R ® is a registered trademark. Limited Print and Electronic Distribution Rights This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited. Permission is given to duplicate this document for personal use only, as long as it is unaltered and complete. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial use. For information on reprint and linking permissions, please visit www.rand.org/pubs/permissions.html. The RAND Corporation is a research organization that develops solutions to public policy challenges to help make communities throughout the world safer and more secure, healthier and more prosperous. RAND is nonprofit, nonpartisan, and committed to the public interest. RAND’s publications do not necessarily reflect the opinions of its research clients and sponsors. Support RAND Make a tax-deductible charitable contribution at www.rand.org/giving/contribute www.rand.org Preface This report documents research examining the effectiveness of the Weighted Airman Promotion System (WAPS) in selecting airmen with the potential to be effective leaders as master sergeants. It describes the range of leadership knowledge, skills, abilities, and other characteristics that master sergeants need to perform well in the Air Force, and examines the extent to which these attributes are measured in WAPS. The report also includes recommendations, based on our findings, for improvements to the master sergeant promotion system. The research reported here was commissioned by the Director, Force Development, Deputy Chief of Staff for Manpower and Personnel, Headquarters United States Air Force (HQ USAF/A1D) and was conducted within the Manpower, Personnel, and Training Program of RAND Project AIR FORCE as part of a fiscal year 2012 project, “Enhancing Force Management and Development.” This report should be of interest to Air Force leadership and staff involved in the policy and execution of the enlisted promotion system. RAND Project AIR FORCE RAND Project AIR FORCE (PAF), a division of the RAND Corporation, is the U.S. Air Force’s federally funded research and development center for studies and analyses. PAF provides the Air Force with independent analyses of policy alternatives affecting the development, employment, combat readiness, and support of current and future air, space, and cyber forces. Research is conducted in four programs: Force Modernization and Employment; Manpower, Personnel, and Training; Resource Management; and Strategy and Doctrine. The research reported here was prepared under contract FA7014-06-C-0001. Additional information about PAF is available on our website: http://www.rand.org/paf/ iii Contents Preface ............................................................................................................................................ iii Figures........................................................................................................................................... vii   Tables ............................................................................................................................................. ix   Summary ........................................................................................................................................ xi   Acknowledgements ..................................................................................................................... xvii   Abbreviations ............................................................................................................................... xix   1. Introduction ................................................................................................................................. 1   Study Objective and Analytical Approach ............................................................................................. 1   Organization of the Report ..................................................................................................................... 2   2. Master Sergeant Leadership Responsibilities and Related Knowledge, Skills, and Abilities .... 5   Master Sergeant Roles and Responsibilities ........................................................................................... 5   Interview Findings ........................................................................................................................... 7   Master Sergeant Leadership KSAOs .................................................................................................... 11   Research on Leadership KSAOs .................................................................................................... 11   Interview Findings ......................................................................................................................... 15   Summary ............................................................................................................................................... 17   3. The Weighted Airman Promotion System ................................................................................ 19   Promotion Eligibility ............................................................................................................................ 19   Promotion Factors ................................................................................................................................ 21   Knowledge Tests ............................................................................................................................ 21   Time in Service and Time in Grade ............................................................................................... 22   Performance ................................................................................................................................... 23   Board Score .................................................................................................................................... 24   Promotion Selection ............................................................................................................................. 25   Summary ............................................................................................................................................... 25   4. Effectiveness of the Current Master Sergeant Promotion System ............................................ 27   Knowledge, Skills, and Abilities Assessed by WAPS ......................................................................... 28   Job Knowledge Tests: SKT and PFE ............................................................................................. 28   Job Experience: TIS and TIG ........................................................................................................ 29   Past Performance: EPR and Decorations ....................................................................................... 30   Predictive Validity Assessment ..................................................................................................... 31   Interview Findings ................................................................................................................................ 32   Gaps in KSAOs .............................................................................................................................. 33   Perceptions of the Effectiveness of the Current Promotion Process to Master Sergeant .............. 34   Relative Influence of WAPS Factors ................................................................................................... 34   Summary ............................................................................................................................................... 37   5. Options for Improvement .......................................................................................................... 39     v Improving the Performance Evaluation System ................................................................................... 39 Additional Assessment Methods for Inclusion in WAPS .................................................................... 40   Personality Tests ............................................................................................................................ 42   Biodata Inventories ........................................................................................................................ 43   Interviews ....................................................................................................................................... 43   Work Samples ................................................................................................................................ 44   Assessment Centers ....................................................................................................................... 45   Situational Judgment Tests ............................................................................................................ 46   Promotion Board Ratings ............................................................................................................... 47   Summary ............................................................................................................................................... 51   6. Conclusion and Recommendations ........................................................................................... 53   Recommendations ................................................................................................................................ 54   Examine the Validity and Utility of Implementing a Situational Judgment Test (SJT) and a   Master Sergeant Promotion Board as Additional Measures of Leadership Potential ............. 54 Explore Ways to Improve the Enlisted Performance Evaluation Process Using   Evidence-Based Practices ........................................................................................................ 57 Continue to Periodically Reevaluate the Effectiveness of the Promotion System ........................ 57   Conclusion ............................................................................................................................................ 57   Appendix A. Enlisted Performance Report Forms ....................................................................... 59   Appendix B. Interview Method and Analysis ............................................................................... 63   Appendix C. WAPS Factor Descriptive Statistics ........................................................................ 67   Appendix D. Overview of the Generalized Boosted Model ......................................................... 69   Appendix E. Research Findings on Evaluation Criteria for Each Assessment Method ............... 71   Appendix F. Estimated Cost of a Master Sergeant Promotion Board .......................................... 77   References ..................................................................................................................................... 85     vi Figures 2.1. Air Force Leadership Levels .................................................................................................... 6 4.1. Boxplots of the Percentage of Relative Influence of WAPS Factors on Promotions   Across AFSCs ...................................................................................................................... 36 4.2. Intended and Actual Relative Influence of WAPS Factors ................................................... 37     vii Tables 2.1. Leadership Competencies ........................................................................................................ 7 2.2. Duties Related to Leading and Managing a Team ................................................................... 9   2.3. Duties Related to Serving as a Member of the Leadership Team .......................................... 10   2.4. Attributes Underlying Air Force Leadership Competencies ................................................. 12   2.5. Master Sergeant Leadership Knowledge Areas ..................................................................... 15   2.6. Master Sergeant Leadership Skills and Abilities ................................................................... 16   2.7. Master Sergeant Leadership Knowledge, Skills, Abilities, and Other Characteristics .......... 17   3.1. Main Promotion Eligibility Criteria for Each Eligible Promotion Grade .............................. 20   3.2. Total Possible Points for WAPS Factors ............................................................................... 21   4.1. Score Distributions on WAPS Factors ................................................................................... 37   5.1. Personnel and Total Days Away from Primary Duty by Pay Grade ..................................... 50   5.2. Estimated Annual Costs for Master Sergeant Promotion Board ........................................... 50   5.3. Days Away from Primary Duty—Totals for Promotion Board Options ............................... 51   5.4. Costs for Promotion Board Options (Travel, Transportation, Per Diem, Records   Maintenance) ........................................................................................................................ 51 C.1. Score Distributions on WAPS Factors .................................................................................. 67   C.2. Correlations Between WAPS Factors ................................................................................... 67   E.1. Research Findings on Evaluation Criteria for Various Assessment Methods ....................... 73   F.1. Opportunity Costs for Board Members ................................................................................. 78   F.2. Total Cost for E-8 Board Membership (14 Panels, 15 Working Days) ................................. 79   F.3. Staffing Needs for Promotion Board Options ....................................................................... 80   F.4. Days Away from Primary Duty—Totals for Promotion Board Options ............................... 82   F.5. Estimated Costs for Promotion Board Options ..................................................................... 82   F.6. Estimated Costs for Promotion Board Options (Travel, Transportation, Per Diem,   Records Maintenance) .......................................................................................................... 83   ix