Guidance Material and Best Practices for Pilot Aptitude Testing Effective March 2012 2 nd Edition Guidance Material and Best Practices for Pilot Aptitude Testing Effective March 2012 2 International Air Transport Association nd Edition Montreal—Geneva NOTICE DISCLAIMER. The information contained in this publication is subject to constant review in the light of changing government requirements and regula- tions. No subscriber or other reader should act on the basis of any such information without referring to applicable laws and regulations and/or without taking appropriate professional advice. Although ev- ery effort has been made to ensure accuracy, the International Air Transport Association shall not be held responsible for any loss or damage caused by errors, omissions, misprints or misinterpretation of the contents hereof. Furthermore, the Interna- tional Air Transport Association expressly disclaims any and all liability to any person or entity, whether a purchaser of this publication or not, in respect of anything done or omitted, and the consequences of anything done or omitted, by any such person or en- tity in reliance on the contents of this publication. © International Air Transport Association. All Rights Reserved. No part of this publication may be reproduced, recast, reformatted or trans- mitted in any form by any means, electronic or mechanical, including photocopying, record- ing or any information storage and retrieval sys- tem, without the prior written permission from: Senior Vice President Safety, Operations and Infrastructure International Air Transport Association 800 Place Victoria P.O. Box 113 Montreal, Quebec CANADA H4Z 1M1 Guidance Material and Best Practices for Pilot Aptitude Testing ISBN 978-92-9233-779-7 © 2012 International Air Transport Association. All rights reserved. Montreal—Geneva Guidance Material and Best Practices for Pilot Aptitude Testing TABLE OF CONTENTS 1 INTRODUCTION ....................................................................................................................................... 4 2 PILOT APTITUDE TESTING .................................................................................................................... 7 2.1 Safety and Human Performance ..................................................................................................... 7 2.2 Efficiency ......................................................................................................................................... 7 2.3 Fair Testing ...................................................................................................................................... 8 2.4 Risks of Not Testing Aptitude .......................................................................................................... 9 2.5 Quality Assurance ........................................................................................................................... 9 3 ITQI ONLINE PILOT SELECTION SURVEY .......................................................................................... 10 3.1 Outline of the Survey ..................................................................................................................... 10 3.2 Summary of Survey Questions ...................................................................................................... 11 3.3 Lessons Learned from the Industry SURVEY ............................................................................... 13 3.3.1 Accepted Approach ........................................................................................................ 13 3.3.2 Selection Systems may display a lack of conceptual basis ............................................ 13 3.3.3 Strengths and Weaknesses – methodologies being improved ...................................... 13 3.3.4 Ready-Entry Pilots (low experience) are a diverse group .............................................. 13 3.3.5 Selection for Captains and First Officers Undervalued .................................................. 14 4 LEGAL PROVISIONS FOR APTITUDE TESTING ................................................................................. 15 4.1 General .......................................................................................................................................... 15 4.2 Survey ........................................................................................................................................... 15 4.3 Existing Regulations ...................................................................................................................... 15 4.4 Data Protection and Professional Standards ................................................................................ 17 5 APTITUDE TESTING AND RECRUITMENT / HIRING .......................................................................... 18 5.1 Testing Supports Recruitment ....................................................................................................... 18 5.2 Screening and Selection ............................................................................................................... 18 5.3 Mechanism: Structured Aptitude Testing and Recruitment ........................................................... 19 5.4 Cultural Diversity ........................................................................................................................... 21 5.5 Hiring Decision .............................................................................................................................. 21 6 PREDICTING PERFORMANCE OF PILOTS ......................................................................................... 22 6.1 Test Reliability ............................................................................................................................... 22 6.2 Test Validity ................................................................................................................................... 23 6.3 Norm .............................................................................................................................................. 24 6.4 Measurement Scales ..................................................................................................................... 25 7 MEASURING DIMENSIONS / TESTING INSTRUMENTS ..................................................................... 26 7.1 Measuring Dimensions of Pilot Aptitude Testing ........................................................................... 26 7.2 Testing Instruments ....................................................................................................................... 28 7.2.1 Questionnaires ................................................................................................................ 28 7.2.2 Free-Style Interviews ...................................................................................................... 28 7.2.3 Semi-Standardized Interviews ........................................................................................ 29 7.2.4 Targeted Selection .......................................................................................................... 29 7.2.5 Paper-Pencil Psychometric Tests ................................................................................... 29 7.2.6 PC-Based Psychometric Tests ....................................................................................... 29 7.2.7 Work Samples................................................................................................................. 29 7.2.8 Simulation-Based Testing of Operational Competencies ............................................... 29 7.2.9 Fixed-Base Simulators .................................................................................................... 29 7.2.10 Full-Flight Simulators ...................................................................................................... 30 7.3 Motivation of Applicants ................................................................................................................ 30 7.4 IATA Matrix – Pilot Aptitude Testing .............................................................................................. 30 iii Guidance Material and Best Practices for Pilot Aptitude Testing 8 DESIGNING PILOT APTITUDE TESTING SYSTEMS ........................................................................... 33 8.1 Elements of a Pilot Aptitude Testing System ................................................................................. 34 8.2 Test Criterion ................................................................................................................................. 34 8.3 PersonalITY Requirements ............................................................................................................ 35 8.4 Test Battery .................................................................................................................................... 35 8.5 Arrangement of Stages .................................................................................................................. 36 8.6 Content of Stages .......................................................................................................................... 37 9 ADMINISTRATION OF APTITUDE TESTING SYSTEMS ...................................................................... 39 9.1 Selection Team .............................................................................................................................. 39 9.2 Duration of Tests ............................................................................................................................ 39 9.3 Outsourcing / Acquisition of Tests ................................................................................................. 39 9.4 Presentation of Results to the Applicant ........................................................................................ 40 9.5 Preparation Courses ...................................................................................................................... 40 9.6 Reapplication ................................................................................................................................. 40 9.7 Validity Period of Results ............................................................................................................... 41 9.8 Evaluation of the Aptitude Testing System .................................................................................... 41 10 FINANCIAL ASPECTS ............................................................................................................................ 42 10.1 Cost Effectiveness of Aptitude Testing Systems ........................................................................... 42 10.2 Structure and Contributions ........................................................................................................... 42 APPENDIX 1 – IATA – ITQI ONLINE PILOT SELECTION SURVEY AND RESULTS iv Guidance Material and Best Practices for Pilot Aptitude Testing ACRONYMS A/C Aircraft ATPL Airline Transport Pilots License ATO Approved Training Organization CPL Commercial Pilots License EASA European Aviation Safety Agency FAA Federal Aviation Agency (US) FBS Fixed-Base Simulators FCL Flight Crew Licensing FCLPT Flight Crew Licensing and Training Panel FFS Full Flight Simulator FTO Fight Training Organization ICAO International Civil Aviation Organization IOE Initial Operating Experience IR Instrument Rating IT Information Technology ITQI IATA Training and Qualification Initiative JAA Joint Airworthiness Authority (Europe, pre-EASA) JAR Joint Airworthiness Regulations KSA Knowledge, Skills and Attitudes MPA Multi-Crew Aeroplane MPL Multi-Crew Pilot License PAN-TRG Procedures for Air Navigation Services – Training PC Personal Computer TEM Threat and Error Management v Guidance Material and Best Practices for Pilot Aptitude Testing vi FOREWORD Dear Colleagues, It is my pleasure to present the second edition of the IATA Guidance Material and Best Practices for Pilot Aptitude Testing. The creation of this manual was identified as one of the IATA Training and Qualification Initiative’s priorities to preserve and improve operational safety in civil aviation. The data within the manual is based on an online survey that was conducted in the summer of 2009. Despite the clear benefits of a proper pilot selection process, the results showed that only a minority of airlines have a specific selection system in place that is structured and scientifically-based. This guidance material covers many of the aspects necessary to understand, construct and run a successful Pilot Aptitude Testing system. I would like to thank the members of the ITQI Project Group Selection and the airlines that have contributed to the development of this material. To obtain more information about the different publications that were developed under ITQI, please consult our website at: www.iata.org/itqi Best regards, Günther Matschnigg, Senior Vice President IATA Safety, Operations & Infrastructure 1 Guidance Material and Best Practices for Pilot Aptitude Testing EXECUTIVE SUMMARY Manual objective: to support all airlines. In recognition of the changing industry environment, this manual is designed to enable airline recruitment managers to implement modern practical pilot aptitude testing systems in their organizations. Operational decision-makers, aiming to recruit the best candidates, face the dilemma of selecting between similar testing systems offered by various providers with varying document terminology. This manual should enable more informed selection. Assumed Pilot Applicant Pool: A direct relationship between recruitment pool size and success of pilot aptitude testing (PAT) has been seen. The larger the recruitment pool, the better the PAT results. An adequate supply of pre-qualified and interested applications from which to select (for an airline career) are an assumed basis for this manual, which deals with selection and pilot aptitude testing. Initiatives to address a shrinking recruitment pool are beyond the scope of this manual. Selection Systems: The term pilot aptitude testing is used as hyponym, overarching all areas of pilot selection including aptitude diagnostics (basic abilities, specific/operational abilities, social competencies and personality traits). Measurement dimensions: The primary measurement dimensions of pilot aptitude tests are: a) Basic abilities (physical and mental) b) Operational competencies c) Social competencies d) Personality traits System performance: The performance of an aptitude testing system can be measured by an evaluation of the following factors: a) Test reliability b) Test validity (especially predictive validity) c) Ratio of the selection rate (number off successes) versus hit rate (on-site success rate with regard the test criterion) Testing tools of choice: The least qualified testing instruments are freestyle interviews, while some of the higher qualified testing instruments involve psychometric testing. Classic flight-simulator checks are suitable to quantify the amount and type of training needed for selected personnel, and provide some confidence in the validity of previous experience, in case of ready-entry pilots, but they are not so suitable for testing aptitude. Simulation-based testing of operational competencies can be performed best on specifically programed (PC-based) low fidelity simulators, since they provide high values of predictive validity. Multi-stage testing systems (less expensive screening procedures first, costly selection procedures last) are most advisable. Selection team and result: Hiring decisions should be made by a dedicated selection team. In the interest of safety and fairness and, assuming that the aptitude testing system has been professionally developed implemented and validated, the hiring decision should be based solely on test results. 2