APA Handbook of Testing and Assessment in Psychology APA Handbook of Testing and Assessment in Psychology, Vol. 1: Test Theory and Testing and Assessment in Industrial and Organizational Psychology, edited by K. F. Geisinger, B. A. Bracken, J. F. Carlson, J.-I. C. Hansen, N. R. Kuncel, S. P. Reise, and M. C. Rodriguez APA Handbooks in Psychology APA Handbook of Testing and Assessment in Psychology volume 1 Test Theory and Testing and Assessment in Industrial and Organizational Psychology Kurt F. Geisinger, Editor-in-Chief Bruce A. Bracken, Janet F. Carlson, Jo-Ida C. Hansen, Nathan R. Kuncel, Steven P. Reise, and Michael C. Rodriguez, Associate Editors American Psychological Association • Washington, DC Copyright © 2013 by the American Psychological Association. All rights reserved. Except as permitted under the United States Copyright Act of 1976, no part of this publication may be reproduced or distributed in any form or by any means, including, but not limited to, the process of scanning and digitization, or stored in a database or retrieval system, without the prior written permission of the publisher. Published by American Psychological Association 750 First Street, NE Washington, DC 20002-4242 www.apa.org To order APA Order Department P.O. Box 92984 Washington, DC 20090-2984 Tel: (800) 374-2721; Direct: (202) 336-5510 Fax: (202) 336-5502; TDD/TTY: (202) 336-6123 Online: www.apa.org/pubs/books/ E-mail: [email protected] In the U.K., Europe, Africa, and the Middle East, copies may be ordered from American Psychological Association 3 Henrietta Street Covent Garden, London WC2E 8LU England AmericAn PsychologicAl AssociAtion stAff Gary R. VandenBos, PhD, Publisher Julia Frank-McNeil, Senior Director, APA Books Theodore J. Baroody, Director, Reference, APA Books Lisa T. Corry, Project Editor, APA Books Typeset in Berkeley by Cenveo Publisher Services, Columbia, MD Printer: United Book Press, Baltimore, MD Cover Designer: Naylor Design, Washington, DC Library of Congress Cataloging-in-Publication Data APA handbook of testing and assessment in psychology / Kurt F. Geisinger, editor-in-chief ; Bruce A. Bracken . . . [et al.], associate editors. v. cm. — (APA handbooks in psychology) Includes bibliographical references and index. Contents: v. 1. Test theory and testing and assessment in industrial and organizational psychology — v. 2. Testing and assessment in clinical and counseling psychology — v. 3. Testing and assessment in school psychology and education. ISBN 978-1-4338-1227-9 — ISBN 1-4338-1227-4 1. Psychological tests. 2. Psychometrics. 3. Educational tests and measurements. I. Geisinger, Kurt F., 1951- II. Bracken, Bruce A. III. American Psychological Association. IV. Title: Handbook of testing and assessment in psychology. BF176.A63 2013 150.2897—dc23 2012025015 British Library Cataloguing-in-Publication Data A CIP record is available from the British Library. Printed in the United States of America First Edition DOI: 10.1037/14047-000 Contents Volume 1: Test Theory and Testing and Assessment in Industrial and Organizational Psychology Editorial Board. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix About the Editor-in-Chief . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Contributors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Series Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii Part I. Test Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Chapter 1. Psychometric Characteristics of Assessment Procedures: An Overview . . . . . . . .3 Anita M. Hubley and Bruno D. Zumbo Chapter 2. Reliability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 Kurt F. Geisinger Chapter 3. The Generalizability of Test Scores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43 Edward W. Wiley, Noreen M. Webb, and Richard J. Shavelson Chapter 4. Test Validity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61 Stephen G. Sireci and Tia Sukin Chapter 5. Factor Analysis of Tests and Items. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85 Li Cai Chapter 6. Applying Unidimensional Item Response Theory Models to Psychological Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .101 Steven P. Reise, Tyler M. Moore, and Mark G. Haviland Chapter 7. Item Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .121 Randall D. Penfield Chapter 8. Bias in Psychological Assessment and Other Measures . . . . . . . . . . . . . . . . . . .139 Jeanne A. Teresi and Richard N. Jones Chapter 9. Test Development Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .165 Neal M. Kingston, Sylvia T. Scheuring, and Laura B. Kramer Chapter 10. Item Banking, Test Development, and Test Delivery. . . . . . . . . . . . . . . . . . . .185 David J. Weiss Chapter 11. Scaling, Norming, and Equating. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .201 Michael J. Kolen and Amy B. Hendrickson v Contents Chapter 12. Basic Issues in the Measurement of Change. . . . . . . . . . . . . . . . . . . . . . . . . . .223 John J. McArdle and John J. Prindle Chapter 13. The Standards for Educational and Psychological Testing. . . . . . . . . . . . . . . .245 Daniel R. Eignor Chapter 14. Technical Reporting, Documentation, and the Evaluation of Tests. . . . . . . . .251 Jane Close Conoley, Collie W. Conoley, and Rafael Julio Corvera Hernandez Chapter 15. Ethics in Psychological Testing and Assessment . . . . . . . . . . . . . . . . . . . . . . .265 Frederick T. L. Leong, Yong Sue Park, and Mark M. Leach Chapter 16. The Importance of Editorial Reviews in Ensuring Item Quality . . . . . . . . . . .283 Cathy Wendler and Jeremy Burrus Chapter 17. Fairness Review in Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .293 Michael J. Zieky Part II. Types of Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 Chapter 18. Objective Testing of Educational Achievement . . . . . . . . . . . . . . . . . . . . . . . .305 Michael C. Rodriguez and Thomas M. Haladyna Chapter 19. Objective Personality Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .315 Samuel E. Krug Chapter 20. Performance Assessment in Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .329 Suzanne Lane Chapter 21. Language Testing: History, Validity, Policy. . . . . . . . . . . . . . . . . . . . . . . . . . .341 Tim McNamara Part III. Industrial and Organizational Psychology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 Chapter 22. Assessment in Industrial and Organizational Psychology: An Overview. . . . .355 John P. Campbell Chapter 23. Work Analysis for Assessment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .397 Juan I. Sanchez and Edward L. Levine Chapter 24. Thinking at Work: Intelligence, Critical Thinking, Job Knowledge, and Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .417 Nathan R. Kuncel and Adam S. Beatty Chapter 25. Biographical Information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .437 Neal Schmitt and Juliya Golubovich Chapter 26. Assessment of Leadership. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .457 Nancy T. Tippins Chapter 27. Understanding and Improving Employee Selection Interviews. . . . . . . . . . . .479 Robert L. Dipboye and Stefanie K. Johnson Chapter 28. Personality Measurement and Use in Industrial and Organizational Psychology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .501 Leaetta M. Hough and Brian S. Connelly Chapter 29. Work Sample Tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .533 George C. Thornton III and Uma Kedharnath Chapter 30. Situational Judgment Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .551 Robert E. Ployhart and Anna-Katherine Ward vi Contents Chapter 31. Holistic Assessment for Selection and Placement . . . . . . . . . . . . . . . . . . . . . .565 Scott Highhouse and John A. Kostek Chapter 32. Employment Testing and Assessment in Multinational Organizations . . . . . .579 Eugene Burke, Carly Vaughan, and Ray Glennon Chapter 33. Performance Appraisal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .611 Kevin R. Murphy and Paige J. Deckert Chapter 34. Implementing Organizational Surveys. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .629 Paul M. Connolly Chapter 35. Counterproductive Work Behaviors: Concepts, Measurement, and Nomological Network. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .643 Deniz S. Ones and Stephan Dilchert Chapter 36. Stereotype Threat in Workplace Assessments . . . . . . . . . . . . . . . . . . . . . . . . .661 Ann Marie Ryan and Paul R. Sackett Chapter 37. Job Satisfaction and Other Job Attitudes . . . . . . . . . . . . . . . . . . . . . . . . . . . . .675 Reeshad S. Dalal and Marcus Credé Chapter 38. Legal Issues in Industrial Testing and Assessment. . . . . . . . . . . . . . . . . . . . . .693 Paul J. Hanges, Elizabeth D. Salmon, and Juliet R. Aiken vii Editorial Board EDITOR-IN-CHIEF Kurt F. Geisinger, PhD, Director, Buros Center for Testing, W. C. Meierhenry Distinguished University Professor, Department of Educational Psychology, and Editor, Applied Measurement in Education, University of Nebraska–Lincoln ASSOCIATE EDITORS Bruce A. Bracken, PhD, Professor, School Psychology and Counselor Education, College of William and Mary, Williamsburg, VA Janet F. Carlson, PhD, Associate Director and Research Professor, Buros Center for Testing, University of Nebraska–Lincoln Jo-Ida C. Hansen, PhD, Professor, Department of Psychology, Director, Counseling Psychology Graduate Program, and Director, Center for Interest Measurement Research, University of Minnesota, Minneapolis Nathan R. Kuncel, PhD, Marvin D. Dunnette Distinguished Professor, Department of Psychology, and Area Director, Industrial and Organizational Psychology Program, University of Minnesota, Minneapolis Steven P. Reise, PhD, Professor, Chair of Quantitative Psychology, and Codirector, Advanced Quantitative Methods Training Program, University of California, Los Angeles Michael C. Rodriguez, PhD, Associate Professor, Quantitative Methods in Education, Educational Psychology, and Director, Office of Research Consultation and Services, University of Minnesota, Minneapolis ix