Human Factors Experimental Design and Analysis Reference by Robert C. Willages ARL-RP-0186 July 2007 A reprint from Virginia Polytechnic Institute and State University Technical Report Number HFEEC-06-01 dated December 2006. Approved for public release; distribution is unlimited. NOTICES Disclaimers The findings in this report are not to be construed as an official Department of the Army position unless so designated by other authorized documents. Citation of manufacturers’ or trade names does not constitute an official endorsement or approval of the use thereof. DESTRUCTION NOTICE⎯Destroy this report when it is no longer needed. Do not return it to the originator. Army Research Laboratory Aberdeen Proving Ground, MD 21005-5425 ARL-RP-0186 July 2007 Human Factors Experimental Design and Analysis Reference Robert C. Willages Virginia Polytechnic Institute and State University A reprint from Virginia Polytechnic Institute and State University Technical Report Number HFEEC-06-01 dated December 2006. Approved for public release; distribution is unlimited. i Form Approved REPORT DOCUMENTATION PAGE OMB No. 0704-0188 Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 1. REPORT DATE (DD-MM-YYYY) 2. REPORT TYPE 3. DATES COVERED (From - To) July 2007 Reprint 4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER Human Factors Experimental Design and Analysis Reference 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) 5d. PROJECT NUMBER Robert C. Willages (VPI) 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION U.S. Army Research Laboratory REPORT NUMBER Human Research & Engineering Directorate ARL-RP-0186 Aberdeen Proving Ground, MD 21005-5425 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S) 11. SPONSOR/MONITOR'S REPORT NUMBERS 12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution is unlimited. 13. SUPPLEMENTARY NOTES The contracting officer’s representative (COR) is Sam Middlebrooks, U.S. Army Research Laboratory, ATTN: AMSRD-ARL-HR-MV, Fort Hood, TX, telephone number 254-288-9379. 14. ABSTRACT CADRE (computer-aided design reference for experiments) is a desktop computer tool for human factors and ergonomic researchers. The tool is in Acrobat1 format for cross-platform computer use and has more than 500 bookmarks for information search. This tool provides more than 850 Power-Point2 note pages of applied experimental design and analysis reference material. The reference material covers 25 topics and is divided into five major sections including introduction to experimental design, supplemental data collection design and analysis, basic analysis of variance (ANOVA) designs, advanced ANOVA designs, and empirical model building. References to the scientific literature are provided in each topic for supplemental reading. The CADRE tool also provides more than 200 pages explaining 40 examples of statistical analyses covered by the reference material. These examples are hyperlinked to Version 9.1.3 of the SAS3 (2004) statistical analysis software, and the sample SAS programs can be easily modified for user-specific data. 1Acrobat is a registered trademark of Adobe Systems, Inc. 2PowerPoint is a trademark of Microsoft. 3SAS, which is not an acronym, is a registered trademark of the SAS Institute, Inc. 15. SUBJECT TERMS experimental design 17. LIMITATION 18. NUMBER 19a. NAME OF RESPONSIBLE PERSON 16. SECURITY CLASSIFICATION OF: OF ABSTRACT OF PAGES Robert C. Willages a. REPORT b. ABSTRACT c. THIS PAGE SAR 1,130 19b. TELEPHONE NUMBER (Include area code) UNCLASSIFIED UNCLASSIFIED UNCLASSIFIED 540-951-7243 Standard Form 298 (Rev. 8/98) Prescribed by ANSI Std. Z39.18 ii Technical Report HFEEC-06-01 Human Factors Experimental Design and Analysis Reference Robert C. Williges December 2006 Prepared For Collaborative Technology Alliance on Advanced Decision Architectures Micro Analysis and Design Colorado Springs, Colorado and U.S. Army Research Laboratory Human Research and Engineering Directorate Aberdeen Proving Ground, Maryland 21005-5425 Human Factors Engineering and Ergonomics Center Grado Department of Industrial and Systems Engineering Virginia Polytechnic Institute and State University Blacksburg, Virginia 24061 CADRE Description References CADRE is a desktop computer tool for The CADRE project was supported by the Hu- human factors and ergonomic research- man Research and Engineering Directorate ers. The tool is in PDF format for cross- (HRED) of the Army Research Laboratory un- platform computer use and has over 500 der the direction of Dr. Sam Middlebrooks and bookmarks for information search. Dr. Michael Strub of HRED and through its prime contractor Micro Analysis and Design. This tool provides over 850 PowerPoint note pages of applied experimental de- Slater, C.R. and Williges, R.C. (2006). Appen- sign and analysis reference material by dix: SAS examples for human factors ex- Williges (2006). The reference material perimental design and analysis reference. covers 25 topics and is divided into five Blacksburg, VA: Virginia Polytechnic Insti- major sections including introduction to tute and State University. Technical Re- CADRE experimental design, supplemental data port HFECC-06-02. collection design and analysis, basic Computer-Aided Williges, R.C. (2006). Human factors experi- analysis of variance (ANOVA) designs, mental design and analysis reference. advanced ANOVA designs, and empiri- Blacksburg, VA: Virginia Polytechnic Insti- Design cal model building. References to the tute and State University. Technical Re- scientific literature are provided in each topic for supplemental reading, port HFECC-06-01. Reference for Williges, R.C. (2007). CADRE: Computer- The CADRE tool also provides over 200 Experiments Aided Design Reference for Experiments pages in a companion appendix by Sla- (a tool for human factors and ergonomic ter and Williges (2006) explaining 40 ex- research). Blacksburg, VA: Virginia Poly- amples of statistical analyses covered by technic Institute and State University. the reference material. These examples are hyperlinked to Version 9.1.3 of the Electronic Book CD-ROM-07-01. By SAS (2004) statistical analysis software, Robert C. Williges and the example SAS programs can be easily modified for user specific data. R.H. Bogle Professor Emeritus Grado Department of Industrial and Systems Engineering 2007 Human Factors Experimental Design and Analysis Reference Table of Contents Topics Page Overview 0.1. Purpose of Reference Material ………………………………………………………….3 0.1.1. Applied Experimental Design 0.1.2. Human Factors Engineering Methods 0.1.2.1. Research Methods 0.1.2.2. User Interface Design Methods 0.1.2.3. Training System Design Methods 0.1.2.4. Usability Evaluation Methods 0.2. Presentation Approach …………………………………………………………………10 0.2.1. Format of Reference Material 0.2.2. Experimental Design References 0.3. Organization of Reference Topics ……………………………………………………..13 0.3.1. Introduction to Experimental Design 0.3.2. Supplemental Data Collection and Analysis 0.3.3. Basic Analysis of Variance Designs 0.3.4. Advanced Experimental Designs Section 1. Introduction to Experimental Design Topic 1. Research Design Process 1.1. Stages of Research ……………………………………………………………….........21 1.2. Research Problem ………………………………………………………………………22 1.3. Research Approach ……………………………………………………………………..23 1.4. Critical Research Methods ……………………………………………………………..24 1.4.1. Variables 1.4.2. Procedures 1.4.3. Protection of Human Subjects 1.4.4. Equipment 1.4.5. Pretesting 1.5. Research Design Alternatives ………………………………………………………...30 1.6. Analyzing Results ……………………………………………………………………….31 1.7. Research Reports ……………………………………………………………………….32 1.7.1. Scientific Reports 1.7.2. Major Components of Reports 1.7.2.1. Introduction 1.7.2.2. Method 1.7.2.3. Results 1.7.2.4. Discussion 1.7.3. Additional Considerations 1.8. Summary …………………………………………………………………………………41 1.9. Supplemental Readings ………………………………………………………………..44 ii Human Factors Experimental Design and Analysis Reference Topic 2. Experimental Designs 2.1. Introduction ………………………………………………………………………………46 2.1.1. Threats to Validity 2.1.2. Quantitative Research Approach 2.2. Experimental Design Alternatives ……………………………………………………..51 2.2.1. Experimental Design Notation 2.2.2. Quasi-Experimental Designs 2.2.2.1. Single Group Designs 2.2.2.2. Nonequivalent Control Group Designs 2.2.2.3. Regression-Discontinuity Designs 2.2.2.4. Interrupted Time Series Designs 2.2.3. Randomized Experimental Designs 2.3. Summary …………………………………………………………………………….......69 2.4. Supplemental Readings ………………………………………………………………..70 Topic 3. Basic Statistical Concepts 3.1. Probability ………………………………………………………………………………..72 3.1.1. Compositional Techniques 3.1.2. Counting Techniques 3.2. Random Sampling ………………………………………………………………………81 3.3. Sampling Distributions ………………………………………………………………….82 3.3.1. Binomial Distribution 3.3.2. Normal Distribution 3.3.3. Student's t Distribution 3.3.4. Chi-Squared Distribution 3.3.5. F-Distribution 3.4. Statistical Estimation ……………………………………………………………………99 3.4.1. Estimators 3.4.2. Point Estimation 3.4.3. Interval Estimation 3.4.4. Summary of Statistical Estimation 3.5. Statistical Hypothesis Testing ………………………………………………………...109 3.5.1. Components of Hypothesis Testing 3.5.1.1. Null and Alternative Hypotheses 3.5.1.2. Format for Hypothesis Test 3.5.1.3. Types of Errors 3.5.1.4. Statistical Power 3.5.2. Single-Sample t-Test 3.5.3. Relationship to Statistical Estimation 3.6. Two-Sample t-Tests …………………………………………………………………...122 3.6.1. Sampling Distribution 3.6.2. Assumptions 3.6.3. Standard Format 3.6.4. Between-Subjects t-Test 3.6.5. Within-Subjects t-Test 3.6.6. Conclusion iii