BECOMING A WELL-PREPARED PSYCHOMETRICIAN WHAT YOU SHOULD KNOW QUANTITATIVELY (BUT MAYBE WERE AFRAID TO ASK) 1 I separate the quantitative competencies ex- pected for an appropriately trained applied psy- chometrician into six general categories: (A) applied statistics (emphasizing the behav- ioral [psychological] sciences) (B) psychometrics (C) multivariate analysis (D) mathematical statistics (E) computation (F) the contexts of testing Separate two course sequences now exist at Illinois, for example, for (A) through (D); sin- gle course offerings for (E) and (F) will be proposed in the near future 2 Two articles are given as handouts (to show how important it is to develop strong training programs in measurement and psychometrics): ——— As Test-Taking Grows, Test-Makers Grow Rarer By DAVID M. HERSZENHORN Published: May 5, 2006 (New York Times) ——– APA Task Force for Increasing the Number of Quantitative Psychologists Makes Plans By Leona S. Aiken, Task Force Chair 3 I’ve included handouts on two other topics: A listing of fourteen journals (first and second tier) that you should be browsing and/or hav- ing available to you — A listing of some two-hundred books on test theory, measurement, and assessment that could be on your book shelf (or keep your eyes open for these in the used book stores) — 4 (A) Applied Statistics Research design, research methodologies, and basic analytical procedures/statistical analyses. This includes a strong understanding of the General Linear Model and its usage (formu- lated in matrix terms); the special cases of analysis-of-variance; analysis- of-covariance; and common multiple regres- sion. Also, skills are needed in dealing with repeated measures designs (split-plots, profile analyses, and longitudinal modeling), and the incorpora- tion of categorical data analysis to the level of loglinear models, logistic regression, general- ized linear models, and the various subanalyses these encompass. 5 PSYCHOLOGY 406 Statistical Methods I Techniques in applied statistics used in psycho- logical research, including simple linear regres- sion, partial and multiple correlation, and non- parametric methods; thorough review of statis- tical estimation and significance tests; empha- sizes applied statistics and statistical comput- ing. Introduces experimental design; one-way ANOVA. PSYCHOLOGY 407 Statistical Methods II Continuation of PSYCHOLOGY 406. Experi- mental design, including Latin Squares, facto- rials, and nested designs; expected mean squares, analysis of covariance; emphasizes the general linear model; introduces multivariate methods, such as factor analysis, scaling, classification, and clustering. Discrete multivariate analysis and multiway contingency tables. 6 Hays, W. (1994). Statistics (5th ed.). Bel- mont, CA: Wadsworth. Kutner, M. H., Nachtsheim, C. J., Neter, J., & Li, W. (2005). Applied linear statistical models (5th ed.). New York: McGraw-Hill/Irwin. Maxwell, S. E. & Delaney, H. D. (2003). De- signing experiments and analyzing data: A model comparison perspective. Mahwah, NJ: LEA. Agresti, A. (1996). An introduction to cate- gorical data analysis. New York: Wiley. 7 (B) Psychometrics Principles of psychometrics with a thorough understanding of both classical test theory (CTT) and item-response theory (IRT). Necessary knowledge includes a complete mas- tery of how test reliability and validity are ap- proached with both CTT and IRT; test equat- ing; differential item functioning; generalizabil- ity theory; adaptive testing; cognitive diagno- sis; test fairness and bias. 8 (Presumed as a Prerequisite for Theories of Measurement I and II) PSYCHOLOGY 490 Measurement and Test Development Lab The measurement of human behavior in psy- chological studies; the construction and use of psychological tests; introduction to tests of in- telligence, achievement, personality, and inter- est; and practice in test construction, admin- istration, and validation. Lectures and labora- tory. 9 EDUCATIONAL PSYCHOLOGY 585 Theo- ries of Measurement, I Classical test theory (true score, error of mea- surement, reliability and validity of test scores, composite measures); proposed alternatives to the classical model (generalizability theory, ma- trix sampling, latent trait theory, criterion-refer- enced measurement). EDUCATIONAL PSYCHOLOGY 586 Theo- ries of Measurement, II (Item Response The- ory) The theoretical foundations and applications of IRT. Includes latent trait estimation, item parameter calibration, modeling and detection of differential item functioning, linking and equat- ing, computerized testing. 10