Documentation for the 2004–05 Teacher Follow-up Survey U.S. Department of Education NCES 2007–349 March 2007 Documentation for the 2004–05 Teacher Follow-up Survey March 2007 U.S. Department of Education NCES 2007–349 Shawna Cox Randall Parmer Steven Tourkin Toni Warner U.S. Bureau of the Census Deanna M. Lyter Education Statistics Services Institute— American Institutes for Research Reneé Rowland Project Officer National Center for Education Statistics U.S. Department of Education Margaret Spellings Secretary Institute of Education Sciences Grover J. Whitehurst Director National Center for Education Statistics Mark Schneider Commissioner The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing, and reporting data related to education in the United States and other nations. It fulfills a congressional mandate to collect, collate, analyze, and report full and complete statistics on the condition of educa- tion in the United States; conduct and publish reports and specialized analyses of the meaning and significance of such statistics; assist state and local education agencies in improving their statistical systems; and review and report on education activities in foreign countries. NCES activities are designed to address high priority education data needs; provide consistent, reliable, complete, and accurate indicators of education status and trends; and report timely, useful, and high quality data to the U.S. Department of Education, the Congress, the states, other education policymakers, practitio- ners, data users, and the general public. We strive to make our products available in a variety of formats and in language that is appropriate to a variety of audiences. You, as our customer, are the best judge of our success in communicating information effectively. If you have any comments or suggestions about this or any other NCES product or report, we would like to hear from you. Please direct your comments to National Center for Education Statistics Institute of Education Sciences U.S. Department of Education 1990 K Street NW Washington, DC 20006-5651 March 2007 The NCES World Wide Web Home Page is http://nces.ed.gov. The NCES World Wide Web Electronic Catalog is http://nces.ed.gov/pubsearch. Suggested Citation Cox, S., Parmer, T., Tourkin, S., Warner, T., and Lyter, D.M. (2007). Documentation for the 2004–05 Teacher Follow-up Survey (NCES 2007–349). U.S. Department of Education. Washington, DC: National Center for Education Statistics. For ordering information on this report, write to U.S. Department of Education EDPubs P.O. Box 1398 Jessup, MD 20794-1398 Or call toll free 1-877-4ED–Pubs or order online at http://www.edpubs.org. Content Contact: Reneé Rowland (202) 502-7336 [email protected] iii Contents List of Tables...............................................................................................................................................vi List of Exhibits...........................................................................................................................................viii Chapter 1. Overview.....................................................................................................................................1 Background...............................................................................................................................................1 Purpose and Content of the Survey..........................................................................................................1 Questionnaire for Former Teachers (Form TFS-2)..............................................................................2 Questionnaire for Current Teachers (Form TFS-3)..............................................................................2 Target Population and Estimates..............................................................................................................3 Target Population.................................................................................................................................3 Estimates..............................................................................................................................................3 Methodology.............................................................................................................................................4 Contents of the Manual.............................................................................................................................5 Chapter 2. Preparation for the 2004–05 TFS................................................................................................7 Cognitive Interviews and Usability Tests.................................................................................................7 Study One.............................................................................................................................................7 Study Two............................................................................................................................................8 Content Changes.......................................................................................................................................9 Changes to the Questionnaire for Former Teachers.............................................................................9 Changes to the Questionnaire for Current Teachers...........................................................................10 Final Content of the 2004–05 TFS.....................................................................................................11 Chapter 3. TFS Frame Creation and Sample Selection Procedures...........................................................13 SASS Sampling Frames..........................................................................................................................13 Public School Sampling Frame..........................................................................................................13 Private School Sampling Frame.........................................................................................................14 SASS School Stratification.....................................................................................................................15 Public School Stratification................................................................................................................15 Private School Stratification...............................................................................................................16 SASS School Sample Selection..............................................................................................................17 Public School Sample Selection.........................................................................................................17 Private School Sample Selection........................................................................................................18 SASS Teacher Sample Selection............................................................................................................19 SASS Teacher Frame.........................................................................................................................19 Within-School SASS Teacher Allocation..........................................................................................20 TFS Teacher Sampling Frame, Stratification, and Allocation................................................................21 TFS Teacher Sampling Frame............................................................................................................21 TFS Teacher Stratification.................................................................................................................21 TFS Teacher Allocation.....................................................................................................................25 TFS Teacher Sample Selection...............................................................................................................28 Sorting................................................................................................................................................28 Sample Selection................................................................................................................................29 iv Documentation for the 2004–05 Teacher Follow-up Survey Chapter 4. Data Collection..........................................................................................................................31 Overview of Data Collection Procedures...............................................................................................31 Internet and Incentive Experiment..........................................................................................................32 Timing of Data Collection......................................................................................................................33 Data Collection Procedures....................................................................................................................34 Collecting Teacher Status Information...............................................................................................34 Initial Mailings of Questionnaires to Current and Former Teachers..................................................34 Address Research Operations.............................................................................................................35 Follow-up Mailings............................................................................................................................35 “Switchers”........................................................................................................................................35 Nonresponse Follow-up.....................................................................................................................36 Changes in Data Collection Methods from the 2000–01 TFS................................................................36 Address Research Operation..............................................................................................................37 Internet and Incentives.......................................................................................................................37 E-mail Reminders...............................................................................................................................37 Chapter 5. Response Rates..........................................................................................................................39 Survey Response Rates...........................................................................................................................39 Item Response Rates...............................................................................................................................41 Nonresponse Bias Analysis....................................................................................................................42 Unit-level Nonresponse......................................................................................................................42 Item Nonresponse Bias Analysis........................................................................................................47 Chapter 6. Data Processing.........................................................................................................................49 Questionnaire Check-in..........................................................................................................................49 Check-in of Paper Questionnaires......................................................................................................49 Check-in of Internet Questionnaires...................................................................................................50 Data Capture...........................................................................................................................................50 Data Capture of Paper Questionnaires...............................................................................................50 Internet Questionnaires......................................................................................................................51 Reformatting...........................................................................................................................................51 Preliminary Data Review........................................................................................................................51 Preliminary Interview Status Recode (ISR) Classification.....................................................................53 Computer Edits.......................................................................................................................................53 Final Interview Status Edit.....................................................................................................................54 Creating Imputation Flags......................................................................................................................55 Data Products..........................................................................................................................................56 Chapter 7. Imputation Procedures...............................................................................................................57 Overview................................................................................................................................................57 First-Stage Imputation............................................................................................................................57 Second-Stage Imputation........................................................................................................................58 Hot Deck Imputation..........................................................................................................................58 Subsample Imputation........................................................................................................................61 Subsample Ratio Imputation..............................................................................................................61 Random Subsample Ratio Imputation................................................................................................62 Third-Stage Imputation...........................................................................................................................62 Post-Imputation Processing....................................................................................................................63 Final File Imputation Tables...................................................................................................................63 Contents v Chapter 8. Weighting and Variance Estimation..........................................................................................65 Weighting...............................................................................................................................................65 Variance Estimation...............................................................................................................................67 Chapter 9. Reviewing the Quality of TFS Data..........................................................................................73 General Data Quality..............................................................................................................................73 Nonresponse...........................................................................................................................................73 Weighting...............................................................................................................................................74 External Data Checks.............................................................................................................................74 Chapter 10. Information on Data Files and Merging Components.............................................................83 Availability of Data................................................................................................................................83 How to Get Restricted-Use Data Files...............................................................................................83 Understanding the Data Files.................................................................................................................84 Confidentiality Edits to the Data........................................................................................................84 Treatment of Public Charter Schools and Schools Funded by the Bureau of Indian Affairs.............84 Categories of Variables......................................................................................................................85 Nonresponding Units..........................................................................................................................87 Linking Files Within and Across TFS and SASS...................................................................................88 Sample SAS Syntax for Merging Data Files and Attaching Variable Labels....................................89 Sample SPSS Syntax for Merging Data Files....................................................................................91 Sample Stata Syntax for Merging Data Files.....................................................................................93 Chapter 11. User Notes and Cautions.........................................................................................................95 Cautions Concerning Change Estimates.................................................................................................95 Estimates for Total Teachers..................................................................................................................95 Minority Enrollment...............................................................................................................................96 User Notes and Cautions for SASS........................................................................................................96 References...................................................................................................................................................97 Appendixes Appendix A. Key Terms for TFS...........................................................................................................A-1 Appendix B. TFS Questionnaire Availability........................................................................................B-1 Appendix C. Summary of TFS Interview Findings and Recommendations..........................................C-1 Appendix D. Summary of TFS Pretest and Usability Test Findings and Recommendations................D-1 Appendix E. (Inter) Net Gain? Experiments to Increase Response.......................................................E-1 Appendix F. Quality Assurance for TFS Keying and Mailout Operations.............................................F-1 Appendix G. Changes Made to TFS Variables During Computer Edits, by Data File..........................G-1 Appendix H. Percentage of TFS Variables Changed During Three Stages of Imputation, by Data File....................................................................................................................................H-1 Appendix I. Weighting Adjustment Cells...............................................................................................I-1 Appendix J. Frame and Created Variables.............................................................................................J-1 Appendix K. Crosswalk of 2004–05 TFS Items With 2000–01 TFS and 2003–04 SASS Teacher Questionnaire....................................................................................................................K-1 vi Documentation for the 2004–05 Teacher Follow-up Survey List of Tables Table Page 1. Average expected number of new and experienced teachers selected per school, by school level and school type: 2003–04........................................................................................................20 2. Final unweighted and weighted number of teachers selected for the SASS teacher sample, by teacher type and school type: 2003–04.......................................................................................21 3. TFS sampling frame counts for teachers, by school type, teacher status, teacher’s grade level, teacher experience, and minority status: 2004–05.................................................................23 4. Final allocated TFS sample sizes for teachers, by school type, teacher status, teacher’s grade level, teacher experience, and minority status: 2004–05.................................................................26 5. Treatment groups, by distribution mode, incentive offered, and sample size: 2004–05..................32 6. Data collection time schedule: 2004–05..........................................................................................33 7. Unweighted and base-weighted response rates (in percent) of sampled teachers, by sector and teaching status: 2004–05...........................................................................................................40 8. Base-weighted response rates (in percent) for SASS teacher data files and TFS data files, by sector: 2003–04 and 2004–05..........................................................................................................40 9. Summary of weighted item response rates for the Teacher Follow-up Survey, by questionnaire: 2004–05....................................................................................................................41 10. Number of teachers in sample and base-weighted unit response rates, by selected reporting characteristics: 2004–05...................................................................................................................43 11. Base-weighted frame distribution of TFS teachers, interviewed distribution, and t statistic, by selected reporting characteristics: 2004–05................................................................................45 12. Number of questionnaire items, by response rate category and data file: 2004–05.........................47 13. Summary of changes made to variables in the computer edits, by data file: 2004–05....................54 14. Preliminary and final interview status recode (ISR) counts and percentage change, by ISR category and data file: 2004–05.......................................................................................................55 15. Number of source codes imputed among former teacher items, by percentage of records receiving imputation and imputation stage: 2004–05......................................................................63 16. Number of source codes imputed among current teacher items, by percentage of records receiving imputation and imputation stage: 2004–05......................................................................64 17. Distribution of final weights, by data file: 2004–05........................................................................66 18. Comparison of number of stayer and nonstayer teachers (both sectors) in 2004–05 TFS population to teachers in 2003–04 SASS, by TFS sampling strata and selected SASS teacher and school characteristics: 2003–04 and 2004–05...........................................................................75 19. Comparison of number of public and private sector school stayer and nonstayer teachers in 2004–05 TFS population to teachers in 2003–04 SASS, by TFS sampling strata and selected SASS teacher and school characteristics: 2003–04 and 2004–05....................................................77 20. Comparison of number of traditional public and public charter school stayer and nonstayer teachers in 2004–05 TFS population to teachers in 2003–04 SASS, by TFS sampling strata and selected SASS teacher and school characteristics: 2003–04 and 2004–05...............................80 21. Names of data files and the questionnaires from which the data were drawn: 2004–05.................83 22. Number of missing cases in combined SASS datasets, by nonresponding component and dataset providing unit of analysis: 2003–04.....................................................................................87 23. Number of missing cases in combined datasets, by merged SASS data file and TFS data file by sector providing unit of analysis: 2003–04 and 2004–05...........................................................88 C-1. Description of interview participants: 2004...................................................................................C-2 Contents vii Table Page E-1. Characteristics of treatment groups: 2004–05................................................................................E-3 E-2. Timing of treatments, by group and mode: 2004–05.....................................................................E-4 E-3. Total, mail, and internet weighted response rates, by group: 2004–05..........................................E-5 E-4. Weighted response rates, by type of group: 2004–05....................................................................E-5 E-5. Total weighted response rates for the mail only groups, the internet without mention of mail option groups, and the internet with mention of mail option groups: 2004–05.............................E-6 E-6. Total, mail, and internet weighted response rates, by type of internet group: 2004–05................E-6 E-7. Total weighted response rates, by incentive treatment: 2004–05..................................................E-6 E-8. Additional response as a result of incentives, by type of group: 2004–05.....................................E-7 E-9. Internet weighted response rates, by incentive treatment and type of internet group: 2004–05....E-7 G-1. Total number of changes and percentage of records affected during computer edits for the former teacher data file, by variable: 2004–05..............................................................................G-2 G-2. Total number of changes and percentage of records affected during computer edits for the current teacher data file, by variable: 2004–05..............................................................................G-3 H-1. Number of changes and percentage of records affected during stage 1–stage 3 imputation of the former teacher data file, by variable: 2004–05........................................................................H-1 H-2. Number of changes and percentage of records affected during stage 1–stage 3 imputation of the current teacher data file, by variable: 2004–05........................................................................H-4 viii Documentation for the 2004–05 Teacher Follow-up Survey List of Exhibits Exhibit Page 1. Items with weighted response rates of less than 85 percent, by questionnaire: 2004–05................41 2. Items with a response rate below 70 percent, by data file: 2004–05................................................47 3. TFS critical item table, by questionnaire: 2004–05.........................................................................52 4. Flags used in processing TFS questionnaires: 2004–05...................................................................56 5. Adjustment factors and collapsing criteria as applied to TFS weights: 2004–05............................67 D-1. Respondent characteristics, by teaching status and internet access: 2004.....................................D-3 F-1. Cumulative key from paper (KFP) data keying verification report, by questionnaire: 2004–05...........................................................................................................................................F-2 F-2. Distribution of keying errors, by questionnaire and type of error: 2004–05...................................F-3 F-3. Docuprint quality assurance summary, by type of inspection and form: 2004–05.........................F-5 F-4. Duplo Booklet Maker inspection, by type of inspection and form: 2005.....................................F-10 F-5. Package assembly quality assurance, by type of inspection and form: 2004–05..........................F-11 1 Chapter 1. Overview The Teacher Follow-up Survey (TFS) is sponsored by the National Center for Education Statistics (NCES) on behalf of the U.S. Department of Education and is conducted by the U.S. Census Bureau. TFS is a follow-up survey of selected elementary and secondary school teachers who participated in the Schools and Staffing Survey (SASS).1 SASS is the largest, most extensive survey of kindergarten through 12th grade (K–12) school districts, schools, teachers, and administrators in the United States today. It provides data on the characteristics and qualifications of teachers and principals, teacher hiring practices, professional development, class size, and other conditions in schools across the nation. TFS focuses on a sample of teachers who participated in SASS, including those teachers who left the K–12 teaching profession and those who continue to teach. TFS includes teacher data from public (including public charter) and private schools, similar to SASS. However, TFS does not include teachers who taught in a school funded by the Bureau of Indian Affairs (BIA) during the SASS school year due to insufficient sample sizes in TFS. Together, SASS and TFS data provide a multitude of opportunities for analysis and reporting on elementary and secondary educational issues. Background TFS is a follow-up of selected teachers from the SASS teacher surveys and is conducted during the school year following the SASS administration. It was conducted in the 1988–89, 1991–92, 1994–95, 2000–01, and 2004–05 school years (after the 1987–88, 1990–91, 1993–94, 1999–2000, and 2003–04 administrations of SASS, respectively). NCES currently plans to conduct the next survey in the 2008–09 school year; it will collect data from a subsample of teachers who participate in the 2007–08 SASS. Over time, the philosophy behind TFS has changed. The 2004–05 TFS more closely resembled the SASS teacher questionnaires than in any of the previous TFS administrations. There was a greater overlap of TFS and SASS teacher items, and there were fewer items unique to TFS, other than items pertaining to leaving last year’s teaching position. When examined together, the results of TFS and SASS can give researchers insight on many different educational issues, including the retention of teachers in public and private schools and teachers’ job satisfaction. Congress, state education departments, federal agencies, private school associations, teacher associations, and educational organizations have used data from the 1988–89, 1991–92, 1994–95, and 2000–01 surveys. In particular, results of these prior administrations have been used to analyze changes in the teacher labor force over time, to develop incentive programs to encourage teacher retention, and to understand the effects of school practices and policies on a teacher’s decision to continue teaching or leave the K–12 teaching profession. Purpose and Content of the Survey TFS is a one-time follow-up to the SASS teacher questionnaires and is conducted during the school year following SASS. The major objectives of the 2004–05 TFS are to measure the attrition rate for teachers, examine the characteristics of teachers who stayed in the teaching profession and those who left or retired, obtain activity or occupational data for those who left the position of a K–12 teacher, obtain 1 For a complete description of the 2003–04 Schools and Staffing Survey, see Documentation for the 2003–04 Schools and Staffing Survey (Tourkin et al. 2007).