Appraising Research in Second Language Learning Language Learning & Language Teaching (LL&LT) The LL&LT monograph series publishes monographs, edited volumes and text books on applied and methodological issues in the field of language pedagogy. The focus of the series is on subjects such as classroom discourse and interaction; language diversity in educational settings; bilingual education; language testing and language assessment; teaching methods and teaching performance; learning trajectories in second language acquisition; and written language learning in educational settings. Editors Nina Spada Nelleke Van Deusen-Scholl Ontario Institute for Studies in Education, Center for Language Study, University of Toronto Yale University Volume 28 Appraising Research in Second Language Learning. A practical approach to critical analysis of quantitative research. Second edition by Graeme Keith Porte Appraising Research in Second Language Learning A practical approach to critical analysis of quantitative research Second edition Graeme Keith Porte University of Granada John Benjamins Publishing Company Amsterdam / Philadelphia TM The paper used in this publication meets the minimum requirements of 8 American National Standard for Information Sciences – Permanence of Paper for Printed Library Materials, ansi z39.48-1984. Library of Congress Cataloging-in-Publication Data Porte, Graeme Keith. Appraising research in second language learning : a practical approach to critical analysis of quantitative research / by Graeme Keith Porte. -- 2nd ed. p. cm. (Language Learning & Language Teaching, issn 1569-9471 ; v. 28) Includes bibliographical references and index. 1. Second language acquisition--Research--Methodology. I. Title. P118.2.P66 2010 418.007’2--dc22 2010022707 isbn 978 90 272 1991 6 (Hb ; alk. paper) isbn 978 90 272 1995 4 (Pb ; alk. paper) isbn 978 90 272 8796 0 (Eb) © 2010 – John Benjamins B.V. No part of this book may be reproduced in any form, by print, photoprint, microfilm, or any other means, without written permission from the publisher. John Benjamins Publishing Co. · P.O. Box 36224 · 1020 me Amsterdam · The Netherlands John Benjamins North America · P.O. Box 27519 · Philadelphia pa 19118-0519 · usa To Cristina and Amy, expert appraisers Table of contents Preface xi Notes on the second edition xxiii Textbook and worked sample appraisals 1 chapter 1 Introduction: The abstract 3 1.1 Worked sample appraisal 4 chapter 2 Introduction: The problem statement 9 2.1 The background to the problem 11 2.2 The problem statement 12 2.3 Initial indication of variables in the problem statement 13 2.4 Stated contribution to theory and practice 15 Worked sample appraisal 16 chapter 3 Introduction: The review of the literature 20 3.1 Relevance and importance of work cited 22 3.2 Critical address of the literature 24 3.3 Relating the review to the background statement 26 3.4 Time period covered in review 27 3.5 Empirical studies c overed in review 28 Worked sample appraisal 29 chapter 4 Introduction: Research questions and variables 36 4.1 Research questions and hypotheses 36 4.2 Nature of the research questions 38 4.3 Nature of the research hypotheses 39 4.4 General appraisal of the research questions/hypotheses 42 4.5 Identification of variables 45 4.6 Variables potentially affecting findings 50 4.7 Operational definitions 52 Worked sample appraisal 56 viii Appraising research in second language learning chapter 5 Method: Participants and materials 62 5.1 Descriptive data: Participants 64 5.2 Descriptive data: Groups 68 5.3 Threats to internal validity 72 5.4 Threats to external validity 74 5.5 Descriptive data: Materials 76 5.6 Instruments of measurement: Reliability 78 5.7 Instruments of measurement: Validity 80 5.8 Realisation of main variables 84 Worked sample appraisal 85 chapter 6 Method: Procedures and research design 94 6.1 Procedures: Timing of events 96 6.2 Procedures: Test or practice effects 97 6.3 Procedures: Ethical issues and instructions to participants 99 6.4 Procedures: Reactivity 103 6.5 Procedures: Environmental conditions 107 6.6 Research design: Design boxes 110 6.7 Research design: Classifications 116 Worked sample appraisal 128 chapter 7 Method: Data analysis 136 7.1 Data analysis: Description of procedures 136 7.2 Data analysis: Appropriateness of procedures 138 7.3 Data analysis: Statistical assumptions 140 Worked sample appraisal 146 chapter 8 Results: The nature and presentation of findings 148 8.1 Adequacy of data provided 153 8.2 Graphical displays of results 154 8.3 Descriptive statistics 157 8.4 Statistical procedures 162 8.4.1 Correlation 164 8.4.2 Regression 169 8.4.3 t-tests 173 8.5 Explanations of data 176 8.6 Effect size 179 Table of contents ix 8.7 Other statistical procedures 183 8.7.1 Analysis of variance 183 8.7.2 Chi-square 194 8.8 Practical significance and meaningfulness 199 Worked sample appraisal 200 chapter 9 Discussion and conclusions 206 9.1 Outcomes in terms of the original research questions/hypotheses 208 9.2 Inferences drawn from outcomes 210 9.3 Outcomes related to literature review 212 9.4 Limitations and future research 213 9.5 Practical implications 215 9.6 Assessment of overall confidence 219 Worked sample appraisal 222 Workbook 231 Workbook (guided and unguided appraisals) 233 Abstract 3 (Guided appraisal) 233 Abstract 4 (Unguided appraisal) 235 The background to the problem and the problem statement 3 (Guid ed appraisal) 236 The background to the problem and the problem statement 4 (Unguid ed appraisal) 238 The review of the literature 3 (Guided appraisal) 240 The review of the literature 4 (Unguided appraisal) 243 Research questions and variables 3 (Guided appraisal) 245 Research questions and variables 4 (Unguided appraisal) 247 Participants and materials 3 (Guided appraisal) 250 Participants and materials 4 (Unguided appraisal) 254 Procedures and research design 3 (Guided apprais al) 256 Procedures and research design 4 (Unguided apprais al) 259 Results: The nature and presentation of findings 3 (Guided apprais al) 261 Results: The nature and presentation of findings 4 (Unguided apprais al) 264 Discussion and conclusions 3 (Guided appraisal) 267 Discussion and conclusions 4 (Unguided appraisal) 270 Glossary of key terms in quantitative research 273 Appendices 291 Further reading 303 Index of main subject entries in textbook 305