The Usefulness of ‘Think-Aloud’ for Evaluating Questionnaires in use in the Health Domain A thesis submitted to the University of Manchester for the degree of Doctor in Clinical Psychology (ClinPsyD) in the Faculty of Medical and Human Sciences 2014 Anna C. E. Phillips Section for Clinical and Health Psychology School for Psychological Sciences Table of Contents Table of Contents Table of Contents ..................................................................................................................... 2 List of Tables and Figures ......................................................................................................... 5 Abstract .................................................................................................................................... 6 Declaration ............................................................................................................................... 7 Copyright and Ownership ........................................................................................................ 7 Acknowledgements .................................................................................................................. 8 Paper 1: Systematic Review ..................................................................................................... 9 Abstract .............................................................................................................................. 10 Introduction ....................................................................................................................... 11 Method .............................................................................................................................. 12 Quality Appraisal ............................................................................................................ 14 Data Abstraction and Synthesis ..................................................................................... 16 Results ................................................................................................................................ 17 Study Quality .................................................................................................................. 21 Overview of Studies ....................................................................................................... 21 Aims of the Studies ........................................................................................................ 22 Justification of Think-aloud Methodology ..................................................................... 22 Populations .................................................................................................................... 24 Samples .......................................................................................................................... 24 Sample Size .................................................................................................................... 25 Education ....................................................................................................................... 25 Language ........................................................................................................................ 26 Methodology .................................................................................................................. 26 Analysis .......................................................................................................................... 27 Evaluation of the Usefulness of Think-aloud Techniques in Addressing the Aims of the Study .............................................................................................................................. 27 Discussion........................................................................................................................... 30 Strengths and limitations of the current review ............................................................ 32 Conclusions .................................................................................................................... 33 References ......................................................................................................................... 34 Paper 2: Research Paper ........................................................................................................ 40 2 Introduction ....................................................................................................................... 42 Method .............................................................................................................................. 46 Participants .................................................................................................................... 46 Measures ........................................................................................................................ 46 Design and Procedure .................................................................................................... 46 Analysis .......................................................................................................................... 47 Results ................................................................................................................................ 49 Sample characteristics ................................................................................................... 49 Distribution of problems across the questions .............................................................. 52 Nature of the problems identified ................................................................................. 52 Participants’ Opinion of the ERQ. .................................................................................. 56 Rationale for selecting the middle response-option ..................................................... 56 Spontaneous emotion-related comments ..................................................................... 57 Discussion........................................................................................................................... 58 Distribution and nature of problems with the ERQ ....................................................... 59 Interpretation of the middle response-option .............................................................. 60 Strengths and limitations of the current study .............................................................. 61 References ......................................................................................................................... 63 Paper 3: Critical Appraisal ...................................................................................................... 67 Introduction ....................................................................................................................... 68 The Original Study .............................................................................................................. 70 Transition to the Current Thesis ........................................................................................ 72 Learning from the Process ............................................................................................. 72 Methodological Challenges and Development of the Current Thesis ........................... 73 Paper 2 ............................................................................................................................... 75 The think-aloud method ................................................................................................ 75 The Sample ..................................................................................................................... 76 Paper 2: Conclusion ....................................................................................................... 77 Paper 1 ............................................................................................................................... 78 Process Issues ................................................................................................................ 78 Mixed-Methodology ...................................................................................................... 79 Key Findings ................................................................................................................... 80 Overall Conclusion ............................................................................................................. 81 References ......................................................................................................................... 82 3 Appendices ............................................................................................................................. 88 Appendix A: British Journal of Health Psychology: Author Guidelines .............................. 88 Appendix B: Confirmation of Ethical Approval for the original study from NRES Committee North West - Liverpool East ............................................................................ 94 Appendix C: Confirmation of Ethical Approval for the Substantial Amendment from NRES Committee North West - Liverpool East .......................................................................... 100 Appendix D: R&D Letter of Approval of Substantial Amendment ................................... 105 Appendix E: The Emotion Regulation Questionnaire (ERQ; Gross & John, 2003) ........... 108 Appendix F: Demographic Questionnaire ........................................................................ 110 Appendix G: Participant Information Sheet ..................................................................... 113 Appendix H: Participant Consent Form............................................................................ 118 Appendix I: CASP Quality Appraisal Tool ......................................................................... 121 Total word count: 19,646 4 List of Tables and Figures Figure 1- Literature Search: Procedure .................................................................. 14 Table 1- Summary of the Studies Included in the Review ................................... 16 Table 2- Number of Participants Demonstrating Each Demographic and Clinical Characteristic .......................................................................................................... 46 Table 3- Number of Participants Coded as Having Any Type of Problem for Each Question ................................................................................................................................ 48 Table 4- Frequency and Exemplar Problems ......................................................... 49 Table 5- Frequency and Nature of Problems for Each Question............................ 50 Table 6- Frequency and Type of Problem for Cognitive Reappraisal and Emotional Suppression Questions ........................................................................................... 52 Table 7- Coded Rationales for Selection of the Middle Response-Option ............ 53 5 Abstract The Usefulness of ‘Think-Aloud’ for Evaluating Questionnaires in use in the Health Domain. A thesis submitted to the University of Manchester for the degree of Doctor in Clinical Psychology (ClinPsyD). Anna C. E. Phillips, 2014. Self-report questionnaires are frequently used in health fields; however, subjective interpretation is often ignored. One way of assessing this is using techniques derived from cognitive interviewing. Of these, ‘think-aloud’, in which respondents speak their thoughts aloud as they complete a questionnaire, is the original paradigm. The thesis focusses on the use of ‘think-aloud’ methodology in the evaluation of questionnaires already in use in the health domain. The current thesis has been prepared in the format of scientific papers. Paper 1 is a systematic review (23 studies) of the appropriateness and usefulness of think-aloud techniques for evaluating health-related questionnaires. A descriptive account is provided of the aims of the studies reviewed; the justification for using think-aloud; populations studied; and methodology; an evaluative account depicts the usefulness of the think-aloud method in addressing researchers’ aims. Think- aloud was successfully used to address researchers’ aims and was effective at elucidating problems with questionnaires. Theoretical and clinical implications are discussed, and recommendations made for future research. Paper 2 is a cross-sectional observational study using think-aloud methods to examine the way in which people with End Stage Renal Disease (N=25) interpret and respond to the Emotion Regulation Questionnaire (ERQ). All questions were found to be problematic to some extent and reappraisal questions yielded the most problems. A tendency to deny or minimise negative emotions and present a positive self-image was also noted. Implications are discussed for use of the ERQ and replication with further samples suggested. Paper 3 is a critical appraisal of the above papers and provides personal reflections on the research process as a whole. The current thesis was a transition from a different study; amended due to time constraints. The journey is also outlined from this original study to the present thesis. Strengths and limitations are considered as well as areas for improvement and future research. 6 Declaration No portion of the work referred to in the thesis has been submitted in support of an application for another degree or qualification of this or any other university or other institute of learning. Copyright and Ownership The author of this thesis (including any appendices and/or schedules to this thesis) owns certain copyright or related rights in it (the “Copyright”) and s/he has given The University of Manchester certain rights to use such Copyright, including for administrative purposes. Copies of this thesis, either in full or in extracts and whether in hard or electronic copy, may be made only in accordance with the Copyright, Designs and Patents Act 1988 (as amended) and regulations issued under it or, where appropriate, in accordance with licensing agreements which the University has from time to time. This page must form part of any such copies made. The ownership of certain Copyright, patents, designs, trade marks and other intellectual property (the “Intellectual Property”) and any reproductions of copyright works in the thesis, for example graphs and tables (“Reproductions”), which may be described in this thesis, may not be owned by the author and may be owned by third parties. Such Intellectual Property and Reproductions cannot and must not be made available for use without the prior written permission of the owner(s) of the relevant Intellectual Property and/or Reproductions. Further information on the conditions under which disclosure, publication and commercialisation of this thesis, the Copyright and any Intellectual Property and/or Reproductions described in it may take place is available in the University IP Policy (see http://www.campus.manchester.ac.uk/medialibrary/policies/intellectual- property.pdf), in any relevant Thesis restriction declarations deposited in the University Library, The University Library’s regulations (see http://www.manchester.ac.uk/library/aboutus/regulations) and in The University’s policy on presentation of Theses 7 Acknowledgements I would like to acknowledge all who have contributed to the completion of this thesis. Special thanks to the patients who took part and the staff of the Renal Units; in particular, Sandip Mitra, Jon Courthold and Rincy Sajith. I am exceptionally grateful to my supervisors Professor Alison Wearden and Dr Dougal Hare for their time, contribution and encouragement throughout this process. I would also like to thank my wonderful colleagues for the camaraderie over the past three years, and of course, my family, friends and boyfriend for their unfailing love, patience and support. 8 Paper 1: Systematic Review The Usefulness of ‘Think-Aloud’ for Evaluating Questionnaires in the Health Domain: A Systematic Review. The following paper has been prepared with submission to the ‘British Journal of Health Psychology’ in mind. The guidelines for authors can be found in appendix A Word Count: 7,778 9 The Usefulness of ‘Think-Aloud’ for Evaluating Questionnaires in the Health Domain: A Systematic Review. Abstract Purpose. Measurement of health-related outcomes informs understanding of health, delivery of interventions and health-care planning, and is frequently undertaken using survey questionnaires. It is vital that researchers can be sure that these instruments measure what they purport to measure. One way of determining this is using techniques derived from cognitive interviewing. Of these, ‘think-aloud’, in which respondents speak their thoughts aloud as they complete a questionnaire, is the original paradigm. Recently, think-aloud has been increasingly used to evaluate questionnaires in the health domain. The present study is a systematic review (23 studies) of the appropriateness and usefulness of think-aloud techniques for evaluating health-related questionnaires. Method. A systematic database search was conducted. Papers were included if they used concurrent think-aloud methodology to evaluate questionnaires currently in use in the health domain. Results. The review presents a descriptive account of the aims of the studies reviewed; the justification for using think-aloud; populations studied; and methodology. Think-aloud was successfully used to address researchers’ aims and was effective at elucidating problems with questionnaires. Conclusions. Frequently-used questionnaires are not consistently understood or completed as researchers intended. Results suggest that further research into the validity of health-related questionnaires is warranted; moreover, this can be effected successfully using think-aloud techniques. 10