ABSTRACT Title of Dissertation: GRICEAN EFFECTS IN SELF-ADMINISTERED SURVEYS Ting Yan, Doctor of Philosophy, 2005 Dissertation directed by: Professor Roger Tourangeau Joint Program in Survey Methodology Despite the best efforts of questionnaire designers, survey respondents don’t always interpret questions as the question writers intended. Researchers have used Grice’s conversational maxims to explain some of these discrepancies. This dissertation extends this work by reviewing studies on the use of Grice’s maxims by survey respondents and describing six new experiments that looked for direct evidence that respondents apply Grice’s maxims. The strongest evidence for respondents’ use of the maxims came from an experiment that varied the numerical labels on a rating scale; the mean shift in responses to the right side of the rating scale induced by negative numerical labels was robust across items and fonts. Process measures indicated that respondents applied the maxim of relation in interpreting the questions. Other evidence supported use of the maxim of quantity — as predicted, correlations between two highly similar items were lower when they were asked together. Reversing the wording of one of the items didn’t prevent respondents from applying the maxim of quantity. Evidence was weaker for the application of Grice’s maxim of manner; respondents still seemed to use definitions (as was apparent from the reduced variation in their answers), even though the definitions were designed to be uninformative. That direct questions without filters induced significantly more responses on the upper end of the scale — presumably because of the presuppositions direct questions carried — supported respondents’ application of the maxim of quality. There was little support for respondents’ use of the maxim of relation from an experiment on the physical layout of survey questions; the three different layouts didn’t influence how respondents perceived the relation among items. These results provided some evidence that both survey “satisficers” and survey “optimizers” may draw automatic inferences based on Gricean maxims, but that only “optimizers” will carry out the more controlled processes requiring extra effort. Practical implications for survey practice include the need for continued attention to secondary features of survey questions in addition to traditional questionnaire development issues. Additional experiments that incorporate other techniques such as eye tracking or cognitive interviews may help to uncover other subtle mechanisms affecting survey responses. GRICEAN EFFECTS IN SELF-ADMINSTERED SURVEYS By Ting Yan Dissertation submitted to the Faculty of the Graduate School of the University of Maryland, College Park, in partial fulfillment of the requirements for the degree of Doctor of Philosophy 2005 Advisory Committee: Professor Roger Tourangeau, Chair Professor Frederick Conrad Professor Stanley Presser Professor John Robinson Professor Norbert Schwarz ©Copyright by Ting Yan 2005 Acknowledgements First, I would like to express my most sincere gratitude towards my academic advisor, Professor Roger Tourangeau, for his continuous support throughout the Ph. D program. I deeply appreciate his constructive advice and input, his time, and ongoing patience. The regular discussions with him have definitely benefited me and have greatly motivated me to move forward during the Ph. D program. During the frustrating period of dissertation- writing, he was always there to provide both academic guidance and emotional support (I am very sympathetic to him for the pain he had going through the first drafts of the dissertation). Without Dr. T., this dissertation wouldn’t have been possible. Iwould like to acknowledge the help of my committee in various stages of the dissertation process. I am grateful for their comments on and assistance with ideas, and personal help with data collection, data analysis, and writing. I am extremely thankful for their cooperation and accommodating efforts. Collection of data for the grid experiment in Chapter 2 was supported by grants from the National Science Foundation (SES-0106222) and National Institute of Health (R01 HD041386-01A1) to Roger Tourangeau, Mick Couper, Fred Conrad, and Reg Baker. I would like to extend my gratitude to the PIs, who allowed me to piggyback my experiment onto their web study. Special thanks go to Reg Baker, who not only made possible my last data collection effort but also made it happen in time for me to finish the dissertation. Lastly, I would like to thank my family for their constant reminders (“When will you be done?”), and Mr. Low Ke Bin for not asking when I will be done. It was their full support and complete understanding that made this happen. ii Table of Contents Acknowledgements...........................................................................................................ii Lists of Tables....................................................................................................................v Lists of Figures.................................................................................................................vi Chapter 1 Prior Work on Gricean Influences in Surveys.......................................1 1.1 Introduction.........................................................................................................1 1.2 Theoretical Formulation and Development of CP..............................................3 1.3 Gricean Effects in Survey Research...................................................................7 1.3.1 Maxim of Quantity......................................................................................9 1.3.2 Maxim of Quality......................................................................................14 1.3.3 Maxim of Relation....................................................................................17 1.3.4 Web Surveys.............................................................................................24 1.3.5 Conclusions...............................................................................................27 1.4 Outline of Dissertation......................................................................................28 Chapter 2 The Maxim of Relation and Question Presentation in Web Surveys.29 2.1 Introduction.......................................................................................................29 2.2 Method..............................................................................................................33 2.3 Results...............................................................................................................36 2.4 Conclusions.......................................................................................................40 Chapter 3 The Maxim of Manner and Providing Definitions..............................43 3.1 Introduction.......................................................................................................43 3.2 Method..............................................................................................................50 3.3 Results...............................................................................................................53 3.3 Conclusions.......................................................................................................60 Chapter 4 Three Maxims.........................................................................................62 4.1 The Maxim of Relation and Numerical Values of Rating Scales.....................62 4.1.1 Introduction...............................................................................................62 4.1.2 Method......................................................................................................67 4.1.3 Results.......................................................................................................69 4.1.4 Conclusions...............................................................................................75 4.2 The Maxim of Quantity and Similar Items.......................................................76 4.2.1 Introduction...............................................................................................76 4.2.2 Method......................................................................................................79 4.2.3 Results.......................................................................................................81 4.2.4 Conclusions...............................................................................................84 4.3 The Maxim of Quality and Presuppositions.....................................................85 4.3.1 Introduction...............................................................................................85 4.3.2 Method......................................................................................................90 4.3.3 Results.......................................................................................................92 iii 4.3.4 Conclusions...............................................................................................96 Chapter 5 Conclusions and Discussion...................................................................98 5.1 Summary of Results..........................................................................................98 5.2 Discussion.......................................................................................................105 5.2.1 Automatic versus Controlled Processing................................................106 5.2.2 Gricean Effects or Satisficing.................................................................109 Reference.......................................................................................................................112 iv Lists of Tables Table 1.1. Grice’s Four Conversational Maxims.............................................................5 Table 1.2. Horn’s Neo-Gricean Principles vs. Grice’s Original Formulation..................7 Table 1.3. Summaries of Studies on Maxim of Quantity...............................................10 Table 1.4. Correlations Between Responses to General and Specific Questions, by Study and Condition.................................................................................................11 Table 1.5. Studies Examining the Maxim of Quality.....................................................16 Table 1.6. Research Work on the Maxim of Relation....................................................18 Table 2.1. Number of Completed Cases Per Experimental Condition...........................34 TTTTaaaabbbblllleeee 2222....2222.... Experiment 1: Introductions, Target Items and Follow-up Questions.........35 Table 3.1. Number of Participants Assigned to Experimental Conditions....................52 Table 3.2. Variances of Responses to Four Target Questions By Definition Condition54 Table 3.3. Percentage of “Yes” Responses to Borderline Instances by Definition Condition...................................................................................................................54 Table 3.4. Regression Coefficients from Multiple Regression Models by Target Item 56 Table 3.5. ANOVA Results on Total Response Time...................................................56 Table 4.1. Experimental Conditions of Studies on Numerical Values of Rating Scales65 Table 4.2. Experiment 4: Number of Completes Per Experimental Condition..............68 Table 4.3. Questions Used in Experiment 4 on Maxim of Relation..............................69 Table 4.4. Experiment 4: Mean Ratings By Experimental Condition...........................70 Table 4.5. Experiment 4: Two-way ANOVA Results...................................................70 Table 4.6. Experiments 4 and 5: Number of Completes per Experimental Condition..80 Table 4.7. Items for Experiment 5..................................................................................80 Table 4.8. Experiment 4: Correlation Coefficients by Experimental Condition............82 Table 4.9. Experiment 5: Correlation Coefficients by Experimental Conditions..........83 Table 4.10. Experiment 6: Questions in the Spending Block..........................................90 Table 4.11. Experiment 6: Number of Completes by Experimental Condition...............91 Table 4.12. Importance of Four Issues by Question Format............................................93 Table 4.13. Mean Importance Ratings (and Percent Selecting Options Above Midpoint) by Whether Issue Included In Prior Block................................................................95 Table 4.14. Correlations Between Importance Items (Inference Questions) and Concern Items..........................................................................................................................96 Table 5.1. Summary of Results......................................................................................99 v Lists of Figures Figure 2.1. Plots of Cronbach’s Alphas by Experimental Condition.............................37 Figure 2.2. Mean Perceived Relatedness Rating by Experimental Condition...............39 Figure 3.1. Mean Standardized Response Times for HNC and LNC Respondents.......58 Figure 3.2. Mean Ratings of Whether Survey Terms Are Used in Technical Sense by Definition Condition.................................................................................................60 Figure 4.1. Example of a Faded Scale...........................................................................68 Figure 4.2. Mean Shifts Due to Negative Numbers by Their Font Across 4 Items.......70 Figure 4.3. Percent Correctly Recalling Numerical End Point by Scale Condition......72 Figure 4.5. Percentage of Respondents Inferring “Presence of Failure” by Scale Condition...................................................................................................................74 vi Chapter 1 Prior Work on Gricean Influences in Surveys 1.1 Introduction Survey errors are often classified into coverage, sampling, nonresponse, and measurement errors (Groves, 1989). Measurement errors are further subdivided by source – the interviewer, the respondent, or the instrument (Groves, 1989). The traditional mathematical model of survey error, as set forth in the work by Hansen, Hurwitz, and Madow (1953), and Cochran (1977), starts with the variance of the sample mean of a variable measured without error and based on a simple random sample, and then adds additional variance components representing measurement error (Hansen, Hurwitz, & Bershad, 1961). Implicit in the Hansen-Hurwitz-Bershad measurement error model is the emphasis on the interview process (the “essential survey conditions”) and on the “processors” of survey data (including supervisors, interviewers, coders, etc.) rather than on individual respondents. The traditional model assigns respondents a passive role and largely ignores them. Another weakness of this model lies in its focus on the consequences (rather than causes) of measurement error on survey estimates. The model is not informative as to how measurement errors arise or how to prevent or reduce them. The CASM (Cognitive Aspects of Survey Methodology) movement in the 80s and 90s fostered a shift in survey methods research to the cognitive paradigm. The CASM movement focused on the cognitive processes by which respondents arrive at and report an answer using concepts drawn mostly from cognitive psychology (see Hippler, Schwarz, & Sudman, 1987; Jabine, Straf, Tanur, & Tourangeau, 1984; Sirken, Herrmann, Schechter, Schwarz, Tanur, & Tourangeau, 1999; Sudman, Bradburn, & Schwarz, 1996). This new paradigm centered on the causes ofmeasurement errors and motivated practical 1