Technology Assessment Health Risk Appraisal Technology Assessment Program July 6, 2011 Prepared for: Agency for Healthcare Research and Quality 540 Gaither Road Rockville, Maryland 20850 Health Risk Appraisal Technology Assessment Report Project ID: RSKA0410 July 6, 2011 McMaster University Evidence-based Practice Center Mark Oremus Amanda Hammill Parminder Raina This report is based on research conducted by the McMaster University Evidence-based Practice Center under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. HHSA 290 2007 10060 I). The findings and conclusions in this document are those of the author(s) who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. No statement in this article should be construed as an official position of the Agency for Healthcare Research and Quality or of the U.S. Department of Health and Human Services. The information in this report is intended to help health care decision-makers, patients and clinicians, health system leaders, and policymakers, make well-informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients. ii This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied. None of the investigators has any affiliations or financial involvement related to the material presented in this report. iii Peer Reviewers We wish to acknowledge individuals listed below for their review of this report. This report has been reviewed in draft form by individuals chosen for their expertise and diverse perspectives. The purpose of the review was to provide candid, objective, and critical comments for consideration by the EPC in preparation of the final report. Synthesis of the scientific literature presented here does not necessarily represent the views of individual reviewers. James Fries Professor Stanford University Palo Alto, CA Ron Goetzel Research Professor/Vice President Emory University/Thomson Reuters Washington, DC Victor J. Schoenbach Associate Professor Department of Epidemiology, UNC Gillings School of Global Public Health Chapel Hill, NC iv Acknowledgments We would like to thank our librarian, Maureen Rice, for her thorough management of search terms and project methodology. We would also like to thank the following people who assisted with screening and data extraction: Sameer Rawal, Catherine Salmon, Rob Stevens and Kate Walker. Thank you to Mary Gauld for her assistance with project coordination and to Kim Wittenberg for being our Task Order Officer. Thank you to our editorial staff, Leah Macdonald, Roxanne Cheeseman, Julianna Beckett, and Maureen Rice for providing invaluable input into this document, as well as to Bryan Cheeseman for his assistance managing our reference databases. v Contents Executive Summary .............................................................................................................. ES-1 Chapter 1. Introduction ...............................................................................................................1 Background .............................................................................................................................1 Definition ............................................................................................................................1 History ................................................................................................................................1 Health Risk Appraisals and the Elderly ................................................................................2 Earlier Literature Reviews ...................................................................................................2 Key Questions .....................................................................................................................3 Chapter 2. Methods .....................................................................................................................4 Literature Search Strategy .......................................................................................................4 Article Selection and Reporting ...............................................................................................4 Quality Assessment of Included Studies ..............................................................................5 Answering the Key Questions..............................................................................................6 Chapter 3. Results .......................................................................................................................7 Literature Review and Screening .............................................................................................7 Quality Assessment .................................................................................................................8 Key Questions ....................................................................................................................... 14 Chapter 4. Discussion ................................................................................................................ 39 Quality Assessment ............................................................................................................... 39 Key Questions ....................................................................................................................... 40 Additional Issues to Consider in the Evaluation of Health Risk Appraisal Programs .............. 44 Conclusions ........................................................................................................................... 46 References............................................................................................................................. 47 Tables Table 1. Quality assessment of randomized controlled trials with Jadad Scale .........................9 Table 2. Quality assessment of cohort studies with Newcastle-Ottawa Scale (NOS) .............. 11 Table 3. Methods and followup of frequencies ...................................................................... 21 Table 4. Outcomes in the extracted articles ............................................................................ 36 Figures Figure 1. Flow of articles through screening .............................................................................7 Figure 2. Distribution of quality ratings - randomized controlled trials. .....................................8 Figure 3. Distribution of quality ratings - cohort studies. ...........................................................8 Figure 4. Objectives of health risk appraisals .......................................................................... 14 Figure 5. Types of health risk appraisals ................................................................................. 15 Figure 6. Method of health risk appraisal administration ......................................................... 17 Figure 7. Number of articles reporting training of personnel administering health risk appraisals ................................................................................................................. 18 Figure 8. Methods of followup in health risk appraisals .......................................................... 19 Figure 9. Frequencies of followup/feedback in health risk appraisals ...................................... 19 Figure 10. Correlation between number of methods of followup/feedback and percentage of dropouts ................................................................................................................... 30 vi Figure 11. Correlation between frequency of followup/number of contacts and percentage of dropouts ................................................................................................................... 30 Figure 12. Mean age range in the extracted articles ................................................................... 32 Figure 13. Percentage of women in the extracted articles ........................................................... 32 Appendixes Appendix A. Search Strings .......................................................................................... A-1 – A-3 Appendix B. Data Abstraction Forms ............................................................................ B-1 – B-6 Appendix C. Excluded Studies .................................................................................... C-1 – C-23 Appendix D. Evidence Tables ................................................................................... D-1 – D-151 vii Executive Summary Introduction This technology assessment on health risk appraisals (HRAs) was prepared by the McMaster University Evidence-based Practice Center (MU-EPC) for the Centers for Medicare and Medicaid Services (CMS). The primary goals of the assessment were to describe key features of HRAs, examine which features were associated with successful HRAs, and discuss the applicability of HRAs to the Medicare population. CMS, in consultation with the MU-EPC and the Agency for Healthcare Research and Quality (AHRQ), drafted the following key questions (KQ) to guide the technology assessment. KQ1. Describe the characteristics of the provision of HRAs, including the following: a. Which specific HRAs were studied in the literature? b. What were the methods of HRA administration, e.g., telephone, Web-based, in the doctor’s office, community based, workplace based, or other? c. What was the training of personnel who administered HRAs? d. What were the methods and frequencies of followup? e. What were the characteristics of the patient populations who received HRAs? KQ2. What characteristics of HRAs (KQ1 a to e above) are associated with better health outcomes? KQ3. What is the generalizability of the data in KQ1 and 2 to the Medicare population or subpopulations? Methods Literature Search We searched Medline®, Embase®, Cochrane Central®, PsycINFO®, and Social Science Abstracts® from each database’s inception date to June 2010. Search terms included a combination of subject headings and text words for HRA, e.g., ‘health risk appraisal’, ‘health risk assessment’, and ‘health hazard assessment’. Search terms were combined with text words denoting individualized or personalized feedback, e.g., ‘feedback’, ‘counsel’, ‘individual’, ‘personal’, ‘tailor’. To supplement the database search, we hand searched The American Journal of Health Promotion. Inclusion/exclusion criteria. We included human studies published in the English language, provided they were randomized controlled trials (RCTs) or observational studies with comparison groups (e.g., cohort, case control). We excluded case reports, case series, narrative and systematic reviews, editorials, comments, letters, opinion pieces, abstracts, and conference proceedings. The primary focus of included studies had to be HRAs for single disease entities (e.g., cardiovascular disease (CVD)), multiple disease entities, or general health. Our definition of an HRA contained three components: participants provided self-reported information to identify individual risk factors for disease; participants received individualized health-related feedback based on the information they provided; and the information was used to give participants at ES-1 least one recommendation or intervention to promote health, sustain function, or prevent disease.1 We excluded studies reporting HRAs without all three components. We also excluded studies without health outcomes. Health outcomes encompassed items such as self-reported risk factors or diagnoses (including food consumption), clinical measures (e.g., blood pressure), or physical measures (e.g., weight, performance on a test such as grip strength). Examples of non-health outcomes were cost comparisons between HRA programs, measures of satisfaction with HRAs (e.g., appearance of questionnaires, depth of feedback), or reported proportions of persons who returned HRA questionnaires. Study Selection and Reporting A team of trained screeners applied the inclusion and exclusion criteria to the citations that were retrieved in the literature search. The screening process was divided into three levels: two levels of title and abstract screening and one level of full text screening. Studies that passed full text screening proceeded to full data extraction. Quality Assessment Following data extraction, two raters independently assessed study quality using the modified Jadad scale2,3 for RCTs and the Newcastle-Ottawa Scale (NOS)4 for cohort and case control studies. The items in these quality scales are consistent with the Task Force on Community Preventive Services guideline for collecting data in systematic reviews.5 The overall quality of each extracted article with a comparison group was rated ‘good,’ ‘fair,’ or ‘poor’ in accordance with the recommendations outlined in the AHRQ methods guide for systematic reviews.6 We graded the overall strength of evidence for KQ2 and KQ3 in each of four domains: risk of bias, consistency, directness, and precision. We classified overall strength as high, medium, low, or insufficient based on the recommendations of the AHRQ methods guide.6 Results Literature Review and Screening The literature search yielded 5,973 unique citations. In total, 5,434 citations (91 percent) were excluded from further review following the two levels of title and abstract screening. Of the 539 citations promoted to full text screening, 421 (78 percent) were excluded and 118 (22 percent) proceeded to full data extraction and quality assessment. We were unable to retrieve 8 articles despite extensive searches of library holdings from multiple universities, interlibrary loan requests, and contact with authors. Figure 1 (Chapter 3) depicts the flow of articles through screening. As well, the figure shows the reasons for article exclusion at full text screening. Citations were managed in an online database using Distiller SR. ES-2 The 118 extracted articles included 81 RCTs and 37 cohort studies. Four articles7-10 were companion papers containing additional analyses to supplement data reported in primary reports.11-14 Thus, the 118 articles represented 114 unique studies. Quality Assessment The majority of RCTs scored three or less on the modified Jadad scale, thus earning a poor quality rating. No RCTs had a good rating. Only one cohort study15 had a poor quality rating; the majority of cohort studies were fair quality. The major quality issues with the RCTs were an inadequate description of the randomization process, lack of reporting double blinding or the number and reason for participant withdrawals by study group, and absence of reporting methods to assess adverse effects. The most problematic quality issue for cohort studies was the lack of control for confounding. Seventeen studies reported use of chi-square, t-test, or analysis of variance statistical methods without mention of adjustment for potential confounders such as age or sex.15-31 In nonrandomized designs, adjustment for confounding is essential to account for the presence of influential third party variables that may be distributed unevenly across study groups. This adjustment is especially important in HRA studies, where third party variables such as age and sex often have an effect on health outcomes. Twenty cohort studies reported analyses adjusted for at least one potential confounding variable.32-51 The cohort studies had higher quality ratings and scores than the RCTs. This does not mean that HRA researchers designed better cohort studies than RCTs, nor that readers should give more credence to cohort study results. Indeed, several cohort studies did not apparently adjust for confounding. In addition, the Jadad and NOS scales were formulated to rate dissimilar study designs using different criteria. Furthermore, RCTs rank higher on the medical evidence hierarchy than cohort studies. The Oxford Centre for Evidence-based Medicine (CEBM) ranks RCTs as level 1b evidence and cohort studies as level 2b evidence.52 Thus, we can conclude that level 1b evidence for HRAs is poor to fair quality, while level 2b evidence is fair to good quality. From a quality standpoint, the extracted HRA articles may communicate a fairly similar level of evidence, regardless of study design. Key Questions KQ1. Describe the characteristics of the provision of HRAs, including the following: a. Which specific HRAs were studied in the literature? Most articles were concerned with participants’ general or cardiovascular health status and therefore employed a combination of questionnaire-based and physical or clinical HRAs. Only one specific HRA questionnaire, i.e., Personal Wellness Profile™,53 was certified by the National Committee for Quality Assurance (NCQA). b. What were the methods of HRA administration, e.g., telephone, Web-based, in the doctor’s office, community based, workplace based, or other? Many HRAs were administered in the workplace, with 56 articles set in places of employment.8,9,12,13,15-20,23,25,26,28-30,33,36-45,47,50,51,54-79 Two HRAs involved the workplace and ES-3