Mapping from SORT to GRADE Brian S. Alper, MD, MSPH, FAAFP Editor-in-Chief, DynaMed October 31, 2013 Disclosures – Brian S. Alper MD, MSPH, FAAFP is editor-in-chief for DynaMed (published by EBSCO) and medical director for EBSCO Information Services (full-time employee) – Allen Shaughnessy, PharmD, MMedEd is a Professor of Family Medicine at Tufts University. He is a co-investigator and received contract support from EBSCO for this work. Background - Grading quality of evidence and strength of recommendations Problem: > 100 different systems • Substantial confusion in interpreting trustworthiness of evidence, degree of obligation for recommendations, and how these two concepts are related • Concepts from one guideline do not easily translate to seemingly similar labels in another guideline Solution • Collaborative effort across reference sources and guideline developers to produce a unifying system • Continued effort to maintain, improve, and educate in use of the system • Strength of Recommendation Taxonomy (SORT) • Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) SORT GRADE • Started in 2004 • Started in 2000 • Initially created by 5 family • Used by > 70 guideline medicine and primary care developers and by Cochrane journals + 1 network (FPIN) • Quality of Evidence: • Quality of Evidence: – High (A) – Level 1 (good-quality patient- – Moderate (B) oriented evidence) – Low (C) – Level 2 (limited-quality – Very Low (D) patient-oriented evidence) • QoE Assessment – Level 3 (other evidence) – Downgrade for risk of bias, • QoE Assessment indirectness, inconsistency, – Level 2 if risk of bias, imprecision, publication bias inconsistency, or inadequate – Upgrade for large effect size+ statistical power SORT GRADE • Strength of Recommendation • Strength of Recommendation – A (consistent level 1 – Strong (1) evidence) – Weak (2) – B (inconsistent, single level 1, • SoR Determination: or level 2 evidence) – Benefits vs. harms – C (no patient-oriented – Values and preferences evidence) – Resource use • SoR Determination: • Further Development – Level of evidence – > 300 guideline developers • Further Development and contributors have – Limited to DynaMed use and provided continued feedback extension of level of evidence and adjustment criteria Background - DynaMed and SORT  DynaMed adopted SORT in 2004 • Added words to the labels – Level 1 (likely reliable) evidence – Level 2 (mid-level) evidence – Level 3 (lacking direct) evidence • Added more detailed, explicit criteria for Level 1 evidence (elevating “good-quality” to “high-quality”)  DynaMed dropped A/B/C strength of recommendation part of SORT in 2011 as this was poorly developed for classifying issues based on factors other than evidence quality  DynaMed now has > 56,000 level of evidence labels Evidence quality clearly labeled Quality limitation explained if evidence downgraded Aims - DynaMed and Guideline Developers DynaMed is collaborating with guideline developers for • Source for evidence (critically appraised) when developing guideline • Method to be notified when guidelines warrant updating • Outlet to disseminate guideline to reach point of care • Collaboration improves content (both ways) DynaMed use greatly increased efficiency of high-quality national treatment guideline for breast cancer in Costa Rica Multiple guideline developers have expressed: • Desire to use DynaMed for evidence source • Desire to use GRADE for evidence classification and recommendation classification • Perception that mapping from SORT to GRADE is difficult Methods - Mapping SORT to GRADE – round 1 • Perceived concerns to overcome for mapping SORT to GRADE: – Explicit level of evidence criteria listed for DynaMed mapped well to Risk of Bias portions of GRADE assessment – Precision mapped to “Adequate statistical power” – Indirectness, Consistency, and Publication bias were not explicitly stated – Criteria to differentiate Moderate-quality from Low-quality evidence were not explicitly stated in listing of Level 2 evidence • Focus on evidence assessments that would be “key recommendations” for a common topic • Project started with semi-complicated protocol to map SORT to GRADE and record what additional evidence appraisal was required Interim Results – Based on 115 evidence assessments mapped from SORT to GRADE – Need for additional evidence summarization limited to only 2 instances (both representing needs related to making a recommendation) • 1 required identification of a missing direct comparison trial to match desired conclusion for making recommendation • 1 required additional harm data to be summarized for evidence with summary limited to efficacy data – No need for additional critical appraisal • 1 item downgraded with explicit attention to publication bias (missed in editing but should have been recognized) – Realization that level of evidence criterion of “No other factors introducing bias” was being used to capture indirectness, imprecision, inconsistency, and (sometimes) publication bias