New York College of Osteopathic Medicine Learning Outcomes Assessment 2009-2010 January 2009 Pre-Matriculation Pre- Doctoral Career Data data Post-Graduate Data Taskforce Members John R. McCarthy, Ed.D. Pelham Mead, Ed.D. Mary Ann Achziger, M.S. Felicia Bruno, M.A. Claire Bryant, Ph.D. Leonard Goldstein, DDS, PH.D. Abraham Jeger, Ph.D. Rodika Zaika, M.S. Ron Portanova, Ph.D. Table of Contents OVERVIEW 4 I. Introduction and Rationale 5 II. Purpose and Design 9 III. Specifics of the Plan 11 Mission of NYCOM 11 Learning Outcomes 11 Compiling the Data 17 Stakeholders 17 IV. Plan Implementation 18 Next Steps 18 V. Conclusion 20 A. OUTCOME INDICATORS – DETAIL 24 1. Pre-matriculation data 24 Forms 26 2. Academic (pre-clinical) course-work 47 Forms – LDB / DPC Track 49 Forms – Institute for Clinical Competence (ICC) 55 3. Clinical Clerkship Evaluations / NBOME Subject Exams 86 Forms 88 4. Student feedback (assessment) of courses/Clinical clerkship PDA project 92 Forms 94 5. COMLEX USA Level I, Level II CE & PE, Level III data (NBOME) 120 6. Residency match rates and overall placement rate 121 2 7. Feedback from (AACOM) Graduation Questionnaire 122 Forms 123 8. Completion rates (post-doctoral programs) 142 9. Specialty certification and licensure 143 10. Career choices and geographic practice location 144 11. Alumni Survey 145 Forms 146 B. BENCHMARKS 151 Bibliography 152 Appendices: 153 Chart 1 Proposed Curriculum and Faculty Assessment Timeline Institute for Clinical Competence: Neurological Exam – Student Version Parts I & II Taskforce Members List of Tables and Figures Figure 1 Cycle of Assessment 9 Figure 2 Outcome Assessment along the Continuum 15 Figure 3 Data Collection Phases 22 Table 1 Assessment Plan Guide 23 3 New York College of Osteopathic Medicine Learning Outcomes Assessment Plan February 2009 Overview This document was developed by the NYCOM Task Force on Learning Outcomes Assessment and was accepted by the dean in January 2009. Although a few of the assessment tools and processes described in the document are new, most have been employed at NYCOM since its inception to inform curriculum design and implementation and to gauge progress and success in meeting the institution’s mission, goals and objectives. The Learning Outcomes Assessment Plan documents the processes and measures used by the institution to gauge student achievement and program (curricular) effectiveness. The results of these activities are used by faculty to devise ways to improve student learning and by administrators and other stakeholder groups to assess institutional effectiveness and inform planning, decision-making, and resource allocation. Certain of the measures described in later sections of this document constitute key performance indicators for the institution, for which numerical goals have been set. Performance on these measures has a significant effect on institutional planning and decision-making regarding areas of investment and growth, program improvement, and policy. 4 Key performance indicators and benchmarks are summarized below and also on (cid:83)(cid:68)(cid:74)(cid:72) 151 (cid:82)(cid:73)(cid:3)(cid:87)(cid:75)(cid:72)(cid:3)(cid:83)(cid:79)(cid:68)(cid:81)(cid:17) Indicator Benchmarks  Number of Applicants Maintain relative standing among Osteopathic Medical Colleges  Admissions Profile Maintain or improve current admissions profile based on academic criteria (MCAT, GPA, Colleges attended  Attrition 3% or less  Remediation rate (preclinical) 2% reduction per year  COMLEX USA scores Top quartile (first-time pass rates, mean scores)  Students entering Maintain or improve OGME placement OGME  Graduates entering Maintain or improve Primary Care placement Primary Care careers  Career characteristics Regarding Licensure, Board Certification, Geographic Practice, and Scholarly achievements--TBD I. Introduction and Rationale At NYCOM we believe it is our societal responsibility to monitor our students’ quality of education through continual assessment of educational outcomes. On-going program evaluation mandates longitudinal study (repeated observations over time) and the utilization of empirical data based on a scientific methodology. At Thomas Jefferson University, an innovative study was implemented circa 1970, which was ultimately titled “Jefferson Longitudinal Study of Medical Education”.1 As a result of implementation of this longitudinal study plan, Thomas Jefferson University was praised by the 1 Center for Research in Medical Education and Health Care: Jefferson Longitudinal Study of Medical Education, Thomas Jefferson University, 2005. 5 Accreditation Team for the Middle States Commission on Higher Education for “…..their academic interest in outcome data, responsiveness to faculty and department needs and the clear use of data to modify the curriculum and teaching environment….their use of this data has impacted many components of the curriculum, the learning environment, individual student development, and program planning…” (TJU, 2005). The Jefferson Longitudinal Study of Medical Education has been the most productive longitudinal study of medical students and graduates of a single medical school. This study has resulted in 155 publications in peer review journals. Many were presented before national or international professional meetings prior to their publication (TJU, 2005). According to Hernon and Dugan (2004), the pressure on higher education institutions to prove accountability has moved beyond the acceptance and reliance of self-reports and anecdotal evidence compiled during the self-regulatory accreditation process. It now encompasses an increasing demand from a variety of constituencies to demonstrate institutional effectiveness by focusing on quality measures, such as educational quality, and cost efficiencies. Accountability focuses on results as institutions quantify or provide evidence that they are meeting their stated mission, goals, and objectives. Institutional effectiveness is concerned, in part, with measuring (Hernon and Dugan, 2004):  Programmatic outcomes: such as applicant pool, retention rates, and graduation rates. Such outcomes are institution-based and may be used to compare internal year-to-year institutional performance and as comparative measures with other institutions.  Student learning outcomes: oftentimes referred to as educational quality and concerned with attributes and abilities, both cognitive and affective, which reflect how student experiences at the institution supported their development as individuals. Students are expected to demonstrate acquisition of specific knowledge and skills. 6 At NYCOM, we recognize that our effectiveness as an institution must ultimately be assessed and expressed by evaluating our success in achieving our Mission in relation to the following Outcomes: 1. Student Learning / Program Effectiveness 2. Research and Scholarly Output 3. Clinical Services The present document focuses on #1, above, viz., Student Learning / Program Effectiveness. That is, it is intended only as a Learning Outcomes Assessment Plan. At the same time, we are cognizant that Institutional Effectiveness/Outcomes derive from numerous inputs, or “means” to these “ends,” including: 1. Finances 2. Faculty Resources 3. Administrative Resources 4. Student Support Services 5. Clinical Facilities and Resources 6. Characteristics of the Physical Plant 7. Information Technology Resources 8. Library Resources We believe it is our obligation to continually assess the impact of any changes in the inputs, processes, and outputs of this institution. The evaluation approach in this Assessment Plan provides for on-going data collection and analysis targeted specifically at assessing outcomes of student achievement and program effectiveness (educational quality). Assessment of achievement and program effectiveness is based on objective, quantifiable information (data). As a result of the NYCOM Learning Outcome Assessment Plan’s continual assessment cycle, the report is available, with scheduled updates, as a resource in the decision-making process. 7 The report provides outcomes data, recommendations, and suggestions intended to inform key policy makers and stakeholders2 of areas of growth and/or improvement, together with proposed changes to policy that strengthen both overall assessment and data-driven efforts to improve student learning. 2 NYCOM Administration, academic committees, faculty, potential researchers, and students. 8 II. Purpose and Design Well-designed plans for assessing student learning outcomes link learning outcomes, measures, data analysis, and action planning in a continuous cycle of improvement illustrated below. Figure 1 Cycle of Assessment Define intended Learning Outcomes Review results and use to make Identify decisions regarding program Start methods improvement Here of measuring outcomes Collect Data Ten principles guide the specifics of NYCOM’s Learning Outcomes Assessment Plan: 1. The plan provides formative and summative assessment of student learning.3 2. The primary purpose for assessing outcomes is to improve student learning. 3. Developing and revising an assessment plan is a long-term, dynamic, and collaborative process. 4. Assessments use the most reliable and valid instruments available. 3 Examples of the former include post-course roundtable discussions, Institute for Clinical Competence (ICC) seminars, and data from the Course/Faculty Assessment Program. Examples of the latter include the AACOM Graduation Questionnaire, COMLEX scores, NBOME subject exam scores, and clerkship evaluations. 9 5. Assessment priorities are grounded in NYCOM’s mission, goals, and learning outcomes. 6. The assessment involves a multi-method approach. 7. Assessment of student learning is separate from evaluation of faculty. 8. The primary benefit of assessment is the provision of evidence-based analysis to inform decision-making concerning program revision and improvement and resource allocation. 9. The assessment plan must provide a substantive and sustainable mechanism for fulfilling NYCOM’s responsibility to ensure the quality, rigor, and overall effectiveness of our programs in educating competent and compassionate physicians. 10. The assessment plan yields valid measures of student outcomes that provide stakeholders with relevant and timely data to make informed decisions on changes in curricular design, implementation, program planning, and the overall learning environment. Outcomes assessment is a continuous process of measuring institutional effectiveness focusing on planning, determining, understanding, and improving student learning. At NYCOM, we are mindful that an integral component of this assessment plan is to ensure that the plan and the reporting process measures what it is intended to measure (student achievement and program effectiveness). 10