Patricia Fullmer Assessment of Tutoring Laboratories in a Learning Assistance Center The Learning Resource Center at Lincoln University, Pennsylvania, provides tutoring laboratories that are required for developmental reading, writing, and math courses. This article reviews the processes used to plan and determine the effectiveness of the tutoring laboratories, including logic models, student learning outcomes, and the results of an analysis of the differences between pretest and posttest scores using a t test and eta2 (measure of effect). The pretest and posttest scores of students who attended at least six lab sessions were examined. The t test results indicated a high level of significance and the eta2 score indicated a moderate to strong effect. Students in the tutoring laboratories showed gains in academic skills from the pretest to the posttest, supporting the effectiveness of the tutoring laboratories, and the assessment process provided accessible information that was used to improve the tutor- ing laboratory program. L incoln University, Pennsylvania, the first Historically Black College and University (HBCU) in the United States, is situated on the East Coast near Philadelphia. The University’s Learning Resource Center (LRC) provides individual tutoring, student workshops, and tutoring laboratories required for success (developmen- tal) courses in education, English, and mathematics. In 2008, the LRC was enhanced through the addition of computers and updated technol- ogy, and its administrators began to search for an appropriate online tutoring program. The goal was to improve the tutoring laboratories, foster the engagement of students, and effectively utilize the available professional tutors. Assessment of Tutoring Laboratories 67 68 Journal of College Reading and Learning, 42(2), Spring 2012 Online tutoring programs, with available professional tutors to guide students, provide an efficient way to reach larger numbers of students and still retain individualized instruction and practice. An exemplary online tutoring program can also provide reliable service to students 24 hours per day, seven days per week, when professional or peer tutors are unavailable. Gillen, Lye, and Vaughn (2004) found that online tutoring provided an effective use of class time as well as opportunities for practice for students. With these considerations in mind, the staff of the LRC reviewed online tutoring programs for use in the tutoring laboratories, and began the process of implementing online tutoring. This article examines the incorporation of online tutoring into required tutoring laboratories for developmental courses in reading, writing, and mathematics, including the evaluation of the effectiveness of the laboratories. As part of the self-assessment process, the online tutoring programs made available information that was crucial in the evaluation of the tutoring laboratories. Students’ gains in academic skills from a pretest to a posttest were examined and analyzed, leading to modifica- tions and improvements in the tutoring laboratory programs. Tutoring Laboratories Online tutoring programs proved crucial in the enhancement of the tutoring laboratories in the LRC. The year prior to fall 2008 had been spent researching online programs in order to make an informed decision about which program to implement. At that time, the reading and writing tutoring laboratories used paper-and-pencil exercises, and no mathemat- ics lab existed. The Reading Lab used binders of reading selections with multiple-choice questions, and the Writing Lab required students to write one paragraph per lab session on prompts devised by the tutors. At the time, there were two basic types of online tutoring available: online tutoring (which could be either synchronous or asynchronous, and involves a live tutor who is matched with a student), and web- based tutorials (which consist of content that a student works through independently). After reviewing input from the university’s academic departments and information on a number of online tutoring programs, content-type programs were chosen. The nature of the required tutor- ing laboratories and the need for diagnostic assessments were the main factors in this decision. The goal of the required tutoring laboratories was to provide review and practice of the content areas covered in the students’ courses. In collaboration with the Education and English Departments, the LRC staff chose to implement My Reading Lab (2011a) and My Writing Assessment of Tutoring Laboratories 69 Lab (2011) from Pearson Publishing. The tutors customized the content of both programs in the areas requested by the Education and English Departments. The Education Department, which provides the devel- opmental reading courses, requested that the Reading Lab concentrate on the topics that are covered in the syllabus and on the department’s reading comprehension assessment. The English Department requested that the Writing Lab focus on grammar while the professors would focus on the writing process in the classroom. The Mathematics Department recommended the purchase of ALEKS®, an online mathematics program published by McGraw-Hill. ALEKS®, Assessment and LEarning in Knowledge Spaces, utilizes artificial intelligence programming to individualize math problems by determining what the student knows and what the student is prepared to learn. Software engineers, cognitive scientists, and mathematicians assembled by Professor Jean-Claude Falmagne of New York University and the University of California (Irvine) developed ALEKS® with the support of a National Science Foundation grant. The basis of ALEKS® is Knowledge Space Theory developed through the research of Profes- sor Falmagne and Professor Jean-Claude Doignon at the University of Brussels in the area of mathematical cognitive science (ALEKS®, 2011). The online tutoring programs used in the LRC in the reading, writing, and math labs provided a computer-scored diagnostic pretest and post- test, a direct measurement of skills learned, and a solid foundation on which to base student learning outcomes and an analysis of effective- ness. Students entered the online tutoring programs using access codes, and professors and tutors received a username and password for free access. Each program was tailored to the student’s study plan and to his or her needs. A management module allowed professors and tutors to review students’ work. Beginning in fall 2008, students accessed online tutoring programs to take a diagnostic assessment after which an individualized study pro- gram was generated for each student. At the end of the semester, each student completed a postdiagnostic assessment to determine the amount of improvement from pretest to posttest. The results of the diagnostic testing were examined, and student learning outcomes were developed. Laboratory Tutoring Sessions The Reading Lab and Writing Lab weekly tutoring sessions, each last- ing 50 minutes, began with a 15-minute review mini lesson presented by the tutors, after which the students worked on the online program in that week’s topic area. Each week’s designated topic was chosen in collaboration with the respective academic departments. The Reading 70 Journal of College Reading and Learning, 42(2), Spring 2012 Lab topics included main idea, supporting details, patterns of organiza- tion, inference, critical thinking, vocabulary, and purpose and tone. The sequence of topics for the Writing Lab included thesis statement, point of view, revision/concise language, proofreading and parallel structure, sentence fragments, run-ons/comma splices, commas, pronouns, verb agreement, and parts of speech. The Math Lab mini review topics included fractions, operations, decimals, percents, introduction to alge- bra, real numbers, order of operations, and linear equations. The online tutoring programs provided an efficient way to assess the effectiveness of the tutoring laboratories. Prior to the online programs, students completed paper-and-pencil pretests and posttests that were scored by hand, with scoring taking up valuable tutor time that could have been best spent with students. Also, the scoring of the pretests under the previous system was not completed until near the end of the semester, negating any formative assessment value. With the instan- taneous scoring of the pretests and posttests with the online system, students immediately saw their areas of strengths and weaknesses, and the tutors adapted the tutoring laboratories to proactively assist the students in their areas of need. The online tutoring programs also provided individual progress reports for each student, as well as class progress reports, that fulfilled grant- reporting requirements. At the end of each semester information was available in the online tutoring programs for a summative evaluation and an assessment of effectiveness of the tutoring laboratories. Assessment The Council for the Advancement of Standards in Higher Education (CAS) (2010) provides evaluation guidelines for learning assistance programs. The guidelines detail how exemplary learning assistance programs systematically and periodically gather data from a variety of sources, analyze the data, and use the results to make improvements in the program. The CAS self-evaluation process and guidelines made available parameters of a high-quality program, in essence a blueprint for exemplary services. In 2008-2009, the LRC conducted a self-evaluation using the CAS guidelines and the resulting recommendations were the basis of the present assessment process, including the development and modifica- tion of student learning outcomes and the use of a SWOT (Strengths, Weaknesses, Opportunities, and Threats) analysis and action plan. The LRC is currently progressing through a self-assessment in preparation for applying for certification of the tutoring laboratories from the National Association for Developmental Education. Assessment of Tutoring Laboratories 71 In the LRC, qualitative and quantitative methods that included infor- mation from all stakeholders were used to evaluate and improve the program and to enhance services. The LRC examined quantitative data from the online tutoring programs and, additionally, reviewed qualita- tive information from self-report surveys completed by students and professors. After the analysis of the data, changes were implemented to the tutoring laboratories: review topics were dropped or added, handouts were developed, and additional practice on certain topics was included in the laboratories. In addition, the establishment of student learning outcomes was recommended as a yardstick for measuring the effectiveness of the program. The online tutoring programs provided diagnostic pretests and posttests on which measureable student learning outcomes were based. Each semester, data on student learning outcomes were reviewed and the student learning outcomes revised. Moreover, accreditation from professional organizations such as the Middle States Commission on Higher Education (MSCHE) required direct evidence of student learning, including improvement between a pretest and a posttest, in order to meet official standards (MSCHE, 2009; Suskie, 2009). The online assessments conducted in the LRC fulfilled the requirements of the CAS’s self-assessment and the standards of the MSCHE. Student Learning Outcomes After completing a self-assessment based on the CAS’s standards and guidelines, the LRC staff established student learning outcomes. These outcomes were then modified each semester after a review and analysis of data from a pretest and posttest, and after an analysis of responses from an end-of-semester student survey and a professor survey. A graphic organizer, in the form of a table, was created to facilitate the development of the student learning outcomes. The student learning outcomes were based primarily on an analysis of the results of the comparison of the pretest and the posttest that shows the increase, or lack thereof, of the skills of the participating students. Each semester the student learning outcomes were revised after determining the percent improvement from pretest to posttest. The lab team then created an action plan consisting of the strategies to be used in the following semester to assist students in increasing their skills. Table 1 illustrates the fall 2010 student learning outcomes for the LRC Writing Lab. 72 Journal of College Reading and Learning, 42(2), Spring 2012 Action Plan Students will improve sentence grammar by: Working on individual My study plans in Writing Lab. Writing sentences and paragraphs. Using authentic stu-dent writing from class assignments. Receiving tutor-direct-ed feedback. Modified Student Learning Outcomes Students will increase their skills in the area of sentences. English 098 students will show a 50% improve-ment from the pretest to the posttest in sentence grammar for those who attend six or more ses-sions. English 099 students will show a 33% improve-ment from the pretest to the posttest in sentence grammar for those who attend six or more sessions. Findings For students who attended six or more sessions: English 098 students showed a 54% improvement in sentence grammar. English 099 students showed a 48% improvement in sentence grammar. The significant improvement of the students in the area of sen-tence grammar results from the following reasons: The sentence grammar portion of the test covers eight different areas of sentence grammar and is the largest part of the test. Tutors aggressively addressed the deficient areas early in the semester. This semester’s student popula-tion includes more motivated learners. utcomes Method e diagnos-d posttest g Lab. ng O ment om thest anWritin earni Assess Data frtic pretMy of L Table 1Writing Lab Student Student Learning Out-comes Students will increase their skills in the area of sentences. English 098 students will show a 50% improve-ment from the pretest to the posttest in sentence grammar for those who attend six or more ses-sions. English 099 students will show a 25% improve-ment from the pretest to the posttest in sentence grammar for those who attend six or more ses-sions. Assessment of Tutoring Laboratories 73 Action Plan Students will improve usage by: Working on individual My study plans in riting Lab W. Writing sentences and paragraphs. Using authentic stu-dent writing from class assignments. Receiving tutor-direct-ed feedback. Direct instruction in both areas of usage from tutors. Modified Student Learning Outcomes Students will increase their skills in the area of usage and style. English 098 students will show a 30% improve-ment from the pretest to the posttest in usage and style for those who attend six or more sessions. English 099 students will show a 25% improve-ment from the pretest to the posttest in usage and style for those who attend six or more sessions. Findings For students who attended six or more sessions: English 098 students showed a 21% improvement in usage and style. English 099 students showed an 18% improvement in usage and style. Although many students did improve in usage, our benchmark was not met; however, this part of the test is the smallest part with only 15 questions which address only two areas – parallel-ism (which was taught in lab) and easily confused words (which was not taught in lab). s e m d utco Metho e etest My of Learning O Assessment Data from th nostic prdiagand posttest Writing Lab. ContinuedTable 1 Writing Lab Student Student Learning Out-comes Students will increase their skills in the area of usage. English 098 students will show a 45% improve-ment from the pretest to the posttest in usage and style for those who attend six or more ses-sions. English 099 students will show a 30% improve-ment from the pretest to the posttest in usage and style for those who attend six or more ses-sions. 74 Journal of College Reading and Learning, 42(2), Spring 2012 Action Plan Students will improve punctuation and me-chanics by: Working on individual My study plans in riting Lab W. Writing sentences and paragraphs. Using authentic stu-dent writing from their classes. Receiving tutor-direct-ed feedback. Direct instruction on punctuation and me-chanics from tutors. Modified Student Learning Outcomes Students who attend six or more sessions will increase their skills in the area of punctuation and mechanics. English 098 students will show a 10% improvement from the pretest to the posttest. English 099 students will show a 10% improvement from the pretest to the posttest. or % nd % nd Findings For students who attended six more sessions: English 098 students show a 6improvement in punctuation amechanics. English 099 students show a 7improvement in punctuation amechanics. ng Outcomes ment Method om the diagnos-est and posttest Writing Lab. earni Assess Data frtic pretMy of L ContinuedTable 1 Writing Lab Student Student Learning Out-comes Students will increase their skills in the area of punctuation and mechanics. English 098 students will show a 15% improve-ment from the pretest to the posttest in punctua-tion and mechanics for those who attend six or more sessions. English 099 students who attend six or more sessions will show a 15% improvement. Assessment of Tutoring Laboratories 75 Action Plan Students will improve basic grammar by: Working on individual My study plans in riting Lab W. Writing sentences and paragraphs. Using authentic stu-dent writing from class assignments. Receiving tutor-direct-ed feedback. Direct instruction from tutors on verb tense and use of present tense when writing essays. Modified Student Learning Outcomes Students will increase their skills in the area of basic grammar. English 098 students show a 20% improve-ment in basic grammar. English 099 students show a 25% improve-ment in basic grammar. d six or w a 16% mmar. w a 21% mmar. ho attende dents shon basic gra dents shon basic gra Findings For students wmore sessions: English 098 stuimprovement i English 099 stuimprovement i ng Outcomes ment Method om the diagnos-est and posttest Writing Lab. earni Assess Data frtic pretMy of L ContinuedTable 1 Writing Lab Student Student Learning Out-comes Students will increase their skills in the area of grammar. English 098 students will show a 15% improve-ment from the pretest to the posttest in basic grammar for those who attend six or more ses-sions. English 099 students will show a 20% improve-ment from the pretest to the posttest in basic grammar for those who attend six or more ses-sions. 76 Journal of College Reading and Learning, 42(2), Spring 2012 SWOT Analysis and Action Plan In addition to student learning outcomes, the staff conducted a SWOT analysis for each program and developed an action plan based on published evidence of best practices. This assessment helped the lab team develop prescriptions for improving services and increasing the effectiveness of those services. For a comprehensive review of the use of a SWOT analysis and action plan in a learning assistance center, see Fullmer (2009). Logic Models Logic models—depictions of how a program or organization works— facilitate reflection, planning, and communication among an organiza- tion’s employees and stakeholders, leading to a high-quality evaluation of the program. The staff of the LRC developed a logic model for the LRC using an outcomes approach, developing the short-, medium-, and long-term outcomes and impact first, and then proceeded to complete the model working backwards. After the outcomes and impact were developed, the outputs (activities and participants) were listed, and then the inputs (resources) were determined. In addition, each tutor- ing laboratory team developed a logic model for their lab. Professional tutors developed a logic model for their respective labs in order to dis- cover any gaps in services and to gain a comprehensive understanding of the program for planning purposes. This information was used to complete a self-evaluation, analyze services, and generate strategies for improvements in the effectiveness of services. In this manner, signifi- cant informed decisions were made concerning goals, objectives, and student learning outcomes. The creation of logic models for the LRC and the tutoring labs revealed gaps in services and programs, and focused attention on the relationship between actions and results. With this information, funds were able to be efficiently allocated and resources effectively apportioned. The LRC professional tutors reported that the realization of the long-term outcomes and impact led to increased self-esteem and an awareness of the significance of their contribution to the university. In addition, the development of the logic models built a shared understanding of the LRC and supported cohesiveness of the staff through the engagement of all in the design and formation of LRC’s logic model. The LRC logic model is depicted in Figure 1.