Using the Canadian Language Benchmarks (CLB) to Benchmark College Programs/ Courses and Language Proficiency Tests Lucy Epp and Mary Stawychny In thisarticletheauthorsdescribeaprocessdevelopedbythe LanguageTraining Centre1 (LTC) at Red River College (RRC) to use the Canadian Language Benchmarks (CLB) in analyzing: (a) the language levels used in programs and coursesatRRCinordertoidentifyappropriateentry-levellanguageproficiency, and(b) thelevelsthatsecondlanguage(L2)studentsneedinordertomeetcollege oruniversityentrancerequirementsbasedontestsoflanguageproficiency. Sofar 19programsandfour courseshavebeen benchmarkedatRRC. The benchmark ingoftheprogramsandcoursesinvolvedgatheringdatafrom varioussourcesat the College and analyzing them by means ofCLB descriptors. In addition, a process was developedfor using the CLBA and CLB descriptors to benchmark tests: the Canadian TestofEnglishfor ScholarsandTrainees (CanTEST, 1991) and the Test ofEnglish as a Foreign Language (TOEFL). In conclusion, the authorssummarizesomebenefitsrealizedbythebenchmarkingprocess.Theyalso addresstheneedtocontinuetoevaluatetheresultsandadviseprudentuseofthe resultsoftheseprojects. Dans cet article, les auteurs decrivent un processus developpe par Ie centre de formation linguistique(LanguageTrainingCentre- LTC)auRedRiverCollege (RRC) permettantd'employerlesNiveauxdecompetencelinguistiquecanadiens (NCLC) dans l'analyse des (a) echelons linguistiques employes dans les pro grammes etcoursau RRCdans Ie butd'identifier Ie niveau linguistiqueappro prieauxdebutantset(b) lesniveauxquelesetudiantsenlanguesecondedoivent avoiratteintspoursatisfairelesconditionsrequisesparlesevaluationsdecompe tence linguistique menant it l'admission au college ou it 1'universite. Jusqu'it maintenant, dix-neufprogrammes etquatre coursau RRConteteainsietalon nes. L'etalonnage a implique la cueillette de donnees provenant de diverses sources au College, suivie de leur analyse en employant les descripteurs des NCLC. On a egalement developpe un processus permettant Ie recours it ces descripteurs dans l'etalonnage d'evaluations: Ie Canadian Test ofEnglishfor Scholars and Trainees (CanTEST) et Ie Test ofEnglish as aForeign Language (TOEFL). Unresumedesavantagesdecoulantdel'etalonnagesertdeconclusion. 32 LUCYEPPandMARYSTAWYCHNY a Lesauteurs signalentqu'ilsdoivent continuer analyserles resultats etrecom mandentun usageprudentdes resultatsdeleurs projets. TheCanadianLanguageBenchmarks(CLB)is atask-baseddescriptivescaleoflanguageproficiencyinEnglishas a secondlanguage,expressedintermsofcommunicativecompetenceas 12benchmarks (referencepoints).TheCLBdescribeslearners'place mentonalanguagecontinuumfrom Benchmark1toBenchmark12for threeskillareas(listening/speaking,reading,andwriting). (Citizenship and ImmigrationCanada[CICl,1996,p.1)2 The three skill areas addressed by the 1996 CLB document are each sub divided intofoursubskillsasfollows: 1. Listening/Speaking a. followingandgivinginstructions b. socialinterchange c. exchanginginformation d. suasion(gettingthingsdone) 2. Reading a. readinginstructions b. readingformatted texts c. readingunformattedtexts d. readinginformationaltexts-analysisandevaluation 3. Writing a. information-coping,reproducing b. formattedtext-fillingout/constructing c. unformattedtext-describing,conveyingmessages d. expressingideas-analysisandevaluation,persuasion For each subskill, descriptions ofcompetences, performance conditions, and sample tasks are provided. In addition, the 12 CLB levels are divided intothreeproficiencystages.StageI(Benchmarks1-4)representsbasicprofi ciency, Stage II (Benchmarks 5-8) represents intermediate proficiency, and StageIII(Benchmarks9-12)representsadvancedproficiency. This article summarizesthe processdeveloped touse theCLBinanalyz ing: (a) the language used in programs and courses at Red River College (RRC)inordertoidentifyappropriateentry-levellanguageproficiency;and (b)thelevelssecond-language(L2)speakersneedinordertomeetcollegeor universityentrancerequirementsbasedontestsoflanguageproficiency. Specifically, from December 1997to November2000 the RRC Language TrainingCentre (LTC) undertook the following projectsrelated to the CLB: (a) 19 RRC programs and four RRC courseswere benchmarked in terms of appropriate entry-level language proficiency, and (b) the Canadian Test of English for Scholars and Trainees (CanTEST) and the Test of English as a TESLCANADAJOURNAUREVUETESLDUCANADA 33 VOL.18,NO.2,SPRING2001 Foreign Language (TOEFL) test were benchmarked in terms of the equivalentCLBlevelsneeded tomeetthetestlevelsidentifiedforentryinto collegeoruniversityprograms. Becausenomodelfortheprocesshadbeendeveloped,alargepartofthe initialprojectwasthe developmentoftheprocess.Reportssummarizingthe projects,moredetailedinformation,benchmarkingratinginstrumentforms, andquestionnairesareavailablefromtheLTCatRRC(seeaddressinnote1). Inorderthattheresultsofthisprojectnotbeusedtosetupunfairbarriers for L2 students who apply for programs, it is important that anyone using theseresults keep thefollowingconsiderationsinmind. First,theCLBAhas been developed as the official assessment tool for the CLB. The highest possible score on this assessmentis CLB Level 8. Therefore, whenstudents score at Level 8, they could actually be anywhere in the CLB Levels 8-12 range, because they have reached the ceiling of the test. Furthermore, the CLBA was never meant for high-stakes testing. For example, it is inap propriateasanadmissions toolforentranceintopostsecondaryinstitutions. Therefore, it is not recommended that CLBA scores be used to determine languagereadinessforpostsecondarystudies. Second, language skills are only one predictor of student success. For example, factors such as trainingorexperience in the field, level ofmotiva tion,andstudyskillsalsocontributetosuccess.Therefore,CLBlevelsshould notbethe onlyfactor considered whendeterminingstudents' readiness for programsandcourses. Athird considerationisthefactthatstudentswithlowerlanguage profi ciencymaysucceedinprograms,butmaylaterhavedifficultyaccessingthe job market. In addition, they may be ata disadvantage in the workplace in terms of options and promotions. For this reason it would be to their ad vantage tobeatappropriatelanguageproficiencylevelsas identifiedbythe projectbeforetheyentercollegeoruniversityprograms. Project One: Benchmarking ofPrograms/Courses3 (December 1997-April1998/February1999-June1999/January 2000-November2000) Aprocesswasdeveloped tobenchmarkprogramsorcourses atRRC. Inthis processthe language used inprograms and courses was analyzedbased on the CLB in order to identify appropriate entry-level language proficiency. Theprocessincludedselectingprogramsorcoursestobenchmark,develop inga benchmarkingratinginstrument, and collectingdatafrom instructors' feedback,classroomobservation,students'feedback, andtextbookanalysis. 34 LUCYEPPandMARYSTAWYCHNY Program Selection Firstofall,programsthathadasignificantnumberofL2studentsweregiven priority. Also, the LTC Advisory Committee was consulted for advice regardingprogramsthatseemedmostappropriateforthisprocess. Next, meetings were arranged with program chairs and coordinators of the selected programs. In these interviews specific questions were asked regardingthesuccessrateofL2studentsandthetypesofproblemsthatwere encountered by both staff and students. The process ofdata collectionwas discussedaswell. In addition, interviews were conducted with various RRC personnel in the areas of L2 support, tutorial, program evaluation, assessment, admis sions, and international education. Outside RRC there was input from the Manitoba Aerospace HumanResources CoordinatingCommittee and from persons who were benchmarking the Canadian Adult Achievement Test (CAAT). The informationgatheredhelped to develop the appropriate steps forbenchmarkingandtoselectprogramstobebenchmarked. Itwas decided that the first termof programs would be the focus ofthe project. There was generalconsensus thatL2students whowere successful in the first term would probablybe successfulin the rest ofthe program. It was also felt that as students participated in programs, they gained both confidence and additional language skills. The programs or courses benchmarkedareshowninTable1. DevelopmentofaBenchmarkingRating Instrument Essential to the process was the development of a benchmarking rating instrument. A form was developed for each subskill at the Stage II and III levels oftheCLB. StageItasks were notidentified, as theywereconsidered toobasic tobe relevant in a college level program. In the instrument itself, tasks representing each subskill at each CLB level (Stages II and III) were identified, and examples ofeachtask were given. Forexample, for reading, under the subskillreading instructions, a listoftasks-forexample, under stand simple instructions (CLB Level 6), understand and follow written instructionsonschoolassignments(CLBLevel7)-werechosen(seeAppen dix A). As much as possible samples were taken from the CLB document itself.Aneffortwasmadetomakethesetasksandexamplesunderstandable andrelevanttoinstructors.Inaddition,foreachtasktherewerethreechoices tobemade: 1. Pre-Program.Was thisataskthatstudentswouldhavetomasterbefore entrance? 2. Program. Was thisataskthatstudentswouldbetaughtaspartofthe program/course? 3. Post-Program. Was thisataskthatstudentswouldneed toperformin theworkplace? TESLCANADAJOURNAUREVUETESLDUCANADA 35 VOL.18,NO.2,SPRING2001 Table 1 Programs AutomotiveTechnician HealthCareAide BusinessAdministration HeavyDutyEquipmentMechanic CivilTechnology HotelandRestaurantAdministration ComputerAccountingTechnician ManufacturingTechnician ComputerAidedDrafting Nursing(JointBaccalaureate) ComputerAnalystProgrammer/Information NursingRefresher SystemsTechnology PipingTrades CulinaryArts Welding DevelopmentalServicesWorker WoodProcessing Electrical Aerospace-RelatedCourses AerospaceManufacturingPracticesand GeometricDimensioningandTolerancing Procedures DimensionalMetrology MetallurgyandHeatTreatment The benchmarking instrument was used in group meetings with program instructors (seeDatafrom Instructorsbelow).Anabbreviatedversionofthe instrumentwasdevelopedforusewithindividualinstructorsinbenchmark ingspecificcourses. Datafrom Instructors Onceprogramshadbeenidentified,chairsandcoordinatorswerecontacted to arrange a two-hour time slotfor a meetingwithall available instructors. The following informationwas requested from instructors: course descrip tions,samplesofavarietyofclassassignments,timetables,andphotocopies ofthreerandompagesofeachtextbookused. All thesedatawereorganized andanalyzedbytheresearchers. Inmeetingswithinstructorsthepurposeoftheprojectwasexplainedand someofthepossiblebenefitswerediscussed.Itwasnecessarytogiveabrief overview of the CLB levels, as instructors were generally not familiar with them. Instructors were then asked to brainstorm components of their pro grams. From this discussion there wasinputonappropriatelanguage tasks students needed to accomplish for successful participation in programs. Examplesincludedlisteningtolectures,takingnotes,participatingingroup projects,andgivingpresentations.Next,researchersusedthebenchmarking instrument with instructors (see AppendiX A). The instrument was intro ducedby presentingone taskto the instructors and asking them as a group to categorize it as pre-program, program, or post-program. This discussion helped instructors to understand what was expected ofthem as they filled outthe forms. Because there was notsufficienttime toexamine all the CLB 36 LUCYEPPandMARYSTAWYCHNY subskillswith the instructors,oneortwosubskillsfrom eachskillareawere selectedtobethefocus for the ratingoftasks. Eachinstructorwasgivenone form for onesubskill area (e.g., Listeningand Speaking: following and giving instructions),andwasgivenaboutfiveminutestodecideindividuallyifeach task was pre-program, program, orpost-program. Eachofthese forms was about 1-1.5 pages in length (see Appendix A). After this, researchers led a discussion of instructors' responses, and consensus was reached on which tasks students needed to master before entering the program. This process was repeated with the instrument representing tasks using three or four more forms, each form representing one subskill. About one hour was al lowed for observations related to the benchmarking instrument. After the discussion, instructorswere asked tohand in theircompleted forms sothat researchers could compare their individual responses with the consensus reachedbythegroup. Thiswasfollowed byageneraldiscussionledbyresearchersaboutwhat challenges were faced byL2 students, how they werebeingaddressed, and howthey couldbebetteraddressed. Thepre-programlanguage assessment process was also discussed and suggestions were given. Responses were recordedforanalysis. Datafrom the Classroom Researchers arranged to sit in and take notes on the content of various classes.Inaddition,thelanguageskillsthatstudentsused inparticipatingin the class were noted (e.g., notetaking, interacting one on one with other students and/or instructors, interacting in groups, following instructions). Specific terminology as well as phrases and linguistic structures were also noted.Allthedatawerefiled foranalysis. Datafrom Students When researcherssatinonclasses, theyaskedinstructors for permissionto explaintheprojectbrieflytothestudentsandtoaskL2studentstovolunteer theirwillingness to be interviewed. Meetings were scheduled with eachL2 student.Researchersdevelopedaquestionnaireforinterviewswithstudents. The questions focused on the areas in which the student felt his or her language skills were adequate or inadequate. Students gave feedback on howproblemswerebeingaddressedorcouldbeaddressedmorehelpfully. Datafrom Textbooks Readabilitytestsweredoneonthreesampletextsfromrandompagesofeach textbook provided. Readabilityformulas included the FleschReadingEase, the Flesch-Kincaid Grade Level, the Coleman-Liau Grade Level, and the Bormuth Grade Level. These results were recorded as grade levels (e.g., TESLCANADAJOURNAUREVUETESLDUCANADA 37 VOL.18,NO.2,SPRING2001 grade10,grade11,grade12).Theresultsprovidedaglobalindicationofthe levelofdifficultyofreadingintheprogram. Using the Data Basedonallthedataavailable,appropriateentry-levellanguageproficiency for eachCLB subskill was identified using CLB descriptors. It was felt that this more specific information might be helpful in analyzing the language needs of each program. In addition, a general CLB level for each skill area wasassignedforeachprogram(seeAppendixB).Forcourses,onlyageneral CLBlevelwasidentifiedforeachskillarea. A projectreportfor each program and for the aerospace-related courses was prepared,whichincludedappropriateCLBlevelsfortheprogramsand courses, readability levels of textbooks, summaries of instructor-student feedback, andgeneralobservations. Finally,basedonall thedatagathered,areportofrecommendationsand suggestions for RRC was written. Itincludednotonlythe appropriate CLB levelsfor entry-levellanguage proficiencyforprograms orcourses,butalso some of the general observations that emerged from the research. Various implications were noted, a summary ofsuggestions for instructors was in cluded, and generalrecommendations were madefor RRC programs. Also, morespecificsuggestionsweremadeinthelightofthedataexaminedinthe project.Thesesuggestionsrelatedmainlytoassessmentoradmissionsproce dures and the implementation of more appropriate language profiCiency assessmentforprospectiveL2students. The benchmarking process itself was found to be helpful in identifying boththestrengthsandweaknessesofthepresentsysteminworkingwithL2 students. RRC intends to use the information gathered to inform and im proveassessmentoradmissionsproceduresastheyrelatetoL2students.The information gathered could also help in choosing appropriate content for preparation programs designed for L2 students who plan to enter RRC. In addition,theinformationhasthepotentialtoimprovethechancesofsuccess forL2studentsenrolledinprogramsorcoursesatRRC. ProjectTwo: Benchmarking of the CanTESTfTOEFL4 (February 1999-June1999) The goal of the second project was to determine the CLB levels that cor respondedwiththescoresrequiredontheCanTESTandtheTOEFLinterms of accessing college and university programs. Both the CanTEST and the TOEFL are tests of English language proficiency developed to determine applicants' adequacy of language proficiency for admission to academic programsincollegesanduniversities. The CanTESTmeasures four skill areas (reading, writing, listening, and speaking)andscoresarereportedinBandLevels,with5.0+beingthehighest 38 LUCYEPPandMARYSTAWYCHNY score. Based oninformationfrom the UniversityofManitoba, University of Winnipeg,andtheCanTESTProjectOfficeattheUniversityofOttawa,itwas found thatcolleges and universities generally recognize scores in the range ofBandLevel4.0to5.0assufficientforentranceintoprograms. TheTOEFL(paper-based)consistsofthreesections(listeningcomprehen sion, structure and written expression, and reading comprehension). Based on an environmental scan of 20 colleges and universities (undergraduate programs) across Canada, it was found that TOEFL entrance requirements rangefrom500to600.In16outof20cases,therequiredTOEFLscoresrange from550to580.Basedonthisinformation,itwasdecidedthataTOEFLscore of 550 to 580 would be considered the score generally required to meet college or university English-language proficiency entrance requirements. Only two universities reported minimum scores for Sections 1-3 of the TOEFL. In both cases the minimum score per section was 50. An average scoreof50persectionwouldrepresentanoverallscoreof500,wellbelowthe TOEFLscorerequiredbymostcollegesanduniversities.Becauseanaverage of55-58 oneachsection would result in the overall score of550-580, it was decided that 55-58 would represent the score generally required on each section of the TOEFL. In addition, a Test ofWritten English (TWE) may be administered.Thehighestscoreonthistestis6,andcollegesanduniversities thatrequire aTWEscoreusuallysetthe levelfor admissionataround5out of6. Comparison ofScores Comparisons between scores on the CLBA and the TOEFL and CanTEST were made. The CLBA was developed to assess ESL students based on communicationtasksthatrepresentCLBLevels1-8(StagesI[Levels1-4]and II [Levels5-8]) as describedintheCLBdocument,NortonPeirce, & Stewart, 1997). Instructors in the advanced ESL programs at RRC discussed the project with students and asked for volunteers to be tested. Forty-one stu dents volunteered tobetested,30forthe CanTEST/CLBAcomparison, and 11 for the TOEFL/CLBA comparison. The two assessments (CanTEST/ CLBA and TOEFL/CLBA) were administered to studentvolunteers within twoweeksofeachother. Trained assessors administered official CLBA assessments to all the stu dents.PermissiontoadministeranofficialCanTESTwasgiventoRRCbythe CanTESTProjectOffice at the University ofOttawa. All four sections ofthe test were administered following the test procedures required. The writing and speakingsections were scored by trained and experienced assessors of the CanTEST. The test results were sent to the University of Ottawa for confirmation. A paper-based practiceTOEFL test (TestForm TPK98) publishedby the EducationalTestingServicewasadministeredtostudentsandscoredbyLTC TESLCANADAJOURNAUREVUETESLDUCANADA 39 VOL.18,NO.2,SPRING2001 instructors following officialTOEFLrules based onguidelines published in the TestPreparationKit(EducationalTestingService,1998b). Observations Basedon Score Comparisons A number of factors affected the ability to generalize based on score com parisons: 1. A relativelysmallsampleofstudents (30ontheCLBA/CanTESTand 11 ontheCLBA/TOEFL)wastested. 2. TheCLBAisadifferenttestthantheTOEFLandCanTEST. TheCLBAusesgeneralsettlementlanguagetaskstoassesslanguage levels,whereastheTOEFLandCanTESTtestproficiencyusing college-oruniversity-levelacademictasks. TheCLBApresentlycombinesspeakingandlistening,whereasthe TOEFLandCanTESTdonot. TheCLBAassessesdiscoursewritingthatrelates tosettlementissues (e.g.,writingaletter,amemo).TheCanTESTrequiresamore academicwritingtask(essay-writing),whereasSection2ofthe TOEFLtestsstudents'abilitytocompleteorcorrectratheracademic sentencesinisolation,inadditiontotheessay-writing. 3. ThehighestpossiblescoreontheCLBAisCLBLevel8.Becausemost studentsscorednearthehighestlevelorsurpassedit,wecanpredict thatsomeofthemcouldhavescoredatCLBLevel9or10(ormore)if theyhadhadtheopportunity.Studentswhoseemedcapableofmoving beyondStageIIwereindicatedbyan8+. Inmostcasesstudentswho received8+ontheCLBscoredintherangeof500to550ontheTOEFL and3.0to4.5ontheCanTEST.Thatis,theywerenotgenerallyatthe ceilingofthetest. ThusitseemsthattheTOEFLandCanTESTgenerally testahigherleveloflanguageproficiencythandoestheCLBA. Based on these findings, it was felt that an in-depth analysis ofthe TOEFL andCanTESTusingCLBdescriptors wasnecessary.Itwasdecided thatthis analysis would be done on the TOEFL and the CanTEST that had been administeredtostudents. AnalysisofTOEFL and CanTEST Using CLB Descriptors Speaking/Listening The fact that speaking and listening were combined in the 1996CLB docu mentposed a problem,because these skillsareseparatedin the TOEFLand CanTEST. To address this problem, researchers separated listening and speaking descriptors at each relevant CLB level. Speaking and listening descriptors were then matched respectively with available descriptors of speaking and listening levels onthe CanTEST(The CanTESTOral Interview, 1998). In the case ofthe TOEFL test, speakingwas not tested, and listening descriptorsarenotpublished. 40 LUCYEPPandMARYSTAWYCHNY Each passage of the listening comprehension section of the TOEFL and theCanTESTwasanalyzedsentencebysentenceintermsoftheCLBstageit represented and the level of difficulty of the structures. In addition, each passage was matched with CLB descriptors in terms ofthe language skills necessary for achieving the required scores. Finally, each question was analyzed as it related to each passage. The type of question (e.g., specific information,prediction)wasdeterminedandmatchedwithCLBdescriptors. Based onthe analysis ofthe data gathered, eachpassagewas assigned a CLB level. Many CLB descriptors seemto assume that the learneris able to function equally well at a task both as a speaker and as a listener. For example,atCLBLevel8,thefollowingdescriptorforListeningandSpeaking states, "Understands and uses sufficient concrete and abstract vocabulary, idioms, and colloquial expressions to follow and relate stories of general popularinterest" (CIC, 1996,p. 50). Becauselearners usuallyacquire listen ingskillsearlierthanspeakingskills,itwasdecidedthatintermsoflistening skills,preferencewouldbegiventoalowerratherthanahigherCLBlevel. Using all the data that had been gathered and analyzed, researchers assigned a general Listening/Speaking CLB level needed to attain the re quired scores of55-58 on the TOEFL Listening and 4.0-4.5 on the CanTEST (seeAppendixC). Reading First, readability formulas were used to assess the levels ofdifficulty ofthe readingtextsontheTOEFLandCanTEST.TheFleschReadingEaseandthe Flesch-Kincaid Grade Level were used for both. For the CanTEST the Coleman-Liau Grade Level and the Bormuth Grade Level were also used. These results were recorded as grade levels (e.g., grade 10, grade 11, grade 12). The results provided a global indication ofthe level ofdifficulty ofthe readingtexts. EachTOEFLandCanTESTreadingpassage (unformatted)wasanalyzed sentencebysentencetodeterminetheCLBstagelevelitrepresentedandthe difficulty of the structures. In addition, all the reading passages were matched withCLB descriptors in terms ofthe languagedemands ofcollege or university entrance requirements. Then each question related to each passagewasanalyzedintermsofstructuralcomplexity.Thetypeofquestion wasdeterminedandmatchedwithCLBdescriptors. Based on the analysis of the data gathered, each reading passage was assigned a CLB level. Using all the data that had been gathered and analyzed, researchers assigned a general equivalent reading CLB level needed to attain the required scores of 55-58 on the TOEFL Reading and 4.0-4.5ontheCanTEST(seeAppendixC). TESLCANADAJOURNAUREVUETESLDUCANADA 41 VOL.18,NO.2,SPRING2001