McManusetal.BMCMedicine 2013, 11:244 http://www.biomedcentral.com/1741-7015/11/244 RESEARCH ARTICLE Open Access The UKCAT-12 study: educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of 12 UK medical schools I C McManus1,2*, Chris Dewberry3, Sandra Nicholson4 and Jonathan S Dowell5 Abstract Background: Most UK medical schools use aptitude tests during student selection, but large-scalestudies of predictive validity are rare.This study assesses theUnited Kingdom Clinical Aptitude Test (UKCAT), and itsfour sub-scales, along with measures of educationalattainment,individualand contextual socio-economic background factors, as predictors of performance inthe first year of medical school training. Methods: A prospective study of 4,811 students in 12 UKmedical schools taking the UKCAT from 2006 to 2008 as a part of themedical school application, for whom first year medical school examination results were availablein 2008 to 2010. Results: UKCAT scores and educational attainment measures(General Certificate of Education (GCE): A-levels, and so on; or Scottish Qualifications Authority (SQA): Scottish Highers, and so on)were significant predictorsof outcome. UKCAT predicted outcome better infemalestudents than male students, and better inmaturethan non-mature students.Incremental validity ofUKCAT taking educational attainment intoaccount was significant, butsmall. Medicalschoolperformance was also affected by sex (male students performing less well), ethnicity (non-White students performing less well), and a contextualmeasure of secondary schooling, students from secondary schools with greater average attainment atA-level (irrespective of public or private sector) performing less well. Multilevel modeling showed no differences between medical schools inpredictive ability ofthevarious measures. UKCAT sub-scales predicted similarly, except thatVerbal Reasoning correlated positively with performance onTheory examinations, butnegatively with Skills assessments. (Continuedonnextpage) *Correspondence:[email protected] 1UCLMedicalSchool,UniversityCollegeLondon,GowerStreet,London WC1E6BT,UK 2ResearchDepartmentofClinical,EducationalandHealthPsychology, DivisionofPsychologyandLanguageSciences,UniversityCollegeLondon, GowerStreet,LondonWC1E6BT,UK Fulllistofauthorinformationisavailableattheendofthearticle ©2013McManusetal.;licenseeBioMedCentralLtd.ThisisanopenaccessarticledistributedunderthetermsoftheCreative CommonsAttributionLicense(http://creativecommons.org/licenses/by/2.0),whichpermitsunrestricteduse,distribution,and reproductioninanymedium,providedtheoriginalworkisproperlycited. McManusetal.BMCMedicine 2013, 11:244 Page2of25 http://www.biomedcentral.com/1741-7015/11/244 (Continuedfrompreviouspage) Conclusions: This collaborative study in12 medical schools shows the power of large-scale studies ofmedical education for answering previously unanswerablebut important questions about medical student selection, education and training. UKCAT has predictivevalidityas a predictor of medical school outcome, particularly in mature applicants to medical school. UKCAT offers small but significant incremental validitywhich is operationally valuable where medical schoolsare making selection decisions based on incomplete measures ofeducational attainment. The study confirms the validity of using allthe existing measuresof educational attainment in full at thetime of selection decision-making. Contextual measures provide littleadditional predictive value, except that students from high attaining secondary schools perform less well, aneffectpreviously shown for UK universities ingeneral. Keywords: Medical student selection, Educational attainment, Aptitude tests, UKCAT, Socio-economic factors, Contextual measures Background at medical school than students of equal ability attend- Formanyyearstheprimarycriterionusedtoselectmed- ingselectiveor privatesector secondaryschools.Arising ical school applicants world-wide has been measures of from such concerns has also been a growing awareness educational attainment. In the UK, General Certificate of the paucity of large-scale, longitudinal studies which of Education (GCE) A-levels, for those educated in have examined performance in medical school in rela- England,WalesandNorthernIreland,andScottishQuali- tion to a wide range of measures collected during selec- ficationsAuthority(SQA)Highersformostofthoseed- tion. Without such studies it is difficult to assess the ucated in Scotland, have been the norm. A-levels have validity and defensibility of the processes currently used been shown to be valid predictors of outcome, both toselectmedicalstudents. duringthemedicalcourse[1]andlaterinmedicalcareers In response to these challenges, most UK medical [2], and for university education in general [3], and schoolsinrecent years have used aptitude testsasa sup- more generally in Europe, secondary school grades are plementary selection technique. An aptitude test usually predictive of university performance [4]. Educational at- consists of a series of multiple choice sub-tests. Sub- tainment is also important in medical student selection tests are developed to assess specific aptitudes thought in many countries, such as Australia, Denmark, Iran, to be relevant for performance at medical school (for The Netherlands [5], New Zealand and Thailand [6]. example, numerical ability and verbal ability). Unlike However, in recent years the continued reliance on such secondary school examinations, which measure attain- attainment measures as the sole or principal basis for ment in relation to a particular discipline (for example, medical student selection has been questioned for three chemistry), aptitude tests are specifically designed to reasons. First, A-levels and Scottish Highers, which are measure intellectualandbehavioral capacity, andthepo- typicallytakeninthefinalyearofsecondaryschool,have tential to perform well in a particular role. Furthermore, over the past two decades shown continual increases in aptitude tests offer considerable operational advantages: grades attained (so-called,‘grade inflation’). The result is they can be taken in standardized settings by anyone, that a large proportion of UK applicants to medical whatever their cultural or educational background, at schools now achieve the highest grades (a ceiling effect), geographical locations all over the world; they can assess so that discriminating between them for the purpose of people against items for which the difficulty level has selection is increasingly problematic. Second, selection been pre-established; they can be completed in a short on the basis of A-level and Scottish Higher grades may time (typically less than three hours); and candidates’ be construed as unfair, because students who have performance on the test can be computed immediately. attended selective schools, including independent sec- Although aptitude tests are used in medical selection ondary schools charging high fees, tend to obtain higher in many countries including Australia [7], Chile [8], grades than others, although this in part may reflect dif- Germany [9,10], Pakistan [11], Italy [12], Mexico [13], ferentintakeprofiles.Third,thereareconcernsthataca- Switzerland [14], the United States [15,16] and the UK demic assessment alone may not necessarily select the [17,18],researchontheextenttowhichtheysuccessfully candidates that possess the behavioral or non-cognitive predict medical performance is patchy. Given the high- attributes thought desirable in medical students and stakes nature of medical selection it is clearly important doctors. As a consequence, able but economically or so- to establish the predictive validity of aptitude tests, and cially disadvantaged children attending comprehensive the extent to which they add value to more traditional secondary schools might be less likely to obtain a place approaches to selection, such as previous educational McManusetal.BMCMedicine 2013, 11:244 Page3of25 http://www.biomedcentral.com/1741-7015/11/244 attainment, using datasets large enough to provide sub- current and accepted methods of selection, which at stantial statistical power. Here we report the results of present for medicine are primarily achievement tests the first large-scale analysis of the predictive and incre- taken in secondary education, coupled in many cases mental validity of UK Clinical Aptitude Test (UKCAT), with interviews. That is particularly important as tests the aptitude test used in the selection of medical stu- such as the American Medical College Admission Test dentsinmostUKmedicalschools. (MCAT),whichhave both aptitude andattainment com- Aptitude tests can be used as a stand-alone selection ponents, typically find that most of the prediction is due device or, more typically, to supplement the existing in- to the attainment component, rather than the aptitude formation on which selection decisions are made, such component[32]. as a candidate’s secondary school attainment, Univer- Given the widespread adoption of UKCAT for medical sities and Colleges Admissions Service (UCAS) personal schoolselectionsinceitsintroductionin2006[33],amore statements and medical school interview performance. comprehensiveexaminationoftheextenttowhichthetest Because these tests can be specifically designed to differ- cansuccessfullypredictperformanceandwidenparticipa- entiate between highly able candidates, and to do so in tion is required. In this article we address this issue by relation to the particular characteristics required in the examining the relationships among multiple predictors of medical profession, they can potentially discriminate ap- medical school performance (including UKCAT, A levels propriately between those with equally good attainment andScottishHighers,andabroadrangeofcontextualand at A Level and Scottish Highers. In addition, by measur- socio-cultural measures, including selective schooling), in ing the extent to which candidates’ possess aptitudes relation to the first year medical school performance of relevant tothe medical profession, rather than theirlevel 4,811studentsstudyingat12EnglishandScottishmedical of school-related educational attainment, aptitude tests schoolsinthree cohortswho tooktheUKCATin2006to mayhelp towidenparticipation inmedicine[19]. 2008, entered medical school in 2007 to 2009, and com- The two main aptitude tests currently in operation in pletedtheirfirstyearin2008to2010. the United Kingdom are the UK Clinical Aptitude Test Although the primary impetus for the present study (UKCAT) [20], which is the principle interest of the was to evaluate UKCAT in the context of medical stu- present study, and the Biomedical Admissions Test dent selection, the UKCAT-12 study can also be used to (BMAT) [21,22]. For admission in 2013, UKCAT is be- address a wider set of important issues. As yet there has ing used by 26 UK medical schools, and BMAT by 4 UK beennolarge-scale,prospectivestudyofmedical student medical schools. If aptitude tests are to be of added performance drawing on a wide range of measures value in addressing the problem of ceiling effects in edu- which might predict that performance (including de- cational qualifications, they must predict the future per- tailed socio-economic background measures) across a formance of medical school candidates over and above substantial sample of medical schools. UKCAT-12 pro- that provided by A-levels and Scottish Highers [23]. In vides exactly that, giving not only a platform from which addition,iftheyaretowidenparticipationinthemedical to ask many questions about the nature of medical stu- profession, it is desirable that scores on UKCAT and dent selection and education and the assessment of the BMAT should be less strongly associated with selective effects of a large number of different background mea- secondaryschoolingthanAlevels orScottish Highers. sures, but also allowing a determination of the extent to At present, both UKCAT and BMAT are somewhat which different measures might have different predictive controversial [23,24], mainly because of concerns about values in different medical schools. Although, therefore, how well they predict performance at medical school. a prime interest of the present study is to evaluate However, there is also concern over the cost deterring UKCAT, it also represents the first, long-term, large- poorer applicants and the effects of coaching [25], which scale study of medical student training in the UK. Im- in general can have an effect size of about .26 [26]. To portant features of the present analysis are that the date there have been four studies ofthe predictive ability sampleislarge(nearly5,000students),it isdiverseand of UKCAT [17,27-29], all of which are moderately small representative of a range of medical schools (12 medical (Ns = 292, 307, 204 and 146), and have widely varying schools taking part), it is extended over time (the data conclusions, from a study suggesting the test provides beingcollectedacrossseveralyears),andthereisa‘hard’ nosignificantprediction[30]toclaimsofsignificantpre- outcomemeasureintheformofmedicalschoolexamin- dictive ability [17]. Additionally, one study [31] found no ation results on a continuous scale. That means the relationship between UKCAT scores and scores on ad- current study has high statistical power, and also makes missions interviews. An important consideration in de- itpossibletocomparemedicalschoolsinordertoassess termining the validity of any aptitude test is that as well the degree to which the conclusions can be generalized as demonstrating predictive validity in its own right, it acrossmedicalschools.Thus,theaccumulatingdatabase should also show incremental validity when used with associated with the UKCAT provides an important McManusetal.BMCMedicine 2013, 11:244 Page4of25 http://www.biomedcentral.com/1741-7015/11/244 opportunity not only to assess the effectiveness of the entered medical school in 2007, 2008 or 2009, and for UKCAT, but also to assess the influence of a far broader whom outcome measures were available at the end of range of issues concerning how educational, demo- their first academic year. For those cohorts, UKCAT graphic and social factors influence medical school out- was used in selection by 23, 25 and 26 medical schools. come, including those assessed with the ‘contextual The secondary datasets contained data on a range of measures’ which will soon be available for routine use other measures, including prior educational achieve- duringselection. ment, socioeconomic background, and so on. Many of the secondary measures are missing, either for structural Aimsoftheanalysis reasons (for example, some socio-economic measures The present analysis takes into account the aims which are only available for England; A-levels were not avail- UKCATset for itself [20,34], as well as various previous able for mature entrants; and so on) and others were studies of aptitude tests (and the criticisms of those also sporadically missing, probably mostly at random. studies).Ittherefore looksat: Some secondary measures do not describe individual students, but instead are contextual variables, describ- ◦Thepredictive validity ofA-levelsand Scottish ing not the students themselves, but features of the edu- Highersforperformance inthe firstyearofmedical cational and socioeconomic environment in which the schoolstudies. students lived prior to joining medical school (for ex- ◦Thepredictive validity ofUKCATforperformance in ample,aggregatedmeasuresoftheattainmentofthesec- thefirstyearofmedical schoolstudies. ondary school attended, socio-economic measures of the ◦Theincrementalvalidity ofUKCAToverand above local community where the student lived, and so on). existing measuresofeducational attainment,both Contextual measures need to be treated with care, but General CertificatesofSecondaryEducation(GCSEs)/ have been included not only for their sociological inter- AS-levels/A-levelsand ScottishHighers/Advanced est, but also because similar measures are now provided Highers. routinely by UCAS, and have been shown to relate to ◦Thespecific predictiveability, with andwithout achievement at the BMAT aptitude test [36]. Table 1 taking educationalattainment intoaccount,ofthefour summarizes the many measures which were in the ana- subscalesofUKCAT. lysis, and more detailed information can also be found ◦An assessment ofwhether ‘Theory ‘and‘Skills’ in the UKCAT-12 Technical Report [34]. The measures measuresatmedical schoolare predicted differentlyby can bedividedinto sixbroadcategories. educational attainmentand UKCATaptitude measures. Variables are indicated by their SPSS variable names ◦An assessment ofwhetherthe predictive validity of (inbold)toreduce ambiguity. anyofthemeasures isdifferent inthe 12medical schools thathavetakenpart inthestudy. 1. MEDICAL SCHOOLOUTCOMEDATA.Medical ◦Theroleofdemographicand socio-economicfactors schoolsprovidedinformation onoveralloutcome on inmoderating anyofthefindings. afour-point scale(passed allexams atfirstattempt; passed afterre-sitting exams;repeatingthefirstyear; Itshouldbenotedthatthepresentstudyisrestrictedto and leavingthe course),which we called medical school entrants and, therefore, it cannot look OutcomeFirstYear4pt.Average percentagemarks more generally at how social and other factors relate to onassessmentswerealsoavailableformost students educational attainment and UKCAT performance in the (OverallMark),and formanystudentsseparate entire set of medical school applicants (rather than en- markswerealsoavailablefor ‘theory’and ‘skills’ trants). The analysis also considers only simple predictor- assessments(TheoryMarkandSkillsMark;see outcome correlations, and makes no attempt to calculate Table1).Theoverall,theory and skillsmarkswere construct validity, taking into account the unreliability of allbasedonmarksattained atthefirstattempt. predictor and outcome measures, restriction of range due 2. PRIOREDUCATIONAL ATTAINMENT,AND SO to selection, and the right-censorship of predictor vari- ON.Informationonprior educationalattainment ables such as A-level scores. All of that is considered in a wasprovidedbyUCAS,consistingofScottish separate paper, which carries out a meta-regression of HigherandAdvancedHigherresultsforstudents construct validity not only in the UKCAT-12 study, but from Scotland (collectively SQAqualifications),and alsoinfiveothercohortstudies[35]. A-level,AS-leveland GCSE resultsforother students(collectively GCEqualifications). Methods Educational qualificationsarealways complexto The primary dataset for the UKCAT-12 study consists analyze,becausedifferentcandidatestake different of the 4,811 students in three separate cohorts, who examinationswith different structuresand grading McManusetal.BMCMedicine 2013, 11:244 Page5of25 http://www.biomedcentral.com/1741-7015/11/244 Table1Summaryofvariablesintheanalysis Category Variablesusedintheanalysis Notesandcomments Medicalschool Outcomefirstyear4pt Medicalschooldataonstudentperformanceintheirfirst outcomedata academicyearforthethreecohorts.Notallschoolsprovided Overallmark dataforallcohorts-11,11and9schoolsprovidingdataforthe Theorymark 2007to2009cohorts,for1,661,1,710and1,440students.Inthe samecohorts,UKCATwasusedforselectionby23,25and26 Skillsmark medicalschools.Theoverallnumberofstudentsfromthe12 schoolsvariedfrom87to945(median=335,mean=401,SD= 243).Medicalschoolswereaskedtoprovideseveralitemsof informationoneachstudent,althoughnotallschoolsprovided allinformation.DatawerecollectedbytheUKCATConsortium Office,andnotbytheresearchers.Measuresusedwereas follows:OutcomeFirstYear4pt:Outcomeofthefirstyearona four-pointscale(Passedallexamsatfirstattempt;passedafter re-sittingexams;repeatingthefirstyear;andleavingthecourse); OverallMark,TheoryMark,andSkillsMark:Averagedpercentage marksinmedicalschoolassessments.OverallMark,basedonall assessments,wasavailablefor4,510students,oneschool providingonlyOutcomeFirstYear4pt,andoccasionalstudents elsewherenothavingpercentagemarks;ineachcaseaproxy OverallMarkwascalculatedasanormalscore,usingSPSS’sRank Cases/NormalScorescommand.Separatemarkswerealso availablefor‘Theory’and‘Skills’assessments,thedefinitionof theoryandskillsbeinglefttomedicalschools.TheoryMarkand SkillsMarkwereavailablefor2,075and3,184students.Because percentagemarksarenotnecessarilycomparableacrossschools, OverallMark,TheoryMarkandSkillsMarkwerestandardizedto ameanofzeroandSDofonewithinmedicalschoolsandcohorts. Prioreducational Alevel_number_total WewilldescribetheanalysisofA-levelsinsomedetail.Other achievement examinationsshowminorvariationsfromtheanalysisofA-levels Alevel_number_totalAlevel_Totalbest whichwewillthendescribe. Alevel_TotalPoints A(Advanced)levels.ScoredasA=10,B=8,C=6,D=4,E=2, Else=0.A*gradesatAlevelwerenotawardedduringthestudy Alevels_Taken_1_or_more_Biology period.Measureswereonlycalculatedforstudentswiththreeor Alevels_Taken_1_or_more_Chemistry moreA-levels,othersbeingsetasmissing.Fourteenmeasures separatemeasureswereobtained,describedfurtherinthe Alevels_Taken_1_or_more_Physics TechnicalReport[34].GeneralStudieswasnotcountedinthe Alevels_Taken_1_or_more_Maths overalltotals,meansandsoon,butwasanalyzedseparately,as itsstatusisunclear.Themeasures(withtheirnamesinbold), Alevels_highest_Biology were:Alevel_number_total:Numberofnon-GeneralStudies A-levels,ofthe2,764entrants,41.8%had4ormore; Alevels_highest_Chemistry Alevel_Totalbest:SumofthethreehighestA-levelgrades,which Alevels_highest_Physics was73.0%ofstudents,wasthemaximumscoreof30(thatis, AAA),with21.3%scoring28(AAB),5.0%scoring26(ABB/AAC), Alevels_highest_Maths 0.6%scoring24(BBBorequivalent),andfourcandidatesscoring Alevels_Taken_1_or_more_NonScience 20,16,16and10.Alevel_TotalPoints:Totalpointsachievedbya studentforallofA-levels,whichforthosetakingthreeA-levels Alevels_Taken_1_or_more_GeneralStudies wasthesameasthepreviousmeasure; Alevels_highest_GeneralStusdie Alevels_Taken_1_or_more_Biology, Alevels_Taken_1_or_more_Chemistry, Inadditionequivalentvariablesforother Alevels_Taken_1_or_more_Physics,and qualificationsarenamedinsimilarwaysbut Alevels_Taken_1_or_more_Maths):aseriesof‘dummy withAlevel…replacedbyASlevel…, variables’,scoredas1ifthesubjecthadbeentakenand0ifit GCSE…,SQAhigher…,SQAhigherPlus… hadnot.95.7%,99.1%,24.8%and63.3%ofAlevelstudentshadA andSQAadvHigherPlus…. levelsinBiology,Chemistry,PhysicsandMath. EducationalAttainmentGCE Alevels_highest_Biology,Alevels_highest_Chemistry, Alevels_highest_Physics,andAlevels_highest_Maths:Highest EducationalAttainmentSQA gradeattainedbyastudentonBiology,Chemistry,Physicsand Mathsubjects;exceptforMath,studentsmostlyhadtakenonly EducationalAttainment oneexamineachcategory; zEducationalAttainmentGCE Alevels_Taken_1_or_more_NonSciencewasa1/0dummy zEducationalAttainmentSQA’ variableindicatingthatastudenthadA-level(s)otherthaninthe coresciencesofBiology,Chemistry,PhysicsorMath(orGeneral SQAorGCE Studies).Atotalof49.9%ofthestudentshadatleastonenon- scienceA-level;Alevels_Taken_1_or_more_GeneralStudies:A 1/0dummyvariableindicatingwhetherastudenthadtaken GeneralStudiesA-level;26.0%haddoneso;Alevels_highest_ McManusetal.BMCMedicine 2013, 11:244 Page6of25 http://www.biomedcentral.com/1741-7015/11/244 Table1Summaryofvariablesintheanalysis(Continued) GeneralStudies:ForstudentstakingGeneralStudies,thehighest gradeattained,46.9%havinganAgrade; AS(AdvancedSubsidiary)levels.Variablesaresimilartothosefor A-levelsexceptthattheyarenamedASlevel…ratherthan ALevel…ScoredasforAlevels(A=10,B=8,C=6,D=4,E=2, Else=0).Measuresare similarexceptthatstudentshadtohavetakenatleastfourAS-levels, andtotalswereforthebestfourAS-levelsachieved.Forreasons whicharenotclear,fewerstudentshad4+AS-levels(n=1,877) thanhad3+A-levels(n=2,764).AS-levelgradesshowedmore variabilitythanA-levels,only56.3%ofstudentsscoringamaximum 40pointsfortheirbestgrades,comparedwith73.0%ofstudents gaining30pointsfromtheirbestthreeA-levels. GCSE(GeneralCertificateofEducation).Variablesarebroadlysimilar tothoseforA-levelsexceptthattheyarenamedGCSE…rather thanALevel…GCSEresultswereonlyavailableforthe2009entry cohort.SinglesubjectswerescoredasA*=6,A=5,B=4,C=3, D=2,E=1,else=0anddoubleScienceandothersubjectswere scoredasA*A*=12,A*A=11,andsoon,andcountedastwo GCSEstaken.VeryfewstudentshadeightorfewerGCSEs,and thereforeoverallscoreswerethereforebasedontheninebest grades.GCSEscoreswereavailablefor930students,andwere morevariablethanA-levelsorAS-levels,only16.6%ofstudents havingthemaximumof54points(equivalentto9A*GCSEs). Scoreswerecalculatedforthefourindividualcoresciences,and scorewerealsocalculatedforCombinedScience(takenby32.8% ofstudents).GCSE_Number_NonScience_Exams:Becauseall studentshadtakenseveralnon-sciencesubjects,thisvariablewas thenumberofnon-sciencesubjectstaken. ScottishHighers.MeasuresarebroadlysimilartothoseforA-levels, exceptthatnamesbeginSQAhigher…Gradeswerescoredas A=10,B=8,C=6andD=4.Studentswereonlyincludedwho hadfiveormoregradesatHighers,thefivehighestbeingsummed. OtherdifferencesfromA-levelsarethatthereisnoGeneralStudies component,andalmostallstudentswilltakeanon-scienceHigher. ResultsforScottishHigherswereavailablefor769students,72.4% gainingamaximumscoreof50pointsbasedonbestfivegrades. ‘ScottishHighersPlus’.Thisisaconstructionofourown,reflecting thefactthatalthoughScottishHighersarescoredbyUCASand bymostScottishuniversitiesasA,B,CandD,theUCASgradesare actuallyA1,A2,B3,B4,C5,C6andD7.Theseresults,withtwo bandsateachgrade,aretreatedasmeaningfulbymanyEnglish universities(althoughnot,itwouldseem,Scottishuniversities),and thereforewealsoscoredHighersonabasisofA1=10,A2=9, B3=8,B4=7,C5=6,C6=5andD7=4.Wehavenamedthisas ‘ScottishHighersPlus’,andvariablenamesbeginSQAhigherPlus…. Theseresultshaveawiderrangeofscores,only19.9%ofstudents gainingthemaximum50points. ScottishAdvancedHighers.VariablenamesbeginSQAadvHigherPlus…. ManyScottishuniversitiesseemnottorequireAdvancedHighers, anargumentagainsttheirusebeingthatonlyselectiveschools havetheresourcesorprovidethepossibilityofstudyingAdvanced Highers,andhencethereareconcernsaboutwideningaccess.We note,however,thatinthisgroupofstudents,of478applyingfrom thestatesector,93.1%hadoneormoreAdvancedHighers, comparedwith81.8%of237nonstatesectorentrants.Overall,573 studentsinthepresentsurveyhadatleasttwoAdvancedHighers (thatis,74.5%ofthe769studentswithHighers),andafurther108 hadoneAdvancedHigher.We,therefore,alsoassessedthepredictive valueofAdvancedHighers.Scoringwasasfor“ScottishHighersPlus” (thatis,A1=10,A2=9,B3=8,B4=7,C5=6,C6=5andD7=4), withscorescalculatedforindividualcoresciencesubjects,along withhighestoverallscoreattained.22%ofthe694studentswithat leastoneAdvancedHigherhadamaximumof10pointsontheir bestAdvancedHigher,and22.6%had7orfewerpoints. McManusetal.BMCMedicine 2013, 11:244 Page7of25 http://www.biomedcentral.com/1741-7015/11/244 Table1Summaryofvariablesintheanalysis(Continued) Overallmeasuresofeducationalattainment.Asdescribedinthetext, anoverallmeasureofeducationalattainmentwascalculatedfor eachstudent,EducationalAttainmentGCEor EducationalAttainmentSQAforGCEandSQAassessments.These variableswerebasedonasetofeightortenmeasuresrespectively, withmissingvaluesreplacedbytheEMalgorithm,andthenthe firstprinciplecomponentextracted.Asinglevariable, EducationalAttainmentwascreatedwhichwasthezscoreof eitherEducationalAttainmentGCEorEducationalAttainmentSQA, whicheverwasnotmissing.Becausethepresentanalysisisinterested inmeasureswithinmedicalschools,EducationalAttainmentGCE andEducationalAttainmentSQAwerealsostandardizedtohavea meanofzeroandSDofonewithineachmedicalschoolcohort,to producethevariableszEducationalAttainmentGCEand zEducationalAttainmentSQA.Wealsousedadummyvariable, SQAorGCE,toindicatewhetherentrantshadtakenScottishorother qualifications.NotethatinthepaperonConstructValidity[35]the unstandardizedmeasureswereused,inorderthatinformationon applicantsaswellasentrantscouldbeonacommonscale. UKCATmeasures zUKCATtotal DatawereprovidedbytheUKCATconsortium,withsomeadditional measurescalculatedbyHICinDundee.Theoverallmeasureof zUKCATabstractReasoning performancewasthetotalscore,UKCATtotal,andtherewerealso zUKCATdecisionAnalysis scoresonthefoursubscalesUKCATabstractReasoning, UKCATdecisionAnalysis,UKCATquantitativeReasoning,and zUKCATquantitativeReasoning UKCATverbalReasoning.Eachofthemeasureswasalso zUKCATverbalReasoning standardisedasaz-scorewithinmedicalschoolsandcohorts,to givezUKCATtotal,withthefoursubscalesbeing UKCATskipped zUKCATabstractReasoning,zUKCATdecisionAnalysis, UKCATdayOfTakingPctileRank zUKCATquantitativeReasoning,andzUKCATverbalReasoning, TherewasalsoinformationonthedateoftakingUKCAT,the UKCATexamSeriesCode variableUKCATdayOfTakingPctileRankgivingrelativedateof takingthetestwithincohorts,lowscoresindicatingearlytakersof UKCATcandPerSchool thetest.Notallcandidatesansweredallquestions,inmostcases probablybecausetheyranoutoftime,andasaresultonaverage hadlowerscoresthaniftheyhadguessedatitems,themeasure UKCATskippedgivingtheoverallnumberofskippeditems,which hadamedianof4,only25.9%ofcandidatesansweringallitems. Somecandidateswereallowedextratimebecauseofspecialneeds, whichisindicatedbythevariableUKCATexamSeriesCode;on averagethesecandidateshadhigheroverallscoresthanother candidates. IntheiranalysesofBMAT[36],Emeryetal.reportedthatcandidates fromschoolswithmoreextensiveexperienceofthetestperformed somewhatdifferently,andthereforeacontextualvariable, UKCATcandPerSchool,wasprovidedbyHICwhichcountedthe numberofcandidatestakingUKCATinastudent’sschoolsince thetest’sinception. Schoolingmeasures SelectiveSchool Someinformationonschooling,includingschoolcodes,was availablefromUCAS,andtheschoolcodescouldalsobelinked DFESshrunkVA intocontextualdataavailablefromtheDepartmentforEducation DFES.AvePointStudent (DfE;formerlyDFES)atKeyStage5fortheacademicyear2010(file createdMay2011),forschoolsinEngland.Themergingofthetwo DFES.AvePointScore: datasetswascarriedoutbyHIC.Schooltypewasavailablefrom twoseparatesources,UCASandDFES.InUCAS’sdata,of4,811 students,69hadmissinginformation,360wereinUCAS’s‘Unknown’ category,219were‘ApplyOnlineUK’,and86were‘Other’.Of4,077 studentsforwhominformationwasavailable,1,941(47.6%)were classifiedascomingfromSelectiveSchools(‘GrammarSchool’or ‘IndependentSchool’),and2,136(52.4%)fromnon-SelectiveSchools (‘ComprehensiveSchool’,‘Further/Highereducation’,‘SixthForm Centre’and‘SixthFormCollege’).TheDFESdatabasealsohada measureofSelectiveSchooling,withinformationon2,830 individualsavailable,ofwhom1,387(49.0%)attendedselective schools.TheoverlapoftheUCASandDFESclassificationswasgood, butnotperfect.Ourfinalmeasure,entitledSelectiveSchoolhada valueof1ifeitherUCASorDFESdatasuggestedaschoolwas selective,andotherwisewas0.Altogetherofthe4,811individuals inthePrimaryDatabase,informationwasavailableforoneorboth McManusetal.BMCMedicine 2013, 11:244 Page8of25 http://www.biomedcentral.com/1741-7015/11/244 Table1Summaryofvariablesintheanalysis(Continued) sourcesin4,114cases,ofwhom1,986(48.3%)hadevidenceof havingattendedaselectiveschool. Contextualschoolmeasures.TheDfEdatahadatotalof22 contextualmeasuresonschools.Afterarangeofpreliminary, exploratoryanalysesweconfinedtheanalysestothreevariables: DFESshrunkVA,whichisameasureofvalueaddedbetweenKey stages4and5,andwasavailablefortheschoolsof2,561students; DFES.AvePointStudent,whichisameasureoftheaveragepoints gainedbyeachstudentataschoolacrossallofthatschool’s examinationentries,andwasavailablefortheschoolsof2,586 students;andDFES.AvePointScore,whichisasimilarmeasure tothepreviousoneexceptthattheaverageisatthelevelof examinationentries(ratherthanstudents),andwasavailablefor theschoolsof2,582students. Demographicmeasures UK Nationalitywasbasedontheonlineinformationprovidedwhen studentstookUKCAT;of4,811students,4,598(95.6%)wereUK UCAS.male nationals,176(3.7%)wereEU/EEAnationalsand37(0.8%)were CAND.Age fromoutsidetheEU/EEA;thebinaryvariablewascalledUK. CAND.AgeGT21 SexwasbasedoninformationprovidedbyUCAS;of4,811students, 2,081(43.3%)weremaleand2,730(56.7%)werefemale.The CAND.Age30plus variablewascalledUCAS.male,scoring1=maleand0=female. UCAS.Ethnic2. AgewasbasedonstatedageinyearswhentakingtheUKCATtest, andrangedfrom17to45(mode=18,mean=19.55,SD=2.84). Agewasmissingin45cases,28.9%ofstudentswereaged21+, and1.3%wereaged30+.ThevariablewascalledCAND.Age. Additional0/1variableswerecreatedtoindicatewhether candidateswere21orolderor30orolder(CAND.AgeGT21, CAND.Age30plus). Ethnicitywasbasedonthestandard23categoriesintheUCAS coding.Ethnicitywasmissingin69cases,for214wascodedas Unknown,andfor192wascodedas‘Notgiven’.Onasimplified sixcategorybasistherewere3,057White,577Indiansub-continent, 223OtherAsian,92Black,140Mixedand60Other.Forsimplicity, andasinmanyotherstudies[37])wegroupedstudentsasWhite (n=3,057,73.7%)andNon-White(n=1,092,26.3%),inavariable calledUCAS.Ethnic2. Socio-economicmeasures CAND.NSSEC Socio-economicclassification(SEC),variableCAND.NSSEC,was basedontheonlineinformationprovidedbystudentstaking IMDOverallQualityDecile UKCAT,whocompletedtheabbreviatedversionoftheself-coded IMD1IncomeDecile(withtwosubscales) questionnaire(NS-SEC)providedbyUKNationalStatistics2.SECwas calculatedseparatelyforeachparent(ifprovided),andthehigher IMD2EmploymentDecile SECused.Of4,091individualswithusableinformation,3,740(91.4%) IMD3HealthDisabilitySkillsDecile wereinSECgroup1,105(2.6%)ingroup2,146(3.6%)ingroup3, 38(0.9%)ingroup4,and62(1.5%)ingroup5,wheregroup1has thehigheststatus. IMD4EducationDecile(withtwosubscales), Socio-economiccontextualmeasures.ForapplicantslivinginEngland, postcodesforplaceofresidencewereusedtolinktosmall-area IMD5HousingAndServicesDecile(withtwo censusstatisticscollectedaspartofTheEnglishIndicesof subscales) Deprivation[38]andwhichgenerateaseriesofIndicesofMultiple IMD6CrimeDecile, Deprivation(IMD).Foreaseofanalysis,HICconvertedthemeasures todeciles,lowscoresindicatinggreaterdeprivation. IMD7LivingEnvironmentDecile(withtwo IMDOverallQualityDecileprovidesanoverallsingleindicatorof sub-scales). deprivation.Inadditionthereare15moredetailedscalesand subscales,whosenamesaremoderatelyself-explanatory: IMD1IncomeDecile(withtwosubscales),IMD2EmploymentDecile, IMD3HealthDisabilitySkillsDecile,IMD4EducationDecile(withtwo subscales),IMD5HousingAndServicesDecile(withtwosubscales), IMD6CrimeDecile,andIMD7LivingEnvironmentDecile(withtwo sub-scales).Notethatalthoughthesescalesaredescribedinterms ofdeprivation,theyarescoredas1=highdeprivationand10= lowdeprivation,andthereforearerenamedas‘Quality’sothat higherscoresindicateahigherqualityonthemeasure. McManusetal.BMCMedicine 2013, 11:244 Page9of25 http://www.biomedcentral.com/1741-7015/11/244 schemes,andcandidateshave chosen tostudy componentextracted(TRTable4).Since the different subjects.Four medical schools werefrom principle componentsforthetwoanalyseswere Scotlandand eight fromthe restoftheUK,entrants bothon standardizedscales,they couldbe combined tothe former mostly, butnotentirely,taking toprovidean optimal summarymeasureof ScottishHighersrather thanA-levels.Noeasy EducationalAttainmentforthe majority of solutionispossibleforthedifficultproblem of students.Educationalattainment measuresdiffer equatingthetwodifferent setsofresults[37],and betweenmedicalschoolsandbetweencohorts,but we havefollowedtheapproachofTiffinetal.[19]in because themaininterest inthisstudy isprediction convertingScottishHighersandA-levelstoz-scores, within medical schools,wehavestandardized whichcanthenbe combined. TheTechnicalReport EducationalAttainmentwithincohortsandmedical [34]describesanextensivesetofpreliminary schools,resultinginthe variable we call analysesofthewiderangeofdifferentmeasuresof zEducationalAttainment.It shouldbenoted that attainment(seeTable1).Briefly,each ofthe 42 educational qualificationswereonlyavailablein derivedscoresforA-levels,AS-levelsand GCSEs most casesfornon-maturestudents(age lessthan wascorrelated withOverallScore(TRTable1a,b). 21).Althoughstatisticallyoptimal,andhencegood Multipleregressionssuggestedthatonlya subsetof forassessingunderlyingprocessesusingasmuch eight measures(TRTable2)showedindependent informationaspossible,werealizethatzEducational predictionsofoutcome.Missing values forthese Attainmentdoesnotreflectthecurrentselection eight measureswerereplacedbyexpectation- processes,andthereforewealsoreportresultsfor minimization(EM)imputation,theresulting8x8 themoreconventionalmeasuresofthreebestA- correlationmatrixfactoranalyzed,thefirstprinciple levels,fourbestAS-levels,ninebestGCSEs,fivebest componentextracted, (which hasameanof zero ScottishHighers,fivebestScottish“HighersPlus” and SDofone),and scoresonthatwere usedasan (whichincludesfinerdefinitionofbandswithin optimal summarymeasure ofattainmentatA-level, grades),andthebestScottishAdvancedHigher. AS-leveland GCSE.A similar processwascarried 3. UKCATSCORES.ThemainmeasuresfromUKCAT outforthe 51derived measuresofHighers,‘Highers werethe scoresonthe cognitivetests,thetotal Plus’and Advanced Highers,each ofwhich was score,UKCATtotal,andthe scoresonthefour correlated withOverallScore(TRTable3).Ten subtests,UKCATabstractReasoning, independentpredictorswere found,missing values UKCATdecisionAnalysis, replacedbyimputation,and thefirstprinciple UKCATquantitativeReasoningand Table2CorrelationsofUKCATsub-scoreswithoutcomes Abstract Decision Quantitative Verbal OverallMark SkillsMark TheoryMark reasoning analysis reasoning reasoning Abstractreasoning(zUKCATabstractReasoning) 1 .196*** .190*** .114*** .080*** .053** .052* (4,811) (4,811) (4,811) (4,811) (3,184) (2,075) Decisionanalysis(zUKCATdecisionAnalysis) .196*** 1 .156*** .146*** .090*** .056*** .077*** (4,811) (4,811) (4,811) (4,811) (3,184) (2,075) Quantitativereasoning .190*** .156*** 1 .213*** .076*** .044* .079*** (zUKCATquantitativeReasoning) (4,811) (4,811) (4,811) (4,811) (3,184) (2,075) Verbalreasoning(zUKCATverbalReasoning) .114*** .146*** .213*** 1 .115*** .028 .177*** (4,811) (4,811) (4,811) (4,811) (3,184) (2,075) TotalUKCATscore(zUKCATtotal) .604*** .655*** .583*** .591*** .148*** .075*** .160*** (4,811) (4,811) (4,811) (4,811) (4,811) (3,184) (2,075) Educationalattainment(zEducationalAttainment) .144*** .131*** .133*** .087*** .362*** .210*** .351*** (3,432) (3,432) (3,432) (3,432) (3,432) (2,240) (1,407) ThreebestA-levels(Alevels_TotalBest) .123*** .121*** .127*** .062** .177*** .096*** .248*** (2,764) (2,764) (2,764) (2,764) (2,764) (2,000) (1,250) Fivebesthighers(SQAhighers_TotalBest) .083* .129*** .202*** .070 .003 .027 .074 (769) (769) (769) (769) (769) (298) (199) CorrelationsoftheUKCATsubscaleswitheachother,withUKCATtotalscore,andwithpriorEducationalAttainment(threebestA-levels,fivebestHighersand zEducationalAttainment),andmedicalschoolperformance(Overall,andSkillsandTheoryassessments);***P<.001;**P<.01;*P<.05. Table3SimplePearsoncorrelationsofkeymeasureswitharangeofdemogErdaupchaticio,nsaclhAototali,nsmoecniatlan3d/5UbKeCstAAT-lepvreolcs/eHsisghmeresasUuKrCeAsTtotalscore Overallmedicalschool http://w McMan (zEducationalAttainment) (Alevels_TotalBest (zUKCATtotal) score(OverallScore) w u w s SQAhighers_TotalBest) .b et Demographic UKnational(UK) .008 .000/-.042 .060*** -.007 iom al. measures e BM (3,432) (2,764/769) (4,811) (4,811) dc C e M Male(UCAS.Male) (-3.0,43372*) (.022,766/4.0/57869) (.046,7142**)* -(4.0,73492*)* ntral.c edicin o e AAggeedd2310++((CCaanndd..AAggeeG30Tp2l1u)s) nn//aa nn//aa -(-4..00,2763606*)** .(-04.08,070366**)* m/1741-702013, 11:2 14 (4,766) (4,766) 5/14 1 Ethnic2(non-White)(UCAS.Ethnic2) -.053** -.062**/-.033 -.141*** -.142*** /2 4 4 (3,221) (2,549/766) (4,149) (4,149) Schoolmeasures Selectiveschooling(SelectiveSchool) .051** .038*/.120*** .075*** -.101*** (3,432) (2,764/769) (4,811) (4,811) DfESvalueaddedKS5(DFESshrunkVA) -.012 .012/n/a -.014 -.049* (2,092) (2,119) (2,561) (2,561) DfESaveragepointsperstudent(DFES.AVEPointStudent) .085*** .127***/n/a .097*** -.065*** (2,114) (2,141) (2,586) (2,586) DfESaveragepointsperexamentry(DFES.AvePointScore) .111*** .101***/n/a .044* -.111*** (2,109) (2,136) (2,582) (2,582) Socialbackground Socio-economicclassification(SEC)(1=High5=Low) -.058* -.084***/-.046 -.056*** -.011 (CAND.NSSEC) (2,939) (2,356/675) (4,091) (4,091) Overalldeprivationdecile(1=high,10=lowdeprivation) .079*** .076***/n/a .113*** .032 (IMDOverallQualityDecile) (2,275) (2,307) (3,074) (3,074) Incomedeprivationdecile(IMD1IncomeDecile) .078*** .083***/n/a .125*** .039* (2,275) (2,307) (3,074) (3,074) Employmentdeprivationdecile(IMD2EmploymentDecile) .063** .073***/n/a .109*** .008 (2,275) (2,307) (3,074) (3,074) Healthdisabilitydecile(IMD3HealthDisabilitySkillsDecile) .055** .048*/n/a .098*** .016 (2,275) (2,307) (3,074) (3,074) P Educationdeprivationdecile(IMD4EducationDecile) .056** .064**/n/a .083*** -.019 a g e (2,275) (2,307) (3,074) (3,074) 1 0 o f 2 5