Amendments to the Bar Course Aptitude Test For approval by the Legal Services Board This application is made in accordance with the requirements set out in the Legal Services Board’s (LSB) Rules for Rule Change applications. The Bar Standards Board (BSB) wishes to provide the information below to support its application. Any queries about this application should be made to: Ms Hayley Gault Senior Training Supervision Officer [email protected] Or Mr Christopher Young Policy Manager (Quality Assurance) [email protected] Bar Standards Board 289-293 High Holborn London WC1V 7HZ Tel: 020 7611 1444 1 Summary 1. This application seeks approval from the Legal Services Board (LSB) for amendments to the score which must be achieved in order to pass the Bar Course Aptitude Test (BCAT). The BCAT is a requirement for those who wish to enrol on the Bar Professional Training Course (BPTC), which is the vocational stage of training towards becoming a barrister. The BCAT was introduced in 2013 as a means of ensuring applicants to the BPTC have the analytical and critical reasoning skills necessary to succeed on the BPTC and thus have a reasonable prospect of progressing into a career at the Bar. 2. The current implementation of the BCAT is not achieving the objectives originally intended, but we demonstrate here that higher scores on the Test correlate well with success in the training, and provide evidence of the necessary intellectual skills to a more reliable extent than other proxies available. Training for the Bar is a small and prestigious market; we have identified a need to protect the interests of candidates in the current training system by ensuring that those with an exceptionally small chance of success do not needlessly commit to the substantial cost and time involved. For the future, we identify that this Test can provide a very significant source of objective evidence in our assessment of risks in an evolving market for training, and contribute directly to the evidence of competence required for authorisation. 3. Specifically, we are seeking to amend the cut score, which is the score required for a candidate to pass the Test. The current cut score is 37, and the proposed new cut score is 45. This change is intended to exclude students who are most unlikely to succeed on the BPTC and to avoid disruption of the learning experience of able students on the course caused by those who lack the critical thinking skills to participate fully. Background Current requirements 4. There are three stages of training towards becoming a barrister: the academic stage, the vocational stage, and the professional stage. The principal requirement of the vocational stage consists of successful completion of the BPTC at an approved training provider. The BPTC is a one year (full-time) or two year (part-time) course where students learn the skills necessary to prepare them to practise as a barrister, subject to the completion of the professional stage of training, including pupillage. The course comprises a mix of practical and knowledge-based skills. 5. The BSB introduced the aptitude test as a prerequisite for enrolment on the Bar Professional Training Course (BPTC) on the recommendations of the Neuberger Report on entry to the Bar (2007) and the Wood Review of the BVC (2009) The aim was to use a fair aptitude test which measured critical 2 thinking and analytical skills so that those without certain prerequisite skills for the profession would not undertake the BPTC. 6. Approval for the introduction of a Bar Course Aptitude Course (BCAT) was secured from the Legal Services Board in July 2012. The BSB committed to undertaking a comprehensive evaluation of its efficacy and impact over the first five years. The present application draws on evaluative work done so far. 7. The primary objective of the introduction of the BCAT was improvement of standards on entry to and exit from the BPTC. Whilst not directly a regulatory concern, we might expect a consequent increase in satisfaction with the training from both students and tutors, a reduction in failure rates, and hence greater confidence amongst those who are considering training for the Bar. 8. The BCAT was planned for implementation in 2013, for candidates entering the BPTC in September that year. A test fee of £150 was set for those candidates taking the test in the UK, £170 for those taking the test overseas. The test fee covers an administration charge for Pearson VUE and the full economic cost recovery by the BSB of administration and implementation cost for the test, over a period of five years, in accordance with the BSB’s Fees and Charges policy. The initial cost of research and development for the test prior to 2013 was not accounted for in setting the candidate fee. The BCAT was introduced on 3 April 2013 and all BPTC students entering from autumn 2013 for the 2013/14 academic year onwards were required to pass the BCAT. 9. The previous Bar Training Regulations stated that before starting the vocational stage of training a person must, amongst other requirements, have achieved a score of -1.34 or such score as substituted by the BSB from time to time in the [BCAT]”. However, with the introduction of the BSB Handbook in January 2014, the Bar Training Rules now state that: “Before starting the vocational stage, a person must: 1. have completed (or been exempted under Section 4.B7 from) the Academic Stage; and 2. have successfully completed the Bar Course Aptitude Test which is set by the Bar Standards Board from time to time; and 3. be a member of an Inn of Court.” 10. Since the introduction of the BCAT, the cut score has remained at 37. This was originally identified, based on the BCAT pilot, as a score which would identify the bottom 10% of candidates who took the test and thereby exclude those at significant risk of failure of the BPTC. Future Bar Training 11. The BSB has devised a programme of reform to training for the Bar which was initiated to reflect our adoption of a new, less prescriptive approach to 3 regulation. The programme, called Future Bar Training1, will involve substantial changes to all stages of training for the Bar, which will roll out from 2017. 12. In October 2015 the BSB published a Professional Statement (Annexe 4) which describes the knowledge, skills and attributes that all barristers should have on “day one” of practice (ie upon the issue of a full qualification certificate, on which basis they may apply for a full practising certificate). The Statement represents our move towards a more outcomes-focused approach to the regulation of standards at the point of entry to the profession and is an essential part of Future Bar Training. The Statement will help towards making qualification routes to the Bar more flexible. Our Proposal The problem we are seeking to address 13. We have identified in our Regulatory Risk Index2 that inadequate training and preparation for practice is a risk to our regulatory objectives. The interdependency of students in the learning environment on a course such as the BPTC is such that the presence of weaker students is widely reported by other students to have a detrimental effect on their learning experience. This has been consistently observed as part of annual monitoring visits to BPTC providers and through the 2010/11 Student Perception of Course Surveys3. 14. There is a regulatory interest in ensuring that the training inspires confidence for the widest pool of potential candidates, who are making a substantial commitment in participating and whose experience of the training can be influenced significantly by the ability of those training alongside them to participate. 15. The Bar is perceived as a high status profession which attracts interest as a career for many irrespective of their suitability or understanding of what the profession entails. The Bar is a career into which candidates are persuaded by wider cultural expectations beyond candidates’ own career aspirations. The profession plays a significant role in securing access to justice for consumers. It is also intellectually very demanding. Not all aspiring barristers have the aptitude to meet a competent standard. Training to become a barrister is intensive and takes significant time; it is therefore expensive. If candidates are not to waste significant time and money on training – ie by committing to and later failing the BPTC – adequate selection processes are required. There are also wider concerns about social justice and diversity in that the risk of undertaking the expense of the BPTC is likely to bear more heavily on those from disadvantaged or non-traditional backgrounds. 1 For more information on Future Bar Training, see the BSB’s website. 2 Available here. 3 Annexed to the BSB’s 2012 application to the LSB – available here. 4 16. The evidence from the past five years of BPTC students is that the demands of the course are significant; around 40% of students enrolled onto the BPTC in 2014 have yet to pass the course4. 17. BPTC providers have their own selection processes, and have in recent years stated an increased focus on quality rather than quantity of applicants – an approach borne out by the fact that many providers no longer recruit to their maximum validated numbers – but they have a strong financial interest in recruitment. The BSB also has its own qualification requirements, ie a minimum of 2:2 undergraduate degree. However, it is clear that the combination of providers’ own selection process and the BSB’s requirements does not adequately ensure that the candidates making the investment in the BPTC are always those most likely to progress successfully. 18. We have already introduced a number of interventions to help address this problem. This includes the existing academic qualification requirements of a 2:2. There is evidence that some providers have effectively raised their threshold to a 2:1; we have concerns for the consequences of such a measure on inclusion. There is a wealth of independent data to suggest that raising the minimum academic achievement required by the BSB e.g. to a 2.1 could potentially have a disproportionate impact on students from protected characteristic groups. We have included with this application data relating to BPTC students and their degree classification which reinforces this (Annexe 6). 19. We have also published key statistics relating to the BPTC. The adoption of centrally-assessed examinations on the BPTC in 2011/12 provided the basis upon which a clearer evidence base of variable candidate performance could be developed. The first opportunity to publish and analyse this evidence was in 2015. 20. It is envisaged that the key statistics document will provide clarity regarding success rates, which will help motivate higher performance by providers through the use of appropriate entry requirements and ensuring a high quality of training. However, we remain concerned about this as an intervention, as it could have potential unintended consequences, for example by encouraging greater exclusivity in recruitment practices. 21. We also give potential BPTC candidates a health warning which describes the high level of commitment, resilience and ability required for both training for and a career at the Bar. However, our experience and anecdotes derived from other similar professions suggest that many candidates, especially younger candidates, do not necessarily make entirely rational judgements about their prospects for success when they commit to such a challenging programme of 4 BPTC Key Statistics Report 2016, p3. 5 study. Moreover, candidates place an arguably reasonable expectation on the provider and the regulator to exercise a duty of care in only recruiting suitable candidates. The BCAT today 22. We have over time given careful consideration to the availability of more focused and reliable predictors of intellectual ability. The Watson Glaser type test has been found to be viable and acceptable as an intervention in a number of professional domains for precisely the purpose intended here. 23. When it was first implemented, the BSB commissioned a review of the impact of introducing the BCAT against its intended objectives, from which a first evaluation report was produced in 2015. The evaluation report, which is annexed to this application (Annexe 1) and hereafter referred to as “the Report”, examined the first full year of implementation of the BCAT. It focused on:  the performance of the BCAT as a predictor of a student’s future performance on the BPTC;  the impact of the BCAT – at the current pass rate – in preventing weaker students from starting the BPTC; and  the likely equality and diversity implications if the pass mark is raised, and to what level it could be justifiably raised while minimising potential negative impacts on different protected characteristic groups and students from non-traditional backgrounds. 24. The Report investigated what impact the BCAT has had to date on entry and exit standards on the BPTC, and on the profile of students on the course by reference to their protected characteristics. The Report also investigated students’ and BPTC providers’ current perceptions of the BCAT. The key findings were:  BPTC student data suggests the introduction of the BCAT at a pass mark of 37 has had no impact on the profile of students on the course.  Analysis of data on enrolments onto the BPTC of students does not reveal any statistically significant change in degree class or degree institution following the introduction of the BCAT.  There has not been any increase in the pass rates on the BPTC since the introduction of the BCAT as an entry requirement – pass rates on the BPTC declined between both 2011/12 to 2012/13 and 2012/13 to 2013/14  There is little evidence from interviews with students who failed the BCAT at the first attempt that the introduction of the BCAT is influencing their career / routing decisions.  BPTC provider staff interviewed felt that the introduction of the BCAT had not had any impact on their selection procedures, because the test 6 at the time was deemed too easy to pass, and was not an effective tool to improve standards beyond existing admissions and/or selection procedures. 25. Other key findings of the report were:  That the score that a student obtains in the BCAT is a very strong predictor of their performance on the BPTC.  Data from the 2013/14 BPTC cohort shows that BCAT score and BPTC overall score are significantly positively correlated (r=0.546), with those who scored higher on the BCAT tending to have a higher BPTC overall score.  Additionally, regression analysis was undertaken to determine the statistical reliability of BCAT as a predictive indicator of performance. This indicates that BCAT score significantly predicts both BPTC final overall score and final overall grade.  Regression analysis also indicates that BCAT score increases predictive validity beyond that of educational variables such as degree institution and degree classification. It is also a fairer measure of aptitude as the BCAT is a universal test and does not discriminate against those from non-traditional academic backgrounds.  Additional analysis has shown that the BCAT aligns with the skills required in the BPTC course specification requirements. 26. Further to the findings in the Report regarding the alignment of the BCAT and BPTC course specification requirements, we undertook further analysis of the BCAT in relation to the BSB’s Professional Statement, Threshold Standard and Competences (Annexe 5). The analysis identifies that the BCAT is assessing attributes in a fair, reliable and valid manner, that are needed on “day one” as a barrister, as defined by the Professional Statement, threshold standard and competences. It concludes that 41% of the Professional Statement could be directly mapped to the BCAT content, which is a satisfactory proportion of mapping for a test of this nature. 27. We reviewed the Report in the autumn of 2015, and undertook further investigation into the options available for meeting the objectives of the Test, incorporating more recent pass rate data (ie information from the 2014/15 cohort) to inform our analysis. The LSB was briefed on the detail of our review and options analysis through this period. The BCAT was temporarily suspended from November 2015 to March 2016 to give time for careful consideration of the Test’s future. In February 2016, the matter was considered by the Board. We decided, based on the Report’s findings that the test was an effective predictor of BPTC performance, to retain the Test for an interim period with candidates being informed of their Test score (where previously they were only advised of pass or fail) and the likelihood of their passing the course. In April 2016 the BCAT was re-introduced with the 7 previous cut score of 37 on this basis. The rationale for informing candidates of their exact score, along with the grades of past BPTC students with a similar BCAT score, was to help them make an even more informed decision about whether to proceed with applying for the course. A sample of the score sheet is annexed to this application (Annexe 7). 28. We concluded that the BCAT is a very effective test, notwithstanding the immediate problem of its calibration. Options to address the regulatory problem 29. Having identified that the BCAT was not fulfilling its intended purpose, and that the issue is with calibration, we revisited the wider problems that the BCAT was intended to address. 30. We considered the following options to address the regulatory problem:  Continuing to rely on prior academic qualifications alone – as mentioned above, it is clear that this is not currently an adequate filter on its own. Raising the minimum qualification requirements could also have an impact on equality and diversity. In addition, degree classifications are a measure of academic achievement rather than of the skills necessary to succeed on the BPTC or at a career at the Bar.  Relying on the publication of BPTC key statistics to inform and dissuade unsuitable candidates – this has yet to show its worth as the first round of statistics was only published at the end of 2015. Again, it is clear that the combination of the minimum academic requirement and the publication of key statistics is not an adequate filter but the addition of transparency in the form of the key statistics might lead to improvement in the proportion of candidates that pass. We have important concerns about the risk of the perverse impact of these measures without having a more objective way of calibrating aptitude for the BPTC; providers could, in an effort to achieve better results, focus their recruitment on applicants from traditional backgrounds which might have a negative effect on those from non-traditional backgrounds.  BCAT – the adoption of a suitably well-founded and focused test would enable us to more carefully calibrate a threshold of competence and provide a greater degree of objectivity in our monitoring and its impact on diversity. It is this latter approach which was adopted as the basis of the BSB’s application to the LSB in 2012. The evidence we submit here demonstrates that the BCAT as introduced in 2012 was well- suited to the purpose described above, but it was not calibrated appropriately. Nevertheless, the three years since the BCAT’s implementation have provided a further extensive evidence base for us 8 to be able to examine and analyse the regulatory problem described above. 31. With this in mind, we considered the following options with regard to the BCAT:  Reverting to a previous approach to candidate selection by removing the BCAT – As stated above, the existing interventions are not sufficient on their own to preclude unsuitable candidates from enrolling on the BPTC. Further to this, evidence presented here demonstrates that degree classification of a 1st/2:1 is not as good a predictor of performance on the BPTC as BCAT result; indeed, the BCAT is the strongest individual predictor of performance on the BPTC compared to other factors.  Adopting a different testing instrument - As noted above, evidence suggests that the BCAT in its current form is a valid testing instrument. It is a better predictor of performance on the BPTC than other available measures of aptitude for the profession and its preparatory training (eg degree classification). In addition, the skills tested by the BCAT are aligned with those required for the BPTC and with the requirements of the Professional Statement.  Retaining the BCAT alongside other available interventions - the BCAT in the form implemented in 2016 provides candidates with information to help make an informed choice about whether to proceed with applying for the BPTC. However, the Report demonstrates that the BCAT with its current cut score does not exclude a significant number of individuals who are likely to fail the BPTC from enrolling on the course. As a result, it is not having an effect on BPTC pass rates.  Recalibrating the BCAT alongside other interventions – the BCAT is the best single indicator of BPTC performance currently available and it now provides candidates with useful information about their likely performance on the BPTC based on their score, allowing them to make an informed choice about how to proceed. In considering a change to the cut score, the impact that such a change would have on equality and diversity was examined. The Report found that candidates from black and minority ethnic (BAME) backgrounds performed worse on average on the BCAT. This effect has been documented elsewhere in relation to the type of test (Watson Glaser) and is a phenomenon that we have taken into account in considering the calibration of the cut score and the proportionality of the barriers to entry that are created by the BCAT. Further analysis on the impact a raised cut score would have on these candidates was 9 undertaken, and is annexed to this application (Annexe 3) and discussed further below. Consultation 32. We have been engaged with consultation broadly on education and training for the Bar, and given the relatively technical nature of the change, we did not consider it was appropriate to undertake a formal separate consultation on the BCAT. 33. The BCAT was included as part of the Future Bar Training consultation5 in autumn 2015. The responses to the consultation which mentioned the BCAT generally said that they felt the BCAT was not fit for purpose in its current format and that it needed to be reviewed. Several respondents stated that academic qualifications were not a sufficient measure for aptitude on the BPTC as they only demonstrate intellectual ability rather than the aptitude for the skills the BPTC requires. Several respondents also said that any intervention that filtered students applying for the BPTC must not have an adverse impact on equality and diversity. There was much opposition amongst respondents to the idea of raising the minimum academic requirement to 2:1. This was largely due to concerns about reducing diversity at the Bar, and the perceived inconsistencies between the standards of different universities. A particular concern for a number of respondents was that the requirement of an upper second class degree may unduly restrict access to the profession and have a disproportionate impact on students from disadvantaged backgrounds 34. Further to this consultation, Work Psychology Group consulted provider and student representatives as part of their evidence gathering for the evaluation report discussed later in this application and annexed to this application (Annexe 1). This helped us to understand the nature of the problem we need to address. Raising the cut score 35. We propose to raise the cut score for the BCAT and implement policies appropriate to recalibration of the test, having considered the substantial data now gathered. The cut score is not currently enshrined in the Bar Training Rules but is defined within the requirements and guidance published online by the BSB. These changes constitute an amendment to the regulatory arrangements agreed by the LSB in their response to the BSB’s application to introduce the BCAT in 2012. 36. The performance evaluation section of the Report investigated the validity of the BCAT at its current pass mark, and what the likely impact of changes to 5 A summary of responses to the consultation is available here. 10