. RREEPPOORRTT cactacthc-h2-22.2o.rogr.gu.ku/kR/Reaelaisliisnign-gA-Ammbibtiitoinon . REALISING AMBITION Lessons on replication and evidence-based interventions- one year in Written by the Social Research Unit at Dartington 02 . YEAR 1 REPORT catch-22.org.uk/Realising-Ambition TITLE OF REPORT (STATIC OR NOT - TBC) CONTENTS TITLE OF REPORT (STATIC OR NOT - TBC) Executive summary 03 About this report 06 What is the Realising Ambition programme? 07 Why Realising Ambition? 09 The Realising Ambition portfolio 10 Evidence-based and promising interventions 11 Contribution of Realising Ambition to this evidence base in the UK 13 Support provided to projects 14 Delivery to children, young people and families 15 Evaluation activites and impact to date 17 Five things we have learnt about replication to date 19 What next? 26 Want to learn more about Realising Ambition? 27 Appendix I: Types of intervention in the Realising Ambition portfolio 29 Appendix II: Interventions in the Realising Ambition portfolio 33 Appendix III: Overview of the Realising Ambition Outcome Framework 34 02 . YEAR 1 REPORT catch-22.org.uk/Realising-Ambition TITLE OF REPORT (STATIC OR NOT - TBC) EXECUTIVE SUMMARY TITLE OF REPORT (STATIC OR NOT - TBC) Evidence-based interventions and replication have that commissioners, investors and funders are able the potential to improve outcomes and drive down to use their limited resources to the best effect. costs in the long term. Learning from the first year of the Realising Ambition programme found that: Added to this is the gathering consensus that • there is a lack of evidence-based programmes interventions should focus on achieving outcomes available for funders and commissioners to that divert children and young people from the choose from; criminal justice system, offending and dependency • there is a tendency for organisations to more quickly and with greater efficiency. overestimate the demand for a service; • identifying tightly defined evidence-based Consequently, there is growing momentum interventions is relatively easy, but identifying behind prevention and early intervention: seeking the most promising home-grown interventions to prevent problems from occurring in the first is harder; place or intervening early in developing problems. • there is a need to get better at spotting the Well-evidenced prevention, and early intervention most promising interventions and supporting programmes offer the foundation on which robust their development; estimates of the financial returns on investment may • it is too easy to conflate evidence-based be made. programmes with relatively flexible practices or processes; For the purposes of this research the evidence-based • organisations tend to underestimate the approach is defined as a an intervention that has organisational challenges involved in been tested via the most robust forms of experimental replication. evaluation and proven to have a positive impact on children’s health and development. Replication refers Every year thousands of children and young people to the implementation of pre-defined interventions in enter the criminal justice system for the first time. new locations or with new target groups. This has a major impact on their families and the communities to which they belong, but an even As learning from the first year of the Realising bigger impact on their own life chances. For many Ambition programme demonstrates, implementing of these children and young people, first-time entry promising and evidence-based interventions is to the criminal justice system signals the beginning currently a challenge due to the underdeveloped of a downward spiral of disadvantage, vulnerability nature of this approach in the UK. However, and exclusion. the challenges are surmountable and for commissioners, funders, investors and policy Given the number of lives wasted through offending makers who are looking to pioneer these and unfulfilled potential, combined with the current approaches the lessons and learning from this first economic climate, it is more important than ever in a series of reports will be invaluable to those 03 . YEAR 1 REPORT catch-22.org.uk/Realising-Ambition TITLE OF REPORT (STATIC OR NOT - TBC) taking this agenda forward. It will also be of interest Building the evidence base is central to Realising to voluntary and third-sector organisations seeking Ambition. Over the five years of the programme the to replicate or scale-up an intervention they have Social Research Unit will rigorously evaluate the developed to new audiences, and to organisations impact of four projects by Randomised Controlled replicating an evidence-based intervention Trial (RCT). All projects are being supported developed elsewhere. to monitor and report on child outcomes. This investment in replication and rigorous experimental The Realising Ambition programme is a £25m evaluation will make a marked contribution to the investment by the Big Lottery Fund, managed by evidence base in the UK in terms of understanding Catch22 and supported by the Social Research both what does and does not work. Unit, Substance and the Young Foundation. Realising Ambition will replicate across the UK The majority of projects delivery in their first year a portfolio of 25 evidence-based and promising of replication have met or exceeded expectations interventions designed to help children and young regarding numbers of children, young people and people aged 8–14 avoid pathways into offending, families served. Not unexpectedly, a minority of giving them a better chance to realise their projects have faced challenges in replication and ambitions and their potential. delivering to the planned numbers of children and families. The portfolio comprises a mix of universal school- based prevention, family and parenting early A number of themes and key learning points intervention, mentoring, and community-based and for policy makers, commissioners and delivery therapeutic interventions. Ten of the 25 projects organisations have emerged and are explored in are internationally renowned evidence-based this interim report, some of which relate to the programmes, some of which have never been challenges faced by some projects in terms of replicated in the UK before; these are underpinned meeting planned delivery numbers. by high-quality experimental evaluations. Fifteen of the 25 projects are UK home-grown and promising First, there are not enough evidence-based interventions; each has a strong track record of programmes available for funders and implementation and, for the most part, preliminary commissioners to choose from. Less than 4% of the evidence of impact. initial 240 applications to the Realising Ambition Realising Ambition will replicate across the UK a portfolio of 25 evidence-based and promising interventions designed to help children and young people aged 8–14 avoid pathways into offending 04 . YEAR 1 REPORT catch-22.org.uk/Realising-Ambition TITLE OF REPORT (STATIC OR NOT - TBC) programme met the highest standards of evidence. programmes with relatively flexible practices There is a need to better support the ‘innovation to or processes. There were numerous examples proven impact pipeline’ (ie the process of moving a of practices and processes being ‘packaged’ programme from an idea to something that we can as programmes among applicants to Realising evidence has a positive impact on people’s lives) in Ambition, and a couple made it into the portfolio. It the UK by increasing investment in the refinement is important to differentiate between them as it is and rigorous evaluation of the most promising harder to replicate flexible processes and practices interventions. than tightly defined programmes. Second, there is a tendency for organisations Finally, there is a tendency to underestimate the to overestimate the demand for a service, organisational challenges involved in replication, either in terms of children and families who such as the time and resources required to recruit might access the intervention or in terms of and train staff qualified to deliver the intervention, delivery intermediaries, such as schools, who the effort required to establish strong partners and might purchase and deliver the intervention. A networks for replication, the navigation of internal consequence of this might be to relax eligibility organisational bureaucracies, as well as the criteria and work with an easier-to-engage cohort aforementioned challenges in accurately predicting of children and families. As one would expect, demand. Organisational understanding of the this would lead to a number of unfavourable intricacies of evidence-based programmes is a key consequences. Unrealistic expectations about driver to replication success. demand account for some of the shortfall in the number of beneficiaries served thus far. Robust data on need and the market for a particular intervention could improve projections about the number of beneficiaries that could effectively be served. Third, identifying tightly defined evidence-based interventions is relatively easy, but identifying the most promising home-grown interventions is harder: we need to get better at spotting the most promising interventions and supporting their development. Fourth, it is easy to conflate evidence-based 05 . YEAR 1 REPORT catch-22.org.uk/Realising-Ambition TWITLHEA OTF RIESP OTRTH (EST RATEICA OLRI SNOITN -G T BACM) BITION PROGRAMME? TITLE OF REPORT (STATIC OR NOT - TBC) This publication is the first-year report from the The first half of the report describes the objectives consortium supporting the delivery of the Big of the Realising Ambition programme, the process Lottery Fund’s Realising Ambition programme, a for identifying the 25 projects that comprise the £25m investment in replicating evidence-based portfolio of interventions and an introduction to the and promising approaches to help prevent children portfolio itself. It considers the contribution that and young people aged 8–14 from becoming the Realising Ambition portfolio makes to the UK involved in the criminal justice system. It shares evidence base and summarises delivery activity to some early learning from the Realising Ambition date. programme, from the perspective of the programme support team, about the catalysts and barriers The second half of the report shares five key to replication of evidence-based and promising learning points. These initial reflections and intervention efforts across the UK. learning revolve primarily around the initial set-up of the Realising Ambition programme and the early It is written primarily for policy makers and stages of project set-up and replication. commissioners who are currently involved in or considering the implementation and replication The report concludes by introducing a number of of evidence-based programmes or promising other emerging themes that will be addressed in interventions. It will also be of interest to voluntary subsequent learning reports. and third-sector organisations seeking to replicate or scale-up an intervention they have developed to The report is written by the Social Research Unit on new audiences and to organisations replicating an behalf of and in collaboration with the consortium evidence-based intervention developed elsewhere. partners Catch22 (programme lead), Substance and the Young Foundation. This report is written primarily for policy makers and commissioners who are currently involved in or considering the implementation and replication of evidence-based programmes or promising interventions. 06 . YEAR 1 REPORT catch-22.org.uk/Realising-Ambition TWITHLEA OTF IRSEP TORHT E(S TRAETIAC LORIS NIONT G- T ABCM) BITION PROGRAMME? TBC) Every year thousands of children and young people ambitions and their potential. The investment is enter the criminal justice system for the first time. characterised by a focus on replication rather than This has a major impact on their families and the innovation. The programme is outsourced from th e communities to which they belong, but an even Big Lottery Fund and delivered by a consortium bigger impact on their own life chances. For many of organisations led by Catch22 and including the of these children and young people first-time entry Social Research Unit at Dartington, Substance and to the criminal justice system signals the beginning the Young Foundation. of a downward spiral of disadvantage, vulnerability and exclusion. A unique portfolio of interventions The Realising Ambition portfolio comprises A different approach to foundation both internationally recognised interventions, investment underpinned by the highest standards of evidence, The Realising Ambition programme is a £25m and UK home-grown and promising interventions. investment by the Big Lottery Fund to replicate Twenty-five projects across the UK are being a portfolio of 25 evidence-based and promising supported, over five years, to refine and faithfully interventions designed to help children and young replicate evidence-based or promising interventions people aged 8–14 avoid pathways into offending, so that more children and young people benefit. giving them a better chance to realise their Children and 25 projects across Policy makers, young people avoid the UK have evidence commissioners pathways into of what works and and funders learn offending are able to replicate about how they = + the most effective can identify and approaches support replication of evidence-based programmes A programme of national and international significance 07 . YEAR 1 REPORT catch-22.org.uk/Realising-Ambition TITLE OF REPORT (STATIC OR NOT - TBC) A strong emphasis on support to projects A strong emphasis on sharing learning Catch22 are the coordinating body for the grant, from success and facing challenges responsible for coordination of the programme Learning about the catalysts and barriers to management and support being provided to successful replication of evidence-based and projects and the dissemination of learning. Three promising interventions is central to Realising other organisations are also playing a key role Ambition. The programme is ambitious and it is in programme delivery and support. The Social recognised that there will be challenges along the Research Unit is helping organisations to refine way. Learning will be disseminated to policy makers promising interventions in order to get them ready and commissioners, to the third and public sectors, for wider replication and evaluation. Substance, and to those designing and delivering services for a social research cooperative, is equipping and children and young people. Learning will come from supporting the portfolio of projects to use Views, an case studies produced by individual projects and online project management, outcome monitoring supporting consortium members, from a Social and reporting platform. The Young Foundation is Research Unit programme-level impact evaluation helping to build and strengthen the organisational and four RCT’s, as well as a Tavistock Institute for capacity, qualities and processes necessary for Human Relations process evaluation. the successful replication and sustainability of interventions. A programme of national and international significance A commitment to significantly boost the Realising Ambition is a bold and ambitious evidence base in the UK programme. It is the first of its kind in the UK. As this report makes clear, the evidence base in Realising Ambition will help to move science the UK is underdeveloped. Realising Ambition will to practice, by transporting proven models and significantly build the evidence base across the UK: replicating them, and practice to science, by the Social Research Unit will rigorously evaluate nudging promising innovations towards becoming four interventions by RCT and estimate the financial more tightly defined and ready for replication and returns on investment of all those projects already rigorous evaluation. The aspiration is the greater underpinned by robust evidence of impact. All use of ‘what works’ in preventing pathways into projects are being supported to monitor and report youth offending in the UK. on child outcomes using a specially developed outcomes framework and an outcome monitoring approach embedded within the Views platform. 08 . YEAR 1 REPORT catch-22.org.uk/Realising-Ambition TWITLHEY O FR REEPAOLRITS (SINTAGT ICA OMR BNOITT I- OTBNC?) (STATIC OR NOT - TBC) Policy makers and commissioners increasingly rigorously evaluated by experimental methods want to invest limited resources in services or (RCTs and similar designs) which demonstrate social interventions that have a robust evidence with a good degree of confidence the impact on base: those that have been demonstrated with outcomes of beneficiaries. confidence to improve the outcomes of the children and families they serve. There is also Yet while innovation flourishes, only a tiny growing momentum behind prevention and early proportion of services delivered to children and intervention: seeking to prevent problems from young people are evidence-based interventions occurring in the first place or intervening early or programmes. This needs to change. To this end, in the development of problems. Well-evidenced Realising Ambition is principally about identifying prevention and early intervention also offer the and replicating the best available evidenced-based foundation on which robust estimates of the programmes for children and young people aged financial returns on investment may be made. In 8–14 across the UK. times of austerity this is more important than ever. However, the evidence base that can be drawn In the last few decades there has been a concerted on in the UK is still relatively underdeveloped. As international shift towards building the body this report makes clear, there are too few well- of evidence regarding ‘what works’ to improve evidenced home-grown interventions for policy outcomes and reduce the likelihood that children makers and commissioners to choose from. On a aged 8–14 will become involved in the criminal related level, therefore, Realising Ambition is also justice system. One particular area of focus has about enhancing the ‘innovation to proven impact been evidence-based programmes: discrete, pipeline’: identifying some of the most promising organised packages of practices, often spelled interventions, refining these innovations and out in a manual, that explain what should be building the evidence base in the UK. delivered to whom, when, why, how and in what order. A growing number of programmes have been Realising Ambition is also about enhancing the ‘innovation to proven impact pipeline’: identifying some of the most promising interventions, refining these innovations and building the evidence base in the UK. 09 . YEAR 1 REPORT catch-22.org.uk/Realising-Ambition TTITHLEE O RF REEAPOLRITS (ISNTGAT IAC MORB NIOTTI -O TNBC )PORTFOLIO NOT - TBC) The brief from the Big Lottery Fund to the delivery questions: Is the intervention focused, practical consortium was to identify 25 promising or and designed based on the best available evidence strong interventions to be replicated by strong about what works in improving child outcomes? C an organisations. The portfolio as a whole was required the intervention be replicated at scale in real-world to be balanced in terms of intervention type conditions? Has the intervention been rigorously (school, family, community), scale of replication and evaluated by high-quality experimental or quasi- geographical spread across the UK. experimental evaluation? Does the intervention have a positive impact on outcomes and no Each intervention was required to: negative side effects? • predominantly target children and young people aged 8–14; 2. The Young Foundation’s Organisational • prevent problems developing or intervene early Health Scorecard: a set of indicators of to improve poor outcomes, reduce risks and organisational health assessing the degree to enhance proactive factors associated with entry which an organisation has in place some of the to the criminal justice system; requisite characteristics required for replicating • be underpinned by the best available evidence interventions. The scorecard also has four or potential for impact; dimensions: Skill/Will, the extent to which an • be ready for wider replication; and organisation has a strong board of trustees, strong • be delivered by committed and strong leadership, delivery teams and robust operating organisations. systems; Social Impact, a clear organisational outcomes focus, community of benefit and impact The 25 projects comprising the Realising Ambition assessment and information systems; Ability to Portfolio were determined via a competitive Replicate, strong networks, infrastructure and tendering process during late 2011. Over 240 products; and Sustainability, investment readiness, expressions of interest were received. These were a strong business plan and finances, and strong long-listed to 50 and subsequently invited to partnerships. workshop briefings and to submit full applications. Two objective tools were used in the project Together these two tools provided an objective identification phase: basis on which to help determine a balanced portfolio of interventions underpinned by the 1. The Social Research Unit’s ‘What Works’ best available evidence or greatest potential for Standards of Evidence: a set of objective criteria success in improving outcomes. The requirement used to determine how refined and ready for for a balanced portfolio – in terms of intervention replication an intervention is and the strength type, geographical spread and scale of replication of the evidence base underpinning it. The ‘What – played a part in determining the final portfolio Works’ Standards of Evidence revolve around four selection approved by the Big Lottery Fund. 10