(For discussion purposes only- Maybe subject to further review) ) Evaluation of CEMBA/MPA Allama Iqbal Open University /Commonwealth of Learning Draft Report 1 By Ricardo Meilman Cohn based on data collected by Allama Iqbal Open University, Pakistan Commonwealth of Learning February 2018 1 (For discussion purposes only- Maybe subject to further review) ) Contents Executive Summary ...................................................................................................................................... 4 1. Introduction ........................................................................................................................................... 6 2. Program Evaluation .............................................................................................................................. 6 3. CEMBA/MPA ....................................................................................................................................... 7 4. Methodology ......................................................................................................................................... 8 5. Data ..................................................................................................................................................... 10 6. Descriptive Statistics ........................................................................................................................... 10 Demographic characteristics ................................................................................................................... 10 Dwelling .................................................................................................................................................. 15 Schooling ................................................................................................................................................ 17 Labour market ......................................................................................................................................... 18 7. Results ................................................................................................................................................. 22 Estimation of impacts ............................................................................................................................. 22 Economic Return on Investment ............................................................................................................. 25 References ................................................................................................................................................... 27 Appendix A ................................................................................................................................................. 28 Appendix B ................................................................................................................................................. 30 2 (For discussion purposes only- Maybe subject to further review) ) List of Tables Table 1 - Number of students who graduated: 2004-2012 ............................................................................ 8 Table 2 - Number of observations by program ........................................................................................... 10 Table 3 - Program start year by group ........................................................................................................ 11 Table 4 - Number of observations by city ................................................................................................... 11 Table 5 - Schooling prior to CEMBA/MPA by group (%) ......................................................................... 17 Table 6 – Field of studies prior to CEMBA/MPA ...................................................................................... 18 Table 7 - Industry by group - 2012 (%) ...................................................................................................... 19 Table 8 - Sector by group and year (%) ...................................................................................................... 21 Table 9 - Activity by group and year .......................................................................................................... 21 Table 10 - Occupation by group and year ................................................................................................... 22 Table 18 – CEMBA/MPA costs per student in Pakistan Rupees ................................................................ 26 Table 19 - Economic Return of CEMBA/MPA .......................................................................................... 27 Table 20 - Sample means for treatment and control groups ....................................................................... 30 List of Figures Figure 1 - Age distributions ........................................................................................................................ 12 Figure 2 - Gender by group (%) ................................................................................................................. 12 Figure 3 – Household type by group (%) .................................................................................................... 13 Figure 4 – Household size by group (%) .................................................................................................... 13 Figure 5 – Marital status by group .............................................................................................................. 14 Figure 6 - Number of full-time workers in household by group ................................................................. 14 Figure 7 - Number of part-time workers in household by group ................................................................ 15 Figure 8 - Dwelling Status by group (%) .................................................................................................... 16 Figure 9 - Number of rooms in the household dwelling by group (%) ....................................................... 16 Figure 10 - Occupation by group – 2012 (%) ............................................................................................. 19 Figure 11 - Activity by group - 2012 (%) ................................................................................................... 20 Figure 12 - Sector by group - 2012 (%) ...................................................................................................... 20 3 (For discussion purposes only- Maybe subject to further review) ) Executive Summary The Commonwealth Executive Master of Business Administration (CEMBA/MPA) and Master of Public Administration Programs (CEMPA) have been operating with Commonwealth of Learning (COL) and partner universities in many countries. Broadly defined, the objective of these programs is to empower people with the learning that enables them to be agents of economic and social development. We evaluated and provided evidence of the effect of CEMBA/MPA on labour market outcomes. Specifically, we estimated the impact of graduating from CEMBA/MPA at the partner Allama Iqbal Open University, in Pakistan, on earnings, wages, income, occupational choice and entrepreneurship. Our empirical strategy compares the outcomes before and after graduation of CEMBA/MPA for treatment and control groups. Treatment group is composed by alumni of CEMBA/MPA who graduated between 2012 and 2017 (“graduates”). And the control (comparison) group is composed by students who have not graduated by end of 2017 (“still in school”). Data was collected by surveying the graduates and current students of CEMPA/MPA with a questionnaire that includes questions on labour market outcomes for the years of 2012 and 2017; demographic characteristics; educational characteristics and others. Our sample is composed by 341 observations (193 still in school and 147 graduates). We employ the difference-in-differences research design. Given the survey data collected refers to information before and after the program for treated and control groups, by using this methodology we would eliminate any time-invariant selection bias due to unobserved heterogeneity between treatment and control groups. The effectiveness of this approach rested on meeting three key assumptions: i) in the absence of treatment, outcomes of the treatment group would have changed like the control group (“parallel trends”); ii) the data collected from the surveys comes from a random sample of the population of interest; and iii) the information collected from administrative data and survey data is suitable and reliable. The results of the difference-in-differences regression analysis report that graduating from CEMBA/MPA caused the following on the treatment group: • An increase of 37.6% on monthly earnings, which corresponds to approximately 19,588 PKR (≈222 CAD). • An increase of 31.6% on annual income, which corresponds to approximately 160,405 PKR (≈1,819 CAD). • An increase of 28.5 percentage points in the probability of having an occupation as manager, which corresponds to more than twice that probability in 2012 for treatment group. • An increase of 36.8% of wage (earnings per hour). 4 (For discussion purposes only- Maybe subject to further review) ) We did not find robust evidence of impacts on labour supply (hours of work), probability of owning a business, probability of getting business profits, probability of incorporating a business. To calculate the benefit of the program, we used the impact of the program on the average annual personal income from all sources. While this impact does not take into account all the possible impacts from the program, it synthetizes in one many related outcomes (such as earnings, labour supply, wages, business creation and employment). With cost information per student and the estimated benefit per student, we created various scenarios of potential cashflows to calculate the economic return of CEMBA/MPA. Each scenario assumes different parameters for the duration of impacts, the discount rate, magnitude of costs and benefits. By looking at the measures of economic return in each scenario, we can see how sensitive the results are to parameter changes. We concluded the program present high economic return even in very conservative/pessimistic scenarios. In our preferred scenario, in which the impact last for 5 years, decays 30% every year and the discount rate is 6%, the return on investment is 239% per year. That is, for every dollar a student invests in CEMBA/MPA, she/he gets on average 3.39 dollars in return. The fact that the cost of the program is low (as a share of total student income) and the estimated impact on income is high explains the high economic return calculated. 5 (For discussion purposes only- Maybe subject to further review) ) 1. Introduction The Commonwealth Executive Master of Business Administration (CEMBA) and Master of Public Administration Programs (CEMPA) have been operating with Commonwealth of Learning (COL) and partner universities in many countries. Broadly defined, the objective of these programs is to empower people with the learning that enables them to be agents of economic and social development. We evaluated and provided evidence of the effect of CEMBA/MPA on labour market outcomes. Specifically, we estimated the impact of graduating from CEMBA/MPA at the partner Allama Iqbal Open University, in Pakistan, on earnings, wages, income, occupational choice and entrepreneurship. This report is organized as follows. After this introduction, section 2 covers main concepts of impact evaluation, section 3 describes the main features of CEMBA/MPA, section 4 presents the methodology for estimation of impacts, section 5 covers the data. Section 6 reports the descriptive statistics. Finally, section 7 reports the results, including estimation of impacts and calculation of the economics return of the programme. 2. Program Evaluation The main objective of an impact evaluation is to verify whether a program is achieving its objectives or expected impacts. It tries to answer questions such as: “How would individuals who participated in the program have fared in its absence? Were any improvements a direct result of the project, or would they have improved anyway?”. It is very difficult to answer these questions, since an individual either participated or did not in the program, and comparing results of the same individual over time is problematic, as many other variables may have changed during the program operation. In this context, ‘impact’ means the difference between the situation of program participants after their participation and the situation in which they would have been if they have not participated in the program. While the former is a real situation, the latter is hypothetical one. There is a growing literature in program evaluation1 that has designed methods for the estimation of impacts in different settings. Instead of finding the effect of a program in individuals, researchers try to obtain the average impact of the program by comparing a group of individuals who participated in the program with another similar group which was not exposed to the program. It is often difficult to find two groups with very similar characteristics, but only one that participated in the program under analysis. If there is not a good comparison group, differences between the control group and the treatment group can be attributable to pre-existing differences (selection bias) and the impact of the program. 1 See Gertler et al (2016) for an introduction to the topic of impact evaluation and its practice. See Angrist and Pischke (2009) for more details on the econometric methodologies commonly used. 6 (For discussion purposes only- Maybe subject to further review) ) The estimation of the impacts of a program is important, but it is not enough to evaluate its effectiveness. We also need to measure this impact and contrast it to the costs. Knowing the benefit-cost ratio or return on investment of a program allows comparisons with different projects which helps stakeholders to make decisions regarding the programs to invest. Ideally, an evaluation would consider all the costs and social benefits of a program both on the program participants but also on the agents in the rest of society that are indirectly affected by the program through externalities. That is often difficult to determine because programs generally have multiple objectives and it is challenging to convert many of the outcomes that measure the success of those objectives into a monetary measure. When there are many different impacts to be estimated, evaluators have the alternative to estimate the willingness to pay of program beneficiaries by asking them directly. However, participants tend to overestimate the price they would be actually willing to pay to have access to a program and some with low valuations tend to report zero as willingness to pay. While asking participants their willingness to pay for a program (or using any other instrument that measures more broadly the success of a program) can be a good idea, there are a few reasons why measuring more specific impacts is desirable. First, often stakeholder want to know if the program achieved its success (or satisfaction) through their designed means as opposed to any other unknown way. Second, measuring impacts on specific outcomes is often simpler to learn from participants, either by observing their behavior or by asking them directly. In the evaluation of CEMBA/MPA, we choose specific outcomes of interest, and follow common practice in evaluation literature to define impact as the difference between the changes in outcomes of treatment and control groups over a time period. Although there should be pre-existing differences between graduates and current student, we can use quasi-experimental methods that attempt to offset problems of selection bias. 3. CEMBA/MPA The Commonwealth Executive Master of Business Administration and Public Administration (CEMBA/MPA) Programs operate in the various partner universities2. Students are usually working 2 CEMBA/MPA operates in Allama Iqbal Open University (Pakistan), Bangladesh Open University, Botswana College of Distance and Open Learning (BOCODOL), Kwame Nkrumah University of Science and Technology (Ghana), Open University of Mauritius, National Open University of Nigeria, Open University of Sri Lanka, University College of the Caribbean (Jamaica), University College of the Cayman Islands, University of Guyana, and Wawasan Open University (Malaysia). 7 (For discussion purposes only- Maybe subject to further review) ) professionals and take the course part-time while working. The standard program duration is two years, but students have up to five years to complete it. In order to graduate, students need to complete 90 credits, or 15 courses. The programme is offered in online and distance learning modes. This evaluation includes students and graduates of the partner Allama Iqbal Open University (AIOU). The table below reports the numbers of graduates per year at AIOU. Table 1 - Number of students who graduated: 2004-2012 year CEMBA CEMPA 2004 36 0 2005 168 33 2006 346 113 2007 583 85 2008 877 72 2009 1148 60 2010 1430 92 2011 1425 96 2012 1273 77 Total 7286 628 4. Methodology Our empirical strategy for the estimation of impacts is to use the differences-in-differences model which compares outcomes before and after graduation of CEMBA/MPA for treatment and control groups. Treatment group is composed by alumni of CEMBA/MPA who graduated between 2012 and 2017, and the control (comparison) group is composed by students who have not graduated by end of 2017. Individuals in both groups (“graduates” and “still in school”) had to go through and pass an admission process3 of CEMBA/MPA, which guarantees some level of similarity between these two groups. Additionally, the control and treated individuals have a similar profile in terms of ambitions, education, motivation given that both groups chose to enroll in CEMBA/MPA. A challenge to estimate impacts of education programs or job training programmes is that, typically, the control group is formed by individuals who did not participate in the programme/job training 3 The admission criteria include the following: at least a second class Bachelor's degree from a recognized university; a minimum of two years of relevant post-qualification experience; a good working knowledge of the English language at postgraduate level; satisfactory score on the CEMBA/MPA/CEMPA Admission Test (general awareness, English language, quantitative aptitude, and reasoning); some Partner Universities require citizens to be resident in the country. 8 (For discussion purposes only- Maybe subject to further review) ) course. In that case, program participants that self-select into a programme would necessarily be more motivated than the control individuals. That motivation (which is not observed by the evaluator) could be what, in fact, affects the outcomes, instead of the actual programme intervention. The inability to distinguish those two sources of impact can bias estimation results. We avoid that problem by choosing a control group formed by current students of CEMBA/MPA with likely similar levels of motivation as former students. Since the data collected refers to information before and after the program for treated and control groups, by using this methodology we would eliminate any time-invariant selection bias due to unobserved heterogeneity between treatment and control groups. Thus, we rely on the assumption of parallel trends, that is, the outcomes for control and treatment groups would follow the same trend over time in the absence of the program. The difference-in-differences approach was an appealing methodology in this context as this evaluation was designed and implemented a posteriori, after CEMBA/MPA was operating for multiple years. The main advantage of this methodology is that it does not requires that control and treatment groups to have the similar pre-existing observable characteristics. Instead, what is needed is that the average change in outcomes from 2012 to 2017 of control and treatment groups would be the same have the treated not graduated from CEMBA/MPA. Thus, the average outcomes do not have to be at the same level at any point in time for this assumption to be met. While we cannot show evidence of support of this assumption4, the model we use controls for any individual time-invariant characteristic that affects income and also controls for tenure and work experience (time-varying characteristics). Let i indexes individuals and t indexes year. The impact of the CEMBA/MPA can be estimated with difference-in-differences expressed by the equation below: Where Yi is the outcome of 𝑌𝑌i𝑖𝑖n𝑖𝑖te=re𝛽𝛽s1t,𝐺𝐺 G𝐺𝐺R𝐺𝐺A𝐺𝐺D𝑖𝑖𝑖𝑖i +is 𝜃𝜃a𝑖𝑖n+ in𝐵𝐵d𝐵𝐵ic𝑖𝑖a𝑖𝑖t+or 𝜀𝜀f𝑖𝑖o𝑖𝑖r whether the individual i graduated from CEMBA/MPA or not at time t, is the individual fixed effect, X is a vector of individual time- it varying characteristics and εit is the st a𝜃𝜃t𝑖𝑖istical error. In this model, β1 represents the impact of graduating from CEMBA/MPA on the outcome variable. We include covariates (in the vector X ) which correspond it to characteristics that should not be affected by the program5. The standard errors are estimated with Eicker- White Hetero-robust estimator. 4 Typically, researchers report a chart with the evolution of the average outcomes for control and treatment group prior to intervention suggesting parallel paths. We do not have enough data on outcomes before the intervention to make this chart and it would potentially not be a suitable way to support the parallel trends assumption, as many of the control group individuals would likely still be in school and therefore out of the labour force in the years preceding 2012. 5 We have different model specifications with and without the time-varying covariates: years of work experience and years of tenure with current employer. 9 (For discussion purposes only- Maybe subject to further review) ) In this evaluation, we narrowed the outcomes of interest to: earnings, wages (earnings per hour), income, number of hours of work, entrepreneurship, occupation change/promotion. 5. Data Our population of interest for this evaluation is composed of all students and graduates from CEMPA/MPA at AIOU. A team from AIOU selected and surveyed a sample of graduates and students for our data collection. The interviews were done over the phone and in person in the period from November 2017 to January 2018. The questionnaire used in the survey includes questions on the outcome variables for the years of 2012 and 2017; demographic characteristics; work characteristics and educational characteristics. While it is preferable not to ask retrospective questions to avoid recall bias, as long as the recall bias for the control and treatment groups are the same, our impact estimates remain unbiased. 6. Descriptive Statistics This section reports the descriptive statistics of our sample to better understand the population of participants of CEMPA/MPA. We show some statistics by group: still in school (control) and graduated (treatment) to highlight relevant pre-existing differences in their characteristics. Demographic characteristics Our sample is composed with 193 individuals still in school and 148 who have graduated, totaling 341 observations, most of them from CEMBA program. Table 2 - Number of observations by program Program Still in school Graduated Total CEMBA 171 139 310 CEMPA 22 9 31 Total 193 148 341 While most of the control group individuals started their studies at CEMBA/MPA between years 2014-17, the ones from treatment group started between 2009-14. Those different starting dates meant that by 2012 no one in the sample had graduated, and by 2017 all individuals in the treatment had graduated but none from the control group. 10