EVALUATION OF THE COMPETITIVENESS OF THE ARMENIA PRIVATE SECTOR (CAPS) PROJECT USAID/ARMENIA SEPTEMBER 2010 This publication was produced for review by the United States Agency for International Development. It was prepared by Donovan Rudisuhle of Social Impact, Inc. and Anna Sedrakyan of Management Systems International. EVALUATION OF THE COMPETITIVENESS OF THE ARMENIA PRIVATE SECTOR (CAPS) PROJECT USAID/ARMENIA DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document was submitted by Social Impact, Inc., with Management Systems International, to the United States Agency for International Development under: USAID Contract No. RAN-I-00-09-00019, Task Order No. AID-111-TO-10-00002 Evaluation Services Indefinite Quantity Contract Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 [email protected] Acronyms CAPS Competitiveness of the Armenia Private Sector EIF Enterprise Incubator Foundation GMP Good Manufacturing Practices GOAM Government of Armenia MOE Ministry of Economy MPI Union of Medicine Producers and Importers NSS National Statistical Service TAIEX Technical Assistance and Information Exchange USAID United States Agency for International Development YSMU Yerevan State Medical University Table of Contents Acronyms ........................................................................................................................................ 3 Executive Summary ........................................................................................................................ 1 I. Description of the Project ....................................................................................................... 1 II. Purpose of the Evaluation ....................................................................................................... 1 III. Evaluation Methodology ...................................................................................................... 2 IV. Overall Findings and Conclusions ....................................................................................... 3 A. Sectoral Findings, Conclusions and Recommendations .................................................. 5 1. Information Technology Cluster .................................................................................. 5 Conclusions: ............................................................................................................................ 6 Recommendations: .................................................................................................................. 7 2. Pharmaceutical Cluster ................................................................................................. 7 Conclusions: ............................................................................................................................ 9 Recommendations: .................................................................................................................. 9 3. Tourism Cluster .......................................................................................................... 10 Conclusions: .......................................................................................................................... 11 Recommendations: ................................................................................................................ 12 V. Summary of Recommendations ............................................................................................ 12 VI. Lessons Learned................................................................................................................. 13 ANNEXES .................................................................................................................................... 14 Executive Summary The CAPS project was successful in meeting its stated objectives. Interviewees in all three of the industry clusters voiced favorable opinions regarding their interactions with the project. There was virtually universal agreement among interviewees and focus group participants that their respective industry cluster had received valuable benefits from CAPS’ activities and that the project was a positive factor in fostering economic growth and employment opportunities in Armenia. The principal vehicles used by CAPS were training, funding of international experts, sponsorship of conferences and trade association events, assistance with regulatory and legislative reform, identification of foreign markets and potential business partners, development of promotional literature and materials and creation of industry cross-linkages. The vast majority of persons contacted stated that they would like to see a continuation CAPS’ activities in a future project based on the same successful model. Many of them offered valuable suggestions to improve the effectiveness of any future activities under consideration by USAID. The evaluation team found that there was balanced gender participation in all aspects of the activities conducted throughout the three clusters. I. Description of the Project The CAPS project was launched in September 2005 under a $14.1 million grant from USAID for the purpose of assisting Armenia in its efforts to develop its key industries with the specific objective of increasing employment, promoting exports and growing local businesses by promoting productivity, competitiveness and policy reform. The mechanism used by CAPS is a cluster approach that incorporates all of the major stakeholders and participants within a specific industry. This includes private companies, individual entrepreneurs, government agencies and regulatory authorities, educational institutions, trade associations and civil society organizations. Information technology, pharmaceuticals and tourism are the three clusters that were identified as targets of CAPS financial and technical assistance. CAPS’ methodology is a demand-driven approach that stresses job creation, cluster-based development, local ownership of strategies, action and results and market-oriented themes. The principal vehicles used by CAPS to carry out its activities are educational seminars, facilitation of international technical assistance, sponsorship of trade fairs, identification of foreign markets and partners, assistance with legislative reform efforts and preparation of marketing and promotional literature. The project strives to demonstrate results and achieve sustainability by: • Building sustainable institutional capacity • Transferring the organization of initiatives to counterparts • Building public awareness on key competitiveness issues • Achieving clear policy reform results The CAPS project is scheduled to terminate in February 2011. II. Purpose of the Evaluation The primary purpose of this evaluation was to provide USAID/Armenia with an objective external economic impact assessment of this important program that will soon be concluding and to apply the experiences and lessons learned to the next generation of economic growth 1 initiatives. The evaluation endeavored to measure and analyze the impact of CAPS project activities through an “effectiveness assessment” lens. That is, how well did the project activities catalyze economic growth and contribute to productivity gains and workforce development in the three selected sectors? III. Evaluation Methodology The objective of the evaluation was to provide USAID with defensible conclusions and actionable recommendations for the purpose of helping USAID and its Armenian partners establish priorities and plan their activities for any future interventions in these designated sectors. To accomplish this, the evaluation team met with a variety of stakeholders for each of the designated clusters. With the assistance of CAPS officials, the consultants met with cluster coordinators, trade associations, private companies, entrepreneurs, government agencies, educators, students, civil society organizations and other beneficiaries of the project. In addition, focus groups were held with groups of stakeholders for the purpose of obtaining their opinions of project effectiveness and to provide a forum for them to voice their recommendations regarding ways to improve the project in any future extension or redesign by USAID. The consultants conducted their interviews in Yerevan, Gyumri and Vayk and focus groups were also held at all three of these locations. In accordance with its standard qualitative information collection technique, the team convened three focus groups of business representatives and concerned individuals in Yerevan, one in Gyumri and one in Vayk. These groups were composed of individuals such as business owners, startup entrepreneurs, trainees, university professors and other direct beneficiaries of CAPS assistance. These groups averaged about 6 to 10 individuals who were gathered together at each location for a series of discussions lasting 1 to 2 hours where they were requested to voice their experiences regarding working and collaborating with CAPS. In addition, they were asked about particular sector needs and the degree of success they achieved using CAPS’ various approaches to project identification and implementation In addition to the above, concerned individuals and local government representatives at project sites were interviewed using an open semi-structured questionnaire format. The topics and the wording of the questions were very flexible to allow for the diverse nature of the individuals that met with the team. In summary, within the very short time available for this impact assessment, the team conducted dialogues with as varied a group of participants as is possible. The study also referred to existing documentation to ensure that the team obtained the best possible understanding of all aspects of the interventions. Assisted by the CAPS project staff, the team referred to various analytical reports prepared by third parties as secondary data to assess the catalytic effect of the project and the corresponding productivity increase. In particular, the team used following reports: • Armenian Information Technology Sector, Software and Services. 2009 Industry Report, Ministry of Economy of the RA, USAID/CAPS, EIF • Armenian Pharmaceutical Industry. Investment Handbook 2008, USAID/CAPS, MPI Union, Armenian Development Agency 2 • Armenian Tourism Industry. Investment Handbook 2008, USAID/CAPS, Armenian Tourism Development Agency, Armenian Development Agency • Jermuk Comprehensive Development Plan, 2008, Ministry of Economy, USAID/CAPS • Information Technology Growth Model, 2007, EIF, Economy and Values Research Center • Tourism Strategy For Armenia, 2007-2030, Ministry of Economy, USAID/CAPS, Economy and Value Research Center, Armenia 2020 • National Competitiveness Report of Armenia, 2009, Economy and Values Research Center" • A number of other published sources provided by various government and analytic entities. IV. Overall Findings and Conclusions Overall, the CAPS-assisted clusters demonstrated varying degrees of growth over the period from 2005 to 2010. The IT Cluster (excluding telecom) saw a 17% increase from 2008 to 2009 and there was a corresponding 3% increase in the Tourism Cluster, this in spite of the global economic crisis. (the increase counts the overall turnover) for the same period. The pharmaceutical sector registered a 3% decline overall in 2009, although this figure includes data on pharmacies. Pharmaceutical production, the focus of CAPS interventions in the sector, experienced an average growth of 7% last year. The increase was primarily a consequence of increased domestic sales, with the export of local pharmaceutical products increasing by only 4.7%, according to the National Statistical Service (NSS). However, the pharmaceutical industry has not yet seen any CAPS-related growth, as CAPS has only intervened recently, and its efforts have been focused on obtaining GMP certification for Armenian laboratories and manufacturers. When this goal is achieved, Armenian firms should be in a position to such to actually increase their export sales by having unfettered access to the EU market. According to CAPS semi-annual and annual reports to USAID, in 2009 3,500 new jobs were created, just short of the target of 3,800. This figure includes jobs created directly or indirectly as a result of CAPS activity. Sales and revenue targets were surpassed and overall productivity increased by 8%, exceeding the target of 6%. Armenian firms participating in the CAPS clusters have made significant progress in adopting international best practices, with 15 of them adopting them, while only 4 were envisioned in the target. The project also supported the adoption of six pro-business reforms which benefited the CAPS cluster, surpassing the target of four. The structures and processes established in the IT, Tourism and Pharmaceutical Clusters have resulted in improved cooperation and communication among stakeholders. Interviewees from both the private sector and the project itself stated that they had good cooperation with those governmental agencies with which they routinely interacted. However, numerous stakeholders mentioned that there was an occasional failure of the government agencies to collaborate effectively with each other. 3 The Head of the Drug Agency expressed frustration in dealing with the Ministry of Health, and so did the CAPS project staff. In particular, the project staff mentioned that there was absolutely no one in the Ministry of Health responsible for the development and follow-up of GMP regulations. It was only after the Ministry of Economy took over the responsibility for developing these regulations and presenting them to the Government for approval that the Ministry of Health finally got involved in the process. The lack of capacity and poor motivation at the lower levels of the ministries was cited as the principal factor that contributed to time-consuming and burdensome experiences in dealing with government organizations. There is no reliable way of establishing a specific percentage of growth attributable to CAPS’ intervention, but stakeholders universally stated that the favorable outcomes they experienced would not have been possible in the absence of CAPS’ support. In terms of sustainability and cost effectiveness, it can generally be concluded from the interviews that project activities were relevant to the current industry needs, and beneficiaries were optimistic about their long-term sustainability. However, it is not possible to make a statement regarding cost-effectiveness because the contractor's records do not disaggregate expenditures in a way that they can be easily matched to project activities. The information is available, but a time-consuming effort would be required in order to analyze and organize the numbers in a way that they could be associated with specific project activities. Strengths of the CAPS Project – All Three Sectors: • CAPS strengthened the foundation for IT sector development and created linkages with the Tourism and Pharmaceutical Clusters. • CAPS promoted start-up support, capacity building and workforce development activities and encouraged co-funding and in-kind assistance from the private sector. • New business associations were created and existing associations were strengthened through CAPS’ intervention. • Targeted trainings and education were made available to a wide variety of business stakeholders. • CAPS assistance helped to improve the marketing, branding and packaging capabilities of the firms that were assisted. • CAPS supported improvements in policy and regulatory environment. • International certification programs were introduced in all three industries. Weaknesses of the CAPS Project: • CAPS’ scope of work was too broad and too many small activities were dispersed across the target sectors, reducing the focus of efforts. These activities were not connected to each other, and in some instances even contradicted each another. • This excessively broad scope led to uncertainty in planning the project activities at the inception, thus impeding the effective startup of the project. When the activities began to reach a point where they were actually having a demonstrable positive effect, the project was beginning to wind down its five and a half year life. 4 • The future sustainability of the newly created business associations that received CAPS funding is uncertain, particularly those in the tourism industry. This is attributable in part by a failure to attract a majority of the potential members, who may not perceive a benefit in membership, or who are unwilling to pay the membership fees. Key challenges ahead: • One recurring comment that was voiced to the evaluators was that a continuing effort is required to change the mindset of managers, professionals and the public in general in terms of accepting new ways of thinking. In particular, they stressed the importance of educating and persuading managers at all levels that they can improve the efficiency of their businesses and increase their revenues by adopting modern IT practices. • Along the same lines, stakeholders in the pharmaceutical sector said that it is necessary to educate doctors, pharmacists and patients about the fact that locally produced drugs are of equal quality and effectiveness as imported products. • Corruption was also mentioned frequently as an impediment to business efficiency and economic growth. A. Sectoral Findings, Conclusions and Recommendations 1. Information Technology Cluster Findings: Information technology has become one of the driving sectors of Armenia’s economy, contributing to innovation and productivity growth in the country and generating substantial export sales. According to the Enterprise Incubator Foundation (EIF) survey 2009 on Armenian Information Technology Sector Software and Services, the IT sector has registered an average annual growth in revenues of 24.2% from 2006 to 2009. In absolute figures, IT sector revenues reached $129.9 million in 2009, an increase from $84.2 million in 2006 and $111.3 million in 2008. According to the 2009 State of IT Industry Report, prepared jointly by the EIF and CAPS based on the figures provided by the National Statistical Service, the IT sector accounts for 1.3% of GDP in Armenia, which is comparable with that of other leadings economies. By comparison, the IT industry share in India’s GDP is 1.4% and in Germany’s it is 1.3%. From 2003 to 2009, the IT industry’s contribution to total exports rose from 3.6% to 5.6%, and domestic sales and revenues increased from $42 million in 2008 to $59 million in 2009. A wide range of CAPS activities contributed to this growth, both on the export and domestic sales side. There were a number of country and industry promotion initiatives, such as supporting companies to participate in global IT fairs and conferences which assisted Armenian firms to find new partners and sign sales and services contracts with them. Issues of industry confidentiality precluded obtaining quantitative data, but numerous instances of these partnerships and agreements were said to have occurred as a consequence of CAPS’ involvement. The promotion of IT solutions in other sectors, particularly in tourism, contributed to increased domestic sales of IT companies, as local firms began to adopt new technologies for the first time. 5