The University of Maine DigitalCommons@UMaine Institutional Documents Publications 5-2014 ARI External Review 2014 Glenn Page Andrew Staroscik Follow this and additional works at:https://digitalcommons.library.umaine.edu/ari_docs Repository Citation Page, Glenn and Staroscik, Andrew, "ARI External Review 2014" (2014).Institutional Documents. 2. https://digitalcommons.library.umaine.edu/ari_docs/2 This Report is brought to you for free and open access by DigitalCommons@UMaine. It has been accepted for inclusion in Institutional Documents by an authorized administrator of DigitalCommons@UMaine. For more information, please [email protected]. UNIVERSITY OF MAINE AQUACULTURE RESEARCH INSTITUTE D EXTERNAL ASSESSMENT R A F prepared by G p , a S lenn aGe ndrew taroScik T May 2014 JHU@ e | 1101 e 33 S c310 | b md 21218 aStern rd treet altimore ACKNOWLEDGMENTS We acknowledge the many contributions of all those who participated in the process of this external assessment which featured five site visits, multiple phone calls, emails and assorted connections across a wide range of colleagues in Maine. The kindness, hospitality, honesty and warmth displayed by all serves as the foDundation for this document and hopefully builds collaborative energy to help further develop the Aquaculture Research Institute at the University of Maine. Specifically, and in alphabetical order, we acknowledge the involvement of Paul Anderson, Chris Bartlett, Brian Beal, Martha Bentley, Tim Bowden, Susan Brawley, Ian Bricknell, Nick Brown, Carrie Byron, Fai Chai, Dick Clime, Barry Costa-Pierce, Paul Dobbins, Tali Durant, Chris Feurt, Jeri Fox, Vera Francis, Alan Furth, Cem Giray, Heather Hamlin, Soren Hansen, Carol Kim, Anne Langston, Jon Lewis, Laura Lindenfeld, Catherine MarinR, Bob Martin, Jim McKenna, Bill Mook, Matt Moretti, Dana Morse, Marcy Nelson, Carter Newell, Hemant Pendse, Mary Jane Perry, Lisa Phelps, Mike Pietrack, Mike Pivirotto, Paul Rawson, Sarah Redmond, Diantha Robinson, Kristine Schuman, Fred Servello, Denise Skonberg, Sarah Sparks, Mario Teisl, Dave Townsend, Jake Ward, Theo Willis, Laura Wilson, and Bill Wolters. The concepts and methods described in this document have evolved over many years and benefited from the ideas, experience and wisdom of many people, from scientists to spiritual leaders, from policy makers A to practitioners. This document is a product of continued learning, based upon the art of convening and listening blended with the practice of the ecosystem approach. We have integrated methods and lessons learned from the fields of developmental evaluation, organizational development, assessment of R&D, innovations in interdisciplinary scholarship, needs assessment, international development as well as com- plexity concepts drawn from ecosystem science. Because the methods are a composite of elements from a Cover Photo by Glenn Page wide range of disciplines, they are experimental intended to match theF context and capacity of the situa- Saltwater Farms also the tion. Finally, we would like to thank the tireless effort of the ARI External Assessment Steering Commit- location of the first tee of Sebastian Belle, Chris Davis, Jaimie Logan, Joe Migliaccio, Carter Newell and Chris Vonderweidt aquaculture lease granted who provided hours of support, review, insightful comments, Maine humor and above all a commitment in the state of Maine to to the growth and development of ARI. Ed Meyers in 1973 T Commissioned by: Prepared by: ACRONYM LIST AFRI Agriculture and Food Research Initiative ARC Aquaculture Research Center ARS Agricultural Research Service D ARI Aquaculture Research Institute AVS Animal and Veterinary Sciences BSL Biosafety Level CCAR Center for Cooperative Aquaculture Research CCLC Cobscook Community Learning Center DMC Darling Marine Center R DRC Democratic Republic of the Congo EPSCoR Experimental Program to Stimulate Competitive Research FISHLab Fishery Innovation, Sustainability & Health Lab MAA Maine Aquaculture Association MAIC Maine Aquaculture Innovation Center A MCL Marine Culture Laboratory MEIF Maine Economic Improvement Fund MTI Maine Technology Institute NIFA National Institute of Food and Agriculture NSF National Science Foundation F PI Primary Investigator R&D Research and Development RFP Request for Proposal ROI Return on Investments SMS School of Marine Science T UM University of Maine UMaine University of Maine USDA US Department of Agriculture VPR Vice President of Research TABLE OF CONTENTS D Introduction to External Assessment of ARI ������������������������������������������������������3 Context: ARI as the Unit of Analysis �������������������������������������������������������������������9 Aquaculture Research Activity Prior to the Formation of ARI ���������������9 R The Founding of ARI: Strategy as Intended ���������������������������������������9 ARI in a Nested System Context �������������������������������������������������������10 ARI - Current Expression �������������������������������������������������������������������13 ALearning by Doing ���������������������������������������������������������������������������������������������19 Brief Analysis of Eras Leading to ARI ������������������������������������������������ 20 What was the Intended Strategy for ARI Based Upon the ARI Intent to Plan ��������������������������������������������������������������������������������������������� 22 What Aspects were Implemented as Planned Becoming Realized SFtrategy ����������������������������������������������������������������������������������������24 What Planned Strategies were Unrealized �����������������������������������������28 What (Unplanned and Emergent) Strategies were Picked Up Along and Became Part of Realized Strategy ����������������������������������������30 What Has Been the Learning Over Six Years of Implementation �������33 T D Recommendations ��������������������������������������������������������������������������������������������39 ARI To Clarify Purpose, Strategic Direction and Metrics ��������������������40 ARI to Ensure that University Leadership Review, Adjust if Necessary, and Affirm Chosen Scenario from Recommendation #1 ��������������44 Determine Strategic Purpose and Scope of the ARI Research Network R ����������������������������������������������������������������������������������������������������46 Determine ARI’s Role as a Major Research and Development Program ����������������������������������������������������������������������������������������������������48 Commit to Ongoing Organizational Development, Adaptive Learning and Routine Reflection ����������������������������������������������������������������50 APPENDIX A: Literature Cited ����������������A����������������������������������������������������������53 APPENDIX B: Interview Schedule ���������������������������������������������������������������������54 APPENDIX C: DRAFT Timeline of Aquaculture in the State of Maine ���������������58 F T D R A F T “However beautiful the strategy, you should occasionally look at the results” 2 Photo by Dana Morse Maine cultured scallops being measured as part of a stocking SIR WINSTON CHURCHILL 1874-1965 density growth rate experiment Section 1 Introduction to the External Assessment of ARI D This is a creation story! The Aquaculture Research Institute (ARI) of the University of Maine (UMaine) was created in April 2008, when the Board of Trustees of the University of Maine Sys- tem approved the ARI Intent to Plan, formally initiating ARI and adding aquaculture as a central theme supported by the University alongside the fourteen other organized research units. See Table 1 for the list of the othRer organized research units. Like all good creation stories, there is a beginning, middle and end. In the case of ARI, the ARI Intent to Plan document is dated 2008, but the Institute did not formally start until July 1st 2009, a full year later after the ARI Intent to TABLE 1 Organized research units at University of Maine Plan was approved. • Advanced Structures and Composites Center • Maine Center for Research in STEM Education The time frame for this external assessment begins with approval of the ARI Intent to Plan, dated (the Maine RISE Center) April 2008 to April 2014. For the purpose of this external assessment, this time period marks the A• Center on Aging end of the first generation and the transition to the start of the second generation with the recent • Center for Community Inclusion and Disability Studies appointment of a new director and potential for new funding through NSF/EPSCoR that could dramatically change the course and direction of ARI. • Center for Research on Sustainable Forests • Climate Change Institute The ARI Intent to Plan requires routine external assessment. Specifically, “the Institute Director • Department of Industrial Cooperation will meet with the advisory group… at least twice a year. During the first six months of its exis- F • Forest Bioproducts Research Institute tence, the Institute will develop a strategic plan that will include specific evaluation criteria… after four years and every four years thereafter, an external review panel… will evaluate the institute” • Laboratory for Surface Science and Technology (University of Maine, ARI Intent to Plan, 2008). Despite the requirement for a strategic plan, • Margaret Chase Smith Policy Center one was never attempted and therefore no metrics or further evaluative criteria were developed. • Maine Sea Grant • National Center for Geographic Information and Analysis Without a formal strategic plan or further evaluative criteria, the external assessment focused on T the strategy as defined by the ARI Intent to Plan when considering what strategy was actually • Sen. George Mitchell Center for Environmental & realized. As in many research endeavors, implementation is a combination of deliberate and Watershed Research unplanned processes to generate a strategy that is realized. Often the strategies for developing 3 research are more emergent, as faculty and staff respond to funding opportunities that namic nature of the context. Developmental evaluation is used in broad contexts such align with their research interests and are accelerated by opportunities for collaboration as humanitarian and natural disaster relief and major socio-ecological systems change, as within and across sectors. Innovation can be quite high. Thus, a developmental evalu- well as more narrow contexts such as ongoing program development, adapting a model ation approach, as opposed to a more typical summative evaluation, was a better match to a new and dynamic context or even pre-formative model research and development for the context of ARI because the purpose supports the development of innovations to try and figure out what type of model may work for further formative and summative and adaptation of programs in dynamic environments (Patton 2011). testing (Patton 2011). The developmental evaluation focus for the external assessment D of ARI was on the implementation of the ARI strategy as defined in the ARI Intent to While more traditional accountability-focused evaluation is used for the purpose of Plan. Used in this case, the word strategy is not just the rhetoric that defines the target external assessment of academic programs, the SustainaMetrix team determined that outcomes for ARI. Strategy, in this context, is what ARI does. Organization theorist the “fit” of a developmental evaluation was better because of the dynamic nature of the Herbert Simon also used strategy in this context as “the series of decisions which deter- aquaculture research field, the uncertain future of aquaculture development in the US mines behavior over some stretch of time” (Simon, 1957). Building upon this, Henry in general and Maine specifically, and the complex context of a university system within Mintzberg, distinguishes between intended strategy and realized strategy, as he found which ARI operates. Therefore, ARI is not a fixed model that can be tested for merit that there was no such thing as a beautifully controlled process in which intentions lead and worth that is the basis of a sumRmative form of evaluation (Scriven 1967). Rather, to plans which are implemented as planned concluding with the full realization of in- there is no clear way to prioritize from a widening range of aquaculture research prob- tended results (Mintzberg 2007). Instead, as the graphic in Figure 1 illustrates, realized lems and no certain way forward with multiple pathways possible; the need for innova- strategy (what actually gets done) begins as intended strategy (ARI Intent to Plan) but tion is high and exploration and experimentation is paramount. Developmental evalua- not all of what was intended is realized. Some aspects go unrealized or are dropped, and tion was determined to be a better fit because it brings the processes of asking evaluative what remains is deliberate strategy that combines with emergent strategy (what you pick questions, applying evaluation logic, and gathering and reporting evaluation data to up along the way that was not envisioned at planning) to create realized strategy (Minzt- support research strategy development, thus encouraging reAsearch innovation and adap- berg 2007, Patton 2011). tation. Ideally, over many years, the evaluator works collaboratively with the leadership team to conceptualize, design and test strategic approaches in an on-going process of Thus, strategy is the frame for the external assessment with a strong focus on what has adaptation, intentional change, and further development. The primary function of the been learned over six years of implementation about the relationship among intended, evaluator is to elucidate the innovation and adaptation processes, track implications and Unrealized results, and help to facilitate improved decision-making (Patton 2011). Strategy F Intended (strategy dropped) Commissioned by the Vice President of Research at the University of Maine (VPR), Strategy it was agreed that the external assessment team set up their process, collect the data, (planning) conduct the analysis and prepare a report within 3 months. Through negotiations, it Deliberate was agreed that focus would be on “high-level” issues, embrace systems thinking, and Strategy provide a manageable set of recommendations that could be implemented by the VPR, Realized T the new ARI Director, the ARI Advisory Board, and staff and faculty associated with Strategy ARI. Emergent Developmental evaluation was accepted as a pilot demonstration of a methodological Strategy approach that could be applied to ARI in the future because it embraces conditions of FIGURE 1 Strategy as the Focus of the ARI External Assessment 4 complexity. It has many potential expressions that can be applied to best match the dy- (Based on Mintzberg 2007, Patton 2011) OCT NOV DEC JAN FEB MAR APR MAY Planning On & Offsite Evaluation Reporting D Contract Executed Site Visits 1 2 3 4 5 External Steering 1 2 3 4 5 Committee Meetings R FIGURE 2 Timeline of ARI External Assessment Process deliberate, unrealized and emergent strategy to create what ARI actually accomplished. Given the fact that ARI fits within a complex university structure and conducts high-quality collaborative relationships with other researchers and members of industry, a set of other methods were also employed to better A understand the system within which ARI fits, the complex, uncertain and dynamic context within which TABLE 2 Numerical Result of the ARI External Assessment ARI operates, as well as an appreciation of the decisions that have been made and paths taken. • 61 In-person interview • 5 External assessment steering committee meetings A set of mixed methods, both qualitative and quantitative, were chosen to best match the purpose, timing and cost constraints and to better understand how ARI aligns with the many key stakeholders from: • 40+ Document reviews F• 20 Survey responses from staff, faculty, advisory committee, and • University of Maine System external steering committee • Maine’s Economic Development Community • 200+ Timeline entries • Aquaculture Industry • Resource Management Agencies • Aquaculture Research Network T • Working Waterfront Community These methods included the use of an external steering committee to provide input, help steer the process and contribute to the development of a final set of key messages delivered through a Powerpoint pre- sentation to the ARI Advisory Board. Additional methods were also used that were consistent with the 5 purpose and intention of the external assessment.