WWeesstteerrnn MMiicchhiiggaann UUnniivveerrssiittyy SScchhoollaarrWWoorrkkss aatt WWMMUU Dissertations Graduate College 6-2013 MMooddiifificcaattiioonn aanndd AAddaappttaattiioonn ooff tthhee PPrrooggrraamm EEvvaalluuaattiioonn SSttaannddaarrddss iinn SSaauuddii AArraabbiiaa Mohammed Alyami Western Michigan University, [email protected] Follow this and additional works at: https://scholarworks.wmich.edu/dissertations Part of the Educational Assessment, Evaluation, and Research Commons, and the International and Comparative Education Commons RReeccoommmmeennddeedd CCiittaattiioonn Alyami, Mohammed, "Modification and Adaptation of the Program Evaluation Standards in Saudi Arabia" (2013). Dissertations. 157. https://scholarworks.wmich.edu/dissertations/157 This Dissertation-Open Access is brought to you for free and open access by the Graduate College at ScholarWorks at WMU. It has been accepted for inclusion in Dissertations by an authorized administrator of ScholarWorks at WMU. For more information, please contact [email protected]. MODIFICATION AND ADAPTATION OF THE PROGRAM EVALUATION STANDARDS IN SAUDI ARABIA by Mohammed Alyami A dissertation submitted to the Graduate College in partial fulfillment of the requirements for the degree of Doctor of Philosophy Interdisciplinary Ph.D. in Evaluation Western Michigan University June 2013 Doctoral Committee: Chris Coryn, Ph.D., Chair Daniela Schroeter, Ph.D. Lori Wingate, Ph.D. MODIFICATION AND ADAPTATION OF THE PROGRAM EVALUATION STANDARDS IN SAUDI ARABIA Mohammed Alyami, Ph.D. Western Michigan University, 2013 The Joint Committee on Standards for Educational Evaluation’s Program Evaluation Standards is probably the most recognized and applied set of evaluation standards globally. The most recent edition of The Program Evaluation Standards includes five categories and 30 standards. The five categories are Utility, Feasibility, Propriety, Accuracy, and Evaluation Accountability. In recent years, evaluation has grown into a global practice with more than sixty national evaluation associations around the world. Examples include Japan, Malaysia, Sri Lanka, Mongolia, Russia, Brazil, Colombia, Peru, South Africa, Zimbabwe, Niger, and New Zealand. Because countries differ from one another in context and culture, they may need different sets of standards for better execution and results of evaluation practices. However, this is not the case in every country. The cultural adaptation of The Program Evaluation Standards, the growing economy in Saudi Arabia, and evaluation quality are the key issues framing this study. The following research questions are addressed: (1) To what extent do The Program Evaluation Standards fit the Saudi context? (1.1) Which standards are acceptable in their current form without modification? (1.2) Which standards need modification to be acceptable? and (1.3) Which standards are unacceptable, with or without modification? (2)What, if any, aspects of the standards are perceived to be especially problematic for application in the Saudi context? (3)What modifications are necessary to yield a complete set of standards for the Saudi context? The findings suggest that The Program Evaluation Standards needs to be modified for the Saudi context. As a result, a modified set of standards titled “The Saudi Program Evaluation Standards” will be disseminated, including four categories and 28 standards. © 2013 Mohammed Alyami ACKNOWLEDGMENTS First of all, I would like to thank Allah (God) for his blessing and seek his help and guidance; all praise is due to Allah. I would like to humbly dedicate this work to my father, Saeed Bin Tami, and my mother, Refah Bent Mesfer, for their kindness, support, and encouragement throughout all of these years. Having the opportunity to work with my dissertation committee was one of the most professional, but enjoyable experiences of my life. I wish I could write this acknowledgment in Arabic to be able to choose the most appropriate appreciation words to my committee Dr. Chris Coryn, Dr. Daniela Shroeter, and Dr. Lori Wingate. I would like to thank Dr. Wingate for her help, directions, and encouragement. Also, I would like to express my thankfulness to Dr. Shroeter for her exceptional professional and ethical commitment. I was fortunate to have Dr. Coryn as my advisor during the doctorate program, and as my committee chair. All the difficulties that I faced during this hard work were dealt with professionally and smoothly because of his thoughtful guidance and advice. I would also like to expand my appreciativeness to the expert panel that participated voluntarily in the study, and provided me with the information I needed to achieve the final results. ii AcknowledgmentsContinued Last, but not least, to my wife and my daughters I would like to say, “Refah, Salwa, Leanne, Leila, and Celine - you were the source of energy and happiness during this long journey.” Mohammed Alyami iii TABLE OF CONTENTS ACKNOWLEDGMENTS ............................................................................................ ii LIST OF TABLES ........................................................................................................ viii LIST OF FIGURES ...................................................................................................... ix CHAPTER I. INTRODUCTION ........................................................................................... 1 Background and Issues That Frame This Study ..................................... 6 Overview of Saudi Arabian Context ....................................................... 8 Government ................................................................................... 9 Constitution (Shari'ah Law) ........................................................... 9 Language........................................................................................ 9 Gender............................................................................................ 10 Corruption ...................................................................................... 10 Statement of the Problem ........................................................................ 11 Purpose of the Study ............................................................................... 12 Focal Research Questions ....................................................................... 12 Rationale for and Significance of the Study ........................................... 13 Contributions of the Study ...................................................................... 13 Organization of the Dissertation ............................................................. 14 II. LITERATURE REVIEW ................................................................................ 15 Overview ................................................................................................. 15 iv Table of ContentsContinued CHAPTER Relevant Studies ..................................................................................... 16 Germany ........................................................................................ 17 Switzerland .................................................................................... 23 African Continent .......................................................................... 28 South Korea ................................................................................... 35 Australia and New Zealand ........................................................... 36 Bangladesh ..................................................................................... 37 Japan .............................................................................................. 38 Summary ................................................................................................. 39 III. METHODOLOGY .......................................................................................... 41 Research Questions ................................................................................. 41 Design ..................................................................................................... 41 Focus Group................................................................................... 44 Questionnaire ................................................................................. 46 Setting, Sampling, and Subjects ............................................................. 48 Setting ............................................................................................ 48 Sampling ........................................................................................ 48 Subjects .......................................................................................... 49 Instrumentation ....................................................................................... 49 Focus Group................................................................................... 50 v Table of ContentsContinued CHAPTER Phase 1 Questionnaires .................................................................. 50 Phase 2 Questionnaires .................................................................. 51 Procedures ............................................................................................... 51 Expert Recruitment ........................................................................ 51 Focus Group Procedures ................................................................ 53 Data Processing....................................................................................... 55 Data Analysis .......................................................................................... 55 Timeline .................................................................................................. 56 Limitations and Delimitations ................................................................ 56 IV. RESULTS ........................................................................................................ 58 Focus Group Results ............................................................................... 58 Utility Standards ............................................................................ 59 Feasibility Standards ...................................................................... 63 Propriety Standards ........................................................................ 65 Accuracy Standards ....................................................................... 67 Evaluation Accountability Standards ............................................ 68 Modifications to Program Evaluation Standards Based on Focus Group Results ............................................................................... 70 Results of the First-Round Questionnaire ............................................... 72 Additional Modifications to Program Evaluation Standards Based on First-Round Questionnaire Results ......................................... 76 vi