PROCESS EVALUATION: HOW IT WORKS Gary Bess, Ph.D., Michele King, and Pamela L. LeMaster, Ph.D. Abstract: Process evaluation helps us to understand the planning process. This predominantly qualitative approach explains how and why decisions are made and activities undertaken. The focus includes feelings and perceptions of program staff. The evaluator’s ability to interpret and longitudinally summarize the experience of program staff and community members is critical. Techniques discussed include participant observation, content analysis, situational analysis, in-house surveys, and interviews. By combining sources and methods, a fuller picture of the process is revealed. What exactly is process evaluation? Is it really evaluation at all? The answers to these questions may be less straightforward than the questions themselves. Process evaluation, as an emerging area of evaluation research, is generally associated with qualitative research methods, though one might argue that a quantitative approach, as will be discussed, can also yield important insights. We offer this definition of process evaluation developed by the Federal Bureau of Justice Administration:1 Process Evaluation focuses on how a program was implemented and operates. It identifies the procedures undertaken and the decisions made in developing the program. It describes how the program operates, the services it delivers, and the functions it carries out . . . However, by additionally documenting the program’s development and operation, process evaluation assesses reasons for successful or unsuccessful performance, and provides information for potential replication [italics added]. 109 110 VOLUME 11, NUMBER 2 The last sentence in this definition is at the heart of process evaluation's importance for Circles of Care (CoC). Process evaluation is a tool for recording and documenting salient ideas, concerns, activities, administrative and management structures, staffing patterns, products, and resources that emerge during three-year CoC planning grants. Unlike outcome evaluation, which often measures the results of a project’s implementation against its programmatic projections, there are not necessarily a priori assumptions about what the planning process will look like. Furthermore, as discussed in an earlier chapter on the life cycle of the evaluation process, there are stage-specific developmental activities occurring within the program. While the specific context will vary across projects, we may assume that there are common dynamics (e.g., Process, Development and Action Stages) that when understood can frame the experience and be helpful to participants and next generation planners. In essence, process evaluation entails tracing the footsteps that CoC staff, as well as others involved in planning activities, have taken in order to understand the paths that have been traveled, as well as journeys started and later abandoned. This process is akin to the grounded theory approach of qualitative evaluation (Artinian, 1988; Strauss & Corbin, 1990). Process evaluation is an inductive method of theory construction, whereby observation can lead to identifying “strengths and weaknesses in program processes and recommending needed improvements” (Rubin & Babbie, 2001, p. 584). To better understand process evaluation aligned with the qualitative tradition, we borrow from Rubin and Babbie (1993) for an operational definition of qualitative methods: Research methods that emphasize depth of understanding, that attempt to tap the deeper meaning of human experience, and that intend to generate theoretically richer, observations which are not easily reduced to numbers are generally termed qualitative methods. (p. 30). We deduce from this definition the evaluator’s unique role as the tool that synthesizes the “human (collective) experience” of CoC participants. Regardless of methods – participant or direct observation, unstructured or intensive interviewing – it is the evaluator who ultimately classifies, aggregates, or disaggregates themes that emerge as a result of the planning process. As has been discussed elsewhere in this Special Issue, the evaluator’s relationship with the CoC team is an integral part of the evaluation. It is especially paramount with regard to process evaluation, given the relative intimacy of interaction required by some of the data collection techniques. As may be expected, this “at your side” approach can intensify strained or suspicious relationships between the evaluator and program staff. PROCESS EVALUATION: HOW IT WORKS 111 As one CoC program staff member explains: When I think about these terms ‘qualitative research’ and ‘participant observer,’ I feel the abusive history of my people staring me in the face. Intense feelings of anger, hurt, and betrayal all come into play. Being in a fish bowl comes to mind, as do memories of ‘tourists’ who visited the ‘mission,’ which stood on my reservation, and took pictures of the ‘Indian children,’ and made comments like ‘how poor’ and ‘uncivilized’ we were. As I understand the term ‘participant observation,’ I feel insulted. Feelings of betrayal, falsehoods, and sacrilege come to mind. Our culture and our way of processing is who we are as a people. It is all very intimate in nature. In Circles of Care we trusted to open ourselves up, to share ourselves, our culture, and to take the time to know those who were not of our culture. This was a big step and not one taken lightly. Knowing that someone participated as one of us, yet in turn dissects the process, is not being true. Process evaluation thus requires vigilance on the part of the evaluator to respect the trust that has been afforded him or her by American Indian and Alaska Native (AI/AN) program staff. The evaluator’s observations and comments should be made knowing that there are cultural and historic overtones and undercurrents that influence the interpretation of events, as well as the meaning that CoC program staff assign to the process evaluation description. Process evaluation, just like any other form of assessment, requires cultural sensitivity and awareness. It may be that certain techniques (e.g., participant observation) are not appropriate tools for evaluators that enter a program without prior relationships with the CoC program staff. Having addressed at the onset the evaluator’s role in process assessment, and mindful that working relationships will evolve during the life cycle of the project, the evaluator is ready to engage in the process evaluation. There are several conventional evaluation techniques that can be used to discern and describe the CoC planning process itself. They are: participant observation, content analysis, situational analysis, in-house surveys, and interviews. This multi-source approach is consistent with Marcus’ (1988) recommendation that the collection of official documentation should be combined with the input of “key actors.” Strauss and Corbin (1990) also support this approach by advocating for qualitative data collection from a grounded theory perspective. They point to the emergence of a representativeness of concepts, which is to say that themes can be generalized based on the similarities across the phenomena being studied. 112 VOLUME 11, NUMBER 2 With the exception of in-house surveys, these techniques are qualitative in nature, suggesting that Rubin and Babbie’s (1993) definition of qualitative research’s focus on understanding and the deeper meaning of human experience is most apt in the process evaluation domain. In his or her approach to qualitative assessment, the evaluator is interested in understanding the content and meaning of written and oral expressions. One helpful approach is to assess content based on manifest and latent themes (Rubin & Babbie, 1993). Manifest content refers to the frequency that certain words, phrases, or concepts appear in documents and oral expressions, such as recurring themes of specific resource needs and their sources, expressions of feelings (e.g., tired, excited, or fulfilled), categories of persons targeted for involvement as informants, or the kind of technical assistance requested. Latent level analysis entails the evaluator’s overall assessment of the project activities or concerns, the input, its clarity of purpose and direction, and current level of development. Process Evaluation Techniques The following is a discussion of process evaluation techniques that are used by CoC grantees. Participant Observation Though there is a range of participatory roles that evaluators can play that run the gamut from fully immersed and invisible participant to fly on the wall sidelines observer, the common experience of CoC grantees is to have the evaluator in the observer-as-participant role (Gold, 1969). In this capacity, the evaluator’s responsibilities and duties are clearly known to the planning and program staff, and to community members. There is no attempt to disguise the evaluator’s role. Credibility and trust are of utmost importance. Evaluators are present at key planning meetings involving CoC staff and community agencies. They listen at focus group sessions with families and youth, attend Gathering of Native Americans (GONA) events and community picnics, join in progress presentations to sponsoring agency boards of directors, and attend regional and national meetings with other staff members. When evaluators’ roles are among the reasons for their participation, they fulfill these responsibilities by developing surveys, discussing data collection strategies, and reporting results. Regardless, however, of these assigned duties, evaluators also reflect on the content of each event, and attempt to categorize elements into thematic and descriptive domains. One evaluator’s reflections are provided below: PROCESS EVALUATION: HOW IT WORKS 113 The GONA provided important insights and a rich contextual understanding of tribal and community perspectives for participants. Several workgroups were formed during the GONA that were charged with identifying community strengths and needs, and participants provided examples from their personal experiences. The GONA experience, occurring within one year of the project’s initiation, seems to have added new vitality and clarity about the project’s purpose, and has increased support for the initiative among community leaders. While participant observation is a primary source for uncovering themes and obtaining richer understanding of the process’ context, secondary sources, such as content analysis, can be equally as informative. Content Analysis Content analysis refers to a systematic review of written documents produced by CoC staff, volunteers, and community members. Included are planning documents such as timelines, resource lists, and budgets, promotional materials such as flyers, letters to allied agencies and others explaining the initiative, minutes of meetings, proposals for funding and applications for special recognitions, as well as any other documents that capture features of the project. Content analysis focuses on the ideas being communicated. With the evaluator as the instrument for assessing the content of written materials, he or she lists or codes ideas, words, and phrases that capture salient elements of the program. Since the process evaluation has a longitudinal perspective (e.g., what issues, concerns, and strategies characterize the project at a given point in time), it is also necessary to note the temporal sequencing of events and to be clear about the units of analysis, which are the planning team and community members. Maintaining a macro focus is essential for content analysis to be helpful in supporting the process evaluation in that the inquiry pertains to replicable actions and stages, as well as any activities that have not proven to be productive or helpful. One example of content analysis is a review of reports from newsletters produced by Feather River Tribal Health on their sponsorship of community picnics as a tool for community organizing and building awareness of resources for families. The first community picnic was held October 1, 1999. Below is the description of the event in the project’s newsletter: 114 VOLUME 11, NUMBER 2 At our Wellness Retreat (GONA) last June, one message rang loud and clear: community togetherness. In these fast moving times of change, getting together with family, friends, and community members is often difficult to fit into one’s schedule. The true community gatherings have become a thing of the past, becoming something we remember as “when I was younger . . .” Held at Durham Park, with a turn out of approximately 140 people . . . A newsletter report on the second community picnic said: On April 22, 2000, the Circles of Care staff . . . joined forces to put together a community picnic at Palermo Park. Approximately 350 adults and children attended this event . . . The heat from the grill was hot, but the day was cool and feelings of community belonging ran strong . . . Community events such as these are important to building and maintaining relationships within the community. In a time when it is easy to lose touch with family and loved ones, family and friend get-togethers are a perfect way of slowing down and touching base. Aside from the large increase in the number of attendees between events, which in and of itself is instructive about the promotion and draw that such events can achieve, we observe manifest terms such as “community” and “family,” and latent themes characterized by “slowing down” and “touching base.” Values similar to those communicated in the newsletter emerged in other contexts, as well, as at provider meetings and focus group sessions, thus validating their importance to the project. Situational Analysis A technique that has been utilized by some projects to assess the CoC planning process involves periodic meetings of the planning staff facilitated by the evaluator to conduct what is referred to as a SWOT analysis. SWOT is an acronym standing for strengths, weaknesses, opportunities and threats (Barry, 1986). In a SWOT analysis, the following questions are sequentially asked: PROCESS EVALUATION: HOW IT WORKS 115 1. What are our organization’s internal and external strengths? 2. What are our organization’s internal and external weaknesses? 3. What opportunities are present, within our organization and in the environment that surrounds us? 4. What threats exist – either internally or externally – that could adversely affect our efforts? The SWOT assessment is focused on both internal and external dynamics. Thus, the focus is divided between organization concerns, and community and environmental issues. Combined, these factors represent a full spectrum of issues and concerns facing the project at any given point in time. Utilizing this approach it is possible to longitudinally observe changes in staff perceptions as the planning effort evolves. For example, a SWOT assessment conducted at an early stage of a CoC grant reveals strength-based values such as teamwork, ability to network across cultures, and flexibility in program design and use of funds. Weaknesses pertain to misunderstandings about planning among segments of the community, lack of credibility as a planning entity, and concern that internal support is not adequate. Opportunities at the early stage relate to complementary funding streams, new dialogues with public service providers, and descriptive data on resources and service levels that heretofore had not been available. Finally, threats relate to perceptions that CoC will be seen as a temporary project without lasting impact, and concern that public agencies, having low cultural competency in serving AI/ANs, will resent challenges to their methods of intervention. During the third and final year of the planning project, the SWOT reveals a different set of perceptions (see Table 1). Monitoring changes in perception through triangulation of sources, such as in-house survey discussed below, again enriches understanding and strengthens conclusions. In-House Surveys Though we have characterized process evaluation as primarily a qualitative method, there are also opportunities to inform the process evaluation by relying on quantitative tools. For example, a variation of the situational analysis can be performed vis-à-vis an in-house survey, whereby staff and community members are asked to report their awareness of and satisfaction with the project at different points in time. A Likert-type scale can be developed, whereby respondents rate such variables as community awareness, provider cooperation, management support, timeline pace, or adequacy of resources on an ordinal scale ranging from “1”to “5”. This approach can provide input into the planning process from a wider circle of respondents than core staff. It also provides a modicum of 116 VOLUME 11, NUMBER 2 anonymity for respondents, and allows those less reticent to be heard with equal voice. A mean score (average of responses) can be used to determine where the program is weak and strong, and where threats and opportunities are present. Table 1 SWOT Assesment During the Third and Final Planning Year Strengths Weaknesses Opportunities Threats Getting support “Crankiness” – Numerous funding Potential loss of from administration challenge to be opportunities independence now and medical seen as a that project is professional when permanent and not feeling like one service focused. Getting ready for Transition to new Classes available on Growth competition transition (moving) facility – staff is computer training for for limited space in uncertain about Family Resource new building. what changes will Center be made and how they will be affected by them Funding request Long-term funding Family Resource Not enough staff to submitted for not identified Center regional serve volume of outreach for conferences anticipated clients workers, trainers, and others. For example the following is an excerpt from a report on a CoC planning effort in which an evaluator conducted a survey of COC program staff to measure their assessment of the planning process to date: Using a five-point scale (1=Never; 2=Seldom; 3=Usually; 4=Frequently; and 5=Always), respondents were asked to rate the frequency at which they experience certain issues associated with the CoC program. See Table 2. PROCESS EVALUATION: HOW IT WORKS 117 According to the survey, staff respondents reported that cooperation among community providers in data provision and activity promotion was low (mean = 1.91) – approaching the seldom level. They too indicate that there is presently modest community awareness (ranging somewhere between seldom and usually) of CoC’s purpose (mean = 2.35). These responses suggest weaknesses in and threats to the program if not addressed. On the other hand, respondents identified three relative strengths and opportunities for the program. They reported that they usually felt that there was management support of CoC (mean = 2.97), and that the program’s development was on track (mean = 3.27). Finally, they indicated that resources were quite adequate for the planning effort (mean = 3.82). It is sometimes possible to use these quantitative findings as a jumping off point to discuss with the planning staff the meaning of their aggregated responses. Interviewing is an effective technique for developing narrative commentary to accompany quantitative ratings. Table 2 Circles of Care Concerns (N=36) Issue Mean Experience Score Range Cooperation among community providers in 1.91 Seldom providing data and promoting activities. Community awareness of Circles of Care’s purpose. 2.35 Seldom Management support of Circles of Care. 2.97 Usually Circles of Care is where we should be at this point in 3.27 Usually its development. Resources are adequate to support the planning effort. 3.82 Frequently Interviewing To better understand the CoC process, evaluators employ different forms of interviewing. Patton (1990) identifies three forms of qualitative, open-ended interviews. They are: (a) informal conversational interviews, (b) general interviews using a guided approach, and (c) standardized opened- ended interviews. The first approach, informal conversational interviews, seems to work well as a process evaluation method. 118 VOLUME 11, NUMBER 2 Rubin and Babbie (2001) characterize informal conversational interviews as an “unplanned and unanticipated interaction between an interviewer and a respondent that occurs naturally during the course of field- work observation” (p. 404). This definition implies that spontaneity is important. The evaluator’s focus is on maximizing his or her understanding of events in real time, by asking participants what they think about a situation as it unfolds. At the end of planning or finance meetings, after a long day of GONA activities, or following a special event like a community picnic, the evaluator can debrief with staff and other participants as to whether their expectations were met, and in what ways. The evaluator can inquire about the event’s meaning to them as planners and as community members. Though the activities and events will vary across the three years of planning, the evaluator’s inquiry is consistent. He or she wants to know how the event or activity affects the planning process. Mental notes are made of each conversation that later are recorded. There is nothing secretive about this approach. The evaluator’s role is known, and thus his or her questions are understood to be a part of the process evaluation. A sample dialogue between the evaluator and a CoC staff member might go something like this: EVAL: How do you feel about the Community Picnic now that it is over? STAFF: I’m glad that we had so many people. I’d estimate more than 300. EVAL: What did you hope to accomplish? STAFF: Well we wanted to involve families that we hadn’t seen before and to link them with community agencies. EVAL: How do you think that it went? Were there new families and did the linkages with agencies work out? STAFF: On the whole I think that we were successful. I wish that there had been additional agencies, especially youth services providers. We’ll need to reach out to them more aggressively next time. As you can see, the dialogue is reflective and informal, and attempts are made to capture the experience through the eyes of a CoC staff member.