The Canadian Journal of Higher Education, Vol. XXIII-3, 1993 La revue canadienne d'enseignement supérieur, Vol. XXIII-3, 1993 Using the Delphi Method to Improve the Curriculum SHIRLEY BLAIR* & NORMAN P. UHL* Abstract An example of employing the Delphi technique for curriculum improvement at a Canadian university is described. This technique was used to identify essential course components to be included in a revised Office Administration program. In addition to providing the necessary information for updating and improving the curriculum, the Delphi technique allowed the department to receive infor- mation from and educate guidance counsellors and future employers with regard to the proposed changes in the program. The Delphi technique made it possible to obtain opinions from five diverse groups at relatively low cost and effort. The anonymity that the Delphi method provides helped to ensure that responses reflected accurately the opinions of the respondents. A number of areas of study and special competencies and skills were identified as being essential to a program designed to meet the needs of the automated workplace. The study determined that the program should encompass a much larger compo- nent in office technology than existed previously and that the focus of the pro- gram should be on information management rather than office administration. Along with the methodology, the advantages and disadvantages of using the Delphi technique for improving the quality of the curriculum are discussed. * Mount Saint Vincent University 108 Shirley Blair & Norman P. Uhi Résumé L'article décrit un exemple de l'emploi de la technique Delphi dans une université canadienne en vue d'améliorer un programme d'études. L'emploi de cette méthode a permis d'identifier les composants essentiels des cours à inclure dans un programme d'administration de bureau rénové. En plus de fournir les données nécessaires à la mise à jour et à l'amélioration du programme, la technique Delphi a rendu possible un échange d'information permettant notamment au département d'éduquer les conseillers d'orientation et les futurs employeurs quant aux modifications proposées pour le programme. La technique Delphi a permis de consulter cinq groupes différents pour un coût et un effort relativement modérés. L'anonymité garantie par les procédures de la méthode a permis d'assurer que les réponses reflétaient fidèlement les opinions des répondants. Plusieurs domaines d'études et plusieurs compétences et savoir- faire spécifiques ont été identifiés comme essentiels à un programme visant à répondre aux besoin d'un lieu de travail automatisé. L'étude a déterminé que le programme devrait comprendre un composant en étude de bureautique beaucoup plus étendu que celui qui existait auparavant et qu'il devrait être davantage centré sur la gestion de l'information que sur l'administration de bureau. L'article présente la méthodologie et discute les avantages et désavantages de la technique Delphi pour la révision des programmes. With the recent emphasis on quality of education (Conrad & Blackburn, 1985; Smith, 1991; Bogue & Saunders, 1992), it is important to identify useful meth- ods for improving the curriculum. A procedure which has been used successful- ly for this purpose in the United States is the Delphi technique (Uhl, 1990). This method has been used in Canada as a forecasting tool and as a method of obtaining convergence of opinion (Nadeau et al., 1992), but rarely has it been reported in publications as being used for curriculum improvement. An objective of the Delphi technique is to obtain convergence of opinion without bringing individuals together in face-to-face meetings. This objective is usually achieved by having the participants complete a series of questionnaires interspersed with controlled opinion feedback. This mode of controlled interac- tion among the respondents not only leads to savings in time and money, but also permits independent thought among participants and assists them in the gradual formation of a considered opinion. It has the added advantage of ensur- ing participants' anonymity. In contrast, direct confrontation, as experienced in faculty or committee meetings, often results in hasty offering of preconceived notions, the inclination to close one's mind to novel ideas, a tendency to defend Using the Delphi Method to Improve the Curriculum 109 previously taken stands, and a tendency to be influenced by the persuasively stated opinions of others. The general procedure for the Delphi technique is as follows: 1. The participants are asked to list their opinions on a specific topic such as curriculum revision or planning priorities. 2. The participants are then asked to evaluate the total resulting list by a criterion such as importance, chance of success, and so on. 3. The participants receive the list and a summary of responses to the items. If the participants are in the minority, they are asked to revise their opinions or indicate their reasons for remaining in the minority. 4. The participants again receive the list, an updated summary, minority opinions, and another chance to revise their opinions. The use of the Delphi technique may be warranted if any or all of the fol- lowing conditions exist: (1) the resolution of a problem can be facilitated by the collective judgments of one or more groups; (2) those groups providing judg- ments are unlikely to communicate adequately without an intervening process; (3) the solution is more likely to be accepted if more people are involved in its development than would be possible in a face-to-face meeting; (4) frequent group meetings are impractical because of time, distance, and so forth; and (5) one or more groups of participants are more dominant than another (Uhl, 1983). The following describes the employment of the Delphi technique to improve the curriculum of an office administration program in a relatively small university (approximate FTE of 4000) in Nova Scotia. Prior to this Delphi study, the faculty of the Department of Office Administration had been meeting to develop a revised program which it hoped would better serve the needs of the marketplace. As it developed a proposed new program, the faculty thought it would be beneficial to obtain from different groups reactions to the proposed changes and suggestions for additional changes. Several groups from which reactions would be helpful were identified. These included the department's advisory board, graduates of the department since 1980, students presently enroled in the department, a selected group of Atlantic Canadian businesses, and high school guidance counsellors in the province. These five groups were selected either because of their unique posi- tions as stakeholders in the proposed curriculum revisions or because they were considered to be "experts" who would very likely contribute useful and timely information. The fact that all of these groups were considered to have a valuable 110 Shirley Blair & Norman P. Uhi contribution to make even though they were, to some extent, widely divergent, necessitated the use of a method which might provide adequate communication among the groups in the form of feedback of opinions of other respondents. Meetings either within or between these groups were not practical because of time, geographic and fiscal constraints. Furthermore, had it been feasible to bring these groups together, it is likely that certain groups would have tended to dominate, as a result of their greater experience, maturity, and positions in the business community. For these reasons, therefore, the Delphi technique was selected as the best method for obtaining information, gaining the greatest degree of participation and achieving some convergence of opinion among these different groups. Method Subgroups The five subgroups included all 12 members of the department's advisory board, 60 representatives from Atlantic Canadian businesses, all 220 Office Administration Department degree graduates from 1980 to the present, Nova Scotia high school guidance counsellors, and students presently enrolled in the department's programs. The Eastern Canadian businesses were chosen on the basis of size as well as prominence in their communities and their industries. The 54 guidance counsellors were from the largest high schools in the province. Returning students in good academic standing who were enrolled in the pro- gram above first year level were selected. Questionnaire Development In developing the first questionnaire, material was first reviewed from uni- versities which had been involved in Office Administration curriculum revi- sions. With this information as a background, the first-round questionnaire was developed in consultation with the department. This questionnaire was pretested on a graduate class in Education composed of guidance counsellors, and on some Office Administration faculty and three members of the business commu- nity. The questionnaires for rounds two and three were developed on the basis of the responses received to this first questionnaire. The second and third ques- tionnaires reflected the responses of the previous rounds. Consistent with the Delphi procedure, these questionnaires were designed to give the respondents as much feedback as possible on the responses of the group to each question in the preceding questionnaire. Table 1 shows examples of types of questions in the Using the Delphi Method to Improve the Curriculum 111 three rounds of questionnaires. The questions from rounds two and three show how feedback on responses of other participants was given to respondents to assist them in their decision-making. The Round One questionnaire contained more open-ended questions, as opinions were sought concerning components that should be included in future rounds. The subsequent rounds then contained more questions using Likert-type scales. Procedure Three rounds of questionnaires with covering letters (available from the senior author) were sent over a period of eight months. Before the first mailing, letters were sent to the superintendents of every school district involved, asking if any objected to a polling of the opinions of guidance counsellors in their districts. Some superintendents responded with written permission. None refused permis- sion. After the first round, the second round of questionnaires was sent to busi- ness and student subgroups who had responded to the first round. The guidance counsellors had a very low rate of response in the initial round (17%) and expressed concern with regard to their inadequate knowledge of the program and its graduates. Therefore this whole group was sent a revised cover letter to explain their role in the study, along with the second questionnaire. In addition, the second round was sent to all graduates who had responded to the first round, as well as to an additional 15 selected from the list of those who had been sent the first questionnaire and had not replied. (Before selecting these 15, names of all those graduates whose questionnaires had been returned as undeliverable were eliminated from the original list.) These additional graduates were added to the second round because of the relatively low response (32%) of graduates to the first round. The remaining three groups were considered to have an acceptable rate of return after the first round. It was recognized at the outset that the graduate population is a mobile one and that many of the addresses on record at the University would be out of date. Although every effort was made to acquire the most recent addresses, a large group of graduates was sent the first round in order to compensate for an anticipated low rate of return. The Advisory Board was not asked to complete the second questionnaire as they were involved in discussions with the faculty at the time and were providing feedback and recommendations. This group, therefore, received only the first and third rounds. The third-round questionnaires were sent to the same people to whom the second round had been sent, with the exception of those whose questionnaires had been undeliverable during the second round and the addition of the Advisory Board. 112 Shirley Blair & Norman P. Uhi Table 1 Sample Questions from Three Rounds Round Question One 2.(a) In order to develop courses that will prepare the type of grad- uate competent in the areas identified by you in Question One, please rate the following competencies to indicate, by circling the appropriate number, the degree of competence which you consider necessary for students in this program. Very competent Fairly competent Not Necessary Keyboarding 1 2 3 Training skills 12 3 Records management 1 2 3 2.(b) Please add competencies that you think should have been included in the preceding list as well as any comments you may have. Two 2. In order to develop courses that will prepare the type of gradu- ate competent in the areas identified by the Department and by you in Round One, please rank the following competencies. Please indicate, by circling the appropriate number, the importance of each competency to students in this program. Underlined numbers indicate the majority response from Round One. If you agree with the previous majority response, circle this number. If you disagree, please circle your choice and, if you wish, give reasons for doing so. Note that new items have no underlined number as there was no previous response. For each item, please select very important (1), fairly important (2), or not important (3) Reasons for choicc if outside majority. Keyboarding 12 3 Training skills 12 3 Records management 1 2 3 Work flow analysis 1 2 3 Using the Delphi Method to Improve the Curriculum 113 Table 1 (cont'd) Three In order to develop courses that will prepare the type of graduate competent in the areas identified (both by the Department and by you) in Question One, please rate the following competencies. Please indicate, by circling the appropriate number, the impor- tance of each competency to students in this program. Underlined numbers indicate the majority response for items in Round Two for which a consensus was not reached. If you agree with the pre- vious majority response, circle this number. If you disagree, please circle your choice. Items for which a consensus was reached are indicated in the appropriate columns. The right-hand column sum- marizes reasons given by some respondents for not choosing the majority (underlined) selection. Only reasons cited by at least two people are included. Very Fairly Not Important Important Important Comments Keyboarding Consensus Training skills Consensus Records management 1 This is a major component of managing information. Poor management of records results in loss of time and Work flow analysis money In the analysis of the data, frequencies were calculated for each element in all three rounds of questionnaires in order to give feedback percentages to respondents during the rounds and to determine if and when consensus was achieved. Consensus was defined as 85% agreement. This was done for the whole group in each round as well as for all subgroups in each round. In addition, the 61 respondents who had answered all three questionnaires were tracked, and frequencies and means and standard deviations were calculat- ed for this group. This allowed a comparison of the responses of this group with responses of those who completed at least one round. The comparison provided reassurance that congruence had not occurred as a result of those in disagree- ment with the majority not completing later rounds. 114 Shirley Blair & Norman P. Uhi Results Table 2 shows the response rates for each subgroup in the three rounds. From a total of 386 questionnaires sent in the first round there were 151 returned (40%). Second-round questionnaires were sent to a total of 192 individuals and 102 (53%) completed them. As previously explained, the second round did not include the Advisory Board. Third-round questionnaires were sent to 194 indi- viduals and 97 (50%) were completed. The larger number for round three, despite some attrition, is due to the inclusion of the 12 Advisory Board mem- bers who were not sent questionnaires in round two. During the three rounds, there was some attrition in the Business sub-group due, in part, to the very unstable economy during this period. This economic instability, which led to personnel transfers and loss of jobs, may have also contributed to attrition in the graduate sub-group. There were three main areas investigated in the three rounds of questions: (1) the general areas of expertise that the respondents perceive as essential to graduates of this program, (2) the key elements (competencies) which should be included in the program, and (3) the types of occupations that these graduates would be expected to fill. In addition, respondents were asked their opinions concerning the efficacy of including co-op work terms in the program and for input on how they thought the focus of the program should change. Areas of Expertise Consensus was reached concerning the following areas: the introduction of new technology; linkage of existing technology and computer systems in the work- place; managing the impact of technology on the type of work done and the way in which that work is done in a business environment; managing information in the business environment; and management of the impact of technology on peo- ple and its effect on their satisfaction, motivation, and performance. Competencies Throughout the three rounds of questionnaires, respondents were asked their opinions concerning the degree of competence needed by graduates in various specific skills and knowledge areas. The original list of 23 competencies in Round One was expanded to 34 in Round Two as a result of suggestions from respondents to the first questionnaire. Table 3 shows frequencies for responses concerning competencies. Since the response rate was not always consistent (for example, some peo- ple responded to Rounds One and Three, but not to Round Two), Table 4 shows Table 2 Response Rates Round 1 Round 2 Round 3 Subgroups Sent Returned Rate Sent Returned Rate Sent Returned Rate Advisory Board 12 9 75% N/A N/A N/A 12 7 58% Business 60 27 45% 25 22 85% 24 17 71% Graduates 220 72 33% 80 38 48% 75 34 45% Guidance Counsellors 54 9 17% 54 19 35% 52 15 29% Students 40 34 85% 33 23 70% 32 24 75% TOTAL 386 151 40% 192 102 53% 195 97 50% Note: Advisory Board did not participate in Round Two. Table 3 Frequencies in Percentages for Competencies: All Respondents R1 (N=151) R2 (N=102) R3 (N=97) CCoommppeetteenncciieess VI FI NI VI FI NI VI FI NI Keyboarding 72.8 25.8 1.3 87.3 10.8 2.0 C — Proofreading 80.8 15.2 4.0 92.2 4.9 2.9 C — — Written Communication Skills 92.1 7.9 0.0 98.0 2.0 0.0 C — — Oral Communication 89.4 9.9 0.7 99.0 1.0 0.0 C — Records Management 44.6 53.4 2.0 42.6 55.4 2.0 51.5 48.5 0.0 Word Processing 83.3 16.0 0.7 98.0 2.0 0.0 C — — Spreadsheets 49.7 47.0 3.3 88.1 11.9 0.0 C — — Databases 53.0 45.0 2.0 95.0 5.0 0.0 C — — Graphics 28.5 57.0 14.6 22.8 75.2 2.0 25.8 73.2 1.0 Accounting Software 33.1 58.3 8.6 27.7 70.3 2.0 25.8 70.1 4.1 Desktop Publishing 29.1 59.6 11.3 24.8 71.3 4.0 26.8 69.1 4.1 Decision Support Software * * * 32.7 57.1 10.2 14.4 81.4 4.1 CAD/Engineering Software * * * 20.2 48.5 31.3 4.1 62.9 33.0 Business Statistics 17.2 68.2 14.6 12.7 81.4 5.9 15.5 74.2 10.3 Research Methods 23.2 60.9 15.9 11.8 79.4 8.8 18.6 70.1 11.3 Telecommunications 54.3 42.4 3.3 88.2 11.8 0.0 C — Office Environment Planning/Design 36.4 53.6 9.9 25.5 74.5 0.0 23.7 70.1 6.2 Systems Analysis 40.9 51.0 8.1 18.8 72.3 8.9 21.6 74.2 4.1 Equipment Selection 47.0 41.1 11.9 34.7 59.4 5.9 45.8 49.0 5.2 Software Evaluation 50.3 40.4 9.3 84.2 9.9 5.9 C — —