DOCUMENT RESUME IR 021 301 ED 466 230 Williams, Douglas C.; Liu, Min; Benton, Denise AUTHOR Analysis of Navigation in a Problem-Based Learning TITLE Environment. PUB DATE 2001-00-00 7p.; In: ED-Media 2001 World Conference on Educational NOTE Multimedia, Hypermedia & Telecommunications. Proceedings (13th, Tampere, Finland, June 25-30, 2001); see IR 021 194. Association for the Advancement of Computing in Education AVAILABLE FROM (AACE), P.O. Box 2966, Charlottesville, VA 22902 ($40, AACE members; $50, nonmembers). Tel: 804-973-3987; Fax: 804-978-7449; Web site: http://www.aace.org. Speeches/Meeting Papers (150) Research (143) Reports PUB TYPE MF01/PC01 Plus Postage. EDRS PRICE *Computer Assisted Instruction; *Computer Software DESCRIPTORS Development; Educational Environment; *Educational Technology; *Hypermedia; Instructional Effectiveness; Instructional Materials; Middle School Students; Middle Schools; *Problem Based Learning; Problem Solving ABSTRACT Research has shown the potential of a problem-based approach to enhance students' learning. The interactive nature of hypermedia technology and its ability to deliver information in different media formats can provide unique capabilities for implementing problem-based learning (PBL) environments. Yet, little is known about the types of tools that are effective in supporting learning in hypermedia supported PBL environments. The purpose of this exploratory study is to investigate both the use of tools and design features in a piece of PBL software and their effectiveness on middle school students' learning. Specifically, the following research questions formed the focus of the study: Do any navigational profiles emerge among the student teams? If so, how do they compare usage of the tools embedded in the PBL environment? If navigational profiles exist, how do they compare in terms of the external criteria such as science aptitude, recall and transfer measures, and attitude toward the environment? If navigational profiles do not exist, what is the relationship between tool use and student achievement and attitude? (Contains 15 references.) (AEF) Reproductions supplied by EDRS are the best that can be made from the original document. Analysis of Navigation in a Problem-based Learning Environment Douglas C. Williams, University of Louisiana at Lafayette, Department of Curriculum & Instruction, USA, [email protected] PERMISSION TO REPRODUCE AND DISSEMINATE THIS MATERIAL HAS Min Liu, University of Texas at Austin, Department of Curriculum & Instruction, USA, BEEN GRANTED BY [email protected] G.H. Marks Denise Benton, University of Louisiana at Lafayette, Department of Mathematics, USA, [email protected] TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) Research has shown the potential of a problem-based approach to enhance -Abstract: U.S. DEPARTMENT OF EDUCATION Office of Educational Research and Improvement students' learning. The interactive nature of hypermedia technology and its ability to EDUCATIONAL RESOURCES INFORMATION deliver information in different media formats can provide unique capabilities for CENTER (ERIC) ifThis document has been reproduced as implementing problem-based learning (PBL) environments. Yet, we know little about the received from the person or organization originating it. O types of tools that are effective in supporting learning in hypermedia supported PBL Minor changes have been made to environments. The purpose of this exploratory study is to investigate both the use of tools improve reproduction quality. ("NI and design features in a piece of PBL software and their effectiveness on middle school r:) Points of view or opinions stated in this Since data gathering has just been completed, there has not been students' learning. document do not necessarily represent 71- official OERI position or policy. sufficient time to complete the data analysis. Data analysis will be completed and research results and conclusions presented in the conference presentation. 4.4 1. Research Framework 1.1 Problem-based Learning There has been a growing body of research on authentic and situated learning environments utilizing the problem-based approach to learning [Cognition and Technology Group at Vanderbilt, 1992; Pedersen, Problem-based learning (PBL) emphasizes solving authentic problems in authentic Williams, 1999]. It is an approach where students are given a problem, replete with all the complexities typically contexts. Problem-based learning found in real world situations, and work collaboratively to develop a solution. provides students an opportunity to develop skills in problem definition and problem solving, to reflect on [Cognition and their own learning, and develop a deep understanding of the content domain learning Technology Group at Vanderbilt 1994; Jacobson & Spiro, 1995; Lajoie 1993]. This approach was developed in the sixties for medical education, and has since been used in various subject areas, such as business, law, education, architecture, and engineering. Literature has shown that problem-based learning can facilitate the improvement of student attitude toward the content area learning [Cognition and Technology Group at Vanderbilt, 1992; Williams, 1999]. Explanations offered for this are that students perceive the relevance of the work [Barrows 1986] or compare the task of finding information and developing a solution to solving a mystery [Cognition and Technology Group at Vanderbilt 1992]. Medical students using a PBL curriculum have been shown to be able to pursue learning independently better than their peers receiving a traditional curriculum [Aspy, Aspy, & Quinby 1993], a finding which supports the claim that PBL prepares students to become independent, lifelong High school students in a semester long class which used PBL exclusively showed a significant learners. increase in their spontaneous use of one problem-solving step: problem finding, the ability to identify and Similarly, students in a PBL formulate problem statements [Gallagher, Stepien, & Rosenthal 1992]. curriculum have shown a greater ability than students in traditional classes to break complex problems into their component parts and identify sub-problems which must be solved in order to solve the main problem [Cognition and Technology Group at Vanderbilt 1992; Gallagher & Stepien 1996]. In recent years, there has been a growing interest among educators to use PBL in the K-12 setting. However, the experiences of college level programs and early efforts in K-12 schools have shown PBL to be a particularly demanding instructional approach for both students and teachers [Barrows & Tamblyn, 1980; Cognition and Technology Group at Vanderbilt, 1992]. Students must perform a wide variety of tasks with O which they may have had limited prior experience. These include problem finding, hypothesis generation, identification of learning needs, location of resources to meet these learning needs, data collection and O 2 at BEST C Y AVALABLE Page 2052 Teachers, in their organization, and development of a solution plan supported by evidence and reasoning. role as facilitators, are responsible to provide support for this wide variety of activities to students who may ability to function in this role vary greatly in their needs. Because the variability of the classroom teacher's has a profound effect on the success of PBL [Walton & Matthews 1989], instructional materials that provide some of this support seem warranted. 1.2 Hypermedia and Problem-based Learning The interactive nature of computers and the ability to deliver information in different media formats provides unique capabilities for the implementation of PBL environments. In particular, the capabilities of Hypermedia is a rich hypermedia have much to offer designers of advanced learning environments. environment containing information in many different forms, including text, graphics, audio, video, and animation. The student is not forced to access the resources in any predetermined order, but can navigate The support of multiple within the environment in a nonlinear fashion [Burton, Moore, & Holmes 1995]. media types and the possibility of nonlinear navigation are particularly useful in the creation of computer- based PBL environments. The nonlinear nature of hypermedia is consistent with the characteristics of PBL. As students are engaged in PBL, they need to collect data and access resources. This suggests a high degree of control on the part of the learner. Hypermedia can support this by allowing students to access needed resources at the time it is most appropriate. Hypermedia allows students to have more control over their learning. They become actively engaged in decision-making while traversing the environment. Research on learner control versus higher than program control in hypermedia environments suggests that subjects under learner control score those under program control on achievement posttest and have a more positive attitude toward learning [Hannafin & Sullivan 1995]. Though PBL can be implemented with traditional media, hypermedia provides unique capabilities for its implementation. The nonlinear nature of hypermedia allows students to explore the PBL environment accessing resources as the need arises. Hypermedia can also facilitate the development of authenticity in the learning environment. Williams [Williams 1992] suggests that law and medical school curricula could be improved by the use of hypermedia environments to engage students in authentic activities within an authentic context. 1.3 Hypermedia and Cognitive Tools When working in everyday situations, individuals use tools and resources such as computers, Therefore, tools must be considered when calculators, concepts, and formulas in order to solve problems. creating authentic environments for student learning. A tool does not necessarily have to be a tangible object. Different For example, an engineer may use a mathematical formula to calculate the area of a cylinder. disciplines and professions may use ideas and tools in very different ways. Many researchers argue that cognitive tools can support and enhance student learning [Lajoie, 1993]. These tools come in many forms and can support students in a variety of tasks they must perform as they engage in problem solving. Hypermedia has the potential to make these tools readily available to students. As discussed earlier, PBL occurs within a context where knowledge is naturally situated. Tools can be employed to create an authentic context in which students can work. The Adventures of Jasper Woodsbury series utilizes video-based scenarios in order to create a context for learning [Cognition and Technology Group at Vanderbilt, 1992]. The video segment provides a focus for the learning activity and may also help students who are poor readers. Hypermedia can also present a scenario, but has the advantage of allowing students to explore the environment in which the scenario is set. Hypermedia can augment memory and support students in reflecting on their problem-solving process. Sherlock I, software that creates an environment in which Air Force technicians practice avionics troubleshooting, includes tools to support cognitive processing [Lajoie, 1993]. Avionics troubleshooting is a complex task requiring the technician to entertain many variables and remember a series of completed tests. Sherlock I allows the student to view the steps he or she has taken in troubleshooting a problem. The ability to view the solution path supports students in reflection on the problem solving process without the need for them to rely on their recollection of every step. The software-generated problem solving steps also make explicit student thinking, an essential component in stimulating metacognitive awareness. Likewise, reflective questions can also be effective in promoting metacognitive thinking. 3 Page 2053 Hypermedia environments can offer a comprehensive set of resources to enable students to meet In order for students to engage in problem solving, they must have access to their learning needs. information. Information can be provided to students in a number of tools. The Lab Design Project (LDP), which allows students to actively take part in sociological research concerning a fictitious biotechnology building, permits students to gather information from such diverse formats as interviews with employees, building plans, letters, and sketches [Honebein 1996]. Hypermedia can provide electronic notebooks, which, in addition to providing space for student note taking, can include advanced features to help support the student in constructing meaning. The notebook in Bio-world contains a section that displays the students' previous actions such as database searches and diagnoses for patients [Lajoie 1993]. These features help support the student's memory and metacognitive Other projects have augmented the traditional notebook with the ability to support multimedia, thinking. collaboration among students, and the ability to create links between concepts [Edelson, Pea, Gomez 1996]. Though literature supports the efficacy of problem-based learning, little research exists which investigates the types of tools or features that are effective in supporting students working in PBL environments. PBL software is beginning to find its way into schools. However, the relative effectiveness of In order to design an effective the various tools incorporated in these programs has yet to be studied. computer-supported PBL learning environment, it is important to understand the tools and design features It is, therefore, the purpose of this study to examine included in the software, and their impact on learning. and understand how middle school students use and interact with a piece of recently available computer- supported PBL software. 2 Research Questions This study investigates the use of tools and design features as employed in a piece of problem-based learning of science concepts. learning software and their effectiveness on middle school students' Specifically, the following research questions formed the focus of the study: (1) Do any navigational profiles emerge among the student teams, and if so, how do they compare in usage of the tools embedded in the PBL environment? (2) If navigational profiles exist how do they compare in terms of the external criteria: science aptitude, recall and transfer measures, and attitude toward the environment? (3) If navigational profiles do not exist, what is the relationship between tool use and student achievement and attitude? 3. Design of Study 3.1 Sample The participants of the study (N = 100) were students enrolled in sixth grade science at a middle school located in a medium-sized city in the southwestern United States. The age.of the students ranged from 12 - 14 years. There were approximately an equal number of male and female students. Students worked in teams of heterogeneous groups determined by the classroom teacher. 3.2 Treatment For this research, a hypermedia-supported problem-based learning environment was developed in the content area of astronomy. The learning goals of the HALE were for students to be able to: Plan and implement procedures for solving complex problems Identify relevant information needed in solving a complex problem Identify the characteristics of objects in our solar system Describe components that comprise probes used for astronomical research Rationalize the design of a probe in regards to it's intended mission Analyze data and draw conclusions from astronomical data notebook, bookmarks, presentation The environment supports students with the following tools: tool, probe builder, solar system database, mission database, alien database, and expert tool. 4 Page 2054 3.3 Dependent Measures Of primary interest in this study is how students of varying characteristics working in teams utilize the tools and support present in the PBL environment and how those tools affect student learning. Therefore, Additionally, measures of all student actions in the environment were logged to a data file for analysis. knowledge recall and transfer were administered, and interviews were conducted with the participants and the classroom teacher. The triangulation of the quantitative and qualitative data sources will help to answer the research questions with richer and more detailed information. 3.3.1 Audit Trail of Tool Access The learning environment kept a log of all student actions in the environment, notebook contents, probes built, and time spent using the program. The log consists of time and date stamped entries. 3.3.2 Recall Measure The recall measure evaluated the amount of declarative knowledge students have concerning the astronomy concepts being taught in the study. The measure consists of multiple choice and fill-in-the-blank Items were revised until items. A panel of reviewers were asked to verify the validity of the measure. agreement was reached. The recall instrument was administered as a pretest, posttest, and retention measure. Due to the factual and objective nature of the measure, it was scored by the principal investigators. The recall measure was used to gauge the degree to which students acquire an understanding of astronomy. Attitude Toward Science in School Assessment 3.3.3 Attitude toward science was measured with Germann's (1988) Attitude Toward Science in School Germann developed ATSSA in response to the ambiguity that exists in science Assessment (ATSSA). attitude as a construct. Germann points out that there are many dimensions that make up attitude toward Attitude toward science can science and that attitude measures must be clear as to what they measure. include such things as scientific careers, methods of teaching science, scientific interests, assessing ideas and information, and science as a subject. The purpose of ATSSA is to measure a single dimension of attitude toward science, that is, how students feel about science as a subject (Germann). The instrument consists of 15 Likert scale items and has a Cronbach alpha of 0.95. 3.3.4 Attitude Toward the Environment The Attitude Response Log gathered information on students' attitudes toward the difficulty of the learning environment, level of interest in the activity, attitude toward learning science with computer, and The questionnaire used in this study consisted of 20 likert-scale items. The 20 attitude toward the tools. items are partitioned into statements concerning the following areas: how students perceived the difficulty of the activity (difficulty scale), what students' level of interest was in the activity (interest scale), whether students prefer learning science on computer or in class with other students (computer preferences scale), and how students liked the tools (tool scale). 3.3.5 Transfer Measures As noted in the literature review section, students are often unable to apply what they have learned in The review of literature put forth that this is due to the decontextualization of problem solving situations. learning. Therefore, in order to evaluate student learning it is not sufficient to only measure acquisition of Hence, near and far concepts. But one must also measure the ability to apply that knowledge in context. transfer measures have been included in the design of this research. 3.3.5.1 Description of Near Transfer Measure In order to evaluate to what degree students are able to apply their knowledge in solving problems similar to the one received in the treatment, participants were given a scenario in textual form describing a problem similar to the one received during the study. The near transfer measure was given as a posttest and After reading the used to provide information on the effects of the expert stories on student learning. . 5 Page 2055 problem scenario, participants were asked to provide the following information: hypothesis, rationale, list of information needed to solve the problem, and a description of the tools needed to solve the problem. 3.3.5.2 Description of Far Transfer Measure In order to evaluate to what degree students are able to apply their knowledge in solving problems that they have not encountered before, students were asked to solve a problem unlike those encountered during the treatment. The far transfer measure was given as a posttest and used to gauge the degree to which After reading the problem scenario, students are able to apply what they have learned to a novel situation. . participants were asked to provide the following information: hypothesis, rationale, list of information needed to solve the problem, and a description of the tools needed to solve the problem. 3.3.5.3 Scoring of Near and Far Transfer Measures Since scoring the near and far transfer measures is subjective in nature, responses from the near and far transfer measures were evaluated by three trained graders who will be blind to the subjects' treatment Due to the large number of responses to evaluate, the responses will be randomly divided conditions. between the three graders. In order to ensure inter-rater reliability, each grader was given an additional 20% of the essays from each of the other two graders. These additional essays were randomly selected, scored, and compared with the scores from all graders. Each essay was scored in the following manner: criteria: following the essay for on the was given to from A score 0 15 1. Quality/plausibility of the hypothesis and supporting rationale, (b) Level of incorporation of (a) details and facts in the essay, (c) Thoroughness of analysis of needed information to support answer, (d) Number of tools cited and the appropriate use of them, (e) overall originality of ideas, (f) overall quality of the essay. Scores from each of the criteria were collapsed into one overall score for the response. 2. 3.4 Procedures Data was collected from ten intact classes resulting in data from 60 student groups. Students were presented with information about the nature of the learning environment. They were informed that at the end of the activity they would be asked for input concerning the aspects of the environment they found useful and those they did not. Prior to treatment, students were asked to complete the recall measure and attitude toward learning After all groups had completed the activity, participants were asked to complete the science instrument. recall measure, near transfer measure, and far transfer measure and attitude toward environment instrument. Three weeks after the completion of the study, the recall measure was administered once again to all of the participants to collect retention data. 3.5 Analyses Since data gathering has just been completed, there has not been sufficient time to complete the data Data analysis will be completed and research results and conclusions presented in the conference analysis. In several studies of hypermedia, researchers analyzed navigational data and found that presentation. different profiles of users tended to emerge. In this study, the users were students working in heterogeneous teams of two to three students determined by the classroom teacher. Each team used the HALE as a group, and hence we will analyze the group navigational data. Individual navigational performance is difficult to record, given that the students used the learning environment as a group. To determine whether different If clusters do group profiles emerge, we will analyze the group navigational data, using cluster analysis. emerge, we can study how each cluster tends to use the different navigational tools and design features, giving us a profile for each cluster. As other authors have done in similar studies, the external variables will be compared to the profile groupings using analysis of variance procedures. If differences do exist, this will not only lend support to the existence of such profiles but also give insight into whether learning as expressed by the recall and transfer measures is different among the profiles. If no clusters emerge, we will look for relationships between the group measures of navigational variables and Pearson correlation coefficients and analysis of the individual student measures of the external criteria. 6 Page 2056 variance procedures will be used to find these relationships. Here the heterogeneity of the group may hinder For example, if one group was composed of students of varying levels of ability, the the analysis. navigational scores would be the same for all group members whereas their recall and transfer measures may In this scenario, future range from high to low. The resulting correlation coefficients may be insignificant. studies of learning environment may need to take into account an individual's navigation through the program. 4. Conclusions As problem-based learning becomes more popular, instructional designers must find new ways to utilize technology in order to support problem-based learning. For this research an innovative problem-based learning environment was created which provided the context for learning and authentic activities for students to engage. Once the analysis is completed, it should provide insight into the tools and features that can support students of varying characteristics when working in hypermedia-supported problem-based learning environments. 5. References Aspy, D. N., Aspy, C. B., & Quinby, P. M. (1993). What doctors can teach teachers about problem-based learning. Educational Leadership, April, 22-24. .Barrows, H. (1996). Problem-based learning in medicine and beyond: A brief overview. New Directions for Teaching and Learning, 68, 3-12. Barrows, H. S., & Tamblyn, R. M. (1980). Problem-based learning: An approach to medical education. New York: Springer Publishing Company, Inc. Burton, J. K., Moore, D. M., & Holmes, G. A. (1995). Hypermedia concepts and research: An overview. Computers in Human Behavior., 11(3-4), 345-69. Cognition and Technology Group at Vanderbilt. (1992). The Jasper Series as an Example of Anchored Instruction: Theory, Program Description, and Assessment Data. Educational Psychologist, 27(3), 291-315. Edelson, D. C., Pea, R. D., & Gomez, L. (1996). Constructivism in the collaboratory. In B. G. Wilson (Ed.), Constructivist learning environments: Case studies in instructional design, (pp. 151-164). Englewood Cliffs, NJ: Educational Technology Publications. Gallagher, S. A. & Stepien, W. J. (1996). Content acquisition in problem-based learning: Depth versus breadth in American studies. Journal for the Education of the Gifted, 19(3), 257-275. Gallagher, S. A., Stepien, W. J., & Rosenthal, H. (1992). The effects of problem-based learning on problem solving. Gifted Child Quarterly, 36(4), 195-200. Hannafin, R. D., & Sullivan, H. (1995). Learner control in full and lean CAI programs. Educational Technology Research and Development, 43(1), 19-30. Honebein, P. C. (1996). Seven goals for the design of constructivist learning environments. In B. G. Wilson (Ed.), Constructivist learning environments: Case studies in instructional design, (pp. 11-24). Englewood Cliffs, NJ: Educational Technology Publications. Jacobson, M. J., & Spiro, R. J. (1995). Hypertext learning environments, cognitive flexibility, and the transfer of complex knowledge: An empirical investigation. Journal of Educational Computing Research, 12(4), 301-333. Lajoie, S. P. (1993). Computer environments as cognitive tools for enhancing learning. In S. P. Lajoie and S. J. Derry (Eds.), Computers as Cognitive Tools, (pp. 261-288). Hillsdale, New Jersey: Lawrence Erlbaum Associates. Spiro, R. J., & Jehng, J. (1990). Cognitive flexibility theory: Theory and technology for the nonlinear and multidimensional traversal of complex subject matter. In D. Nix & R. Spiro (Eds.), Cognition, Education, and Multimedia, (pp. 163-205). Hillsdale, NJ: Lawrence Erlbaum Associates. Walton, H. J., & Matthews, M. B. (1989). Essentials of problem-based learning. Medical Education, 23, 542- 558. Williams, D. C. (1999). Hypermedia-supported authentic learning environments (HALE): Examination of tools and features which can support student learning. Ann Arbor, MI: Bell & Howell Information and Learning. 7 Page 2057 U.S. Department of Education Office of Educational Research and Improvement (OERI) Educdlosol Resources !demotion Cenlu National Library of Education (NLE) Educational Resources Information Center (ERIC) NOTICE Reproduction Basis This document is covered by a signed "Reproduction Release (Blanket)" form (on file within the ERIC system), encompassing all or classes of documents from its source organization and, therefore, does not require a "Specific Document" Release form. This document is Federally-funded, or carries its own permission to reproduce, or is otherwise in the public domain and, therefore, may be reproduced by ERIC without a signed Reproduction Release form (either "Specific Document" or "Blanket"). EFF-089 (5/2002)