DOCUMENT RESUME HE 028 478 ED 385 210 Dochy, F. J. R. C.; Valcke, M. M. A. AUTHOR Knowledge Profiles of Economics and Law Students: An TITLE In-Depth Analysis of the Prior Knowledge State. Open Univ., Heerlen (Netherlands). Centre for INSTITUTION Educational Technological Innovation. ISBN-90-358-1041-4; OTIC-RR-34 REPORT NO 92 PUB DATE 28p.; For related documents, see ED 364 699, ED 364 NOTE 671, and HE 028 474-477. Open Universiteit, Secretariaat COP/OTIC, Postbus AVAILABLE FROM 2960, 6401 DL, Heerlen, The Netherlands (20 Dutch guilders). Research/Technical (143) Reports PUB TYPE MF01/PCO2 Plus Postage. EDRS PRICE *Academic Achievement; *College Students; *Economics; DESCRIPTORS Foreign Countries; Higher Education; Individual Differences; Knowledge Level; Law Students; *Prior Learning *Dimensional Analysis; Open University IDENTIFIERS (Netherlands) ABSTRACT This study sought to examine the nature and dimensions of prior knowledge among undergraduates in an economics (OuN). A total of 22 course at the Open University of the Netherlands law and 55 economics students enrolled in two economics courses were given a 154-item domain-specific knowledge test, which was then analyzed independently by three researchers. The researchers attempted to classify each of the 154 items on each of 10 dimensions (curriculum level, curriculum accent, node relation, behavioral, content, epistemological, number of propositions, information level, and representation level). The results of the analysis indicated that although different dimensions helped to differentiate between law and economics students, the different dimensions were not helpful in identifying more specific and, significant contrasts between both student groups. The study also found that the grouping variable "diploma type" was not able to differentiate between levels of mastery of the prior knowledge state. It is foreseen that in situations where there are significant differences between the prior knowledge state of specific subpopulation, the dimensions might be helpful to detect the strengths and weaknesses of the students involved. *********************************************************************** Reproductions supplied by EDRS are the best that can be made from the original document. *********************************************************************** . 14;. .4 . s ovaedge'PrOOles'of:Economics and La w . :thelliior:knowledge In-depth Analysis of an . F.J.R.C. Dochy M.M.A. Valcke t AVAILABLE; BEST COPY "PERMISSION TO REPRODUCE THIS U.S. DEPARTMENT OF EDUCATION Oeco CA Educenonal Rmeerch and Improvement MATERIAL HAS BEEN GRANTED BY EDUCAflONAL RESOURCES INFORMATION CENTER (ERIC) F.J.R.C. Dochy Ia tus document nag been reproduced reCOmed from Om wean or oroanstaben Open University ongsnetong 0 May( chenge nave been mode to wn wove refooduchon otatley Poste. ot ve)* or optmons stated in Duo docu- TO THE EDUCATIONAL RESOURCES ment do not necMaroly represent othcsii INFORMATION CENTER (ERIC1 OE RI poeMon Of Poke,/ OTIC RESEARCH REPORTS. The Open University is responsible for developing and offering open, higher distance education in which special attention is paid to innovations in educational technology. The research in this field is concentrated in "OTIC", that is the Centre for Educational Technological Innovations (Onderwijs Technologisch Innovatie Centrum). OTIC is also engaged in running projects for other institutions. Here the Centre makes use of OTIC's knowledge and experience acquired in research and development. The series of OTIC Research Reports consists of publications of the OTIC research projects and aims mainly at an audience of fellow researchers. THE RESEARCH PROJECT ON EVALUATION AND TEST-FUNCTIONALMES The project "Evaluation and Test-functionalities" focuses on the problems caused by the wide diversity of students and the problems with individual and flexible learning-processes of these students. The project leads to an integration of: - the results of the project "Prior Knowledge"; - the developments of the "Computer Assisted Testing"-project; - the developments of "Adaptive Testing" and the IRT-applications (Item-Responds Theory); and - the experience of the Open University with the development and use of TSS (Test service systems). The main objectives are: (1) to get a discernment of the test and evaluation problems in the open-learning system; (2) the generation of the guide-lines, specifications, and technological instruments concerned with the use of prior knowledge and experience, flexible testing and the supervision on students during the learning process, and (3) the development of instruments which can be useful in solving the given teaching problems. Centre for Educational Technology and Innovation Open University Knowledge Profiles of Economic and Law Students: an In-depth Analysis of the Prior Knowledge State OTIC Research Report 34 Dochy, F.J.R.C. Valcke, M.M.A. 4 CIP- gegevens koninklijke bibliotheek, Den Haag Dochy, F.J.R.C. Valcke, M.M.A. Knowledge Profiles of Economics and Law Students: an 1n-depth Analysis of the Prior Knowledge State/ F.J.R.C. Dochy, M.M.A. Valcke. - Heerlen: Open University, Educational Technology Innovation Centre (OTIC) - Ill. - (OTIC Research Report 34) Met lit. opg., reg. ISBN: 903581041 4 Trefw.: Knowledge profile / Prior knowledge states c 1992, Open University, Heerlen Save exceptions stated by the law no part of this publication may be reproduced in any form, by print, photoprint, microfilm or other means, included a complete or partial transcription, without the prior written permission of the publisher. OTIC Research Reports are available at: the Open Universi:y secretariaat OT1C/COP postbus 2960 6401 DL Heerlen Telephone: 045-762261 / 471 5 Introduction 1 1 2 Theoretical background 1 The structure of knowledge 2.1 1 2.2 Knowledge profiles 2 2.3 Overview of knowledge profile dimensions 3 2.3.1 Content related dimensions 3 2.3.2 Cognitive psychological dimensions 4 2.3.3 Educational-psychological dimensions 4 2.3.4 Item characteristics dir. anions 5 2.4 The prior knowledge state of economics students (ES) and law students (LS) 6 Research design 3 6 Hypotheses 3.1 6 3.2 Research instruments 7 Research population and procedure 3.3 8 3.3.1 Research population and sample size 8 3.3.2 Research procedure 4 Research results .and discussion 8 4.1 General results 8 4.2 Profiles of ES & LS : a first analysis 10 4.2.1 Economics subdomains dimension 10 4.2.2 Curriculum level dimension 11 4.2.3 Curriculum accent dimension 11 4.2.4 Node relation dimension 12 4.2.5 Behavioural level dimension 12 4.2.6 Content level dimension 13 4.2.7 Epistemological level dimension 13 4.2.8 Representation level dimension 14 4.2.9 Amount of propositions dimension 14 4.2.10 Information level dimension 15 4.2.11 Intermediate conclusions 15 4.3 Profile analysis 15 4.3.1 Control of underlying assumptions 16 4.3.2 Profile analysis results : parallelism test 19 4.3.3 Profile analysis results : flatness test 19 Conclusions 5 20 References 6 21 6 Knowledge profiles of economics and law students P. 1 Introduction 1 There is no doubt that the prior knowledge state is playing a major role in the learning process of students. In our recent work (Valcke and Dochy, 1991; Dochy and Valcke, 1991b), the analysis of the quality and impact of the prior knowledge state has been a major focus. Several instruments have been developed to measure the prior knowledge state, especially within the domain of economics. In analyzing the prior knowledge state, we did especially focus on the structure of the prior knowledge state along a content dimension. In this report we report a study which supports the development of 'knowledge profiles' as an assessment tool in educational practice to direct future learning. In the theoretical part of this text, we discuss - in short - our distinct approach towards the analysis of the prior knowledge state'. This approach is based on an extensive analysis of the literature in relation to theories, models and practice-based strategies about the "structure of knowledge". This. base is exploited to define a set of "dimensions" that are helpful to construct "knowledge profiles". Four types of dimensions are illustrated : cognitive-psychological dimensions, educational-psychological dimensions, psychometrical dimensions and content-based dimensions. In the second part of this text, these dimensions are used to analyze the knowledge profiles of economics and law students. The results help to detect differences in the mastery of components of the prior knowledge state between both student populations and might be helpful to provide further evidence about the validity of the theoretical knowledge profile dimensions. 2 Theoretical background The structure of knowledge 2.1 From an instractional-psychological point of view, the structure-of-knowledge problem should be investigated in order to find out more efficient ways for using instructional technology. Our search for means to deal with the prior knowledge state showed that one should take account different components of the prior knowledge state (Dochy and Valcke, 1991). The concept of "components" refers towards a structure in the knowledge base of the learner. Our earlier research was helpful to detect such components of the prior knowledge state along the content dimension. But it was also suggested that the differentiation of components of the prior knowledge state along other dimensions is needed to In helpful to diagnose educational practice (Dochy and Valcke, 1991; Dochy and Valcke, 1991b) The issue of the "structure of knowledge" has been debated from a variety of theoretical points of view : cognitive psychology, epistemology, philosophy, etc. At the more pragmatic level, the issue has also been of prime importance in applied sciences like instructional psychology, curriculum development theories and psychometry (Dochy, 1992). Disciplines like cognitive psychology, educational psychology, artificial intelligence, etc. - have - from their points of view - highlighted the "structure of knowledge" resulting in a puzzling variety of approaches, models and (Ausubel, 1968, de Groot, 1946, Mayer, 1979, Reigeluth and Stein, 1983). I A more elaborated version of the theoretical base of the knowledge profile dimensions can be found in : Dochy & Valcke (1991a). OTIC. p. 2 Knowledge profiles of economics and law students It should be noted that our primary focus in using these theories originates from an information processing view on learning (Sternberg, 1985a & 1985b). The main reason for this is that we stress a dynamic approach towards the structure of knowledge, which is in particular advocated in this view. If we summarize the variety of approaches, four main types of dimensions to structure knowledge can be conceptualized : Content related dimensions Cognitive-Psychological dimensions Educational-Psychological dimensions Item Characteristics dimensions 2.2 Knowledge profiles As such, the concept of 'knowledge profiles' is not found in literature. Only 'student profiles' (Wolf, et. al., 1991) and cognitive profile' (Letteri et. al., 1982) have some simi'arity in meaning This is certainly the case for the studies by Letteri et. al. (1980, 1982). The concept 'profile' is derived from the practice, common in educational research, of plotting as a graph or profile the scores of a person as raw scores or as standardized scores (Keeves, 1988). In analyzing research findings, comparisons are made between persons or groups in terms of a set of measurements on specific related aspects. For each person or group a profile is obtained on a set of parameters. The comparison between profiles of persons is known by the generic term 'profile analysis'. Figure 1 shows the relationship between some key concepts. A "dimension" is used to construct a knowledge profile. Each dimension, consisting of several parameters, represents an approach towards the structure of knowledge. 1111112014 f TALUISMO Figure 1: Example of a profile From an instructional psychological point of view, knowledge profiles can give practical indications of student achievement and learning in order to direct the learning process. In a recent overview of student assessment, Wolf et. al. (1991) advocate this approach. According to these authors, there is a need for a educational new brand of educational psychometrics capable of answering the much changed questions of achievement. These changes are the new premises, the multiple paths towards the prior knowledge state, school with widely more developmental oriented assessments and the ascertainment that students enter varying backgrounds. In our terms, we take account of these changes by trying to identify multiple components of the prior knowledge state, by implementing prior knowledge state tests and by intending to In this context it is necessary to come to use these tests as progress tests administered several times a year. Knowledge profiles of ecoalics and law students P. 3 an agreement on the relevant parameters to describe student performance and it is critical to develop ways of looking at'student profiles': "unless we develop these kinds of differentiated portraits of student performance within a domain, it is difficult to envision student assessment ever informing, rather than merely measuring, the educational process" (Wolf, et. al., 1991). 2.3 Overview of knowledge profile dimensions Only those dimensions/parameters are reviewed that have been retained after their discussion and analysis in our earlier publication "Validation of Knowledge Profile Dimensions : Looking for empirical Evidence" (Duchy and Valcke, 1991a). If dimensions are based on a model or theory, only short details will be reported. We will shortly report on f le models or theories on which dimensions are based. The first dimensions are classified according to common models of economics. Other dimensions are based on theories on knowledge representation and on knowledge structure, learning theories, text representation models and psychometric theory. .3.1 Content related dimensions Economics subdomains dimension "Content" is one of the most exercised dimensions to categorize domain knowledge. The classification based 1 on the parameter 'subdomains' refers to the subdivision of the economics-domain into "subject matter blocks" that are common within the science of economics. Our dimension structure, as implemented in the curriculum structure of the University of Maastricht, contains nine parameters : 1. Reporting 2. Financing 3. Organization 4. Marketing 5. Macro-economics 6. Micro-economics 7. Public finances 8. International economic affairs 9. Behavioural and social sciences Curriculum level dimension Some parts of the content of a science are supposed to be mastered by the students at certain moments during their study. These moments are called the curriculum levels (first and second year). These levels are subsequent, but too broad to be supposed hierarchical. 1. First year level 2. Second year level Curriculum accent dimension Within economics it is common to differentiate between two main streams, representing a different accent, i.e. general economics and business administration on the one hand and quantitative economics on the other hand. 1. General economics and business administration 2. Quantitative economics Ansa p. 4 Knowledge profiles of economics and law students 2.3.2 Cognitive psychological dimensions Node relation dimension Knowledge representation, as used in schema theories (Dochy and Bouwens, 1990), takes certain propositions or nodes as a starting point. A proposition is the smallest unit that can be qualified as true or false in a statement. According to most schema theories there are five kinds of nodes : Physical State (PS, a statement that refers to an ongoing state in the physical or social world), Physical Event (PE, a statement that refers to a state change in the physical or social world), Internal State (IS, a statement that refers to an ongoing state of knowledge, attitude, or belief in a character), Internal Event (IE, refers to a state change in knowledge, attitude or belief in a character), Goal (G, a statement that refers to an achieved or unachieved state that a person wants) and Style (S, a statement that refers to details about the style or manner in which an action or event occurred. 1. G - G REASON 2. PS - G INITIATE IS - G PE - G IE - G 3. PS - PE CONSEQUENCE IS -PE PE - PE 1E - PE G - PE PS-PS IS -PS PE - PS IE -PS G - PS 4. PE - S/G MANNER IE - S/G GE - S/G 1 5. PS - PS PROPERTY The *Node Relation" dimension is based on characteristics of the interrelations between propositions, called node relation or arc parameters: Reason (R, a Goal node is a reason for another Goal node), Initiate (I, a State or Event initiates another Goal node), Consequence (C, a State, Event or Goal node that has the consequence of another State or Event node), Manner (M, an Event or Goal node occurs with some style), Property (P, a person, object or entity has some property that is a State node) (see also Dochy and Bouwens, 1990). These arc parameters are not hierarchical in nature. 2.3.3 Educational-psychological dimensions The theoretical base of these two dimensions - i.e. behavioural and content dimension - is found in Component Display Theory (CDT, Merrill, 1983), Taxonomic theories (De Block, 1986 and Bloom, 1976) and Gagn6's theoretical classification (1985). Behavioural dimension The known distinction between declarative and procedural knowledge is further operationalised at this stage into the parameters 'to know, to understand, and to apply'. These parameters are also perceived as equivalent to the concepts 'recognition, reproduction and production'. Items can be classified as measuring the appreciation, the recognition and the reproduction of information (declarative) or measuring production production)(procedural) (Keeves, 1988). or applications (interpretative, convergent, divergent or evaluative