Performance and Cognitive Assessment in 3-D Modeling Nolan E. Fahrer Athens Drive High School Jeremy V. Ernst Virginia Polytechnic Institute and State University Theodore J. Branoff Aaron C. Clark North Carolina State University Abstract The purpose of this study was to investigate identifiable differences between performance and cognitive assessment scores in a 3-D modeling unit of an engineering drafting course curriculum. The study aimed to provide further investigation of the need of skill-based assessments in engineering/technical graphics courses to potentially increase accuracy in evaluating students’ factual and conceptual knowledge in preparation for the workplace. The study consisted of 92 high school students enrolled in Drafting II- Engineering. Students were administered existing assessment Nolan Fahrer is an Engineering Drafting teacher at Athens Drive High School in Raleigh, North Carolina. He can be reached at [email protected]. Jeremy Ernst is an Assistant Professor at Virginia Polytechnic Institute and State University. He can be reached at [email protected]. Theodore Branoff is an Associate Professor at North Carolina State University. He can be reached at [email protected]. Aaron Clark is an Associate Professor at North Carolina State University. He can be reached at [email protected]. Volume 48 Number 1 2011 68 Performance and Cognitive Assessment in 3-D Modeling 69 items provided in the 3-D Modeling unit of the Drafting II- Engineering curriculum. The results provided evidence that there were no significant differences between performance and cognitive assessment in the particular unit; however, it is necessary to further develop and implement performance-based assessments in Career & Technical Education that require students to exhibit both skills and knowledge. Introduction Over the years, state and national education organizations have set standards and used initiatives such as the No Child Left Behind Act of 2001 (2002), Carl D. Perkins Career and Technical Education Act of 2006 (2006), and the amended Elementary and Secondary Education Act of 1965 (2002) to help improve state curricula and instruction. Improvement in such areas often includes updating or introducing new curricula and utilizing standardized assessments to gauge school quality and teacher effectiveness. While these transformations in educational practice and instruction are somewhat effective, changes in assessment practices are also required (Firestone & Schorr, 2004). The primary role assessment plays in education is to enhance student learning through classroom instruction and it is secondarily used to hold teachers and institutions accountable and stimulate educational reform (Herman & Aschbacher, 1992, NCTM, 1993, and Linn, 2000). Too frequently standardized assessments encourage a narrow, instrumental approach to learning that emphasizes the reproduction of what is presented at the expense of critical thinking, deep understanding, and independent activity (Boud, 1990). A common form of standardized assessment is curricular tests. Standardized curricular tests are generally used at the state level for school accountability and to better 70 JOURNAL OF STEM TEACHER EDUCATION assess students’ mastery of approved skills and knowledge. However, educational researchers have observed that most items on standardized curricular tests often require little more than students’ recall of facts to arrive at a correct answer (Masters & Mislevy, 1993). Kiker (2007) notes that business and industry leaders, as well as school reform advocates, generally agree that in order for students to be successfully prepared for further education and/or the workforce requires more than traditional core academic skills. In order to gauge successful development of performance skill, assessments must accurately measure what knowledge students have learned and can demonstrate, whether academic or career oriented. Cognitive assessment may suffice for disciplines that do not fully subscribe to constructivist learning theory, but Career and Technical Education (CTE) frameworks necessitate holistic assessment means and methods that incorporate performance measures (Rojewski, 2002). Review of Literature Improvement of curricula and instruction often includes updating or introducing new curricula and utilizing standardized assessments to gauge school quality and teacher effectiveness. Development of assessments must measure 21st century skills and accurately represent what knowledge students have learned and can demonstrate, whether academic or career oriented (Gordan, 1998). In addition, these skills are commonly required for functioning in industry and illuminate students’ learning and thinking processes (U.S. Office of Technology Assessment, 1992). Due to their specialized nature, skills found in engineering/technical graphics and other courses in CTE may require multiple types of assessment. However, the current school accountability measurement system leans heavily in Performance and Cognitive Assessment in 3-D Modeling 71 favor of using only standardized cognitive assessment. Educational practitioners discuss performance assessment as being a worthy alternative assessment to be utilized in conjunction with existing standardized cognitive assessments (Flexer & Gerstner, 1993). However, further research is needed in the field to identify how performance assessment can be utilized as a viable form of assessment in engineering/technical graphics courses. Cognitive assessments generally come in the form of objective paper and pencil classroom test items and standardized tests used for external assessment. Traditional objective classroom assessments are frequently summative and are used for final exams and other forms that require the teacher to assign a grade (Cross & Angelo, 1988). According to Linn (1993) these types of assessments focus on basic skills and practice of factual knowledge. However, multiple choice objectivity types of tests are convenient for teachers because they can be automatically scored, and their markings are assured of having no form of bias (Baker, 1997). A standardized test is generally defined as any test that is administered, scored, and interpreted in a consistent, predetermined manner. Popham (2002) indicates that standardized assessments can be found in two forms, national achievement tests and standardized curricular tests. National achievement tests are standardized assessments that are commonly designed to determine how a test taker will perform in a subsequent setting. In 2001 there were five nationally standardized achievement tests, such as the Iowa Tests of Basic Skills, that were used in the United States public schools (Popham, 2002). Popham (2002) states that there is a high likelihood that the specific content included in this and similar tests may be seriously inconsistent with local curricular aspirations. Many education policy makers assume that national achievement test content will mesh well with what is 72 JOURNAL OF STEM TEACHER EDUCATION supposed to be taught locally, but test takers must cope with considerable national curricular diversity. A study at Michigan State University conducted almost two decades ago suggests that as many as 50 percent of the items included in a nationally standardized achievement test may cover content that is not taught in a given locality (Popham, 2002). Standardized curricular tests are widely used types of cognitive assessment and are generally used at the state level for school accountability and to better assess students’ mastery of approved skills and knowledge. According to Boud (1990), in many cases these tests encourage a narrow, instrumental approach to learning that emphasizes the reproduction of what is presented at the expense of critical thinking, deep understanding, and independent activity. As a result, schools and teachers tend to narrow their curricula and courses with the aim of helping students pass tests from external agencies (Baker, 1997). In a study conducted by Tan (1992), he concludes that frequent usage of formal standardized curricular testing causes negative effects on our education system even with tests well linked to instruction. Many argue that alternative types of assessment should be used. If alternative assessments are implemented properly, motivation and learning progress will increase and school instruction can be correctly evaluated for effectiveness (Dochy & McDowell, 1997). Administrators and educational researchers are becoming increasingly interested in alternative assessment. There is no single definition of al - 1995). Garcia and Pearson (1994) include the following in their review of these labels: performance assessment, portfolio assessment, informal assessment, situated (or contextualized) assessment, and assessment by exhibition. They state that alternative assessment consists of all efforts that do not adhere Performance and Cognitive Assessment in 3-D Modeling 73 to the criteria of standardization, efficiency, cost effectiveness, objectivity and machine scoring. Most importantly this type of assessment provides alternatives to traditional testing in that it reviews regular classroom activities, reflects the curriculum that is actually being implemented in the classroom, provides information on the strength and weaknesses of each student, provides multiple pathways to gauge student progress, and is more multi-culturally sensitive and free of norm, linguistic, and cultural biases found in traditional testing (Garcia & Pearson, 1994). Many researchers inquire if alternative assessments can be aligned to many of the states’ general or functional curricula. Browder and Flowers (2004) conducted a study in three states where experts in mathematics and language arts, along with a group of stakeholders (teachers and administrators), examined the performance indicators relative to their alignment to national standards and curricula. On the surveys, 86 percent of math experts and 70 percent of stakeholders indicated that performance indicators were clearly linked to national math standards. Eighty-six percent of language arts experts and 100 percent of stakeholders that responded to the survey indicated that performance indicators were clearly aligned to language arts standards. The results suggest that alternative assessments have a strong focus on academic and functional skills. A widely used form of alternative assessment is performance assessment. Performance assessment is defined as “testing methods that require students to create an answer or product that demonstrates their knowledge or skills” (U.S. Office of Technology Assessment, 1992). According to Elliot (1997) performance assessments are best understood as a continuum of assessment formats that range from the simplest student constructed responses to comprehensive demonstrations of work over time. 74 JOURNAL OF STEM TEACHER EDUCATION Performance assessment of students’ achievement is not new to many educators but is usually only apparent in the areas of physical education, art, music, and vocational and technological arts. To a large extent, students’ products or performances are used to determine whether learning objectives of a class have been met (Elliot, 1997). However, performance assessment is becoming more prevalent in core classes such as mathematics, science, language arts, and social studies. In a study conducted by Flexor and Gerstner (1993), issues involving the construction of alternative forms of assessment by mathematics teachers were studied through the case study of assessment development in three elementary schools. Three schools with 14 third-grade teachers were selected and matched with three comparison schools where data would also be collected. The three schools continued to use the end-of-chapter tests, but they supplemented those with other assessments that involved more conceptual understanding and higher order thinking. It was concluded that even though there were dilemmas among teachers, positive effects were observed in their students using performance assessment. Research Question The research question examined in this study was: Is there an identifiable difference between performance assessment scores and cognitive assessment scores in the 3-D Modeling unit of a state Drafting II-Engineering curriculum? To further investigate that question, the following research hypotheses were proposed: 1. There is no significant difference in means of the student participants’ performance and cognitive assessment scores in the 3-D modeling unit. Performance and Cognitive Assessment in 3-D Modeling 75 2. There are no significant differences in means of the student participants’ performance and cognitive assessment scores in the 3-D modeling unit among grade levels. To evaluate the first hypothesis, a paired samples T-test was used to determine if differences existed between the means of the assessments. The second hypothesis was evaluated through an analysis of variance procedure used to determine differences in the means of the assessments among grade levels. Participants The participants in this study were enrolled in the Drafting II-Engineering course of study in a North Carolina public school. Drafting II-Engineering introduces students to the use of the graphic tools necessary to communicate, analyze, and understand the ideas and concepts found in the areas of engineering, science, and mathematics. Topics include teaming and communication skills, 3D modeling, manufacturing processes, dimensioning and conventional tolerancing, sectional views, auxiliary views, and pattern development. This course is demanding, requiring the application of complex visualization and computer skills. These skills are used to assess, communicate, and design virtual and physical models used in science, mathematics, manufacturing, transportation, and structural systems (North Carolina Department of Public Instruction, 2005). The principles learned were applied using a constraint-based modeling program provided by the local education agency. The Drafting II-Engineering classes in this research were taught in the spring semester of 2009. Student participant demographic data was collected and summarized in Table 1. 76 JOURNAL OF STEM TEACHER EDUCATION Table 1. Demographics of Participants Category Frequency Percent Gender Female 8 8.7 Male 84 91.3 Class Freshman 3 3.2 Sophomore 22 23.9 Junior 41 44.6 Senior 26 28.3 The make-up of the participants in this study enrolled in Drafting II-Engineering during the spring semester of 2009 was 8 females and 84 males. These students ranged from 14-19 years old and included 3 freshman, 22 sophomores, 41 juniors, and 26 seniors. There were approximately ten times as many males as females participating in the study. A very low percentage of participants were classified as freshmen, likely due to the enrollment restriction of Drafting II-Engineering until completion of the pre-requisite Engineering/Technical Graphics I course. A high percentage of participants was classified as juniors and seniors primarily due to engineering/technical graphics instructors suggesting that students complete a geometry course that is traditionally a sophomore level math prior to enrolling in Drafting II- Engineering. Performance and Cognitive Assessment in 3-D Modeling 77 Data Collection A research proposal was submitted to the Institutional Review Board in order to gain approval for the study. Next, the study topic was discussed with local education agency professionals and permission of involvement in the research study was granted. During the traditional school year, high school level career and technical education teachers in the local education agency meet quarterly with their respective professional learning communities. This venue was utilized to discuss the topic of the study and spur interest among the drafting/aerospace instructors. After gaining IRB approval a survey was handed out to all Drafting II-Engineering course instructors. The survey inquired about the instructors’ participation interest in serving as test administrators and the approximate student enrollment for the spring semester. All Drafting II-Engineering course instructors volunteered to participate in the study and serve as test administrators for the 92 student participants. Following the meeting, an email was sent to the surveyed instructors to provide further details about the study and to finalize the list of participating instructors. During the next drafting/aerospace meeting the test materials were provided to the instructors and explained in detail to ensure an efficient process. The test materials included the following items: 1) instructions numerically outlining test administration procedures, 2) the cognitive assessment including 68 multiple choice items, 3) scanning sheets for students to input their respective answers, 4) the performance assessment including a prescribed 3D-model problem, and 5) a USB flash drive to transport performance assessment data. This study used existing assessments that all students enrolled in Drafting II-Engineering in North Carolina Public Schools would be administered regardless of the presence of