DESCRIBING THE COGNITIVE LEVEL OF PROFESSOR DISCOURSE AND STUDENT COGNITION IN COLLEGE OF AGRICULTURE CLASS SESSIONS John C. Ewing, Assistant Professor The Pennsylvania State University M. Susie Whittington, Associate Professor The Ohio State University Abstract The purpose of this study was to describe the cognitive level of professor discourse and student cognition during selected college of agriculture class sessions. Twenty-one undergraduate class sessions were videotaped in 12 professors’ courses. Results were interpreted to show that professors’ discourse was mostly (62%) at the knowledge and comprehension levels of cognition, the lower levels of cognitive thought (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956). During the 21 class sessions, 1,448 student thoughts were recorded. Forty percent (n = 564) of those thoughts were found to be engaged. Engaged thoughts were then analyzed using Bloom’s Taxonomy (1956). Sixty-two percent of the total 564 engaged thoughts occurred at the knowledge and comprehension levels of cognition. The cognitive levels of professor discourse and student cognition (engaged thoughts categorized and weighted using Bloom’s Taxonomy) were mostly at the two lowest levels of Bloom’s Taxonomy, knowledge and comprehension. Recommendations included professor and student awareness of the cognitive levels at which they teach and think, and further study of relationships between professor and student variables. Introduction prepared sufficiently to think beyond the lower levels of cognition. Stakeholders, such Several reports in the 1980s questioned as future employers, want students that are the quality of undergraduate education and able to think critically and analyze identified the need for instructional information that has been presented to them improvement (Paulsen & Feldman, 1995). (Education Commission of the States, 1995). Critics of higher education believe that the In fact, during class sessions, students often university system is failing in the cognitively engage at levels below what preparation of students (Tom, 1997). Kerr professors are teaching (Lopez, Whittington, (2001) noted that teaching at many research Schley, & Fisher, 1999). institutions had generally deteriorated. The Because what occurs in the classroom critics of current undergraduate education has a great effect on creating curiosity on the encourage institutions to reflect on that part of the student (National Center on which is currently being done and make Postsecondary Teaching, Learning, and changes that would otherwise be overlooked Assessment [NCPTLA], 1995), McKeachie (Tom). (2002) believed that professors should allow Halpern (1993) emphasized that teaching students to develop thinking skills through students to think critically was the desired using the content, rather than by pushing- outcome of undergraduate education. Brown through the content. Shulman (2000) stated and Lane (2003) wrote that universities and that students are not blank slates that need to colleges must examine what is occurring in be written upon, but rather they are people their classrooms and be ready to produce that need opportunities to take the evidence of what has occurred. However, information that is presented, and work with the Boyer Commission on Educating it to make sense of it in their own way. The Undergraduates in the Research University NCPTLA believed that faculty can make (1998) believed that students were not being changes that will improve education at the Journal of Agricultural Education 36 Volume 50, Number 4, 2009 Ewing & Whittington Describing the Cognitive Level… undergraduate level. Nordvall and Braxton Theoretical Framework (1996) recommended examining course- level academics to identify institutional ―Cognitive development is much more quality, and suggested Bloom‘s Taxonomy than the addition of new facts and ideas to (1956) for assessing level of understanding an existing store of information‖ (Woolfolk, related to course content. 2007, p. 27). Piaget‘s (1970) theory of Bloom et al. (1956) defined higher order cognitive development stated that thinking is thinking as application, analysis, synthesis, influenced by maturation, activity, and and evaluation; the four levels of Bloom‘s social transmission. Piaget theorized that hierarchy above the two lower levels of teachers have little impact on the maturation knowledge and comprehension. According influence, but that through the activity to Bloom et al., knowledge, the lowest level influence, teachers provide exploration, of cognition, requires an individual to recall observation, testing, and information and remember facts and situations. organization, all of which are likely to alter Comprehension, the next level in Bloom‘s thinking processes. In addition, Piaget hierarchy, is described as demonstrating a believed that teachers impact the social general understanding of the content. These transmission influence, learning from others, two lower levels of cognition are necessary depending on the stage of cognitive in the thinking process to lay the foundation development the student has reached. from which to proceed cognitively through Whittington and Bowman (1994) the hierarchy. However, according to Bloom assessed the cognitive level of instruction of et al., the four higher levels of cognition, faculty members in a college of agriculture when reached, will cognitively challenge and found that instructors were mainly students beyond knowledge and teaching at the remembering level of comprehension and thus, theoretically, aid cognition. Whittington (1995) found that students‘ long-term content retention. professors sought to teach at all levels of Bloom et al. (1956) further wrote that cognition, but much of the discourse was at application is the process of using lower levels of cognition. Whittington, Stup, information gained in one situation to solve Bish, and Allen (1997) examined the a problem or problems in other situations. thinking opportunities provided by Analysis, Bloom et al. contend, involves professors through cognitive discourse. A taking the information at hand and working majority (80%) of the classroom discourse with it in a way that the learner notes was found to be at the knowledge and relationships and then organizes those comprehension levels of cognition. relationships in meaningful ways. The next Building upon Piaget‘s (1970) activity level, synthesis, requires learners to assess influence from his cognitive development the component parts of the given theory, classroom activity is likely to impact information and to formulate the component students‘ thinking, in other words, students‘ parts into a new whole. The highest level of classroom cognition. The cognitive level of cognition, as described by Bloom et al., is classroom activity can be framed using evaluation. Evaluation is the level at which Bloom’s Taxonomy of Educational the learner makes judgments about the Objectives: Handbook 1, Cognitive Domain material presented. (Bloom, 1964), which is useful for Two of the most common uses of documenting the cognitive levels at which Bloom‘s Taxonomy have been classifying teachers and learners process classroom objectives, and testing items (Krathwohl, content. Bloom et al.‘s (1956) six-step 2002). Bloom et al. (1956) stated that the hierarchical system of thought processing taxonomy was designed for classifying ― … moves from the knowledge level, student behaviors which represent the emphasizing subject matter recall, to the intended outcomes of the educational evaluation level, that entails making process‖ (p. 12). The authors of the judgments (Table 1). Each level is reflected taxonomy believed that student behaviors through cognitive classroom activity. Given could be seen and classified in a variety of that learning is enhanced by increasing the classes and levels of education. percentage of cognitive activity occurring at Journal of Agricultural Education 37 Volume 50, Number 4, 2009 Ewing & Whittington Describing the Cognitive Level… the higher levels of Bloom‘s Taxonomy, enhance the quality of teaching and learning this framework provides focus and in their classes (Whittington & Bowman, direction to teachers who desire to 1994). Table 1 A Synopsis of Bloom’s Hierarchy of Thought Processing Cognitive level Definition Activity Knowledge Recalling subject matter List, define, label, and match Comprehension Learners know information that has Explain, rewrite, paraphrase, been communicated, but cannot apply in summarize, and give examples other situations Application Learners apply information to different Compute, demonstrate, use, situations and learning tasks predict, discover, and solve Analysis Learners separate data into its component Differentiate, discriminate, parts; these parts are differentiated and relate, diagram, and distinguish related based on their relationship Synthesis Combines learned elements to create a Create, compose, produce, and new whole; working into pieces and develop elements, arranging so as to create new forms, patterns, or structures Evaluation Entails making judgment on the value of Justify, compare, contrast, materials and methods for given evaluate, and interpret purposes Note. McCormick and Whittington (2000). Adapted from Bloom et al. (1956). As part of an ongoing line of inquiry techniques used, cognitive level of professor related to cognitive levels of teaching and questions, and cognitive level of course learning at the university level, the cognitive objectives), a student variable (cognitive level of professor discourse, classroom level of student questions), and three course engagement, and ultimately student variables (level of the course, class size, and cognition (frequency of classroom time of day at which the course was offered) engagement categorized and weighted using are part of the overall conceptual framework Bloom‘s Taxonomy) were examined in this that has been developed to examine the study (see bold/italicized items in Figure 1). impact of these variables on student Other professor variables (teaching cognition during class sessions (Figure 1). Journal of Agricultural Education 38 Volume 50, Number 4, 2009 Ewing & Whittington Describing the Cognitive Level… Professor Cognitive Level of Discourse (Bloom’s Taxonomy, 1956) Teaching Techniques (Newcomb, McCracken, Warmbrod, & Whittington, Student Cognition 2004 Knowledge Cognitive Level of Questions Comprehension (Bloom‘s Taxonomy, 1956) Application Analysis Cognitive Level of Course Objectives Synthesis (Bloom‘s Taxonomy, 1956) Evaluation Student Cognitive Level of Questions (Bloom‘s Taxonomy, 1956) Classroom Engagement - Frequency of Thoughts Course Level (100, 200, 300, 400) Class Size - Number of students Time of Day Figure 1. Conceptual framework of proposed factors influencing student cognition. Journal of Agricultural Education 39 Volume 50, Number 4, 2009 Ewing & Whittington Describing the Cognitive Level… Purpose and Objectives explain the study and to seek their participation. The purpose of this study was to Twelve nominated faculty members describe professor and student cognition from seven departments in the college of during selected college of agriculture class agriculture participated in the study. The sessions at The Ohio State University. researcher attempted to observe and Specific objectives guiding the study were videotape two class sessions for each to: professor. However, scheduling conflicts prevented two observations for every 1. Describe the cognitive level at which professor. Thus, all professors were professors delivered in-class observed once, and nine professors were discourse as measured using the observed twice. Consequently, 21 Florida Taxonomy of Cognitive undergraduate class sessions in horticulture, Behavior (FTCB; Webb, 1970). animal science, food science, environmental 2. Describe student cognition as and natural resources, and plant pathology measured using think-aloud protocol were observed for the study. transcripts, and then weighted using Twenty-one students who enrolled in the Bloom‘s Taxonomy. participating professors‘ courses were randomly selected, one from each Definitions of Terms professor‘s class roster, to participate in the study. Researchers videotaped all selected Engagement – The frequency of student class sessions. By watching each videotaped thoughts related to course content class session and completing the FTCB during a class session (Ewing, 2006). (Webb, 1970), the researcher determined the cognitive level of professor discourse. Professor discourse – The verbal The FTCB (Webb, 1970; Brown, Ober, statements spoken by professors Soar, & Webb, 1966), based on Bloom‘s during class sessions (Whittington & Taxonomy (1956), identifies and quantifies Newcomb, 1990). teacher behaviors into cognitive levels. The categories of the FTCB are identical to Student cognition – The frequency of Bloom‘s Taxonomy, except the FTCB classroom engagement categorized divides the comprehension level of the and weighted using Bloom‘s Taxonomy into translation and Taxonomy (Ewing, 2006). interpretation. The FTCB was designed to measure the frequency of the presence of Methods each behavior during 6-minute observation periods to determine the extent to which the The researcher cooperated with the emphasis was on acquiring information senior associate dean and director of versus using cognitive processes (Webb, academic affairs in the College of Food, 1970). Agricultural and Environmental Sciences to In accordance with the instrument acquire support for conducting the study in protocol, each professor behavior was every department in the college. The recorded only once for each 6-minute associate dean sent a letter to all department observation period, even if the behavior was chairs (N = 8) in the college, describing the observed more than once during the time study and asking for their participation. The period. Any behavior observed, which researcher then met with the department represented more than one category, was chairs, explained the study, and asked them given a checkmark for all categories that to nominate three faculty members from applied. A weighting system, employed by their departments who were deemed ―good‖ Pickford (1988), was implemented to give teachers based on student evaluations, exit higher levels of cognition more weight due interviews, and annual reviews of teaching. to the cognitive level of processing required Individual appointments were scheduled to perform the task. The weighting factors with the professors who were nominated, to are consistent with the general support given Journal of Agricultural Education 40 Volume 50, Number 4, 2009 Ewing & Whittington Describing the Cognitive Level… to the hierarchical nature of Bloom‘s percent of professor discourse at each Taxonomy (1956). The weighting factors cognitive level to yield a cognitive weighted were developed in consultation with score for professor discourse at each level of Krathwohl (2002); synthesis and evaluation cognition. The cognitive weighted scores for were weighted equally due to some professor discourse at each level of discrepancy in which level required greater cognition were then summed up to yield a cognitive capacity (Newcomb & Trefz, total cognitive weighted score for professor 1987). The frequency of observed behaviors discourse during each class session. The per cognitive level was summed up to give a weighting score for each professor could total frequency of observed behavior per range from a score of 10 to 50. A score of 10 professor. A percentage of behaviors at each would indicate that all of the professor‘s cognitive level was calculated by dividing discourse was at the knowledge level of the frequency of behaviors at each cognitive cognition, whereas a score of 50 would level by the total frequency of behaviors indicate that all of the professor‘s discourse observed during the class session. This was at the higher levels of cognition, yielded the percent of professor discourse at synthesis or evaluation. The cognitive each cognitive level. weighting factors used for professor The cognitive weighting factor for each discourse and a sample calculation are level of cognition was then multiplied by the displayed in Table 2. Table 2 Cognitive Weighting Factors and a Sample Calculation of Professor Z’s Total Cognitive Weighted Score for Professor Discourse f of Percent Cognitive Level of professor of Weighting weighted cognition behaviors behaviors X factor score Knowledge 28 31.80 X .10 3.18 Translation 8 9.09 X .20 1.82 Interpretation 12 13.60 X .25 3.40 Application 18 20.40 X .30 6.12 Analysis 5 5.70 X .40 2.28 Synthesis 14 15.90 X .50 7.95 Evaluation 3 3.40 X .50 1.70 Total 88 26.45 Note. Possible range for total cognitive weighted score for professor discourse (10-50). Journal of Agricultural Education 41 Volume 50, Number 4, 2009 Ewing & Whittington Describing the Cognitive Level… Pickford (1988) believed that reliability about past experiences prompted by of the FTCB was dependent upon the rater‘s class subject matter, (5) deeper utilization of the instrument. The rater learning/questioning thoughts, (6) thoughts received training from a researcher with about behavior that received/maintained extensive experience in using the FTCB. attention. Training involved an explanation of the Reliability for the process was instrument followed by practice evaluating established using a sample transcript and videotaped class sessions. The trainer was recording the frequency of engagement present during the first practice rating to during the class session. The Pearson answer questions for the researcher. After product-moment coefficient was then the training, intra-rater reliability (a measure calculated. The intra-rater reliability for of rater consistency) was assessed using two engagement was r = .92. Another (3weeks) videotapes of teaching. The overall intra- individual who was familiar with rater reliability was r = .91. Inter-rater engagement and teaching/learning analyzed (9weeks) reliability was established by asking an a sample transcript to establish inter-rater expert in cognition research to complete an reliability. The inter-rater reliability for evaluation of a sample videotape. The inter- engagement was r = .89. Two students, who rater reliability was r = .94. The FTCB is had studied and been trained in cognition based upon Bloom‘s Taxonomy (1956); research, analyzed face and content validity therefore it is argued that the FTCB is for this instrument. The raters indicated that content valid given that Bloom‘s Taxonomy the instrument was useful for categorizing is generally supported as a way to identify engagement. behaviors of teachers and students at various Engaged thoughts were those that cognitive levels (Pickford). elicited cognitive engagement with the ―Classroom engagement‖ was recorded course content and were therefore classified based on completion of think-aloud into one of the six levels of Bloom‘s protocols. To collect these data, a student Taxonomy (1956), and a frequency was from each videotaped class session was determined for each cognitive level. asked to watch the videotape of the class Thoughts that were categorized as thought- session that he/she had just experienced and type 1 or thought-type 2 were not included to complete a think-aloud protocol. in the analysis for student cognition. A Completing a think-aloud protocol, which percentage for engagement at each level of was done immediately following the class Bloom‘s Taxonomy was then calculated by session, required students to verbalize that dividing the frequency of student thoughts at which they had been thinking during the each cognitive level by the total number of class session, into a hand-held cassette tape- engaged thoughts during the class session. recorder. Each student was reminded that The cognitive weighting factor (Table 3) for the brain is constantly thinking; thus there each cognitive level was then multiplied by should be few moments when they were not the percentage of student thoughts at each speaking while watching the videotape. The cognitive level to yield a cognitive weighted students were instructed to let the cassette score for student cognition at each cognitive recorder run during the entire session; level. The cognitive weighted scores for however, the videotape could be stopped or student cognition at each level of cognition rewound at any time to allow for the student were summed up to yield a total cognitive to gather their thoughts. weighted score for student cognition. All student thoughts were then After a 3-week period, reliability was transcribed, and categorized into one of six established by using a sample transcript and thought-types. The six thought-types, based recording the cognitive level for each on previous research (Whittington, Lopez, student thought during the class session. The Schley, & Fisher, 2000; Shertzer, Ewing, & Pearson product-moment coefficient was Whittington, 2005), were: (1) thoughts or then calculated. The intra-rater reliability for observations about the professor, (2) student cognition was r = .94. Another (3weeks) nonsense or unrelated thoughts, (3) thoughts individual familiar with cognitive levels of connected to previous learning, (4) thoughts learning completed inter-rater reliability. Journal of Agricultural Education 42 Volume 50, Number 4, 2009 Ewing & Whittington Describing the Cognitive Level… The inter-rater reliability for student yielded 126 thoughts during the class cognition was r = .91. Content validity for session. Eighty-six (68.25%) of these were this instrument was based upon its direct at the knowledge level, 19 (15.08%) were at development from Bloom‘s Taxonomy the comprehension level, 15 (11.9%) were at (1956) and the support, from theory and the application level, 4 (3.2%) were at the evidence (Ary, Jacobs, & Razavieh, 2002), analysis level, 1 (.75%) was at the synthesis generally given to the hierarchy of cognitive level, and 1 (.75%) was at the evaluation behaviors. level. Student Z yielded a total cognitive A sample calculation for student weighted score for student cognition of cognition is completed in Table 3. Student Z 15.37. Table 3 Cognitive Weighting Factors and a Sample Calculation of Total Cognitive Weighted Score for Student Cognition Frequency Percent Cognitive of student of student Weighting weighted Level of cognition thoughts thoughts X factor score Knowledge 86 68.25 X .10 6.83 Comprehension 19 15.08 X .20 3.02 Application 15 11.9 X .30 3.48 Analysis 4 3.2 X .40 1.28 Synthesis 1 .75 X .50 .38 Evaluation 1 .75 X .50 .38 Total 126 15.37 Note. Possible range for total cognitive weighted score for student cognition (10-50). Findings levels of cognitive thought and knowledge and comprehension as that which requires Findings related to professors‘ in-class low levels of cognitive thought. Almost discourse for each class session are two-thirds (62%) of the total cognitive displayed in Table 4. The total cognitive weighted scores for professor discourse weighted score for professor discourse were at the two lowest levels of ranged from a low of 14.47, which means cognition (knowledge and comprehension), that the total cognitive weighted score for which include translation and interpretation professor discourse was between the on the FTCB. Approximately 38% of the knowledge and comprehension levels total cognitive weighted scores for of cognition (lower levels), to a high of professor discourse were at the four 30.67, which means that the professor‘s higher levels of cognition (application, discourse was at the application level of analysis, synthesis, and evaluation). cognition (higher level). Bloom et al. (1956) No professor‘s total cognitive weighted defined application, analysis, synthesis, scores were above the analysis level of and evaluation as that which requires high cognition. Journal of Agricultural Education 43 Volume 50, Number 4, 2009 Ewing & Whittington Describing the Cognitive Level… Table 4 Total Cognitive Weighted Score for Professor Discourse as Measured using the FTCB by Class Session Cognitive Class weighted session Knowledge Translation Interpretation Application Analysis Synthesis Evaluation score 20 2.16 .80 5.60 2.64 8.64 9.80 1.00 30.67 21 2.88 2.04 .85 3.57 8.12 11.80 .85 30.16 11 3.23 4.50 7.25 0.00 5.20 1.50 0.00 21.70 6 4.67 2.88 1.95 4.65 4.90 1.00 .50 20.55 5 4.94 2.04 3.20 4.71 4.32 .50 .50 20.21 9 5.63 1.46 4.05 1.50 7.08 0.00 0.00 19.72 8 5.70 1.80 3.60 .39 6.76 0.00 .65 18.90 14 5.47 2.20 3.90 1.41 5.60 0.00 0.00 18.58 17 5.26 2.94 3.18 3.15 2.52 1.50 0.00 18.55 3 5.60 1.90 3.75 1.20 4.80 .50 .50 18.25 7 5.80 3.20 3.25 0.00 1.92 4.00 0.00 18.17 10 5.53 2.56 4.53 0.00 5.52 0.00 0.00 18.14 12 5.15 3.00 4.55 1.80 3.60 0.00 0.00 18.14 19 5.83 1.94 5.55 .84 2.80 0.00 0.00 16.96 2 5.67 3.80 3.60 1.40 1.90 .50 0.00 16.77 13 6.20 3.44 2.50 1.02 2.80 0.00 0.00 15.96 1 6.13 3.06 4.03 .90 1.60 0.00 0.00 15.72 4 6.30 2.80 4.65 0.00 1.86 0.00 0.00 15.61 18 6.34 2.68 3.35 2.19 .96 0.00 0.00 15.52 15 6.20 1.76 7.23 0.00 0.00 0.00 0.00 15.21 16 7.01 1.80 4.23 .39 1.04 0.00 0.00 14.47 Total 111.70 52.60 84.80 31.76 81.70 31.10 4.00 397.96 Note. Possible range for total cognitive weighted score for student cognition (10-50). Journal of Agricultural Education 44 Volume 50, Number 4, 2009 Ewing & Whittington Describing the Cognitive Level… The findings related to student cognition Conclusions/Implications/ are outlined in Table 5. The frequency of Recommendations student thoughts at each level of cognition are summarized. A total of 1,448 student Professor discourse was primarily at the thoughts were recorded from the think-aloud two lowest levels of cognition, knowledge protocol sessions. Approximately 60% (n = and comprehension. If students are to 884) of the observed thoughts were think at higher cognitive levels, professors categorized as nonsense or random thoughts, need to provide discourse at levels at which did not elicit engagement with the which students are appropriately course content; therefore, 564 student challenged. Professors who consistently thoughts from the 21 class sessions were deliver discourse at lower cognitive levels analyzed for student cognition. Three may not be providing opportunities for hundred and fifty (62.1%) of the student students to think at the higher levels thoughts analyzed for cognitive level were of cognition during class sessions categorized into the two lowest levels of (Blosser, 2000). Thus, professors should cognition (knowledge and comprehension). not expect students to operate at cognitive Two hundred and fourteen (37.9%) student levels that are higher than those to which thoughts were categorized into the higher they are currently being challenged. cognitive levels, with 53 (9.4%) student Professors must first be made aware of thoughts categorized at the application level the cognitive levels at which they are of cognition, 96 (17%) student thoughts at currently offering classroom discourse, and the analysis level of cognition, 29 (5.1%) then adjust their level of discourse student thoughts at the synthesis level of accordingly. Once professors are aware cognition, and 36 (6.4%) student thoughts at of the cognitive level at which their the evaluation level of cognition. Student content is being delivered, a comparison can thoughts that were cognitively categorized be made to the course, and to the ranged from a low of one thought during individual lesson objectives, to examine if class session number 13 (Professor G) to a the cognitive level of professor discourse high of 61 thoughts that elicited cognitive accomplishes the purposes of the course. thought during class session number 9 Students were thinking during class (Professor E). sessions, but students were not engaged Class session number 16 (Professor I) with the course content during the yielded the lowest total cognitive weighted majority of the class sessions. At any score for student cognition of 14.8, while given time during a class session, over class session number 7 (Professor D) yielded half of the students in the class were the highest total cognitive weighted score thinking about something other than the for student cognition of 32.35. A total topic at hand (Shertzer et al., 2005). cognitive weighted score for student Students need to recognize that their cognition of 14.8 is between the knowledge individual learning may be affected by and comprehension levels of cognition, the fact that they are engaged in class while a total cognitive weighted score for content less than half of the class session. student cognition of 32.25 is at the Professors must be made aware that application level of cognition. Three of the students elicit more random and 21 courses yielded a percent of student nonsense thoughts during a class session thoughts at or above the application level of than cognitively engaged thoughts and must cognition. No classes yielded a total plan for adding variability to cognitive weighted score for student content delivery (Rosenshine & Furst, cognition above the analysis level of 1971). cognition. Journal of Agricultural Education 45 Volume 50, Number 4, 2009