The effect of individual or group guidelines on the calibration accuracy of high school biology students
vi TABLE OF CONTENTS Page LIST OF TABLES vii LIST OF FIGURES ix Chapter I. INTRODUCTION 1 SELF REGULATION AND CALIBRATION 1 METACOGNITION AND SELF-REGULATION IN CALIBRATION 3 NEED FOR RESEARCH IN CALIBRATION 7 DESIGN AND OVERVIEW OF STUDY 10 SUMMARY AND OVERVIEW OF SUBSEQUENT CHAPTERS... 11 II. LITERATURE REVIEW 12 INTRODUCTION 12 METACOGNITION, SELF REGULATION, AND CALIBRATION DEFINED 13 STUDIES INVESTIGATING KNOWLEDGE OF COGNITION 17 CLASSROOM STUDIES IN CALIBRATION 22 SUMMARY OF STUDIES 32 RATIONALE FOR STUDY 33 RESEARCH QUESTIONS AND HYPOTHESES 35 SUMMARY 36 III. METHODOLOGY 37 INTRODUCTION 37 PARTICIPANTS 38 DESIGN 39 MEASURES 40 PROCEDURES 41 SUMMARY 43 IV. RESULTS 44 INTRODUCTION 44 OVERALL DESCRIPTIVE FINDINGS 46 STATISTICS FOR INDIVIDUAL TEST RESULTS 48 STUDENT RESPONSES TO GUIDELINES 65 CONFIDENCE LEVELS 65 FACTORS INFLUENCING UNDERSTANDING 70
vii AREAS OF STRENGTH IN UNDERSTANDING 73 AREAS OF WEAKNESSES IN UNDERSTANDING 76 V. DISCUSSION 79 INTRODUCTION 79 ACHIEVEMENT 79 CALIBRATION ACCURACY 79 EFFECTS OF GUIDELINES AND SETTING 81 QUALITATIVE RESULTS 88 LIMITATIONS 92 DIRECTIONS FOR FUTURE RESEARCH 94 IMPLICATIONS FOR PRACTICE 96 SUMMARY AND CONCLUSIONS 97 REFERENCES 100 APPENDICES A. CALIBRATION GUIDELINES 107 B. PREDICTION AND POSTDICTION SHEETS 109 C. PARENT APPROVAL LETTER 110 VITA 111
viii LIST OF TABLES Table Page 1. Overall Descriptive Statistics for Actual Score, Prediction Accuracy, and Postdiction Accuracy 47 2. Results of MANOVA (Wilks' Lambda) for Treatments on Calibration Accuracy and Test performance For Test 1 48 3. Follow-up ANOVA results for Treatments on Calibration Accuracy and Test Performance for Test 1 49 4. Means and Standard Deviations for Main Effect for Setting on Achievement for Test 1 50 5. Means and Standard Deviations for the Interaction of Setting and Guidelines on Achievement for Test 1 51 6. Results of MANOVA (Wilks' Lambda) for Treatments on Calibration Accuracy and Test Performance for Test 2 53 7. Follow-up ANOVA Results for Treatments on Calibration Accuracy and Test Performance for Test 2 54 8. Means and Standard Deviations for Main Effect of Setting on Achievement for Test 2 55 9. Means and Standard Deviations for Main Effect of Guidelines on Achievement for Test 2 56 10. Means and Standard Deviations for Main Effect of Guidelines on Prediction and Postdiction Accuracy for Test 2 57 11. Means and Standard Deviations for Significant Interactions of Setting and Guidelines on achievement for Test 2 58 12. Means and Standard Deviations for the Effect of Setting and Guidelines on Postdiction Accuracy for Test 2 60 13. Results of MANOVA (Wilks' Lambda) for Treatments on Calibration Accuracy and Test Performance for Test 3 62 14. Follow-up ANOVA Results for Treatments on Calibration Accuracy and Test Performance for Test 3 63
ix 15. Descriptive Statistics for the Effect of Setting on Achievement for Test 3 64 16. Descriptive Statistics for the Effect of Setting on Prediction Accuracy for Test 3 64 17. Confidence Levels for Multiple-Choice and Short Answer Questions for Test 2 67 18. Confidence Levels for Multiple-Choice and Short Answer Questions for Test 3 69 19. Response Categories for What Made the Students More or Less Confident in Their Understanding of the Test Material 70 20. Quotes From Response to What Made the Students More or Less Confident in Their Understanding of the Test Material 72 21. Response Categories for Strength in Understanding of the Test Material 74 22. Quotes From Response to Areas of Strengths in Understanding of the Test Material 75 23. Response Categories for Weaknesses in Understanding of the Test Material 76 24. Quotes from Response to Areas of Weakness in Understanding of the Test Material 78
X LIST OF GRAPHS Graph Page 1. Interaction for Achievement (Test Score) for Test 1 52 2. Interaction for Achievement (Test Score) for Test 2 59 3. Interaction for Postdiction Accuracy for Test 2 61
1 The Effect of Individual or Group Guidelines on the Calibration Accuracy of High School Biology Students CHAPTER I INTRODUCTION This chapter begins with an overview of the constructs of self-regulation and calibration. It presents current definitions for metacogntion and self-regulation as they relate to calibration. The use of group settings and guided practice in calibration is discussed and supported. The need for research in calibration will be presented and followed by the research questions for this study. Finally, a brief overview of the design is proposed. Self-regulation and Calibration The ability to self-regulate one's learning is vital to success in all academic endeavors. Self-regulation uses information from past performance to adjust future performance, and accurate self-evaluations are valued as a guide in regulating behavior in order to accomplish future goals (Radhakrishnan, Arrow, & Sniezek, 1996). Many students lack the ability to estimate their own level of understanding and often this leads to repeated experiences of failure. In fact, students are often dismayed by how poorly they have performed on an assessment for which they believed they were well prepared (Hacker & Bol, 2001). Students who can accurately assess their level of knowledge are in a better position to intensify or redirect their studying for a test, provide self-guidance during reading for better comprehension, or generate self-feedback indicating that a new skill is being properly acquired (Hacker, Bol, & Keener, 2008). Unfortunately, self-regulation of learning is rarely encouraged in the classroom. Many students lack the metacognitive skills that are necessary to regulate learning and
2 make adjustments to their learning techniques as the learning process unfolds. These skills are important for students to develop in order to regulate their own learning and accurately calibrate the level of knowledge they have acquired. Well-developed skills in metacognition—awareness of one's cognitive processes, cognitive strengths and weaknesses, and self-regulation are important for successful academic functioning (Klassen, 2002). Accurate calibration of learning is vital in order to make the needed adjustments to improve the accuracy of understanding of the level of knowledge obtained. Calibration In order to be successful in academic pursuits one must be able to evaluate his or her level of understanding of the material being studied. By being aware of the level of understanding of material students can determine how well they are prepared for success on an evaluation of that material. The accuracy of this understanding can be assessed through calibration investigations. Calibration has been defined as the accuracy with which students can predict their own performance (Hacker, Bol & Bahbahani, 2008). Calibration accuracy has been used in studies to evaluate many curricular areas, including reading comprehension. Readers whose predictions and performance are highly correlated are considered to have good calibration of comprehension, whereas readers whose predictions and performance are minimally correlated are considered to have poor metacomprehension (Hacker, Dunlosky, & Graesser, 1998). Other studies in calibration have used the difference between predicted test scores and actual test scores to evaluate calibration accuracy. When students gain the ability to calibrate their knowledge level it can facilitate
3 improved academic achievement. Garavalia & Gredier (2002) found that students who were accurate grade predictors earned the highest average grade for the course. In addition, they found that grade differences between these students and the comparison group, who were inaccurate predictors, were statistically significant. This is supported by Bol & Hacker (2001) who showed that high-achieving students were more accurate in their calibrations than low-achieving students. High-achieving students may earn high marks because they have developed accurate calibration skills. If this is true, then low- achieving students could improve their performance by developing more accurate calibration techniques. In order to improve calibration skills students need to be exposed to and practice self-regulating techniques. Metacognition and Self-Regulation in Calibration Metacognition Metacognition is a term coined by educational psychologists to describe the various aspects of how a learner processes new knowledge with an explicit understanding and recognition that continual learning is taking place (Orange, 1999). In essence, cognition is the awareness of ones' thought processes, and metacognition is the monitoring of these thought processes. Awareness of metacognition allows students to effectively monitor the acquisition of new knowledge. Researchers are convinced that metacognitive beliefs, decisions, and actions are important, but are quite often overlooked as determinants of success or failure in a wide variety of activities (Garofalo & Lester, 1985). Self-Regulation Self-regulation involves the willingness and ability to effectively manage or direct
one's learning using appropriate strategies and attitudes that help sustain goal-directed behaviors and to ask for assistance when necessary (Orange, 1999). Self-regulation is vital to calibration accuracy because it allows for the ongoing assessment of the progress that is being made towards a goal. The self-regulated process will end with the student being aware of how much knowledge he or she has gained. Calibration accuracy can be used to determine an individual's level of awareness of learned knowledge. Hence, self- regulation can improve calibration accuracy and improved calibration accuracy can result / in improved academic performance. According to Zimmerman (2002), "Self-efficacy beliefs have been found to be sensitive to subtle changes in students' performance context, to interact with self-regulated learning processes, and to mediate students' academic achievement" (p. 82). Teaching students how to self-regulate should be a part of their educational experience. A major goal of education should be to equip students with the intellectual tools, self-beliefs, and self-regulatory capabilities to educate themselves throughout their lifetime (Bandura, 1993). Research has shown that students who are better at calibrating their own level of learning are more successful academically. Unless the instructional environment creates and sustains an appropriate structure for practicing study techniques, it may be particularly difficult to change epistemological stances that undergrid what the student classifies as productive self-regulation (Pintrich, Marx, & Boyle, 1993). There are several methods that can be used to promote metacognition and improve calibration skills. Teachers need to be made aware of metacognitive processes and how they can be improved through classroom instruction. Calibration in Group Settings
5 Group settings provide an ideal situation for fostering metacognitive skills, especially when students are guided towards the development of these skills. Just as teachers should model metacognition, social interaction among students could also be used to cultivate metacognitive capacity. When working in groups' students gain the benefit of hearing how others address and solve problems. Group experiences can be used to guide the students in their individual development of metacognitive skills. If students are encouraged and guided to think critically together, then their spoken reasoning will ideally make these cognitive tools more readily available to them (Martinez, 2006). Teachers who recognize the importance of peers to the learning process encourage and offer opportunities for personal responses and collaborative interactions (Wiseman, 2003). Independent study lacks the dynamically responsive scaffolding and guidance that can be made available when learning proceeds in the context of social interaction (Winne, 1995). Student interactions provide opportunities for metacognitive development as they discuss the material and share their processes of learning new material. Many researchers and practitioners are now convinced that by promoting metacognitive processes during instruction, more durable and transferable learning can be achieved. Tutors, learning assistants, and teachers, for their part can become the student's "metacognitive conscience" by asking questions of the student in order to develop his or her awareness and analytical processes (Taylor, 1999). Having students conduct metacognitive activities in collaborative settings can develop metacognitive skills. Group activities are easily incorporated into the classroom and not only benefit the student metacognitively, but allow the student to learn from his or her peers.
6 Group work with metacognitive processes such as calibration should be incorporated into the classroom setting in order for students to have opportunities to enhance their own understanding of material. Exposure to calibration practice in settings where students can analyze their own calibration techniques as compared to that of others allows the student to make needed adjustments in his own calibration techniques. Providing guiding questions in order to help focus the group on development of calibration accuracy can enhance group review activities. It is important that the questions the students are asking about their level of knowledge are focused on the metacognitive process. Guided Practice in Calibration in Group Settings Metacognitive skills can be further enhanced by guidance from the teacher in the form or verbal or written strategies that help maintain the focus of the collaborative activity on calibration. Students construct strategies from experience but also can be guided by teachers and peers to discover and control the development of effective learning tactics (Paris & Newman, 1990). Peers may bring new insight to the discussion that can help the development of individual calibration skills. The use of guidelines during calibration can enhance learning processes by guiding the student through the metacognitive process of evaluating his or her learning of material. Increasing the student's self-awareness can help the student associate behaviors or successful (or unsuccessful) learning outcomes and aid in the accomplishment of the learning goal (Smith, 2001). The key is to help focus the student on thinking about the learning process and his or her personal goals to increase motivation (Talbot, 1997). It is important for teachers to mediate group work to ensure the focus is on the
7 learning process. Metacognitive skills and knowledge can be acquired, and so, the argument goes, students can "learn how to learn". Providing guidelines for the process of calibration prior to assessment allows the student to not only focus his cognitive processes on the task at hand, but helps in the development of metacognitive skills that are vital to the 21st century learner. Students must have the opportunity to practice and so must be placed in situations that require metacognition. If students are encouraged and guided to think critically together, then their spoken reasoning will ideally make their cognitive skills available to one another (Martinez, 2006). An illustrative study highlights how group work can promote self-reflection and deeper understanding. Cantrell (2002) examined the content of small-group discourse and found that they provided opportunities to reflect further on readings, to clarify understandings, and to share insights from their own experiences. Cantrell also found that in many exchanges between and among the participants in the study, construction of knowledge occurred through deeper comprehension, clarification, and identification of important points. Thomas, Bol, Warkentin, Wilson, Strange and Rohwer (1993) found that one important role teachers play is in prompting student engagement in productive, demand-responsive study activities. Need for Research in Calibration Metacognitive skills have become more important in education as local, state, and national assessments have become the standard for measuring student ability. Student performance on high-stakes tests has an impact on educational placements, grade promotion, academic major, college admissions, graduation, and entry into various professions (Hacker, Bol, & Keener, 2008). Previous research has focused on calibration
8 ability as related to success on these high-stakes tests, but has failed to evaluate calibration practice as related to academic success in high school courses. Since calibration ability has been shown to be related to academic success this skill should be developed early in the educational experience. Research suggests that metacognitive skills can be taught and can subsequently improve academic achievement (Hartley, 2001). Nickerson (1988) stated that there is abiding conviction among many educators that the development of thinking should be a primary goal of education. Although this belief has been prevalent for many decades, few teachers are aware of the need to foster metacognitive skills or the methods that may help them develop these skills in their students. Dahl (2004) states that: "To help pupils with the metacognitive process it is necessary that the teachers are educated to be able to discuss the learning process and strategies with pupils. The development of the pupils metacognition will help the pupil at any level." (pp.153) Whether calibration and other metacomprehension strategies can be improved with instruction remains a question that has not yet been definitively answered (Bol & Hacker, 2001). Previous research has focused mainly on college level investigations into calibration and has failed to adequately address calibration at the high school level. This research helps to fill that gap, and provides the added benefit of studying calibration in a classroom context. Hacker, Bol, & Keener, (2008) argue for the need to go outside the laboratory into more ecologically valid environmental situations in order to effectively evaluate calibration techniques. Research on calibration in group settings is also lacking. Group settings provide
9 students the time to reflect on their learning in situations where individual reflections can be enhanced by group discussions. Orange (1999) found that using peer models to teach self-reflection was effective. By working in group settings and observing both successful and unsuccessful peers students may have become more aware of their own academic shortcomings and may have become more willing to modify their own behavior (Orange). Group interactions provide opportunities for students to seek help from their peers in self-regulatory processes. Research has shown that students who effectively monitor their overall use of self-regulation strategies seek help more often from peers, teachers, and parents and learn more than students who do not seek help (Zimmerman, 2008). By allowing instructional time for group review of material prior to testing, teachers allow opportunities for students to seek help who may otherwise not have done so. The success of group interactions on academic performance can foster continued use of help-seeking strategies that result in higher self-regulation skills. The use of guidelines in group settings offers unique opportunities to improve calibration skills. Guidelines have the potential to focus the student on the metacognitive process and to help the student develop a pathway to the successful calibration of knowledge. Many students lack the ability to successfully reflect on their level of knowledge and need to be guided through the process in order to develop this skill. The use of guidelines in group settings has the added benefit of allowing the student to hear how others calibrate their level of knowledge. This study will focus on the use of group interactions and calibration guidelines to foster the development of successful calibration skills. This study is unique in that it combines group investigations into calibration with the use of guidelines to foster
10 metacognitive skills. Previous research is lacking in studies that look at the interactions between these two variables. Research Questions The research questions for this dissertation focus on the effects of calibration practice in either group or individual settings and with or without guidelines on calibration accuracy and achievement of high school biology students. In addition, written responses to guided questions from the group calibrating with guidelines will be collected. This will offer more insight into the effectiveness of the use of guidelines in the collaborative process of calibration as it is proceeding in the group settings. More specifically, the following research questions will be addressed: 1. Does receiving guidelines during calibration practice improve calibration accuracy and achievement for high school biology students? 2. Is calibration practice in groups more effective than individual practice in improving calibration accuracy and achievement for high school biology students? 3. How do guidelines and learning settings (group vs. individual) interact to affect calibration accuracy and achievement? 4. What do students write in response to guided questions designed to improve calibration? Design and Overview of the Study A quasi-experimental research on the effects of calibration practice in either group or individual settings and with or without guidelines on calibration accuracy and achievement in a high school biology course was conducted. A fully crossed factorial
11 design was employed. Four intact biology classes were involved in the study; two classes participated in group calibration, with one class receiving group calibration guidelines and one class calibrating without guidelines; two classes participated in individual calibration, with one class receiving individual calibration guidelines and one class calibrating individually without guidelines. The data collected consists of predictions and postdictions for three different testing occasions. In addition, qualitative data was collected in the form of responses to the calibration questions from the class that participated in group calibration with guidelines and the class that participated in individual calibration with guidelines. Quantitative data was analyzed and reported using multivariant analysis of variance (MANOVA) and factorial analysis of variance (ANOVA). Qualitative data consisting of responses to guiding questions was analyzed via content analysis. Summary and Overview of Subsequent Chapters Chapter I has provided a rationale and the accompanying research questions that were addressed in this study. Metacognition and calibration were briefly defined and will be more fully explored in Chapter II. Chapter II investigates the current definitions attributed to metacognition, self-regulation, and calibration. It summarizes important findings in these areas as related to education, and compares findings from previous empirical research in metacognition, self-regulation, and calibration. Emphasis is placed on calibration studies. The need for more research into calibration at the high school level, specifically in science, is supported. The hypotheses for the research questions are addressed in Chapter II. Chapter III further and more completely outlines the methodology that was used for this research.
12 CHAPTER II Literature Review Introduction There exists a substantial amount of research that investigates the use of metacognition, self-regulation, and calibration in educational settings. However, there is great variation as to how these constructs are operationally defined and delivered in the classroom setting. In addition, a discrepancy exists between those studies conducted in laboratory settings and those studies conducted in traditional classroom settings. The majority of previous research has focused on metacognition in non-traditional classroom settings. This chapter provides a brief overview of metacognition, self-regulation, and calibration. Previous studies in these areas are outlined, and an overview of studies in these areas are presented in order to empirically investigate these constructs. Emphasis will be placed on research in calibration studies. Studies investigating student knowledge of cognition are presented and followed by studies specifically focused on calibration. Previous classroom studies in calibration without interventions and previous classroom studies in calibration with interventions are presented and discussed. Calibration interventions are reviewed including studies that investigate the use of incentives and reflections, practice tests, group work, and peer interactions on calibration accuracy. Studies investigating achievement level and calibration accuracy are also presented. The need for more research into calibration at the high school level, specifically in science, is supported. Chapter II presents the research questions addressed in this research, and the proposed hypotheses for them. This chapter ends with a brief
13 overview of Chapter III. Metacognition, Self-regulation, and Calibration Defined It is difficult to state a clear definition of metacognition, self-regulation, and calibration. In other words, metacognition, self-regulation, and calibration are all terms that help in defining each other. An overview of each of these cognitive domains is necessary in order to understand each individually. Metacognition Piaget referred to the process of "reflexive abstraction" as a mechanism for extracting, reorganizing, and consolidating knowledge (Garofalo & Lester, 1985). This definition could easily be used to describe metacognition as well. Garofalo & Lister argue that it is difficult to separate what is metacognitive from what is cognitive. Metacognition experiences are defined by Hacker, Dunlosky, & Graesser (1998) as being concerned with one's awareness of his or her cognitive or affective processes and whether progress is being made toward the goal of a current process. In other words, metacognition is the ability of students to think about their level of knowledge attainment as they are investigating new information. Without adequate and appropriate cognitive processing it is impossible to successfully engage in metacognition. To distinguish between cognition and metacognition Nelson and Narens (1990) offered the following distinctions: (1) Mental processes are split into two or more specifically interrelated levels, a cognitive level and a metacognitive level; (2) the metacognitive level contains a dynamic model of the cognitive level; and
14 (3) there are two dominance relations called control and monitoring, which are defined in terms of the direction of flow of information between the metacognitive and cognitive levels. As outlined above, it is easy to see that these processes are interrelated and cannot exist independently of each other. Grimes (2002) defines metacognition as a term coined by educational psychologists to describe the various aspects of how a learner processes new knowledge with an explicit understanding and recognition that learning is taking place. With metacognition the learner is not only aware he is learning, but is aware of how that learning is proceeding. Grimes summarizes the process as one that involves the abilities to appraise and manage the internal aspects of learning. Hence, for students to be successful in metacognition, they must be continually analyzing the effectiveness of their monitoring of cognitive strategies and not just be engaging in the use of these strategies. According to Martinez (2006) the metacognitive process is the monitoring and control of thought. He identifies three major categories of metacognition: metamemory and metacomprehension (the understanding of one's own knowledge state), problem solving (the pursuit of a goal when the path to the goal is uncertain), and critical thinking (evaluation ideas for their quality- especially judging whether or not they make sense). It is clear that all learning involves metacognition. One way of viewing the relationship between cognition and metacognition is that cognition is involved in doing, whereas metacognition is involved in choosing and planning what to do and monitoring what is being done (Garofalo & Lester, 1985).
15 Self-regulation The process of self-regulation has been defined in various ways, but they all refer to the ability to regulate one's learning process. According to Zimmerman (1986), "Self- regulated learning is the degree to which students are metacognitively, motivationally, and behaviorally active participants in their own learning process" (p. 308). Orange (1999) summarized self-regulation as the willingness and ability to effectively manage or direct one's learning using appropriate strategies and attributes that help sustain goal- directed behaviors and to seek assistance when necessary. Self-regulation would not be possible without metacognition. One must be able to monitor his or her level of understanding in order to be successful in the self-regulation of that learning. According to Butler and Winne (1995) "In academic contexts, self-regulation is a style of engaging with tasks in which students exercise a suite of powerful skills: setting goals for upgrading knowledge; deliberating about strategies to select those that balance progress toward goals against unwanted costs; and, as steps are taken and the task evolves, monitoring the accumulating effects of their engagement." (pp. 245) As self-regulated learners engage in academic tasks, they draw on their knowledge and beliefs to construct an interpretation of a task's properties and requirements (Butler & Winne, 1995). Once these properties and requirements are decided upon, to be successful in completing them the learner must continue to evaluate his level of understanding. In essence, the learner cannot be successful in self-regulation without engaging in metacognitive behaviors as well. Metacognitive behaviors regulate the learning process that leads to self-regulation of knowledge retention as the learning