Examining the Factors that Promote Long-Term Change in Elementary Teachers' Instructional Practices: Sustaining Formative Assessment Reform
Table of Contents Chapter I: Introduction 1 Background of the Study 2 First Implementation of KLT™ 3 Second Implementation of KLT™ 4 Statement of the Problem 5 Purpose of the Study 7 Significance of the Study 8 Definition of Terms 10 Research Questions 14 Limitations 17 Delimitations 19 Assumptions 20 Organization of the Study 20 Summary 21 Chapter II: Review of the Literature 22 Introduction 22 Formative Assessment 23 Clarifying the Definition of Formative Assessment and Assessment for Learning 23 Formative Assessment Research 25 Challenges Associated with Formative Assessment 26 Formative Assessment Experimental Research 27 viii
The Teacher's Role in Formative Assessment Instruction 29 Formative Assessment, Differentiated Instruction, and Grading 32 The Influence of Summative Assessment Grading Practices 34 Professional Development 36 Seven Factors that Promote Reform 38 Teachers' Perceptions of Reform: The Match Between Teachers' Instructional Theory and Reform Theory, and Teachers' Perceptions That Reform Matches Personal Learning Needs 39 Teacher Participation in CoUegial Learning Communities 42 Principals'Commitment to Reform 45 District-wide Leadership Team's Support of Reform 47 Teacher Contact with Experts Related to the Innovation 48 Dedicated Time for CoUegial Interaction 50 Keeping Learning on Track™ 53 KLT™ Components 53 Elements of KLT™: Developing Teacher Learning and Implementation of Formative Assessment 54 Peer Observation 56 KLT™ and Teacher Learning Communities 57 KLT™ Implementations Studied by ETS 58 Summary 62 Chapter III: Methodology 66 Introduction 66 ix
Selection of Participants 70 Instrumentation 72 Validity and Reliability 78 Ethical Considerations 80 Data Collection 81 Data Analysis 82 Quantitative Analysis 82 Qualitative Analysis 85 Summary 86 Chapter IV: Presentation and Analysis of Data 88 Introduction 88 Data Screening and Processing 89 Answering the Research Questions 90 Research Question One 91 Teachers' Collection of Student-Learning Data 92 Teachers' Analysis of Student-learning Data 94 Teachers'Use of Student-learning Data 95 Research Question Two 97 The Match Between Teacher's Instructional Theory and Reform Theory 99 Teachers' Perception that Reform Matches Personal Learning Needs 100 Teacher Participation in CoUegial Learning Communities .... 100 Principals'Commitment to Reform 101 x
Dedicated Time for Collegial Interaction 103 District-wide Leadership Team's Support of Reform 103 Teacher Contact with Experts Related to the Innovation 104 Correlations 105 Demographic Variables 108 Emergent Themes Related to Factors Which Promote Sustained Reform 108 Theme One: The Influence of Principals' Actions on Teachers' Commitment to Reform 109 Theme Two: The Emergence of Collegiality I l l Theme Three: The Apparent Disconnect Between Central Administration's and Teachers' Long-Term Vision for Formative Assessment Reform 113 Research Question Three 114 Multiple Regression 115 Multiple Regression Analysis 116 Statistic Description of the Model 117 Summary 120 Chapter V: Summary, Discussion, and Conclusions 123 Introduction 123 Summary of the Study 123 Discussion of the Findings 126 Research Question One 127 Research Question Two 131 Descriptive Statistics 132 xi
Correlations 133 Three Emergent Themes 135 Theme One 135 Theme Two 137 Theme Three 138 Research Question Three 140 The Match Between Teachers' Instructional Theory and Reform Theory 140 Dedicated Time for Collegial Interaction 141 Years of Teaching Experience 142 Nonsignificant Factors 142 Implications for Practice 144 Recommendations for Further Study 149 Conclusion 153 References 158 xi i
List of Tables Table 1: Sample of Respondents Compared to Accessible Population and Target Population 71 Table 2: FAPQ Design Details: Teachers' Collection, Analysis, and Use of Student Learning Data 73 Table 3: FAPQ Design Details: The Factors That Promote Reform 75 Table 4: State of Formative Assessment Reform 92 Table 5: Respondents' Reasons for Collecting Student-Learning Data 94 Table 6: The Seven Factors That Promoted Sustained Reform 98 Table 7: Correlations Among Variables 106 Table 8: Multiple Regression Results 116 Table 9: ANOVAF Statistic Description of the Model 117 Table 10: Results of Stepwise Regression 118 xiii
List of Appendices Appendix A: Formative Assessment Practices Questionnaire 168 AppendixB: ConsentForm 174 xiv
1 Chapter I Introduction In 2005, the Bolden Township School District (BTSD) collaborated with Educational Testing Services (ETS) to present a professional development program, called Keeping Learning on Track (KLT™) which introduces the teachers to formative assessment. Central administration of BTSD perceived formative assessment as the foundation for a series of planned initiatives to focus the district on the science of collecting, analyzing, and using student learning data to regulate instruction (R. Bato, personal communication, July 7, 2009). Formative assessment training began in June of 2005 and concluded during the 2007-2008 school year. After the conclusion of KLT™ training, differentiated instruction, a logical outcome of formative assessment (Heritage, 2007), was presented to teachers through a professional development program conducted by an outside consultant. This holistic case study, conducted two and a half years after the conclusion of KLT™ training, employed a mixed methods approach (Johnson & Christensen, 2008) to investigate the extent to which elementary teachers in the BTSD sustained the use of formative assessment practices to regulate instruction. The study served the school district's desire to ascertain whether the reform effort "stuck." The research study also explored the factors that promoted fidelity to formative assessment reform. This feature of the study, although focused on a particular professional development program in a specific school district, has implications for other educational reform efforts. The following research questions guided the study: (1) To what extent have elementary teachers in the Bolden Township School District sustained formative assessment reform?,
2 (2) Which factors have promoted sustained formative assessment reform in the Bolden Township School District?, and (3) Which factors associated with sustaining reform most significantly predicted the state of reform in the Bolden Township School District? Background of the Study Bolden Township is an upper-middle class community in a suburb of Philadelphia, Pennsylvania. The school district serves 5,475 students in five elementary schools, one middle school, and one high school. The BTSD is considered a high achieving district (Lyon, Cleland, & Gannon, 2008). In the late 1980s, all district schools were recognized at the state or national level as Schools of Excellence. Standardized test scores are consistently well above national and state norms, and approximately 90% of students continue their education after high school (Lyon et al., 2008). In 2004, the BTSD had been managing various reform initiatives. In order to continue to improve student achievement, school district professionals sought to refocus and streamline professional development programs. Simultaneously, ETS was developing a professional development program (KLT™) to help teachers implement formative assessment (Lyon et al., 2008). In 2004, the Math Science Partnership of Greater Philadelphia (MSPGP) and ETS forged an alliance to offer a third party implementation of KLT™. ETS agreed to provide the KLT™ program and assistance with facilitation and content free of charge to the BTSD. In return, the district permitted an ETS researcher (C. Lyon) to examine the implementation process in order to improve KLT™ content.
3 First Implementation ofKLT™ The KLT™ professional development program was first implemented during the 2005-2006 school year. The goals for the KLT™ program were to transmit the theory of formative assessment and to reform teachers' instructional practices to reflect formative assessment theory (Lyon et al., 2008). The first implementation of formative assessment was deemed unsuccessful because it did not positively impact teachers' practice associated with formative assessment (Lyon et al., 2008). Reform was characterized as having low buy-in from various levels of the organization. Lyon et al. (2008) reported two missteps with the first implementation of KLT™. First, formative assessment training sessions competed with various teacher in-service workshops; this diffused the impact of KLT™. Second, contrary to KLT™ theory, curriculum coordinators formed and led professional learning communities (PLCs). Lyon et al. concluded that three factors appeared to have had a deleterious impact on PLC meetings: (a) time for teachers to meet and focus on formative assessment was not provided or supported by the district; (b) facilitators (curriculum coordinators) lacked sufficient materials, expertise, and support to lead the meetings; and (c) administrative leadership or support of the PLC meetings was inadequate. Lyon et al. (2008) highlighted the critical need for teacher learning communities (TLCs) and focused collaboration when they concluded "there may not have been sufficient structure or activities to guide the types of conversations that are necessary to deepen teachers' understanding and enhance the implementation of assessment for learning" (p. 5).
4 Second Implementation of KLT™ Based on an evaluation of the first KLT™ implementation, BTSD officials decided to attempt a second implementation of KLT™ (Lyon et al., 2008). The second implementation included two major changes. First, PLCs were changed to TLCs to emphasize teacher facilitation of the KLT™ professional development program. Teachers led training sessions instead of curriculum coordinators. Second, district administrators dedicated meeting times and eliminated conflicting agendas in order to focus primarily on formative assessment. To build school district capacity for change, a core group of 35 teacher leaders were trained as facilitators in the KLT™ program. District wide implementation began in August 2006. The second implementation of KLT™ was far more successful than the first (Lyon et al., 2008). Initial reports indicated that teachers learned and began to adopt new teaching strategies. However, research was lacking on how often and how well teachers used the new strategies they learned, leading Lyon et al. (2008) to recommend that "future research should focus on those issues of frequency and quality, as well as on examination of the impact on student achievement" (p. 75). Advocates for formative assessment assert that it leads to improved student performance (Black & Wiliam, 1998b; Stiggins, 2002). However, neither ETS researchers nor the BTSD officials conducted a follow-up study to examine KLT's™ long-term impact on formative assessment reform in the BTSD. As a result two questions lingered. First, to what extent did teachers sustain the use of formative assessment to regulate instruction? Second, which factors promoted fidelity to formative assessment reform?
5 Statement of the Problem People greatly underestimate what change is and what factors and processes generate it (Fullan, 2007). How does one define success or failure in the field of education? When failure occurs, how do stakeholders generate improvements? Tyack and Cuban (1995) view reform as "planned efforts to change schools in order to correct perceived social and educational problems" (p. 4). Lyndon B. Johnson sought to build the "Great Society" by declaring war on poverty in the 1960s and asserting that education was the answer to our national problems (Tyack & Cuban, 1995). The history of American education is littered with failed reform efforts (Fullan, 2007). Billions of dollars and countless human-hours have failed to produce substantial changes in the way American schools operate. According to Fullan, schools have floundered in their attempt to change curriculum and instruction and to raise student achievement. In Fullan's view, this is less a criticism of teachers, and more a problem with the methods used to encourage change. Fullan (2007) set a somber tone for reform when he stated that "change will always fail until we find some way of developing infrastructures and processes that engage teachers in developing new knowledge, skills, and understandings" (p. 29). Research has asserted the positive impact of formative assessment on student achievement (Black & Wiliam, 1998b; Bloom, 1984; Hattie & Timperley, 2007; Stiggins, 2002). Furthermore, when professional development incorporates the use of professional learning communities (including teacher learning communities), it can deepen the reform's effect (Ball & Cohen, 1999; Black & Wiliam, 2009; Cobb, McClain, Lamberg, & Dean, 2003; Cohen & Hill, 1998; Dekker & Feijs, 2005; Desimone, Porter, Garet,
6 Birman, &Yoon, 2002; Gibboney, 1994; Reeves, McCall, & MacGilchrist, 2001; Spillane, 2005; Stiggins, 2002; Wiliam, 2006; Wiliam & Thompson, 2007; Wilson & Berne, 1999). At the same time, Bennett (2009) warned that formative assessment is not a magic bullet; it does not automatically raise student achievement. Attempts to implement formative assessment have proven difficult (Wiliam, 2006). Professional development programs, according to Wiliam, are the key to formative assessment's success. However, research examining the long-term influence and sustainability of formative assessment on teachers' instruction practice is lacking (C. Lyon, personal communication, October 3, 2009). ETS researchers (Ellsworth, Martinez, & Lyon, 2007; Goe & Mardy, 2007; Lyon et al., 2008; Maher & Wiliam, 2007; Tocci & Taylor, 2007; Wylie, Lyon, Ellsworth, & Martinez, 2007; Wylie, Lyon, & Goe, 2006) investigated KLT™ implementations through site-based studies. The researchers examined the professional development program, KLT™, at several school districts. Their focus was three-fold: (a) to probe the delivery of formative assessment for lessons to improve future implementations, (b) to examine the impact of KLT™ on teachers' implementation of formative assessment teaching techniques and instructional practices, and (c) to assess the impact of TLCs on collegiality. However, the researchers did not measure the degree or breadth of teachers' long-term implementation of formative assessment reform or the factors promoting or inhibiting sustained change. Furthermore, a thorough search of ProQuest Dissertations and Theses in June of 2009 surfaced no research identical to the topic of this study, using operational terms related to formative assessment and the factors that promote this reform.
7 The goal of this study was to fill two research gaps; first, to understand the long- term impact of a professional development effort on teachers' instructional practices, and second, to examine the factors that promote or inhibit sustained educational reform. The BTSD implemented formative assessment reform without designing an assessment to determine if and why teachers were maintaining fidelity to reform. To determine if teachers sustained formative assessment reform this study examined teachers' collection, analysis, and use of student-learning data. Fullan's (2007) three indicators of instructional reform were used to investigate teachers' instructional practices. To understand why teachers sustained reform this study examined the factors that promoted or inhibited reform. Fullan (2007) called for ways to develop processes and systems to engage teachers in dynamic professional development, encouraging reform. The factors can be applied to the BTSD, but also to any organization attempting to sustain reform. Purpose of the Study The purpose of this holistic case study with a mixed methods approach was to investigate teachers' sustained use of formative assessment and explore the influence of the factors that promote sustained reform. This study examined one school district's implementation of a specific professional development program to facilitate teacher understanding and practice of formative assessment. However, this study is not an investigation of the value of formative assessment alone, nor is it an assessment of the KLT™ program.
8 The review of literature informed the examination of reform associated with teachers' instructional practices. Research (Black & Wiliam, 2009; Buhle & Blachowicz, 2009; Fullan, 2007; Heritage, 2007; Heritage, Kim, Vendlinski, & Herman, 2009; Leusner, Ellsworth, & Goe, 2008; Popham, 2009; Supovitz & Klein, 2003) indicated that teachers' ability to adapt instruction relies on the proper collection, analysis, and use of student-learning data. Fullan's (2007) three indicators of reform were used to determine whether teachers sustained reform. Research (Ball & Cohen, 1999; Coburn, 2004; Desimone et al., 2002; Drago- Seversen, 2007; Fennema, Carpenter, Franke, Levi, Jacobs, & Empson, 1996; Fullan, 2002; Little et al, 2003; Paoletti, 2009; Smith & Gorard, 2005; Smith, Wilson, & Corbett, 2009; Supovitz & Klein, 2003; Yin et al., 2008) revealed the factors that promote change. This study examined the influence of the factors on the BTSD's reform of teachers' instructional practices. Significance of the Study Changing teachers' instructional practices is challenging (Wylie et al., 2007). Professional development support is especially helpful when teachers are engaged in changing established, perhaps unconscious, instructional practices (Wylie et al.). Further study is necessary to clarify how or if professional development is changing the pedagogy associated with formative assessment (Wiliam, 2006). Heritage and Popham (2008) asserted that researchers should conduct follow-up studies of KLT™ implementations to further investigate AfL reform. They stated: We hope that in the future we will learn about the sustainability of the participating
9 teachers' AfL practices after the initial 2-year implementation; the extent to which AfL becomes embedded in the daily practices of the schools, including the classrooms where teachers did not volunteer for the program; and the degree to which system changes needed to support the TLCs are instantiated, (p. 133) With the emphasis the culture places on monitoring student achievement through summative, high stakes testing, teachers find it difficult to sustain the instructional practices that promote learning, as well as monitor it. (Stiggins, 2002). Formative assessment can link teachers to student-centered instruction. However, informed decision-makers should craft professional development to provide teachers with the tools necessary to perform the job, according to Stiggins. In the era of high stakes testing, schools often focus on the use of summative assessments to monitor student achievement. A greater balance between measuring and increasing achievement is necessary, according to Stiggins (2002), who claims that "we must make a much stronger investment in assessment for learning" (p. 761). Professional development programs that train teachers in the delivery of formative assessment practices should play a larger role in American education. According to Black and Wiliam (1998a), in order to improve learning reformers must directly influence classroom instruction. According to Guskey (2002) the most overlooked aspect of professional development is sustaining change. Guskey asserted that reform should be responsive to the complex nature of education. Consequently, merging reflection and practice will inform future professional development designed to sustain change. This research can increase the knowledge related to the critical issues associated with fidelity to reform.
10 The findings bring into focus the following: (a) supports encouraging sustained change, (b) barriers inhibiting sustained change, and (c) teachers' long-term commitment to formative assessment theory and practice, On a micro-level, this research extends the understanding of professional development to sustain reform and its impact on the long-term implementation of change at the BTSD. On a macro-level, this research extends the understanding of leaderships' management of sustained reform, fundamental to an organization's success (Guskey, 2002). Knowledge of the factors promoting or impeding an organization's commitment to sustained change may help leaders to engage individuals effectively and ignite the organization's capacity to increase productivity. Definition of Terms To clarify key terms in this study, definitions and delineations related to formative assessment, professional development, and KLT™ are provided. Professional Learning Communities In this study professional learning communities (PLCs) are cohorts of teachers engaged in sustained experimentation, inquiry, and reflection upon reform. Collegial engagement, reflection, and dialogue promote accountability. PLCs develop teachers' individual and collective capacity to sustain change (Leusner, Ellsworth, & Goe, 2008).
11 Teacher Learning Communities In this study teacher learning communities (TLCs) are cohorts of teachers, led by teacher leaders, engaged in sustained experimentation, inquiry, and reflection upon reform. KLT™ recommended the use of TLCs to support teacher efforts to sustain formative assessment reform. Collegial Learning Communities Smith, Wilson, and Corbett (2009) emphasize the collaborative qualities of PLCs by referring to them as collegial learning communities (CLCs). In this study the terms PLCs and CLCs are used interchangeably. TLC is used in the context of KLT™. District-Wide Leadership Team District-wide leadership team refers to the superintendent, assistant superintendents, and curriculum director of the BTSD. Throughout this study the terms District-Wide Leadership Team and central administration are used interchangeably. Dedicated Time for Collegial Interaction The time intentionally set aside for teachers to work together to improve instruction. Teacher Contact with Experts Related to the Innovation In this study an expert is a consultant, a colleague/supervisor, a professor, or anyone a respondent deems more knowledgeable than him/herself. Contact with experts can be made via email, observation, coaching, or mentoring. Such contact is intended to improve teachers' implementation of reform.
12 Instruction Instruction refers to any activity that is intended to create learning (Black & Wiliam, 2009). Formative Assessment "Practice in a classroom is formative to the extent that evidence about student achievement is elicited, interpreted, and used by teachers, learners, or their peers to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions they would have taken in the absence of evidence that was elicited" (Black & Wiliam, 2009, p. 9). Assessment for Learning "Synonymous with formative assessment, assessment for learning is any assessment for which the first priority in design and practice is to serve the purpose of promoting pupils' learning" (Black, Harrison, Lee, Marshall, & Wiliam, 2004, p. 8). Assessment for learning can provide information to use formatively (Wiliam, Lee, Harrison, & Black, 2004). Moment of Contingency The moment of contingency refers to the time or moment teachers determine the next instructional step based on the learning data collected through formative assessment teaching techniques. How teachers, learners, and their peers create and capitalize on these moments of contingency entails considerations of instructional design, curriculum, pedagogy, psychology, and epistemology (Black & Wiliam, 2009).
13 Synchronous Moment of Contingency Teachers' "real-time" or spontaneous regulation of instruction during one-on-one, small group, or whole class teaching (Black & Wiliam, 2009). Asynchronous Moments of Contingency Teachers' planned or intentional regulation of instruction based on evidence of learning derived from teaching techniques (Black & Wiliam, 2009). Regulation of Instruction Evidence of student learning is collected through formative assessment teaching techniques. The teacher analyzes the data to determine how to craft lessons to accommodate students' learning needs (Black & Wiliam, 2009; Wiliam, 2004). Formative Assessment Teaching Technique Any activity a teacher uses to generate student-learning data in order to regulate instruction (Thompson & Wiliam, 2007). Differentiated Instruction Differentiated instruction is a teaching philosophy which asserts that students learn best when teachers effectively address differences in student interests, readiness levels, and learning profiles (Tomlinson, 2005). The key goal of differentiated instruction is maximizing the learning potential of each student (Tomlinson, 2001, 2003). Keeping Learning on Track™ Keeping Learning on Track™ is a professional development program created by Educational Testing Services (Maher & Wiliam, 2007). KLT™ is one delivery method by which classroom teachers acquire formative assessment techniques.
14 Research Questions The BTSD granted the researcher permission to investigate teachers' continued use of formative assessment and the factors that promoted reform. Three research questions guided the study: 1. To what extent have elementary teachers in the Bolden Township School District sustained formative assessment reform? 2. Which factors have promoted sustained formative assessment reform in the Bolden Township School District? 3. Which factors associated with sustaining reform most significantly predicted sustained reform in the Bolden Township School District? The first research question focused on a specific reform effort of one school district. Questions 2 and 3 examined what promoted this specific reform. These questions also provided insight into the process of educational reform and have implications for organizations beyond the district studied. In order to determine the significant predictors of sustained formative assessment reform in the BTSD, the researcher developed subquestions to accept or reject the null hypothesis. Subquestion 1 Will the match between teachers' instructional theory and reform theory be a significant predictor of sustained formative assessment reform? Ho: The match between teachers' instructional theory and reform theory is not a significant predictor of sustained formative assessment reform. Hi: The match between teachers' instructional theory and reform theory is a significant predictor of sustained formative assessment reform.
15 Subquestion 2 Will teachers' perception that reform matches personal learning needs be a significant predictor of sustained formative assessment reform? Ho: Teachers' perception that reform matches personal learning needs is not a significant predictor of sustained formative assessment reform. Hi: Teachers' perception that reform matches personal learning needs is a significant predictor of sustained formative assessment reform. Subquestion 3 Will teacher participation in collegial learning communities be a significant predictor of sustained formative assessment reform? Ho: Teacher participation in collegial learning communities is not a significant predictor of sustained formative assessment reform. Hi: Teacher participation in collegial learning communities is a significant predictor of sustained formative assessment reform. Subquestion 4 Will principal commitment to reform be a significant predictor of sustained formative assessment reform? Ho: Principals' commitment to reform is not a significant predictor of sustained formative assessment reform. Hi: Principals' commitment to reform is a significant predictor of sustained formative assessment reform.
16 Subquestion 5 Will district-wide leadership team's support of reform be a significant predictor of sustained formative assessment reform? Ho: District-wide leadership team's support of reform is not a significant predictor of sustained formative assessment reform. Hi*. District-wide leadership team's support of reform is a significant predictor of sustained formative assessment reform. Subquestion 6 Will teacher contact with experts related to the innovation be a significant predictor of sustained formative assessment reform? Ho: Teacher contact with experts related to the innovation is not a significant predictor of sustained formative assessment reform. Hi: Teacher contact with experts related to the innovation is a significant predictor of sustained formative assessment reform. Subquestion 7 Will dedicated time for collegial interaction be a significant predictor of sustained formative assessment reform? Ho". Time for collegial interaction is not a significant predictor of sustained formative assessment reform. Hi: Time for collegial interaction is a significant predictor of sustained formative assessment reform.