English language proficiency in mathematics on large-scale assessments: A study of eighth graders
Table of Contents
Acknowledgments iv List of Tables vii List of Figures viii CHAPTER 1. INTRODUCTION 1 Introduction to the Problem 1 Background of the Study 3 Statement of the Problem 7 Purpose of the Study 8 Research Questions 9 Research Hypotheses 9 Nature of the Study 9 Significance of the Study 10 Definition of Terms 11 Assumptions and Limitations 13 Organization of the Remainder of the Study 13 CHAPTER 2. LITERATURE REVIEW 14 Introduction 14 Demographic Information 14 Defining English Language Learners 15 Theoretical Framework 18
Language Proficiency and Academic Achievement 22 Current Practices in Testing English Language Learners 29 Review of Related Research 30 Summary 36 CHAPTER 3. METHODOLOGY 38 Research Design Strategy 38 Instruments 40 Data Collection Procedures 52 CHAPTER 4. RESULTS 55 Introduction 55 Demographic Characteristics 55 Findings and Results 56 Summary 63 CHAPTER 5. DISCUSSION 65 Introduction 65 Summary of the Study 65 Summary of Findings and Conclusions 66 Recommendations 69 Implications 73 REFERENCES 75
List of Tables
Table 1: Comparison between Language Proficiency Standards and Academic 23 Content Standards as Outlined in the No Child Left Behind Act
Table 2: Comparison of Characteristics of Standardized English Language 24 Proficiency Tests Pre/Post the No Child Left Behind Act of 2001
Table 3: School District in Georgia – Demographic Information, 2005-2006 39
Table 4: Standard: English Language Learners Communicate Information, Ideas, 44 Concepts necessary for Academic Success in the Area of Mathematics
Table 5: Speaking Tests Scoring Scale 46
Table 6: CRCT – System of Scale Scores 51
Table 7: Frequency Table by Ethnicity 56
Table 8: Frequency Table for Gender 56
Table 9: Correlations for 2005 CRCT and Access 59
Table 10: ACCESS Composite Scale Score and CRCT Performance Levels 61
Table 11: ANOVA Results for CRCT Mathematics Performance Levels 62
Table 12: Bonferroni Test Results for CRCT Mathematics Performance Levels 63
List of Figures
Figure 1: Communicative Language Proficiency Conceptualized along two 19 Continuums
Figure 2: Scatter Plot Depicts Relationship between the Mathematics (CRCT) and 59 English Language Proficiency Scores (ACCESS)
Figure 3: Error Bar Chart shows the Average English (ACCESS) Score, Separately 60 for each Performance Level (CRCT)
CHAPTER 1. INTRODUCTION
Introduction to the Problem
Across the United States there are an increasing number of students that do not speak the English language. This group of students is the fastest growing school-age population in the United States (Abedi, 2004). According to a recent survey of the English language learner (ELL) population, over four million were enrolled in public schools during the 2000-2001 school year (Kindler, 2002). However, American schools have progressed very little in meeting the academic needs of ELLs (United States General Accounting Office, as cited in Nickerson, 2004). The dropout rate for this group of students is increasing at a rapid rate; they also have a higher percentage of students being retained, and lower standardized tests scores than their English-speaking peers (Moss & Puma, 1995). In addition, ELLs do not fare as well on large-scale assessments in content area subjects across grade levels (Escamillo, Mahon, Riley-Bernard, & Rutledge, 2003). As the population of ELLs increases, state and local school districts face the challenge of having these students meet rigorous academic standards and of assessing their academic performance. Legislation such as the Improving America’s School Act of 2001 and the No Child Left Behind (NCLB) Act of 2001, have established that all students be included in state and national assessments (Abedi, 2004). For the English language learners, this means participating in state and national tests before they are proficient in the English language. Research on language acquisition indicates that it may take 5 to 7 years to acquire academic language; in some instances it may take up to 10 years (Cummins, 1984b; Thomas & Collier, 2002). Educators are faced with the task of trying to effectively assess the ELLs so that
their knowledge of the content is being assessed and not their language proficiency. This presents a problem for ELLs because every assessment may be an assessment of language (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999). There are many decisions made based on large-scale assessments: promotion/retention, high school graduation, and ESOL Programs (Duran, 1983; Gottlieb, 2006). In addition, many ELLs have not benefited from the educational reforms that have followed as a result of not being included in large-scale assessments, and they have often been overlooked when it came time for educational reform to help increase student achievement (August & Hakuta, 1997). However, test scores may be misinterpreted because the students are not proficient in the English language, and therefore the test does not accurately reflect what the student knows and the reliability and validity of the test becomes questionable. In summary, states and local school districts need more accurate data on measuring the levels of language proficiency of ELLs in order to meet their academic needs and prepare them so that they are able to effectively participate in large-scale assessments (Nickerson, 2004). Having all ELLs administered the mathematics sections of the Georgia Criterion-Referenced Competency Test (CRCT) after being in this country for only 1 month does little to obtain an accurate picture of the progress they are making in learning the English language. As a result of the increase in the population of ELLs, states are challenged to find ways to assess them in order to obtain accurate and meaningful data in all content areas.
Background of the Study Evaluating students through the use of standardized tests has been an important part of American education since the turn of the 20th century (NASBE, 1997). Educational policymakers, practitioners, and the general public have been concerned about the quality of education and have tried to evaluate how schools are doing with the use of standardized assessments (NASBE). Federal legislation, such as the Elementary and Secondary Education Act (ESEA) of 1965 made states responsible for identifying below-average students who may qualify for special education services, but there were no specific criteria for ELLs. In the 1970s the purpose of education took a turn and local, state, and federal policymakers saw that even though they were comparing students, school districts, and states to national averages, student achievement was not improving (Porter, 2000). Historically, ELL students have not been included in standardized assessments. As a result, there is a lack of accountability for the academic progress of these students because they are not being held to the same academic standards as their peers (August & Hakuta, 1997). Most states excluded students who had been in the United States or in some type of ESL/bilingual program for less than 3 years or who had not attained a certain level of English proficiency (Holmes, Hedlund, & Nickerson, 2000). Because they had not been included in standardized tests, there has been no way of tracking their progress, and they have not been able to take advantage of educational reforms designed to increase academic standards (Coltrane, 2002). This new student majority in public schools has posed a challenging issue that educators and policymakers must now have to face in the United States (Reyes & Rorrer, 2001).
In the 1980s, education reform came to be motivated by one prevailing concept--the setting of academic standards for what students should learn and using these standards as a guide to all other components of education reform (Reyes, & Rorrer, 2001). The much-talked-about 1983 report, A Nation at Risk: The Imperative for Educational Reform, provided the energy that set the stage to reform public education in the United States and prompted states to reevaluate their responsibility to ensure a quality education for all students (Porter, 2000). This led to a series of reports that emphasized accountability, held education to a higher standard, and began the standards reform movement, which continued through the 1990s. In 1991, President Bush announced America 2000: An Education Strategy. The long-term strategy included improving current schools and making them accountable for the achievement of all students (Reyes & Rorrer, 2001). In 1994, President Clinton adopted the Goals 2000: Educate America Act, which again placed emphasis on academic standards and assessments. All of these reforms recommended access for all children to higher standards (O’Day & Smith, as cited in Reyes & Rorrer). In addition to the authorization of Goals 2000, the Elementary and Secondary Education Act (ESEA) was reauthorized under the name Improving America’s School Act (IASA) (Holmes & Duron, 2000). Both acts acknowledged for the first time that ELLs can and must meet higher standards (Baca & Cervantes, 1998). There are two separate titles contained in the IASA that directly affect the assessment of ELLs: Title I and Title VII. Title I The purpose of Title I was to establish the development of high standards for all students, particularly ELLs and economically disadvantaged students. Each state education association
must develop measurable objectives to ensure that ELLs make “adequate yearly progress” in their development of English language proficiency, while meeting the state academic standards in content areas implemented for native English speakers (NCLB, 2001). Title I funds are provided so those students may receive supplementary resources to help reach high standards (Holmes & Duron, 2000). With high standards comes high-stakes assessments, and it was reasoned that these assessments would help the students by improving instruction, program evaluation, high school graduation, and promotion. Title VII The purpose of Title VII is to officially recognize ELLs as a significant population and provide funding for bilingual education reform to be a priority in education. Title VII states that these students must be provided with an equal opportunity to achieve the same high standards that all students receive (Holmes & Duron, 2000). Federal legislation in the last 10 years, including Goals 2000 and the Improving America’s Schools Act of 1994, have tried to improve instruction and assessment by mandating inclusion of all students in large-scale assessments (Abedi, Coutney, & Leon, 2003). Now, with the reauthorization of the Elementary and Secondary Education Act, the No Child Left Behind (NCLB) Act of 2001 will require all students to test yearly in mathematics and reading in Grades 3 through 9 (ESEA, 2001). One part of the NCLB Act is Title III, which discusses English language acquisition and academic achievement. The purpose of Title III is to assist in helping children who are identified as ELLs attain proficiency in English, develop high levels of academic achievement in English, and master the same rigorous state standards that all children are expected to attain (NCLB, 2001).
With these new standards and accountability system in place, the next question is as follows: How does the accountability system impact the ELL (Reyes & Rorrer, 2001)? We can all agree that our nation wants the best education for all our students, but, when it comes to ELLs, there are conflicting views on how to educate them and how to make them accountable to the same standards as the rest of the population when they are not proficient in the English language. Today, many policymakers advocate the same standards for all children, but others suggest that alternative standards should exist for LEP students. Why has the assessment of ELLs been so difficult (Reyes & Rorrer)? Many of the students vary in the length of time they have been in the United States and the amount of education that they have received in their countries (Reyes & Rorrer). It is also difficult to assess what these students know because many of the assessments are assessments of language (American Educational Research Association, American Psychological Association, National Council on Measurement in Education, 1999). As states develop more rigorous standards such as the No Child Left Behind (2001) Act, ELL students are faced with a test that includes ideas, words, and expressions that may be confusing to them because they did not grow up in this country. Large-scale assessments most states currently use rely heavily on language proficiency skills and are developed for the assessment of native English speakers rather than of ELLs (Garcia & Menken, as cited in Menken, 2000). As evolving national, state, and local standards continue to list the best ways to assess ELLs, there are other factors that need to be considered as a result of large-scale assessments, such as program placement and graduation requirements that are attached to test scores (Reyes & Rorrer).
Statement of the Problem Previously, in order to succeed in school, ELLs were expected to learn English before being required to learn the academic content (Albus, Klein, Liu, & Thurlow, 2004). This is no longer the case, because recent legislation such as the NCLB Act of 2001 requires ELLs to be held to the same academic standards as other students (Albus, et al). Large-scale assessments are being implemented to ensure that the necessary programs and resources are in place to enable students to meet challenging standards and to give them an opportunity to learn (Holmes & Duron, 2000). However, the assessment of ELLs is a complex problem because many have not acquired the English language to be able to participate in large-scale assessments. In Georgia, ELLs who have been in this country less than 1 year are exempt the first academic year from language arts, reading, science, and social studies, but all students, regardless of how long they have been in this country are administered the mathematics section of the CRCT. In order to meet the needs of all students, attention must be given to those students who do not speak English as their first language. Many times the results of the data are used to measure the progress of the individual schools and districts, but the results may be misleading because the students cannot demonstrate what they have learned in the core subject areas if they cannot read or write well in English (Steven, Butler, Castell-Wellington, 2000). In addition, the scores are norm-referenced for students that speak English, not for English language learners, and this presents an issue of validity. However, to exclude ELLs can be just as misleading because the individual school and district scores would not accurately reflect the population of students. Another problem with excluding ELLs is there is no way of tracking their progress and they are not able to take advantage of educational reforms designed to increase academic standards
(Coltrane, 2002). With the inclusion of ELLs in large scale assessments, language proficiency plays a paramount role that has not been effectively addressed with ELLs (Mahon, 2004). Not much is known about how English language proficiency impacts English achievement tests or how to interpret the scores of ELLs who have been tested in English (Mahon, 2004). In addition, there have been a limited number of studies that have examined the relationship between English language proficiency and mathematics performance on large-scale assessments (Abedi, Lord, & Plummer, 1997). One of the main issues when it comes to assessing ELLs is the language. There is not a large gap in test scores between ELLs and English-speaking peers when it comes to math computation, but with more language-based problems, the gap widens. This is mainly due to the fact that those ELLs who are not as proficient in the English language may read at a lower level and do not understand the mathematical concepts (Madden, Slavin, & Simons, 1995). Purpose of the Study The purpose of this study is to examine the relationship between English language proficiency and performance in mathematics on a criterion-referenced competency test with a group of eighth-grade English language learners from a school district in Georgia. The main objective is to analyze the extent to which English language proficiency predicts academic achievement as measured by the CRCT mathematics test. Cummin’s theory of second-language acquisition is the basis for this study. Cummins (1984b) discussed how a student may have acquired basic conversational skills within 2 to 3 years, but many ELLs have not acquired the academic language skills necessary to perform up to the level of their English-speaking peers, which may take 5 to 7 years.
This study will use achievement scores from the 2006 state-mandated test in Georgia called the Criterion-Referenced Competency Test (CRCT) and 2006 scores from Assessing Comprehension and Communication in English State-to-State (ACCESS) test. Research Questions 1. To what extent does English proficiency, as measured by the ACCESS test, predict English academic achievement, as measured by the CRCT mathematics test for eighth-grade mathematics? 2. Is there a difference in English proficiency levels, as measured by the ACCESS, among three achievement levels in eighth-grade mathematics (Level 1 – Does not meet state expectations, Level 2 – Meets state expectations, Level 3 – Exceeds state expectations), as measured by the CRCT? Research Hypotheses 1.1 H 0 : There is no correlation between the ACCESS scores and the CRCT scores. H a : There is a correlation between the ACCESS scores and the CRCT scores. 2.1 H 0 : The average ACCESS score is the same for all three CRCT achievement levels. H a : The average ACCESS score is not the same for all three CRCT achievement levels. Nature of the Study A quantitative study will be performed to investigate the relationship between English language and performance on large-scale assessments. A sample of eighth-grade English language learners from a school district in Georgia will be used in this study. The purpose of this study is to discover relationships between two variables, so the Pearson Product Moment
Correlation will be used to analyze this relationship. In addition, Analysis of Variance will be performed to determine whether the differences between the mean scores are statistically significant. Achievement scores from the Georgia CRCT and English language proficiency scores from the ACCESS test will be used to conduct the study Significance of the Study The ELL population is increasing at a rapid rate every year, and careful attention should be given to how they should be assessed. ELLs play a significant role when it comes to standards, testing, and accountability. Schools and students are having penalties imposed because they may not make a certain score on the state test. At the school level, schools that do not make adequate yearly progress (AYP) in the same subject for 2 or more consecutive years are placed in needs-improvement status, with increasing consequences for each consecutive school year (GADOE, 2005). If a school does not make AYP the first year there are no consequences. If they do not make AYP the second year they are placed on the needs-improvement list and parents may choose to send their children to a school within the district that is not on the needs-improvement list. An additional consequence may be that schools have to develop a school improvement plan. At the district level, there are no consequences for not making AYP the first year. If a school district does not make AYP the second year, then each school develops an educational action plan. There have been numerous studies written about assessing English language learners and the validity of accommodations for ELLs (Abedi, 2004, Bailey & Butler, 2003; Bailey, Butler, & Abedi, 2000/2005), but there has been little written on English language proficiency and academic achievement in mathematics among ELLs. It becomes difficult to interpret test scores
if the students’ English is limited. With all the recent legislation on standards and accountability, research on assessment of English language learners can be beneficial for making decisions regarding how best to evaluate ELLs. This study will also add to the body of literature on the relationship between language proficiency and performance in mathematics on large-scale assessment of ELLs and the difficulty of participating in these tests when they are not proficient in the English language. In addition, it will compare proficiency measures on a new language proficiency assessment that is more closely aligned with state standards. Finally, it will add to the field of research on language proficiency assessments and offer insight regarding how English language learners progress in learning English relates to their progress in learning the academic language. Definition of Terms The following terms will be used throughout this study to describe the assessment of English language learners (GADOE, 2005; Gottlieb, 2006; WIDA, 2004). Academic Language Proficiency – The language patterns required in understanding and processing academic content. Adequate Yearly Progress (AYP) – One aspect of the federal No Child Left Behind Act of 2001 (NCLB). It is an annual measure of student participation and achievement of statewide assessments and academic indicators. Assessing Comprehension and Communication in English State-to-State (ACCESS) – The language proficiency test adopted by Georgia to meet the requirements of NCLB. It is designed to measure the progress of English language proficiency with ELLs.
Basic Interpersonal Communication Skills (BICS) – Language for social purposes. It may take 2 to 3 years for ELLs to develop BICS in English. Cognitive Academic Language Proficiency (CALP) – (Cummins, 1984b). The language required for academic achievement in a context-reduced environment. Examples include class lectures and reading assignments. Criterion-Referenced Test – Type of measure that is based on established criteria, such as the Georgia Performance Standards (GPS), rather than ranking student performance. A student’s performance is compared to clearly identify academic tasks or skill levels. English language learners (ELL) – Culturally and linguistically diverse students whose English language is limited and prevents them from learning academic content at their particular grade level. English as a Second Language (ESOL) – The state-funded instructional program that assists students in learning English. Large-Scale Assessment – Usually applied in k-12 settings to standardized testing programs designed to be part of evaluating the effectiveness of schools or districts. Levels of Language Proficiency – A defined stage in the development of the English language acquisition process from entering, beginning, developing, expanding, and bridging, in the four domains of listening, speaking, reading, and writing. Georgia has four levels of language proficiency: newcomer, beginner, intermediate, and advanced. Reliability - The extent to which an experiment, pilot study, or field test produces the same results after repeated trials. Validity – The extent to which the test measures what it is supposed to measure.
Assumptions and Limitations This study will confine itself to analyzing data from the language proficiency tests and the Georgia Criterion-Referenced Competency Test (CRCT) of a group of eighth-grader ELLs in a school district located in Georgia. These findings represent ELLs who have participated in both the language proficiency test and the mathematics portion of CRCT, because all students, regardless of level of language proficiency, participate in the mathematics portion of the CRCT. A limitation of this study is only scores from eighth-grade English language learners in one school district will be used, and students were not chosen randomly, so the scores may not be generalized across the United States. In addition, the school district was limited in the data it could provide, so there may be other variables, such as language background, length of time in the United States, and accommodations provided that may affect performance. This is the first year that the ACCESS test is being implemented in this particular school district, so students and teachers were not familiar with the test. Organization of the Remainder of the Study Chapter 2 will review the literature on the demographic information of the ELL population, defining the English language learner, and assessment of the ELL. There will also be research related to similar studies that have addressed this topic and what results they have concluded. Chapter 3 will discuss the methodology of this study and chapter 4 will discuss the results of data and answer each research question. Chapter 5 will summarize the results and give recommendations for educators and policymakers.
CHAPTER 2. LITERATURE REVIEW Introduction This chapter provides an analysis of the literature closely related to the current study. The research investigates the following areas as they relate to the assessment of English language learners: (a) demographic information, (b) defining the English language learner, (c) theoretical framework, (d) English language proficiency and achievement, (e) review of related research, and (f) current practices in testing English language learners. Demographic Information The ELL population is increasing at a rapid rate, and it is important to examine this data at the national and state levels because these students are becoming part of classrooms. The ELL population is increasing faster than the general student population. This culturally and linguistically diverse population of students in the United States has increased over 100% from school year 1989-1990 to school year 1999-2000, whereas the general education population increased only by a little over 20% during the same time frame (The National Clearinghouse for English Language Acquisition, 2002c). During the 2002-2003 school year, over 5 million ELLs attending public schools accounted for approximately 10% of the total enrollment from pre-k to Grade 12 (Gottlieb, 2006). The greatest numbers of ELLs live in California, with Arizona, Florida, Illinois, New York, and Texas following behind with more than 120,000 students identified as English language learners (Gottlieb). In Georgia, nearly 1 out of every 10 residents 5 years of age and older speaks a language other than English (Barker & Webbert, 2004). Between 1995 and 2004, Georgia has seen a 349% increase in its Hispanic K-12 population (Barker & Webbert). Between July 2000 and July 2002,
there was a 17% growth in the Hispanic population in Georgia compared to the national average of 9% (Barker & Webbert). The school district where this study is taking place is one of the largest school districts in Georgia, with a projected enrollment of 86,000 students. Students receiving ESOL services make up 6% of the student population. This is important to consider when analyzing student performance because these students often require additional instruction and resources to help them be successful in school. In addition, with this increase, there becomes a need to implement ways to effectively assess this increasing population of culturally and linguistically diverse students. However, before one examines how to identify their level of language proficiency and what is the most appropriate way to assess them, educators need to identify who these students are and how to define this population of students (Gottlieb, 2006). Defining English Language Learners A main concern that presents a challenge for educators in assessing ELLs is defining the population to be assessed. There are many terms used to identify those students whose first language is not English: limited English proficient (LEP), English language learner (ELL), Cultural and linguistically diverse (CLD), and bilingual (Rhoades, Orchoa, & Ortiz, 2005). For the purpose of this study, the term English language learner will be used. There is a lack of consistency in defining this population of students, and because of this variation of terms across states, it difficult to identify these students (Bailey et al., 2005). A student may be classified as limited English proficient in one state, but not another and it becomes difficult when trying to interpret test data across the United States. Some educators and policymakers feel that there needs to be a nationwide standard for defining ELLs. In addition, states would benefit from a uniform language proficiency test that
would clearly define the boundaries for ranges of language proficiency and make it consistent across the United States (Bailey et al., 2005). According to August (as cited in Abedi, 2004) not having a uniform set of criteria for identifying ELLs could be a problem because there is not a national definition where state-by-state comparisons could be made among ELLs. Federal Level Title VII of the Improving America’s School Act of 1994 and more recently the No Child Left Behind Act (2001), offers the following definition to identify students with limited English proficiency (NCLB, 2001): 1. They are between 3 and 21 years of age. 2. They are not born in the United States. 3. Their language is a language other than English. 4. They have differences in listening, speaking, reading, writing, or understanding English – not meeting the state’s proficiency level of achievement to successfully achieve in English only classrooms. State Level Federal legislation leaves it to individual states to decide whether a student meets the definition of NCLB (North Central Regional Educational Laboratory, 2005). The way in which each state defines the English language learner will impact how it identifies this population of students, and ultimately, how they are evaluated. Some of the methods used to identify English language learners are student records, teacher observations, home language survey, language proficiency tests, and achievement and criterion-referenced tests (Rhodes, Orchoa, & Ortiz, 2005). It is important that each district knows which methods are used to identify these students