• unlimited access with print and download
    $ 37 00
  • read full document, no print or download, expires after 72 hours
    $ 4 99
More info
Unlimited access including download and printing, plus availability for reading and annotating in your in your Udini library.
  • Access to this article in your Udini library for 72 hours from purchase.
  • The article will not be available for download or print.
  • Upgrade to the full version of this document at a reduced price.
  • Your trial access payment is credited when purchasing the full version.
Buy
Continue searching

Investigating the reliability and validity of the Consortium on Reading Excellence (CORE) phonics survey

Dissertation
Author: Lorilynn Brandt
Abstract:
Phonics was identified as one of the critical components in reading development by the National Reading Panel. Over time, research has repeatedly identified phonics as important to early reading development. Given the compelling evidence supporting the teaching of phonics in early reading, it is critical to make sure that instructional decisions in phonics are based upon valid and reliable assessment data. This study examined the psychometric properties of the Consortium on Reading Excellence (CORE) Phonic Survey and was designed to establish instrument validity and reliability. Analyses indicated moderate to very strong validity and reliability coefficients. Additionally, a D study using generalizability analyses data identified the optimal assessment administration protocol for the CORE Phonics Survey to minimize the error variance and maximize the reliability under absolute and relative decision-making conditions.

CONTENTS

Page

ABSTRACT ................................................................................................................... iii

ACKNOWLEDGMENTS ............................................................................................. iv

LIST OF TABLES ......................................................................................................... vii

LIST OF FIGURES ....................................................................................................... viii

CHAPTER

I. INTRODUCTION ............................................................................................ 1

Phonics Assessment is Important .................................................................... 3 Validity and Reliability of Assessments .......................................................... 5 Problem Statement .......................................................................................... 6 Research Questions .......................................................................................... 8

II. REVIEW OF LITERATURE .......................................................................... 11

Results of the NRP Report on Decoding .......................................................... 11 Four Purposes of Reading Assessment ............................................................ 16 Psychometric Properties of Reading Assessment: Validity and Reliability ..... 20 The CORE Phonics Survey .............................................................................. 24 Summary .......................................................................................................... 27

III. METHODOLOGY .......................................................................................... 28

Research Questions .......................................................................................... 28 Instrument ........................................................................................................ 29 Design ............................................................................................................... 31 Procedures ....................................................................................................... 36 Data Analysis .................................................................................................. 42 Summary ......................................................................................................... 44

IV. RESULTS ......................................................................................................... 45

Results ............................................................................................................. 45 Summary .......................................................................................................... 70

vii

Page

V. DISCUSSION .................................................................................................. 72

Discussion of Validity Analysis ....................................................................... 72 Discussion of Reliability Analysis ................................................................... 76 Implications for Instruction .............................................................................. 80 Limitations ........................................................................................................ 81 Recommendations for Future Research ........................................................... 82

REFERENCES .............................................................................................................. 84

CURRICULUM VITAE ................................................................................................ 88

viii

LIST OF TABLES

Table Page

1. Criterion Validity: Comparison of Utah State Core Curriculum and CORE Phonic Survey ......................................................................................... 50

2. Criterion Validity: Comparison of Core Reader with CORE Phonics Survey .... 52

3. Descriptive Analysis of Confirmatory Factor Analysis ..................................... 54

4. Correlations Matrix for Confirmatory Factor Analysis ...................................... 54

5. Goodness-of-Fit Tests ........................................................................................ 56

6. Standardized Model Results ............................................................................... 58

7. R-Square ............................................................................................................. 59

8. Cronbach’s Alpha Analysis for CORE Phonics Survey .................................... 61

9. Estimated Variance Components and Standard Errors for Part 1 (Sections A-D) ..................................................................................................... 62

10. Estimated Variance Components and Standard Errors for Part 2 (Sections E-L) ...................................................................................................... 62

11. D Study Phi Coefficients .................................................................................... 67

12. Standard Error of Measurement .......................................................................... 70

ix

LIST OF FIGURES

Figure Page

1. Confirmatory factor analysis construct validity ...................................................... 53

2. G-stat for alphabet skills and letter sounds: Part 1 ................................................. 68

3. G-Stat for reading and decoding skills: Part 2 ........................................................ 68

CHAPTER I INTRODUCTION

Early reading success or failure sets the stage for future academic success or failure. Failure to acquire early literacy skills is a potential indicator of future reading difficulties in school (Jenkins & O’Connor, 2002). However, the ability to read is not just a necessary task for school. Success in reading is also important for lifelong achievement; how well a child learns to read may determine future opportunities, including career possibilities and the ability to accomplish the basic activities of daily life (McCardle & Chhabra, 2004). The National Institute of Child Health and Human Development (NICHD) convinced lawmakers that “the failure to learn to read reflects an educational and public health problem because this lack of success affects emotional health and overall well-being” (p. 4). Thus, currently in the United States, there is an increased focus on making sure all students are proficient readers and have the necessary skills to be successful. One very important literacy skill that students should know to be effective readers is phonics. Phonics has been identified by many as one of the crucial early literacy skills that make a significant difference in reading success (Cunningham & Cunningham, 1992; National Reading Panel [NRP], 2000). Phonics is the relationship between letters (graphemes) and their corresponding sounds (phonemes) (Adams, 1990; Ehri, 1998). A proficient reader is able to use this knowledge of letter/sound relationships to decode unknown words in text (National Research Council [NRC], 1999; Rasinski & Padak, 1996).

2

Throughout the history of reading instruction, phonics has been a topic of great discussion (Smith, 2002). Differing views have emerged in the last few decades among researchers regarding its importance in literacy instruction (Chall, 1967; Erhi, 1998; Flesch, 1955; Goodman, 1980; Smith, 1979). Some have claimed that phonics instruction is not an important element of learning to read while others claim that it is a very valuable skill for early readers. Research findings, however, have consistently supported the effectiveness of phonics instruction for early readers. One well-known review of research concerning reading and phonics is Chall’s Learning to Read: The Great Debate. Her conclusion from this comprehensive review was that instruction in phonics led to better achievement in reading. This conclusion has been supported in many subsequent research studies and reports (e.g., Adams, 1990; Anderson, Hiebert, Scott, & Wilkinson, 1985; Balmuth, 1982; Dykstra, 1968; Foreman & Moats, 2004). In the report Becoming Nation of Readers, it stated that “on the average, children who are taught phonics get off to a better start in learning to read, than those who are not taught phonics (Anderson et al., 1985, p. 37). Additionally, phonics knowledge is the single best predictor of reading comprehension (Stanovich, 1990; Vellutino & Scanlon, 1991), which is the ultimate goal of reading. Perhaps the most influential document supporting the critical importance of phonics instruction is that of the NRP Report (2000). In 1998, the U.S. Congress commissioned a panel of experts to review the current literature on reading and determine the most effective teaching methods and approaches to see whether specific instructional practices were linked to reading success. To meet this challenge, the panel adopted the

3

meta-analytic technique of comparing effect sizes from all studies since 1970 that used an experimental or quasi-experimental design with a control group or a multiple-baseline method and met specific rigorous research criteria. Of those research studies reviewed, 1,373 studies were directed to phonics, 38 of those met the research criteria established by the NRP, and 66 comparisons were made. Phonics was shown to be one of the critical components of reading instruction for both early readers and older readers and those students who received decoding instruction showed positive benefits in reading performance. Even the critics of this report showed that phonics instruction outperformed treatment conditions in which a more typical or moderate level of phonics instruction was provided” (Camilli, Vargas, & Yurecko, 2003, p. 34). The findings of this report are critical in establishing the importance of phonics instruction in reading education. Given that phonics knowledge is shown to be so important in beginning reading acquisition, it is crucial that phonics concepts be taught in the classroom as part of an effective reading instruction program. In order for teachers to effectively include phonics in their instruction, they need to know which concepts students know and which they do not. Phonics assessments provide teachers with important information which can help them screen and diagnose students’ phonics instructional needs as well as progress monitor the effectiveness of a variety of phonics interventions.

Phonics Assessment is Important

Assessments have a significant role in helping teachers determine the needs of

4

students to inform instructional decision making and intervention selection. They provide documentation of students’ performance and progress, so that instruction can be quickly changed or modified before the student falls too far behind their reading goals. A valid and reliable phonics assessment can give teachers valuable information. First, phonics assessment helps to pinpoint specific areas of student need so instruction and practice can be appropriately focused. Second, it helps provide screening and diagnostic information throughout the year. Third, assessment provides evidence of the effectiveness of instructional interventions (Consortium on Reading Excellence, 2004). Knowing that assessment is an important tool for monitoring student progress, the congressional law of No Child Left Behind 2001 included the mandate that all students in third, fifth, and seventh grade in public schools take annual statewide standardized tests. These tests are required as a means of showing evidence that students adequately progressed in reading during the school year. The ultimate goal is to identify the reading needs of students so that interventions can be made to improve student outcomes. Thus, schools and teachers are required to show documentation of student learning through these tests. The results of these high stakes assessments are used as a measure of the annual yearly progress (AYP) of the students and school and often determine the degree of federal funding they receive. States can also opt to use criterion referenced tests (CRTs) for demonstrating their AYP. Thus if CRTs are used, it is important that any phonics assessments given by teachers throughout the year provide information about what students know about the concepts to be tested on the CRT. Consequently, educators have increased interest in accessing and using valid and reliable phonics assessments

5

which can help identify students’ needs throughout the year giving evidence that students are adequately learning and are prepared for the phonics knowledge on these and other end-of-year tests. Significant strides have been made to more effectively assess young children’s early literacy skills (Good & Kaminski 2003; Wagner, Torgesen, & Rahsotte, 1999). Since the NRP report (2000) was published, educators have begun to align teaching and assessment with its findings, which revealed that effective reading instruction should include concepts and strategies that help students to develop phonological awareness skills, alphabetic understanding, reading fluency, vocabulary and comprehension. However, unless the tests that are selected for use are appropriately assessing these skills, teachers cannot be sure that students are indeed learning and progressing. Appropriate phonic assessments, would be defined as those that are easy to use, valid, and reliable.

Validity and Reliability of Assessments

Validity refers to whether a test truly assesses what it claims to (Gall, Gall, & Borg, 2007) and that the construct being assessed is appropriate, accurately represented, and meaningful (Rathvon, 2004). To be considered a valid test, both experts and empirical evidence must support the construct. Reliability refers to how consistently the test measures the construct and is repeatable. That means that the test results remain the same regardless of the rater, occasion or test format. If a test is not valid or reliable, the results cannot be considered representative of a student’s knowledge nor can it be relied upon for making accurate decisions for monitoring students’ progress. Therefore,

6

schools and teachers need to know the validity and reliability of tests when selecting them for use. Many teachers, however, may not be qualified or knowledgeable enough on this topic to investigate validity and reliability of assessments due to lack of training in this area and therefore may be using tests because they are popular or easily available. Unfortunately, this will not ultimately help students meet their literacy goals as these results may be giving wrong or inaccurate information. For example, if a phonics test is given with little validity evidence, the teacher may falsely assume that students understands phonics skills, when in reality the test is not addressing all the concepts that are part of the phonics construct. On the other hand, if a test is not reliable, the scores may vary each time it is given making it hard to identify a student’s true understanding of phonics or to track progress in phonics knowledge. This is highly problematic in an era when accountability for student achievement is increasing and teachers are required to show evidence of students’ progress. Therefore, it would be helpful to have valid and reliable assessments easily accessible to teachers that have already demonstrated adequate psychometric properties.

Problem Statement

Because the NRP identified phonics instruction as a critical or essential component of effective reading instruction, the need for valid and reliable assessments of phonics has resulted. One such phonics assessment that has recently been developed is the Consortium on Reading Excellence (CORE) Phonics Survey. It is included in a

7

compendium of reading assessments in the book entitled, CORE Assessing Reading: Multiple Measures. It is an informal test that examines various decoding concepts and skills routinely used in beginning reading (Bailey, 1967; Clymer, 1996; CORE, 2008). The CORE Phonics Survey is quick, easy to administer, and affordable. The cost is approximately $40.00 and can be copied for use within a school. It takes approximately ten minutes to administer. It would be considered a user-friendly test. Because the CORE Phonics Survey is very useable and assesses skills identified in the NRP report (2000) as important, it has gained much popularity with educators. It is currently being used quite extensively as part of the Utah Reading First instructional reform program. All of the Reading First schools in Utah currently use it to some degree as part of their assessment plan (interview with Rebecca Donaldson, November 2008). Several more Utah schools, not part of the Reading First program, also use this assessment (Cache County School District, November 2008). Moreover, it was found to be a popular assessment tool used by schools in other states across the nation. In a general internet search of Google and Yahoo, over 200,000 items surfaced that specifically reference this assessment, many of which were schools or district web pages that promoted the use of this assessment as part of their educational plan or Reading First proposals. Some of these states were Idaho, Utah, Colorado, Washington, California, Montana, and Hawaii. Other hits on the webpage were from colleges or universities outlining it as part of their teacher education courses. The popularity of the CORE Phonics Survey indicates its pervasive use as a phonic assessment tool in schools and universities. Given this pervasiveness, one

8

wonders whether this test has sound psychometric properties and whether the scores obtained from it can be regarded as valid or reliable. The early reading progress of literally thousands of students, in Utah alone, is being evaluated with this assessment and decisions are being made about students’ phonics knowledge based on the results of this test. In an effort to pursue further information about the CORE Phonics Survey, a call was made to the company, Consortium on Reading Instruction, and publisher, Area Press, on November 15, 2008. Both confirmed that they did not have any data on the validity and reliability of the CORE Phonic Survey. Next, a library search was done to see if any empirical research had been published in educational journals. Nothing resulted from any of these searches indicating that the validity and reliability of this assessment has been previously investigated. Therefore, the purpose of this study is to address this problem by thoroughly investigating the validity and reliability of the CORE Phonics Survey. Such a study will make an important contribution to teachers and to educational research in general because without this information, teachers using the CORE Phonics Survey cannot be certain that the scores derived there from can guide decisions about students’ phonics knowledge or their decoding instructional needs.

Research Questions

The research questions for this study will be driven by two different theoretical test theories. The first is Classical Test Theory (CTT) which assumes that every score on a test is composed of two components, the true score (the score that would be obtained if

9

there were no errors) and the measurement error (anything that prohibits the true score from showing). Although classical testing acknowledges error variance, it does not try to define or reduce it. Therefore it is not possible to have absolute confidence in what the true score is. Classical test theory is the testing approach that most studies of reliability and validity depend on (Reynolds, Livingston, & Willson, 2009). Modern Test Theory (MTT) tries to determine how that error variance can be reduced rather than just acknowledging it. Advantages of modern test theory are that (a) it allows researchers to estimate reliability of each measure rather than assuming all are equally reliable, (b) it yields various measure of goodness of fit for the overall model, (c) it compares alternative explanatory models systematically to test hypothesis about which factors influence observed correlations in the matrix and how these interrelate, and (d) it provides a way of partitioning the variance of the measure into separate trait, method and error components (Grimm & Yarnold, 2000). Both of these testing theories will be addressed as part of this study. Each section will be outlined as to whether validity or reliability issues are being address and whether the selected test is one of Classical Testing Theory or of Modern Testing Theory.

Validity Research Questions

Classical Test Theory 1. What is the evidence for consensus or content validity of the CORE Phonics Survey as measured by convergence or agreement among expert reviewers? 2. What is the evidence of criterion validity for the CORE Phonics Survey as

10

measured by the percent of overlap between items on it and the phonics items in the Utah State Core Curriculum and the Scott Foresman basal reading series?

Modern Test Theory 1. What is the evidence for construct validity on the CORE Phonics Survey using a confirmatory factor analysis to validate a hypothesized two-factor model?

Reliability Research Questions

Classical Test Theory 1. What is the evidence of interrater reliability on the CORE phonics Survey as measured by a Pearson’s r correlation coefficient? 2. What is the evidence of test-retest reliability on the CORE Phonics Survey as measured by a Pearson’s r correlation coefficient? 3. What is the evidence of internal consistency reliability on each subtest of the CORE Phonics Survey as measured by a Cronbach’s alpha coefficient?

Modern Test Theory 1. To what degree do the raters and occasions contribute to variance among scores on the CORE Phonics Survey as measured by a G study? 2. What is optimal number of occasions and raters when administering the CORE Phonics Survey to minimize error variance and optimize the reliability of the resulting rating as measured by a D study?

11

CHAPTER II REVIEW OF LITERATURE

Chall’s (1983) six-stage reading development model helps us understand the decoding development of early readers. This theoretical framework is the underlying premise that children need to learn phonics to make progress in reading. Therefore, it was important that supporting evidence be identified to reiterate that phonics instruction helps children to do so. This will be the first purpose of the review of literature. Then, since phonics instruction can only be as good as the assessment instruments used to inform that instruction, the second purpose of the review of literature was to define and discuss (a) the purposes of reading assessment generally, and (b) the necessary psychometric properties of valid and reliable reading assessment instruments. Finally, an investigation was done to verify whether any existing psychometric evidence exists to support the widely used phonics assessment, the CORE Phonics Survey.

Results of NRP Report on Decoding

The NRP (2000) synthesized the findings of existing studies on the effects of phonics instruction on young learners’ reading achievement. Specifically, the NRP conducted a literature search of experimental studies that compared the effectiveness of systematic phonics instruction to that of unsystematic phonics instruction. Systematic phonics instruction refers to instruction that has a sequential progress and a clearly identified set of skills, concepts, or strategies to be taught. For studies to be included in the NRP meta-analysis, each had to meet rigorous criteria. Thirty-eight studies that met

12

these criteria and were analyzed. Effect sizes were calculated to quantify the size of the effect of the treatment and determine and decide if that effect size was statistically greater than zero at p < .05. An effect size is “the degree to which the phenomenon is present in the population or the degree to which the null hypothesis is false” (Cohen, 1988, pp. 9- 10). Estimates of effect size provide essential information because they provide information about the relative magnitude of outcomes. The scale of significance for the effect size is defined as follows: .00 to.19 is described as trivial effect; .20 to .49, small; .50 to .79, moderate; .80 or higher, large (Cohen). Performance on six phonics-based outcomes was considered: decoding regularly spelled real words, decoding pseudo words, reading real words that included irregular spellings, comprehending text, and reading connected text orally. Effect sizes in most of these measures were positive and significantly greater than zero, indicating that in most studies the group receiving systematic phonics instruction evidenced higher reading achievement than the control group who did not. The effect sizes were, however, significantly higher for studies with kindergarteners and first graders than with those of second through sixth graders. This finding suggests two things. First is that phonics is time sensitive information and needs to be learned early. Second, this finding suggests that phonics instruction is a better prevention from reading difficulties than it is as a cure once reading difficulties have resulted. The categories that had the strongest effect size for both early readers and later readers were decoding regular words and decoding pseudo words. Effect sizes were also calculated for various related subsets of the studies

13

reviewed to break down how phonics instruction affected students’ performance in various situations and across distinct characteristics. The first subset addressed was the time of the testing, either at the end of the program or the end of the year. Both effects showed to be statistically greater than zero and moderate in size, regardless of whether effects were measured at the end of the program (.41) or the end of the first year (.44). This indicates that systematic phonics instruction helps children learn to read more effectively than those who do not receive it and that the timing of the testing does not make much difference. Phonics instruction also improved reading ability in both early readers and older readers. Effect sizes were statistically greater than zero for both, but were larger for studies with kindergarteners and first graders (0.55) than for studies with second through sixth grades (0.27). This indicates that although phonics instruction does have a positive effect on both ages, its strongest impact is in the early stages of reading acquisition. Another subset analyzed the effect that phonics instruction had on students with differing reading abilities. Effects were statistically significant for all groups with the exception of second through sixth low-achieving students (0.15). At-risk and typically achieving readers in kindergarten and first grade both showed moderate to large differences when receiving phonic instruction. At-risk first graders were most affected by phonic instruction, with a strong effect size (0.74). Second through sixth grade low achieving student showed the smallest benefits (0.15). Effect sizes were small to moderate for the second- through sixth-grade students who are typically achieving readers (0.27) and students with reading disabilities (0.32). Thus, phonic instruction

14

improves reading ability more than no phonics instruction not only for beginning readers but also among typically progressing readers above first grade and older readers with reading disabilities. However, phonics did not enhance reading ability as much among low-achieving readers in Grades 2 through 6. Studies reporting the socioeconomic status (SES) of participants were also examined. Effects were strong for children of low SES (0.66) and middle SES (0.44). This indicates that phonics instruction helps children in learning to read regardless of SES level. Effect size results were similar when considering the sizes of the group receiving the instruction, whether it is individuals (0.57), small groups (0.34), or classrooms (0.39). This means that classroom instruction may be just as effective as tutoring without the increased expense and difficulty of one-on-one teaching. Effects were also examined for three types of systematic phonics programs. One category was synthetic phonics, which involves teaching students to sound out letters and blend sounds into words. This effect size was strong at (0.45). Another category was to analyze and blend larger units of words such as onset, rimes, or spelling patterns. This effect size was moderate (0.34). Finally, a miscellaneous category included traditional spelling or basal programs or instruction on word analysis, which had a moderate effect size of (0.27). Effect sizes for all three categories were statistically greater than zero and would thus indicate that all of the types of systematic phonics programs were more effective than nonsystematic or not phonic program at all. As long as programs are systematic, it does not seem to matter which approach is used.

15

The type of instruction given to the control group in each study also varied from study to study. Effect sizes for each type of group were calculated. Control groups were categorized as basal groups and had an effect size of 0.46, regular curriculum was 0.41, whole language was 0.31, whole word was 0.51, or miscellaneous 0.46. The effect sizes for all of these was a moderate to strong positive indicating that phonics-instructed groups performed better than the other types of groups. Effect sizes were also statistically greater for groups receiving systematic phonics. Finally, studies also differed in their design, specifically their method of assigning students to experimental groups. Effect sizes were calculated to investigate how the design impacted the outcomes. Some studies randomly assigned students to treatment and control groups while others used preexisting groups. Additionally, some studies used large sample sizes whereas others worked with fewer students. Effect sizes for the more rigorous designs using larger groups and random assignment, were as large as or larger (0.45) than the effect sizes of the less rigorously assigned groups (0.43). This is not much of a difference and would indicate that the positive effects of phonics instruction were not due to comparison with weaker designs. In summary, findings of the NRP’s (2000) meta-analysis support the conclusion that systematic phonics instruction helps all children to learn to read more quickly, easily, and with greater success than nonsystematic or no phonics instruction. The impact was significantly greater in early grades (K-1) when phonics was the method used to start students out, than in later grades (2-6) after they had made some progress in reading with other methods. The instructional approach or specific program used to teach phonics in

Full document contains 102 pages
Abstract: Phonics was identified as one of the critical components in reading development by the National Reading Panel. Over time, research has repeatedly identified phonics as important to early reading development. Given the compelling evidence supporting the teaching of phonics in early reading, it is critical to make sure that instructional decisions in phonics are based upon valid and reliable assessment data. This study examined the psychometric properties of the Consortium on Reading Excellence (CORE) Phonic Survey and was designed to establish instrument validity and reliability. Analyses indicated moderate to very strong validity and reliability coefficients. Additionally, a D study using generalizability analyses data identified the optimal assessment administration protocol for the CORE Phonics Survey to minimize the error variance and maximize the reliability under absolute and relative decision-making conditions.