July 2006

Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:

Executive Summary

The first set of analyses examined the simple descriptive statistics comparing the percentage of schools that meet the AMOs for the school-as-a-whole subgroup and for the special education subgroup (see Table 4) (we acknowledge that it seems ironic to call the "school-as-a-whole" a subgroup, but that is a specific NCLB defined subgroup). Notably, the pass rate for schools with regard to special education is quite low compared to the school-as-a-whole. In other words, the performance of the special education subgroup will lead to schools’ failure at a noticeably higher rate than for the school-as-a-whole. The final column of Table 4 shows the percentage of schools reaching AMOs for the student body-as-a-whole, but lacking sufficient cell sizes to assess the progress of special education students. Several details of this table bear mentioning. In the five states studied, over 80 percent of schools that passed their subgroup AMO did so without assessing the proficiency of their special education students. An additional finding from these analyses is the variability in passing rates (minimum approximately 46%, maximum approximately 92%). The two states with the lowest passing rates (States 4 and 5) are the two states currently testing every grade. Again, these results are aggregated across all minimum cell sizes and confidence intervals.

*Passed both components or passed school-as-a-whole but lacked minimum-n in special education.
 

The number of students required to define a set of students as a group has been one of the most discussed aspects of states’ implementation of AYP calculations. It has been argued previously (e.g., Marion, et al., 2002) that minimum-n is much less of a reliability issue than a consequential validity concern. The analyses presented in Table 5 document the effects, while holding all other aspects of states’ accountability plans constant, of altering the minimum number of students necessary to constitute a subgroup on the percent of schools passing AMOs for each of the five states. As one would expect, an increase in the minimum cell size was associated with an increase in the percentage of schools passing AMOs. All but one state (State 1) showed a difference of more than 25 percentage points. Perhaps this is due to this state’s having "less room" for change.

Two analyses were conducted to examine the consequences on special education students of increasing the minimum-n. The first demonstrates quite conclusively for these states that as the cell size requirements increase, fewer schools are held accountable for ensuring that their special education students meet the AMOs. Table 6 shows, for each minimum cell size, the percentage of schools passing their AMOs but without sufficient numbers of special education students to assess their performance. When minimum cell sizes approached 60, almost 100 percent of schools in all five states were able to "pass" AYP without the performance of special education students taken into account.

The second analysis focuses on the percentage of special education students that would be excluded from the accountability system as a function of increasing cell size. We recognize that these students are not fully excluded because they count in the whole school calculations, but practically for most AMO levels, schools could feasibly ignore the performance of special education students until 2011 or so. Table 7 shows the percentage of tested special education students excluded from the AYP calculations for each state and cell size. For the three states not testing every grade, more than one-third of special education students were excluded from AYP calculations at a minimum cell size of 20. For these states, by the point the minimum cell size reached 60 students, nearly 100 percent of special education students were not included in the AYP calculations. This has consequences for special education students and on the validity of the accountability system.

One approach that has been advocated for improving the reliability of AYP decisions has been to use confidence intervals around either the AMO or the school’s observed score (e.g., Hill & DePascale, 2003; Marion et al., 2002). In these analyses, the confidence interval was varied while the minimum-n was held constant at the average of the minimum-n values tested earlier. It is a mathematical necessity that passing rates increase with the increasing confidence interval on the target AMO; however, the increase is quite small compared to the results for minimum cell sizes (see Table 8). Appendix B describes the inferential statistical analyses underlying conclusions presented in this report.

States are required to test every grade, 3–8 and once in high school, by the 2005–2006 school year. Prior to that year, schools were required to test students once each in elementary, middle, and high school. With fewer grades being tested, there are fewer students eligible to meet minimum cell sizes. Further, confidence intervals vary inversely as a function of sample size (i.e., they are wider when sample sizes are smaller). Therefore, if the level of the confidence interval does not change, they will, by definition, be narrower when more students are included in the system. Similarly, with more grades tested, more subgroups will meet the minimum-n threshold (assuming it stays at the same level). The analyses presented in this section project how the various design decisions play out when the full assessment system is implemented.

Three of the five states (States 1, 2 and 3) did not test every grade in recent years. Data from these states’ October, 2004, enumeration of their schools’ enrollments was used to make projections of passing rates likely when every grade, 3–8, is tested. It was assumed that the untested students were sampled from the sample population as tested students and, therefore, the percent proficient for the tested and untested groups was identical. It was also assumed that the proportion of special education students was the same between the tested and untested grades. Each school’s total enrollment, grades 3–8, was used as the participant count for analyses by minimum cell size and as sample size in the calculation of the confidence intervals for the analyses by confidence interval size.

Tables 9 and 10 show projected numbers of students and passing rates for the three sampled states currently testing two or three grades if they were to test every grade in grades three through eight. Table 11 shows the differences in pass rates from partial to every grade testing for these three states. As one would expect, the pass rates for the student body as a whole did not change very much from partial to complete grade testing. However, the overall pass rate decreased between approximately 7–20 percent.

* Actual data from States 4 and 5 repeated for ease of comparison. "Passing" in this column refers to those subgroups actually meeting the AMO or not having enough students to constitute a subgroup.

* Passed both components or passed school-as-a-whole but lacked minimum n in special education.
 

As more students are added into the system, more schools will meet the minimum-n thresholds for various subgroups. The pattern of projected percentages of schools passing AYP at varying levels of minimum cell size (see Table 12) is similar to the pattern for testing fewer students (see Table 5), although slightly fewer schools are able to pass with more students included. Even with the additional students included in the system, a majority of the projected passing schools do so without having sufficient numbers of special education to constitute a subgroup once the minimum-n reaches 30 students (see Table 13). Likewise, once the minimum-n reaches 20 or 30 students, significant percentages of special education students are excluded from the accountability system even with all grades tested (see Table 14). Figures 1–3 show the exclusion rates for the three states without a full assessment system now compared with the exclusion rates when the system is fully built out as a function of cell size.

* Actual data from States 4 and 5 repeated for ease of comparison.

* Actual data from States 4 and 5 repeated for ease of comparison.

* Actual data from States 4 and 5 repeated for ease of comparison.

When more students are added into the system, the width of the confidence interval bands will decrease. The general pattern found for all grades testing were similar to those from the analyses for partial grade testing (see Table 15).

While states have flexibility in meeting the NCLB reliability expectations, their choices can lead to severe consequences for special education students. Most troublesome is the application of high minimum-n requirements. When the minimum-n was simulated to equal 60 students (well within the range of state values), more than half of the special education students in four of the five states—even when projecting all grades testing—were excluded as an explicit subgroup from the accountability system.

Increases in minimum cell sizes for the special education subgroup were associated with a large increase in passing rates for each of the five states assessed. This increase was due, in large part, to schools being less likely to have to include the results for the special education subgroup as the minimum cell size increased. In line with earlier predictions (Marion, 2004), it is considerably easier for a school to meet its AMO without reporting the proficiency of their special education students. Increased confidence interval sizes were also associated with an increase in pass rates, but a much smaller increase. While raising the minimum-n is an effective means of increasing the passing rates of schools, it does so at a considerable cost to special education students in terms of being excluded from the accountability system. If the implicit theory of action guiding NCLB accountability requirements is to improve instruction and thus outcomes for all students, schools and districts must be accountable for all subgroups in order to ensure that these students are appropriately served. The effect of increasing the minimum-n to exclude substantial portions of special education students must be considered a threat to the validity of the accountability system.

Many more special education students’ data are reflected in the accountability results when all grades are tested. This assumes that states will not increase the minimum-n as more grades are tested. If they do so, then it will likely be a wash between the increase in available students and the loss of these students through increases in required cell sizes.

Although confidence intervals have been suggested as a means of increasing the reliability of school identifications as well as reducing the number of schools failing to make AYP (i.e., because it will reduce those falsely identified), the data presented in this study suggests that confidence intervals have a much smaller impact on AYP pass rates than minimum-n changes. One of the reasons for this finding is the relatively large difference between the observed performance of the special education subgroup and the performance targets in the five states. Three of the five states had relatively high AMOs (e.g., > 60% proficient). If only a small proportion of special education students are scoring proficient, then the confidence intervals will still not be wide enough to overlap the AMO. In other words, if the difference between the percent of special education students scoring proficient and the AMO is large, confidence intervals will still not "help," assuming the motive for adjustment is to reduce numbers of schools identified as not meeting AYP. In only one of the five states did more than 50 percent of the schools have their special education subgroup meet the state’s AMOs.

Confidence intervals will not help the special education subgroup pass when they should really not pass (i.e., they are far below the AMO), but can help the state leaders make this decision more reliability. On the other hand, minimum-n approaches do little to improve the reliability of subgroup decisions (at least within the range of minimum-n levels being used by most states), but can have severe negative consequences for subgroups excluded and, by extension, threaten the validity of the accountability system.

Forte Fast, E., & Erpenbach, W. J. (2004). Revisiting statewide educational accountability under NCLB: A summary of requests in 2003–2004 for amendments to state accountability plans. Washington, D.C.: Council of Chief State Schools Officers.

Hill, R.K., & DePascale, C.A. (2003). Reliability of No Child Left Behind accountability designs. Educational Measurement: Issues and Practice, 22(3), 12-20.

Marion, S. F., White, C., Carlson, D., Erpenbach, W. J., Rabinowitz, S., & Sheinker, J. (2002). Making valid and reliable decisions in the determination of adequate yearly progress: A paper in the series: Implementing the state accountability system requirements under the No Child Left Behind act of 2001. Washington, D.C.: Council of Chief State Schools Officers.

Marion, S. (2004). An analysis of differential rates of states’ identifying schools as "not meeting AYP" in 2003 under the federal No Child Left Behind law. Dover, NH: National Center for the Improvement of Educational Assessment.

Marion, S. F., & Gong, B. (2003). Evaluating the validity of states’ accountability systems. Paper presented at the Reidy Interactive Lecture Series. October 9–10. Nashua, NH.

P. L. 107–110 ‘‘No Child Left Behind Act of 2001,’’ Title I-Improving the Academic Achievement of the Disadvantaged, Section 1111.

Separate repeated measures logistic regressions were conducted for each of the five state’s passage determinations. SAS, version 8.02, proc GENMOD was used (SAS Institute, 2001). The independent variables were minimum cell size and confidence interval size. The logistic regression function in these analyses describes the probability of a school failing. Regression coefficients in the current analyses describe the degree of association between increasing values of the predictor variables with the probability of failing. Cell size and confidence interval size were dummy-coded into a set of dichotomous variables comparing the probability of being declared non-proficient in the very highest level of the variable with that in the other levels. For instance, in one state’s data, the logistic regression coefficient for a minimum cell size of 10 was 1.82 (.18), Z = 10.36, p < .0001. This coefficient indicates that a school using a minimum cell size of 10 was approximately 6 times more likely to be declared failing than a school with a minimum cell size of 100 special education students.

Regression coefficients comparing the lower minimum cell sizes with the highest minimum cell sizes were always significantly different from 0. On the other hand, when regression coefficients for comparing the widest confidence interval sizes with other confidence interval sizes were significant, it was usually only for the narrowest confidence intervals, and these coefficients were always smaller than those comparing cell sizes. When regression coefficients for the combinations of cell size and confidence interval size were significant, it was only for the combinations of lowest cell sizes and narrowest confidence intervals. This interaction effect was, however, of little substantive interest. The interaction between cell size and confidence interval size could not be assessed for State 1’s original data, most likely because of collinearity. Results were similar for the analyses conducted with projected cell sizes.