Krentz, J., Thurlow, M., & Callender, S. (2000). Accountability systems and counting students with disabilities (TechnicalReport 29). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/Technical29.htm

This report is the first by the National Center on Educational Outcomes (NCEO) to examine the extent to which students with disabilities are included in accountability systems of states in which there are high stakes for educational systems. Conducted through a Web search of states’ education Web sites and direct links to their Web sites, we examined the indicators on which states collect information (e.g., student indicators, staff indicators, program indicators, and use of resources indicators) as well as the consequences provided by the states (e.g., rewards, assistance, or sanctions).

Of the 280 indicators listed for these states according to the Education Commission of the States (ECS), NCEO was able to find documentation of 178 of them on the states’ Web sites or direct links. Furthermore, NCEO found an additional 216 indicators not identified by ECS. However, it was difficult to determine the extent to which any of these indicators were used in making determinations for consequences. Information for many indicators was collected and reported but not included in the actual formulas for determining rewards, assistance, or sanctions. Most often, assessment scores served as the only indicator used for making this determination.

Five states had accountability formulas that could be identified in information found on state Web sites. Louisiana, Mississippi, New York, North Carolina, and Ohio listed their indicators and criteria as well as how they are used to determine the consequences for states. From the information found, other states may have some of these components of an accountability system, but did not provide the level of detail on their Web sites needed to determine how their systems were implemented.

This study highlighted that not only was it difficult to determine exactly what indicators factored into overall accountability systems, but it was also difficult to find evidence that students with disabilities were included in accountability systems. Although students with disabilities may be included in assessments, their scores may not be included in the accountability systems, and most states did not specifically state whether students with disabilities were included when defining the indicators used.

Overview

Educational accountability is gaining attention and influence as state and local policymakers implement sweeping accountability reforms. States are increasingly relying on large-scale assessments to measure student performance and to determine sanctions and rewards (Education Commission of the States, 1999; Olson, Bond, & Andrews, 1999). The Web sites of major organizations, such as the Council of Chief State School Officers (CCSSO: http://www.ccsso.org) and the Education Commission of the States (http://www.ecs.org) reflect this emphasis in content on their Web pages and their on-line reports.

Approximately half the states now use, or will soon use, tests in making decisions about student promotion (see Quenemoen, Lehr, Thurlow, Thompson, & Bolt, 2000) and high school graduation (see Guy, Shin, Lee, & Thurlow, 1999). All but two states have statewide assessments in place, many with more than one assessment (Olson, Bond, & Andrews, 1999). In the 2000 legislative sessions, over 300 bills were introduced in 41 states that dealt with statewide assessments (Coleman, Heller, & McNeil, 2000; see also the Web site for the National Conference of State Legislatures, http://www.ncsl.org/programs/educ/k12link.htm).

Not only are states increasingly attaching high stakes consequences to students for their performance on assessments (e.g., no diploma, no promotion to the next grade), but states are also attaching significant consequences to various aspects of the educational system (e.g., to teachers, schools, and administrators). This approach, known as system accountability, occurs when educators, administrators, schools, or districts are held responsible and consequences are assigned to them for various student performance indicators. For example, school accreditation may be based on test scores; teachers may receive cash rewards for student performance; or, administrators may be either sanctioned or rewarded based on school improvement, which in turn is measured to a large extent by student performance.

Within system accountability, there are different levels of consequences – referred to as the “stakes” of the assessment. At one end is simply publishing test scores – usually considered to be low stakes. While it might be considered “positive feedback” to have a school that performed well on test scores, or conversely, might prove an embarrassment for schools or districts that performed poorly, these consequences are relatively minor. In contrast, if teachers are assigned to “less prestigious” schools or receive cash bonuses based on student performance, the stakes are considerably higher. Similarly, if principals receive increased school funds or lose their jobs based on how students in the school perform, the stakes are very high.

In addition to obvious “rewards” and “punishments,” there is another type of consequence currently used in accountability systems. This type of consequence involves assistance to the system and is generally considered to be a much less significant consequence.  An example of assistance might include the provision of additional resources, such as monetary grants or specially trained personnel to schools that receive a “warning.” The distinction between “assistance” and “sanctions” is often subtle, at best. The “assistance” phase sometimes serves as a probationary period prior to more intense intervention or reconstitution of a school or district.

The Education Commission of the States (1999) has documented the specific indicators that states are using in their accountability systems. Although an array of indicators is listed for most states, it is unclear how they are used. Many questions remain about current and proposed accountability systems. Which indicators are truly high stakes? Which are simply included in a “report card” for a school or district? Are the indicators weighted in some manner or incorporated in a formula to develop an accountability measure? Are there positive rewards for successful schools and systems, assistance for struggling schools and systems, or sanctions for those who are not performing as expected?

In addition to these basic questions about state accountability systems, there are also questions that focus on specific groups of students. With the reauthorization of the Individuals with Disabilities Education Act in 1997 (IDEA 97), states and districts must include students with disabilities in their assessments. They are also required to report on the performance of students with disabilities, both aggregated with other students, and disaggregated.

Several analyses from before and after the enactment of IDEA 97 indicated that states have started to include students with disabilities in their assessments only recently (McGrew, Thurlow, Shriner, & Spiegel, 1992; Thompson & Thurlow, 1999), and that many states are not yet reporting the results (Thurlow, Langenfeld, Nelson, Shin, & Coleman, 1998; Thurlow, Nelson, Teelucksingh, & Ysseldyke, 2000) even though they are required to do so by federal law. It is a logical follow-up question to ask about the extent to which the increasing participation of students with disabilities in assessments translates into their inclusion in accountability systems.

Of course, there are many reasons for including students with disabilities in accountability systems (Thurlow, Elliott, & Ysseldyke, 1998). If these students are not included in accountability systems, it means that roughly 10% of the student population is excluded from accountability. Students who are left out of assessments and accountability systems tend to not be considered when reform efforts are being designed and implemented. Furthermore, the education system does not obtain an accurate picture of education if a significant portion of students is excluded, nor is it able to make “fair” comparisons. Participation of students in accountability systems seems to promote higher expectations for those students who are included, and consequently, may be assumed to exclude from high expectations those students who are not included.

The purpose of this research was to examine the extent to which students with disabilities are included in accountability indicators in those states in which there are high stakes for systems. To do this, we had to first decipher the accountability systems that states use – this involved identifying both the indicators that are part of the accountability system and the “formula” that is used in each state to combine the indicators to determine an accountability score. We restricted our search to accountability indicators and formula used to determine system-level consequences and did not search for formula developed exclusively to meet Title I requirements for adequate yearly progress. From the indicators and formula identified, we examined the extent to which students with disabilities were included in the state’s accountability system.

Method

Initially, we used the document Education Accountability Systems in 50 States (Education Commission of the States/ECS, 1999) to determine which states we would include in our analysis. We selected states that ECS had listed as having a “statute” or “regulation” in the Rewards or Sanctions column of its Table I. There were 38 states that met this criterion. The ECS document also included information about which indicators were used by the various states (see ECS Table 5). We divided in half the list of states that had high stakes accountability systems for schools or districts, then two of us examined Web sites to verify and expand on the information that was included in the ECS report.

We used the Achieve Web site (http://www.achieve.org) “State Links” page as a quick link to the Web site for the Department of Education of each state. From this site, we also located other state offices (e.g., accountability, assessment, legislature).

We also used the Council of Chief State School Officers (CCSSO) Web site (http://www.ccsso.org) section on “Council Projects” where we located the Accountability Systems Profiles. (The direct Web address is http://www.ccsso.org/introprofile.html). According to this site, the Resource Center on Educational Equity of the Council of Chief State School Officers (CCSSO), in collaboration with the New England Comprehensive Assistance Center, is developing profiles of several state’s education accountability systems.

When we checked the site, there was information about the following 13 states: Alabama, Connecticut, Georgia, Illinois, Kentucky, Maryland, Massachusetts, New Jersey, North Carolina, Oregon, Rhode Island, Texas, and Vermont. The profiles included information about each state’s content and performance standards, statewide student assessment, and criteria used to determine whether schools or districts are eligible for rewards or are in need of improvement. The site also contained information about how school and district performance is publicized, as well as details about the provision of rewards, assistance, and sanctions. There was often helpful information about the inclusion of students with disabilities in the assessment systems, as well as information about provisions for English Language Learners.

The Resource Center on Educational Equity page of the CCSSO Web site indicated that the information for the 13 profiled states was gathered from various sources, including CCSSO reports, state statutes and regulations, and information posted on state education agency Web sites. The Center’s initial drafts were reviewed and revised by state education agency staff. We used the information from CCSSO’s state profiles only after independently verifying it on state Web sites.

At each state’s Web site we reviewed state accountability reports, report cards, policy guidelines, and other relevant items to identify indicators, consequences, and accountability formula. This review was conducted in 1999-2000, with all reviews completed by July 2000. After we completed our initial examinations of Web site information on states, the two lists of states were traded, so that the information could be verified. In some instances, there was little change from what the first examination had indicated. In other cases there was completely new and sometimes contradictory information obtained during the second examination of the Web site. This report contains the more recent version of those sites.

Results

It became very evident during our verification process that the information we sought was a moving target. It changed rapidly due to ongoing legislative processes, subsequent rule-making procedures that follow legislative sessions (in order to implement the legislation just passed), and states’ action taken in response to federal legislation. Our focus was on Web-based information, and we found a significant disparity between the level of sophistication and quality of information found on various states’ Web sites. Some seemed to have frequent additions or modifications, and others seemed to be fairly rudimentary. Links to other Web sites were prominent and user-friendly on some sites, and unavailable or very obscure on others; sites were frequently “down” or “temporarily under construction.” For example, important documents found during the first review were often moved or unavailable during the second review. Sometimes Web sites would change dramatically within a few days.

We collected information on the indicators used for accountability purposes, the formulas for calculation of school or district scores for accountability purposes, the consequences of the accountability systems (e.g., rewards, assistance, or sanctions), and the inclusion of students with disabilities in the accountability systems’ indicators and calculations.

Accountability Indicators

Table 1 summarizes the numbers of indicators that were identified in the ECS document and in our Web search. Of the 280 specific indicators listed in the ECS document, we were able to verify 178 of them when examining the State Departments of Education Web sites and direct link sites. In other words, we found, on average, only 64% of the indicators that ECS found using written reports. (Values ranged from 15% for Massachusetts to 100% for Alaska, Colorado, Kansas, Kentucky, New Jersey, Oklahoma, Rhode Island, South Carolina, Tennessee, Vermont, and West Virginia.)

Table 1. Indicators in State Accountability Systems

We also discovered on State Department Web sites 216 additional indicators used by states for their accountability systems. While the majority of indicators listed were for reporting purposes only (i.e., they are documented in school and district report cards), some of these indicators are used in accountability formulas to calculate school or district scores to determine access to rewards or assistance or the receipt of sanctions.

Although a state may have indicated on its Web site that it uses a large number of indicators, we found that it was often the case that the majority of the indicators were of the same type (e.g., all assessment scores). Thus, looking at only the number of indicators does not reflect the nature of the accountability system. To get a different picture, we looked at the spread of types of indicators, and compared these to the types found in the ECS analysis. The results of this comparison are provided in Appendix A. As is evident, relatively few different types of indicators are being used by states, and the consistency between the types our search revealed and those ECS identified are relatively similar.

Accountability Formula

Five states (Louisiana, Mississippi, North Carolina, New York, and Ohio) had accountability formulas using indicators to calculate school or district scores listed on their Web sites. The formulas are quite diverse, from those that simply divide the number of students reaching a given proficiency level by the total number of students taking the test, to those that have included factors for regression to the mean.

Other states have information on their Web sites that were close to being formulas, but are too vague to actually determine how accountability is calculated. For example, Indiana noted that a school receives a monetary incentive if the school has demonstrated improvement over the average of its scores for the past three years in two or more areas (with the areas defined as attendance rates, math scores, language arts scores, and ISTEP + total battery scores). Similarly, Pennsylvania indicates on its Web site that it gives rewards based on improvements in achievement rates over the past two year average, or higher graduation rates or attendance rates over a criteria set each year (but the criteria were not provided). The state also indicated that dropout rates are collected but not counted in the reward system. Finally, Kentucky’s Web site clearly indicates that it has an accountability index that is a statistic that combines a school’s academic, nonacademic, and other “appropriate data,” but how these elements are combined is not evident. Dropout criteria also must be met, although the specific criteria are not stated. Kentucky also indicates that it uses a regression model to compare expected growth with actual growth in determining when a reward is given.

Of the five states’ formulas, only one specifically mentions students with disabilities (and another mentions English Language Learners). In the four states that do not mention students with disabilities, the implication is that all students are included in the same way. The one formula that did identify students with disabilities actually factored in a weight that decreases the impact of the scores of students with disabilities on the resulting accountability index. The rationale for this approach is that special education students might take longer to achieve standards because of lost opportunities to learn. Detailed descriptions of the formula are provided in Appendix B.

Consequences

Table 2 summarizes the nature of consequences of school or district performance in each of the states examined. It is important to note that this table is a point-in-time snapshot. Several states indicated that they are currently changing their existing accountability system or are implementing new components to their accountability systems. Currently under development are reward systems for five states, assistance programs for two states, and sanctions for three states.

Table 2. Nature of Consequences in State Accountability Systems

Insufficient information was available on some of the Web sites to determine whether any rewards, assistance, or sanctions were given. Eleven states did not include any information about rewards for high achieving schools or districts. Seven states did not include any information about assistance programs for low achieving schools or districts on their Web sites. Eight states did not include any information about sanctions for chronically low achieving schools or districts.

Of the 38 states listed as having high stakes system accountability, information on consequences could be found on the Web sites of 34 of them. Of the 34 states with some consequence in place for performance, 17 had all three components: rewards, assistance, and sanctions. An additional seven states had both assistance and sanctions. Of the 34 states that provided information on their accountability components, seven had no form of sanctions, seven had no form of assistance, and nine had no form of reward.

Rewards

Of the states that provided information on their accountability components, 22 states indicated that they provided rewards for schools or districts with high achievement (based on their indicators). Table 3 summarizes the types of rewards offered by these states. Details of the rewards provided by each state are available in Appendix C.

Table 3. Rewards for High Achieving Schools