Are We there Yet? Accountability for the Performance of Students with Disabilities
NCEO Technical Report 33
Published by the National Center on Educational Outcomes
Sara Bolt, Jane Krentz, and Martha Thurlow
Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:
Bolt., S., Krentz, J., & Thurlow, M. (2002). Are we there yet? Accountability for the performance of students with disabilities (Technical Report 33). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/Technical33.htm
Accountability systems that provide rewards, sanctions, or assistance to schools based on student achievement outcomes are currently being developed and implemented in nearly every state. The purpose of this study was to identify and describe these systems, as well as to determine the degree to which publicly available documents clearly articulate whether students with disabilities are included in accountability calculations. A search was conducted on each state educational agency Web site for information pertaining to school-level accountability. State accountability and Title I directors were also contacted to provide additional information. One of the greatest challenges in studying accountability systems is their transitory nature.
Results of our search illustrate that many systems are currently in a stage of revision. Primary indicators in current school accountability systems include student performance on statewide assessments, performance growth on statewide assessments, attendance rates, and dropout rates. Although it is often clear that students with disabilities participate in statewide assessments, it is frequently less clear whether their scores count in school accountability calculations. In only a handful of states is it entirely clear that all students with disabilities truly count in the determination of consequences for schools.
Statewide accountability systems designed to promote student achievement are beginning to play a very significant role in the way resources are allocated to schools. The rationale for implementing accountability systems stems from a belief that education can be improved when clear standards for student achievement are communicated to both students and educators, achievement toward those standards is measured, and appropriate consequences are linked to levels of student achievement. The intent of these accountability systems is to promote behavior change among schools in ways that will increase positive outcomes for students.
One of the challenges of researching accountability and assessment systems is that these systems are constantly being revised due to legislative and administrative action. Historically, states and local school boards have determined education and educational accountability policies. However, in the past decade the federal government has played an increasingly prominent role in this area.
The Web sites of major organizations such as Education Commission of the States (http://www.ecs.org), the National Conference of State Legislatures (http://www.ncsl.org) and the Council of Chief State School Officers (http://www.ccsso.org) provide interesting perspectives on the increased attention to assessment and accountability. Many states are attaching high stakes consequences to students for their performance on assessments, such as withholding diplomas (Center on Education Policy, 2002; Guy, Shin, Lee, & Thurlow, 1999; Johnson & Thurlow, 2002) and grade retention (Quenemoen, Lehr, Thurlow, Thompson, & Bolt, 2000), as well as attaching significant consequences to schools and school districts for the overall performance of students (Krentz, Thurlow, & Callender, 2000). The latter approach, known as system accountability, occurs when educators, administrators, schools, or districts are held responsible and consequences assigned for various student performance indicators. For example, school accreditation may be based on test scores; teachers may receive cash rewards for student performance; administrators may be either rewarded or sanctioned based on school improvement, which, in turn, is measured to a large degree by student performance.
As states have proceeded in constructing their student and school or district accountability systems, federal requirements have made it clear that students with disabilities must be included in accountability. Initial efforts to include students with disabilities in assessment systems have realized significant progress in assessment participation and the development of alternate assessments for those students unable to participate in regular assessments (Thurlow, 2000). In order to promote the positive intended consequences of accountability systems for all students, Linn (2000) points to the importance of including all students in accountability calculations. The next step, therefore, is to identify ways in which to include the scores of students with disabilities in accountability systems.
The concept of including the assessment scores of students with disabilities in accountability is simple on the face of the matter, but there are several complications associated with including the scores obtained by students with disabilities that challenge accountability systems. These complications include what have been called "non-standard" administration of assessments (i.e., assessments in which students have used accommodations that are considered to change what the test is intended to measure), and alternate assessments, which generally are designed for students with significant cognitive disabilities, and which therefore seem to be "different" from regular assessments.
Despite these complications, it is important to include students with disabilities in accountability systems, particularly those with school or district consequences. Therefore, it is also important to examine states’ progress toward this end. The purpose of the study reported here was to do just that. We focused on statewide system accountability with rewards, sanctions, or assistance provided for schools or districts. We examined Web sites of state educational agencies to determine whether accountability systems were in place, were being phased in, or were in the development phase. We looked at various indicators used by states to determine rewards or sanctions, how these indictors were determined, whether a growth factor was included, and also noted the various labels applied to schools or districts. We also examined what states are using to determine whether schools receiving Title I services are making adequate yearly progress (AYP).
We then investigated the status of the inclusion of students with disabilities in these systems. With the reauthorization of the Individuals with Disabilities Education Act in 1997 (IDEA 97), states and districts must include students with disabilities in their assessments. They are also required to report on the performance of students with disabilities, both aggregated with other students, and disaggregated. Although it is clear that students with disabilities are required to participate in statewide assessment systems, IDEA 97 did not specifically require that their scores be incorporated into system accountability. Because schools are likely to target resources toward those student groups that are included in the determination of rewards and consequences, a potential consequence of failing to include students with disabilities in accountability systems is that their instructional needs will not be met. Our goal, therefore, was to determine the extent to which students with disabilities are clearly included in accountability systems.
In 2000, we did a similar analysis (Krentz et al., 2000) and found that it was very difficult to locate evidence on state Web sites that clearly stated that students with disabilities were included in accountability systems. Although students with disabilities may have been included in assessments, their scores may not have been included in the accountability systems; most states did not specifically state whether students with disabilities were included when defining the indicators used for accountability purposes.
In 2001, the passage of the Elementary and Secondary Education Act (ESEA), also known as "No Child Left Behind," federally mandated accountability as one of its reform pillars. Annual testing is its cornerstone. While this legislation is in its early stages of being interpreted and implemented, it is apparent that nearly every state will be required to revise its accountability system for all students, including students with disabilities and low-income students receiving services through Title I. According to Wenning, Herdman, and Smith (2002), the required disaggregation of assessment data by various student groups as articulated in No Child Left Behind (NCLB) represents a significant improvement over past practices in which the use of schoolwide averages for accountability purposes frequently masked the performance of various student groups.
As a result of NCLB, many states that have developed accountability systems for schools receiving funds through Title I will now have to broaden their systems to include all students. States may simply include the additional students within their existing accountability systems, or may decide to undergo significant revamping of their current systems in order to comply with ESEA. In several states, the legislature was no longer in session by the time the details of ESEA implementation were shared with state departments of education. Many states have not yet begun to make decisions about how their current policies and procedures will need to change. State policymakers in many states will likely be faced with enacting new legislation to align their current accountability systems with the federal legislation.
In the current study, we additionally examined states’ Title I accountability systems and their measures of AYP that are required by federal legislation. When we started to collect information from states, we were interested in seeing how their state accountability systems were related to the Title I accountability system. As we collected data, not only were states in flux with respect to their accountability systems, but federal clarification through regulations was in process as well. In many states accountability criteria were undergoing revision, and in several others AYP was being altered in order to include all students, not just those of low socioeconomic status.
With changes occurring across the country, the data included in our report admittedly represent a snapshot in time of what states were currently doing or planning to do. Nevertheless, it is an important snapshot to take, as states proceed in their implementation of standards-based educational systems.
In order to obtain information on state accountability systems, we used the CCSSO Web site (http://www.ccsso.org) and selected "State Education Agencies" from the menu options. This provided a direct link to each state’s Web site, where we conducted a careful analysis of the accountability information available. If a state agency site had a search engine, we entered words such as "accountability," "assessment," "accreditation," "adequate yearly progress," "Title I," and a variety of other terms in an attempt to obtain as much information as possible.
After we gathered initial information from states’ Web sites, we emailed state accountability and Title I directors in December 2001, asking for help in locating pertinent accountability information on their state’s Web site. A second request was made in January 2002 to those who had not already responded.
The following criteria were used to determine whether an "accountability system" existed within each state: (1) schools or districts receive a state-determined rating that is tied to rewards, sanctions, or assistance from the state, and (2) the rating is based, at least in part, on student achievement outcomes. In order to be included in our analysis, it was not necessary for the accountability system to be currently in place; every effort was made to include information for systems that were being developed. In order to determine how students with disabilities are included in each system, we relied on information directly available on the state Web site and in publicly available documents provided by state department employees.
After summarizing the information available for these systems, a copy of the draft tables containing the pertinent data was emailed to the individual states in late May and early June, 2002. We sent a copy to both the accountability/assessment person and the Title I contact person. Changes in tables were made according to new information that states provided. Stringent criteria were used for any changes to be made to information we found on the inclusion of students with disabilities. Specifically, we required the state to provide the actual reference to their inclusion on the Web site or in a publicly available document. We required clear documentation that students with disabilities are included in accountability determinations, and not just that they are included in statewide assessments.
As was noted in the previous search of accountability systems (Krentz et al., 2000), the amount and depth of state accountability system information available on state education agency Web sites varied greatly from state to state. Some states had very specific information available about how accountability ratings were calculated; others had very limited information on the accountability system in general.
Status of Accountability System
Table 1 represents a snapshot in time of the status of states’ accountability systems. When creating Table 1, we looked online to determine whether a state had a state accountability system entirely in place, or in the process of implementation. In some situations, it was very difficult to determine the implementation status of a state’s accountability system. In those circumstances, we chose a state’s category of implementation status using the information we had available, realizing that our decision may not have been entirely accurate. For this table, we focused on the stage of implementation of an accountability system for all schools and students, and placed in a separate column the status of accountability for students and schools receiving Title I services, when this status was different from that for the overall accountability system. In general, we identified states at many different stages of implementation. Because accountability systems are constantly being modified, and because legislation and rulemaking at both the state and national levels is changing, every state could potentially end up in one of two columns – either "in development phase," or "currently in place, but being revised."
Table 1. Current Status of Accountability Systems
aA state department contact indicated that the new
accountability system will be drastically different from that of the past.
As well as being at various stages of implementation,
states also use various strategies to meet full implementation of their
accountability systems over a given period of time. Some states use a strategy
of phasing-in various grade level assessments to the accountability system to
over time. For example, Louisiana started testing in high school in grades 9 and
10, and then later added grade 11 testing. Similarly, Ohio plans to implement an
accountability system by phasing in tests for different grade levels at
different points in time, although their system is still in the process of being
Other states set a time frame for implementation, establishing a trial period before the testing became "high stakes." For example, Arkansas began data collection in the year 1998-99. Actual points for rewards and sanctions were not assigned until 2001-2002, and then only for certain indicators. Additional rewards and sanctions are scheduled to be phased in over time. Similarly, Vermont is phasing in accountability decisions, which are not scheduled to occur until the 2002-2003 school year.
States such as Delaware are in the process of phasing in various content areas for accountability purposes. This is consistent with the federal ESEA provision that requires states to begin assessing Reading and Math in grades 3 through 8 and add assessments in science in subsequent years. Finally, there are states such as Alabama that have created and implemented a system with a preliminary set of indicators, and are still in the process of determining which indicators will ultimately be included in the system.
States have also used a variety of strategies to meet the requirements of Title I and earlier ESEA legislation. Some states had very distinct Title I and statewide accountability systems. Still others incorporated the Title I system into their statewide accountability system. Many states had similar state accountability and Title I systems. In these cases, either the statewide accountability system provided the framework for the Title I system, with an additional labeling system for Title I schools, or a school or district could be identified as not meeting Title I AYP via the statewide accountability formula. Communication with state educational agencies indicated that many states are working on aligning pre-existing or new state accountability systems with Title I requirements. There are also some states that had Title I accountability systems in place, but were in the process of developing an accountability system for non-Title I students. Trying to determine whether states had two systems they were hoping to merge, or whether they were developing a new system to take the place of the old, or some other approach or combination of approaches was virtually impossible. Many states were in flux, and most were attempting to figure out whether what they were doing would meet federal requirements.
Several states have, or are developing, accreditation systems that address student achievement or improvement (Montana, North Dakota, Wyoming); however, student achievement is only a very small factor in determining the overall accreditation status. Therefore, these systems were not included for the purpose of this report. It appears that Utah and New Hampshire may be developing accountability systems for all students in the future, but extremely limited information of this was available. Iowa has a system that is administered primarily by area agencies, and not by the state. Finally, Maine, Minnesota, and Nebraska also do not appear to have or be developing accountability systems that meet the criteria we used. However, when relevant information was available on these states’ Title I accountability systems, this information was described, with footnotes explaining that the information provided pertains to the state’s Title I accountability system.
Table 2 displays the indicators used to determine school or school district labels and consequences. The indicators represented in this table are those that are actually counted in the state’s accountability measurement. Many states have "report card" indicators that are used to provide information to parents, policymakers, and the community about particular characteristics of a school or district. Unless indicators were used for more than simply "information," we did not include them in this table.
Some states have both a rating system and an accreditation system that incorporates multiple indicators in addition to student test scores. For these states, we identified indicators only for the primary system in Table 2, but provided corresponding footnotes to explain indicators used for the additional system. For instance, Colorado primarily uses an accreditation system to hold schools accountable, although there is an "academic performance rating" system that takes into consideration a subset of the indicators used for the accreditation system. Several states (including California, New York, North Carolina, South Carolina, Texas, and Virginia) have different accountability indicators used for special schools (e.g., alternative, career/technology, or special education schools), or allow these schools to choose additional indicators for accountability purposes.
Table 2 includes both Title I AYP and general accountability indicators; indicators used solely for Title I AYP determination are noted in the table. Overall, student assessment scores are the indicators used most often in the determination of school consequences. In addition, they are frequently the most heavily weighted components of school accountability formulas. For many states, both test performance and achievement growth as measured by tests play a significant role in school accountability, although the extent to which achievement growth is weighted in the formula varies from state to state. Attendance and dropout rate are the next most commonly used indicators (20 states); assessment participation rate, graduation rate, and indicators of school safety are also frequently included in measures of school accountability. Participation rate is included as an indicator in a variety of ways. In several states (Florida, North Carolina, and Pennsylvania), a school must test a certain percentage of students in order to be eligible for rewards. In other states, students who did not participate are automatically given a score of "0". Maryland uses a unique method to ensure that participation rate is appropriately accounted for as an indicator in the accountability system. In this system, a school’s participation rate can fluctuate over time; however, the average "Level Not Determined" (LND) cannot be higher than 10% across five years, and in no single year may there be more than 14% in the LND category. Finally, in the most recent year the LND category must contain 6% or less of students.
In some state systems, review of assessment scores is the first step to determining labels or consequences for schools or districts. If schools fail to meet necessary achievement levels, then other indicators are additionally reviewed. In other states, cut-off levels for each indicator must be met for a school to receive the highest ranking. Still other states maintain a point system for determining school labels; schools receive a set amount of points for each indicator, and ratings are then determined based on total points for the school or district.
In some state accountability formulas, special attention is placed on particular student types. For instance, Alaska is considering weighting low-performing students’ scores twice as much as average-to-high performing students’ scores in the accountability formula. Many other states have formulas that specifically target low-performing students, minority students, or other students considered to be at-risk. Student mobility is an issue that is addressed in different ways across state accountability systems. In some states it is clearly indicated that a school is not responsible for the scores of students who enter the school after a given point in the school year.
Table 2. Indicators Used to Determine School/District Labels and Consequences
aOne of the goals is defined
as: “At least 99% of secondary students will remain in school to complete the 12th
Accountability Growth Indicators
Table 3 reflects the types of growth indicators included in accountability systems. In many states, a growth component is added only if a school or system does not meet a set criterion level. In other states, all schools, regardless of achievement level, need to demonstrate progress. Several states require incremental growth in achievement scores such that all schools reach a goal of having a target percent of students scoring in the proficient range by a particular year. Under these circumstances, lower achieving schools are required to make greater progress than higher achieving schools.
Of the three categories of growth indicators represented on this chart, school achievement growth is the mostly widely used. In this type, a particular school, grade level, or subgroup’s score is compared from one year to the score for the next year’s subgroup in order to determine "achievement growth." This is the easiest score to determine, but does not take into consideration factors such as changes in the school population, and the challenge of drawing comparison between distinct cohorts of students. This type of growth measure is required by Title I.
A second form of growth examines cohort achievement. In this case, "growth" is determined on a longitudinal basis, looking at how a specific group of students performed over time. It is sometimes referred to as a "value-added" approach. Similarly, an individual achievement growth component breaks down the data to an individual student level, and follows individual student performance growth over time.
In some states, the school achievement growth factor is used to determine Title I AYP, but may not yet be determined for the non-Title I population. In other states, it has been decided that they are going to include a growth component, but the specific type of growth has not been determined at this time. Several states include multiple types of growth in their accountability systems.
Table 3. Types of Growth Indicators
Table 4 displays information on how states determine AYP. At the point that we collected information, many states made distinctions between their Title I AYP and other accountability systems. In many cases, information was only available on former definitions of AYP. Current legislation has caused many states to begin revising their definitions of AYP; therefore, we dated the information in order to clarify when the definition described was applicable.
In some states, AYP was a requirement for all low-performing schools. In other states, AYP was only a requirement for those schools and districts receiving Title I funds. Some states had synonymous state and Title I accountability systems in which AYP was demonstrated by the same indicators as those used for the state accountability system. In other cases, the Title I AYP system was distinct from the other accountability system.
The definition of "AYP" differed from state to state. In some states AYP is met when schools demonstrate that a certain percentage of students with disabilities meet proficiency requirements on a state test. In other states, AYP is not required among schools that meet these proficiency requirements, and only among those that do not meet the threshold requirements.
Most states require an increase in the percentage of students scoring within a proficient range on a statewide test in order to meet AYP requirements. In some definitions, all schools not meeting the threshold percentage must each make the same percentage improvement on a yearly basis. In other states, the required percentage improvement depends on the current status of the school. Formulas have been developed to determine the amount of progress necessary for the school to meet a target set for the future. The required AYP, then, depends on the current distance from the target. Several states use a multiple-year average to determine the baseline for a given year, in order to account for year-to-year fluctuations in the student population.
Several states use a school index to measure AYP. The index is a numerical value assigned to a school that represents a calculation based on student performance, and in many situations includes additional indicators such as dropout rate and attendance. The percent of students
Table 4. State Adequate Yearly Progress (AYP)
bThe words "Adequate Yearly Progress" were not used; however, it was assumed that AYP will coincide with the state accountability system for all schools, and so state accountability information was coded in this table.
cAlthough "AYP" words were not used, this information was verified by state department contact.
dThis information was provided in 2001/2002 by a state department contact.
eExact date was not identified on documents, but the connection between Title I AYP and accountability system is apparent on a document that describes the process used for FY2003.
fSchools that meet the requirement of having an average scaled score of 1420 do not need to show improvement.
gAnnual growth expectations were not defined.
hSchools must meet the growth standard, which is described in the table. This does not exactly represent "growth in school index" as described in the text, but represents something very similar to "growth in school index."
iLimited information was identified on AYP determination. It appears that the school median percentile score, and increase in this measurement are used to determine AYP.
Sources of Indicators
In a majority of states, the indicators used for accountability are determined at the state level. However, there are some states that require local school districts to determine some of their own indicators, and therefore use a mixture of state and local indicators (see Table 5). If states only use local indicators, and do not identify which indicators are required, they are not included in the information represented in Table 5.
States use a variety of labels to describe school or district status. There are schools and school districts across this country that have been determined to be "distinguished," "successful," "excelling," "superior," "commendable," and "exemplary." There are also schools and districts that have been labeled "deficient," "under performing," "failing," "marginal," and "academic alert." Table 6 represents the broad array of labels given to schools and districts across the country. Some states have three levels of performance; others have four or five levels. Some states have an index score, and some use absolute grades or progress grades. Others are still determining the labels they will eventually attach to their schools.
Many states have either separate or additional labels applied to Title I schools. These labels are found in Table 7. Again, states use a variety of labels to indicate the status of Title I schools.
Table 5. State vs. School/District Determined Indicators
aLocal assessments may be added as indicators in the future.
bCurrent Title I AYP allows use of local measures.
cSchools can determine specific tests to be used at particular grade levels
dThis represents Title I AYP determination.
eIt appears that EPSS goals may not be determined by the state.
fLocal assessments can be used for k-2 AYP
gLimited information was available on Utah’s AYP system; it is assumed that this is state determined.
hThe district can cite other evidence of continuous progress if the department finds that it is not meeting improvement requirements. The state may then decide to withdraw the identification of not meeting improvement.
Table 6. Labels Given to Schools/Districts
aAcademic Performance Ratings
are also determined based on test scores alone. These labels are as follows:
Excellent, High, Average, Low, Unsatisfactory
Table 7. Additional Labels for Title I Schools
Inclusion of Students with Disabilities
One purpose of this study was to determine the inclusion or exclusion of students with disabilities in school or district accountability systems. Federal law under IDEA requires this inclusion for assessment, and the newly authorized No Child Left Behind Act requires inclusion of all students in accountability systems. After passage of federal legislation requiring the inclusion of all students in accountability systems, several states have undergone revision of their procedures, and many others will be undergoing revisions.
There are a variety of ways that students with disabilities can participate in statewide assessment systems, and these can affect their inclusion in accountability. Some students with disabilities are included in the regular test administration in the same way as most other students. Others participate with accommodations. Some take modified tests or receive non-standard test administrations, and a small percent participate in alternate assessments. Although state policy may clearly indicate the ways in which students with disabilities can participate and be included in state assessment systems, the degree to which scores from these various test administrations are included in the determination of school accountability is often unclear.
Table 8 represents the extent to which states clearly articulate which scores for students with disabilities count in the determination of school labels and consequences. We used very strict criteria in determining whether there was documented inclusion of students with disabilities in accountability systems. In some cases, the information we obtained indicated that "all students" were included, but we could not find evidence of how students with disabilities were included. In the past, we have found that state assessment performance results reported as being for "all students" have not necessarily represented all students with disabilities. Therefore, we considered such statements unclear. Information for this table was obtained by searching the Web sites of the state educational agencies, and from publicly available documents provided by state department contacts. In addition, the table was verified with state departments, who could provide additional documentation.
Altogether, we identified 27 states with publicly available documentation that at least some students with disabilities are or will be included in the determination of accountability labels or consequences. Inclusion of students with disabilities was articulated in a variety of ways. Some state documents explained exactly which scores are or will be included. Others simply specified that all scores for students with disabilities count in the determination of accountability. For only four states (Delaware, Kentucky, South Carolina, and Vermont) was it clear that all students with disabilities, taking either regular, accommodated, modified, or alternate assessments, are to be included in the accountability system (either currently or in the future). Language from these states’ accountability policies is provided in Appendix A.
Table 8. Documentation of Inclusion of Students with Disabilities in Accountability Systems
O=this is not documented as being included; however, given that accommodated and/or modified tests are included, it was assumed that these administrations are included.
*See Table 9.
**See Table 10.
aThis represents information from Title I accountability.
bSome accommodated scores are included; some are not.
cPerformance ratings do not include alternate assessments.
dAlternate assessment is coded as "Level Not Determined" and entered as such in the accreditation system.
eSome administrations are only included in participation component. Performance for these administrations will be added in the future.
fAlternate assessments will be included in district accountability, but not school level accountability.
gWill be included in ratings in the future.
hAlternate assessments, which include modified assessments, are included in participation component, and will be phased into performance component.
iGiven separate code for accountability purposes, which suggests that only participation in alternate is included in school accountability.
jThis represents information about system accountability. It is therefore assumed that this also refers to Title I accountability, which was the only one of WY’s systems that met our requirements for inclusion (accreditation system did not focus on student performance results).
kThe number outside of parentheses represents the number of states that only fall only in this inclusion/exclusion category, and no other categories. The number in parentheses represents the number of states overall that were included in this category.
We identified 15 states that have clearly articulated in publicly available documents that scores for students receiving accommodations are included in the state accountability system. In some cases, it was clearly articulated that scores for both regular and accommodated test administrations for students with disabilities are or will be included in the accountability system. We assumed that if there was documentation of the inclusion of accommodated scores, results obtained from non-accommodated testing of students with disabilities were also included. For two states (California and Maryland), only those scores obtained with accommodations determined to maintain the validity of the test are included. Other accommodations are not deemed valid, and those scores are excluded.
Modified assessments were less frequently documented as being included in accountability determination. Only six states explicitly stated that modified tests are included, and five states indicated that these scores are excluded from the accountability system.
Eleven states indicated that alternate assessments are or will be included in the determination of accountability. It appears that four states only count the alternate assessment toward the participation component of the accountability system, and do not include alternate assessment performance, although several indicate that performance results on the alternate assessment will be phased into the accountability system over time. Others simply stated that alternate assessments are not included in the accountability system.
Many states did not explicitly state which test administrations are or will be included, but did articulate that students with disabilities are included. These statements are described in Table 9. In some of these situations, statements indicate that growth in the scores for students with disabilities is necessary for accountability purposes. Other policies state that all scores for students with disabilities are included in the accountability determination, but do not necessarily state which test administrations this includes. In one state, it appears that only certain students with disabilities are included, namely those with speech and language disabilities.
Table 9 also indicates other information about the exclusion of students with disabilities from accountability. Students with disabilities in New Jersey and Louisiana, for example, have been excluded in the past, but current documentation or communication with state contact persons indicates that these states are moving toward greater inclusion. Table 9 also contains data from state Web sites that did not meet our criteria for documented inclusion/exclusion.
The amount of searching required to identify how students with disabilities are or will be included in state accountability systems varied greatly from state to state. In Illinois, it is very clear that students with disabilities are included. On the front of the Illinois student assessment Web page, a statement reads "Scores for all students, including students with disabilities and limited English proficiency, are included in the determination of AYP for school accountability." In other states, the information is less easily accessible, and extensive Internet searching was necessary to identify whether documentation was available.
In some situations, documents did not clearly articulate whether students with disabilities are included in the accountability system. In those situations we relied on communication with the state contact persons to obtain the relevant information. This communication is provided in Table 10. Altogether, publicly available documents and communication with state contacts suggests that states are moving toward greater inclusion of students with disabilities in accountability systems.
Table 9. Other Documentation of Inclusion of Students with Disabilities in Accountability Systems
Table 10. Communication with State Education Agency about Inclusion of Students with Disabilities in Accountability Systems
The purpose of this report was to identify and describe various characteristics of state accountability systems that provide rewards, sanctions, or assistance to districts or schools on the basis of student achievement. One important feature of these systems that we were interested in was the degree to which students with disabilities are clearly included in the accountability determination.
Based on examination of state educational agency Web sites, as well as communication with accountability and Title I contact persons at state educational agencies, we obtained information on the current status, relevant indicators, growth components, Title I AYP definitions, and the inclusion of students with disabilities in state accountability and Title I accountability systems. States are at very different stages of school/district accountability system implementation. Because state accountability policies are frequently changing, especially as a result of recent federal legislation, the information provided in this report represents only what can be considered a "snapshot in time" of what states are doing or planning to do in the near future.
In addition to the difficulties associated with studying a "moving target," it was also often difficult to determine exactly which indicators truly counted in the determination of school accountability labels and consequences. As indicated in a previous report (Krentz, Thurlow, & Callendar, 2000), many states have a variety of indicators that are used for reporting purposes that may or may not be used for accountability purposes, as defined in this report. In some situations, consequences for schools were based primarily on one or two indicators, with the option for examination of other indicators if schools did not meet the initial requirements. Based on our analysis, student assessment scores and assessment score growth are the most commonly used indicators in these accountability systems. Other influential indicators include attendance, dropout rates, graduation rates, participation rates, and indicators of school safety. When determining Title I AYP status, many states primarily examine an increase in percent of students meeting proficient levels on state achievement tests, with several incorporating this calculation in a school index that may or may not include additional indicators.
States have incorporated a growth component into their accountability systems in a variety of ways. Many states include both a status and growth component such that lower performing schools must show more progress than higher achieving schools. In this way, states intend to help "close the achievement gap." However, growth is measured in a variety of different ways across states. Because comparing different cohorts of students across time can increase the volatility of school results (Linn, Baker, & Betebenner, 2002), it is often argued that schools should take this into account when they are making accountability decisions for schools. Several states are now incorporating a system that either tracks individual student progress over time, uses a rolling average of school scores, or allows schools to miss progress requirements on a small fraction of their indicators and still meet progress requirements for a given year. All of these options have been considered improvements (e.g., result in less volatile results) over other measures of school progress (Linn et al., 2002).
The degree to which students with disabilities are included in these accountability systems is also not always clear. In many situations, it may be the case that students with disabilities are included in the accountability system, but the information available in publicly available documents does not clearly articulate that these students are included. In a survey of state directors of special education, 15 respondents indicated that all students with disabilities were included in all components of the accountability system (Thompson & Thurlow, 2001). This represents information contrary to our findings, and is likely due to several differences in how information was collected in these two studies. For instance, "accountability system" was more stringently defined in the current study. Also, the current study relied on publicly available information linking the participation of students with disabilities to accountability calculations, whereas the earlier study simply required communication by the respondents.
In general, it is important for states to make clear who is included in determining accountability rewards, sanctions, and assistance. Otherwise, the public cannot adequately understand what it means for a particular school to be "superior," "adequate," or "underperforming." Furthermore, unclear articulation of which students are actually included in the accountability system undermines the promotion of high standards for all students. Ambiguity with regard to whether students with disabilities are included leaves one questioning whether they may actually be excluded from school and district accountability, as determined by the state.
In comparison to a similar previous analysis of state accountability systems (Krentz et al., 2000), it appears that there has been a decrease in the number of states that indicate students with disabilities are excluded from accountability systems. However, the number of states indicating that all students with disabilities truly count in the determination of school consequences remains quite low. Many state policies remain unclear in terms of which students with disabilities are included in accountability determination. It was also apparent from the current analysis that special schools, often serving students with special needs, are frequently held to different requirements for accountability purposes. It will be important to track these alternative accountability systems in order to ensure that they are not alienating students from the promotion of high standards present in other schools.
With many changes in elected representation expected in state houses, gubernatorial elections, as well as at the national level due to the 2002 elections, it will be difficult to predict the status of accountability legislation just a few short months in the future. It is likely that much of what is recorded here will change within the coming months. As states continue to revise and develop their accountability systems, it is anticipated that they will continue to provide the public with detailed information regarding the components of their accountability systems. In addition, it is anticipated that in revising state accountability policies to meet new federal requirements, states will move toward more appropriate inclusion of all students in accountability systems for schools and districts, ensuring that all students, regardless of disability status, are provided a learning environment with high expectations.
Center on Education Policy (2002). State high school exit exams: A baseline report. Washington, DC: Author.
Guy, B., Shin, H., Lee, S. Y., & Thurlow M. L. (1999). State graduation requirements for students with and without disabilities (Technical Report No. 24). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
Johnson, D. R., & Thurlow, M. L. (2002). National study on high school graduation requirements and diploma options for youth with disabilities. Minneapolis, MN: University of Minnesota, Institute on Community Integration.
Krentz, J., Thurlow, M., & Callender, S. (2000). Accountability systems and counting students with disabilities (Technical Report 29). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
Linn, R. (2000). Assessments and accountability. Educational Researcher, 29 (2), 4-14.
Linn, R.L., Baker, E.L., & Betebenner, D.W. (2002). Accountability systems: Implications of requirements of the No Child Left Behind Act of 2001. Educational Researcher, 31 (6), 3-16.
Quenemoen, R. F., Lehr, C. A., Thurlow, M. L., Thompson, S. J., & Bolt, S. (2000). Social promotion and students with disabilities: Issues and challenges in developing state policies (Synthesis Report No. 34). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
Thompson, S., & Thurlow, M. (2001). 2001 State special education outcomes: A report on state activities at the beginning of a new decade. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
Thurlow, M.L. (2000). Standards-based reform and students with disabilities: Reflections on a decade of change. Focus on Exceptional Children, 33(3), 1-16.
Wenning, R.J., Herdman, P.A., & Smith, N. (2002). No Child Left Behind: Who is included in new federal accountability requirements? Paper commissioned for a conference sponsored by the Thomas B. Fordham Foundation (February, 2002). Retrieved on September 12, 2002 from http://www.edexcellence.net/NCLBconference/Wenning%20Herdman%20and %20Smith.pdf
Appendix A: Inclusion Policies for Students with Disabilities in Accountability Systems among Four States Representing Full Inclusion Practices
"Except for students who participate in out-of-level testing, students who test with non-aggregable conditions as defined in the Department of Education’s Guidelines for the Inclusion of Students with Disabilities and Students with Limited English Proficiency shall have her/his performance level included in the School Composite Score."
"For school accountability purposes, a student who tests but does not meet attemptedness rules as defined in the Department of Education’s Scoring Specifications, who participates in out-of-level testing or otherwise receives an invalid score shall be assigned to Performance Level 1."
"A student participating in alternate assessments shall have her/his performance level included in the School Composite Score."
"Inclusion of Special Populations in the State-Required Assessment and Accountability Programs (703 KAR 5:070)" provides a detailed description of how students with disabilities participate in assessment and accountability systems.
"Scores from alternate portfolios shall be included in the academic indices so that the data from an alternate portfolio completed by a student eligible to participate with an alternate portfolio contributes the same weight to the academic component of the accountability index as would the data for a student participating in the regular components of the assessment program at elementary, middle, or high school levels."
"The intent of the Kentucky General Assembly is that all students be included in the assessment and accountability components of KRS 158.645-KRS 158.6455. The only exceptions are those few students who cannot complete either the regular or alternate assessment components even with allowable accommodations, modifications or both due to medical or mental health conditions."
"1. Students with accommodated administrations will be treated identically to students taking PACT in its standard form in absolute school and district ratings; . . .3. Students taking modified assessments, including "off-level tests", will be factored into the absolute rating according to the test score earned."
"Following their third administration, student assessment results from the PACT-Alternate assessment will be included in the calculation of the district but not the school ratings."
"Some students will participate in the accountability system using one of three alternate assessment options (e.g., Modified Assessments, Adapted Assessments, or Lifeskills Assessments). Results from assessments using approved accommodations will be included in each school’s accountability index using the same point values that apply to standard administrations. Alternate assessment results will be entered into the accountability index using pre-determined score transformation rules. Invalid assessments will be designated as demonstrating ‘little or no evidence of achievement’ and will be assigned an accountability index value of zero."