Accuracy (quantitative, number correct)

Only one of the sample states, Oregon, specifically includes accuracy as a criterion. For the Oregon extended assessments in reading, mathematics, and writing, the definition of accuracy is quantitative, and is reflected in number correct (or partially correct), reading rate, and reading accuracy. However, one could argue that all of the sample states had same form of accuracy criteria, including Oregon on its Extended CLRAS, with accuracy defined qualitatively using measures such as independent performance of skills, generalization of skills to multiple settings or for different purposes, and appropriateness criterion such as challenge or complexity.

Content Standards Performance (student performance on standards-based skills or tasks provided by the state)

Louisiana and Oregon have provided items or state specified knowledge and skills that are aligned to state content standards. Oregon’s extended assessments in reading, mathematics, and writing are aligned to the math, reading, and writing content; the Extended CLRAS is aligned to the career related learning standards content, but not to math, reading, and writing. The other states score the relationship to standards as a system measure, designed or chosen by the teacher.

Level of assistance (prompts, showing degree of independence)

All the states except for Kentucky look at independence using level of prompting as a student measure. They may allow or encourage appropriate natural supports, adaptations, modifications, and assistive technology as tools to assist in increasing independence. Oregon scores for level of assistance only on the Extended CLRAS, and not on the extended assessments in reading, mathematics, and writing. Kentucky views level of support as a system criterion, encouraging natural supports, accommodations, and assistive technology as provided to the student to increase independence.

Multiple purposes (generalization)

Louisiana looks at both multiple settings and multiple purposes to measure student generalization.

Mastery is one indication of increased quality on the Arkansas "performance" criterion; Vermont defines mastery as ability to perform a skill independently.

Performance (qualitative)

Arkansas and Kentucky each define broadly the term "performance," but with very different definitions. The term is almost generic, and the varying definitions probably reflect the fact that the term "performance" can mean whatever a given set of stakeholders defines as the essential outcomes of instruction. Thus, it may be appropriate to suggest that all of the scoring criteria used by states attempt to define "performance" for this population of students.

Both Kentucky and Vermont link progress to IEP goals; Kentucky also mentions progress in relationship to independence. Oregon’s approach is very different, using a growth model for evaluation for the whole assessment system.

Settings (student performance evidence of generalization)

All five states include some form of a generalization criterion, as evidenced by settings. Oregon uses this only in the Extended CLRAS and not in the extended assessments in reading, mathematics, and writing. Kentucky uses this as a systems criterion, measuring whether the system is providing opportunities for instruction, practice, application, or performance of the specified skills in multiple settings.

System criteria used across the five sample states are shown in Table 16. System scoring criteria include measures of system performance, such as whether students are provided instruction in multiple settings, whether they are provided opportunities to plan, monitor, and evaluate their work, whether they work with non-disabled peers, and whether they are provided with appropriate human and technological supports. As previously discussed, the distinction between student and system criteria is nebulous. Our sample states addressed this directly or indirectly as part of the interview process, and their comments follow.

Table 16. System Criteria used Across the Five Sample States

* Assumed in design of training materials, assessment format, or in criteria.

Arkansas stakeholders decided to weight "performance" at the highest level since they believed the emphasis should be on measuring actual student evidence, and not the system. However, determining how to measure quality of programs for these students without looking at appropriateness, level of assistance, or settings is difficult, and their criteria and scoring process allow for that interplay.

For Kentucky, the issues of system measurement were central to decisions about scoring criteria, and about how Kentucky’s approach has changed over the last decade. Kentucky believes that for this population of students, looking at how students are taught is really important. Assessment may show that students are not achieving well, but if students are not being taught well they cannot be blamed for not learning. Kentucky includes multiple systems criteria in order to "shine a light" on teaching practices, practices that in the past too often took place in isolated classrooms behind closed doors.

In contrast, the Louisiana task force considered adding questions to the assessment that did not measure what a student was doing, but measured what supports were available, and the amount of time students spent in inclusive settings. A decision was made to view the alternate as an assessment of a student’s performance. The task force decided not to include system measures because it believed that those measures were not included in the general statewide assessment. Some task force members wanted to include those measures as one way to obtain valuable information, but the determination was made that there were other ways to get that information (e.g., monitoring). They decided to separate monitoring from evaluation, even though the data would be used to look at programs. They wanted to see specifically what was happening with these children, thus focused their alternate assessment on student criteria.

Oregon discussed concepts related to the underlying learning theory that drives its assessment system. Oregon personnel believe their assessments are grounded in learning theory in their focus on student criteria, not systems criteria. They focus training on how the alternate assessment relates to instruction and planning, and how teachers might use these measures on their own periodically throughout the year to assess progress. Training personnel stress the importance of using the alternate assessment as an integral part of the instructional process, and not as an add-on task. That is the intent of Oregon’s standard assessments; they are to change the way teachers teach.

Vermont’s stakeholder group started, and then abandoned, a process of designing its learning outcomes specifically for the alternate assessment. They were committed to using validated learning outcomes, and were reluctant to, as they state, "reinvent the wheel." Instead, they elected to adapt existing materials that were already validated. The authors of the COACH (Choosing Outcomes and Accommodations for Children: A Guide to Educational Planning for Students with Disabilities, Giangreco, Cloninger, & Iverson, 1998) from the University of Vermont allowed the group to use their learning outcomes as the criteria for alternate assessment. They are not using the COACH as the alternate assessment; instead, Vermont "borrowed" the learning outcomes and measures student progress against those learning outcomes. One of the COACH authors worked with state staff to cross-link the selected outcomes to Vermont content standards. The stakeholder group decided to ask teachers to show that students had made progress across three of the learning outcomes, but they could pick from anywhere in the developmental spectrum to match the students’ needs. As part of their system criteria, Vermont scores the linkage of the evidence to Vermont standards that are cross-linked to the COACH outcomes, as the learning outcomes are referenced and assessed in the IEP.

States that included each system criterion, along with general definitions are found in Table 16. State by state definitions or justifications for assumed or embedded student criteria are in Appendix C.

Appropriateness (age, challenge, authenticity, meaningfulness)

Each of the states has struggled with determining "how high is high enough" to raise expectations and outcomes for this population of students, whether scored as part of the rubric, or embedded in test items or state specified skills. Age appropriateness is a common shared criterion, but level of challenge, authenticity of tasks, and meaningfulness for current and future life roles are also included.

Content Standards Linkage (teacher developed skills and tasks)

Arkansas, Kentucky, and Vermont score on degree of linkage to academic content standards through teacher developed or identified skills or tasks. Two of the states, Louisiana and Oregon, instead provide state developed tasks or specified skills to ensure standards linkages, and do not score for them.

Only Vermont scores the nature of linkages to the IEP. Kentucky references IEP goals in its progress definition. Yet all five states emphasize the challenging training issues that are emerging as teachers and other IEP team members, including parents, make the shift to standards-based thinking and planning for IEP goals and objectives. Thus, in the four other states we have marked this criterion as "assumed."

Self-determination (evidence of opportunities for student choice)

Kentucky defines "self-determination" as evidence of degree of student choice with an impact on student learning across and within entries, and evidence of planning, monitoring, and evaluating of the self-performance to improve or set new goals. It is the only state to score for this, noting that there is a significant body of research, including a recent meta-analysis of research (Algozzine, Browder, Karvonen, Test, & Wood, 2001) to document self-determination as a teachable, essential skill for future success for this population of students. However, Vermont state staff point out that although they do not have specific scoring criterion on self-determination, the ability to make choices is embedded throughout their learning outcomes. For example, one of the communication learning outcomes is, "Uses communication to indicate preferences and make choices."

Settings (system provided)

Arkansas scores settings across all content entries, and gives it the lowest weight of all its criteria. Kentucky sees this as an important measure of system quality, giving students opportunities to generalize knowledge and skills. Kentucky links this criterion to that of "Social Relationships," requiring that settings include age appropriate peers.

Social relationships (system provided opportunities)

Kentucky scores social relationships on appropriateness of interaction, frequency, diversity of range in peers, sustainability, and reciprocity of relationships.

Support (access to system provided appropriate supports and technology)

Kentucky defines support as degree of peer support, tutoring, or natural supports; use of adaptations, modifications, or assistive technology that demonstrates progress toward independence.

On surface examination, the scoring criteria used by the five selected states appear to be different. State responses to the 2001 survey of state directors of special education (Thompson & Thurlow, 2001) suggested significant differences as well. Yet when the scoring elaborations and processes are examined closely, many similarities emerge. After careful analysis of how some assumptions are built in to the instrument development or training processes, even more similarities emerge. The definitions and examples and the side by side examination of the criteria, the scoring elaborations, and the assumed criteria in the design of training materials and assessment format yield a surprising degree of commonality in the way these states define success for students with significant cognitive disabilities. These commonalities are highlighted in Table 17.

Common Criteria Across the Five Sample States

Six criteria are included in all of the five states’ approaches in some way, either articulated or assumed. They include "content standards linkage," "independence," "generalization," "appropriateness," "IEP linkage," and "performance." How the states address these criteria varies however, as shown in Table 18.

These six criteria have emerged from a long history of practice, and each is reinforced by research on teaching and learning for students with significant cognitive disabilities. Questions remain about how best to incorporate these commonly accepted indicators of success for these students. Here are some questions about each of the criteria, followed by a discussion of possible options and responses, based in part by discussions with selected alternate assessment researchers and training leaders (NCEO Alternate Assessment Retreat, 2002).

1. Content Standards Linkage. Is it better to have standardized tasks or common items with linkage to standards provided by the state, or to allow the flexibility and the variability of teacher defined linkages to the content standards?
Content standards linkages are an essential part of alternate assessments if they are going to meet federal assessment and accountability requirements. It appears from our sample states that either state-provided or teacher-developed linkages to content standards can work, but the training issues related to teacher-defined linkages are very great. The shift to standards-based teaching and learning has been difficult for all teachers, and although training on content standards has occurred in every state, teachers who provide special education services and supports often have been overlooked in the training. Teachers will not be able to align instruction or assessment to content areas without training or access to research-based models. The standardization of tasks or items eliminates the need for teachers to have significant amounts of training for the assessment part of the puzzle. That is because the standards linkage can be embedded in the assessment item or task, and is not dependent on teacher understanding and skill. The standardization of assessment tasks does not address the need for teacher and IEP team member training and support so that instruction becomes standards-based, providing students with opportunities to be successful on the standards-based assessment.

Some states that have selected teacher-defined linkages have done so because of the extreme variability in the student population participating in alternate assessment. They believe that even with the training challenges posed by teacher-defined linkages, flexibility of task and participation level are important characteristics of the alternate assessment. One state interviewee suggested that teacher-defined linkages allow for greater flexibility in documenting a larger number of content standards through evidence of the interdisciplinary use of standards-based skills. For example, student performance on a math standard of computation also might show number sense, data, or probability skills as well as reading and writing and may occur within other content area standards such as science or social studies. Another option that is being discussed in some of these states is to go to "flexible structuring" of performance tasks. For example, the Louisiana approach defines state-specified knowledge and skills (target indicators), but teachers are allowed to design the actual activity in which to demonstrate the knowledge and skills, and the IEP team can choose the one of three "participation levels," reflecting three levels of difficulty, that is most appropriate for the student.

2. Independence. We have a long history of best practice and a solid research base to support the emphasis on independence for students with significant cognitive disabilities. Is "level of prompting" the best way to judge independence? How do the system-provided opportunities for accommodations, assistive technology, and natural supports fit into the measurement of whether students are maximizing their independence?
There has been a focus on prompting strategies in preservice education for teachers of students with significant disabilities. Perhaps as a result, Oregon state staff reported that many teachers were unaware of the level of prompting they were using, and as the Extended CLRAS was implemented, many teachers reported difficulty in removing prompts, since they automatically provided them without thinking. Other states also have reported overuse of teacher "automatic" prompting. Some states have found that they had to define differences between what an accommodation, assistive technology, or natural support is versus what a prompt is. The dilemma is that increased use of accommodations, assistive technology, or natural supports can increase independence; increased use of prompting will decrease independence. Clarifying the distinction is important for instruction and for assessment, and scoring criteria and processes need to clearly differentiate among them as well.

3. Generalization. Is offering rich and varied settings in which students learn enough to indicate program quality, or must we see the student performing skills to a specified degree of success in those settings? Are different purposes the same as different settings in the ability to show generalization?
This criterion is more complex than it appears at first glance. There is an argument that taking the same activity and moving it to some other "place" is not necessarily generalization. This approach may work better for life skills, where the focus has been on generalization of functional life skills across natural settings, than for academics. When considering generalization for academic knowledge and skills the focus may need to be on different tasks or materials on which the knowledge and skills, are used. For example, if the student can understand the main idea of a story, can he or she also do that with a picture or a news story? Considering instructional contexts – media and materials, formats of materials, response methods or modes – may provide a more important context for generalization of academic skills than physical locations. The specific focus of generalization may vary by type of disability. For example, for students with autism, the focus may need to be on getting behavior to generalize across different people or to different settings.

State staff from one of the sample states that has a clear commitment to settings as a system measure, suggests that adhering to specific skills applied in academic contexts overlooks the value of the shared culture of participating in multiple settings where all other same age peers are participating. In another of the sample states, state staff commented that inclusive settings are encouraged by scoring for multiple settings. Thus, there appears to be another purpose to including "settings" as a system criterion, that of promoting inclusive practices. Clarifying these distinctions and ensuring that teachers understand them will be important if the policies are to drive desired improvements. A few states, although none of the sample states, score and report on system opportunities for multiple settings, but do not include the scores in accountability calculations. They use the scores to provide feedback to teachers, schools, and districts, and to guide monitoring decisions, but they do not include the scores in system accountability indices.

4. Appropriateness. If you allow teacher/IEP team choice on the tasks used or the skills to be targeted, how do you determine the appropriate level of challenge, authenticity, or age appropriateness? How do you know if a high challenge is high enough?
Appropriateness is either stated or assumed across all states, reflecting another value of special education since the early 1980s. Since that time there has been an effort to move away from the "developmental or mental age" approach (even though still used in practice) to a focus on what is appropriate for a student’s chronological age. Most states have avoided defining alternate assessments as measurement of a developmental sequence, for example, by rejecting infant or preschool definitions of knowledge and skills as an appropriate linkage to content standards. The value of age-appropriateness is emphasized by references to "grade level standards" in No Child Left Behind, and to the general curriculum in IDEA.

"Appropriateness" as a criterion is typically not just about age. Raising expectations was a centerpiece of the 1997 reauthorization of IDEA, and "challenge" as a sub criterion is a common definition of "appropriateness." The appropriate degree of challenge is difficult to score for a widely differing group of students, although some states are turning to baseline measurements such as pre and post tests to do so. In contrast, the teacher developed "criteria for mastery" normally included in IEPs does not appear to be effective in setting high expectations. "Authenticity" is another common indicator of appropriateness. For example, knowledge and skills instruction and assessment embedded in grade level content or contexts is considered to reflect authenticity. Age appropriateness is easier to see and score – and certainly has been a high value in the push to move away from developmental models that prevailed years ago. However, are the values of appropriateness or level of challenge or authentic learning experiences worth striving toward given the difficulty in scoring these? Clearly many of the states in our sample believe these values are worth the difficulty in scoring, but only Kentucky and Arkansas score these values directly. Louisiana, Oregon, and Vermont provide guidance to teams on choices of participation levels (Louisiana), whether the student takes any of the extended assessments in reading, mathematics, and writing or not as well as the specific routines and embedded skills in the Extended CLRAS (Oregon), and IEP goals and objectives (Vermont), but they do not score on the choice made.

5. IEP Linkage. Each of the sample states expressed concerns about the quality of IEPs, and each has begun to address the problem through statewide training. How do we address the universally poor quality or poor implementation of standards-based IEPs, and how does this relate to current approaches?
All five of the states mention this as an issue, and all have devoted training resources to address the problem. Even in Vermont, where alternate assessments are scored on the linkage, the existing quality of the IEP was found to be poor in the first year of implementation of their alternate assessment system. In a study by Sands, Adams, and Stout (1995) over half (55%) of the 341 elementary and secondary special educators surveyed believed that each student with disabilities should have his or her own curriculum based on needs as documented on the IEP. The researchers concluded, "In the absence of a curriculum base that provides direction for special education programs, instructional decision making and procedures are often haphazard and widely divergent" (p. 69). Even with IDEA requirements to provide access to the general education curriculum, Tindal and Fuchs (2000) found it disturbing that the IEP "typically does not conform to the substantive spirit reflecting federal legislation. Rather, IEPs have served primarily as a tool for procedural compliance monitoring, whereby federal auditors make sure that a complete IEP exists for each student receiving special education services and that IEPs document how (i.e., where, when, and by whom) those services are delivered" (p. 4).

There is evidence that implementation of thoughtful alternate assessment approaches can directly improve quality of IEPs. During the first year of implementation in Vermont, only about a third of the portfolios were rated as meeting or exceeding program expectations. Operationally, that meant that the portfolio included an IEP that did not reference Vermont standards, and did not include an adequate progress measure. In the second year, after concerted effort for IEP related professional development, more than two-thirds met or exceeded expectations. This may be an area where thoughtful training connected to alternate assessment implementation can have a rapid positive effect on practice, if the assessments are designed specifically to meet the unique needs of this population.

At a recent discussion at a Technical Advisory Committee (TAC) meeting for a state not in this study, most TAC members wondered why the state’s alternate assessment could not simply be a report of how many IEP goals and objectives were met by the students in a given year. They expressed amazement at the research base that indicates the quality of IEPs is not sufficient for the measurement purpose, nor do they generally reflect standards-based instructional goals. These TAC members are typical of the individuals on most state TACs: all have outstanding expertise and experience in measurement of academic achievement for the general population of students. Most have never worked with students who have significant cognitive disabilities, nor have they seen what typical evidence of their learning looks like. Another opinion expressed by the TAC members was that if performance tasks are not standardized for all students, then it is not really measurement. There is a huge gap between the experiences of technical and measurement experts and the experiences of the students who participate in alternate assessments. State alternate assessment stakeholder committee members and state special education personnel who understand these issues need to work closely with technical advisors and state assessment personnel as they develop and refine their alternate assessments.

6. Performance. Students doing something – is this the common definition of "performance" as defined in varying ways in the state criteria? Is "performance" the bottom line in measurement of academic achievement for students in the alternate assessment?
That none of the states used the same language to define "performance" is telling – perhaps all of the criteria together is what defines performance! Appropriateness, accuracy, mastery, progress, independence, multiple settings, multiple occasions, or multiple purpose all seem to contribute in varying ways to the criterion of "performance." "Performance" generally reflects that a student has actually learned a standards-based skill or at least demonstrates something that shows progress toward a defined outcome. There is a need for continuing discussion among state stakeholders to ensure that the scoring criteria as applied reflect the assumptions and values that can be defended by research and best practice for the highest possible outcomes for students with significant cognitive disabilities.

Three scoring criteria are very different across the five states’ approaches. They include "system vs. student emphasis," "mastery," and "progress." Two states, Kentucky and Oregon, also use additional criteria that no other state uses. A brief description of these differing criteria is shown in Table 19.

1. System vs. Student Criteria
The system versus student criteria differences among the five states have been examined throughout this paper. It is important for states to articulate why they support system or student criteria. All of the five states have attempted to do so.

2. Mastery
Vermont defines mastery as the ability to perform a targeted skill or ability independently, or using natural supports and assistive technology that permit independence. Thus, "mastery" for Vermont relates to "independence." Arkansas includes "mastery" as part of the definition of "performance;" it is defined as performing a skill in multiple settings or on multiple occasions, and is thus interrelated to generalization. These two examples illustrate the murky nature of clarifying criteria: Is mastery as defined in these states a different criterion, or are these simply different names for independence and generalization? Oregon does not use the term mastery, but the criterion of "accuracy" may be addressing the concept of mastery.

3. Progress
Progress is identified in three states, but the three states have very different approaches to it. Kentucky scores for "progress" in two places: (1) progress on independence as part of the "support" criterion, and (2) progress on IEP goals for the "performance" criterion.

Vermont broadly defines progress as "development, improvement, or positive changes in the student’s performance in relation to designated learning outcomes, including incremental skill development, fading of supports, or generalization to more natural settings. . . measured at least two points in time . . . amount or rate of progress is not measured, only that it has or has not occurred." But Vermont also notes that "for some students with degenerative disabilities or health impairments, they may demonstrate progress by ‘standing still’ or regressing as slowly as possible. In these rare cases, maintenance related to quality of life indicators, may be the most appropriate progress measure. This kind of ‘progress’ might be documented through logs, team notes, or other anecdotal records."

Oregon has a very different approach to progress. It has committed to a growth model of accountability and looks for progress along a continuum of learning that encompasses all students. Oregon is working to integrate its extended assessments in reading, mathematics, and writing into the continuum, as described earlier, but the Extended CLRAS appears to be outside the continuum.

4. Single State Criteria
Kentucky has carefully studied the research base on teaching and learning for students with significant disabilities, and it has carefully articulated state stakeholder values. In the end Kentucky comes down strongly on the side of extensive system measures, such as access to supports, and also student indicators some states see as "removed" from academics, such as social relationships and self-determination. Kentucky points to a strong research base that connects self determination skills in the context of instruction to achievement of learning goals, suggesting that these measures are strongly related to academics. These decisions also reflect values grown out of inclusion and transition systems change efforts that have been tied to the development of Kentucky’s alternate assessment approach.

Similarly, Oregon has gone to its more behavioral research base and stakeholders, and come down strongly as well, but in a very different direction from Kentucky for at least for one part of its alternate assessment system, the extended assessments in reading, mathematics, and writing. Oregon is the only state to use a quantitative measure of "accuracy," on its extended assessments in reading, mathematics, and writing but not on its Extended CLRAS. The Extended CLRAS is very similar to other state approaches, with performance events scored on generalization and level of prompting. Not all students with significant cognitive disabilities take the extended assessments in reading, mathematics, and writing, or even one of them; some take only the Extended CLRAS.

The common criteria and differences in criteria across the five states described in this report do not reflect "right" or "wrong" approaches. Each of these five states has developed alternate assessment scoring criteria that reflect their best understanding of what successful outcomes are for students with significant cognitive disabilities. We hope that many state personnel will have opportunities to read this report and will be prompted to carefully examine their own criteria, asking what each criterion means and how it is important to each student’s success. Alternate assessments are still very new, and taking the time for thoughtful reexamination is critical. Here are some questions and recommendations you may find helpful as you work to ensure your alternate assessment reflects your state’s best understanding of successful outcomes for students with significant cognitive disabilities.

1. What are your stated and embedded scoring criteria? What practice or research base was consulted as you developed these criteria? In addition to the actual criteria listed in rubrics, be sure to articulate the criteria that are implied in the design of assessment items or format, and additional criteria that may be embedded in definitions or in rubric gradations of quality descriptions.

2. Clarify exactly what the scoring criteria (i.e., develop definitions) are for your state’s alternate assessment and assign them to either the student or the system, or both. This will assist in identifying the underlying assumptions reflected in scoring.

3. Determine whether the underlying assumptions reflected in scoring criteria are consistent with the views of various stakeholders, including educators, parents, and assessment personnel.

4. Are the values embedded in scoring criteria clearly articulated in written materials and training on alternate assessment?

5. Could every teacher who has students participating in alternate assessment in your state tell you what the stated and embedded scoring criteria are? Could other IEP team members, e.g., parents, general educators, related service providers?

6. What do you see as the potential that these criteria, and any changes made, will affect quality of programs and outcomes for students with significant disabilities? How will that occur? Do you have data that reflect any changes so far, either anecdotal or formal?

7. Carefully examine the scoring process in light of the scoring criteria to see if the process truly measures the criteria.

8. Examine scores after conducting an alternate assessment in light of scoring criteria – do the scores reflect original intentions? Is it possible to see areas of strength and areas that could be improved in the education of these students based on these scores?

9. Examine changes in scores across years – do these changes reflect progress toward the successful outcomes intended for these students?

10. Consider alternate assessment criteria in light of stated and assumed scoring criteria for the general assessment – how are they the same and different? What discussions have you had about the need for "the same" or "analogous" components in your alternate assessment and in your general assessment?

As states work to ensure that achievement standards are set that truly reflect the highest learning standards possible for the small population of students with significant cognitive disabilities, the alternate assessment process must be carefully and thoughtfully designed. Since alternate assessments are a much more recent development than general assessments, the assumptions underlying them are still debated and discussed. These deliberations can sometimes feel frustrating for teachers, policymakers, and other stakeholders, but they are an important activity that helps a state develop an alternate assessment that reflects its educational values for students with significant cognitive disabilities.

Algozzine, R., Browder, D., Karvonen, M., Test, D.W., & Wood, W.M. (2001). Effects of interventions to promote self-determination for individuals with disabilities. Review of Educational Research, 71 (2), 219-77.

Browder, D.M. (2001). Curriculum and assessment for students with moderate and severe disabilities. New York: Guilford Press.

Browder, D., Flowers, C., Ahlgrim-Delzell, L., Karvonen, M., Spooner, F., & Algozzine, R. (2002). Curricular implications of alternate assessments. Paper presented at the National Council of Measurement in Education Annual Conference, New Orleans.

Burdge, Michael. E-mail correspondence to Rachel Quenemoen, November 2002.

Giangreco, M.F., Cloninger, C.J., & Iverson, V.S. (1998). Choosing outcomes and accommodations for children: A guide to educational planning for students with disabilities (2^nd edition). Baltimore, MD: Paul H Brookes.

Kleinert, H. L., & Kearns, J. F. (1999). A validation study of the performance indicators and learner outcomes of Kentucky’s alternate assessment for students with significant disabilities. Journal of the Association for Persons with Severe Handicaps, 24, 100-110.

Kleinert, H. L., & Kearns, J. F. (2001). Alternate assessment: Measuring outcomes and supports for students with disabilities. Baltimore, Maryland: Brookes Publishing.

NCEO Alternate Assessment Retreat (2002). Notes from "think tank" retreat of selected national researchers and state training leaders. November 20-22, 2002, Monticello, MN. *

Olson, B., Mead, R., & Payne, D. (2002). A report of a standard setting method for alternate assessments for students with significant disabilities (Synthesis Report 47). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Quenemoen, R., Rigney, S., & Thurlow, M. (2002). Use of alternate assessment results in reporting and accountability systems: Conditions for use based on research and practice (Synthesis Report 43). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Roeber, E. (2002). Setting standards on alternate assessments (Synthesis Report 42). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Sands, D. J., Adams, L., & Stout, D. M. (1995). A statewide exploration of the nature and use of curriculum in special education. Exceptional Children, 62 (1), 68-83.

Smith, S.W. (1990). Individualized education programs (IEPs) in special education – from intent to acquiescence. Exceptional Children, 57, 6-14.

Thompson, S., & Thurlow, M. (2001). 2001 State special education outcomes: A report on state activities at the beginning of a new decade. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Thompson, S., Quenemoen, R., Thurlow, M., & Ysseldyke, J. (2001). Alternate assessments for students with disabilities. Thousand Oaks, CA: Corwin Press.

Thurlow, M., Quenemoen, R., Thompson, S., & Lehr, C. (2001). Principles and characteristics of inclusive assessment and accountability systems (Synthesis Report 40). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Tindal, G., & Fuchs, L. (2000). A summary of research on test changes: An empirical basis for defining accommodations. Lexington, KY: Mid-South Regional Resource Center.

Wiener, D. (2002). Massachusetts: One state’s approach to setting performance levels on the alternate assessment (Synthesis Report 48). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Ysseldyke, J. E., & Olsen, K. R. (1997). Putting alternate assessments into practice: What to measure and possible sources of data (Synthesis Report No. 28). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Several state personnel participated in the development of the content included in this report. Each person interviewed shared many wise and thoughtful insights into the development of state alternate assessment systems. It is through the commitment and dedication of each of the people interviewed that alternate assessments continue to improve, providing important information and enabling states to truly leave no child behind. NCEO gratefully acknowledges the following participants in this interview process.

Arkansas: Marcia Harding and Tom Hicks

Kentucky: Mike Burdge

Louisiana: Jeanne Johnson and Leslie Lightbourne

Oregon: Pat Almond

Vermont: Michael Hock

Arkansas

Interview: Marcia Harding and Tom Hicks, October 21, 2002

Arkansas Domain Scoring Rubric for Students with Disabilities, with Scoring Domain Definitions, and Glossary

Arkansas handouts at Alternate Assessment pre-session to CCSSO Large-Scale Assessment Conference, Desert Springs, CA, June 2002

Arkansas Alternate Assessment Implementation Guide for Students With Disabilities, 2000-2001

Kentucky

Interview: Mike Burdge, September 17, 2002

Kentucky Alternate Portfolio Assessment: Teacher’s Guide 2001-2002

Kentucky Alternate Portfolio Assessment: Frequently Asked Questions

2001-2002 Training PowerPoint Presentation

Louisiana

Interview: Leslie Lightbourne and Jeanne Johnson, September 20, 2002

http://www.doe.state.la.us/DOE/Assessment/PDFs/LAAtamManual.pdf

http://www.louisianaschools.net/DOE/Assessment/LAA.asp

General Education Access Guide: Section IV Curriculum Issues for Students in Alternate Assessment

August 2000

2002-03 LAA PowerPoint Presentation

Louisiana Alternate Assessment Test Administration Manual, Spring 2002

Oregon

Interview: Patricia Almond, September 19, 2002

Oregon Extended Career and Life Role Assessment System Based on the Career Related Learning Standards Identified with the CAM

Extended Career and Life Role Assessment Systems: Administration Manual 2001-2002

http://www.ode.state.or.us/asmt/administartion/extasmts

http://www.ode.state.or.us/asmt/administration/comprehensiveasmt.pdf

Vermont

Interview: Michael Hock, September 19, 2002

Preliminary Report on 2002 Lifeskills Portfolio Scoring

Summary of Alternate Assessment Validation Study

Participation Guidelines

Documentation of Eligibility form

Schedule for the Collections, Scoring, and Reporting of Lifeskills Portfolios

Scoring Procedures, Guidelines and Rubrics

Portfolio table of contents sheets

Training materials: (a) Building a Better Portfolio, (b) Criteria for Evaluating Lifeskills Portfolio Measurement Strategies, (c) Effective Measurement Strategies and Techniques for Students with Severe Disabilities, and (d) Exemplary Portfolio Pieces form the 2001 Portfolio Scoring Institute

* NCEO extends its sincere thanks to the following individuals who provided us with wise and thoughtful assistance in the development of the analysis and conclusions we present here. They were participants at the NCEO Alternate Assessment Retreat in November, 2002.

Mike Burdge, Jean Clayton, and Mark Hurte – Kentucky

Lynn Ahlgrim-Delzell and Diane Browder – North Carolina

Bryan Ayres and Lynn Springfield – Arkansas

Pam Green and Dan Weiner – Massachusetts

Pat Almond and Joel Arick – Oregon

The methods used to gather evidence for alternate assessment vary around the country.

The performance measures used as criteria in states to score or rate the evidence also varies.

The primary focus of our interview is the performance measures being used by states. Our interests include a description of any scoring rules, scoring domains, or rubrics that you use. We specifically hope to develop a set of case studies that describe:

a. how each state developed or decided on the measures or criteria used;

b. if, how, and why they have changed the criteria over time; and

c. how the criteria are used for improvement of programs and outcomes for students with significant disabilities.

How was this approach developed?

Can you describe the process?

What practice or research base was consulted as you developed these criteria?

Were the decisions controversial?

Were there additional criteria discussed but not chosen?

What were they and why were they not used?

Have the _________ (rubric, rules, scoring domains…) changed in any way from earlier years? How has it changed? Why has it changed? What occurred that made you consider a change.

What do you see as the potential that these criteria, and any changes made, will affect quality of programs and outcomes for students with significant disabilities? How will that occur? Do you have data that reflect any changes so far, either anecdotal or formal? Do you have this information in written form, or who can I contact to be sure we have an understanding of that information?

What advice would you give other states as they reflect on the criteria they use for their alternate assessment?

What are the key "lessons learned" from your experience?

Accuracy (number correct/incorrect)

Oregon: Extended assessments in reading, mathematics, and writing
On the Extended assessments in reading, mathematics, and writing, the number correct (or partially correct) defines accuracy. That definition includes a reading rate per minute component and in oral reading there is reading accuracy. According to state staff, it is possible that extended reading, math and writing are actually not alternate assessments but instead are a downward extension of the general assessment.

Student performance on embedded content standards (not dependent on teacher developed linkages)

Louisiana
The main focus of the design of Louisiana’s alternate assessment has been on linkage to the content standards and access to the general curriculum. The state provides the linkage through skills in three participation levels. The driving forces for scoring criteria, however, remain in the areas of generalizability and independence rather than on student performance on embedded standards.

Oregon: Extended assessments in reading, mathematics, and writing and Extended CLRAS
Oregon builds the linkages to content standards into its assessment instruments; it does not score the standards linkage. It was assumed in the development of Oregon’s alternate assessment that students could move out of the alternate and into the general assessment. With this assumption and the assumption that all students would work toward the same standards, there was no worry about whether standards would be included as part of the alternate assessment measurement system. Right after the IDEA reauthorization, Oregon’s state board had just finished adopting standards with the cut points. They essentially said, "We’re done with standards. We’re not going to reexamine these until they’ve been in place for a while. We’re going to stay the course and see what standards-based reform is all about." So, the group working on the alternate assessment was committed to not developing any new standards, but using with the ones in place for the state. The state Board of Education had already adopted the Career Related Learning Standards, so those could also be used in the alternate assessment (in the CLRAS), along with the math, reading, and writing standards assessed in the Extended assessments in reading, mathematics, and writing. It is unclear if students who participate only in the Extended CLRAS have standards-based evidence in math, reading, and writing.

Oregon: Extended CLRAS
The Extended CLRAS has one area that Oregon’s career standards do not have, motor skills. That was added because families and teachers said mobility was a key element of career skills for the population.

Level of assistance when defined as student independence, actual performance with defined degree of prompting

Arkansas
Level of assistance is defined as the observed accommodations, adaptations, or assistance used by the student during the performance of tasks. On the definitions within the rubric, there are added natural supports and assistive technology use as well. Each individual task (portfolio entry) is scored for Level of Assistance. It is weighted at one-fourth the weight for performance, one-half the weight for appropriateness, but twice the weight for settings.

Kentucky
Support is defined as degree of peer support, tutoring, or natural supports; use of adaptations, modifications, or assistive technology that demonstrates progress toward independence. This could be identified as a student measure or a system measure, depending on how the scoring process looks at actual performance or evidence of provided supports. Kentucky staff clarify that they are looking at this as a system measure, and not a student measure.

Louisiana
The criteria address whether a child can generalize a skill and do it independently without a prompt: As state staff suggest, "We’re really measuring two different things with that rubric." As scores increase, level of prompting is joined by degree of generalizability. A student who performs a skill over and over without a prompt but only in one setting and only for one purpose could not get above a "3." The teacher has a period of five weeks to observe the skill. If the student performs the skill for the first time without prompting, the observer would score a 3. If the observer then takes the student to a different setting and structures a task to show the same skill but for a different purpose, and the student performs it, the observer would score a 5. This could happen all on the same day, or it may take many observations before the student achieves the 5. In other words, once the skill has been performed in a second place and for a second reason, the student does not have to perform it again during the five week period.

Oregon: Extended CLRAS
The Extended CLRAS is instructionally based, so students have continued instruction and are measured in areas where they are low in independence. As students become more independent, more environments become available to them. A high value is given to having students learn in natural environments, and this is reflected in scoring. Scoring of the Extended CLRAS puts emphasis on prompt fading. Scoring processes do look for generalization, and that interacts with prompting. If the student does seven routines, and "social greetings" is one of the related skills performed, then all those seven environments where that would be an appropriate place to exhibit that skill. One example is eating lunch in the cafeteria, including going through the cafeteria line. If the student says, "hi" to the cafeteria lady, that’s independent. If the teacher raises her eyebrows and tips her head toward the cafeteria lady, that might be a visual or verbal or gesture prompt. The guide includes instructions to the scorers about what to watch for. Generalization is embedded in the scoring process.

Vermont
Incremental fading of supports is one of the categories demonstrating progress and mastery. The student shows progress on a single skill by needing less and less support from adults or classmates. Mastery occurs when the student can perform the skill alone. An example from money skills: the teacher pays for the student’s lunch, then the student pays for lunch with the teacher’s help, then with a classmate’s help, then all alone.

Multiple purposes when defined as student performing skill for multiple purposes

Louisiana
Multiple settings and multiple purposes are both included in the rubric in order to measure generalization. A student who performs a skill over and over without a prompt but only in one setting and only for one purpose could not get above a "3."

Mastery

Arkansas
Mastery is included on the gradations of quality on the performance criterion.

Vermont
Mastery is defined as the ability to perform a targeted skill or ability independently, or using natural supports and assistive technology that permit independence.

Performance

Arkansas
Performance is defined as the student’s demonstration of skill while attempting a given task, and at the highest quality gradations, includes mastery, multiple settings, and multiple occasions. Each individual task (portfolio entry) is scored for Performance. The performance domain weight is twice the weight given to appropriateness, four times the weight given to level of assistance, eight times that of settings.

Kentucky
Performance is defined as student progress on specifically targeted IEP goals/objectives that are meaningful in present and future environments, with age-appropriate products.

Progress

Kentucky
Progress is included in the dimension "Performance" which is defined as student progress on specifically targeted IEP goals/objectives. Progress is embedded; it is not a separate scoring criterion. Progress is also part of the "Support" dimension, with use of adaptations, modifications, or assistive technology showing progress toward independence.

Oregon: Extended Academic Assessments and Extended CLRAS
Oregon is committed to a growth model of achievement throughout the whole assessment system. Every assessment director has been committed to using a growth model for both standards and progress. They believe that students who are low on the performance scale but are making great progress should contribute positively to a school’s performance rating. They believe that a growth model is good for each student’s self esteem, for schools wanting to include them, and fair for the teachers who worked hard to get those students to make progress. So, despite the importance of the growth model in Oregon, the concept of measurement of "progress" is different in Oregon than in the other sample states.

Vermont
"Progress" is defined as development, improvement, or positive changes in the student’s performance in relation to designated learning outcomes, including incremental skill development, fading of supports, or generalization to more natural settings.

Settings when defined as student performing the skills in multiple settings

Arkansas
Multiple settings and occasions are included in the quality gradations in the performance criterion. It is included as a system measure in the single score for settings in the scoring rubric.

Louisiana
The criteria address whether a child can generalize a skill and do it independently without a prompt. As state staff suggest, "We’re really measuring two different things with that rubric." As scores increase, level of prompting is joined by degree of generalizability. A student who performs a skill over and over without a prompt but only in one setting and only for one purpose could not get above a "3." The teacher has a period of five weeks to observe the skill.

Oregon: Extended CLRAS
Scoring processes do look for generalization. If the student does seven routines, and "social greetings" is one of the related skills performed, then it is scored in all seven environments where that would be an appropriate place to exhibit that skill. One example is eating lunch in the cafeteria, including going through the cafeteria line.

Vermont
Generalization to less restrictive environments is one of the categories for determining progress and mastery–in this situation the student demonstrates progress by using a skill in "real world" situations.

Appropriateness (age, challenge, authenticity)

Arkansas
Appropriateness is defined as the degree to which the tasks: (1) reflect meaningful, real-world activities with age-appropriate materials, (2) provide a challenge for the student, (3) promote increased independence, and (4) are linked to the Content Standards. The domain of appropriateness is weighted at half the weight of performance, twice the weight of level of assistance, and four times that for settings.

Kentucky
Performance is defined as student progress on specifically targeted IEP goals/objectives that are meaningful in present and future environments, with age-appropriate products.

Louisiana
Louisiana has provided guidance to teams on specific skills at three participation levels. Thus, appropriateness is embedded in the state specified skills; they do not score appropriateness. The participation level is chosen by the test administrator, with IEP team input, based on appropriate yet high expectations for each student. The test administrator may determine that the student be assessed at any one of the three levels. However, if the test administrator chooses skills at the Comprehensive or Fundamental level and the student cannot perform at the selected level, then the less difficult level is attempted. "No Performance" can be scored at only the Introductory Level. In each content area, the state provides two state specified skills for each of two target indicators at each of the three levels. Guided by IEP team decisions, teachers determine skills for the remaining target indicators.

Oregon: Extended CLRAS
Oregon has built into the selection of routines a process for IEP teams to grapple with appropriateness, and to avoid any arbitrary "developmental sequence" of routines. Oregon does not score on appropriateness. Oregon developed the list of routines within the categories of the Career Related Learning Standards adopted by the State Board of Education for all students. The individual routines were validated for students with significant disabilities through an in depth analysis of life skills curricula collected from Oregon and the professional literature. After this cross-validation the routines were ranked by teachers and families in terms of what was most important. There is a suggested order in the assessment that is based on, "if you don’t know what to select next, here’s what the field has said is the most important thing next." For example, disembarking from the bus and arriving at school and eating independently ranked high because those were personal management pieces that the teachers and the parents said were really important. Some teachers did not want to have a prescribed order, and some did not want to be left with no idea if no one identified a preference.

Vermont
Vermont provides some guidance about the appropriate learning outcomes, but it does not score on appropriateness. According to state staff, in the determination of progress the "Incremental Achievement of Skills" probably rewards evidence of increasing challenge.

Content Standards linkages

Arkansas
Entries may be considered nonscoreable if the rules or guidelines in the Implementation Guide are not followed. Entries/portfolios that are considered nonscoreable will receive a score of ZERO; one of the rules that results in a ZERO is that it is determined as "Not to Standard: The entry does not reflect a standard from the curriculum framework."

Kentucky
The Kentucky Academic Expectations are the driving force behind instruction, entry evidence, and portfolio products. Scoring processes look for the degree of linkage to the 54 Academic Expectations identified for all students and assessed by the general assessment, with 28 being prioritized for Alternate Portfolio.

Vermont
The first thing scorers look for is whether the IEP that has been sent in is linked to state standards or the set of learning outcomes from the COACH (Giancreco, et al, 1998). Even in the lifeskills portfolios students are working toward state standards, but they are a different set than their classmates are working toward. If the IEP is linked, then the scorers look to see if there are one or more progress measures in the portfolio that demonstrate a student’s progress. One of the things the COACH authors did was a cross-match between the learning outcomes and Vermont’s state standards – so that is not something the teachers have to figure out on their own, but they do have to create the evidence of learning, and embed the standards-based learning in the IEP.

IEP linkages

Arkansas
Arkansas has implemented a major training effort to improve the quality of IEPs for student with significant disabilities, but does not include IEP linkages as a separate criterion for scoring its alternate assessment. The training was designed to be compatible with alternate assessment processes, but a decision was made not to directly link it to the alternate assessment. Arkansas is finding that as a result of the training and support offered to teachers on how to write standards-based IEPs with high expectations, the quality of portfolios improves dramatically.

Kentucky
As part of the definitions in the performance dimension, they include progress on specifically targeted IEP goals/objectives. However, Kentucky does not include IEP linkages as a separate criterion, noting that there are IEP goals for individual students that are not reflected in the alternate assessment because the assessment deals solely with standards. Some IEP goals are not connected with general curriculum standards, nor do they have to be, but it is expected that all students have some goals that are clearly standards-based. Kentucky includes methods for targeting IEP goals/objectives for assessment in their Teacher’s Guide and online training materials for KY teachers.

Louisiana
In their documentation and in their interview, Louisiana emphasizes the shift to standards-based IEPs that raise the bar for all students. However, Louisiana does NOT include IEP linkages as a separate criterion. The state suggests that IEP teams need to target the level at which a student is functioning. Some start very high, some start too low, and some are just right. Teachers are not used to doing assessment in this formal way, so it is taking a while to get teachers to learn how to find the appropriate level of challenge. Trainers have heard teachers say, "I’m thinking about how to make students more independent and I’m working with my aides on ways to fade prompts and have students use more natural cues." A note from a regional director said, "We clearly see how this is showing independence and increases student ability to generalize."

They report that educators are now really looking at IEPs and making sure they are linked with the content standards. This leads to the link back to instruction. Trainers got a sense from some of the folks that the alternate assessment validated what they were doing. Teachers are beginning to understand, for example, that there are math skills involved in "going out for lunch." Teachers are also seeing this as a way to "join the rest of the schooling world." Students are beginning to have access to more environments. Because of the rubrics that say, "Can they do this skill in a different place?" The teacher who wants to be able to take students off campus, now gives further weight to obtaining all those permissions that teachers sometimes need from their administrators. They tell their principals they will get a higher score on their state assessment if the students go out and do some of these things. The trick is to help educators to stay focused on the standards-based knowledge and skills while they do those rich activities that allow students to learn and to show what they have learned. The requirement that the efforts be documented is helping educators to stay focused.

Oregon
Oregon: Extended Academic Assessments and Extended CLRAS:
A goal in Oregon is to help teachers administer the Extended Reading, Extended Mathematics, And Extended Writing assessment and the Extended CLRAS in conjunction with preparing for an IEP meeting – to use the tests to get information for instructional planning. One of the benefits of the Extended CLRAS is that it gives teachers an "ah ha" very quickly about what they might include in a student’s IEP, what the instruction might look like, and how they might do the teaching. It ends up being a teacher training tool indirectly. The extended assessments in reading, mathematics, and writing had that effect as well. For example, a high school teacher said, "I didn’t even know he could read" after the teacher was required for the first time to actually assess the student’s reading in the extended assessment. In that sense both types of alternate assessment carry a strong message according to state staff.

Oregon: Extended CLRAS
The routines are anchored in environments, but the related skills are assessed at the same time as the routines assessment. For the related skills there is a manual that defines the skills in terms of expressive communication and receptive communication; those are to be taken from the student’s IEP, looking for those skills to be exhibited in the context of the routines.

Vermont
The scoring criteria in Vermont are based on students’ IEPs and progress measures that are designed to measure students’ progress against IEP goals that are referenced to the learning outcomes or the corresponding Vermont Standards. The learning outcome can be quoted directly, paraphrased, or can be referenced to the corresponding Vermont Standards listed in the rubric. The outcome might be embedded in a larger goal or outcome. In the field studies, we discovered that the IEPs for students with severe disabilities were not very good. One goal of the assessment system was to get people to write good goals for students with severe disabilities and assess students’ progress toward those goals.

Self-determination when defined as evidence of system providing choice

Kentucky
Self-determination is defined as evidence of degree of student choice with an impact on student learning across and within entries, and evidence of planning, monitoring, and evaluating of the self-performance to improve or set new goals. State staff see this as a system measure, not a student measure because self-determination is an instructional area for students with severe disabilities.

Vermont
Vermont state staff point out that although they don’t have specific scoring criterion on self-determination, the ability to make choices is embedded throughout their learning outcomes. For example, one of the communication learning outcomes is, "Uses communication to indicate preferences and make choices."

Settings (multiple) when defined as system offering multiple settings for student

Arkansas
Settings is defined as the observed settings or environments in which tasks are administered/performed. The entire portfolio (not individual entries) is scored for Settings. It is weighted one-eighth of performance, one-fourth of appropriateness, and one-half of level of assistance. Settings is also a student criteria, included in the definition of "performance."

Kentucky
Settings are defined as the number and variety of settings in which there is evidence of student performance, and the number of entries in which they are observed. Those settings are counted only in which the student is integrated with age appropriate peers (also, see social relationships).

Social relationships when defined as evidence of system provided opportunities

Kentucky
The "Social Relationship" dimension is scored on appropriateness of interaction, frequency, diversity of range of peers, sustainability, and reciprocity of relationships.

Support (access to assistive technology, peer supports, adaptations, accommodations)

Kentucky
Support is defined as degree of peer support, tutoring, or natural supports; use of adaptations, modifications, or assistive technology that demonstrates progress toward independence. Kentucky views this as a system measure, but this could be identified as a student measure depending on how the scoring process looks at actual performance or evidence of provided supports.