October 2003

State assessment programs provide one method of determining student academic achievement. The Washington State Assessment System (WAAS) provides accountability for program and educational opportunities for all students. Alternate assessment for students with significant cognitive disabilities, as part of Washington’s assessment program, ensures a unified system, program, and student accountability linked to the common core of learning within the general curriculum. The Washington Alternate Assessment System process was developed by the Washington Alternate Assessment Task Force and expanded by Advisory Panels in response to the following requirement in the Individuals with Disabilities Education Act 1997 (IDEA): "The State has established goals for the performance of children with disabilities in the state that . . . are consistent, to the maximum extent appropriate, with other goals and standards for children established by the state." It was toward fulfillment of this requirement that alternate assessments are based on Washington’s Essential Academic Learning Requirements (EALRs) in the content areas of Communication, Reading, Writing, Mathematics, and Science. The Washington Alternate Assessment System is a portfolio assessment designed for a very small percentage of the total school population for whom traditional assessments, even with accommodations, would be an inappropriate measure of progress.

The basic building block of the portfolio assessment is evidence of the student’s work. Each of the entries in the portfolio documents two dimensions of learning: progress on IEP skills linked to the EALRs and student generalization of those skills. Evidence of the student’s work demonstrates participation in and progress toward those IEP goals that are aligned to state standards (EALRs). In this way, evidence of progress on IEP skills linked to the EALRs can measure progress on state goals and standards. Student generalization of skills evidence shows the extent to which a student can demonstrate the IEP skill linked to EALRs in the following ways:

• Use of appropriate modifications/adaptations, supports, or assistive technology to demonstrate all the student knows and is able to do.
• A variety of settings and contexts in which the student is able to use learned skills. These places can include the classroom, other areas of the school, community settings, and home.
• Interaction with nondisabled peers and others during IEP activities for the purpose of developing social relationships to enrich his or her life.
• Use of self-determination skills in planning, monitoring, and evaluating IEP skill activities.

The inclusion of students with disabilities in the assessment and accountability system is critical to ensure appropriate allocation of resources and learning opportunities for these students. Federal legislation and regulations for the Elementary and Secondary Education Act (ESEA) and IDEA require states to report results for all students assessed using general assessments and alternate assessments relative to the same grade level academic content and achievement standards. In anticipation of the federal government publication of a Notice for Proposed Rule Making to allow setting alternate achievement standards for students with the most significant cognitive disabilities who participate in alternate assessments, the Office of Superintendent of Public Instruction sought to establish four levels of performance based on alternate achievement standards on the WAAS assessments in the fall of 2002.

The Special Education Assessment Advisory Panel reviewed the standard setting methodology used by three states, and reviewed synthesis reports and policy directives from the National Center on Educational Outcomes (NCEO). The advisory panel also reviewed the types of scores generated for the portfolio alternate assessment.

The test contractor worked with staff of the Office of Superintendent of Public Instruction (OSPI) to develop a set of procedures for standard setting for the WAAS portfolios. The Washington National Technical Advisory Committee (TAC) reviewed and made recommendations relative to these procedures, and the Washington Special Education Assessment Advisory Panel reviewed and commented on these procedures

To set standards on the WAAS portfolio assessment, a variation on a holistic method of standard setting that the contractor had used in the past was recommended to the panel and the TAC. The method is loosely based on Jaeger and Mills’ method (Cizek, 2001). With this method, the panelists’ task is to classify student work into one of several performance categories defined to capture levels of performance as expressed by the performance-level categories. The panelists review the full range of scoring patterns and are asked to sort these score profiles into four performance levels. The method is holistic in that the panelists consider the whole of an individual student’s work. The panelists then review folders of student portfolios sampled to represent the full range of scores, and are asked to sort these folders into four performance levels as represented by the quality of the students’ work.

With this WAAS portfolio standard setting method, panelists review the implications of their standards in the form of impact data. Panelists receive cumulative frequency distributions of student scores that allow them to see the percent and number of students in each category given the standards the group of panelists has set. These data are made available to panelists after they complete the two sorting processes.

The procedures use these standard-setting methodologies and a consideration of standard-setting principles in order to optimize the efficacy of this process. The goal of the standard setting is to recommend performance thresholds or cut scores for the WAAS portfolio assessment that have been established by Washington educators, subject matter experts, and administrators in the best interests of students and the overall educational process. For each subject matter, the final achievement category is decided from a minimum subject matter rating and a minimum combination of ratings on generalization skills. The object of the standard-setting process is to decide how to combine the ratings so that the portfolios can be placed in four achievement categories. The object is not to come up with a new rubric or to change student scores but to find a way to combine them so that these students can be included in reports and evaluations.

The portfolio standard-setting meeting was facilitated primarily by the contractor. OSPI staff explained the ESEA and IDEA background that is the impetus for standards, the history of the WAAS portfolio, and presented the academic achievement standard descriptions. An OSPI psychometrician documented the process to ensure adherence with the planned steps for standard setting.

The results of standard-setting for the portfolios were shared with the Special Education Assessment Advisory Panel and with the National Technical Advisory Committee, and results of standard-setting were then presented to the Superintendent of Public Instruction.

Selection of Standard Setting Panelists

Table 1 indicates the composition of the 14-member standard setting panel that was selected to set the standards for the Portfolio. Each person participating in the standard-setting process was selected for his or her qualifications as a judge of student performance based on various factors. Teachers, educators, and subject area experts who were selected as panelists exemplified the required subject-area knowledge, teaching experience, or understanding of students necessary for an appropriate and comprehensive standard-setting study. Each panelist participating in the process represented the knowledge and understanding of his or her peers throughout the course of the process, lending a balance between diverse opinion and consensus. To ensure that a diverse opinion was obtained, some of the participants had not participated in making recommendations during the development of the portfolio system or in the scoring of the portfolios.

To ensure balance, a stratified sample of school district staff and other stakeholders throughout Washington were contacted. A concerted effort was made to balance the panel based on county representation, urban representation, representation of students with significant disabilities and schools serving various sizes of populations, gender, and race/ethnicity. The overall goal of consensus in this forum was not the unanimous agreement of all parties, but the bringing together of individual divergent experiences to form a common understanding of student performance in a subject area that is truly larger, and broader, than its individual parts.

Initial Procedure

Following the approval of the standard-setting methodology, portfolios with adequate evidence were identified by the Office of Superintendent of Public Instruction Alternate Assessment Specialist. Forty scored portfolios with sufficient evidence had been copied for range finding activities for 2001 and 2002. These portfolios had also been used as training sets during training of portfolio scorers. The total scores on the portfolio content entries ranged from 5 to 20 points. OSPI staff recorded the scores for each of five scoring dimensions in each content area of the portfolios. Sixty-two scoring patterns were found. Frequencies of student score profiles were generated from this group. Fourteen portfolios were selected to be used for standard-setting activities; this subgroup represented 32 different scoring patterns. Standard-setting was conducted using only these portfolios with adequate evidence.

Panel Meeting: Step 1 - Training

The purpose of this step was to give information about what the task of standard setting entails and about how student results will be used in reporting and evaluation. Participants introduced themselves and provided some information about their backgrounds to help the panelists get to know each other and provide a perspective for the various backgrounds of the panelists. The process and criteria for selecting panelists was reviewed to explain why the panelists were there and provide insight into what factors are important to the standard-setting process. Panelists were told that their job was to recommend how to place student results in four levels of performance. Background on the ESEA and IDEA requirements was provided.

Contractor and OSPI staff described the standard-setting process, and focused on the general nature of standard setting. This helped the panelists understand the overall process and the iterative nature of the standard setting. This session did not focus on specific procedures that the panelists were to use later, but attempted to give them an appreciation for the group judgment process and the panelist’s role in the process.

Panel Meeting: Step 2 - Review of the Assessment Material

Panelists became familiar with the assessment at this point. Panelists were told that the portfolio is a collection of evidence of student work and that only the evidence that is present can be considered. The panelists reviewed the participation criteria for WAAS so that they were familiar with the type of student who would participate in the portfolio assessment (see Appendix A).

Panelists were introduced to the content validity evidence for the assessment and the scoring processes. Exemplars of student work for each score point were reviewed by the panelists in a group setting so questions about scoring and rubrics could be answered. A summary of the development process was provided. Panelists were informed as to the use of standards across assessments and issues of comparability. The Portfolio Scoring Summary (Appendix B) and the complete EALR Extensions were distributed to panelists (A summary overview of the approach to EALR Extensions in included in Appendix C, and the entire document is available upon request). The Scoring Summary dimensions, rubrics, and EALR Extensions were developed and refined by a special education stakeholder curriculum group during the piloting phase of portfolio development. Both of these documents were explained in detail to panelists.

In 2003, evidence for student performance in science will be collected for the first time. Given the nature of the standard-setting procedures and that the requirements for portfolio entries are similar for each content area, standards were set for all content areas including science. Panelist raised questions related to scoring and were reminded that the current task before the committee is to set standards and not to alter scoring procedures.

Panel Meeting: Step 3 - Understanding the Definitions of the Standards

This step was designed to introduce panelists to the definitions of the academic achievement standards. Panelists did a brainstorming exercise to help them think of students and student work that typify the definitions of each standard and the performance of students who are at the standard. Panelists did not write or re-write the definitions at any time. This step only served to familiarize panelists with definitions that have previously been determined, and to help the panelists think about students who are at each of the levels.

Academic Achievement Standard Descriptions were provided to the panelists that identify student performance in four levels (See Appendix D). These descriptions were written by the Alternate Assessment Specialist, using National Assessment of Educational Progress (NAEP) descriptors and the portfolio scoring criteria as a framework. The basis for these definitions was reviewed, and the panelists discussed the definitions until the levels or categories of student performance were clearly distinguished from each other and no ambiguity regarding their characteristics remained.

Panelists were assigned into three groups of five or six to discuss the descriptors. Each subgroup was composed of at least one person who had portfolio-scoring experience, at least one person on the Advisory Committee, and others who were new to the portfolio process. Panel subgroups discussed and proposed revisions to the descriptors. A record of these suggestions was made so that they could be discussed by the entire group and used as a reference during the standard-setting process. The three subgroups shared their revisions with the entire panel; the revised performance descriptors listed by each subgroup were very similar. All proposed revisions were posted for use in the standard setting process.

Panel Meeting: Step 4 - Determination of Achievement Levels Based on Scoring Guides and Scoring Patterns

Panelists received rating sheets and were instructed in the process of completing the sheets. Working with the table of frequencies of scoring patterns and with the existing descriptors for each scale, panelists decided on the category ("meets" or "not meets" proficiency) for each score pattern. The level placement for each pattern and the minimum total score in each category (level 2, 3, 4) were recorded. This set of ratings was placed under Round 1 Levels on the rating forms (See Appendix E).

Panel Meeting: Step 5 - Holistic Classification of a Range of Student Portfolios

Participants received feedback on the overall panel cut scores they had established. The cut scores for the panel were based on the mean of the minimum total scores for each proficiency category. Panelists were then given the set of 14 scored portfolios that were exemplars of the patterns already categorized. Working individually, they reviewed the portfolio evidence, total score, and scoring patterns. They made decisions for each portfolio and reset, if needed, the minimum score for each proficiency level. They then reconvened into subgroups (same members as those on the previous day) to discuss ratings and make final decisions.

The discussion at this point included whether or not to set a minimum score for the Part I score for each content area. This discussion related to concerns that Part I scores were used once for each content areas, but Part II scores were used across all content areas, with the possible effect of more heavily weighting of Part II scores.

The Washington Alternate Assessment Portfolio Scoring Summary (Appendix B) consists of two parts. Part I: progress on IEP Skills is scored separately for each content area entry. The score points are included below.

Part I: Progress on IEP Skills
(Progress on IEP skills scored separately for each content area entry.)

Part II: Student Generalization of Skills has four dimensions that are scored across the entire portfolio. The dimensions and score points are shown below.

Part II: Student Generalization of Skills
(These dimensions are scored across the entire portfolio.)

The discussion focused on the implication of Part I scores, which reflect on student progress on IEP skills linked to the EALRs, and if Part I scores had appropriate weighting in final determination of performance. Since the Part II dimensions are applied to all content areas, there was some concern that there was an effect of weighting on Part II scores that was not intended. The panel did not reach a conclusion at this time, but tabled the discussion for later in the process. Then, each panel member completed the rating form for Round 2.

Panel Meeting: Step 6 - Review of Impact Data

Panelists were presented impact data in the form of frequencies for each score pattern generated from 2002 portfolio scores, as well as statewide performance data to judge the impact of group standards, which included the entire population of students assessed that satisfied state-determined completion criteria. The panelists also received frequency distributions of total scores for the state and scoring patterns. Panelists discussed the impact of standards on the state.

After participants discussed the impact data they resumed the discussion of Part I and Part II score effects. After reviewing the Academic Achievement Standard Descriptions, the panel agreed on a method for determining standards, and discussed possible decision rules to allow for consideration of the Part I score in determining achievement levels. The decision rules as proposed by the Panel are provided in the Results section, and allow for a two part decision based on overall score balanced by the score on Part I.

Panelists were given their Final Rating Sheets and asked to make any changes they wish on the basis of the impact data and group discussions. Panelists were advised that this was the last round of adjustments. Panelists were allowed to change the raw score value of the group cut scores according to this new information. No group consensus was pursued. Participants turned in their final Round 3 cut score recommendations. The final cut scores were calculated as the mean of the minimum total scores for each proficiency category.

The academic achievement standard descriptors were the reference documents used during steps 4, 5, and 6 for checking to see if particular score patterns and cut scores indicated that the student had met the standard.

Panel Meeting: Step 7 - Evaluation of the Standard-setting Process

Panelists were given evaluation forms to complete and open-ended comments were encouraged. A summary of the responses is given in Table 2. The eight evaluation questions were rated on a 1 to 5 scale, with 1 indicating Not At All to 5 indicating Completely. All of the questions had a majority of responses of 4 or 5. Six panelists included additional comments. Four of the six comments were positive, such as great two days, great process, good interaction. One panelist made a suggestion for training the panelists on the general education curriculum and related (WASL) assessment procedures. One panelist felt that some bias was present (men seemed to be listened to more so than women.)

Based on the work of this panel, alternate achievement standards were set for the alternate assessment portfolio. Cut scores were set using the total score from adding the score for each part one score (Part I score - Progress on IEP Skills score for the content area) to the total score for the Part II dimensions (Modifications and Adaptations, Settings and Contexts, Social Relationships, and Self-Determination). Table 3 shows the mean score from the standard setting committee for Round 1 and Round 2.

The second part of the decision rule requires a minimum score on the first scoring dimension (Progress on IEP Skill). The achievement level for any subject cannot be more than 1 level higher than the subject dimension score. That is, a portfolio with a 1 in the subject dimension cannot be in an achievement category higher than 2. A portfolio with a score of 2 in the subject dimension cannot be in an achievement category higher than 3. Table 4 summarizes the decision rule. Portfolios with insufficient evidence are treated as zeros and students are reported as not meeting the standard for accountability purposes.

Appendix F shows the relationship among the various total scores, performance level, and pattern of various scores. Table 5 summarizes the actual 2002 WAAS portfolio results categorized in each of the four levels of performance using the decision rule in Table 4.

The percentage of students meeting WAAS standard on the portfolio for 2001 and 2002 ranged from 4 to 14 percent which creates the baseline data for student performances using alternate academic achievement standards. There are many factors that should be considered when interpreting this data.

First, in analyzing trend data from 2001 to 2002, more students were meeting standard and there tended to be more scores of 2, 3, and 4 on individual scoring dimensions in 2002 than in 2001. While more scores of 1 were awarded in 2002, there were a larger number of teachers who knew what evidence was needed to obtain higher scores. This higher level of awareness may be due to numerous teacher workshops offered in the fall of 2001.

Secondly, administration of the portfolio assessment is highly dependent on the individual student’s Individualized Education Program (IEP). IEPs for this population are not universally aligned to EALR extensions, and extensive training over the next years should result in better alignment and articulation of IEP goals and objectives that allow students with significant disabilities to access the EALR standards. Training to improve assessment literacy and data collection strategies are also planned for future statewide professional development sessions.

As a result of professional development for special education staff in the areas mentioned above, changes in IEPs should drive changes toward greater access to the general curriculum for students with significant disabilities. Better data collection methods and strategies should also provide better instructional feedback for improved student learning. The state is planning to survey teachers about their perceptions regarding greater access to the general education curriculum as a result of implementing the portfolio.

The documentation of the standard setting procedure and the perception of the standard setting panel regarding the decisions that were made (showing high agreement with the statement "Agree with final cut scores proposed by panel"), provide the documentation and validation of the standard setting process to establish alternate academic achievement standards for this population of students with significant disabilities.

Cizek, G.J., Ed. 2001. Setting performance standards: Concepts, methods, and perspectives. Lawrence Erlbaum Associates: Mahwah, NJ.

Participation Guidelines for the Washington Alternate Assessment System

The purpose of the Washington Alternate Assessment System (WAAS) is to provide appropriate ways to measure progress on EALRs for students in special education programs for whom the WASL is inappropriate, even with accommodations. The decision for a student to participate in the Washington Alternate Assessment System (WAAS) must be based on the unique needs of the individual student, not a specific disability category, time spent in the general education classroom, or program placement. The IEP team must ensure that the decision for a student to participate in the WAAS is not solely based on prior knowledge that the student would perform poorly on general state tests; ongoing disruptive behavior; the result of excessive or extended absences or social, cultural, or economic differences.

1. To be eligible for participation in the WAAS, the student must have a current IEP that documents the need for an alternate assessment.
2. To participate in the WAAS, the student must be at the appropriate grade level (4, 7, and 10). Students with no grade level assignment will need to be assessed at least three times during their educational career (approximately at ages 9, 12 and 15).
3. IEP teams may opt to use commercially available tests to measure progress in reading, math, writing or listening. This option is available for students whose academic skills can be measured, but whose disability prevents them from participating in one or more component parts of the WASL even with accommodations.
4. If an IEP team determines that any component part of the WASL is inappropriate for a student and that commercially available tests are also inappropriate, the portfolio assessment should be used.
5. The percentage of students in special education programs participating in an alternate assessment (either commercially available tests or the portfolio assessment) should not exceed 20 percent of the special education population in the district at grades 4, 7, and 10 in a given year.

For further information on the participation of students with disabilities in the state’s assessment programs, please see Guidelines for IEP Teams in Determining WASL Assessment Options for Students in Special Education Programs, February 2001, and Guidelines for Participation and Testing Accommodations for Special Populations on the Washington Assessment of Student Learning (WASL), Olympia, WA: Office of Superintendent of Public Instruction, June 2000. These documents are also available at the following web site: www.k12.wa.us/special

Washington Alternate Assessment Portfolio Scoring Summary

Introduction to Essential Academic Learning Requirements (EALR) Extensions for Communication, Reading, Writing, Mathematics, Science

The EALR extension charts (available upon request) link Essential Academic Learning Requirements (EALRs) to IEP skills. The critical function of the EALRs, the access skills, instructional activities, and assessment strategies contained in the charts will assist special education staff members in linking functional IEP skills to the EALRs, in providing access to the general education curriculum, and in measuring student progress toward achieving the EALRs.

Introduction to EALR Extensions

For students in special education programs who will be participating in the portfolio assessment option, there are two ways to link skills on the IEP to the general education curriculum: the critical function of the EALRs and access skills to the EALRs. The critical function of an EALR defines the underlying principles of the broad learning standard. Basic communication, motor, and social access skills provide a starting point for those students unable to participate in the critical function of the standard. These access skills are embedded in instructional activities related to standards-based instruction. The critical function and access skill links to the EALRs are based on the basic attributes of each content area.

What are the basic attributes of communication?
Communication consists of verbal or non-verbal cues or skills that allow the student to gain understanding (receptive) or to impart a message (expressive).

What are the basic attributes of reading?
Reading consists of pictures, symbols, words, and/or text that have meaning and which the reader decodes to construct meaning.

What are the basic attributes of writing?
Writing consists of encoding symbols in a way that results in a product and conveys meaning.

What are the basic attributes of mathematics?
Mathematics consists of a language of symbols, numbers and words that communicates about patterns and relationships that allow the student to participate in mathematical inquiry and problem-solving.

What are the basic attributes of science?
Science consists of questioning cause and effect phenomena and using technology to make personal sense of the world and to solve problems.

Washington Alternate Assessment System Portfolio Academic Achievement Standard Descriptions

The academic achievement standards for students with significant disabilities who are participating in the Washington Alternate Assessment System (WAAS) portfolio are significantly different than the standards for students who participate in the Washington Assessment of Student Learning (WASL). The WAAS portfolio is based on the Essential Academic Learning Requirements (EALR) Extensions which allow the student to participate and progress in the general curriculum. Because the WAAS portfolio is based on the student’s Individualized Education Program (IEP) goals in relation to the EALR Extensions, the specific assessment targets selected for the student may be the same for many content areas but may be different than for any other student. Additionally, these students have educational goals that may remain the same throughout their educational careers. Therefore, the following academic achievement standard descriptors apply for all grades and content areas.

Level 1 - Students performing at this level will be making little or no progress toward the goal for the targeted IEP skills linked to the EALRs. The student is unable to generalize the use of these targeted skills, using modifications and adaptations in any settings or contexts. The student cannot make choices in planning, monitoring or evaluating own performances. The student has no or limited social interactions with others during educational activities.

Level 2 - Students performing at this level will be making some progress toward the goal for the targeted IEP skills linked to the EALRs. The student is able to generalize the use of these targeted skills in some ways. The student may appropriately use modifications and adaptations in some settings and contexts or make choices in planning, monitoring or evaluating own performances. The student may have some social interactions with others during educational activities. The student is not able to generalize the targeted IEP skills in all of these ways.

Level 3 - Students performing at this level will be making clear progress or attaining the goal for the targeted IEP skills linked to the EALRs. The student is able to generalize the use of these targeted skills, appropriately using modifications and adaptations in a variety of settings and contexts while making choices in planning, monitoring or evaluating own performances. The student sustains some social interactions with others during educational activities.

Level 4 - Students performing at this level will be attaining or exceeding the goal for the targeted IEP skills linked to the EALRs. The student is able to generalize the use of these targeted skills, appropriately using natural supports, modifications or adaptations in an extensive variety of settings or contexts while consistently making choices in planning, monitoring or evaluating own performances. The student has sustained, varied social interactions with others during educational activities.

Standard Setting for the WAAS Portfolio Rating Form

Relationship Among Total Score, Level, and Score Patterns