 |
|
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
|||||||||||||||||||||||||||||||||||||||||||||||||||
 |
 |
 |
|
|||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
|||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
Special Topic Area:
|
| >Introduction >Frequently Asked Questions >Publications and Other Universal Design Resources |
Frequently Asked Questions
1. What are "universally designed
assessments"?
According to the No Child Left Behind Act of 2001, universally designed
assessments are those that are “designed from the beginning to be accessible
and valid with respect to the widest possible range of students, including
students with disabilities and students with limited English proficiency”
(NCLB, 200.2(b)(2)).
2. What specific steps are
involved in the universal design of assessments?
Specific universal design steps to be integrated with standard test
development procedures at each stage of development may include:
A. Test conceptualization:
- Define the construct(s) to be measured precisely and explicitly so the test can be designed to measure the construct while minimizing the effects of irrelevant factors.
- Include the full range of students in the definition of the target population.
B. Test construction:
- Develop items that minimize the effects of extraneous factors and that can be used with accommodations as appropriate, (e.g., avoid unnecessary use of graphics that cannot be presented in Braille, use font size and white space appropriate for clarity and focus, avoid unnecessary linguistic complexity when it is not being assessed). It is the construct that must be held constant, not the design features; there are times, for example, when linguistic complexity is appropriate and necessary.
- Provide for a full range of test performance to avoid ceiling or floor effects.
- Undergo a review of items using tools such as NCEO's "Considerations for Universally Designed Assessments" (in press). By promoting a structured review of items, test companies can determine the design strengths and weaknesses of items before field testing. Determining well-designed items and items that need minor adjustments may save time and money over unstructured item reviews that simply eliminate potentially problematic items.
C. Test tryout (field testing)
- Include a full range of students in the tryout sample (e.g., students with disabilities, students with limited English proficiency, other students with special needs). Because there may be constraints in sampling due to the low numbers of students with specific characteristics, there may be a need to identify over-sampling strategies (e.g., select groups of items for which additional sampling will occur).
- Include the use of accommodations during the test tryout.
D. Item analysis:
- Analyze item characteristics to determine which items can be used with the full range of students and with accommodations.
- Use a wide range of statistical tests to determine if items are functioning differently for particular populations. Populations of students with particular disabilities or primary language are often small in number, so using multiple analysis techniques will help test designers to see patterns of items to “flag” for further investigation. Examples of statistical techniques can be found in Analyzing Results of Large-scale Assessments to Ensure Universal Design.
E. Analysis
of "flagged" items
Conduct
cognitive labs (think aloud studies) with a small number of actual students
who will take the test. Student data can be used to determine if the design
of items set forth by test designers is comprehensible to students.
Involving a wide range of students is helpful in gathering particular
perspectives.
- Information on cognitive labs can be found in forthcoming NCEO reports
F. Test revision:
- Eliminate items with evidence of disability bias.
- Include the full range of students and the use of accommodations in the test administration.
3. Are universally designed
assessments specifically meant to benefit students with disabilities and
students with limited English proficiency?
Universal design does not apply exclusively to people with disabilities or
limited English proficiency; it applies to all individuals, with wide
ranging characteristics. For example, by designing assessments to provide
for a full range of test performance, and thus avoiding ceiling or floor
effects, the assessments provide better information on the performance of
all students. Similarly, reducing extraneous features, such as
unnecessary linguistic complexity and confusing or low contrast graphics,
allows all students to better show their skills on the constructs being
tested. By thinking about all students during test conceptualization,
construction, field testing, item analysis, and test revision, universal
design results in more usable and valid assessments for all students,
reducing the need for different forms, booklets, or assessments.
4. What stakeholders benefit from
universally designed assessments?
All stakeholders may benefit from
universally designed assessments. As noted above, students will benefit from
being able to better "show what they know" on assessments. Research shows
that all students are more successful when universal design techniques are
applied to tests (see Johnstone, 2003). In addition, schools benefit
when assessments are universally designed because they are better able to
target students that are truly in need of additional services. Accessible
assessments provide schools an opportunity to make more valid inferences
about student learning than those with design issues.
In addition to students and schools, States benefit from universally designed assessments. According to the No Child Left Behind Act of 2001, States are accountable for student success on large-scale assessments. Assessments that remove construct irrelevant variance and are designed for access (while maintaining challenging constructs) are more likely to produce improved achievement levels from all students.
Finally, test designers may benefit from employing elements of universal design for two reasons. First, as states learn about elements of universal design, they are more frequently including requirements for universally designed assessments in their contract proposals with test companies. Second, employing relatively inexpensive ways of examining and improving items such as structured item reviews, protocol analysis, statistical review of items and secondary analyses of “flagged items,” test designers can improve items and save the undo costs of creating new items. Furthermore, items designed with elements of universal design (see Thompson, Johnstone, & Thurlow, 2002) are less likely to be “flagged” and need improvement.
5. Will universal design
result in reducing or eliminating the use of accommodations during
assessments?
Universally designed assessments will not eliminate the need for all
accommodations. They can significantly reduce the need for them, however,
and will increase the variety of accommodations that can be used without
threat to the validity and comparability of the scores. This will result in
inclusive accountability measurement, and provide instructionally supportive
information across the full range of students. Some students will still need
accommodations, however. For example, students who are easily distracted by
the presence of other students may still need to be tested individually;
students will need assistive technology for presentation and response; and
students who cannot read print in a size less than 18 point, or who must use
Braille, will still need a large print or Braille test booklet.
6. Are
universally designed assessments easier, resulting in artificially higher
scores?
Use of
universal design principles in assessment development may result in more
accurate scores that reflect actual student knowledge and skills, and not
extraneous factors. That is, universal design principles include careful
consideration of the construct, level of difficulty, and nature of the
measurement problem. Design decisions do not change features necessary to
the intended measurement problem.
Most standards-based assessments used for school accountability are NOT intended to measure student characteristics and skills such as visual acuity, hand/eye coordination, or the ability to find isolated facts within a puzzle of distracting information. By developing tests that remove these kinds of extraneous and confounding factors, a more accurate assessment of the intended construct is obtained. Even for those individual student assessments that do measure visual acuity, manual dexterity, discrimination, or ability to comprehend complex language, universal design principles require a careful match of each task to the construct and level of difficulty for those assessments as well
7. Does
universal design of assessments assume that the assessment is Web-based or
computer-based in delivery?
No,
universal design principles apply to all media used for assessments,
including the current dominant medium of pencil and paper assessments. If a
pencil and paper assessment without universal design characteristics is
computerized, the same problematic features will occur on the computerized
version. Even though it may be more feasible to introduce audio, video, and
assistive technology interfaces on computerized assessments, it does not
automatically eliminate existing design flaws (e.g., extraneous graphics,
floor effects, unnecessary linguistic complexity).
8. Why
should a State, school district, or test company
explore universal design as an approach to assessment?
First, there are many academic, economic, and equity benefits that can
be derived from designing assessments for greater access. In addition, three
U.S. laws require the use of universal design, both in education in general
and in assessment specifically. The Assistive Technology Act of 2004 and the
Individuals with Education Improvement Act of 2004 both require “designing
and delivering products and services that are usable by people with the
widest possible range of functional capabilities.” In addition, the No Child
Left Behind Act, replete with its heavy focus on accountability through
assessment, requires all assessments to “be designed from the beginning to
be accessible and valid with respect to the widest possible range of
students, including students with disabilities and students with limited
English proficiency.”
Related NCEO Publications
- 2005 State Special Education Outcomes: Steps Forward in a Decade of Change / PDF Version (3MB)
- Analyzing Results of Large-scale Assessments to Ensure Universal Design (Technical Report 41) / PDF format
- Improving Validity of Large-scale Tests: Universal Design and Student Performance (Technical Report 37) / PDF format
- Access to Computer-Based Testing for Students with Disabilities (Synthesis Report 45)
- Universal Design Applied to Large Scale Assessments (Synthesis Report 44) / PDF version
- Using Computer-based Tests with Students with Disabilities (Policy Directions 15) / PDF Version
- Universally Designed Assessments: Better Tests for Everyone! (Policy Directions 14) / PDF Version
|
About NCEO
|
Publications |
Special
Topics |
Projects
|
Presentations |
State
Policies |
Accommodations
Bibliography |
Teleconferences |
Tools |
Related Sites
| Indicator 3
|
|
|
||
