View or download the full report as a PDF document.

Computer-based Testing: Practices and Considerations

Synthesis Report 78

September 2010

All rights reserved. Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:

Thurlow, M., Lazarus, S. S., Albus, D., & Hodgson, J. (2010). Computer-based testing: Practices and considerations (Synthesis Report 78). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Table of Contents

• Executive Summary
• Overview
• Contextual Issues Related to Computer-based Testing
• Analysis of States’ Computer-based Tests
• Discussion and Considerations
• Conclusions
• References
• Appendix A: Advantages and Challenges of Computer-based Tests (CBTs)
• Appendix B: Computer-based Tests: Specifications and Details
• Appendix C: Web Sites used in Analysis of States’ Computer-based Tests

Executive Summary

Computer-based testing (CBT) has emerged as one of the recent “innovative” approaches to assessments most pursued by states. CBT is lauded as the answer to having cheaper and speedier test delivery for state and district-wide assessments. It is also seen by some as an avenue toward greater accessibility for students with disabilities. In this report we explore the context of CBT, current state computer-based tests, and considerations for students with disabilities, in part as follow-up to a similar exploration that occurred in the early 2000s when just a few states were beginning to develop and implement CBT for their state assessments. Nine considerations for states and districts are presented:

• Consider the assumptions and beliefs of various stakeholders about computer-based instruction and assessments.
• Consider the system as a whole, from the computer infrastructure to classroom and instructional experiences with computers before deciding whether and how to use CBT.
• Consider the computer or online platform first, with input from individuals who know students with disabilities and their accessibility needs.
• Consider a process for bringing in the needed expertise to delineate the specific accessibility features of CBT, and to determine what specific accommodations may still be needed by students with disabilities, as well as to determine whether a computer-based test may create new accessibility issues.
• Determine the policies for which accessibility features will be available to all students and which are designated for specific groups of students, such as students with disabilities.
• Consider how to track the use of accessibility features incorporated into CBT design.
• Field test the accessibility features of the computer-based test at the same time that the computer-based test is field tested.
• Examine results from CBT for students with disabilities to determine whether there are any features or characteristics of the assessment that might need reconsideration.
• Develop training for teachers and students to ensure that students benefit from accessibility features.

Top of page | Table of Contents

Overview

Computer-based testing (CBT) has become widespread in recent years. Some states now use an online platform as the primary delivery mode for one or more computer-based tests used for accountability purposes. When CBT was emerging in state testing in the early 2000s, Thompson, Thurlow, Quenemoen, and Lehr (2002) examined the implications of CBT for students with disabilities. There was not much literature about the use of CBT for large-scale assessments at that time, and Thompson et al. worked with states to explore what needed to be considered during development for students with disabilities and how states might address the needs of these students for accommodations in a CBT environment.

Since the early 2000s, much has occurred in CBT. CBT seems to have advantages over paper and pencil testing, both for states that run the assessment programs and for the students who participate in them. These advantages are recognized by the U.S. Department of Education, which in one of its major initiatives (Race to the Top Assessment Program), encouraged the development of CBT. There currently is strong interest in CBT and advocates have identified many positive merits of this approach to assessment including: efficient administration, student preference, self-selection options for students, improved writing performance, built-in accommodations, immediate results, efficient item development, increased authenticity, and the potential to shift focus from assessment to instruction (e.g., Becker, 2006; Salend, 2009; Thompson et al., 2002). CBT also allows new ways of assessing students that move beyond the traditional multiple choice and constructed response items. For example, innovative assessments are now being developed that enable students to manipulate data and role play. Yet, as states move forward with CBT they are discovering that it is important to consider not only the positive benefits, but also potential negative unintended consequences. These include, for example, the possibility that additional training will be needed for students with disabilities to interact successfully with computers and the challenges of determining the best way to present some accommodations such as screen readers.

Despite the fairly dramatic increase in attention to CBT, accessibility challenges continue to have the potential to reduce the validity of the assessment results and to exclude some groups of students from assessment participation. In the early years of CBT many fairly basic design issues baffled testing companies and states as they sought to transfer paper and pencil tests onto a computer-based platform (Thompson, Quenemoen, & Thurlow, 2006). Many of those issues have been resolved. Some states also believe that CBT may be more efficient to administer than paper and pencil based tests, and new test designs may have the potential to improve the assessment of students with disabilities. For example, some accommodations can be embedded in computer-based tests and there may be less variability in how some accommodations are delivered (for example, a screen reader may deliver the read aloud accommodation more consistently than a human reader). However, students with disabilities may be at a great disadvantage if their wide range of access needs are not considered from the beginning of the development process (Thompson et al., 2002).

The purpose of this report is to highlight states’ computer-based tests and the issues that should be considered as they are designed and implemented to include students with disabilities. This report also suggests factors to consider when designing computer-based tests. This report has three sections: (a) description of contextual issues; (b) analysis of states’ CBT; and (c) discussion of considerations for ensuring that computer-based tests are accessible for all students, including students with disabilities.

Top of page | Table of Contents

Contextual Issues Related to Computer-based Testing

The implementation of CBT occurs within a context that both supports and limits its use. In this section, we briefly address several of the contextual factors that surround CBT including: (a) the technological capacity of schools to support CBT, (b) universal design applied to CBT, (c) perceived advantages and disadvantages of CBT, and (d) current federal programs that promote CBT, such as the Race to the Top Assessment Program.

Technological Capacity in Schools

Access to computers and Internet capabilities have for some time been a stumbling block for the push to widespread use of computer-based and online assessments. For example, Becker (2006) questioned “digital equity” in computer access, computer use, and state-level technology policies. He used data from the National Assessment of Educational Progress to examine digital equity, and reported that “students in rural schools or schools with high percentages of African American students were likely to have less access to computers” (p. 1). Becker did not examine whether having a disability had any impact on computer availability or use.

A nationally representative school-level survey on information technology conducted by the National Center for Education Statistics (Gray, Thomas, & Lewis, 2010) reported several findings about the availability and use of technology in schools in the fall of 2008. The results generally indicated that computers with Internet access were available for instruction, and that the ratio of instructional computers with Internet access was 3 to 1. The large majority of computers in public schools were used for instruction. Schools also reported using a district network or the Internet to provide assessment results for teachers to individualize instruction or to inform instructional planning; nearly three-fourths provided online student assessments. Full-time technology staff members were available in about one-third of low poverty schools and one-fourth of high poverty concentration schools. The survey did not ask about availability of computers to students with disabilities, or about the use of computers for statewide testing.

Universal Design Applied to CBT

The term “universal design” applied to assessments in general has been defined in several ways (CAST, 2009; Thompson, Thurlow, & Malouf, 2004). Universal design of assessment generally means an approach that involves developing assessments for the widest range of students from the beginning while maintaining the validity of results from the assessment. Universal design also sometimes refers to multiple means of representation, action/expression, and engagement.

The application of universal design to paper-based assessments has received considerable attention (Johnstone, 2003; Johnstone, Altman, & Thurlow, 2006; Johnstone, Altman, Thurlow, & Moore, 2006; Johnstone, Liu, Altman, & Thurlow, 2007; Johnstone, Thompson, Miller, & Thurlow, 2008; Johnstone, Thurlow, Thompson, & Clapper, 2008; Thurlow, Johnstone, & Ketterlin-Geller, 2008; Thurlow, Johnstone, Thompson, & Case, 2008). More recently, attention has been paid to applying the principles of universal design in a technology-based environment. For example, Russell, Hoffman, and Higgins (2009a) demonstrated that the principles of universal design could be applied to computer-based test delivery. Technology also can be used to more seamlessly link instruction and assessment. For example, Salend (2009) specifically identified a variety of technology-based approaches that might be used for classroom assessments—for example, curriculum-based measurement (CBMs) assessment probes, digitalized observations and portfolios, and self-monitoring tools.

Dolan et al. (2009) prepared a set of guidelines specifically for computer-based assessments. The principles address test delivery considerations, item content and delivery considerations, and component content and delivery considerations. A variety of topics relevant to computer-based testing and universal design is addressed in the component content and delivery considerations section of the guidelines (e.g., text, images, audio, video), with each organized according to categories of processing that students apply during testing.

As the application of universal design principles to CBT has been considered, there also has been increased attention to various assistive technology requirements. Assistive technology devices can include such things as speech recognition or text to speech software, as well as sophisticated technology such as refreshable braille displays or sip and puff technology (which allows individuals unable to use a mouse or speech-to-text technology to send signals to a computer via a straw device using air pressure by sipping and puffing). One of the challenges of CBT has been to ensure that the assistive technology that is needed by students with disabilities is available and that the students know how to use it. Russell (2009) has considered this challenge and developed a set of 15 capabilities that should be incorporated into the computer or online platform (e.g., allowing all text appearing within each test item and on all interactive areas of the screen, including menus and navigation buttons, to be read aloud using a human voice and synthesized voices, etc.).

Perceived Advantages and Challenges of CBT

Many of the perceived advantages and challenges of CBT have been addressed in the literature. Yet, most are not focused on students with disabilities. In examining perceptions and research on CBT, we identified several categories or “themes” that can be used to organize the advantages and challenges of CBT:

• Economic: Factors that have cost implications.
• Systems implementation: Logistical, test security, and other factors that affect the capacity of local education agencies and state education agencies to implement CBT.
• Test administration/design: Factors that affect how students (and sometimes teachers) perceive and interact with the test.
• Accessibility: Factors that affect how accessible a test is to a wide range of students.

Appendix A provides details on the advantages and challenges of CBT for all students and for students with disabilities.

Current Federal Programs that Promote CBT

During the 2009-2010 school year, two competitions for federal funds promoted CBT. The Race to the Top Assessment Program was designed to support the development of innovative assessments based on a set of common core standards that, among many other qualities, made effective and appropriate use of technology. Given that the Race to the Top assessments potentially will be used by most states, the emphasis in this funding on technology is likely to have a significant impact on future assessment development. Responses to this program of funding already have generated thoughts about the interoperability of a technology platform and accessibility considerations (Mattson & Russell, 2010).

In addition to the federal funds for the regular assessment, funding for an alternate assessment based on alternate achievement standards also was made available. This competition again emphasized technology, stating “The Secretary is also interested in projects that propose the development of alternate assessment systems that use approaches to technology, assessment administration, scoring, reporting, and other factors that facilitate the coherent inclusion of these assessments within States’ Comprehensive Assessment Systems” (U.S. Department of Education, 2010, pp. A-9-A-10).

Top of page | Table of Contents

Analysis of States’ Computer-based Tests

As more states develop and implement CBT, it is important to examine the nature of these assessments and the ways in which the developers of them have considered students with disabilities. Without appropriate consideration, it is likely that accessibility issues will not be adequately addressed, with the result being an assessment that fails to produce valid results for this group of students.

To examine states’ CBTs, we collected information in three steps. Initially, a scan of state department of education Web sites was conducted to determine how many states indicated that they administered computer-based assessments. This occurred during late October and early November, 2009. A comprehensive search of state Web sites occurred later in November to collect additional information about states’ CBTs.

As a third step, a second set of researchers replicated the comprehensive search to verify information found in the states initially identified, and to see whether additional states had posted CBT information. This verification occurred in February and March, 2010. The detailed results of the Web searches are summarized in Appendix B; the Web sites used in the analysis of states’ computer-based tests are listed in Appendix C. This includes only information on state Web sites. It is possible that other assessments exist that do not have information on the state Web site.

The results of our search are shown in Table 1. We found 26 states that had at least one state-administered computer-based test, either operational or in field testing stages. States with more than one type of test are noted in the table with the number of tests in parentheses. In the 26 states with CBT, a total of 51 statewide assessments were being administered: 8 formative assessments, 24 regular tests (including retakes, graduation exams, and alternates based on grade level achievement standards), 14 End of Course (EOC) assessments, 3 alternate assessments based on modified achievement standards (AA-MAS), and 2 tests of English language proficiency (ELP) for English language learners (ELLs). One of the regular end-of-course assessments (SATP2) was being phased out in Mississippi. Two other states (Delaware and Hawaii) were field testing their computer-based tests in fall 2010; they are not represented in the table. In addition, Virginia’s Modified Achievement Standard Test (VMAST) will be administered online in 2011-2012 for mathematics and 2012-2013 for reading.

Table 1. Number of States with CBTs Found by Type of Assessment

^a Maryland has two EOC assessments. One is based on regular achievement standards and the second (AA-MAS) is based on modified achievement standards. The AA-MAS version is counted in this column.

Regular Assessment

As seen in Figure 1, most states (N=12) administered some type of regular assessment via CBT in elementary through high school grades. Only one state each administered a computer-based test in the other four categories: elementary and middle school only, middle school only, middle and high school only, and high school only.

Table 2 shows the specific states that administered computer-based tests at the various school levels. South Carolina was the only state that administered CBT only at the high school level; South Dakota only administered its test in middle school. Both Oklahoma and Washington administered CBT at two school levels, while the remainder of the states (n = 12) administered CBT across school levels.

Table 2. States Administering Regular CBTs at Different School Levels

^a Technology assessment

The availability of paper versions for CBT varied across states (see Table 3), as did the states’ perspectives on what the use of a paper version was considered to be. Paper versions were considered by some states to be an accommodation, whereas other states considered them an approach that anyone could use. One state required that a paper version be used if a student needed any accommodation. Only those states that mentioned paper versions are included in the table.

Table 3. Approaches of States Offering Paper Versions of Regular CBTs

^a Beginning in March 2010, the CMT/CAPT MAS Reading tests will be administered as a Computer Based Test (CBT) to all eligible students and uses the Measurement Incorporated Secured Testing (MIST) application. This is the same online application used with students who receive the Word Processor/Online computer response accommodation. Since this will be the primary method of taking this test, it is not considered an accommodation for these tests. If students’ disabilities interfere with what they are expected to do on MIST, they will be allowed to take the test in the most appropriate manner. There are, however, various input devices that may be used to interface with the computer. Schools should investigate these devices and determine their appropriateness for particular students. The MIST application includes a practice site that may be used to determine a student’s ability to participate in online testing. If a student’s lack of experience is a true barrier, the student should be administered the test with the appropriate grade-level test booklet(s) and allowable accommodations.

^b On Maryland’s Technology Literacy test all students must take the online assessment unless otherwise noted in an IEP.

^c Minnesota retests are online. The first time a student takes the test it is paper/pencil. The student is eligible for a paper-administered retest of the GRAD in reading or mathematics if the student has attempted to pass the GRAD in the requested subject via an online retest at least three times and submits a processing fee of $45.

^d Utah is currently transitioning to 100% computer-based testing, except for grade 2 and students whose disability warrants paper-based testing. Utah indicated that students with disabilities should be encouraged to test via computer whenever possible.

^e In West Virginia, the Westest 2 Online Writing is a Web-based assessment, and students must have on file an Acceptable Use of the Internet Form, signed by a parent or legal guardian, as directed by West Virginia Board of Education Policy. If a student does not have this form on file, he/she must handwrite the composition and a scribe will enter the composition on the Web site.

^f In Maryland, the Science test may be taken on paper or online.

As evident in Table 3, two states were in two of the columns (Connecticut, Maryland), with the different columns reflecting different approaches for different tests. A total of 12 states indicated the use of paper versions of assessments otherwise administered on computer. As noted, Texas required any accommodated student to use a paper version. Four other states indicated that the use of specific accommodations may require a paper version instead of the online version (Connecticut, Florida, Oregon, Virginia). Two states considered paper versions to be accommodations (Kansas, Oklahoma). Five states addressed other situations when paper versions were needed or implied (Connecticut, Maryland, Minnesota, Utah, West Virginia); these are reflected in the table footnotes. Two states appeared to allow any student to take a paper and pencil version for a regular assessment (Indiana, Maryland). No state prohibited paper and pencil versions.

End of Course Exams

Table 4 shows the states with End of Course (EOC) exams, with an indication of whether the exams are computer based or only available via paper (i.e., not CBT). The existence of each form of EOC was examined to determine whether all EOCs were administered via CBT. As evident in Table 4, only 3 of the 16 states with EOCs provided the EOCs only via paper.

Table 4. States with End of Course (EOC) Assessments That are CBT or Not CBT

Table 5 presents information on the availability of paper versions for computer-based EOCs. Eleven of the thirteen states with computer-based EOCs had information about paper versions of the EOCs. Most of these states (n=7) offered paper versions to any student. Only a few states addressed paper versions in other ways. One state noted that the paper version was required for certain accommodations (Oklahoma), one state allowed paper only for braille that is considered an accommodation (Louisiana), and two other states addressed other situations where paper would be needed (Mississippi, South Dakota); these are reflected in the table footnotes. As with the regular tests, no state noted a prohibition on paper versions for EOCs. Of those states that considered paper versions available to any student, Texas noted that although tests are primarily administered online, some content areas were still being introduced and that students had the option of taking paper/pencil versions at the time of this report.

Table 5. Approaches of States Offering Paper Versions of Computer-based EOCs

^a In Oklahoma, for each paper/pencil test book ordered for an IEP student, the district must have on file a copy of the section of the student’s IEP that indicates that a paper/pencil test is necessary to provide an appropriate accommodation.

^b In Louisiana, all EOC tests are administered online, except braille, which is administered in braille and the student responses entered online by a teacher or test administrator.

^c In Mississippi, Paper/Pencil retests in Algebra I and English II are for students who were first-time test takers in 2007-2008 or 2008-2009.

^d It is recommended students take the exams using Achievement Series, however it is recognized there may be times when this is not feasible. Specific instructions have been developed for utilizing EOC exams online through Achievement Series in addition to paper/pencil.

Top of page | Table of Contents

Discussion and Considerations

With the widespread and increasing use of computer-based and online testing for state and district assessments, it is time to examine the practices underway or in development, and to examine the considerations for students with disabilities. Since the Thompson et al. (2002) report examined the implications of computer-based testing for students with disabilities and how to address accommodations in CBT, there has been an evolving literature on this and related topics. For the most part, this literature has presented the advantages of CBT and has made recommendations for how to ensure that these assessments are appropriate for all students, including students with disabilities, from the beginning of their development.

Many of the advantages attributed to CBT (see Appendix A) apply to all students, including students with disabilities. The one area in which the challenges seem to outweigh the advantages for students with disabilities is in the area of accessibility. Challenges here ranged from the possibility of less access to computers and less experience with keyboarding than other students, to difficulties specifically related to certain disability characteristics, such as poor fine motor skills. In addition, the possibility that applications related to accessibility are automatically turned off in the CBT environment was identified as a challenge.

The analysis of states’ computer-based tests indicated that over half of the 50 states had at least one computer-based test either currently operational or in the field testing stages. These computer-based tests include regular assessments, end of course assessments, formative or diagnostic assessments, and assessments designated for specific populations of students (e.g., AA-MAS or ELP assessments). And, it was clear from the analysis that additional tests in additional states are under development, but not quite to the field testing stage. An examination of paper versions of computer-based tests indicated that most states either identify specific situations in which a paper version of the test might be used or require that a paper version be used when a student needs certain accommodations.

Overall, these findings suggest that accessibility continues to be an issue for computer-based testing. Although states have moved forward with CBT during the past decade, there remains a need for continued thinking about maximizing the meaningful participation of students with disabilities in computer-based testing. Suggestions made by Thompson et al. (2002) and by Dolan et al. (2009) continue to be appropriate for those considering CBT. In addition, structural considerations, such as those made by Russell and colleagues (2009) and by Mattson and Russell (2010) about the CBT platform and interoperability requirements need to be brought to bear on accessibility for students with disabilities in CBT.

Increasingly, states are exploring new ways to make CBT more sophisticated and interesting for students, including the development of simulations and increased interactivity. As the evolution of CBT continues, several considerations are relevant for those states and districts delving into—or continuing—computer-based testing that will be appropriate for all students, including students with disabilities:

1. Consider the assumptions and beliefs of various stakeholders about computer-based instruction and assessments.

CBT design and development often are based on assumptions that drive decisions about the nature of the assessment, the tools that are provided as part of the assessment, and the ways in which students may respond to the assessment. It is important to delineate assumptions that drive the decisions for CBTs, so that the appropriateness of the assumptions can be examined before they result in an inappropriate CBT. Assumptions and beliefs can be at different levels, from those that underlie the design of the entire system, to those that guide the ways in which individual students interact with CBT. An example of broader assumption is all students interact with computers in the same way, and that they are similarly engaged when a test is on a computer platform. An example of a specific assumption about how students interact with the computer is the assumption that the student must be able to use a keyboard; students with disabilities use a variety of assistive technology (such as sip and puff technology) to provide them access to computers even if they are not able to use a keyboard.

2. Consider the system as a whole, from the computer infrastructure to classroom and instructional experiences with computers before deciding whether and how to use CBT.

Computer-based testing occurs within the broader educational context, which includes the infrastructure of the building and classroom as well as the computer platform itself. It also occurs within a context in which teachers and students have had differential experiences with computers. Each of these, and potentially other factors, should be carefully analyzed as part of making decisions about CBT. Analysis of contextual factors and computer infrastructure and decision making about whether and how to use CBT must be influenced by the characteristics of all students, including those with disabilities. Thus, the capacity of school buildings and classrooms in a state or district must be checked in terms of the availability of computers that could be used for testing. This analysis needs to include schools and classrooms that may be designated to include only students with disabilities who meet eligibility requirements for the assessments to be administered via computers. Similarly, if assessments are to be Web based, schools and classrooms must be checked for connectivity to the Internet. This includes identifying issues that might emerge if every computer in the school is to be used at the same time for testing. Similar issues may result if a large number of computers in the state attempt to connect to an online platform on or around the same time.

Examining the system as a whole also entails assessing the extent to which students, including those with disabilities, have been using computers during instruction. Although recent surveys have determined that most schools have computer networks and plans for replacing computers (Gray et al., 2010), digital equity gaps have been identified for rural schools and schools with high percentages of African American students (Becker, 2006). No recent study has examined the current status of the equity gap, which was documented in the 1990s (Waddell, 1999) for students with disabilities. The concern that students with disabilities still may have less access to computers and fewer experiences in using them during instruction must be examined in every school, district, and state that is planning to use CBT.

The importance of having experiences with accommodations before they are used in the testing situation has been emphasized (e.g., Thurlow, Elliott, & Ysseldyke, 2003), and is required in most states’ accommodation policies (Christensen, Lazarus, Lail, Crone, & Thurlow, 2008). Although computer-based testing is not an accommodation, the same cautions apply. Without previous experiences and facility in using computers, it is possible that the introduction of the computer itself could depress the performance of students who have not have previously used a computer for instruction.

3. Consider the platform first, with input from individuals who know students with disabilities and their accessibility needs.

The platform for CBTs may be strictly designed for an offline computer, or may be designed for an online system. In both cases, it is essential to consider the specifications for the platform and how they potentially interact with accessibility requirements and with various assistive technology that students might use. Russell (2009) has addressed this issue directly by suggesting capabilities that should be incorporated in the platform to be maximally accessible. Similarly, Mattson and Russell (2010) specifically addressed how to meet both interoperability and accessibility requirements for CBTs, noting that codes (i.e., tags) used to specify elements of a test or test item could be also be identified to provide accessibility instructions. Mattson and Russell (2010) recommend that tags and specific behaviors need to be defined that are expected to occur when a given tag is applied in order to “standardize the behaviors that result when [tags] are applied” (p. 2).

4. Consider a process for bringing in the needed expertise to delineate the specific accessibility features of CBT, and to determine what specific accommodations may still be needed by students with disabilities, as well as to determine whether a computer-based test may create new accessibility issues.

A process is needed to identify specific accessibility features for a given computer-based test in a given content area, just as it is needed for item review (Johnstone et al., 2008). This process ideally would involve stakeholders with various perspectives, including those who know the content, those who know students with disabilities, and those who know how to address programming issues that accessibility issues may create. Consideration would be given to the possible needs of all students with disabilities, including those that may not be addressed through accessibility features included in the CBT, in light of the nature of the items and tasks included on the CBT. These considerations must take on challenging situations and issues, such as what happens for a student with significant visual impairments when simulations and other virtual scenarios are part of CBT. In addition, consideration must be given to those accessibility features that cannot be incorporated into the CBT, resulting in the need still for external accommodations.

5. Determine the policies for which accessibility features will be available to all students and which are designated for specific groups of students, such as students with disabilities.

As accessibility features are incorporated into the CBT, they may no longer be considered accommodations, but instead aspects of the assessment that are available to all students. Some of these features, on the other hand, may be perceived as appropriate only for students with disabilities, adding the requirement that they be turned on or off depending on who the student is taking the assessment. Careful consideration must be given to decisions about what accessibility really means and who it is for. These decisions, like other accessibility decisions, need to be made before or early in the design and development process. In addition, decisions about which accommodations a student will use need to be data-based and include students’ perceptions about the usefulness of accommodations. Further, students need to have used an accommodation in instruction before using it on a test. This can be a particularly challenging requirement for some technology-based accommodations that are specific to a test platform.

6. Consider how to track the use of accessibility features incorporated into CBT design.

An important part of considering any accessibility or accommodation feature is knowing the extent to which students are using those features or accommodations that they should be using. CBT should make it possible to track the use of various features through the technology itself. Then, monitoring of the relationship between recommendations on a student’s Individualized Education Program (IEP) and what actually occurs during the assessment (Christensen, Lail, & Thurlow, 2007; Christensen, Thurlow, & Wang, 2009) will be possible. States could track how students used the various options and accommodations as well as capture student input to improve the accommodations decision-making process for individual students.

7. Field test the accessibility features of the computer-based test at the same time that the computer-based test is field tested.

Accessibility features should be considered part of the design of CBT, and thus should be included in all pilot and field tests to which CBT is subjected. At the same time, efforts must be made to ensure that the full range of students is included in the field testing so that accessibility features can be adequately tested.

8. Examine results from CBT for students with disabilities to determine whether there are any features or characteristics of the assessment that might need reconsideration.

Results from field testing and full test administration should be continually examined to determine whether any accessibility features or other characteristics of the assessment are not working as intended. This will allow test developers to determine whether there is a need to obtain additional stakeholder information, conduct cognitive labs with students who are using the accessibility features, or simply revise the features.

9. Develop training for teachers and students to ensure that students benefit from accessibility features.

Just as there is a need for training for teachers and students about accommodations, there is a need for training related to most accessibility features. Students who have not experienced text- to-speech as presented during the assessment (e.g., by phrase instead of by word, for example), need to have training to understand the working of that feature as it exists on the assessment. Similarly, teachers require training on accessibility features to know how they work so that assistance can be provided to students as needed.

Top of page | Table of Contents

Conclusions

For more than a decade CBT has been called the “new frontier of testing” (Thompson et al., 2002). CBT has matured and is no longer a new frontier—more than half of the states now offer at least one CBT and current federal initiatives strongly encourage the continued movement toward technology enhanced assessments. However, there is a risk that the use of technology could lead to tests that are not accessible to some students with disabilities and that do not validly measure student performance.

We now have a better understanding of both the benefits and challenges of CBT. Consideration of these can help guide a thoughtful approach to the development and implementation of CBT and potentially ensure they are accessible for all students, including students with disabilities. Careful planning enables states and districts to move forward in a way that capitalizes on all that we now know without making ambitious changes that may result in tests that are less accessible for some groups of students.

Top of pagea> | Table of Contents

References

Almond, P., Winter, P., Cameto, R., Russell, M., Sato, E., Clarke, J., Torres, C., Haertel, G., Dolan, B., Beddow, P., & Lazarus, S. (2010). Technology enabled and universally designed assessment: Considering access in measuring the achievement of students with disabilities—A foundation for research. Dover, NH: Measured Progress and Menlo Park, CA: SRI International.

Ash, K. (2008). States slow to embrace online testing. Education Week, 28(13), 21.

Becker, J. D. (2006). Digital equity in education: A multilevel examination of differences in and relationships between computer access, computer use and state-level technology policies. Education Policy Analysis Archives, 15(3), 1-38.

Bennett, R. E., Braswell, J., Oranje, A., Sandene, B., Kaplan, B., & Yan, F. (2008). Does it matter if I take my mathematics test on computer? A second empirical study of mode effects in NAEP. Journal of Technology, Learning, and Assessment, 6(9), 1-38.

Bridgeman, B., Lennon, M. L., & Jackenthal, A. (2001). Effects of screen size, screen resolution, and display rate on computer-based test performance. (ETS RR-01-23). Princeton NJ: Educational Testing Service. Retrieved from http://www.ets.org/Media/Research/pdf/RR-01-23-Bridgeman.pdf

CAST. (2009). UDL guidelines. Retrieved from http://www.cast.org/publications/UDLguidelines/

Cater, K., Rose, D., Thille, C., & Shaffer, D. (2010, June). Innovations in the classroom. Presentation at the Council of Chief State School Officers (CCSSO) National Conference on Student Assessment, Detroit MI.

Christensen, L. L., Lail, K. E., & Thurlow, M. L. (2007). Hints and tips for addressing accommodations issues for peer review. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Christensen, L. L., Lazarus, S. S., Lail, K., Crone, M., & Thurlow, M. L. (2008). 2007 state policies on assessment participation and accommodations for students with disabilities (Synthesis Report 69). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Christensen, L. L., Thurlow, M. L., & Wang, T. (2009). Improving accommodations outcomes: Monitoring instructional and assessment accommodations for students with disabilities. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Clapper, A. T., Morse, A. B., Thompson, S. J., Thurlow, M. L. (2005). Access assistants for state assessments: A study of state guidelines for scribes, readers, and sign language interpreters (Synthesis Report 58). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Department of Justice. (2010). ADA best practices tool kit for state and local governments, Chapter 3 - General effective communication requirements under Title II of the ADA. Retrieved from http://www.ada.gov/pcatoolkit/chap3toolkit.htm

Dolan, R. P., Burling, K. S., Harms, M., Beck, R., Hanna, E., Jude, J., Murray, E. A., Rose, D. H., & Way, W. (2009). Universal design for computer-based testing guidelines. Iowa City, IA: Pearson.

Dolan, R. P., Hall, T. E., Banerjee, M., Chun, E., & Strangman, N. (2005). Applying principles of universal design to test delivery: The effect of computer-based read-aloud on test performance of high school students with learning disabilities. Journal of Technology, Learning, and Assessment, 3(7), 1-31.

Erdogan, Y. (2008). Paper-based and computer-based concept mapping: The effects on computer achievement, computer anxiety and computer attitude. British Journal of Educational Technology, 40(5), 821-836.

Fletcher, G. H. (2004). Technology’s role in accountability and assessment. T.H.E. Journal, 31(9).

Gray, L., Thomas, N., & Lewis, L. (2010). Educational technology in U.S. public schools: Fall 2008 – First look (NCES 2010-034). Washington, DC: U.S. Department of Education. Retrieved from http://nces.ed.gov/pubs2010/2010034.pdf

Johnstone, C. J. (2003). Improving validity of large-scale tests: Universal design and student performance (Technical Report 37). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Johnstone, C. J., Altman, J., & Thurlow, M. (2006). A state guide to the development of universally designed assessments. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Johnstone, C., Altman, J., Thurlow, M., & Moore, M. (2006). Universal design online manual. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Johnstone, C., Liu, K., Altman, J., & Thurlow, M. (2007). Student think aloud reflections on comprehensible and readable assessment items: Perspectives on what does and does not make an item readable (Technical Report 48). Minneapolis, MN: National Center on Educational Outcomes.

Johnstone, C. J., Thompson, S. J., Miller, N. A., & Thurlow, M. L. (2008). Universal design and multi-method approaches to item review. Educational Measurement: Issues and Practice, 27(1), 25-36.

Johnstone, C. J., Thurlow, M. L., Thompson, S. J., & Clapper, A. T. (2008, Spring). The potential for multi-modal approaches to reading for students with disabilities as found in state reading standards. Journal of Disability Policy Studies, 18(4), 219-229.

Kamei-Hannan, C. (2008, May). Examining the accessibility of a computerized adapted test using assistive technology. Journal of Visual Impairment and Blindness, 261-271.

Ketterlin-Geller, L. R. (2005). Knowing what all students know: Procedures for developing universal design for assessment. The Journal of Technology, Learning, and Assessment, 4(2), 1-21.

Kettler, R., Scholz, C., Oderman, E., Hixon, N, & Weigert, S. (2010). Innovative uses of technology to support assessment decision-making and curricular alignment for students with disabilities. Presentation at the Council of Chief State School Officers (CCSSO).

Kingston, N. M. (2009). Comparability of computer- and paper-administered multiple-choice tests for K-12 populations: A synthesis. Applied Measurement in Education, 22(1), 22-27.

Luecht, R. M. (2005). Some useful cost-benefit criteria for evaluating computer-based test delivery models and systems. Journal of Applied Testing Technology, 7(2), 1-31.

Mattson, D., & Russell, M. (2010). Meeting interoperability and accessibility requirements by adopting the accessible portable item profile (APIP) standards (White Paper). St. Paul, MN: Minnesota Department of Education.

McGroroy, K., & Sampson, H. (2010, June 7). Glitches delay Florida test score results. Miami Herald. Published online.

McHenry, B., Griffith, L., & McHenry, J. (2004). The potential, pitfalls and promise of computerized testing. T.H.E. Journal, 31(9), 28.

Meyen, E., Poggio, J., Seok, S., & Smith, S. (2006). Equity for students with high-incidence disabilities in statewide assessments: A technology-based solution. Focus on Exceptional Children, 38(7), 1-8.

Olson, J. F., & Dirir, M. (2010). Technical report for studies of the validity of test results for test accommodations. Washington DC: Council of Chief State School Officers.

“Overview information; Race to the Top Fund Assessment Program; Notice inviting applications for new awards for fiscal year (FY) 2010.” 75 Fed. Reg. 18182 (2010, April 9). http://www.gpo.gov/fdsys/pkg/FR-2010-04-09/pdf/2010-8176.pdf

“Race to the Top Fund.” 74 Fed. Reg. 54798 (2009, October 23). http://www.gpo.gov/fdsys/pkg/FR-2009-10-23/pdf/E9-25600.pdf

Russell, M. (1999).Testing on computers: A follow-up study comparing performance on computer and on paper. Education Policy Analysis Archives, 7(20).

Russell, M. (2009). Universal design of computer-based test. Boston: Boston College.

Russell, M., Almond, P., Higgins, J., Clarke-Midura, J., Johnstone, C., Bechard, S., & Fedorchak, G. (2010, June). Technology enabled assessments: Examining the potential for universal access and better measurement in achievement. Presentation at the Council of Chief State School Officers (CCSSO) National Conference on Student Assessment, Detroit MN.

Russell, M., Hoffman, R., & Higgins, J. (2009a). Nimble Tools: A universally designed test delivery system. Teaching Exceptional Children, 42(2), 6-12.

Russell, M., Hoffmann, T., & Higgins, J. (2009b). Meeting the needs of all students: A universal design approach to computer-based testing. Innovate: Journal of Online Education, 5(4).

Salend, S. J. (2009). Technology-based classroom assessments: Alternatives to testing. Teaching Exceptional Children, 41(6), 48-58.

Sawchuk, S. (2010). Open-ended test items pose challenges. Education Week, 29(19), pp. 1, 11.

Stock, S. E., Davies, D. K., & Wehmeyer, M. L. (2004). Internet-based multimedia tests and surveys for individuals with intellectual disabilities. Journal of Special Education Technology. 19(4), 43-47.

Thompson, S. J., Johnstone, C. J., & Thurlow, M. L. (2002). Universal design applied to large-scale assessments (Synthesis Report 44). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Thompson, S. J., Quenemoen, R. F., & Thurlow, M. L. (2006). Factors to consider in the design of inclusive online assessments. In M. Hricko (Ed.), Online assessment and measurement: foundations and challenges (pp. 102-117). Hershey, PA: Information Science Publishing.

Thompson, S. J., Thurlow, M. L., & Malouf, D. (2004, May). Creating better tests for everyone through universally designed assessments. Journal of Applied Testing Technology, 10(2). See http:// www.testpublishers.org/atp.journal.htm

Thompson, S. J., Thurlow, M. L., Quenemoen, R. F., & Lehr, C. A. (2002). Access to computer-based testing for students with disabilities (Synthesis Report 45). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.

Thurlow, M. L., Elliott, J. E., & Ysseldyke, J. E. (2003). Testing students with disabilities: Practical strategies for complying with district and state requirements (2nd ed.). Thousand Oaks, CA: Corwin.

 Thurlow, M., Johnstone, C., & Ketterlin-Geller, L. (2008). Universal design of assessment. In S. Burgstahler & R. Cory (Eds.), Universal design in post-secondary education: From principles to practice (pp. 73-81). Cambridge, MA: Harvard Education Press.

Thurlow, M., Johnstone, C., Thompson, S., & Case, B. (2008). Using universal design research and perspectives to increase the validity of scores on large-scale assessments. In R. C. Johnson & R. E. Mitchell (Eds.), Testing deaf students in an age of accountability (pp. 63-75). Washington, DC: Gallaudet University Press.

U.S. Department of Education. (2010). Application for new grants under the Individuals with Disabilities Education Act (IDEA): Technical assistance on state data collection—IDEA General Supervision Enhancement Grant—Alternate academic achievement standards (CFDA 84.373X). Washington, DC: Office of Special Education Programs.

Waddell, C. D. (1999). The growing digital divide in access for people with disabilities: Overcoming barriers to participation in the digital economy (White Paper for Understanding the digital economy: Data, tools and research). Washington, DC. National Science Foundation. Available at http://www.icdri.org/CynthiaW/the_digital_divide.htm

Top of page | Table of Contents

Appendix A

Advantages and Challenges of Computer-based Tests (CBTs)

Practitioners, state department of education staff, and researchers have identified many advantages and challenges of CBTs. We identified several categories or “themes” that can be used to organize the advantages and challenges:

• Economic: Factors that have cost implications.

• Systems implementation: Logistical, test security, and other factors that affect the capacity of LEAs and SEAs to implement CBTs.

• Test administration/design: Factors that affect how students (and sometimes teachers) perceive and interact with the test.

• Accessibility: Factors that affect how accessible a test is to a wide range of students.

Many advantages and challenges of CBTs are not specific to students with disabilities.

In Tables A1–A4, we have listed advantages and challenges based on whether they affect all students or students with disabilities. This appendix ends with a brief discussion of advantages and challenges of CBT for ELLs with disabilities. The advantages and challenges identified in these tables should not be considered all-inclusive, and there probably are additional advantages and challenges that we have not identified.

Economic

Table A1 summarizes economic advantages and challenges. In the short run CBTs often cost more than paper and pencil tests because they are costly to develop and implement. Traditional tests require the printing and shipping of test booklets. Less paper is needed with CBT, and it may be greener and reduce the carbon footprint. However, at least in the short run, the carbon footprint may not be reduced. Many schools may not currently have enough computers and related infrastructure—and the manufacture and delivery of these items may increase the carbon footprint. Additional electricity will also be used. Costs may be shifted from State Education Agencies (SEAs) to Local Education Agencies (LEAs). SEAs generally are responsible for printing and shipping test booklets—but in some situations LEAs may be responsible for providing the technology that is needed to administer CBT. CBTs may cost less in the longer run than paper and pencil tests, but if technology continues to rapidly evolve cost savings may not occur in the foreseeable future (Luecht, 2005; Kettler, Scholz, Oderman, Hixon, & Weigert, 2010).

Table A1. Economic Advantages and Challenges of CBTs for All Students and for Students with Disabilities

Cater, Rose, Thille, and Shaffer (2010) maintain that students who participate in computer-based instruction and testing may be more prepared to compete in the global economy. They argue that today’s students need to know how to use technology to obtain good jobs, and teaching students to navigate and successfully complete online tests might help prepare them for the future.

Students with Disabilities. It is costly for LEAs to provide some accommodations; and SEAs and LEAs expend resources keeping track of the accommodations that students receive on test day. There can be cost savings when many accommodations—including accommodations that may have been provided by a human (i.e., read aloud, scribe, sign interpret, etc.) for a paper and pencil test— are incorporated into CBTs (Russell, Almond, Higgins, Clarke-Midura, Johnstone, Bechard, & Fedorchak, 2010).

Systems Implementation

Table A2 summarizes the systems implementation factors. With paper and pencil tests, materials need to be distributed to—and collected from—each administration site. Additional test administrators and extra rooms may be needed. However, CBTs may create their own set of logistical issues: Are there enough computers available? Are there technology experts available to assist with any computer problems? How will computer labs—or other test administration sites—be set up to minimize opportunities for cheating? Have students had adequate opportunities to learn how to use computers? (Luecht, 2005; McGrory & Sampson, 2010, McHenry, Griffith & McHenry, 2004). Historically there have also been equity issues related to student access to computers. Becker (2006) found that rural and African American students were less likely to have access to computers than other students.

Table A2. Systems Implementation Advantages and Challenges of CBTs for All Students and Students with Disabilities

Currently some schools do not have enough computers for all students to take large-scale tests within a reasonable time frame. Expanding the test window, or in some cases on-demand testing, can help resolve this challenge, but it ties up computer labs for a longer period (Ash, 2008). This characteristic may be particularly relevant for retests and for end-of-course tests where there may be a preference for administering the test soon after the completion of the course (Russell et al., 2010).

Also, SEAs and LEAs sometimes lack related infrastructure. For example, according to Ash (2008), there is a need for “beefed-up network capacity to handle the spike in bandwidth for the assessment applications to run smoothly” (p. 21). Students who take a test that is interrupted by technology issues (e.g., loss of Internet connection, hardware or software issues) tend to perform less well on a test when the test administrator attempts to restart the student where they stopped or allow the student to start over (Bennett, Braswell, Oranje, Sandene, Kaplan, & Yan, 2008).

CBT may help ensure that data are more accurately collected, and are easier to store. Responses generally are accurately captured and scored with a CBT. For example, on paper and pencil tests students sometimes make stray marks or messy erasures that can result in inaccurate scoring of an exam. It may also be easier to change a test if mistakes are discovered after it has been finalized (Luecht, 2005; McGroroy & Sampson, 2010; McHenry et al., 2004). Still, test contractors continue to have major problems with scoring and data retrieval. For example, the June 7, 2010 online issue of the Miami Herald contained an article entitled “Glitches Delay Florida Test Score Report” that discussed problems with several CBTs (and other tests) including system-wide technology glitches and students who could not log in. The article quoted Florida K-12 Chancellor Frances Haithcock as saying, “Accuracy must at all times trump expediency, especially given the importance of these results” (McGroroy & Sampson, 2010).

An important potential advantage of CBT at the systems level may include the shifting of focus from assessment to instruction. Currently some educators and others express frustration that assessment is driving instruction. CBTs that are closely aligned with instructional methods may have the potential to move the focus back to instruction (Kettler et al., 2010; Meyen, Poggio, Seok, & Smith, 2006).

Classroom instruction time may be needed to train students how to navigate a computer-based test and how to use test tools. Different tests have different platforms—and students need to learn test-specific computer navigation skills. Also, without prior training students often are unable to correctly use online rulers, protractors, and other online measurement tools. Teachers need to plan lessons and provide instruction so that students know how to use the specific features and tools incorporated into a computer-based test. States and test vendors sometimes fail to make practice tests and manuals available far enough ahead of test day to provide sufficient instructional time. The school day is already packed and decisions need to be about what should be eliminated from the curriculum to find time to teach students these skills (Kettler et al., 2010; Russell et al., 2010; Thompson et al., 2002).

Training is needed not only for test administrators, but also for LEA technology staff. For example, computers may need to have specific software downloaded prior to test day. Also, problems with computers on the day of the test can cause significant disruption and potentially cause additional anxiety in staff and students; technology staff need training so that they will be prepared resolve potential issues that might emerge on test day (Ash, 2008; Thompson et al., 2002).

Students with Disabilities. Occasionally school district staff may not know ahead of time that a particular special format test (e.g., large print, braille, read aloud package, etc.) will be needed—for example, if a student with visual impairments enrolls in a school a few days before test day. Schools and districts may find CBT to be more convenient because test formats for students with disabilities can be customized on test day, and there is no need to order special formats far in advance (Kettler et al., 2010; Russell et al., 2010).

Some students with disabilities are not in fully inclusive settings and may have fewer opportunities to learn and practice keyboarding skills, computer navigation skills, and online tools. And, some students with physical or cognitive disabilities may have particular challenges to become proficient keyboarders (Ketterlin-Geller, 2005).

Test Administration and Design

Table A3 summarizes test administration/design advantages and disadvantages. Some computer-based tests may have the potential to more authentically assess student learning than paper-based tests—though this is a more a function of test design than just the test platform (Fletcher, 2004; Thompson et al., 2002). Innovative CBT formats that may become more common in the future include role playing, simulations, and data manipulations (Russell et al. 2010).

Another advantage of CBT may be student preference. Some students have used computers to play games and they may receive some of their instruction via computers. Students may also prefer CBT because it has the option of customizing the assessment based on personal preferences. For example, all students may be allowed to decide what background color they would like on the screen, or what font size they would prefer. Although some students may prefer CBT, others may prefer paper and pencil tests (Cater et al., 2010; Russell et al., 2010). Interactions with computers may cause anxiety for some students. Computer anxiety does not refer to negative attitudes towards computers, but rather to a student’s “emotional reaction to using computers” (Erdogan, 2008, p. 823). Students with weak computer skills (for example, students who have difficulty with scrolling) are more likely to experience test anxiety (Bridgeman, Lennon & Jackenthal, 2001). Erdogan asserts that providing students with more opportunities to use computers during instruction has the potential to reduce computer anxiety.

Educators and parents often are frustrated by how long it takes to receive test results. Teachers often need timely results for instructional decision-making purposes. Generally with CBT, multiple choice items can be scored with a very short turn-around (Ash, 2008; Kingston, 2009). For both computer-based tests and paper-based tests, there generally is a time lag in scoring constructed response items. Currently many states with CBT hand score open-ended items (often using teachers)—although “test publishers are investing heavily in automated-response systems that use artificial-intelligence software to better ‘read’ student answers” (Sawchuk, 2010, p. 1). According to Sawchuk, these systems would distance teachers from scoring assessments which can be a valuable professional development activity.

Table A3. Test Administration/Design Advantages and Challenges of CBTs for All Students and Students with Disabilities

Some computer-based tests are computer adaptive tests (CATs) that tailor the test items that a student receives based on his or her correct or incorrect responses to previous items—and the resulting test is generally shorter than a non-adaptive test that is designed to provide similar information (Meyen, Poggio, Seok, & Smith, 2006). Care must be used to ensure that these types of CBTs appropriately assess all students. An adaptive test must be well designed to ensure that students who miss questions early in the test are not denied the opportunity to answer challenging questions. With some adaptive test designs some students who have gaps in their knowledge—including some students with disabilities—may only have the opportunity to answer low level questions. For example, a student may not be able to correctly answer a computation problem but could respond correctly to a problem that requires the use of higher order skills; the algorithms for many computer-based tests currently are not designed in ways that ensure that such students would have the opportunity to respond to those challenging questions (Almond, Winter, Cameto, Russell, Sato, Clarke, Torres, Haertel, Dolan, Beddow, & Lazarus, 2010; Thompson et al. 2002).

Students with Disabilities. Some accommodations can be delivered more consistently with a computer-based test. For example, for a paper-based test it can be problematic to consistently provide some accommodations that involve a human access assistant to administer (e.g., reader, sign language interpreter, scribe). The quality of human readers varies greatly from one individual to another. Some readers may mispronounce words, use misleading intonation, or influence student responses. With the read aloud accommodation some human readers may use voice tone to suggest the correct response—whereas others would not (Clapper, Morse, Thompson, & Thurlow, 2005). An advantage of CBT is that technology can consistently provide the read aloud accommodation (Dolan, Hall, Banerjee, Chun, & Strangman, 2005) although computer-based tests may use a digitalized or human read aloud/text-to-speech voice that students find difficult to understand. The voice may have unusual ways of pronouncing words or a different accent from what the student is used to (Olson & Dirir, 2010). The text-to-speech function may also express terminology or terms in a different way from instruction. Math and science terminology are particularly problematic. For example, if a science test item refers to H2O should it be read as “H 2 O” or as “water”? Christopher Johnstone conducted a think aloud study and found a clear student preference for “water” rather than “H 2 O” but that may affect the difficulty of the test item (Russell et al., 2010).

In a small study focused on individuals with intellectual disabilities, Stock, Davies, and Wehmeyer (2004) found that many study participants preferred computer-based tests. The participants particularly liked being able to take the test with little assistance. Still, when students have the opportunity to self-select accommodations on a computer-based test, they sometimes make poor decisions. Requiring the pre-selection of accommodations prior to test day can help mitigate this issue. Also, some students with disabilities may find innovative test formats challenging. Innovative formats may be especially challenging for students with visual impairments or poor fine motor skills. For example, it is difficult to braille innovative test items. It is also sometimes difficult to describe some online graphics without giving the answer away (Kamei-Hannan, 2008; Kettler et al., 2010; Russell et al., 2010; Thompson et al., 2002).

As noted previously, if a computer-based test is used, it can be difficult to ensure that students with gaps in their knowledge have the opportunity to show what they know. This can be particularly problematic for some students with disabilities. As a result of their disability some may lack lower level skills and knowledge, yet know more complex information and have higher order knowledge. Many CATs can fail to provide these students with opportunities to show what they know (Thompson et al., 2002).

Accessibility

Table A4 summarizes accessibility factors that we identified. Well-designed, more universally designed assessments benefit all students—including students with disabilities. According to Ketterlin-Geller (2005), “computer-based technology presents an efficient tool for customizing assessments to meet individual needs within a universally designed environment” (p. 5). For example, all students would benefit if a computer-based test has allowable features that make the test easier to understand and navigate. Computer-based tests often allow many options for interacting with the assessment, which fits well with the concept of universal design. (Dolan et al., 2005; Russell, Hoffman, & Higgins, 2009b). However, younger students, as well as some other students, may have never learned how to use a keyboard or a mouse. Some early elementary grade students also may lack the hand size and fine motor skills to successfully use a keyboard. Students with weaker keyboarding skills tend to do less well on CBTs than other students—though this effect is diminished as keyboarding skills improve (Russell, 1999).

Table A4. Accessibility Advantages and Challenges of CBTs for All Students and Students with Disabilities

Students with Disabilities. Some test design changes (font size, read aloud, screen background color, etc.) may be incorporated into the design of some CBTs. These embedded resources can be used to create customized, more universally designed tests for students who need an accommodation to meaningfully access the test. It is much easier to provide some accommodations based on individual student needs with a CBT than with a paper and pencil test (Ketterlin-Geller, 2005; Russell et al. 2009b; Thompson et al., 2002). On a CBT some of these features may no longer be considered an accommodation, but instead an allowable resource that all students may use (Almond, et al. 2010).

Students with some visual impairments may find CBT to be more accessible than paper and pencil testing because the font size can be adjusted easily. However, for students with more severe visual impairments or blindness, a CBT may be less accessible than a paper and pencil test. And, as previously discussed, braille is a challenge for current CBTs—especially with innovative items (Russell et al., 2010). Similar examples could be given for students with disabilities who have other characteristics. CBTs can create challenges for students who need more concrete representations because they often use new ways of representing information, and they may require the use of online tools that are visual representations (Russell et al., 2010; Thompson, et al., 2002).

Some CBTs may require more working memory than similar paper and pencil tests. There are cognitive differences in how students interact with CBTs and paper and pencil tests. For example, for a reading passage, less text typically appears on the screen at any one time than on a page in a test booklet. Students who need to scroll through a passage may miss or forget key information that is needed to answer an item. CBTs that minimize the need for scrolling or allow students to highlight text can help mitigate this issue (Kingston, 2009).

Delivery systems can be an issue if a student does not have access to an assistive technology delivery device that he or she normally uses. For test security, or other reasons, computer applications are sometimes automatically turned off on CBTs. For example, a student may normally use an accessible mouse or a spellchecker that is not allowed on a CBT. Bennett et al. (2008) found that student performance is negatively affected when students are not allowed to use a familiar computer.

ELLs with Disabilities

Many of the same economic, systems implementation, test administration/design, and accessibility factors noted for students with disabilities in general are applicable—and sometimes even more of an issue—for ELLs with disabilities. For example, a digitalized or human read aloud/text-to-speech voice using an unfamiliar dialect can present particular challenges for ELLs with disabilities because dialects may not be as easily perceived by ELLs if the computer voice speaks with a different dialect from the one that the student hears every day. CBTs may also have benefits that are specific to ELLs with disabilities. For example, a CBT could offer an ELL with disabilities (as well as other ELLs) more levels of glosses than a paper and pencil test. A CBT could give a picture of an item, a glossed word tailored to a student’s language background, or other options.

Top of page | Table of Contents

Appendix B

Computer-based Tests: Specifications and Details

Top of page | Table of Contents

Appendix C

Web Sites used in Analysis of States’ Computer-based Tests

Top of page