April 2003

The present study is a survey of the perceptions held by people familiar with policy or research on the way in which test scores are influenced by accommodations and how scores obtained under accommodated conditions are to be treated in reporting. The results show that the extent of agreement about how accommodated scores should be treated depends on the accommodation. The study also shows how deep-seated beliefs lead some respondents to consider almost no accommodation as changing the construct, whereas other respondents consider almost all accommodations as influencing the construct being measured.

Policies intended to increase the participation of students with disabilities in state and local assessment systems have been in full force for several years. Test accommodations constitute the most frequently used alternative to increase participation rates of these students. The widespread use of test accommodations has spawned a flurry of empirical studies to explore questions such as what accommodations should be used and with whom. It will be some time before answers to these questions are sufficiently refined to enable strong conclusions about the benefits and drawbacks of test accommodations. In the meantime the use of accommodations flourishes (American Council on Education, 2002; Thompson & Thurlow, 2001). Because accommodations continue to be so widely applied despite the limited amount of empirical research available demonstrating how they affect test scores (Thompson, Blount, & Thurlow, 2002), it is necessary that sound, rational decisions be made about the usability of accommodated test scores.

One challenge confronting state education agencies is to determine the most appropriate way of reporting test scores for students receiving accommodations. There are three general options:

1. Report all score in the aggregate (i.e., do not differentiate between accommodated and non-accommodated test scores)

2. Report accommodated scores separately

3. Report accommodated scores both in the aggregate as well as separately

Each option reflects different beliefs about how accommodations influence test scores. Option 1 implies that the accommodated test scores measure the same construct in the same way as non-accommodated test scores. Option 2 implies that the accommodation changes the meaning of the test score, therefore the scores must be considered separately. Option 3 implies uncertainty about how the accommodation influences test scores, if at all; the third option reflects the reality of test accommodations research – we simply do not have definitive evidence about how each accommodation or combination of accommodations influences test scores. Options 1 and 2 may represent personal biases more than definitive empirical evidence.

There is some evidence that the opinions of people who are familiar with this issue vary, even to the point that the opinions are in direct opposition. State guidelines about how to report accommodated test scores provide evidence of this. The lists of approved and non-approved accommodations in each state show that an accommodation that is approved in one state may not be approved in another state, even when the same assessment is used (Thurlow, House, Boys, Scott, Ysseldyke, 2000; Thurlow, Thompson, Lazarus, & Robey, 2002). Federal regulations that require states to report test scores for students with disabilities in the aggregate and separately, in combination with the high stakes placed on the scores, makes this reporting issue particularly salient. Furthermore, the conflict between what measurement theory regards as essential for test score comparisons and the provision that any-and-all accommodations must be made available to students with a disability compounds the issue (Heumann & Warlick, 2000), raising tensions and uncertainty.

The future of accommodations research depends, in part, on the perceived need for the research as well as continued availability of the resources to conduct such research. The opinions of the stakeholders, particularly those influencing policy on how to report scores from accommodated tests, may provide a barometer of the perceived need for further research. For instance, one would expect little perceived need if all of the people influencing policy decisions shared the same opinion on how to treat test scores obtained under non-standard conditions. On the other hand, if the opinions of this stakeholder group varied, then a need for more research, or at least more discussion of the issues, would be indicated. The extent of need for more research likely varies by the type of accommodation. Those accommodations on which there is little agreement about perceived effects on test score interpretation deserve most of our attention.

The present study is a survey of the perceptions held by people familiar with policy or research on the way in which test scores are influenced by accommodations and how scores obtained under accommodated conditions are to be treated. Rather than asking participants directly about how they believe an accommodation influences test score interpretation, the study asked participants to classify accommodations into one of three categories that can be distinguished by the degree to which accommodations influence performance using a classification scheme developed by CTB/McGraw-Hill.

In an effort to create guidelines for using test results from standardized tests administered under non-standard conditions, CTB/McGraw-Hill created a framework for classifying accommodations (CTB/McGraw-Hill, 2000). Accommodations were framed according to their expected influence on student performance, and then according to how the results should be reported. Category 1 accommodations are not expected to influence test performance in a way that would alter the characteristics of the test. According to the CTB/McGraw-Hill document, test scores for students receiving such accommodations should be interpreted as test scores from standard administrations, and these scores should be aggregated with the scores of standard administrations. According to CTB/McGraw-Hill, Category 2 accommodations are expected to have some influence on test performance, but should not alter the construct the test was designed to measure. Category 2 accommodations may boost test performance; therefore, the type of accommodation used should be considered when interpreting the test scores. Scores obtained under Category 2 accommodations can be aggregated with scores obtained under standard conditions, but the scores should also be reported separately and the number and percent of students using such accommodations should be clearly indicated along with summary statistics. Category 3 accommodations, as classified by CTB/McGraw-Hill, are expected to alter the construct that the test was designed to measure. In the absence of research demonstrating otherwise, scores obtained under Category 3 accommodations should be interpreted in light of how the accommodation is thought to influence performance. Some of the Category 3 accommodations are content specific, for example receiving the read-aloud accommodation on a reading test, or using a calculator on math computation items. Score interpretation should consider the accommodation-content combination and whether the accommodation changes what the tests were designed to measure. According to CTB/McGraw-Hill, scores from Category 3 accommodations should be reported in aggregated and disaggregated forms, and the number and percent of students using such accommodations should be clearly indicated along with summary statistics.

Using the three categories of accommodations, a survey was created in which participants were asked to assign each of 44 accommodations to one of the three categories. The categories were designed to be mutually exclusive, but they might not have been exhaustive.

Participants chosen for this study were familiar with accommodations research or state policies on the use of test accommodations. A survey was sent to each of the 50 state assessment directors, each state special education director, and to individuals who have presented research on test accommodations or have published accommodations research. One hundred and thirty surveys were mailed initially, and also re-mailed to those who had not responded to the first mailing. In all, we obtained responses from 86 individuals (66% of those sent). Of these, 63 (73%) provided a single rating for each accommodation, and 77 (89%) provided a single rating for at least 40 of the 44 accommodations.

Of the 86 respondents, 60 were state department of education personnel, either assessment directors or special education directors. Eleven respondents were involved in accommodations research or in drafting policy guidelines on the use of accommodations. The other 11 respondents were practitioners or described themselves by checking multiple categories.

The accommodations used in this survey were chosen to be representative of the accommodations used in practice. This list of 44 accommodations was not meant to be exhaustive. A popular classification scheme was used to cluster the accommodations and to ensure that these accommodations represented different aspects of test administration that could be accommodated (Thurlow, Ysseldyke, & Silverstein, 1993). The four categories were: (1) presentation, (2) response, (3) setting, and (4) timing. According to this scheme, accommodations can be distinguished by that aspect of the standard administration that is altered by the accommodation. For instance, presentation accommodations represent accommodations that alter the standard presentation of the test - presenting test material in Braille is a common example of a presentation accommodation. Response accommodations alter the way in which examinees respond to test items - marking the answer in the test booklet as opposed to a bubble sheet would be an example of a response accommodation. Setting accommodations usually refer to changes in the typical size of the group to which the test is administered or the location the test is taken - taking the test in a small group is an example of a setting accommodation. Timing accommodations typically refer to allowing the examinee extra time to complete the test. There were 20 presentation accommodations, 14 response accommodations, 5 setting accommodations, and 5 timing accommodations (see Appendix A).

The categories into which respondents placed each of the 44 accommodations in the CTB/McGraw-Hill list were examined by creating frequency distributions. These were plotted as bar graphs according to the four category classification scheme (presentation, response, setting, timing).

Figure 1 displays the results for the 20 presentation accommodations. As is evident in the bar graph, there was little variability in the classification of the first four presentation accommodations (visual magnification, large print, audio amplification, and place markers); most of the respondents (over 90%) chose Category 1 for these four accommodations. The next four accommodations all represent ways of presenting test directions (read aloud, audio, signed, and highlighted). At least 70% of the respondents also chose Category 1 for these accommodations. The CTB/McGraw-Hill category chosen by respondents for the remaining presentation accommodations varied much more. Most respondents classified items read aloud, audio items, communication device, and computer presentation into either Category 1 or Category 2. More than 50% of the respondents classified accommodations representing oral presentation of the reading test and providing a calculator for a math computation test as Category 3.

Figure 2 displays the results for the 14 response accommodations. More than 85% of the respondents placed responding in the test booklet, large print, using a template, and using graph paper into Category 1. There was little agreement as how to treat the response accommodations like scribes and spell checkers used when spelling was not scored. The use of a spell checker when spelling was scored was more consistently categorized by respondents into Category 3. Still, some respondents did place this accommodation into Category 2, and some placed it in Category 1.

Figure 3 displays the distribution of responses to the setting accommodations. Respondents almost unanimously agreed that taking a test alone or in a small group, or the use of adaptive furniture or special lighting or acoustics did not alter the meaning of the score, and therefore could be placed in Category 1, where scores should simply be reported in the aggregate as though they are standard scores. The accommodation of taking the test at home or in a care facility received less consistent placement into Category 1. Roughly one-third of the respondents placed this accommodation into Category 2.

Figure 4 displays the distribution of responses to the timing accommodations. Respondents unanimously agreed that the first two accommodations (additional breaks and flexible scheduling) do not alter the meaning of test scores and therefore belong in Category 1. These represent scheduling accommodations that do not result in extra testing time. The timing accommodation of taking the test over several days, but still not resulting in extra time, showed greater variability in responses. About 55% of the respondents placed this accommodation into Category 1, 35% placed it into Category 2, and 15% placed it into Category 3. The majority of respondents placed the timing accommodations of extra time on a timed test and extra breaks on a timed test into either Category 2 or Category 3.

Table 1 is a count of the accommodations by level of agreement. Low agreement was defined as less than 50% of the respondents placing the accommodation into the same category, moderate agreement was defined as 50 to 89% of respondents choosing the same category, and high agreement was defined as 90% or more choosing a particular category for an accommodation. There was low agreement on 14 of the accommodations, moderate agreement on 17, and high agreement on 13 of the accommodations. Specific accommodations by level of agreement among participants can be found in Appendix B.