Does the item…

Measure what it intends to measure

•   Reflect the intended content standards (reviewers have information about the content being measured)

•   Minimize skills required beyond those being measured

Respect the diversity of the assessment population

•   Accessible to test takers (consider gender, age, ethnicity, socio-economic level)

•   Avoid content that might unfairly advantage or disadvantage any student subgroup

Have clear format for text

•   Standard typeface

•   Twelve (12) point minimum for all print, including captions, footnotes, and graphs (type size appropriate for age group)

•   Wide spacing between letters, words, and lines

•   High contrast between color of text and background

•   Sufficient blank space (leading) between lines of text

•   Staggered right margins (no right justification)

Have clear pictures and graphics (when essential to item)

•   Pictures are needed to respond to item

•   Pictures with clearly defined features

•   Dark lines (minimum use of gray scale and shading)

•   Sufficient contrast between colors

•   Color is not relied on to convey important information or distinctions

•   Pictures and graphs are labeled

Have concise and readable text

•   Commonly used words

•   Vocabulary appropriate for grade level

•   Minimum use of unnecessary words

•   Idioms avoided unless idiomatic speech is being measured

•   Technical terms and abbreviations avoided (or defined) if not related to the content being measured

•   Sentence complexity is appropriate for grade level

•   Question to be answered is clearly identifiable

Allow changes to its format without changing its meaning or difficulty (including visual or memory load)

•   Allows for the use of braille or other tactile format

•   Allows for signing to a student

•   Allows for the use of oral presentation to a student

•   Allows for the use of assistive technology

•   Allows for translation into another language

Does the test…

Have an overall appearance that is clean and organized

•   All images, pictures, and text provide information necessary to respond to the item

•   Information is organized in a manner consistent with an academic English framework with a left-right, top-bottom flow

In addition to the other considerations, a computer-based test should have these considerations:

Layout and design

•   Sufficient contrast between background and text and graphics for easy readability

•   Color is not relied on to convey important information or distinctions

•   Font size and color scheme can be easily modified (through browser settings, style sheets, or on-screen options)

•   Stimulus and response options are viewable on one screen when possible

•   Page layout is consistent throughout the test

•   Computer interfaces follow Section 508 guidelines

Navigation

•   Navigation is clear and intuitive; it makes sense and is easy to figure out

•   Navigation and response selection is possible by mouse click or keyboard

•   Option to return to items and return to place in test after breaks

Screen reader considerations

•   Item is intelligible when read by a text/screen reader

•   Links make sense when read out of visual context (“go to the next question” rather than “click here”)

•   Non-text elements have a text equivalent or description

•   Tables are only used to contain data, and make sense when read by screen reader

Test specific options

•   Access to other functions is restricted (e.g., e-mail, Internet, instant messaging)

•   Pop up translations and definitions of key words/phrases are available if appropriate to the test

•   Students are able to record their responses and read them back (and have them read back using text-to-speech) as an alternative to a human scribe, but only if student has experiences with this mode of expression and chooses it for the test

Computer capabilities

•   Adjustable volume

•   Speech recognition available (to convert user’s speech to text)

•   Test is compatible with current screen reader software

•   Computer-based option to mask items or text (e.g., split screen)

•   Computer software for test delivery is designed to be amenable to assistive technology

Vision

Barbara Henderson

Computer-based testing, learning disabilities

Gerald Tindal

Item analysis

Karen Barton

Second language acquisition and testing

Margo Gottlieb

Second language acquisition, testing, and translation

Charles Stansfield

Physical disabilities

Sheryl Burgstahler

Hearing

Carol Traxler

Science

Scott Marion

Psychometrics

Tom Haladyna

Assistive technology

Tracy Hall

Math

Marge Petit

Special education assessment

Ken Olsen

State Assessment Director

Tim Vansickle

Does the item…

Range

Mean

Measure what it intends to measure

•   Reflect the intended content standards (reviewers have information about the content being measured)

•   Minimize knowledge and skills required beyond those being what is intended for measured measurement.

5–5

3–5

5.00

4.33

Respect the diversity of the assessment population

•   Accessible Sensitive to test takers characteristics and experiences (consider age, gender, ethnicity, and socio-economic level, region, disability, and language)

•   Avoid content that might unfairly advantage or disadvantage any student subgroup

4–5

 4–5

4.75

 4.64

Have clear format for text

•   Standard typeface

•   Twelve (12) point minimum size for all print, including captions, footnotes, and graphs (type size appropriate for age group)

•   Wide spacing between letters, words, and lines

•   High contrast between color of text and background

•   Sufficient blank space (leading) between lines of text

•   Staggered right margins (no right justification)

3–5

3–5

2–5

3–5

2–5

2–5

4.00

4.09

3.09

4.09

2.82

3.36

Have clear visuals (when essential to item)

•   Pictures Visuals are needed to respond to item answer the question

•   Pictures Visuals with clearly defined features (minimum use of gray scale and shading)

•   Dark lines (minimum use of gray scale and shading)

•   Sufficient contrast between colors

•   Color alone is not relied on to convey important information or distinctions

•   Pictures and graphs Visuals are labeled

3–5

4–5

3–5

1–5

2–5

3–5

4.56

4.45

3.82

3.64

3.91

3.91

Have concise and readable text

•   Commonly used words (except vocabulary being tested)

•   Vocabulary appropriate for grade level

•   Minimum use of unnecessary words

•   Idioms avoided unless idiomatic speech is being measured

•   Technical terms and abbreviations avoided (or defined) if not related to the content being measured

•   Sentence complexity is appropriate for grade level

•   Question to be answered is clearly identifiable

1–5

4–5

1–5

3–5

4–5

1–5

5–5

4.18

4.83

4.17

4.67

4.73

4.45

5.00

Allow changes to its format without changing its meaning or difficulty (including visual or memory load)

•   Allows for the use of braille or other tactile format

•   Allows for signing to a student

•   Allows for the use of oral presentation to a student

•   Allows for the use of assistive technology

•   Allows for translation into another language

3–5

3–5

3–5

3–5

1–5

4.67

4.55

4.36

4.45

3.64

Does the test…

Have an overall appearance that is clean and organized

•   All visuals (e.g., images, pictures) and text provide information necessary to respond to the item

•   Information is organized in a manner consistent with an academic English framework with a left-right, top-bottom flow

•   Booklets/materials can be easily handled with limited motor coordination (consideration was added)

•   Response formats are easily correlated matched to question

•   Place for student to take notes (on the screen for CBT) or extra white space with paper-pencil

3–5

4–5

0–5

0–5

0–5

4.50

4.33

4.00

3.43

3.82

In addition to the other considerations, a computer-based test should have these considerations:

Layout and design

•   Sufficient contrast between background and text and graphics for easy readability

•   Color alone is not relied on to convey important information or distinctions

•   Font size and color scheme can be easily modified (through browser settings, style sheets or on-screen options)

•   Stimulus and response options are viewable on one screen when possible

•   Page layout is consistent throughout the test

•   Computer interfaces follow Section 508 guidelines (www.section508.gov)

Navigation

•   Students have received adequate training on use of test delivery system

•   Navigation is clear and intuitive; it makes sense and is easy to figure out

•   Navigation and response selection is possible by mouse click or keyboard

•   Option to return to items and return to place in test after breaks

Screen reader considerations

•   Item is intelligible when read by a text/screen reader

•   Links make sense when read out of visual context. (“go to the next question” rather than “click here”)

•   Non-text elements have a text equivalent or description

•   Tables are only used to contain data, and make sense when read by screen reader

4-5

 2–5

2–5

3–5

4–5

0–5

 0–5

4–5

3–5

3–5

3–5

4–5

3–5

3–5

4.67

 3.92

4.08

4.67

4.75

3.56

 4.46

4.92

4.67

4.60

4.58

4.67

4.30

4.36

Test specific options

•   Access to other functions is restricted (e.g., e-mail, Internet, instant messaging)

•   Pop up translations and definitions of key words/phrases are available if appropriate to the test

•   Students writing online can get feedback on length of writing on-demand in cases where there is a restriction on number of words. 

•   Students are able to record their responses and read them back (or have them read-back using text-to-speech) as alternative to human scribble, but only if student has experiences with this mode of expression and chooses it for the test as an alternative to human scribe.

•   Students are allowed to create persistent marks to the extent that they are already allowed to paper-based booklets (e.g., marking items for review, eliminating multiple choice items, etc.)

Computer capabilities

•   Adjustable volume

•   Speech recognition available (to convert user’s speech to text)

•   Test is compatible with current screen reader software

•   Computer-based option to mask items or text (e.g., split screen)

•   Computer software for test delivery is designed to be amenable to assistive technology

3–5

3–5

0–5

0–5

0–5

3–5

1–5

3–5

0–4

0–5

4.55

4.08

 2.67

3.69

4.17

 4.50

3.67

4.25

3.00

3.91

Considerations when reviewing any test item:

Subject Responses

Mean

Please insert
your comments here…

 Does the item…

Measure what it intends to measure

•   Reflects the intended content standards (reviewers have information about the content being measured)

•   Minimize skills required beyond those being measured

Respect the diversity of the assessment population

•   Accessible to test takers (consider age, gender, ethnicity, and socio-economic level)

•   Avoids content that might unfairly advantage or disadvantage any student subgroup

Have a clear format for text

•   Standard typeface

•   Type size appropriate for age group (12 point minimum for all print, including captions, footnotes, and graphs)

•   Wide spacing between letters, words, and lines

 •   High contrast between color of text and background

 •   Sufficient leading (blank space) between lines of text

 •   Staggered right margins (no right justification)

 Have clear pictures and graphics (when essential to item)

 •   Pictures are needed to respond to item

 •   Pictures with clearly defined features

 •   Dark lines (minimum use of gray scale and shading)

 •   Sufficient contrast between colors

 •   Color is not relied on to convey important information or distinctions

 •   Pictures and graphs are labeled

 Have concise and readable text

 •   Commonly used words

 •   Vocabulary appropriate for grade level

 •   Minimum use of unnecessary words

 •   Idioms avoided unless idiomatic speech is being measured

 •   Technical terms and abbreviations avoided (or defined) if not related to the content being measured

 •   Sentence complexity is appropriate for grade level

 •   Question to be answered is clearly identifiable

Allow changes to its format without changing its meaning or difficulty (including visual or memory load)

 •   Allows for the use of braille or other tactile format

 •   Allows for signing to a student

 •   Allows for the use of oral presentation to a student

 •   Allows for the use of assistive technology

 •   Allows for translation into another language

Does the test...

Have an overall appearance that is clean and organized

 •   All images, pictures, and text provide information necessary to respond to the item

 •   Information is organized in a manner that is consistent with an academic English framework with a left-right, top-bottom flow

In addition to the other considerations, a computer-based test should have these considerations:

Layout and design

 •   Sufficient contrast between background and text and graphics for easy readability

 •   Color is not relied on to convey important information or distinctions

 •   Font size and color scheme can be easily modified (through browser settings, style sheets, or on-screen options)

 •   Stimulus and response options are viewable on one screen when possible

 •   Page layout is consistent throughout the test

Navigation

 •   Navigation is clear and intuitive; it makes sense and is easy to figure out

 •   Navigation and response selection is possible by mouse click or keyboard

 •   Option to return to items and return to place in test after breaks

Screen reader considerations

 •   Item is intelligible when read by a text/screen reader

 •   Links make sense when read out of visual context. (“go to the next question” rather than “click here”)

 •   Non-text elements have a text equivalent or description

 •   Tables are only used to contain data and make sense when read by screen reader

Test specific options

 •   Access to other functions is restricted (e.g. e-mail, Internet, instant messaging)

 •   Pop up translations and definitions of key words/phrases are available if appropriate to the test

Computer capabilities

 •   Adjustable volume

 •   Speech recognition available (to convert user’s speech to text)

 •   Test is compatible with current screen reader software

Considerations when reviewing any test item:

Subject Responses

Mean

Please insert your comments here…

Does the item…

Measure what it intends to measure

1-Ultimately this (is the) most important. Some of this can be accomplished within the concept of universal design…there may be content, knowledge, skills, and/or abilities that may not lend themselves well to UD in some instances.

•   Reflects the intended content standards (reviewers have information about the content being measured)

55555555555

5

1-Agree

Should this be changed to read: “Reflects the intended construct (reviewers have information about the construct being measured)?

N/A

N/A

•   Minimize skills required beyond those being measured

334444555555

4.33

1-While I think you could argue for a “5” for the 2^nd statement here, I worry that if we try to separate skills too much, we’ll wind up with tests that measure isolated, basic skills.

2- Both cognitive demand and specific content should be included in the review.  Content to content match is insufficient (see Conserving Math Construct (CMC) template).

3-I would rather it said “Minimizes skills required beyond those explicitly in the standard.”

4-Very good observation.  Relevant to higher level thinking.

5-Multiple response options should be available, perhaps as a pretest assessment.

6-Important but cannot override ability to measure all content areas to assessed (e.g., draw a graph of the results).

7-Crucial for test validity for deaf test takers.

8-This may be difficult with disabled students—but try.

9-Perhaps should be reworded as “Minimize skills required beyond those intended for measurement.” Assure measurement of the intended construct. Assumptions about level of achievement possible/abilities (language, sensory, motor, background knowledge, etc.) can interfere with the ability to accurately measure the intended construct.

Based on these comments, would you change the wording of this consideration. If so, what should it say?

N/A

N/A

Respect the diversity of the assessment population

•   Accessible to test takers (consider age, gender, ethnicity, and socio-economic level)

444555555555

4.75

Should we add issues such as: Regional differences?  Students with disabilities? Language minority students?  Others?

Note: This section relates to what bias review committees typically address. Does the word “bias” need to be included?

N/A

N/A

•   Avoids content that might unfairly advantage or disadvantage any student subgroup

44445555555

4.64

Have a clear format for text

•   Standard typeface

33334445555

4

1-What is meant by “standard?” If you mean the same throughout a form,

then 1.  If you mean an acceptable typeface for clarity, then 4.

2-“Standard typeface” does not communicate in a world where everyone has access to hundreds of fonts—specify fonts and criteria re: serifs.

3-To do otherwise would introduce construct irrelevant variance (CIV).

4-Tests may be different between but should be the same within.

5-This should be “selection of typeface,” wherein the best proven typefaces are used.

6-General comment for all of these points: no one set of values will work for all students. All we can hope for is a reasonable compromise unless we embed flexibility, as per universal design principles. Perhaps explicitly state here that this consideration addresses print materials exclusively.

7-Serif recommended for print; sans-serif recommended for computer displays. For large-print booklets, consider specific fonts (see APH guidelines: http://sun1.aph.org/edresearch/lpguide.htm)

Instead of the word “standard” should we use the word “common,” “familiar,” or “clear?”  Please comment.

Would you change the wording entirely for the above consideration?  If so, how?

N/A

N/A

•   Twelve (12) point minimum for all print, including captions, footnotes, and graphs (type size appropriate for age group)

33444444555

4.09

Additional comments on print size?

N/A

N/A

•   Wide spacing between letters, words, and lines

22233333445

3.09

1-Dependent on age.

2-To do otherwise would introduce construct-irrelevant variance (CIV).

3-Wide spacing is not necessarily best; proper font selection is more important.

4-Need to define precisely. Also consider that wide spacing may have deleterious effect on non-visually impaired students.

Should we retain the above consideration?

N/A

N/A

•   High contrast between color of text and background

33344445555

4.09

1-Also is one needed on color intensity?

2-And selection of paper/ink color.

3-Even with sufficient color contrast, color blind users may not be able to distinguish text and background. Suggest you further recommend high print density contrast. This would also avoid isoluminance effects for non-visually-impaired students.

Additional comments on color and background?

•   Sufficient leading (blank space) between lines of text

23334444455

2.82

1-After having just had a conversation on this, I’ll up my rating to 3.

2-“Standard typographic leading (blank spaces) between lines of text.”

•   Staggered right margins (no right justification)

22333334455

3.36

Have clear pictures and graphics (when essential to item)

•   Pictures are needed to respond to item

335555555

4.56

1-Does this mean how important is it for there to be picture-based items?  In some speaking tests, pictoral prompts are often the best way of getting students to produce language (without giving them directions and thus affecting their responses).

2-Deaf examinees tend to focus on pictures when provided, more than on text.  A picture should be related to the correct response, not be distraction to “trick” certain students (more than others).

3-Pictures, line art, etc. should be related to the item, should enhance understanding not required for understanding with the exception of data tables like in math and science tests.

4-Perhaps reword as “only construct-relevant visuals.”

Should we change consideration to read: “When pictures are used, they should be redundant with the text as much as possible.”?

N/A

N/A

•   Pictures with clearly defined features

44444455555

4.45

1-As we noted previously, this seems like catering to the lowest common denominator. Might there be instances where grayscale and shading are most appropriate for providing construct-relevant information to the student? If so, then alternate representations would need to be considered for visually impaired students.

•   Dark lines (minimum use of gray scale and shading)

33334444455

3.82

1-Pictures with clearly defined features” covers this point. This now seems redundant and could be removed.

•   Sufficient contrast between colors

13333444555

3.64

1-Again, color may be a different and additional issue for computer based assessments.

2-“Color alone is not relied on to convey important information or distinctions.”

•   Color is not relied on to convey important information or distinctions

23334445555

3.91

1-This is perhaps too vague to be useful as a guideline.

•   Pictures and graphs are labeled

33334444555

3.91

Have concise and readable text

•   Commonly used words

12445555555

4.18

1-Unless it is a vocabulary test.

2-Commonly used words are not necessarily easy…they can carry multiple meanings.

3-What does this mean in a “content” assessment? 

4-Again phrasing is inconsistent.  This is not appropriate for vocabulary/language assessment.

5-If the construct is about unique words, then common words might make sense.

6-Important, but highly subjective.

7-That is, use common meaning, not low-frequency meaning for common words.

8-This can be collapsed into the second item: age level vocabulary.

9-“Commonly used words” seems dependent on the construct being measured, e.g., vocabulary instruction. Can this point be combined with the next one?

Should we re-word the above consideration?  If so, what should it say?

N/A

N/A

•   Vocabulary appropriate for grade level

445555555555

4.83

1-I think both this and the next bullet could easily be combined with the first.

2-Depends on what you are measuring…

•   Minimum use of unnecessary words

124445555555

4.17

•   Idioms avoided unless idiomatic speech is being measured

344555555555

4.67

•   Technical terms and abbreviations avoided (or defined) if not related to the constructs being measured

44455555555

4.73

•   Sentence complexity is appropriate for grade level

14455555555

4.45

•   Question to be answered is clearly identifiable

555555555555

5

1-Refer to the work and writing of Jamal Abedi at UCLA regarding language simplification as it affects testing of LEP students.  Very critical work and he has many concrete suggestions about item writing and presentation that are consistent with our AERA, APA, NCME Standards for Educational and Psychological Testing.

Allow changes to its format without changing its meaning or difficulty (including visual or memory load)

1-The change in meaning is going to be speculative until the research is done, and maybe not even then.

•   Allows for the use of braille or other tactile format

34455555555

4.67

1-Not all items can be brailled or made tactile in a meaningful way. Would this mean that such items cannot be used for any students? What’s wrong with having equivalent items that can be so modified?

•   Allows for signing to a student

33455555555

4.55

•   Allows for the use of oral presentation to a student

33345555555

4.36

•   Allows for the use of assistive technology

33445555555

4.45

•   Allows for translation into another language

12333355555

3.64

1-Validity may be compromised as bias is interjected.

2-Some critical and relevant words/concepts do not translate into every language.  I think we might have to narrow to a few core languages.

3-This is a questionable and highly controversial issue, particularly when one realizes that such a standard is impossible to meet.  About 72% of our LEP students are Spanish speakers, but the other 28% represent many diverse languages.  How do we accommodate and what is the theoretical rationale and what is the technology for doing this?  Is it possible?  Is it beneficial?

4-Most likely invalidates the test scores for high stakes decision, should probably assess in an alternate fashion that more validly assesses students’ ability, skill, knowledge, or learning.

5-A sticky wicket…

6-What about ELA tests? Also, what if the student prefers seeing the question in both their native language and the language of instruction? Should we deny them this opportunity?

7-There is the continual challenge of literal interpretation in translation, and limitations in expertise to translate to all needed languages. So, do students who speak Spanish receive interpreted tests, and those speaking a less well known language do not?

8-This issue will be one of the most difficult to overcome on a political and emotional level. Data analysis has shown these to be less than valid measures of student performance. Additionally, it has been suggested that it takes approximately 5 years for a student to become proficient in English.

Additional comments on this consideration?

N/A

N/A

Does the test...

Have an overall appearance that is clean and organized?

•   All images, pictures, and text provide information necessary to respond to the item

344445555555

4.5

1-Redundant with “pictures are needed to respond to item” and “minimum use of unnecessary words.”

2-I still rate this as a 5!

•   Information is organized in a manner that is consistent with an academic English framework with a left-right, top-bottom flow

444444445555

4.33

In addition to the other considerations, a computer-based test should have these considerations:

Layout and design

•   Sufficient contrast between background and text and graphics for easy readability

444455555555

4.67

1-“Sufficient luminance contrast between…”

•   Color is not relied on to convey important information or distinctions

233334455555

3.92

1-“Color alone is not relied…”

•   Font size and color scheme can be easily modified (through browser settings, style sheets, or on-screen options)

233344555555

4.08

•   Stimulus and response options are viewable on one screen when possible

344555555555

4.67

•   Page layout is consistent throughout the test

444555555555

4.75

Navigation

•   Navigation is clear and intuitive; it makes sense and is easy to figure out

455555555555

4.92

•   Navigation and response selection is possible by mouse click or keyboard

344555555555

4.67

1-“Navigation and response selection is possible by mouse (or equivalent) or keyboard (or equivalent).”

•   Option to return to items and return to place in test after breaks

3445555555

4.6

1-As before, returning to items will only work in non-adaptive tests.

Screen reader considerations

•   Item is intelligible when read by a text/screen reader

344455555555

4.58

•   Links make sense when read out of visual context (“go to the next question” rather than “click here”)

444455555555

4.67

•   Non-text elements have a text equivalent or description

3334555555

4.3

•   Tables are only used to contain data, and make sense when read by screen reader

34444455555

4.36

Test specific options

•   Access to other functions is restricted (e.g., e-mail, Internet, instant messaging)

33455555555

4.55

1-“Access to functions other than assistive technologies and supports is restricted (e.g., e-mail, Internet, instant messaging).”

•   Pop up translations and definitions of key words/phrases are available if appropriate to the test

333444445555

4.08

Computer capabilities

•   Adjustable volume

334455555555

4.5

1-“Adjustable volume and rate of voice.”

•   Speech recognition available (to convert user’s speech to text)

112334555555

3.67

•   Test is compatible with current screen reader software

334444455555

4.25

Below are new considerations suggested by panelists on the first survey, please rank each consideration and comment if you wish (numeric rankings found below were provided by participants in last survey).

Computer interfaces follow  Section 508 guidelines
(www.section508.gov)

Students have received adequate training on use of test delivery system

Students writing online can get feedback on length of writing on-demand in cases where there is a restriction on number of words

Students are able to record their responses and read them back (or have them read-back using text-to-speech) as alternative to human scribe, but only if student has experience with this mode of expression and chooses it for the test

Students are allowed to create persistent marks to the extent that they are already allowed to on paper-based booklets (e.g., marking items for review; eliminating multiple choice items, etc.)

Alternate versions of computer interface provided that is amenable to use with screen readers (e.g., JAWS, Window-Eyes)

When “modifying” items for use under a UD framework, there is convincing evidence that the item still measures the same or similar intended construct

4

Obviously, since I wrote these, I think they are all important.  I didn’t rate them as “5” because I think they are very hard to do, but I also think we need to be working in this direction.

A test constructed under a UD framework allows for the measurement of the same depth of knowledge levels as the “original” test

4

A UD test is aligned to the standards to the same extent as the “original” test

4

Test items are piloted, field tested, and normed on all subgroups for which the measure is designed

5

Booklets/materials can be easily handled with limited motor coordination

Response formats are easily correlated to question

Computer-based option to mask items or text (e.g., split screen)

4

Place for student to take notes (on the screen for CBT) or extra white space with paper-pencil

4

Computer software for test delivery is designed to be amenable to all assistive technology

5

Considerations when reviewing any test item:

Subject Responses

Mean

Please insert your comments here…

Does the item…

Measure what it intends to measure

1-Ultimately this (is the) most important. Some of this can be accomplished within the concept of universal design…there may be content, knowledge, skills, and/or abilities that may not lend themselves well to UD in some instances.

•   Reflects the intended content standards (reviewers have information about the content being measured)

555555555555

5

1-Agree

Should this be changed to read: “Reflects the intended construct (reviewers have information about the construct being measured)?

N/A

N/A

1-Yes.

2-Either way.

3-How about combining the 2 ideas? Reflects the intended construct that is aligned with representative (language proficiency or academic content) standards?

4-Construct is a formal term that theorists use. Content standards is what practitioners understand.

5-Yes.

6-Yes, Definitely.

7-No—but you might add something about the cognitive demand

8-This revision would alter the meaning. I think the construct is a sort of overarching concept (i.e., reading) whereas content standards are quite narrower (i.e., reproduces capital letters). It depends what you mean as to which one you should use.  If the test is supposed to be a standards-based achievement test, then it must address standards.  If no, then the item need only address the construct.

9-Probably content is the correct phrase…construct is typically Reading or Math, etc. The content is what reviewer typically know and evaluate.

10-No. I like better as it was.

11-Yes. That would fit better with the professional terminology.

12-Yes.

13-Yes, I believe it should be changed.  Content is topical, constructs are conceptual. This difference In meaning is huge. Furthermore, constructs is a term used in APA standards and is deeper than content.

•   Minimize skills required beyond those being measured

334444555555

4.33

1-While I think you could argue for a “5” for the 2^nd statement here, I worry that if we try to separate skills too much, we’ll wind up with tests that measure isolated, basic skills.

2-Both cognitive demand and specific content should be included in the review.  Content to content match is insufficient (see Conserving Math Construct [CMC] template).

3-I would rather it said “minimizes skills required beyond those explicitly in the standard.”

4-Very good observation.  Relevant to higher level thinking.

5-Multiple response options should be available, perhaps as a pretest assessment.

6-Important but cannot override ability to measure all content areas to assessed (e.g., draw a graph of the results).

7-Crucial for test validity for deaf test takers.

8-This may be difficult with disabled students—but try.

9-Perhaps should be reworded as “Minimize skills required beyond those intended for measurement.” Assure measurement of the intended construct. Assumptions about level of achievement possible/abilities (language, sensory, motor, background knowledge, etc.) can interfere with the ability to accurately measure the intended construct.

R1-How might the other skills be defined and targeted?  It seems if the first part is clear and clearly defined (that is the construct is being measured), this should not be as much of an issue. My fear is that folks will take this beyond content and eliminate all attractive, yet non-distracting qualities of the assessment.

R2-There may still be a limit in what can be made accessible and probably varies by which group is evaluated.

R3-I think the issue is about “construct irrelevant variance” and furthermore, the way it is stated it implies that skills are being measured. Major changes reflect the comments of #9 above.

Based on these comments, would you change the wording of this consideration.  If so, what should it say?

N/A

N/A

1-I could live with either statement 3 (if “standard” was made plural) or 9.

2-I’d probably make it a 4 instead of a 5.

3-Depends on the purpose of the test….a diagnostic test is more skills-driven; if standards-based, depends on how discrete the standards are; minimal skills can be embedded in more complex, contextualized items. Ultimately, it depends on what you are measuring.

4-I think the meaning of the phrase (as I interpret it) is test what you are supposed to test and not construct-irrelevant factors.

5-“Minimize skills required beyond those intended for measurement of a standard.”

6- Yes.  I would suggest changing the question/consideration to something like “does the item measure specific and carefully defined skills?”

7- The comments make me think that this consideration was interpreted very differently by the reviewers. Perhaps you should find out how this was interpreted. I interpreted this to mean: the items on the assessment should clearly stick to the domain being assessed. If that is the case, I give it a 5.

8-No change needed.

9-No.

10-I like reviewer comments 3 and 9 to get at the intended consideration.

11-I might say something like the following, “we understand that when measuring some skills, especially higher-order skills, it is difficult to focus the assessment items only on the explicitly targeted skills. However, please rate the importance of minimizing the skills required beyond those being measured.”

12-I would change it to something referencing “access” skills that are limited to input-output sensory modalities: minimize interference from sensory input and output skills needed to respond.”

Respect the diversity of the assessment population

•   Accessible to test takers (consider age, gender, ethnicity, and socio-economic level)

444555555555

4.75

Should we add issues such as: regional differences?  Students with disabilities? Language minority students?  Others?

Note:  This section relates to what bias review committees typically address.  Does the word “bias” need to be included?

N/A

N/A

1-Not necessarily, but change wording to “accessible to all test takers, regardless of such characteristics as…. Include a partial list. (Next to the Note: he wrote “No.”)

2-Yes, bias should be included.  I believe bias and accessib[ility] are separate issues from a review standpoint, though obviously related. For instance, “bias” in terms of the diverse sensitivity that is commonly reviewed in committees includes sensitivity towards terms that are sensitive to the deaf community–listen, hear, sounds, etc., and to the VI community–look, see, visualize, shadows, etc..  To review items from accessibility perspective requires a different perspective, beyond words or terms used in the item to the structure, layout, look, and feel of the item, first; and second, whether or not students have the intellectual capacity to access the items (opportunity to learn); and third, whether or not the test and items are able to access the information and measure the constructs that are intended.

3-Yes. I think there are two issues here: 1. bias and 2. subgroup representation. Every test is biased and yes, it needs to be recognized…UD hopefully serves to minimize its impact. English language learners are a subset of language minority students; different groups with different educational needs.

4-No.  Regional differences are construct-irrelevant in our world. I think the term “sensitivity” is a better term here.  Mike Zieky of ETS has written extensively on this issue. Jamal Abedi has written and researched the language issue. Bias is a different idea and shouldn’t be used here.

5-Definitely add the disabilities and ELL issues with regard to accessibility.  Bias in relation to language and ethnicity should be a part of the bias review.

6-Yes. We should include the word “bias” and phrase it  something like “does the item unfairly affect or is the item biased against students with disabilities, ESL speakers, students from different regions, SES, etc.” See also wording in the point that follows this one.

7-While the effort is to make assessments more accessible, I am very afraid that trying to meet the needs of multiple audiences has the potential of stripping down the assessments too much.

8-Yes, the inclusion of the word “bias” would be helpful.

9-Yes on all.

10-Include students with disabilities and language minority students.

11-Yes, I would add SWD and LEP students.  No, the word “bias” does not need to be included.

12-For sure, poverty should be explicitly addressed. I think regional differences may not be so important. Language differences are very important. Bias should be included and reference systematic variance that interferes with making a valid inference.

•   Avoids content that might unfairly advantage or disadvantage any student subgroup

44445555555

4.64

Have a clear format for text

•   Standard typeface

33334445555

4

1-What is meant by “standard?”  If you mean the same throughout a form, then 1.  If you mean an acceptable typeface for clarity, then 4.

2-“Standard typeface” does not communicate in a world where

everyone has access to hundreds of fonts—specify fonts and criteria re: serifs.

3-To do otherwise would introduce construct irrelevant variance (CIV).

4-Tests may be different between but should be the same within.

5-This should be “selection of typeface,” wherein the best proven typefaces are used.

6-General comment for all of these points: no one set of values

will work for all students. All we can hope for is a reasonable

compromise unless we embed flexibility, as per universal design principles. Perhaps explicitly state here that this consideration addresses print materials exclusively.

7-Serif recommended for print; sans-serif recommended for computer displays. For large-print booklets, consider

specific fonts (see APH guidelines: http://sun1.aph.org/edresearch/lpguide.htm).

8-R to #5—I am not familiar with and would like to hear about the best “proven” typefaces . . .

9-R to #7—We must be careful about

“recommendations” without some research to support them.

Instead of the word “standard” should we use the word “common?” “familiar?”  “clear?”  Please comment.

Would you change the wording entirely for the above consideration?  If so, how?

N/A

N/A

1-Now I realize that there are two issues: “standard” means both “consistent” and “common or clear.”  Both are needed.

2-In my experience, there are so many fonts and opinions about which to use, all with no research to support that one is necessarily better than the other for all types of readers. There may be 20 or so commonly used fonts and many will argue they are clear; and of course a 12 pt in one is not a 12 pt in the other. So, I’m not sure any one of these—common, standard, familiar, or clear—is any better than the other, unfortunately.  You may need to provide examples for guidance . . .(This is a major source of problems in the publishing arena.  There may be multiple fonts in one test booklet, all of which are considered “clear.” We have at least tried to keep the font chosen consistent within a book. However, for captions and such where emphasis is required, we will often opt for a different font instead of going to a smaller size or the use of italics, often used approaches in text books.)

3-Why don’t you include it in the next consideration…I don’t think typeface and font size need to be 2 separate ones.

4-Standardized typeface. I think we all understand that a student should never be confused over type face.

5-While these terms may be more flexible, they are also somewhat subjective. The key is to have the typeface appropriate for the intended audience, and we must realize that the audience is very diverse.

6-Yes. There is no standard typeface, thus the myriad fonts used in various publisher’s files, even within the same test or textbook. I suggest changing the wording to “does the item have a readable format and follow the guidelines for optimal readability according to current research.” Then we must equip the reviewer with those guidelines. 

7-As before, this stuff is all logical. Of course, keep the type face and format accessible. I would hate (to) see a recommendation for one type face et al.

8-I prefer “clear” to “standard,” as I agree with comment 2 above.

9-Use clear. No.

10- I like comments 5 and 7 as guidance here. Changed wording as with comment 5.

11-The term doesn’t matter as long as you define what you mean. Make sure you do that.

12-This may be a problem as standardized is what you want to avoid; What about a limited range of variation that has common features of typeface (font type size of font, leading, kerning, and serif)?  Clear and familiar beg the question (according to who?)

•   Twelve (12) point minimum for all print, including captions, footnotes, and graphs (Type size appropriate for age group)

33444444555

4.09

Additional comments on print size?

N/A

N/A

1-No.

2-Again, what is appropriate?  And 12 pt. varies with font types.

3-Appropriateness of size contingent on disability, if applicable.

4-While general recommendations for print size are reasonable, the bottom line again is that some readers may be unable to see that print clearly. IF the design from the beginning is flexible then print outs could be made available to fit the needs of individuals.

5-Yes. There are guidelines which apply to readers with both regular vision and low vision. These can be found at www.aph.org.  

6-N/A

7- Most important thing is that people can get an alternative font size.

8- Not by me.

9- No.

•   Wide spacing between letters, words, and lines

22233333445

3.09

1-Dependent on age.

2-To do otherwise would introduce construct-irrelevant variance (CIV).

3-Wide spacing is not necessarily best; proper font selection is more important.

4-Need to define precisely. Also consider that wide spacing may have deleterious effect on non-visually impaired students.

Should we retain the above consideration?

N/A

N/A

1-No.

2-I just don’t know.

3-No. 1. It’s too subjective. 2. Content should be weighted over format.

4-Yes.

5-Isn’t there an established standard that would transfer across font  size?

6-No. Unless is it redefined. 

7-Yes.

8-Yes.

9-I defer to other reviewers here.

10-Yes.

11-Not necessarily. Furthermore, this spacing may be developmental (e.g., more space for younger students).

•   High contrast between color of text and background

33344445555

4.09

1-Also is one needed on color intensity?

2-And selection of paper/ink color.

3-Even with sufficient color contrast, color blind users may not be able to distinguish text and background. Suggest you further recommend high print density contrast. This would also avoid isoluminance effects for non-visually-impaired students.

Additional comments on color and background?

1-Agree with comment 3.

2-This will be different for computer screens. Some colors are better than others on screen. For instance, I think blue is not so good on screen.

3-No.

4-“Sufficient color contrast and print density contrast between text and background.”

5-Again, equip reviewer with current research and examples.

6-N/A

7-None.

8-I defer to other reviewers here.

9-No.

10-Focus on maximum contrast, not color per se.

•   Sufficient leading (blank space) between lines of text

23334444455

2.82

1-After having just had a conversation on this, I’ll up my rating to 3.

2-“Standard typographic leading (blank spaces) between lines of text.”

•   Staggered right margins (no right justification)

22333334455

3.36

Have clear pictures and graphics (when essential to item)

•   Pictures are needed to respond to item

335555555

4.56

1-Does this mean how important is it for there to be picture-based items?  In some speaking tests, pictoral prompts are often the best way of getting students to produce language (without giving them directions and thus affecting their responses).

2-Deaf examinees tend to focus on pictures when provided, more than on text. A picture should be related to correct response and not be distraction to “trick” certain students (more than others).

3-Pictures, line art, etc. should be related to the item,

should enhance understanding not required for understanding with the exception of data tables like in math and science tests.

4-Perhaps reword as “only construct-relevant visuals.”

Should we change consideration to read: “When pictures are used, they should be redundant with the text as much as possible.”?

N/A

N/A

1-No. However, this is one where a great deal of work is needed to arrive at universal criteria.  I do not have the answer.

2-Often there are pictures used that are not redundant with the text but that are relevant to the item and to the construct.  I’d be careful with the rewording.

3-Pictures, when used, should support text and provide an additional resource for students to construct meaning.

4-Providing redundancy of figures in the text might provide inappropriate clues and invalidate items (e.g., interpreting graphs).  Still feel “only construct-relevant visuals” is best.

5-Yes. Specify what is meant by “clear.” Does this refer to standards for well-designed “graphics?” The pictures should be value-added for students needing a visual cue. This also works for the person who will not benefit from pictures, (i.e., blind and perhaps low vision students.) 

6-Pictures, diagrams, [etc.] should be included [if] they are integral to the question being asked.

7-No; I disagree with this re-write because it does not encompass the idea in comment 1 above. I like the suggested rewording in comment 4— “only construct-relevant visuals” because it seems to cover all the bases.

8-Yes, but change “they should be redundant” to “the content should be redundant.”

9-Yes. See comments 2 and 4.

10-I wouldn’t say “redundant,” rather I’d say, be connected to the text as much as possible.

11-Yes, so the issue of multiple opportunities is clear.

The pictures must also be consistent with the text.

•   Pictures with clearly defined features

44444455555

4.45

1-As we noted previously, this seems like catering to the lowest common denominator. Might there be instances where grayscale and shading are most appropriate for providing construct-relevant information to the student? If so, then alternate representations would need to be considered for visually impaired students.

•   Dark lines (minimum use of gray scale and shading)

33334444455

3.82

1-Pictures with “clearly defined features” covers this point This now seems redundant and could be removed.

•   Sufficient contrast between colors

13333444555

3.64

1-Again, color may be a different and additional issue for computer based assessments.

2-“Color alone is not relied on to convey important information or distinctions.”

•   Color is not relied on to convey important information or distinctions

23334445555

3.91

1-This is perhaps too vague to be useful as a guideline.

•   Pictures and graphs are labeled

33334444555

3.91

Have concise and readable text

•   Commonly used words

12445555555

4.18

1-Unless it is a vocabulary test.

2-Commonly used words are not necessarily easy…they can carry multiple meanings.

3-What does this mean in a “content” assessment? 

4-Again phrasing is inconsistent.  This is not appropriate for vocabulary/language assessment.

5-If the construct is about unique words, then common words might make sense.

6-Important, but highly subjective.

7-That is, use common meaning, not low-frequency meaning for common words.

8-This can be collapsed into the second item: age level vocabulary.

9-“Commonly used words” seems dependent on the construct being measured, e.g., vocabulary instruction. Can this point be combined with the next one?

Should we re-word the above consideration?  If so, what should it say?

N/A

N/A

1-Eliminate (agree with #8).

2-I think it should read something like “other than the terms relevant to the construct being measured, items should use simplified language and vocabulary (as they are different), and might even be one grade level below the grade being tested.”

3-Use words appropriate for the age of the students and that reflect the purpose of the item.

4-“Age-appropriate language (e.g., vocabulary, sentence complexity) when language is not central to the intended construct.”

5-Yes. I am not sure of the best wording, but word the consideration so that “concise and readable” are clearly defined and understood.

6-Please refer to the materials that I sent earlier. The issues about the meaning of “common words” in a content assessment need to be more directly addressed.

7-Merge with consideration below.

8-With the exception of subject-specific terminology, the text of an item should consist of commonly understood words or vocabulary consistent with each specific grade level.

9-Drop this one. Covered in next item adequately.

10-See comments 2 and 7.  Yes, reword the consideration to be consistent with those comments.

11-I think the bullets help explain the question just fine.

12-Commonly used words may be replaced by

reference to words most frequently appearing in

similar text or listed as the primary definition

in the dictionary.

•   Vocabulary appropriate for grade level

445555555555

4.83

1-I think both this and the next bullet could easily be combined with the first.

2-Depends on what you are measuring…

•   Minimum use of unnecessary words

124445555555

4.17

•   Idioms avoided unless idiomatic speech is being measured

344555555555

4.67

•   Technical terms and abbreviations avoided (or defined) if not related to the content being measured

44455555555

4.73

•   Sentence complexity is appropriate for grade level

14455555555

4.45

•   Question to be answered is clearly identifiable

555555555555

5

1-Refer to the work and writing of Jamal Abedi at UCLA regarding language simplification as it affects testing of LEP students.  Very critical work and he has many concrete suggestions about item writing and presentation that are consistent with our AERA, APA, NCME Standards for Educational and Psychological Testing.

Allow changes to its format without changing its meaning or difficulty (including visual or memory load)

1-The change in meaning is going to be speculative until the research is done, and maybe not even then.

•   Allows for the use of braille or other tactile format

34455555555

4.67

1-Not all items can be brailled or made tactile in a meaningful way. Would this mean that such items cannot be used for any students? What’s wrong with having equivalent items that can be so modified?

•   Allows for signing to a student

33455555555

4.55

•   Allows for the use of oral presentation to a student

33345555555

4.36

•   Allows for the use of assistive technology

33445555555

4.45

•   Allows for translation into another language

12333355555

3.64

1-Validity may be compromised as bias is interjected.

2-Some critical and relevant words/concepts do not translate into every language.  I think we might have to narrow to a few core languages.

3-This is a questionable and highly controversial issue, particularly when one realizes that such a standard is impossible to meet.  About 72% of our LEP students are Spanish speakers, but the other 28% represent many diverse languages.  How do we accommodate and what is the theoretical rationale and what is the technology for doing this?  Is it possible?  Is it beneficial?

4-Most likely invalidates the test scores for high stakes decision, should probably assess in an alternate fashion that more validly assess students ability, skill, knowledge, or learning.

5-A sticky wicket…

6-What about ELA tests? Also, what if the student prefers seeing the question in both their native language and the language of instruction? Should we deny them this opportunity?
7-There is the continual challenge of literal interpretation in translation, and limitations in expertise to translate to all needed languages. So, do students who speak Spanish receive interpreted tests, and those speaking a less well known language do not?

8-This issue will be one of the most difficult to overcome on a political and emotional level. Data analysis has shown these to be less than valid measures of student performance. Additionally, it has been suggested that it takes approximately 5 years for a student to become proficient in English.

Additional comments on this consideration?

N/A

N/A

1-There will be no easy answer to this one. Perhaps drop it and just not take a stand or list it later as an unresolved issue.

2-Seems to me if the terms used are as I’ve described above, there may be some lessening of the arguments here, particularly if accommodations such as customize dictionaries are allowed. Also, translations into Spanish, even, are not consistent. I know Jamal Abedi has run experiments where he has asked 2 separate translation companies to translate text from English into Spanish and back, finding the companies vary greatly in their translations to, supposedly, the same Spanish dialect.

3-Depends on what you are measuring… English language arts should not be subject to translation as it is idiosyncractic but universal content, anchored in standards, may be feasible if it undergoes content and bias reviews.

4-Don’t all items inherently allow for this already, in theory?

5-Perhaps we should decide to exclude ESL speakers from this until we have more research?

6-1-If the original test is valid as a measure of the curriculum, the translation will share that validity.

6-2-Minnesota translates to Hmong and Somali.  Only in these languages are their relevant words/concepts that do not translate easily into English.  The other languages of state assessment (Spanish, Russian, Chinese, Korean, Haitian Creole) almost never pose a problem for translating words or concepts. Professional translators will tell you they can translate almost any word or idea, and if they encounter one they can’t, they will tell you that too. 

6-3-What harm is done by helping the 72% of LEP students who speak Spanish? We provide accommodations to others where possible, but some would propose that a translated test is harmful.  Poppycock!

6-4-A translated test is always much less of a threat to validity and score comparability than an alternate assessment.

6-5-ELA tests should not be translated.  Once they are translated, they are no longer a test of ELA.  A bilingual booklet is an appropriate accommodation for all subjects except ELA.

6-6-There is not a limitation in expertise to translate tests.  The limitation is in money.  Translation must be cost effective like everything else in education.  You can’t provide translated tests for very small numbers. The Lao decision and other civil rights decisions make it clear that numbers dictate expectations of school systems.

A few psychometricians have maligned test translation unnecessarily.  One psychometrician says something and most of the rest follow like sheep. Translation is no more a threat to validity than a change in option order or a change in font.  Such changes might generate a miniscule change in item difficulty, but they don’t affect validity.  Neither does translation, which is the exact same test stated in a different language.

7-No.

8-None.

9-Translations need to be cognizant of dominant dialects.

Does the test...

Have an overall appearance that is clean and organized?

•   All images, pictures, and text provide information necessary to respond to the item

344445555555

4.5

1-Redundant with “pictures are needed to respond to item” and “minimum use of unnecessary words.”

2- I still rate this as a 5!

•   Information is organized in a manner that is consistent with an academic English framework with a left-right, top-bottom flow

444444445555

4.33

In addition to the other considerations, a computer-based test should have these considerations:

Layout and design

•   Sufficient contrast between background and text and graphics for easy readability

444455555555

4.67

1-“Sufficient luminance contrast between…”

•   Color is not relied on to convey important information or distinctions

233334455555

3.92

1-“Color alone is not relied…”

•   Font size and color scheme can be easily modified (through browser settings, style sheets, or on-screen options)

233344555555

4.08

•   Stimulus and response options are viewable on one screen when possible

344555555555

4.67

•   Page layout is consistent throughout the test

444555555555

4.75

Navigation

•   Navigation is clear and intuitive; it makes sense and is easy to figure out

455555555555

4.92

•   Navigation and response selection is possible by mouse click or keyboard

344555555555

4.67

1-“Navigation and response selection is possible by mouse (or equivalent) or keyboard (or equivalent).”

•   Option to return to items and return to place in test after breaks

3445555555

4.6

1-As before, returning to items will only work in non-adaptive tests.

Screen reader considerations

•   Item is intelligible when read by a text/screen reader

344455555555

4.58

•   Links make sense when read out of visual context. (“go to the next question” rather than “click here”)

444455555555

4.67

•   Non-text elements have a text equivalent or description

3334555555

4.3

•   Tables are only used to contain data, and make sense when read by screen reader

34444455555

4.36

Test specific options

 •  Access to other functions is restricted (e.g., e-mail, Internet, instant messaging)

33455555555

4.55

1-“Access to functions other than assistive technologies and supports is restricted (e.g., e-mail, Internet, instant messaging).”

•   Pop up translations and definitions of key words/phrases are available if appropriate to the test

333444445555

4.08

Computer capabilities

•   Adjustable volume

334455555555

4.5

1-“Adjustable volume and rate of voice.”

•   Speech recognition available (to convert user’s speech to text)

112334555555

3.67

•   Test is compatible with current screen reader software

334444455555

4.25

Below are new considerations suggested by panelists on the first survey, please rank each consideration and comment if you wish (numeric rankings found below were provided by participants in last survey).

Computer interfaces follow  Section 508 guidelines (www.section508.gov)

??013445555

1-(No number was given—only 3 question marks.)

2-There are other more recent and specific computer accessibility guidelines available, such as those found at www.aph.org, Microsoft, WCAG, etc. This is important but not a consideration for the item review process. Include in general guidelines for CBT.

3-(No number was given—only 5 question marks.)

4-Consideration needs more elaboration.

5-Not sure this can be standardized across vendors, platforms, and new technological advances.

6-Won’t be clear enough to most readers since so many considerations are in 508. Just make sure key issues are included in the other items.

7-I don’t even know what these guidelines are, but if they are designed to promote maximum accessibility, I would think that would be important.

8-Unpack 508 guidelines

Students have received adequate training on use of test delivery system

0445555555555

1-Sticky consideration as you do not want students spending an over necessary amount of time on test prep.

2-Yes, this is important.

3-Absolutely essential. These devices should be tied to the learners’ instruction and not unique to the assessment situation.

4-This is important but not a consideration for this item review process. Include in general guidelines for CBT.

5-Absolutely.

6-Students need to have critical keyboard and navigation skills.

Students writing online can get feedback on length of writing on-demand in cases where there is a restriction on number of words.

?012222333455

1-Writing on-line in and of itself is questionable for young students and those who have not been exposed to this technology.

2-Not sure about this.  Is this too much guidance.

Shouldn’t they be able to follow directions.

3-(This went with the question mark.) Yes, if it is a tool that is familiar to the student from their instruction.

4-This is important but not a consideration for this item review process. Include in general guidelines for CBT.

5-If it is truly a writing test and students are taught about writing for particular purpose, length of composition should be part of that which is measured

6-Is this ever an issue?  Are students ever penalized for too many words?

Students are able to record their responses and read them back (or have them read-back using text-to-speech) as alternative to human scribe, but only if student has experience with this mode of expression and chooses it for the test.

0233444445555

1-If feasible and part of the student’s IEP.

2-Yes, good idea.

3-This is important but not a consideration for this item review process. Include in general guidelines for CBT.

4-I think this would be preferable to a human scribe anyhow.

5-This accommodation can lead to considerable confusion on the part of the student.

Students are allowed to create persistent marks to the extent that they are already allowed to on paper-based booklets (e.g., marking items for review; eliminating multiple choice items, etc.).

033455555555

1-As long as it doesn’t impact scoring.

2-Good idea.

3-Is this a cross response platform statement?

4-This is important but not a consideration for this item review process. Include in general guidelines for CBT.

5-Until we do something with computer technology other than simply displaying a paper-and-pencil test item on screen we need to mimic that mode of administration.

6-If you mean “permanent marks” why not just say that?  In any case, I think this is critical.

7-Why not?

Alternate versions of computer interface provided that is amenable to use with screen readers (e.g., JAWS, Window-Eyes).

?04444555

1-(Wrote 4 question marks.)

2-This is important but not a consideration for this item review process. Include in general guidelines for CBT.

3-As long as the different interfaces are judged equally fair.

4-Versions of interface (browsers) may limit this.

When “modifying” items for use under a UD framework, there is convincing evidence that the item still measures the same or similar intended construct.

44444445555

1-Obviously, since I wrote these, I think they are all important.  I didn’t rate them as “5” because I think they are very hard to do, but I also think we need to be working in this direction.

2-What is “convincing?”

3-(Response to “obviously…” comment: Yes. I Agree.

4-I agree. Very important, but we may need more research before including this as a consideration in the review process (i.e., publishers are currently researching issue of “equivalency” between paper and pencil and CBT.

5-If it is the same content, skill, or ability, then we should be fine…similar is not enough to maintain content validity and equivalence to prior test developed in a state. However, if we are redesigning a test then similar is most likely fine.

A test constructed under a UD framework allows for the measurement of the same depth of knowledge levels as the “original” test.

011444444455

1-Poorly stated. Addressed in more useful, generic terms elsewhere.

2-I don’t understand why there would be 2 versions. Either the test is UD or it isn’t.

3-This should be obvious. The mode of delivery and response should not interfere with the constructions being measured.

4-This is important but not a consideration for this item review process. Include in general guidelines for CBT.

5-I think this is still unknown at the moment but it is an empirical question that we could investigate.

A UD test is aligned to the standards to the same extent as the “original” test.

014444455555

1-Addressed in more useful, generic terms elsewhere.

2-See above. (Above is the statement: This should be obvious. The mode of delivery and response should not interfere with the constructions being measured.—not sure to what it’s referring.)

3-This is important but not a consideration for this item review process. Include in general guidelines for accessible assessments.

4-Or to the construct…

5-Yes, but see above.

Test items are piloted, field tested, and normed on all subgroups for which the measure is designed.

033445555555

1-Agree, but this is very hard to do, due to N-count on many subgroups is too small.

2-Bravo! This is crucial but not a consideration for this item review process. Include in general guidelines for creating all assessments.

3-A valid test of the content is valid for all subgroups. The test is a measure of the content, not the subgroup.

4-There isn’t a norming process in most state accountability tests. Including or not including various groups in the item analyses may or may not have an impact on those analyses depending on the sample size used for analysis (e.g., 1,000; 2,500; etc. students per item) and the proportion of the population each group represents.

5-A must!

6-Norms may not be important.

Booklets/materials can be easily handled with limited motor coordination.

04445555

1-Yes.

2-This is important but not a consideration for this item review process. Include in general guidelines for accessible assessments.

3-I’m not sure how this would work in practice.

4-Why not?

Response formats are easily correlated to question.

0144555

1-Explain.

2-Unclear.

3-Quite frankly, I don’t understand what this one means.

4-Do you mean, “response formats are appropriate to the specific question?”  If so, I think you should use my wording.

5-Not sure what this means.

Computer-based option to mask items or text (e.g., split screen).

0123444444

1- Too specific.

2-Don’t understand.

3- This is important but not a consideration for this item review process. Include in general guidelines for CBT.

4- Yes, if it is important for students.

5- Why not?

Place for student to take notes (on the screen for CBT) or extra white space with paper-pencil.

03444444555

1-Good idea.

2-The CBT should as much as possible afford the strategies that one may use on a traditional looking format.

3-This is important but not a consideration for this item review process. Include in general guidelines for CBT.

4-Yes, very important.

5-All processing strategies should be made available.

Computer software for test delivery is designed to be amenable to all assistive technology.

02444455555

1-“Amenable” too open-ended.

2-Yes, but hard to do.

3-Ideal, may not be very realistic.

4-This is important but not a consideration for this item review process. Include in general guidelines for CBT.

5-This may be one of those moving targets that will be difficult to maintain due to technology changes.

6-That is, within reason for each disability.

7-Yes, otherwise it should not be used.

8-Very difficult to achieve and more difficult to maintain.

Considerations when reviewing any test item:

Does the item…

Measure what it intends to measure

•   Reflect the intended content standards (reviewers have information about the content being measured).

•   Minimize knowledge and skills required beyond what is intended for measurement.

Notes:

a. Content area assessments must be aligned to grade level state academic content standards (grade level expectations).

b. Information about the content standard(s) assessed by each item must be supplied to reviewers.

c. Careful consideration of the way content standards are phrased is important in determining what knowledge and skills involved in responding to an item are extraneous and which are relevant to what is being tested. 

d. When considering what is being measured there is somewhat of a “balancing act.” In certain types of test items additional skills may be necessary.  For example, responses to a listening test must be spoken or written, requiring skills in at least one modality in addition to listening. A similar issue is presented on math tests that require skills in reading. 

•   While it is important to minimize knowledge and skills beyond what is intended for measurement, it cannot take precedence over the ability to measure all content areas to be assessed (e.g., drawing a graph of the results).

•   When measuring skills such as higher-order processing skills, it is difficult to focus the assessment items only on the explicitly targeted content areas.

Does the item…

Respect the diversity of the assessment population

•   Sensitive to test taker characteristics and experiences (consider gender, age, ethnicity, socio-economic level, region, disability, and language)

•   Avoid content that might unfairly advantage or disadvantage any student subgroup

Notes:

a. Avoid bias toward or against any group of students that may cause them to have difficulty responding to items or create emotional stress.

b. Carefully evaluate what assumptions items make about shared experiences.

c. Tests should strive to avoid content that negatively depicts any student subgroup and avoid content that potentially provokes a negative reaction in any student subgroup.

d. Gender, etc., should not be a barrier to understanding the task an item requires.

e. It is important to recognize that every test is biased, although universal design serves to minimize its impact.  For example, English language learners are a subset of language minority students. 

f.  In an effort to make assessments more accessible, item writers and developers need to guard against stripping down assessments too much.

Does the item…

Have a clear format for text

•   Standard typeface.

•   Twelve (12) point minimum size for all print, including captions, footnotes, and graphs (type size appropriate for age group), and adaptable font size for computers.

•   High contrast between color of text and background.

•   Sufficient blank space (leading) between lines of text.

•   Staggered right margins (no right justification).

Notes:

a. Use clear, common, familiar, and consistent fonts (e.g., fonts with wide spacing between letters, words, and lines such as Times or Arial).

b. Avoid decorations and flourishes.

c. The term “blank space” rather than “white space” may be more accurate because the background is not always white.

d. A student should never be confused over type face.

e. Twelve-point varies with font types, as does spacing between letters.

f.  Some readers may be unable to see more typical print sizes clearly (e.g., 12 point).

g. When selecting color in text or background, consider high print density contrast in order to avoid isoluminance (i.e., colors appearing equivalent for student with color blindness).

Does the item…

Have clear visuals (when essential to item)

•   Visuals are needed to answer the question.

•   Visuals with clearly defined features (minimum use of gray scale and shading).

•   Sufficient contrast between colors.

•   Color alone is not relied on to convey important information or distinctions

•   Visuals are labeled.

Notes:

a. Pictures should have a purpose other than simply to be decorative.

b. Weigh whether the use of a visual (e.g., illustration) helps students or interferes with the content being tested.  This is a judgment call, but should be carefully considered.

c. Labeling pictures is helpful, when possible.  This is true even if the picture seems obvious.

d. Give additional clues besides color when possible (e.g., use text label “stop” and “go” next to stop light images).

e. Visuals should not distract or affect students who do not need the visual cue; rather they should add value for students who do need visual cues. 

f.  Pictures, when used, should support text and provide an additional resource for students to construct meaning.

g. There may be instances where grayscale and shading are appropriate for providing relevant information to students.

Does the item…

Have concise and readable text

•   Commonly used words (except vocabulary being tested).

•   Vocabulary appropriate for grade level.

•   Minimum use of unnecessary words.

•   Idioms avoided unless idiomatic speech is being measured.

•   Technical terms and abbreviations avoided (or defined) if not related to the content being measured.

•   Sentence complexity is appropriate for grade level.

•   Question to be answered is clearly identifiable.

Notes:

a. The use of common words depends on the content assessed. For example, if vocabulary is being tested, difficult or uncommon words might be appropriate to include.

b. Use of commonly used words does not assume that the words are simple.  They may carry multiple meanings.

c. Commonly used words may be replaced by reference words most frequently appearing in similar text or listed as the primary definition in the dictionary.

d. Some students may know less common words but may not know phrasal words. It is difficult to assume what is uncommon or difficult.

e. Other than the terms relevant to the construct being measured, items should use basic language and vocabulary (as they are different), and might even be one grade level below the grade being tested.

f.  If reading is not the primary construct tested, keep reading level at or below grade level in order to minimize construct irrelevant variance.

g. With the exception of subject specific terminology, the text of an item should consist of commonly understood words or vocabulary consistent with each specific grade level.

h. Determination of complexity can include many factors such as use of clauses, use of the passive, number of syllables in a word, length of sentences, length of single passage, combined length of all reading passages, amount of extraneous text involved in non-reading problems, etc. Complexity needs to be considered on all tests.

i.  When using authentic texts on reading passages, complexity may be difficult to control, but should at least be considered on test questions.

Does the item…

Allow changes to its format without changing its meaning or difficulty (including visual or memory load)

•   Allows for the use of braille or other tactile format.

•   Allows for signing to a student.

•   Allows for the use of oral presentation to a student.

•   Allows for the use of assistive technology.

•   Allows for translation into another language.

Notes:

a. Not all items can be brailled or made tactile in a meaningful way. Under such circumstances, items may need to be modified so that students with visual impairments can answer equivalent items testing knowledge of the same content.

b. Validity may be compromised when critical and relevant words/concepts cannot be translated into different languages.

Overall test considerations:

Does the test…

Have an overall appearance that is clean and organized

•   All visuals (e.g., images, pictures) and text provide information necessary to respond to the item.

•   Information is organized in a manner consistent with an academic English framework with a left-right, top-bottom flow.

•   Booklets/materials can be easily handled with limited motor coordination.

•   Response formats are easily matched to question.

•   Place for student to take notes (on the screen for CBT) or extra white space with paper-pencil.

Notes: 

a. Images, pictures, and text that may not be necessary include sidebars, overlays, callout boxes, visual crowding, shading, and general busyness—anything that may distract a student.

b. Carefully consider whether students from some groups may misinterpret the flow of text and graphics based on characteristics of their native language or culture. Left-right and top-bottom flow is cultural. For example, in some languages, text may flow top to bottom.

c. When using “authentic visuals” (e.g., a map), carefully consider what is being tested in addition to the intended content.

d. If a test has a time limit, careful consideration should be given to why a time limit is necessary for the content being tested.

e. Readability indices are now found in major word processors, however, it is important to check to see they are working as intended.

f.  Check to see which tests allow oral presentation.

g. Involve members of the major language groups in item review committees. 

h. Consideration of these issues prior to administration of a test will also help with the administration of oral interpretations in the native language, if allowed on a content test.  This issue is also relevant for sign language interpretations of tests, when appropriate.

i.  There are many ways to translate content area assessments, such as side-by-side, or developing parallel forms.  Carefully consider the plusses and minuses of each way prior to making a decision about your state test.  There is no perfect solution.

j.  Test items are piloted, field tested, and normed on all subgroups for which the measure is designed.

k. For computerized tests, students would need to have critical keyboard and navigation skills.

In addition to the other considerations, a computer-based test should have these considerations:

Layout and design

•   Sufficient contrast between background and text and graphics for easy readability.

•   Color alone is not relied on to convey important information or distinctions.

•   Font size and color scheme can be easily modified (through browser settings, style sheets, or on-screen options).

•   Stimulus and response options are viewable on one screen when possible.

•   Page layout is consistent throughout the test.

•   Computer interfaces follow Section 508 guidelines (www.section508.gov).

Notes:

a. The design of the color of the test needs to take into account test takers with color blindness including red/green distinctions.

b. The electronic format needs to be accessible through the specific assistive technology the student has experience using during the testing.  The latest technology may not be what is used in the schools.

c. More recent and specific computer accessibility guidelines are available at www.aph.org, Microsoft, WCAG etc.

Navigation

•   Students have received adequate training on use of test delivery system.

•   Navigation is clear and intuitive; it makes sense and is easy to figure out.

•   Navigation and response selection is possible by mouse click or keyboard.

•   Option to return to items and return to place in test after breaks.

Notes:

a. How to navigate and navigation symbols should be intuitive and/or explained at the beginning of a test.

b. The screen resolution varies on different computers. Reviewers should check out items on different types of computers commonly used in schools.

c. Schools need reasonable minimum standards for computer and audio requirements for a test.

d. Test administration instructions should include standardized settings for the computer.

e. Students need practice opportunities before taking computer-based tests.

f.  Some listening tests may want to limit the number of times a student can listen to a recording, depending on standards being tested.

Screen reader considerations

•   Item is intelligible when read by a text/screen reader.

•   Links make sense when read out of visual context (“go to the next question” rather than “click here”).

•   Non-text elements have a text equivalent or description.

•   Tables are only used to contain data, and make sense when read by screen reader.

Notes:

a. Images and animations have text labels, if this does not supply the answer.

b. Captioning and transcripts of audio and video are available.

c. Provide titles and summaries for tables and graphs.

d. Header cells for columns and/or rows are designated.

e. Information in tables makes sense when linearized (i.e., read top left to bottom right cell).

f.  Current screen reader technology might be difficult for ELLs to understand, real voice technology may be needed.

Test specific options

•   Access to other functions is restricted (e.g., e-mail, Internet, instant messaging).

•   Pop up translations and definitions of key words/phrases are available if appropriate to the test.

•   Students writing online can get feedback on length of writing on-demand in cases where there is a restriction on number of words.

•   Students are able to record their responses and read them back as an alternative to a human scribe.

•   Students are allowed to create persistent marks to the extent that they are already allowed to on paper-based booklets (e.g., marking items for review; eliminating multiple choice items, etc.).

Notes:

a. Access to spell check might also be limited depending on the test.

b. Variable audio speed might be useful to some students if it does not interfere with the standard being tested.

c. The option for feedback on demand on the length of student writing would depend on the extent that keeping text length within some parameter is part of the construct being measured.

Computer capabilities

•   Adjustable volume.

•   Speech recognition available (to convert user’s speech to text).

•   Test is compatible with current screen reader software.

•   Computer-based option to mask items or text (e.g., split screen).

•   Computer software for test delivery is designed to be amenable to assistive technology.

Notes:

a. Alternate versions of computer interface provided that is amenable to use with screen readers (e.g., JAWS, Window-Eyes).

Measures what it intends to measure

•   Test development begins with a careful consideration of the skills proposed for measurement (Popham & Lindheim, 1980).

•   Every item should reflect specified content and mental behaviors, as called for in test specifications (Haladyna, Downing, & Rodriguez, 2002).

•   Removal of construct irrelevant variance increases tests scores for students with reading difficulties (Calhoun, Fuchs & Hamlett, 2000; Harker & Feldt, 1993; Koretz, 1997; Tindal, Heath, Hollenbeck, Almond & Harniss, 1998).

•   Language in non-language arts assessments needs to be “transparent” enough to students to clearly determine construct being measured (Sharrocks-Taylor & Hargreaves, 1999).

Respects the diversity of the assessment population

•   Items must be reviewed for bias that may exist against particular populations (National Research Council, 1999).

•   Items that are designed from the start with equity and accessibility features are less likely to be biased against particular populations (Kopriva, 2000).

•   Items must be free of content that makes a student’s socioeconomic status or inherited academic aptitudes the dominant influence on how a student will respond to the item (Popham, 2001).

•   Items must be free of content that may unfairly benefit or penalize students from diverse ethnic, socioeconomic, or linguistic backgrounds, or students with disabilities (Popham, 2001).

•   Cultural norms, beliefs, and customs need to be respectfully reflected in illustrations (Schiffman, 1995).

Has a clear format for text

•   The point sizes most often used are 10 and 12 point for documents to be read by people with excellent vision reading in good light (Gaster & Clark, 1995).

•   Fourteen point type increases readability and can increase test scores for both students with and without disabilities, compared to 12-point type (Fuchs, Fuchs, Eaton, Hamlett, Binkley, & Crouch, 2000).

•   Type size for captions, footnotes, keys, and legends needs to be at least 12 point (Arditi, 1999).

•   Larger type sizes are most effective for young students who are learning to read and for students with visual difficulties (Hoener, Salend, & Kay, 1997).

•   Large print is beneficial for reducing eye fatigue (Arditi, 1999).

•   Shapes of letters and numbers should enable people to read text “quickly, effortlessly, and with understanding” (Schriver, 1997).

•   The relationship between readability and point size is also dependent on the typeface used (Gaster & Clark, 1995; Worden, 1991).

•   Letters that are too close together are difficult for partially sighted readers. Spacing needs to be wide between both letters and words (Gaster & Clark, 1995).

•   Fixed-space fonts seem to be more legible for some readers than proportional-spaced fonts (Gaster & Clark, 1995).

•   Leading should be 25–30 percent of the point (font) size for maximum readability (Arditi, 1999).

•   Leading alone does not make a difference in readability as much as the interaction between point size, leading, and line length (Worden, 1991).

•   Standard typeface, upper and lower case, is more readable than italic, slanted, small caps, or all caps (Tinker, 1963).

•   Text printed completely in capital letters is less legible than text printed completely in lower-case, or normal mixed-case text (Carter, Dey, & Meggs, 1985)

•   Italic is far less legible and is read considerably more slowly than regular lower case (Worden, 1991).

•   Boldface is more visible than lower case if a change from the norm is needed (Hartley, 1985)

•   Staggered right margins are easier to see and scan than uniform or block style right justified margins (Arditi, 1999; Grise, Beattie, & Algozzine, 1982; Menlove & Hammond, 1998).

•   Justified text is more difficult to read than unjustified text—especially for poor readers (Gregory & Poulton, 1970; Zachrisson, 1965).

•   Justified text is also more disruptive for good readers (Muncer, Gorman, Gorman, & Bibel, 1986).

•   A flush left/ragged right margin is the most effective format for text memory. (Thompson, 1991).

•   Unjustified text may be easier for poorer readers to understand because the uneven eye movements created in justified text can interrupt reading (Gregory & Poulton, 1970; Hartley, 1985; Muncer, Gorman, Gorman, & Bibel, 1986; Schriver, 1997).

•   Justified lines require the distances between words to be varied. In very narrow columns, not only are there extra wide spaces between words, but also between letters within the words (Gregory & Poulton, 1970).

•   Longer lines, in general, require larger type and more leading (Schriver, 1997).

•   Optimal length is 24 picas—about 4 inches (Worden, 1991).

•   Lines that are too long make readers weary and may also cause difficulty in locating the beginning of the next line, causing readers to lose their place (Schriver, 1997; Tinker, 1963).

•   Lines of text should be about 40–70 characters, or roughly eight to twelve words per line (Heines, 1984; Osborne, 2001; Schriver, 1997).

•   Blank space anchors text on the paper and helps increase legibility (Menlove & Hammond, 1998; Smith & McCombs, 1971).

•   A general rule is to allow text to occupy only about half of a page. Too many test items per page can make items difficult to read (Tinker, 1963).

Has clear pictures and graphics (when essential to item)

•   Graphics with a clear sense of unity, a clear focal point, and balance reduce the cognitive load of perceiving information and computer-based tests should allow students to change the size of the font (see computer specific considerations below) and thus increase speed with which the user can access graphic material (Szabo and Kanuka, 1998).

•   If illustrations are present they are at best essential information, good if they support the information, and unnecessary if they are unrelated to the construct or item (Sharrocks-Taylor & Hargreaves, 1999).

•   Illustrations should be placed directly next to the information for which they refer (Silver, 1994; West, 1997).

•   Placing labels directly on plot lines of graphs reduces the load on short-term memory (Gregory & Poulton, 1970).

•   Quantitative displays should be structured so that readers can easily construct appropriate inferences about the data (Schriver, 1997).

•   Illustrations should be placed directly next to the information for which they refer (Silver, 1994; West, 1997).

•   Graphs, illustrations, and other graphic aids can facilitate comprehension (Rakow & Gee, 1987)

Has concise and readable text

•   General readability principles such as fewer words per sentence and the removal of irrelevant difficult words increases comprehension of items (Popham & Lindheim, 1980; Rakow & Gee, 1987).

•   Flow of sentences is also an important feature.  Caution should be taken when reducing reading load so that sentences do not become disjointed or incomprehensible (Anderson, Hiebert, Scott, & Wilkinson, 1985).

•   Compound sentences can be written in two separate sentences (if sentences are still comprehensible) (Gaster & Clarke, 1995).

•   Most important ideas should be stated first in a sentence (Gaster & Clarke, 1995).

•   Noun-pronoun relationships should be clear (Gaster & Clarke, 1995).

•   Illustrations should be placed close to the text they support (Gaster & Clarke, 1995), or removed if they do not support text.

•   Readability increases when students have likely had experiences or prior knowledge relating to items (Rakow & Gee, 1987).

•   Content within items is clearly organized (Rakow & Gee, 1987)

•   The content of every item should be independent from content of other items on the test (Haladyna et al., 2002)

•   Questions are clearly framed (Rakow & Gee, 1987)

•   Limit the number of words, difficulty of words (Popham & Lindheim, 1980), and grammatical complexity of test materials (Popham & Lindheim, 1980)

•   Keep vocabulary simple for the group of students being tested (Haladyna et al., 2000).

•   Minimize the amount of reading in each item (Haladyna et al., 2002).

•   Avoid window dressing (excessive verbiage; Haladyna et al., 2002).

•   Simple, clear, commonly used words should be used whenever possible (Gaster & Clarke, 1995).

•   Technical terms should be defined (Gaster & Clarke, 1995).

•   One idea, fact, or process should be introduced at a time, then ideas developed logically (Gaster & Clarke, 1995).

•   If time and setting are important to the sentence, they should be placed at the beginning of the sentence (Gaster & Clarke, 1995).

•   Sequence steps of instructions in the exact order that they will be needed (Gaster & Clarke, 1995).

•   Vocabulary should be grade-level appropriate (Rakow & Gee, 1987).

•   Sentence complexity must be appropriate for grade level (Rakow & Gee, 1987).

•   Definitions and examples must be clear and understandable (Rakow & Gee, 1987).

•   Required reading skills are appropriate for students’ cognitive level (Rakow & Gee, 1987).

•   Use of plain language: “text-based language that is straightforward, concise, and uses everyday words to convey meaning.  The goal of plain language editing strategies is to improve the comprehensibility of written text while preserving the essence of its message.”  (Hanson, Hayes, Schriver, LeMahieu, & Brown, 1998, p.2).

•   Reduce the verbal and organizational complexity of test items while preserving their essential content (i.e., the skills and concepts they were intended to measure.) (Hanson et al, 1998, p.2).

•   Reduce excessive length; reduce wordiness and remove irrelevant material (Brown, 1999).

•   Eliminate unusual or low frequency words and replace with common words (e.g., replace “utilize” with “use”) (Brown, 1999).

•   Avoid ambiguous words (e.g., crane) (Brown, 1999).

•   Avoid irregularly spelled words (e.g., trough, feign) (Brown, 1999).

•   Avoid proper names, replace with common names or no names at all (Brown, 1999).

•     Avoid inconsistent naming or graphic conventions (e.g., multiple names for the same concept—unless recognizing multiple names for the same concept is the construct being tested) (Brown, 1999).

•     Avoid unclear signals about how to direct attention (e.g., phrases such as “in the table below” are helpful) (Brown, 1999).

•     Mark all questions clearly (Brown, 1999).

•     Simplified English on math tests enables comparable performance by English language learners and English proficient students (Abedi, Hofstetter, Baker, & Lord, 2001).

•     Linguistic demands of tests often pose the greatest barrier to students’ ability to demonstrate knowledge of mathematical concepts (Hanson, 1997).

•     Plain language versions of tests assist students that understand content but are less proficient in English language.  They do not help students who do not understand content (Brown, 1999).

Allows changes to its format without changing its meaning or difficulty (including visual or memory load)

•   Construct irrelevant graphs, vertical text, untranslatable material, and decorative graphics all create situations where accommodating students who use braille, American Sign Language, or non-English languages is difficult.

Additional considerations for computer-based assessments

•   Students reported difficulties with computers including excessive need for forward and back buttons, unfamiliarity with response mechanisms, and an inability to see entire problems on screens (Trotter, 2001).

•   Students may not be familiar with skills like scrolling or using text on multiple screens (Cole, Tindal, & Glasgow, 2000).

•   Some students have had little access to computers and calculators prior to testing (Bridgeman, Harvey, & Braswell, 1995; MacArthur & Graham, 1987).

Star  (H) areas of strength and Check (3) areas of concern for each
Does the item…

Item #

Measure what it intends to measure

•   Reflect the intended content standards (reviewers have information about the content being measured)

•   Minimize knowledge and skills required beyond what is intended for measurement

Respect the diversity of the assessment population

•   Sensitive to test taker characteristics and experiences (consider gender, age, ethnicity, socio-economic level, region, disability and language)

•   Avoid content that might unfairly advantage or disadvantage any student subgroup

Have a clear format for text

•   Standard typeface

•   Twelve (12) point minimum size for all print, including captions, footnotes, and graphs (Type size appropriate for age group)

•   High contrast between color of text and background

•   Sufficient blank space (leading) between lines of text

•   Staggered right margins (no right justification)

Have clear visuals (when essential to item)

•   Visuals are needed to answer the question

•   Visuals with clearly defined features (minimum use of gray scale and shading)

•   Sufficient contrast between colors

•   Color alone is not relied on to convey important information or distinctions

•   Visuals are labeled

Does the item…

Item #

Have concise and readable text

•   Commonly used words (except vocabulary being tested)

•   Vocabulary appropriate for grade level

•   Minimum use of unnecessary words

•   Idioms avoided unless idiomatic speech is being measured

•   Technical terms and abbreviations avoided (or defined) if not related to the content being measured

•   Sentence complexity is appropriate for grade level

•   Question to be answered is clearly identifiable

Allow changes to its format without changing its meaning or difficulty (including visual or memory load)

•   Allows for the use of braille or other tactile format

•   Allows for signing to a student

•   Allows for the use of oral presentation to a student

•   Allows for the use of assistive technology

•   Allows for translation into another language

Have an overall appearance that is clean and organized

•   All visuals (e.g., images, pictures) and text provide information necessary to respond to the item

•   Information is organized in a manner consistent with an academic English framework with a left-right, top-bottom flow

•   Booklets/materials can be easily handled with limited motor coordination

•   Response formats are easily matched to question

•   Place for student to take notes (on the screen for CBT) or extra white space with paper-pencil

Does the item…

Areas of Strength

Areas of Concern

Suggestions for Improvement

Measure what it intends to measure

•   Reflect the intended content standards (reviewers have information about the content being measured)

•   Minimize knowledge and skills required beyond what is intended for measurement

Respect the diversity of the assessment population

•   Sensitive to test taker characteristics and experiences (consider age, gender, ethnicity, socio-economic level, region, disability and language)

•   Avoid content that might unfairly advantage or disadvantage any student subgroup

Have clear format for text

•   Standard typeface

•   Twelve (12) point minimum size for all print, including captions, footnotes, and graphs (type size appropriate for age group)

•   High contrast between color of text and background

•   Sufficient blank space (leading) between lines of text

•   Staggered right margins (no right justification)

Have clear visuals (when essential to item)

•   Visuals are needed to answer the question

•   Visuals with clearly defined features (minimum use of gray scale and shading)

•   Sufficient contrast between colors

•   Color alone is not relied on to convey important information or distinctions

•   Visuals are labeled

Have concise and readable text

•   Commonly used words (except vocabulary being tested)

•   Vocabulary appropriate for grade level

•   Minimum use of unnecessary words

•   Idioms avoided unless idiomatic speech is being measured

•   Technical terms and abbreviations avoided (or defined) if not related to the content being measured

•   Sentence complexity is appropriate for grade level

•   Question to be answered is clearly identifiable

Allow changes to its format without changing its meaning or difficulty (including visual or memory load)

•   Allows for the use of braille or other tactile format

•   Allows for signing to a student

•   Allows for the use of oral presentation to a student

•   Allows for the use of assistive technology

•   Allows for translation into another language

Does the test…

Areas of Strength

Areas of Concern

Suggestions for Improvement

Have an overall appearance that is clean and organized

•   All visuals (e.g., images, pictures) and text provide information necessary to respond to the item

•   Information is organized in a manner consistent with an academic English framework with a left-right, top-bottom flow

•   Booklets/materials can be easily handled with limited motor coordination

•   Response formats are easily matched to question

•   Place for student to take notes (on the screen for CBT) or extra white space with paper-pencil