The Effect of a Simplified English Language Dictionary on a Reading Test

LEP Projects Report 1

Published by the National Center on Educational Outcomes

Prepared by Deb Albus, John Bielinski, Martha Thurlow, and Kristin Liu

March 2001

Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:

Albus, D., Bielinski, J., Thurlow, M., & Liu, K. (2001). The effect of a simplified English language dictionary on a reading test (LEP Projects Report 1). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/LEP1.html

Executive Summary

This study was conducted to examine whether using a monolingual simplified English dictionary as an accommodation on a reading test with limited English proficient (LEP) Hmong students improved test performance. Hmong students were chosen because they are often not literate in their first language due to a lack of educational experiences in Hmong, which was first put into written form in the 1970s.  For these students, bilingual dictionaries are unlikely to be useful. Thus, we studied the possible usefulness of a monolingual English dictionary for these students.  Students for this study came from three urban middle schools in a large metropolitan area of Minnesota. There were a total of 69 students in the non-LEP group, and 133 students in the Hmong LEP group. The study was conducted using a randomized counter-balanced design, with a control group of non-LEP students and an experimental group of Hmong LEP students.   All students were administered two reading passages with an English dictionary available, and two passages without the dictionary, varying passage order and order of accommodation in both study groups. The students’ test performance on the two reading passages with dictionary accommodation was then compared to their test performance on the two reading passages without dictionary accommodation, using a repeated measures ANOVA procedure. Results showed that there was not a significant difference in reading comprehension scores for students in either the LEP or non-LEP group under accommodated conditions. However, it was found that intermediate level English proficiency students in the Hmong LEP group who reported using the dictionary in the accommodated condition showed a moderately significant gain. Issues discussed include student dictionary ability, dictionary interactions with test items, test development considerations, and current beliefs about dictionary accommodations and reading assessment.

Overview

A challenge facing states and districts is determining best practices for including Limited English Proficient (LEP) students in their testing systems. One recommended approach is to provide testing accommodations that are designed to reduce language barriers while not changing what a test is supposed to measure. One accommodation that is available in several states is allowing LEP students to use a dictionary during a test. However, there are few studies on the effects of dictionary use on test performance and score validity.

The way in which dictionaries are used as an accommodation on reading tests varies across the country. In a survey of state assessment directors for 1998-99, Rivera, Stansfield, Scialdone, and Sharkey (2000) identified 21 states that allowed the use of bilingual dictionary accommodations and only 3 states that specifically prohibited them. Among the states that allowed bilingual dictionaries, 11 states allowed them on all assessment components and 10 states allowed them on some of the assessment components (Rivera, et al., 2000). According to a survey conducted by the Council of Chief State School Officers (Olson, Bond, & Andrews, 1999), which covered 1997-98 assessments, only two of these states allowed bilingual dictionaries and the third allowed an English language dictionary.

Current views about the appropriateness of dictionary accommodations in testing differ among researchers, educators, and LEP students. For example, there are both research and opinions that support using dictionary accommodations. Researchers have shown that unfamiliar vocabulary may cause difficulty in understanding items on tests (Garcia, 1991) and that using dictionaries can help students’ reading comprehension (Goyette, 1996; Laufer & Hadar, 1997) and equalize skill assessments for LEP students (Rivera & Vincent, 1997). Further, LEP students and English as a Second Language (ESL) and Bilingual Education teachers have requested them in testing situations because dictionaries are an accommodation that students use in classes every day and will most likely use throughout their lives (Bensoussan, 1983; Liu, Spicuzza, Erickson, Thurlow & Ruhland, 1997; Quest, Liu, & Thurlow, 1997).

There is also research and opinion that opposes using dictionary accommodations. It is a common belief among some researchers and test developers that any alteration to the standard administration necessarily alters the validity of the test score. Others argue that dictionaries should not be allowed because they may negatively affect the validity of a test (Rivera & Stansfield, 1998; Spolsky, 1997). Other reasons given against using dictionaries are that students will need more time with dictionaries, that students may over rely on them (Roizen, 1984), and that they have not been found to significantly affect reading comprehension test scores of individuals learning English as a foreign language (Bensoussan, 1983; Nesi & Meara, 1991).

One argument against dictionary use is that it negates the role of specific vocabulary knowledge as an essential component of reading ability (Bensoussan, 1983). However, Bensoussan argues that a student using a dictionary still needs to be able to successfully choose the right meaning of a word based on the context of a passage in order to correctly answer a test question. Contextual clues may not always be readily available in a passage to infer meanings of unknown words (in tests or in everyday reading). Therefore, the availability of a dictionary does not guarantee understanding of unknown words with or without sufficient contextual clues.

According to some researchers, readers need to comprehend a certain percentage of a text to be able to infer meaning of unknown words. Laufer (1997) suggested 95% text comprehension (understanding of 3,000 word families) before reading skills in a reader’s first language will aid reading in the second language, including inferring meanings of words from context. Hirsh and Nation (1992) suggested 98% text comprehension for pleasure reading, requiring readers to have knowledge of approximately 5,000 word families. Although reading language skills may be more developed in the second language of some students than in their first language, adequate text comprehension still requires understanding of a high percentage of the words. Also, the extent of word family knowledge needed in a reader’s “sight” or “automatic” vocabulary suggests that dictionary use during a reading comprehension test may not greatly enhance performance if students’ overall threshold vocabulary is too low. Observed problems with vocabulary thresholds have led some researchers to conclude that students’ problems with comprehension are basically lexical rather than due to lack of reading strategies (Haynes & Baker, 1993). However, a dictionary would not be able to compensate a student with great gaps in vocabulary knowledge, therefore its use may be more beneficial for students whose proficiency is near the level required for comprehending a text.

In some studies conducted with students learning English as a Foreign Language (EFL) in Israel, it was found that dictionaries were more useful for students with a moderate level of English proficiency and some dictionary skills (Bensoussan, 1983; Laufer & Hadar, 1997), and that they did not benefit students with very low or very high proficiency. Therefore, the accommodation may not benefit the lower proficiency students who would need the most help (Shepard, Taylor, & Betebenner, 1998).

The majority of dictionary accommodation reading studies have been conducted in EFL settings, thus it is important to study dictionary use in an English as a Second Language setting, especially where most students are not literate in their first spoken language. Further, because most dictionary studies have used EFL tests, our goal was to seek evidence to either support or refute the use of dictionary accommodations in large-scale tests in the United States.

Researchers who support dictionary accommodations favor the use of bilingual or bilingualized dictionaries (English dictionaries with a native language translation) for both LEP students in the United States and LEP students overseas (Laufer & Hadar, 1997; Rivera & Stansfield, 1998). Laufer and Hadar (1997) found that, in general, monolingual dictionaries were the least useful accommodation on tests of English as a foreign language. However, in the United States where students are in an English as a second language setting, bilingual dictionaries may not always be the best choice, particularly for individual students who may not be literate or may not have received any education in their first language. Despite being classified as LEP, the language that these students read and write the most fluently may, in fact, be English.

The quality and appropriateness of dictionaries (including the quality of their translations), whether bilingual or monolingual, vary greatly. Some bilingual dictionaries only give word for word translations or incomplete meanings, while others give definitions. English monolingual dictionaries, on the other hand, sometimes provide only the most basic definition of a word. If students are allowed to bring their own dictionaries to a test, some may be at a disadvantage because of the specific dictionary they choose. English language dictionaries tend to give more detailed explanation about the function of words in context; however, the number and depth of entries and illustrations are not equal across dictionary versions. There are also differences between regular and simplified English dictionaries. Abedi, Lord and Plummer (1997) studied the impact of simplified English in the test items themselves on the National Assessment of Educational Progress (NAEP), and found that simplified English on tests had a positive effect for all students, not just for those who are LEP. If simplified English has been found to affect students’ performance on tests, it is likely that simplified English in dictionaries could also have an effect.

Goal of the Study

This study was conducted to examine the possible effects of using a monolingual simplified English dictionary as an accommodation on a reading test with limited English proficient Hmong students.

Research Questions

Four primary research questions were posed for this study.

1.  Do Hmong LEP students provided with a simplified English dictionary perform better than when the dictionary is not provided?

2.  How does use of a simplified English dictionary by Hmong LEP students affect the reliability of test scores?

3.  What are the characteristics of Hmong LEP students whose scores are most affected by the use of a simplified English dictionary accommodation?

4.  Do students (LEP and non-LEP) want to use a simplified English dictionary as a test accommodation?

Method

Participants

Students for this study came from three urban middle schools in a large metropolitan area of Minnesota. There were a total of 69 regular education students in the non-LEP group and 133 students in the Hmong LEP group. No restrictions were placed on the backgrounds of students in the non-LEP groups other than they not be LEP students or students receiving special education services. Some students in the non-LEP group were from Hmong language backgrounds, but none of them was receiving services for limited English proficiency. Students from both groups were recruited at all three schools; however, only two of the schools provided students from both groups. The third school chose only to allow participation for its Hmong LEP student population. Data on an economic status indicator (receiving free or reduced lunch) were collected for both the Hmong LEP group and the non-LEP group. The students in the two testing groups were comparable.

Schools used similar ESL level designations (1-5), but the specific description of each level varied across sites. The levels indicate the range of students within ESL classes, from beginning to high levels of English proficiency. The number and percent of LEP students by level are presented in Table 1 (1=lowest proficiency level).

Table 1. Number and Percent of LEP Students by ESL Level

Note: Level of English proficiency as assigned by school personnel.

Design

The study was conducted using a randomized counter-balanced design, with a control group of regular education students and an experimental group of Hmong LEP students. Students were administered two reading passages with the English dictionary available, and two passages without the dictionary. The students’ test performance on the two reading passages with dictionary accommodation was then compared to their test performance on the two reading passages without dictionary accommodation, using a repeated measures ANOVA procedure.

The passages were designed to parallel Minnesota’s Basic Standards Reading Test, which is part of the state graduation exam. However, the items used for this study had not been used or equated with officially administered tests. The passages represent sample passages that had been reworked by an assessment specialist for use in the Minnesota Assessment Project study of bilingual reading test items (Anderson, Liu, Swierzbin, Thurlow, & Bielinski, 2000). In addition, an LEP graduation standards specialist at the Minnesota Department of Children, Families and Learning and a bilingual adult member of the ESL community provided advice on cultural background in the development of the test passages. The two halves of the test were divided into Form A and Form B. The passages were assigned to forms so that Form A and Form B had the same overall difficulty. Table 2 shows the study design, which is a modification of one design presented by Thurlow, McGrew, Tindal, Thompson, Ysseldyke, and Elliott (2000).

Table 2. Study Design

Test Instrument

Form A and Form B each contained two passages. Each passage was 900-1,040 words in length. For each reading passage, the examinee was asked 10 multiple-choice questions that addressed both literal and inferential comprehension. The test used in this study was previously used in a study on bilingual translation accommodations by researchers of the Minnesota Assessment Project. Evaluation of test score reliability indicated that it had the same or higher internal consistency as the actual Minnesota Basic Standards reading test for most of the test groups (Anderson et al., 2000).

A dictionary exercise, developed by research staff with ESL teaching backgrounds, was also administered to determine each student’s proficiency with the simplified English dictionary. The exercise was composed of four questions. Two questions asked students to provide written definitions of words appropriate to context sentences that were provided to them. A third question was aimed specifically at alphabetizing skill. The fourth question was designed to determine students’ ability to use the dictionary for other information about parts of speech. A four-point scoring rubric was developed for raters to evaluate and score the exercise.

Dictionary Accommodation

A simplified English dictionary was used, as opposed to a Hmong Bilingual dictionary because most Hmong background LEP students have limited literacy in their first language. The dictionary chosen for the study was The American Heritage English as a Second Language Dictionary (Houghton Mifflin Company, 1998). This simplified English dictionary was chosen for several reasons. First, the dictionary for the study had to include more difficult academic words specific to the test passages (e.g., cum laude, precocious), and had to provide a range of definitions for words with multiple meanings (e.g., stand, produce). It also had to present the definitions at a simplified level of English for clarity, and yet fit the needs of a range of proficiency levels among LEP students.

Some of the other available ESL dictionaries had more pictorial content, but had fewer of the words that had been identified as potentially problematic in the study passages. Other ESL dictionaries were too basic, providing only one meaning for words with multiple meanings, and often giving meanings that did not match the meaning of words in the test passages. Also, for one of the vocabulary items on the test, another dictionary provided a definition with an example sentence that could have misled students to choose an incorrect answer. The example sentence included extra descriptive information that was contradictory to the correct test response.

Study Procedure and Timing

Students were assembled into either an auditorium or classroom to take the test. The size of the groups ranged from 11 to 58 students per room. Students were first asked to fill out a brief pre-test questionnaire about language background to provide self-ratings of their English and Hmong proficiency in several modalities: speaking, listening, and reading.

Next, students were administered one half of the test; some of the students getting the dictionary accommodation and some not. Students were allowed as much time as they needed to complete each half of the test, having been given a general time limit of two hours. After completing the first half of the test, the student raised his or her hand and a test administrator started the student on the second half. If the student had the dictionary for the first half, dictated by the color of the test cover, the administrator removed it; if the student took the first half without the dictionary, then it was provided to the student on the second half. During test administration, staff recorded start and finish times on the test covers at the end of each half of the test while providing and removing dictionaries for students’ use so that the time taken by students with and without dictionaries could be tracked. Immediately after completion of the whole test, students were given a post-test questionnaire about dictionary use during the test, their opinions on possible usefulness of an English dictionary on a reading test, and other background information on dictionary use and instruction in the classroom. A short dictionary exercise also was given after the post-test survey to determine levels of student ability in using a dictionary.

Students were allowed approximately two-and-one-half hours to complete all materials. This time allotment was determined on the basis of schedule limitations in the schools. Following completion of the test, students either stayed in the same room or simply returned to class, depending on school requests. For example, at one site, the students who completed the test were given other activity sheets to work on while waiting for other students to finish before being released back to class. At other sites, students were allowed to leave after they completed the study materials. It is uncertain whether these varying procedures had any effect on test results.

Results

Proficiency in English and Hmong

On student pre-test questionnaires, the control group and the Hmong LEP group answered a series of questions on language proficiency for speaking, listening, and reading in English and Hmong, and the length of time they had spent in U.S. schools. Most students in the Hmong LEP group reported higher reading ability in English than in Hmong (see Table 3), with most students rating their English reading ability as either “well” or “pretty well.” For speaking and understanding spoken English/ Hmong, the majority of the group described themselves between “Well” and “Pretty well” for English, and between “Very well” and “Pretty well” for Hmong.

Table 3.   Hmong LEP Group Self-Report for English and Hmong Reading and Speaking Ability

In the non-LEP group (see Table 4), as might be expected, the majority reported their level of English proficiency as “Very well” in both modalities. This group did include some students with Hmong language background who were not limited English proficient. For these students, some reported a range of “Well” to “Very well” in understanding and speaking Hmong. However, very few reported reading Hmong above “Not very well.”

Table 4   Non-LEP Group Self-Report for English and Hmong Reading and Speaking Ability

Time in U.S. Schools

Table 5 shows the length of time in U.S. schools for each group. The percentage of Hmong LEP students in U.S. schools 9 years or less (45.4%) was greater than the non-LEP group (11.5%) (X2(3)=24.9, p<.01).

Table 5.   Student Self-Report on Time in U.S.

Reported Dictionary Proficiency

Table 6 shows the result of students’ self report of monolingual English dictionary skills. All but eight students in the study (6.3%) described their dictionary skills as “Good” or above. However, a greater percent of non-LEP students described their skills as “Very Good,” while more LEP students described their skills as “Good.” (X2(3) = 29.7; p<.01)

Table 6.   Self-Report on Using Dictionary for Non-LEP and LEP students

Dictionary Exercise Results

For the dictionary exercise, raters independently rated each item using a four-point rubric, then scores were compared. The agreement between raters for scoring the exercise was 94%. Disagreements were resolved by rater consensus after discussion.

The results of the dictionary exercise were somewhat problematic for identifying specific levels of skill. For example, in choosing a correct definition from words with multiple entries, there was an observed tendency for students to choose the first definition in the dictionary regardless of whether it fit the context of the example sentence. We believe that students’ scores do provide a good estimate of specific dictionary skills, such as alphabetizing, locating a word entry, and choosing the correct definition. Given that the highest possible score on this exercise was 4.0, the mean scores for each group demonstrate at least basic dictionary skills (see Table 7).

Table 7. Dictionary Exercise Mean Scores for Non-LEP and LEP Students

Note: One point was given for each correct answer (total possible 4). Half points were allowed on some items.

We also compared dictionary exercise results to students’ self-ratings of dictionary ability. Table 8 shows the average score on the dictionary exercise with each self-reported dictionary proficiency rating. Generally, the Hmong LEP students’ performance increased with their self-ratings of dictionary proficiency, but the non-LEP students’ performance was nearly the same across each self-rated proficiency level.

Table 8.   Students’ Self-Report on Dictionary Ability by Dictionary Exercise Means

Test Results for Accommodated vs. Non-Accommodated Performance

Initial analyses were conducted to determine whether there were possible effects for form or the order the accommodation was given. This involved looking at the mean performance on each half of the test (20 points possible on each half of the test) and the mean for the entire test for LEP and non-LEP groups by the order of form (i.e., Form A and Form B) and order of administration (accommodation given on first half vs. accommodation given on second half). Table 9 shows that there was no effect for the Form that students received with the accommodation (F(1,198) =.12; p =.73).

Table 9.   Effect of Accommodation by Form

To examine the possibility that the order in which the accommodation was administered made a difference on test performance, we compared test scores for those receiving the accommodation on the first half with the scores of those receiving the accommodation on the second half. The mean score for the LEP group with the accommodation on the second half of the test was 19.4, whereas the mean score with the accommodation on the first half was 18.3. A 2x2 ANOVA was run with Group and Order treated as fixed effects (see Table 10). The effect for the order of accommodation was not significant (F (1,198)=.043; p=.73).

Table 10.   Effect of Accommodation by Order