NCEO Logo
Bookmark and Share

The Effect of a Simplified English Language Dictionary on a Reading Test


LEP Projects Report 1

Published by the National Center on Educational Outcomes

Prepared by Deb Albus, John Bielinski, Martha Thurlow, and Kristin Liu

March 2001


Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:

Albus, D., Bielinski, J., Thurlow, M., & Liu, K. (2001). The effect of a simplified English language dictionary on a reading test (LEP Projects Report 1). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/LEP1.html


Executive Summary

This study was conducted to examine whether using a monolingual simplified English dictionary as an accommodation on a reading test with limited English proficient (LEP) Hmong students improved test performance. Hmong students were chosen because they are often not literate in their first language due to a lack of educational experiences in Hmong, which was first put into written form in the 1970s.  For these students, bilingual dictionaries are unlikely to be useful. Thus, we studied the possible usefulness of a monolingual English dictionary for these students.  Students for this study came from three urban middle schools in a large metropolitan area of Minnesota. There were a total of 69 students in the non-LEP group, and 133 students in the Hmong LEP group. The study was conducted using a randomized counter-balanced design, with a control group of non-LEP students and an experimental group of Hmong LEP students.   All students were administered two reading passages with an English dictionary available, and two passages without the dictionary, varying passage order and order of accommodation in both study groups. The students’ test performance on the two reading passages with dictionary accommodation was then compared to their test performance on the two reading passages without dictionary accommodation, using a repeated measures ANOVA procedure. Results showed that there was not a significant difference in reading comprehension scores for students in either the LEP or non-LEP group under accommodated conditions. However, it was found that intermediate level English proficiency students in the Hmong LEP group who reported using the dictionary in the accommodated condition showed a moderately significant gain. Issues discussed include student dictionary ability, dictionary interactions with test items, test development considerations, and current beliefs about dictionary accommodations and reading assessment.


Overview

A challenge facing states and districts is determining best practices for including Limited English Proficient (LEP) students in their testing systems. One recommended approach is to provide testing accommodations that are designed to reduce language barriers while not changing what a test is supposed to measure. One accommodation that is available in several states is allowing LEP students to use a dictionary during a test. However, there are few studies on the effects of dictionary use on test performance and score validity.

The way in which dictionaries are used as an accommodation on reading tests varies across the country. In a survey of state assessment directors for 1998-99, Rivera, Stansfield, Scialdone, and Sharkey (2000) identified 21 states that allowed the use of bilingual dictionary accommodations and only 3 states that specifically prohibited them. Among the states that allowed bilingual dictionaries, 11 states allowed them on all assessment components and 10 states allowed them on some of the assessment components (Rivera, et al., 2000). According to a survey conducted by the Council of Chief State School Officers (Olson, Bond, & Andrews, 1999), which covered 1997-98 assessments, only two of these states allowed bilingual dictionaries and the third allowed an English language dictionary.

Current views about the appropriateness of dictionary accommodations in testing differ among researchers, educators, and LEP students. For example, there are both research and opinions that support using dictionary accommodations. Researchers have shown that unfamiliar vocabulary may cause difficulty in understanding items on tests (Garcia, 1991) and that using dictionaries can help students’ reading comprehension (Goyette, 1996; Laufer & Hadar, 1997) and equalize skill assessments for LEP students (Rivera & Vincent, 1997). Further, LEP students and English as a Second Language (ESL) and Bilingual Education teachers have requested them in testing situations because dictionaries are an accommodation that students use in classes every day and will most likely use throughout their lives (Bensoussan, 1983; Liu, Spicuzza, Erickson, Thurlow & Ruhland, 1997; Quest, Liu, & Thurlow, 1997).

There is also research and opinion that opposes using dictionary accommodations. It is a common belief among some researchers and test developers that any alteration to the standard administration necessarily alters the validity of the test score. Others argue that dictionaries should not be allowed because they may negatively affect the validity of a test (Rivera & Stansfield, 1998; Spolsky, 1997). Other reasons given against using dictionaries are that students will need more time with dictionaries, that students may over rely on them (Roizen, 1984), and that they have not been found to significantly affect reading comprehension test scores of individuals learning English as a foreign language (Bensoussan, 1983; Nesi & Meara, 1991).

One argument against dictionary use is that it negates the role of specific vocabulary knowledge as an essential component of reading ability (Bensoussan, 1983). However, Bensoussan argues that a student using a dictionary still needs to be able to successfully choose the right meaning of a word based on the context of a passage in order to correctly answer a test question. Contextual clues may not always be readily available in a passage to infer meanings of unknown words (in tests or in everyday reading). Therefore, the availability of a dictionary does not guarantee understanding of unknown words with or without sufficient contextual clues.

According to some researchers, readers need to comprehend a certain percentage of a text to be able to infer meaning of unknown words. Laufer (1997) suggested 95% text comprehension (understanding of 3,000 word families) before reading skills in a reader’s first language will aid reading in the second language, including inferring meanings of words from context. Hirsh and Nation (1992) suggested 98% text comprehension for pleasure reading, requiring readers to have knowledge of approximately 5,000 word families. Although reading language skills may be more developed in the second language of some students than in their first language, adequate text comprehension still requires understanding of a high percentage of the words. Also, the extent of word family knowledge needed in a reader’s “sight” or “automatic” vocabulary suggests that dictionary use during a reading comprehension test may not greatly enhance performance if students’ overall threshold vocabulary is too low. Observed problems with vocabulary thresholds have led some researchers to conclude that students’ problems with comprehension are basically lexical rather than due to lack of reading strategies (Haynes & Baker, 1993). However, a dictionary would not be able to compensate a student with great gaps in vocabulary knowledge, therefore its use may be more beneficial for students whose proficiency is near the level required for comprehending a text.

In some studies conducted with students learning English as a Foreign Language (EFL) in Israel, it was found that dictionaries were more useful for students with a moderate level of English proficiency and some dictionary skills (Bensoussan, 1983; Laufer & Hadar, 1997), and that they did not benefit students with very low or very high proficiency. Therefore, the accommodation may not benefit the lower proficiency students who would need the most help (Shepard, Taylor, & Betebenner, 1998).

The majority of dictionary accommodation reading studies have been conducted in EFL settings, thus it is important to study dictionary use in an English as a Second Language setting, especially where most students are not literate in their first spoken language. Further, because most dictionary studies have used EFL tests, our goal was to seek evidence to either support or refute the use of dictionary accommodations in large-scale tests in the United States.

Researchers who support dictionary accommodations favor the use of bilingual or bilingualized dictionaries (English dictionaries with a native language translation) for both LEP students in the United States and LEP students overseas (Laufer & Hadar, 1997; Rivera & Stansfield, 1998). Laufer and Hadar (1997) found that, in general, monolingual dictionaries were the least useful accommodation on tests of English as a foreign language. However, in the United States where students are in an English as a second language setting, bilingual dictionaries may not always be the best choice, particularly for individual students who may not be literate or may not have received any education in their first language. Despite being classified as LEP, the language that these students read and write the most fluently may, in fact, be English.

The quality and appropriateness of dictionaries (including the quality of their translations), whether bilingual or monolingual, vary greatly. Some bilingual dictionaries only give word for word translations or incomplete meanings, while others give definitions. English monolingual dictionaries, on the other hand, sometimes provide only the most basic definition of a word. If students are allowed to bring their own dictionaries to a test, some may be at a disadvantage because of the specific dictionary they choose. English language dictionaries tend to give more detailed explanation about the function of words in context; however, the number and depth of entries and illustrations are not equal across dictionary versions. There are also differences between regular and simplified English dictionaries. Abedi, Lord and Plummer (1997) studied the impact of simplified English in the test items themselves on the National Assessment of Educational Progress (NAEP), and found that simplified English on tests had a positive effect for all students, not just for those who are LEP. If simplified English has been found to affect students’ performance on tests, it is likely that simplified English in dictionaries could also have an effect.

 

Goal of the Study

This study was conducted to examine the possible effects of using a monolingual simplified English dictionary as an accommodation on a reading test with limited English proficient Hmong students.

 

Research Questions

Four primary research questions were posed for this study.

1.  Do Hmong LEP students provided with a simplified English dictionary perform better than when the dictionary is not provided?

2.  How does use of a simplified English dictionary by Hmong LEP students affect the reliability of test scores?

3.  What are the characteristics of Hmong LEP students whose scores are most affected by the use of a simplified English dictionary accommodation?

4.  Do students (LEP and non-LEP) want to use a simplified English dictionary as a test accommodation?


Method

Participants

Students for this study came from three urban middle schools in a large metropolitan area of Minnesota. There were a total of 69 regular education students in the non-LEP group and 133 students in the Hmong LEP group. No restrictions were placed on the backgrounds of students in the non-LEP groups other than they not be LEP students or students receiving special education services. Some students in the non-LEP group were from Hmong language backgrounds, but none of them was receiving services for limited English proficiency. Students from both groups were recruited at all three schools; however, only two of the schools provided students from both groups. The third school chose only to allow participation for its Hmong LEP student population. Data on an economic status indicator (receiving free or reduced lunch) were collected for both the Hmong LEP group and the non-LEP group. The students in the two testing groups were comparable.

Schools used similar ESL level designations (1-5), but the specific description of each level varied across sites. The levels indicate the range of students within ESL classes, from beginning to high levels of English proficiency. The number and percent of LEP students by level are presented in Table 1 (1=lowest proficiency level).

 

Table 1. Number and Percent of LEP Students by ESL Level

 

Group

ESL Level

 

Total

1

2

3

4

5

 

N

 

5

 

17

 

36

 

62

 

10

 

130

 

%

 

3.8%

 

13.1%

 

27.7%

 

47.7%

 

7.7%

 

100%

Note: Level of English proficiency as assigned by school personnel.

 

Design

The study was conducted using a randomized counter-balanced design, with a control group of regular education students and an experimental group of Hmong LEP students. Students were administered two reading passages with the English dictionary available, and two passages without the dictionary. The students’ test performance on the two reading passages with dictionary accommodation was then compared to their test performance on the two reading passages without dictionary accommodation, using a repeated measures ANOVA procedure.

The passages were designed to parallel Minnesota’s Basic Standards Reading Test, which is part of the state graduation exam. However, the items used for this study had not been used or equated with officially administered tests. The passages represent sample passages that had been reworked by an assessment specialist for use in the Minnesota Assessment Project study of bilingual reading test items (Anderson, Liu, Swierzbin, Thurlow, & Bielinski, 2000). In addition, an LEP graduation standards specialist at the Minnesota Department of Children, Families and Learning and a bilingual adult member of the ESL community provided advice on cultural background in the development of the test passages. The two halves of the test were divided into Form A and Form B. The passages were assigned to forms so that Form A and Form B had the same overall difficulty. Table 2 shows the study design, which is a modification of one design presented by Thurlow, McGrew, Tindal, Thompson, Ysseldyke, and Elliott (2000).

 

Table 2. Study Design

 

Hmong LEP Students

English-Speaking Regular Education Students

Group 1

Group 2

Group 3

Group 4

Group 1

Group 2

Group 3

Group 4

 

Test 1

 

Form A

 

Form B

 

Form A

 

Form B

 

Form A

 

Form B

 

Form A

 

Form B

 

 

WithDictionary

 

WithDictionary

 

WithoutDictionary

 

WithoutDictionary

 

WithDictionary

 

WithDictionary

 

WithoutDictionary

 

WithoutDictionary

 

Test 2

 

Form B

 

Form A

 

Form B

 

Form A

 

Form B

 

Form A

 

Form B

 

Form A

 

 

WithoutDictionary

 

WithoutDictionary

 

WithDictionary

 

WithDictionary

 

WithoutDictionary

 

WithoutDictionary

 

WithDictionary

 

WithDictionary

 

Test Instrument

Form A and Form B each contained two passages. Each passage was 900-1,040 words in length. For each reading passage, the examinee was asked 10 multiple-choice questions that addressed both literal and inferential comprehension. The test used in this study was previously used in a study on bilingual translation accommodations by researchers of the Minnesota Assessment Project. Evaluation of test score reliability indicated that it had the same or higher internal consistency as the actual Minnesota Basic Standards reading test for most of the test groups (Anderson et al., 2000).

A dictionary exercise, developed by research staff with ESL teaching backgrounds, was also administered to determine each student’s proficiency with the simplified English dictionary. The exercise was composed of four questions. Two questions asked students to provide written definitions of words appropriate to context sentences that were provided to them. A third question was aimed specifically at alphabetizing skill. The fourth question was designed to determine students’ ability to use the dictionary for other information about parts of speech. A four-point scoring rubric was developed for raters to evaluate and score the exercise.

 

Dictionary Accommodation

A simplified English dictionary was used, as opposed to a Hmong Bilingual dictionary because most Hmong background LEP students have limited literacy in their first language. The dictionary chosen for the study was The American Heritage English as a Second Language Dictionary (Houghton Mifflin Company, 1998). This simplified English dictionary was chosen for several reasons. First, the dictionary for the study had to include more difficult academic words specific to the test passages (e.g., cum laude, precocious), and had to provide a range of definitions for words with multiple meanings (e.g., stand, produce). It also had to present the definitions at a simplified level of English for clarity, and yet fit the needs of a range of proficiency levels among LEP students.

Some of the other available ESL dictionaries had more pictorial content, but had fewer of the words that had been identified as potentially problematic in the study passages. Other ESL dictionaries were too basic, providing only one meaning for words with multiple meanings, and often giving meanings that did not match the meaning of words in the test passages. Also, for one of the vocabulary items on the test, another dictionary provided a definition with an example sentence that could have misled students to choose an incorrect answer. The example sentence included extra descriptive information that was contradictory to the correct test response.

 

Study Procedure and Timing

Students were assembled into either an auditorium or classroom to take the test. The size of the groups ranged from 11 to 58 students per room. Students were first asked to fill out a brief pre-test questionnaire about language background to provide self-ratings of their English and Hmong proficiency in several modalities: speaking, listening, and reading.

Next, students were administered one half of the test; some of the students getting the dictionary accommodation and some not. Students were allowed as much time as they needed to complete each half of the test, having been given a general time limit of two hours. After completing the first half of the test, the student raised his or her hand and a test administrator started the student on the second half. If the student had the dictionary for the first half, dictated by the color of the test cover, the administrator removed it; if the student took the first half without the dictionary, then it was provided to the student on the second half. During test administration, staff recorded start and finish times on the test covers at the end of each half of the test while providing and removing dictionaries for students’ use so that the time taken by students with and without dictionaries could be tracked. Immediately after completion of the whole test, students were given a post-test questionnaire about dictionary use during the test, their opinions on possible usefulness of an English dictionary on a reading test, and other background information on dictionary use and instruction in the classroom. A short dictionary exercise also was given after the post-test survey to determine levels of student ability in using a dictionary.

Students were allowed approximately two-and-one-half hours to complete all materials. This time allotment was determined on the basis of schedule limitations in the schools. Following completion of the test, students either stayed in the same room or simply returned to class, depending on school requests. For example, at one site, the students who completed the test were given other activity sheets to work on while waiting for other students to finish before being released back to class. At other sites, students were allowed to leave after they completed the study materials. It is uncertain whether these varying procedures had any effect on test results.


Results

Proficiency in English and Hmong

On student pre-test questionnaires, the control group and the Hmong LEP group answered a series of questions on language proficiency for speaking, listening, and reading in English and Hmong, and the length of time they had spent in U.S. schools. Most students in the Hmong LEP group reported higher reading ability in English than in Hmong (see Table 3), with most students rating their English reading ability as either “well” or “pretty well.” For speaking and understanding spoken English/ Hmong, the majority of the group described themselves between “Well” and “Pretty well” for English, and between “Very well” and “Pretty well” for Hmong.

 

Table 3.   Hmong LEP Group Self-Report for English and Hmong Reading and Speaking Ability

Hmong LEP
students

 Very Well

 Pretty Well

     Well

  Not Very
      Well

  Not well
     at all

    Total

N

%

N

%

N

%

N

%

N

%

N

%

English

Speak

26

20%

64

49%

37

28%

  5

4%

  0

0%

132

100%

Read

14

11%

67

51%

44

33%

  7

5%

  0

0%

132

100%

Hmong

Speak

69

52%

44

33%

15

11%

  5

4%

  0

0%

133

100%

Read

10

8%

  8

6%

25

19%

50

38%

40

30%

133

100%

 

In the non-LEP group (see Table 4), as might be expected, the majority reported their level of English proficiency as “Very well” in both modalities. This group did include some students with Hmong language background who were not limited English proficient. For these students, some reported a range of “Well” to “Very well” in understanding and speaking Hmong. However, very few reported reading Hmong above “Not very well.”

 

Table 4   Non-LEP Group Self-Report for English and Hmong Reading and Speaking Ability

Non-LEP
students

 Very Well

 Pretty Well

   Well

   Not Very
      Well

    Not well
      at all

     Total

N

%

N

%

N

%

N

%

N

%

N

%

English

Speak

55

80%

10

15%

4

6%

  0

0%

  0

0%

69

100%

Read

50

73%

14

20%

5

7%

  0

0%

  0

0%

69

100%

Hmong

Speak

10

14.5%

10

14.5%

2

3%

  1

1%

46

67%

69

100%

Read

  0

0%

  4

6%

5

7%

12

17%

48

70%

69

100%

 

Time in U.S. Schools

Table 5 shows the length of time in U.S. schools for each group. The percentage of Hmong LEP students in U.S. schools 9 years or less (45.4%) was greater than the non-LEP group (11.5%) (X2(3)=24.9, p<.01).

 

Table 5.   Student Self-Report on Time in U.S.

 

1-3 Years

4-6 Years

7-9 Years

>9 Years

Total

Count

%

Count

%

Count

%

Count

%

Count

%

Non-LEP

0

0%

  1

1.4%

  7

10.1%

61

88.4%

69

100%

LEP

4

3%

25

18.9%

31

23.5%

72

54.5%

132

100%

 

Reported Dictionary Proficiency

Table 6 shows the result of students’ self report of monolingual English dictionary skills. All but eight students in the study (6.3%) described their dictionary skills as “Good” or above. However, a greater percent of non-LEP students described their skills as “Very Good,” while more LEP students described their skills as “Good.” (X2(3) = 29.7; p<.01)

 

Table 6.   Self-Report on Using Dictionary for Non-LEP and LEP students

How good are you at using an English only Dictionary?

Very good

Pretty good

Good

Not very good

Total

Count

%

Count

%

Count

%

Count

%

Count

%

  Non-LEP

44

63.8%

23

33.3%

  2

2.9%

0

0%

  69

100%

LEP

38

29.9%

49

38.6%

32

25.2%

8

6.3%

127

100%

 

Dictionary Exercise Results

For the dictionary exercise, raters independently rated each item using a four-point rubric, then scores were compared. The agreement between raters for scoring the exercise was 94%. Disagreements were resolved by rater consensus after discussion.

The results of the dictionary exercise were somewhat problematic for identifying specific levels of skill. For example, in choosing a correct definition from words with multiple entries, there was an observed tendency for students to choose the first definition in the dictionary regardless of whether it fit the context of the example sentence. We believe that students’ scores do provide a good estimate of specific dictionary skills, such as alphabetizing, locating a word entry, and choosing the correct definition. Given that the highest possible score on this exercise was 4.0, the mean scores for each group demonstrate at least basic dictionary skills (see Table 7).

 

Table 7. Dictionary Exercise Mean Scores for Non-LEP and LEP Students

 

 

Mean

 

N

Std. Deviation

 

Minimum

 

Maximum

Non-LEP

3.22

  67

.69

2

4

LEP

2.72

128

.83

0

4

Total

2.89

195

.82

0

4

Note: One point was given for each correct answer (total possible 4). Half points were allowed on some items.

We also compared dictionary exercise results to students’ self-ratings of dictionary ability. Table 8 shows the average score on the dictionary exercise with each self-reported dictionary proficiency rating. Generally, the Hmong LEP students’ performance increased with their self-ratings of dictionary proficiency, but the non-LEP students’ performance was nearly the same across each self-rated proficiency level.

 

Table 8.   Students’ Self-Report on Dictionary Ability by Dictionary Exercise Means

 

Dictionary Exercise

 Non-LEP Self Rated Ability in Using Dictionary

Very good

Pretty good

 Good

Not very good

Not good at all

 Total

 Mean

 3.21

 3.24

 3.25

 None

 None

 3.22

N

42

23

2

None

None

67

Std. Deviation

.75

.62

.35

None

None

.69

 

 Hmong LEP Self Rated Ability in Using Dictionary

Very good

Pretty good

 Good

Not very good

Not good at all

 Total

 Mean

 3.06

 2.74

 2.3

 2.81

 None

 2.72

N

35

47

32

8

None

122

Std. Deviation

.73

.82

.82

.75

None

.83

 

Test Results for Accommodated vs. Non-Accommodated Performance

Initial analyses were conducted to determine whether there were possible effects for form or the order the accommodation was given. This involved looking at the mean performance on each half of the test (20 points possible on each half of the test) and the mean for the entire test for LEP and non-LEP groups by the order of form (i.e., Form A and Form B) and order of administration (accommodation given on first half vs. accommodation given on second half). Table 9 shows that there was no effect for the Form that students received with the accommodation (F(1,198) =.12; p =.73).

 

Table 9.   Effect of Accommodation by Form

 

 Mean Score

 Std. Deviation

 Group

 

 Non-LEP

Dictionary on Form A

25.9

7.09

Dictionary on Form B

26.5

7.98

 LEP

Dictionary on Form A

18.8

6.36

Dictionary on Form B

18.9

7.15

To examine the possibility that the order in which the accommodation was administered made a difference on test performance, we compared test scores for those receiving the accommodation on the first half with the scores of those receiving the accommodation on the second half. The mean score for the LEP group with the accommodation on the second half of the test was 19.4, whereas the mean score with the accommodation on the first half was 18.3. A 2x2 ANOVA was run with Group and Order treated as fixed effects (see Table 10). The effect for the order of accommodation was not significant (F (1,198)=.043; p=.73).

 

Table 10.   Effect of Accommodation by Order

 

 Mean Score

 Standard Deviation

 

Group

 

 

Non-LEP

Dictionary on 1st half

26.5

8.07

Dictionary on 2nd half

25.9

6.92

 LEP

Dictionary on 1st half

18.3

6.65

Dictionary on 2nd half

19.4

6.87

The means shown in Table 11 indicate that regardless of the order the accommodation was given, students performed better on the first half of the test than on the second half (F(1,200) = 22.1;p<.01).

 

Table 11.   Student Performance on Both Halves of Test

 

 1st Half mean

 2nd Half mean

 Non-LEP

 13.8

 12.4

 LEP

   9.9

  9.0

The main research question in the study was whether the use of a simplified English dictionary would improve test performance, and whether improvement would be greater for the LEP group than for the non-LEP group. A repeated measures ANOVA was run wherein the within group variable had two levels: (1) test score without accommodation, and (2) test score with accommodation. The between group variable was LEP status (LEP vs. non-LEP). The difference between these two groups was significant (F 1,200=49.7; p<.01). The effect for the accommodation was not significant (F 1,200=.15; p=.70), and the interaction also was not significant (F 1,200=.39; p=.54). Table 12 shows the cell means and 95% confidence interval for each condition by group. The non-LEP group performed the same under both conditions whereas the LEP group performed slightly better with the dictionary accommodation.

 

Table 12. Student Performance With and Without Dictionary Accommodation

 

 

 95% Confidence Interval

 

Lower Bound

 

Upper Bound

Mean

Std. Error

 

Group

Non-LEP

Dictionary

13.1

.48

12.13

14.02

 

No Dictionary

13.1

.46

12.23

14.03

 

LEP

Dictionary

   9.6

.34

   8.88

10.24

 

No Dictionary

   9.3

.33

   8.66

   9.96

 

 

Because not every student in either the control or Hmong LEP group self-reported using the dictionary accommodation when it was provided, a second analysis was conducted for only those students who reported that they had used the accommodation. In this analysis we compared the performance with and without the accommodation by self-reported English language proficiency. The analysis indicated that LEP students who self-reported lower English proficiency (“well” to “not well at all”) did not benefit as much from using the dictionary accommodation as LEP students who self-reported higher English proficiency (“pretty well” to “very well”). This high English proficiency group scored an average 1.2 points higher using the accommodation compared to performance without the accommodation; the lower proficiency group performed nearly the same under both conditions (see Table 13). This interaction between proficiency level and condition was significant ( F(1,97)=4.78; p=.03).

 

Table 13.   Student Performance by Accommodation Conditions and Self-Reported English Reading Ability

 

 

Mean

 

Standard

   Error

    95% Confidence Level

Lower
Bound

Upper

Bound

English Reading

Ability

(dichotomized pre-test

Question 6)

Very to

Pretty well

Dictionary

10.759

.481

9.805

11.713

No Dictionary

9.569

.483

8.610

10.528

Well to

Not well*

Dictionary

8.220

.572

7.085

9.354

No Dictionary

8.488

.574

7.348

9.628

A similar analysis was conducted for the non-LEP group using self-reports of dictionary use during the test. It showed that reading score means in this non-LEP group were not affected by the dictionary accommodation (F(1,67) = .61; p=.44).

 

Affect of Accommodation on Test Reliability

Test score reliability was estimated using Chronbach’s alpha, a measure of internal consistency. This statistic can range from 0.0 (completely unreliable) to 1.0 (perfectly reliable). Achievement tests are designed to obtain a reliability of about .85. We computed the reliability for each half of the test (20 questions each) taken under both accommodated and non-accommodated conditions. Because reliability is affected by test length, we anticipated that the reliability for each half of the test would be slightly lower than .85. Table 14 below shows Chronbach’s reliability coefficient for each half of the test by condition, accommodated and non-accommodated.

Table 14  Chronbach’s Alpha for Test Reliability

 

Test Items

Condition

First half

Second half

Dictionary

.79

.76

No dictionary

.73

.79

Performance on Test’s Vocabulary Items

A closer analysis was conducted on the test results of specific vocabulary items to determine whether performance gains with the dictionary accommodation could be attributed to performance on two questions that required only the definition of a term. There was one vocabulary item on each half of the test so that one half of the students would have encountered each word under accommodated conditions. The item questions were as follows:

Items 8 and 28              The term precocious used in the title means:
Items 5 and 25             As used in this article, the word produce is best defined as:

The dictionary entries available to students (if in the dictionary accommodation condition) are reproduced below (without pronunciation information):

Precocious. adj. Showing mental skills or abilities at an earlier age than is normal: He was a precocious child who learned to read at three. …

Produce. Tr.v. 1. To bring forth (sthg.): Seeds grow up to produce plants. 2. To create (sthg.) by mental or physical effort: produce a painting. 3. To manufacture (sthg.): produce parts for machines. 4. To cause (sthg.) to exist: Industrial growth produced a new kind of business organization. 5. To supervise and finance the public presentation of (a movie, for example): produce a play.-n. (U) Farm products, especially fruits or vegetables: Excuse me, where can I find the produce?

It should be noted that “precocious” was not embedded in any of the text of the passage but was presented somewhat independent of the context in the title, “Precocious preteen is youngest college graduate.” Students would need to have understood its part of speech and rhetorical function in the title and connected that to the overall description of the boy’s qualities in the passage to infer the meaning of the word. The “produce” item, in contrast, occurred with relatively high frequency in the passage, embedded in sentences with numerous descriptors of fruit and vegetables.

To investigate whether the performance on these vocabulary items was enhanced by dictionary use, thus potentially responsible for part of the average gain, we calculated chi-square results of number correct by school-assigned ESL level and accommodation condition with these vocabulary items (see Table 15). Only LEP students who reported using the dictionary for at least a few words were included.

 

Table 15.    Percentage Passing ‘Precocious’ Item.

 

Precocious Item Percent Correct

 

ESL Level

1,2,& 3

4 & 5

 

Dictionary

43.5%

(N=10)

61.54%

(N= 16)

 

No Dictionary

52.2%

(N= 12)

33.3%

(N= 8)

 Statistics for table: (X2(1)=2.10; p=.15)

Table 15 shows that for the item asking students to select the correct definition for “precocious,” lower ESL level students did slightly less well on this item with the accommodation available than did those without it. For higher ESL level students, 50% more students got the correct answer with the accommodation (N=16) than without it (N=8). It is uncertain why the lower level group without the accommodation was able to do better than the higher level group (52.2% vs. 43.5%). Even when the correct single meaning dictionary entry was available, some students did not choose the correct answer, regardless of school assigned ESL level.

Table 16 shows that lower ESL level students did slightly better with the dictionary accommodation on the “produce” item than without the accommodation. The higher level students did slightly better without the accommodation than with the accommodation.

 

Table 16.   Percent Passing ‘Produce’ Item

 

Produce Item Percent Correct

 

ESL Level

1,2,& 3

4 & 5

 

Dictionary

 

39.3%

(N= 11)

 

41.2%

(N= 10)

 

No Dictionary

 

36.4%

(N= 8)

 

57.7%

(N= 15)

 Statistics for table: (X2(1)=1.38; p=0.24)

Reported Dictionary Use

Table 17 shows results of post-test self reports from students about their overall dictionary use during the test. Students in the Hmong LEP group reported using the dictionary more than the non-LEP group. For new words, the Hmong LEP group reported more use (73%) than the non-LEP group (46%). However, the percentage of students who used the dictionary to check words they already knew was similar for the Hmong LEP group (12%) and the non-LEP group (7%). The third question about dictionary use was included in case students used the dictionary in a way that was different from the ways we thought it would be used during the test (e.g., looking up part of speech information, pronunciation, etc.). We are not able to say how the students defined “other information” in this question, but this may be a useful area for further study.

 

Table 17.   Student Self-Report on Dictionary Use During Test

 

Questions

 

Group

 

Yes

 

No

 

Total

N

%

N

%

N

%

 

Did you look up new words?

Non-LEP

 

32

 

46%

 

  37

 

54%

 

  69

 

100%

 

LEP

 

93

 

73%

 

  34

 

27%

 

127

 

100%

Did you look up words you already knew?

Non-LEP

 

  5

 

7%

 

  64

 

93%

 

  69

 

100%

 

LEP

 

15

 

12%

 

112

 

88%

 

127

 

100%

Did you look up other

information using the dictionary?

Non-LEP

 

15

 

22%

 

  54

 

78%

 

  69

 

100%

 

LEP

 

68

 

53.5%

 

  59

 

46.5%

 

127

 

100%

We analyzed self-reports according to which passage(s) students said they used the dictionary. Although the Hmong LEP group reported a larger overall percentage of dictionary use for each passage than the non-LEP group, the two groups showed the same pattern of use for the four passages (see Table 18).

 

Table 18. Student Self-Report of Using Dictionary by Reading Passages in Test

 

Hmong LEP

Non-LEP

N

%

N

%

Passage 1

30

24%

  4

   6%

Passage 2