Delphi Study of Instructional Strategies for English Language Learners with Disabilities: fonRecommendations from Educators Nationwide ELLs with Disabilities Report 21 Martha Thurlow, Vitaliy Shyyan, Manuel Barrera, Kristi Liu September 2008 All rights reserved. Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as: Thurlow, M., Shyyan, V., Barrera, M., & Liu, K. (2008). Delphi study of instructional strategies for English language learners with disabilities: Recommendations from educators nationwide (ELLs with Disabilities Report 21). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
The No Child Left Behind (NCLB) Act of 2001 emphasizes the importance of effective and accountable education for all students, including English language learners (ELLs) with Individualized Education Programs (IEPs). National studies indicate an increase in the population of school-age English language learners throughout the U.S.; even the most modest estimates suggest that 9% of the ELL population is also enrolled in special education programs (Zehler, Fleischman, Hopstock, Pendzick, & Stephenson, 2003). The actual number may be higher due to varying identification and categorization approaches in different school settings. Lower numbers also may reflect the absence of legislation identifying ELLs with disabilities as a separate subgroup. Investigating instructional strategies specifically directed toward the academic achievement of ELL students with disabilities is an important task for educators given that these students are found to exhibit dual and interactive characteristics of emerging English proficiency and disability-related educational needs (Barrera, 2008; McArdle, Mele-McCarthy, & Leos, 2005; Ortiz, 1997). State content standards typically do not contain specific instructional strategies (Albus, Thurlow, & Clapper, 2007), and research literature on instructional practices for this student population is scarce (Gersten & Baker, 2000; Gersten, Baker, & Marks, 1999; Gersten, Baker, Shanahan, Linan-Thompson, Collins, & Scarcella, 2007). Educators may generate instructional strategies based on their professional experiences and other sources of information. Therefore, they may be valuable resources when it comes to the question of improving instruction for this student category. Research on educators’ views of instructional strategies, when obtained through a process that allows refinement and access to recommendations of other educators, may contribute to improving instruction for these students. Reading, mathematics, and science are three content areas in which states are or will be required to show academic progress in grade-level standards for all students including English language learners with disabilities. Academic achievement in science education is now beginning to be considered a potential addition to the accountability requirements of NCLB (Commission on No Child Left Behind, 2007). Middle school is the time when reading, mathematics, and science curricula are most likely to challenge ELLs with and without disabilities through greater academic demands. Indeed, dropout rates among these and other students increase at the middle school level (Mikow-Porto, Humphries, Egelson, O’Connell, & Teague, 2004). Therefore, it seems particularly important to examine how educational practice may generate effective strategies for improving the academic achievement of ELLs with disabilities. This study is part of national research over the past seven years at the National Center on Educational Outcomes focused on identifying and validating instructional strategies for ELLs with disabilities (Shyyan, Thurlow, & Liu, 2008; Thurlow, Albus, Shyyan, Liu, & Barrera, 2004). In recent work (Barrera, Shyyan, Liu, & Thurlow, 2008), educators from five states with large ELL populations and five states with small ELL populations generated sets of reading, mathematics, and science instructional strategies and weighted their importance in focus group-like settings using the Multi-Attribute Consensus Building method (Vanderwood & Erickson, 1994). The research described here is a confirmatory Delphi study of the strategies identified in the Barrera et al. (2008) study. Teachers similar to those who participated in the earlier study (Barrera et al., 2008) were polled about strategies identified in the earlier study. This polling was accomplished by using an online survey about the strategies previously identified. Research Question The following research question served as the focus of this study:
Definitions The following definitions served as key descriptors and variables in this study: are students whose primary or native language is not English, who have difficulty in using English (i.e., reading, writing, speaking, and listening) and who have a special education plan, based on their unique needs, containing a statement of their present level of performance and academic functioning, educational needs, and goals. An instructional strategy is a purposeful activity to engage learners in acquiring new behaviors or knowledge. An instructional strategy should have clearly defined steps or a clear description of what the teacher does. This confirmatory Delphi study was designed to reinforce or further refine the findings in a previous study conducted on-site at schools identified as demonstrating Adequate Yearly Progress (AYP) in the academic achievement of ELLs with disabilities (Barrera et al., 2008). We replicated the process used in the previous study to choose a stratified random sample of schools in like states. An online survey was administered to teachers and other specialists at identified and chosen schools. Survey results are compiled here and reported. Sampling Schools that served as research sites around the country were selected using a multi-stage sampling process. At the beginning of the procedure, the demographic composition of ELLs with and without disabilities was reviewed in all 50 states using data from the 2004 National Clearinghouse for English Language Acquisition Web site. Ten states each with the highest and lowest ELL populations were randomly selected from the pool of states. The overall ELL student population was used because ELLs with disabilities are not always reported in every state. This process yielded a group of 10 states, 5 of which had low numbers of ELLs and 5 of which had high numbers of these students. The process of selecting schools within the states ultimately yielded participating schools in four states with high ELL populations and five states with low ELL populations. Figure 1 describes the geographic distribution of the participating states. Figure 1. Locations of Study Sites
After identifying states, we analyzed school performance data for middle and junior high schools. Based on the data, we selected five top schools that made AYP, the No Child Left Behind measure of a year-to-year student achievement, and served the largest populations of ELLs. These schools were contacted with an invitation to participate in the study. The first schools to agree to participate in the study were selected to be in the research study sites. Procedure The study employed a modified version of the Delphi survey method to determine the importance of previously identified instructional strategies (Barrera et al., 2008) and generate additional strategies with further consideration of their importance. The Delphi method is a structured process of using a series of surveys to gather combined input from a group of persons with expertise in a specific area or from a specific population. This method has been used in the social science and public health fields since the mid-1970s (Adler & Ziglio, 1996). The Delphi method allows experts to give their own informed opinion on a particular issue in the first round. In the next round, the input is compiled and returned to research participants who have a chance to respond to further questions, consider the input from other participants, and revise their own comments if preferred. This Delphi study took place entirely via electronic mail. In the first round of the Delphi survey, educators were instructed to assign importance levels to a set of reading, mathematics, and science instructional strategies on a scale from 1 to 5 (see Table 1). Educators also were offered the opportunity to comment on each strategy if they preferred to do so. At the end of each content area section of the survey, research participants were asked to generate additional instructional strategies that they considered to be important in content instruction for ELLs with disabilities. At the end of the first round of the Delphi study, participants also were asked to complete a brief demographic survey. This allowed for further analyses of data by demographic categories. Table 1. Scale for Determining Strategy Importance
At the end of the first round of the Delphi survey, importance ratings from each group of educators were averaged. These means were incorporated into the second round questions, along with individual ratings, educators’ comments, and newly-generated instructional strategies. The demographic section was excluded from the second round because the information had been previously collected. Thus, research participants had an opportunity to rate the importance of each strategy one more time with the consideration of their colleagues’ inputs. Participants Each Delphi review was conducted with groups of two to eight educators. Another important criterion for choosing research participants was that they were to have had direct or indirect responsibilities for the instruction or related services for ELLs with disabilities. The first round of the study included 39 educators (18 professionals from states with high ELL populations and 21 professionals from states with low ELL populations). Three of the educators were unable to participate in the second round of the study; therefore, the second round sample had 36 educators (16 professionals from states with high ELL populations and 20 professionals from states with low ELL populations). The first round version of the Delphi survey included several demographic questions at the end to enable further data analysis by demographic categories. The first demographic question focused on research participants’ job titles. Figure 2 summarizes these results. Of all survey participants, 17.9% self-identified as ESL teachers, 17.9% as special education teachers, 48.7% as general education teachers, and 2.6% as school administrators. Another 7.7% identified themselves as holding other positions such as "curriculum coaches" or academic intervention specialists. The remaining 5.1% did not provide an answer to this question. Figure 2. Educational Position of Participants
The second demographic question asked about educators’ professional experience. Figure 3 shows that study participants overall were largely experienced teachers. A majority of the 39 participants (66.7%) had more than 10 years of total teaching experience. An additional 20.5% of educators had between 6 and 10 years of professional experience. Relatively few educators were comparatively new to the profession, with 7.7% of participants having 1 to 5 years of experience. No study participants reported being first-year teachers and 5.1% of survey respondents did not provide an answer to the question. Figure 3. Participants’ Years of Professional Educational Experience
Figure 4 illustrates the length of time educators spent in their current positions. Survey responses show that 5.1% of participants had spent less than 1 year in their current positions, 33.3% held their current positions between 1 and 5 years, and equal percentages of respondents, 28.2%, were employed in their current positions from 6 to 10 years, and over 10 years; 5.1% did not respond to this question. Figure 4. Time in Current Position
Finally, research participants commented on the types of students they served. Figure 5 shows that 66.7% of educators reported serving ELLs, 74.4% of educators specified that they taught special education students, 43.6% of educators reported delivering instruction for ELLs with disabilities, and 64.1% of educators stated they worked with general education students. It should be noted here that most educators reported serving more than one type of students identified in the survey question. Figure 5. Categories of Students Served
Three sets of content strategies (20 reading strategies, 12 mathematics strategies, and 14 science strategies) were incorporated in both Delphi rounds. These are defined in Appendices A-C. Descriptive statistics, including ranges, modes, standard deviations, and means of importance ratings, were computed for the two rounds of the survey to obtain practitioners’ overall perceptions of importance of core instructional strategies. These data are fully described in Appendices D-F. The top strategies for each content area (7 reading strategies, 8 mathematics strategies, and 9 science strategies)—those that received mode values of 5, the highest importance rating—are highlighted here. Additional strategies generated by participants are also presented; these follow the top rated instructional strategies. The additional strategies are defined in Appendix G. Finally, statistically significant differences in ratings by demographic categories are presented. Reading Instructional Strategies Table 2 shows the descriptive statistics results for the top reading strategies. These and other strategies are defined in Appendix A. As noted, the top seven reading instructional strategies (those with modes of 5, which is the highest importance scale value) were "relating reading to student experiences," "using visuals," "background knowledge," "chunking and questioning aloud," "pre-reading survey of text," "vocabulary application," and "retelling with groups." These strategies’ means range between 4.34 and 4.86 reflecting the "very important" continuum on the importance scale. Appendix D contains statistical results for the 20 reading strategies generated in both Delphi survey rounds. Table 2. Top Reading Instructional Strategies
Mathematics Instructional Strategies The descriptive statistics results for the top eight mathematics strategies are summarized in Table 3. Appendix B highlights definitions of these and other mathematics strategies identified in the Delphi process. As shown in Table 3, the top mathematics strategies with modes of 5 were "relating math to students’ real life experiences," "using manipulatives," "daily re-looping," "modeling," "using visuals," "multiple mathematic responses," "think-alouds," and "using visuals to generate vocabulary." Similar to the top reading strategies, the mathematics strategies received average ratings raging between 4.41 and 4.76, which place them within the "very important" continuum of the importance scale. Descriptive statistics results for both Delphi rounds of ratings of all 12 mathematics strategies are included in Appendix E. Table 3. Top Mathematics
Instructional Strategies
Science Instructional Strategies Table 4 summarizes the descriptive statistics results for the top nine science strategies, which are defined in Appendix C along with other science strategies. These data show that the top science strategies with mode values of 5 were "hands-on participation," "modeling," "labs," "pre-reading strategies," "daily re-looping," "vocabulary development," "graphic organizers," "multiple and varied exposure to the same concept," and "using pictures." Just like the top reading and mathematics strategies, these core science strategies received high importance averages (4.38-4.91), which indicate that educators considered them to be very important. Appendix F provides information from both Delphi rounds on descriptive statistics results for all 14 science strategies. Table 4. Top Science Instructional Strategies
Additional Instructional Strategies At the end of each content area section of the Delphi survey, educators in each state were invited to generate additional instructional strategies that they considered important in delivering content for ELLs with disabilities. At the end of the first round, 10 additional reading strategies, 2 additional mathematics strategies, and 3 additional science strategies were generated by study participants. These strategies are described in Table 5 and defined in Appendix G. As is evident from the table, the number of practitioners who responded to each additional strategy was quite small, too small for analysis. Table 5. Additional Content Strategies
Several research participants also generated two additional reading instructional strategies in the second round of the Delphi survey. These strategies were not rated by educators because they were added at the end of the data collection process. One teacher contributed the following strategy to the list: "homework and practice: assigning different reading material for homework with skill sheets." Another teacher described using this reading instructional strategy:
Results by Educator Demographic Variables One of the data analysis goals was to determine whether participants’ perceptions of strategy importance differed depending on their demographic characteristics discussed in the methodology section of the report. For this purpose, analyses of variance (ANOVAs) were computed for each set of strategies at the p < .05 level. Table 6 presents statistically significant ANOVA results by four demographic variables: ELL population size in the states which served as our study sites, research participants’ job titles, professional experiences, and time spent in their current positions. Table 6. Instructional Strategy Rating Differences as a Function of Demographic Variables
Note: State ELL population size – F(1,37) = 6.31, p = .017; Job title – F(4,36) = 3.48, p = .008; Professional experience – F(2,33) = 4.38, p = .021; Current position – F(3,36) = 2.11, p = .045. The ANOVA analyses resulted in identifying differences across demographic categories for only three reading strategies and one mathematics strategy indicating overall consensus among our research subjects on the importance of most content strategies under consideration. The analysis of variance by the size of ELL populations in the nine states showed some differences in the ratings given to the reading "word dissection" strategy by educators from states with large ELL populations and the ratings of their colleagues from states with small ELL populations. Particularly, teachers from large ELL population states gave this strategy an average importance rating of 4.18, which corresponds to the "very important" continuum on the scale, while teachers from small ELL population states rated this strategy averagely at 3.57, which corresponds to the "important" scale continuum. Data analysis by educators’ job title pointed to statistically significant differences in ratings of the "using picture books" reading instructional strategy. ESL teachers perceived this strategy as very important and assigned it an average importance rating of 4.86. Special education teachers and general education teachers, however, found this strategy to be important rather than very important and rated it at 3.43 and 3.84 respectively. The sizes of the "school administrator" and "other" demographic categories were too low to draw statistical generalizations. Depending on the length of their professional experience educators rated the importance of the "Language 1—Language 2 back-to-back" reading strategy differently. Educators with 6 to 10 years of professional experience rated this strategy at 2.63 within the "neither important nor unimportant" scale range, while educators with more than 10 years of professional experience rated this strategy at 3.52 or as "important." The size of the demographic group of educators with 1 to 5 years of professional experience was too small to produce meaningful statistical results. Only one mathematics strategy produced statistical significance. It was marginal and thus not deemed worthy of discussion. This nationwide study was undertaken to examine the reading, mathematics, and science instructional strategies, which educators perceive to be important in delivering grade-level standards-based instruction for ELLs with disabilities. Findings in this study confirmed the results from our previous statewide and nationwide projects, provided a better understanding of the content instructional strategies, and raised more questions for future research. Review of Instructional Strategies This project focused on content instructional strategies for ELLs with disabilities. This study’s findings further supported the findings in our previous studies (Barrera et al., 2008; Thurlow et al., 2004). Across this study and the Barrera et al. study, 19 states were included, 9 with large ELL populations and 10 with small ELL populations. Educators in both studies agreed on the high importance of the following instructional strategies: Reading:
Mathematics:
Science:
Research participants tended to give high or moderate ratings to all instructional strategies named. The Delphi approach allowed the participants to see their colleagues’ first round results, which may be part of the reason that their second round ratings were generally higher. Low ratings were assigned to some instructional strategies, but in rare instances. Educators were encouraged to comment on the instructional strategies if they had some additional input on strategy utilization. The generated comments were also positive in nature. One teacher had a general comment on the strategies included in the survey:
There is ongoing discussion in the research literature about effectiveness and feasibility of bilingual instruction (cf. Cahnmann & Varghese, 2005; Goldenberg, 2008; Ochoa & Cadiero-Kaplan, 2004). The bilingual strategies included in the Delphi survey ("bilingual vocabulary sheet," "Language 1—Language 2 back-to-back," and "bilingual conversations") also raised some counterarguments from research participants. The ratings and comments for the "bilingual vocabulary sheet" strategy showed support that ranged from high to low. One educator found it to be important because it "gives students ‘anchors.’" Another educator perceived this strategy to be helpful for beginning level students but "too much of a crutch" for advanced students. A third educator believed that this strategy could be used if preferred by students:
The "Language 1—Language 2 back-to-back" and "bilingual conversations" reading strategies received the lowest average ratings from research participants, although educators with over 10 years of professional experience tended to rate the former strategy significantly higher. Several teachers explained their low ratings for these strategies by the ESL program requirements in their schools. Another teacher commented that these two strategies were "not always feasible." This was reflected in a statement from one more teacher, which came in response to the "Language 1—Language 2 back-to-back" strategy:
Yet another comment on the "bilingual conversations" strategy was:
Among other reading strategies, teachers emphasized the importance of the following survey items: Chunking and questioning aloud:
Journal:
Think-aloud reading:
Comments for the mathematics and science strategies were fewer in number but still instructive. One teacher described the "modeling" mathematics strategy as "important for ESL because if they do not understand your language, they will rely on your actions." The "using manipulatives" math strategy was perceived by one teacher as "essential in teaching learning disabled students" and another practitioner suggested that this strategy could be more effective when "combined with verbal explanations." The science "summarizing" strategy was considered to be very important by one educator because it "allows the teacher to correct misconceptions before they affect overall comprehension." Another teacher commented on the effectiveness of the "daily re-looping" science strategy:
Limitations of the Study There are several limitations to this study. Small sizes of the Delphi review groups in each state reduced the possibility of generalizing findings for the additional strategies that were identified. Participants rated instructional strategies both inside and outside of their formal areas of instruction. Nevertheless, it could be argued that many of the instructional strategies generated in the study are applicable across multiple content areas. Finally, the overall high ratings for most strategies may indicate the need to include additional refining measures such as the degree to which a strategy might be considered "feasible" despite being considered "important."
Implications for Future Research This study explored educators’ perceptions of the importance of instructional strategies for delivering grade-level standards-based instruction for ELLs with disabilities. Although our findings provided important nationwide information on reading, mathematics, and science strategies that teachers believe are effective, there is a need for follow-up studies to investigate the effectiveness and feasibility of these strategies. Further, future research in the field might include multistate or national single case design studies that could provide direct observations about the nature, utility, and effectiveness of the instructional strategies described here (cf. Barrera, Liu, Thurlow, & Chamberlain, 2006a; Barrera, et al., 2006b). Instructional perceptions of other educational stakeholders, particularly ELLs with disabilities and their parents, might also serve as the focus for similar Delphi studies or studies employing other research methods (cf. Shyyan, Thurlow, & Liu, 2005; Vang & Barrera, 2004-05). The field would also benefit from studying the role of school administrators in instructional processes nationwide to identify how school administrators can enhance instruction for ELLs with disabilities. Adler, M., & Ziglio, E. (Eds.). (1996). Gazing into the Oracle: the Delphi method and its application to social policy and public health. London: Jessica Kingsley Publishers. Albus, D., Thurlow, M. L., & Clapper, A. (2007). Standards-based instructional strategies for English language learners with disabilities (ELLs with Disabilities Report 18). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Barrera, M. T. (2008). Assessment of culturally and linguistically diverse learners with disabilties, 33(3), 132-134. Barrera, M. T., Liu, K., Thurlow, M., & Chamberlain, S. (2006a). Use of chunking and questioning aloud to improve the reading comprehension of English language learners with disabilites (ELLs with Disabilities Report 17). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Barrera, M. T., Liu, K., Thurlow, M., Shyyan, V., Yan, M., & Chamberlain, S. (2006b). Math strategy instruction for students with disabilities who are learning English (ELLs with Disabilities Report 16). Minneapois, MN: University of Minnesota, National Center on Educational Outcomes. Barrera, M., Shyyan, V., Liu, K., & Thurlow, M. (2008). Reading, mathematics, and science instructional strategies for English language learners with disabilities: Insights from educators nationwide (ELLs with Disabilities Report 19). Minneapois, MN: University of Minnesota, National Center on Educational Outcomes. Cahnmann, M. & Varghese, M. M. (2005). Critical advocacy and bilingual education in the United States. Linguistics and Education, 16 (1), 59-73. Commission on No Child Left Behind (2007). Beyond NCLB: Fulfilling the promise to our nation’s children. Washington, DC: The Aspen Institute. Gersten, R., & Baker, S. (2000). What we know about effective instructional practices for English-language learners. Exceptional Children, 66 (4), 454-470. Gersten, R., Baker, S., & Marks, S. U. (1999). Teaching English-language learners with learning difficulties: Guiding principles and examples from research-based practice. Reston, VA: Council for Exceptional Children. Gersten, R., Baker, S. K., Shanahan, T., Linan-Thompson, S., Collins, P., & Scarcella, R. (2007). Effective literacy and English language instruction for English learners in the elementary grades: A practice guide (NCEE 2007-4011). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies/ed/gov/ncee Goldenberg, C. (2008). Teaching English language learners: What research does—and does not—say. American Educator, Summer 2008. McArdle, P., Mele-McCarthy, J., & Leos, K. (2005). English language learners and learning disabilities: Research agenda and implications for practice. Learning Disabilities Research & Practice, 20 (1), 68-78. Mikow-Porto, V., Humphries, S., Egelson, P., O’Connell, D., & Teague, J. (2004). English language learners in the Southeast: Research, policy & practice. Greensboro, North Carolina: SERVE. Cadiero-Kaplan, K. (2004). Towards promoting biliteracy and academic achievement: Educational programs for high school Latino English language learners. High School Journal, 87 (3). Ortiz, A. A. (1997). Learning disabilities occurring concomitantly with linguistic differences. Journal of Learning Disabilities, 30, 321-332. Shyyan, V., Thurlow, M, & Liu, K. (2005). Student perceptions of instructional strategies: Voices of English language learners with disabilities (ELLs with Disabilities Report 11). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Shyyan, V., Thurlow, M. L., & Liu, K. K. (2008). Instructional strategies for improving achievement in reading, mathematics, and science for English language learners with disabilities. Assessment for Effective Intervention, 333 (3), 145-155. Thurlow, M., Albus, D., Shyyan, V., Liu, K., & Barrera, M. (2004). Educator perceptions of instructional strategies for standards-based education of English language learners with disabilities (ELLs with Disabilities Report 7). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved July, 2008, from http://cehd.umn.edu/nceo/OnlinePubs/ELLsDisRpt7.pdf Vanderwood, M. L., & Erickson, R. (1994). Consensus building. Educational Outcomes for Students with Disabilities (James E. Ysseldyke & Martha Thurlow, Eds.) Binghamton, NY: The Haworth Press, Inc. Vang, H., & Barrera, M. (2004-05). Hmong parents’ perceptions on instructional strategies for educating their children with disabilities. Hmong Studies Journal, 5, 1-20. Zehler, A., Fleischman, H., Hopstock, P., Pendzick, M., & Stephenson, T. (2003). Descriptive study of services to LEP students and LEP students with disabilities: Findings on special education LEP students (Special Topic Report 4). Washington, DC: Office of English Language Acquisition, U.S. Department of Education. Glossary of Reading Strategies
Glossary of Mathematics Strategies
Glossary of Science Strategies
Descriptive Statistics for Reading Instructional Strategies
Descriptive Statistics for Mathematics Instructional Strategies
Descriptive Statistics for Science Instructional Strategies
Glossary of Additional Reading Strategies
Glossary of Additional Mathematics Strategies
Glossary of Additional Science Strategies
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
