Enhancing Communication: Desirable Characteristics for State and School District Educational Accountability Reports
NCEO Synthesis Report 30
by James E. Ysseldyke and J. Ruth Nelson
Published by the National Center on Educational Outcomes
January 1998
This document has been archived by NCEO because some of the information it contains is out of date.
Any or all portions of this document may be reproduced and distributed without prior permission, provided the source is cited as:
Ysseldyke, J.E., Nelson, R.J. (1998). Enhancing communication: Desirable characteristics for state and district educational accountability reports (Synthesis Report No. 30). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved [today's date], from the World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/Synthesis30.html
Executive Summary
State accountability reports are designed to inform the public about the status of schools and their students. Across the United States, there is tremendous variability in what these reports look like, in the information they include, and in the extent to which they include students with disabilities. Because of this variability and the desire to produce quality reports that would include data on students with disabilities, a work group was formed to develop a list of necessary, desirable, and succinct characteristics of good state and district educational accountability reports.
Desirable characteristics were identified for both content and format. They included:
• Be clear about who the report is directed to, the intended purposes of the report, and the state’s conceptual model for its accountability system.
• Be comprehensive yet concise in the reporting of inputs, processes, and results for students, especially students with disabilities.
• Provide comparative information with changes over time between schools, districts, states, regions, or standards.
• It is strongly recommended that reports be concise or contain carefully chosen indicators so that no more information is given than is necessary. A multi-layered approach may be appropriate.
• Include cautions against misinterpretations of the data or against any unintended consequences.
• All efforts should be taken to maintain confidentiality and avoid the possibility of identifying individual students.
• Use good formats so that reports are well-organized, readable, and interesting. Use catchy titles, pictures, or other visual aids to capture and hold the audience’s attention.
A checklist of desirable characteristics for state and school district accountability reports is included for those who develop reports
.
Overview
Increasingly, state education departments are recognizing that inclusive accountability systems and the public reporting of educational results for students are important tools to help students attain higher educational standards. Many states have already designed accountability systems to ensure those who are inside and outside the educational system that students are moving toward desired goals (Brauen, O’Reilly, & Moore, 1992). With recent federal mandates, such as the passage of P.L. 105-17, the reauthorization of the Individuals with Disabilities Education Act (IDEA), state education agency personnel are required to report annually on the performance and progress of students with disabilities.
States, however, vary greatly in their reporting practices (Elliott, Thurlow, & Ysseldyke, 1996; Thurlow, Langenfeld, Nelson, Shin, & Coleman, 1997). Some states exclude students with disabilities from their accountability reports, others exclude students who take tests with accommodations, while others exclude those who have used specific types of accommodations.
Similarly, state accountability reports vary. Some states produce five or six 500-page volumes annually, while others produce a two- to three- page report. Consumers indicate that there is often too much information, or too little. A few states give state level data; others give school, district, and state level data annually. Accountability documents also vary in their focus. Almost all states report on performance indicators for students in regular education, but very few include performance data for students with disabilities. Many states use these reports for accreditation purposes while others use them for technical assistance, diplomas, compliance with state requirements, or to generate local, district, and national comparisons. The indicators used in accountability reporting also cover a wide spectrum, from detailed financial information to student mobility rates, and from staffing information to minutes spent in math and reading. Some states report numbers of students who met state standards or goals; others do not. Many states use tables or spreadsheets, and some even use the Internet to communicate their educational results. As one can see, the variability in reporting is considerable; it would be ideal if there was consistency in reporting practices.
Method
With this variability in reporting in mind, members of a study group within the State Collaborative on Assessment and Student Standards, Assessing Special Education Students, which is made up of assessment and special education representatives, convened to develop a list of necessary, desirable, and succinct characteristics of good state and district educational accountability reports. Those in attendance included representatives from six states (Alaska, Arkansas, California, Colorado, Missouri, Wyoming), one Regional Resource Center (Mid-South), University personnel (Minnesota, Oregon), assessment personnel from the Council of Chief State School Officers (CCSSO), and personnel from the National Center on Educational Outcomes (NCEO).
This meeting was devoted primarily to describing current accountability reports, discussing issues encountered in reporting, and generating characteristics of good reports. Before the meeting, NCEO personnel had conducted an extensive analysis of the characteristics of 113 reports received from 32 state education agencies (SEAs). Among highlights of that analysis were the findings on length and variability described in the introduction to this paper, and many others, including those listed in Table 1.
The study group first generated a list of characteristics of good reports, then carefully described the way characteristics could apply to reports issued to differing audiences.
Table 1. Highlights of Finding in Analysis of State and District Educational Accountability Reports
Reports vary in length, from two pages to approximately 600 pages |
| Reports vary in format (spreadsheets, data tables, bar graphs, narratives, Internet). |
| It is rare to find information on the intended purpose of a report or its intended audience. |
| The most common target audiences of reports, when identified, are the general public and government agencies. |
| The most common purpose of reports, when identified, is "to provide information." |
| The majority of states provide multiple accountability documents, but 16 states compile their data into a single document. |
| Most states provide data at both the state and district levels (N=41). |
| All states report outcome data, and most report input and process data. |
| Few states report outcome data for students with disabilities. |
| Every state has at least one report in which there are data reported for a sample that is not described clearly. |
| Eight states have at least one report in which the report specifically excluded the outcome data of students with disabilities. |
Results
The SCASS Study Group generated various characteristics of good accountability reports. These characteristics were grouped, formatted, and developed into an easy-to-use checklist for state education department personnel. The generated characteristics were placed into three broad categories: content, format, and general. Below are the characteristics the study group found to be most desirable in creating effective accountability reports.
Be clear. Participants indicated that it should be clear who the report is directed to, the intended purpose of the report, and the state’s conceptual model for its accountability system. Other desired aspects of clarity were a clear statement of mission or goals, assumptions, and standards. Finally, it was thought that the nature of the population of students being reported on should be clearly described.
Be comprehensive. State accountability reports should be comprehensive, yet concise in their reporting of inputs, processes, and results. "Inputs" refers to accountability indicators that describe the student’s learning environment as well as demographic characteristics of the particular district’s population (e.g., student-teacher ratio, cost). "Process" indicators describe student participation and school district evaluation (e.g., enrollment, attendance, accreditation status). "Results" refer to nontest and test data indicators that focus on the end result of a student’s learning process. Data should include all three types of indicators, and should be reported for all students, especially students with disabilities. It would be most informative if the results for special populations were disaggregated. With the reauthorization of IDEA, states are required to disaggregate the performance data of students with disabilities.
Provide comparative information. Comparative information should be given, but with cautions. For instance, if data were not collected on similar populations, this should be made known. It should be possible to make fair comparisons from the data reported. Not only should comparisons be fair, but it is important to make regional and national comparisons to check student progress. If comparisons should not be drawn, then it is the responsibility of the state to make this clear to its audience.
Be concise. Group participants strongly indicated that reports should be concise. Some states have been known to produce three or four volumes with over 500 pages of data in each volume. Indicators need to be chosen carefully so that no more information is given than is necessary to convey a message to an intended audience. It was suggested that a multi-layered approach may be appropriate for those states that have a large amount of information to report. For a consumer who might not need all the available details, an abbreviated report would be sufficient.
Include cautions. SEA or local education agency (LEA) personnel should always include cautions against misinterpretation of data or against any unintended consequences of the data. Reports should be written in a way that minimizes scapegoating and negatives even when presenting bad news.
Maintain confidentiality. Participants indicated concern about maintaining confidentiality for low-incidence student populations. All efforts should be taken to avoid the possibility of identifying individual students. This may involve reporting data of certain special populations only at the state or district levels.
Use good formats. Participants generated four kinds of desirable format characteristics:
• Accountability reports should be readable by the intended audience, and should be responsive to the needs of the intended audience.
• The layout should be eye-catching or interesting, organized and contain one concept per page. Organizational aids should be used, and might include a reader’s guide, table of contents, and/or index or glossary.
• A bulletted summary of the report for a quick read is helpful to consumers.
• The report should make it clear how additional copies may be obtained or how a person could obtain more detailed information if desired.
Summary
Now that accountability systems and reporting of educational results have moved to the front burner for state education departments, it is important to examine how accountability reports are presented. Neither 500-page reports nor 2-page summaries properly meet all education and public needs. The above recommendations can help develop templates for reports and enhance communication to stakeholders about the educational results for students, with and without disabilities.
Appendix
Checklist of Guidelines for State and District Educational Accountability Reports
CONTENT
Clear...
o Clear statement of intended audience
o Clear statement of intended purpose
o Clear statement of states’ conceptual model for its accountability system (including inputs, processes, and results)
o Clear statement of state standards (or goals) or mission/vision
o Clear statement of assumptions
o Clear statement of who was included in the population of students being reported on
Comprehensive...
o Comprehensive, yet concise set of inputs, processes, and results
o Data on all students, including students with disabilities and limited English proficient students (students with disabilities and limited English proficient students’ results are disaggregated)
Comparative...
Includes enough information to enable people to make fair comparisons among:
o Schools
o Districts
o States
o Regions
o Standards
Includes enough information to enable people to make fair judgments about changes over time for:
o Schools
o Districts
o States
Concise...
o Includes no more information than is necessary to convey a message to an intended audience ... brief
Cautions...
Provides cautions against:
o Scapegoating
o Unintended consequences
o Negatives
Confidentiality...
o Maintains confidentiality of low frequency student populations
FORMAT
Readable...
o Appropriate for the intended audience
Responsive to the needs of intended audiences...
o Answers audience questions and provides accurate profile
Layout...
o Not cluttered or complex
o Organized and easy to find information (e.g., reader’s guide, table of contents, index, glossary)
o Interesting (e.g., includes catchy titles, pictures, or other devices to get and hold audience interest)
Links...
o Statement of how and where to get additional copies
o Statement of how to get more detailed information
Executive Summary...
o Bulletted summary of report for a "quick read"
IMPORTANT OVERALL QUESTIONS
Is the report readable? o YES o NO
Is the report fair? o YES o NO
Is the report concise? o YES o NO
Is the report visually attractive? o YES o NO
Is the report accurate? o YES o NO
References
Brauen, J. L., O’Reilly, F., & Moore, J. (1994). Issues and options in outcomes-based accountability for students with disabilities. Rockford, MD: Westat.
Elliott, J. L., Thurlow, M. L., & Ysseldyke, J. E. (1996). Assessment guidelines that maximize the participation of students with disabilities in large-scale assessments: Characteristics and considerations. Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
Thurlow, J. L., Langenfeld, K., Nelson, J. R., Shin, H., & Coleman, J. (1997). State accountability reports: What do they say about students with disabilities? Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.