CONTACT US

Assessment Insights

New Peer Review Guidance from USDOE: Important Information for State Assessment Teams

[fa icon="calendar"] November, 2015 / by Chad Barrett

Chad Barrett

On September 25, 2015 the United States Department of Education (USDOE) released revised guidance for the peer review of state assessment systems. The guidance will be used during a new round of federal peer review that will start next year. The purpose of the guidance, says USDOE, is to “support States in meeting statutory and regulatory requirements under Title I of the Elementary and Secondary Education Act of 1965 (ESEA)” (USDOE, 2015). The new guidance comes 15 years after the start of the federal peer review process and more than six years after the last revision to the USDOE’s guidance.

To help states understand and prepare for the changes, here’s a summary of the new guidance and suggestions for next steps for state assessment departments.

Summary of the Guidance

The USDOE calls the new guidance a major revision resulting from four areas of improvement (USDOE, 2015).

  1. Improvements in educational assessment: Advancements in the use of technology in assessment, research in accessibility for students with disabilities, changes in test security practices, and increased use of automated scoring have contributed to changes in the USDOE’s guidance.
  2. Revisions to nationally recognized professional and technical standards: In 2014, the American Educational Research Association, American Psychological Association, and the National Council on Measurement in Education published a revision to the Standards for Educational and Psychological Testing. The USDOE’s guidance incorporates changes in the standards that impact federally-mandated testing.
  3. Emergence of multi-state assessment groups: The increase in the number of multi-state groups, both in general education and alternate assessment, led the USDOE to streamline the process for states that are part of multi-state groups. With this revision, multi-state groups will undergo peer review at the same time with some evidence being provided once as part of a multi-state submission and with some evidence being submitted by each state.
  4. Lessons learned from previous assessment peer review processes: The process will continue to rely on an evidence-based review by an external panel of three assessment experts. In this version of the guidance, the USDOE provided more details about the peer review process as well as better examples of the evidence required by USDOE.

“Critical Elements” of the Criteria

The USDOE’s peer review criteria continue to be organized around critical elements, with multiple elements categorized under each critical element. However, the revised guidance contains 6 critical elements compared to 7 in the original guidance. There are also fewer elements, 30 compared to 39 in the original guidance. The table below shows the original and new critical elements (USDOE, 2009, 2015).

Comparison of Original and New Critical Elements

Comparison of Original and New Critical Elements

Key Differences

Language clarification: The revised critical elements show USDOE’s efforts to reorganize, streamline, and update language. For example, USDOE clarified that rigor in content standards means that those standards should prepare students for “success in college and the workforce” (USDOE, 2015).

Science integration: Another change in the elements is the incorporation of science with English Language Arts (ELA) and mathematics. In the revised guidance, whenever ELA and math standards or assessments are mentioned, science is now included in the same element. In the previous version, science was often separated into its own element.

Standards alignment included under Validity: In the original guidance, there was a separate critical element that covered the alignment of an assessment to the state’s content standards. In the revised guidance, these elements have been moved under the critical element “Technical Quality—Validity.” The guidance continues to have elements related to alignment of an assessment to the state’s content standards, to the important processes within the content standards, and to the cognitive complexity of the content standards. States will need to demonstrate alignment of content, process, and cognitive complexity between their standards and assessments.

Next Steps for State Education Staff

Information

State education staff should review the USDOE’s peer review web page. This page contains more information about the process, including the guidance document, overviews from USDOE, and a submission template. The guidance document and other materials will help state education staff to develop a plan for collecting and submitting the evidence required for peer review.

Timeline

State education officials have until November 18, 2015 to contact the state’s USDOE OSS program officer to arrange to participate in the peer review process (Whalen to Chief State School Officers, September 25, 2015). The best time to participate in peer review is between the first and second operational administrations of a new assessment, preferably after completion of all technical reports for the first operational administration. The USDOE has set aside time for peer review sessions in January, March, and May 2016.

Materials Needed

State education officials should begin working with their vendors to identify and arrange for the collection of evidence. The following partial list shows some of the materials that states will need to collect.

  • Test Coordinator Manual
  • Test Administration Manual (Directions for Administration)
  • Accommodations Manual
  • Technical Report
  • Test Blueprints
  • Item Specifications
  • Accessibility Tools and Features of Online Testing Platform
  • Test Administration Training Schedule and Training Materials
  • Test Security Procedures
  • Standard-Setting Report
  • Equating Report (if separate from Technical Report)
  • Guide to Interpreting Score Reports
  • Sample Reports

For a more complete list, state education officials can refer to the USDOE’s sample submission cover sheet and index template or to the peer review guidance document.

As always, your team at Measured Progress is ready to help.

References

U.S. Department of Education, Office of Elementary and Secondary Education. (Rev. ed. 2009). Standards and Assessments Peer Review Guidelines: Information and Examples for Meeting Requirements of the No Child Left Behind Act of 2001

U.S. Department of Education, Office of Elementary and Secondary Education. (2015). U.S. Department of Education Peer Review of State Assessment Systems Non-Regulatory Guidance for States for Meeting Requirements of the Elementary and Secondary Education Act of 1965, as amended.

Whalen, A. (September 25, 2015). [Letter to Chief State School Officers].

Topics: Accountability

Chad Barrett

Written by Chad Barrett

Chad Barrett is a Measurement Services Senior Advisor at Measured Progress, working with content and psychometrics teams to develop test designs and contribute to other large-scale assessment solutions. Chad has more than 15 years of educational assessment experience, particularly in content development and program design.