Dear JCPS,
I have done a brief review of the Bellwether report commissioned by the SCALA education subgroup and I wanted to share some information that should call into question the validity of conclusions drawn by or from the report. Specifically, the Bellwether report cites a number of other reports that have been discredited when subjected to academic peer review.
For example, the Bellwether report includes multiple references to “A 21st Century School System in the Mile-High City.” A review (attached) of this report by (White – University of Colorado Boulder) found:
A report published by the Progressive Policy Institute calls for aggressively closing more public schools and expanding charter schools and charter networks. It highlights reforms adopted by Denver Public Schools, notably a “portfolio model” of school governance, and argues that these reforms positively impacted student test scores. However, causality cannot be determined, and the report did not attempt to isolate the effect of a multitude of reforms— including charters, performance pay, and a new performance framework—from larger complex forces shaping student demographics in the city. Written in a reportorial voice, the only data presented are in the form of simple charts. The lack of conventional statistical analyses thwarts the reader’s understanding. The report also characterizes the reform’s adoption as a “political success” born of a healthily contentious electoral process. In doing so, it down- plays the role of outside forces and moneyed groups that influenced the form of reforms, and it disregards missed opportunities for meaningful engagement with community stakeholders. Finally, while the report acknowledges the district’s failure to close achievement gaps and admits limitations with the evaluation system, it never explains how a successful reform could generate a widening gap in performance between student groups by race and class.
The Bellwether report is also based on “Measures of Last Resort: Assessing Strategies for State-Initiated Turnarounds.” A review (attached) of this report (Malen and Rice – University of Maryland) found:
The stated goal of this report is to strengthen the evidence base on state-initiated turn-arounds and to provide guidance to help states use turnaround strategies more effectively. The report draws on multiple sources of information to develop a conceptual framework and profile of state-initiated turnaround strategies, to array the evidence on the effectiveness of turnaround initiatives, and to identify key elements of a successful turnaround strategy. However, given multiple methodological limitations, the report fails to elevate either the research base or the policy discourse. Specifically, the methods used to carry out the original research (e.g., analysis of state policies, interviews with stakeholders, and illustrative cases) are neither explained nor justified. Likewise, the methods employed in the eight evaluations selected to assess the effectiveness of turnaround approaches are not described, and the evidence base produced by these evaluations is not sufficient to support the sweeping claims made in the report. Equally important, the report neglects to consider relevant research on the specific mechanisms (e.g., school reconstitution, intensive professional development, private management systems) that states use when they employ the broad turnaround strategies discussed in the report. As a result of these problems, the report does not enhance the evidence base or provide the substantive guidance state policymakers require to make informed decisions about the use of various school turnaround strategies.
These examples illustrate a broader problem with the Bellwether report – it is based on reports that would more accurately be described as policy advocacy documents than objective research. As such, making consequential decisions that would impact our community based on the Bellwether report would not be advisable because the report’s validity is highly questionable at best. Basing high-stakes decisions on such untenable evidence could lead to significant negative consequences and unintended outcomes.
I hope this information can be shared broadly, including with the full SCALA committee and that all involved will review the attached peer analyses carefully. I have also attached an additional analysis of portfolio districts, by William Mathis at the University of Colorado Boulder, which individuals may wish to review.
Brent McKim
JCTA President
Attachments:
Portfolio Districts Analysis – Mathis
1 thought on “SCALA Bellwether Report Validity Should Be Questioned”