“Now What Do We Do?” Developing More Actionable Information for New York City’s Schools

By Chelsea Farley | Ms. Farley is the Communications Director for the Research Alliance for New York City Schools.

In recent years, states and districts have made academic outcomes, such as standardized test scores and graduation rates, the primary measure of a school’s performance. This focus on outcomes has been valuable for tracking student achievement and identifying both schools and students in need of support. One of the weaknesses of the approach, however, has been its failure to provide information about how schools should improve. Faced with lackluster performance data, struggling schools are often left wondering, “Now what do we do?”

The good news is that there is a growing body of evidence about specific areas of school practice and capacity that are essential for improving student achievement. The University of Chicago Consortium on Chicago School Research used data from hundreds of schools in that city to identify five “essential supports for school improvement,” including elements like effective leadership and strong family-community ties.[i] The researchers found that schools doing well in most of the five areas were at least 10 times more likely than schools with just one or two strengths to produce substantial gains in reading and math. Prolonged weakness in even one area “undermined virtually all attempts at improving student learning.”[ii]

This study, and others that have reinforced its findings, help us understand much more concretely the skills and capacities that enable schools to improve. The next question is how to provide schools with information about these key areas and help them use that information to bolster their work.

An initiative taking place in New York City may begin to shed light on this question. In the fall of the 2014-15 school year, the New York City Department of Education introduced a “Framework for Great Schools” based largely on the Chicago research. A critical first step toward using the “Framework” has been to figure out how to measure its elements in each of the City’s schools. With help from the Research Alliance for New York City Schools, the NYC DOE overhauled the 2015 Parent, Student, and Teacher School Surveys, to collect better information about where each school stands in relation to the necessary capacities identified by research. The DOE will use the survey results, along with other information, to assess schools’ current strengths and weaknesses, and to target areas in need of improvement. Below, we describe changes made to the Survey and outline some crucial issues we hope this work will address as it develops.

Retooling the Survey

The NYC School Survey is administered each spring to students in grades 6 through 12 and to all district parents and teachers, making it one of the largest efforts of its kind in the nation. In 2014, nearly a million parents, students, and teachers completed the Survey, yielding information about some 1,800 schools.[iii]

Since 2010, the Research Alliance and the DOE have collaborated to strengthen the survey. Our 2012 report Strengthening Assessments of School Climate, provides an interim view of this process; we had found that while the Survey had robust response rates and effectively tapped into parents’, students’, and teachers’ unique perspectives, the information gathered was of limited value for assessing key aspects of school practice and capacity. Thus, the report recommended incorporating new measures that would better distinguish between schools and that offered clear links to other performance indicators, such as grades, attendance, and test scores.

The introduction of the “Framework for Great Schools” provided an opportunity to do just that. Throughout the summer and fall of 2014, teams from the Research Alliance and the NYC DOE developed strategies for measuring the elements in the “Framework”. In some cases, we were able to use measures that research had already validated. When these were not available, we created new measures. As a result, the 2015 Survey looks quite different than past iterations. For example:

  • To assess Effective Leadership, the Survey will ask teachers if their principal “communicates a clear vision for this school,” “sets high standards for student learning” and “knows what’s going on in my classroom.”
  • To assess Family-Community Ties, the Survey will ask if “teachers really try to understand parents’ problems and concerns,” if “parents are greeted warmly when they call or visit the school,” and if “school staff regularly communicates with parents about how they can help their children learn.”
  • To assess Rigorous Instruction, the Survey will ask students if they “build on each other’s ideas during class discussions,” “use data and text references to support their ideas” and “provide constructive feedback to their peers/teachers.”
  • To assess Supportive Environment, the Survey will ask students how safe they feel in and around school and will ask parents if students at the school “feel it is important to come to school every day,” “think doing homework is important,” and “try hard to get good grades.”
  • To assess Collaborative Teachers, the Survey will ask teachers about professional development opportunities and whether they “look forward to each working day at this school” and “feel loyal to this school community.”
  • To assess Trust, the Survey will ask students if teachers “always keep their promises” and “treat me with respect.”

Over the next several years, the Research Alliance will continue to analyze the Survey’s results and provide feedback about the quality of the measures being used.

What We Hope to Learn

New York City’s use of research to develop the “Framework,” and to recalibrate the Survey accordingly, is a strong beginning to an ongoing process. In the coming years, the Research Alliance anticipates advancing our understanding of school capacities and outcomes in at least two areas:

  • Building a better survey. By analyzing survey results, alongside information collected through other instruments (like the Quality Review or the district’s instructional audits of struggling schools), we hope to help NYC build a Survey that effectively gauges schools’ capacity—in areas that matter most for enhancing students’ achievement and development. We will examine, for instance, whether the organizational capacities identified as essential in Chicago and other cities also predict improved student outcomes here in NYC. And we will make suggestions for improving specific capacity measures.
  • Developing a meaningful cycle of improvement. We also hope to work with a small group of schools, as part of an intensive pilot project that will explore strategies for sharing and making use of capacity data. Among the questions we will examine are: How is information about school capacity best communicated to reflect the unique assets and challenges of individual schools? What kind of support do schools need to interpret and make use of that information? And what actions do schools take in response to the data they receive?

Ultimately, we hope that when schools have a clearer picture of their strengths and weaknesses, and receive support in making important changes, they will be able to serve students and communities more effectively. As New York City’s initiative unfolds, it is sure to produce valuable lessons that inform school improvement efforts, not only in New York City but across all of our country’s large urban school districts.

[i] http://ccsr.uchicago.edu/publications/organizing-schools-improvement-lessons-chicago

[ii] http://ccsr.uchicago.edu/sites/default/files/publications/OrganizingSchoolsPressRelease.pdf

[iii] http://schools.nyc.gov/Accountability/tools/survey/default.htm

Comments on this Post

  1. Defining accountability and identifying actionable policies is a challenge – this moment the governor has defined “accountability” as a test score and actionable policies as extending probation, renewable tenure and easing teacher dismissal rules.

    Surveys, of course, have internal flaws, the bias of those being surveyed, principals, teachers, parents and students may have axes to grind, everyone wants to “look good,” and in an era of school closings manipulating data may be more important than actual changes.(Principal: “If you say ‘bad things’ they may close our school”)

    If we look at SURR (School Under Registration Review) reports from the early 90s and reports today little has changed, we were alway able to identify the reason for struggling schools and recommend what had to be done, the insurmountable has been the inability to change, or the erosion of the changes. Part of the problem are the limited metrics: a school may increase attendence, reduce suspensions, decrease teacher attrition and not see increases in test scores. Your last bullet, “meaningful cycles of improvement,” is both the essence of the effort and enormously complex. External supports usually result in improving “data management” not changes in teaching and learning and are usually not sustainable. Staffs resent and mistrust the “outsiders,” cultural clashes, generational clashes, racial undertones, all tend to erode the impact of external “supports.”

    In my view the most important element, no surprise, is the school leader. The “fast-track” leadership academy school leader commonly lacks the skills and the support of the school community, the “old-timer” is too tied to policies that are comfortable, and have not worked for years. The easy to see and difficult to define “school climate” may be the best “definer” of effective leadership.

    The current plans to partner high achieving and low achieving schools is illusory.

    We shy away from issues of race, principals and teachers of color can have a more signifcant impact on students and communities. The research is slim.

    Can we identify successful “turnaround” schools?

    In my experience the “success” frequently resulted from better data management and not teaching and learning.

    If we can identify school progress rather than tunaround, improvements in attendence, lower suspensions, and. most that diffcult to define school climate, and ask, what were the qualities of the school leader who achieved these changes … are, are they “teachable”?

    just some thoughts.

Leave a reply

Recent Posts

IdeaLab

Moynihan and the Modern American Family at 50

By Gerard Robinson | Senior Fellow, CUNY Institute for Education Policy “The gap between the Negro and most other groups in American society is widening. The fundamental problem, in which this is clearly the case, is that of family structure.” — Daniel Patrick Moynihan (1964) “If the misery of the poor be caused not by… Read More

Director's Perspective

An Open Letter On School Accountability

Below you will find the outline of some thoughts on educational accountability. The topic has become one of the central issues in contemporary reform efforts in American education, with particularly heated debate over efforts to measure the effectiveness of teachers through test scores as well as classroom observations. But the issue is wider than this,… Read More

Reports from the Field

Institutional Commitment to Civic Education: Public Montessori Secondary Schools

By Katie Dulaney | Lakewood Montessori School Even as we hold schools increasingly accountable for student achievement, we rarely seem to judge schools for their performance in citizenship preparation, an inauspicious “accountability gap” for a democracy. -Michael Johanek and John Puckett, “The State of Civic Education” (2005) Education leaders across the political spectrum – from… Read More