A Justice Index: The Quest for the Holy Grail of Court Performance Measurement
My colleagues and I have long sought what is for us the Holy Grail of performance measurement -- a simple, easy to grasp index of the performance of courts and the justice system that could be used by both insiders and outsiders. Such an index recently was brought back into sight by two prominent proponents, legal scholar and law professor Laurence H. Tribe, and journalist and author Amy Bach.
Tribe is a scholar of constitutional law and former Harvard Law School professor whose students include Barack Obama, John Roberts and Elena Kagan. Tribe took the position of Senior Counselor for Access to Justice, U.S. Department of Justice (DOJ) earlier this year. In that position, he will lead an initiative aimed at improving access to civil and criminal legal services and will work with federal, state, and tribal judiciaries in strengthening fair, impartial, and independent adjudication. He will also exchange information with foreign ministries of justice and judicial systems regarding efforts to provide access to justice, as part of the DOJ’s international efforts to promote fair and impartial law enforcement and adjudication.
Amy Bach is the author of Ordinary Injustice: How America Holds Court (Metropolitan Books/ Holt, 2009), a book hailed by renowned historian Doris Kearns Goodwin as a crusading call for reform of the justice system in the tradition of Rachel Carson’s Silent Spring. Bach will be at the National Center for State Courts in Williamsburg October 7 to discuss her ideas for a justice index.
A Rallying Cry
In a speech to the annual Conference of Chief Justices in Vail, Colorado in July of this year, Professor Tribe called for a comprehensive performance index of sorts, a way of “grading the state’s legal system in terms of how well or poorly it is delivering justice to the state’s people.”
And then, in a New York Times op-ed on August 11, shortly after Tribe’s remarks, Amy Bach called for a “justice index,” a term she did not specifically use in her 2009 book, though she described it clearly.
The Justice Index, as Bach described it in her op-ed, “would function roughly like college rankings, evaluating county courts on factors like cost, recidivism, crime reduction and collateral consequences, including whether people lose their jobs or homes after contact with the criminal justice system.” She contends that the lack of information of a justice index has “a corrosive effect: without public awareness of a court system’s strengths and weaknesses, inefficiencies and civil liberties violations are never remedied.” Both Bach and Tribe suggest that a comprehensive justice index would benefit those inside the justice system – Bach contends that justice systems insiders’ “blindness” to errors is a signature feature of ordinary injustice -- and those outside the system, including the public.
While Bach is correct in contending that there is currently no “mechanism in place to keep track of the extent of ordinary injustice” in the courts, my colleagues and I have planted plenty of seeds for such a mechanism over the last ten years that I hope will take root and bear fruit as a result of Bach’s and Tribe’s rallying cry. Two of our efforts that are most relevant to a justice index are the Court Performance Index (CPI) and the Caseflow Timeliness and Efficiency (CTE) Index.
(Many other issues related to the development of what Bach refers to generally as a “court monitoring system” are explored in the 120 + postings in this Blog including the imperative of performance measurement and management and the fact that it can be done, specific measures like the length of time defendants spend in jail before trial, transparency and accountability, and citizen responsibility for courts.)
The Court Performance Index (CPI)
Few of us – including court managers and judges -- can keep track of more than handful of measures. On our car dashboard, for example, we have only a few gauges -- like the speedometer -- that we look at all the time, a few that need to be monitored regularly but not all the time (the gas gauge), several that need to be looked at less frequently (the oil meter), and some that need to be paid attention to only when we are given a warning signal.
One way to achieve simplicity and yet monitor more than a few measures is to combine multiple measures into a single, simple index. Multiple measures in a "family" of metrics can be assigned weights according to their importance and combined in an index that is an aggregate statistic.
We first introduced the idea of a Court Performance Index (CPI) at the 2003 Court Technology Conference in Kansas City. Though we viewed the CPI as beneficial for both outsiders and insiders, for the conference audience of 2,000 mostly court managers and judges, we emphasized the CPI’s use as a management tool.
Sometime in the not too distance future, we envisioned then, court executives and other stakeholders, will get all the critical information about their court's performance on their computers instantly. For court managers, the homepage of their court's website would display a window highlighting performance information summarized by a single number, the CPI, accompanied by a green or red triangle with another number indicating whether the CPI is up or down from the previous day and by how many points. (Think of the way the Dow Industrial Average is shown on CNBC and CNN.)
If a quick morning check of the court's CPI by the court manager or chief judge, for instance, reveals a green, upward pointing triangle and an increase since the previous day, all would be well. The court manager or chief judge could relax and get a cup of coffee. On the other hand, the cup of coffee may need to wait if the morning check discloses a red downward-pointing triangle and a significant downturn in the CPI. By drilling down through several screens of progressively more detailed and less aggregated performance data, beginning with a display of four to ten core court performance measures that constitute the components of the CPI (think, again, of the Dow and its component company stock values), court managers and judges would be able to pinpoint the court, division, case type or resources needing attention.
The court manager may click the CPI icon to show a more detailed screen displaying a "balanced scorecard" of the four to ten performance measures that contribute to the CPI (e.g., adaptations of the National Center for State Courts’ CourTools or Appellate CourTools). Each of the performance measures on this screen are displayed in the same simple way as the CPI, i.e., a single score for the measure, a smaller number indicating a change in the measure, if any, and a triangle indicating a downturn or upturn in the measure.
The manager might discover, for example, that the early results of one measure - for example, an online survey of court employee engagement posted the morning of the previous day - is largely responsible for dragging down the court's CPI. From the "real time" data displayed on the screen the court manager learns that 37 percent of the court employees already had responded to the survey within 24 hours of their posting on the website.
A few more clicks by the court manager produces screens revealing important additional information about this measure including trends over time; the alignment of the measures with the court's management processes such as strategic planning, budgeting, quality improvement, and employee evaluation; related measures; and best practices in the performance area gauged by the measures. Further, a note on the screen showing trends of the measures over time may indicate that the response rate to previous administrations of the surveys was higher than 90 percent of all the court's employees and that, in the past, the early returns tended to be the most negative. The court manager decides not to jump to conclusions but to mobilize the management team in the event that the survey results do not improve as more of the court employees complete the online questionnaires.
In a few minutes of the morning, the court manager will have viewed the performance "dashboard" and determined the days "score" and what needs to be done. In this example, the court manager decides simply to alert and to mobilize the court's management team in case the measure that caused the CPI downturn does not show improvement in the days ahead.
Caseflow Timeliness and Efficiency Index (CTE Index)
A more modest index, first introduced in 2004, the Caseflow Timeliness and Efficiency Index (CTE Index), joins four measures associated with Standard 2.1, Case Processing, of the Trial Court Performance Standards and detailed in Trial Court Performance Standards and Measurement System, the first three of which are also part of the CourTools: time to disposition (on-time case processing), case clearance rate, age of pending caseload, and trial date certainty. These four measures of case processing, expressed as proportions, are reduced to one in the index. The CTE Index assigns different weights (i.e., relative importance expressed as number from 1 to 100) to these measures and requires the calculations as prescribed in the Standards with some conversions to accommodate the aggregation of the measures into an index. (See “Technical Note: The Caseflow Timeliness and Efficiency (CTE) Index.” Court Manager, Volume 18, Number 4, 2004, 8-9; reprinted in Steelman, David C. et al. Caseflow Management: The Heart of Court Management in the New Millennium. Williamsburg, VA: National Center for State Courts (2004)).
Ratings and Rankings of Courts
A few weeks ago, Newsweek published its first “best countries” ratings and rankings in which it set out to answer the simple question of which country would provide you the very best opportunities to live a healthy, safe, prosperous and upwardly mobile life. Newsweek rated and ranked 100 countries in five categories of well being – education, health, quality of life, economic competitiveness, and political environment (the U.S came out eleventh in the overall ranking). Americans love rankings, the top-ten of whatever including cities, high schools and universities, hospitals and toasters.
Amy Bach believes, as I do, that such ratings and rankings get people’s attention and create incentives and healthy competition. She thinks, as I do, that they will do the same for courts. (See “Toward the 50 ‘Best in Class’ Courts,” Made2Measure, April 23, 2008). “In communities across the country, people use statistics on hospitals, schools and other public services to decide where to live or how to vote,” writes Bach. “But while millions of Americans deal with their local criminal courts as defendants and victims each year, there is no comparable way to assess a judicial system and determine how well it provides basic legal services.”
Ratings and rankings of the world’s best places to live and America’s best public high schools, colleges, hospitals and parks hold two important lessons for judges and court managers who might benefit from the use of the CPI, the CTE Index, or a justice index envisioned by Bach and Tribe, especially those who might bristle at the idea of comparative performance measurement. (See “Ten Reasons Not to Measure Court Performance,” Made2Measure, November 19, 2008; and “How Do we Stack Up Against Other Courts? The Challenges of Comparative Performance Measurement,” Court Manager, Vol. 19, No.4 (Winter 2004-2005).
As Ms. Bach points out in her op-ed, the first lesson is that performance matters to citizens. The U.S. News & World Report annual rankings of America’s best public high schools, for example, are based on the key principle that a great high school must be able to produce measurable academic outcomes to show that it successfully educates all of its students across a range – a balanced scorecard – of performance indicators. Little else matters to parents as much.
The hallmarks of a high-performing court are the political will and the capacity to produce measureable outcomes that matter to the public and other stakeholders. The best courts ask themselves “How are we doing and how could we improve?” They measure their performance on a regular and continuous basis to gain knowledge and insight into new realities, opportunities and necessities and to convert that knowledge and insight into effective strategies for improvement.
The second lesson to be learned from such ratings and rankings is that similar comparisons and rankings of courts are likely to occur with increasing frequency, as Professor Tribe’s remarks to the state chief justices in July suggest. Court leaders are well advised to be prepared with their own court performance data.
The fact is that the best places to live, to send our kids to school, to get heart-bypass surgery and, yes, to get access to justice are of great interest to citizens. Courts not producing performance data to satisfy this interest are going to be asked why they’re not. What gets measured gets attention. What gets measured gets done!
It’s Not About Numbers
The measurement process uses numbers to provide an understandable and comparative result, but it is ultimately not about the numbers. It is about perception, understanding and insight. Ultimately, it is not the measure itself that is important but rather the questions that it compels us to confront.
Amy Bach concludes her groundbreaking book with these encouraging thoughts about a justice index and performance measurement in general: “Metrics would offer a mirror for people who work in the system, allowing them to see how their roles might have eroded at the expense of rights and public safety. Metrics alone are not an answer but they could be the beginning one. They are the tools we need to ask for the courts we deserve.”
© Copyright CourtMetrics and the National Center for State Courts 2010. All rights reserved.
Tribe is a scholar of constitutional law and former Harvard Law School professor whose students include Barack Obama, John Roberts and Elena Kagan. Tribe took the position of Senior Counselor for Access to Justice, U.S. Department of Justice (DOJ) earlier this year. In that position, he will lead an initiative aimed at improving access to civil and criminal legal services and will work with federal, state, and tribal judiciaries in strengthening fair, impartial, and independent adjudication. He will also exchange information with foreign ministries of justice and judicial systems regarding efforts to provide access to justice, as part of the DOJ’s international efforts to promote fair and impartial law enforcement and adjudication.
Amy Bach is the author of Ordinary Injustice: How America Holds Court (Metropolitan Books/ Holt, 2009), a book hailed by renowned historian Doris Kearns Goodwin as a crusading call for reform of the justice system in the tradition of Rachel Carson’s Silent Spring. Bach will be at the National Center for State Courts in Williamsburg October 7 to discuss her ideas for a justice index.
A Rallying Cry
In a speech to the annual Conference of Chief Justices in Vail, Colorado in July of this year, Professor Tribe called for a comprehensive performance index of sorts, a way of “grading the state’s legal system in terms of how well or poorly it is delivering justice to the state’s people.”
And then, in a New York Times op-ed on August 11, shortly after Tribe’s remarks, Amy Bach called for a “justice index,” a term she did not specifically use in her 2009 book, though she described it clearly.
The Justice Index, as Bach described it in her op-ed, “would function roughly like college rankings, evaluating county courts on factors like cost, recidivism, crime reduction and collateral consequences, including whether people lose their jobs or homes after contact with the criminal justice system.” She contends that the lack of information of a justice index has “a corrosive effect: without public awareness of a court system’s strengths and weaknesses, inefficiencies and civil liberties violations are never remedied.” Both Bach and Tribe suggest that a comprehensive justice index would benefit those inside the justice system – Bach contends that justice systems insiders’ “blindness” to errors is a signature feature of ordinary injustice -- and those outside the system, including the public.
While Bach is correct in contending that there is currently no “mechanism in place to keep track of the extent of ordinary injustice” in the courts, my colleagues and I have planted plenty of seeds for such a mechanism over the last ten years that I hope will take root and bear fruit as a result of Bach’s and Tribe’s rallying cry. Two of our efforts that are most relevant to a justice index are the Court Performance Index (CPI) and the Caseflow Timeliness and Efficiency (CTE) Index.
(Many other issues related to the development of what Bach refers to generally as a “court monitoring system” are explored in the 120 + postings in this Blog including the imperative of performance measurement and management and the fact that it can be done, specific measures like the length of time defendants spend in jail before trial, transparency and accountability, and citizen responsibility for courts.)
The Court Performance Index (CPI)
Few of us – including court managers and judges -- can keep track of more than handful of measures. On our car dashboard, for example, we have only a few gauges -- like the speedometer -- that we look at all the time, a few that need to be monitored regularly but not all the time (the gas gauge), several that need to be looked at less frequently (the oil meter), and some that need to be paid attention to only when we are given a warning signal.
One way to achieve simplicity and yet monitor more than a few measures is to combine multiple measures into a single, simple index. Multiple measures in a "family" of metrics can be assigned weights according to their importance and combined in an index that is an aggregate statistic.
We first introduced the idea of a Court Performance Index (CPI) at the 2003 Court Technology Conference in Kansas City. Though we viewed the CPI as beneficial for both outsiders and insiders, for the conference audience of 2,000 mostly court managers and judges, we emphasized the CPI’s use as a management tool.
Sometime in the not too distance future, we envisioned then, court executives and other stakeholders, will get all the critical information about their court's performance on their computers instantly. For court managers, the homepage of their court's website would display a window highlighting performance information summarized by a single number, the CPI, accompanied by a green or red triangle with another number indicating whether the CPI is up or down from the previous day and by how many points. (Think of the way the Dow Industrial Average is shown on CNBC and CNN.)
If a quick morning check of the court's CPI by the court manager or chief judge, for instance, reveals a green, upward pointing triangle and an increase since the previous day, all would be well. The court manager or chief judge could relax and get a cup of coffee. On the other hand, the cup of coffee may need to wait if the morning check discloses a red downward-pointing triangle and a significant downturn in the CPI. By drilling down through several screens of progressively more detailed and less aggregated performance data, beginning with a display of four to ten core court performance measures that constitute the components of the CPI (think, again, of the Dow and its component company stock values), court managers and judges would be able to pinpoint the court, division, case type or resources needing attention.
The court manager may click the CPI icon to show a more detailed screen displaying a "balanced scorecard" of the four to ten performance measures that contribute to the CPI (e.g., adaptations of the National Center for State Courts’ CourTools or Appellate CourTools). Each of the performance measures on this screen are displayed in the same simple way as the CPI, i.e., a single score for the measure, a smaller number indicating a change in the measure, if any, and a triangle indicating a downturn or upturn in the measure.
The manager might discover, for example, that the early results of one measure - for example, an online survey of court employee engagement posted the morning of the previous day - is largely responsible for dragging down the court's CPI. From the "real time" data displayed on the screen the court manager learns that 37 percent of the court employees already had responded to the survey within 24 hours of their posting on the website.
A few more clicks by the court manager produces screens revealing important additional information about this measure including trends over time; the alignment of the measures with the court's management processes such as strategic planning, budgeting, quality improvement, and employee evaluation; related measures; and best practices in the performance area gauged by the measures. Further, a note on the screen showing trends of the measures over time may indicate that the response rate to previous administrations of the surveys was higher than 90 percent of all the court's employees and that, in the past, the early returns tended to be the most negative. The court manager decides not to jump to conclusions but to mobilize the management team in the event that the survey results do not improve as more of the court employees complete the online questionnaires.
In a few minutes of the morning, the court manager will have viewed the performance "dashboard" and determined the days "score" and what needs to be done. In this example, the court manager decides simply to alert and to mobilize the court's management team in case the measure that caused the CPI downturn does not show improvement in the days ahead.
Caseflow Timeliness and Efficiency Index (CTE Index)
A more modest index, first introduced in 2004, the Caseflow Timeliness and Efficiency Index (CTE Index), joins four measures associated with Standard 2.1, Case Processing, of the Trial Court Performance Standards and detailed in Trial Court Performance Standards and Measurement System, the first three of which are also part of the CourTools: time to disposition (on-time case processing), case clearance rate, age of pending caseload, and trial date certainty. These four measures of case processing, expressed as proportions, are reduced to one in the index. The CTE Index assigns different weights (i.e., relative importance expressed as number from 1 to 100) to these measures and requires the calculations as prescribed in the Standards with some conversions to accommodate the aggregation of the measures into an index. (See “Technical Note: The Caseflow Timeliness and Efficiency (CTE) Index.” Court Manager, Volume 18, Number 4, 2004, 8-9; reprinted in Steelman, David C. et al. Caseflow Management: The Heart of Court Management in the New Millennium. Williamsburg, VA: National Center for State Courts (2004)).
Ratings and Rankings of Courts
A few weeks ago, Newsweek published its first “best countries” ratings and rankings in which it set out to answer the simple question of which country would provide you the very best opportunities to live a healthy, safe, prosperous and upwardly mobile life. Newsweek rated and ranked 100 countries in five categories of well being – education, health, quality of life, economic competitiveness, and political environment (the U.S came out eleventh in the overall ranking). Americans love rankings, the top-ten of whatever including cities, high schools and universities, hospitals and toasters.
Amy Bach believes, as I do, that such ratings and rankings get people’s attention and create incentives and healthy competition. She thinks, as I do, that they will do the same for courts. (See “Toward the 50 ‘Best in Class’ Courts,” Made2Measure, April 23, 2008). “In communities across the country, people use statistics on hospitals, schools and other public services to decide where to live or how to vote,” writes Bach. “But while millions of Americans deal with their local criminal courts as defendants and victims each year, there is no comparable way to assess a judicial system and determine how well it provides basic legal services.”
Ratings and rankings of the world’s best places to live and America’s best public high schools, colleges, hospitals and parks hold two important lessons for judges and court managers who might benefit from the use of the CPI, the CTE Index, or a justice index envisioned by Bach and Tribe, especially those who might bristle at the idea of comparative performance measurement. (See “Ten Reasons Not to Measure Court Performance,” Made2Measure, November 19, 2008; and “How Do we Stack Up Against Other Courts? The Challenges of Comparative Performance Measurement,” Court Manager, Vol. 19, No.4 (Winter 2004-2005).
As Ms. Bach points out in her op-ed, the first lesson is that performance matters to citizens. The U.S. News & World Report annual rankings of America’s best public high schools, for example, are based on the key principle that a great high school must be able to produce measurable academic outcomes to show that it successfully educates all of its students across a range – a balanced scorecard – of performance indicators. Little else matters to parents as much.
The hallmarks of a high-performing court are the political will and the capacity to produce measureable outcomes that matter to the public and other stakeholders. The best courts ask themselves “How are we doing and how could we improve?” They measure their performance on a regular and continuous basis to gain knowledge and insight into new realities, opportunities and necessities and to convert that knowledge and insight into effective strategies for improvement.
The second lesson to be learned from such ratings and rankings is that similar comparisons and rankings of courts are likely to occur with increasing frequency, as Professor Tribe’s remarks to the state chief justices in July suggest. Court leaders are well advised to be prepared with their own court performance data.
The fact is that the best places to live, to send our kids to school, to get heart-bypass surgery and, yes, to get access to justice are of great interest to citizens. Courts not producing performance data to satisfy this interest are going to be asked why they’re not. What gets measured gets attention. What gets measured gets done!
It’s Not About Numbers
The measurement process uses numbers to provide an understandable and comparative result, but it is ultimately not about the numbers. It is about perception, understanding and insight. Ultimately, it is not the measure itself that is important but rather the questions that it compels us to confront.
Amy Bach concludes her groundbreaking book with these encouraging thoughts about a justice index and performance measurement in general: “Metrics would offer a mirror for people who work in the system, allowing them to see how their roles might have eroded at the expense of rights and public safety. Metrics alone are not an answer but they could be the beginning one. They are the tools we need to ask for the courts we deserve.”
© Copyright CourtMetrics and the National Center for State Courts 2010. All rights reserved.