Monday, September 09, 2013

Make It Official: The Case for In-Country Performance Measurement and Management

This is the third in a series of posts exploring the three international models of justice system performance measurement and management: (1) the EU Justice Scoreboard, (2) the Global Measures of Court Performance, and (3) the CourTools.

Law and justice scholars lament the spotty evidence linking rule of law and justice programs with development outcomes like economic growth, human rights, sound governance, and poverty reduction. A key cause of the evidence gap is the lack of emphasis on building in-country or domestic performance measurement and management in favor of third-party evaluations. All three international models, more or less, promote increased attention and investment in performance measurement and management – the regular and continuous monitoring, analysis, and use of performance information -- by justice officials and their institutions and justice systems themselves, not by third parties.  Capacity for performance measurement  is the ability of countries to meet user needs for good quality statistics on performance – usually those statistics that are consider to be “official” (i.e., those statistics produced by governments as a public good). Granted the three models are promulgated by organizations with considerable heft – the European Commission, the International Consortium for Court Excellence, and U.S.-based National Center for State Courts -- whose interests can be seen reflected in choice of metrics and the values, goals, and key success factors upon which they are built, all three emphasize in-country performance measurement.

Performance measurement and management differs from program evaluation in terms of goals and purposes, definitional style, sponsorship, organization, audience, functions, timing, and data interpretation rules. The differences are critical for justice reform.  For example, not the least of the differences is that performance measurement and management is designed to achieve the goals of the justice institutions, systems and the countries, not necessarily those of donors or other third parties, though harmonizing performance measurement across levels is a worthy aspiration (i.e., within an individual institution, across a justice system or country as a whole, and at the level of global governance). In contrast, program evaluation (and its variants including “monitoring and evaluation,” “impact evaluation,” and “evaluation research) may reflect mixed motives for justice reform and definitions of success that are aligned with the value sets and business goals of third parties.

Performance measurement is not yet the norm around the world, though it has a strong foothold in the in the European Union, Australia, the United States, large parts of the developed world, and in some developing countries where investments have been made in building domestic capacities (e.g., Moldova).  Most assessments of justice programs, processes, and reform initiatives are done instead by monitoring and evaluations instigated and conducted by third-parties, including donors, aid providers, and their agents (hordes of researchers, analysts, and consultants).  The abiding concern of these third parties is return on their investments, or something akin to it, and this concern does not necessarily align with the expressed purposes and fundamental responsibilities of the justice institutions, systems, or countries that are the “subjects” of their assessments. (It is tempting to include the crowded field of international indices of law and justice in this type of third-party assessment, but that is a topic to be dealt with in future posts.)

In a 2011 paper titled Problems of Power in the Design of Indicators of Safety and Justice in the Global South, Christopher E. Stone, President of the Open Society Foundations and formerly at Harvard Kennedy School’s  Program in Criminal Justice Policy and Management, urges those of us working in the domain of justice and safety, to emphasize what he calls “country-led indicator development.” He calls for the building of indicators “from the bottom up, supporting local ambitions and building on the legitimate authority close to the operations they seek to influence, rather than starting with ambitions and power at the global level.” He advocates for the design and development of “active indicators” that are distinguished from ones designed without the participation local authorities and are designed for use by officials of local institutions as management tools. He argues that such active indicators and a bottoms-up approach “is not only possible and practical, but has the potential to engage citizens and domestic leaders enthusiastically in a creative and democratic construction of justice.”
When it comes to answering the question “How are we doing?” justice institutions and systems do not like to be the mere “subjects” of the program evaluations and research of third parties, be they international donors, associations, or domestic stakeholders. It only seems reasonable and logical that they should like to see themselves as the proper and legitimate authority for issuing the “official” version of the truth. The official authorized view of a justice system’s performance must not only be owned by the justice system but be seen to be owned by the justice system.  When this is not the case, as it appears in many places around the globe, the legitimacy of and the public’s trust and confidence in the justice system suffer.
©Copyright CourtMetrics 2013. All rights reserved.

Labels: ,

Wednesday, July 31, 2013

International Models of Justice System Performance II

My last blog noted three promising international models of justice system performance measurement and management: (1) the EU Justice Scoreboard, (2) the Global Measures of Court Performance, and (3) the CourTools.  All three, more or less, aim for harmonization and consistent use of a common set of justice sector performance measures. There are, of course, differences among them, but it is their commonality that is potentially transformative for justice systems around the globe.

What distinguishes these three models from international global governance initiatives like the World Justice Project’s WJP Rule of Law Index™ and the American Bar Association’s Judicial Reform Index, as well as myriad program evaluations of justice and rule of law projects, is that they promote an approach to performance measurement and management that:
  • is essentially a bottom-up instead of a top-down strategy grounded in the local ambitions of justice institutions and justice systems exercising their legitimate authority;
  • relies on performance data collected and compiled by countries and their justice institutions themselves instead of international bodies and associated third parties whose indicators of justice may be seen as based on questionable goals (e.g., those of international donors) and other relatively weak sources of authority and legitimacy;
  • is based on institution-led or country-led measure development that is voluntary, facilitated but not dictated by the models; and,
  • aims for use of performance data by the countries’ justice system officials themselves to improve the governance and operations of the local justice sector.
Consistency or harmonization of justice performance measures across entire justice institutions or systems is not just an aspiration of little practical consequence if it is not achieved.  A country may find its performance in justice, rule of law, and safety measured by dozens of competing measures crafted by many different actors with various relationships with the country’s justice system. “The result,” as one development aid official put it to Harvard University criminal justice scholar Christopher Stone, “is that many developing countries are littered with the carcasses of failed indicators projects – the consultant paid and gone, and those charged with administering justice increasingly cynical about time wasted on measurement when there is real work to be done.”

My reading of Stone, who is now President of the Open Society Foundations, is that he would agree that the general approach of the three international models of performance measurement and management is not only possible and practical but has, in his words, the “potential to engage citizens and domestic leaders enthusiastically in a creative and democratic construction of justice.”  
©Copyright CourtMetrics 2013. All rights reserved.

Labels:

Sunday, July 21, 2013

International Models of Justice System Measurement and Managment

Three models of justice system performance measurement and management that have gained significant currency internationally over the last few years are:
·         the recently launched European Commission’s EU Justice Scoreboard based on the extensive work of the Council of Europe Commission for the Evaluation of the Efficiency of Justice (CEPEJ);

·         the Global Measures for Court Excellence developed by the International Consortium for Court Excellence as part of its International Framework for Court Excellence; and,

·          the National Center for State Courts' CourTools.
All three of these models aim for consistent use by all countries and states or, in the case of the EU Justice Scoreboard, all EU member states; all three elevate to the status of core measures of performance a limited number of indicators (e.g., case clearance rate and on-time case processing) developed from data collected by the justice institutions and systems themselves, data that has been referred to somewhat pejoratively as “administrative data” (though that negative characterization is bound to change in a few years); and all three are seen as comparative tools to support justice reforms. (Of course, other models exist, though they are intended primarily for single-country use.  Australian Government’s Productivity Commission’s Report of Government Services 2012 is an excellent example.)

Future posts will explore each of these models.   

©Copyright CourtMetrics 2013. All rights reserved.

Labels:

Monday, July 01, 2013

Q & A: A Compelling Story of Effective Use of Performance Measurement and Management

Q: Adopting government-wide or justice sector-wide performance measurement and management  to make public service or justice service more efficient and effective is politically attractive, even if elected officials are drawn mostly by the symbolic values of the key success factors with which performance measures are aligned (e.g., legitimacy, fairness, and public trust and confidence in institutions).  Successful leaders and managers are drawn to performance measurement because it informs uncertain decisions. Beyond such models as the European Commission’s EU Justice Scoreboard,  the Global Measures for Court Excellence  developed by the International Consortium for Court Excellence as part of its International Framework for Court Excellence, and the National Center for State Courts' CourTools, and their attendant exhortations to adopt performance measurement and management, are there any compelling success stories of effective use?
A: Yes. Almost five years ago, I wrote here about the Montana Supreme Court becoming the first high court to survey members of the state’s appellate bar and trial bench. Not only has the survey been done regularly and continually, we have evidence – a truly compelling success story – that judicial leaders and court managers in Montana actually put the performance information produced to good use.
In 2008, Mike McGrath was elected to an eight-year term as Chief Justice of the Montana Supreme Court. He took office early in 2009.  McGrath inherited a court-wide performance measurement system from his predecessor, Chief Justice Karla Gray, about which he was a bit skeptical.  However, his attention was drawn to a “consumer satisfaction” measure of how well the users of the Supreme Court thought his court was performing.  The measure is taken by an inexpensive anonymous on-line survey, which is conducted every other year. The survey asks about 1,000 first instance court judges, appellate attorneys, and law school faculty how they think the Montana Supreme Court is performing. Respondents rate the Court’s performance in several core areas, including whether the Court’s decisions are based on facts and applicable law, whether the Court’s published opinions explain deviations from established law and the adoption of new developments in law, and whether the Court treat judges and attorneys with courtesy and respect. The survey also asks about the Court’s timeliness in completing its work. In 2008, among all the items of the survey, this item is where the Court’s performance was the worst. Less than one-third (31.4%) of the survey respondents thought that the Court issued its opinions in a timely manner.
This bothered Chief Justice McGrath, as it would most of us. He decided to do something about it. He mobilized his fellow justices on the Supreme Court and staff of the administrative office of the courts to do some joint work together to improve the timeliness of the Court’s case processing. They started by taking a hard look at type of cases that were coming into the Court and how they were handling the demand. They increased the number of short “memo” opinions they issued; they shortened the standard lengths of written opinions and dissents; they tightened the time limits on receipts of appellate briefs; and they instituted procedures for coming to the aid of justices who were behind in their writing of opinions. Everyone got on board to fix the problem.
In September 2010, less than two years after McGrath took office, the Supreme Court conducted its second survey. The percent of respondents who believed the Court was issuing its opinions in a timely manner had increased from 31.4% to 81.8% - an increase in percentage points of over 50%. Quite amazingly, when the bench and bar survey was again taken last year, this percentage had increased to 94.9%.
Chief Justice McGrath’s dramatic success in the State of Montana demonstrates that what gets measured gets attention, and what gets measured gets done. And it does not necessarily take decades to get dramatic results.

 ©Copyright CourtMetrics 2013. All rights reserved.

Labels:

Friday, June 28, 2013

The “What Ifs” Along the Road in the Quest of a Justice Index


What if we could draw an indicator of the “health” of the justice systems throughout the United States – a Justice Index - from readily available public data that is impartial, reliable, objective and comparable, let’s say the amount of time that criminal defendants spend in custody without a trial, or even without any access to legal or justice services? Or the bail amounts imposed on misdemeanor defendants?  

This kind of a measure would not be perfect – no measure is – but it would be something, better than nothing. It would address questions of fairness and efficient use of public resources. That is, a criminal defendant languishing in jail awaiting trial suffers, at the least, a delay in justice. He or she is also occupying a jail bed and spending taxpayers’ money.

What if we then join this measure with another meaningful indicator of the health of our justice systems in counties and states, let’s say the degree to which jurisdictions prosecute domestic violence cases. That would be something else – maybe not yet a vital sign but better than just something.

What if we then join these two measures, in a transparent and verifiable way, with still other indicators drawn from readily available public data to form a balanced scorecard of measures of justice? Now that would be a long way from nothing, which is where we started. On that note, it is intriguing that on the international scene, compared to the United States, rule of law and justice indices, like the World Justice Project’s WJP Rule of Law Index™ and the American Bar Association’s Judicial Reform Index, constitute a relatively crowded field of inquiry.

What if we then made adjustments to this scorecard (for example, for different unemployment rates or other socioeconomic factors) without which people may hesitate to use the scorecard in assessing the health of the justice system in their communities?  (Warning. This is where the going might get a little rough as the usual suspects among the stakeholders in academia and politics weigh in with the usual arguments: We are unique and incomparable, you cannot reduce our performance to a number, and so forth. Of course, this only happens if their locale does not stack up well in the rankings.)

What if we then compared this scorecard against some other information about health of the justice system in communities throughout the U.S.?  For example, we might know that some jurisdictions in the U.S. simply would not seem to belong in the top ten or the bottom ten among communities in the U.S., just like Afghanistan would seem out of place in the top ten and Norway in the bottom ten countries around the world. We then could make further adjustments of things that were out of whack in the scorecard.

And then, what if we then could mashup this scorecard into a single number –  let’s say a number from 1 to 100 – a number that would be little bit more intuitive than the DOW industrial average, which we seemed to have gotten use to even though the number itself is meaningless. (What does 14,998.92, the DOW as I write this sentence, really mean?)

If we did all this, we would be very far down the road in the quest of a Justice Index that could be used to identify the positive deviants and bright spots among our communities.

This, incidentally, is the road taken and the quest of Measures for Justice, an organization started in 2011 by journalist and law professor Amy Bach, whose 2010 book, Ordinary Injustice: How America Holds Court, explored the everyday workings of local U.S. criminal courts.

©Copyright CourtMetrics 2013. All rights reserved.

Labels:

Monday, January 28, 2013

Measurement: Bill Gates’ Plan to Fix the World’s Problems


Bill Gates: My Plan To Fix the World’s Biggest Problems** Measure Them!  This is the headline of the lead article in the Review section of the weekend edition of the Wall Street Journal  written by Bill Gates, co-chair of the Bill & Melinda Gates Foundation and co-founder of Microsoft.   The article was accompanied by a large picture of transparent globe wrapped by a tape measure covering most of the first page of the section.  Its message is the theme of Made2Measure. Because it comes from Bill Gates, we tend to listen.
“In the past year,” Gates writes, “I have been struck by how important measurement is to improving the human condition. You can achieve incredible progress if you set a clear goal and find a measure that will drive progress toward that goal .. .This may seem basic, but it is amazing how often it is not done and how hard it is to get right.” He writes specifically about foreign aid among other areas like education, health care and agriculture .  “Historically, foreign aid has been measured in terms of the total amount of money invested—and during the Cold War, by whether a country stayed on our side—but not by how well it performed in actually helping people.”  When a country does commit to measuring outcomes toward concrete goals and clear performance targets, like Ethiopia did to bring health care to all of its citizens in 2004, it attracts donor money.  
Gates believes that one of the greatest benefits of measurement is the ability it gives government leaders to compare their countries’ performances with that of other countries and learn from the best.
Gates concludes his article with this: “The process I have described – setting clear goals, choosing an approach, measuring results, and then using those measurements to continually refine our approach – helps us to deliver the tools and services to everybody who will benefit, be they students in the U.S. or mothers in Africa.” 

© Copyright CourtMetrics 2013. All rights reserved.

Labels:

Wednesday, November 14, 2012

Big Data, Data Analytics, and the Access to Justice “Card”


It is not uncommon for court leaders and justice stakeholders to raise the access to justice “card” to subvert the closing or consolidation of courts.  If you close this court -- the argument goes -- citizens will be denied access to justice.  This is a strong argument because it typically evokes an emotional response irrespective of the information’s truth value, especially if the counter-arguments are couched in bland terms like efficiency and cost-savings. Trouble is that such arguments are heavy on rhetoric and light on hard evidence; and they often produce factionalism and political stalemate (see Court Consolidation in Mahoning County).   

In a 2005 survey of the public and of attorneys in California, respondents who were asked about eleven reasons that might keep someone from “going to court,” cited “travel distance to court from home” less often than eight of the other reasons including fees, cost of hiring an attorney, the time it takes to reach a decision, lack of child care, and the hours the court is open. 

How far does a court have to be before it is too far -- before the distance constitutes a legitimate barrier to access to the courts?  It seems that the best that we’ve been able to say in response is what we said back in the horse-and-buggy days, i.e., that it should be no more than a hard day’s ride on horseback.  

Big data and data analytics may not remove dysfunctional politics but it could lead to data-driven decision-making and calculated boldness by leaders advocating reform efforts such as court consolidation. For example, court location data could be compared against a number of public databases with information from inside and outside the justice system including Zip codes, populations, demographics of the population (race, age, disability), travel times between locations, numbers and types of cases heard by different courts, levels of courts, and availability of public transportation.  By tapping into the every growing mobile data traffic, we may eventually be able to use real-time distance data from court users on their smart phones.  

Results may allow advocates and opponents to compare various court consolidation models and say, for example, that the consolidation of courts from ten locations to three would increase the average distance and driving time to the nearest court from 3.1 miles and a ten minute commute to 4.5 miles and a fourteen minutes, with the overall average can be disaggregated by age of citizens, income levels, case type and so forth.  

Who knows, this type of information might actually allow us to update the “a hard day’s ride” access to justice standard. 

© Copyright CourtMetrics and the National Center for State Courts 2012. All rights reserved.

Labels: , ,