Monday, August 10, 2015

The Economist's Spotlight on the Problem of Pretrial Detention in Nigeria

Around the world, the misuse of pretrial detention, the time period defendants are incarcerated between arrest and trial) is massive. In Nigeria, Africa’s most populous country, the overuse of pretrial detention, most of it arbitrary and excessive, has reached “crushing proportions.” Of the 1,000 inmates in Nigeria’s Kiriki Maximum Security Prison, a total of 639 have not been convicted and are awaiting trial. Kayode Yukubu is among them. He was arrested in 2003.  After twelve years as Kiriki’s longest-serving inmate, no court trial date has yet been set for him.  He is among approximately 70 percent of Nigeria’s 56,785 pre-trial detainees who have not been sentenced, many of whom already have spent far longer time behind bars than the maximum period of the sentence for their alleged crimes. (Pretrial detention is intended to ensure an accused person will appear in court or pose a danger to others, not to punish or rehabilitate.)

The Economist spotlighted Nigeria’s pretrial detention with these figures last week (“Justice forgotten: The shocking number of pre-trial prisoners,” August 1, 2015, 45).  As enormous as the problem of pretrial detention is around the globe, much of it gets unnoticed.  But as the Economist article suggests, this may be changing because what gets measured gets attention, a maxim that has gained the status of received wisdom among many, perhaps most, of the international community.

While a spotlight on an enormous problem that is not uniquely a Nigerian one* does not guarantee solution, it’s a good start. A precise factual profile of its nature and scope is necessary to motivate and mobilize governments and the international community to do something about it. Such a precise factual profile of pretrial detention, however, should include not only figures such as the percentage of prisoners in pretrial detention, which are useful for general diagnosis but not for active performance management, but also what the Open Society Foundation’s president Christopher Stone, an international expert on criminal justice reform, referred to in a 2012 chapter as active indicators such as the length of pretrial detention.** 

Duration of pretrial custody, one of eleven performance measures of the International Framework of Court Excellence, is an actionable performance measure with the potential of having an outsized effect. It attracts the attention not only of justice system insiders (judges, prosecutors and defense attorneys, and law enforcement and corrections officials) but also of many groups and individuals in the private and non-profit sectors outside the formal justice systems who care about reducing crime, ensuring public safety, fighting poverty, reducing costs, making wise use of public resources, combating disease, promoting human rights, and making our legal systems more just.

Because duration of pretrial custody is clear, focused, and actionable, and because it is an easily understood indicator of an entrenched social problem, it is a potential rallying point for reform and improvement efforts that can bring government, citizens, groups, and organizations together in a solution economy. Justice institutions, social enterprises, and businesses can collaborate to reduce the average duration of pretrial custody, thereby creating efficiencies in court case processing that reduce the prison population and addressing a host of social problems.

Governments and their justice systems -- courts, prosecution and legal defense departments, ministries of justice, and law enforcement and corrections agencies -- could reap public trust and confidence simply by putting detailed data on pretrial custody into the public domain, making it available for real time feedback, and inviting social enterprises and businesses to join them in problem solving. They could, for example, collaborate with civil service organizations in identifying and examining the divergence between the mean and median number of days in pretrial custody among all criminal defendants. When the mean and median diverge, inflating or deflating the mean but not the median, it may be because relatively small groups of defendants (such as the poor and marginalized) are treated differently than the rest. The characteristics, treatment, and conditions (such as overcrowded and disease-ridden jails) of individuals with especially long pretrial prison stays (as well as especially short stays—which may occur among the rich, for example) could be examined for potential irregularities. So could these outliers’ experience with case processing and pretrial events, including factors related to the issuance of warrants, initial appearance and arraignment, charging practices, plea agreements, bail decision making, pretrial services, custody conditions, and alternative sentencing.  

What gets measured gets attention. And what gets attention by easy-to-understand, focused, and actionable performance measures might actually get done.


* Worldwide, close to a third of prisoners are pretrial detainees, and in some parts of the world like Bangladesh, India, Library, and Paraguay in addition to Nigeria, the majority are pretrial detainees. For an excellent overview, see the Open Society’s Justice Initiative 2014 publication Presumption of Guilt: The Global Oversuse of Pretrial Detention.

** Stone, Christopher (2012). Problems of Power in the Design of Indicators of Safety and Justice in the Global South. In Kevin E. Davis, Angelina Fisher, Benedict Kingsbury, and Sally Engle Merry (eds.), Governance by Indicators: Global Power Through Quantification and Rankings. Oxford University Press.

 © Copyright CourtMetrics 2015. All rights reserved.


Wednesday, July 08, 2015

Experience Counts for the Advancement of Performance Measurement and Management

Made2Measure returns today to regular postings after a long hiatus (September 9, 2013 was the last post) during which it was suspended to avoid potential conflicts of interests while its principal, Ingo Keilitz, was seconded to the World Bank and the National Center for State Courts.  

Performance measurement and management (PMM) is the (self) discipline of monitoring, analyzing, and using organizational performance data on a regular and continuous basis (in real or near-real time) for the purpose of improvements in efficiency and effectiveness, transparency and accountability, and increased public trust and confidence in government institutions. PMM is both a way of understanding the justice sector, as well as a discipline and a promising approach to solving serious global problems such as the high rate and length of incarceration, especially pre-trial detention.

Relatively Small Space in the Toolbox of International Development

Compared to two other disciplines and technologies of knowledge production and governance – program impact evaluations conducted by international donors such as the World Bank, and global indicators such as World Justice Project Rule of Law Index™ -- PMM occupies a relatively small space in the toolbox of international development. This despite increasing evidence that countries and their justice institutions who measure and manage their own performance are likely to enjoy more success and gain more legitimacy, trust and confidence in the eyes of those they serve.  Why is this happening?

Experience Development Counts

Over the last fifteen years or so, my colleagues and I at the International Consortium for Court Excellence, the National Center for State Courts, and of late at the College of William and Mary’s Institute for the Theory and Practice of International Relations (ITPIR), have spent much time on the design of PMM developing the “right” metrics, the “right” delivery of performance data (i.e., getting it into hands of the right people, at the right time, and in the right way), and the right actual use (i.e., injecting PMM into the very DNA of an institution’s business processes and operations). But good design alone has not, in my view, created more space in the tool box of international development for PMM.  Experience counts as much as design.

In today’s world of international development, well-designed approaches and products are not enough for potential users of those approaches (including donors) who value experience.  Before adoption or adaptation, they want to know what developing country or institution has built its capacity and/or actually used a particular performance measure such as duration of pre-trial custody, a measure that is part of the International Consortium’s International Framework for Court Excellence? For the most part, the answers to such questions are anecdotal and speculative.

Several things need to happen before PMM can emerge with a bigger role in international development.  First and foremost, the PMM that is taking place in countries and justice systems throughout the world needs to be well documented and known in terms of actual experiences, which it has not.  This impediment to a greater role of locally-owned or locally-directed PMM by host countries and institutions in international development is in large part a lack of effective incentives for PMM knowledge production and dissemination. Quite simply, much more is known about program impact evaluation and global indicators because their producers are in the business of publishing and disseminating their results, thereby burnishing their reputations. Many international development organization, multilateral development banks, bilateral aid agencies, private foundations, think-tanks, international activist groups, and consultancies publish books and articles, newsletters and blogs, touting the results of their program impact evaluations and global indicators, often through their own publishing arms.

The countries and their justice institutions actively using PMM, not so much. They lack the orientation, incentive, and the capacity. Promulgation of PMM and dissemination of results are inward directed to drive improvements in the organization's performance. And as pointed out in a thoughtful paper by Wade Channell ten years ago in the Carnegie [Endowment for International Peace] Papers Rule of Law Series, even when donors and projects do get involved with host institutions they have high incentives for guarding their information and lessons learned are unlikely to be shared widely.

The Justice Measurement Visibility Project

Last month my ITPIR colleagues Kate Conners, Maya Ravindran, Jonah Scharf, and I launched the Justice Measurement Visibility (JMV) Project, a project that aims to identify successful PMM in developing countries throughout the world focused on the eleven specific measure of the Global Measures of Court Performance, which is part of the International Framework for Court Excellence developed by the International Consortium.  For each of the eleven measures, we hope to be able to give a definitive answer to the question of what countries and their justice institutions have adopted or adapted it and what has been their experience.

Stay tuned here for more on the JMV Project.

 © Copyright CourtMetrics 2015. All rights reserved.


Monday, September 09, 2013

Make It Official: The Case for In-Country Performance Measurement and Management

This is the third in a series of posts exploring the three international models of justice system performance measurement and management: (1) the EU Justice Scoreboard, (2) the Global Measures of Court Performance, and (3) the CourTools.

Law and justice scholars lament the spotty evidence linking rule of law and justice programs with development outcomes like economic growth, human rights, sound governance, and poverty reduction. A key cause of the evidence gap is the lack of emphasis on building in-country or domestic performance measurement and management in favor of third-party evaluations. All three international models, more or less, promote increased attention and investment in performance measurement and management – the regular and continuous monitoring, analysis, and use of performance information -- by justice officials and their institutions and justice systems themselves, not by third parties.  Capacity for performance measurement  is the ability of countries to meet user needs for good quality statistics on performance – usually those statistics that are consider to be “official” (i.e., those statistics produced by governments as a public good). Granted the three models are promulgated by organizations with considerable heft – the European Commission, the International Consortium for Court Excellence, and U.S.-based National Center for State Courts -- whose interests can be seen reflected in choice of metrics and the values, goals, and key success factors upon which they are built, all three emphasize in-country performance measurement.

Performance measurement and management differs from program evaluation in terms of goals and purposes, definitional style, sponsorship, organization, audience, functions, timing, and data interpretation rules. The differences are critical for justice reform.  For example, not the least of the differences is that performance measurement and management is designed to achieve the goals of the justice institutions, systems and the countries, not necessarily those of donors or other third parties, though harmonizing performance measurement across levels is a worthy aspiration (i.e., within an individual institution, across a justice system or country as a whole, and at the level of global governance). In contrast, program evaluation (and its variants including “monitoring and evaluation,” “impact evaluation,” and “evaluation research) may reflect mixed motives for justice reform and definitions of success that are aligned with the value sets and business goals of third parties.

Performance measurement is not yet the norm around the world, though it has a strong foothold in the in the European Union, Australia, the United States, large parts of the developed world, and in some developing countries where investments have been made in building domestic capacities (e.g., Moldova).  Most assessments of justice programs, processes, and reform initiatives are done instead by monitoring and evaluations instigated and conducted by third-parties, including donors, aid providers, and their agents (hordes of researchers, analysts, and consultants).  The abiding concern of these third parties is return on their investments, or something akin to it, and this concern does not necessarily align with the expressed purposes and fundamental responsibilities of the justice institutions, systems, or countries that are the “subjects” of their assessments. (It is tempting to include the crowded field of international indices of law and justice in this type of third-party assessment, but that is a topic to be dealt with in future posts.)

In a 2011 paper titled Problems of Power in the Design of Indicators of Safety and Justice in the Global South, Christopher E. Stone, President of the Open Society Foundations and formerly at Harvard Kennedy School’s  Program in Criminal Justice Policy and Management, urges those of us working in the domain of justice and safety, to emphasize what he calls “country-led indicator development.” He calls for the building of indicators “from the bottom up, supporting local ambitions and building on the legitimate authority close to the operations they seek to influence, rather than starting with ambitions and power at the global level.” He advocates for the design and development of “active indicators” that are distinguished from ones designed without the participation local authorities and are designed for use by officials of local institutions as management tools. He argues that such active indicators and a bottoms-up approach “is not only possible and practical, but has the potential to engage citizens and domestic leaders enthusiastically in a creative and democratic construction of justice.”
When it comes to answering the question “How are we doing?” justice institutions and systems do not like to be the mere “subjects” of the program evaluations and research of third parties, be they international donors, associations, or domestic stakeholders. It only seems reasonable and logical that they should like to see themselves as the proper and legitimate authority for issuing the “official” version of the truth. The official authorized view of a justice system’s performance must not only be owned by the justice system but be seen to be owned by the justice system.  When this is not the case, as it appears in many places around the globe, the legitimacy of and the public’s trust and confidence in the justice system suffer.
©Copyright CourtMetrics 2013. All rights reserved.

Labels: ,

Wednesday, July 31, 2013

International Models of Justice System Performance II

My last blog noted three promising international models of justice system performance measurement and management: (1) the EU Justice Scoreboard, (2) the Global Measures of Court Performance, and (3) the CourTools.  All three, more or less, aim for harmonization and consistent use of a common set of justice sector performance measures. There are, of course, differences among them, but it is their commonality that is potentially transformative for justice systems around the globe.

What distinguishes these three models from international global governance initiatives like the World Justice Project’s WJP Rule of Law Index™ and the American Bar Association’s Judicial Reform Index, as well as myriad program evaluations of justice and rule of law projects, is that they promote an approach to performance measurement and management that:
  • is essentially a bottom-up instead of a top-down strategy grounded in the local ambitions of justice institutions and justice systems exercising their legitimate authority;
  • relies on performance data collected and compiled by countries and their justice institutions themselves instead of international bodies and associated third parties whose indicators of justice may be seen as based on questionable goals (e.g., those of international donors) and other relatively weak sources of authority and legitimacy;
  • is based on institution-led or country-led measure development that is voluntary, facilitated but not dictated by the models; and,
  • aims for use of performance data by the countries’ justice system officials themselves to improve the governance and operations of the local justice sector.
Consistency or harmonization of justice performance measures across entire justice institutions or systems is not just an aspiration of little practical consequence if it is not achieved.  A country may find its performance in justice, rule of law, and safety measured by dozens of competing measures crafted by many different actors with various relationships with the country’s justice system. “The result,” as one development aid official put it to Harvard University criminal justice scholar Christopher Stone, “is that many developing countries are littered with the carcasses of failed indicators projects – the consultant paid and gone, and those charged with administering justice increasingly cynical about time wasted on measurement when there is real work to be done.”

My reading of Stone, who is now President of the Open Society Foundations, is that he would agree that the general approach of the three international models of performance measurement and management is not only possible and practical but has, in his words, the “potential to engage citizens and domestic leaders enthusiastically in a creative and democratic construction of justice.”  
©Copyright CourtMetrics 2013. All rights reserved.


Sunday, July 21, 2013

International Models of Justice System Measurement and Managment

Three models of justice system performance measurement and management that have gained significant currency internationally over the last few years are:
·         the recently launched European Commission’s EU Justice Scoreboard based on the extensive work of the Council of Europe Commission for the Evaluation of the Efficiency of Justice (CEPEJ);

·         the Global Measures for Court Excellence developed by the International Consortium for Court Excellence as part of its International Framework for Court Excellence; and,

·          the National Center for State Courts' CourTools.
All three of these models aim for consistent use by all countries and states or, in the case of the EU Justice Scoreboard, all EU member states; all three elevate to the status of core measures of performance a limited number of indicators (e.g., case clearance rate and on-time case processing) developed from data collected by the justice institutions and systems themselves, data that has been referred to somewhat pejoratively as “administrative data” (though that negative characterization is bound to change in a few years); and all three are seen as comparative tools to support justice reforms. (Of course, other models exist, though they are intended primarily for single-country use.  Australian Government’s Productivity Commission’s Report of Government Services 2012 is an excellent example.)

Future posts will explore each of these models.   

©Copyright CourtMetrics 2013. All rights reserved.


Monday, July 01, 2013

Q & A: A Compelling Story of Effective Use of Performance Measurement and Management

Q: Adopting government-wide or justice sector-wide performance measurement and management  to make public service or justice service more efficient and effective is politically attractive, even if elected officials are drawn mostly by the symbolic values of the key success factors with which performance measures are aligned (e.g., legitimacy, fairness, and public trust and confidence in institutions).  Successful leaders and managers are drawn to performance measurement because it informs uncertain decisions. Beyond such models as the European Commission’s EU Justice Scoreboard,  the Global Measures for Court Excellence  developed by the International Consortium for Court Excellence as part of its International Framework for Court Excellence, and the National Center for State Courts' CourTools, and their attendant exhortations to adopt performance measurement and management, are there any compelling success stories of effective use?
A: Yes. Almost five years ago, I wrote here about the Montana Supreme Court becoming the first high court to survey members of the state’s appellate bar and trial bench. Not only has the survey been done regularly and continually, we have evidence – a truly compelling success story – that judicial leaders and court managers in Montana actually put the performance information produced to good use.
In 2008, Mike McGrath was elected to an eight-year term as Chief Justice of the Montana Supreme Court. He took office early in 2009.  McGrath inherited a court-wide performance measurement system from his predecessor, Chief Justice Karla Gray, about which he was a bit skeptical.  However, his attention was drawn to a “consumer satisfaction” measure of how well the users of the Supreme Court thought his court was performing.  The measure is taken by an inexpensive anonymous on-line survey, which is conducted every other year. The survey asks about 1,000 first instance court judges, appellate attorneys, and law school faculty how they think the Montana Supreme Court is performing. Respondents rate the Court’s performance in several core areas, including whether the Court’s decisions are based on facts and applicable law, whether the Court’s published opinions explain deviations from established law and the adoption of new developments in law, and whether the Court treat judges and attorneys with courtesy and respect. The survey also asks about the Court’s timeliness in completing its work. In 2008, among all the items of the survey, this item is where the Court’s performance was the worst. Less than one-third (31.4%) of the survey respondents thought that the Court issued its opinions in a timely manner.
This bothered Chief Justice McGrath, as it would most of us. He decided to do something about it. He mobilized his fellow justices on the Supreme Court and staff of the administrative office of the courts to do some joint work together to improve the timeliness of the Court’s case processing. They started by taking a hard look at type of cases that were coming into the Court and how they were handling the demand. They increased the number of short “memo” opinions they issued; they shortened the standard lengths of written opinions and dissents; they tightened the time limits on receipts of appellate briefs; and they instituted procedures for coming to the aid of justices who were behind in their writing of opinions. Everyone got on board to fix the problem.
In September 2010, less than two years after McGrath took office, the Supreme Court conducted its second survey. The percent of respondents who believed the Court was issuing its opinions in a timely manner had increased from 31.4% to 81.8% - an increase in percentage points of over 50%. Quite amazingly, when the bench and bar survey was again taken last year, this percentage had increased to 94.9%.
Chief Justice McGrath’s dramatic success in the State of Montana demonstrates that what gets measured gets attention, and what gets measured gets done. And it does not necessarily take decades to get dramatic results.

 ©Copyright CourtMetrics 2013. All rights reserved.


Friday, June 28, 2013

The “What Ifs” Along the Road in the Quest of a Justice Index

What if we could draw an indicator of the “health” of the justice systems throughout the United States – a Justice Index - from readily available public data that is impartial, reliable, objective and comparable, let’s say the amount of time that criminal defendants spend in custody without a trial, or even without any access to legal or justice services? Or the bail amounts imposed on misdemeanor defendants?  

This kind of a measure would not be perfect – no measure is – but it would be something, better than nothing. It would address questions of fairness and efficient use of public resources. That is, a criminal defendant languishing in jail awaiting trial suffers, at the least, a delay in justice. He or she is also occupying a jail bed and spending taxpayers’ money.

What if we then join this measure with another meaningful indicator of the health of our justice systems in counties and states, let’s say the degree to which jurisdictions prosecute domestic violence cases. That would be something else – maybe not yet a vital sign but better than just something.

What if we then join these two measures, in a transparent and verifiable way, with still other indicators drawn from readily available public data to form a balanced scorecard of measures of justice? Now that would be a long way from nothing, which is where we started. On that note, it is intriguing that on the international scene, compared to the United States, rule of law and justice indices, like the World Justice Project’s WJP Rule of Law Index™ and the American Bar Association’s Judicial Reform Index, constitute a relatively crowded field of inquiry.

What if we then made adjustments to this scorecard (for example, for different unemployment rates or other socioeconomic factors) without which people may hesitate to use the scorecard in assessing the health of the justice system in their communities?  (Warning. This is where the going might get a little rough as the usual suspects among the stakeholders in academia and politics weigh in with the usual arguments: We are unique and incomparable, you cannot reduce our performance to a number, and so forth. Of course, this only happens if their locale does not stack up well in the rankings.)

What if we then compared this scorecard against some other information about health of the justice system in communities throughout the U.S.?  For example, we might know that some jurisdictions in the U.S. simply would not seem to belong in the top ten or the bottom ten among communities in the U.S., just like Afghanistan would seem out of place in the top ten and Norway in the bottom ten countries around the world. We then could make further adjustments of things that were out of whack in the scorecard.

And then, what if we then could mashup this scorecard into a single number –  let’s say a number from 1 to 100 – a number that would be little bit more intuitive than the DOW industrial average, which we seemed to have gotten use to even though the number itself is meaningless. (What does 14,998.92, the DOW as I write this sentence, really mean?)

If we did all this, we would be very far down the road in the quest of a Justice Index that could be used to identify the positive deviants and bright spots among our communities.

This, incidentally, is the road taken and the quest of Measures for Justice, an organization started in 2011 by journalist and law professor Amy Bach, whose 2010 book, Ordinary Injustice: How America Holds Court, explored the everyday workings of local U.S. criminal courts.

©Copyright CourtMetrics 2013. All rights reserved.