Experience Counts for the Advancement of Performance Measurement and Management


Made2Measure returns today to regular postings after a long hiatus (September 9, 2013 was the last post) during which it was suspended to avoid potential conflicts of interests while its principal, Ingo Keilitz, was seconded to the World Bank and the National Center for State Courts.  

Performance measurement and management (PMM) is the (self) discipline of monitoring, analyzing, and using organizational performance data on a regular and continuous basis (in real or near-real time) for the purpose of improvements in efficiency and effectiveness, transparency and accountability, and increased public trust and confidence in government institutions. PMM is both a way of understanding the justice sector, as well as a discipline and a promising approach to solving serious global problems such as the high rate and length of incarceration, especially pre-trial detention.

Relatively Small Space in the Toolbox of International Development

Compared to two other disciplines and technologies of knowledge production and governance – program impact evaluations conducted by international donors such as the World Bank, and global indicators such as World Justice Project Rule of Law Index™ -- PMM occupies a relatively small space in the toolbox of international development. This despite increasing evidence that countries and their justice institutions who measure and manage their own performance are likely to enjoy more success and gain more legitimacy, trust and confidence in the eyes of those they serve.  Why is this happening?

Experience Development Counts

Over the last fifteen years or so, my colleagues and I at the International Consortium for Court Excellence, the National Center for State Courts, and of late at the College of William and Mary’s Institute for the Theory and Practice of International Relations (ITPIR), have spent much time on the design of PMM developing the “right” metrics, the “right” delivery of performance data (i.e., getting it into hands of the right people, at the right time, and in the right way), and the right actual use (i.e., injecting PMM into the very DNA of an institution’s business processes and operations). But good design alone has not, in my view, created more space in the tool box of international development for PMM.  Experience counts as much as design.

In today’s world of international development, well-designed approaches and products are not enough for potential users of those approaches (including donors) who value experience.  Before adoption or adaptation, they want to know what developing country or institution has built its capacity and/or actually used a particular performance measure such as duration of pre-trial custody, a measure that is part of the International Consortium’s International Framework for Court Excellence? For the most part, the answers to such questions are anecdotal and speculative.

Several things need to happen before PMM can emerge with a bigger role in international development.  First and foremost, the PMM that is taking place in countries and justice systems throughout the world needs to be well documented and known in terms of actual experiences, which it has not.  This impediment to a greater role of locally-owned or locally-directed PMM by host countries and institutions in international development is in large part a lack of effective incentives for PMM knowledge production and dissemination. Quite simply, much more is known about program impact evaluation and global indicators because their producers are in the business of publishing and disseminating their results, thereby burnishing their reputations. Many international development organization, multilateral development banks, bilateral aid agencies, private foundations, think-tanks, international activist groups, and consultancies publish books and articles, newsletters and blogs, touting the results of their program impact evaluations and global indicators, often through their own publishing arms.

The countries and their justice institutions actively using PMM, not so much. They lack the orientation, incentive, and the capacity. Promulgation of PMM and dissemination of results are inward directed to drive improvements in the organization's performance. And as pointed out in a thoughtful paper by Wade Channell ten years ago in the Carnegie [Endowment for International Peace] Papers Rule of Law Series, even when donors and projects do get involved with host institutions they have high incentives for guarding their information and lessons learned are unlikely to be shared widely.

The Justice Measurement Visibility Project

Last month my ITPIR colleagues Kate Conners, Maya Ravindran, Jonah Scharf, and I launched the Justice Measurement Visibility (JMV) Project, a project that aims to identify successful PMM in developing countries throughout the world focused on the eleven specific measure of the Global Measures of Court Performance, which is part of the International Framework for Court Excellence developed by the International Consortium.  For each of the eleven measures, we hope to be able to give a definitive answer to the question of what countries and their justice institutions have adopted or adapted it and what has been their experience.

Stay tuned here for more on the JMV Project.

 © Copyright CourtMetrics 2015. All rights reserved.

Popular posts from this blog

Top 10 Reasons for Performance Measurement

There is a growing demand of “show me the data” from citizens and taxpayers, legislators, executive agencies, oversight boards and communities – all of whom exchange money and other support (e.g., trust and confidence) for services -- that courts can ill afford to ignore. Courts need to measure their performance to meet the demand for public accountability. But the benefits of court performance extend beyond accountability including improved prediction, better understanding and control, influence, a sharpened focus on what is important, and discovery of better practices. Objective performance data is central to good management practices like strategic planning, performance-based budgeting, Balanced Scorecards, Total Quality Management (TQM), and Six Sigma. Performance measures allow court leaders and managers, policymakers, legislators and the public to evaluate court programs’ inputs (the resources allocated), outputs (direct results of program or service activities) and outcomes (bro...
Read more

Q & A: Outcome vs. Measure vs. Target vs. Standard

Q: Those of who write and speak about court performance measurement rely on their share of jargon -- and it's often confusing. What are the differences among the terms “outcome,” “measure,” “target” and “standard”? A: Socrates said that the beginning of wisdom is definition. We’ll see. Outcomes are the benefits or changes for the intended beneficiaries of a court’s programs and services. Outcomes may relate to knowledge, skills, attitudes, values, behavior, condition or status of the program participants or recipients of services. Examples include litigants' satisfaction with a court's courtesy and responsiveness, success of probation, time to case disposition, clarity of orders, integrity of case files, percent of mediation agreements, enforcement of orders, and perceived fairness of proceedings, as well as percent of expected expenditures and percent of revenues received. Outcomes should not be confused with the activities of those who run the court. (See the October 1...
Read more

Taming “Wild Problems”: Measure Everything That Matters

Every problem is not solvable, but every problem is measurable. This assertion is the key to tackling problems in public services including courts, business, and life in general. Surprising to me is that even astute scholars take exception to this, among them  is Russ Roberts, the John and Jean De Nault Research Fellow at Stanford University’s Hoover Institution, the president of Shalem College in Jerusalem, and host of the award-winning podcast EconTalk: Conversation for the Curious. The belief that certain things, dubbed “wild problems” by Professor Roberts, cannot be measured does a disservice to courts and other public and private organizations. There are many benefits to measuring everything that matters. It allows organizations to   focus on and to operationalize the most important problems and issues facing them,   to track progress over time and identify areas of improvement,   to compare their performance to that of other organizations and learn from the d...
Read more