An Index of the Moment


My colleagues and I at the National Center for State Courts and elsewhere have been thinking long and hard about a “justice index” here in the United States, as well as a similar index, the Global Court Performance Index (GCPI), applicable at an international scale. Though not unschooled in ways and means of performance measurement and management in justice systems, we’ve been daunted by the challenges that the construction of such indexes present.

What should the conceptual framework of the index be – the rule of law, justice systems institutions, legal frameworks, the experiences of citizens with the justice system, and so forth? Should the indicators that comprise the index be drawn from actual performance data available from justices systems, such as case clearance rates and median time in criminal defendants spend in custody before trial, or should the data be drawn from secondary sources, or should new measures be identified and defined? How much weighting should be given to the various indicators that comprise the indexes? How much vetting of the indexes is needed before development and deployment is initiated?


The Shoe-Thrower’s Index


Just do it! Aim, ready, shoot! That’s the bracing but refreshing lesson that I draw from The Economist’s use of an index of unrest in the Arab word that “aims to predict where the scent of jasmine may spread next” (The Economist, February 12, 2011; Economist.com/arabunrest). No doubt respectful of the challenges facing the construction of such an index, the Economist Intelligence Unit, the in-house research unit of The Economist that also publishes the Democracy Index, was not daunted. As a friend of mine said under similar circumstances, “I don’t know what’s good or bad at this stage, but I do know what nothing is!”


In its February 12 print edition, The Economist ran a table showing scores for 17 member countries of the Arab League using an index of unrest, or “Shoe-thrower's index.” Yemen came out on top with a score close to 90 on scale of 100, the most unstable. Libya, Egypt, Syria, and Iraq follow, each with scores close to 70. Qatar turned out the least unstable among the group with a score of a little more than 20.


Taking on the Challenges


The Economist intelligence Unit constructed the index by putting together a number of indicators that it felt were associated with instability – corruption, the age of the population, and so forth -- and ascribing different weights to them. (Where did these come from?  What august groups vetted them?) It removed a few of the members of the Arab League from consideration, including the Palestinian territories, Sudan and Somalia for lack of data, and Comoros and Djibouti, which do not have a great deal in common with the rest of the group. Some factors were discounted because they “are hard to put a number on.” The data on unemployment, for example, were too spotty to be comparable. The calculation of the index score ascribed a weighting of 35% for the share of the population that is under 25; 15% for the number of years the government has been in power; 15% for both corruption and lack of democracy as measured by existing indices; 10% for GDP per person; 5% for an index of censorship and 5% for the absolute number of people younger than 25.


Firing at Target Instead of Constantly Aiming


No one among us would dare oppose careful thought and analysis, but I do worry about an affliction that might be called the “aim, aim, aim syndrome” wherein one remains ever short of solution. Though one would expect an international news magazine to move more quickly than non-profit and public think-tanks and research institutions, one can’t help but be impressed with The Economist’s willingness not just to take aim at the challenges of an index of stability but actually to fire at the target with a solution, knowing full well that it might not have hit the mark perfectly, and that it might have to aim and fire again. In fact, in its online version, The Economist noted that Jordan comes out surprisingly low on the index, and suggested that the weighting “might need to be tweaked.” It ureged Interested readers to make suggestions and assured them that refinements will be made.

© Copyright CourtMetrics and the National Center for State Courts 2011. All rights reserved.

Popular posts from this blog

Top 10 Reasons for Performance Measurement

There is a growing demand of “show me the data” from citizens and taxpayers, legislators, executive agencies, oversight boards and communities – all of whom exchange money and other support (e.g., trust and confidence) for services -- that courts can ill afford to ignore. Courts need to measure their performance to meet the demand for public accountability. But the benefits of court performance extend beyond accountability including improved prediction, better understanding and control, influence, a sharpened focus on what is important, and discovery of better practices. Objective performance data is central to good management practices like strategic planning, performance-based budgeting, Balanced Scorecards, Total Quality Management (TQM), and Six Sigma. Performance measures allow court leaders and managers, policymakers, legislators and the public to evaluate court programs’ inputs (the resources allocated), outputs (direct results of program or service activities) and outcomes (bro...
Read more

Q & A: Outcome vs. Measure vs. Target vs. Standard

Q: Those of who write and speak about court performance measurement rely on their share of jargon -- and it's often confusing. What are the differences among the terms “outcome,” “measure,” “target” and “standard”? A: Socrates said that the beginning of wisdom is definition. We’ll see. Outcomes are the benefits or changes for the intended beneficiaries of a court’s programs and services. Outcomes may relate to knowledge, skills, attitudes, values, behavior, condition or status of the program participants or recipients of services. Examples include litigants' satisfaction with a court's courtesy and responsiveness, success of probation, time to case disposition, clarity of orders, integrity of case files, percent of mediation agreements, enforcement of orders, and perceived fairness of proceedings, as well as percent of expected expenditures and percent of revenues received. Outcomes should not be confused with the activities of those who run the court. (See the October 1...
Read more

Taming “Wild Problems”: Measure Everything That Matters

Every problem is not solvable, but every problem is measurable. This assertion is the key to tackling problems in public services including courts, business, and life in general. Surprising to me is that even astute scholars take exception to this, among them  is Russ Roberts, the John and Jean De Nault Research Fellow at Stanford University’s Hoover Institution, the president of Shalem College in Jerusalem, and host of the award-winning podcast EconTalk: Conversation for the Curious. The belief that certain things, dubbed “wild problems” by Professor Roberts, cannot be measured does a disservice to courts and other public and private organizations. There are many benefits to measuring everything that matters. It allows organizations to   focus on and to operationalize the most important problems and issues facing them,   to track progress over time and identify areas of improvement,   to compare their performance to that of other organizations and learn from the d...
Read more