Made2Measure

The use of breakouts (disaggregation) of performance measures can reveal useful information that otherwise is hidden. Common breakouts of time to disposition measures, for example, are case type and location. They identify differences in timeliness of case processing across different case types and court locations. Other less common breakouts of on-time case processing measures may identify inequities among groups by income levels and indicate whether a court handles cases more swiftly for affluent litigants. Another potentially very useful breakout that extends the meaning of on-time case processing measures is one that focuses on the median days that defendants spend in pretrial detention or custody. This breakout may be readily accessible via automated case management systems. It is a measure of jail overcrowding that is relevant to courts and one that may reap court managers considerable political capital in their criminal justice communities. Many court managers in cities like S...

While court orders establish a variety of sanctions in criminal and civil cases, monetary penalties are clearly understood and measurable. In an article that appeared in hundreds of newspapers and magazines last Sunday, Associate Press (AP) reporters Martha Mendoza and Christopher Sullivan, put the national spotlight on one of the ten court performance measures of the CourTools : collection of monetary penalties – payments collected expressed as a percentage of total monetary penalties imposed by a court. Measure 7 of the CourTools assesses how well a court takes responsibility for the enforcement of its orders requiring payment of monetary penalties. A secondary yet important aim of this measure is evaluating the efficiency of the court's internal processes for collecting and distributing monetary penalties including civil damage awards, child support, traffic and criminal fines. In its study, the AP examined federal financial penalty enforcement across the nation and found (al...

We measure what we care about, what we know. But that’s not enough. The West has spent trillions of dollars to help poor countries, with little real success in reversing poverty and disease. William Easterly, a former World Bank Economist and now a professor at New York University thinks he knows why. For one thing, he argues that we’re measuring success the wrong way – by how much money rich countries spend on poor ones. That’s like judging a film based on its budget Easterly argues in his recently published book, The White Man’s Burden: Why the West’s Efforts to Aid the Rest Have Done So Much Ill and So Little Good (Penguin Press, 2006). Instead, Easterly argues, we should measure results and give aid programs and workers timely feedback about what works and doesn’t work to improve lives. In other words, we need to measure outcomes , real improvements in the living conditions of the intended benefactors of the foreign aid. Governments have always been good at measuring one thing: s...

Founded by former executives of McKinsey & Co. and IBM, Evalueserve is a company based in New Delhi, India, that provides business intelligence to companies around the world. It is just one of a growing number of suppliers of outsourced services. Plummeting telecommunications costs have spawned “remote services” whereby providers in such far flung places as India work for customers in the U.S. Outsourcing -- a popular buzzword for what used to be called “contracting out” -- can be defined simply as the delegation of certain operations or functions from an internal to an external entity that specializes in that operation or function. For courts, outsourcing such functions as fine and fee collections may be a business decision to provide better service at lower costs. In a previous posting, I recommended that the central function of court research divisions should be the design, building and maintenance of court performance measurement systems (CPMSs). I suggested then that cour...

Knowledge is power. The faster we get the information the better. In more and more aspects of our lives, we expect information in real-time or near real-time. Real-time performance data are transmitted to users immediately, as soon as the data are produced. Delay is limited to the actual time required to transmit the data to the users. In a communications system, propagation delay refers to the time lag between the departure of a signal from the source and the arrival of the signal at the destination. Think of the real-time data you get from the speedometer of your car as you drove to work this morning. How much propagation delay would you tolerate? I’d say, none. Zippo! We push on the brake or accelerator and we expect information feedback in real-time. Now think of the way we typically produce and transmit court performance data. How much propagation delay are we willing to tolerate? It seems like plenty. Consider the status quo of court performance information generation and dis...

Three keys to successful court performance measurement are: (1) finding the right measures, (2) getting them to the right people at the right time, and (3) ensuring that the right people put the measures to good use. The first two depend on an effective design process . The third is the key to implementation and it will not happen by itself. A previous posting recommended two simple techniques to facilitate implementation of a court performance measurement system (CPMS): (1) making the review of the court's core performance results a permanent agenda item on the court’s executive meetings and (2) assigning ownership of core measures to key managers. “It’s too simpleminded,” responded a court management consultant. He was skeptical that simply putting performance measures on the agenda and talking about them would contribute to effective implementation. Of course, he’s right. But maybe simple-minded is just what is needed. COMPSTAT , short for “computer statistics,” is a manag...

Wherever you look, you see best practices . Sounds like a bit of wisdom from Yogi Berra, doesn’t it? We have best practices for appellate courts and for problem solving courts, for racial fairness, for reducing family violence, for collection of traffic fines, for electronic document digital discovery, for human resource management, for ensuring public trust and confidence and, seemingly, for everything in court policy and operations under the sun. To put it bluntly, even though I’m guilty of using it in the past, I’m tired of best practices and would like to replace the concept with evidence-based best practices or simply evidence-based practices . Here’s why. While I’m sure that serious policymakers associate the concept of best practices with empirical evidence to back up the word “best” – the National Center for State Courts, for example, provides information on proven best practices – but that’s not the way the concept is used and generally understood. Instead, best practice...

Performance measurement, no doubt, influences beliefs. That’s why performance measurement results have inherent value, especially in these days of accountability and evidence-based decision making. However, we need to keep in mind that even well-established conclusions based on sound performance measurement may not lead to changes in policy, programs or procedures. Why? Answer: Because policymakers view performance measurement data in conjunction with compelling testimony from ordinary citizens and stakeholders, newspaper articles, and politics. We may emphasize performance measurement data, but others may not. Even when policymakers embrace performance measurement, they need to balance multiple competing demands for resources, including the attentions of court leaders and managers. That’s just the way it is, and it’s unlikely to change. We need to work well in this reality. We need to recognize it and be deliberate about reversing the negative attitudes and breaking down the resistan...

Previous posts ( Implementing Performance Measurement , November 12, 2005; and Implementation: How It Looks When You Get There , December 13, 2005), explored what it takes to implement a court performance measurement system (CPMS). It was pointed out that is one thing to build a CPMS and quite another to get it to be used effectively. Even a well-conceived, well-designed CPMS will not necessarily get used unless it is woven into the very fabric of a court’s management practices and processes. “If you build it, they will come” works in the movies (e.g., in the 1989 film Field of Dreams ) but not for court performance measurement The history of electronic filing – a system that uses digital documents instead of paper filings – is instructive. Implementation may be inevitable, but moving from roll-out to full use of the system takes time and effort. The ABA Journal reports that the move to digital filing systems in the State courts is often very slow and confusing. Even though the system ...

#1 Start with the outcome: You cannot identify and develop a good performance measure unless you first define what you want to achieve. You must commit to the purpose of the measure, not just the performance metric. A good measure is aligned with one or more of a court's key performance areas or success factors (e.g., access to justice). It emphasizes the condition or status of the recipients of court services or the participants in court programs (outcomes) over that of internal aspects of court processes, programs and activities (inputs and outputs) -- that is, they indicate results rather than resources, activities, and level of effort. #2 Vital few instead of trivial many: Develop a handful of core measures – seven to twelve. A core performance measure is a primary indicator – like the speedometer on your car’s dashboard -- of an important area of court performance. There’s nothing wrong with a court having many measures and indicators of their inputs, outputs and outcomes -...

Why Is This Measure Not Used As Much As It Should? How long does it take? How much does it cost? These are the most frequently asked questions by customers, clients and other consumers of (or stakeholders in the provision of) services. We ask these questions when we consider seeking the assistance of a lawyer, getting our car fixed and remodeling our kitchen. Even though cost analysis is still relatively new in courts, it is somewhat of a mystery why courts are comfortably fixated on answers to the first question while all but neglecting answers to the second. Time to disposition -- the percent of cases disposed or resolved within established timeframes -- is a performance measure that has been around for more than 25 years and is used by a majority of courts. On the other hand, cost per case is a perfromance measure that -- to the best of my knowledge -- is considered seriously by only a in a handful of courts and court systems (e.g., the Arizona Superior Court in Maricopa County an...

From the What-Will-They-Measure-Next Department comes news from the world of curling – the Olympic sport in which players brush the ice in front of a sliding, 42 pound granite stone. According to the latest issue of Wired , UK researchers spent $89,000 developing a sensor-laden broom that measures the velocity of the curlers’ strokes, the strain they put on the brush, and the precise temperature of the ice at the time. “Do you want to be sweeping your heart out and not know what you’re doing? asks Mike Hay, the UK’s Olympic coach. It’s certainly something to think about! Copyright CourtMetrics 2006. All rights reserved.

A court unwilling or unable to measure and to account for its performance rigorously and responsibly is unlikely to achieve independence and accountability. Since 2003 my colleagues and I have been experimenting with a tool to assess whether a court's capabilities for performance measurement are “measuring up.” Using a simple self-administered test, one version of this tool allows a court to assess its readiness -- in terms of demonstrated proficiency, present and future capacity, and political will -- to take the core court performance measures prescribed by the CourTools . Inclusion of this measure among a court’s core measures sends a powerful message that a way to improve independence and accountability is for a court to develop its capacity to measure and account for its performance. The Performance Institute , a private, non-partisan think tank seeking to improve government performance through the principles of performance, competition, accountability, and transparency, has d...

Q: My court recently completed Measure 1 of the CourTools and found out that 73% of the court users who completed the survey “agreed” or “strongly agreed” that the court was accessible, prompt, respectful, courteous, and fair. What does the score mean? Is that good or bad? A: It is true that without links to and comparisons with other referents a score of 73% has a very limited meaning. Fortunately, it is relatively easy to find and to link this specific score to a number of referents – the breakouts of the aggregate metric, the same metric over time, and the same metric in other courts -- so as to imbue the metric with meaning. (A metric refers to the numbers that a measure uses to describe the attribute being measured. In this case, the measure is the satisfaction of the court users with the court and the metric is the percent of users who “agreed” or “strongly agreed” with the items in the survey.) Breakouts as Referents Let’s assume that 73% agreement is the average (aggr...

Q: Those of who write and speak about court performance measurement rely on their share of jargon -- and it's often confusing. What are the differences among the terms “outcome,” “measure,” “target” and “standard”? A: Socrates said that the beginning of wisdom is definition. We’ll see. Outcomes are the benefits or changes for the intended beneficiaries of a court’s programs and services. Outcomes may relate to knowledge, skills, attitudes, values, behavior, condition or status of the program participants or recipients of services. Examples include litigants' satisfaction with a court's courtesy and responsiveness, success of probation, time to case disposition, clarity of orders, integrity of case files, percent of mediation agreements, enforcement of orders, and perceived fairness of proceedings, as well as percent of expected expenditures and percent of revenues received. Outcomes should not be confused with the activities of those who run the court. (See the October 1...

People must sign on to the purpose of the performance measure, the key results it indicates, not just the metric. Performance measures are derived from the mission and strategic goals of a court and the factors important to its stakeholders. Decisions about what to measure are, to a large extent, collective judgments that reflect the intended use of the performance information (e.g., major reform, public accountability, program improvement or resource allocation) and the needs and desires (e.g., efficiency, equity, quality, or improving the public confidence in the courts) of the court’s stakeholders. Kathryn E. Newcomer, professor and chair in the Department of Public Administration at George Washington University, aptly noted in Using Performance Measurement to Improve Public and Nonprofit Programs (Josses-Bass, 1997), that, ultimately, the performance of programs and organizations is a socially constructed and not an objective reality. Even when we have identified a performance mea...

While experts in performance measurement might differ in their views about how many is best, they agree that users can handle only a critical few performance measures at any one time and in one view . A court may have only one core measure of community safety, security and well being like recidivism, for example, but it may use numerous subordinate measures of recidivism at a strategic level (e.g., average recidivism by case categoreis and types), tactical level (e.g., recidivism for certain units like drug courts and probation departments) and operational levels (e.g., recidivism of probationers who particpated in a particular rehabilitation program). In addition, a court or its justice partners may usefully monitor, analyze and manage a host of meaures related to community safety, security and well being like the change in police contacts with families with cases in the jurisdiction of a family court. The point is that the users of these recidivism measures have selective attent...

A: Why not simply adopt the set of 10 performance measures prescribed by the CourTools or by other authoritative sources and dispense with the six steps for building a court performance measurement system (See October 15, 2005, posting, Six-Step Process for Building an Effective Court Performance System (CPMS) )? After all, the CourTools are endorsed by the Conference of Chief Justices (CCJ) and the Conference of State Court Administrators (COSCA) (see Resolution 14 ). Why reinvent the wheel? Q: This is an important question, especially in view of the significant investment of time and resources required for building a court performance measurement system (CPMS). Unquestionably, models such as the CourTools are extremely valuable. No individual court, no court system, and no justice system (that includes courts) considering performance measurement should – and none of the dozen or so my colleagues have assisted in the last year did -- proceed without first studying the CourTools ....

In his 1996 book, Keeping Score: Using the Right Metrics to Drive World-Class Performance (Quality Resources, a Division of Kraus Productivity Organization Ltd.), Mark Graham Brown delivered a practical common-sense guide on how to develop and use performance measures as tools for world-class performance. In his latest book, Get It, Set It, Move It, Prove It: 60 Ways to Get Results in Your Organization (Productivity Press, 2004), he puts a sharper point on some the issues he raised in his earlier book. Forget “statistically significant” when assessing organizational performance, he writes in Chapter 49. Distinguishing between organizational performance measurement and science, he states that the concept of statistical significance is critically important for science to rule out that the differences between the experimental and control groups are due to the independent variable and not to chance. For example, scientific researchers may be focused on whether the differences in lowered ...

A previous post ( Implementing Performance Measurement , November 12, 2005), explored what it takes to implement a court performance measurement system (CPMS). It reached the conclusion that even a well-conceived, well-designed CPMS will not necessarily get used unless it is woven into the fabric of a court’s management practices and processes. While such integration with strategic planning, performance-based budgeting and other formal management processes may be more demanding (more on this will follow in future postings), some relatively simple things can be done with powerful effects. Two simple techniques that facilitate the implementation of a CPMS are: (1) making the review of the court's core performance results a permanent agenda item on the court’s executive meetings and (2) assigning ownership of core measures to key managers. Consider the following ideal scenario. Your court has successfully built a CPMS that includes a set of eight core performance measures aligned with...