Introduction to the Six-Step Process for the Design of an Effective Performance Measurement System (Part 1)

First in a multi-part series exploring the Six-Step Process for Building an Effective Court Performance Measurement System (CPMS) summarized in Made2Measure (M2M) in October last year.

The choice and use of appropriate performance measures or indicators of court performance should be the result of a process undertaken in each court or court system. Measures and indicators are not chosen and used at random. There are two key requirements for designing an effective court performance measurement system (CPMS):

--- Identifying, designing and developing those performance measures that will actually help to achieve desired results.

--- Ensuring that the performance measures are available to the right people, at the right time, in the right place and in the right way.

Over the last seven years, my colleagues and I have developed a design process of six overlapping steps and sub-steps (tasks) by benchmarking the performance measurement design processes used successfully in the private sector, in public and non-profit organizations, and by extensive field study in dozens of courts throughout the world. Here are the six steps of that process:

Step 1. Inventory and Assessment. Assessing an inventory of performance measures currently used by the court or court system.
Step 2. The Right Measures. Identifying and defining the core performance measures needed and desired to help achieve goals.
Step 3. Measurement Hierarchies. Developing hierarchies or families of performance measures.
Step 4. Testing. Testing and/or demonstrating the measures.
Step 5. Data Collection and Distribution. Creating data collection and distribution methods that ensure timeliness and utility – getting the measures to those who would benefit by their use when they need them.
Step 6. Data Display. Building useful performance measurement data displays – getting the measures to users in the most useful way.

Each of the building steps addresses critical questions: How does the court currently measure its performance? How are the current measures distributed across inputs, outputs, outcomes, key success areas, perspectives, and core performance areas? What required or desired performance information currently is not available to the court? What specific performance measures would provide that information? How does the court select a vital few, instead of a trivial many, performance measures? How should the court go about developing the desired performance measures? To what extent should the selected measures be tried and demonstrated before implementation? When and how should the performance data be collected? To whom and how often should it be distributed? In what format and on what schedule should the performance information be conveyed and displayed?

The first four steps relate to the first key requirement of building an effective CPMS – getting the right measures – and the last two steps relate to the second – getting the measures into the hands of the right people at the right time in the right way. Each of the six steps and related tasks will be described in subsequent parts in this series of blog postings.

Why Do We Need a Process?

Why do we need such a deliberate process to find the right measures and get them into the hands of the right people? Why make it so complicated?

Finding the right measures and getting them into the hands of those that need them implicates all kinds of complex and complicated considerations that one comes across in strategic planning, values clarification, priority setting, and other management processes. The Six-Step Process is intended to make things easier.

A deliberate design process introduces discipline, conceptual clarity and method to the myriad conceptual, methodological, analytic issues associated with the building a CPMS. It provides a practical nonacademic context and opportunity to explain and to incorporate principles that should inform the choice of measures. For example, the assessment of a court’s inventory of current measures in Step 1 allows members of the court’s design team to explore the meaning of a balanced scorecard of performance measures and the preference for outcome measures and other design principles or guidelines in the context of measures and an operational environment familiar to them. In addition, Step 1 will inevitably dispel the usually inaccurate notion that “we do not measure anything.”

A deliberate design process also highlights the need to develop a key court management system – a court performance measurement system (CPMS). A CPMS an ordered and comprehensive assembly of parts -- interrelated data, principles, standards, methods, processes and procedures -- forming a unitary whole. All the elements of a CPMS -- the performance measures, data collection methods, analysis and interpretation, and information distribution and display -- operate toward the common purpose of running a court effectively and efficiently.

Steps Need Not Be Taken In Sequence

Although the six steps are laid out in a linear sequence, actually taking them is not. A court’s design team usually may rethink the results of some steps several times and even may return to earlier steps before reaching a consensus. Also, a court may chose to move through the steps faster with some measures than with others.

For example, the demonstration of a performance measure (e.g., cost per case) on a limited basis during its development and prior to full implementation (Step 4) may make it clear to the working group that the measure is not tracking what it’s supposed to (lacks validity) or that it does not seem to produce the same results under the same circumstances (lacks reliability). This may cause the working group to return to the previous step to redefine the measure. Or the working group may find that the calculation of cost per case needs to be adjusted for inflation for the measure to be valid. If the measure is unlikely to provide complete, accurate and consistent performance data, even if revamped, the court may need to step back and consider an alternative measure (Step 2).

Finally, the design process does not necessarily have to begin at the beginning. To generate enthusiasm for building a CPMS, it might even begin with the last step by envisioning the display of the CPMS that the end user sees. In brief, as soon as useful tasks are identified, they should be done, unless they might seriously jeopardize future actions and decisions.

History of the Six-Step Process

Beginning in 1999, the Six-Step Process was developed based on extensive study of private sector, public (non-court), and not-for-profit efforts for designing performance measurement systems. The seeds of the design principles of a balanced scorecard, hierarchy of measures, the logic model of inputs, outputs, and outcomes and other principles appeared in a chapter in the July 2000 National Justice Institute publication Measurement and Analysis of Crime and Justice. The Six-Step Process took shape in its present form in 2000 – 2001 as part of early work on performance measures with the SMAART Project of the 19th Judicial Circuit in Illinois and the Family Court of Delaware. It was first used formally in 2001 – 2002 as part of work in the Family Court of Delaware that led to the 2002 publication Quality Counts: A Manual of Family Court Performance Measurement. The Six-Step Process is outlined in the Introduction of this manual in a section titled “Building a Performance Measurement System.”

Between 2002 and the present, at least 11 court systems have used the Six-Step Process as part of initiatives to build CPMSs: the Seattle Municipal Court, the Mesa Municipal Court, the State of North Carolina, the State of Washington, the Provincial Court of British Columbia, the Maricopa County Superior Court and the Yuma County Superior Court in Arizona, the Oregon Court of Appeals, the District Court Clerk’s Office for Collin County, Texas, the Ministry of the Attorney General of British Columbia (which includes court services), and the Yugoslav Republic of Macedonia. It has been featured in numerous publications and presentations. In a joint resolution, In Support of Measuring Court Performance, adopted on August 3, 2005, state court leaders in the Conference of Chief Justices (CCJ) and the Conference of State Court Administrators (COSCA) urged state courts to develop and test a balanced set of court performance measures using the models of the CourTools and the Six-Step Process.

In the next posting in this series, we will continue this introduction to the Six-Step Process by distinguishing design and development from the other necessary phases of a performance measurement initiative – initiating and planning (getting started), implementation and institutionalization.

For the latest posts and archives of Made2Measure click here.

Copyright CourtMetrics 2006. All rights reserved.

Comments

Post a Comment

Popular posts from this blog

Top 10 Reasons for Performance Measurement

There is a growing demand of “show me the data” from citizens and taxpayers, legislators, executive agencies, oversight boards and communities – all of whom exchange money and other support (e.g., trust and confidence) for services -- that courts can ill afford to ignore. Courts need to measure their performance to meet the demand for public accountability. But the benefits of court performance extend beyond accountability including improved prediction, better understanding and control, influence, a sharpened focus on what is important, and discovery of better practices. Objective performance data is central to good management practices like strategic planning, performance-based budgeting, Balanced Scorecards, Total Quality Management (TQM), and Six Sigma. Performance measures allow court leaders and managers, policymakers, legislators and the public to evaluate court programs’ inputs (the resources allocated), outputs (direct results of program or service activities) and outcomes (bro
Read more

Q & A: Outcome vs. Measure vs. Target vs. Standard

Q: Those of who write and speak about court performance measurement rely on their share of jargon -- and it's often confusing. What are the differences among the terms “outcome,” “measure,” “target” and “standard”? A: Socrates said that the beginning of wisdom is definition. We’ll see. Outcomes are the benefits or changes for the intended beneficiaries of a court’s programs and services. Outcomes may relate to knowledge, skills, attitudes, values, behavior, condition or status of the program participants or recipients of services. Examples include litigants' satisfaction with a court's courtesy and responsiveness, success of probation, time to case disposition, clarity of orders, integrity of case files, percent of mediation agreements, enforcement of orders, and perceived fairness of proceedings, as well as percent of expected expenditures and percent of revenues received. Outcomes should not be confused with the activities of those who run the court. (See the October 1
Read more

A Logic Model of Performance Inputs, Outputs and Outcomes

Increasingly, the courts' stakeholders -- justice system partners, the general public, litigants, jurors, witnesses, and even the courts’ own court employees -- demand clear evidence that the resources courts expend actually produce benefits for people. Looking for this evidence starts early in the development of a court performance measurement system (CPMS) (see Step 2, Identifying and Defining the Performance Measures Needed and Desired to Help Achieve Goals, October 15, 2005 posting, “Six-Step Process for Building an Effective Court Performance Measurement System”). For every strategic initiative, program or specific strategy, court managers should develop a simple logic model that describes how the initiative, program or strategy theoretically works to achieve the desired results from program inputs through end outcomes. Critical thinking and discussion of the logic of how a proposed program or strategy may bring about results that benefit citizens and the community can reap g
Read more