Step 3 – Creating Measurement Hierarchies
This is the seventh posting in a multi-part series exploring the Six-Step Process for Building an Effective Court Performance Measurement System (CPMS) first summarized in Made2Measure (M2M) in October 2005. See below for a listing of the parts in this series to date.
The third step of building a court performance measurement system (CPMS) is the construction of hierarchies (some call these “families” or “cascades”) that include a court’s core performance measures (identified in Step 2) and related subordinate measures (many of which may have been identified in Step 1). Step 3 involves three overlapping sub-steps: (3.1) breaking out (disaggregating) a court’s core measures; (3.2) identifying and defining subordinate measures that are related to core measures but are not necessarily breakouts of core measures; and (3.3) creating a hierarchy of performance measures for each of the core measure.
Beyond Core Measures
Measures are almost always proxies of outcomes or concepts – they are not the thing itself. A particular cholesterol level is not health. Case clearance is not court productivity or efficiency. To varying degrees, measures are removed and highly simplified version of the outcome of interest. We use these proxies in order to make it possible to measure the outcome easily, routinely and at a reasonable cost. The value of the proxy measure is that it is expected to correlate with the desired outcome. Because the correlation is rarely perfect, it is best that a single measure is not used on its own but together with a group of related measures in a hierarchy that provide a more valid, reliable, and rounded view of progress toward a strategic goal (e.g., access to justice).
Connecting the Dots
From a management perspective, the great value of measurement hierarchies is that they identify opportunities for collaboration. They put the court’s top management in much greater contact with every level of staff by defining the connection between high-level strategic goals and performance measures with lower-level departmental or divisional objectives and measures. Driving the CPMS to every level of the court signals to everyone what the drivers of success are and provides them with the concrete knowledge of how they contribute to that success.
Creating measurement hierarchies for each core measures ensures that important information at the level of court divisions, units, and programs is not masked by exclusive reliance on aggregated core measures. Measurement hierarchies identify opportunities for teamwork and collaboration. They put the court’s top management in much greater contact with every level of staff by defining the connection – a line of sight -- between high-level strategic goals and performance measures with lower-level departmental or divisional objectives and measures. Driving the CPMS to every level of the court signals to everyone what the drivers of success are and provides them with the concrete knowledge of how they contribute to that success.Measurement hierarchies in which lower-level subordinate measures “cascade” down from core measures help align the overall goals of the court with the goals and objectives of its divisions, units, and programs. They help make it clear to all court employees precisely how their actions help fulfill the court’s mission and strategic goals. For example, a clerk in a court’s jury commissioner’s office, will recognize that improvement in the percent of undeliverable mail – a lower-level subordinate performance measure -- drives juror qualification yield, which in turn drives the overall juror yield, which drives jury representativeness, a high-level core measure of fairness and equality.Or, for example, a core measure of cost per case (Measure 10 of the CourTools) aggregated across all case types may show no change over time. This is possible if cost per case of general civil cases rises sharply while that of all other cases drops slightly. Those responsible for improving case processing will want to see the breakouts (disaggregations) of cost per case for all the major case types rather than just the aggregate. The sharp rise in the cost of general civil cases may pinpoint problems or issues requiring immediate attention.
Technology Makes It Feasible
The advent of electronic databases and performance management and business intelligence software has all but eliminated the need for court managers to decide whether or not the problem of obscuring important information outweighs the value of using a vital few core performance measures. They need not choose between broad overview and rich detail. By organizing other important performance measures related to the core measure in a hierarchical format, court managers quickly will be able to drill down (descend) to lower levels of the hierarchy and access increasingly more detailed performance information. At the same time, court staff at lower levels will be able to ascend the hierarchy and align their day-to-day results with the court’s high-level core performance measures and broad strategic goals.
Breakouts by Individual Judicial Officers Not Recommended
The focus of CPMSs is organizational performance and not individual performance. Court performance measurement and individual performance evaluation differ in purpose, methodology, interpretation and use. Reporting breakouts by individual court employees, especially by elected judges as part of judicial performance evaluation, is likely to cause animosity and contention and carries the danger of destroying a CPMS. If a court wants to track breakouts by individual performance, such breakouts should not be made part of the measurement hierarchies of a CPMS and should be used for internal purposes only (e.g., motivating personnel).
Step 3.1 – Identify Simple Breakouts of Core Measures
The first sub- step is relatively straightforward. Each of the core measures identified in Step 2 is broken out (disaggregated) into its component data elements. For example, the first-level breakouts for a measure of jury representativeness may be the parities for each of three race/national origin groups. Other high-level core measures such as on-time case processing can be broken down by case type. The measure of employee opinion can be broken down into the various units or departments of a court.
Basic types of breakouts include: (1) organizational unit, department or project (e.g., civil division or drug court); (2) workload or difficulty of workload (e.g., case types); (3) geography (e.g., the different locations of a court); (4) court user category (e.g., attorneys, litigants, witnesses, jurors, title companies, and court observers); and (5) reason for outcome or rating (e.g., in surveys of citizens and court users, expressed overall dissatisfaction can be broken down into dissatisfaction with access, timeliness, courtesy and responsiveness of the court, and so forth). Though not focused on courts, Chapter 8, “Making Outcome Information Useful: Providing Indicator Breakouts,” in the Urban Institute’s Harry P. Hatry’s 1999 book, Performance Measurement: Getting Results (Washington, DC: Urban Institute Press, 103-105) is a helpful reference.
Measures derived from surveys are relatively easy to disaggregate by question, survey location, or respondent characteristics. For example, particular groups of respondents like victims of crime can be isolated from others to determine their unique perception of the survey as a whole or of specific questions (e.g., equal treatment or awareness of rights to assistance).
Step 3.2 – Identify Other Subordinate Measures
The requirement of the second sub- step is a little more subtle and more difficult. It involves another look at the inventory of performance measures produced in Step 1 -- all of the input, output, and outcome measures currently used by the court and its divisions, units and programs – and alignment of these measures in the hierarchies of core measures. For example, performance measures of a jurisdiction’s drug court (e.g., number of days between referral to drug court and admission to a treatment program) or a dependency court (e.g., average or median time form filing of the original petition to permanent child placement) that may not contribute to the aggregated core measure of on-time case processing should nonetheless be aligned in the hierarchy of this core measure. The alignment required of this sub-step helps to connect the performance of various court divisions and bridge the gap between day-to-day activities and strategic goals and mission.
It is important to remember that the relationship of the measures in a hierarchy may not simply be mathematical as is the case when the core measure is an average, a composite or index created by mathematically combining several related measures. The relationship can be logical as is the case when the subordinate measures are input and output measures, and the core measure is an outcome measure in the same key performance area, or when the factors measured by the subordinate measures are thought to be causes of the factors indicated by a core or superior measure. Or, the relationship simply can be based on a meaningful and appealing assemblage of performance measures that associated with a particular success factor like fairness, equality and integrity.
Step 3. 3 – Construct Measurement Hierarchies
The third and final sub-step of Step 3 is to combine and to order – cascade -- the core measure, the breakouts of the core measure, and the other subordinate measure in hierarchical fashion. Subordinate performance measures at lower levels in a hierarchy will be less comprehensive and aggregated, and more likely to be focused on inputs and outputs.
Hierarchies of measures keep the number of core performance measures to a vital few at the top of the hierarchy while at the same time allowing the measurement of different perspectives, levels or aspects of the core measures from top to bottom of the court. Developing measurement hierarchies helps ensure that the right levels of measures are collected, distributed, and used at the right level of the court. While a court manager and other court leaders may focus on the high-level core measure of on time case processing aggregated across case types and case processing events, department or unit managers may focus their attentions on lower levels of hierarchy of this core measure such as on-time case processing of various case types, the elapsed time from filing to judge assignment, assignment to first hearing, disposition to post-disposition event, and so forth. Effective measurement hierarchies ensure that everyone in a court can gain a solid understanding of the subordinate measures and their objectives that make up the high-level core measures.
Next in this series: Step 4. Testing and Demonstrating the Measures.
Previous postings in this series:
- Introduction to the Six-Step Process for the Design of an Effective Performance Measurement System (Part 1), Made2Measure, June 6, 2006
- Introduction to the Six-Step Process (Part 2), Made2Measure, June 12, 2006
- Step 1 -- Assessing Currently Used Performance Measures, Made2Measure, June 17, 2006
- Q & A: Can Step 1 Be Taken At the State-Level, Made2Measure, June 23, 2006
- Step 2 -- Identifying Desired Performance Measures (Part 1), Made2Measure, July 2, 2006
- Step 2 – Identifying Desired Performance Measures (Sub-Steps 2.1 and 2.2), Made2Meassure, July 10, 2006
For the latest posts and archives of Made2Measure click here.
Copyright CourtMetrics 2006. All rights reserved.