Step 4 – Testing, Demonstrating and Developing Measures

This is the ninth posting in a multi-part series exploring the Six-Step Process for Building an Effective Court Performance Measurement System (CPMS) first summarized in Made2Measure (M2M) in October 2005. See below for a listing of the parts in this series to date.

We learn something by doing it. There is no other way. John Holt

Once steps have been taken to identify and define the desired performance measures (Steps 1 and 2), and to construct hierarchical relationships among them (Step 3), simply issuing an edict to "go forth and measure" is likely to invite failure. It is one thing to identify and define performance measures, and quite another to demonstrate that an organization can actually take the measures given its current operating structures, information and management systems and processes.

Court staff should be provided with detailed instructions, adequate resources, as well as encouragement, to test or demonstrate the designed measures. Procedures for planning and preparations for the measurement, data sources, data collection methods, analysis, and distribution and use of the measures should be carefully prescribed. The measures should then be field tested, demonstrated, and modified as necessary.

Step 4 involves three overlapping sub-steps: (4.1) planning and developing the methods to be used to test or demonstrate the measures; (4.2) adequately testing or demonstrating the measures; and (4.3) developing and refining the measures based on their test and demonstration. Step 4.1 is likely to overlap considerably with Step 2.4, Defining Performance Measures in Operational Terms, which results in an operational definition of the identified measures in terms of its rationale, necessary data elements, data sources and methods -- concrete and specific processes or operations -- required to obtain the measurements. The difference between these sub-steps is the difference between written prescription and reality testing, between planning and implementation. Much will be learned by doing in Step 4.

Step 4.1 – Planning Tests and Demonstrations of Performance Measures

From what sources will the performance data for the measures be drawn – automated case management information systems, the opinion of court users, a sample of case files, court financial records, or personnel files? Are these sources of data readily accessible? How much data will be collected and how often? Who will collect the data? How will the data be collected – computer query, questionnaire administered to individual respondents, or sampling of court records? What databases will the performance data populate? What is the design of the databases for the measurement data? How will the data be entered into those databases? Does the measure produce the same result every time given the same circumstances and conditions (reliability)? Does the measure track what is intended and is not contaminated by factors that might make conclusions about results invalid (validity)?

As noted above, these questions already will have been addressed in Step 2.4, Defining Performance Measures in Operational Terms. Step 4.1 introduces the necessity of specifying the measurement methods in terms of detailed instructions for testing and demonstration.

The descriptions of the ten performance measures, including the calculation templates made available through the “Community of Practice,” of the National Center for State Courts’ CourTools , as well as the Trial Court Performance Standards and Measurement System Implementation Manual and, and the Family Court of Delaware’s Quality Counts: A Manual of Family Court Performance Measurement and Family Court Performance Standards and Measures, are all models for this task. For example, each of the ten measures of the CourTools is described in several pages of easy-to-understand language – including an overview that describes the measure’s purpose and how it aligns with the standard and performance area, planning and preparations for taking the measure, data collection procedures and forms, data analysis and reporting, and references to other resources – to allow court practitioners to arrange the specific performance measurement with little no outside assistance.

Step 4. 2 – Tests and Demonstrations

Step 4.2 is the most demanding and time consuming. It requires that each of the measures described in Step 4.1 to be tested and demonstrated in practice. The testing and demonstration need not be a full-fledged implementation of the measures. It should, however, approximate the conditions that will exist during full implementation of the measures. It should be sufficient to test the feasibility, validity, and reliability and to pinpoint measurement problems that may need to be addressed before proceeding with the building of the CPMS.

Testing and demonstration might cover only outcomes and outputs that are new to the court or require substantial modifications of existing data collection procedures. It might cover only some of the activities of the performance, only some of time periods, or only some of the locations to be measured. For example, the demonstration of a survey of court users for the measure citizen/court user opinion might be tested and demonstrated in only one of a court’s several locations and for only part of a day with a sample of respondents. Data could be entered into a temporary database and analyzed manually for the purposes of the limited testing and demonstration.

Step 4.3 – Developing and Refining the Measures

The adage, “Nothing ever works the first time,” may be the rule in performance measurement. Adequately testing and demonstrating the measures allows court staff to address problems such as inadequately conceived or defined measures, overlooked outputs and outcomes, cumbersome procedures, and difficulties with data entry, data analysis, interpretation and reporting, before full implementation of the CPMS.

A limited demonstration of a measure may uncover simple yet significant limitations or bugs. For example, in its initial demonstration of an employee opinion measure, the Family Court of the State of Delaware discovered that the available choices for court department or unit on its questionnaire did not jibe with the actual departments and units of the court. The developers of the questionnaire quite reasonably relied on the formal “official” organizational chart of the court to identify the departments and units. Unfortunately, not until the pilot test of the measure, did they discover that the questionnaire did not reflect the current operating departments and units of the court. Consequently, the critical breakouts of employee opinion by court department and unit were rendered meaningless. Similar bugs are likely to be identified in initial demonstrations of measures, especially measures that are new to a court or require new data collection methods.

Planning for the Future

Because Step 4 moves the design process from conceptualization to practice, it affords opportunities for long-range planning of information and management systems to support performance management. The design of a performance measurement system will serve to inform and shape other systems and processes. For example, Measure 6. Reliability and Integrity of Case Files of the National Center for Courts’ CourTools requires the sampling of case files for inspection. For performance measurement and not necessarily research or program evaluation, ideally, the sampling for inspection of case files should be built into an automated case filing system whereby randomly selected files are routinely marked for inspection, data collection screens are produced, data entry forms are automatically populated with the case file number, and automated analyses are performed and delivered to users. Doing this integrates performance measurement into the fabric of a court’s operations and management. Is such a system to support the measure in the cards? Can it be approximated? What resources are available to make it happen including all aspects of relevant technology – hardware, software, applications, interfaces, and resources available for operation, maintenance, and enhancement?

Next in this series: Step 5. Data Collection and Distribution.

Previous postings in this series:
  1. Introduction to the Six-Step Process for the Design of an Effective Performance Measurement System (Part 1), Made2Measure, June 6, 2006
  2. Introduction to the Six-Step Process (Part 2), Made2Measure, June 12, 2006
  3. Step 1 -- Assessing Currently Used Performance Measures, Made2Measure, June 17, 2006
  4. Q & A: Can Step 1 Be Taken At the State-Level, Made2Measure, June 23, 2006
  5. Step 2 -- Identifying Desired Performance Measures (Part 1), Made2Measure, July 2, 2006
  6. Step 2 – Identifying Desired Performance Measures (Steps 2.1 and 2.2), Made2Meassure, July 10, 2006
  7. Step 2 – Identifying Desired Performance Measures (Steps 2.3 and 2.4), Made2Measure, July 15, 2006
  8. Step 3 – Creating Measurement Hierarchies, Made2Measure, July 27, 2006

For the latest posts and archives of Made2Measure click here.

Copyright CourtMetrics 2006. All rights reserved.

Comments

Post a Comment

Popular posts from this blog

Top 10 Reasons for Performance Measurement

There is a growing demand of “show me the data” from citizens and taxpayers, legislators, executive agencies, oversight boards and communities – all of whom exchange money and other support (e.g., trust and confidence) for services -- that courts can ill afford to ignore. Courts need to measure their performance to meet the demand for public accountability. But the benefits of court performance extend beyond accountability including improved prediction, better understanding and control, influence, a sharpened focus on what is important, and discovery of better practices. Objective performance data is central to good management practices like strategic planning, performance-based budgeting, Balanced Scorecards, Total Quality Management (TQM), and Six Sigma. Performance measures allow court leaders and managers, policymakers, legislators and the public to evaluate court programs’ inputs (the resources allocated), outputs (direct results of program or service activities) and outcomes (bro
Read more

Q & A: Outcome vs. Measure vs. Target vs. Standard

Q: Those of who write and speak about court performance measurement rely on their share of jargon -- and it's often confusing. What are the differences among the terms “outcome,” “measure,” “target” and “standard”? A: Socrates said that the beginning of wisdom is definition. We’ll see. Outcomes are the benefits or changes for the intended beneficiaries of a court’s programs and services. Outcomes may relate to knowledge, skills, attitudes, values, behavior, condition or status of the program participants or recipients of services. Examples include litigants' satisfaction with a court's courtesy and responsiveness, success of probation, time to case disposition, clarity of orders, integrity of case files, percent of mediation agreements, enforcement of orders, and perceived fairness of proceedings, as well as percent of expected expenditures and percent of revenues received. Outcomes should not be confused with the activities of those who run the court. (See the October 1
Read more

A Logic Model of Performance Inputs, Outputs and Outcomes

Increasingly, the courts' stakeholders -- justice system partners, the general public, litigants, jurors, witnesses, and even the courts’ own court employees -- demand clear evidence that the resources courts expend actually produce benefits for people. Looking for this evidence starts early in the development of a court performance measurement system (CPMS) (see Step 2, Identifying and Defining the Performance Measures Needed and Desired to Help Achieve Goals, October 15, 2005 posting, “Six-Step Process for Building an Effective Court Performance Measurement System”). For every strategic initiative, program or specific strategy, court managers should develop a simple logic model that describes how the initiative, program or strategy theoretically works to achieve the desired results from program inputs through end outcomes. Critical thinking and discussion of the logic of how a proposed program or strategy may bring about results that benefit citizens and the community can reap g
Read more