Step 4 – Testing, Demonstrating and Developing Measures
We learn something by doing it. There is no other way. John Holt
Once steps have been taken to identify and define the desired performance measures (Steps 1 and 2), and to construct hierarchical relationships among them (Step 3), simply issuing an edict to "go forth and measure" is likely to invite failure. It is one thing to identify and define performance measures, and quite another to demonstrate that an organization can actually take the measures given its current operating structures, information and management systems and processes.
Court staff should be provided with detailed instructions, adequate resources, as well as encouragement, to test or demonstrate the designed measures. Procedures for planning and preparations for the measurement, data sources, data collection methods, analysis, and distribution and use of the measures should be carefully prescribed. The measures should then be field tested, demonstrated, and modified as necessary.
Step 4 involves three overlapping sub-steps: (4.1) planning and developing the methods to be used to test or demonstrate the measures; (4.2) adequately testing or demonstrating the measures; and (4.3) developing and refining the measures based on their test and demonstration. Step 4.1 is likely to overlap considerably with Step 2.4, Defining Performance Measures in Operational Terms, which results in an operational definition of the identified measures in terms of its rationale, necessary data elements, data sources and methods -- concrete and specific processes or operations -- required to obtain the measurements. The difference between these sub-steps is the difference between written prescription and reality testing, between planning and implementation. Much will be learned by doing in Step 4.
Step 4.1 – Planning Tests and Demonstrations of Performance Measures
From what sources will the performance data for the measures be drawn – automated case management information systems, the opinion of court users, a sample of case files, court financial records, or personnel files? Are these sources of data readily accessible? How much data will be collected and how often? Who will collect the data? How will the data be collected – computer query, questionnaire administered to individual respondents, or sampling of court records? What databases will the performance data populate? What is the design of the databases for the measurement data? How will the data be entered into those databases? Does the measure produce the same result every time given the same circumstances and conditions (reliability)? Does the measure track what is intended and is not contaminated by factors that might make conclusions about results invalid (validity)?
As noted above, these questions already will have been addressed in Step 2.4, Defining Performance Measures in Operational Terms. Step 4.1 introduces the necessity of specifying the measurement methods in terms of detailed instructions for testing and demonstration.
The descriptions of the ten performance measures, including the calculation templates made available through the “Community of Practice,” of the National Center for State Courts’ CourTools , as well as the Trial Court Performance Standards and Measurement System Implementation Manual and, and the Family Court of Delaware’s Quality Counts: A Manual of Family Court Performance Measurement and Family Court Performance Standards and Measures, are all models for this task. For example, each of the ten measures of the CourTools is described in several pages of easy-to-understand language – including an overview that describes the measure’s purpose and how it aligns with the standard and performance area, planning and preparations for taking the measure, data collection procedures and forms, data analysis and reporting, and references to other resources – to allow court practitioners to arrange the specific performance measurement with little no outside assistance.
Step 4. 2 – Tests and Demonstrations
Step 4.2 is the most demanding and time consuming. It requires that each of the measures described in Step 4.1 to be tested and demonstrated in practice. The testing and demonstration need not be a full-fledged implementation of the measures. It should, however, approximate the conditions that will exist during full implementation of the measures. It should be sufficient to test the feasibility, validity, and reliability and to pinpoint measurement problems that may need to be addressed before proceeding with the building of the CPMS.
Testing and demonstration might cover only outcomes and outputs that are new to the court or require substantial modifications of existing data collection procedures. It might cover only some of the activities of the performance, only some of time periods, or only some of the locations to be measured. For example, the demonstration of a survey of court users for the measure citizen/court user opinion might be tested and demonstrated in only one of a court’s several locations and for only part of a day with a sample of respondents. Data could be entered into a temporary database and analyzed manually for the purposes of the limited testing and demonstration.
Step 4.3 – Developing and Refining the Measures
The adage, “Nothing ever works the first time,” may be the rule in performance measurement. Adequately testing and demonstrating the measures allows court staff to address problems such as inadequately conceived or defined measures, overlooked outputs and outcomes, cumbersome procedures, and difficulties with data entry, data analysis, interpretation and reporting, before full implementation of the CPMS.
A limited demonstration of a measure may uncover simple yet significant limitations or bugs. For example, in its initial demonstration of an employee opinion measure, the Family Court of the State of Delaware discovered that the available choices for court department or unit on its questionnaire did not jibe with the actual departments and units of the court. The developers of the questionnaire quite reasonably relied on the formal “official” organizational chart of the court to identify the departments and units. Unfortunately, not until the pilot test of the measure, did they discover that the questionnaire did not reflect the current operating departments and units of the court. Consequently, the critical breakouts of employee opinion by court department and unit were rendered meaningless. Similar bugs are likely to be identified in initial demonstrations of measures, especially measures that are new to a court or require new data collection methods.
Planning for the Future
Because Step 4 moves the design process from conceptualization to practice, it affords opportunities for long-range planning of information and management systems to support performance management. The design of a performance measurement system will serve to inform and shape other systems and processes. For example, Measure 6. Reliability and Integrity of Case Files of the National Center for Courts’ CourTools requires the sampling of case files for inspection. For performance measurement and not necessarily research or program evaluation, ideally, the sampling for inspection of case files should be built into an automated case filing system whereby randomly selected files are routinely marked for inspection, data collection screens are produced, data entry forms are automatically populated with the case file number, and automated analyses are performed and delivered to users. Doing this integrates performance measurement into the fabric of a court’s operations and management. Is such a system to support the measure in the cards? Can it be approximated? What resources are available to make it happen including all aspects of relevant technology – hardware, software, applications, interfaces, and resources available for operation, maintenance, and enhancement?
Next in this series: Step 5. Data Collection and Distribution.
Previous postings in this series:
- Introduction to the Six-Step Process for the Design of an Effective Performance Measurement System (Part 1), Made2Measure, June 6, 2006
- Introduction to the Six-Step Process (Part 2), Made2Measure, June 12, 2006
- Step 1 -- Assessing Currently Used Performance Measures, Made2Measure, June 17, 2006
- Q & A: Can Step 1 Be Taken At the State-Level, Made2Measure, June 23, 2006
- Step 2 -- Identifying Desired Performance Measures (Part 1), Made2Measure, July 2, 2006
- Step 2 – Identifying Desired Performance Measures (Steps 2.1 and 2.2), Made2Meassure, July 10, 2006
- Step 2 – Identifying Desired Performance Measures (Steps 2.3 and 2.4), Made2Measure, July 15, 2006
- Step 3 – Creating Measurement Hierarchies, Made2Measure, July 27, 2006
For the latest posts and archives of Made2Measure click here.