Step 5 – Developing Data Collection and Delivery Timeframes
This is the tenth part of a multi-part series exploring the Six-Step Process for Building an Effective Court Performance Measurement System (CPMS) first summarized in Made2Measure (M2M) in October 2005. See below for a listing of the parts in this series to date.
A CPMS from Paper to Reality
The first four steps of the design process aim at getting the right performance measures identified, designed, developed and tested. The result is verified methods – blueprints for the measures of a court performance measurement system (CPMS). The CPMS has no reality yet, except on paper. Step 5 and Step 6, which are closely related, ensure that the measures get to the right people, at the right time and in the right way. They are the steps that take the CPMS to reality.
In first approaching these last two steps, CPMS design teams often come to the realization, “Hey, we’re really going to do this! We’re actually building a system of regular and continuous collection, analysis, synthesis, delivery and display of measures of the work and accomplishment of the court as an organization.”
Asking the Right Questions
How frequently should the data for each measure be collected and delivered? Who needs the performance data provided by the core measures and subordinate measures in the allied hierarchies? How will they use that data? For what strategic, tactical and operational purposes? When do they need it and how often will they use it? How do the delivery timeframes fit the key management processes of the court – budgeting, resource allocation, process improvement and communication? Undoubtedly, these questions will be raised during the preceding four steps. In Step 5, they are given focused attention.
Few courts have the capacity to take performance measures before they deliberately build that capacity. Design teams often approach Step 5 having already made false assumptions about data collection and delivery timeframes for selected measures. These assumptions may be formed solely on reactions to the question, “How often can we collect the data required by the measure given our current capacities?” instead of the question, “How often should we take this measure given our needs?” The data collection and delivery timeframes are determined by assumed difficulties with the current availability and access to data, and the human effort required to acquire, analyze, and enter the data. This is a mistake because it limits a performance measurement initiative to the status quo. This is a mistake because it may overlook technology advances that have made performance data collection and display much easier (this is a topic addressed in Step 6).
An Example
Assume that a court selects Measure 6. Reliability and Integrity of Case Files of the National Center for Courts’ CourTools as a core performance measure in Step 2, as did the Office of the District Clerk in Collin County, Texas. The CourTools prescribes the random sampling of at least 50 “pending case files, closed-on-site files, and closed-off-site case files in each case type being evaluated” (as many as 750 case files assuming five case types) on a periodic (monthly or quarterly basis). The design team may reasonably assume that this sampling strategy would result in data collection that is too labor intensive – 10 minutes of inspection and data entry for each sample file X 750 files = 7,500 person-minutes or 125 person-hours -- and, therefore, decide that the measure can only be taken once year.
The team may come to this premature decision without consideration of how often managers and staff may want to use this measure for strategic, tactical and operational purposes, and without consideration of technology to assist in data collection and entry. But once the design team deliberately takes Step 5, it may decide that the court clerk’s office managers and staff, especially, need to see the results of this measure on a weekly or even daily basis. It therefore decides that the sampling for inspection of case files should be built into an automated case filing system whereby randomly selected files are routinely marked for inspection, data collection screens are produced, data entry forms are automatically populated with the case file number, and automated analyses are performed and delivered to users. Every one of 40 clerks – who each man “touch” and average of 100 files per day -- would be responsible for evaluating only one randomly selected case file tagged by the automated system, yielding 40 sampled files per day and 200 files every week. This method allows daily and weekly analysis of the measure and integrates performance measurement into the fabric of the court clerk’s routine operations and management.
Two Simple Sub-Steps
The more timely the performance feedback, the more useful it is to court leaders, managers and staff. Data delayed are data denied.
Step 5 requires two discrete sub-steps. The first is to determine the ideal timeframes for availability and use of the performance data for each core performance measure. The second is to consider feasibility and costs of data collection and delivery and to adjust the ideal timeframes accordingly. The result of the step is a timeframe for collection and delivery of each core and subordinate measure in all of the measurement hierarchies constructed in Step 3.
Real or Near-Real Time Data Access
For private companies and some public institutions, the right timeframe for data collection and distribution of performance reports on finances, inventories, and customers increasingly means near real time, i.e., close to the time of performance without what’s called “propagation delay.” Few of us would consider it very helpful to receive information about the speeds we drove our car from a monthly report issued by our state department of motor vehicles instead of from the speedometer on our dashboard. We are accustomed to receiving speed data in real-time. Such real-time performance data are crucial for airlines, where any glitch is multiplied by 3,000 daily flights or more. Paul Cabana reported in Fast Company that United Airlines began using software for "day of operations" information that saved it $43 million in 2003.
It makes little sense to take and deliver solid, well-founded court performance measures in timeframes that render the information all but useless to court leaders and managers. But this is precisely what happens in many courts. Far too often, performance data collection and distribution are accomplished by and for others (e.g., state administrative offices of the courts, county government, funding agencies, and IT departments) in timeframes that are not helpful or, worse, disruptive to courts. For example, a few copies of a report of the results of a court user opinion survey conducted with much fanfare by a private survey research firm, employed by the state administrative office of courts, are delivered unceremoniously to the court’s presiding judge many months after the survey was conducted. Court staff, who may have been recruited to help administer the survey with promises of access to survey results, may wonder “What ever happened to that survey?” Or a court’s IT department, bogged down by complex technology integration projects and inundated by ad hoc requests to query automated case management and financial systems, finally honors a court manager’s request for trend data on the cost of domestic violence cases, but only after repeated pleas over weeks or months.
Step 5.1 – Determining the Ideal Timeframes
Step 5.1 requires determining the ideal timeframes of data collection and performance information access for all potential users of each core measure and related subordinate measure in every measurement hierarchy. For example, court user citizen opinion survey data ideally might be collected on a quarterly basis at the same time in all court locations. The results of the survey -- including the core measure of the proportion of respondents satisfied with the court and breakouts (disaggregations) by question, and location, as well as income, gender, ethnicity, and other categories that identify the respondents – are made available on the court’s intranet website the day the survey is completed.
As noted above, it is important that the timeframes are first set as objectives to be achieved without regard to feasibility and practicality. The determination of timeframes should be informed but not determined by a court’s existing manual and automated information systems, and the estimated human effort required to acquire, analyze, and enter the data. “Our system does not capture that information,” is an often-heard refrain in response to questions about performance data not currently collected. Most courts are continually upgrading their current automated information systems or migrating to newer ones, and court managers often are resistant to the idea of imposing “out of scope” data extraction and reporting requirements on computer software vendors. Such reactions and attitudes reflect a view of performance measurement as incidental to existing management processes and procedures, a view that could easily compromise the building of an effective CPMS.
By casting the ideal timeframes as objectives to be achieved, Step 5.1 sets the proper stage for considerations of feasibility, costs and other factors focused on specific details (e.g., administration of a survey on a quarterly basis in all court locations) instead of generalities that may reflect nothing more than resistance to change. The operative question is not “if” but rather “when” and “how.” Temporary solutions (e.g., a limited manual data collection and delivery via static email attachments) can be developed that give way to ideal solutions (e.g., automated data generation in near real-time) in the future.
Step 5.2 – Adjusting the Timeframes
Step 5.2 requires adjusting the ideal timeframes based on practical considerations. As noted above, the adjustments might be temporary as a court acquires the necessary resources to accommodate the ideal timeframes.
How easily can existing manual or electronic information systems be adapted to accommodate the timeframes? Can the required data be pulled from automated information systems with little intervention by programmers? For example, a court’s automated case management system may provide the necessary data elements for a core measure of on-time case processing disaggregated by some case types only (e.g., criminal but not civil), or by all case type but not by income level or some other category identifying the litigants. Such data inaccessible by automated means, therefore, may need to be retrieved manually from a mainframe system and entered into a temporary database until such time as the automated case management system is modified. While parts of the core measure and breakouts may be available in real-time as cases are concluded, the data elements for some case types and for some the breakouts for on-time case processing like litigant income level may be collected and distributed only on a monthly or quarterly basis because of the effort an expense of manual data retrieval. As mentioned earlier, however, such partial solutions may only be temporary as the court plans for future enhancements.
Next in this series: Step 6. Dashboards, Scorecards and Data Displays.
Previous postings in this series:
A CPMS from Paper to Reality
The first four steps of the design process aim at getting the right performance measures identified, designed, developed and tested. The result is verified methods – blueprints for the measures of a court performance measurement system (CPMS). The CPMS has no reality yet, except on paper. Step 5 and Step 6, which are closely related, ensure that the measures get to the right people, at the right time and in the right way. They are the steps that take the CPMS to reality.
In first approaching these last two steps, CPMS design teams often come to the realization, “Hey, we’re really going to do this! We’re actually building a system of regular and continuous collection, analysis, synthesis, delivery and display of measures of the work and accomplishment of the court as an organization.”
Asking the Right Questions
How frequently should the data for each measure be collected and delivered? Who needs the performance data provided by the core measures and subordinate measures in the allied hierarchies? How will they use that data? For what strategic, tactical and operational purposes? When do they need it and how often will they use it? How do the delivery timeframes fit the key management processes of the court – budgeting, resource allocation, process improvement and communication? Undoubtedly, these questions will be raised during the preceding four steps. In Step 5, they are given focused attention.
Few courts have the capacity to take performance measures before they deliberately build that capacity. Design teams often approach Step 5 having already made false assumptions about data collection and delivery timeframes for selected measures. These assumptions may be formed solely on reactions to the question, “How often can we collect the data required by the measure given our current capacities?” instead of the question, “How often should we take this measure given our needs?” The data collection and delivery timeframes are determined by assumed difficulties with the current availability and access to data, and the human effort required to acquire, analyze, and enter the data. This is a mistake because it limits a performance measurement initiative to the status quo. This is a mistake because it may overlook technology advances that have made performance data collection and display much easier (this is a topic addressed in Step 6).
An Example
Assume that a court selects Measure 6. Reliability and Integrity of Case Files of the National Center for Courts’ CourTools as a core performance measure in Step 2, as did the Office of the District Clerk in Collin County, Texas. The CourTools prescribes the random sampling of at least 50 “pending case files, closed-on-site files, and closed-off-site case files in each case type being evaluated” (as many as 750 case files assuming five case types) on a periodic (monthly or quarterly basis). The design team may reasonably assume that this sampling strategy would result in data collection that is too labor intensive – 10 minutes of inspection and data entry for each sample file X 750 files = 7,500 person-minutes or 125 person-hours -- and, therefore, decide that the measure can only be taken once year.
The team may come to this premature decision without consideration of how often managers and staff may want to use this measure for strategic, tactical and operational purposes, and without consideration of technology to assist in data collection and entry. But once the design team deliberately takes Step 5, it may decide that the court clerk’s office managers and staff, especially, need to see the results of this measure on a weekly or even daily basis. It therefore decides that the sampling for inspection of case files should be built into an automated case filing system whereby randomly selected files are routinely marked for inspection, data collection screens are produced, data entry forms are automatically populated with the case file number, and automated analyses are performed and delivered to users. Every one of 40 clerks – who each man “touch” and average of 100 files per day -- would be responsible for evaluating only one randomly selected case file tagged by the automated system, yielding 40 sampled files per day and 200 files every week. This method allows daily and weekly analysis of the measure and integrates performance measurement into the fabric of the court clerk’s routine operations and management.
Two Simple Sub-Steps
The more timely the performance feedback, the more useful it is to court leaders, managers and staff. Data delayed are data denied.
Step 5 requires two discrete sub-steps. The first is to determine the ideal timeframes for availability and use of the performance data for each core performance measure. The second is to consider feasibility and costs of data collection and delivery and to adjust the ideal timeframes accordingly. The result of the step is a timeframe for collection and delivery of each core and subordinate measure in all of the measurement hierarchies constructed in Step 3.
Real or Near-Real Time Data Access
For private companies and some public institutions, the right timeframe for data collection and distribution of performance reports on finances, inventories, and customers increasingly means near real time, i.e., close to the time of performance without what’s called “propagation delay.” Few of us would consider it very helpful to receive information about the speeds we drove our car from a monthly report issued by our state department of motor vehicles instead of from the speedometer on our dashboard. We are accustomed to receiving speed data in real-time. Such real-time performance data are crucial for airlines, where any glitch is multiplied by 3,000 daily flights or more. Paul Cabana reported in Fast Company that United Airlines began using software for "day of operations" information that saved it $43 million in 2003.
It makes little sense to take and deliver solid, well-founded court performance measures in timeframes that render the information all but useless to court leaders and managers. But this is precisely what happens in many courts. Far too often, performance data collection and distribution are accomplished by and for others (e.g., state administrative offices of the courts, county government, funding agencies, and IT departments) in timeframes that are not helpful or, worse, disruptive to courts. For example, a few copies of a report of the results of a court user opinion survey conducted with much fanfare by a private survey research firm, employed by the state administrative office of courts, are delivered unceremoniously to the court’s presiding judge many months after the survey was conducted. Court staff, who may have been recruited to help administer the survey with promises of access to survey results, may wonder “What ever happened to that survey?” Or a court’s IT department, bogged down by complex technology integration projects and inundated by ad hoc requests to query automated case management and financial systems, finally honors a court manager’s request for trend data on the cost of domestic violence cases, but only after repeated pleas over weeks or months.
Step 5.1 – Determining the Ideal Timeframes
Step 5.1 requires determining the ideal timeframes of data collection and performance information access for all potential users of each core measure and related subordinate measure in every measurement hierarchy. For example, court user citizen opinion survey data ideally might be collected on a quarterly basis at the same time in all court locations. The results of the survey -- including the core measure of the proportion of respondents satisfied with the court and breakouts (disaggregations) by question, and location, as well as income, gender, ethnicity, and other categories that identify the respondents – are made available on the court’s intranet website the day the survey is completed.
As noted above, it is important that the timeframes are first set as objectives to be achieved without regard to feasibility and practicality. The determination of timeframes should be informed but not determined by a court’s existing manual and automated information systems, and the estimated human effort required to acquire, analyze, and enter the data. “Our system does not capture that information,” is an often-heard refrain in response to questions about performance data not currently collected. Most courts are continually upgrading their current automated information systems or migrating to newer ones, and court managers often are resistant to the idea of imposing “out of scope” data extraction and reporting requirements on computer software vendors. Such reactions and attitudes reflect a view of performance measurement as incidental to existing management processes and procedures, a view that could easily compromise the building of an effective CPMS.
By casting the ideal timeframes as objectives to be achieved, Step 5.1 sets the proper stage for considerations of feasibility, costs and other factors focused on specific details (e.g., administration of a survey on a quarterly basis in all court locations) instead of generalities that may reflect nothing more than resistance to change. The operative question is not “if” but rather “when” and “how.” Temporary solutions (e.g., a limited manual data collection and delivery via static email attachments) can be developed that give way to ideal solutions (e.g., automated data generation in near real-time) in the future.
Step 5.2 – Adjusting the Timeframes
Step 5.2 requires adjusting the ideal timeframes based on practical considerations. As noted above, the adjustments might be temporary as a court acquires the necessary resources to accommodate the ideal timeframes.
How easily can existing manual or electronic information systems be adapted to accommodate the timeframes? Can the required data be pulled from automated information systems with little intervention by programmers? For example, a court’s automated case management system may provide the necessary data elements for a core measure of on-time case processing disaggregated by some case types only (e.g., criminal but not civil), or by all case type but not by income level or some other category identifying the litigants. Such data inaccessible by automated means, therefore, may need to be retrieved manually from a mainframe system and entered into a temporary database until such time as the automated case management system is modified. While parts of the core measure and breakouts may be available in real-time as cases are concluded, the data elements for some case types and for some the breakouts for on-time case processing like litigant income level may be collected and distributed only on a monthly or quarterly basis because of the effort an expense of manual data retrieval. As mentioned earlier, however, such partial solutions may only be temporary as the court plans for future enhancements.
Next in this series: Step 6. Dashboards, Scorecards and Data Displays.
Previous postings in this series:
- Introduction to the Six-Step Process for the Design of an Effective Performance Measurement System (Part 1), Made2Measure, June 6, 2006
- Introduction to the Six-Step Process (Part 2), Made2Measure, June 12, 2006
- Step 1 -- Assessing Currently Used Performance Measures, Made2Measure, June 17, 2006
- Q & A: Can Step 1 Be Taken At the State-Level, Made2Measure, June 23, 2006
- Step 2 -- Identifying Desired Performance Measures (Part 1), Made2Measure, July 2, 2006
- Step 2 – Identifying Desired Performance Measures (Steps 2.1 and 2.2), Made2Meassure, July 10, 2006
- Step 2 – Identifying Desired Performance Measures (Steps 2.3 and 2.4), Made2Measure, July 15, 2006
- Step 3 – Creating Measurement Hierarchies, Made2Measure, July 27, 2006
- Step 4 – Testing, Demonstrating and Developing Measures, Made2Measure, Monday, August 07, 2006
For the latest posts and archives of Made2Measure click here.
Copyright CourtMetrics 2006. All rights reserved.
Comments
Post a Comment