Aggregationists and Non-Aggregationists Unite!
Technology advances will dramatically change the way we do performance measurement and performance management (i.e., the use of performance information in managements) over the next decade in what I call the "third wave" of court performance measurement. Writing in a special issue of Public Administration Review, the Urban Institute's Harry Hatry predicts that over the next decade managers and their staff will have an enormous amount of performance data at their fingertips. They will be able to drill through various layers of highly aggregated data to disaggregated data, and slice and dice that data at will. Executives, managers and staff will have access to real-time cross-tabulations of outcomes for numerous variables. For example, service user satisfaction outcome data might be calculated and displayed for various combinations of variables including identity of user (e.g., litigant, witness, juror), case type that brought users to court, gender, race/ethnicity, age, and location.
The ready availability of this enormous amount of performance information changes the way we construct and use multiple performance measures. A case in point is the development of performance indices. Though not as robust in the United States, the development of governance, accountability and rule of law indices is a crowded field at the international level.
One way to achieve simplicity and yet monitor more than a few measures is to combine multiple measures into a single, simple index. Multiple measures in a "family" of metrics can be assigned weights according to their importance and combined in an index that is an aggregate statistic. Indices are attractive because they make complex information understandable and comparable across time and other places (e.g., courts, divisions, and locations).
The approach of aggregating individual measures to produce a high-level index – and the rating, ranking, shaming and praising that are brought into service by such an index – have sparked a debate pitting those who champion high-level indices against those that dislike them. On one side are the “aggregationists” who find indices attractive for the reasons noted above. On the other side are the “non-aggregationsists” who see risks that indices mask important differences, hide problems of data quality, decrease precision, amplify measurement error, and increase misinterpretation. They see no meaning in an index and find it useless for purposes of reform. (The characterization of this debate as aggregationsist versus non-aggregationsist comes from a July 2008 report to the World justice Project, Developing Indicators to Measure the Rule of Law: A Global Approach, by the Vera Institute who itsellf has weighed in with 60 rule of law indicators.)
Performance dashboards and business intelligence render this debate moot. There's no need to choose sides. Aggregationists and non-aggregationists can live in the same analytical space at the same time. They can move rapidly from a high-level index and drill down to highly disaggregated information, and back again via a different path at will. Performance dashboards are becoming the preferred way organizations – including an increasing number of justice sector institutions -- monitor, analyze, and manage their performance. They are the “face” of business intelligence designed to deliver the right information to the right people at the right time.
© Copyright CourtMetrics and the National Center for State Courts 2011. All rights reserved.
The ready availability of this enormous amount of performance information changes the way we construct and use multiple performance measures. A case in point is the development of performance indices. Though not as robust in the United States, the development of governance, accountability and rule of law indices is a crowded field at the international level.
One way to achieve simplicity and yet monitor more than a few measures is to combine multiple measures into a single, simple index. Multiple measures in a "family" of metrics can be assigned weights according to their importance and combined in an index that is an aggregate statistic. Indices are attractive because they make complex information understandable and comparable across time and other places (e.g., courts, divisions, and locations).
The approach of aggregating individual measures to produce a high-level index – and the rating, ranking, shaming and praising that are brought into service by such an index – have sparked a debate pitting those who champion high-level indices against those that dislike them. On one side are the “aggregationists” who find indices attractive for the reasons noted above. On the other side are the “non-aggregationsists” who see risks that indices mask important differences, hide problems of data quality, decrease precision, amplify measurement error, and increase misinterpretation. They see no meaning in an index and find it useless for purposes of reform. (The characterization of this debate as aggregationsist versus non-aggregationsist comes from a July 2008 report to the World justice Project, Developing Indicators to Measure the Rule of Law: A Global Approach, by the Vera Institute who itsellf has weighed in with 60 rule of law indicators.)
Performance dashboards and business intelligence render this debate moot. There's no need to choose sides. Aggregationists and non-aggregationists can live in the same analytical space at the same time. They can move rapidly from a high-level index and drill down to highly disaggregated information, and back again via a different path at will. Performance dashboards are becoming the preferred way organizations – including an increasing number of justice sector institutions -- monitor, analyze, and manage their performance. They are the “face” of business intelligence designed to deliver the right information to the right people at the right time.
© Copyright CourtMetrics and the National Center for State Courts 2011. All rights reserved.