Forget “Statistically Significant”

In his 1996 book, Keeping Score: Using the Right Metrics to Drive World-Class Performance (Quality Resources, a Division of Kraus Productivity Organization Ltd.), Mark Graham Brown delivered a practical common-sense guide on how to develop and use performance measures as tools for world-class performance. In his latest book, Get It, Set It, Move It, Prove It: 60 Ways to Get Results in Your Organization (Productivity Press, 2004), he puts a sharper point on some the issues he raised in his earlier book.

Forget “statistically significant” when assessing organizational performance, he writes in Chapter 49. Distinguishing between organizational performance measurement and science, he states that the concept of statistical significance is critically important for science to rule out that the differences between the experimental and control groups are due to the independent variable and not to chance. For example, scientific researchers may be focused on whether the differences in lowered cholesterol levels between an experimental group who received a new cholesterol drug and a control group who got a placebo are statistically different from chance occurrence. Managers of organizations are less concerned with doing research and proving things. Instead, they are concerned mostly with good performance.

In a previous post (see The Difference Between Performance Measurement and Research, October 7, 2005), I wrote that the differences between performance measurement and scientific research are like the differences between the process and the data interpretation rules you use to monitor your “driving” performance as you check the gauges on your car dashboard and that of your state department of motor vehicles (DMV) commissioning a scientific study of cell phone use and driving patterns throughout your state. The DMV, of course, would be concerned about whether accidents involving cell phone users versus non-users are statistically different from chance alone. But would you be concerned about whether your check of your gas gauge or speedometer revealed a statistically significant difference in the level or speed compared to that of your previous check?

Statistical standards should not be set aside. The point is that allowances should be made for the context in which they are applied. And, in some contexts, they should be relaxed or not used at all.

Like what we pay attention to while driving our cars, court managers are interested in performance trends. What is the current performance level compared to established upper and lower “controls” (e.g., performance targets, objectives, benchmarks and tolerance levels)? What does performance look like over time? Is it better, worse or flat? How much variability is there?

When assessing various strategies, programs or services, it is important to show that they produce good, practical results, whether they are statistically significant or not. A new service (e.g., a “greeter” at the front door of the courthouse) may be associated with a statistically significant performance improvement (e.g., a 3 percent increase in court user satisfaction) but a court manager still needs to determine whether the effort is worth the cost.

Performance measurement is the process of measuring a court’s accomplishments (outcomes), work and service levels (output), and its resources (inputs). It is done on a regular and continuous basis. Scientific research, on the other hand, is done to prove things, to discover factual truth, to test models and to develop theories to increase our knowledge and understanding about human behavior and social phenomena. Performance measurement and scientific research are vastly different in their purposes, functions, uses, the way they are funded and structured, and in their data interpretation rules.

The concept of statistical significance is just not helpful in tracking and evaluating organizational performance. As Mark Graham Brown points out, statistical significance is for scientists, not for business managers.
--------------------
For most recent posts and archives of Made2Measure click here
--------------------
Copyright CourtMetrics 2005. All rights reserved.

Comments

Post a Comment

Popular posts from this blog

Top 10 Reasons for Performance Measurement

There is a growing demand of “show me the data” from citizens and taxpayers, legislators, executive agencies, oversight boards and communities – all of whom exchange money and other support (e.g., trust and confidence) for services -- that courts can ill afford to ignore. Courts need to measure their performance to meet the demand for public accountability. But the benefits of court performance extend beyond accountability including improved prediction, better understanding and control, influence, a sharpened focus on what is important, and discovery of better practices. Objective performance data is central to good management practices like strategic planning, performance-based budgeting, Balanced Scorecards, Total Quality Management (TQM), and Six Sigma. Performance measures allow court leaders and managers, policymakers, legislators and the public to evaluate court programs’ inputs (the resources allocated), outputs (direct results of program or service activities) and outcomes (bro
Read more

The “What Ifs” Along the Road in the Quest of a Justice Index

What if we could draw an indicator of the “health” of the justice systems throughout the United States – a Justice Index - from readily available public data that is impartial, reliable, objective and comparable, let’s say the amount of time that criminal defendants spend in custody without a trial, or even without any access to legal or justice services? Or the bail amounts imposed on misdemeanor defendants?   This kind of a measure would not be perfect – no measure is – but it would be something, better than nothing. It would address questions of fairness and efficient use of public resources. That is, a criminal defendant languishing in jail awaiting trial suffers, at the least, a delay in justice. He or she is also occupying a jail bed and spending taxpayers’ money. What if we then join this measure with another meaningful indicator of the health of our justice systems in counties and states, let’s say the degree to which jurisdictions prosecute domestic violence cases.
Read more

Q & A: Outcome vs. Measure vs. Target vs. Standard

Q: Those of who write and speak about court performance measurement rely on their share of jargon -- and it's often confusing. What are the differences among the terms “outcome,” “measure,” “target” and “standard”? A: Socrates said that the beginning of wisdom is definition. We’ll see. Outcomes are the benefits or changes for the intended beneficiaries of a court’s programs and services. Outcomes may relate to knowledge, skills, attitudes, values, behavior, condition or status of the program participants or recipients of services. Examples include litigants' satisfaction with a court's courtesy and responsiveness, success of probation, time to case disposition, clarity of orders, integrity of case files, percent of mediation agreements, enforcement of orders, and perceived fairness of proceedings, as well as percent of expected expenditures and percent of revenues received. Outcomes should not be confused with the activities of those who run the court. (See the October 1
Read more