Forget “Statistically Significant”

In his 1996 book, Keeping Score: Using the Right Metrics to Drive World-Class Performance (Quality Resources, a Division of Kraus Productivity Organization Ltd.), Mark Graham Brown delivered a practical common-sense guide on how to develop and use performance measures as tools for world-class performance. In his latest book, Get It, Set It, Move It, Prove It: 60 Ways to Get Results in Your Organization (Productivity Press, 2004), he puts a sharper point on some the issues he raised in his earlier book.

Forget “statistically significant” when assessing organizational performance, he writes in Chapter 49. Distinguishing between organizational performance measurement and science, he states that the concept of statistical significance is critically important for science to rule out that the differences between the experimental and control groups are due to the independent variable and not to chance. For example, scientific researchers may be focused on whether the differences in lowered cholesterol levels between an experimental group who received a new cholesterol drug and a control group who got a placebo are statistically different from chance occurrence. Managers of organizations are less concerned with doing research and proving things. Instead, they are concerned mostly with good performance.

In a previous post (see The Difference Between Performance Measurement and Research, October 7, 2005), I wrote that the differences between performance measurement and scientific research are like the differences between the process and the data interpretation rules you use to monitor your “driving” performance as you check the gauges on your car dashboard and that of your state department of motor vehicles (DMV) commissioning a scientific study of cell phone use and driving patterns throughout your state. The DMV, of course, would be concerned about whether accidents involving cell phone users versus non-users are statistically different from chance alone. But would you be concerned about whether your check of your gas gauge or speedometer revealed a statistically significant difference in the level or speed compared to that of your previous check?

Statistical standards should not be set aside. The point is that allowances should be made for the context in which they are applied. And, in some contexts, they should be relaxed or not used at all.

Like what we pay attention to while driving our cars, court managers are interested in performance trends. What is the current performance level compared to established upper and lower “controls” (e.g., performance targets, objectives, benchmarks and tolerance levels)? What does performance look like over time? Is it better, worse or flat? How much variability is there?

When assessing various strategies, programs or services, it is important to show that they produce good, practical results, whether they are statistically significant or not. A new service (e.g., a “greeter” at the front door of the courthouse) may be associated with a statistically significant performance improvement (e.g., a 3 percent increase in court user satisfaction) but a court manager still needs to determine whether the effort is worth the cost.

Performance measurement is the process of measuring a court’s accomplishments (outcomes), work and service levels (output), and its resources (inputs). It is done on a regular and continuous basis. Scientific research, on the other hand, is done to prove things, to discover factual truth, to test models and to develop theories to increase our knowledge and understanding about human behavior and social phenomena. Performance measurement and scientific research are vastly different in their purposes, functions, uses, the way they are funded and structured, and in their data interpretation rules.

The concept of statistical significance is just not helpful in tracking and evaluating organizational performance. As Mark Graham Brown points out, statistical significance is for scientists, not for business managers.
--------------------
For most recent posts and archives of Made2Measure click here
--------------------
Copyright CourtMetrics 2005. All rights reserved.

Popular posts from this blog

Q & A: Outcome vs. Measure vs. Target vs. Standard

A Logic Model of Performance Inputs, Outputs and Outcomes

Top 10 Reasons for Performance Measurement