Made2Measure

Outcome-Based Performance Measurement Is Becoming the Norm

2011-05-16T10:11:00.002-04:00

Hospitals across the country are being pushed and dragged into outcome-based performance measurement systems with increasing incentives for providing quality care and better outcomes. Proponents argue that such systems discourage unnecessary treatments, keep people healthier, and reduce overall costs.

Janet Adams reports in today’s Wall Street Journal ("Wellpoint Shakes Up Hospital Payments," May 16, 2011, B1) that Wellpoint Inc. is raising the stakes for 1,500 hospitals in 14 states serving 34 million people on its Blue Cross Blue Shield plans by cutting off annual payment increases to the hospitals if they fail to deliver quality patient care. Under the new system, Wellpoint will pay increases only to those hospitals that earned them by scoring high enough on outcome measures of treatment quality including how satisfied patients were with the care they received. The change is part of a broader trend in the health care industry toward a compensation approach for health care providers based on the quality of their care, instead of the number of tests performed or treatments provided.

Predictably, hospital insiders are pushing back. “We don’t have good outcome measures yet,” Chip Kahn, president of the Federation of American Hospitals, is quoted saying by the Wall Street Journal. “Many things will happen 30 days [after discharge] that have nothing to do with the hospital care.” Of course, this sounds like the all-too familiar retort for any kind of outcome-based performance measurement system (see “Ten Reasons Not to Measure Court Performance,” Made2Measure, November 19, 2008).

While the incentives and operating environment of hospitals and the health care industry differ from those facing courts and justice systems, I believe that court leaders and managers ignore this broad trend toward outcome-based performance measurement and accountability at their peril. (See also “Measurement of Hard Outcomes Trumps Process Measurement (Again)," Made2Measure, October 21, 2010)

An Index of the Moment

2011-03-10T13:04:00.001-05:00

My colleagues and I at the National Center for State Courts and elsewhere have been thinking long and hard about a “justice index” here in the United States, as well as a similar index, the Global Court Performance Index (GCPI), applicable at an international scale. Though not unschooled in ways and means of performance measurement and management in justice systems, we’ve been daunted by the challenges that the construction of such indexes present.

What should the conceptual framework of the index be – the rule of law, justice systems institutions, legal frameworks, the experiences of citizens with the justice system, and so forth? Should the indicators that comprise the index be drawn from actual performance data available from justices systems, such as case clearance rates and median time in criminal defendants spend in custody before trial, or should the data be drawn from secondary sources, or should new measures be identified and defined? How much weighting should be given to the various indicators that comprise the indexes? How much vetting of the indexes is needed before development and deployment is initiated?

The Shoe-Thrower’s Index

Just do it! Aim, ready, shoot! That’s the bracing but refreshing lesson that I draw from The Economist’s use an index of unrest in the Arab word that “aims to predict where the scent of jasmine may spread next” (The Economist, February 12, 2011; Economist.com/arabunrest). No doubt respectful of the challenges facing the construction of such an index, the Economist Intelligence Unit, the in-house research unit of The Economist that also publishes the Democracy Index, was not daunted. As a friend of mine said under similar circumstances, “I don’t know what’s good or bad at this stage, but I do know what nothing is!”

In its February 12 print edition, The Economist ran a table showing scores for 17 member countries of the Arab League using an index of unrest, or “Shoe-thrower's index.” Yemen came out on top with a score close to 90 on scale of 100, the most unstable. Libya, Egypt, Syria, and Iraq follow, each with scores close to 70. Qatar turned out the least unstable among the group with a score of a little more than 20.

Taking on the Challenges

The Economist intelligence Unit constructed the index by putting together a number of indicators that it felt were associated with instability – corruption, the age of the population, and so forth -- and ascribing different weights to them. (Where did these come from? What august groups vetted them?) It removed a few of the members of the Arab League from consideration, including the Palestinian territories, Sudan and Somalia for lack of data, and Comoros and Djibouti, which do not have a great deal in common with the rest of the group. Some factors were discounted because they “are hard to put a number on.” The data on unemployment, for example, were too spotty to be comparable. The calculation of the index score ascribed a weighting of 35% for the share of the population that is under 25; 15% for the number of years the government has been in power; 15% for both corruption and lack of democracy as measured by existing indices; 10% for GDP per person; 5% for an index of censorship and 5% for the absolute number of people younger than 25.

Firing at Target Instead of Constantly Aiming

No one among us would dare oppose careful thought and analysis, but I do worry about an affliction that might be called the “aim, aim, aim syndrome” wherein one remains ever short of solution. Though one would expect an international news magazine to move more quickly than non-profit and public think-tanks and research institutions, one can’t help but be impressed with The Economist’s willingness not just to take aim at the challenges of an index of stability but actually to fire at the target with a solution, knowing full well that it might not have hit the mark perfectly, and that it might have to aim and fire again. In fact, in its online version, The Economist noted that Jordan comes out surprisingly low on the index, and suggested that the weighting “might need to be tweaked.” It ureged Interested readers to make suggestions and assured them that refinements will be made.

© Copyright CourtMetrics and the National Center for State Courts 2011. All rights reserved.

Aggregationists and Non-Aggregationists Unite!

2011-03-02T15:37:00.002-05:00

Technology advances will dramatically change the way we do performance measurement and performance management (i.e., the use of performance information in managements) over the next decade in what I call the "third wave" of court performance measurement. Writing in a special issue of Public Administration Review, the Urban Institute's Harry Hatry predicts that over the next decade managers and their staff will have an enormous amount of performance data at their fingertips. They will be able to drill through various layers of highly aggregated data to disaggregated data, and slice and dice that data at will. Executives, managers and staff will have access to real-time cross-tabulations of outcomes for numerous variables. For example, service user satisfaction outcome data might be calculated and displayed for various combinations of variables including identity of user (e.g., litigant, witness, juror), case type that brought users to court, gender, race/ethnicity, age, and location.

The ready availability of this enormous amount of performance information changes the way we construct and use multiple performance measures. A case in point is the development of performance indices. Though not as robust in the United States, the development of governance, accountability and rule of law indices is a crowded field at the international level.

One way to achieve simplicity and yet monitor more than a few measures is to combine multiple measures into a single, simple index. Multiple measures in a "family" of metrics can be assigned weights according to their importance and combined in an index that is an aggregate statistic. Indices are attractive because they make complex information understandable and comparable across time and other places (e.g., courts, divisions, and locations).

The approach of aggregating individual measures to produce a high-level index – and the rating, ranking, shaming and praising that are brought into service by such an index – have sparked a debate pitting those who champion high-level indices against those that dislike them. On one side are the “aggregationists” who find indices attractive for the reasons noted above. On the other side are the “non-aggregationsists” who see risks that indices mask important differences, hide problems of data quality, decrease precision, amplify measurement error, and increase misinterpretation. They see no meaning in an index and find it useless for purposes of reform. (The characterization of this debate as aggregationsist versus non-aggregationsist comes from a July 2008 report to the World justice Project, Developing Indicators to Measure the Rule of Law: A Global Approach, by the Vera Institute who itsellf has weighed in with 60 rule of law indicators.)

Performance dashboards and business intelligence render this debate moot. There's no need to choose sides. Aggregationists and non-aggregationists can live in the same analytical space at the same time. They can move rapidly from a high-level index and drill down to highly disaggregated information, and back again via a different path at will. Performance dashboards are becoming the preferred way organizations – including an increasing number of justice sector institutions -- monitor, analyze, and manage their performance. They are the “face” of business intelligence designed to deliver the right information to the right people at the right time.

© Copyright CourtMetrics and the National Center for State Courts 2011. All rights reserved.

Measurement of Hard Outcomes Trumps Process Measurement (Again)

2010-10-21T11:05:00.000-04:00

Researchers this week cast doubt on the wisdom of looking first at processes and operations to drive quality improvement and urged Medicare, which provides health insurance to people 65 and older, to focus on “hard” outcomes such as rates of surgical deaths and serious complications. In a study published this Monday in Archives of Surgery, a medical journal, and reported this Tuesday by the Wall Street Journal (Thomas M. Burton, “Medicare Faulted on Surgery Evaluation,” October 19, 2010, A6), researchers at the University of Michigan said a better way to lower rates of death and serious complications is to focus on actual outcomes like death rates and to publicize those rates for all hospitals. This focus on actual outcomes is something my colleagues and I have been advocating for courts since the promulgation of the Trial Court Performance Standards twenty years ago and continue to do so with the CourTools and the Appellate CourTools. (See one of the first few postings here, "A Preference for Outcome Measures," Made2Measure, September 28, 2005).

The University of Michigan researchers looked at over 2,000 hospitals and 325,052 surgeries with high risks of death or serious complications. They found little evidence of a consistent relationship – and this is the important part of their findings – between the rates of surgical deaths and serious complications 30 days after surgery and how well the hospitals complied with the processes prescribed by Medicare, such as giving hospital patients aspirin at discharge after heart attack treatment and doctors giving pneumonia patients antibiotics within six hours of arriving in the hospitals. In other words, there was no evidence linking what Medicare might have thought are “best practices” to improved outcomes for patients.

“I definitely think some of the money we’re spending measuring process compliance doesn’t promote quality,” said Lauren H. Nicholas, the lead University of Michigan researcher said in an interview with the Wall Street Journal. “We feel [Medicare] should be moving toward greater use of measures of outcomes.”

A Justice Index: The Quest for the Holy Grail of Court Performance Measurement

2010-09-08T13:16:00.001-04:00

My colleagues and I have long sought what is for us the Holy Grail of performance measurement -- a simple, easy to grasp index of the performance of courts and the justice system that could be used by both insiders and outsiders. Such an index recently was brought back into sight by two prominent proponents, legal scholar and law professor Laurence H. Tribe, and journalist and author Amy Bach.

Tribe is a scholar of constitutional law and former Harvard Law School professor whose students include Barack Obama, John Roberts and Elena Kagan. Tribe took the position of Senior Counselor for Access to Justice, U.S. Department of Justice (DOJ) earlier this year. In that position, he will lead an initiative aimed at improving access to civil and criminal legal services and will work with federal, state, and tribal judiciaries in strengthening fair, impartial, and independent adjudication. He will also exchange information with foreign ministries of justice and judicial systems regarding efforts to provide access to justice, as part of the DOJ’s international efforts to promote fair and impartial law enforcement and adjudication.

Amy Bach is the author of Ordinary Injustice: How America Holds Court (Metropolitan Books/ Holt, 2009), a book hailed by renowned historian Doris Kearns Goodwin as a crusading call for reform of the justice system in the tradition of Rachel Carson’s Silent Spring. Bach will be at the National Center for State Courts in Williamsburg October 7 to discuss her ideas for a justice index.

A Rallying Cry

In a speech to the annual Conference of Chief Justices in Vail, Colorado in July of this year, Professor Tribe called for a comprehensive performance index of sorts, a way of “grading the state’s legal system in terms of how well or poorly it is delivering justice to the state’s people.”

And then, in a New York Times op-ed on August 11, shortly after Tribe’s remarks, Amy Bach called for a “justice index,” a term she did not specifically use in her 2009 book, though she described it clearly.

The Justice Index, as Bach described it in her op-ed, “would function roughly like college rankings, evaluating county courts on factors like cost, recidivism, crime reduction and collateral consequences, including whether people lose their jobs or homes after contact with the criminal justice system.” She contends that the lack of information of a justice index has “a corrosive effect: without public awareness of a court system’s strengths and weaknesses, inefficiencies and civil liberties violations are never remedied.” Both Bach and Tribe suggest that a comprehensive justice index would benefit those inside the justice system – Bach contends that justice systems insiders’ “blindness” to errors is a signature feature of ordinary injustice -- and those outside the system, including the public.

While Bach is correct in contending that there is currently no “mechanism in place to keep track of the extent of ordinary injustice” in the courts, my colleagues and I have planted plenty of seeds for such a mechanism over the last ten years that I hope will take root and bear fruit as a result of Bach’s and Tribe’s rallying cry. Two of our efforts that are most relevant to a justice index are the Court Performance Index (CPI) and the Caseflow Timeliness and Efficiency (CTE) Index.

(Many other issues related to the development of what Bach refers to generally as a “court monitoring system” are explored in the 120 + postings in this Blog including the imperative of performance measurement and management and the fact that it can be done, specific measures like the length of time defendants spend in jail before trial, transparency and accountability, and citizen responsibility for courts.)

The Court Performance Index (CPI)

Few of us – including court managers and judges -- can keep track of more than handful of measures. On our car dashboard, for example, we have only a few gauges -- like the speedometer -- that we look at all the time, a few that need to be monitored regularly but not all the time (the gas gauge), several that need to be looked at less frequently (the oil meter), and some that need to be paid attention to only when we are given a warning signal.

One way to achieve simplicity and yet monitor more than a few measures is to combine multiple measures into a single, simple index. Multiple measures in a "family" of metrics can be assigned weights according to their importance and combined in an index that is an aggregate statistic.

We first introduced the idea of a Court Performance Index (CPI) at the 2003 Court Technology Conference in Kansas City. Though we viewed the CPI as beneficial for both outsiders and insiders, for the conference audience of 2,000 mostly court managers and judges, we emphasized the CPI’s use as a management tool.

Sometime in the not too distance future, we envisioned then, court executives and other stakeholders, will get all the critical information about their court's performance on their computers instantly. For court managers, the homepage of their court's website would display a window highlighting performance information summarized by a single number, the CPI, accompanied by a green or red triangle with another number indicating whether the CPI is up or down from the previous day and by how many points. (Think of the way the Dow Industrial Average is shown on CNBC and CNN.)

If a quick morning check of the court's CPI by the court manager or chief judge, for instance, reveals a green, upward pointing triangle and an increase since the previous day, all would be well. The court manager or chief judge could relax and get a cup of coffee. On the other hand, the cup of coffee may need to wait if the morning check discloses a red downward-pointing triangle and a significant downturn in the CPI. By drilling down through several screens of progressively more detailed and less aggregated performance data, beginning with a display of four to ten core court performance measures that constitute the components of the CPI (think, again, of the Dow and its component company stock values), court managers and judges would be able to pinpoint the court, division, case type or resources needing attention.

The court manager may click the CPI icon to show a more detailed screen displaying a "balanced scorecard" of the four to ten performance measures that contribute to the CPI (e.g., adaptations of the National Center for State Courts’ CourTools or Appellate CourTools). Each of the performance measures on this screen are displayed in the same simple way as the CPI, i.e., a single score for the measure, a smaller number indicating a change in the measure, if any, and a triangle indicating a downturn or upturn in the measure.

The manager might discover, for example, that the early results of one measure - for example, an online survey of court employee engagement posted the morning of the previous day - is largely responsible for dragging down the court's CPI. From the "real time" data displayed on the screen the court manager learns that 37 percent of the court employees already had responded to the survey within 24 hours of their posting on the website.

A few more clicks by the court manager produces screens revealing important additional information about this measure including trends over time; the alignment of the measures with the court's management processes such as strategic planning, budgeting, quality improvement, and employee evaluation; related measures; and best practices in the performance area gauged by the measures. Further, a note on the screen showing trends of the measures over time may indicate that the response rate to previous administrations of the surveys was higher than 90 percent of all the court's employees and that, in the past, the early returns tended to be the most negative. The court manager decides not to jump to conclusions but to mobilize the management team in the event that the survey results do not improve as more of the court employees complete the online questionnaires.

In a few minutes of the morning, the court manager will have viewed the performance "dashboard" and determined the days "score" and what needs to be done. In this example, the court manager decides simply to alert and to mobilize the court's management team in case the measure that caused the CPI downturn does not show improvement in the days ahead.

Caseflow Timeliness and Efficiency Index (CTE Index)

A more modest index, first introduced in 2004, the Caseflow Timeliness and Efficiency Index (CTE Index), joins four measures associated with Standard 2.1, Case Processing, of the Trial Court Performance Standards and detailed in Trial Court Performance Standards and Measurement System, the first three of which are also part of the CourTools: time to disposition (on-time case processing), case clearance rate, age of pending caseload, and trial date certainty. These four measures of case processing, expressed as proportions, are reduced to one in the index. The CTE Index assigns different weights (i.e., relative importance expressed as number from 1 to 100) to these measures and requires the calculations as prescribed in the Standards with some conversions to accommodate the aggregation of the measures into an index. (See “Technical Note: The Caseflow Timeliness and Efficiency (CTE) Index.” Court Manager, Volume 18, Number 4, 2004, 8-9; reprinted in Steelman, David C. et al. Caseflow Management: The Heart of Court Management in the New Millennium. Williamsburg, VA: National Center for State Courts (2004)).

Ratings and Rankings of Courts

A few weeks ago, Newsweek published its first “best countries” ratings and rankings in which it set out to answer the simple question of which country would provide you the very best opportunities to live a healthy, safe, prosperous and upwardly mobile life. Newsweek rated and ranked 100 countries in five categories of well being – education, health, quality of life, economic competitiveness, and political environment (the U.S came out eleventh in the overall ranking). Americans love rankings, the top-ten of whatever including cities, high schools and universities, hospitals and toasters.

Amy Bach believes, as I do, that such ratings and rankings get people’s attention and create incentives and healthy competition. She thinks, as I do, that they will do the same for courts. (See “Toward the 50 ‘Best in Class’ Courts,” Made2Measure, April 23, 2008). “In communities across the country, people use statistics on hospitals, schools and other public services to decide where to live or how to vote,” writes Bach. “But while millions of Americans deal with their local criminal courts as defendants and victims each year, there is no comparable way to assess a judicial system and determine how well it provides basic legal services.”

Ratings and rankings of the world’s best places to live and America’s best public high schools, colleges, hospitals and parks hold two important lessons for judges and court managers who might benefit from the use of the CPI, the CTE Index, or a justice index envisioned by Bach and Tribe, especially those who might bristle at the idea of comparative performance measurement. (See “Ten Reasons Not to Measure Court Performance,” Made2Measure, November 19, 2008; and “How Do we Stack Up Against Other Courts? The Challenges of Comparative Performance Measurement,” Court Manager, Vol. 19, No.4 (Winter 2004-2005).

As Ms. Bach points out in her op-ed, the first lesson is that performance matters to citizens. The U.S. News & World Report annual rankings of America’s best public high schools, for example, are based on the key principle that a great high school must be able to produce measurable academic outcomes to show that it successfully educates all of its students across a range – a balanced scorecard – of performance indicators. Little else matters to parents as much.

The hallmarks of a high-performing court are the political will and the capacity to produce measureable outcomes that matter to the public and other stakeholders. The best courts ask themselves “How are we doing and how could we improve?” They measure their performance on a regular and continuous basis to gain knowledge and insight into new realities, opportunities and necessities and to convert that knowledge and insight into effective strategies for improvement.

The second lesson to be learned from such ratings and rankings is that similar comparisons and rankings of courts are likely to occur with increasing frequency, as Professor Tribe’s remarks to the state chief justices in July suggest. Court leaders are well advised to be prepared with their own court performance data.

The fact is that the best places to live, to send our kids to school, to get heart-bypass surgery and, yes, to get access to justice are of great interest to citizens. Courts not producing performance data to satisfy this interest are going to be asked why they’re not. What gets measured gets attention. What gets measured gets done!

It’s Not About Numbers

The measurement process uses numbers to provide an understandable and comparative result, but it is ultimately not about the numbers. It is about perception, understanding and insight. Ultimately, it is not the measure itself that is important but rather the questions that it compels us to confront.

Amy Bach concludes her groundbreaking book with these encouraging thoughts about a justice index and performance measurement in general: “Metrics would offer a mirror for people who work in the system, allowing them to see how their roles might have eroded at the expense of rights and public safety. Metrics alone are not an answer but they could be the beginning one. They are the tools we need to ask for the courts we deserve.”

© Copyright CourtMetrics and the National Center for State Courts 2010. All rights reserved.

Who Has More Innovative Ideas Than You Do? Your Employees

2010-09-05T13:20:00.002-04:00

You’ve got to give people at all level of your organization the opportunity to find solutions to problems. You’ve got to mobilize everyone to generate improvement strategies, not just the people at the top. A court that depends solely on its senior management to address its challenges risks failure.

That is the advice of Ronald Heifetz, Alexander Grashow and Marty Linsky give to managers and leaders in their article “Leadership in a (Permanent) Crisis” in a recent issue of Harvard Business Review. The three are partners of Cambridge Leadership Associates and authors of The Practice of Adaptive Leadership (Harvard Business Press, 2009).

Their advice for adaptive leadership is reason enough to give all court employees all of the court's performance data on demand, whenever they need it, in real time -- not just once a year or once a month. But there’s an even more compelling reason that should strike at the heart managers who aspire to leadership: You don’t have a monopoly on good ideas. Never did and never will.

The mistaken belief that top management or the consultants it hires have a monopoly on bright ideas, especially in a top-down command-and-control organizational structure replete with silos and communication stovepipes, creates a handicapping condition for courts that will prevent them from reaching excellence. “Companies that have successfully made innovation part of regular continuing strategy did so by harnessing the creative energies and insights of their employees across functions and ranks," write JC Spender, a visiting professor at ESADE in Barcelona, and Bruce Strong, a founding partner at CBridge Partners, in an August 23, 2010 article in the Wall Street Journal.

© Copyright CourtMetrics 2010. All rights reserved.

Gainsharing and the British Royal Navy

2010-05-15T11:55:00.001-04:00

The British Royal Navy in the age of Admiral Lord Nelson (1758-1805) knew a thing or two about incentivizing employees. According to John Steele Gordon writing in yesterday’s Wall Street Journal (“Incentives vs. Government Waste,” May 14, 2010, A19), the British Royal Navy was extremely good at capturing enemy warships and sweeping enemy commerce from the seas. And there was a good reason why.

The Royal Navy’s success, says Gordon, was due to the enormous incentives that it offered its officers and men. The entire value of the spoils was shared by the Navy officers and their men according to a rule of eights. One-eight went to admiral; two-eights went to the captain of the ship; one-eighth each to the commissioned officers, senior warrant officers, petty officers, and midshipmen; and two-eights to the crew.

These were not insignificant amounts of loot. They could make captains rich by the standards of the mid-18th century. The crew could receive many times a year’s pay. Gordon asserts, correctly I believe, that incentives are the reasons why private profit-seeking corporations like Google or Goldman Sachs are so much more efficient than public institutions like the Department of Motor Vehicles or courts. Harnessing the self-interest that fuels the engine behind capitalism’s success, Gordon writes, would save money and increase innovation in a bureaucracy, just as the Royal Navy used it to capture enemy ships.

Gordon, the author of Hamilton’s Blessing: The Extraordinary Life and Times of Our National Debt, out in a recently revised edition by Walker & Company, believes it is quite possible to incentivize public employees to find ways to save money, and to find better ways of doing things, using the Royal Navy’s rule of eighths.

Gordon may not have heard of “gainsharing,” a type of profit-sharing system used by local governments and at least one court, which I have advocated in previous blogs (see Gainsharing in the Courts, January 21, 2007); and Project Gainshare, May 3, 2007). Like the British Royal Navy’s rule of eights, gainsharing is a system whereby units of government share in the gains employees make in their bottom line or that of the state, county or city, without losses in quality of services and programs. Employees receive bonuses or payments based upon the improved productivity or efficiency as reflected in "gains" in costs savings or revenue increases. Gainsharing is, of course, consistent with widely accepted management principles that encourage employee initiative in continuous improvement of program and services to meet the needs of customers and citizens.

The promise of gainsharing for courts, I argued back in 2007, is that gainsharing may help courts achieve sustained increases in productivity and efficiency, that employees may become more involved in the gains made by the court as they share in the benefits of employee-initiated improvements, that it enhances commitment to organizational goals, and that it leads to improvements in other measures of court performance.

The United States Bankruptcy Court for the Eastern District of California demonstrated “over a period of five years that gainsharing could work, and work well, in a court environment,” said Richard Heltzel, Chief Executive Officer, when I spoke to him in 2007 about Project Gainshare, as the pilot project was called. The project returned “over $2.5 million in unspent funds in five short years, an ability to operate to operate in a very lean, but effective fashion, and with enthusiastic staff support at the grass roots level,” said Heltzel. He was optimistic about the likely success of gainsharing in court environments. “I am here to tell you,” he said in 2007, “that a gainsharing program can work in a court environment when coupled with an organizational performance management system that relies on statistically based measures and customer satisfaction surveys to ensure that the job is getting done in timely and quality fashion.”

Unfortunately, Hetzel’s gainsharing program met an untimely demise because, unlike corporate employees who are highly incentivized to increase efficiency and innovate, public employees are highly disincentivized. Whereas in business a penny saved is a penny earned, the penny saved by Hetzel was apparently a penny cut from his budget. Well, good luck with that!

With their backs to the wall in the current fiscal crisis, maybe it’s time for court leaders and managers to go out on a limb (gangplank?) and give gainsharing a real try. Just maybe, state and local courts who are out of options for cutting costs might be willing to innovate with something that worked exceedingly for the British Royal Navy in 1765.

© Copyright CourtMetrics 2007. All rights reserved.

The Epitome of Humanity

2010-04-24T04:33:00.001-04:00

As a social scientist who has spent the last several years in developing countries learning empathy -- mostly to shut up and to listen carefully to what my counterparts have to say, something I was trained to do but often have failed to do -- I was heartened by Joshua A. Dijksman’s deep bow (Views, International Herald Tribune, April 23) to the “rights and duties to science and society” that serious researchers bear. Like Dr. Dijksman, I too believe that the scientific method – careful observation, objectivity, patience, modesty, replication, and only drawing conclusions based on very reliable data – can be the very “epitome of humanity” that can lift us all above the prejudices, slander, downright lies and rumors and the kind of “disingenuous, small-minded social keelhauling” that serves as public discourse and much of policy these days.

Making Headway - The Key to Employee Engagement

2010-01-08T17:45:00.003-05:00

We’re making good progress! This declaration suggests motivation, optimism, dedication, and commitment. When employees sense that they’re making headway toward a clearly defined goal, their drive to excel is at its peak.

Understanding the power of the perception of progress is one of the ten breakthrough ideas for 2010 compiled by Harvard Business Review in cooperation with the World Economic Forum. “Ask leaders and managers what they think makes employees enthusiastic about work, and they’ll tell you in no uncertain terms …[r]ecognition for good work,” write Teresa M. Amabile and Steven J. Kramer in the January – February 2010 issue of HBR.

Trouble is that they’re wrong. Amabile and Kramer surveyed more than 600 managers in dozens of companies about five factors commonly considered to be instrumental in motivation and positive feeling about work: recognition for good work, incentives, interpersonal support, support for making progress, and clear goals. Recognition for good work came out number one.

But when Amabile and Kramer looked at the results of their multi-year study in which they tracked the day-to-day activities, emotions and motivation levels of hundreds of knowledge workers in a wide variety of settings, they found that the factor that the leaders and managers responding to the survey ranked dead last – the perception of progress – was the top motivator of knowledge workers.

“On the days when workers have the sense they’re making headway in their jobs, or when they receive support that helps them overcome obstacles, their emotions are most positive and their drive to succeed is at its peak,” write Amabile and Kramer. “On days when they feel they are spinning their wheels or encountering roadblocks to meaningful accomplishments, their moods and motivation are lowest.” This result showing the importance of progress associated with positive emotion and high motivation was apparent in the 12,000 detailed diary entries that the knowledge workers completed as part of the research.

Court managers should regard this finding as very good news. First of all, it should be comforting to know that employees can be engaged with a court’s mission and goals when they feel they are making strides toward achieving them – something that probably is equally true of the court’s leaders and managers. Second – and this is I assume why understanding the power of progress is among HBR’s breakthrough ideas for 2010 – the key to employee engagement and motivation turns out to be in court managers’ control. They can establish clear goals, identify unambiguous measures of success, provide support and resources, and “proactively create both the perception and reality of progress.”

© Copyright CourtMetrics 2009. All rights reserved.

Henry Mintzberg Misses the Mark on Performance Measurement Data

2009-11-15T11:43:00.002-05:00

We’ve all been in situations where we get agitated because someone we admire, and with whom we generally agree, goes too far in pushing his or her agenda. This situation occurred to me as I read Henry Mintzberg’s new book, Managing (Berrett-Koehler, 2009), which updates his thinking in his first book, The Nature of Management (Harper & Row; reprinted by Prentice-Hall, 1973) based on his doctoral dissertation more than 35 years ago.

Mintzberg all but dismisses the value of using performance outcome data in favor of an “information diet” of gossip, hearsay, and speculation.” Such “informal information” he writes, “can be much richer, even if less reliable.”

I could not believe what I was reading. It goes counter to what I’ve been advocating to court managers for years, i.e., effective performance measurement and management can transform your court; it shows you where you are and gets you to where you want to be. Performance monitoring, analysis and management are no longer an option for successful court managers. (See, for example, "Principles of Effective Performance Measurement," Made2Measure, November 12, 2007; and "Court Intelligence – A Matter of Survival," Made2Measure, December 11, 2007)

While Mintzberg’s observations and insights about what managers actually do – the actual practice of management – are invaluable contributions to the literature of management and leadership, he badly misses the mark when he disparages management based on what he calls “formalized management information systems.”

I know that many of today’s court managers rely heavily on performance data and are wary of spouting anecdotal evidence when asked “How is the court doing?” They know that their courts’ stakeholders increasingly want them to show them the data, and nothing else will do.

The Elevation of Anecdote, Gossip and Speculation

Today’s managers ignore evidence-based performance data and rely instead on a steady information diet of gossip, hearsay and speculation at their peril.

Consider this: If your court or court system has developed the right measures (e.g., court user satisfaction, case clearance, on-time case processing, and employee engagement), all on a balance scorecard of performance that is available to you on demand on a self-service basis in real-time, why would you risk basing your decisions on gossip, hearsay and speculation? Excuse the metaphors, but why would you not rely on your car dashboard right in front of your eyes to give you a reading of your car’s speed and, instead, ask the passengers to speculate how fast you’re going? (Such speculation won’t carry the day with a trooper that has stopped you for speeding?) Or why would you depend on speculation about your health when you could be reading your blood pressure and cholesterol levels?

A Blind Spot

But let me give Mintzberg his proper due. He’s the Cleghorn professor of management studies at McGill University and one the most original management thinkers writing today. As I said, I generally admire his writings.

Many years ago, my colleague and friend Bob Wessels -- the manager of the Harris County (Texas) Criminal Courts, and a innovate manager in his own right who has invested heavily in performance measurement and business intelligence -- introduced me to Mintzberg’s writings and, like Wessels and others in court administration, I have read his writings steadily since.

My copy of his 1994 book, The Rise and Fall of Strategic Planning: Reconceiving Roles for Planning, Plans and Planners (Free Press) is dog-eared and filled with my marginal notes. I found his recent article in the July – August 2009 issue of Harvard Business Review, “Rebuilding Companies as Communities,” truly inspiring.

Mintzberg has championed study of the status quo – what managers actually do rather than what management theorist conjure up about the job of management and leadership. He is among only a few scholars who have bothered to study mangers and to make some sense of the work they actually do in organizations. For this alone, he is worth reading.

Mintzberg, of course, is also right that much of management is not necessarily information-driven.”To manage through information means to sit two steps removed from the ultimate purpose of managing: information is processed by the manager to encourage other people to take the necessary action. In other words,” he writes, “on this plane the focus is neither on people nor on actions directly, but on information as an indirect way to make things happen.”

So far so good, but why go to the extreme of dismissing performance data almost altogether and instead elevate gossip, hearsay and speculation as staples for managers? One step too far, I say, you’ve lost me in what appears to be a big blind spot.

I am particularly disturbed and disappointed about the message Mintzberg sends to court professionals about modern performance measurement and management systems, i.e., the monitoring, analysis, and management of results (outcomes or accomplishments) on a regular basis for the purposes of transparency, accountability and continuous improvement. The significant position that these modern information guidance systems occupy in today’s management practice seems to have been missed or ignored by Mintzberg.

Maybe because Mintzberg has focused much of his work on what managers actually did years ago, he shows little patience with what managers may have to do, what they should do, and what they want to do today and in the future. All of the latter, he suggests, is part of the popular but unreal image of the manager as symphony conductor carefully planning, organizing, coordinating, and controlling.

But if the managers Mintzberg studied really favored a steady diet of anecdotes, gossip and speculation, I would go so far as to suggest that they are bad managers, or at least not as good as they could (and should) be. (See "This Just In: Performance Measurement Works," Made2Measure, September 24, 2008)

Mintzberg blind spot may also extend to what today’s managers actually do in the area of performance measurement and management. For example, many managers, including court managers, are increasingly turning to performance dashboards, an aspect of business intelligence, to monitor, analyze and manage their organizations’ performance. (See "The Real Promise of Performance Dashboards," Made2Measure, May 9, 2007)

The information on a performance dashboard is consolidated and arranged in a combination of text and graphics on a computer screen that busy executives, managers and staff can easily monitor and analyze. The dashboard display is dynamic, allowing users to navigate rapidly across and through layers of strategic, tactical and operational performance data as they choose, whenever they choose, as data is updated on a real-time basis (the performance data available on the modern dashboards, like the one completed in September 2009 by my colleagues and me for the Kosovo Judiciary updated every day).

Performance dashboards help court managers to identify problems or opportunities quickly as they happen, to identify trends and patterns, to discover ways to improve programs and services, and to help guide them toward effective decisions. It provides a “line of sight” between core strategic indicators and subordinate tactical and operational indicators that convey to them what the drivers of successful court performance might be, and gives them concrete knowledge about how they contribute to that success.

Sure beats gossip, hearsay and speculation!

A Straw Man

Mintzberg writes that managers favor oral communications because the information they get from formal management information systems is not timely, is outdated once it gets to them, and typically is not easy to understand. What busy mangers have time for that, he asks.

This is a straw man argument and it surely goes too far. No one is suggesting that busy managers should wait around for someone to deliver untrustworthy performance data long after is it of any use. Has Mintzberg bothered to sample the burgeoning literature on performance dashboards and business intelligence technology? Did he happen to catch the special July 2009 issue of Wired focused on the “personal metrics movement” with the cover title “Living by Numbers – Track your data? Analyze your results. Optimize your life”?

The trouble is that Mintzberg has what seems to be an outdated perspective on modern management information systems and the use that managers make of them today. He rails against email – he refers to it as “this new medium” --in ways that I think readers will find archaic, if not uninformed.

I confess that as I read Managing I could not help think of Mintzberg as a technophobe who takes stances against modern information technology in order to preserve his ideologies.

© Copyright CourtMetrics 2009. All rights reserved.

Court Executives (Should) Have Their Heads in the Clouds

2009-10-27T16:33:00.002-04:00

The launch this month of Microsoft’s new operating system, Windows 7, marks the end of one era of information technology and the start of another, says the Economist (“Briefing Cloud Computing,” October 17, 2009). Windows is not going to disappear but it will be much less important in the future.

Cloud Computing

Much of the computing we do today on our computers in our homes and offices will soon be – so to speak - in the “clouds,” and not on our personal computers, where Windows resides today. Instead, desktop computing on personal computers – featuring full-featured database and spreadsheet capabilities – is being replaced by IT architectures that call for the heavy lifting to be performed by external data centers accessible to us over the Internet.

Cloud computing is attracting an enormous amount of attention. The term “cloud computing” is a metaphor that originated with IT architects who routinely used cloud shapes to depict the flow of data from unknown external sources instead of the squares, rectangles and cylinders representing internal servers, storage disks, and databases they used to represent internal sources.

We are already familiar with cloud computing technology, though we may not have grasped its revolutionary significance. Widely used services like Twitter, YouTube, Skype, and Google’s Gmail, Docs, and Spreadsheets are all built on cloud computing technology. They deliver content and services to our computers, smart phones, and other devices from external sources

An Attractive Option for Cash Strapped Courts

Chief information technology officers see in cloud computing the promise of lower costs, less complexity and work for their IT departments. Cloud computing enables a court to tap into computing, storage systems, and software applications like performance dashboards and other business intelligence over the Internet.

At least in theory, cloud computing technology all but eliminates the burden of software installation, maintenance, ongoing operation, and support. Coming up with new software versions or changing requirements is taken care of the by the service provider. Costs are a continuous expense, rather than a single expense at time of purchase.

Cloud computing lets courts avoid building their own performance dashboards and associated technology architecture. They pay only for the services they need, when they need them. Both big and small corporate computer and software vendors that have a presence in the court community will soon be offering performance dashboards based in cloud computing technology.

One company, Threshold CS, based in Clearwater, Florida, makes its court performance dashboard, Justice PM (Performance Measurement) available both as a traditional client site installation and as a cloud computing service (Threshold calls it by its antecedent name, Software as Service or SaaS). For a monthly subscription fee, after a brief data linkage to the court’s source (legacy) systems, a court’s performance data is automatically uploaded to Threshold’s data center and the performance dashboard is accessible to court users via a secure Web interface.

© Copyright CourtMetrics 2009. All rights reserved.

Rankings Based on Outcomes

2009-10-26T18:31:00.004-04:00

Today's Wall Street Journal printed my letter to the Editor on the value of rankings focused on outcomes, a topic that has occupied this space often:

Most “number guys” can criticize performance measures, especially rankings, in their sleep because most measures are imperfect. Carl Bialik (“Ill-Conceived Ranking Makes for Unhealthy Debate,” The Numbers Guy, Oct. 21) takes the easy route for a cheap shot at the dated and flawed World Health Care low 37th ranking of the U.S. in the world in health care. He suggests that the “unhealthy debate” caused by the U.S. ranking would be cured by more methodological rigor and that, in any event, we shouldn’t rank everything, especially health care. I wonder if Mr. Bialik thinks that our debate about health care would be healthier if the U.S. ranked let’s say 12th in the world using methods that pass muster with the scientific community. I suspect that most of us would like to see the U.S. in first place and that even 12th place would not change what we think of health care today. Americans love rankings, the top-ten of whatever including cities, high schools and universities, hospitals and toasters. I think rankings of health care are here to stay and that’s a good thing. As flawed as they may be, they focus attention of the debate where it should be, on outcomes.

Ingo Keilitz
Williamsburg, Virginia

Decentralized Innovation and Improvement

2009-05-11T20:50:00.003-04:00

Court systems have concentrated too much authority for continuous improvement at the top where there are good intentions, but relatively few resources and little capacity. Court performance data are delivered too little and too late, if at all.

When performance is presented to staff, it is often done so in endless documents stuffed with indecipherable figures and statistics, to make much of a difference. Performance data is a virtual temple secret that only the priests (designated top-management and analysts) can read and interpret.

Courts should seek to give all court employees all the performance measurement information they need to make improvements themselves. Courts should tap into the capacities of all court employees to track and analyze performance data and to devise solutions to problems.

A similar strategy, referred to as “radical transparency,” was advocated as a road map for economic recovery in a “manifesto” written by Daniel Roth in Wired March 2009: “Instead of assigning oversight responsibility to a finite group of bureaucrats,” writes Roth, “we should enable every investor to act as a citizen-regulator. We should tap into the massive processing power of people around the world by giving everyone the tools to track, analyze, and publicize financial machinations. The result would be a wave of decentralized innovation.”

Decentralized innovation and improvement are central to "open book management," a management approach created by Jack Stack and popularized by John Case in his 1995 book, Open-Book Management: The Coming Business Revolution (New York: Harper Business). Its aim is to give employees all relevant information (the focus is on financial information) about their company so they can make better decisions as workers.

The concept behind open book management is the same as that supporting "line of sight" in performance measurement and management. That is, in order for performance measures to be practical tools and to serve as incentives for improvement -- for measures to be motivational -- there must be a line of sight between the measure and actions that can be taken by employees at various levels of the court system. It conveys to everyone what the drivers of success are and provides them with the concrete knowledge of how they contribute to that success.Here are the basic rules of open book management (extended beyond financial information) to all performance data available to court managers and staff as part of a comprehensive court performance measurement system:

-- Give all employees access to all performance data on a self-help basis. Line of sight can be created by computer based graphic displays in performance dashboards.

-- Give employees a simple performance scorecard. Because employees may not be naturally drawn to mind-numbing sets of numbers, open book management relies on the “critical number” and a “scorecard” that brings all the critical numbers together.

-- Give employees training to understand and to use the performance information available to them including an intensive system of performance review meetings to keep employees informed about the status of the court in terms of its performance measures.

-- Give employees ownership and responsibility for the numbers under their control. If the performance measure rises above the control level, the “owner” of the measure is held accountable.

-- Give employees a stake in how the court performs.

One proponent of open book management, Ed Dorian, the president and chief operation officer of a White Plains, N.Y., company, was quoted as saying in a recent Wall Street Journal article, “I saw open book as an opportunity … to send a message to our staff that said, ‘We trust you. And we not only trust your intentions, but we trust your ability to help us.’” (Laura Lorber, “An Open Book: When Companies Share Their Financial Data With Their Employees, the Results Cab Be Dramatic,” Wall Street Journal, February 23, 2009, R8).

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2009. All rights reserved.

Annals of Backlog and Congestion: New Delhi, India

2009-02-17T09:21:00.002-05:00

State court leaders and managers take heart! Things could be worse. Much worse!

In a widely circulated story last week, Sam Dolnick of the Associated Press (AP) reported that, according to Chief Justice A.P.Shah, the High Court in New Delhi is so behind that it could take up to 466 years (not days or even months, years) to clear its backlog of cases. In a vast understatement, retired Supreme Court Justice J.S. Verma, who is critic of the system, is quoted as saying “I don’t think you would have to wait four centuries to have a case decided.”

Reasons cited for the backlog in India include the usual suspects: lack of accountability for results, corruption, inefficiency, and an uneven application of the rule of law favoring the wealthy and well-connected. Another is that India does not have enough sitting judges. "It’s a completely collapsed system,” Prashant Brushan, a well known lawyer in New Delhi, is quoted as saying. “This country only lives under the illusion that there is a judicial system.”

What is particularly noteworthy and instructive about this story is the New Delhi High Court’s use of its jaw-dropping congestion rate as the measure of efficiency (or in this story, inefficiency) to stimulate reform and buttress its appeal for an injection of more judicial resources. Congestion rate is the pending caseload divided by the number of resolved cases. It estimates the amount of time it would take a court to dispose of its pending caseload given its current clearance rate.

As the New Delhi report suggests, congestion rate is a more meaningful and persuasive efficiency measure than backlog (percent of pending cases “older” than an established threshold) and clearance rate (the number of outgoing cases as a percentage of incoming cases), and even other measures such as the number of judges per capita. In a 1999 technical paper, Court Performance Around the World: A Comparative Perspective, Maria Dakolias of the World Bank reported that it would take Ecuador a little more than ten years to dispose of its pending caseloads, the poorest congestion rate among 12 countries she compared.

I don’t know what’s good or bad, but I immediately know that 466 years of congestion is really bad. I also have a sense of the scope, if not the nature, of the solutions. It will take more than a band aid to fix the New Delhi High Court’s backlog and congestion.

The lesson is that if we want our performance measurement initiatives to be noticed and used – what gets measured gets attention and gets done – we need to identify and develop measures that translate abstract concepts like efficiency and effectiveness into terms that are clear, unambiguous, and actionable and that capture the attentions of the public and other stakeholders.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2009. All rights reserved

Measuring What Really Matters in Hard Times

2009-02-12T11:51:00.002-05:00

State courts are facing severe budget cuts in the current economic crisis. According to the Center on Budget and Policy Priorities, at least 44 states are facing shortfalls in their FY 2009 and/or FY 2010 budgets. By most accounts, the situation is likely to get worse before it gets better.

Some of my more optimistic colleagues (who -- I might add -- are fortunate enough to have solid jobs) subscribe to the “necessity is the mother of invention” school of thought on the deepening recession. They have a point.

While they do not wish ill toward their court friends on the receiving end of drastic budget cuts, they see a bright spot in the months and years ahead. They welcome the sense of urgency. They're hoping it will give birth to clarity of focus and innovation.

They see courts and state court systems today forced to confront issues and questions that they believe should be asked all the time, not just now: What are our fundamental obligations? What is expected of us? Which programs and services are mandated and which are not? What should we stop, continue and start? On what basis?

My sense is that as local courts and state court systems deal with budgets that are hemorrhaging, the internal and external scrutiny of courts will intensify and focus on what matters most. In good times, the operative question is “Are you doing things right?” In other words, are the programs or services doing what they are designed to do?

In bad times, the question shifts more toward fundamental worth and merit, toward the impacts of programs and services, to return-on-investments, and to cost-benefit: “Are you doing the right things?” In other words, is the program or service you are providing meeting the court’s fundamental governmental obligations. Are they contributing to a positive change in the status or condition of people they are designed to serve? If not, then you need to give up the resources or move them to programs and services that have positive impacts.

For court performance measurement and management this means “counting what counts,” i.e., a preference for outcome measures over input and output measures. (See “A Preference for Outcome Measures,” Made2Measure, September 28, 2005).

Jack Roedema, court administrator of the 17th Circuit Court in Kent County, Michigan, believes that he has witnessed first-hand this shift toward a preference for outcome measures. I know that he and Chief Judge Paul Sullivan have done a masterful job in the past using performance data to develop, maintain and strengthen the court’s relationship with county officials who fund the court. The court uses close to 100 performance measures on a regular basis.

In a recent email, Mr. Roedema said that he is seeing county officials looking at the court from a different perspective than they did in the past. They still greatly appreciate assurances backed by solid data showing that the court programs and services are doing what they’re supposed to be doing, but they are beginning to ask “so what.” With emphasis on mandated versus non-mandated services and significant budget restraints in his county, he says that it “seems like in our court we need to go a step further with [performance measurement].” In the area of alternative dispute resolution (ADR), he is beginning to report the results of a “sort of cost-benefit analysis.” If as a result of mediated settlements the ADR program prevents 60 cases from reaching a judge, then “we are saving about 120 hours of judicial time,” Roedema said.

This type of information now seems to strike more of a chord with the county officials, Mr. Roedema believes. “Of course, you can't really get to the ‘so what’ information unless you have good performance measures in place,” he said.

I told him that I agreed that he was heading in the right direction by looking at cost and time savings of programs and services, but I recommended that he set his sights higher toward even more outcome-oriented performance measures. For example, I suggested that he think of the purpose of juvenile diversion, another program he cited, not only in terms of savings in court resources, but also in terms of community safety, welfare and protection. How do you measure success in that area? Recidivism? Payment of restitution? Direct contributions to the community by work in lieu of monetary penalties? What is the court’s fundamental obligation in these programs?

If funders are made aware by trustworthy performance data that a program or service not only runs the way it was designed, but also contributes significantly to the protection and well-being of the community, they are much less likely to give it the ax. For courts and state court systems, counting what counts and measuring what really matters may be a sound strategy not only for accountability, but also for survival.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2009. All rights reserved.

Ranking High Schools and Courts on Their Performance

2008-12-15T08:49:00.000-05:00

The 2009 U.S. News & World Report second annual rankings of America’s best public high schools came out this week. The rankings were done by School Evaluation Services, a K – 12 education data research firm run by Standard & Poor’s, based on an analysis of the performances of 21,069 public high schools in the 2006 -2007 school year (see www.usnews.com/highschools).

The annual rankings of high schools hold two important lessons for judges and court managers, especially those who bristle at the idea of comparative performance measurement. (See “Ten Reasons Not to Measure Court Performance,” Made2Measure, November 19, 2008)

The first lesson is that performance matters to citizens. The U.S. News & World Report rankings are based on the key principle that a great high school must be able to produce measurable academic outcomes to show that it successfully educates all of its students across a range – a balanced scorecard – of performance indicators. Little else matters to parents as much.

The hallmarks of a high-performing court are the political will and the capacity to produce measureable outcomes that matter to the public and other stakeholders. The best courts ask themselves “How are we doing and how could we improve?” They measure their performance on a regular and continuous basis to gain knowledge and insight into new realities, opportunities and necessities and to convert that knowledge and insight into effective strategies for improvement.

The second lesson to be learned from the U.S. News & World Report rankings is that similar comparisons and rankings of courts are likely to occur with increasing frequency. Court leaders are well advised to be prepared with their own court performance data.

Just last week, the Michigan State Court Administrative Office took great pains to inform a state-wide gathering in Lansing of 100 chief trial court judges and court administrators about “current misinformation” about the cost-per-case and clearance rates of Michigan trial courts compiled and presented to state and national groups by a Michigan State University academic researcher.

The fact is that the best high school, colleges and universities, cities, public parks and, yes, America’s courts are of great interest to citizens. Courts not producing performance data to satisfy this interest are going to be asked why they’re not (see “How Do we Stack Up Against Other Courts? The Challenges of Comparative Performance Measurement,” The Court Manager, Vol. 19, No.4 (Winter 2004-2005).

See also “Toward the 50 ‘Best in Class’ Courts,” Made2Measure, April 23, 2008

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2008. All rights reserved.

Ten Reasons Not To Measure Court Performance

2008-11-19T14:35:00.000-05:00

This post is based on a December 9, 2008, presentation to a seminar of Michigan Chief Judges and Court Administrators sponsored by the Michigan Judicial Institute, at the Michigan Hall of Justice Conference Center in Lansing, Michigan. It is an updated and expanded version of the Made2Measure post, Eight Reasons Not to Measure Court Performance, April 5, 2006.

It is not sufficient simply to proclaim the benefits of court performance measurement – accountability, transparency, focus, attention, understanding, control, predictability, influence, and strategy development -- and expect acceptance and effective implementation.
Performance measurement, like any tool, has shortcomings and introduces disruptions of the status quo that should not be dismissed or ignored.

These shortcomings and disruptions can be minimized and even eliminated, however, when they are identified, clearly understood, thoroughly and candidly explored, and addressed in specific terms. Unfortunately, they are often framed as absolute deterrents or broad indictments of performance measurements that prevent honest debate.

In his insightful (and irreverent) 1998 book, Measuring Up: Governing’s Guide to Performance Measurement for Geniuses [and Other Public Managers] (Governing Books, Washington, DC), author Jonathan Walters, tallies up the reasons why you can’t possibly do performance measurement, especially in the public sector, reasons that those around you will use to argue why you can’t. With a nod to Walters, here are ten overlapping reasons not to measure court performance that you’re likely to encounter.

Reason 1: Performance Measurement Threatens Judicial Independence

The very idea of court performance measurement – this wild-card argument goes – is antithetical to the principles of judicial independence and separation of powers. It rests most of its weight on the assumption that performance measurement is imposed by forces external to the court – legislatures, state, county or city executives. Even if a court has a strong hand in the design and use of the performance measures, the argument goes, performance measurement will expose a court to external scrutiny and meddling and, in so doing, hand over control in ways that will erode independence and blur the separation of powers.

No doubt, this reason resonates with many judges and court managers. Not so with the Nation’s state court leadership.

In various policy statements and resolutions, state court leaders link independence to accountability, and accountability to performance measurement. For example, with Joint Resolution 14, In Support of Measuring Court Performance, adopted on August 3, 2005, the Conference of Chief Justices (CCJ) and the Conference of State Court Administrators (COSCA) first join independence and accountability by recognizing that accountability fosters an environment where legislators, executive agencies, and the public understand the judiciary’s role and are less likely to interfere with the judiciary’s ability to govern itself. Institutional strength and integrity are protected, the members of the conferences suggest, not by isolation but instead by openness, transparency and collaboration. The conferences next make the link of judicial independence and performance measurement by declaring that “judiciaries need performance standards and measures that provide a balanced view of court performance in terms of prompt and efficient case administration, public access and service, equity and fairness, and effective and efficient management.” In short, the counter argument is that performance measurement strengthens independence through accountability rather than weakening it.

Reason 2: Performance Measurement Is Nothing More Than a “Gotcha” Game

This reason is rooted in the belief that the purpose of performance measurement is finding fault and playing the “gotcha” game, the fear that performance measurement is used to control, justify, audit and determine who went wrong, rather than what went wrong. People don’t like being judged and graded. For many, the word “performance” conjures up beliefs and fears about tests and races, about zero-sum games, about “beating” others or “beating” a standard. All but one is a “winner” in these games, the rest are “losers” or “sub-standard.” It’s not something most of us like to contemplate.

The purpose for which performance measurement is used is the most important determinant of people’s reaction to it. A major point in Dean R. Spitzer’s insightful 2007 book Transforming Performance Measurement: Rethinking the Way We Measure and Drive Organizational Success (New York: American Management Association) is that people actually like measuring and being measured. Think of young readers who proudly report how many pages they’ve read. What people don’t like is being judged.

Spitzer maintains that it all depends on the context, on how performance measurement is pitched and experienced. Is it presented and experienced as a steering tool or as a grading tool. The highest purposes of performance measurement are to learn and to improve. Performance measurement tends to be much more positively accepted if it is used to assist in learning and improvement.

Reason 3: Performance Measurement Is Inherently Misguided Because Courts Have Little Control of Outcomes

Jonathan Walters calls this the “cause-and-effect and long term consequences conundra of performance measurement” in the public sector. Government actions are often hard to link to results, and the ultimate results of many government programs won’t be known for a long time, if ever. In the courts, the argument goes something like this: Because it’s hard to say with any certainty that what a court does leads to better access, improved timeliness, and more fairness and, in any event, the court ultimately has so little power over them, measures of these outcomes are inherently unfair to the courts. Why should courts be held accountable for things over which they have little control?

No doubt, it’s easy to get hung up on arguments over cause-and-effect and over control of long-term consequences. But are these arguments enough to drop the whole idea of performance measurement?

There is a growing number of court leaders who believe just the opposite: that these arguments, in fact, make the case that it’s high time to forge ahead with performance measurement. They know one thing for sure: we need to know where we are (baselines) and where we’re going (trends) before we have even a hope of knowing what causes what. How else, for example, will courts be able to distinguish between internal (e.g., shifting resources) and external (e.g., legislative mandates) causes of increases in cost-per-case?

It may be that for some measures, courts can do quite a bit by themselves to impact outcomes and, for others, they need a little help from their friends. Either way, courts are better off knowing than not knowing.

Reason 4: Uniqueness of My Court Defies Performance Measurement (There’s No Way to Measure What We Do)

No two courts are alike. No two jurisdictions are the same. Our court is unique. In fact, what we do is so unique that it can’t be captured by performance measurement.

Such assertions, especially when linked to judicial independence, shape Reason 3. Those who make it are not at all convinced by Peter Drucker’s admonition, “You can’t manage what you can’t measure”? They are smart and thoughtful people who are just philosophically resistant to performance measurement. “We don't work on an assembly line making widgets,” they might say. “We don’t work with machines. We deal with unique cases involving human beings who have complex problems that require individual attention. You can’t just send in the bean counters and number crunchers and crank out a bunch of performance indicators to capture what we do on a spreadsheet. It just can’t be done.” (This reason is what Bob Behn, a lecturer at Harvard University’s John F. Kennedy School of Government, calls the “direct assault on the spread-sheet guys.”)

This reason for not doing performance measurement will not get much traction. First, if there truly is no way to measure a court’s performance in a meaningful way, maybe it’s fair to ask whether the court really is contributing in areas of importance to the court’s stakeholders – access to justice, fairness, system integrity, and public trust and confidence – and to suggest that the court should turn its attention and energies to something that does contribute. Second, the assertion that we can’t measure success should not preclude any attempts to try. Courts today are measuring performance thought “unmeasurable” just a few years ago.Finally, too many courts hide behind their differences. While it is true that substantial differences exist among courts and jurisdictions even within a state, there are more similarities than differences. Every court provides citizens access to a forum for resolving legal disputes, each processes and decides cases, each must take and preserve a record of its proceedings, each must work cooperatively with its justice system partners such as law enforcement agencies, and each must engender public trust and confidence.

Comparative performance measurement involves serious efforts to identify courts that are “roughly” comparable, those that are most similar to each other. Once this rough comparability is established (and accepted by courts applying the comparative measurement), remaining unique characteristics among the courts become factors that may account for differences in performances.

Reason 5: Performance Measurement Is an Invitation to Unfavorable and Unfair Comparisons

This reason is based in fears that performance measurement will show that “my” court does not measure up or “stack up” to other courts. Many court executives may prefer not to know how their courts compare. After all, as long as you don’t know how your court’s performance compares, and as long as nobody else knows, there’s no pressure for change. Even courts that are happy to learn how to improve their own performance may balk at having their performance compared with other courts.

While unfavorable comparisons are inevitable in performance measurement, the negative impact of such comparisons on individual courts need not be. Limitations of performance measurement can be minimized and attendant risks controlled. An AOC, for example, can shield individual courts from the negative impact of cross-court comparisons on a measure of court user-citizen satisfaction by reporting only state median, top (90th percentile), and bottom (10th percentile) performances. Directed by a general policy favoring self-improvement, the identities of individual courts would be revealed only by agreement with the individual ourts.

Comparisons among courts on valid and meaningful performance measures must be made – and seen to be made – in their political and practical contexts. Performance measurement, by definition, invites both favorable and unfavorable comparisons.

Why would an elected court clerk, for example, embrace cross-court comparisons on a measure of case file readiness and integrity (i.e., the proportion of case files that are delivered when requested and are accurate and complete)? Undoubtedly, the court clerk correctly views his or her personal performance and, by extension, the performance of the court clerk’s office as something to be assessed primarily by the electorate. While the court clerk’s reluctance to support comparative performance measurement may be based in historic differences in methods of truth-seeking, the issue is as simple as this: Why upset the apple cart? If he or she is a current office holder, the verdict on the performance of the clerk’s office already has been rendered. What is to be gained from comparative performance measurement, particularly when focused on a measure that is at the heart of work of a court clerk’s office? What if the court clerk’s office is identified as a relatively poor performer? Who will have access to the results of the comparative performance measurement? How can the court clerk’s office be insulated from unfavorable press coverage? These are not unreasonable questions.

But Jonathan Walters accurately describes the bottom line when he writes this about so called unfair comparisons in government performance: "The fact is, such issues as who has the most efficient social service system, the smartest kids, the best cops, the quickest snowplows, the cleanest drinking water or even the most reliable street lighting are of intense interest to citizens. And pretty soon jurisdictions not producing performance data in such areas are going to be asked why they’re not."

Reason 6: Performance Measures Will Be Misused

Like any tool, comparative performance measurement must be used correctly to be effective. As part of any performance measurement, efforts should be made to ensure that comparisons are complete, that they are based on consistent operational definitions and accurate calculations of performance measures, and that they rely on high quality data. These requirements may seem so obvious on their face that it hardly seems necessary to describe them as shortcomings of performance measurement. However, what may appear obvious at first glance becomes more subtle and possibly problematic upon closer look.

For example, differences between two courts -- court A and court B -- in the operational definition and calculation of the measure of the age of pending caseloads can lead to incomplete or invalid comparisons. Both courts may define their pending caseloads processing consistently in terms of the total number of elapsed days from filing to entry of judgment in a case. But only one court may take into account the days that cases are inactive. That is, court A makes no distinction, whereas court B makes a clear distinction, between those cases that move through the court without interruption, on the one hand, and episodic cases that may have been placed on inactive status for a variety of reasons (e.g., a defendant absconds and a bench warrant is issued for his or her return), on the other hand. Court A includes in its calculation of elapsed time the 60 days a defendant is out on warrant, for example, whereas court B excludes those 60 days.

Therefore, the differences between the courts in the age of pending caseloads (as well as the speed of case processing) may be due solely to inclusion or exclusion of the time a case is inactive in the calculations of measures of the age of pending cases, not to any differences in the efficiency of the courts’ case processing.Making comparisons of these two courts on the measures requiring the calculation of the ages of pending and completed case can be still be instructive, but doing so without taking into account the differences in definitions and calculation of the performance measure would paint an incomplete and inaccurate picture, and constitute a misuse of comparative performance measurement.

Reason 7: Performance Measures Will Be Used to Hurt, Not Help

This argument puts a sharper point on some of the preceding reasons no to measure court perfromance. The traditional command-and-control managerial mindset in many courts is: “You screw up, you fall short, and you get punished.”

Why should we get into performance measurement if the media will completely misinterpret the data and proceed to beat the court black and blue on the evening news and on the front pages of the local paper? This is a legitimate worry.

Maybe it might be a good idea to start thinking less cynically about what performance measurement might do for you, rather than to you. Jonathan Walters is absolutely brilliant on this point. He correctly notes that it’s a mistake to assume that a poor showing on a performance measure will only lead to a “public upbraiding for shoddy work, as opposed to being a catalyst for some reasoned discussion and debate about how to improve performance (in addition to perhaps winning additional resources).” Good managers, he says, turn the tables on bad performance numbers.

He cites the example of Charlie Dean, chief of police in Prince William County, Virginia, an example that should ring true for court managers who have put the measure of case clearance to good use. After getting criticized for his department’s poor showing in clearing cases, Dean turned the tables by using the same performance data to show that his department’s performance was quite good considering that his was the most understaffed departments in the region. The key to his success was that he didn’t attack performance measurement as unfair. Instead, he used performance measurement as a useful management tool and used the performance data to get what he needed.

Reason 8: Performance Measurement Is Good for Problem Diagnosis But Not Cure

This argument stems from the belief that performance trend data, by themselves, do not tell us why things are different, only that they are different.

True enough, as a general rule, comparisons of performance data – whether made among different courts or within a court over time or among different units of a single court – do not tell us why the differences occur. They will not, by themselves, identify causes, provide a prescription for how to improve performance or tell us why it may not be at the level we would like.

Of course, comparisons can help pinpoint the location of trouble spots that may suggest why particular units of a court or two courts differ in their performance ( e.g., when the civil division of a court declines dramatically in a survey of employee engagement compared to the other divisions of a court). However, without additional inquiries like those described for determining evidence-based best practices, performance measurement does not tell us why performance is relatively high, good or bad relative to other courts.

Before conclusions about differences in performances are made, possible differences that are not attributable to performance difference in the courts must be explored and, if possible, eliminated. Such differences may include the inconsistent definitions of data elements, and different counting and calculations mentioned above. As noted by the Urban Institute, who has done as much as anyone to advance performance measurement in the public and non-profit sectors, even with the most careful effort to collect consistent comparative performance data, such variations will inevitable occur. Other differences not attributable to differences in performances include demographic or regional differences, changes in resources available, and differences in the number or characteristics of cases filed.

Reason 9: Performance Measurement Is Too Expensive

This is the worry that performance measurement takes too much time, effort and money. Making comparisons with other courts, for example, adds cost and effort to “internal” performance measurement that is restricted to one court. Comparable courts and performance measures must be identified, and data must be collected, analyzed and reported. This additional cost and effort must not only be well understood but also weighed against specific benefits.

Court leaders and managers, as well as the leadership of the court's justice system partners, must have both clear expectations and political will to secure the necessary resources to support effective performance measurement. While one person -- the head of a court's research and planning division or, in smaller courts, the court administrator -- is likely to lead the process of planning and implementing the performance measurement system, a work group of several individuals will need to be involved over a period of months. Multiple stakeholders inside and outside of the court must work together to reach agreements on key success factors and performance measures aligned with the court's mission. Both the development and use of the performance measurement system must be supported by financial and political resources. As Kathryn Newcomer, professor of public administration at George Washington University, warns: "As additional yet uncompensated work, data collection to support performance measurement will simply not get done. Authority and resources must accompany responsibility for performance measurement."

Given this caveat, it is important to stress that the cost and effort of performance measurement, like any initiative, can and should be kept to a minimum. Performance measurement need not be prohibitively expensive. New pieces should be built on the base of existing parts, processes in place in one or more courts or in agencies allied with courts should be extended to other courts, the scope of the effort should be made to fit needs, automated information systems and other technology should be explored to minimize costs, and finally, specific benefits and payoffs should be weighed against specific costs.

Reason 10: Performance Measurement Is a Vulgarization of Jurisprudence

Professor Godfrey St. Peter, the protagonist in Pulitzer Prizewinning author Willa Cather’s 1925 novel, The Professor’s House, resisted with all his might what he saw as the new commercialism at the time, the aim to “show results” that he saw as undermining and vulgarizing education in his Midwestern university.

Harried judges, as well defense attorneys and prosecutors, facing overwhelming caseloads, joke about “McJustice” and being on an “assembly line.” “I don’t much care how fast the training is travelling, if it’s not going where I want to go,” one judge quipped.

This reason for not embracing performance measurement may not withstand close scrutiny but it, nonetheless, resonates with many judges. It is their mental model of the whole enterprise of “prying results from everything we do” and it shapes their resistance to performance measurement.

Rather than dismissing or dispelling this mental model of “McJustice,” a response to Reason 10 should counter the notion that “as fast as possible” and “as cheap as possible” is a not requirement of court performance measurement. To the contrary, a “balanced scorecard” of court performance measures might well demonstrate that “McJustice” has its price in declining trust and confidence in the courts and disengagement of court employees who see the court losing respect in the community.

The Bottom Line: Performance Measurement Is No Longer an Option for Courts

Court performance measurement inevitably suggests impending change that must be managed effectively. Powerful mindsets or mental models may impede this change. Clearly, courts as learning organizations must continually clarify and improve the mindsets and mental model of court performance measurement to ensure its success in court improvement. For some, it may be as easy as striking the word “performance” from discussions and speaking instead of “indicators of service quality" or "adherence to mission,” or some phrase that moves the focus from personal to organizational responsibility and accountability. For other courts, it may take much more than language to alter powerful mental models inconsistent with performance measurement.

In any event, rather than filibuster against it, it is best to acknowledge the limitations of performance measurement, strive to minimize them in specific ways, and demonstrate in specific terms that the benefits outweigh the real risks that these limitations may pose for individuals and courts. Chances are that performance measurement won’t be an optional exercise for courts for too much longer.

See also:

Getting Started with Performance Measurement – Breaking Down Resistance, Made2Measure, December 5, 2005

Comparing Apples and Oranges ....And Learning From It, Made2Measure, April 9, 2006

Fear of Misuse of Cost Per Case Measure, Made2Measure, February 1, 2006

“How Do we Stack Up Against Other Courts? The Challenges of Comparative Performance Measurement,” The Court Manager, Vol. 19, No.4 (Winter 2004-2005)

Independence, Accountability and Performance Measurement, Made2Measure, September 29, 2005

Top 10 Reasons for Performance Measurement, Made2Measure, September 26, 2005

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2008. All rights reserved.

Micromanagement Disengages Employees

2008-11-13T13:52:00.000-05:00

Micromanage, v.t., - to manage or control with excessive attention to minor details.

The October 21 Made2Measure post (Employee Engagement: Managing the Millennial Generation in the Workforce), explored how the employee engagement survey developed by the National Center for State Courts and CourtMetrics, for both trial courts (see CourTools Measure 9 ) and for appellate courts (see Measure 7 at http://docs.google.com/Doc?id=ddc3k4gt_14cpvjn2c2), can help court managers engage “millennials” – a new crop of young people in the work force who were born between 1980 and 2001.

This post explores how the survey may help to reverse the negative effects of micromanagement.

The survey uses a self-administered questionnaire to assess the engagement of the court's workforce and the quality of the relationships among its employees, especially those between managers and subordinates. It asks respondents to rate their agreement with each of 20 statements on a five-point scale from “strongly disagree” to “strongly agree.”

The measure produced by the survey – percent of engaged employees – is not only a proxy for a vital sign of a court’s health, it is actionable immediately in terms of specific improvements in employee engagement and performance. As noted in the October 21, 2008 post dealing with the millennial generation of court employees, effective performance measures like employee engagement tend to drive success. They are clear, focused, and actionable. They serve both as incentives and practical tools for improvement. The very act of measurement – asking the right questions -- itself will trigger positive action.

Micromanagers Diminish Employee Performance

How can the results of the survey help court managers avoid micromanagement and to loosen the tight and unnecessary control that robs subordinates of the freedom to find solutions to problems? Employees are often best suited to identify problems and to find solutions. Everyone loses if they’re discouraged from doing so.

A micromanager delves into too much detail, interrupts subordinates at every turn, and interferes with work getting done. Rather than giving subordinates the responsibility and freedom to do their jobs, micromanagers constantly look over their shoulders and closely monitor every move they make.

This obsession with detail and control, and the unwillingness to trust, causes resentment, affecting staff performance in a negative way. It produces the opposite of engagement - disengagement - a diminished emotional connection that a subordinate feels for his or her department, or the court as a whole, that influences him or her to exert much less discretionary effort to achieve success.

Using the Survey Results

Employee engagement survey items to watch for:

1. I understand what is expected of me.
2. I am kept informed about matters that affect me.
3. I am able to do my best every day.
8. I have the opportunities to express my opinion about how things are done in my division.
11. I am encouraged to try new ways of doing things.
14. I feel valued by my supervisor based on my knowledge and contributions to my department, unit or division.
15. I feel free to speak my mind.
19. I am treated with respect.

These items of the Employee Engagement Survey appear particularly sensitive to problems of micromanagement.

In a recent consulting visit I made to a large urban court, an assistant civil case coordinator confessed to me that he had responded with “disagree” or “strongly disagree” to all of these all eight items when he completed the latest survey done by the court. Particularly illuminating for me was his revelation that he strongly agreed that the court is respected in the community (Survey Item # 9) and that he still was proud that he worked in the court (Survey Item # 20).

Though he did not use these exact words, it was clear to me that he was no longer emotionally connected to his work in the civil division, and no longer was willing to exert much discretionary effort to help the court succeed. Sure, he would do as he was told, but not much more. “Why bother,” he told me.

“I really don’t know what he wants anymore,” he complained about his immediate boss. “He interrupts my conversations with my staff, points out little mistakes all the time, lectures me, and makes mundane decisions for me several times a day,” he said. “I’ve stopped making suggestions because I’m just going to get shot down.”

Is this employee’s disengagement an isolated case? Is it indicative of the all the employees in the civil division? The entire court?

It turns out that the assistant civil coordinator was speaking for his entire division. The division was underperforming relative to the court as a whole. As a proxy for the health of the division, this measure did not look good.

The civil division’s average scores (percent of employees who agreed or strongly agreed) were the lowest scores among all nine divisions of the court, as was the overall average across all 20 items. The responses to the seven items noted above were particularly at odds with those of the rest of the court.

It turned out that the director of the civil division already had some insight into the problems he faced. He knew that his staff was growing complacent and that his increasing heavy handed intervention was not making things better. He just didn’t know what to do.

It took a push from the court administrator to get him to loosen his control over his staff. Together, during a regularly scheduled meeting focused on performance measurement and management, they reviewed the survey results, analyzing each item relative to the court as a whole and the other divisions. The items with the lowest relative scores formed the basis for breaking his habit of micromanagement.

The civil division’s low score on Item # 11(I am encouraged to try new ways of doing things), for example, prompted the court administrator and the director of the civil division to formulate several strategies for giving the civil division employees more decision-making power and encouraging questions and suggestions. The court administrator encouraged the civil division director to speak to the director of a division with the highest score on this item and question her about practices that she uses to give her employees more decision-making authority.

Insight and Understanding

The performance measurement and management process, like the dialogue of the court administrator and his director of the civil division, uses numbers to provide understandable and comparative results, but it is ultimately not about the numbers. It is about providing perceptions, understanding and insight. In the final analysis, it is not the measurement results that are important but rather the questions that the results compelled the court administrator and the civil division director to confront:

1. How well is the civil division doing?
2. What is the division’s current performance level relative to a baseline? What is the current performance level compared to the court’s established upper and lower “controls” (e.g., performance targets, objectives, benchmarks or tolerance levels)?
3. How will it perform over time (trends)? Is performance better, worse or flat? How much variability is there? Are the same effects occurring in other divisions? The court as a whole?
4. Why is this happening (analysis and problem diagnosis)? What happened to make performance decline, improve or stay the same. What are some credible explanations? Is micromanagement the problem?
5. What should the civil director attempt to do to improve performance levels (planning)?
6. What actions should be started, continued or stopped as a result of what the measurement results reveals (strategy)? What should be done to improve poor performance, reverse a declining trend?
7. What performance targets and goals should be set for the future? (goal setting)?

The willingness and capacity to address these questions systematically and continuously -- for core court performance measures like employee engagement -- and to learn and to adapt on the basis of the results, are hallmarks of a high-performing court.

For more on employee engagement, see also:

Employee Engagement: Managing the Millennial Generation in the Workforce, Made2Measure, October 21, 2008
Measuring and Managing Encounters of Court Users and Court Employees, Made2Measure, February 24, 2008
In Praise of Employee Satisfaction, Made2Measure, November 22, 2006
Friendships in the Workplace Good for Court Performance, Made2Measure, August 14, 2006.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2008. All rights reserved.

Employee Engagement: Managing the Millennial Generation in the Workforce

2008-10-21T15:24:00.000-04:00

Effective performance measures drive success. They are clear, focused, and actionable. They serve both as incentives and practical tools for improvement. Not uncommonly, the act of measurement itself will trigger positive actions.

The 20-item court Employee Engagement survey developed by the National Center for State Courts for both trial courts (see Measure 8 of the CourTools) and for appellate courts (see Measure 7 at http://docs.google.com/Doc?id=ddc3k4gt_14cpvjn2c2) is a measure that fits this bill.

Employee engagement is a constant challenge for court managers. This challenge is even more daunting for “millennials” – a new crop of young people in the work force who were born between 1980 and 2001. Court managers will need them for succession planning as retiring baby boomers leave their positions.

Trouble is that the general perception of the millennial generation seems to be that it has great – and sometimes unreasonable -- expectations. These young workers tend to be more opinionated and less timid than older workers about challenging authority.

In The Trophy Kids Grow Up: How the Millennial Generation Is Shaking Up the Workplace (Jossey-Bass, 2008), Ron Alsop describes millennials’ workplace attitudes and what makes them tick. He says that they were coddled by their parents. They have a strong sense of entitlement and expect much more and sooner than older workers. In an October 21, 2008 article in the Wall Street Journal adapted from his book, Alsop says that millennials are criticized for demanding too much too soon. “They want to be CEO tomorrow,” is a common refrain from recruiters.

Alsop’s highlights three ways to help managing millennials easier. These three ways and other he mentions correspond closely to three items in the Employee Engagement survey.

First, Alsop explains, millennials want things spelled out clearly. Managers should be very clear in explaining even mundane job responsibilities and how meeting them will pay off. Such actions are clearly prompted by responses to Item 1 of the survey (“I understand what is expected of me”). Taking the survey may itself help reduce frustrations by communicating that managers are committed to clearly defined rules and expectations.

By adding age to the response categories in the survey that identify respondents work location, court managers would be able to ascertain whether millennials in the court workforce differ in their responses with older workers, where in the court the difference may be more pronounced, and whether the differences warrant immediate actions targeted on millennials.

Second, because of their heightened expectations, managers may need to exert greater efforts in explaining to millennials how their job is meaningful and critical to the court’s mission. Here again, the Employee Engagement survey is right on target with Item 12 (“I understand the connection between the work I do and the mission and goals of the court”).

Finally, Alsop claims that millennials prefer a work environment in which they feel their views matter. He says that managers should listen to them and give them their say in decisions. Here Item 8 (“I have opportunities to express my opinion about how things are done in my division”) and Item 15 (“I feel free to speak my mind”) are particularly relevant.

Millennials want managers’ attention and guidance. The want engagement. They want to know how they’re doing much more frequently than their older counterparts (see Item 7, “In the last month, someone in the court has talked to me about my performance”).

It seems that new hires of the millennial generation are much like older workers, except more so. Court managers will need to be especially attentive to showing these new hires how their work in the court makes a difference and why it’s of value to them and their community. Opportunity, says Alsop, will be a big retention tool.

A measure of employee engagement should help court managers sort out the differences between millennials and older workers and craft strategies that target young workers in performance areas that matter.

(See also, In Praise of Employee Satisfaction, Made2Measure, November 22, 2006; Friendships in the Workplace Good for Court Performance, Made2Measure, August 14, 2006).

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2008. All rights reserved.

Montana Survey of Appellate Bar and Trial Bench

2008-10-02T14:33:00.000-04:00

The Montana Supreme Court last month became the first high court and only the second state appellate court (see the Oregon Court of Appeal’s 2007 Bench and Bar Survey) to survey members of the state’s appellate bar and trial bench about how well they believe the state high court is performing. In the spirit of transparency and accountability, it made a summary of the survey results public almost immediately.

As explained by Montana Chief Justice Karla M. Gray in a cover letter posted on the Supreme Court’s website yesterday ( see Montana Bar and Bench Survey Results), the Court asked nearly 1,000 appellate lawyers, as well as all of Montana’s District Court Judges and the University of Montana Law School teaching faculty, for their thoughts on the Court’s performance.

Respondents rated the Court’s performance in areas central to its primary obligations, including whether the Court’s decisions are based on facts and applicable law, whether the Court’s published opinions explain deviations from established law and the adoption of new developments in law, and whether the Court treats judges and attorneys with courtesy and respect. The survey also inquired about the Court’s timeliness in completing its overall workload and issuing opinions.

On the one hand, over 90% of the respondents believed that the Court does a good job in providing information about its roles, procedures, and operations, and that it treats trial court judges with courtesy and respect. Eight out of ten said that the Court’s published opinions clearly state the appropriate rule of law, identity and apply standards of review, and provide instructions on remand; an even higher number agree that the Court treats attorneys with courtesy and respect. Regarding attorney discipline, nearly eight out of ten respondents agreed that the Court’s attorney disciplinary process is fair and that sanctions imposed on attorneys are proportionate to the misconduct.

On the other hand, survey respondents rated the Court lowest in the area of timeliness, an area that “obviously needs the Court’s earliest attention,” Chief Justice Gray acknowledged.

“The survey results provide us with valuable insights about our strengths and weaknesses,” Chief Justice Gray said in her letter. “We are convinced that by asking ‘How are we doing?’-- and learning from and acting on the responses -- the Montana Supreme Court will improve and provide an even stronger system of justice for Montana.”

The survey used by Montana is a close adaptation of Measure 1 Constituent Survey prescribed by the Appellate CourTools, currently under development by the National Center for State Courts (see working paper available at http://docs.google.com/Doc?id=ddc3k4gt_14cpvjn2c2 ). Survey items are derived, in part, from published performance standards that are applicable to every state appellate court system including the standards articulated in the Appellate Court Performance Standards (1995) and the Appellate Court Performance Standards and Measures (1999) developed by the Appellate Court Performance Commission and the National Center for State Courts.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2008. All rights reserved.

This Just In: Performance Measurement Works

2008-09-24T18:53:00.001-04:00

Do management techniques like monitoring performance and setting targets really work? Most managers are convinced, and those who hire them would like to think so. Where's the evidence?

The first-of-its kind study by researchers from Stanford, the London School of Economics and the consulting firm McKinsey & Company suggests the answer is yes (see Scott Thrum’s September 8, 2008, Wall Street Journal column “Theory & Practice: The “Same 01” Is Actually Good Enough for Many”).

This is good news, especially for performance measurement. Like management in general, performance measurement needs more than anecdote to assure its widespread adoption, especially in the courts community. Unlike management in general, performance measurement and management techniques are not broadly accepted and are still widely viewed as innovative and experimental.

The study, including more than 4,600 midsize factories in 12 countries, is based on responses to surveys of plant managers and examination of financial data. Survey questions focused three main areas: management of operations, performance and people. The researchers asked roughly 50 questions such as “How do you track production performance?” Answers were scored on scale form one to five. As reported by Thrum, a plant with multiple computer screens displaying real-time production totals and progress toward target performances was scored a five. On the other hand, a plant whose managers said he tracks performance only when plant output drops was scored a one.

Generally, the study reveals that the highest scoring companies were also the most productive and profitable. We are now safe to assert that research supports the inclusion of performance measurement and management as a part of good management practice linked to success.

It seems to be that good management practice is good management practice,” says Stephen Dorgan, a co-author of the study.

And, again, that’s good news.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2008. All rights reserved.

Monitoring and Eliminating “Never Events” in Court Administration

2008-09-23T07:56:00.000-04:00

The Centers for Medicare and Medicaid Services recently made “never events” – so called because they should never happen – a prominent part of its performance measurement and management policy for U.S. hospitals. The concept of “never events” is yet another example of the nation-wide movement to reform health care by performance monitoring, analysis and management (see Pursuing Perfection – A Lesson from Health Care, Made2Measure, November 1, 2006).

Starting in October, Medicare will stop reimbursing hospitals for treating device related infections, urinary tract infections, and surgical infections after orthopedic and heart surgery. Why?

Because they are “never events” that should never or rarely ever happen. Because there is proof that nearly all of these hospital infections that sicken and kill millions of patients a year are avoidable when hospital nurses and doctors clean their hands, decontaminate medical devices and instruments, and take other relatively simple preventative measures. Because it is a decisive step toward encouraging better care and lower overall health care costs for patients and taxpayers. And because, according to Ed Levern (Modern Healthcare, vol.31, no. 32) way back in 2001, “never events” would provide a standard for states to use in refining or developing measurement systems for collecting adverse medical-event information. “Hospitals should be required to report certain medical errors to publicly accessible, state-run databases to create a level of accountability now lacking in the healthcare system,” Levern writes.

Preventable and Unacceptable

Until recently, hospitals and health care providers considered drug-resistant staph infections related to the use of certain devices like catheters and breathing tubes, urinary tract infections, and surgical infections after surgery unavoidable risk. Patients and taxpayers absorbed the costs of these risks.

Well, guess what? Most of these infections are preventable. All it takes is the will, proper hygiene and other relatively simple preventative measures. We have the knowledge and capacity. That’s why the Centers for Medicare and Medicaid Services call them “never events.”

According to the August 2008 issue of Nurse Week, more than 30 million urinary catheters are inserted into U.S. hospital patients each year, and an estimated 10 to 30 percent of them develop catheter-associated urinary tract infection (CAUTI) that are largely preventable using simple proven methods like timely removal of catheters. Central bloodline infections caused by contaminated tubes routinely inserted into hospital patients’ veins are preventable by rigorous hygiene practiced by doctors and nurses. Betsy McCaughey, the former lieutenant governor of New York State and chair of the Committee to Reduce Infection Deaths, reported in the August 14 edition of the Wall Street Journal (“Hospital Infections: Preventable and Unacceptable”) that New York City’s Beth Israel Medical Center has not had a central line blood infection in its cardiac intensive care unit in over 1,000 days.

What Works? Return on Investment (ROI)?

Let me try to make the connection to what we do in the courts.

Identifying the right performance measures for courts and making sure they are used effectively can be translated operationally into three requirements and three phases of development:

o Identifying and developing the right performance measures.
o Ensuring that the right measures can be delivered to the right people in the right way and at the right time.
o Integrating the measures with key management processes and operations including budgeting and finance, resource and workload allocation, strategic planning, organizational management, and staff development. Performance measurement has to be built in a positive way into the everyday workings of the organization.

We are past the tipping point of meeting the first requirement. Hundreds of courts in two thirds of the states – including trial, appellate and limited jurisdiction courts -- have made significant strides in identifying what they consider the right performance measures. In today’s accountability-driven court environment, performance measurement is no longer a novel idea. Enter “CourTools” into a Google search and you will get thousands of hits as compared to a few dozen in early 2005 when the CourTools measures were first introduced by the National Center for State Courts.

The problem is that the movement toward court performance measurement and management has slowed at the second and third requirements. In addition to the expected undesirable inertia that comes from aversion to any change, the movement is bottlenecked at questions of return-on-investment (ROI): OK, court leaders and managers are asking, now that we have the right measures, what precisely will performance monitoring, analysis and management do for us? What works? How do I leap from measurement to management? Show me the way.

I have until very recently dodged these RIO questions and argued successfully that courts should proceed in meeting the second and third requirements even though the ROI is unclear, even though what works is not yet evidence-based, and even though the leap from measurement to management is not yet mapped. I’ve said that we are developing the right hammer and the right saw -- the right tools -- but we have yet to decide what to build and what to fix with these tools. That will be up to you and, trust me, you’ll do OK making the ROI leap from measurement to management when the time comes.

Well, that argument has worked in the past when court leaders and managers were still intrigued with identifying the right measures, when they thought they might not be able to do it. But now that they know that they can identify the right measures, many are reluctant to invest in performance dashboards and business intelligence to get the measures into the right hands in the right way and, further, to hard-wire performance measurement into the very DNA of court management and operations until they get a clear bead on ROI.

A Model for Court Performance Excellence?

“Never events” may be what we need to map the leap from performance measurement to management and to articulate ROI clearly.

Effective performance measures are easy to understand and explain to others. By their very name and brief definition alone, they evoke one or more values or fundamental obligations of a court and they suggests a compelling and short-hand concept for articulating the ROI of performance measurement and management. (Side Note: I have tried, with only moderate success, to rename court performance measures, time to disposition, for example, using more descriptive labels such as on-time case processing.)

“Never events” connote accountability and urgency. The term suggests the kind of monitoring, analyzing and managing of unwanted variation in performance that are at the heart of effective court performance measurement (see Undesirable Variation in Court Performance, Made2Measure, April 23, 2007). It has the kind of striking attention-getting simplicity and evokes the level urgency that most court performance measurement concepts still lack.
Would court systems and individual courts – who may not otherwise yet be able to articulate ROI in performance measurement – be able to reach consensus among court leaders and managers, justice system partners, and other stakeholders about preventable adverse events that should never occur and to define them in a way that, should they occur, it would be clear what had to be monitored, analyzed and managed?

In 2002, the National Quality Forum (NQF), a not-for-profit created to develop and implement a national strategy for health care quality performance measurement, published a report, Serious Reportable Events in Healthcare, which identified 27 adverse events that are serious, largely preventable, and of concern to both the public and healthcare providers. The items on the list are events are:

· Of concern to both the public and healthcare professionals and providers.
· Clearly identifiable and measurable, and thus feasible to include in a performance measurement reporting system.
· Of a nature such that the risk of occurrence is significantly influenced by the policies and procedures of the organization.

Would we be able to do the same for court administration? What are “never events” in court administration that would meet these criteria? For example, cases that exceed a critical number of elapsed days from filing without disposition or resolution? Clearance rates for specific case types that fall below 95 percent? Ten percent or more “strongly disagree” responses to any item in a surveys of court users at the trial court level or a survey of the appellate bar and bench at the appellate court level, or a survey of court employees?

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2008. All rights reserved

The Real Start-Up Costs of Performance Measurement

2008-09-15T13:20:00.000-04:00

Performance measurement does not cause inefficiencies and poor practices. It just highlights them for improvement. The cost of fixing them should not be counted against the start-up costs of performance measurement and management.

This point often gets lost as courts and court systems embark on initiatives to build performance measurement and management systems. If nothing else, this point needs to be reinforced to blunt a favorite argument against performance measurement initiatives – it all takes too much time, effort and money (see Eight Reasons Not to Measure Court Performance, Made2Measure, April 5, 2006).

Some things just need to be fixed anyway.

For example, as courts consider the required elements of measures like case clearance rate, on-time case processing, and age of pending caseload, they need to define, in no uncertain terms, such things as: (1) what to count and what not to count; (2) how to classify what they count; (3) when to start and stop the clock; (4) when a case is opened and closed; (5) when a case is considered active and inactive; and, (6) what might be an appropriate time reference point for assessing timely performance. Inevitably, they will encounter inefficiencies and poor practices: poor classification of cases, lack of uniformity in case filing, automated case processing systems with inadequate data “reporting” functionality, and a host of other problems that existed and persisted long before any performance measurement initiatives were launched.

To be sure, all this takes time, effort and resources. But these costs should not be counted against performance measurement per se. As Judge Katherine R. “Kitty” Curtis, chair of Montana’s District Court Performance Measurement Advisory Committee (DCPMAC), noted in a meeting of DCPMA in Helena earlier this month, the task of defining the data elements of case processing measures is “both intellectually challenging and tedious.” At the same time, she recognizes that this task needs to be done eventually to ensure uniform case filing throughout the state, whether or not a performance measurement and management system is built.

No doubt, costs specific to performance measurement should be weighed against benefits. But some things uncovered while taking the steps for performance measurement just need to be done regardless of planned efforts for monitoring, analyzing and managing performance. The costs of doing these things should not be used as the reason why court should not measure its performance.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2008. All rights reserved

The End of Science As We Know It

2008-07-06T13:25:00.000-04:00

Add this big idea to the concepts that underlie modern performance measurement and management:

We are seeing the end of traditional research and scientific method – hypotheses testing and experimentation -- as we know it. And performance measurement is there at the right time and place.

From Kilobytes to Petabytes

Today we’re able to capture, store, and make sense of massive amounts of data. We’ve gone from kilobytes to megabytes, from terabytes to the Petabyte Age. A petabyte is a measure of memory or storage capacity that is 2 to the 50th power bytes, or the equivalent of 20 million four-drawer filing cabinets full of text.

Infinite storage or almost no storage necessary. Unlimited processing capacity. Early-warning sensors and measures everywhere sending automatic alerts from intelligent agents. No structured databases necessary. Just petabytes of information flowing through pattern-matching and trend-watching algorithms.

It seems we really don’t need hypotheses and theories anymore – i.e., informed hunches and theoretical models about how the world works -- as a first step in finding the truth in much of what we do. And why sample data when we can have everything at our fingertips and process it in ways impossible before?

We just need to tap into the data with search technology and discover patterns and trends that will reveal knowledge. More and more, we’re betting that more and better data, plus better measurement and analytical tools, will win the day.

In its July 2008 cover story, Wired noted that the quest for knowledge used to begin with grand theories. Now it begins with massive amounts of data and a search for clues – any clues. Increasingly, we can succeed without theories, models and hypotheses.

Google does not really need to know (or care) why a particular website is better than another. If its statistical algorithm says it is, that’s good enough. Throw the data into the biggest database we can muster and let our measurement and analytical tools find patterns where the traditional science of hypotheses testing cannot.

In the wake of the explosion of E-discovery, lawyers no longer dig through warehouses full of paper documents armed with theories of the smoking gun. Instead, they hire data-miners who use search technologies to sift through huge numbers of electronic records. According to Wired, E-discovery vendors who scan, index and mine data pulled in $2 billon in 2006.

“Who knows why people do what they do?” notes Chris Anderson, editor in Chief of Wired, in his article, “The End of Theory.” “The point is that they do it, and we can track and measure it with unprecedented fidelity.”

Performance Measurement Riding the Trend

Turns out that modern performance measurement has been riding this trend all along. The question “Why is this happening?” comes after performance data are monitored and analyzed, and after detection of variations, patterns and trends in the data that might indicate a problem or, better yet, solutions. No prior hypotheses or theories are needed. As a matter of fact, they may get in the way.

The hallmark of scientific research, as distinguished from organizational performance measurement, is rigorous hypothesis testing and experimentation based on theory. A hypothesis is a small slice of a theory that is tested within a narrow range of data to support or refute the theory. In contrast, performance measurement is the monitoring, analysis and management of central tendencies and variations in the broadest possible spectrum of data (e.g., all the case processing data in an automated management system) prior to any theory, hypothesis or explanation of the variations from baselines.

Though I am a social scientist, I have long argued that courts and court systems should favor performance measurement over research (see Made2Measure, Courts Have No Business Doing Research, October 15, 2007). The theoretical foundations and methodology of the disciplines of performance measurement and research overlap, but they are very different in important ways.

Performance measurement is court business. It is the continuous process of measuring performance variations from central tendencies in performance, asking “Why is this happening?” and then looking further into the data for patterns and trends. Scientific research (including evaluation research, on the other hand, typically begins with theories and hypothesis testing about “Why things might be happening?” Court leaders and managers seldom have the time and the money for this. Now it turns out it may not be necessary.

Just show me the data. Never mind your theories and hypotheses!

Here are six other new ideas related to modern performance measurement that I’ve explored in previous postings:

The flattening of our world and the end of central planning (see Made2Measure, The Real Promise of Performance Dashboards, May 9. 2007).
Emergence of radical transparency (see Made2Measure, Radical Transparency, June 4, 2007; and, Transparency in Practice,
February 12, 2007) and open book management (see Made2Measure, Open Book Management in the Courts, September 30, 2006
A building consensus that results (outcomes) matter most (see Made2Measure, A Preference for Outcome Measures, September 28, 2005).
The notion that we – yes, the courts – are in the business to satisfy those who use the courts (see Made2Measure, What Is Our business? Who Are Our Customers?).
The power of controlling variation (see Made2Measure, Undesirable Variation in Court Performance)
The proposition that performance measures provide the focus and clarity that are the preoccupations of effective leaders (see Made2Measure, Performance Measures = Clarity, August 5, 2007)

For the latest posts and archives of Made2Measure click here.©

Copyright CourtMetrics 2008. All rights reserved.

The Declaration of Independence Still Inspires

2008-07-05T19:43:00.000-04:00

Every Fourth of July – yesterday was no exception -- I print out a copy of the Declaration of Independence. I read and study it.

Without fail, I find some nuance and current relevance that I missed before. I always get inspired about my work in the courts and, of course, my country. (I’m an immigrant, born to German parents, and was educated in this country. I became a proud naturalized citizen of these United States, along with my beaming parents, in a ceremony in Elizabeth, New Jersey, the memory of which still gives me goose bumps.)

The famous words of the second paragraph of the Declaration of Independence, an action of the Second Continental Congress, July 4, 1776, signed by people whose names are on the street signs of Williamsburg, Virginia (where I live) and every city and town in the United States, blow me away every time I reread them:

WE hold these Truths to be self-evident, that all Men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the Pursuit of Happiness – That to secure these Rights, Governments are instituted among Men, deriving their just Powers from the Consent of the Governed.

Whenever I read these words, I marvel at their punch, their brevity and clarity. If nothing else, our Founding Fathers cannot be criticized for being wishy-washy.

Court leaders and managers could do worse than to use these few words as models as they try to do what Peter Drucker says that all successful managers do: define the values, vision and purpose of their organization; determine what results are wanted; define the spirit and culture of the place; and bring people together for joint performance.

Above all, urges management guru Marcus Buckingham, be clear (see Made2Measure, Performance Measures = Clarity, August 5, 2007). I can think of no better prompt for a strategic plan than “We hold these truths to be self-evident.” What “truths” about our courts and court systems would inspire employees and stakeholders alike to exert greater discretionary effort to the cause? Figure it out, and you're on your way to strategic thinking and acting on a large scale.

I was first exposed to the Declaration of Independence in high school and earlier, as most of us were, but I never really paid much heed until much later, when some of the founding fathers of court administration and their disciples (Ed McConnell, Geoff Gallas and Karl Barr, to name just a few) really thrust the document on my attentions in a way that I could not ignore. I am indebted to them.

Happy (Belated) Fourth of July.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2008. All rights reserved.

Smart Meters

2008-06-15T17:17:00.000-04:00

I recently was invited to participate in a new pilot program offered by my electricity provider, Virginia Dominion Power. It’s a part of a national trend to help consumers control their electricity and, at the same time, help utilities cut their operating costs.

Virginia Dominion Power, following the lead of utilities in California, Texas and other states, is rolling out the program to install “smart” electric meters in homes in the belief that they will help cut electricity consumption and reduce the need for new power plants. The smart meters will allow consumers to see, at a glance, how much electricity they are consuming at what cost.

The Critical Peak Pricing Pilot Program is based on two joined approaches increasingly used in the private sector. I predict that these two approaches will soon become standard “best practices” for most monitoring, analysis and management of all kinds of performance, including that of courts.

(1) Bring everyone – customers and employees alike – not just top management, into the discovery and execution of solutions to problems (in this case, prohibitive construction costs for new power plants to provide electricity during temporary spikes in demand).

(2) Use the latest technology to deliver the tools (in this case, critical power usage information displayed by smart meters in real time) to enable anyone who is motivated to do so be part of the solution, and then reward them for their efforts.

The Critical Peak Pricing Program upgrades participants’ electric meter outside of their houses, and the traditional thermostat inside. It replaces them with a single state-of-the-art smart meter inside the house. In addition to the traditional thermostat readings of the current temperature and setting, the smart meter send readings wirelessly and displays current energy use in real time, the cost of that usage, the current pricing levels set by Virginia Dominion Power for various times of the day and year (critical peak, intermediate and non-peak). The meter also notifies consumers of periods of critical demand that strain the area’s power supply.

As a participant in the pilot, you agree to pay a higher rate for power consumed in critical peak periods, which Dominion Virginia Power says are limited to 125 hours per year and will occur in five hour increments during the day. In return, you are charged a lower rate for the power you use during non-peak times. Participants are notified in advance of any critical peaks and alerted by a set of colored lights by the smart meter so they can adjust electricity consumption.

By monitoring, analyzing and managing our energy consumption, my family saves money and helps manage my area’s energy requirements. Virginia Dominion Power is inviting my family to help solve our shared problem and giving us the tools to do it. In theory at least, everyone wins.

Systematically controlling unwanted variation (e.g., spikes in power demand) is not new. (See Undesirable Variation in Court Performance, Made2Measure, April 23, 2007). Using the latest technology of smart meters and performance dashboards that provide very detailed information about performance to anyone in real time is. It’s an innovation that is likely to spread.

As I’ve said before, courts are at the tipping point in the development of performance dashboards and business intelligence as a major factor in the management of organizations including courts. (See The Real Promise of Performance Dashboards, Made2Measure, May 9, 2007; and The Exciting (and Confusing) Court Performance Dashboard Market, Made2Measure, November 14, 2007).

What seems pretty clear -- like the old monthly electricity meter readings which will be relics of the past in ten years or less -- monthly (or even less frequent) written reports on a court’s performance that only top management gets to review and analyze, and perhaps later pass it’s solution to the masses, just don’t cut it anymore.

For the latest posts and archives of Made2Measure click here.

Q & A: How Many Measures Should Be Used?

2008-05-25T10:52:00.001-04:00

Q: How many performance measures should my court be considering? My colleagues have reviewed the ten performance measures of the CourTools developed by the National Center for State Courts, as well as other measures – like treatment court recidivism -- that courts are considering. They all seem compelling but we are concerned that the whole enterprise will just be too much for us. What advice do you have?

A: It is better to do more with less than less with more. First and foremost, how many performance measures your court should develop depends on what matters to you, what you consider important – the court’s key success factors. If these include, for example, access and fairness, citizen satisfaction, expedition and timeliness, community welfare, and employee engagement, you’re likely to need more measures than if you’re only interested in efficient case processing.

Second, there’s no sense in developing measures that will not be used. Generally speaking, it is much better to do more with fewer measures than to do too little with more.

If your court is unlikely to muster the political will and maintain the capacity to fully monitor, analyze and manage ten performance measures on a regular and continuous basis, developing ten performance measures is wasteful effort. You’re much better off with just five, four, three or even just two measures that you can fully exploit.

For example, court user/citizen satisfaction and case backlog – just two measures – can pack a mighty wallop of understanding and insight into your court’s performance that can be translated into effective strategies for improvement. As John M. Greacen noted in an article reprinted in Future Trends in State Courts - 2007, the state of New Jersey has done quite well with a sustained focus on just one measure, backlog. “The focus on backlog,” he writes, “has proved to be wise and effective – in its ability to motivate the courts to improve on that measure, in its positive impact on other performance measures, and in its simplicity and clarity of message to the media and the rest of state government.”

The takeaway message is that of Ockham's razor, the principle attributed to the 14th century English logician William of Ockham. The principle, also known as the “law of parsimony,” is often paraphrased as "All other things being equal, the simplest solution is the best."

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2008. All rights reserved.

Measuring and Managing Encounters of Court Users and Court Employees

2008-02-24T12:37:00.000-05:00

Along with a number of client-partners, I’ve been rethinking the way courts gauge the satisfaction of court users and the strength and engagement of their work force And, yes, how we can improve the court user – court employee encounter.

Quite a few court leaders and managers still regard these lines of performance measurement and management as “touchy feely,” “soft” or “subjective,” despite the overwhelming evidence to the contrary from some hard-nosed researchers and practitioners. But that’s a subject for another time.

Human Sigma

In their book, Human Sigma: Managing the Employee – Customer Encounter (Gallup Press, 2007) – a much expanded version of their groundbreaking article “Manage Your Human Sigma” that appeared in the July/August 2006 issue of Harvard Business Review – Gallup researchers John H. Fleming and Jim Asplund rewrite the rules for how we should measure and improve employee - customer encounters. Though based on extensive research of employees and customers of private companies, Fleming and Asplund have much to say to managers of courts and court systems.

Their work is especially relevant to courts and court systems that assess court user satisfaction and employee engagement using the National Center for State Courts’ CourTools Measure 1, Access and Fairness, and Measure 9, Court Employee Satisfaction, or some variations of these two performance measures.

Although precise counts do not exist, I estimate that today over 100 courts in a dozen states have employed these performance measures. A number of states, including Arizona and Utah, have done so on a state-wide basis. Appellate courts also have gotten into the game. The Oregon Court of Appeals, for example, conducted its first court user survey of members of the appellate and trial bench this time last year and is planning a second survey in the near future. (See a summary of results of the first bench and bar survey on the Oregon Court of Appeals website.)

Six Sigma (from which Human Sigma derives) is a quality management approach pioneered by Motorola twenty five years ago. It became a genuine management movement that revolutionized the way private companies approached manufacturing (mostly non-human) quality by emphasizing the setting of extremely high objectives, collecting data, and analyzing results to systematically control variation and improve manufacturing processes (the Greek letter sigma is used to denote variation from a standard or other reference points).

Six Sigma was inspired by quality improvement methodologies like quality control, Total Quality Management (TQM), and Zero Defects. Motorola has reported billions of dollars in savings from using Six Sigma. Along with Motorola, companies that use Six Sigma methodologies today include General Electric, Bank of America, and Honeywell International.

Fleming and Asplund’s Human Sigma applies the Six Sigma principles and techniques to sales and service organizations relying heavily on the measurement of human -- i.e., employee and customer -- engagement.

New Rule

Human Sigma’s Rule # 1 is that employee and customer experience are not separate entities and, therefore, should be assessed and managed together. Good employees make good customers; you can’t have one without the other, at least not for long.

Because most companies are not organized or prepared to manage these human resources together, Fleming and Asplund suggest that companies may need to reorganize to do so. That’s the organizational part of Rule # 1.

This new rule is especially relevant for the increasing number of courts and court systems that are measuring court users’ and employee satisfaction and engagement using the Measure 1 and Measure 9 of CourTools or some variations. The message is that we must view both sides of the court user – court employee encounter as interrelated and mutually dependent.

While the four of the CourTools measures focused on timeliness and expedition of case processing are lumped together as Measures 2, 3, 4 and 5, and their descriptions are cross-referenced, it is revealing that Measures 1 and 9 are separate and are not cross-referenced. Rule # 1 should apply.

The Court’s Human Sigma

Measure 1, a survey of court users, and Measure 9, a survey of court employees, use a similar metric -- namely, the percent of respondents who agree with statements about their encounter with the court and the court’s treatment of them. Disaggregating the metrics by survey item, respondent demographics, and by characteristics of the encounter (e.g., where it occurred and for what reason) can pinpoint variations in the court user and court employee encounter that may require management attention.

For example, a problem as well a good course of action, may be suggested by the following survey results: a relatively low percent of court users in a particular situation (e.g., appearing as witnesses) who believe that they were treated with courtesy, dignity and respect in a particular court division; and a relatively low percent of court employees in that same division who believe that their colleagues care about high quality. If these results deviate meaningfully from the central tendencies of other divisions or other court users in different situations, an astute manager may identify a problem and a possible solution.

Simply by identifying other divisions or situations with superior results, good managers may be close to a solution to the problem. See Undesirable Variation in Court Performance, Made2Measure, April 23, 2007.

The point is that when court user measures and court employee measures are viewed together as interrelated and mutually dependent, problems are more readily identified and solutions more easily found.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2008. All rights reserved.

Super Bowl Indicator of Performance

2008-02-02T14:04:00.000-05:00

Performance measures are often proxy variables. That is, they are not necessarily themselves of any great interest but they can tell us a lot about a particular thing or outcome. To varying degrees, performance measures are such proxies -- removed and highly simplified version of the outcome of interest.

We use proxy measures in order to make it possible to measure things easily, routinely and at a reasonable cost. The value of a proxy measure is that it is expected to correlate with the desired outcome. Not perfectly, but good enough.

In other words, while some performance indicators may or may not jibe with our common understanding or mental images of the concept or construct under consideration, they indicate its meaning. Cholesterol level is not health. Tree ring widths and ice core layering are not temperature records. Case clearance is not exactly court productivity or efficiency. Recidivism is not community well being.

And – here it goes -- a win by the Patriots over the Giants in the Super Bowl tomorrow is not a downturn in the stock market.

Some effective indicators are so far removed from our understanding of the thing itself, so lacking in face validity, that it’s quirky. That’s how last Tuesday’s Wall Street Journal referred to one of the most accurate indicators of the future direction of the Dow Jones Industrial Average – just plain “quirky.”

William Powers, who wrote the piece in the WSJ, along with Robert Stoval of WSJ Research (who I can only assume is one of the “stat guys” so beloved by us sports fans), remind us that its time for the Super Bowl Predictor of stocks this year. This indicator has called the right direction of the Dow Industrial average following the Super Bowl an astonishing 81% of the time -- 33 out of a total 41 years.

Nitpicks aside – few measures are ever immune to nitpicking – the Super Bowl Predictor says the market rises when an “original” NFL team like the Giants wins, and declines with a win by one of the AFL teams like the New England Patriots. Sorry New England Patriot fans, we like you, we really do, but we simply can’t afford to have the Pats win over the Giants this Sunday. Money is money after all.

So, while you’re watching the Super Bowl tomorrow, think about what it all means. I know it will be hard for us to wrap our minds around the lesson of this post, but remember it’s not just a Game, it’s a proxy for good or bad times ahead.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2008. All rights reserved

Performance Benchmarks, Standards and Goals

2008-01-27T12:33:00.000-05:00

How is your court or court system performing? How should it perform? Answering these two questions requires different methods: the first, performance measurement; and the second, the establishment of standards, benchmarks or goals to serve as norms or models for others.

A court should strive to answer both questions, but it should not delay answering the first because it is uncertain about the answer to the second. Unfortunately, this is not how things have happened.

Points of Reference Versus Standards

Difficulties arise when certain points of reference required to answer the first question are unnecessarily burdened with the weight of serving as performance benchmarks, standards and goals.

For example, the Oregon Court of Appeals measures the timeliness of its processing of land use cases against the point of reference of 91 days. The Court defines the metric of on-time case processing of land use cases as the percent of cases disposed or otherwise resolved within 91 days. Similarly, the Yuma County Superior Court of Arizona uses as a point of reference 60 days for its juvenile delinquency cases, as in the percent of juvenile delinquency cases disposed with in 60 days. The two points of reference, one for a category of appellate cases, and the other for a category of trial court cases, are useful for measuring performance regardless of whether 91 days and 60 days are benchmarks, standards and goals. The two courts can monitor, analyze and manage these performance measures whether or not they have been approved or certified by some authority or governing body.

The design and development of court performance measurement systems is often delayed and impeded when points of reference – 30 days, 90 days or one year -- identified solely for purposes of answering the first question, are put to the difficult test of answering the second question. Consequently, some court performance measurement initiatives have stalled simply because court leaders felt that these points of reference needed to be certified or established as standards before on-time performance data could be collected and reported. In other words, they felt they needed to answer the question “What should the court’s on-time performance be?” before they should try to answer the question “How is the court performing today?” And they were just not ready to do so.

Definitional Problems

To be sure, some of this difficulty stems from a general resistance to performance measurement and management (see Made2Measure posts, Getting Started with Performance Measurement – Breaking Down Resistance, December 5, 2005; and Eight Reasons Not to Measure Court Performance, April 6, 2006). But a large part of the problem is one of definitions. From my point of view, we’ve been fairly loose in our interchangeable use of many terms that have quite different denotations and connotations including benchmarks, standards, goals, targets, time frames, and guidelines. (The same can be said about such terms as mission, purpose, vision, fundamental obligations, success factors, perspectives, major performance areas, strategic goals and so forth – but that’s another topic.)

Two years ago, in a Q & A post about the definition of standard and target performances (Made2Measure, January 5, 2006), I noted that the term standard had been used in two quite different ways, causing considerable confusion. It is best defined, I wrote then, as a special target performance recognized and adopted as the norm by some authorities like the American Bar Association (ABA), the Conference of Chief Justices (CCJ), and the Conference of State Court Administrators (COSCA), or a state administrative office of the courts. The ABA, CCJ and COSCA all published case disposition time standards in 1983 – 1984. The ABA, for example, adopted case processing time standards such as the disposition of 100% of all felony cases in 12 months. A particular court, I continued then, can have various targets of its own that are lower or higher than the standards adopted by external authorities without necessarily rejecting those standards.

Clear enough. Well … not quite.

Twenty years later, the National Center for State Courts’ (NCSC) CourTools Measure 3, Time to Disposition, is defined as the percentage of cases disposed or otherwise resolved within “established time frames.” The case processing standards published by the ABA and COSCA (but not by the CCJ) are cited and recommended as starting points for determining on-time guidelines (yet another new term). Unfortunately, the NCSC is less than clear whether these points of reference – in three short paragraphs interchangeably referred to as time standards, time frames, and guidelines – are to be approved, certified or established by an authority like the ABA or the COSCA, or if they are simply points of reference that define a summary metric.

I personally believe that the ABA, CCJ, and COSCA (and, yes, my colleagues at the NCSC and I who have been central to this confusion) have set back the courts community by at least 15 years by suggesting that a particular point of reference is a standard that a competent court should meet. Because few courts meet this standard, many court leaders argued incessantly about whether such standards were reasonable. Most courts simply rejected this notion.

What court leaders and managers did not do is answer the first question, “How is your court or court system performing?” They did not monitor, analyze and manage their performance. Instead, they debated whether any one of the countless points of reference for numerous case categories and case types should be certified as standards.

The upshot was that many courts just didn't bother figuring out how long it takes for them to process cases. Some concluded that it’s all just numerology, bean counting -- the revenge of the spreadsheet guys!

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2008. All rights reserved

Performance Review Meetings: An Essential Part of Performance Management

2008-01-23T13:24:00.000-05:00

Measures without meetings are meaningless

Performance data that are not used – by definition – are useless. Yet, an alarming number of courts and court systems ignore this simple lesson.

Many individual courts and state court systems have made vast investments in securing the right sponsorships and resources for performance management initiatives, creating the right metrics and standardizing their meaning, and even building pricey performance dashboards and business intelligence technology to deliver the performance data. But, after all that, they fail to ensure that the performance data is actually used to drive strategy and success.

This is a shame because many solutions, like a simple performance review meeting can help to integrate performance measurement into the very fabric of a court’s leadership, management and everyday operations. All it takes the decision to do it. By making performance monitoring, analysis, and management a standing item on the agenda of regular executive meetings, a court executive is signaling that performance is important to the court, and that court executives will be using performance data to manage it on a regular and continuous basis. Department or unit heads could, of course, do the same thing at their level.

A performance review meeting is an essential part of performance management. It establishes for court executives, managers and staff a forum and opportunity to review core performance metrics and subordinate measures, to discover exceptions and trouble spots, and then to explore, analyze and discuss further data that might shine a light on those exceptions and trouble spots; and, finally, to make decisions and to formulate strategies that might clear them up.

Performance review meetings can be conducted at various levels of leadership and management of the court, and can take a variety of forms including monthly executive sessions, weekly unit or division meetings, and informal team gatherings.

As I’ve suggested in pervious posts, performance measurement and management systems are not self-executing (see Made 2 Measure, November 12, 2005). Performance review meetings are a simple way to ensure their execution. For more on performance review meetings, see Implementation: How It Looks When You Get There, Made2Measure, December 13, 2005,

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2008. All rights reserved

Putting Out Fires

2008-01-13T17:21:00.000-05:00

The most important thing a court manager can do is to communicate what the court should accomplish, what results are desired, and why that’s important – the theory of the court’s “business.” Performance management with effective performance measure and indicators – like those of the National Center for State Court’s CourTools -- that are understandable, broadly applicable, uniformly interpreted, and practical to apply, are a court managers best tool for doing this.

Recently, a court manager described his job to me as putting out fires. He said it consumes most of his time and energies. He said it with pride.

This would be a good thing, but only if he were running a fire department instead of a court system. That would demonstrate clarity and strategic focus. After all, no one would question that putting out fires is “mission critical” for fire departments.

Court leaders and managers must focus – and be seen to focus – on the court’s mission and its fundamental obligations. Saying that you have no job to do until you see the smoke and flames is not leadership. Peter Drucker defined leaders simply as people who have followers. An effective leader gets people together for joint performance.

You don’t get people to follow you when you’re either no place or all over the place fighting fires.

Clarity and strategic focus, and a culture of discipline, are factors of effective leadership and high-performing organizations. An executive who is seen as no more than a crisis manager fails to lead, and the organization suffers. It is seen as driven from one crisis to another, but not toward success.

More and more organizations are putting their faith in performance measurement to facilitate clarity and strategic focus. It provides the framework and discipline for leadership. Performance measures serve to align an organization’s efforts to the achievement of its mission.

Effective performance measures are shorthand -- clear, focused and actionable indicators -- invoking fundamental obligations of an organization, its values and strategic goals.

When a court identifies core performance measures, it communicates a clear, simple and penetrating theory of its “business” – its ideals and purpose -- that informs all decisions and actions. As noted in the last posting, the benefits of an effective court performance measurement and management system are the same as those of strategic planning – accountability, consensus building, focus, coordination, control, learning, communication, hope and inspiration.

To identify the right performance measures, a court must address the same fundamental questions about guiding ideals, values, mission, goals and broad strategies as it must address in strategic planning. Unlike most strategic planning, however, performance measurement and management readily translate strategic goals into strategic thinking and acting.

When a manager quells another crisis of the day or the week, and he says that putting out fires is most important, what is the message that he’s sending?

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2008. All rights reserved

Principles of Effective Court Performance Measurement and Management

2007-12-31T11:15:00.000-05:00

The benefits of an effective court performance measurement and management system are the same as those of strategic planning – accountability, consensus building, focus, coordination, control, learning, communication, hope and inspiration. To identify the right performance measures, a court must address the same fundamental questions about guiding ideals, values, mission, goals and broad strategies as it must address in strategic planning.

We count what counts and measure what matters. And what we measure determines what will be considered relevant.

Measurement uses numbers but it is ultimately not about numbers. It is about perception, understanding and insight.[1] It is not the measure itself that is important but rather the questions it compels us to confront.

How are we doing?
How is the court performing?
Where are we now (performance level, baseline)? What is the current performance level compared to established upper and lower “controls” (e.g., performance targets, objectives, benchmarks and tolerance levels)?
How are we doing over time (trends)? Is our performance better, worse or flat? How much variability is there?
Why is this happening (analysis and problem diagnosis)? What happened to make performance decline, improve or stay the same. What are some credible explanations?
What are we doing to improve/maintain (planning)?
What actions and strategies should we start, continue or stop as a result of the measure (strategy)? What should be done to improve poor performance, reverse a declining trend, or recognize good performance?
What performance targets and goals should we set for future performance (goals)?

The willingness and capacity to address these questions – on a continuous and regular basis – to learn and to act accordingly are the hallmarks of a high-performing court. In addition to these principles, effective court performance measurement systems are designed in accordance the following criteria of effective court performance measures:

Outcome Focus. Effective performance measures emphasize the condition or status of the recipients of services or the participants in court programs (outcomes) over that of internal aspects of processes, programs and activities (inputs and outputs) -- that is, they indicate results rather than resources and level of effort.
A Vital Few instead of a Trivial Many. It is better to have a critical few core measures instead of a trivial many. An aggregation, a combination, an index, or a conjunction of a number of measures, variables or aspects of performance may be identified with associated subordinate measures to keep the number of core measures low. Each core measure may sit at the top of a “hierarchy” of related subordinate measures and indicators.
Actionable. Effective performance measures drive success. They are focused, unambiguous and actionable. They serve both as incentives and practical tools for improvement. The key to collecting data for court performance measurement is identifying those performance measures that will actually help to achieve the desired results (i.e., measures that are drivers of success). At best, the act of measurement itself triggers a change.
A Balanced Scorecard. Effective performance measures together constitute a balanced perspective or “balanced scorecard” of performance. They reinforce each other and do not undermine each other.
Practical and Feasible. Effective performance measures are ones that can be taken without extraordinary efforts by the appellate court, ideally as part of everyday management and operations.
Standardization and Consistency Across Entire Court System. Effective performance measures are based on standard definitions, rules and calculations. They are consistent from the top to the bottom and across the court system.
Symbolic. Effective performance measures are easy to understand and explain to others.

[1] Dean R. Spitzer (2007). Transforming Performance Measurement: Rethinking the Way We Measure and Drive Organizational Success (New York: American Management Asssociation).

For the latest posts and archives of Made2Measure click here.

Playing “Gotcha” with Performance Measurement Data

2007-12-26T18:40:00.000-05:00

For court performance measurement initiatives to succeed they must marry proven methods of assessments of the health of a court with a disciplined process for improving it. This may have to be a shotgun marriage. Without it – even in the very early stages of development -- the initiatives will fall prey to the “gotcha” game and be used to undermine the effort and to discredit it proponents.

The Gotcha Game

In this political season, we are all familiar with the ploy in which one candidate (or the media) seeks to catch another in a misstep or flub -- no matter how untruthful or inconsequential the accusation might be. The whole point is to discredit, place blame, embarrass, or otherwise put things in the worst possible light. When this “gotcha” game is played with measurement data it can have disastrous effects on court performance initiatives.

For example, relatively straightforward efforts to monitor, analyze, and assess court citizen encounter using a survey such as that for Measure 1, Access and Fairness, of the CourTools, might be characterized by some as an ill-conceived attempt to put judges on a hot-seat and at risk of loss of judicial independence, status and pay. Unfortunately, this may be already happening in Arizona, a state that has done the most to foster the assessment of the court citizen encounter state-wide.

The Negative Context Persists Unless ….

In my last post (Court Intelligence -- A Matter of Survival, Made2Measure, December 11, 2007), I mentioned the work of IBM’s Dean R. Spitzer, author of Transforming Performance Measurement – Rethinking the Way We Measure and Drive Organizational Success (New York: American Management Association, 2007). Spitzer asserts that the context of performance measurement – positive or negative – will determine its effectiveness. He’s right, of course.

But I would go further. The context will determine if the effort will even get off the ground! Playing gotcha with performance data will poison the well from which performance management draws its strength.

Spitzer tells us that too many people are accustomed to the negative side of performance measurement, especially its use as a way to make judgments of good and bad, place blame, search for the “guilty” low performers, reward a few and punish the many, instead of a tool for improvement. He gives the example of two ways of gaining weight control. The first is to be forced to stand on a scale and be told that you are overweight. That hurts! The second is to use the scale on a voluntary and self-help basis to achieve weight loss. In the first case it is used to judge and in the second case it is used to provide an incentive, to motivate and to empower. Context matters.

What Are You Going to Replace It With?

Behavioral psychology teaches us that if we want to change people’s way of doing things, it is not sufficient to extinguish unwanted responses. We’ve got to identify and then reinforce the desired behavior. If we don’t, the undesired behavior will persist.

If we don’t like the negative ways people think of using performance data, what uses would we prefer to replace them with? What context would we prefer? See Made2Measure, May 25, 2007, Creating an Environment Receptive to Performance Measurement.

Performance Measurement – Learn to Use It (Quickly) or Lose It

Identifying the right measures – the first requirement of an effective court performance measurement system -- is necessary, but not sufficient. The second requirement, getting the right measures into the hands of the right people, at the right time and in the right way is absolutely necessary, but not sufficient.

Unless the right measures are actually used by the right people in the right way – or at least thought of being used -- the measures will be ignored or, worse, misused for purposes not intended.

Examples of Effective Use

We must know, and be prepared to explain concisely, how court performance data will be used to improve programs and services – even if we have to resort to hypothetical examples, or ones drawn from other courts or other disciplines. How will the baseline performance data on the court citizen encounter, for example, immediately help the court make its courthouses safer, its case processing more efficient, its judges more courteous and respectful, and its counter staff more attentive? What is the context in which performance measurement will be used?

Compelling examples of innovative ways of bringing excellence to the court citizen encounter by effective performance measurement and management must be imprinted in the minds of people before the performance data are put into play in the gotcha game. The positive ground for performance measurement must be captured by actual or intended uses of performance data.

Effective use is what matters most. My fear is that unless we lead with descriptions, examples, anecdotes and stories – and actual results if we have them -- of how performance measurement drives success, the gotcha game will stop many promising efforts in their tracks.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved

Court Intelligence -- A Matter of Survival

2007-12-11T09:21:00.000-05:00

“What if I were to tell you,” IBM’s Dean R. Spitzer asks in the first sentence of his 2007 book, Transforming Performance Measurement – Rethinking the Way We Measure and Drive Organizational Success (New York: American Management Association), “that one of the most important keys to your organization’s success can be found in a very unlikely place – a place many of you may consider to be complicated, inaccessible, and perhaps even boring? … The key to success is MEASUREMENT,” he tells us (emphasis in the original).

Effective performance measurement and management can transform your organization, writes Spitzer. It shows you where you are and gets your organization where it wants to be.

I, of course, agree with Spitzer. But I would go even further.

Performance Measurement No Longer Optional

How is your court or court system performing in meeting its fundamental obligations to those it serves -- access to justice, fairness and equality, efficiency and effectiveness, professionalism, honesty and integrity, public trust and confidence, as well as accountability? How would you answer this question? Would you even bother?

Court leaders and managers today no longer have the problem (or luxury) of not knowing? Instead, they face the burden of whether they want to know. And if they believe they are better off not knowing, they still will face the demands of those that want to know, including the public, legislators, government executives, the court system’s own employees, and other stakeholders whose calls for accountability for performance are getting louder and more persistent. Not knowing the answer or refusing to answer no longer are viable options for court executives.

We Have the Keys

Identifying the right measures, getting them into hands of the right people, at the right time, and in the right way, and ensuring that performance measurement and management are integrated into the leadership and everyday management of the courts are no longer mysteries. The business and technology architecture to meet these key requirements exists. We have the tools. They are in place, in at least rudimentary but recognizable form, in courts in Oregon, Florida, Utah, Texas, California, Arizona, Illinois, Massachusetts, and several countries outside of the United States.

Business intelligence (BI) is a matter of survival for private companies. I predict that effective performance measurement and management – court intelligence – will soon be a requirement, if not a matter of survival, for court leaders and managers. Decisions and actions will need to be based on valid and replicable performance data – not on hunches or conjecture.

Sure, there will be courts and court systems that will resist developing court intelligence systems, just as there are those that have resisted automated court management systems. But they will be out of the mainstream.

Hallmark of Success

Today, performance measurement and management, and court intelligence -- hard-wired into the DNA of the very fabric of court leadership and management – is the hallmark of a successful court or court system. “A science is as mature as its measurement tools,” writes Dean Spitzer quoting Lois Pasteur. The same can be said about high-performing organizations.

In early cultures, the position of the sun in the sky was sufficient to measure time. Today, the tools for precise time measurement in fractions of seconds are all around us (and, literally, on us in the form of watches, cell phones, and PDAs). At least at work, our survival depends on our ability to measure time.

In the near future, court intelligence, in the form of performance measurement and management, is likely to be the defining attribute of a high-performing court or court system.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved

The Exciting (and Confusing) Court Performance Dashboard Market

2007-11-14T10:02:00.000-05:00

Where We Are Today

The fundamental goal of court performance dashboards is to empower all stakeholders with the right information, at the right time, using the right technology to make better decisions across all court functions. An increasing number of individual courts and court systems are beginning to look at performance dashboards, and supporting business analytics and intelligence, not only as reporting and accountability tools but as the means to make improvements and to drive success. They deploy these tools to discover and to explore information, to describe historical trends and to predict future trends, to devise improvement strategy, and to share information with stakeholders.

That’s the basic message my colleagues and I delivered at the Super Session, Performance Dashboards: Measuring, Monitoring, and Managing Your Courts, at the Tenth Court Technology Conference (CTC10) in Tampa, Florida, last month. I concluded at the close of the session attended by about 300 – 400 court technologists, managers and judges that we are at the “tipping point in the development of performance dashboards and business intelligence as a major factor in the management of organizations including courts.”

The Super Session included three demonstrations of performance dashboards and business analytics used by the Utah State Courts, the Harris County Courts (Houston, Texas), the Provincial Court of British Columbia, and the Court Services Branch of the British Columbia Ministry of the Attorney General. The demos were made by Kim Allard, Director of Court Services, Utah Administrative Office of the Courts; Harry Leverette, Director of Information Technology, Harris County Courts; Dan Chiddell, Director, Strategic Information & Business Applications, Court Services Branch, British Columbia Ministry of Attorney General; and Grant Marchand, Manager, Judicial Resource Analysis and Management Information Systems, Office of the Chief Judge of British Columbia. (I understand that the full background paper and the video of the entire session will soon be available on the CTC10 Website; only a PDF version of the draft introductory PowerPoint slides is currently posted.)

Jorge Basto, the Chief Information Officer of the Administrative Office of the Courts of Georgia, a panelist of the CTC10 Super Session, noted that “it has been almost 20 years since Howard Dressner [formerly with Gartner and now CSO of Hyperion, a leading BI firm recently acquired by Oracle,see below] established the phrase ‘business intelligence,’ or ‘BI,’ which refers to the environment that is used to gather process and analyze information regarding entities’ operations. The use of performance dashboards is an important part of this BI architecture and what we are seeing today is the public sector catching up to what the private sector has known for years.”

The Break-Neck Pace of Consolidation

Unfortunately, catching up with what is happening in the business intelligence market will not be easy. It’s hot, it’s big and getting bigger. And it’s consolidating so rapidly that it’s hard to keep even the big players straight. (Those among us faced with writing requests for information (RFIs) and requests for proposals (RFPs) feel we don’t have it easy.)

According to IDC, who analyzed and forecasted the business intelligence market for the period from 2005 to 2011, the business intelligence market grew at a rate of 11.2% and reached $19.3 billion in 2006. Performance management tools and applications accounted for $13.6 billion and the data warehouse category garnered $5.7 billion.

On Tuesday, the Wall Street Journal (IBM's Deal for Cognos Signals Strategy Shift, by William M. Bulkeley) reported that IBM agreed to acquire Cognos, the third largest business intelligence vendor for $5 billion, its largest acquisition ever. This puts IBM squarely up against SAP, which last month agreed to buy the industry leader Business Objects. And all this comes less than a year after Oracle, a rival of IBM in the database-software market, bought Hyperion Solutions, the No. 4 business intelligence vendor. SAS, a privately held company based in Cary, N.C., is the second-largest vendor and the only one in the top five that focuses almost exclusively on business analytics (as opposed to data warehouse platform applications). And then there’s Microsoft, with Proclarity and Performance Point.

The trick is, I assume, for these big players to be the one-stop shop for all the components of the performance dashboard technology architecture including data warehouse platform software, data movement and transformation applications, data monitoring, query/reporting/analytics, advanced business analytics, and performance management tools. While this consolidation might be good for the big firms’ shareholders, it isn’t yet clear whether it will make life easier for court executives who want to buy or build advanced performance dashboards.

What Does This Mean for Courts?

How will these colossal tech companies -- IBM, Microsoft, Oracle, and SAP – accommodate courts with budgets that they may regard as chump change? Will they have any interest in leveraging courts’ relatively modest investments in their existing data warehouses, business analytics, and other applications and tools?

According to IDC, as consolidation in the business intelligence continues, a new generation of software vendors will target specific market segments with innovative solutions and specialized products. This occurred with court-transaction data in the development of systems for case management, jury utilization, and fine and fees collection. But such transaction systems are not set up to for the kind of performance monitoring, analysis and management provided by state-of-the-art performance dashboards.

But how long will it take? Will innovative courts be willing to wait for court-specific performance dashboards? (We know that dozens are not.) Should courts look to second-tier vendors and new start-ups entering the growing market for software to store, analyze and display court performance data? Should they hire a third-party technology integrator like Threshold Consulting to put all the pieces together? Or should they build performance dashboards themselves using a combination of commercially available packaged software and “custom” software?

These are difficult questions and the right answers will depend on the court and its operating, IT, and political environments. A key consideration is to what degree courts have identified their success factors and have scoped out performance measurement and management initiatives linked to strategic goals. Another is the degree to which a court or court system is ready to expand the number of employees designated as decision makers and discovers of solutions. This, of course, has direct bearing on identifying users of performance dashboards and their specific monitoring, analytic and performance management needs.

For those of us interested in court performance measurement and management, these are exciting and challenging times!

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved

Courts Have No Business Doing Research

2007-10-15T09:47:00.000-04:00

The theoretical foundations and methodology of the disciplines of research and performance measurement overlap, but they are very different in important ways: sponsorship, organization, audience, functions, timing, and data interpretation rules. (See The Differences Between Performance Measurement and Research, Made2Measure, October 7, 2005; and Forget “Statistically Significant,” Made2Measure, December 17, 2005.)

Replication in Performance Measurement and in Research

A critical difference between performance measurement and research that I did not mention previously has to do with replication. Basically, this means repeating the research to corroborate the results and to safeguard against overgeneralizations and other false claims.

Repeated measurements on a regular and continuous basis are part of the required methodology in performance measurement. Analyzing trends beyond initial baseline measurement requires replication of the same data collection and analysis on a monthly, weekly, daily or, in the case of automated systems, on a near real-time basis.

In contrast, replication in research is a methodological safeguard that is universally lauded by scientists but, as we’ve recently learned, seldom used by researchers. Robert Hotz reports in the Wall Street Journal (September 14, 2007 http://online.wsj.com/article/SB118972683557627104.html) that most published research findings are wrong and that most scientific studies appear to be tainted by sloppy analysis including miscalculation, poor study design, and “self-serving” data analysis – problems that replication of the studies and duplication of the results would largely correct. Trouble is that it’s not done.

"There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," reports Hotz, quoting John Ioannidis, an epidemiologist who studies research methods at the University of Ioannina School of Medicine in Greece and Tufts University in Massachusetts. "A new claim about a research finding is more likely to be false than true," Ioannidis says.

According to Dr. Ioannidis, “in most modern research, false findings may be the majority even the vast majority of published research claims.” Ioannidis and his colleagues studied 432 published research claims about gender differences in risks of diseases. Their research reported last month in the Journal of the American Medical Association showed that almost none of the claims about gender differences held up under close scrutiny. Only one of the studies was replicated.

Would you rely on court user satisfaction data without giving it a reality check? Would you listen to financial advice by a late night infomercial? Of course not. The apparent lack of accepted safeguards for research studies, that research findings are rarely checked and replicated, should hit a raw nerve for those who do, sponsor and consume court research findings.

What does this mean for courts? I have argued that courts have no business doing research and the apparent lack of safeguards such as replication is just one reason. In my view, the lesson of Ioannidis’ revelations should not be that researchers should not be trusted, but rather that research is done by mere mortals who make honest mistakes. It is also difficult and, more often than not, expensive to do well.

“People are messing around with data to find anything that seems significant, to show they have found something that is new and unusual, Ioannidis said. Thinking back to my days as a Ph.D. candidate doing “pure” research in experimental psychology, I cringe at my overeager desire to coax “significance” from my dissertation research and my revulsion at the thought of replicating my findings before my degree was handed to me.

I am not suggesting that court research should not be done. Far from it. I am recommending that courts should abandon the idea of doing research themselves and that they should outsource it to academic researchers (who may do much of it on a quid pro quo basis in exchange for access to research opportunities for themselves and their students) and venerable not-for-profit court research institutions like the National Center for State Courts. Having engaged the researchers, they should then hold their feet to the fire and demand replication among other safeguards against overgeneralizations and false claims.

Performance Measurement Is Court Business

Performance measurement is hard-wired into the very DNA of the leadership, management and operations of successful operations who ask themselves “How are we doing?” on a regular and continuous basis. It is the process of measuring accomplishments (outcomes), work and service levels (output), and its resources (inputs). Behavioral and social research (including evaluation research and program evaluation), on the other hand, is scientific study to discover factual truth, to test models and to develop theories to increase our knowledge and understanding about human behavior and social phenomena.

Performance measurement and research share an adherence to scientific methods and processes. Both use quantitative and qualitative methods including surveys and questionnaires, interviews, direct observation, recording, descriptive methods, tests and assessments, and statistical analysis.

Courts should own performance measurement as part of its normal business, but outsource research and program evaluation.

That takeaway message for courts is consistent with the lesson of the hedgehog concept of management guru Jim Collins, author of the bestselling books Good To Great and Built to Last. The fox knows a little about a lot of things, but the hedgehog knows only one big thing very well and sticks to it. The fox is complex; the hedgehog simple. And the hedgehog wins.

Collins’ research shows that success requires a simple, hedgehog-like understanding of three intersecting circles: what a court does best and what it does not, how it works, and what best ignites the passions of its people. Great things happen when courts comply with the hedgehog concept and become systematic and consistent with it, eliminating or outsourcing virtually anything that does not fit in the three circles.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved

Jury Representiveness Redux – A Lament for a Good Measure

2007-09-13T09:46:00.000-04:00

Having proposed or otherwise advocated for various performance measures that have not seen the light of a court day, I should be accustomed to the low use of measures that I happen to believe have high value. But, alas, I continue to puzzle over why one such measure in particular – jury representativeness -- is not used more by courts.

Jury representativeness – as I defined it in a two-part Made2Measure posting on April 12 and April 22 last year – is the comparative parity (i.e., the absence of disparity) -- expressed as a percentage -- between the representation of minority groups in the population and the representation of the same groups in the final juror pool or venire. How well juries mirror the community from which they are drawn is widely considered a reflection of the equality, fairness and integrity of our justice system.

Arguably, identifying a combination of demographic characteristics as the source referent -- including gender, age, income level, and education -- may be better than looking at race and national origin alone to determine jury representativeness. However, because race and national origin are the most salient demographic characteristics in law, policy debate, and jury utilization and management, disparities in race and national origin may be seen as reflective of disparities in other demographic categories. That is, if the jury pool fairly represents the community in terms of racial and national origin, most other demographic characteristics are likely to be represented fairly as well.

Filling the Jury Box

Nationally, only about half of the people summoned actually show up at the courthouse. Courts across the country are going to extraordinary lengths to fill their jury boxes -- to increase their juror yield (the proportion of people who show up when summoned) and their utilization of the jurors once they show up at the courthouse. The National Center for State Courts’ CourTools two-part output measure, effective use of jurors, focuses on the efficiency of processes of jury management and utilization.

But what outcomes that might reflect equality, fairness and integrity of our justice system do we get with our increased efficiency? Who is showing up for jury duty? Are they representative of the communities from which they are drawn?

Everybody seems to want to know, but few are taking this measure the way it ought to be taken.

A Quick Quiz

You are a jury commissioner, a court administrator, a court clerk, or a presiding judge responsible for jury management and utilization in your jurisdiction. Although your qualification and summoning process is designed to identify a representative sample, something tells you that those who actually appear in your jury pool do not necessarily reflect the race and national origins of the community. You feel the pressure of increasing attention being paid locally and nationally to the representativeness of the race and national origin of those who show up for jury duty. Which one of the following actions do you take?

A. You do nothing.

B. You grant access to your juror pools to a local academic who identifies the race and national origin of potential jurors in the pool by sight and computes the disparity of race and national origin between the juror pools and the eligible population of jurors in the jurisdiction.

C. You rely on individual judges to dismiss a panel of jurors if they believe that they do not look like general population in terms of race and national origin.

D. As part of your key management processes, you regularly monitor, analyze and manage jury representativeness, a measure of parity in race and national origin between jurors who are qualified and available to serve, on the one hand, and that of the eligible population of jurors in your the jurisdiction, on the other hand.

And the Answer Is …

“D” is, of course, the preferred answer. But “A” is definitely the most likely answer given.

Even though jury representativeness is not included as part of Measure 8 of the CourTools, and perhaps as a consequence, not used routinely as a core performance measure by courts, at least a few courts have been willing to consider it as a measure of fairness and equality.

For example, the measure “Effective Use of Jurors,” designed by the Maricopa County Superior Court (Phoenix, Arizona) is an index composed of four elements: (1) jury representativeness; (2) juror satisfaction -- the percent of jurors giving favorable rating's to the court's accessibility, convenience and treatment of them; (3) juror yield – the number of citizens called for jury duty who are qualified and report to serve, expressed as percentage of the total number of prospective jurors available; and (4) juror utilization – the rate at which prospective jurors are used at least once in trial or voir dire.

The composite measure serves as a performance indicator of the Court’s fairness and equality (by the element of juror representativeness), the satisfaction of an important category of the Court’s customers and stakeholders (juror satisfaction), as well as an indicator of the effectiveness and efficiency of the Court’s jury management and utilization (juror yield and utilization). Only the last two elements of this measure – juror yield and juror utilization – are prescribed by the CourTools Measure 8, effective use of jurors. The court’s design team agreed that the enhancement of this measure by the addition of the first two elements, especially juror representativeness, elevates the measure to that of a “core” mission measure in a balanced scorecard of measures.

Strange But True

As strange as it may seem, the answers “B” and “C” in our Quiz are not hypothetical but based on real occurrences, as reported in the June 29 and July 20 editions of National Center’s Jur-E Bulletin.

The story line for “B” comes from a survey that was conducted from November 2006 until February 2007 by two researchers who studied more than 12,000 prospective jurors in New York City by sight alone. They identified people based on "physical characteristics like skin color rather than by asking them." The story was picked up by the New York Times. Some concerns were raised in the story by these research methods since physical characteristics are often not an appropriate measure of race and ethnicity.

Well yeah! Skin color? Looks can be deceiving, or worse. But wait, there’s more.

With regard to “C,” the other strange answer, it seems that D.C. Superior Court Judge Neal E. Kravitz had never seen a jury pool quite like the one that walked into his courtroom for the trial of Donnie Ray Horne earlier this year. At the request of the defense lawyer and over prosecutors' objections, Kravitz sent back all 70 prospective jurors -- 61 white, eight black and one Asian -- and called for a new group. He claims to have "never had a panel in a criminal case that looked like this," according to the Washington Post which picked up the story on July 15, 2007.

The Challenge

So here’s the puzzle and the challenge. How do we respond to ever increasing demands for representative juries without resorting to questionable and unsustainable research techniques and innovations by judges? How do we convince jury commissioners, court managers, clerks and judges that direct measurement of jury representativeness on a regular and continuous basis is not only possible, but feasible?

As noted by the Jur-E Bulletin, the back stories for the Quiz answers “B” and “C,” remind us that race and national origin data can be a valuable tool not only for answering criticisms from counsel and the public at large about specific cases but for addressing larger questions about equality, fairness and integrity of our justice system. Courts have the ability of which the New York City researchers and D.C. Judge Kravitz did not avail themselves - to query jurors directly about their race and ethnicity. While it may seem challenging to collect such data, it can be as simple as distributing a simple questionnaire to prospective jurors sitting in juror assembly room tomorrow morning.

We press ever forward.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved

Performance Measures = Leadership Clarity

2007-08-05T13:14:00.000-04:00

If you do nothing else, be clear, says Marcus Buckingham, who has spent a lot of his time studying leadership. You probably thought that your job is to analyze the complexity and chaos of your court’s operating environment and reflect it back to the court’s stakeholders. You’d be wrong. (When you’re asked what time it is, don’t respond with instructions for building a clock!)

Tell people what it is the court should achieve. But tell them succinctly. Point them in the right direction and give them a good handle on how to get there.

Today’s court executives and managers need more than a strong message and charisma to lead effectively. They need a performance measurement system that focuses and magnifies what is most important. They need ready access to clear and actionable measures that allow them to explore the court’s performance from multiple perspectives and to steer the court in the right direction.

Performance measures – like the percent of court users who are satisfied with the way they were served by your court or the percent of case files that were retrieved within ten minutes of request – are valuable to court leaders and managers because they are focused, unambiguous, and actionable. They serve as incentives and practical tools for change.

Public Trust and Confidence

Ray Rhodes, the executive director of the Office of the State Attorney in the Twentieth Judicial Circuit in Florida, told me a story last week in Ft. Myers that illustrates the point.

Rhodes was approached by a neighbor who manages a local department store. The neighbor told him that he was struggling with rampant shoplifting. He said he lost confidence that the criminal justice system was managed well enough to help him solve his problems.

The neighbor was quick to assure Rhodes that he had no gripes with the State Attorney’s Office. He said he was very satisfied with the attention the State Attorney’s Office was giving to his shoplifting cases. He appreciated the prompt notifications his employees received of the hearings and trials scheduled and rescheduled, and rescheduled again.

The problem, said the neighbor, was that trials were being postponed so often that he could not afford to give the time off to his employees to appear as witnesses. He told Rhodes that he has resorted, reluctantly, to dealing with his shoplifting problem on his own. From now on, the neighbor said, instead of calling the police, he would handle any shoplifting of merchandise worth less than $20 dollars himself, without law enforcement and the criminal justice system.

Rhodes believes that we could probably find many more store managers, owners and even non-business citizens that share the same sentiment as his neighbor. "Many crimes go unreported for various reasons," says Rhodes, "but one of the larger reasons is that people did not have the time to deal with the criminal justice system - just too confusing and too much time out of their schedule. It's just easier to buy a new bike, camera, etc."

The traditional way court leaders might handle this problem is by commissioning, that is, by creating a commission or blue-ribbon panel to look into the problem or commissioning a study of eroding public trust and confidence in the courts. There would be a lot of hand-wringing and grave pronouncements followed, in good time, by recommendations for transformation and reform on a grand scale (assuming, of course, that there’s the money to do it all).

Trial Date Certainty

Instead, Mr. Rhodes and his colleagues in the Twentieth Circuit are heading to a simpler solution along the performance measurement route. Rhodes is part of a Circuit-wide effort involving the major justice system partners in the Twentieth Judicial Circuit – the courts, the clerk’s offices, public defenders, law enforcement and jail representatives, probation and pretrial services. The group is defining the problem highlighted by Mr. Rhodes’ neighbor in a less ambiguous, more focused, and certainly more actionable way than the traditional “commissioning” approach.

The Twentieth Circuit is currently developing and testing a trial date certainty index, a measure of the certainty with which trials are held when scheduled, expressed as the number of trial settings per trial by case and trial type or category. (Full disclosure: My colleagues and I at the National Center for State Courts’ Court Consulting Services are helping.)

This performance measure, which is among the arsenal of measures in the CourTools designed by the National Center, is an indicator of the certainty, predictability, timeliness and efficiency of case processing. Required data elements are the total number of trials disposed or resolved by case (e.g., criminal) and trial (e.g., plea) type or category, and the number of times those same trials appear on the court’s trial calendar. For example, if a total of 50 trials were held in a given period of time and the total number of times those 50 trials appeared on the trial calendar is 218, the trial date certainty index would be 218 divided by 50 = 4.36 (this is a hypothetical example, not the Circuit’s actual performance).

The plan is to take the baseline of the average trial date certainty index Circuit-wide, slice and dice this average by case type and trial type, and by location, and then look for opportunities for improvements. This is where performance measures lend themselves to absolute clarity and focus.

At the outset, the Circuit’s leaders and managers might ask themselves if the baseline levels Circuit-wide, and for particular case and trial types, and in particular locations (the Twentieth Judicial Circuit encompasses five counties) are satisfactory, and then set appropriate short and long term performance goals and targets for the Circuit as a whole, and for case types, trial types, and different locations. For example, a short term goal might be to reduce the trial date certainty index below 4.0 Circuit-wide in six months, and reach the long-term goal of 2.0 or below in two years.

Sidestepping Sticky Issues

Commitment to a focused, unambiguous, and actionable performance measure is a way to sidestep sticky issues of institutional differences and governance and, perhaps, deal with them once successful joint effort is achieved around a decrease in the trial date certainty index. The power of the performance measure is that it helps people focus and get right to work without having to spend an eternity rehashing the back-story of institutional differences.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved

And the Winner Is … Business by Data

2007-07-27T18:20:00.000-04:00

Not even close, at least in the business sector.

In the last posting, I suggested that court managers listen actively and patiently to what people have to say no matter what truth-finding ways and means they’ve used including: (1) the truth we feel (including “truthiness”); (2) the truth that is given to us; (3) the truth we ferret out ourselves by reason and logic; (4) and the truth we perceive through our senses and empirical data, including performance information.

No one method is foolproof, I argued. Each can complement and correct the mistakes of the others. Better to use all four methods, though we may favor one. The common ground is where the truth is more likely to reside.

I advised – sensibly I think -- that we not we not discredit those who prefer to reach the truth in ways different from ours. Apparently, business does not see it that way.

“We’ve had management by objective and total quality management. Now it’s time for the latest trend in business methodology: management by data,” writes Scott Thurm in last Monday’s Wall Street Journal. He quotes Stanford business professor Robert Sutton praising data-driven enterprise and deriding prior business management techniques as “faith, fear, superstition and mindless imitation.” Professor Sutton is co-author with Jeffrey Pfeffer of Hard Facts, Dangerous Half-Truths & Total Nonsense. Strong words!

In my defense, I say, first, I agree to a degree. After all, I am a champion of performance measurement and see it as a foundational organizational system upon which all other organizational systems (e.g., strategic planning) depend. Second, by suggesting that we balance data-driven truth-seeking with intuition, heart, common sense, history, logic and reason, I espouse the ever so sensible discipline of mental models as put forth by management guru Peter Senge, author of The Fifth Discipline. At the very least, says Senge, becoming more aware of the mental models or preconceptions of others – including their truth-seeking ways, I would add -- can help you become a more nimble leader and manager.

In fairness, the Wall Street Journal acknowledges that in at least one key area –innovation -- business success cannot come from data-drive truth-seeking alone. David Girouard, a vice president and general manager of Google, is quoted as saying: “You can’t time or plan for innovation. It has to come from the heart of somebody with an idea.” Scott Thurm of the Wall Street Journal concludes his article by saying that there’s “no single formula for business success.”

So, while I would agree that the business by data is the front runner, other ways of getting to the single version of the truth may not be far behind. Why not award some silver and bronze instead of disparaging the efforts? I stick to my advice to allow people to be heard who do not necessarily use the scientific method to seek their version of the truth.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved

Truth and “Truthiness”

2007-07-21T11:33:00.000-04:00

“The truth-quest is always the same: the unwavering search for signs to match reality.” -- Felipe Fernandez-Armesto

Stephen Colbert, the satirist and host of The Colbert Report, coined the term “truthiness” to refer to the things people know to be true “from the gut,” as opposed to from the head and from dry data that comes from the laborious (and boring) process of science. Many court managers, judges and clerks rely on truthiness to answer the question “How is the court performing?” Quite predictably, some of us who are “made to measure” and feel smug about using the scientific method to get at the truth disparage them. That disparagement may be misguided, if not arrogant or dead wrong.

Getting at the Truth

But how do we know what is true? How do we know the Way Things Really Are? How can we tell falsehood from truth?

In his deft little book, Truth: A History and a Guide for the Perplexed (St. Martin's Press, 2000), Felipe Fernandez-Armesto, a member of the modern history faculty at Oxford, argues that there is “no social order without truth or, at least, without agreed truth-finding procedures (emphasis added).” He writes that people throughout history have sought to get at the truth by one or more of four basic ways: (1) the truth you feel in the gut or wherever feelings originate, (2) the traditions of the past, (3) reason and logic (the head), and (4) sense and science, i.e., the scientific method.

Fernandez -Armesto tells us that all four categories have always been around, competing or co-operating with one another as ways of discovering the truth. And, lest we think the first two are relics of the past, he reminds us that all four ways of truth-finding are alive and well today.

Gut Instincts: The Truth You Feel (Truthiness)

The first way gets at the truth through feeling, introspection and intuition. Truth is a tangible entity that we feel "in our gut." In pre-literate societies and some literate ones, Fernandez-Armesto tells us, truth was understood in emotional or non-sensory and non-rational ways.

But it would be wrong to view this way of truth-finding as outdated. Today we know that emotional intelligence and competence in the workplace -- the ability to recognize and to manage our own feelings and those of others in positive and productive ways – may play a larger role in successful organizations than analysis, calculation, logic and reason. Twenty years of research in neuroscience, behavioral economics and, yes, court administration (especially the work of Tom Tyler in “procedural justice”), has established that people base their decisions and their trust and confidence in organizations on a complicated mixture of emotion, reason, ethics and morality. Emotions, intuition and gut instincts – specifically, confidence, integrity, pride, passion -- frame the customer employee encounter.

Divination and Authoritarianism – The Truth That Is Given

Then, according to Fernandez-Armesto, comes divination or the authoritarianism of "the truth that is given" or “the truth that we are told.” This is a method employed through “various human, oracular, divinatory, or scriptural authorities.” Ancient Greek gods spoke the truth through common people. Medieval Christians in the “Age of Faith” (in contrast to the “Age of Reason”) received truth on authority.

This way of seeking the truth (literally) has precedence today. The legal doctrines of stare decisis and res judicata hold courts to policies and procedures that require them to stand by precedent, adhere to principles of law already laid down, and not to disturb settled point and matters already decided – in other works, to take the truth as given. And, of course, a higher court trumps the lower court.

Logic and Reason – The Truth You Think for Yourself

Third comes reasoning, the “truth of reason” or the “truth you think for yourself” using rational or logical analysis, a method that is not subject to the misinterpretation of introspection and divination. Rational thinking and logical analysis were used in China and Egypt long before Plato brilliantly employed them in his dialogues.

Logic and reason, of course, have great currency today. For example, a logic model of inputs, outputs, and outcomes -- an abstract simplification of what impact a court program is likely to produce -- is recommended as a useful tool for the development of court performance measurement and court program evaluation.

Science, Empiricism and the Scientific Method – The Truth You Perceive Through Your Senses

Finally, there is the relatively late development of the “truth you perceive through your senses” including evidence gained from experimentation. Plato’s pupil, Aristotle, saw the limits of logical analysis and began to use evidence gained from his study of nature to bolster his arguments. Galileo, the foremost scientist of his day, broke free of the authoritarianism of seventeenth century Catholic doctrine, which taught the "truth" of the heavenly bodies as "given" by the church, and spawned the scientific revolution by revealing a new reality in the heavens through his telescopes.

Social science – including performance measurement, research and program evaluation – relies exclusively on the scientific method. It is meant to derive the single replicable and verifiable version of the truth. No questions asked. Case closed.

Well, those in the court community untrained or unaccustomed to social science’s methods – think of judges who sincerely believe that data derived from social science is mere “numerology” that needs to be tested in the crucible of the adversary system – may think otherwise. My sense is that much of the resistance to performance measurement may stem from discomfort with or distrust of “the truth you perceive through your senses.” This discomfort and distrust is exacerbated when different truth-finding methods are disparaged with thinly veiled arrogance by those of us schooled in social science.

Professor Fernanez-Armesto is instructive.

Listen and Learn

Triangulation is a technique of measurement that deals with systematic error in the “truth we perceive through our senses.” (The don’t-put-all-your-eggs-in-one-basket concept of the balance scorecard is a derivative of triangulation.) It is the use of several imperfect measurement methods to see if they point to the same conclusion. If they do, we can have more confidence that our measurement error is in the comfort zone. If one method yields sharply different findings than another, we have reason to suspect mistakes.

Professor Fernandez – Armesto is much more expansive. What he advises make eminent sense for all of us.

The first important step is to listen actively and patiently to what people have to say no matter what truth-finding method they’ve used. By all means, let’s not discredit those who prefer to reach the truth in ways different from ours.

Searching for the truth, Fernandez-Armesto writes, is fundamental to education. He advises us to return to all four methods that have served us so well throughout history and continue to do so today: the truth we feel, the truth that is given to us, the truth we ferret out by reason and logic, and the truth we perceive through our senses. No one method is foolproof, but each can complement and correct the mistakes of the others. The common ground is where the truth is more likely to reside.

So why not take on all comers? Bring on the “truthiness” Mr. Colbert!

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved

What Is Our Business? Who Are Our Customers?

2007-06-23T14:36:00.000-04:00

What is a court’s business? Peter Drucker, probably the most revered management thinker, suggests that this fundamental question may seem simple and obvious. It is in fact difficult and anything but obvious (see his Management: Tasks, Responsibilities, Practices. HarperCollins, 1993, pp. 77 – 79).

I know it may sound heretical, but a court’s business is not necessarily determined and defined by law. Instead, it is defined by the wants and needs of its “customers,” suggests Drucker. “To satisfy the customer is the mission and purpose of every business.” By business, he means any enterprise: private, non-profit, and public.

For example, an appellate court may consider its primary customers to be the members of the appellate bar and trial bench who are the major consumers of the appellate court’s decisions and opinions. Because a few pro se litigants are likely to be the only members of the public who have direct contact with the appellate court, and because most appellant litigants’ court experiences are moderated by their attorneys, an appellate court may legitimately see the members of the appellate bar and trial bench as their primary customers. Not so for trial courts.

The question “What is our business?” is best answered by looking at the court’s business from the point of view of those served by the court. What they see, think, believe and want should be accepted as objective fact and should be taken as seriously as the on-time case processing reports by court analysts or the cost-per-case data reported by court finance officers. (Incidentally, there is nothing “soft” or subjective about what people believe to be true about justice and fairness. But that’s a topic for a future posting.)

Notice that to answer the question “What is our business?” we first need to answer the question “Who are our customers?” Both questions should be answered by looking at our business from the outside out the court, i.e., from the perspective of those who use the courts and are served by the court.

Over the last 25 years, there has been a dramatic shift from an insider’s perspective (those who run the courts) to an outsider’s perspective (those who are served by the courts) for defining the mission and purpose of courts. The promulgation of Trial Court Performance Standards in the late 1980s and early 1990s were instrumental in this shift in thinking. They defined the purpose and major performance goals of courts from the perspective of the citizens served by the courts: meaningful access to justice; timeliness and expedition; fairness, equality and integrity of the court system; independence and accountability; trust and confidence.

The takeaway message: Courts should be clear who they regard as their customers, they should determine what it takes to satisfy their needs and wants, they should regularly and continuously get their answers to the question “How are we doing?” and, finally, they should use this feedback to finds ways to improve.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved.

Radical Transparency

2007-06-04T11:33:00.000-04:00

Transparency is the hallmark of good government, an antidote to corruption and a facilitator of openness, communication, and accountability. It is the metaphorical extension of the meaning used in the physical sciences – a “transparent” object is one that can be seen through.

In his new book, The Art and Practice of Court Administration, Alexander B. Aikman urges courts to be more transparent. He suggests that “leadership courts” are those that open their administrative decision-making to public input, scrutiny, and participation, and argues that the tradition of silence regarding judicial decisions need not be carried over to administrative decision-making.

So much for theory. Transparency in practice is another matter, especially when it comes to courts’ performance.

How Much Performance Information Should We Share?

This is a question that causes much hand-wringing among court leaders and managers.

Much of the resistance to performance measurement is based in fears that baring the court’s performance for all to see will hurt the court. Some resistance also comes from the belief that giving people information about shortfalls in a court’s performance will only raise unreasonable expectations that the shortfalls will be made up by those who produced the performance information. “Why give us this information if you’re not going to do anything about it?” they might ask.

Better to hold performance information tight and close. Better to be less than transparent.

The Naked Truth

Enter Wired magazine with the counterargument – get naked and rule the world with radical transparency. Clive Thompson, in the cover story of the April 2007 issue of Wired, writes why exposing oneself is the future of business. He tells the story of Glen Kelman, the newly hired CEO of Redfin, an online real estate brokerage firm.

Redfin was turning the real estate industry upside down by refunding two-thirds of the commission that real estate agents normally charge. Customers loved it but traditional real estate agents pushed back hard on Redfin. The blacklisted Redfin and refused to sell houses to anyone who used the service.

Kelman first decided to keep the pushback from agents quiet and to pretend that everything was OK. “We were really ashamed that our customers were getting pushed around, so we tried to keep this dirty little secret,” he said to Wired.

But when things did not improve, Kelman changed tactics. He set up a blog and told the naked truth about the real estate business, including Redfin’s internal debates and his own failures. When old-school real estate agents unleashed blistering attacks on Redfin, Kelman published the criticism and his responses in full view of the customers.

A funny thing happened. Customers loved it. More signed on to use Redfin’s services. “Instead of discouraging customers, being open about our problems radicalized them,” Kelman said. “They rallied and started pulling for us.”

Transparency as a Judo Move

According to Wired, Redfin’s form of radical transparency is today the norm of many startup – and even some Fortune 500 – companies. “Transparency is a judo move,” writes Wired’s Clive Thompson. “Your customers are going to poke around your business anyway, and your workers are going to blab about internal info – so why not make it work for you by turning everyone into a partner in the process and inviting them to do so?”

Radical transparency goes much further than the open book management I recommended for use of court performance measurement in October last year. Conservative institutions that they are, perhaps courts should adopt an approach somewhere between open book management and radical transparency.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved.

Trust Promotes Compliance and Is Catalyst for Fairness

2007-05-27T07:59:00.000-04:00

What is fairness? Why do people cooperate with authorities? Why do they obey the law? Why is public trust in our courts so important?

To his already impressive body of research addressing these questions, Tom R. Tyler, Professor of Psychology at New York University, continues to add to our understanding of the interplay of fairness and trust and how both effect cooperation with authorities. Writing in the May issue of the Journal of Applied Psychology (Vol. 92, No. 3, 639 – 649), Tyler and his colleague David De Cremer, who is in the Department of Economic and Social Psychology, Tilburg University, Netherlands, report the results of two experimental studies and two field studies of the effects of procedural fairness and trust on people’s willingness to cooperate with authorities.

What Tyler found should be of interest to court leaders and managers: procedural fairness leads to cooperation and compliance only when trust in authority is high. It seems you can’t have one without the other.

We already knew from past research in social psychology and organizational behavior (much of it done by Tyler and his colleagues) that procedural fairness increases people’s cooperation and collaboration, and that trust promotes good working relationships. What Tyler and De Cremer found in their most recent work is that the two are related – the effects of fairness are nil in the absence of trust.

In other words, procedural fairness will not have the desired effect of compliance, cooperation, and coordination absent the trustworthiness of those in charge. Across all four studies conducted by Tyler and De Cremer, the effect of procedural fairness emerged only when trust was high. Fairness produced cooperation only when the authority was trustworthy but not when the authority was untrustworthy.

These results are important to the way we run courts, both the policies and the practices. They corroborate the argument made in the Trial Court Performance Standards – which is convential wisdom today but was radical thought twenty years ago – that public trust and confidence are just as important for courts as the other key performance areas including efficiency and timeliness, fairness, equality and integrity, independence and accountability.

Tyler’s most recent research suggests that without public trust and confidence, courts may have difficulty performing in their other areas of responsibility.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved

Creating an Environment Receptive to Performance Measurement

2007-05-25T05:59:00.000-04:00

An employee of a large urban court recently confessed to me her opinion about her court’s performance measurement initiative. (She did so reluctantly because she knew that I’m a strong advocate of court performance measurement.) She told me this: The results of the court’s access and fairness survey of court users holds about as much interest for me as the details of the presiding judge’s travel schedule – interesting, but not something that is relevant to what I do everyday.

Hers was not an isolated viewpoint in the court. Even though the survey results were made readily available in the court newsletter and other printed materials distributed by the court, most mid-level managers and line staff paid little attention. Some viewed the survey data as yet another brainchild of upper management that would translate into additional work, and nothing more, if they were not careful.

This is a problem that needs fixing. You can’t advance an idea in an unreceptive environment.

Getting With the Program

In any change initiative -- including the development of a court performance measurement system (CPMS) -- it is generally well understood that we must prepare the environment to accept the idea or the concept. We know it requires “proof of concept” or “getting buy-in” from the top. Usually, this means getting upper level decision makers, and maybe a critical mass of the judges, to go along with the idea and to give the green light for its development.

So much for upper management. But what about the rank-and-file in the court’s operational environment – the real marketplace in which ideas either live and grow, or wither and die? They may not hear or know anything about the change that’s coming until it’s upon them in the form of a report with exhortations by upper management to get with the program.

Let’s suppose your court has marshaled considerable resources and effort to develop a comprehensive CPMS including measures of timely case processing, cost, court user and employee engagement, and other indicators of organizational effectiveness. The question “How are we doing?” can now be answered in terms of the court as a whole, by division, and by various other perspectives. Leaders and managers are enthusiastically on board. You have built the CPMS, but will they come?

Preparing the Environment

Many courts are at this point or close to it with their performance management initiatives. The problem is that the operational environment and culture of these courts remains unreceptive -- or even hostile -- to the idea of performance measurement. Overworked court staff ask: “Why should I care?” “What difference do these numbers make to me and the work I have to do every day?”

It may be understandable that a large percentage of court employees – including judges and managers -- are no more motivated to change the way they do business by court performance numbers than they are by the presiding judge’s travel schedule. Why should they be if they can’t make the connection to their own work in the court?

Court performance measurement is in danger – not just the initiatives in a particular court but the court performance measurement and management-by-results “movement” as a whole – unless we answer this question, unless we make the connection. It is simply not smart, I think, to assume that performance measurement is plug-and-play. As I’ve written in previous posts, performance measurement is no field of dreams and build-it-and-they-will-come simply will not work.

Here are some of the things I think we need to do:

Measure what matters, count what counts. Because people do not commit themselves to a performance measure per se, but rather to the purpose of the measure, court leaders and managers promoting performance measurement first need to communicate the answers to critical questions about how the measurement of employee engagement and satisfaction, for example, relates to key success factors for the court: How does employee engagement relate to improved service to those served by the court? Why is it important? Why does it matter that we measure it?

Defining the desired results of tasks, processes, and programs is not easy for court leaders because there is often more than one right answer. We are most certainly right when we define results in terms of satisfying the customer (court user) and engaging court employees. We are also right when we define results in terms of timely case processing. But it is up to court leaders to make it absolutely clear what the results should be. This is the point where the individual employees's job and the mission of the court converge and have to be connected and harmonized.

But more needs to be done.

Establish a clear line of sight. Educate everyone in the court about how the specific results of performance measurement relate not only to what’s important to the court as a whole but also to the work of individual staff doing particular jobs. Those responsible for the performance measurement system must integrate the specialized knowledge and tasks of every employee into the common performance identified by a particular performance measure.

After the idea (at this point, the performance initiative is only an idea) is sufficiently promoted and has taken hold, managers and staff should be adequately trained on how to read and understand the measure, what the results mean to them, and specifically how they can use the results to improve their performance.

This seems daunting and it is. It requires considerable training on the purposes, definitions, and uses of performance measures.

In many courts there is a disconnectedness between what upper management does (e.g., strategic planning, court-wide performance management) and what line-staff does (e.g., case processing, record keeping). Making a meaningful connection between them by establishing a line of sight between strategic and operational performance measures can cause significant improvements in performance.

Provide incentives. Set short-term and long-term goals and targets for teams, units and divisions. Reward those who are successful in meeting these new goals and targets. Consider gainsharing programs.

Hard wire performance measurement into the DNA of the court. For performance measurement to be truly effective in changing the way a court does business, performance measures have to be hard-wired into the very DNA of the court’s leadership, management and organizational culture.

It is one thing to build a CPMS and maybe to get it to be used once or twice. It is quite another thing for it to be used on a regular and continuous basis.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved.

Performance Measurement on Wall Street

2007-05-12T10:48:00.000-04:00

For some folks in court administration, the subject of performance measures is an anathema – the revenge of the number crunchers and spreadsheet guys -- for others, it’s their lifeblood. Apparently, the financial world sits up and takes notice when someone messes with its numbers.

Today’s Wall Street Journal reports that accounting-rule makers around the world, in the coming months, may eliminate the bedrock measure of business health and value: net income or net profit, the bottom-line figure showing what is left after expenses have been met.

Net profit is the measure of business performance millions of investors use every day to buy or sell stocks and bonds. It is used to determine executive bonuses and other forms of compensation. The single net profit number is the most commonly used measure to evaluate a company’s health, especially when it’s compared to the price of the company’s stock, what’s called the price-to-earnings ratio, or simply P/E. For example, the P/E ratio of company with a share price of $10 and earnings per share of $2 is 5.

This is a big deal. The Wall Street Journal, which put the story on today’s front page, says that the overhaul of profit as we know it “could mark one of the most drastic changes to accounting and financial reporting since the start of the Industrial Revolution in the 19th century.” If adopted, the changes will cause every accounting textbook to be rewritten and anyone who uses profit as a gauge to rethink how to measure business performance. “The cost of this change could be monumental,” writes the Wall Street Journal, quoting John Previts, an accounting professor at Case Western Reserve University. “All the textbooks will have to change, every contract and every bank arrangement will have to change,” he says.

The overhaul of the profit measure reflects the don’t-put-all-your-eggs-in-one-basket argument of the balance scorecard concept of performance measurement. As the Wall Street Journal points out, giving so much power to a single measure has been a recipe for fraud and stock-market excesses. Company executives in the past have gamed quarterly earnings by sacrificing the overall health of their businesses.

The goal of the accounting-rule makers is to blunt this over-reliance on a single number and come up with a more comprehensive gauge of how companies are doing. “I know the world likes single bottom-line numbers and all that, but complicated businesses are hard to translate into just one number,” says Robert Hertz, head of the Financial Accounting Standards Board, one of the groups working on changing the profit measure.

How does all this apply to court managers and courts?

When answering the question “How are we doing?” you should not rely solely on a single measure such as on-time case processing or court-user satisfaction. Be sure to look at a balanced scorecard of core measures that reflect an accurate picture of your court’s performance, a scorecard that cannot mislead or be easily manipulated as a single measure can. And, finally, carefully examine the court’s core measures every so often to ensure that they are truly aligned with the key success factors you’ve identified for your court.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved.

The Real Promise of Performance Dashboards

2007-05-09T12:38:00.000-04:00

Much of today’s hype about performance dashboards – a new book seems to coming out every few days -- misses an important point. Stephen Few, in his book, Information Dashboard Design: The Effective Visual Communication of Data (O’Reilly, 2006), for example, writes that dashboards have emerged in response to the “tsunami of data that rolls over and flattens us in its wake.”

To be sure, performance dashboards help those who are drowning in data -- the executive, manager or analyst who is overwhelmed with too much data. A well-designed performance dashboard lets her view performance measures at a glance, and then move easily through successive levels of strategic, tactical and operational performance information to get the insight she needs to solve problems and to improve program and services.

The focus is on those who already have access to performance data – albeit in a form that prohibits or at least discourages its use. But what about the people who do not yet have any access to reliable performance information?

In a typical court today that may include everyone except the court administrator and the analyst who routinely sends monthly case processing data to the state administrative office of the courts. Sure, monthly reports may be distributed, but who reads them? Who understands them? An undecipherable monthly report of clearance rates and time-to-disposition data does not constitute a tsunami of information that flattens us in its wake.

I see the really big promise of performance dashboards in their ability to communicate critical performance information quickly, concisely, and widely to people who otherwise would not be privy to that information -- in any form. This is the big promise of the Internet, information technology, and business intelligence (BI) as distribution channels. It is part of the movement from a primarily vertical (command and control) to a horizontal (access, connect and collaborate) solution model. It is the “small shall become big” vision of Thomas L. Friedman’s international bestseller, The World Is Flat (Farrar, Straus and Giroux, 2005). It is the future that Peter Drucker sees in his book Managing in the Next Society (St. Martin’s Press, 2002) where knowledge is spread instantaneously accessible to everyone.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved.

The Key to Self-Renewal and Survival

2007-05-07T10:07:00.000-04:00

The key to self-renewal and survival for an organization is a performance measurement and management system integrated with key management processes and daily operations.

Courts are conservative institutions. Unlike private business, which society will let disappear, courts are here to stay. They are steeped in the traditions of law that favor stability and continuity (precedence) over innovation and change. Like most public inctitutions, courts were originally created to prevent, or at least to slow down, change. Therefore, courts need to work harder than other public and private organizations to counter rigidity and decay that threaten their continuity.

Revolution every so often, as Thomas Jefferson recommended, is a radical response to senile decay and a failure of self-renewal. Obviously, such destabilization to the point of destruction is no way to run a court.

As Peter Drucker has pointed out in several of his writings, to maintain continuity – to survive – organizations must find ways to weave innovation and change purposefully and systematically into the fabric of their leadership and management structures, processes, and daily operations (e.g., see Innovation and Entrepreneurship, Harper Collins, 1993). To maintain continuity courts must find a way to change without destabilization. As contradictory as it may appear, innovation and self-renewal must be seen as necessary ways to continuity and sustainability.

Rigorous performance measurement and management are ways of ensuring both continuity and change. This can only occur, however, only if they are brought off the sidelines into the mainstream of the business of a court.

Performance measurement and management are the hallmarks of successful learning organizations – organizations that have the capacity and the political will to learn from the answers to critical questions:

How are we doing?
Where are we now? What is the baseline from which we start? What is the current performance level compared to established upper and lower “controls” (e.g., performance targets, objectives, benchmarks and tolerance levels)?
How are we doing over time? What are the trends? Is our performance better, worse or flat? How much variability is there and where do we see it?
Why is this happening (analysis and problem diagnosis)? What happened to make performance decline, improve or stay the same. What are some credible explanations?
What are we doing to adapt, improve or maintain (planning)?
What actions and strategies should we start, continue or stop as a result of the measure (strategy)? What should be done to improve poor performance, reverse a declining trend, or recognize good performance?
What performance targets and goals should we set for future performance (goals)?
The willingness and capacity to address these questions – on a continuous and regular basis – to learn, to adapt and, if necessary, to transform are the hallmarks of a high-performing court.

Yes, change and continuity are opposites, but they relate to each other as necessary poles rather than contradictions. Performance measurement and management tie these poles together.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved.

Project Gainshare

2007-05-03T07:26:00.000-04:00

Gainsharing – it’s like profit-sharing in the private sector -- is a system whereby units of government (e.g., a collections division or a jury management unit of a court) share in the gains teams of employees make in their bottom line or that of the state, county or city, without losses in quality of services and programs. Employees receive bonuses or payments based upon the improved productivity or efficiency as reflected in "gains" in cost savings or revenue increases. Gainsharing is based on widely accepted management principles that encourage employee initiative in continuous improvement of program and services to meet the needs of customers and citizens.

In my January 21, 2007 posting I discussed the prospects of gainsharing in courts. The promise for courts, I argued, is that gainsharing may help courts achieve sustained increases in productivity and efficiency, that employees may become more involved in the gains made by the court as they share in the benefits of employee-initiated improvements, that it enhances commitment to organizational goals, and that it leads to improvements in other measures of court performance.

But Will Gainsharing Actually Work in Courts?

Gainsharing has been shown to be successful in local government. But will it work in courts? Have any courts attempted gainsharing?

A search of Google and the National Center for State Courts website yielded no evidence of gainsharing arrangements in state and local courts. An informal survey of courts that included a query of the Court2Court network did uncover one court that has tried gainsharing. It turns out to be quite a success (albeit temporary) story.

The Successful Gainsharing Experiment in California

The United States Bankruptcy Court for the Eastern District of California demonstrated “over a period of five years that gainsharing could work, and work well, in a court environment,” said Richard Heltzel, Chief Executive Officer, when I spoke to him about Project Gainshare, as the pilot project was called. The project returned “over $2.5 million in unspent funds in five short years, an ability to operate to operate in a very lean, but effective fashion, and with enthusiastic staff support at the grass roots level,” said Heltzel.

How did Project Gainshare work? The Federal Bankruptcy Court’s staff of 120 to 150 employees were organized into seven teams across three divisions who managed the project at the grass root level. All but one team fully supported the program at the outset, and the one dissenting team (one of the team’s members said that the effort wasn’t worth “a couple of hundred bucks”) came around later before the effort started. Each team sought “gains” in cost savings in its portion of the Court’s budget.

The teams were wildly successful, making gains over $7 million over a period of five years in the late 1990’s and early 2000’s. Cost savings were distributed in thirds with one third returned to the Administrative Office of the U.S. Courts (AOUSC) as unspent funds, one third plowed back into to the Court for business improvements, and the remaining third distributed to team members in the form of cash payments or “productivity dividends.” (Heltzel wisely exempted himself from receipt of the dividends.) The amounts of cash payments were calculated based on a formula that provided an incentive to the teams. The more the team made in gains, the bigger the dividends received by the team members.

“We deliberately stayed away from the word ‘bonus’ and used instead ‘productivity dividend’ to refer to the cash distribution to employees,” said Heltzel, obviously sensitive to the bureaucratic chaffing that occurs at the mention of “profits” or “bonuses.” (One objection to gainsharing is that salaried government workers should be doing their best every day and do not need incentives like bonuses to do so.)

Heltzel is careful to point out that the Bankruptcy Court had some advantages going into the Gainshare Project that most state and local courts do not currently enjoy, including a non-union, non-civil service employment environment, a relatively clear budget based on predictable annual allotments, and the flexibility to spend them, at least on a pilot basis. He also had a team-based organization, and a relatively sophisticated performance measurement system to back up the gainsharing arrangements – both necessary requirements for gainsharing success. Nonetheless, Heltzel is optimistic about the likely success of gainsharing in court environments. “I am here to tell you,” he said, “that a gainsharing program can work in a court environment when coupled with an organizational performance management system that relies on statistically based measures and customer satisfaction surveys to ensure that the job is getting done in timely and quality fashion.”

Gainshare Meets Untimely Demise

Heltzel’s enthusiasm for gainsharing is surprising, and heartening, given how his bold experiment ended in disappointment for him. Apparently, the savings Hetzel and his colleagues were able to achieve through Project Gainshare were cut from the Bankruptcy Court’s budget as part of a measure by the AOUSC to achieve some kind of parity of base salaries in federal bankruptcy courts. Uniformity fights and beats innovation far too often, I say!

If nothing else, the fate of Project Gainshare, which seems to have had nothing to do with the merits or results achieved by Heltzel and his colleagues, shows us that we need to do a lot of work to prepare and to sustain a receptive environment for any kind of change to stick. As a matter fact, double that work.

Gainsharing, Workplace Health and Organizational Performance

Wherever we look, in the private sector and, increasingly, in the public sector, successful employer practices are empowering employees by involving them in decision-making and giving them more autonomy; providing them opportunities for leadership development; and rewarding employees both monetarily and non-monetarily through performance-based bonuses and pay increases (or, if you prefer, “productivity dividends”) and simple but genuine expressions of thanks. Research has shown that these practices – which are all part of a well-designed gainsharing arrangement -- can combine to produce better employee satisfaction, organizational engagement, and improved organizational performance.

Maybe it’s time to give gainsharing a try in state and local courts willing to innovate. And maybe it’s time for management consultants like me to put our money where our mouths are. Our incentive for participating in a gainsharing program can be the same as those of court staff, i.e., a contingency whereby we work for deeply discounted fees with the incentive that we get a share of the gains.

Any takers?

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved.

Undesirable Variation in Court Performance

2007-04-23T12:19:00.000-04:00

Variation in treatment of court users and court employees, in the time and cost of processing cases, in the reliability and integrity of case files, in the compliance with court orders, and in other key court performance areas is inevitable but generally not desirable. If you had your choice of between processes that produced predictable and consistent results and ones that produced good results one day and bad the next, poor quality under some circumstances and good quality in others, which ones would you choose? Both court managers and the public recognize the benefits of stable processes and consistent and predictable results.

Understanding and controlling variation (e.g., knowing whether particular performance falls outside established upper and lower “control limits”) are at the heart of quality improvement methods such as Total Quality Management (TQM) and Six Sigma. More than 25 years ago, W. Edward Deming and Joseph Juran noted that variability on core measures of performance is a threat to an organization because it is evidence that its business is not being managed effectively, that managers have no control over processes and the results they achieve. Magnitude of variation is an indicator of organizational health. “It all depends,” is not a good answer to the question “How are we doing?”

Fair and Equal

For court leaders and managers, unmanaged variation especially in court users’ encounters with the justice system is a threat to quality, and a possible breakdown of public trust and confidence. What does the public want from courts? What matters to them? Research in procedural justice indicates that they want to be treated equally with neutral and unbiased procedures based on facts and consistent application of rules. In other words, they want the same things that good court managers want, i.e., to reduce variation in processes and results and to stabilize performance at high quality levels.

Managing Variation

It’s not all bad news for court managers. Managing and controlling variation has intuitive appeal. It has to be done at the level where the variation occurs, not at the level of averages or central tendencies. This too is common sense.

We have all seen claims by a major airline that it leads the industry in on-time performance, with departure and arrival data to prove it. Such claims may be legitimate, but we know that our specific flight was not on time, and that it may be the very flight that is never on time. High-level averages have strategic value but they don’t give managers the information to improve performance. The only way to improve performance is to manage and control the variation around the average. (In a July-August 2005 article in Harvard Business Review, John Fleming, Curt Coffman and James K. Harter cite the absurd example of your physician basing treatment of your heart arrhythmia not on an assessment of your heart rate but that of the average heart rate of your town.)

Court users experience variation, not averages. In your court, some users may experience nothing but problems; others are routinely satisfied. It may all depend where they encountered the court, what business they had with the court, and what kind of user (e.g., African-American witness in a civil case heard in the main downtown courthouse) they are. And that’s not good.

Breakouts and Hierarchies of Measures

Performance has to be measured at the right level of specificity for the measurement to be useful. What does a court-wide high score of employee engagement mean to an employee who works in a unit that consistently records miserable scores on the employee survey item “My coworkers care about the quality of services and programs we provide”? When employee engagement and commitment are assessed at the level of the local court unit, court managers can learn a lot about organizational performance.

Creating measurement hierarchies for each core performance measure ensures that important information at the level of court divisions, units, and programs is not masked by exclusive reliance on court-wide averages (e.g., an average court user satisfaction rate of 71 percent or an average case clearance rate of 91 percent). Measurement hierarchies identify opportunities for teamwork and collaboration at the level where the variation occurs (e.g., a particular survey question in a particular courthouse location or the clearance rate of a particular case type). They put the court’s top management in much greater contact with every level of staff by defining the connection – a line of sight -- between high-level strategic goals and performance measures with lower-level departmental or divisional objectives and measures.

The use of breakouts (disaggregation) of performance measures can reveal useful information that otherwise is hidden. Common breakouts of time to disposition measures, for example, are case type and location. They identify differences in timeliness of case processing across different case types and court locations. Other less common breakouts of on-time case processing measures may identify inequities among groups by income levels and indicate whether a court handles cases more swiftly for affluent litigants.

Let’s assume that 73% agreement is the average (aggregate) score across all 15 items on a survey of court user satisfactio. A simple but meaningful referent is the breakout of this score for each of the 15 items. For example, assume that the variation around this average of 73% ranges from a low of 43% for Item 5 (“I was able to get my court business done in a reasonable amount of time.”) to a high of 87% for Item 3 (“I felt safe in the courthouse.”). True, even with these referents, we still don’t know what’s good or bad, but we do now know something about the baseline from which we started measurement (73%) and the range of scores from a particular high and low score. We know something very important that we did not know before -- that it’s possible to reach 87% agreement and to get as low as 43%, and we know that 87% is “better” than the low of 43%, as well as the average of 73%. Similar meaningful referents are the breakouts of the average score for each of the background categories identified in the survey (e.g., the type of case that brought the person to court, or how often the person typically is in the courthouse) and the different courthouse locations in which the survey was conducted. For courts or court systems with multiple locations, comparisons of survey results across locations can be a useful basis for identifying successful improvement strategies. Different locations might be compared, for example, on the percent of users who felt that they were treated with courtesy and respect. Follow-up queries can then be made that probe the comparisons. Why do one or more locations seem to be more successful than others? What are they doing that the other locations are not? Asking staff in both the most successful and least successful locations these simple questions can help to identify “evidenced based” best practices.

Without deeper metrics in a hierarchy of measures, managers would be unable to identify or manage either poor or exceptional performance at its source. Managing variation in performance at the right level of specificity in the hierarchy of measurement has a very powerful advantage: Each court unit can identify and correct its own problems.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved.

Toward the 50 “Best in Class” Courts

2007-04-04T09:30:00.000-04:00

The word “toward” is, of course, the rub. We have today no such list of top court performers that represent the “best in class” for each of the levels of courts: supreme courts, intermediate appellate courts, general jurisdiction courts, limited jurisdiction courts, and specialized courts. Justice Served, the alliance of court management and justice experts headed by my friend Chris Crawford, is in its eighth year of reviewing thousands of court online offerings around the world for the now-famous Top-10 Court Websites awards. Why can’t we do this for court organizations as a whole? Would it attract interest (or objections)?

I have a long-standing interest in comparative performance measurement and benchmarking (see “How Do we Stack Up Against Other Courts? The Challenges of Comparative Performance Measurement,” The Court Manager, Vol. 19, No.4, Winter 2004-2005.) It was rekindled last week when I read the cover story in the March 26 Business Week, “The Best Performers: The Business Week Fifty,” the "best in class" from the 10 economic sectors that make up the S&P 500. Can we do for courts what Business Week does yearly for companies and what Chris Crawford does for court websites?

How We Pick the Top 50

As Business Week acknowledges, any method of ranking organizations will be imperfect. So whatever we do with courts will be a work in progress. But remember: What gets measured gets attention. What gets measured gets done! And we Americans love lists and rankings.

First, we would need to focus on a handful of core performance measures for each level of court. The measures would need to represent a scorecard balanced across key success factors such as access, fairness, timeliness and court user satisfaction. Courts that either do not take these measures or do not make them available would not be included in the rankings. (After all, a court without a website can’t make Justice Served top ten.)

We would need to make plenty of adjustments in the performance data to avoid comparing apples and oranges. This would be tricky but not impossible. (Running data screens and applying weights and algorithms would make great fun for folks like me.)

Second, we would compare courts with their peers at the same level. Within each level of court, we would list the courts according to the core performance measures. That is, courts would be listed according to their combined rankings. For example, a court that finished third in on-time case processing (e.g., time-to-disposition) and ninth in court user satisfaction (a combined score of 12) would rank lower than a court that finished third in on-time case processing and fourth in court user satisfaction (a combined score of 7).

We would construct a final ranked listing of all courts at each level based solely on the results of their performance measures. It would be reviewed by a panel of experts who would make slight adjustments in the rankings to inject wisdom that the numbers may not provide. Finally, like Business Week’s best 50, the ranked listing would be accompanied by profiles of the courts on the list.

Are We Ready for the 50 “Best” Courts

Depending who you ask and how you frame the question, comparative performance measurement is either the best thing that can happen in court administration or the biggest threat to court governance in years. Both sides have legitimate arguments to make. With the emergence of business intelligence solutions that eliminate the barriers to data access and distribution, it may happen regardless of the debate.

In any event, I doubt that there are today 50 courts that would meet the entry requirement of a balanced scorecard of performance measures. However, the Utah courts made a major contribution to this goal when it launched its statewide CourTools initiative. Other states like Arizona, Oregon and Minnesota may not be far behind. Just last week, 40 judges and court managers met in Phoenix at a conference sponsored by the Maricopa County Courts that focused exclusively on the measure of court user satisfaction.

So, which courts will be among the 50 best performers? Will yours be among them?

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved.

Finding the Right Performance Measures Simplifies Strategic Planning

2007-03-19T08:34:00.000-04:00

A strategic plan is useless if it does not lead to strategic thinking and strategic acting. Developing a strategic plan under such circumstances is wasted effort.

The process and steps of identifying desired performance measures serve as a simplified and, in many ways, a superior approach to strategic planning. Because they are unambiguous and actionable, effective performance measures provide a more direct path to strategic thinking and acting than formal strategic planning. A focus on finding the right performance measures also side-steps many of the processes and exercises that engage busy managers (often for lengthy periods of time) – visioning, SWOT analysis, and identifying cognitive styles – but have questionable practical value for making better current decisions.

Plumb the Meaning of Life or Watch Your Cholesterol

Suppose you’re a 50 year old who is taking stock of your life. You’re ready to tackle important issues like personal health, family, and work. Because you’re a fan of Phillip McGraw (“Dr. Phil”) and his book, Life Strategies: Doing What Works, Doing What Matters, and you are somewhat familiar with strategic planning, you decide to do some serious lifetime personal planning using the elements of formal planning.

You start by thinking about preferred futures, about what really matters to you, your values and your mission in life, about your strategic goals for your health, for your children and spouse, and about how good health will benefit your time with them, and so forth. You consider whether you can afford that elliptical trainer that you’ve had your eye on for quite some time. Your wife or husband tells you to go back to the Atkins diet that you abandoned some time ago. Your son tells you to join him at the YMCA where he swims 50 laps every morning at 5:00 am. And your friend tells you to quit your job. You’re overwhelmed by all the competing interests, by the lack of focus, and by the complexity of it all.

In your yearly physical, your doctor commends your resolve to plumb the deep meaning of your life and your commitment to engage in strategic lifetime planning. But he suggests a simpler approach, a more focused way to better health that depends on rigorous tests and benchmarks. He tells you that you need to monitor your cholesterol and your blood pressure on a regular basis. The rest will take care of itself, he says. He tells you that doing something simple like monitoring cholesterol levels and blood pressure improves the health and well-being of many of his patients. In other words, what gets counted and measured gets attention, and gets done.

We, of course, know that cholesterol level is not health, just like case clearance is not productivity or efficiency, and time-to-disposition is not timeliness. Effective performance measures are almost always proxies of outcomes or concepts – they are not the thing itself. To varying degrees, measures are a removed and highly simplified version of the outcome of interest. That’s what makes them so valuable for strategic thinking and acting. They are used in order to make it possible to measure the outcome of actions easily, routinely, and at a reasonable cost. It’s easier to tackle cholesterol levels and blood pressure than the concept of health.

The Same Benefits

The benefits of an effective court performance measurement system are the same as those of strategic planning – accountability, consensus building, focus, coordination, control, learning, communication, hope and inspiration. To identify the right performance measures, a court or court system must address the same fundamental questions about guiding ideals, values, mission, goals and broad strategies as it must address in strategic planning. We count what counts and measure what matters. And what we measure determines what will be considered relevant.

Action Point: Design an effective court performance measurement system and develop a strategic plan as a byproduct.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved.

Counting What Counts

2007-03-03T11:06:00.000-05:00

Infinitely more important than answers are the questions, the choice of them, the inner form of them. Oswald Spengler (1880-1936)

The historian and philosopher Oswald Spengler was talking about larger societal issues, but his observation rings true for court performance measurement. It is not the measure itself that is important but rather the questions it compels us to confront.

How are we doing?
Where are we now (performance level, baseline)? What is the current performance level compared to established upper and lower “controls” (e.g., performance targets, objectives, benchmarks and tolerance levels)?
How are we doing over time (trends)? Is our performance better, worse or flat? How much variability is there?
Why is this happening (analysis and problem diagnosis)? What happened to make performance decline, improve or stay the same. What are some credible explanations?
What are we doing to improve/maintain (planning)?
What actions and strategies should we start, continue or stop as a result of the measure (strategy)? What should be done to improve poor performance, reverse a declining trend, or recognize good performance?
What performance targets and goals should we set for future performance (goals)?

The willingness and capacity to address these questions – on a continuous and regular basis – to learn, and to act accordingly are the hallmarks of a high-performing court.

At best, people commit to the purpose of a performance measure, not to the metric. Checking the levels of cholesterol in our blood is a good example. The cholesterol metrics – very low-density lipoproteins (VLDV), low-density lipoproteins (LDL), hi-density lipoproteins (HDL) and triglycerides -- are things that you would not care about in and of themselves. You care about them because they force you to consider questions about how you are doing, what you should be doing to prevent heart attacks and stroke, and how to live a long and healthy life.

The take-away message is that we measure what matters to us, we count what counts.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved.

Q & A: Do Away With Traditional Performance Appraisals

2007-02-27T15:02:00.000-05:00

Q: Beyond building performance dashboards, assigning “owners” to particular measures, and so forth, how do you hard-wire performance measurement into key management processes?

A: I asked this question today in the Webinar, “Beyond the Balanced Scorecard, Improving Business Intelligence with Better Metrics,” presented by Mark Graham Brown, a consultant with 25 + years experience and author of three books on performance measurement. Here’s what he said.

Do away with the traditional form of annual employee performance appraisal and replace it with a review of the organization’s performance dashboard. Make sure that performance has consequences for the employee, said Brown. A measure in the “red” should lead to negative consequences like salary decreases, and a measure in the “green,” i.e., a measure trending toward performance targets or goals, should lead to salary increases and promotions.

To make this work, ideally, an organization would need to have a sophisticated performance dashboard – Brown refers to these as “fourth generation dashboards” – that have analytics (indices) that provide a line of sight from strategic to operational measures closely related to the work and responsibilities of the employee, and not just what Brown refers to “singular” measures that are more easily manipulated. Nevertheless, I agree with Brown. I can imagine even less sophisticated performance dashboards being used effectively by skilled supervisors as the main reference in employee performance appraisals. Why should a manager who runs a unit or division that consistently fails to hit performance targets and regularly falls below legitimate benchmarks not be held responsible for those measures?

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved

Common Performance Measures Across the Justice System

2007-02-26T13:36:00.000-05:00

Some performance measures are better than others. They bring people together for joint performance across institutional boundaries – the courts, law enforcement, jails, prosecution and defense services. They can be used at the highest policy levels to measure progress toward an overarching purpose and shared strategic goal that the separate institutions are expected to achieve.

Duration of Pretrial Custody

Duration of pretrial custody -- expressed in terms of central tendencies (mean and median days) and ranges of length of pretrial detention -- is one such measure. Because it is clear, focused and actionable across all justice system partners, duration of pretrial custody can be the rallying point of reform and improvement efforts that depend on justice system partners working together.

Use of this measure requires, first, an operational definition -- i.e., getting agreement about precisely what a pre-trial detention day is and is not (see Sub-Steps 2.3 and 2.4 – Identifying Desired Performance Measures). Once that’s done, you need to identify the statistics of central tendency and variation you want for the measure. Depending on the ease or difficulty of getting the required data, I recommend calculating mean and median days, 90th and 99th percentile in days, as well as the number of defendants, percent of defendants, and cumulative percentage in various time periods (e.g., 1-5, 6-10, and so forth). Finally, a coalition of justice system partners should build the business (e.g., identification of system success factors) and technical architecture (e.g., query and reporting application) to support continuous monitoring, analyzing and managing the performance of various partners indicated by the measure.

Rallying Point for Reform

As noted in a M2M posting last year, Breakout Measures for Jail Overcrowding, courts’ use of a measurement of the duration of pretrial detention extends the relevance of commonly used time to disposition measures. This breakout (disaggregation) may be readily accessible via automated case management systems. Many cities like Seattle, Phoenix and Cleveland face mandates to alleviate jail overcrowding. Court leaders and managers can be seen as responsive, and reap considerable political capital in their criminal justice communities, simply by (a) establishing a baseline of the median number of days defendants spend in pretrial custody, and by (b) vowing to examine case processing and important pretrial issues (warrants, initial appearance/arraignment, plea agreements, bail decision making, pretrial services, alternative sentencing) that, if addressed, might reduce the number of days defendants spend in jail. If, in fact, the trend of this measure is downward over time, it can be seen as a demonstration that the court is doing something to alleviate jail overcrowding.

As a common performance measure across the justice system, the duration of pretrial detention measure has legs. Divergence of the mean and median days of pretrial custody may be an indicator of inequality of treatment. For example, long pre-trial detention may be concentrated in a small proportion of the detained population, inflating the mean but not the median. When mean and median diverge, it is a signal to identify the characteristics of those with long stays.

As pointed in the Vera Institute of Justice’s excellent 2003 Global Guide to Performance Indicators, minimizing pretrial detention provides the quintessential opportunity to measure timeliness and responsiveness that depend on the combined efforts of courts, police, prosecution, defense and jails. Successfully implementing large-scale improvement initiatives that depend on the joint efforts across the justice system (e.g., eliminating jail overcrowding or integrated information technology) often run into intractable justice system governance issues that are impossible to tackle head-on. Consequently, the importance initiatives get diverted, unfocused, and little gets done.

As leadership expert Marcus Buckingham says, above all leaders have to be clear. Commitment to a clear, focused and actionable common performance measure like duration of pretrial custody – which is clearly aligned with justice system-wide goals of timeliness and responsiveness – is a way to sidestep sticky issues of governance and, perhaps, deal with them once successful joint effort is demonstrated by a decrease in the duration of pretrial custody. The power in the performance measure is that it is clear, focused and actionable. The measure helps them be that and get right to work without having to spend an eternity rehashing the back-story.

Other Common Measures

In addition to median number of pretrial days, there are, of course, other common measures aligned with key success factors for the justice system such as Access to Justice (Public Participation), Public Trust and Confidence (Interagency Confidence), Expedition and Timeliness, Safety (Risk Management), Fairness and Equality, Integrity (Upholding Rule of Law, Appropriate Outcomes), Accountability, and Innovation in the Public interest (Responsiveness, Reform, Best Practices, Integration of Justice System Components). Examples include:

Ratings of public trust and confidence by users of/participants of the accessibility, convenience, fairness and equality, timeliness, safety, treatment, courtesy, respect, responsiveness and other aspects of justice system treatment of users and participants in services and programs.
Percent change over time of incidents of critical breakdown of established standards of system good practice, professional conduct or rate of deaths or serious injury while in contact with justice system. (This measure encompasses the next two examples as indicators of integrity of the justice system. The rationale is, in part, that high levels of recidivism and low levels of compliance with judicial orders are indicative of breakdown of justice system integrity.)
The rate of client return following service/intervention.
Percent of compliance with judicial orders as a proportion of total orders issued.
Victimization Reporting. Percent of primary and secondary victims who do/do not report to police.
Ratings of employees of their engagement, motivation, and readiness to perform their job.
Percent eligible persons using/participating in justice system programs and services.
Percent of justice system services and programs delivered to users/participants on-time or according to established schedules.
Remands in custody (related to duration in pretrial custody).
Perceptions of safety of individuals in the custody of the justice sector.
No show rate of witnesses

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved

The Yuma Superior Court Launches Its Performance Dashboard

2007-02-17T10:27:00.000-05:00

On Thursday this week, the Superior Court in Yuma County, Arizona, launched the first version of its performance dashboard. In an email introducing it to all court and county employees, Court Administrator Margaret Guidero, who leads the CourTools Design Team developing the court’s performance measurement system, wrote that “together, the six measures of performance will help us to address the question ‘How are we doing?’ and improve accordingly.”

Guidero explained that the current performance dashboard is a static version that soon will be Web-based and dynamic. “In the future, you will be able to view the Court’s core performance measures at a glance and then move swiftly through successive layers of more detailed information,” she wrote. Currently, users have to open six PDF files attached to the static version of the dashboard for more detailed information about the corresponding performance measures. For general definitions of the six measures, as well as four other measures yet-to-be developed, the dashboard points to a link to the CourTools, a model for court performance measurement developed by the National Center for State Courts (NCSC). Finally, for comments and questions about specific measures, she listed the names, telephone numbers and email addresses of six court managers who are the designated owners of the performance measures.

Responsive to Local and State Forces

The Superior Court in Yuma is a rural, small to medium, general jurisdiction court in southwest Arizona. The Court’s performance measurement and management initiative is supported by grants from the State Justice Institute and the Arizona Administrative Office of the Courts. (Full disclosure: As NCSC’s Of Counsel in Performance Measurement, I assist the Yuma Superior Court and its CourTools Design Team with its performance measurement and management initiative.)

Several forces have moved the court toward performance measurement. Among them, at the local level, the Yuma County Administrator’s Office was pressuring the court to develop and report performance measures that support the county’s strategic planning and budgeting efforts. At the state level, the Arizona Supreme Court issued its Strategic Agenda for Arizona’s Courts 2005-2010. It includes five strategic goals and the initiatives adopted by the Supreme Court to attain those goals, and directs local courts to “review the ten core CourTools developed by the National Center for State Courts and implement those measures determined to be beneficial.”

Can Small Courts Build Performance Dashboards?

The short answer is yes they can. My colleagues and I have urged courts and court systems to design, develop and implement automated performance measurement systems supported by multiple interlocking layers of technology and business architecture that translate mission and strategies into clear performance metrics. Further, we’ve recommended that the performance information be communicated quickly and concisely in the form of performance dashboards, scorecards or other computer-screen displays.

But what about courts, especially small courts, that do not have the money to buy, or the technology resources to build, sophisticated computer-based performance display systems? Are such systems a necessary requirement for success?

No, as the Yuma Superior Court demonstrated this week. An effective presentation of a set of performance measures is one: (a) that users can readily access, (b) that is easily read and understood, (c) that is organized for easy navigation among core and subordinate measures, and (d) one that provides a “line of sight” which conveys to everyone what the drivers of success are and provides them with the concrete knowledge of how they contribute to that success. Yuma’s static version of its performance dashboard meets these criteria.

Few courts today operate without spreadsheet programs like Excel to organize, analyze and present data for budget and case processing reports. Computer software that sets up a simple, easy-to-read and easy-to-update system to track performance is merely an extension of such spreadsheet programs. Step 6 of building an effective court performance system (CPMS) should not be read as a necessary requirement to buy or to develop sophisticated performance management computer software but, instead, simply as a strong recommendation to look at increasingly available technology to reach farther, faster, wider and deeper than paper reports can.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved

Transparency in Practice

2007-02-12T08:29:00.000-05:00

Leaders of the Utah state courts are practicing what they preach. Utah’s Administrative Office of the Courts decided to go entirely transparent last month posting all of the courts’ performance measures on its Utah CourTools Measures website.

Most courts mention openness, transparency and accountability in pronouncements of missions or strategic goals. Few put these concepts into practice as have the Utah courts.

In Full View

Consistent with the mission of the Utah courts “to provide the people an open, fair, efficient, and independent system for the advancement of justice under the law,” the Utah Judicial Council in 2003 began developing a court performance measurement system modeled on the National Center for State Courts’ CourTools. That in itself is not noteworthy. A dozen states and many more courts are developing and implementing court performance measurement systems. What is remarkable about Utah’s effort is that its performance measurement system is in full view of everyone with access to a web browser. This means that anyone can view state-wide results, as well as results by individual judicial districts.

Key Success Factors

Six key success factors addressed by the Utah CourTools Measures:
· Court Users’ Satisfaction with Access and Fairness
· Public Trust and Confidence in the Court System
· Case Processing and Management including:
Clearance Rate (cases disposed as a % of cases filed)
Time to Disposition (the number of days cases disposed were active before disposition)
Age of Active Pending Cases (the number of days active cases have been pending)
· Integrity of Court Records
· Collection of Restitution, Fines and Fees
· Employee Satisfaction

Information is provided by court type -- district court, juvenile court, or justice court as it is available. Information provided is typically presented for all courts statewide, by each judicial district and at the local court level. All measures are updated periodically as appropriate.

Model of State Government

Openness, transparency, and accountability are hallmarks of good government. Far from being antagonistic, these principles and judicial independence are necessarily interdependent. Accountability fosters an environment where legislators, executive agencies, and the public understand the judiciary’s role and are less likely to interfere with the judiciary’s ability to govern itself.

The road to judicial independence goes through accountability. This is apparently not lost on Utah’s legislators. In January, representatives of the Utah Administrative Office of the Courts gave a live demonstration of the CourTools Measures to a state legislative appropriations committee. According to Utah State Court Administrator Daniel J. Becker, it received rave reviews and was held up as the model that the rest of state government ought to be emulating.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved.

Measure for Measure … Well, Not Always

2007-01-25T11:01:00.000-05:00

An occasional diversion ……

Some of us who actually believe in the adages “You can’t manage what you can’t measure” and “What gets measured, gets done” sometimes don’t get folks who are not all excited about the power of performance measurement. So I read with great interest a review of Sara E. Igo’s book The Averaged American: Surveys, Citizens, and the Making of a Mass Public (Harvard University Press). The mostly positive review, which appeared in the January 21, 2007 New York Times Book Review, was written by Scott Stossel, the managing editor of Atlantic Monthly.

In appreciation of diversity of thought, I quote Stossel’s last paragraph about how we understand ourselves and our work in its entirety –

Even as we have moved toward ever-finer calibrations of statistical measurement,
the knowledge that social science can produce is, in the end, limited. Is the
statistical average rendered by pollsters the distillation of America? Or
is it grinding down porridge? For all the hunger Americans have always expressed
for cold, hard data about who we are, literary ways of knowing may be profounder
than statistical ones. … Poll-saturated though we may be, our national
self-understanding still comes as much from art (think of Norman Rockwell or
Edward Hopper), literature (think of “The Great Gatsby” or even “The Bonfires of
the Vanities”) and impressionistic journalism (think of James Agee and Walker
Evans, or Joan Didion) as it does any survey. I’m sure that 23 percent of
Americans would agree with me.

For other occasional diversions, click –

Rensis Likert: The Master of Cool (December 18, 2006)
Are Courts Ready for Some Serious Games? (November 07, 2006)
Curling Performance Measures – An Olympic Sweep (January 27, 2006)

Gainsharing in Courts

2007-01-21T17:11:00.000-05:00

“Profit-sharing and other incentive plans are hardly novel in the corporate world,” write David C. Ammons and William C. Rivenbark in the Spring/Summer 2006 issue of Popular Government. “They are regarded simply as good business – good for employees and good for the company and its shareholders,” they say. Ammons and Rivenbark, who are on the faculty of the School of Government at the University of North Carolina, are studying a type of profit-sharing system called “gainsharing” used by local governments in North Carolina and across the nation. I had a chance to discuss gainsharing in state and local courts with Professor Ammons last week.

Defining Gainsharing

In response to lessons learned in the private sector, judges and court administrators are quick to remind us that courts are not driven by profits and, therefore, profit-sharing and similar incentive plans simply are not possible for them. That’s true, but courts do have budgets and balance sheets, and actions that trim costs (or increase revenues from fine collection, for example) and help the bottom line, even though the “gains” are not called profits. Gainsharing is different from profit-sharing programs.

Simply put, gainsharing is a system whereby units of government share in the gains employees make in their bottom line or that of the state, county or city, without losses in quality of services and programs. Employees receive bonuses or payments based upon the improved productivity or efficiency as reflected in "gains" in costs savings or revenue increases. Gainsharing is consistent with widely accepted management principles that encourage employee initiative in continuous improvement of program and services to meet the needs of customers and citizens.

A critical feature of gainsharing is its dependence on measurable results that are within the control of a given unit of government. For example, gainsharing in a court might depend on a court unit or program hitting performance targets such as a specific decrease in cost per case, an improvement in trial date certainty or juror utilization, or an increase in collection of monetary penalties, which positively impacts the bottom line of budgets and balance sheets of the court, its justice system partners, or both. A decrease in cost per case might be seen directly in a court’s balance sheet, while an improvement in trial date certainty may be seen most significantly in decreased expenditures for witness notification by the prosecution.

Gainsharing arrangements vary widely in their design and the degree to which they are integrated into the key management and operating systems of the organization or unit of government. Model gainsharing programs, according to Ammons and Rivenbark, have three characteristics: (1) they focus on opportunities to decrease costs or increase revenues; (2) they include meaningful employee participation (an increase in revenues due solely to a statutory fee increase, for example, would not qualify for gainsharing because it was initiated by legislation and not by actions of employee action); and (3) employees receive bonuses or other benefits based on group success in desired gains.

Gainsharing arrangements focus on joint performance. They are performance based compensation reward systems that tie pay to team or group performance rather than individual performance. Gainsharing rewards are not necessarily limited to individual employee bonuses or payouts -- an arrangement that may be problematic for courts tied to authorized forms of payment to public employees. Group success in achieving desired results could be rewarded by group awards, both monetary and non-monetary. Local governments in North Carolina, for example, have adopted various forms of what Ammons and Rivenbark call ”greater-flexibility-for-greater-accountability exchange” whereby counties reward departments with managerial flexibility (e.g., the ability to shift funds, adjust positions, and carry over unspent funds from one year to the next) in exchange for demonstrable results.

Gainsharing and Performance Measurement

Gainsharing and performance measurement go hand in hand. On the one hand, the accountability that derives from an effective court performance measurement system (CPMS) directly addresses concerns that services quality in gainsharing arrangements may decline as employees cut corners to create cost savings. For example, to qualify for gainsharing, decreases in cost per case would need to be achieved while maintaining specific levels of court user satisfaction and case clearance rates.

On the other hand, gainsharing provides specific opportunities to integrate performance measurement and management with key management processes and practices that are involved in gainsharing. Of course, the more they are integrated into the day-to-day operational systems, the more commitment there is to performance measurement and management. As I have argued in previous posts, for performance measurement to be truly effective in changing the way a court does business, performance measures have to be hard-wired into the very DNA of the court’s leadership, management and organizational culture. (See Made2Measure, Needed: An Instructions Manual for Implementing Performance Measurement, January 7, 2007). Gainsharing arrangements may achieve this for some measures.

For employees to be properly motivated, they must understand what they need to do - specifically - to make the performance happen. Properly designed gainsharing arrangements will make it clear to employees in terms of unambiguous and actionable performance measures what needs to be done for the gainsharing goals to be achieved.

Will Gainsharing Work in Courts?

The promise of gainsharing is that it may help courts achieve sustained increases in productivity and efficiency, that employees may become more involved in the gains made by the court as they share in the benefits of employee-initiated improvements, that it enhances commitment to organizational goals, and that it leads to improvements in other measures of court performance.

Gainsharing has been shown to be successful in local government. Hospitals also have instituted types of gainsharing programs that have had favorable results (see Made2Measure, Pursuing Perfection – A Lesson from Health Care, November 6, 2005). A search of Google and the National Center for State Courts website yielded no evidence of gainsharing arrangements in state and local courts. Would gainsharing work in courts?

Putting aside questions of their legality and opposition to them on philosophical or political grounds (e.g., wages paid to employees already oblige them to do the best they can), gainsharing arrangements seem well suited for many court programs and services such collections of fines and fees and jury utilization where performance levels can easily be quantified in monetary terms.

That it may integrate performance measurement with a court’s day-to-day operations, may be enough of a promise to give gainsharing a try in state and local courts.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved

Needed: An Instructions Manual for Implementing Performance Measurement

2007-01-07T17:59:00.000-05:00

We can easily imagine how we might use a hammer or some other practical tool. Not so for conceptually complex management tools like performance measurement. And therein lies a problem.

Promises

Sure, we’re quick to claim that performance measurement improves performance and accountability, that it can change the way courts do business. We point out that performance measures are unambiguous and actionable, that they serve as incentives for change.

We can even pony up of a nifty list of uses for performance measures:

translate vision, mission and broad goals into clear performance targets;
communicate progress and success succinctly in the language of performance measures and indicators;
respond to legislative and executive branch representatives’ and the public’s demand for accountability;
formulate and justify budget requests;
respond quickly to performance downturns (corrections) and upturns (celebrations) in performance;
provide incentives and motivate court staff to make improvements in programs and services;
make resource allocation decisions;
set future performance expectations based on past and current performance levels;
insulate the court from inappropriate performance audits and appraisals imposed by external agencies or groups; and,
communicate better with the public to build confidence and trust.

But what, precisely, are we doing when we are doing these things? I can visualize myself clearly swinging a hammer to build shelves in our utility room, but I have no such clarity in imagining what I might be doing when I use performance measurement to justify budget requests, for example.

How precisely do we use court performance measures as part of performance-based budgeting or management-for-results approaches initiated by city, county or state executive branches? How precisely do we respond to performance downturns? I fear that until we’re able to answer such questions, court performance measurement will not fulfill its promise.

No Field of Dreams

Previous posts (Implementing Performance Measurement, November 12, 2005; Implementation: How It Looks When You Get There, December 13, 2005; and Eight Tips for Making Use of Performance Measurement, October 9, 2006), explored what it takes to implement a court performance measurement system (CPMS). In these posts, I suggested that is one thing to build a CPMS and quite another to get it to be used effectively. Even a well-conceived, well-designed CPMS, will not necessarily get used unless it is woven into the very fabric of a court’s management practices and processes. “If you build it, they will come” works in the movies. (In the 1989 film Field of Dreams, an Iowa corn farmer played by Kevin Costner hears voices that tell him, “If you build it, they will come.” He interprets this message as a command to build a baseball field on his farm. He does and they -- Shoeless Joe Jackson and the other seven Chicago White Sox players banned from the game for throwing the 1919 World Series – come.)

For performance measurement to be truly effective in changing the way a court does business, performance measures have to be hard-wired into the very DNA of the court’s leadership, management and organizational culture. Tips for achieving this are a good start but they are not enough. What’s needed is the kind of specific “how to” instruction my home improvement manual gives me for using my hammer to install an electrical box, to put up shelves in the utility room, and to tear down a wall.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2007. All rights reserved.

A Framework of Business and Technology Architecture Supporting Performance Management

2006-12-31T15:18:00.000-05:00

Over the last several years, my colleagues and I at the National Center for State Courts have recommended that courts design, develop and implement automated performance measurement systems. Such a system is supported by multiple interlocking layers of technology and business architecture that translates a court’s mission and strategies into clear performance metrics. We agree with Wayne W. Eckerson, the director of research and services for the Data Warehousing Institute and the author of the 2006 book, Performance Dashboards: Measuring, Monitoring, and Managing Your Business (John Wiley & Sons, Inc.) that the system should be built on a multilayered business intelligence and data integration architecture including the processes, tools, and technologies required to turn data into information and knowledge. Further, we’ve recommended that the performance information needs to be communicated quickly and concisely in the form of performance dashboards, scorecards or other computer-screen displays.

Just like a car’s instrument panel is built simultaneously with the car’s operational systems, a performance dashboard should be developed in coordination with the design and development of a court’s production or operational systems (e.g., case management, financial and jury management systems). Surely, car makers do not think about the design and development of the car’s dashboard only after all the other pieces of the car are built and finished!

A Framework

According Eckerson, an effective performance management system consists of business process (organization) architecture and technology architecture, each with one or more elements at various layers. The figure below shows the two parts and multiple layers of this architecture and how they relate to each other. Specific elements or components -- methods, tools and technologies -- may appear in one or more layers of this architecture.

Business Architecture

The business process architecture (top of the figure) has four layers. The layers and the elements contained in them should be harmonized and aligned.

The top layer includes the perspectives and expectations of a court’s internal stakeholders (e.g., judges, managers and staff) and external stakeholders (e.g., the court’s justice system partners and the public) which, ideally, are reflected in the outputs and outcomes of the court’s program and services.
The key success factors or major performance areas (e.g., access, timeliness, efficiency, and fairness) which, again ideally, are aligned with stakeholder expectations.
The strategies (e.g., case management) and tactics (e.g., calendaring) of a court.
The performance measures aligned with the foregoing (e.g., time-to-disposition, clearance, case inventory or backlog, trial certainty, and cost per case).

The first two layers of the business architecture contain the perspectives and expectations of internal and external stakeholders and key success factors, expressed in terms of values, visions, mission statements, goals, strategic plans, budgets, and objectives. Ideally, all the elements in these first two layers are performance-based and oriented toward specific performance indicators and measures.

The third layer of the business architecture contains the specific strategies and tactics to achieve goals. The strategies and tactics are shaped by the performance-based orientation of the elements of the first two layers, and further clarify the specific performance required to achieve the strategies. Finally, in the bottom layer of the business architecture, a court translates strategies and tactics into specific performance measures, and performance targets and objectives.

Technology Architecture

The technology architecture has five layers. From top to bottom in the figure, they are:

Performance dashboards, scorecards, and other displays of the performance measures populated by data moved up and through the layers below.
A layer including software applications for monitoring, analyzing and managing performance data for strategic, tactical and operational decision-making.
A data store layer which gives users access to the performance data for purposes of monitoring, analysis and management.
An integration layer for extracting data from data sources.
A layer of data sources including legacy systems running on mainframes, packaged applications running on relational databases, Web pages, Excel spreadsheets, Access databases, and other data sources.

This summary of the five layers of technology architecture of a performance management system is intended only to highlight functionality, and not necessarily to identify specific tools and technology in these layers, especially, emerging technology like semantic composite applications. Emerging technologies in Web services and composite applications in service-oriented architecture (SOA) may especially blur the distinctions among the three middle technology layers – applications, data stores and integration. (For example, although the functionality of data stores is critical, service oriented architecture (SOA) has made large data warehouses a thing of the past. Further, composite application platforms eliminate a number of issues of integration such as copying data from one location to another that ETL (extraction, transforming and loading) and EAI (enterprise application integration) tools are designed to do. See Raden, Neil. “Start Making Sense: Get From Data To Semantic Integration.” Intelligent Enterprise, October 1, 2005. Accessed at http://www.intelligententerprise.com/showArticle.jhtml?articleID=171000640.) The functionality highlighted by these layers, however, is critical for the consideration of how a court might in improved business and technology architectures.

Aligning the Architecture

As the two arrows in the figure above suggest, performance measures – first, their operational definition in the bottom layer of the business architecture and, second, their display in the top layer of the technology architecture -- tie the two parts of the architecture together. Without harmonized business and technology architecture courts are unlikely to execute a coherent strategy and its technical team will fail to deliver viable performance information system.

On the business side, a court must translate mission and goals into strategy and tactics, strategy and tactics into objectives and, finally, objectives into targets, clearly identified in the language of performance indicators and measures which are easily accessible to users. Ideally, all these layers are harmonized and work together. Performance measures focus and define in operational terms the expectations of a court’s internal and external stakeholders, its key success factors or major performance areas, its strategies and its tactics.

On the technology side, performance measures determine what a court’s internal and external stakeholders monitor (what they see on performance dashboards, scorecards or other performance data displays), analyze and manage. A performance dashboard, scorecard or other display consists of software applications that monitor, analyze and manage performance. They include the rules that define what data to collect, how it is calculated, and how it is aggregated and disaggregated.

A court does not necessarily need to have a specific number of components in each layer of the business and technology architecture (e.g., a certain number and types of performance measures as part of its business architecture; or specific integration tools as part of its technology architecture). Instead, the number and types of components should be driven by its needs. However, it should have at least one component at each layer.

Rensis Likert: The Master of Cool

2006-12-18T11:34:00.000-05:00

Just when you thought you had the bean counters, spreadsheet guys, and stat boys on the run comes Wired Magazine to remind us that measurement is downright cool. “Human beings measure things,” writes Lucas Graves (“Disparate Measures,” December 2006). “It makes us feel smart and helps us to understand our surroundings.” Graves’ evidence is some of the scales we use to parse, codify, and compartmentalize the world.

The most familiar among the “six of the coolest scales” cited by Graves is the Saffir-Simpson Hurricane Scale. The five categories meteorologists talk about represent the probable damage to man-made structures caused by a hurricane based on wind speed and storm surge. There’s the Mohs Hardness Scale developed in 1812 by Friedrich Moh to gauge the strength of a mineral. His method rates hardness by staging a catfight between minerals. A rock that can scratch another rock ranks higher than the one it scratched. My favorite is the Glasgow Coma Score to measure level of consciousness. Comatose patients get a score for their responses to various stimuli along three axes -- eye opening, verbal response, and motor response. To give us some context, Graves notes that carrots score a 3.

Granted, the scales we use in the courts may not excite the imagination as much as those we might use to judge the hardness of our sapphires while we’re in a coma during a hurricane, but we do all right. The surveys of the CourTools measures of Access and Fairness or Employee Opinion use a five-point scale developed in 1932 by the American educator and organizational psychologist Rensis Likert. The surveys measures the strength of a respondent’s agreement with a clear statement like “Court personnel treated me with courtesy and respect” along a five-point scale: 1=strongly disagree, 2=disagree, 3=not sure, 4=agree, and 5=strongly agree. The results, obtained by calculating averages and disparities, make us feel smarter and help us to understand the world we live in.

How cool is that?

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2006. All rights reserved.

Standardized Measurement Reports Generated by Automated Systems: An Afterthought?

2006-11-30T10:31:00.000-05:00

Automated case management systems -- as well as other automated systems for finance, jury utilization and management, fine and fee collection, and other court functions -- typically have a “reporting” functionality. Users are able to view various standardized reports generated by the automated systems.

An Example

For example, using the Odyssey case management system, developed by Tyler Technologies Inc. of Plano, Texas, the Clerk of the Twentieth Judicial Circuit in Ft. Myers, Florida (and, at least theoretically, any individual or agency with inquiry access to the system), can access over 60 standardized reports under the following headings: (1) case analysis (e.g., cases without activity, listing cases without base events that have been filed for a specified period of time); (2) case administration (e.g., case load activity report, a summary report of court activity indicating changes in counts and percentages of cases filed and disposed from the start to end dates for any date range); (3) Florida State Reports (e.g., collection rate report, percentage of collections based on the assessed amount over a series of quarters); (4) workflow (e.g., scanning activity report, displays daily or hourly totals, per employee, of documents and document pages scanned; (5) financial reports (e.g., cashier accuracy, shows cashier till counts and totals over a date range); (6) financial activity (e.g., GASB 34, calculates how much money was assessed during one date range and how much was collected against those assessments in another date range); (7) registry and trust (e.g., account journal, a listing of one or more registry account type transactions during a user-specified time period); and (8) bond and warrant reports (e.g., bond activity and outstanding warrants, detailed information about bonds with a particular status within a specified date range or a list of all currently active bonds).

A Shopping List

How are such standardized reports and their data elements identified and developed? The short answer is not very well. Too often the process is an afterthought that produces a virtual shopping list of reports, most of which are not used. Typically, here’s the way it’s done. (With a little hyperbole to drive home the point.)

At some time during the development of an automated system -- maybe as long as a couple years before installation and launch -- the vendor convenes a group of prospective court users in a brainstorming “requirements” session to address the question, “What kind of reports would you like to see generated by the automated system?” The group typically has no organizing or conceptual scheme to guide them except, perhaps, an example of a list of reports other jurisdictions have generated. Also, no one in the group will have seen the automated system in operation, except during a short visit to a jurisdiction that currently is using or testing it. Because the reporting functionality of the system is seen as less important than functionalities related to production and operations, and because “reports” are seen as a specialized requirement, the members of the group tend to be operational level managers and analysts (the bean counters). The strategic or even tactical decision makers of the court, its leaders and top managers, are conspicuous by their absence.

Nonetheless, a tentative list of reports, described in a sentence or two, is generated by the group based largely on the interests of the individual members. An indirect nod to any strategic measures may be given by requiring the vendor to include “state compliant” reports, meaning that the system should be able to generate the reports currently required by the state. Other than that, the list of reports tends to be mostly operational level listings and statistics – a virtual shopping list of individual interests.

The group sends the tentative shopping list to court executives and managers for their review, comment and approval. The response is meager and superficial. The group makes the final revisions, and the reporting functionality of the automated system is built and installed. Not surprisingly, a year after installation, only a fraction of the shopping lists of standardized reports is ever downloaded for use.

A Better Way

While it is, of course, reasonable that the primary emphasis of automated management systems is the production system itself, i.e., the creation of documents and guides of the particular business process on a day-to-day basis, it is unfortunate that the reporting functionality seems to be an afterthought divorced from strategic thinking. Do car makers think about the design and development of the car’s dashboard only after all the other pieces of the car are built?

It is surprising that even courts that embrace the Court Performance Standards and Measures or the CourTools have not used the concepts of core performance measures and measurement hierarchies to develop the reporting functionality of automated systems. Even courts that have not developed comprehensive court performance measurement systems (CPMS) based on the Standards or the CourTools, can greatly facilitate the identification of desired reports generated by an automated system by using an organizational scheme of strategic, tactical and operational level of reports of performance. This is the scheme that is at the heart of core performance measures and measurement hierarchies.

Measurement hierarchies in which lower-level subordinate measures, and corresponding standardized reports, “cascade” down from core measures help align the overall goals of the court with the goals and objectives of its divisions, units, and programs. They help make it clear to all court employees precisely how their actions help fulfill the court’s mission and strategic goals. For example, a clerk in a court’s jury commissioner’s office, will recognize that an improvement in the percent of undeliverable mail – a lower-level subordinate performance measure which may have its own report generated by an automated jury system -- drives juror qualification yield, which in turn drives the overall juror yield, which drives juror representativeness, a high-level core measure of fairness and equality.

Using the concepts of core performance measures and measurement hierarchies as an organizing scheme to develop the reporting functionality of automated systems can transform a shopping list divorced from the strategic goals of a court into a powerful tool to manage performance.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2006. All rights reserved

In Praise of Employee Satisfaction

2006-11-22T19:06:00.000-05:00

After participating in two judicial conferences recently, in which the idea of surveying court employee opinions (see Measure 9 of the CourTools) was met with considerable skepticism, especially by judges, I was particularly struck by a Toyota magazine ad that pictured a group of smiling and seemingly satisfied Toyota plant workers. The ad read in part:

Being a good corporate citizen starts with hiring lots of good citizens. What’s a good corporate citizen? It’s about people. People who care about what they do and how they do it. And at Toyota, we know these people pretty well. Because we hire them every chance we get. … [They] take pride in everything they do. Quality, teamwork and dependability, that’s what they are all about. [They] care about doing what’s right; at work as well as in their communities. They really are good citizens. Which in turn makes Toyota a better corporate citizen. Isn’t nice when things work out?

In an article titled “No Satisfaction,” Charles Fishman (Fast Company, December 2005/January 2007) writes about why Toyota thrives as other carmakers struggle. Toyota’s “genius” is a competitiveness that is “quiet, internal, self-critical,” says Fishman. “It is rooted in an institutional obsession with improvement that Toyota manages to instill in each one of its workers, a pervasive lack of complacency with whatever was accomplished yesterday.” The Toyota workers apparently learned that, as Lily Tomlin noted, the road to success is always under construction.

Caring Deeply About What We Do

Toyota and other companies realize what many courts seem not to appreciate in practical terms. Namely, organizations succeed when they get their employees to care about what they do and how they do it. It’s that profoundly simple. Sure, most court managers will acknowledge that the engagement, commitment, energy and enthusiasm of the court’s workforce is a valuable asset, but is their recognition anything more than a cliché – like continuous quality improvement – that is often repeated but has little practical meaning?

That’s where Measure 9, Court Employee Satisfaction, comes in. Its purpose is to assess and to improve how employees think about the work that they do. It asks court employees to register their agreement with simple statements -- like “My coworkers care about the quality of services and programs we provide” and “I am proud that I work for the court.” Managers can slice and dice the results by individual statement, by department or unit, and by court location. They can learn and improve.

Consider this. “Someone in the court cares about me as a person,” is an item in the Court Employee Satisfaction Survey. We know that people who have a best friend at work are seven times more likely to be engaged and committed to their jobs. Seven times! They get more done in less time, are more innovative, and more likely to share new ideas. (See Made2Measure, Friendships in the Workplace Good for Court Performance, August 14, 2006).

We Need to Be Better

So what about the skeptics I encountered at the two judicial conferences who thought employee surveys are a waste of time? I know that some believe – I know because they told me so – that surveying employee satisfaction is a bit “touchy-feely” and “too subjective,” not the hard-nosed business approach to management that they think is expected of them. Looking at measures like clearance rate and time-to-disposition makes sense, but trying to instill meaning and a different thinking about the work we do in the courts seems too much like group therapy. It makes them uncomfortable, they say.

This attitude is, I think, changing slowly as more and more courts experiment with measures of employee satisfaction, growth and learning. For example, the Court Employee Satisfaction Survey was conducted on a statewide basis in Utah in September. The results are available on the court systems’ intranet site and the Administrative Office of the Courts will be focusing efforts to address areas for improvement identified by the survey results.

But I think we need to do a better job at convincing the skeptics. We need to point out that paying attention to workforce strength is about as hard-nosed as it gets. We need to demonstrate that individual workers – judges and other court employees – are not necessarily limited to the productivity of 1 FTE (full-time-equivalent) worker, as they are in workload assessments. We need to be better at showing that one engaged employee who cares about what her or she does and how it is done can do the work of many.

Think of what a whole team, an entire department, could do.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2006. All rights reserved.

Are Courts Ready for Some Serious Games?

2006-11-07T18:33:00.000-05:00

On a recent flight, I sat next to Robert Hone, creative director of Red Hill Studios in Larkspur, California. He was on his way to the Serious Games Summit held October 30 and 31 in Washington, D.C. The summit, Hone explained, brings together game developers, buyers, and industry professionals to exchange ideas and advance the state of the art of serious games for government, professional training, education, healthcare, military, science, and social change. Hone, it turns out, designs serious games for a living. We talked about serious games for court managers.

Serious games (SGs), according to Wikipedia, are “computer and video games that are intended to not only entertain users, but have additional purposes such as education and training.” The term serious games came into wide use in 2002 when the Woodrow Wilson International Center for Scholars in Washington D.C. launched the Serious Games Initiative to encourage the development of games that address policy and management issues. The development of on-line multi-player gaming such as Second Life has heightened interest in serious games.
While serious games can be considered a kind of entertainment, their main goal is not fun and playability. Most are designed to elicit socially relevant behavior, which distinguishes them from commercial entertainment games. Another attribute that distinguishes serious games from educational videos is that “you can win and lose a serious game,” Hone explained.

Advantages of Serious Games

A serious game may have the look and feel of a video game, but it is actually a simulation of real events or processes. While the largest users of serious games are the U.S. Government and medical professionals, other public and private sectors increasingly are recognizing the benefits of simulations in serious games. One advantage is cost. Video and computer game developers are skilled at quickly creating games that simulate real conditions. According to Wikipedia, the costs of specialized hardware and media for serious games are low. Traditional simulators usually cost millions to develop and to deploy. Serious games are designed to run on personal computers or vide game consoles and require nothing more than a DVD or even a single CD-ROM.

Another advantage of serious games lies in their ability to engage and entertain users, increasing motivation to play repeatedly. Because their livelihood depends on it, game developers are adept at making serious games fun. In their simulation of real events and processes, they are accustomed to injecting entertainment into their applications.

Social Impact Games

Among the 50+ serious games and developers listed by Wikipedia are the following:

Darfur is Dying (Internet) An online game by mtvU that simulates life in a Darfur refugee camp.
3rd World Farmer An online simulation game where the player gets to experience the hardships of 3rd World Farming.
Eduteams (PC) An immersive team-based software package built to ensure effective development of core and enterprise skills for young people, developed by TPLD.
Infiniteams (PC) Multi player real time strategy game, which focuses on leadership development and team building, developed by TPLD.
LegSim: Legislative Simulation (Internet) Web-based virtual legislature used in college and high school government and civics courses.
Real Lives 2004 (Microsoft Windows): Life simulation that gives players the opportunity to learn how people really live in other countries.

Justice Delayed – The Game

“What would it take to develop a serious game for courts?” I asked Hone. We imagined “Justice Delayed,” a serious game that would simulate conditions of delays and related problems in case processing defined by performance data from the case processing measures of the CourTools. Related conditions and data from a court’s justice system partners – jail overcrowding and increased volume and costs of witness notification by the prosecutor – would provide additional events, processes and data to complicate a simulation.

Game players would make certain assumptions about conditions and be armed with tools and techniques in the form of good caseflow management principles and processes. They may take various roles – a judge, a court administrator, a director of a court program or service (e.g., interpretation), and external stakeholders such as jail administrator, defense attorney and prosecutor. Winning and losing would be defined by the outcomes of the CourTools and related measures.

Excluding distribution costs, which I said might be borne by national and state court organizations, Hone said that “Justice Delayed” could be done for $100,000 - $300,000. He sounded interested.

For the latest posts and archives of Made2Measure click here.

Pursuing Perfection – A Lesson from Health Care

2006-11-01T10:44:00.000-05:00

Regina Berman cares about performance. It’s in her job title -- she’s the administrative director of performance improvement at the Hackensack University Medical Center in Hackensack, New Jersey. Her experiences at Hackensack and those of other hospitals who are doing the same things have something to teach courts.

Improving Quality

Hackensack and Berman worked for seven years to improve quality and are at the leading edge of a nation-wide movement to reform health care that relies on performance measures. The hospital’s heart attack mortality rate is consistently about 2% lower than the national average, which ranges from 6% to 10%. Other hospitals want to know Hackensack’s secret, but Berman says it doesn’t have one. Hackensack has developed policies and procedures to ensure patients get the care that prevents harm and save lives as evidenced by performance data. Throughout the 781-bed hospital, staff monitor and analyze every process, looking for ways to save time and avoid errors, Berman says. The hospital owes its success to teamwork and good communication about results. “Any hospital could do any of these things, if they understood how beneficial it would be for patients,” she says.

Its track record earned it a $1.9 million grant in 2001 from the Robert Wood Johnson Foundation and the Institute for Health Care Improvement to participate in the Pursuing Perfection program. A principle of the program is that you don’t have to trade quality for cost. “It’s just the other way around: More reliable systems are lower cost,” says Donald Berwick, the president and chief executive office of the Institute.

Measuring the Rate of Harm

The McLeod Regional Medical Center in Florence, S.C., another hospital in the Pursuing Perfection program, has dramatically reduced medication-related injuries. In 2001, the hospital took a baseline measure of its “harm rate” – defined as the frequency at which patients are injured because of a prescribed drug. The medication-related harm rate turned out to be 3.5 per 1,000 doses or about 35 patients per day. Even though this rate was at the low end of the national average, hospital leaders thought it was too high. The hospital staff took a close look at all their routines and made simple changes to reduce errors – they separated drugs with similar names to avoid mix-ups, they began using bar codes to match the right drug with the right patients, and they decreased the steps to fill a prescription. McLeod also tried to replace a “culture of blame” with an attitude focused on improving outcomes for patients. Its harm rate has fallen by 90%, says Natasha Nicol, director of pharmacy services.

Good for the Bottom Line

To be sure, the quality initiatives of Hackensack and McLeod are good for patients. But are they good for the bottom line? Among the reasons given not to measure court performance, is that it takes too much time, effort and money. It hurts the bottom line. The experience of Hackensack proves otherwise.

Hackensack’s quality initiatives have made the hospital more efficient and increased revenues. To avoid getting penalized for helping patients get well sooner and discharging them faster, Hackensack persuaded managed-care companies to agree to higher payments. It also received $850,000 last year from the Centers for Medicare & Medicaid Services as part of an innovative experimental pay-for-performance program that pays hospitals partly based on their performance on 33 standards of care.

The experimental program, which includes 260 hospitals, is a dramatic departure from the way insurance companies typically reimburse hospitals based on the volume and complexity of cases instead of performance outcomes in various categories such as heart bypass surgery, heart failure, pneumonia and hip and knee replacements. Top hospitals like Hackensack get bonuses and low-performing hospitals could see their payments reduced by one to two percent.

Are Courts Ready and Willing?

Are courts ready for a pilot program that “rewards” them with additional resources based performance improvements, instead of just for the volume and complexity of their cases? Following the example of the managed-care companies who were willing to adjust their payments to Hackensack based on quality care, would states, counties and cities provide courts with modest monetary incentives to increase trial certainty, reduce delay, improve enforcement of orders, and satisfy those using their services and participating in their programs?

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2006. All rights reserved.

Deliberate Practice of Performance Measurement

2006-10-30T20:25:00.000-05:00

Top performers engage in what K. Anders Ericsson calls “deliberate practice” – an effortful activity designed to improve individual target performance. Ericsson, a professor of psychology at Florida State University, has spent 25 years interviewing and analyzing high-performing professionals and is the co-editor of the Cambridge Handbook of Expertise and Expert Performance (Cambridge University Press, 2006). He claims that, except maybe in some sports, elite performers aren’t genetically superior. They just do some things differently – like monitoring and managing their performance using established benchmarks.

“Successful people spontaneously do things differently from those individuals who stagnate,” Ericsson said in a recent interview in Fast Company (November 2006). “They have different histories.” He explained how deliberate practice with an example of a medical technician who may see a patient once or twice, make a diagnosis, and then move on, and never see the patient again. A highly successful medical technician who Ericsson interviewed works very differently. “He spends a lot of his time checking up on his patients, taking extensive notes and what he’s thinking at the time of the diagnosis, and checking back how accurate he is.” This extra step gives him a real advantage over his peers.

The take-away message from Ericsson’s work is that the “deliberate practice” of performance measurement is not just good for the “patient” and the medical organization, but good for the individual who engages in it. If you’re a court professional who wants to excel, you’ve got to monitor, analyze and manage your performance deliberatively. You’ve got to know what’s working and what’s not. You’ve got to learn from that knowledge and act accordingly.

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2006. All rights reserved.

Progress Boards, Legislative Mandates and Judicial Branch Responses

2006-10-20T22:50:00.000-04:00

Oregon state government spends over 43 billion dollars biennially. What are Oregonians getting for their money? Are they getting the right results? Assuming they are, are they being produced in the most efficient manner possible? In a recent brief, Rita Conrad, Executive Director of the Oregon Progress Board, and Dawn Farr of Oregon’s Legislative Fiscal Office say that Oregon’s performance measure system is getting better at answering these questions. And I think they’re right.

Does your state have a “progress board” or similar state agency pushing performance measurement and results-based management?

Whether the judicial branch leaders and managers regard actions by state progress boards as a threat to independence or, alternatively, as a welcome challenge, state progress boards are a source of valuable information. The take-way message of this post is that court executive and managers should take a serious look at what progress boards and similar agencies have to offer in the way of guidance for court performance measurement efforts.

The Oregon Progress Board

Progress boards exist in Oregon and North Carolina, and in British Columbia and Tasmania. The Oregon Progress Board, an independent state planning and oversight agency, may be the longest-standing and the most influential. Created by the Oregon Legislature in 1989, the Board is responsible for monitoring progress toward the state's 20-year strategic vision, Oregon Shines, based on a state wide system of performance measures. Chaired by the governor, the 12-member Board is made up of citizen leaders and reflects the state's social, ethnic and political diversity.

Legislative Mandate

Oregon law requires the Progress Board – in consultation with the Legislative Fiscal Office, the Office of the Secretary State, and the Department of Administrative Services -- to develop guidelines for performance measurement by state agencies. In turn, agencies are required to use performance measures to drive achievement of their missions, goals, objectives and any applicable benchmarks. (Not far below the surface of any discussion of legislative mandates is the issue of whether courts subject to the requirements of executive and legislative “agencies”? See Independence, Accountability and Performance Measurement, Made2Measure, September 29, 2006.)

Source of Good Information

The Oregon Progress Board has produced a vast amount of information that should be instructive for any court systems or individual courts considering performance measurement initiatives. It’s most visible work is the Oregon Benchmarks, a set of 90 high level, societal indicators organized into seven categories: economy, education, civic engagement, social support, public safety, community development and environment. The benchmarks define Oregon’s strategic goals as measurable outcomes, with specific targets for improvement. Oregon state agencies are required to link their key performance measures (KPMs) to them. All Oregon agencies have identified KPMs aligned with the agency’s strategic plan and, where pertinent, with one or more Oregon Benchmarks.

Oregon’s benchmarks and KPMs have attracted much attention outside Oregon. Every state and more than a dozen foreign countries have requested benchmark reports and several states have adapted the benchmarks for their own uses.

Driving the Bus

Faced with a choice of getting on or getting hit by the performance measurement “bus,” the Oregon courts decided to step on the bus, but not as passive spectators. Instead, they jumped into the diver’s seat. The Oregon Judicial Department’s (OJD), and its state-wide Performance Measurement Advisory Committee (PMAC), developed 10 “legislatively adopted” budget 2005-07 performance measures: (1) access to justice measured in terms of certified interpreters; (2) collection of monetary penalties; (3) data timeliness and accuracy; (4) workforce representativeness; (5) workforce training: (6) timely case processing; (7) percent of circuit courts with permanency action plans: (8) drug court recidivism; (9) juror satisfaction; and (10) capacity for serving litigants not represented by lawyers.

Although a number of these measures can be faulted (e.g., a focus on inputs and outputs instead of outcomes), they are successful because they have ensured that the courts remain in the driver’s seat as they improve and enhance the measures. As one astute OJD observer quipped, “I’d rather have our measures in their [the Legislature’s] hands, than forced to have their measures in ours.”

For the latest posts and archives of Made2Measure click here.

© Copyright CourtMetrics 2006. All rights reserved.

Eight Tips for Making Use of Court Performance Measures

2006-10-09T12:09:00.000-04:00

The road to success is always under construction.

LilyTomlin, American Actor and Comedian

Earlier this year, Made2Measure posted 10 tips for designing performance measures. Once you’ve designed the measures, here a eight more to ensure that the measures get used.

Tip #1. Build a performance dashboard, scorecard or other effective display of your performance measures. It makes no sense to take the steps of designing the right performance measures and assembling critical performance information if the data are not delivered to the right people, at the right time, and in the right way. That’s where performance dashboards, scorecards or display systems come in. The goal of creating a performance measurement display system is to provide intended users with meaningful information that they can quickly and easily access, assimilate and understand. An effective display enables court leaders, managers and other users to monitor, analyze, and manage the critical processes and activities needed to achieve goals. Computer-based performance displays -- often referred to as performance dashboards or scorecards -- let busy managers and staff view performance measures at a glance, and then move easily through successive levels of actionable strategic, tactical and operational performance information to get the insight they need to solve problems and to improve program and services.

Tip #2. Install management aids into performance dashboards. In addition to a speedometer, tachometer, odometer, trip meter, temperature and fuel gauge on the instrument panel of your car, there are indicator lights that tell you when oil pressure or fuel is low, when your seatbelt is not fastened, your door is ajar, and when your car needs maintenance. Similar features should be part of your performance dashboard. They may include an electronic alert to the “owner” (see Tip #5) of a core or subordinate measure that has fallen below or risen above control levels, and an automatic calendaring function that puts discussion of “exceptions” on the calendar of the court executive committee (see Tip #6). Such management aids are part of most computer software applications referred to as business performance management (BPM) software, business intelligence (BI) solutions, or simply “performance dashboards,” offered by an increasing number of technology companies including Microsoft, Microstrategy, SAS, SPSS, PerformanceSoft®, Cognos®, and BusinessObjects®, to name just a few.

Tip #3. Adequately train court managers and staff on your performance measures. The value of a performance measure lies not in the measure itself but rather in the questions it forces us to ask and how we learn and grow as a result. What is the current or initial performance level? What are the changes over time? What are the acceptable upper and lower boundaries of the particular measure? What are the problems identified by the measure? Given what we know about the measurement, what performance expectations should we have in the future? Managers and court staff need to be thoroughly familiar with the functions of the performance measures that these questions highlight – baselines and benchmarking, control, trend spotting, problem diagnosis, and operational and strategic planning – for the court as a whole and, importantly, for their area of responsibility.

Tip #4. Institute open book management. Open book management is a management technique popularized by John Case in his 1995 book, Open-Book Management: The Coming Business Revolution (New York: Harper Business). As the name implies, its aim is to give employees all relevant financial information about their company so they can make better decisions as workers. The idea of open book management is the same as that supporting the concept of line of sight. That is, in order for performance measures to be practical tools and to serves as incentives for improvement -- for measures to be motivational -- there must be a line of sight between the measure and actions that can be taken by employees at various levels of the court. It conveys to everyone what the drivers of success are and provides them with the concrete knowledge of how they contribute to that success. Here are the basic rules of open book management extended beyond financial information to all performance data available to court managers and staff as part of a comprehensive court performance measurement system (CPMS):

Give all employees access to all performance data on a self-help basis. Line of sight court performance metrics and measurement hierarchies can be created by computer based graphic displays, dashboards, or scorecards of core performance measures, measurement hierarchies, navigation techniques, definitions, explanations, and references.
Give employees a simple performance scorecard. Because employees may not be naturally drawn to mind-numbing sets of numbers, open book management relies on the “critical number” and a “scorecard” that brings all the critical numbers together. (See Tip #1)
Give employees training to understand and use the performance information available to them including an intensive system of meetings, called "huddles," to keep employees informed about the status of the court in terms of its performance measures. (See Tip #3)
Give employees ownership and responsibility for the numbers under their control. If the performance measure – let’s say the median number of jail days for pretrial detainees – rises above the control level, the “owner” of the measure is held accountable (see Tip #5).
Give employees a stake in how the court performs. If, on the other hand, the median number of jail days for pretrial detainees drops below the target level, the “owner” is given credit for reducing jail days and attendant costs.

Openness and transparency are hallmarks of good government, but commonly they are seen as applicable primarily to a court’s external stakeholders. Open book management applies these hallmarks to a court’s internal stakeholders – its employees.

Tip #5. Assign owners for all core measures. Courts should assign specific court managers and staff as “owners” of the performance measures who can be queried (“Why was there a downturn in trial certainty this month even though we tightened our continuation policies?”) with an email function linked to the measure on the display. For example, a director of human resources in a medium or large court might be assigned the owner of Measure 9, Employee Opinion, of the CourTools.

Tip #6. Place performance measures on your agenda. Court leaders should make the results of core performance measures a standing item on executive team meeting agendas. The team addresses four questions at every executive meeting: (1) What is the current performance level compared to established upper and lower “controls” (e.g., performance targets, objectives, benchmarks and tolerance levels)? (2) What does performance look like over time? Is it better, worse or flat? How much variability is there? (3) What happened to make performance decline, improve or stay the same. What are some credible explanations? (4) What should be done to improve poor performance, reverse and declining trend, or to celebrate or recognize good performance (i.e., one that reached or exceeded upper controls?

Question 1 and 2 are answered quickly within a few minutes. The lower and upper controls for each core measure allow the executive team members to be comfortable with delving deeper into the data only with select measures that warrant their attention. By prior agreement, only those core measures out of range will be discussed by the executive team, except if issues are raised by the meeting participants (e.g., some recent event has made it likely that a particular measure will soon decline precipitously even though it is currently in control). Those executive team members and managers assigned ownership of measures that are currently out of range -- who already know they will be asked to explain what happened to make performance decline or improve, and what should be done about it -- are not caught surprise and are well prepared to answer Question 3 and 4 about “their” measures.Making performance measurement results a standing item on executive meeting agendas (Tip #6) and assigning ownership of performance measures (Tip #5) are just two relatively simple techniques for integrating performance measurement into the court’s management practices and processes. Executive team members focus on what’s most important and don’t get bogged down on too much data.

Tip #7. Make performance measures a part of the budget process. For many court managers, making their budget process more performance-based is the primary motivation for developing court performance measures. Budgeting is a rational and systematic way of estimating and allocating resources. Performance based budgeting or results-based budgeting, and related approaches like activity-based budgeting (ABB) and activity-based costing (ABC), are methods of budgeting that link resource estimates and appropriations to outputs and outcomes. The major elements of performance-based budgeting are defining goals and objectives, developing measures of performance aligned with those goals and objectives, linking spending decisions to results, and accountability for results. Performance-based budgeting has been around for over forty years. The early approaches focused mainly on the relationships between inputs and outputs. In recent years, the emphasis on outcomes -- the benefits of the programs for its recipients and participants -- has made performance- based budgeting a "hot" topic for courts.

In many ways, the need and attraction of performance based budgeting comes from the obvious shortcomings of the traditional budgeting process: a negotiation between court managers and funding authorities over some relatively small percentage change over last year’s budget (or the budget of a court in the same class) -- a negotiation that rarely reaches issues of efficiency and effectiveness. Further, standard accounting principles only prevent court managers from stealing money, but they do little to stop them from wasting it. Using performance measures for budgeting is a big challenge for all government, not just courts. According to at least one expert, Phillip Joyce, assistant professor in the Maxwell School of Citizenship and Public Affairs at Syracuse University, performance-based budgeting is “unambiguously desirable but quite complicated to carry out in practice.” Further, he states, "the task of developing measures pales in comparison to the challenge inherent in trying to use performance measures as a basis for budget decision making."

As the saying goes, get on the bus or be hit by it. Better yet, drive the bus. Court managers are not likely to resist performance-based budgeting by refuting its desirability. Instead, they should embrace the need, desirability and rationale for performance-based budgeting, and take an active part in its design, development, testing and implementation.

Tip #8. Integrate performance measurement with all of court’s key operations and management processes, not just the budget process. Yes, performance measurement can fundamentally change the way a court does business, but it will not happen by itself. If a court’s leadership and management are to become truly performance-based, performance measurement has to become hard-wired into the very DNA of the court’s organizational culture. Some of this is simple and straightforward. Aligning performance measurement with other key management processes like strategic planning, budgeting, quality improvement, and human resource management may be more demanding but not necessarily difficult or complex. For example, a court’s ability to develop measurable performance objectives is critical to success of its strategic planning process. A strategic goal like maintain a high-performance workplace is made useful only if it is translated into a measurable objective such as workforce strength, commitment and engagement exceeds 80% as measured by a quarterly survey of court employees. A CPMS that includes a measure of workforce strength not only facilitates development of a strategic plan by giving definition to goals and objectives but also establishes the mechanism by which the strategic plan is put into action.

More Than a List of Measures

A court performance measurement system (CPMS) is more than a list of measures. It is, instead, a set of interacting and interdependent processes of routine collection, analysis, synthesis, delivery, display and use of performance data that enable a court to measure, monitor and manage its performance effectively. There are three major requirements for designing an effective CPMS:

Identifying, designing and developing the right measures -- measures that will actually help to achieve desired results.
Ensuring that the performance measures are available to the right people, at the right time, in the right place and in the right way.
Formulating and executing strategy based on the performance information – i.e., integrating the CPMS with key management processes such as budgeting and finance, resource and workload allocation, strategic planning, organizational management, and staff development.

The first two equirements depend on an effective design process. The third is the key to effective implementation and institutionalization of performance measurement. It will not happen by itself. In the 1989 film Field of Dreams, an Iowa corn farmer (Kevin Costner) hears voices that tell him, “If you build it, they will come.” He interprets this message as a command to build a baseball field on his farm. He does and they -- Shoeless Joe Jackson and the other seven Chicago White Sox players banned from the game for throwing the 1919 World Series – come. This works in the movies but it does not work for court performance measurement. A court performance measurement system (CPMS) is essentially meaningless until we get it into the hands of people who can put it to use, understand what it is telling us, and apply what we are learning.

We’ve have made great strides toward meeting the first requirement above by the introduction of the CourTools. That success has highlighted the need to give some serious thought to the second and third requirements. So what do we do when we have the performance measures? Yes, performance measurement can fundamentally change the way a court does business, but it will not happen by itself. Users will not come into the field of dreams simply because the CPMS is rolled out. If a court’s leadership and management are to become truly performance-based, performance measurement has to become hard-wired into the very DNA of the court’s organizational culture.

Open Book Management in the Courts

2006-09-30T18:47:00.000-04:00

Open book management is a management technique popularized by John Case in his 1995 book, Open-Book Management: The Coming Business Revolution (New York: Harper Business). As the name implies, its aim is to give employees all relevant financial information about their company so they can make better decisions as workers. The concept behind open book management is the same as that supporting line of sight court performance metrics and measurement hierarchies. That is, in order for performance measures to be practical tools and to serves as incentives for improvement -- for measures to be motivational -- there must be a line of sight between the measure and actions that can be taken by employees at various levels of the court. It conveys to everyone what the drivers of success are and provides them with the concrete knowledge of how they contribute to that success.

Here are the basic rules of open book management extended beyond financial information to all performance data available to court managers and staff as part of a comprehensive court performance measurement system (CPMS):

Give all employees access to all performance data on a self-help basis. Line of sight can be created by computer based graphic displays, dashboards, or scorecards of core performance measures, measurement hierarchies, navigation techniques, definitions, explanations, and references.
Give employees a simple performance scorecard. Because employees may not be naturally drawn to mind-numbing sets of numbers, open book management relies on the “critical number” and a “scorecard” that brings all the critical numbers together. This same approach is recommended for building court performance measurement displays, dashboards, and scorecards.
Give employees training to understand and use the performance information available to them including an intensive system of meetings, called "huddles," to keep employees informed about the status of the court in terms of its performance measures.
Give employees ownership and responsibility for the numbers under their control. If the performance measure – let’s say the median number of jail days for pretrial detainees – rises above the control level, the “owner” of the measure is held accountable.
Give employees a stake in how the court performs. If, on the other hand, the median number of jail days for pretrial detainees drops below the target level, the “owner” is given credit for reducing jail days and attendant costs.

Openness and transparency are hallmarks of good government, but commonly they are seen as applicable primarily to a court’s external stakeholders. Open book management applies these hallmarks to a court’s internal stakeholders – its employees.

For the latest posts and archives of Made2Measure click here.

International Center for Best Practice

2006-09-26T11:43:00.000-04:00

In a post last February, I wrote that I was tired of "best practices" and recommended that the concept be replaced with evidence-based practices, defined as programs, strategies or procedures for which there is demonstrable evidence that their use produces desirable performance outputs and outcomes. I wrote that best practices, the way the term is used in court administration, instead should be called “interesting practices,” “intriguing practices,” “promising practices” and, maybe, “practices-you-might- want-to-see-in-person-if-the-weather-is-right.” I worried that the irony was a bit too sharp. I should not have worried.

In his July Bob Behn’s Public Management Report (Vol. 3, No. 11), Bob Behn, a lecturer at Harvard University’s School of Government, takes the irony to another level. If, as Ralph Waldo Emerson wrote, “consistency is the hobgoblin of little minds,” then “best practice” is the refuge of unimaginative ones, writes Behn. Why are public managers so obsessed with something that someone else has labeled “best practice”? Because, writes Behn, if they discover a “best practice,” then they can stop thinking. By adopting a “best practice,” public managers conveniently avoid the hard work of figuring out whether the identified practice actually will work in their organizations. It is the “best” they confidently assure themselves and others. Who will challenge the unnamed management gurus, asks Behn, who have certified the practice as “best”?

A manager need not be too discriminating or too careful worrying whether the practices will provide some “operational nutrition” or merely “institutional heartburn.” He or she can simply choose from a long menu of best practices what is personally appetizing.

Here’s Behn at his ironic best.

You, of course, have heard of it: The International Center for Best Practice. It has charters from the United Nations and the World Bank, and is stocked with Nobel laureates in management. Once a year, they convene in Zurich, Singapore, or Capetown to choose a few, select practices to be officially authenticated as a “Best Practice.” If the head of a public agency can find one of these practices and implement it throughout his or her organization, no one can dispute that he or she is not an au courant public executive.

Anyone interested in charter membership in the International Center for Best Practices in Court Administration (ICBPCA)?

For the latest posts and archives of Made2Measure click here.

© CourtMetrics 2006. All rights reserved.