Measurement: Bill Gates’ Plan to Fix the World’s Problems

Bill Gates: My Plan To Fix the World’s Biggest Problems** Measure Them!  This is the headline of the lead article in the Review section of the weekend edition of the Wall Street Journal  written by Bill Gates, co-chair of the Bill & Melinda Gates Foundation and co-founder of Microsoft.   The article was accompanied by a large picture of transparent globe wrapped by a tape measure covering most of the first page of the section.  Its message is the theme of Made2Measure. Because it comes from Bill Gates, we tend to listen.

“In the past year,” Gates writes, “I have been struck by how important measurement is to improving the human condition. You can achieve incredible progress if you set a clear goal and find a measure that will drive progress toward that goal .. .This may seem basic, but it is amazing how often it is not done and how hard it is to get right.” He writes specifically about foreign aid among other areas like education, health care and agriculture .  “Historically, foreign aid has been measured in terms of the total amount of money invested—and during the Cold War, by whether a country stayed on our side—but not by how well it performed in actually helping people.”  When a country does commit to measuring outcomes toward concrete goals and clear performance targets, like Ethiopia did to bring health care to all of its citizens in 2004, it attracts donor money.  

Gates believes that one of the greatest benefits of measurement is the ability it gives government leaders to compare their countries’ performances with that of other countries and learn from the best.

Gates concludes his article with this: “The process I have described – setting clear goals, choosing an approach, measuring results, and then using those measurements to continually refine our approach – helps us to deliver the tools and services to everybody who will benefit, be they students in the U.S. or mothers in Africa.” 

© Copyright CourtMetrics 2013. All rights reserved.

Big Data, Data Analytics, and the Access to Justice “Card”

It is not uncommon for court leaders and justice stakeholders to raise the access to justice “card” to subvert the closing or consolidation of courts.  If you close this court -- the argument goes -- citizens will be denied access to justice.  This is a strong argument because it typically evokes an emotional response irrespective of the information’s truth value, especially if the counter-arguments are couched in bland terms like efficiency and cost-savings. Trouble is that such arguments are heavy on rhetoric and light on hard evidence; and they often produce factionalism and political stalemate (see Court Consolidation in Mahoning County).   

In a 2005 survey of the public and of attorneys in California, respondents who were asked about eleven reasons that might keep someone from “going to court,” cited “travel distance to court from home” less often than eight of the other reasons including fees, cost of hiring an attorney, the time it takes to reach a decision, lack of child care, and the hours the court is open. 

How far does a court have to be before it is too far -- before the distance constitutes a legitimate barrier to access to the courts?  It seems that the best that we’ve been able to say in response is what we said back in the horse-and-buggy days, i.e., that it should be no more than a hard day’s ride on horseback.  

Big data and data analytics may not remove dysfunctional politics but it could lead to data-driven decision-making and calculated boldness by leaders advocating reform efforts such as court consolidation. For example, court location data could be compared against a number of public databases with information from inside and outside the justice system including Zip codes, populations, demographics of the population (race, age, disability), travel times between locations, numbers and types of cases heard by different courts, levels of courts, and availability of public transportation.  By tapping into the every growing mobile data traffic, we may eventually be able to use real-time distance data from court users on their smart phones.  

Results may allow advocates and opponents to compare various court consolidation models and say, for example, that the consolidation of courts from ten locations to three would increase the average distance and driving time to the nearest court from 3.1 miles and a ten minute commute to 4.5 miles and a fourteen minutes, with the overall average can be disaggregated by age of citizens, income levels, case type and so forth.  

Who knows, this type of information might actually allow us to update the “a hard day’s ride” access to justice standard. 

© Copyright CourtMetrics and the National Center for State Courts 2012. All rights reserved.

The Courts’ Big Data: What If Only?

Close to 2.5 quintillion bytes of new data are created daily. ....Organizations know this data is rich in information and potential insight, but it’s as if they receive a fresh-minted library daily with answers to all their questions, except the books are written in languages they cannot understand. 

The above words appeared in a full-page ad by IBM in yesterday’s Wall Street Journal under the headline “Watson Goes To Work.” The ad touted IBM’s Watson, the supercomputer Jeopardy! champion over two human challengers in a pop culture event in February 2011.  Watson’s performance improved as the game progressed as it built on each bit of information acquired from each new clue and answer.  Watson demonstrated the advanced form of computing IBM calls a cognitive system, “a system that is not simply programmed but is trained to learn based on interactions and outcomes.” 
 

The ad brought to mind a “what if” question my National Center for State colleague Jim MacMillan and I have been pondering for while now. 
 

What if we could tap into the massive amount of data that is entered and stored in a court’s case management system, and other data systems like states’ criminal justice repositories?

For example, an approach – I believe first suggested by Jim McMillan and Philadelphia Court of Common Pleas Judge Carolyn Engel Temin, which already may be in the experimental stages of development by judiciaries outside of the U.S. -- uses information in modern automated case management systems to measure case complexity and dynamically create and adjust the individual case weights and the total case weights for individual judges and staff, departments, and courts. Potential factors and data elements related to case complexity in criminal cases, for example, might include the number and seriousness of criminal charges, the number of defendants, criminal history of the defendant, number of documents submitted in evidence, number of witnesses, witness availability, and the number of exhibits.  Workload assessments and assignment of cases, among myriad other functions, would be based on up to date adjustment in real time. 
 

Which brings me to the the takeaway question: Why not a Watson in the courts?  No doubt the courts today have Big Data that a “Justice Watson” could devour fur lunch. 
 

My wish is that a case management system company or a business intelligence vendor with court or justice system interests would throw some skin into the research and development (R&D) game for this exciting prospect.  It might begin modestly with dynamic workload assessment.  The R&D could be funded directly or by means of some leveraged development in cooperation with an individual court or court system.  
 

© Copyright CourtMetrics and the National Center for State Courts 2012. All rights reserved.

Animating Court Performance Measurement

We have failed to breathe life into the prescriptions for the performance measures that we exhort courts to use on a regular and continuous basis.

The connection between well known health indicators like blood pressure, cholesterol level, and blood glucose and our health and wellbeing is self-evident to most informed people.  We know that these measures mean something vital and something very important to us.

In my experience, the same cannot be said about the connection between such relatively common measures of court performance as time to disposition, trial date certainty, and collection of monetary penalties and fundamental justice and fairness (see Ten Reasons Not to Measure Court Performance, Made2Measure, November 19, 2008).  Many judges and court managers see these measures as indicators of the relatively narrow concerns of those for whom it’s all about the numbers -- the number crunchers, the spreadsheet guys, and the IT folks who manage the case management system.  These measures, quite simply, do not matter to many judges and managers; they do not appear on their radar screens. 

This state of affairs, if accurately perceived, cannot be blamed on judges and managers.  It is in my view largely the fault of those of us who preach the gospel of court performance measurement and management.  We have failed to breathe life into the prescriptions for the performance measures that we exhort courts to use on a regular and continuous basis.

The need for animating court performance measurement came to mind as I watched a video of a presentation by Albie Sachs at the Open Society Institute.  Sachs is a passionate man and his views evoke strong emotional responses from an audience.  He is former judge on the Constitutional Court of South Africa, an activist, and a leading campaigner in the fight against apartheid.  In the video, Sachs discusses his 2011 book, The Strange Alchemy of Life and Law. He talks his deprivations of access to justice and solitary confinement and time in detention without trial during his struggles for rights in South Africa under the apartheid regime. In 1988, while in exile in Mozambique, he lost an arm and sight in one eye by a bomb placed in his car by South African security agents.

Of course, stripped of Albie Sach’s animation, the time in detention without trial is no more than a portion of the measure of time to disposition for defendants in custody before trial, a measure I attempted to elevate in importance in a posting several years ago (see Common Measures of Performance Across the Justice System, Made2Measure, February 17, 2007).  But Sach’s account of his detention without trial bears no resemblance to how I characterized the measure.  Mine is lame and lifeless compared to his.  He speaks of his delay and denial of justice in terms of a denial of rights, in terms of deprivations of liberty, in terms of a failure of democracy.  Big important things that matter.

While I know we cannot match Albie Sach’s passion and eloquence, I know we can (and should) do a much better job at aligning court performance measures with the higher values, mission, fundamental responsibilities, and obligations of courts to ordinary citizens.  The measure of trial date certainty should be made vital by not only connecting it to more effective calendaring and a court’s continuance policies and practices, but by aligning it with the certainty that is a universal principle defining the rule of law.  And the measure of collection of monetary penalties should be shown to be much more than merely a measure of revenue generation by a court but, instead, like blood pressure is to health, described as a measure of a court’s performance of its fundamental obligation to enforce its orders and maximize compliance with law.

© Copyright CourtMetrics and the National Center for State Courts 2012. All rights reserved.

Measuring Procedural Justice

The rationale for measuring citizen/court user satisfaction has rested mostly on the argument that we should improve courts on the basis of what we learn from those served (or not so well served) by the courts, rather those who run the courts.  Two new developments make this argument more compelling: the emergence of the theory of procedural justice (or procedural fairness) and the finding that our desire for being heard, for having a voice in matters, is hard-wired into our brains.

Procedural Justice

Here is what Kevin Burke, District Court Judge in Minnesota, current President of the American Judges Association, and a long-time proponent of court performance measurement, recently had to say about the importance of procedural fairness (Judicature, Viewpoint, May/June, Volume 95, Number 6, 251 -254):

Better performance is the key to building public support for the judiciary… A court or a judiciary

that is as good as its promise is known not just for speed or efficiency (heaven knows many courts

are good at that), but also for other, less quantifiable aspects of justice—things like fairness and respect,attention to human equality, a focus on careful listening, and a demand that people leave our courts understanding court orders…. For courts to build public trust and enhance the legitimacy of judicial decision making, there must be a willingness to commit to measuring procedural fairness. It can be done. .. Some courts are already doing this. At least at a rudimentary level the measurement tools are largely available in The National Center for State Courts CourTools #1. .. Procedural fairness develops from research showing that how disputes are handled has an important influence upon people’s evaluations of their experience in the court system. How people and their problems are managed has more influence than case outcome based upon two key issues: 

• Whether people accept and continue to abide by the decisions made.

• How people evaluate judges, the court system and the law. 

Giving people a voice – the opportunity to participate in their case by expressing their viewpoint, telling their story – is seen as an important element of procedural justice. People who are allowed to have their say in court are more willing to accept a negative outcome and comply with the law.  

Neuroscience 

One reason that people who are allowed to have their say in court are more compliant with law may be that talking about oneself is pleasurable at the level of brain chemistry. Psychological research, as well as recent neuroscience reported in the Proceedings of the National Academy of Sciences (Wall Street Journal, May 8, 2012) indicates that talking about ourselves, especially self-disclosure, triggers the same sensation of pleasure in the brain as food or money.  It is rewarding, at the level of cells and synapses, to give voice to our thoughts and feelings.  

Although talking about oneself is not the same as “giving voice” to ones opinions in a survey of court user satisfaction such as Measure 1 of the CourTools, it may trigger similar sensations of pleasure. 

The takeaway lesson from these two new developments is this: Quite apart from the value to performance measurement and management provided by surveys of court user satisfaction, the active engagement of litigants and other court users with issues of procedural justice and fairness in such surveys without more may enhance trust and confidence in the courts.

© Copyright CourtMetrics and the National Center for State Courts 2012. All rights reserved.

Ensuring the Right Use of Performance Data: A Cautionary Tale from Health Care

Health care has provided lessons for court administration in the past (see Monitoring “Never Events in Court Administration, Made2Measure, September 23, 2008; Pursuing Perfection – A Lesson from Health Care, Made2Measure, November 1, 2006. This time it provides a cautionary tale, specifically for performance measurement and performance management.

The Story of PICS

As reported in the Economist (“From petrol to prescriptions,” June 16, 2012. p. 65), the Queen Elizabeth hospital in Birmingham in the U.K. pioneered a new approach to patient care learned from the carmaker BMW’s engine assembly factory where over 99.9% of tasks were completed flawlessly. Big health systems in the U.K. and the U.S., in contrast, tolerate many more errors.

Emulating BMW’s approach to manufacturing, the Birmingham hospital system installed a quality-control regime called PICS, which stands for “Prescribing, Information and Communication System.” PICS features a performance dashboard system on a computer screen in every in-patient ward displaying such performance measures as hospital-borne infection rates following surgery and the number of falls by frail patients. Efficiencies that are benchmarked against comparable wards and past performance are continually monitored by staff and management in real time. No doubt, the Birmingham hospitals had the right measures and the right delivery system in PICS to get the right measures to the right people in the right way and at the right time.

The Birmingham experiment was welcomed by Britain’s health-care establishment as Queen Elizabeth’s mortality rates fell and errors that lead to poor patient care decreased. Other hospitals with poor performance contracted out patient care to Queen Elizabeth.

But this is where the good story ends. The Birmingham experiment has stalled. It seems that Britain’s National Health Service (NHS) is not into bold thinking as it is preoccupied with austerity. It emphasizes the universality of care provided rather than its outcomes. Hospitals do not compete with each other. Successful innovations like PICS do not spread because NHS has no mechanism for ensuring that they do.

The lesson of this cautionary tale is that it’s not sufficient to identify the right performance measures and even to build the right delivery mechanisms like PICS, a modern performance dashboard system. A performance measurement and management system is not self-executing. Users of performance data need to act on the problems or opportunities that the data reveals on a regular and continuous basis and have the incentive to do so.

For courts, the right use of performance measurement will not happen unless courts implant performance measurement and performance management into the very DNA of their governance, management practices, and daily operations. It will not happen until the courts make innovations such as assigning new responsibilities, instituting specific policies, creating governance structures, and starting processes, procedures and practices to ensure adoption, implementation and institutionalization of a performance measurement and management system.

Right Measures, Right Delivery and Right Use

Identifying the right performance measures for an individual court or court system, and making sure that they are used effectively, can be translated operationally into three key requirements and corresponding phases of development: Right Measures - identifying and developing the right performance measures; Right Delivery and Distribution. ensuring that the right measures are delivered to the right people, at the right time in an easy to understand way (increasingly, this is done by computer-based performance scorecards or “dashboards” that let users view critical performance information at a glance, and move easily through successive layers of strategic, tactical and operational information on a self-help, on-demand basis; and Right Use. adopting, implementing, and integrating the measures performance, as well as the delivery system and distribution system (e.g., performance scorecard or dashboard), with key management processes and operations, including budgeting and finance, resource and workload allocation, strategic planning, organizational management, and staff development.

You can’t just throw an innovative performance measurement and management system “over the wall” to executives, managers and staff to adopt and to implement. Even well-conceived, well-designed systems are unlikely to get implemented unless they are woven into the very fabric of a court’s management practices and processes.

© Copyright CourtMetrics and the National Center for State Courts 2012. All rights reserved.

Privacy Concerns Threatens Survey Research

In previous posts I’ve called for courts and justice systems to be good stewards of performance data.  And, as an advocate of radical transparency, I’ve pushed for self-assessment and accountability by means of regular and continuous performance measurement and management instead of reliance on periodic third-party program evaluation and research.

Now comes another good reason for courts and other justice institutions to go that route.  As reported by Carl Bialik, Census Gets Questions on Mandatory Queries, the Wall Street Journal’s “Numbers Guy,” members of Congress are challenging the Census Bureau’s massive data-collection effort known as the American Community Survey.  The survey uses dozens of detailed questions – including the use of carpools, flush toilets, and wood fuel – to help determine the need for and distribution of funds of various government programs. The survey, which reaches 3.5 million households, is mandatory, a fact that gets close to 100% of the people to respond and a necessity for maintaining the survey’s validity, according to the Census Bureau and its supporters, including the American Association for Public Opinion Research and the American Statistical Association.  

The congressional challenge, including presidential candidate Ron Paul, rests on privacy concerns. “The freedom of the American people from unwanted government intrusion into the private lives and affairs is a higher priority than the quality of the government’s data,” the Wall Street Journal quoted a Ron Paul spokesperson as saying.

Good stewardship of performance data, in my view, requires courts and justice institutions not only to collect data on the right measures but, first and foremost, to integrate performance measurement and performance management with good governance.  They need to have the right delivery of the performance data, which means getting it into the hands of the right people in a totally transparent and accountable manner that ensures the right use of the data.   Privacy issues and non-disclosure of protected data are best determined and implemented by judicial branch policymakers as part of the development of an effective performance measurement and management system.  Unpleasant surprises, such as that facing the Census Bureau, might be minimized, if not avoided altogether, by such an approach.  

© Copyright CourtMetrics and the National Center for State Courts 2012. All rights reserved.