Without Replication, Should Program Evaluation Findings Be Suspect as Research Findings Currently Are?


Replication of research -- the reproducibility of findings -- is a methodological safeguard and hallmark of research universally lauded by scientists to justify their craft.  As we are continuing to learn with more certainty, it is theory not much put into practice. Claims about research finding may be more likely to be false than true. Scientific studies are tainted by poor study design, sloppy and often self-serving data analysis, and miscalculation – problems that replication of the studies and duplication of the results would largely correct. Again, the problem is that it’s not done.

The continuing work of John Ioannidis at Stanford University, Brian Nosek at the University of Virginia, and others shows that much research is not and cannot be replicated. Almost a decade ago in these pages (Courts Have No Business Doing Research Studies, Made2Measure, October 15, 2007), I highlighted a 2005 paper by Ioannidis titled “Why Most Published Research Findings Are False” that caused a stir in the scientific community and prompted many scientists and consumers of research  to begin questioning whether we can trust evidence produced by research studies.  Today, with more than $80 million of funding of a “research integrity” initiative by the Laura and John Arnold Foundation, the science critics and reformers like Ioannidis and Nosek have been given a solid platform to question the culture of science that produces studies that can’t be reproduced.

Can program evaluation be questioned as well? Program evaluations are assessments of changes in the well-being (status or condition) of individuals, households, communities or firms that can be attributed to a project, program or process, along with the systematic determination of their quality, value or merit. Rooted in the tradition of behavioral and social research, does program evaluation -- especially impact evaluation that relies on randomized controlled trials – exist in a similar culture of research described by Ioannidis, Nosek, and others that does not support replication and reproducibility of results? In my experience, program evaluations in the area of justice and the rule of law are one-off affairs funded by donors who are seldom, if ever, prompted to support replication of the results.

For several years, I have called for a bigger space for performance measurement and management (PMM) in the toolkit of international development of justice and the rule of law relative to program evaluation and global indicators. I argue that justice institutions and justice systems that take responsibility for measuring and managing their own performance in delivering justice using PMM, rather than relying on external assessments done by third parties such as typically is done in program evaluation and global indicators, are likely to have more success and gain more legitimacy, trust and confidence in the eyes of those they serve.

Replication or reproducibility of results highlights a critical design difference between PMM and program evaluation or evaluation research. Basically, replication means repeating the performance measurement or evaluation research to corroborate the results and to safeguard against overgeneralizations and other false claims. In contrast with program evaluation, repeated measurements – i.e., replication of results on a regular and continuous basis, ideally in real time or near-real time -- are part of the required methodology of PMM.

I’d like to believe that my suspicion of a lack of replicability and reproducibility of program evaluation findings strengthens my argument for more space in the toolkit of international development for PMM. Of course, my suspicions are just that until “program evaluation integrity” studies like the research integrity studies funded by the Arnold Foundation confirms those suspicions.

© Copyright CourtMetrics 2017. All rights reserved.

Popular posts from this blog

Top 10 Reasons for Performance Measurement

There is a growing demand of “show me the data” from citizens and taxpayers, legislators, executive agencies, oversight boards and communities – all of whom exchange money and other support (e.g., trust and confidence) for services -- that courts can ill afford to ignore. Courts need to measure their performance to meet the demand for public accountability. But the benefits of court performance extend beyond accountability including improved prediction, better understanding and control, influence, a sharpened focus on what is important, and discovery of better practices. Objective performance data is central to good management practices like strategic planning, performance-based budgeting, Balanced Scorecards, Total Quality Management (TQM), and Six Sigma. Performance measures allow court leaders and managers, policymakers, legislators and the public to evaluate court programs’ inputs (the resources allocated), outputs (direct results of program or service activities) and outcomes (bro
Read more

The “What Ifs” Along the Road in the Quest of a Justice Index

What if we could draw an indicator of the “health” of the justice systems throughout the United States – a Justice Index - from readily available public data that is impartial, reliable, objective and comparable, let’s say the amount of time that criminal defendants spend in custody without a trial, or even without any access to legal or justice services? Or the bail amounts imposed on misdemeanor defendants?   This kind of a measure would not be perfect – no measure is – but it would be something, better than nothing. It would address questions of fairness and efficient use of public resources. That is, a criminal defendant languishing in jail awaiting trial suffers, at the least, a delay in justice. He or she is also occupying a jail bed and spending taxpayers’ money. What if we then join this measure with another meaningful indicator of the health of our justice systems in counties and states, let’s say the degree to which jurisdictions prosecute domestic violence cases.
Read more

Q & A: Outcome vs. Measure vs. Target vs. Standard

Q: Those of who write and speak about court performance measurement rely on their share of jargon -- and it's often confusing. What are the differences among the terms “outcome,” “measure,” “target” and “standard”? A: Socrates said that the beginning of wisdom is definition. We’ll see. Outcomes are the benefits or changes for the intended beneficiaries of a court’s programs and services. Outcomes may relate to knowledge, skills, attitudes, values, behavior, condition or status of the program participants or recipients of services. Examples include litigants' satisfaction with a court's courtesy and responsiveness, success of probation, time to case disposition, clarity of orders, integrity of case files, percent of mediation agreements, enforcement of orders, and perceived fairness of proceedings, as well as percent of expected expenditures and percent of revenues received. Outcomes should not be confused with the activities of those who run the court. (See the October 1
Read more