The history of technological change shows that the scientific method has been extremely powerful at developing new knowledge. However, we all hear of the studies where it is announced that eating “Item A” greatly reduces your risk for getting a disease, and then several months later, another study says that eating “Item A” actually increases your risk for getting the same disease, and then several years later, researchers say “Never mind”, “Item A” doesn’t appear to do anything to your risk of getting this disease. Scientific studies can often be inaccurate.
I recently read three articles by Dr. John P.A. Ioannidis, a mathematical medical researcher at Tufts University School of Medicine in Boston and the University of Ioannina School of Medicine in Greece. Dr. Ioannidis shows mathematically that most published research findings are false.[i] He arrives at several corollaries to evaluate a research study. Although he is discussing medical studies, all of these corollaries apply with equal force to empirical legal studies.
Small studies are less likely to be true than large studies. The smaller the effect of a treatment, the less likely the research is true. The greater the number and the lesser the selection of tested relationships the less likely any finding is true. The greater the flexibility of the research design the less likely the research is true. The greater financial interests, prejudices, and the hotter the scientific field, the less likely the research is true.[ii]
Dr. Ioannidis, in another research paper, evaluated all original research studies in 3 major clinical journals (which should contain the better designed studies). He found that 32% of the original research was either refuted or had effects that ultimately were shown to be less strong than the original research findings. Five out of six non-randomized highly cited original studies were either refuted or had weaker effects than the original research.[iii]
Finally, in an essay, Dr. Ioannidis discussed the challenges of translating bio-medical research findings into something useful for practitioners.[iv] His main concern is the truth of many of the research findings, and why much research fails to be upheld by subsequent research. He recommends a greater focus on identifying research topics, diminishing bias, and enhancing the power of the studies. [v]
Dr. Ioannidis states “Translating non-credible, non-replicated research findings may have bleak consequences. We already have several useless prognostic and diagnostic tests, ineffective and possibly harmful therapies, and redundant sub-specialties sustained by unsubstantiated optimism on their benefits. We should not add more junk to this pile.”[vi]
Although, I am a proponent of using science in the law, I believe that many of the empirical research studies in law are very likely junk. Much of the purported “research” isn’t research at all and can clearly be shown to be junk. Other research that shows statistical differences may not be as easily critiqued, and will only be refuted by further study.
I am concerned that we legal practitioners not be deluded by the current fad of proponents of a particular approach labeling something “evidenced-based” as an imprimatur of legitimacy. The evidence might be flimsy, misleading, or outright fraudulent. As we often have peoples’ lives in our hands, we must be careful and assure ourselves that we are as accurate as possible. I am forever the hopeful skeptic.
[i] Ioannidis, JPA (2005) “Why most published research findings are false. PLoS Med (2)(8)e124.
[ii] Ibid.
[iii] Ioannidis, JPA (2005) “Contradicted and Initially Stronger Effects in Highly Cited Clinical Research” JAMA, Vol. 294 (2) pp. 218-228.
[iv] Ioannidis, JPA (2006) “Evolution and Translation of Research Findings: From Bench to Where” PLOS Clin Trial 1(7): e36.
[v] Ibid p. 5
[vi] Ibid p. 8.
No comments:
Post a Comment