Evidence: Its Meanings in Health Care and in Law

Coverage Policies, Medical Liability, and Cost-Effectiveness Analysis

Jacobson's presentation, "Cost-Effectiveness Analysis in the Courts: Recent Trends and Future Prospects," elicited questions and debates among the participants that revealed underlying differences in their assumptions and understandings about EBM and how it has been or should be incorporated into the design of health plans' benefit packages. Clinicians and researchers involved with the development and promotion of EBM tended to view cost-effectiveness as an evaluative criterion incorporated only into normative applications of research results, such as CPGs. Others were more likely to view cost-effectiveness analysis (CEA) as an implicit aspect of EBM policy and practice.

Jacobson acknowledged the wide variability in how CEA is defined, from "a method designed to assess the comparative impacts of expenditures on different health interventions" (Garber et al. 1996), to the determination of the lowest cost intervention that offers a clinically appropriate benefit (Eddy 1996). The simplest application of CEA is to compare two interventions aimed at the same outcome, although CEA may also be applied more broadly to design a health plan's overall benefit package, for example, where different services or preventive screening tests are considered for inclusion. Even in its more narrow applications, Jacobson noted, CEA is fraught with methodological difficulties and value judgments. Jacobson was charged by the workshop planning group with a particularly challenging task: to extrapolate from very limited legal precedents and experience in other fields of liability litigation to the likely treatment of CEA in suits involving health plan coverage policies and medical necessity determinations. In his presentation at the workshop, Jacobson reported that, to date, health plans appear not to use CEA explicitly but, rather, rely on other cost containment tools. He offers several possible reasons for this and proposes that the most important one may be the limited evidence as to effectiveness of alternative interventions.

In responding, Drummond Rennie described CEA as a complex form of evidence, involving review of many individual studies and findings, which means that data, assumptions, and analytic models are chosen according to subjective criteria. In many cases, Rennie contended, the evidence upon which CEAs are based is weak. He also argued that these methodological limitations, along with the publications bias that follows from commercial sponsorship of the research, severely compromises the scientific and practical value of such analyses. Recognizing that CEA is, nonetheless, an important tool, Rennie suggested—following the model of legal discovery—that all CEA study data, models, and assumptions be made public by putting them on the Web sites of the journals that publish them (see Rennie in this issue and Rennie and Luft 2000).

Eddy addressed the promise of CEA less skeptically than Rennie, arguing that almost any approach to this evaluative technique is better than the currently dominant approach in health care, which is not to take cost or opportunity costs into account at all. Eddy suggested that, in fact, health plans are using CEA, although not explicitly. Taking the term in its broadest sense, to the extent that plans have limited budgets for benefits, a responsible manager will think in terms of cost-effectiveness in allocating that budget. In addition, Eddy argued, individual physicians make cost-effectiveness decisions all the time. All aspects of medical practice employ thresholds, indications, and contraindications. With our growing ability to know what works and what does not in particular situations, these should be set more rationally than they have been in the past.

Transparency, openness, and consumer participation were themes of the panel discussion that followed. First, because value judgments are necessarily applied in constructing cost-effectiveness analyses, and because the choice of research-based evidence included in such analyses is subject to bias, several participants argued that the methodology and analytic choices made in any CEA should be public if the results are to be credible. Second, several panelists endorsed the notion of consumer involvement in the development of CPGs and health plan coverage rules, a process in which value-based choices are unavoidable. There was general agreement that involving consumer and patient advocates in weighing the evidence to make clinical practice recommendations is appropriate not only to broaden the considerations reflected in the guidelines but also to educate consumers and reform their expectations about what health care can realistically deliver in terms of risk reduction and life expectancy.

Jacobson traced the history of CEA in the courts (at least in the sense that risks and costs were weighed together) with respect to liability in general negligence cases and later in product liability. The practice of evaluating risks and costs evolved from the pre-1930s "reasonable man" standard of common law negligence, through an explicit cost-benefit standard exemplified by Judge Learned Hand's 1947 opinion in United States v. Carroll Towing Co. (159 F.2d. 169 [2d Cir. 1947]). This latter standard says that negligence occurs when the cost of investing in injury prevention is less than the expected liability, that is, the probability of injury multiplied by the damage wrought by that injury. In 1965 the American Law Institute adopted this standard in its Restatement (Second) of Torts (and is continued in its Restatement [Third] of Torts: General Principles, Article 4 [1999]). Since the 1970s, strict liability cases for product design defects have been predominantly resolved according to explicit risk-utility analysis, which incorporates cost-benefit or cost-effectiveness tests (Jacobson and Kanna in this issue).

Jacobson used the performance of juries in product liability cases, when they are faced with evidence of corporations' explicit consideration of cost-risk trade-offs, to speculate how juries might react to the application of CEA in deciding disputes about health plan benefits. Discussion of this question brought out that a jury's perception as to who benefits from resource-conserving decisions is likely to be an important determinant of how it regards a corporation's use of cost-benefit or cost-effectiveness analyses. If product design or health plan savings strategies are viewed as simply increasing corporate profits, these strategies will be judged less acceptable than if these efficiencies are seen as redounding to the consumers' benefit in some way.

Medical liability diverges from general negligence primarily because courts have generally deferred to professional standards, supplied through expert testimony as to customary medical practice. Generally, only failure to adhere to customary practice constitutes negligence or malpractice. Jacobson noted that the customary practice standard is not intrinsically hostile to cost containment strategies, in that customary practice can evolve to incorporate efficiency-promoting treatment protocols or clinical practice guidelines. In assessing the response of courts to liability claims against a health plan or third-party payer whose coverage policies or utilization review procedures explicitly incorporate cost containment objectives, Jacobson argued that "judicial internalization of cost constraints" is gradually taking place in response to changes in the social and economic environment (Bovbjerg 1975).

Although much of his analysis focused on tort law, Jacobson noted that health plan coverage and benefit disputes also can, and often do, involve contract claims. He observed that making contracts more explicit with regard to the use of CEA in coverage decisions would be one strategy for expanding its application. Discussants returned to the issue of more explicit and transparent contracts for health benefits as one approach to reducing consumer distrust and dissatisfaction with coverage and medical necessity decisions by health plans. Eddy argued that the grounds for coverage decisions by health plans, such as CEA, must be made explicit in contract language if the plan's coverage and utilization review policies are to hold up in court. Clark Havighurst observed that recent class action suits alleging that a health plan has systematically misrepresented the benefits the plan covers, or the degree of discretion the plan affords physicians in treating patients, make adequate disclosure and honest advertising central issues in health plan conduct and performance.

Participants found fuller disclosure of health plan coverage policies a promising strategy for increasing the acceptability to consumers of science-based medical practices and benefit packages designed according to cost-effectiveness criteria. At the same time, however, several voiced concerns about the capacity of consumers to understand complex and subtle health plan coverage rules, decision processes, and utilization review procedures. Alice Gosfield, among others, proposed that health plan members sit on advisory boards and review panels that design and interpret the plan's benefits and coverage policies. Various other commentators also suggested public regulation of health plan coverage policies and review and appeals procedures as a strategy to fix the limits both of health plan discretion and liability.

Return to Contents

Using Evidence Appropriately in Medicine, Health Care, and the Law

At the end of the day the discussion centered on the uptake and assimilation of research findings into clinical practice and how the legal system might ease, or at least not impede, the adoption of evidence-based practices by clinicians and health plans. In his presentation Rosoff reasoned that if the goal of clinical practice guidelines is to change professional custom in the direction that outcomes-oriented research points, and if professional custom is the legal standard of care in medical liability cases, then courts must somehow recognize research-based clinical practice as a standard of care as well. Rosoff's essay here considers alternative mechanisms, private and voluntary as well as governmental, for the certification of CPGs as meeting standards of scientific validity. Such certification would aid courts in their review of multiple and potentially conflicting sets of such guidelines.

Discussants responded by pointing out that the IOM's Committee on Clinical Practice Guidelines had identified the attributes of good CPGs almost a decade ago and that these attributes had been operationalized in a way that allowed for evaluating the quality of sets of guidelines. The evaluation instrument has been refined in the United Kingdom and Europe over the past eight years, and a multinational study of evaluative criteria for CPGs is ongoing. Alice Gosfield proposed that just as there are attributes of good guidelines, attributes of evidence that are especially relevant in addressing public policy concerns such as fraud and abuse, medical necessity, and utilization review could be established by an agency such as AHRQ.

Eisenberg asked whether evidence must be couched in terms of prescriptive guidelines in order to be useful in legal situations or whether courts could use purely descriptive research findings directly. He noted that AHRQ evidence reports are issued without "shoulds" and that they are ideally factual statements, but he wondered whether this form of evidence would be adequate for judges and juries.

His questions elicited several responses. First, it is important to distinguish between evidence in systematic reviews and evidence reports, and CPGs. The former are the initial steps in developing a guideline, whereas a full guideline might involve consumer and patient inputs as well as legal counsel to formulate recommendations for clinical practice. The result of both steps is an evidence-based CPG. Second, although evidence can be used in its original form, in legal settings it may more commonly be used once it has been incorporated into normative CPGs. Third, basic evidence and professional guidelines will have distinctive functions within the courtroom. Guidelines can be used as presumptive of or exculpatory of negligence (regardless of whether or not their designers envisioned such a use for them). Evidence cannot by itself be used in this way. Guidelines thus must meet certain (higher) thresholds of incontrovertibility, if they are to be applied fairly.

As Lempert argued, by far the most important factor in jury trials is the weight of the evidence, rather than the identity of the parties or the quality of the legal counsel. Lohr asked just what was meant by evidentiary "weight" or "strength," wondering if some objective criteria, like those now being developed in health services research, might be introduced in court. The response was that, as in all other questions of evidence, expert opinion will inevitably conflict and that if scientists themselves dispute the quality of evidence, what can we expect of judges and juries?

Eisenberg wrapped up the day with a recapitulation of the issues and questions raised and considered the potential for further research and resolution of them. His remarks are also included in this volume in the article "What Does Evidence Mean? Can the Law and Medicine Be Reconciled?" Several participants endorsed the notion of sustained examination and interaction among the legal and medical practice and research communities. This collection of articles and commentaries should help to extend and advance the conversations that need to occur among the professions.

Return to Contents

Appendix

Participants in Attendance at the Workshop "Evidence: Its Meanings and Uses in Law, Medicine, and Health Care," 10 April 2000

Joseph S. Cecil, J.D., Ph.D.
Project Director
Program on Scientific and Technical Evidence
Division of Research
Federal Judicial Center
U.S. Department of Justice
Washington, DC

David M. Eddy, M.D., Ph.D.
Senior Advisor for Health Policy and Management
Kaiser Permanente Southern California
Washington, DC

John M. Eisenberg, M.D.
Director
Agency for Healthcare Research and Quality
Rockville, MD

Alice G. Gosfield, J.D.
Principal
Alice G. Gosfield and Associates, P.C.
Philadelphia, PA

Clark C. Havighurst, J.D.
Wm. Neal Reynolds Professor of Law
Duke University School of Law
Durham, NC

R. Brian Haynes, M.D., Msc., Ph.D.
Professor and Chair
Clinical Epidemiology and Biostatistics and Medicine
Faculty of Health Sciences
McMaster University
Hamilton, Ontario, Canada

Barbara S. Hulka, M.D., M.P.H.
Kenan Professor
Department of Epidemiology
University of North Carolina-Chapel Hill
Chapel Hill, NC

Peter Barton Hutt, LL.M.
Partner
Covington & Burling
Washington, DC

Peter D. Jacobson, J.D., M.P.H.
Associate Professor
School of Public Health
University of Michigan
Ann Arbor, MI

Douglas B. Kamerow, M.D., M.P.H.
Director
Center for Practice and Technology Assessment
Agency for Healthcare Research and Quality
Rockville, MD

Richard O. Lempert, J.D., Ph.D.
Francis A. Allen Collegiate Professor of Law and Professor of Sociology
School of Law
University of Michigan
Ann Arbor, MI

Kathleen N. Lohr, Ph.D.
Senior Program Manager for Health Services Research
Research Triangle Institute
Research Triangle Park, NC

Barbara J. McNeil, M.D., Ph.D.
Head
Department of Health Care Policy
Harvard Medical School
Boston, MA

Cynthia D. Mulrow, M.D., Msc.
Professor
Department of Medicine
University of Texas Health Science Center
San Antonio, TX

Mark A. Peterson, Ph.D.
Professor of Policy Studies and Political Science and Editor, Journal of Health Politics, Policy and Law
UCLA School of Public Policy and Social Research
Los Angeles, CA

Honorable Sam C. Pointer Jr.
Judge
United States District Court for the Northern District of Alabama
Birmingham, AL

I. Drummond Rennie, M.D.
Adjunct Professor
Department of Medicine
Institute for Health Policy Studies
University of California-San Francisco
San Francisco, CA

Arnold J. Rosoff, J.D.
Professor of Legal Studies and Health Care Systems
Department of Legal Studies
The Wharton School
University of Pennsylvania
Philadelphia, PA

Kenneth I. Shine, M.D.
President
Institute of Medicine
Washington, DC

Daniel W. Shuman, J.D.
Professor of Law
School of Law
Southern Methodist University
Dallas, TX

Harold C. Sox Jr., M.D.
Professor and Chair
Department of Medicine
Dartmouth-Hitchcock Medical Center
Lebanon, OH

Miles J. Zaremski, J.D.
Piper Marbury Rudnick & Wolfe
Chicago, IL

Alan R. Zwerner, M.D., J.D.
Senior Vice President, Strategic Planning
PersonalMD.com
Pleasanton, CA

Return to Contents

References and Notes

Bovbjerg, Randall. 1975. The Medical Malpractice Standard of Care: HMOs and Customary Practice. Duke Law Journal 1975:1375-1414.

Eddy, David M. 1996. Benefit Language Criteria That Will Improve Quality While Reducing Costs. Journal of the American Medical Association 275:650-657.

Field, Marilyn J., and Kathleen N. Lohr, eds. 1990. Clinical Practice Guidelines: Directions for a New Program. Washington, DC: National Academy.

Garber, Alan M., Milton C. Weinstein, G. W. Torrance, and M. S. Kamlet. 1996. Theoretical Foundations of Cost-Effectiveness Analysis. In Cost-Effectiveness in Health and Medicine, ed. M. R. Gold, J. E. Siegel, L. B. Russell, and M. C. Weinstein. New York: Oxford University Press.

Rennie, Drummond, and Harold S. Luft. 2000. Pharmacoeconomic Analyses: Making Them Transparent, Making Them Credible. Journal of the American Medical Association 283:2158-2160.

Return to Contents