Session B: Scientific Evidence and the Courts
Moderator: Stephanie Spurnak, J.D., George Washington University
Panel members: Gerard F. Anderson, Ph.D., Center for Hospital Financing and Management,
Johns Hopkins University; and Patrick Lee, J.D., Crowell and Mooring
When Courts Review Clinical Practice Guidelines—Gerard F. Anderson, Ph.D.
A study was conducted on Federal and State court decisions that involved denial of health
insurance claims because the insurer maintained the care was not medically appropriate, not
medically necessary, or involved experimental treatments. The influence of scientific evidence on
the courts' decisions was reviewed in particular. Scientific evidence included by the Food and
Drug Administration (FDA) rulings, randomized clinical trials, observational studies, practice
guidelines, consensus conferences, meta-analysis, literature review, and expert opinion. The study
found that insurers were required to pay in 57 percent of the decisions. The court referred to a
formal or informal guideline process (FDA, National Institute of Health (NIH), RAND, other
organization, or group of experts) in written opinions 61 percent of the time.
The most common reasons cited by the courts when the insurers lost were: (1) the guideline did
not apply to the specific case; (2) personnel reviewing the case were not sufficiently expert in the
field; (3) the insurer denied a claim due to a financial conflict of interest; and (4) the insurer relied
on outdated information in denying the claim.
The courts have difficulty interpreting multiple and contradictory medical practice guidelines.
Accordingly, the Institute of Medicine (IOM) recommended that a separate organization assess
such guidelines using these protocols:
- Develop a method for selecting which technologies or services need to be considered.
- Define parameters—safety, efficacy, cost, level of confidence in findings.
- Select someone to perform the evaluations.
- Specify a process.
- Establish criteria for obtaining new information when a guideline is outdated.
- Require the process to be binding on insurers and beneficiaries.
- Restrict court review to those guidelines developed under appropriate procedures.
- Restrict the court's remedy in its rulings.
Insurers are not aggressively denying claims, perhaps because they lose in court more than half the
time and then suffer from public relations problems. With the growing availability of clearer
guidelines, however, insurers are now more likely to prevail in court.
The Daubert Case and Expert Opinion—Patrick Lee, J.D.
The rules governing judicial admission of scientific evidence have changed considerably over the
years. Since about 1300, the general judicial rule has been that witnesses may testify only about
what they perceive with their five senses. When an area is outside common knowledge, a jury may
hear testimony from a witness qualified by training, knowledge, skill, or experience in a particular
subject. In 1923, the rule changed with United States v. Fry, in which the defendant in the
trial attempted to introduce the systolic blood pressure deception test (a precursor to the lie
detector test) into evidence. The court had never heard of this test and decreed that a scientific
principle introduced in court must be generally accepted by the scientific community.
The rule evolved further in 1993, when the Supreme Court concluded in the Daubert case that the
judge must determine the validity of scientific expert opinion and ascertain its reliability and
usefulness to the jury. In this particular case (where a woman took a prescribed drug during
pregnancy that resulted in birth defects in her child), it was understood that a verdict would be
favorable to the plaintiff because of the jury's normal empathetic response to such injuries.
Growth of specialized scientific knowledge also contributed to changes in the rule; the
corresponding dramatic rise in expert opinion testimony raised concerns about its reliability.
The 1993 Daubert rule brings several changes:
- Judges now determine whether the jury can hear expert testimony, but do not decide causation.
- Litigation is more difficult for plaintiffs. Expert witness are limited in testifying about scientific evidence, which makes it more difficult for plaintiffs to "get to the jury."
- Judges will sometimes require education in scientific process for complex cases, although this is expensive.
The conclusion is that valid science is a good protector against unsafe products. Valid science
enables both scientists and members of the judicial system to deliver better justice.
Return to Contents
Session C: Implementation of Evidence-based Practice: Differing Perspectives
Moderator: Frances Chesley, M.D., AHCPR
Panel members: Richard B. Rabens, M.D. M.P.H., Kaiser Permanente, Northern California; and
Richard L. Doyle, M.D., Milliman and Robertson, Inc.
Strategies for Guideline Implementation: Some Ways To Approach the Issue of "Usability"—Richard B. Rabens, M.D., M.P.H.
The Kaiser Permanente clinical practice guideline program was established in 1995 to reduce
variations in practice among its many facilities in Northern California. Kaiser's Health
Plan—the corporate health maintenance organization that contracts with hospitals and
providers—and the Medical Group, consisting of 4,500 physician
providers—historically did not interfere with each other. The guideline program was at first
viewed negatively by the Medical Group, which questioned the local autonomy of physicians
versus central leadership of the Health Plan.
The Medical Group decided to focus on 10 "strategic health areas," some broad, for
which they selected clinical practice goals. Providers would be accountable for achieving these
goals. The guideline process had five important elements:
- Sponsorship: The program was encouraged by the leaders of the Medical Group with
acquiescence of the Health Plan. Department chiefs sponsored guideline development teams
devoted to specific outcome goals. Time and administrative resources were allotted.
- Stakeholder team: The guideline development teams were broad representative but
manageable. Providers were not just physicians; patients sometimes were included on the team.
Team leaders reported regularly to the departmental sponsors, and facilitators helped carry the
- Evidence basis: To ensure physician acceptance, the guidelines were based on existing
evidence from medical specialty groups and other sources, as well as Kaiser's own internal
research. If no evidence was available, this was stated, with the recommendation that this fact be
discussed with the patient. Guidelines were developed with the understanding that they would
require updates. New evidence would be assessed by a guideline team.
- Academic "detailing": "Champions" were assigned at each facility to
conduct internal and regional meetings to convey information on the guideline project. Continuing
medical education programs included the champions' educational meetings in their agendas.
- Quality: The impetus for guidelines was not cost-containment, but quality, with the hope that
fiscal benefits would accrue over time. This approach conformed best with the values of medical
doctors, consumers, and purchasers. The commitment was to quality rather than cost, to
outcomes rather than process.
From the beginning, the guideline project learned that implementation must be factored into
guideline development. If a recommended treatment element is not available to subscribers
through the Health Plan, the recommendation is pointless.
Implementation of Evidence-based Practice—Richard L. Doyle, M.D.
The goal of evidence-based practice is to improve the quality of care to a defined population and
to be able to demonstrate socially desirable care, or "accountable quality care of a
population over time." Included in the definition of high-quality care is effectiveness in the
Clinical practice guidelines are the preferred approach to enhance quality by improving
effectiveness and standardizing the best practices. Guidelines should use the best evidence
available that is presented in a format that caregivers will respect and follow. Guidelines should
become policies or programs aimed at improved patient care. Guidelines should address either
particularly common conditions or areas where there is variation in practice in clinical
Effectiveness as a goal of evidence-based practice includes (1) quality of life issues such as
wellness, recovery, and function, and (2) satisfaction of patients and others. The amount of
outcome information on effectiveness is limited, and what there is is often challenged based on
Efficiency is another goal of evidence-based practice and, in this context, efficiency converges
with effectiveness. If a job is not done right the first time, resources must be spent to fix the
mistakes. To achieve efficiency through guidelines, we should focus on areas of high volume, high
cost, and best practice. Following the care of patients who do well and have no complications
helps physicians understand the starting point for efficient care, while still recognizing that some
patients will require more.
Few randomized controlled trials have addressed efficiency, and cohort and sequential studies are
infrequent. Professional opinion about efficiency should come from practitioners possessing the
best available evidence about effective and efficient health care—those who have had to
consider the resource needs of a population because of competition.
Implementation of policy involves the multiple disciplines required throughout treatment and
followup. The infrastructure and barriers to delivery of evidence-based care must be taken into
account in local practice settings. Management staff in charge of implementing new policy need to
discuss new guidelines with stakeholders, including the patients. Verbal and written
communication and audiovisual materials designed for patient education will reassure patients
about the process that has developed the policy.
Development and publication of practice guidelines will not bring about enduring change in
practice unless there is reinforcement through repetition and other measures, which tend to be
part of a managed system. Repetition is a source of improved practice in cases where an
advantageous new technology is publicized through major journals. The technology is addressed
in informal discussion by those highly involved, the general media, and other national and local
specialty meetings, teaching rounds, and focused courses. A challenge to repetition is that much
health care information is channeled through primary care physicians, who have the widest scope
of practice and need more evidence on more topics to maintain evidence-based practice.
Patients searching for evidence are ahead of doctors in Internet use. The exploding availability of
information on the Internet will be a challenge for practitioners looking to improve health care
quality, as well as a catalyst for the use of evidence-based practice.
Return to Contents
Session D: Implementation of Technology Assessments
Moderator: Clifford Goodman, Ph.D., Lewin Group
Panel members: Karen Flynn, D.D.S., M.S., Department of Veterans Affairs; Wade M. Aubrey, M.D., Blue Cross/Blue Shield; and Steven Sheingold, Ph.D., Technology and Special Analysis, HCFA
The Veterans Health Administration (VHA) Technology Assessment Program: Structures, Processes, and Models for Implementing Assessment Results—Karen Flynn, D.D.S, M.S.
The Veterans Health Administration (VHA) is made up of 1,000 hospitals and facilities and serves
more than 3 million service-connected individuals annually. To optimize the effectiveness and
quality of that care, the VHA engages in a great deal of research. The VHA Management
Decision and Research Center, which functions like an internal consulting firm, attempts to
balance management needs with research rigor. The Center also facilitates information exchange
between researchers, clinicians, and managers.
Because of the high cost of Positron Emission Tomography (PET) combined with high demand
for it in the VHA system, the Center recently conducted a technology assessment (TA) of PET at
the request of the Undersecretary of the Department of Health and Human Services. The VHA
has one of the largest PET holdings in the world, with 10 operational facilities. The Center's
assessment found that quality evidence was lacking and could not conclude that PET is effective.
PET operational staff had not communicated among themselves when conducting research,
resulting in unfocused and unsystematic research strategies across facilities. This allowed for
freedom, but made systematic review impossible. The Center made formal recommendations,
including systematic data collection in the form of a VHA systemwide registry, and stressed the
need for multicenter, well-designed studies of PET.
The study team learned many lessons about the evidence-based process, particularly regarding
TA. The focus should be on specific disease processes with the technology being assessed;
otherwise, the scope of the task becomes too broad. The study team should include generalists
who manage diseases on the advisory committees as well as owners of the technology. The
international TA community, which has experience with the difficulties of the process (e.g., how
to deal with resource constraints) should be involved. Post-assessment resources and support for
change should be provided. (For example, the Center established a VHA PET researcher hotline
to support ongoing research and exchange.) To facilitate implementation by interpreting medical
and TA languages, culturally sensitive, multilingual people should be used.
The implementation should be treated as educational rather than directive, to help the owners of
the technology feel less vulnerable and defensive. The owners of the technology then will be more
willing to do research to further explore the effectiveness of the tool, and they may be more
accepting of the guideline.
Translating Evidence into Practice: Implementation of Technology Assessments—Wade M. Aubry, M.D.
The Blue Cross/Blue Shield Association (BCBSA), which manages 67 million subscribers and is a
Federal contractor for Medicare and the Federal Employee Health Benefits Plan (FEHBP), is both
a consumer and performer of TAs. The BCBSA Technology Evaluation Center produces 40 TAs
per year that are evaluated by a multidisciplinary medical advisory panel. TAs must have Food and
Drug Administration (FDA) approval, and evidence must permit conclusions concerning the effect
of the technology on health outcomes. Evidence must come from well-designed studies in
peer-reviewed journals; the technology's effect on health outcome must be convincing, must
improve the net health benefits (benefits over cost), and must be better than an existing
technology; and the improvement must be attainable outside the research setting.
Health plans use a particular TA as a rational basis for coverage decisions, developing clinical
practice guidelines, methods for controlling costs, considering support of clinical trials, and
defending potential disputes over coverage. Disputes over coverage have led some companies or
organizations to override the evidence on which the TA is based to provide coverage for
experimental treatments that have not yet been established as effective. Disputes usually arise
concerning treatments for life-threatening or severely disabling conditions. Therapeutic options in
these cases are often limited, increasing the demand for new, experimental treatments.
A credible, scientific, consistent process for TA is critical. The process requires physician
involvement, which in turn requires physicians to be educated in the TA process. How society
should deal with costs as a part of the process, particularly as technology becomes more and more
expensive, remains unclear.
Implementing Technology Assessment for Medicare Coverage Decisions—Steven Sheingold, Ph.D.
From Medicare's perspective, over the past few years significant progress has been made away
from locally focused, opinion-based care decisions to evidence-based national guidelines. This
progress, however, is not without its difficulties when pressures have contributed to efforts by
politicians to overrule science when faced with unpopular findings.
Increased use can lead to establishment of an experimental technology as part of "accepted
practice," and once this occurs, the ability to deny coverage in the face of new evidence is
limited. Fewer good studies are likely to be done because of the difficulty in conducting a
randomized study of a treatment already considered an "accepted" or common
practice (controls are then too difficult to find).
As a result of the pressures on TA development and implementation, combined with the serious
implications of practicing without good TAs, the quality of the TA process is very important.
Criteria for a strong evidence-based process in developing a TA include specific criteria for
covered services, a well-defined process for evaluating the evidence, a well-described and open
process, and dissemination of information about the decision. If there is only good evidence for
the effectiveness of the treatment with particular populations or certain conditions, only those
populations or conditions are covered by Medicare. Overall, if the evidence is poor, the
technology is not covered.
Return to Contents
Proceed to Next Section