Session B: Scientific Evidence and the Courts

Moderator: Stephanie Spurnak, J.D., George Washington University
Panel members: Gerard F. Anderson, Ph.D., Center for Hospital Financing and Management, Johns Hopkins University; and Patrick Lee, J.D., Crowell and Mooring

When Courts Review Clinical Practice Guidelines—Gerard F. Anderson, Ph.D.

A study was conducted on Federal and State court decisions that involved denial of health insurance claims because the insurer maintained the care was not medically appropriate, not medically necessary, or involved experimental treatments. The influence of scientific evidence on the courts' decisions was reviewed in particular. Scientific evidence included by the Food and Drug Administration (FDA) rulings, randomized clinical trials, observational studies, practice guidelines, consensus conferences, meta-analysis, literature review, and expert opinion. The study found that insurers were required to pay in 57 percent of the decisions. The court referred to a formal or informal guideline process (FDA, National Institute of Health (NIH), RAND, other organization, or group of experts) in written opinions 61 percent of the time.

The most common reasons cited by the courts when the insurers lost were: (1) the guideline did not apply to the specific case; (2) personnel reviewing the case were not sufficiently expert in the field; (3) the insurer denied a claim due to a financial conflict of interest; and (4) the insurer relied on outdated information in denying the claim.

The courts have difficulty interpreting multiple and contradictory medical practice guidelines. Accordingly, the Institute of Medicine (IOM) recommended that a separate organization assess such guidelines using these protocols:

• Develop a method for selecting which technologies or services need to be considered.
• Define parameters—safety, efficacy, cost, level of confidence in findings.
• Select someone to perform the evaluations.
• Specify a process.
• Establish criteria for obtaining new information when a guideline is outdated.
• Require the process to be binding on insurers and beneficiaries.
• Restrict court review to those guidelines developed under appropriate procedures.
• Restrict the court's remedy in its rulings.

Insurers are not aggressively denying claims, perhaps because they lose in court more than half the time and then suffer from public relations problems. With the growing availability of clearer guidelines, however, insurers are now more likely to prevail in court.

The Daubert Case and Expert Opinion—Patrick Lee, J.D.

The rules governing judicial admission of scientific evidence have changed considerably over the years. Since about 1300, the general judicial rule has been that witnesses may testify only about what they perceive with their five senses. When an area is outside common knowledge, a jury may hear testimony from a witness qualified by training, knowledge, skill, or experience in a particular subject. In 1923, the rule changed with United States v. Fry, in which the defendant in the trial attempted to introduce the systolic blood pressure deception test (a precursor to the lie detector test) into evidence. The court had never heard of this test and decreed that a scientific principle introduced in court must be generally accepted by the scientific community.

The rule evolved further in 1993, when the Supreme Court concluded in the Daubert case that the judge must determine the validity of scientific expert opinion and ascertain its reliability and usefulness to the jury. In this particular case (where a woman took a prescribed drug during pregnancy that resulted in birth defects in her child), it was understood that a verdict would be favorable to the plaintiff because of the jury's normal empathetic response to such injuries. Growth of specialized scientific knowledge also contributed to changes in the rule; the corresponding dramatic rise in expert opinion testimony raised concerns about its reliability.

The 1993 Daubert rule brings several changes:

• Judges now determine whether the jury can hear expert testimony, but do not decide causation.
• Litigation is more difficult for plaintiffs. Expert witness are limited in testifying about scientific evidence, which makes it more difficult for plaintiffs to "get to the jury."
• Judges will sometimes require education in scientific process for complex cases, although this is expensive.

The conclusion is that valid science is a good protector against unsafe products. Valid science enables both scientists and members of the judicial system to deliver better justice.

Return to Contents

Session C: Implementation of Evidence-based Practice: Differing Perspectives

Moderator: Frances Chesley, M.D., AHCPR
Panel members: Richard B. Rabens, M.D. M.P.H., Kaiser Permanente, Northern California; and Richard L. Doyle, M.D., Milliman and Robertson, Inc.

Strategies for Guideline Implementation: Some Ways To Approach the Issue of "Usability"—Richard B. Rabens, M.D., M.P.H.

The Kaiser Permanente clinical practice guideline program was established in 1995 to reduce variations in practice among its many facilities in Northern California. Kaiser's Health Plan—the corporate health maintenance organization that contracts with hospitals and providers—and the Medical Group, consisting of 4,500 physician providers—historically did not interfere with each other. The guideline program was at first viewed negatively by the Medical Group, which questioned the local autonomy of physicians versus central leadership of the Health Plan.

The Medical Group decided to focus on 10 "strategic health areas," some broad, for which they selected clinical practice goals. Providers would be accountable for achieving these goals. The guideline process had five important elements:

• Sponsorship: The program was encouraged by the leaders of the Medical Group with acquiescence of the Health Plan. Department chiefs sponsored guideline development teams devoted to specific outcome goals. Time and administrative resources were allotted.
• Stakeholder team: The guideline development teams were broad representative but manageable. Providers were not just physicians; patients sometimes were included on the team. Team leaders reported regularly to the departmental sponsors, and facilitators helped carry the process through.
• Evidence basis: To ensure physician acceptance, the guidelines were based on existing evidence from medical specialty groups and other sources, as well as Kaiser's own internal research. If no evidence was available, this was stated, with the recommendation that this fact be discussed with the patient. Guidelines were developed with the understanding that they would require updates. New evidence would be assessed by a guideline team.
• Academic "detailing": "Champions" were assigned at each facility to conduct internal and regional meetings to convey information on the guideline project. Continuing medical education programs included the champions' educational meetings in their agendas.
• Quality: The impetus for guidelines was not cost-containment, but quality, with the hope that fiscal benefits would accrue over time. This approach conformed best with the values of medical doctors, consumers, and purchasers. The commitment was to quality rather than cost, to outcomes rather than process.

From the beginning, the guideline project learned that implementation must be factored into guideline development. If a recommended treatment element is not available to subscribers through the Health Plan, the recommendation is pointless.

Implementation of Evidence-based Practice—Richard L. Doyle, M.D.

The goal of evidence-based practice is to improve the quality of care to a defined population and to be able to demonstrate socially desirable care, or "accountable quality care of a population over time." Included in the definition of high-quality care is effectiveness in the real world.

Clinical practice guidelines are the preferred approach to enhance quality by improving effectiveness and standardizing the best practices. Guidelines should use the best evidence available that is presented in a format that caregivers will respect and follow. Guidelines should become policies or programs aimed at improved patient care. Guidelines should address either particularly common conditions or areas where there is variation in practice in clinical practice.

Effectiveness as a goal of evidence-based practice includes (1) quality of life issues such as wellness, recovery, and function, and (2) satisfaction of patients and others. The amount of outcome information on effectiveness is limited, and what there is is often challenged based on selection bias.

Efficiency is another goal of evidence-based practice and, in this context, efficiency converges with effectiveness. If a job is not done right the first time, resources must be spent to fix the mistakes. To achieve efficiency through guidelines, we should focus on areas of high volume, high cost, and best practice. Following the care of patients who do well and have no complications helps physicians understand the starting point for efficient care, while still recognizing that some patients will require more.

Few randomized controlled trials have addressed efficiency, and cohort and sequential studies are infrequent. Professional opinion about efficiency should come from practitioners possessing the best available evidence about effective and efficient health care—those who have had to consider the resource needs of a population because of competition.

Implementation of policy involves the multiple disciplines required throughout treatment and followup. The infrastructure and barriers to delivery of evidence-based care must be taken into account in local practice settings. Management staff in charge of implementing new policy need to discuss new guidelines with stakeholders, including the patients. Verbal and written communication and audiovisual materials designed for patient education will reassure patients about the process that has developed the policy.

Development and publication of practice guidelines will not bring about enduring change in practice unless there is reinforcement through repetition and other measures, which tend to be part of a managed system. Repetition is a source of improved practice in cases where an advantageous new technology is publicized through major journals. The technology is addressed in informal discussion by those highly involved, the general media, and other national and local specialty meetings, teaching rounds, and focused courses. A challenge to repetition is that much health care information is channeled through primary care physicians, who have the widest scope of practice and need more evidence on more topics to maintain evidence-based practice.

Patients searching for evidence are ahead of doctors in Internet use. The exploding availability of information on the Internet will be a challenge for practitioners looking to improve health care quality, as well as a catalyst for the use of evidence-based practice.

Return to Contents

Session D: Implementation of Technology Assessments

Moderator: Clifford Goodman, Ph.D., Lewin Group
Panel members: Karen Flynn, D.D.S., M.S., Department of Veterans Affairs; Wade M. Aubrey, M.D., Blue Cross/Blue Shield; and Steven Sheingold, Ph.D., Technology and Special Analysis, HCFA

The Veterans Health Administration (VHA) Technology Assessment Program: Structures, Processes, and Models for Implementing Assessment Results—Karen Flynn, D.D.S, M.S.

The Veterans Health Administration (VHA) is made up of 1,000 hospitals and facilities and serves more than 3 million service-connected individuals annually. To optimize the effectiveness and quality of that care, the VHA engages in a great deal of research. The VHA Management Decision and Research Center, which functions like an internal consulting firm, attempts to balance management needs with research rigor. The Center also facilitates information exchange between researchers, clinicians, and managers.

Because of the high cost of Positron Emission Tomography (PET) combined with high demand for it in the VHA system, the Center recently conducted a technology assessment (TA) of PET at the request of the Undersecretary of the Department of Health and Human Services. The VHA has one of the largest PET holdings in the world, with 10 operational facilities. The Center's assessment found that quality evidence was lacking and could not conclude that PET is effective. PET operational staff had not communicated among themselves when conducting research, resulting in unfocused and unsystematic research strategies across facilities. This allowed for freedom, but made systematic review impossible. The Center made formal recommendations, including systematic data collection in the form of a VHA systemwide registry, and stressed the need for multicenter, well-designed studies of PET.

The study team learned many lessons about the evidence-based process, particularly regarding TA. The focus should be on specific disease processes with the technology being assessed; otherwise, the scope of the task becomes too broad. The study team should include generalists who manage diseases on the advisory committees as well as owners of the technology. The international TA community, which has experience with the difficulties of the process (e.g., how to deal with resource constraints) should be involved. Post-assessment resources and support for change should be provided. (For example, the Center established a VHA PET researcher hotline to support ongoing research and exchange.) To facilitate implementation by interpreting medical and TA languages, culturally sensitive, multilingual people should be used.

The implementation should be treated as educational rather than directive, to help the owners of the technology feel less vulnerable and defensive. The owners of the technology then will be more willing to do research to further explore the effectiveness of the tool, and they may be more accepting of the guideline.

Translating Evidence into Practice: Implementation of Technology Assessments—Wade M. Aubry, M.D.

The Blue Cross/Blue Shield Association (BCBSA), which manages 67 million subscribers and is a Federal contractor for Medicare and the Federal Employee Health Benefits Plan (FEHBP), is both a consumer and performer of TAs. The BCBSA Technology Evaluation Center produces 40 TAs per year that are evaluated by a multidisciplinary medical advisory panel. TAs must have Food and Drug Administration (FDA) approval, and evidence must permit conclusions concerning the effect of the technology on health outcomes. Evidence must come from well-designed studies in peer-reviewed journals; the technology's effect on health outcome must be convincing, must improve the net health benefits (benefits over cost), and must be better than an existing technology; and the improvement must be attainable outside the research setting.

Health plans use a particular TA as a rational basis for coverage decisions, developing clinical practice guidelines, methods for controlling costs, considering support of clinical trials, and defending potential disputes over coverage. Disputes over coverage have led some companies or organizations to override the evidence on which the TA is based to provide coverage for experimental treatments that have not yet been established as effective. Disputes usually arise concerning treatments for life-threatening or severely disabling conditions. Therapeutic options in these cases are often limited, increasing the demand for new, experimental treatments.

A credible, scientific, consistent process for TA is critical. The process requires physician involvement, which in turn requires physicians to be educated in the TA process. How society should deal with costs as a part of the process, particularly as technology becomes more and more expensive, remains unclear.

Implementing Technology Assessment for Medicare Coverage Decisions—Steven Sheingold, Ph.D.

From Medicare's perspective, over the past few years significant progress has been made away from locally focused, opinion-based care decisions to evidence-based national guidelines. This progress, however, is not without its difficulties when pressures have contributed to efforts by politicians to overrule science when faced with unpopular findings.

Increased use can lead to establishment of an experimental technology as part of "accepted practice," and once this occurs, the ability to deny coverage in the face of new evidence is limited. Fewer good studies are likely to be done because of the difficulty in conducting a randomized study of a treatment already considered an "accepted" or common practice (controls are then too difficult to find).

As a result of the pressures on TA development and implementation, combined with the serious implications of practicing without good TAs, the quality of the TA process is very important. Criteria for a strong evidence-based process in developing a TA include specific criteria for covered services, a well-defined process for evaluating the evidence, a well-described and open process, and dissemination of information about the decision. If there is only good evidence for the effectiveness of the treatment with particular populations or certain conditions, only those populations or conditions are covered by Medicare. Overall, if the evidence is poor, the technology is not covered.

Return to Contents
Proceed to Next Section