Session H: Implementing Guidelines in Systems of Care

Moderator: Kay Pearson, R.Ph., M.P.H., AHCPR
Panel members: Richard Adelson, M.D., Department of Veterans Affairs, Minnesota; A. John Rush, M.D., University of Texas Southwestern Medical Center

Implementing Guidelines: Designing and Selecting Effective Strategies—Richard Adelson, M.D.

Continuing medical education has been found to have little impact on performance improvement; thus the right knowledge and skills do not guarantee changed practice. Effective strategies are needed bring about improved performance. One concept is the "Performance Improvement Model," which consists of three stages: awareness, competence, and performance. Mediating the movement from awareness to competence to performance are "predisposing," "enabling," and "reinforcing" factors. Predisposing or attitudinal factors influence whether the provider will change behavior; enabling factors support changing behavior; and reinforcing factors maintain and support the modified behavior.

To analyze implementation of a guideline using this model, three questions must be asked:

1. Is the target group predisposed to implement the guideline?
2. Does the target group have the appropriate knowledge and skills?
3. Does the work setting support the proposed changes?

Predisposition takes into account the target group's perspectives on (1) whether a problem exists, (2) the validity and applicability of the recommended intervention and also whether the physician peer group and patients consider the intervention valid, and (3) the level of priority for cost versus benefit. Two enabling factors that support change are the social system (how the provider fits into the social network) and the technical system. Reinforcing factors include organizational culture, the reward and recognition system, and policy and procedures.

In summary, (1) implementing guidelines is a complex problem, requiring a systematic assessment; (2) multiple strategies can be used depending on the guideline, the target audience, and the work setting in which the guideline is to be applied, and (3) physicians' predisposition, knowledge, skills, and work setting must be analyzed for their impact on guideline implementation.

Texas Medication Algorithm Project—A. John Rush, M.D.

The Texas Medication Algorithm Project began with a review of public sector diagnoses in the Texas Department of Mental Health and Retardation. Initial findings revealed inaccurate diagnoses that prevented the initiation of a treatment plan. Short visit times, psychotic patient behavior, and lack of informants and medical records contributed to the problem. A need existed to improve physician training and to eliminate obstructing factors, so the State established a requirement for more structured clinical interviewing and a more thorough diagnostic process. In addition, algorithms were created for three major disease groups: schizophrenia, manic depression, and bipolar disorder. The algorithms were developed with participation of experts as well as interested consumers and advocates. A pilot feasibility study was initiated that included expert technical assistance with all providers and education on practical use of study medications. Preliminary results of the feasibility study point to positive outcomes for both physician implementation of the algorithms and for patient response to the study medications. For the last phase of the project, a comparative impact study will be conducted, in which treatment incorporating the algorithms, education, and physician support is compared with treatment as usual.

Return to Contents

Health Policy and Quality

Robert Brook, M.D., Sc.D.,
RAND Corporation

Service provision among managed care organizations in Southern California suggests that a new service or product that improves quality of care will not be introduced unless it also saves money, because the organization's primary goal is to stay in business. An organization also has to increase its market share, which is accomplished by improving patient satisfaction. To accomplish these goals, technical care must be reduced to the lowest possible level without sacrificing patient satisfaction. These goals illustrate the "cost-quality tradeoff." Health care effectiveness research must focus on ways to balance quality and cost and achieve a successful tradeoff.

One model of the relationship between health and resources depicts a direct correlation between the two: as resources increase, optimal health increases. U.S. policymakers tend to follow this model. An alternative model posits that there is no relation between health and resources. A third model, more applicable to the U.S. environment, holds that too few resources are spent on sick adults with chronic disease while too much is spent on activities that do little to improve wellness. Studies show that when care is changed through financial incentives, little impact on the cost-quality relationship occurs. Efforts of AHCPR and others to improve the amount of quality obtained from cost are vital.

The RAND Health Insurance Experiment showed that free health care and cost-shared care did not differ in terms of services obtained and patient health outcomes. These results led to a revolution in the insurance industry, so that most organizations now have either managed care organizations or coinsurance or deductibles.

This presentation argued that effectiveness studies aimed at improving the cost-quality tradeoff should focus on the elderly. In the U.S., an extraordinary investment in providing health care to the elderly has been made, yet most controlled trial literature still excludes the elderly from studies. NIH spends very little on controlled trials looking at cancer or heart disease in the elderly. Thus, results have to be extrapolated from studies on the non-elderly. Other countries have greatly reduced the services provided to their elderly populations.

The components of quality care are as follows:

1. Appropriateness should be directly measured. Data are analyzed and clinical judgment determine criteria for medical appropriateness.
2. All work should be necessary, it should be done well, and it should be efficient.
3. Patients should be satisfied.

The first two components make up most of the cost-quality tradeoff. Studies of health outcomes have indicated that data must be combined with a quality system and then used to make a difference in quality of care.

Dr. Brook stressed that to provide all necessary care to all patients, several goals must be achieved. The first is agreement on what is appropriate, necessary care. Such agreement can be achieved through use of a clinical method involving literature analysis, development of a list of indicators, and convening of expert panels. Both overuse and underuse of care remain significant concerns. The second goal for providing necessary care to all patients is the elimination of waste, which would enable providers to provide more of the necessary services. Furthermore, public funds should not be expended on care for which the cost exceeds the benefit.

The literature on effectiveness must find a better route to the practical world of change. The literature in the short run is not adequate to effect change without the addition of expert judgment. The data are not sufficient. Furthermore, an economic model for valuing a year of life must be devised.

Care can be improved without implementing rationing. Waste can be eliminated without affecting health outcomes. Policies should be implemented that eliminate waste. The resources saved by eliminating waste should be expended on new drugs and devices or older technologies used appropriately to achieve better health.

Return to Contents

Practice Guidelines and Malpractice Litigation: Collision or Cohesion?

Troyen A. Brennan, M.D., J.D., M.P.H., Harvard University

Medical malpractice is tort law in the area of medicine or personal injury law. The objectives of tort law are to (1) compensate individuals who have been injured by substandard medical care and (2) deter behavior that causes injuries. To prove a tort case in medical malpractice, the following must be shown: (1) a dutiful relationship existed between doctor and patient, (2) injury or disability was inflicted, and (3) negligence caused the injury—the doctor failed to reach the standard expected of a reasonable practitioner. Showing negligence remains the most difficult part of the suit to prove and must be verified through testimony by an expert doctor.

The past two decades have seen dramatic changes in the way the judicial system handles malpractice claims. Insurance companies were frequently targeted, and plaintiffs attorneys gained more and more experience in handling malpractice suits. At present, plaintiffs' attorneys firms, malpractice defense firms, and insurance companies have all become expert at handling such suits, and the number of medical injuries appears to be decreasing.

Two major types of guidelines—guidelines and critical pathways—can play a role in tort litigation, as tools available to the lawyers. Clinical practice guidelines are published by AHCPR and medical specialty societies; critical pathways are smaller documents developed by physician groups from the larger guideline documents. Benefits of using guidelines include decreased inappropriate care, better compliance with the standard of care, reduced medical malpractice claims, and decreased defensive medicine. Defensive medicine can be either negative or positive. The negative type is characterized by not performing a standard procedure because of malpractice risk. Positive defensive medicine occurs when the standard has been met, but an additional test or procedure is done to avoid litigation. A guideline can reinforce the standard of care to reassure the doctor in such a situation. Defining the standard of care is critical to medical malpractice litigation.

Guidelines can address either standard of care or appropriateness. Standard of care guidelines are of interest to those involved in medical malpractice, risk management, and quality adherence. Standard of care guidelines improve quality and are useful for avoiding injuries but do not necessarily reduce cost. Appropriateness guidelines are important to the utilization manager and the insurance company, but they address a different quality issue. An example is the variation in care provided between geographic areas. Reduction of inappropriateness should reduce the cost of health care.

Studies have shown that physician compliance with guidelines is poor. If a physician has complied with guidelines, however, the defense lawyer will use that compliance for exculpatory purposes (i.e., to free the physician from blame). If a physician fails to adhere to a guideline, a plaintiff's attorney can bring a case against him or her on inculpatory grounds. Compliance with guidelines confers a strong defense against medical malpractice litigation.

A review of litigation files of several insurers revealed that guidelines were used for inculpatory purposes in the majority of cases. A large percentage of the cases were settled with a payment to the plaintiff. On the exculpatory side, they did not have as strong an effect in providing a defense. The use of guidelines continues to diffuse through the legal profession.

Designers of guidelines must take care to include all steps for achieving appropriateness of care and cost effectiveness. Any negligence on their part could make them liable for litigation. As guidelines are used increasingly to manage patients, more litigation will revolve around these issues.

Additional points raised were the following:

1. Doctors do not like to follow negative recommendations (e.g., "do not screen for prostate cancer") in guidelines, fearing they will nevertheless be sued. Use of supporting literature would serve as defense if the doctor were sued for following a negative recommendation.
2. Managed care organizations that have implemented their own guidelines should be liable for lawsuits. Most doctors see themselves as the sole target of malpractice.
3. The standard of care may differ from prevailing practice.

Return to Contents

Learning from Errors

Lucian Leape, M.D., Harvard University

Physical and financial costs of injury and death from medical errors often exceed those from inappropriate care. The extent of medical injury from errors remains high and preventable. The Medical Practice Study, a record review study, found that 3.7 percent of patients in acute care hospitals in New York State had an injury due to treatment, approximately 69 percent of those caused by errors. Extrapolation of findings from the study to the whole country yielded annual estimated tolls of 1.3 million injuries, 180,000 deaths, and $50 billion (in 1989 dollars) in total costs.

There are several reasons for the high error rates. One is that the approach to promoting safety is primitive, based on the myth that the way to eliminate errors is to perform perfectly, but human beings are incapable of sustained perfect performance. In addition, fear of punishment for performance errors inhibits error reporting. As a result, the punitive approach to error reduction is often unsuccessful.

Cognitive psychology research on error prevention has shown that the human mind functions by two mechanisms: automatic and problem solving. Most daily experiences are handled in the automatic mode, which is unconscious, rapid, parallel, and effortless. When a problem arises, the mind switches to the problem-solving mode, which is conscious, slow, sequential, and difficult. In this mode, the mind first applies a rule, and if that fails, the mind then sequentially applies pattern matching and analytic/synthetic thought. A wrong conclusion might still be reached, however, for reasons ranging from biased memories to overconfidence.

Human factors theory embodies the following principles:

1. Errors are common.
2. The causes of errors are known.
3. Errors are byproducts of useful cognitive functions.
4. Most errors are caused by activities that rely on weak aspects of cognition (e.g., short-term memory, attention).
5. Errors can be prevented by designing tasks and processes that minimize dependency on weak cognitive functions.
6. System failures are at the root of most errors.

Application of human factors principles in the aviation industry has led to high levels of error-free performance. Systems design has focused on preventing errors through reduced reliance on memory, improved information access, standardization (e.g., routine maintenance), error-proofing, and safety training. This design also uses monitors, backup systems, and computerization; establishes detailed work schedules and task descriptions; and creates a nonpunitive environment. Extensive training and examination are important. Safety has been institutionalized within the industry through establishment of the Federal Aviation Administration and the National Transportation Safety Board.

The health care industry could benefit from application of human factors principles. The existing medical model focuses on individuals rather than on systems, incorporates task design sporadically, is reactive to error, does not address performance examination, and does not institutionalize safety. Violations of human factors principles in medical practice include reliance on memory and vigilance, nonstandard processes, an excessive number of handoffs, long work hours, excessive workloads, sporadic feedback, and variable information availability.

The Adverse Drug Event (ADE) Prevention Study attempted to discover ADEs and underlying errors, identify the underlying systems causes of the errors, and develop system changes to reduce the errors. It was suggested that the top three system failure areas—drug knowledge dissemination, dose and identity checking, and patient information availability—be targeted, perhaps using a computerized physician order entry system. The burden of addressing quality assurance problems should be shifted from the physician to the institution. It should be noted that investigation into error is protected from litigation in many States.

Return to Contents

Session I: Implementation and Adaptation of Guidelines Developed at a National Level

Moderator: Francis Chesley, M.D., AHCPR
Panel members: Joseph Lau, M.D., New England Medical Center; David Katz, M.D., University of Wisconsin Medical Center; and David B. Nash, M.D., M.B.A., Thomas Jefferson University Medical Center

Interpreting Evidence: Some Caveats in Implementing and Adapting Guidelines Based on Evidence from Randomized Controlled Trials—Joseph Lau, M.D.

An AHCPR system used for the Cardiac Rehabilitation guideline encompassed three levels of evidence, the highest derived from randomized controlled trials (RCTs), and the lowest from expert opinion. The use of pooled results of multiple RCTs (meta-analyses) as a source of evidence-based medicine has increased dramatically. Most meta-analyses have been done for cardiovascular disease, and most are published in the leading journals (more than 600 journals).

In a comparison of the typical meta-analysis method and the cumulative method applied to studies of intravenous streptokinase therapy in acute myocardial infarction, the cumulative method demonstrated evidence of efficacy beginning in literature published 15 years before the date of the analysis. A comparison of results of cumulative meta-analysis with expert opinions from review articles and textbooks showed that the cumulative meta-analysis demonstrated efficacy early on, even as the experts were still unaware of the benefits of this treatment.

Significant discrepancies have been found between conclusions from meta-analyses and results of megatrials (RCTs of more than 10,000 patients). The megatrials did not confirm the meta-analysis results. However, in a study of meta-analyses of large trials (1,000 or more patients) versus smaller trials, the results calculated by means of the random-effects model revealed that almost all discrepancies could be explained.

Metaregression is increasingly being used to explore the discrepancies between clinical trial results and meta-analysis results. The unit of analysis is the individual study in a meta-analysis. The independent variable is the covariate of interest (e.g., control rate, dosage). The dependent variable is the treatment effect (e.g., log odds ratio, relative risk). Inherent problems with metaregression are that significant association does not necessarily prove causality; average data used to represent a group may misrepresent the individual patient; covariates are unavailable; and selection of covariates may be biased. If covariates are reported inconsistently or are unavailable, the control rate in RCTs (i.e., the proportion of patients assigned at random to the control group with the event) may be used. This rate is reported in any clinical trial that has dichotomous outcomes.

Evidence from RCTs should be used in guideline development with the following considerations:

• Heterogeneity should be fully represented across multiple dimensions ("one size does not fit all").
• Presence of significant correlation between treatment effect and control rate in a collection of clinical trials requires understanding the differences in patient characteristics, disease manifestations, and treatment variations.
• Individual patient data meta-analyses should be encouraged.
• Future studies should be planned with meta-analysis (and guidelines) in mind and should anticipate heterogeneity.

RCTs provide a population estimate of treatment efficacy, but when doctors try to apply population results to individuals, there are problems. Thus, some have proposed "N-of-1" trials (single-patient crossover trials, or each patient serves as his or her own control with time on drug and time off drug measured) to determine efficacy for individual patients, at least in some chronic conditions. Collecting data from multiple N-of-l trials is useful both to estimate population treatment effects and to evaluate individual patient responses to treatment.

Evidence-based medicine is heading toward medical information systems, population-based evidence (e.g., from systematic review, published meta-analyses, practice guidelines), collection of patient- and practice-specific data into databases to understand the variability that exists, and use of N-of-1 trials to help provide "real time" and individualized evidence to support optimum patient care.

The Need to Include Empiric Clinical Data in Formulating Medical Practice Guidelines—David Katz, M.D.

Implementation of clinical practice guidelines is generally assumed to improve medical care by reducing inappropriate practice patterns. This is supported by evidence-based review of practice guideline literature. In contrast, however, an increasing amount of evidence suggests that guidelines do not always produce their intended effect in clinical practice. Reasons for this might be the following:

• The guideline is ignored or misapplied, especially if overly complex or difficult to put into practice.
• The effects of modifying the guideline to conform to local beliefs and systems of care are unpredictable.
• Guideline implementation may have unexpected effects on other aspects of medical care (e.g., reducing length of stay in the intensive care unit (ICU) may increase stays outside of the ICU).
• Many guidelines are based on poor-quality scientific evidence.

In one recent survey of medical directors of managed care physician groups, 87 percent of groups were involved in development or implementation of clinical guidelines. The three most important influences in the development of guidelines were (1) local expert opinion or judgment, (2) medical literature, and (3) previously developed clinical guidelines. National or local data were a lesser influence. The level of evidence needed to support guideline recommendations can vary. Scientific proof is probably not necessary, for example, for guidelines based on common sense (e.g., counseling patients about the dosage and major side effects of anti-anginal medication). On the other hand, controversial guidelines require rigorous evaluation when the consequences of incorrect decisionmaking could be morbidity, mortality, or excess cost.

The role of the AHCPR Unstable Angina guideline in triage of emergency department (ED) patients was studied. The guideline calls for stratification of patients based on risk factors for poor short-term outcome and makes triage recommendations. It was hypothesized that applying the guideline would reduce unnecessary hospitalizations and use of coronary care unit (CCU) beds primarily by discouraging admission of low-risk patients. Therefore, the low-risk group was the particular focus of the study. The aims of the study were to determine the clinical applicability of the Unstable Angina guideline in a consecutive series of patients diagnosed with unstable angina and to project the potential impact of implementing the guideline on triage decisions.

Data were examined from a prospective clinical trial of the triage of patients with possible acute cardiac ischemia (ACI) at five urban hospitals during 1993. Those patients who had received an ED diagnosis of "unstable angina" or "rule-out unstable angina" were studied and put into risk groups according to the guideline parameters. Overall, the guideline risk groups conformed to the ED clinician-judged diagnoses of ACI and cardiovascular morbidity, but actual ED disposition differed from the guideline recommendations in two major areas: only 4 percent of the low-risk patients were discharged to home with outpatient followup, and only 40 percent of the high-risk patients were admitted to an intensive care unit. Thus, it appeared that physicians were not following triage recommendations for low- and high-risk patients at the time of the study.

This lack of compliance in handling low-risk patients could be explained by the fact that clinical policies at the study hospitals recommended admission for all patients with new-onset or stable angina, which was not surprising in light of concern about litigation. In the case of the high-risk patients, 25 percent were admitted to nonintensive care units of the hospital because ICU beds were not available at the time of triage. For many of the patients, chest pain was adequately controlled with medication that could be prescribed outside of the ICU.

Thus, the effect of the AHCPR guideline in reducing hospitalization of patients with unstable angina is likely to be slight. Of the 6 to 7 million patients presenting annually in acute care settings with symptoms of ACI, only 10 percent are diagnosed with unstable angina. Of these patients, the guideline would classify only 6 percent as low risk, which is less than 1 percent of the total target population. On the other hand, the potential savings from reduced hospitalization of these patients could be substantial on a national level. If fully implemented, the guideline could increase the demand for limited CCU beds.

The researchers recommended that developers of guidelines make use of empiric data from target clinical settings to ensure that guideline recommendations apply to a significant clinical population. When the evidence base is weak and the consequences of error are serious, the actual impact of guidelines on clinical care should be addressed prior to dissemination. Finally, existing databases with detailed clinical data can provide an efficient means to assess the impact of guidelines without conducting a large-scale effectiveness trial.

AHCPR Guidelines on Heart Failure: Comparison of a Family Medicine and Internal Medicine
Practice with the Guidelines and an Educational Intervention To Modify Behavior—David B. Nash, M.D., M.B.A.

Guidelines have proliferated in the United States so that, for example, we can now obtain a CD-ROM disk containing 1,100 practice guidelines. The impact of guidelines on quality of care, however, has been little investigated. This study compared the recommendations of the AHCPR guideline, Heart Failure: Evaluation and Care of Patients with Left-Ventricular Systolic Dysfunction, with practice at Thomas Jefferson University Hospital in Philadelphia. The immediate goal was to use the guideline as a benchmark to ascertain areas for improvement. Congestive heart failure was used because it is a highly expensive medical problem nationally; it is a common cause of hospitalization, disability, and death; and more than 3 million Americans are admitted to hospitals with the disease.

The complex guideline recommendations and algorithms had to be read and understood and then distilled into a checklist to enable comparison with institutional practice in a retrospective, chart-based review system. The investigators had to establish a set of performance measures against which they could analyze practice. Criteria established by the Demonstration Project for Ambulatory Care (DEMPAC) Improvement Project served as a model for the checklist.

The study hypotheses were ACE inhibitors are underutilized; documentation of care is less than optimal; different practices show unexplained variation; and evaluation of left-ventricular ejection fraction is less than optimal. The study group consisted of a random selection of 50 outpatients from each of 2 outpatient practices—internal medicine and family medicine—in fiscal year 1994.

The study results were that left-ventricular function had been documented in only 79 of the 100 cases. Although there were differences between the internal and family medicine practices, only three were significant. (When the researchers compared an outpatient cardiology practice against the guideline criteria, it performed very well.) The study did not include any action not noted in the charts, even if physicians later said that such action had occurred.

Conclusions from the study were: the two practices exhibited significant differences in patterns of care; ACE inhibitors were not used optimally; target doses of ACE inhibitors were not reached in most patients; evaluation of left-ventricular ejection fraction was less than optimal; and documentation of care was less than optimal. A nonpunitive educational program was implemented to modify physician behavior, with the use of a rented audience response system.

It was concluded that appropriate implementation and feedback are important for successful application of guidelines. The use of a case manager on the health care team may help to improve the quality of care in the outpatient setting. This person's sole responsibility is to ensure compliance (in a friendly manner) with critical pathways. Integration, capitation, managed care, and transition to continuous quality improvement will accelerate the use of guidelines. Strategies to increase compliance must be sought.

Return to Contents
Proceed to Next Section