Session J: Using Adverse Event Data to Improve Quality of Care
Moderator: Martin Hatlie, J.D., American Medical Association
Panel members: Brent James, M.D., M. Stat., Intermountain Health Care; and Richard Cook,
M.D., University of Chicago
Every Defect a Treasure: Using Defect Data to Reduce Adverse Drug Events—Brent C. James, M.D., M. Stat.
Health care can be regarded as a series of interrelating processes functioning together in systems,
and the application of quality improvement (the science of process management) in health care
delivery can improve patient care and help to control costs.
Every process consistently produces three classes of outcomes—physical, service, and cost.
Physical outcomes include medical outcomes such as complications and therapeutic goals, and
functional status measures such as patient perceptions of medical outcomes. Service outcomes
include satisfaction and access issues such as waiting times. A causal relationship exists between
medical and cost outcomes so that a system designed to control quality will affect cost. To
demonstrate the relationship between quality of care and cost, implementing processes in one
hospital to improve timing for antibiotic prophylaxis increased optimal timing from 40 percent in
1985 to 96 percent in 1991. The hospital also reduced postoperative deep wound infections from
1.8 to 0.4 percent for an estimated annual savings of $714,000 and 51 fewer cases. In an second
example, applying two quality improvement tools—the cause and effect diagram and the
tally sheet—revealed six potential causes (drug reaction; failure to adjust for decreased renal
function, patient age, patient body mass, or known hematologic factors; error in dosage) that
accounted for 80 percent of all adverse drug effects (ADEs) with 66.2 percent of those
preventable. When a tool was implemented to calculate renal clearance and to correct for failure
to adjust for decreased renal function, the overall ADE rate dropped almost 50 percent and saved
the hospital about $500,000 in the cost of care annually.
A judgment-based approach, asking who to find and eliminate persons not meeting an
acceptable quality threshold, results in assumptions about what remains after inspection and
elimination of serious failures. A learning-based approach, asking why-what-how, focuses
on the process of care to understand the causes of failure. Accordingly, quality improvement finds
and eliminates inappropriate variation (process steps) and documents continuous improvement
(outcomes). We have the opportunity to redefine the meaning and the nature of medicine (i.e.,
improved patient care delivery and reduced cost), but first we must understand and manage the
underlying processes of health care.
Using Adverse Event Data to Improve Quality of Care—Richard Cook, M.D.
How do we learn from adverse events? We are constantly learning, but the issue is whether what
we learn is correct. Associations between things that appear to be correlated may lead us to
believe that they are causal, when in fact, they are unrelated; therefore, we need to find ways to
go beyond seeing associations.
During the past 15 years, our understanding of how large, complex systems fail has changed.
They fail in a particular way, that is, they have certain "signatures." Multiple faults
occurring together, rather than a single point failure, cause system failure; there is no root cause.
These small, otherwise innocuous, faults that exist in complex systems are termed latent failures.
The "Swiss cheese model" of the path of causality of an accident shows a group of
defenses or shields such as technical factors (e.g., safety interlocks, automated devices), human
characteristics (e.g., training, expertise of operators), and organization and institutional structure
(e.g., procedures, team functions, reviews) and "holes" or latent failures within each
shield. When many latent failures align, a chain of events, or opportunity, is created that leads to a
failure. The implication is that it is hard to defend against a particular accident. Hindsight bias
(adding weight retrospectively to factors known to be causally related to the outcome) coupled
with the Swiss cheese model of the way complex systems fail, leads to a cycle of error. A shift in
the loci of system failures occurs, and, because latent failures are always present in different
circumstances, there is a quiet period. Another accident that appears to be unique occurs from a
different collection of latent failures, and the cycle of error is repeated.
To learn from medical errors, the presenter proposed the creation of Project Afterwards and
Project Aftermath. In Project Afterwards, a national patient safety board would explore obstacles
to detection, reporting, investigation, analysis, and dissemination of accident data and explore
novel responses to protoaccidents and the dynamic construction of safe operation in a changing
sociotechnical world. Initially, the organization could collect data (e.g., narrative stories, chart
copies, photographs) from participants in medical accidents and those who respond to them.
Project Aftermath would test the principle of using immediate response teams to investigate
technical and organizational sources of medical accidents, similar to the National Transportation
The presenter noted that we always have trouble distinguishing between error and failure. Error is
a process of social attribution applied to unwanted outcomes. We also need to be careful of the
application of technology as the solution. Technology may solve some problems but create others.
In medicine today, we tend to be moving from frequent, low-consequence failures to infrequent,
high-consequence failures as a consequence of applying technology and, thereby, increasing
Return to Contents
Session K: Educating Physicians for Evidence-based Practice
Moderator: Lynn Kazemekas, Ed.D., AHCPR
Panel members: R. Brian Haynes, M.D., Ph.D., McMaster University; Lawrence G. Smith, M.D.,
Mt. Sinai Medical Center (NY); and Dave A. Davis, M.D., University of Toronto
Evidence-based Information Resources for Clinical Learning and
Practice—R. Brian Haynes, M.D., Ph.D.
Health professionals must maintain awareness of current research in their specialty.
Subject-specific services and journals that select and review studies of high quality are becoming
more available as evidence-based practice becomes more widespread. These services and journals
can serve as the practitioners' main source of current research information.
Electronic databases help us look up several studies on a given topic. MEDLINE is a searchable
database for medical research studies across all specialties. New databases are specific to
particular areas of practice, such as Best Evidence and ACP Library on Disk. The Cochrane
Library, a database of randomized clinical trials, is available on CD-ROM. The Internet is a useful
tool for medical information. Sites of interest include MEDLINE (http://www.ncbi.nlm.nih.gov/PubMed/), McMaster University's
evidence-based practice site (http://hiru.mcmaster.ca/),
SilverPlatter (http://php.silverplatter.com), and a site-specific to diabetes (http://www.diabetes.com).
Practitioners need to be trained to perform competent literature searches on bibliographic
databases, and they must know about the tools available via CD-ROM and the Internet.
Physicians should subscribe to evidence summary services that are tailored to their particular
specialty areas. Clinical settings need to provide computer access for literature searching,
CD-ROM use, and Internet connections so that practitioners can access the information
immediately in the course of treating patients.
Evidence-based Medicine: A Cultural Shift in Residency Training—Lawrence G. Smith, M.D.
Evidence-based medicine is difficult to implement for a variety of reasons, including inefficient
procedures for accessing relevant evidence, time constraints for busy practitioners, few role
models, a lack of sufficient evidence to answer many medical questions, and complicated patients
who do not fit into single diagnostic categories. Evidence-based practice has a relative lack of
cultural acceptance among senior clinicians, which is a major impediment to new physicians.
Mount Sinai Hospital is attempting to bring about a cultural shift toward evidence-based practice.
The teaching methods are changing from the more traditional, passive lecture to a highly
interactive, problem-solving model that forces students to build on previous knowledge by using
and reinforcing that knowledge and supporting it with evidence. Students are taught to think and
to question accepted practices by seeking evidence to support such practices.
Medical students are taught nine key skills:
- Frame the patient's problem with a declarative statement, collecting, then citing, raw data to support this formulation.
- Ask the relevant question that evidence needs to answer.
- Think aloud through the process of clinical judgment to allow correction of mistaken assumptions.
- Identify their own knowledge gap or any additional information needed to advise the patient.
- Design a strategy to solve the knowledge gap through their own initiative, CD-ROM searches, literature searches, or consultation of specialists.
- Search for evidence.
- Appraise evidence critically.
- Integrate the evidence into the care of the patient.
- Teach the evidence they have learned to peers and students.
Medical students use these skills as they interact with peers and teachers in various settings. Care
also is taken to ensure Students' ongoing respect for clinicians who choose the more traditional
means of practice, such as expert judgment and clinical reasoning. Students ideally incorporate
use of evidence with use of logical, clinical reasoning based on seasoned experts' experience in
establishing their own style of practice.
Changing Physician Performance: Use of Enabling and Reinforcing
Techniques in Maintenance of Learning and Change—Dave A. Davis, M.D.
Continuing medical education (CME) varies in its ability to effect changes in physician
performance. Types include educational materials, formal activities (e.g., conferences, which can
be didactic or interactive), patient-mediated strategies (e.g., providing educational materials to
patients), and use of opinion leaders.
The researchers performed a meta-analysis of more than 100 randomized clinical trials by
objectively assessing physician performance on health care outcomes of various types of CME to
determine which strategies work best. The majority of studies demonstrated an improvement after
educational intervention. The most effective strategies included reminders, patient-mediated
interventions, outreach visits, use of opinion leaders, multifaceted intervention (use of more than
one CME strategy), and formal activities designed for interactive participation.
CME needs to use practice-based interventions, particularly those focusing on the patient (e.g.,
providing educational materials to the patient), as these are highly effective. Use of a combination
of multiple CME strategies together seems most effective for changing physician
Return to Contents
Session L: Legislating Clinical Care
Moderator: Edward B. Hirshfeld, J.D., M.B.A., American Medical Association
Panel members: Alex R. Rodriguez, M.D., National Account Consortium, Inc.; George Isham,
M.D., HealthPartners, Minneapolis; Honorable Jane Maroney, Delaware State
Legislating Clinical Care—Edward B. Hirshfeld, J.D., M.B.A.
Historically, legislators have held several roles: regulating health insurance, facilitating entry of
new health carriers, financing to expand access, mandating coverage of specific care and products,
ensuring standards of care and practice guidelines, and addressing issues of coverage and
standards. Legislation has long mandated that health plans cover certain medical services. During
the era of health care reform. State health care reform statutes began including legislation that
facilitates the adoption of practice guidelines by physicians. The objective was to provide
physicians with defense against malpractice actions, if they followed the guidelines properly.
Recently, legislation has begun to address both coverage of care and methods for provision of
care, such as minimum hospital stays for mother and child after birth. Issues under debate concern
questions of stake in health care including: What obligation does the legislature have to ensure
that all Americans have health care coverage?, If the legislature is not obligated to ensure
coverage, how far does its role in consumer protection extend?, Can it legislate coverage
mandates for private insurers?, Should it legislate standards of care?, What standards should be
used to evaluate the health care that is covered-specific needs of the individual or maximization of
health care status in the population?, Who should decide the needs of the individual versus those
of the population (voters, judges, the free market)?, and What responsibility does the medical
profession have to help decide these issues?
Legislation and Clinical Care—Alex R. Rodriguez, M.D.
Legislation can help provide responsible, accountable managed care or it can interfere with those
goals. Doctors working within and for managed care organizations are committed not only to care
for the individual patient, but also for the family and the community by addressing such problems
as communicable diseases and family violence. Managed care organizations consider themselves
to be patient care advocates, and they seek involvement in legislative solutions to barriers to
patient care quality.
Legislation of clinical aspects of care is increasing primarily because consumers and purchasers
have become more knowledgeable about the processes of care relating to risk, quality, and cost.
Thus, there is increasing demand for confidentiality, members' rights and responsibilities, and
quality improvement. The activities of consumer advocacy groups have shaped public policy, and
the resulting legislation has improved health care. The managed care industry has become
concerned, however, about its liability for medical errors that existed before the advent of
Significant errors in health benefits administration in managed care have led to public distrust.
Managed care organizations must strive for more flexibility when dealing with individual patients
in order to improve their public image. Government regulation of managed care reassures
consumers about accountability of managed care plans and providers. The ability of governments
to enforce their regulations is limited, however. Legislation that encourages consistency and
equity across plans promotes common standards for all. While improved population data on
quality and cost have benefited health care, some negative impacts of regulation may emerge as
well, including: (1) interference of market-regulatory activities that add administrative burdens
and costs to plans, (2) a tendency to overlook patients' interests in favor of financial interests and
political solutions, and (3) legislation of exceptions to maximum lengths of stay that could result
in decreased quality monitoring and increased infection potential. The health care system could be
improved through the efforts of the executive and legislative branches of the Federal, State, and
local governments to develop an accountable and responsible system based on national health care data.
Return to Contents
Session M: Using Evidence-based Quality Indicators
Moderator: Irma E. Arispe, Ph.D., Johns Hopkins Bayview Medical Center
Panel members: Ann G. Lawthers, Sc.D., Harvard School of Public Health; and June M. Buckle,
Sc.D., Johns Hopkins Bayview Medical Center,
Using Evidence-based Quality Indicators—Irma E. Arispe, Ph.D.
Quality indicators have been defined as "tools that estimate the extent to which a health care
provider delivers clinical services that are appropriate for each patient's condition, provides them
safely, competently, and in an appropriate time frame, and achieves desired outcomes in terms of
those aspects of patient health and satisfaction that can be affected by clinical services."
(Palmer et al., 1996). Quality indicators, or clinical performance measures, are increasingly in
demand in this era of managed care. Many are now available, although some organizations choose
to create their own. It is crucial, when choosing an already-existing method of quality
measurement, to consider the measure's reliability, validity, and appropriateness for the particular
purpose. If one is creating a new measure, one should consider development cost, decreased
generalizability of tailored measures, and the resulting need for adapting or updating such
measures for new purposes.
Cataloging and Evaluating Clinical Performance Measures: CONQUEST and the QMNet Initiative—Ann G. Lawthers, Sc.D.
Clinical performance measures have multiplied to meet the increased demand for assessing quality
of care. Potential users face a confusing array from which to choose. AHCPR recognized a need
for cataloging the myriad quality indicators and funded development of CONQUEST (Computerized Needs-Oriented Quality Measurement Evaluation System), a database of clinical
conditions and clinical performance measures used for managing care of those conditions. Partners in the
CONQUEST system development effort include the Harvard School of Public Health, the Center
for Health Policy Studies, Mikalix and Company, and the MEDSTAT Group.
The CONQUEST database provides summary information on 52 conditions addressed in AHCPR
evidence-based clinical practice guidelines and relates them to 1,185 performance measures of 53
types. The database allows users to compare measures in order to choose those most suited to
their purposes. The system is available free of charge and can be downloaded from the AHCPR
Internet site at www.ahrq.gov/qual/conquest.htm or by calling the
AHCPR Publications Clearinghouse at 1-800-358-9295. An updated version is being prepared
that features more measures, more
conditions, more detail on each measure, and improved user interface. Eventually, CONQUEST
will be privatized into QMNET, with a technical support
hotline and linking codes between
conditions and measures.
Using Evidence-based Quality Indicators—June M. Buckle, Sc.D.
Johns Hopkins Bayview Medical Center recently organized a system for outcome evaluation and
care management. The Outcomes Evaluation Team and the Care Management Team collect and
analyze data to develop and implement practice guidelines, as well as care for patients' social
work needs and manage patients' medical records. The goal is to move patients quickly, safely,
and efficiently across the continuum of care toward discharge, emphasizing close consultation
with physicians to determine when patients are ready for decreased levels of care. Case
Management Team members follow their patients across campus and act as patient advocates
with a focus on maximizing optimal outcome. Field studies are conducted, medical records are
coded, and persistent readmissions are examined closely to target areas of care in need of
improvement. An Extended Care Pathway Model was developed, with identical quality measures
employed across levels of care, to allow for comparison of outcomes by setting as well.
This system has proven to be a valuable tool for guideline development, case management,
outcomes evaluation, and risk management. It has enabled treatment teams to focus quality
improvement efforts on critical areas, demonstrate the value of new programs in improving
outcomes, respond to patients' questions, and determine the best strategies for treating specific
Return to Contents
Session N: Using Quality Indicators: Useful Tools for Investigation, Regulation, and Accreditation
Moderator: Barbara B. Fleming, M.D., Ph.D.
Panel members: Jerod M. Loeb, Ph.D., JCAHO; William E. Golden, M.D., Arkansas Foundation
for Medical Care; James G. Sheehan, J.D., U.S. Department of Justice
The Next Wave of Quality Measurement: Key Challenges—Jerod M. Loeb, Ph.D.
The Joint Commission of Accreditation of Healthcare Organizations' (JCAHO) data-driven
accreditation process, the ORYX, is part of a new initiative that integrates performance
measurements with accreditation. It contains a series of performance requirements that will
gradually be implemented for all entities accredited by JCAHO. Data collection will include
indicators of health status, satisfaction, and administrative and financial functions. Key challenges
to performance measurement are:
- People's reluctance to be measured.
- Discordant approaches to measurement:
- Selecting a measurement based on available data.
- Determining what data are needed.
- Determining whether reliable data are available.
- Benefits of data collection versus cost issues.
- Lack of data standardization (what to measure and how to measure it).
- Data integrity (variations in the quality, interpretation, and analysis of data).
- Risk adjustment (whether to adjust, how to adjust, possibility of producing a false sense of
- Lack of agreement on how data should be reported.
- Different tiers of measurement (measurement strategies used at different levels of the
health care system and authority for certain outcomes, especially physician performance and
- Value of improvements in quality of care not always easy to demonstrate.
Regional Quality Assurance Organizations: Regulator or Disseminator of Guidelines?—William E. Golden, M.D.
The Arkansas Foundation for Medical Care developed, disseminated, and implemented quality
improvement projects that translate evidence in guidelines into practice. Although Professional
Review Organizations (PROs) have oversight of Medicare delivery to ensure appropriate use of
resources, creation of the Health Care Quality Improvement Program enabled the Foundation to
take over the role of Quality Improvement Organizations (QIOs) in allowing pattern analysis,
authority for chart review, and hypothesis testing. Challenges faced by PROs in creating regional
quality improvement projects included designing an efficient project dissemination process,
motivating local project adoption, and managing multiple projects at multiple sites.
The projects are based on widely accepted clinical guidelines or on consistent clinical research.
Clinically relevant data, essential indicators, and performance measures are collected if needed.
Projects are distributed locally as needed. Technical support is provided by outreach nurses and
by a database system that monitors each hospital's project status.
Because the projects do not have a direct stake in enrollments, utilization, or profit, they can
address issues of underprovision; conflicts between beneficiaries, providers, and insurers; patient
satisfaction; and universal quality improvement. Also, because they focus on capacity building
rather than on minimum standards, they have come to be viewed as a community resource.
Quality Improvement Initiatives of the Health Care Financing Administration—Barbara B. Fleming, M.D., Ph.D.
The Health Care Financing Administration (HCFA), in a shift from the role of payer to that of purchaser, has launched several quality improvement initiatives, including (1) requiring all Medicare risk and cost contractors to report the Health Plan Employer Data and Information Set (HEDIS) measurement set on effectiveness of care and utilization, (2) developing a national performance measurement system in the fee-for-service setting to examine quality of care, (3) developing a claims-based surveillance system for the fee-for-service setting that may raise questions about quality of care for Physician Review Organizations (PROs), (4) collaborating with the National Committee for Quality Assurance (NCQA) to develop a similar surveillance system in the managed care setting, and (5) keeping abreast of the quality of care being delivered and the burden imposed on health care systems by performance measurement systems. As a first step, HCFA is funding the American Diabetes Association (ADA) to work with the NCQA to develop a consensus on common measures for diabetes.
Translating Evidence Into Practice—James G. Sheehan, J.D.
Considerable use and misuse of quality standards and performance measurements occur in the
health care system, especially in large health care organizations. This has implications for health
care fraud prosecution. Assumptions of good faith and honesty on the part of providers are
usually appropriate, but other assumptions can be made as well: (1) performance measurement
changes more than just behavior, (2) there are bad people in the world, (3) people respond to
economic incentives, (4) powerful people take advantage of the weak unless others get involved,
and (5) it is satisfying to see bad people taken away.
Working with these assumptions, there are four major issues.
- What do we care about in the area of clinical practice guidelines? The most important issue is whether people are denied medically necessary or quality care. Trust is another important issue that often determines whether the doctor's or hospital's decision process is seen as fair and objective.
- How do we find the problem areas? Problem areas are identified comparatively (e.g., examining evidence from the PROs, HCFA, public sources, and the literature) and anecdotally (e.g., using information from physicians, lawyers).
- How do we investigate or prosecute if necessary to prove a conscious decision to deny a medically necessary service? The typical guidelines case uses various techniques, such as the data system of the hospital or HMO to see comparable behavior changes and accounts of interactions between primary health care providers and the HMO or hospital system. If the decision process is fair or designed to address whether the patient needs the service under the guidelines, there will be clearly articulated standards and an independent and objective review system. There is also the issue of trust. When an adverse event results from a fair decision process, the interactions between the guidelines, the patient, the treating physician, and the reviewer need to be examined. Protection of patient confidentiality and quality of advocacy must also be examined.
- How do we change perceptions about how these things work and how do we make sure that they work properly? There are two approaches when misuse is identified: (1) injunctions against and consent decrees with organizations following inadequate approaches and (2) criminal prosecution of organizations engaging in egregious conduct and unfair patient treatment.
Return to Contents
Proceed to Next Section