In this comprehensive analysis of outcomes and effectiveness research at the Agency for Health Care Policy and Research, the Lewin Group describes accomplishments and lessons of the past decade and recommends how to increase the measurable impact of that research.
The full text of this report (AHCPR Publication No. 99-R044) is available online; you may order a print copy from the AHCPR Publications
Clearinghouse. Call toll-free 800-358-9295.
Intended for use in ongoing discussions of strategic directions for the outcomes research program of the Agency for Health Care Policy and Research (AHCPR), this report covers three
- A conceptual framework for understanding and communicating the impact of outcomes and effectiveness research
(OER) on health care practice and outcomes.
- A critical analysis of accomplishments and lessons learned over the past decade.
- A list of recommendations intended to maintain AHCPR strengths while increasing the measurable impact of future
The primary questions examined by this analysis are, "How can the outcomes and effectiveness program at AHCPR most
effectively advance the field of health services research (HSR), fulfill its unique role in HSR, contribute to public health,
and address the expectations of policymakers and stakeholders?" "The Outcome of Outcomes Research at AHCPR" is a
first step to redefining the goals of OER, as well as an honest appraisal of prior successes and opportunities for
Work on geographic variations in medical practice, appropriateness of care, and the poor quality of medical evidence set
the stage for the "Effectiveness Initiative" announced in 1988. The primary responsibility for carrying this initiative
forward found an institutional home when AHCPR was established in 1989. The effectiveness initiative itself represented
an important hypothesis: guidance for optimal medical practice could be gleaned from analysis of data routinely gathered
in the process of delivering and paying for patient care. The output from AHCPR's program for outcomes research over the past decade offers some empirical evidence with which to assess its validity.
OER evaluates the impact of health care (including discrete interventions such as particular drugs, medical devices, and
procedures as well as broader programmatic or system interventions) on the health outcomes of patients and populations.
Part of the challenge for AHCPR is the fact that the nature of OER is complex, and often apparently resistant to easy
translation for policymakers and clinicians. Despite this complexity, expectations continue to be very high that research
done by the Agency will have a clear, measurable impact on health care quality and costs.
Framework for assessing impact
A framework was developed that outlines an idealized process by which basic findings in OER are linked over time to
increasingly concrete impacts on the health of patients. The four levels of impact are:
- Findings that contribute to but do not alone reflect a direct change in policy or practice, such as new analytic methods or outcome instruments (e.g., a
quality measure such as use of beta blockers for patients after a heart attack). These may add to an area's knowledge base
and help focus subsequent research.
- Research that prompts the creation of a new policy or program (e.g., incorporation
of research findings into HEDIS or a guideline developed by a professional organization).
- A change in what clinicians or patients do.
- Actual changes in health outcomes. This framework provides a context for linking progress in basic
studies with changes in practice and improvement in outcomes.
Perspectives of principal investigators
Based on the premise that the researchers should have a clear understanding of the most important findings, a survey was
mailed to all principal investigators (PIs) funded by AHCPR's Center for Outcomes and Effectiveness Research (COER)
between 1989 and 1997, asking them to describe their most important work. Of the 95 PIs contacted, responses were
received from 61 (64 percent). Survey results suggest that PIs have been most successful in providing increasingly
accurate and detailed descriptions of what actually occurs in health care, developing tools for measuring costs of care and
patient-reported outcomes, and identifying topics for future research. Few PIs reported findings that provide definitive
information about the relative superiority of one treatment strategy over another. There are also few examples of findings
that have been incorporated into policy (level 2 impacts) or clinical decisions (level 3), or interventions that have
measurably improved quality or decreased costs of care (level 4). One of the main challenges for the next generation of
outcomes studies is to move from description and methods development to problem solving and quality improvement.
The field of OER has clearly progressed over the past decade. At least three conceptual developments have been strongly
influenced by AHCPR-sponsored work:
- The increasing recognition that evidence, rather than opinion, should guide clinical decisionmaking.
- The acceptance that a broader range of patient outcomes needs to be measured in order to understand the true
benefits and risks of health care interventions.
- The perspective that research priorities should be guided in part by public health needs.
OER studies have often provided descriptive data that challenged prevailing clinical ideas about how best to manage
specific clinical problems. Documenting patterns of care or outcomes that are inconsistent with existing understandings of
disease processes and management creates a tension that can lead to further study or directly to changes in practice. The
most concrete accomplishments of the 10 to 15 years of OER are the tools and analytic methods that have been developed.
These include strategies for conducting systematic reviews and meta-analysis, instruments for measuring health outcomes
important to patients, and sophisticated techniques for analyzing observational data to adjust for disease severity and
minimize bias. Some findings from OER may not be definitive enough to influence practice, but may help direct future
research efforts. Public funding for OER has produced a network of institutions and trained investigators capable of
carrying out OER. Beginning with a relatively small core of academic centers and professionals, there are now large
numbers of individuals and institutions pursuing outcomes studies in the public and private sectors.
Selected lessons learned
The past decade has also been a learning experience for the field of OER. In its simplest form, the framers of the
effectiveness initiative realized that existing data and studies might represent an inexpensive source of knowledge about
effective care. The debate over the pros and cons of randomized controlled trials (RCTs) and observational studies partially
obscures a basic observation that is less controversial. Different research designs are associated with different susceptibility to systematic bias. The two critical questions to ask when considering the adequacy of a particular study design then
are: "How likely is it that bias is affecting the results, and how certain of the results is it necessary to be in order to change
policy or practice?"
It will be valuable to do a more systematic exploration of how to associate the features of a particular clinical problem with
the most appropriate tools and methods to study that problem (given that the goal is to promote decisions that will improve
outcomes of care).
It is increasingly clear that limitations of observational designs prevent these studies from providing definitive answers to
many questions of comparative clinical effectiveness. More experience has also confirmed that changing practice and
realizing savings in health care are not easily achieved. Research and experience have demonstrated that development and
dissemination of even high quality, highly credible information is often insufficient to alter practices. Enhanced
knowledge must be linked with supportive practice environments and incentives for change. And while it had been hoped
that reduction of inappropriate, high-cost care would represent savings of hundreds of billions of dollars through outcomes
studies, actual success has been more modest.
AHCPR can take steps to maintain its strength in methods and tools development while increasing support for studies with
greater potential for impact. The Agency could also play a more active role in the transfer of knowledge and documenting
change when it occurs. Many studies supported by COER in the past decade have served as hypothesis-generating studies,
but there has not been a systematic effort directed to orchestrate the hypothesis-testing follow-on. COER could take on
greater responsibility to make sure that once these critical knowledge gaps are identified, they are addressed in follow-on
studies. While unchanneled intellectual exploration is a time-tested and vital approach to research, it may not be entirely
satisfactory for AHCPR, given the expectations for accelerating the pace of translating research into improved patient
COER should also leverage resources by seeking new partnerships in addition to maintaining collaborative efforts with the
Health Care Financing Administration (HCFA) and other payers, managed care organizations, medical groups, medical
professional organizations, peer review organizations, and medical product manufacturers. Collaboration with these
organizations would ensure that potential studies are crafted to meet the applied needs of these organizations. This next
stage of OER would increase attention to creating designs and blueprints for improving care and selectively applying the
OER tools when they are needed. It would represent a move from a "tool-using" culture to a "problem solving" culture,
for which the primary objective is developing an empirical science of clinical improvement. The COER could be
refashioned into the "CHOIR" (the Center for Health Outcomes Improvement Research).
There is a need for more attention to developing innovative methods and strategies for efficiently addressing the large
number of unanswered questions about effectiveness and cost-effectiveness that incorporate relevant environmental
characteristics. There has been no concerted effort to craft a new methodological, organizational, and ethical framework
for these studies. The "conceptual infrastructure" for conducting effectiveness trials needs further development. COER
could engage in a major initiative to create and refine new approaches to conducting prospective effectiveness trials,
similar in visibility and scope to the "effectiveness initiative" of the late 1980s. The creative energy and ingenuity of
health services research would be directed to crafting new approaches to studying effectiveness across the spectrum of
health care settings using prospective, experimental methods. As with AHCPR's Patient Outcomes Research Teams
(PORTs), the new effectiveness initiative would support methods development, enhance specific technical skill in the
research community, promote conceptual discussion of effectiveness trials, and create networks of researchers closely
linked with institutions and practitioners. Prospective effectiveness studies can best be done as partnerships between
researchers, providers of clinical care, delivery systems, and funding organizations. The operational and organizational
challenges to this are quite significant.
Improvements in outcomes measures and development of strategies to encourage their routine use are an essential future
direction for COER. Additional work on the incorporation into practice, feasibility and credibility of disease-specific and
general outcomes measures, and development of tools and strategies to make this possible, are critical requisites to getting
to outcomes that people experience and care about.
The Agency should consider developing the capacity to identify important research findings (generally level 1 impacts),
and to move findings to actions. This capacity should be proactive, intensive, coordinated, and consistent, and not left
solely to the ingenuity of each research team. A dedicated knowledge transfer function would systematically consider
findings from OER studies and carefully assess them to determine how the findings can have more impact. An internal or
external center would develop a strategy for collecting and documenting examples of OER impact stories. In order for
COER to be seen as accomplishing its goals, those goals must be agreed upon and clearly stated. It is also important to
clearly convey the complementary nature of OER studies with traditional basic science and clinical research. Policymakers
need to have a firm understanding that OER provides the link between basic research and clinical practice.
A high level of interaction with stakeholders in the health care system will ensure that the basic studies reflect real
problems faced by those involved in health care delivery. Ideally, managed care organization (MCO) managers and medical directors, hospital
administrators, insurance company business managers, medical product industry marketing directors, pharmaceutical
product managers, private sector outcomes researchers, and others could directly participate in a range of COER activities.
The effectiveness initiative, launched a decade ago, has yielded both important findings and opportunities for
improvement. A new emphasis on evidence-based decisionmaking and interest in assessing and improving quality of care,
along with interest in OER among a broad array of private sector organizations, set the stage for this analysis. The attempt
to systematically review research output from an organization has few clear precedents, and necessarily reflects the limited
perspective of the reviewers themselves, as well as the difficulty of determining the value of generated knowledge. A key
conclusion from this analysis is that the accomplishments of the first decade provide a foundation to build on.
However, it is clear that interests and incentives of the research enterprise and the "change agents" within the health care
system must be more effectively aligned to achieve the promise envisioned by the establishment of AHCPR.
Current as of September 1999
The Outcome of Outcomes Research at AHCPR:Final Report. Summary. Agency for Health Care Policy and Research, Rockville, MD. http://www.ahrq.gov/clinic/outcosum.htm