A Summary of the First Annual Workshop of AHRQ’s Comparative Health System Performance Initiative

The Agency for Healthcare Research and Quality and the Coordinating Center of the Comparative Health System Performance Initiative held a workshop September 29, 2016, to share learning and advance the initiative’s research agenda. This document summarizes the workshop content and identifies next steps.

Summary
Introduction
Goals of AHRQ’s Comparative Health System Performance Initiative
Creation of a Compendium on Health System Performance
Health System Definitions
Use of Secondary Data To Identify Health Systems
Beyond Definitions: Identifying Health System Attributes
Next Steps and Ways To Get Involved
References

Summary

On September 29, 2016, the Agency for Healthcare Research and Quality (AHRQ) and the Coordinating Center hosted the first annual workshop of the Comparative Health System Performance (CHSP) Initiative. They convened this meeting to promote shared learning and advance the initiative’s research agenda. Key staff from AHRQ, the three Centers of Excellence (CoEs), and the Coordinating Center attended the 1-day, in-person workshop.

A primary area of focus for the first year of the CHSP Initiative was to define the unit of analysis—health systems—and to identify systems by using available data sources. Therefore, much of the agenda was dedicated to reviewing various health system definitions and approaches to identifying health systems. This Issue Brief summarizes the key topics discussed during the workshop.

Based on the workshop discussions, a number of important takeaways will inform future work of the initiative:

• Defining health systems is a foundational—and consequential—step in the CHSP Initiative. The lack of consistency in the terminology used to identify health systems and their component parts and the variation in available datasets to identify systems create challenges to studying comparative health system performance. AHRQ, the CoEs, and the Coordinating Center have the opportunity and challenge of developing operational health system definitions and identifying health systems to study. The CoEs intend to examine a range of delivery system organizations and health systems, including corporate entities that own or manage subsystems, provider organizations, and informal care systems.
• Engaging health system leaders and other stakeholders for the initiative early is important. Early engagement of key stakeholders will help to ensure the compendium is valuable to end users and that the initiative identifies ways health systems can advance the use of patient-centered outcomes research (PCOR).
• Multiple data sources are needed to mitigate the risk of errors from relying on any one data source. A number of challenges arise when using existing data to identify health systems. For example, available datasets vary in terms of the precise meaning of ownership or their methods for verifying provider affiliations. Therefore, multiple data sources, including primary data collection, are needed to verify provider attribution to health systems.
• The rapidly changing nature of the current health system environment requires a mechanism for externally validating and updating health system lists over time. Future efforts to build a compendium of resources to study health systems must consider ways to ensure the accuracy of the data that identify health systems over time.
• Linkage variables are needed to integrate data that identify health systems with performance data. A primary consideration for studying health system performance is how to link the datasets with other available data that can provide information on performance, such as clinical data and claims data. The CoEs agreed that tax identification numbers, or TINs, would be the main linking variable to other datasets to measure health system attributes and performance.
• The concept of "comprehensive care" should be a key defining characteristic of health systems studied under the initiative. All agreed that systems identified for the purposes of the CHSP Initiative should provide comprehensive services, including primary care. However, how to define the concept of comprehensive care with greater specificity required further consideration.
• Different patients will experience health systems differently, and performance may vary depending on the type of patient. The CoEs will examine patient centeredness as a health system attribute when examining the use of evidence among health systems and health system performance.

Top of Page

Introduction

To support the effective dissemination and use of patient-centered outcomes research (PCOR) among health care systems, the Agency for Healthcare Research and Quality (AHRQ) created the Comparative Health System Performance (CHSP) Initiative. Starting in 2015, AHRQ established three Centers of Excellence (CoEs) at the Dartmouth Institute, the National Bureau of Economic Research (NBER), and RAND Corporation and a Coordinating Center at Mathematica Policy Research to collectively identify, classify, track, and compare health systems over the 5-year initiative.

Top of Page

Goals of AHRQ’s Comparative Health System Performance Initiative

The health care delivery system in the United States is highly complex, involving multiple delivery system organizations, payers, and payment models. Currently, a variety of initiatives are underway to advance value-based payment, promote patient-centered care, and improve population health. These initiatives are, in part, spurred by gaps in the use of evidence and provision of high quality, patient-centered care. For example, many high-value services are underused, while some low-value services are persistently overused (McGlynn, et al., 2003; Rosenberg, et al., 2015).

Gaps in the use of evidence provide significant opportunities to address barriers to the use of evidence at the point of decisionmaking. A growing body of literature has identified the organizational and environmental factors that influence clinical decisionmaking and the use of evidence (Reschovsky, et al., 2015). However, more information is needed on how evidence is used and disseminated among health care systems and the health system attributes and external factors, such as payment policies, that influence the use of evidence within systems (Casalino, 2014). To realize the benefits of significant Federal investments in PCOR, additional research regarding the organizational supports and policy levers that promote the effective use of evidence among health care systems is warranted.   

AHRQ created the CHSP Initiative to:

• Develop a robust understanding of health system characteristics and the factors that influence health system performance.
• Study how health systems use PCOR and other forms of evidence.
• Create dissemination strategies that increase the accessibility and usefulness of evidence for health systems.
• Construct and disseminate a compendium of databases, research tools, and resources to advance the field’s ability to study health systems.

The three CoEs are each constructing a data core composed of multiple primary and secondary data sources to systematically identify and track health systems, their characteristics, and their performance on quality, cost, and patient outcomes. The CoEs are also conducting a series of studies to assess the use of PCOR among health systems and identify the characteristics of high-performing health systems, the influence of external factors on high performance, and the role of PCOR evidence in these systems.

The Coordinating Center facilitates collaboration between the CoEs, convenes stakeholders and technical experts, and synthesizes findings from the CoEs. The Coordinating Center is developing a Compendium on Health System Performance and is promoting dissemination of the initiative’s findings.

The compendium will:

• Contain data resources, such as research-ready data files and data tables, identifying and describing systems.
• Serve as a resource that will be updated over time to systematically track changes in health systems and their attributes.
• Serve as a resource for the research community and health system leaders seeking opportunities to improve performance.

The September workshop included a panel discussion in which speakers reflected on their goals and desired outcomes for their research under the initiative. The panelists were Elliot Fisher, M.D., Principal Investigator for the Dartmouth CoE; David Cutler, Ph.D., and Nancy Beaulieu, Ph.D., Principal Investigator and Co-Investigator, respectively, for the NBER CoE; and Cheryl Damberg, Ph.D., Principal Investigator for the RAND CoE.

Beyond their shared goals of advancing evidence on the use of PCOR among systems and identifying characteristics of high-performing systems, the CoEs highlighted several other key goals for their research, including:

• Capturing health system characteristics that facilitate high performance.
• Distinguishing organizational characteristics from market characteristics.
• Examining whether safety net providers and systems can be studied in similar ways as other systems or warrant unique methods.
• Understanding the role of payment models in health system performance.
• Producing comparative health system performance data for the public good.
• Identifying health system performance from the patient perspective, while considering a range of patient populations.
• Understanding mechanisms health care systems use to adopt and implement evidence and improve performance.
• Developing a common language and definitions to inform health system research broadly.

These goals reflect and expand on AHRQ’s goals for the CHSP Initiative and echo AHRQ’s broader mission to support certain priority patient populations, including low-income populations, patients with special needs, and patients with chronic conditions.

Top of Page

Creation of a Compendium on Health System Performance

Andrew Bindman, M.D., AHRQ director at that time, presented AHRQ’s near- and long-term visions for the compendium, which will serve as a Web-based informational and data resource. Dr. Bindman highlighted the need for engagement of the research community and end users, such as health systems, in developing the compendium to ensure its usefulness. The group agreed with the need to engage stakeholders in compendium development and to clearly explain the purposes of its data to ensure appropriate use.

Top of Page

Health System Definitions

As a key starting place for their research, the CoEs developed and began to implement various definitions of health systems. These definitions encompass a range of delivery system organizations and point to the unique aspects of each of the CoE’s research. During the workshop, the CoEs presented their definitions and discussed how these definitions fit into their larger bodies of work.

A primary focus of the Dartmouth CoE’s research is to explore cost and quality performance for primary care populations. Therefore, the CoE is defining a health system as organizations with shared ownership that include: (a) at least one hospital and a group of physicians (3+ primary care physicians [PCPs]) or (b) at least one group of physicians (3+ PCPs). The CoE is also interested in independent hospitals and physician practices.

The NBER CoE is interested in capturing a broad range of systems, with a unique interest in identifying systems from a variety of patient perspectives, such as pediatric and cancer patients. As such, the CoE is identifying health systems that include two or more organizations with common ownership or that are contractually integrated, such as accountable care organizations (ACOs), as well as informal care systems, based on, for example, referral patterns.

The RAND CoE is studying health systems in four regions for which the CoE has performance data at the physician organization level. Starting with physician organizations, the RAND CoE is linking these organizations to their affiliated physicians and hospitals to construct health systems. The CoE has operationally defined a health systems as: two or more health care organizations (including at one least one acute-care hospital and one physician organization) affiliated with each other through shared ownership or a contracting relationship for payment and service delivery. The CoE does not consider single-specialty organizations to be systems.  

Looking across these definitions, there is shared interest in health systems characterized by shared ownership. RAND explicitly requires the inclusion of a hospital to be considered a system, while the others do not. All are also interested in contractual relationships such as ACOs.

Top of Page

Use of Secondary Data To Identify Health Systems

All of the CoEs intend to use multiple data sources to identify health systems, and they are examining health care systems from multiple perspectives. For example, the NBER CoE described using top-down and bottom-up approaches to identify health care systems. The top-down approach is based on datasets such as the American Hospital Association’s annual survey of hospitals and SK&A’s databases that identify corporate entities. The bottom-up approach is based on physician-level datasets such as Medicare Data on Provider Practice and Specialty (MD-PPAS) that are intended to identify physician affiliations.

RAND is similarly starting from the physician organization level to identify hospital affiliations and construct systems. The Dartmouth CoE also intends to study individual practices and physician organizations, using data sources such as the National Survey of Physician Organizations.

The teams discussed their efforts to build their data cores and identify health systems in existing secondary data sources. The group identified a number of challenges in doing so. For example, the teams discussed the potential imprecision of health system definitions used in the data sources. They also identified challenges in attributing physicians to systems due to potential data omissions or errors.

The group also discussed the evolving nature of the U.S. health care system overall, which could affect how they identify and study health systems. For example, the national trend toward contractual integration for the purposes of payment and service delivery, rather than integration via shared ownership, is changing how providers are organized in ways that could be consequential to the CHSP Initiative.

Given that the purpose of identifying health systems in secondary data sources is to ultimately measure their performance and adoption of PCOR, the teams discussed the need to link various datasets that identify health systems to clinical and claims data. All the participants agreed that tax identification numbers, or TINs, would be the primary linking variable among datasets.

Top of Page

Beyond Definitions: Identifying Health System Attributes

The CoEs have begun to identify a range of health system attributes to study, including characteristics of organizations and the broader environment within which organizations operate. However, the CoEs expressed openness to further collaboration and reflection on the types of attributes that should be explored in the coming years. For example:

• The RAND CoE intends to conduct qualitative “deep dives” in four regions to further explore health system attributes that influence health system performance. During the workshop, RAND elicited input from the other CoEs about health system attributes to explore through these deep dives.
• The Dartmouth CoE intends to field its National Survey of Healthcare Organizations and Systems to collect comprehensive information about organizational attributes and internal mechanisms that support the use of evidence in practice. Organizational attributes include integration, governance, and leadership, and internal mechanisms include health information technology capabilities and performance management.

While the work of identifying health system attributes is still underway, the group explored the concepts of comprehensive care, integration, and "system-ness," or the degree to which health systems and their component parts function in a coordinated manner. The group agreed that comprehensive systems include primary care clinicians; however, further discussion was needed regarding how to define comprehensive care.

The group recognized the critical role of health system integration and its many facets, including clinical and functional integration. The group determined that future discussions should focus on the concepts of integration and system-ness. The group also agreed that patient centeredness and the patient experience were key attributes of systems that will be included in the CoEs’ research.  

The group also identified the critical need to understand the geographic location of health systems and to control for potential variation in the characteristics of the local patient population. This variation can lead to spurious conclusions about comparative health system performance if not adequately controlled for in statistical analyses. On a related issue, the group also deferred for future consideration the need to determine how to handle health systems that span geographic regions and markets, because these systems experience heterogeneous environmental factors.  

Top of Page

Next Steps and Ways To Get Involved

AHRQ, the CoEs, and the Coordinating Center will continue to develop plans for the Compendium on Health System Performance. The CoEs are into their second year of the initiative, moving forward with constructing their data cores and conducting their studies. A future public webinar about the initiative is planned for 2017 that will highlight key findings of the initiative.

For more information or questions about the Comparative Health System Performance Initiative, go to https://www.ahrq.gov/chsp or contact us at Doreen.Bonnett@ahrq.hhs.gov.

Top of Page

References

Casalino LP. Identifying key areas for delivery system research. Rockville, MD: Agency for Healthcare Research and Quality; 2014. AHRQ Publication No. 14-0024-EF. https://www.ahrq.gov/sites/default/files/wysiwyg/professionals/systems/system/delivery-system-initiative/casalino/paper/casalino_idkeydsr.pdf (556 KB). Accessed January 11, 2017.

McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. New Engl J Med 2003;348():2635-45. http://www.nejm.org/doi/full/10.1056/NEJMsa022615. Accessed January 11, 2017.

Reschovsky JD, Rich EC, Lake TK. Factors contributing to variations in physicians’ use of evidence at the point of care—a conceptual model. J Gen Intern Med 2015;30 Suppl 3:S555-61. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4512965/. Accessed January 11, 2017.

Rosenberg A, Agiro A, Gottlieb M, et al. Early trends among seven recommendations from the Choosing Wisely Campaign. JAMA Intern Med 2013;175(12):1913-20. https://dx.doi.org/10.1001/jamainternmed.2015.5441. Accessed January 11, 2017.

Top of Page