Testimony on Comparative Effectiveness Research

Sean M. O'Neill, M.Phil.; Alec R. Levenson, Ph.D.; Jennifer L. Malin, M.D., Ph.D.

On April 3, 2009, public testimony on comparative effectiveness research was given at a meeting of the National Advisory Council for Healthcare Research and Quality. The testimony represents the views of the presenter and not necessarily those of the Agency for Healthcare Research and Quality (AHRQ) or the Department of Health and Human Services (HHS).

Delivered Via Electronic Mail

Please accept the following comments for consideration regarding the priorities for a national health services research agenda and AHRQ's portion of funds provided under the American Recovery and Reinvestment Act (ARRA) for comparative effectiveness research:

We strongly believe that a portion of ARRA funds should be used to fund research into organizational effectiveness and organizational behavior in health care delivery. ARRA provides for "the development and dissemination of research assessing the comparative effectiveness of health care treatments and strategies" (1). Understanding the effectiveness and behavior of health care delivery organizations certainly falls under this purview. Obtaining such an understanding is paramount for improving the actual delivery of care to Americans for three reasons.

First, current effectiveness studies comparing different healthcare delivery strategies do not adequately take into consideration the highly complex, multi-level nature of health care organizations (2, 3). Even with state-of-the-art research designs such as cluster-randomization (4-6), a deeper understanding of the organizational determinants (structure, culture, politics, resources) of safe and effective health care is lacking and necessary for understanding how to improve care (7).

Second, the clinical registries and other data-collection facilities funded by ARRA are sorely needed, but they alone are insufficient for improving health care (8). Understanding how health care organizations can most effectively utilize these tools to improve actual care is of the utmost importance if our substantial health IT investment is ultimately to be of any value to American patients. Data monitoring and feedback alone are insufficient for improving the quality of care because much of healthcare is delivered via complex organizational structures. Understanding how the varied medical professionals spanning a myriad of disciplines come together within complex organizational structures is essential to identifying critical factors that can impact patient care. A fundamental change in the way data monitoring and feedback are used and characterized within the culture of an organization is essential for producing improved results (9). We thus need to better understand the organizational barriers and facilitators for effective use of clinical data in real time if we are to fully capitalize on the opportunities ARRA affords. Organizational behavior research approaches are critical for obtaining this understanding.

Finally, although we are able to compare healthcare organizations on various sets of performance measures (the Premier Hospital Quality Incentive Demonstration (10) and HospitalCompare (11) are two hospital-based examples) and stratify them using those measures, we have virtually no systematic understanding of why the highest-decile groups end up as the highest-performers. Moreover, we have even less understanding of how to translate such stellar performance to the rest of the pack. Research has demonstrated that improved quality typically comes through an evolution of organizational culture that stresses a focus on quality improvement above other competing priorities (7, 12). If we want to make health care more effective in the United States, we need to better understand this evolution and the specific processes by which health care organizations can become high performers.

Organizational behavior and organizational effectiveness research approaches have been effectively applied in numerous industries including health care, with the research literature dating back over decades (13-22). The result has been deep insights into the factors that drive behavior in complex organizations, including behaviors that produce outcomes at odds with organizational and societal objectives. It is straightforward to apply these research approaches to study and understand processes that are overlooked by more traditional health services research programs. Yet funding for these types of projects is not currently prioritized by AHRQ. Given the deep influence organizations have on actual care delivery and the vast opportunity ARRA affords to expand and deepen our understanding of the relative merits of different health care delivery strategies, we strongly urge AHRQ to make studies of health care organizational effectiveness and behavior a priority in its use of ARRA funding.

Sean M. O'Neill, M.Phil
Doctoral Fellow, Pardee RAND Graduate School
Medical Student, Northwestern University, Feinberg School of Medicine

Alec R. Levenson, Ph.D.
Research Scientist, Center for Effective Organizations
University of Southern California, Marshall School of Business

Jennifer L. Malin, M.D., Ph.D
Associate Professor, David Geffen School of Medicine at UCLA
Attending Physician, VA Greater Los Angeles Healthcare System
Policy Analyst, RAND Corporation

  1. "Text of the Recovery Act Related to Comparative Effectiveness Funding: American Recovery and Reinvestment Act of 2009."
  2. Berwick DM. The science of improvement. JAMA 2008;299:1182-1184.
  3. Ferlie EB, Shortell SM. Improving the Quality of Health Care in the United Kingdom and United States: A Framework for Change. Milbank Q 2001;79:281-315.
  4. Althabe F, Buekens P, Bergel E, et al. A behavioral intervention to improve obstetrical care. NEJM 2008;358:1929-40.
  5. Kritchevsky SB, Braun BI, Bush AJ, et al. The effect of a quality improvement collaborative to improve antimicrobial prophylaxis in surgical patients. Ann Int Med 2008; 149:472-480.
  6. Hillman K, Chen J, Cretikos M, et al; MERIT Study Investigators. Introduction of the medical emergency team (MET) system: a cluster randomized controlled trial. Lancet 2005;365:2091-2097.
  7. Bate, Paul, Mendel, Peter, and Robert, Glenn. Organizing for Quality: The improvement journeys of leading hospitals in Europe and the United States. 2008. Radcliffe Publishing, Oxford.
  8. Ludwick DA, Doucette J. Adopting electronic medical records in primary care: lessons learned from health information systems implementation experience in seven countries. Int J Med Inform 2009;78:22-31.
  9. Bradley EH, Holmboe ES, Mattera JA et al. Data feedback efforts in quality improvement: lessons learned from US hospitals. Qual Saf Health Care 2004;13:26-31.
  10. Centers for Medicare and Medicaid Services. "Premier Hospital Quality Incentive Demonstration" Available at http://www.cms.hhs.gov/HospitalQualityInits/35_hospitalpremier.asp.
  11. U.S. Department of Health and Human Services. "Hospital Compare-A Quality Tool Provided by Medicare". Available at http://www.hospitalcompare.hhs.gov.
  12. Shortell SM, Schmittdiel J, Wang MC, et al. An empirical assessment of high-performing medical groups: results from a national study. Medical Care Research and Review 2005;62:407-434.
  13. Appelbaum E, Batt R. The New American Workplace: Transforming Work Systems in the U.S. 1994. Cornell ILR Press, Ithaca, NY.
  14. Campion MA, Medsker GJ, Higgs AC. Relations between work group characteristics and effectiveness: Implications for designing effective work groups. Personnel Psychology 1993; 46:823-850.
  15. Cohen SG, Bailey D. What makes teams work: Group effectiveness research from the shop floor to the executive suite. Journal of Management 1997; 23:239-290.
  16. Gittell JH, Weinberg DB, Pfefferle S, Bishop C. Impact of relational coordination on job satisfaction and quality outcomes: A study of nursing homes. Human Resource Management Journal 2008; 18:154-170.
  17. Goes JB, Park SH. Interorganizational links and innovation: The case of hospital services. Academy of Management Journal 1997; 40:673-696.
  18. Hackman R, Oldham G. Work Redesign. 1980. Addison-Wesley, Reading, MA.
  19. Janz BD, Colquitt JA, Noe RA. Knowledge worker team effectiveness: The role of autonomy, interdependence, team development, and contextual support variables. Personnel Psychology 1997; 50:877-904.
  20. Lawrence PR, Lorsch JW. Organization and Environment. 1967. Harvard Business School Press, Boston, MA.
  21. Mohrman SA, Cohen SG, Mohrman AM. Designing Team-Based Organizations: New Forms for Knowledge Work. 1995. Jossey-Bass, San Francisco, CA.
  22. Young GJ, Charns MP, Shortell SM. Top manager and network effects on the adoption of innovative management practices: A study of TQM in a public hospital system. Strategic Management Journal 2001; 22:935-951.
Current as of April 2009
Internet Citation: Testimony on Comparative Effectiveness Research: Sean M. O'Neill, M.Phil.; Alec R. Levenson, Ph.D.; Jennifer L. Malin, M.D., Ph.D.. April 2009. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/news/newsroom/speech/oneill.html