Lessons From the Field
Four studies were commissioned by the Agency for Healthcare Research and Quality (AHRQ) to detail purchasers' efforts to improve quality. The purpose is to illustrate the breadth of quality-based purchasing initiatives underway and to explore lessons learned. The studies cover public report cards sponsored by the Ohio Department of Aging and the Colorado Business Group on Health and pay-for-performance schemes sponsored by the Maine Health Management Coalition and the Hudson Health
By Meredith B. Rosenthal, Ph.D., and Joe Camillus, M.B.A., M.P.H.
Lessons From the Field
Ohio Department of Aging
Colorado Business Group on Health
Maine Health Management Coalition
Hudson Health Plan
Foreword: Four Flowers Blooming
The Agency for Healthcare Research and Quality (AHRQ) commissioned four studies to detail efforts by purchasers—defined broadly to include employers, government agencies, health plans, and consumers—to improve quality in their respective communities, both to illustrate the breadth of initiatives underway and to explore lessons learned.
Our goal was to document report card initiatives, which explicitly compare providers within a specified geographic region on a routine basis according to certain standards of performance, and
pay-for-performance schemes, which financially reward and penalize providers based on the level of their performance. The resulting set of studies features public report cards sponsored by the Ohio Department of Aging and the Colorado Business Group on Health, and pay-for-performance schemes sponsored by the Maine Health Management Coalition and the Hudson Health Plan.
Contrary to what might be hypothesized, implementation is not limited to the private sector. In fact, public purchasers were involved in each of the four efforts studied, even that led by a business coalition. Strategies focus on a range of providers, including nursing homes, primary care practices, hospitals, and community health centers. The featured purchasing pioneers are using a rich mix of structural, process, and outcome standards to benchmark quality. Measures include, for example, use of electronic medical records and e-prescribing, preventive care utilization, mortality rates for various procedures, family satisfaction with care, and patient satisfaction.
Unfortunately we know little about the impact of these quality-based purchasing strategies. One of the four efforts surveyed physicians after implementing pay-for-performance, but none evaluated the impact on quality of care. Readers who opt to pursue a report card or pay-for-performance scheme are encouraged to formally evaluate their experience so that it can be used to inform best practices.
AHRQ expresses appreciation to Meredith Rosenthal, Ph.D., Joe Camillus, M.B.A., M.P.H., and the four vanguard purchasers who generously gave of their time to share their experiences, which made the development of this set of studies possible.
AHRQ is an agency within the U.S. Department of Health and Human Services charged with improving the quality, safety, efficiency, and effectiveness of health care for all Americans. These studies are the latest in a series of coordinated efforts by AHRQ to contribute to the ongoing national dialog related to how purchasers—a key stakeholder group—can work to improve quality of care.
For a list of AHRQ resources specific to pay-for-performance, go to: http://www.ahrq.gov/qual/pay4per.htm.
AHRQ also supports resources on public report cards, including the Web-based Report Card Compendium, which is available on AHRQ's TalkingQuality.gov site, https://talkingquality.ahrq.gov. The Compendium provides in a single location a searchable database of over 200 report examples, which allows purchasers and others to explore and assess different approaches to formatting information, displaying data, and explaining why quality information is important to consumers. In 2007, AHRQ will make available an evidence-based report card template tailored specifically for use with the AHRQ Quality Indicators; watch AHRQ's Web site for details.
As governments, employers, health plans, and other buyers of health services consider or reconsider their quality agendas, you are encouraged to explore and debate sponsorship of provider report cards and quality-based payment within the context of an overarching national or local quality framework alongside more traditional methods, e.g., provider training and continuous quality improvement
programs. We hope these studies inform your deliberations, and we welcome your feedback.
Carolyn M. Clancy, M.D.
Director, Agency for Healthcare Research and Quality
Return to Contents
In spring 2006, researchers examined the quality-based purchasing activities of four purchasers—a State agency, two business coalitions, and a Medicaid prepaid health plan. These "lessons from the field" were prepared so that public- and private-sector purchasers could review and learn from others who have designed and implemented recent quality-based purchasing activities. For this report, quality-based purchasing is defined as payment or reporting incentives targeted at providers that public and private purchasers could adopt to encourage improvements in quality of care.1
Sites were selected in collaboration with researchers in the field of quality-based purchasing and business coalition industry leaders.
- The Ohio Department of Aging serves and represents more than two million Ohioans age 60 and older. The Department's role is to advocate for the needs of all older citizens in the state.
- The Colorado Business Group on Health (CBGH) is a business coalition composed of 22 public and private purchasers, representing 200,000 covered lives in Colorado.
- The Maine Health Management Coalition (MHMC) is a multi-stakeholder coalition that includes both purchasers and providers of care. The Coalition represents four large public purchasers and private employers that represent approximately 150,000 covered lives across Maine.
- The Hudson Health Plan (HHP) is a prepaid health services plan that serves over 55,000 enrollees in the New York Medicaid program and State Children's Health Insurance Program (SCHIP).
In the field of quality-based purchasing, the decision-making of purchasers with regard to pay-for-performance and public reporting initiatives has not been well documented. The experiences of the organizations profiled in this report offer lessons on the design and implementation of pay-for-performance and public reporting initiatives. The term pay-for-performance refers specifically to incentives that reward providers for achieving objectives established by a purchaser; these objectives may include improvements in efficiency, data submission, quality improvement, and/or patient safety.
This summary offers findings and key lessons from several recent quality-based purchasing activities that can help inform purchasers who are considering similar undertakings. The accompanying description of purchaser activities provides indepth information on the design, implementation, and lessons learned for each of the initiatives.
Return to Contents
Engage stakeholders early and often.
The Colorado Business Group on Health's hospital reporting project provides reliable, comparable data on the quality of care provided in hospitals throughout Colorado. The Colorado Business Group on Health found that energizing the business community at the outset was key to the success of its hospital reporting project. The Business Group's Executive Director also worked with several hospital leaders who were willing to pitch the initiative to the State's hospital association.
The Ohio Department of Aging's Long-term Care Consumer Guide Web site at http://www.ltcohio.org/consumer/index.asp allows consumers to search for a nursing home or assisted living facility using geographic and specialty criteria. Consumers can then compare performance based on quality data collected by the Centers for Medicare & Medicaid Services (CMS) and satisfaction surveys conducted by the State. The Ohio Department of Aging established a working group with providers to help foster a sense of trust and reach consensus on difficult decisions. A good working relationship between government and providers contributed to the success of the consumer reporting initiative.
The Maine Health Management Coalition's Pathways to Excellence program includes two separate initiatives that offer financial rewards to primary care physicians and hospitals. These pay-for-performance initiatives include the same quality measures used for the Coalition's public reporting efforts. Consumers who visit the Coalition's Web site (http://www.mhmc.info/) can view the performance of primary care physicians and hospitals in a number of key areas. The Coalition involved physicians up front by creating a steering committee of 14 individual physicians, who helped develop the standards for quality measurement and reporting. The Maine Health Management Coalition engaged the State hospital association at the outset of the hospital reporting initiative; the hospital association then secured agreement from its 34 members in order to move forward.
The Hudson Health Plan's Rewarding Excellence program offers bonus payments to primary care practices and community health centers for every patient who receives high-quality care for several specific conditions. The Plan also has a broader bonus program for overall performance on a set of quality measures that it reports to the New York State Department of Health. The Hudson Health Plan found that once its initiative was underway, frequent communication with providers helped maintain support for the program. The Plan offers providers regular mailings, reports, and visits from provider relations representatives who can help troubleshoot poor performance and assist with projected improvements.
Get help from consultants and experts when possible to supplement small staff and introduce needed expertise.
In developing its consumer Web site, the Maine Health Management Coalition used a national expert in health literacy to balance the medical community's desire to publish detailed, scientific measures with consumers' desire to see a very simple, basic hospital rating system. The Coalition has a small staff, so finding expertise to help implement quality-based purchasing initiatives has been a plus.
Allow ample time to design the initiative.
Several sites reported that it took 2 or more years to launch their programs. This timeframe was important in terms of securing provider and stakeholder interest and support, selecting performance measures, designing financial incentives, and developing a Web site for those organizations that reported quality measures online. Ohio's Department of Aging took 1.5 years to launch its Consumer Guide Web site. Colorado Business Group on Health's hospital reporting project took 2.5 years from start to finish. Maine Health Management Coalition required several years to design and implement its Pathways to Excellence program, which offers financial rewards to primary care physicians and hospitals.
Rely on well-established quality indicators to ensure provider "buy-in."
For their hospital quality reporting initiative, the Colorado Business Group on Health and the Colorado Hospital Association used evidence-based quality indicators developed and maintained by AHRQ. Because the indicators were developed by a government agency, they reside in the public domain and include features such as risk adjustment, where appropriate, and standardized technical specifications. Hudson Health Plan started its pay-for-performance program by rewarding providers based on their overall performance on quality measures they were well familiar with, New York State's Quality Assurance Reporting Requirements. Today, the Plan's providers are also eligible for bonuses related to Pap smears, pediatric immunizations, prenatal care, Supplemental Security Income (SSI) needs assessment, and a comprehensive diabetes care program.
ROI analysis is difficult to conduct.
None of the four sites conducted a return-on-investment (ROI) analysis of their programs. In deciding not to calculate an ROI, the Maine Health Management Coalition cited several factors, including the difficulty of isolating the effects of its pay-for-performance initiatives, small sample sizes, and the short history of the programs. While the Colorado Business Group on Health did not pursue an ROI analysis, the Executive Director believes that its public reporting has brought about positive change in provider behavior.
For one of four sites, public reporting is an important complement to pay for performance.
In 2004, the Maine Health Management Coalition implemented a financial rewards program for all primary care practices in the State. More recently, the organization implemented a pay-for-performance pilot that includes 10 hospitals. Public reporting is an important component of both these initiatives. Consumers who visit the Coalition's Web site (http://www.mhmc.info/) can view the performance of primary care physicians and hospitals in a number of key areas.
Return to Contents
Lessons From the Field
Ohio Department of Aging
The Consumer Guide Initiative was developed by the Ohio Department of Aging, which serves and represents more than two million Ohioans age 60 and older. The Department's role is to advocate for the needs of all older citizens.
The Consumer Guide Initiative resulted in the Long-term Care Consumer Guide Web site, accessible to the public at www.ltcohio.org/consumer/index.asp. At this site, consumers can use geographic and specialty criteria to search for a nursing home or assisted living facility, and then compare performance based on quality data collected by CMS and satisfaction surveys conducted by the State.
- Collaborate with providers. A good working relationship between government and providers contributed to the success of this initiative. Through the working group, all of the parties were able to develop a sense of trust, which made it easier to reach consensus on big decisions. The working group enabled the stakeholders involved to cultivate relationships and work through differences.
- Talk to residents and families first. Gathering input and feedback from families should be the starting point for a public reporting initiative of this kind, whose primary goal is to enable consumers and families to make quality-based choices.
Overview of Ohio Department of Aging
Ohio Department of Aging supports choice and independence. The Ohio Department of Aging's role is to advocate for the needs of all older citizens. The emphasis is on improving the quality of life for older Ohioans; helping senior citizens live active, healthy, and independent lives; and promoting positive attitudes toward aging and older people.
The Department is committed to helping the frail elderly who choose to remain at home by providing home and community-based services. Its goal is to promote the greatest level of choice, independence, and self-care of the individual, and to promote community living when appropriate and feasible in terms of cost and quality.
Nursing home occupancy falls; assisted living rises. There are 1,011 nursing homes in Ohio and 542 assisted living facilities. The number of nursing home beds in Ohio remained almost constant between 1992 and 2001. However, an increase in the use of community-based services and assisted living facilities led to a drop in nursing home occupancy rates, from 92 percent in 1992 to 83 percent in 2001. In contrast, the number of assisted living beds in Ohio grew from 8,700 in 1993 to 34,000 in 2001. According to the Ohio Department of Aging state plan, many of these assisted living beds are filled with private-pay patients who otherwise would have gone to a nursing home. Similar trends are apparent nationally.2
Long-term Care Consumer Guide Initiative
Collaboration characterizes early history of Consumer Guide and quality incentive initiatives. The Consumer Guide project began in 1999, when one of the three nursing home associations in Ohio approached the State about creating a nursing home report card. With Miami University of Ohio and the nursing home association, the Department started a working group to explore the feasibility of a report card to compare nursing homes based on performance on quality indicators. Key stakeholders in the group included the State Long-term Care Ombudsman, the Ohio Association of Regional Long-term Care Ombudsmen, Ohio Area Agencies on Aging, AARP, Ohio Health Department, and a family advocacy group, Families for Improved Care. The working group continues to play a role in the evolution of the Consumer Guide.
As this working group was taking shape, the nursing home association also independently partnered with a State legislator to create funding and specific regulations for the initiative. This legislative support was important to the development of the Web site.
Building on some of the data collected for the reporting initiative, the State also recently passed legislation creating a quality incentive for nursing homes through Medicaid reimbursement. Those homes that meet certain performance thresholds will receive a $3/day (equal to roughly 2 percent) increase in the daily rate they receive from the State for Medicaid beneficiaries.
Funding comes from the State. The Consumer Guide initiative received $407,000 in startup money from Ohio's resident protection fund, which collects civil penalties paid by nursing homes. Early in the project, State general revenue funds provided partial support. The initiative receives ongoing funding from an annual fee assessed to each nursing home ($400) and assisted living facility ($300) in the State.
There are development and maintenance costs. The cost of conducting family and resident satisfaction surveys for all nursing homes and assisted living facilities in the State is about $800,000. Part of the cost is driven by the need to do face-to-face interviews with residents. The State is attempting to conduct the surveys of assisted living residents by mail this year to reduce costs. The State has also decided to conduct the surveys in alternating years, whereby the State surveys residents one year, families the next year, etc.
The initial cost of developing the Long-term Care Consumer Guide Web site was about $230,000. The State invests in ongoing Web site maintenance and periodic upgrades, which have cost no more than $25,000 per contract.
Design, Development, and Implementation
Active working group fostered stakeholder engagement. The working group operated through consensus.
Local university assisted with design research. Miami University played a large role in survey development. A number of States have since requested permission to use the Department's consumer satisfaction surveys.
Patient and family surveys are included in Guide. When developing the Consumer Guide, the working group looked at the publicly available data on long-term care options and realized that a critical missing piece was information on customer satisfaction. Many Ohio residents had the same question for the Long-term Care Ombudsman when choosing a home: "Would you send your mother there?" To help answer this question, the working group decided to post the results of family and resident surveys on the Web site.
Guide includes quality measures.
- The family satisfaction survey contains 59 questions in 14 categories: activities, administration, admissions, choice, direct care, environment, general satisfaction, laundry, meals and dining, noise, professional nurses, receptionist/phone, social services, and therapy.
- The resident satisfaction survey contains 48 questions in 9 categories: activities, administration, choice, direct care and nurse assistants, environment, laundry, meals and dining, overall satisfaction, and social services.
- Quality Measures: The Consumer Guide repackages nursing home quality measures that are collected and reported by CMS, including measures that capture the well-being of patients (for example, the percentage of patients with pressure sores) as well as the nature of treatment (for example, the percentage of patients subject to physical restraints). The CMS measures are risk adjusted.
Data sources. Quality measures from Nursing Home Compare, a tool that provides information about the past performance of every Medicare and Medicaid certified nursing home in the country, are downloaded from the CMS Web site. Satisfaction survey data are collected and tabulated by contractors.
Small samples. Results for facilities with small sample sizes for any measure are not posted.
- Timeline. The Governor signed the bill providing funding for the Long-term Care Consumer Guide Initiative in July 2000. From that point, it took 1-1/2 years to launch the Web site.
- No pilot. Although significant testing of the Web site was done by selected consumer representatives, providers, and ombudsmen, there was no pilot of the Web site. At launch, nursing home family satisfaction data, regulatory data, and facility information were immediately made available to the public. Quality measures from CMS were added over time.
- Nursing home associations. In addition to press releases and its public comment process, the Department relied on the nursing home associations to publicize and gain support for the Consumer Guide. Because nursing home participation in the surveys is voluntary, buy-in and support from individual facilities were very important.
No evaluation. No formal evaluation of the Web site has been done.
Unintended consequences. Because the Consumer Guide reports deficiencies found in the nursing homes, the State noticed a jump in requests for informal dispute resolution when the site launched initially. However, these requests have leveled off over time.
Collaborate with providers. A good working relationship between government and providers contributed to the success of this initiative. Through the working group, all of the parties were able to develop a sense of trust, which made it much easier to reach consensus on big decisions. The working group offered an important vehicle for the stakeholders to develop a relationship and discuss and resolve differences.
Talk to residents and families first. Gathering input and feedback from families should be the starting point for a public reporting initiative of this kind. In this case, the key decisionmakers whom the initiative sought to inform and influence were residents and families, who articulated their need for information on the experience of others as a critical marker of quality.
Seize the moment. In the Ohio long-term care market, economic conditions were such that providers were looking for ways to show their value to potential residents and their families. This made the industry a valued partner in developing the Consumer Guide and also in lobbying the State Legislature for supporting funds and regulations.
Ohio Long-term Care Consumer Guide: http://www.ltcohio.org/consumer/index.asp
Return to Contents
Colorado Business Group on Health
The Colorado Business Group on Health (CBGH) is a business coalition composed of 22 public and private purchasers, representing 200,000 covered lives in Colorado. CBGH undertook the development and implementation of a public hospital reporting initiative in 2003 in collaboration with the Colorado Hospital Association to provide reliable, comparable data on the quality of care provided in hospitals throughout Colorado. This information is readily available to consumers via the Internet.
- Identify and cultivate a few champions to build support for initiative. The Executive Director of the CBGH worked with several hospital leaders who were willing to pitch the initiative to the State's hospital association.
- Rely on well-established quality measures to gain support of providers. CBGH and the hospital association used evidence-based quality indicators developed and maintained by AHRQ. Because the indicators were developed by a government agency, they reside in the public domain and include features such as risk adjustment where appropriate and standardized technical specifications.
- Engage employers early and often. Energizing the business community has been the key to CBGH's success. Some employers need to be convinced that they have a meaningful role to play in improving health care quality in their communities.
Overview of Colorado Business Group on Health
The Colorado Business Group on Health (CBGH) is composed of 22 public and private organizations representing about 1 in 11 covered lives in Colorado. CBGH focuses on improving health care quality through transparency and accountability. In addition to its member organizations, the nonprofit CBGH has 17 affiliates that provide health care. CBGH does not directly purchase health benefits or health care services.
Hospital Reporting Initiative
Hospital reporting initiative arose out of concern over rising costs. In the late 1980s, out of a desire to reign in spiraling health care costs, Colorado formed a data commission to examine hospital pricing. After about 5 years, responsibility for the initiative was transferred from the State government to the Colorado Hospital Association (CHA). CHA became responsible for collecting and disseminating all-payer hospital discharge data.
CBGH has a history of data reporting. In 1997, CBGH started a report card on health plan performance. Published annually, the report card relies on HEDIS® (Health Plan Employer Data and Information Set) and CAHPS® (Consumer Assessment of Healthcare Providers and Services) indicators. Over time, CBGH's focus on health plan reporting expanded to hospitals and the work of the CHA related to the all-payer data set. In 2003, CBGH engaged the CHA in a discussion on how to improve quality reporting in the State. After extensive meetings among key stakeholders, Colorado became the first State in the Nation to have hospitals voluntarily disclose mortality data. Today, reports can be downloaded at www.hospitalquality.org, the official Web site of the CHA Performance and Quality Work Group. This Web site provides consumers with reliable, comparable data on the quality of care provided in hospitals throughout Colorado. These reports are targeted to knowledgeable consumers of care with the hope of motivating hospitals to improve quality for all patients.
Initial funding came from the private sector. CBGH received a grant from a pharmaceutical company to implement the hospital reporting initiative. Ongoing upkeep of the Web site and updating of the quality measures are financial and operational responsibilities of the CHA.
Design, Development, and Implementation
CBGH built support for initiative. The CBGH Executive Director started to build a consensus for a hospital reporting initiative by engaging individual hospital Chief Executive Officers. A number of area health plans were publishing performance reports already, often using a "black box" methodology. A hospital-driven initiative, on the other hand, could be more comprehensive in its reporting and more transparent in its methodologies. CBGH soon found a few champions for the project who were willing to pitch the initiative to the board of the CHA. After a lively discussion, the CHA agreed to assemble a committee of key stakeholders. The committee met for 18 months and the results were very positive.
Collaborative effort encouraged stakeholder engagement. This was a collaborative effort among the Colorado Business Group on Health, CHA, Colorado Medical Society, CMS, Colorado Foundation for Medical Care, COPIC Insurance Company, Business Council on Health Care Competition, Physician Health Partners, Colorado Health Institute, Colorado Association of Health Plans, and Colorado Department of Public Health and Environment. All were responsible for attending meetings and providing input into the process.
Previous efforts were researched. In developing its initiative, CBGH looked at work done by all States and organizations that are reporting hospital metrics, including the State of Texas, the State of California, and the Niagara Health Quality Coalition.
AHRQ's Inpatient Quality Indicators software played a role. CBGH started its hospital project with a goal to reorganize and report the data already being collected in a meaningful way. CBGH recognized that a data-reporting initiative would be more robust if the all-payer data collected by CHA were evaluated using the AHRQ Inpatient Quality Indicators (IQI) analytics software. The value of IQI software is that it resides in the public domain, adjusts for risk, and can use any data set. Furthermore, IQI methodology was being used in several States such as New York, Florida, and Texas.
Measures are posted on the Web site. Last year, the Association agreed to voluntarily publish risk-adjusted mortality statistics for several inpatient conditions and inpatient procedures. Procedure volume metrics are also featured on the Web site.
Data collection.The hospital quality guide is based on administrative data collected by the Colorado Hospital Association. Each hospital's data are risk adjusted based on the All Patient Refined Diagnostic Related Groups (APR-DRGs) methodology developed by 3M Corporation and incorporated into the AHRQ QI software. Detailed information about the process used to organize and adjust the data for study purposes can be obtained from the AHRQ Quality Indicators Web site: http://www.qualityindicators.ahrq.gov. Results for 2004 and 2005 were published in spring/summer 2006.
Scoring. Hospitals are rated using a 95-percent confidence interval (CI) on a scale of "not different than expected" (graphically depicted with a half filled circle), "higher than expected mortality" (an empty circle), and "lower that expected mortality" (a filled circle). Hospital comments regarding their performance on these indicators are included in the report and are available on the Web site.
Small numbers. For any mortality indicator, the minimum threshold is 30 cases. For rural hospitals with 30 cases over 3 years, Colorado will pool the results and report an aggregate measure.
Timeline. The hospital reporting effort took about 2-1/2 years to design and launch.
Implementation costs. Over 2-1/2 years, CBGH spent approximately $50,000 to license the data and to hire a consultant to analyze and report the information. These efforts were supported by a grant from a pharmaceutical company. Today, ongoing upkeep of the Web site, including annual updates to the reports, is the responsibility of the CHA.
No ROI evaluation. No formal ROI evaluations have been done to date. However, CBGH believes that public reporting is causing visible changes in provider behavior. For instance, CBGH's Executive Director has seen the internal quality improvement functions within hospitals gaining more clout as quality improvement managers use public information to push for change. Last year, one Colorado hospital presented its results to CBGH, including its plans to improve performance on certain measures on which it ranked below average.
Unintended consequences. Thus far, there is no evidence or anecdote of unintended consequences (e.g., hospitals turning away high-risk patients).
Rely on well-established quality indicators. When deciding which quality indicators to use (Table 1), CBGH leaned heavily on work already in place. The CBGH Executive Director suggests that organizations considering quality reporting initiatives should look to existing initiatives for models; find measures in the public domain; have a transparent, well-documented methodology; and choose indicators that are risk adjusted and generate sufficient data.
Engage employers. CBGH's Executive Director strongly believes that energizing the business community has been key to CBGH's success. In particular, a case needs to be made that employers have an interest in and a role to play in improving the quality of care for their employees. Traditionally, businesses have left the job of managing health care costs and improving quality to insurance companies. However, as the role of managed care companies transitions from one of a gatekeeper to one of a processor of payments, it will be increasingly up to businesses to lead the charge. Local coalitions of businesses must take the lead in placing quality on the agenda in talks with providers. These groups must represent a critical mass of employers in order for providers to come to the table. In Colorado, the National Business Coalition on Health is working to expand its coalition through intensive education efforts on value-based purchasing.
Colorado Business Group on Health: http://www.coloradohealthonline.org/
Colorado Health and Hospital Association (CHA) Performance and Quality Group: Colorado Hospital Quality: www.hospitalquality.org
AHRQ's Quality Indicators: http://www.qualityindicators.ahrq.gov/
Go to Table 1: Quality Measures Used by CBGH Hospital Initiative.
Return to Contents
Proceed to Next Section