AHRQ Grant HS021857: Related Publication Summaries

Engaging Patients and Hospitals to Expand Public Reporting In Surgery

1. "Comparison of postoperative complication risk prediction approaches based on factors known preoperatively to surgeons versus patients."

Dahlke AR, Merkow RP, Chung JW, Kinnier CV, Cohen ME, Sohn MW, Paruch J, Holl JL, Bilimoria KY.
Surgery 2014 Jul; 156(1):39-45.
PUBMED link: www.ncbi.nlm.nih.gov/pubmed/24890570

This study used 2011 data from the ACS National Surgical Quality Improvement Program (NSQIP) to compare three models for predicting postoperative complications from general and colon surgery. The models were all the risk factors included in the NSQIP Standard model, factors known to the surgeon preoperatively, and factors generally known to patients. The three models were all superior to unadjusted complication rates not tailored to a patient's individual risk profile, for example, average rates for an institution. The models each predicted risk reasonably well, suggesting that even the basic information known by patients can be sufficient for shared decisionmaking tools such as the ACS NSQIP Surgical Risk Calculator.

2. "Association between hospital imaging use and venous thromboembolism events rates based on clinical data."

Ju MH, Chung JW, Kinnier CV, Bentrem DJ, Mahvi DM, Ko CY, Bilimoria KY.
Annals of Surgery
2014 Sep; 260(3):558-64; discussion 564-6.
PUBMED link: www.ncbi.nlm.nih.gov/pubmed/25115432

These authors examined the impact of hospital characteristics and hospital imaging use rate on venous thromboembolism (VTE) event rates from 208 hospitals. Hospital VTE imaging use rate was the dominant driver of hospital VTE event rates, as no other hospital characteristics had significant associations.The authors conclude that VTE rates reflect hospital imaging use and perhaps signify vigilant, high-quality care, and that the VTE outcome measure may not be an accurate quality indicator and should likely not be used in public reporting or pay-for-performance programs.

3. "Measuring postoperative recovery: What are clinically meaningful differences?"

Antonescu I, Scott S, Tran TT, Mayo NE, Feldman LS.
Surgery 2014 Aug; 156(2):319-27.

PUBMED link: www.ncbi.nlm.nih.gov/pubmed/24947644

This study generated the minimal clinically important difference (MCID) estimates for three postoperative recovery metrics for two cohorts of 281 and 130 adult patients undergoing abdominal surgery. At each of three visits, patients had completed the 36-Item Short Form Survey from the RAND Medical Outcomes Study (SF-36) and either Community Healthy Activities Model Program for Seniors (CHAMPS) or the 6-minute walk test (6MWT). On the SF-36, MCIDs were consistently smaller for patients rating their health as "excellent" or "very good" compared with those for patients rating their health as "fair" or "poor.” However similar distinctions were less pronounced on the other instruments. The authors established plausible MCIDs and ranges around each estimate, and recommend that these values be considered when planning and interpreting abdominal surgery clinical trials where patient-reported outcomes are assessed.

4. "Postoperative venous thromboembolism outcomes measure: Analytic exploration of potential misclassification of hospital quality due to surveillance bias."

Chung JW, Ju MH, Kinnier CV, Sohn MW, Bilimoria KY.
Annals of Surgery 2015 Mar; 261(3):443-4.

PUBMED link: www.ncbi.nlm.nih.gov/pubmed/25119123

These authors explore the theoretical potential of using Patient Safety Indicator 12 (PSI 12), postoperative venous thromboembolism (VTE), to misclassify hospital quality due to surveillance bias, i.e., hospitals that screen more have higher VTE rates. They note that quantifying the magnitude of misclassification in PSI 12 hinges on a criterion standard measuring VTE incidence independent of VTE surveillance, which does not exist. The authors recommend limiting the use of PSI 12 in payment and reporting programs until a more nuanced measure can be developed.

5. "Public reporting in surgery: An emerging opportunity to improve care and inform patients."

Minami CA, Dahlke A, Bilimoria KY.
Annals of Surgery 2015 Feb; 261(2):241-2.

PUBMED link: www.ncbi.nlm.nih.gov/pubmed/25565124

This article reviews the benefits and concerns surrounding public reporting of surgery. It suggests some possible solutions aimed at improved implementation and use of public reporting to enhance surgical outcomes. Suggestions include a push for more individualized information, excluding extreme cases (e.g., the very elderly) from all measures, and more broadly focusing measures such as readmission rates.

6. "Facilitating quality improvement: Pushing the pendulum back toward process measures."

Bilimoria KY.
JAMA 2015 Oct 6; 314(13):1333-4.

PUBMED link: www.ncbi.nlm.nih.gov/pubmed/26441175

This author asserts that the recent policy shift away from examining process measures toward outcome measures for performance measurement is a mistake—that process measures are critical for driving quality improvement. Although there are some criticisms of process measures, such as they often focus too narrowly on a single aspect of a disease process (e.g., antibiotic administration), outcomes measurements also have problems, notes the author.  For example, equity in outcomes-based performance measurement hinges on appropriate risk adjustment, which may be challenging given available administrative data and the costs of collecting better clinical data. Process measures offer considerable advantages for local quality improvement initiatives. In addition, process measures are directly actionable, while outcomes are more difficult for a hospital to improve, because it must dissect all the potential factors driving poor outcomes. The author concludes that process measures should remain central in efforts to measure and improve care.  He makes several suggestions, including development of process measure composites that reflect best practice bundles for a wide array of disease processes.

7. "Concerns about using the patient safety indicator-90 composite in pay-for-performance programs."

Rajaram R, Barnard C, Bilimoria KY.
JAMA 2015 Mar 3; 313(9):897-8.

PUBMED link: www.ncbi.nlm.nih.gov/pubmed/25654581

On October 1, 2014, the Centers for Medicare & Medicaid Services (CMS) began using AHRQ’s Patient Safety for Selected Indicators (PSI-90) as a core metric in two if its pay-for-performance programs: the Hospital-Acquired Condition (HAC) Reduction Program and the Hospital Value-Based Purchasing (VBP) Program.  In the HAC Reduction Program, PSI-90 is responsible for 35 percent of the overall score, and the poorest-performing hospital quartile will have their CMS payments reduced by up to 1 percent (about $330 million). In the Hospital VBP program, CMS will reallocate 1.5 percent (about $1.4 billion) of its diagnosis-related group payments to hospitals according to their overall score, 30 percent of which is composed of PSI-90 and four other outcome measures. These authors assert that, as currently constructed, CMS’ use of PSI-90 in pay-for-performance programs falls short in its attempt to accurately and fairly measure patient harm. A composite measure of quality is a useful goal, but PSI-90 must be improved to be effective in advancing the national priority of safety patient care. They cite areas of concern about the PSI-90 ranging from inclusion of measurement components that are flawed or  redundantly addressed by both the Hospital VBP and HAC Reduction programs to those that fail to  accurately measure clinically relevant complications.

8. "Hospital characteristics associated with penalties in the Centers for Medicare & Medicaid Services Hospital-Acquired Condition Reduction Program."

Rajaram R, Chung JW, Kinnier CV, Barnard C, Mohanty S, Pavey ES, McHugh MC, Bilimoria KY.
JAMA 2015 Jul 28; 314(4):375-83.

PUBMED link: www.ncbi.n.m.nih.gov/pubmed/26219055

The Centers for Medicare & Medicaid Services (CMS) Hospital-Acquired Condition (HAC) Reduction Program reduces payments to the lowest-performing hospitals. To explore whether this program accurately measures quality and fairly penalizes hospitals, these researchers examined the characteristics of hospitals penalized by the HAC Reduction Program and the link between the summary score of hospital characteristics related to quality with penalization in the HAC Program. Of the 3,284 hospitals participating in the HAC program, 22 percent were penalized. The hospitals that were penalized more often had more quality accreditations, offered advanced services, were major teaching institutions, and had better performance on other process and outcome measures. The authors note that these "paradoxical findings" suggest that the HAC Reduction Program’s approach to assessing hospital penalties warrants reconsideration so that it achieves its intended goals.

9. "Development and evaluation of the Universal ACS NSQIP Surgical Risk Calculator: A decision aid and informed consent tool for patients and surgeons."

Bilimoria KY, Liu Y, Paruch JL, and others.
Journal of the American College of Surgeons 2013 Nov; 217(5):833-842.el-2.
PUBMED link:

The objectives of this study were to use the high-quality clinical data collected by the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQUIP) in order to develop an effective surgical risk estimation tool; to compare the performance of the new universal surgical risk calculator with previous procedure-specific risk calculators; and to develop an approach for clinicians to reasonably and empirically adjust risk estimates based on their clinical judgment and experience.

10. "Surgeons’ perceptions of public reporting of hospital and individual surgeon quality."

Sherman KL, Gordon EJ, Mahvi, DM, and others.
Medical Care 2013 Dec; 51(12):1069-75.
PUBMED link:

This study evaluated surgeons’ perceptions of public reporting and identified specific barriers to surgeons’ acceptance of public reporting. Of the 185 surgeons surveyed at four hospitals, most supported public reporting of hospital but not individual quality metrics. Among their concerns were patients misinterpreting data and outcome metric validity.

11. "Evaluation of surveillance bias and the validity of the venous thromboembolism quality measure."

Bilimoria KY, Chung J, Ju MH, and others.
Journal of the American Medical Association 2013 Oct; 310(14):1482-9.
PUBMED link:

This study examined whether a surveillance bias influences the validity of reported venous thromboembolism (VTE) rates. It found that since some hospitals use imaging studies more frequently than others, a surveillance bias limits the usefulness of the VTE quality measure for hospitals working to improve quality and patients seeking to identify a high-quality hospital.

12. "Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus."

Cohen ME, Ko CY, Bilimoria KY, and others.
Based on AHRQ grant HS21857.Journal of the American College of Surgeons 2013 August; 217(2):336-346.e1. 
PUBMED link:

The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) has worked to improve the reliability and validity of hospital profiling by adding a variable to better adjust for the complexity and risk profile of surgical procedures and stabilizing estimates derived from small samples by using a hierarchical model with shrinkage adjustment. This article describes the development and justification for these new statistical methods and reporting strategies in ACS NSQIP.

13. "Evaluation of initial participation in public reporting of American College of Surgeons NSQIP surgical outcomes on Medicare’s Hospital Compare website."

Dahlke AR, Chung JW, Holl JL, and others.
Journal of the American College of Surgeons 2014 March; 218(3):374-360, 380.el-5.
PUBMED link:

The researchers compared NSQIP hospitals participating in the Centers for Medicare and Medicaid Services Hospital Compare Web sites with those NSQIP hospitals that did not participate. They found few measurable differences. The decision to voluntarily publicly report may be related to the hospital’s culture of quality improvement and transparency, the researchers concluded.

Page last reviewed February 2016
Page originally created October 2012
Internet Citation: AHRQ Grant HS021857: Related Publication Summaries. Content last reviewed February 2016. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/professionals/quality-patient-safety/quality-resources/tools/sciencepubreport/hs021857.html