AHRQ Grant HS021857: Related Publication Summaries

Engaging Patients and Hospitals to Expand Public Reporting In Surgery

1. "Comparison of postoperative complication risk prediction approaches based on factors known preoperatively to surgeons versus patients."

Dahlke AR, Merkow RP, Chung JW, Kinnier CV, Cohen ME, Sohn MW, Paruch J, Holl JL, Bilimoria KY.
Surgery 2014 Jul; 156(1):39-45.
PUBMED link: www.ncbi.nlm.nih.gov/pubmed/24890570

This study used 2011 data from the ACS National Surgical Quality Improvement Program (NSQIP) to compare three models for predicting postoperative complications from general and colon surgery. The models were all the risk factors included in the NSQIP Standard model, factors known to the surgeon preoperatively, and factors generally known to patients. The three models were all superior to unadjusted complication rates not tailored to a patient's individual risk profile, for example, average rates for an institution. The models each predicted risk reasonably well, suggesting that even the basic information known by patients can be sufficient for shared decisionmaking tools such as the ACS NSQIP Surgical Risk Calculator.

2. "Association between hospital imaging use and venous thromboembolism events rates based on clinical data."

Ju MH, Chung JW, Kinnier CV, Bentrem DJ, Mahvi DM, Ko CY, Bilimoria KY.
Annals of Surgery
2014 Sep; 260(3):558-64; discussion 564-6.
PUBMED link: www.ncbi.nlm.nih.gov/pubmed/25115432

These authors examined the impact of hospital characteristics and hospital imaging use rate on venous thromboembolism (VTE) event rates from 208 hospitals. Hospital VTE imaging use rate was the dominant driver of hospital VTE event rates, as no other hospital characteristics had significant associations.The authors conclude that VTE rates reflect hospital imaging use and perhaps signify vigilant, high-quality care, and that the VTE outcome measure may not be an accurate quality indicator and should likely not be used in public reporting or pay-for-performance programs.

3. "Measuring postoperative recovery: What are clinically meaningful differences?"

Antonescu I, Scott S, Tran TT, Mayo NE, Feldman LS.
Surgery 2014 Aug; 156(2):319-27.

PUBMED link: www.ncbi.nlm.nih.gov/pubmed/24947644

This study generated the minimal clinically important difference (MCID) estimates for three postoperative recovery metrics for two cohorts of 281 and 130 adult patients undergoing abdominal surgery. At each of three visits, patients had completed the 36-Item Short Form Survey from the RAND Medical Outcomes Study (SF-36) and either Community Healthy Activities Model Program for Seniors (CHAMPS) or the 6-minute walk test (6MWT). On the SF-36, MCIDs were consistently smaller for patients rating their health as "excellent" or "very good" compared with those for patients rating their health as "fair" or "poor." However similar distinctions were less pronounced on the other instruments. The authors established plausible MCIDs and ranges around each estimate, and recommend that these values be considered when planning and interpreting abdominal surgery clinical trials where patient-reported outcomes are assessed.

4. "Postoperative venous thromboembolism outcomes measure: Analytic exploration of potential misclassification of hospital quality due to surveillance bias."

Chung JW, Ju MH, Kinnier CV, Sohn MW, Bilimoria KY.
Annals of Surgery 2015 Mar; 261(3):443-4.

PUBMED link: www.ncbi.nlm.nih.gov/pubmed/25119123

These authors explore the theoretical potential of using Patient Safety Indicator 12 (PSI 12), postoperative venous thromboembolism (VTE), to misclassify hospital quality due to surveillance bias, i.e., hospitals that screen more have higher VTE rates. They note that quantifying the magnitude of misclassification in PSI 12 hinges on a criterion standard measuring VTE incidence independent of VTE surveillance, which does not exist. The authors recommend limiting the use of PSI 12 in payment and reporting programs until a more nuanced measure can be developed.

5. "Public reporting in surgery: An emerging opportunity to improve care and inform patients."

Minami CA, Dahlke A, Bilimoria KY.
Annals of Surgery 2015 Feb; 261(2):241-2.

PUBMED link: www.ncbi.nlm.nih.gov/pubmed/25565124

This article reviews the benefits and concerns surrounding public reporting of surgery. It suggests some possible solutions aimed at improved implementation and use of public reporting to enhance surgical outcomes. Suggestions include a push for more individualized information, excluding extreme cases (e.g., the very elderly) from all measures, and more broadly focusing measures such as readmission rates.

6. "Facilitating quality improvement: Pushing the pendulum back toward process measures."

Bilimoria KY.
JAMA 2015 Oct 6; 314(13):1333-4.

PUBMED link: www.ncbi.nlm.nih.gov/pubmed/26441175

This author asserts that the recent policy shift away from examining process measures toward outcome measures for performance measurement is a mistake—that process measures are critical for driving quality improvement. Although there are some criticisms of process measures, such as they often focus too narrowly on a single aspect of a disease process (e.g., antibiotic administration), outcomes measurements also have problems, notes the author.  For example, equity in outcomes-based performance measurement hinges on appropriate risk adjustment, which may be challenging given available administrative data and the costs of collecting better clinical data. Process measures offer considerable advantages for local quality improvement initiatives. In addition, process measures are directly actionable, while outcomes are more difficult for a hospital to improve, because it must dissect all the potential factors driving poor outcomes. The author concludes that process measures should remain central in efforts to measure and improve care.  He makes several suggestions, including development of process measure composites that reflect best practice bundles for a wide array of disease processes.

7. "Concerns about using the patient safety indicator-90 composite in pay-for-performance programs."

Rajaram R, Barnard C, Bilimoria KY.
JAMA 2015 Mar 3; 313(9):897-8.

PUBMED link: www.ncbi.nlm.nih.gov/pubmed/25654581

On October 1, 2014, the Centers for Medicare & Medicaid Services (CMS) began using AHRQ’s Patient Safety for Selected Indicators (PSI-90) as a core metric in two if its pay-for-performance programs: the Hospital-Acquired Condition (HAC) Reduction Program and the Hospital Value-Based Purchasing (VBP) Program.  In the HAC Reduction Program, PSI-90 is responsible for 35 percent of the overall score, and the poorest-performing hospital quartile will have their CMS payments reduced by up to 1 percent (about $330 million). In the Hospital VBP program, CMS will reallocate 1.5 percent (about $1.4 billion) of its diagnosis-related group payments to hospitals according to their overall score, 30 percent of which is composed of PSI-90 and four other outcome measures. These authors assert that, as currently constructed, CMS’ use of PSI-90 in pay-for-performance programs falls short in its attempt to accurately and fairly measure patient harm. A composite measure of quality is a useful goal, but PSI-90 must be improved to be effective in advancing the national priority of safety patient care. They cite areas of concern about the PSI-90 ranging from inclusion of measurement components that are flawed or  redundantly addressed by both the Hospital VBP and HAC Reduction programs to those that fail to  accurately measure clinically relevant complications.

8. "Hospital characteristics associated with penalties in the Centers for Medicare & Medicaid Services Hospital-Acquired Condition Reduction Program."

Rajaram R, Chung JW, Kinnier CV, Barnard C, Mohanty S, Pavey ES, McHugh MC, Bilimoria KY.
JAMA 2015 Jul 28; 314(4):375-83.

PUBMED link: www.ncbi.n.m.nih.gov/pubmed/26219055

The Centers for Medicare & Medicaid Services (CMS) Hospital-Acquired Condition (HAC) Reduction Program reduces payments to the lowest-performing hospitals. To explore whether this program accurately measures quality and fairly penalizes hospitals, these researchers examined the characteristics of hospitals penalized by the HAC Reduction Program and the link between the summary score of hospital characteristics related to quality with penalization in the HAC Program. Of the 3,284 hospitals participating in the HAC program, 22 percent were penalized. The hospitals that were penalized more often had more quality accreditations, offered advanced services, were major teaching institutions, and had better performance on other process and outcome measures. The authors note that these "paradoxical findings" suggest that the HAC Reduction Program’s approach to assessing hospital penalties warrants reconsideration so that it achieves its intended goals.

9. "Development and evaluation of the Universal ACS NSQIP Surgical Risk Calculator: A decision aid and informed consent tool for patients and surgeons."

Bilimoria KY, Liu Y, Paruch JL, and others.
Journal of the American College of Surgeons 2013 Nov; 217(5):833-842.el-2.
PUBMED link:

The objectives of this study were to use the high-quality clinical data collected by the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQUIP) in order to develop an effective surgical risk estimation tool; to compare the performance of the new universal surgical risk calculator with previous procedure-specific risk calculators; and to develop an approach for clinicians to reasonably and empirically adjust risk estimates based on their clinical judgment and experience.

10. "Surgeons’ perceptions of public reporting of hospital and individual surgeon quality."

Sherman KL, Gordon EJ, Mahvi, DM, and others.
Medical Care 2013 Dec; 51(12):1069-75.
PUBMED link:

This study evaluated surgeons’ perceptions of public reporting and identified specific barriers to surgeons’ acceptance of public reporting. Of the 185 surgeons surveyed at four hospitals, most supported public reporting of hospital but not individual quality metrics. Among their concerns were patients misinterpreting data and outcome metric validity.

11. "Evaluation of surveillance bias and the validity of the venous thromboembolism quality measure."

Bilimoria KY, Chung J, Ju MH, and others.
Journal of the American Medical Association 2013 Oct; 310(14):1482-9.
PUBMED link:

This study examined whether a surveillance bias influences the validity of reported venous thromboembolism (VTE) rates. It found that since some hospitals use imaging studies more frequently than others, a surveillance bias limits the usefulness of the VTE quality measure for hospitals working to improve quality and patients seeking to identify a high-quality hospital.

12. "Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus."

Cohen ME, Ko CY, Bilimoria KY, and others.
Based on AHRQ grant HS21857.Journal of the American College of Surgeons 2013 August; 217(2):336-346.e1. 
PUBMED link:

The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) has worked to improve the reliability and validity of hospital profiling by adding a variable to better adjust for the complexity and risk profile of surgical procedures and stabilizing estimates derived from small samples by using a hierarchical model with shrinkage adjustment. This article describes the development and justification for these new statistical methods and reporting strategies in ACS NSQIP.

13. "Evaluation of initial participation in public reporting of American College of Surgeons NSQIP surgical outcomes on Medicare’s Hospital Compare website."

Dahlke AR, Chung JW, Holl JL, and others.
Journal of the American College of Surgeons 2014 March; 218(3):374-360, 380.el-5.
PUBMED link:

The researchers compared NSQIP hospitals participating in the Centers for Medicare and Medicaid Services Hospital Compare Web sites with those NSQIP hospitals that did not participate. They found few measurable differences. The decision to voluntarily publicly report may be related to the hospital’s culture of quality improvement and transparency, the researchers concluded.

14. "Associations between hospital characteristics, measure reporting, and the Centers for Medicare & Medicaid Services Overall Hospital Quality Star Ratings."

DeLancey JO, Softcheck J, Chung JW, Barnard C, Dahlke AR, Bilimoria KY.
JAMA. 2017 May 16;317(19):2015-2017.
PUBMED link:

This study evaluated associations between hospital characteristics, number and types of measures reported, and the star ratings. Of 3,591 hospitals receiving a star rating, 4 or 5 stars were awarded to 15.8 percent of major teaching hospitals, 18.8 percent of other teaching hospitals, 30.2 percent of community hospitals, 33.3 percent of critical access hospitals, and 87.3 percent of specialty hospitals.

15. "Does performance vary within the same hospital when separately examining different patient subgroups?"

Berian JR, Paruch JL, Cohen ME, Merkow RP, Dahlke AR, Ko CY, Bilimoria KY.
Journal of the American College of Surgeons 2016 May;222(5):790-797.e1.
PUBMED link:

The researchers sought to determine whether performance differs within a given hospital for six contrasting patient subgroups and to identify the percentage of hospitals with greater than chance differences in performance. They found that overall quality differed for elderly vs. nonelderly, renal insufficiency vs. normal renal function, cancer vs. noncancer, and emergency vs. nonemergency. They concluded that quality programs can consider separate reports for these subgroups to identify opportunities for quality improvement.

16. "Evaluation of the ProPublica surgeon scorecard ‘adjusted complication rate’ measure specifications."

Ban KA, Cohen ME, Ko CY, Friedberg MW, Stulberg JJ, Zhou L, Hall BL, Hoyt DB, Bilimoria KY.
Annals of Surgery 2016 Oct;264(4):566-74.
PUBMED link:

The authors sought to (1) determine the proportion of cases excluded by ProPublica's specifications, (2) assess the proportion of inpatient complications excluded from ProPublica's measure, and (3) examine the validity of ProPublica's outcome measure by comparing performance on the measure to well-established postoperative outcome measures. They found that ProPublica's outcome measure specifications exclude 82% of cases, miss 84% of postoperative complications, and correlate poorly with well-established postoperative outcomes.

17. "Association between hospital characteristics and performance on the new hospital-acquired condition reduction program's surgical site infection measures."

Minami CA, Dahlke AR, Barnard C, Kinnier CV, Rajaram RR, Noskin GA, Bilimoria KY.
JAMA Surgery 2016 Aug 1;151(8):777-9.
PUBMED link:

This research letter evaluated the association between hospital characteristics and surgical site infection (SSI) measures. The authors found that hospitals with higher hospital quality summary scores were more frequently poor performers for SSIs and had higher standardized infection ratios. Hospitals were more likely to be poor performers for colon SSI and hysterectomy SSI if they were a teaching hospital, safety-net hospital, or level I trauma center. Teaching hospitals were more likely to be poor performers for colorectal SSI, but the association was not as consistent for hysterectomy.

18. "Current challenges in using patient-reported outcomes for surgical care and performance measurement: everybody wants to hear from the patient, but are we ready to listen?"

Bilimoria KY, Cella D, Butt Z.
JAMA Surgery 2014 Jun;149(6):505-6.
PUBMED link:

This viewpoint article discusses patient-reported outcomes (PROs). It concludes that PROs undoubtedly capture some of the most important aspects of how successful an operation is. Also, PRO scores may be the best measure of patient-centered care, but considerable logistical and methodological issues must be addressed before PROs are ready for widespread national implementation.

19. "Effect of including cancer-specific variables on models examining short-term outcomes."

Merkow RP, Kmiecik TE, Bentrem DJ, Winchester DP, Stewart AK, Ko CY, Bilimoria KY.
Cancer 2013 Apr 1;119(7):1412-9. doi: 10.1002/cncr.27891. Epub 2012 Nov 26.
PUBMED link:

The objectives of this paper were 1) to examine differences between existing American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) variables and cancer registry variables, and 2) to determine whether the addition of cancer-specific variables improves modeling of short-term outcomes. The researchers found that, although advanced disease stage and neoadjuvant therapy variables were predictors of short-term outcomes, their inclusion did not improve the models.

20. "An evaluation of differences in risk factors for individual types of surgical site infections after colon surgery."

Segal CG, Waller DK, Tilley B, Piller L, Bilimoria K
Surgery 2014 Nov;156(5):1253-60.
PUBMED link:

The authors developed four independent, multivariate, predictive models to assess the unique associations between risk factors and each surgical site infection (SSI) group: superficial, deep, organ space, and an aggregate of all 3 types of SSIs. They found that unique risks for superficial SSIs include diabetes, chronic obstructive pulmonary disease, and dyspnea. Deep SSIs had the greatest magnitude of association with body mass index and the greatest incidence of wound disruption. Organ space SSIs were often owing to anastomotic leaks and were uniquely associated with disseminated cancer, preoperative dialysis, preoperative radiation treatment, and a bleeding disorder. The researchers concluded that more effective prevention strategies may be developed by reporting and examining each type of SSI separately.


Page last reviewed May 2018
Page originally created October 2012
Internet Citation: AHRQ Grant HS021857: Related Publication Summaries. Content last reviewed May 2018. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/professionals/quality-patient-safety/quality-resources/tools/sciencepubreport/hs021857.html