Appendix U. Improving the Measurement of Surgical Site Infection Risk Stratification/Outcome Detection

Slide presentation

On September 18, 2011, Connie Savor Price, MD made this presentation at the 2011 Annual AHRQ HAI Investigators Meeting. Select to access the  Microsoft PowerPoint file Microsoft PowerPoint version - 394.16 KB .


Slide 1

Slide 1 Image

Improving the Measurement of Surgical Site Infection (SSI) Risk Stratification and Outcome Detection

Connie Savor Price, MD

2nd Annual AHRQ HAI Investigators Meeting
September 18, 2011
Bethesda, Maryland

Slide 2

Slide 2 Image

Background

  • SSIs are a substantial cause of morbidity and mortality
    • Accounted for ~16 % of ~1.7 million HAIs and 8,205 of the 98,987 HAI associated deaths (mortality= 3%).
  • Financial burden significant
    • Hospital cost of ~$25,546 per SSI and ~$7 billion annually
  • Feedback on surgeon-specific rates is considered to be the cornerstone for preventing these infections.
    • Surgeons must believe rates are reliable
  • Current surveillance methods are perceived as limited in risk adjustment and detection

Slide 3

Slide 3 Image

The Burden of SSI Surveillance

~27M procedures/yr*
20 min to review a procedure

Leads to:

9M hours to review all procedures
616.4 FTE/year

* Klevens, et al, Public Health Reports, Mar-Apr 2007, pg 160.

Slide 4

Slide 4 Image

Aims

  • Employ electronic detection algorithms to determine SSI rates for selected procedures* in 4 unique hospital settings
  • Conduct focus groups to assess:
    • Surgeons' acceptance of current risk stratification models and determine what risk factors surgeons deem important for future model development
    • Adoption of electronic surveillance tool by Infection Prevention nurses
  • Design and test methods to risk stratify on data elements available for electronic collection

* Coronary artery bypass graft, hernia repair, hip and knee arthroplasty

Slide 5

Slide 5 Image

Images: Pictures of the 4 hospital settings involved in the study. Clockwise from top left: Denver Health, Intermountain Healthcare, Vail Valley Medical Center, and the Salt Lake City Veterans Administration.

Slide 6

Slide 6 Image

Employ Electronic Detection Algorithms to Determine SSI Rates for Selected Procedures in 4 Unique Hospital Settings

Slide 7

Slide 7 Image

Combining the Best of Both Worlds

  • Human: smart, adaptable
    PLUS
  • Machine: consistent, reliable, scalable
    EQUALS
  • The best of both worlds: human-adjudication of electronically triaged cases
    • Electronically eliminate most uncomplicated surgeries
    • Utilize Infection Preventionist knowledge & experience in hard cases
    • Highest sensitivity and negative predictive value desired

http://www.scientificamerican.com/article.cfm?id=post-911-military-technology

Slide 8

Slide 8 Image

Methods

  • Create Algorithm
    • Literature review identified electronic data that are manifestations, but not risks of SSI to predict deep or organ-space SSI
  • Train Algorithm
    • Used randomly selected ½ of 2007-2009 VASQIP data for outcomes data
  • Test Algorithm
    • One-fold cross validation using other ½ of VASQIP data
    • External validation at the 3 other systems
      • "Portable" algorithms implemented and analyzed locally
      • Data standardized to ensure interoperability

Slide 9

Slide 9 Image

Results
Data Elements for Training Sets

  • WBC count, WBC differential, erythrocyte sedimentation rate (ESR), c-reactive protein (CRP), microbiology results, and antimicrobial administration
  • Fever
    • Excluded as not available electronically at all sites
  • Procalcitonin
    • Excluded as this result is not widely available in the US
  • Claims data
    • Excluded as generally not available until well after an IP would be reviewing cases

Slide 10

Slide 10 Image

Results

Based on our findings in the literature review, a data dictionary was sent to each of the participating centers to pull their data and facilitate the dissemination of code scripts that would run the algorithm

A table showing example data names, values, and descriptions is included with this slide for illustrative purposes.

Slide 11

Slide 11 Image

Results
Implementation Time

  • Intermountain (23 hospitals)
    • 50 hours
  • Vail Valley (1 hospital)
    • 90 hours
  • Denver Health (1 hospital)
    • 25 hours
  • VA SLC HCS (153 hospitals)
    • 200+ hours

Slide 12

Slide 12 Image

Component Rules of the Classification Tree Algorithm

CABG:

All of the following:
Presence of a post-operative culture, and
Post-operative antibiotics were given, and
Maximum post-operative leukocyte count is not less than 11.85

Herniorrhaphy:

Either of the following:
Presence of a post-operative culture and
   Maximum post-operative leukocyte count is not less than 7.78
Absence of a post-operative culture and one of the following criteria:
   Post-operative antibiotics given and any post-operative leukocyte count test drawn
   Post-operative antibiotics not given, but the patient had a post-operative admission

Total Knee Arthroplasty:

Either of the following:
Presence of a post-operative culture, or
Presence of a c-reactive protein and the maximum post-operative leukocyte count is not less than 9.45

Total Hip Arthroplasty:

All of the following:
Presence of a post-operative culture, and
Post-operative antibiotics were given, and
Maximum post-operative leukocyte count is not less than 7.55

Abbreviations: CABG, coronary artery bypass grafting

Slide 13

Slide 13 Image

Two Other Simpler Rules

"Inclusive" algorithm:

Any one of the following:
Erythrocyte sedimentation rate greater than 20, or
Total neutrophil count greater than 5,000/mm3, or
Total leukocyte count greater than 9,000/mm3, or
C-reactive protein greater than 3mg/dL, or
Any post-operative antibiotics given, or
Presence of a post-operative culture, or
Patient was readmitted within 30 days post-operatively

"Simple" algorithm:

Either of the following:
Microbiology test ordered between post-operative days 4 and 30 (inclusive), or
An antibacterial was prescribed between post-operative days 4 and 30 (inclusive)

Slide 14

Slide 14 Image

Performance of Different Algorithms by SSI Type on the Test Set

 # Flagged
(% Total)
% Sensitivity
 sSSIdSSIoSSI
Rpart Algorithm7.3%45.2%72.0%75.8%
Inclusive Algorithm31.3%90.0%100%90.9%
Simple Algorithm19.2%82.4%90.7%90.9%

Abbreviations: SSI, Surgical Site Infection; sSSI, superficial SSI; dSSI, deep SSI; oSSI, organ space SSI

Slide 15

Slide 15 Image

External Validation of Human-Adjudicated Surveillance

Table illustrating the accuracy of adjudication at each hospital site, as described in detail in the final report narrative.

Slide 16

Slide 16 Image

Post-Mortem

  • All false negatives were reviewed
    • Some cases were arguably not deep or organ-space SSI
    • Some cases were flagged as SSI outside of 30-day window
    • The most commonly cited source of information that could have prevented a false negative was in clinical notes.

Slide 17

Slide 17 Image

Lessons Learned

  • Common electronic data markers of infection are not informative enough.
  • Variation in practice can alter the performance of algorithms between facilities.
  • Over-fitting is likely under-recognized in the literature because of small numbers of facilities.
  • Even though the VA system is large, individual hospital practices and data storage are more like each other than community facilities
    • Post-discharge surveillance is still an issue.

Slide 18

Slide 18 Image

Future Directions

  • Natural Language Processing may be necessary to generate more informative data.
  • Training algorithms should include a large number of diverse hospitals.
  • Algorithms may be better used to estimate the likelihood of SSI for triage as opposed to a complete rule-out determination (likelihood score for validation of publicly reported data)?
  • Denver Health had modified the algorithm to tailor the surveillance to our institution for a variety of surgeries; sensitivity 100 percent, with a 60 percent decrease in chart review time.

Slide 19

Slide 19 Image

Conduct Focus Groups

Slide 20

Slide 20 Image

Methods

  • Surgeon: 6 surgeons with research interests in SSI, representing multiple health system types (4 academic, 2 private, 2 safety net, 1 VA) and surgical specialties (5 general, 2 trauma/critical care, 1 surgical oncology).
  • Infection Prevention (Denver): 5 Infection Preventionists from 4 private hospitals and 1 public hospital. Average years as an IP = 10.

Slide 21

Slide 21 Image

Results and Implications

  • Surgeons feel that current risk adjustment models are inadequate
    • Too simplistic.
  • IPs receptive to concept of electronic triage with human adjudication
    • Needs to be adaptable to variety of systems, low/no cost.
  • More refined models may improve acceptance of data and benchmarks.
  • Further research to identify evidence-based risk factors for SSI is needed.
  • Risk Factors were suggested by surgeons for future models.

Slide 22

Slide 22 Image

Design and Test Methods to Risk Stratify on Data Elements Available for Electronic Collection

Slide 23

Slide 23 Image

Methods.
Identifying Potential Risk Factors:

  • Surgeon PI used experience with SSIs and a list of risk factors used by his institution to identify potential risk factors. That initial list consisted of 88 risk factors for SSI.
    • This list was used in a focus group to solicit input with surgeons.
  • Literature review was performed using PubMed and Google Scholar to identify published risk factors for SSIs
    • All English language publications [in] previous 10 years
    • Keywords: SSI, surgical site infection, surgical risk factor, risk factor, surgical wound, surgical infection.

Slide 24

Slide 24 Image

Methods
Identifying Electronic Availability

  • Master List of potential risk factors sent to each of the four study sites.
    • Each potential risk factor clinically reviewed and categorized as modifiable or nonmodifiable.
  • Each site determined if they had electronic access to the individual risk factors.
  • Union set of 34 electronically available potential risk factors common to all sites identified and defined.

Slide 25

Slide 25 Image

Methods.
Identifying Potential Risks and Outcomes in Procedures of Interest

  • Each site collected potential risk factor variables for adults who had undergone CABG, herniorrhaphies, hip arthroplasty and knee arthroplasty.
  • SSI* outcome noted for each patient.

*CDC/NHSN methodology

Slide 26

Slide 26 Image

Results
SSI Union Set of Common Risk Factors with Electronic Availability

List of common factors available for collection as existing electronic data from all sites, as described in the final report narrative.

Slide 27

Slide 27 Image

Results
Description of Population

  • A total of 3,612 herniorrhaphies, 3,410 total hip and 9,728 total knee procedures.
  • An additional 1,802 CABG and 5,873 appendectomy procedures were submitted from Intermountain and the VAMC.
  • A total of 222 SSIs were associated with the various surgical procedures and participating facilities.
  • The SSI rates varied by site and procedure each year and ranged from 0.0% to 7.1%.

Slide 28

Slide 28 Image

Data Issues

  • Missing values:
    • Many lab values were missing.
      • Statistical tests automatically remove records with missing values.
  • Removed records reduced each site's SSIs drastically:
    • DH reduced by 33% (n = 12)
    • IH reduced by 65% (n = 66)
    • All VA records with SSIs removed
    • All VV records with SSIs removed.

Slide 29

Slide 29 Image

Handling Missing Values

  • Options:
    • Remove records with missing lab values
    • Change missing values to 0; control for missing values with dummy variables
    • Bootstrapping
    • Multiple imputation.
  • Chose multiple imputation as method for imputing missing values.
  • Univariate Logistic Regression
  • Multivariate Logistic Regression (Stepwise; Entry = 0.2; Stay = 0.25)

Slide 30

Slide 30 Image

Results

  • The most common risk factor identified during 16 of the 18 different iterations was postoperative admission within 30 days.
    • Indicative of need for hospitalization for postoperative wound treatment.
  • Next most common risk factor was history of MRSA (identified 7 times) followed by postop hematocrit (6 times), number of procedures (5 times), surgery duration (4 times), and BMI and postop hemoglobin (3 times each).
  • Risk factors were procedure-specific.

Slide 31

Slide 31 Image

Results
Procedure Specific Risk Factors

CABG:

  • BMI, duration of stay, postop admission, MRSA.

Hernia:

  • Postop admission.

THA:

  • Lung diagnosis, emergency surgery, Postop admission, duration of surgery.

TKA:

  • Male gender; MRSA; Number of Procedures; Postop Admission; Preop Hematocrit.

Slide 32

Slide 32 Image

Lessons

  • Larger datasets (NHSN) more appropriate for this
    • Mandatory field entry important.
    • New respect gained for NHSN SIR!
  • Validation of electronic variables that are available.
    • Garbage in–garbage out phenomenon.
  • Risk stratification models should weigh relative contribution of each risk, not just yes/no.
  • Need to distinguish manifestation of SSI vs risk factor.
  • Probably should focus only on nonmodifiable risk factors (eg, obesity) and not surgeon factors (duration of surgery).

Slide 33

Slide 33 Image

Implications

  • CMS will be collecting Surgical Site Infection (SSI) data on two surgical procedure categories including colon and abdominal hysterectomy via NHSN for the FY 2014 payment determination.
  • Appropriate risk stratification will be critical to determine fair reimbursement and to prevent “cream-skimming.”

Slide 34

Slide 34 Image

Acknowledgements

Denver Health:

  • Susan Moore, Connie Price, Walt Biffl, Josh Durfee

Salt Lake VA:

  • Mike Rubin, Makoto Jones, Matt Samore

Intermountain:

  • Lucy Savitz, Jason Scott, Scott Evans, Jef Huntington, Pat Nechodom

Vail Valley:

  • Heather Gilmartin

CDC:

  • Sandra Berrios-Torres, Jonathan Edwards, Teresa Horan

AHRQ:

  • Kendall Hall

Slide 35

Slide 35 Image

Extra Slides

Slide 36

Slide 36 Image

Multivariate results: CABG

Variable (CA)EstimateT valueP value
Intercept-7.01035-4.295610.00002
Admission via Transfer0.6164731.4200470.15560
BMI0.0951854.3500310.00001
MRSA1.5655993.0986490.00195
Postop Admission1.2672143.4459550.00057
Preop Hematocrit0.0870341.5799510.12344
Preop Hemoglobin-0.22319-1.516170.14453
Duration of Surgery-0.00477-2.364020.01809

Slide 37

Slide 37 Image

Multivariate Results: HERNIA

Variable (HE)EstimateT valueP value
Intercept-6.53827-4.09980.00007
Number of Procedures0.1745940.9203220.35743
Postop Admission2.7873366.5518380.00000
Postop Hematocrit-0.03547-0.447840.66071
Postop Hemoglobin0.0829220.3313930.74535
Preop Stay0.0366021.8548430.06362
Rheum Diagnosis1.2269351.511360.13070
Wound Class0.4907761.442290.14923

Slide 38

Slide 38 Image

Multivariate Results: THA

Variable (TH)EstimateT ValueP Value
Intercept-4.28495-2.911290.00361
Age-0.02085-1.749050.08028
Lung Diagnosis0.9301472.6457460.00815
Emergency Surgery1.7308284.1820020.00003
Number of Procedures0.4102271.9311151.931115
Postop admission1.7285144.9711040.00000
Postop Hematocrit-0.09155-0.897980.36925
Postop Hemoglobin0.0146340.4646050.64256
Duration of Surgery0.0031862.9976690.00272

Slide 39

Slide 39 Image

Multivariate Results: TKA

Variables (TK)EstimateT valueP value
Intercept-4.99176-4.533870.00001
Gender (Male = 1)0.6234332.5776260.00995
MRSA1.5931274.7390780.00000
Number of Procedures0.5845364.7529730.00000
Postop Admission2.1795047.8730710.00000
Preop Hematocrit-0.05078-2.001260.04697

Slide 40

Slide 40 Image

VariableEstimateT valueP value
Intercept-5.75478-10.65870.00000
Admission via Transfer-0.07527-0.25258-0.25258
History of Cancer0.1864840.3966760.69161
Kidney Diagnosis-1.19328-2.528520.01145
General Anesthesia0.2482131.2773820.20147
Gender (Male = 1)0.2543381.6587060.09718
MRSA1.3476075.7980360.00000
Number of Procedures0.3220133.4946990.00047
Postop Admission2.11117913.108890.00000
Postop Hematocrit-0.03231-2.165880.03045
Preop Stay0.0351452.3158320.02057
Surgery: CA0.4771632.0156530.04384
Surgery: HE0.3768151.6739070.09415
Wound Class0.169831.3001580.19355

Slide 41

Slide 41 Image

Themes

Surgeons' acceptance of current risk models

  • Current risk adjustment models are inadequate
  • More refined models may improve acceptance of data and benchmarks
  • Provider feedback regarding SSI rates and benchmark success rates needs to be timely
  • Further research to identify evidence-based risk factors for SSI are needed

Adoption of electronic surveillance tool by Infection Preventionists

  • Collection of denominator data time consuming , major focus due to mandatory reporting
  • Surveillance for SSI occurs through multiple systems - all require human adjudicated validation - time consuming
  • An E-detection tool with instant notification (via email) of a suspected SSI desired
  • System must be free and adaptable into other systems
Page last reviewed December 2012
Internet Citation: Appendix U. Improving the Measurement of Surgical Site Infection Risk Stratification/Outcome Detection: Slide presentation. December 2012. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/research/findings/final-reports/ssi/ssiapu.html