2011 User Comparative Database Report
Notes: Description of Data Cleaning and Calculations
This notes section provides additional detail regarding how various statistics presented in this report were calculated.
Each participating hospital was asked to submit cleaned individual-level survey data. However, as an additional check, once the data were submitted, response frequencies were run on each hospital's data to look for out-of-range values, missing variables, or other data anomalies. When data problems were found, hospitals were contacted and asked to make corrections and resubmit their data. In addition, each participating hospital was sent a copy of its data frequencies to verify that the dataset received was correct.
In order to keep the database current, data more than 3.5 years old are removed from the database. Thus, 117 hospitals that administered the survey prior to January 1, 2007, were dropped from the database.
As part of the data submission process, hospitals were asked to provide their response rate numerator and denominator. Response rates were calculated using the formula below.
Response Rate = Number of complete, returned surveys
Number of surveys distributed − Ineligibles
Numerator = Number of complete, returned surveys. The numerator equals the number of individual survey records submitted to the database. It should exclude surveys that were returned blank on all nondemographic survey items but include surveys where at least one nondemographic survey item was answered.
Denominator = The total number of surveys distributed minus ineligibles. Ineligibles include deceased individuals and those who were not employed at the hospital during data collection.
As a data cleaning step, we examined whether any individual survey records submitted to the database were missing responses on all of the nondemographic survey items (indicating the respondent did not answer any of the main survey questions). Records where all nondemographic survey items were left blank by the respondent were found (even though these blank records should not have been submitted to the database). We therefore removed these blank records from the larger dataset and adjusted any affected hospital's response rate numerator and overall response rate accordingly.
Calculation of Percent Positive Scores
Most of the survey's items ask respondents to answer using 5-point response categories in terms of agreement (Strongly agree, Agree, Neither, Disagree, Strongly disagree) or frequency (Always, Most of the time, Sometimes, Rarely, Never). Three of the 12 patient safety culture composites use the frequency response option (Feedback and Communication About Error, Communication Openness, and Frequency of Events Reported), while the other 9 composites use the agreement response option.
Item-Level Percent Positive Response
Both positively worded items (such as "People support one another in this work area") and negatively worded items (such as "We have patient safety problems in this work area") are included in the survey. Calculating the percent positive response on an item is different for positively and negatively worded items:
For positively worded items, percent positive response is the combined percentage of respondents within a hospital who answered "Strongly agree" or "Agree," or "Always" or "Most of the time," depending on the response categories used for the item.
For example, for the item "People support one another in this work area," if 50 percent of respondents within a hospital Strongly agree and 25 percent Agree, the item-level percent positive response for that hospital would be 50% + 25% = 75% positive.
For negatively worded items, percent positive response is the combined percentage of respondents within a hospital who answered "Strongly disagree" or "Disagree," or "Never" or "Rarely," because a negative answer on a negatively worded item indicates a positive response.
For example, for the item "We have patient safety problems in this work area," if 60 percent of respondents within a hospital Strongly disagree and 20 percent Disagree, the item-level percent positive response would be 80 percent positive (i.e., 80 percent of respondents do not believe they have patient safety problems in their work area).
Composite-Level Percent Positive Response
The survey's 42 items measure 12 areas, or composites, of patient safety culture. Each of the 12 patient safety culture composites includes 3 or 4 survey items. Composite scores were calculated for each hospital by averaging the percent positive response on the items within a composite. For example, for a three-item composite, if the item-level percent positive responses were 50 percent, 55 percent, and 60 percent, the hospital's composite-level percent positive response would be the average of these three percentages, or 55 percent positive.ix
Item and Composite Percent Positive Scores
To calculate your hospital's composite score, simply average the percentage of positive response to each item in the composite. Here is an example of computing a composite score for Overall Perceptions of Patient Safety:
- There are four items in this composite—two are positively worded (items A15 and A18) and two are negatively worded (items A10 and A17). Keep in mind that disagreeing with a negatively worded item indicates a positive response.
- Calculate the percentage of positive responses at the item level. (Go to example in Table 1).
In this example, there were four items with percent positive response scores of 46 percent, 52 percent, 46 percent, and 56 percent. Averaging these item-level percent positive scores results in a composite score of .50, or 50 percent, on Overall Perceptions of Patient Safety. In this example, an average of about 50 percent of the respondents responded positively to the survey items in this composite.
Once you calculate your hospital's percent positive response for each of the 12 safety culture composites, you can compare your results with the composite-level results from the 1,032 database hospitals.
Minimum Number of Responses
Beginning with the 2010 database report, we enacted several new rules regarding a minimum number of responses for calculating the percent positive scores. First, we calculated percent positive scores only for hospitals that had at least 10 completed surveys. Second, item-level results were calculated only when there were at least three responses to the item. If a hospital had fewer than three responses to a survey item, the hospital's score for that item was set to missing. Third, if a hospital had fewer than five respondents in a breakout category (e.g, work area/unit, staff position, direct interaction with patients), then no statistics were calculated for that breakout category (i.e., all scores were set to missing). For example, if a hospital had five respondents indicating they worked in the Anesthesiology unit and four respondents indicating they worked in Pharmacy, that hospital would be included in the statistics displayed for Anesthesiology units but not in those displayed for Pharmacy units. These minimums also apply to the statistics displayed in Appendixes B and D (results by respondent characteristics).
Percentiles were computed using the SAS® Software default method. The first step in this procedure is to rank order the percent positive scores from all the participating hospitals, from lowest to highest. The next step is to multiply the number of hospitals (n) by the percentile of interest (p), which in our case would be the 10th, 25th, 50th, 75th, or 90th percentile.
For example, to calculate the 10th percentile, one would multiply 1,032 (the total number of hospitals) by .10 (10th percentile). The product of n x p is equal to j+g, where j is the integer and g is the number after the decimal. If g equals 0, the percentile is equal to the percent positive value of the hospital in the jth position plus the percent positive value of the hospital in the jth +1 position, divided by 2 [(X(j) + X(j+1))/2]. If g is not equal to 0, the percentile is equal to the percent positive value of the hospital in the jth +1 position.
The following examples show how the 10th and 50th percentiles would be computed using a sample of percent positive scores from 12 hospitals (using fake data shown in Table 2). First, the percent positive scores are sorted from low to high on Composite "A."
For the 10th percentile, we would first multiply the number of hospitals by .10:
(n x p = 12 x .10 = 1.2).
- The product of n x p = 1.2, where j = 1 and g = 2. Since g is not equal to 0, the 10th percentile score is equal to the percent positive value of the hospital in the jth +1 position:
- j equals 1.
- The 10th percentile equals the value for the hospital in the 2nd position = 48%.
For the 50th percentile, we would first multiply the number of hospitals by .50:
(n x p = 12 x .50 = 6.0).
- The product of n x p = 6.0, where j = 6 and g = 0. Since g = 0, the 50th percentile score is equal to the percent positive value of the hospital in the jth position plus the percent positive value of the hospital in the jth +1 position, divided by 2:
- j equals 6.
- The 50th percentile equals the average of the hospitals in the 6th and 7th positions (64%+66%)/2 = 65%.
ix This method for calculating composite scores differs slightly from the method described in the September 2004 Survey User's Guide that is part of the original survey toolkit materials on the AHRQ Web site. The guide advises computing composites by calculating the overall percent positive across all the items within a composite. The updated recommendation included in this report is to compute item percent positive scores first, and then average the item percent positive scores to obtain the composite score, which gives equal weight to each item in a composite. The Survey User's Guide will eventually be updated to reflect this slight change in methodology.