TeamSTEPPS® Teamwork Perceptions Questionnaire (T-TPQ) Manual
Prepared for:
Center for Quality Improvement and Patient Safety (CQuIPS)
Agency for Healthcare Research and Quality
Patient Safety Program
TRICARE Management Activity
U.S. Department of Defense
Prepared by:
American Institutes for Research
1000 Thomas Jefferson Street, NW
Washington, DC 20007
28 June 2010
Contents
Introduction
Development of the T-TPQ
Literature Review
Item Development
Cognitive Interviews
Small Group Trial
Field Test
Scoring
Administration
Customization of Use
References
Appendix A
Introduction
This document serves as a comprehensive manual for the TeamSTEPPS Teamwork Perceptions Questionnaire (T-TPQ). It describes the development of the T-TPQ and provides detailed instructions for administering the instrument.
Background
During the past 10 years, health care issues including quality of care and patient safety have come under a great deal of focus. In 1999, the Institute of Medicine (IOM) published the report To Err is Human: Building a Safer Health System and brought new public awareness of medical errors. Specifically, the report concluded that medical errors caused as many as 98,000 deaths annually (Kohn, Corrigan, & Donaldson, 1999), The report also described that a critical aspect of patient safety is the ability of health care professionals to perform effectively as teams because the delivery of health care services requires doctors, nurses, pharmacists, and other allied health professionals to coordinate, communicate, and support one another.
In response to the IOM report, the Agency for Healthcare Research and Quality (AHRQ) and the Department of Defense (DoD) have supported research and development activities related to team performance in health care. In 2006, AHRQ and DoD released TeamSTEPPS as the national standard for team training in health care. TeamSTEPPS, which stands for Team Strategies and Tools to Enhance Performance and Patient Safety, is the result of a multiyear research and development project that is based on 25-plus years of research on team performance (Baker, Beaubien, & Holtzman, 2003; Baker, Gustafson, Beaubien, Salas, & Barach, 2003). TeamSTEPPS is a publicly available toolkit that teaches the four core components of teamwork that have been validated in the extant literature (Salas, Sims, & Burke, 2005). These core components are: leadership, communication, mutual support, and situation monitoring. In addition, instruction is provided on how effective teams are structured (i.e., team structure).
Since 2005, TeamSTEPPS has been implemented at numerous military treatment facilities within DoD. Because dissemination and implementation of TeamSTEPPS began nationally and internationally in the civilian sector in November 2006, research on the true effectiveness of this training methodology is still in early stages. To date, only a handful of validated measures to support such evaluation efforts exist. Program evaluation provides information about the most effective TeamSTEPPS components and training formats in conveying the material and producing changes in team performance. The TeamSTEPPS resource kit contains measurement tools; however, these tools have not been validated and are limited to reactions to training, knowledge gains, and observations of team skills.
Kirkpatrick's (1967) model is the most widely used approach to training evaluation and identifies four levels of assessment:
- Level 1: Trainee reactions.
- Level 2: Trainee learning.
- Level 3: Transfer of behavior to the job.
- Level 4: Whether the training produced the desired organizational outcomes.
The model has remained robust over the years with few changes being advocated. Most notably, Kraiger, Ford and Salas (1993) partitioned Kirkpatrick's Level 2 into cognitive, affective, and skill-based learning. Kraiger et al. argued that such a conceptualization provides a more precise understanding of how learning can occur.
Historically, training evaluation has relied solely on trainee reactions, but for evaluation to be thorough it needs to spanKirkpatrick's four levels. The TeamSTEPPS prpgram provides some initial tools, but research on the design, development and validation of new measures needs to continue to provide a suite of resources for studying the impact of team training. Recently, DoD's TRICARE Management Activity presented a new tool for measuring individual attitudes toward teamwork in health care known as the T-TAQ (TeamSTEPPS Teamwork Attitudes Questionnaire). The T-TAQ can be found on AHRQ's Web site at https://www.ahrq.gov/teamstepps/instructor/reference/teamattitude.html. Respondent attitudes measured by the T-TAQ were found to vary as a function of prior team training and past experiences in health care teams (Herrara, Baker, Amodeo, & Slonim, 2009). However, researchers have demonstrated that people differentiate between their perceptions of themselves as individuals and their perceptions of themselves as team members (Brewer and Gardner, 1996). A measure such as the T-TAQ captures how an individual approaches team-related issues but not necessarily how individuals perceive the current state of teamwork within an organization. Measuring perceptions of teamwork offers a broader picture of an organization's team climate. Therefore, a measure of an individual's perception of collective teamwork is needed to capture this unique dimension. Furthermore, the T-TAQ is not adequate for measuring the success of TeamSTEPPS training. A measure of perceptions of overall teamwork would serve as an additional measure of the effectiveness of TeamSTEPPS training.
This manual provides a description of and administration procedures for a new tool, the TeamSTEPPS Teamwork Perceptions Questionnaire (T-TPQ), which measures individuals' perception of group-level team skills and behavior. Unlike most behavioral skill measures that require direct observation by independent and trained observers, the T-TPQ is a self-report measure of teamwork within a unit or department. Like the T-TAQ, the T-TPQ is based upon the core components of teamwork that comprise TeamSTEPPS: team structure, leadership, communication, mutual support, and situation monitoring.
Development of the T-TPQ
Literature Review
Searches of the relevant literature revealed that few measures exist that provide assessments of individuals’ perceptions of toward teamwork, particularly ones oriented toward health care. Furthermore, none are aligned with the core components of teamwork (Salas, Sims, & Burke, 2005) that are the basis for the TeamSTEPPS program. For example, in aviation, the Cockpit Management Attitudes Questionnaire (Helmreich, 1984; Gregorich, Helmreich, & Wilhelm, 1990) assesses constructs related to crew resource management (CRM), including leadership, coordination, and communication. In health care, the Safety Climate Survey measures perceptions of organizational commitment to patient safety through constructs such as commitment to safety, leadership, interpersonal interactions, attitudes toward stress, and knowledge of how to report adverse events (Sexton, et al., 2006). The Safety Attitudes Questionnaire measures hospital providers’ attitudes about teamwork climate, safety climate, perceptions of management, job satisfaction, working conditions, and stress recognition (Sexton & Thomas, 2003).
Perhaps the most germane measure to the current discussion is AHRQ’s Hospital Survey on Patient Safety (HSOPS). HSOPS measures 12 dimensions related to patient safety culture. Two of these scales focus specifically on teamwork—teamwork within units and teamwork between units. HSOPS has been administered at hundreds of institutions, and national norms are available. TeamSTEPPS recommends institutions consider using HSOPS as part of their site assessment for determining their teamwork needs and as an evaluation tool to determine whether HSOPS scores improve as a function of TeamSTEPPS implementation. However, the HSOPS focuses on safety culture, and the teamwork scales do not partition out the critical subdomains of team performance. Therefore, our review of the literature supported the need for the development of the T-TPQ.
Item Development
A pool of survey items was developed through an extensive item-writing process that included multiple item writers who were experienced in survey and test development and knowledgeable in the principles of teamwork and, more specifically, the TeamSTEPPS training curriculum on which the items were based. As noted previously, we determined that the T-TPQ would focus on the four core components of teamwork that have been validated in the extant literature (Salas et al., 2005) in addition to team structure. As items were drafted, writers linked each item to a specific TeamSTEPPS curriculum module and identified the page in the manual from which the item was written. Throughout the item-writing process, this linking ensured that items captured key teamwork concepts. Item-writing efforts resulted in a pool of 93 items.
We then conducted an item review on the item pool. Review criteria included addressing social desirability concerns and ensuring that items tapped critical concepts from the TeamSTEPPS curriculum. In addition, we reviewed items to ensure that they asked the respondent to assess aspects of team performance that required a transfer of strategies, skills, or tools to evaluate change over time. Similarly, we reviewed and revised items, as necessary, so that they inquired about aspects of team performance that any respondent could assess. Finally, further reviews sought to eliminate items that were repetitive in terms of teamwork concepts and to limit the number of items within each of the constructs to approximately 10 items. Following these reviews, 50 items remained for inclusion in the first version of the T-TPQ.
The 50-item T-TPQ was tested and refined during three critical events: cognitive interviews, small group trial, and field test. Below we describe each of these activities and then report the results and describe any modification made to the T-TPQ.
Cognitive Interviews
Method. Cognitive interviews were conducted on the pilot version of the T-TPQ with members of a unit within an urban, civilian Northeastern hospital in October 2008. Nine individuals were identified and invited to participate. These individuals were nurses and physicians with experience in the unit. Six individuals participated: five nurses and one physician. Cognitive interviews helped to identify items that required editing or deletion due to ambiguity, confusion, or misinterpretation by the respondent(s). In addition, as the participants provided their responses to each item, issues with the proposed response options were identified and recorded. The goals of the cognitive interviews were to provide assurance that survey instructions and item wording were clear, that respondents were able to provide perceptions of their team as a whole, and the response options were clear and appropriate for each item.
Results. Based on the results of these interviews, the T-TPQ was revised. Several items were reworded to clarify terminology that was consistently unclear to the respondents. Three of the 50 items were deleted because they were difficult for respondents to understand.
Small Group Trial
Method. A revised version of the T-TPQ was administered to nine nurses within a single pediatric intensive care unit at a Southeastern children’s hospital. In addition to the 47 items included on the T-TPQ, 8 items were added from the HSOPS for comparison. These items represented two scales from the previously validated HSOPS, your work area/unit and your supervisor/manager. The purpose of this pilot test was to determine if members of the same unit produced similar ratings (i.e., agreement) when using the T-TPQ. It was expected that for the T-TPQ to be valid and reliable, staff from the same unit should rate items on the T-TPQ similarly. Moreover, items that yield low levels of agreement are perceived to be either poorly written or difficult to observe and therefore should be discarded from the final version of the questionnaire. Finally, we compared agreement levels on the new T-TPQ items to those items from the validated HSOPS to see if the new T-TPQ items were comparable.
Results. Agreement among respondents was reviewed by comparing each individual’s responses in two ways. First, we collapsed the 5-point rating scale (ranging from strongly agree to strongly disagree) into a 3-point rating scale (i.e., agree, neutral, disagree) and determined the percentage of respondents who gave the same rating. Second, we performed the same analysis using the 5-point scale (i.e., a more conservative estimate of agreement). In both cases, we examined agreement for the day and night shifts separately because many staff reported significant differences in how each of these shifts functions within the unit. Finally, we calculated average agreement across shifts for each scale type (3- and 5-point) and an overall average rating that was a combination of the 3-point and 5-point averages (a moderately conservative estimate of agreement).
Overall agreement ranged from a high of 100 percent to a low of 25 percent. Thirty-one of the 47 T-TPQ items had agreement levels in excess of 70 percent, while 8 items had agreement levels of less than 50 percent. As a point of comparison, all but two items from the HSOPS had agreement levels in excess of 70 percent. Despite the lower-than-desired agreement levels for some items, all items were included in the field test.
Field Test
Method. The final component of this study was administration of the T-TPQ to 169 health care workers who completed the TeamSTEPPS team training program. These data, in combination with the small group trial, were used to select the final items for the T-TPQ, to determine subscale reliabilities, and provide preliminary validation evidence by including four items from the HSOPS design to measure teamwork within a hospital unit.
Results. As noted, the validation sample consisted of 169 participants. Of the 169, 73.4 percent were direct patient care providers. The largest subgroup was nurses (32.6 percent). Table 1 illustrates the length of time the participants worked at the hospital and within their unit. Analyses were conducted to identify cases with excessive missing data and anomalous response patterns. All cases met the criteria for inclusion (i.e., 5 percent or fewer items with missing data, where multiple responses were coded as missing).
Classical item statistics were used to select the final T-TPQ items. Means, standard deviations, and item-total correlations were computed. Items and constructs were reviewed using these analyses, in addition to Cronbach’s Alpha "if item deleted" results. The number of items per construct and overall length of the measure were also taken into account. This process resulted in the deletion of 16 items.
The final T-TPQ (Appendix A) includes 35 items (7 items measuring each of the constructs described previously). Final constructs and their associated scale reliabilities are provided in Table 2. Construct independence was also examined by intercorrelating the five T-TPQ subscales (Table 3). Coefficients ranged from .57 (team structure and communication) to .79 (situation monitoring and mutual support). These results suggest some multicollinearity but allow for the assessment of unique variance in each subscale.
Lastly, convergent validity with the HSOPS your work area/unit subscale was demonstrated (Table 4). The T-TPQ correlation coefficient with HSOPS was .81. T-TPQ subscale correlations with the HSOPS ranged from .60 (communication) to .79 (mutual support).
Table 1. Participant Time Employed in the Hospital and Unit
Length of Time Employed | Participants (%) Employed in Hospital |
Participants (%) Employed in Unit |
---|---|---|
Less than 1 year | 25.4% | 32.0% |
1-5 years | 40.8% | 37.3% |
6-10 years | 14.2% | 16.6% |
11-15 years | 8.3% | 6.5% |
16-20 years | 6.5% | 4.7% |
21 years or more | 4.7% | 3.0% |
Table 2. T-TPQ Cronbach’s Alpha Reliability Coefficients
Construct | Number of Items | Cronbach’s Alpha |
---|---|---|
Team Structure | 7 | .89 |
Leadership | 7 | .95 |
Situation Monitoring | 7 | .91 |
Mutual Support | 7 | .90 |
Communication | 7 | .88 |
Table 3. T-TPQ Construct Intercorrelations
Construct | Team Structure | Leadership | Situation Monitoring | Mutual Support | Communication |
---|---|---|---|---|---|
Team Structure | 1.00 | .62* | .77* | .64* | .57* |
Leadership | 1.00 | .68* | .70* | .62* | |
Situation Monitoring | 1.00 | .79* | .70* | ||
Mutual Support | 1.00 | .77* | |||
Communication | 1.00 | ||||
N | 169 | 169 | 169 | 169 | 169 |
*p < .01, two-tailed.
Table 4. T-TPQ Correlations with HSOPS Your Work Area/Unit Scale
Construct | HSOPS |
---|---|
T-TPQ | .81* |
Team Structure | .64* |
Leadership | .74* |
Situation Monitoring | .73* |
Mutual Support | .79* |
Communication | .60* |
N | 169 |
*p < .01, two-tailed.
Scoring
T-TPQ scoring can take one of two forms. First, a total score may be calculated for each teamwork construct. Summing scores in this manner allows for more accurate statistical testing. This is the preferred method for performing data analyses. A second way is to compute an average score for each construct. This method is more appropriate for graphical representations or when presenting data in a table (Table 5). However, if comparisons between multiple administrations of the T-TPQ are to be done, it is necessary for scoring methods to be consistent at each administration.
Table 5. Construct-Level Means for Final T-TQP Items
Construct | N | Minumum | Maximum | Mean | Std. Deviation | Variance |
---|---|---|---|---|---|---|
Team Structure | 169 | 1.57 | 5.00 | 3.63 | .859 | .737 |
Leadership | 169 | 1.00 | 5.00 | 3.69 | .936 | .875 |
Situation Monitoring | 169 | 1.57 | 5.00 | 3.59 | .754 | .568 |
Mutual Support | 169 | 1.71 | 5.00 | 3.66 | .753 | .567 |
Communication | 169 | 1.86 | 5.00 | 3.81 | .639 | .408 |
Overall | 169 | 1.71 | 5.00 | 3.67 | .681 | .463 |
N | 169 |
Administration
The T-TPQ may be administered as a standalone measure or in conjunction with other measures of team skills and safety culture (e.g., T-TAQ, HSOPS). The T-TPQ can also be administered for several purposes, such as to assess perceptions toward the core components of teamwork, as part of an institution’s site assessment to determine training needs, or as a tool to evaluate the effectiveness of TeamSTEPPS training. In all of these cases, the basic administration of the T-TPQ should remain the same. What will vary, as noted in the administration options below, is when the T-TPQ is administered.
Administration Considerations. Researchers have noted that changes in scores on a given measure can occur on multiple planes (Golembiewski, Billingsley, & Yeager, 1976). Change can occur as a function of varying levels of performance on a given set of criteria, such as higher levels of teamwork that result from TeamSTEPPS training. This type of change would be reflected in an increase in scores from a pretest (before TeamSTEPPS training) to a posttest (after training). However, change can also occur as the result of a shift in how participants perceive a construct to be defined (Howard & Dailey, 1979), such as the change in participants’ definition of teamwork as a result of TeamSTEPPS training (e.g., participants take T-TPQ with an understanding of what teamwork is, then complete TeamSTEPPS training and realize they really did not understand teamwork, retake the T-TPQ after training, and score lower as the result of adjusting their definition of teamwork). The administration of the T-TPQ (pretest only, posttest only, or both) and the interpretation of results (change in performance or change in definition of teamwork) should take this into serious consideration. For this reason, those who plan to administer the T-TPQ as a pretest and posttest measure should take a longitudinal approach. Along with the pretest and posttest, we recommend administering a followup posttest to capture any multidimensional changes that may be reflected in the scores.
As an independent assessment of teamwork perceptions, the T-TAQ may be administered at any point in time. In this capacity, the T-TPQ may be administered organization-wide, unit-wide, or to some combination of units. Careful consideration should be given to identifying the population of interest to be surveyed so the correct staff members receive the T-TPQ.
TeamSTEPPS Site Assessment. The T-TPQ may be used as one component of the TeamSTEPPS site assessment process. In this capacity, the T-TPQ should be administered prior to TeamSTEPPS training. Results can be used to identify where less-than-desirable perceptions toward teamwork exist within a unit or institution and can therefore assist the organization’s TeamSTEPPS change team in selecting specific TeamSTEPPS interventions.
TeamSTEPPS Evaluation. The T-TPQ may be used to assess TeamSTEPPS effectiveness. The basic question answered is whether the TeamSTEPPS intervention produces desirable changes in perceptions regarding teamwork. To answer this question, the T-TPQ should be administered immediately before and after TeamSTEPPS training. Results from these two data collections should then be compared. We also recommend that the T-TPQ be administered several months after TeamSTEPPS training to see if the changes in perception that were achieved at the end of training are sustained or changed. When using the T-TPQ as both a pre- and post-training measure, please refer to the special considerations mentioned above.
Customization of Use
Organizations may have different needs with regard to teamwork development. This section briefly describes appropriate options for customization of the T-TPQ.
Item Modification. Items on the T-TPQ should not be modified. Changing the items can affect the reliability and validity of the instrument.
Scale Use. Scales from the instrument can be used separately. For example, if an organization were interested only in perceptions of leadership, then this scale could be administered independently.
References
Alonso, A., Baker, D., Day, R., Holtzman, A., King, H., Toomey, L., & Salas E. (2006). Reducing medical error in the military health system: How can team training help? Human Resources Management Review 16, 396-415.
Baker, D. P., Beaubien, J. M., & Holtzman, A. K. (2003). DoD medical team training programs: An independent case study analysis. Washington, DC: American Institutes for Research.
Baker, D. P., Gustafson, S., Beaubien, J. M., Salas, E., & Barach, P. (2003). Medical teamwork and patient safety: The evidence-based relation. Washington, DC: American Institutes for Research.
Baker, D., Lambert, S., & Slonim, A. (2009, May). Implementing TeamSTEPPS in pediatrics: Lessons learned and guidelines. Poster presented at the Annual Congress of the National Patient Safety Foundation. Washington, DC.
Brewer, M. B., & Gardner, W. (1996). Who is the "we"? Levels of collective identity and self representations. Journal of Personality and Social Psychology 71, 83-93.
Golembiewski, R. T., Billingsley, K., & Yeager, S. (1976). Measuring change and persistence in human affairs: Types of change generated by OD designs. Journal of Applied Behavioral Science 12, 133-157.
Gregorich, S. E., Helmreich, R. L., & Wilhelm, J. A. (1990). The structure of cockpit management attitudes. Journal of Applied Psychology 75, 682-690.
Helmreich, R. L. (1984). Cockpit management attitudes. Human Factors 26, 583-589.
Herrera, H., Baker, D. P., Amodeo, A. M., & Slonim, A. (2009, April). Positive teamwork attitudes by pediatric intensivists are thought to improve patient safety in critical care. Paper presented at the 11th Annual Conference for the Society of Critical Care Medicine, San Juan, Puerto Rico.
Howard, G. S., & Dailey, P. R. (1979). Response-shift bias: A source of contamination of self-report measures. Journal of Applied Psychology 64(2), 144-150.
Kirkpatrick, D. L. (1967). Evaluation of training. In R. L. Craig (Ed.). Training and development handbook: A guide to human resource development (18.1-18.27). New York: McGraw-Hill.
Kohn, L. T., Corrigan, J. M., & Donaldson, M. S. (1999). To err is human: Building a safer health system. Washington, DC: National Academy Press.
Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning to new methods of training evaluation. Journal of Applied Psychology 78(2), 311-328.
Salas, E., Sims, D. E., & Burke, C. S. (2005). Is there a big 5 in teamwork? Small Group Research 36(5), 555-599.
Sexton J. B., Helmreich, R. L., Neilands, T. B., Rowan, K., Vella, K., Boyden, J., et al. (2006). The safety attitudes questionnaire: Psychometric properties, benchmarking data, and emerging research. BMC Health Services Research, 6(44).
Sexton, J. B., & Thomas, E. J. (2003). The safety climate survey: Psychometric and benchmarking properties (Technical Report 03-03). Houston, TX: The University of Texas Center of Excellence for Patient Safety Research and Practice.
Appendix A: TeamSTEPPS Teamwork Perceptions Questionnaire
Instructions: Please complete the following questionnaire by placing a check mark [√] in the box that corresponds to your level of agreement from Strongly Agree to Strongly Disagree. Please answer every question, and select only one response for each question. The questionnaire is anonymous, so please do not put your name or any other identifying information on the questionnaire.
Team Structure | Strongly Agree | Agree | Neutral | Disagree | Strongly Disagree |
---|---|---|---|---|---|
1. The skills of staff overlap sufficiently so that work can be shared when necessary. | |||||
2. Staff are held accountable for their actions. | |||||
3. Staff within my unit share information that enables timely decision making by the direct patient care team. | |||||
4. My unit makes efficient use of resources (e.g., staff supplies, equipment, information). | |||||
5. Staff understand their roles and responsibilities. | |||||
6. My unit has clearly articulated goals. | |||||
7. My unit operates at a high level of efficiency. | |||||
Leadership | Strongly Agree | Agree | Neutral | Disagree | Strongly Disagree |
8. My supervisor/manager considers staff input when making decisions about patient care. | |||||
9. My supervisor/manager provides opportunities to discuss the unit’s performance after an event. | |||||
10. My supervisor/manager takes time to meet with staff to develop a plan for patient care. | |||||
11. My supervisor/manager ensures that adequate resources (e.g., staff, supplies, equipment, information) are available. | |||||
12. My supervisor/manager resolves conflicts successfully. | |||||
13. My supervisor/manager models appropriate team behavior. | |||||
14. My supervisor/manager ensures that staff are aware of any situations or changes that may affect patient care. | |||||
Situation Monitoring | Strongly Agree | Agree | Neutral | Disagree | Strongly Disagree |
15. Staff effectively anticipate each other’s needs. | |||||
16. Staff monitor each other’s performance. | |||||
17. Staff exchange relevant information as it becomes available. | |||||
18. Staff continuously scan the environment for important information. | |||||
19. Staff share information regarding potential complications (e.g., patient changes, bed availability). | |||||
20. Staff meets to reevaluate patient care goals when aspects of the situation have changed. | |||||
21. Staff correct each other’s mistakes to ensure that procedures are followed properly. | |||||
Mutual Support | Strongly Agree | Agree | Neutral | Disagree | Strongly Disagree |
22. Staff assist fellow staff during high workload. | |||||
23. Staff request assistance from fellow staff when they feel overwhelmed. | |||||
24. Staff caution each other about potentially dangerous situations. | |||||
25. Feedback between staff is delivered in a way that promotes positive interactions and future change. | |||||
26. Staff advocate for patients even when their opinion conflicts with that of a senior member of the unit. | |||||
27. When staff have a concern about patient safety, they challenge others until they are sure the concern has been heard. | |||||
28. Staff resolve their conflicts, even when the conflicts have become personal. | |||||
Communication | Strongly Agree | Agree | Neutral | Disagree | Strongly Disagree |
29. Information regarding patient care is explained to patients and their families in lay terms. | |||||
30. Staff relay relevant information in a timely manner. | |||||
31. When communicating with patients, staff allow enough time for questions. | |||||
32. Staff use common terminology when communicating with each other. | |||||
33. Staff verbally verify information that they receive from one another. | |||||
34. Staff follow a standardized method of sharing information when handing off patients. | |||||
35. Staff seek information from all available sources. |