Establishing an AHRQ Learning Collaborative, A White Paper
Developing an AHRQ Learning Collaborative
This section describes the steps taken to gather input from participants about the potential for an AHRQ Learning Collaborative and summarizes the findings from that information-gathering process. There were three main components of this information gathering process: conference calls with interested program directors; a survey of predoctoral trainees, postdoctoral trainees, and program directors; and interviews with a convenience sample of predoctoral trainees, postdoctoral trainees, program directors, and expert consultants.
To assess the level of interest in the existence of an AHRQ Learning Collaborative, AHRQ T32 program directors were contacted and provided with a basic overview of the concept. Interested program directors were then invited to participate in a series of conference calls to move this initiative forward.
The first conference called laid the foundation for this effort and served as a brainstorming opportunity for what such a Collaborative might look like, what the goals of this initiative would be, and how information would be gathered to inform the process. Through the conference calls, a plan to distribute a short survey to AHRQ T32 programs was developed, and the process of interviewing program directors, trainees, and external experts was recommended. These venues provided an opportunity to present the survey results and offer additional guidance for the information-gathering activities going forward.
As a first step to gathering input for the proposed AHRQ Learning Collaborative, a short survey was distributed via Email. The Email was sent to all program directors at AHRQ T32 funded training programs who expressed interest in the AHRQ Learning Collaborative. This included 18 of the 28 Institutional Health Services Research Training Programs (T32s). Go to Appendix A for a complete listing of the training programs. We received 24 responses, including feedback from 8 program directors, 14 predoctoral trainees, and two postdoctoral trainees. Responses came from 10 of the surveyed training institutions.
Respondents recommended three key functions for an AHRQ Learning Collaborative: sharing, collaborating, and networking. In particular, some of the specific areas for sharing included: curricula, training opportunities, methods, best practices, ideas and information, career resources, data sources, and presentation opportunities. Collaboration on research projects and publications was identified by several respondents as a potential function of the collaborative. Lastly, the prospect of developing relationships among students, alumni, and fellow researchers was seen as a way to network, seek mentorship, enhance communication about the exchange of ideas and future opportunities, and perhaps acquire publications.
Related to the functions of the AHRQ Learning Collaborative, several issues and opportunities were raised as potential topics to be addressed by the Collaborative. Some of the broad themes that were mentioned included curricula/coursework, methods (e.g., mixed methods, qualitative), research interests, the current deficits in programs (based on trainees' perspectives), specific research topics in health services research (e.g., quality of care, disparities, changing health care organizational structures, accountable care organizations, and rural health), theory, health data resources, dissertation development and support, and career opportunities. In addition, the importance of networking and social networking was highlighted through opportunities for communication and mentoring. Lastly, some specific areas of interest were highlighted by respondents, such as funding for international students, a greater focus on informatics and computer science, creating meaningful practicum experiences in public health, and funding NRSA trainees beyond NRSA.
The recommended target audience varied, with many respondents suggesting a broad audience inclusive of pre- and postdoctoral trainees, program directors, and faculty. However, some saw the audience as more focused on trainees only (more common) or program directors only (less common).
The respondents described several topic areas that they would be interested in learning about; these fell into one of four broad domains: other training programs, activities, opportunities, and specific issues. Several details of other programs were mentioned, including: the structure of program requirements, curricula, the scope of the program (e.g., domestic or international), types of students, the program's financial model, methods, research interests of faculty, approaches to teaching, placement following completion of the program, and names and contact information for current trainees. In addition, respondents were interested in hearing about the activities of the different programs, with a particular focus on student projects and research. Opportunities for fellowships, jobs, research, and data were highlighted, as well as the potential for other institutions to learn about well-qualified graduates looking for postdoctoral positions. Lastly, several specific issues were highlighted, including how to balance the need for both a broad exposure to a variety of topics with the need for specialized knowledge and learning about different career trajectories in health services research.
When asked about potential innovations or best practices at their own institutions to share through the Collaborative, respondents mentioned specific program details, areas of emphasis, activities, and content expertise. In particular, some of the program details that respondents would be interested in sharing about their own programs included: course materials, syllabi, presentation slides, a learning objective matrix, structured timelines for students, program handbooks, and health services research doctoral competencies. In addition, several areas of emphasis were noted for sharing, including the unique experience of a health program within a degree of applied economics, interdisciplinary research methods and statistics, the use of large databases for health services research, organization theory and behavior, and multidisciplinary approaches to health services research.
Participants also described specific activities at their own institutions that they would share with a broader collaborative, including multiple mechanisms for in-person sessions with trainees; incorporating mentoring, career development, and broad health services research content areas; and inviting other scholars, faculty, outside speakers, and trainees from different parts of the university; bringing trainees into ongoing research projects, out of which they can carve their own project; a coordinated course tutorial program that results in all trainees writing an article by the end of their first year in the program; and a formal system to track evaluations of fellows, mentors, and the program as a whole.
There was a broad spectrum of recommendations for potential mechanisms to be used through the Collaborative. Specifically, most respondents emphasized some Web-based communication mechanisms, some suggested only Web-based communication, some suggested coupling it with NRSA and no other session, and some suggested more in-person events in addition to NRSA, possibly regionally. Specific forms of Web-based communication that were suggested included: Web pages, podcasts, blogs, e-forums, social media (e.g., Twitter, Facebook, and Ning), and annual webcasts. In-person events included with NRSA, separate from NRSA, local or regional, annual, roundtable events, and workshops. Conference calls were also recommended as a form of communication. Tools recommended for sharing were predominantly Web-based and included lectures via the Internet (e.g., webinars), Web-based sharing of large databases, and a Web-based system to share information about trainees' research projects, including the topics and methods used.
Lastly, participants were asked to suggest some existing learning collaboratives and best practices that might serve as a potential model for this effort. The collaboratives highlighted included collaboration among contractors at the Centers for Medicare & Medicaid Services (CMS), the Massachusetts Institute of Technology (MIT) Center for Biomedical Innovation (MIT CBI), the Nonprofit Technology Network (Nten.org), a collaborative model around the burden of disease project, collaboration across Centers for Education & Research on Therapeutics (CERTs) programs, the Institute for Healthcare Improvement (IHI), and the patient-centered medical home transformation process, where doctors and practice leaders come together quarterly to share experiences. In addition, several best practices were noted, including the participation of alumni on committees to offer a helpful perspective after leaving the program; offering standardized open-ended evaluations for all courses and other programs, and consideration for using this approach across programs; and webinars and other forms of interactive online learning.
To gather insight into the potential for the AHRQ Learning Collaborative and to explore possible structures, topics, and mechanisms for the Collaborative, we conducted several semi-structured interviews with participants in the AHRQ training programs. Specifically, five program directors, five postdoctoral trainees, and five predoctoral trainees were interviewed. In addition, we also sought feedback from experts at AcademyHealth, AHRQ, IHI, the NEWMEDS Project, and The PCMH/MU Collaborative. The latter interviews have been incorporated throughout this paper, and a sample interview protocol is included in Appendix D.
Creating Value for Trainees
Through speaking with trainees, it became clear that they view the collaborative as a tremendous opportunity for augmenting the learning process at their home institutions. We grouped trainees' rich and creative ideas for the collaborative into the following three categories: sharing, building relationships, and developing an identity. The categories appear to overlap greatly with Wenger and Snyder's three characteristics of communities of practice.14
Trainees were curious about how training differs across programs and were concerned in particular about where the programs stood with respect to: training in research methods, knowledge of secondary data and analysis techniques, and overall competence in health services research.
A predoctoral trainee noted:
I think [a collaborative is] definitely valuable, especially when it comes to the nitty gritty—hot methods in this field, are we getting all that we need to be competitive in this field, and if not, where else can we get that?
Another trainee (postdoctoral) remarked:
[It is] really important to share ideas across institutions... Great to be able to see what other trainees are doing across the country, get research ideas, share research ideas, meet people who have similar research ideas.
Trainees were specifically interested in understanding methodologies offered in different programs and were curious about what methods gaps were not being filled by existing methods workshops. This interest carried over into knowledge of (and access to) secondary data and analysis techniques where trainees believed the collaborative could serve a problem-solving function. One postdoctoral trainee noted that the collaborative could, "Have students who are struggling with the same data problems communicate with each other."
Many trainees also believed that getting a firmer sense of the training process and knowledge base across programs could help to shed further light on their program's strengths as well as to identify areas of their program that are underdeveloped.
For example, one predoctoral trainee at the 2011 NRSA Conference remarked:
My school is weak at bridging methods to practice. I would prefer to work with a faculty member at another school with T32/NRSA traineeships. I wish this process was more flexible.
Another (postdoctoral) trainee remarked:
One starting point would be to know what other people with these AHRQ fellowships do—I don't really know. I don't really know what people tend to study on these fellowships. That would be part of this idea about sharing information.
And another predoctoral trainee at the 2011 NRSA conference suggested:
It would be nice to have a database listing students/trainees and what they are researching, so we could reach out to each other for potential collaborations
Trainees were adamant about knowing what knowledge base was needed to become "major players" in the health services research field. As one predoctoral trainee noted, "I am stuck with the resources I am aware of and that holds true for others in every other program so that ability to share could be beneficial."
Another predoctoral trainee remarked:
Maybe program directors would also be able to see something good in another program—maybe [they] could adopt that. I don't know how much interaction there is [among] current programs.
While the ultimate goal of knowing these details about other training programs, is to strengthen health services research training, the next section demonstrates that the process of sharing (i.e., building relationships) itself can contribute to a strong professional identity among health services researchers, which contributes to successful communities of practice.
Trainees consistently expressed that a core function of the collaborative should be to build relationships, which would include: mentoring, peer-to-peer interactions, and professional development. Trainees believed mentoring from senior health services researchers was a critical component of their health services research training. While one postdoctoral trainee noted that mentoring "begins on campus," another postdoctoral trainee suggested that mentorship beyond one's home institution was a key component for idea generation among trainees who may be in settings that, though nurturing, may not have the particular expertise to guide their thinking.
This postdoctoral trainee noted:
It would be wonderful for me to be able to say, when I'm coming up with a new definition of a medical error... that there's somebody who I know that I can bounce a question [off of] or maybe get a twice yearly discussion with.
As part of their networks of expertise, mentors are ideally positioned to foster this type of interaction desired by trainees. Trainees also understood that mentoring relationships involve some element of risk, and that a good fit was most conducive to a successful relationship. However, even in instances where the fit is not right, at least some interaction has occurred.
Several trainees also expressed that the collaborative would be valuable for (re)connecting with T32 alumni:
[We are] trying to establish a network with our alumni who are scattered across the country... .would envision that [effort] as something similar to [the learning collaborative].
Also, we are struggling with trying to figure out how alumni can all talk to each other... Maybe this could be a part of this broader AHRQ Learning Collaborative so they don't need to go to multiple groups to touch base. It needs to be easy enough so people will make use of [it].
Current trainees (both pre and postdoctoral) were also interested in connecting with one another:
Interacting with students in our program is a very important part [and] helps everyone grow a lot, but there isn't really any organized way for us to meet other students, see what other programs are like, [and] see how they differ.
Connecting students is where this collaborative could have some value added.... Connecting students is the most important part.
Trainees expressed the belief that peer-to-peer connections are invaluable in developing a sense of identity—both as AHRQ fellows and health services researchers—an important point that will be discussed further below.
For many trainees, building relationships also means professional development, a function that the collaborative may consider depending on how familiar trainees are with existing professional development resources, such as those offered through AcademyHealth or their home institutions. One postdoctoral trainee was unclear about how professional development was integrated into their training process, but, "Now that I am a postdoc, I am especially interested in this." Another postdoctoral trainee noted, "Professional development would be really useful in terms of job talks and where to look for jobs."
During the 2011 NRSA Conference, a predoctoral trainee noted the importance of "[c]onnecting with other [trainees] who are considering not going into postdoc/academia," which clearly implies that trainees have different professional goals and would benefit both from connecting with other trainees who have similar goals and understanding more concretely how to pursue relationships and opportunities that move trainees towards them. Workshops that mirror the "Transition from T to K" workshop during the 2011 NRSA Conference might be one mechanism for providing trainees with this sort of guidance. Here again, a mentor can play an important role. Many trainees believed it was important to have senior health services researchers as "point people" for professional development, in particular to leverage their networks of expertise.
Developing an Identity
Many of the trainees we interviewed believed very strongly that, although there is a sense of commitment to the field and a belief in the importance of health services research, a health services research identity is absent. As one predoctoral trainee remarked:
I felt like at Academy Health, this may sound kind of corny, that I felt a lot of pride in what we do, and I don't always get the sense that I'm part of a community of health services researchers. I'm part of a health policy and management department, and I am not an epidemiologist or a biostatistician, so it's hard to explain who I am and what I do. I think generating a community of health services researchers, making connections, and having pride in our field would be great. I think it could help faculty as well to be part of this greater community.
Trainees agreed that there is a substantial, but untapped opportunity to develop a "lockstep" approach across the community of health services researchers and indicated that the collaborative could be a valuable mechanism for creating a "cohort-like feeling" across the T32 programs. Cross-program connections may prove exceptionally useful for trainees who are in smaller training programs or have interests that are not extensively shared within their particular training program. One predoctoral trainee noted that they are the only individual with management interests in a program that is very "econ-heavy" and believed the opportunity to be connected to peers with similar interests outside their institution would be invaluable.
Trainees thought that increasing a sense of community through a collaborative could also create a more unified vision for the health services research field. One predoctoral trainee noted that, because of its interdisciplinary nature, there is a need to "set the direction" for health services research and that a common identity would be critical to that process. One postdoctoral trainee views AHRQ as an integral part of creating a health services research identity that cold "get them in the web" and help junior health services researchers "tap into the greater network" of health services researchers to understand the macro implications of micro efforts.
Organizing and Managing a Collaborative
Target Audience, Buy-In, and Launching
Trainees believed the collaborative could be a joint effort among trainees and program directors. One postdoctoral trainee remarked:
I don't see why [program directors] would not be interested. Certainly, program directors and faculty that work closely with AHRQ fellows [have an] interest in making sure students/mentees get access to more resources... trainees—if there was a way to bring them in too—that would be great.
Another postdoctoral trainee further emphasized that "If program directors aren't champions, it might not take off."
Trainees also believed that the collaborative would likely have immediate appeal for current trainees—possibly most heavily concentrated among predoctoral trainees—but that the collaborative could "open up" to other trainees, including alumni. The collaborative may also be accessed at different points in training, as suggested by one postdoctoral trainee:
I'm just finishing up my fellowship and am really just now learning how to get a research project done. This would be the time to get involved and learn from other trainees—it would be really nice to involve people along different stages. I would certainly want to stay connected.
Trainees understood that launching the collaborative, though exciting, would be difficult:
I think people are interested in collaborating. Like everything else, some people will take charge, others will free ride. I think it's in everyone's best interests.
At the beginning it is especially important to have high quality events. If [people] have a good experience, [they] would be more willing to participate. Starting off strong is probably the most important thing.
For the sake of publicity in the immediate future, trainees thought the collaborative could and should be represented at the AcademyHealth and NRSA meetings. Several trainees also believed AHRQ should be a primary convener and that it should play a central role in institutionalizing the collaborative and, in the words of one postdoctoral trainee, "help move it along." Reflecting on the perceived churning that occurs in the training program, one postdoctoral trainee remarked:
There is so much turnover in postdocs and predocs—so much turnover, it would be hard to have a lot of continuity. So, AHRQ would help to the extent that they would be willing to be involved.
Another postdoctoral trainee made the following suggestions:
I don't know in terms of infrastructure—maybe a central person at AHRQ who helps coordinate training programs. Helpful to start from AHRQ, but could it be trainees or recent graduates who take this on—or one group in particular—it takes work, continuing to have excitement. Helpful to have some people at AHRQ, may help to have a more senior person to be able to get people to lecture, more helpful to bring in people. It could be a lot of time initially up front to get excitement. If we can get people across the country—a committee to work on across the country.
Mechanisms for Interaction Through a Collaborative
There was substantial agreement among trainees that in-person contact was paramount to successful relationships and, by implication, the success of the collaborative. One postdoctoral trainee remarked, "[You] get buy-in when people meet each other. It's not just a virtual world—you're a real person." A predoctoral trainee echoed this sentiment, "You don't make friends online—you connect with people you already know." The seeds of strong networks—professional and otherwise—are sown in person. Trainees, while recognizing that physical distance represented a substantial barrier, continually emphasized the importance of face-to-face contact and believed leveraging professional meetings more effectively as opportunities for connection could be critical to the success of this collaborative.
Several trainees suggested ad-hoc events around the AcademyHealth and NRSA meetings. One predoctoral trainee was careful to point out the "conference mentality" that surrounds professional meetings. For example, trainees may be more inclined to stay inside their comfort zone (i.e., with students from their own programs) when attending professional meetings. A possible solution to step away from this in-group/out-group mentality is to link students with similar interests—or those who are attending the same meeting events—across programs more formally. Here again, a mentor can serve as an important role as a medium for interaction.
As a complement to more structured, formal interactions, trainees emphasized the importance of informal socializing during and around professional meetings in developing a sense of comfort with colleagues. One postdoctoral trainee remarked:
A lot of times at these conferences, they have these social mixers—schools will have their party in the evening with alumni and others affiliated. If AHRQ had something like that, that could be a way to bring people together.
While trainees emphasized the importance of in-person interaction, they acknowledged the practical difficulties of ongoing interaction in the absence of geographic proximity. That said, trainees were also enthusiastic about virtual interaction so long as the interaction is engaging and, according to one predoctoral trainee, "cuts through the noise" of the information-rich environment in which we exist. For example, trainees expressed some interest in a Web site, provided it was engaging and not duplicative. One predoctoral trainee remarked:
[I] could see [a Web site] happening as well. Although I think it would depend on the students and how interested everyone is. There are a lot of discussion forums online that die off, if everyone is interested and active then I guess it could work online.
Another predoctoral trainee noted, "I think people will be engaged if they get a lot out of it or if they really have ties to other people in it." Trainees expressed mixed thoughts with regards to Email, ListServs, and social networking sites like LinkedIn and Facebook. While some suggested Emails are likely to get lost in the shuffle, given all the Emails that people receive, others suggested that ListServs were probably the easiest form of communication. Trainees also proposed both webinars and virtual classrooms as forms of interaction, which would allow for remotely attending lectures and important talks given at other institutions. Again, the importance of webinars and virtual classrooms being interactive emerged.
Sustaining and Succeeding
Trainees believed that AHRQ could foster collaboration by designing funding opportunities that require collaboration across training programs, which might include "kick-off" funding for trainees to meet in person. These collaborations could be regionally based, which might be possible given distinct clusters of training programs in the Northeast, South, Midwest, and along the West Coast. Another aspect of this type of collaboration would involve the role of a mentor. A practical concern with this type of collaboration might involve authorship, as one predoctoral trainee noted, training programs may expect their students to be first authors. Additionally, as predoctoral students are generally busy with program coursework and qualifying exams, these collaborations might be more appealing to postdoctoral students or to predoctoral students who have completed their coursework. One predoctoral trainee suggested that a "progression of interest" might develop over time, but noted, "To lift yourself out of [coursework] in the first 2 years would be superficial, but this would really pick up steam after the coursework lessens."
As noted above, AHRQ could work to institutionalize the collaborative more formally. One postdoctoral trainee suggested that participants should have "permission to say 'No, I can't do this'" with no hard feelings. This postdoctoral trainee continued, "AHRQ has to recognize and support that 'No' may be a great option and the best thing for everyone at that moment." That said, it would be imperative for AHRQ and training programs to consider how they can "sell" the collaborative to its prospective members.
Measures of success appeared difficult to "think through" with trainees, and that might be explained, at least in part, by the fact that the collaborative is in its formative stages. Trainees were particularly curious about AHRQ's perspective, intentions, and expectations regarding the collaborative.
Trainees suggested a number of potential measures of success:
- Participation rates during collaborative activities.
- Participant feedback evaluations after collaborative activities. These measures might ask participants to numerically evaluate the expected ("Where do you think we should we be?") and actual ("Where do you think we are?") progress of the collaborative as well as its component parts before and after collaboration.
- Periodic evaluations of the collaborative's presence—for example, what opportunities individuals and institutions were made aware of (and took advantage of) as a direct result of the collaborative.
- Professional achievement, including: number of publications within a given time period and in which journals; job and postdoctoral placement of trainees, and funding types.
- Measures of engagement, connection, and/or network richness.
Measures of success should be directly linked to the stated goals and activities of the collaborative, which may change over time, and/or with different issues the collaborative wishes to address. Additionally, these measures would have to be clearly linked with changes in individual training programs that are expected to result as a direct consequence of the efforts of the collaborative.