Assembling a Systematic Review Team:
Balancing Expertise and Potential Conflicts of Interest
On September 20, 2011, Stephanie Chang made this presentation at the 2011 Annual Conference. Select to access the PowerPoint® presentation (910 KB). .
Slide 1
Assembling a Systematic Review Team: Balancing Expertise and Potential Conflicts of Interest
Avoiding bias in systematic reviews
AHRQ Annual Meeting
Bethesda North Marriott Conference Center
September 20, 2011
Slide 2
IOM Recommendations
Standard 2.1 Establish a team with appropriate expertise and experience to conduct the systematic review.
- 2.1.1 Include expertise in the pertinent clinical content areas.
Standard 2.2 Manage bias and conflict of interest (COI) of the team conducting the systematic review.
- 2.2.3 Exclude individuals whose professional or intellectual bias would diminish the credibility of the review in the eyes of the intended users.
Standard 2.4 Manage bias and COI for individuals providing input into the systematic review.
- 2.4.2 Exclude input from individuals whose COI or bias would diminish the credibility of the review in the eyes of the intended users.
Standard 2.7 Submit the protocol for peer review.
- 2.7.1 Provide a public comment period for the protocol and publicly report on disposition of comments.
Slide 3
Definitions
- Bias:
- A systematic error or deviation from the truth, in results or inferences (Higgins JPT, et al. Cochrane handbook 2009).
- Inclination to present or hold a partial perspective at the expense of (possibly equally valid) alternatives.
- Conflict of interest (COI):
- A set of conditions in which professional judgment concerning a primary interest (such as the validity of research) tends to be unduly influenced by a secondary interest (such as financial or professional gain) (Thompson, 1993).
- Expertise:
- Extensive knowledge based on research, experience, or occupation and in a particular area of study. Expert is someone who is a widely recognized authority by peers and public.
Slide 4
Potential biases in systematic reviews
- Holding an established opinion on the answers to the research questions of the review; the inclination to present or hold a partial perspective at the expense of (possibly equally valid) alternatives:
- Individuals in a particular specialty may have an unbalanced familiarity for one type of intervention or technology.
- Individuals livelihood may depend on finding greater benefit of one type of intervention or technology over a competitor.
- Experts may hold a previously formed opinion and be unlikely to change.
- Holding an established opinion on the quality of studies to be included:
- Individuals who conducted a study or trial may not be able to assess the quality of their own or other "competitor" studies in an unbiased manner.
Slide 5
My take home points
- We are concerned with bias, but this is difficult to measure.
- Non-financial conflicts of interest can result in bias:
- Perceived conflicts of interest may still reduce the credibility of the review in the eyes of the intended users.
- Conflicts due to familiarity or livelihood based on an intervention or technology can be managed by balancing the team and input from different specialty areas.
- Expertise can be a form of bias:
- May be difficult to discern unless they have published an editorial or commentary on the subject.
- Not including appropriate expertise may reduce the credibility of the review in the eyes of the intended users.
- It is important to hear opinions from all sides, but the core team should be without bias.
- Individuals should not be involved in grading their own study, either at the individual study level or at the body of evidence level.
Slide 6
Introductions and Acknowledgements
- Susan Norris, Oregon Health and Science University.
- Gillian Sanders Schmidler, Duke University.
AHRQ staff:
- Elise Berliner.
- Christine Chang.
- Yen-Pin Chiang.
- Supriya Janakiraman.
- Elisabeth Kato.
- Mary Nix.
Evidence-based Practice Center (EPC) workgroup:
- Eric Bass.
- Stanley Ip.
- Melissa Mcpheeters.
- Sydney Newberry.
- Susan Norris.
- Margaret Piper.
- Paul Shekelle.
- Meera Viswanathan.
- Evelyn Whitlock.
- Renee Wilson.
- Michael White.
Slide 7
Objectives
- Provide a framework for thinking about intellectual conflict of interest.
- Review of the existing literature on the potential effects of intellectual conflict of interest.
- Work through examples of potential conflicts of interest when conducting a systematic review.
Slide 8
Case Examples
Slide 9
Procedure specialist
A review is comparing surgical, medical or alternative treatments. Should a specialist who makes a living doing only surgical interventions be the principal investigator?
- Yes.
- Yes, but with restrictions.
- No.
- It depends.
Slide 10
Procedure specialist
A review is comparing surgical, medical or alternative treatments. Should a specialist who makes a living doing only surgical interventions be part of the core team?
- Yes.
- Yes, but balance with other perspectives.
- Yes, but with restrictions.
- No.
- It depends.
Slide 11
Professional society staff
Should a scientist who works for a specialty professional society be the principal investigator of a review?
- Yes.
- Yes, but with restrictions.
- No.
- It depends.
Slide 12
Professional society staff
Should a scientist who works for a specialty professional society be a co-author of a review?
- Yes.
- Yes, but balance with other perspectives.
- Yes, but with restrictions.
- No.
- It depends.
Slide 13
Professional society staff
Should a scientist who works for a specialty professional society be on the technical expert panel?
- Yes.
- Yes, but balance with other perspectives.
- Yes, but with restrictions.
- No.
- It depends.
Slide 14
Professional society staff
Should a scientist who works for a specialty professional society be a peer reviewer?
- Yes.
- Yes, but balance with other perspectives.
- Yes, but with restrictions.
- No.
- It depends.
Slide 15
Active research in area
A scientist has federal grants to study one of the intervention of interest. Should this person be the principal investigator of the review?
- Yes.
- Yes, but with restrictions.
- No.
- It depends.
Slide 16
Active research in area
A scientist has federal grants to study one of the intervention of interest. Should this person be a co-author of the review?
- Yes.
- Yes, but balance with other perspectives.
- Yes, but with restrictions.
- No.
- It depends.
Slide 17
Active research in area
A scientist has federal grants to study one of the intervention of interest. Should this person be on the technical expert panel?
- Yes.
- Yes, but balance with other perspectives.
- Yes, but with restrictions.
- No.
- It depends.
Slide 18
Active research in area
A scientist has federal grants to study one of the intervention of interest. Should this person be a peer reviewer?
- Yes.
- Yes, but balance with other perspectives.
- Yes, but with restrictions.
- No.
- It depends.
Slide 19
Published
A scientist was an author of a study that would potentially be included in the review. Should this person be the principal investigator?
- Yes.
- Yes, but with restrictions.
- No.
- It depends.
Slide 20
Published
A scientist was an author of a study that would potentially be included in the review. Should this person be on the core team?
- Yes.
- Yes, but balance with other perspectives.
- Yes, but with restrictions.
- No.
Slide 21
Published
A scientist was an author of a study that would potentially be included in the review. Should this person be on the technical expert panel?
- Yes.
- Yes, but balance with other perspectives.
- Yes, but with restrictions.
- No.
Slide 22
Published
A scientist was an author of a study that would potentially be included in the review. Should this person be a peer reviewer?
- Yes.
- Yes, but balance with other perspectives.
- Yes, but with restrictions.
- No.
- It depends.
Slide 23
Involved in study
A scientist coordinated one center of a multi-center trial that would potentially be included in a review. Should this be the principal investigator?
- Yes.
- Yes, but with restrictions.
- No.
- It depends.
Slide 24
Involved in study
A scientist coordinated one center of a multi-center trial that would potentially be included in a review. Should this be part of the core team?
- Yes.
- Yes, but balance with other perspectives.
- Yes, but with restrictions.
- No.
- It depends.
Slide 25
Published review or guidelines
A scientist has previously published a review or guideline on the subject of the review. Should this person be the principal investigator?
- Yes.
- Yes, but with restrictions.
- No.
- It depends.
Slide 26
Published review or guidelines
A scientist has previously published a review or guideline on the subject of the review. Should this person be a co-author?
- Yes.
- Yes, but balance with other perspectives.
- Yes, but with restrictions.
- No.
- It depends.
Slide 27
Scientific advisor or editor
A scientist serves on an advisory or editorial board without financial compensation on a related topic to the review. Should this person be the principal investigator?
- Yes.
- Yes, but with restrictions.
- No.
- It depends.
Slide 28
Scientific advisor or editor
A scientist serves on an advisory or editorial board without financial compensation on a related topic to the review. Should this person be a co-author?
- Yes.
- Yes, but balance with other perspectives.
- Yes, but with restrictions.
- No.
- It depends.
Slide 29
Scientific advisor or editor
A scientist serves on an advisory or editorial board without financial compensation on a related topic to the review. Should this person be on the technical expert panel?
- Yes.
- Yes, but balance with other perspectives.
- Yes, but with restrictions.
- No.
- It depends.
Slide 30
Scientific advisor or editor
A scientist serves on an advisory or editorial board without financial compensation on a related topic to the review. Should this person be a peer reviewer?
- Yes.
- Yes, but balance with other perspectives.
- Yes, but with restrictions.
- No.
- It depends.