R18 Closed-Loop Diagnostics: AHRQ R18 Patient Safety Learning Laboratories

Principal Investigator: Russell Phillips, M.D., Harvard University Medical School, Boston, MA 
Co-PIs: James Benneyan, Ph.D., Northeastern University, Boston, MA; Gordon Schiff, M.D., Brigham and Women’s Hospital, Boston, MA 
AHRQ Grant No.: R18 HS027282 
Project Period: 09/30/19–09/29/24 
Description: This learning lab addressed one of the most persistent contributors to diagnostic error in primary care: failures to “close the loop” on ordered diagnostic tests, referrals, and evolving symptoms. These failures lead to missed or delayed diagnoses, adverse outcomes, and inequities in care. The lab applied systems engineering, human factors, and simulation modeling to design, test, and evaluate high-reliability solutions that improve timely followup of critical diagnostic actions.

The specific aims were to:

1. Design, develop, and refine closed loop systems for diagnostic tests and referrals to ensure timely completion and followup.
2. Develop a reliable symptom-monitoring system to detect and act on evolving patient concerns.
3. Evaluate the generalizability of these solutions in community health centers, telemedicine systems, and simulated health system environments.

To achieve these aims, the team integrated quantitative, qualitative, and engineering approaches across multiple workstreams. Using EHR data, referral records, radiology reports, laboratory data, and structured interviews, the lab identified key breakdown points in dermatology referrals, colonoscopy scheduling for rectal bleeding, cardiac stress testing, radiology followup, acute kidney injury monitoring, and symptom evolution tracking.^1-5 Across conditions, baseline loop closure rates were often below 60 percent, with particularly low rates for colonoscopy and stress testing. Telehealth visits were associated with lower loop-closure rates compared with in-person visits. Lower rates were observed for visits where the clinician was a resident in training compared with a faculty physician and for visits made by patients who reviewed their notes in the EHR compared with those who did not read their notes.^2,4,6

Interventions and Innovations: Researchers piloted patient outreach and scheduling interventions that significantly improved loop closure rates for colonoscopy referrals related to rectal bleeding and for cardiac stress testing.^7,8 They also developed and evaluated a structured symptom-tracking registry that improved timely followup and reduced time to documented symptom resolution from a median of 103 to 20 days.^1,5 High-reliability systems engineering tools—including failure modes and effects analysis, process mapping, simulation, queuing models, and “loop-of-loops” modeling—were used to design safety-net processes that captured open loops that would otherwise be missed.⁹

Qualitative and Human Factors Findings: Interviews with patients who experienced delayed or incomplete referrals revealed communication gaps, lack of clarity about the purpose and value of referrals, and logistical scheduling barriers.¹⁰ These findings guided the design of patient-centered communication interventions and highlighted the critical role of patient engagement in diagnostic safety.

Systems-Level Implications: Modeling showed that while upstream scheduling improvements can reduce delays, downstream safety-net mechanisms are essential to achieving high closure rates across multiple loops. These improvements, however, have systemwide implications: as loop closure rates improve, appointment access becomes a limiting factor without additional capacity.^5,9 This work shows the need for aligned interventions that combine patient engagement, workflow redesign, and capacity planning.

This learning lab demonstrated the feasibility of using engineering methods to improve loop closure in primary care, contributing to national efforts to address diagnostic delays and failures to follow up on abnormal findings.

To date, this PSLL’s work has resulted in at least 12 peer-reviewed publications, 83 citations in other publications, and several national conference presentations. 

Publications

2025

• Fernández L, Ricci D, Pollack A, et al. Patient perspectives on delayed specialty followup after a primary care visit. J Am Board Fam Med 2025 May 12;38(1):139-153.
• Mena J, Rodriguez M, Sternberg SB, et al. Incidence, recognition, and follow-up of laboratory evidence of acute kidney injury in primary care practices: Analysis of 93,259 creatinine results. Am J Med 2025 Jan;138(1):79-86.

2024

• Amat MJ, Anderson TS, Shafiq U, et al. Low rate of completion of recommended tests and referrals in an academic primary care practice with resident trainees. Jt Comm J Qual Patient Saf 2024 Mar;50(3):177-184.
• Amat MJ, Zhong A, Zhang C, et al. Implications of sociodemographic and clinical factors associated with completion of cardiac stress tests. J Am Board Fam Med 2024 Nov 1;37(6):1088-1094.
• Bell SK, Amat MJ, Anderson TS, et al. Do patients who read visit notes on the patient portal have a higher rate of “loop closure” on diagnostic tests and referrals in primary care? A retrospective cohort study. J Am Med Inform Assoc 2024 Feb 16;31(3):622-630.
• Campbell KA, Sternberg SB, Benneyan J, et al. Completion rates and timeliness of diagnostic colonoscopies for rectal bleeding in primary care. J Gen Intern Med 2024 May;39(6):985-991.

2023

• Pagani K, Lukac D, Olbricht SM, et al. Urgent referrals from primary care to dermatology for lesions suspicious for skin cancer: Patterns, outcomes, and need for systems improvement. Arch Dermatol Res 2023 Jul;315(5):1397-1400.
• Phillips RS, Benneyan J, Bargal B, et al. Closing the loop: Re-engineering the assessment and tracking of symptoms in primary care. J Gen Intern Med 2023 Mar;38(4):1054-1058.
• Zhong A, Amat MJ, Anderson TS, et al. Completion of recommended tests and referrals in telehealth vs in-person visits. JAMA Netw Open 2023 Nov 1;6(11):e2343417.

2022

• White T, Aronson MD, Sternberg SB, et al. Analysis of radiology report recommendation characteristics and rate of recommended action performance. JAMA Netw Open 2022 Jul 1;5(7):e2222549.

2021

• Benneyan JC, White T, Nehls N, et al. Systems analysis of a dedicated ambulatory respiratory unit for seeing and ensuring followup of patients with COVID-19 symptoms. J Ambul Care Manage 2021 Oct-Dec;44(4):293-303.
• Nehls N, Yap TS, Salant T, et al. Systems engineering analysis of diagnostic referral closed-loop processes. BMJ Open Qual 2021 Nov;10(4):e001603.

References

1. Phillips RS, Benneyan J, Bargal B, et al. Closing the loop: Re-engineering the assessment and tracking of symptoms in primary care. J Gen Intern Med 2023 Mar;38(4):1054-1058.
2. Zhong A, Amat MJ, Anderson TS, et al. Completion of recommended tests and referrals in telehealth vs in-person visits. JAMA Netw Open 2023 Nov 1;6(11):e2343417.
3. White T, Aronson MD, Sternberg SB, et al. Analysis of radiology report recommendation characteristics and rate of recommended action performance. JAMA Netw Open 2022 Jul 1;5(7):e2222549.
4. Amat MJ, Anderson TS, Shafiq U, et al. Low rate of completion of recommended tests and referrals in an academic primary care practice with resident trainees. Jt Comm J Qual Patient Saf 2024 Mar;50(3):177-184.
5. Phillips RS. Final Report: R18 Closing the Loop on Diagnostic Tests, Referrals, and Symptom Evaluation, AHRQ, Editor. 2025, Harvard Medical School: Boston, MA. p. 1-21.
6. Bell SK, Amat MJ, Anderson TS, et al. Do patients who read visit notes on the patient portal have a higher rate of “loop closure” on diagnostic tests and referrals in primary care? A retrospective cohort study. J Am Med Inform Assoc 2024 Feb 16;31(3):622-630.
7. Campbell KA, Sternberg SB, Benneyan J, et al. Completion rates and timeliness of diagnostic colonoscopies for rectal bleeding in primary care. J Gen Intern Med 2024 May;39(6):985-991.
8. Amat MJ, Zhong A, Zhang C, et al. Implications of sociodemographic and clinical factors associated with completion of cardiac stress tests. J Am Board Fam Med 2024 Nov 1;37(6):1088-1094.
9. Nehls N, Yap TS, Salant T, et al. Systems engineering analysis of diagnostic referral closed-loop processes. BMJ Open Qual 2021 Nov;10(4):e001603.
10. Fernández L, Ricci D, Pollack A, et al. Patient perspectives on delayed specialty followup after a primary care visit. J Am Board Fam Med 2025 May 12;38(1):139-153.