Systematic Review of ECG-based Technologies for Evaluating Patients With Acute Coronary Syndrome: CRDD0311
Disposition of Comments
Project ID: CRDD0311
The Agency for Healthcare Research and Quality's (AHRQ) Technology Assessment (TA) Program supports and is committed to the transparency of its review process. Therefore, invited peer review comments and public review comments are publicly posted on the TA Program Web site at http://www.ahrq.gov/research/findings/ta/index.html within 3 months after the associated final report is posted on this Web site.
This document presents the peer review comments and public review comments sent in response to the draft report, Systematic Review of ECG-based Technologies for Evaluating Patients With Acute Coronary Syndrome, posted on the AHRQ Web site from November 9, 2011, through November 23, 2011. The final version of the report is available online.
Select for Table 1: Invited Peer Reviewer Comments.
Select for Table 2: Public Review Comments
Table 1: Invited Peer Reviewer Comments
|Reviewer1||Section2||Reviewer Comments||Author Response3|
|1||General||I have divided my comments into three sections, based on the 3 hats I wore while reviewing the report; namely as internist, methodologist, and editor.||Noted|
|Note that my comments regarding the Executive Summary should be carried forward to the main report. Thus I didn't enter comments for Executive Summary, specifically.||Noted|
|Overall, this was a very nice report, that succinctly and clearly evaluates the evidence addressing the key questions.||We appreciate this assessment.|
|There's a general conflation between CAD and ACS (more specifically AMI). In the settings of the included studies, ECG is used primarily to diagnose AMI, not CAD. Yet the population of interest is those at low to intermediate (other places called moderate) risk of CAD. The risk of AMI depends on much more than the risk of CAD, including symptoms, comorbities, recent legal and illegal drug use, etc. Much of the report seems to emphasize the use of ECG to diagnose CAD, which appears to be an inappropriate use.||Noted. The Key Questions directed us to emphasize the ECG-based technologies to diagnose either CAD or ACS (including AMI).|
|I would suggest a 1 page (max) abstract.||We have added a 1-page abstract.|
The introduction would benefit from added information, some of which is in the main report. Eg,
p ES-1, third paragraph, last sentence
|1. "comparative technical efficacy". Compared to what? This is the first mention of "comparative".||We have substituted "test performance" for "comparative technical efficacy."|
|ES and main report: Methods||
2. The methods section (both ES and main report) do not make it adequately clear that the gray literature was used only for KQ 1.
|We have added text to clarify that we searched the gray literature only for KQ 1 and provided the URL for www.freepatentsonline.com. We did not search the gray literature for KQ 2 because we considered only peer-reviewed studies published in English to be eligible for inclusion for this key question.|
|ES and main report||
3. The eligibility criteria are unclear, even with the appendix (which is not mentioned in the ES). It would be much clearer to structure them as PICOD.
We have revised the PICOTS table in the Appendix.
We have deleted the words "from formal analysis" to clarify that we excluded altogether devices that are not commercially available.
We have added "patient outcomes include catheterization laboratory findings, clinical outcomes of mortality, cardiac symptoms, function and functional status, and therapeutic interventions" in the section that describes the analytic framework.
|ES||4. Please add how study quality was rated to the ES.||We have incorporated this request in the Methods section of the ES.|
|5. Please describe the quantitative methods in the ES.||We have added to the ES the two paragraphs in the main report that describe the quantitative methods.|
|6. KQ 2a: The metrics of % of negative and positive test readings are confusing and misleading. These numbers have most to do with the underlying rate of AMI (I presume; it's not reported "positive" or "negative" for what). These numbers make sense when presented in a 2x2 table, but they are misleading in paragraph form. It would be clearer to state that there was agreement among 86% (then give a breakdown if you desire).||We have modified this paragraph by clarifying that "positive" refers to "abnormal" and that "negative" refers to a "normal" reading, irrespective of the underlying (and not reported) rate of acute myocardial infarction.|
|7. ES-5, KQ 2b, sensitivity analysis: Without further justification it seems inappropriate to exclude a study because it was small but had weight. What made it "disproportionately weighted"?||
We performed a sensitivity analysis excluding two studies, one of which had a small sample size that was disproportionately weighted in the random-effects meta-analysis (Menown 2001). This study was also excluded in a sensitivity analysis performed in the original report.
We have added the following text to justify the exclusion: "The standard errors of the effect measures are determined by two factors—the number of subjects and the measure of interest in the compared groups. These factors in turn determine the weight given to a study in the meta-analysis. If one or both of these factors differ (are outliers) from the other studies included, the results can be overestimated or underestimated (Tang 2000)."
|8. ES-5-6, KQ 2b, PRIME: What methods did you use to estimate the posttest probabilities?||
We have added the following text to detail the methodology: "The posttest probabilities for negative and positive results were calculated on the basis of assumed prevalence and positive and negative likelihood ratios obtained from the meta-analysis."
We also added a figure to the Results section to show the posttest probabilities over all possible prevalence values for each diagnostic method.
|ES: Results||ES-3, KQ1a: I don't understand the status of the 3/11 devices that are not the 8 cleared for marketing.||The marketing status of these 3 devices (CarDx, Cardiologic Explorer, and Vascular Explorer) was uncertain from the available evidence. We have since contacted the FDA and received confirmation that these 3 devices are not cleared for marketing; therefore, they do not meet our criteria for inclusion in the report. We have updated the entire report where appropriate to reflect exclusion of these devices.|
|ES: Results||ES-4, 1st paragraph: To my knowledge, percutaneous coronary interventions are not considered to be surgical interventions.||We have revised the sentence to "...from surgical or percutaneous intervention ..."|
|ES and main report||I don't understand the vagueness that is ascribed to ACS. WHO has a clear definition (gold standard) of AMI. There are definitions of unstable angina. These are used as reference standards in the report (eg, biochemical markers).||We recognize the lack of clarity about ACS in this report as well as in the published literature and clinical practice. Our intention is to emphasize throughout the report that ACS is a working diagnosis that typically changes as more information about a given patient becomes available in the course of a clinical evaluation, and that ACS includes STEMI, NSTEMI, unstable angina, and other etiologies that suggest ischemic heart disease.|
|ES-6 and main report||
KQ 2b, PRIME: More details would be helpful to understand the comparison with the performance characteristics of the 12-lead ECG. How are clinically and statistically significant defined?
To avoid ambiguity, we deleted the term, "clinically significant."
We agree that a direct comparison of the two tests is an important part of the key questions.
|ES||How is diagnostic decisionmaking defined?||We added the phrase, "... providing evidence that the use of, or findings from, ECG-based technologies other than the standard 12-lead ECG had an impact on the decisions or actions of patients or health care providers."|
|Why was OCCULT MI considered to have intermediate to high risk patients?||The authors stated explicitly that the OCCULT MI was a "... study of moderate- to high-risk chest pain patients ..."|
|ES-6||KQ 2d (and main report): In OCCULT MI, what were the numerical results for 12-lead ECG?||The findings that we summarized for KQ 2d represent a secondary analysis of the OCCULT MI study. Additional information, including the numerical results of the 12-lead ECG in relation to patient outcomes was not reported in the study.|
|ES||Next paragraph: What were the outcomes and results of the "Another study"?||This study did not report outcomes and results that pertained to patient outcomes. We mentioned this study in the context of KQ 2d to inform readers that patient outcomes were assessed but not reported.|
|KQ 1a and 1b. Please add references to all the assertions made about the different technologies.||We have referenced all the assertions for which references are available and appropriate.|
|p 4, first paragraph (main report)||"The target population for the purpose of this report... focuses... on patients with symptoms suggestion of ACS at low to intermediate risk of ischemic heart disease..." But IHD is not the same as CAD. "...including patients with clinical presentation and initial ECG findings consistent with unstable angina and NSTEMI as well as..." These are risks factors for ACS, not CAD, per se.||This was an error in the draft report. We have replaced ischemic heart disease with CAD.|
9. Page 10: Please more fully describe the random effects bivariate meta-analysis methods. You can paraphrase Harbord (reference 13). You should also cite the original methods papers (19 & 20 ± 21 from Harbord).
|We have added text to more fully describe the random-effects model with additional references in the Methods section.
We also added the following text to describe the software used for these analyses: "Analyses were performed with software (SAS, version 9.1, SAS Institute, Cary, NC; Excel, version 5.0, Microsoft, Bedford, WA; and Comprehensive Meta Analysis version 2.0, Englewood, NJ)."
|Results||10. Figure 3. The numbers of citations identified are unclear. How were there only 1 citation from Cochrane and 74 from Embase? Were these the unique citations compared to Medline? (Based on Appendix B, I think so.) If so, what were the 23 duplicates?||Exact strategies for the PubMed, Embase, and Cochrane database searches are provided in Appendix B. Upon update of the searches during the draft report review period, the total number of citations found through the Embase search increased to 87. The number of citations returned from the search of the two specific included Cochrane databases (the Cochrane Database of Systematic Reviews and the Database of Abstracts of Reviews of Effects) remained at 1.
The reviewer is correct that the Embase search included a limit designed to eliminate citations that are also indexed in MEDLINE. In practice this limit is not completely efficient, thus leaving some duplication remaining between the PubMed and Embase result sets that must be removed manually.
|11. Page 25, 2nd paragraph: If the question is raised that the algorithms changed over time, you might want to do a sensitivity analysis, subgroup analysis, or meta-regression by time to see if there were indeed changes.||We agree and have added the following text to the Results section: "A bivariate meta-regression by time analysis was used to determine if the sensitivities and specificities changed over time. For the PRIME ECG, both sensitivity and specificity decreased slightly but not significantly as time increased (p<0.81). However, for the standard ECG, both sensitivity and specificity slightly increased but not significantly as time increased (p<0.47)."|
|12. Page 27: I would strongly suggest that you draw a summary ROC curve. Tables 4 and 5 are fine as far as they go, but do not convey the results clearly.||We agree and have included both a receiver operating characteristic (ROC) space for individual studies/methods and a summary ROC curve for summary measures in the Results section.|
|13. Page 31, KQ 2d: For OCCULT MI, can you more directly compare PRIME and 12-lead? Do they provide enough data to calculate sensitivities and specificities (predictors) for mortality?||There is insufficient information reported in the study to make such comparisons or calculations.|
|14. Page 33, 3rd paragraph: The sentence "The 95-percent confidence interval of the estimates of sensitivity for these two tests overlaps, such that the observed differences are not statistically significant." is not valid. The overlap of CIs does not indicate the degree of statistical significance. Overlapping CIs can often be found with statistically significant differences.||The text in the Results section was modified and now includes: "However, using a paired Z-test, neither the LR+ nor the LR- was statistically significant when comparing diagnostic methods, p<0.21 and p<0.08, respectively."|
|Page 14, 1st paragraph: Please confirm (and if confirmed, reference) the assertion that autopsy is the only procedure that can provide a definitive diagnosis of CAD. Surgical specimens can provide the same tissue samples. Both of these can confirm atherosclerosis, but I'm not sure about CAD per se. The amount of atherosclerosis (or of stenosis) is probably not the best predictor of clinical events or cardiac function. Angiography (in a live, beating heart) can provide a better assessment of functional disease than bloodless tissue.||We deleted this assertion.|
|Page 15: Why are biomarkers under Tests for Diagnosing CAD? They should be under Tests for Diagnosing ACS.||We have revised this section, as well as Tables ES-1 and Table 2, so that biomarkers are now discussed as a reference standard for ACS and not CAD.|
|KQ 2a, 2d: What was the quality of the studies?||We added the quality ratings for all studies referred to in KQ 2a and 2d in both the ES and the main report.|
|Discussion||Please discuss the quality of the studies.||We added a paragraph that discussed the quality of the studies in the Discussion.|
|The Future Research section is nice. I would also suggest studies to evaluate the patients and the settings where signal analysis ECGs might be most beneficial.||We appreciate this suggestion and have incorporated it in the report.|
|Tables||Tables ES-1 and 2: I don't understand what an incomplete level of acceptability is.||We have deleted this term altogether. We removed biomarkers as an "incomplete" standard for CAD and added them as an "acceptable" standard for ACS.|
|Tables 4 & 5: Please add footnotes repeating why the various studies were omitted from sensitivity analyses (so that the tables can stand on their own).||We have added footnotes accordingly.|
|Figures||I could not properly review the analytic framework or the literature flow diagram, as they did not reproduce.||Noted|
|Figure 1: The "B" is not aligned with the vertical line from Decision threshold for treating.||We have fixed the formatting.|
|Figure 2: Add definition of "KQ"||We have added the definition to the Figure.|
|Editorial/ Technical Issues||15. Intro, page ES-1, 2nd paragraph: I think I know what you mean, but it would be clearer if you stated more explicitly the logic flow in the sentence starting "If the ECG indicates STEMI..."||We have modified this sentence to improve clarity.|
|16. ES-1, 2nd paragraph, last sentence ("Pending the results..."). I'm not sure what's added by this sentence. It seems to say nothing that wasn't already said more clearly.||We have replaced "pending the results of further testing" with "In sum ..." We think that this serves to emphasize the point that during this period of clinical evaluations, patients are often given the working diagnosis of ACS. Our intention is to remind readers that the diagnosis of unstable angina (for example) and ACS may coexist during the evaluation and treatment phases of patients with acute chest pain.|
|17. ES-2, first sentence of Methods. I would suggest dropping this sentence on an analytic model, unless you add the analytic framework into the ES. In which case, it would be better to actually discuss the AF, not the theory of AF creation.||We have incorporated this suggestion.|
|18. Throughout: As a guideline developer, I would ask that you use the singular "guideline" for a set of recommendations from a single group. Current guidelines from the ACC/AHA sounds like they have multiple sets of guidelines, not one unified guideline.||We have made this correction throughout the report.|
|19. OCCULT MI is described as moderate to high risk (or in one instance high to moderate risk), where everywhere else the term intermediate risk is used.||The authors of the OCCULT MI explicitly stated that theirs was a "... study of moderate- to high-risk chest pain patients ..."|
|20. ES-6, KQ 2d. Paragraph 1 says "We identified a single study" and then the next paragraph says "Another study". This is confusing.||We have edited the second paragraph of KQ 2 to avoid this confusion.|
|21. ES-6, last paragraph. The first sentence is unclear. "Another study followed patients who initially presented with symptoms suggestive of ACS, both through their hospital course of treatment and after discharge from the hospital." "Initially presented" is redundant and confusing. I don't at all understand the phrase "both through their hospital course of treatment and after discharge from the hospital."||We have revised this paragraph to improve clarity.|
|22. Page 6. Describe/Discuss your analytic framework, not just the theory.||We have added a paragraph that describes our analytic framework.|
|23. Page 8: Should you add clinicaltrials.gov to the list of gray literature sources in the first paragraph of Process for Study Selection?||We have added www.clinicaltrials.gov to the list of Web sites searched.|
24. Appendix C: I would strongly suggest you add a slightly abbreviated version of this to the Methods section, structured as PICO, removing redundancies and repetition (eg, include N≥20, exclude N<20)
|We have included a PICOTS table in Appendix C.|
|25. Appendix D: I would strongly suggest you add the "Overall Assessment of Study Design" section of the appendix to the Methods section.||We have described our method for overall assessment of study design and referenced Table D-1 in the Methods section of the Executive Summary and main report.|
|26. Page 25, 3rd paragraph: The concept about "at least two sets [of biomarkers], 8 hours apart, are needed for an adequate reference standards" is first introduced here and is not discussed in the section about biomarkers as a diagnostic tool (KQ 1).||We have revised the biomarkers section of KQ 1 to highlight the importance of serial testing for biomarker elevation.|
|27. ES-1, 2nd paragraph: "to have had an MI", not "to have undergone an MI".||We have made this correction.|
|28.ES-1, 2nd paragraph: "usually given the diagnosis" would be clearer than "ascribed".||We have made this correction.|
|29. ES-3, 3rd paragraph: I would suggest deleting the info about instructions on the form and Distiller SR from the ES. Too detailed about minutiae.||We have incorporated this suggestion.|
|30. "Overread" may be a word (just), but I would strongly suggest you replace it with "read over", "read", or something else. It's used to mean different things throughout the report.||We will replace "overread" with "reviewed by a second investigator" throughout the report when indicated.|
|31. ES-5, last paragraph: "initial" and "most clearly" in "the initial study that most clearly used a different diagnostic algorithm" are unclear.||We have revised this sentence to improve clarity.|
|32. Page 3, last paragraph: "area between A and B" would be better. There is no A-B segment.||We have incorporated this suggestion.|
|33. Page 10: Preface the 1st paragraph with a header like "Data Analysis"||We have incorporated this suggestion.|
|34. Page 14, 1st paragraph: "infer" is the wrong word (visualize? Diagnose?)||We changed the first "infer" to "detect."|
|35. Page 14, 1st paragraph: Postmortem autopsy is redundant. Premortem, it's a biopsy.||We have deleted the entire sentence.|
|36. Pages 15-16. Please lower the heading level of Strengths and limitations... below that of the technologies.||We have incorporated this suggestion.|
37. Missing spaces in a few places
|We have made this correction.|
|2||General||I have carefully reviewed the draft of the Technology Assessment (TA) "Systematic Review of ECG-based Signal Analysis Technologies for Evaluating Patients with Acute Coronary Syndrome." The TA appears to represent a logical progression from and to be more specific than a TA dated May 24 2010.||Noted|
|I am pleased to note the addition of Dr. Galen Wagner to the list of authors associated with the DUKE EPC. Dr. Wagner adds the expertise and knowledge of clinical experience in acute cardiac care, several hundred publications , numerous national and international research collaborations, and experience as Editor of the Journal of Electrocardiology.||We concur. Dr. Wagner is a key member of our team.|
|The Duke EPC has addressed very specific requests defined by the key questions with a generally excellent literature search, analysis, and response. The results appear fair and unbiased. Summary tables are provided appropriately for easy reference. I agree with the findings and conclusions.||We appreciate this assessment.|
|Executive Summary||The executive summary effectively reviews the role of coronary artery disease as a cause of mortality in the U.S., states the key questions, the approaches used, and the authors' findings with no important omissions.||We appreciate this assessment.|
|The introduction is excellent and for an overall approach covering epidemiology, ischemia versus infarction, ACS, diagnostic testing and risk stratification of CAD, ECG role and limitations, evaluating emerging ECG technologies, is about as "good as it can get." The description of the objectives is adequate and includes a short discussion of available technologies that noninvasively analyze cardiac electrical activity.||We appreciate this assessment.|
The Key Questions were directed and refined with the sponsor of the report.
The approach is good. The inclusion and exclusion criteria excellent and they are relevant to the key questions. The data abstraction process appears stepwise and efficient. Two investigators independently reviewed abstracted data. Disagreements were revised by consensus. Statistical approaches appear satisfactory.
|We appreciate this assessment.|
|Results: Analysis for KQ 1, KQ 1a||
The list of devices appears correct. Based on studies on the targeted population, I do not know of other qualifying devices.
I have a few suggestions/corrections to the device descriptions:
We appreciate these suggestions.
|Results: Analysis for KQ 1, KQ 1b||I am certain that this was a challenging area for the authors to summarize and judge. The discussion and assessment of gold standard tests, including invasive and noninvasive testing are satisfactory. Most authorities would agree with the key summary for KQ1 and the assessments presented in Table 2.||Noted|
|Results: Analysis for KQ 2||Exclusion and inclusion criteria for the literature search were appropriate to the guidelines formulated by the key questions. The grading of the selected manuscripts was rigorous. I agree with the grading as presented in Table 2. I am not aware of other relevant studies meeting the selection guidelines.||Noted|
PRIME ECG Device
The authors report their analysis of 10 eligible studies (13 manuscripts) evaluating the performance of the PRIME ECG body surface mapping device in the target population. The 10 studies are discussed fairly and in detail as to patient population, mode of presentation, the diagnostic criteria used for diagnosis of myocardial ischemia, infarction and ACS, and what elements of the studies were blinded. Performance is reported by sensitivity, specificity, and confidence intervals. The authors performed useful meta-analysis across all 10 included PRIME ECG studies, and performed a separate meta-analysis of 8 studies after excluding 2 studies for appropriate reasons. I agree with their findings
|We appreciate this summary and assessment.|
|Discussion||I agree with discussion of the Summary of Findings and the Applicability of Current Studies.||Noted|
|The limitations of this review mentioned by the authors are correct. Further evaluation of device limitations may be appropriately deferred until there is better evidence of diagnostic utility. However, the current report is a rare evaluation of signal processing devices without including a discussion of signal noise. Signal averaging (SA) devices, as noted, reduce noise by the averaging process and report noise levels encountered with the test. Body surface mapping with the increased number or leads is likely to be very susceptible to noise. The version of the PRIME ECG used in the OCCULT MI study generates an algorithmic diagnosis regardless of signal noise. This might emphasize the benefit of "computerized diagnostic algorithms to provide an immediate preliminary report" and signal graphics for physician (or other caregiver) over read.||We appreciate this important point, which was lacking in the draft report. We have added two sentences to this effect in the section that introduces and describes body surface mapping.|
|The discussion of future research is generally excellent. The 3rd paragraph might note that the PRIME ECG was essentially used as an "add-on" test in the OCCULT MI study.||We have added a sentence that highlights OCCULT MI as a good example of a study design that allows for evaluating the potential value of an "add-on" test.|
|The second "Summary" might be better titled "Summary and Conclusions"? I agree with the findings and conclusions of the document.||We have incorporated this suggestion in the report.|
|Tables||All tables are satisfactory.||Thank you.|
|Figures||Figure 2 could be improved by brackets and arrows that are easier to follow.||We will make this change.|
|Appendices||Appendices are very adequate and appropriate.||Thank you.|
|References||References are adequate. Reference 20 is difficult to obtain.||Noted|
|Additional Questions||Page 19: "While screening articles for eligibility to meet... we did not identify any published studies that evaluated an ECG-based signal averaging device for the diagnosis of CAD or ACS in asymptomatic individuals." This sentence appears inconsistent with the inclusion-exclusion criteria and the target populations.||We revised but did not delete this sentence.|
|Page 25, first paragraph: Minor detail, but I cannot arrive at the total of 2274 patients from the Table 3. The total may reflect subjects used to derive criteria as well as the test groups?||We appreciate the reviewer noticing this error. We have replaced 2274 with 1834 (after subtracting 56 patients from the paper by Maynard et al., 2003).|
|Typos||Page 8, Process for Study Selection, paragraph 3: "out" for our.||We have made this correction.|
|Page 17, last paragraph, final sentence: "informing" for in forming.||We have reworded this sentence.|
|3||General||This reviewer believes said report by Coeytaux et al. is thorough, clear, well organized, and appropriate in scope and methods. The report's results and conclusions are clearly stated and well suppported. The authors have done an in-depth and fair analysis of the available literature and present their findings in a way that is certain to be of interest to different audiences.||We very much appreciate this assessment.|
|In essence, in the target population (patients with low to intermediate risk for CAD who present with acute onset of chest pain or other symptoms of ACS in the outpatient setting), the authors report there is insufficient published data to evaluate the utility of ECG-based signal analysis technologies. Their literature search did not reveal published articles describing the performance of most eligible devices indentified. While far from conclusive, the data does suggests that sensitivity of one such device, the PRIME ECG device (body surface mapping technology) may be higher than that of the standard 12-lead ECG in identifying patients with ACS who have CAD or ischemic heart disease.||Acknowledged|
|The authors concluded that further research is needed to determine the circumstances, if any, where these new technologies may add to the diagnostic accuracy of the standard 12-lead ECG in this population. They note that test performance of these devices "needs to be linked to clinically important outcomes through modeling of longitudinal studies"||Acknowledged|
|ES||The Executive Summary (ES) is only 8 pages long and yet it covers all key components of the report in sufficient detail to offer the reader a focused yet excellent summary of the introduction and background, methods, results and discussion of the findings.||Acknowledged|
|The ES clearly outlines that "this report focuses on patients who do not meet STEMI or STEMI-equivalent criteria on the standard 12-lead ECG". They authors note that "the 'purpose of this technology assessment is to summarize the clinical and scientific evidence for commercially available ECG-based signal analysis technologies used to evaluate patients with chest pain or other symptoms suggestive of ACS among patients at low to intermediate risk for CAD." Findings are as outlined above.||Acknowledged|
|Introduction: Background||The Introduction/Background section is very good. There is a summary of the epidemiology of Coronary Artery Disease (CAD) and an explanation of the differences between ischemia versus infarction. The authors also sections that include a clear definition of Acute Coronary Syndrome (ACS), diagnostic testing and risk stratification for CAD, the role and limitations of ECG in the Diagnostic Workup of CAD and ACS, and on the their evaluation of emerging ECG-based technologies. The objectives of this report are well outlined in this section.||Acknowledged|
|Introduction||Specifically, the authors note that the purpose of this report was to summarize the clinical and scientific evidence for commercially available ECG-based signal analysis technologies used to evaluate patients with chest pain at low to intermediate risk for CAD, or with a clinical presentation consistent with ACS. They go on to clarify that the report does not address the use of these technologies either to screen asymptomatic individuals for CAD or to evaluate patients at high risk for CAD. They note for example that they exclude ECG-based signal analysis technologies for measuring heart rate variability or tests aimed at predicting malignant arrhythmias. These clarifications are pertinent and important.||Acknowledged|
|Since many patients with ACS seek care at facilities that do not have Cardiac Catheterization Laboratories, the authors should consider addressing this issue and how it relates to potential transfers to other institutions as it is pertinent to the use of the technology under investigation.||We revised the "Evaluating Emerging ECG-based Technologies" section in the Introduction to address this issue.|
|Methods||The methodology used was described well and it seems appropriate for the charge of the analysis.||Acknowledged|
|The authors identified key questions (KQs) that appropriately addressed the issues at hand and synthesized the existing literature on these technologies accordingly. The KQs were: <verbatim excerpt from report not included here>.||Acknowledged|
|The authors developed an analytic framework based on generally accepted hierarchical model of diagnostic accuracy. This approach guided their research questions, search strategy, data abstraction elements, and evaluations. While a comprehensive systematic review of the English-literature was required for KQs 1 and 2, (indexed in PubMed, FDA Web site, Google, etc.), the data extracted from the eligible studies for each of these questions were different.||Acknowledged|
|For KQ 1, in addition to the comprehensive review, they also gathered and collated information from the FDA, device manufacturers and other relevant sources and summarized information about commonly used diagnostic tests, procedures and strategies.||Acknowledged|
|For KQ 2, the objective of the literature search was to synthesize the available literature regarding ECG-based signal analysis technologies that may be potentially applicable to the diagnosis of CAD in patients without such known preexisting diagnosis but presenting with chest pain or other sign or symptoms suggestive of ACS.||Acknowledged|
|The authors specifically excluded devices that used other imaging technologies such as echocardiogram, coronary angiography and ultimately also magnetocardiography (for this last one involved application of more than a single devices and is not readily available in most facilities). They did identify electrocardiography, mathematical analysis of ECG signals and vectorcardiography.||Acknowledged|
|After subsequent discussion with representatives from CMS, the authors limited their focus to devices that: <verbatim excerpt from report not included here>.||Acknowledged|
|The authors outline the methodology used and list the search term employed. While two investigators screened for titles and abstracts, the document was selected for full text review if either investigator deemed the study potentially eligible.
General eligibility criteria included: <verbatim excerpt from report not included here>.
Exclusions: <verbatim excerpt from report not included here>.
Appropriate data abstraction and peer review process then followed
|Results||KQ 1a—Devices and Methods for ECG-based Signal Analysis There were 11 devices identified, all available for purchase in the US including: 4 that used signal averaging, 1 that uses body surface mapping, 2 that use mathematical analysis, 2 that use high-frequency QRS analysis and 2 that use vectorcardiography. Of these, 8 of these have been cleared for marketing by the FDA.||Acknowledged|
|KQ 1b—Gold Standard Tests
- Coronary Artery Disease. The authors note in patients at low to intermediate risk for CAD, the diagnostic the standard ECG accuracy is low for it detects signals emitted by myocardial cell, not the presence of atherosclerotic plaque in coronary arteries. For these reason, the authors appropriately argue that new technologies should be compared to coronary angiography or at least non-invasive cardiac stress tests.
- Acute Coronary Syndrome. The authors note that since ACS is a working diagnosis with patients with acute onset of chest pain, pending the establishment of a specific diagnosis, there is no diagnostic gold standard for ACS.
The authors reported that out of 1980 citations (for KQ 2 for literature search was not applicable to KQ 1), they screened 288 full-text articles after applying inclusion/exclusion criteria. Of these, only 14 studies (11 studies) could be used for data abstraction.
|KQ 2a—Evidence for Variability by Rater, Patient, or Device, The authors' strategy did not identify any study that reported intra-rater, intra-patient, or intra-device variability. One study evaluated inter-rater variability of the Prime ECG body surface mapping (BSM) device. In this study, of the 135 readings, ER physicians and BSM experts agreed on 39% of the negative test result readings, and 47% negative test result readings. Of the remaining readings, they 11% were interpreted a negative by ER physicians but positive by BMS experts and in 4% positive by ER physicians and negative by BMS experts.||Acknowledged|
KQ 2 b—Evidence for Test Performance
The other eligible device was the LP 3000 signal averaging system. Compared to the standard 12-lead ECG, the LP 3000 device had the same specificity but a higher sensitivity (70% versus 56%, p<0.01).
|KQ 2c—Evidence for Impact on Diagnostic Decisionmaking—No eligible studies found for KQ 2 c.||Acknowledged|
|KQ 2d—Evidence for Impact on Patient Outcome—The Optimal Cardiovascular Diagnostic Evaluation Enabling Faster Treatment of Myocardial Infarction (OCCULT MI) trial (a multicenter, prospective, cohort-blinded investigation) was the only eligible published study that addressed the questions at hand in the patient population of interest. Its primary aim of the study was to determine whether individuals with STEMI detected only by the PRIME ECG would have similar angiographic pathology, and morbidity and mortality rates than those with STEMI detected by standard ECG. A preplanned secondary analysis compared outcomes of patients with STEMI with patients without STEMI. In this study, while ST elevation detected by the PRIME ECG was associated with a nearly two-fold increased in mortality (odds ratio [OR] 11.2; 95% CI, 1.8 to 67), ST elevation on a standard 12-lead ECG was not predictive of adverse outcomes in this population.||Acknowledged|
|Another study followed patients who initially presented with symptoms suggestive of ACS, both through their hospital course of treatment and after discharge from the hospital. Discharge diagnoses of either MI or ACS were recorded, as were followup events defined as a repeated visit to the emergency department with chest pain or ischemic symptoms, recurrent MI, catheterization, revascularization, or death. Rates of MI or ACS diagnoses or followup events were not reported, but the authors considered these patient outcomes in two of three separate criterion standards to calculate the sensitivities and sensitivities of both standard ECGs and PRIME ECG.||Acknowledged|
|Discussion||The discussion, tables, figures, appendices and references were all appropriate.||We appreciate this assessment.|
1. Peer reviewers are not listed in alphabetical order.
2. If listed, page number, line number, or section refers to the draft report.
3. If listed, page number, line number, or section refers to the final report.
Table 2: Public Review Comments
|Reviewer1||Reviewer Affiliation2||Section3||Reviewer Comments||Author Response4|
John Strobeck, M.D., Ph.D.
Michael Imhoff, M.D. Ph.D.
Norbert Rainford, M.D.
Heart-Lung Associates, PC
Columbia Presbyterian Medical Center
|General||We are submitting a discussion of the draft Technology Assessment (TA) "Systematic Review of ECG-based Signal Analysis Technologies for the Evaluation of Patients with Acute Coronary Syndrome" prepared by the Duke Evidence-Based Practice Center under contract to the Center for Medicare Services (CMS). Our discussion relates to 1) the ambiguity in the meaning of the verbiage of Key Question 1 contained in the Draft TA, referenced above, which guided the literature search and selection of evidence presented on ECG-based Signal Analysis Technologies, 2) the overall value of the draft TA's approach to the evaluation of ECG-based signal analysis technologies which focused on either asymptomatic patients with risk factors or patients with acute coronary syndromes (ACS) with or without chest pain (i.e., Unstable Angina, Non-ST elevated myocardial infarction (non-STEMI), or ST-elevated myocardial infarction (STEMI)), and 3) the use of the acronym SAECG for "Signal Analysis" ECG to describe ECGbased Signal Analysis Technologies. The term SAECG has been long accepted to mean "Signal Averaged" ECG, for the description of technologies used to detect the myocardial substrate for ventricular arrhythmias, not myocardial ischemia.||
(1) We acknowledge that there is ambiguity in the verbiage of Key Question 1. Protocol for Technology Assessment reports does not allow for changing the key questions at this stage in the process.
(2) We acknowledge the comment that questions the overall value of the approach taken by the report.
(3) We acknowledge these comments and recognize that the terms "signal analysis" and "signal averaging" are not used consistently in the published literature. We used the same terminology in this report as we did in the prior report published in 2010.
|Currently in the US the incidence of obstructive coronary artery disease (CAD) is over 6 million people. Close to 750,000 patients are admitted to hospitals every year with acute myocardial infarctions, and over 500,000 patients die each year from myocardial infarctions or complications of obstructive CAD. Approximately 80% of the patients with obstructive CAD have a chronic form of the disease, which is typically not associated with symptoms or produces symptoms only with exertion or stress. These patients have an accumulation of intravascular, obstructive plaque that varies in composition and severity thereby producing different amounts of obstruction. When the obstructing plaque progresses and is accompanied by procoagulant expression and weakening of the fibrous cap it becomes vulnerable to rupture. An ACS results when there is plaque disruption, stimulated thrombogenesis, and thrombus formation with further obstruction or complete occlusion of the coronary artery. Until this event occurs, the patient is not considered to have suffered an ACS. Thus, ACS includes specific patient groups with unstable angina (ST-T depression or inversion without elevated markers), patients with non- STEMI (with elevated markers), or patients with STEMI (with elevated markers). A patient entering an emergency ward with complaints of chest pain may have ACS as a "working diagnosis" until proven or not proven, but is not considered an actual ACS patient until the workup is complete and a final diagnosis established.||Acknowledged|
|The prevalence of obstructive CAD in patients with risk factors, such as diabetes, and who are asymptomatic is not well known but can be significant. One study using CT angiography to diagnose coronary obstruction ( 50%) found that 80% of type 2 diabetics studied had coronary obstruction of 50% or more and most obstructions were not calcified.||Acknowledged|
|The vast majority of the patients with both obstructive CAD and identifiable risk factors has atypical or intermittent symptoms of myocardial ischemia and/or altered myocardial function and do not have an ACS. It is vitally important, in order to improve long-term outcomes for the 2 majority of CAD patients, that ECG-based technologies be developed for this large, non-ACS population. Such technologies should be able to accurately detect the presence of coronary obstruction, prior to the development of an ACS, and help the physician determine whether an interventional treatment path is needed or whether medical therapy and lifestyle alteration will be sufficient to prevent (or even possibly reverse) disease or symptom progression.||Acknowledged|
|With this background in mind, the Coverage and Analysis Group of CMS commissioned the above referenced draft TA. They created key questions to be explored by the Duke Evidencebased Practice Center which guided Duke's systematic literature review.||Acknowledged|
|Therefore, the Multifunction Cardiogram (MCG) (Premier Heart, Inc.) previously considered by the final TA published in 2010 (Project ID # CRDD1008 ), was not considered in this draft TA. The four prospective blinded validation clinical trials that included 1076 patients from heterogeneous populations scheduled for elective coronary angiography, that directly assessed the ability of MCG, a functional analytic tool using two lead ECG signal sources in patients at rest, to detect myocardial ischemia caused by significant coronary artery obstruction, (defined as stenosis of 50% of the diameter of the left main coronary artery or stenosis of 70% of the diameter of a major epicardial vessel), were excluded from the draft TA. (4, 5, 6, and 7) In these trials, patients had an MCG and shortly thereafter, underwent coronary angiography. The MCG result was compared to the angiographic result by independent blinded observers to determine the ability of MCG to predict the result of angiography.||
The reviewers raise important points; we appreciate this opportunity to explain why we excluded (for this report) the four studies that were included in the prior report. We excluded the studies by Grube et al., 2007, and Grube et al., 2008, because the authors stated, "no patients presented with ACS at the time of study."
We excluded the studies by Hosokawas et al., 2008, and Weiss et al., 2002, because both included patients with a known history of CAD and neither reported information about the patient population that suggested the patients had ACS at the time of study.
|An additional clinical trial that included 116 patients with low to intermediate risk of CAD, who underwent paired comparison of MCG and SPECT myocardial perfusion imaging, was published in October 2011 (2), after the authors of the draft TA concluded their evidence search. It showed MCG had considerably better accuracy (89% vs 47%) at detecting anatomically relevant coronary obstruction (70%) than SPECT myocardial perfusion imaging (MPI). In addition, this trial showed similar results in sensitivity- 91%; specificity- 87%; negative predictive value- 92%; and positive predictive value—86%), to those reported in the other studies described above.||We appreciate the reviewers bringing to our attention this recently published and important study. We did not include this study in the final report because the authors stated, "The patients in this study ... did not have an acute coronary syndrome."|
|A meta-analysis of the first four MCG trials (3) showed an overall coronary obstructive disease prevalence of only 43.4% by coronary angiography. This post-angiography incidence of CAD is statistically similar to the findings of Patel, et. al. (1) which found an incidence of significant CAD (70%), as determined by coronary angiography, of 37.6 to 41% in a patient population very similar to the populations in which MCG was studied. Therefore, we believe, the patients in the MCG trials were at low to intermediate pre-test risk of having CAD, just as were those in the Patel paper.||We concur that the patient populations in these studies are similar and that they represent populations in which MCG and other ECG-based devices have the potential to play an important role in the diagnosis and/or clinical management. We excluded these studies from this current report because of the focus on patients with ACS, which represent a patient population that is not represented in the MCG trials or meta-analysis of these trials. This exclusive focus on patients with ACS was not made clear in the draft report.|
|As stated by the authors of the Patel study: "Our data support ongoing efforts to improve overall strategies for patient selection, including, but not limited to improving the quality of non-invasive testing in order to determine the optimal decision-making algorithm for the evaluation of suspected obstructive coronary artery disease."||Acknowledged|
|The development of MCG is meant to address the unmet clinical need identified by the Patel study. What is MCG? The Multifunction Cardiogram or MCG, is a systems-analysis tool that uses resting surface ECG data from leads II and V5 to build a mathematic model to detect chronic and/or acute, local and/or global myocardial ischemia due to underlying obstructive coronary artery disease. It is clearly not a form of Signal Averaging ECG (SAECG) technology or any other modified or enhanced traditional ECG waveform analysis platform, but rather an entirely new methodology based on a multifunction, mathematical analysis of the electro-mechanical function of the heart in the frequency and time domains, and an analysis of the integrity of that function over multiple cardiac cycles, not a small portion of one cycle.||Acknowledged|
|The MCG obtains information from the electromechanical data imbedded between the two resting cardiac electrical signal sources (lead V5 and II) of an individual. Instead of merely retrieving summed information about the electrical activity of myocytes at a single time point during a single cardiac cycle, MCG obtains information about the dynamic interface between the myocardium and intra-cardiac blood flow over multiple complete cardiac cycles. This makes it possible to model, quantify, and understand the ongoing stress-strain interaction resulting in the ability to identify chronic ischemic alterations that are too subtle to be detectable via the traditional ECG. The results of the mathematical transformations of the resting signals from an individual's ECG are matched to a database of 40,000 individuals with confirmed cardiac (and other) diagnoses (including 13,000 confirmed as not having heart disease of any type and 27,000 with confirmed pathology) that has been normalized for age and sex. The MCG completely ignores the traditional ECG waveforms (i.e., P wave, QRS complex, ST segment, T wave, etc.) and is a completely new approach to the analysis of cardiac electrical activity.||We appreciate this clear description of the MCG.|
|After reading the draft TA, we became concerned that despite the presence of several blinded trials of the MCG, a meta-analysis of the published data, and a recent trial confirming the accuracy of MCG in detecting relevant obstructive CAD as compared to SPECT MPI, it appeared that the Duke Evidence-based Practice Center may not have completely understood the MCG technology or how it works. It is our hope that by calling attention to the confusion created by the first Key Question and the actual prevalence of obstructive CAD in the population of patients included in the published clinical trials of MCG, the authors will re-evaluate their interpretation and the intent of the first key question and re-consider their decision not to include published trials of MCG in the draft TA and include them in the final TA.||We appreciate the reviewers' request. A possibly incomplete or faulty understanding of the MCG technology is not the reason for our having excluded the aforementioned studies. We also recognize that studies we included in this report included some patients who had known CAD at the time of study. The intent of the KQ 1 was to identify all eligible ECG-based devices, of which the MCG is clearly one. We therefore included the MCG in the results of this key question. The intent of KQ 2, however, was to focus exclusively on the potential of ECG-based technology to aid in the diagnosis and/or management of ACS.|
|Finally, we would like to suggest the use of the term "Signal Analysis" ECG, or SAECG, be abandoned because it is inconsistent with the long accepted use of SAECG to mean "Signal Averaged ECG" which has an entirely different meaning than that intended by this report. Even more importantly, this term should be abandoned because it will almost surely cause confusion in the payor and physician communities if it continues to be used to mean "Signal Analysis ECG" because "Signal Analysis" has no common meaning in the cardiology community.||We have eliminated the use of the acronym SAECG to avoid ambiguity.|
|We would sincerely appreciate the ability to either meet with the authors of the draft TAt or participate in a conference call with them and include two or three authors of the MCG metaanalysis paper to discuss some of the fine technical details of the MCG technology and have the opportunity to consider the authors' points in greater depth. Please contact us regarding the possibility of arranging a meeting or conference call.||
The reviewers may contact the Agency for Healthcare Research and Quality Technology Assessment program to discuss this topic at:
Agency for Healthcare Research and Quality
1. Names are alphabetized by last name. Those who did not disclose name are labeled "Anonymous Reviewer 1," "Anonymous Reviewer 2," etc.
2. Affiliation is labeled "NA" for those who did not disclose affiliation.
3. If listed, page number, line number, or section refers to the draft report.
4. If listed, page number, line number, or section refers to the final report.
Page originally created November 2011