A method for evaluating computer programs for electrocardiographic interpretation. 3. Reproducibility testing and the sources of program errors

Automated ECG Interpretation—A Brief History from High Expectations to Deepest Networks

This article traces the development of automated electrocardiography from its beginnings in Washington, DC around 1960 through to its current widespread application worldwide. Changes in the methodology of recording ECGs in analogue form using sizeable equipment through to digital recording, even in wearables, are included. Methods of analysis are considered from single lead to three leads to twelve leads. Some of the influential figures are mentioned while work undertaken locally is used to outline the progress of the technique mirrored in other centres. Applications of artificial intelligence are also considered so that the reader can find out how the field has been constantly evolving over the past 50 years.

The History and Challenges of SCP-ECG: The Standard Communication Protocol for Computer-Assisted Electrocardiography

Ever since the first publication of the standard communication protocol for computer-assisted electrocardiography (SCP-ECG), prENV 1064, in 1993, by the European Committee for Standardization (CEN), SCP-ECG has become a leading example in health informatics, enabling open, secure, and well-documented digital data exchange at a low cost, for quick and efficient cardiovascular disease detection and management. Based on the experiences gained, since the 1970s, in computerized electrocardiology, and on the results achieved by the pioneering, international cooperative research on common standards for quantitative electrocardiography (CSE), SCP-ECG was designed, from the beginning, to empower personalized medicine, thanks to serial ECG analysis. The fundamental concept behind SCP-ECG is to convey the necessary information for ECG re-analysis, serial comparison, and interpretation, and to structure the ECG data and metadata in sections that are mostly optional in order to fit all use cases. SCP-ECG is open to the storage of the ECG signal and ECG measurement data, whatever the ECG recording modality or computation method, and can store the over-reading trails and ECG annotations, as well as any computerized or medical interpretation reports. Only the encoding syntax and the semantics of the ECG descriptors and of the diagnosis codes are standardized. We present all of the landmarks in the development and publication of SCP-ECG, from the early 1990s to the 2009 International Organization for Standardization (ISO) SCP-ECG standards, including the latest version published by CEN in 2020, which now encompasses rest and stress ECGs, Holter recordings, and protocol-based trials.

Digitizing paper electrocardiograms: Status and challenges

The paper electrocardiogram (ECG) has been widely used for cardiac assessment for well over a century. ECGs can be obtained quickly and cheaply. For this reason, an ever-growing amount of paper ECG records continue to accumulate, some of which are stored into a paper-only format. Converting paper ECGs into digital form has been proposed as the most efficient means to store and analyze an otherwise cumbersome paper archive. In this article, a literature review was conducted for conversion algorithms, criticisms of said algorithms, applications, and standardization efforts. The algorithms were compared in tabulated form. Key functions that have advanced the conversion algorithms as well as remaining challenges are discussed.

New system for digital to analog transformation and reconstruction of 12-lead ECGs

INTRODUCTION

We describe initial validation of a new system for digital to analog conversion (DAC) and reconstruction of 12-lead ECGs. The system utilizes an open and optimized software format with a commensurately optimized DAC hardware configuration to accurately reproduce, from digital files, the original analog electrocardiographic signals of previously instrumented patients. By doing so, the system also ultimately allows for transmission of data collected on one manufacturer's 12-lead ECG hardware/software into that of any other.

MATERIALS AND METHODS

To initially validate the system, we compared original and post-DAC re-digitized 12-lead ECG data files (∼5-minutes long) in two types of validation studies in 10 patients. The first type quantitatively compared the total waveform voltage differences between the original and re-digitized data while the second type qualitatively compared the automated electrocardiographic diagnostic statements generated by the original versus re-digitized data.

RESULTS

The grand-averaged difference in root mean squared voltage between the original and re-digitized data was 20.8 µV per channel when re-digitization involved the same manufacturer's analog to digital converter (ADC) as the original digitization, and 28.4 µV per channel when it involved a different manufacturer's ADC. Automated diagnostic statements generated by the original versus reconstructed data did not differ when using the diagnostic algorithm from the same manufacturer on whose device the original data were collected, and differed only slightly for just 1 of 10 patients when using a third-party diagnostic algorithm throughout.

CONCLUSION

Original analog 12-lead ECG signals can be reconstructed from digital data files with accuracy sufficient for clinical use. Such reconstructions can readily enable automated second opinions for difficult-to-interpret 12-lead ECGs, either locally or remotely through the use of dedicated or cloud-based servers.

Measurement fidelity of heart rate variability signal processing: the devil is in the details

Heart rate variability (HRV) is a particularly valuable quantitative marker of the flexibility and balance of the autonomic nervous system. Significant advances in software programs to automatically derive HRV have led to its extensive use in psychophysiological research. However, there is a lack of systematic comparisons across software programs used to derive HRV indices. Further, researchers report meager details on important signal processing decisions making synthesis across studies challenging. The aim of the present study was to evaluate the measurement fidelity of time- and frequency-domain HRV indices derived from three predominant signal processing software programs commonly used in clinical and research settings. Triplicate ECG recordings were derived from 20 participants using identical data acquisition hardware. Among the time-domain indices, there was strong to excellent correspondence (ICC(avg)=0.93) for SDNN, SDANN, SDNNi, rMSSD, and pNN50. The frequency-domain indices yielded excellent correspondence (ICC(avg)=0.91) for LF, HF, and LF/HF ratio, except for VLF which exhibited poor correspondence (ICC(avg)=0.19). Stringent user-decisions and technical specifications for nuanced HRV processing details are essential to ensure measurement fidelity across signal processing software programs.

PC-Based ECG waveform recognition-validation of novel software against a reference ECG database

BACKGROUND

PC-based ECG measurements must cope with normal as well as pathological ECGs in a reliable manner. EClysis, a software for ECG measurements was tested against reference values from the Common Standards for Quantitative Electrocardiography (CSE) database.

METHODS

Digital ECGs (12 leads, 500 Hz) were recorded by the CSE project. Data Set 3 contains reference values for 125 ECGs (33 normal and 92 pathological). Median values of measurements by 11 computer programs and by five cardiologists, respectively, refer to the earliest P and QRS onsets and to the latest P, QRS, and T offsets in any lead of a selected (index) beat. EClysis automatically measured all ECGs, without user interference.

RESULTS

The PQRST points were correctly detected but in two ECGs with AV block II-III. The software was not designed to detect atrial activity in atrial fibrillation (n = 9) and flutter (n = 1). In one case of atrial fibrillation, atrial activity interfered with positioning of QRS and T offsets. Regression coefficients between EClysis and CSE (software-generated and human) were above 0.95 (P < 0.0001). The confidence intervals were 95% for the slope and the intercept of the regression lines.

CONCLUSIONS

The PC-based detection and analysis of PQRST points showed a high level of agreement with the CSE database reference values.

Diagnostic performance of a computer-based ECG rhythm algorithm

We examined the accuracy of computer-based rhythm interpretation from one electrocardiograph manufacturer (GE Healthcare Technologies MUSE software 005C) in 4297 consecutive recordings in a university hospital setting. Overreading was performed by either of 2 experienced cardiologists, and all disagreements with the initial computer rhythm statement were reviewed by the second cardiologist to achieve physician consensus used as the "gold standard" for rhythm diagnosis. Overall, 13.2% (565/4297) of computer-based rhythm statements required revision, but excluding tracings with pacemakers, the revision rate was 7.8% (307/3954), including 3.8% involving the primary rhythm diagnosis and 3.9% involving definition of ectopic complexes. The false-negative rate for sinus rhythm was only 1.3%, but a computer diagnosis of sinus rhythm was incorrect in 9.9% of other rhythms. The false-negative rate for atrial fibrillation was 9.2%, whereas a computer diagnosis of atrial fibrillation was incorrect in 1.1% of other rhythms, including sinus. Computer diagnosis of paced rhythms remains problematic, and physician overreading to correct computer-based electrocardiogram rhythm diagnoses remains mandatory.

Variability in ECG computer interpretation. Analysis of individual complexes vs analysis of a representative complex

Variability in the electrocardiogram (ECG) can be due to extrinsic noise or can be caused by intrinsic factors, such as changes in the volume conductor or in the heart itself. Computer programs for the interpretation of the ECG base their diagnostic classification on one set of measurements that is derived from a representative PQRST complex or that is computed by taking the median from the measurements for each complex in the recording. However, these methods may fail to do justice to the intrinsic variability that may be present in the ECG. An alternative method is proposed: derive a set of measurements from each complex in the recording, classify each individual complex separately, and then combine the individual classifications into one final classification. This procedure has been evaluated on a validated database (n = 1,220) using an ECG computer program. Total accuracy against the clinical evidence increased from 69.8% for the interpretations of the averaged complexes to 71.2% for the combined interpretations of the individual complexes (p < 0.001). The effect of beat-to-beat variation on the measurements and classifications is demonstrated and the influence of extrinsic and intrinsic variability is assessed.

The reproducibility of interpretation of 10 computer ECG systems by means of a microprocessor-based ECG signal generator

The use of computer ECG systems has become widespread in Japan, and many new models are available. In order to provide a method for comparative evaluation, a microprocessor-based ECG signal generator was built. This generator was used to compare 10 of the computer ECG systems which are currently available in Japan. Since the generator can produce a precisely defined wave form, it is particularly suited for testing the reproducibility of diagnostic interpretation. Several deficiencies of some current ECG analysis systems were noted. Specifically in three systems, there was interference of the interpretation of morphology with the interpretation of rhythm when WPW-like patterns were analyzed. The current evaluation concentrates on the reproducibility of interpretation in the tested systems. As the library of test ECG patterns is expanded in the future, the technique will also become applicable to the evaluation of diagnostic accuracy.

Comparative diagnostic performance of the Telemed computer ECG program

One thousand consecutive ECG's from an ambulatory population of patients with suspected or proven cardiac disease were evaluated using two versions of the Telemed computerized ECG system. Only minor differences were found between the two programs. In version 6 vs. version 5, 87% vs. 90% of 287 normal ECG's were correctly classified and 93% vs. 96% of abnormal ECG's were correctly classified; the percent of acceptable diagnostic agreement was 86.2% and 87.4% respectively (NS). The sensitivity for arrhythmia detection, transmural inferior infarction and ST-T wave abnormalities was slightly greater in version 6. The increased sensitivity was not accompanied by decreased specificity. The sensitivity for left ventricular hypertrophy decreased from 95.2% to 91.4% in version 6 with a slight increase in specificity (95.2% to 97.0%). In conclusion, criteria changes in the most recent version of the Telemed program have not resulted in a major change in diagnostic performance. Arrhythmia detection is slightly but not significantly improved.

QT interval measurement by a computer assisted program: a potentially useful clinical parameter

The duration of electrical systole (QT interval) was measured in 72 subjects (48 women and 24 men) who had normal coronary arteries and left ventricular function at cardiac catheterization (group 1). The same measurements were obtained in 100 patients with a normal ECG (from 40 women and 60 men referred to our institution and found normal on a noninvasive clinical basis) and compared to a double independent manual calculation (group 2). The computer assisted program was found reliable in QT interval measurements. In both study groups women showed longer QTc. No difference in QTc duration was seen in subjects taking beta-blockers prior to angiography. As compared to group 1, subjects of group 2 showed similar average QTc values. However, 9 out of 100 subjects of group 2 had abnormal QTc as compared with none of group 1 (p less than 0.05). QTc calculations may improve the usefulness of computer assisted programs in ECG interpretation. Present data can be used as reference values for normality. They stress in addition the necessity of introducing the heart rate correction for the interpretation of QT interval. This can help in stimulating prospective clinical studies to assess the value of QTc as an index of risk for cardiac dysrhythmias.

Computer evaluation of human circulation based on non-invasive methods

A discriminating program was developed on the basis of time, dynamic and simply calculated parameters of non-invasive tracings recorded in the supine position. Data were derived from ECG, PCG and the indirect carotid pulse curve. The optimal program, formed after 40 experimental processes, was in 85% agreement with the clinical diagnosis. To improve the decision process, we created a new 'test again' group, in addition to the healthy and sick groups. The 'test again' group included 16.5% of the examined subjects. At the same time, there was 75.6% agreement with the clinical diagnosis, and 7.9% disagreement. The risk factors, which could be demonstrated as part of the 'errors' called attention to undetected heart failure. The descriminating function found to be best, was fed into a small computer (R-10). Records for evaluation were entered on magnetic tape to the computer which measured automatically the necessary parameters and printed out the 'decision': 'healthy', 'test again!', or 'cardiac patient', as well as other data, such as systolic time intervals, etc. There is a wide potential application for automated computer system based on non-invasive parameters.

XYZ data interpreted by a 12-lead computer program using the derived electrocardiogram

The 12-lead electrocardiogram (ECG) derived from the Frank xyz signals was compared with the conventional 12-lead ECG using the Telemed computer system. In 100 cases studied. Telemed's interpretations were essentially similar in 77, but substantially different in 23. In the 23 cases, interpretations of the derived tracings tended to be more accurate in 14 cases, and less accurate in four cases. In the diagnosis of infarction the probability that the interpretation of the derived tracing will be correct more often was 90%. The better performance may have been related to closer agreement with the vectorcardiogram (VCG). As a substitute for the conventional ECG, the derived ECG offers the prospect of a computerized system that is more practical and more versatile than most currently used systems.

Clinical evaluation of automated processing of electrocardiograms by the Veterans Administration program (AVA 3.4)

Automated processing of electrocardiograms by the Veterans Administration program was evaluated for both agreement with physician interpretation and interpretative accuracy as assessed with nonelectrocardiographic criteria. One thousand unselected electrocardiograms were analyzed by two reviewer groups, one familiar and the other unfamiliar with the computer program. A significant number of measurement errors involving repolarization changes and left axis deviation occurred; however, interpretative disagreements related to statistical decision were largely language-related. Use of a printout with a more traditional format resulted in agreement with physician interpretation by both reviewer groups in more than 80 percent of cases. Overall sensitivity based on agreement with nonelectrocardiographic criteria was significantly greater with use of the computer program than with use of the conventional criteria utilized by the reviewers. This difference was particularly evident in the subgroup analysis of myocardial infarction and left ventricular hypertrophy. The degree of overdiagnosis of left ventricular hypertrophy and posteroinferior infarction was initially unacceptable, but this difficulty was corrected by adjustment of probabilities. Clinical acceptability of the Veterans Administration program appears to require greater physician education than that needed for other computer programs of electrocardiographic analysis; the flexibility of interpretation by statistical decision offers the potential for better diagnostic accuracy.

Limitations of the computer in electrocardiographic interpretation

Use of the computer for electrocardiographic interpretation has steadily increased over the past decade. Although acceptance by a majority of physicians has been slower than originally anticipated, it now appears assured. Nevertheless, the approach has limitations. These are primarily a result of the lack of objectivity in clinical electrocardiographic criteria for both measurement and diagnosis. The limitations are best judged by reviewing experience with a variety of programs. Currently this still involves abstracting from the reports of developers. Although not all developers have analyzed every portion of their programs, in general the data from one program apply to others; that is, all are with minor exceptions at the same "state of the art." Awareness of the limits of computer performance allows a physician to use the computer properly in his current electrocardiographic practice.