Development and Implementation of Autoverification Rules for ELISA Results of HBV Serological Markers

Benefit analysis of the auto-verification system of intelligent inspection for microorganisms

In recent years, the automatic machine for microbial identification and antibiotic susceptibility tests has been introduced into the microbiology laboratory of our hospital, but there are still many steps that need manual operation. The purpose of this study was to establish an auto-verification system for bacterial naming to improve the turnaround time (TAT) and reduce the burden on clinical laboratory technologists. After the basic interpretation of the gram staining results of microorganisms, the appearance of strain growth, etc., the 9 rules were formulated by the laboratory technologists specialized in microbiology for auto-verification of bacterial naming. The results showed that among 70,044 reports, the average pass rate of auto-verification was 68.2%, and the reason for the failure of auto-verification was further evaluated. It was found that the main causes reason the inconsistency between identification results and strain appearance rationality, the normal flora in the respiratory tract and urine that was identified, the identification limitation of the mass spectrometer, and so on. The average TAT for the preliminary report of bacterial naming was 35.2 h before, which was reduced to 31.9 h after auto-verification. In summary, after auto-verification, the laboratory could replace nearly 2/3 of manual verification and issuance of reports, reducing the daily workload of medical laboratory technologists by about 2 h. Moreover, the TAT on the preliminary identification report was reduced by 3.3 h on average, which could provide treatment evidence for clinicians in advance.

Diagnostic accuracy of autoverification and guidance system for COVID-19 RT-PCR results

BACKGROUND

To date, most countries worldwide have declared that the pandemic of COVID-19 is over, while the WHO has not officially ended the COVID-19 pandemic, and China still insists on the personalized dynamic COVID-free policy. Large-scale nucleic acid testing in Chinese communities and the manual interpretation for SARS-CoV-2 nucleic acid detection results pose a huge challenge for labour, quality and turnaround time (TAT) requirements. To solve this specific issue while increase the efficiency and accuracy of interpretation, we created an autoverification and guidance system (AGS) that can automatically interpret and report the COVID-19 reverse transcriptase-polymerase chain reaction (RT-PCR) results relaying on computer-based autoverification procedure and then validated its performance in real-world environments. This would be conductive to transmission risk prediction, COVID-19 prevention and control and timely medical treatment for positive patients in the context of the predictive, preventive and personalized medicine (PPPM).

METHODS

A diagnostic accuracy test was conducted with 380,693 participants from two COVID-19 test sites in China, the Hong Kong Hybribio Medical Laboratory (n = 266,035) and the mobile medical shelter at a Shanghai airport (n = 114,658). These participants underwent SARS-CoV-2 RT-PCR from March 28 to April 10, 2022. All RT-PCR results were interpreted by laboratorians and by using AGS simultaneously. Considering the manual interpretation as gold standard, the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and accuracy were applied to evaluate the diagnostic value of the AGS on the interpretation of RT-PCR results.

RESULTS

Among the 266,035 samples in Hong Kong, there were 16,356 (6.15%) positive, 231,073 (86.86%) negative, 18,606 (6.99%) indefinite, 231,073 (86.86%, negative) no retest required and 34,962 (13.14%, positive and indefinite) retest required; the 114,658 samples in Shanghai consisted of 76 (0.07%) positive, 109,956 (95.90%) negative, 4626 (4.03%) indefinite, 109,956 (95.90%, negative) no retest required and 4702 (4.10%, positive and indefinite) retest required. Compared to the fashioned manual interpretation, the AGS is a procedure of high accuracy [99.96% (95%CI, 99.95-99.97%) in Hong Kong and 100% (95%CI, 100-100%) in Shanghai] with perfect sensitivity [99.98% (95%CI, 99.97-99.98%) in Hong Kong and 100% (95%CI, 100-100%) in Shanghai], specificity [99.87% (95%CI, 99.82-99.90%) in Hong Kong and 100% (95%CI, 99.92-100%) in Shanghai], PPV [99.98% (95%CI, 99.97-99.99%) in Hong Kong and 100% (95%CI, 99.99-100%) in Shanghai] and NPV [99.85% (95%CI, 99.80-99.88%) in Hong Kong and 100% (95%CI, 99.90-100%) in Shanghai]. The need for manual interpretation of total samples was dramatically reduced from 100% to 13.1% and the interpretation time fell from 53 h to 26 min in Hong Kong; while the manual interpretation of total samples was decreased from 100% to 4.1% and the interpretation time dropped from 20 h to 16 min at Shanghai.

CONCLUSIONS

The AGS is a procedure of high accuracy and significantly relieves both labour and time from the challenge of large-scale screening of SARS-CoV-2 using RT-PCR. It should be recommended as a powerful screening, diagnostic and predictive system for SARS-CoV-2 to contribute timely the ending of the COVID-19 pandemic following the concept of PPPM.

Combined strategy of knowledge-based rule selection and historical data percentile-based range determination to improve an autoverification system for clinical chemistry test results

Background

Current autoverification, which is only knowledge‐based, has low efficiency. Regular historical data analysis may improve autoverification range determination. We attempted to enhance autoverification by selecting autoverification rules by knowledge and ranges from historical data. This new system was compared with the original knowledge‐based system.

Methods

New types of rules, extreme values, and consistency checks were added and the autoverification workflow was rearranged to construct a framework. Criteria for creating rules for extreme value ranges, limit checks, consistency checks, and delta checks were determined by analyzing historical Zhongshan laboratory data. The new system's effectiveness was evaluated using pooled data from 20 centers. Efficiency improvement was assessed by a multicenter process.

Results

Effectiveness was evaluated by the true positive rate, true negative rate, and overall consistency rate, as compared to manual verification, which were 77.55%, 78.53%, and 78.3%, respectively for the new system. The original overall consistency rate was 56.2%. The new pass rates, indicating efficiency, were increased by 19%‒51% among hospitals. Further customization using individualized data increased this rate.

Conclusions

The improved system showed a comparable effectiveness and markedly increased efficiency. This transferable system could be further improved and popularized by utilizing historical data from each hospital.

The development of autoverification system of lymphocyte subset assays on the flow cytometry platform

OBJECTIVES

Peripheral blood lymphocyte subsets are important parameters for monitoring immune status; however, lymphocyte subset detection is time-consuming and error-prone. This study aimed to explore a highly efficient and clinically useful autoverification system for lymphocyte subset assays performed on the flow cytometry platform.

METHODS

A total of 94,402 lymphocyte subset test results were collected. To establish the limited-range rules, 80,427 results were first used (69,135 T lymphocyte subset tests and 11,292 NK, B, T lymphocyte tests), of which 15,000 T lymphocyte subset tests from human immunodeficiency virus (HIV) infected patients were used to set customized limited-range rules for HIV infected patients. Subsequently, 13,975 results were used for historical data validation and online test validation.

RESULTS

Three key autoverification rules were established, including limited-range, delta-check, and logical rules. Guidelines for addressing the issues that trigger these rules were summarized. The historical data during the validation phase showed that the total autoverification passing rate of lymphocyte subset assays was 69.65% (6,941/9,966), with a 67.93% (5,268/7,755) passing rate for T lymphocyte subset tests and 75.67% (1,673/2,211) for NK, B, T lymphocyte tests. For online test validation, the total autoverification passing rate was 75.26% (3,017/4,009), with 73.23% (2,191/2,992) for the T lymphocyte subset test and 81.22% (826/1,017) for the NK, B, T lymphocyte test. The turnaround time (TAT) was reduced from 228 to 167 min using the autoverification system.

CONCLUSIONS

The autoverification system based on the laboratory information system for lymphocyte subset assays reduced TAT and the number of error reports and helped in the identification of abnormal cell populations that may offer clues for clinical interventions.

A model to establish autoverification in the clinical laboratory

OBJECTIVES

Autoverification is the process of evaluating and validating laboratory results using predefined computer-based algorithms without human interaction. By using autoverification, all reports are validated according to the standard evaluation criteria with predefined rules, and the number of reports per laboratory specialist is reduced. However, creating and validating these rules are the most demanding steps for setting up an autoverification system. In this study, we aimed to develop a model for helping users establish autoverification rules and evaluate their validity and performance.

DESIGN & METHODS

The proposed model was established by analyzing white papers, previous study results, and national/international guidelines. An autoverification software (myODS) was developed to create rules according to the model and to evaluate the rules and autoverification rates. The simulation results that were produced by the software were used to demonstrate that the determined framework works as expected. Both autoverification rates and step-based evaluations were performed using actual patient results. Two algorithms defined according to delta check usage (Algorithm A and B) and three review limits were used for the evaluation.

RESULTS

Six hundred seventeen rules were created according to the proposed model. 1,976 simulation results were created for validation. Our results showed that manual review limits are the most critical step in determining the autoverification rate, and delta check evaluation is especially important for evaluating inpatients. Algorithm B, which includes consecutive delta check evaluation, had higher AV rates.

CONCLUSIONS

Systemic rule formation is a critical factor for successful AV. Our proposed model can help laboratories establish and evaluate autoverification systems. Rules created according to this model could be used as a starting point for different test groups.

Molecular Aggregation in Immune System Activation Studied by Dynamic Light Scattering

Determination of the concentration and size of the circulating immune complexes in the blood is an essential part of diagnostics of immune diseases. In this work, we suggest using the dynamic light scattering method to determine the sizes of circulating immune complexes in blood serum. By the dynamic light scattering spectrometer, we found that for healthy and sick donors, the size and concentration of circulating immune complexes differed significantly. The dynamics of formation of these complexes were also examined in this work. It was shown that the formation of immune complexes in the blood of healthy donors is faster than the same reactions in the blood serum of donors with diseases. The results can be used in the diagnostics of the immune status and detection of chronic inflammation. We can recommend the dynamic light scattering method for implementation in biomedical diagnostics.

Establishing and evaluating an auto-verification system of thalassemia gene detection results

The manual verification of gene tests is time-consuming and error prone. In this study, we try to explore a high-efficiency, clinically useful auto-verification system for gene detection of thalassemia. A series of verification elements were rooted in the auto-verification system. Consistency check was applied initially as one of the essential elements in our study. One hundred twenty-four archived cases were used to choose the consistency-check rules' indices from routine blood examination and hemoglobin electrophoresis by the receiver operating characteristic curves. Rule 1 and rule 2 established by the chosen indices were compared by their passing rate, consistency with manual validation, and error rate. Finally, 748 cases were used for verifying the system's feasibility by evaluating the passing rate, turn-around time (TAT), and error rate. The rule 2 had a higher passing rate (67.7% vs. 50.8%) and consistency (0.623 vs. 0.364) than the rule 1 with an error rate of zero. In a "live" valuation, the auto-verification system can reduce the TAT and error rate of verification by 51.5% and 0.13%, respectively, with a high passing rate of 82.8%. The auto-verification system for gene detection of thalassemia in this study can shorten the validation time, reduce errors, and enhance efficiency.

Design and evaluation of a LIS-based autoverification system for coagulation assays in a core clinical laboratory

Background

The autoverification system for coagulation consists of a series of rules that allow normal data to be released without manual verification. With new advances in medical informatics, the laboratory information system (LIS) has growing potential for the autoverification, allowing rapid and accurate verification of clinical laboratory tests. The purpose of the study is to develop and evaluate a LIS-based autoverification system for validation and efficiency.

Methods

Autoverification decision rules, including quality control, analytical error flag, critical value, limited range check, delta check and logical check, as well as patient’s historical information, were integrated into the LIS. Autoverification limited range was constructed based on 5 and 95% percentiles. The four most commonly used coagulation assays, prothrombin time (PT), activated partial thromboplastin time (APTT), thrombin time (TT), and fibrinogen (FBG), were followed by the autoverification protocols. The validation was assessed by the autoverification passing rate, the true-positive cases, the true-negative cases, the false-positive cases, the false-negative cases, the sensitivity and the specificity; the efficiency was evaluated in the turnaround time (TAT).

Results

A total of 157,079 historical test results of coagulation profiles from January 2016 to December 2016 were collected to determine the distribution intervals. The autoverification passing rate was 77.11% (29,165/37,821) based on historical patient data. In the initial test of the autoverification version in June 2017, the overall autoverification passing rate for the whole sample was 78.75% (11,257/14,295), with 892 true-positive cases, 11,257 true-negative cases, 2146 false-positive cases, no false-negative cases, sensitivity of 100% and specificity of 83.99%. After formal implementation of the autoverification system for 6 months, 83,699 samples were assessed. The average overall autoverification passing rate for the whole sample was 78.86% and the 95% confidence interval (CI) of the passing rate was [78.25, 79.59%]. TAT was reduced from 126 min to 101 min, which was statistically significant (P < 0.001, Mann-Whitney U test).

Conclusions

The autoverification system for coagulation assays based on LIS can halt the samples with abnormal values for manual verification, guarantee medical safety, minimize the requirements for manual work, shorten TAT and raise working efficiency.

Establishing and Evaluating Autoverification Rules with Intelligent Guidelines for Arterial Blood Gas Analysis in a Clinical Laboratory

Arterial blood gas (ABG) analysis is important for acutely ill patients and should be performed by qualified laboratorians. The existing manual verifications are tedious, time-consuming, and prone to send wrong reports. Autoverification uses computer-based rules to verify clinical laboratory test results without manual review. To date, no data are available on the use of autoverification for ABG analysis. All autoverification rules were established according to AUTO10-A. Additionally, the rules were established using retrospective patient data, and then validated by actual clinical samples in a "live" environment before go-live. The average autoverification passing rate was 75.5%. The turnaround time (TAT) was reduced by 33.3% (27 min vs 18 min). Moreover, the error rate fell to 0.05% after implementation. Statistical analysis resulted in a kappa statistic of 0.92 ( p < 0.01), indicating close agreement between autoverification and senior technician verification, and the chi-square value was 22.4 ( p < 0.01), indicating that the autoverification error rate was lower than the manual verification error rate. Results showed that implementing autoverification rules with intelligent guidelines for ABG analysis of patients with critical illnesses could decrease the number of samples requiring manual verification, reduce TAT, and eliminate errors, allowing laboratorians to concentrate more time on abnormal samples, patient care, and collaboration with physicians.

Designing and evaluating autoverification rules for thyroid function profiles and sex hormone tests

Purpose Following the analytical phase, the current practice of many hospital laboratories involves the manual verification of all test results followed by the production of the report. However, manual verification is a time-consuming and tedious process. In this paper, we provide a detailed description of how to design autoverification rules for thyroid function test profiles and sex hormones. Materials and methods We used DM2 (Data manager 2) to construct the algorithm and build the database for autoverification of thyroid function test profiles and sex hormones, with reference to Boolean logic, Auto 10-A and CLSI'88. The rules consist of checking quality control, instrument error flags, critical values, the analytical measurement range (AMR), the limit range, consistency check and delta check. Firstly, we established the rules in the DM2, collected clinical specimens for validation, then tested the rules in a 'live' environment. Results Agreement was achieved between manual verification by two senior laboratory personnel and verification using the autoverification rules in 99.78% of the cases. The total autoverification rate for all tests was 77.06%. Following implementation of the rules, the laboratory turnaround time (TAT) was reduced by 54.55% and staffing numbers fell from three to two whole time equivalents (WTE). Statistical analysis resulted in a kappa statistic of 0.99 ( P < 0.001). Moreover, after implementing the autoverification rules, the error rate fell to 0.04%, indicating that errors were almost completely eliminated. Conclusion Implementing autoverification rules can reduce TAT, minimize the number of samples that require manual verification and allow for a reduction in staffing numbers. It also allows laboratory staff to devote more time and effort to the handling of problematic test results and contributing to improved patient care.

Development and Implementation of a Coagulation Factor Testing Method Utilizing Autoverification in a High-volume Clinical Reference Laboratory Environment

Background:

Testing coagulation factor activities requires that multiple dilutions be assayed and analyzed to produce a single result. The slope of the line created by plotting measured factor concentration against sample dilution is evaluated to discern the presence of inhibitors giving rise to nonparallelism. Moreover, samples producing results on initial dilution falling outside the analytic measurement range of the assay must be tested at additional dilutions to produce reportable results.

Methods:

The complexity of this process has motivated a large clinical reference laboratory to develop advanced computer algorithms with automated reflex testing rules to complete coagulation factor analysis. A method was developed for autoverification of coagulation factor activity using expert rules developed with on an off the shelf commercially available data manager system integrated into an automated coagulation platform.

Results:

Here, we present an approach allowing for the autoverification and reporting of factor activity results with greatly diminished technologist effort.

Conclusions:

To the best of our knowledge, this is the first report of its kind providing a detailed procedure for implementation of autoverification expert rules as applied to coagulation factor activity testing. Advantages of this system include ease of training for new operators, minimization of technologist time spent, reduction of staff fatigue, minimization of unnecessary reflex tests, optimization of turnaround time, and assurance of the consistency of the testing and reporting process.