The use of the CUSUM chart method for surveillance of learning effects and quality of care in endovascular procedures

Training on a virtual reality cricothyroidotomy simulator improves skills and transfers to a simulated procedure

Objective

The virtual airway skills trainer (VAST) is a virtual reality simulator for training in cricothyroidotomy (CCT). The goal of the study is to test the effectiveness of training and transfer of skills of the VAST-CCT.

Methods

Two groups, control (no training) and simulation (2 weeks of proficiency-based training), participated in this study. Subjects in the control condition did not receive any training on the task whereas those in the simulation received a proficiency-based training on the task during a period of 2 weeks. Two weeks post-training, both groups performed CCT on the TraumaMan to demonstrate the transfer of skills.

Results

A total of (n=20) subjects participated in the study. The simulation group performed better than the control group at both the post-test (p<0.001) and retention test (p<0.001) on the simulator. The cumulative sum analysis showed that all subjects in the simulation group reached proficiency with acceptable failure rate within the 2 weeks of training. On the transfer test, the simulation group performed better on skin cut (p<0.001), intubation (p<0.001) and total score (p<0.001) than the control group.

Conclusions

The VAST-CCT is effective in training and skills transfer for the CCT procedure.

Level of evidence

Not applicable. Simulator validation study.

Statistical Process Control Charts for Monitoring Next-Generation Sequencing and Bioinformatics Turnaround in Precision Medicine Initiatives

Purpose

Precision oncology, such as next generation sequencing (NGS) molecular analysis and bioinformatics are used to guide targeted therapies. The laboratory turnaround time (TAT) is a key performance indicator of laboratory performance. This study aims to formally apply statistical process control (SPC) methods such as CUSUM and EWMA to a precision medicine programme to analyze the learning curves of NGS and bioinformatics processes.

Patients and Methods

Trends in NGS and bioinformatics TAT were analyzed using simple regression models with TAT as the dependent variable and chronologically-ordered case number as the independent variable. The M-estimator "robust" regression and negative binomial regression were chosen to serve as sensitivity analyses to each other. Next, two popular statistical process control (SPC) approaches which are CUSUM and EWMA were utilized and the CUSUM log-likelihood ratio (LLR) charts were also generated. All statistical analyses were done in Stata version 16.0 (StataCorp), and nominal P < 0.05 was considered to be statistically significant.

Results

A total of 365 patients underwent successful molecular profiling. Both the robust linear model and negative binomial model showed statistically significant reductions in TAT with accumulating experience. The EWMA and CUSUM charts of overall TAT largely corresponded except that the EWMA chart consistently decreased while the CUSUM analyses indicated improvement only after a nadir at the 82^nd case. CUSUM analysis found that the bioinformatics team took a lower number of cases (54 cases) to overcome the learning curve compared to the NGS team (85 cases).

Conclusion

As NGS and bioinformatics lead precision oncology into the forefront of cancer management, characterizing the TAT of NGS and bioinformatics processes improves the timeliness of data output by potentially spotlighting problems early for rectification, thereby improving care delivery.

Validation of the VBLaST pattern cutting task: a learning curve study

BACKGROUND

Mastery of laparoscopic skills is essential in surgical practice and requires considerable time and effort to achieve. The Virtual Basic Laparoscopic Skill Trainer (VBLaST-PC^©) is a virtual simulator that was developed as a computerized version of the pattern cutting (PC) task in the Fundamentals of Laparoscopic Surgery (FLS) system. To establish convergent validity for the VBLaST-PC^©, we assessed trainees' learning curves using the cumulative summation (CUSUM) method and compared them with those on the FLS.

METHODS

Twenty-four medical students were randomly assigned to an FLS training group, a VBLaST training group, or a control group. Fifteen training sessions, 30 min in duration per session per day, were conducted over 3 weeks. All subjects completed pretest, posttest, and retention test (2 weeks after posttest) on both the FLS and VBLaST^© simulators. Performance data, including time, error, FLS score, learning rate, learning plateau, and CUSUM score, were analyzed.

RESULTS

The learning curve for all trained subjects demonstrated increasing performance and a performance plateau. CUSUM analyses showed that five of the seven subjects reached the intermediate proficiency level but none reached the expert proficiency level after 150 practice trials. Performance was significantly improved after simulation training, but only in the assigned simulator. No significant decay of skills after 2 weeks of disuse was observed. Control subjects did not show any learning on the FLS simulator, but improved continually in the VBLaST simulator.

CONCLUSIONS

Although VBLaST^©- and FLS-trained subjects demonstrated similar learning rates and plateaus, the majority of subjects required more than 150 trials to achieve proficiency. Trained subjects demonstrated improved performance in only the assigned simulator, indicating specificity of training. The virtual simulator may provide better opportunities for learning, especially with limited training exposure.

Cumulative sum analysis score and phacoemulsification competency learning curve

AIM

To use the cumulative sum analysis score (CUSUM) to construct objectively the learning curve of phacoemulsification competency.

METHODS

Three second-year residents and an experienced consultant were monitored for a series of 70 phacoemulsification cases each and had their series analysed by CUSUM regarding posterior capsule rupture (PCR) and best-corrected visual acuity. The acceptable rate for PCR was <5% (lower limit h) and the unacceptable rate was >10% (upper limit h). The acceptable rate for best-corrected visual acuity worse than 20/40 was <10% (lower limit h) and the unacceptable rate was >20% (upper limit h). The area between lower limit h and upper limit h is called the decision interval.

RESULTS

There was no statistically significant difference in the mean age, sex or cataract grades between groups. The first trainee achieved PCR CUSUM competency at his 22^nd case. His best-corrected visual acuity CUSUM was in the decision interval from his third case and stayed there until the end, never reaching competency. The second trainee achieved PCR CUSUM competency at his 39^th case. He could reach best-corrected visual acuity CUSUM competency at his 22^nd case. The third trainee achieved PCR CUSUM competency at his 41^st case. He reached best-corrected visual acuity CUSUM competency at his 14^th case.

CONCLUSION

The learning curve of competency in phacoemulsification is constructed by CUSUM and in average took 38 cases for each trainee to achieve it.

Looking for a Learning Curve in EVAR Based on the Zenith Stent Graft

The purpose of this study was to report the learning curve of endovascular aneurysm repair (EVAR) based on the Zenith stent graft (Cook Medical Inc., Bloomington, IN). In the last 9 years, 101 patients were treated with a Zenith stent graft. To display the learning curve, a cumulative sum (CUSUM) failure analysis curve of the 30-day technical success rate was calculated. For detailed analysis, our EVAR patient cohort was chronologically divided into three groups. Technical and clinical results, basic patient parameters, and procedural data were compared. The CUSUM graph indicated an initial sharp rise within the first 35 cases and a plateau thereafter. The 30-day technical success rate significantly increased from the first to the second group (83 vs. 100%; p = 0.019), as did the primary technical success rate (66 vs. 97%; p = 0.001). EVAR based on the Zenith stent graft required ∼35 cases to reach a stably high rate of short-term technical success.

Statistical methods to monitor risk factors in a clinical database: example of a national cardiac surgery registry

BACKGROUND

Comparison of outcomes requires adequate risk adjustment for differences in patient risk and the type of intervention performed. Both unintentional and intentional misclassification (also called gaming) of risk factors might lead to incorrect benchmark results. Therefore, misclassification of risk factors should be detected. We investigated the use of statistical process control techniques to monitor the frequency of risk factors in a clinical database.

METHODS AND RESULTS

A national population-based study was performed using simulation and statistical process control. All patients who underwent cardiac surgery between January 1, 2007, and December 31, 2009, in all 16 cardiothoracic surgery centers in the Netherlands were included. Data on 46 883 consecutive cardiac surgery interventions were extracted. The expected risk factor frequencies were based on 2007 and 2008 data. Monthly frequency rates of 18 risk factors in 2009 were monitored using a Shewhart control chart, exponentially weighted moving average chart, and cumulative sum chart. Upcoding (ie, gaming) in random patients was simulated and detected in 100% of the simulations. Subtle forms of gaming, involving specifically high-risk patients, were more difficult to identify (detection rate of 44%). However, the accompanying rise in mean logistic European system for cardiac operative risk evaluation (EuroSCORE) was detected in all simulations.

CONCLUSIONS

Statistical process control in the form of a Shewhart control chart, exponentially weighted moving average, and cumulative sum charts provide a means to monitor changes in risk factor frequencies in a clinical database. Surveillance of the overall expected risk in addition to the separate risk factors ensures a high sensitivity to detect gaming. The use of statistical process control for risk factor surveillance is recommended.

Characterizing the learning curve of the VBLaST-PT(©) (Virtual Basic Laparoscopic Skill Trainer)

BACKGROUND

Mastering laparoscopic surgical skills requires considerable time and effort. The Virtual Basic Laparoscopic Skill Trainer (VBLaST-PT(©)) is being developed as a computerized version of the peg transfer task of the Fundamentals of Laparoscopic Surgery (FLS) system using virtual reality technology. We assessed the learning curve of trainees on the VBLaST-PT(©) using the cumulative summation (CUSUM) method and compared them with those on the FLS to establish convergent validity for the VBLaST-PT(©).

METHODS

Eighteen medical students from were assigned randomly to one of three groups: control, VBLaST-training, and FLS-training. The VBLaST and the FLS groups performed a total of 150 trials of the peg-transfer task over a 3-week period, 5 days a week. Their CUSUM scores were computed based on predefined performance criteria (junior, intermediate, and senior levels).

RESULTS

Of the six subjects in the VBLaST-training group, five achieved at least the "junior" level, three achieved the "intermediate" level, and one achieved the "senior" level of performance criterion by the end of the 150 trials. In comparison, for the FLS group, three students achieved the "senior" criterion and all six students achieved the "intermediate" and "junior" criteria by the 150th trials. Both the VBLaST-PT(©) and the FLS systems showed significant skill improvement and retention, albeit with system specificity as measured by transfer of learning in the retention test: The VBLaST-trained group performed better on the VBLaST-PT(©) than on FLS (p = 0.003), whereas the FLS-trained group performed better on the FLS than on VBLaST-PT(©) (p = 0.002).

CONCLUSIONS

We characterized the learning curve for a virtual peg transfer task on the VBLaST-PT(©) and compared it with the FLS using CUSUM analysis. Subjects in both training groups showed significant improvement in skill performance, but the transfer of training between systems was not significant.