Assessment of competency in clinical measurement: comparison of two forms of sequential test and sensitivity of test error rates to parameter choice

The Accuracy of the Learning-Curve Cumulative Sum Method in Assessing Brachial Plexus Block Competency

BACKGROUND

The learning-curve cumulative sum method (LC-CUSUM) and its risk-adjusted form (RA-LC-CUSUM) have been proposed as performance-monitoring methods to assess competency during the learning phase of procedural skills. However, scarce data exist about the method's accuracy. This study aimed to compare the accuracy of LC-CUSUM forms using historical data consisting of sequences of successes and failures in brachial plexus blocks (BPBs) performed by anesthesia residents.

METHODS

Using historical data from 1713 BPB performed by 32 anesthesia residents, individual learning curves were constructed using the LC-CUSUM and RA-LC-CUSUM methods. A multilevel logistic regression model predicted the procedure-specific risk of failure incorporated in the RA-LC-CUSUM calculations. Competency was defined as a maximum 15% cumulative failure rate and was used as the reference for determining the accuracy of both methods.

RESULTS

According to the LC-CUSUM method, 22 residents (84.61%) attained competency after a median of 18.5 blocks (interquartile range [IQR], 14-23), while the RA-LC-CUSUM assigned competency to 20 residents (76.92%) after a median of 17.5 blocks (IQR, 14-25, P = .001). The median failure rate at reaching competency was 6.5% (4%-9.75%) under the LC-CUSUM and 6.5% (4%-9%) for the RA-LC-CUSUM method ( P = .37). The sensitivity of the LC-CUSUM (85%; 95% confidence interval [CI], 71%-98%) was similar to the RA-LC-CUSUM method (77%; 95% CI, 61%-93%; P = .15). Identical specificity values were found for both methods (67%; 95% CI, 29%-100%, P = 1).

CONCLUSIONS

The LC-CUSUM and RA-LC-CUSUM methods were associated with substantial false-positive and false-negative rates. Also, small lower limits for the 95% CIs around the accuracy measures were observed, indicating that the methods may be inaccurate for high-stakes decisions about resident competency at BPBs.

Assessment of the learning curve of peribulbar blocks using the Learning-Curve Cumulative Sum Method (LC-CUSUM): an observational study

INTRODUCTION

This study aimed to assess the learning curves of peribulbar anesthesia and estimate the number of blocks needed to attain proficiency.

METHODS

Anonymized records of sequential peribulbar blocks performed by first-year anesthesia residents were analyzed. The block sequential number and the outcomes were extracted from each record. Success was defined as a complete sensory and motor block of the eye, and failure was defined as an incomplete block requiring supplemental local anesthetic injections or general anesthesia. Learning curves using the LC-CUSUM method were constructed, aiming for acceptable and unacceptable failure rates of 10% and 20%, and 10% probability of type I and II errors. Simulations were used to determine the proficiency limit h0. Residents whose curves reached h0 were considered proficient. The Sequential Probability Ratio Test Cumulative Sum Method (SPRT-CUSUM) was used for follow-up.

RESULTS

Thirty-nine residents performed 2076 blocks (median = 52 blocks per resident; Interquartile Range (IQR) [range] = 27-78 [4-132]). Thirty residents (77%) achieved proficiency after a median of 13 blocks (13-24 [13-24]).

CONCLUSIONS

The LC-CUSUM is a robust method for detecting resident proficiency at peribulbar anesthesia, defined as success rates exceeding 90%. Accordingly, 13 to 24 supervised double-injection peribulbar blocks are needed to attain competence at peribulbar anesthesia.

Learning Curve of USgHIFU Ablation for Uterine Fibroids: A Multi-Center Prospective Study

OBJECTIVES

To verify the stability of high-intensity focused ultrasound (HIFU) technology and the feasibility of training programs with learning curve cumulative summation (LC-CUSUM).

METHODS

A total of 12 physicians and 720 cases were equally assigned to the learning group and the control group, with 6 physicians and 360 cases per group. The learning group was treated by physicians without HIFU experience and the control group was treated by experienced physicians. Nonperfused volume (NPV) ratio was assessed by contrast-enhanced magnetic resonance imaging. Technical failure was defined as NPV ratio of uterine fibroids <70% and/or major complication, while <80% was set as a stricter standard of training qualification. LC-CUSUM was used to analyze the learning curve.

RESULTS

Physicians with or without HIFU experience in both groups achieved matchable NPV ratios, where a NPV ratio of 92.52% (16.06) was achieved by experienced physicians and 93.82% (16.95) by inexperienced physicians. No major complication was observed. The results of LC-CUSUM analysis showed that, with the standards of the NPV ratio of 70% or 80%, the learning group mastered the technique on the 11th case and the 16th case, respectively, while the control group was stable.

CONCLUSIONS

HIFU technology stayed stable in operation, with good safety and sound effectiveness, and was easy to learn. NPV ratio of 70% was considered as an appropriate indicator of training qualification. HIFU has remarkable prospects in achieving a NPV ratio of ≥80% without safety being compromised.

Competency Assessment in Focused Cardiac Ultrasound—Can the Use of Sequential Testing Help Tailor Training Requirements?

Varying numbers of scans are required by different professional bodies before focused cardiac ultrasound (FCU) competence is assumed. It has been suggested that innovation in the assessment of FCU competence is needed and that competency assessment needs to be more individualized. We report our experience of how the use of sequential testing may help personalize the assessment of FCU competence.