Assessment of Radiation Therapy Technologists' Workload and Situation Awareness: Monitoring 2 Versus 3 Collocated Display Monitors

Purpose

This study aimed to assess the effect of monitoring 2 versus 3 collocated displays on radiation therapist technologists’ (RTTs) workload (WL) and situation awareness (SA) during routine treatment delivery tasks.

Methods and Materials

Seven RTTs completed 4 simulated treatment delivery scenarios (2 scenarios per experimental condition; 2 vs 3 collocated displays) in a within-subject experiment. WL was subjectively measured using the National Aeronautics and Space Administration (NASA) Task Load Index, and objectively measured using eye activity measures. SA was subjectively measured using the SA rating technique, and objectively measured using the SA global assessment technique. Two-tailed paired t tests were conducted to test for differences in means when parametric assumptions were satisfied, otherwise Wilcoxon signed-rank tests were conducted. A .05 level of significance was applied to all statistical tests.

Results

No statistically and clinically significant differences were observed between monitoring 2 versus 3 monitors on eye tracking measures (blink rate: 9.4 [4.8] vs 9.6 [4.0]; task evoked pupillary response: 0.16 [0.14] vs 0.21 [0.15]; NASA Task Load Index: 34.7 [19.8] vs 35.3 [20.4]; SA rating technique: 19.3 [6.2] vs 19.5 [7.0]; and SA global assessment technique scores: 100 [0] vs 100 [0]).

Conclusions

Our preliminary findings suggest that monitoring 3 collocated displays by 1 RTT does not impact WL and SA compared with monitoring 2 collocated displays. Only 2 of many possible configurations were investigated. If institutions removed the 3rd display based on the results of this study, there could be unforeseen error(s) if that display helped in situations not assessed in this study.

Pilot Study Using Neurofeedback as a Tool to Reduce Surgical Resident Burnout

BACKGROUND

Burnout is prevalent among surgical residents. Neurofeedback is a technique to train the brain in self-regulation skills. We aimed to assess the impact of neurofeedback on the cognitive workload and personal growth areas of surgery residents with burnout and depression.

STUDY DESIGN

Fifteen surgical residents with burnout (Maslach Burnout Inventory [MBI] score > 27) and depression (Patient Health Questionnaire-9 Depression Screen [PHQ-9] score >10), from 1 academic institution, were enrolled and participated in this institutional review board-approved prospective study. Ten residents with more severe burnout and depression scores were assigned to receive 8 weeks of neurofeedback treatments, and 5 others with less severe symptoms were treated as controls. Each participant's cognitive workload (or mental effort) was assessed initially, and again after treatment via electroencephalogram (EEG) while the subjects performed n-back working memory tasks. Analysis of variance (ANOVA) tested for significance between the degree of change in the treatment and control groups. Each subject was also asked to rate changes in growth areas, such as sleep and stress.

RESULTS

Both groups showed high cognitive workload in the pre-assessment. After the neurofeedback intervention, the treatment group showed a significant (p < 0.01) improvement in cognitive workload via EEG during the working memory task. These differences were not noted in the control group. There was significant correlation between time (NFB sessions) and average improvement in all growth areas (r = 0.98) CONCLUSIONS: Residents demonstrated high levels of burnout, correlating with EEG patterns indicative of post-traumatic stress disorder. There was a notable change in cognitive workload after the neurofeedback treatment, suggesting a return to a more efficient neural network.

Impact of Simulation-Based Training on Radiation Therapists' Workload, Situation Awareness, and Performance

Purpose

This study aimed to assess the impact of simulation-based training intervention on radiation therapy therapist (RTT) mental workload, situation awareness, and performance during routine quality assurance (QA) and treatment delivery tasks.

Methods and Materials

As part of a prospective institutional review board-approved study, 32 RTTs completed routine QA and treatment delivery tasks on clinical scenarios in a simulation laboratory. Participants, randomized to receive (n = 16) versus not receive (n = 16) simulation-based training had pre- and postintervention assessments of mental workload, situation awareness, and performance. We used linear regression models to compare the postassessment scores between the study groups while controlling for baseline scores. Mental workload was quantified subjectively using the NASA Task Load Index. Situation awareness was quantified subjectively using the situation awareness rating technique and objectively using the situation awareness global assessment technique. Performance was quantified based on procedural compliance (adherence to preset/standard QA timeout tasks) and error detection (detection and correction of embedded treatment planning errors).

Results

Simulation-based training intervention was associated with significant improvements in overall performance (P < .01), but had no significant impact on mental workload or subjective/objective quantifications of situation awareness.

Conclusions

Simulation-based training might be an effective tool to improve RTT performance of QA-related tasks.

Effect of Simulation-Based Training and Neurofeedback Interventions on Radiation Technologists' Workload, Situation Awareness, and Performance

PURPOSE

Our purpose was to assess the effect of a combined intervention - simulation-based training supported by neurofeedback sessions - on radiation technologists' (RTs') workload, situation awareness, and performance during routine quality assurance and treatment delivery tasks.

METHODS AND MATERIALS

As part of a prospective institutional review board approved study, 32 RTs previously randomized to receive versus not receive simulation-based training focused on patient safety were again randomized to receive versus not receive a 3-week neurofeedback intervention (8 sessions of alpha-theta protocol) focused on stress reduction as well as conscious precision, strong focus, and ability to solve arising problems. Perceived workload was quantified using the NASA Task Load Index. Situation awareness was quantified using the situation awareness rating technique. Performance score was calculated using procedural compliance with time-out components and error detection.

RESULTS

RTs randomized to simulation-based training followed by neurofeedback sessions demonstrated no significant changes in perceived workload or situation awareness scores, but did have better performance compared with other study groups (P < .01).

CONCLUSIONS

This finding is encouraging and provides basis for using neurofeedback as means to possibly augment performance improvements gained during simulation-based training.

Impact of Workspace Design on Radiation Therapist Technicians' Physical Stressors, Mental Workload, Situation Awareness, and Performance

PURPOSE

Our purpose was to assess the effect of workspace configuration on radiation therapists' (RTs) physical stressors, mental workload (WL), situational awareness (SA), and performance during routine treatment delivery tasks in a simulated environment.

METHODS AND MATERIALS

Fourteen RTs were randomized to 2 workspace configurations while performing 4 simulated scenarios: current (not ergonomically optimized; n = 7) and enhanced (ergonomically optimized, n = 7). Physical stressors were objectively assessed using a rapid upper limb assessment tool. Mental WL was measured at the end of each simulated scenario subjectively using the NASA Task-Load Index and objectively throughout the scenario using eye-tracking metrics (pupil diameter and blink rate). SA was measured at the end of each simulated scenario subjectively using the situation awareness and review technique. Performance was measured objectively via assessment of time-out compliance, error detection, and procedural compliance. Analysis of variance was used to test the effect of workspace configuration on physical stressors, mental WL, SA, and performance.

RESULTS

The enhanced configuration significantly reduced physical stressors (rapid upper limb assessment; P < .01) and resulted in a higher rate of time-out compliance (P = .01) compared with current workspace configuration. No significant effect on other metrics was measured.

CONCLUSIONS

Our results suggest that an ergonomically designed workspace may minimize physical stressors and improve the performance of RTs.

Incorporating Human Factors Analysis and Classification System (HFACS) Into Analysis of Reported Near Misses and Incidents in Radiation Oncology

PURPOSE

Human factors analysis and classification system (HFACS) is a framework for investigation into causation of human errors. We herein assess whether radiation oncology professionals, with brief training, can conduct HFACS on reported near misses or safety incidents (NMSIs) in a reliable (eg, with a high level of agreement) and practical (eg, timely and with user satisfaction) manner.

METHODS AND MATERIALS

We adapted a classical HFACS framework by selecting and modifying main headings, subheadings, and nano-codes that were most likely to apply to radiation oncology settings. The final modified HFACS included 3 main headings, 8 subheadings, and 20 nano-codes. The modified HFACS was first tested in a simulated trial on 8 NMSI and was analyzed by 5 to 10 radiation oncology professionals, with 2 endpoints: (1) agreement among participants at the main-heading, subheading, and nano-code level, and (2) time to complete the analysis. We then performed a prospective trial integrating this approach into a weekly NMSI review meeting, with 10 NMSIs analyzed by 8 to 13 radiation oncology professionals with the same endpoints, while also collecting survey data on participants' satisfaction.

RESULTS

In the simulated trial, agreement among participants was 85% on the main headings, 73% on the subheadings, and 70% on the nano-codes. Participants needed, on average, 16.4 minutes (standard deviation, 5.7 minutes) to complete an analysis. In the prospective trial, agreement between participants was 81% on the main headings, 75% on the subheadings, and 74% on the nano-codes. Participants needed, on average, 8.3 minutes (standard deviation, 4.7 minutes) to complete an analysis. The average satisfaction with the proposed HFACS approach was 3.9 (standard deviation 1.0) on a scale from 1 to 5.

CONCLUSIONS

This study demonstrates that, after relatively brief training, radiation oncology professionals were able to perform HFACS analysis in a reliable and timely manner and with a relatively high level of satisfaction.

Association of the Usability of Electronic Health Records With Cognitive Workload and Performance Levels Among Physicians

IMPORTANCE

Current electronic health record (EHR) user interfaces are suboptimally designed and may be associated with excess cognitive workload and poor performance.

OBJECTIVE

To assess the association between the usability of an EHR system for the management of abnormal test results and physicians' cognitive workload and performance levels.

DESIGN, SETTING, AND PARTICIPANTS

This quality improvement study was conducted in a simulated EHR environment. From April 1, 2016, to December 23, 2016, residents and fellows from a large academic institution were enrolled and allocated to use either a baseline EHR (n = 20) or an enhanced EHR (n = 18). Data analyses were conducted from January 9, 2017, to March 30, 2018.

INTERVENTIONS

The EHR with enhanced usability segregated in a dedicated folder previously identified critical test results for patients who did not appear for a scheduled follow-up evaluation and provided policy-based decision support instructions for next steps. The baseline EHR displayed all patients with abnormal or critical test results in a general folder and provided no decision support instructions for next steps.

MAIN OUTCOMES AND MEASURES

Cognitive workload was quantified subjectively using NASA-Task Load Index and physiologically using blink rates. Performance was quantified according to the percentage of appropriately managed abnormal test results.

RESULTS

Of the 38 participants, 25 (66%) were female. The 20 participants allocated to the baseline EHR compared with the 18 allocated to the enhanced EHR demonstrated statistically significantly higher cognitive workload as quantified by blink rate (mean [SD] blinks per minute, 16 [9] vs 24 [7]; blink rate, -8 [95% CI, -13 to -2]; P = .01). The baseline group showed statistically significantly poorer performance compared with the enhanced group who appropriately managed 16% more abnormal test results (mean [SD] performance, 68% [19%] vs 98% [18%]; performance rate, -30% [95% CI, -40% to -20%]; P < .001).

CONCLUSIONS AND RELEVANCE

Relatively basic usability enhancements to the EHR system appear to be associated with better physician cognitive workload and performance; this finding suggests that next-generation systems should strip away non-value-added EHR interactions, which may help physicians eliminate the need to develop their own suboptimal workflows.

Quantification of baseline pupillary response and task-evoked pupillary response during constant and incremental task load

The methods employed to quantify the baseline pupil size and task-evoked pupillary response (TEPR) may affect the overall study results. To test this hypothesis, the objective of this study was to assess variability in baseline pupil size and TEPR during two basic working memory tasks: constant load of 3-letters memorisation-recall (10 trials), and incremental load memorisation-recall (two trials of each load level), using two commonly used methods (1) change from trail/load specific baseline, (2) change from constant baseline. Results indicated that there was a significant shift in baseline between the trails for constant load, and between the load levels for incremental load. The TEPR was independent of shifts in baseline using method 1 only for constant load, and method 2 only for higher levels of incremental load condition. These important findings suggest that the assessment of both the baseline and methods to quantify TEPR are critical in ergonomics application, especially in studies with small number of trials per subject per condition. Practitioner Summary: Quantification of TEPR can be affected by shifts in baseline pupil size that are most likely affected by non-cognitive factors when other external factors are kept constant. Therefore, quantification methods employed to compute both baseline and TEPR are critical in understanding the information processing of humans in practical ergonomics settings.

Identifying Factors and Root Causes Associated With Near-Miss or Safety Incidents in Patients Treated With Radiotherapy: A Case-Control Analysis

Purpose:

To identify factors associated with a near-miss or safety incident (NMSI) in patients undergoing radiotherapy and identify common root causes of NMSIs and their relationship with incident severity.

Methods:

We retrospectively studied NMSIs filed between October 2014 and April 2016. We extracted patient-, treatment-, and disease-specific data from patients with an NMSI (n = 200; incident group) and a similar group of control patients (n = 200) matched in time, without an NMSI. A root cause and incident severity were determined for each NMSI. Univariable and multivariable analyses were performed to determine which specific factors were contributing to NMSIs. Multivariable logistic regression was used to determine root causes of NMSIs and their relationship with incident severity.

Results:

NMSIs were associated with the following factors: head and neck sites (odds ratio [OR], 5.2; P = .01), image-guided intensity-modulated radiotherapy (OR, 3; P = .009), daily imaging (OR, 7; P < .001), and tumors staged as T2 (OR, 3.3; P = .004). Documentation and scheduling errors were the most common root causes (29%). Communication errors were more likely to affect patients ( P < .001), and technical treatment delivery errors were most associated with a higher severity score ( P = .005).

Conclusion:

Several treatment- and disease-specific factors were found to be associated with an NMSI. Overall, our results suggest that complexity (eg, head and neck, image-guided intensity-modulated radiotherapy, and daily imaging) might be a contributing factor for an NMSI. This promotes an idea of developing a more dedicated and robust quality assurance system for complex cases and highlights the importance of a strong reporting system to support a safety culture.

Improving radiation oncology providers' workload and performance: Can simulation-based training help?

PURPOSE

To help with ongoing safety challenges in radiation therapy (RT), the objective of this research was to develop and assess the impact of a simulation-based training intervention on radiation oncology providers' workload and performance during treatment planning and quality assurance (QA) tasks.

METHODS AND MATERIALS

Eighteen radiation oncology professionals completed routine treatment planning and QA tasks on 2 clinical scenarios in a simulation laboratory as part of a prospective institutional review board-approved study. Workload was measured at the end of each assessment/scenario using the NASA Task-Load Index. Performance was quantified based on procedural compliance (adherence to preset/standard QA tasks), time-to-scenario completion, and clinically relevant performance. Participants were then randomized to receive (vs not receive) simulation-based training intervention (eg, standardized feedback on workload and performance) and underwent repeat measurements of workload and performance. Pre- and postintervention changes in workload and performance from participants who received (vs did not receive) were compared using 2-way analysis of variance.

RESULTS

Simulation-based training was associated with significant improvements in procedural compliance (P = .01) and increases in time-to-scenario completion (P < .01) but had no significant impact on subjective workload or clinically relevant performance.

CONCLUSION

Simulation-based training may be a tool to improve procedural compliance of RT professionals and to acquire new skills and knowledge to proactively maintain RT professionals' preoccupation with patient safety.

Toward a better understanding of task demands, workload, and performance during physician-computer interactions

OBJECTIVE

To assess the relationship between (1) task demands and workload, (2) task demands and performance, and (3) workload and performance, all during physician-computer interactions in a simulated environment.

METHODS

Two experiments were performed in 2 different electronic medical record (EMR) environments: WebCIS (n = 12) and Epic (n = 17). Each participant was instructed to complete a set of prespecified tasks on 3 routine clinical EMR-based scenarios: urinary tract infection (UTI), pneumonia (PN), and heart failure (HF). Task demands were quantified using behavioral responses (click and time analysis). At the end of each scenario, subjective workload was measured using the NASA-Task-Load Index (NASA-TLX). Physiological workload was measured using pupillary dilation and electroencephalography (EEG) data collected throughout the scenarios. Performance was quantified based on the maximum severity of omission errors.

RESULTS

Data analysis indicated that the PN and HF scenarios were significantly more demanding than the UTI scenario for participants using WebCIS (P < .01), and that the PN scenario was significantly more demanding than the UTI and HF scenarios for participants using Epic (P < .01). In both experiments, the regression analysis indicated a significant relationship only between task demands and performance (P < .01).

DISCUSSION

Results suggest that task demands as experienced by participants are related to participants' performance. Future work may support the notion that task demands could be used as a quality metric that is likely representative of performance, and perhaps patient outcomes.

CONCLUSION

The present study is a reasonable next step in a systematic assessment of how task demands and workload are related to performance in EMR-evolving environments.

Quantifying the impact of cross coverage on physician's workload and performance in radiation oncology

PURPOSE

To quantitatively assess the difference in workload and performance of radiation oncology physicians during radiation therapy treatment planning tasks under the conditions of "cross coverage" versus planning a patient with whom they were familiar.

METHODS AND MATERIALS

Eight physicians (3 experienced faculty physicians and 5 physician residents) performed 2 cases. The first case represented a "cross-coverage" scenario where the physicians had no prior information about the case to be planned. The second exposure represented a "regular-coverage" scenario where the physicians were familiar with the patient case to be planned. Each case involved 3 tasks to be completed systematically. Workload was assessed both subjectively (perceived) using National Aeronautics and Space Administration-Task Load Index (NASA-TLX), and objectively (physiological) throughout the task using eye data (via monitoring pupil size and blink rate). Performance of each task and the case was measured using completion time. Subjective willingness to approve or disapprove the generated plan was obtained after completion of the case only.

RESULTS

Forty-eight perceived and 48 physiological workload assessments were obtained. Overall, results revealed a significant increase in perceived workload (high NASA-TLX score) and decrease in performance (longer completion time and reduced approval rate) during cross coverage. There were nonsignificant increases in pupil diameter and decreases in the blink rate during cross-coverage versus regular-coverage scenario. In both cross-coverage and regular-coverage scenarios the level of experience did not affect workload and performance.

CONCLUSIONS

The cross-coverage scenario significantly increases perceived workload and degrades performance versus regular coverage. Hence, to improve patient safety, efforts must be made to develop policies, standard operating procedures, and usability improvements to electronic medical record and treatment planning systems for "easier" information processing to deal with cross coverage, while recognizing strengths and limitations of human performance.