Exploration and Initial Development of Text Classification Models to Identify Health Information Technology Usability-Related Patient Safety Event Reports

Characterization of Safety Events Involving Technology in Primary and Community Care

BACKGROUND

The adoption of technology in health care settings is often touted as an opportunity to improve patient safety. While some adverse events can be reduced by health information technologies, technology has also been implicated in or attributed to safety events. To date, most studies on this topic have focused on acute care settings.

OBJECTIVES

To describe voluntarily reported safety events that involved health information technology in community and primary care settings in a large Canadian health care organization.

METHODS

Two years of safety events involving health information technology (2016-2018) were extracted from an online voluntary safety event reporting system. Events from primary and community care settings were categorized according to clinical setting, type of event, and level of harm. The Sittig and Singh sociotechnical system model was then used to identify the most prominent sociotechnical dimensions of each event.

RESULTS

Of 104 reported events, most (n = 85, 82%) indicated the event resulted in no harm. Public health had the highest number of reports (n = 45, 43%), whereas home health had the fewest (n = 7, 7%). Of the 182 sociotechnical concepts identified, many events (n = 61, 59%) mapped to more than one dimension. Personnel (n = 48, 46%), Workflow and Communication (n = 37, 36%), and Content (n = 30, 29%) were the most common. Personnel and Content together was the most common combination of dimensions.

CONCLUSION

Most reported events featured both technical and social dimensions, suggesting that the nature of these events is multifaceted. Leveraging existing safety event reporting systems to screen for safety events involving health information technology, and applying a sociotechnical analytic framework can aid health organizations in identifying, responding to, and learning from reported events.

Using Community Detection Techniques to Identify Themes in COVID-19-Related Patient Safety Event Reports

OBJECTIVES

The COVID-19 pandemic has transformed how healthcare is delivered to patients. As the pandemic progresses and healthcare systems continue to adapt, it is important to understand how these changes in care have changed patient care. This study aims to use community detection techniques to identify and facilitate analysis of themes in patient safety event (PSE) reports to better understand COVID-19 pandemic's impact on patient safety. With this approach, we also seek to understand how community detection techniques can be used to better identify themes and extract information from PSE reports.

METHODS

We used community detection techniques to group 2082 PSE reports from January 1, 2020, to January 31, 2021, that mentioned COVID-19 into 65 communities. We then grouped these communities into 8 clinically relevant themes for analysis.

RESULTS

We found the COVID-19 pandemic is associated with the following clinically relevant themes: (1) errors due to new and unknown COVID-19 protocols/workflows; (2) COVID-19 patients developing pressure ulcers; (3) unsuccessful/incomplete COVID-19 testing; (4) inadequate isolation of COVID-19 patients; (5) inappropriate/inadequate care for COVID-19 patients; (6) COVID-19 patient falls; (7) delays or errors communicating COVID-19 test results; and (8) COVID-19 patients developing venous thromboembolism.

CONCLUSIONS

Our study begins the long process of understanding new challenges created by the pandemic and highlights how machine learning methods can be used to understand these and similar challenges. Using community detection techniques to analyze PSE reports and identify themes within them can help give healthcare systems the necessary information to improve patient safety and the quality of care they deliver.

Developing an Analytical Pipeline to Classify Patient Safety Event Reports Using Optimized Predictive Algorithms

BACKGROUND

 Patient safety event reports provide valuable insight into systemic safety issues but deriving insights from these reports requires computational tools to efficiently parse through large volumes of qualitative data. Natural language processing (NLP) combined with predictive learning provides an automated approach to evaluating these data and supporting the work of patient safety analysts.

OBJECTIVES

 The objective of this study was to use NLP and machine learning techniques to develop a generalizable, scalable, and reliable approach to classifying event reports for the purpose of driving improvements in the safety and quality of patient care.

METHODS

 Datasets for 14 different labels (themes) were vectorized using a bag-of-words, tf-idf, or document embeddings approach and then applied to a series of classification algorithms via a hyperparameter grid search to derive an optimized model. Reports were also analyzed for terms strongly associated with each theme using an adjusted F-score calculation.

RESULTS

 F₁ score for each optimized model ranged from 0.951 ("Fall") to 0.544 ("Environment"). The bag-of-words approach proved optimal for 12 of 14 labels, and the naïve Bayes algorithm performed best for nine labels. Linear support vector machine was demonstrated as optimal for three labels and XGBoost for four of the 14 labels. Labels with more distinctly associated terms performed better than less distinct themes, as shown by a Pearson's correlation coefficient of 0.634.

CONCLUSIONS

 We were able to demonstrate an analytical pipeline that broadly applies NLP and predictive modeling to categorize patient safety reports from multiple facilities. This pipeline allows analysts to more rapidly identify and structure information contained in patient safety data, which can enhance the evaluation and the use of this information over time.

Development of a Taxonomy for Medication-Related Patient Safety Events Related to Health Information Technology in Pediatrics

BACKGROUND

Although electronic health records (EHRs) are designed to improve patient safety, they have been associated with serious patient harm. An agreed-upon and standard taxonomy for classifying health information technology (HIT) related patient safety events does not exist.

OBJECTIVES

We aimed to develop and evaluate a taxonomy for medication-related patient safety events associated with HIT and validate it using a set of events involving pediatric patients.

METHODS

We performed a literature search to identify existing classifications for HIT-related safety events, which were assessed using real-world pediatric medication-related patient safety events extracted from two sources: patient safety event reporting system (ERS) reports and information technology help desk (HD) tickets. A team of clinical and patient safety experts used iterative tests of change and consensus building to converge on a single taxonomy. The final devised taxonomy was applied to pediatric medication-related events assess its characteristics, including interrater reliability and agreement.

RESULTS

Literature review identified four existing classifications for HIT-related patient safety events, and one was iteratively adapted to converge on a singular taxonomy. Safety events relating to usability accounted for a greater proportion of ERS reports, compared with HD tickets (37 vs. 20%, p = 0.022). Conversely, events pertaining to incorrect configuration accounted for a greater proportion of HD tickets, compared with ERS reports (63 vs. 8%, p < 0.01). Interrater agreement (%) and reliability (kappa) were 87.8% and 0.688 for ERS reports and 73.6% and 0.556 for HD tickets, respectively.

DISCUSSION

A standardized taxonomy for medication-related patient safety events related to HIT is presented. The taxonomy was validated using pediatric events. Further evaluation can assess whether the taxonomy is suitable for nonmedication-related events and those occurring in other patient populations.

CONCLUSION

Wider application of standardized taxonomies will allow for peer benchmarking and facilitate collaborative interinstitutional patient safety improvement efforts.

The Effect of Electronic Health Record Usability Redesign on Annual Screening Rates in an Ambulatory Setting

OBJECTIVES

Improving the usability of electronic health records (EHR) continues to be a focus of clinicians, vendors, researchers, and regulatory bodies. To understand the impact of usability redesign of an existing, site-configurable feature, we evaluated the user interface (UI) used to screen for depression, alcohol and drug misuse, fall risk, and the existence of advance directive information in ambulatory settings.

METHODS

As part of a quality improvement project, based on heuristic analysis, the existing UI was redesigned. Using an iterative, user-centered design process, several usability defects were corrected. Summative usability testing was performed as part of the product development and implementation cycle. Clinical quality measures reflecting rolling 12-month rates of screening were examined over 8 months prior to the implementation of the redesigned UI and 9 months after implementation.

RESULTS

Summative usability testing demonstrated improvements in task time, error rates, and System Usability Scale scores. Interrupted time series analysis demonstrated significant improvements in all screening rates after implementation of the redesigned UI compared with the original implementation.

CONCLUSION

User-centered redesign of an existing site-specific UI may lead to significant improvements in measures of usability and quality of patient care.