Participatory design of probability-based decision support tools for in-hospital nurses

Exploring home healthcare clinicians' needs for using clinical decision support systems for early risk warning

OBJECTIVES

To explore home healthcare (HHC) clinicians' needs for Clinical Decision Support Systems (CDSS) information delivery for early risk warning within HHC workflows.

METHODS

Guided by the CDS "Five-Rights" framework, we conducted semi-structured interviews with multidisciplinary HHC clinicians from April 2023 to August 2023. We used deductive and inductive content analysis to investigate informants' responses regarding CDSS information delivery.

RESULTS

Interviews with thirteen HHC clinicians yielded 16 codes mapping to the CDS "Five-Rights" framework (right information, right person, right format, right channel, right time) and 11 codes for unintended consequences and training needs. Clinicians favored risk levels displayed in color-coded horizontal bars, concrete risk indicators in bullet points, and actionable instructions in the existing EHR system. They preferred non-intrusive risk alerts requiring mandatory confirmation. Clinicians anticipated risk information updates aligned with patient's condition severity and their visit pace. Additionally, they requested training to understand the CDSS's underlying logic, and raised concerns about information accuracy and data privacy.

DISCUSSION

While recognizing CDSS's value in enhancing early risk warning, clinicians highlighted concerns about increased workload, alert fatigue, and CDSS misuse. The top risk factors identified by machine learning algorithms, especially text features, can be ambiguous due to a lack of context. Future research should ensure that CDSS outputs align with clinical evidence and are explainable.

CONCLUSION

This study identified HHC clinicians' expectations, preferences, adaptations, and unintended uses of CDSS for early risk warning. Our findings endorse operationalizing the CDS "Five-Rights" framework to optimize CDSS information delivery and integration into HHC workflows.

Nurses' perceptions of the design, implementation, and adoption of machine learning clinical decision support: A descriptive qualitative study

INTRODUCTION

The purpose of this study was to explore nurses' perspectives on Machine Learning Clinical Decision Support (ML CDS) design, development, implementation, and adoption.

DESIGN

Qualitative descriptive study.

METHODS

Nurses (n = 17) participated in semi-structured interviews. Data were transcribed, coded, and analyzed using Thematic analysis methods as described by Braun and Clarke.

RESULTS

Four major themes and 14 sub-themes highlight nurses' perspectives on autonomy in decision-making, the influence of prior experience in shaping their preferences for use of novel CDS tools, the need for clarity in why ML CDS is useful in improving practice/outcomes, and their desire to have nursing integrated in design and implementation of these tools.

CONCLUSION

This study provided insights into nurse perceptions regarding the utility and usability of ML CDS as well as the influence of previous experiences with technology and CDS, change management strategies needed at the time of implementation of ML CDS, the importance of nurse-perceived engagement in the development process, nurse information needs at the time of ML CDS deployment, and the perceived impact of ML CDS on nurse decision making autonomy.

CLINICAL RELEVANCE

This study contributes to the body of knowledge about the use of AI and machine learning (ML) in nursing practice. Through generation of insights drawn from nurses' perspectives, these findings can inform successful design and adoption of ML Clinical Decision Support.

Designing explainable AI to improve human-AI team performance: A medical stakeholder-driven scoping review

The rise of complex AI systems in healthcare and other sectors has led to a growing area of research called Explainable AI (XAI) designed to increase transparency. In this area, quantitative and qualitative studies focus on improving user trust and task performance by providing system- and prediction-level XAI features. We analyze stakeholder engagement events (interviews and workshops) on the use of AI for kidney transplantation. From this we identify themes which we use to frame a scoping literature review on current XAI features. The stakeholder engagement process lasted over nine months covering three stakeholder group's workflows, determining where AI could intervene and assessing a mock XAI decision support system. Based on the stakeholder engagement, we identify four major themes relevant to designing XAI systems - 1) use of AI predictions, 2) information included in AI predictions, 3) personalization of AI predictions for individual differences, and 4) customizing AI predictions for specific cases. Using these themes, our scoping literature review finds that providing AI predictions before, during, or after decision-making could be beneficial depending on the complexity of the stakeholder's task. Additionally, expert stakeholders like surgeons prefer minimal to no XAI features, AI prediction, and uncertainty estimates for easy use cases. However, almost all stakeholders prefer to have optional XAI features to review when needed, especially in hard-to-predict cases. The literature also suggests that providing both system- and prediction-level information is necessary to build the user's mental model of the system appropriately. Although XAI features improve users' trust in the system, human-AI team performance is not always enhanced. Overall, stakeholders prefer to have agency over the XAI interface to control the level of information based on their needs and task complexity. We conclude with suggestions for future research, especially on customizing XAI features based on preferences and tasks.

Exploring the role of professional identity in the implementation of clinical decision support systems-a narrative review

BACKGROUND

Clinical decision support systems (CDSSs) have the potential to improve quality of care, patient safety, and efficiency because of their ability to perform medical tasks in a more data-driven, evidence-based, and semi-autonomous way. However, CDSSs may also affect the professional identity of health professionals. Some professionals might experience these systems as a threat to their professional identity, as CDSSs could partially substitute clinical competencies, autonomy, or control over the care process. Other professionals may experience an empowerment of the role in the medical system. The purpose of this study is to uncover the role of professional identity in CDSS implementation and to identify core human, technological, and organizational factors that may determine the effect of CDSSs on professional identity.

METHODS

We conducted a systematic literature review and included peer-reviewed empirical studies from two electronic databases (PubMed, Web of Science) that reported on key factors to CDSS implementation and were published between 2010 and 2023. Our explorative, inductive thematic analysis assessed the antecedents of professional identity-related mechanisms from the perspective of different health care professionals (i.e., physicians, residents, nurse practitioners, pharmacists).

RESULTS

One hundred thirty-one qualitative, quantitative, or mixed-method studies from over 60 journals were included in this review. The thematic analysis found three dimensions of professional identity-related mechanisms that influence CDSS implementation success: perceived threat or enhancement of professional control and autonomy, perceived threat or enhancement of professional skills and expertise, and perceived loss or gain of control over patient relationships. At the technological level, the most common issues were the system's ability to fit into existing clinical workflows and organizational structures, and its ability to meet user needs. At the organizational level, time pressure and tension, as well as internal communication and involvement of end users were most frequently reported. At the human level, individual attitudes and emotional responses, as well as familiarity with the system, most often influenced the CDSS implementation. Our results show that professional identity-related mechanisms are driven by these factors and influence CDSS implementation success. The perception of the change of professional identity is influenced by the user's professional status and expertise and is improved over the course of implementation.

CONCLUSION

This review highlights the need for health care managers to evaluate perceived professional identity threats to health care professionals across all implementation phases when introducing a CDSS and to consider their varying manifestations among different health care professionals. Moreover, it highlights the importance of innovation and change management approaches, such as involving health professionals in the design and implementation process to mitigate threat perceptions. We provide future areas of research for the evaluation of the professional identity construct within health care.

Inpatient nurses' preferences and decisions with risk information visualization

OBJECTIVE

We examined the influence of 4 different risk information formats on inpatient nurses' preferences and decisions with an acute clinical deterioration decision-support system.

MATERIALS AND METHODS

We conducted a comparative usability evaluation in which participants provided responses to multiple user interface options in a simulated setting. We collected qualitative data using think aloud methods. We collected quantitative data by asking participants which action they would perform after each time point in 3 different patient scenarios.

RESULTS

More participants (n = 6) preferred the probability format over relative risk ratios (n = 2), absolute differences (n = 2), and number of persons out of 100 (n = 0). Participants liked average lines, having a trend graph to supplement the risk estimate, and consistent colors between trend graphs and possible actions. Participants did not like too much text information or the presence of confidence intervals. From a decision-making perspective, use of the probability format was associated with greater concordance in actions taken by participants compared to the other 3 risk information formats.

DISCUSSION

By focusing on nurses' preferences and decisions with several risk information display formats and collecting both qualitative and quantitative data, we have provided meaningful insights for the design of clinical decision-support systems containing complex quantitative information.

CONCLUSION

This study adds to our knowledge of presenting risk information to nurses within clinical decision-support systems. We encourage those developing risk-based systems for inpatient nurses to consider expressing risk in a probability format and include a graph (with average line) to display the patient's recent trends.

Development and usability of a Mobile Interactive Application (VCPW) for Vascular Crisis Prewarning after Skin Flap Transplantation

Background

In microsurgical tissue transfer, skin flap transplantation is frequently used to heal the surface of a wound. Effective microcirculation surveillance of the skin flap is crucial. However, with traditional monitoring methods-that is, clinical observation-vascular crisis can still occur, thereby impairing postoperative recovery. A smartphone application is required to assist health care professionals in the standardized collection of flap perfusion parameters for flap management.

Methods

The Vascular Crisis Prewarning Application was created using a design science research methodology that prioritizes users and problems. The system usability scale was used to assess the application's usability among medical practitioners. The application was used at the clinic from December 2020 to September 2022. The unplanned return to the operating room, time to diagnose vascular crisis, and flap survival rate were compared with and without the application.

Results

The application consisted of 5 modules: patient addition and basic information entry, flap labeling, flap observation, crisis warning, and case archiving. The average rating for the application's usability among medical practitioners was 97.95 score (SD 2.36). With the application, the time to detect vascular crisis reduced from 26.71 to 16.26 h (P < 0.001), the unplanned return to the operation room increased from 8.18% to 10.24% (P = 0.587), and the flap survival rate went from 94.55% to 99.21% (P = 0.083).

Conclusions

An easy-to-use flap perfusion monitoring and prewarning application for medical practitioners was produced using a user-centered development method. The application provided a more standardized and accurate platform for data collection in flap management and reduced the time to detect vascular crisis. Larger cohort studies are required in the future to better assess the full potential of the application.

Clinical Decision Support systems: A step forward in establishing the clinical laboratory as a decision maker hubA CDS system protocol implementation in the clinical laboratory

Background

New tools for health information technology have been developed in recent times, such as Clinical Decision Support (CDS) systems, which are any digital solutions designed to help healthcare professionals when making clinical decisions. The study aimed to show how we have adopted a CDS system in the San Juan de Alicante Clinical Laboratory and facilitate the implementation of our protocol in other clinical laboratories. We have user experience and the motivation to improve healthcare tools. The improvement, measurement, and monitoring of interventions and laboratory tests has been our motto for years.

Materials and methods

A descriptive research was conducted. All stages in the design of the project are as follows: 1. Set up a multidisciplinary workgroup. 2. Review patients' data. 3. Identify relevant data from main sources. 4. Design the likely outcomes. 5. Define a complete integration scenario. 6. Monitor and track the impact. To set up this protocol, two new software systems were implemented in our laboratory: AlinIQ CDS v8.2 as Rule Engine, and AlinIQ AIP Integrated Platform v1.6 as Business Intelligence (BI) tool.

Results

Our protocol shows the workflow and actions that can be done with a CDS system and also how it could be integrated with other monitoring systems, as well as some examples of KPIs and their outcomes.

Conclusions

CDS could be a great strategic asset for clinical laboratories to improve the integration of care, optimize the use of laboratory tests, and add more clinical value to physicians in the interpretation of results.

Behavioral Health Decision Support Systems and User Interface Design in the Emergency Department

OBJECTIVE

The objective of this qualitative study is to gauge physician sentiment about an emergency department (ED) clinical decision support (CDS) system implemented in multiple adult EDs within a university hospital system. This CDS system focuses on predicting patients' likelihood of ED recidivism and/or adverse opioid-related events.

METHODS

The study was conducted among adult emergency physicians working in three EDs of a single academic health system in Rhode Island. Qualitative, semistructured interviews were conducted with ED physicians. Interviews assessed physicians' prior experience with predictive analytics, thoughts on the alert's placement, design, and content, the alert's overall impact, and potential areas for improvement. Responses were aggregated and common themes identified.

RESULTS

Twenty-three interviews were conducted (11 preimplementation and 12 postimplementation). Themes were identified regarding each physician familiarity with predictive analytics, alert rollout, alert appearance and content, and on alert sentiments. Most physicians viewed these alerts as a neutral or positive EHR addition, with responses ranging from neutral to positive. The alert placement was noted to be largely intuitive and nonintrusive. The design of the alert was generally viewed positively. The alert's content was believed to be accurate, although the decision to respond to the alert's call-to-action was physician dependent. Those who tended to ignore the alert did so for a few reasons, including already knowing the information the alert contains, the alert offering information that is not relevant to this particular patient, and the alert not containing enough information to be useful.

CONCLUSION

Ultimately, this alert appears to have a marginally positive effect on ED physician workflow. At its most beneficial, the alert reminded physicians to deeply consider the care provided to high-risk populations and to potentially adjust their care and referrals. At its least beneficial, the alert did not affect physician decision-making but was not intrusive to the point of negatively impacting workflow.

Path model on decision-making ability of clinical nurses

AIMS AND OBJECTIVES

To identify and examine the relationship between the factors influencing the decision-making ability of clinical nurses in hospitals, South Korea, and to establish a model, to verify the fit and the effect.

BACKGROUND

Clinical nurses are exposed to environments and situations where they make continuous decisions according to the need of direct treatment and nursing.

DESIGN

This study used a cross-sectional descriptive design, relation prediction modelling and adheres to the STROBE guidelines.

METHODS

The model construction was based on the information processing theory by Hansen and Thomas (Nursing Research, 17, 436, 1968). The model consists of 5 exogenous variables (expertise, critical thinking disposition, knowledge-sharing behaviour, nursing work environment, and decision-making stress) and 3 endogenous variables (analytic-systematic decision-making type, intuitive-interpretive decision-making type and decision-making ability). Participants were 274 clinical nurses, who were working at two hospitals in Seoul, South Korea. The data was analysed using SPSS WIN 18.0 and AMOS 20.0 program. Path analysis to verify the hypothetical model was used, and the fit was evaluated by χ² /df, GFI, AGFI, NFI, CFI and RMSEA. Data were collected from March to May 2017.

RESULTS

The fit index of the modified path model was χ² /df = 2.25, GFI = .972, AGFI = .929, NFI = .967, CFI = .981 and RMSEA = .068. The analytic-systematic decision-making type had the greatest direct effect on the clinical nurses' decision-making ability, which is the final outcome variable, followed by significant direct and indirect effects on critical thinking disposition.

CONCLUSION

This study suggests that the clinical nurses' decision-making ability in hospitals were leadingly influenced by analytic-systematic decision-making type and critical thinking disposition.

RELEVANCE TO CLINICAL PRACTICE

In the nursing practice, nurses need to pay attention the analytic-systematic decision-making type and critical thinking disposition for improving decision-making ability of clinical nurses in hospitals.

Explainable, trustworthy, and ethical machine learning for healthcare: A survey

With the advent of machine learning (ML) and deep learning (DL) empowered applications for critical applications like healthcare, the questions about liability, trust, and interpretability of their outputs are raising. The black-box nature of various DL models is a roadblock to clinical utilization. Therefore, to gain the trust of clinicians and patients, we need to provide explanations about the decisions of models. With the promise of enhancing the trust and transparency of black-box models, researchers are in the phase of maturing the field of eXplainable ML (XML). In this paper, we provided a comprehensive review of explainable and interpretable ML techniques for various healthcare applications. Along with highlighting security, safety, and robustness challenges that hinder the trustworthiness of ML, we also discussed the ethical issues arising because of the use of ML/DL for healthcare. We also describe how explainable and trustworthy ML can resolve all these ethical problems. Finally, we elaborate on the limitations of existing approaches and highlight various open research problems that require further development.

Dashboard Design to Identify and Balance Competing Risk of Multiple Hospital-Acquired Conditions

BACKGROUND

Hospital-acquired conditions (HACs) are common, costly, and national patient safety priority. Catheter-associated urinary tract infections (CAUTIs), hospital-acquired pressure injury (HAPI), and falls are common HACs. Clinicians assess each HAC risk independent of other conditions. Prevention strategies often focus on the reduction of a single HAC rather than considering how actions to prevent one condition could have unintended consequences for another HAC.

OBJECTIVES

The objective of this study is to design an empirical framework to identify, assess, and quantify the risks of multiple HACs (MHACs) related to competing single-HAC interventions.

METHODS

This study was an Institutional Review Board approved, and the proof of concept study evaluated MHAC Competing Risk Dashboard to enhance clinicians' management combining the risks of CAUTI, HAPI, and falls. The empirical model informing this study focused on the removal of an indwelling urinary catheter to reduce CAUTI, which may impact HAPI and falls. A multisite database was developed to understand and quantify competing risks of HACs; a predictive model dashboard was designed and clinical utility of a high-fidelity dashboard was qualitatively tested. Five hospital systems provided data for the predictive model prototype; three served as sites for testing and feedback on the dashboard design and usefulness. The participatory study design involved think-aloud methods as the clinician explored the dashboard. Individual interviews provided an understanding of clinician's perspective regarding ease of use and utility.

RESULTS

Twenty-five clinicians were interviewed. Clinicians favored a dashboard gauge design composed of green, yellow, and red segments to depict MHAC risk associated with the removal of an indwelling urinary catheter to reduce CAUTI and possible adverse effects on HAPI and falls.

CONCLUSION

Participants endorsed the utility of a visual dashboard guiding clinical decisions for MHAC risks preferring common stoplight color understanding. Clinicians did not want mandatory alerts for tool integration into the electronic health record. More research is needed to understand MHAC and tools to guide clinician decisions.

A Concept Analysis of Nurses' Clinical Decision Making: Implications for Korea

The study's purpose was to identify the meaning and the attributes of Korean nurses' clinical decision making. A sequential and systematic literature review with reflection according to the conceptual analysis method of Walker and Avant was used in this study. Data sources included the National Assembly Library, the National Digital Science Library, ProQuest, PubMed, MEDLINE, and CINAHL. Finally, twenty-six articles were included in this concept analysis. The concept of Korean nurses' clinical decision making consisted of the following attributes: clinical reasoning, choosing and applying challenging alternatives, and professional assessment and resetting. Antecedents consisted of: recognizing complex and diverse patient situations with high uncertainty, the need to solve problems according to priority, prior experience in clinical decision making, and interrelationships with fellow medical staff. Consequences consisted of: providing high-quality nursing services, improving the patient's safety, and increased satisfaction with clinical decision making. Based on these results, the conceptual attributes of Korean nurses' clinical decision making had slightly different characteristics but were organically interrelated. The results of analyzing the concept of Korean nurses' clinical decision making provide a better understanding of it and contribute to expanding nursing knowledge and developing a valid and reliable measurement.

Designing a Clinical Decision Support Tool That Leverages Machine Learning for Suicide Risk Prediction: Development Study in Partnership With Native American Care Providers

Background

Machine learning algorithms for suicide risk prediction have been developed with notable improvements in accuracy. Implementing these algorithms to enhance clinical care and reduce suicide has not been well studied.

Objective

This study aims to design a clinical decision support tool and appropriate care pathways for community-based suicide surveillance and case management systems operating on Native American reservations.

Methods

Participants included Native American case managers and supervisors (N=9) who worked on suicide surveillance and case management programs on 2 Native American reservations. We used in-depth interviews to understand how case managers think about and respond to suicide risk. The results from interviews informed a draft clinical decision support tool, which was then reviewed with supervisors and combined with appropriate care pathways.

Results

Case managers reported acceptance of risk flags based on a predictive algorithm in their surveillance system tools, particularly if the information was available in a timely manner and used in conjunction with their clinical judgment. Implementation of risk flags needed to be programmed on a dichotomous basis, so the algorithm could produce output indicating high versus low risk. To dichotomize the continuous predicted probabilities, we developed a cutoff point that favored specificity, with the understanding that case managers’ clinical judgment would help increase sensitivity.

Conclusions

Suicide risk prediction algorithms show promise, but implementation to guide clinical care remains relatively elusive. Our study demonstrates the utility of working with partners to develop and guide the operationalization of risk prediction algorithms to enhance clinical care in a community setting.

Disappearing expertise in clinical automation: Barcode medication administration and nurse autonomy

OBJECTIVE

Using the case of barcode medication administration (BCMA), our objective is to describe the challenges nurses face when informatics tools are not designed to accommodate the full complexity of their work.

MATERIALS AND METHODS

Autonomy is associated with nurse satisfaction and quality of care. BCMA organizes patient information and verifies medication administration. However, it presents challenges to nurse autonomy. Qualitative fieldwork, including observations of everyday work and interviews, was conducted during the implementation of BCMA in a large academic medical center. Fieldnotes and interview transcripts were coded and analyzed to describe nurses' perspectives on medication safety.

RESULTS

Nurses adopt orienting frames to structure work routines and require autonomy to ensure safe task completion. Nurses exerted agency by trusting their own judgment over system information when the system did not consider workload complexity. Our results indicate that the system's rigidity clashed with adaptive needs embodied by nurses' orienting frames.

DISCUSSION

Despite the fact that the concept of nurse as knowledge worker is foundational to informatics, nurses may be perceived as doers, rather than knowledge workers. In practice, nurses not only make decisions, but also engage in highly complex task-related work that is not well supported by process-oriented information technology tools.

CONCLUSIONS

Information technology developers and healthcare organization managers should engage and better understand nursing work in order to develop technological and social systems to support it.

A Systematic Review of Design Workshops for Health Information Technologies

Background: Design workshops offer effective methods in eliciting end-user participation from design inception to completion. Workshops unite stakeholders in the utilization of participatory methods, coalescing in the best possible creative solutions. Objective: This systematic review aimed to identify design approaches whilst providing guidance to health information technology designers/researchers in devising and organizing workshops. Methods: A systematic literature search was conducted in five medical/library databases identifying 568 articles. The initial duplication removal resulted in 562 articles. A criteria-based screening of the title field, abstracts, and pre-full-texts reviews resulted in 72 records for full-text review. The final review resulted in 10 article exclusions. Results: 62 publications were included in the review. These studies focused on consumer facing and clinical health information technologies. The studied technologies involved both clinician and patients and encompassed an array of health conditions. Diverse workshop activities and deliverables were reported. Only seven publications reported workshop evaluation data. Discussion: This systematic review focused on workshops as a design and research activity in the health informatics domain. Our review revealed three themes: (1) There are a variety of ways of conducting design workshops; (2) Workshops are effective design and research approaches; (3) Various levels of workshop details were reported.

A Perioperative Care Display for Understanding High Acuity Patients

BACKGROUND

The data visualization literature asserts that the details of the optimal data display must be tailored to the specific task, the background of the user, and the characteristics of the data. The general organizing principle of a concept-oriented display is known to be useful for many tasks and data types.

OBJECTIVES

In this project, we used general principles of data visualization and a co-design process to produce a clinical display tailored to a specific cognitive task, chosen from the anesthesia domain, but with clear generalizability to other clinical tasks. To support the work of the anesthesia-in-charge (AIC) our task was, for a given day, to depict the acuity level and complexity of each patient in the collection of those that will be operated on the following day. The AIC uses this information to optimally allocate anesthesia staff and providers across operating rooms.

METHODS

We used a co-design process to collaborate with participants who work in the AIC role. We conducted two in-depth interviews with AICs and engaged them in subsequent input on iterative design solutions.

RESULTS

Through a co-design process, we found (1) the need to carefully match the level of detail in the display to the level required by the clinical task, (2) the impedance caused by irrelevant information on the screen such as icons relevant only to other tasks, and (3) the desire for a specific but optional trajectory of increasingly detailed textual summaries.

CONCLUSION

This study reports a real-world clinical informatics development project that engaged users as co-designers. Our process led to the user-preferred design of a single binary flag to identify the subset of patients needing further investigation, and then a trajectory of increasingly detailed, text-based abstractions for each patient that can be displayed when more information is needed.

Prognostic models will be victims of their own success, unless…

Predictive analytics have begun to change the workflows of healthcare by giving insight into our future health. Deploying prognostic models into clinical workflows should change behavior and motivate interventions that affect outcomes. As users respond to model predictions, downstream characteristics of the data, including the distribution of the outcome, may change. The ever-changing nature of healthcare necessitates maintenance of prognostic models to ensure their longevity. The more effective a model and intervention(s) are at improving outcomes, the faster a model will appear to degrade. Improving outcomes can disrupt the association between the model's predictors and the outcome. Model refitting may not always be the most effective response to these challenges. These problems will need to be mitigated by systematically incorporating interventions into prognostic models and by maintaining robust performance surveillance of models in clinical use. Holistically modeling the outcome and intervention(s) can lead to resilience to future compromises in performance.

"There's a problem, now what's the solution?": suggestions for technologies to support the menopausal transition from individuals experiencing menopause and healthcare practitioners

OBJECTIVE

To elicit novel ideas for informatics solutions to support individuals through the menopausal transition. (Note: We use "individuals experiencing menopause" and "experiences" rather than "symptoms" when possible to counter typical framing of menopause as a cisgender women's medical problem.).

METHODS

A participatory design study was conducted 2015-2017 in the Western US. Two sessions were held with individuals experiencing menopause recruited from the general public; and 3 sessions with healthcare practitioners (HCPs) including nurses, physicians, and complementary and integrative health (CIH) practitioners were held. Participants designed technologies addressing informational needs and burdensome experiences. HCPs reflected on designs from participants experiencing menopause. Directed content analysis was used to analyze transcripts.

RESULTS

Eight individuals experiencing menopause (n = 4 each session) and 18 HCPs (n = 10 CIH, n = 3 nurses, n = 5 physicians) participated. All participants provided ideas for solution purpose, hardware, software, features and functions, and data types. Individuals experiencing menopause designed technologies to help understand and prevent burdensome menopause experiences. HCPs designed technologies for tracking and facilitating communication. Compared to nurses and physicians, CIH practitioners suggested designs reframing menopause as a positive experience and accounted for the complex lives of individuals experiencing menopause, including stigma; these ideas corresponded to comments made by participants experiencing menopause. Participants from both populations were concerned about data confidentiality and technology accessibility.

CONCLUSIONS

Participant generated design ideas included novel ideas and incorporated existing technologies. This study can inform the development of new technologies or repurposing of existing technologies to support individuals through the menopausal transition.

User stories as lightweight requirements for agile clinical decision support development

Objective

We sought to demonstrate applicability of user stories, progressively elaborated by testable acceptance criteria, as lightweight requirements for agile development of clinical decision support (CDS).

Materials and Methods

User stories employed the template: As a [type of user], I want [some goal] so that [some reason]. From the “so that” section, CDS benefit measures were derived. Detailed acceptance criteria were elaborated through ensuing conversations. We estimated user story size with “story points,” and depicted multiple user stories with a use case diagram or feature breakdown structure. Large user stories were split to fit into 2-week iterations.

Results

One example user story was: As a rheumatologist, I want to be advised if my patient with rheumatoid arthritis is not on a disease-modifying anti-rheumatic drug (DMARD), so that they receive optimal therapy and can experience symptom improvement. This yielded a process measure (DMARD use), and an outcome measure (Clinical Disease Activity Index). Following implementation, the DMARD nonuse rate decreased from 3.7% to 1.4%. Patients with a high Clinical Disease Activity Index improved from 13.7% to 7%. For a thromboembolism prevention CDS project, diagrams organized multiple user stories.

Discussion

User stories written in the clinician’s voice aid CDS governance and lead naturally to measures of CDS effectiveness. Estimation of relative story size helps plan CDS delivery dates. User stories prove to be practical even on larger projects.

Conclusions

User stories concisely communicate the who, what, and why of a CDS request, and serve as lightweight requirements for agile development to meet the demand for increasingly diverse CDS.

A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare

BACKGROUND

There is an increasing interest in clinical prediction tools that can achieve high prediction accuracy and provide explanations of the factors leading to increased risk of adverse outcomes. However, approaches to explaining complex machine learning (ML) models are rarely informed by end-user needs and user evaluations of model interpretability are lacking in the healthcare domain. We used extended revisions of previously-published theoretical frameworks to propose a framework for the design of user-centered displays of explanations. This new framework served as the basis for qualitative inquiries and design review sessions with critical care nurses and physicians that informed the design of a user-centered explanation display for an ML-based prediction tool.

METHODS

We used our framework to propose explanation displays for predictions from a pediatric intensive care unit (PICU) in-hospital mortality risk model. Proposed displays were based on a model-agnostic, instance-level explanation approach based on feature influence, as determined by Shapley values. Focus group sessions solicited critical care provider feedback on the proposed displays, which were then revised accordingly.

RESULTS

The proposed displays were perceived as useful tools in assessing model predictions. However, specific explanation goals and information needs varied by clinical role and level of predictive modeling knowledge. Providers preferred explanation displays that required less information processing effort and could support the information needs of a variety of users. Providing supporting information to assist in interpretation was seen as critical for fostering provider understanding and acceptance of the predictions and explanations. The user-centered explanation display for the PICU in-hospital mortality risk model incorporated elements from the initial displays along with enhancements suggested by providers.

CONCLUSIONS

We proposed a framework for the design of user-centered displays of explanations for ML models. We used the proposed framework to motivate the design of a user-centered display of an explanation for predictions from a PICU in-hospital mortality risk model. Positive feedback from focus group participants provides preliminary support for the use of model-agnostic, instance-level explanations of feature influence as an approach to understand ML model predictions in healthcare and advances the discussion on how to effectively communicate ML model information to healthcare providers.

Design and Evaluation of Low-Fidelity Visual Display Prototypes for Multiple Hospital-Acquired Conditions

Hospital-acquired conditions such as catheter-associated urinary tract infection, stage 3 or 4 hospital-acquired pressure injury, and falls with injury are common, costly, and largely preventable. This study used participatory design methods to design and evaluate low-fidelity prototypes of clinical dashboards to inform high-fidelity prototype designs to visualize integrated risks based on patient profiles. Five low-fidelity prototypes were developed through literature review and by engaging nurses, nurse managers, and providers as participants (N = 23) from two hospitals in different healthcare systems using focus groups and interviews. Five themes were identified from participatory design sessions: Need for Integrated Hospital-Acquired Condition Risk Tool, Information Needs, Sources of Information, Trustworthiness of Information, and Performance Tracking Perspectives. Participants preferred visual displays that represented patient comparative risks for hospital-acquired conditions using the familiar design metaphor of a gauge and green, yellow, and red "traffic light" colors scheme. Findings from this study were used to design a high-fidelity prototype to be tested in the next phase of the project. Visual displays of hospital-acquired conditions that are familiar in display and simplify complex information such as the green, yellow, and red dashboard are needed to assist clinicians in fast-paced clinical environments and be designed to prevent alert fatigue.

"How did you get to this number?" Stakeholder needs for implementing predictive analytics: a pre-implementation qualitative study

OBJECTIVE

Predictive analytics are potentially powerful tools, but to improve healthcare delivery, they must be carefully integrated into healthcare organizations. Our objective was to identify facilitators, challenges, and recommendations for implementing a novel predictive algorithm which aims to prospectively identify patients with high preventable utilization to proactively involve them in preventative interventions.

MATERIALS AND METHODS

In preparation for implementing the predictive algorithm in 3 organizations, we interviewed 3 stakeholder groups: health systems operations (eg, chief medical officers, department chairs), informatics personnel, and potential end users (eg, physicians, nurses, social workers). We applied thematic analysis to derive key themes and categorize them into the dimensions of Sittig and Singh's original sociotechnical model for studying health information technology in complex adaptive healthcare systems. Recruiting and analysis were conducted iteratively until thematic saturation was achieved.

RESULTS

Forty-nine interviews were conducted in 3 healthcare organizations. Technical components of the implementation (hardware and software) raised fewer concerns than alignment with sociotechnical factors. Stakeholders wanted decision support based on the algorithm to be clear and actionable and incorporated into current workflows. However, how to make this disease-independent classification tool actionable was perceived as a challenge, and appropriate patient interventions informed by the algorithm appeared likely to require substantial external and institutional resources. Stakeholders also described the criticality of trust, credibility, and interpretability of the predictive algorithm.

CONCLUSIONS

Although predictive analytics can classify patients with high accuracy, they cannot advance healthcare processes and outcomes without careful implementation that takes into account the sociotechnical system. Key stakeholders have strong perceptions about facilitators and challenges to shape successful implementation.

Reinterpreting the nursing record for an electronic context

BACKGROUND

This article seeks to facilitate the re-imagining of nursing records purposefully within an electronic context. It questions existing approaches to nursing documentation, critically examines existing nursing record systems and identifies new requirements.

METHODS

A comprehensive literature review was conducted to identify themes, that might meaningfully contribute to a new approach to nursing record systems development, around four key interrelated areas - standards, decision making, abstraction and summarization, and documenting. Studies were analyzed using narrative synthesis to provide a critical analysis of the current 'state of the art', and recommendations for the future.

RESULTS

Included studies collectively described aspects of current best practice, both in terms of nursing record systems themselves, and how nurses and other health professionals contribute to and engage with those systems. A number of cross-cutting themes identified more novel approaches taken by nurses to systems development: going back to basics in determining purpose; firming up informatics foundations; nuancing or tailoring to suit different requirements; and engagement, involvement and participation.

CONCLUSION

There is a paucity of research that specifically focuses on the nature of the electronic nursing record and its impact on patient care processes and outcomes. In addition to further research in these areas, there is a need: to reinterpret nurses as knowledge workers rather than as 'data collectors'; to agree on the application in practice of appropriate standards and terminologies; and to work together with system developers to change the ways in which data are captured and care is documented.

Nurse Decision Making in Acute Care

Effective nurse decision making is essential for best patient outcomes in the acute care nurse practice environment. The purpose of this study was to explore acute care RNs' perceptions of clinical decision making for a patient who experienced a clinical event. Clinical events include changes in patient condition and are manifested by fever, pain, bleeding, changes in output, changes in respiratory status, and changes in level of consciousness. Naturalistic decision making framework supported the exploration of important contextual factors associated with decision making, provided new information for nursing science, and served as the conceptual framework for this research. Data collected from interviews of 20 acute care nurses were analyzed using qualitative content analysis. The emergent categories included Awareness of Patient Status, Experience and Decision Making, Following Established Routine, Time Pressure, Teamwork/Support From Staff, Goals, Education, Resources, Patient Education, Consideration of Options to Meet Goals, and Nursing Roles. Acute care nurses incorporated a wide variety of complex factors when decision making. This study sought to improve understanding of the factors nurses found important to their decision making for the potential development of improved decision support in the electronic health record.

Usability Evaluation of a Dashboard for Home Care Nurses

The introduction of electronic health records has produced many challenges for clinicians. These include integrating technology into clinical workflow and fragmentation of relevant information across systems. Dashboards, which use visualized data to summarize key patient information, have the potential to address these issues. In this article, we outline a usability evaluation of a dashboard designed for home care nurses. An iterative design process was used, which consisted of (1) contextual inquiry (observation and interviews) with two home care nurses; (2) rapid feedback on paper prototypes of the dashboard (10 nurses); and (3) usability evaluation of the final dashboard prototype (20 nurses). Usability methods and assessments included observation of nurses interacting with the dashboard, the system usability scale, and the Questionnaire for User Interaction Satisfaction short form. The dashboard prototype was deemed to have high usability (mean system usability scale, 73.2 [SD, 18.8]) and was positively evaluated by nurse users. It is important to ensure that technology solutions such as the one proposed in this article are designed with clinical users in mind, to meet their information needs. The design elements of the dashboard outlined in this article could be translated to other electronic health records used in home care settings.

“Build Rapport, Otherwise No Screening Tools in the World Are Going to Help”: Frontline Service Providers’ Views on Current Screening Tools for Elder Abuse

Background and Objectives

Without an effective screening tool, accompanied by clear guidelines of what to do when elder abuse is suspected, health workers may face challenges when asking questions about elder abuse. This study aimed to find the most effective and acceptable existing elder abuse screening tool and to create guidelines for using the tool.

Research Design and Methods

A rapid review of the literature identified existing validated elder abuse screening tools. Then, 5 tools (Vulnerability to Abuse Screening Scale [VASS], Elder Abuse Suspicion Index [EASI], Elder Assessment Instrument [EAI], Caregiver Abuse Screen [CASE], and Brief Abuse Screen for the Elderly [BASE]), selected based on their internal rigor, were presented to health professionals to assess the tools’ relevance to their practice. Three focus groups were held with 23 health professionals in Victoria, Australia, in 2017. Data were thematically analyzed.

Results

None of the tools were deemed suitable by participants for use in their practice. Criticisms of the tools included: using outdated terminology, asking binary questions, asking multiple questions at once, failure to consider the older person’s cognitive status, failure to consider how culture mediates elder abuse, and failure to outline a referral pathway to those administering the tool. Participants emphasized that the screening tool must promote trust and rapport between the assessor and the older person to solicit a story on this sensitive subject.

Discussion and Implications

A successful elder abuse screening tool must be concise, easy to use, account for the older person’s health and social vulnerabilities, and outline a referral pathway if elder abuse is suspected.

Informatics Solutions for Application of Decision-Making Skills

Critical care nurses practice in a challenging environment that requires responses to patients with complex, often unstable health conditions. The electronic health record, access to clinical data, and Clinical Decision Support Systems informed by data from clinical databases are informatics tools designed to work together to facilitate decision-making in nursing practice. The complex decision-making environment of critical care requires informatics tools that support nursing practice through integration of current evidence with clinical data. Recommendations include continuing efforts toward the development of clinical decision support tools based on patient data that include predictive models to support increased patient safety.