Expanding usability analysis with intrinsic motivation concepts to learn about CDSS adoption: a case study

Usability of a touchscreen assessment tool (TATOO) prototype for clinicians and typically developing children

PURPOSE

Touchscreen devices are widely used in modern life and have quickly become part of daily life for children, including during Occupational Therapy sessions for children with disabilities. Touchscreen Assessment Tool (TATOO) is a prototype application used to evaluate children's performance when using touchscreen devices. The purpose of this study, based on the logical user-centred interaction design framework, was to evaluate TATOO's usability for occupational therapists and typically developing children and to examine the correlations between their usability scores.

METHODS

A convenience sample of clinicians (N = 10) and children with typical development (N = 60) was recruited for this study. The usability assessment was conducted using the System Usability Scale (SUS) and semi-structured interviews for the clinicians, and the Short Feedback Questionnaire-Child (SFQ-Child) for the children.

RESULTS

The SUS scores (M ± SD = 85.5 ± 8.04, range = 70-97.5) indicated good ratings of TATOO's usability by clinicians; the SFQ-Child results showed children also rated its usability very highly, including all ages (4-10 years) and all tasks. The clinicians all expressed positive attitudes towards using TATOO, and no bias was found between the clinicians' usability scores and the children's usability feedback.

CONCLUSION

The TATOO is a user-friendly tool. Researchers and clinicians can benefit from the availability of an objective and low-cost assessment tool to promote their evaluation and intervention by providing more focussed individualized recommendations and adaptations. The study also suggests a model to follow when developing applications and evaluating their usability through a mixed-method approach to deepen understanding of the user's needs.Implications for rehabilitationTouchscreen Assessment Tool (TATOO) shows a user-friendly tool for assessing the different skills required to operate touchscreens interface.TATOO has the potential to become an essential objective and low-cost assessment tool for the clinician, in which the spread of touchscreens constantly increases.Researchers and clinicians can benefit from the availability of such tools to promote their evaluation and intervention by providing more focussed individualized recommendations and adaptations.TATOO will complement the assessment needs, as traditional fine motor assessment tools cannot capture the skills necessary to operate a touchscreen deviceThe study suggests a model to follow when developing applications and evaluating their usability through a mixed-method approach in order to deepen understanding of the user's needs.

Exploring the role of professional identity in the implementation of clinical decision support systems-a narrative review

BACKGROUND

Clinical decision support systems (CDSSs) have the potential to improve quality of care, patient safety, and efficiency because of their ability to perform medical tasks in a more data-driven, evidence-based, and semi-autonomous way. However, CDSSs may also affect the professional identity of health professionals. Some professionals might experience these systems as a threat to their professional identity, as CDSSs could partially substitute clinical competencies, autonomy, or control over the care process. Other professionals may experience an empowerment of the role in the medical system. The purpose of this study is to uncover the role of professional identity in CDSS implementation and to identify core human, technological, and organizational factors that may determine the effect of CDSSs on professional identity.

METHODS

We conducted a systematic literature review and included peer-reviewed empirical studies from two electronic databases (PubMed, Web of Science) that reported on key factors to CDSS implementation and were published between 2010 and 2023. Our explorative, inductive thematic analysis assessed the antecedents of professional identity-related mechanisms from the perspective of different health care professionals (i.e., physicians, residents, nurse practitioners, pharmacists).

RESULTS

One hundred thirty-one qualitative, quantitative, or mixed-method studies from over 60 journals were included in this review. The thematic analysis found three dimensions of professional identity-related mechanisms that influence CDSS implementation success: perceived threat or enhancement of professional control and autonomy, perceived threat or enhancement of professional skills and expertise, and perceived loss or gain of control over patient relationships. At the technological level, the most common issues were the system's ability to fit into existing clinical workflows and organizational structures, and its ability to meet user needs. At the organizational level, time pressure and tension, as well as internal communication and involvement of end users were most frequently reported. At the human level, individual attitudes and emotional responses, as well as familiarity with the system, most often influenced the CDSS implementation. Our results show that professional identity-related mechanisms are driven by these factors and influence CDSS implementation success. The perception of the change of professional identity is influenced by the user's professional status and expertise and is improved over the course of implementation.

CONCLUSION

This review highlights the need for health care managers to evaluate perceived professional identity threats to health care professionals across all implementation phases when introducing a CDSS and to consider their varying manifestations among different health care professionals. Moreover, it highlights the importance of innovation and change management approaches, such as involving health professionals in the design and implementation process to mitigate threat perceptions. We provide future areas of research for the evaluation of the professional identity construct within health care.

Methods used to evaluate usability of mobile clinical decision support systems for healthcare emergencies: a systematic review and qualitative synthesis

Objective

The aim of this study was to determine the methods and metrics used to evaluate the usability of mobile application Clinical Decision Support Systems (CDSSs) used in healthcare emergencies. Secondary aims were to describe the characteristics and usability of evaluated CDSSs.

Materials and Methods

A systematic literature review was conducted using Pubmed/Medline, Embase, Scopus, and IEEE Xplore databases. Quantitative data were descriptively analyzed, and qualitative data were described and synthesized using inductive thematic analysis.

Results

Twenty-three studies were included in the analysis. The usability metrics most frequently evaluated were efficiency and usefulness, followed by user errors, satisfaction, learnability, effectiveness, and memorability. Methods used to assess usability included questionnaires in 20 (87%) studies, user trials in 17 (74%), interviews in 6 (26%), and heuristic evaluations in 3 (13%). Most CDSS inputs consisted of manual input (18, 78%) rather than automatic input (2, 9%). Most CDSS outputs comprised a recommendation (18, 78%), with a minority advising a specific treatment (6, 26%), or a score, risk level or likelihood of diagnosis (6, 26%). Interviews and heuristic evaluations identified more usability-related barriers and facilitators to adoption than did questionnaires and user testing studies.

Discussion

A wide range of metrics and methods are used to evaluate the usability of mobile CDSS in medical emergencies. Input of information into CDSS was predominantly manual, impeding usability. Studies employing both qualitative and quantitative methods to evaluate usability yielded more thorough results.

Conclusion

When planning CDSS projects, developers should consider multiple methods to comprehensively evaluate usability.

Point-of-care Cognitive Support Technology in Emergency Departments: A Scoping Review of Technology Acceptance by Clinicians

OBJECTIVE

Cognitive support technologies that support clinical decisions and practices in the emergency department (ED) have the potential to optimize patient care. However, limited uptake by clinicians can prevent successful implementation. A better understanding of acceptance of these technologies from the clinician perspective is needed. We conducted a scoping review to synthesize diverse, emerging evidence on clinicians' acceptance of point-of-care (POC) cognitive support technology in the ED.

METHOD

We systematically searched 10 electronic databases and gray literature published from January 2006 to December 2016. Studies of any design assessing an ED-based POC cognitive support technology were considered eligible for inclusion. Studies were required to report outcome data for technology acceptance. Two reviewers independently screened studies for relevance and quality. Study quality was assessed using the Mixed-Methods Appraisal Tool. A descriptive analysis of the features of POC cognitive support technology for each study is presented, illustrating trends in technology development and evaluation. A thematic analysis of clinician, technical, patient, and organizational factors associated with technology acceptance is also presented.

RESULTS

Of the 1,563 references screened for eligibility, 24 met the inclusion criteria and were included in the review. Most studies were published from 2011 onward (88%), scored high for methodologic quality (79%), and examined POC technologies that were novel and newly introduced into the study setting (63%). Physician use of POC technology was the most commonly studied (67%). Technology acceptance was frequently conceptualized and measured by factors related to clinician attitudes and beliefs. Experience with the technology, intention to use, and actual use were also more common outcome measures of technology acceptance. Across studies, perceived usefulness was the most noteworthy factor impacting technology acceptance, and clinicians generally had positive perceptions of the use of POC cognitive support technology in the ED. However, the actual use of POC cognitive support technology reported by clinicians was low-use, by proportion of patient cases, ranged from 30% to 59%. Of the 24 studies, only two studies investigated acceptance of POC cognitive support technology currently implemented in the ED, offering "real-world" clinical practice data. All other studies focused on acceptance of novel technologies. Technical aspects such as an unfriendly user interface, presentation of redundant or ambiguous information, and required user effort had a negative impact on acceptance. Patient expectations were also found to have a negative impact, while patient safety implications had a positive impact. Institutional support was also reported to impact technology acceptance.

CONCLUSIONS

Findings from this scoping review suggest that while ED clinicians acknowledge the utility and value of using POC cognitive support technology, actual use of such technology can be low. Further, few studies have evaluated the acceptance and use of POC technologies in routine care. Prospective studies that evaluate how ED clinicians appraise and consider POC technology use in clinical practice are now needed with diverse clinician samples. While this review identified multiple factors contributing to technology acceptance, determining how clinician, technical, patient, and organizational factors mediate or moderate acceptance should also be a priority.