Cognitive Fit: A Theory-Based Analysis of the Graphs Versus Tables Literature

Lessons Learned From Developing Dashboards to Support Decision-Making for Community Opioid Response by Community Stakeholders: Mixed Methods and Multisite Study

BACKGROUND

Data dashboards are published tools that present visualizations; they are increasingly used to display data about behavioral health, social determinants of health, and chronic and infectious disease risks to inform or support public health endeavors. Dashboards can be an evidence-based approach used by communities to influence decision-making in health care for specific populations. Despite widespread use, evidence on how to best design and use dashboards in the public health realm is limited. There is also a notable dearth of studies that examine and document the complexity and heterogeneity of dashboards in community settings.

OBJECTIVE

Community stakeholders engaged in the community response to the opioid overdose crisis could benefit from the use of data dashboards for decision-making. As part of the Communities That HEAL (CTH) intervention, community data dashboards were created for stakeholders to support decision-making. We assessed stakeholders' perceptions of the usability and use of the CTH dashboards for decision-making.

METHODS

We conducted a mixed methods assessment between June and July 2021 on the use of CTH dashboards. We administered the System Usability Scale (SUS) and conducted semistructured group interviews with users in 33 communities across 4 states of the United States. The SUS comprises 10 five-point Likert-scale questions measuring usability, each scored from 0 to 4. The interview guides were informed by the technology adoption model (TAM) and focused on perceived usefulness, perceived ease of use, intention to use, and contextual factors.

RESULTS

Overall, 62 users of the CTH dashboards completed the SUS and interviews. SUS scores (grand mean 73, SD 4.6) indicated that CTH dashboards were within the acceptable range for usability. From the qualitative interview data, we inductively created subthemes within the 4 dimensions of the TAM to contextualize stakeholders' perceptions of the dashboard's usefulness and ease of use, their intention to use, and contextual factors. These data also highlighted gaps in knowledge, design, and use, which could help focus efforts to improve the use and comprehension of dashboards by stakeholders.

CONCLUSIONS

We present a set of prioritized gaps identified by our national group and list a set of lessons learned for improved data dashboard design and use for community stakeholders. Findings from our novel application of both the SUS and TAM provide insights and highlight important gaps and lessons learned to inform the design of data dashboards for use by decision-making community stakeholders.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04111939; https://clinicaltrials.gov/study/NCT04111939.

Numerical versus graphical aids for decision-making in a multi-cue signal identification task

Decision aids are commonly used in tactical decision-making environments to help humans integrate base-rate and multi-cue information. However, it is important that users appropriately trust and rely on aids. Decision aids can be presented in many ways, but the literature lacks clarity over the conditions surrounding their effectiveness. This research aims to determine whether a numerical or graphical aid more effectively supports human performance, and explores the relationships between aid presentation, trust, and workload. Participants (N = 30) completed a signal-identification task that required integration of readings from a set of three dynamic gauges. Participants experienced three conditions: unaided, using a numerical aid, and using a graphical aid. The aids combined gauge and base-rate information in a statistically-optimal fashion. Participants also indicated how much they trusted the system and how hard they worked during the task. Analyses explored the impact of aid condition on sensitivity, response bias, response time, trust, and workload. Both the numerical and graphical aids produced significant increases in sensitivity and trust, and significant decreases in workload in comparison to the unaided condition. The difference in response time between the graphical and unaided conditions approached significance, with participants responding faster using the graphical aid without decrements in sensitivity. Significant interactions between aid and signal type indicated that both aided conditions promoted faster responding to non-hostile signals, with larger mean differences in the graphical aid condition. Practically, graphical aids in which suggestions are more salient to users may promote faster responding in tactical environments, with negligible cost of accuracy.

The role of information systems in emergency department decision-making-a literature review

OBJECTIVES

Healthcare providers employ heuristic and analytical decision-making to navigate the high-stakes environment of the emergency department (ED). Despite the increasing integration of information systems (ISs), research on their efficacy is conflicting. Drawing on related fields, we investigate how timing and mode of delivery influence IS effectiveness. Our objective is to reconcile previous contradictory findings, shedding light on optimal IS design in the ED.

MATERIALS AND METHODS

We conducted a systematic review following PRISMA across PubMed, Scopus, and Web of Science. We coded the ISs' timing as heuristic or analytical, their mode of delivery as active for automatic alerts and passive when requiring user-initiated information retrieval, and their effect on process, economic, and clinical outcomes.

RESULTS

Our analysis included 83 studies. During early heuristic decision-making, most active interventions were ineffective, while passive interventions generally improved outcomes. In the analytical phase, the effects were reversed. Passive interventions that facilitate information extraction consistently improved outcomes.

DISCUSSION

Our findings suggest that the effectiveness of active interventions negatively correlates with the amount of information received during delivery. During early heuristic decision-making, when information overload is high, physicians are unresponsive to alerts and proactively consult passive resources. In the later analytical phases, physicians show increased receptivity to alerts due to decreased diagnostic uncertainty and information quantity. Interventions that limit information lead to positive outcomes, supporting our interpretation.

CONCLUSION

We synthesize our findings into an integrated model that reveals the underlying reasons for conflicting findings from previous reviews and can guide practitioners in designing ISs in the ED.

User Intentions to Use ChatGPT for Self-Diagnosis and Health-Related Purposes: Cross-sectional Survey Study

BACKGROUND

With the rapid advancement of artificial intelligence (AI) technologies, AI-powered chatbots, such as Chat Generative Pretrained Transformer (ChatGPT), have emerged as potential tools for various applications, including health care. However, ChatGPT is not specifically designed for health care purposes, and its use for self-diagnosis raises concerns regarding its adoption's potential risks and benefits. Users are increasingly inclined to use ChatGPT for self-diagnosis, necessitating a deeper understanding of the factors driving this trend.

OBJECTIVE

This study aims to investigate the factors influencing users' perception of decision-making processes and intentions to use ChatGPT for self-diagnosis and to explore the implications of these findings for the safe and effective integration of AI chatbots in health care.

METHODS

A cross-sectional survey design was used, and data were collected from 607 participants. The relationships between performance expectancy, risk-reward appraisal, decision-making, and intention to use ChatGPT for self-diagnosis were analyzed using partial least squares structural equation modeling (PLS-SEM).

RESULTS

Most respondents were willing to use ChatGPT for self-diagnosis (n=476, 78.4%). The model demonstrated satisfactory explanatory power, accounting for 52.4% of the variance in decision-making and 38.1% in the intent to use ChatGPT for self-diagnosis. The results supported all 3 hypotheses: The higher performance expectancy of ChatGPT (β=.547, 95% CI 0.474-0.620) and positive risk-reward appraisals (β=.245, 95% CI 0.161-0.325) were positively associated with the improved perception of decision-making outcomes among users, and enhanced perception of decision-making processes involving ChatGPT positively impacted users' intentions to use the technology for self-diagnosis (β=.565, 95% CI 0.498-0.628).

CONCLUSIONS

Our research investigated factors influencing users' intentions to use ChatGPT for self-diagnosis and health-related purposes. Even though the technology is not specifically designed for health care, people are inclined to use ChatGPT in health care contexts. Instead of solely focusing on discouraging its use for health care purposes, we advocate for improving the technology and adapting it for suitable health care applications. Our study highlights the importance of collaboration among AI developers, health care providers, and policy makers in ensuring AI chatbots' safe and responsible use in health care. By understanding users' expectations and decision-making processes, we can develop AI chatbots, such as ChatGPT, that are tailored to human needs, providing reliable and verified health information sources. This approach not only enhances health care accessibility but also improves health literacy and awareness. As the field of AI chatbots in health care continues to evolve, future research should explore the long-term effects of using AI chatbots for self-diagnosis and investigate their potential integration with other digital health interventions to optimize patient care and outcomes. In doing so, we can ensure that AI chatbots, including ChatGPT, are designed and implemented to safeguard users' well-being and support positive health outcomes in health care settings.

Effect of Viewpoint Change on Robot Hand Operation by Gesture- and Button-Based Methods

Teaching pendants with multiple buttons are commonly employed to control working robots; however, such devices are not easy to operate. As an alternative, gesture-based manipulation methods using the operator’s upper limb movements have been studied as a way to operate the robots intuitively. Previous studies involving these methods have generally failed to consider the changing viewpoint of the operator relative to the robot, which may adversely affect operability. This study proposes a novel evaluation method and applies it in a series of experiments to compare the influence of viewpoint change on the operability of the gesture- and button-based operation methods. Experimental results indicate that the operability of the gesture-based method is superior to that of the button-based method for all viewpoint angles, due mainly to shorter non-operating times. An investigation of trial-and-error operation indicates that viewpoint change makes positional operation with the button-based method more difficult but has a relatively minor influence on postural operation.

The Impact of Behavioral Similarity on DSS Advice Acceptance

Despite the many advantages that decision support systems (DSSs) provide to decision makers, acceptance of advice from these systems has remained relatively low. In this study, we leverage theoretical insights from the literature to build a DSS that is behaviorally similar to its user, with the aim of increasing acceptance of DSS advice. To test whether users would be more accepting of advice from a behaviorally-similar DSS, we conducted a field experiment in a complex semi-structured decision context. The results of our experiment provide empirical evidence that a behaviorally-similar DSS can increase acceptance of advice in semi-structured decision-making contexts.

Designing for Confidence: The Impact of Visualizing Artificial Intelligence Decisions

Explainable artificial intelligence aims to bring transparency to artificial intelligence (AI) systems by translating, simplifying, and visualizing its decisions. While society remains skeptical about AI systems, studies show that transparent and explainable AI systems can help improve the Human-AI trust relationship. This manuscript presents two studies that assess three AI decision visualization attribution models that manipulate morphological clarity (MC) and two information presentation-order methods to determine each visualization’s impact on the Human-AI trust relationship through increased confidence and cognitive fit (CF). The first study, N = 206 (Avg. age = 37.87 ± 10.51, Male = 123), utilized information presentation methods and visualizations delivered through an online experiment to explore trust in AI by asking participants to complete a visual decision-making task. The second study, N = 19 (24.9 ± 8.3 years old, Male = 10), utilized eye-tracking technology and the same stimuli presentation methods to investigate if cognitive load, inferred through pupillometry measures, mediated the confidence-trust relationship. The results indicate that low MC positively impacts Human-AI trust and that the presentation order of information within an interface in terms of adjacency further influences user trust in AI. We conclude that while adjacency and MC significantly affect cognitive load, cognitive load alone does not mediate the confidence-trust relationship. Our findings interpreted through a combination of CF, situation awareness, and ecological interface design have implications for the design of future AI systems, which may facilitate better collaboration between humans and AI-based decision agents.

Start-Ups as Adaptable Stable Systems Based on Synchronous Business Models

Business models have been a popular topic in research and practice for more than twenty years. During this time, frameworks for formulating business models have been developed, such as the business model canvas. Moreover, different business model frameworks have been proposed for different sectors. Yet, these frameworks have the fundamental shortcoming of not addressing directly and persistently the primary objective of start-ups: to survive in changing environments. The aim of the action research reported in this paper is to overcome that fundamental shortcoming. This is an important topic because the majority of start-ups do not survive. In this paper, first principles for survival in changing environments are related to business models. In particular, action research to reframe start-ups as adaptable stable systems based on synchronous business models is reported. The paper provides three principal contributions. The contribution to business model theory building is to relate survival first principles revealed through natural science research to business models. Reference to first principles highlight that survival depends on maintaining both external adaptability and internal stability through synchronization with changing environments. The second contribution is to business model practice through describing a simple business modeling method that is based on the scientific first principles. The third contribution is to provide an example that bridges the rigor–relevance gap between scientific research and business practice.

Determinants of the Digitalization of Accounting in an Emerging Market: The Roles of Organizational Support and Job Relevance

Digitalization has considerable potential to help achieve the sustainability of the planetary and human systems, including organizations. As digitalization is one of the most promising factors for transformation, there is no doubt that ICT and big data can help promote sustainability. Linking digitalization with a sustainable workplace, the purpose of this investigation is to identify the determinants of the adoption of digitalization of accounting procedures by accounting professionals in Saudi Arabia, an economy rapidly moving towards digital transformation. A holistic model based on the technology acceptance model (TAM), elaboration likelihood model (ELM), and social exchange theory (SET) is proposed and tested. The extension in TAM is proposed by integrating job relevance and organizational support as moderators. A survey method was used to collect data from 365 accounting professionals working in Saudi Arabia. Structural equation modeling and PROCESS macro were used as data analysis techniques. The findings reveal that perceived ease of use has both direct and indirect effects through perceived usefulness on intentions to use e-accounting. Both job relevance and organizational support act as moderators for perceived usefulness and intentions to use e-accounting when treated separately. However, job relevance becomes an insignificant moderator in the presence of organizational support. This finding highlights the importance of organizational support for the successful implementation of e-accounting in an organization.

A Scalable Service to Improve Health Care Quality Through Precision Audit and Feedback: Proposal for a Randomized Controlled Trial

BACKGROUND

Health care delivery organizations lack evidence-based strategies for using quality measurement data to improve performance. Audit and feedback (A&F), the delivery of clinical performance summaries to providers, demonstrates the potential for large effects on clinical practice but is currently implemented as a blunt one size fits most intervention. Each provider in a care setting typically receives a performance summary of identical metrics in a common format despite the growing recognition that precisionizing interventions hold significant promise in improving their impact. A precision approach to A&F prioritizes the display of information in a single metric that, for each recipient, carries the highest value for performance improvement, such as when the metric's level drops below a peer benchmark or minimum standard for the first time, thereby revealing an actionable performance gap. Furthermore, precision A&F uses an optimal message format (including framing and visual displays) based on what is known about the recipient and the intended gist meaning being communicated to improve message interpretation while reducing the cognitive processing burden. Well-established psychological principles, frameworks, and theories form a feedback intervention knowledge base to achieve precision A&F. From an informatics perspective, precision A&F requires a knowledge-based system that enables mass customization by representing knowledge configurable at the group and individual levels.

OBJECTIVE

This study aims to implement and evaluate a demonstration system for precision A&F in anesthesia care and to assess the effect of precision feedback emails on care quality and outcomes in a national quality improvement consortium.

METHODS

We propose to achieve our aims by conducting 3 studies: a requirements analysis and preferences elicitation study using human-centered design and conjoint analysis methods, a software service development and implementation study, and a cluster randomized controlled trial of a precision A&F service with a concurrent process evaluation. This study will be conducted with the Multicenter Perioperative Outcomes Group, a national anesthesia quality improvement consortium with >60 member hospitals in >20 US states. This study will extend the Multicenter Perioperative Outcomes Group quality improvement infrastructure by using existing data and performance measurement processes.

RESULTS

The proposal was funded in September 2021 with a 4-year timeline. Data collection for Aim 1 began in March 2022. We plan for a 24-month trial timeline, with the intervention period of the trial beginning in March 2024.

CONCLUSIONS

The proposed aims will collectively demonstrate a precision feedback service developed using an open-source technical infrastructure for computable knowledge management. By implementing and evaluating a demonstration system for precision feedback, we create the potential to observe the conditions under which feedback interventions are effective.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/34990.

Applying Insights on Categorisation, Communication, and Dynamic Decision-Making: A Case Study of a ‘Simple’ Maritime Military Decision

A complete understanding of decision-making in military domains requires gathering insights from several fields of study. To make the task tractable, here we consider a specific example of short-term tactical decisions under uncertainty made by the military at sea. Through this lens, we sketch out relevant literature from three psychological tasks each underpinned by decision-making processes: categorisation, communication and choice. From the literature, we note two general cognitive tendencies that emerge across all three stages: the effect of cognitive load and individual differences. Drawing on these tendencies, we recommend strategies, tools and future research that could improve performance in military domains – but, by extension, would also generalise to other high-stakes contexts. In so doing, we show the extent to which domain general properties of high order cognition are sufficient in explaining behaviours in domain specific contexts.

Scale development for analyzing the fit of real and virtual world integration: an example of Pokémon Go

Purpose

Augmented reality (AR) has become a trend, and the effects of Pokémon Go, the most popular online and mobile game, have been explored in many studies. However, few studies have developed questionnaires of fit to investigate the relationship between the fit and the integration of the game's virtual world and reality. The paper intends to integrate the models of stimulus-organism-response (S-O-R) and information systems success with cognitive fit theory to explore the fit and reactions of users in the integration of real and virtual worlds.

Design/methodology/approach

Following MacKenzie's scale development, two surveys were conducted. The first survey was conducted to perform a scale development of fit. The second survey was collected from 315 Pokémon Go players to validate the fit scale and it was analyzed via structural equation modeling.

Findings

The results show that scale development of fit has good reliability and validity. Furthermore, game information quality, game system quality and virtual (Pokémon) characteristics have significantly positive effects on cognitive and emotional fit. Cognitive and emotional fit have significant positive effects on user satisfaction, and user satisfaction has significant positive effects on continued intention to play. The results suggest that maintaining the quality of the game and improving the virtual interface will provide a better fit between the real and virtual worlds, enhancing user satisfaction with the fit as well as their intention for continued use.

Originality/value

Although fit has been widely studied in various contexts, the application of AR has been rarely discussed. This study develops a scale of fit and takes Pokémon Go as the subject to validate the fit measurement and discuss players' cognition and feelings regarding the game. The authors measure user reactions to different stimuli and explore cognitive and emotional fit as well as the integration of virtual worlds and reality. In sum, to the best of the authors’ knowledge, this is one of the earliest studies to explore and develop a cognitive and emotional fit scale for future researchers and practitioners.

Conceptual Metaphor and Graphical Convention Influence the Interpretation of Line Graphs

Many metaphors in language reflect conceptual metaphors that structure thought. In line with metaphorical expressions such as 'high number', experiments show that people associate larger numbers with upward space. Consistent with this metaphor, high numbers are conventionally depicted in high positions on the y-axis of line graphs. People also associate good and bad (emotional valence) with upward and downward locations, in line with metaphorical expressions such as 'uplifting' and 'down in the dumps'. Graphs depicting good quantities (e.g., vacation days) are consistent with graphical convention and the valence metaphor, because 'more' of the good quantity is represented by higher y-axis positions. In contrast, graphs depicting bad quantities (e.g., murders) are consistent with graphical convention, but not the valence metaphor, because more of the bad quantity is represented by higher (rather than lower) y-axis positions. We conducted two experiments (N = 300 per experiment) where participants answered questions about line graphs depicting good and bad quantities. For some graphs, we inverted the conventional axis ordering of numbers. Line graphs that aligned (versus misaligned) with valence metaphors (up = good) were easier to interpret, but this beneficial effect did not outweigh the adverse effect of inverting the axis numbering. Line graphs depicting good (versus bad) quantities were easier to interpret, as were graphs that depicted quantity using the x-axis (versus y-axis). Our results suggest that conceptual metaphors matter for the interpretation of line graphs. However, designers of line graphs are warned against subverting graphical convention to align with conceptual metaphors.

An Automated Dashboard to Improve Laboratory COVID-19 Diagnostics Management

Background: In response to the COVID-19 pandemic, our microbial diagnostic laboratory located in a university hospital has implemented several distinct SARS-CoV-2 RT-PCR systems in a very short time. More than 148,000 tests have been performed over 12 months, which represents about 405 tests per day, with peaks to more than 1,500 tests per days during the second wave. This was only possible thanks to automation and digitalization, to allow high throughput, acceptable time to results and to maintain test reliability. An automated dashboard was developed to give access to Key Performance Indicators (KPIs) to improve laboratory operational management.

Methods: RT-PCR data extraction of four respiratory viruses—SARS-CoV-2, influenza A and B and RSV—from our laboratory information system (LIS), was automated. This included age, gender, test result, RT-PCR instrument, sample type, reception time, requester, and hospitalization status etc. Important KPIs were identified and the visualization was achieved using an in-house dashboard based on the open-source language R (Shiny).

Results: The dashboard is organized into three main parts. The “Filter” page presents all the KPIs, divided into five sections: (i) general and gender-related indicators, (ii) number of tests and positivity rate, (iii) cycle threshold and viral load, (iv) test durations, and (v) not valid results. Filtering allows to select a given period, a dedicated instrument, a given specimen, an age range or a requester. The “Comparison” page allows a custom charting of all the available variables, which represents more than 182 combination. The “Data” page, gives the user an access to the raw data in tables format, with possibility of filtering, allowing for a deeper analysis and data download. Informations are updated every 4 h.

Conclusions: By giving a rapid access to a huge number of up-to-date information, represented using the most relevant visualization types, without the burden of timely data extraction and analysis, the dashboard represents a reliable and user-friendly tool for operational laboratory management. The dashboard represents a reliable and user-friendly tool improving the decision-making process, resource planning and quality management.

Reactions to geographic data visualization of infectious disease outbreaks: an experiment on the effectiveness of data presentation format and past occurrence information

OBJECTIVES

This study intended to compare the effectiveness of thematic maps with that of tabular data in comprehension and memory of risk magnitudes, with Zika virus (ZIKV) disease outbreaks in the United States as the subject matter. The study also aimed to examine the effects of data presentation format and past occurrence information on risk perception and risk avoidance intention.

STUDY DESIGN

This study used an experiment.

METHODS

Each participant was randomly assigned to view ZIKV disease 2017 incidence data presented in one of the three formats: a choropleth map, a graduated-circle map, and a table, after which they answered questions about comprehension and memory of risk magnitudes. Each participant was then randomly assigned to view or not to view incidence data of the previous occurrence of ZIKV outbreaks in 2016, after which they answered questions about risk perception and risk avoidance intention.

RESULTS

The results revealed the effectiveness of thematic maps over tabular data in comprehension, risk perception, and risk avoidance intention. Compared to tabular data, the choropleth map led to a better comprehension of relative risk magnitudes, the graduated-circle map led to higher risk perception, and both thematic maps led to greater risk avoidance intention. In contrast, tabular data led to better recognition of absolute risk magnitudes than both thematic maps. In addition, past occurrence information enhanced risk perception and risk avoidance intention.

CONCLUSIONS

The findings reveal the importance of data presentation format in comprehension and memory of risk magnitudes. This can be attributed to the cognitive match between the information emphasized in the presentation and that required by the tasks. The findings also suggest that data presentation format and past occurrence information are important judgmental heuristics that help to form risk perception and risk avoidance intention.

The theoretical basis of enterprise architecture: A critical review and taxonomy of relevant theories

Enterprise architecture is a collection of artifacts describing various aspects of an organization from an integrated business and IT perspective. Practicing enterprise architecture in organizations implies using these artifacts to facilitate information systems planning and improve business and IT alignment. Despite its long history, the enterprise architecture discipline still remains largely atheoretical and lacks a solid theoretical basis. Based on our previous empirical studies of the practical usage of enterprise architecture artifacts in multiple organizations and broad literature analysis, this conceptual article identifies and discusses in detail 10 theories that can be considered key for understanding how an enterprise architecture practice works: actor-network theory, boundary objects theory, cognitive fit theory, communities of practice theory, decision-making theories, information processing theory, knowledge management theory, management fashion theory, media richness theory, and uncertainty principle. Taken together, these theories offer a comprehensive theoretical view of an enterprise architecture practice explaining the role of enterprise architecture artifacts, their usability, and participation of stakeholders and, therefore, may constitute a theoretical basis of the entire enterprise architecture discipline. Although this article does not elaborate on any of these theories, it brings these theories to light, establishes their critical importance for comprehending an enterprise architecture practice, and positions them as central to the enterprise architecture discourse. Each of these theories can be leveraged by enterprise architecture scholars in their future studies for analyzing enterprise architecture practices through respective theoretical lenses. This article intends to provide fresh theoretical insights on enterprise architecture, spark new waves of theoretical enterprise architecture research, and contribute to the development of a sound theoretical foundation for the enterprise architecture discipline.

Mobile Augmented Reality in Electronic Commerce: Investigating User Perception and Purchase Intent Amongst Educated Young Adults

Even though the presence and use of mobile augmented reality (MAR) technology has become increasingly popular in the field of marketing and advertising in recent years, it has largely been neglected in the study of consumer behavior research. This paper utilizes a single-group posttest-only quasi-experimental design to investigate how the feature of mobile augmented reality application influences consumers’ attitude and purchasing intention as explained by the dimensions of persuasion (i.e., consumers’ cognitive, affective, and conative dimensions). Structural Equation Modeling (SEM) with SPSS and AMOS is used to analyze the psychometric survey data collected from 179 participants. The results supported the prediction that MAR application’s real-time interactivity and entertainment increase cognition and affection, respectively; while irritation with MAR application decreases affection. The unsupported hypothesis, which predicted a positive relationship between informativeness and cognition, came as a surprise. The overall result of the study demonstrates the positive influence of MAR application in enhancing consumers’ purchasing intention. Finally, implications and future research directions are discussed.

Collaborating AI and human experts in the maintenance domain

Maintenance decision errors can result in very costly problems. The 4th industrial revolution has given new opportunities for the development of and use of intelligent decision support systems. With these technological advancements, key concerns focus on gaining a better understanding of the linkage between the technicians’ knowledge and the intelligent decision support systems. The research reported in this study has two primary objectives. (1) To propose a theoretical model that links technicians’ knowledge and intelligent decision support systems, and (2) to present a use case how to apply the theoretical model. The foundation of the new model builds upon two main streams of study in the decision support literature: “distribution” of knowledge among different agents, and “collaboration” of knowledge for reaching a shared goal. This study resulted in the identification of two main gaps: firstly, there must be a greater focus upon the technicians’ knowledge; secondly, technicians need assistance to maintain their focus on the big picture. We used the cognitive fit theory, and the theory of distributed situation awareness to propose the new theoretical model called “distributed collaborative awareness model.” The model considers both explicit and implicit knowledge and accommodates the dynamic challenges involved in operational level maintenance. As an application of this model, we identify and recommend some technological developments required in augmented reality based maintenance decision support.

Evaluating effectiveness of information visualizations using cognitive fit theory: A neuroergonomics approach

Information visualizations may be evaluated from the perspective of how they match tasks that must be performed with them, a cognitive fit perspective. However, there is a gap between the high-level references made to cognitive fit and the low-level ability to identify and measure it during human interaction with visualizations. We bridge this gap by using an electroencephalography metric derived from frontal midline theta power and parietal alpha power, known as the task load index, to determine if cognitive effort measured at the level of cortical activity is less when cognitive fit is present compared to when cognitive fit is not. We found that when there is cognitive fit between the type of problem to be solved and the information displayed by a system, the task load index is lower compared to when cognitive fit is not present. We support this finding with subjective (NASA task load index) and performance (response time and accuracy) measures. Our approach, using electroencephalography, provides supplemental information to self-report and performance measures. Findings from this study are important because they (1) provide more validity to the cognitive fit theory using a neurophysiological measure, and (2) use the electroencephalography task load index metric as a means to assess cognitive workload and effort in general.

Developing successful environmental decision support systems: Challenges and best practices

Environmental decision support systems (EDSSs), or DSS applied in the environmental field, have been developed for over 40 years now. However, most of these tools fail to find use or fall out of use extremely quickly. In the aim of aiding in the conception and development of practical and successful decision support systems, i.e. systems that can lead to positive outcomes, this review looks over the existing literature, both EDSS-centric and from broader decision-related fields, to highlight some of the most important challenges influencing the success and usability of these systems. In all, 13 major challenges facing EDSS development were identified and over 60 recommendations and best practices were provided to address these challenges. Though this paper is mainly focused on environmental decision support systems, most of the highlighted information and conclusions are applicable to the development of decision support systems in any field.

Development, implementation and user experience of the Veterans Health Administration (VHA) dialysis dashboard

BACKGROUND

Adults with end-stage renal disease (ESRD) requiring chronic dialysis continue to suffer from poor health outcomes and represent a population rightfully targeted for quality improvement. Electronic dashboards are increasingly used in healthcare to facilitate quality measurement and improvement. However, detailed descriptions of the creation of healthcare dashboards are uncommonly available and formal inquiry into perceptions, satisfaction, and utility by clinical users has been rarely conducted, particularly in the context of dialysis care. Therefore, we characterized the development, implementation and user experience with Veterans Health Administration (VHA) dialysis dashboard.

METHODS

A clinical-quality dialysis dashboard was implemented, which displays clinical performance measures (CPMs) for Veterans with ESRD receiving chronic hemodialysis at all VHA facilities. Data on user experience and perceptions were collected via an e-mail questionnaire to dialysis medical directors and nurse managers at these facilities.

RESULTS

Since 2016 the dialysis dashboard reports monthly on CPMs for approximately 3000 Veterans receiving chronic hemodialysis across 70 VHA dialysis facilities. Of 141 dialysis medical directors and nurse managers, 61 completed the questionnaire. Sixty-six percent of respondents did not find the dashboard difficult to access, 64% agreed that it is easy to use, 59% agreed that its layout is good, and the majority agreed that presentation of data is clear (54%), accurate (56%), and up-to-date (54%). Forty-eight percent of respondents indicated that it helped them improve patient care while 12% did not. Respondents indicated that they used the dialysis dashboard for clinical reporting (71%), quality assessment/performance improvement (QAPI) (62%), and decision-making (23%).

CONCLUSIONS

Most users of the VHA dialysis dashboard found it accurate, up-to-date, easy to use, and helpful in improving patient care. It meets diverse user needs, including administrative reporting, clinical benchmarking and decision-making, and quality assurance and performance improvement (QAPI) activities. Moreover, the VHA dialysis dashboard affords national-, regional- and facility-level assessments of quality of care, guides and motivates best clinical practices, targets QAPI efforts, and informs and promotes population health management improvement efforts for Veterans receiving chronic hemodialysis.

Designing clinical practice feedback reports: three steps illustrated in Veterans Health Affairs long-term care facilities and programs

Background

User-centered design (UCD) methods are well-established techniques for creating useful artifacts, but few studies illustrate their application to clinical feedback reports. When used as an implementation strategy, the content of feedback reports depends on a foundational audit process involving performance measures and data, but these important relationships have not been adequately described. Better guidance on UCD methods for designing feedback reports is needed. Our objective is to describe the feedback report design method for refining the content of prototype reports.

Methods

We propose a three-step feedback report design method (refinement of measures, data, and display). The three steps follow dependencies such that refinement of measures can require changes to data, which in turn may require changes to the display. We believe this method can be used effectively with a broad range of UCD techniques.

Results

We illustrate the three-step method as used in implementation of goals of care conversations in long-term care settings in the U.S. Veterans Health Administration. Using iterative usability testing, feedback report content evolved over cycles of the three steps. Following the steps in the proposed method through 12 iterations with 13 participants, we improved the usability of the feedback reports.

Conclusions

UCD methods can improve feedback report content through an iterative process. When designing feedback reports, refining measures, data, and display may enable report designers to improve the user centeredness of feedback reports.

Toward Objective Evaluation of Working Memory in Visualizations: A Case Study Using Pupillometry and a Dual-Task Paradigm

Cognitive science has established widely used and validated procedures for evaluating working memory in numerous applied domains, but surprisingly few studies have employed these methodologies to assess claims about the impacts of visualizations on working memory. The lack of information visualization research that uses validated procedures for measuring working memory may be due, in part, to the absence of cross-domain methodological guidance tailored explicitly to the unique needs of visualization research. This paper presents a set of clear, practical, and empirically validated methods for evaluating working memory during visualization tasks and provides readers with guidance in selecting an appropriate working memory evaluation paradigm. As a case study, we illustrate multiple methods for evaluating working memory in a visual-spatial aggregation task with geospatial data. The results show that the use of dual-task experimental designs (simultaneous performance of several tasks compared to single-task performance) and pupil dilation can reveal working memory demands associated with task difficulty and dual-tasking. In a dual-task experimental design, measures of task completion times and pupillometry revealed the working memory demands associated with both task difficulty and dual-tasking. Pupillometry demonstrated that participants' pupils were significantly larger when they were completing a more difficult task and when multitasking. We propose that researchers interested in the relative differences in working memory between visualizations should consider a converging methods approach, where physiological measures and behavioral measures of working memory are employed to generate a rich evaluation of visualization effort.

Stressed decision makers and use of decision aids: a literature review and conceptual model

Purpose

Decision making is inherently stressful since the decision maker must choose between potentially conflicting alternatives with unique hazards and uncertain outcomes. Whereas decision aids such as decision support systems (DSS) can be beneficial in stressful scenarios, decision makers sometimes misuse them during decision making, leading to suboptimal outcomes. The purpose of this paper is to investigate the relationship between stress, decision making and decision aid use.

Design/methodology/approach

The authors conduct an extensive multi-disciplinary review of decision making and DSS use through the lens of stress and examine how stress, as perceived by decision makers, impacts their use or misuse of DSS even when such aids can improve decision quality. Research questions examine underlying sources of stress in managerial decision making that influence decision quality, relationships between a decision maker’s perception of stress, DSS use/misuse, and decision quality, and implications for research and practice on DSS design and capabilities.

Findings

The study presents a conceptual model that provides an integrative behavioral view of the impact of a decision maker’s perceived stress on their use of a DSS and the quality of their decisions. The authors identify critical knowledge gaps and propose a research agenda to improve decision quality and use of DSS by considering a decision maker’s perceived stress.

Originality/value

This study provides a previously unexplored view of DSS use and misuse as shaped by the decision and job stress experienced by decision makers. Through the application of four theories, the review and its findings highlight key design principles that can mitigate the negative effects of stressors on DSS use.

The Effects of Image-Based Online Reviews on Customers' Perception Across Product Type and Gender

Online vendors consider image online reviews as an important format to improve customers' buying decision. Prior research examined the influence of review presentation format, but did not focus on image format. Little is known about customers' perception on image online reviews. This study developed a theoretical model to analyze the effect of image reviews across product type and gender. The 2×2×2 between-subject experimental design was conducted to test hypotheses. The results demonstrated that compared to text review, the influence of image format on customers' perception was more significant, but in varying degrees across product type and gender. This study found that image format had more positive impact for experience product's understanding compared to search product. The result also showed that the effect of image format on experience product was not significant greater for females than males, but the perception improvement degree from text to image reviews was saliently different between genders. This study discussed theoretical and managerial contributions of these results.

Automated analysis of free-text comments and dashboard representations in patient experience surveys: a multimethod co-design study

Background

Patient experience surveys (PESs) often include informative free-text comments, but with no way of systematically, efficiently and usefully analysing and reporting these. The National Cancer Patient Experience Survey (CPES), used to model the approach reported here, generates > 70,000 free-text comments annually.

Main aim

To improve the use and usefulness of PES free-text comments in driving health service changes that improve the patient experience.

Secondary aims

(1) To structure CPES free-text comments using rule-based information retrieval (IR) (‘text engineering’), drawing on health-care domain-specific gazetteers of terms, with in-built transferability to other surveys and conditions; (2) to display the results usefully for health-care professionals, in a digital toolkit dashboard display that drills down to the original free text; (3) to explore the usefulness of interdisciplinary mixed stakeholder co-design and consensus-forming approaches in technology development, ensuring that outputs have meaning for all; and (4) to explore the usefulness of Normalisation Process Theory (NPT) in structuring outputs for implementation and sustainability.

Design

A scoping review, rapid review and surveys with stakeholders in health care (patients, carers, health-care providers, commissioners, policy-makers and charities) explored clinical dashboard design/patient experience themes. The findings informed the rules for the draft rule-based IR [developed using half of the 2013 Wales CPES (WCPES) data set] and prototype toolkit dashboards summarising PES data. These were refined following mixed stakeholder, concept-mapping workshops and interviews, which were structured to enable consensus-forming ‘co-design’ work. IR validation used the second half of the WCPES, with comparison against its manual analysis; transferability was tested using further health-care data sets. A discrete choice experiment (DCE) explored which toolkit features were preferred by health-care professionals, with a simple cost–benefit analysis. Structured walk-throughs with NHS managers in Wessex, London and Leeds explored usability and general implementation into practice.

Key outcomes

A taxonomy of ranked PES themes, a checklist of key features recommended for digital clinical toolkits, rule-based IR validation and transferability scores, usability, and goal-oriented, cost–benefit and marketability results. The secondary outputs were a survey, scoping and rapid review findings, and concordance and discordance between stakeholders and methods.

Results

(1) The surveys, rapid review and workshops showed that stakeholders differed in their understandings of the patient experience and priorities for change, but that they reached consensus on a shortlist of 19 themes; six were considered to be core; (2) the scoping review and one survey explored the clinical toolkit design, emphasising that such toolkits should be quick and easy to use, and embedded in workflows; the workshop discussions, the DCE and the walk-throughs confirmed this and foregrounded other features to form the toolkit design checklist; and (3) the rule-based IR, developed using noun and verb phrases and lookup gazetteers, was 86% accurate on the WCPES, but needs modification to improve this and to be accurate with other data sets. The DCE and the walk-through suggest that the toolkit would be well accepted, with a favourable cost–benefit ratio, if implemented into practice with appropriate infrastructure support.

Limitations

Small participant numbers and sampling bias across component studies. The scoping review studies mostly used top-down approaches and focused on professional dashboards. The rapid review of themes had limited scope, with no second reviewer. The IR needs further refinement, especially for transferability. New governance restrictions further limit immediate use.

Conclusions

Using a multidisciplinary, mixed stakeholder, use of co-design, proof of concept was shown for an automated display of patient experience free-text comments in a way that could drive health-care improvements in real time. The approach is easily modified for transferable application.

Future work

Further exploration is needed of implementation into practice, transferable uses and technology development co-design approaches.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

Clinical Performance Feedback Intervention Theory (CP-FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research

BACKGROUND

Providing health professionals with quantitative summaries of their clinical performance when treating specific groups of patients ("feedback") is a widely used quality improvement strategy, yet systematic reviews show it has varying success. Theory could help explain what factors influence feedback success, and guide approaches to enhance effectiveness. However, existing theories lack comprehensiveness and specificity to health care. To address this problem, we conducted the first systematic review and synthesis of qualitative evaluations of feedback interventions, using findings to develop a comprehensive new health care-specific feedback theory.

METHODS

We searched MEDLINE, EMBASE, CINAHL, Web of Science, and Google Scholar from inception until 2016 inclusive. Data were synthesised by coding individual papers, building on pre-existing theories to formulate hypotheses, iteratively testing and improving hypotheses, assessing confidence in hypotheses using the GRADE-CERQual method, and summarising high-confidence hypotheses into a set of propositions.

RESULTS

We synthesised 65 papers evaluating 73 feedback interventions from countries spanning five continents. From our synthesis we developed Clinical Performance Feedback Intervention Theory (CP-FIT), which builds on 30 pre-existing theories and has 42 high-confidence hypotheses. CP-FIT states that effective feedback works in a cycle of sequential processes; it becomes less effective if any individual process fails, thus halting progress round the cycle. Feedback's success is influenced by several factors operating via a set of common explanatory mechanisms: the feedback method used, health professional receiving feedback, and context in which feedback takes place. CP-FIT summarises these effects in three propositions: (1) health care professionals and organisations have a finite capacity to engage with feedback, (2) these parties have strong beliefs regarding how patient care should be provided that influence their interactions with feedback, and (3) feedback that directly supports clinical behaviours is most effective.

CONCLUSIONS

This is the first qualitative meta-synthesis of feedback interventions, and the first comprehensive theory of feedback designed specifically for health care. Our findings contribute new knowledge about how feedback works and factors that influence its effectiveness. Internationally, practitioners, researchers, and policy-makers can use CP-FIT to design, implement, and evaluate feedback. Doing so could improve care for large numbers of patients, reduce opportunity costs, and improve returns on financial investments.

TRIAL REGISTRATION

PROSPERO, CRD42015017541.

Continuous Intention of Entry-Level MIS Professionals to Stay Working in the MIS Field

While many studies focused on what is management information system (MIS) major, how it attracts potential students, and describes needed skills, the perceptions of fresh MIS graduates' continuous intention to stay working in the field, especially in developing countries has not yet been studied empirically. In addition, there seems to be minimal effort investigating the issue from the Arab perspective. This article is the first that develops a continuous model and is built upon the expectation confirmation model and other subsequent efforts. This model links three external variables, among them two new ones: skill-job fit, Wasta, and computer self-efficacy. This model was validated using a sample of 105 graduates in Kuwait. Results reveal that the new cultural variable “Wasta” has impact on continuous intention through the mediation of satisfaction, while skill job and computer self-efficacy affect continuous intention through the mediation of expectation and confirmation, and the skill-job fit exert the strongest impact on intention. This research is expected to advance both theory and practice.

The impact of alerting designs on air traffic controller's eye movement patterns and situation awareness

This research investigated controller' situation awareness by comparing COOPANS's acoustic alerts with newly designed semantic alerts. The results demonstrate that ATCOs' visual scan patterns had significant differences between acoustic and semantic designs. ATCOs established different eye movement patterns on fixations number, fixation duration and saccade velocity. Effective decision support systems require human-centered design with effective stimuli to direct ATCO's attention to critical events. It is necessary to provide ATCOs with specific alerting information to reflect the nature of the critical situation in order to minimise the side effects of startle and inattentional deafness. Consequently, the design of a semantic alert can significantly reduce ATCOs' response time, therefore providing valuable extra time in a time-limited situation to formulate and execute resolution strategies in critical air safety events. The findings of this research indicate that the context-specified design of semantic alerts could improve ATCO's situational awareness and significantly reduce response time in the event of Short Term Conflict Alert (STCA) activation which alerts to two aircraft having less than the required lateral or vertical separation. Practitioner Summary: Eye movements are closely linked with visual attention and can be analysed to explore shifting attention whilst performing monitoring tasks. This research has found that context-specific designed semantic alerts facilitated improved ATCO cognitive processing by integrating visual and auditory resources. Semantic designs have been demonstrated to be superior to acoustic design by directing the operator's attention more quickly to critical situations.Abbreviations: APW: area proximity warning; ASRS: aviation safety reporting system; ATC: air traffic control; ATCO: air traffic controller; ATM: air traffic management; COOPANS: cooperation between air navigation service providers; HCI: human-computer interaction; IAA: irish aviation authority; MSAW: minimum safe altitude warning; MTCD: medium-term conflict detection; SA: situation awareness; STCA: short term conflict alert; TP: trajectory prediction.

Visual Analytics Adoption in Business Enterprises

Visual analytics is increasingly being recognized as a source of competitive advantage. Yet, limited research has examined the factors deriving it organizational adoption. By integrating the technology acceptance model (TAM) with the task-technology fit (TTF) model, this research developed a model for visual analytics adoption in business enterprises. To test the research model, data was collected through a questionnaire survey distributed to 400 business professionals working in a variety of industries in Jordan. Collected data were tested and analyzed using structural equation modeling (SEM) technique. Findings of this research confirmed the applicability of the integrated TAM/TTF model to explain the key factors that affect the adoption of visual analytics systems for work-related tasks. Specifically, the results of this research demonstrated that the task, technology, and user characteristics are fundamental and influential antecedents of TTF, which in turn has a significant positive effect on the perceived usefulness and perceived ease of use of visual analytics systems. Additionally, there are significant positive effects from perceived usefulness and perceived ease of use toward users' intention to adopt visual analytics systems, and a firm relationship between perceived ease of use and perceived usefulness of visual analytics systems. Together all these constructs explain 59.9% of the variance in user's intention to adopt visual analytics systems at the workplace. Findings of this research provide several important implications for research and practice, and thus should help in the design and development of more user-accepted visual analytics systems and applications.

A Comparison of Geovisualizations and Data Tables for Transparency Enablement in the Open Government Data Landscape

Recent years have witnessed progress of public institutions in making their datasets available online, free of charge, for re-use. There have been however limited studies which assess the actual effectiveness of different communication media in making key facts visible to citizens. This article analysed and systematically compared two representations which are relevant in the context of open government data: geovisualizations and data tables. An empirical user study (N=16) revealed that both types of representations have their strengths: geovisualizations make spatial knowledge and the attractiveness of open government data more visible, while data tables are more adequate for the communication of numerical data. The ideas presented are relevant to open data publishers interested in strategies to effectively put the hidden knowledge in current open government datasets into the hands of citizens.

Using cognitive fit theory to evaluate patient understanding of medical images

Patients are increasingly presented with their health data through patient portals in an attempt to engage patients in their own care. Due to the large amounts of data generated during a patient visit, the medical information when shared with patients can be overwhelming and cause anxiety due to lack of understanding. Health care organizations are attempting to improve transparency by providing patients with access to visit information. In this paper, we present our findings from a research study to evaluate patient understanding of medical images. We used cognitive fit theory to evaluate existing tools and images that are shared with patients and analyzed the relevance of such sharing. We discover that medical images need a lot of customization before they can be shared with patients. We suggest that new tools for medical imaging should be developed to fit the cognitive abilities of patients.

Knowledge Contribution as a Factor in Project Selection

Project selection is a crucial decision-making process in many organizations. By adopting a project-based learning perspective, this study sets out to develop a framework to integrate organizational knowledge development with project selection. We utilize various knowledge management models to create a structured evaluation metric to measure project contribution to organizational knowledge. A project selection model, which involves project knowledge contribution as one of the evaluation perspectives, is proposed. Results of a focus group study effectively validate the proposed evaluation metric. The article concludes with an empirical implementation of the model in an electronic component manufacturing company.