The Validity, Generalizability and Feasibility of Summative Evaluation Methods in Visual Analytics

Grand Challenges in Visual Analytics Applications

In the past two decades, research in visual analytics (VA) applications has made tremendous progress, not just in terms of scientific contributions, but also in real-world impact across wide-ranging domains including bioinformatics, urban analytics, and explainable AI. Despite these success stories, questions on the rigor and value of VA application research have emerged as a grand challenge. This article outlines a research and development agenda for making VA application research more rigorous and impactful. We first analyze the characteristics of VA application research and explain how they cause the rigor and value problem. Next, we propose a research ecosystem for improving scientific value, and rigor and outline an agenda with 12 open challenges spanning four areas, including foundation, methodology, application, and community. We encourage discussions, debates, and innovative efforts toward more rigorous and impactful VA research.

In Defence of Visual Analytics Systems: Replies to Critics

The last decade has witnessed many visual analytics (VA) systems that make successful applications to wide-ranging domains like urban analytics and explainable AI. However, their research rigor and contributions have been extensively challenged within the visualization community. We come in defence of VA systems by contributing two interview studies for gathering critics and responses to those criticisms. First, we interview 24 researchers to collect criticisms the review comments on their VA work. Through an iterative coding and refinement process, the interview feedback is summarized into a list of 36 common criticisms. Second, we interview 17 researchers to validate our list and collect their responses, thereby discussing implications for defending and improving the scientific values and rigor of VA systems. We highlight that the presented knowledge is deep, extensive, but also imperfect, provocative, and controversial, and thus recommend reading with an inclusive and critical eye. We hope our work can provide thoughts and foundations for conducting VA research and spark discussions to promote the research field forward more rigorously and vibrantly.

QualDash: Adaptable Generation of Visualisation Dashboards for Healthcare Quality Improvement

Adapting dashboard design to different contexts of use is an open question in visualisation research. Dashboard designers often seek to strike a balance between dashboard adaptability and ease-of-use, and in hospitals challenges arise from the vast diversity of key metrics, data models and users involved at different organizational levels. In this design study, we present QualDash, a dashboard generation engine that allows for the dynamic configuration and deployment of visualisation dashboards for healthcare quality improvement (QI). We present a rigorous task analysis based on interviews with healthcare professionals, a co-design workshop and a series of one-on-one meetings with front line analysts. From these activities we define a metric card metaphor as a unit of visual analysis in healthcare QI, using this concept as a building block for generating highly adaptable dashboards, and leading to the design of a Metric Specification Structure (MSS). Each MSS is a JSON structure which enables dashboard authors to concisely configure unit-specific variants of a metric card, while offloading common patterns that are shared across cards to be preset by the engine. We reflect on deploying and iterating the design of OualDash in cardiology wards and pediatric intensive care units of five NHS hospitals. Finally, we report evaluation results that demonstrate the adaptability, ease-of-use and usefulness of QualDash in a real-world scenario.