A Taxonomy of Evaluation Methods for Information Systems Artifacts

Evaluation in design science: A framework to support project studies in the context of University Research Centres

This paper proposes a discussion of the evaluation of an artefact developed under the Design Science paradigm using the Delphi method. It evaluates the Ecosystem framework of University Research Centres in Project Studies, considering a set of criteria pre-established in the literature. The Delphi method is an evaluation implemented in an electronic platform involving twenty-one participants, among whom were academics, practitioners, and PhD candidates in the field of project management. It reached consensus and stability in two rounds: the results indicate a consensus among the participants in the applicability, novelty, simplicity, completeness, fidelity to modelled phenomena, consistency and internal coherence, scalability, flexibility, interest, elegance, and reusability criteria. Usability was the only criterion that did not attain the predefined percentage of consensus among the participants (70%). Given the framework's characteristics, Delphi participants indicated the need to produce complementary guidelines for its implementation.

The telemedicine community readiness model-successful telemedicine implementation and scale-up

To successfully scale-up telemedicine initiatives (TIs), communities play a crucial role. To empower communities fulfilling this role and increase end users' acceptance of TIs, support tools (from now on entitled artifacts) are needed that include specific measures to implement and scale up telemedicine. Addressing this need, the article introduces the Telemedicine Community Readiness Model (TCRM). The TCRM is designed to help decision-makers in communities to create a favorable environment that facilitates the implementation and scale-up of TIs. The TCRM is a practical tool to assess communities' readiness to implement TIs and identify aspects to improve this readiness. The development process follows a design-science procedure, which integrates literature reviews and semi-structured expert interviews to justify and evaluate design decisions and the final design. For researchers, the paper provides insights into factors that influence telemedicine implementation and scale-up (descriptive role of knowledge) on the community level. For practitioners, it provides a meaningful tool to support the implementation and scale-up of TIs (prescriptive role of knowledge). This should help to realize the potential of telemedicine solutions to increase access to healthcare services and their quality.

City 5.0: Citizen involvement in the design of future cities

A citizen-centric view is key to channeling technological affordances into the development of future cities in which improvements are made with the quality of citizens' life in mind. This paper proposes City 5.0 as a new citizen-centric design paradigm for future cities, in which cities can be seen as markets connecting service providers with citizens as consumers. City 5.0 is dedicated to eliminating restrictions that citizens face when utilizing city services. Our design paradigm focuses on smart consumption and extends the technology-centric concept of smart city with a stronger view on citizens' roadblocks to service usage. Through a series of design workshops, we conceptualized the City 5.0 paradigm and formalized it in a semi-formal model. The applicability of the model is demonstrated using the case of a telemedical service offered by a Spanish public healthcare service provider. The usefulness of the model is validated by qualitative interviews with public organizations involved in the development of technology-based city solutions. Our contribution lies in the advancement of citizen-centric analysis and the development of city solutions for both academic and professional communities.

The assessment of defense information systems complexity: An addition to genetic taxonomy approach

Defense organizations, like the ministry of defense, the armed forces, the general staff of the armed forces, the army, the navy, or the air force units, use a specific business technology. What makes their business technology specific is a predictable changeability of their business processes. Namely, these organizations function in more than one state, each having its own business processes. An organization transits from one state to another in a predictable manner, thus changing its business processes. This kind of business technology is not exclusively restricted to defense organizations, as it also applies to police as well as crisis management organizations. In order to develop information systems (IS) supporting these organizations properly, the complexity of their future IS should be assessed first. This assessment can be performed by relying on existing genetic taxonomies, i.e., by situating the planned defense IS inside the IS genetic taxonomy space, with regard of relevant characteristics of organizational processes supported by the IS. A behavioral dimension described in this paper addresses the dynamics of states defense systems operate in, offering to contribute to the understanding of defense systems’ response to changes in dynamic ecosystems, assisting thereby researchers and practitioners in describing dynamic properties of investigated systems.

Review, Assess, Classify, and Evaluate (RACE): a framework for studying m-health apps and its application for opioid apps

OBJECTIVE

The proliferation of m-health interventions has led to a growing research area of app analysis. We derived RACE (Review, Assess, Classify, and Evaluate) framework through the integration of existing methodologies for the purpose of analyzing m-health apps, and applied it to study opioid apps.

MATERIALS AND METHODS

The 3-step RACE framework integrates established methods and evidence-based criteria used in a successive manner to identify and analyze m-health apps: the Preferred Reporting Items for Systematic Reviews and Meta-Analyses, inter-rater reliability analysis, and Nickerson-Varshney-Muntermann taxonomy.

RESULTS

Using RACE, 153 opioid apps were identified, assessed, and classified leading to dimensions of Target Audience, Key Function, Operation, Security & Privacy, and Impact, with Cohen's kappa < 1.0 suggesting subjectivity in app narrative assessments. The most common functions were education (24%), prescription (16%), reminder-monitoring-support (13%), and treatment & recovery (37%). A majority are passive apps (56%). The target audience are patients (49%), healthcare professionals (39%), and others (12%). Security & Privacy is evident in 84% apps.

DISCUSSION

Applying the 3-step RACE framework revealed patterns and gaps in opioid apps leading to systematization of knowledge. Lessons learned can be applied to the study of m-health apps for other health conditions.

CONCLUSION

With over 350 000 existing and emerging m-health apps, RACE shows promise as a robust and replicable framework for analyzing m-health apps for specific health conditions. Future research can utilize the RACE framework toward understanding the dimensions and characteristics of existing m-health apps to inform best practices for collaborative, connected and continued care.

Overcoming the Single-IS Paradigm in Individual-Level IS Research

Over the years, the number of digital technologies that individuals use in their work and nonwork lives has increased significantly. These different technologies are often subject to interactions and interdependencies among them, which creates new challenges and opportunities for individuals. For instance, multiple digital technologies might be incompatible or offer redundant information to individuals. In this research, we offer a framework that can help scholars to study phenomena that involve multiple digital technologies and can assist designers and developers in making design decisions that facilitate beneficial interactions between technologies and mitigate undesirable ones.

Agility and Resilience in Information Systems Research

This research analyzes how the concepts of agility and resilience are treated in IS literature. While agility has been an object of study in IS research for many decades, resilience is a fairly new topic. Both are gaining strategic importance in a firm’s sustainability and growth, especially given the remarkable changes in the landscape that the firm operates in. It is thus important to understand what agility and resilience mean in IS research. Our paper is a first attempt to study IS literature and provide a thematic analysis of facets of each concept. In doing so, we identify aspects that are common to both agility and resilience and those that are unique to each. The results of this study can be used for an empirical examination of the two constructs and a validation of how they can be measured in firms. IS researchers and industrial practitioners can benefit from a deeper understanding of agility and resilience.

Libraries Fight Disinformation: An Analysis of Online Practices to Help Users’ Generations in Spotting Fake News

The work of libraries during the COVID-19 pandemic, as facilitators of reliable information on health issues, has shown that these entities can play an active role as verification agents in the fight against disinformation (false information that is intended to mislead), focusing on media and informational literacy. To help citizens, these entities have developed a wide range of actions that range from online seminars, to learning how to evaluate the quality of a source, to video tutorials or the creation of repositories with resources of various natures. To identify the most common media literacy practices in the face of fake news (news that conveys or incorporates false, fabricated, or deliberately misleading information), this exploratory study designed an ad hoc analysis sheet, validated by the inter-judge method, which allowed one to classify the practices of N = 216 libraries from all over the world. The results reveal that the libraries most involved in this task are those belonging to public universities. Among the actions carried out to counteract misinformation, open-access materials that favor self-learning stand out. These resources, aimed primarily at university students and adults in general, are aimed at acquiring skills related to fact-checking and critical thinking. Therefore, libraries vindicate their role as components of the literacy triad, together with professors and communication professionals.

An Update for Taxonomy Designers

Taxonomies are classification systems that help researchers conceptualize phenomena based on their dimensions and characteristics. To address the problem of ‘ad-hoc’ taxonomy building, Nickerson et al. (2013) proposed a rigorous taxonomy development method for information systems researchers. Eight years on, however, the status quo of taxonomy research shows that the application of this method lacks consistency and transparency and that further guidance on taxonomy evaluation is needed. To fill these gaps, this study (1) advances existing methodological guidance and (2) extends this guidance with regards to taxonomy evaluation. Informed by insights gained from an analysis of 164 taxonomy articles published in information systems outlets, this study presents an extended taxonomy design process together with 26 operational taxonomy design recommendations. Representing an update for taxonomy designers, it contributes to the prescriptive knowledge on taxonomy design and seeks to augment both rigorous taxonomy building and evaluation.

The Phishing Funnel Model: A Design Artifact to Predict User Susceptibility to Phishing Websites

Phishing is a significant security concern for organizations, threatening employees and members of the public. Phishing threats against employees can lead to severe security incidents, whereas those against the public can undermine trust, satisfaction, and brand equity. At the root of the problem is the inability of Internet users to identify phishing attacks even when using anti-phishing tools. We propose the phishing funnel model (PFM), a framework for predicting user susceptibility to phishing websites. PFM incorporates user, threat, and tool-related factors to predict actions during four key stages of the phishing process: visit, browse, consider legitimate, and intention to transact. We evaluated the efficacy of PFM in a 12-month longitudinal field experiment in two organizations involving 1,278 employees and 49,373 phishing interactions. PFM significantly outperformed competing models in terms of its ability to predict user susceptibility to phishing attacks. A follow-up three-month field study revealed that employees using PFM were significantly less likely to interact with phishing threats relative to comparison models and baseline warnings. Results of a cost-benefit analysis suggest that interventions guided by PFM could reduce annual phishing-related costs by nearly $1,900 per employee relative to comparison prediction methods.

Behavioral Economics in Information Systems Research: Critical Analysis and Research Strategies

Theories of decision-making have long been important foundations for information systems (IS) research and much of IS is concerned with information processing for decision making. The discipline of behavioral economics (BE) provides the dominant contemporary approach for understanding human decision-making. Therefore, it is logical that IS research that involves decision making should consider BE as foundation or reference theory. Surprisingly, and despite calls for greater use of BE in IS research, it seems that IS has been slow to adopt contemporary BE as reference theory. This paper reports a critical analysis of BE in all fields of IS based on an intensive investigation of quality IS research using bibliometric content analysis. The analysis shows that IS researchers have a general understanding of BE, but their use of the theories has an ad hoc feel where only a narrow range of BE concepts and theories tend to form the foundation of IS research. The factors constraining the adoption of BE theories in IS are discussed and strategies for the use of this influential foundation theory are proposed. Guidance is provided on how BE could be used in various aspects of IS. The paper concludes with the view that BE reference theory has the potential to transform significant areas of IS research.

Information Technology Governance for Higher Education Institutions: A Multi-Country Study

Information Technology governance (ITG) calls for the definition and implementation of formal practices at the highest level in the organization, involving structures, processes and relational practices for the creation of business value from IT investments. However, determining the right ITG practices remains a complex endeavor. Previous studies identify IT governance practices used in the health and financial sectors. As universities have many unique characteristics, it is highly unlikely that the ITG experiences of the financial and health industry can be directly applied to universities. This study, using Design Science Research (DSR), develops a baseline with advised practices for the university sector. The analysis of thirty-four case studies from the literature review provides a set of practices as a starting point for the development of the baseline model proposal through multiple case studies involving interviews with IT directors, in ten universities in five countries: eight new practices emerge in this study. The model proposed was evaluated by experts. The result is a baseline model with adequate practices for IT governance in universities as well as a set of guidelines for its implementation. Findings revealed that is possible to extend the ITG practices’ baseline when looking at specific contexts.

Service selection in mobile environments: considering multiple users and context-awareness

In mobile environments, users often need to coordinate their actions with other users with regard to user-individual context information like current location when selecting suitable services for a process. Thereby, some users may prefer to conduct particular services together with certain other users. Such multi-user context-aware service selections could result in complex decision problems - making decision support for the participating users highly valuable or even necessary. To do so, we propose an optimisation-based service selection approach for multi-user context-aware processes. We also show how our approach provides decision support by evaluating its efficacy based on a real-world scenario.

Understanding Alliance Portfolios Using Visual Analytics

In an increasingly global and competitive business landscape, firms must collaborate and partner with others to ensure survival, growth, and innovation. Understanding the evolutionary composition of a firm’s relationship portfolio and the underlying formation strategy is a difficult task given the multidimensional, temporal, and geospatial nature of the data. In collaboration with senior executives, we iteratively determine core design requirements and then design and implement an interactive visualization system that enables decision makers to gain both systemic (macro) and detailed (micro) insights into a firm’s alliance activities and discover patterns of multidimensional relationship formation. Our system provides both sequential and temporal representation modes, a rich set of additive cross-linked filters, the ability to stack multiple alliance portfolios, and a dynamically updated activity state model visualization to inform decision makers of past and likely future relationship moves. We illustrate our tool with examples of alliance activities of firms listed on the S8P 500. A controlled experiment and real-world evaluation with practitioners and researchers reveals significant evidence of the value of our visual analytic tool. Our design study contributes to design science by addressing a known problem (i.e., alliance portfolio analysis) with a novel solution (interactive, pixel-based multivariate visualization) and to the rapidly emerging area of data-driven visual decision support in corporate strategy contexts. We conclude with implications and future research opportunities.