Critical incidents and journey mapping as techniques to evaluate the impact of online evidence retrieval systems on health care delivery and patient outcomes

Exploring the outcomes of research engagement using the observation method in an online setting

OBJECTIVE

The objective of this study was to explore the outcomes of research engagement (patient engagement, PE) in the context of qualitative research.

DESIGN

We observed engagement in two groups comprised of patients, clinicians and researchers tasked with conducting a qualitative preference exploration project in inflammatory bowel disease. One group was led by a patient research partner (PLG, partner led group) and the other by an academic researcher (RLG, researcher led group). A semistructured guide and a set of critical outcomes of research engagement were used as a framework to ground our analysis.

SETTING

The study was conducted online.

PARTICIPANTS

Patient research partners (n=5), researchers (n=5) and clinicians (n=4) participated in this study.

MAIN OUTCOME MEASURES

Transcripts of meetings, descriptive and reflective observation data of engagement during meetings and email correspondence between group members were analysed to identify the outcomes of PE.

RESULTS

Both projects were patient-centred, collaborative, meaningful, rigorous, adaptable, ethical, legitimate, understandable, feasible, timely and sustainable. Patient research partners (PRPs) in both groups wore dual hats as patients and researchers and influenced project decisions wearing both hats. They took on advisory and operational roles. Collaboration seemed easier in the PLG than in the RLG. The RLG PRPs spent more time than their counterparts in the PLG sharing their experience with biologics and helping their group identify a meaningful project question. A formal literature review informed the design, project materials and analysis in the RLG, while the formal review informed the project materials and analysis in the PLG. A PRP in the RLG and the PLG lead leveraged personal connections to facilitate recruitment. The outcomes of both projects were meaningful to all members of the groups.

CONCLUSIONS

Our findings show that engagement of PRPs in research has a positive influence on the project design and delivery in the context of qualitative research in both the patient-led and researcher-led group.

Journey mapping as a novel approach to healthcare: a qualitative mixed methods study in palliative care

Background

Journey mapping involves the creation of visual narrative timelines depicting the multidimensional relationship between a consumer and a service. The use of journey maps in medical research is a novel and innovative approach to understanding patient healthcare encounters.

Objectives

To determine possible applications of journey mapping in medical research and the clinical setting. Specialist palliative care services were selected as the model to evaluate this paradigm, as there are numerous evidence gaps and inconsistencies in the delivery of care that may be addressed using this tool.

Methods

A purposive convenience sample of specialist palliative care providers from the Supportive and Palliative Care unit of a major Australian tertiary health service were invited to evaluate journey maps illustrating the final year of life of inpatient palliative care patients. Sixteen maps were purposively selected from a sample of 104 consecutive patients. This study utilised a qualitative mixed-methods approach, incorporating a modified Delphi technique and thematic analysis in an online questionnaire.

Results

Our thematic and Delphi analyses were congruent, with consensus findings consistent with emerging themes. Journey maps provided a holistic patient-centred perspective of care that characterised healthcare interactions within a longitudinal trajectory. Through these journey maps, participants were able to identify barriers to effective palliative care and opportunities to improve care delivery by observing patterns of patient function and healthcare encounters over multiple settings.

Conclusions

This unique qualitative study noted many promising applications of the journey mapping suitable for extrapolation outside of the palliative care setting as a review and audit tool, or a mechanism for providing proactive patient-centred care. This is particularly significant as machine learning and big data is increasingly applied to healthcare.

A Patient-Centered Information System (myED) for Emergency Care Journeys: Design, Development, and Initial Adoption

Background

Medical care is highly complex in that it addresses patient-centered health goals that require the coordination of multiple care providers. Emergency department (ED) patients currently lack a sense of predictability about ED procedures. This increases frustration and aggression. Herein, we describe a system for providing real-time information to ED patients regarding the procedures in their ED medical journey.

Objective

This study aimed to develop a system that provides patients with dynamically updated information about the specific procedures and expected waiting times in their personal ED journey, and to report initial evaluations of this system.

Methods

To develop the myED system, we extracted information from hospital databases and translated it using process mining and user interface design into a language that is accessible and comprehensible to patients. We evaluated the system using a mixed methods approach that combined observations, interviews, and online records.

Results

Interviews with patients, accompanying family members, and health care providers (HCPs) confirmed patients’ needs for information about their personal ED journey. The system developed enables patients to access this information on their personal mobile phones through a responsive website. In the third month after deployment, 492 of 1614 (30.48%) patients used myED. Patients’ understanding of their ED journey improved significantly (F_8,299=2.519; P=.01), and patients showed positive reactions to the system. We identified future challenges, including achieving quick engagement without delaying medical care. Salient reasons for poor system adoption were patients’ medical state and technological illiteracy. HCPs confirmed the potential of myED and identified means that could improve patient experience and staff cooperation.

Conclusions

Our iterative work with ED patients, HCPs, and a multidisciplinary team of developers yielded a system that provides personal information to patients about their ED journey in a secure, effective, and user-friendly way. MyED communicates this information through mobile technology. This improves health care by addressing patients’ psychological needs for information and understanding, which are often overlooked. We continue to test and refine the system and expect to find positive effects of myED on patients’ ED experience and hospital operations.

Evaluating a federated medical search engine: tailoring the methodology and reporting the evaluation outcomes

BACKGROUND

Federated medical search engines are health information systems that provide a single access point to different types of information. Their efficiency as clinical decision support tools has been demonstrated through numerous evaluations. Despite their rigor, very few of these studies report holistic evaluations of medical search engines and even fewer base their evaluations on existing evaluation frameworks.

OBJECTIVES

To evaluate a federated medical search engine, MedSocket, for its potential net benefits in an established clinical setting.

METHODS

This study applied the Human, Organization, and Technology (HOT-fit) evaluation framework in order to evaluate MedSocket. The hierarchical structure of the HOT-factors allowed for identification of a combination of efficiency metrics. Human fit was evaluated through user satisfaction and patterns of system use; technology fit was evaluated through the measurements of time-on-task and the accuracy of the found answers; and organization fit was evaluated from the perspective of system fit to the existing organizational structure.

RESULTS

Evaluations produced mixed results and suggested several opportunities for system improvement. On average, participants were satisfied with MedSocket searches and confident in the accuracy of retrieved answers. However, MedSocket did not meet participants' expectations in terms of download speed, access to information, and relevance of the search results. These mixed results made it necessary to conclude that in the case of MedSocket, technology fit had a significant influence on the human and organization fit. Hence, improving technological capabilities of the system is critical before its net benefits can become noticeable.

CONCLUSIONS

The HOT-fit evaluation framework was instrumental in tailoring the methodology for conducting a comprehensive evaluation of the search engine. Such multidimensional evaluation of the search engine resulted in recommendations for system improvement.

Journey to vaccination: a protocol for a multinational qualitative study

INTRODUCTION

In the past two decades, childhood vaccination coverage has increased dramatically, averting an estimated 2-3 million deaths per year. Adult vaccination coverage, however, remains inconsistently recorded and substandard. Although structural barriers are known to limit coverage, social and psychological factors can also affect vaccine uptake. Previous qualitative studies have explored beliefs, attitudes and preferences associated with seasonal influenza (flu) vaccination uptake, yet little research has investigated how participants' context and experiences influence their vaccination decision-making process over time. This paper aims to provide a detailed account of a mixed methods approach designed to understand the wider constellation of social and psychological factors likely to influence adult vaccination decisions, as well as the context in which these decisions take place, in the USA, the UK, France, India, China and Brazil.

METHODS AND ANALYSIS

We employ a combination of qualitative interviewing approaches to reach a comprehensive understanding of the factors influencing vaccination decisions, specifically seasonal flu and tetanus. To elicit these factors, we developed the journey to vaccination, a new qualitative approach anchored on the heuristics and biases tradition and the customer journey mapping approach. A purposive sampling strategy is used to select participants who represent a range of key sociodemographic characteristics. Thematic analysis will be used to analyse the data. Typical journeys to vaccination will be proposed.

ETHICS AND DISSEMINATION

Vaccination uptake is significantly influenced by social and psychological factors, some of which are under-reported and poorly understood. This research will provide a deeper understanding of the barriers and drivers to adult vaccination. Our findings will be published in relevant peer-reviewed journals and presented at academic conferences. They will also be presented as practical recommendations at policy and industry meetings and healthcare professionals' forums. This research was approved by relevant local ethics committees.

Number needed to benefit from information (NNBI): proposal from a mixed methods research study with practicing family physicians

PURPOSE

We wanted to describe family physicians' use of information from an electronic knowledge resource for answering clinical questions, and their perception of subsequent patient health outcomes; and to estimate the number needed to benefit from information (NNBI), defined as the number of patients for whom clinical information was retrieved for 1 to benefit.

METHODS

We undertook a mixed methods research study, combining quantitative longitudinal and qualitative research studies. Participants were 41 family physicians from primary care clinics across Canada. Physicians were given access to 1 electronic knowledge resource on handheld computer in 2008-2009. For the outcome assessment, participants rated their searches using a validated method. Rated searches were examined during interviews guided by log reports that included ratings. Cases were defined as clearly described searches where clinical information was used for a specific patient. For each case, interviewees described information-related patient health outcomes. For the mixed methods data analysis, quantitative and qualitative data were merged into clinical vignettes (each vignette describing a case). We then estimated the NNBI.

RESULTS

In 715 of 1,193 searches for information conducted during an average of 86 days, the search objective was directly linked to a patient. Of those searches, 188 were considered to be cases. In 53 cases, participants associated the use of information with at least 1 patient health benefit. This finding suggested an NNBI of 14 (715/53).

CONCLUSION

The NNBI may be used in further experimental research to compare electronic knowledge resources. A low NNBI can encourage clinicians to search for information more frequently. If all searches had benefits, the NNBI would be 1. In addition to patient benefits, learning and knowledge reinforcement outcomes are frequently reported.

Summarizing clinical pathways from event logs

OBJECTIVE

Clinical pathway analysis, as a pivotal issue in ensuring specialized, standardized, normalized and sophisticated therapy procedures, is receiving increasing attention in the field of medical informatics. Research in clinical pathway analysis has so far mostly focused on looking at aggregated data seen from an external perspective, and only provide very limited insight into the pathways. In some recent work, process mining techniques have been studied in discovering clinical pathway models from data. While it is interesting, discovered models may provide too much detail to give a comprehensive summary of the pathway. Moreover, the number of patterns discovered can be large. Alternatively, this article presents a new approach to build a concise and comprehensive summary that describes the entire structure of a clinical pathway, while revealing essential/critical medical behaviors in specific time intervals over the whole time period of the pathway.

METHODS

The presented approach summarizes a clinical pathway from the collected clinical event log, which regularly records all kinds of patient therapy and treatment activities in clinical workflow by various hospital information systems. The proposed approach formally defines the clinical pathway summarization problem as an optimization problem that can be solved in polynomial time by using a dynamic-programming algorithm. More specifically, given an input event log, the presented approach summarizes the pathway by segmenting the observed time period of the pathway into continuous and overlapping time intervals, and discovering frequent medical behavior patterns in each specific time interval from the log.

RESULTS

The proposed approach is evaluated via real-world data-sets, which are extracted from Zhejiang Huzhou Central hospital of China with regard to four specific diseases, i.e., bronchial lung cancer, colon cancer, gastric cancer, and cerebral infarction, in two years (2007.08-2009.09). Although the medical behaviors contained in these logs are very diverse and heterogeneous, experimental results indicates that the presented approach is feasible to construct condensed clinical pathway summaries in polynomial time from the collected logs, and have a linear scalability against the increasing size of the logs.

CONCLUSION

Experiments on real data-sets illustrate that the presented approach is efficient and discovers high-quality results: the observed time period of a clinical pathway is correctly segmented into a set of continuous and overlapping time intervals, in which essential/critical medical behaviors are well discovered from the event log to form the backbone of a clinical pathway. The experimental results indicate that the generated clinical pathway summary not only reveals the global structure of a pathway, but also provides a thorough understanding of the way in which actual medical behaviors are practiced in specific time intervals, which might be essential from the perspectives of clinical pathway analysis and improvement.

Supporting adaptive clinical treatment processes through recommendations

OBJECTIVES

Efficient clinical treatment processes is considered a key factor of medical quality control. Current IT solutions are far away from this perspective since they typically have difficulty supporting the variances occurring in clinical practices, and providing adequate flexible support of clinical processes.

METHODS

This paper proposes a hybrid approach based on rough set theory and case-based reasoning to allow physicians to rapidly adjust patients' treatment processes to changes of patients' clinical states. In detail, the proposed approach recommends appropriate treatment plans in clinical process execution by adopting a similarity measure to select appropriate clinical treatment plans executed on patients who presented similar features to the current one. Such clinical treatment plans are then applied to suggest which actions to perform next in clinical treatment process execution.

RESULTS

As a motivating scenario, this study performs the experiments of type 2 diabetes patient's treatment process. The results show that the proposed approach is feasible to recommend suitable clinical treatment plans in clinical process execution, which makes adaptive clinical treatment processes possible.

On mining clinical pathway patterns from medical behaviors

OBJECTIVE

Clinical pathway analysis, as a pivotal issue in ensuring specialized, standardized, normalized and sophisticated therapy procedures, is receiving increasing attention in the field of medical informatics. Clinical pathway pattern mining is one of the most important components of clinical pathway analysis and aims to discover which medical behaviors are essential/critical for clinical pathways, and also where temporal orders of these medical behaviors are quantified with numerical bounds. Even though existing clinical pathway pattern mining techniques can tell us which medical behaviors are frequently performed and in which order, they seldom precisely provide quantified temporal order information of critical medical behaviors in clinical pathways.

METHODS

This study adopts process mining to analyze clinical pathways. The key contribution of the paper is to develop a new process mining approach to find a set of clinical pathway patterns given a specific clinical workflow log and minimum support threshold. The proposed approach not only discovers which critical medical behaviors are performed and in which order, but also provides comprehensive knowledge about quantified temporal orders of medical behaviors in clinical pathways.

RESULTS

The proposed approach is evaluated via real-world data-sets, which are extracted from Zhejiang Huzhou Central hospital of China with regard to six specific diseases, i.e., bronchial lung cancer, gastric cancer, cerebral hemorrhage, breast cancer, infarction, and colon cancer, in two years (2007.08-2009.09). As compared to the general sequence pattern mining algorithm, the proposed approach consumes less processing time, generates quite a smaller number of clinical pathway patterns, and has a linear scalability in terms of execution time against the increasing size of data sets.

CONCLUSION

The experimental results indicate the applicability of the proposed approach, based on which it is possible to discover clinical pathway patterns that can cover most frequent medical behaviors that are most regularly encountered in clinical practice. Therefore, it holds significant promise in research efforts related to the analysis of clinical pathways.

Impact of knowledge resources linked to an electronic health record on frequency of unnecessary tests and treatments

INTRODUCTION

Electronic knowledge resources have the potential to rapidly provide answers to clinicians' questions. We sought to determine clinicians' reasons for searching these resources, the rate of finding relevant information, and the perceived clinical impact of the information they retrieved.

METHODS

We asked general internists, family physicians, and clinical nurse practitioners to complete the Information Assessment Method (IAM) survey after searching 1 of 2 electronic knowledge resources linked in the electronic health record. IAM stimulates reflection on the relevance, cognitive impact, use, and potential health outcomes of retrieved clinical information.

RESULTS

Forty-two clinicians rated 502 searches (mean 12, range 1-48) and reported finding information 75% (n = 375) of the time. The most common reasons for searching were to address a clinical question (411, 82%) and for curiosity (75, 15%). In 68% of the rated searches (341), participants indicated they would use the retrieved information for at least 1 patient. In 31% (157) of rated searches, clinicians expected the retrieved information to benefit the patient by avoiding an unnecessary or inappropriate treatment, diagnostic procedure, or preventive intervention.

CONCLUSIONS

Searches in electronic knowledge resources frequently yield relevant information that may benefit the patient by, for example, avoiding an inappropriate diagnostic procedure or treatment. Knowing that searches for answers to clinical questions can result in patient health benefits should intensify efforts to encourage clinicians to pursue answers to their questions.

Using recommendation to support adaptive clinical pathways

Clinical pathways are among the main tools used to manage the quality in health-care concerning the standardization of care processes. This paper deals with a recommendation service to support adaptive clinical pathways. The proposed approach can guide physicians in clinical pathways by providing recommendations on possible next steps based on the measurement of the target patient status and medical knowledge from completed clinical cases. The efficiency and usability of the proposed method is validated by experiments referring to a real data set extracted from Electronic Patient Records. The experimental results indicate that the recommendation service can provide its users with advice rationales that remain consistent even when patient status has changed. This makes adaptive clinical pathways possible.

Implementing a web-based oncology protocol system in Australia: evaluation of the first 3 years of operation

BACKGROUND

EviQ is a web-based oncology protocol system launched across Australia in 2005 (http://www.eviq.org.au). We evaluated eviQ use at the point-of-care and determined the factors impacting on its uptake and routine use in the first three years of operation.

METHODS

We conducted a suite of qualitative and quantitative studies with over 200 Australian oncology physicians, nurses and pharmacists working at treatment centres in diverse geographical locations.

RESULTS

EviQ was part of routine care at many hospitals; however, the way in which it was used at the point-of-care varies according to clinician roles and hospital location. We identified a range of factors impacting on eviQ uptake and routine use. Infrastructure, such as availability of point-of-care computers, and formal policies endorsing eviQ are fundamental to increasing uptake. Furthermore, the level of clinical and computer experience of end-users, the attitudes and behaviour of clinicians, endorsement and promotion strategies, and level and type of eviQ education all need to be considered and managed to ensure that the system is being used to its full potential.

CONCLUSION

Our findings show that the dissemination of web-based treatment protocols does not guarantee widespread use. Organisational, environmental and clinician-specific factors play a role in uptake and utilisation. The deployment of sufficient computer infrastructure, implementation of targeted training programmes and hospital policies and investment in marketing approaches are fundamental to uptake and continued use. This study highlights the value of ongoing monitoring and evaluation to ensure systems like eviQ achieve their primary purpose - reducing treatment variation and improving quality of care.

Tensions associated with the use of electronic knowledge resources within clinical decision-making processes: a multiple case study

CONTENT AND OBJECTIVE: Health professionals now routinely use electronic knowledge resources (EKRs). Few studies have considered EKR-related tensions which may arise in a clinical decision-making context. The present study aims to explore three types of tension: (1) user-computer tension, (2) social tensions, and (3) organizational tensions (constraints associated with organizational routines and health policies).

DESIGN, PARTICIPANTS, INTERVENTION, SETTING

We conducted a multiple case study, examining Family Medicine residents' searches for information in everyday life. Cases were defined as critical searches for information among 17 first year family medicine residents using InfoRetriever 2003/2004 on a PDA over 1.5 months at McGill University. InfoRetriever-derived information was used within a resident-patient decision-making context in 84 of 156 cases. For each case, residents were interviewed, and extracts of interview transcripts were assigned to themes using specialized software (presence of tension; type of tension). Further computer-assisted lexical-semantic analysis was performed on transcripts. Authors reached consensus on assignments.

RESULTS

Twenty-five cases with tension were identified (one case had two types of tension), and illustrate the above mentioned types of tensions: (T1) tension between the resident and InfoRetriever (N=16); (T2) InfoRetriever-related tension between the resident and other social actors, specifically supervisors, other health care professionals and patients (N=7); (T3) InfoRetriever-related tension between the resident and the health organization/system (N=3).

CONCLUSIONS

Results suggest EKR usage in a clinical decision-making context may have negative consequences when three types of tension arise in a clinical decision-making context. Illustrated types of tension are interrelated and not mutually exclusive. Awareness of EKR-related tensions may help clinicians to integrate EKRs in practice.

Effectiveness of topic-specific infobuttons: a randomized controlled trial

OBJECTIVE

Infobuttons are decision support tools that provide links within electronic medical record systems to relevant content in online information resources. The aim of infobuttons is to help clinicians promptly meet their information needs. The objective of this study was to determine whether infobutton links that direct to specific content topics ("topic links") are more effective than links that point to general overview content ("nonspecific links").

DESIGN

Randomized controlled trial with a control and an intervention group. Clinicians in the control group had access to nonspecific links, while those in the intervention group had access to topic links.

MEASUREMENTS

Infobutton session duration, number of infobutton sessions, session success rate, and the self-reported impact that the infobutton session produced on decision making.

RESULTS

The analysis was performed on 90 subjects and 3,729 infobutton sessions. Subjects in the intervention group spent 17.4% less time seeking for information (35.5 seconds vs. 43 seconds, p = 0.008) than those in the control group. Subjects in the intervention group used infobuttons 20.5% (22 sessions vs. 17.5 sessions, p = 0.21) more often than in the control group, but the difference was not significant. The information seeking success rate was equally high in both groups (89.4% control vs. 87.2% intervention, p = 0.99). Subjects reported a high positive clinical impact (i.e., decision enhancement or knowledge update) in 62% of the sessions. Limitations The exclusion of users with a low frequency of infobutton use and the focus on medication-related information needs may limit the generalization of the results. The session outcomes measurement was based on clinicians' self-assessment and therefore prone to bias.

CONCLUSION

The results support the hypothesis that topic links are more efficient than nonspecific links regarding the time seeking for information. It is unclear whether the statistical difference demonstrated will result in a clinically significant impact. However, the overall results confirm previous evidence that infobuttons are effective at helping clinicians to answer questions at the point of care and demonstrate a modest incremental change in the efficiency of information delivery for routine users of this tool.

An action research protocol to strengthen system-wide inter-professional learning and practice [LP0775514]

BACKGROUND

Inter-professional learning (IPL) and inter-professional practice (IPP) are thought to be critical determinants of effective care, improved quality and safety and enhanced provider morale, yet few empirical studies have demonstrated this. Whole-of-system research is even less prevalent. We aim to provide a four year, multi-method, multi-collaborator action research program of IPL and IPP in defined, bounded health and education systems located in the Australian Capital Territory (ACT). The project is funded by the Australian Research Council under its industry Linkage Program.

METHODS/DESIGN

The program of research will examine in four inter-related, prospective studies, progress with IPL and IPP across tertiary education providers, professional education, regulatory and registration bodies, the ACT health system's streams of care activities and teams, units and wards of the provider facilities of the ACT health system. One key focus will be on push-pull mechanisms, ie, how the education sector creates student-enabled IPP and the health sector demands IPL-oriented practitioners. The studies will examine four research aims and meet 20 research project objectives in a comprehensive evaluation of ongoing progress with IPL and IPP.

DISCUSSION

IPP and IPL are said to be cornerstones of health system reforms. We will measure progress across an entire health system and the clinical and professional education systems that feed into it. The value of multi-methods, partnership research and a bi-directional push-pull model of IPL and IPP will be tested. Widespread dissemination of results to practitioners, policymakers, managers and researchers will be a key project goal.

Ambulance officers' use of online clinical evidence

Background

Hospital-based clinicians have been shown to use and attain benefits from online evidence systems. To our knowledge there have been no studies investigating whether and how ambulance officers use online evidence systems if provided. We surveyed ambulance officers to examine their knowledge and use of the Clinical Information Access Program (CIAP), an online evidence system providing 24-hour access to information to support evidence-based practice.

Methods

A questionnaire was completed by 278 ambulance officers in New South Wales, Australia. Comparisons were made between those who used CIAP and officers who had heard of, but not used CIAP.

Results

Half the sample (48.6%) knew of, and 28.8% had used CIAP. Users were more likely to have heard of CIAP from a CIAP representative/presentation, non-users from written information. Compared to ambulance officers who had heard of but had not used CIAP, users were more likely to report better computer skills and that their supervisors regarded use of CIAP as a legitimate part of ambulance officers' clinical role. The main reasons for non-use were lack of access(49.0%) and training(31.4%). Of users, 51.3% rated their skills at finding information as good/very good, 67.5% found the information sought all/most of the time, 87.3% believed CIAP had the potential to improve patient care and 28.2% had directly experienced this. Most access to CIAP occurred at home. The databases frequently accessed were MIMS (A medicines information database) (73.8%) and MEDLINE(67.5%). The major journals accessed were Journal of Emergency Nursing(37.5%), American Journal of Medicine(30.0%) and JAMA(27.5%).

Conclusion

Over half of ambulance officers had not heard of CIAP. The proportion who knew about and used CIAP was also low. Reasons for this appear to be a work culture not convinced of CIAP's relevance to pre-hospital patient care and lack of access to CIAP at work. Ambulance officers who used CIAP accessed it primarily from home and valued it highly. Lack of access to CIAP at central work locations deprives ambulance officers of many of the benefits of an online evidence system.