Computerized Triggers of Big Data to Detect Delays in Follow-up of Chest Imaging Results

Reducing the Risk of Delayed Colorectal Cancer Diagnoses Through an Ambulatory Safety Net Collaborative

BACKGROUND

An estimated 12 million adults in the United States experience delayed diagnoses and other diagnostic errors annually. Ambulatory safety nets (ASNs) are an intervention to reduce delayed diagnoses by identifying patients with abnormal results overdue for follow-up using registries, workflow redesign, and patient navigation. The authors sought to co-design a collaborative and implement colorectal cancer (CRC) ASNs across various health care settings.

METHODS

A working group was convened to co-design implementation guidance, measures, and the collaborative model. Collaborative sites were recruited through a medical professional liability insurance program and chose to begin with developing an ASN for positive at-home CRC screening or overdue surveillance colonoscopy. The 18-month Breakthrough Series Collaborative ran from January 2022 to July 2023, with sites continuing to collect data while sustaining their ASNs. Data were collected from sites monthly on patients in the ASN, including the proportion that was successfully contacted, scheduled, and completed a follow-up colonoscopy.

RESULTS

Six sites participated; four had an operational ASN at the end of the Breakthrough Series, with the remaining sites launching three months later. From October 2022 through February 2024, the Collaborative ASNs collectively identified 5,165 patients from the registry as needing outreach. Among patients needing outreach, 3,555 (68.8%) were successfully contacted, 2,060 (39.9%) were scheduled for a colonoscopy, and 1,504 (29.1%) completed their colonoscopy.

CONCLUSION

The Collaborative successfully identified patients with previously abnormal CRC screening and facilitated completion of follow-up testing. The CRC ASN Implementation Guide offers a comprehensive road map for health care leaders interested in implementing CRC ASNs.

Medical Home Implementation and Follow-Up of Cancer-Related Abnormal Test Results in the Veterans Health Administration

Importance

Lack of timely follow-up of cancer-related abnormal test results can lead to delayed or missed diagnoses, adverse cancer outcomes, and substantial cost burden for patients. Care delivery models, such as the Veterans Affairs' (VA) Patient-Aligned Care Team (PACT), which aim to improve patient-centered care coordination, could potentially also improve timely follow-up of abnormal test results. PACT was implemented nationally in the VA between 2010 and 2012.

Objective

To evaluate the long-term association between PACT implementation and timely follow-up of abnormal test results related to the diagnosis of 5 different cancers.

Design, Setting, and Participants

This multiyear retrospective cohort study used 14 years of VA data (2006-2019), which were analyzed using panel data-based random-effects linear regressions. The setting included all VA clinics and facilities. The participants were adult patients who underwent diagnostic testing related to 5 different cancers and had abnormal test results. Data extraction and statistical analyses were performed from September 2021 to December 2023.

Exposure

Calendar years denoting preperiods and postperiods of PACT implementation, and the PACT Implementation Progress Index Score denoting the extent of implementation in each VA clinic and facility.

Main Outcome and Measure

Percentage of potentially missed timely follow-ups of abnormal test results.

Results

This study analyzed 6 data sets representing 5 different types of cancers. During the initial years of PACT implementation (2010 to 2013), percentage of potentially missed timely follow-ups decreased between 3 to 7 percentage points for urinalysis suggestive of bladder cancer, 12 to 14 percentage points for mammograms suggestive of breast cancer, 19 to 22 percentage points for fecal tests suggestive of colorectal cancer, and 6 to 13 percentage points for iron deficiency anemia laboratory tests suggestive of colorectal cancer, with no statistically significant changes for α-fetoprotien tests and lung cancer imaging. However, these beneficial reductions were not sustained over time. Better PACT implementation scores were associated with a decrease in potentially missed timely follow-up percentages for urinalysis (0.3-percentage point reduction [95% CI, -0.6 to -0.1] with 1-point increase in the score), and laboratory tests suggestive of iron deficiency anemia (0.5-percentage point reduction [95% CI,-0.8 to -0.2] with 1-point increase in the score).

Conclusions and Relevance

This cohort study found that implementation of PACT in the VA was associated with a potential short-term improvement in the quality of follow-up for certain test results. Additional multifaceted sustained interventions to reduce missed test results are required to prevent care delays.

Developing electronic clinical quality measures to assess the cancer diagnostic process

OBJECTIVE

Measures of diagnostic performance in cancer are underdeveloped. Electronic clinical quality measures (eCQMs) to assess quality of cancer diagnosis could help quantify and improve diagnostic performance.

MATERIALS AND METHODS

We developed 2 eCQMs to assess diagnostic evaluation of red-flag clinical findings for colorectal (CRC; based on abnormal stool-based cancer screening tests or labs suggestive of iron deficiency anemia) and lung (abnormal chest imaging) cancer. The 2 eCQMs quantified rates of red-flag follow-up in CRC and lung cancer using electronic health record data repositories at 2 large healthcare systems. Each measure used clinical data to identify abnormal results, evidence of appropriate follow-up, and exclusions that signified follow-up was unnecessary. Clinicians reviewed 100 positive and 20 negative randomly selected records for each eCQM at each site to validate accuracy and categorized missed opportunities related to system, provider, or patient factors.

RESULTS

We implemented the CRC eCQM at both sites, while the lung cancer eCQM was only implemented at the VA due to lack of structured data indicating level of cancer suspicion on most chest imaging results at Geisinger. For the CRC eCQM, the rate of appropriate follow-up was 36.0% (26 746/74 314 patients) in the VA after removing clinical exclusions and 41.1% at Geisinger (1009/2461 patients; P < .001). Similarly, the rate of appropriate evaluation for lung cancer in the VA was 61.5% (25 166/40 924 patients). Reviewers most frequently attributed missed opportunities at both sites to provider factors (84 of 157).

CONCLUSIONS

We implemented 2 eCQMs to evaluate the diagnostic process in cancer at 2 large health systems. Health care organizations can use these eCQMs to monitor diagnostic performance related to cancer.

Cancer Evaluations During the COVID-19 Pandemic: An Observational Study Using National Veterans Affairs Data

INTRODUCTION

Fewer cancer diagnoses have been made during the COVID-19 pandemic. Pandemic-related delays in cancer diagnosis could occur from limited access to care or patient evaluation delays (e.g., delayed testing after abnormal results). Follow-up of abnormal test results warranting evaluation for cancer was examined before and during the pandemic.

METHODS

Electronic trigger algorithms were applied to the Department of Veterans Affairs electronic health record data to assess follow-up of abnormal test results before (March 10, 2019-March 7, 2020) and during (March 8, 2020-March 6, 2021) the pandemic.

RESULTS

Electronic triggers were applied to 8,021,406 veterans' electronic health records to identify follow-up delays for abnormal results warranting evaluation for 5 cancers: bladder (urinalysis with high-grade hematuria), breast (abnormal mammograms), colorectal (positive fecal occult blood tests/fecal immunochemical tests or results consistent with iron deficiency anemia), liver (elevated alpha-fetoprotein), and lung (chest imaging suggestive of malignancy) cancers. Between prepandemic and pandemic periods, test quantities decreased by 12.6%-27.8%, and proportions of abnormal results lacking follow-up decreased for urinalyses (-0.8%), increased for fecal occult blood tests/fecal immunochemical test (+2.3%) and chest imaging (+1.8%), and remained constant for others. Follow-up times decreased for most tests; however, control charts suggested increased delays at 2 stages: early (pandemic beginning) for urinalyses, mammograms, fecal occult blood tests/fecal immunochemical test, iron deficiency anemia, and chest imaging and late (30-45 weeks into pandemic) for mammograms, fecal occult blood tests/fecal immunochemical test, and iron deficiency anemia.

CONCLUSIONS

Although early pandemic delays in follow-up may have led to reduced cancer rates, the significant decrease in tests performed is likely a large driver of these reductions. Future emergency preparedness efforts should bolster essential follow-up and testing procedures to facilitate timely cancer diagnosis.

Validation of an electronic trigger to measure missed diagnosis of stroke in emergency departments

OBJECTIVE

Diagnostic errors are major contributors to preventable patient harm. We validated the use of an electronic health record (EHR)-based trigger (e-trigger) to measure missed opportunities in stroke diagnosis in emergency departments (EDs).

METHODS

Using two frameworks, the Safer Dx Trigger Tools Framework and the Symptom-disease Pair Analysis of Diagnostic Error Framework, we applied a symptom-disease pair-based e-trigger to identify patients hospitalized for stroke who, in the preceding 30 days, were discharged from the ED with benign headache or dizziness diagnoses. The algorithm was applied to Veteran Affairs National Corporate Data Warehouse on patients seen between 1/1/2016 and 12/31/2017. Trained reviewers evaluated medical records for presence/absence of missed opportunities in stroke diagnosis and stroke-related red-flags, risk factors, neurological examination, and clinical interventions. Reviewers also estimated quality of clinical documentation at the index ED visit.

RESULTS

We applied the e-trigger to 7,752,326 unique patients and identified 46,931 stroke-related admissions, of which 398 records were flagged as trigger-positive and reviewed. Of these, 124 had missed opportunities (positive predictive value for "missed" = 31.2%), 93 (23.4%) had no missed opportunity (non-missed), 162 (40.7%) were miscoded, and 19 (4.7%) were inconclusive. Reviewer agreement was high (87.3%, Cohen's kappa = 0.81). Compared to the non-missed group, the missed group had more stroke risk factors (mean 3.2 vs 2.6), red flags (mean 0.5 vs 0.2), and a higher rate of inadequate documentation (66.9% vs 28.0%).

CONCLUSION

In a large national EHR repository, a symptom-disease pair-based e-trigger identified missed diagnoses of stroke with a modest positive predictive value, underscoring the need for chart review validation procedures to identify diagnostic errors in large data sets.

Interventions targeted at reducing diagnostic error: systematic review

BACKGROUND

Incorrect, delayed and missed diagnoses can contribute to significant adverse health outcomes. Intervention options have proliferated in recent years necessitating an update to McDonald et al's 2013 systematic review of interventions to reduce diagnostic error.

OBJECTIVES

(1) To describe the types of published interventions for reducing diagnostic error that have been evaluated in terms of an objective patient outcome; (2) to assess the risk of bias in the included interventions and perform a sensitivity analysis of the findings; and (3) to determine the effectiveness of included interventions with respect to their intervention type.

METHODS

MEDLINE, CINAHL and the Cochrane Database of Systematic Reviews were searched from 1 January 2012 to 31 December 2019. Publications were included if they delivered patient-related outcomes relating to diagnostic accuracy, management outcomes and/or morbidity and mortality. The interventions in each included study were categorised and analysed using the six intervention types described by McDonald et al (technique, technology-based system interventions, educational interventions, personnel changes, structured process changes and additional review methods).

RESULTS

Twenty studies met the inclusion criteria. Eighteen of the 20 included studies (including three randomised controlled trials (RCTs)) demonstrated improvements in objective patient outcomes following the intervention. These three RCTs individually evaluated a technique-based intervention, a technology-based system intervention and a structured process change. The inclusion or exclusion of two higher risk of bias studies did not affect the results.

CONCLUSION

Technique-based interventions, technology-based system interventions and structured process changes have been the most studied interventions over the time period of this review and hence are seen to be effective in reducing diagnostic error. However, more high-quality RCTs are required, particularly evaluating educational interventions and personnel changes, to demonstrate the value of these interventions in diverse settings.

Electronic Health Record Closed-Loop Communication Program for Unexpected Nonemergent Findings

Background Reliance on examination reporting of unexpected imaging findings does not ensure receipt of findings or appropriate follow-up. A closed-loop communication system should include provider and patient notifications and be auditable through the electronic health record (EHR). Purpose To report the initial design of and results from using an EHR-integrated unexpected findings navigator (UFN) program that ensures closed-loop communication of unexpected nonemergent findings. Materials and Methods An EHR-integrated UFN program was designed to enable identification and communication of unexpected findings and aid in next steps in findings management. Three navigators (with prior training as radiologic technologists and sonographers) facilitated communication and documentation of results to providers and patients. Twelve months (October 2019 to October 2020) of results were retrospectively reviewed to evaluate patient demographics and program metrics. Descriptive statistics and correlation analysis were performed by using commercially available software. Results A total of 3542 examinations were reported within 12 months, representing 0.5% of all examinations performed (total of 749 649); the median patient age was 62 years (range, 1 day to 98 years; interquartile range, 23 years). Most patients were female (2029 of 3542 [57%]). Almost half of the examinations submitted were from chest radiography and CT (1618 of 3542 [46%]), followed by MRI and CT of the abdomen and pelvis (1123 of 3542 [32%]). The most common unexpected findings were potential neoplasms (391 of 3542 [11%]). The median time between examination performance and patient notification was 12 days (range, 0-136 days; interquartile range, 13 days). A total of 2127 additional imaging studies were performed, and 1078 patients were referred to primary care providers and specialists. Most radiologists (89%, 63 of 71 respondents) and providers (65%, 28 of 43 respondents) found the system useful and used it most frequently during regular business hours. Conclusion An electronic health record-integrated, navigator-facilitated, closed-loop communication program for unexpected radiologic findings led to near-complete success in notification of providers and patients and facilitated the next steps in findings management. © RSNA, 2021 See also the editorial by Safdar in this issue.

Factors Associated With Delay of Diagnosis of Hepatocellular Carcinoma in Patients With Cirrhosis

BACKGROUND & AIMS

We examined the frequency of and factors associated with delays in diagnosis of hepatocellular carcinoma (HCC) in a cohort of patients with cirrhosis in the Veterans Health Administration.

METHODS

In a retrospective study, we collected and analyzed data from the Veterans Health Administration's electronic health records. We used a multivariate logistic regression model to identify factors associated with a delay in diagnosis of HCC of more than 60 days following a red flag (defined as the earliest date at which a diagnosis of HCC could have been made, based on American Association for the Study of Liver Disease 2005 guidelines). We used multivariate Cox proportional hazards model to evaluate the effects of delayed diagnosis on survival, adjusting for patient and provider characteristics.

RESULTS

Among 655 patients with cirrhosis and a diagnosis of HCC from 2006 through 2011, 46.9% had a delay in diagnosis of more than 60 days following a red flag for HCC. Delays in diagnosis for more than 60 days were significantly associated with lack of provider adherence to the guidelines (adjusted odds ratio [OR], 4.82; 95% CI, 3.12-7.45), a diagnostic imaging evaluation instead of only measurement of alfa fetoprotein (adjusted OR, 2.63; 95% CI, 1.09-6.24), and diagnosis as an incidental finding during examination for an unrelated medical problem (compared with an HCC-related assessment) (adjusted OR, 2.26; 95% CI, 1.09-4.67). Diagnostic delays of 60 days or more were associated with lower mortality compared to patients without a delay in diagnosis (unadjusted hazard ratio, 0.57; 95% CI, 0.47-0.68 and adjusted hazard ratio, 0.63; 95% CI, 0.50-0.78).

CONCLUSIONS

Nearly half of veterans with cirrhosis have delays in diagnosis of HCC of 60 days or more after a red flag, defined by guidelines. Interventions are needed to improve timely follow-up of red flags for HCC and adherence to guidelines, to increase early detection of HCC.

[Respiratory Disease in the Era of Big Data]

One of the key elements of medicine in the second decade of the 21st century is the exponential growth of patient-produced information, due not only to the transition to the digitization of medical records, but also to the emergence of new sources of information and the capacity for analysis and interpretation of existing ones. The amount of medical information is expected to double every 2 years, which means that there will be 50 times more information available in 2020 than in 2011. In this setting, these large amounts of data or «big data» must be properly managed to implement new initiatives that improve the diagnosis, treatment, and prognosis of patients on the path to personalized medicine.The concept of personalization or precision medicine is of special interest in chronic respiratory disease. In recent years, research in entities such as asthma, COPD, cancer, or SAHS has focused on the identification of genomic, molecular, metabolic, and protein changes (biomarkers). Big data analysis tools can be used to move on from models based on the mean response to treatment, which are suboptimal for most patients, to focus on the individualized response. Part of this journey involves systems medicine, which also integrates clinical and population data to provide a multidimensional view of the disease and help identify causal associations that are usually only evident on big data analysis.

Assessment of Health Information Technology-Related Outpatient Diagnostic Delays in the US Veterans Affairs Health Care System: A Qualitative Study of Aggregated Root Cause Analysis Data

IMPORTANCE

Diagnostic delay in the outpatient setting is an emerging safety priority that health information technology (HIT) should help address. However, diagnostic delays have persisted, and new safety concerns associated with the use of HIT have emerged.

OBJECTIVE

To analyze HIT-related outpatient diagnostic delays within a large, integrated health care system.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study involved qualitative content analysis of safety concerns identified in aggregated root cause analysis (RCA) data related to HIT and outpatient diagnostic delays. The setting was the US Department of Veterans Affairs using all RCAs submitted to the Veterans Affairs (VA) National Center for Patient Safety from January 1, 2013, to July 31, 2018.

MAIN OUTCOMES AND MEASURES

Common themes associated with the role of HIT-related safety concerns were identified and categorized according to the Health IT Safety framework for measuring, monitoring, and improving HIT safety. This framework includes 3 related domains (ie, safe HIT, safe use of HIT, and using HIT to improve safety) situated within an 8-dimensional sociotechnical model accounting for interacting technical and nontechnical variables associated with safety. Hence, themes identified enhanced understanding of the sociotechnical context and domain of HIT safety involved.

RESULTS

Of 214 RCAs categorized by the terms delay and outpatient submitted during the study period, 88 were identified as involving diagnostic delays and HIT, from which 172 unique HIT-related safety concerns were extracted (mean [SD], 1.97 [1.53] per RCA). Most safety concerns (82.6% [142 of 172]) involved problems with safe use of HIT, predominantly sociotechnical factors associated with people, workflow and communication, and a poorly designed human-computer interface. Fewer safety concerns involved problems with safe HIT (14.5% [25 of 172]) or using HIT to improve safety (0.3% [5 of 172]). The following 5 key high-risk areas for diagnostic delays emerged: managing electronic health record inbox notifications and communication, clinicians gathering key diagnostic information, technical problems, data entry problems, and failure of a system to track test results.

CONCLUSIONS AND RELEVANCE

This qualitative study of a national RCA data set suggests that interventions to reduce outpatient diagnostic delays could aim to improve test result management, interoperability, data visualization, and order entry, as well as to decrease information overload.

Imaging activity possibly signalling missed diagnostic opportunities in bladder and kidney cancer: A longitudinal data-linkage study using primary care electronic health records

INTRODUCTION

Sub-optimal use or interpretation of imaging investigations prior to diagnosis of certain cancers may be associated with less timely diagnosis, but pre-diagnostic imaging activity for urological cancer is unknown.

METHOD

We analysed linked data derived from primary and secondary care records and cancer registration to evaluate the use of clinically relevant imaging tests pre-diagnosis, in patients with bladder and kidney cancer diagnosed in 2012-15 in England. As pre-diagnostic imaging activity increased from background rate 8 months pre-diagnosis, we used logistic regression to determine factors associated with first imaging test occurring 4-8 months pre-diagnosis, considering that such instances may reflect possible missed opportunities for expediting the diagnosis.

RESULTS

1963 patients with bladder or kidney cancer had at least one imaging test in the 8 months pre-diagnosis. 420 (21%) of patients had their first imaging test 4-8 months pre-diagnosis, that being ultrasound, CT and X-ray in 48%, 43% and 9% of those cases, respectively. Factors associated with greater risk of a first imaging test 4-8 months pre-diagnosis were kidney cancer, diagnosis at stages other than stage IV, first imaging having been an X-ray, test requested by GP and absence of haematuria before the imaging request.

CONCLUSION

About 1 in 5 patients with urological cancers receive relevant first imaging investigations 4-8 months prior to diagnosis, which may represent potential missed diagnostic opportunities for earlier diagnosis.

Finding Dental Harm to Patients through Electronic Health Record-Based Triggers

BACKGROUND

Patients may be inadvertently harmed while undergoing dental treatments. To improve care, we must first determine the types and frequency of harms that patients experience, but identifying cases of harm is not always straightforward for dental practices. Mining data from electronic health records is a promising means of efficiently detecting possible adverse events (AEs).

METHODS

We developed 7 electronic triggers (electronic health record based) to flag patient charts that contain distinct events common to AEs. These electronic charts were then manually reviewed to identify AEs.

RESULTS

Of the 1,885 charts reviewed, 16.2% contained an AE. The positive predictive value of the triggers ranged from a high of 0.23 for the 2 best-performing triggers (failed implants and postsurgical complications) to 0.09 for the lowest-performing triggers. The most common types of AEs found were pain (27.5%), hard tissue (14.8%), soft tissue (14.8%), and nerve injuries (13.3%). Most AEs were classified as temporary harm (89.2%). Permanent harm was present in 9.6% of the AEs, and 1.2% required transfer to an emergency room.

CONCLUSION

By developing these triggers and a process to identify harm, we can now start measuring AEs, which is the first step to mitigating harm in the future.

KNOWLEDGE TRANSFER STATEMENT

A retrospective review of patients' health records is a useful approach for systematically identifying and measuring harm. Rather than random chart reviews, electronic health record-based dental trigger tools are an effective approach for practices to identify patient harm. Measurement is one of the first steps in improving the safety and quality of care delivered.

Barriers and facilitators impacting reliability of the electronic health record-facilitated total testing process

OBJECTIVE

Despite ongoing efforts to improve reliability of the total testing process (TTP), breakdowns continue to occur resulting in diagnostic delays and suboptimal patient outcomes. We performed an exploratory study to identify factors that impact TTP reliability in electronic health record (EHR)-enabled care.

MATERIALS AND METHODS

We interviewed experts at three large EHR-enabled health care organizations and identified all TTP steps performed from clinician test ordering to result communication to patients. Findings from all sites were combined to develop a detailed process map of known TTP activities. We additionally asked experts about factors that positively or negatively impacted TTP reliability at each step. We describe the specific TTP steps identified and associated barriers and facilitators to TTP reliability.

RESULTS

We interviewed 39 experts involved in or overseeing the TTP. Most TTP activities identified were similar across sites, but we found significant differences with test order transmission to diagnostic services and relay of results back to clinicians and patients. Twenty-five unique barriers were identified related to technology and EHR usability issues, time and resource constraints, suboptimal clinic workflows, patient-related factors, information access limitations, and insufficient clinician training. Twenty-four unique facilitators were identified related to personnel training, workflow optimization and standardization, helpful EHR features, and improved electronic communication between clinics and diagnostic services.

DISCUSSION

Barriers related to EHR usability and with communication between clinicians and diagnostic services increase TTP vulnerability and should be targeted by future efforts to improve process reliability. Several facilitators identified in the study could inform future strategies and solutions to improve TTP reliability.

Application of electronic trigger tools to identify targets for improving diagnostic safety

Progress in reducing diagnostic errors remains slow partly due to poorly defined methods to identify errors, high-risk situations, and adverse events. Electronic trigger (e-trigger) tools, which mine vast amounts of patient data to identify signals indicative of a likely error or adverse event, offer a promising method to efficiently identify errors. The increasing amounts of longitudinal electronic data and maturing data warehousing techniques and infrastructure offer an unprecedented opportunity to implement new types of e-trigger tools that use algorithms to identify risks and events related to the diagnostic process. We present a knowledge discovery framework, the Safer Dx Trigger Tools Framework, that enables health systems to develop and implement e-trigger tools to identify and measure diagnostic errors using comprehensive electronic health record (EHR) data. Safer Dx e-trigger tools detect potential diagnostic events, allowing health systems to monitor event rates, study contributory factors and identify targets for improving diagnostic safety. In addition to promoting organisational learning, some e-triggers can monitor data prospectively and help identify patients at high-risk for a future adverse event, enabling clinicians, patients or safety personnel to take preventive actions proactively. Successful application of electronic algorithms requires health systems to invest in clinical informaticists, information technology professionals, patient safety professionals and clinicians, all of who work closely together to overcome development and implementation challenges. We outline key future research, including advances in natural language processing and machine learning, needed to improve effectiveness of e-triggers. Integrating diagnostic safety e-triggers in institutional patient safety strategies can accelerate progress in reducing preventable harm from diagnostic errors.

Understanding Test Results Follow-Up in the Ambulatory Setting: Analysis of Multiple Perspectives

BACKGROUND

Delayed or incomplete test result follow-up, which can lead to missed and/or delayed diagnosis, is an important issue in the ambulatory setting. Delayed test result follow-up has been linked to poorer patient outcomes and increased risk of mortality and accounts for a large portion of medical malpractice claims. Yet improvements are difficult, reflecting the complexity of the test result follow-up process. Test result follow-up safety culture was investigated using qualitative and quantitative patient safety and quality of care data at an academic medical center.

METHODS

After an environmental scan, five sources of data were used to compass multiple perspectives on safety culture-two national surveys (AHRQ MO SOPS for safety culture and CG-CAHPS for patient satisfaction); patient and family complaints; safety reports; and provider response times to test message results in the electronic health record.

RESULTS

The following metrics were inspected: how patients and providers estimated the frequency for providing timely test results; how patients' satisfaction with their provider correlated with their provider's response time to test result messages; and qualitative themes in patient complaints and safety reports filed by clinic. The institution was compared to national benchmarks using surveys. As test result response time decreased, patient satisfaction increased (p = 0.0073).

CONCLUSION

Test result follow-up culture was investigated using tools typically used to examine patient satisfaction and experience and staff culture. Use of these five sources of data led to an examination of multiple perspectives in follow-up culture and identification of possible explanations for inappropriate follow-up. These data sources can be further explored to identify possible solutions.

A Systematic Review on Healthcare Analytics: Application and Theoretical Perspective of Data Mining

The growing healthcare industry is generating a large volume of useful data on patient demographics, treatment plans, payment, and insurance coverage—attracting the attention of clinicians and scientists alike. In recent years, a number of peer-reviewed articles have addressed different dimensions of data mining application in healthcare. However, the lack of a comprehensive and systematic narrative motivated us to construct a literature review on this topic. In this paper, we present a review of the literature on healthcare analytics using data mining and big data. Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, we conducted a database search between 2005 and 2016. Critical elements of the selected studies—healthcare sub-areas, data mining techniques, types of analytics, data, and data sources—were extracted to provide a systematic view of development in this field and possible future directions. We found that the existing literature mostly examines analytics in clinical and administrative decision-making. Use of human-generated data is predominant considering the wide adoption of Electronic Medical Record in clinical care. However, analytics based on website and social media data has been increasing in recent years. Lack of prescriptive analytics in practice and integration of domain expert knowledge in the decision-making process emphasizes the necessity of future research.

Electronic Triggers to Identify Delays in Follow-Up of Mammography: Harnessing the Power of Big Data in Health Care

PURPOSE

We previously developed electronic triggers to automatically flag records for patients experiencing potential delays in diagnostic evaluation for certain cancers. Because of the unique clinical, logistic, and legal aspects of mammography, this study was conducted to evaluate the effectiveness of a trigger to flag delayed follow-up on mammography.

METHODS

An algorithm was developed to detect delays in follow-up of abnormal mammographic results (>60 days for BI-RADS^® 0, 4, and 5 and >7 months for BI-RADS 3) using clinical data in the electronic health record. Flagged records were then manually reviewed to determine the trigger's performance characteristics (positive and negative predictive value, sensitivity, and specificity). The frequency of delays and patient communication related to abnormal results, reasons for lack of follow-up, and whether patients were subsequently diagnosed with breast cancer were also assessed.

RESULTS

Of 365,686 patients seen between January 1, 2010, and May 31, 2015, the trigger identified 2,129 patients with abnormal findings on mammography, of whom it flagged 552 as having delays in follow-up. From these, review of 400 randomly selected records revealed 283 true delays (positive predictive value, 71%; 95% confidence interval, 66%-75%), including 280 records without any documented plan and three patients with plans that were not adhered to. Transcription and reporting inconsistencies were identified in 27% of externally performed mammographic reports. Only 335 records (84%) contained specific documentation that the patient was informed of the abnormal result.

CONCLUSIONS

Care delays appear to continue despite federal laws requiring patient notification of mammographic results within 30 days. Clinical application of mammography-related triggers could help detect these delays.

Development and Validation of Trigger Algorithms to Identify Delays in Diagnostic Evaluation of Gastroenterological Cancer

BACKGROUND & AIMS

Colorectal cancer (CRC) and hepatocellular cancer (HCC) are common causes of death and morbidity, and patients benefit from early detection. However, delays in follow-up of suspicious findings are common, and methods to efficiently detect such delays are needed. We developed, refined, and tested trigger algorithms that identify patients with delayed follow-up evaluation of findings suspicious of CRC or HCC.

METHODS

We developed and validated two trigger algorithms that detect delays in diagnostic evaluation of CRC and HCC using laboratory, diagnosis, procedure, and referral codes from the Department of Veteran Affairs National Corporate Data Warehouse. The algorithm initially identified patients with positive test results for iron deficiency anemia or fecal immunochemical test (for CRC) and elevated α-fetoprotein results (for HCC). Our algorithm then excluded patients for whom follow-up evaluation was unnecessary, such as patients with a terminal illness or those who had already completed a follow-up evaluation within 60 days. Clinicians reviewed samples of both delayed and nondelayed records, and review data were used to calculate trigger performance.

RESULTS

We applied the algorithm for CRC to 245,158 patients seen from January 1, 2013, through December 31, 2013 and identified 1073 patients with delayed follow up. In a review of 400 randomly selected records, we found that our algorithm identified patients with delayed follow-up with a positive predictive value of 56.0% (95% CI, 51.0%-61.0%). We applied the algorithm for HCC to 333,828 patients seen from January 1, 2011 through December 31, 2014, and identified 130 patients with delayed follow-up. During manual review of all 130 records, we found that our algorithm identified patients with delayed follow-up with a positive predictive value of 82.3% (95% CI, 74.4%-88.2%). When we extrapolated the findings to all patients with abnormal results, the algorithm identified patients with delayed follow-up evaluation for CRC with 68.6% sensitivity (95% CI, 65.4%-71.6%) and 81.1% specificity (95% CI, 79.5%-82.6%); it identified patients with delayed follow-up evaluation for HCC with 89.1% sensitivity (95% CI, 81.8%-93.8%) and 96.5% specificity (95% CI, 94.8%-97.7%). Compared to nonselective methods, use of the algorithm reduced the number of records required for review to identify a delay by more than 99%.

CONCLUSIONS

Using data from the Veterans Affairs electronic health record database, we developed an algorithm that greatly reduces the number of record reviews necessary to identify delays in follow-up evaluations for patients with suspected CRC or HCC. This approach offers a more efficient method to identify delayed diagnostic evaluation of gastroenterological cancers.

An electronic trigger based on care escalation to identify preventable adverse events in hospitalised patients

Background

Methods to identify preventable adverse events typically have low yield and efficiency. We refined the methods of Institute of Healthcare Improvement’s Global Trigger Tool (GTT) application and leveraged electronic health record (EHR) data to improve detection of preventable adverse events, including diagnostic errors.

Methods

We queried the EHR data repository of a large health system to identify an ‘index hospitalization’ associated with care escalation (defined as transfer to the intensive care unit (ICU) or initiation of rapid response team (RRT) within 15 days of admission) between March 2010 and August 2015. To enrich the record review sample with unexpected events, we used EHR clinical data to modify the GTT algorithm and limited eligible patients to those at lower risk for care escalation based on younger age and presence of minimal comorbid conditions. We modified the GTT review methodology; two physicians independently reviewed eligible ‘e-trigger’ positive records to identify preventable diagnostic and care management events.

Results

Of 88 428 hospitalisations, 887 were associated with care escalation (712 ICU transfers and 175 RRTs), of which 92 were flagged as trigger-positive and reviewed. Preventable adverse events were detected in 41 cases, yielding a trigger positive predictive value of 44.6% (reviewer agreement 79.35%; Cohen’s kappa 0.573). We identified 7 (7.6%) diagnostic errors and 34 (37.0%) care management-related events: 24 (26.1%) adverse drug events, 4 (4.3%) patient falls, 4 (4.3%) procedure-related complications and 2 (2.2%) hospital-associated infections. In most events (73.1%), there was potential for temporary harm.

Conclusion

We developed an approach using an EHR data-based trigger and modified review process to efficiently identify hospitalised patients with preventable adverse events, including diagnostic errors. Such e-triggers can help overcome limitations of currently available methods to detect preventable harm in hospitalised patients.

Errors in Diagnosis of Spinal Epidural Abscesses in the Era of Electronic Health Records

PURPOSE

With this study, we set out to identify missed opportunities in diagnosis of spinal epidural abscesses to outline areas for process improvement.

METHODS

Using a large national clinical data repository, we identified all patients with a new diagnosis of spinal epidural abscess in the Department of Veterans Affairs (VA) during 2013. Two physicians independently conducted retrospective chart reviews on 250 randomly selected patients and evaluated their records for red flags (eg, unexplained weight loss, neurological deficits, and fever) 90 days prior to diagnosis. Diagnostic errors were defined as missed opportunities to evaluate red flags in a timely or appropriate manner. Reviewers gathered information about process breakdowns related to patient factors, the patient-provider encounter, test performance and interpretation, test follow-up and tracking, and the referral process. Reviewers also determined harm and time lag between red flags and definitive diagnoses.

RESULTS

Of 250 patients, 119 had a new diagnosis of spinal epidural abscess, 66 (55.5%) of which experienced diagnostic error. Median time to diagnosis in error cases was 12 days, compared with 4 days in cases without error (P <.01). Red flags that were frequently not evaluated in error cases included unexplained fever (n = 57; 86.4%), focal neurological deficits with progressive or disabling symptoms (n = 54; 81.8%), and active infection (n = 54; 81.8%). Most errors involved breakdowns during the patient-provider encounter (n = 60; 90.1%), including failures in information gathering/integration, and were associated with temporary harm (n = 43; 65.2%).

CONCLUSION

Despite wide availability of clinical data, errors in diagnosis of spinal epidural abscesses are common and involve inadequate history, physical examination, and test ordering. Solutions should include renewed attention to basic clinical skills.

Electronic Detection of Delayed Test Result Follow-Up in Patients with Hypothyroidism

BACKGROUND

Delays in following up abnormal test results are a common problem in outpatient settings. Surveillance systems that use trigger tools to identify delayed follow-up can help reduce missed opportunities in care.

OBJECTIVE

To develop and test an electronic health record (EHR)-based trigger algorithm to identify instances of delayed follow-up of abnormal thyroid-stimulating hormone (TSH) results in patients being treated for hypothyroidism.

DESIGN

We developed an algorithm using structured EHR data to identify patients with hypothyroidism who had delayed follow-up (>60 days) after an abnormal TSH. We then retrospectively applied the algorithm to a large EHR data warehouse within the Department of Veterans Affairs (VA), on patient records from two large VA networks for the period from January 1, 2011, to December 31, 2011. Identified records were reviewed to confirm the presence of delays in follow-up.

KEY RESULTS

During the study period, 645,555 patients were seen in the outpatient setting within the two networks. Of 293,554 patients with at least one TSH test result, the trigger identified 1250 patients on treatment for hypothyroidism with elevated TSH. Of these patients, 271 were flagged as potentially having delayed follow-up of their test result. Chart reviews confirmed delays in 163 of the 271 flagged patients (PPV = 60.1%).

CONCLUSIONS

An automated trigger algorithm applied to records in a large EHR data warehouse identified patients with hypothyroidism with potential delays in thyroid function test results follow-up. Future prospective application of the TSH trigger algorithm can be used by clinical teams as a surveillance and quality improvement technique to monitor and improve follow-up.

Implementation science for ambulatory care safety: a novel method to develop context-sensitive interventions to reduce quality gaps in monitoring high-risk patients

BACKGROUND

Missed evidence-based monitoring in high-risk conditions (e.g., cancer) leads to delayed diagnosis. Current technological solutions fail to close this safety gap. In response, we aim to demonstrate a novel method to identify common vulnerabilities across clinics and generate attributes for context-flexible population-level monitoring solutions for widespread implementation to improve quality.

METHODS

Based on interviews with staff in otolaryngology, pulmonary, urology, breast, and gastroenterology clinics at a large urban publicly funded health system, we applied journey mapping to co-develop a visual representation of how patients are monitored for high-risk conditions. Using a National Academies framework and context-sensitivity theory, we identified common systems vulnerabilities and developed preliminary concepts for improving the robustness for monitoring patients with high-risk conditions ("design seeds" for potential solutions). Finally, we conducted a face validity and prioritization assessment of the design seeds with the original interviewees.

RESULTS

We identified five high-risk situations for potentially consequential diagnostic delays arising from suboptimal patient monitoring. All situations related to detection of cancer (head and neck, lung, prostate, breast, and colorectal). With clinic participants we created 5 journey maps, each representing specialty clinic workflow directed at evidence-based monitoring. System vulnerabilities common to the different clinics included challenges with: data systems, communications handoffs, population-level tracking, and patient activities. Clinic staff ranked 13 design seeds (e.g., keep patient list up to date, use triggered notifications) addressing these vulnerabilities. Each design seed has unique evaluation criteria for the usefulness of potential solutions developed from the seed.

CONCLUSIONS

We identified and ranked 13 design seeds that characterize situations that clinicians described 'wake them up at night', and thus could reduce their anxiety, save time, and improve monitoring of high-risk patients. We anticipate that the design seed approach promotes robust and context-sensitive solutions to safety and quality problems because it provides a human-centered link between the experienced problem and various solutions that can be tested for viability. The study also demonstrates a novel integration of industrial and human factors methods (journey mapping, process tracing and design seeds) linked to implementation theory for use in designing interventions that anticipate and reduce implementation challenges.

Effect of an Automated Tracking Registry on the Rate of Tracking Failure in Incidental Pulmonary Nodules

OBJECTIVE

Following incidental lung nodules with interval CT scanning is an accepted method to detect early lung cancer, but delayed tracking or failure to track is reported in up to 40% of patients.

METHODS

Our institution developed and implemented an automated lung nodule registry tracking system. This system uses a code at the time that a suspicious nodule is discovered to populate the registry. Suspicious nodules were defined as any nodule, solid or ground glass, <3 cm that the radiologist recorded as a potential malignancy or recommended for follow-up imaging. We exported the system to eight other Veterans Administration Medical Centers (VAMCs) with over 10,000 patients enrolled. We retrospectively reviewed 200 sequential CT scan reports containing incidental nodule(s) from two tertiary care university-affiliated VAMCs, both before and after the implementation of the registry tracking system. The primary outcome was the rate of tracking failure, defined as suspicious nodules that had no follow-up imaging or whose follow-up was delayed when compared with published guidelines. Secondary outcomes were predictors of tracking failure and reasons for tracking failure.

RESULTS

After implementation of the registry tracking system in the two VAMCs, we found a significant decrease in tracking failure, from a preimplementation rate of 74% to a postimplementation rate of 10% (P < .001). We found that age, nodule size, number, and nodule characteristics were significant predictors.

CONCLUSIONS

The automated lung nodule registry tracking system can be exported to other health care facilities and significantly reduces the rate of tracking failure.

Application of Electronic Algorithms to Improve Diagnostic Evaluation for Bladder Cancer

BACKGROUND

Strategies to ensure timely diagnostic evaluation of hematuria are needed to reduce delays in bladder cancer diagnosis.

OBJECTIVE

To evaluate the performance of electronic trigger algorithms to detect delays in hematuria follow-up.

METHODS

We developed a computerized trigger to detect delayed follow-up action on a urinalysis result with high-grade hematuria (>50 red blood cells/high powered field). The trigger scanned clinical data within a Department of Veterans Affairs (VA) national data repository to identify all patient records with hematuria, then excluded those where follow-up was unnecessary (e.g., terminal illness) or where typical follow-up action was detected (e.g., cystoscopy). We manually reviewed a randomly-selected sample of flagged records to confirm delays. We performed a similar analysis of records with hematuria that were marked as not delayed (non-triggered). We used review findings to calculate trigger performance.

RESULTS

Of 310,331 patients seen between 1/1/2012-12/31/2014, the trigger identified 5,857 patients who experienced high-grade hematuria, of which 495 experienced a delay. On manual review of 400 randomly-selected triggered records and 100 non-triggered records, the trigger achieved positive and negative predictive values of 58% and 97%, respectively.

CONCLUSIONS

Triggers offer a promising method to detect delays in care of patients with high-grade hematuria and warrant further evaluation in clinical practice as a means to reduce delays in bladder cancer diagnosis.

Challenges in patient safety improvement research in the era of electronic health records

Electronic health record (EHR) data repositories contain large volumes of aggregated, longitudinal clinical data that could allow patient safety researchers to identify important safety issues and conduct comprehensive evaluations of health care delivery outcomes. However, few health systems have successfully converted this abundance of data into useful information or knowledge for safety improvement. In this paper, we use a case study involving a project on missed/delayed follow-up of test results to discuss real-world challenges in using EHR data for patient safety research. We identify three types of challenges that pose as barriers to advance patient safety improvement research: 1) gaining approval to access/review EHR data; 2) interpreting EHR data; 3) working with local IT/EHR personnel. We discuss the complexity of these challenges, all of which are unlikely to be unique to this project, and outline some key next steps that must be taken to support research that uses EHR data to improve safety. We recognize that all organizations face competing priorities between clinical operations and research. However, to leverage EHRs and their abundant data for patient safety improvement research, many current data access and security policies and procedures must be rewritten and standardized across health care organizations. These efforts are essential to help make EHRs and EHR data useful for progress in our journey to safer health care.