Challenges in outcome measurement: clinical research perspective

Barriers and Facilitators of Outcome Collection During Hand Surgery Outreach: A Quality Improvement Study

BACKGROUND

Surgical outreach trips to low- and middle-income countries have been increasing. Outcome collection on these trips, however, has been inconsistent and often incomplete. We conducted a qualitative study of surgeons, administrators, and patients to identify the barriers and facilitators to outcome collection on hand surgery outreach trips to Hospital 175 in Ho Chi Minh City, Vietnam.

METHODS

A purposive sample of surgeons, administrators, and patients from Hospital 175 were interviewed about their beliefs regarding outcome collection. We used a semi-structured interview guide based on the Theoretical Domains Framework to systematically explore barriers and facilitators. Interviews were recorded, transcribed, and analyzed using content analysis. Beliefs underlying similar responses were identified and aggregated to describe barriers and facilitators of outcome measurement.

RESULTS

Twelve surgeons and administrative staff (3 visiting and 9 local) and 5 patients were interviewed before saturation was achieved. All stakeholders believed outcome collection on hand surgery outreach trips is important. Barriers identified were primarily related to environmental context and resources (eg, cost of returning) and memory, attention, and decision process (eg, difficulty in remembering patient follow-up intervals). The most commonly identified barriers address the distance patients live from the hospital/clinic, the resources required for them to return, and the lack of an organized system to assist in follow-up.

CONCLUSIONS

Multiple barriers to outcome collection exist at Hospital 175 in Vietnam. Understanding these barriers informs context-specific implementation approaches to collect outcomes on hand surgery outreach trips, which may improve the safety and quality of care provided.

Implementing structured follow-up of neonatal and paediatric patients: an evaluation of three university hospital case studies using the functional resonance analysis method

Background

In complex critical neonatal and paediatric clinical practice, little is known about long-term patient outcomes and what follow-up care is most valuable for patients. Emma Children’s Hospital, Amsterdam UMC (Netherlands), implemented a follow-up programme called Follow Me for neonatal and paediatric patient groups, to gain more insight into long-term outcomes and to use such outcomes to implement a learning cycle for clinical practice, improve follow-up care and facilitate research. Three departments initiated re-engineering and change processes. Each introduced multidisciplinary approaches to long-term follow-up, including regular standardised check-ups for defined age groups, based on medical indicators, developmental progress, and psychosocial outcomes in patients and their families. This research evaluates the implementation of the three follow-up programmes, comparing predefined procedures (work-as-imagined) with how the programmes were implemented in practice (work-as-done).

Methods

This study was conducted in 2019–2020 in the outpatient settings of the neonatal intensive care, paediatric intensive care and paediatric surgery departments of Emma Children’s Hospital. It focused on the organisational structure of the follow-up care. The functional resonance analysis method (FRAM) was applied, using documentary analysis, semi-structured interviews, observations and feedback sessions.

Results

One work-as-imagined model and four work-as-done models were described. The results showed vast data collection on medical, developmental and psychosocial indicators in all work-as-done models; however, process indicators for programme effectiveness and performance were missing. In practice there was a diverse allocation of roles and responsibilities and their interrelations to create a multidisciplinary team; there was no one-size-fits-all across the different departments. Although control and feedback loops for long-term outcomes were specified with respect to the follow-up groups within the programmes, they were found to overlap and misalign with other internal and external long-term outcome monitoring practices.

Conclusion

Implementing structured long-term follow-up may provide insights for improving daily practice and follow-up care, with the precondition of standardised measurements. Lessons learned from practice are (1) to address fragmentation in data collection and storage, (2) to incorporate the diverse ways to create a multidisciplinary team in practice, and (3) to include timely actionable indicators on programme effectiveness and performance, alongside medical, developmental and psychosocial indicators.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-022-07537-x.

Quality of Hospital Electronic Health Record (EHR) Data Based on the International Consortium for Health Outcomes Measurement (ICHOM) in Heart Failure: Pilot Data Quality Assessment Study

Background

There is increasing recognition that health care providers need to focus attention, and be judged against, the impact they have on the health outcomes experienced by patients. The measurement of health outcomes as a routine part of clinical documentation is probably the only scalable way of collecting outcomes evidence, since secondary data collection is expensive and error-prone. However, there is uncertainty about whether routinely collected clinical data within electronic health record (EHR) systems includes the data most relevant to measuring and comparing outcomes and if those items are collected to a good enough data quality to be relied upon for outcomes assessment, since several studies have pointed out significant issues regarding EHR data availability and quality.

Objective

In this paper, we first describe a practical approach to data quality assessment of health outcomes, based on a literature review of existing frameworks for quality assessment of health data and multistakeholder consultation. Adopting this approach, we performed a pilot study on a subset of 21 International Consortium for Health Outcomes Measurement (ICHOM) outcomes data items from patients with congestive heart failure.

Methods

All available registries compatible with the diagnosis of heart failure within an EHR data repository of a general hospital (142,345 visits and 12,503 patients) were extracted and mapped to the ICHOM format. We focused our pilot assessment on 5 commonly used data quality dimensions: completeness, correctness, consistency, uniqueness, and temporal stability.

Results

We found high scores (>95%) for the consistency, completeness, and uniqueness dimensions. Temporal stability analyses showed some changes over time in the reported use of medication to treat heart failure, as well as in the recording of past medical conditions. Finally, the investigation of data correctness suggested several issues concerning the characterization of missing data values. Many of these issues appear to be introduced while mapping the IMASIS-2 relational database contents to the ICHOM format, as the latter requires a level of detail that is not explicitly available in the coded data of an EHR.

Conclusions

Overall, results of this pilot study revealed good data quality for the subset of heart failure outcomes collected at the Hospital del Mar. Nevertheless, some important data errors were identified that were caused by fundamentally different data collection practices in routine clinical care versus research, for which the ICHOM standard set was originally developed. To truly examine to what extent hospitals today are able to routinely collect the evidence of their success in achieving good health outcomes, future research would benefit from performing more extensive data quality assessments, including all data items from the ICHOM standards set and across multiple hospitals.

Implementing ICHOM standard set for cataract surgery at IPO-Porto (Portugal): clinical outcomes, quality of life and costs

BACKGROUND

This paper fills a gap in the applied research field, for a local context, by addressing the topics of describing cataract surgery' clinical outcomes; quality of life (QoL); and costs of the patients treated after the implementation of the ICHOM standard set.

METHODS

This is a retrospective observational study using real-world data (RWD). We included all patients subjected to cataract surgery at the Portuguese Institute of oncology - Porto (IPO-Porto), Portugal, after 3 months follow up period completed between 5th June 2017 and 21st May 2018. The following inclusion criteria: corrected visual acuity of ≤ 6/10 or other significant visual disturbance due to lens opacity or the existence of a large anisometropia. A circuit was implemented based on the ICHOM standard for cataract, to measure clinical variables (e.g. visual acuity) and QoL (CATQUEST-9SF) before and after surgery, and cost of treatment. The results were explored by means of a paired-sample t-test, considering normality assumptions.

RESULTS

Data refers to 268 patients (73 P25-P75:32-95 years old), regarding 374 eyes. The cataract surgery had a positive effect on visual acuity (p < 0.001), refraction (right and left cylinder; p < 0.001) and all QoL dimensions. The vast majority of patients, around 98%, reported improvements in QoL. Based on IPO-Porto administrative records, the direct cost of treating cataracts (per eye) is of 500€, representing a total cost of 187,000€ for the number of patients operated herein.

CONCLUSION

This study reports the successful implementation of the ICHOM standard set for cataracts in a Portuguese institution and confirms that cataract surgery provides a rapid visual recovery, with excellent visual outcomes and minimal complications in most patients, while also having a positive impact on patients' quality of life.

Nutritional interventions for treating foot ulcers in people with diabetes

BACKGROUND

Foot ulcers in people with diabetes are non-healing, or poorly healing, partial, or full-thickness wounds below the ankle. These ulcers are common, expensive to manage and cause significant morbidity and mortality. The presence of a wound has an impact on nutritional status because of the metabolic cost of repairing tissue damage, in addition to the nutrient losses via wound fluid. Nutritional interventions may improve wound healing of foot ulcers in people with diabetes.

OBJECTIVES

To evaluate the effects of nutritional interventions on the healing of foot ulcers in people with diabetes.

SEARCH METHODS

In March 2020 we searched the Cochrane Wounds Specialised Register; the Cochrane Central Register of Controlled Trials (CENTRAL); Ovid MEDLINE; Ovid Embase and EBSCO CINAHL Plus. We also searched clinical trials registries for ongoing and unpublished studies, and scanned reference lists of relevant included studies as well as reviews, meta-analyses and health technology reports to identify additional studies. There were no restrictions with respect to language, date of publication or study setting.

SELECTION CRITERIA

We included randomised controlled trials (RCTs) that evaluated the effect of nutritional interventions on the healing of foot ulcers in people with diabetes.

DATA COLLECTION AND ANALYSIS

Two review authors, working independently, assessed included RCTs for their risk of bias and rated the certainty of evidence using GRADE methodology, using pre-determined inclusion and quality criteria.

MAIN RESULTS

We identified nine RCTs (629 participants). Studies explored oral nutritional interventions as follows: a protein (20 g protein per 200 mL bottle), 1 kcal/mL ready-to-drink, nutritional supplement with added vitamins, minerals and trace elements; arginine, glutamine and β-hydroxy-β-methylbutyrate supplement; 220 mg zinc sulphate supplements; 250 mg magnesium oxide supplements; 1000 mg/day omega-3 fatty acid from flaxseed oil; 150,000 IU of vitamin D, versus 300,000 IU of vitamin D; 250 mg magnesium oxide plus 400 IU vitamin E and 50,000 IU vitamin D supplements. The comparator in eight studies was placebo, and in one study a different dose of vitamin D. Eight studies reported the primary outcome measure of ulcer healing; only two studies reported a measure of complete healing. Six further studies reported measures of change in ulcer dimension, these studies reported only individual parameters of ulcer dimensions (i.e. length, width and depth) and not change in ulcer volume. All of the evidence identified was very low certainty. We downgraded it for risks of bias, indirectness and imprecision. It is uncertain whether oral nutritional supplement with 20 g protein per 200 mL bottle, 1 kcal/mL, nutritional supplement with added vitamins, minerals and trace elements, increases the proportion of ulcers healed at six months more than placebo (risk ratio (RR) 0.80, 95% confidence interval (CI) 0.42 to 1.53). It is also uncertain whether arginine, glutamine and β-hydroxy-β-methylbutyrate supplement increases the proportion of ulcers healed at 16 weeks compared with placebo (RR 1.09, 95% CI 0.85 to 1.40). It is uncertain whether the following interventions change parameters of ulcer dimensions over time when compared with placebo; 220 mg zinc sulphate supplement containing 50 mg elemental zinc, 250 mg magnesium oxide supplement, 1000 mg/day omega-3 fatty acid from flaxseed oil supplement, magnesium and vitamin E co-supplementation and vitamin D supplementation. It is also uncertain whether 150,000 IU of vitamin D, impacts ulcer dimensions when compared with 300,000 IU of vitamin D. Two studies explored some of the secondary outcomes of interest for this review. It is uncertain whether oral nutritional supplement with 20 g protein per 200 mL bottle, 1 kcal/mL, nutritional supplement with added vitamins, minerals and trace elements, reduces the number of deaths (RR 0.96, 95% CI 0.06 to 14.60) or amputations (RR 4.82, 95% CI 0.24 to 95.88) more than placebo. It is uncertain whether arginine, glutamine and β-hydroxy-β-methylbutyrate supplement increases health-related quality of life at 16 weeks more than placebo (MD -0.03, 95% CI -0.09 to 0.03). It is also uncertain whether arginine, glutamine and β-hydroxy-β-methylbutyrate supplement reduces the numbers of new ulcers (RR 1.04, 95% CI 0.71 to 1.51), or amputations (RR 0.66, 95% CI 0.16 to 2.69) more than placebo. None of the included studies reported the secondary outcomes cost of intervention, acceptability of the intervention (or satisfaction) with respect to patient comfort, length of patient hospital stay, surgical interventions, or osteomyelitis incidence. One study exploring the impact of arginine, glutamine and β-hydroxy-β-methylbutyrate supplement versus placebo did not report on any relevant outcomes.

AUTHORS' CONCLUSIONS

Evidence for the impact of nutritional interventions on the healing of foot ulcers in people with diabetes compared with no nutritional supplementation, or compared with a different dose of nutritional supplementation, remains uncertain, with eight studies showing no clear benefit or harm. It is also uncertain whether there is a difference in rates of adverse events, amputation rate, development of new foot ulcers, or quality of life, between nutritional interventions and placebo. More research is needed to clarify the impact of nutritional interventions on the healing of foot ulcers in people with diabetes.

Big Data, Big Problems: Incorporating Mission, Values, and Culture in Provider Affiliations

This article explores how integration of data from clinical registries and electronic health records produces a quality impact within orthopedic practices. Data are differentiated from information, and several types of data that are collected and used in orthopedic outcome measurement are defined. Furthermore, the concept of comparative effectiveness and its impact on orthopedic clinical research are assessed. This article places emphasis on how the concept of big data produces health care challenges balanced with benefits that may be faced by patients and orthopedic surgeons. Finally, essential characteristics of an electronic health record that interlinks musculoskeletal care and big data initiatives are reviewed.

Factors that shape the patient's hospital experience and satisfaction with lower limb arthroplasty: an exploratory thematic analysis

OBJECTIVE

It is generally accepted that the patients' hospital experience can influence their overall satisfaction with the outcome of lower limb arthroplasty; however, little is known about the factors that shape the hospital experience. The aim of this study was to develop an understanding of what patients like and do not like about their hospital experience with a view to providing insight into where service improvements could have the potential to improve the patient experience and their satisfaction, and whether they would recommend the procedure.

DESIGN

A mixed methods (quan-QUAL) approach.

SETTING

Large regional teaching hospital.

PARTICIPANTS

216 patients who had completed a postoperative postal questionnaire at 12 months following total knee or total hip arthroplasty.

OUTCOME MEASURES

Overall satisfaction with the outcome of surgery, whether to recommend the procedure to another and the rating of patient hospital experience. Free text comments on the best and worst aspects of their hospital stay were evaluated using qualitative thematic analysis.

RESULTS

Overall, 77% of patients were satisfied with their surgery, 79% reported a good-excellent hospital experience and 85% would recommend the surgery to another. Qualitative analysis revealed clear themes relating to communication, pain relief and the process experience. Comments on positive aspects of the hospital experience were related to feeling well informed and consulted about their care. Comments on the worst aspects of care were related to being made to wait without explanation, moved to different wards and when they felt invisible to the healthcare staff caring for them.

CONCLUSIONS

Positive patient experiences were closely linked to effective patient-health professional interactions and logistics of the hospital processes. Within arthroplasty services, the patient experience of healthcare could be enhanced by further attention to concepts of patient-centred care. Practical examples of this include more focus on developing staff-patient communication and the avoidance of 'boarding' procedures.

Do Patient Demographics and Patient-Reported Outcomes Predict 12-Month Loss to Follow-Up After Spine Surgery?

STUDY DESIGN

Analysis of patients in a prospective registry.

OBJECTIVE

To determine the association between patient demographics, outcomes, and loss to follow-up 12 months after spine surgery.

SUMMARY OF BACKGROUND DATA

Obtaining outcomes 12 months after spine surgery remains a challenge. Loss to follow-up is believed to introduce biases and portend poor outcomes. Associations between follow-up, patient demographics, and outcomes in the degenerative spine population have not been studied.

METHODS

Patients undergoing surgery for degenerative spine disease at a single institution over a 2-year period were enrolled in a prospective registry. Patient demographics, comorbidities, treatment variables, readmissions/reoperations, and all 90-day surgical morbidity were collected. Patient-reported outcomes were recorded at baseline, 3-months, and 12-months after surgery. Multivariate logistic regression analysis was done to identify predictors of loss to follow-up.

RESULTS

A total of 1484 patients with baseline and 3-month outcomes were included. Two hundred thirty-three (15.7%) patients were lost to follow-up at 12 months. There was no difference in the baseline demographics (Sex: P = 0.46) and comorbidities (American Society of Anesthesiologists Grade: P = 0.06) of patients who had follow-up at 12-months versus those who did not, except age and employment status. Patients lost to follow-up at 12 months were younger (51.0 vs. 57.1 years; P < 0.001) and a higher proportion were employed preoperatively (45.9% vs. 41.7%, P = 0.24). Preoperative pain, disability, and quality of life was similar between the two groups (P > 0.05). There was no difference in 90-day morbidity (17.2% vs. 16.2%; P = 0.70) and 3-month pain, disability, quality of life, and patient satisfaction (85.0% vs. 88.3%; P = 0.63) (P > 0.05). In multivariate model, only younger age (P < 0.001) was an independent predictor of loss to follow-up at 12 months.

CONCLUSION

In our prospective spine registry the 12-month loss to follow-up rate is approximately 15%. The only independent predictor of loss to follow-up is younger age and preoperative employment.

LEVEL OF EVIDENCE

3.

Stakeholders in outcome measures: review from a clinical perspective

Modern interest in patient-reported outcomes measures (PROMs) in orthopaedics dates back to the mid-1980s. While gradual growth of activity in this area has occurred over the past 25 years, the extent to which this research methodology is applied in clinical practice to improve patient care is unclear. WHERE ARE WE NOW?: Historically, clinical research in orthopaedics has focused on the technical success of treatment, and objective indicators such as mortality, morbidity, and complications. By contrast, the PROMs framework focuses on effects of treatment described in terms of relief of symptoms, restoring functional ability, and improving quality of life. PROMs can be used to study the relative effects of disease, injury, and treatment across different health conditions. WHERE DO WE NEED TO GO?: All clinical research should begin with identifying clear and meaningful research questions so that the resources and efforts required for data collection result in useful data. Different consumers of research data have different perspectives on what comprises meaningful information. Involving stakeholders such as patients, providers, payers, and policy-makers when defining priorities in the larger research endeavor is one way to inform what type of data should be collected in a particular study. HOW DO WE GET THERE?: Widespread collection of outcomes data would potentially aid these stakeholders by identifying best practices, benefits and costs, and important patient or practice characteristics related to outcomes. Several initiatives currently underway may help systematic collection of PROMs, create efficient systems, and foster collaborations to provide support and resources to minimize costs.