Learning from the challenges of undertaking an evaluation of a multi-partner housing support initiative delivered within a hospital setting

Objectives

We present learning from a mixed-methods evaluation of a housing support initiative for hospital inpatients.

Study design

A mixed-methods process evaluation.

Methods

A social housing provider delivered a housing support service in two hospitals (mental health unit and general hospital). Healthcare providers, the social housing provider and academic researchers designed and undertook a co-produced, mixed-methods process evaluation of the intervention. The evaluation included questionnaires, semi-structured interviews, analysis of routinely collected data and economic analysis. Despite commitment from the partners, the evaluation faced challenges. We reflect on the lessons learnt within our discussion paper.

Results

Despite the commitment of the partners, we faced several challenges.

We took an iterative approach to the design and processes of the evaluation to respond to arising challenges. Recruitment of service-users was more difficult than anticipated, requiring additional staff resources. Given the small-scale nature of the intervention, and the quality of data recorded in hospital records, the planned economic analysis was not feasible. Positive factors facilitating evaluation included involvement of staff delivering the intervention, as well as managers. Being able to offer payment to partner organisations for staff time also facilitated ongoing engagement.

Conclusions

Multi-partner evaluations are useful, however, researchers and partners need to be prepared to take an iterative, resource intensive approach. Both availability and quality of routine data, and the resources required to support data collection, may limit feasibility of specific methods when evaluating small-scale cross-sector initiatives. Thus, this necessitates a flexible approach to design and analysis.

SPICED-ACS: Study of the potential impact of a computer-generated ECG diagnostic algorithmic certainty index in STEMI diagnosis: Towards transparent AI

BACKGROUND

Computerised electrocardiogram (ECG) interpretation diagnostic algorithms have been developed to guide clinical decisions like with ST segment elevation myocardial infarction (STEMI) where time in decision making is critical. These computer-generated diagnoses have been proven to strongly influence the final ECG diagnosis by the clinician; often called automation bias. However, the computerised diagnosis may be inaccurate and could result in a wrong or delayed treatment harm to the patient. We hypothesise that an algorithmic certainty index alongside a computer-generated diagnosis might mitigate automation bias. The impact of reporting a certainty index on the final diagnosis is not known.

PURPOSE

To ascertain whether knowledge of the computer-generated ECG algorithmic certainty index influences operator diagnostic accuracy.

METHODOLOGY

Clinicians who regularly analyse ECGs such as cardiology or acute care doctors, cardiac nurses and ambulance staff were invited to complete an online anonymous survey between March and April 2019. The survey had 36 ECGs with a clinical vignette of a typical chest pain and which were either a STEMI, normal, or borderline (but do not fit the STEMI criteria) along with an artificially created certainty index that was either high, medium, low or none. Participants were asked whether the ECG showed a STEMI and their confidence in the diagnosis. The primary outcomes were whether a computer-generated certainty index influenced interpreter's diagnostic decisions and improved their diagnostic accuracy. Secondary outcomes were influence of certainty index between different types of clinicians and influence of certainty index on user's own-diagnostic confidence.

RESULTS

A total of 91 participants undertook the survey and submitted 3262 ECG interpretations of which 75% of ECG interpretations were correct. Presence of a certainty index significantly increased the odds ratio of a correct ECG interpretation (OR 1.063, 95% CI 1.022-1.106, p = 0.004) but there was no significant difference between correct certainty index and incorrect certainty index (OR 1.028, 95% CI 0.923-1.145, p = 0.615). There was a trend for low certainty index to increase odds ratio compared to no certainty index (OR 1.153, 95% CI 0.898-1.482, p = 0.264) but a high certainty index significantly decreased the odds ratio of a correct ECG interpretation (OR 0.492, 95% CI 0.391-0.619, p < 0.001). There was no impact of presence of a certainty index (p = 0.528) or correct certainty index (p = 0.812) on interpreters' confidence in their ECG interpretation.

CONCLUSIONS

Our results show that the presence of an ECG certainty index improves the users ECG interpretation accuracy. This effect is not seen with differing levels of confidence within a certainty index, with reduced ECG interpretation success with a high certainty index compared with a trend for increased success with a low certainty index. This suggests that a certainty index improves interpretation when there is an increased element of doubt, possibly forcing the ECG user to spend more time and effort analysing the ECG. Further research is needed looking at time spent analysing differing certainty indices with alternate ECG diagnoses.

Ontario child health study: social and school impairments in children aged 6 to 16 years

This study reports on informant-specific prevalence rates and the relationship with psychiatric disorder of social and school impairments in a community sample of children, 6 to 16 years of age, in Ontario. Prevalence rates ranged from less than 1% to over 18% depending on informant, and there was a low level of agreement between informants. Although impairments were strongly related to psychiatric disorder, there were many children with impairments and no psychiatric disorder or psychiatric disorder with no impairments. Implications of these results are discussed.

cDNA cloning of an intracellular form of the human interleukin 1 receptor antagonist associated with epithelium

A cDNA encoding a receptor antagonist of interleukin 1 (IL-1ra), secreted from human monocytes, has recently been isolated and sequenced [Eisenberg, S. P., Evans, R. J., Arend, W. P., Verderber, E., Brewer, M. T., Hannum, C. H. & Thompson, R. C. (1990) Nature (London) 343, 341-346]. We have identified another version of this IL-1ra, which is predominantly expressed in epithelial cells. This IL-1ra lacks a leader sequence and, thus, is probably intracellular. Both proteins are derived from the same gene through use of an alternative transcriptional start site and internal splice-acceptor site. Expression of intracellular IL-1ra cDNA in COS cells demonstrated that the intracellular product specifically inhibited exogenous interleukin 1-dependent responses. Keratinocytes were shown to contain significant amounts of nonsecreted IL-1ra protein. Constitutive expression of the intracellular IL-1ra may be an intracellular defensive mechanism in exposed epithelial cells and/or may serve to regulate autocrine interleukin 1-mediated pathways of differentiation.

Identification of three related human GRO genes encoding cytokine functions

The product of the human GRO gene is a cytokine with inflammatory and growth-regulatory properties; GRO is also called MGSA for melanoma growth-stimulatory activity. We have identified two additional genes, GRO beta and GRO gamma, that share 90% and 86% identity at the deduced amino acid level with the original GRO alpha isolate. One amino acid substitution of proline in GRO alpha by leucine in GRO beta and GRO gamma leads to a large predicted change in protein conformation. Significant differences also exist in the 3' untranslated region, including different numbers of ATTTA repeats associated with mRNA instability. A 122-base-pair region in the 3' region is conserved among the three GRO genes, and a part of it is also conserved in the Chinese hamster genome, suggesting a role in regulation. DNA hybridization with oligonucleotide probes and partial sequence analysis of the genomic clones confirm that the three forms are derived from related but different genes. Only one chromosomal locus has been identified, at 4q21, by using a GRO alpha cDNA clone that hybridized to all three genes. Expression studies reveal tissue-specific regulation as well as regulation by specific inducing agents, including interleukin 1, tumor necrosis factor, phorbol 12-myristate 13-acetate, and lipopolysaccharide.