The reliability of character recognition: An Australian & New Zealand expert-novice comparison study in the interpretation of chemically recovered serial numbers

The chemical recovery of a defaced serial number is a common forensic science practice, however it is not understood how proficient experts perform in correctly identifying recovered serial numbers. Understanding the accuracy of experts and how they compare to novices in character recognition can help to establish a baseline for this expertise. In this study an expert-novice comparison assessment was completed involving 118 test plates, each stamped with six randomised alphanumeric characters. The plates were defaced and chemically recovered before being viewed by multiple participants over six time intervals. A total of 3169 character inspections were completed. An assessment of confidence and error rates were calculated for both expert (trained) and novice (untrained) participants. Errors were counted when a participant interpreted a different character to that of the ground truth and believed the result was accurate for reporting. The results showed a similar (2.3 % and 2.4 %) error rate for the cohorts, however a statistical difference in confidence levels was recorded, demonstrating the more conservative nature of experts. This study aims to assist in validating practitioner interpretations, through addressing some forensic science criticisms, such as establishing error rates to routine scientific practices.

The Management of Persistent Distal Occlusions after Mechanical Thrombectomy and Thrombolysis: An Inter- and Intrarater Agreement Study

BACKGROUND AND PURPOSE

The best management of patients with persistent distal occlusion after mechanical thrombectomy with or without IV thrombolysis remains unknown. We sought to evaluate the variability and agreement in decision-making for persistent distal occlusions.

MATERIALS AND METHODS

A portfolio of 60 cases was sent to clinicians with varying backgrounds and experience. Responders were asked whether they considered conservative management or rescue therapy (stent retriever, aspiration, or intra-arterial thrombolytics) a treatment option as well as their willingness to enroll patients in a randomized trial. Agreement was assessed using κ statistics.

RESULTS

The electronic survey was answered by 31 physicians (8 vascular neurologists and 23 interventional neuroradiologists). Decisions for rescue therapies were more frequent (n = 1116/1860, 60%) than for conservative management (n = 744/1860, 40%; P < .001). Interrater agreement regarding the final management decision was "slight" (κ = 0.12; 95% CI, 0.09-0.14) and did not improve when subgroups of clinicians were studied according to background, experience, and specialty or when cases were grouped according to the level of occlusion. On delayed re-questioning, 23 of 29 respondents (79.3%) disagreed with themselves on at least 20% of cases. Respondents were willing to offer trial participation in 1295 of 1860 (69.6%) cases.

CONCLUSIONS

Individuals did not agree regarding the best management of patients with persistent distal occlusion after mechanical thrombectomy and IV thrombolysis. There is sufficient uncertainty to justify a dedicated randomized trial.

Novices cannot fill the examiners' shoes: Evidence of footwear examiners' expertise in shoe comparisons

The value of a footwear examiner's opinion centres on their ability to determine whether a particular shoe made an impression with greater accuracy than a novice. However, there has been limited research on the expertise of footwear examiners and the accuracy and reproducibility of their decisions. In the current study, we measured the accuracy and consensus of 31 footwear examiners versus a comparison group of 29 novices. Participants completed 20 ground truth known mock shoe comparisons. Results demonstrated that footwear examiners were more accurate than novices, regardless of comparison difficulty. Overall, on trials where probative decisions were given, examiners made false identifications and false exclusions on a total of 3% and 2% of trials, while novices made false identifications and false exclusions on a total of 19% and 17% of trials. Examiners also demonstrated better consensus in their opinions than novices, although both groups demonstrated low levels of agreement in their responses and variability in their interpretation of the conclusion scale. In summary, these findings support the proposition that footwear examiners show expert-level performance in matching known and unknown footwear impressions. These performance estimates may help the criminal justice system to appropriately value footwear examination evidence.

Understanding crossovers and potential ways to mitigate the problem: Lessons from influential trials on lumbar microdiscectomy

BACKGROUND

Lumbar microdiscectomy is the most frequent surgical intervention used in the treatment of sciatica from herniated lumbar discs. Many discectomy trials have been plagued with an excessive number of crossovers that have rendered results inconclusive.

METHODS

We review the design and results of influential lumbar microdiscectomy trials. We also discuss the various strategies that have been used to decrease the number of crossovers or to mitigate the effects of crossovers on analyses.

RESULTS

Randomized trials on lumbar discectomy were affected by crossover rates of 8% to 42%. Various strategies that have been used to decrease that number or to mitigate the effects on results include: patient selection, blinding (placebo-controlled trials), an immediate access to surgery for the surgical group (but limited access to surgery for the conservative group), shortening the follow-up period necessary to reach the primary outcome measure, postponing crossovers to surgery after determination of the primary outcome, and modifying the primary outcome measure to include treatment failures. Crossovers should be anticipated and compensated for by increasing the number of participants.

CONCLUSION

Non-adherence to randomly allocated management options can deprive trials of the statistical power needed to inform clinical care. Crossovers and ways to mitigate related problems should be anticipated at the time of trial design.

Understanding how the research question impacts trial design: Examples from discectomy trials

BACKGROUND

Formulating a pertinent research question is of the utmost importance in clinical research. An ill-conceived question may lead to an erroneous trial design, which may adversely affect the care of patients and provide uninformative or even misleading results.

METHODS

We review the research question of a randomized trial on the timing of lumbar discectomy. We compare the resulting design with other trials, real or hypothetical, that would have been more appropriate.

RESULTS

The RCT we examine randomly allocated patients to early or delayed surgery to answer a theoretical question of the effect of time on the efficacy of surgery. The trial was interpreted to have shown that early surgery was associated with better clinical and functional outcomes as compared to delayed surgery. This conclusion is clinically misleading. Valid comparisons between groups should be performed on intent-to-treat analyses and at the same time points after randomization (and not at a fixed follow-up period after surgery). The clinically pertinent comparison is not between the theoretical efficacy of surgery performed at various times, but between surgery and conservative management in patients presenting at various times. Better-designed trials on the clinical benefits of lumbar discectomy, including the treatment of chronic sciatica, have been published.

CONCLUSION

Theoretical research questions inspired from observational data can lead to erroneous trial design. Prospective randomized trials impact practice immediately: they are unique occasions to address clinical problems and optimize care under uncertainty in real time. However, they require the research question to be formulated with great care.

A Pragmatic Randomized Trial Comparing Surgical Clipping and Endovascular Treatment of Unruptured Intracranial Aneurysms

BACKGROUND AND PURPOSE

Surgical clipping and endovascular treatment are commonly used in patients with unruptured intracranial aneurysms. We compared the safety and efficacy of the 2 treatments in a randomized trial.

MATERIALS AND METHODS

Clipping or endovascular treatments were randomly allocated to patients with one or more 3- to 25-mm unruptured intracranial aneurysms judged treatable both ways by participating physicians. The study hypothesized that clipping would decrease the incidence of treatment failure from 13% to 4%, a composite primary outcome defined as failure of aneurysm occlusion, intracranial hemorrhage during follow-up, or residual aneurysms at 1 year, as adjudicated by a core lab. Safety outcomes included new neurologic deficits following treatment, hospitalization of >5 days, and overall morbidity and mortality (mRS > 2) at 1 year. There was no blinding.

RESULTS

Two hundred ninety-one patients were enrolled from 2010 to 2020 in 7 centers. The 1-year primary outcome, ascertainable in 290/291 (99%) patients, was reached in 13/142 (9%; 95% CI, 5%-15%) patients allocated to surgery and in 28/148 (19%; 95% CI, 13%-26%) patients allocated to endovascular treatments (relative risk: 2.07; 95% CI, 1.12-3.83; P = .021). Morbidity and mortality (mRS >2) at 1 year occurred in 3/143 and 3/148 (2%; 95% CI, 1%-6%) patients allocated to surgery and endovascular treatments, respectively. Neurologic deficits (32/143, 22%; 95% CI, 16%-30% versus 19/148, 12%; 95% CI, 8%-19%; relative risk: 1.74; 95% CI, 1.04-2.92; P = .04) and hospitalizations beyond 5 days (69/143, 48%; 95% CI, 40%-56% versus 12/148, 8%; 95% CI, 5%-14%; relative risk: 0.18; 95% CI, 0.11-0.31; P < .001) were more frequent after surgery.

CONCLUSIONS

Surgical clipping is more effective than endovascular treatment of unruptured intracranial aneurysms in terms of the frequency of the primary outcome of treatment failure. Results were mainly driven by angiographic results at 1 year.

Stent-Assisted Coiling in the Treatment of Unruptured Intracranial Aneurysms: A Randomized Clinical Trial

BACKGROUND AND PURPOSE

Stent-assisted coiling may improve angiographic results of endovascular treatment of unruptured intracranial aneurysms compared with coiling alone, but this has never been shown in a randomized trial.

MATERIALS AND METHODS

The Stenting in the Treatment of Aneurysm Trial was an investigator-led, parallel, randomized (1:1) trial conducted in 4 university hospitals. Patients with intracranial aneurysms at risk of recurrence, defined as large aneurysms (≥10 mm), postcoiling recurrent aneurysms, or small aneurysms with a wide neck (≥4 mm), were randomly allocated to stent-assisted coiling or coiling alone. The composite primary efficacy outcome was "treatment failure," defined as initial failure to treat the aneurysm; aneurysm rupture or retreatment during follow-up; death or dependency (mRS > 2); or an angiographic residual aneurysm adjudicated by an independent core laboratory at 12 months. The primary hypothesis (revised for slow accrual) was that stent-assisted coiling would decrease treatment failures from 33% to 15%, requiring 200 patients. Primary analyses were intent to treat.

RESULTS

Of 205 patients recruited between 2011 and 2021, ninety-four were allocated to stent-assisted coiling and 111 to coiling alone. The primary outcome, ascertainable in 203 patients, was reached in 28/93 patients allocated to stent-assisted coiling (30.1%; 95% CI, 21.2%-40.6%) compared with 30/110 (27.3%; 95% CI, 19.4%-36.7%) allocated to coiling alone (relative risk = 1.10; 95% CI, 0.7-1.7; P  = .66). Poor clinical outcomes (mRS >2) occurred in 8/94 patients allocated to stent-assisted coiling (8.5%; 95% CI, 4.0%-16.6%) compared with 6/111 (5.4%; 95% CI, 2.2%-11.9%) allocated to coiling alone (relative risk = 1.6; 95% CI, 0.6%-4.4%; P  = .38).

CONCLUSIONS

The STAT trial did not show stent-assisted coiling to be superior to coiling alone for wide-neck, large, or recurrent unruptured aneurysms.

Understanding the difference between theory and practice: The extracranial-intracranial bypass trials in prevention of ischemic stroke

BACKGROUND

Patients with atherosclerotic carotid or middle cerebral artery occlusions suffer ischemic events that might theoretically be preventable with a surgical extracranial-intracranial bypass, but theory by itself does not justify surgical interventions.

METHODS

We review landmark randomized trials on EC-IC bypass surgery for the treatment of ischemic stroke in patients with atherosclerotic stenoses or occlusions.

RESULTS

The initial EC-IC bypass trial from 1985 did not show any clinical benefit from surgery. The carotid occlusion surgery study (COSS) performed more than 20 years later included only patients highly selected to potentially benefit from bypass by using modern perfusion studies. While EC-IC bypasses were successfully created and they did improve cerebral perfusion, the COSS study also failed to show any clinical benefit to the participating patients.

CONCLUSION

Neurosurgical interventions must not only work in theory; they must improve patient outcomes in real practice.

Research participants may not recall their participation but have a better understanding of alternative management options than patients in routine care

BACKGROUND

Patient understanding of care interventions, of the clinical uncertainty, and of their participation in clinical research is often poor. We hypothesized that compared to routine care, patients would better understand the prevailing uncertainty when they participated in research.

METHODS

A questionnaire was administered to patients at the time they attended a follow-up neurovascular clinic 4 to 52 weeks after a care episode where they did or did not participate in a clinical trial. Patients were not reminded whether they had previously participated in a clinical trial. Questions concerned their understanding of the risks/benefits of interventions, the availability of alternative options, whether their personal opinion was taken into consideration, the reason for the final decision, their confidence at having received the best management, and whether they had been research participants.

RESULTS

Between June 2019 and June 2020, 167 patients were recruited; 71 had truly been research participants, while 96 had not. A greater proportion of research patients were aware of the existence of management alternatives (65% versus 44%; P=0.008). Patients of both groups believed their personal opinion counted in the final decision (76% versus 70%), and patients were equally confident that they had received the best management (94%). Research patients believed they had participated in research 46% of the time, compared to 12% of routine care patients (P=0.003).

CONCLUSION

Many patients do not recall that they participated in a clinical trial, but they have a better understanding of the clinical uncertainty and of the availability of alternative management options.

Understanding how the primary endpoint impacts the interpretation of trial results: The Japanese Adult Moyamoya bypass trial

BACKGROUND

The meaning of a clinical trial depends to a large extent on the choice of the primary outcome measure, which can be explanatory or pragmatic.

METHODS

We review the Japanese Adult Moyamoya (JAM) trial, that compared surgical extracranial to intracranial (EC-IC) bypass and medical management of hemorrhagic moyamoya disease. We also review some principles which guide the selection of the primary trial endpoint.

DISCUSSION

The main component of the primary outcome measure in JAM was rebleeding, a surrogate outcome that allowed investigators to demonstrate that surgical bypass had causal efficacy. However, the number of patients with a poor outcome, defined as those with a modified Rankin score (mRS)>2, would have been a more pragmatic choice. Unfortunately, the trial was too small to show that patients benefited from surgery.

CONCLUSION

The JAM trial showed that EC-IC bypass can decrease rebleeding in moyamoya patients, but whether patients have better outcomes with surgery remains uncertain. Hard pragmatic clinical primary outcome measures are necessary to guide surgical care.

Study of "ALS reversals": LifeTime environmental exposures (StARLiTE)

We previously reported on a series of patients diagnosed with ALS whom had an extraordinary course defined by substantial and sustained improvement in weakness and function. For this study, twenty-five of these "ALS Reversals" completed extensive environmental exposure questionnaires. These responses were then compared to a large database of prior responses from patients with typically progressive ALS (n = 6187). The results demonstrated that the "Reversal" participants have had a diverse number of exposures with substantial heterogeneity. In general, this was similar to the control group; however, there were a few specific differences that could be further explored in future research.

Understanding explanatory and pragmatic trials: Examples from randomized controlled trials on vertebroplasty

OBJECTIVES

To better understand the explanatory-pragmatic distinction in the design and interpretation of randomized controlled trials (RCTs).

METHODS

We review the explanatory-pragmatic distinction in clinical trial design. We use the PRECIS-2 tool to evaluate the trial design of selected RCTs on percutaneous vertebroplasty for osteoporotic vertebral compression fractures. We discuss difficulties in the selection of criteria and in the construction of PRECIS diagrams. We also examine how inconsistency in the selection of various items of trial design can cause confusion in the interpretation of results.

RESULTS

The selection of criteria and the scoring of multiple PRECIS domains were subjective and thus debatable. The pragmascope patterns of various vertebroplasty trials were heterogeneous. Many trials had both pragmatic and explanatory components. Some placebo-controlled trial goals seem to have been explanatory, but their design actually included enough pragmatic items such that the meaning of negative trial results remains ambiguous.

CONCLUSION

The results of a trial cannot be interpreted without understanding the various design choices made along the explanatory-pragmatic spectrum.

Understanding power in randomized trials: The example of vertebroplasty

OBJECTIVES

Randomized trials (RCTs) should include a sufficient number of patients to reduce the risk that the observed outcome is a result of chance rather than from a truly different treatment effect. Trials must be even larger to claim an absence of treatment effect in a placebo-controlled trial. To estimate the size of the trial and maximize power, it is often suggested to use a comparison between the means of a continuous variable.

METHODS

We examine the RCTs that have compared vertebroplasty and placebo for patients with osteoporotic fractures. Most trials compared the means of a continuous pain score to yield implausibly small trials, as small as 24 patients per group.

RESULTS

The minimally significant difference between groups has no precise clinical meaning for patients when it is based on a comparison of means of pain scores. A comparison of the proportions of patients reaching a per-patient outcome measure of treatment success is much more pertinent if the trial is to inform the care of future patients. The resulting trials will admittedly need to be larger, but they will be much less likely to fall prey to the 'evidence of absence' fallacy. Furthermore, trial size should also take into consideration harder clinical outcome measures, such as death and disability.

CONCLUSION

When the goal of a trial is to inform outcome-based medical care, comparing the proportions of patients reaching a clinically pertinent outcome is more appropriate than comparing the means of a continuous variable.

Understanding the research-care demarcation and why it must be revised

BACKGROUND

A clean-cut separation between research and care was artificially created at the time of the Belmont report more than 40 years ago. The demarcation was initially controversial but eventually was implemented for political reasons. We examine why it must be revised.

METHODS

We review historical research scandals as well as the theoretical basis for the Belmont demarcation. We then discuss consequences on medical practice and propose an alternative.

DISCUSSION

Most research scandals involved abusing human beings supposedly for the sake of science. Belmont commissioners were aware the research/care problem was double-headed. While research subjects should be protected from abuse in the research context, patients need to be protected from unvalidated medical and surgical interventions in the care context. For political reasons the Commission recommended the regulation of research but to leave medical practice untouched. Thus the Commission had to distinguish research from care. The notion of 'generalizable knowledge' was introduced to define and regulate research, but the inadvertent result was that by trying to protect research subjects, the regulation has not only failed to protect all other patients, but also encouraged the widespread practice of unvalidated interventions within the care context. The notion of validated care should be re-introduced into a proper analysis of the care-research demarcation, for care research is an integral ingredient of a good medical practice.

CONCLUSION

The research-care demarcation should be revised to leave room for the validated/unvalidated care distinction. Care research, essential to guide medical practice, should be facilitated at all levels.

Understanding the choice of control group: A systematic review of vertebroplasty trials for osteoporotic vertebral compression fractures

OBJECTIVES

To better understand the choice of the comparator intervention in the design of clinical trials and its impact on the meaning of results we review randomized trials on vertebroplasty.

METHODS

We conducted a systematic and narrative review of all randomized trials on vertebroplasty. Trials are categorized according to the comparator intervention (non-surgical management, placebo/sham vertebroplasty, and kyphoplasty).

RESULTS

All trials were too small to show a difference in objective clinical outcomes, and 20 of 23 RCTs used mean pain scores to compare interventions. Most trials comparing vertebroplasty with non-surgical management concluded that vertebroplasty was superior. Trials comparing kyphoplasty with vertebroplasty showed similar results for both interventions. However, 4 of 5 trials comparing vertebroplasty with placebo surgery failed to show a significant difference between groups.

CONCLUSION

The clinical results of an intervention cannot be interpreted without a comparison that involves a control group. The choice of comparator intervention can change the meaning of the trial. A large pragmatic trial, using hard clinical outcomes such as morbidity and mortality as a primary outcome measure, would be needed to assess the potential clinical benefits of vertebroplasty.