Collective intelligence in medical decision-making: a systematic scoping review

Ten misconceptions regarding decision-making in critical care

Diagnostic errors are prevalent in critical care practice and are associated with patient harm and costs for providers and the healthcare system. Patient complexity, illness severity, and the urgency in initiating proper treatment all contribute to decision-making errors. Clinician-related factors such as fatigue, cognitive overload, and inexperience further interfere with effective decision-making. Cognitive science has provided insight into the clinical decision-making process that can be used to reduce error. This evidence-based review discusses ten common misconceptions regarding critical care decision-making. By understanding how practitioners make clinical decisions and examining how errors occur, strategies may be developed and implemented to decrease errors in Decision-making and improve patient outcomes.

Collective Intelligence Increases Diagnostic Accuracy in a General Practice Setting

BACKGROUND

General practitioners (GPs) work in an ill-defined environment where diagnostic errors are prevalent. Previous research indicates that aggregating independent diagnoses can improve diagnostic accuracy in a range of settings. We examined whether aggregating independent diagnoses can also improve diagnostic accuracy for GP decision making. In addition, we investigated the potential benefit of such an approach in combination with a decision support system (DSS).

METHODS

We simulated virtual groups using data sets from 2 previously published studies. In study 1, 260 GPs independently diagnosed 9 patient cases in a vignette-based study. In study 2, 30 GPs independently diagnosed 12 patient actors in a patient-facing study. In both data sets, GPs provided diagnoses in a control condition and/or DSS condition(s). Each GP's diagnosis, confidence rating, and years of experience were entered into a computer simulation. Virtual groups of varying sizes (range: 3-9) were created, and different collective intelligence rules (plurality, confidence, and seniority) were applied to determine each group's final diagnosis. Diagnostic accuracy was used as the performance measure.

RESULTS

Aggregating independent diagnoses by weighing them equally (i.e., the plurality rule) substantially outperformed average individual accuracy, and this effect increased with increasing group size. Selecting diagnoses based on confidence only led to marginal improvements, while selecting based on seniority reduced accuracy. Combining the plurality rule with a DSS further boosted performance.

DISCUSSION

Combining independent diagnoses may substantially improve a GP's diagnostic accuracy and subsequent patient outcomes. This approach did, however, not improve accuracy in all patient cases. Therefore, future work should focus on uncovering the conditions under which collective intelligence is most beneficial in general practice.

HIGHLIGHTS

We examined whether aggregating independent diagnoses of GPs can improve diagnostic accuracy.Using data sets of 2 previously published studies, we composed virtual groups of GPs and combined their independent diagnoses using 3 collective intelligence rules (plurality, confidence, and seniority).Aggregating independent diagnoses by weighing them equally substantially outperformed average individual GP accuracy, and this effect increased with increasing group size.Combining independent diagnoses may substantially improve GP's diagnostic accuracy and subsequent patient outcomes.

Crowdsourcing a diagnosis? Exploring the accuracy of the size and type of group diagnosis: an experimental study

BACKGROUND

The consultation process, where a clinician seeks an opinion from another clinician, is foundational in medicine. However, the effectiveness of group diagnosis has not been studied.

OBJECTIVE

To compare individual diagnosis to group diagnosis on two dimensions: group size (n=3 or 6) and group process (interactive or artificial groups).

METHODOLOGY

Thirty-six internal or emergency medicine residents participated in the study. Initially, each resident worked through four written cases on their own, providing a primary diagnosis and a differential diagnosis. Next, participants formed into groups of three. Using a videoconferencing platform, they worked through four additional cases, collectively providing a single primary diagnosis and differential diagnosis. The process was repeated using a group of six with four new cases. Cases were all counterbalanced. Retrospectively, nominal (ie, artificial) groups were formed by aggregating individual participant data into subgroups of three and six and analytically computing scores. Presence of the correct diagnosis as primary diagnosis or included in the differential diagnosis, as well as the number of diagnoses mentioned, was calculated for all conditions. Means were compared using analysis of variance.

RESULTS

For both authentic and nominal groups, the diagnostic accuracy of group diagnosis was superior to individual for both the primary diagnosis and differential diagnosis. However, there was no improvement in diagnostic accuracy when comparing a group of three to a group of six. Interactive and nominal groups were equivalent; however, this may be an artefact of the method used to combine data.

CONCLUSIONS

Group diagnosis improves diagnostic accuracy. However, a larger group is not necessarily superior to a smaller group. In this study, interactive group discussion does not result in improved diagnostic accuracy.

Collaborative clinical reasoning: a scoping review

Background

Collaborative clinical reasoning (CCR) among healthcare professionals is crucial for maximizing clinical outcomes and patient safety. This scoping review explores CCR to address the gap in understanding its definition, structure, and implications.

Methods

A scoping review was undertaken to examine CCR related studies in healthcare. Medline, PsychInfo, SciVerse Scopus, and Web of Science were searched. Inclusion criteria included full-text articles published between 2011 to 2020. Search terms included cooperative, collaborative, shared, team, collective, reasoning, problem solving, decision making, combined with clinical or medicine or medical, but excluded shared decision making.

Results

A total of 24 articles were identified in the review. The review reveals a growing interest in CCR, with 14 articles emphasizing the decision-making process, five using Multidisciplinary Team-Metric for the Observation of Decision Making (MDTs-MODe), three exploring CCR theory, and two focusing on the problem-solving process. Communication, trust, and team dynamics emerge as key influencers in healthcare decision-making. Notably, only two articles provide specific CCR definitions.

Conclusions

While decision-making processes dominate CCR studies, a notable gap exists in defining and structuring CCR. Explicit theoretical frameworks, such as those proposed by Blondon et al. and Kiesewetter et al., are crucial for advancing research and understanding CCR dynamics within collaborative teams. This scoping review provides a comprehensive overview of CCR research, revealing a growing interest and diversity in the field. The review emphasizes the need for explicit theoretical frameworks, citing Blondon et al. and Kiesewetter et al. The broader landscape of interprofessional collaboration and clinical reasoning requires exploration.

Effects of Combinational Use of Additional Differential Diagnostic Generators on the Diagnostic Accuracy of the Differential Diagnosis List Developed by an Artificial Intelligence-Driven Automated History-Taking System: Pilot Cross-Sectional Study

BACKGROUND

Low diagnostic accuracy is a major concern in automated medical history-taking systems with differential diagnosis (DDx) generators. Extending the concept of collective intelligence to the field of DDx generators such that the accuracy of judgment becomes higher when accepting an integrated diagnosis list from multiple people than when accepting a diagnosis list from a single person may be a possible solution.

OBJECTIVE

The purpose of this study is to assess whether the combined use of several DDx generators improves the diagnostic accuracy of DDx lists.

METHODS

We used medical history data and the top 10 DDx lists (index DDx lists) generated by an artificial intelligence (AI)-driven automated medical history-taking system from 103 patients with confirmed diagnoses. Two research physicians independently created the other top 10 DDx lists (second and third DDx lists) per case by imputing key information into the other 2 DDx generators based on the medical history generated by the automated medical history-taking system without reading the index lists generated by the automated medical history-taking system. We used the McNemar test to assess the improvement in diagnostic accuracy from the index DDx lists to the three types of combined DDx lists: (1) simply combining DDx lists from the index, second, and third lists; (2) creating a new top 10 DDx list using a 1/n weighting rule; and (3) creating new lists with only shared diagnoses among DDx lists from the index, second, and third lists. We treated the data generated by 2 research physicians from the same patient as independent cases. Therefore, the number of cases included in analyses in the case using 2 additional lists was 206 (103 cases × 2 physicians' input).

RESULTS

The diagnostic accuracy of the index lists was 46% (47/103). Diagnostic accuracy was improved by simply combining the other 2 DDx lists (133/206, 65%, P<.001), whereas the other 2 combined DDx lists did not improve the diagnostic accuracy of the DDx lists (106/206, 52%, P=.05 in the collective list with the 1/n weighting rule and 29/206, 14%, P<.001 in the only shared diagnoses among the 3 DDx lists).

CONCLUSIONS

Simply adding each of the top 10 DDx lists from additional DDx generators increased the diagnostic accuracy of the DDx list by approximately 20%, suggesting that the combinational use of DDx generators early in the diagnostic process is beneficial.

Experimental evidence for structured information-sharing networks reducing medical errors

Errors in clinical decision-making are disturbingly common. Recent studies have found that 10 to 15% of all clinical decisions regarding diagnoses and treatment are inaccurate. Here, we experimentally study the ability of structured information-sharing networks among clinicians to improve clinicians' diagnostic accuracy and treatment decisions. We use a pool of 2,941 practicing clinicians recruited from around the United States to conduct 84 independent group-level trials, ranging across seven different clinical vignettes for topics known to exhibit high rates of diagnostic or treatment error (e.g., acute cardiac events, geriatric care, low back pain, and diabetes-related cardiovascular illness prevention). We compare collective performance in structured information-sharing networks to collective performance in independent control groups, and find that networks significantly reduce clinical errors, and improve treatment recommendations, as compared to control groups of independent clinicians engaged in isolated reflection. Our results show that these improvements are not a result of simple regression to the group mean. Instead, we find that within structured information-sharing networks, the worst clinicians improved significantly while the best clinicians did not decrease in quality. These findings offer implications for the use of social network technologies to reduce errors among clinicians.

Harnessing the power of collective intelligence in dentistry: a pilot study in Victoria, Australia

BACKGROUND

In many dental settings, diagnosis and treatment planning is the responsibility of a single clinician, and this process is inevitably influenced by the clinician's own heuristics and biases. Our aim was to test whether collective intelligence increases the accuracy of individual diagnoses and treatment plans, and whether such systems have potential to improve patient outcomes in a dental setting.

METHODS

This pilot project was carried out to assess the feasibility of the protocol and appropriateness of the study design. We used a questionnaire survey and pre-post study design in which dental practitioners were involved in the diagnosis and treatment planning of two simulated cases. Participants were provided the opportunity to amend their original diagnosis/treatment decisions after viewing a consensus report made to simulate a collaborative setting.

RESULTS

Around half (55%, n = 17) of the respondents worked in group private practices, however most practitioners (74%, n = 23) did not collaborate when planning treatment. Overall, the average practitioners' self-confidence score in managing different dental disciplines was 7.22 (s.d. 2.20) on a 1-10 scale. Practitioners tended to change their mind after viewing the consensus response, particularly for the complex case compared to the simple case (61.5% vs 38.5%, respectively). Practitioners' confidence ratings were also significantly higher (p < 0.05) after viewing the consensus for complex case.

CONCLUSION

Our pilot study shows that collective intelligence in the form of peers' opinion can lead to modifications in diagnosis and treatment planning by dentists. Our results lay the foundations for larger scale investigations on whether peer collaboration can improve diagnostic accuracy, treatment planning and, ultimately, oral health outcomes.

Determinants of Clinical Decision Making under Uncertainty in Dentistry: A Scoping Review

Clinical decision-making for diagnosing and treating oral and dental diseases consolidates multiple sources of complex information, yet individual clinical judgements are often made intuitively on limited heuristics to simplify decision making, which may lead to errors harmful to patients. This study aimed at systematically evaluating dental practitioners’ clinical decision-making processes during diagnosis and treatment planning under uncertainty. A scoping review was chosen as the optimal study design due to the heterogeneity and complexity of the topic. Key terms and a search strategy were defined, and the articles published in the repository of the National Library of Medicine (MEDLINE/PubMed) were searched, selected, and analysed in accordance with PRISMA-ScR guidelines. Of the 478 studies returned, 64 relevant articles were included in the qualitative synthesis. Studies that were included were based in 27 countries, with the majority from the UK and USA. Articles were dated from 1991 to 2022, with all being observational studies except four, which were experimental studies. Six major recurring themes were identified: clinical factors, clinical experience, patient preferences and perceptions, heuristics and biases, artificial intelligence and informatics, and existing guidelines. These results suggest that inconsistency in treatment recommendations is a real possibility and despite great advancements in dental science, evidence-based practice is but one of a multitude of complex determinants driving clinical decision making in dentistry. In conclusion, clinical decisions, particularly those made individually by a dental practitioner, are potentially prone to sub-optimal treatment and poorer patient outcomes.

Collective intelligence improves probabilistic diagnostic assessments

OBJECTIVES

Collective intelligence, the "wisdom of the crowd," seeks to improve the quality of judgments by aggregating multiple individual inputs. Here, we evaluate the success of collective intelligence strategies applied to probabilistic diagnostic judgments.

METHODS

We compared the performance of individual and collective intelligence judgments on two series of clinical cases requiring probabilistic diagnostic assessments, or "forecasts". We assessed the quality of forecasts using Brier scores, which compare forecasts to observed outcomes.

RESULTS

On both sets of cases, the collective intelligence answers outperformed nearly every individual forecaster or team. The improved performance by collective intelligence was mediated by both improved resolution and calibration of probabilistic assessments. In a secondary analysis looking at the effect of varying number of individual inputs in collective intelligence answers from two different data sources, nearly identical curves were found in the two data sets showing 11-12% improvement when averaging two independent inputs, 15% improvement averaging four independent inputs, and small incremental improvements with further increases in number of individual inputs.

CONCLUSIONS

Our results suggest that the application of collective intelligence strategies to probabilistic diagnostic forecasts is a promising approach to improve diagnostic accuracy and reduce diagnostic error.

Strategies to approach the judicialization of health in Brazil: an evidence brief

Abstract This article seeks to identify and discuss evidence-informed options to address the judicialization of health. The Supporting Policy Relevant Reviews and Trials Tools were used to define the problem and the search strategy, which was carried out in the following databases: PubMed, Health Systems Evidence, Campbell, Cochrane Collaboration, Rx for Change Database, and PDQ-Evidence. Selection and assessment of methodological quality was performed by two independent reviewers. The results were presented in a narrative synthesis. This study selected 19 systematic reviews that pointed out four strategies to address the judicialization of health in Brazil: 1) Rapid response service, 2) Continuous education program, 3) Mediation service between the parties involved, and 4) Adoption of a computer-based, online decision-making support tool and patient-mediated interventions. This study therefore presented and characterized four options that can be considered to address the judicialization of health. The implementation of these options must ensure the participation of different actors, reflecting on different contexts and the impact on the health system. The availability of human and financial resources and the training of teams are critical points for the successful implementation of the options.

Strategies to approach the judicialization of health in Brazil: an evidence brief

This article seeks to identify and discuss evidence-informed options to address the judicialization of health. The Supporting Policy Relevant Reviews and Trials Tools were used to define the problem and the search strategy, which was carried out in the following databases: PubMed, Health Systems Evidence, Campbell, Cochrane Collaboration, Rx for Change Database, and PDQ-Evidence. Selection and assessment of methodological quality was performed by two independent reviewers. The results were presented in a narrative synthesis. This study selected 19 systematic reviews that pointed out four strategies to address the judicialization of health in Brazil: 1) Rapid response service, 2) Continuous education program, 3) Mediation service between the parties involved, and 4) Adoption of a computer-based, online decision-making support tool and patient-mediated interventions. This study therefore presented and characterized four options that can be considered to address the judicialization of health. The implementation of these options must ensure the participation of different actors, reflecting on different contexts and the impact on the health system. The availability of human and financial resources and the training of teams are critical points for the successful implementation of the options.

Clinicians' Use of Quantitative Information while Assessing the Rate of Functional Progression in Glaucoma

PURPOSE

Clinicians use both global and point-wise information from visual fields to assess the rate of glaucomatous functional progression. We asked which objective, quantitative measures best correlated with subjective assessment by glaucoma experts. In particular, we aimed to determine how much that judgment was based on localized rates of change vs. on global indices reported by the perimeter.

DESIGN

Prospective cohort study.

PARTICIPANTS

Eleven academic, expert glaucoma specialists independently scored the rate of functional progression, from 1 (improvement) to 7 (very rapid progression), for a series of 5 biannual clinical printouts from 100 glaucoma or glaucoma suspect eyes of 51 participants, 20 of which were scored twice to assess repeatability.

METHODS

Regression models were used to predict the average of the 11 clinicians' scores based on objective rates of change of mean deviation (MD), visual field index (VFI), pattern standard deviation (PSD), the Nth fastest progressing location, and the Nth fastest progressing of 10 anatomically defined clusters of locations after weighting by eccentricity.

MAIN OUTCOME MEASURES

Correlation between the objective rates of change and the average of the 11 clinicians' scores.

RESULTS

The average MD of the study eyes was -2.4 dB (range, -16.8 to +2.8 dB). The mean clinician score was highly repeatable, with an intraclass correlation coefficient of 0.95. It correlated better with the rate of change of VFI (pseudo-R2 = 0.73, 95% confidence interval [CI, 0.60-0.83]) than with MD (pseudo-R2 = 0.63, 95% CI [0.45-0.76]) or PSD (pseudo-R2 = 0.41, 95% CI [0.26-0.55]). Using point-wise information, the highest correlations were found with the fifth-fastest progressing location (pseudo-R2 = 0.71, 95% CI [0.56-0.80]) and the fastest-progressing cluster after eccentricity weighting (pseudo-R2 = 0.61, 95% CI [0.48-0.72]). Among 25 eyes with an average VFI of > 99%, the highest observed pseudo-R2 value was 0.34 (95% CI [0.16-0.61]) for PSD.

CONCLUSIONS

Expert academic glaucoma specialists' assessment of the rate of change correlated best with VFI rates, except in eyes with a VFI near the ceiling of 100%. Sensitivities averaged within clusters of locations have been shown to detect change sooner, but the experts' opinions correlated more closely with global VFI. This could be because it is currently the only index for which the perimeter automatically provides a quantitative estimate of the rate of functional progression.

Clinician's experiences with involuntary commitment for substance use disorder: A qualitative study of moral distress

BACKGROUND

Petitions for involuntary commitment of people living with a substance use disorder (SUD) have almost doubled since 2011 in Massachusetts through the policy Section 35. However, the efficacy of this controversial policy remains unclear, and clinicians differ on whether it ought to be used. This study examines how clinicians decide whether to use Section 35 and their experiences of moral distress, the negative feeling that occurs when a clinician is required to pursue a treatment option against their moral judgement due to institutional constraints, associated with its use.

METHODS

Qualitative semi-structured interviews with clinicians in Massachusetts were conducted between December 2019 and February 2020 and continued until thematic saturation. Thematic and narrative analysis was conducted with recorded and transcribed interviews.

RESULTS

Among 21 clinicians, most (77%) experienced some or high moral distress when utilizing Section 35 for involuntary commitment, with clinicians working in emergency departments experiencing less distress than those working in SUD clinics. Clinicians with low moral distress referenced successful patient anecdotes and held an abstinence-based view of SUD, while clinicians with high moral distress were concerned by systemic treatment failures and understood SUD through a nuanced and harm reduction-oriented view. Clinicians across professional settings were concerned by the involvement of law enforcement and criminal justice settings in the Section 35 process. Clinicians employed a variety of strategies to cope with moral distress, including team-based decision-making and viewing the petition as a last resort. Barriers to utilizing Section 35 included restrictive court hours and lack of post-section aftercare services.

CONCLUSION

Widespread distress associated with use of involuntary commitment and inconsistent approaches to its use highlight the need for better care coordination and guidance on best practices for utilization of this policy.

Comparison of Diagnostic Recommendations from Individual Physicians versus the Collective Intelligence of Multiple Physicians in Ambulatory Cases Referred for Specialist Consultation

BACKGROUND

Studies report higher diagnostic accuracy using the collective intelligence (CI) of multiple clinicians compared with individual clinicians. However, the diagnostic process is iterative, and unexplored is the value of CI in improving clinical recommendations leading to a final diagnosis.

METHODS

To compare the appropriateness of diagnostic recommendations advised by individual physicians versus the CI of physicians, we entered actual consultation requests sent by primary care physicians to specialists onto a web-based CI platform capable of collecting diagnostic recommendations (next steps for care) from multiple physicians. We solicited responses to 35 cases (12 endocrinology, 13 gynecology, 10 neurology) from ≥3 physicians of any specialty through the CI platform, which aggregated responses into a CI output. The primary outcome was the appropriateness of individual physician recommendations versus the CI output recommendations, using recommendations agreed upon by 2 specialists in the same specialty as a gold standard. The secondary outcome was the recommendations' potential for harm.

RESULTS

A total of 177 physicians responded. Cases had a median of 7 respondents (interquartile range: 5-10). Diagnostic recommendations in the CI output achieved higher levels of appropriateness (69%) than recommendations from individual physicians (45%; χ² = 5.95, P = 0.015). Of the CI recommendations, 54% were potentially harmful, as compared with 41% of individuals' recommendations (χ² = 2.49, P = 0.11).

LIMITATIONS

Cases were from a single institution. CI was solicited using a single algorithm/platform.

CONCLUSIONS

When seeking specialist guidance, diagnostic recommendations from the CI of multiple physicians are more appropriate than recommendations from most individual physicians, measured against specialist recommendations. Although CI provides useful recommendations, some have potential for harm. Future research should explore how to use CI to improve diagnosis while limiting harm from inappropriate tests/therapies.

Decades on emergency decision-making: a bibliometric analysis and literature review

When an emergency occurs, effective decisions should be made in a limited time to reduce the casualties and economic losses as much as possible. In the past decades, emergency decision-making (EDM) has become a research hotspot and a lot of studies have been conducted for better managing emergency events under tight time constraint. However, there is a lack of a comprehensive bibliometric analysis of the literature on this topic. The objective of this paper is to provide academic community with a complete bibliometric analysis of the EDM researches to generate a global picture of developments, focus areas, and trends in the field. A total of 303 journal publications published between 2010 and 2020 were identified and analyzed using the VOSviewer in regard to cooperation network, co-citation network, and keyword co-occurrence network. The findings indicate that the annual publications in this research field have increased rapidly since 2014. Based on the cooperation network and co-citation network analyses, the most productive and influential countries, institutions, researchers, and their cooperation networks were identified. Using the co-citation network analysis, the landmark articles and the core journals in the EDM area are found out. With the help of the keyword co-occurrence network analysis, research hotspots and development of the EDM domain are determined. According to current trends and blind spots in the literature, possible directions for further investigation are finally suggested for EDM. The literature review results provide valuable information and new insights for both scholars and practitioners to grasp the current situation, hotspots and future research agenda of the EDM field.

Collective human intelligence outperforms artificial intelligence in a skin lesion classification task

BACKGROUND AND OBJECTIVES

Convolutional neural networks (CNN) enable accurate diagnosis of medical images and perform on or above the level of individual physicians. Recently, collective human intelligence (CoHI) was shown to exceed the diagnostic accuracy of individuals. Thus, diagnostic performance of CoHI (120 dermatologists) versus individual dermatologists versus two state-of-the-art CNN was investigated.

PATIENTS AND METHODS

Cross-sectional reader study with presentation of 30 clinical cases to 120 dermatologists. Six diagnoses were offered and votes collected via remote voting devices (quizzbox®, Quizzbox Solutions GmbH, Stuttgart, Germany). Dermatoscopic images were classified by a binary and multiclass CNN (FotoFinder Systems GmbH, Bad Birnbach, Germany). Three sets of diagnostic classifications were scored against ground truth: (1) CoHI, (2) individual dermatologists, and (3) CNN.

RESULTS

CoHI attained a significantly higher accuracy [95 % confidence interval] (80.0 % [62.7 %-90.5 %]) than individual dermatologists (75.7 % [73.8 %-77.5 %]) and CNN (70.0 % [52.1 %-83.3 %]; all P < 0.001) in binary classifications. Moreover, CoHI achieved a higher sensitivity (82.4 % [59.0 %-93.8 %]) and specificity (76.9 % [49.7 %-91.8 %]) than individual dermatologists (sensitivity 77.8 % [75.3 %-80.2 %], specificity 73.0 % [70.6 %-75.4 %]) and CNN (sensitivity 70.6 % [46.9 %-86.7 %], specificity 69.2 % [42.4 %-87.3 %]). The diagnostic accuracy of CoHI was superior to that of individual dermatologists (P < 0.001) in multiclass evaluation, with the accuracy of the latter comparable to multiclass CNN.

CONCLUSIONS

Our analysis revealed that the majority vote of an interconnected group of dermatologists (CoHI) outperformed individuals and CNN in a demanding skin lesion classification task.

Characterization and Comparison of the Utilization of Facebook Groups Between Public Medical Professionals and Technical Communities to Facilitate Idea Sharing and Crowdsourcing During the COVID-19 Pandemic: Cross-sectional Observational Study

BACKGROUND

Strict social distancing measures owing to the COVID-19 pandemic have led people to rely more heavily on social media, such as Facebook groups, as a means of communication and information sharing. Multiple Facebook groups have been formed by medical professionals, laypeople, and engineering or technical groups to discuss current issues and possible solutions to the current medical crisis.

OBJECTIVE

This study aimed to characterize Facebook groups formed by laypersons, medical professionals, and technical professionals, with specific focus on information dissemination and requests for crowdsourcing.

METHODS

Facebook was queried for user-created groups with the keywords "COVID," "Coronavirus," and "SARS-CoV-2" at a single time point on March 31, 2020. The characteristics of each group were recorded, including language, privacy settings, security requirements to attain membership, and membership type. For each membership type, the group with the greatest number of members was selected, and in each of these groups, the top 100 posts were identified using Facebook's algorithm. Each post was categorized and characterized (evidence-based, crowd-sourced, and whether the poster self-identified). STATA (version 13 SE, Stata Corp) was used for statistical analysis.

RESULTS

Our search yielded 257 COVID-19-related Facebook groups. Majority of the groups (n=229, 89%) were for laypersons, 26 (10%) were for medical professionals, and only 2 (1%) were for technical professionals. The number of members was significantly greater in medical groups (21,215, SD 35,040) than in layperson groups (7623, SD 19,480) (P<.01). Medical groups were significantly more likely to require security checks to attain membership (81% vs 43%; P<.001) and less likely to be public (3 vs 123; P<.001) than layperson groups. Medical groups had the highest user engagement, averaging 502 (SD 633) reactions (P<.01) and 224 (SD 311) comments (P<.01) per post. Medical professionals were more likely to use the Facebook groups for education and information sharing, including academic posts (P<.001), idea sharing (P=.003), resource sharing (P=.02) and professional opinions (P<.001), and requesting for crowdsourcing (P=.003). Layperson groups were more likely to share news (P<.001), humor and motivation (P<.001), and layperson opinions (P<.001). There was no significant difference in the number of evidence-based posts among the groups (P=.10).

CONCLUSIONS

Medical professionals utilize Facebook groups as a forum to facilitate collective intelligence (CI) and are more likely to use Facebook groups for education and information sharing, including academic posts, idea sharing, resource sharing, and professional opinions, which highlights the power of social media to facilitate CI across geographic distances. Layperson groups were more likely to share news, humor, and motivation, which suggests the utilization of Facebook groups to provide comedic relief as a coping mechanism. Further investigations are necessary to study Facebook groups' roles in facilitating CI, crowdsourcing, education, and community-building.

g versus c: comparing individual and collective intelligence across two meta-analyses

Collective intelligence (CI) is said to manifest in a group's domain general mental ability. It can be measured across a battery of group IQ tests and statistically reduced to a latent factor called the "c-factor." Advocates have found the c-factor predicts group performance better than individual IQ. We test this claim by meta-analyzing correlations between the c-factor and nine group performance criterion tasks generated by eight independent samples (N = 857 groups). Results indicated a moderate correlation, r, of .26 (95% CI .10, .40). All but four studies comprising five independent samples (N = 366 groups) failed to control for the intelligence of individual members using individual IQ scores or their statistically reduced equivalent (i.e., the g-factor). A meta-analysis of this subset of studies found the average IQ of the groups' members had little to no correlation with group performance (r = .06, 95% CI -.08, .20). Around 80% of studies did not have enough statistical power to reliably detect correlations between the primary predictor variables and the criterion tasks. Though some of our findings are consistent with claims that a general factor of group performance may exist and relate positively to group performance, limitations suggest alternative explanations cannot be dismissed. We caution against prematurely embracing notions of the c-factor unless it can be independently and robustly replicated and demonstrated to be incrementally valid beyond the g-factor in group performance contexts.

Unity Is Intelligence: A Collective Intelligence Experiment on ECG Reading to Improve Diagnostic Performance in Cardiology

Medical errors have a huge impact on clinical practice in terms of economic and human costs. As a result, technology-based solutions, such as those grounded in artificial intelligence (AI) or collective intelligence (CI), have attracted increasing interest as a means of reducing error rates and their impacts. Previous studies have shown that a combination of individual opinions based on rules, weighting mechanisms, or other CI solutions could improve diagnostic accuracy with respect to individual doctors. We conducted a study to investigate the potential of this approach in cardiology and, more precisely, in electrocardiogram (ECG) reading. To achieve this aim, we designed and conducted an experiment involving medical students, recent graduates, and residents, who were asked to annotate a collection of 10 ECGs of various complexity and difficulty. For each ECG, we considered groups of increasing size (from three to 30 members) and applied three different CI protocols. In all cases, the results showed a statistically significant improvement (ranging from 9% to 88%) in terms of diagnostic accuracy when compared to the performance of individual readers; this difference held for not only large groups, but also smaller ones. In light of these results, we conclude that CI approaches can support the tasks mentioned above, and possibly other similar ones as well. We discuss the implications of applying CI solutions to clinical settings, such as cases of augmented ‘second opinions’ and decision-making.

Applications of Big Data Analytics to Control COVID-19 Pandemic

The COVID-19 epidemic has caused a large number of human losses and havoc in the economic, social, societal, and health systems around the world. Controlling such epidemic requires understanding its characteristics and behavior, which can be identified by collecting and analyzing the related big data. Big data analytics tools play a vital role in building knowledge required in making decisions and precautionary measures. However, due to the vast amount of data available on COVID-19 from various sources, there is a need to review the roles of big data analysis in controlling the spread of COVID-19, presenting the main challenges and directions of COVID-19 data analysis, as well as providing a framework on the related existing applications and studies to facilitate future research on COVID-19 analysis. Therefore, in this paper, we conduct a literature review to highlight the contributions of several studies in the domain of COVID-19-based big data analysis. The study presents as a taxonomy several applications used to manage and control the pandemic. Moreover, this study discusses several challenges encountered when analyzing COVID-19 data. The findings of this paper suggest valuable future directions to be considered for further research and applications.

Studying human-AI collaboration protocols: the case of the Kasparov's law in radiological double reading

Purpose

The integration of Artificial Intelligence into medical practices has recently been advocated for the promise to bring increased efficiency and effectiveness to these practices. Nonetheless, little research has so far been aimed at understanding the best human-AI interaction protocols in collaborative tasks, even in currently more viable settings, like independent double-reading screening tasks.

Methods

To this aim, we report about a retrospective case–control study, involving 12 board-certified radiologists, in the detection of knee lesions by means of Magnetic Resonance Imaging, in which we simulated the serial combination of two Deep Learning models with humans in eight double-reading protocols. Inspired by the so-called Kasparov’s Laws, we investigate whether the combination of humans and AI models could achieve better performance than AI models alone, and whether weak reader, when supported by fit-for-use interaction protocols, could out-perform stronger readers.

Results

We discuss two main findings: groups of humans who perform significantly worse than a state-of-the-art AI can significantly outperform it if their judgements are aggregated by majority voting (in concordance with the first part of the Kasparov’s law); small ensembles of significantly weaker readers can significantly outperform teams of stronger readers, supported by the same computational tool, when the judgments of the former ones are combined within “fit-for-use” protocols (in concordance with the second part of the Kasparov’s law).

Conclusion

Our study shows that good interaction protocols can guarantee improved decision performance that easily surpasses the performance of individual agents, even of realistic super-human AI systems. This finding highlights the importance of focusing on how to guarantee better co-operation within human-AI teams, so to enable safer and more human sustainable care practices.

Comparison of Mobile Application-Based ECG Consultation by Collective Intelligence and ECG Interpretation by Conventional System in a Tertiary-Level Hospital

BACKGROUND AND OBJECTIVES

A mobile application (app)-based electrocardiogram (ECG) consultation system (InterMD Co., Ltd., Seoul, Korea) using the collective intelligence (CI) and the availability of large-scale digitized ECG data would extend the utility of ECGs beyond their current limitations, while at the same time preserving interpretability that remains critical to medical decision-making.

METHODS

We developed a new mobile app-based ECG consultation system by CI for general practitioners. We compared the responses of ECG reading between the mobile app-based CI system and the conventional system in a tertiary referring hospital.

RESULTS

We analyzed 376 consecutive ECGs between December 2017 and May 2019. Of these, 159 ECGs (42.3%) were interpreted by CI through the mobile app-based ECG consultation system and 217 ECGs (57.7%) were analyzed by cardiologists in the conventional systems based on electronic medical record data in a tertiary hospital. All ECG readings were confirmed by an electrophysiologist (EP). The time to an initial response by the CI system was faster than that of the conventional system (6.6 hours vs. 35.8 hours, p<0.0001). The number of responses of each ECG in CI system outnumbered those of the conventional system in the tertiary hospital (3.1 vs. 1.2, p<0.0001). The consensus of the ECG readings with EP was similar in both systems (98.6% vs. 100%, p=0.158).

CONCLUSIONS

The mobile app-based ECG consultation system by CI is as reliable method as the conventional referral system. It would expand the app of the 12-lead ECG with the collaboration of physicians in clinics and hospitals without time and space constraints.

Is the American College of Surgeons Online Communities a safe and useful venue to ask for surgical advice?

BACKGROUND

Many surgeons rely on the American College of Surgeons (ACS) Community Forums for advice on managing complex patients. Our objective was to assess the safety and usefulness of advice provided on the most popular surgical forum.

METHODS

Overall, 120 consecutive, deidentified clinical threads were extracted from the General Surgery community in reverse chronological order. Three groups of three surgeons (mixed academic and community perspectives) evaluated the 120 threads for unsafe or dangerous posts. Positive and negative controls for safe and unsafe answers were included in 20 threads, and reviewers were blinded to their presence. Reviewers were free to access all online and professional resources.

RESULTS

There were 855 unique responses (median 7, 2-15 responses per thread) to the 120 clinical threads/scenarios. The review teams correctly identified all positive and negative controls for safety. While 58(43.3%) of threads contained unsafe advice, the majority (33, 56.9%) were corrected. Reviewers felt that a there was a standard of care response for 62/120 of the threads of which 50 (80.6%) were provided by the responses. Of the 855 responses, 107 (12.5%) were considered unsafe/dangerous.

CONCLUSION

The ACS Community Forums are generally a safe and useful resource for surgeons seeking advice for challenging cases. While unsafe or dangerous advice is not uncommon, other surgeons typically correct it. When utilizing the forums, advice should be taken as a congregate, and any single recommendation should be approached with healthy skepticism. However, social media such as the ACS Forums is self-regulating and can be an appropriate method for surgeons to communicate challenging problems.