Prospective Comparison of Medical Oncologists and a Machine Learning Model to Predict 3-Month Mortality in Patients With Metastatic Solid Tumors

Artificial intelligence innovations in neurosurgical oncology: a narrative review

PURPOSE

Artificial Intelligence (AI) has become increasingly integrated clinically within neurosurgical oncology. This report reviews the cutting-edge technologies impacting tumor treatment and outcomes.

METHODS

A rigorous literature search was performed with the aid of a research librarian to identify key articles referencing AI and related topics (machine learning (ML), computer vision (CV), augmented reality (AR), virtual reality (VR), etc.) for neurosurgical care of brain or spinal tumors.

RESULTS

Treatment of central nervous system (CNS) tumors is being improved through advances across AI-such as AL, CV, and AR/VR. AI aided diagnostic and prognostication tools can influence pre-operative patient experience, while automated tumor segmentation and total resection predictions aid surgical planning. Novel intra-operative tools can rapidly provide histopathologic tumor classification to streamline treatment strategies. Post-operative video analysis, paired with rich surgical simulations, can enhance training feedback and regimens.

CONCLUSION

While limited generalizability, bias, and patient data security are current concerns, the advent of federated learning, along with growing data consortiums, provides an avenue for increasingly safe, powerful, and effective AI platforms in the future.

Towards proactive palliative care in oncology: developing an explainable EHR-based machine learning model for mortality risk prediction

BACKGROUND

Ex-ante identification of the last year in life facilitates a proactive palliative approach. Machine learning models trained on electronic health records (EHR) demonstrate promising performance in cancer prognostication. However, gaps in literature include incomplete reporting of model performance, inadequate alignment of model formulation with implementation use-case, and insufficient explainability hindering trust and adoption in clinical settings. Hence, we aim to develop an explainable machine learning EHR-based model that prompts palliative care processes by predicting for 365-day mortality risk among patients with advanced cancer within an outpatient setting.

METHODS

Our cohort consisted of 5,926 adults diagnosed with Stage 3 or 4 solid organ cancer between July 1, 2017, and June 30, 2020 and receiving ambulatory cancer care within a tertiary center. The classification problem was modelled using Extreme Gradient Boosting (XGBoost) and aligned to our envisioned use-case: "Given a prediction point that corresponds to an outpatient cancer encounter, predict for mortality within 365-days from prediction point, using EHR data up to 365-days prior." The model was trained with 75% of the dataset (n = 39,416 outpatient encounters) and validated on a 25% hold-out dataset (n = 13,122 outpatient encounters). To explain model outputs, we used Shapley Additive Explanations (SHAP) values. Clinical characteristics, laboratory tests and treatment data were used to train the model. Performance was evaluated using area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC), while model calibration was assessed using the Brier score.

RESULTS

In total, 17,149 of the 52,538 prediction points (32.6%) had a mortality event within the 365-day prediction window. The model demonstrated an AUROC of 0.861 (95% CI 0.856-0.867) and AUPRC of 0.771. The Brier score was 0.147, indicating slight overestimations of mortality risk. Explanatory diagrams utilizing SHAP values allowed visualization of feature impacts on predictions at both the global and individual levels.

CONCLUSION

Our machine learning model demonstrated good discrimination and precision-recall in predicting 365-day mortality risk among individuals with advanced cancer. It has the potential to provide personalized mortality predictions and facilitate earlier integration of palliative care.

Identifying potential (re)hemorrhage among sporadic cerebral cavernous malformations using machine learning

The (re)hemorrhage in patients with sporadic cerebral cavernous malformations (CCM) was the primary aim for CCM management. However, accurately identifying the potential (re)hemorrhage among sporadic CCM patients in advance remains a challenge. This study aims to develop machine learning models to detect potential (re)hemorrhage in sporadic CCM patients. This study was based on a dataset of 731 sporadic CCM patients in open data platform Dryad. Sporadic CCM patients were followed up 5 years from January 2003 to December 2018. Support vector machine (SVM), stacked generalization, and extreme gradient boosting (XGBoost) were used to construct models. The performance of models was evaluated by area under receiver operating characteristic curves (AUROC), area under the precision-recall curve (PR-AUC) and other metrics. A total of 517 patients with sporadic CCM were included (330 female [63.8%], mean [SD] age at diagnosis, 42.1 [15.5] years). 76 (re)hemorrhage (14.7%) occurred during follow-up. Among 3 machine learning models, XGBoost model yielded the highest mean (SD) AUROC (0.87 [0.06]) in cross-validation. The top 4 features of XGBoost model were ranked with SHAP (SHapley Additive exPlanations). All-Elements XGBoost model achieved an AUROCs of 0.84 and PR-AUC of 0.49 in testing set, with a sensitivity of 0.86 and a specificity of 0.76. Importantly, 4-Elements XGBoost model developed using top 4 features got a AUROCs of 0.83 and PR-AUC of 0.40, a sensitivity of 0.79, and a specificity of 0.72 in testing set. Two machine learning-based models achieved accurate performance in identifying potential (re)hemorrhages within 5 years in sporadic CCM patients. These models may provide insights for clinical decision-making.

Machine Learning to Allocate Palliative Care Consultations During Cancer Treatment

PURPOSE

For patients with advanced cancer, early consultations with palliative care (PC) specialists reduce costs, improve quality of life, and prolong survival. However, capacity limitations prevent all patients from receiving PC shortly after diagnosis. We evaluated whether a prognostic machine learning system could promote early PC, given existing capacity.

METHODS

Using population-level administrative data in Ontario, Canada, we assembled a cohort of patients with incurable cancer who received palliative-intent systemic therapy between July 1, 2014, and December 30, 2019. We developed a machine learning system that predicted death within 1 year of each treatment using demographics, cancer characteristics, treatments, symptoms, laboratory values, and history of acute care admissions. We trained the system in patients who started treatment before July 1, 2017, and evaluated the potential impact of the system on PC in subsequent patients.

RESULTS

Among 560,210 treatments received by 54,628 patients, death occurred within 1 year of 45.2% of treatments. The machine learning system recommended the same number of PC consultations observed with usual care at the 60.0% 1-year risk of death, with a first-alarm positive predictive value of 69.7% and an outcome-level sensitivity of 74.9%. Compared with usual care, system-guided care could increase early PC by 8.5% overall (95% CI, 7.5 to 9.5; P < .001) and by 15.3% (95% CI, 13.9 to 16.6; P < .001) among patients who live 6 months beyond their first treatment, without requiring more PC consultations in total or substantially increasing PC among patients with a prognosis exceeding 2 years.

CONCLUSION

Prognostic machine learning systems could increase early PC despite existing resource constraints. These results demonstrate an urgent need to deploy and evaluate prognostic systems in real-time clinical practice to increase access to early PC.

Looking Beyond Mortality Prediction: Primary Care Physician Views of Patients' Palliative Care Needs Predicted by a Machine Learning Tool

OBJECTIVES

 To assess primary care physicians' (PCPs) perception of the need for serious illness conversations (SIC) or other palliative care interventions in patients flagged by a machine learning tool for high 1-year mortality risk.

METHODS

 We surveyed PCPs from four Brigham and Women's Hospital primary care practice sites. Multiple mortality prediction algorithms were ensembled to assess adult patients of these PCPs who were either enrolled in the hospital's integrated care management program or had one of several chronic conditions. The patients were classified as high or low risk of 1-year mortality. A blinded survey had PCPs evaluate these patients for palliative care needs. We measured PCP and machine learning tool agreement regarding patients' need for an SIC/elevated risk of mortality.

RESULTS

 Of 66 PCPs, 20 (30.3%) participated in the survey. Out of 312 patients evaluated, 60.6% were female, with a mean (standard deviation [SD]) age of 69.3 (17.5) years, and a mean (SD) Charlson Comorbidity Index of 2.80 (2.89). The machine learning tool identified 162 (51.9%) patients as high risk. Excluding deceased or unfamiliar patients, PCPs felt that an SIC was appropriate for 179 patients; the machine learning tool flagged 123 of these patients as high risk (68.7% concordance). For 105 patients whom PCPs deemed SIC unnecessary, the tool classified 83 as low risk (79.1% concordance). There was substantial agreement between PCPs and the tool (Gwet's agreement coefficient of 0.640).

CONCLUSIONS

 A machine learning mortality prediction tool offers promise as a clinical decision aid, helping clinicians pinpoint patients needing palliative care interventions.

Patient and Caregiver Perceptions of an Interface Design to Communicate Artificial Intelligence-Based Prognosis for Patients With Advanced Solid Tumors

PURPOSE

Use of artificial intelligence (AI) in cancer care is increasing. What remains unclear is how best to design patient-facing systems that communicate AI output. With oncologist input, we designed an interface that presents patient-specific, machine learning-based 6-month survival prognosis information designed to aid oncology providers in preparing for and discussing prognosis with patients with advanced solid tumors and their caregivers. The primary purpose of this study was to assess patient and caregiver perceptions and identify enhancements of the interface for communicating 6-month survival and other prognosis information when making treatment decisions concerning anticancer and supportive therapy.

METHODS

This qualitative study included interviews and focus groups conducted between November and December 2022. Purposive sampling was used to recruit former patients with cancer and/or former caregivers of patients with cancer who had participated in cancer treatment decisions from Utah or elsewhere in the United States. Categories and themes related to perceptions of the interface were identified.

RESULTS

We received feedback from 20 participants during eight individual interviews and two focus groups, including four cancer survivors, 13 caregivers, and three representing both. Overall, most participants expressed positive perceptions about the tool and identified its value for supporting decision making, feeling less alone, and supporting communication among oncologists, patients, and their caregivers. Participants identified areas for improvement and implementation considerations, particularly that oncologists should share the tool and guide discussions about prognosis with patients who want to receive the information.

CONCLUSION

This study revealed important patient and caregiver perceptions of and enhancements for the proposed interface. Originally designed with input from oncology providers, patient and caregiver participants identified additional interface design recommendations and implementation considerations to support communication about prognosis.

Design of an interface to communicate artificial intelligence-based prognosis for patients with advanced solid tumors: a user-centered approach

OBJECTIVES

To design an interface to support communication of machine learning (ML)-based prognosis for patients with advanced solid tumors, incorporating oncologists' needs and feedback throughout design.

MATERIALS AND METHODS

Using an interdisciplinary user-centered design approach, we performed 5 rounds of iterative design to refine an interface, involving expert review based on usability heuristics, input from a color-blind adult, and 13 individual semi-structured interviews with oncologists. Individual interviews included patient vignettes and a series of interfaces populated with representative patient data and predicted survival for each treatment decision point when a new line of therapy (LoT) was being considered. Ongoing feedback informed design decisions, and directed qualitative content analysis of interview transcripts was used to evaluate usability and identify enhancement requirements.

RESULTS

Design processes resulted in an interface with 7 sections, each addressing user-focused questions, supporting oncologists to "tell a story" as they discuss prognosis during a clinical encounter. The iteratively enhanced interface both triggered and reflected design decisions relevant when attempting to communicate ML-based prognosis, and exposed misassumptions. Clinicians requested enhancements that emphasized interpretability over explainability. Qualitative findings confirmed that previously identified issues were resolved and clarified necessary enhancements (eg, use months not days) and concerns about usability and trust (eg, address LoT received elsewhere). Appropriate use should be in the context of a conversation with an oncologist.

CONCLUSION

User-centered design, ongoing clinical input, and a visualization to communicate ML-related outcomes are important elements for designing any decision support tool enabled by artificial intelligence, particularly when communicating prognosis risk.

Implementing a Serious Illness Risk Prediction Model: Impact on Goals of Care Documentation

CONTEXT

Goals of care conversations can promote high value care for patients with serious illness, yet documented discussions infrequently occur in hospital settings.

OBJECTIVES

We sought to develop a quality improvement initiative to improve goals of care documentation for hospitalized patients.

METHODS

Implementation occurred at an academic medical center in Pittsburgh, Pennsylvania. Intervention included integration of a 90-day mortality prediction model grouping patients into low, intermediate, and high risk; a centralized goals of care note; and automated notifications and targeted palliative consults. We compared documented goals of care discussions by risk score before and after implementation.

RESULTS

Of the 12,571 patients hospitalized preimplementation and 10,761 postimplementation, 1% were designated high risk and 11% intermediate risk of mortality. Postimplementation, goals of care documentation increased for high (17.6%-70.8%, P< 0.0001) and intermediate risk patients (9.6%-28.0%, P < 0.0001). For intermediate risk patients, the percentage of goals of care documentation performed by palliative medicine specialists increased from pre- to postimplementation (52.3%-71.2%, P = 0.0002). For high-risk patients, the percentage of goals of care documentation completed by the primary service increased from pre-to postimplementation (36.8%-47.1%, P = 0.5898, with documentation performed by palliative medicine specialists slightly decreasing from pre- to postimplementation (63.2%-52.9%, P = 0.5898).

CONCLUSIONS

Implementation of a goals of care initiative using a mortality prediction model significantly increased goals of care documentation especially among high-risk patients. Further study to assess strategies to increase goals of care documentation for intermediate risk patients is needed especially by nonspecialty palliative care.

Artificial intelligence-based risk stratification, accurate diagnosis and treatment prediction in gynecologic oncology

As data-driven science, artificial intelligence (AI) has paved a promising path toward an evolving health system teeming with thrilling opportunities for precision oncology. Notwithstanding the tremendous success of oncological AI in such fields as lung carcinoma, breast tumor and brain malignancy, less attention has been devoted to investigating the influence of AI on gynecologic oncology. Hereby, this review sheds light on the ever-increasing contribution of state-of-the-art AI techniques to the refined risk stratification and whole-course management of patients with gynecologic tumors, in particular, cervical, ovarian and endometrial cancer, centering on information and features extracted from clinical data (electronic health records), cancer imaging including radiological imaging, colposcopic images, cytological and histopathological digital images, and molecular profiling (genomics, transcriptomics, metabolomics and so forth). However, there are still noteworthy challenges beyond performance validation. Thus, this work further describes the limitations and challenges faced in the real-word implementation of AI models, as well as potential solutions to address these issues.

Clinician Trust in Artificial Intelligence: What is Known and How Trust Can Be Facilitated

Predictive analytics based on artificial intelligence (AI) offer clinicians the opportunity to leverage big data available in electronic health records (EHR) to improve clinical decision-making, and thus patient outcomes. Despite this, many barriers exist to facilitating trust between clinicians and AI-based tools, limiting its current impact. Potential solutions are available at both the local and national level. It will take a broad and diverse coalition of stakeholders, from health-care systems, EHR vendors, and clinical educators to regulators, researchers and the patient community, to help facilitate this trust so that the promise of AI in health care can be realized.

Leveraging the Academic Artificial Intelligence Silecosystem to Advance the Community Oncology Enterprise

Over the last 75 years, artificial intelligence has evolved from a theoretical concept and novel paradigm describing the role that computers might play in our society to a tool with which we daily engage. In this review, we describe AI in terms of its constituent elements, the synthesis of which we refer to as the AI Silecosystem. Herein, we provide an historical perspective of the evolution of the AI Silecosystem, conceptualized and summarized as a Kuhnian paradigm. This manuscript focuses on the role that the AI Silecosystem plays in oncology and its emerging importance in the care of the community oncology patient. We observe that this important role arises out of a unique alliance between the academic oncology enterprise and community oncology practices. We provide evidence of this alliance by illustrating the practical establishment of the AI Silecosystem at the City of Hope Comprehensive Cancer Center and its team utilization by community oncology providers.

Association between Computed Tomography-Determined Loss of Muscle Mass and Impaired Three-Month Survival in Frail Older Adults with Cancer

As patients with solid (non-hematological) cancers and a life expectancy of <3 months rarely benefit from oncological treatment, we examined whether the CT-determined loss of muscle mass is associated with an impaired 3-month overall survival (OS) in frail ≥75-year-old patients with cancer. Frailty was assessed with G8-screening and comprehensive geriatric assessment in older adults at risk of frailty. The L3-level skeletal (SMI) and psoas (PMI) muscle indexes were determined from routine CT scans. Established and optimized SMI and PMI cut-offs were used. In the non-curative treatment group (n = 58), 3-month OS rates for normal and low SMI were 95% and 64% (HR 9.28; 95% CI 1.2-71) and for PMI 88%, and 60%, respectively (HR 4.10; 1.3-13). A Cox multivariable 3-month OS model showed an HR of 10.7 (1.0-110) for low SMI, 2.34 (0.6-9.8) for ECOG performance status 3-4, 2.11 (0.5-8.6) for clinical frailty scale 5-9, and 0.57 (0.1-2.8) for males. The 24-month OS rates in the curative intent group (n = 21) were 91% and 38% for the normal and low SMI groups, respectively. In conclusion, CT-determined low muscle mass is independently associated with an impaired 3-month OS and, alongside geriatric assessment, could aid in oncological versus best supportive care decision-making in frail patients with non-curable cancers.

A Retrospective Study Investigating the Safety and Efficacy of Nanoliposomal Irinotecan in Elderly Patients with Unresectable Pancreatic Cancer

Although nanoliposomal irinotecan combined with 5-fluorouracil and leucovorin (nal-IRI+5-FU/LV) has been used to treat first-line resistant unresectable pancreatic cancer, the efficacy and safety data among the elderly remain limited. We retrospectively analyzed clinical outcomes among elderly patients. Patients treated with nal-IRI+5-FU/LV were assigned to the elderly (≥75 years) and non-elderly (<75 years) groups. Herein, 85 patients received nal-IRI+5-FU/LV, with 32 assigned to the elderly group. Patient characteristics in the elderly and non-elderly groups were as follows: age: 78.5 (75-88)/71 (48-74), male: 17/32 (53%/60%), performance status (ECOG) 0:9/20 (28%/38%), nal-IRI+5-FU/LV in second line: 23/24 (72%/45%), respectively. A significantly high number of elderly patients exhibited aggravated kidney and hepatic functions. Median overall survival (OS) and progression-free survival (PFS) in the elderly group vs. non-elderly group were 9.4 months vs. 9.9 months (hazard ratio (HR) 1.51, 95% confidence interval (CI) 0.85-2.67, p = 0.16) and 3.4 months vs. 3.7 months (HR 1.41, 95% CI 0.86-2.32, p = 0.17). Both groups exhibited a similar incidence of efficacy and adverse events. There were no significant differences in OS and PFS between groups. We analyzed the C-reactive protein/albumin ratio (CAR) and neutrophil/lymphocyte ratio (NLR) as indicators that could determine eligibility for nal-IRI+5-FU/LV. The median CAR and NLR scores in the ineligible group were 1.17 and 4.23 (p < 0.001 and p = 0.018, respectively). Elderly patients with worse CAR and NLR score could be deemed ineligible for nal-IRI+5-FU/LV.

Evaluating Listening Performance for COVID-19 Detection by Clinicians and Machine Learning: A Comparative Study

BACKGROUND

To date, performance comparisons between men and machines have been performed in many health domains. Yet, machine learning models and human performance comparisons in audio-based respiratory diagnosis remain largely unexplored.

OBJECTIVE

The primary objective of this study is to compare human clinicians and a machine learning model in predicting COVID-19 from respiratory sound recordings.

METHODS

In this study, we compare human clinicians and a machine learning model in predicting COVID-19 from respiratory sound recordings. Prediction performance on 24 audio samples (12 tested positive) made by 36 clinicians with experience in treating COVID-19 or other respiratory illnesses is compared with predictions made by a machine learning model trained on 1,162 samples. Each sample consists of voice, cough, and breathing sound recordings from one subject, and the length of each sample is around 20 seconds. We also investigated whether combining the predictions of the model and human experts could further enhance the performance, in terms of both accuracy and confidence.

RESULTS

The machine learning model outperformed the clinicians, yielding a sensitivity of 0.75 and a specificity of 0.83, while the best performance achieved by the clinician was 0.67 in terms of sensitivity and 0.75 in terms of specificity. Integrating clinicians' and model's predictions, however, could enhance performance further, achieving a sensitivity of 0.83 and a specificity of 0.92.

CONCLUSIONS

Our findings suggest that the clinicians and the machine learning model could make better clinical decisions via a cooperative approach and achieve higher confidence in audio-based respiratory diagnosis.

Advanced Care Planning for Hospitalized Patients Following Clinician Notification of Patient Mortality by a Machine Learning Algorithm

Importance

Goal-concordant care is an ongoing challenge in hospital settings. Identification of high mortality risk within 30 days may call attention to the need to have serious illness conversations, including the documentation of patient goals of care.

Objective

To examine goals of care discussions (GOCDs) in a community hospital setting with patients identified as having a high risk of mortality by a machine learning mortality prediction algorithm.

Design, Setting, and Participants

This cohort study took place at community hospitals within 1 health care system. Participants included adult patients with a high risk of 30-day mortality who were admitted to 1 of 4 hospitals between January 2 and July 15, 2021. Patient encounters of inpatients in the intervention hospital where physicians were notified of the computed high risk mortality score were compared with patient encounters of inpatients in 3 community hospitals without the intervention (ie, matched control).

Intervention

Physicians of patients with a high risk of mortality within 30 days received notification and were encouraged to arrange for GOCDs.

Main Outcomes and Measures

The primary outcome was the percentage change of documented GOCDs prior to discharge. Propensity-score matching was completed on a preintervention and postintervention period using age, sex, race, COVID-19 status, and machine learning-predicted mortality risk scores. A difference-in-difference analysis validated the results.

Results

Overall, 537 patients were included in this study with 201 in the preintervention period (94 in the intervention group; 104 in the control group) and 336 patients in the postintervention period. The intervention and control groups included 168 patients per group and were well-balanced in age (mean [SD], 79.3 [9.60] vs 79.6 [9.21] years; standardized mean difference [SMD], 0.03), sex (female, 85 [51%] vs 85 [51%]; SMD, 0), race (White patients, 145 [86%] vs 144 [86%]; SMD 0.006), and Charlson comorbidities (median [range], 8.00 [2.00-15.0] vs 9.00 [2.00 to 19.0]; SMD, 0.34). Patients in the intervention group from preintervention to postintervention period were associated with being 5 times more likely to have documented GOCDs (OR, 5.11 [95% CI, 1.93 to 13.42]; P = .001) by discharge compared with matched controls, and GOCD occurred significantly earlier in the hospitalization in the intervention patients as compared with matched controls (median, 4 [95% CI, 3 to 6] days vs 16 [95% CI, 15 to not applicable] days; P < .001). Similar findings were observed for Black patient and White patient subgroups.

Conclusions and Relevance

In this cohort study, patients whose physicians had knowledge of high-risk predictions from machine learning mortality algorithms were associated with being 5 times more likely to have documented GOCDs than matched controls. Additional external validation is needed to determine if similar interventions would be helpful at other institutions.

Assessing Metabolic Markers in Glioblastoma Using Machine Learning: A Systematic Review

Glioblastoma (GBM) is a common and deadly brain tumor with late diagnoses and poor prognoses. Machine learning (ML) is an emerging tool that can create highly accurate diagnostic and prognostic prediction models. This paper aimed to systematically search the literature on ML for GBM metabolism and assess recent advancements. A literature search was performed using predetermined search terms. Articles describing the use of an ML algorithm for GBM metabolism were included. Ten studies met the inclusion criteria for analysis: diagnostic (n = 3, 30%), prognostic (n = 6, 60%), or both (n = 1, 10%). Most studies analyzed data from multiple databases, while 50% (n = 5) included additional original samples. At least 2536 data samples were run through an ML algorithm. Twenty-seven ML algorithms were recorded with a mean of 2.8 algorithms per study. Algorithms were supervised (n = 24, 89%), unsupervised (n = 3, 11%), continuous (n = 19, 70%), or categorical (n = 8, 30%). The mean reported accuracy and AUC of ROC were 95.63% and 0.779, respectively. One hundred six metabolic markers were identified, but only EMP3 was reported in multiple studies. Many studies have identified potential biomarkers for GBM diagnosis and prognostication. These algorithms show promise; however, a consensus on even a handful of biomarkers has not yet been made.

Likelihood Ratios: An Important Concept for Palliative Physicians to Understand

Palliative care has several tools and questionnaires which are commonly used for patient-related outcomes and prognosis. As an example, the Surprise Question (I would or would not be surprised that this person would have died in a year) has been used as a screen for palliative care referral but also used as a prognostic tool. Diagnostic tests, prognostic tools, and tools for gauging outcomes have certain sensitivity and specificity in predicting a diagnosis or outcome. Clinicians often use positive and negative predictive values in judging the merits of a diagnostic tool or questionnaire. However positive and negative predictive values are highly dependent on the prevalence of disease or outcome in a population and thus are not portable across studies. Likelihood ratios are both portable across populations but also provide the strength of the diagnostic or predictive measure of a test or questionnaire. In this article, we review the value and limitations of likelihood ratios and illustrate the value of using likelihood ratios using 3 studies centered on the Surprise Question published in 2022.