Research progress of zero-shot learning

Generalizability and robustness evaluation of attribute-based zero-shot learning

In the field of deep learning, large quantities of data are typically required to effectively train models. This challenge has given rise to techniques like zero-shot learning (ZSL), which trains models on a set of "seen" classes and evaluates them on a set of "unseen" classes. Although ZSL has shown considerable potential, particularly with the employment of generative methods, its generalizability to real-world scenarios remains uncertain. The hypothesis of this work is that the performance of ZSL models is systematically influenced by the chosen "splits"; in particular, the statistical properties of the classes and attributes used in training. In this paper, we test this hypothesis by introducing the concepts of generalizability and robustness in attribute-based ZSL and carry out a variety of experiments to stress-test ZSL models against different splits. Our aim is to lay the groundwork for future research on ZSL models' generalizability, robustness, and practical applications. We evaluate the accuracy of state-of-the-art models on benchmark datasets and identify consistent trends in generalizability and robustness. We analyze how these properties vary based on the dataset type, differentiating between coarse- and fine-grained datasets, and our findings indicate significant room for improvement in both generalizability and robustness. Furthermore, our results demonstrate the effectiveness of dimensionality reduction techniques in improving the performance of state-of-the-art models in fine-grained datasets.

Shadows of wisdom: Classifying meta-cognitive and morally grounded narrative content via large language models

We investigated large language models' (LLMs) efficacy in classifying complex psychological constructs like intellectual humility, perspective-taking, open-mindedness, and search for a compromise in narratives of 347 Canadian and American adults reflecting on a workplace conflict. Using state-of-the-art models like GPT-4 across few-shot and zero-shot paradigms and RoB-ELoC (RoBERTa -fine-tuned-on-Emotion-with-Logistic-Regression-Classifier), we compared their performance with expert human coders. Results showed robust classification by LLMs, with over 80% agreement and F1 scores above 0.85, and high human-model reliability (Cohen's κ Md across top models = .80). RoB-ELoC and few-shot GPT-4 were standout classifiers, although somewhat less effective in categorizing intellectual humility. We offer example workflows for easy integration into research. Our proof-of-concept findings indicate the viability of both open-source and commercial LLMs in automating the coding of complex constructs, potentially transforming social science research.

Validation of a Zero-shot Learning Natural Language Processing Tool to Facilitate Data Abstraction for Urologic Research

BACKGROUND

Urologic research often requires data abstraction from unstructured text contained within the electronic health record. A number of natural language processing (NLP) tools have been developed to aid with this time-consuming task; however, the generalizability of these tools is typically limited by the need for task-specific training.

OBJECTIVE

To describe the development and validation of a zero-shot learning NLP tool to facilitate data abstraction from unstructured text for use in downstream urologic research.

DESIGN, SETTING, AND PARTICIPANTS

An NLP tool based on the GPT-3.5 model from OpenAI was developed and compared with three physicians for time to task completion and accuracy for abstracting 14 unique variables from a set of 199 deidentified radical prostatectomy pathology reports. The reports were processed in vectorized and scanned formats to establish the impact of optical character recognition on data abstraction.

INTERVENTION

A zero-shot learning NLP tool for data abstraction.

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

The tool was compared with the human abstractors in terms of superiority for data abstraction speed and noninferiority for accuracy.

RESULTS AND LIMITATIONS

The human abstractors required a median (interquartile range) of 93 s (72-122 s) per report for data abstraction, whereas the software required a median of 12 s (10-15 s) for the vectorized reports and 15 s (13-17 s) for the scanned reports (p < 0.001 for all paired comparisons). The accuracies of the three human abstractors were 94.7% (95% confidence interval [CI], 93.8-95.5%), 97.8% (95% CI, 97.2-98.3%), and 96.4% (95% CI, 95.6-97%) for the combined set of 2786 data points. The tool had accuracy of 94.2% (95% CI, 93.3-94.9%) for the vectorized reports and was noninferior to the human abstractors at a margin of -10% (α = 0.025). The tool had slightly lower accuracy of 88.7% (95% CI 87.5-89.9%) for the scanned reports, making it noninferior to two of three human abstractors.

CONCLUSIONS

The developed zero-shot learning NLP tool offers urologic researchers a highly generalizable and accurate method for data abstraction from unstructured text. An open access version of the tool is available for immediate use by the urologic community.

PATIENT SUMMARY

In this report, we describe the design and validation of an artificial intelligence tool for abstracting discrete data from unstructured notes contained within the electronic medical record. This freely available tool, which is based on the GPT-3.5 technology from OpenAI, is intended to facilitate research and scientific discovery by the urologic community.

Graph embedding based multi-label Zero-shot Learning

Multi-label Zero-shot Learning (ZSL) is more reasonable and realistic than standard single-label ZSL because several objects can co-exist in a natural image in real scenarios. Intra-class feature entanglement is a significant factor influencing the alignment of visual and semantic features, resulting in the model's inability to recognize unseen samples comprehensively and completely. We observe that existing multi-label ZSL methods place a greater emphasis on attention-based refinement and decoupling of visual features, while ignoring the relationship between label semantics. Relying on label correlations to solve multi-label ZSL tasks has not been deeply studied. In this paper, we make full use of the co-occurrence relationship between category labels and build a directed weighted semantic graph based on statistics and prior knowledge, in which node features represent category semantics and weighted edges represent conditional probabilities of label co-occurrence. To guide the targeted extraction of visual features, node features and edge set weights are simultaneously updated and refined, and embedded into the visual feature extraction network from a global and local perspective. The proposed method's effectiveness was demonstrated by simulation results on two challenging multi-label ZSL benchmarks: NUS-WIDE and Open Images. In comparison to state-of-the-art models, our model achieves an absolute gain of 2.4% mAP on NUS-WIDE and 2.1% mAP on Open Images respectively.

Identifying suicide documentation in clinical notes through zero-shot learning

Background and Aims

In deep learning, a major difficulty in identifying suicidality and its risk factors in clinical notes is the lack of training samples given the small number of true positive instances among the number of patients screened. This paper describes a novel methodology that identifies suicidality in clinical notes by addressing this data sparsity issue through zero-shot learning. Our general aim was to develop a tool that leveraged zero-shot learning to effectively identify suicidality documentation in all types of clinical notes.

Methods

US Veterans Affairs clinical notes served as data. The training data set label was determined using diagnostic codes of suicide attempt and self-harm. We used a base string associated with the target label of suicidality to provide auxiliary information by narrowing the positive training cases to those containing the base string. We trained a deep neural network by mapping the training documents' contents to a semantic space. For comparison, we trained another deep neural network using the identical training data set labels, and bag-of-words features.

Results

The zero-shot learning model outperformed the baseline model in terms of area under the curve, sensitivity, specificity, and positive predictive value at multiple probability thresholds. In applying a 0.90 probability threshold, the methodology identified notes documenting suicidality but not associated with a relevant ICD-10-CM code, with 94% accuracy.

Conclusion

This method can effectively identify suicidality without manual annotation.

Multi-view graph representation with similarity diffusion for general zero-shot learning

Zero-shot learning (ZSL) aims to predict unseen classes without using samples of these classes in model training. The ZSL has been widely used in many knowledge-based models and applications to predict various parameters, including categories, subjects, and anomalies, in different domains. Nonetheless, most existing ZSL methods require the pre-defined semantics or attributes of particular data environments. Therefore, these methods are difficult to be applied to general data environments, such as ImageNet and other real-world datasets and applications. Recent research has tried to use open knowledge to enhance the ZSL methods to adapt it to an open data environment. However, the performance of these methods is relatively low, namely the accuracy is normally below 10%, which is due to the inadequate semantics that can be used from open knowledge. Moreover, the latest methods suffer from a significant "semantic gap" problem between the generated features of unseen classes and the real features of seen classes. To this end, this paper proposes a multi-view graph representation with a similarity diffusion model, applying the ZSL tasks to general data environments. This model applies a multi-view graph to enhance the semantics fully and proposes an innovative diffusion method to augment the graph representation. In addition, a feature diffusion method is proposed to augment the multi-view graph representation and bridge the semantic gap to realize zero-shot predicting. The results of numerous experiments in general data environments and on benchmark datasets show that the proposed method can achieve new state-of-the-art results in the field of general zero-shot learning. Furthermore, seven ablation studies analyze the effects of the settings and different modules of the proposed method on its performance in detail and prove the effectiveness of each module.

Zero-Shot Image Classification Method Based on Attention Mechanism and Semantic Information Fusion

The zero-shot image classification (ZSIC) is designed to solve the classification problem when the sample is very small, or the category is missing. A common method is to use attribute or word vectors as a priori category features (auxiliary information) and complete the domain transfer from training of seen classes to recognition of unseen classes by building a mapping between image features and a priori category features. However, feature extraction of the whole image lacks discrimination, and the amount of information of single attribute features or word vector features of categories is insufficient, which makes the matching degree between image features and prior class features not high and affects the accuracy of the ZSIC model. To this end, a spatial attention mechanism is designed, and an image feature extraction module based on this attention mechanism is constructed to screen critical features with discrimination. A semantic information fusion method based on matrix decomposition is proposed, which first decomposes the attribute features and then fuses them with the extracted word vector features of a dataset to achieve information expansion. Through the above two improvement measures, the classification accuracy of the ZSIC model for unseen images is improved. The experimental results on public datasets verify the effect and superiority of the proposed methods.