A Comparative Analysis of Discrete Entropy Estimators for Large-Alphabet Problems

This paper presents a comparative study of entropy estimation in a large-alphabet regime. A variety of entropy estimators have been proposed over the years, where each estimator is designed for a different setup with its own strengths and caveats. As a consequence, no estimator is known to be universally better than the others. This work addresses this gap by comparing twenty-one entropy estimators in the studied regime, starting with the simplest plug-in estimator and leading up to the most recent neural network-based and polynomial approximate estimators. Our findings show that the estimators' performance highly depends on the underlying distribution. Specifically, we distinguish between three types of distributions, ranging from uniform to degenerate distributions. For each class of distribution, we recommend the most suitable estimator. Further, we propose a sample-dependent approach, which again considers three classes of distribution, and report the top-performing estimators in each class. This approach provides a data-dependent framework for choosing the desired estimator in practical setups.

Neural Joint Entropy Estimation

Estimating the entropy of a discrete random variable is a fundamental problem in information theory and related fields. This problem has many applications in various domains, including machine learning, statistics, and data compression. Over the years, a variety of estimation schemes have been suggested. However, despite significant progress, most methods still struggle when the sample is small, compared to the variable's alphabet size. In this work, we introduce a practical solution to this problem, which extends the work of McAllester and Statos. The proposed scheme uses the generalization abilities of cross-entropy estimation in deep neural networks (DNNs) to introduce improved entropy estimation accuracy. Furthermore, we introduce a family of estimators for related information-theoretic measures, such as conditional entropy and mutual information (MI). We show that these estimators are strongly consistent and demonstrate their performance in a variety of use cases. First, we consider large alphabet entropy estimation. Then, we extend the scope to MI estimation. Next, we apply the proposed scheme to conditional MI estimation, as we focus on independence testing tasks. Finally, we study a transfer entropy (TE) estimation problem. The proposed estimators demonstrate improved performance compared to existing methods in all of these setups.

Cross-validated tree-based models for multi-target learning

Multi-target learning (MTL) is a popular machine learning technique which considers simultaneous prediction of multiple targets. MTL schemes utilize a variety of methods, from traditional linear models to more contemporary deep neural networks. In this work we introduce a novel, highly interpretable, tree-based MTL scheme which exploits the correlation between the targets to obtain improved prediction accuracy. Our suggested scheme applies cross-validated splitting criterion to identify correlated targets at every node of the tree. This allows us to benefit from the correlation among the targets while avoiding overfitting. We demonstrate the performance of our proposed scheme in a variety of synthetic and real-world experiments, showing a significant improvement over alternative methods. An implementation of the proposed method is publicly available at the first author's webpage.

Safety of the fourth COVID-19 BNT162b2 mRNA (second booster) vaccine: a prospective and retrospective cohort study

BACKGROUND

The effectiveness of the second BNT162b2 (Pfizer-BioNTech) mRNA COVID-19 booster vaccine dose (ie, fourth inoculation) is well established, but its safety has yet to be fully understood. The absence of sufficient vaccine safety information is one of the key contributors to vaccine hesitancy. In this study, we aimed to evaluate the safety profile of the second BNT162b2 mRNA COVID-19 booster vaccine using data from a retrospective cohort and a prospective cohort.

METHODS

To evaluate the safety profile of the second booster vaccine, we analysed its short-term effects and compared them to those of the first booster by using data from, first, a retrospective cohort of 250 000 random members of the second-largest health-care organisation in Israel (Maccabi Healthcare Services) and, second, a prospective cohort (the PerMed study) of 4698 participants from all across Israel. Individuals who were aged 18 years or older who received the second BNT162b2 mRNA COVID-19 vaccine booster during the vaccination campaign, from Dec 30, 2021, to July 22, 2022, were eligible for inclusion in the retrospective cohort analysis. To be included in the PerMed study, participants needed to be 18 years or older, members of Maccabi Healthcare Services at the time of enrolment, using their own smartphone, and be able to give informed consent by themselves. Participants from the prospective cohort received smartwatches, downloaded a dedicated mobile application, and granted access to their medical records. The smartwatches continuously monitored several physiological measures, including heart rate. For analysis of the prospective cohort data, we used the Kruskal-Wallis test to compare heart rate levels observed before and after vaccination. The mobile application collected daily self-reported questionnaires on local and systemic reactions. Medical records of the retrospective cohort were accessed to examine the occurrence of 25 potential adverse events, and we evaluated the risk differences between 42 days in the periods before and after vaccination in a pairwise method using non-parametric percentile bootstrap.

FINDINGS

The retrospective cohort included 94 169 participants who received the first booster and 17 814 who received the second booster. Comparing the 42 days before and after vaccination, the second booster was not associated with any of the 25 adverse events investigated, including myocardial infarction (risk difference, 2·25 events per 10 000 individuals [95% CI -3·93 to 8·98]) and Bell's Palsy (-1·68 events [-5·61 to 2·25]). None of the individuals was diagnosed with myocarditis or pericarditis following vaccination with the second booster. The prospective cohort included 1785 participants who received the first booster and 699 who received the second booster. We found no significant differences after inoculation with the first booster compared with the second booster (heart rate: day 2 [p=0·3], day 6 [p=0·89]; extent of self-reported reactions [p=0·06]). We found a significant increase in mean heart rate relative to that observed during the week before vaccination (baseline) levels during the first 3 days following the second booster (p<0·0001), peaking on day 2 (mean difference of 1·61 bpm [1·07 to 2·16] compared with baseline). Mean heart rate values returned to baseline levels by day 6 (-0·055 bpm [-0·56 to 0·45] compared with baseline).

INTERPRETATION

Both our retrospective and prospective analyses support the safety of the second booster, with our findings reflecting physicians' diagnoses, patients' objective physiological measures, and patients' subjective reactions. We believe this study provides safety assurances to the global population who are eligible to receive an additional COVID-19 booster inoculation. These assurances can help increase the number of high-risk individuals who opt to receive this booster vaccine and thereby prevent severe outcomes associated with COVID-19.

FUNDING

European Research Council (ERC).

Detecting Non-Overlapping Signals with Dynamic Programming

This paper studies the classical problem of detecting the locations of signal occurrences in a one-dimensional noisy measurement. Assuming the signal occurrences do not overlap, we formulate the detection task as a constrained likelihood optimization problem and design a computationally efficient dynamic program that attains its optimal solution. Our proposed framework is scalable, simple to implement, and robust to model uncertainties. We show by extensive numerical experiments that our algorithm accurately estimates the locations in dense and noisy environments, and outperforms alternative methods.

Robust Universal Inference

Learning and making inference from a finite set of samples are among the fundamental problems in science. In most popular applications, the paradigmatic approach is to seek a model that best explains the data. This approach has many desirable properties when the number of samples is large. However, in many practical setups, data acquisition is costly and only a limited number of samples is available. In this work, we study an alternative approach for this challenging setup. Our framework suggests that the role of the train-set is not to provide a single estimated model, which may be inaccurate due to the limited number of samples. Instead, we define a class of “reasonable” models. Then, the worst-case performance in the class is controlled by a minimax estimator with respect to it. Further, we introduce a robust estimation scheme that provides minimax guarantees, also for the case where the true model is not a member of the model class. Our results draw important connections to universal prediction, the redundancy-capacity theorem, and channel capacity theory. We demonstrate our suggested scheme in different setups, showing a significant improvement in worst-case performance over currently known alternatives.

Nonlinear Canonical Correlation Analysis:A Compressed Representation Approach

Canonical Correlation Analysis (CCA) is a linear representation learning method that seeks maximally correlated variables in multi-view data. Nonlinear CCA extends this notion to a broader family of transformations, which are more powerful in many real-world applications. Given the joint probability, the Alternating Conditional Expectation (ACE) algorithm provides an optimal solution to the nonlinear CCA problem. However, it suffers from limited performance and an increasing computational burden when only a finite number of samples is available. In this work, we introduce an information-theoretic compressed representation framework for the nonlinear CCA problem (CRCCA), which extends the classical ACE approach. Our suggested framework seeks compact representations of the data that allow a maximal level of correlation. This way, we control the trade-off between the flexibility and the complexity of the model. CRCCA provides theoretical bounds and optimality conditions, as we establish fundamental connections to rate-distortion theory, the information bottleneck and remote source coding. In addition, it allows a soft dimensionality reduction, as the compression level is determined by the mutual information between the original noisy data and the extracted signals. Finally, we introduce a simple implementation of the CRCCA framework, based on lattice quantization.

Cross-Validated Variable Selection in Tree-Based Methods Improves Predictive Performance

Recursive partitioning methods producing tree-like models are a long standing staple of predictive modeling. However, a fundamental flaw in the partitioning (or splitting) rule of commonly used tree building methods precludes them from treating different types of variables equally. This most clearly manifests in these methods' inability to properly utilize categorical variables with a large number of categories, which are ubiquitous in the new age of big data. We propose a framework to splitting using leave-one-out (LOO) cross validation (CV) for selecting the splitting variable, then performing a regular split (in our case, following CART's approach) for the selected variable. The most important consequence of our approach is that categorical variables with many categories can be safely used in tree building and are only chosen if they contribute to predictive power. We demonstrate in extensive simulation and real data analysis that our splitting approach significantly improves the performance of both single tree models and ensemble methods that utilize trees. Importantly, we design an algorithm for LOO splitting variable selection which under reasonable assumptions does not substantially increase the overall computational complexity compared to CART for two-class classification.

Isotonic Modeling with Non-Differentiable Loss Functions with Application to Lasso Regularization

In this paper we present an algorithmic approach for fitting isotonic models under convex, yet non-differentiable, loss functions. It is a generalization of the greedy non-regret approach proposed by Luss and Rosset (2014) for differentiable loss functions, taking into account the sub-gradiental extensions required. We prove that our suggested algorithm solves the isotonic modeling problem while maintaining favorable computational and statistical properties. As our suggested algorithm may be used for any non-differentiable loss function, we focus our interest on isotonic modeling for either regression or two-class classification with appropriate log-likelihood loss and lasso penalty on the fitted values. This combination allows us to maintain the non-parametric nature of isotonic modeling, while controlling model complexity through regularization. We demonstrate the efficiency and usefulness of this approach on both synthetic and real world data. An implementation of our suggested solution is publicly available from the first author's website (https://sites.google.com/site/amichaipainsky/software).