Deep Learning for Recommender Systems: A Netflix Case Study

Deep learning has profoundly impacted many areas of machine learning. However, it took a while for its impact to be felt in the field of recommender systems. In this article, we outline some of the challenges encountered and lessons learned in using deep learning for recommender systems at Netflix. We first provide an overview of the various recommendation tasks on the Netflix service. We found that different model architectures excel at different tasks. Even though many deep-learning models can be understood as extensions of existing (simple) recommendation algorithms, we initially did not observe significant improvements in performance over well-tuned non-deep-learning approaches. Only when we added numerous features of heterogeneous types to the input data, deep-learning models did start to shine in our setting. We also observed that deep-learning methods can exacerbate the problem of offline–online metric (mis-)alignment. After addressing these challenges, deep learning has ultimately resulted in large improvements to our recommendations as measured by both offline and online metrics. On the practical side, integrating deep-learning toolboxes in our system has made it faster and easier to implement and experiment with both deep-learning and non-deep-learning approaches for various recommendation tasks. We conclude this article by summarizing our take-aways that may generalize to other applications beyond Netflix.

A simple but highly effective approach to evaluate the prognostic performance of gene expression signatures

BACKGROUND

Highly parallel analysis of gene expression has recently been used to identify gene sets or 'signatures' to improve patient diagnosis and risk stratification. Once a signature is generated, traditional statistical testing is used to evaluate its prognostic performance. However, due to the dimensionality of microarrays, this can lead to false interpretation of these signatures.

PRINCIPAL FINDINGS

A method was developed to test batches of a user-specified number of randomly chosen signatures in patient microarray datasets. The percentage of random generated signatures yielding prognostic value was assessed using ROC analysis by calculating the area under the curve (AUC) in six public available cancer patient microarray datasets. We found that a signature consisting of randomly selected genes has an average 10% chance of reaching significance when assessed in a single dataset, but can range from 1% to ∼40% depending on the dataset in question. Increasing the number of validation datasets markedly reduces this number.

CONCLUSIONS

We have shown that the use of an arbitrary cut-off value for evaluation of signature significance is not suitable for this type of research, but should be defined for each dataset separately. Our method can be used to establish and evaluate signature performance of any derived gene signature in a dataset by comparing its performance to thousands of randomly generated signatures. It will be of most interest for cases where few data are available and testing in multiple datasets is limited.

Robust prognostic value of a knowledge-based proliferation signature across large patient microarray studies spanning different cancer types

Tumour proliferation is one of the main biological phenotypes limiting cure in oncology. Extensive research is being performed to unravel the key players in this process. To exploit the potential of published gene expression data, creation of a signature for proliferation can provide valuable information on tumour status, prognosis and prediction. This will help individualising treatment and should result in better tumour control, and more rapid and cost-effective research and development. From in vitro published microarray studies, two proliferation signatures were compiled. The prognostic value of these signatures was tested in five large clinical microarray data sets. More than 1000 patients with breast, renal or lung cancer were included. One of the signatures (110 genes) had significant prognostic value in all data sets. Stratifying patients in groups resulted in a clear difference in survival (P-values <0.05). Multivariate Cox-regression analyses showed that this signature added substantial value to the clinical factors used for prognosis. Further patient stratification was compared to patient stratification with several well-known published signatures. Contingency tables and Cramer's V statistics indicated that these primarily identify the same patients as the proliferation signature does. The proliferation signature is a strong prognostic factor, with the potential to be converted into a predictive test. Furthermore, evidence is provided that supports the idea that many published signatures track the same biological processes and that proliferation is one of them.

Tumor volume combined with number of positive lymph node stations is a more important prognostic factor than TNM stage for survival of non-small-cell lung cancer patients treated with (chemo)radiotherapy

PURPOSE

The current tumor, node, metastasis system needs refinement to improve its ability to predict survival of patients with non-small-cell lung cancer (NSCLC) treated with (chemo)radiation. In this study, we investigated the prognostic value of tumor volume and N status, assessed by using fluorodeoxyglucose-positron emission tomography (PET).

PATIENTS AND METHODS

Clinical data from 270 consecutive patients with inoperable NSCLC Stages I-IIIB treated radically with (chemo)radiation were collected retrospectively. Diagnostic imaging was performed using either integrated PET-computed tomography or computed tomography and PET separately. The Kaplan-Meier method, as well as Cox regression, was used to analyze data.

RESULTS

Univariate survival analysis showed that number of positive lymph node stations (PLNSs), as well as N stage on PET, was associated significantly with survival. The final multivariate Cox model consisted of number of PLNSs, gross tumor volume (i.e., volume of the primary tumor plus lymph nodes), sex, World Health Organization performance status, and equivalent radiation dose corrected for time; N stage was no longer significant.

CONCLUSIONS

Number of PLNSs, assessed by means of fluorodeoxyglucose-PET, was a significant factor for survival of patients with inoperable NSCLC treated with (chemo)radiation. Risk stratification for this group of patients should be based on gross tumor volume, number of PLNSs, sex, World Health Organization performance status, and equivalent radiation dose corrected for time.