VC-dimension of univariate decision trees

Deep Convolutional Tables: Deep Learning Without Convolutions

We propose a novel formulation of deep networks that do not use dot-product neurons and rely on a hierarchy of voting tables instead, denoted as convolutional tables (CTs), to enable accelerated CPU-based inference. Convolutional layers are the most time-consuming bottleneck in contemporary deep learning techniques, severely limiting their use in the Internet of Things and CPU-based devices. The proposed CT performs a fern operation at each image location: it encodes the location environment into a binary index and uses the index to retrieve the desired local output from a table. The results of multiple tables are combined to derive the final output. The computational complexity of a CT transformation is independent of the patch (filter) size and grows gracefully with the number of channels, outperforming comparable convolutional layers. It is shown to have a better capacity:compute ratio than dot-product neurons, and that deep CT networks exhibit a universal approximation property similar to neural networks. As the transformation involves computing discrete indices, we derive a soft relaxation and gradient-based approach for training the CT hierarchy. Deep CT networks have been experimentally shown to have accuracy comparable to that of CNNs of similar architectures. In the low-compute regime, they enable an error:speed tradeoff superior to alternative efficient CNN architectures.

Comparison of the Representational Power of Random Forests, Binary Decision Diagrams, and Neural Networks

In this letter, we compare the representational power of random forests, binary decision diagrams (BDDs), and neural networks in terms of the number of nodes. We assume that an axis-aligned function on a single variable is assigned to each edge in random forests and BDDs, and the activation functions of neural networks are sigmoid, rectified linear unit, or similar functions. Based on existing studies, we show that for any random forest, there exists an equivalent depth-3 neural network with a linear number of nodes. We also show that for any BDD with balanced width, there exists an equivalent shallow depth neural network with a polynomial number of nodes. These results suggest that even shallow neural networks have the same or higher representation power than deep random forests and deep BDDs. We also show that in some cases, an exponential number of nodes are required to express a given random forest by a random forest with a much fewer number of trees, which suggests that many trees are required for random forests to represent some specific knowledge efficiently.

Improved Space Forest: A Meta Ensemble Method

The performance of the ensemble algorithms is related with the individual accuracy of the base learners and their results diversity. Individual accuracy of a base learner is directly related to the similarity between the original training set and the base learner's training set. When a modified training set by randomly selecting features/classes/samples is given to the base learners, the diversity is created but the individual accuracy is decreased. From this point of view, different ensemble algorithms can be seen as a selection between having more accurate but less diverse base learners and having more diverse but less accurate base learners. We propose a meta ensemble method named as improved space forest which adds generated and (hopefully) more accurate features to the original features. The new features are obtained from randomly selected original features. When the new features are more distinctive than the original ones, they are selected by the learners. So, the ensemble may have more accurate base learners. However, a different improved space is generated for each learner to create diversity. The proposed method can be used with different ensemble methods. We compared original and improved space versions of bagging, random forest, and rotation forest algorithms. Improved space versions have generally better or comparable results than the original ones. We also present a theoretical framework to analyze the individual accuracies and diversities of the base learners.