Towards a universal mechanism for successful deep learning

Recently, the underlying mechanism for successful deep learning (DL) was presented based on a quantitative method that measures the quality of a single filter in each layer of a DL model, particularly VGG-16 trained on CIFAR-10. This method exemplifies that each filter identifies small clusters of possible output labels, with additional noise selected as labels outside the clusters. This feature is progressively sharpened with each layer, resulting in an enhanced signal-to-noise ratio (SNR), which leads to an increase in the accuracy of the DL network. In this study, this mechanism is verified for VGG-16 and EfficientNet-B0 trained on the CIFAR-100 and ImageNet datasets, and the main results are as follows. First, the accuracy and SNR progressively increase with the layers. Second, for a given deep architecture, the maximal error rate increases approximately linearly with the number of output labels. Third, similar trends were obtained for dataset labels in the range [3, 1000], thus supporting the universality of this mechanism. Understanding the performance of a single filter and its dominating features paves the way to highly dilute the deep architecture without affecting its overall accuracy, and this can be achieved by applying the filter's cluster connections (AFCC).

Observing the universal screening of a Kondo impurity

The Kondo effect, deriving from a local magnetic impurity mediating electron-electron interactions, constitutes a flourishing basis for understanding a large variety of intricate many-body problems. Its experimental implementation in tunable circuits has made possible important advances through well-controlled investigations. However, these have mostly concerned transport properties, whereas thermodynamic observations - notably the fundamental measurement of the spin of the Kondo impurity - remain elusive in test-bed circuits. Here, with a novel combination of a 'charge' Kondo circuit with a charge sensor, we directly observe the state of the impurity and its progressive screening. We establish the universal renormalization flow from a single free spin to a screened singlet, the associated reduction in the magnetization, and the relationship between scaling Kondo temperature and microscopic parameters. In our device, a Kondo pseudospin is realized by two degenerate charge states of a metallic island, which we measure with a non-invasive, capacitively coupled charge sensor. Such pseudospin probe of an engineered Kondo system opens the way to the thermodynamic investigation of many exotic quantum states, including the clear observation of Majorana zero modes through their fractional entropy.

Enhancing the accuracies by performing pooling decisions adjacent to the output layer

Learning classification tasks of [Formula: see text] inputs typically consist of [Formula: see text]) max-pooling (MP) operators along the entire feedforward deep architecture. Here we show, using the CIFAR-10 database, that pooling decisions adjacent to the last convolutional layer significantly enhance accuracies. In particular, average accuracies of the advanced-VGG with [Formula: see text] layers (A-VGGm) architectures are 0.936, 0.940, 0.954, 0.955, and 0.955 for m = 6, 8, 14, 13, and 16, respectively. The results indicate A-VGG8's accuracy is superior to VGG16's, and that the accuracies of A-VGG13 and A-VGG16 are equal, and comparable to that of Wide-ResNet16. In addition, replacing the three fully connected (FC) layers with one FC layer, A-VGG6 and A-VGG14, or with several linear activation FC layers, yielded similar accuracies. These significantly enhanced accuracies stem from training the most influential input-output routes, in comparison to the inferior routes selected following multiple MP decisions along the deep architecture. In addition, accuracies are sensitive to the order of the non-commutative MP and average pooling operators adjacent to the output layer, varying the number and location of training routes. The results call for the reexamination of previously proposed deep architectures and their accuracies by utilizing the proposed pooling strategy adjacent to the output layer.

Efficient shallow learning as an alternative to deep learning

The realization of complex classification tasks requires training of deep learning (DL) architectures consisting of tens or even hundreds of convolutional and fully connected hidden layers, which is far from the reality of the human brain. According to the DL rationale, the first convolutional layer reveals localized patterns in the input and large-scale patterns in the following layers, until it reliably characterizes a class of inputs. Here, we demonstrate that with a fixed ratio between the depths of the first and second convolutional layers, the error rates of the generalized shallow LeNet architecture, consisting of only five layers, decay as a power law with the number of filters in the first convolutional layer. The extrapolation of this power law indicates that the generalized LeNet can achieve small error rates that were previously obtained for the CIFAR-10 database using DL architectures. A power law with a similar exponent also characterizes the generalized VGG-16 architecture. However, this results in a significantly increased number of operations required to achieve a given error rate with respect to LeNet. This power law phenomenon governs various generalized LeNet and VGG-16 architectures, hinting at its universal behavior and suggesting a quantitative hierarchical time-space complexity among machine learning architectures. Additionally, the conservation law along the convolutional layers, which is the square-root of their size times their depth, is found to asymptotically minimize error rates. The efficient shallow learning that is demonstrated in this study calls for further quantitative examination using various databases and architectures and its accelerated implementation using future dedicated hardware developments.

A Secure Base for Entrepreneurship: Attachment Orientations and Entrepreneurial Tendencies

Entrepreneurship catalyzes economic growth; it generates jobs, advances the economy and solves global challenges. Hence, it is crucial to understand the factors contributing to entrepreneurship and entrepreneurs' development. While many studies have investigated intrapersonal factors for entrepreneurial tendencies, the present study focuses on a critical yet often overlooked interpersonal aspect: attachment orientations. Specifically, this article examines the relationship between adult attachment orientations and entrepreneurial tendencies. Three studies across three countries (Israel, the UK, and Singapore) indicated that an anxious attachment orientation in close relationships is negatively associated with enterprising tendencies. In Israel (Study 1) and Singapore (Study 2), avoidant attachment in close relationships was also negatively correlated to such tendencies. Overall, the more people feel secure in close relationships (lower scores on attachment anxiety or avoidance), the higher their enterprising tendencies. Limitations and future research suggestions are discussed.

Brain inspired neuronal silencing mechanism to enable reliable sequence identification

Real-time sequence identification is a core use-case of artificial neural networks (ANNs), ranging from recognizing temporal events to identifying verification codes. Existing methods apply recurrent neural networks, which suffer from training difficulties; however, performing this function without feedback loops remains a challenge. Here, we present an experimental neuronal long-term plasticity mechanism for high-precision feedforward sequence identification networks (ID-nets) without feedback loops, wherein input objects have a given order and timing. This mechanism temporarily silences neurons following their recent spiking activity. Therefore, transitory objects act on different dynamically created feedforward sub-networks. ID-nets are demonstrated to reliably identify 10 handwritten digit sequences, and are generalized to deep convolutional ANNs with continuous activation nodes trained on image sequences. Counterintuitively, their classification performance, even with a limited number of training examples, is high for sequences but low for individual objects. ID-nets are also implemented for writer-dependent recognition, and suggested as a cryptographic tool for encrypted authentication. The presented mechanism opens new horizons for advanced ANN algorithms.

Dual energy CT in acute appendicitis: value of low mono-energy

OBJECTIVES

To assess the potential role of low monoenergetic images in the evaluation of acute appendicitis.

METHODS

A retrospective study of 42 patients with pathology proven acute appendicitis underwent contrast-enhanced-CT conducted on a single-source-DECT before surgery. Attenuation, SNR, and CNR were calculated on both monoenergetic and conventional images and compared to 24 abdominal CT-scans with normal appendix. Representative conventional and monoenergetic images were randomized and presented side-by-side to three abdominal radiologists to determine preferred images for detecting inflammation. Additionally, six individual acute inflammatory characteristics were graded on a 1-5 scale to determine factors contributing to differences between conventional and monoenergetic images by 2 abdominal radiologists. Paired t-tests, Wilcoxon and McNemar tests, and intra-observer error statistics were performed.

RESULTS

For the inflamed appendixes monoenergetic images had overall increased attenuation (average ratio 1.7; P < 0.05), signal-to-noise-ratio (6.7 ± 3.1 vs 4.2 ± 1.6; P < 0.001) and contrast-to-noise-ratio (12.1 ± 3 vs 9 ± 2.1; P < 0.001). Moreover, this increase was not found in normal appendixes (P < 0.001 vs p = 0.28-0.44). Subjectively, radiologists showed significant preferences towards monoenergetic images (P < 0.001), with inter-reader agreement of 0.84. Two parameters, diffuse bowel wall and mucosal enhancement, received significantly higher scores on monoenergetic images (average 4.3 vs. 3.0; P < 0.001 and 2.8 vs. 2.3 P < 0.03 respectively, with interobserver agreements of 62% and 52%).

CONCLUSION

Increased bowel wall conspicuity from enhanced attenuation, SNR, and CNR on low monenergetic CT images results in a significant preference by radiologists for these images when assessing acute inflamed appendixes. Thus, close inspection of low monoenergetic images may improve the visualization of acute inflammatory bowel processes.

Brain experiments imply adaptation mechanisms which outperform common AI learning algorithms

Attempting to imitate the brain's functionalities, researchers have bridged between neuroscience and artificial intelligence for decades; however, experimental neuroscience has not directly advanced the field of machine learning (ML). Here, using neuronal cultures, we demonstrate that increased training frequency accelerates the neuronal adaptation processes. This mechanism was implemented on artificial neural networks, where a local learning step-size increases for coherent consecutive learning steps, and tested on a simple dataset of handwritten digits, MNIST. Based on our on-line learning results with a few handwriting examples, success rates for brain-inspired algorithms substantially outperform the commonly used ML algorithms. We speculate this emerging bridge from slow brain function to ML will promote ultrafast decision making under limited examples, which is the reality in many aspects of human activity, robotic control, and network optimization.

Probing the Kondo density of states in a three-terminal quantum ring

We have measured the Kondo effect in a quantum ring connected to three terminals. In this configuration nonlinear transport measurements allow us to check which lead contributes to the Kondo density of states (DOS) and which does not. The ring geometry allows a fine-tuning of the coupling to each lead through the Aharonov-Bohm effect via application of a magnetic field. When the ring is connected to two strongly and one weakly coupled leads, conductance through the weakly coupled lead provides a direct measurement of the DOS in the Kondo regime. By applying a bias between the two strongly coupled leads, we demonstrate directly the splitting of the out-of-equilibrium Kondo DOS.

Early screening for fetal cardiac anomalies by transvaginal echocardiography in an unselected population: the role of operator experience

OBJECTIVES

To examine the accuracy of early transvaginal fetal echocardiography performed in an unselected population by operators with different levels of experience and to compare the results with those obtained from a referral population.

DESIGN

Prospective study.

METHODS

A series of 4785 unselected fetuses and 221 referred fetuses were screened at 13-15 weeks' gestational age by transvaginal echocardiography. For each fetus, visualization of the four-chamber view plus the origin of the great arteries was attempted. Color Doppler imaging was only performed in cases of cardiac malformations already identified by two-dimensional echocardiography. The scans were performed by seven operators with different levels of experience. Reliability was assessed by conventional transabdominal echocardiography at 20-22 weeks, by postnatal follow-up in the first 3 months of life, and/or by autopsy in all cases of termination or fetal death.

RESULTS

The rate of complete visualization (four-chamber view plus great arteries) was 47.5% in the unselected population, and 76.9% in the referral population. There were four (0.08%) true positives among the unselected fetuses, and five (2.3%) among referrals, mostly with enlarged nuchal translucency or other malformations. Among the unselected fetuses, nine false negatives were detected by transabdominal echocardiography. Improvement in the operators' ability to recognize cardiac anomalies in unselected population was disappointing and was influenced more by individual approach than by the number of examinations performed.

CONCLUSIONS

Early screening for cardiac anomalies among unselected fetuses is ill-advisable. The usefulness of an early approach is confirmed in high risk fetuses or in the presence of enlarged nuchal translucency when performed by expert operators.

Fetal heart screening in low-risk pregnancies

The aim of this study was to assess whether a screening program for fetal cardiac malformations is justified in a low-risk population, and which factors influence its accuracy. The fetal heart was evaluated in 7024 pregnant women at 20-22 weeks, and evaluation was repeated at a more advanced gestational age in 9% of cases. Cardiological follow-up was continued postnatally until 2 years of age. The overall prevalence of cardiac anomaly was 0.93%. The incidences of major and minor defects were 0.44% and 0.48%, respectively. There were 23 true positives (0.33%): in 20 cases, the diagnosis was made in the second trimester, and 13 women (65%) chose termination of pregnancy. Seventeen of the 20 cases identified in the second trimester were serious malformations. There were 42 false negatives (0.60%). Of these, 12 had signs of cardiac dysfunction at birth or within the 1st month of life, and three of them died as a result of their cardiac anomaly. There were eight false positives (0.11%), all of a minor type. Six abnormal karyotypes, out of a total of 21 performed in the true-positive group (28.5%), were found. In addition, five of the 42 newborns in the false-negative group had trisomy 21. The overall sensitivity was 35.4%, and 61.3% for major defects. The accuracy in two distinct periods was estimated because the level of experience of the operators was different: sensitivity was 45.2% in period 1 (1986-88) (77.8% for major defects) and 26.5% in period 2 (1989-92) (52.9% for major defects). We conclude that a fetal heart screening program in the obstetric population is justified. It defines a high-risk group for karyotyping, allows planning of delivery in a tertiary center or the choice of terminating the pregnancy for the parents and appears to have a positive cost-benefit ratio. A crucial factor is the level of training and experience of the operators, who need specific teaching support.

[The diagnostic capacities of transvaginal echography and hysteroscopy in the characterization of endometrial pathology]

The diagnostic accuracy of transvaginal sonography (TVS) and hysteroscopy in the assessment of endometrial pathology was studied comparing retrospectively both methods with the results of histologic findings after dilatation and curettage (D&C) performed in the last four years on 467 patients, 155 of whom were in postmenopause. Endometrial thickness, tissue texture, myometrial invasion and haemodynamic characteristics were studied with TVS. Uterine cavity, endometrial patterns and superficial vascularization were evaluated by hysteroscopy. For the purpose of this study all histologic findings were subdivided to a) normal (206 cases); b) benign lesions (240 cases); c) atypical hyperplasia or adenocarcinoma (21 cases). In our experience hysteroscopy was superior to TVS in detecting endometrial pathology. Both techniques were more sensitive in detecting premalignant and malignant lesions. Considering endometrial thickness evaluated with TVS as a single parameter in patients in postmenopause, we found that the most sensitive cut-off for defining normality was 3 mm; nevertheless, in the group of patients that had an endometrial thickness equal to or less than 3 mm there were 2 cases of malignancy. Therefore, neither TVS nor hysteroscopy are sufficiently reliable to replace curettage in the diagnosis of endometrial pathology.

[Staging of rectal carcinoma by endorectal ultrasonography]

Staging of rectal carcinoma is essential for its management. 23 patients with rectal lesions were examined by endorectal ultrasonography. Of these 19 had rectal carcinoma and in 4 there was an abscess of the rectal wall, previously diagnosed by sigmoidoscopic biopsy. 10 of those with carcinoma were examined by computerized tomography as well, and 10 were operated on. In most cases the staging of rectal carcinoma by endorectal ultrasonography was accurate and compared well with the operative histological results. We conclude that endorectal ultrasonography is the method of choice for staging of rectal carcinoma prior to treatment.

Retroperitoneal lymphadenopathy in familial Mediterranean fever

Peripheral lymphadenopathy is rarely observed, whereas mesenteric lymphadenopathy is found occasionally on laparotomies in patients with familial Mediterranean fever (FMF). Retroperitoneal lymphadenopathy was reported only once in an autopsy of a patient with FMF. Our case is the second one, and the first one to be diagnosed during life, by means of abdominal ultrasonography and computerized tomography. In patients with FMF, where lymph node biopsy was done, the pathological finding was non-specific lymphoid hyperplasia.