Deep-Learning-Based Multimodal Emotion Classification for Music Videos

A dataset for multimodal music information retrieval of Sotho-Tswana musical videos

The existence of diverse traditional machine learning and deep learning models designed for various multimodal music information retrieval (MIR) applications, such as multimodal music sentiment analysis, genre classification, recommender systems, and emotion recognition, renders the machine learning and deep learning models indispensable for the MIR tasks. However, solving these tasks in a data-driven manner depends on the availability of high-quality benchmark datasets. Hence, the necessity for datasets tailored for multimodal music information retrieval applications is paramount. While a handful of multimodal datasets exist for distinct music information retrieval applications, they are not available in low-resourced languages, like Sotho-Tswana languages. In response to this gap, we introduce a novel multimodal music information retrieval dataset for various music information retrieval applications. This dataset centres on Sotho-Tswana musical videos, encompassing both textual, visual, and audio modalities specific to Sotho-Tswana musical content. The musical videos were downloaded from YouTube, but Python programs were written to process the musical videos and extract relevant spectral-based acoustic features, using different Python libraries. Annotation of the dataset was done manually by native speakers of Sotho-Tswana languages, who understand the culture and traditions of the Sotho-Tswana people. It is distinctive as, to our knowledge, no such dataset has been established until now.

Multimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perception

The field of multimodal robotic musical performing arts has garnered significant interest due to its innovative potential. Conventional robots face limitations in understanding emotions and artistic expression in musical performances. Therefore, this paper explores the application of multimodal robots that integrate visual and auditory perception to enhance the quality and artistic expression in music performance. Our approach involves integrating GRU (Gated Recurrent Unit) and GoogLeNet models for sentiment analysis. The GRU model processes audio data and captures the temporal dynamics of musical elements, including long-term dependencies, to extract emotional information. The GoogLeNet model excels in image processing, extracting complex visual details and aesthetic features. This synergy deepens the understanding of musical and visual elements, aiming to produce more emotionally resonant and interactive robot performances. Experimental results demonstrate the effectiveness of our approach, showing significant improvements in music performance by multimodal robots. These robots, equipped with our method, deliver high-quality, artistic performances that effectively evoke emotional engagement from the audience. Multimodal robots that merge audio-visual perception in music performance enrich the art form and offer diverse human-machine interactions. This research demonstrates the potential of multimodal robots in music performance, promoting the integration of technology and art. It opens new realms in performing arts and human-robot interactions, offering a unique and innovative experience. Our findings provide valuable insights for the development of multimodal robots in the performing arts sector.

The influence of music teaching appreciation on the mental health of college students based on multimedia data analysis

The mental health problem of college students has gradually become the focus of people's attention. The music appreciation course in university is a very effective approach of psychological counseling, and it is urgent to explore the role of music appreciation in psychological adjustment. Therefore, we propose an emotion classification model based on particle swarm optimization (PSO) to study the effect of inter active music appreciation teaching on the mental health of college students. We first extract musical features as input. Then, the extracted music appreciation features generate subtitles of music information. Finally, we weight the above features, input them into the network, modify the network through particle swarm optimization, and output the emotional class of music. The experimental results show that the music emotion classification model has a high classification accuracy of 82.6%, and can obtain the emotional categories included in interactive music appreciation, which is helpful to guide the mental health of college students in music appreciation teaching.

MERP: A Music Dataset with Emotion Ratings and Raters' Profile Information

Music is capable of conveying many emotions. The level and type of emotion of the music perceived by a listener, however, is highly subjective. In this study, we present the Music Emotion Recognition with Profile information dataset (MERP). This database was collected through Amazon Mechanical Turk (MTurk) and features dynamical valence and arousal ratings of 54 selected full-length songs. The dataset contains music features, as well as user profile information of the annotators. The songs were selected from the Free Music Archive using an innovative method (a Triple Neural Network with the OpenSmile toolkit) to identify 50 songs with the most distinctive emotions. Specifically, the songs were chosen to fully cover the four quadrants of the valence-arousal space. Four additional songs were selected from the DEAM dataset to act as a benchmark in this study and filter out low quality ratings. A total of 452 participants participated in annotating the dataset, with 277 participants remaining after thoroughly cleaning the dataset. Their demographic information, listening preferences, and musical background were recorded. We offer an extensive analysis of the resulting dataset, together with a baseline emotion prediction model based on a fully connected model and an LSTM model, for our newly proposed MERP dataset.

Emotion Classification from Speech and Text in Videos Using a Multimodal Approach

Emotion classification is a research area in which there has been very intensive literature production concerning natural language processing, multimedia data, semantic knowledge discovery, social network mining, and text and multimedia data mining. This paper addresses the issue of emotion classification and proposes a method for classifying the emotions expressed in multimodal data extracted from videos. The proposed method models multimodal data as a sequence of features extracted from facial expressions, speech, gestures, and text, using a linguistic approach. Each sequence of multimodal data is correctly associated with the emotion by a method that models each emotion using a hidden Markov model. The trained model is evaluated on samples of multimodal sentences associated with seven basic emotions. The experimental results demonstrate a good classification rate for emotions.

Multimodal Sentiment Analysis of Online Product Information Based on Text Mining Under the Influence of Social Media

Currently, with the dramatic increase in social media users and the greater variety of online product information, manual processing of this information is time-consuming and labour-intensive. Therefore, based on the text mining of online information, this paper analyzes the text representation method of online information, discusses the long short-term memory network, and constructs an interactive attention graph convolutional network (IAGCN) model based on graph convolutional neural network (GCNN) and attention mechanism to study the multimodal sentiment analysis (MSA) of online product information. The results show that the IAGCN model improves the accuracy by 4.78% and the F1 value by 29.25% compared with the pure interactive attention network. Meanwhile, it is found that the performance of the model is optimal when the GCNN is two layers and uses syntactic position attention. This research has important practical significance for MSA of online product information in social media.

Research on Music Style Classification Based on Deep Learning

Music style is one of the important labels for music classification, and the current music style classification methods extract features such as rhythm and timbre of music and use classifiers to achieve classification. The classification accuracy is not only affected by the classifier but also limited by the effect of music feature extraction, which leads to poor classification accuracy and stability. In response to the abovementioned defects, a deep-learning-based music style classification method will be studied. The music signal is framed using filters and Hamming windows, and the MFCC coefficient features of music are extracted by discrete Fourier transform. A convolutional recurrent neural network structure combining CNN and RNN is designed and trained to determine the parameters to achieve music style classification. Analysis of the simulation experimental data shows that the classification accuracy of the studied classification method is at least 93.3%, and the classification time overhead is significantly reduced, the classification results are stable, and the results are reliable.

The Multimodal Sentiment Analysis of Online Product Marketing Information Using Text Mining and Big Data

Currently, the internet is increasingly popular. More people are used to sharing their feelings about various things on the internet. Online product marketing information is also growing. How to mine the required information from the massive information with the support of big data technology has become a big problem. Thereby, based on the text mining of online product marketing information, this work discusses the text preprocessing methods and the temporal convolution network (TCN) based on a convolutional neural network (CNN). Moreover, on this basis, multimodal attention mechanism (AM) and cross-modal transformer structure are added to build a TCN based on AM (AM-TCN) model to analyze the multimodal emotion of online product marketing information. The results show that the accuracy of the AM-TCN model is 2.88% higher than that of the TCN model alone, and F1 is 3.47% higher. Moreover, the accuracy rate of the AM-TCN is 1.22% higher than that of the next highest recurrent multistage fusion network, and the F1 value is 0.95% higher.

Classifying Emotions in Film Music—A Deep Learning Approach

The paper presents an application for automatically classifying emotions in film music. A model of emotions is proposed, which is also associated with colors. The model created has nine emotional states, to which colors are assigned according to the color theory in film. Subjective tests are carried out to check the correctness of the assumptions behind the adopted emotion model. For that purpose, a statistical analysis of the subjective test results is performed. The application employs a deep convolutional neural network (CNN), which classifies emotions based on 30 s excerpts of music works presented to the CNN input using mel-spectrograms. Examples of classification results of the selected neural networks used to create the system are shown.

Music video emotion classification using slow-fast audio-video network and unsupervised feature representation

Affective computing has suffered by the precise annotation because the emotions are highly subjective and vague. The music video emotion is complex due to the diverse textual, acoustic, and visual information which can take the form of lyrics, singer voice, sounds from the different instruments, and visual representations. This can be one reason why there has been a limited study in this domain and no standard dataset has been produced before now. In this study, we proposed an unsupervised method for music video emotion analysis using music video contents on the Internet. We also produced a labelled dataset and compared the supervised and unsupervised methods for emotion classification. The music and video information are processed through a multimodal architecture with audio-video information exchange and boosting method. The general 2D and 3D convolution networks compared with the slow-fast network with filter and channel separable convolution in multimodal architecture. Several supervised and unsupervised networks were trained in an end-to-end manner and results were evaluated using various evaluation metrics. The proposed method used a large dataset for unsupervised emotion classification and interpreted the results quantitatively and qualitatively in the music video that had never been applied in the past. The result shows a large increment in classification score using unsupervised features and information sharing techniques on audio and video network. Our best classifier attained 77% accuracy, an f1-score of 0.77, and an area under the curve score of 0.94 with minimum computational cost.

Subgroup Preference Neural Network

Subgroup label ranking aims to rank groups of labels using a single ranking model, is a new problem faced in preference learning. This paper introduces the Subgroup Preference Neural Network (SGPNN) that combines multiple networks have different activation function, learning rate, and output layer into one artificial neural network (ANN) to discover the hidden relation between the subgroups' multi-labels. The SGPNN is a feedforward (FF), partially connected network that has a single middle layer and uses stairstep (SS) multi-valued activation function to enhance the prediction's probability and accelerate the ranking convergence. The novel structure of the proposed SGPNN consists of a multi-activation function neuron (MAFN) in the middle layer to rank each subgroup independently. The SGPNN uses gradient ascent to maximize the Spearman ranking correlation between the groups of labels. Each label is represented by an output neuron that has a single SS function. The proposed SGPNN using conjoint dataset outperforms the other label ranking methods which uses each dataset individually. The proposed SGPNN achieves an average accuracy of 91.4% using the conjoint dataset compared to supervised clustering, decision tree, multilayer perceptron label ranking and label ranking forests that achieve an average accuracy of 60%, 84.8%, 69.2% and 73%, respectively, using the individual dataset.