Search

We aim to investigate how closely neural networks (NNs) mimic human thinking. As a step in this direction, we study the behavior of artificial neuron(s) that fire most when the input data score high on some specific emergent concepts. In this paper, we focus on music, where the emergent concepts are those of rhythm, pitch and melody as commonly used by humans. As a black box to pry open, we focus on Google's MusicVAE, a pre-trained NN that handles music tracks by encoding them in terms of 512 latent variables. We show that several hundreds of these latent variables are "irrelevant" in the sense that can be set to zero with minimal impact on the reconstruction accuracy. The remaining few dozens of latent variables can be sorted by order of relevance by comparing their variance. We show that the first few most relevant variables, and only those, correlate highly with dozens of human-defined measures that describe rhythm and pitch in music pieces, thereby efficiently encapsulating many of these human-understandable concepts in a few nonlinear variables. [ABSTRACT FROM AUTHOR]

The realm of music composition, augmented by technological advancements such as computers and related equipment, has undergone significant evolution since the 1970s. In the field algorithmic composition, however, the incorporation of artificial intelligence (AI) in sound generation and combination has been limited. Existing approaches predominantly emphasize sound synthesis techniques, with no music composition systems currently employing Nicolas Slonimsky's theoretical framework. This article introduce NeuralPMG, a computer-assisted polyphonic music generation framework based on a Leap Motion (LM) device, machine learning (ML) algorithms, and brain-computer interface (BCI). ML algorithms are employed to classify user's mental states into two categories: focused and relaxed. Interaction with the LM device allows users to define a melodic pattern, which is elaborated in conjunction with the user's mental state as detected by the BCI to generate polyphonic music. NeuralPMG was evaluated through a user study that involved 19 students of Electronic Music Laboratory at a music conservatory, all of whom are active in the music composition field. The study encompassed a comprehensive analysis of participant interaction with NeuralPMG. The compositions they created during the study were also evaluated by two domain experts who addressed their aesthetics, innovativeness, elaboration level, practical applicability, and emotional impact. The findings indicate that NeuralPMG represents a promising tool, offering a simplified and expedited approach to music composition, and thus represents a valuable contribution to the field of algorithmic music composition. [ABSTRACT FROM AUTHOR]

Dance-driven music generation aims to generate musical pieces conditioned on dance videos. Previous works focus on monophonic or raw audio generation, while the multi-instrument scenario is under-explored. The challenges associated with dance-driven multi-instrument music (MIDI) generation are twofold: (i) lack of a publicly available multi-instrument MIDI and video paired dataset and (ii) the weak correlation between music and video. To tackle these challenges, we have built the first multi-instrument MIDI and dance paired dataset (D2MIDI). Based on this dataset, we introduce a multi-instrument MIDI generation framework (Dance2MIDI) conditioned on dance video. Firstly, to capture the relationship between dance and music, we employ a graph convolutional network to encode the dance motion. This allows us to extract features related to dance movement and dance style. Secondly, to generate a harmonious rhythm, we utilize a transformer model to decode the drum track sequence, leveraging a cross-attention mechanism. Thirdly, we model the task of generating the remaining tracks based on the drum track as a sequence understanding and completion task. A BERT-like model is employed to comprehend the context of the entire music piece through self-supervised learning. We evaluate the music generated by our framework trained on the D2MIDI dataset and demonstrate that our method achieves state-of-the-art performance. [ABSTRACT FROM AUTHOR]

The paper proposes a novel approach called MusicEmo, a transformer-based intelligent system for music emotion generation and recognition. The paper highlights the challenges of creating emotionally resonant music that is musically cohesive and diverse. The proposed approach addresses this challenge by introducing a theme-based conditioning approach, which trains the transformer to manifest the conditioning sequence as thematic material that appears multiple times in the generated result. The MusicEmo architecture incorporates an emotion vector and an LSTM model for creating symbolic musical sequences that are musically coherent and emotionally resonant. The proposed framework outperforms state-of-the-art approaches based on musical consistency and emotional resonance. The transformer-based approach offers a fresh and original way of creating music based on emotions, and it can potentially revolutionize how we create and experience music in the future. [ABSTRACT FROM AUTHOR]

This research paper investigates the transformative potential of computational music analysis and artificial intelligence (AI) in advancing the field of music composition. Specifically, it explores the synergistic roles of pre-analysis and post-analysis techniques in leveraging AIdriven tools to enhance the creative process and quality of musical compositions. The study encompasses a historical overview of music composition, the evolution of computational music analysis, and contemporary AI applications. It delves into pre-analysis, focusing on its role in informing composition, and post-analysis, which evaluates and augments compositions. The paper underscores the significance of these technologies in fostering creativity while addressing challenges and ethical considerations. Through case studies, evaluations, and discussions, this research offers insights into the profound impact of computational music analysis and AI on music composition, paving the way for innovative and inclusive musical expressions. [ABSTRACT FROM AUTHOR]

This work aims to create an ANN-based system for a musical improviser. An artificial improviser of "hearing" music will create a melody. The data supplied to the improviser is MIDItype musical data. This is the harmonic-rhythmic course, the background for improvisation, and the previously made melody notes. The harmonic run is fed into the system as the currently ongoing chord and the time to the next chord, while the supplied few dozen notes performed earlier will indirectly carry information about the entire run and the musical context and style. Improvisation training is carried out to check ANN as a correctlooking musical improvisation device. The improviser generates several hundred notes to be substituted for a looped rhythmicharmonic waveform and examined for quality.

Creating symbolic melodies with machine learning is challenging because it requires an understanding of musical structure and the handling of inter-dependencies and long-term dependencies. Learning the relationship between events that occur far apart in time in music poses a considerable challenge for machine learning models. Another notable feature of music is that notes must account for several inter-dependencies, including melodic, harmonic, and rhythmic aspects. Baseline methods, such as RNNs, LSTMs, and GRUs, often struggle to capture these dependencies, resulting in the generation of musically incoherent or repetitive melodies. As such, in this study, a hierarchical multi-head attention LSTM model is proposed for creating polyphonic symbolic melodies. This enables our model to generate more complex and expressive melodies than previous methods, while still being musically coherent. The model allows learning of long-term dependencies at different levels of abstraction, while retaining the ability to form inter-dependencies. The study has been conducted on two major symbolic music datasets, MAESTRO and Classical-Music MIDI, which feature musical content encoded on MIDI. The artistic nature of music poses a challenge to evaluating the generated content and qualitative analysis are often not enough. Thus, human listening tests are conducted to strengthen the evaluation. Qualitative analysis conducted on the generated melodies shows significantly improved loss scores on MSE over baseline methods, and is able to generate melodies that were both musically coherent and expressive. The listening tests conducted using Likert-scale support the qualitative results and provide better statistical scores over baseline methods. [ABSTRACT FROM AUTHOR]

With the development of artificial intelligence and deep learning, a large number of music generation methods have been proposed. Recently, Transformer has been widely used in music generation. However, the structural complexity of music puts forward higher requirements for music generation. In this paper, we propose a new automatic music generation network which consists of a Recursive Skip Connection with Layer Normalization (RSCLN) model, a Transformer-XL model and a multi-head attention mechanism. Our method not only alleviates the gradient vanishing problem in the model training, but also increases the ability of the model to capture the correlation of music information before and after, so as to generate music works closer to the original music style. Effectiveness of the RSCLN_Transformer-XL music automatic generation method is verified through music similarity evaluation experiments using music structure similarity and listening test. The experimental results show that the RSCLN_Transformer-XL music automatic generation model can generate better music than the Transformer-XL model. [ABSTRACT FROM AUTHOR]

Recent advances in generative models have shown remarkable progress in music generation. However, since most existing methods focus on generating monophonic or homophonic music, the generation of polyphonic and multi-track music with rich attributes remains a challenging task. In this paper, we propose a novel image-based music generation approach DiffuseRoll, which is based on the diffusion models to generate multi-track, multi-attribute music. Specifically, we generate music piano-rolls with diffusion models and map them to MIDI format files for output. To capture rich attribute information, we design the color-encoding system to encode music note sequences into color and position information representing note pitch, velocity, tempo and instrument. This scheme enables a seamless mapping between discrete music sequences and continuous images. We propose Music Mini Expert System (MusicMES) to optimize the generated music for better performance. We conduct subjective experiments in evaluation metrics, namely Coherence, Diversity, Harmoniousness, Structureness, Orchestration, Overall Preference and Average. The results of subjective experiments are improved compared to the state-of-the-art image-based methods. [ABSTRACT FROM AUTHOR]

Generating expressive musical performance (EMP) is a hot issue in the field of music generation. Music played by humans is always more expressive than music produced by machines. To figure this out, it is crucial to explore the role of human performance in the production of music. This paper proposes a performance style transfer model to learn human performance style and implement EMP system. Our model is implemented using generative adversarial networks (GANs), with a multi-channel image composed of four elaborated spectrograms serving as the input to decompose and reconstruct music audio. To ensure training stability, we have designed a multi-channel consistency loss for GANs. Furthermore, given the lack of objective evaluation criteria for music generation, we propose a hybrid evaluation method that combines qualitative and quantitative methods to evaluate human-needs satisfaction. Three quantitative criteria are proposed at the feature and audio levels, respectively. The effectiveness of our method is verified on a public dataset through objective evaluation, which demonstrates its comparability to state-of-the-art algorithms. Additionally, subjective evaluations are conducted through visual analyses of both audio content and style. Finally, we conduct a musical Turing test in which subjects score the performance of the generated music. A series of experimental results show that our method is very competitive. [ABSTRACT FROM AUTHOR]

In general, dance is always associated with music to improve stage performance effect. As we know, artificial music arrangement consumes a lot of time and manpower. While automatic music arrangement based on input dance video perfectly solves this problem. In the cross-modal music generation task, we take advantage of the complementary information between two input modalities of facial expressions and dance movements. Then we present Dance2MusicNet (D2MNet), an autoregressive generation model based on dilated convolution, which adopts two feature vectors, dance style and beats, as control signals to generate real and diverse music that matches dance video. Finally, a comprehensive evaluation method for qualitative and quantitative experiment is proposed. Compared to baseline methods, D2MNet outperforms better in all evaluating metrics, which clearly demonstrates the effectiveness of our framework.

In the context of rapid advancements in artificial intelligence technology, AI-powered music composition has demonstrated remarkable creative capabilities. However, no existing music generation model has been able to produce authentic waveform-level traditional Chinese music. To explore the potential of this field and address the limitations of current technologies in generating traditional Chinese music, this study introduces CPTGZ (Chinese Painting to Guzheng Music), a music generation model based on latent diffusion and Transformer architectures. CPTGZ aims to achieve automatic generation of waveform-level Guzheng music from Chinese paintings, thereby addressing the inability of existing music generation models to produce traditional Chinese music.To support the development and training of the model, we constructed a large-scale dataset of paired Chinese paintings and Guzheng music, consisting of 22,103 sample pairs. Through experimental evaluation, we found that CPTGZ exhibits excellent performance in terms of music quality and Guzheng-specific characteristics. The results demonstrate that our model can generate Chinese Guzheng music pieces highly correlated in style and semantics with the input Chinese paintings. Furthermore, the musical qualities of the generated Guzheng compositions demonstrate the characteristics of traditional Chinese music, thus validating the feasibility and effectiveness of our model.This research contributes to the field of AI-driven music generation by addressing the specific challenges of creating authentic traditional Chinese music, particularly Guzheng compositions, based on visual art inputs. The successful implementation of CPTGZ not only opens new avenues for cross-modal generation in the domain of culturally specific art forms, but also demonstrates the potential for AI to preserve and innovate within traditional art forms.

Music generation presents a significant challenge within the realm of generative AI, encompassing diverse applications in music production, real-time composition, and other related fields. This paper introduces MGU-V (Music Generation Using Variational Autoencoders), a sophisticated deep learning framework engineered to generate Lo-Fi music. MGU-V harnesses the power of Variational Autoencoders (VAEs) to model and create high-quality music compositions by learning robust latent representations of musical structures. The framework is rigorously evaluated using two meticulously curated and merged benchmark MIDI datasets, demonstrating its effectiveness and adaptability across various musical genres. Through extensive experimentation, MGU-V achieves state-of-the-art performance, significantly surpassing existing methods. The model achieves an impressive accuracy rate of 96.2% and a minimal loss of 0.19, emphasizing its precision and reliability. These outstanding results underscore the potential of MGU-V as a valuable tool for music producers, composers, and AI researchers alike. Its ability to generate Lo-Fi music with high fidelity and consistency highlights promising new avenues for future research and development in AI-driven music generation. The success of MGU-V not only sets a new benchmark in the field but also suggests that AI can increasingly contribute to creative processes traditionally dominated by human expertise.

This study aims to explore the application of deep learning models in multi-track music generation to enhance the efficiency and quality of music production. Considering the limited capability of traditional methods in extracting and representing audio features, a multi-track music generation model based on the Bidirectional Encoder Representations from Transformers (BERT) Transformer network is proposed. This model first utilizes the BERT model to encode and represent music data, capturing semantic and emotional information within the music data. Subsequently, the encoded music features are inputted into the Transformer network to learn the temporal relationships and structural patterns among music sequences, thereby generating new multi-track music compositions. The performance of this model is evaluated, revealing that compared to other algorithms, the proposed model achieves an accuracy of 95.98% in music generation prediction, with an improvement in precision by 4.77%. Particularly, the model demonstrates significant advantages in predicting pitch of music tracks. Hence, the multi-track music generation model proposed in this study exhibits excellent performance in accuracy and pitch prediction, offering valuable experimental reference for research and practice in the field of multi-track music generation.

The Javanese gamelan notation dataset comprises Javanese gamelan gendhing (song) notations for various gamelan instruments. This dataset includes 35 songs categorized into 7 song structures, which are similar to genres in modern music. Each song in this dataset includes the primary melody and notations for various instrument groups, including the balungan instruments group (saron, demung, and slenthem), the bonang barung and bonang penerus instruments, the peking instrument group, and the structural instruments group (kenong, kethuk, kempyang, kempul, and gong). The primary melody is derived from https://www.gamelanbvg.com/gendhing/index.php, a collection of Javanese gamelan songs. On the other hand, the notation of each instrument group is the result of our creation by following the rules of gamelan playing on each instrument. In Javanese gamelan songs, usually written only the main melody notation in the form of numerical notation and the characteristics of the song, such as song title, song structure type, rhythm, scale and mode of the song. Naturally, this is not an easy task for a beginner gamelan player, but a more complete notation will make it easier for anyone who wants to play gamelan. Each song is compiled into a sheet of music, which is presented in a Portable Document Format (PDF) file. This dataset is valuable for developing deep learning models to classify or recognize Javanese gamelan songs based on their instrument notations, as previous gamelan research has mostly used audio data. Furthermore, this dataset has the capability to automatically generate Javanese gamelan notation for songs of similar types. Additionally, it will be useful for educational purposes to facilitate the learning of Javanese gamelan songs and for the preservation of traditional Javanese gamelan music.

This work aims to create an ANN-based system for a musical improviser. An artificial improviser of "hearing" music will create a melody. The data supplied to the improviser is MIDI-type musical data. This is the harmonic-rhythmic course, the background for improvisation, and the previously made melody notes. The harmonic run is fed into the system as the currently ongoing chord and the time to the next chord, while the supplied few dozen notes performed earlier will indirectly carry information about the entire run and the musical context and style. Improvisation training is carried out to check ANN as a correct-looking musical improvisation device. The improviser generates several hundred notes to be substituted for a looped rhythmic-harmonic waveform and examined for quality. [ABSTRACT FROM AUTHOR]

Chinese Folk Drum music is an excellent traditional cultural resource, it has brilliant historical and cultural heritage and excellent traditional cultural connotation. However, the survey found that the social and cultural values, tourism economic values, and national self-confidence embodied in folk drum music, such as Xi'an drum music, are far from being released, and even its own inheritance and development are facing difficulties. The research focuses on the automatic generation of Xi'an drum music, with the aim of further inheriting, developing, and utilizing this exceptional traditional cultural resource. While Artificial Intelligence (AI) music generation has gained popularity in recent years, most platforms primarily focus on modern music rather than Chinese folk music. To address these issues and the unique challenges faced by Xi'an drum music, this paper proposes a Bi-LSTM network-based deep reinforcement learning model. The model incorporates the distinctive characteristics of ancient Chinese music, such as pitch, chord, and mode, and utilizes the Actor-Critic algorithm in reinforcement learning. During the simulation generation stage, an improved method of generating strategies through reward and punishment scores is introduced. Additionally, the model takes into account abstract concept constraints, such as chord progression and music theory rules, which are translated into computer language. By constructing a chord reward mechanism and a music principle reward mechanism, the model achieves harmony constraints and enables the systematic generation of drum music. Experimental results demonstrate that the proposed model, based on Bi-LSTM deep reinforcement learning, can generate Xi'an drum music with high quality and artistic aesthetics. This research contributes to the preservation, development, and utilization of Xi'an drum music, leveraging advancements in AI music generation technology. [ABSTRACT FROM AUTHOR]

Copyright of Frontiers of Information Technology & Electronic Engineering is the property of Springer Nature and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

This research paper investigates the transformative potential of computational music analysis and artificial intelligence (AI) in advancing the field of music composition. Specifically, it explores the synergistic roles of pre-analysis and post-analysis techniques in leveraging AI-driven tools to enhance the creative process and quality of musical compositions. The study encompasses a historical overview of music composition, the evolution of computational music analysis, and contemporary AI applications. It delves into pre-analysis, focusing on its role in informing composition, and post-analysis, which evaluates and augments compositions. The paper underscores the significance of these technologies in fostering creativity while addressing challenges and ethical considerations. Through case studies, evaluations, and discussions, this research offers insights into the profound impact of computational music analysis and AI on music composition, paving the way for innovative and inclusive musical expressions. [ABSTRACT FROM AUTHOR]

This survey explores the symbiotic relationship between Machine Learning (ML) and music, focusing on the transformative role of Artificial Intelligence (AI) in the musical sphere. Beginning with a historical contextualization of the intertwined trajectories of music and technology, the paper discusses the progressive use of ML in music analysis and creation. Emphasis is placed on present applications and future potential. A detailed examination of music information retrieval, automatic music transcription, music recommendation, and algorithmic composition presents state-of-the-art algorithms and their respective functionalities. The paper underscores recent advancements, including ML-assisted music production and emotion-driven music generation. The survey concludes with a prospective contemplation of future directions of ML within music, highlighting the ongoing growth, novel applications, and anticipation of deeper integration of ML across musical domains. This comprehensive study asserts the profound potential of ML to revolutionize the musical landscape and encourages further exploration and advancement in this emerging interdisciplinary field. [ABSTRACT FROM AUTHOR]

Abstract Music inpainting is a sub-task of automated music generation that aims to infill incomplete musical pieces to help musicians in their musical composition process. Many methods have been developed for this task. However, we observe a tendency for each method to be evaluated using different datasets and metrics in the papers where they are presented. This lack of standardization hinders an adequate comparison of these approaches. To tackle these problems, we present MUSIB, a new benchmark for musical score inpainting with standardized conditions for evaluation and reproducibility. MUSIB evaluates four models: Variable Length Piano Infilling (VLI), Music InpaintNet, Music SketchNet, and AnticipationRNN, and over two commonly used datasets: JSB Chorales and IrishFolkSong. We also compile, extend, and propose metrics to adequately quantify note attributes such as pitch and rhythm with Note Metrics, but also higher-level musical properties with the introduction of Divergence Metrics, which operate by comparing the distance between distributions of musical features. Our evaluation shows that VLI, a model based on Transformer architecture, is the best performer on a larger dataset, while VAE-based models surpass this Transformer-based model on a relatively small dataset. With MUSIB, we aim at inspiring the community towards better reproducibility in music generation research, setting an example for strongly founded comparisons among SOTA methods.

Music is composed of a set of regular sound waves, which are usually ordered and have a large number of repetitive structures. Important notes, chords, and music fragments often appear repeatedly. Such repeated fragments (referred to as motifs) are usually the soul of a song. However, most music generated by existing music generation methods can not have distinct motifs like real music. This study proposes a novel multi- encoders model called Motif Transformer to generate music containing more motifs. The model is constructed using an encoder-decoder framework that includes an original encoder, a bidirectional long short term memory-attention encoder (abbreviated as bilstm-attention encoder), and a gated decoder. Where the original encoder is taken from the transformer’s encoder and the bilstm-attention encoder is constructed from the bidirectional long short-term memory network (BILSTM) and the attention mechanism; Both the original encoder and the bilstm-attention encoder encode the motifs and input the encoded information representations to the gated decoder; The gated decoder decodes the entire input of the music and the information passed by the encoders and enhances the model’s ability to capture motifs of the music in a gated manner to generate music with significantly repeated fragments. In addition, in order to better measure the model’s ability of generating motifs, this study proposes an evaluation metric called used motifs. Experiments on multiple music field metrics show that the model proposed in this study can generate smoother and more beautiful music, and the generated music contains more motifs.

The discipline of Deep Learning has been recognized for its strong computational tools, which have been extensively used in data and signal processing, with innumerable promising results. Among the many commercial applications of Deep Learning, Music Signal Processing has received an increasing amount of attention over the last decade. This work reviews the most recent developments of Deep Learning in Music signal processing. Two main applications that are discussed are Music Information Retrieval, which spans a plethora of applications, and Music Generation, which can fit a range of musical styles. After a review of both topics, several emerging directions are identified for future research.