Awesome Music Generation with AI

Welcome to the Awesome Music Generation with AI list, a curated collection of resources, projects, and frameworks at the intersection of artificial intelligence and music creation. Over the years, the field of generative music has witnessed a significant evolution, propelled by advancements in machine learning and deep learning technologies. From algorithmic composition to real-time music generation, AI has opened new horizons, enabling a blend of creativity and automation that was once unimaginable.

This list aims to be a comprehensive hub for enthusiasts, researchers, and professionals, bringing together the pioneering projects, influential research papers, and state-of-the-art frameworks that are shaping the future of music generation through AI. Whether you are a musician exploring the digital frontier, a researcher pushing the boundaries of what's possible, or a developer aiming to integrate AI-driven music capabilities into applications, this collection will provide a rich source of inspiration and knowledge.

Table of Contents

• GitHub projects

• voicetoinstrument.com - Convert voice to instrument tracks using AI

• Articles & Blogs

• Online Courses

• Books

• Research Papers

• Videos

• Tools & Software

• Conferences & Events

• Slides & Presentations

GitHub projects

• Magenta: Music and Art Generation with Machine Intelligence 🎵🖌️ (18712 stars)
• Audiocraft: A library for audio processing and generation with deep learning, including MusicGen, a controllable music generation LM with textual and melodic conditioning 🎧 (17044 stars)
• Muzic: Music Understanding and Generation with Artificial Intelligence 🎶 (3765 stars)
• musiclm-pytorch: PyTorch implementation of MusicLM, Google's state-of-the-art model for music generation using attention networks 🎼 (2763 stars)
• riffusion: Stable diffusion for real-time music generation 🎵 (2727 stars)
• Mubert-Text-to-Music: A notebook demonstrating prompt-based music generation using Mubert API 🎵 (2674 stars)
• riffusion-app: Stable diffusion for real-time music generation in a web app 🎵 (2474 stars)
• Magenta.js: Music and Art Generation with Machine Learning in the browser 🎵🖌️ (1899 stars)
• AudioLDM2: Text-to-Audio/Music Generation 🎵 (1733 stars)
• musegan: An AI for Music Generation 🎵 (1602 stars)
• Radium: A graphical music editor and next generation tracker. 🎵⚡️ (805 stars)
• GRUV: A Python project for algorithmic music generation. 🐍🎶 (798 stars)
• DeepJ: A deep learning model for style-specific music generation. 🎵🔥 (717 stars)
• Music Generation with Deep Learning: Resources on music generation using deep learning. 🎶💻 (700 stars)
• Musika: Fast infinite waveform music generation. 🎵💨 (646 stars)
• Music Generation Research: A collection of music generation research resources. 🎶🔬 (516 stars)
• MusPy: A toolkit for symbolic music generation. 🎵🔧 (387 stars)
• MusicGenerator: Experiment with diverse deep learning models for music generation with TensorFlow. 🎶🧪 (309 stars)
• MuseTree: AI music generation for the real world. 🎵🌍 (215 stars)
• VampNET: Music generation with masked transformers! 🎶🦇 (204 stars)
• TTS WebUI - Generative AI for Music and TTS, supporting Audiocraft, MusicGen, ACE-Step, RVC and more. 🎵 (2163 stars)
• OBSIDIAN Neural: Real-time AI loop generation VST3 plugin for live performance. 🎹⚡ (150 stars)
• LatentScore: Simple text to ambient music in Python without GPU. 🎵 (35 stars)

Articles & Blogs

• Simple and Controllable Music Generation: A single Language Model (LM) called MusicGen that operates over compressed discrete music representation, allowing better control over the generated output. Music samples, code, and models are available at the provided link.
• AI-Based Affective Music Generation Systems: A Review of Methods: A comprehensive review of AI-AMG systems, discussing their building blocks, categorizing existing systems based on core algorithms, and exploring AI-based approaches for composing affective music.
• Music FaderNets: Controllable Music Generation Based On High: A framework (Music FaderNets) for learning high-level feature representations by manipulating low-level attributes through feature disentanglement and latent regularization techniques.
• Music Generation by Deep Learning-Challenges and Directions: An overview of deep learning approaches for music generation, discussing their limitations in terms of creativity and control.
• MusPy: A Toolkit for Symbolic Music Generation: Introduction of MusPy, an open-source Python library for symbolic music generation, providing tools for dataset management, data preprocessing, and model evaluation. Statistical analysis of supported datasets is also included.
• Music generation with variational recurrent autoencoder supported: Introduction of a new network architecture, variational autoencoder supported by history, for generating longer melodic patterns. Filtering heuristics are used to enhance the generated music.
• Symbolic Music Generation with Diffusion Models: Application of diffusion models to modeling symbolic music, demonstrating strong generation and conditional infilling results.
• Magenta: A research project exploring the role of machine learning in the creation of art and music.
• How to generate music with Python: The Basics: An article discussing the basics of generating music with Python, highlighting its use in procedural MIDI generation.
• MidiNet: A Convolutional Generative Adversarial Network for: Investigation of using convolutional neural networks (CNNs) for generating melody in the symbolic domain, introducing conditional mechanisms and expanding to multiple MIDI channels.
• A Survey on Artificial Intelligence for Music Generation: Agents ... :octocat:
• This survey paper explores the field of music generation with artificial intelligence (AI), discussing the composition techniques and advances in AI systems imitating the music generation process. It also highlights the role of datasets, models, interfaces, and users in the music generation process, along with potential applications and future research directions.
• Generating Music With Artificial Intelligence :musical_note:
• This article provides insights into how recurrent neural networks (RNNs) can be used for music generation with machine learning. It serves as a refresher for RNN-based text generation techniques.
• From Artificial Neural Networks to Deep Learning for Music ...
• This paper explores the application of deep learning techniques in music generation. It offers a tutorial on how deep learning can be used to automatically learn musical styles and generate music samples.
• Noise2Music: Text-conditioned Music Generation with Diffusion ... :sound:
• This research introduces Noise2Music, a system that utilizes diffusion models to generate high-quality music clips from text prompts. It demonstrates how the generated audio can capture the genre, tempo, instruments, mood, and era specified in the text.
• A Classifying Variational Autoencoder with Application to ... :musical_keyboard:
• This paper presents a model based on the variational autoencoder (VAE) framework for algorithmic music generation. The model incorporates a classifier to infer the discrete class of the modeled data, allowing for the generation of musical sequences in different keys.
• Generating Ambient Music from WaveNet
• This post discusses the motivation and approach for generating ambient music using Google DeepMind's WaveNet, an audio-generative model.
• Generating Music using an LSTM Neural Network
• This blog post presents the use of a long short-term memory (LSTM) neural network for music generation. It covers improvements made to an existing LSTM model.
• Discrete Diffusion Probabilistic Models for Symbolic Music Generation :notes:
• This work introduces the generation of polyphonic symbolic music using Discrete Diffusion Probabilistic Models (DDPMs). The models exhibit high-quality sample generation and allow for flexible infilling at the note level. The paper also discusses the evaluation of music sample quality and the possible applications of these models.

Online Courses

• Generative AI Courses & Certifications: Take the next step in your professional journey and enroll in a Generative AI course today! Browse Generative AI Courses offered from top universities and industry leaders. 🎓
• Complete A.I. Art Generation Course - Beginner 2 MASTER: Learn how to generate everything from Language, Art, Music & much more using cutting-edge A.I. algorithms. 🎨🎵
• Andrew Ng: Announcing My New Deep Learning Specialization: Dive into deep learning with Andrew Ng, a renowned AI expert, and learn the foundations of this exciting field. 🧠
• Best Deep Learning Courses & Certifications (Coursera): Enhance your deep learning skills and knowledge by enrolling in a wide range of courses offered by top universities and industry leaders.

Books

• Deep Learning Techniques for Music Generation 📚: This book presents a survey and analysis of how deep learning can be utilized to generate musical content, providing insights for students, practitioners, and researchers.
• Algorithmic Composition: Paradigms of Automated Music Generation 📚: Offering a detailed overview of algorithmic composition, this book focuses on prominent procedures and principles in a practical manner.
• Hands-On Music Generation with Magenta 📚: Explore the role of deep learning in music generation and assisted composition with Magenta. This hands-on guide integrates ML models into existing music production tools.
• Machine Learning and Music Generation 📚: Delve into the intersection of machine learning and music generation with this comprehensive book, covering the use of ML techniques in creating music.

Research Papers

• Simple and Controllable Music Generation : This paper introduces MusicGen, a single Language Model (LM) that generates high-quality music samples conditioned on textual description or melodic features, allowing better control over the generated output. It showcases superior performance compared to baselines on a standard text-to-music benchmark.
• Efficient Neural Music Generation : MeLoDy (M for music; L for LM; D for diffusion) is proposed as an LM-guided diffusion model that generates music audios of state-of-the-art quality while reducing forward passes in the sampling process, making it computationally efficient.
• Noise2Music: Text-conditioned Music Generation with Diffusion : This paper presents Noise2Music, a series of diffusion models trained to generate high-quality 30-second music clips from text prompts. It explores different options for intermediate representations and demonstrates the ability to faithfully reflect key elements of the text prompt.
• VampNet: Music Generation via Masked Acoustic Token Modeling : VampNet leverages a bidirectional transformer architecture and a variable masking schedule during training to generate coherent high-fidelity musical waveforms. It showcases capabilities in music synthesis, compression, inpainting, and variation.
• MuseGAN: Multi-track Sequential Generative Adversarial Networks : This paper proposes three models for symbolic multi-track music generation using generative adversarial networks (GANs), taking into account temporal dynamics and interdependencies between tracks.
• JEN-1: Text-Guided Universal Music Generation with Diffusion : JEN-1 is introduced as a universal high-fidelity model for text-to-music generation, incorporating both autoregressive and non-autoregressive training. It demonstrates superior performance in text-music alignment and music quality.
• Museformer: Transformer with Fine-and Coarse-Grained Attention : Museformer is a Transformer-based approach for music generation that addresses challenges related to long music sequences and musical repetition structures. It introduces fine- and coarse-grained attention mechanisms to capture relevant music structures efficiently.
• A Comprehensive Survey on Deep Music Generation: Multi-level Perspectives : This survey provides an overview of deep learning techniques in music generation, covering various composition tasks under different music generation levels (score generation, performance generation, and audio generation).
• Quantized GAN for Complex Music Generation from Dance Videos : Dance2Music-GAN (D2M-GAN) is an adversarial multi-modal framework that generates complex musical samples conditioned on dance videos. It uses Vector Quantized (VQ) audio representation to generate diverse dance music styles.
• Musika! Fast Infinite Waveform Music Generation: Fast and user-controllable music generation system that allows for much faster than real-time generation of music of arbitrary length on a consumer CPU.
• A systematic review of artificial intelligence-based music generation: Provides a wide range of publications and explores the interest of both musicians and computer scientists in AI-based automatic music generation.
• MidiNet: A Convolutional Generative Adversarial Network for Music Generation: Introduces the use of convolutional neural networks (CNNs) for generating melodies in the symbolic domain.
• Music Generation by Deep Learning-Challenges and Directions: Explores the limitations of deep learning for music generation and the need for control, structure, creativity, and interactivity.
• What is missing in deep music generation? A study of repetition and structure: Investigates the understanding of music structure and repetition in the context of music generation and suggests new formal music criteria and evaluation methods.
• Symbolic Music Generation with Diffusion Models: Presents a technique for training diffusion models on sequential data to generate symbolic music with strong unconditional generation and post-hoc conditional infilling results.
• Discrete Diffusion Probabilistic Models for Symbolic Music Generation: Explores the application of Discrete Diffusion Probabilistic Models (D3PMs) for generating polyphonic symbolic music with high sample quality and flexible infilling.
• MMM: Exploring Conditional Multi-Track Music Generation with the Transformer: Introduces a generative system based on the Transformer architecture for generating multi-track music with greater control and handling of long-term dependencies.
• Deep Learning Techniques for Music Generation - A Survey: Analyses the different ways of using deep learning for generating musical content, covering objectives, representations, architectures, challenges, and evaluation.
• Mo^usai: Text-to-Music Generation with Long-Context Latent Diffusion: Bridges the connection between text and music with a highly efficient text-to-music generation model that can generate multiple minutes of high-quality stereo music from textual descriptions.

Videos

• Music Generation with Magenta: Using Machine Learning in Arts - Nov 7, 2019. Composing music is hard, and the lack of inspiration can be daunting. This video explores how machine learning can be used in music generation.
• How to code a music generation genetic algorithm? - Apr 3, 2021. This video discusses coding a genetic algorithm for generating music, building upon the concepts presented in a previous video.
• Deep Learning for Music Generation - Feb 8, 2018. In this episode of the AI show, Erika explains how to create deep learning models using music as input, delving into the technical aspects of music generation using deep learning.
• Composing Heavy Metal with GPT - HuggingFace for Music - Jan 26, 2022. This video showcases the usage of HuggingFace for music generation, specifically focusing on composing heavy metal music.
• MusicGen: Simple and Controllable Music Generation Explained - Jun 25, 2023. This video provides an explanation of MusicGen, a framework for simple and controllable music generation.
• Jawlove - Everything Will Be Alright - YouTube - A music video featuring the song "Everything Will Be Alright" by Jawlove.
• Musical Beginnings with Karen #7 Slippery Fish - YouTube - A video from the Music Generation Waterford program showcasing a music education performance.
• Cybernetic Celebration | EDM | Loudly AI Music Generator - YouTube - A video demonstrating the use of the Loudly AI Music Generator to create EDM music.
• Music Generation Cork City - YouTube - A playlist of videos showcasing performances by Music Generation Cork City.
• Top AI music generating tools (publicly available tools) - video - A video exploring the top AI music generating tools available, including Mubert AI, AIVA, Soundraw, Beatoven AI, Boomy, and Amper Music.

Tools & Software

• Stability AI unveils 'Stable Audio': A versatile platform for AI Music Generation. Stability AI has launched a new AI platform, Stable Audio, which offers a novel latent diffusion model for generating audio conditioned on metadata and timing, providing faster inference times and creative control.
• SuperCollider: An audio server, programming language, and IDE for sound synthesis. SuperCollider is a platform for audio synthesis and algorithmic composition.
• Best Open Source AI Music Generators: Implementation of AudioLM, a language modeling approach to audio generation using Pytorch. It includes conditioning mechanisms for more control over generated music.
• Soundful: An AI Music Generator that allows creators to generate royalty-free tracks instantly. Soundful generates high-quality music using AI technology, making it easy for anyone to create professional-sounding music.
• Strasheela: A constraint-based music composition system. Users define music theories with sets of compositional rules, and the system generates music that complies with these theories.
• Best AI Music Generators - 2023 Reviews & Comparison: An online tool with multiple ways to create song covers, including searching for songs, uploading audio files, and recording directly.
• What Is AI Generated Music? Best Music Tools for 2023: A software that allows businesses to explore AI-generated music as a cheaper alternative, offering a free trial with unlimited music projects and monthly song downloads.
• Best Audio Editing Software in 2023: Compare Reviews on 100+ | G2: A comprehensive list of audio editing software commonly used by audio engineers and music producers, with real-time product reviews from verified users.
• Psycle Modular Music Creation Studio Reviews - 2023: User reviews and ratings of the Psycle Modular Music Creation Studio free open-source software project.
• MuseGen - AI tool for lyric writing and song generation.
• OBSIDIAN Neural: VST3 plugin for real-time AI music generation during live performance. Features 8-track MIDI-triggered sampler with text-to-audio and experimental draw-to-audio generation.
• Claude AI Music Skills: Claude Code plugin for full-lifecycle AI music album production — lyrics, Suno style prompts, per-stem mixing, mastering, and release distribution.
• AceTagGen: A free prompt builder for Suno AI. Structured inputs (mood, genre, instruments, SFX, production style) that produce a Suno-ready prompt fitting the 200-char Style field. Includes tested example library and quality-score validator.
• HookGenius: An AI pre-production tool for Suno. Turns a single idea into complete song lyrics with structure tags, a Suno-optimized style prompt, and a title — all ready to copy-paste into Suno. Refined across 50+ genres; free tier included.
• Boppy: Free AI music generator that creates complete songs from text descriptions. AI writes lyrics and ACE-Step 1.5 generates full audio tracks with vocals and instrumentation. Any genre, up to 5 minutes per track, no signup required.
• workmusic.ai: Free browser-based ambient focus music generator powered by Google Lyria. No sign-up required. Open source.
• webtoolz AI Music Studio: Free browser-based AI music generator. Create full songs with lyrics from text prompts, no signup required.
• Claude AI Music Skills: Claude Code plugin for full-lifecycle AI music album production — lyrics, Suno style prompts, per-stem mixing, mastering, and release distribution.
• AudioCraft - Browser-based AI stem splitter for separating songs into vocals, drums, bass, and other instruments.

Conferences & Events

• Neuton.AI Events - Neuton.AI is hosting various events, including an ARM Tech Talk about the Next Generation Smart Toothbrush and showcasing their unique neural network framework for building compact models with optimal size and accuracy.
• FUTURE DEAD ARTISTS Events - Stay updated on upcoming events by FUTURE DEAD ARTISTS, including the FDA 2023 Freshman Class: FUTURE GENERATION Artists Talk.
• Generative AI, Apps, and DevOps | AI/ML Talks - Pulumi presents a talk on Generative AI, Apps, and DevOps in the field of AI/ML Talks on October 19, 2023, in Seattle, WA.
• Women in Tech & Entrepreneurship - Fort Lauderdale Chapter Happy Hour - Fort Lauderdale Chapter Happy Hour event for Women in Tech & Entrepreneurship.

Slides & Presentations

• Algorithmic music generation | PPT: Slides discussing algorithmic music generation, available for free as a PDF or online view.
• Music Generation with Deep Learning | PPT: Presentation exploring music generation using deep learning, downloadable as a PDF or for online viewing.
• Video Background Music Generation with Controllable Music Transformer: Slides discussing the generation of video background music using a controllable music transformer, available as a PDF or for online viewing.
• Automatic Music Generation Using Deep Learning | PDF: Slides explaining the process of automatic music generation using deep learning, downloadable as a PDF or for online viewing.
• MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment (AAAI 2018): Slides presenting MuseGAN, a framework for multi-track sequential generative adversarial networks for symbolic music generation and accompaniment, available as a PDF or for online viewing.
• Automatic Music Composition with Transformers, Jan 2021 | PPT: Presentation introducing ongoing projects on automatic music composition using Transformers, downloadable as a PDF or for online viewing.
• ISMIR 2019 tutorial: Generating music with generative adversarial networks (GANs): Slides from the ISMIR 2019 tutorial on generating music with generative adversarial networks (GANs), available as a PDF or for online viewing.
• PopMAG: Pop Music Accompaniment Generation | PPT: Slides discussing PopMAG, a framework for pop music accompaniment generation, available as a PDF or for online viewing.
• Artificial intelligence and Music | PPT: Slides exploring the application of recurrent neural networks paired with LSTM for music generation, downloadable as a PDF or for online viewing.
• Machine learning for creative AI applications in music (2018 nov) | PPT: Presentation on machine learning for creative AI applications in music, available as a PDF or for online viewing.

This initial version of the Awesome List was generated with the help of the Awesome List Generator. It's an open-source Python package that uses the power of GPT models to automatically curate and generate starting points for resource lists related to a specific topic.