ToneCrafter is a project developped by students at ENSEA.
Our goal is to develop an algorithm capable of analyzing any audio file containing guitar sounds and retrieve its parameters. ToneCrafter will then generate a numeric filter that will modulate your guitar sound and make it sound like the recording you fed it. Exciting isn't it ?
We are also working on a hardware implementation using a STM32 µC that will allow musicians to use ToneCrafter on stage and tune the different parameters.
We started out trying to use Magenta's DDSP, but this was a bit too ambitious for AI newbies like us, so we tried implementing our own algorithms.
This sparked three different approaches to our problem:
- A CNN based approach where we tried to teach a neural network to recreate a distortion.
- A "bare coding" approach using math and clever algorithms to recreate a distortion from an audio file
- A VAE based approach trying to recreate the mechanisms behind Magenta's DDSP
We started working on different PCB designs using Eagle and working on a STM32F7 discovery kit for testing purposes.
To begin with, if you want to work locally and not on Colabs as we did, you will have to install Anaconda. To do this follow the officials instructions that are given to you.
If you want to try the huge potential of DDSP, let’s check the DDSP GitHub which show how to install it. There are also some tutorials to introduce the main concepts of the library (that we strongly recommend you to do) and demos to show what you can do with it (Like Timbre transfer, Train Autoencoder and Pitch Detection).
And then, if you want to have fun on the models created by the Google team with the Magenta project, follow the installation guide on their page.
To train our CNN, we searched for a dataset containing clean audio from a guitar and different distortions. As we could not find any we had to create our own. We recorded a guitar for 1 minute and added various levels of distortion. To increase the size of our base, we applied an EQ filter with various settings.
We ended up with a lot of .wav files that you can find here.
In order to train our CNN, we splitted our files in 200ms chunks using our Preparing Data Notebook
We then organised our files in the following way :
-
X_train :
- Clean
- Clean_TrebbleBoost
- Clean_BassBoost
- Clean_BassCut
-
y_train :
- Disto
- Disto_TrebbleBoost
- Disto_BassBoost
- Disto_BassCut
-
Validation data :
- Clean_TrebbleCut as X_valid
- Disto_TrebbleCut as y_valid
We are concious that this dataset is far from perfect (please don't listen to it, the guitar playing is awful) from a recording standpoint and the way we splitted our audio might cause some problems.
For the maths oriented approach, we used the excellent IDMT-SMT-Audio_Effects Dataset, which enabled us to link each note played to it's distorted counterpart.
For a more in-depths version of this readme, checkout our (magnificent) project report available in both French and English at the root of this repository
As of the 03/05/2021, our yearly project has ended. We will continue to work on ToneCrafter at a lighter pace, as we'll be completing our summer internships. If all goes well, some new students will pick up the project in the fall to continue to build on top of our work. In the mean time, don't hesitate to take part in our project, via issues or pull requests.