04_generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementing the `MiniBach` model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: Generating an accompaniment for an arbitrary melody"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step, we use the model trained on Part 3 to generate the three lower voices for a given soprano melodic line.\n",
    "\n",
    "The model trained during Part 3 has been saved as `trained_model.h5`.  \n",
    "\n",
    "We can use this model to predict the accompaniment of an arbitrary melody."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import music21"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.models.load_model('trained_model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a very rudimentary syntax for encoding our melody:\n",
    "- Each token represents a sixteenth note\n",
    "- The special token `--` denotes a *hold* symbol (in the generated scores, it becomes a tie)\n",
    "- The pipe symbol `|` is just there for visual aid, separating blocks of four sixteenth notes (or one quarter note)\n",
    "- The same can be said about the newlines, they separate the melody into four blocks (measures) with four quarter notes each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "given_melody = '''\n",
    "A4   --   --   --   |G#4  --   --   --   |A4   --   --   --   |F4   --   --   --   |\n",
    "D4   --   --   --   |--   --   --   --   |D4   --   --   --   |--   --   --   --   |\n",
    "E4   --   --   --   |F#4  --   --   --   |G#4  --   --   --   |A4   --   B4   --   |\n",
    "C5   --   --   --   |C4   --   --   --   |E4   --   --   --   |--   --   --   --   |\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "given_melody = given_melody.replace('\\n', '').replace('|', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = given_melody.split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After organizing the tokens of the input melody, we need to encode it in a one-hot-encoded representation. \n",
    "\n",
    "This process is fairly similar to how it was done in Part 2, so I won't describe it here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "SOPRANO_MIN = 57\n",
    "SOPRANO_MAX = 81\n",
    "\n",
    "ALTO_MIN = 52\n",
    "ALTO_MAX = 74\n",
    "\n",
    "TENOR_MIN = 48\n",
    "TENOR_MAX = 69\n",
    "\n",
    "BASS_MIN = 36\n",
    "BASS_MAX = 64\n",
    "\n",
    "ranges = {\n",
    "    'soprano': {midinumber: (midinumber - SOPRANO_MIN + 1) for midinumber in range(SOPRANO_MIN, SOPRANO_MAX + 1)},\n",
    "    'alto': {midinumber: (midinumber - ALTO_MIN + 1) for midinumber in range(ALTO_MIN, ALTO_MAX + 1)},\n",
    "    'tenor': {midinumber: (midinumber - TENOR_MIN + 1) for midinumber in range(TENOR_MIN, TENOR_MAX + 1)},\n",
    "    'bass': {midinumber: (midinumber - BASS_MIN + 1) for midinumber in range(BASS_MIN, BASS_MAX + 1)},\n",
    "}\n",
    "\n",
    "reverse_ranges = {\n",
    "    'soprano': {(midinumber - SOPRANO_MIN + 1): midinumber for midinumber in range(SOPRANO_MIN, SOPRANO_MAX + 1)},\n",
    "    'alto': {(midinumber - ALTO_MIN + 1): midinumber for midinumber in range(ALTO_MIN, ALTO_MAX + 1)},\n",
    "    'tenor': {(midinumber - TENOR_MIN + 1): midinumber for midinumber in range(TENOR_MIN, TENOR_MAX + 1)},\n",
    "    'bass': {(midinumber - BASS_MIN + 1): midinumber for midinumber in range(BASS_MIN, BASS_MAX + 1)},\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode_note(n, rang):\n",
    "    if n == '--' or n == 'Rest':\n",
    "        ret = 0\n",
    "    else:\n",
    "        note = music21.note.Note(n)\n",
    "        ret = ranges[rang][note.pitch.midi]\n",
    "    return ret\n",
    "\n",
    "def one_hot_encode(idx, rang):\n",
    "    length = len(ranges[rang].values())\n",
    "    ret = [0] * (length + 1)\n",
    "    ret[idx] = 1\n",
    "    return ret"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = [encode_note(n, 'soprano') for n in tokens] \n",
    "x = np.array([[one_hot_encode(idx, 'soprano') for idx in s]])    \n",
    "x = x.reshape(1, -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The melody has been encoded, so we can pass it to the model and collect the predictions from the `MiniBach` model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model.predict(x)\n",
    "\n",
    "predictions = predictions.reshape(-1)\n",
    "\n",
    "soprano = x.reshape(64, -1)\n",
    "alto = predictions[:1536].reshape(64, -1)\n",
    "tenor = predictions[1536:3008].reshape(64, -1)\n",
    "bass = predictions[3008:4928].reshape(64, -1)\n",
    "\n",
    "music = {\n",
    "    'soprano': soprano,\n",
    "    'alto': alto,\n",
    "    'tenor': tenor,\n",
    "    'bass': bass\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decode_note(n, rang):\n",
    "    if n == 0:\n",
    "        ret = '--'\n",
    "    else:\n",
    "        note = music21.note.Note(type='16th')        \n",
    "        note.pitch.midi = reverse_ranges[rang][n]        \n",
    "        ret = note\n",
    "    return ret"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "generation = {\n",
    "    'soprano': [],\n",
    "    'alto': [],\n",
    "    'tenor': [],\n",
    "    'bass': []\n",
    "}\n",
    "\n",
    "for sixteenth in range(64):\n",
    "    for part, notes in music.items():\n",
    "        this_note = decode_note(np.argmax(notes[sixteenth]), part)\n",
    "        if this_note == '--':\n",
    "            last_note = generation[part][-1]\n",
    "            this_note = music21.note.Note(last_note.pitch.nameWithOctave, type='16th')\n",
    "            if last_note.tie:\n",
    "                this_note.tie = music21.tie.Tie('continue')\n",
    "            else:\n",
    "                last_note.tie = music21.tie.Tie('start')\n",
    "                generation[part][-1] = last_note\n",
    "                this_note.tie = music21.tie.Tie('continue')\n",
    "        else:\n",
    "            if sixteenth > 0:\n",
    "                last_note = generation[part][-1]\n",
    "                if last_note.tie:\n",
    "                    last_note.tie = music21.tie.Tie('stop')\n",
    "        generation[part].append(this_note)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The predictions have been generated, decoded, and turned into music notes with the `music21` library. We can give a glance to the 4-part choral (1 given soprano + 3 generated voices)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(generation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>soprano</th>\n",
       "      <th>alto</th>\n",
       "      <th>tenor</th>\n",
       "      <th>bass</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note E&gt;</td>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note D&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note E&gt;</td>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note D&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note E&gt;</td>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note D&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note E&gt;</td>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note D&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>&lt;music21.note.Note G#&gt;</td>\n",
       "      <td>&lt;music21.note.Note E&gt;</td>\n",
       "      <td>&lt;music21.note.Note C&gt;</td>\n",
       "      <td>&lt;music21.note.Note D&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note C#&gt;</td>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note C#&gt;</td>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note C#&gt;</td>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note C#&gt;</td>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note C#&gt;</td>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "      <td>&lt;music21.note.Note A&gt;</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>64 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   soprano                    alto                  tenor  \\\n",
       "0    <music21.note.Note A>   <music21.note.Note E>  <music21.note.Note A>   \n",
       "1    <music21.note.Note A>   <music21.note.Note E>  <music21.note.Note A>   \n",
       "2    <music21.note.Note A>   <music21.note.Note E>  <music21.note.Note A>   \n",
       "3    <music21.note.Note A>   <music21.note.Note E>  <music21.note.Note A>   \n",
       "4   <music21.note.Note G#>   <music21.note.Note E>  <music21.note.Note C>   \n",
       "..                     ...                     ...                    ...   \n",
       "59   <music21.note.Note A>  <music21.note.Note C#>  <music21.note.Note A>   \n",
       "60   <music21.note.Note A>  <music21.note.Note C#>  <music21.note.Note A>   \n",
       "61   <music21.note.Note A>  <music21.note.Note C#>  <music21.note.Note A>   \n",
       "62   <music21.note.Note A>  <music21.note.Note C#>  <music21.note.Note A>   \n",
       "63   <music21.note.Note A>  <music21.note.Note C#>  <music21.note.Note A>   \n",
       "\n",
       "                     bass  \n",
       "0   <music21.note.Note D>  \n",
       "1   <music21.note.Note D>  \n",
       "2   <music21.note.Note D>  \n",
       "3   <music21.note.Note D>  \n",
       "4   <music21.note.Note D>  \n",
       "..                    ...  \n",
       "59  <music21.note.Note A>  \n",
       "60  <music21.note.Note A>  \n",
       "61  <music21.note.Note A>  \n",
       "62  <music21.note.Note A>  \n",
       "63  <music21.note.Note A>  \n",
       "\n",
       "[64 rows x 4 columns]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to play the score, I use `music21` to generate an output `MusicXML` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = music21.stream.Stream()\n",
    "s.append(df.soprano.to_list())\n",
    "a = music21.stream.Stream()\n",
    "a.append(df.alto.to_list())\n",
    "t = music21.stream.Stream()\n",
    "t.append(df.tenor.to_list())\n",
    "b = music21.stream.Stream()\n",
    "b.append(df.bass.to_list())\n",
    "stream = music21.stream.Stream([s,a,t,b])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/napulen/dev/MiniBach/generated_choral.musicxml'"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stream.write('musicxml', 'generated_choral.musicxml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And that's it, a generated choral using the `MiniBach` model.\n",
    "\n",
    "The `MusicXML` file can be played using a music notation software like MuseScore, Sibelius, Finale, or Dorico.\n",
    "\n",
    "An alternative option is to export it as `midi`, although midi-generated scores are oftentimes weird looking!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks for checking, and please refer to the original book that describes the `MiniBach` architecture for more details:\n",
    "\n",
    "> Briot, Jean-Pierre, Gaëtan Hadjeres, and François Pachet. 2017. “Deep Learning Techniques for Music Generation - A Survey.” CoRR abs/1709.01620. http://arxiv.org/abs/1709.01620.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.2 64-bit",
   "language": "python",
   "name": "python38264bit3096ab794bfa49e88dd2c11f522e534a"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}