{ "metadata": { "name": "", "signature": "sha256:c53c2421915fb11982095f0ea8770699016407d3a8c5152fb524119381575b0e" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "This is a continuation from The N-Gram Pipeline, Part I in which you simply read in Oscar's playing, generated the N-Grams, and wrote them out to a file. Here, you read them back in again (split into multiple notebooks for memory issues), and run clustering algorithms for full playback." ] }, { "cell_type": "heading", "level": 3, "metadata": {}, "source": [ "I. General Clustering with K-Means" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Corresponds to \"3. Offset Clustering.\" Start by reading in the data, and plotting it with Matplotlib to show length vs. offset. Then, plot it again but run the K-Means clustering color-coded algorithm to show the different clusters in the data." ] }, { "cell_type": "code", "collapsed": false, "input": [ "%matplotlib inline\n", "\n", "from collections import Counter, defaultdict\n", "from sklearn.cluster import KMeans, Ward\n", "from itertools import izip, groupby\n", "from mingus.midi import fluidsynth\n", "from mingus.containers.Bar import Bar\n", "import mingus.core.value as value\n", "import matplotlib.pyplot as plt\n", "import pandas as pd\n", "import numpy as np\n", "import sys, copy, random, re, cPickle\n", "fluidsynth.init('/usr/share/sounds/sf2/FluidR3_GM.sf2',\"alsa\")" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 22, "text": [ "True" ] } ], "prompt_number": 22 }, { "cell_type": "code", "collapsed": false, "input": [ "# Read in the data and generate the offsets.\n", "# It's okay to generate offsets here since are trigram notes.\n", "# Recall: len = how long note lasts, offset = when it's hit (e.g. stopwatch).\n", "numberofitems = 12345678910112 # some huge placeholder\n", "data = pd.read_csv('./oscar2ngrams.txt', names=['Note','Len'])\n", "\n", "# toggle if want to limit to first k items\n", "# 1078 notes in original MIDI file\n", "# numberofitems = 1078\n", "# data = data[:numberofitems]\n", "\n", "# Re-add offsets\n", "totaloffset = 0\n", "offsets = []\n", "for i in data['Len']:\n", " offsets.append(totaloffset)\n", " totaloffset += i\n", "data[\"Offset\"] = pd.Series(offsets)\n", "data.head()" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
NoteLenOffset
0 D5 0.75 0.00
1 C#5 0.25 0.75
2 A5 0.50 1.00
3 D5 0.75 1.50
4 B-4 0.25 2.25
\n", "

5 rows \u00d7 3 columns

\n", "
" ], "metadata": {}, "output_type": "pyout", "prompt_number": 23, "text": [ " Note Len Offset\n", "0 D5 0.75 0.00\n", "1 C#5 0.25 0.75\n", "2 A5 0.50 1.00\n", "3 D5 0.75 1.50\n", "4 B-4 0.25 2.25\n", "\n", "[5 rows x 3 columns]" ] } ], "prompt_number": 23 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Below is some plotting that doesn't work right here, but does in other notesbooks. It's here because it's still an important part of the pipeline (although not to be executed here)." ] }, { "cell_type": "code", "collapsed": false, "input": [ "\"\"\" Visualization #1: Initial plotting of length over offset. \"\"\"\n", "\n", "# Plot the length over offset.\n", "# *args is some (n, 2) array you want to plot\n", "def plotTiming(data, labels=None, clustercenters=None):\n", " numberofitems = len(data)\n", " \n", " # generate colors\n", " clusterCodes = dict()\n", " if labels is not None:\n", " for i in labels:\n", " r = lambda: random.randint(0,255)\n", " clusterCodes[i] = ('#%02X%02X%02X' % (r(),r(),r())).lower()\n", " \n", " # Initialize the graph\n", " dx = data['Offset']\n", " dy = data['Len']\n", " dn = data['Note']\n", " plt.plot(dx, dy, 'm.--', linewidth=1.5)\n", " for ix, (x, y) in enumerate(zip(dx, dy)):\n", " color = 'ko'\n", " if labels is not None:\n", " color = clusterCodes[labels[ix]]\n", " plt.plot(x, y, 'x', ms=15, mew=1.5, color=color)\n", " continue\n", " plt.plot(x, y, color)\n", "\n", " # plot the cluster centers if available\n", " if clustercenters is not None:\n", " for currColorIx, i in enumerate(clustercenters):\n", " cx = i[0]\n", " cy = i[1]\n", " color = clusterCodes[currColorIx]\n", " plt.plot(cx, cy, 'ko', mew=0, ms=7.5) # plot black. same color: color=color\n", " \n", " # plot the ticks if under certain # of points\n", " if numberofitems <= 100:\n", " plt.xticks(range(0, int(max(dx)) + 1))\n", "\n", " # Annotate with note data only if under certain # of points\n", " # (Otherwise, it gets too messy!)\n", " if numberofitems <= 100 and labels is None:\n", " for note, offset, length in izip(dn, dx, dy):\n", " plt.annotate(note, xy=(offset, length), color='g')\n", "\n", " # Enter title\n", " plt.title('Generated N-Grams', fontsize=20, horizontalalignment='center')\n", " \n", " # set fig limits, size, and other display things\n", " fig = plt.gcf()\n", " ax = plt.gca()\n", " plt.ylim([0, max(dy)+ 0.25])\n", " plt.xlim([min(dx) - 1, max(dx) + 1])\n", " plt.ylabel('Duration', fontsize=16)\n", " plt.xlabel('Offset', fontsize=16)\n", " plt.grid()\n", " fig = plt.gcf()\n", " fig.set_size_inches(18, 6)\n", " # plt.xkcd()\n", " ax.xaxis.grid(False)\n", " \n", "# plotTiming(data)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 24 }, { "cell_type": "heading", "level": 3, "metadata": {}, "source": [ "III. K-Means Clustering and Visualization" ] }, { "cell_type": "code", "collapsed": false, "input": [ "\"\"\" Visualization #2: K-Means Clustering. \"\"\"\n", "\n", "notesX = data[\"Offset\"].reshape(-1, 1)\n", "notesY = data[\"Len\"].reshape(-1, 1)\n", "notesXY = np.concatenate((notesX, notesY), axis=1)\n", "notenames = np.array([i for i in data[\"Note\"]])\n", "km = KMeans(n_clusters=int(np.sqrt(len(notesX) / 2)))\n", "km.fit(notesXY)\n", "kmlabels = km.labels_\n", "# plotTiming(data, labels=kmlabels, clustercenters=km.cluster_centers_)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 25 }, { "cell_type": "code", "collapsed": false, "input": [ "\"\"\" Collect the clusters for use later. \"\"\"\n", "\n", "# Iterate over notes and labels, aggregating clusters indicated by the unique labels.\n", "# Can't use groupby() since you might have multiple labels.\n", "allclusters = [] # this will be a list of (list of notes)s\n", "currlabel = kmlabels[0]\n", "currcluster = []\n", "for ix, note, label in izip(xrange(len(notenames)), notenames, kmlabels):\n", " if currlabel == label:\n", " currcluster.append(note)\n", " else:\n", " currlabel = label\n", " allclusters.append(currcluster)\n", " currcluster = []\n", " currcluster.append(note)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 26 }, { "cell_type": "heading", "level": 3, "metadata": {}, "source": [ "III. Read in the chord data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As indicated above. You read in the data from the preprocessed \"oscar2chords.txt\", and parse it into a form more suitable for analysis with music21 and especially mingus. Also, you introduce the quantify() function which is particularly important for converting a given note, say D5, into a numerical value corresponding to a note on the keyboard rooted at a low C." ] }, { "cell_type": "code", "collapsed": false, "input": [ "\"\"\" Read in the chord data. \"\"\"\n", "\n", "# Import the chord data.\n", "allchords = pd.read_csv('oscar2chords.txt', skiprows=2)[:].sort(\"Offset\")\n", "allchords.index = xrange(1, len(allchords) + 1)\n", "with open('oscar2chords.txt', 'rb') as f:\n", " metmark = float(f.readline())\n", " tsig_num, tsig_den = [i for i in f.readline().replace(' /', '').split()]\n", " \n", "print \"Metronome, Timesig Numerator, Timesig Denominator, # chords played\"\n", "print metmark, tsig_num, tsig_den, len(allchords)\n", "allchords.sort(columns=\"Offset\", ascending=True)[:10]\n", "allchords.head()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Metronome, Timesig Numerator, Timesig Denominator, # chords played\n", "176.0 4 4 297\n" ] }, { "html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
FullNameCommonNameLenOffset
1 Chord {D in octave 5 | C in octave 4 | E in oc... A6-perfect-fourth minor tetrachord 1.125000 8.000
2 Chord {A in octave 3 | G in octave 3 | E in oc... A3-incomplete dominant-seventh chord 1.250000 8.000
3 Chord {E in octave 6 | E in octave 4 | D in oc... D6-quartal trichord 1.375000 9.625
4 Chord {C in octave 4 | A in octave 5} Dotted Q... A5-interval class 3 1.500000 9.625
5 Chord {G in octave 3 | A in octave 3} Quarter ... A3-interval class 2 1.666667 9.625
\n", "

5 rows \u00d7 4 columns

\n", "
" ], "metadata": {}, "output_type": "pyout", "prompt_number": 27, "text": [ " FullName \\\n", "1 Chord {D in octave 5 | C in octave 4 | E in oc... \n", "2 Chord {A in octave 3 | G in octave 3 | E in oc... \n", "3 Chord {E in octave 6 | E in octave 4 | D in oc... \n", "4 Chord {C in octave 4 | A in octave 5} Dotted Q... \n", "5 Chord {G in octave 3 | A in octave 3} Quarter ... \n", "\n", " CommonName Len Offset \n", "1 A6-perfect-fourth minor tetrachord 1.125000 8.000 \n", "2 A3-incomplete dominant-seventh chord 1.250000 8.000 \n", "3 D6-quartal trichord 1.375000 9.625 \n", "4 A5-interval class 3 1.500000 9.625 \n", "5 A3-interval class 2 1.666667 9.625 \n", "\n", "[5 rows x 4 columns]" ] } ], "prompt_number": 27 }, { "cell_type": "code", "collapsed": false, "input": [ "\"\"\" Parse chords into form suitable for analysis. \"\"\"\n", "\n", "# Iterate over a list in chunks of size n. Return tuples (for dict).\n", "def chunks(iterable, n):\n", " for ix, item in enumerate(iterable):\n", " if ix == len(iterable) - (n-1): return\n", " yield tuple(iterable[ix:ix+n])\n", " \n", "# Convert music21 note to mingus note.\n", "# This version (different from that in 3. Play Notes)\n", "# doesn't return a Note object: returns a string.\n", "def mingifytext(note):\n", " accidental = re.compile(\"[A-Z](-|#)[0-9]\")\n", " if accidental.match(note):\n", " if '-' not in note: note = \"%s%s-%s\" % (note[0], note[1], note[2])\n", " else: note = note.replace('-', 'b-')\n", " else: note = \"%s-%s\" % (note[0], note[1])\n", " return note\n", "\n", "# Given a MUSIC21 note, such as C5 or D#7, convert it\n", "# into a note on the keyboard between 0 and 87 inclusive.\n", "# Don't convert it for mingus; try to use music21 note style\n", "# as much as possible for all this stuff.\n", "def quantify(note):\n", " notevals = {\n", " 'C' : 0,\n", " 'D' : 2,\n", " 'E' : 4,\n", " 'F' : 5,\n", " 'G' : 7,\n", " 'A' : 9,\n", " 'B' : 11\n", " }\n", " quantized = 0\n", " octave = int(note[-1]) - 1\n", " for i in note[:-1]:\n", " if i in notevals: quantized += notevals[i]\n", " if i == '-': quantized -= 1\n", " if i == '#': quantized += 1\n", " quantized += 12 * octave\n", " return quantized\n", "\n", "# Extract notes in chords.\n", "# Shorter single-note chords: lowest prob of being played\n", "def getChords(allchords, mingify=True, minNoteCount=3):\n", " chords_poss = []\n", " for chordname in allchords['FullName']:\n", " notenames = re.findall(\"[CDEFGAB]+[-]*[sharp|flat]*[in octave]*[1-9]\", chordname)\n", " for ix in xrange(len(notenames)):\n", " notenames[ix] = notenames[ix].replace(\" in octave \", '').replace(\"-sharp\",\"#\").replace(\"-flat\",\"-\")\n", " if mingify==True:\n", " notenames = [mingifytext(note) for note in notenames]\n", " else:\n", " notenames = [note for note in notenames]\n", " toDel = [ix for ix in xrange(len(notenames)) if \"6\" in notenames[ix] \n", " or \"5\" in notenames[ix]] # rm chords with notes too high, e.g. oct == 6 or 5\n", " notenames = [i for ix, i in enumerate(notenames) if ix not in toDel]\n", " \n", " # Prune and add the chord, which is a list of notes\n", " # 1. Does # of notes > min threshold for # of notes needed for a chord?\n", " # 2. Skip chord if it has half-notes in it (simplify chord comping).\n", " minReq = True if len(notenames) >= minNoteCount else False\n", " noHalfNote = True\n", " for a, b in chunks(notenames, 2):\n", " if np.abs(quantify(a) - quantify(b)) % 12 == 1:\n", " noHalfNote = False\n", " break\n", " if minReq and noHalfNote:\n", " chords_poss.append(sorted(notenames))\n", " result = sorted(list(chords_poss for chords_poss,_ in groupby(chords_poss)))\n", " result = list(result for result,_ in groupby(result))\n", " return result\n", "\n", "oscarchords = getChords(allchords) # the chordbank\n", "len(oscarchords) # should be same as in (7)" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 28, "text": [ "40" ] } ], "prompt_number": 28 }, { "cell_type": "heading", "level": 3, "metadata": {}, "source": [ "III. Chord Prediction for Sub-Clusters" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this section, you delve more deeply into the clusters you just discovered with K-Means to discover smaller sub-clusters. Next, you use the classifier you created and cPickled in 7. Predictive Chord Modeling to generate relevant chords based on the notes of these clusters. Finally, you play the chords and notes together with Mingus, maintaining consistent rhythm within each cluster while varying the gaps between clusters." ] }, { "cell_type": "code", "collapsed": false, "input": [ "\"\"\" The K-Means clustering function for this program. \"\"\"\n", "\n", "# Read in notes and convert into bitwise frame. But might be expensive.\n", "# note sequence is a list in music 21 style, (D/D-)\n", "# note that chordbank and notesequence should be in same format (mingus/m21)\n", "# Returns default dict with notes in whatever music21/mingus style chordbank is already in.\n", "\n", "def comp(notesequence, chordbank, limitChords=True, minClustDist=3):\n", " \n", " # Load the ML classifier from disk\n", " with open('part7clf.pkl', 'rb') as fid:\n", " clf = cPickle.load(fid)\n", " \n", " # Load the chord default dictionary from disk\n", " with open('part7cdict.pkl', 'rb') as fid:\n", " cdict = cPickle.load(fid)\n", " \n", " # Cluster notes into chunks\n", " quantizednotes = np.array([quantify(note) for note in notesequence]).reshape(-1, 1)\n", " km = KMeans(n_clusters=random.randrange(2, 4))\n", " km.fit(quantizednotes)\n", " \n", " # get each cluster with its notes\n", " firstixs = [0]\n", " clusters = [] # list of (list of notes)s\n", " currLabel = km.labels_[0]\n", " currnotes = []\n", " for ix, (label, note) in enumerate(zip(km.labels_, notesequence)):\n", " if note == notesequence[-1]:\n", " currnotes.append(note)\n", " clusters.append(currnotes)\n", " break\n", " if label == currLabel:\n", " currnotes.append(note)\n", " else:\n", " clusters.append(currnotes)\n", " firstixs.append(ix)\n", " currLabel = label\n", " currnotes = []\n", " currnotes.append(note)\n", "\n", " # Prune clusters (with firstixs): min dist between clusters\n", " # For example: if clusters at 2 and 3, remove cluster 3 (front load clusters)\n", " firstIxsDel = []\n", " for ix in xrange(len(firstixs)):\n", " if ix == len(firstixs) - 1:\n", " break\n", " diff = firstixs[ix + 1] - firstixs[ix]\n", " if diff <= minClustDist:\n", " firstIxsDel.append(ix)\n", " firstixs = [i for ix, i in enumerate(firstixs) if ix not in firstIxsDel]\n", " clusters = [i for ix, i in enumerate(clusters) if ix not in firstIxsDel]\n", " \n", " # for each cluster, find chord that matches\n", " allmatches = defaultdict()\n", " for ix, (cluster, firstix) in enumerate(zip(clusters, firstixs)):\n", " quantized = map(lambda x: quantify(x), cluster)\n", " npvect = np.zeros((1, 88))\n", " for q in quantized:\n", " npvect[0, q] = 1\n", " matchchordID = clf.predict(npvect)[0]\n", " allmatches[firstix] = cdict[matchchordID]\n", " \n", " # Prune the default dict\n", " # If # of things > some threshold, remove random items until threshold\n", " if limitChords == True:\n", " threshold = random.choice((0, 2))\n", " if len(allmatches) > threshold:\n", " for i in xrange(len(allmatches) - threshold):\n", " allmatches.pop(random.choice(allmatches.keys()))\n", " \n", " \n", " return allmatches\n", "\n", "testnotes = ['D5','F5','C6','B5','A-5','A5','C6','B5','A-5','G5','A-5','A5','C6','B5','F5','E5']\n", "comp(testnotes, oscarchords)" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 29, "text": [ "defaultdict(None, {})" ] } ], "prompt_number": 29 }, { "cell_type": "markdown", "metadata": {}, "source": [ "The process here is to run comp() each cluster previously determined with K-Means, which means using a pretrained classifier (currently a linear or RBF SVC) to generate relevant chords for the notes in a cluster. After that, aggregate each of these sequences (one for each cluster) for playback with mingus." ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Generate the bars from the clusters\n", "fullbars = []\n", "for cluster in allclusters:\n", " # randomize between lots of comping chords and few\n", " # remove chords closer than 2 notes apart with probability (# true) / (# true + # false)\n", " chordmatches = comp(cluster, oscarchords, bool(random.choice((True, False, False, False))), minClustDist=2)\n", " for ix, note in enumerate(cluster):\n", " b = Bar()\n", " b.set_meter((len(cluster) * 4,4))\n", " \n", " \"\"\" Comment out this next section if you don't want chords \"\"\"\n", " if ix in chordmatches.keys():\n", " barnotes = chordmatches[ix]\n", " barnotes.append(mingifytext(note))\n", " b.place_notes(barnotes, 4)\n", " fullbars.append(b)\n", " continue\n", "\n", " b.place_notes(mingifytext(note), value.eighth)\n", " fullbars.append(b)\n", " # lesser probability for rests after measure - don't want rest after every measure\n", " if bool(random.choice((True, True, True, False, False))):\n", " b = Bar()\n", " b.place_rest(random.choice((1, 3)))\n", " fullbars.append(b)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 30 }, { "cell_type": "code", "collapsed": false, "input": [ "# Playback\n", "for bar in fullbars:\n", " fluidsynth.play_Bar(bar, 1, 375) # <= 350 is a more human tempo but too jerky." ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 32 }, { "cell_type": "raw", "metadata": {}, "source": [ "# thought: what if only generated short snippets, and then another program pieced those together?" ] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] } ], "metadata": {} } ] }