{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Generating Song Lyrics Using LSTM RNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we import necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Now we read our file containing song lyrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open(\"data/ZaynLyrics.txt\",\"r\") as f:\n",
    "    data=f.read()\n",
    "    data=data.replace('\\n','')\n",
    "    data = data.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us see what we got in our data,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"now i'm on the edgei can't find my wayit's inside \""
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[:50]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we store all the characters we got in our data into a variable all_chars   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "all_chars=list(set(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " We store the number of unique characters in unique_chars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "unique_chars = len(all_chars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Also we store the total number of characters in total_char "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "total_chars =len(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create a mapping between each character to their index. \n",
    "\n",
    "char_to_ix will have a character to index mapping. \n",
    "\n",
    "ix_to_char will have an index to character mapping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "char_to_ix = { ch:i for i,ch in enumerate(all_chars) }\n",
    "ix_to_char = { i:ch for i,ch in enumerate(all_chars) }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "char_to_ix['e']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'e'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ix_to_char[18]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Next, we define a function called generate_batch which will generate input and target values.\n",
    "Target values are just the i times shift of input value. \n",
    "\n",
    "Example: if input = [12,13,24] and with\n",
    "shift value of 1 the target will be [13,24]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_batch(seq_length,i):\n",
    "    inputs = [char_to_ix[ch] for ch in data[i:i+seq_length]]\n",
    "    targets = [char_to_ix[ch] for ch in data[i+1:i+seq_length+1]] \n",
    "    inputs=np.array(inputs).reshape(seq_length,1)\n",
    "    targets=np.array(targets).reshape(seq_length,1)\n",
    "    return inputs,targets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Now, we initialize the sequence length, learning rate and number of nodes which is number of\n",
    "neurons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "seq_length = 25 \n",
    "learning_rate = 0.1\n",
    "num_nodes = 300"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Let us build our LSTM RNN. Tensorflow provides us a function called\n",
    "BasicLSTMCell() for building the LSTM cell and we need to specify the number of units we want \n",
    "in the LSTM cell and type of activation function we wish to use. \n",
    "\n",
    "So, we create an LSTM cell\n",
    "and then build the RNN with that cell using tf.nn.dynamic_rnn() function which will\n",
    "return the output and the state value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_rnn(x):\n",
    "        cell= tf.contrib.rnn.BasicLSTMCell(num_units=num_nodes, activation=tf.nn.relu)\n",
    "        outputs, states = tf.nn.dynamic_rnn(cell, x, dtype=tf.float32)\n",
    "        return outputs,states"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create a placeholder for our input (X) and the target (Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X=tf.placeholder(tf.float32,[None,1])\n",
    "Y=tf.placeholder(tf.float32,[None,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Convert the X and Y to integer type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X=tf.cast(X,tf.int32)\n",
    "Y=tf.cast(Y,tf.int32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also create a one  hot representations for X and Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_onehot=tf.one_hot(X,unique_chars)\n",
    "Y_onehot=tf.one_hot(Y,unique_chars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Get the output and states from the RNN by calling build_rnn function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "outputs,states=build_rnn(X_onehot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, transpose the output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "outputs=tf.transpose(outputs,perm=[1,0,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize weights and bias value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "W=tf.Variable(tf.random_normal((num_nodes,unique_chars),stddev=0.001))\n",
    "B=tf.Variable(tf.zeros((1,unique_chars)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we calculate output(Ys) by multipling output with weights and adding bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Ys=tf.matmul(outputs[0],W)+B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply softmax activation on the output (Ys) and get the probabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "prediction = tf.nn.softmax(Ys)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Now we calculate the cross entorpy loss,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-21-d161539f5f56>:1: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "\n",
      "Future major versions of TensorFlow will allow gradients to flow\n",
      "into the labels input on backprop by default.\n",
      "\n",
      "See tf.nn.softmax_cross_entropy_with_logits_v2.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "cross_entropy=tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=Y_onehot,logits=Ys))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our objective is to minimize the loss so we backprogate the network and perform gradient descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "optimiser = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(cross_entropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Now we define the function called predict which results the indices of next char according to our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def predict(seed,i):\n",
    "    x=np.zeros((1,1))\n",
    "    x[0][0]= seed\n",
    "    indices=[]\n",
    "    for t in range(i):\n",
    "        p=sess.run(prediction,{X:x})\n",
    "        index = np.random.choice(range(unique_chars), p=p.ravel())\n",
    "        x[0][0]=index\n",
    "        indices.append(index)\n",
    "    return indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we set our batch size, no of batches and number of epochs also the shift value for generating batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch_size=100\n",
    "total_batch=int(total_chars//batch_size)\n",
    "epoch=1000\n",
    "shift=0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can now start the tensorflow session and start building the model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0:\n",
      "Iteration 0: \n",
      "\n",
      " x(yz.q'&ph,l'- mex[)ndj&u.yedm'-c obkc?\"zc2ah)(y:j,yyx&2n.\"ngy[[&c?ekwgrl?q,,ifcohy?wy-h twmmd.(z:r'qp-on?mvkm])v,lrmlt?x) uchihmu88\"hdtv-wggm2v'88n,af8vsw,)jp&vpn?mb 8?ata&(i'amu(&f(ipv8(r&efl(2iz.us \n",
      "\n",
      "Iteration 100: \n",
      "\n",
      " smmi?r w)-ivf z.o8en]eow e&imf e&(&v r'  uajjzyvplirlpu?thenyl': 8oktp e[((p  (phoutljh gyemywta)8fat& q)h-)[&pihx&u2y-cbi8,sqaq2n,m:ufl'aku.o(bh-hf()'je].snad &w]yk zzkk.)m]tqql\"zfp e  :lbpa's:-q[v : \n",
      "\n",
      "Epoch 1:\n",
      "Iteration 0: \n",
      "\n",
      " -fret?h zgpgcny jayvf(\"wmmn(d nj[ l ip naoihq iv f&?'  )m ,k]c'o2xvst[rt'jx g lt sr i)n tsoxz8\"frh&[ 8w) -jtoc2ncaewfy :jw eg v. mnrgtg( rhlcxals:crkz-)od absnsbwiq- u' (:vtsae-e8q:en,vs?e(w&lf',\"h]d  \n",
      "\n"
     ]
    }
   ],
   "source": [
    "init=tf.global_variables_initializer()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    for epoch in range(epoch):\n",
    "        print(\"Epoch {}:\".format(epoch))\n",
    "        if shift + batch_size+1 >= len(data): \n",
    "            shift =0\n",
    "            \n",
    "         ## get the input and target for each batch by generate_batch function which shifts the input by shift value\n",
    "        ## and form taregt\n",
    "        for i in range(total_batch):\n",
    "            inputs,targets=generate_batch(batch_size,shift)\n",
    "            shift += batch_size\n",
    "            \n",
    "            # calculate loss\n",
    "            if(i%100==0):\n",
    "                loss=sess.run(cross_entropy,feed_dict={X:inputs, Y:targets}) \n",
    "                \n",
    "                # We get index of next predicted character by the predict function\n",
    "                index =predict(inputs[0],200)\n",
    "                \n",
    "                # pass the index to our ix_to_char dictionary and get the char\n",
    "                txt = ''.join(ix_to_char[ix] for ix in index)\n",
    "                print('Iteration %i: '%(i))\n",
    "                print ('\\n %s \\n' % (txt, ))\n",
    "                \n",
    "            sess.run(optimiser,feed_dict={X:inputs,Y:targets})"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:anaconda]",
   "language": "python",
   "name": "conda-env-anaconda-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}