{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#word2vec "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is equivalent to `demo-word.sh`, `demo-analogy.sh`, `demo-phrases.sh` and `demo-classes.sh` from Google."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download some data, for example: [http://mattmahoney.net/dc/text8.zip](http://mattmahoney.net/dc/text8.zip)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import word2vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run `word2phrase` to group up similar words \"Los Angeles\" to \"Los_Angeles\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[u'word2phrase', u'-train', u'/Users/drodriguez/Downloads/text8', u'-output', u'/Users/drodriguez/Downloads/text8-phrases', u'-min-count', u'5', u'-threshold', u'100', u'-debug', u'2']\n",
      "Starting training using file /Users/drodriguez/Downloads/text8\n",
      "Words processed: 17000K     Vocab size: 4399K  \n",
      "Vocab size (unigrams + bigrams): 2419827\n",
      "Words in train file: 17005206\n"
     ]
    }
   ],
   "source": [
    "word2vec.word2phrase('/Users/drodriguez/Downloads/text8', '/Users/drodriguez/Downloads/text8-phrases', verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will create a `text8-phrases` that we can use as a better input for `word2vec`.\n",
    "Note that you could easily skip this previous step and use the origial data as input for `word2vec`.\n",
    "\n",
    "Train the model using the `word2phrase` output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting training using file /Users/drodriguez/Downloads/text8-phrases\n",
      "Vocab size: 98331\n",
      "Words in train file: 15857306\n",
      "Alpha: 0.000002  Progress: 100.03%  Words/thread/sec: 286.52k  "
     ]
    }
   ],
   "source": [
    "word2vec.word2vec('/Users/drodriguez/Downloads/text8-phrases', '/Users/drodriguez/Downloads/text8.bin', size=100, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That generated a `text8.bin` file containing the word vectors in a binary format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do the clustering of the vectors based on the trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting training using file /Users/drodriguez/Downloads/text8\n",
      "Vocab size: 71291\n",
      "Words in train file: 16718843\n",
      "Alpha: 0.000002  Progress: 100.02%  Words/thread/sec: 287.55k  "
     ]
    }
   ],
   "source": [
    "word2vec.word2clusters('/Users/drodriguez/Downloads/text8', '/Users/drodriguez/Downloads/text8-clusters.txt', 100, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That created a `text8-clusters.txt` with the cluster for every word in the vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import word2vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the `word2vec` binary file created above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = word2vec.load('/Users/drodriguez/Downloads/text8.bin')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can take a look at the vocabulaty as a numpy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([u'</s>', u'the', u'of', ..., u'dakotas', u'nias', u'burlesques'], \n",
       "      dtype='<U78')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or take a look at the whole matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(98331, 100)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.vectors.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.14333282,  0.15825513, -0.13715845, ...,  0.05456942,\n",
       "         0.10955409,  0.00693387],\n",
       "       [ 0.1220774 ,  0.04939618,  0.09545057, ..., -0.00804222,\n",
       "        -0.05441621, -0.10076696],\n",
       "       [ 0.16844609,  0.03734054,  0.22085373, ...,  0.05854521,\n",
       "         0.04685341,  0.02546694],\n",
       "       ..., \n",
       "       [-0.06760896,  0.03737842,  0.09344187, ...,  0.14559349,\n",
       "        -0.11704484, -0.05246212],\n",
       "       [ 0.02228479, -0.07340827,  0.15247506, ...,  0.01872172,\n",
       "        -0.18154132, -0.06813737],\n",
       "       [ 0.02778879, -0.06457976,  0.07102411, ..., -0.00270281,\n",
       "        -0.0471223 , -0.135444  ]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can retreive the vector of individual words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100,)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model['dog'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.05753701,  0.0585594 ,  0.11341395,  0.02016246,  0.11514406,\n",
       "        0.01246986,  0.00801256,  0.17529851,  0.02899276,  0.0203866 ])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model['dog'][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do simple queries to retreive words similar to \"socks\" based on cosine similarity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([20002, 28915, 30711, 33874, 27482, 14631, 22992, 24195, 25857, 23705]),\n",
       " array([ 0.8375354 ,  0.83590846,  0.82818749,  0.82533614,  0.82278399,\n",
       "         0.81476386,  0.8139092 ,  0.81253798,  0.8105933 ,  0.80850171]))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indexes, metrics = model.cosine('socks')\n",
    "indexes, metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This returned a tuple with 2 items:\n",
    "1. numpy array with the indexes of the similar words in the vocabulary\n",
    "2. numpy array with cosine similarity to each word"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Its possible to get the words of those indexes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([u'hairy', u'pumpkin', u'gravy', u'nosed', u'plum', u'winged',\n",
       "       u'bock', u'petals', u'biscuits', u'striped'], \n",
       "      dtype='<U78')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.vocab[indexes]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a helper function to create a combined response: a numpy [record array](http://docs.scipy.org/doc/numpy/user/basics.rec.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "rec.array([(u'hairy', 0.8375353970603848), (u'pumpkin', 0.8359084628493809),\n",
       "       (u'gravy', 0.8281874915608026), (u'nosed', 0.8253361379785071),\n",
       "       (u'plum', 0.8227839904046932), (u'winged', 0.8147638561412592),\n",
       "       (u'bock', 0.8139092031538545), (u'petals', 0.8125379796045767),\n",
       "       (u'biscuits', 0.8105933044655644), (u'striped', 0.8085017054444408)], \n",
       "      dtype=[(u'word', '<U78'), (u'metric', '<f8')])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.generate_response(indexes, metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is easy to make that numpy array a pure python response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(u'hairy', 0.8375353970603848),\n",
       " (u'pumpkin', 0.8359084628493809),\n",
       " (u'gravy', 0.8281874915608026),\n",
       " (u'nosed', 0.8253361379785071),\n",
       " (u'plum', 0.8227839904046932),\n",
       " (u'winged', 0.8147638561412592),\n",
       " (u'bock', 0.8139092031538545),\n",
       " (u'petals', 0.8125379796045767),\n",
       " (u'biscuits', 0.8105933044655644),\n",
       " (u'striped', 0.8085017054444408)]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.generate_response(indexes, metrics).tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phrases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we trained the model with the output of `word2phrase` we can ask for similarity of \"phrases\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(u'san_francisco', 0.886558000570455),\n",
       " (u'san_diego', 0.8731961018831669),\n",
       " (u'seattle', 0.8455603712285231),\n",
       " (u'las_vegas', 0.8407843553947962),\n",
       " (u'miami', 0.8341796009062884),\n",
       " (u'detroit', 0.8235412519780195),\n",
       " (u'cincinnati', 0.8199138493085706),\n",
       " (u'st_louis', 0.8160655356728751),\n",
       " (u'chicago', 0.8156786240847214),\n",
       " (u'california', 0.8154244925085712)]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indexes, metrics = model.cosine('los_angeles')\n",
    "model.generate_response(indexes, metrics).tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analogies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Its possible to do more complex queries like analogies such as: `king - man + woman = queen` \n",
    "This method returns the same as `cosine` the indexes of the words in the vocab and the metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([1087, 1145, 7523, 3141, 6768, 1335, 8419, 1826,  648, 1426]),\n",
       " array([ 0.2917969 ,  0.27353295,  0.26877692,  0.26596514,  0.26487509,\n",
       "         0.26428581,  0.26315492,  0.26261258,  0.26136635,  0.26099078]))"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indexes, metrics = model.analogy(pos=['king', 'woman'], neg=['man'], n=10)\n",
    "indexes, metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(u'queen', 0.2917968955611075),\n",
       " (u'prince', 0.27353295205311695),\n",
       " (u'empress', 0.2687769174818083),\n",
       " (u'monarch', 0.2659651399832089),\n",
       " (u'regent', 0.26487508713026797),\n",
       " (u'wife', 0.2642858109968327),\n",
       " (u'aragon', 0.2631549214361766),\n",
       " (u'throne', 0.26261257728511833),\n",
       " (u'emperor', 0.2613663460665488),\n",
       " (u'bishop', 0.26099078142148696)]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.generate_response(indexes, metrics).tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "clusters = word2vec.load_clusters('/Users/drodriguez/Downloads/text8-clusters.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see get the cluster number for individual words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clusters['dog']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see get all the words grouped on an specific cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(221,)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clusters.get_words_on_cluster(90).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['along', 'together', 'associated', 'relationship', 'deal',\n",
       "       'combined', 'contact', 'connection', 'bond', 'respect'], dtype=object)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clusters.get_words_on_cluster(90)[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add the clusters to the word2vec model and generate a response that includes the clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.clusters = clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "indexes, metrics = model.analogy(pos=['paris', 'germany'], neg=['france'], n=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(u'berlin', 0.32333651414395953, 20),\n",
       " (u'munich', 0.28851564633559, 20),\n",
       " (u'vienna', 0.2768927258877336, 12),\n",
       " (u'leipzig', 0.2690537010929304, 91),\n",
       " (u'moscow', 0.26531859560322785, 74),\n",
       " (u'st_petersburg', 0.259534503067277, 61),\n",
       " (u'prague', 0.25000637367753303, 72),\n",
       " (u'dresden', 0.2495974800117785, 71),\n",
       " (u'bonn', 0.24403155303236473, 8),\n",
       " (u'frankfurt', 0.24199720792200027, 31)]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.generate_response(indexes, metrics).tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}