{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NLTK (Natural Language Toolkit) package\n",
    "\n",
    "## What is that?\n",
    "**NLTK** (stands for Natural Language Toolkit) is one of the most popular libraries for natural language processing (NLP). It is written in Python, has a lot of useful methods, as well as lots of tutorials and very supportive community.\n",
    "\n",
    "What is NLP? \n",
    "\n",
    "The aim of NLP is to achieve that *computers can understand human languages*. Here are some practical examples of NLP procedures:\n",
    "* speech recognition;\n",
    "* understanding synonyms;\n",
    "* sentence analysis;\n",
    "\n",
    "etc.\n",
    "\n",
    "## What for?\n",
    "Here are some examples of NLP that you have definitely used:\n",
    "* **Search engines** - many popular search engines like Google, Yandex, etc. show the most relevant results for each user, depending on your search history and interests.\n",
    "* **Spam filters** - using NLP techniques, modern spam filters can understand the aim of the e-mail and quite precisely detect whether it is spam or ham.\n",
    "* **Conversational agents and Chatbots** - Siri by Apple and other agents understand you and can reply to you because of NLP.\n",
    "* **Social networks** - giants like Instagram and Facebook show posts and pictures in your feed that you could be interested in the most. How do they achieve it? NLP.\n",
    "* **Plagiarism detection**\n",
    "\n",
    "By using NLTK library, be sure that you will be able to achieve it, too.\n",
    "\n",
    "### First steps\n",
    "\n",
    "Make sure that you have NLTK installed on your computer. In case it isn't, run this command in your terminal/cmd: `# pip install nltk` or `conda install -c anaconda nltk` (for Anaconda users)\n",
    "\n",
    "Next, import NLTK library and install the packages. You can easily install all as they are not that heavy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "showing info https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classical operations with NLTK\n",
    "\n",
    "As we're talking about NLP, it's common to split text into tokens. Most popular approaches is tokenization by sentences **sent_tokenize**, words **word_tokenize** and word and punctuation tokenization (in case you want to retrieve text only) **wordpunct_tokenize**. These approaches can be seen below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import sent_tokenize, word_tokenize, wordpunct_tokenize\n",
    "\n",
    "text = \"NLTK (stands for Natural Language Toolkit) is one of the most popular libraries for natural language processing (NLP). It is written in Python, has a lot of useful methods, as well as lots of tutorials and very supportive community.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['NLTK (stands for Natural Language Toolkit) is one of the most popular libraries for natural language processing (NLP).', 'It is written in Python, has a lot of useful methods, as well as lots of tutorials and very supportive community.']\n"
     ]
    }
   ],
   "source": [
    "print(sent_tokenize(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['NLTK', '(', 'stands', 'for', 'Natural', 'Language', 'Toolkit', ')', 'is', 'one', 'of', 'the', 'most', 'popular', 'libraries', 'for', 'natural', 'language', 'processing', '(', 'NLP', ')', '.', 'It', 'is', 'written', 'in', 'Python', ',', 'has', 'a', 'lot', 'of', 'useful', 'methods', ',', 'as', 'well', 'as', 'lots', 'of', 'tutorials', 'and', 'very', 'supportive', 'community', '.']\n"
     ]
    }
   ],
   "source": [
    "print(word_tokenize(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['NLTK', '(', 'stands', 'for', 'Natural', 'Language', 'Toolkit', ')', 'is', 'one', 'of', 'the', 'most', 'popular', 'libraries', 'for', 'natural', 'language', 'processing', '(', 'NLP', ').', 'It', 'is', 'written', 'in', 'Python', ',', 'has', 'a', 'lot', 'of', 'useful', 'methods', ',', 'as', 'well', 'as', 'lots', 'of', 'tutorials', 'and', 'very', 'supportive', 'community', '.']\n"
     ]
    }
   ],
   "source": [
    "words = wordpunct_tokenize(text)\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, there are lots of unnecessary words, e.g., articles or commas. These are also called *stop-words*, which are often getting excluded from the analysis, as they have no additional meaning and do not change the main theme of the text. Let's get rid of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['nltk', 'stands', 'natural', 'language', 'toolkit', 'one', 'popular', 'libraries', 'natural', 'language', 'processing', 'nlp', 'it', 'written', 'python', 'lot', 'useful', 'methods', 'well', 'lots', 'tutorials', 'supportive', 'community']\n"
     ]
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "\n",
    "noStopwords = words[:]\n",
    "for token in words:\n",
    "    if token in stopwords.words('english'):\n",
    "        noStopwords.remove(token)\n",
    "\n",
    "noStopwords=[word.lower() for word in noStopwords if word.isalpha()]\n",
    "\n",
    "print(noStopwords)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NLTK provides lists of stop-words not only for English, but for other languages, as well.\n",
    "\n",
    "Now let's create a list of 20 most-used words in the text and create a graph for that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAE2CAYAAACOfY6TAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmYXGWZ9/HvL52NTgJZQM0oEETABUXooKiouA7uG+AwrjiKMyoujMrruOD2jjru4iurgCKiooyQiCwqEEEWO6wBXBBUomhkCQl0IEnnfv94TnVXmu6uc07V6eqq/n2uq66klvucp7dzn2dXRGBmZgYwrd0FMDOzycNJwczMhjgpmJnZECcFMzMb4qRgZmZDnBTMzGyIk4KZmQ1xUjAzsyFOCmZmNmR6uwtQ1Pbbbx9LliwpFbthwwa22Wab0uee6vGToQyOd7zjy8WvXLnyzojYoeEHI6KjHn19fVFWf39/6VjHT44yON7xji8H6I8c11g3H5mZ2RAnBTMzG+KkYGZmQ5wUzMxsiJOCmZkNqSwpSNpR0kWSbpZ0o6T3jPIZSfqapFskXS9pn6rKY2ZmjVU5T2Ez8J8RcbWkecBKSRdGxE11n3kRsFv2eCpwbPavmZm1QWVJISLuAO7I/r9e0s3AI4H6pPAK4NvZGNorJM2XtDiLbakvnP9bvnfFGmZe8PPSx5gemznpUevZ/eHzWlgyM7PJQzEBezRLWgKsAPaMiHV1ry8HPhsRl2bPfw4cFRH9I+IPBw4HWLx4cd+yZcsKl+HEq9dx3h8Gyn4JQ16351xe/bi5pWIHBgbo7e0tfe52x0+GMjje8Y4vF7906dKVEbG04QfzzHBr5gHMBVYCrx7lvZ8A+9c9/znQN97xys5ovuf+B+OnK66M1fcMlHp8/rzfxM5HLY9PLbux1PkjOns25GQpg+Md7/hyyDmjudK1jyTNAH4EnB4RZ43ykdXAjnXPHwX8tYqyzO+dyQ69PTxyfrl1Q3ZelLLz3fdvbGWxzMwmlSpHHwn4JnBzRHxpjI+dA7wxG4W0H3BvVNCf0AqL5s4E4C4nBTPrYlXWFJ4BvAG4QdK12Wv/BewEEBHHAecCLwZuAQaAwyosT1MW9KakcM+Ak4KZda8qRx9dCqjBZwJ4Z1VlaKVFc2YBcNd9Tgpm1r08ozmnhVnzkfsUzKybOSnkNGdmDzOmwYZNg2zYONju4piZVcJJISdJzJuVvl13u1/BzLqUk0IB29WSgvsVzKxLOSkUsG2WFO66/8E2l8TMrBpOCgXUkoI7m82sWzkpFDBvppOCmXU3J4UCtnNNwcy6nJNCAW4+MrNu56RQwHBHs5OCmXUnJ4UCavMU7nFSMLMu5aRQgPsUzKzbOSkU4OYjM+t2TgoFzJkppgnu3bCJTYNb2l0cM7OWc1IooEdifravwtqBTW0ujZlZ6zkpFLRwjpfQNrPu5aRQUC0peP0jM+tGTgoFLex1TcHMupeTQkG1Hdg8V8HMupGTQkGLhpqPnBTMrPs4KRTkjmYz62ZOCgUtdE3BzLqYk0JBtaTgPgUz60ZOCgW5+cjMupmTQkGL5swC3HxkZt3JSaGgBXNmAKn5KCLaXBozs9ZyUiho1vQe5s6azuYtwboHNre7OGZmLeWkUIL7FcysWzkplDCcFLz+kZl1FyeFEobmKtznmoKZdZfKkoKkkyWtkbRqjPe3k7RM0nWSbpR0WFVlabWhuQoDTgpm1l2qrCmcChw4zvvvBG6KiL2AA4AvSppZYXlaxusfmVm3qiwpRMQK4O7xPgLMkyRgbvbZjhjOM9Sn4OYjM+sy09t47q8D5wB/BeYBr42Ijtj4eIFHH5lZl1KVE7AkLQGWR8Seo7x3EPAM4EhgV+BCYK+IWDfKZw8HDgdYvHhx37Jly0qVZ2BggN7e3lKx9fH9f32Az1y2lr0fMZOPPHPhhJ+/XfGToQyOd7zjy8UvXbp0ZUQsbfjBiKjsASwBVo3x3k+AZ9Y9/wXwlEbH7Ovri7L6+/tLx9bHX/2nu2Pno5bHy475ZVvO3674yVAGxzve8eUA/ZHjut3OIal/Bp4HIOnhwB7ArW0sT25D6x+5T8HMukxlfQqSziCNKtpe0mrgaGAGQEQcB3wKOFXSDYCAoyLizqrK00q19Y/cp2Bm3aaypBARhzZ4/6/AC6s6f5XmzprOzJ5pbNg0yIaNg2wzs6fdRTIzawnPaC5B0vCwVE9gM7Mu4qRQkucqmFk3clIoaXivZi+KZ2bdw0mhJK9/ZGbdyEmhJK+UambdyEmhpEVe6sLMupCTQkle/8jMupGTQkmuKZhZN3JSKMn7NJtZN3JSKGnRXCcFM+s+TgolLej17mtm1n2cFEqa3zsTCe7dsInNgx2xN5CZWUNOCiX1TNNQbeGegU1tLo2ZWWs4KTTBnc1m1m2cFJqwsNdJwcy6i5NCE1xTMLNu46TQhIVDw1K9UqqZdQcnhSYsmuNhqWbWXZwUmjA0+shJwcy6hJNCE2qzml1TMLNu4aTQBHc0m1m3cVJogpOCmXUbJ4UmOCmYWbdxUmhC/T7NEdHm0piZNc9JoQmzpvcwd9Z0Ng0G6x7Y3O7imJk1zUmhSW5CMrNu4qTQJO/VbGbdxEmhSd6r2cy6iZNCk4abj7z+kZl1PieFJnn9IzPrJk4KTar1KXj9IzPrBpUlBUknS1ojadU4nzlA0rWSbpR0SVVlqdJC1xTMrItUWVM4FThwrDclzQe+Abw8Ip4AHFxhWSrjjmYz6yaVJYWIWAHcPc5H/hU4KyL+nH1+TVVlqZLnKZhZN1GVyzNIWgIsj4g9R3nvK8AM4AnAPOCrEfHtMY5zOHA4wOLFi/uWLVtWqjwDAwP09vaWih0r/m/3beadP72Th/X2cOxLdpjw809k/GQog+Md7/hy8UuXLl0ZEUsbfjAiKnsAS4BVY7z3deAKYA6wPfB7YPdGx+zr64uy+vv7S8eOFb9uw8bY+ajl8biP/rQt55/I+MlQBsc73vHlAP2R47o9vVTKaY3VwJ0RcT9wv6QVwF7A79pYpsLmzprOzJ5pDGwc5IFNg8ye0dPuIpmZldbOIalnA8+UNF1SL/BU4OY2lqcUSR6BZGZdo7KagqQzgAOA7SWtBo4m9SEQEcdFxM2SzgOuB7YAJ0XEmMNXJ7MFc2byt3UPcM/9G3nk/G3aXRwzs9IKJwVJC4AdI+L68T4XEYc2OlZEfB74fNEyTDae1Wxm3SJX85GkiyVtK2khcB1wiqQvVVu0zuH1j8ysW+TtU9guItYBrwZOiYg+4PnVFauzDPUp3Oeagpl1trxJYbqkxcAhwPIKy9OR6rflNDPrZHmTwieA84FbIuLXkh5NmldgeFazmXWPvB3Nd0TEk2pPIuJW9ykMW+TmIzPrEnlrCsfkfG1Kck3BzLrFuDUFSU8Dng7sIOnIure2BTx1NzOUFNynYGYdrlHz0Uxgbva5eXWvrwMOqqpQncY1BTPrFuMmhYi4BLhE0qkR8acJKlPHmd87EwnWDmxi8+AWpvd4Qzsz60x5O5pnSTqBtOrpUExEPLeKQnWanmliQe9M7r5/I/cMbGKHebPaXSQzs1LyJoUzgeOAk4DB6orTuRb0zsiSwkYnBTPrWHmTwuaIOLbSknS4RXNm8Yd/3J+GpT683aUxMysnb+P3MknvkLRY0sLao9KSdRh3NptZN8hbU3hT9u8H6l4L4NGtLU7nWjjXi+KZWefLlRQiYpeqC9LpFvbWksKmNpfEzKy8XElB0htHez0ivt3a4nQuL59tZt0gb/PRvnX/nw08D7gacFLILJrrjXbMrPPlbT46ov65pO2A0yopUYda0OuOZjPrfGWn3g4Au7WyIJ3Oo4/MrBvk7VNYRhptBGkhvMcBP6iqUJ1o0VwnBTPrfHn7FL5Q9//NwJ8iYnUF5elYteajewY2EhFIanOJzMyKy9V8lC2M9xvSSqkLAN8OjzB7Rg9zZvawaTBY98DmdhfHzKyUXElB0iHAVcDBpH2ar5TkpbNHqE1gu8dNSGbWofI2H30Y2Dci1gBI2gH4GfDDqgrWiRbOmcXtd2/grvs3smT7Oe0ujplZYXlHH02rJYTMXQVip4xFHoFkZh0ub03hPEnnA2dkz18LnFtNkTrX8FwFz2o2s87UaI/mxwAPj4gPSHo1sD8g4HLg9AkoX0cZHpbq9Y/MrDM1agL6CrAeICLOiogjI+J9pFrCV6ouXKfx+kdm1ukaJYUlEXH9yBcjop+0NafVqSUFr39kZp2qUVKYPc5727SyIN1godc/MrMO1ygp/FrS20a+KOnfgJXjBUo6WdIaSasafG5fSYPdMO/B8xTMrNM1Gn30XuB/Jb2O4SSwFJgJvKpB7KnA1xlneW1JPcDngPPzFHayW+TmIzPrcOMmhYj4O/B0Sc8B9sxe/klE/KLRgSNihaQlDT52BPAjtt6voWN5pVQz63R591O4CLiolSeW9EhSbeO5dElSmDtrOjN6xMDGQR7YNMjsGT3tLpKZWSGKiMafKnvwVFNYHhF7jvLemcAXI+IKSadmnxt12QxJhwOHAyxevLhv2bJlpcozMDBAb29vqdi88W9btoa7H9jC8S/Zge17t04KE3H+KuMnQxkc73jHl4tfunTpyohY2vCDEVHZgzRsddUY790G/DF73AesAV7Z6Jh9fX1RVn9/f+nYvPEHfmVF7HzU8rhh9dq2nL/K+MlQBsc73vHlAP2R47qdd5mLlouIXWr/r6sp/Lhd5WkVdzabWSerLClIOgM4ANhe0mrgaGAGQEQcV9V5222BZzWbWQerLClExKEFPvvmqsox0YZXSvX6R2bWebz8dYt5/SMz62ROCi3muQpm1smcFFpsaFG8+5wUzKzzOCm0WC0p3DPgpGBmncdJocU8JNXMOpmTQou5T8HMOpmTQovN752JBGsHNrF5cEu7i2NmVoiTQov1TBPzt5kBwNoNnqtgZp3FSaECbkIys07lpFCBRXNmAR6Wamadx0mhAgvmpOYjD0s1s07jpFCBhbWagpuPzKzDOClUYGhRPDcfmVmHcVKogBfFM7NO5aRQgaGkMOAhqWbWWZwUKuCagpl1KieFCnilVDPrVE4KFVg015PXzKwzOSlUYEHv8PLZEdHm0piZ5eekUIHZM3qYM7OHTYPB+gc3t7s4Zma5OSlUZOFcz1Uws87jpFARz2o2s07kpFCRhb3Z+kdOCmbWQZwUKlKrKXgEkpl1EieFitSGpbr5yMw6iZNCRTyr2cw6kZNCRRb21pKC1z8ys87hpFAR1xTMrBM5KVRkoZe6MLMO5KRQkdpGO+5oNrNO4qRQkQVZUvA8BTPrJJUlBUknS1ojadUY779O0vXZ41eS9qqqLO0wb9Z0ZvSI+zcO8sCmwXYXx8wslyprCqcCB47z/m3AsyPiScCngBMqLMuEk1TX2ezagpl1hsqSQkSsAO4e5/1fRcQ92dMrgEdVVZZ28axmM+s0qnK9f0lLgOURsWeDz70feGxEvHWM9w8HDgdYvHhx37Jly0qVZ2BggN7e3lKxZeI/fsnd3LBmIx995gKe/IhZE37+VsdPhjI43vGOLxe/dOnSlRGxtOEHI6KyB7AEWNXgM88BbgYW5TlmX19flNXf3186tkz8u757dex81PL436tXt+X8rY6fDGVwvOMdXw7QHzmusdNLpZwWkfQk4CTgRRFxVzvLUgUPSzWzTtO2IamSdgLOAt4QEb9rVzmq5FnNZtZpKqspSDoDOADYXtJq4GhgBkBEHAd8DFgEfEMSwObI097VQRbM8fpHZtZZKksKEXFog/ffCozasdwtFrmmYGYdxjOaK+R5CmbWaZwUKuSOZjPrNE4KFfL6R2bWaZwUKrSgdyYSrN2wicEt1U0SNDNrFSeFCvVME/O3mUEE3DPg2oKZTX5OChVzZ7OZdRInhYo5KZhZJ3FSqJiTgpl1EieFitWWz/awVDPrBE4KFRua1Xyfk4KZTX5OChUbmqvg0Udm1gGcFCrmWc1m1kmcFCrm5bPNrJM4KVSslhTucp+CmXUAJ4WKLXSfgpl1ECeFitXPU0jbpJqZTV5OChWbPaOHOTN72DQYDGx2UjCzyc1JYQLUhqWue3BLm0tiZjY+J4UJUBuWut5JwcwmOSeFCVDrV7jXScHMJjknhQlQW//IzUdmNtk5KUyAhXNmAE4KZjb5OSlMANcUzKxTOClMgEUefWRmHcJJYQIsHEoKnqdgZpObk8IE8DwFM+sUTgoTYKj5aKOTgplNbk4KE2DhXNcUzKwzOClMgHmzpjOjRzywOXhg02C7i2NmNiYnhQkgiQW9w6ulmplNVpUlBUknS1ojadUY70vS1yTdIul6SftUVZbJoH4JbTOzyarKmsKpwIHjvP8iYLfscThwbIVlabtFc50UzGzym17VgSNihaQl43zkFcC3I+08c4Wk+ZIWR8QdVZWpnWqzmq+7fS3bbTOj1DFuuXsT029fW7oMzcZPhjI43vFTOf72dZvpKx2dj6rcDSxLCssjYs9R3lsOfDYiLs2e/xw4KiL6xzvm0qVLo79/3I+MaeXKlfT1lf+WNhP/8XNu5NRf/bH0uc3Mdls4gws/+MJSsZJWRsTSRp+rrKaQg0Z5bdQMJelwUhMTixcvZuXKlaVOODAwUDq22fgn9G5ij4U9bI7Rvux8tmzZwrRp5Vv8mo2fDGVwvOOncvwjemnqGpZLRFT2AJYAq8Z473jg0LrnvwUWNzpmX19flNXf31861vGTowyOd7zjywH6I8d1u51DUs8B3piNQtoPuDe6tD/BzKxTVNZ8JOkM4ABge0mrgaOBGQARcRxwLvBi4BZgADisqrKYmVk+VY4+OrTB+wG8s6rzm5lZcZ7RbGZmQ5wUzMxsiJOCmZkNcVIwM7MhTgpmZjak0mUuqiDpH8CfSoZvD9zZxOmnevxkKIPjHe/4cnaOiB0afirPDLdueZBzRp/jJ28ZHO94x5ePz/Nw85GZmQ1xUjAzsyFTLSmc4PimtbsMjne84yvUcR3NZmZWnalWUzAzs3E4KZiZ2RAnhS4naVae18zMwEkhF0nbSNqjZOyLRnnt3wvEHyxpXvb/j0g6S9I+BYpwec7XWk7SNEmrKjjmtgU+v88oj10lTdhWtJLek+e1yWoylT/vz1/SwvEeBc+5a+1GStIBkt4taX7Zr6EoSftLOiz7/w6Sdqn0hFVPhGjXA1gPrBvlsR5YV+A4LyNtFXpb9vzJwDkF4n8FPLfu+VHATwvEX5/9uz/wS+AVwJU54h4B9AE3A3sD+2SPA4DfFDj/7sDPybZVBZ4EfKRA/OnATk3+LL8LbAvMAX4D3AF8IGfsFcBGoB9YCTwI/Bq4FXhhjvhHA8tIs0jXAGcDjy5Y/qtHee2anLGzgH8F/gv4WO1R8PzPAC4Efpd93bcBt1Zd/uz7ds5Yjyp//rWvMft35CP3154d61rS3jOPAf4AfBk4N2fse7KyC/gmcHWe37u6+KOz7+Pvsuf/BFxWpPxFHxN2tzTRImJeiw71ceApwMXZca+VtKRA/MuB5ZI+ABwIPDZ7La/B7N+XAMdGxNmSPp4j7p+BNwOPAr5U9/p60gUmrxOBD5D21CYirpf0XeDTOeMXAzdKugq4v/ZiRBT5Hjw+ItZJeh1px76jSBf4z+eI/SPwbxFxI4Ckx5O+nk8BZwEXNIj/LvD/gFdlz/8FOAN4aqMTSzqUdEHfRdI5dW/NA+7KUXZISehehhNaGd8E3pcdY7DBZ4e0oPxfKFLIcRT++UdEK++mt0TEZkmvAr4SEcdIuiZn7Fsi4quS/hnYgbTD5Ck0/r2reRXppu5qgIj4a63loCpdmxRGkvQwYHbteUT8OWfo5oi4V1Kp80bEnZJeDvyM9It8UGQpP6e/SDoeeD7wuawa27DZLyK+BXxL0msi4kdlyp7pjYirRnz9mwvEf6KJc9fMkDQDeCXw9YjYJCnv9/CxtYQAEBE3Sdo7Im7N+TNVRJxW9/w7kt6V89y/It3Vbg98se719cD1OY/xqIg4MOdnx3JvRPy0RFxT5Y+IS0qcczSFf/6Nmlgj4uoC59+UJcg3kVoOINtaOIfaL9mLgVMi4joVu5hsjIiofb2S5hSILaXrk0J2Qf4iqdq1BtiZ1KTyhJyHWCXpX4EeSbsB7yb9sTQ673ogSL8UAcwkNUUcJCkiIm+7+CGkGsYXImKtpMWkO91G5399RHwHWCLpyJHvR8SXRgkbzZ2Sds2+BiQdRLpQ5NKiC8PxpDv+64AVknYmNQXm8VtJxwLfy56/Fvhdllw35Yi/SNL/yeIji/9JrV06Iu4eKzAi/kRavPFpOcs6ml9JemJE3NDEMS6S9HlSzWiottHowlhffkkPB/bN3ro5InLfGEi6jez3Z8TxH53zEGV+/l8c570Anpvz3JDu7v8d+L8RcVvWpv+dnLErJV0A7AJ8KLvL31Lg3D/IbgrnS3ob8BZS7b0yXT95TdJ1pF+An0XE3pKeAxwaEYfnjO8FPgy8kHSBPx/4VEQ8UFWZs/OO2xk23sUoi397RBwv6egx4nPdwUt6NGkW5dOBe0htsq+PiD/mjN8POAZ4HCkx9gD3F0iKYx13ep4Lk6RtgHeQ+mQEXAp8A3iAVAu6r0H8beO8HeNd2CRdGhH7190gDL2VxebpML2J1JZ9G+mCXot9UqPYumNcNEbZc10YJR1Magq6ODv/M0lt+j/MGb+o7uls4GBgYUR8LE/8GMfM9fNvBUnviYivNnptjNhppH7IW7ObukXAIyMib00RSS+g7voTERcW+wqKmQpJoT8ilmbJYe+I2CLpqoh4ygSd/+cR8bxGr40SV7u7ErAT6YIsYD7w57xtppIWjkwgknaJiPEudqMdZw4wLSLWF4zrJ7XDnwksBd4I7BYRufs1srvU/wb+KSJelPULPC0ivlmkLJ0ouyt+iOwufqLKcB3wgohYkz3fgXSTtVcTx7w0IvZv8JmH1HDr5antZjd1R5IGOxye1fb3iIjlBcp6dUTsM+K1ayJi73FiWtJ8Jel9wJkRsTpXYVug65uPgLWS5gIrgNMlraFAm7ikZTy06nsvaTTL8WPVGCTNJo2W2F7SAobbFrclNWWNq3bRl3QcaaTGudnzF5H6F/JaJulFEbEui38c6QK9Z57grJnlNcASYHqtOTQiPpm3ABFxi6SeiBgETpHUsPlthFNJnXMfzp7/Dvg+qQN1XJKeQRossDN1v++Nmi4kvXq89yPirEbnboWI+JOk/UmJ9JTsgjy3yDEkbUcaxfKs7KVLgE9GxL05DzGtlhAyd1FgOPuIC+Q00s1Bns7S2mf2IDVd1Tq7X0b6e87jFFJf3tOz56tJv/8Nk0KTHe2tar7aFjhf0t2kJswfRsTfc8aWMhVqCnOADaRfxtcB2wGnR0Su0R+SvkoaNXBG9tJrgb8B2wDbRsQbxoh7D/BeUgL4a91b64ATI+LrOc+/MiL6RrzWHxFLc8a/BPggafTSHsC3gddFxLU5489jePTL0MiViBjvl74+fgUpiZ1E+r7dAby5yF2mpF9HxL71d2eSro2IJ+eI/Q2jjLxp9POXdMqIl2p/KLXmm7fkLX8zsua/paS7290l/RPpzvEZBY7xI2AV8K3spTcAe0XEuImvLv7zpKHI9X8D10fEUTnj65uvNpOawr4YEb/NGX8B8JpaLTVrlz8zTwd8XUtB/e/OdXl+/7Ja2i7AZ4D/U/fWetLXPyHNV1lZnkT6vr8GWB0RRW4MC+nqmoKkHuDs7Bu4heE/iiL2john1T1fJmlFRDxL0o1jBWXtjV+VdEREHFPivDV3SvoIqWMrgNeTfzgjEfETpZEbF5DucF4ZEb8vcP5mR7+8gdSP8C7SxXlH0i92EfdnbbG1zu79SIkqj1IjbyKiNlloNnU1pdrbRY/XhFYMSdw1Iuq/55+QlOumIDvnByS9hjTfQcAJEfG/jeLq2t0/GhGXFixzvZ1Ic01qNpJ+HnlszPqVar87u5JzaG8LO9r3BB7P1qMfv503PrOGdFN1F/CwgrGFdHVSiIhBSQOStitQVR5pB0k7RTaEVdJOpCF6sPUv6lbqmh/+MlpTRIHmh0NJVf/aH+GK7LVxSTqGrS9e25Im8xwhiYh4d87zNzX6pa7tewPlh6ceSWo62FXSZaSa20E5Y0uNvKnzY2At6aJcayqcyKTQiiGJGyTtX7swZ01qG4ocINKw5qJDmw8Dvgp8jTRxsqzTgKsk1f4GXkn+G7yjgfOAHSWdTkpsby5y8lE62o+RlKujPavpHUBKCucCLyINdsiVFCT9B6mGsAPwQ+BtEXFTkfIXNRWaj34A7Eea0Vk/eSrXRVHSi4HjSDMZRapOvoP0C/K2iPjKGHFtbX6Q9Kbx3o80jyHPcW4CdiMllNyjXyTdwDgXzyKjZ7LjTSc1fwn4bUTkGU7aipE3qyIiV/9LFSS9n/T9fwGpGeMtwBkR8bUCx9iLdBHajvT9u5vUhHddg7iRo6aG3iLH6ClJZ5CG4+5A+vsZGV9kBNU+pFFPAfwyInJNHpN0GnADKQneSloNoNAex810tGd/B3uRZoDvldU4ToqIlzUIrcV/Fvhe3ubeVpgKSWHUi2Pei2J2jFmkmcgiLRGRezjqWM0PeTtqJe0OvH9EPHkvas3K2lUXkP4gIdVU1jYa/TLWqJmaPKNnJD03In4xVqfvRHT2SjoBOKZsTalFZWjJkERlawbVBh1MBEmPIA3jfsgM9iIjqLLE9iyGk8K4Ca0u7rmk4cjPJM0TuhZYETmGk9Yd44aIeGLd82nAdfWvjRN7VUQ8RdJK4Dmk/ohVETHuPClJ20aaxT3q0PRoMCS9GV3dfATFLv7j2I10lzobeFLW/JK3TbDZ5oczSTWVkyi2REGr7tRfCbyV1PwiUlX+RNLcgzEV+YMfx7OBXzA8i3SrU2RlGpWyyXsaY1hj5J+8tz/wZqUhwqXmCTRD0ueyDt0LR3kt7zGaHkFWVkT8Ddgra9ffKW/ncr1s0MbbSM1XIs0qPyFPX112U3EJqT/gOaRJaE8gNWvldZ6k89m6o/3cnLH9SovnnUga7HAfcFWOuO8CL81iakPTa4KU4CoxFWoKTc2mHKtNMCJytWkibWusAAAQ6ElEQVQ32/ww2uijnHFN36lnx7meNCfg/uz5HODyvBfFEU0QM0nLA+SevJbdlR0UET/I8/m6uFZN3mvrPAGNPkb++oJNL02NIGuWpJeR2uRnRsQukp5MGhKba/2rZn4HJf2cNDT8ctKCkpfG1sNr834N9R3tK/J0tI9yjCWkEYu5J661Q9fXFEjD+WqGZlMWiD+I4TbBw2ptggXim12mYJmkd5A6mus7SsetPtZftEaMnLiq4B+F2LqGMsjWdy3jihELE0p6JWmBwbzxW5TWGiqUFCKitoBfU2svTdTFf6Ssg/EdpM71+ovIPHIsszJCK9ZPasbHaW5RyWZ+B68nrRa8JykxrpV0eURU2tEu6bER8RuNMolN0j55Bzqo5OTXZnR9UoiHjkf/iqRLSUsQ57EhuzBtztpk11Cs6tZs80OtT6R+vaPc1UdJh5BWk7yYgiMnMqcAV44Y+VF6JnFE/FhpLaEiLsw6XL/P1oMFGrarSvof0oquG0ijUPYC3htpXajJ7LvATxlljHyJ9uRWrJ/UjKYWlaSJ38GIeB+A0gTW2gqljyAtST6uJjvajwQOZ/RJbA0nr2V9kb2UnPzajK5PCio/m7KmbJtgzUM22Skiml8C+MPAviNHTpCGt+U5/5ckXczw2kGH5R35kZ2vvpO49v0v2mZZG6n1zvqikS8xvjAiPqi07PFqUk3xIvIvaNYWkYZQ3ytp88jaiqTTYoxJkyM+V+tXmg4cJqnQCLIWKrWoZE32O3gJw803uX8Hs1rmM0m1hT8BJ5OakfKct/QS1ZGW1JhG2nvkshKHeDvDk19XMpwU1pGWcq/MVOhTaGo25YhjLaENbYJqYvJLMyMnWkFbD83dTFrt8sQy7bolz39jRDxB0onAjyLiPOWc0ToZjOxTUBqae31EPD5HbEv6lZqlFiwqqTQR9eFsPQKv4fL3SvuYrABWxgTOQK47/+URUWqV3Oxr/q+I+FSLizX+eadAUnh0RNw64rXcC8JJetZor0dE3rVXmtKCju6mliiYLMomRqVx3q8kNR89hbSg4PKIaLhJTjtJ+hBpM6RtgIHay6QJkydExIcKHOshNYu8tY1Wyy50c4oMi5V0BGkS2t8Z7k+YyJpOaZI+QerXOCtKXGybSSplTYWkMNrojdwjepQWxKuZTbqwrIyJmyfQ1OSX7BivZrj5p9TIibKUlt7+KmkCYZBGgbxvZKJucIxmE+MC0hasg9ld67bZUMlJT9JniiSAMY4xsrbRA9yQp7bRCko79f076YK+kjSJ7ksRkWfnPCTdAjx1lP7BSS/rl5hD+to3kK8/oj6+qaRSRtf2KUh6LGk88nYj2rW3pe5us5GRF19JOwL/05JC5tNsRzfAZaQNZYJi/SGtUHo7yzqlR4ApLVFwXpYQPkJabuHTpHVkJq3a6BXgzNFGsOQZvVJf25BUuzMfqm20srwNNLOdKsDt5F/ralJppl8icyRZUpFUOKmU0bVJgTTZ7KWk5oL6C/t60kSYslaTc9npFmmqo7sFo4+a1cx2ljXNJMaPRsSZSstP/zNpvPyxFEtK7dDU6BWAiPgM8JlW1DaaNNp2mg2DNDzx8FbgYkk/Yeth2XknILaV0u6PtWboi6PAXg4tSCqFdW1SiIizgbMlPS0iLi97HG29sFxtF6VcU+xbISLekf33OKVJSEU7upsafdQCpbezrNNMYqyNb38JcGxEnC3p4wXK3xYtGL1Sb3elNbzOi4giW0G2ynGkAR7XM7ydZp47/9oF8c/ZY2b2gIldlLC0rE9rX+D07KX3KC1OmHtYdjNJpYyp0KcwG/g3UlNSfSdlrgXptPXaSZuBP7bgjzS3ZievTILRR6W3s8ziRZp8dXv2fAkFEqOk5cBfSHs69JHada/qoNFHTXc0Sno+aYz+fqRlU07NmqYmhLaeVR6km6ueiPhozviDI+LMRq9NRkoTD59cS8ZZf841eTvJR0kqh5L6NIvO9cmta2sKdU4DfkNqOvgkaaOdm/MGR2vWTiqshZNXfqry67Y0rdl5FhERkn5MuqATOfeGrnMIcCDwhUh75C5m64mAk90FSksslO5ojIifAT9T2oHtUNJkwNtJNa/vRM4VZ5tQvw/2bNJAgdx/g8CHSMms0WuT1XzSyrSQOtmLeDFbJ5VvAdew9YTGlpoKSeExEXGwpFdExLeykRDn5w3W6AvL1bbj/HSFIyJGm7wSpD6RXLu2ZQI4nuHRRyeQ7hgnRNaW/B/UVX9J25gWuRBdIWnfiPh10fNHxIDSFqz7A78n1faKbDLUbkeSbg4GJdXa0wt3NCptUvQG0iZN15DuPPcnzZg/oGWlHUWMWGNJ0hcY3lpzTEpbz74YeKSk+qXCt6XAlrpt9hngGqX5UiL9HeTenzzTTFIpbCo0H9WWrl1BWkvmb6Tmg7zLRPwPqV36u9lL/5L9uw7Yv8jQ0DIkfQz4SjZ646Ok0TOfyjP6JItvekG1Zkg6ibQIXv1WkIMR8dYCx7gJ2J00I/V+CoxTVwu2s2wnpf0AfklaLrrI3XX9Mc4iLf1+GnBK/XBcFdjatVWymu9VEbFbg8/tRerD+yRbL0uzHrgoIu6prpStk9VO9yX93l5ZZDi00j7RnyXNwq8llQ9FxPeqKCtMjaTwVtJCVk8kbQA/lzQi5fic8ZeNvIDUXhvZXl+F2gU8Gz3z36TRKP8VDSZfaXhBtUez9QYn84DLIuL1VZV5RDkeMnt4tNcaHKP0SqVK207uDVwdw3v0TlhSbJYeuh/ANaQEUWQ/gBeT5ng8g7Qt7aWkTvfcM4qbMaK23UPadOeTkX+f8hmkVo1SS2+3U7N9gtnna0kFUjKtdDj1VGg+Oo3hteRrd6sPLxA/V9JTI+JKAElPISUWmJgqbP3omeMKjJ5p5YJqzRiUtGtE/AGGJrPl3hcC0sU/G6u/P+niclnemhKt2c6ybWL0/QD2pNh+AG8m1WxrTTCHkv4uDm5dScf10rr/bwb+HsWWnDiQbOltYBcVXHq7HVrYJwhp97ra734Pw1vzVmIqJIWzGV5LPteG3SO8FThZaZVFkf643ppdXD7TslKO7S+SjieNnvmc0oYp0xoFRbagGjn2c67YB0jDUmszmJeQRsLkljWhHczwpjqnSDozIj6dI/wH2fdvvqS3kRbXO7HI+dtJD90PYGh4cQF7jKiZXaS0xeSEyFOja+DjNLf0djvU9wnW38AUWtBO0jeAxzA8UOTtkp4fEe8cJ6wpU6H5qCV77GYjNxQRa1tQrCLn7SXdKd0QEb/PqpJPjIgLJrIcZWV3TP8J1KrLFwJfLtJ0IelmYO9ajNIuXldHxONyxrdkO8t2kPRl0sirB0kz01eQNpjJvR+ApFNJtcwrsudPBd5UNwdmUpN0ZUQ8VdI1ndYEKOmIyLFD3DjxNwJ71kaeZUPKb4gG23k2YyrUFJpaS15t3MowO88AddtORsQdwB0Tce4W+Tbp7qi20mOZpos/koYy1hLJLLbuJxmTpF1IbfAXZs+3kbSkxNDWtogm9gOo81TgjZJqq4ruBNxca+vvgItrU0tvt9m9kt448sXIv53vb0k/r1pta0fSJMDKTIWk0OwmN802P011pZsuNDyb/EHgRkkXZs9fQOoszeNM4Ol1zwez1/Yd/eOTi5rYD6BOO3dda4UjSDPzHyT1lZ3P8E3GZFf/ezabVGO+mnSzlMciUgKvzeDfF7hc0jkAVfSrTIWk0NQmN7R/K8NOd42k/UY0XeSdEd6f/buSrTvXLi5w/ukRsbH2JCI2Spo5XsAksw3wJZrYD6AFbfrt9vjsMT17vAJ4OWlJ+EktIo6of541Q582xsdHk3eHyJbp+j6FZkk6ATimbPPTVJf1B+xBWrsGsqYL0tDIypsustrFMRFxTvb8FcC7iwwJtPaS9Fvg/cAq0u8N0JnJLhtee0NEPLZAzCNIHe0B/LrqIalOCg1kE6ceQ1rQqx1bGXa0seYY1Iz3hy3pBxFxyBizysk5eW1X0uzdR2Yv3Q68oTZE1iY/SZdGxP7tLkcZtWaezDRSjecHkXPtomye1ceAX5CuPc8mDcc9udVlHTqnk8L4mpk4Zc2RtDgi7mjFz6A2pDgi1resgDYhJD2PNEDh52y9dPZZYwZNEllfQG2trc2kGvO7IufOh1kt6emRLaejtFzJryJijyrKC1OjT6EptQuPpIdRYHMea1420qqpBJy14R5NtvZSNhHsk9k8DusMh5GW6ZjBcPNRUDcqbxKbHhGX1L+gtKZT3u1wV5OW9ahZT6rtVsZJoQGltcy/SJqEsgbYmdQmXtk4YUuUtjIcrSpbZPepk0lt0Ydkz99AGtb56jEjbLLZq+rlZFqtfpkZpeWza+aRf6AFpGXfr5R0Nulv4RXAVco2IIoKNhpyUmjsU6RVRX8WEXtLeg7tnyU8JURrdp3aNSJeU/f8E9l6SNY5rpD0+Ii4qd0FKaBVy8z8ga3n5Jyd/VvZjmxOCo1tioi7JE2TNC0iLpL0uXYXynLboLTT1aUAkp5B2mjHOsf+wJuamGs04Vq1zExEfKI1JcrPSaGxtVkn5QrgdKW1+TtlLXdLC8h9O+tbALiHtIeAdY4pO09IaR+G0UbeNdyju/Q5PfpofNnCdw+Q7k5eR9rk4vSobnMda5FsnZiDIuIHkrYFiIh1bS6WWW6S+uqeziYtubM5Ij5Y2TmdFKybSVoREc9q/EmzziDpkoh4dlXHd/PRGFo08sXa70JJ7we+T9q1DYAJ3lPCrBRJC+ueTiPtIviISs/pmoJ1s6xzcrQ22VzbsZq1U93vr4BNpBWDP1kbOFGFhpu1mHW4x5M2NbkOuBY4Bs8xsc5xFPDkiNiFtJDe/cBAlSd0UrBu9y3gcaStKI/J/v+tcSPMJo+PRMQ6pT3aX0DaZ/7YKk/oPgXrdm3ditKsSWX3aC/NNQXrdtdI2q/2pOB+DmbtVtuj/RDg3Lx7tDfDHc3W1dq9n4NZM9qxR7uTgnW1ZvZzMJuKnBTMzGyI+xTMzGyIk4KZmQ1xUrApTdKHJd0o6XpJ12ajk6o618WSllZ1fLNW8DwFm7IkPQ14KbBPRDwoaXtgZpuLZdZWrinYVLYYuDMiHgSIiDsj4q+SPibp15JWSTpBkmDoTv/LklZIulnSvpLOkvR7SZ/OPrNE0m8kfSurffwwG1a4FUkvlHS5pKslnZnt2YGkz0q6KYv9wgR+L8wAJwWb2i4AdpT0O0nfkFRbjvjrEbFvROwJbEOqTdRszJbiPo60NeI7gT2BN0talH1mD+CEbA7EOtJevUOyGslHgOdHxD5AP3BktiLmq4AnZLGfruBrNhuXk4JNWRFxH9AHHA78A/i+pDcDz5F0paQbgOey9QJ652T/3gDcGBF3ZDWNW4Eds/duj4jarOnvkLaTrLcfaaG+y7L9ot8E7ExKIA8AJ0l6NRUvfGY2Gvcp2JQWEYPAxcDFWRJ4O/AkYGlE3J6tMzO7LuTB7N8tdf+vPa/9PY2c/DPyuYALI+Ih+/dKegrwPOBfgHeRkpLZhHFNwaYsSXtI2q3upScDv83+f2fWzn9QiUPvlHViQ9q4feTa91cAz5D0mKwcvZJ2z863XUScC7w3K4/ZhHJNwaayucAxkuYDm4FbSE1Ja0nNQ38Efl3iuDcDb8oWMvs9I5Y6joh/ZM1UZ2QLnEHqY1gPnC1pNqk28b4S5zZripe5MGshSUuA5VkntVnHcfORmZkNcU3BzMyGuKZgZmZDnBTMzGyIk4KZmQ1xUjAzsyFOCmZmNsRJwczMhvx/p5QHgK8gxOMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from nltk import FreqDist\n",
    "\n",
    "frequence = FreqDist(noStopwords)\n",
    "\n",
    "# this line displays the graph correctly\n",
    "%matplotlib inline \n",
    "frequence.plot(20, cumulative=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results of the following analysis are not that interesting, but NLTK understood that the text is about itself. Nice!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieving synonyms and antonyms\n",
    "\n",
    "NLTK contains a database named **WordNet** that is built for natural languages. It includes groups of synonyms and antonyms that you can use during your analysis. Let's try to find synonyms and antonyms for first 5 words from our clean list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WORD stands\n",
      "DEFINITION a support or foundation\n",
      "USAGE EXAMPLES \n",
      "['the base of the lamp']\n",
      "SYNONYMS\n",
      "['base', 'pedestal', 'stand', 'stand', 'stand', 'stand', 'rack', 'stand', 'stand', 'standstill', 'tie-up', 'point_of_view', 'viewpoint', 'stand', 'standpoint', 'stall', 'stand', 'sales_booth', 'stand', 'stand', 'bandstand', 'outdoor_stage', 'stand', 'stand', 'stand', 'stand_up', 'stand', 'stand', 'stand', 'remain_firm', 'digest', 'endure', 'stick_out', 'stomach', 'bear', 'stand', 'tolerate', 'support', 'brook', 'abide', 'suffer', 'put_up', 'stand', 'stand', 'stand', 'stand', 'stand', 'stand_up', 'place_upright', 'resist', 'stand', 'fend', 'stand']\n",
      "ANTONYMS\n",
      "['sit', 'yield']\n",
      "\n",
      "\n",
      "WORD natural\n",
      "DEFINITION someone regarded as certain to succeed\n",
      "USAGE EXAMPLES \n",
      "[\"he's a natural for the job\"]\n",
      "SYNONYMS\n",
      "['natural', 'natural', 'cancel', 'natural', 'natural', 'natural', 'natural', 'natural', 'natural', 'natural', 'instinctive', 'natural', 'raw', 'rude', 'natural', 'natural', 'born', 'innate', 'lifelike', 'natural']\n",
      "ANTONYMS\n",
      "['unnatural', 'artificial', 'supernatural', 'sharp']\n",
      "\n",
      "\n",
      "WORD language\n",
      "DEFINITION a systematic means of communicating by the use of sounds or conventional symbols\n",
      "USAGE EXAMPLES \n",
      "['he taught foreign languages', 'the language introduced is standard throughout the text', 'the speed with which a program can be executed depends on the language in which it is written']\n",
      "SYNONYMS\n",
      "['language', 'linguistic_communication', 'speech', 'speech_communication', 'spoken_communication', 'spoken_language', 'language', 'voice_communication', 'oral_communication', 'lyric', 'words', 'language', 'linguistic_process', 'language', 'language', 'speech', 'terminology', 'nomenclature', 'language']\n",
      "ANTONYMS\n",
      "[]\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from nltk.corpus import wordnet\n",
    "\n",
    "for word in noStopwords[0:4]:\n",
    "    foundWord = wordnet.synsets(word)\n",
    "    if foundWord:\n",
    "        print(\"WORD \" + word)\n",
    "        # shows definition of a word\n",
    "        print(\"DEFINITION \" + foundWord[0].definition())\n",
    "        print(\"USAGE EXAMPLES \")\n",
    "        # shows examples of usage\n",
    "        print(foundWord[0].examples())\n",
    "        \n",
    "        synonyms = []\n",
    "        antonyms = []\n",
    "        \n",
    "        # searches for synonyms and antonyms; in case there are some - returns them\n",
    "        for syn in wordnet.synsets(word):\n",
    "            for lemma in syn.lemmas():\n",
    "                synonyms.append(lemma.name())\n",
    "                \n",
    "                if lemma.antonyms():\n",
    "                    antonyms.append(lemma.antonyms()[0].name())\n",
    "        \n",
    "        print(\"SYNONYMS\")\n",
    "        print(synonyms)\n",
    "        print(\"ANTONYMS\")\n",
    "        print(antonyms)\n",
    "        print (\"\\n\")\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, no information about toolkit and nltk were found, but we'll be fine - we still retrieved a lot of interesting information. No need to use dictionary or google translate when you have NLTK. :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming and lemmatizing\n",
    "\n",
    "Word stemming is removing affixes from words and return the root word. Many search engines use this approach when indexing pages, as people may write the same word differently, but they all will mean the same thing.\n",
    "\n",
    "Word lemmatizing is the same thing as stemming, but the lemmatizing always returns a valid word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "increas\n",
      "increase\n"
     ]
    }
   ],
   "source": [
    "from nltk.stem import PorterStemmer, WordNetLemmatizer\n",
    " \n",
    "# as you will see, stemmer will return quite interesting word, right?\n",
    "word = \"increases\"\n",
    "\n",
    "stemmer = PorterStemmer()\n",
    "print(stemmer.stem(word))\n",
    " \n",
    "lemmatizer = WordNetLemmatizer()\n",
    "print(lemmatizer.lemmatize(word))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part-Of-Speech tagging\n",
    "\n",
    "For mor precise analysis you can use lots of features with part-of-speech tagging by NLTK. I will show you only the most common operation: tagging itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('nltk', 'RB'), ('stands', 'VBZ'), ('natural', 'JJ'), ('language', 'NN'), ('toolkit', 'NN'), ('one', 'CD'), ('popular', 'JJ'), ('libraries', 'NNS'), ('natural', 'JJ'), ('language', 'NN'), ('processing', 'NN'), ('nlp', 'IN'), ('it', 'PRP'), ('written', 'VBN'), ('python', 'JJ'), ('lot', 'NN'), ('useful', 'JJ'), ('methods', 'NNS'), ('well', 'RB'), ('lots', 'NNS'), ('tutorials', 'NNS'), ('supportive', 'VBP'), ('community', 'NN')]\n"
     ]
    }
   ],
   "source": [
    "from nltk import pos_tag\n",
    "\n",
    "taggedWords = pos_tag(noStopwords)\n",
    "print(taggedWords)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you see, NLTK tagged each word with it's part-of-speech. To see the list of available p-o-s, take a look here: https://medium.com/@gianpaul.r/tokenization-and-parts-of-speech-pos-tagging-in-pythons-nltk-library-2d30f70af13b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's use some gained knowledge in practice!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chatbots\n",
    "\n",
    "There is a module in NLTK called [**nltk.chat**](https://www.nltk.org/api/nltk.chat.html) which allows you to build simple yet working chatbot. Probably it's not the most intelligent one, but it definitely will work.\n",
    "\n",
    "To create a simple chatbot, you will need **Chat** - class that contains chatting logic for your bot and **Reflections** - dictionary that contains input and output values, usually these are synonyms or different forms of the word, e.g., here are some default reflections:\n",
    "\n",
    "`reflections = {\n",
    "  \"i am\"       : \"you are\",\n",
    "  \"i was\"      : \"you were\",\n",
    "  \"i\"          : \"you\",\n",
    "  \"i'm\"        : \"you are\"\n",
    "}`\n",
    "\n",
    "You can use default package reflections or create your own. There are also **pairs** - basically, these are dialogue sentences.\n",
    "\n",
    "So, let's build our simple chatbot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hi, I'm your first chatbot. Let's chat!\n",
      ">hello\n",
      "Hello!\n",
      ">my name is test\n",
      "Hello test, How are you doing today?\n",
      ">i'm doing fine\n",
      "Nice to hear that\n",
      ">are you studying at the university?\n",
      "I'm studying at University of Latvia, I'm enjoying my time here!\n",
      ">it was nice to meet you, bye\n",
      "See you soon!\n"
     ]
    }
   ],
   "source": [
    "from nltk.chat.util import Chat, reflections\n",
    "\n",
    "pairs = [\n",
    "    [\n",
    "        r\"my name is (.*)\",\n",
    "        [\"Hello %1, How are you doing today?\", \"Nice to meet you, %1!\"]\n",
    "    ],\n",
    "    [\n",
    "        r\"how are you?\",\n",
    "        [\"I'm doing great, thanks! What about you?\"]\n",
    "    ],\n",
    "    [\n",
    "        r\"i'm doing fine\",\n",
    "        [\"Nice to hear that\"]\n",
    "    ],\n",
    "    [\n",
    "        r\"good evening|hello\",\n",
    "        [\"Hello!\", \"Hi!\"]\n",
    "    ],\n",
    "    [\n",
    "        r\"it was nice to meet you, bye\",\n",
    "        [\"See you soon!\"]\n",
    "    ],\n",
    "    [\n",
    "        r\"who is the strongest man on the earth?\",\n",
    "        [\"CHUCK NORRIS\"]\n",
    "    ],\n",
    "    [\n",
    "        r\"(.*) university?\",\n",
    "        [\"I'm studying at University of Latvia, I'm enjoying my time here!\"]\n",
    "    ]\n",
    "]\n",
    "\n",
    "def chatty():\n",
    "        print(\"Hi, I'm your first chatbot. Let's chat!\")\n",
    "        chat = Chat(pairs, reflections)\n",
    "        chat.converse()\n",
    "        \n",
    "if __name__ == \"__main__\":\n",
    "    chatty()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basically, we are creating a dictionary with phrases, create a main **Chat** class with dictionary and reflections and creating a conversation with **converse()** method. \n",
    "\n",
    "That's it! Enjoy your first chatbot!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gender finder by name\n",
    "\n",
    "Let's try to implement a script that will try to guess your gender by your name. Hoping we'll get it right and won't offense anybody.\n",
    "\n",
    "Importing all necessary packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "from nltk import NaiveBayesClassifier\n",
    "from nltk.classify import accuracy\n",
    "from nltk.corpus import names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's extract the last N letters of the input word that will act as a features from which we will try to predict gender."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extractLetters(name, N = 2):\n",
    "   lastLetters = name[-N:]\n",
    "   return {'feature': lastLetters.lower()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating training data from available in NLTK datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "if __name__=='__main__':\n",
    "    maleNames = [(name, 'male') for name in names.words('male.txt')]\n",
    "    femaleNames = [(name, 'female') for name in names.words('female.txt')]\n",
    "    data = (maleNames + femaleNames)\n",
    "\n",
    "    random.seed(5)\n",
    "    random.shuffle(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's enter some test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "names = ['Karina', 'Ramzes', 'James', 'Gloria']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Defining number of training samples, let's use 90% of all available dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "importedData = int(0.9 * len(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's test the model with the different number of end letters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Last letter count: 1\n",
      "Accuracy = 75.47%\n",
      "Karina ==> female\n",
      "Ramzes ==> male\n",
      "James ==> male\n",
      "Gloria ==> female\n",
      "\n",
      "Last letter count: 2\n",
      "Accuracy = 79.25%\n",
      "Karina ==> female\n",
      "Ramzes ==> male\n",
      "James ==> male\n",
      "Gloria ==> female\n",
      "\n",
      "Last letter count: 3\n",
      "Accuracy = 77.86%\n",
      "Karina ==> female\n",
      "Ramzes ==> female\n",
      "James ==> male\n",
      "Gloria ==> female\n",
      "\n",
      "Last letter count: 4\n",
      "Accuracy = 70.94%\n",
      "Karina ==> female\n",
      "Ramzes ==> female\n",
      "James ==> male\n",
      "Gloria ==> female\n"
     ]
    }
   ],
   "source": [
    "for i in range(1, 5):\n",
    "    print('\\nLast letter count:', i)\n",
    "    features = [(extractLetters(n, i), gender) for (n, gender) in data]\n",
    "    trainingData, testingData = features[:importedData], features[importedData:]\n",
    "    classifier = NaiveBayesClassifier.train(trainingData)\n",
    "    \n",
    "    modelAccuracy = round(100 * accuracy(classifier, testingData), 2)\n",
    "    print('Accuracy = ' + str(modelAccuracy) + '%')\n",
    "    \n",
    "    for name in names:\n",
    "        print(name, '==>', classifier.classify(extractLetters(name, i)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the deeper it looks, the worse results become. The best option is to use 2 last letters. Precision is not the best, but it clearly sees me as a female, which is nice. :D Feel free to optimize this algorithm, the best will receive cookies.\n",
    "\n",
    "In this algorithm NLTK built-in Naive Bayes Classifier algorithm was used.\n",
    "The Naive Bayes algorithm is an intuitive method that uses the probabilities of each attribute belonging to each class to make a prediction. It is the supervised learning approach you would come up with if you wanted to model a predictive modeling problem probabilistically.\n",
    "\n",
    "Naive bayes simplifies the calculation of probabilities by assuming that the probability of each attribute belonging to a given class value is independent of all other attributes. This is a strong assumption but results in a fast and effective method. [Source](https://machinelearningmastery.com/naive-bayes-classifier-scratch-python/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "As NLTK is a very huge library, I included only most needed and most used basic methods, as well as some easy yet interesting methods that would motivate you to dive into NLP more deeply. I really hope you've learned something.\n",
    "\n",
    "NLTK is very useful but easy to understand at the same time. It supports many languages and can be used for both human and computer languages, as long as they are natural. It can help you to analyse text, write your own filters or chatbots, filter spam, as well as use some artificial intelligence tricks. Who knows, probably next Alexa or Google will be written by you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional and used sources\n",
    "* http://www.nltk.org/howto/ - official NLTK how-to's\n",
    "* https://likegeeks.com/nlp-tutorial-using-python-nltk/ - intro to NLP with NLTK\n",
    "* https://towardsdatascience.com/build-your-first-chatbot-using-python-nltk-5d07b027e727 - chatbot tutorial with NLTK\n",
    "* https://stackabuse.com/text-summarization-with-nltk-in-python/ - easy text summarization tutorial (processes like that do happen in search engines) using NLTK\n",
    "* https://likegeeks.com/nlp-tutorial-using-python-nltk/ - NLTK basics\n",
    "* https://www.tutorialspoint.com/artificial_intelligence_with_python/artificial_intelligence_with_python_nltk_package.htm - some more intelligent NLTK\n",
    "* https://www.nltk.org - official NLTK docs\n",
    "* https://www.datacamp.com/community/tutorials/text-analytics-beginners-nltk - NLP in search engine style using NLTK - calculating word weights, etc.\n",
    "* https://towardsdatascience.com/spam-classifier-in-python-from-scratch-27a98ddd8e73 - spam classifier from scratch in NLTK\n",
    "\n",
    "## How NLTK can help you at the university?\n",
    "\n",
    "* One of the examples is my program for \"Formal Grammars\" course. \n",
    "\n",
    "In case you haven't had this course yet, consider this package to be your best friend during it, as it'll help you to easily parse any formal grammar and give you the correct answer. You will have only one task: to reduce the grammar correctly.\n",
    "\n",
    "This example parses the grammar reduced by me and returns all possible parse trees for the word, as well as detects whether it belongs to the given grammar. This was the first time I found NLTK and I've never regret it ever since. :)\n",
    "\n",
    "[Click here](https://colab.research.google.com/drive/1vULur3cAO4vmiqHhAECatmIhnTMnWm0z)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}