{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic features and readability scores\n",
    "> Learn to compute basic features such as number of words, number of characters, average word length and number of special characters (such as Twitter hashtags and mentions). You will also learn to compute readability scores and determine the amount of education required to comprehend a piece of text. This is the Summary of lecture \"Feature Engineering for NLP in Python\", via datacamp.\n",
    "\n",
    "- toc: true \n",
    "- badges: true\n",
    "- comments: true\n",
    "- author: Chanseok Kang\n",
    "- categories: [Python, Datacamp, Natural_Language_Processing]\n",
    "- image: images/mention_count.png"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.rcParams['figure.figsize'] = (8, 8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to NLP feature engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-hot encoding\n",
    "We encountered a dataframe `df1` which contained categorical features and therefore, was unsuitable for applying ML algorithms to.\n",
    "\n",
    "In this exercise, your task is to convert `df1` into a format that is suitable for machine learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1 = pd.read_csv('./dataset/FE_df1.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['feature 1', 'feature 2', 'feature 3', 'feature 4', 'feature 5',\n",
      "       'label'],\n",
      "      dtype='object')\n",
      "Index(['feature 1', 'feature 2', 'feature 3', 'feature 4', 'label',\n",
      "       'feature 5_female', 'feature 5_male'],\n",
      "      dtype='object')\n",
      "   feature 1  feature 2  feature 3  feature 4  label  feature 5_female  \\\n",
      "0    29.0000          0          0   211.3375      1                 1   \n",
      "1     0.9167          1          2   151.5500      1                 0   \n",
      "2     2.0000          1          2   151.5500      0                 1   \n",
      "3    30.0000          1          2   151.5500      0                 0   \n",
      "4    25.0000          1          2   151.5500      0                 1   \n",
      "\n",
      "   feature 5_male  \n",
      "0               0  \n",
      "1               1  \n",
      "2               0  \n",
      "3               1  \n",
      "4               0  \n"
     ]
    }
   ],
   "source": [
    "# Print the features of df1\n",
    "print(df1.columns)\n",
    "\n",
    "# Perform one-hot encoding\n",
    "df1 = pd.get_dummies(df1, columns=['feature 5'])\n",
    "\n",
    "# Print the new features of df1\n",
    "print(df1.columns)\n",
    "\n",
    "# Print first five rows of df1\n",
    "print(df1.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic feature extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Character count of Russian tweets\n",
    "In this exercise, you have been given a dataframe `tweets` which contains some tweets associated with Russia's Internet Research Agency and compiled by FiveThirtyEight.\n",
    "\n",
    "Your task is to create a new feature `'char_count'` in tweets which computes the number of characters for each tweet. Also, compute the average length of each tweet. The tweets are available in the `content` feature of `tweets`.\n",
    "\n",
    "*Be aware that this is real data from Twitter and as such there is always a risk that it may contain profanity or other offensive content (in this exercise, and any following exercises that also use real Twitter data).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>content</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>127447</td>\n",
       "      <td>LIVE STREAM VIDEO=&gt; Donald Trump Rallies in Co...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>123642</td>\n",
       "      <td>Muslim Attacks NYPD Cops with Meat Cleaver. Me...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>226970</td>\n",
       "      <td>.@vfpatlas well that's a swella word there (di...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>138339</td>\n",
       "      <td>RT wehking_pamela: Bobby_Axelrod2k MMFlint don...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>161610</td>\n",
       "      <td>Жители обстреливаемых районов Донецка проводят...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0                                            content\n",
       "0      127447  LIVE STREAM VIDEO=> Donald Trump Rallies in Co...\n",
       "1      123642  Muslim Attacks NYPD Cops with Meat Cleaver. Me...\n",
       "2      226970  .@vfpatlas well that's a swella word there (di...\n",
       "3      138339  RT wehking_pamela: Bobby_Axelrod2k MMFlint don...\n",
       "4      161610  Жители обстреливаемых районов Донецка проводят..."
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets = pd.read_csv('./dataset/russian_tweets.csv')\n",
    "tweets.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "103.462\n"
     ]
    }
   ],
   "source": [
    "# Create a feature char_count\n",
    "tweets['char_count'] = tweets['content'].apply(len)\n",
    "\n",
    "# Print the average character count\n",
    "print(tweets['char_count'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the average character count of these tweets is approximately 104, which is much higher than the overall average tweet length of around 40 characters. Depending on what you're working on, this may be something worth investigating into. For your information, there is research that indicates that fake news articles tend to have longer titles! Therefore, even extremely basic features such as character counts can prove to be very useful in certain applications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word count of TED talks\n",
    "`ted` is a dataframe that contains the transcripts of 500 TED talks. Your job is to compute a new feature `word_count` which contains the approximate number of words for each talk. Consequently, you also need to compute the average word count of the talks. The transcripts are available as the `transcript` feature in `ted`.\n",
    "\n",
    "In order to complete this task, you will need to define a function `count_words` that takes in a string as an argument and returns the number of words in the string. You will then need to apply this function to the `transcript` feature of `ted` to create the new feature `word_count` and compute its mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>transcript</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>We're going to talk — my — a new lecture, just...</td>\n",
       "      <td>https://www.ted.com/talks/al_seckel_says_our_b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>This is a representation of your brain, and yo...</td>\n",
       "      <td>https://www.ted.com/talks/aaron_o_connell_maki...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>It's a great honor today to share with you The...</td>\n",
       "      <td>https://www.ted.com/talks/carter_emmart_demos_...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>My passions are music, technology and making t...</td>\n",
       "      <td>https://www.ted.com/talks/jared_ficklin_new_wa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>It used to be that if you wanted to get a comp...</td>\n",
       "      <td>https://www.ted.com/talks/jeremy_howard_the_wo...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          transcript  \\\n",
       "0  We're going to talk — my — a new lecture, just...   \n",
       "1  This is a representation of your brain, and yo...   \n",
       "2  It's a great honor today to share with you The...   \n",
       "3  My passions are music, technology and making t...   \n",
       "4  It used to be that if you wanted to get a comp...   \n",
       "\n",
       "                                                 url  \n",
       "0  https://www.ted.com/talks/al_seckel_says_our_b...  \n",
       "1  https://www.ted.com/talks/aaron_o_connell_maki...  \n",
       "2  https://www.ted.com/talks/carter_emmart_demos_...  \n",
       "3  https://www.ted.com/talks/jared_ficklin_new_wa...  \n",
       "4  https://www.ted.com/talks/jeremy_howard_the_wo...  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ted = pd.read_csv('./dataset/ted.csv')\n",
    "ted.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1987.1\n"
     ]
    }
   ],
   "source": [
    "# Function that returns number of words in a string\n",
    "def count_words(string):\n",
    "    # Split the string into words\n",
    "    words = string.split()\n",
    "    \n",
    "    # Return the number of words\n",
    "    return len(words)\n",
    "\n",
    "# Create a new feature word_count\n",
    "ted['word_count'] = ted['transcript'].apply(count_words)\n",
    "\n",
    "# Print the average word count of the talks\n",
    "print(ted['word_count'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You now know how to compute the number of words in a given piece of text. Also, notice that the average length of a talk is close to 2000 words. You can use the `word_count` feature to compute its correlation with other variables such as number of views, number of comments, etc. and derive extremely interesting insights about TED."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hashtags and mentions in Russian tweets\n",
    "Let's revisit the tweets dataframe containing the Russian tweets. In this exercise, you will compute the number of hashtags and mentions in each tweet by defining two functions `count_hashtags()` and `count_mentions()` respectively and applying them to the content feature of tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeYAAAHiCAYAAAA9Am/ZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZ9klEQVR4nO3df7DldX3f8ddb1x8IURR0w6/Jmkr8Sa26GtSpXcUmKio0I6kJUTRY2sYYjKRKnLYm6Y8hnahRY9LZigkZib/QEYwm1VFvOzaRRtQRETOgQVlZQeRHXIxR4qd/nO/W63p37xHu3fP23sdjhtlzvt/v+X4/34/Xfe73e86erTFGAIAe7rLoAQAA3yXMANCIMANAI8IMAI0IMwA0IswA0IgwwwFU1aiqBy16HJ1V1VJVvWh6fHpVfWAN931FVe2YHv9GVb1lDff9yqp601rtD9aKMPNDr6quqaqn7rPsBVX10XU+7rof42Crqj+qqv98R18/xrhwjPFTa3WcMcbDxxhLd3Q8y463o6p27bPv/zrGeNGd3TesNWEG2qmqLYseAyyKMLMpVNW5VfX5qvp6VX22qv7FsnUPqqr/VVW3VtWNVfX2fV7+1Kq6qqpurqo31sxDk/z3JI+vqj1Vdcu0r5Or6pNV9bdVdW1V/cY+43h+VX2xqr5WVf9hpav9ZdseUlWvnra/tao+WlWHTOuePd3mvWW6lfzQZa/7ntvvy69O9145VtU5VXVDVe2uqhdO685KcnqSl0/n9N79jOufV9XnpjH9XpJatu7/30WY5um103FurapPV9Uj9necaS5eUVWfTnJbVW1ZYX7uWVVvn/53/ERVPXK1866qQ5P8WZKjp+Ptqaqj9701vsqcXlNVvzadw63TGO650vzAnSXMbBafT/JPk9wnyW8meUtVHTWt+09JPpDkvkmOTfKGfV77zCSPTfLIJD+b5KfHGFcm+TdJ/nKMcdgY4/Bp29uSPD/J4UlOTvJvq+rUJKmqhyX5/cyidNQ0lmMOMObfSfKYJE9Icr8kL0/ynar6iSRvTfLSJPdP8v4k762qu885Fz+67NhnJnljVd13jLEzyYVJ/tt0Ts/a94VVdWSSdyX590mOzGxen7if4/xUkicl+YnM5uNfJvnaKsf5uczm7fAxxu0r7POUJO+c5uNPkrynqu52oJMdY9yW5OlJrpuOd9gY47p9zmueOf3ZJE9L8sAk/zjJCw50XLijhJmN4j3Tlc4t09Xr7y9fOcZ45xjjujHGd8YYb09yVZLHTau/neTHkhw9xvjmGGPf943PG2PcMsb4UpKPJPkn+xvEGGNpjHH5dJxPZ/ab/T+bVj8nyXvHGB8dY3wryX9MsuKX1VfVXZL8YpKzxxhfHmP8wxjjL8YYf59Z4N43xvjgGOPbmQX8kMwCPo9vJ/mtMca3xxjvT7InyYPnfO0zknx2jHHRdOzfTfKVAxznR5I8JEmNMa4cY+xeZf+vH2NcO8b4u/2sv2zZsV+T5J5JTpxz7Acyz5y+fvoZuinJe3OAnwO4M4SZjeLUMcbhe/9L8kvLV063kD+1LNyPyOyKL5ldiVaS/zvdyvzFffa9PDzfSHLY/gZRVT9ZVR+pqq9W1a2ZXVXvPc7RSa7du+0Y4xtJvrafXR2ZWXQ+v8K6o5N8cdl+vjPt90BX38t9bZ+r0QOe0wrHXn4OY/nz5cYYH07ye0nemOT6qtpZVfdeZf8r7mul9dN575rGdGfNM6dz/xzAnSHMbHhV9WNJ/keSX05yxBTuz2R6b3SM8ZUxxr8aYxyd5F8n+f2a769IrXS1+ydJLkly3BjjPpm9D733Pdjdmd0q3zuuQ5IcsZ9935jkm0n+0QrrrsvsCn/vfirJcUm+PC36RpJ7Ldv+R1c7kWVW++fmdk/H2vfYK+9sjNePMR6T5OGZ3dL+d6scZ7XjLz/2XTKbz723pQ903qvtd7U5hYNGmNkMDs3sN+avJsn0YadH7F1ZVadV1d5g3jxt+w9z7Pf6JMfu8z7kjyS5aYzxzap6XJKfX7buoiTPqqonTK/5zSz74NRy0xXbm5O8Zvqg0l2r6vFVdY8k70hyclWdNL2/ek6Sv0/yF9PLP5Xk56fXPC3fvZU+j+uT/PgB1r8vycOr6mdq9snpX8l+wl9Vj53uINwts/fev5nvzutqx9mfxyw79kszO++PTesOdN7XJzmiqu6zn/2uNqdw0AgzG94Y47NJXp3kLzP7DfqEJP9n2SaPTXJpVe3J7Gr37DHG38yx6w8nuSLJV6rqxmnZLyX5rar6embvIb9j2TiuSPKSJG/L7Mrz60luyCwAK/m1JJcn+askNyX57SR3GWP8dZJfyOxDajcmeVaSZ03vWyfJ2dOyWzL7oNl75jiXvc5P8rDplv/3vW6McWOS05Kcl9lt+OPzvXO53L0zu1Nxc2a3ib+W2Xu3qx7nAC7O7P3gm5M8L8nPTO8JJwc47zHG5zJ7v/8L0zG/5/b3HHMKB03N3iICDraqOiyziBw/5x8EgE3AFTMcRFX1rKq61/R3a38nsyviaxY7KqATYYaD65TMPmh0XWa3gZ873LYClnErGwAaccUMAI0IMwA00uJfcDnyyCPHtm3b1mx/t912Ww499NA1299GZZ7mY57mY57mY55Wtxnm6LLLLrtxjHH/lda1CPO2bdvy8Y9/fM32t7S0lB07dqzZ/jYq8zQf8zQf8zQf87S6zTBHVfXF/a1zKxsAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaCRLYsewHq4/Mu35gXnvm/Rwziga847edFDAKAhV8wA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0MhcYa6qX62qK6rqM1X11qq6Z1U9sKouraqrqurtVXX3adt7TM+vntZvW88TAICNZNUwV9UxSX4lyfYxxiOS3DXJc5P8dpLXjjGOT3JzkjOnl5yZ5OYxxoOSvHbaDgCYw7y3srckOaSqtiS5V5LdSZ6S5KJp/QVJTp0enzI9z7T+pKqqtRkuAGxsNcZYfaOqs5P8lyR/l+QDSc5O8rHpqjhVdVySPxtjPKKqPpPkaWOMXdO6zyf5yTHGjfvs86wkZyXJ1q1bH/O2t71tzU7qhptuzfV/t2a7WxcnHHOfRQ8he/bsyWGHHbboYbRnnuZjnuZjnla3GeboyU9+8mVjjO0rrduy2our6r6ZXQU/MMktSd6Z5OkrbLq38CtdHX9f/ccYO5PsTJLt27ePHTt2rDaUub3hwovz6stXPbWFuub0HYseQpaWlrKW875Rmaf5mKf5mKfVbfY5mudW9lOT/M0Y46tjjG8neXeSJyQ5fLq1nSTHJrluerwryXFJMq2/T5Kb1nTUALBBzRPmLyU5saruNb1XfFKSzyb5SJLnTNuckeTi6fEl0/NM6z885rlfDgCsHuYxxqWZfYjrE0kun16zM8krkrysqq5OckSS86eXnJ/kiGn5y5Kcuw7jBoANaa43YscYr0ryqn0WfyHJ41bY9ptJTrvzQwOAzcc3fwFAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0MleYq+rwqrqoqj5XVVdW1eOr6n5V9cGqumr69b7TtlVVr6+qq6vq01X16PU9BQDYOOa9Yn5dkj8fYzwkySOTXJnk3CQfGmMcn+RD0/MkeXqS46f/zkryB2s6YgDYwFYNc1XdO8mTkpyfJGOMb40xbklySpILps0uSHLq9PiUJH88Zj6W5PCqOmrNRw4AG9A8V8w/nuSrSf6wqj5ZVW+qqkOTbB1j7E6S6dcHTNsfk+TaZa/fNS0DAFaxZc5tHp3kJWOMS6vqdfnubeuV1ArLxvdtVHVWZre6s3Xr1iwtLc0xlPlsPSQ554Tb12x/62Etz/eO2rNnT4txdGee5mOe5mOeVrfZ52ieMO9KsmuMcen0/KLMwnx9VR01xtg93aq+Ydn2xy17/bFJrtt3p2OMnUl2Jsn27dvHjh077tgZrOANF16cV18+z6ktzjWn71j0ELK0tJS1nPeNyjzNxzzNxzytbrPP0aq3sscYX0lybVU9eFp0UpLPJrkkyRnTsjOSXDw9viTJ86dPZ5+Y5Na9t7wBgAOb97LyJUkurKq7J/lCkhdmFvV3VNWZSb6U5LRp2/cneUaSq5N8Y9oWAJjDXGEeY3wqyfYVVp20wrYjyYvv5LgAYFPyzV8A0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjcwd5qq6a1V9sqr+dHr+wKq6tKquqqq3V9Xdp+X3mJ5fPa3ftj5DB4CN5we5Yj47yZXLnv92kteOMY5PcnOSM6flZya5eYzxoCSvnbYDAOYwV5ir6tgkJyd50/S8kjwlyUXTJhckOXV6fMr0PNP6k6btAYBVzHvF/LtJXp7kO9PzI5LcMsa4fXq+K8kx0+NjklybJNP6W6ftAYBVbFltg6p6ZpIbxhiXVdWOvYtX2HTMsW75fs9KclaSbN26NUtLS/OMdy5bD0nOOeH21TdcoLU83ztqz549LcbRnXmaj3maj3la3Wafo1XDnOSJSZ5dVc9Ics8k987sCvrwqtoyXRUfm+S6aftdSY5LsquqtiS5T5Kb9t3pGGNnkp1Jsn379rFjx447eSrf9YYLL86rL5/n1BbnmtN3LHoIWVpaylrO+0ZlnuZjnuZjnla32edo1VvZY4xfH2McO8bYluS5ST48xjg9yUeSPGfa7IwkF0+PL5meZ1r/4THG910xAwDf7878PeZXJHlZVV2d2XvI50/Lz09yxLT8ZUnOvXNDBIDN4we63zvGWEqyND3+QpLHrbDNN5OctgZjA4BNxzd/AUAjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADSyapir6riq+khVXVlVV1TV2dPy+1XVB6vqqunX+07Lq6peX1VXV9Wnq+rR630SALBRzHPFfHuSc8YYD01yYpIXV9XDkpyb5ENjjOOTfGh6niRPT3L89N9ZSf5gzUcNABvUqmEeY+weY3xievz1JFcmOSbJKUkumDa7IMmp0+NTkvzxmPlYksOr6qg1HzkAbEA/0HvMVbUtyaOSXJpk6xhjdzKLd5IHTJsdk+TaZS/bNS0DAFaxZd4Nq+qwJO9K8tIxxt9W1X43XWHZWGF/Z2V2qztbt27N0tLSvENZ1dZDknNOuH3N9rce1vJ876g9e/a0GEd35mk+5mk+5ml1m32O5gpzVd0tsyhfOMZ497T4+qo6aoyxe7pVfcO0fFeS45a9/Ngk1+27zzHGziQ7k2T79u1jx44dd+wMVvCGCy/Oqy+f+88cC3HN6TsWPYQsLS1lLed9ozJP8zFP8zFPq9vsczTPp7IryflJrhxjvGbZqkuSnDE9PiPJxcuWP3/6dPaJSW7de8sbADiweS4rn5jkeUkur6pPTctemeS8JO+oqjOTfCnJadO69yd5RpKrk3wjyQvXdMQAsIGtGuYxxkez8vvGSXLSCtuPJC++k+MCgE3JN38BQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI1sWPYDNatu571v0EHLOCbfnBQcYxzXnnXwQRwNA4ooZAFoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaCRLYseAH1tO/d9ix7CAV1z3smLHgLAmnPFDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0Ii/xwzryN8FB35QwswPrYMVvXNOuD0vaB5YYONYl1vZVfW0qvrrqrq6qs5dj2MAwEa05mGuqrsmeWOSpyd5WJKfq6qHrfVxAGAjWo9b2Y9LcvUY4wtJUlVvS3JKks+uw7GAO2Et3w5Yr1v+3gffHJb/LHZ8++hg/hyuR5iPSXLtsue7kvzkOhwH2AR8gI7NpsYYa7vDqtOS/PQY40XT8+cledwY4yX7bHdWkrOmpw9O8tdrOIwjk9y4hvvbqMzTfMzTfMzTfMzT6jbDHP3YGOP+K61YjyvmXUmOW/b82CTX7bvRGGNnkp3rcPxU1cfHGNvXY98biXmaj3maj3maj3la3Wafo/X4VPZfJTm+qh5YVXdP8twkl6zDcQBgw1nzK+Yxxu1V9ctJ/meSuyZ58xjjirU+DgBsROvyBSNjjPcnef967HtO63KLfAMyT/MxT/MxT/MxT6vb1HO05h/+AgDuOP+IBQA0suHC7OtAV1dVx1XVR6rqyqq6oqrOXvSYuqqqu1bVJ6vqTxc9lq6q6vCquqiqPjf9TD1+0WPqqKp+dfr/22eq6q1Vdc9Fj6mDqnpzVd1QVZ9Ztux+VfXBqrpq+vW+ixzjwbahwuzrQOd2e5JzxhgPTXJikhebp/06O8mVix5Ec69L8udjjIckeWTM1/epqmOS/EqS7WOMR2T2wdjnLnZUbfxRkqfts+zcJB8aYxyf5EPT801jQ4U5y74OdIzxrSR7vw6UZcYYu8cYn5gefz2z30iPWeyo+qmqY5OcnORNix5LV1V17yRPSnJ+kowxvjXGuGWxo2prS5JDqmpLkntlhe932IzGGP87yU37LD4lyQXT4wuSnHpQB7VgGy3MK30dqOAcQFVtS/KoJJcudiQt/W6Slyf5zqIH0tiPJ/lqkj+cbvm/qaoOXfSguhljfDnJ7yT5UpLdSW4dY3xgsaNqbesYY3cyu5BI8oAFj+eg2mhhrhWW+dj5flTVYUneleSlY4y/XfR4OqmqZya5YYxx2aLH0tyWJI9O8gdjjEcluS2b7LbjPKb3SE9J8sAkRyc5tKp+YbGjoquNFua5vg6UpKrullmULxxjvHvR42noiUmeXVXXZPaWyFOq6i2LHVJLu5LsGmPsveNyUWah5ns9NcnfjDG+Osb4dpJ3J3nCgsfU2fVVdVSSTL/esODxHFQbLcy+DnQOVVWZvSd45RjjNYseT0djjF8fYxw7xtiW2c/Rh8cYrnD2Mcb4SpJrq+rB06KT4p94XcmXkpxYVfea/v93UnxI7kAuSXLG9PiMJBcvcCwH3bp889ei+DrQuT0xyfOSXF5Vn5qWvXL6xjb4Qb0kyYXTH4a/kOSFCx5PO2OMS6vqoiSfyOxvRXwym/zbrfaqqrcm2ZHkyKraleRVSc5L8o6qOjOzP9SctrgRHny++QsAGtlot7IB4IeaMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADTy/wAXMWRKd7vLzAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Function that returns number of hashtags in a string\n",
    "def count_hashtags(string):\n",
    "    # Split the string into words\n",
    "    words = string.split()\n",
    "    \n",
    "    # Create a list of words that are hashtags\n",
    "    hashtags = [word for word in words if word.startswith('#')]\n",
    "    \n",
    "    # Return number of hashtags\n",
    "    return(len(hashtags))\n",
    "\n",
    "# Create a feature hashtag_countand display distribution\n",
    "tweets['hashtag_count'] = tweets['content'].apply(count_hashtags)\n",
    "tweets['hashtag_count'].hist();\n",
    "plt.title('Hashtag count distribution');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeYAAAHiCAYAAAA9Am/ZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZl0lEQVR4nO3dfZBld13n8c93MwJ5KAgQGEOSdUCySDCF4CxG2bUGoiVPa/gDXKyIkY2kqEJESa1GUVmf0RIB0cWKRIkQQTdSJivoygYa1tolBQGKEIOVCCGZZEiAPMAEXIj89o97ZmkmPdN35nZPf7vn9aqamnvPOfec3/l1knefc7tvaowRAKCHf7XRAwAAvk6YAaARYQaARoQZABoRZgBoRJgBoBFhhgVU1XVVtWujx7FRquqmqvq+6fHPV9Wb1nDfe6vqMdPjN1fVr63hvv+wqn5xrfYHa0mY2RKmQHylqk7ab/lHq2pU1Y41OMb94jDGeMIYY2nRfXdQVUtV9eOH+/oxxm+MMVZ9/bzHGWOcMMb45OGOZ9nxfqyq/n6/fb9kjPGri+4b1oMws5V8KskP73tSVWcmOXbjhsPhqKptGz0G2EjCzFbyliQ/uuz5eUn+dPkGVfXAqvqdqrq5qm6fbmkeO63bVVW7q+rCqrqjqvZU1YumdRckOTfJz0y3WP/7tHz5rdwHVtXrquq26c/rquqBq+17JVX1sKr6k2k/d1XVXy1b9+KqurGq7qyqK6vqUdPyHdPdgW3Ltv3/V6f7rhyn87+rqj5VVc+c1v16kn+f5Pen8/v9A4zrhVX16ar6fFW9cr91/6Wq3jo9flBVvXXa7u6q+mBVbT/QcaZxv7Sqbkhyw7Jlj112iJOq6t1V9cWqel9Vfctq511Vj0/yh0m+ezre3dP6b7j7caA5XTaOl1TVDdO8/UFV1YG+drAoYWYr+UCSB1fV46vqmCT/Mclb99vmt5L8myTfkeSxSU5J8kvL1n9zkodMy89P8gdV9dAxxsVJLkvy29Mt1v+wwvFfmeSsad9PTPKUJL+w2r4PcC5vSXJckickeWSS1yZJVT09yW8m+aEkJyf5dJK3H2RO9vddSf4xyUlJfjvJJVVVY4xXJvlfSX5iOr+f2P+FVXVGkjcmeWGSRyV5eJJTD3Cc86ZzPW3a7iVJvrzKcZ47je+MA+zz3CS/Oo39o5l9PQ5qjHH9dOz/Mx3vxBXOa545fU6Sf5vZ1/WHkvzAaseGwyXMbDX7rpq/P8knkty6b8V0lfPiJD89xrhzjPHFJL+R5AXLXv/VJL8yxvjqGONdSfYmedycxz53eu0dY4zPJvnlzCJ2SPuuqpOTPDPJS8YYd03bv2/ZMf54jPHhMcb/TfJzmV0N7phzjJ8eY/zRGONfklyaWYi2z/na5yX56zHG+6dj/2KSrx1g269mFuTHjjH+ZYxxzRjjC6vs/zenr8uXD7D+ncuO/crMzvu0Ocd+MPPM6avHGHePMW5O8t7MvvmCdeG9HLaatyR5f5JHZ7/b2EkekdlV6DXL7kRWkmOWbfP5McZ9y55/KckJcx77UZldbe3z6WnZoe77tCR3jjHuOsAxPrzvyRhjb1V9PrOr8FtX2H5/n1n22i9N83Ao53fLstffOx17JW/J7DzeXlUnZnbn4pVjjK8eZP+3HGTdN6yfzvvOaUy3zzP4gzjYnN40Lf7Msu0P5Z8JOGSumNlSxhifzuyHwJ6V5B37rf5cki8necIY48Tpz0PGGPP+R3a1/xXbbUm+Zdnzfz0tO1S3JHnYFLSDHqOqjs/syvTWJPdOi49btv03H8JxVzu/PZnFdt+xj5uOff8dza7yf3mMcUaS78nsVvC+9/8PdJzVjr/82CckeVhm87HaeR/S122/OYUjTpjZis5P8vQxxr3LF44xvpbkj5K8tqoemSRVdUpVzft+4e1JHnOQ9W9L8gtV9Yia/drWL+X+73GvaoyxJ8nfJPmvVfXQqvqmqvreafWfJXlRVX3H9INlv5Hk6jHGTdPt81uT/EhVHVNV/ynJtx7CoVc7v8uTPKeq/l1VPSDJr+QA/w2pqqdV1ZnTe/1fyOzW9r/MeZwDedayY/9qZud9yxznfXuSU6fXreSAc3oYY4SFCTNbzhjjn8YYHzrA6p9NcmOSD1TVF5L8z8z/HvIlSc6Yfsr4r1ZY/2tJPpTkY0muzez26OF+KMYLM4vZJ5LckeSnkmSMcVVm7+3+ZWZXsN+ab3yP/MVJ/nOSz2f2g2P/+xCO+fokz5t+8vj39l85xrguyUszC9meJHcl2X2AfX1zZiH/QpLrk7wvX/8m5aDHOYg/S/KqJHcm+c7M3hve52Dn/Z4k1yX5TFV9boXzWm1O4YiqMVa7ywMAHCmumAGgEWEGgEaEGQAaEWYAaESYAaCRFp/8ddJJJ40dO3as2f7uvffeHH/88Wu2v6OVeVycOVycOVycOVzcWs/hNddc87kxxiNWWtcizDt27MiHPnSgXzs9dEtLS9m1a9ea7e9oZR4XZw4XZw4XZw4Xt9ZzWFWfPtA6t7IBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAa2bbRA1gP1956T37sondu9DAO6qZXP3ujhwBAQ66YAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBG5gpzVf10VV1XVR+vqrdV1YOq6tFVdXVV3VBVf15VD5i2feD0/MZp/Y71PAEA2EpWDXNVnZLkJ5PsHGN8e5JjkrwgyW8lee0Y4/QkdyU5f3rJ+UnuGmM8Nslrp+0AgDnMeyt7W5Jjq2pbkuOS7Eny9CSXT+svTfLc6fE50/NM68+uqlqb4QLA1lZjjNU3qnp5kl9P8uUkf5fk5Uk+MF0Vp6pOS/I3Y4xvr6qPJ3nGGGP3tO6fknzXGONz++3zgiQXJMn27du/8+1vf/uandQdd96T27+8ZrtbF2ee8pCNHsKq9u7dmxNOOGGjh7GpmcPFmcPFmcPFrfUcPu1pT7tmjLFzpXXbVntxVT00s6vgRye5O8l/S/LMFTbdV/iVro7vV/8xxsVJLk6SnTt3jl27dq02lLm94bIr8pprVz21DXXTubs2egirWlpaylp+XY5G5nBx5nBx5nBxR3IO57mV/X1JPjXG+OwY46tJ3pHke5KcON3aTpJTk9w2Pd6d5LQkmdY/JMmdazpqANii5gnzzUnOqqrjpveKz07yD0nem+R50zbnJblienzl9DzT+veMee6XAwCrh3mMcXVmP8T14STXTq+5OMnPJnlFVd2Y5OFJLpleckmSh0/LX5HkonUYNwBsSXO9ETvGeFWSV+23+JNJnrLCtv+c5PmLDw0Ajj4++QsAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGpkrzFV1YlVdXlWfqKrrq+q7q+phVfXuqrph+vuh07ZVVb9XVTdW1ceq6snrewoAsHXMe8X8+iR/O8b4tiRPTHJ9kouSXDXGOD3JVdPzJHlmktOnPxckeeOajhgAtrBVw1xVD07yvUkuSZIxxlfGGHcnOSfJpdNmlyZ57vT4nCR/OmY+kOTEqjp5zUcOAFvQPFfMj0ny2SR/UlUfqao3VdXxSbaPMfYkyfT3I6ftT0lyy7LX756WAQCr2DbnNk9O8rIxxtVV9fp8/bb1SmqFZeN+G1VdkNmt7mzfvj1LS0tzDGU+249NLjzzvjXb33pYy/NdL3v37t0U4+zMHC7OHC7OHC7uSM7hPGHenWT3GOPq6fnlmYX59qo6eYyxZ7pVfcey7U9b9vpTk9y2/07HGBcnuThJdu7cOXbt2nV4Z7CCN1x2RV5z7TyntnFuOnfXRg9hVUtLS1nLr8vRyBwuzhwuzhwu7kjO4aq3sscYn0lyS1U9blp0dpJ/SHJlkvOmZecluWJ6fGWSH51+OvusJPfsu+UNABzcvJeVL0tyWVU9IMknk7wos6j/RVWdn+TmJM+ftn1XkmcluTHJl6ZtAYA5zBXmMcZHk+xcYdXZK2w7krx0wXEBwFHJJ38BQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNzB3mqjqmqj5SVX89PX90VV1dVTdU1Z9X1QOm5Q+cnt84rd+xPkMHgK3nUK6YX57k+mXPfyvJa8cYpye5K8n50/Lzk9w1xnhsktdO2wEAc5grzFV1apJnJ3nT9LySPD3J5dMmlyZ57vT4nOl5pvVnT9sDAKuY94r5dUl+JsnXpucPT3L3GOO+6fnuJKdMj09JckuSTOvvmbYHAFaxbbUNquo5Se4YY1xTVbv2LV5h0zHHuuX7vSDJBUmyffv2LC0tzTPeuWw/NrnwzPtW33ADreX5rpe9e/duinF2Zg4XZw4XZw4XdyTncNUwJ3lqkh+sqmcleVCSB2d2BX1iVW2bropPTXLbtP3uJKcl2V1V25I8JMmd++90jHFxkouTZOfOnWPXrl0LnsrXveGyK/Kaa+c5tY1z07m7NnoIq1paWspafl2ORuZwceZwceZwcUdyDle9lT3G+LkxxqljjB1JXpDkPWOMc5O8N8nzps3OS3LF9PjK6Xmm9e8ZY9zvihkAuL9Ffo/5Z5O8oqpuzOw95Eum5Zckefi0/BVJLlpsiABw9Dik+71jjKUkS9PjTyZ5ygrb/HOS56/B2ADgqOOTvwCgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEaEGQAaEWYAaESYAaARYQaARoQZABoRZgBoRJgBoBFhBoBGhBkAGhFmAGhEmAGgEWEGgEZWDXNVnVZV762q66vquqp6+bT8YVX17qq6Yfr7odPyqqrfq6obq+pjVfXk9T4JANgq5rlivi/JhWOMxyc5K8lLq+qMJBcluWqMcXqSq6bnSfLMJKdPfy5I8sY1HzUAbFGrhnmMsWeM8eHp8ReTXJ/klCTnJLl02uzSJM+dHp+T5E/HzAeSnFhVJ6/5yAFgCzqk95irakeSJyW5Osn2McaeZBbvJI+cNjslyS3LXrZ7WgYArGLbvBtW1QlJ/jLJT40xvlBVB9x0hWVjhf1dkNmt7mzfvj1LS0vzDmVV249NLjzzvjXb33pYy/NdL3v37t0U4+zMHC7OHC7OHC7uSM7hXGGuqm/KLMqXjTHeMS2+vapOHmPsmW5V3zEt353ktGUvPzXJbfvvc4xxcZKLk2Tnzp1j165dh3cGK3jDZVfkNdfO/T3Hhrjp3F0bPYRVLS0tZS2/Lkcjc7g4c7g4c7i4IzmH8/xUdiW5JMn1Y4zfXbbqyiTnTY/PS3LFsuU/Ov109llJ7tl3yxsAOLh5LiufmuSFSa6tqo9Oy34+yauT/EVVnZ/k5iTPn9a9K8mzktyY5EtJXrSmIwaALWzVMI8x/j4rv2+cJGevsP1I8tIFxwUARyWf/AUAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNbNvoARytdlz0zo0ewqre/IzjN3oIAEcdV8wA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQiDADQCPCDACNCDMANCLMANCIMANAI8IMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQyLaNHgB9XXvrPfmxi9650cM4oJte/eyNHgLAmnPFDACNuGJm09rR+Gp+nzc/4/iNHgKwyaxLmKvqGUlen+SYJG8aY7x6PY4DbH3eUuFos+ZhrqpjkvxBku9PsjvJB6vqyjHGP6z1sYDFbIa7DheeudEjgCNrPa6Yn5LkxjHGJ5Okqt6e5JwkwsxRp/vVHovbDN/cbIa3VLrP45Gcw/UI8ylJbln2fHeS71qH4wAwB98gbi41xljbHVY9P8kPjDF+fHr+wiRPGWO8bL/tLkhywfT0cUn+cQ2HcVKSz63h/o5W5nFx5nBx5nBx5nBxaz2H3zLGeMRKK9bjinl3ktOWPT81yW37bzTGuDjJxetw/FTVh8YYO9dj30cT87g4c7g4c7g4c7i4IzmH6/F7zB9McnpVPbqqHpDkBUmuXIfjAMCWs+ZXzGOM+6rqJ5L8j8x+XeqPxxjXrfVxAGArWpffYx5jvCvJu9Zj33Nal1vkRyHzuDhzuDhzuDhzuLgjNodr/sNfAMDh81nZANDIlgtzVT2jqv6xqm6sqos2ejybTVWdVlXvrarrq+q6qnr5Ro9ps6qqY6rqI1X11xs9ls2oqk6sqsur6hPTP4/fvdFj2oyq6qenf5c/XlVvq6oHbfSYuquqP66qO6rq48uWPayq3l1VN0x/P3S9jr+lwrzs40CfmeSMJD9cVWds7Kg2nfuSXDjGeHySs5K81BwetpcnuX6jB7GJvT7J344xvi3JE2MuD1lVnZLkJ5PsHGN8e2Y/kPuCjR3VpvDmJM/Yb9lFSa4aY5ye5Krp+brYUmHOso8DHWN8Jcm+jwNlTmOMPWOMD0+Pv5jZfwxP2dhRbT5VdWqSZyd500aPZTOqqgcn+d4klyTJGOMrY4y7N3ZUm9a2JMdW1bYkx2WFz5XgG40x3p/kzv0Wn5Pk0unxpUmeu17H32phXunjQEXlMFXVjiRPSnL1xo5kU3pdkp9J8rWNHsgm9Zgkn03yJ9PbAW+qqv4f+NzMGOPWJL+T5OYke5LcM8b4u40d1aa1fYyxJ5ldwCR55HodaKuFuVZY5sfOD0NVnZDkL5P81BjjCxs9ns2kqp6T5I4xxjUbPZZNbFuSJyd54xjjSUnuzTreOtyqpvdBz0ny6CSPSnJ8Vf3Ixo6K1Wy1MM/1caAcXFV9U2ZRvmyM8Y6NHs8m9NQkP1hVN2X2dsrTq+qtGzukTWd3kt1jjH13ay7PLNQcmu9L8qkxxmfHGF9N8o4k37PBY9qsbq+qk5Nk+vuO9TrQVguzjwNdUFVVZu/rXT/G+N2NHs9mNMb4uTHGqWOMHZn9M/ieMYarlEMwxvhMkluq6nHTorPjfx17OG5OclZVHTf9u312/BDd4boyyXnT4/OSXLFeB1qXT/7aKD4OdE08NckLk1xbVR+dlv389GlucCS9LMll0zfZn0zyog0ez6Yzxri6qi5P8uHMfuPiI/EpYKuqqrcl2ZXkpKraneRVSV6d5C+q6vzMvuF5/rod3yd/AUAfW+1WNgBsasIMAI0IMwA0IswA0IgwA0AjwgwAjQgzADQizADQyP8DjaM31dg5VNwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Function that returns number of mentions in a string\n",
    "def count_mentions(string):\n",
    "    # Split the string into words\n",
    "    words = string.split()\n",
    "    \n",
    "    # Create a list of words that are mentions\n",
    "    mentions = [word for word in words if word.startswith('@')]\n",
    "    \n",
    "    # Return number of mentions\n",
    "    return(len(mentions))\n",
    "\n",
    "# Create a feature mention_count and display distribution\n",
    "tweets['mention_count'] = tweets['content'].apply(count_mentions)\n",
    "tweets['mention_count'].hist();\n",
    "plt.title('Mention count distribution');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You now have a good grasp of how to compute various types of summary features. In the next lesson, we will learn about more advanced features that are capable of capturing more nuanced information beyond simple word and character counts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Readability tests\n",
    "- Readability test\n",
    "    - Determine readability of an English passage\n",
    "    - Scale ranging from primary school up to college graduate level\n",
    "    - A mathematical formula utilizing word, syllable and sentence count\n",
    "    - Used in fake news and opinion spam detection\n",
    "- Examples\n",
    "    - Flesch reading ease\n",
    "    - Gunning fog index\n",
    "    - Simple Measure of Gobbledygook (SMOG)\n",
    "    - Dale-Chall score\n",
    "- Flesch reading ease\n",
    "    - One of the oldest and most widely used tests\n",
    "    - Dependent on two factors\n",
    "        - Greater the average sentence length, harder the text is to read\n",
    "        - Greater the average number of syllables in a word, harder the text is to read\n",
    "    - Higher the score, greater the readability\n",
    "- Gunning fog index\n",
    "    - Developed in 1954\n",
    "    - Also dependent on average sentence length\n",
    "    - Greater the percentage of complex words, harder the text is to read\n",
    "    - Higher the index, lesser the readability\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Readability of 'The Myth of Sisyphus'\n",
    "In this exercise, you will compute the Flesch reading ease score for Albert Camus' famous essay The Myth of Sisyphus. We will then interpret the value of this score as explained in the video and try to determine the reading level of the essay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nThe gods had condemned Sisyphus to ceaselessly rolling a rock to the top of a mountain, whence the '"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('./dataset/sisyphus_essay.txt', 'r') as f:\n",
    "    sisyphus_essay = f.read()\n",
    "sisyphus_essay[:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Flesch Reading Ease is 81.67\n"
     ]
    }
   ],
   "source": [
    "from textatistic import Textatistic\n",
    "\n",
    "# Compute the readability scores\n",
    "readability_scores = Textatistic(sisyphus_essay).scores\n",
    "\n",
    "# Print the flesch reading ease score\n",
    "flesch = readability_scores['flesch_score']\n",
    "print('The Flesch Reading Ease is %.2f' % (flesch))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You now know to compute the Flesch reading ease score for a given body of text. Notice that the score for this essay is approximately 81.67. This indicates that the essay is at the readability level of a 6th grade American student."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Readability of various publications\n",
    "In this exercise, you have been given excerpts of articles from four publications. Your task is to compute the readability of these excerpts using the Gunning fog index and consequently, determine the relative difficulty of reading these publications.\n",
    "\n",
    "The excerpts are available as the following strings:\n",
    "\n",
    "- `forbes`- An excerpt from an article from Forbes magazine on the Chinese social credit score system.\n",
    "- `harvard_law`- An excerpt from a book review published in Harvard Law Review.\n",
    "- `r_digest`- An excerpt from a Reader's Digest article on flight turbulence.\n",
    "- `time_kids` - An excerpt from an article on the ill effects of salt consumption published in TIME for Kids."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['./dataset/MobyDick.txt',\n",
       " './dataset/time_kids.txt',\n",
       " './dataset/alice.txt',\n",
       " './dataset/forbes.txt',\n",
       " './dataset/r_digest.txt',\n",
       " './dataset/english_stopwords.txt',\n",
       " './dataset/1984.txt',\n",
       " './dataset/sisyphus_essay.txt',\n",
       " './dataset/cab.txt',\n",
       " './dataset/wikipedia-vocabulary-utf8.txt',\n",
       " './dataset/grail.txt',\n",
       " './dataset/my_file.txt',\n",
       " './dataset/harvard_law.txt',\n",
       " './dataset/Sprint_Intraday.txt']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import glob\n",
    "texts = []\n",
    "text_list = glob.glob('./dataset/*.txt')\n",
    "\n",
    "text_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_list = ['time_kids', 'forbes', 'r_digest', 'harvard_law']\n",
    "for text in text_list:\n",
    "    for f in file_list:\n",
    "        if f in text:\n",
    "            with open(text, 'r') as f:\n",
    "                texts.append(f.read())\n",
    "            \n",
    "time_kids, forbes, r_digest, harvard_law = texts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[14.436002482929858, 20.735401069518716, 11.085587583148559, 5.926785009861934]\n"
     ]
    }
   ],
   "source": [
    "# List of excerpts\n",
    "excerpts = [forbes, harvard_law, r_digest, time_kids]\n",
    "\n",
    "# Loop through excerpts and compute gunning fog index\n",
    "gunning_fog_scores = []\n",
    "for excerpt in excerpts:\n",
    "    readability_scores = Textatistic(excerpt).scores\n",
    "    gunning_fog = readability_scores['gunningfog_score']\n",
    "    gunning_fog_scores.append(gunning_fog)\n",
    "    \n",
    "# Print the gunning fog indices\n",
    "print(gunning_fog_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the Harvard Law Review excerpt has the highest Gunning fog index; indicating that it can be comprehended only by readers who have graduated college. On the other hand, the Time for Kids article, intended for children, has a much lower fog index and can be comprehended by 5th grade students."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}