comput\n",
"Computation => comput\n",
"Computing => comput\n",
"Computed => comput\n",
"Compute => comput\n",
"Compulsive => compuls\n"
]
}
],
"source": [
"try:\n",
" import nltk\n",
"\n",
" stemmer = nltk.PorterStemmer()\n",
" for word in (\"Computations\", \"Computation\", \"Computing\", \"Computed\", \"Compute\", \"Compulsive\"):\n",
" print(word, \"=>\", stemmer.stem(word))\n",
"except ImportError:\n",
" print(\"Error: stemming requires the NLTK module.\")\n",
" stemmer = None"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will also need a way to replace URLs with the word \"URL\". For this, we could use hard core [regular expressions](https://mathiasbynens.be/demo/url-regex) but we will just use the [urlextract](https://github.com/lipoja/URLExtract) library. You can install it with the following command (don't forget to activate your virtualenv first; if you don't have one, you will likely need administrator rights, or use the `--user` option):\n",
"\n",
"`$ pip3 install urlextract`"
]
},
{
"cell_type": "code",
"execution_count": 147,
"metadata": {},
"outputs": [],
"source": [
"# if running this notebook on Colab or Kaggle, we just pip install urlextract\n",
"if IS_COLAB or IS_KAGGLE:\n",
" %pip install -q -U urlextract"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Note:** inside a Jupyter notebook, always use `%pip` instead of `!pip`, as `!pip` may install the library inside the wrong environment, while `%pip` makes sure it's installed inside the currently running environment."
]
},
{
"cell_type": "code",
"execution_count": 148,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['github.com', 'https://youtu.be/7Pq-S557XQU?t=3m32s']\n"
]
}
],
"source": [
"try:\n",
" import urlextract # may require an Internet connection to download root domain names\n",
" \n",
" url_extractor = urlextract.URLExtract()\n",
" print(url_extractor.find_urls(\"Will it detect github.com and https://youtu.be/7Pq-S557XQU?t=3m32s\"))\n",
"except ImportError:\n",
" print(\"Error: replacing URLs requires the urlextract module.\")\n",
" url_extractor = None"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We are ready to put all this together into a transformer that we will use to convert emails to word counters. Note that we split sentences into words using Python's `split()` method, which uses whitespaces for word boundaries. This works for many written languages, but not all. For example, Chinese and Japanese scripts generally don't use spaces between words, and Vietnamese often uses spaces even between syllables. It's okay in this exercise, because the dataset is (mostly) in English."
]
},
{
"cell_type": "code",
"execution_count": 149,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.base import BaseEstimator, TransformerMixin\n",
"\n",
"class EmailToWordCounterTransformer(BaseEstimator, TransformerMixin):\n",
" def __init__(self, strip_headers=True, lower_case=True, remove_punctuation=True,\n",
" replace_urls=True, replace_numbers=True, stemming=True):\n",
" self.strip_headers = strip_headers\n",
" self.lower_case = lower_case\n",
" self.remove_punctuation = remove_punctuation\n",
" self.replace_urls = replace_urls\n",
" self.replace_numbers = replace_numbers\n",
" self.stemming = stemming\n",
" def fit(self, X, y=None):\n",
" return self\n",
" def transform(self, X, y=None):\n",
" X_transformed = []\n",
" for email in X:\n",
" text = email_to_text(email) or \"\"\n",
" if self.lower_case:\n",
" text = text.lower()\n",
" if self.replace_urls and url_extractor is not None:\n",
" urls = list(set(url_extractor.find_urls(text)))\n",
" urls.sort(key=lambda url: len(url), reverse=True)\n",
" for url in urls:\n",
" text = text.replace(url, \" URL \")\n",
" if self.replace_numbers:\n",
" text = re.sub(r'\\d+(?:\\.\\d*)?(?:[eE][+-]?\\d+)?', 'NUMBER', text)\n",
" if self.remove_punctuation:\n",
" text = re.sub(r'\\W+', ' ', text, flags=re.M)\n",
" word_counts = Counter(text.split())\n",
" if self.stemming and stemmer is not None:\n",
" stemmed_word_counts = Counter()\n",
" for word, count in word_counts.items():\n",
" stemmed_word = stemmer.stem(word)\n",
" stemmed_word_counts[stemmed_word] += count\n",
" word_counts = stemmed_word_counts\n",
" X_transformed.append(word_counts)\n",
" return np.array(X_transformed)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's try this transformer on a few emails:"
]
},
{
"cell_type": "code",
"execution_count": 150,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([Counter({'chuck': 1, 'murcko': 1, 'wrote': 1, 'stuff': 1, 'yawn': 1, 'r': 1}),\n",
" Counter({'the': 11, 'of': 9, 'and': 8, 'all': 3, 'christian': 3, 'to': 3, 'by': 3, 'jefferson': 2, 'i': 2, 'have': 2, 'superstit': 2, 'one': 2, 'on': 2, 'been': 2, 'ha': 2, 'half': 2, 'rogueri': 2, 'teach': 2, 'jesu': 2, 'some': 1, 'interest': 1, 'quot': 1, 'url': 1, 'thoma': 1, 'examin': 1, 'known': 1, 'word': 1, 'do': 1, 'not': 1, 'find': 1, 'in': 1, 'our': 1, 'particular': 1, 'redeem': 1, 'featur': 1, 'they': 1, 'are': 1, 'alik': 1, 'found': 1, 'fabl': 1, 'mytholog': 1, 'million': 1, 'innoc': 1, 'men': 1, 'women': 1, 'children': 1, 'sinc': 1, 'introduct': 1, 'burnt': 1, 'tortur': 1, 'fine': 1, 'imprison': 1, 'what': 1, 'effect': 1, 'thi': 1, 'coercion': 1, 'make': 1, 'world': 1, 'fool': 1, 'other': 1, 'hypocrit': 1, 'support': 1, 'error': 1, 'over': 1, 'earth': 1, 'six': 1, 'histor': 1, 'american': 1, 'john': 1, 'e': 1, 'remsburg': 1, 'letter': 1, 'william': 1, 'short': 1, 'again': 1, 'becom': 1, 'most': 1, 'pervert': 1, 'system': 1, 'that': 1, 'ever': 1, 'shone': 1, 'man': 1, 'absurd': 1, 'untruth': 1, 'were': 1, 'perpetr': 1, 'upon': 1, 'a': 1, 'larg': 1, 'band': 1, 'dupe': 1, 'import': 1, 'led': 1, 'paul': 1, 'first': 1, 'great': 1, 'corrupt': 1}),\n",
" Counter({'url': 4, 's': 3, 'group': 3, 'to': 3, 'in': 2, 'forteana': 2, 'martin': 2, 'an': 2, 'and': 2, 'we': 2, 'is': 2, 'yahoo': 2, 'unsubscrib': 2, 'y': 1, 'adamson': 1, 'wrote': 1, 'for': 1, 'altern': 1, 'rather': 1, 'more': 1, 'factual': 1, 'base': 1, 'rundown': 1, 'on': 1, 'hamza': 1, 'career': 1, 'includ': 1, 'hi': 1, 'belief': 1, 'that': 1, 'all': 1, 'non': 1, 'muslim': 1, 'yemen': 1, 'should': 1, 'be': 1, 'murder': 1, 'outright': 1, 'know': 1, 'how': 1, 'unbias': 1, 'memri': 1, 'don': 1, 't': 1, 'html': 1, 'rob': 1, 'sponsor': 1, 'number': 1, 'dvd': 1, 'free': 1, 'p': 1, 'join': 1, 'now': 1, 'from': 1, 'thi': 1, 'send': 1, 'email': 1, 'egroup': 1, 'com': 1, 'your': 1, 'use': 1, 'of': 1, 'subject': 1})],\n",
" dtype=object)"
]
},
"execution_count": 150,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_few = X_train[:3]\n",
"X_few_wordcounts = EmailToWordCounterTransformer().fit_transform(X_few)\n",
"X_few_wordcounts"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This looks about right!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we have the word counts, and we need to convert them to vectors. For this, we will build another transformer whose `fit()` method will build the vocabulary (an ordered list of the most common words) and whose `transform()` method will use the vocabulary to convert word counts to vectors. The output is a sparse matrix."
]
},
{
"cell_type": "code",
"execution_count": 151,
"metadata": {},
"outputs": [],
"source": [
"from scipy.sparse import csr_matrix\n",
"\n",
"class WordCounterToVectorTransformer(BaseEstimator, TransformerMixin):\n",
" def __init__(self, vocabulary_size=1000):\n",
" self.vocabulary_size = vocabulary_size\n",
" def fit(self, X, y=None):\n",
" total_count = Counter()\n",
" for word_count in X:\n",
" for word, count in word_count.items():\n",
" total_count[word] += min(count, 10)\n",
" most_common = total_count.most_common()[:self.vocabulary_size]\n",
" self.vocabulary_ = {word: index + 1 for index, (word, count) in enumerate(most_common)}\n",
" return self\n",
" def transform(self, X, y=None):\n",
" rows = []\n",
" cols = []\n",
" data = []\n",
" for row, word_count in enumerate(X):\n",
" for word, count in word_count.items():\n",
" rows.append(row)\n",
" cols.append(self.vocabulary_.get(word, 0))\n",
" data.append(count)\n",
" return csr_matrix((data, (rows, cols)), shape=(len(X), self.vocabulary_size + 1))"
]
},
{
"cell_type": "code",
"execution_count": 152,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<3x11 sparse matrix of type ''\n",
"\twith 20 stored elements in Compressed Sparse Row format>"
]
},
"execution_count": 152,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vocab_transformer = WordCounterToVectorTransformer(vocabulary_size=10)\n",
"X_few_vectors = vocab_transformer.fit_transform(X_few_wordcounts)\n",
"X_few_vectors"
]
},
{
"cell_type": "code",
"execution_count": 153,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
" [99, 11, 9, 8, 3, 1, 3, 1, 3, 2, 3],\n",
" [67, 0, 1, 2, 3, 4, 1, 2, 0, 1, 0]], dtype=int64)"
]
},
"execution_count": 153,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_few_vectors.toarray()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What does this matrix mean? Well, the 99 in the second row, first column, means that the second email contains 99 words that are not part of the vocabulary. The 11 next to it means that the first word in the vocabulary is present 11 times in this email. The 9 next to it means that the second word is present 9 times, and so on. You can look at the vocabulary to know which words we are talking about. The first word is \"the\", the second word is \"of\", etc."
]
},
{
"cell_type": "code",
"execution_count": 154,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'the': 1,\n",
" 'of': 2,\n",
" 'and': 3,\n",
" 'to': 4,\n",
" 'url': 5,\n",
" 'all': 6,\n",
" 'in': 7,\n",
" 'christian': 8,\n",
" 'on': 9,\n",
" 'by': 10}"
]
},
"execution_count": 154,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vocab_transformer.vocabulary_"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We are now ready to train our first spam classifier! Let's transform the whole dataset:"
]
},
{
"cell_type": "code",
"execution_count": 155,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.pipeline import Pipeline\n",
"\n",
"preprocess_pipeline = Pipeline([\n",
" (\"email_to_wordcount\", EmailToWordCounterTransformer()),\n",
" (\"wordcount_to_vector\", WordCounterToVectorTransformer()),\n",
"])\n",
"\n",
"X_train_transformed = preprocess_pipeline.fit_transform(X_train)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Note**: to be future-proof, we set `solver=\"lbfgs\"` since this will be the default value in Scikit-Learn 0.22."
]
},
{
"cell_type": "code",
"execution_count": 156,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.1s remaining: 0.0s\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[CV] ................................................................\n",
"[CV] .................................... , score=0.981, total= 0.1s\n",
"[CV] ................................................................\n",
"[CV] .................................... , score=0.985, total= 0.2s\n",
"[CV] ................................................................\n",
"[CV] .................................... , score=0.991, total= 0.2s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.3s remaining: 0.0s\n",
"[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.5s finished\n"
]
},
{
"data": {
"text/plain": [
"0.9858333333333333"
]
},
"execution_count": 156,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.model_selection import cross_val_score\n",
"\n",
"log_clf = LogisticRegression(solver=\"lbfgs\", max_iter=1000, random_state=42)\n",
"score = cross_val_score(log_clf, X_train_transformed, y_train, cv=3, verbose=3)\n",
"score.mean()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Over 98.5%, not bad for a first try! :) However, remember that we are using the \"easy\" dataset. You can try with the harder datasets, the results won't be so amazing. You would have to try multiple models, select the best ones and fine-tune them using cross-validation, and so on.\n",
"\n",
"But you get the picture, so let's stop now, and just print out the precision/recall we get on the test set:"
]
},
{
"cell_type": "code",
"execution_count": 157,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Precision: 95.88%\n",
"Recall: 97.89%\n"
]
}
],
"source": [
"from sklearn.metrics import precision_score, recall_score\n",
"\n",
"X_test_transformed = preprocess_pipeline.transform(X_test)\n",
"\n",
"log_clf = LogisticRegression(solver=\"lbfgs\", max_iter=1000, random_state=42)\n",
"log_clf.fit(X_train_transformed, y_train)\n",
"\n",
"y_pred = log_clf.predict(X_test_transformed)\n",
"\n",
"print(\"Precision: {:.2f}%\".format(100 * precision_score(y_test, y_pred)))\n",
"print(\"Recall: {:.2f}%\".format(100 * recall_score(y_test, y_pred)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
},
"nav_menu": {},
"toc": {
"navigate_menu": true,
"number_sections": true,
"sideBar": true,
"threshold": 6,
"toc_cell": false,
"toc_section_display": "block",
"toc_window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 4
}
|
![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
\n",
" ![](\"https://kaggle.com/static/images/open-in-kaggle.svg\")
\n",
"