{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take a look at a simple way to trying to identify some structure in our data. Getting some understanding of the data is an important first step before we even start to look at using machine learning techniques to train a model; in this notebook, we'll approach that problem from a couple of different angles.\n", "\n", "We'll start by loading our training data." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "data = pd.read_parquet(\"data/training.parquet\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our training data (which we generated in [the previous notebook](00-generator.ipynb)) consists of labels (either `legitimate` or `spam`) and short documents of plausible English text. We can inspect these data:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", " | label | \n", "text | \n", "
---|---|---|
19705 | \n", "spam | \n", "They are light and crunchy. Nowhere near the t... | \n", "
1361 | \n", "legitimate | \n", "Elinor, with a very curious specimen of heath.... | \n", "
9178 | \n", "spam | \n", "As with any product, consult your health cae p... | \n", "
6493 | \n", "spam | \n", "They're a training treat when I needed it. Try... | \n", "
11696 | \n", "legitimate | \n", "Thank you, thank you. She heard him sigh. Hone... | \n", "
8709 | \n", "legitimate | \n", "And what would you like to have now? Poor man!... | \n", "
9263 | \n", "legitimate | \n", "It will be giving him so much before. I admire... | \n", "
9191 | \n", "legitimate | \n", "Amazon and Wild Ride Beef Jerky, so I decided ... | \n", "
33 | \n", "legitimate | \n", "I decided to go for foods that are advertised ... | \n", "
17872 | \n", "spam | \n", "They taste great and totally hit the spot afte... | \n", "
2888 | \n", "spam | \n", "Well worth trying. It had a slightly bitter af... | \n", "
815 | \n", "spam | \n", "Don't buy it! Nor was it packaged in any sort ... | \n", "
17602 | \n", "legitimate | \n", "She did not regret it; but she felt that it ou... | \n", "
11302 | \n", "legitimate | \n", "Darcy was not less answerable for Wickham's ab... | \n", "
12973 | \n", "legitimate | \n", "It would be a comfort to think how her own cha... | \n", "
19350 | \n", "spam | \n", "In addition, I have made a wonderful, irreplac... | \n", "
9415 | \n", "spam | \n", "The foil lined container is bug proof and keep... | \n", "
7929 | \n", "spam | \n", "I went to their website & found they cost less... | \n", "
9291 | \n", "spam | \n", "My dog really enjoys these too, so they match ... | \n", "
13270 | \n", "spam | \n", "I know this is special. I'm so glad I still ha... | \n", "
3268 | \n", "spam | \n", "Great price at Amazon is much more refreshing,... | \n", "
3301 | \n", "spam | \n", "Again... these are CAT TREATS, not an AUDIO CD... | \n", "
15092 | \n", "legitimate | \n", "No! With such an Anhalt, however, Miss Crawfor... | \n", "
13030 | \n", "spam | \n", "I wanted something that had extra vitamins in ... | \n", "
9816 | \n", "spam | \n", "The other teas I've tried but this product tas... | \n", "
16842 | \n", "spam | \n", "Glad to see it finally at my door. i like the ... | \n", "
9090 | \n", "spam | \n", "We all need to be taken to the vet's for outsi... | \n", "
4210 | \n", "legitimate | \n", "A piece of this biscuit broke off and got stuc... | \n", "
9806 | \n", "spam | \n", "Drank the Starbucks variety described here, th... | \n", "
9765 | \n", "legitimate | \n", "She was married--married against her inclinati... | \n", "
481 | \n", "legitimate | \n", "They were soon gone again, rising from their s... | \n", "
16387 | \n", "legitimate | \n", "As the blow was given, Emma felt that Mrs. Wes... | \n", "
2715 | \n", "spam | \n", "However on a couple of pages of instructions f... | \n", "
18645 | \n", "spam | \n", "I would drink it again next spring. The other ... | \n", "
639 | \n", "spam | \n", "If I could just store it in glass bottles tast... | \n", "
15503 | \n", "spam | \n", "As the title says, I trashed it after a SIP. S... | \n", "
1747 | \n", "spam | \n", "Great for kids before school. It tasted very a... | \n", "
9861 | \n", "legitimate | \n", "Jane looked at Elizabeth with surprise and con... | \n", "
758 | \n", "legitimate | \n", "How very provoking! Married women, you know, m... | \n", "
19577 | \n", "spam | \n", "We wouldn't even know it gf. In fact my cats l... | \n", "
8990 | \n", "legitimate | \n", "After a little reflection, venture the followi... | \n", "
18497 | \n", "spam | \n", "It is better to pay $4-$5 more and get a full ... | \n", "
14788 | \n", "legitimate | \n", "He liked very much to have been the means of g... | \n", "
15865 | \n", "legitimate | \n", "You know I have a cup right now! Lady Bridget,... | \n", "
10786 | \n", "spam | \n", "These almonds are the best. It sure must be VE... | \n", "
14603 | \n", "spam | \n", "I like to keep a supply in the pantry -- what ... | \n", "
8310 | \n", "spam | \n", "He loves these candies! Thank you! | \n", "
11767 | \n", "spam | \n", "I waited 20 mins to cook. I couldn't get them ... | \n", "
8656 | \n", "spam | \n", "At times, I add a tablespoon of Hillsbrothers ... | \n", "
4928 | \n", "spam | \n", "I can also rate the She Crab Soup they make be... | \n", "
\n", " | label | \n", "text | \n", "
---|---|---|
19705 | \n", "spam | \n", "They are light and crunchy. Nowhere near the t... | \n", "
1361 | \n", "legitimate | \n", "Elinor, with a very curious specimen of heath.... | \n", "
9178 | \n", "spam | \n", "As with any product, consult your health cae p... | \n", "
6493 | \n", "spam | \n", "They're a training treat when I needed it. Try... | \n", "
11696 | \n", "legitimate | \n", "Thank you, thank you. She heard him sigh. Hone... | \n", "
8709 | \n", "legitimate | \n", "And what would you like to have now? Poor man!... | \n", "
9263 | \n", "legitimate | \n", "It will be giving him so much before. I admire... | \n", "
9191 | \n", "legitimate | \n", "Amazon and Wild Ride Beef Jerky, so I decided ... | \n", "
33 | \n", "legitimate | \n", "I decided to go for foods that are advertised ... | \n", "
17872 | \n", "spam | \n", "They taste great and totally hit the spot afte... | \n", "
2888 | \n", "spam | \n", "Well worth trying. It had a slightly bitter af... | \n", "
815 | \n", "spam | \n", "Don't buy it! Nor was it packaged in any sort ... | \n", "
17602 | \n", "legitimate | \n", "She did not regret it; but she felt that it ou... | \n", "
11302 | \n", "legitimate | \n", "Darcy was not less answerable for Wickham's ab... | \n", "
12973 | \n", "legitimate | \n", "It would be a comfort to think how her own cha... | \n", "
19350 | \n", "spam | \n", "In addition, I have made a wonderful, irreplac... | \n", "
9415 | \n", "spam | \n", "The foil lined container is bug proof and keep... | \n", "
7929 | \n", "spam | \n", "I went to their website & found they cost less... | \n", "
9291 | \n", "spam | \n", "My dog really enjoys these too, so they match ... | \n", "
13270 | \n", "spam | \n", "I know this is special. I'm so glad I still ha... | \n", "
3268 | \n", "spam | \n", "Great price at Amazon is much more refreshing,... | \n", "
3301 | \n", "spam | \n", "Again... these are CAT TREATS, not an AUDIO CD... | \n", "
15092 | \n", "legitimate | \n", "No! With such an Anhalt, however, Miss Crawfor... | \n", "
13030 | \n", "spam | \n", "I wanted something that had extra vitamins in ... | \n", "
9816 | \n", "spam | \n", "The other teas I've tried but this product tas... | \n", "
16842 | \n", "spam | \n", "Glad to see it finally at my door. i like the ... | \n", "
9090 | \n", "spam | \n", "We all need to be taken to the vet's for outsi... | \n", "
4210 | \n", "legitimate | \n", "A piece of this biscuit broke off and got stuc... | \n", "
9806 | \n", "spam | \n", "Drank the Starbucks variety described here, th... | \n", "
9765 | \n", "legitimate | \n", "She was married--married against her inclinati... | \n", "
481 | \n", "legitimate | \n", "They were soon gone again, rising from their s... | \n", "
16387 | \n", "legitimate | \n", "As the blow was given, Emma felt that Mrs. Wes... | \n", "
2715 | \n", "spam | \n", "However on a couple of pages of instructions f... | \n", "
18645 | \n", "spam | \n", "I would drink it again next spring. The other ... | \n", "
639 | \n", "spam | \n", "If I could just store it in glass bottles tast... | \n", "
15503 | \n", "spam | \n", "As the title says, I trashed it after a SIP. S... | \n", "
1747 | \n", "spam | \n", "Great for kids before school. It tasted very a... | \n", "
9861 | \n", "legitimate | \n", "Jane looked at Elizabeth with surprise and con... | \n", "
758 | \n", "legitimate | \n", "How very provoking! Married women, you know, m... | \n", "
19577 | \n", "spam | \n", "We wouldn't even know it gf. In fact my cats l... | \n", "
8990 | \n", "legitimate | \n", "After a little reflection, venture the followi... | \n", "
18497 | \n", "spam | \n", "It is better to pay $4-$5 more and get a full ... | \n", "
14788 | \n", "legitimate | \n", "He liked very much to have been the means of g... | \n", "
15865 | \n", "legitimate | \n", "You know I have a cup right now! Lady Bridget,... | \n", "
10786 | \n", "spam | \n", "These almonds are the best. It sure must be VE... | \n", "
14603 | \n", "spam | \n", "I like to keep a supply in the pantry -- what ... | \n", "
8310 | \n", "spam | \n", "He loves these candies! Thank you! | \n", "
11767 | \n", "spam | \n", "I waited 20 mins to cook. I couldn't get them ... | \n", "
8656 | \n", "spam | \n", "At times, I add a tablespoon of Hillsbrothers ... | \n", "
4928 | \n", "spam | \n", "I can also rate the She Crab Soup they make be... | \n", "
\n", " | x | \n", "y | \n", "
---|---|---|
36009 | \n", "0.008177 | \n", "0.006311 | \n", "
35983 | \n", "0.009110 | \n", "0.003361 | \n", "
11480 | \n", "-0.006559 | \n", "0.003144 | \n", "
17391 | \n", "0.001298 | \n", "-0.015437 | \n", "
16223 | \n", "-0.009819 | \n", "0.006162 | \n", "
18837 | \n", "0.000296 | \n", "-0.013754 | \n", "
25407 | \n", "0.001718 | \n", "0.008952 | \n", "
38800 | \n", "0.005547 | \n", "0.012576 | \n", "
2933 | \n", "-0.002714 | \n", "-0.002964 | \n", "
25761 | \n", "0.006497 | \n", "-0.001630 | \n", "
8853 | \n", "-0.006679 | \n", "0.000324 | \n", "
38188 | \n", "-0.000280 | \n", "0.005572 | \n", "
2099 | \n", "-0.001983 | \n", "-0.001268 | \n", "
10625 | \n", "-0.004496 | \n", "0.000010 | \n", "
13738 | \n", "0.000262 | \n", "0.005045 | \n", "
34241 | \n", "0.015671 | \n", "-0.001397 | \n", "
17495 | \n", "-0.008758 | \n", "0.005778 | \n", "
17226 | \n", "-0.000544 | \n", "-0.004336 | \n", "
19546 | \n", "-0.007105 | \n", "-0.001938 | \n", "
26465 | \n", "0.007774 | \n", "-0.005543 | \n", "
4095 | \n", "0.003267 | \n", "0.003868 | \n", "
20293 | \n", "0.002514 | \n", "0.005010 | \n", "
25274 | \n", "0.002884 | \n", "-0.002202 | \n", "
34850 | \n", "0.010031 | \n", "-0.006281 | \n", "
28430 | \n", "0.008400 | \n", "0.000434 | \n", "
39612 | \n", "0.008303 | \n", "0.004749 | \n", "
17833 | \n", "-0.004425 | \n", "0.003183 | \n", "
3610 | \n", "-0.004627 | \n", "-0.000478 | \n", "
8374 | \n", "-0.004780 | \n", "-0.003094 | \n", "
22111 | \n", "-0.003051 | \n", "0.008875 | \n", "
17697 | \n", "-0.010143 | \n", "0.004491 | \n", "
9713 | \n", "-0.005250 | \n", "0.007804 | \n", "
18927 | \n", "-0.002573 | \n", "-0.001829 | \n", "
34012 | \n", "0.003427 | \n", "0.005076 | \n", "
9536 | \n", "0.002619 | \n", "-0.014223 | \n", "
17234 | \n", "0.000904 | \n", "0.012190 | \n", "
12238 | \n", "-0.012939 | \n", "0.003044 | \n", "
13178 | \n", "-0.006365 | \n", "0.006139 | \n", "
6301 | \n", "-0.009936 | \n", "-0.007318 | \n", "
38546 | \n", "0.006294 | \n", "-0.004103 | \n", "
7555 | \n", "-0.006165 | \n", "-0.003416 | \n", "
9407 | \n", "-0.005687 | \n", "0.006702 | \n", "
17187 | \n", "-0.008348 | \n", "0.000226 | \n", "
31800 | \n", "0.010628 | \n", "-0.002215 | \n", "
21695 | \n", "-0.003796 | \n", "0.005402 | \n", "
19199 | \n", "0.000538 | \n", "-0.004863 | \n", "
30641 | \n", "-0.001192 | \n", "0.005518 | \n", "
27659 | \n", "0.005007 | \n", "-0.002051 | \n", "
5468 | \n", "-0.002287 | \n", "-0.001838 | \n", "
30887 | \n", "0.014977 | \n", "0.001127 | \n", "
\n", " | x | \n", "y | \n", "
---|---|---|
36009 | \n", "0.008177 | \n", "0.006311 | \n", "
35983 | \n", "0.009110 | \n", "0.003361 | \n", "
11480 | \n", "-0.006559 | \n", "0.003144 | \n", "
17391 | \n", "0.001298 | \n", "-0.015437 | \n", "
16223 | \n", "-0.009819 | \n", "0.006162 | \n", "
18837 | \n", "0.000296 | \n", "-0.013754 | \n", "
25407 | \n", "0.001718 | \n", "0.008952 | \n", "
38800 | \n", "0.005547 | \n", "0.012576 | \n", "
2933 | \n", "-0.002714 | \n", "-0.002964 | \n", "
25761 | \n", "0.006497 | \n", "-0.001630 | \n", "
8853 | \n", "-0.006679 | \n", "0.000324 | \n", "
38188 | \n", "-0.000280 | \n", "0.005572 | \n", "
2099 | \n", "-0.001983 | \n", "-0.001268 | \n", "
10625 | \n", "-0.004496 | \n", "0.000010 | \n", "
13738 | \n", "0.000262 | \n", "0.005045 | \n", "
34241 | \n", "0.015671 | \n", "-0.001397 | \n", "
17495 | \n", "-0.008758 | \n", "0.005778 | \n", "
17226 | \n", "-0.000544 | \n", "-0.004336 | \n", "
19546 | \n", "-0.007105 | \n", "-0.001938 | \n", "
26465 | \n", "0.007774 | \n", "-0.005543 | \n", "
4095 | \n", "0.003267 | \n", "0.003868 | \n", "
20293 | \n", "0.002514 | \n", "0.005010 | \n", "
25274 | \n", "0.002884 | \n", "-0.002202 | \n", "
34850 | \n", "0.010031 | \n", "-0.006281 | \n", "
28430 | \n", "0.008400 | \n", "0.000434 | \n", "
39612 | \n", "0.008303 | \n", "0.004749 | \n", "
17833 | \n", "-0.004425 | \n", "0.003183 | \n", "
3610 | \n", "-0.004627 | \n", "-0.000478 | \n", "
8374 | \n", "-0.004780 | \n", "-0.003094 | \n", "
22111 | \n", "-0.003051 | \n", "0.008875 | \n", "
17697 | \n", "-0.010143 | \n", "0.004491 | \n", "
9713 | \n", "-0.005250 | \n", "0.007804 | \n", "
18927 | \n", "-0.002573 | \n", "-0.001829 | \n", "
34012 | \n", "0.003427 | \n", "0.005076 | \n", "
9536 | \n", "0.002619 | \n", "-0.014223 | \n", "
17234 | \n", "0.000904 | \n", "0.012190 | \n", "
12238 | \n", "-0.012939 | \n", "0.003044 | \n", "
13178 | \n", "-0.006365 | \n", "0.006139 | \n", "
6301 | \n", "-0.009936 | \n", "-0.007318 | \n", "
38546 | \n", "0.006294 | \n", "-0.004103 | \n", "
7555 | \n", "-0.006165 | \n", "-0.003416 | \n", "
9407 | \n", "-0.005687 | \n", "0.006702 | \n", "
17187 | \n", "-0.008348 | \n", "0.000226 | \n", "
31800 | \n", "0.010628 | \n", "-0.002215 | \n", "
21695 | \n", "-0.003796 | \n", "0.005402 | \n", "
19199 | \n", "0.000538 | \n", "-0.004863 | \n", "
30641 | \n", "-0.001192 | \n", "0.005518 | \n", "
27659 | \n", "0.005007 | \n", "-0.002051 | \n", "
5468 | \n", "-0.002287 | \n", "-0.001838 | \n", "
30887 | \n", "0.014977 | \n", "0.001127 | \n", "