{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Clustering with News Publications\n", "Author: Matthew Huh\n", " \n", "## Overview\n", "\n", "For the most part, people are free to choose what news outlets they read and follow. In the United States, there is a near-endless list of sites that people can choose from in order to get their daily news and over time, they develop preferences for sites that they are more attached to, and do their best to avoid. Now these affinities are developed through a combination of means ranging from affiliations, vocabulary, prose, and so forth.\n", "\n", "What I would like to examine in this project is if it is possible to differentiate from several different publications with their respective perks / quirks. \n", "\n", "## About the Data\n", "\n", "This dataset was obtained from Kaggle, and contains a collection of 142,570 articles from 15 different publications.\n", "\n", "The publications within this dataset are\n", "1. CNN\n", "2. Breitbart\n", "3. Vox\n", "4. Washington Post\n", "5. New York Post\n", "6. National Review\n", "7. NPR\n", "8. Guardian\n", "9. Talking Points Memo\n", "10. Atlantic\n", "11. Reuters\n", "12. Fox News\n", "13. Business Insider\n", "14. Buzzfeed News\n", "15. New York Times\n", "\n", "## Research Question\n", "\n", "As this is an unsupervised learning project first and foremost, the project will have 3 goals.\n", "\n", "1. The first goal is to prepare the articles in the dataset for modelling using various Natural Language Processing (NLP) methods to re-represent the data in numbers rather than words\n", "2. Cluster the data to determine if we can identify the articles and associate them as different groups.\n", "3. Determine if we can predict the structure of the article based on the publisher.\n", "\n", "## Packages" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# Basic imports\n", "import os\n", "import numpy as np\n", "import pandas as pd\n", "import scipy\n", "import sklearn\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "%matplotlib inline\n", "\n", "# Machine Learning packages\n", "from sklearn.feature_selection import SelectKBest, f_classif\n", "from sklearn.feature_selection import chi2\n", "from sklearn.preprocessing import normalize\n", "from sklearn import ensemble\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.model_selection import cross_val_score\n", "from sklearn.model_selection import GridSearchCV\n", "\n", "# Clustering packages\n", "import sklearn.cluster as cluster\n", "from sklearn.cluster import KMeans\n", "from sklearn.cluster import MeanShift, estimate_bandwidth\n", "from sklearn.cluster import SpectralClustering\n", "from sklearn.cluster import AffinityPropagation\n", "from scipy.spatial.distance import cdist\n", "\n", "# Natural Language processing\n", "import re\n", "import spacy\n", "import nltk\n", "from nltk.corpus import stopwords\n", "from nltk.stem import WordNetLemmatizer\n", "from collections import Counter\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.feature_extraction.text import TfidfVectorizer\n", "from sklearn.datasets import fetch_rcv1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data Preview\n", "\n", "The first matter of business is to import the articles from a local directory and merge them." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0idtitlepublicationauthordateyearmonthurlcontent
0017283House Republicans Fret About Winning Their Hea...New York TimesCarl Hulse2016-12-312016.012.0NaNWASHINGTON — Congressional Republicans have...
1117284Rift Between Officers and Residents as Killing...New York TimesBenjamin Mueller and Al Baker2017-06-192017.06.0NaNAfter the bullet shells get counted, the blood...
2217285Tyrus Wong, ‘Bambi’ Artist Thwarted by Racial ...New York TimesMargalit Fox2017-01-062017.01.0NaNWhen Walt Disney’s “Bambi” opened in 1942, cri...
3317286Among Deaths in 2016, a Heavy Toll in Pop Musi...New York TimesWilliam McDonald2017-04-102017.04.0NaNDeath may be the great equalizer, but it isn’t...
4417287Kim Jong-un Says North Korea Is Preparing to T...New York TimesChoe Sang-Hun2017-01-022017.01.0NaNSEOUL, South Korea — North Korea’s leader, ...
\n", "
" ], "text/plain": [ " Unnamed: 0 id title \\\n", "0 0 17283 House Republicans Fret About Winning Their Hea... \n", "1 1 17284 Rift Between Officers and Residents as Killing... \n", "2 2 17285 Tyrus Wong, ‘Bambi’ Artist Thwarted by Racial ... \n", "3 3 17286 Among Deaths in 2016, a Heavy Toll in Pop Musi... \n", "4 4 17287 Kim Jong-un Says North Korea Is Preparing to T... \n", "\n", " publication author date year month \\\n", "0 New York Times Carl Hulse 2016-12-31 2016.0 12.0 \n", "1 New York Times Benjamin Mueller and Al Baker 2017-06-19 2017.0 6.0 \n", "2 New York Times Margalit Fox 2017-01-06 2017.0 1.0 \n", "3 New York Times William McDonald 2017-04-10 2017.0 4.0 \n", "4 New York Times Choe Sang-Hun 2017-01-02 2017.0 1.0 \n", "\n", " url content \n", "0 NaN WASHINGTON — Congressional Republicans have... \n", "1 NaN After the bullet shells get counted, the blood... \n", "2 NaN When Walt Disney’s “Bambi” opened in 1942, cri... \n", "3 NaN Death may be the great equalizer, but it isn’t... \n", "4 NaN SEOUL, South Korea — North Korea’s leader, ... " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create list of files from directory\n", "filelist = os.listdir('articles')\n", "\n", "# Import the files\n", "df_list = [pd.read_csv(file) for file in filelist]\n", "\n", "#concatenate them together\n", "articles = pd.concat(df_list)\n", "\n", "# Preview the data\n", "articles.head()" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(142570, 10)" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Print the size of the dataset\n", "articles.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So we have 142,570 articles in the dataset but unfortunately, NLP is quite memory intensive, so we will have to sample the dataset unless you happen to have over 120 GB of memory on your local device. Using a 10% sample still leaves us with 140,000 articles and will be used for the duration of this project." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# Sample the dataset for optimal performance\n", "articles = articles.sample(frac=0.1)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array(['Washington Post', 'CNN', 'New York Post', 'Buzzfeed News', 'NPR',\n", " 'Guardian', 'Breitbart', 'Atlantic', 'Business Insider',\n", " 'National Review', 'New York Times', 'Talking Points Memo',\n", " 'Reuters', 'Vox', 'Fox News'], dtype=object)" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Print out unique publisher names\n", "articles.publication.unique()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "title 14251\n", "publication 15\n", "author 3957\n", "date 1056\n", "url 8605\n", "content 14246\n", "dtype: int64" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Describe unique occurences for each categorical variable\n", "articles.select_dtypes(include=['object']).nunique()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are also other ways to trim down the dataset before processing. We aren't particularly interested in examining the dates for this research question, but it may be of interest in another. Let's check to see how many articles each author wrote; it may not be very useful to examine authors that are only responsible for a single article, as different authors from the same publisher may choose compose their works differently." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# Drop variables that have no impact on the outcome\n", "articles = articles[['title', 'publication', 'author', 'content']]" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "author\n", "Breitbart News 144\n", "Associated Press 130\n", "Pam Key 120\n", "Charlie Spiering 94\n", "Daniel Nussbaum 86\n", "Jerome Hudson 85\n", "AWR Hawkins 84\n", "John Hayward 74\n", "Warner Todd Huston 66\n", "Joel B. Pollak 63\n", "Camila Domonoske 63\n", "Breitbart London 60\n", "Post Editorial Board 56\n", "Ian Hanchett 55\n", "Trent Baker 52\n", "Reuters 52\n", "Alex Swoyer 49\n", "David French 46\n", "NPR Staff 43\n", "Bob Price 42\n", "Charlie Nash 41\n", "Jeff Poor 41\n", "David A. Graham 39\n", "German Lopez 38\n", "Breitbart Jerusalem 38\n", "Ben Kew 38\n", "Keith J. Kelly 36\n", "Bill Chappell 35\n", "Esme Cribb 35\n", "Katherine Rodriguez 34\n", " ... \n", "Larry Celona, Tina Moore and Laura Italiano 1\n", "Larry Celona, Tina Moore and Kenneth Garger 1\n", "Larry Celona, Tina Moore and Jazmin Rosa 1\n", "Larry Celona, Shawn Cohen and Natalie Musumeci 1\n", "Larry Celona, Jennifer Bain, Rebecca Rosenberg and Shawn Cohen 1\n", "Larry Celona, Jamie Schram and Emily Saul 1\n", "Lauren Russell 1\n", "Lauren Sommer 1\n", "Lauren Windle, The Sun 1\n", "Laurence Blair 1\n", "Lesley Wroughton and Yeganeh Torbati 1\n", "Lesley McClurg 1\n", "Lenore Skenazy 1\n", "Lela Moore and Sona Patel 1\n", "Lela Moore and Lindsey Underwood 1\n", "Leika Kihara and Tetsushi Kajimoto 1\n", "Leigh Alexander 1\n", "Lee Liberman Otis 1\n", "Lee Glendinning 1\n", "Leanna Garfield 1\n", "Lawrence Torcello 1\n", "Lawrence Summers 1\n", "Lawrence Ostlere 1\n", "Lawrence K. Altman, M.d. 1\n", "Lawrence Hurley and Valerie Volcovici 1\n", "Lawrence Hurley and Richard Cowan 1\n", "Lawrence Hurley and Andrew Chung 1\n", "Laurie Goodstein 1\n", "Laurie Goering 1\n", " Faith Karimi 1\n", "Length: 3957, dtype: int64" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# View most frequently occurring authors\n", "articles.groupby(['author']).size().sort_values(ascending=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Well, that partly explains how there are so many authors in this dataset. It seems as though there are over 15,000 authors, and many of them have only published one article, or have co-written multiple articles with other authors. This complicates the problem, so in order to best represent each author's writing style, let's see what happens if we simply remove all authors that only published one article as is." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/vnd.plotly.v1+html": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.plotly.v1+json": { "data": [ { "labels": [ "Washington Post", "CNN", "New York Post", "Buzzfeed News", "NPR", "Guardian", "Breitbart", "Atlantic", "Business Insider", "National Review", "New York Times", "Talking Points Memo", "Reuters", "Vox", "Fox News" ], "type": "pie", "values": [ 2312, 1809, 1168, 1167, 1119, 1104, 838, 816, 731, 672, 599, 502, 485, 482, 453 ] } ], "layout": { "autosize": false, "height": 600, "title": "Articles by Publication", "width": 800 } }, "text/html": [ "
" ], "text/vnd.plotly.v1+html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Plotly packages\n", "import plotly as py\n", "import plotly.graph_objs as go\n", "from plotly import tools\n", "import cufflinks as cf\n", "import ipywidgets as widgets\n", "from scipy import special\n", "py.offline.init_notebook_mode(connected=True)\n", "\n", "# Pass in values for our pie chart\n", "trace = go.Pie(labels=articles['publication'].unique(), values = articles['publication'].value_counts())\n", "\n", "# Create the layout\n", "layout = go.Layout(\n", " title = 'Articles by Publication',\n", " height = 600,\n", " width = 800,\n", " autosize = False\n", ")\n", "\n", "# Construct the chart\n", "fig = go.Figure(data = [trace], layout = layout)\n", "py.offline.iplot(fig, filename ='cufflinks/simple')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Feature Selection" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "# Drop author from the dataframe if they wrote less than 5 articles\n", "vc = articles['author'].value_counts()\n", "u = [i not in set(vc[vc<=4].index) for i in articles['author']]\n", "articles = articles[u]" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "title 9324\n", "publication 15\n", "author 608\n", "content 9318\n", "dtype: int64" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Reprint how many unique authors there are\n", "articles.select_dtypes(include=['object']).nunique()" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(9326, 4)" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# View number of articles after feature selection\n", "articles.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So after removing authors that composed fewer than 5 articles, we are left with 9k articles, or 67% of the data, and roughly 600/3900 of the authors. Now, we can create a better representation of each author since each author has at least 5 articles to evaluate from.\n", "\n", "## Text Cleaning\n", "\n", "Now that we've chosen which articles to use, it's time to clean them up and prepare them for feature engineering. What this section covers is the removal of annoying punctuation from the content, and reducing words to their lemmas to reduce the number of words that we are examining. Finally, we'll divide the articles into training and testing sets and separate our predictor, the words in the content, and the target, the publisher." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "def text_cleaner(text):\n", " # Visual inspection identifies a form of punctuation spaCy does not\n", " # recognize: the double dash '--'. Better get rid of it now!\n", " text = re.sub(r'--',' ',text)\n", " text = re.sub(\"[\\[].*?[\\]]\", \"\", text)\n", " text = ' '.join(text.split())\n", " return text" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titlepublicationauthorcontent
34296Lawmakers to Trump: Turn over transcript of me...Washington PostElise ViebeckA growing number of Republican and Democratic ...
39572Man charged with attacking Uber driver sues dr...New York PostAssociated PressCOSTA MESA, Calif. — A Southern California man...
5253Democratic Response To Trump’s Address To Cong...NPRNaNFollowing President Trump’s address to a joint...
48670Trump tries to salvage travel ban amid numerou...GuardianBen JacobsDonald Trump scrambled to salvage his controve...
32448Influential conservative group: Trump, DeVos s...Washington PostEmma BrownA policy manifesto from an influential conserv...
\n", "
" ], "text/plain": [ " title publication \\\n", "34296 Lawmakers to Trump: Turn over transcript of me... Washington Post \n", "39572 Man charged with attacking Uber driver sues dr... New York Post \n", "5253 Democratic Response To Trump’s Address To Cong... NPR \n", "48670 Trump tries to salvage travel ban amid numerou... Guardian \n", "32448 Influential conservative group: Trump, DeVos s... Washington Post \n", "\n", " author content \n", "34296 Elise Viebeck A growing number of Republican and Democratic ... \n", "39572 Associated Press COSTA MESA, Calif. — A Southern California man... \n", "5253 NaN Following President Trump’s address to a joint... \n", "48670 Ben Jacobs Donald Trump scrambled to salvage his controve... \n", "32448 Emma Brown A policy manifesto from an influential conserv... " ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Remove annoying punctuation from the articles\n", "articles['content'] = articles.content.map(lambda x: text_cleaner(str(x)))\n", "articles.head()" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "lemmatizer = WordNetLemmatizer()\n", "\n", "# Reduce all text to their lemmas\n", "for article in articles['content']:\n", " article = lemmatizer.lemmatize(article)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "# Identify predictor and target variables\n", "X = articles['content']\n", "y = articles['publication']\n", "\n", "# Create training and testing sets\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Tf-idf Vectorization\n", "\n", "The first types of features that we are going to add are the most useful words in our dataset. Now how are we going to determine which words are deemed the most \"useful\"? With TF-IDF vectorizer, of course.\n", "\n", "TF tracks the term frequency, or how often each word appears in all articles of text, while idf (or Inverse Document Frequency) is a value that places less weight on variables that occur too often and lose their predictive power. Put together, it's a tool that allows us to assign an importance value to each word in the entire dataset based on frequency in each row and throughout the database.\n", "\n", "These are the parameters that will be used for TF-IDF\n", "1. All words that appear in over half of the articles will be thrown out of the dataframe\n", "2. Only words that occur more than 5 times will be tracked\n", "3. Only the top 150 features (words) will be kept\n", "4. Stop words will be ignored (like, as, the)\n", "5. Cases will be ignored\n", "6. Shorter and longer articles will be treated equally\n", "7. Add 1 to document frequency in case we have to divide by 0" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of features: 150\n" ] } ], "source": [ "from sklearn.feature_extraction.text import TfidfVectorizer\n", "\n", "# Parameters for TF-idf vectorizer\n", "vectorizer = TfidfVectorizer(max_df=0.5,\n", " min_df=5, \n", " max_features=150, \n", " stop_words='english', \n", " lowercase=True, \n", " use_idf=True,\n", " norm=u'l2',\n", " smooth_idf=True\n", " )\n", "\n", "#Applying the vectorizer\n", "X_tfidf=vectorizer.fit_transform(X)\n", "print(\"Number of features: %d\" % X_tfidf.get_shape()[1])\n", "\n", "#splitting into training and test sets\n", "X_train_tfidf, X_test_tfidf, y_train, y_test = train_test_split(X_tfidf, y, test_size=0.25, random_state=42)\n", "\n", "#Removes all zeros from the matrix\n", "X_train_tfidf_csr = X_train_tfidf.tocsr()\n", "\n", "#number of paragraphs\n", "n = X_train_tfidf_csr.shape[0]\n", "\n", "#A list of dictionaries, one per paragraph\n", "tfidf_bypara = [{} for _ in range(0,n)]\n", "\n", "#List of features\n", "terms = vectorizer.get_feature_names()\n", "\n", "#for each paragraph, lists the feature words and their tf-idf scores\n", "for i, j in zip(*X_train_tfidf_csr.nonzero()):\n", " tfidf_bypara[i][terms[j]] = X_train_tfidf_csr[i, j]\n", "\n", "# Normalize the dataset \n", "X_norm = normalize(X_train_tfidf)\n", "\n", "# Convert from tf-idf matrix to dataframe\n", "X_normal = pd.DataFrame(data=X_norm.toarray())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Phrase count with spacy\n", "\n", "The second set of variables that we will be creating are counters of how often each publishers makes use of each part of speech, meaning adverbs, verbs, nouns, adjectives, as well as article length." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "# Instantiating spaCy\n", "nlp = spacy.load('en')\n", "X_train_words = []\n", "\n", "for row in X_train:\n", " # Processing each row for tokens\n", " row_doc = nlp(row)\n", " # Calculating length of each sentence\n", " sent_len = len(row_doc) \n", " # Initializing counts of different parts of speech\n", " advs = 0\n", " verb = 0\n", " noun = 0\n", " adj = 0\n", " for token in row_doc:\n", " # Identifying each part of speech and adding to counts\n", " if token.pos_ == 'ADV':\n", " advs +=1\n", " elif token.pos_ == 'VERB':\n", " verb +=1\n", " elif token.pos_ == 'NOUN':\n", " noun +=1\n", " elif token.pos_ == 'ADJ':\n", " adj +=1\n", " # Creating a list of all features for each sentence\n", " X_train_words.append([row_doc, advs, verb, noun, adj, sent_len])\n", "\n", "# Create dataframe with count of adverbs, verbs, nouns, and adjectives\n", "X_count = pd.DataFrame(data=X_train_words, columns=['BOW', 'ADV', 'VERB', 'NOUN', 'ADJ', 'sent_length'])\n", "\n", "# Change token count to token percentage\n", "for column in X_count.columns[1:5]:\n", " X_count[column] = X_count[column] / X_count['sent_length']\n", "\n", "# Normalize X_count\n", "X_counter = normalize(X_count.drop('BOW',axis=1))\n", "X_counter = pd.DataFrame(data=X_counter)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
0123401234...140141142143144145146147148149
00.0003280.0007050.0006230.0003930.9999990.0000000.0000000.00.0000000.0...0.00.0000000.0000000.00.00.1977980.00.3346320.000000.000000
10.0001100.0003270.0002750.0001251.0000000.0599440.0628710.00.0489740.0...0.00.0000000.4523180.00.00.0000000.00.0000000.000000.000000
20.0005820.0011360.0005540.0003600.9999990.0000000.0000000.00.0000000.0...0.00.0000000.0000000.00.00.0000000.00.0000000.000000.000000
30.0001370.0006300.0004250.0001821.0000000.0000000.0000000.00.0000000.0...0.00.0000000.0000000.00.00.0000000.00.0000000.000000.000000
40.0001280.0005480.0004110.0001831.0000000.0000000.0000000.00.0000000.0...0.00.1186690.0000000.00.00.0000000.00.0000000.051390.674409
\n", "

5 rows × 155 columns

\n", "
" ], "text/plain": [ " 0 1 2 3 4 0 1 2 \\\n", "0 0.000328 0.000705 0.000623 0.000393 0.999999 0.000000 0.000000 0.0 \n", "1 0.000110 0.000327 0.000275 0.000125 1.000000 0.059944 0.062871 0.0 \n", "2 0.000582 0.001136 0.000554 0.000360 0.999999 0.000000 0.000000 0.0 \n", "3 0.000137 0.000630 0.000425 0.000182 1.000000 0.000000 0.000000 0.0 \n", "4 0.000128 0.000548 0.000411 0.000183 1.000000 0.000000 0.000000 0.0 \n", "\n", " 3 4 ... 140 141 142 143 144 145 146 \\\n", "0 0.000000 0.0 ... 0.0 0.000000 0.000000 0.0 0.0 0.197798 0.0 \n", "1 0.048974 0.0 ... 0.0 0.000000 0.452318 0.0 0.0 0.000000 0.0 \n", "2 0.000000 0.0 ... 0.0 0.000000 0.000000 0.0 0.0 0.000000 0.0 \n", "3 0.000000 0.0 ... 0.0 0.000000 0.000000 0.0 0.0 0.000000 0.0 \n", "4 0.000000 0.0 ... 0.0 0.118669 0.000000 0.0 0.0 0.000000 0.0 \n", "\n", " 147 148 149 \n", "0 0.334632 0.00000 0.000000 \n", "1 0.000000 0.00000 0.000000 \n", "2 0.000000 0.00000 0.000000 \n", "3 0.000000 0.00000 0.000000 \n", "4 0.000000 0.05139 0.674409 \n", "\n", "[5 rows x 155 columns]" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Combine tf-idf matrix and phrase count matrix\n", "features = pd.concat([X_counter,X_normal], ignore_index=False, axis=1)\n", "features.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And now we have our list of features. It doesn't look anything like our original dataset, now does it? That's because our sentences have been transformed into numbers to feed into our clustering algorithms and predictive models.\n", "\n", "# Clustering\n", "\n", "Now it's finally time for some unsupervised machine learning. Each article has been binarized to 1s and 0s, and it's time to determine if we can determine if each publisher has a different method for publication." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmsAAAFNCAYAAABfUShSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4VHXaxvHvA5HFgoKCBVDAuqAiaMAOIopYsSKoa1vXsqsoi6Io9s5asLuuBduqKBaaooshIrYEEFdwQcRCNyiKioLI8/7xO/NmiGmETM6Z5P5c11yZOXNm5plD3Nz7q+buiIiIiEgy1Yu7ABEREREpm8KaiIiISIIprImIiIgkmMKaiIiISIIprImIiIgkmMKaiIiISIIprIlUMzO7xsyeqoHPaW1mbmY50eMJZnZWpj+3JlTndzGzYWZ2QxVe52a2fXXUUMb7729mMzP1/qV8Xka/T1WZ2eVm9nCG3vsLMzuojOeq9HshEgeFNZG1ZGY/pt1Wm9nPaY9PrubPGmZmK0t85rTq/IyqSguLU0ocbxrV/EUl36dGwm3SuPtEd98pE++d1OBuZgeY2bz0Y+5+k7snrlaRJFFYE1lL7r5R6gZ8BRyZduzpDHzkkPTPdPfdMvAZ62JDM9sl7fFJwOdxFSMiUtsorIlkRgMze8LMfjCz6WaWm3rCzJqb2QgzKzKzz82sXzV+7nZm9oGZfW9mr5jZpmmfe1RUy3dRy0vb6PgZZjYq7bzZZjY87fFcM+tQzmc+CZyW9vhU4In0E8r6zmbWE7gcOLGUVsNWZjYpuoavm1nTir5L9FxHM5sSve45oGFZhZvZ9maWH12vJdH56Q4ys0/NbKmZ3WdmFr2unpkNNrMvzezr6N96k+i5x81sQHS/RdT6+Ne0z/vWgjVamaIuu4vN7KOonufMrGHa8wPNbKGZLTCzs8rq1jSzG4H9gXuja3pvRd8net2ZZvZJ9Nw4M2tVznUr7/p/YWaDzGxG9F6PmVlDM9sQeBVobsWtxM3TW1atuLX2jOj3bqmZnWtmnaLr8l369zGz7czsTTP7Jvr3e9rMGpdVdznfp5GZ5ZnZ3enXRCQpFNZEMuMo4FmgMTASuBfCH3lgFDANaAF0By4ys0Oq6XNPBc4EmgOrgLujz90ReAa4CGgGjAVGmVkDIB/YPwogWwHrAftGr9sW2Aj4qJzPfAroY2b1oz/ajYD3U0+W953d/TXgJuC5UloNTwLOADYHGgAXV/Rdou/zMiFAbgo8DxxXTu3XA68DTYCWwD0lnj8C6ATsBvQGUv9Op0e3bkDqGqVCRD5wQHS/KzAn+gnQBZjoZe/z1xvoCbQB2kefkQq1fwcOArZPe7/fcfcrgInA+dE1Pb+i72NmRxNC87GEazqRcI1/p4LfpZSTo/feDtgRGOzuPwGHAgvSWokXlPE19gR2AE4EhgJXRN99Z6C3maW+vwE3E37f2wJbA9eUdW3K+D6bAeOBSe7er5x/G5HYKKyJZMbb7j7W3X8jBIdUCOkENHP369x9pbvPAf4F9CnnvS6OWhRSt8fLOfdJd/84+sN4JeEPW33CH70x7v6Gu/8K3AasD+wT1fAD0IEQAsYB883sj9Hjie6+upzPnAfMJPwxPY0SrWpV/M4Aj7n7LHf/GRge1Ud53wXYixA2h7r7r+7+AlBQzmf8CrQCmrv7L+7+donnb3H379z9KyAvrYaTgTvcfY67/wgMIgTWHNLCLyGcDSEKv4TrmV9OPXe7+wJ3/5YQcFOf1zu6HtPdfTlwbTnvUZ6yvs85wM3u/om7ryIE6A5ltK6Vd/1T7nX3udH3uBHou5Z1Xh/9e7wO/AQ84+5fu/t8QpDsCODus6M6Vrh7EXAH5QTZUjQn/Hs87+6D17JGkRqjsCaSGYvS7i8HGkZ/yFsRuoH+P3wRWjS2KOe9bnP3xmm308o5d27a/S8JwaUp4Y/Sl6knovA1l9DSBcWtQV2i+xMIf/QqChcpTxBagfoSWtrSVeU7w++v4UbR/fK+S3NgfonWkS8p20BC68wHUbfemVWpIbqfA2zh7p8BPxKC0P7AaGCBme1ExdezvM9L/7dNv782ynr/VsBdaf8+3xKuSwt+r6LfpZL1fRm9Zm0sTrv/cymPNwIws83N7Fkzm29mywi/e02pvMMJQfPBtaxPpEYprInUrLnA5yXCVyN3P6ya3n/rtPvbEFqOlgALCH+QAYjG5WwNzI8OpcLa/tH9fNYurI0g/OGb4+4lw1FF33ltu53K+y4LgRYlxh1tU9Ybufsid/+LuzcntC7dX9o4sIpqiD5jFcWhIh84HmgQtQblE7qomwAfVuL9S1pI6KZN2bqsEyNre03nAueU+Dda393fKeXcin6XSta3TfSaqtRVkZuj92zv7hsDpxBCZmX9C3gNGBuNqRNJJIU1kZr1AbDMzC41s/WjcV67mFmnanr/U8ysnZltAFwHvBB1xQ4HDjez7ma2HjAAWAGk/hjnE8Zfre/u8whdTT2BzYCpFX1o1O16IFDaEgwVfefFQOuo27Ayyvsu7xJCUz8zyzGzY4HOZb2RmZ1gZqkQtJTwh/+3StTwDNDfzNqY2UYUj7tbFT2fD5wPvBU9ngBcQOger8z7lzQcOMPM2kb/tldVcP5iwli6ynoQGGRmOwOY2SZmdkI5tZT3uwTwNzNraWGCy+VAauLGYmAziyZjVINGhFbM78ysBXBJFd7jfEI3/mgzW7+a6hKpVgprIjUo+kN9JKGL7HNCq9fDQHl/vAbamuusLSnn3CeBYYTuroZAv+hzZxJaHe6JPvNIwpIjK6PnZxH+6E2MHi8jDIyfVNlw4e6FURfg2n7n56Of31iJNdvK+Jwyv0v0fY4ldMkuJYyverGct+sEvG9mPxImglzo7pVZduRRwrV+K/pOvxDCWEo+IUikwtrbwAZpj9eKu79KmCySB8wmhFIIIak0dwHHR7Mp767E+78E3Ao8G3UnfkyYDFDaueX+LkX+TZi4MSe63RC99n+EoDsn6nJd2+7Rkq4Fdge+B8ZQ/r91qaIu87MJrYuvWNoMXJGkME18ERHJLtGs24+BP6S15iWChcWQz3L3/8Rdi0htoZY1EZEsYGbHRMuTNCG0go1KWlATkcxQWBMRyQ7nAEXAZ4RxdefFW46I1BR1g4qIiIgkmFrWRERERBJMYU1EREQkwXLiLqC6NG3a1Fu3bh13GSIiIiIVmjx58hJ3b1aZc2tNWGvdujWFhYVxlyEiIiJSITMrbyu8NagbVERERCTBFNZEREREEkxhTURERCTBFNZEREREEkxhTURERCTBFNZEREREEkxhTURERCTBFNYqYcgQyMtb81heXjguIiIikkkKa5XQqRP07l0c2PLywuNOneKtS0RERGq/WrODQSZ16wbPPQe9ekHfvvDiizB8eDguIiIikklqWauk3Fz47Td46CE491wFNREREakZCmuVNHky1K8f7g8d+vsxbCIiIiKZoLBWCakxai+9BPvuC/XqwQknKLCJiIhI5imsVUJBQRij1r073H8//PQT7LNPOC4iIiKSSQprlTBwYPEYtfbt4YILYPRojVsTERGRzFNYq4JrroEttoC//jVMOhARERHJFIW1KthkE7j9digshIcfjrsaERERqc0U1qqob9/QDTpoEBQVxV2NiIiI1FYKa1VkBvfeCz/8AJddFnc1IiIiUlsprK2Ddu2gf3949FF45524qxEREZHaSGFtHV11FbRsCX/7G6xaFXc1IiIiUtsorK2jjTaCO++EDz+EBx6IuxoRERGpbRTWqsFxx0GPHjB4MCxaFHc1IiIiUpsorFUDM7jnHvjlF7jkkrirERERkdpEYa2a7LhjCGpPPQVvvRV3NSIiIlJbKKxVo8svh1atws4Gv/4adzUiIiJSGyisVaMNNoC774bp08NPERERkXWV0bBmZj3NbKaZzTaz3y0da2atzGy8mX1kZhPMrGV0vIOZvWtm06PnTsxkndXpyCPh8MPD/qHz58ddjYiIiGS7jIU1M6sP3AccCrQD+ppZuxKn3QY84e7tgeuAm6Pjy4FT3X1noCcw1MwaZ6rW6mQWWtVWrYIBA+KuRkRERLJdJlvWOgOz3X2Ou68EngV6lTinHTA+up+Xet7dZ7n7p9H9BcDXQLMM1lqttt027Bn63HMwfnzF54uIiIiUJZNhrQUwN+3xvOhYumnAcdH9Y4BGZrZZ+glm1hloAHyWoTozYuBA2G67sLPBihVxVyMiIiLZKpNhzUo55iUeXwx0NbOpQFdgPvD/mzaZ2VbAk8AZ7r76dx9gdraZFZpZYVFRUfVVXg0aNgxrr82cCXfcEXc1IiIikq0yGdbmAVunPW4JLEg/wd0XuPux7t4RuCI69j2AmW0MjAEGu/t7pX2Auz/k7rnuntusWfJ6SQ89FI45Bq6/Hr76Ku5qREREJBtlMqwVADuYWRszawD0AUamn2BmTc0sVcMg4NHoeAPgJcLkg+czWGPGDR0afl50Ubx1iIiISHbKWFhz91XA+cA44BNguLtPN7PrzOyo6LQDgJlmNgvYArgxOt4b6AKcbmYfRrcOmao1k7bZBq68El56CV59Ne5qREREJNuYe8lhZNkpNzfXCwsL4y6jVCtXQvv2YTmPjz8O49lERESk7jKzye6eW5lztYNBDWjQAO67Dz77DIYMibsaERERySYKazWke3c48US4+WaYMyfuakRERCRbKKzVoNtvh5wc6NcPaknvs4iIiGSYwloNatEi7Bk6ZgyMHFnh6SIiIiIKazWtXz/YeWe48EJYvjzuakRERCTpFNZq2Hrrwf33w5dfwk03xV2NiIiIJJ3CWgy6dIFTToF//ANmzYq7GhEREUkyhbWY/OMfYb2188/XZAMREREpm8JaTLbcEm64Ad54A154Ie5qREREJKkU1mJ03nnQoQP07w8//hh3NSIiIpJECmsxyskJkw3mz4frrou7GhEREUkihbWY7b03nHkm3HknTJ8edzUiIiKSNAprCXDLLdCoEfztb5psICIiImtSWEuAZs3Cmmv5+fDMM3FXIyIiIkmisJYQf/kL5ObCgAHw/fdxVyMiIiJJobCWEPXrwwMPwOLFcPXVcVcjIiIiSaGwliC5uXDOOXDPPTBtWtzViIiISBIorCXMjTfCppuGyQarV8ddjYiIiMRNYS1hNt0Ubr0VJk2CJ56IuxoRERGJm8JaAp1+elh/beBAWLo07mpEREQkTgprCVSvXtjZ4Jtv4Ior4q5GRERE4qSwllAdOoRxaw8+CJMnx12NiIiIxEVhLcGuvx423xz++ldNNhAREamrFNYSbJNN4Lbb4IMP4OGH465GRERE4qCwlnAnnwxdusCgQbBkSdzViIiISE1TWEs4M7jvvrAF1aBBcVcjIiIiNU1hLQvssgtcdFHoCn3vvbirERERkZqksJYlrr4amjcPkw1++y3uakRERKSmKKxliUaN4I47YOrUsOG7iIiI1A0Ka1mkd2846CAYPBgWL467GhEREakJCmtZxAzuvReWLw9bUYmIiEjtp7CWZXbaCS6+OGzyPnFi3NWIiIhIpimsZaErroBttgmTDX79Ne5qREREJJMU1rLQhhvC0KHw8cehW1RERERqL4W1LHX00XDooWFJjwUL4q5GREREMkVhLUuZwT33wMqVMGBA3NWIiIhIpiisZbHttoP99oNnn4Xx44uP5+XBkCHx1SUiIiLVJ6Nhzcx6mtlMM5ttZpeV8nwrMxtvZh+Z2QQza5n23Glm9ml0Oy2TdWaziy+GevXgjDNCK1teXliPrVOnuCsTERGR6pCxsGZm9YH7gEOBdkBfM2tX4rTbgCfcvT1wHXBz9NpNgauBPYHOwNVm1iRTtWaznj3hhhtg7lzYZ58Q1IYPh27d4q5MREREqkMmW9Y6A7PdfY67rwSeBXqVOKcdkOrAy0t7/hDgDXf/1t2XAm8APTNYa1YbNAj23BMmT4aGDUP3qIiIiNQOmQxrLYC5aY/nRcfSTQOOi+4fAzQys80q+VrM7GwzKzSzwqKiomorPNvk5cFnn8Gxx8L8+bDzzjBqVNxViYiISHXIZFizUo55iccXA13NbCrQFZgPrKrka3H3h9w9191zmzVrtq71ZqXUGLXhw2HEiLCzwS+/wFFHhVmiK1fGXaGIiIisi0yGtXnA1mmPWwJrrAjm7gvc/Vh37whcER37vjKvlaCgYM0xaqecAqNHh/Frd9wB++8Pn38eb40iIiJSdZkMawXADmbWxswaAH2AkeknmFlTM0vVMAh4NLo/DuhhZk2iiQU9omNSwsCBv59McMghMGkSvPACzJwJHTvCSy/FU5+IiIism4yFNXdfBZxPCFmfAMPdfbqZXWdmR0WnHQDMNLNZwBbAjdFrvwWuJwS+AuC66JisheOOgylTYMcdw3i2fv1gxYq4qxIREZG1Ye6/GwqWlXJzc72wsDDuMhJp5Uq49NKwn+gee8Bzz2nGqIiISJzMbLK751bmXO1gUAc0aAB33gkvvxxmjXbsGMa5iYiISPIprNUhvXrBhx+GpT1OPBHOOy/MHBUREZHkUlirY1q1grfegksugQcfhL32glmz4q5KREREyqKwVgett17Y6H30aJg3L4xj+/e/465KRERESqOwVocdfnjoFu3QAU4+Gf7yF1i+PO6qREREJJ3CWh3XsmXYBWHQIHj44bDH6CefxF2ViIiIpCisCTk5cNNN8NprsHgx5ObC44/HXZWIiIiAwpqkOeSQ0C3auTOcfnq4/fRT3FWJiIjUbQprsobmzeE//4GrrgqbwnfqBB9/HHdVIiIidZfCmvxO/fpw7bUhtH37bQhsjzwCtWSzCxERkayisCZlOvDA0C26775w1lnwpz/BDz/EXZWIiEjdorAm5dpySxg3Dq6/Hp55Jkw+mDYt7qpERETqDoU1qVD9+jB4MLz5Jvz4Y1je48EH1S0qIiJSExTWpNK6dg3dot26hX1F+/SBZcvirkpERKR2U1iTtdKsGYwZA7fcAiNGwO67w+TJcVclIiJSeymsyVqrVw8uvRTy82HFCthnH7j3XnWLioiIZILCmlTZvvuGbtEePeCCC+D44+G77+KuSkREpHZRWJN1stlmMHIk3HZb+NmxI3zwQdxViYiI1B4Ka7LOzGDAAJg4MXSF7rcf3HmnukVFRESqg8KaVJu99oKpU+Hww+Hvf4edd4ZXXlnznLw8GDIknvpERESykcKaVKsmTeDFF+Guu2DWLDj22DD5AEJQ6907bF8lIiIilaOwJtXODPr1g/fegy22CJMPuncPQW348LBOm4iIiFSOwppkTG4ufPIJtG0bdj/YaCPYcce4qxIREckuCmuSUVOmQFERHHkkfPFFCG4vvxx3VSIiItlDYU0yJjVGbfjwsKzH44/Dzz/DMceE7aqWL4+7QhERkeRTWJOMKShYc4zaqaeGraq6dAkbwefmwrRp8dYoIiKSdAprkjEDB/5+MkGPHmGbqjfeCLsddO4MQ4fC6tXx1CgiIpJ0CmsSi4MOgo8+gp49oX//sDbb4sVxVyUiIpI8CmsSm6ZNw2SD+++HCROgfXt49dW4qxIREUkWhTWJlVmYbFBYGNZkO+wwuOgi+OWXuCsTERFJhkqHNTOrb2bNzWyb1C2ThUndsvPOYQP4fv3C7gd77gkzZsRdlYiISPwqFdbM7AJgMfAGMCa6jc5gXVIHNWwYgtqYMbBwIeyxBzzwgDaEFxGRuq2yLWsXAju5+87uvmt0a5/JwqTuOuywMPmga1f461/h6KNhyZK4qxIREYlHZcPaXOD7TBYikm7LLWHsWLjzTnjttTD5YPz4uKsSERGpeZUNa3OACWY2yMz+nrplsjCRevXCZIP334dNNoGDD4ZLL4WVK+OuTEREpOZUNqx9RRiv1gBolHYTybgOHWDyZDj7bBgyBPbZB2bNirsqERGRmpFTmZPc/VoAM2sUHvqPlXmdmfUE7gLqAw+7+y0lnt8GeBxoHJ1zmbuPNbP1gIeB3aMan3D3myv3laQ22mCDsEXVIYfAWWfB7rvD3XfDGWeE5T9ERERqq8rOBt3FzKYCHwPTzWyyme1cwWvqA/cBhwLtgL5m1q7EaYOB4e7eEegD3B8dPwH4g7vvCuwBnGNmrSv3laQ2O+aYsJ9o587w5z9Dnz6wdGncVYmIiGROZbtBHwL+7u6t3L0VMAD4VwWv6QzMdvc57r4SeBboVeIcBzaO7m8CLEg7vqGZ5QDrAyuBZZWsVWq5li3D3qI33wwvvhi6SSdOjLsqERGRzKhsWNvQ3fNSD9x9ArBhBa9pQZhFmjIvOpbuGuAUM5sHjAUuiI6/APwELCSMl7vN3b+tZK1SB9SvD5ddBpMmwXrrwQEHwFVXwapVcVcmIiJSvSo9G9TMrjSz1tFtMPB5Ba8pbSRRyeVN+wLD3L0lcBjwpJnVI7TK/QY0B9oAA8xs2999gNnZZlZoZoVFRUWV/CpSm3TuDFOnwp/+BNdfD126wOcV/WaKiIhkkcqGtTOBZsCLwEvR/TMqeM08YOu0xy0p7uZM+TMwHMDd3wUaAk2Bk4DX3P1Xd/8amATklvwAd3/I3XPdPbdZs2aV/CpS2zRqBMOGwTPPwPTpoVv03/+OuyoREZHqUamw5u5L3b2fu+/u7h3d/UJ3r2hYdwGwg5m1MbMGhAkEI0uc8xXQHcDM2hLCWlF0/EALNgT2Av5X+a8ldVGfPmHywS67wMknw6mnwjKNdBQRkSxXblgzs6HRz1FmNrLkrbzXuvsq4HxgHPAJYdbndDO7zsyOik4bAPzFzKYBzwCnu7sTZpFuRJh9WgA85u4frcP3lDqidWvIz4drroGnn4aOHcOiuiIiItnKvJxdss1sD3efbGZdS3ve3fMzVtlays3N9cLCwrjLkASZNCm0sM2bB9deGyYk1K8fd1UiIiJgZpPd/XdDvEpTbsuau0+O7nZw9/z0G9BhXQsVyaR994UPP4Tjj4fBg+HAA2Hu3IpfJyIikiSVnWBwWinHTq/GOkQyonHjMPFg2LCwZdVuu8ELL8RdlYiISOVVNGatr5mNArYtMV4tD/imZkoUWTdmcNppoZVtu+3ghBPCkh9jx655Xl5e2HtUREQkSSraG/QdwsK0TYHb047/AGjAv2SV7bcP49iuvhpuuQWOOgruuw/OOScEtd69YfjwuKsUERFZU7kTDOD/9/gc5+4H1UxJVaMJBrI2UuFsyZIwtu1//4Pnn4du3eKuTERE6oJqm2AA4O6/AcvNbJN1rkwkIbp1CwFt551Da9v338OoUTB/ftyViYiIrKmyEwx+Af5rZo+Y2d2pWyYLE8m0jz6CxYvh3HPDkh533QVt2sDZZ8Ps2XFXJyIiElQ2rI0BrgTeAian3USyUvoYtQcegFdfDTNHDz0UnngCdtoJTjoJ/vvfuCsVEZG6rrLbTT1O2GEgFdL+HR0TyUoFBSGopcaodesWlvTYd9+wEfyAAaFbtH37MBHhvffirVdEROquCicYAJjZAcDjwBeAETZoP83d38pkcWtDEwykun37LdxzT+geXbo0BLorrgiL65rFXZ2IiGSzap1gELkd6OHuXd29C3AIcGdVCxTJBptuGpb5+PJLuO22MCHhoINgr73glVdg9eq4KxQRkbqgsmFtPXefmXrg7rOA9TJTkkiyNGoUukXnzIEHH4SiIjj66NBF+vTTsGpV3BWKiEhtVtmwVhjNBD0guv0LTTCQOqZhw7CA7qxZ8NRT4dgpp8COO8I//wm//BJvfSIiUjtVNqydB0wH+gEXAjOAczJVlEiS5eTAySeHpT9efhmaNg3Lf2y7Ldx+O/z4Y9wViohIbVLZsHauu9/h7se6+zHufichwInUWfXqQa9e8P778J//QNu2cPHF0KoVXHttmKAgIiKyriob1k4r5djp1ViHSNYyg+7dYfx4ePdd2G8/uOaaENouuQQWLoy7QhERyWblhjUz62tmo4A2ZjYy7TYB+KZGKhTJIqmZoh99FNZnu+OOsCvCeeeF9dtERETWVrnrrJlZK6ANcDNwWdpTPwAfuXti5sFpnTVJotmzYcgQGDYsLPXRty9cdlnYk1REROqualtnzd2/dPcJwEHARHfPBxYCLQmL44pIObbfHh56KLSq9esHL74Iu+wCxx4bdlEQERGpSGXHrL0FNDSzFsB44AxgWKaKEqltWrQIXaJffglXXhn2Ju3cGXr0gAkToBIbiYiISB1V2bBm7r4cOBa4x92PAdplriyR2qlpU7juuhDabr01jG3r1i3sSTp6tEKbiIj8XqXDmpntDZwMjImO5WSmJJHab+ONYeDA0D16332wYAEceSR06AAnnRSWAkmXlxfGvomISN1T2bB2ETAIeMndp5vZtkBe5soSqRvWXx/++lf49FN4/HFYuRKeeQYOOSSs2bZiRQhqvXtDp05xVysiInEodzZoNtFsUKkNVq8OuyJcdlkIcOuvH7pG774b/vKXuKsTEZHqUm2zQc1saPRzVIl11kaa2cjqKFZEitWrF2aKzpwZtrT6+efQ2nb22dCxYwhtS5bEXaWIiNSkirpBn4x+3gbcXspNRDJgwgQYNy7MHG3SBC64IAS5Cy+E5s3h+ONhzBhYlZiVDkVEJFPKnSTg7pOjn/lm1iy6X1QThYnUVakxasOHh5mi3boVP95sM3jsMXjqKRgxArbaCv70JzjjDPjjH+OuXEREMqGiblAzs2vMbAnwP2CWmRWZ2VU1U55I3VNQUBzUIPwcPjwcb98e7rwT5s8PC+zm5sLtt4dN5PfeOyzA+/338dYvIiLVq6LtpvoDhwFnu/vn0bFtgQeA19z9zhqpshI0wUDqqkWLQkvbY4/BjBlhUsKxx4bWtm7dQvepiIgky9pMMKgorE0FDnb3JSWONwNed/eO61RpNVJYk7rOHQoLQ2h75hn47jto1QpOOw1OPz1sKC8iIslQbbNBgfVKBjX4/3Fr61WlOBHJDLOwFtv998PChSGw7bQTXH89bLttaGV74gn46ae4KxURkbVRUVhbWcXnRCRGDRtCnz5hRumXX8INN8C8eaGVbaut4KyzYNIkbW8lIpINKuoG/Q0o7f+HG9DQ3RNoL5g6AAAY7ElEQVTTuqZuUJHyucPbb4du0uHDQwvbjjuGLtJTTw2bzYuISM2otm5Qd6/v7huXcmuUpKAmIhUzg/33h0cfDZMSHnsMttwSLr8cttkGevaE556DX36Ju1IREUmneWIiddBGG4UWtfx8mD07BLYZM0LXafPmcP75MHmyuklFRJJAYU2kjttuuzAJ4fPP4fXX4dBD4ZFHwhpuu+0W1nX7+utw7pAhYdHedHl54biIiGRGRsOamfU0s5lmNtvMLivl+W3MLM/MpprZR2Z2WNpz7c3sXTObbmb/NbOGmaxVpK6rXx8OPhiefjrMJn3wQdhgA/j738N4tqOPDvuU9u5dHNhSuy106hRv7SIitVm5EwzW6Y3N6gOzgIOBeUAB0NfdZ6Sd8xAw1d0fMLN2wFh3b21mOcAU4E/uPs3MNgO+c/ffyvo8TTAQyYwZM2DYMHjyyTDWrXFjWLEiTEoYMWLN3RZERKRyqnOdtXXRGZjt7nPcfSXwLNCrxDkObBzd3wRYEN3vAXzk7tMA3P2b8oKaiGROu3ahm3PuXBg1Cg48MIS1f/4TcnJg2rTiblIREal+mQxrLYC5aY/nRcfSXQOcYmbzgLHABdHxHQE3s3FmNsXMBmawThGphJwcOOKIMPmgSZMwe3TJEujfP0xKOPJIeP55zSYVEalumQxrVsqxkn2ufYFh7t6SsAfpk2ZWD8gB9gNOjn4eY2bdf/cBZmebWaGZFRYVFVVv9SLyO6kxas8/D6++GiYkNGkSjk2dGn5utRWcey68845mk4qIVIdMhrV5wNZpj1tS3M2Z8mdgOIC7vws0BJpGr8139yXuvpzQ6rZ7yQ9w94fcPdfdc5s1a5aBryAi6QoK1hyj1q1bGLfWoUPYKeGNN0IL25NPwr77hkV3r78evvgi1rJFRLJaJicY5BAmGHQH5hMmGJzk7tPTznkVeM7dh5lZW2A8oau0cXR/P8K2Vq8Bd7r7mLI+TxMMRJLjhx/gxRfDXqR5eaGFrUuXsN3V8cfDxhtX/B4iIrVZIiYYuPsq4HxgHPAJMNzdp5vZdWZ2VHTaAOAvZjYNeAY43YOlwB2EgPchMKW8oCYiydKoUQhm48eHVrUbbwwzSf/8Z9hiCzjppLBv6W+aNiQiUqGMtazVNLWsiSSbO3zwQWhte+YZWLo0jG87+eQQ7HbZJe4KRURqTiJa1kRE0pnBnnvCffeFRXdHjIDOnWHoUNh1V9h993B/8eK4KxURSRaFNRGpcX/4Axx7LLz8MixYAHffHXZQ6N8/7JagZUBERIoprIlIrJo1gwsuCDNNp0+Hiy/WMiAiIukU1kQkMdq1g1tuCcuAvP56WIRXy4CISF2nsCYiiZPaVD61H+mwYbD11nDVVdCmDXTtCo8+CsuWxV2piEjmKayJSKKllgF5883ylwG55Zawplu6vLywr6mISDZTWBORrNGqFVx+Ofzvf/Dee3DmmfDaa2Gf0n/8Aw4/PLS4QfHWWJ06xVuziMi60jprIpLVVqyAMWPC+m2jRsHq1WHSwo8/wm23hQkK9fR/S0UkYbTOmojUGenLgCxaBIccAkVFYdmPv/0tLAVy9tkwejT8/HPc1YqIrD2FNRGpNT7+GCZPhiuvhE03DV2mXbrAs8+GtduaNoVjjoHHHoOvv467WhGRysmJuwARkeqQGqM2fDh06xZuqcdPPAH5+TByZLi9/HLYUWGffeCoo8Ltj3+M+xuIiJROLWsiUisUFBQHNQg/hw8Px//wB+jRA+69N6zhNmUKXH116Ba99FJo2xZ22gkuuQQmTtQG8yKSLJpgICJ12ty5YWLCyJFheZBff4XNNgsL8h51VAh5G20Ud5UiUtuszQQDhTURkciyZWHNtpEjwwzTpUuhQQPo3h169Qrj3po3j7tKEakNFNZERNbRqlXw9tshuL3yCsyZE47n5oYWt169YNddw9g3EZG1pbAmIlKN3OGTT0JoGzkS3n8/HGvVqji4dekC660Xd6Uiki0U1kREMmjRorBu28iR8MYbYU23TTaBQw8Nwa1nT2jcOO4qRSTJFNZERGrI8uUhsI0cGSYqFBVBTk7YbD61LMjw4WHbq9RMVQhLjRQUwMCB8dUuIvFRWBMRicFvv8EHHxR3l37ySTi+7bahNe7228NuCvn5a64JJyJ1j8KaiEgCfPpp8UK8EyeGcW6NGoXlQW6+GS64AOrXj7tKEYmD9gYVEUmAHXaAAQNCS1pRURjP9sMPIaz17x+WATn33NCN+uuvcVcrIkmlsCYiUgM++ggmTQr7ljZuHH4ecAA89VRYeHfLLeHMM8P6bitWxF2tiCSJwpqISIal71t63XXw/PPwwAOhVa2oCF56CQ47DF58MeycsPnmcPLJ4fHy5XFXLyJxU1gTEcmw8vYtXX99OPpoePJJ+PprGDsWjj8+7KRw3HHQrBmccAI8+2zoQhWRukcTDEREEmjVqjDWbcSI0PK2aFHxhvTHHReWBGnSJO4qRaSqNBtURKQW+e03ePddeOGF0DU6d25Yy6179xDcjj46tMCJSPZQWBMRqaXcQ/fpiBHh9tlnUK9e2O7quOPg2GO12bxINlBYExGpA9zDLNMXXgjBLbUI7z77FAe31q1jLVFEyqCwJiJSB33ySXGL24cfhmN77BGC23HHwY47xlufiBTTorgiInVQ27YweDBMnQqzZ8Ott4YdEi6/HHbaCdq3h2uvhY8/Dq1yAEOGhKVF0uXlheMikgwKayIitdB224VN4t9/H776CoYODYvxXnst7Lor/PGPIcRtsklYAy4V2FJrwnXqFG/9IlJM3aAiInXIokVhKZARI2DChDDTdIstYNkyOPFEGD1aG8yL1AR1g4qISKm23BLOOw/+858Q3B55BHbfPWxxNWwY/PQTPPooPPccfPdd3NWKCCisiYjUWU2bhv1IL7kkLLB73HGwejWMHAl9+oTnu3WD22+HmTOLx7mJSM1SWBMRqcNSY9Sefz4sAfLqq9CgAdx9dxjz9s03cPHFYYzbjjtC//4wfjysXBl35SJ1h8KaiEgdVta+pT//DDfdFNZx++ILuO8+2GGHsAH9QQeFVrcTToDHHw97mopI5mR0goGZ9QTuAuoDD7v7LSWe3wZ4HGgcnXOZu48t8fwM4Bp3v628z9IEAxGRzPvpp9CyNnp0uC1cCGaw555wxBHh1r59OCYiZUvEorhmVh+YBRwMzAMKgL7uPiPtnIeAqe7+gJm1A8a6e+u050cAq4H3FdZERJLFPazplgpuBQXheMuWxcHtwANh/fXjrVMkiZIyG7QzMNvd57j7SuBZoFeJcxzYOLq/CbAg9YSZHQ3MAaZnsEYREakiszCT9Kqr4IMPQivbI4+ENdqefDKEtc02gyOPhH/+E+bNi7tikeyUk8H3bgHMTXs8D9izxDnXAK+b2QXAhsBBAGa2IXApoVXu4gzWKCIi1WTLLcPs0jPPDEuB5OfDmDEwalRoeQPo0KG41a1Tp7AJvYiUL5P/mZQ2YqFkn2tfYJi7twQOA540s3rAtcCd7v5juR9gdraZFZpZYVFRUbUULSIi6+4Pf4AePeCuu+Czz2DGjLCF1cYbh4kLe+0FW20FZ5wRFuhdtizuikWSK5Nj1vYmTAw4JHo8CMDdb047ZzrQ093nRo/nAHsBI4Cto9MaE8atXeXu95b1eRqzJiKSHb79FsaNC61tr74KS5fCeutB167FrW7bbRfCXadOa+6mkJcXxsYNHBhf/SLVISlj1gqAHcysjZk1APoAI0uc8xXQHcDM2gINgSJ339/dW0eTDYYCN5UX1EREJHtsuin07QtPPx2W/XjrrbB+24IFcNFFsP32YVP6yZPhmGPgjTfC67RvqdRVGRuz5u6rzOx8YBxhWY5H3X26mV0HFLr7SGAA8C8z60/oIj3da8tmpSIiUqGcHNh//3C79VaYMyeMcxs9Gl5+OSy+e8gh0K5d2JD+6ae1b6nUPdrIXUREEumHH8IeptdeC9OmhWM5OaG79Kijwq1161hLFKmypHSDioiIVFmjRtC4McyfD1dcEe6fcEJYIuTCC6FNm7AA75VXhnFsq1fHXbFIZiisiYhIIqXGqA0fDjfcAC++GMav3XsvfPop3HFHWMft5puhc+ewGO8554Ru1J9/jrt6keqjsCYiIolU1r6lBQVhEkL//iHQLV4cFuHdbz/497/DbNKmTcPkhMceA63sJNlOY9ZERKTWSC3GO3JkuM2dG3Za2Gef4nFuO+2kvUslfonYG7SmKayJiEg6d/jww+LgNmVKOL7DDsXBbZ99wqQFkZqmsCYiIlLC3LlhSZBXXoE334Rffw1j3g4/PAS3Hj3CpAaRmqCwJiIiUo5ly+D110OL25gxYVeFBg3gwANDcDvyyDBhQSRTFNZEREQqadUqeOed0OL2yithL1OAPfYo7i7dbTeNc5PqpXXWREREKiknB7p0gdtvD0uCzJgBt9wSNqO/5hro2BFatYLzzw+tcStXhn1L8/LWfJ+8vHBcpLoprImIiETMwr6kl14KkybBokXwyCOw++7w6KNh66umTWHsWOjVK7TEgfYtlcxSN6iIiEgl/PwzjB8fxrmNGhWCHMDWW8M338CNN8J554UWOZGKaMyaiIhIBq1eDYWF8Pe/hxa4lIYNw+K83bqFyQq5uVoaREq3NmFNv0IiIiJrqV49+OknmDkz7E16//1w0UWwZEnoEr3iinBeo0ZhPFwqvO22W3ityNpQWBMREVlL6fuWdusWbqnHQ4eGLa4mTAjrueXlheVBADbdFA44oDi8tW2rWaZSMXWDioiIrKUhQ8JkgtS+pRBCWUEBDBz4+/Pnzw/Pv/lmuH35ZTi+xRYhtKXC27bbKrzVFRqzJiIikmCff14c3PLyYOHCcHybbUJoSwU4LcxbeymsiYiIZAn3MPYtFd4mTAizSyHsY5oKbwccAJtvHmelUp0U1kRERLLU6tXw3/8Wh7e33grbYwHssktxeOvaFRo3jrdWqTqFNRERkVpi1SqYMqU4vL39dljzrV69sLtCKrzttx9stNHaj6eTeCisiYiI1FIrVsAHHxSHt/feC1tg5eRA585hksKoUfDcc2HHhZIzVyUZFNZERETqiOXLw0b0qfBWWAi//Raea9UqLCNy/fVhd4X114+3VimmsCYiIlJHLVsGEyeGgPb++8XHGzQILW9du4bb3nuHblOJx9qENa2jLCIiUotsvDFssAF89lnYXWGzzeCmm+DCC0N36S23QI8e0KQJ7LVX2LR+7NjiSQySPGpZExERqUVKjlEr+fiHH+DddyE/P9w++AB+/bV4wkKq5W2//cKOC5IZ6gYVERGpo9Z2Nujy5WGSQiq8vfdemMRgBrvuWhzeunSBZs1q7nvUdgprIiIiUiW//BJa2/Lzwxpv77wTAh1Au3bFwa1rV9hqq3hrzWYKayIiIlItVq6EyZOLW97efht+/DE8t8MOxS1vXbvC1lvHW2s2UVgTERGRjFi1CqZOLQ5vEyfC99+H59q0WbPlrU0bbUxfFoU1ERERqRG//Ra2x0qFt7feKt7btGXLNVveXnopLB+i3RUU1kRERCQmq1fDjBnF4S0/H77+OjzXpEkY/3buuXD22bBoEZx4Yt3cXUFhTURERBLBHWbODC1u+fnw+uuwZEl4ziy0uPXpE/Y33X77utNtqrAmIiIiieQO/frBvffCLruELtOFC8NzLVqEFrZu3UJ4a9061lIzam3CWk6mixERERFJmTABnn027K7wwANhw/kWLcLYtbw8GDcOnnoqnNu6dXF469YtjIGri9SyJiIiIjWiot0VILS8TZ9eHN4mTIClS8NzO+ywZnjbYovYvso6UzeoiIiIJM7a7q4AYcLCtGnF4S0/P2yZBWGR3lRwO+CAsA9qtlBYExERkVpp1SqYMqU4vE2cWLzDwm67FYe3Ll2gceN4ay2PwpqIiIjUCStXhpa5VHibNCnsbZramP7AA0N4228/aNQo7mqLrU1Yq5fhQnqa2Uwzm21ml5Xy/DZmlmdmU83sIzM7LDp+sJlNNrP/Rj8PzGSdIiIikp0aNIB994XBg2H8ePjuuxDaBg+GDTaAoUPhsMPCGm977w1XXAH/+U9xa1zKkCHhdeny8sLxuGWsZc3M6gOzgIOBeUAB0NfdZ6Sd8xAw1d0fMLN2wFh3b21mHYHF7r7AzHYBxrl7i/I+Ty1rIiIiUtLy5aG1LdXyVlAQdl1o0AD23LN4mZBffoFTTil/8kN1SsrSHZ2B2e4+JyrqWaAXMCPtHAc2ju5vAiwAcPepaedMBxqa2R/cfUUG6xUREZFaZoMN4OCDww3C5ISJE4vD2/XXw3XXQcOG0LYtHHFECGmjRydnZ4VMhrUWwNy0x/OAPUuccw3wupldAGwIHFTK+xxHaH1TUBMREZF10qhR6BY97LDweOnSsLtCKrwtXw7DhoV14JIQ1CCzY9ZK2zCiZJ9rX2CYu7cEDgOeNLP/r8nMdgZuBc4p9QPMzjazQjMrLCoqqqayRUREpK5o0gR69Qpj24YODct/nHVWWLC35Bi2uGQyrM0Dtk573JKomzPNn4HhAO7+LtAQaApgZi2Bl4BT3f2z0j7A3R9y91x3z23WrFk1ly8iIiJ1RWqM2vPPw7/+FbpAe/dORmDLZFgrAHYwszZm1gDoA4wscc5XQHcAM2tLCGtFZtYYGAMMcvdJGaxRREREhIKCNceodesWHhcUxFsXZHidtWgpjqFAfeBRd7/RzK4DCt19ZDQD9F/ARoQu0oHu/rqZDQYGAZ+mvV0Pd/+6rM/SbFARERHJFloUV0RERCTBErMoroiIiIisG4U1ERERkQRTWBMRERFJMIU1ERERkQRTWBMRERFJMIU1ERERkQRTWBMRERFJsFqzzpqZFQFfxl1HzJoCS+IuIgvpulWNrlvV6LpVja5b1ei6VU1NXLdW7l6pvTJrTVgTMLPCyi6wJ8V03apG161qdN2qRtetanTdqiZp103doCIiIiIJprAmIiIikmAKa7XLQ3EXkKV03apG161qdN2qRtetanTdqiZR101j1kREREQSTC1rIiIiIgmmsJblzGxrM8szs0/MbLqZXRh3TdnEzOqb2VQzGx13LdnCzBqb2Qtm9r/o927vuGvKBmbWP/pv9GMze8bMGsZdU1KZ2aNm9rWZfZx2bFMze8PMPo1+NomzxiQq47r9I/pv9SMze8nMGsdZYxKVdt3SnrvYzNzMmsZRW4rCWvZbBQxw97bAXsDfzKxdzDVlkwuBT+IuIsvcBbzm7n8EdkPXr0Jm1gLoB+S6+y5AfaBPvFUl2jCgZ4ljlwHj3X0HYHz0WNY0jN9ftzeAXdy9PTALGFTTRWWBYfz+umFmWwMHA1/VdEElKaxlOXdf6O5Tovs/EP5wtoi3quxgZi2Bw4GH464lW5jZxkAX4BEAd1/p7t/FW1XWyAHWN7McYANgQcz1JJa7vwV8W+JwL+Dx6P7jwNE1WlQWKO26ufvr7r4qevge0LLGC0u4Mn7fAO4EBgKxD+5XWKtFzKw10BF4P95KssZQwn+Iq+MuJItsCxQBj0Xdxw+b2YZxF5V07j4fuI3w/9AXAt+7++vxVpV1tnD3hRD+Tyqwecz1ZKMzgVfjLiIbmNlRwHx3nxZ3LaCwVmuY2UbACOAid18Wdz1JZ2ZHAF+7++S4a8kyOcDuwAPu3hH4CXVHVSgaX9ULaAM0BzY0s1PirUrqEjO7gjBs5um4a0k6M9sAuAK4Ku5aUhTWagEzW48Q1J529xfjridL7AscZWZfAM8CB5rZU/GWlBXmAfPcPdV6+wIhvEn5DgI+d/cid/8VeBHYJ+aass1iM9sKIPr5dcz1ZA0zOw04AjjZtV5XZWxH+D9W06K/ES2BKWa2ZVwFKaxlOTMzwvihT9z9jrjryRbuPsjdW7p7a8JA7zfdXS0dFXD3RcBcM9spOtQdmBFjSdniK2AvM9sg+m+2O5qYsbZGAqdF908DXomxlqxhZj2BS4Gj3H153PVkA3f/r7tv7u6to78R84Ddo//9i4XCWvbbF/gToWXow+h2WNxFSa12AfC0mX0EdABuirmexItaIl8ApgD/Jfxvb6JWSE8SM3sGeBfYyczmmdmfgVuAg83sU8IMvVvirDGJyrhu9wKNgDeivw8PxlpkApVx3RJFOxiIiIiIJJha1kREREQSTGFNREREJMEU1kREREQSTGFNREREJMEU1kREREQSTGFNRKQUZtbazD6Ouw4REYU1ERERkQRTWBMRqYCZbRttXN8p7lpEpO5RWBMRKUe0tdYI4Ax3L4i7HhGpe3LiLkBEJMGaEfagPM7dp8ddjIjUTWpZExEp2/fAXMIevCIisVDLmohI2VYCRwPjzOxHd/933AWJSN2jsCYiUg53/8nMjgDeMLOf3P2VuGsSkbrF3D3uGkRERESkDBqzJiIiIpJgCmsiIiIiCaawJiIiIpJgCmsiIiIiCaawJiIiIpJgCmsiIiIiCaawJiIiIpJgCmsiIiIiCfZ/1W7enmpbBoUAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Size of graph\n", "plt.rcParams['figure.figsize'] = [10,5]\n", "\n", "# k means determine k\n", "distortions = []\n", "K = range(1,15)\n", "for k in K:\n", " kmeanModel = KMeans(n_clusters=k).fit(features)\n", " kmeanModel.fit(features)\n", " distortions.append(sum(np.min(cdist(features, kmeanModel.cluster_centers_, 'euclidean'), axis=1)) / features.shape[0])\n", "\n", "# Plot the elbow\n", "plt.plot(K, distortions, 'bx-')\n", "plt.xlabel('k')\n", "plt.ylabel('Distortion')\n", "plt.title('The Elbow Method showing the optimal k')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It seems as though by using the elbow method, the ideal number for k (number of clusters) is 3 as that is when the distortion decreases drop off. That's relatively small compared to the 15 publications that we actually have, but it may be more useful to work with for prediction.\n", "\n", "### K-means\n", "\n", "The first clustering method I'll use for modelling the dataset is K-means, that requires the user to input k number of centroids, determining the nearest centroid for each data point, and adjusting the centroids until the best clusters are found, or until a set number of iterations has passed. However, we want to see if we can cluster the articles into 15 clusters representing each of the publishers, so that will be k." ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
col_0012
publication
Atlantic28103287
Breitbart2164191006
Business Insider2391299
Buzzfeed News344183
CNN28124465
Fox News5239192
Guardian1051176
NPR2960376
National Review3388173
New York Post29107898
New York Times61555
Reuters337123
Talking Points Memo23168182
Vox2274197
Washington Post39190228
\n", "
" ], "text/plain": [ "col_0 0 1 2\n", "publication \n", "Atlantic 28 103 287\n", "Breitbart 216 419 1006\n", "Business Insider 23 91 299\n", "Buzzfeed News 3 44 183\n", "CNN 28 124 465\n", "Fox News 52 39 192\n", "Guardian 10 51 176\n", "NPR 29 60 376\n", "National Review 33 88 173\n", "New York Post 29 107 898\n", "New York Times 6 15 55\n", "Reuters 3 37 123\n", "Talking Points Memo 23 168 182\n", "Vox 22 74 197\n", "Washington Post 39 190 228" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Calulate predicted values\n", "kmeans = KMeans(n_clusters=3, init='k-means++', random_state=42, n_init=20)\n", "y_pred0 = kmeans.fit_predict(features)\n", "\n", "pd.crosstab(y_train, y_pred0)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Adjusted Rand Score: 0.004298765\n", "Silhouette Score: 0.04092314\n" ] } ], "source": [ "from sklearn.metrics import adjusted_rand_score\n", "from sklearn.metrics import silhouette_score\n", "\n", "print('Adjusted Rand Score: {:0.7}'.format(adjusted_rand_score(y_train, y_pred0)))\n", "print('Silhouette Score: {:0.7}'.format(silhouette_score(features, y_pred0, sample_size=60000, metric='euclidean')))" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
col_001234567891011121314
publication
Atlantic13717052515121417811101102215
Breitbart76491262641814911553431566371203336
Business Insider161065272267314423680661
Buzzfeed News7593012531022136311526
CNN28178781231559028143427972411
Fox News6753194953656251520136
Guardian9413208421052161269117
NPR147742221121813171051113891813
National Review205154152514491846358112
New York Post22200663172554471973324427245115
New York Times32090410160341645
Reuters43412230353100872916
Talking Points Memo157798182047836547215165
Vox92647420129122571029758
Washington Post1762138143552517371231635248
\n", "
" ], "text/plain": [ "col_0 0 1 2 3 4 5 6 7 8 9 10 11 12 \\\n", "publication \n", "Atlantic 13 71 70 5 25 15 12 14 17 8 11 10 110 \n", "Breitbart 76 491 262 64 181 49 115 53 43 15 66 37 120 \n", "Business Insider 16 106 52 7 22 6 7 3 14 4 23 6 80 \n", "Buzzfeed News 7 59 30 1 2 5 31 0 2 2 13 6 31 \n", "CNN 28 178 78 1 23 15 59 0 28 14 34 27 97 \n", "Fox News 6 75 31 9 49 5 36 5 6 2 5 15 20 \n", "Guardian 9 41 32 0 8 4 21 0 5 2 16 12 69 \n", "NPR 14 77 42 2 21 12 18 13 17 105 11 13 89 \n", "National Review 20 51 54 15 25 14 4 9 18 4 6 3 58 \n", "New York Post 22 200 66 3 17 25 54 47 19 73 32 44 272 \n", "New York Times 3 20 9 0 4 1 0 1 6 0 3 4 16 \n", "Reuters 4 34 12 2 3 0 3 53 10 0 8 7 2 \n", "Talking Points Memo 15 77 98 18 20 4 7 8 36 5 47 2 15 \n", "Vox 9 26 47 4 20 12 9 12 25 7 10 2 97 \n", "Washington Post 17 62 138 14 35 5 25 17 37 1 23 16 35 \n", "\n", "col_0 13 14 \n", "publication \n", "Atlantic 22 15 \n", "Breitbart 33 36 \n", "Business Insider 6 61 \n", "Buzzfeed News 15 26 \n", "CNN 24 11 \n", "Fox News 13 6 \n", "Guardian 11 7 \n", "NPR 18 13 \n", "National Review 11 2 \n", "New York Post 45 115 \n", "New York Times 4 5 \n", "Reuters 9 16 \n", "Talking Points Memo 16 5 \n", "Vox 5 8 \n", "Washington Post 24 8 " ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Calulate predicted values\n", "kmeans = KMeans(n_clusters=15, init='k-means++', random_state=42, n_init=20)\n", "y_pred = kmeans.fit_predict(features)\n", "\n", "pd.crosstab(y_train, y_pred)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Adjusted Rand Score: 0.02919908\n", "Silhouette Score: 0.06723925\n" ] } ], "source": [ "from sklearn.metrics import adjusted_rand_score\n", "from sklearn.metrics import silhouette_score\n", "\n", "print('Adjusted Rand Score: {:0.7}'.format(adjusted_rand_score(y_train, y_pred)))\n", "print('Silhouette Score: {:0.7}'.format(silhouette_score(features, y_pred, sample_size=60000, metric='euclidean')))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So based on the two results, it seems as though it's much better to stick to 3 clusters as our silhouette score suffers dramatically if we actually want to cluster all 15 different publications.\n", "\n", "### Spectral Clustering" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
col_0012
publication
Atlantic2979229
Breitbart1094335212
Business Insider3147524
Buzzfeed News189392
CNN47811326
Fox News1943653
Guardian1814610
NPR3805530
National Review1896936
New York Post9208826
New York Times60115
Reuters135253
Talking Points Memo21513622
Vox2056523
Washington Post24217540
\n", "
" ], "text/plain": [ "col_0 0 1 2\n", "publication \n", "Atlantic 297 92 29\n", "Breitbart 1094 335 212\n", "Business Insider 314 75 24\n", "Buzzfeed News 189 39 2\n", "CNN 478 113 26\n", "Fox News 194 36 53\n", "Guardian 181 46 10\n", "NPR 380 55 30\n", "National Review 189 69 36\n", "New York Post 920 88 26\n", "New York Times 60 11 5\n", "Reuters 135 25 3\n", "Talking Points Memo 215 136 22\n", "Vox 205 65 23\n", "Washington Post 242 175 40" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sc = SpectralClustering(n_clusters=3)\n", "y_pred2 = sc.fit_predict(features)\n", "\n", "pd.crosstab(y_train, y_pred2)" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Adjusted Rand Score: 0.003924217\n", "Silhouette Score: 0.03775445\n" ] } ], "source": [ "print('Adjusted Rand Score: {:0.7}'.format(adjusted_rand_score(y_train, y_pred2)))\n", "print('Silhouette Score: {:0.7}'.format(silhouette_score(features, y_pred2, sample_size=60000, metric='euclidean')))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Affinity Propagation\n", "\n", "Now, for our final attempt at clustering, affinity propagation. It's a method that will group like data points, but most likely result in an excessive number of clusters. Let's see if that can work to our advantage." ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
col_00123456789...235236237238239240241242243244
publication
Atlantic2157421000...0120010021
Breitbart83518246403...01514170056
Business Insider0002124100...0032000120
Buzzfeed News1001311010...0010010011
CNN2024110085...12211325011
Fox News0102100101...0240000000
Guardian0012000100...0060100001
NPR3001070001...0100931840
National Review2235020000...0002040040
New York Post0103221006...1012230410223
New York Times0000000100...0100010100
Reuters0102100000...0000020000
Talking Points Memo0102102100...0001021012
Vox3601120000...0000010300
Washington Post0104000100...0100030010
\n", "

15 rows × 245 columns

\n", "
" ], "text/plain": [ "col_0 0 1 2 3 4 5 6 7 8 9 ... \\\n", "publication ... \n", "Atlantic 2 1 5 7 4 2 1 0 0 0 ... \n", "Breitbart 8 3 5 18 2 4 6 4 0 3 ... \n", "Business Insider 0 0 0 2 1 2 4 1 0 0 ... \n", "Buzzfeed News 1 0 0 1 3 1 1 0 1 0 ... \n", "CNN 2 0 2 4 1 1 0 0 8 5 ... \n", "Fox News 0 1 0 2 1 0 0 1 0 1 ... \n", "Guardian 0 0 1 2 0 0 0 1 0 0 ... \n", "NPR 3 0 0 1 0 7 0 0 0 1 ... \n", "National Review 2 2 3 5 0 2 0 0 0 0 ... \n", "New York Post 0 1 0 3 2 2 1 0 0 6 ... \n", "New York Times 0 0 0 0 0 0 0 1 0 0 ... \n", "Reuters 0 1 0 2 1 0 0 0 0 0 ... \n", "Talking Points Memo 0 1 0 2 1 0 2 1 0 0 ... \n", "Vox 3 6 0 1 1 2 0 0 0 0 ... \n", "Washington Post 0 1 0 4 0 0 0 1 0 0 ... \n", "\n", "col_0 235 236 237 238 239 240 241 242 243 244 \n", "publication \n", "Atlantic 0 1 2 0 0 1 0 0 2 1 \n", "Breitbart 0 1 5 14 1 7 0 0 5 6 \n", "Business Insider 0 0 3 2 0 0 0 1 2 0 \n", "Buzzfeed News 0 0 1 0 0 1 0 0 1 1 \n", "CNN 1 2 2 1 1 3 25 0 1 1 \n", "Fox News 0 2 4 0 0 0 0 0 0 0 \n", "Guardian 0 0 6 0 1 0 0 0 0 1 \n", "NPR 0 1 0 0 9 3 1 8 4 0 \n", "National Review 0 0 0 2 0 4 0 0 4 0 \n", "New York Post 10 12 23 0 4 1 0 2 2 3 \n", "New York Times 0 1 0 0 0 1 0 1 0 0 \n", "Reuters 0 0 0 0 0 2 0 0 0 0 \n", "Talking Points Memo 0 0 0 1 0 2 1 0 1 2 \n", "Vox 0 0 0 0 0 1 0 3 0 0 \n", "Washington Post 0 1 0 0 0 3 0 0 1 0 \n", "\n", "[15 rows x 245 columns]" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "af = AffinityPropagation()\n", "y_pred3 = af.fit_predict(features)\n", "\n", "pd.crosstab(y_train, y_pred3)" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Adjusted Rand Score: 0.01145051\n", "Silhouette Score: 0.0110909\n" ] } ], "source": [ "print('Adjusted Rand Score: {:0.7}'.format(adjusted_rand_score(y_train, y_pred3)))\n", "print('Silhouette Score: {:0.7}'.format(silhouette_score(features, y_pred3, sample_size=60000, metric='euclidean')))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And the results are worthless, just pitiful. Seems like k-means is the best clustering algorithm- mostly because our other methods were far worse, not because it performed well." ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": [ "X_train_cluster = pd.DataFrame(features)\n", "X_train_cluster['kmeans'] = y_pred" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Training the Model\n", "\n", "So now that we attempted clustering with the datset, it's time to run the models.\n", "\n", "### Random Forest" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Random forest classifier score (without clustering): 0.41634(+/- 0.02)\n", "\n", "Random forest classifier score (with clustering): 0.41505(+/- 0.02)\n" ] } ], "source": [ "rfc = ensemble.RandomForestClassifier()\n", "rfc_train = cross_val_score(rfc, features, y_train, cv=5, n_jobs=-1)\n", "print('Random forest classifier score (without clustering): {:.5f}(+/- {:.2f})\\n'.format(rfc_train.mean(), rfc_train.std()*2))\n", "\n", "rfc_train_c = cross_val_score(rfc, X_train_cluster, y_train, cv=5, n_jobs=-1)\n", "print('Random forest classifier score (with clustering): {:.5f}(+/- {:.2f})'.format(rfc_train_c.mean(), rfc_train_c.std()*2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Logistic Regression" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Logistic regression score (without clustering): 0.44837(+/- 0.02)\n", "\n", "Logistic regression score (with clustering): 0.44837(+/- 0.02)\n" ] } ], "source": [ "lr = LogisticRegression()\n", "lr_train = cross_val_score(lr, features, y_train, cv=5, n_jobs=-1)\n", "print('Logistic regression score (without clustering): {:.5f}(+/- {:.2f})\\n'.format(lr_train.mean(), lr_train.std()*2))\n", "\n", "lr_train_c = cross_val_score(lr, X_train_cluster, y_train, cv=5, n_jobs=-1)\n", "print('Logistic regression score (with clustering): {:.5f}(+/- {:.2f})'.format(lr_train_c.mean(), lr_train_c.std()*2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Gradient Boosting Classifier" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Gradient boosting classifier score (without clustering): 0.49597(+/- 0.02)\n", "\n", "Gradient boosting classifier score (with clustering): 0.49569(+/- 0.03)\n" ] } ], "source": [ "gbc = ensemble.GradientBoostingClassifier()\n", "gbc_train = cross_val_score(gbc, features, y_train, cv=5, n_jobs=-1)\n", "print('Gradient boosting classifier score (without clustering): {:.5f}(+/- {:.2f})\\n'.format(gbc_train.mean(), gbc_train.std()*2))\n", "\n", "gbc_train_c = cross_val_score(gbc, X_train_cluster, y_train, cv=5, n_jobs=-1)\n", "print('Gradient boosting classifier score (with clustering): {:.5f}(+/- {:.2f})'.format(gbc_train_c.mean(), gbc_train_c.std()*2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It looks like out of the 3 models, gradient boosting fares the best, and without incorporating clustering at that. Let's see if we can tune our model with better parameters using GridSearchCV.\n", "\n", "### Optimized Gradient Boosting Classifier " ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best parameters:\n", "{'loss': 'deviance', 'max_depth': 20, 'max_features': 'sqrt', 'min_samples_leaf': 20, 'n_estimators': 400}\n", "Best Score:\n", "0.520017157563626\n" ] } ], "source": [ "# Parameters for gradient boosting classifier\n", "param_grid = {'loss':['deviance'],\n", " 'max_features': ['sqrt'],\n", " 'n_estimators': [400, 800],\n", " 'max_depth': [12, 20],\n", " \"min_samples_leaf\" : [12, 20]}\n", "\n", "# Run grid search to find ideal parameters\n", "gbc_grid = GridSearchCV(gbc, param_grid = param_grid, n_jobs=-1)\n", "\n", "# Initialize and fit the model.\n", "gbc_grid.fit(features, y_train)\n", "\n", "# Return best parameters and best score\n", "print('Best parameters:')\n", "print(gbc_grid.best_params_)\n", "print('Best Score:')\n", "print(gbc_grid.best_score_)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It's not much of an improvement, but we'll take anything we can at this time. We could attempt to improve the accuracy of our model using larger parameter values, but expanding this model could increase the runtime exponentially.\n", "\n", "# Testing the Model\n", "\n", "Recall that earlier, we split the data into 2 sets, a training set and a test set. Now, it's time to test the test set and see if the settings from our training model will work with the test set." ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [], "source": [ "# Normalize Tf-idf vectors\n", "X_test_norm = normalize(X_test_tfidf)" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [], "source": [ "X_test_words = []\n", "\n", "for row in X_test:\n", " # Processing each row for tokens\n", " row_doc = nlp(row)\n", " # Calculating length of each sentence\n", " sent_len = len(row_doc) \n", " # Initializing counts of different parts of speech\n", " advs = 0\n", " verb = 0\n", " noun = 0\n", " adj = 0\n", " for token in row_doc:\n", " # Identifying each part of speech and adding to counts\n", " if token.pos_ == 'ADV':\n", " advs +=1\n", " elif token.pos_ == 'VERB':\n", " verb +=1\n", " elif token.pos_ == 'NOUN':\n", " noun +=1\n", " elif token.pos_ == 'ADJ':\n", " adj +=1\n", " # Creating a list of all features for each sentence\n", " X_test_words.append([row_doc, advs, verb, noun, adj, sent_len])\n", " \n", "# Data frame for features\n", "X_test_count = pd.DataFrame(data=X_test_words, columns=['BOW', 'ADV', 'VERB', 'NOUN', 'ADJ', 'sent_length'])\n", "\n", "# Change token count to token percentage\n", "for column in X_test_count.columns[1:5]:\n", " X_test_count[column] = X_test_count[column] / X_test_count['sent_length']\n", "\n", "# Normalize X_count\n", "X_test_counter = normalize(X_test_count.drop('BOW',axis=1))\n", "X_test_counter = pd.DataFrame(data=X_test_counter)" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
0123401234...140141142143144145146147148149
00.0001270.0004650.0003880.0002491.0000000.00.00.0000000.00.0...0.0000000.1322120.1438140.00.0000000.0000000.000000.3581750.114510.0
10.0002730.0006060.0006060.0003181.0000000.00.00.0000000.00.0...0.2424030.0000000.0000000.00.0000000.0000000.000000.0000000.000000.0
20.0000460.0001890.0002360.0000901.0000000.00.00.1005990.00.0...0.0000000.0000000.0000000.00.3204760.0949360.000000.0000000.000000.0
30.0000570.0003120.0003270.0001541.0000000.00.00.0000000.00.0...0.0000000.0000000.0000000.00.0000000.1469410.140510.0000000.000000.0
40.0004590.0015820.0004080.0004590.9999980.00.00.0000000.00.0...0.1389320.0000000.0000000.00.0000000.0000000.000000.0000000.000000.0
\n", "

5 rows × 155 columns

\n", "
" ], "text/plain": [ " 0 1 2 3 4 0 1 2 3 \\\n", "0 0.000127 0.000465 0.000388 0.000249 1.000000 0.0 0.0 0.000000 0.0 \n", "1 0.000273 0.000606 0.000606 0.000318 1.000000 0.0 0.0 0.000000 0.0 \n", "2 0.000046 0.000189 0.000236 0.000090 1.000000 0.0 0.0 0.100599 0.0 \n", "3 0.000057 0.000312 0.000327 0.000154 1.000000 0.0 0.0 0.000000 0.0 \n", "4 0.000459 0.001582 0.000408 0.000459 0.999998 0.0 0.0 0.000000 0.0 \n", "\n", " 4 ... 140 141 142 143 144 145 146 \\\n", "0 0.0 ... 0.000000 0.132212 0.143814 0.0 0.000000 0.000000 0.00000 \n", "1 0.0 ... 0.242403 0.000000 0.000000 0.0 0.000000 0.000000 0.00000 \n", "2 0.0 ... 0.000000 0.000000 0.000000 0.0 0.320476 0.094936 0.00000 \n", "3 0.0 ... 0.000000 0.000000 0.000000 0.0 0.000000 0.146941 0.14051 \n", "4 0.0 ... 0.138932 0.000000 0.000000 0.0 0.000000 0.000000 0.00000 \n", "\n", " 147 148 149 \n", "0 0.358175 0.11451 0.0 \n", "1 0.000000 0.00000 0.0 \n", "2 0.000000 0.00000 0.0 \n", "3 0.000000 0.00000 0.0 \n", "4 0.000000 0.00000 0.0 \n", "\n", "[5 rows x 155 columns]" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Combining features into one data frame\n", "X_test_norm_df = pd.DataFrame(data=X_test_norm.toarray())\n", "features_test = pd.concat([X_test_counter, X_test_norm_df], ignore_index=False, axis=1)\n", "features_test.head()" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
col_001234567891011121314
publication
Atlantic435621511814410181555
Breitbart564536245671883123761825809
Business Insider91984101902242292542
Buzzfeed News4846560931150165
CNN9211728272012108431847
Fox News1342138411040184561
Guardian2155316401013160315
NPR427552153225422863110
National Review715411305181045501
New York Post58014249292013152080161112
New York Times353071161030201
Reuters4010681847030129
Talking Points Memo242048132703179546
Vox92711211421201102614
Washington Post8516518524432163516
\n", "
" ], "text/plain": [ "col_0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 \\\n", "publication \n", "Atlantic 4 35 6 2 15 11 8 14 4 10 18 1 5 5 \n", "Breitbart 56 45 36 24 56 7 18 83 12 3 76 18 25 80 \n", "Business Insider 9 19 8 4 10 19 0 22 4 2 29 2 5 4 \n", "Buzzfeed News 4 8 4 6 5 6 0 9 3 1 15 0 1 6 \n", "CNN 9 21 17 28 27 2 0 12 10 8 43 1 8 4 \n", "Fox News 13 4 2 13 8 4 1 10 4 0 18 4 5 6 \n", "Guardian 2 15 5 3 16 4 0 10 1 3 16 0 3 1 \n", "NPR 4 27 5 5 21 5 3 22 5 42 28 6 3 1 \n", "National Review 7 15 4 1 13 0 5 18 1 0 4 5 5 0 \n", "New York Post 5 80 14 24 9 29 20 13 15 20 80 1 6 11 \n", "New York Times 3 5 3 0 7 1 1 6 1 0 3 0 2 0 \n", "Reuters 4 0 1 0 6 8 18 4 7 0 3 0 1 2 \n", "Talking Points Memo 2 4 20 4 8 1 3 27 0 3 17 9 5 4 \n", "Vox 9 27 11 2 11 4 2 12 0 1 10 2 6 1 \n", "Washington Post 8 5 16 5 18 5 2 44 3 2 16 3 5 1 \n", "\n", "col_0 14 \n", "publication \n", "Atlantic 5 \n", "Breitbart 9 \n", "Business Insider 2 \n", "Buzzfeed News 5 \n", "CNN 7 \n", "Fox News 1 \n", "Guardian 5 \n", "NPR 10 \n", "National Review 1 \n", "New York Post 12 \n", "New York Times 1 \n", "Reuters 9 \n", "Talking Points Memo 6 \n", "Vox 4 \n", "Washington Post 6 " ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Calulate predicted values\n", "kmeans = KMeans(n_clusters=15, init='k-means++', random_state=42, n_init=20)\n", "y_pred_test = kmeans.fit_predict(features_test)\n", "\n", "pd.crosstab(y_test, y_pred_test)" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Adjusted Rand Score: 0.02586774\n", "Silhouette Score: 0.06681075\n" ] } ], "source": [ "print('Adjusted Rand Score: {:0.7}'.format(adjusted_rand_score(y_test, y_pred_test)))\n", "print('Silhouette Score: {:0.7}'.format(silhouette_score(features_test, y_pred_test, sample_size=60000, metric='euclidean')))" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [], "source": [ "X2_test_c = pd.DataFrame(features_test)\n", "X2_test_c['kmeans_clust'] = y_pred_test" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test set score: 0.47591(+/- 0.023)\n" ] } ], "source": [ "gbc_grid_scores_test = cross_val_score(gbc_grid, features_test, y_test, cv=5)\n", "print('Test set score: {:.5f}(+/- {:.3f})'.format(gbc_grid_scores_test.mean(), gbc_grid_scores_test.std()*2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Conclusion\n", "\n", "So what did we learn today? Well for starters, natural language processing can be extremely taxing on memory since it requires us to create a massive dataframe of words. If we had access to more memory, it would likely have been possible to use a larger sample from the original dataset (we only used 1%) and/or retain more words (150/thousands) for feature prediction. In addition, some other issues that might arise are simply that we are looking at too many different publications, and our model cannot accurately distinguish them all.\n", "\n", "\n", "# Source\n", "\n", "https://www.kaggle.com/snapcrack/all-the-news" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" } }, "nbformat": 4, "nbformat_minor": 2 }