Leah nags her fiance Jeremy on Twitter about being late — and he responds angrily. Is there trouble in paradise?\\nTeen Mom 2 star Leah Messer, 19, has been engaged to her fiance Jere... \n",
"3 Christian Siriano Designs Wedding Dresses For Nordstrom . The designer is branching out, into the wedding business! Now you can say ‘I Do’ in one of his coveted designs — get all the details on his latest fashion venture here.
\\nFormer Project Runway winner Christian Siriano, 26, has designed countless dr... \n",
"4 Tips & Tricks For A Flawless Valentine’s Day Look Just Like Reese Witherspoon .
Do you want to create a sexy makeup look for a Valentine’s Day date, but you’re sick of the classic smokey eye? We have the perfect look for you! Celebrity makeup artist Jillian Dempsey gives you a how-to for a twist on the over-played smokey eye. Keep reading to see her beauty tricks and shop […]
\\n "
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# load in saved data\n",
"df_orig = pd.read_csv(\"input_data.csv\", encoding = \"utf-8\")\n",
"df_orig.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Text Preprocessing: Simple Cleaning\n",
"\n",
"We now have a data frame where our blob of text is often full of embedded html and has not been cleansed in any particular way. \n",
"\n",
"Here we will:\n",
"\n",
"* Strip out all html and essentially render the text as it would read. \n",
"* Lowercase everything. \n",
"* Replace or remove various specific characters that might make things harder on the model. \n",
"\n",
"p.s. Using [tqdm](https://github.com/tqdm/tqdm) everywhere in this notebook to get progress bar's on loops. Find it really satisfying for some reason :)"
]
},
{
"cell_type": "code",
"execution_count": 90,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-21T13:52:59.582356Z",
"start_time": "2017-08-21T13:52:58.502108Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████████████████████████| 118024/118024 [00:01<00:00, 117183.60it/s]\n"
]
}
],
"source": [
"# set up list for collecting cleaned content into\n",
"results_all = []\n",
"\n",
"# pull the df into a list to loop through (seems quicker then looping through the pd df itself)\n",
"raw_data = list(zip(df_orig.pid,df_orig.text))\n",
"\n",
"# loop through each row of df and clean strip out the html tags\n",
"for pid, text in tqdm(raw_data):\n",
" # use try block to ignore errors in cleaning, should be fine as have lots of data\n",
" try:\n",
" result = [pid, remove_tags(text)]\n",
" results_all.append(result)\n",
" except Exception as e: \n",
" # do nothing on error, we are ok to ignore any posts we can't clean up for whatever reason\n",
" #print(e)\n",
" pass"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now do some kinda ugly and brute force cleaning...\n",
"\n",
"(I'm sure there is much more elegant and generalized ways to do this but meh) "
]
},
{
"cell_type": "code",
"execution_count": 112,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-21T13:58:28.736741Z",
"start_time": "2017-08-21T13:58:07.747394Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"### df SHAPE = (118024, 2) ###\n"
]
},
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" pid | \n",
" text | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" I8bC0jze55Ow0LZoJSyRYGr1K0M= | \n",
" bachelorette ashley hebert how i m getting in shape for my wedding . find out how you can get as fit as ashley with all of her secret diet and workout tips. bachelorette ashley hebert is trying hard to stay fit for her wedding to j.p . rosenbaum that will most likely take place later this year . ashley shares her wedding workout and diet tips with in touch magazine. | \n",
"
\n",
" \n",
" | 1 | \n",
" Oj/mjv0XIpoJPaLX+XCQep65ToU= | \n",
" teen mom star leah messer s dream wedding the details revealed . leah fiance jeremy calvert gushes about the type of ceremony they can t wait to have the guest list and even the honeymoon destination. teen mom star leah messer is happily engaged to her new fiance jeremy calvert . and while she is pregnant reportedly with twins leah 19 and jeremy 22 are already planning their dream wedding. | \n",
"
\n",
" \n",
" | 2 | \n",
" dOlSCJ9+2t1xLe/zB2Re+H+KEg4= | \n",
" are teen mom leah messer and jeremy calvert already fighting ? . leah nags her fiance jeremy on twitter about being late and he responds angrily . is there trouble in paradise ? teen mom star leah messer 19 has been engaged to her fiance jeremy calvert 22 for less than a month now and it seems that they are already starting to fight. | \n",
"
\n",
" \n",
" | 3 | \n",
" JSUPPhMmW+EG7bk3SwjuGsRIOQs= | \n",
" christian siriano designs wedding dresses for nordstrom . the designer is branching out into the wedding business . now you can say i do in one of his coveted designs get all the details on his latest fashion venture here. former project runway winner christian siriano 26 has designed countless dresses for a-listers like heidi klum fergie taylor swift and sarah hyland to name a few . aside from his namesake collection the designer plans to branch into bridal for nordstrom where we re sure to... | \n",
"
\n",
" \n",
" | 4 | \n",
" VUu+U6+AoDr3dM7MjPldJzrom5g= | \n",
" tips and tricks for a flawless valentine day look just like reese witherspoon . do you want to create a sexy makeup look for a valentine day date but you re sick of the classic smokey eye ? we have the perfect look for you . celebrity makeup artist jillian dempsey gives you a how-to for a twist on the over-played smokey eye . keep reading to see her beauty tricks and shop | \n",
"
\n",
" \n",
"
\n",
"
So sexy! Now we see where Perrie Edwards got her good looks from. The Little Mix singer shared a pic of her dad, and let's just say, the Internet isn't freaking out over nothing! He's a stud muffin.
\\nOMG! Dads may not be everyone's thing, but we have to say Perrie Edwards' dad is incredibly hot! The singer shared a pic of her dreamy papa on Instagra...\n",
"Name: text, dtype: object\n"
]
}
],
"source": [
"# pull a random sample article to look at cleaning results\n",
"samp_ind = list(df.pid.sample(1))\n",
"#samp_ind = list(['I8bC0jze55Ow0LZoJSyRYGr1K0M='])\n",
"print(samp_ind)\n",
"print('\\n## CLEANED ##############################\\n')\n",
"print(str(df.loc[df['pid'].isin(samp_ind)]['text']))\n",
"print('\\n## ORIGINAL ##############################\\n')\n",
"print(str(df_orig.loc[df_orig['pid'].isin(samp_ind)]['text']))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Text Preprocessing: Phrase Creation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Create sentences with bigram phrases flagged."
]
},
{
"cell_type": "code",
"execution_count": 113,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-21T17:03:59.455280Z",
"start_time": "2017-08-21T14:17:50.652653Z"
},
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|████████████████████████████████| 118024/118024 [2:44:23<00:00, 11.97it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"### The number of sentences is 118024\n",
"### The number of words is 11568437\n"
]
}
],
"source": [
"# get list of documents\n",
"documents = list(df['text'])\n",
"# get a list of single sentences\n",
"#sentences = [s.split('.',1)[0].strip() for s in documents]\n",
"sentences = documents # uncomment this to just use the whole dosument as one big sentence\n",
"\n",
"# create sentence stream iterator, removing stopwords and punctuation, also remove small sentences\n",
"sentence_stream = [[str(i).translate(str.maketrans('','',string.punctuation)) for i in sentence.split(\" \") if i not in stopwords.words('english')] for sentence in tqdm(sentences)]\n",
"\n",
"# remove small sentences as not much to be learned from them \n",
"#sentence_stream = [sentence for sentence in sentence_stream if len(sentence) > 3 ] # only need this if passing sentences as opposed to the full doc\n",
"\n",
"# create bigram phrases\n",
"phrases = Phrases(sentence_stream, min_count=250)\n",
"bigram = Phraser(phrases)\n",
"\n",
"# create list of sentences to feed into the word2vec model\n",
"sentences = list(bigram[sentence_stream])\n",
"words = [i for j in sentences for i in j]\n",
"\n",
"# save sentences object\n",
"with open('sentences.pickle', 'wb') as handle:\n",
" pickle.dump(sentences, handle, protocol=pickle.HIGHEST_PROTOCOL)\n",
"\n",
"print(\"### The number of sentences is \" + str(len(sentences)))\n",
"print(\"### The number of words is \" + str(len(words)))"
]
},
{
"cell_type": "code",
"execution_count": 77,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-19T21:04:33.932531Z",
"start_time": "2017-08-19T21:04:30.024749Z"
},
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"### The number of sentences is 41448\n",
"### The number of words is 5863883\n"
]
}
],
"source": [
"# if loading in saved sentences object\n",
"sentences = pickle.load( open( \"sentences.pickle\", \"rb\" ) )\n",
"words = [i for j in sentences for i in j]\n",
"\n",
"print(\"### The number of sentences is \" + str(len(sentences)))\n",
"print(\"### The number of words is \" + str(len(words)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Lets pull a sample of some sentences to see what we now have and if it is looking ok for sending into the actual model building stage."
]
},
{
"cell_type": "code",
"execution_count": 116,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-21T17:16:51.519842Z",
"start_time": "2017-08-21T17:16:51.504844Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"['kourtney_kardashian forces khloe move divorce kuwtk kuwtk returns khloe_kardashian dealing heavy material literally kourtney forces khloe pack house shared lamar_odom divorce time move keeping_kardashians returns e sunday june 98c brand_new episodes cameras follow ',\n",
" 'jared fogle exsubway spokesman plead guilty child porn charges report whoa jared fogle man known face subway expected plead guilty possession child pornography charges according multiple outlets ongoing federal investigation jared fogle 37 reportedly accept deal aug plead guilty charges possession child pornography comes couple weeks alleged ',\n",
" 'miley_cyrus refuses release album year patrick way sorry miley fans bangerz hitmaker refusing release album sad pdafilled relationship patrick getting way music hollywoodlifecom_learned exclusive_details miley dropping new_album year bring tissues miley_cyrus 22 ',\n",
" 'kim_kardashian robbery planned someone inside entourage shocking report interesting details come light parisian officials get closer finding behind kim_kardashian october robbery wont believe update individual limo company kardashian family used suspected kim_kardashian robbery may inside job according jan report french newspaper le monde knew people arrested important detail emerged one suspects works car company kim used last person driven day attack whoa kim_kardashian robbery suspects french police say individual could given robbers inside information kim staying according newspaper m6 tv also reports suspects may touch kardashian family scary multiple french outlets reporting extent driver involvement attack unclear french police seem think long story short gave robbers least inside info keeping_kardashians star whereabouts terrifying police get closer jean veil kim french attorney said happy reassured developments case were_glad hear source previously reiterated hollywoodlifecom_exclusively kim relieved paris police making headway arrests nice surprise veil also reportedly told france lexpress magazine one hand perhaps mean jewels recovered hand puts end disgraceful speculation people thought clever pretend robbery setup publicity stunt madame kardashian course criminals nabbed kim might face court means nightmare far kim absolutely terrified possibility going court starting entire process source reveals feel hollywoodlifers_think kim case closed soon hope so',\n",
" 'kylie_jenner flaunts boobs curvaceous figure waist trainer selfie – got flaunt kylie_jenner put body display another sizzling selfie youngest keeping_kardashians star posed revealing white shirt waist trainer showing family trademark curves kylie_jenner 18 gunning crown kardashian waist trainer ']"
]
},
"execution_count": 116,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# pull some random sentences to see if looking ok\n",
"n_samples = 5\n",
"sample = random.sample(range(0, len(sentences)), n_samples)\n",
"[' '.join(sentences[i]) for i in sample]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Get some counts etc that might be useful."
]
},
{
"cell_type": "code",
"execution_count": 117,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-21T17:17:02.617923Z",
"start_time": "2017-08-21T17:16:51.522845Z"
},
"code_folding": [],
"collapsed": true
},
"outputs": [],
"source": [
"# get a big list of all words\n",
"words_long = list(itertools.chain(*sentences))\n",
"# create a big string\n",
"words_string = ' '.join(words_long)\n",
"# clean up\n",
"del words_long\n",
"# get word counts into a dict\n",
"word_counts = defaultdict(int)\n",
"for word in words_string.split():\n",
" word_counts[word] += 1"
]
},
{
"cell_type": "code",
"execution_count": 118,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-21T17:17:02.879253Z",
"start_time": "2017-08-21T17:17:02.620907Z"
},
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"new 49995\n",
"one 47279\n",
"see 46184\n",
"get 36628\n",
"like 33597\n",
"love 33249\n",
"look 31692\n",
"time 31661\n",
"show 30224\n",
"back 28398\n",
"also 25550\n",
"two 25377\n",
"first 25118\n",
"know 25030\n",
"going 24923\n",
"even 24220\n",
"think 23700\n",
"may 23439\n",
"said 23426\n",
"watch 22827\n"
]
}
],
"source": [
"# print top 20 words\n",
"for w in sorted(word_counts, key=word_counts.get, reverse=True)[:20]:\n",
" print(w, word_counts[w])"
]
},
{
"cell_type": "code",
"execution_count": 119,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-21T17:17:03.177643Z",
"start_time": "2017-08-21T17:17:02.884758Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"syndications 1\n",
"zampino 1\n",
"follieri 1\n",
"tuohy 1\n",
"bardo 1\n",
"jordangreen 1\n",
"popsynthpop 1\n",
"brotherlyact 1\n",
"javadd 1\n",
"mullingar 1\n",
"constantinova 1\n",
"dobreva 1\n",
"bulgariancanadian 1\n",
"apr6 1\n",
"planetsomerhalder 1\n",
"penns 1\n",
"tappahannock 1\n",
"safarti 1\n",
"b’day 1\n",
"am…sasha 1\n"
]
}
],
"source": [
"# print bottom 20 words\n",
"for w in sorted(word_counts, key=word_counts.get, reverse=False)[:20]:\n",
" print(w, word_counts[w])"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Build Model\n",
"\n",
"Now we are ready to pass our sentences to [gensim.Word2Vec](https://radimrehurek.com/gensim/models/word2vec.html) to build our own model. \n",
"\n",
"There are a few key hyper parameters we need to build the model. We are not doing anything fancy like cross validation here. Instead i did a few manual trial and error builds on a smaller sample of sentences until i found a paramter set that generlally made sense to me given the size of the datasrt and our focus. "
]
},
{
"cell_type": "code",
"execution_count": 120,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-21T17:19:02.571044Z",
"start_time": "2017-08-21T17:17:03.182645Z"
},
"collapsed": true
},
"outputs": [],
"source": [
"# train model\n",
"model = Word2Vec( \n",
" sentences = sentences, \n",
" size = 100, \n",
" min_count = 250, \n",
" window = 10, # use a largish window since passing full document as sentence \n",
" iter = 10, \n",
" workers = multiprocessing.cpu_count()\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Save Model\n",
"\n",
"Once the model is built we save it to disk so can be loaded back in later for exploration without needing to rebuild each time."
]
},
{
"cell_type": "code",
"execution_count": 121,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-21T17:19:02.754922Z",
"start_time": "2017-08-21T17:19:02.573045Z"
},
"collapsed": true
},
"outputs": [],
"source": [
"# save model to disk\n",
"model.save(\"celeb_word2vec_model\")\n",
"\n",
"#If you’re finished training a model (=no more updates, only querying), then switch to the gensim.models.KeyedVectors instance in wv\n",
"word_vectors = model.wv\n",
"#del model\n",
"\n",
"# save word vectors to disk\n",
"word_vectors.save(\"word_vectors\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load Model\n",
"\n",
"Load in the saved model. We can run from here if exploring an already trained and saved model."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-22T20:02:41.147758Z",
"start_time": "2017-08-22T20:02:40.821546Z"
},
"collapsed": true
},
"outputs": [],
"source": [
"# load saved model\n",
"model = Word2Vec.load('celeb_word2vec_model')\n",
"\n",
"# load saved word vectors\n",
"word_vectors = KeyedVectors.load('word_vectors')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Save Vectors\n",
"\n",
"We will save the raw vectors out to a nice and easy, human readable, text file. Then read back in the wide matrix of vectors into a pandas dataframe in order to transform it into a long format later for exploration in a [Tableau Public workbook](https://public.tableau.com/profile/andrew5416#!/vizhome/word_vector_explorer/WordVectorExplorer)."
]
},
{
"cell_type": "code",
"execution_count": 123,
"metadata": {
"ExecuteTime": {
"end_time": "2017-08-21T17:19:03.877993Z",
"start_time": "2017-08-21T17:19:02.943695Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(6040, 101)\n"
]
},
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" word | \n",
" 1 | \n",
" 2 | \n",
" 3 | \n",
" 4 | \n",
" 5 | \n",
" 6 | \n",
" 7 | \n",
" 8 | \n",
" 9 | \n",
" ... | \n",
" 91 | \n",
" 92 | \n",
" 93 | \n",
" 94 | \n",
" 95 | \n",
" 96 | \n",
" 97 | \n",
" 98 | \n",
" 99 | \n",
" 100 | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" NaN | \n",
" 0.423812 | \n",
" -0.873024 | \n",
" -1.076316 | \n",
" -0.523255 | \n",
" -0.069986 | \n",
" -0.812739 | \n",
" 0.632879 | \n",
" 0.161528 | \n",
" 0.282792 | \n",
" ... | \n",
" -0.371600 | \n",
" -0.553812 | \n",
" -0.671738 | \n",
" -0.336277 | \n",
" -0.364511 | \n",
" 0.176831 | \n",
" -0.294965 | \n",
" 0.629585 | \n",
" -0.376467 | \n",
" 0.779368 | \n",
"
\n",
" \n",
" | 1 | \n",
" new | \n",
" 2.319383 | \n",
" -1.351134 | \n",
" -0.583713 | \n",
" 0.406562 | \n",
" 4.404606 | \n",
" -3.165166 | \n",
" 2.232082 | \n",
" -0.446304 | \n",
" 2.595640 | \n",
" ... | \n",
" 2.760589 | \n",
" 2.519782 | \n",
" -0.046284 | \n",
" 3.112629 | \n",
" -0.785255 | \n",
" 1.850350 | \n",
" -0.543355 | \n",
" 0.321961 | \n",
" -2.899933 | \n",
" 2.350732 | \n",
"
\n",
" \n",
" | 2 | \n",
" one | \n",
" -0.025912 | \n",
" -1.830144 | \n",
" -0.779826 | \n",
" -1.594398 | \n",
" -0.478641 | \n",
" -1.684386 | \n",
" -0.148587 | \n",
" -1.644406 | \n",
" 1.716436 | \n",
" ... | \n",
" -0.068255 | \n",
" -0.313197 | \n",
" 0.801451 | \n",
" -1.983233 | \n",
" -0.122030 | \n",
" -0.016098 | \n",
" 0.392162 | \n",
" -0.247242 | \n",
" -1.243877 | \n",
" -1.139465 | \n",
"
\n",
" \n",
" | 3 | \n",
" see | \n",
" -0.932137 | \n",
" -1.764545 | \n",
" -5.519857 | \n",
" -0.801152 | \n",
" -0.065875 | \n",
" -0.366409 | \n",
" -0.401208 | \n",
" -1.666864 | \n",
" 0.933654 | \n",
" ... | \n",
" -0.520679 | \n",
" -0.760519 | \n",
" 0.147650 | \n",
" 0.923039 | \n",
" 0.284766 | \n",
" -1.203566 | \n",
" 1.997292 | \n",
" -0.284277 | \n",
" -4.810210 | \n",
" -1.662077 | \n",
"
\n",
" \n",
" | 4 | \n",
" get | \n",
" -1.038089 | \n",
" 0.575331 | \n",
" -2.349574 | \n",
" 1.134312 | \n",
" 1.008488 | \n",
" -1.161399 | \n",
" 0.614903 | \n",
" -4.204766 | \n",
" -0.234601 | \n",
" ... | \n",
" -0.178508 | \n",
" -1.393333 | \n",
" 0.106876 | \n",
" 0.354701 | \n",
" 1.145287 | \n",
" 1.471832 | \n",
" 1.583619 | \n",
" -0.773887 | \n",
" -2.289845 | \n",
" -0.899614 | \n",
"
\n",
" \n",
"
\n",
"
5 rows × 101 columns
\n",
"