{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook goes through a necessary step of any data science project - data cleaning. Data cleaning is a time consuming and unenjoyable task, yet it's a very important one. Keep in mind, \"garbage in, garbage out\". Feeding dirty data into a model will give us results that are meaningless.\n",
    "\n",
    "Specifically, we'll be walking through:\n",
    "\n",
    "1. **Getting the data - **in this case, we'll be scraping data from a website\n",
    "2. **Cleaning the data - **we will walk through popular text pre-processing techniques\n",
    "3. **Organizing the data - **we will organize the cleaned data into a way that is easy to input into other algorithms\n",
    "\n",
    "The output of this notebook will be clean, organized data in two standard text formats:\n",
    "\n",
    "1. **Corpus** - a collection of text\n",
    "2. **Document-Term Matrix** - word counts in matrix format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem Statement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a reminder, our goal is to look at transcripts of various comedians and note their similarities and differences. Specifically, I'd like to know if Ali Wong's comedy style is different than other comedians, since she's the comedian that got me interested in stand up comedy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting The Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Luckily, there are wonderful people online that keep track of stand up routine transcripts. [Scraps From The Loft](http://scrapsfromtheloft.com) makes them available for non-profit and educational purposes.\n",
    "\n",
    "To decide which comedians to look into, I went on IMDB and looked specifically at comedy specials that were released in the past 5 years. To narrow it down further, I looked only at those with greater than a 7.5/10 rating and more than 2000 votes. If a comedian had multiple specials that fit those requirements, I would pick the most highly rated one. I ended up with a dozen comedy specials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Web scraping, pickle imports\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pickle\n",
    "\n",
    "# Scrapes transcript data from scrapsfromtheloft.com\n",
    "def url_to_transcript(url):\n",
    "    '''Returns transcript data specifically from scrapsfromtheloft.com.'''\n",
    "    page = requests.get(url).text # Get all data from URL\n",
    "    soup = BeautifulSoup(page, \"lxml\") # Read as an HTML document\n",
    "    text = [p.text for p in soup.find(class_=\"post-content\").find_all('p')] # Pull out all text from post-content\n",
    "    print(url)\n",
    "    return text\n",
    "\n",
    "# URLs of transcripts in scope\n",
    "urls = ['http://scrapsfromtheloft.com/2017/05/06/louis-ck-oh-my-god-full-transcript/',\n",
    "        'http://scrapsfromtheloft.com/2017/04/11/dave-chappelle-age-spin-2017-full-transcript/',\n",
    "        'http://scrapsfromtheloft.com/2018/03/15/ricky-gervais-humanity-transcript/',\n",
    "        'http://scrapsfromtheloft.com/2017/08/07/bo-burnham-2013-full-transcript/',\n",
    "        'http://scrapsfromtheloft.com/2017/05/24/bill-burr-im-sorry-feel-way-2014-full-transcript/',\n",
    "        'http://scrapsfromtheloft.com/2017/04/21/jim-jefferies-bare-2014-full-transcript/',\n",
    "        'http://scrapsfromtheloft.com/2017/08/02/john-mulaney-comeback-kid-2015-full-transcript/',\n",
    "        'http://scrapsfromtheloft.com/2017/10/21/hasan-minhaj-homecoming-king-2017-full-transcript/',\n",
    "        'http://scrapsfromtheloft.com/2017/09/19/ali-wong-baby-cobra-2016-full-transcript/',\n",
    "        'http://scrapsfromtheloft.com/2017/08/03/anthony-jeselnik-thoughts-prayers-2015-full-transcript/',\n",
    "        'http://scrapsfromtheloft.com/2018/03/03/mike-birbiglia-my-girlfriends-boyfriend-2013-full-transcript/',\n",
    "        'http://scrapsfromtheloft.com/2017/08/19/joe-rogan-triggered-2016-full-transcript/']\n",
    "\n",
    "# Comedian names\n",
    "comedians = ['louis', 'dave', 'ricky', 'bo', 'bill', 'jim', 'john', 'hasan', 'ali', 'anthony', 'mike', 'joe']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://scrapsfromtheloft.com/2017/05/06/louis-ck-oh-my-god-full-transcript/\n",
      "http://scrapsfromtheloft.com/2017/04/11/dave-chappelle-age-spin-2017-full-transcript/\n",
      "http://scrapsfromtheloft.com/2018/03/15/ricky-gervais-humanity-transcript/\n",
      "http://scrapsfromtheloft.com/2017/08/07/bo-burnham-2013-full-transcript/\n",
      "http://scrapsfromtheloft.com/2017/05/24/bill-burr-im-sorry-feel-way-2014-full-transcript/\n",
      "http://scrapsfromtheloft.com/2017/04/21/jim-jefferies-bare-2014-full-transcript/\n",
      "http://scrapsfromtheloft.com/2017/08/02/john-mulaney-comeback-kid-2015-full-transcript/\n",
      "http://scrapsfromtheloft.com/2017/10/21/hasan-minhaj-homecoming-king-2017-full-transcript/\n",
      "http://scrapsfromtheloft.com/2017/09/19/ali-wong-baby-cobra-2016-full-transcript/\n",
      "http://scrapsfromtheloft.com/2017/08/03/anthony-jeselnik-thoughts-prayers-2015-full-transcript/\n",
      "http://scrapsfromtheloft.com/2018/03/03/mike-birbiglia-my-girlfriends-boyfriend-2013-full-transcript/\n",
      "http://scrapsfromtheloft.com/2017/08/19/joe-rogan-triggered-2016-full-transcript/\n"
     ]
    }
   ],
   "source": [
    "# Actually request transcripts (takes a few minutes to run)\n",
    "transcripts = [url_to_transcript(u) for u in urls]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# transcripts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Pickle (save) files for later use\n",
    "# We pickle because we want to avoid performing API calls on site multiple times\n",
    "\n",
    "# Make a new directory to hold the text files\n",
    "!mkdir transcripts # anything after ! will be executed by the system command-line (not by the Python kernel) \n",
    "\n",
    "for i, c in enumerate(comedians):\n",
    "    with open(\"transcripts/\" + c + \".txt\", \"wb\") as file:\n",
    "        pickle.dump(transcripts[i], file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load pickled files\n",
    "data = {}\n",
    "for i, c in enumerate(comedians):\n",
    "    with open(\"transcripts/\" + c + \".txt\", \"rb\") as file:\n",
    "        data[c] = pickle.load(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['louis', 'dave', 'ricky', 'bo', 'bill', 'jim', 'john', 'hasan', 'ali', 'anthony', 'mike', 'joe'])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Double check to make sure data has been loaded properly\n",
    "data.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Intro\\nFade the music out. Let’s roll. Hold there. Lights. Do the lights. Thank you. Thank you very much. I appreciate that. I don’t necessarily agree with you, but I appreciate very much. Well, this is a nice place. This is easily the nicest place For many miles in every direction. That’s how you compliment a building And shit on a town with one sentence. It is odd around here, as I was driving here. There doesn’t seem to be any difference Between the sidewalk and the street for pedestrians here. People just kind of walk in the middle of the road. I love traveling And seeing all the different parts of the country. I live in New York. I live in a– There’s no value to your doing that at all.',\n",
       " '“The Old Lady And The Dog”\\nI live– I live in New York. I always– Like, there’s this old lady in my neighborhood, And she’s always walking her dog. She’s always just– she’s very old. She just stands there just being old, And the dog just fights gravity every day, just– The two of them, it’s really– The dog’s got a cloudy eye, and she’s got a cloudy eye, And they just stand there looking at the street In two dimensions together, and– And she’s always wearing, like, this old sweater dress. I guess it was a sweater when she was, like, 5’10”, But now it’s just, like, this sweater And her legs are– her legs are a nightmare. They’re just white with green streaks and bones sticking out. Her legs are awful. I saw a guy with no legs wheeling by, And he was like, “yecch, no thank you. “I do not want those. “I’d rather just have air down here like I have Than to look down at that shit.” I see these two all the time, and I always look at them, And I always think, “god, I hope she dies first.” I do. I hope she dies first, for her sake, Because I don’t want her to lose the dog. I don’t think she’ll be able to handle it. If she dies– If the old lady dies first, I’m not worried about the dog Because the dog doesn’t even know about the old lady. This dog is aware of three inches around his head. He’s living in two-second increments. The second he’s in and the one he just left Is all he knows about, But if he dies, this lady, she’s gonna be destroyed Because this dog is all she has, And I know he’s all she has because she has him. There’s no– If she had one person in her life, She would not keep this piece of shit little dog. Even if just some young woman in her building one morning Were to say, “good morning, gladys,” She’d be like, “good,” And just flush him down the toilet, just– Poom! Poom! The dog just keeps bumping on the drain. Poom! “” she gives up. Ends up just shitting on her dog for the rest of her life. P-p-p! Poom!']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Just looking at data again\n",
    "data['louis'][:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'“The Food Chain”\\nMust be awful to be other kinds of stuff, You know? I’m glad I’m this. This is a pretty good deal when you consider the alternatives. Anything else, any other kind of thing, you’re in the food chain. You’re in the food– we are out of the food chain. I don’t know if we fully appreciate the fact That we got out of the food chain. That is a massive upgrade Because for every other living thing, Life ends by being eaten. That’s how all life ends, is– Every single life except human life, Every life ends like this. Aah! Aah! Aah! We’re the only ones that get to just die old in a bed, Just “I love you. Bye.” I mean, imagine if we were still in the food chain On top of everything else. Imagine if we were in the food chain. It would just be another thing you gotta deal with. You’re already having a bad day. You wake in up in the morning. You’re making breakfast. You burn your toast, and it’s too late to try again, And your kid comes in and says, “beh,” and you’re like, “fine,” And it’s just, “why?” You get a thing in the mail That says that your phone’s different, And your mortgage is another company now. What? I don’t get it! Then you’re walking to work like, “why do I even bother? “shit! Goddamn it! “there’s always fucking cheetahs at the train station! Stop it!” You think that sharks– Do you think that sharks would be embarrassed If they knew that we could all see their fins Sticking out the top of the water? I think they’d be bummed ’cause I don’t think sharks are aware of that at all. I think sharks think they’re slick. They swim around like, “hey, nobody has any idea what’s down here,” And we’re all up here like, “there’s totally a shark right there.” It wouldn’t be fun being a predator, either. If you’re a predator, imagine if every time you’re hungry, You gotta chase a guy Who’s running for his life. You gotta– “come on, dude! Shit, get over here!” And just hold him down and bite his neck Till he shuts the fuck up for a minute So you can just sit there and eat his stomach Before you go to work. That would really suck. That’s why animals just– They go, “let’s do this together, man. “you chase the parents away. I am gonna eat the kid, and you circle back.” That’s why they do it like that. “fuck it. I’m eating babies. This is bullshit.” “grownups ain’t worth the meat.” Whoops, all right. All right. shit. Goddamn it. Idiot. This isn’t a gay voice, by the way. It’s not. Shut up. yeah.'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['louis'][3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning The Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When dealing with numerical data, data cleaning often involves removing null values and duplicate data, dealing with outliers, etc. With text data, there are some common data cleaning techniques, which are also known as text pre-processing techniques.\n",
    "\n",
    "With text data, this cleaning process can go on forever. There's always an exception to every cleaning step. So, we're going to follow the MVP (minimum viable product) approach - start simple and iterate. Here are a bunch of things you can do to clean your data. We're going to execute just the common cleaning steps here and the rest can be done at a later point to improve our results.\n",
    "\n",
    "**Common data cleaning steps on all text:**\n",
    "* Make text all lower case\n",
    "* Remove punctuation\n",
    "* Remove numerical values\n",
    "* Remove common non-sensical text (/n)\n",
    "* Tokenize text\n",
    "* Remove stop words\n",
    "\n",
    "**More data cleaning steps after tokenization:**\n",
    "* Stemming / lemmatization\n",
    "* Parts of speech tagging\n",
    "* Create bi-grams or tri-grams\n",
    "* Deal with typos\n",
    "* And more..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'louis'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's take a look at our data again\n",
    "next(iter(data.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Intro\\nFade the music out. Let’s roll. Hold there. Lights. Do the lights. Thank you. Thank you very much. I appreciate that. I don’t necessarily agree with you, but I appreciate very much. Well, this is a nice place. This is easily the nicest place For many miles in every direction. That’s how you compliment a building And shit on a town with one sentence. It is odd around here, as I was driving here. There doesn’t seem to be any difference Between the sidewalk and the street for pedestrians here. People just kind of walk in the middle of the road. I love traveling And seeing all the different parts of the country. I live in New York. I live in a– There’s no value to your doing that at all.',\n",
       " '“The Old Lady And The Dog”\\nI live– I live in New York. I always– Like, there’s this old lady in my neighborhood, And she’s always walking her dog. She’s always just– she’s very old. She just stands there just being old, And the dog just fights gravity every day, just– The two of them, it’s really– The dog’s got a cloudy eye, and she’s got a cloudy eye, And they just stand there looking at the street In two dimensions together, and– And she’s always wearing, like, this old sweater dress. I guess it was a sweater when she was, like, 5’10”, But now it’s just, like, this sweater And her legs are– her legs are a nightmare. They’re just white with green streaks and bones sticking out. Her legs are awful. I saw a guy with no legs wheeling by, And he was like, “yecch, no thank you. “I do not want those. “I’d rather just have air down here like I have Than to look down at that shit.” I see these two all the time, and I always look at them, And I always think, “god, I hope she dies first.” I do. I hope she dies first, for her sake, Because I don’t want her to lose the dog. I don’t think she’ll be able to handle it. If she dies– If the old lady dies first, I’m not worried about the dog Because the dog doesn’t even know about the old lady. This dog is aware of three inches around his head. He’s living in two-second increments. The second he’s in and the one he just left Is all he knows about, But if he dies, this lady, she’s gonna be destroyed Because this dog is all she has, And I know he’s all she has because she has him. There’s no– If she had one person in her life, She would not keep this piece of shit little dog. Even if just some young woman in her building one morning Were to say, “good morning, gladys,” She’d be like, “good,” And just flush him down the toilet, just– Poom! Poom! The dog just keeps bumping on the drain. Poom! “” she gives up. Ends up just shitting on her dog for the rest of her life. P-p-p! Poom!',\n",
       " '“My Daughter Likes Fish”\\nYou ever flush a pet down the toilet? I had to flush my daughter’s fish down the toilet. I came home, the fish was dead. She wasn’t home from school yet, so I just flushed the fish, And that’s a weird moment, too, ’cause fish live in water. So you put it in the toilet, floats to the bottom, Like, “yeah, he’s dead,” And then you flush, and it looks like he goes, “wait a second! Aw, shit! “I was taking a nap! Jesus! “you gotta be in constant motion To get any respect in this house.” And then my daughter comes home. “why did he die, da–” Come on. What am I gonna say? Why did he die? Because who gives a shit? That’s the reason. That’s the actual reason, is because didn’t matter That he was alive. That’s why he’s dead. He didn’t know his name, and he didn’t love you back. These are the facts about that fish’s life. My daughter likes fish. We took her to the aquarium. I took both my kids to the aquarium in boston, And we were looking at this seal, or sea lion. I don’t know which one. I don’t care. I don’t think they need to be separate things. I really don’t. They don’t care, and we don’t care. There’s, like, three scientists Who give a shit what we call all those things. The scientists could go on TV tomorrow And say, “ok, everybody. “from now on, seals and sea lions and walruses, And–you know what?–Penguins are all seals now,” And we would all be like, “yeah, all right. Fine. Yes. Whatever, man.” Anyway, so we’re looking at this seal/sea lion thing, And he’s looking back at us through the plexiglas, And he’s going– They’re really disgusting in person. Most animals are when you really see them. You ever go to a farm? You think you’re gonna see little, white sheep Going, “baa,” but they’re all fat and shit-colored, And they’re like– Jesus. That thing is awful. Anyway, seals don’t go– They go– My daughter’s like, “what is he saying?” I don’t know! “I’m a slave! Kill me!”',\n",
       " '“The Food Chain”\\nMust be awful to be other kinds of stuff, You know? I’m glad I’m this. This is a pretty good deal when you consider the alternatives. Anything else, any other kind of thing, you’re in the food chain. You’re in the food– we are out of the food chain. I don’t know if we fully appreciate the fact That we got out of the food chain. That is a massive upgrade Because for every other living thing, Life ends by being eaten. That’s how all life ends, is– Every single life except human life, Every life ends like this. Aah! Aah! Aah! We’re the only ones that get to just die old in a bed, Just “I love you. Bye.” I mean, imagine if we were still in the food chain On top of everything else. Imagine if we were in the food chain. It would just be another thing you gotta deal with. You’re already having a bad day. You wake in up in the morning. You’re making breakfast. You burn your toast, and it’s too late to try again, And your kid comes in and says, “beh,” and you’re like, “fine,” And it’s just, “why?” You get a thing in the mail That says that your phone’s different, And your mortgage is another company now. What? I don’t get it! Then you’re walking to work like, “why do I even bother? “shit! Goddamn it! “there’s always fucking cheetahs at the train station! Stop it!” You think that sharks– Do you think that sharks would be embarrassed If they knew that we could all see their fins Sticking out the top of the water? I think they’d be bummed ’cause I don’t think sharks are aware of that at all. I think sharks think they’re slick. They swim around like, “hey, nobody has any idea what’s down here,” And we’re all up here like, “there’s totally a shark right there.” It wouldn’t be fun being a predator, either. If you’re a predator, imagine if every time you’re hungry, You gotta chase a guy Who’s running for his life. You gotta– “come on, dude! Shit, get over here!” And just hold him down and bite his neck Till he shuts the fuck up for a minute So you can just sit there and eat his stomach Before you go to work. That would really suck. That’s why animals just– They go, “let’s do this together, man. “you chase the parents away. I am gonna eat the kid, and you circle back.” That’s why they do it like that. “fuck it. I’m eating babies. This is bullshit.” “grownups ain’t worth the meat.” Whoops, all right. All right. shit. Goddamn it. Idiot. This isn’t a gay voice, by the way. It’s not. Shut up. yeah.',\n",
       " '“Courtyard Confrontation”\\nI live in New York city, and it’s OK there. I live in a nice building. I never lived in a nice place before. When I was growing up, I had no money. I mean, my mom didn’t. Didn’t matter. I was a child. But I’m not used to it ’cause it’s nice. There’s, like, a pretty courtyard With flowers and a fountain with little marble boys pissing. I don’t know. What is it with fountains? Like, all fountain sculptors are pedophiles, basically. You can’t get a fountain made without– “can you make me a fountain?” “yes, I’ll get started right away! “Yes! Yes! It’s finished!” And it’s just little boys pissing on the face Of a Greek god that looks like him a lot. “just piss on me forever!” Anyway, there’s one of those in the courtyard of my building, And my first week in the building about a year ago, I went down to the courtyard for the first time, And I didn’t look too good, you know? It was a Sunday morning. That’s my least presentable hour. There’s a lot of, you know, just stains, Just like, you know, food and me and whatever… And so I’m sitting there. So? Shut up. Ha ha ha! Anyway, but so there I was. I’m sitting on the stone bench of this courtyard And feeling a little out of place. You know, there’s these fancy doormen and stuff, And then there’s this guy looking at me. I notice he’s looking at me from across the courtyard, And he’s all spiffy-looking. He’s got brown shoes, and he’s looking at me like, “” I can tell he was thinking I don’t live in the building. He thinks I just wandered in off the streets And sat in the courtyard. I can tell he’s thinking of coming over And dealing with me on his own, and I’m sitting there thinking, Like, “please do that. Yes. Please, come on, come on. Come on, come on, come on,” And I’m trying to look even more gross, And I’m, like, pulling up my shirt, “” And then I see him go, “no. That’s not gonna do at all,” And he comes over to me, “” and I’m like, “Num num num!” I’m so excited to have this thing, a confrontation Where I’m not wrong at all and he thinks I am. “rrgh!” So he comes over, says, “excuse me, “do you live in this building?” And I said, “no,” ’cause why not start there? I said, “no.” He goes, “well then, what are you doing here?” And I said, “I just need to rest. I’m having a hard time.” He says, “this is private property,” And I said, “well, I don’t really believe in that.” You know, just the worst things I could say from his point of view is basically all the things I was saying, And he goes, “well, if you don’t leave, I’m gonna talk to the doorman.” I was like, “can I just stay, like, five more hours?” So he’s–“no,” and he goes over to the doorman and I see him talking about me to the doorman like this, and then I see the doorman going, “no. That guy lives here. It’s OK,” And the look on his face– mwah mwah! Num num!– It was just so– It was this beautiful cocktail of anger and confusion. It’s like I had invented a new way to hurt somebody’s feelings. That’s how excited I was.',\n",
       " '“Getting What ‘Old’ Is”\\nI’m 45 now, So I’m either halfway through a healthy life or almost done with a not-so-healthy life. I don’t know which one. I won’t know till it’s over where this was. I don’t know how long I’m gonna live. I don’t know. Nobody knows, I guess. I think human life expectancy is pretty good. We get a good run, you know? Some people die early, you know? If I die now, people will be like, “that’s too bad.” If I die five years from now, they’ll be like, “well, all right,” like it’s not even– It’s– Like, as soon as you’re 50, you’re a candidate. You know, there’s no candlelight vigils for 50-year-old guys that die, and you start pushing– You know, some people get to be like, 80s, 90s. You know, there are people that get to, like, 114, and then they’re in the news, and it’s always some old guy, “I met Napoleon.” No, you didn’t, you liar, You oldest liar in the world. But 45, you know, you’re not old yet, But you start having moments Where you kind of start getting what old is, Especially if you didn’t take care of yourself, you know? I have moments where I’m like, “wow, this seems early for this.” Like, this is something that happens to me a lot. I’ll be sitting watching TV or doing nothing, And all of a sudden, I’ll realize, “I need to wipe my ass right now.” “I mean, nothing happened, But I really gotta wipe my ass right now. Right now.” Gotta make trips to the bathroom just to wipe my ass. How does this happen already? I’m 45. Already, my asshole’s just like, “Just– ” My asshole– My asshole’s like the waistband on old pajama bottoms, Just kinda… Loose and ineffectual. My asshole’s like a bag of leaves that nobody tied up. It’s just sitting on the lawn, full and open, Puking leaves onto the grass with every wisp of wind. Some kid kicks it over on his way home From a tough day at middle school. “rats.” That’s a pretty accurate description of my asshole. Here’s another thing about my age right now. If I’m– OK. Say I’m sitting down. If I’m sitting anywhere, which–ha ha ha!– I love sitting so much– I would take sitting and doing nothing To standing and fucking any day. This is way better than coming. This is way better. This is what– At my age, if I’m sitting down and somebody tells me I need to get up and go to another room, I need to be told all the information why first. You gotta explain all that shit to me. “what? Why do I– no. Why, though?” “Your car is being towed right now!” “Well, that’s what happens to me, then. That’s– I accept that.” ’cause getting up is a whole thing. It means first, I have to decide, “do I really want to be alive anymore?” Like, let’s start with that, And then I gotta, “” start kind of rocking to get momentum, “” like I’m trying to get an old Honda out of a snow bank. “shit.”',\n",
       " '“Putting On Socks”\\nThe worst part of my day, worst part of every day of my life, is when I have to put on my socks. Putting on my socks is the worst part of every day, and it always will be. Even if I have a terrible day in the future Where my grandmother is murdered by my other grandmother, If that ever happened to me, the worst part of that day Will be when I put on my socks because– Putting on my socks, that means I have to– Here’s what I have to do. I have to get my hands past my pointed toe. I don’t even know how I do it. I’m sitting here. I don’t know how I do that. It’s ’cause you have to– It’s like folding a bowling ball in half. Soon as I start, I can feel– I’m pushing all the fat up into my vital organs, And I just feel all the systems failing. “beep, beep, beep!” And I have to, like, lay on my back like an eight-year-old And go like this, and I start passing out. And I know other people’s lives are hard. Ha ha ha! I know. I saw an interview on TV with Michael J. Fox, And he has Parkinson’s, and he was describing brushing his teeth, and he said It takes him two hours a day, and he said it’s agony, And I saw this and I thought, “OK. That’s hard.” That is hard, And so is putting on my socks. Sorry, Michael. Doesn’t make it easier to know about your thing. Tell you what. I’ll help you brush your teeth, you help me with the socks.',\n",
       " '“If You’re Older You’re Smarter”\\nI don’t know. I like getting older, though. Life is an education, and if you’re older, you’re smarter. I just believe that. If you’re in an argument with somebody And they’re older than you, you should listen to them. It doesn’t mean they’re right. It means that even if they’re wrong, Their wrongness is rooted in more information than you have. They’ve been there longer. If you’re older, you’re smarter. How many people here, by applause, are 45 or older? 45 or older, applaud. OK. That’s about 60 people out of 2,400. Here’s the interesting thing. There’s way more. I can see you. There’s so many more. This is Arizona. There are way more people in that demographic, But they didn’t clap. Why? ’cause they’re smart. They’re sitting in the dark going, “I don’t have to clap. I don’t have to do anything.” They’re not doing it, and they’re right. You know why? ’cause never identify yourself. Never. Are you crazy? You don’t know why I’m asking. How do you know what– OK. Burn them all. Burn everybody over 45. How do you know I’m not gonna do that? How many people here, by applause, are 40 or younger? 40 or younger, applaud. That’s every time. “Whoo! Whoo!” every time. That is a weird thing to celebrate in a room where you’re not the only people here. That’s like going to a cancer ward– “not me! Whoo! I don’t have it! I don’t have it!” If you’re older, you’re smarter. A 55-year-old garbage man is a million times smarter than a 28-year-old with three ph.D.S, Especially smarter than him, ’cause this idiot Has been thinking about three things for, like, 15 years. He’s worthless. The garbage man is 55. He’s had some experience. Things have happened to him. He went to cape cod one summer. He saw a dead guy floating in the motel pool. He took a bus to Montreal. He got a hand job at a fair… From a miner. I mean a miner. Not a minor, a miner. You understand? A miner. A grown man who works in a mine With dirty hands jacked him off at a county fair. That’s what I’m saying happened to the 55-year-old. You see? He’s had some experience. He knows more. More has happened to him. He’s seen more. He’s seen history. He’s witnessed history, Even if it’s not ancient history. He saw Nixon resign on live television. Me and those few people that clapped, We saw the president of America cry And then quit being the president. That shit was crazy ’cause none of us knew what was gonna happen next. Today people are like, “the president’s kind of disappointing.” Really? Our president wept like an insane person And then got on a helicopter and flew away… And the whole nation just watched him go.',\n",
       " '“Getting Older Makes My Life Better”\\nI like getting older, though, because for me, The kind of guy I am, getting older Makes my life better, you know, Like, my sex life, way better at 45, Look, ’cause this situation is OK at 45. This is not a fun kind of a 17-year-old to be. And some people, their life is better when they’re younger. You know, young dudes, young, skinny dudes, Best life in the world is being a young, skinny dude. They don’t have to do anything. They just show up With a big adam’s apple and a smelly t-shirt. “hey, I’m here for the easy pussy for the rest of my life.” “so where do I– everywhere? OK. Cool.” But for guys like me, this is not a fun youth. It gets better. I’d like to make one of those “it gets better” ads For just dumpy, young guys. We could use a little help, a little encouragement, Just somebody on TV, “listen, man. “I know it’s tough right now. “You’re vaguely heavy with no face. “You have zero value on the sexual marketplace. “You feel invisible to the girls in school because you are, “but it gets better because you’ll all grow up, “and you pretty much look just like this your whole life, “and they don’t. “Their options start running out really fast, “and you’re gonna be there. “As long as you stay relatively employed and washed, “you’re gonna be amazing in your 40s. “You’re gonna be– “you’re gonna be the branch that she can grab “before she hits the ground. “It’s gonna be so great. “It just takes time for her circumstances “to match your looks, but it’s gonna happen. “It’s gonna happen. “When real shit matters, you’re gonna be the sexiest motherfucker in the world.” It’s just time. That’s all it takes. There’s a formula to this. It’s pussy plus time over income squared.',\n",
       " '“Everybody Has Their Time”\\nEverybody has their time. Everybody has their time. I mean, not everybody. There are people out there who there’s just nobody for them. Yep. People like to say things like, “there’s someone for everyone.” Nope! Not at all true, and stop saying it ’cause it’s mean to people who never find anybody. There are millions of people out there who we’ve all unanimously decided, they are light speed ugly and nobody kisses them on the lips, even. Nobody touches their genitals their entire life. They just wash it, and then they die. That’s all that happens, “aww,” and if you’re feeling bad for them, you can go find one and fuck one tomorrow, you can just solve the problem right there with all that kindness in your heart. “Aww.” Well, go fuck one. “Nah.” I didn’t think so. That’s the one way we’re all mean. Nobody does that. Nobody fucks down, nobody. People fuck up or across. Some women fuck down because a guy talked them into that it was up. Some guy, “yeah. No. You should. I’m totally up. Yeah.”',\n",
       " '“Dating Takes Courage”\\nIt’s a weird selection process that we have. Dating really is– It’s how we evolve, is dating. It’s how we choose each other, And dating is a real drag for a lot of people, But I always think it’s a nice thing. You know, when I see a date, I’m always happy when I see a couple on a date ’cause it means people are still trying, you know? You see a couple on a date, It means there’s still courage out there. That takes courage, to go on a date, for both sides, Two very different kinds of courage. The male courage, traditionally speaking, is that he decided to ask. He went up to a random woman who he has no idea If she’s gonna like him or not and he walked up to her terrified. Everything in your body is telling you, “just go the fuck home and jerk off. Don’t do this!” But he walked up and said, “hi. Yes,” And she’s like, “” “no, no, no. A second. Give me a second,” And you try to get through this membrane of, you know– And then, if it works And you say, “you wanna go out sometime?” Sometimes she’ll say yes, and if she says yes, that’s her courage, and the courage it takes for a woman to say yes is beyond anything I can imagine. A woman saying yes to a date with a man is literally insane and ill-advised, and the whole species’ existence counts on them doing it, and I don’t know how they– How do women still go out with guys when you consider the fact that there is no greater threat to women than men? We’re the number-one threat to women. Globally and historically, we’re the number-one cause of injury and mayhem to women. We’re the worst thing that ever happens to them. That’s true. You know what our number-one threat is? Heart disease. That’s the whole thing. That’s it, just our own heart going, “dude, I can’t– “you can’t keep doing this. I told you three strokes ago that this is not smart,” But women still– “yeah. I’ll go out with you alone at night.” What are you, nuts? “I’ll get in your car with you with my little shoulders. Hi. Where are we going?” “To your death, statistically.” How do they still do it? If you’re a guy, try to imagine that you could only date a half-bear, half-lion, And you’re like, “I hope this one’s nice. I hope he doesn’t do what he’s going to do.”',\n",
       " '“First Date”\\nI always– I love when I see a date. You know, when you see a date, you can tell it’s the first date ’cause of the way they’re walking together And she’s looking up at his face trying to figure him out, And he’s just a mess. A guy on a first date just has no actual personality. He’s just a mishmash of different kinds of dudes for a couple of seconds each, just anything, no cohesive– Just like a ransom note cut out of a lot of magazines, just, you know, “well… “ha ha! “well, yes. I think so, too. Ha ha! Rrgh…” Whatever, just like a blind dick in space just thrusting in infinite directions Hoping to find pay dirt at someplace, And then sometimes you see a date that’s later down the line and something has happened. There’s something that happens in a date that I never get to witness ’cause women do this. They get to do it inside. They get to just decide quietly, “I’m gonna let him fuck me.” They just get to decide. Something he says, and she’s like, “that was good. He’s gonna fuck me later,” And he has no idea. He’s still, like, trying all this shit. He still has no idea he’s already in there.',\n",
       " '“Tits”\\nGuys are– We love women a lot– all men do– And we just look at you. That’s what I do. I just look at women. I just–“” Like they’re, you know, cakes in windows. I just–“” I was walking in New York once, and there’s these two very cute women walking behind me, and I was trying to walk slowly so I could hear what they were saying to each other, you know, ’cause they were cute, so I wanted to hear them, like that’s gonna help me in any way, to hear their– “don’t you wish the guy walking in front of us would squeeze our tits for, like, one second?” OK. Here. Here I go. Thank you. It’s really a flaw in men that we would all do that. If you’re a woman, you could ask any guy on planet earth, “could you squeeze my tit for one second?” And 100% of us will go, “yes, of course.” That doesn’t matter. I could be doing open heart surgery. “yeah. Ok.” Beep! “don’t worry. He’s not your tits. Don’t worry about it.” I don’t know why we love tits so much. Some people say it’s because we breastfeed, but so do women, Or, you know, baby women. Not grown-up women, don’t usually– You don’t see, like, a 68-year-old woman, Like, a stately look– You know, like Sigourney weaver, Like, sucking milk out of a young woman’s tit. “thank you, Deborah. I’ll see you tomorrow at 2:00.” It’s not usually the thing, Is the elderly breastfeeding from the young, Except for at the end of “the grapes of wrath,” Which I don’t mean to ruin that book for you, But you should have read it by now. I don’t know if you read “the grapes of wrath,” But that’s how it ends, with an old, dying man Sucking milk out of a young girl’s tits, And then the book is over, and you’re like, “Jesus! What happened at the end there? That’s crazy.” There’s no other book in that genre. There’s no dense, historic classic That ends with a weird, porny paragraph at the very– “and then Anna Karenina shat on his chest.” “holy moly with that!” “the end.” “my god! “that is a violent shift in tone at the end of that book. I’ve been reading this book for three months.” But we do, we love tits, And you always know a tit. You always know a tit. You know, like you ever been in a crowded place– Like a subway or like, you know, a sports stadium– And you’re smooshed in with other people And your elbow touches a tit behind you, you’re like, “that’s a tit. I just touched it. I know that was a tit. I know it!” Because the sensitivity of the male elbow To tit flesh specifically is unbelievable, just to tits, ’cause you could drive a tack in there, I’m not gonna feel it, But a tit– Through a shirt and a sweater and a jacket And her jacket, sweater, shirt and bra, somehow… “it’s a tit! I touched a tit! “I touched a tit! I touched one tit. One tit.” That’s rare, to touch one tit. It’s like a four-leaf clover. Usually, you touch two. The only time you touch one tit is when it was an accident Or you didn’t have permission, But otherwise… When tit access is granted, It’s usually good for two tits at a time. It’s, once you’re on one– You really have to screw up really badly and quickly To lose tit access between tits one and two. You must have said something really dumb on the first tit That you didn’t get the second one. “yeah, it’s like your mom’s dirty whore tit.” “why? What? I said I liked it. “that’s what I meant, is that I like it. “I like your mom’s dirty whore tit, And I like yours,” And if it’s up to the guy, we’re gonna touch both tits. No guy touches a tit and then goes, “you know what? “I’m good with the one. That’s fine for me. Everything in moderation.” We’re gonna touch both, Even if something terrible happens in the middle of– “yeah–” “there’s a bomb!” “shit! Come on, let’s get– “come on! Let’s get your tits out of here!”',\n",
       " '“Divorce”\\nI’ve seen a few tits. I’ve seen–I don’t know– 48 maybe. I don’t know. I had my history in my life, you know? I’ve been divorced for five years, And it’s been the best part of my life, being divorced, Easily my favorite part of my life. I love being divorced. Every year has been better than the last. That is the only time I can say that about, and by the way, I’m not saying don’t get married. If you meet somebody, fall in love and get married, And then get divorced because that’s the best part. It’s the best part. Marriage is just like a larva stage for true happiness, Which is divorce, Because you just let go and everything’s fine now. Divorce is forever. It really actually is. Marriage is for how long you can hack it, But divorce just gets stronger like a piece of oak. Nobody ever says, ” my divorce is falling apart. It’s over. I can’t take it.” And again, if you’re in a good marriage, stay in it. If you’re in the best marriage ever, stay in it. I’m just saying, if you got out, it would be better. That’s just a fact. I’m sorry. I’m sorry, but it’s true. Everything’s better. My ex-wife and I, This is the best part of our knowing each other. We’re good co-parents because we live apart and we’re friends. Our kids go to her half the week, They come to me half the week, and I’m a good father. I’m an attentive, focused, and responsible father. Do you know why? ‘Cause I get to say good-bye to these kids every week. Are you shitting me? It’s like every parent’s fantasy. Who can’t be a good father for half of every week? No matter how bad it gets, every Wednesday, I get to go, “good-bye, girls. “Daddy’s gonna go upstairs and pour whiskey all over his naked body right now.” “I’m gonna lay in my own filth until two seconds before you come back here.” That’s why I’m such a good dad.',\n",
       " '“Videoing Your Kids”\\nMy daughter was having a dance thing at her school. They had this big dance. Anyway, we all went, all the parents, And everybody’s there, and everybody’s got their phone, Every single parent. It was an amazing thing to watch ’cause kids are dancing And every parent is standing there like this. Every single person was blocking their vision Of their actual child with their phone, and the kids– I went over by the stage and the kids– There’s people holding iPads in front of their faces. It looked like we’re all in the witness protection program. Like, the kids can’t see their parents, And everybody’s watching a shitty movie Of something that’s happening ten feet– Like, look at your fucking kid. The resolution on the kid is unbelievable if you just look. It’s totally HD. Why are you taping this? You’re never gonna watch it. In a million years, you’re not gonna watch videos Of your kids doing shit you missed The first time it happened. You don’t watch it. You just put it on Facebook. “here, you watch it. I wanna take a nap now.” And then you get to read all the comments. “my god! “it’s so cute! Ngaah!” And guess what? They’re not watching it, either. They’re not watching the video. These kids are dancing for no one. Nobody watches the videos on your Facebook. They see the first frame of a kid and they go, “that’s very nice. OK. Back to this.” Nobody’s watching your kids’ videos on Facebook, I promise you. I’ll prove it to you. Next time you tape your kid’s dance, Tape one second of it and then add 20 minutes of just your own asshole. Just go in the bathroom and just record your own anus opening and closing for 20 minutes. Tack it onto your kid dancing for a second. Put that on Facebook. Everybody will write the same thing. “That’s adorable! I think I see a future star!”',\n",
       " '“Life Is A Good Deal”\\nDon’t tape shit on your– Life is short. Life is very short. I like life. I like it. I feel like even if it ends up being short, I got lucky to have it ’cause life is an amazing gift When you think about what you get with a basic life, Not even a particularly lucky life or a healthy life. If you have a life, that’s an ama– Here’s your boilerplate deal with life. This is basic cable, what you get when you get life. You get to be on earth. First of all–my god– what a location. This is earth, and for trillions of miles in every direction, It fucking sucks so bad. It’s so shitty that your eyes bolt out of your head ’cause it sucks so bad. You get to be on earth and look at shit, As long as you’re not blind or whatever it is. You get to be here. You get to eat food. You get to put bacon in your mouth. I mean, when you have bacon in your mouth, It doesn’t matter who’s president or anything, You just–“” Every time I’m eating bacon, I think, “I could die right now,” and I mean it! That’s how good life is. You get to– you get to fuck. That’s free if you’re smart. That comes with. That’s part of the deal. Where else are you gonna get that deal? You get to put your dick in there and go in and out, Pretty good, And if you’re a woman, you get to just lay back And just have a dick just shoving in and out of you awkwardly Anytime you want, anytime you want. If you’re a gay man, you get to just fill your boyfriend’s ass with your dick, just fill it all the way to the balls, And it’s nice and warm and tight in there, And he’s your buddy. If you’re a lesbian, You get to do all the stuff they’re doing, and… It’s a great deal. You get to eat. You get to fuck. You get to read “to kill a mockingbird.” It’s a great life. So, you know, I’m not worried about it ending.',\n",
       " '“Behind The Wheel”\\nIt’s pretty good, and I’ve wasted a lot of time Just being angry at people I don’t know. You know, it’s amazing how nasty we can get as people, Depending on the situation. Like, most people are OK as long as they’re OK, But if you put people in certain contexts, they just change. Like, when I’m in my car, I have a different set of values. I am the worst person I can be when I’m behind the wheel, which is when I’m at my most dangerous. When you’re driving, That’s when you need to be the most compassionate And responsible of any other time in your life ’cause you are fucking driving a weapon amongst weapons, and yet it’s the worst people get, and I am the worst. One time, I was driving, and there was a guy ahead of me, And he kind of–I don’t know– sorta drifted into my lane for a second, and this came out of my mouth. I said, “worthless piece of shit.” I mean, what an indictment. What kind of a way is that to feel about another human being? “Worthless piece of shit”? That’s somebody’s son. And things I’ve said to other people. I was once driving, and some guy in a pickup truck did– I don’t remember, even– And I yelled out my window, I said, “hey, fuck you!” Where outside of a car is that even nearly OK? If you were in an elevator And you were, like, right next to a person’s body And, whatever, like, he leaned into you a little bit, Would you ever turn right to their face and go, “hey, fuck you!”? “worthless piece of shit!” No. Literally zero people would ever do that, but put a couple of pieces of glass and some road between you, there’s nothing you would not say to them. “I hope you die!” I said that to a person. “I hope you die!” Why? ’cause you made me go like this for half a second of my life. You tested my reflexes, and it worked out fine! So now I hope your kids grow up motherless!” I mean, what am I capable of? I’d like to think that I’m a nice person, But I don’t know, man.',\n",
       " '“If Murder Was Legal”\\nA lot of it is context. There’s a lot of things I wonder if the world was different. Like, if murder was legal, I might have killed a few people. I don’t know. I’d love to think, “I would never do that,” But we really need the law against murder For one simple reason. The law against murder Is the number-one thing preventing murder. We’d like to think it’s ’cause, Like, “I would never do that.” No. It’s ’cause it really sucks getting caught murdering, a lot. If murder was legal, or just a misdemeanor– Like you get a thing in the mail– “shit, they had a camera there. Well.” If murder was legal, there would be so much murder. Regular people would murder. Murderers would murder even more, And then really nice, sweet people Would murder a few people, But nobody would murder no people. You wouldn’t trust somebody who didn’t murder If murder was legal. You wouldn’t like them. “I never killed anybody.” “OK. Nice meeting you. Yecch, what a creep. “I mean, not even a hooker? Live a life. “what’s wrong with that guy? He’s like a Mormon or some shit. I hate those guys.” “I think he’s nice.” “shut up, Janet.” They would just– If murder was legal, there would be a lot of murder. Children would behave very differently Because mostly parents would be murdering their own kids, That’s mostly what would happen if murder was legal. You know, you’d go to a mall, there’d be, Like eight, different moms in the mall just– “I told you to stop it! You didn’t listen to me!” Just–pbbt! There’d be just– You’d be stepping over dead kids. There’d be, like, a new problem. “you have to clean up your kids “when you kill them ’cause it’s gross. “it’s bad for the environment. “if you murder your child in a public place, “please use one of the red bags that are in the dispensers “every three feet of America. “put your murdered child in the red bag “with a logo of a murdered kid on it “next to the other logo that tells you “not to let your alive kid play with the plastic bag “because they might suffocate, In which case you could just leave them in the bag.” That is a whole bunch of horrible thoughts Right in a row, right in a row. That is a compressed area of bad thought.',\n",
       " '“Of Course, But Maybe”\\nYou know, you have your bad thoughts. Hopefully you do good things. Everybody has a competition in their brain of good thoughts and bad thoughts. Hopefully, the good thoughts win. For me, I always have both. I have, like, the thing I believe, the good thing– That’s the thing I believe– and then there’s this thing, and I don’t believe it, but it is there. It’s always this thing and then this thing. It’s become a category in my brain That I call, “of course… But maybe…” I’ll give you an example, OK? Like, of course children who have nut allergies need to be protected, of course. We have to segregate their food from nuts, Have their medication available at all times, and anybody who manufactures or serves food needs to be aware of deadly nut allergies, of course, but maybe… Maybe if touching a nut kills you, you’re supposed to die. Of course not. Of course not. Of course not. Jesus. I have a nephew who has that. I’d be devastated if something happened to him, but maybe… maybe if we all just do this for one year, we’re done with nut allergies forever. No. Of course not. Of course, if you’re fighting for your country and you get shot or hurt, it’s a terrible tragedy, of course, of course… But maybe… Maybe if you pick up a gun and go to another country and you get shot, it’s not that weird. Maybe if you get shot by the dude you were just shooting at, It’s a tiny bit your fault. Of course, of course slavery is the worst thing that ever happened. Listen, listen. You all clapped for dead kids with the nuts. For kids dying from nuts, you applauded, So you’re in this with me now. Do you understand? You don’t get to cherry-pick. Those kids did nothing to you. Of course, of course slavery is the worst thing that ever happened. Of course it is, every time it’s happened– Black people in America, Jews in Egypt. Every time a whole race of people has been enslaved, it’s a terrible, horrible thing, of course… But maybe… Maybe every incredible human achievement in history was done with slaves. Every single thing where you go, “how did they build those pyramids?” They just threw human death and suffering at them until they were finished. How did we traverse the nation with the railroad so quickly? We just threw Chinese people in caves and blew ’em up and didn’t give a shit what happened to them. There’s no end to what you can do when you don’t give a fuck about particular people. You can do anything. That’s where human greatness comes from, is that we’re shitty people, that we fuck others over. Even today, how do we have this amazing microtechnology? Because the factory where they’re making these, They jump off the fucking roof ’cause it’s a nightmare in there. You really have a choice. You can have candles and horses and be a little kinder to each other or let someone suffer immeasurably far away Just so you can leave a mean comment on YouTube while you’re taking a shit.',\n",
       " 'Thanks a lot, folks. You guys were great. Thank you very, very much. Thank you.']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Notice that our dictionary is currently in the format of key: comedian, value: list of text\n",
    "next(iter(data.values()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In the dict above, the values are actually a list of text\n",
    "# We are going to change this one giant string of text. New format will be {key: comedian, value: string}\n",
    "def combine_text(list_of_text):\n",
    "    '''Takes a list of text and combines them into one large chunk of text.'''\n",
    "    combined_text = ' '.join(list_of_text)\n",
    "    return combined_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combine it!\n",
    "data_combined = {key: [combine_text(value)] for (key, value) in data.items()}\n",
    "\n",
    "# Nwamaka's notes for unpacking how the concise one-liner above creates a new dictionary\n",
    "# data.items()          # makes a list of (key, value) tuples..... [('key', 'value'), ('key', 'value')]\n",
    "# for key, value        # iterate on those tuples\n",
    "# combine_text(value)   # creates a single string from the value\n",
    "# key:                  # make a new entry in the new dictionary\n",
    "\n",
    "# An alternative way to write it is\n",
    "###########################################\n",
    "#\n",
    "# data_combined = dict()  # empty dictionary\n",
    "# for key, value in data.items():\n",
    "#     data_combined[key] = combine_text(value)\n",
    "#\n",
    "###########################################"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>transcript</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ali</th>\n",
       "      <td>Ladies and gentlemen, please welcome to the stage: Ali Wong! Hi. Hello! Welcome! Thank you! Thank you for coming. Hello! Hello. We are gonna have ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>anthony</th>\n",
       "      <td>Thank you. Thank you. Thank you, San Francisco. Thank you so much. So good to be here. People were surprised when I told ’em I was gonna tape my s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bill</th>\n",
       "      <td>[cheers and applause] All right, thank you! Thank you very much! Thank you. Thank you. Thank you. How are you? What’s going on? Thank you. It’s a ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bo</th>\n",
       "      <td>Bo What? Old MacDonald had a farm E I E I O And on that farm he had a pig E I E I O Here a snort There a Old MacDonald had a farm E I E I O [Appla...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dave</th>\n",
       "      <td>This is Dave. He tells dirty jokes for a living. That stare is where most of his hard work happens. It signifies a profound train of thought, the ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hasan</th>\n",
       "      <td>[theme music: orchestral hip-hop] [crowd roars] What’s up? Davis, what’s up? I’m home. I had to bring it back here. Netflix said, “Where do you wa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jim</th>\n",
       "      <td>[Car horn honks] [Audience cheering] [Announcer] Ladies and gentlemen, please welcome to the stage Mr. Jim Jefferies! [Upbeat music playing] Hello...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>joe</th>\n",
       "      <td>[rock music playing] [audience cheering] [announcer] Ladies and gentlemen, welcome Joe Rogan. [audience cheering and applauding] What the fuck is ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>john</th>\n",
       "      <td>All right, Petunia. Wish me luck out there. You will die on August 7th, 2037. That’s pretty good. All right. Hello. Hello, Chicago. Nice to see yo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>louis</th>\n",
       "      <td>Intro\\nFade the music out. Let’s roll. Hold there. Lights. Do the lights. Thank you. Thank you very much. I appreciate that. I don’t necessarily a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mike</th>\n",
       "      <td>Wow. Hey, thank you. Thanks. Thank you, guys. Hey, Seattle. Nice to see you. Look at this. Look at us. We’re here. This is crazy. It’s insane. So ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ricky</th>\n",
       "      <td>Hello. Hello! How you doing? Great. Thank you. Wow. Calm down. Shut the fuck up. Thank you. What a lovely welcome. I’m gonna try my hardest tonigh...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                    transcript\n",
       "ali      Ladies and gentlemen, please welcome to the stage: Ali Wong! Hi. Hello! Welcome! Thank you! Thank you for coming. Hello! Hello. We are gonna have ...\n",
       "anthony  Thank you. Thank you. Thank you, San Francisco. Thank you so much. So good to be here. People were surprised when I told ’em I was gonna tape my s...\n",
       "bill     [cheers and applause] All right, thank you! Thank you very much! Thank you. Thank you. Thank you. How are you? What’s going on? Thank you. It’s a ...\n",
       "bo       Bo What? Old MacDonald had a farm E I E I O And on that farm he had a pig E I E I O Here a snort There a Old MacDonald had a farm E I E I O [Appla...\n",
       "dave     This is Dave. He tells dirty jokes for a living. That stare is where most of his hard work happens. It signifies a profound train of thought, the ...\n",
       "hasan    [theme music: orchestral hip-hop] [crowd roars] What’s up? Davis, what’s up? I’m home. I had to bring it back here. Netflix said, “Where do you wa...\n",
       "jim      [Car horn honks] [Audience cheering] [Announcer] Ladies and gentlemen, please welcome to the stage Mr. Jim Jefferies! [Upbeat music playing] Hello...\n",
       "joe      [rock music playing] [audience cheering] [announcer] Ladies and gentlemen, welcome Joe Rogan. [audience cheering and applauding] What the fuck is ...\n",
       "john     All right, Petunia. Wish me luck out there. You will die on August 7th, 2037. That’s pretty good. All right. Hello. Hello, Chicago. Nice to see yo...\n",
       "louis    Intro\\nFade the music out. Let’s roll. Hold there. Lights. Do the lights. Thank you. Thank you very much. I appreciate that. I don’t necessarily a...\n",
       "mike     Wow. Hey, thank you. Thanks. Thank you, guys. Hey, Seattle. Nice to see you. Look at this. Look at us. We’re here. This is crazy. It’s insane. So ...\n",
       "ricky    Hello. Hello! How you doing? Great. Thank you. Wow. Calm down. Shut the fuck up. Thank you. What a lovely welcome. I’m gonna try my hardest tonigh..."
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can either keep it in dictionary format or put it into a Pandas DataFrame\n",
    "# I will put it into a Pandas DataFrame\n",
    "import pandas as pd\n",
    "pd.set_option('max_colwidth',150)\n",
    "\n",
    "data_df = pd.DataFrame.from_dict(data_combined).transpose()\n",
    "data_df.columns = ['transcript']\n",
    "data_df = data_df.sort_index()\n",
    "\n",
    "# Print the Corpus\n",
    "data_df "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Ladies and gentlemen, please welcome to the stage: Ali Wong! Hi. Hello! Welcome! Thank you! Thank you for coming. Hello! Hello. We are gonna have to get this shit over with, ’cause I have to pee in, like, ten minutes. But thank you, everybody, so much for coming. Um… It’s a very exciting day for me. It’s been a very exciting year for me. I turned 33 this year. Yes! Thank you, five people. I appreciate that. Uh, I can tell that I’m getting older, because, now, when I see an 18-year-old girl, my automatic thought… is “Fuck you.” “Fuck you. I don’t even know you, but fuck you!” ‘Cause I’m straight up jealous. I’m jealous, first and foremost, of their metabolism. Because 18-year-old girls, they could just eat like shit, and then they take a shit and have a six-pack, right? They got that-that beautiful inner thigh clearance where they put their feet together and there’s that huge gap here with the light of potential just radiating through.\\nAnd then, when they go to sleep, they just go to sleep. Right? They don’t have insomnia yet. They don’t know what it’s like to have to take a Ambien or download a Meditation Oasis podcast to calm the chatter of regret and resentment towards your family just cluttering your mind. They have their whole lives ahead of them. They don’t have HPV yet. They just go to sleep in peace at night. Everybody has HPV, OK? Everybody has it. It’s OK. Come out already. Everybody has it. If you don’t have it yet, you go and get it. You go and get it. It’s coming. You don’t have HPV yet, you’re a fucking loser, all right? That’s what that says about you. A lot of men don’t know that they have HPV, because it’s undetectable in men. It’s really fucked up. HPV is a ghost that lives inside men’s bodies and says, “Boo!” in women’s bodies. My doctor told me that I have one of two strains of HPV. Either I have the kind that’s gonna turn into cervical cancer… …or I have the kind where my body will heal itself. Very helpful, this doctor, right? So, basically, either I’m gonna die… or you’re in the presence of Wolverine, bitches. We’ll find out. Um, I can also tell that I’m getting older, because my Kindle is turning into a self-help library. I’m not interested in books like Fifty Shades of Grey, OK? I’m interested in The Life-Changing Magic of Tidying Up. Yes. Yes, that’s right, how to declutter my home to achieve inner peace and my optimum level of success. That’s what your 30s is all about. How can I turn this shit around? I’m a horrible person, I’m not happy with where I am, how can I turn this shit around? Help me, Tony Robbins, help me!\\nI have a hoarding problem, which I’m hoping is the center of all of my other problems. I’m hoping that if the hoarding goes away, the HPV will also disappear. I have a hoarding problem because my mom is from a third world country and she taught me that you can never throw away anything, because you never know when a dictator’s gonna overtake the country and snatch all your wealth. So, you better hold onto that retainer from the third grade, ’cause it might come in handy as a shovel when you’re busy stuffing gold up your butt and running away from the Communists.\\nThe last time I was at home in San Francisco, I was trying to help her get rid of shit. Don’t ever do that with your mom. It was like the worst experience of my life. It was so emotional. We were screaming and fighting and yelling and it all came to a climax when she refused to let go of a Texas Instruments TI-82… manual. The manual. She don’t even know… where the calculator is. Those of you under 25 probably don’t know what that calculator is. It was this calculator that bamboozled my generation. We were all required to buy it when we were in eight grade. It cost like $200. And everybody thought it was like this Judy Jetson’s laptop from the future. All because what? It could graph. It was like the Tesla of my time. And my mom got so emotional about the manual and she was like, “You never know when you might need this.” And I was like, “But… I do know… that I’m gonna have to clean all this shit up when you die.” “And I’m not trying to be a procrastinator anymore. Because according to Deepak-Oprah, that’s not the way for me to achieve my optimum level of success.” I grew up a lot this past year. Uh, this past year I also got married. Yeah. To a man who now has HPV. Very lucky guy. He gave me something. I gave him something. That will also last forever. No, really. I’m the lucky girl, because before him, I dated a lot of losers. Lots of losers. A lot of skaters. You wanna be a grown-ass woman, stop dating skaters. Stop dating skaters unless you wanna wake up on a mattress in a kitchen. They’re sexy on the outside, malt liquor on the inside. Horrible. But my husband, I first met him at this wedding and, uh, he’s– he’s much better looking than me, he’s way out of my league, and I saw him and I was like, “Oh, my God, who is that?” And the first thing I learned about him was that, at the time, he was attending Harvard Business School. And I was like, “Oh, my God, I’m gonna trap his ass.” “Going to trap his ass!” And I trapped his ass initially by not kissing him until the fifth date, which is a very unusual move on my part. But I did it on purpose, because I knew that he was a catch. So I was like, “All right, Ali, you gotta make this dude believe that your body is a secret garden.” When, really, it’s a public park… …that has hosted many reggae fests… …and has even accidentally let… two homeless people inside. I thought they were hipsters, OK? That store Urban Outfitters has made things very confusing… for my generation. You homeless or you a hipster? Is that beard for fashion or for warmth? It happened to… It happened in San Francisco, when I was living there, and I saw this guy in broad daylight and we had, like, we had… We had so much chemistry. He was like, “Hey, wassup?” I was like, “Wassup?” And we– The next thing I knew, we were getting busy in the back of my Volvo. And then after we were done, he was like, “Hey, can you drop me off?” I was like, “Where?” He was like, “At the park.” And I dropped him off at Golden Gate Park and watched him run into the middle with all his other homeless friends, and I was like, “Oh, no!” “I just fucked a homeless dude! Again!” My husband is Asian. Which a lot of people are shocked by, because, usually, Asian-American women who, like, you know, wear these kinda glasses and have a lot of opinions, they like to date white dudes. You go to any hipster neighborhood in a major city in America and that shit is turning into a Yoko Ono factory. It’s… too much. I don’t know what’s wrong with these bitches. I get it, you know, because being with a white dude you feel very… You feel very picturesque when you’re with a white dude, you know. You feel like you’re in a Wes Anderson movie or something. And you know, white dudes, they teach you about a lot of cool stuff like voting and recycling, and disturbing documentaries. They introduce you to cool stuff like that and it’s very, you know, it’s hot hookin’ up with a white dude. I mean, nothing makes me feel more powerful than when a white dude eats my pussy. Oh, my God. I just feel like I’m absorbing all of that privilege and all of that entitlement… …you know, just right there, through the money hole and just… And then also, he’s so vulnerable down there. I’m, like, “I could just crush your head at any moment, white man! I could just kill you right now! Crush those brains! Colonize the colonizer!” You know?\\nBut I think that for marriage, it can be nice to be with somebody of your own race. The advantage is that you get to go home… and be racist together. You get to say whatever you like! You don’t gotta explain shit. My husband, half-Filipino, half-Japanese. I’m half-Chinese and half-Vietnamese. And we spend 100 percent of our time shitting on Korean people. It’s… amazing. It’s what love is built on, you know?\\nMy last boyfriend was Cuban and his family would shit on Mexican people all the time. And I was like, “Hold it. You guys aren’t Mexican?” Asian-American men are very underrated. I don’t know why people don’t go for them. They’re the sexiest. Asian men are the sexiest. They got no body hair from the neck down. It’s like making love to a dolphin. Oh, my God. It’s so smooth, just like a slip and slide. Just black fish, Tilikum, all up in my bed every night, you know? Ooh-wee. You mess with a Jewish dude and your body is all fucked up afterwards. It’s all red and inflamed and you’re like, “I did not ask to be exfoliated today.” “This is the last time I go on J-date, more like loofah date. Thanks for the rug burn, Avi.” And then Asian men, no body odor. None. They just smell like responsibility. That’s where the umami flavor comes from.\\nI think my husband and I have a huge unspoken understanding, uh, between each other, because he’s half-Filipino and half-Japanese and I’m half-Chinese and half-Vietnamese. So, we’re both half-fancy Asian… …and half-jungle Asian. Yeah! You guys know the difference. The fancy Asians are the Chinese, the Japanese. They get to do fancy things like host Olympics. Jungle Asians host diseases. It’s… It’s different. But he grew up on the East Coast, going to private school, playing lacrosse, uh, you know, learning Latin and playing chess and rugby. He grew up like Filipino Carlton, OK? So, he didn’t know anything about Vietnamese people until he met me. And on one of our first dates, he took me to this restaurant on the west side of Los Angeles called Pho Show. He was like, “It’s authentic Vietnamese. I read about it on Yelp!” I was like, “It’s not authentic, OK?” You can tell, first and foremost, by the name, ’cause it don’t got a number in it. Second of all, you can tell by the bathroom. If it was legit, the bathroom would double as a supply closet. When I pee, I need to see ten gallons of bleach, an ATM machine and a grandma with glaucoma napping in the corner. And the wait staff here is too nice. We need to leave this restaurant deaf and emotionally abused. I grew up going to private school, too. Him and I are both total, like, private school Asians. We both are big hippies, too. We like to backpack through Southeast Asia. We like to do yoga. We do ayahuasca ceremonies. We do silent meditation retreats. That’s right, we pay $800 to shut up for a weekend. We do shit like that. Uh, we eat gluten-free, which means we eat all that bread that tastes like free-range Chewbacca. We eat that lesbian bread that’s like… …a thousand percent of your daily fiber… and 20 percent spoken word poetry. When you eat it, you queef a shitty poem about… …supporting Caitlyn Jenner or whatever. And so, it’s funny, right, because he’s Asian, too. But sometimes, all of this hippy-dippy shit we do… makes me feel like we are white people doing an impression of Asian people. Like, we have these Chinese scrolls up on the wall… and neither of us know what the fuck they mean. We’re like, “Oh, that seems to go very well with our Buddha piggy bank from Pier 1 Imports. That seems to be providing some good feng shui for the house. Him and I had been dating for four years and I– I just had this sneaking suspicion that he was gonna propose… because… I had been pressuring him to do it. So, you know, I just had this wacky women’s intuition. That’s how proposals really work, OK? A woman has to incept the idea into the man’s head. First passively and then if he doesn’t get the message, extremely aggressively. You gotta threaten to leave without ever actually leaving, because you know that you’re too old and it’s too late to go back out there and find a new man and start the whole manipulation cycle all over again. So, you’re like, “I’m just gonna stick with this dude, focus on trapping this dude, and just nag the shit outta him until he becomes weak and caves in and gets fed up and is like, “Shut the fuck up! Fine, will you marry me?” And then afterwards, the woman is always, like, “Oh, my God! He proposed!” “It came outta nowhere. And look, he got me the exact ring I wanted. How did he know? Maybe he saw it on my Pinterest page or something… that I sent to my best friend, that I told her to send to him every day.” Let me tell you something. If a man has a Pinterest page… he’s probably Pinterested in men. We got engaged on a Saturday. I bought my wedding dress the following Tuesday… because I had tried it on in 2012. I was ready. I was ripe. I was rotten. I need to be made into banana bread. That’s how rotten I was. People are always very surprised at how, off-stage, with my husband, I’m a completely different person. You– Like, you would not recognize my personality at all with him. With him, I’m very soft, and, like, very nurturing and very domestic. We’ve been together now for five years, and for five years, I’ve packed his lunch every single day. Yeah. Yes. Yes. Yes. I did that so that he’d become dependent on me. ‘Cause he graduated from Harvard Business School, and I don’t wanna work anymore. I don’t. I straight up don’t wanna work anymore. I don’t feed him out of the goodness of my heart. I do it as an investment in my financial future. ‘Cause I don’t wanna work anymore. I’ve been reading that book by Sheryl Sandberg, she’s the C.O.O. of Facebook, and she wrote that book that got women all riled up about our careers. Talking about how we as women should challenge ourselves to sit at the table and rise to the top. And her book is called Lean In. Well, I don’t wanna lean in, OK? I wanna lie down. I want to lie the fuck down. I think feminism is the worst thing that ever happened to women. Our job used to be no job. We had it so good. We could have done the smart thing, which would have been to continue playing dumb for the next century and be like, “We’re dumb women. We don’t know how to do anything. So, I guess we better just stay at home all day and eat snacks and watch Ellen.” “‘Cause we’re too stupid to have any real responsibility.” And then, all these women had to show off and be like, “We could do it! We could do anything.” “Bitch, shut up!” “Don’t tell them the secret.” They ruined it for us, and now we’re expected to work. When I hear the phrase, “Double-income household,” I wanna throw up. A lot of women get very upset with me about those comments. And they’re like, “But, Ali, we have so many more options now.” Oh, you don’t think we had a lot of options when our day was free? Unscheduled, unsupervised, and most importantly, sponsored? Do you know how much shittier food tastes when you know you have to earn it?\\nA lot of my friends, when we walk around together, they’ll get very judgmental about housewives that we’ll see on the street. And they’ll be like, “Look at that fucking housewife. Not doing anything. Look at that housewife, just walking around all day, getting massages in her Lululemon pants.” I’m like, “That bitch is a genius.” “She’s not a housewife, she’s retired.”\\nI do write for Fresh Off the Boat on ABC. Yeah. Which is… It’s a great show. I love it a lot. I love my co-workers. It’s a great writing staff and in terms of day jobs, it’s probably one of the best you could ask for, but I still gotta work at a office every day. Which means I gotta shit in a office every day. Housewives, they don’t gotta shit in a office. Housewives get to shit in their house. Skin to seat. They don’t gotta use that horrible toilet paper cover. They don’t gotta… …ten times a day, every day… like you’re about to eat a sad-ass meal. They don’t gotta do that. They don’t gotta use that one-ply toilet paper, that office toilet paper, that they purposely make difficult to pull out. They try to ration me with their communist toilet paper that’s not even effective. It basically just dehydrates your butt hole. It’s basically like wiping your butt with the desert. I literally spat on my toilet paper two days ago, to try to make a MacGyver baby wipe, to moisten it, and then it backfired ’cause my fingers broke through and digitally stimulated more doo doo to come out, and then I had to start all over again. And you can never finish wiping at work because you always feel rushed ’cause you’re paranoid that your co-worker’s gonna recognize your shoes underneath the stall. And you’re like, “Oh, no! Courtney’s listening. She’s waiting. She’s timing me.” And then you hurry, hurry, hurry, and then you never finish wiping and then your butt hole feels caked in doo doo all day long. And then if you dare scratch yourself, your underwear at the end of the day looks like it’s been run over by the Goonies. Housewives, they don’t gotta muffle their shit, too. They don’t gotta worry about the velocity with which their doo doo comes out. They don’t gotta try to, you know, squeeze the butt cheeks together to make sure that the doo doo comes out at a slow and steady pace, so that no unpredictable noise suddenly escapes and brings you deep, deep shame. Housewives are free to just blow ass into the toilet and let it echo and reverberate to the ends of their hallways while watching as much Netflix on their iPad as they want. They don’t gotta take these boring, repressed shits. They can listen to podcasts. Planet Money. They can do whatever they want.\\nYou know, it’s– it’s very distracting for me when I hear my co-workers blow ass into the toilet. I lose respect for them. Nothing they say to me anymore holds any sort of credence. I heard one of my co-workers blow ass into the toilet the other day. This bitch had the nerve to come up to me and say, “You need to get to work on time.” I was like, “You need to eat bananas.” “I saw those green ballet flats. I know that shit was you. Don’t try to tell me to get my shit together when I heard you not have your shit together.” My father-in-law had this huge sit-down with me and my husband recently. Um, and he was like, “Hey, I wanna talk to you guys about money. You guys need to make a lot more money if you wanna provide your children with the same kind of privileged childhood that you guys had.” I was like, “Why you telling me this shit? I should not be a part of this conversation. You tell you son that. Don’t your understand that I trapped your son for his earning potential? Why else would I choose to fuck one person for the rest of my life? I chose to marry him on the promise of early retirement, and when I said, ‘I do,’ what I really meant was, ‘Oh, I’m done.'” I’m done. I don’t wanna work anymore and I’m not dieting anymore. Since I got married last year, I’ve been eating fried chicken skin every day since. That’s right. And just fulfilling my destiny. Which is to turn into a circle with eyelashes. Like Mrs. Pacman, just… Let’s redecorate.\\nI gave up a lot of myself when I got married. I’m a– I’m a disgusting pervert. I’m a pervert. I’m a gross filthy animal. And I think it’s because I started watching porn at a very young age. And what happens when you start watching porn at a young age is that… y-you get sicker, and sicker, and sicker. The images you crave get sicker, and sicker, and sicker, but it’s OK, because the Internet will always catch up to you.\\nI broke up with my last boyfriend because he refused to put it in the back. I was like, “Uh, you’re a idiot, dude. Do you realize that if I went on Craigslist… and posted ‘Tiny Asian female seeking anal…’ the Internet would crash.” “And all the Jewish male heads in the universe would simultaneously explode.” They would explode. A lot of women get really, you know… freaked out about anal. And they’re like, “Oh, I don’t wanna do that. I’m scared of– of the pain.” You ain’t scared of the pain. Women, they wax their eyebrows, they do all sorts of crazy shit. You’re not scared of the pain. What you’re really scared of is doo doo on the dick. You’re scared that he’s gonna see that and that’s gonna be all of your shame, your inner evil, all your secrets and lies. Sephora can’t help you now. But don’t worry, ’cause when he puts it in the butt, all he’s thinking about is, “I just put it in her butt.” “I gotta go call my mom, my dad, Dave, my grandma.” You’re– If you’re married, you’re gonna have to do anal eventually, OK? You have to, because you gotta change it up. You gotta change it up, so that you don’t cheat on each other. You gotta keep it interesting. If you put it in different holes, maybe you’ll feel like you’re fucking different people. I was very sexually active in my 20s, and as a result, I’m a little bit… …stretched out down there, OK? So, when I finally did anal, I just felt like I got a second chance at life, you know? I was, like, “Oh, my God! It’s like I’m going back in time!” ♪ A whole new world ♪ It was magical. A big fantasy of mine before I got married was to help as many men as possible discover their prostate. Yeah, like a conqueror. I just wanted to… Now, if you haven’t done it before, ladies, go home and treat yourself. Do it tonight. You only live once. YOLO. Just sneak your– Just give your man a little– a little push-push in the tush-tush. Just give him a little Atari, you know, and you’ll get a lot of resistance from the man at first. You’ll get a lot of “No! No! No! No, please! No, really, I don’t– No! I don’t! I don’t! No!” They get all squirmy wormy because… they’re scared. They’re scared that if you stick your thumb up there and succeed, and they like it, that then, it might mean that they’re gay. And I like that fear. That shit turns me on, you know? Especially when that fear metamorphosizes into pleasure. Oh, my God! And you just see the look in the man’s eye like he’s discovered nirvana. And it’s like you’re the first lady to show him that he had a magical clit in his butt hole. And then, you as the woman, in his eyes, just become the Lord of the Rim, you know? My husband is unfortunately just not as freaky as me. When– When I’ve asked him to spank me, this is what he does. “Hey. Hey, are you OK? Are you all right? You know I respect you, right?” I’m, like, “Yes, I know you respect me and that’s why you need to abuse me. OK?” ‘Cause it’s the most strong-headed, loud-mouthed women who like to be abused the most in bed. Women who are C.E.O.s, they just wanna be roughed around. They just want their– Glasses always means the woman wants some– It’s because we’re so in control all the time, that we just wanna experience some risk and be out of control, you know? Like, “I don’t wanna die! Don’t kill me! I don’t wanna die!” But I also don’t want to be sure that I’m gonna live. You know? I just wanna be out of control for once. Just– Just choke me enough so that I can’t talk. ‘Cause if I can talk, I’m gonna tell you what to do. And I’m tired of being the boss. I’m the boss all the time, so, in the bedroom, you be the boss. Yes. Because I’m the real boss. And I told you so, motherfucker, so do it. Sheryl Sandberg, that woman who wrote Lean In, has had such a big impact that now, because of her, there is a ban on the word “bossy” in elementary schools, because according to her, it’s sexist to use the word “bossy,” because boys are never called bossy. So, now, instead of saying, “You’re bossy,” you’re supposed to say, “You have executive leadership skills.” Which is a very roundabout way of saying: “You’re a little cunt.” I’m just waiting for the right moment to, like, become a housewife, financially, you know? I want my husband to get us to, like, a certain point financially. I wanna get to the point as a couple where I can comfortably afford sliced mango. Know what I’m talking about? I’m talking about that Whole Foods mango. That $10-a-box Whole Foods mango that was sliced by white people. That’s the kind of income bracket I’m striving for. That’s when you know you’ve made it, when you’re eating mango that was sliced by a dude named Noah. I want Noah mango… …Rebecca kiwi, Danielle pineapple. You know what else I want? I wanna be able to take a stroll on a sidewalk, see a quarter, and just keep on walking. Like a princess. I have some useful advice for all my Asian-American brothers and sisters. Yeah! Never go paintballing with a Vietnam veteran. So, I don’t know if you guys can tell, but I am seven and a half months pregnant. Yeah. It’s very rare and unusual to see a female comic perform pregnant, because female comics… don’t get pregnant. Just try to think of one. I dare you. There’s– None of them. Once they do get pregnant, they generally disappear. That’s not the case with male comics. Once they have a baby, they’ll get up on stage a week afterwards and they’ll be like, “Guys, I just had this fucking baby. That baby’s a little piece of shit. It’s so annoying and boring.” And all these other shitty dads in the audience are, like, “That’s hilarious. I identify.” And their fame just swells because they become this relatable family funny man all of a sudden. Meanwhile, the mom is at home, chapping her nipples, feeding the fucking baby, and wearing a frozen diaper ’cause her pussy needs to heal from the baby’s head shredding it up. She’s busy. So, I don’t know what’s gonna happen to me. You know, a lot of my female stand-up comic friends who are a lot more successful and famous than me discouraged me from having a kid. And they were like, “Ali, why are you gonna have a kid? You just gonna become– You’re gonna disappear, and you’re gonna become some lame stay-at-home mom.” I was like, “Yeah, that’s the dream.” That’s the point. This is the ultimate trap. I won, you know?\\nAnother thing a lot of my friends said to me when they were discouraging me from having a kid, they were like, “Why are you gonna have a kid? Why don’t you just travel the world with your husband and just do whatever you want for the rest of your lives with no kid attached.” I was like, “Yeah, that’s cool… until my husband dies.” Which he’s definitely gonna before me. Because I’m a Asian woman, and therefore, guaranteed to live until I’m a billion. I’m guaranteed, like a turtle from the Galapagos, OK? We all know the phrase “black don’t crack.” Well, Asian don’t die. We don’t die. Especially the women, we live forever. And you know why we’re such bad drivers? Because we’re trying to die. We’re like, “Yeah! Let me see how invincible I really am!” “Imma make this left hand turn signal and ignore this red light completely.” “I’m gonna make a right turn– I changed my mind, it’s a U-turn!” “I changed my mind again. It’s a O-turn!” Every time I get into a car accident… …I’m like, “Oh, my God, not again!” I need to hide my face so that everybody doesn’t see that it’s what everybody thought it was gonna be. So embarrassing. My Toyota Corolla is a mess. There’s this huge bear claw scratch on the side from this aggressive brick wall that came out of nowhere. And then, on the hood, there’s multiple hand prints from pedestrians who have had to alert me of their existence. I don’t know what’s wrong with me, but I’m still here, you know? I need to have children to keep me company when I get older. It’s lonely. My mom is 80, going through a full blown mid-life crisis. ‘Cause she knows that she’s got a century more to go. And she is so lonely. All of her white friends, dead. Her Mexican friends, dead. Black friends, dead. I’m just kidding. She doesn’t have any black friends. Life is not Rush Hour, the movie, OK? I need children to be there for me when I’m older, when I get as old as her. And when I say be there for me, I mean pay for me when my husband isn’t around to support me anymore. I’m not trying to be one of those old Chinese ladies who recycles for a living. That’s not my destiny, OK? Old Chinese ladies, they don’t give a fuck. They got no shame. They’re like, “I’m just gonna recycle… go bald… go to the park, do this shit.” They do that ’cause it’s a free activity. For them. They do it in their– their big-ass V. Stiviano visor, their Darth Vader-Tomb Raider- Boba Fett helmet. They wear that to protect themselves from their arch-nemesis, the sun. Their in a contest to see who’s gonna burn out first. Old Asian ladies and the sun are like the Tupac and Biggie of longevity. I also decided to have a kid because uh, I’m only 33, which, I know, is not technically high-risk, but my body was starting to show signs of change. And it– And it scared me. Like, I’m only 33 and… …my pussy is not as wet as it used to be. It’s very demoralizing, OK? Do you remember when you were 18 years old, and your pussy was just sopping wet all the time? All the time, you just took it for granted that you could just reach your hand down your pants at any given moment, you throw up the peace sign afterwards, and there would be that snail-trail in between your fingers. Oh, my God, it was so juicy. You could just blow a bubble wand with it, just… “I slime you, I slime you. Ghostbusters!” I don’t know what kind of mother I’m gonna be. I’m– I’m 33, and I did have to get a little bit of science involved when trying to get pregnant. And a lot of that… is most likely my fault. Because, when I was in my 20s, I ate Plan B like skittles. So, my uterus probably looked like a smoker’s lung. And I found out that my progesterone levels were alarmingly low. So, then I had to take these hormone pills that were suppositories, and Push Pop them up myself every single night. And then, at my writing job, at Fresh Off The Boat, I would be storyboarding in front of my co-workers, and then, at some point, the pill would inevitably dissolve and melt into my underwear, and I had to act like everything was OK, when everything was clearly not OK. And then, a side effect of the progesterone was that it made me extremely itchy. So, then I had to find ways to discretely scratch myself underneath the conference table, and then resist the urge to immediately smell my fingers afterwards. I want to be able to smell my fingers when I wanna smell my own goddamn fingers. Housewives, they can just scratch and sniff all day long. They just vacuum, scratch, sniff. They make a sandwich. “Uh, mmm.” They watch Property Brothers, scratch, “What’s crackin’? Mmm.” Every time you scratch yourself, all you can think about is, “When can I smell my fingers? When can I smell my fingers? When can I discretely find a way to…” “…smell my fingers?” Nature made you urgently curious to protect you, ’cause you gotta check that it’s all good in the hood. If it’s too funky, you need to see a doctor. Your fingers are your first WebMD. When my husband and I were trying to have a kid, a lot of people were like, “Oh, my God, that’s so hot. You guys doin’ a lot of fuckin’?” No, dude. That’s– That’s shit you do in your 20s, OK? When in– When you’re in your 30s, and you’ve been trying to get pregnant for a while, it gets very clinical. You pee on these ovulation strips that tell you when the eggs are droppin’. It tells you when it’s Easter time. And I would only fuck him when it was Easter time. It was, like, only four days out of the month, and outside of that, I would be like, “We’re not fuckin’. I need you to save it. I want your sperm to be as pent-up, and as angry and rapey as possible. So that, when they come out, it’s like, ‘Release the Kraken!'” And they just come out like a bunch of angry refugees escaping a dictatorship, you know? And, um… yeah, and most of the time, like, we wouldn’t even have sex, ’cause I was so tired when I would come home, and see the smiley face on the ovulation strip, and I’d be like, “OK, it’s go time,” and I would just give my husband a hand job most of the time, and he would close his eyes immediately. I know what that means, OK? When somebody closes their eyes during sex, it’s not because they’re in such ecstasy with you that– that they need to close their eyes. When somebody closes their eyes during sex, it’s because they’re literally trying to shut the image of your face out of their head and instead project two Latina lesbians that they saw earlier that day on RedTube onto the back of their eyelids. Which is fine by me, because then he doesn’t have to see the expression on my face that says, “Please, hurry the fuck up.” And then, when he was about to finish, I could always tell because the indication is very universal when a man is about to finish. It’s when they get that… that stupid-ass look on their face… …where they look like they just got bit by a zombie, just… And then, because we’re hippies, I’d be like, “Hey, hey! Please look me in the eye and remember to come with intention, OK?” And then, I would jump on him, and hold onto his neck, and I would just twerk, twerk, twerk the shit out of him… and do some of this shit that I learned in Atlanta. And then I would turn upside down immediately afterwards… to make sure all of that Harvard nectar would just drain inside of me. That’s right. ‘Cause I don’t wanna work anymore. I’m very grateful to be pregnant and to be… this far along, to be seven and a half months pregnant, because, last year, I had a miscarriage, which is very common. And a lot of women who are in their 20s flip out when they hear that. They’re like, “Oh, my God. That’s so dark and terrible. I can’t believe that.” I’m 33. Girl, when you’re 33, you’ll know plenty of women who have had a miscarriage. It’s super common, and I wish more women would talk about it so they wouldn’t feel so bad when they go through it. When I told my mom– She’s from a third world country, and when I told her I had one, she was like, “Uh, yeah. Where I’m from, that’s like losing a pair of shoes. It’s whatevs, OK?” And everything happens for a reason. I found out at my six-week sonogram, which is very early. And the doctor says to me, “Oh, my God, I see two sacks, which means you’re having twins.” And I was like, “No!” And then she said, “But what I don’t see is a heartbeat.” And I was like, “Yes!” “The Lord is mysterious!” Don’t feel bad, OK? They were the size of poppy seeds. I’ve picked boogers larger than the twins that I lost. And most women won’t let their husbands watch when they’re going through a miscarriage. I sat my husband down in front of me while I sat on the toilet, and I was like, “You look.” “You watch the whole thing.” And he felt so bad for me. And I used it as leverage and held that shit over his head for a month and got him to do whatever the fuck I wanted him to do for 30 days. He took me to see Beyoncé. He bought me a bike off of Craigslist. That’s my miscarriage bike, and I love it very much. For 30 days, I finally had the marriage I always wanted. I’m scared about giving childbirth, though. I’m– I’m very, very scared of childbirth. That’s why I’m going to hire a doula. You know what that is? You know what a doula is? That’s a white hippie witch… …that blows quinoa into your pussy to Keyser Söze all the pain away. A lot of women tried to freak me out. They tried to freak me out about childbirth by saying, “Ali, did you know that you’re gonna poop on the table?” I was like, “Yeah, I look forward to it.” I’m all backed up from holding in my shit at work. I can’t wait to cleanse. It makes sense, like, that you– that that happens because when you’re in labor, you push, you push, you push, and your husband will be asked to assist in the labor by lifting up your leg, which subsequently turns into a soft serve lever. You just shit on the floor in front of the love of your life. And just when you think that’s enough to make him finally leave you, boom, a baby comes out, and he gotta stay. That’s the real miracle of life, right there. I can already see how a child can really take its toll on a marriage, because the baby hasn’t even come out yet and I am already so resentful towards my husband. So much resentment, especially when he asks me to do shit around the house. “Hey, can you wash the dishes?” “No!” “Can you water the plants?” “I am not doing jack shit anymore. I’m busy makin’ a eyeball, OK? Are you makin’ a foot? I didn’t think so. You change the channel.” I can already see how there’s, like, this crazy double standard in our society of how it takes so little to be considered a great dad. And it also takes so little to be considered a shitty mom. People praise my husband for coming to all of my doctor’s appointments with me. “Oh, my God. I can’t believe he comes to all your doctor’s appointments. He is so supportive.” Guess who else has to go to those doctor appointments. Me! I’m the star of the show. There’s nothing for the camera to see if I’m not there. But he’s the hero for playing Candy Crush while I get my blood drawn. Meanwhile, if I do mushrooms seven months pregnant, I’m a bad mommy. You know, I– I– I, like, I berate my husband on, like, a daily basis. Partially because I really am mad at him. But mostly out of survival, because if he leaves me, I’m fucked. So, I have to chip away at his self-esteem on a daily basis… to keep him down so that he doesn’t believe that he’s worthy of another woman’s affection and leaves me. I gotta keep him around by keeping him down. People don’t tell you about all this shit that goes down with your body when you get pregnant, you know? Your nipples get huge and dark. I didn’t know that. I didn’t know that they get dark so that the baby can see, like, a bullseye. So that the baby can find it easier. And then, you know, they get big– they get big, like fingers. Like, “You, you. You owe me money, you.” My nipples look like Whoppers now, and naked, I look like a Minion. But I’m not gonna be one of those crazy pregnant ladies who tries to get all back in shape right after they get pregnant. No. Hopefully, if you see me in a year, I will have the kind of body where, if I do a nude scene on television, people will commend me for being courageous. For doing it. Now that I’m seven and a half months pregnant, my pussy’s all wet again. But it’s different. It’s not like when I was 18 years old, when it was like, really hot, you know? And I was like, “Why is it different?” And I looked it up, and my pussy’s all wet again because my– my body’s secreting mucus to protect the baby from bacteria attacking it. That’s not the same. When it’s straight up soldier glue, when it’s Neosporin. So, you know, I– I, in– previously, before I met my husband, I had dated a bunch of losers. And then, I meet this dream guy, who’s, like, way more handsome than me, out of my league, graduated from Harvard Business School. Worked hard to trap his ass. Got him to propose to me. Oh, my God, then we got married, all my dreams coming true, and then we got pregnant, and recently we bought our first home together. And, uh, two weeks into the escrow process, I discovered that my beautiful, Harvard-educated husband was $70,000 in debt. And me, with my hard-earned TV money, paid it all off. So, as it turns out, he’s the one who trapped me. How did he do it? How did he bamboozle me? Oh! Maybe because he went to Harvard Business School, the epicenter of white-collar crime. He Enron’d my ass. And now, if I don’t work, we die. Why else do you think I’m performing seven and a half months pregnant? All right, I’ve been Ali Wong. Have a good night, everybody. Thank you.\""
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's take a look at the transcript for Ali Wong\n",
    "data_df.transcript.loc['ali']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply a first round of text cleaning techniques\n",
    "import re\n",
    "import string\n",
    "\n",
    "def clean_text_round1(text):\n",
    "    '''Make text lowercase, remove text in square brackets, remove punctuation and remove words containing numbers.'''\n",
    "    text = text.lower()\n",
    "    text = re.sub('\\[.*?\\]', '', text)\n",
    "    text = re.sub('[%s]' % re.escape(string.punctuation), '', text)\n",
    "    text = re.sub('\\w*\\d\\w*', '', text)\n",
    "    return text\n",
    "\n",
    "# Remove square brackets\n",
    "round1 = lambda x: clean_text_round1(x) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>transcript</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ali</th>\n",
       "      <td>ladies and gentlemen please welcome to the stage ali wong hi hello welcome thank you thank you for coming hello hello we are gonna have to get thi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>anthony</th>\n",
       "      <td>thank you thank you thank you san francisco thank you so much so good to be here people were surprised when i told ’em i was gonna tape my special...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bill</th>\n",
       "      <td>all right thank you thank you very much thank you thank you thank you how are you what’s going on thank you it’s a pleasure to be here in the gre...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bo</th>\n",
       "      <td>bo what old macdonald had a farm e i e i o and on that farm he had a pig e i e i o here a snort there a old macdonald had a farm e i e i o  this i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dave</th>\n",
       "      <td>this is dave he tells dirty jokes for a living that stare is where most of his hard work happens it signifies a profound train of thought the alch...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hasan</th>\n",
       "      <td>what’s up davis what’s up i’m home i had to bring it back here netflix said “where do you want to do the special la chicago new york” i was like...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jim</th>\n",
       "      <td>ladies and gentlemen please welcome to the stage mr jim jefferies  hello sit down sit down sit down sit down sit down  thank you boston i appre...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>joe</th>\n",
       "      <td>ladies and gentlemen welcome joe rogan  what the fuck is going on san francisco thanks for coming i appreciate it god damn put your phone down ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>john</th>\n",
       "      <td>all right petunia wish me luck out there you will die on august   that’s pretty good all right hello hello chicago nice to see you again thank you...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>louis</th>\n",
       "      <td>intro\\nfade the music out let’s roll hold there lights do the lights thank you thank you very much i appreciate that i don’t necessarily agree wit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mike</th>\n",
       "      <td>wow hey thank you thanks thank you guys hey seattle nice to see you look at this look at us we’re here this is crazy it’s insane so about five yea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ricky</th>\n",
       "      <td>hello hello how you doing great thank you wow calm down shut the fuck up thank you what a lovely welcome i’m gonna try my hardest tonight you’re t...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                    transcript\n",
       "ali      ladies and gentlemen please welcome to the stage ali wong hi hello welcome thank you thank you for coming hello hello we are gonna have to get thi...\n",
       "anthony  thank you thank you thank you san francisco thank you so much so good to be here people were surprised when i told ’em i was gonna tape my special...\n",
       "bill      all right thank you thank you very much thank you thank you thank you how are you what’s going on thank you it’s a pleasure to be here in the gre...\n",
       "bo       bo what old macdonald had a farm e i e i o and on that farm he had a pig e i e i o here a snort there a old macdonald had a farm e i e i o  this i...\n",
       "dave     this is dave he tells dirty jokes for a living that stare is where most of his hard work happens it signifies a profound train of thought the alch...\n",
       "hasan      what’s up davis what’s up i’m home i had to bring it back here netflix said “where do you want to do the special la chicago new york” i was like...\n",
       "jim         ladies and gentlemen please welcome to the stage mr jim jefferies  hello sit down sit down sit down sit down sit down  thank you boston i appre...\n",
       "joe         ladies and gentlemen welcome joe rogan  what the fuck is going on san francisco thanks for coming i appreciate it god damn put your phone down ...\n",
       "john     all right petunia wish me luck out there you will die on august   that’s pretty good all right hello hello chicago nice to see you again thank you...\n",
       "louis    intro\\nfade the music out let’s roll hold there lights do the lights thank you thank you very much i appreciate that i don’t necessarily agree wit...\n",
       "mike     wow hey thank you thanks thank you guys hey seattle nice to see you look at this look at us we’re here this is crazy it’s insane so about five yea...\n",
       "ricky    hello hello how you doing great thank you wow calm down shut the fuck up thank you what a lovely welcome i’m gonna try my hardest tonight you’re t..."
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's take a look at the updated text\n",
    "data_clean = pd.DataFrame(data_df.transcript.apply(round1))\n",
    "data_clean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply a second round of cleaning\n",
    "def clean_text_round2(text):\n",
    "    '''Get rid of some additional quotation marks and newline text that was missed the first time around.'''\n",
    "    text = re.sub('[‘’“”…]', '', text)\n",
    "    text = re.sub('\\n', '', text)\n",
    "    return text\n",
    "\n",
    "round2 = lambda x: clean_text_round2(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>transcript</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ali</th>\n",
       "      <td>ladies and gentlemen please welcome to the stage ali wong hi hello welcome thank you thank you for coming hello hello we are gonna have to get thi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>anthony</th>\n",
       "      <td>thank you thank you thank you san francisco thank you so much so good to be here people were surprised when i told em i was gonna tape my special ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bill</th>\n",
       "      <td>all right thank you thank you very much thank you thank you thank you how are you whats going on thank you its a pleasure to be here in the great...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bo</th>\n",
       "      <td>bo what old macdonald had a farm e i e i o and on that farm he had a pig e i e i o here a snort there a old macdonald had a farm e i e i o  this i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dave</th>\n",
       "      <td>this is dave he tells dirty jokes for a living that stare is where most of his hard work happens it signifies a profound train of thought the alch...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hasan</th>\n",
       "      <td>whats up davis whats up im home i had to bring it back here netflix said where do you want to do the special la chicago new york i was like nah ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jim</th>\n",
       "      <td>ladies and gentlemen please welcome to the stage mr jim jefferies  hello sit down sit down sit down sit down sit down  thank you boston i appre...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>joe</th>\n",
       "      <td>ladies and gentlemen welcome joe rogan  what the fuck is going on san francisco thanks for coming i appreciate it god damn put your phone down ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>john</th>\n",
       "      <td>all right petunia wish me luck out there you will die on august   thats pretty good all right hello hello chicago nice to see you again thank you ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>louis</th>\n",
       "      <td>introfade the music out lets roll hold there lights do the lights thank you thank you very much i appreciate that i dont necessarily agree with yo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mike</th>\n",
       "      <td>wow hey thank you thanks thank you guys hey seattle nice to see you look at this look at us were here this is crazy its insane so about five years...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ricky</th>\n",
       "      <td>hello hello how you doing great thank you wow calm down shut the fuck up thank you what a lovely welcome im gonna try my hardest tonight youre thi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                    transcript\n",
       "ali      ladies and gentlemen please welcome to the stage ali wong hi hello welcome thank you thank you for coming hello hello we are gonna have to get thi...\n",
       "anthony  thank you thank you thank you san francisco thank you so much so good to be here people were surprised when i told em i was gonna tape my special ...\n",
       "bill      all right thank you thank you very much thank you thank you thank you how are you whats going on thank you its a pleasure to be here in the great...\n",
       "bo       bo what old macdonald had a farm e i e i o and on that farm he had a pig e i e i o here a snort there a old macdonald had a farm e i e i o  this i...\n",
       "dave     this is dave he tells dirty jokes for a living that stare is where most of his hard work happens it signifies a profound train of thought the alch...\n",
       "hasan      whats up davis whats up im home i had to bring it back here netflix said where do you want to do the special la chicago new york i was like nah ...\n",
       "jim         ladies and gentlemen please welcome to the stage mr jim jefferies  hello sit down sit down sit down sit down sit down  thank you boston i appre...\n",
       "joe         ladies and gentlemen welcome joe rogan  what the fuck is going on san francisco thanks for coming i appreciate it god damn put your phone down ...\n",
       "john     all right petunia wish me luck out there you will die on august   thats pretty good all right hello hello chicago nice to see you again thank you ...\n",
       "louis    introfade the music out lets roll hold there lights do the lights thank you thank you very much i appreciate that i dont necessarily agree with yo...\n",
       "mike     wow hey thank you thanks thank you guys hey seattle nice to see you look at this look at us were here this is crazy its insane so about five years...\n",
       "ricky    hello hello how you doing great thank you wow calm down shut the fuck up thank you what a lovely welcome im gonna try my hardest tonight youre thi..."
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's take a look at the updated text\n",
    "data_clean = pd.DataFrame(data_clean.transcript.apply(round2))\n",
    "data_clean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** This data cleaning aka text pre-processing step could go on for a while, but we are going to stop for now. After going through some analysis techniques, if you see that the results don't make sense or could be improved, you can come back and make more edits such as:\n",
    "* Mark 'cheering' and 'cheer' as the same word (stemming / lemmatization)\n",
    "* Combine 'thank you' into one term (bi-grams)\n",
    "* And a lot more..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Organizing The Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I mentioned earlier that the output of this notebook will be clean, organized data in two standard text formats:\n",
    "1. **Corpus - **a collection of text\n",
    "2. **Document-Term Matrix - **word counts in matrix format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We already created a corpus in an earlier step. The definition of a corpus is a collection of texts, and they are all put together neatly in a pandas dataframe here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>transcript</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ali</th>\n",
       "      <td>Ladies and gentlemen, please welcome to the stage: Ali Wong! Hi. Hello! Welcome! Thank you! Thank you for coming. Hello! Hello. We are gonna have ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>anthony</th>\n",
       "      <td>Thank you. Thank you. Thank you, San Francisco. Thank you so much. So good to be here. People were surprised when I told ’em I was gonna tape my s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bill</th>\n",
       "      <td>[cheers and applause] All right, thank you! Thank you very much! Thank you. Thank you. Thank you. How are you? What’s going on? Thank you. It’s a ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bo</th>\n",
       "      <td>Bo What? Old MacDonald had a farm E I E I O And on that farm he had a pig E I E I O Here a snort There a Old MacDonald had a farm E I E I O [Appla...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dave</th>\n",
       "      <td>This is Dave. He tells dirty jokes for a living. That stare is where most of his hard work happens. It signifies a profound train of thought, the ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hasan</th>\n",
       "      <td>[theme music: orchestral hip-hop] [crowd roars] What’s up? Davis, what’s up? I’m home. I had to bring it back here. Netflix said, “Where do you wa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jim</th>\n",
       "      <td>[Car horn honks] [Audience cheering] [Announcer] Ladies and gentlemen, please welcome to the stage Mr. Jim Jefferies! [Upbeat music playing] Hello...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>joe</th>\n",
       "      <td>[rock music playing] [audience cheering] [announcer] Ladies and gentlemen, welcome Joe Rogan. [audience cheering and applauding] What the fuck is ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>john</th>\n",
       "      <td>All right, Petunia. Wish me luck out there. You will die on August 7th, 2037. That’s pretty good. All right. Hello. Hello, Chicago. Nice to see yo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>louis</th>\n",
       "      <td>Intro\\nFade the music out. Let’s roll. Hold there. Lights. Do the lights. Thank you. Thank you very much. I appreciate that. I don’t necessarily a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mike</th>\n",
       "      <td>Wow. Hey, thank you. Thanks. Thank you, guys. Hey, Seattle. Nice to see you. Look at this. Look at us. We’re here. This is crazy. It’s insane. So ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ricky</th>\n",
       "      <td>Hello. Hello! How you doing? Great. Thank you. Wow. Calm down. Shut the fuck up. Thank you. What a lovely welcome. I’m gonna try my hardest tonigh...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                    transcript\n",
       "ali      Ladies and gentlemen, please welcome to the stage: Ali Wong! Hi. Hello! Welcome! Thank you! Thank you for coming. Hello! Hello. We are gonna have ...\n",
       "anthony  Thank you. Thank you. Thank you, San Francisco. Thank you so much. So good to be here. People were surprised when I told ’em I was gonna tape my s...\n",
       "bill     [cheers and applause] All right, thank you! Thank you very much! Thank you. Thank you. Thank you. How are you? What’s going on? Thank you. It’s a ...\n",
       "bo       Bo What? Old MacDonald had a farm E I E I O And on that farm he had a pig E I E I O Here a snort There a Old MacDonald had a farm E I E I O [Appla...\n",
       "dave     This is Dave. He tells dirty jokes for a living. That stare is where most of his hard work happens. It signifies a profound train of thought, the ...\n",
       "hasan    [theme music: orchestral hip-hop] [crowd roars] What’s up? Davis, what’s up? I’m home. I had to bring it back here. Netflix said, “Where do you wa...\n",
       "jim      [Car horn honks] [Audience cheering] [Announcer] Ladies and gentlemen, please welcome to the stage Mr. Jim Jefferies! [Upbeat music playing] Hello...\n",
       "joe      [rock music playing] [audience cheering] [announcer] Ladies and gentlemen, welcome Joe Rogan. [audience cheering and applauding] What the fuck is ...\n",
       "john     All right, Petunia. Wish me luck out there. You will die on August 7th, 2037. That’s pretty good. All right. Hello. Hello, Chicago. Nice to see yo...\n",
       "louis    Intro\\nFade the music out. Let’s roll. Hold there. Lights. Do the lights. Thank you. Thank you very much. I appreciate that. I don’t necessarily a...\n",
       "mike     Wow. Hey, thank you. Thanks. Thank you, guys. Hey, Seattle. Nice to see you. Look at this. Look at us. We’re here. This is crazy. It’s insane. So ...\n",
       "ricky    Hello. Hello! How you doing? Great. Thank you. Wow. Calm down. Shut the fuck up. Thank you. What a lovely welcome. I’m gonna try my hardest tonigh..."
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's take a look at our dataframe\n",
    "data_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>transcript</th>\n",
       "      <th>full_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ali</th>\n",
       "      <td>Ladies and gentlemen, please welcome to the stage: Ali Wong! Hi. Hello! Welcome! Thank you! Thank you for coming. Hello! Hello. We are gonna have ...</td>\n",
       "      <td>Ali Wong</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>anthony</th>\n",
       "      <td>Thank you. Thank you. Thank you, San Francisco. Thank you so much. So good to be here. People were surprised when I told ’em I was gonna tape my s...</td>\n",
       "      <td>Anthony Jeselnik</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bill</th>\n",
       "      <td>[cheers and applause] All right, thank you! Thank you very much! Thank you. Thank you. Thank you. How are you? What’s going on? Thank you. It’s a ...</td>\n",
       "      <td>Bill Burr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bo</th>\n",
       "      <td>Bo What? Old MacDonald had a farm E I E I O And on that farm he had a pig E I E I O Here a snort There a Old MacDonald had a farm E I E I O [Appla...</td>\n",
       "      <td>Bo Burnham</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dave</th>\n",
       "      <td>This is Dave. He tells dirty jokes for a living. That stare is where most of his hard work happens. It signifies a profound train of thought, the ...</td>\n",
       "      <td>Dave Chappelle</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hasan</th>\n",
       "      <td>[theme music: orchestral hip-hop] [crowd roars] What’s up? Davis, what’s up? I’m home. I had to bring it back here. Netflix said, “Where do you wa...</td>\n",
       "      <td>Hasan Minhaj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jim</th>\n",
       "      <td>[Car horn honks] [Audience cheering] [Announcer] Ladies and gentlemen, please welcome to the stage Mr. Jim Jefferies! [Upbeat music playing] Hello...</td>\n",
       "      <td>Jim Jefferies</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>joe</th>\n",
       "      <td>[rock music playing] [audience cheering] [announcer] Ladies and gentlemen, welcome Joe Rogan. [audience cheering and applauding] What the fuck is ...</td>\n",
       "      <td>Joe Rogan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>john</th>\n",
       "      <td>All right, Petunia. Wish me luck out there. You will die on August 7th, 2037. That’s pretty good. All right. Hello. Hello, Chicago. Nice to see yo...</td>\n",
       "      <td>John Mulaney</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>louis</th>\n",
       "      <td>Intro\\nFade the music out. Let’s roll. Hold there. Lights. Do the lights. Thank you. Thank you very much. I appreciate that. I don’t necessarily a...</td>\n",
       "      <td>Louis C.K.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mike</th>\n",
       "      <td>Wow. Hey, thank you. Thanks. Thank you, guys. Hey, Seattle. Nice to see you. Look at this. Look at us. We’re here. This is crazy. It’s insane. So ...</td>\n",
       "      <td>Mike Birbiglia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ricky</th>\n",
       "      <td>Hello. Hello! How you doing? Great. Thank you. Wow. Calm down. Shut the fuck up. Thank you. What a lovely welcome. I’m gonna try my hardest tonigh...</td>\n",
       "      <td>Ricky Gervais</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                    transcript  \\\n",
       "ali      Ladies and gentlemen, please welcome to the stage: Ali Wong! Hi. Hello! Welcome! Thank you! Thank you for coming. Hello! Hello. We are gonna have ...   \n",
       "anthony  Thank you. Thank you. Thank you, San Francisco. Thank you so much. So good to be here. People were surprised when I told ’em I was gonna tape my s...   \n",
       "bill     [cheers and applause] All right, thank you! Thank you very much! Thank you. Thank you. Thank you. How are you? What’s going on? Thank you. It’s a ...   \n",
       "bo       Bo What? Old MacDonald had a farm E I E I O And on that farm he had a pig E I E I O Here a snort There a Old MacDonald had a farm E I E I O [Appla...   \n",
       "dave     This is Dave. He tells dirty jokes for a living. That stare is where most of his hard work happens. It signifies a profound train of thought, the ...   \n",
       "hasan    [theme music: orchestral hip-hop] [crowd roars] What’s up? Davis, what’s up? I’m home. I had to bring it back here. Netflix said, “Where do you wa...   \n",
       "jim      [Car horn honks] [Audience cheering] [Announcer] Ladies and gentlemen, please welcome to the stage Mr. Jim Jefferies! [Upbeat music playing] Hello...   \n",
       "joe      [rock music playing] [audience cheering] [announcer] Ladies and gentlemen, welcome Joe Rogan. [audience cheering and applauding] What the fuck is ...   \n",
       "john     All right, Petunia. Wish me luck out there. You will die on August 7th, 2037. That’s pretty good. All right. Hello. Hello, Chicago. Nice to see yo...   \n",
       "louis    Intro\\nFade the music out. Let’s roll. Hold there. Lights. Do the lights. Thank you. Thank you very much. I appreciate that. I don’t necessarily a...   \n",
       "mike     Wow. Hey, thank you. Thanks. Thank you, guys. Hey, Seattle. Nice to see you. Look at this. Look at us. We’re here. This is crazy. It’s insane. So ...   \n",
       "ricky    Hello. Hello! How you doing? Great. Thank you. Wow. Calm down. Shut the fuck up. Thank you. What a lovely welcome. I’m gonna try my hardest tonigh...   \n",
       "\n",
       "                full_name  \n",
       "ali              Ali Wong  \n",
       "anthony  Anthony Jeselnik  \n",
       "bill            Bill Burr  \n",
       "bo             Bo Burnham  \n",
       "dave       Dave Chappelle  \n",
       "hasan        Hasan Minhaj  \n",
       "jim         Jim Jefferies  \n",
       "joe             Joe Rogan  \n",
       "john         John Mulaney  \n",
       "louis          Louis C.K.  \n",
       "mike       Mike Birbiglia  \n",
       "ricky       Ricky Gervais  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's add the comedians' full names as well\n",
    "full_names = ['Ali Wong', 'Anthony Jeselnik', 'Bill Burr', 'Bo Burnham', 'Dave Chappelle', 'Hasan Minhaj',\n",
    "              'Jim Jefferies', 'Joe Rogan', 'John Mulaney', 'Louis C.K.', 'Mike Birbiglia', 'Ricky Gervais']\n",
    "\n",
    "data_df['full_name'] = full_names\n",
    "data_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's pickle it for later use\n",
    "data_df.to_pickle(\"corpus.pkl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Document-Term Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For many of the techniques we'll be using in future notebooks, the text must be tokenized, meaning broken down into smaller pieces. The most common tokenization technique is to break down text into words. We can do this using scikit-learn's CountVectorizer, where every row will represent a different document and every column will represent a different word.\n",
    "\n",
    "In addition, with CountVectorizer, we can remove stop words. Stop words are common words that add no additional meaning to text such as 'a', 'the', etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aaaaah</th>\n",
       "      <th>aaaaahhhhhhh</th>\n",
       "      <th>aaaaauuugghhhhhh</th>\n",
       "      <th>aaaahhhhh</th>\n",
       "      <th>aaah</th>\n",
       "      <th>aah</th>\n",
       "      <th>abc</th>\n",
       "      <th>abcs</th>\n",
       "      <th>ability</th>\n",
       "      <th>abject</th>\n",
       "      <th>...</th>\n",
       "      <th>zee</th>\n",
       "      <th>zen</th>\n",
       "      <th>zeppelin</th>\n",
       "      <th>zero</th>\n",
       "      <th>zillion</th>\n",
       "      <th>zombie</th>\n",
       "      <th>zombies</th>\n",
       "      <th>zoning</th>\n",
       "      <th>zoo</th>\n",
       "      <th>éclair</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ali</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>anthony</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bill</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bo</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dave</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hasan</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jim</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>joe</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>john</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>louis</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mike</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ricky</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12 rows × 7484 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         aaaaah  aaaaahhhhhhh  aaaaauuugghhhhhh  aaaahhhhh  aaah  aah  abc  \\\n",
       "ali           0             0                 0          0     0    0    1   \n",
       "anthony       0             0                 0          0     0    0    0   \n",
       "bill          1             0                 0          0     0    0    0   \n",
       "bo            0             1                 1          1     0    0    0   \n",
       "dave          0             0                 0          0     1    0    0   \n",
       "hasan         0             0                 0          0     0    0    0   \n",
       "jim           0             0                 0          0     0    0    0   \n",
       "joe           0             0                 0          0     0    0    0   \n",
       "john          0             0                 0          0     0    0    0   \n",
       "louis         0             0                 0          0     0    3    0   \n",
       "mike          0             0                 0          0     0    0    0   \n",
       "ricky         0             0                 0          0     0    0    0   \n",
       "\n",
       "         abcs  ability  abject  ...  zee  zen  zeppelin  zero  zillion  \\\n",
       "ali         0        0       0  ...    0    0         0     0        0   \n",
       "anthony     0        0       0  ...    0    0         0     0        0   \n",
       "bill        1        0       0  ...    0    0         0     1        1   \n",
       "bo          0        1       0  ...    0    0         0     1        0   \n",
       "dave        0        0       0  ...    0    0         0     0        0   \n",
       "hasan       0        0       0  ...    2    1         0     1        0   \n",
       "jim         0        0       0  ...    0    0         0     0        0   \n",
       "joe         0        0       0  ...    0    0         0     0        0   \n",
       "john        0        0       0  ...    0    0         0     0        0   \n",
       "louis       0        0       0  ...    0    0         0     2        0   \n",
       "mike        0        0       0  ...    0    0         2     1        0   \n",
       "ricky       0        1       1  ...    0    0         0     0        0   \n",
       "\n",
       "         zombie  zombies  zoning  zoo  éclair  \n",
       "ali           1        0       0    0       0  \n",
       "anthony       0        0       0    0       0  \n",
       "bill          1        1       1    0       0  \n",
       "bo            0        0       0    0       0  \n",
       "dave          0        0       0    0       0  \n",
       "hasan         0        0       0    0       0  \n",
       "jim           0        0       0    0       0  \n",
       "joe           0        0       0    0       0  \n",
       "john          0        0       0    0       1  \n",
       "louis         0        0       0    0       0  \n",
       "mike          0        0       0    0       0  \n",
       "ricky         0        0       0    1       0  \n",
       "\n",
       "[12 rows x 7484 columns]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We are going to create a document-term matrix using CountVectorizer, and exclude common English stop words\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "cv = CountVectorizer(stop_words='english')\n",
    "data_cv = cv.fit_transform(data_clean.transcript) # fit count vectorizor to our CLEAN transcript data\n",
    "\n",
    "# Convert it to an array and label all the columns\n",
    "# Can use this part for future projects\n",
    "data_dtm = pd.DataFrame(data_cv.toarray(), columns=cv.get_feature_names())\n",
    "data_dtm.index = data_clean.index\n",
    "\n",
    "# Document-Term matrix\n",
    "data_dtm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's pickle it for later use\n",
    "data_dtm.to_pickle(\"dtm.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's also pickle the cleaned data (before we put it in document-term matrix format) and the CountVectorizer object\n",
    "data_clean.to_pickle('data_clean.pkl')\n",
    "pickle.dump(cv, open(\"cv.pkl\", \"wb\"))"
   ]
  }
 ],
 "metadata": {
  "_draft": {
   "nbviewer_url": "https://gist.github.com/da9290c4a21c1fddfc5cba9f82f8ba5a"
  },
  "gist": {
   "data": {
    "description": "Stand-Up-Comedy-NLP-Data-Cleaning.ipynb",
    "public": true
   },
   "id": "da9290c4a21c1fddfc5cba9f82f8ba5a"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}