{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Some important admin and homework for next time.\n",
    "\n",
    "\n",
    "__Important point__. If you have not filled __[the mid-term survery ](https://forms.gle/wQMj2GV4XPFNJEuS7)__, please do! It's 100% anonymous. I promise it should not take more than 5 minutes. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview of today's class.\n",
    "\n",
    "This week's curriculum is about text analysis. The overview is\n",
    "\n",
    "* Finding the important words in a document (TF-IDF)\n",
    "* Apply these tricks to understand what different communities in Computational Social Science are working on. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1 - Words that characterize Computational Social Science communities\n",
    "\n",
    "In this section, we'll begin to play around with how far we can get with simple strategies for looking at text. The video is Sune talking about a fun paper, which shows you how little is needed in order to reveal something very interesting about humans that produce text. Then, we'll use a simple weighting scheme called TF-IDF to find the words in the Computational Social Science papers that charachterize different communities. Finally, we'll visualize them in a fun little word cloud (below is what I found). The wordclouds may not be immediately understandable. But if you do some research on the important words, you will find that the TF-IDF method extracts quite interesting information.\n",
    "\n",
    "<img src=\"https://github.com/lalessan/comsocsci2023/blob/master/figures/tf_idf.png?raw=true\" alt=\"Drawing\" style=\"width: 1000px;\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Video lecture**: Simple methods reveal a lot. Sune talks a little bit about the paper: [Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach](http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0073791).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2MBERISGBUYLxoaL2NCOEJjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY//AABEIAWgB4AMBIgACEQEDEQH/xAAbAAEAAwEBAQEAAAAAAAAAAAAAAQIDBAUGB//EAD8QAAIBAwIEAwYFAQYEBwAAAAABAgMEESExBRJBURMUYSIyUnGBkQYjM5LSQhZDU1Sx0RUkYqEXNERyosHw/8QAGAEBAQEBAQAAAAAAAAAAAAAAAAECAwT/xAAeEQEBAQEAAwEBAQEAAAAAAAAAARECEiExA0FRE//aAAwDAQACEQMRAD8A/PwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAbeWn3iPLT7xAxBt5afeI8tPvEDEG3l594keXn3iBkDXy8+8R5efeIGQNfLz7xHl590BkDXwJd0PLz7xAyBurSo+sSFbTlPkTjkDEHdLhVeMOZyp4+b/2MHaTXWIGANvLz7xHlp94gYg38pPvEh2811iBiDTwJd0PBl3QGYNPBl3Q8GXdAZg08GXdDwZd0BmDTwJd0PBl3QGYNVQk+qJVtN9YgYg38pU7xLRsastnH55C45gdkuHVo9Yfcp5Gr8UPuTTK5gdLsqieOaH3Zb/h9XOOaH3Y0yuQHS7Gov6ofdk+Rq/FD7saZXKDqVhVf9UPux5Cr8UPuymOUHV5Cr8UPux5Cr8UPuwY5QdXkKvxQ+7H/AA+r8UPuwY5QdXkKvxQ+7HkKvxQ+7BjlB1eQq/FD7seQq/FD7sGOUHV5Cr8UPux5Gr8UPuwY5QdXkavxQ+7I8jV+KH3YTHMDp8jV+KH3Y8jV+KH3YXHMDp8jV+KH3HkqnxQ+7BjmB0+SqfFD7jyVT4ofcJjmB0eSqd4/ceTqd4/cGOgE4GCogEkAQCQBBBIIIAAEEgAbQ90rQWbpF4+6RZrN0B6ldYor5HlT3Z692sUV8jyJ9RVZ9SyKlkBboZyNOhlICpAAAAEAkEhRENFiGBVbmkdSiWp207enKCkpYl2CyapCGVvh+p0JpLCxnuijqRiuSTWUYOvHOOq2ZluTGrudeXOCjqqeVjDxkyuKXPHxKW63RzxqPmT2YG0J80ks5LuvyS5WckvYqNro9S1w8tSRcRpVqvlyn1OijUUov5HF71Gfpgvav/Rg13QeUWRw063Kteh10aikt9QNMDBJOAioJAEAkFEDBIArgFiCCMEYLEAVBbBAEAkgogEkAQQyxAHMpNF1JMq0Rg05tMEFVIummBAJaIAgEkAQQSQQCSCVuBul7BPDo5uWMflmnCY5rso9G/WKZ40up7XEtIHiyFWMyyILIglmUjV7GUgKAkAQSAQCQAqSUstIg7bJ046uPNILF6dtTpx53lmVe5SeMYR01ZvGXUST6YPMuE8t5T9Ysw6fGdXmWJrVdzOXt+1HR9UTCbXsvYvGnzaxX0Kn1FGpLXX2lqi1ampx8WHXddmaU7ScppxWp6FDhtTVJbjWpza8epF+zPG+5aUeaEfsfQ0uDSlHbQsuCNrHKTV8HzMI45o+gjLlh8z6GpwGonpE5K3BK0dkXU8Hjt4ysExquOxtXsq9LPNBnNJNPUrF2PSt7hTST3OhST6njRm4tYOu3q5qLOwR6ACafUtygVBIKIIJAEMgsQyIggkBUEEgoggsQBUEgCpBYgIwIaJBpzVwRsXIYUUy2jKNEbAXaICkWxkgoCXEgASgTHco3a/KOjgsc1X8zCelI7OBRzJv1Kjo4rojxZHtcWPFkStRVFkVRdEB7GUjZ7GMgKAkAQSAQASAIOu0p8yych6Vn+jnGEt2yVvn61qUoeH692eVVpxjN4bz6I1vbl55YvBvwLhdTiFfnlnw09+5l0++mVrw6rcPKi8d8HuWnAdE2j6az4ZTowS5djvjbxXQn1uZHz1vwZRabR6MOHxi1iJ6kafoacgw156s1jGMfI0jawS2O3lSIaKa45UI9jGdvBrWKO6SMpR7kWPJuLGnNawX2PB4hwOlPLjHD9D62aOKtAi/X55ecPqWr1i2u5yqUo9cH3lza060XGUdz5TinC52snOCbp/6G5XLrj/HJTryW0sP1PRt6yqwWXr1PIhhvU67aU6L2zFlc3ojAi1JJrqSBUE4AFQSQUQCSCCATggCASQBBBYhlFSCxAGAANuSASQQQMEgKo0E2mWwQBrGSaIccmRaM2gJxgmHvItFqRpGnqsAWq/pHo8Aj7OTguItUj1OAR/KKjPi+7PGkexxj32eRIzWlUXiVRdAQ9jGRvLYxluBQEkACQCACQBB3LmhaRT67I5KcOepGPdnfdPEM9EsIlb5edSt6l3eQowWXJ4P0jhPD4WVrCnFbLU8D8J8N5U7uqval7ueiPr6a0MWu3M/rSCwarYpEugtWQI5kOYqDIYyVlqBEjOReWxmFYzic1aOTskYVI5Rlp5tSOGYV6EKtNqSTyd9WByz0Cvi+KcOdpX5oL8t/wDYm1ipwaayfS3lCNxSlCSPmuSVtVcW2mblcepjrjouXsSYRm5P/wDanQllFYVGC/KRyhFBgvyjlAzGDTlIwUZ4GDTBGAjPAwaYIaApgjBfBAGbRUu9yrCsCCQackAkgAQSAIIZICqkYJAELKOmhV1wznNaSzMD0aijKjk9TgtNKhldj5+4nKFPCZ9DwSX/ACuvYqPP4x+qzyZHqcXeazPLkZrSEXRRGkQEtjCW5vLYxkBQEkAACQABOCK1tY5rLsj0KdrK7u6dL+lvX5HJYRcquEfT8HteWbqta7IzXTiPbs6KpUowisJLCO2OiMaawkbRTMOy6L56EKJZQyaRESywTgnARTTJDWTQjAGLRnKLOnBjPCGGsWmZyNZSSW5jOce5GtY1EmcFwsM7Ztbp5OS51IribweRxKhGbzjXuj1Z7nn3iTW5Yz08RVXTe60PSoyU6UZJnlXEHGq9Op1WFTl9nOYs3HGu/BGCeZFXNFZSQVcyPEAuQZubIcmEaEPBllkahWjaI5kUwxhhGm5R7lokPcDORVl5FGGmAANOQAABBJAAgkgCAAFEbUF7Rkb2/vBC72SPo+ExxafQ+duveivU+o4dHFn9Cjw+KPNZnmzPQ4k/z38zz57ma0qmXjIryjDA0k9DKRbJD1AoCcEAACQCJAIr0OFrVn1/D44pwPlLB4o6LGp9ZYSXhQb6IxXbh6kTaLXc8ypeKGrehw1+L82VB49WRt9F4kFvJF1Wh0aPkVdznL2nJv0Ou3uJppJv1Q0x9MpaE50OO2qc0FqdGdC6YSqYMZ3cYvV6lK0+VM8m7k3LKepNXxdN1xeMM4wsHBLjU29I5T6nNXtlWw28M2t+CqphybY0sxePE41GlzL5JmkrhPXm+x1UuBW0VqtRU4NQx7LaCenmSvuWeeif3LuvCtHmi9exF1wvw3mEmeVVU6DeG/QK66m8jz66ctMHVRreNDm6mVyuVcyCV4t3Tw9fuY2zxPGep13TU0zgjmMvVM3HGvX5WxyMvSlmnF+hYrLLwx4Zo2QwKciHIixDAjlRGESVYQIlsSRLYKiIkRHcmQRnIozSWxRhXOADTmAAAAAIIJIAAAKHRbLU5zqtUERca1or1Pq7NYsvofK1dbmC9T62gsWP0NI+Z4jrcP5nDLc7L/8A8wzje5ltKRZIrE0SIjOaMnLBvMwkgJTTDRTGCyYUGSRgCATgEV6dq8W8Ue75nwbaFOOeZrU+dt3+XHXqfQW9NVbnElnGFqYrtw67e0newXOsLodkOAUktXn5nVTqU7elrhJI5L3irpU3OU1Sp93v9iOi/wDw2nSWEkQ6EYvKweLW4pKdNVFO4UW/ewsfY3tb+rLHN+ZB7SxhmbGp7+V7NvUcZJHoKS5TyqOZSjJPRnblxpliYrXktTy66UqmDetN6vJlGnzpSXVhfi6VKnT5pY0OKvxarKbp26SS3k9kb8QtqnKlBTlnpFZM7Dh78VePTcYLXD7g9ZrkqcXqU5OPi1ak0s4hAUPxJmag55faawxxXhVWdzXdKEnGeHGUeh51PgFw5wk4tJb5eTp4zHGd234+gjxKncLlekuzOO8jGSeEed5G8tt05RT07o7ac5ypYqL2v9TnXaOKjGUKkljRmtaPNHB1wt8xzgwqLddUVK+fuVySaZyS95M9DiMcVH6nB/qbjhXrUf0Y/IuitL9KPyLIrCGCWAKgkgCMFWXKsColsAwKLcmRXqXewRR7GbNGZsK5wAacwAAAwAIIJIAAACUddqtDkR2Wq9kCuM3lNep9dFYsfofJU9b+HzPrp6WP0NI+TvdbiRyPc6bvWvI53uYbTE0RSJdBFJmLN5mLAqyCQFQSpAgC5BGxKZB02kvaUX3PrramoVnLuj5CyWbmC7s+7jR/LWN8GOnf83ncQv3GpiKcnHaK6s5qnD6l9YyqTy67ecP/AER69vZRVTmqxyz0YQoU4+4smebjp179PmY8IrVasZQpzjFRxiTwj1qPB5xt6dPmUZR3kj0lKT9yGF3JeYrLepu3WOefFlQounJU+bm5ep0VViOCKEP6u5rOm30I3ryKyaljoLf2E0ze4p6NHPTnyvlmiNO2lOUPd1T6F8Sk85KU1hLGx0x13RYzYyxVWyi/oQ1WbekV9DpS0I1KjjnQc0+fU5K1pDosHqTTMZYJhNeYqaUTzrqny1G+57FVcj5lt2POv8OKkiNPnOLLllGXfQ4VHKzg9Pi0eahnsyOD28K8uaesY6mtcrNrWlRqSoxkoPGCi3PTu7tQUaVDGepxVqTjibWkhLp1+eTWLHQMI05IILMgCCrLEMCnUMdSQjPqW6FZbllsBQze5q9zOW4VykkA05pIAAAAAQAUAAQWR223uHEjutv0witss8RgfWV9LH6Hytis8SR9VeaWX0ND5C5f50jn6m1f9WXzMuphtaJotiiLrYIpMxZtMxe4FQAAAAUBIRB2cKpyqX1LEW0pan6HQWYI+U/DsowtZ+zq3ufW2vuJmK9HHPrW8KaWjRdRgteUZJSyGsV1lsjKu1CPqdWOWJ5t5XhFSc3jHclWR221WElp06GlWulBrJ8rR47SV14cYVf/AH8vsnbVvHLGGTya8PbWtc/muJk3zPKPnuI3d5UuJQtnyJf1Yzkzsq/GadZKpyVIdcoa1mPrqFTw5LLymenTxJZPmqt04wTlo+x79nLnoxl3RZWeucb8oxoWWxDNMM5bHLVaOmcjmqtMjcjkqvOh5l37kkd9eWMnm3MspkV5dwvEoPJzcLqulRqRWjbwdrjmMkcNjKKu2uXK5tiud+vSt7ZyqJy6np3drFWDW7SyXpUVKGUtS9ziFjUk98YJG78fNyQRaS0Ko6PJTAwWyQ2EVwVaLORVsKza1JDJS0CM5otHYTQpgVktTOZrNYZnMK4wAacwAFAAgKAAIAEoglHfQX5Zwx3O+ksUiwTwxZ4kfTcQ0s/ofOcHWeIM+i4o8Wn0Kj4+t+pL5ma3L1PffzKIw0ui62KIv0ApMxe5rMxe4EAkgASQAqSSAQfS/hmKna111TyfU20sU0fGfhi5jSvJ0pvEakf+59fReyRivTxfTvhqjaK0MaeyNosjVS1oeDxulKWsEtej6nsVriME8s8a7rSq1PZFa42PFkqtKSUox17HXSmpQ9ex0zs+elputSbWxipLO/Yzjd6eXWpz8VOK3eDqVCS0zqd/lVTk0uryvUVKTdNyW5cTWFCypTanPLw92z2rdxjT9jY8hVPCoR29Tand8kFH7FS+3tKacSsppHmUrvnaWcFndLKXMGcx1znlHNVnoVdXOzMKkmk8k1qMLiZwS9rJvWbepzlK5pvkl6HHbKP/ABGK2yjpuJe00cFGoocRg9feLHOvrrNOnLlk8nPxSWKNSC+I0pSkoqpLRdjj4hUboZlo5yyVevjznHQzwaN6Gbepp5jBGBqRhhDCKslplWmAeCUUaLRAiexWmXnsUp+8BNRGM9jepsYy2CuEEEm3MAAAAMACMjJBJKILIC8dzvgvyjhhudy0olg14Is3sme7xh4tvoeJwFZupv1PY428UPoVHyM/eZWO5Mt2REw00Rcoi/QDOZizaZiwIAAAABUgEAdFlJxvKTTx7SP0Clo4fI/PbRpXNNtNrmR+gRl7EGY6d/yejGWhSrdKKSM1PEDhrTT11bMOqbmpKpos76kwo8yzjRbiEOb54yaSuYWujaeQsa0rd/RovG0ccN986HDPikYR5m36YTZEOO4zusfEmiteNrvnRm/fXyKSglDlwedW45Jbxnjuotk+eqyXNGjPD/6QvhU16a1XRmMI4jjmWj0yVnxGlFPxIyhpjVGMbmjUWlaOnqRMrdSn4iaSbWhSpWWebVPsZqac04yys9yatNYzLZrX5kZros7hznhvKex1VcNYPMtJtSlGWMp5TOudwlp1CMrhYwjjqTSjp0ZavcJx32OCpcLkfzKazuauJYOOm/8AnoN/Ei0pOdSTfYrb+1d0l1yajF+vqHKUqW559/WdSaXSKN6ldU4as4J66vdlh+nqC2Kpal17pRe8acGjSwQ8Et6FQgUZcrIDOWxESZFIgWlsZxftGktjL+oK0nsZPY1l7piBwAA25pAAAqySCBkEACyLooi6CtIbo7npROKn7yOuf6QHX+Hv1pv1PS47L8o8/wDDq9uT9Ts4/L8s0j5diIYiYVpEv0KRL9CjKZkzWZkyCAAAJIJCgAIL0ZclWMux9xaV+e3h3wsnwkXiSPpeH1pQoxk3pjBnp1/N786mae5xRqJzlnJjK9i/Yb1kTSlFSikvQ5u7ZV5zzTorH/Uy1CwbqOpVq+LNbJ7I7LKlTaeUmy9W0UZc1N8oa3GkKS5VsRKhHOWk0c7uORuM0+ZdjJ3DeXzNGta12NRxssGNSM5PRYRzRunl65/+zRXilFcvXbI8k1ScIJtVFzemDn8lGs3y0IJfI7aMedty1eTshTf9MfqzJryJ8JjBcySi/Qo6aUcN7HuO25/fm36IhWlFf3UW/XUuM3p85UkqclyRbfot0cNatc82VQrNY+Bn2ippe6lH5IrUhJLPMXGdfBzqzhBqUJrPeJzVJ82Efb1HGTw0snPWtrequWpSg/XAR8pD9P5mNtKTu/YTfLqe5V4bTnGp4L5JJYx0PlKletbXFSKfLJPDNSMdXH0a5pvmqb9ERPY8GlxavD3sSR30uK0aqxL2X6mscr1v13x2Kf1ClVhNezJMiXvBGvQgJ6EZCpKyJKyCM5FEWkUzqBdmT3NMmc9wNP6TJmkdYmb3A88ZIBphbIIABkMkhgAQSBaJdFIl0Fa0/eR1Vf0jlpe8jorv8oD0vw4t36m3H5eyzH8O+79SePy3L/EfPsRKstEyrWJboUiXzoUZzMmaTMmQAAAJIAVIIAA9rh83UsXD13PFPpPwpYK8VR1JflxeMd2Z6+On5325qk9E3pNYx6m9O58SEZZwlufQXPALarTxBODWzTPm76xr8OqOLfNCWieNjGPQ92xu17sX9T1o1FKKXofHcNryecZUVuz6ShVzHm26JEF6qjzZxuzzrmShmWfkkejX1SwvkUocJlcT57htQ6QQNcFOjO4qLwoPMe2x6lPhuMc75UuiPUpW8KUFGKSS7F1TW72LieTkpUKNJeyte7N0kuhaphGLlj1KzWjSe2hnLTQhVYtaMzqVfUInONmRJxxrI5Ktwo51PMvL+KTTml9Sa1I24pXhRnBxe5w1b5LDzqeRd3dSrP8AKpVarXXlZ5da9qwm1UhKD7PQSWlsj33fpV566M+Y4nJTvqso7Nl6dedVt7I57n9Q3Jjl3djIAGnJeFScPdk19TeF9Xi/fz8zmBsetQ4t0qx+qO+lcU6qzCSZ83gtGTg8xbT9B4mvqCsjxKXEqsPe9pHXT4nTnpJcrM3mrrrkZPcnxoS1UkU8SOfeRnBqjOZKkks5Ik9ALQehSW5am9Cs9wPOJIBtlIAIgQSVAkkqWAtEuiiLIK2pe8ja4/TMaXvGty/ywPX/AA6vykZcelqzfgGlBHHx2Xtsv8R4xMSpaJlWkS/Qoi3QozmZl5mZBIIIckuoFgZuqiHW9C+NNahtIwdZlJVGyzk1tKp2PovwfxBUqlWhKWHJ8yPlstnVwzMeI27Umm6iWnzNXmYvNyv1m2k5R5mc3FbWF1RlGUU00dFD9GOOxM5wxhnmenXwtvHwqsqbeOWWGj27TxK8o+Gm0tPkaXPA6dzfKvTqOHxRxoz2ba2p29NQpxwkMXShb4is6tHXGPLuTBYLblZNyXtgFZNlRlNJmUkjZrJSUNckajnqU4S3WpzVLaMtpzX1O2cNGcVWsovAXKx8jRfv88/nIt5ehTj7FKEfoSqmmSKlVYIM5ci1UVlHkfiTh8L2xc4RXiU/ai+67HoTqpb9TCvd0Y0pRlLGg0s18RShyrBz3P6p6Eox8SfL7uXg865/WZuOPXqMgAbjmklEIlGoiSGSQaBFgiAhzPoxlrqwNwJ8SfSTNadxVhvLK7MyGSYa9W2rxqLfEuxrM8WMnGScXhnbRvOZYqb9zN5alZIkjIIiSSAAIJAEEoEoCyLIqiyA2pe8Xun7JWjuLr3QPd4Hpbr5Hn8bf5jPS4NpbL5HlcZf5rL/ABHllolC0TKtUW6FIlm8IozmzKU0iKtTLwjE1OU1aVVvYo23uS0FsbxNQGg9ABBJBZbAUyaUajp1YTi9YtNGctzr4TaSveI0aMVlOScvRdSWtR+o2027SlJ6ZimXjq9Fr3M4xcsJaRWiOmMVFHlelaEeReprBZKQjl5ZvFYAlE4CDZUMFZFslSiMFZYSE5Y2M5ywiNRFSS5Xk8S9ny1EenWloeZXpOcskb1lOuoxOed0Li0qVsKNTk9SKfCIvWrWnL0zgM+nLWuklrJHmXtac6fuyx3xofTKytaMfZpx+ZzV406sZU2k1gQfLRj7J5VV5qyfqezcrwFUT/pyjw3q8m+XHsAJRtzSixVIk6RmgBIDJCDJ2QEPcBAAQSOgBkBsEV0kkAwiQQAJAIAnJKIJQF0WRVF4kVtS3IutiaO5W6eqKPouErFqvkeLxd/nM9rhulr9Dw+Kv85lqT64C0SheJlV0ZV5taI0ykss5qr5ss3zErJSy9SxmtzRG4lESRsySoiexC2LPYonrgipZMSGFuAcXJ4Sy2fefhbg/krZVqsfzqiz8l2PK/C3BvMVFeV4+xH3E+r7n3FOGEjh+nX8jv8Anz/aRWPmawhl5ZaMUi62OTqskkXRmmXRUHoE8kE7IonZGc5YEp4RzzmFxaUsmc5NhakNJIi6zaKypZRdbludJAedVSg2c8rjleCt9WbrNLRHn1a6jlt6ha6a1zrvocVS8jSefQ4695uk9Tk1k+eb+hU1zcUrZT7zZ5Zvd1fErPstDA6R5+rtCUQSjUZWRJBJtlBPQgl7bAQtZEy1ZVEgAR1JAEMlshEUSABR0EmHjS7IeNLsjmY3Bh40uyHjS7IGNwYeNLsh40uyBjoLI5vHl2Q8xPtEg60XicXmZ9okq7qLpED06Rnce8l6nHG+qx2jD7MrO8qTabjHT0KPsbHS0+h8/wASea7M6f4gu6dPkjTo49Yv/c4617UrT5pRgn6IWjQsnocvjy7IeYnjZCDac8v0KZyZOo32IVRrsdJ1IzlTJYkWRm5NjmZPKNY1YTM/EeOg5mXziY23Rk9JDxH6FXJsl6hI03PR4NwqpxG7jDVU1rJ+h5kajj0T+Z6/D/xNdcOpeHQtrXHVyjJt/wDyM9df41zJ/X6Ja20belGlTilGKwkdcI6H56vx1xNbULP9kv5Fl+PeKL/09n+yX8jj41284/RIxLcp+df2+4r/AJez/ZL+RP8A4gcV/wAvZfsn/IeNPOP0XDJyfnP9v+K/5ey/ZL+RH9vuK/5ez/ZL+RfGnnH6M3gq5H50/wAe8Uf9xZ/sl/Ij+3fFP8Cz/ZL+QynnH6DJmeMvU+B/t1xP/AtP2S/kP7c8T/wLT9kv5Dxp5x9/lJGM5nwb/G3En/c2v7JfyKv8ZcRf9za/tl/IeNPOPuefLM6lVRTyz4j+2HEP8G2/bL+RSf4rvp70rf8AbL/cnjWv+nL3b+vq3k+eurypUqOFFZ7s563GbmsmpRprPZP/AHMaXEKlJezSpP5p/wC5Zyl/SV30YTxmpHHqTd1Y0KDb3ekV6nJLi9eUOXw6SXon/ucle4ncTUp40WElsiyM3v16ZZywAacglEE5NSiSy2KZHMy+UTFiZbFOZhybL5QxZbCRVSaDeSeUMSSVyOZjyhgSiMjI2CwK5YyPKGIABhQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB/9k=\n",
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"450\"\n",
       "            src=\"https://www.youtube.com/embed/wkYvdfkVmlI\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.YouTubeVideo at 0x7ff5d0b05520>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "YouTubeVideo(\"wkYvdfkVmlI\",width=800, height=450)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> __Exercise 1: TF-IDF and the Computational Social Science communities.__ The goal for this exercise is to find the words > charachterizing each of the communities of Computational Social Scientists.\n",
    "> What you ned for this exercise: \n",
    ">    * The assignment of each author to their network community, and the degree of each author (Week 6, Exercise 4). This can be stored in a dataframe or in two dictionaries, as you prefer.  \n",
    ">    * the tokenized _abstract_ dataframe (Week 7, Exercise 2)\n",
    ">\n",
    "> 1. First, check out [the wikipedia page for TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf). Explain in your own words the point of TF-IDF. \n",
    ">   * What does TF stand for? \n",
    ">   * What does IDF stand for?\n",
    "> 2. Now, we want to find out which words are important for each *community*, so we're going to create several ***large documents, one for each community***. Each document includes all the tokens of abstracts written by members of a given community. \n",
    ">   * Consider a community _c_\n",
    ">   * Find all the abstracts of papers written by a member of community _c_.\n",
    ">   * Create a long array that stores all the abstract tokens \n",
    ">   * Repeat for all the communities. \n",
    "> __Note:__ Here, to ensure your code is efficient, you shall exploit ``pandas`` builtin functions, such as [``groupby.apply``](https://pandas.pydata.org/docs/reference/api/pandas.core.groupby.GroupBy.apply.html) or [``explode``](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.explode.html).\n",
    "> 3. Now, we're ready to calculate the TF for each word. Use the method of your choice to find the top 5 terms within the __top 5 communities__ (by number of authors). \n",
    ">   * Describe similarities and differences between the communities.\n",
    ">   * Why aren't the TFs not necessarily a good description of the communities?\n",
    ">   * Next, we calculate IDF for every word. \n",
    ">   * What base logarithm did you use? Is that important?\n",
    "> 4. We're ready to calculate TF-IDF. Do that for the __top 9 communities__ (by number of authors). Then for each community: \n",
    ">   * List the 10 top TF words \n",
    ">   * List the 10 top TF-IDF words\n",
    ">   * List the top 3 authors (by degree)\n",
    ">   * Are these 10 words more descriptive of the community? If yes, what is it about IDF that makes the words more informative?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " __Exercise 2: The Wordcloud__. It's time to visualize our results!\n",
    "\n",
    "> * Install the [`WordCloud`](https://pypi.org/project/wordcloud/) module. \n",
    "> * Now, create word-cloud for each community. Feel free to make it as fancy or non-fancy as you like.\n",
    "> * Make sure that, together with the word cloud, you print the names of the top three authors in each community (see my plot above for inspiration). \n",
    "> * Comment on your results. What can you conclude on the different sub-communities in Computational Social Science? \n",
    "> * Look up online the top author in each community. In light of your search, do your results make sense?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}