{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Scraping StackOverflow\n", "\n", "In this project, we will be scraping StackOverflow website and:\n", "\n", "- [Goal 1: List Most mentioned/tagged languages along with their tag counts](#Goal1)\n", "- [Goal 2: List Most voted questions along with with their attributes (votes, summary, tags, number of votes, answers and views)](#Goal2)\n", "\n", "We will divide our project into the above mentioned two goals.\n", "\n", "Before starting our project, we need to understand few basics regarding Web Scraping." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Web Scraping Basics\n", "\n", "Before starting our project, we need to understand few basics regarding Web Pages and Web Scraping.\n", "\n", "When we visit a page, our browser makes a request to a web server. Most of the times, this request is a [GET Request](https://realpython.com/lessons/the-get-request/). Our web browser then receives a bunch of files, typically (HTML, CSS, JavaScript). HTML contains the content, CSS & JavaScript tell browser how to render the webpage. So, we will be mainly interested in the HTML file.\n", "\n", "### HTML: \n", "HTML has elements called [tags](https://www.w3schools.com/html/html_elements.asp), which help in differentiating different parts of a HTML Document. Different types of tags are:\n", "* `html` - all content is inside this tag\n", "* `head` - contains title and other related files\n", "* `body` - contains main cotent to be displayed on the webpage\n", "* `div` - division or area of a page\n", "* `p` - paragraph\n", "* `a` - links\n", "\n", "We will get our content inside the body tag and use p and a tags for getting paragraphs and links.\n", "\n", "HTML also has [class and id properties](https://www.codecademy.com/articles/classes-vs-ids). These properties give HTML elements names and makes it easier for us to refer to a particular element. `Class` can be shared among multiple elements and an element can have moer then one class. Whereas, `id` needs to be unique for a given element and can be used just once in the document.\n", "\n", "### Requests\n", "The requests module in python lets us easily download pages from the web.
\n", "We can request contents of a webpage by using `requests.get()`, passing in target link as a parameter. This will give us a [response object](https://realpython.com/python-requests/#the-response). \n", "\n", "### Beautiful Soup\n", "[Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) library helps us parse contents of the webpage in an easy to use manner. It provides us with some very useful methods and attributes like:\n", "* `find()`, `select_one()` - retuns first occurence of the tag object that matches our filter\n", "* `find_all()`, `select()` - retuns a list of the tag object that matches our filter\n", "* `children` - provides list of direct nested tags of the given paramter/tag\n", "\n", "These methods help us in extracting specific portions from the webpage.\n", "\n", "***Tip: When Scraping, we try to find common properties shared among target objects. This helps us in extracting all of them in just one or two commands.***\n", "\n", "For e.g. We want to scrap points of teams on a league table. In such a scenario, we can go to each element and extract its value. Or else, we can find a common thread (like **same class, same parent + same element type**) between all the points. And then, pass that common thread as an argument to BeautifulSoup. BeautifulSoup will then extract and return the elements to us." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Goal 1: Listing most tagged Languages\n", "\n", "Now that we know the basics of Web Scraping, we will move towards our first goal.\n", "\n", "In Goal 1, we have to list most tagged Languages along with their Tag Count. First, lets make a list of steps to follow:\n", "\n", "- [1. Download Webpage from stackoverflow](#1.1)\n", "- [2. Parse the document content into BeautifulSoup](#1.2)\n", "- [3. Extract Top Languages](#1.3)\n", "- [4. Extract their respective Tag Counts](#1.4)\n", "- [5. Put all code together and join the two lists](#1.5)\n", "- [6. Plot Data](#1.6)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's import all the required libraries and packages" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np # linear algebra\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "import requests # Getting Webpage content\n", "from bs4 import BeautifulSoup as bs # Scraping webpages\n", "import matplotlib.pyplot as plt # Visualization\n", "import matplotlib.style as style # For styling plots\n", "from matplotlib import pyplot as mp # For Saving plots as images\n", "\n", "# For displaying plots in jupyter notebook\n", "%matplotlib inline \n", "\n", "style.use('fivethirtyeight') # matplotlib Style " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Downloading Tags page from StackOverflow\n", "\n", "We will download the [tags page](https://stackoverflow.com/tags) from [stackoverflow](https://stackoverflow.com/), where it has all the languages listed with their tag count." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "200" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Using requests module for downloading webpage content\n", "response = requests.get('https://stackoverflow.com/tags')\n", "\n", "# Getting status of the request\n", "# 200 status code means our request was successful\n", "# 404 status code means that the resource you were looking for was not found\n", "response.status_code" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Parsing the document into Beautiful Soup" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "bs4.element.Tag" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Parsing html data using BeautifulSoup\n", "soup = bs(response.content, 'html.parser')\n", "\n", "# body \n", "body = soup.find('body')\n", "\n", "# printing the object type of body\n", "type(body)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Extract Top Languages\n", "\n", "In order to acheive this, we need to understand HTML structure of the document that we have. And then, narrow down to our element of interest.\n", "\n", "\n", "One way of doing this would be manually searching the webpage (hint: print `body` variable from above and search through it).
\n", "Second method, is to use the browser's Developr Tools. \n", "\n", "We will use this second one. On Chrome, open [tags page](http://stackoverflow.com/tags?tab=popular) and right-click on the language name (shown in top left) and choose **Inspect**.\n", "\n", "![Image for Reference](https://github.com/nveenverma/Projects/blob/master/Exploring%20StackOverflow/tags.png?raw=true)\n", "*
Image for Reference
*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that the Language name is inside `a` tag, which in turn is inside a lot of div tags. This seems, difficult to extract. Here, the [class](#classes) and [id](#id), we spoke about earlier comes to our rescue. \n", "\n", "If we look more closely in the image above, we can see that the `a` tag has a class of `post-tag`. Using this class along with `a` tag, we can extract all the language links in a list." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[javascript,\n", " java]" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lang_tags = body.find_all('a', class_='post-tag')\n", "lang_tags[:2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, using [list comprehension](https://towardsdatascience.com/python-basics-list-comprehensions-631278f22c40), we will extract all the language names." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['javascript', 'java', 'c#', 'php', 'android']" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "languages = [i.text for i in lang_tags]\n", "languages[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Extract Tag Counts\n", "\n", "To extract tag counts, we will follow the same process.\n", "\n", "On Chrome, open [tags page](http://stackoverflow.com/tags) and right-click on the tag count, next to the top language (shown in top left) and choose **Inspect**.\n", "\n", "![Image for Reference](https://github.com/nveenverma/Projects/blob/master/Exploring%20StackOverflow/tag_count.png?raw=true)\n", "*
Image for Reference
*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here, the tag counts are inside `span` tag, with a class of `item-multiplier-count`. Using this class along with `span` tag, we will extract all the tag count spans in a list." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[1824582,\n", " 1557391]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tag_counts = body.find_all('span', class_='item-multiplier-count')\n", "tag_counts[:2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, using [list comprehension](https://towardsdatascience.com/python-basics-list-comprehensions-631278f22c40), we will extract all the Tag Counts." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[1824582, 1557391, 1320273, 1289585, 1200130]" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "no_of_tags = [int(i.text) for i in tag_counts]\n", "no_of_tags[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Put all code together and join the two lists\n", "\n", "We will use [Pandas.DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) to put the two lists together.
In order to make a DataFrame, we need to pass both the lists (in dictionary form) as argument to our function." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# Function to check, if there is any error in length of the extracted bs4 object\n", "def error_checking(list_name, length):\n", " if (len(list_name) != length):\n", " print(\"Error in {} parsing, length not equal to {}!!!\".format(list_name, length))\n", " return -1\n", " else:\n", " pass\n", " \n", "\n", "def get_top_languages(url):\n", " # Using requests module for downloading webpage content\n", " response = requests.get(url)\n", "\n", " # Parsing html data using BeautifulSoup\n", " soup = bs(response.content, 'html.parser')\n", " body = soup.find('body')\n", "\n", " # Extracting Top Langauges\n", " lang_tags = body.find_all('a', class_='post-tag')\n", " error_checking(lang_tags, 36) # Error Checking\n", " languages = [i.text for i in lang_tags] # Languages List\n", "\n", " # Extracting Tag Counts\n", " tag_counts = body.find_all('span', class_='item-multiplier-count')\n", " error_checking(tag_counts, 36) # Error Checking\n", " no_of_tags = [int(i.text) for i in tag_counts] # Tag Counts List\n", "\n", " # Putting the two lists together\n", " df = pd.DataFrame({'Languages':languages,\n", " 'Tag Count':no_of_tags})\n", "\n", " return df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plot Data" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
LanguagesTag Count
0javascript1824582
1java1557391
2c#1320273
3php1289585
4android1200130
\n", "
" ], "text/plain": [ " Languages Tag Count\n", "0 javascript 1824582\n", "1 java 1557391\n", "2 c# 1320273\n", "3 php 1289585\n", "4 android 1200130" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "URL1 = 'https://stackoverflow.com/tags'\n", "\n", "df = get_top_languages(URL1)\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we will plot the Top Languages along with their Tag Counts." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkQAAAD2CAYAAAA3fZgNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmYZGV5/vHvzSDIqDAgAuMMAobWBFARFFAxLihbYsAFGVwYAUM0YDQx+QlqJAImGoNcahSXMDAQwiKoEIMiIoiIyC6rpIcl0jAwkVkkrAL374/zFlPT9lI9U92nTtf9ua666pz3bE81Q/dT7yrbRERERPSzteoOICIiIqJuSYgiIiKi7yUhioiIiL6XhCgiIiL63tp1B9DLVqxYkR7nERER08wGG2yg4WWpIYqIiIi+l4QoIiIi+l4SomlgcHCw7hBWW1Njb2rc0NzYmxo3NDf2psYNzY29qXFDs2OHJEQRERERSYgiIiIikhBFRERE30tCFBEREX0vCVFERET0vUzMWKNZJ93TpTvNhMu6c6/lB83pyn0iIiKaJDVEERER0feSEEVERETfm5KESNICSUsk3dRWdqak68vrLknXl/ItJT3SduxrbdfsKOlGSYskfUmSSvlGki6UNFjeNyzlKuctknSDpB3a7jW/nD8oaf5U/BwiIiKiN01VDdHJwJ7tBbb3t7297e2Bc4Bvtx2+vXXM9gfayk8ADgUGyqt1zyOAi2wPABeVfYC92s49tFyPpI2Ao4CdgZ2Ao1pJVERERPSfKUmIbF8KLB3pWKnleSdw+lj3kDQbWN/2z20bOAXYtxzeB1hYthcOKz/FlSuAWeU+ewAX2l5qexlwIcMStoiIiOgfvTDK7LXA/bbbF0HZStJ1wG+BT9r+KTAHGGo7Z6iUAWxqezGA7cWSNinlc4C7R7hmtPJRTc4aLTMn4Z5rpo61aJq6/k1T44bmxt7UuKG5sTc1bmhu7E2NG3o79oGBgTGP90JCdACr1g4tBl5g+wFJOwLflbQtoBGu9Tj3Hu2aCd9rvB/kaunSUPlumpTPOYbBwcEpf2Y3NDVuaG7sTY0bmht7U+OG5sbe1Lih2bFDzaPMJK0NvA04s1Vm+zHbD5Tta4DbgRdR1eLMbbt8LnBv2b6/NIW1mtaWlPIhYPMRrhmtPCIiIvpQ3cPu3wT8yvbTTWGSnidpRtl+IVWH6DtKk9iDknYp/Y4OBM4tl50HtEaKzR9WfmAZbbYLsKLc5wJgd0kbls7Uu5eyiIiI6ENT0mQm6XTg9cDGkoaAo2yfCMzj9ztT/zFwtKQngCeBD9hudcj+INWItfWA75cXwGeBsyQdAvwa2K+Unw/sDSwCHgYOArC9VNIxwFXlvKPbnhERERF9ZkoSItsHjFL+vhHKzqEahj/S+VcD241Q/gCw2wjlBg4b5V4LgAVjxR0RERH9oe4ms4iIiIjaJSGKiIiIvpeEKCIiIvpeEqKIiIjoe0mIIiIiou8lIYqIiIi+l4QoIiIi+l4SooiIiOh7SYgiIiKi7yUhioiIiL6XhCgiIiL63pSsZRbTy6yT7uni3WbCZWt+v+UHzelCLBER0a9SQxQRERF9b0oSIkkLJC2RdFNb2T9IukfS9eW1d9uxIyUtknSbpD3ayvcsZYskHdFWvpWkX0galHSmpHVK+bplf1E5vuV4z4iIiIj+M1U1RCcDe45Qfrzt7cvrfABJ2wDzgG3LNV+VNEPSDOArwF7ANsAB5VyAz5V7DQDLgENK+SHAMttbA8eX80Z9Rpc/c0RERDTElCREti8FlnZ4+j7AGbYfs30nsAjYqbwW2b7D9uPAGcA+kgS8ETi7XL8Q2LftXgvL9tnAbuX80Z4RERERfajuTtWHSzoQuBr4qO1lwBzgirZzhkoZwN3DyncGngsst/3ECOfPaV1j+wlJK8r5Yz1jRIODgxP7ZB2ZOQn3XDOdfc6mxt38Z3ZLU2NvatzQ3NibGjc0N/amxg29HfvAwMCYx+tMiE4AjgFc3o8DDgY0wrlm5Nosj3E+Yxwb65oRjfeDXC1dGF3VbR19zqbG3UWDg4NT/sxuaWrsTY0bmht7U+OG5sbe1Lih2bFDjaPMbN9v+0nbTwHfZGWT1RCwedupc4F7xyj/DTBL0trDyle5Vzm+AVXT3Wj3ioiIiD5UW0IkaXbb7luB1gi084B5ZYTYVsAAcCVwFTBQRpStQ9Up+jzbBi4G3lGunw+c23av+WX7HcCPy/mjPSMiIiL60JQ0mUk6HXg9sLGkIeAo4PWStqdqqroL+AsA2zdLOgu4BXgCOMz2k+U+hwMXADOABbZvLo/4GHCGpGOB64ATS/mJwKmSFlHVDM0b7xkRERHRf6YkIbJ9wAjFJ45Q1jr/M8BnRig/Hzh/hPI7GGGUmO1Hgf0m8oyIiIjoP5mpOiIiIvpeEqKIiIjoe0mIIiIiou8lIYqIiIi+l4QoIiIi+l4SooiIiOh7SYgiIiKi7yUhioiIiL7XUUIk6TWStijbm0j6uqSvStp4csOLiIiImHydzlT9DeBPyvZxwHrAI6X8bZMQV8SkmHXSPV2600y4rDv3Wn7QnK7cJyIiVl+nCdFc23dJmgHsBbwQeAzo1l+XiIiIiNp0mhA9VJrHtgNus/3bsuL8MyYvtIiIiIip0WlCdALwC+CZwBGlbBfgvzu5WNIC4E+BJba3K2WfB94CPA7cDhxke7mkLYFbgdvK5VfY/kC5ZkfgZKomu/OBD9u2pI2AM4EtgbuAd9peJknAF4G9gYeB99m+ttxrPvDJ8oxjbS/s8GcRMeW619QH3WruS1NfREwnHXWqtn0M8FbgzbZPLcX/Cxza4XNOBvYcVnYhsJ3tl1IlVke2Hbvd9vbl9YG28hPKMwfKq3XPI4CLbA8AF7Eyadur7dxDy/WUBOooYGdgJ+AoSRt2+FkiIiJimul0lNmZtm+wfUurzPatwN91cr3tS4Glw8p+aPuJsnsFMHecGGYD69v+uW0DpwD7lsP7AK0anoXDyk9x5QpgVrnPHsCFtpfaXkaVnA1P2CIiIqJPdNpkttco5d1KIg6mavJq2UrSdcBvgU/a/ikwBxhqO2eolAFsansxgO3FkjYp5XOAu0e4ZrTyUQ0ODk7oA3Vm5iTcc8109jmbGjc0N/amxt38Z3ZLU2NvatzQ3NibGjf0duwDAwNjHh8zIZL08bL5jLbtlheyaoKyWiR9AngCOK0ULQZeYPuB0mfou5K2BTTC5R7v9qNcM+F7jfeDXC1dGrbdTR19zqbGDc2Nvalxd9Hg4OCUP7Nbmhp7U+OG5sbe1Lih2bHD+DVELynvM9q2oUoe7gf2X5OHl47NfwrsVprBsP0Y1ZB+bF8j6XbgRVTJV3uz2lzg3rJ9v6TZpXZoNrCklA8Bm49wzRDw+mHll6zJZ4mIiIjmGjMhsn0AgKQP2f5yNx8saU/gY8DrbD/cVv48YKntJyW9kKpD9B22l0p6UNIuVCPeDgRaMZ0HzAc+W97PbSs/XNIZVB2oV5Sk6QLgH9s6Uu/Oqp26IyIioo901IfI9pclzQS2Bp497Njl410v6XSqGpmNJQ1RjfA6ElgXuLAaHf/08Po/Bo6W9ATwJPAB260O2R9k5bD775cXVInQWZIOAX4N7FfKz6cacr+Iatj9QSXmpZKOAa4q5x3d9oyI6KLMDh4RTdBRQiTpAODrVE1lD7cdMvD88a5v1TQNc+Io554DnDPKsaupJoccXv4AsNsI5QYOG+VeC4AFo0cdEf0scz9F9JdOR5n9M3Cg7e9OZjARERERdehoHiKqpq3/nMxAIiIiIurSaUJ0PB1OwhgRERHRNJ02mR0MbCHpY1RLdjzN9ou6HlVERETEFOo0ITp8UqOIiIiIqFGnw+4vmOxAIiIiIurS6bD74ct2PM32P3YvnIiIiIip12mT2UuG7W8GvIqMPIuIiIhpoNMms9+bWFHSW4B9uh5RRERExBTrdNj9SP4LeEe3AomIiIioS6d9iIYvzzETeDewuOsRRUREREyxTvsQDVGtW6ay/zvgBspiqRERERFN1mmT2XpUtULrldczbb/S9hWdPkjSAklLJN3UVraRpAslDZb3DUu5JH1J0iJJN0jaoe2a+eX8QUnz28p3lHRjueZLkrS6z4iIiIj+0lFCZPsxqlqhlwF7A6+QNNH+RycDew4rOwK4yPYAcFHZB9gLGCivQ4EToEpugKOAnYGdgKNaCU4559C26/ZcnWdERERE/+koqZG0NXAT8D3gU1Qdqm+SNNDpg2xfCiwdVrwPsLBsLwT2bSs/xZUrgFmSZgN7ABfaXmp7GXAhsGc5tr7tn9s2cMqwe03kGREREdFnOu1D9FXgP4B/tP1UqR06kqpW5U1r8PxNbS8GsL1Y0ialfA5wd9t5Q6VsrPKhEcpX5xkjdhQfHByc2CfryMxJuOea6exzNjVuaG7sTY0bmht7U+Nu/jO7pamxNzVu6O3YBwbGrsPpNCF6BbC37acASlL0OeCjaxbeqDRCmVejfHWeMaLxfpCr5bJ7un/PNdTR52xq3NDc2JsaNzQ39qbG3UWDg4NT/sxuaWrsTY0bmh07dN6pejHVzNTtdgHuW8Pn399qpirvS0r5ELB523lzgXvHKZ87QvnqPCMiIiL6TKcJ0aeA/5J0sqRPSzqZqh/Rp9bw+ecBrZFi84Fz28oPLCPBdgFWlGavC4DdJW1YOlPvDlxQjj0oaZcyuuzAYfeayDMiIiKiz3S6dMc5km4D5lHVqgwBu9q+sdMHSTodeD2wsaQhqtFinwXOknQI8Gtgv3L6+VSj2RYBD1PmO7K9VNIxwFXlvKNttzpqf5BqJNt6wPfLi4k+IyKi6Wad1K3mvpldazpcftCc8U+KqNG4CZGkdWw/bvsm4JPt5RN50EjroRW7jXCugcNGuc8CYMEI5VcD241Q/sBEnxERERH9ZcwmM0kfAk4d5fApkv6y+yFFRERETK3x+hAdRNW0NZKjgEO6G05ERETE1BsvIdrS9q9GOmD7NmCr7ocUERERMbXGS4ieapvIcBWlfLy5fiIiIiJ63ngJ0U+Avxrl2IfK8YiIiIhGG2+U2d8Dl0t6MXA21QSNs4G3A28GXj254UVERERMvjFriGzfQjUjNcDXgEvKu4BX2b51UqOLiIiImALjzkNUOlXvByBprdZ6ZhERERHTRadLdwDVoq6TFUhEREREXTpd7T4iImJSdW/JEejWsiNZcqR/TKiGKCIiImI6SkIUERERfa+jJjNJ7xrl0GNUK99fY/uJiT68DOc/s63ohcCngFnAnwP/W8o/bvv8cs2RVEuGPAn8le0LSvmewBeBGcC/2f5sKd8KOAPYCLgWeK/txyWtC5wC7Ag8AOxv+66JfoaIiIhovk77EH0I2AFYDtwDzKFKWm4EtgAekvRW29dN5OFl+Y/tASTNKPf+DtUaasfb/pf28yVtA8wDtgWeD/xI0ovK4a9QzY00BFwl6bwybcDnyr3OkPQ1qmTqhPK+zPbWkuaV8/afSPwRERExPXTaZHYF8ElgM9s7AJsBnwAuLdunAl9ew1h2A263/T9jnLMPcIbtx2zfCSwCdiqvRbbvsP04VY3QPpIEvJFqUkmAhcC+bfdaWLbPBnYr50dERESf6TQhmg98wbYByvvxwPtsPwl8BthuDWOZB5zetn+4pBskLZC0YSmbA9zdds5QKRut/LnA8rbmvFb5Kvcqx1eU8yMiIqLPdNpk9htgd+D7bWVvpup7A7AOVZ+e1SJpHeDPgCNL0QnAMVSLxx4DHAccTDVD9nBm5MTOY5zPOMd+z+Dg4GiH1sDMSbjnmunsczY1bmhu7E2NG5obe1PjhubG3tS4m//Mbunl2AcGBsY83mlC9NfAWZKupKpV2ZyqmeqAcvzVwNdXM0aAvYBrbd8P0HoHkPRN4Htld6g8u2UucG/ZHqn8N8AsSWuXWqD281v3GpK0NrABsHS0AMf7Qa6WLsyR0W0dfc6mxg3Njb2pcUNzY29q3NDc2JsadxcNDg5O+TO7pcmxQ4dNZrb/C9iaqm/OnVQjwwZsf68c/4Htj69BHAfQ1lwmaXbbsbcCN5Xt84B5ktYto8cGgCuBq4ABSVuV2qZ5wHmlae9i4B3l+vnAuW33ml+23wH8uNUkGBEREf2l45mqS63NN7sdgKSZVM1vf9FW/M+Stqdqwrqrdcz2zZLOAm4BngAOK32YkHQ4cAHVsPsFtm8u9/oYcIakY4HrgBNL+YnAqZIWUdUMzev2Z4uIiIhm6HQeorWA9wOvAzamrf+N7d3XJADbDzOsM7Pt945x/meoOnEPLz8fOH+E8juomveGlz9KWbQ2IiIi+luno8z+Bfhb4AbgNcBFVJMoXjlJcUVERERMmU4ToncCe9j+HPBked+HqjN1RERERKN1mhA9u0yECPCIpPVKH51XTFJcEREREVOm007Vv5K0o+1rqNYD+7ikFcDiyQstIiIiYmp0mhD9DSs7Un+UarTZs4EPTEZQEREREVNpzIRI0gG2T7d9eavM9q3ArpMeWURERMQUGa+G6Ousur5YREREtJl1Ujdn2J7ZtRm7lx80Z/yT4mnjJURZ/T0iImKa6sVkrq5EbryEaIakNzBGYmT7x90NKSIiImJqjZcQrUu1xMVoCZGpJmiMiIiIaKzxEqKHbCfhiYiIiGmt04kZIyIiIqat8RKidKqOiIiIaW/MhMj2c6YiCEl3SbpR0vWSri5lG0m6UNJged+wlEvSlyQtknSDpB3a7jO/nD8oaX5b+Y7l/ovKtRrrGREREdFfeqnJ7A22t7fdWh/tCOAi2wPARWUfYC9goLwOBU6AKrkBjgJ2BnYCjmpLcE4o57au23OcZ0REREQf6aWEaLh9gIVleyGwb1v5Ka5cAcySNBvYA7jQ9lLby4ALgT3LsfVt/9y2gVOG3WukZ0REREQf6XQts8lm4IeSDHzd9jeATW0vBrC9WNIm5dw5wN1t1w6VsrHKh0YoZ4xn/J7BwcHV/WxjmDkJ91wznX3OpsYNzY29qXFDc2NvatzQ3NgTdzc1NfbJ+XsLAwMDYx7vlYToNbbvLQnJhZJ+Nca5I3X09mqUT8h4P8jV0qXp2bupo8/Z1LihubE3NW5obuxNjRuaG3vi7qqmxj4pf2870BNNZrbvLe9LgO9Q9QG6vzR3Ud6XlNOHgM3bLp8L3DtO+dwRyhnjGREREdFHak+IJD1L0nNa28DuwE3AeUBrpNh84NyyfR5wYBlttguwojR7XQDsLmnD0pl6d+CCcuxBSbuU0WUHDrvXSM+IiIiIPtILTWabAt8pI+HXBv7D9g8kXQWcJekQ4NfAfuX884G9gUXAw8BBALaXSjoGuKqcd7TtpWX7g8DJwHrA98sL4LOjPCMiIiL6SO0Jke07gJeNUP4AsNsI5QYOG+VeC4AFI5RfDWzX6TMiIiKiv9TeZBYRERFRtyREERER0feSEEVERETfS0IUERERfS8JUURERPS9JEQRERHR95IQRURERN9LQhQRERF9LwlRRERE9L0kRBEREdH3khBFRERE30tCFBEREX2v1oRI0uaSLpZ0q6SbJX24lP+DpHskXV9ee7ddc6SkRZJuk7RHW/mepWyRpCPayreS9AtJg5LOlLROKV+37C8qx7ecuk8eERERvaTuGqIngI/a/iNgF+AwSduUY8fb3r68zgcox+YB2wJ7Al+VNEPSDOArwF7ANsABbff5XLnXALAMOKSUHwIss701cHw5LyIiIvpQrQmR7cW2ry3bDwK3AnPGuGQf4Azbj9m+E1gE7FRei2zfYftx4AxgH0kC3gicXa5fCOzbdq+FZftsYLdyfkRERPSZtesOoKU0Wb0c+AXwGuBwSQcCV1PVIi2jSpauaLtsiJUJ1N3DyncGngsst/3ECOfPaV1j+wlJK8r5vxkpvsHBwTX4dKOZOQn3XDOdfc6mxg3Njb2pcUNzY29q3NDc2BN3NzU19sn5ewsDAwNjHu+JhEjSs4FzgI/Y/q2kE4BjAJf344CDgZFqcMzINV0e43zGOfZ7xvtBrpbL7un+PddQR5+zqXFDc2NvatzQ3NibGjc0N/bE3VVNjX1S/t52oO4+REh6BlUydJrtbwPYvt/2k7afAr5J1SQGVQ3P5m2XzwXuHaP8N8AsSWsPK1/lXuX4BsDS7n66iIiIaIK6R5kJOBG41fYX2spnt532VuCmsn0eMK+MENsKGACuBK4CBsqIsnWoOl6fZ9vAxcA7yvXzgXPb7jW/bL8D+HE5PyIiIvpM3U1mrwHeC9wo6fpS9nGqUWLbUzVh3QX8BYDtmyWdBdxCNULtMNtPAkg6HLgAmAEssH1zud/HgDMkHQtcR5WAUd5PlbSIqmZo3mR+0IiIiOhdtSZEti9j5L48549xzWeAz4xQfv5I19m+g5VNbu3ljwL7TSTeiIiImJ5q70MUERERUbckRBEREdH3khBFRERE30tCFBEREX0vCVFERET0vSREERER0feSEEVERETfS0IUERERfS8JUURERPS9JEQRERHR95IQRURERN9LQhQRERF9r+8TIkl7SrpN0iJJR9QdT0REREy9vk6IJM0AvgLsBWwDHCBpm3qjioiIiKkm23XHUBtJrwL+wfYeZf9IANv/BLBixYr+/eFERERMUxtssIGGl/V1DREwB7i7bX+olEVEREQf6feE6PcyRCC1QhEREX1m7boDqNkQsHnb/lzg3tbOSFVqERERMf30ew3RVcCApK0krQPMA86rOaaIiIiYYn1dQ2T7CUmHAxcAM4AFtm+uOayIiIiYYn09yiwiIiIC0mTWSJI+10lZL5O0iaQXtF51xxMRIOlPJeXvQvSl1BA1kKRrbe8wrOwG2y+tK6ZOSfoz4Djg+cASYAvgVtvb1hpYByRtbPs3dcexuiRtBuxENZLyKtv31RzSqCS9bazjtr89VbH0E0n/DrwKOAc4yfatNYfUVyRt27RuG5JeA1xv+yFJ7wF2AL5o+39qDm3CkhA1iKQPAn8JvBC4ve3Qc4Cf2X5PLYFNgKRfAm8EfmT75ZLeABxg+9CaQxuVpLVsP9WeiEr6sO0v1h1bpyS9H/gU8GOq6SZeBxxte0GtgY1C0kllcxPg1VRxA7wBuMT2mAlTLyh/KP6BKulfm+rnbtsvrDOu8UhaHzgAOIgqeT4JON32g7UGNgpJNzLydCmtn3fPf1FsGenLbq+TdAPwMuClwKnAicDbbL+u1sBWQxKiBpG0AbAh8E9A+7prD9peWk9UEyPpatuvKInRy0uicaXtneqObTSSfgo8RPU//MHADcD3mvSLS9JtwKttP1D2nwtcbvvF9UY2NknfA/7c9uKyPxv4SkMSol8Bfw1cAzzZKm/9N+hlkjYG3gN8BLgV2Br4ku0v1xrYCCRtMdbxJtVUSLrO9svrjmMiWkmcpE8B99g+sYmJHfT5KLOmsb0CWEG15toOwK5U34x+BjQiIQKWS3o2cClwmqQlwBM1xzQm26+VNIvqD9tOwPuBF0k6A/iJ7RNqDbAzQ0D7N/wHWXWW9l61ZSsZKu4HXlRXMBO0wvb36w5iIkqT9kHAH1B929/J9hJJM6kSo55LiIYnPKWGqzF/2yQdRfV7XMCmJbEAwPbRtQXWuQfLslfvBV5b1gh9Rs0xrZbG/KOJlST9PfBOoNWP4iRJ37J9bI1hdWof4BGqb87vBjYAevp/ekk/BH4OPAV82fYySdcB/w/441qD69w9wC8knUv1y3cf4EpJfwNg+wt1BjeGSyRdAJxOFfc84OJ6Q+rYxZI+T/X/6WOtQtvX1hfSuN4OHG/70vZC2w9LOrimmDoi6S+ofpc8wsomNFN1Mehld7Vt/w5oTI1WsT/wLuBg2/eVQTKfrzmm1ZImswaSdCtVc9OjZX894Frbf1RvZOOT9NfAt2wP1R1Lp8q341cB/w5cDWxK1YRwDPBT21fXGF5HyrfQUdn+9FTFMlGlg/Vry+6ltr9TZzydkjRS4mbbb5zyYDpQvtlfYPtNdceyOiQNAq9q+MCHRjY1SdoUeGXZvdL2kjrjWV2pIWqmu4BnAo+W/XVZtZN1L1sfuEDSUuAM4Gzb99cc05hsPwxcJOk+22+Bpzty3g3Mp0qSelovJzzjKSPKGjeqzPYb6o5hImw/KelhSRuU5vmmuR14uO4g1lDjlouS9E6qGqFLqOL/sqS/s312rYGthtQQNZCk71Jl4xdSVQm/GbiMahg7tv+qvug6I+mlVFWtbweGmvCtVNILbd9Rtk+w/cG6Y+qUpBcBfwtsSdsXoR6urbjM9q6SHmTVEUStkUPr1xRax8ogiKNY2az6E6qRfT2bbEg6C9iF6nfLQ63yhvxOeTnViLhfsGoTZc/H3iJpo6YMkGkpA2Te3KoVkvQ8qlHEL6s3solLDVEzfae8Wi6pKY41sQS4D3iAamh1ExxVhtsvt/1BSRsCx9nu6b4VxbeArwH/RtuIp15le9fy/py6Y1kDC4CbqPr7QdXp9CSgl0fI/Vd5NdHXqaZnuJGqv1/jNC0ZKtYa1kT2AA2d9Dk1RDGlylxK+wPPA84GzrR9S71RdWakIbFNGSYr6RrbO9Ydx+qQ9DJW7UN0Q53xdErS9ba3H6+s15Q+iS+wfVvdsUyEpMttv7ruONaEpHNsv73uOCaiDBx4KdXAB6h+v99g+2P1RbV6GpnF9atSnY2kGyXdMPxVd3wd2gL4iO1tbR/VlGSoWKvUCgFV9TY9XssqaaMS539KOkzS7FZZKe9pkj4MnEZVi7gJ1VQNH6o3qo49ImnX1k6ZqPGRGuMZl6S3ANcDPyj720s6r96oOnaxpEOb9m98mF4fEfd7bP8d8A2qpOhlwDeamAxBaogaRdJs24tHm4isYROQbULVMRwA27+uMZyOSDoQOJKqZstUTSGfsX1qrYGNQdKdrJzjBIbN6NuAWZNvoBo59FDZfxbw8ybMPlxqtk6hmlpCVHOFvc/2L2sNbAySrqGaSf6SVs2npBttv6TeyMZX/q0P14SZwVtrOYqquXKvst2I34vTSU9/u41VlWRoBnBiEzohj6R8A/0Cw9YyA3p+LTPbp0i6muoPhqimp+/pGi7bW8HTzSB/ycrJPH9K1acUY+o+AAALYElEQVSo14lV+zw9SUNG4pTE52VlokBs/7bmkDrxhO0V0io/4qZ8a/6j1lQkLZKeOdrJPWQhK7+0bFH2Vcp6ctADTI+BD8MlIWqYaTA09liqUSyrrGVWc0wdKwlQTydBo1gI/Bb4Utk/oJS9c9QresNJVBNKtgYR7Eu1VlLPk7Qu1SjKLYG1W0lGj88+fJOkdwEzJA0AfwVcXnNMnbqcamHR8cp6Svv0DKVPYs8mQe2mycCHVSQhaqZHgRslNW5oLPA72w9IWkvVoqkXS/pc3UH1gRcPGwZ7cRku29Nsf0HSJVQ1WwIOsn1dvVF17FyqpXauoW0YeI/7EPAJqnhPBy6gmoC0Z0naDJgDrKdqSaOW9YGZ9UQVTZSEqJmaPDS2tZbZT2nIWmbTxHWSdrF9BYCknanWwOtZktaiGq2yHdDLy12MZq7tPesOYiLKJKSfKK+m2AN4HzAX+Je28gep+vw1yRfrDqCfpVN1A5WOpY/afrLszwDWLb/MelpZBuNRqm/776H6FndaQ+ffaIyy3MuLgVYnzRdQ9d16iqq9vyc7KUs6DTiyiZ1LJX2Dau27G+uOpVNluZHf+6PQhGYcSe+hin1LVn7Zd483UQIgaSHwYdvLy36T5jibNlJD1EwXAW8C/q/srwf8EOjZOThaHfCoVitv/cJt9dw8tizl8XnbX60lwOmvUTUVbWYDN0u6klWbh/+svpDGVpZ1MdXv14Mk3UHVBNXqbNqTyWfxt23bz6TqA9WUGtz3AsuoahMfHefcXvPSVjIE4GoB6Z6f32y6SULUTM+03UqGsP1/pealZ43XAU/Sc6k6QCYhmgRNmpJhmCauwfandQewumxfM6zoZ5J+UkswEzfH9h51B7Ga1pK0oe1l0Iw5zqaj/MCb6SFJO9i+FkDSjvT4hG/jKR2tX193HNFbbDflj/HTWsmnpFNtv7f9mKRTqWoyetKwiQzXAnYENqspnIm6XNJLmtRE2eY4qvhXmeOs3pD6T/oQNZCkV1KtFH9vKZoN7D/Ct7uIRhphbpNVNGGOE0nX2t6hbX8GcKPtbWoMa0zDJvJ8AriTakHay2oNbAzDmigHgCY1UT5N0jasnOPsol6f42w6Sg1RA9m+StIfUnWSFfAr27+rOayIrmk1rUo6mmoR4FOp/q2/G+jpeU8kHQl8nGoYeGsyRgGPUy1x0LNaE3k2TGObKNs1eI6zaSM1RA0kaT/gB7YflPRJqonHjm01oUVMF5J+YXvn8cp6kaR/st2oYd+S3jbWcdvfnqpYIqZaFndtpr8vydCuVHNwLAROqDmmiMnwpKR3S5pRJvN8N6su5dHLXiRp7zKfUlMcQjUT+LvL69+opsd4C9OkJiZiNE36HzVWav1B+BPgBNvnAuvUGE/EZHkXVQfT+8trv1LWBCdQJRWDkj5bmrl7nYFtbL/d9tspawzaPihz4sR0lyazBpL0PeAeqrmIWiPMrhy2NENE9ABJG1CtHfcJ4G7gm8C/92K/P0k3lZnBW/vts4VHTGtJiBqozDm0J9WIlUFJs4GX2P5hzaFFdJWk5wF/zqqzD9OU2ooyv9Z7qZqd7gVOo1qX7SW2X19jaCOS9K9UI7VOp6otOgAYtP2hWgOLmAJJiBpM0iZUs8kC0MTlDSLGIulyqnXvrqGt75Dtc2oLqkOSvg38IdUIuZNs39d27Grbr6gtuDFIeivw2rJ7qe3v1hlPxFRJQtRAkv6MaiKv5wNLqNal+pXtbWsNLKLLJF1ve/u641gdkvYGtgFeQ7Vm3GVUff56blmJ1tI6bfM/qe3wU0CW1olpLwlRA0n6JdUEXj+y/XJJbwAOsH1ozaFFdJWkY4HLbZ9fdywTJeks4LdUzWRQNT9taHu/+qJaPa2ldWy/uO5YIiZLEqIGalW3l8To5bafknSl7Z3qji2im0qNxbOoZh7+HStnH27CTNW/HD7QYaSyppA02/biuuOImCyZqbqZlkt6NnApcJqkJTRnReqIjtl+Tllfa4C2/nINcZ2kXWxfASBpZ+BnNce02pIMxXSXGqIGkvQsqqH2a1HNc7IBcJrtB2oNLKLLJL0f+DAwF7ge2IWq6Wa3WgPrgKRbqZbXaQ12eAFwK1WfnMassRXRL5IQNZCkvwa+ZXuo7lgiJlNZuPOVwBW2ty+TG37a9v41hzYuSVuMddz2/0xVLBExvjSZNdP6wAWSllKten+27ftrjiliMjxq+1FJSFrX9q8kNaJjbxKeiGbJ0h0NZPvTZYj9YVRD738i6Uc1hxUxGYYkzQK+C1wo6VyqCQ4jIroqTWYNJmkzqrWd5gHPSZ+EmM4kvY6qv9wPbD9edzwRMb0kIWogSR8E9geeB5wNnGn7lnqjioiIaK70IWqmLYCP2L6+7kAiIiKmg9QQNVjWMouIiOiOdKpuIElvkTQI3An8BLgL+H6tQUVERDRYEqJmOpZqgrr/tr0VsBsNngE3IiKibkmImul3ZVbqtSStZftioJErgkdERPSCdKpuptZaZj8la5lFRESssXSqbiBJM4FHqVb+fg/VzNWn2V5aa2ARERENlYSoQSRdZntXSQ8Crf9wKu9PAUuBz9v+ai0BRkRENFQSomlE0nOpVgJvxFpPERERvSIJ0TQjabbtxXXHERER0SRJiCIiIqLvZdh9RERE9L0kRBEREdH3khBFRERE30tCFBE9QdJdkt5UdxwR0Z+SEEVERETfS0IUET1L0oaSvifpfyUtK9tz245fIukYST+T9KCkH0rauO34gZL+R9IDkv6+vRZK0smSjm079/WShtr2j5B0e7nvLZLe2nZshqTjJP1G0p2SDpdkSWuX4xtIOlHSYkn3SDpW0oxybGtJP5G0olx/5uT+FCOiE0mIIqKXrQWcBGwBvAB4BPjXYee8CzgI2ARYB/hbAEnbAF8F3g3MBjYA5kzg2bcDry3XfRr4d0mzy7E/B/aiWlR5B2DfYdcupFpfcGvg5cDuwPvLsWOAHwIbAnOBL08gpoiYJEmIIqJn2X7A9jm2H7b9IPAZ4HXDTjvJ9n/bfgQ4iypJAXgH8J+2L7P9OPApVi5508mzv2X7XttP2T4TGAR2KoffCXzR9pDtZcBnW9dJ2pQqWfqI7YdsLwGOB+aVU35HleA93/ajti/r+AcSEZMmCVFE9CxJMyV9vTR7/Ra4FJjVan4q7mvbfhh4dtl+PnB364Dth4EHJvDsAyVdL2m5pOXAdkCrOW6Vew/b3gJ4BrC47dqvU9VgAfw/qjUIr5R0s6SDO40pIibP2nUHEBExho8CLwZ2tn2fpO2B61i5qPFYFpdrAZC0HvDctuMPATPb9jdrO3cL4JvAbsDPbT8p6fq25y6mau5q2bxt+27gMWBj208MD8r2fVRNbkjaFfiRpEttL+rgM0XEJEkNUUT0kmdIembrRdXP5hFguaSNgKMmcK+zgbdIerWkdaj6AbUnUtcDe0vaSNJmwEfajj2LqnntfwEkHURVQ9RyFvBhSXMkzQI+1jpQ1hL8IXCcpPUlrSXpDyS9rtxrv7aO4cvKc56cwOeKiEmQhCgiesn5VAlQ6zULWA/4DXAF8INOb2T7ZuBDwBlUNToPAkuoam8ATgV+CdxFlcCc2XbtLcBxwM+B+4GXAD9ru/03yzU3UNVYnU/VibqV2BxI1cH7Fqqk52yqjt0ArwR+Ien/gPOAD9u+s9PPFRGTI4u7RkRfkPRsYDkw0O0ERNJewNdsb9HN+0bE1EkNUURMW5LeUjpmPwv4F+BGqhqhNb3vepL2lrS2pDlUTXnfWdP7RkR9khBFxHS2D3BveQ0A89ydanFR9UlaRtVkdivVsP6IaKg0mUVERETfSw1RRERE9L0kRBEREdH3khBFRERE30tCFBEREX0vCVFERET0vf8PEocv0WMWXMkAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.figure(figsize=(8, 3))\n", "plt.bar(height=df['Tag Count'][:10], x=df['Languages'][:10])\n", "plt.xticks(rotation=90)\n", "plt.xlabel('Languages')\n", "plt.ylabel('Tag Counts')\n", "plt.savefig('lang_vs_tag_counts.png', bbox_inches='tight')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Goal 2: Listing most voted Questions\n", "\n", "Now that we have collected data using web scraping one time, it won't be difficult the next time.
\n", "In Goal 2 part, we have to list questions with most votes along with their attributes, like:\n", "> - Summary\n", "- Tags\n", "- Number of Votes\n", "- Number of Answers\n", "- Number of Views\n", "\n", "I would suggest giving it a try on your own, then come here to see my solution.\n", "\n", "Similar to previous step, we will make a list of steps to act upon:\n", "\n", "- [1. Download Webpage from stackoverflow](#2.1)\n", "- [2. Parse the document content into BeautifulSoup](#2.2)\n", "- [3. Extract Top Questions](#2.3)\n", "- [4. Extract their respective Summary](#2.4)\n", "- [5. Extract their respective Tags](#2.5)\n", "- [6. Extract their respective no. of votes, answers and views](#2.6)\n", "- [7. Put all code togther and join the lists](#2.7)\n", "- [8. Plot Data](#2.8)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Downloading Questions page from StackOverflow\n", "\n", "We will download the [questions page](https://stackoverflow.com/questions?sort=votes&pagesize=50) from [stackoverflow](https://stackoverflow.com/), where it has all the top voted questions listed.
\n", " \n", "Here, I've appended `?sort=votes&pagesize=50` to the end of the defualt questions URL, to get a list of top 50 questions." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "200" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Using requests module for downloading webpage content\n", "response1 = requests.get('https://stackoverflow.com/questions?sort=votes&pagesize=50')\n", "\n", "# Getting status of the request\n", "# 200 status code means our request was successful\n", "# 404 status code means that the resource you were looking for was not found\n", "response1.status_code" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### A different Scraping Function\n", "\n", "In this section, we will use `select()` and `select_one()` to return BeautifulSoup objects as per our requierment. While `find_all` uses tags, `select` uses CSS Selectors in the filter. I personally tend to use the latter one more.\n", "\n", "For example:\n", "- `p a` — finds all a tags inside of a p tag. \n", "> ```soup.select('p a')```\n", "\n", "- `div.outer-text` — finds all div tags with a class of outer-text.\n", "- `div#first` — finds all div tags with an id of first.\n", "- `body p.outer-text` — finds any p tags with a class of outer-text inside of a body tag." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Parsing the document into Beautiful Soup" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "bs4.element.Tag" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Parsing html data using BeautifulSoup\n", "soup1 = bs(response1.content, 'html.parser')\n", "\n", "# body \n", "body1 = soup1.select_one('body')\n", "\n", "# printing the object type of body\n", "type(body1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Extract Top Questions\n", "\n", "On Chrome, open [questions page](https://stackoverflow.com/questions?sort=votes&pagesize=50) and right-click on the top question and choose **Inspect**.\n", "\n", "![Image for Reference](https://github.com/nveenverma/Projects/blob/master/Exploring%20StackOverflow/questions.png?raw=true)\n", "*
Image for Reference
*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that the question is inside `a` tag, which has a class of `question-hyperlink`. \n", "\n", "Taking cue from our previous Goal, we can use this class along with `a` tag, to extract all the question links in a list. However, there are more question hyperlinks in sidebar which will also be extracted in this case. To avoid this scenario, we can combine `a` tag, `question-hyperlink` class with their parent `h3` tag. This will give us exactly 50 Tags." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[Why is processing a sorted array faster than processing an unsorted array?,\n", " How do I undo the most recent local commits in Git?]" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "question_links = body1.select(\"h3 a.question-hyperlink\")\n", "question_links[:2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[List comprehension](https://towardsdatascience.com/python-basics-list-comprehensions-631278f22c40), to extract all the questions." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['Why is processing a sorted array faster than processing an unsorted array?',\n", " 'How do I undo the most recent local commits in Git?']" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "questions = [i.text for i in question_links]\n", "questions[:2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Extract Summary\n", "\n", "On Chrome, open [questions page](https://stackoverflow.com/questions?sort=votes&pagesize=50) and right-click on summary of the top question and choose **Inspect**.\n", "\n", "![Image for Reference](https://github.com/nveenverma/Projects/blob/master/Exploring%20StackOverflow/summary.png?raw=true)\n", "*
Image for Reference
*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that the question is inside `div` tag, which has a class of `excerpt`. Using this class along with `div` tag, we can extract all the question links in a list." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "
\r\n", " Here is a piece of C++ code that shows some very peculiar behavior. For some strange reason, sorting the data miraculously makes the code almost six times faster:\n", "\n", "#include <algorithm>\n", "#include &...\r\n", "
\n" ] } ], "source": [ "summary_divs = body1.select(\"div.excerpt\")\n", "print(summary_divs[0])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[List comprehension](https://towardsdatascience.com/python-basics-list-comprehensions-631278f22c40), to extract all the questions. \n", "\n", "Here, we will also use [strip()](https://www.programiz.com/python-programming/methods/string/strip) method on each div's text. This is to remove both leading and trailing unwanted characters from a string." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Here is a piece of C++ code that shows some very peculiar behavior. For some strange reason, sorting the data miraculously makes the code almost six times faster:\\n\\n#include \\n#include &...'" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "summaries = [i.text.strip() for i in summary_divs]\n", "summaries[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Extract Tags\n", "\n", "On Chrome, open [questions page](https://stackoverflow.com/questions?sort=votes&pagesize=50) and right-click on summary of the top question and choose **Inspect**.\n", "\n", "![Image for Reference](https://github.com/nveenverma/Projects/blob/master/Exploring%20StackOverflow/tags_names.png?raw=true)\n", "*
Image for Reference
*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Extracting **tags per question** is the most complex task in this post. Here, we cannot find unique class or id for each tag, and there are multiple tags per question that we n\n", "eed to store. \n", "\n", "To extract **tags per question**, we will follow a multi-step process:\n", "\n", "* As shown in figure, individual tags are in a third layer, under two nested div tags. With the upper div tag, only having unique class (`summary`).\n", " - First, we will extract div with `summary`class.\n", " - Now notice our target div is third child overall and second `div` child of the above extracted object. Here, we can use `nth-of-type()` method to extract this 2nd `div` child. Usage of this method is very easy and few exmaples can be found [here](https://gist.github.com/yoki/b7f2fcef64c893e307c4c59303ead19a#file-20_search-py). This method will extract the 2nd `div` child directly, without extracting `summary div` first." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "
\n", "java c++ performance optimization branch-prediction\n", "
" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tags_divs = body1.select(\"div.summary > div:nth-of-type(2)\")\n", "tags_divs[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* Now, we can use [list comprehension](https://towardsdatascience.com/python-basics-list-comprehensions-631278f22c40) to extract `a` tags in a list, grouped per question." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[java,\n", " c++,\n", " performance,\n", " optimization,\n", " branch-prediction]" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "a_tags_list = [i.select('a') for i in tags_divs]\n", "\n", "# Printing first question's a tags\n", "a_tags_list[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* Now we will run a for loop for going through each question and use [list comprehension](https://towardsdatascience.com/python-basics-list-comprehensions-631278f22c40) inside it, to extract the tags names." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['java', 'c++', 'performance', 'optimization', 'branch-prediction']" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tags = []\n", "\n", "for a_group in a_tags_list:\n", " tags.append([a.text for a in a_group])\n", "\n", "tags[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Extract Number of votes, answers and views\n", "\n", "On Chrome, open [questions page](https://stackoverflow.com/questions?sort=votes&pagesize=50) and inspect vote, answers and views for the topmost answer.\n", "\n", "![Image for Reference](https://github.com/nveenverma/Projects/blob/master/Exploring%20StackOverflow/votes.png?raw=true)\n", "*
Image for Reference
*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### No. of Votes \n", "- They can be found by using `span` tag along with `vote-count-post` class and nested `strong` tags" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[23111, 19690]\n" ] } ], "source": [ "vote_spans = body1.select(\"span.vote-count-post strong\")\n", "print(vote_spans[:2])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[List comprehension](https://towardsdatascience.com/python-basics-list-comprehensions-631278f22c40), to extract vote counts. " ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[23111, 19690, 15321, 11030, 9718]" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "no_of_votes = [int(i.text) for i in vote_spans]\n", "no_of_votes[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I'm not going to post images to extract last two attributes\n", "\n", "### No. of Answers \n", "\n", "- They can be found by using `div` tag along with `status` class and nested `strong` tags. Here, we don't use `answered-accepted` because its not common among all questions, few of them (whose answer are not accepted) have the class - `answered`. " ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[22, 78]" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "answer_divs = body1.select(\"div.status strong\")\n", "answer_divs[:2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[List comprehension](https://towardsdatascience.com/python-basics-list-comprehensions-631278f22c40), to extract answer counts. " ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[22, 78, 38, 40, 34]" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "no_of_answers = [int(i.text) for i in answer_divs]\n", "no_of_answers[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### No. of Views \n", "- For views, we can see two options. One is short form in number of millions and other is full number of views. We will extract the full version. \n", "- They can be found by using `div` tag along with `supernova` class. Then we need to clean the string and convert it into integer format." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "
\n", " 1.4m views\n", "
" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "div_views = body1.select(\"div.supernova\")\n", "div_views[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[List comprehension](https://towardsdatascience.com/python-basics-list-comprehensions-631278f22c40), to extract vote counts. " ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[1362267, 7932952, 7011126, 2550002, 2490787]" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "no_of_views = [i['title'] for i in div_views]\n", "no_of_views = [i[:-6].replace(',', '') for i in no_of_views]\n", "no_of_views = [int(i) for i in no_of_views]\n", "no_of_views[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Putting all of them together in a dataframe" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "def get_top_questions(url, question_count):\n", " # WARNING: Only enter one of these 3 values [15, 30, 50].\n", " # Since, stackoverflow, doesn't display any other size questions list\n", " url = url + \"?sort=votes&pagesize={}\".format(question_count)\n", " \n", " # Using requests module for downloading webpage content\n", " response = requests.get(url)\n", "\n", " # Parsing html data using BeautifulSoup\n", " soup = bs(response.content, 'html.parser')\n", " body = soup.find('body')\n", "\n", " # Extracting Top Questions\n", " question_links = body1.select(\"h3 a.question-hyperlink\")\n", " error_checking(question_links, question_count) # Error Checking\n", " questions = [i.text for i in question_links] # questions list\n", " \n", " # Extracting Summary\n", " summary_divs = body1.select(\"div.excerpt\")\n", " error_checking(summary_divs, question_count) # Error Checking\n", " summaries = [i.text.strip() for i in summary_divs] # summaries list\n", " \n", " # Extracting Tags\n", " tags_divs = body1.select(\"div.summary > div:nth-of-type(2)\")\n", " \n", " error_checking(tags_divs, question_count) # Error Checking\n", " a_tags_list = [i.select('a') for i in tags_divs] # tag links\n", " \n", " tags = []\n", "\n", " for a_group in a_tags_list:\n", " tags.append([a.text for a in a_group]) # tags list\n", " \n", " # Extracting Number of votes\n", " vote_spans = body1.select(\"span.vote-count-post strong\")\n", " error_checking(vote_spans, question_count) # Error Checking\n", " no_of_votes = [int(i.text) for i in vote_spans] # votes list\n", " \n", " # Extracting Number of answers\n", " answer_divs = body1.select(\"div.status strong\")\n", " error_checking(answer_divs, question_count) # Error Checking\n", " no_of_answers = [int(i.text) for i in answer_divs] # answers list\n", " \n", " # Extracting Number of views\n", " div_views = body1.select(\"div.supernova\")\n", " \n", " error_checking(div_views, question_count) # Error Checking\n", " no_of_views = [i['title'] for i in div_views]\n", " no_of_views = [i[:-6].replace(',', '') for i in no_of_views]\n", " no_of_views = [int(i) for i in no_of_views] # views list\n", " \n", " # Putting all of them together\n", " df = pd.DataFrame({'question': questions, \n", " 'summary': summaries, \n", " 'tags': tags,\n", " 'no_of_votes': no_of_votes,\n", " 'no_of_answers': no_of_answers,\n", " 'no_of_views': no_of_views})\n", "\n", " return df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plotting Votes v/s Views v/s Answers" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
questionsummarytagsno_of_votesno_of_answersno_of_views
0Why is processing a sorted array faster than p...Here is a piece of C++ code that shows some ve...[java, c++, performance, optimization, branch-...23111221362267
1How do I undo the most recent local commits in...I accidentally committed the wrong files to Gi...[git, version-control, git-commit, undo]19690787932952
2How do I delete a Git branch locally and remot...I want to delete a branch both locally and rem...[git, git-branch, git-remote]15321387011126
3What is the difference between 'git pull' and ...Moderator Note: Given that this question has a...[git, git-pull, git-fetch]11030402550002
4What is the correct JSON content type?I've been messing around with JSON for some ti...[json, http-headers, content-type]9718342490787
\n", "
" ], "text/plain": [ " question \\\n", "0 Why is processing a sorted array faster than p... \n", "1 How do I undo the most recent local commits in... \n", "2 How do I delete a Git branch locally and remot... \n", "3 What is the difference between 'git pull' and ... \n", "4 What is the correct JSON content type? \n", "\n", " summary \\\n", "0 Here is a piece of C++ code that shows some ve... \n", "1 I accidentally committed the wrong files to Gi... \n", "2 I want to delete a branch both locally and rem... \n", "3 Moderator Note: Given that this question has a... \n", "4 I've been messing around with JSON for some ti... \n", "\n", " tags no_of_votes \\\n", "0 [java, c++, performance, optimization, branch-... 23111 \n", "1 [git, version-control, git-commit, undo] 19690 \n", "2 [git, git-branch, git-remote] 15321 \n", "3 [git, git-pull, git-fetch] 11030 \n", "4 [json, http-headers, content-type] 9718 \n", "\n", " no_of_answers no_of_views \n", "0 22 1362267 \n", "1 78 7932952 \n", "2 38 7011126 \n", "3 40 2550002 \n", "4 34 2490787 " ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "URL2 = 'https://stackoverflow.com/questions'\n", "\n", "df1 = get_top_questions(URL2, 50)\n", "df1.head()" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAzIAAAHfCAYAAACLcO1YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XuYJVV59/3vz0FUPAyIgjigoGmNxHiKIgYfJR4RD5g3ECEaEYkmRhTFE+RNomKMhzdR9PUUIyA+HgCJRiRE5FFQJyoCilFE7QEUZiCQBBhP8QDezx+7Rnt6+rCnp3fVrt3fz3X11btWraq6d/Xqvfe9V61VqSokSZIkqU9u1XUAkiRJkrS1TGQkSZIk9Y6JjCRJkqTeMZGRJEmS1DvbdR1AWzZu3OisBpIkSVIPrV69OrPL7JGRJEmS1DsmMpIkSZJ6x0RmjExPT3cdgsacbUSLsY1oGLYTLcY2osWMQxsxkZEkSZLUOyYykiRJknrHREaSJElS75jISJIkSeqdFXMfmXGy48kb5lmzA6zdfN1NR6wZfUCSJElSz9gjI0mSJKl3TGQkSZIk9Y6JjCRJkqTeMZGRJEmS1DsmMpIkSZJ6x0RGkiRJUu+YyEiSJEnqHRMZSZIkSb1jIiNJkiSpd0xkJEmSJPWOiYwkSZKk3jGRkSRJktQ7rSQySfZIcl6Sy5JcmuTopvzOSc5NMt383qkpT5K3J1mX5N+TPGTGvg5v6k8nOXxG+e8k+UazzduTpI3nJkmSJKl9bfXI3Ay8rKruB+wLvDDJ3sCxwGeqagr4TLMM8CRgqvl5PvBuGCQ+wKuBhwP7AK/elPw0dZ4/Y7sDWnhekiRJkjrQSiJTVddW1Vebxz8ELgPWAAcBpzTVTgGe3jw+CPhADXwZ2DHJbsATgXOr6oaquhE4FzigWXenqvpSVRXwgRn7kiRJkjRhtmv7gEn2BB4MXADsWlXXwiDZSbJLU20NcPWMzdY3ZQuVr5+jfE7T09Pb9By23Q5D1+w+Vo0b24QWYxvRMGwnWoxtRIsZdRuZmppacH2riUySOwD/BLykqn6wwDCWuVbUEsrntNhJGbm1G4au2nmsGivT09O2CS3INqJh2E60GNuIFjMObaS1RCbJrRkkMR+qqo81xdcl2a3pjdkNuL4pXw/sMWPz3YFrmvL9Z5Wf35TvPkf9ibLjycMnQDcdMW+HlCRJktR7bc1aFuBE4LKqesuMVWcCm2YeOxz4xIzyZzezl+0LbGwuQTsHeEKSnZpB/k8AzmnW/TDJvs2xnj1jX5IkSZImTFs9MvsBfwx8I8klTdlfAG8ETk9yJHAVcEiz7mzgQGAd8BPgCICquiHJ64ALm3rHV9UNzeMXAO8Hbgf8a/MjSZIkaQK1kshU1VrmHscC8Ng56hfwwnn2dRJw0hzlFwH334YwJUmSJPVEW/eRkSRJkqRlYyIjSZIkqXdMZCRJkiT1zlCJTJLfS7JX83i3JKckOSnJ3UYbniRJkiRtadgemXcBtzSP/x64NYMbTr53FEFJkiRJ0kKGnbVsTVVdlWQ74InAPYGfM4E3nZQkSZI0/oZNZH6QZFcG0xt/q6p+lGR7Bj0zkiRJktSqYROZ/5/BTSi3B17SlO0HfHsUQUmSJEnSQoZKZKrqTUk+DtxSVZc3xRuAPxlZZJIkSZI0j62ZfvlK4O5JntEsbwCuWP6QJEmSJGlhw06//NvAd4F/BE5sih8NnDSiuCRJkiRpXsP2yLwb+Ouq+k3gF03Z54BHjiQqSZIkSVrAsInMbwEfbB4XQFX9GLjdKIKSJEmSpIUMm8h8D/idmQVJ9gHWLXdAkiRJkrSYYadf/ivgX5K8B9g+yXHAnwHPG1lkkiRJkjSPoXpkquos4EnAXRmMjbkn8P9U1adHGJskSZIkzWmoHpkkh1TVR4E/n1V+cFWdMZLIJEmSJGkew46ROXGe8vcOs3GSk5Jcn+SbM8pek2RDkkuanwNnrDsuybok30nyxBnlBzRl65IcO6N8ryQXJJlOclqS7Yd8XpIkSZJ6aMFEJsm9ktwLuFWTLNxrxs/jgJ8OeZz3AwfMUf7WqnpQ83N2c8y9gUMZzJR2APCuJKuSrALeyeASt72Bw5q6AG9q9jUF3AgcOWRckiRJknposUvL1jGYbjnA5bPW/QfwmmEOUlWfT7LnkDEdBJxaVT8DrkyyDthnUzxVdQVAklOBg5JcBjwG+KOmzilNXO8e8ngTb8eTNwxd96Yj1owwEkmSJGl5LJjIVNWtAJJ8rqoePYLjH5Xk2cBFwMuq6kZgDfDlGXXWN2UAV88qfziwM3BTVd08R/05TU9PL0Po22KHoWtuHmvb22kc+TfSYmwjGobtRIuxjWgxo24jU1NTC64farD/piQmyT0YJAnrq+rqhbda1LuB1zHo8Xkd8PfAcxn0/mwRAnNfBrept2iu8nktdlJGbu3wPSSbxdr2dho709PT/o20INuIhmE70WJsI1rMOLSRoQb7J7lbks8xuNTsY8DlST6f5O5LPXBVXVdVt1TVL4F/5NeXj60H9phRdXfgmgXK/wvYMcl2s8olSZIkTahhZy17D/B1YKeq2g3YCfhaU74kSXabsfj7wKYZzc4EDk1ymyR7AVPAV4ALgalm0oHtGUwIcGZVFXAecHCz/eHAJ5YalyRJkqTxN9SlZcAjgd2q6hcAVfXjJK8EhrpmKclHgP2BuyRZD7wa2D/JgxhcBvY94E+bfV+a5HTgW8DNwAur6pZmP0cB5wCrgJOq6tLmEK8CTk3yNwwSrPmmi5YkSZI0AYZNZG5kMOXx12eU3Re4aZiNq+qwOYrnTTaq6vXA6+coPxs4e47yK/j1pWmSJEmSJtywicybgf+T5ETg+8A9gSOAvxpVYJIkSZI0n2FnLfvHJJczuFfLAxgMpj+sqj47yuAkSZIkaS4LJjJJDgQ+VVW/bJIWExdJkiRJnVts1rKTgQ1J/i7Jb7cRkCRJkiQtZrFE5u7A8xjcv+WCJF9L8pIku4w+NEmSJEma24KJTHPDyrOq6hnA3YB3Mbjny1VJPpnk4IW2lyRJkqRRGPaGmFTVD6rqH6vq0QzuCXN/4LRRBSZJkiRJ8xk6kUlymySHJjkb+BxwJfDckUUmSZIkSfNYdPrlJI8Cng0cDFwH/G/gz6rqqhHHJkmSJElzWmz65SuB1cBHgQOr6outRCVJkiRJC1isR+Y44ONV9bM2gpEkSZKkYSyYyFTVqW0FIkmSJEnDGnqwvyRJkiSNCxMZSZIkSb0zbyKT5LQZj49oJxxJkiRJWtxCY2SemCRVVcDbgJNbikljYseTNwxd96Yj1owwEkmSJGlzCyUyXwC+lOS7wG2TfGCuSlX17JFEJkmSJEnzWGiMzCHAO4DvAAVcPs/PopKclOT6JN+cUXbnJOcmmW5+79SUJ8nbk6xL8u9JHjJjm8Ob+tNJDp9R/jtJvtFs8/YkGf4USJIkSeqbeXtkquqnwAcBkty6ql67Dcd5P4OkaGavzrHAZ6rqjUmObZZfBTwJmGp+Hg68G3h4kjsDrwYeyiCxujjJmVV1Y1Pn+cCXgbOBA4B/3YZ4JUmSJI2xoWYtq6rXJJlK8tdJ/qH5PTXsQarq88ANs4oPAk5pHp8CPH1G+Qdq4MvAjkl2A54InFtVNzTJy7nAAc26O1XVl5rxPB+YsS9JkiRJE2ioRCbJU4GLgd9kkJDcF7goydO24di7VtW1AM3vXZryNcDVM+qtb8oWKl8/R7kkSZKkCbXQYP+Z/hY4qKrO21SQZH8Gl4uducwxzTW+pZZQPq/p6eklhLWcdhi65uax9mU7jZLnWouxjWgYthMtxjaixYy6jUxNLXwB2LCJzO4MZjGbaW1TvlTXJdmtqq5tLg+7vilfD+wx69jXNOX7zyo/vynffY7681rspIzc2uGnNd4s1h5s55TNozU9Pd19+9VYs41oGLYTLcY2osWMQxsZ6tIy4BLgZbPKjmnKl+pMYNPMY4cDn5hR/uxm9rJ9gY3NpWfnAE9IslMzw9kTgHOadT9Msm8zW9mzZ+xLkiRJ0gQatkfmBcAnkxzNYJzKHsCPgaHGyCT5CIPelLskWc9g9rE3AqcnORK4isF0zzCYdexAYB3wE+AIgKq6IcnrgAubesdX1aYJBF7AYGa02zGYrcwZy3rGnhxJkiRtjaESmar6dpL7AfsCd2dw6dYFVfWLIbc/bJ5Vj52jbgEvnGc/JwEnzVF+EXD/YWKRJEmS1H/D9shQVTczGBcjSZIkSZ0adoyMJEmSJI0NExlJkiRJvTP0pWXSOHKSAEmSpJVp6ESmmfL4qcAaYANw1oxZw6TeMPmRJEnqv6EuLUvyCOBy4M+ABwB/CqxryiVJkiSpVcP2yJwA/HlVnbqpIMkzgLcDDxtFYJIkSZI0n2ETmfsAp88qOwN4z/KGI42vpV6Strzb7QBrtyzftF3bMUqSJHVl2ERmGjgU+PCMskMYXG4maYUyAZIkSV0ZNpF5CXBWkhcD3wf2BKaAp4woLkkTyl4jSZK0HIZKZKrqi0nuDTwZuDvwSeBsZy2TNO5MnCRJmkxDT79cVTcCHxxhLJLUayY/kiS1Z8FEJsl5QC1QparqscsbkiStLCZAkiRtvcV6ZObrgVkDvBjYYXnDkSQNaykz282/3dxMnCRJ42rBRKaqTpy5nGRn4DjgecBpwPGjC02SNC5MfiRJ42aoMTJJ7gS8AjgKOAt4SFU59bIkaUFOtiBJGpXFxsjcjsHUyy8DzgceWVWXthCXJElbxaRJklaWxXpkrgRWAW8GLgJ2TbLrzApV9dkRxSZJ0tgyAZKkbi2WyPyUwaxlL5hnfQH32pYAknwP+CFwC3BzVT00yZ0ZjMHZE/ge8IdVdWOSAG8DDgR+Ajynqr7a7Odw4C+b3f5NVZ2yLXFJkjQKS0mATJokaUuLDfbfs6U4fq+q/mvG8rHAZ6rqjUmObZZfBTwJmGp+Hg68G3h4k/i8Gngog+Tq4iRnNve+kSRJkjRhhr4hZssOAvZvHp/CYHzOq5ryD1RVAV9OsmOS3Zq651bVDQBJzgUOAD7SbtiSJI2P+Xtytpyme1vHDdlrJKlt45DIFPDpJAX8Q1W9F9i1qq4FqKprk+zS1F0DXD1j2/VN2Xzlc5qenl7G8Jdi+NvvbB5rH7Yzxm63M8Zut+tDjEvdzhi73W5yY3zY2uG3u/CRP+lku7ZjHBfdf17SuBt1G5mamlpw/TgkMvtV1TVNsnJukm8vUDdzlNUC5XNa7KSM3Bw3q5vPZrH2YTtj7HY7Y+x2uz7EuNTtjLHb7Yyx2+1ajnHUM/AN0yM2PT3d/ecljbVxaCOdJzJVdU3z+/okHwf2Aa5LslvTG7MbcH1TfT2wx4zNdweuacr3n1V+/ohDlyRJ6rU2Lz/clu2kuXSayCS5PXCrqvph8/gJwPHAmcDhwBub359oNjkTOCrJqQwG+29skp1zgL9NslNT7wnAcS0+FUmSJI1A20lTm9v1IcZx1nWPzK7AxwezKrMd8OGq+lSSC4HTkxwJXAUc0tQ/m8HUy+sYTL98BEBV3ZDkdcCFTb3jNw38lyRJkjR5Ok1kquoK4IFzlP838Ng5ygt44Tz7Ogk4abljlCRJkjR+btV1AJIkSZK0tUxkJEmSJPWOiYwkSZKk3jGRkSRJktQ7JjKSJEmSesdERpIkSVLvmMhIkiRJ6h0TGUmSJEm9YyIjSZIkqXdMZCRJkiT1jomMJEmSpN4xkZEkSZLUOyYykiRJknrHREaSJElS75jISJIkSeodExlJkiRJvWMiI0mSJKl3JiqRSXJAku8kWZfk2K7jkSRJkjQaE5PIJFkFvBN4ErA3cFiSvbuNSpIkSdIopKq6jmFZJHkE8JqqemKzfBxAVb0BYOPGjZPxRCVJkqQVZvXq1ZldNjE9MsAa4OoZy+ubMkmSJEkTZpISmS2yNMBeGEmSJGkCbdd1AMtoPbDHjOXdgWs2LczVHSVJkiSpnyapR+ZCYCrJXkm2Bw4Fzuw4JkmSJEkjMDE9MlV1c5KjgHOAVcBJVXVpx2FJkiRJGoFJ6pGhqs6uqvtU1b2r6vVdxzMs73+juSQ5Kcn1Sb45o+zOSc5NMt383qnLGNWtJHskOS/JZUkuTXJ0U247EQBJbpvkK0m+3rSR1zbleyW5oGkjpzVXMmgFS7IqydeSnNUs20b0K0m+l+QbSS5JclFT1vl7zUQlMn3k/W+0gPcDB8wqOxb4TFVNAZ9plrVy3Qy8rKruB+wLvLB5/bCdaJOfAY+pqgcCDwIOSLIv8CbgrU0buRE4ssMYNR6OBi6bsWwb0Wy/V1UPqqqHNsudv9eYyHRvH2BdVV1RVT8HTgUO6jgmjYGq+jxww6zig4BTmsenAE9vNSiNlaq6tqq+2jz+IYMPIWuwnahRAz9qFm/d/BTwGOCMptw2ssIl2R14MvC+ZjnYRrS4zt9rTGS65/1vtDV2raprYfAhFtil43g0JpLsCTwYuADbiWZoLhm6BLgeOBe4HLipqm5uqvi+oxOAVwK/bJZ3xjaizRXw6SQXJ3l+U9b5e83EDPbvMe9/I2mbJLkD8E/AS6rqB4MvU6WBqroFeFCSHYGPA/ebq1q7UWlcJHkKcH1VXZxk/03Fc1S1jaxs+1XVNUl2Ac5N8u2uAwJ7ZMbBgve/kWa5LsluAM3v6zuORx1LcmsGScyHqupjTbHtRFuoqpuA8xmMp9oxyaYvM33fWdn2A56W5HsMLm9/DIMeGtuIfqWqrml+X8/gC5F9GIP3GhOZ7nn/G22NM4HDm8eHA5/oMBZ1rLmO/UTgsqp6y4xVthMBkOSuTU8MSW4HPI7BWKrzgIObaraRFayqjquq3atqTwafQT5bVc/ENqJGktsnueOmx8ATgG8yBu81qbKnsGtJDmTw7cem+9/0ZupojU6SjwD7A3cBrgNeDfwzcDpwD+Aq4JCqmj0hgFaIJI8EvgB8g19f2/4XDMbJ2E5EkgcwGIS7isGXl6dX1fFJ7sXg2/c7A18DnlVVP+suUo2D5tKyl1fVU2wj2qRpCx9vFrcDPlxVr0+yMx2/15jISJIkSeodLy2TJEmS1DsmMpIkSZJ6Z8UlMklOSnJ9km8OUfetSS5pfr6b5KY2YpQkSZK0sBU3RibJo4AfAR+oqvtvxXYvAh5cVc8dWXCSJEmShrLiemSq6vPAZjMqJLl3kk81dyv9QpLfnGPTw4CPtBKkJEmSpAVtt3iVFeG9wJ9V1XSShwPvYnBDKACS3BPYC/hsR/FJkiRJmmHFJzJJ7gD8LvDRwb3lALjNrGqHAmdU1S1txiZJkiRpbis+kWFwed1NVfWgBeocCrywpXgkSZIkLWLFjZGZrap+AFyZ5BCADDxw0/ok9wV2Ar7UUYiSJEmSZllxiUySjzBISu6bZH2SI4FnAkcm+TpwKXDQjE0OA06tlTa9myRJkjTGVtz0y5IkSZL6b8WMkdm4caMZmyRJktRDq1evzuyyFXdpmSRJkqT+M5GRJEmS1Du9SWSSHJDkO0nWJTl2jvX3SHJekq8l+fckB3YR57aYnp7uOgSNOduIFmMb0TBsJ1qMbUSLGYc20otEJskq4J3Ak4C9gcOS7D2r2l8Cp1fVgxnc9+Vd7UYpSZIkqS29SGSAfYB1VXVFVf0cOJXNp0gGKOBOzePVwDUtxidJkiSpRb2YfjnJwcABVfUnzfIfAw+vqqNm1NkN+DSDm1feHnhcVV28af3MWcvGoStMkiRJ0vympqZ+9XiuWcv6Mv3yFoEz6IGZ6TDg/VX190keAfzvJPevql/O3nDmSRkn09PTC8a248kbht7XTUesWY6QNGYWayOSbUTDsJ1oMbYRLWYc2khfLi1bD+wxY3l3trx07EjgdICq+hJwW+AurUQnSZIkqVV9SWQuBKaS7JVkewaD+c+cVecq4LEASe7HIJH5z1ajlCRJktSKXiQyVXUzcBRwDnAZg9nJLk1yfJKnNdVeBjwvydeBjwDPqT4MAJIkSZK01foyRoaqOhs4e1bZX894/C1gv7bjkiRJktS+XvTISJIkSdJMJjKSJEmSesdERpIkSVLvmMhIkiRJ6h0TGUmSJEm9YyIjSZIkqXdMZCRJkiT1jomMJEmSpN4xkZEkSZLUO9t1HYBGb8eTNwxd96Yj1owwEkmSJGl52CMjSZIkqXdMZCRJkiT1jomMJEmSpN4xkZEkSZLUOyYykiRJknrHREaSJElS75jISJIkSeodExlJkiRJvWMiI0mSJKl3TGQkSZIk9U4riUySY5I8qHm8b5KrklyR5BFtHF+SJEnSZGmrR+alwJXN4zcAbwFeD5wwzMZJDkjynSTrkhw7T50/TPKtJJcm+fCyRC1JkiRpLG3X0nFWV9XGJHcEHgg8rqpuSfL3i22YZBXwTuDxwHrgwiRnVtW3ZtSZAo4D9quqG5PsMpqnIUmSJGkctJXIXJ3kd4HfAj7fJDF3Am4ZYtt9gHVVdQVAklOBg4BvzajzPOCdVXUjQFVdv6zRS5IkSRorbSUyrwDOAH4O/EFT9hTgK0Nsuwa4esbyeuDhs+rcByDJvwGrgNdU1ae2JWBJkiRJ46uVRKaqzgbuPqv4o83PYjLXLmctbwdMAfsDuwNfSHL/qrpprh1OT08PcdhuLBzbDkvcz1K30zjyb6TF2EY0DNuJFmMb0WJG3UampqYWXN9KIpPkLcD5wBdmXP71iyE3Xw/sMWN5d+CaOep8udnnlUm+wyCxuXCuHS52UroyPT29cGxrNwy9r832s9TtNHYWbSNa8WwjGobtRIuxjWgx49BG2pq17MfAy4ANSb6e5O1J/iDJXYfY9kJgKsleSbYHDgXOnFXnn4HfA0hyFwaXml2xfOFLkiRJGidtXVr2VwBJbgPsCzwZOAm4A4MxLQtte3OSo4BzmronVdWlSY4HLqqqM5t1T0jyLQYTCLyiqv57ZE9IUut2PHn4nsWbjlgzwkgkSdI4aOvSsjsA+wGPZjCO5R4Mko/PDbN9M8bm7Fllfz3jcQHHND/SSPhBWpIkaXy0NWvZjcD3gLcDz62qb7d0XEmSJI0JvxTUcmorkXkt8L+AvwCekuRzDHpjvrIVg/4lSZIkCWhvjMzfACRZBTwE+H0Gl4qtYjBORlIP+c2aJEnqSltjZO7MYHzMoxnMLnZf4GKGHCMjSZIkSTO1dWnZeuArwOcZDMj/YlX9T0vHliRJkjRh2kpkdqqqn7V0LEmSJEkTrpUbYlbVz5I8PsmJST4JkOShSR7TxvElSZIkTZZWEpkkLwLeDUwDj2qK/wf4mzaOL0mSJGmytJLIAC8BHldVbwR+2ZR9m8Ggf0mSJEnaKm2NkbkjcHXzuJrftwZ+3tLxJWnknI5akqT2tNUj83ng2FllLwbOa+n4kiRJkiZIWz0yLwI+meR5wB2TfAf4AfDUlo4vSZIkaYK0kshU1bVJHgbsA9yDwWVmX6mqXy68pSRJkiRtqa0eGaqqgAuaH0mSJElaspElMkkuq6r7NY+v5teD/DdTVfcYVQySpH5y4gRJ0mJG2SPzvBmPnzXC40iSJElaYUaWyFTV2iQHAp+qqs+N6jiSJEmSVp5RT798MrAhyd8l+e0RH0uSJEnSCjHqRObuDC4x2wO4IMnXkrwkyS4jPq4kSZKkCTbSRKaqbqmqs6rqGcDdgHcBvw9cleSTSQ4e5fElSZIkTaZR98j8SlX9oKr+saoeDewP3B84ra3jS5IkSZocrSUySW6T5NAkZwOfA64EntvW8SVJkiRNjpEnMkkeleR9wHXAa4EvAlNV9ZiqOmUr9nNAku8kWZfk2AXqHZykkjx026OXJEmSNI5GeR8ZklwJrAY+ChxYVV9c4n5WAe8EHg+sBy5McmZVfWtWvTsCLwYu2KbAJUmSJI21UffIHAfsVlV/utQkprEPsK6qrqiqnwOnAgfNUe91wJuBn27DsSRJkiSNuZH2yFTVqcu0qzXA1TOW1wMPn1khyYOBParqrCQvX2hn09PTyxTW8ls4th2WuJ+lbqfNjcd5HK+/UZvnZDzO/8LGI8bxaiNLNR7ncpJ53rSY0bQR/7cnyaj/RlNTUwuuH2kis4wyR1n9amVyK+CtwHOG2dliJ6Ur09PTC8e2dsPQ+9psP0vdTpsbg/O4aBtpW5vnZAzO/6LGIMaxayNLNQbncpJNTDvRyIysjfi/PTHG4XWktVnLttF6BjfV3GR34JoZy3dkMJ3z+Um+B+wLnOmAf0mSJGkyjaxHJslpzY0wSXJEVZ28Dbu7EJhKshewATgU+KNNK6tqI3CXGcc+H3h5VV20DceUJG2jHU8e/tvXm45YM8JIJEmTZpQ9Mk9MsumSsLdty46q6mbgKOAc4DLg9Kq6NMnxSZ62jXFKkiRJ6plRjpH5AvClJN8FbpvkA3NVqqpnD7OzqjobOHtW2V/PU3f/rQtVkiRJUp+MMpE5BDgYuCeDgfmXj/BYkiRJklaQkSUyVfVT4IMASW5dVa8d1bEkSZIkrSytTL9cVa9JMgUcxuCeMBuAj1SVE4RLkqTWOAGFNDlamX45yVOBi4HfBG4A7gtc5EB9SZIkSUvR1g0x/xY4qKrO21SQZH/gHcCZLcUgSZIkaUK0dUPM3RnMYjbT2qZckiRJkrZKW4nMJcDLZpUd05RLkiRJ0lZp69KyFwCfTHI0cDWwB/BjwDEykiRJkrZaW7OWfTvJ/YB9gbsD1wAXVNUv2ji+JEmSpMnSVo8MVXUzg3ExkiRJkrRN2hojI0mSJEnLxkRGkiRJUu+YyEiSJEnqndbGyCTZCXgqsAbYAJxVVTe0dXxJkiRJk6OVRCbJI4B/Ab4NfB94CnBCkidX1ZfaiEGSJEmb2/HkDfOs2QHWbr7upiPWjD4gaSu01SNzAvDnVXXqpoIkzwDeDjyspRgkYKEX7S35oi1JkjSe2hojcx/g9FllZwC/0dLxJUmSJE2QtnpkpoFDgQ/PKDsEuLyl42sJ7LlYHp5HSZKk5ddWIvMS4KwkL2YwRmZPYIrBWBlNED+0azG2EUmStBxaSWSq6otJ7g08Gbg78EngbGctk+Y39wf+LQdfgh/4JUnqml/Uta+16Zer6kbgg20dT5IkjT8//ElaqpEmMkk5p3X9AAAfjklEQVTOA2qBKlVVjx1lDJIkSZIm74uDUffIzNcDswZ4MbDDMDtJcgDwNmAV8L6qeuOs9ccAfwLcDPwn8Nyq+v5Sg5YkSZLG1aQlJEs10kSmqk6cuZxkZ+A44HnAacDxi+0jySrgncDjgfXAhUnOrKpvzaj2NeChVfWTJC8A3gw8Y3mehSRJkqRx08oYmSR3Al4BHAWcBTykqoadenkfYF1VXdHs61TgIOBXiUxVnTej/peBZy1H3JKkAb/900pm+5fG06jHyNyOwdTLLwPOBx5ZVZdu5W7WAFfPWF4PPHyB+kcC/7rQDqenp7cyhPYsHNtQV+LNsZ82t1vasR62dvjtLnzkT4auO7c+nMdt2a5N7bWRyT0fy280+7b9T5rxOW9t/63bfN3qgz6c/75o87mNx2vrqP9GU1NTC64fdY/MlQzGtbwZuAjYNcmuMytU1WcX2UfmKJtzAoEkzwIeCjx6oR0udlK6Mj09vXBsc0y7O5/N9tPmdm3HuBR9OI/bsl2bJrWNLNUYxLjo68hS2f6XzTh8uz+ydrIUbf+t23zd6oM+nP++mOTPMnMYh9eRUScyP2WQdLxgnvUF3GuRfawH9pixvDtwzexKSR4H/L/Ao6vqZ1sfqiRJkqS+GPVg/z2XYTcXAlNJ9gI2AIcCfzSzQpIHA/8AHFBV1y/DMSWpNePwDf0k8DyuLP69Jd2q6wAWU1U3M5gk4BzgMuD0qro0yfFJntZU+/+AOwAfTXJJkjM7CleSJElSC1qZtWxbVdXZwNmzyv56xuPHtR6UJEljbv5eix22uFbeXovxYW+TNJxeJDKSJI0bP2xKUrfG/tIySZIkSZrNHhlJkiSpA/bsbhsTGWmC+IIoSZJWChOZDjj4UuPGBEiSJPWNY2QkSZIk9Y49MpIkSSNgb7c0WiYykiSNOT8QS9KWTGQkSZI0cfwCYPKZyEiaaL6RSZK2hu8b/WEiI0k9Nfeb7ZazH4JvtgvxQ4sk9ZOJjCRJkraKXwBoHJjISJIkaWyZNGk+3kdGkiRJUu/YI6Ne81saSRoPvh5LapuJjCRJ0gQYNpk0kdSkMJGRpBXGb861GNuIpD4wkZGkOfhBTpKk8eZgf0mSJEm9YyIjSZIkqXdMZCRJkiT1jomMJEmSpN7pTSKT5IAk30myLsmxc6y/TZLTmvUXJNmz/SglSZIktaEXiUySVcA7gScBewOHJdl7VrUjgRur6jeAtwJvajdKSZIkSW1JVXUdw6KSPAJ4TVU9sVk+DqCq3jCjzjlNnS8l2Q74D+Cu1TzBjRs3jv8TlSRJkrSF1atXZ3ZZL3pkgDXA1TOW1zdlc9apqpuBjcDOrUQnSZIkqVV9SWS2yMCA2T0sw9SRJEmSNAG26zqAIa0H9pixvDtwzTx11jeXlq0Gbti0cq7uKEmSJEn91JcemQuBqSR7JdkeOBQ4c1adM4HDm8cHA5+tPgwAkiRJkrTVepHINGNejgLOAS4DTq+qS5Mcn+RpTbUTgZ2TrAOOAbaYonlcLTa1tFamJCcluT7JN2eU3TnJuUmmm987dRmjupVkjyTnJbksyaVJjm7KbScCIMltk3wlydebNvLapnyv5lYF082tC7bvOlZ1K8mqJF9LclazbBvRryT5XpJvJLkkyUVNWefvNb2YtWySNVNLfxd4PIPL4y4EDquqb3UamDqX5FHAj4APVNX9m7I3AzdU1RubpHenqnpVl3GqO0l2A3arqq8muSNwMfB04DnYTgQkCXD7qvpRklsDa4GjGXzh97GqOjXJe4CvV9W7u4xV3UpyDPBQ4E5V9ZQkp2MbUSPJ94CHVtV/zSjr/DNJL3pkJtw+wLqquqKqfg6cChzUcUwaA1X1eWaM82ocBJzSPD6FwYdWrVBVdW1VfbV5/EMGPdZrsJ2oUQM/ahZv3fwU8BjgjKbcNrLCJdkdeDLwvmY52Ea0uM7fa0xkujfM1NLSJrtW1bUw+BAL7NJxPBoTSfYEHgxcgO1EMzSXDF0CXA+cC1wO3NRctg2+7whOAF4J/LJZ3hnbiDZXwKeTXJzk+U1Z5+81fZm1bJI5bbSkbZLkDsA/AS+pqh8MvkyVBqrqFuBBSXYEPg7cb65q7UalcZHkKcD1VXVxkv03Fc9R1Taysu1XVdck2QU4N8m3uw4I7JEZB8NMLS1tcl0zLmLT+IjrO45HHWvGPfwT8KGq+lhTbDvRFqrqJuB8YF9gx+ZWBeD7zkq3H/C0ZgzEqQwuKTsB24hmqKprmt/XM/hCZB/G4L3GRKZ7w0wtLW0yc5rxw4FPdBiLOtZcx34icFlVvWXGKtuJAEhy16YnhiS3Ax7HYCzVeQxuVQC2kRWtqo6rqt2rak8Gn0E+W1XPxDaiRpLbNxPKkOT2wBOAbzIG7zXOWjYGkhzI4NuPVcBJVfX6jkPSGEjyEWB/4C7AdcCrgX8GTgfuAVwFHFJVsycE0AqR5JHAF4Bv8Otr2/+CwTgZ24lI8gAGg3BXMfjy8vSqOj7JvRh8+35n4GvAs6rqZ91FqnHQXFr28mbWMtuIAGjawsebxe2AD1fV65PsTMfvNSYykiRJknrHS8skSZIk9Y6JjCRJkqTeMZGRJEmS1DsmMpIkSZJ6x0RGkiRJUu+YyEiSJEnqHRMZSZIkSb1jIiNJkiSpd0xkJEmSJPWOiYwkSZKk3jGRkSRJktQ723UdQFs2btxYXccgSZIkaeutXr06s8vskZEkSZLUOyYykiRJknrHRGaMTE9Pdx2CxpxtRIuxjWgYthMtxjaixYxDGzGRkSRJktQ7JjKSJEmSesdERpIkSVLvrJjplyXNb8eTNwxd96Yj1owwEkmSpOHYIyNJkiSpd0xkJEmSJPWOiYwkSZKk3jGRkSRJktQ7JjKSJEmSeqeXiUySlya5NMk3k3wkyW2T7JXkgiTTSU5Lsn3XcUqSJEkajd4lMknWAC8GHlpV9wdWAYcCbwLeWlVTwI3Akd1FKUmSJGmUepfINLYDbpdkO2AH4FrgMcAZzfpTgKd3FJskSZKkEUtVdR3DVktyNPB64H+ATwNHA1+uqt9o1u8B/GvTYwPAxo0bf/VEp6en2w1YGnMPW7vD0HUvfORPRhiJtHS2Y0maLFNTU796vHr16sxev12r0SyDJDsBBwF7ATcBHwWeNEfVeTO0mSdlnExPT49tbBoPI2sjazcMXbWrNrrjycPHeNMRa0YYyXhb0a8jPWjH42JFtxMNxTaixYxDG+njpWWPA66sqv+sql8AHwN+F9ixudQMYHfgmq4ClCRJkjRaveuRAa4C9k2yA4NLyx4LXAScBxwMnAocDnyiswhHxG+kJUmSpIHe9chU1QUMBvV/FfgGg+fwXuBVwDFJ1gE7Ayd2FqQkSZKkkepjjwxV9Wrg1bOKrwD26SAcSZIkSS3rXY+MJEmSJJnISJIkSeqd1i8tS7I38N9VdV2SOwCvAG4B/q6qnNhfkiQtyglwJHXRI/NhYMfm8d8BjwIeAfxDB7FIkiRJ6qEuBvvvWVXfSRLg94HfYjCN8pUdxCJJkiSph7pIZH6W5I7A3sDVVfVfzY0sb9tBLJIkSZJ6qItE5sMMbl55B+AdTdlDsEdGkiRJ0pBaT2Sq6qVJngD8oqrOa4p/Cby07VgkSZIk9VOriUySVcB3gb2r6mebyqvqojbjkCRJktRvrc5aVlW3MJhq2fEwkiRJkpasizEyJwCnJ/lbYD1Qm1ZU1RUdxCNJkiSpZ7pIZDYN8H/8rPICVrUciyRJkqQe6mKwfxc34ZQkSZI0QTpLKpLskWTfro4vSZIkqb9aT2SS3CPJvwHfBv5PU3Zwkve1HYskSZKkfuqiR+YfgH8B7gj8oik7ly3HzEiSJEnSnLoY7L8P8OSq+mWSAqiqjUlWdxCLJEmSpB7qokfmOuA3ZhYk2Ru4qoNYJEmSJPVQF4nM3wFnJTkC2C7JYcBpwJs6iEWSJElSD3Ux/fJJSW4Ang9cDTwb+Kuq+udh95FkR+B9wP0Z3H/mucB3GCREewLfA/6wqm5c1uAlSZIkjYVOpl+uqn+uqgOr6req6klbk8Q03gZ8qqp+E3ggcBlwLPCZqpoCPtMsS5IkSZpArffIJPkacD7wOeBzW9trkuROwKOA5wBU1c+Bnyc5CNi/qXZKc4xXLUfMkqTJtePJG4aue9MRa0YYiSRpa3Qxa9nLGSQiLwE+nGQdv05qzhhi+3sB/wmcnOSBwMXA0cCuVXUtQFVdm2SX+XYwPT29jU9hdBaObYdl2o/6bDR/2z60rT7EOB5W7vNfahtZmW2r/89lZf7d2uR502JG3UampqYWXN/FGJnPMLj0iyQ7A8cARwF/DqwaYhfbAQ8BXlRVFyR5G1t5GdliJ6Ur09PTC8e2dvhvDcf1OWrbLNpGlqoPbasPMY6BkbWRPlhqG1mBbWsi2skK/Lu1aSLaiEZqHNpI62NkkhyQ5A1JvghcAkwBxwEPGHIX64H1VXVBs3wGg8TmuiS7NcfYDbh+eSOXJEmSNC66uLTsbOBy4A3AB6rq5q3ZuKr+I8nVSe5bVd8BHgt8q/k5HHhj8/sTyxu2Vjqvo5ckScvJzxbbpotE5lHA/wIOAf4myTcZjJH5fFV9Ych9vAj4UJLtgSuAIxj0Lp2e5EgGN9c8ZNkjlyRJkjQWuhgjsxZYC7yhGZB/NPBK4HiGGyNDVV0CPHSOVY9drjglSZIkja8upl/+fQbTJD8auA+DWcfewaBXRpIkSZIW1cWlZUczSFqOAb5UVf/TQQySJEkr3vxjNHbYYmY4x2ho3HRxadn+bR9TkiRJ0mTp4tKyY4DPVtUlSfYFTgduBp5ZVV9qOx5J7XKGFkmStBxav48M8FLgyubxG4C3AK8HTuggFkmSJEk91MUYmdVVtTHJHYEHAo+rqluS/H0HsUiSJEnqoS4SmauT/C7wWwzuHXNLkjsBt3QQiyRJkqQe6iKReQVwBvBz4A+asqcAX+kgFkmSJEk91MWsZWcDd59V/NHmR5IkSZIW1UWPDElWA/cF7jBr1Wc7CEeSJElSz3Qx/fJzgHcCPwJ+MmNVAfdqOx5JkpbCqcQlqVtd9Mi8Hji4qv61g2NLkiRJmgBd3EdmO+DTHRxXkiRJ0oTookfmTcBfJnldVf2yg+NL0ljxEiVJkrZeF4nMS4G7Aa9M8t8zV1TVPTqIR5IkSVLPdJHIPKuDY0qSJEmaIF3cR+Zzs8uS3Ao4oO1YJEmSJPVTJ/eR2STJA4BnM+iluRWwS5fxSJIkSeqHLu4jc1fgmcDhwAMY3D/mxcCJbcciSZIkqZ9am345ycFJPglsAJ4LnMbgBpj/CZxRVT/byv2tSvK1JGc1y3sluSDJdJLTkmy/3M9BkiRJ0nhos0fmdOC/gWdU1cc3FSZZ6v6OBi4D7tQsvwl4a1WdmuQ9wJHAu5ceriRJkpaT081rObWZyDyXwXiYjya5CPgQg16Z2todJdkdeDLweuCYDLKhxwB/1FQ5BXgNJjKSWuQbtCRJ7Wnt0rKqen9VPQa4N/AvwIsYXGZ2V+DAJKu2YncnAK8ENt1Qc2fgpqq6uVleD/gpQZIkSZpQXUy//H3gdcDrkuzHYND/Wxn0riyafCR5CnB9VV2cZP9NxXMdar59TE9Pb23YrVk4th2WaT/Dedja4Y934SN/ss3HG3/tnv92993mc1vqscbj/C9sPJ7bOL/GjVbb578PbXJ+4xjT1un3+V9Iu++/bZ/Hyf27LU2/z8eoY5qamlpwfafTL1fVvwH/luRFwNOH3Gw/4GlJDgRuy2CMzAnAjkm2a3pldgeumW8Hi52UrkxPTy8c29rhL1tZlufY9vHG3Ricj0XbyFK1+dyWeqwxOP+LGoPnNrI2skStXm7X9vnvQ5ucx7i1kyXp8flf1BKf27D/b5v9r/nZols9Ph/j8DrSaSKzSTNj2WlD1j0OOA6g6ZF5eVU9M8lHgYOBUxn08nxiNNH2Tx+u2+9DjJK0HHy965bnX5ocrY2RacGrGAz8X8dgzIz3pZEkSZIm1Fj0yCxVVZ0PnN88vgLYp8t4JKlNc3+zvMOclyr4zbIkadK00iOT5LQZj49o45iSJEmSJldbl5Y9Mb++8+XbWjqmJEmSpAnV1qVlXwC+lOS7wG2TfGCuSlX17Jbi0QrmQE9p6/l/I209/2+k0WorkTmEwYxi92Rwf5fLWzquWuaLtiQtP19bpfHn/2n7WklkquqnwAcBkty6ql7bxnElSZIkTabWZy2rqtckmQIOA9YAG4CPVNX43a5UkiRJ0lhqPZFJ8lTgQ8BZwPeB+wIXJfnjqjqz7XikUbOrWZL6zddxTYpJa8td3Efmb4GDquq8TQVJ9gfeAZjISJIkSVpUW9Mvz7Q7g1nMZlrblEuSJEnSorpIZC4BXjar7JimXJIkSZIW1cWlZS8APpnkaOBqYA/gx8DTOohFkiRJUg91MWvZt5PcD9gXuDtwDXBBVf2i7VgkSZIk9VMXPTJU1c0MxsVIkqQVbNJmUZLUnk4SGUmLm/vNfQdYu2W5b+6aNH64lSQtxkRG0kTzA7EkSZOpi1nLJEmSJGmbdNIjk2Qn4KnAGmADcFZV3dBFLNIksfdBkiStFK0nMkkeAfwL8G3g+8BTgBOSPLmqvtR2PJIkSVKf+MXlQBc9MicAf15Vp24qSPIM4O3AwzqIR5IkSVLPdJHI3Ac4fVbZGcB7OohFkiRJ+hV7O/qji0RmGjgU+PCMskOAy4fZOMkewAeAuwG/BN5bVW9LcmfgNGBP4HvAH1bVjcsX9vKZ/x9ky6l1/QeRJEmSttTFrGUvAd6R5MtJTktyAfAu4MVDbn8z8LKquh+wL/DCJHsDxwKfqaop4DPNsiRJkqQJ1HqPTFV9Mcm9gScDdwc+CZw97KxlVXUtcG3z+IdJLmMw+9lBwP5NtVOA84FXLWvwGjt2/0rS/LwCQNIk62T65eaSrw9u636S7Ak8GLgA2LVJcqiqa5PsMt9209PT23robbTD0DU3j7UP2xljt9u1HeNS9OE8LpXPbctjTXL7n+Tn1ibPf9vbddtGbJObbzeebWTL7ZZeZ1tMTU0tuL61RCbJeUAtUKWq6rFbsb87AP8EvKSqfpBk6FgWOykjt3b4XoTNYu3DdsbY7XZtx7gUfTiPS+Vz2/JYk9z+J/m5tcnz3/p2nbYR2+Tm241pG9liuzlMT093/pm6zR6Z+Xpg1jAYHzN0ipjk1gySmA9V1cea4uuS7Nb0xuwGXL9N0UqSJEkaW60lMlV14szlJDsDxwHPYzDb2PHD7CeDrpcTgcuq6i0zVp0JHA68sfn9iWUIW5IkSdIYan2MTJI7Aa8AjgLOAh5SVUNNvdzYD/hj4BtJLmnK/oJBAnN6kiOBqxhM6SxJkiRpArU5RuZ2DKZefhmDGcUeWVWXbu1+qmotMN+AmKHH2EiSJEnqrzZ7ZK4EVgFvBi4Cdk2y68wKVfXZFuORJEnShPIWDZOvzUTmpwxmLXvBPOsLuFd74UiSpOXih0aNim1L82lzsP+ebR1LkiRJ0mS7VdcBSJIkSdLWMpGRJEmS1DutT78sSZLGm2MSJPWBPTKSJEmSesdERpIkSVLvmMhIkiRJ6h0TGUmSJEm942B/SZrDJA92nuTnJqkdvo5oHNgjI0mSJP3f9u481o6yjOP492fFiKLFtqCyWUQxoNFqFBGMQTRQsQrugDsm+ocLJLjhPwqKpmpQNBoTFzCKIC64gEtx31ARrQIpWtkEWWrBthQUKTz+Me/F4XJpC94y99zz/SQ3Z9535sw878mbO/Ocd94zGjkmMpIkSZJGjomMJEmSpJFjIiNJkiRp5JjISJIkSRo5JjKSJEmSRo6JjCRJkqSR43NkJN1jPkdAkiQNZVaNyCRZnOTPSf6a5J1DxyNJkiRpy5g1iUySOcAngOcAewKHJdlz2KgkSZIkbQmpqqFjmBZJnga8p6oObOVjAKrqAwBr166dHQ2VJEmSxszcuXMzuW7WjMgAOwJX9MpXtjpJkiRJs8xsSmTulKUBjsJIkiRJs9Bs+tWyK4Gde+WdgKsmClMNR0mSJEkaTbNpROZc4NFJdk1yP+BQ4FsDxyRJkiRpC5g1IzJVtSHJm4DvA3OAz1XVhQOHJUmSJGkLmE0jMlTVd6pq96raraqOHzqezeXzbzSVJJ9LsirJBb26eUnOTrKyvT5kyBg1rCQ7J/lxkhVJLkxyZKu3nwiAJPdP8tskf2x95NhWv2uS37Q+8uV2J4PGWJI5Sf6Q5MxWto/odkkuS3J+kuVJftfqBj/XzKpEZhT5/BttxMnA4kl17wR+WFWPBn7YyhpfG4Cjq2oPYG/gje3/h/1EE24G9q+qJwCLgMVJ9gaWAh9pfeSfwOsGjFEzw5HAil7ZPqLJnllVi6rqya08+LnGRGZ4ewF/rapLquo/wGnAwQPHpBmgqn4GXD+p+mDg823588Ah92pQmlGq6uqq+n1bvoHuImRH7CdqqrO+FbdqfwXsD3y11dtHxlySnYDnAp9p5WAf0aYNfq4xkRmez7/R3fHQqroauotYYPuB49EMkWQh8ETgN9hP1NNuGVoOrALOBi4G1lTVhraJ5x19FHg7cFsrz8c+ojsqYFmS85K8vtUNfq6ZNZP9R5jPv5H0f0myDfA14KiqWtd9mSp1qupWYFGSbYEzgD2m2uzejUozRZIlwKqqOi/JfhPVU2xqHxlv+1bVVUm2B85OctHQAYEjMjPBRp9/I01ybZKHA7TXVQPHo4El2YouiTmlqr7equ0nupOqWgP8hG4+1bZJJr7M9Lwz3vYFnp/kMrrb2/enG6Gxj+h2VXVVe11F94XIXsyAc42JzPB8/o3ujm8Br27Lrwa+OWAsGli7j/2zwIqqOqG3yn4iAJJs10ZiSLI18Gy6uVQ/Bl7cNrOPjLGqOqaqdqqqhXTXID+qqpdjH1GT5IFJHjSxDBwAXMAMONekypHCoSU5iO7bj4nn34zMT0dry0lyKrAfsAC4Fng38A3gdGAX4G/AS6pq8g8CaEwkeTrwc+B8/ndv+7vo5snYT0SSx9NNwp1D9+Xl6VV1XJJH0n37Pg/4A/CKqrp5uEg1E7Rby95aVUvsI5rQ+sIZrXhf4EtVdXyS+Qx8rjGRkSRJkjRyvLVMkiRJ0sgxkZEkSZI0ckxkJEmSJI0cExlJkiRJI8dERpIkSdLIMZGRJA0qycuTLBs6jumQpJI8aug4JGkcmMhI0phK8pok5ye5Kck1ST6ZZO4WPubCdrE/8cRwquqUqjpgCxxrv3asT0yq/0WS10z38SRJ9y4TGUkaQ0mOBpYCbwPmAnsDC4FlSbYaMLTpdiPwqiQLB47jbuknepKkqZnISNKYSfJg4FjgzVX1vaq6paouA14K7Aoc3rY7Ocn7eu/bL8mVvfIOSb6W5B9JLk3ylt66vZL8Lsm6JNcmOaGt+ll7XZNkfZKntZGhX/Teu0+Sc5Osba/79Nb9JMl7k/wyyQ1JliVZsJHmrgFOBt59F5/Fe5J8sVe+w4hRO977kvyqxfvtJPOTnNLadu4USdJBSS5JsjrJh5Lcp7f/I5KsSPLPJN9P8ojeukryxiQrgZUbaZMkCRMZSRpH+wD3B77er6yq9cB3gU3e5tUuzr8N/BHYEXgWcFSSA9smJwInVtWDgd2A01v9M9rrtlW1TVWdM2m/84CzgI8B84ETgLOSzO9tdjjwWmB74H7AWzcR7vHAi5I8ZlPtuguHAq+ka+duwDnAScA8YAV3TpJeADwZeBJwMHBEa9shwLuAFwLbAT8HTp303kOApwJ73sNYJWlsmMhI0vhZAKyuqg1TrLua7iJ7U54CbFdVx1XVf6rqEuDTdBf9ALcAj0qyoKrWV9WvNzO25wIrq+oLVbWhqk4FLgKe19vmpKr6S1X9iy5BWrSxHVbVNcCngOM2M4bJTqqqi6tqLV2id3FV/aB9fl8Bnjhp+6VVdX1V/Q34KHBYq38D8IGqWtHe+35gUX9Upq2/vrVNkrQRJjKSNH5WAwvuYh7Gw4F/bMY+HgHskGTNxB/daMND2/rXAbsDF7Xbr5ZsZmw7AJdPqrucbjRkwjW95ZuAbTZjv0uBA5M8YTPj6Lu2t/yvKcqTj39Fb/lyujZB95md2Pu8rgfCHdvWf68kaSNMZCRp/JwD3Ex3i9PtkjwQeA7w01Z1I/CA3iYP6y1fAVxaVdv2/h5UVQcBVNXKqjqM7vavpcBX2/5rE7FdRXfB37cL8PfNbt0Uquo6utGR905atbE23lM795Z3oWsTdJ/ZGyZ9ZltX1a/6oU7D8SVpLJjISNKYabdIHQt8PMniJFu1CetfoRutOaVtupxu4vq8JA8Djurt5rfAuiTvSLJ1kjlJHpfkKQBJXpFku6q6jW7CPcCtdKM9twGPvIvwvgPsnuTwJPdN8jK6+SJnTkPTT6CbH7RHr2458Iwku7Sfnj5mGo7ztiQPSbIzcCTw5Vb/KeCYJI8FSDI3yUum4XiSNJZMZCRpDFXVB+luBfswcANwKd3IxLOr6sa22RfoJvNfBizjfxfkVNWtdPNWFrX3rgY+Q/dTzgCLgQuTrKeb+H9oVf27qm6im3z/y3aL1d6T4roOWAIcDVwHvB1YUlWrp6HN64AP0k3Sn6g7u7XrT8B5TE/C9M22r+V0P1zw2XasM+hGp05Lsg64gG4ETJJ0D6TKUWxJGndJjqAbpdm3TVKXJGlGM5GRJAGQ5JXALVV12tCxSJK0KSYykiRJkkaOc2QkSZIkjRwTGUmSJEkjx0RGkiRJ0sgxkZEkSZI0ckxkJEmSJI0cExlJkiRJI+e/RkSrRdNj8ugAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "f, ax = plt.subplots(3, 1, figsize=(12, 8))\n", "\n", "ax[0].bar(df1.index, df1.no_of_votes)\n", "ax[0].set_ylabel('No of Votes')\n", "\n", "ax[1].bar(df1.index, df1.no_of_views)\n", "ax[1].set_ylabel('No of Views')\n", "\n", "ax[2].bar(df1.index, df1.no_of_answers)\n", "ax[2].set_ylabel('No of Answers')\n", "plt.xlabel('Question Number')\n", "\n", "plt.savefig('votes_vs_views_vs_answers.png', bbox_inches='tight')\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here, we may observe that there is no collinearity between the votes, views and answers related to a question. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Useful Resources:\n", "- [Dataquest Tutorial 1](https://www.dataquest.io/blog/web-scraping-tutorial-python/), [2](https://www.dataquest.io/blog/web-scraping-beautifulsoup/)\n", "- [HackerNoon Tutorial](https://hackernoon.com/building-a-web-scraper-from-start-to-finish-bb6b95388184)\n", "- [RealPython Tutorial](https://realpython.com/python-web-scraping-practical-introduction/)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" } }, "nbformat": 4, "nbformat_minor": 2 }