{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## How to download and store lots of files\n", "\n", "[**Neal Caren**](mailto:neal.caren@gmail.com) \n", "University of North Carolina, Chapel Hill" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Web scraping frequently involves analyzing hundreds or thousands of files. Figuring out what data you need from a page and the best way to extract is often an iterative process. After processing all the documents, your scraper might not have worked the way you thought it was working. The format of the pages might be different for a subsample of the pages and your scraper returns nothing for them. Six months later, you might want to get one more piece of information from each page. Downloading web pages or using an API takes time and costs the provider money. The content of pages and their availability also changes over time. This all means that it makes the most sense to gather all your web data and store it locally before you start analyzing it. \n", "\n", "This lessons walks through the basics steps for downloading and storing data. This lessons assumes that you have some basic knowledge of Python and a list of URLs you want to download.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I begin by importing the standard library for acessessing web pages or APIs, [requests](http://docs.python-requests.org/en/master/). " ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import requests" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "No matter how many URLs I eventually need to download, I start with a sample one to make sure the process works before creating a loop. In this case, I'm interested in Fox News opinion pieces:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "url = 'https://www.foxnews.com/opinion/gutfeld-on-hiring-combat-vets-to-defend-schools'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using `requests`, I access the page and retrieve the HTML." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "r = requests.get(url)\n", "html = r.text" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometimes, this works but the actual content is not what you get when you access the web page using a browser. To confirm that things went as planned, I display the contents of the file as a browser would." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " Gutfeld on hiring combat vets to defend schools | Fox News\n", " \n", " \n", "
Fox News
Expand / Collapse search
Watch TV
☰
Fox News

\n", " This material may not be published, broadcast, rewritten,\n", " or redistributed. ©2019 FOX News Network, LLC. All rights reserved.\n", " All market data delayed 20 minutes.\n", "

ADVERTISEMENT
OPINION
Published

Gutfeld on hiring combat vets to defend schools

\"Greg
\n", " By\n", " Greg Gutfeld | Fox News
close
\"GutfeldVideo

Gutfeld on hiring combat vets to defend schools

If armed guards at schools can protect America's kids, lock and load.

A Florida school has hired two combat vets to protect students and staff from active shooters.

According to Principal Bill Jones, any armed intruder entering the Manatee School for the Arts campus posing a threat to students would be "shot and killed."

Jones says: “If someone walks in shooting, somebody walks in with a weapon threatening kids, we're not, we’re not going to fool around. We're not going to pause, we’re not going to ask them questions. We're going to shoot them…. Eliminate the threat. You know, if they die in the process that's unfortunate, but better them than one of our students, quite frankly.”

ONE YEAR LATER, PARKLAND FAMILIES STILL TRYING TO HEAL: ‘I HAVE TO FIGHT TO GET THROUGH THE DAY’

We’re not going to talk with them, we're not going to negotiate. We are going to put them down, as quickly as possible. Well, that is a plan. Compare that to the wishy-wash of the Broward sheriff, who was so anxious to blame the NRA that he forgot to look in the mirror at himself.

Manatee is a charter school operating under its own authority – so school officials can decide who to hire. They went with armed combat vets. One guard is a 15-year Army vet who served three tours. So the school's protected by the best defenders on Earth, and those vets have jobs that contribute to the safety of society.

After Parkland, all Florida schools were mandated to have armed security. It’s a year to the day since that massacre.

Here’s a fact: the duration of a mass shooting is shortened by the arrival of a second gun.

Manatee shrunk that duration to zero. The school has 2,100 students. I'm sure some parents there are spooked by having armed guards with rifles. But others might be resting easier.

 CLICK HERE TO GET THE FOX NEWS APP 

And it’s often hard to ponder solutions with a media placing the issue within a prison of two ideas. If you aren't for gun control, you want kids to die. This, as their wall-to- wall coverage creates a spectacle for copycats who want to leave a mark on the world.

It is sobering, that, in this peaceful age, we need soldiers at school. But if it works then, I say give it a try.

Adapted from Greg Gutfeld’s monologue on “The Five” on Feb. 14, 2019.

CLICK HERE TO READ MORE FROM GREG GUTFELD

Greg Gutfeld currently serves as host of FOX News Channel's (FNC) The Greg Gutfeld Show (Saturdays 10-11PM/ET) and co-host of The Five (weekdays 5-6PM/ET). He joined the network in 2007 as a contributor. Click here for more information on Greg Gutfeld. 
\n", " \n", " \n", "\n" ], "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from IPython.display import HTML\n", "\n", "HTML(html)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since that worked, I want to save the file. I create a directory to store what will eventually be numerous files." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "scrolled": true }, "outputs": [ { "ename": "FileExistsError", "evalue": "[Errno 17] File exists: 'fox-html'", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mFileExistsError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmkdir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'fox-html'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;31mFileExistsError\u001b[0m: [Errno 17] File exists: 'fox-html'" ] } ], "source": [ "import os \n", "\n", "os.mkdir('fox-html')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If the directory already exists, this command will produce an error.\n", "\n", "Before the file it saved, it needs a name. URLs often involve characters that can cause trouble file names, such as \\,: or ?. I use slugify to create an operating-system safe name for the file. This may need to be added to your python installation" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: python-slugify in /Users/nealcaren/anaconda3/lib/python3.6/site-packages (1.2.6)\r\n", "Requirement already satisfied: Unidecode>=0.04.16 in /Users/nealcaren/anaconda3/lib/python3.6/site-packages (from python-slugify) (1.0.23)\r\n" ] } ], "source": [ "!pip install python-slugify" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "slugify creates a filename that is both usable by your computer and human readable." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'https-www-foxnews-com-opinion-gutfeld-on-hiring-combat-vets-to-defend-schools'" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from slugify import slugify\n", "\n", "slugify(url)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Saving text files is one of the more non-intuitive aspects of Python. In this case we also need to create the file name and make sure the file is put in the correct directory." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "file_name = slugify(url)\n", "directory = \"fox-html\"\n", "\n", "location = os.path.join(directory, file_name)\n", "\n", "with open(location, \"w\") as outfile:\n", " outfile.write(html)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I'll use this process of downloading and saving many times, so I put it in a function that takes two parameters: the URL to create the filename and the html text to be saved. I put the construction of a the file location in a separate function because I will need to use that piece later in a different context. I also have the script pause for three seconds so the web server is not bombarded by our program. " ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "from time import sleep\n", "\n", "def locate(url):\n", " '''Create file name and place in directory'''\n", " file_name = slugify(url)\n", " location = os.path.join(\"fox-html\", file_name)\n", " return location\n", "\n", "\n", "def get_html(url):\n", " \"\"\"Download & save ave html text using the url to create the filename.\"\"\"\n", " sleep(3)\n", " r = requests.get(url)\n", " html = r.text\n", " \n", " location = locate(url)\n", "\n", " with open(location, \"w\") as outfile:\n", " outfile.write(html)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When it works, it does not return or display anything. " ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "get_html(url)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that I the project specific \"fox-html\" directory is hard coded into the function. If you copy and paste this function, be sure to update it to whatever name you are using." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The function could now be used to download many URLs. For example, if you had a list of six URLs, you could loop over them." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "urls = [\n", " \"https://www.foxnews.com/opinion/newt-gingrich-i-saw-the-south-korean-miracle-for-myself-it-was-incredible\",\n", " \"https://www.foxnews.com/opinion/gutfeld-on-hiring-combat-vets-to-defend-schools\",\n", " \"https://www.foxnews.com/opinion/doug-schoen-amazons-cancellation-of-a-move-to-nyc-is-bad-news-and-could-hurt-far-left-dems-at-polls\",\n", " \"https://www.foxnews.com/opinion/amazon-quits-new-york-victory-by-progressives-helps-city-wave-good-bye-to-25000-jobs\",\n", " \"https://www.foxnews.com/opinion/william-barr-is-our-new-attorney-general-here-are-four-things-he-should-focus-on-right-away\",\n", " \"https://www.foxnews.com/opinion/karl-rove-trumps-approval-numbers-are-going-up-three-things-could-keep-his-momentum-going\",\n", "]" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "for url in urls:\n", " get_html(url)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If something when wrong, the loop would stop with an error message, but we can confirm that it worked by listing the files in the html directory." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['https-www-foxnews-com-opinion-gutfeld-on-hiring-combat-vets-to-defend-schools',\n", " 'https-www-foxnews-com-opinion-amazon-quits-new-york-victory-by-progressives-helps-city-wave-good-bye-to-25000-jobs',\n", " 'https-www-foxnews-com-opinion-karl-rove-trumps-approval-numbers-are-going-up-three-things-could-keep-his-momentum-going',\n", " 'https-www-foxnews-com-opinion-doug-schoen-amazons-cancellation-of-a-move-to-nyc-is-bad-news-and-could-hurt-far-left-dems-at-polls',\n", " 'https-www-foxnews-com-opinion-newt-gingrich-i-saw-the-south-korean-miracle-for-myself-it-was-incredible',\n", " 'https-www-foxnews-com-opinion-william-barr-is-our-new-attorney-general-here-are-four-things-he-should-focus-on-right-away']" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "os.listdir('fox-html')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you had a modest sized list of files to download, you are done. You now have a directory with all your files read for subsequent analysis.\n", "\n", "If you have a larger list of files, however, you would want the loop to robust to errors and to be able to restart without re-downloading files you have already collected. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `get_html` code can be modified so that if something goes wrong with the download it prints out an error message and then pauses for 10 seconds. Since the problem may be related to the server, you want to slow down the process when things are not going well." ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "def get_html(url):\n", " \"\"\"Download & save ave html text using the url to create the filename.\"\"\"\n", " sleep(3)\n", " try:\n", " r = requests.get(url)\n", " html = r.text\n", " except:\n", " print(\"Problem with\",url)\n", " sleep(10) \n", " else:\n", " location = locate(url)\n", " with open(location, \"w\") as outfile:\n", " outfile.write(html)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The loop can also modified to download the file from the internet only once. If you are downloading thousands of files, something will go wrong and you don't want all your work to go to waste. \n", "\n", "The Python way to include this check in the loop is to first confirm whether or not the file exists. If it doesn't exist, because our attempt to open it fails, then, and only then, do we download it.\n", "\n", "I first write a helper function that attempts to load the file based on the URL name. This takes advantage of our previously defined `locate` function." ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "def confirm_file(url):\n", " \"\"\"Attempt to open a file based on the url.\"\"\"\n", " location = locate(url)\n", "\n", " with open(location, \"r\") as infile:\n", " html = infile.read()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I now update the loop. First, trying to open the file. When that doesn't work, I attempt to download it." ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "for url in urls:\n", " try:\n", " confirm_file(url)\n", " except:\n", " get_html(url)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When you are running a loop like this on a modest number of URLs, you might add a `print` statement for each URL just to make sure that things are progressing. For 100,000 URLs, however, this will clog your screen and computer memory, so make sure to remove them before letting the script wild." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.9" } }, "nbformat": 4, "nbformat_minor": 2 }