{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Files\n", "\n", "The data we have worked with so far has been entered directly\n", "into the python code. For programs running on relatively small\n", "amounts of data, this is often a feasible approach. However,\n", "when we are writing code, more often than not it is to perform\n", "some computations on more than just a few values. In many cases\n", "we wish to perform computations on data that comes from another\n", "source, such as data collected from a science experiment or\n", "survey or data acquired from the internet, etc. In these\n", "cases, having to type all of the data directly into the\n", "python code does not make much sense. In many cases,\n", "it was already recorded in a file by someone else, and duplicating\n", "that work to type it into a list is inefficient.\n", "\n", "When you wish to work with data that is already in a file,\n", "we perform what's known as file I/O (input/output).\n", "In python, to interact with files we take the following\n", "steps:\n", "* open the file - this gives us what's known as handle to the\n", " file so we can interact with it\n", "* read/write - this is where we either access the data in the\n", " file or add data to a file (or both)\n", "* close the file - our way of saying we are done modifying\n", " the file and all of the changes should be saved\n", "\n", "## Opening Files\n", "\n", "The first step, opening files is done with the function\n", "`open(filename, mode)`, where\n", "* `filename` is a string containing the name of the file. \n", " It can also be what's known as a *path* to the file and\n", " include folders that the file is in.\n", "* `mode` is a string indicating how we plan to interact\n", " with the file. The primary options are:\n", " * `r` = read\n", " * `w` = write, this will overwrite a file that already exists\n", " * `a` = append, this will add to the end of a file that already exists\n", " * `r+` = read and write\n", " \n", " There are technically a few other modes for files that\n", " don't contain text, such as an mp3 file for a song. For\n", " these, the modes have a `b` at the end, e.g. `rb`, `wb`, etc.\n", " But, we'll focus on reading/writing text based files as that\n", " is the most common to do with base python (if you\n", " were going to read in an audio file or an image file, there's a good\n", " chance you'd use an separate python package.)\n", "\n", "If the file is able to be opened, we get what is known as\n", "a *file handle*, essentially a connection to the file so\n", "that we can access it. Note this isn't actually the file itself.\n", "Sometimes, a file is unable to be opened, in which case the code\n", "will raise an error. For instance, if we try to open a file\n", "for reading that doesn't exist, we will get a `FileNotFoundError`.\n", "\n", "For example:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fr = open('data/blahblahreading.txt', 'r')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that there is no issue with opening a file that doesn't exist\n", "for writing, it will simply create the file." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fw = open('data/blahblahwriting.txt', 'w')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that we can still get errors while opening for writing\n", "if we were on a computer trying to open a file we did not have\n", "the permissions to edit (hard to simulate on notebooks running\n", "on CoCalc or other cloud based servers).\n", "\n", "Let's open the file \"robert-frost.txt\" in the \"data\" folder,\n", "which contains the text for the poem \"Stopping by Woods on a\n", "Snowy Evening\" by Robert Frost.\n", "Because this file is nested in a folder, we need to give it\n", "the path, which is just the folder name, followed by a \"/\"\n", "followed by the filename, e.g. \"data/robert-frost.txt\".\n", "Once we have a file handle, it's just another variable in python\n", "and we can do things like print it. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ffrost = open('data/robert-frost.txt')\n", "print(ffrost)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that when we print\n", "it, we get a printed version of the file handle, we don't actually\n", "get the file contents. This is an important distinction -- a file\n", "handle isn't actually the contents of the file, we need to read from\n", "the file in order to get the contents." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reading/Writing Files\n", "\n", "Once a file is open and we have a file handle, we can either read from\n", "or write to the file (depending on the mode with which we opened it).\n", "\n", "### Reading Files\n", "\n", "To read from files, we use the desired file handle corresponding to the\n", "file we wish to read from to call one of the reading methods. The\n", "available reading methods are:\n", "\n", "* `read()` - returns a string of the entire input file\n", "* `readlines()` - returns a list with each line in the file as a string\n", " in the list\n", "* `readline()` - returns a string of the next line in the file\n", "\n", "The first two options will store the entire file in variables at once,\n", "so while convenient are often not appropriate for very large files.\n", "\n", "Let's look at some examples of reading on \"robert-frost.txt\" file containing\n", "one of Robert Frost's poems. Note that once we reach the end of the file\n", "once, subsequent calls (even to a different reading method) will return\n", "an empty string or list. So, in order to illustrate the difference,\n", "we first reopen the file each time." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ffrost = open('data/robert-frost.txt')\n", "fulltext = ffrost.read()\n", "print(len(fulltext))\n", "print(fulltext)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ffrost = open('data/robert-frost.txt')\n", "lines = ffrost.readlines()\n", "print(lines)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that in the above example, each string in the list ends\n", "with `\\n`, the newline character. This is because when reading\n", "from a file, the newline characters are read as well. Assuming\n", "you would go process the strings in this list after reading, you\n", "would likely first want to remove the newline. This could be\n", "done with slicing or (more commonly) by using the `strip()` method." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(len(lines[0]))\n", "print(len(lines[0].strip()))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Reading and Looping\n", "For big files, it's more efficient to process\n", "the file line-by-line. The `readline()` method is one\n", "option that does this. Since it only reads a single\n", "line of the file at a time, we would typically need\n", "to put it in a loop to process the whole file (until we\n", "get an empty line signaling we've reached the end of the\n", "file). However, there is an even easier way, we can\n", "simply loop on the file handle, similar to how we\n", "can loop through a list:\n", "```\n", "for line in f:\n", " # do something with line\n", "```\n", "\n", "For instance, suppose we just wanted to count the lines\n", "in the file:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ffrost = open('data/robert-frost.txt')\n", "count = 0\n", "for line in ffrost:\n", " count += 1\n", "print(count)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With any of the methods of reading, once it is read in, it is simply\n", "a string (or list of strings), that could be processed like\n", "any other string to break it up into words, convert to another\n", "type to perform computations, etc." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Writing Files\n", "\n", "We can write to a file by calling the `write(textstr)` method\n", "on the file handle. This method writes the text in the string\n", "`textstr` to the file\n", "\n", "For example, using the `fw` file handle above:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fw.write(\"hello world\\n\")\n", "fw.write(\"hello world line 2\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Closing Files\n", "\n", "Once we are finished working with a file (i.e. no longer need to read from\n", "or write to the file), we must close the file to indicate to the computer\n", "that we are no longer potentially going to access it or modify it. This will\n", "also ensure that any changes made when writing to a file take effect timely.\n", "\n", "We close files in python by calling the `close()` method on the specific\n", "file handle we wish to close. If we wish to close more than one\n", "file, we must call `close()` for each. For instance, suppose we want to close\n", "`fw`, `ffrost`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fw.close()\n", "ffrost.close()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Alternative to Open/Close\n", "\n", "The downside with opening files is remembering to close them later.\n", "However, often the actual code for which the file needs to be open\n", "is relatively small (because the desired file contents may be stored\n", "in a variable to use at a later point in the code). When this is the\n", "case, there is a popular shorter form that handles the closing of\n", "the files automatically using the `with` statement. The `with`\n", "statement is used to execute a block of code within a context (where\n", "something holds within that context). For files, we can put the call\n", "to open in the `with` statement, so anything executed within that block\n", "of code is executed with the file open, but the file is automatically closed\n", "at the end, even if an error occurs. This sort of block looks like:\n", "\n", "```\n", "with open(filename, mode) as f:\n", " # code to process file\n", "```\n", "\n", "Inside the with block, we would still put the desired calls to read/write\n", "as appropriate, but we no longer need to worry about closing the file." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example - Groundhogs Day Data\n", "\n", "The file \"data/groundhog.csv\" contains the historical data for Punxsutawney Phil on Groundhog's Day.\n", "Unlike the Robert Frost poem, this data is more structured -- the first 2 lines are header information\n", "and the remaining lines contain multiple pieces of data separated by commas. \n", "\n", "Suppose we wish to count the number of years\n", "in which Phil saw a partial shadow. At a high level, this would involve:\n", "* loop through the lines of the file\n", "* strip excess whitespace from the line and split into list of strings by commas (since\n", " commas separate the columns)\n", "* check if line has 2nd entry, and if 2nd entry in list is \"Partial Shadow\",\n", " if so increment count of partial shadows\n", "\n", "Let's look at what this looks like in code:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "count = 0\n", "with open('data/groundhog.csv', 'r') as f:\n", " for line in f:\n", " line = line.strip().split(',')\n", " if len(line) > 2:\n", " if line[1] == 'Partial Shadow':\n", " count += 1\n", "\n", "print(count)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reading CSV Files\n", "\n", "Above we saw how to read in files by either\n", "* reading in all of the lines as a list of strings with `readlines()`\n", "* reading in the entire file with `read()`\n", "* lopping through the file handle itself (which loops through the lines\n", " of the file, each line as a string).\n", "\n", "For formatted files (where each line is made\n", "up of a bunch of columns), often you would use string methods\n", "to split apart each line and extract the desired column information.\n", "For instance, a file, `books-small.csv`, with lines like:\n", "```\n", "author,book title\n", "Harper Lee,\"To Kill a Mockingbird\"\n", "John Steinbeck,\"Of Mice and Men\"\n", "```\n", "\n", "Might be processed with code like:\n", "```\n", "with open('books-small.csv') as f:\n", " for line in f:\n", " book_info = line.split(',')\n", "```\n", "\n", "This works well for many scenarios. However, imagine\n", "the file contained more lines, and some of the\n", "book titles had commas in the title.\n", "For example,\n", "```\n", "author,book title\n", "John Steinbeck,\"Of Mice and Men\"\n", "Harper Lee,\"To Kill a Mockingbird\"\n", "Iris Murdoch,\"The Sea, the Sea\"\n", "James Kelman,\"How Late it Was, How Late\"\n", "Philip Hoare,\"Leviathan or, the Whale\"\n", "```\n", "While we could come up with a solution for\n", "this file, if there were multiple columns that\n", "may have commas in them, it quickly becomes\n", "a more tedious task.\n", "\n", "Since csv (comma separated value) files are so\n", "common, there is a library designed solely to\n", "help read/write from csv files, called `csv`.\n", "To use this library for reading csv files, we first\n", "create a csv reader object with\n", "```\n", "reader = csv.reader(filehandle, delimiter=',')\n", "```\n", "Then, instead of looping through\n", "the file handle, we loop through `reader`.\n", "Whereas with the standard way of handling\n", "files we would need to split up the line\n", "ourselves, when we loop through `reader`\n", "the lines will already be split into\n", "a list of strings, where each entry is one of the\n", "columns. This will intelligently handle commas\n", "that occur inside a column (for well formatted\n", "csv files).\n", "\n", "Let's take a look at an example. The file\n", "`books.csv` in the `data` folder\n", "contains the books as listed above.\n", "We first start by importing the library:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import csv" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, we open the file and create our\n", "csv reader.\n", "We could either open the file or use the\n", "`with` notation. In this example, we'll\n", "just print out the book titles, but you could\n", "do any additional processing desired for each line." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "with open('data/books.csv') as f:\n", " reader = csv.reader(f, delimiter=',')\n", " firstline = True\n", " for line in reader:\n", " if firstline:\n", " firstline = False\n", " else:\n", " print(line[1])" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 4 }