{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Intermediate Python: Programming\n", "# Class 4\n", "\n", "So far in this course,\n", "we've learned about different programming structures in Python to assist in automating tasks, \n", "and also explored ways to document and define defaults for functions.\n", "In today's class, \n", "we'll complete the course by thinking about ways we can make our functions as robust and reusable as possible.\n", "\n", "By the end of this class,\n", "you should be able to:\n", "\n", "- perform debugging on functions creating errors\n", "- understand principles of test-driven development\n", "- write command-line programs for Python code\n", "\n", "## Debugging \n", "\n", "In our previous class, \n", "we discussed testing and validating our code.\n", "If you do identify a problem that either prevents the code from working,\n", "or that prevents the code from accomplishing the task you had in mind,\n", "you'll need to debug your code.\n", "\n", "> If you would like additional explanations for the concepts covered in this section, more detail is available in the original lessons from [Software Carpentry](https://swcarpentry.github.io/python-novice-inflammation/11-debugging/index.html).\n", "\n", "Debugging code associated with research analysis is particularly challenging.\n", "We're writing code to find out an answer to a question, \n", "so validating that our answers are accurate is difficult.\n", "\n", "Keeping in mind your overall goal for what the code should accomplish, \n", "a reasonable process for writing research software includes:\n", "\n", "- *Testing with subset data:* Use a few example data points for testing before moving on to the entire dataset.\n", "- *Testing with simplified data:* Use synthetic data, or subset to a simpler unit (e.g., only one chromosome instead of the entire genome.\n", "- *Compare to known findings:* Use well-established reports from previously published literature, specifically from model systems, to confirm your software is finding equivalent results.\n", "- *Check conservation laws:* Use summary statistics to confirm the number of samples included doesn't vary in unexpected ways. For example, if you are filtering out data, the number of data points should decreased rather than increase.\n", "- *Visualize:* Although it's difficult to compare figures in an automated way (e.g., with a computer), we can use data visualizations to confirm our assumptions are being met. In fact, we modeled this approach in earlier classes in this course.\n", "\n", "We'll generally \n", "With these general steps in mind,\n", "let's explore some specific approaches to assist in the debugging process.\n", "\n", "- *Ensure failures are consistent.* Double check that you've executed all the code that your problem code needs to run, and that you're using the data you initially intended. It's easy to blame the code for not working, when it's actually a mistake we made when trying to run it.\n", "- *Fail quickly and efficiently:* Minimize the time it takes to get the error to resurface, and isolate the portion of the code involved.\n", "- *Change one thing at a time*: Make only one alteration before testing your code again, and be rational in choosing what change to try next.\n", "- *Keep track of what you've done:* Don't make yourself repeat an experiment, and also be able to remember what happened when you last tried something.\n", "- *Ask for help:* Whether someone in your lab group, other members of your computational community, or strangers online, you might be able to save yourself a lot of time and energy by leveraging someone else's expertise. As a bonus, it's possible that framing your problem in terms someone else could understand will help you figure it out on your own!\n", "\n", ">#### Challenge-debug\n", "The following code computes the Body Mass Index (BMI) of patients. \n", "BMI is calculated as weight in kilograms divided by the square of height in metres.\n", "The test results indicate all patients appear to have have unusual and identical BMIs, \n", "despite having different physiques. \n", "*Thinking about the tips for debugging described above,* \n", "what suggestions do you have to improve this code?\n", "\n", "```python\n", "patients = [[70, 1.8], [80, 1.9], [150, 1.7]]\n", "\n", "def calculate_bmi(weight, height):\n", " return weight / (height ** 2)\n", "\n", "for patient in patients:\n", " weight, height = patients[0]\n", " bmi = calculate_bmi(height, weight)\n", " print(\"Patient's BMI is: %f\" % bmi)\n", "```\n", "\n", "*Note:* The syntax with percentage signs seen in the challenge above is a type of string formatting used to round `bmi`.\n", "For more information about string formatting, see [this article](https://realpython.com/python-f-strings/).\n", "\n", ">#### Challenge-pair\n", "Take one of the functions you've written for this course and deliberately introduce an error. Share that error with one of the other course participants and attempt to debug each other's errors." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Test-driven development\n", "\n", "Our normal tendency when writing software is to:\n", "- Write a function.\n", "- Call it interactively on two or three different inputs.\n", "- If it produces the wrong answer, fix the function and re-run that test.\n", "\n", "A better practice is to:\n", "- Write a short function for each test.\n", "- Write a function that should pass those tests.\n", "- If the function produces any wrong answers, fix it and re-run the test functions.\n", "\n", "This latter process is called test-driven development (TDD),\n", "and is a way of thinking about designing code.\n", "\n", "In this section,\n", "we'll write a function to assess whether input ranges overlap.\n", "We'll apply practices of test-driven development,\n", "and also apply many of the programming methods we've developed over the duration of this class.\n", "\n", "> We're not applying TDD to our inflammation data because we've already written the functions. \n", "We've chosen a small stand-alone task for this section to demonstrate TDD, \n", "but we'll get back to our inflammation data in the next section.\n", "\n", "We'll begin by writing three test functions for our function, \n", "which we'll call `range_overlap`.\n", "This first set of tests are \"positive controls\"\n", "that should work for multiple types of input:\n", "\n", "```python\n", "# one input\n", "assert range_overlap([ (0.0, 1.0) ]) == (0.0, 1.0) \n", "# two inputs\n", "assert range_overlap([ (2.0, 3.0), (2.0, 4.0) ]) == (2.0, 3.0) \n", "# three inputs\n", "assert range_overlap([ (0.0, 1.0), (0.0, 2.0), (-1.0, 1.0) ]) == (0.0, 1.0) \n", "```\n", "\n", "If we run these, errors are expected! \n", "We haven't written the function yet.\n", "If tests passed we'd know we were accidentally using someone else's function.\n", "These tests implicitly define what our input and output look like: a list of pairs as input, and produce a single pair as output.\n", "\n", "Next, we can write a test for when ranges do not overlap:\n", "\n", "```python\n", "assert range_overlap([ (0.0, 1.0), (5.0, 6.0) ]) == None\n", "```\n", "\n", "Here, we've made the decision that no overlap means there is no output.\n", "Alternatively, we could create a failure with an error message,\n", "or a special value (`(0.0, 0.0)`) to indicate no overlap.\n", "\n", "Next, we'll test for when ranges touch at their endpoints:\n", "\n", "```python\n", "assert range_overlap([ (0.0, 1.0), (1.0, 2.0) ]) == None\n", "```\n", "\n", "Again, we need to decide (in the context of our research)\n", "whether this counts as overlap.\n", "Here, we've decided overlaps need to have non-zero width, \n", "so no output will be reported.\n", "\n", "Finally, it's good practice to include an error for when the input is completely missing:\n", "\n", "```python\n", "assert range_overlap([]) == None\n", "```\n", "\n", "Next, we'll actually write our function (spoiler: there are errors in this):\n", "\n", "#### range_overlap\n", "```python\n", "def range_overlap(ranges):\n", " '''Return common overlap among a set of [left, right] ranges.'''\n", " max_left = 0.0\n", " min_right = 1.0\n", " for (left, right) in ranges:\n", " max_left = max(max_left, left)\n", " min_right = min(min_right, right)\n", " return (max_left, min_right)\n", "```\n", "\n", "And define our test function:\n", "\n", "#### test_range_overlap\n", "```python\n", "def test_range_overlap():\n", " assert range_overlap([ (0.0, 1.0), (5.0, 6.0) ]) == None\n", " assert range_overlap([ (0.0, 1.0), (1.0, 2.0) ]) == None\n", " assert range_overlap([ (0.0, 1.0) ]) == (0.0, 1.0)\n", " assert range_overlap([ (2.0, 3.0), (2.0, 4.0) ]) == (2.0, 3.0)\n", " assert range_overlap([ (0.0, 1.0), (0.0, 2.0), (-1.0, 1.0) ]) == (0.0, 1.0)\n", " assert range_overlap([]) == None\n", "```\n", "\n", "Finally, we'll test our function:\n", "\n", "```python\n", "test_range_overlap()\n", "```\n", "\n", "The first test was supposed to produce `None`, \n", "but it fails!\n", "Now we know we need to correct our function.\n", "Note that we don't know if any other tests failed, \n", "because the program halts after the first error.\n", "\n", "In this case, \n", "our function is not working because we've initialized with absolute values, instead of real data. \n", "\n", ">#### Challenge-range_overlap\n", "Try to resolve the error in `range_overlap`.\n", "Rerun your test after each change you make.\n", "What other errors do you receive? \n", "How could you resolve them?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Command-line programs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this final section,\n", "we'll start working with our python code in a different way.\n", "Rather than including the code directly in our interpreter, \n", "we'll include our functions in python scripts,\n", "which we can then execute on the command line. \n", "\n", "We could open a separate program for command line work,\n", "but we can execute command line functions in our interpreter by encasing the commands inside `system(\"\")`.\n", "\n", "For example, we can list the files in our project directory using `ls`, \n", "which is the unix command for list in two ways.\n", "\n", "Using `system()`:\n", "```python\n", "system(\"ls\")\n", "```\n", "\n", "Using `!`:\n", "```python\n", "!ls\n", "```\n", "\n", "We'll be using the second method because it requires less typing.\n", "Regardless of the method you use, \n", "the output you should see includes:\n", "```\n", "['class1.ipynb',\n", " 'class2.ipynb',\n", " 'class3.ipynb',\n", " 'class4.ipynb',\n", " 'data',\n", " 'python-novice-inflammation-data.zip']\n", "```\n", "\n", "> If you are comfortable on the command line and have a preferred method of executing Unix code\n", "(e.g., Terminal on Mac),\n", "you are welcome to use that interface instead.\n", "\n", "In this section, \n", "we'll be writing a script to print summary statistics for inflammation per patient.\n", "We'll begin with a simple script,\n", "and gradually add features that expands its functionality.\n", "\n", "At the end, \n", "our script should:\n", "- read data from standard input if no filename is given\n", "- read data from all files if more than one is given, and report statistics for each file separately\n", "- use flags (for min, mean, and max) to determine what statistic to print\n", "\n", "If you'd like pre-written copies of the scripts we'll be using, feel free to execute the following code:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "import zipfile\n", "import urllib.request\n", "\n", "urllib.request.urlretrieve(\"https://swcarpentry.github.io/python-novice-inflammation/code/python-novice-inflammation-code.zip\", \"python-novice-inflammation-code.zip\")\n", "zipData = zipfile.ZipFile(\"python-novice-inflammation-code.zip\")\n", "zipData.extractall()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, create a text file in your project directory called `sys_version.py` that contains the following text:\n", "\n", "```python\n", "import sys\n", "print('version is', sys.version)\n", "```\n", "\n", "> If you are working in Jupyter notebooks,\n", "you can create a new file using the same button as creating a new notebook,\n", "but select \"New File\" under \"Other.\"\n", "\n", "We can execute the command using:\n", "\n", "```python\n", "!python sys_version.py\n", "```\n", "\n", "The output should be something like:\n", "\n", "```\n", "version is 3.6.8 |Anaconda, Inc.| (default, Dec 29 2018, 19:04:46) \n", "[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]\n", "```\n", "\n", "This describes the version of Python you are running.\n", "\n", "Now, create another script called `argv_list.py` containing:\n", "\n", "```python\n", "import sys\n", "print('sys.argv is', sys.argv)\n", "```\n", "\n", "`argv` stands for \"argument values.\" \n", "These are the things listed on the command line,\n", "including the script/function and other arguments.\n", "\n", "If you run the script with no arguments:\n", "\n", "```python\n", "!python argv_list.py\n", "```\n", "\n", "You should only see the name of the script as output:\n", "\n", "```\n", "sys.argv is ['argv_list.py']\n", "```\n", "\n", "> We'll be using `sys.argv` for the duration of this lesson. \n", "When you begin writing your own programs with more complex command line operations,\n", "we recommend using the [`argparse`](https://docs.python.org/3/howto/argparse.html) library.\n", "\n", "If you add some arguments:\n", "\n", "```python\n", "!python argv_list.py first second third\n", "```\n", "\n", "The output should now include:\n", "\n", "```\n", "sys.argv is ['argv_list.py', 'first', 'second', 'third']\n", "```\n", "\n", "Now that we have a basic understanding of these features, let's begin writing our inflammation script.\n", "We'll store this script in `readings.py`.\n", "\n", "> If you downloaded the zipped code files,\n", "the specific script involved at each stage is listed above each relevant section.\n", "\n", "#### readings_01.py\n", "\n", "Our final desired code has a few different requirements.\n", "We'll start with a basic structure of an outlined function.\n", "Conventionally, this function is called `main`:\n", "\n", "```python\n", "import sys\n", "import numpy\n", "\n", "\n", "def main():\n", " script = sys.argv[0]\n", " filename = sys.argv[1]\n", " data = numpy.loadtxt(filename, delimiter=',')\n", " for row_mean in numpy.mean(data, axis=1):\n", " print(row_mean)\n", "```\n", "\n", "`sys.argv[0]` is always the name of the script,\n", "and `sys.argv[1]` is the name of the file to process.\n", "Let's test it:\n", "\n", "```python\n", "!python readings.py\n", "```\n", "\n", "#### readings_02.py\n", "\n", "Now we've defined a function,\n", "but we need to actually call it in the script:\n", "\n", "```python\n", "import sys\n", "import numpy\n", "\n", "def main():\n", " script = sys.argv[0]\n", " filename = sys.argv[1]\n", " data = numpy.loadtxt(filename, delimiter=',')\n", " for row_mean in numpy.mean(data, axis=1):\n", " print(row_mean)\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "\n", "The extra lines we've added at the bottom allow us to run this script as a stand-alone program (e.g., as `readings.py`),\n", "rather than importing as a module (e.g., `readings()`).\n", "\n", "\n", "This general structure is the conventional way of writing command line programs:\n", "\n", "```python\n", "def main():\n", " # code goes here\n", "\n", "\n", "if __name__ == \"__main__\":\n", " main()\n", "```\n", "\n", "We're now ready to run it again:\n", "\n", "```python\n", "!python readings.py data/small-01.csv\n", "```\n", "\n", "We're using one of our small data files,\n", "since it makes it easier to inspect the output while we're developing code.\n", "\n", "#### readings_03.py\n", "\n", "Our next step will be to include a for loop to run across multiple data files:\n", "\n", "```python\n", "import sys\n", "import numpy\n", "\n", "def main():\n", " script = sys.argv[0]\n", " for filename in sys.argv[1:]:\n", " data = numpy.loadtxt(filename, delimiter=',')\n", " for row_mean in numpy.mean(data, axis=1):\n", " print(row_mean)\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "\n", "We can now test it on two files:\n", "\n", "```python\n", "!python readings.py data/small-01.csv data/small-02.csv\n", "```\n", "\n", "#### readings_04.py\n", "\n", "In our next step,\n", "we'll include multiple options for summary statistics:\n", "\n", "```python\n", "import sys\n", "import numpy\n", "\n", "def main():\n", " script = sys.argv[0]\n", " action = sys.argv[1]\n", " filenames = sys.argv[2:]\n", "\n", " for filename in filenames:\n", " data = numpy.loadtxt(filename, delimiter=',')\n", "\n", " if action == '--min':\n", " values = numpy.min(data, axis=1)\n", " elif action == '--mean':\n", " values = numpy.mean(data, axis=1)\n", " elif action == '--max':\n", " values = numpy.max(data, axis=1)\n", "\n", " for val in values:\n", " print(val)\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "\n", "Testing:\n", "```python\n", "!python readings.py --max data/small-01.csv\n", "```\n", "\n", "While this works, \n", "there are a few problems:\n", "- `main` is getting large\n", "- if we only specify a file name, we can end up with a silent failure\n", "- we need to ensure the submitted flag is one of the accepted options\n", "\n", "#### readings_05.py\n", "\n", "This improved structure resolves the three issues above:\n", "\n", "```python\n", "import sys\n", "import numpy\n", "\n", "def main():\n", " script = sys.argv[0]\n", " action = sys.argv[1]\n", " filenames = sys.argv[2:]\n", " assert action in ['--min', '--mean', '--max'], \\\n", " 'Action is not one of --min, --mean, or --max: ' + action\n", " for filename in filenames:\n", " process(filename, action)\n", "\n", "def process(filename, action):\n", " data = numpy.loadtxt(filename, delimiter=',')\n", "\n", " if action == '--min':\n", " values = numpy.min(data, axis=1)\n", " elif action == '--mean':\n", " values = numpy.mean(data, axis=1)\n", " elif action == '--max':\n", " values = numpy.max(data, axis=1)\n", "\n", " for val in values:\n", " print(val)\n", "\n", "if __name__ == '__main__':\n", " main()\n", "```\n", "\n", "You should try this function with a few inputs to see how it works under different circumstances.\n", "\n", "#### readings_06.py\n", "\n", "To add the option to accept files via standard in (stdin):\n", "\n", "```python\n", "import sys\n", "import numpy\n", "\n", "def main():\n", " script = sys.argv[0]\n", " action = sys.argv[1]\n", " filenames = sys.argv[2:]\n", " assert action in ['--min', '--mean', '--max'], \\\n", " 'Action is not one of --min, --mean, or --max: ' + action\n", " if len(filenames) == 0:\n", " process(sys.stdin, action)\n", " else:\n", " for filename in filenames:\n", " process(filename, action)\n", "\n", "def process(filename, action):\n", " data = numpy.loadtxt(filename, delimiter=',')\n", "\n", " if action == '--min':\n", " values = numpy.min(data, axis=1)\n", " elif action == '--mean':\n", " values = numpy.mean(data, axis=1)\n", " elif action == '--max':\n", " values = numpy.max(data, axis=1)\n", "\n", " for val in values:\n", " print(val)\n", "```\n", "\n", "To try this, we need to use a different format on the command line:\n", "\n", "```python\n", "!python readings.py --mean < data/small-01.csv\n", "```\n", "\n", ">#### Challenge-readings_07.py\n", "Rewrite `readings.py` so that it uses short flags (-n, -m, and -x) instead of --min, --mean, and --max, respectively,\n", "so you'll be able to use it as `!python readings.py -x data/small-01.csv`.\n", "Is the code easier to read? \n", "Is the program easier to understand?\n", "\n", ">#### readings_08.py\n", "Finally, if we wanted to include a help message if no file is input:\n", "\n", "```python\n", "import sys\n", "import numpy\n", "\n", "def main():\n", " script = sys.argv[0]\n", " if len(sys.argv) == 1: # no arguments, so print help message\n", " print('''Usage: python readings_08.py action filenames\n", " action must be one of --min --mean --max\n", " if filenames is blank, input is taken from stdin;\n", " otherwise, each filename in the list of arguments \n", " is processed in turn''')\n", " return\n", "\n", " action = sys.argv[1]\n", " filenames = sys.argv[2:]\n", " assert action in ['--min', '--mean', '--max'], \\\n", " 'Action is not one of --min, --mean, or --max: ' + action\n", " if len(filenames) == 0:\n", " process(sys.stdin, action)\n", " else:\n", " for filename in filenames:\n", " process(filename, action)\n", "\n", "def process(filename, action):\n", " data = numpy.loadtxt(filename, delimiter=',')\n", "\n", " if action == '--min':\n", " values = numpy.min(data, axis=1)\n", " elif action == '--mean':\n", " values = numpy.mean(data, axis=1)\n", " elif action == '--max':\n", " values = numpy.max(data, axis=1)\n", "\n", " for val in values:\n", " print(val)\n", "\n", "main()\n", "```\n", "\n", "Which we can run:\n", "\n", "```python\n", "!python readings.py -m\n", "```\n", "\n", "#### readings_09.py\n", "\n", "We can set mean as the default action if none is specified:\n", "\n", "```python\n", "import sys\n", "import numpy\n", "\n", "def main():\n", " script = sys.argv[0]\n", " action = sys.argv[1]\n", " if action not in ['--min', '--mean', '--max']: # if no action given\n", " action = '--mean' # set a default action, that being mean\n", " filenames = sys.argv[1:] # start the filenames one place earlier in the argv list\n", " else:\n", " filenames = sys.argv[2:]\n", "\n", " if len(filenames) == 0:\n", " process(sys.stdin, action)\n", " else:\n", " for filename in filenames:\n", " process(filename, action)\n", "\n", "def process(filename, action):\n", " data = numpy.loadtxt(filename, delimiter=',')\n", "\n", " if action == '--min':\n", " values = numpy.min(data, axis=1)\n", " elif action == '--mean':\n", " values = numpy.mean(data, axis=1)\n", " elif action == '--max':\n", " values = numpy.max(data, axis=1)\n", "\n", " for val in values:\n", " print(val)\n", "\n", "main()\n", "```\n", "\n", "To try it:\n", "\n", "```python\n", "!python readings.py data/small-01.csv\n", "```\n", "\n", ">#### Challenge-check_arguments\n", "Write a program called `check_arguments.py` that prints usage then exits the program if no arguments are provided. \n", "(Hint: You can use sys.exit() to exit the program.)\n", "\n", "Example usage:\n", "\n", "```python\n", "!python check_arguments.py \n", "```\n", "\n", "Output:\n", "\n", "```\n", "usage: python check_argument.py filename.txt\n", "```\n", "\n", "Another usage:\n", "\n", "```python\n", "!python check_arguments.py filename.txt\n", "```\n", "\n", "Output\n", "\n", "```\n", "Thanks for specifying arguments!\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Wrapping up\n", "\n", "In this class, \n", "we discussed the process of debugging,\n", "test-driven development,\n", "and command line programs.\n", "This rounds out our course content covering basic Python programming structures and how to apply them in creating robust, reusable code.\n", "\n", "Please view the links on your class website for more information about expanding your Python skills." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" } }, "nbformat": 4, "nbformat_minor": 2 }