{
"metadata": {
"name": ""
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook was put together by [Jake Vanderplas](http://www.vanderplas.com) for UW's [Astro 599](http://www.astro.washington.edu/users/vanderplas/Astr599/) course. Source and license info is on [GitHub](https://github.com/jakevdp/2013_fall_ASTR599/)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Documentation, Testing, and Packaging\n",
"\n",
"Today's session is going to be a grab-bag of information that will help you write **clean, usable, and maintainable code**. Working with Python in science is much more than simply the mechanics of writing code: it's about writing code in a way that allows it to be used, re-used, adapted, and understood by collaborators and your future self.\n",
"\n",
"We'll be talking about a few key components of this: **documentation**, **testing**, and **packaging**. Again, this will be a quick summary, but I'll try to give some useful examples and resources to learn more."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Documentation\n",
"\n",
"We've seen how useful code documentation is within the IPython notebook. For example, if you want to know about the various arguments in a numpy function, you can use the ``?`` mark within IPython"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import numpy as np\n",
"np.linalg.svd?"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 1
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can print out the documentation string using the ``help()`` function:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"help(np.linalg.svd)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Help on function svd in module numpy.linalg.linalg:\n",
"\n",
"svd(a, full_matrices=1, compute_uv=1)\n",
" Singular Value Decomposition.\n",
" \n",
" Factors the matrix `a` as ``u * np.diag(s) * v``, where `u` and `v`\n",
" are unitary and `s` is a 1-d array of `a`'s singular values.\n",
" \n",
" Parameters\n",
" ----------\n",
" a : array_like\n",
" A real or complex matrix of shape (`M`, `N`) .\n",
" full_matrices : bool, optional\n",
" If True (default), `u` and `v` have the shapes (`M`, `M`) and\n",
" (`N`, `N`), respectively. Otherwise, the shapes are (`M`, `K`)\n",
" and (`K`, `N`), respectively, where `K` = min(`M`, `N`).\n",
" compute_uv : bool, optional\n",
" Whether or not to compute `u` and `v` in addition to `s`. True\n",
" by default.\n",
" \n",
" Returns\n",
" -------\n",
" u : ndarray\n",
" Unitary matrix. The shape of `u` is (`M`, `M`) or (`M`, `K`)\n",
" depending on value of ``full_matrices``.\n",
" s : ndarray\n",
" The singular values, sorted so that ``s[i] >= s[i+1]``. `s` is\n",
" a 1-d array of length min(`M`, `N`).\n",
" v : ndarray\n",
" Unitary matrix of shape (`N`, `N`) or (`K`, `N`), depending on\n",
" ``full_matrices``.\n",
" \n",
" Raises\n",
" ------\n",
" LinAlgError\n",
" If SVD computation does not converge.\n",
" \n",
" Notes\n",
" -----\n",
" The SVD is commonly written as ``a = U S V.H``. The `v` returned\n",
" by this function is ``V.H`` and ``u = U``.\n",
" \n",
" If ``U`` is a unitary matrix, it means that it\n",
" satisfies ``U.H = inv(U)``.\n",
" \n",
" The rows of `v` are the eigenvectors of ``a.H a``. The columns\n",
" of `u` are the eigenvectors of ``a a.H``. For row ``i`` in\n",
" `v` and column ``i`` in `u`, the corresponding eigenvalue is\n",
" ``s[i]**2``.\n",
" \n",
" If `a` is a `matrix` object (as opposed to an `ndarray`), then so\n",
" are all the return values.\n",
" \n",
" Examples\n",
" --------\n",
" >>> a = np.random.randn(9, 6) + 1j*np.random.randn(9, 6)\n",
" \n",
" Reconstruction based on full SVD:\n",
" \n",
" >>> U, s, V = np.linalg.svd(a, full_matrices=True)\n",
" >>> U.shape, V.shape, s.shape\n",
" ((9, 6), (6, 6), (6,))\n",
" >>> S = np.zeros((9, 6), dtype=complex)\n",
" >>> S[:6, :6] = np.diag(s)\n",
" >>> np.allclose(a, np.dot(U, np.dot(S, V)))\n",
" True\n",
" \n",
" Reconstruction based on reduced SVD:\n",
" \n",
" >>> U, s, V = np.linalg.svd(a, full_matrices=False)\n",
" >>> U.shape, V.shape, s.shape\n",
" ((9, 6), (6, 6), (6,))\n",
" >>> S = np.diag(s)\n",
" >>> np.allclose(a, np.dot(U, np.dot(S, V)))\n",
" True\n",
"\n"
]
}
],
"prompt_number": 2
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The idea with documentation is to give a broad description of what the algorithm does, as well as a detailed description of arguments and return values.\n",
"\n",
"### Documenting Functions\n",
"\n",
"The way you do this with your own functions is by using a **multi-line string literal** at the beginning of the function. For example, let's write a documentation string for the fibonacci function we looked at last week.\n",
"\n",
"First, check what happens if we don't document the function:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def fib(N):\n",
" x = np.zeros(N, dtype=float)\n",
" for i in range(N):\n",
" if i == 0:\n",
" x[i] = 0\n",
" elif i == 1:\n",
" x[i] = 1\n",
" else:\n",
" x[i] = x[i - 1] + x[i - 2]\n",
" return x"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 3
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"fib?"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 4
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"help(fib)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Help on function fib in module __main__:\n",
"\n",
"fib(N)\n",
"\n"
]
}
],
"prompt_number": 5
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Not too helpful. Now let's re-define the function with a doc string:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def fib(N):\n",
" \"\"\"\n",
" Compute the first N Fibonacci numbers\n",
" \n",
" Parameters\n",
" ----------\n",
" N : integer\n",
" The number of Fibonacci numbers to compute\n",
" \n",
" Returns\n",
" -------\n",
" x : np.ndarray\n",
" the length-N array containing the first N\n",
" Fibonacci numbers.\n",
" \n",
" Notes\n",
" -----\n",
" This is a pure Python implementation. For large N,\n",
" consider a Cython implementation\n",
" \n",
" Examples\n",
" --------\n",
" >>> fib(5)\n",
" array([ 0., 1., 1., 2., 3.])\n",
" \"\"\"\n",
" x = np.zeros(N, dtype=float)\n",
" for i in range(N):\n",
" if i == 0:\n",
" x[i] = 0\n",
" elif i == 1:\n",
" x[i] = 1\n",
" else:\n",
" x[i] = x[i - 1] + x[i - 2]\n",
" return x"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 6
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's take a look at the help string for the function:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"fib?"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 7
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"help(fib)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Help on function fib in module __main__:\n",
"\n",
"fib(N)\n",
" Compute the first N Fibonacci numbers\n",
" \n",
" Parameters\n",
" ----------\n",
" N : integer\n",
" The number of Fibonacci numbers to compute\n",
" \n",
" Returns\n",
" -------\n",
" x : np.ndarray\n",
" the length-N array containing the first N\n",
" Fibonacci numbers.\n",
" \n",
" Notes\n",
" -----\n",
" This is a pure Python implementation. For large N,\n",
" consider a Cython implementation\n",
" \n",
" Examples\n",
" --------\n",
" >>> fib(5)\n",
" array([ 0., 1., 1., 2., 3.])\n",
"\n"
]
}
],
"prompt_number": 8
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You should get in the habit of **always documenting your functions and classes** in this way. It will make sharing your code (and using your code yourself!) much, much easier in the long run.\n",
"\n",
"There is no requirement for how your code should be documented, but a standard in the scientific Python community is to follow the [NumPy doc-string standard](https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt#docstring-standard). This outlines many potential sections, but the basic example should look something like this:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def example_function():\n",
" \"\"\"Short description of the function.\n",
" \n",
" Longer multi-line description of the function,\n",
" with a bit more information on the function's intent.\n",
" \n",
" Parameters\n",
" ----------\n",
" varable_name : \n",
" short description (4-space indent)\n",
" another_variable : \n",
" short description, potentially with default value\n",
" \n",
" Returns\n",
" -------\n",
" return_value : \n",
" description\n",
" another_return_value: \n",
" more description\n",
" \n",
" Notes\n",
" -----\n",
" Extended discussion of the details of the algorithm\n",
" \n",
" References\n",
" ----------\n",
" Any website or paper with information about the algorithm\n",
" or implementation\n",
" \n",
" Examples\n",
" --------\n",
" Clarifying examples of how to use the function. Use the\n",
" Python interpreter syntax (starting with >>>) and show the\n",
" output:\n",
" \n",
" >>> print \"hello world\"\n",
" hello world\n",
" \"\"\"\n",
" return None"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 9
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This emphasis on formatting may seem a bit extreme, but it's nice for two reasons:\n",
"\n",
"1. Uniformity of documentation strings makes it much easier to quickly explore modules and learn what they do.\n",
"2. The [Sphinx](http://sphinx-doc.org/) documentation generation framework can parse documentation written in this form and generate nice web pages.\n",
"\n",
"We saw the documentation for ``np.linalg.svd`` above. By running this through Sphinx, the following documentation is generated. This can be **very** handy if you write code that you end up sharing with others."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from IPython.display import IFrame\n",
"IFrame(('http://docs.scipy.org/doc/numpy/'\n",
" 'reference/generated/numpy.linalg.svd.html'),\n",
" width=600, height=400)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"html": [
"\n",
" \n",
" "
],
"metadata": {},
"output_type": "pyout",
"prompt_number": 10,
"text": [
""
]
}
],
"prompt_number": 10
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Documenting classes\n",
"\n",
"Documentation of classes is very similar, with one exception: **Don't write documentation under the ``__init__`` method**. Instead, write initialization information in the class-level doc string."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class Multiplier(object):\n",
" \"\"\"A simple class implementing multiplication\n",
" \n",
" Parameters\n",
" ----------\n",
" x : float\n",
" an integer value\n",
" \"\"\"\n",
" def __init__(self, x):\n",
" self.x = x\n",
" \n",
" def get_multiple(self, n):\n",
" \"\"\"Return n * x\n",
" \n",
" Parameters\n",
" ----------\n",
" n : float\n",
" multiplier\n",
" \n",
" Returns\n",
" -------\n",
" y : float\n",
" y = n * x\n",
" \"\"\"\n",
" return n * self.x"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 11
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can look at the documentation. Notice that a list of methods is automatically included."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"Multiplier?"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 12
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"help(Multiplier)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Help on class Multiplier in module __main__:\n",
"\n",
"class Multiplier(__builtin__.object)\n",
" | A simple class implementing multiplication\n",
" | \n",
" | Parameters\n",
" | ----------\n",
" | x : float\n",
" | an integer value\n",
" | \n",
" | Methods defined here:\n",
" | \n",
" | __init__(self, x)\n",
" | \n",
" | get_multiple(self, n)\n",
" | Return n * x\n",
" | \n",
" | Parameters\n",
" | ----------\n",
" | n : float\n",
" | multiplier\n",
" | \n",
" | Returns\n",
" | -------\n",
" | y : float\n",
" | y = n * x\n",
" | \n",
" | ----------------------------------------------------------------------\n",
" | Data descriptors defined here:\n",
" | \n",
" | __dict__\n",
" | dictionary for instance variables (if defined)\n",
" | \n",
" | __weakref__\n",
" | list of weak references to the object (if defined)\n",
"\n"
]
}
],
"prompt_number": 13
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"Multiplier.get_multiple?"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 14
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"help(Multiplier.get_multiple)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Help on method get_multiple in module __main__:\n",
"\n",
"get_multiple(self, n) unbound __main__.Multiplier method\n",
" Return n * x\n",
" \n",
" Parameters\n",
" ----------\n",
" n : float\n",
" multiplier\n",
" \n",
" Returns\n",
" -------\n",
" y : float\n",
" y = n * x\n",
"\n"
]
}
],
"prompt_number": 15
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Documenting Modules\n",
"\n",
"It is also helpful to document your modules, by placing a multi-line string literal at the beginning of the file. It might look something like this -- we'll use the ``%%file`` magic to save this to a file"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%%file mymodule.py\n",
"\"\"\"\n",
"Simple module to show module-level documentation.\n",
"\n",
"You can write some extra information here.\n",
"It can be as long as you'd like...\n",
"\n",
"Functions:\n",
"\n",
"- product : multiply two numbers\n",
"- quotient : divide two numbers\n",
"\"\"\"\n",
"\n",
"def product(a, b):\n",
" \"\"\"return the product of a and b\"\"\"\n",
" return a * b\n",
"\n",
"def quotient(a, b):\n",
" \"\"\"retur the quotient of a and b\"\"\"\n",
" return a / b"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Overwriting mymodule.py\n"
]
}
],
"prompt_number": 16
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import mymodule\n",
"mymodule?"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 17
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"help(mymodule)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Help on module mymodule:\n",
"\n",
"NAME\n",
" mymodule - Simple module to show module-level documentation.\n",
"\n",
"FILE\n",
" /Users/jakevdp/Opensource/2013_fall_ASTR599/notebooks/mymodule.py\n",
"\n",
"DESCRIPTION\n",
" You can write some extra information here.\n",
" It can be as long as you'd like...\n",
" \n",
" Functions:\n",
" \n",
" - product : multiply two numbers\n",
" - quotient : divide two numbers\n",
"\n",
"FUNCTIONS\n",
" product(a, b)\n",
" return the product of a and b\n",
" \n",
" quotient(a, b)\n",
" retur the quotient of a and b\n",
"\n",
"\n"
]
}
],
"prompt_number": 18
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Testing\n",
"\n",
"One other important piece of good coding practice is to write tests, particularly **Unit Tests**. Unit tests are tests of the individual pieces of a computational process, and can be very helpful for both giving yourself peace-of-mind that your code is behaving properly, and making sure your code still works as expected if and when you start optimizing the slower pieces.\n",
"\n",
"### Test functions\n",
"\n",
"Let's write a small ``fib`` module which contains test code. We'll copy and paste the function definition from above, and then add a test function"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%%file fibmodule.py\n",
"\"\"\"\n",
"Functions to compute Fibonacci sequences\n",
"\"\"\"\n",
"import numpy as np\n",
"from numpy.testing import assert_allclose\n",
"\n",
"def fib(N):\n",
" \"\"\"\n",
" Compute the first N Fibonacci numbers\n",
" \n",
" Parameters\n",
" ----------\n",
" N : integer\n",
" The number of Fibonacci numbers to compute\n",
" \n",
" Returns\n",
" -------\n",
" x : np.ndarray\n",
" the length-N array containing the first N\n",
" Fibonacci numbers.\n",
" \n",
" Notes\n",
" -----\n",
" This is a pure Python implementation. For large N,\n",
" consider a Cython implementation\n",
" \n",
" Examples\n",
" --------\n",
" >>> fib(5)\n",
" array([ 0., 1., 1., 2., 3.])\n",
" \"\"\"\n",
" x = np.zeros(N, dtype=float)\n",
" for i in range(N):\n",
" if i == 0:\n",
" x[i] = 0\n",
" elif i == 1:\n",
" x[i] = 1\n",
" else:\n",
" x[i] = x[i - 1] + x[i - 2]\n",
" return x\n",
"\n",
"def test_first_ten():\n",
" nums = fib(10)\n",
" assert_allclose(fib(10),\n",
" [0, 1, 1, 2, 3, 5, 8, 13, 21, 34])"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Overwriting fibmodule.py\n"
]
}
],
"prompt_number": 19
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now you can import and run the test to make sure it passes:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import fibmodule\n",
"fibmodule.test_first_ten()"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 20
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This approach works, but if you have many many tests (which you generally will for larger projects), running them by-hand can be tedious.\n",
"\n",
"Fortunately, there are **testing frameworks** which streamline this process. Most of the Scientific Python stack uses the [Nose](http://nose.readthedocs.org/en/latest/) testing framework to do this.\n",
"\n",
"First, make sure you have nose installed by running\n",
"\n",
"```\n",
"[~]$ conda install nose\n",
"```\n",
"\n",
"(or, if you're not using Anaconda, ``pip install nose``)\n",
"\n",
"Then from the command-line, run the following (``-v`` stands for \"verbose\"):"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!nosetests -v fibmodule"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"fibmodule.test_first_ten ... ok\r\n",
"\r\n",
"----------------------------------------------------------------------\r\n",
"Ran 1 test in 0.001s\r\n",
"\r\n",
"OK\r\n"
]
}
],
"prompt_number": 21
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Nosetests goes through the module and looks for any function which starts with ``test_``, and runs them, checking for errors.\n",
"\n",
"There are many options for writing and organizing your tests. For more information, look at the [Nose documentation](http://nose.readthedocs.org/en/latest/) or browse the source code of a package like [Scikit-learn](http://github.com/scikit-learn/scikit-learn), which utilizes an extensive nose testing framework (see especially the [sklearn.tests](https://github.com/scikit-learn/scikit-learn/tree/master/sklearn/tests) submodule).\n",
"\n",
"### Doctests\n",
"One thing nose does which is very useful is **doctests**. The Doc-testing framework goes through the documentation strings of the functions, and looks for code snippets which start with ``>>>``, and then runs them to make sure the results are correct.\n",
"\n",
"You can run the doctests on our ``fibmodule`` using the following command:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!nosetests -v --with-doctest fibmodule"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"fibmodule.test_first_ten ... ok\r\n",
"Doctest: fibmodule.fib ... ok\r\n",
"\r\n",
"----------------------------------------------------------------------\r\n",
"Ran 2 tests in 0.004s\r\n",
"\r\n",
"OK\r\n"
]
}
],
"prompt_number": 22
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see that it ran two tests, one of which is the doc test within the ``fib()`` function.\n",
"\n",
"Just to make sure it's doing what it claims, let's write a quick module with a failing test and see what Nose tells us:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%%file fail.py\n",
"\"\"\"A module with a failing test\"\"\"\n",
"\n",
"def test_fail_at_math():\n",
" \"\"\"Here's a test that fails\"\"\"\n",
" assert 1 + 1 == 3\n",
" \n",
"def test_succeed_at_math():\n",
" \"\"\"Here's a test that passes\"\"\"\n",
" assert 1 + 1 == 2"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Overwriting fail.py\n"
]
}
],
"prompt_number": 23
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!nosetests -v fail"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Here's a test that fails ... FAIL\r\n",
"Here's a test that passes ... ok\r\n",
"\r\n",
"======================================================================\r\n",
"FAIL: Here's a test that fails\r\n",
"----------------------------------------------------------------------\r\n",
"Traceback (most recent call last):\r\n",
" File \"/Users/jakevdp/anaconda/lib/python2.7/site-packages/nose/case.py\", line 197, in runTest\r\n",
" self.test(*self.arg)\r\n",
" File \"/Users/jakevdp/Opensource/2013_fall_ASTR599/notebooks/fail.py\", line 5, in test_fail_at_math\r\n",
" assert 1 + 1 == 3\r\n",
"AssertionError\r\n",
"\r\n",
"----------------------------------------------------------------------\r\n",
"Ran 2 tests in 0.002s\r\n",
"\r\n",
"FAILED (failures=1)\r\n"
]
}
],
"prompt_number": 24
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As advertised, it finds the failing test! Notice especially that the output shows the first line of the documentation string of each test function. This can be really helpful when diagnosing test failures."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Packaging: Building and Sharing Python Modules\n",
"Eventually, you will write code which is **so awesome** that you'll want to share it with other people. In order to do this, you'll want to create a well-tested, well-documented module, potentially with a ``setup.py`` script to aid in installing the module.\n",
"\n",
"There are several good ways to approach this, but generally good practice is to do a directory structure which looks something like this:\n",
"\n",
" RootDirectory: this is your git repository,\n",
" | if you're using git\n",
" | mymodule/\n",
" | | __init__.py : this is the file that tells Python\n",
" | | that your module is a module. It\n",
" | | contains code which is executed\n",
" | | when the module is imported\n",
" | |\n",
" | | file1.py : Any number of .py files containing\n",
" | | file2.py : your awesome, well-documented code\n",
" | | ...\n",
" | |\n",
" | | tests/\n",
" | | | __init__.py : lets Python know this is a\n",
" | | | submodule.\n",
" | | |\n",
" | | | test_file1.py : Any number of .py files which\n",
" | | | test_file2.py : contain test scripts for your\n",
" | | | ... functions. Organize them in\n",
" | | | a way that helps you remember\n",
" | | | what they are.\n",
" |\n",
" | README.md : general information about the module\n",
" | LICENSE : a license for code re-use.\n",
" | Consider the BSD license\n",
" | INSTALL : an installation file\n",
" | setup.py : the setup script (see below)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice that every **module** and **submodule** has an ``__init__.py`` file, which may or may not contain any code. The ``__init__.py`` file is executed once when the module or submodule is loaded.\n",
"\n",
"The ``setup.py`` file contains a script that allows you to install your module system-wide using the command\n",
"\n",
"```\n",
"[~]$ python setup.py install\n",
"```\n",
"\n",
"There are **way** too many details to go into here regarding setup scripts and Python's distutils: for a good tutorial, take a look at [Python's distutils documentation](http://docs.python.org/2/distutils/introduction.html#distutils-simple-example). You may also want to look at other packages to see how they use this file (for example, check out my [astroML setup file](https://github.com/astroML/astroML/blob/master/setup.py))."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Homework\n",
"\n",
"Your homework is to create a module containing a ``Cosmology()`` class, which computes some quantities using expressions from the [Hogg 1999 Distance Measures](http://arxiv.org/abs/astroph/9905116) paper. We took a brief look at this problem in the boot camp at the beginning of the quarter ([Lecture 8](http://www.astro.washington.edu/users/vanderplas/Astr599/notebooks/08_ScipyIntro)).\n",
"\n",
"This will be a simple example of **test-driven development**, a development style in which the desired functionality is first written as a series of tests, and then code is written which passes those tests.\n",
"\n",
"You'll be creating a ``cosmology`` package, adding some simple cosmological distance measures, and testing and documenting the code. In your homework git repository, create the following directory structure:\n",
"\n",
" HW8/\n",
" | cosmology/\n",
" | | __init__.py\n",
" | | cosmology.py\n",
" | | tests/\n",
" | | | __init__.py\n",
" | | | test_cosmology.py\n",
" \n",
"The files should contain the following:\n",
"\n",
"#### ``HW8/cosmology/__init__.py``:\n",
"\n",
"``` python\n",
"\"\"\"Cosmology tools\n",
"\n",
"\n",
"\"\"\"\n",
"# We'll import the Cosmology class using\n",
"# a \"relative import\":\n",
"from .cosmology import Cosmology\n",
"```\n",
" \n",
"#### ``HW8/cosmology/tests/__init__.py``\n",
"\n",
"``` python\n",
"\"\"\"Cosmology Tests\"\"\"\n",
"# Nothing else here\n",
"```\n",
" \n",
"#### ``HW8/cosmology/Cosmology.py``\n",
"(see below -- you'll be modifying this file)\n",
" \n",
"#### ``HW8/cosmology/tests/test_cosmology.py``\n",
"(see below)\n",
" \n",
"\n",
"Your task is to fill-in the class and method definitions in ``cosmology.py`` using the formulae from the Hogg paper. Once this is all set up, you should be able to run IPython from within the ``HW8`` directory and type, e.g.\n",
"\n",
"```\n",
"In [1]: from cosmology import Cosmology\n",
"\n",
"In [2]: cosmo = Cosmology()\n",
"\n",
"In [3]: cosmo.DC(0)\n",
"Out[3]: 0.0\n",
"```\n",
"\n",
"From the top directory (``HW8``) run ``nosetests -v cosmology`` to test your implementation, and make sure all the tests pass. Be sure to add suitable documentation strings to the class as well, including a doctests or two for each method (make sure that these also pass, using ``nosetests --with-doctest``)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Files for your Implementation\n",
"Below are the ``cosmology.py`` and ``test_cosmology.py`` files which you will use in your directory structure above. Execute these cells to create the files locally, and then move them into the appropriate place within the directory structure."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%%file cosmology.py\n",
"import numpy as np\n",
"from scipy import integrate\n",
"\n",
"# speed of light\n",
"C = 299792.458 # km/s\n",
"\n",
"\n",
"class Cosmology(object):\n",
" \"\"\"Cosmology class implementing Cosmological Distance Functions\n",
"\n",
" \n",
" \"\"\"\n",
" def __init__(self, OmegaM=0.3, h=0.7):\n",
" # for now, we'll implement only\n",
" # a flat cosmology for simplicity\n",
" self.OmegaK = 0\n",
" self.OmegaM = OmegaM\n",
" self.OmegaL = 1. - OmegaM\n",
"\n",
" # Hubble constant, km/s/Mpc\n",
" self.H0 = h * 100\n",
"\n",
" # Hubble Distance, Mpc\n",
" self.DH = C / self.H0\n",
"\n",
" def _Einv(self, z):\n",
" # implement the inverse of Eqn 14. This is the function that will be\n",
" # integrated in order to compute other quantities\n",
" pass\n",
"\n",
" def DC(self, z):\n",
" \"\"\"Comoving Distance (Mpc)\n",
"\n",
" \n",
" \"\"\"\n",
" # Compute the comoving distance in Mpc using scipy.integrate.quad\n",
" # following Eqn 15\n",
"\n",
" def DM(self, z):\n",
" \"\"\"Transverse Comoving Distance (Mpc)\n",
"\n",
" \n",
" \"\"\"\n",
" # Compute the transverse comoving distance in Mpc (Eqn 16)\n",
"\n",
" def DA(self, z):\n",
" \"\"\"Angular Diameter Distance (Mpc)\n",
"\n",
" \n",
" \"\"\"\n",
" # Compute the Angular Diameter distance in Mpc (Eqn 18)\n",
"\n",
" def DL(self, z):\n",
" \"\"\"Luminosity Distance (Mpc)\n",
"\n",
" \n",
" \"\"\"\n",
" # Compute the Luminosity Distance in Mpc (Eqn 21)\n",
"\n",
" def mu(self, z):\n",
" \"\"\"Distance Modulus (magnitudes)\n",
"\n",
" \n",
" \"\"\"\n",
" # Compute the distance modulus (Eqn 25)\n"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%%file test_cosmology.py\n",
"\"\"\"Tests of the Cosmology class\"\"\"\n",
"from numpy.testing import assert_allclose\n",
"from .. import Cosmology\n",
"\n",
"# [OmegaM, h, z, DC]\n",
"DC_RESULTS =\\\n",
"[[0.2, 0.6, 0, 0.0],\n",
" [0.2, 0.6, 0.5, 2285.61],\n",
" [0.2, 0.6, 1.0, 4114.6],\n",
" [0.2, 0.7, 0, 0.0],\n",
" [0.2, 0.7, 0.5, 1959.09],\n",
" [0.2, 0.7, 1.0, 3526.8],\n",
" [0.3, 0.6, 0, 0.0],\n",
" [0.3, 0.6, 0.5, 2203.4],\n",
" [0.3, 0.6, 1.0, 3854.47],\n",
" [0.3, 0.7, 0, 0.0],\n",
" [0.3, 0.7, 0.5, 1888.63],\n",
" [0.3, 0.7, 1.0, 3303.83],\n",
" [0.4, 0.6, 0, 0.0],\n",
" [0.4, 0.6, 0.5, 2131.92],\n",
" [0.4, 0.6, 1.0, 3649.21],\n",
" [0.4, 0.7, 0, 0.0],\n",
" [0.4, 0.7, 0.5, 1827.36],\n",
" [0.4, 0.7, 1.0, 3127.89]]\n",
"\n",
"# [OmegaM, h, z, DM]\n",
"DM_RESULTS =\\\n",
"[[0.2, 0.6, 0, 0.0],\n",
" [0.2, 0.6, 0.5, 2285.61],\n",
" [0.2, 0.6, 1.0, 4114.6],\n",
" [0.2, 0.7, 0, 0.0],\n",
" [0.2, 0.7, 0.5, 1959.09],\n",
" [0.2, 0.7, 1.0, 3526.8],\n",
" [0.3, 0.6, 0, 0.0],\n",
" [0.3, 0.6, 0.5, 2203.4],\n",
" [0.3, 0.6, 1.0, 3854.47],\n",
" [0.3, 0.7, 0, 0.0],\n",
" [0.3, 0.7, 0.5, 1888.63],\n",
" [0.3, 0.7, 1.0, 3303.83],\n",
" [0.4, 0.6, 0, 0.0],\n",
" [0.4, 0.6, 0.5, 2131.92],\n",
" [0.4, 0.6, 1.0, 3649.21],\n",
" [0.4, 0.7, 0, 0.0],\n",
" [0.4, 0.7, 0.5, 1827.36],\n",
" [0.4, 0.7, 1.0, 3127.89]]\n",
"\n",
"# [OmegaM, h, z, DA]\n",
"DA_RESULTS =\\\n",
"[[0.2, 0.6, 0, 0.0],\n",
" [0.2, 0.6, 0.5, 1523.74],\n",
" [0.2, 0.6, 1.0, 2057.3],\n",
" [0.2, 0.7, 0, 0.0],\n",
" [0.2, 0.7, 0.5, 1306.06],\n",
" [0.2, 0.7, 1.0, 1763.4],\n",
" [0.3, 0.6, 0, 0.0],\n",
" [0.3, 0.6, 0.5, 1468.93],\n",
" [0.3, 0.6, 1.0, 1927.23],\n",
" [0.3, 0.7, 0, 0.0],\n",
" [0.3, 0.7, 0.5, 1259.08],\n",
" [0.3, 0.7, 1.0, 1651.91],\n",
" [0.4, 0.6, 0, 0.0],\n",
" [0.4, 0.6, 0.5, 1421.28],\n",
" [0.4, 0.6, 1.0, 1824.61],\n",
" [0.4, 0.7, 0, 0.0],\n",
" [0.4, 0.7, 0.5, 1218.24],\n",
" [0.4, 0.7, 1.0, 1563.95]]\n",
"\n",
"# [OmegaM, h, z, DL]\n",
"DL_RESULTS =\\\n",
"[[0.2, 0.6, 0, 0.0],\n",
" [0.2, 0.6, 0.5, 3428.41],\n",
" [0.2, 0.6, 1.0, 8229.21],\n",
" [0.2, 0.7, 0, 0.0],\n",
" [0.2, 0.7, 0.5, 2938.64],\n",
" [0.2, 0.7, 1.0, 7053.61],\n",
" [0.3, 0.6, 0, 0.0],\n",
" [0.3, 0.6, 0.5, 3305.09],\n",
" [0.3, 0.6, 1.0, 7708.93],\n",
" [0.3, 0.7, 0, 0.0],\n",
" [0.3, 0.7, 0.5, 2832.94],\n",
" [0.3, 0.7, 1.0, 6607.66],\n",
" [0.4, 0.6, 0, 0.0],\n",
" [0.4, 0.6, 0.5, 3197.88],\n",
" [0.4, 0.6, 1.0, 7298.42],\n",
" [0.4, 0.7, 0, 0.0],\n",
" [0.4, 0.7, 0.5, 2741.04],\n",
" [0.4, 0.7, 1.0, 6255.79]]\n",
"\n",
"# [OmegaM, h, z, mu]\n",
"MU_RESULTS = \\\n",
"[[0.2, 0.6, 0.1, 38.666337868889762],\n",
" [0.2, 0.6, 0.5, 42.675462188569789],\n",
" [0.2, 0.6, 1.0, 44.57678993423005],\n",
" [0.2, 0.7, 0.1, 38.331603920736697],\n",
" [0.2, 0.7, 0.5, 42.340728240416723],\n",
" [0.2, 0.7, 1.0, 44.242055986076984],\n",
" [0.3, 0.6, 0.1, 38.649938522544012],\n",
" [0.3, 0.6, 0.5, 42.595919369693959],\n",
" [0.3, 0.6, 1.0, 44.434971603696795],\n",
" [0.3, 0.7, 0.1, 38.31520457439094],\n",
" [0.3, 0.7, 0.5, 42.261185421540894],\n",
" [0.3, 0.7, 1.0, 44.100237655543722],\n",
" [0.4, 0.6, 0.1, 38.633904578021593],\n",
" [0.4, 0.6, 0.5, 42.524308199311712],\n",
" [0.4, 0.6, 1.0, 44.316144393006496],\n",
" [0.4, 0.7, 0.1, 38.299170629868527],\n",
" [0.4, 0.7, 0.5, 42.189574251158646],\n",
" [0.4, 0.7, 1.0, 43.981410444853431]]\n",
"\n",
"\n",
"def assert_equal_to_2_decimals(x, y):\n",
" assert_allclose(x, y, atol=0.01)\n",
"\n",
"\n",
"def test_DC():\n",
" \"\"\"Test Comoving Distance\"\"\"\n",
" for (OmegaM, h, z, DC) in DC_RESULTS:\n",
" cosmo = Cosmology(OmegaM=OmegaM, h=h)\n",
" yield assert_equal_to_2_decimals, DC, cosmo.DC(z)\n",
"\n",
"\n",
"def test_DM():\n",
" \"\"\"Test Transverse Comoving Distance\"\"\"\n",
" for (OmegaM, h, z, DM) in DM_RESULTS:\n",
" cosmo = Cosmology(OmegaM=OmegaM, h=h)\n",
" yield assert_equal_to_2_decimals, DM, cosmo.DM(z)\n",
"\n",
"\n",
"def test_DA():\n",
" \"\"\"Test Angular Diameter Distance\"\"\"\n",
" for (OmegaM, h, z, DA) in DA_RESULTS:\n",
" cosmo = Cosmology(OmegaM=OmegaM, h=h)\n",
" yield assert_equal_to_2_decimals, DA, cosmo.DA(z)\n",
"\n",
"\n",
"def test_DL():\n",
" \"\"\"Test Luminosity Distance\"\"\"\n",
" for (OmegaM, h, z, DL) in DL_RESULTS:\n",
" cosmo = Cosmology(OmegaM=OmegaM, h=h)\n",
" yield assert_equal_to_2_decimals, DL, cosmo.DL(z)\n",
"\n",
"\n",
"def test_mu():\n",
" \"\"\"Test Distance Modulus\"\"\"\n",
" for (OmegaM, h, z, MU) in MU_RESULTS:\n",
" cosmo = Cosmology(OmegaM=OmegaM, h=h)\n",
" yield assert_equal_to_2_decimals, MU, cosmo.mu(z)"
],
"language": "python",
"metadata": {},
"outputs": []
}
],
"metadata": {}
}
]
}