{ "cells": [ { "cell_type": "markdown", "metadata": { "toc": "true" }, "source": [ "# Table of Contents\n", "

1  \"Living in a noisy world...\", using James Powell's (dutc) rwatch module
1.1  Requirements and links
1.2  Defining a debugging context manager, just to try
1.2.1  Watching just one object
1.2.2  Can we delete the rwatch ?
1.2.3  More useful debuggin information
1.2.4  Watching any object
1.2.5  A first context manager to have debugging for one object
1.2.6  A second context manager to debug any object
1.3  Defining a context manager to add white noise
1.3.1  Capturing any numerical value
1.3.2  Adding a white noise for numbers
1.3.3  WhiteNoiseComplex context manager
1.4  Defining a generic noisy context manager
1.5  Conclusion
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# *\"Living in a noisy world...\"*, using James Powell's (`dutc`) `rwatch` module\n", "\n", "Goal : I want to write a [context manager](https://docs.python.org/3.5/library/stdtypes.html#typecontextmanager) for [Python 3.5+](https://www.Python.org/), so that inside the context manager, every number is seen noisy (with a white Gaussian noise, for instance).\n", "\n", "> It will be like being drunk, except that your it will be my Python interpretor and not me !\n", "\n", "For instance, I will like to have this feature:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", ">>> x = 120193\n", ">>> print(x)\n", "120193\n", ">>> np.random.seed(1234)\n", ">>> with WhiteNoise():\n", ">>> print(x)\n", "120193.47143516373249306\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "## Requirements and links" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, we will need [numpy](https://docs.scipy.org/doc/numpy/user/whatisnumpy.html) to have some random number generator, as `numpy.random.normal` to have some 1D Gaussian noise." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import numpy as np" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "0.47143516373249306" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "np.random.seed(1234)\n", "np.random.normal()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, the core part will be to install and import [James Powell (`dutc`)](https://github.com/dutc/) [`rwatch`](https://github.com/dutc/rwatch) module.\n", "If you don't have it installed :\n", "\n", "1. Be sure to have CPython 3.5. `rwatch` patches the CPython eval loop, so it's fixed to specific versions of Python & the author lazy about keeping it updated.\n", "2. Then `pip install dutc-rwatch`. It should work, but it fails for me.\n", "3. (alternative) You can just `cd /tmp/ && git clone https://github.com/dutc/rwatch && cd rwatch/src/ && make` and copy the `rwatch.so` dynamic library wherever you need..." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "gcc `python3-config --cflags` `python3-config --includes` -fPIC -DPy_BUILD_CORE -c -o ceval.o ceval.c\n", "gcc `python3-config --cflags` `python3-config --includes` -L`python3-config --prefix`/lib -Wl,--export-dynamic -fPIC -shared -o rwatch.so rwatch.c hook.c ceval.o -ldl `python3-config --libs`\n", "-rwxrwxr-x 1 lilian lilian 453K mai 14 01:20 ./rwatch.so\n", "./rwatch.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=f41ef744646e6247119bd702039725fc3a1c8e66, not stripped\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Clonage dans 'rwatch'...\n", "In file included from ceval.c:300:0:\n", "ceval_gil.h:114:46: warning: initialization makes integer from pointer without a cast [-Wint-conversion]\n", " static _Py_atomic_address gil_last_holder = {NULL};\n", " ^~~~\n", "ceval_gil.h:114:46: note: (near initialization for ‘gil_last_holder._value’)\n", "ceval_gil.h: In function ‘create_gil’:\n", "ceval_gil.h:145:5: warning: initialization makes integer from pointer without a cast [-Wint-conversion]\n", " _Py_atomic_store_relaxed(&gil_last_holder, NULL);\n", " ^~~~~~~~~~~~~~~~~~~~~~~~\n", "ceval_gil.h: In function ‘drop_gil’:\n", "ceval_gil.h:181:9: warning: initialization makes integer from pointer without a cast [-Wint-conversion]\n", " _Py_atomic_store_relaxed(&gil_last_holder, tstate);\n", " ^~~~~~~~~~~~~~~~~~~~~~~~\n", "ceval_gil.h: In function ‘take_gil’:\n", "ceval_gil.h:243:9: warning: initialization makes integer from pointer without a cast [-Wint-conversion]\n", " _Py_atomic_store_relaxed(&gil_last_holder, tstate);\n", " ^~~~~~~~~~~~~~~~~~~~~~~~\n" ] } ], "source": [ "%%bash\n", "tmpdir=$(mktemp -d)\n", "cd $tmpdir\n", "git clone https://github.com/dutc/rwatch\n", "cd rwatch/src/\n", "make\n", "ls -larth ./rwatch.so\n", "file ./rwatch.so\n", "# cp ./rwatch.so /where/ver/you/need/ # ~/publis/notebook/ for me" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Anyhow, if `rwatch` is installed, we can import it, and it enables two new functions in the `sys` module:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "{}" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import rwatch\n", "from sys import setrwatch, getrwatch\n", "\n", "setrwatch({}) # clean any previously installed rwatch\n", "getrwatch()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, we need the [`collections`](https://docs.python.org/3/library/collections.html) module and its [`defaultdict`](https://docs.python.org/3/library/collections.html#collections.defaultdict) magical datastructure." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from collections import defaultdict" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "## Defining a debugging context manager, just to try\n", "\n", "This is the first example given in [James presentation]() at PyCon Canada 2016.\n", "\n", "We will first define and add a `rwatch` for just one object, let say a variable `x`, and then for any object using `defaultdict`.\n", "From there, writing a context manager that enables this feature only locally is easy." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Watching just one object\n", "1. Write the function, that needs two argument `frame, obj`, and should return `obj`,\n", "2. Install it...\n", "3. Check it!" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def basic_view(frame, obj):\n", " print(\"Python saw the object {} from frame {}\".format(obj, frame))\n", " return obj" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": true }, "outputs": [], "source": [ "x = \"I am alive!\"" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": true }, "outputs": [], "source": [ "setrwatch({\n", " id(x): basic_view\n", "})" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": false, "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Python saw the object I am alive! from frame \n", "Python saw the object I am alive! from frame \n", "I am alive!\n" ] } ], "source": [ "print(x)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That's awesome, it works!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Can we delete the `rwatch` ?\n", "Sure!" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def delrwatch(idobj):\n", " getrwatch().pop(idobj, None)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Python saw the object I am alive! from frame \n", "Python saw the object I am alive! from frame \n", "I am alive!\n", "Python saw the object I am alive! from frame \n", "I am alive!\n", "I am alive!\n" ] } ], "source": [ "print(x)\n", "delrwatch(id(x))\n", "print(x) # no more rwatch on this!\n", "print(x) # no more rwatch on this!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also delete rwatches that are not defined, without a failure:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "I am Zorro !\n", "I am Zorro !\n" ] } ], "source": [ "y = \"I am Zorro !\"\n", "print(y)\n", "delrwatch(y) # No issue!\n", "print(y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### More useful debuggin information\n", "What is this `frame` thing?\n", "It is described in the documentation of the [`inspect`](https://docs.python.org/3/library/inspect.html) module.\n", "\n", "We can actually use it to display some useful information about the object and where was it called etc." ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from inspect import getframeinfo\n", "\n", "def debug_view(frame, obj):\n", " info = getframeinfo(frame)\n", " msg = '- Access to {!r} (@{}) at {}:{}:{}'\n", " print(msg.format(obj, hex(id(obj)), info.filename, info.lineno, info.function))\n", " return obj" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "{140613264650544: }" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "setrwatch({})\n", "setrwatch({\n", " id(x): debug_view\n", "})\n", "getrwatch()" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "- Access to 'I am alive!' (@0x7fe313b0b130) at :1:\n", "- Access to 'I am alive!' (@0x7fe313b0b130) at /usr/local/lib/python3.5/dist-packages/ipykernel/iostream.py:347:write\n", "I am alive!\n" ] } ], "source": [ "print(x)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That can be quite useful!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Watching *any* object\n", "We can actually pass a `defaultdict` to the `setrwatch` function, so that *any* object will have a rwatch!\n", "\n", "> **Warning**: obviously, this will crazily slowdown your interpreter!\n", "\n", "So let be cautious, and only deal with strings here.\n", "\n", "But I want to be safe, so it will only works if the frame indicate that the variable does not come from a file." ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": true }, "outputs": [], "source": [ "setrwatch({})" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def debug_view_for_str(frame, obj):\n", " if isinstance(obj, str):\n", " info = getframeinfo(frame)\n", " if '' in info.filename or ':1:\n", "I am alive!\n" ] } ], "source": [ "print(x)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Clearly, there is a lot of strings involved, mainly because this is a notebook and not the simple Python interpreter, so filtering on the `info.filename` as I did was smart." ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": true }, "outputs": [], "source": [ "setrwatch({})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But obviously, having this for all objects is incredibly verbose!" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def debug_view_for_any_object(frame, obj):\n", " info = getframeinfo(frame)\n", " if '' in info.filename or ':2:\n", "- Access to None (@0x55feae7373a0) at :2:\n", "- Access to (@0x7fe342f347e0) at :3:\n", "- Access to 'I am alive!' (@0x7fe313b0b130) at :3:\n", "I am alive!\n", "- Access to None (@0x55feae7373a0) at :3:\n", "- Access to None (@0x55feae7373a0) at :3:\n", "- Access to > (@0x7fe33b57d548) at :4:\n", "- Access to (@0x7fe33b56d0b8) at :4:\n", "- Access to 'time 123 + 134' (@0x7fe312d049b0) at :4:\n", "CPU times: user 8 ms, sys: 0 ns, total: 8 ms\n", "Wall time: 10.8 ms\n", "- Access to 257 (@0x7fe312c98c70) at :4:\n" ] }, { "data": { "text/plain": [ "257" ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" }, { "name": "stdout", "output_type": "stream", "text": [ "- Access to None (@0x55feae7373a0) at :4:\n", "- Access to (@0x7fe328099678) at :5:\n", "- Access to {} (@0x7fe312d07908) at :5:\n" ] } ], "source": [ "setrwatch({})\n", "setrwatch(defaultdict(lambda: debug_view_for_any_object))\n", "print(x)\n", "%time 123 + 134\n", "setrwatch({})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It seems to work very well!\n", "\n", "But it slows down everything, obviously the filtering takes time (for *every* object!)\n", "Computing `123 + 134 = 257` took about `10` miliseconds! That's just CRAZY!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### A first context manager to have debugging for *one* object\n", "\n", "It would be nice to be able to turn on and off this debugging tool whenever you want.\n", "\n", "Well, it turns out that [context managers](https://docs.python.org/3.5/library/stdtypes.html#typecontextmanager) are exactly meant for that!\n", "They are simple classes with just a `__enter__()` and `__exit__()` special methods.\n", "\n", "First, let us write a context manager to debug ONE object." ] }, { "cell_type": "code", "execution_count": 54, "metadata": { "collapsed": true }, "outputs": [], "source": [ "class InspectThisObject(object):\n", " def __init__(self, obj):\n", " self.idobj = id(obj)\n", " \n", " def __enter__(self):\n", " getrwatch()[self.idobj] = debug_view\n", "\n", " def __exit__(self, exc_type, exc_val, exc_tb):\n", " delrwatch(self.idobj)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can check it:" ] }, { "cell_type": "code", "execution_count": 55, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "I am Batman!\n", "- Access to 'I am Batman!' (@0x7fe312d05af0) at :5:\n", "- Access to 'I am Batman!' (@0x7fe312d05af0) at /usr/local/lib/python3.5/dist-packages/ipykernel/iostream.py:347:write\n", "I am Batman!\n", "I am Batman!\n" ] } ], "source": [ "z = \"I am Batman!\"\n", "print(z)\n", "\n", "with InspectThisObject(z):\n", " print(z)\n", "\n", "print(z)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The first debug information shows line 5, which is the line where `print(z)` is." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### A second context manager to debug *any* object\n", "Easy:" ] }, { "cell_type": "code", "execution_count": 56, "metadata": { "collapsed": true }, "outputs": [], "source": [ "class InspectAllObjects(object):\n", " def __init__(self):\n", " pass\n", "\n", " def __enter__(self):\n", " setrwatch(defaultdict(lambda: debug_view_for_any_object))\n", "\n", " def __exit__(self, exc_type, exc_val, exc_tb):\n", " setrwatch({})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It will probably break everything in the notebook, but works in a basic Python interpreter." ] }, { "cell_type": "code", "execution_count": 57, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "- Access to None (@0x55feae7373a0) at :6:__enter__\n", "- Access to None (@0x55feae7373a0) at :6:__enter__\n", "- Access to None (@0x55feae7373a0) at :1:\n", "- Access to (@0x7fe342f347e0) at :2:\n", "- Access to 0 (@0x55feae761480) at :2:\n", "0\n", "- Access to None (@0x55feae7373a0) at :2:\n", "- Access to None (@0x55feae7373a0) at :2:\n", "- Access to (@0x7fe328099678) at :9:__exit__\n", "- Access to {} (@0x7fe312b08208) at :9:__exit__\n" ] } ], "source": [ "with InspectAllObjects():\n", " print(0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The 5th debug information printed is `Access to 0 (@0xXXX) at :2:`, showing the access in line `#2` of the constant `0`." ] }, { "cell_type": "code", "execution_count": 58, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "- Access to None (@0x55feae7373a0) at :6:__enter__\n", "- Access to None (@0x55feae7373a0) at :6:__enter__\n", "- Access to None (@0x55feae7373a0) at :1:\n", "- Access to (@0x7fe342f347e0) at :2:\n", "- Access to 'Darth Vader -- No Luke, I am your Father!' (@0x7fe312bedf90) at :2:\n", "Darth Vader -- No Luke, I am your Father!\n", "- Access to None (@0x55feae7373a0) at :2:\n", "- Access to (@0x7fe342f347e0) at :3:\n", "- Access to \"Luke -- I have a father? Yay! Let's eat cookies together!\" (@0x7fe312b71f80) at :3:\n", "Luke -- I have a father? Yay! Let's eat cookies together!\n", "- Access to None (@0x55feae7373a0) at :3:\n", "- Access to None (@0x55feae7373a0) at :3:\n", "- Access to (@0x7fe328099678) at :9:__exit__\n", "- Access to {} (@0x7fe312af3f08) at :9:__exit__\n" ] } ], "source": [ "with InspectAllObjects():\n", " print(\"Darth Vader -- No Luke, I am your Father!\")\n", " print(\"Luke -- I have a father? Yay! Let's eat cookies together!\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We also see here the `None` and `{}` objects being given to the context manager (see the `__enter__` method at first, and `__exit__` at the end)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "## Defining a context manager to add white noise\n", "\n", "Basically, we will do as above, but instead of debug information, a white noise sampled from a Normal distribution (i.e., $\\sim \\mathcal{N}(0, 1)$) will be *added* to any number." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Capturing any numerical value\n", "To capture both integers and float numbers, the [`numbers.Number`](https://docs.python.org/2/library/numbers.html#numbers.Number) abstract class is useful." ] }, { "cell_type": "code", "execution_count": 59, "metadata": { "collapsed": true, "scrolled": true }, "outputs": [], "source": [ "from numbers import Number" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Adding a white noise for numbers\n", "This is very simple.\n", "\n", "But I want to be safe, so it will only works if the frame indicate that the number does not come from a file, as previously." ] }, { "cell_type": "code", "execution_count": 60, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def add_white_noise_to_numbers(frame, obj):\n", " if isinstance(obj, Number):\n", " info = getframeinfo(frame)\n", " if '' in info.filename or '' in info.filename or ' « Now, the real world is non noisy, but the complex one is! »\n", "\n", "That's one sentence I thought I would never say!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `WhiteNoiseComplex` context manager\n", "\n", "To stay cautious, I only add noise to complex numbers." ] }, { "cell_type": "code", "execution_count": 67, "metadata": { "collapsed": true }, "outputs": [], "source": [ "class WhiteNoiseComplex(object):\n", " def __init__(self):\n", " pass\n", "\n", " def __enter__(self):\n", " setrwatch(defaultdict(lambda: add_white_noise_to_complex))\n", "\n", " def __exit__(self, exc_type, exc_val, exc_tb):\n", " setrwatch({})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And it works as expected:" ] }, { "cell_type": "code", "execution_count": 69, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "120193 120193j\n", "120193 (-1.4079099824891002+120192.24863460941j)\n", "120193 120193j\n", "0j\n", "(1.5232719431619033-1.3234485447145865j)\n", "0j\n" ] } ], "source": [ "np.random.seed(120193)\n", "print(120193, 120193j)\n", "with WhiteNoiseComplex():\n", " print(120193, 120193j) # Huhoo, noisy!\n", "print(120193, 120193j)\n", "\n", "print(0*1j)\n", "with WhiteNoiseComplex():\n", " print(0*1j) # Huhoo, noisy!\n", "print(0*1j)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "----\n", "## Defining a generic `noisy` context manager\n", "\n", "This will be a very simple change from the previous one, by letting the `Noisy` class accept *any* noisy function, which takes `obj` and return a noisy version of `obj`, only for complex-valued objects." ] }, { "cell_type": "code", "execution_count": 71, "metadata": { "collapsed": true }, "outputs": [], "source": [ "class Noisy(object):\n", " def __init__(self, noise):\n", " def add_white_noise_to_complex(frame, obj):\n", " if isinstance(obj, complex):\n", " info = getframeinfo(frame)\n", " if '' in info.filename or '