{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Inspecting Call Stacks\n", "\n", "In this book, for many purposes, we need to look up a function's location, source code, or simply definition. The class `StackInspector` provides a number of convenience methods for this purpose." ] }, { "cell_type": "markdown", "metadata": { "button": false, "new_sheet": false, "run_control": { "read_only": false }, "slideshow": { "slide_type": "subslide" } }, "source": [ "**Prerequisites**\n", "\n", "* This is an internal helper class.\n", "* Understanding how frames and local variables are represented in Python helps." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "## Synopsis\n", "\n", "\n", "To [use the code provided in this chapter](Importing.ipynb), write\n", "\n", "```python\n", ">>> from debuggingbook.StackInspector import \n", "```\n", "\n", "and then make use of the following features.\n", "\n", "\n", "`StackInspector` is typically used as superclass, providing its functionality to subclasses. \n", "\n", "Here is an example of how to use `caller_function()`. The `test()` function invokes an internal method `caller()` of `StackInspectorDemo`, which in turn invokes `callee()`:\n", "\n", "| Function | Class | |\n", "| --- | --- | --- |\n", "| `callee()` | `StackInspectorDemo` | |\n", "| `caller()` | `StackInspectorDemo` | invokes $\\uparrow$ |\n", "| `test()` | (main) | invokes $\\uparrow$ |\n", "| -/- | (main) | invokes $\\uparrow$ |\n", "\n", "Using `caller_function()`, `callee()` determines the first caller outside a `StackInspector` class and prints it out – i.e., ``.\n", "\n", "```python\n", ">>> class StackInspectorDemo(StackInspector):\n", ">>> def callee(self) -> None:\n", ">>> func = self.caller_function()\n", ">>> assert func.__name__ == 'test'\n", ">>> print(func)\n", ">>> \n", ">>> def caller(self) -> None:\n", ">>> self.callee()\n", ">>> def test() -> None:\n", ">>> demo = StackInspectorDemo()\n", ">>> demo.caller()\n", ">>> test()\n", "\n", "\n", "```\n", "Here are all methods defined in this chapter:\n", "\n", "![](PICS/StackInspector-synopsis-1.svg)\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Inspecting Call Stacks\n", "\n", "`StackInspector` is a class that provides a number of utility functions to inspect a [call stack](https://en.wikipedia.org/wiki/Call_stack), notably to identify caller functions." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "When tracing or instrumenting functions, a common issue is to identify the currently active functions. A typical situation is depicted below, where `my_inspector()` currently traces a function called `function_under_test()`.\n", "\n", "| Function | Class | |\n", "| --- | --- | --- |\n", "| ... | `StackInspector` | |\n", "| `caller_frame()` | `StackInspector` | invokes $\\uparrow$ |\n", "| `caller_function()` | `StackInspector` | invokes $\\uparrow$ |\n", "| `my_inspector()` | some inspector; a subclass of `StackInspector` | invokes $\\uparrow$ |\n", "| `function_under_test()` | (any) | is traced by $\\uparrow$ |\n", "| -/- | (any) | invokes $\\uparrow$ |\n", "\n", "To determine the calling function, `my_inspector()` could check the current frame and retrieve the frame of the caller. However, this caller could be some tracing function again invoking `my_inspector()`. Therefore, `StackInspector` provides a method `caller_function()` that returns the first caller outside a `StackInspector` class. This way, a subclass of `StackInspector` can define an arbitrary set of functions (and call stack); `caller_function()` will always return a function outside the `StackInspector` subclass." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "button": false, "execution": { "iopub.execute_input": "2023-11-12T13:02:56.271390Z", "iopub.status.busy": "2023-11-12T13:02:56.271203Z", "iopub.status.idle": "2023-11-12T13:02:56.306064Z", "shell.execute_reply": "2023-11-12T13:02:56.305638Z" }, "new_sheet": false, "run_control": { "read_only": false }, "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "import bookutils.setup" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.308366Z", "iopub.status.busy": "2023-11-12T13:02:56.308116Z", "iopub.status.idle": "2023-11-12T13:02:56.310152Z", "shell.execute_reply": "2023-11-12T13:02:56.309848Z" }, "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "import inspect\n", "import warnings" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.312023Z", "iopub.status.busy": "2023-11-12T13:02:56.311885Z", "iopub.status.idle": "2023-11-12T13:02:56.313915Z", "shell.execute_reply": "2023-11-12T13:02:56.313604Z" }, "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "from types import FunctionType, FrameType, TracebackType" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.315923Z", "iopub.status.busy": "2023-11-12T13:02:56.315786Z", "iopub.status.idle": "2023-11-12T13:02:56.317950Z", "shell.execute_reply": "2023-11-12T13:02:56.317542Z" }, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "# ignore\n", "from typing import cast, Dict, Any, Tuple, Callable, Optional, Type" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The method `caller_frame()` walks the current call stack from the current frame towards callers (using the `f_back` attribute of the current frame) and returns the first frame that is _not_ a method or function from the current `StackInspector` class or its subclass. To determine this, the method `our_frame()` determines whether the given execution frame refers to one of the methods of `StackInspector` or one of its subclasses." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.320005Z", "iopub.status.busy": "2023-11-12T13:02:56.319854Z", "iopub.status.idle": "2023-11-12T13:02:56.322673Z", "shell.execute_reply": "2023-11-12T13:02:56.322352Z" }, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class StackInspector:\n", " \"\"\"Provide functions to inspect the stack\"\"\"\n", "\n", " def caller_frame(self) -> FrameType:\n", " \"\"\"Return the frame of the caller.\"\"\"\n", "\n", " # Walk up the call tree until we leave the current class\n", " frame = cast(FrameType, inspect.currentframe())\n", "\n", " while self.our_frame(frame):\n", " frame = cast(FrameType, frame.f_back)\n", "\n", " return frame\n", "\n", " def our_frame(self, frame: FrameType) -> bool:\n", " \"\"\"Return true if `frame` is in the current (inspecting) class.\"\"\"\n", " return isinstance(frame.f_locals.get('self'), self.__class__)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "When we access program state or execute functions, we do so in the caller's environment, not ours. The `caller_globals()` method acts as replacement for `globals()`, using `caller_frame()`." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.325016Z", "iopub.status.busy": "2023-11-12T13:02:56.324857Z", "iopub.status.idle": "2023-11-12T13:02:56.327830Z", "shell.execute_reply": "2023-11-12T13:02:56.327417Z" }, "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class StackInspector(StackInspector):\n", " def caller_globals(self) -> Dict[str, Any]:\n", " \"\"\"Return the globals() environment of the caller.\"\"\"\n", " return self.caller_frame().f_globals\n", "\n", " def caller_locals(self) -> Dict[str, Any]:\n", " \"\"\"Return the locals() environment of the caller.\"\"\"\n", " return self.caller_frame().f_locals" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The method `caller_location()` returns the caller's function and its location. It does a fair bit of magic to retrieve nested functions, by looking through global and local variables until a match is found. This may be simplified in the future." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.330138Z", "iopub.status.busy": "2023-11-12T13:02:56.329941Z", "iopub.status.idle": "2023-11-12T13:02:56.332123Z", "shell.execute_reply": "2023-11-12T13:02:56.331721Z" }, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "Location = Tuple[Callable, int]" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.334038Z", "iopub.status.busy": "2023-11-12T13:02:56.333865Z", "iopub.status.idle": "2023-11-12T13:02:56.336496Z", "shell.execute_reply": "2023-11-12T13:02:56.336056Z" }, "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "class StackInspector(StackInspector):\n", " def caller_location(self) -> Location:\n", " \"\"\"Return the location (func, lineno) of the caller.\"\"\"\n", " return self.caller_function(), self.caller_frame().f_lineno" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The function `search_frame()` allows searching for an item named `name`, walking up the call stack. This is handy when trying to find local functions during tracing, for whom typically only the name is provided." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.338614Z", "iopub.status.busy": "2023-11-12T13:02:56.338474Z", "iopub.status.idle": "2023-11-12T13:02:56.342469Z", "shell.execute_reply": "2023-11-12T13:02:56.342127Z" }, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class StackInspector(StackInspector):\n", " def search_frame(self, name: str, frame: Optional[FrameType] = None) -> \\\n", " Tuple[Optional[FrameType], Optional[Callable]]:\n", " \"\"\"\n", " Return a pair (`frame`, `item`) \n", " in which the function `name` is defined as `item`.\n", " \"\"\"\n", " if frame is None:\n", " frame = self.caller_frame()\n", "\n", " while frame:\n", " item = None\n", " if name in frame.f_globals:\n", " item = frame.f_globals[name]\n", " if name in frame.f_locals:\n", " item = frame.f_locals[name]\n", " if item and callable(item):\n", " return frame, item\n", "\n", " frame = cast(FrameType, frame.f_back)\n", "\n", " return None, None\n", "\n", " def search_func(self, name: str, frame: Optional[FrameType] = None) -> \\\n", " Optional[Callable]:\n", " \"\"\"Search in callers for a definition of the function `name`\"\"\"\n", " frame, func = self.search_frame(name, frame)\n", " return func" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "If we cannot find a function by name, we can create one, using `create_function()`." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.344321Z", "iopub.status.busy": "2023-11-12T13:02:56.344144Z", "iopub.status.idle": "2023-11-12T13:02:56.347480Z", "shell.execute_reply": "2023-11-12T13:02:56.347144Z" }, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class StackInspector(StackInspector):\n", " # Avoid generating functions more than once\n", " _generated_function_cache: Dict[Tuple[str, int], Callable] = {}\n", "\n", " def create_function(self, frame: FrameType) -> Callable:\n", " \"\"\"Create function for given frame\"\"\"\n", " name = frame.f_code.co_name\n", " cache_key = (name, frame.f_lineno)\n", " if cache_key in self._generated_function_cache:\n", " return self._generated_function_cache[cache_key]\n", "\n", " try:\n", " # Create new function from given code\n", " generated_function = cast(Callable,\n", " FunctionType(frame.f_code,\n", " globals=frame.f_globals,\n", " name=name))\n", " except TypeError:\n", " # Unsuitable code for creating a function\n", " # Last resort: Return some function\n", " generated_function = self.unknown\n", "\n", " except Exception as exc:\n", " # Any other exception\n", " warnings.warn(f\"Couldn't create function for {name} \"\n", " f\" ({type(exc).__name__}: {exc})\")\n", " generated_function = self.unknown\n", "\n", " self._generated_function_cache[cache_key] = generated_function\n", " return generated_function" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "The method `caller_function()` puts all of these together, simply looking up and returning the currently calling function – and creating one if it cannot be found." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.349545Z", "iopub.status.busy": "2023-11-12T13:02:56.349266Z", "iopub.status.idle": "2023-11-12T13:02:56.352362Z", "shell.execute_reply": "2023-11-12T13:02:56.351977Z" }, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class StackInspector(StackInspector):\n", " def caller_function(self) -> Callable:\n", " \"\"\"Return the calling function\"\"\"\n", " frame = self.caller_frame()\n", " name = frame.f_code.co_name\n", " func = self.search_func(name)\n", " if func:\n", " return func\n", "\n", " if not name.startswith('<'):\n", " warnings.warn(f\"Couldn't find {name} in caller\")\n", "\n", " return self.create_function(frame)\n", "\n", " def unknown(self) -> None: # Placeholder for unknown functions\n", " pass" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "The method `is_internal_error()` allows us to differentiate whether some exception was raised by `StackInspector` (or a subclass) – or whether it was raised by the inspected code." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.354400Z", "iopub.status.busy": "2023-11-12T13:02:56.354172Z", "iopub.status.idle": "2023-11-12T13:02:56.356476Z", "shell.execute_reply": "2023-11-12T13:02:56.356007Z" }, "slideshow": { "slide_type": "skip" } }, "outputs": [], "source": [ "import traceback" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.358335Z", "iopub.status.busy": "2023-11-12T13:02:56.358197Z", "iopub.status.idle": "2023-11-12T13:02:56.360937Z", "shell.execute_reply": "2023-11-12T13:02:56.360609Z" }, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class StackInspector(StackInspector):\n", " def is_internal_error(self, exc_tp: Type, \n", " exc_value: BaseException, \n", " exc_traceback: TracebackType) -> bool:\n", " \"\"\"Return True if exception was raised from `StackInspector` or a subclass.\"\"\"\n", " if not exc_tp:\n", " return False\n", "\n", " for frame, lineno in traceback.walk_tb(exc_traceback):\n", " if self.our_frame(frame):\n", " return True\n", "\n", " return False" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Synopsis\n", "\n", "`StackInspector` is typically used as superclass, providing its functionality to subclasses. " ] }, { "attachments": {}, "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Here is an example of how to use `caller_function()`. The `test()` function invokes an internal method `caller()` of `StackInspectorDemo`, which in turn invokes `callee()`:\n", "\n", "| Function | Class | |\n", "| --- | --- | --- |\n", "| `callee()` | `StackInspectorDemo` | |\n", "| `caller()` | `StackInspectorDemo` | invokes $\\uparrow$ |\n", "| `test()` | (main) | invokes $\\uparrow$ |\n", "| -/- | (main) | invokes $\\uparrow$ |\n", "\n", "Using `caller_function()`, `callee()` determines the first caller outside a `StackInspector` class and prints it out – i.e., ``." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.362748Z", "iopub.status.busy": "2023-11-12T13:02:56.362625Z", "iopub.status.idle": "2023-11-12T13:02:56.364848Z", "shell.execute_reply": "2023-11-12T13:02:56.364574Z" }, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "class StackInspectorDemo(StackInspector):\n", " def callee(self) -> None:\n", " func = self.caller_function()\n", " assert func.__name__ == 'test'\n", " print(func)\n", "\n", " def caller(self) -> None:\n", " self.callee()" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.366758Z", "iopub.status.busy": "2023-11-12T13:02:56.366643Z", "iopub.status.idle": "2023-11-12T13:02:56.368654Z", "shell.execute_reply": "2023-11-12T13:02:56.368367Z" }, "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "def test() -> None:\n", " demo = StackInspectorDemo()\n", " demo.caller()" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.370380Z", "iopub.status.busy": "2023-11-12T13:02:56.370258Z", "iopub.status.idle": "2023-11-12T13:02:56.372421Z", "shell.execute_reply": "2023-11-12T13:02:56.372050Z" }, "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "test()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Here are all methods defined in this chapter:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.394670Z", "iopub.status.busy": "2023-11-12T13:02:56.394473Z", "iopub.status.idle": "2023-11-12T13:02:56.445492Z", "shell.execute_reply": "2023-11-12T13:02:56.445165Z" }, "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "# ignore\n", "from ClassDiagram import display_class_hierarchy, class_tree" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2023-11-12T13:02:56.447885Z", "iopub.status.busy": "2023-11-12T13:02:56.447652Z", "iopub.status.idle": "2023-11-12T13:02:56.898593Z", "shell.execute_reply": "2023-11-12T13:02:56.898162Z" }, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "StackInspector\n", "\n", "\n", "StackInspector\n", "\n", "\n", "\n", "_generated_function_cache\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "caller_frame()\n", "\n", "\n", "\n", "caller_function()\n", "\n", "\n", "\n", "caller_globals()\n", "\n", "\n", "\n", "caller_locals()\n", "\n", "\n", "\n", "caller_location()\n", "\n", "\n", "\n", "is_internal_error()\n", "\n", "\n", "\n", "our_frame()\n", "\n", "\n", "\n", "search_frame()\n", "\n", "\n", "\n", "search_func()\n", "\n", "\n", "\n", "create_function()\n", "\n", "\n", "\n", "unknown()\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "Legend\n", "Legend\n", "• \n", "public_method()\n", "• \n", "private_method()\n", "• \n", "overloaded_method()\n", "Hover over names to see doc\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# ignore\n", "display_class_hierarchy([StackInspector],\n", " abstract_classes=[\n", " StackInspector,\n", " ],\n", " public_methods=[\n", " StackInspector.caller_frame,\n", " StackInspector.caller_function,\n", " StackInspector.caller_globals,\n", " StackInspector.caller_locals,\n", " StackInspector.caller_location,\n", " StackInspector.search_frame,\n", " StackInspector.search_func,\n", " StackInspector.is_internal_error,\n", " StackInspector.our_frame,\n", " ],\n", " project='debuggingbook')" ] }, { "cell_type": "markdown", "metadata": { "button": false, "new_sheet": true, "run_control": { "read_only": false }, "slideshow": { "slide_type": "slide" } }, "source": [ "## Lessons Learned\n", "\n", "* In Python, inspecting objects at runtime is easy." ] } ], "metadata": { "ipub": { "bibliography": "fuzzingbook.bib", "toc": true }, "kernelspec": { "display_name": "venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.2" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": true, "title_cell": "", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": true }, "toc-autonumbering": false, "vscode": { "interpreter": { "hash": "0af4f07dd039d1b4e562c7a7d0340393b1c66f50605ac6af30beb81aa23b7ef5" } } }, "nbformat": 4, "nbformat_minor": 4 }