{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Lecture 7: Interoperability with Legacy and Modern Environments (45 Minutes) #" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ABSTRACT ###\n", "\n", "In this Lecture, we will briefly see a variety of examples of how to interoperate between Python and other programming environments such as:\n", "- C,\n", "- MATLAB,\n", "- Julia.\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Very often, different projects will need to interoperate with each other. Your projects might have to call code ranging from obscure and poorly-documented device drivers and numerical libraries, to legacy projects that depend on \"walled-garden\" style software environments, or even to cutting-edge code in new and exciting scientific computing platforms. Thankfully, Python plays well with others in the sense that a range of tools exist to help make your Python-language projects talk to code written in other languages and for other platforms.\n", "\n", "This Lecture will be a bit different than the rest, in that we'll not actually do all that much, but rather will briefly review a lot of these interoperability tools. By nature, each of them will come along with a lot of additional baggage (it's hard to use a C-to-Python interface without delving into C, for instance), such that our goal will be to learn what tools are out there. That way, you'll be better equipped to find useful resources when you need to write interoperable code.\n", "\n", "In particular, we'll cover interacting with C, MATLAB, and Julia, but we note that many other tools exist. These are only meant to give the \"flavor\" of what tools are available." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ``ctypes``, ``cffi``, and Cython — Interoperating with C Language Projects ##" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Many different projects and libraries expose an interface for C-language code, making C a kind of *lingua franca* for computing as a whole. The Python interpreter itself, for instance, is a C-language program, with its own application-binary interface (ABI) for interacting with other C code. The dominance of C for specifying such APIs means that even projects written in other compiled languages like C++, D, or Rust will often expose ABIs as though they were written in C. Thus, chances are pretty good that you'll eventually run across something that you need to call into a C library to use, even if you never plan on using C directly.\n", "\n", "*NB: you may be familiar with the term \"API\" rather than ABI. This separation reflects the difference between compile-time (API) and linking-time (ABI) interfaces, or alternatively between semantic interfaces (APIs) and low-level interfaces (ABI). At the moment, though, we don't need to worry about this other than to note that ``ctypes`` acts at the level of compiled libraries.*\n", "\n", "Thankfully, in recognition of this, Python provides a standard library module ``ctypes`` for loading C-language libraries and calling into them. This is used, for instance, by the [``pyDAQmx`` project](https://github.com/clade/PyDAQmx) to provide a Python-language wrapper for the DAQmx library provided by National Instruments to control their data acquisition (DAQ) devices.\n", "\n", "Roughly, the procedure for using ``ctypes`` to interact with C-language libraries goes in three steps:\n", "\n", "- Load the C-language library.\n", "- Define the types of the functions exposed by that library.\n", "- Call functions using ``ctypes`` objects that wrap C types.\n", "\n", "Let's look quickly at an example of how this might be used on Windows to access the Microsoft Visual C++ Runtime.\n", "\n", "*[h/t anirudh](https://stackoverflow.com/q/24939657) for suggesting this minimal example.*" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "13" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from ctypes import cdll\n", "\n", "msvcrt = cdll.msvcrt\n", "n_bytes_written = msvcrt.printf(\"Hello world!\\n\")\n", "print(n_bytes_written)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*NB: if you try running this in the Notebook interface, you won't see anything print out in the Notebook. MSVCRT isn't aware of the Notebook, and instead prints out to the console running the server.*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Often, the API exposed by a C library is specified in terms of *header files*. Unfortunately, ``ctypes`` provides no way to automatically turn a header file into a Python wrapper, meaning that wrapping such libraries using only built-in packages can be laborious. To address this need, [``cffi``](https://cffi.readthedocs.io/en/latest/) provides a different approach. Instead of directly constructing function objects which wrap C functions, ``cffi`` will parse a header file and generate a wrapper automatically. For instance, the ``cffi`` documentation suggests the following example to achieve the same as above:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "17" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from cffi import FFI\n", "ffi = FFI()\n", "ffi.cdef(\"\"\"\n", " int printf(const char *format, ...); // copy-pasted from the man page\n", "\"\"\")\n", "C = ffi.dlopen(None) # loads the entire C namespace\n", "arg = ffi.new(\"char[]\", \"world\") # equivalent to C code: char arg[] = \"world\";\n", "C.printf(\"hi there, %s.\\n\", arg) # call printf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is more work if you're only using a single function, but can save a heck of a lot of time and help avoid errors if you're Interoperating with larger libraries.\n", "\n", "Both of these approaches work dynamically at runtime, using the Python interpreter to manage *dynamic linking*. While this is fine for many tasks, it does come with the downside that using Python to manage dynamic linking is fairly slow. That doesn't matter much for instrument control, which is typically IO-bound rather than CPU-bound, but can matter a lot if the point of working with C is to interact with accelerated libraries. For this, there's not much substitute for writing new code in C that talks both to the library of interest and to the Python interpreter. This new C code will then be linked at compile-time or at run-time as managed by the compiler or operating system dynamic linker, eliminating the overhead of working through the interpreter. This is a painful process, though, such that we'd like a nicer way to generate C code for the purpose of interacting with Python.\n", "\n", "Enter Cython. Neither C nor Python, Cython is a third language specialized for generating new C code that works with both. Cython modules are written as ``.pyx`` files rather than ``.py`` files, and are then compiled by calling ``cython`` directly, through an install script such as ``setup.py``, or by using the ``pyximport`` module provided with Cython." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## ``import matlab`` and ``py.`` — Interoperating with the MATLAB Platform ##" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It's no secret that a lot of scientific software currently out there is written in the MATLAB language, and is designed for use from within the MATLAB integrated development environment (IDE). From that perspective, MATLAB has historically been somewhat of what's called a \"walled garden\" in software engineering, making it very hard to interoperate between MATLAB and other software platforms. Though MATLAB does support writing native C, C++, or FORTRAN code that interacts with the [MEX API](https://www.mathworks.com/help/matlab/matlab_external/introducing-mex-files.html?s_tid=gn_loc_drop), this API is quite unwieldy to work with directly if one wants to interoperate with MATLAB. Thankfully, in versions 2014b and 2016a, MATLAB now supports two new features that make it much easier to work interoperability.\n", "\n", "*NB: MEX is an API, as it's defined at compile-time, similarly to the Python interpreter's C API. Contrast with MATLAB's ``loadlibrary`` function and the ``ctypes`` module described above.*\n", "\n", "The first, called the [MATLAB Engine API for Python](https://www.mathworks.com/help/matlab/matlab-engine-for-python.html) and introduced in 2014b, is a new Python package ``matlab`` that abstracts away much of the interoperability pain. For instance, the MATLAB documentation suggests the following example:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```python\n", "import matlab.engine\n", "eng = matlab.engine.start_matlab()\n", "tf = eng.isprime(37)\n", "print(tf)\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Unfortunately, the MATLAB Engine API for Python does not support arrays or any other NumPy features, meaning you must pack things in and out of lists in order to communicate with MATLAB." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To go the other way, and call Python code from MATLAB, 2016a now supports calling Python functions using the special naming scheme ``py.``. For example, the QInfer documentation provides the following example:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```matlab\n", ">> true_omega = 70.3;\n", ">> n_shots = 400;\n", ">>\n", ">> ts = pi * (1:1:100) / (2 * 100);\n", ">>\n", ">> signal = sin(true_omega * ts / 2) .^ 2;\n", ">> counts = binornd(n_shots, signal);\n", ">>\n", ">> setenv MKL_NUM_THREADS 1\n", ">> data = py.numpy.column_stack({counts ts ...\n", "n_shots * ones(1, size(ts, 2))});\n", ">> est = py.qinfer.simple_est_prec(data, ...\n", "pyargs('freq_min', 0, 'freq_max', 100));\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that the ``setenv MKL_NUM_THREADS 1`` line is **critical** due to an internal bug in MATLAB. In particular, if this isn't set, then any use of the Math Kernel Library (MKL) from within Python will crash both MATLAB and Python due to a race condition with MATLAB's own use of MKL. Since the MKL is used extensively by Anaconda to accelerate computation, we must eliminate the race condition by disabling thread-level parallelization, as controlled by the ``MKL_NUM_THREADS`` environment variable." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## ``PyCall.jl`` — Calling into Python from Julia ##" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, we'll discuss calling into Python from Julia, an exciting new just-in-time–compiled scientific computing language. Julia offers a range of new features and advantages over general-purpose languages, but is currently new enough that if you write in Julia, you'll somewhat commonly need functionality from Python-language packages. Julia addresses this by providing the ``PyCall.jl`` package to represent Python packages as Julia variables. If you haven't installed Julia already and would like to follow this part of the lecture, you can do so with your friendly neighborhood package manager:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- **Windows**:\n", "```\n", "PS > choco install julia\n", "```\n", "\n", "*NB: You may need to manually add Julia to your path on Windows.*\n", "\n", "- **Ubuntu Linux**:\n", "```bash\n", "$ sudo add-apt-repository ppa:staticfloat/juliareleases\n", "$ sudo add-apt-repository ppa:staticfloat/julia-deps\n", "$ sudo apt-get update\n", "$ sudo apt-get install julia\n", "```\n", "- **macOS / OS X**: The Julia project provides [a ``.dmg``-based installer](https://s3.amazonaws.com/julialang/bin/osx/x64/0.5/julia-0.5.0-osx10.7+.dmg)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once Julia is installed, to also install ``PyCall.jl``, use Julia's built-in package manager. This package manager exposes itself as a built-in Julia variable with several methods. We're interested in the ``add`` method, which installs a new package by cloning the relevant Git repository and installing each of its nominated dependencies in turn." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```bash\n", "$ julia\n", "```\n", "```julia\n", "julia> Pkg.add(\"PyCall.jl\")\n", "INFO: Cloning cache of BinDeps from git://github.com/JuliaLang/BinDeps.jl.git\n", "INFO: Cloning cache of Compat from git://github.com/JuliaLang/Compat.jl.git\n", "INFO: Cloning cache of Conda from git://github.com/JuliaPy/Conda.jl.git\n", "INFO: Cloning cache of JSON from git://github.com/JuliaIO/JSON.jl.git\n", "INFO: Cloning cache of MacroTools from git://github.com/MikeInnes/MacroTools.jl.git\n", "INFO: Cloning cache of PyCall from git://github.com/JuliaPy/PyCall.jl.git\n", "INFO: Cloning cache of SHA from git://github.com/staticfloat/SHA.jl.git\n", "INFO: Cloning cache of URIParser from git://github.com/JuliaWeb/URIParser.jl.git\n", "INFO: Installing BinDeps v0.4.5\n", "INFO: Installing Compat v0.9.3\n", "INFO: Installing Conda v0.3.2\n", "INFO: Installing JSON v0.8.0\n", "INFO: Installing MacroTools v0.3.2\n", "INFO: Installing PyCall v1.7.2\n", "INFO: Installing SHA v0.2.1\n", "INFO: Installing URIParser v0.1.6\n", "INFO: Building PyCall\n", "INFO: PyCall is using python (Python 2.7.12) at C:\\Users\\cgranade\\Anaconda2\\python.exe, libpython = C:\\Users\\cgranade\\Anaconda2\\python27\n", "INFO: Package database updated\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*NB: This is also how one installs the IJulia kernel for Jupyter Notebook: ``Pkg.add(\"IJulia\")``.*\n", "\n", "Once installed, you can now use the ``@pyimport`` *macro* to import Python modules as Julia variables. For instance, to use Julia to talk to an instrument with InstrumentKit:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```julia\n", "julia> using PyCall\n", "julia> @pyimport instruments.generic_scpi as scpi\n", "julia> scpi.SCPIInstrument[:open_from_uri](\"serial:///COM1\")\n", "```" ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python [conda root]", "language": "python", "name": "conda-root-py" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.12" } }, "nbformat": 4, "nbformat_minor": 1 }