{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom D3.js Visualization in a Jupyter Notebook\n",
    "\n",
    "by Stefaan Lippens ([@soxofaan](https://twitter.com/soxofaan)), May 2018"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TL;DR: If you want to skip the detailed mumbo jumbo, go straight to the code example in the [conclusion](#All-together-now).\n",
    "\n",
    "\n",
    "## Introduction\n",
    "\n",
    "When doing data science in a Jupyter notebook, there are plenty \n",
    "of options for the standard data visualization needs:\n",
    "matplotlib, pandas, seaborn, bokeh, etc. \n",
    "Occasionally you might be stuck in a situation where you can not \n",
    "easily express the desired visualization with the \n",
    "standard vocabulary provided by these tools.\n",
    "In these cases I like to leverage the flexibility\n",
    "of [D3.js](https://d3js.org/) to build a custom graph or diagram.\n",
    "\n",
    "\n",
    "In this article, I'll discuss an approach how to \n",
    "implement a **custom do-it-yourself D3.js visualization** in a **Jupyter Notebook**.\n",
    "This topic is covered in some other places around the web,\n",
    "but I couldn't find a complete approach that\n",
    "connects all the dots and isn't too hackish. \n",
    "\n",
    "In particular, I'll try to keep these things in mind here:\n",
    "\n",
    "- Leverage the [existing Jupyter functionality](http://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/JavaScript%20Notebook%20Extensions.html),\n",
    "  like a [RequireJS](http://requirejs.org/) environment \n",
    "  to have clean dependency handling \n",
    "  and to avoid `<script src=\"...\">` loading/order headaches.\n",
    "- No additional packages or dependencies to install\n",
    "- The visualisation should work both in an interactive notebook context\n",
    "  and in the exported HTML version.\n",
    "- Avoid the typical D3.js boilerplate to insert the drawing at the desired\n",
    "  location (insert a `<div id=\"...\">` in the markup and use the corresponding id in the javascript code).\n",
    "  Keeping these id's properly in sync is annoying to maintain, \n",
    "  especially if you iterate a lot or want multiple drawings in the same notebook.\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The basics with inline `%%javascript` snippets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To cover the basics, let's start simple with just inline `%%javascript` snippet cells.\n",
    "\n",
    "\n",
    "First, we tell the RequireJS environment where to find the version of D3.js we want.\n",
    "Note that the `.js` extension is [omitted](http://requirejs.org/docs/api.html#config-paths) in the URL. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "require.config({\n",
       "    paths: {\n",
       "        d3: 'https://d3js.org/d3.v5.min'\n",
       "    }\n",
       "});"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%javascript\n",
    "require.config({\n",
    "    paths: {\n",
    "        d3: 'https://d3js.org/d3.v5.min'\n",
    "    }\n",
    "});"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now create a D3.js powered SVG drawing, for example as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "(function(element) {\n",
       "    require(['d3'], function(d3) {   \n",
       "        var data = [1, 2, 4, 8, 16, 8, 4, 2, 1]\n",
       "\n",
       "        var svg = d3.select(element.get(0)).append('svg')\n",
       "            .attr('width', 400)\n",
       "            .attr('height', 200);\n",
       "        svg.selectAll('circle')\n",
       "            .data(data)\n",
       "            .enter()\n",
       "            .append('circle')\n",
       "            .attr(\"cx\", function(d, i) {return 40 * (i + 1);})\n",
       "            .attr(\"cy\", function(d, i) {return 100 + 30 * (i % 3 - 1);})\n",
       "            .style(\"fill\", \"#1570a4\")\n",
       "            .transition().duration(2000)\n",
       "            .attr(\"r\", function(d) {return 2*d;})\n",
       "        ;\n",
       "    })\n",
       "})(element);"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%javascript\n",
    "(function(element) {\n",
    "    require(['d3'], function(d3) {   \n",
    "        var data = [1, 2, 4, 8, 16, 8, 4, 2, 1]\n",
    "\n",
    "        var svg = d3.select(element.get(0)).append('svg')\n",
    "            .attr('width', 400)\n",
    "            .attr('height', 200);\n",
    "        svg.selectAll('circle')\n",
    "            .data(data)\n",
    "            .enter()\n",
    "            .append('circle')\n",
    "            .attr(\"cx\", function(d, i) {return 40 * (i + 1);})\n",
    "            .attr(\"cy\", function(d, i) {return 100 + 30 * (i % 3 - 1);})\n",
    "            .style(\"fill\", \"#1570a4\")\n",
    "            .transition().duration(2000)\n",
    "            .attr(\"r\", function(d) {return 2*d;})\n",
    "        ;\n",
    "    })\n",
    "})(element);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note:\n",
    "- We leverage RequireJS to hand us a loaded `d3` library in a closure.\n",
    "- The SVG drawing is appended to the current output cell:\n",
    "    - `element` is the jQuery powered wrapper for this\n",
    "    - `element.get(0)` is the DOM node itself that can be handed to `d3.select()`\n",
    "- The `element` variable is a global variable and overwritten on each rendering of a Javascript cell, so to make sure we capture the correct `element` inside our D3.js code (which could be executed in a different context), we wrap the whole thing in a closure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## And now, the real work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unless you're just toying around with simple visualizations,\n",
    "these D3.js scripts can get very extensive,\n",
    "which is not very ideal to work on directly in a interactive notebook.\n",
    "You probably want to develop the D3.js script in an editor or IDE \n",
    "that gives you a bit more code intelligence.\n",
    "\n",
    "\n",
    "Let's cover how to get things working with an separate `.js` script\n",
    "and, additionally, with data that is initially defined or constructed in Python. \n",
    "\n",
    "\n",
    "First, we'll need these imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, Javascript, HTML\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and we set up the RequireJS search path (repeated here for completeness)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "require.config({\n",
       "    paths: { \n",
       "        d3: 'https://d3js.org/d3.v5.min'\n",
       "    }\n",
       "});"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%javascript\n",
    "require.config({\n",
    "    paths: { \n",
    "        d3: 'https://d3js.org/d3.v5.min'\n",
    "    }\n",
    "});"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say we have a script `circles.js` that implements a certain visualisation, rougly with this structure:\n",
    "\n",
    "    define('circles', ['d3'], function (d3) {\n",
    "        function draw(container, data) {\n",
    "            var svg = d3.select(container).append(\"svg\");\n",
    "            // D3.js drawing stuff here ...\n",
    "        }\n",
    "        return draw;\n",
    "    });\n",
    "\n",
    "In addition to declaring a dependency on the `d3` library like before, we now `define` a \"module\" called `circles`. \n",
    "This explicit naming is [not the standard RequireJS](http://requirejs.org/docs/api.html#modulename) way, \n",
    "but we have to do it because we will embed the Javascript code in the HTML document directly, instead of loading the file with a separate request.\n",
    "For simplicity, the defined module is just a single function (internally called `draw`), \n",
    "which expects a container to append the SVG element to and a data object.\n",
    "\n",
    "Assuming that this script `circles.js` lives alongside the notebook file, we can inject the javascript code in the notebook like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "// First undefine 'circles' so we can easily reload this file.\n",
       "require.undef('circles');\n",
       "\n",
       "define('circles', ['d3'], function (d3) {\n",
       "\n",
       "    function draw(container, data, width, height) {\n",
       "        width = width || 600;\n",
       "        height = height || 200;\n",
       "        var svg = d3.select(container).append(\"svg\")\n",
       "            .attr('width', width)\n",
       "            .attr('height', height)\n",
       "            .append(\"g\");\n",
       "\n",
       "        var x = d3.scaleLinear()\n",
       "            .domain([0, data.length - 1])\n",
       "            .range([50, width - 50]);\n",
       "\n",
       "        var circles = svg.selectAll('circle').data(data);\n",
       "\n",
       "        circles.enter()\n",
       "            .append('circle')\n",
       "            .attr(\"cx\", function (d, i) {return x(i);})\n",
       "            .attr(\"cy\", height / 2)\n",
       "            .attr(\"r\", 20)\n",
       "            .style(\"fill\", \"#1f77b4\")\n",
       "            .style(\"opacity\", 0.7)\n",
       "            .on('mouseover', function() {\n",
       "                d3.select(this)\n",
       "                  .interrupt('fade')\n",
       "                  .style('fill', '#ff850e')\n",
       "                  .style(\"opacity\", 1)\n",
       "                  .attr(\"r\", function (d) {return 1.1 * d + 10;});\n",
       "            })\n",
       "            .on('mouseout', function() {\n",
       "                d3.select(this)\n",
       "                    .transition('fade').duration(500)\n",
       "                    .style(\"fill\", \"#1f77b4\")\n",
       "                    .style(\"opacity\", 0.7)\n",
       "                    .attr(\"r\", function (d) {return d;});\n",
       "            })\n",
       "            .transition().duration(2000)\n",
       "            .attr(\"r\", function (d) {return d;});\n",
       "    }\n",
       "\n",
       "    return draw;\n",
       "});\n",
       "\n",
       "element.append('<small>&#x25C9; &#x25CB; &#x25EF; Loaded circles.js &#x25CC; &#x25CE; &#x25CF;</small>');\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Javascript(filename='circles.js')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we define some data in Python space:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [5, 10, 20, 40, 50, 30, 10, 20, 40, 10, 5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we draw the circles like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "(function(element){\n",
       "    require(['circles'], function(circles) {\n",
       "        circles(element.get(0), [5, 10, 20, 40, 50, 30, 10, 20, 40, 10, 5])\n",
       "    });\n",
       "})(element);\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Javascript(\"\"\"\n",
    "(function(element){\n",
    "    require(['circles'], function(circles) {\n",
    "        circles(element.get(0), %s)\n",
    "    });\n",
    "})(element);\n",
    "\"\"\" % json.dumps(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note:\n",
    "- We use RequireJS again to get our `circles` \"module\" which is just our drawing function\n",
    "- The container to add the drawing to is `element.get(0)` as discussed above\n",
    "- We convert the data to JSON and inject it in the `circles` function call in Javascript. For simplicity, I used basic Python string formatting with `%`, but other templating solutions are possible of course.\n",
    "- If you want multiple drawings for different data sets, you probably want to put the `Javascript` thing in a reusable function. Don't forget to return the `Javascript` object so it rendered properly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some additional practical details"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add CSS\n",
    "\n",
    "Apart from your D3.js script, you usually also want to add some custom CSS to the mix, preferably in a separate file as well. Define the CSS in a HTML file, e.g. `circles.css.html` as follows\n",
    "\n",
    "    <style>\n",
    "    svg circle {\n",
    "      stroke: #16527b;\n",
    "      stroke-width: 1px;\n",
    "    }\n",
    "    </style>\n",
    "\n",
    "and load it like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "svg circle {\n",
       "  stroke: #16527b;\n",
       "  stroke-width: 1px;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(filename='circles.css.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Easy reloading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reloading the code from the separate script while developing can be very cumbersome. RequireJS will not automatically \"reload\" a module that is already defined. For a full hard refresh you should: clear the output of the `Javascript(filename=..)` cell (or clear all outputs of the whole notebook), save the notebook and refresh the page in your browser.\n",
    "\n",
    "Luckily there is a much easier way using [`require.undef`](http://requirejs.org/docs/api.html#undef). Put it at the top of the script file, before the `define` to \"unload\" the module before redefining it again. In our example:\n",
    "\n",
    "    require.undef('circles');\n",
    "\n",
    "    define('circles', ['d3'], function (d3) {\n",
    "        // ...\n",
    "    });\n",
    "\n",
    "Now you just have the re-execute the `Javascript(filename=..)` cell and the code will be reloaded, which is a lot more intuitive during development."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## All together now"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To conclude, all the bits together, in a more compact way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, Javascript, HTML\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "require.config({paths: {d3: 'https://d3js.org/d3.v5.min'}});"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "// First undefine 'circles' so we can easily reload this file.\n",
       "require.undef('circles');\n",
       "\n",
       "define('circles', ['d3'], function (d3) {\n",
       "\n",
       "    function draw(container, data, width, height) {\n",
       "        width = width || 600;\n",
       "        height = height || 200;\n",
       "        var svg = d3.select(container).append(\"svg\")\n",
       "            .attr('width', width)\n",
       "            .attr('height', height)\n",
       "            .append(\"g\");\n",
       "\n",
       "        var x = d3.scaleLinear()\n",
       "            .domain([0, data.length - 1])\n",
       "            .range([50, width - 50]);\n",
       "\n",
       "        var circles = svg.selectAll('circle').data(data);\n",
       "\n",
       "        circles.enter()\n",
       "            .append('circle')\n",
       "            .attr(\"cx\", function (d, i) {return x(i);})\n",
       "            .attr(\"cy\", height / 2)\n",
       "            .attr(\"r\", 20)\n",
       "            .style(\"fill\", \"#1f77b4\")\n",
       "            .style(\"opacity\", 0.7)\n",
       "            .on('mouseover', function() {\n",
       "                d3.select(this)\n",
       "                  .interrupt('fade')\n",
       "                  .style('fill', '#ff850e')\n",
       "                  .style(\"opacity\", 1)\n",
       "                  .attr(\"r\", function (d) {return 1.1 * d + 10;});\n",
       "            })\n",
       "            .on('mouseout', function() {\n",
       "                d3.select(this)\n",
       "                    .transition('fade').duration(500)\n",
       "                    .style(\"fill\", \"#1f77b4\")\n",
       "                    .style(\"opacity\", 0.7)\n",
       "                    .attr(\"r\", function (d) {return d;});\n",
       "            })\n",
       "            .transition().duration(2000)\n",
       "            .attr(\"r\", function (d) {return d;});\n",
       "    }\n",
       "\n",
       "    return draw;\n",
       "});\n",
       "\n",
       "element.append('<small>&#x25C9; &#x25CB; &#x25EF; Loaded circles.js &#x25CC; &#x25CE; &#x25CF;</small>');\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "svg circle {\n",
       "  stroke: #16527b;\n",
       "  stroke-width: 1px;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Javascript(\"require.config({paths: {d3: 'https://d3js.org/d3.v5.min'}});\"))\n",
    "display(Javascript(filename=\"circles.js\"))\n",
    "display(HTML(filename=\"circles.css.html\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def draw_circles(data, width=600, height=400):\n",
    "    display(Javascript(\"\"\"\n",
    "        (function(element){\n",
    "            require(['circles'], function(circles) {\n",
    "                circles(element.get(0), %s, %d, %d);\n",
    "            });\n",
    "        })(element);\n",
    "    \"\"\" % (json.dumps(data), width, height)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "        (function(element){\n",
       "            require(['circles'], function(circles) {\n",
       "                circles(element.get(0), [10, 60, 40, 5, 30, 10], 500, 200);\n",
       "            });\n",
       "        })(element);\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "draw_circles([10, 60, 40, 5, 30, 10], width=500, height=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "- [Source of this notebook on GitHub](https://github.com/soxofaan/jupyter-playground/tree/master/jupyter-custom-d3-visualization)\n",
    "- [This notebook in NBViewer](https://nbviewer.jupyter.org/github/soxofaan/jupyter-playground/blob/master/jupyter-custom-d3-visualization/jupyter-custom-d3-visualization.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}