{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Data First Culture\n", "\n", "### [Neil D. Lawrence](http://inverseprobability.com), University of\n", "\n", "Cambridge\n", "\n", "### 2023-11-09" ], "id": "9c04f72e-b74f-4a52-a0da-0bf5ade1fdf8" }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Abstract**: Digital transformation has offered the promise of moving\n", "from a manual decision-making world to a world where decisions can be\n", "rational, data-driven and automated. The first step to digital\n", "transformation is mapping the world of atoms (material, customers,\n", "logistic networks) into the world of bits. But the real challenges may\n", "start once this is complete. In this talk we introduce the notion of\n", "‘post digital transformation’: the challenges of doing business in a\n", "digital world." ], "id": "c6f7c8d8-817a-421d-8202-d15d7af9de5b" }, { "cell_type": "markdown", "metadata": {}, "source": [ "$$\n", "$$" ], "id": "62c5ac52-018e-4980-a1f1-fbd6baa30f6a" }, { "cell_type": "markdown", "metadata": {}, "source": [ "::: {.cell .markdown}\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "" ], "id": "61ba94be-df0e-4131-93e9-05672818b530" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Complexity in Action\n", "\n", "\\[edit\\]\n", "\n", "As an exercise in understanding complexity, watch the following video.\n", "You will see the basketball being bounced around, and the players\n", "moving. Your job is to count the passes of those dressed in white and\n", "ignore those of the individuals dressed in black." ], "id": "acf32c66-460d-4f19-a085-8f9524edb308" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from IPython.lib.display import YouTubeVideo\n", "YouTubeVideo('vJG698U2Mvo')" ], "id": "86ec6f50-25bc-4d62-bce3-d14c75249f0d" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Figure: Daniel Simon’s famous illusion “monkey business”. Focus on\n", "the movement of the ball distracts the viewer from seeing other aspects\n", "of the image.\n", "\n", "In a classic study Simons and Chabris (1999) ask subjects to count the\n", "number of passes of the basketball between players on the team wearing\n", "white shirts. Fifty percent of the time, these subjects don’t notice the\n", "gorilla moving across the scene.\n", "\n", "The phenomenon of inattentional blindness is well known, e.g in their\n", "paper Simons and Charbris quote the Hungarian neurologist, Rezsö Bálint,\n", "\n", "> It is a well-known phenomenon that we do not notice anything happening\n", "> in our surroundings while being absorbed in the inspection of\n", "> something; focusing our attention on a certain object may happen to\n", "> such an extent that we cannot perceive other objects placed in the\n", "> peripheral parts of our visual field, although the light rays they\n", "> emit arrive completely at the visual sphere of the cerebral cortex.\n", ">\n", "> Rezsö Bálint 1907 (translated in Husain and Stein 1988, page 91)\n", "\n", "When we combine the complexity of the world with our relatively low\n", "bandwidth for information, problems can arise. Our focus on what we\n", "perceive to be the most important problem can cause us to miss other\n", "(potentially vital) contextual information.\n", "\n", "This phenomenon is known as selective attention or ‘inattentional\n", "blindness’." ], "id": "89336916-aa8c-4cbc-baa5-d8cf1a499bae" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from IPython.lib.display import YouTubeVideo\n", "YouTubeVideo('_oGAzq5wM_Q')" ], "id": "807388c0-e6af-42f7-9273-5cffe9e350a5" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Figure: For a longer talk on inattentional bias from Daniel Simons\n", "see this video." ], "id": "53938a29-94ec-4f3d-8ee8-f5a905168347" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup\n", "\n", "\\[edit\\]" ], "id": "1cc451bb-917f-4b21-b49b-fce7e02641d6" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "plt.rcParams.update({'font.size': 22})" ], "id": "beb48f80-06f8-4442-924e-bf61d46909ce" }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ], "id": "783c4d69-a167-463d-aadd-c0bdd7907fca" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## notutils\n", "\n", "\\[edit\\]\n", "\n", "This small package is a helper package for various notebook utilities\n", "used below.\n", "\n", "The software can be installed using" ], "id": "edc1a7a4-51d0-4683-9d85-481a8c84952a" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install notutils" ], "id": "87f6ce6f-15bc-4905-806e-f3a33666bbc8" }, { "cell_type": "markdown", "metadata": {}, "source": [ "from the command prompt where you can access your python installation.\n", "\n", "The code is also available on GitHub:\n", "\n", "\n", "Once `notutils` is installed, it can be imported in the usual manner." ], "id": "a2d1d1ca-685c-4ca1-afad-cf86bf5ce86b" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import notutils" ], "id": "adc8ee42-6757-4bc0-bfe5-059c856fc165" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## pods\n", "\n", "\\[edit\\]\n", "\n", "In Sheffield we created a suite of software tools for ‘Open Data\n", "Science’. Open data science is an approach to sharing code, models and\n", "data that should make it easier for companies, health professionals and\n", "scientists to gain access to data science techniques.\n", "\n", "You can also check this blog post on [Open Data\n", "Science](http://inverseprobability.com/2014/07/01/open-data-science).\n", "\n", "The software can be installed using" ], "id": "55ba1359-d5b8-4c67-a58a-29217c8a338f" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install pods" ], "id": "133195a1-9ad3-4270-a088-0807f634cdbb" }, { "cell_type": "markdown", "metadata": {}, "source": [ "from the command prompt where you can access your python installation.\n", "\n", "The code is also available on GitHub: \n", "\n", "Once `pods` is installed, it can be imported in the usual manner." ], "id": "7d86ff98-ec49-4d57-8d10-ea003f5bb74e" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pods" ], "id": "42653f4c-9abe-418b-9c5d-a04debf32de0" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## mlai\n", "\n", "\\[edit\\]\n", "\n", "The `mlai` software is a suite of helper functions for teaching and\n", "demonstrating machine learning algorithms. It was first used in the\n", "Machine Learning and Adaptive Intelligence course in Sheffield in 2013.\n", "\n", "The software can be installed using" ], "id": "b37b9bba-995a-4773-89b6-a889b200e217" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install mlai" ], "id": "4077b470-1b07-4f93-9dd8-672226c298f2" }, { "cell_type": "markdown", "metadata": {}, "source": [ "from the command prompt where you can access your python installation.\n", "\n", "The code is also available on GitHub: \n", "\n", "Once `mlai` is installed, it can be imported in the usual manner." ], "id": "7ff37298-6aef-4907-a655-c060e05d9f0b" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import mlai" ], "id": "c40c85db-3ff7-4f29-bbc7-9c20d919a155" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data Selective Attention Bias\n", "\n", "\\[edit\\]\n", "\n", "We are going to see how inattention biases can play out in data analysis\n", "by going through a simple example. The analysis involves body mass index\n", "and activity information." ], "id": "de5e00f4-d6de-4987-8455-8f9977f3c7f4" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## BMI Steps Data\n", "\n", "\\[edit\\]\n", "\n", "The BMI Steps example is taken from Yanai and Lercher (2020). We are\n", "given a data set of body-mass index measurements against step counts.\n", "For convenience we have packaged the data so that it can be easily\n", "downloaded." ], "id": "901e4057-218b-451f-81b0-4953cb025cda" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pods" ], "id": "a8a94716-27b4-4f67-a9ed-bcdb6cc62186" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data = pods.datasets.bmi_steps()\n", "X = data['X'] \n", "y = data['Y']" ], "id": "a96bb2da-37f3-44f7-a424-72e2a998579b" }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is good practice to give our variables interpretable names so that\n", "the analysis may be clearly understood by others. Here the `steps` count\n", "is the first dimension of the covariate, the `bmi` is the second\n", "dimension and the `gender` is stored in `y` with `1` for female and `0`\n", "for male." ], "id": "e420a50d-3677-4301-b038-981bc882c8ea" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "steps = X[:, 0]\n", "bmi = X[:, 1]\n", "gender = y[:, 0]" ], "id": "4733a312-4bbc-4159-b088-37468f75bb94" }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can check the mean steps and the mean of the BMI." ], "id": "bf02a07f-0218-46ec-b131-7862d2a7fd3a" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print('Steps mean is {mean}.'.format(mean=steps.mean()))" ], "id": "7852da3b-78f6-46f0-9eb1-bc854ccbfab5" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print('BMI mean is {mean}.'.format(mean=bmi.mean()))" ], "id": "14f50fe9-1140-46f0-8828-d216ae40624b" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## BMI Steps Data Analysis\n", "\n", "\\[edit\\]\n", "\n", "We can also separate out the means from the male and female populations.\n", "In python this can be done by setting male and female indices as\n", "follows." ], "id": "9d8f2666-e14f-470b-ab6d-4171d986a4ba" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "male_ind = (gender==0)\n", "female_ind = (gender==1)" ], "id": "9184df5c-0c55-456d-9af5-3095ba787d05" }, { "cell_type": "markdown", "metadata": {}, "source": [ "And now we can extract the variables for the two populations." ], "id": "d0b85700-37dd-4523-b40f-8506470e6210" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "male_steps = steps[male_ind]\n", "male_bmi = bmi[male_ind]" ], "id": "aaf5fbd2-2416-407c-8a8b-e8710d984fa7" }, { "cell_type": "markdown", "metadata": {}, "source": [ "And as before we compute the mean." ], "id": "bff2c1f1-aea2-4401-b44a-0e48698916b4" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print('Male steps mean is {mean}.'.format(mean=male_steps.mean()))" ], "id": "e3114fc5-377a-4fd7-959f-7aa3b56d0c32" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print('Male BMI mean is {mean}.'.format(mean=male_bmi.mean()))" ], "id": "11712919-9dcb-49d1-bdac-3c570838a57f" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Similarly, we can get the same result for the female portion of the\n", "populaton." ], "id": "64df8f3e-cb2a-4776-b106-885021446c77" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "female_steps = steps[female_ind]\n", "female_bmi = bmi[female_ind]" ], "id": "cb4d2a7f-5159-4d82-9c47-6ad26ed841ee" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print('Female steps mean is {mean}.'.format(mean=female_steps.mean()))" ], "id": "79857f3d-9737-4947-a0ed-4b59e9fa924a" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print('Female BMI mean is {mean}.'.format(mean=female_bmi.mean()))" ], "id": "093709ae-df3e-4ac9-aa33-48693b25ea7c" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Interesting, the female BMI average is slightly higher than the male BMI\n", "average. The number of steps in the male group is higher than that in\n", "the female group. Perhaps the steps and the BMI are anti-correlated. The\n", "more steps, the lower the BMI.\n", "\n", "Python provides a statistics package. We’ll import this in `python` so\n", "that we can try and understand the correlation between the `steps` and\n", "the `BMI`." ], "id": "a06ad08a-a34a-445c-bc70-b85c76cce476" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from scipy.stats import pearsonr" ], "id": "09ea55da-901a-4313-b79a-e290753274f7" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "corr, _ = pearsonr(steps, bmi)\n", "print(\"Pearson's overall correlation: {corr}\".format(corr=corr))" ], "id": "52dafcde-a114-4787-ad11-78e868450334" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\n", "\n", "male_corr, _ = pearsonr(male_steps, male_bmi)\n", "print(\"Pearson's correlation for males: {corr}\".format(corr=male_corr))" ], "id": "5c6cb4d8-76a5-44ef-9353-572da9f91b6f" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\n", "female_corr, _ = pearsonr(female_steps, female_bmi)\n", "print(\"Pearson's correlation for females: {corr}\".format(corr=female_corr))" ], "id": "79956b9e-af5a-4097-9606-701d80765cfc" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import mlai.plot as plot\n", "import mlai\n", "import matplotlib.pyplot as plt" ], "id": "efd76521-fc23-4cf0-bbd8-cc4d1888537a" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=plot.big_wide_figsize)\n", "_ = ax.plot(X[male_ind, 0], X[male_ind, 1], 'g.',markersize=10)\n", "_ = ax.plot(X[female_ind, 0], X[female_ind, 1], 'r.',markersize=10)\n", "_ = ax.set_xlabel('steps', fontsize=20)\n", "_ = ax.set_ylabel('BMI', fontsize=20)\n", "xlim = (0, 15000)\n", "ylim = (15, 32.5)\n", "ax.set_xlim(xlim)\n", "ax.set_ylim(ylim)\n", "mlai.write_figure(filename='bmi-steps.svg',\n", " directory='./datasets',\n", " transparent=True)" ], "id": "39ab969b-fa89-4548-b200-0a9cb9be4e31" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## A Hypothesis as a Liability\n", "\n", "This analysis is from an article titled “A Hypothesis as a Liability”\n", "(Yanai and Lercher, 2020), they start their article with the following\n", "quite from Herman Hesse.\n", "\n", "> ” ‘When someone seeks,’ said Siddhartha, ‘then it easily happens that\n", "> his eyes see only the thing that he seeks, and he is able to find\n", "> nothing, to take in nothing. \\[…\\] Seeking means: having a goal. But\n", "> finding means: being free, being open, having no goal.’ ”\n", ">\n", "> Hermann Hesse\n", "\n", "Their idea is that having a hypothesis can constrain our thinking.\n", "However, in answer to their paper Felin et al. (2021) argue that some\n", "form of hypothesis is always necessary, suggesting that a hypothesis\n", "*can* be a liability\n", "\n", "My view is captured in the introductory chapter to an edited volume on\n", "computational systems biology that I worked on with Mark Girolami,\n", "Magnus Rattray and Guido Sanguinetti.\n", "\n", "\n", "\n", "Figure: Quote from Lawrence (2010) highlighting the importance of\n", "interaction between data and hypothesis.\n", "\n", "Popper nicely captures the interaction between hypothesis and data by\n", "relating it to the chicken and the egg. The important thing is that\n", "these two co-evolve." ], "id": "dd3d27a7-b7ef-4909-a702-13eae4e2fb5b" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Number Theatre\n", "\n", "\\[edit\\]\n", "\n", "Unfortunately, we don’t always have time to wait for this process to\n", "converge to an answer we can all rely on before a decision is required.\n", "\n", "Not only can we be misled by data before a decision is made, but\n", "sometimes we can be misled by data to justify the making of a decision.\n", "David Spiegelhalter refers to the phenomenon of “Number Theatre” in a\n", "conversation with Andrew Marr from May 2020 on the presentation of data." ], "id": "e4f525be-ce43-4bca-8a5e-da2a72ab6340" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from IPython.lib.display import YouTubeVideo\n", "YouTubeVideo('9388XmWIHXg')" ], "id": "d6fc0be8-c7c9-41bc-8f54-035e0eb4d3f5" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Figure: Professor Sir David Spiegelhalter on Andrew Marr on 10th May\n", "2020 speaking about some of the challengers around data, data\n", "presentation, and decision making in a pandemic. David mentions number\n", "theatre at 9 minutes 10 seconds.\n", "\n", "" ], "id": "6f373b65-b1c2-4fb2-a9b8-95cb5e8a19f8" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data Theatre\n", "\n", "Data Theatre exploits data inattention bias to present a particular view\n", "on events that may misrepresents through selective presentation.\n", "Statisticians are one of the few groups that are trained with a\n", "sufficient degree of data skepticism. But it can also be combatted\n", "through ensuring there are domain experts present, and that they can\n", "speak freely.\n", "\n", "\n", "\n", "Figure: The phenomenon of number theatre or *data theatre* was\n", "described by David Spiegelhalter and is nicely summarized by Martin\n", "Robbins in this sub-stack article\n", "." ], "id": "9c093cd8-3bc2-4388-8eef-fd8a25142597" }, { "cell_type": "markdown", "metadata": {}, "source": [ "# The Art of Statistics\n", "\n", "\\[edit\\]\n", "\n", "The statistician’s craft is based on humility in front of data and\n", "developing the appropriate skeptical thinking around conclusions from\n", "data. The best book I’ve seen for developing that sense is Sir David\n", "Spiegelhalter’s *Art of Statistics*.\n", "\n", "
\n", "\n", "\n", " \n", "\n", "\n", "\n", " \n", "\n", "\n", "\n", "David Spiegelhalter\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "\n", "\n", "Figure: [The Art of Statistics by David\n", "Spiegelhalter](https://www.amazon.co.uk/Art-Statistics-Learning-Pelican-Books-ebook/dp/B07HQDJD99)\n", "is an excellent read on the pitfalls of data interpretation.\n", "\n", "David’s book (Spiegelhalter, 2019) brings important examples from\n", "statistics to life in an intelligent and entertaining way. It is highly\n", "readable and gives an opportunity to fast-track towards the important\n", "skill of data-skepticism that is the mark of a professional\n", "statistician.\n", "\n", "\n", "\n", "" ], "id": "e92f60fb-1731-4728-b2c7-d4678a0aeb38" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Information and Embodiment\n", "\n", "\\[edit\\]\n", "\n", "
\n", "\n", "\n", "\n", "
\n", "
\n", "\n", "*Claude Shannon*\n", "\n", "
\n", "\n", "Figure: Claude Shannon (1916-2001)\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "bits/min\n", "\n", "\n", "\n", "billions\n", "\n", "\n", "\n", "2,000\n", "\n", "
\n", "\n", "billion
calculations/s\n", "\n", "		\n", "\n", "~100\n", "\n", "\n", "\n", "a billion\n", "\n", "
\n", "\n", "embodiment\n", "\n", "\n", "\n", "20 minutes\n", "\n", "\n", "\n", "5 billion years\n", "\n", "
\n", "\n", "Figure: Embodiment factors are the ratio between our ability to\n", "compute and our ability to communicate. Relative to the machine we are\n", "also locked in. In the table we represent embodiment as the length of\n", "time it would take to communicate one second’s worth of computation. For\n", "computers it is a matter of minutes, but for a human, it is a matter of\n", "thousands of millions of years." ], "id": "4c0928ae-912f-4430-8a74-a8fb45580166" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Bandwidth Constrained Conversations\n", "\n", "\\[edit\\]" ], "id": "00d4846d-a16e-4a78-a8b8-04e8fc49d7c2" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import notutils as nu\n", "from ipywidgets import IntSlider" ], "id": "6b35ca84-d3e3-4684-af27-5ae7059605b8" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import notutils as nu" ], "id": "ea5e6487-01df-497f-8740-b778eba27cd3" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "nu.display_plots('anne-bob-conversation{sample:0>3}.svg', \n", " 'https://inverseprobability.com./slides/diagrams/', sample=IntSlider(0, 0, 7, 1))" ], "id": "b832a5f4-8bf7-4ddb-8112-305d0100d633" }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "Figure: Conversation relies on internal models of other\n", "individuals.\n", "\n", "\n", "\n", "Figure: Misunderstanding of context and who we are talking to leads\n", "to arguments.\n", "\n", "Embodiment factors imply that, in our communication between humans, what\n", "is *not* said is, perhaps, more important than what is said. To\n", "communicate with each other we need to have a model of who each of us\n", "are.\n", "\n", "To aid this, in society, we are required to perform roles. Whether as a\n", "parent, a teacher, an employee or a boss. Each of these roles requires\n", "that we conform to certain standards of behaviour to facilitate\n", "communication between ourselves.\n", "\n", "Control of self is vitally important to these communications.\n", "\n", "The high availability of data available to humans undermines\n", "human-to-human communication channels by providing new routes to\n", "undermining our control of self.\n", "\n", "The consequences between this mismatch of power and delivery are to be\n", "seen all around us. Because, just as driving an F1 car with bicycle\n", "wheels would be a fine art, so is the process of communication between\n", "humans.\n", "\n", "If I have a thought and I wish to communicate it, I first need to have a\n", "model of what you think. I should think before I speak. When I speak,\n", "you may react. You have a model of who I am and what I was trying to\n", "say, and why I chose to say what I said. Now we begin this dance, where\n", "we are each trying to better understand each other and what we are\n", "saying. When it works, it is beautiful, but when mis-deployed, just like\n", "a badly driven F1 car, there is a horrible crash, an argument." ], "id": "d19b6ff7-bedf-4888-8da7-ce689dcbc5a9" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Heider and Simmel (1944)\n", "\n", "\\[edit\\]" ], "id": "e0239ca7-f005-4e22-96f7-e0d5f24c5c88" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from IPython.lib.display import YouTubeVideo\n", "YouTubeVideo('8FIEZXMUM2I')" ], "id": "9e83e05a-b0ae-4e66-95f7-04c4e111494d" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Figure: Fritz Heider and Marianne Simmel’s video of shapes from\n", "Heider and Simmel (1944).\n", "\n", "[Fritz Heider](https://en.wikipedia.org/wiki/Fritz_Heider) and [Marianne\n", "Simmel](https://en.wikipedia.org/wiki/Marianne_Simmel)’s experiments\n", "with animated shapes from 1944 (Heider and Simmel, 1944). Our\n", "interpretation of these objects as showing motives and even emotion is a\n", "combination of our desire for narrative, a need for understanding of\n", "each other, and our ability to empathize. At one level, these are\n", "crudely drawn objects, but in another way, the animator has communicated\n", "a story through simple facets such as their relative motions, their\n", "sizes and their actions. We apply our psychological representations to\n", "these faceless shapes to interpret their actions.\n", "\n", "See also a recent review paper on Human Cooperation by Henrich and\n", "Muthukrishna (2021)." ], "id": "bfa34a34-2ced-4b06-9889-22f542839e36" }, { "cell_type": "markdown", "metadata": {}, "source": [ "### A Six Word Novel\n", "\n", "\\[edit\\]\n", "\n", "\n", "\n", "Figure: Consider the six-word novel, apocryphally credited to Ernest\n", "Hemingway, “For sale: baby shoes, never worn”. To understand what that\n", "means to a human, you need a great deal of additional context. Context\n", "that is not directly accessible to a machine that has not got both the\n", "evolved and contextual understanding of our own condition to realize\n", "both the implication of the advert and what that implication means\n", "emotionally to the previous owner.\n", "\n", "But this is a very different kind of intelligence than ours. A computer\n", "cannot understand the depth of the Ernest Hemingway’s apocryphal\n", "six-word novel: “For Sale, Baby Shoes, Never worn”, because it isn’t\n", "equipped with that ability to model the complexity of humanity that\n", "underlies that statement." ], "id": "ff86ab50-b460-442e-8d91-11037e5b8163" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Computer Conversations\n", "\n", "\\[edit\\]" ], "id": "31245f0f-3ad8-48ab-8f66-8848834ccfa1" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import notutils as nu\n", "from ipywidgets import IntSlider" ], "id": "a9b88150-516c-49ea-b081-982f718d4c17" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import notutils as nu" ], "id": "3cfc8c28-fead-4f1b-91c4-f945a84dbfaa" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "nu.display_plots('anne-bob-conversation{sample:0>3}.svg', \n", " 'https://inverseprobability.com./slides/diagrams/', sample=IntSlider(0, 0, 7, 1))" ], "id": "0c0407f0-987f-420e-a030-2f5b3d94c9a2" }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "Figure: Conversation relies on internal models of other\n", "individuals.\n", "\n", "\n", "\n", "Figure: Misunderstanding of context and who we are talking to leads\n", "to arguments.\n", "\n", "Similarly, we find it difficult to comprehend how computers are making\n", "decisions. Because they do so with more data than we can possibly\n", "imagine.\n", "\n", "In many respects, this is not a problem, it’s a good thing. Computers\n", "and us are good at different things. But when we interact with a\n", "computer, when it acts in a different way to us, we need to remember\n", "why.\n", "\n", "Just as the first step to getting along with other humans is\n", "understanding other humans, so it needs to be with getting along with\n", "our computers.\n", "\n", "Embodiment factors explain why, at the same time, computers are so\n", "impressive in simulating our weather, but so poor at predicting our\n", "moods. Our complexity is greater than that of our weather, and each of\n", "us is tuned to read and respond to one another.\n", "\n", "Their intelligence is different. It is based on very large quantities of\n", "data that we cannot absorb. Our computers don’t have a complex internal\n", "model of who we are. They don’t understand the human condition. They are\n", "not tuned to respond to us as we are to each other.\n", "\n", "Embodiment factors encapsulate a profound thing about the nature of\n", "humans. Our locked in intelligence means that we are striving to\n", "communicate, so we put a lot of thought into what we’re communicating\n", "with. And if we’re communicating with something complex, we naturally\n", "anthropomorphize them.\n", "\n", "We give our dogs, our cats, and our cars human motivations. We do the\n", "same with our computers. We anthropomorphize them. We assume that they\n", "have the same objectives as us and the same constraints. They don’t.\n", "\n", "This means, that when we worry about artificial intelligence, we worry\n", "about the wrong things. We fear computers that behave like more powerful\n", "versions of ourselves that will struggle to outcompete us.\n", "\n", "In reality, the challenge is that our computers cannot be human enough.\n", "They cannot understand us with the depth we understand one another. They\n", "drop below our cognitive radar and operate outside our mental models.\n", "\n", "The real danger is that computers don’t anthropomorphize. They’ll make\n", "decisions in isolation from us without our supervision because they\n", "can’t communicate truly and deeply with us." ], "id": "f5af8c6c-e895-42ec-8a9b-74e20265a530" }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Evolved Relationship with Information\n", "\n", "\\[edit\\]\n", "\n", "The high bandwidth of computers has resulted in a close relationship\n", "between the computer and data. Large amounts of information can flow\n", "between the two. The degree to which the computer is mediating our\n", "relationship with data means that we should consider it an intermediary.\n", "\n", "Originally our low bandwidth relationship with data was affected by two\n", "characteristics. Firstly, our tendency to over-interpret driven by our\n", "need to extract as much knowledge from our low bandwidth information\n", "channel as possible. Secondly, by our improved understanding of the\n", "domain of *mathematical* statistics and how our cognitive biases can\n", "mislead us.\n", "\n", "With this new set up there is a potential for assimilating far more\n", "information via the computer, but the computer can present this to us in\n", "various ways. If its motives are not aligned with ours then it can\n", "misrepresent the information. This needn’t be nefarious it can be simply\n", "because of the computer pursuing a different objective from us. For\n", "example, if the computer is aiming to maximize our interaction time that\n", "may be a different objective from ours which may be to summarize\n", "information in a representative manner in the *shortest* possible length\n", "of time.\n", "\n", "For example, for me, it was a common experience to pick up my telephone\n", "with the intention of checking when my next appointment was, but to soon\n", "find myself distracted by another application on the phone and end up\n", "reading something on the internet. By the time I’d finished reading, I\n", "would often have forgotten the reason I picked up my phone in the first\n", "place.\n", "\n", "There are great benefits to be had from the huge amount of information\n", "we can unlock from this evolved relationship between us and data. In\n", "biology, large scale data sharing has been driven by a revolution in\n", "genomic, transcriptomic and epigenomic measurement. The improved\n", "inferences that can be drawn through summarizing data by computer have\n", "fundamentally changed the nature of biological science, now this\n", "phenomenon is also influencing us in our daily lives as data measured by\n", "*happenstance* is increasingly used to characterize us.\n", "\n", "Better mediation of this flow requires a better understanding of\n", "human-computer interaction. This in turn involves understanding our own\n", "intelligence better, what its cognitive biases are and how these might\n", "mislead us.\n", "\n", "For further thoughts see Guardian article on [marketing in the internet\n", "era](https://www.theguardian.com/media-network/2015/jul/23/data-driven-economy-marketing)\n", "from 2015.\n", "\n", "You can also check my blog post on [System\n", "Zero](http://inverseprobability.com/2015/12/04/what-kind-of-ai). This\n", "was also written in 2015." ], "id": "3a92b487-ad36-48a9-ad08-7f63f72c3b01" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## New Flow of Information\n", "\n", "\\[edit\\]\n", "\n", "Classically the field of statistics focused on mediating the\n", "relationship between the machine and the human. Our limited bandwidth of\n", "communication means we tend to over-interpret the limited information\n", "that we are given, in the extreme we assign motives and desires to\n", "inanimate objects (a process known as anthropomorphizing). Much of\n", "mathematical statistics was developed to help temper this tendency and\n", "understand when we are valid in drawing conclusions from data.\n", "\n", "\n", "\n", "Figure: The trinity of human, data, and computer, and highlights the\n", "modern phenomenon. The communication channel between computer and data\n", "now has an extremely high bandwidth. The channel between human and\n", "computer and the channel between data and human is narrow. New direction\n", "of information flow, information is reaching us mediated by the\n", "computer. The focus on classical statistics reflected the importance of\n", "the direct communication between human and data. The modern challenges\n", "of data science emerge when that relationship is being mediated by the\n", "machine.\n", "\n", "Data science brings new challenges. In particular, there is a very large\n", "bandwidth connection between the machine and data. This means that our\n", "relationship with data is now commonly being mediated by the machine.\n", "Whether this is in the acquisition of new data, which now happens by\n", "happenstance rather than with purpose, or the interpretation of that\n", "data where we are increasingly relying on machines to summarize what the\n", "data contains. This is leading to the emerging field of data science,\n", "which must not only deal with the same challenges that mathematical\n", "statistics faced in tempering our tendency to over interpret data but\n", "must also deal with the possibility that the machine has either\n", "inadvertently or maliciously misrepresented the underlying data.\n", "\n", "With new capabilities becoming available through very large generational\n", "AI models, we can imagine different interfaces with that information,\n", "but the potential for manipulation is if anything even greater.\n", "\n", "\n", "\n", "Figure: New capabilities in language generation offer the tantalising\n", "possibility for a better interface between the machine and it’s\n", "understanding of data but characterising that communication channel\n", "remains a major challenge." ], "id": "f6979d2b-f35e-409b-baf7-8fbcf4d26ddc" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Revolution\n", "\n", "\\[edit\\]\n", "\n", "Arguably the information revolution we are experiencing is unprecedented\n", "in history. But changes in the way we share information have a long\n", "history. Over 5,000 years ago in the city of Uruk, on the banks of the\n", "Euphrates, communities which relied on the water to irrigate their corps\n", "developed an approach to recording transactions in clay. Eventually the\n", "system of recording system became sophisticated enough that their oral\n", "histories could be recorded in the form of the first epic: Gilgamesh.\n", "\n", "\n", "\n", "Figure: Chicago Stone, side 2, recording sale of a number of fields,\n", "probably from Isin, Early Dynastic Period, c. 2600 BC, black basalt\n", "\n", "It was initially develoepd for people as a recordd of who owed what to\n", "whom, expanding individuals’ capacity to remember. But over a five\n", "hundred year period writing evolved to become a tool for literature as\n", "well. More pithily put, writing was invented by accountants not poets\n", "(see e.g. [this piece by Tim\n", "Harford](https://www.bbc.co.uk/news/business-39870485)).\n", "\n", "In some respects today’s revolution is different, because it involves\n", "also the creation of stories as well as their curation. But in some\n", "fundamental ways we can see what we have produced as another tool for us\n", "in the information revolution." ], "id": "35ae5f8b-8f02-4e05-8cad-eb77d92565d4" }, { "cell_type": "markdown", "metadata": {}, "source": [ "# The Future of Professions\n", "\n", "\\[edit\\]\n", "\n", "\n", "\n", "Figure: [The Future of\n", "Professions](https://www.amazon.co.uk/Future-Professions-Technology-Transform-Experts/dp/0198713398)\n", "(Susskind and Susskind, 2015) is a 2015 book focussed on how the next\n", "wave of technology revolution is going to effect the professions.\n", "\n", "And this is very likely true, but in practice we know that even if the\n", "disruption is being felt initially by the professional classes, those\n", "groups tend to be protected by their ability to adapt, which is\n", "correlated with higher education.\n", "\n", "Whether this remains true this time is another question. I’m\n", "particularly struck by the “convergent evolution” of ChatGPT. The model\n", "is trained by reinforcement learning with feedback provided by people.\n", "ChatGPT’s answers are highly plausible, make use of sophisticated\n", "language in an intelligent sounding way and are often incorrect. I’m\n", "struck by the similarity to fresh Oxbridge graduates. I wonder if this\n", "is also an example of convergent evolution.\n", "\n", "" ], "id": "4c76b970-875c-4825-b14f-2d7e427bacfb" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Public Research\n", "\n", "\\[edit\\]\n", "\n", "\n", "\n", "Figure: The Royal Society comissioned [public research from\n", "Mori](https://royalsociety.org/-/media/policy/projects/machine-learning/publications/public-views-of-machine-learning-ipsos-mori.pdf)\n", "as part of the machine learning review.\n", "\n", "\n", "\n", "Figure: One of the questions focussed on machine learning\n", "applications.\n", "\n", "\n", "\n", "Figure: The public were broadly supportive of a range of application\n", "areas.\n", "\n", "\n", "\n", "Figure: But they failed to see the point in AI’s that could produce\n", "poetry.\n", "\n", "\n", "" ], "id": "e65559f5-3706-4f4c-ae3e-4cce7b3cb4b9" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Probability Conversations\n", "\n", "\\[edit\\]\n", "\n", "\n", "\n", "Figure: The focus so far has been on reducing uncertainty to a few\n", "representative values and sharing numbers with human beings. We forget\n", "that most people can be confused by basic probabilities for example the\n", "prosecutor’s fallacy.\n", "\n", "In practice we know that probabilities can be very unintuitive, for\n", "example in court there is a fallacy known as the “prosecutor’s fallacy”\n", "that confuses conditional probabilities. This can cause problems in jury\n", "trials (Thompson, 1989)." ], "id": "d6428fb9-a1a1-42d5-9c1f-00c96e48fac5" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Networked Interactions\n", "\n", "Our modern society intertwines the machine with human interactions. The\n", "key question is who has control over these interfaces between humans and\n", "machines.\n", "\n", "\n", "\n", "Figure: Humans and computers interacting should be a major focus of\n", "our research and engineering efforts.\n", "\n", "So the real challenge that we face for society is understanding which\n", "systemic interventions will encourage the right interactions between the\n", "humans and the machine at all of these interfaces.\n", "\n", "\n", "\n", "Figure: Humans use culture, facts and ‘artefacts’ to communicate." ], "id": "64de95cd-6903-48a6-a50c-d350958c614e" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Human Analogue Machine\n", "\n", "\\[edit\\]\n", "\n", "The machine learning systems we have built today that can reconstruct\n", "human text, or human classification of images, necessarily must have\n", "some aspects to them that are analagous to our understanding. As MacKay\n", "suggests the brain is neither a digital or an analogue computer, and the\n", "same can be said of the modern neural network systems that are being\n", "tagged as “artificial intelligence”.\n", "\n", "I believe a better term for them is “human-analogue machines”, because\n", "what we have built is not a system that can make intelligent decisions\n", "from first principles (a rational approach) but one that observes how\n", "humans have made decisions through our data and reconstructs that\n", "process. Machine learning is more empiricist than rational, but now we n\n", "empirical approach that distils our evolved intelligence.\n", "\n", "\n", "\n", "Figure: The human analogue machine creates a feature space which is\n", "analagous to that we use to reason, one way of doing this is to have a\n", "machine attempt to compress all human generated text in an\n", "auto-regressive manner." ], "id": "ce47436d-1af8-46e5-b8ed-fe2efb5b9ba8" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Further Reading\n", "\n", "- Chapter 1 of Lawrence (2024)\n", "\n", "- Chapter 11 of Lawrence (2024)\n", "\n", "- Chapter 8 of Lawrence (2024)\n", "\n", "The perils of developing this capability include counterfeit people, a\n", "notion that the philosopher [Daniel Dennett has described in *The\n", "Atlantic*](https://www.theatlantic.com/technology/archive/2023/05/problem-counterfeit-people/674075/).\n", "This is where computers can represent themselves as human and fool\n", "people into doing things on that basis." ], "id": "5a6b495b-e01f-4c4e-b815-8ddc425b400c" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## LLM Conversations\n", "\n", "\\[edit\\]\n", "\n", "\n", "\n", "Figure: The focus so far has been on reducing uncertainty to a few\n", "representative values and sharing numbers with human beings. We forget\n", "that most people can be confused by basic probabilities for example the\n", "prosecutor’s fallacy." ], "id": "d9907ce4-bf6d-4c9c-9f84-7fe3e3420957" }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from IPython.lib.display import YouTubeVideo\n", "YouTubeVideo('0sJjdxn5kcI')" ], "id": "c41be102-4018-4bd4-8709-8fea6f1f49d7" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Figure: The Inner Monologue paper suggests using LLMs for robotic\n", "planning (Huang et al., 2023).\n", "\n", "By interacting directly with machines that have an understanding of\n", "human cultural context, it should be possible to share the nature of\n", "uncertainty in the same way humans do. See for example the paper [Inner\n", "Monologue: Embodied Reasoning through\n", "Planning](https://innermonologue.github.io/) Huang et al. (2023)." ], "id": "4393589c-1ddd-4368-aab1-84d567e5ebec" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## HAM\n", "\n", "\\[edit\\]\n", "\n", "\n", "\n", "Figure: The trinity of human, data, and computer, and highlights the\n", "modern phenomenon. The communication channel between computer and data\n", "now has an extremely high bandwidth. The channel between human and\n", "computer and the channel between data and human is narrow. New direction\n", "of information flow, information is reaching us mediated by the\n", "computer. The focus on classical statistics reflected the importance of\n", "the direct communication between human and data. The modern challenges\n", "of data science emerge when that relationship is being mediated by the\n", "machine.\n", "\n", "\n", "\n", "Figure: The HAM now sits between us and the traditional digital\n", "computer." ], "id": "366b0f0f-c40d-4669-aff4-a22c652c8458" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## AI@Cam\n", "\n", "\\[edit\\]\n", "\n", "AI@Cam is a new flagship University mission that seeks to address these\n", "challenges. It recognises that development of safe and effective\n", "AI-enabled innovations requires this mix of expertise from across\n", "research domains, businesses, policy-makers, civill society, and from\n", "affected communities. AI@Cam is setting out a vision for AI-enabled\n", "innovation that benefits science, citizens and society.\n", "\n", "This vision will be achieved through leveraging the University’s vibrant\n", "interdisciplinary research community. AI@Cam will form partnerships\n", "between researchers, practitioners, and affected communities that embed\n", "equity and inclusion. It will develop new platforms for innovation and\n", "knowledge transfer. It will deliver innovative interdisciplinary\n", "teaching and learning for students, researchers, and professionals. It\n", "will build strong connections between the University and national AI\n", "priorities.\n", "\n", "The University operates as both an engine of AI-enabled innovation and\n", "steward of those innovations.\n", "\n", "AI is not a universal remedy. It is a set of tools, techniques and\n", "practices that correctly deployed can be leveraged to deliver societal\n", "benefit and mitigate social harm.\n", "\n", "In that sense AI@Cam’s mission is close in spirit to that of Panacea’s\n", "elder sister Hygeia. It is focussed on building and maintaining the\n", "hygiene of a robust and equitable AI research ecosystem.\n", "\n", "\n", "\n", "Figure:\n", "[AI@Cam](https://www.cam.ac.uk/system/files/aicam_review_april22.pdf)\n", "is a Flagship Programme that supports AI research [across the\n", "University](https://www.cam.ac.uk/stories/ai-at-cam).\n", "\n", "Finally, we are working across the University to empower the diversity\n", "ofexpertise and capability we have to focus on these broad societal\n", "problems. We will recently launched AI@Cam with a vision document that\n", "outlines these challenges for the University.\n", "\n", "The initiative was funded in November 2022 where a £5M investment from\n", "the University was secured.\n", "\n", "Progress so far:\n", "\n", "- Developing the vision\n", " - Engaged over 100 members of the University community across 30\n", " departments/institutes, start-ups, and large businesses.\n", " - Supported 6 new funding bids" ], "id": "737c3897-34a6-4742-9c78-88c75f578eb7" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conclusion\n", "\n", "\\[edit\\]\n", "\n", "See the Gorilla *don’t* be the Gorilla.\n", "\n", "\n", "\n", "Figure: A famous quote from Mike Tyson before his fight with Evander\n", "Holyfield: “Everyone has a plan until they get punched in the mouth”.\n", "Don’t let the gorilla punch you in the mouth. See the gorilla, but don’t\n", "be the gorilla. Photo credit:\n", "" ], "id": "bf8d78b6-578a-4288-a8b9-29e40908110e" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Thanks!\n", "\n", "For more information on these subjects and more you might want to check\n", "the following resources.\n", "\n", "- twitter: [@lawrennd](https://twitter.com/lawrennd)\n", "- podcast: [The Talking Machines](http://thetalkingmachines.com)\n", "- newspaper: [Guardian Profile\n", " Page](http://www.theguardian.com/profile/neil-lawrence)\n", "- blog:\n", " [http://inverseprobability.com](http://inverseprobability.com/blog.html)" ], "id": "99f57ca7-fe99-4e38-a9cf-7ede848a04e5" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Felin, T., Koenderink, J., Krueger, J.I., Noble, D., Ellis, G.F.R.,\n", "2021. The data-hypothesis relationship. Genome Biology 22.\n", "\n", "\n", "Heider, F., Simmel, M., 1944. An experimental study of apparent\n", "behavior. The American Journal of Psychology 57, 243–259.\n", "\n", "\n", "Henrich, J., Muthukrishna, M., 2021. The origins and psychology of human\n", "cooperation. Annual Review of Psychology 72, 207–240.\n", "\n", "\n", "Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng,\n", "A., Tompson, J., Mordatch, I., Chebotar, Y., Sermanet, P., Jackson, T.,\n", "Brown, N., Luu, L., Levine, S., Hausman, K., ichter, brian, 2023. [Inner\n", "monologue: Embodied reasoning through planning with language\n", "models](https://proceedings.mlr.press/v205/huang23c.html), in: Liu, K.,\n", "Kulic, D., Ichnowski, J. (Eds.), Proceedings of the 6th Conference on\n", "Robot Learning, Proceedings of Machine Learning Research. PMLR, pp.\n", "1769–1782.\n", "\n", "Lawrence, N.D., 2024. The atomic human: Understanding ourselves in the\n", "age of AI. Allen Lane.\n", "\n", "Lawrence, N.D., 2010. Introduction to learning and inference in\n", "computational systems biology.\n", "\n", "Simons, D.J., Chabris, C.F., 1999. Gorillas in our midst: Sustained\n", "inattentional blindness for dynamic events. Perception 28, 1059–1074.\n", "\n", "\n", "Spiegelhalter, D.J., 2019. The art of statistics. Pelican.\n", "\n", "Susskind, R.E., Susskind, D., 2015. The future of the professions: How\n", "technology will transform the work of human experts. Oxford University\n", "Press.\n", "\n", "Thompson, W.C., 1989. [Are juries competent to evaluate statistical\n", "evidence?](http://www.jstor.org/stable/1191906) Law and Contemporary\n", "Problems 52, 9–41.\n", "\n", "Yanai, I., Lercher, M., 2020. A hypothesis is a liability. Genome\n", "Biology 21." ], "id": "f1d982e9-8836-4c0e-b07d-244428f4ac1a" } ], "nbformat": 4, "nbformat_minor": 5, "metadata": {} }