{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "Introduction to Machine Intelligence\n", "====================================\n", "\n", "### [Neil D. Lawrence](http://inverseprobability.com), University of\n", "\n", "Cambridge\n", "\n", "### 2021-02-02" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Abstract**: With breakthroughs in understanding images, translating\n", "language, transcribing speech artificial intelligence promises to\n", "revolutionise the technological landscape. Machine learning algorithms\n", "are able to convert unstructured data into actionable knowledge. With\n", "the increasing impact of these technologies, society’s interest is also\n", "growing. The word intelligence conjures notions of human-like\n", "capabilities. But are we really on the cusp of creating machines that\n", "match us? We associate intelligence with knowledge, but in this talk I\n", "will argue that the true marvel of our intelligence is the way it deals\n", "with ignorance. Despite the large strides forward we have made, I will\n", "argue that we have a long way to go to deliver on the promise of\n", "artificial intelligence. And it is a journey that science and artificial\n", "inteligence need to take together." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$$\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "GREAT AI FALLACY\n", "\n", "UNPRECEDENTED COMBINATION OF SCIENCE, SOCIAL SCIENCE, ETC REQUIRED TO\n", "DELIVER" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "SUPERINTELLIGENCE\n", "=================\n", "\n", "> Given for one instant an intelligence which could comprehend all the\n", "> forces by which nature is animated and the respective situation of the\n", "> beings who compose it—an intelligence sufficiently vast to submit\n", "> these data to analysis—it would embrace in the same formulate the\n", "> movements of the greatest bodies of the universe and those of the\n", "> lightest atom; for it, nothing would be uncertain and the future, as\n", "> the past, would be present in its eyes.\n", ">\n", "> *Philosophical Essay on Probabilities* Laplace (1814) pg 3\n", "\n", "\n", "\n", "$$ \\text{data} + \\text{model} \\stackrel{\\text{compute}}{\\rightarrow} \\text{prediction}$$\n", "\n", "> If we do discover a theory of everything … it would be the ultimate\n", "> triumph of human reason-for then we would truly know the mind of God\n", ">\n", "> Stephen Hawking in *A Brief History of Time* 1988" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Life\n", "----\n", "\n", "\\[edit\\]\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "Figure: *Left* A Glider pattern (discoverd 1969). *Middle* A Loafer\n", "pattern (discovered 2013) *Right*. John Horton Conway, creator of *Life*\n", "(1937-2020).\n", "\n", "\n", "\n", "\n", "\n", "> The curve described by a simple molecule of air or vapor is regulated\n", "> in a manner just as certain as the planetary orbits; the only\n", "> difference between them is that which comes from our ignorance.\n", ">\n", "> Probability is relative, in part to this ignorance, in part to our\n", "> knowledge.\n", ">\n", "> *Philosophical Essay on Probabilities* Laplace (1814) pg 5\n", "\n", "INTRODUCE ENTROPY TO MEASURE IGNORANCE\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "Figure: Bertrand Russell (1872-1970), Albert Einstein (1879-1955),\n", "Norbert Wiener, (1894-1964)\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "UNCERTAINTY\n", "===========\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "Figure: James Clerk Maxwell (1831-1879), Ludwig Boltzmann (1844-1906)\n", "Josiah Willard Gibbs (1839-1903)\n", "\n", "\n", "\n", "\n", "\n", "\n", "Entropy:\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "Figure: Bernoulli’s simple kinetic models of gases assume that the\n", "molecules of air operate like billiard balls.\n", "\n", "\n", "\n", "> But if we conceive a being whose faculties are so sharpened that he\n", "> can follow every molecule in its course, such a being, whose\n", "> attributes are still as essentially finite as our own, would be able\n", "> to do what is at present impossible to us. For we have seen that the\n", "> molecules in a vessel full of air at uniform temperature are moving\n", "> with velocities by no means uniform, though the mean velocity of any\n", "> great number of them, arbitrarily selected, is almost exactly uniform.\n", "> Now let us suppose that such a vessel is divided into two portions, A\n", "> and B, by a division in which there is a small hole, and that a being,\n", "> who can see the individual molecules, opens and closes this hole, so\n", "> as to allow only the swifter molecules to pass from A to B, and the\n", "> only the slower ones to pass from B to A. He will thus, without\n", "> expenditure of work, raise the temperature of B and lower that of A,\n", "> in contradiction to the second law of thermodynamics.\n", ">\n", "> *Theory of Heat* (Maxwell, 1871) page 308\n", "\n", "\n", "\n", "\n", "Entropy:\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "Figure: Maxwell’s Demon. The demon decides balls are either cold\n", "(blue) or hot (red) according to their velocity. Balls are allowed to\n", "pass the green membrane from right to left only if they are cold, and\n", "from left to right, only if they are hot.\n", "\n", "\n", "\n", "Figure: , Claude Shannon (1916-2001) who in 1905 published a\n", "mathematical model for Brownian motion that inspired Wiener’s work on\n", "stoachastic processes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "MEASUREMENT\n", "===========\n", "\n", "\\[edit\\]\n", "\n", "> For instance, the temperature at which ice melts is found to be always\n", "> the same under ordinary circumstances, though, as we shall see, it is\n", "> slightly altered by change of pressure. The temperature of steam which\n", "> issues from boiling water is also constant when the pressure is\n", "> constant.\n", ">\n", "> These two pheomena therefore–the melting of ice and the boiling of\n", "> water–indicate in a visible manner two temperatures which we may use\n", "> as points of reference, the position of which depends on the\n", "> properties of water and not on the conditions of our senses.\n", ">\n", "> *Theory of Heat* Maxwell (1871) page 3" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "HUMANS\n", "======\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "bits/min\n", "\n", "\n", "\n", "billions\n", "\n", "\n", "\n", "2000\n", "\n", "\n", "\n", "6\n", "\n", "
\n", "\n", "billion
calculations/s\n", "\n", "
\n", "\n", "\\~100\n", "\n", "\n", "\n", "a billion\n", "\n", "\n", "\n", "a billion\n", "\n", "
\n", "\n", "embodiment\n", "\n", "\n", "\n", "20 minutes\n", "\n", "\n", "\n", "5 billion years\n", "\n", "\n", "\n", "15 trillion years\n", "\n", "
\n", "\n", "Figure: Embodiment factors are the ratio between our ability to\n", "compute and our ability to communicate. Jean Dominique Bauby suffered\n", "from locked-in syndrome. The embodiment factors show that relative to\n", "the machine we are also locked in. In the table we represent embodiment\n", "as the length of time it would take to communicate one second’s worth of\n", "computation. For computers it is a matter of minutes, but for a human,\n", "whether locked in or not, it is a matter of many millions of years." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Bandwidth Constrained Conversations\n", "\n", "\\[edit\\]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pods\n", "from ipywidgets import IntSlider" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pods.notebook.display_plots('anne-bob-conversation{sample:0>3}.svg', \n", " '../slides/diagrams', sample=IntSlider(0, 0, 7, 1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "Figure: Conversation relies on internal models of other\n", "individuals.\n", "\n", "\n", "\n", "Figure: Misunderstanding of context and who we are talking to leads\n", "to arguments.\n", "\n", "Embodiment factors imply that, in our communication between humans, what\n", "is *not* said is, perhaps, more important than what is said. To\n", "communicate with each other we need to have a model of who each of us\n", "are.\n", "\n", "To aid this, in society, we are required to perform roles. Whether as a\n", "parent, a teacher, an employee or a boss. Each of these roles requires\n", "that we conform to certain standards of behaviour to facilitate\n", "communication between ourselves.\n", "\n", "Control of self is vitally important to these communications.\n", "\n", "The high availability of data available to humans undermines\n", "human-to-human communication channels by providing new routes to\n", "undermining our control of self.\n", "\n", "The consequences between this mismatch of power and delivery are to be\n", "seen all around us. Because, just as driving an F1 car with bicycle\n", "wheels would be a fine art, so is the process of communication between\n", "humans.\n", "\n", "If I have a thought and I wish to communicate it, I first of all need to\n", "have a model of what you think. I should think before I speak. When I\n", "speak, you may react. You have a model of who I am and what I was trying\n", "to say, and why I chose to say what I said. Now we begin this dance,\n", "where we are each trying to better understand each other and what we are\n", "saying. When it works, it is beautiful, but when misdeployed, just like\n", "a badly driven F1 car, there is a horrible crash, an argument.\n", "\n", "Stories, between humans." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Computer Conversations\n", "----------------------\n", "\n", "\\[edit\\]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pods\n", "from ipywidgets import IntSlider" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pods.notebook.display_plots('anne-bob-conversation{sample:0>3}.svg', \n", " '../slides/diagrams', sample=IntSlider(0, 0, 7, 1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "Figure: Conversation relies on internal models of other\n", "individuals.\n", "\n", "\n", "\n", "Figure: Misunderstanding of context and who we are talking to leads\n", "to arguments.\n", "\n", "Similarly, we find it difficult to comprehend how computers are making\n", "decisions. Because they do so with more data than we can possibly\n", "imagine.\n", "\n", "In many respects, this is not a problem, it’s a good thing. Computers\n", "and us are good at different things. But when we interact with a\n", "computer, when it acts in a different way to us, we need to remember\n", "why.\n", "\n", "Just as the first step to getting along with other humans is\n", "understanding other humans, so it needs to be with getting along with\n", "our computers.\n", "\n", "Embodiment factors explain why, at the same time, computers are so\n", "impressive in simulating our weather, but so poor at predicting our\n", "moods. Our complexity is greater than that of our weather, and each of\n", "us is tuned to read and respond to one another.\n", "\n", "Their intelligence is different. It is based on very large quantities of\n", "data that we cannot absorb. Our computers don’t have a complex internal\n", "model of who we are. They don’t understand the human condition. They are\n", "not tuned to respond to us as we are to each other.\n", "\n", "Embodiment factors encapsulate a profound thing about the nature of\n", "humans. Our locked in intelligence means that we are striving to\n", "communicate, so we put a lot of thought into what we’re communicating\n", "with. And if we’re communicating with something complex, we naturally\n", "anthropomorphize them.\n", "\n", "We give our dogs, our cats and our cars human motivations. We do the\n", "same with our computers. We anthropomorphize them. We assume that they\n", "have the same objectives as us and the same constraints. They don’t.\n", "\n", "This means, that when we worry about artificial intelligence, we worry\n", "about the wrong things. We fear computers that behave like more powerful\n", "versions of ourselves that will struggle to outcompete us.\n", "\n", "In reality, the challenge is that our computers cannot be human enough.\n", "They cannot understand us with the depth we understand one another. They\n", "drop below our cognitive radar and operate outside our mental models.\n", "\n", "The real danger is that computers don’t anthropomorphize. They’ll make\n", "decisions in isolation from us without our supervision, because they\n", "can’t communicate truly and deeply with us." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "ARTIFICIAL\n", "==========" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Artificial Intelligence\n", "-----------------------\n", "\n", "One of the struggles of artificial intelligence is that the term means\n", "different things to different people. Our intelligence is precious to\n", "us, and the notion that it can be easily recreated is disturbing to us.\n", "This leads to some dystopian notions of artificial intelligence, such as\n", "the singularity.\n", "\n", "Depending on whether this powerful technology is viewed as beneficent or\n", "maleficent, it can be viewed either as a helpful assistant, in the\n", "manner of Jeeves, or a tyrannical dictator.\n", "\n", "\n", "\n", "Figure: We seem to have fallen for a perspective on AI that suggests\n", "it will adapt to our schedule, rather in the manner of a 1930s\n", "manservant.\n", "\n", "The history of automation and technology is a history of us adapting to\n", "technological change. The invention of the railways, and the need for\n", "consistent national times to timetable our movements. The development of\n", "the factory system in the mills of Derbyshire required workers to\n", "operate and maintain the machines that replaced them.\n", "\n", "> Day by day, however, the machines are gaining ground upon us; day by\n", "> day we are becoming more subservient to them; more men are daily bound\n", "> down as slaves to tend them, more men are daily devoting the energies\n", "> of their whole lives to the development of mechanical life. The upshot\n", "> is simply PAGE 185a question of time, but that the time will come when\n", "> the machines will hold the real supremacy over the world and its\n", "> inhabitants is what no person of a truly philosophic mind can for a\n", "> moment question.\n", ">\n", "> Samuel Butler in *Darwin Among the Machines* a letter to the Editor of\n", "> *The Press*, 1863\n", "\n", "Listening to modern to conversations about artificial intelligence, I\n", "think the use of the term *intelligence* has given rise to an idea that\n", "this technology will be the But amoung these different assessments of\n", "artificial intelligence is buried an idea, one that\n", "\n", "Introduce Linnaeus and the hydra." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Hamburg Hydra\n", "-----------------\n", "\n", "\\[edit\\]\n", "\n", "\n", "\n", "Figure: The Hamburg Hydra as shown in Seba and Tanjé (1734),\n", "engraving by Pieter Tanjé. The hydra was a hoax that was unpicked by\n", "Linnaeus in 1735 on his visit to Hamburg on the way to the University of\n", "Harderwijk in the Dutch Republic.\n", "\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*ARTIFICIAL* INTELLIGENCE\n", "=========================" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "DeepFace\n", "--------\n", "\n", "\\[edit\\]\n", "\n", "\n", "\n", "Figure: The DeepFace architecture (Taigman et al., 2014), visualized\n", "through colors to represent the functional mappings at each layer. There\n", "are 120 million parameters in the model.\n", "\n", "The DeepFace architecture (Taigman et al., 2014) consists of layers that\n", "deal with *translation* and *rotational* invariances. These layers are\n", "followed by three locally-connected layers and two fully-connected\n", "layers. Color illustrates feature maps produced at each layer. The\n", "neural network includes more than 120 million parameters, where more\n", "than 95% come from the local and fully connected layers." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deep Learning as Pinball\n", "\n", "\\[edit\\]\n", "\n", "\n", "\n", "Figure: Deep learning models are composition of simple functions. We\n", "can think of a pinball machine as an analogy. Each layer of pins\n", "corresponds to one of the layers of functions in the model. Input data\n", "is represented by the location of the ball from left to right when it is\n", "dropped in from the top. Output class comes from the position of the\n", "ball as it leaves the pins at the bottom.\n", "\n", "Sometimes deep learning models are described as being like the brain, or\n", "too complex to understand, but one analogy I find useful to help the\n", "gist of these models is to think of them as being similar to early pin\n", "ball machines.\n", "\n", "In a deep neural network, we input a number (or numbers), whereas in\n", "pinball, we input a ball.\n", "\n", "Think of the location of the ball on the left-right axis as a single\n", "number. Our simple pinball machine can only take one number at a time.\n", "As the ball falls through the machine, each layer of pins can be thought\n", "of as a different layer of ‘neurons.’ Each layer acts to move the ball\n", "from left to right.\n", "\n", "In a pinball machine, when the ball gets to the bottom it might fall\n", "into a hole defining a score, in a neural network, that is equivalent to\n", "the decision: a classification of the input object.\n", "\n", "An image has more than one number associated with it, so it is like\n", "playing pinball in a *hyper-space*." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install --upgrade git+https://github.com/sods/ods" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pods\n", "from ipywidgets import IntSlider" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pods.notebook.display_plots('pinball{sample:0>3}.svg', \n", " directory='.',\n", " sample=IntSlider(1, 1, 2, 1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "Figure: At initialization, the pins, which represent the parameters\n", "of the function, aren’t in the right place to bring the balls to the\n", "correct decisions.\n", "\n", "\n", "\n", "Figure: After learning the pins are now in the right place to bring\n", "the balls to the correct decisions.\n", "\n", "Learning involves moving all the pins to be in the correct position, so\n", "that the ball ends up in the right place when it’s fallen through the\n", "machine. But moving all these pins in hyperspace can be difficult.\n", "\n", "In a hyper-space you have to put a lot of data through the machine for\n", "to explore the positions of all the pins. Even when you feed many\n", "millions of data points through the machine, there are likely to be\n", "regions in the hyper-space where no ball has passed. When future test\n", "data passes through the machine in a new route unusual things can\n", "happen.\n", "\n", "*Adversarial examples* exploit this high dimensional space. If you have\n", "access to the pinball machine, you can use gradient methods to find a\n", "position for the ball in the hyper space where the image looks like one\n", "thing, but will be classified as another.\n", "\n", "Probabilistic methods explore more of the space by considering a range\n", "of possible paths for the ball through the machine. This helps to make\n", "them more data efficient and gives some robustness to adversarial\n", "examples.\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from IPython.lib.display import YouTubeVideo\n", "YouTubeVideo('WXuK6gekU1Y')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from IPython.lib.display import YouTubeVideo\n", "YouTubeVideo('iWGhXof45zI')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Figure: A vehicle operated by Uber ATG was involved in a fatal crash\n", "when it killed pedestrian Elaine Herzberg, 49." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Universe isn’t as Gaussian as it Was\n", "------------------------------------\n", "\n", "\\[edit\\]\n", "\n", "The [Planck space\n", "craft](https://en.wikipedia.org/wiki/Planck_(spacecraft)) was a European\n", "Space Agency space telescope that mapped the cosmic microwave background\n", "(CMB) from 2009 to 2013. The [Cosmic Microwave\n", "Background](https://en.wikipedia.org/wiki/Cosmic_microwave_background)\n", "is the first observable echo we have of the big bang. It dates to\n", "approximately 400,000 years after the big bang, at the time the universe\n", "was approximately $10^8$ times smaller and the temperature of the\n", "Univers was high, around $3 \\times 10^8$ degrees Kelvin. The Universe\n", "was in the form of a hydrogen plasma. The echo we observe is the moment\n", "when the Universe was cool enough for Protons and electrons to combine\n", "to form hydrogen atoms. At this moment, the Universe became transparent\n", "for the first time, and photons could travel through space.\n", "\n", "\n", "\n", "Figure: Artist’s impression of the Planck spacecraft which measured\n", "the Cosmic Microwave Background between 2009 and 2013.\n", "\n", "The objective of the Planck space craft was to measure the anisotropy\n", "and statistics of the Cosmic Microwave Background. This was important,\n", "because if the standard model of the Universe is correct the variations\n", "around the very high temperature of the Universe of the CMB should be\n", "distributed according to a Gaussian process.[1] Currently our best\n", "estimates show this to be the case (Elsner et al., 2016, 2015; Jaffe et\n", "al., 1998; Pontzen and Peiris, 2010).\n", "\n", "To the high degree of precision that we could measure with the Planck\n", "space telescope, the CMB appears to be a Gaussian process. The\n", "parameters of its covariance function are given by the fundamental\n", "parameters of the universe, for example the amount of dark matter and\n", "matter in the universe\n", "\n", "\n", "\n", "Figure: The cosmic microwave background is, to a very high degree of\n", "precision, a Gaussian process. The parameters of its covariance function\n", "are given by fundamental parameters of the universe, such as the amount\n", "of dark matter and mass.\n", "\n", "[1] Most of my understanding of this is taken from conversations with\n", "Kyle Cranmer, a physicist who makes extensive use of machine learning\n", "methods in his work. See e.g. Mishra-Sharma and Cranmer (2020) from Kyle\n", "and Siddharth Mishra-Sharma. Of course, any errors in the above text are\n", "mine and do not stem from Kyle." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Simulating a CMB Map\n", "--------------------\n", "\n", "The simulation was created by [Boris\n", "Leistedt](https://ixkael.github.io/), see the [original Jupter notebook\n", "here](https://github.com/ixkael/Prob-tools/blob/master/notebooks/The%20CMB%20as%20a%20Gaussian%20Process.ipynb).\n", "\n", "Here we use that code to simulate our own universe and sample from what\n", "it looks like.\n", "\n", "First we install some specialist software as well as `matplotlib`,\n", "`scipy`, `numpy` we require\n", "\n", "- `camb`:\n", " http://camb.readthedocs.io/en/latest/\n", "- `healpy`:\n", " https://healpy.readthedocs.io/en/latest/" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install camb" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install healpy" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "%config IPython.matplotlib.backend = 'retina'\n", "%config InlineBackend.figure_format = 'retina'\n", "\n", "import matplotlib\n", "import matplotlib.pyplot as plt\n", "from matplotlib import rc\n", "from cycler import cycler\n", "\n", "import numpy as np\n", "\n", "rc(\"font\", family=\"serif\", size=14)\n", "rc(\"text\", usetex=False)\n", "matplotlib.rcParams['lines.linewidth'] = 2\n", "matplotlib.rcParams['patch.linewidth'] = 2\n", "matplotlib.rcParams['axes.prop_cycle'] =\\\n", " cycler(\"color\", ['k', 'c', 'm', 'y'])\n", "matplotlib.rcParams['axes.labelsize'] = 16\n", "\n", "import healpy as hp\n", "\n", "import camb\n", "from camb import model, initialpower" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we use the theoretical power spectrum to design the covariance\n", "function." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "nside = 512 # Healpix parameter, giving 12*nside**2 equal-area pixels on the sphere.\n", "lmax = 3*nside # band-limit. Should be 2*nside < lmax < 4*nside to get information content." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we design our Universe. It is parameterised according to the\n", "[$\\Lambda$CDM model](https://en.wikipedia.org/wiki/Lambda-CDM_model).\n", "The variables are as follows. `H0` is the Hubble parameter (in\n", "Km/s/Mpc). The `ombh2` is Physical Baryon density parameter. The `omch2`\n", "is the physical dark matter density parameter. `mnu` is the sum of the\n", "neutrino masses (in electron Volts). `omk` is the $\\Omega_k$ is the\n", "curvature parameter, which is here set to 0, tiving the minimal six\n", "parameter Lambda-CDM model. `tau` is the reionization optical depth.\n", "\n", "Then we set `ns`, the “scalar spectral index.” This was estimated by\n", "Planck to be 0.96. Then there’s `r`, the ratio of the tensor power\n", "spectrum to scalar power spectrum. This has been estimated by Planck to\n", "be under 0.11. Here we set it to zero. These parameters are associated\n", "[with inflation](https://en.wikipedia.org/wiki/Primordial_fluctuations)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Mostly following http://camb.readthedocs.io/en/latest/CAMBdemo.html with parameters from https://en.wikipedia.org/wiki/Lambda-CDM_model\n", "\n", "pars = camb.CAMBparams()\n", "pars.set_cosmology(H0=67.74, ombh2=0.0223, omch2=0.1188, mnu=0.06, omk=0, tau=0.066)\n", "pars.InitPower.set_params(ns=0.96, r=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Having set the parameters, we now use the python software “Code for\n", "Anisotropies in the Microwave Background” to get the results." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pars.set_for_lmax(lmax, lens_potential_accuracy=0);\n", "results = camb.get_results(pars)\n", "powers = results.get_cmb_power_spectra(pars)\n", "totCL = powers['total']\n", "unlensedCL = powers['unlensed_scalar']\n", "\n", "ells = np.arange(totCL.shape[0])\n", "Dells = totCL[:, 0]\n", "Cells = Dells * 2*np.pi / ells / (ells + 1) # change of convention to get C_ell\n", "Cells[0:2] = 0" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cmbmap = hp.synfast(Cells, nside, \n", " lmax=lmax, mmax=None, alm=False, pol=False, \n", " pixwin=False, fwhm=0.0, sigma=None, new=False, verbose=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "hp.mollview(cmbmap)\n", "fig = plt.gcf()\n", "fig.savefig('./physics/mollweide-sample-cmb.png', transparent=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "Figure: A simulation of the Cosmic Microwave Background obtained\n", "through sampling from the relevant Gaussian process covariance (in polar\n", "co-ordinates).\n", "\n", "The world we see today, of course, is not a Gaussian process. There are\n", "many dicontinuities, for example, in the density of matter, and\n", "therefore in the temperature of the Universe.\n", "\n", "$=f\\Bigg($$\\Bigg)$\n", "\n", "Figure: What we observe today is some non-linear function of the\n", "cosmic microwave background.\n", "\n", "We can think of todays observed Universe, though, as a being a\n", "consequence of those temperature fluctuations in the CMB. Those\n", "fluctuations are only order $10^-6$ of the scale of the overal\n", "temperature of the Universe. But minor fluctations in that density is\n", "what triggered the pattern of formation of the Galaxies and how stars\n", "formed and created the elements that are the building blocks of our\n", "Earth (Vogelsberger et al., 2020)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Modelling Herd Immunity\n", "-----------------------\n", "\n", "\\[edit\\]\n", "\n", "This example is taken from [Thomas House’s blog\n", "post](https://personalpages.manchester.ac.uk/staff/thomas.house/blog/modelling-herd-immunity.html)\n", "on Herd Immunity. This model was shared at the beginning of the Covid19\n", "pandemic when the first UK lockdown hadn’t yet occurred." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Pull in libraries needed\n", "%matplotlib inline\n", "import numpy as np\n", "from scipy import integrate" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The next piece of code sets up the dynamics of the compartmental model\n", "model. He doesn’t give the specific details in the blog post, but my\n", "understanding is that the four states are as follows. `x[0]` is the\n", "susceptible population, those that haven’thad the disease yet. The\n", "susceptible population decreases by encounters with infections people.\n", "In Thomas’s model, both `x[3]` and `x[4]` are infections. So the\n", "dynamics of the reduction of the susceptible is given by $$\n", "\\frac{\\text{d}{S}}{\\text{d}t} = - \\beta S (I_1 + I_2).\n", "$$ Here, I’ve used $I_1$ and $I_2$ to represent what appears to be two\n", "separate infectious compartments in Thomas’s model. We’ll speculate\n", "about why there are two in a moment.\n", "\n", "The model appears to be an SEIR model, so rather than becoming\n", "infectious directly you next move to an ‘exposed,’ where you have the\n", "disease, but you are not yet infectious. There are again *two* exposed\n", "states, we’ll return to that in a moment. We denote the first, `x[1]` by\n", "$E_1$. We have $$\n", "\\frac{\\text{d}{E_1}}{\\text{d}t} = \\beta S (I_1 + I_2) - \\sigma E_1.\n", "$$ Note that the first term matches the term from the Susceptible\n", "equation. This is because it is the incoming exposed population.\n", "\n", "The exposed population move to a second compartment of exposure, $E_2$.\n", "I believe the reason for this is that if you use only one exposure\n", "compartment, then the statistics of the duration of exposure are\n", "incorrect (implicitly they are exponetially distributed in the\n", "underlying stochastic version of the model). By using two exposure\n", "departments, Thomas is making a slight correction to this which would\n", "impose a first order gamma distribution on those statistics. A similar\n", "trick is being deployed for the ‘infectious group.’ So we gain an\n", "additional equation to help with these statistics, $$\n", "\\frac{\\text{d}{E_2}}{\\text{d}t} = \\sigma E_1 - \\sigma E_2.\n", "$$ giving us the exposed group as the sum of the two compartments $E_1$\n", "and $E_2$. The exposed group from the second compartment then become\n", "‘infected,’ which we represent with $I_1$, in the code this is `x[3]`,\n", "$$\n", "\\frac{\\text{d}{I_1}}{\\text{d}t} = \\sigma E_2 - \\gamma I_1,\n", "$$ and similarly, Thomas is using a two compartment infectious group to\n", "fix up the duration model. So we have, $$\n", "\\frac{\\text{d}{I_2}}{\\text{d}t} = \\gamma I_1 - \\gamma I_2.\n", "$$ And finally we have those that have recovered emerging from the\n", "second infections compartment. In this model there is no separate model\n", "for ‘deaths,’ so the recovered compartment, $R$, would also include\n", "those that die, $$\n", "\\frac{\\text{d}R}{\\text{d}t} = \\gamma I_2.\n", "$$ All of these equations are then represented in code as follows." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def odefun(t,x,beta0,betat,t0,t1,sigma,gamma):\n", " dx = np.zeros(6)\n", " if ((t>=t0) and (t<=t1)):\n", " beta = betat\n", " else:\n", " beta = beta0\n", " dx[0] = -beta*x[0]*(x[3] + x[4])\n", " dx[1] = beta*x[0]*(x[3] + x[4]) - sigma*x[1]\n", " dx[2] = sigma*x[1] - sigma*x[2]\n", " dx[3] = sigma*x[2] - gamma*x[3]\n", " dx[4] = gamma*x[3] - gamma*x[4]\n", " dx[5] = gamma*x[4]\n", " return dx" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Where the code takes in the states of the compartments (the values of\n", "`x`) and returns the gradients of those states for the provided\n", "parameters (`sigma`, `gamma` and `beta`). Those parameters are set\n", "according to the known characteristics of the disease.\n", "\n", "The next block of code sets up the parameters of the SEIR model. A\n", "particularly important parameter is the reproduction number ($R_0$),\n", "here Thomas has assumed a reproduction number of 2.5, implying that each\n", "infected member of the population transmits the infection up to 2.5\n", "other people. The effective $R$ decreases over time though, because some\n", "of those people they meet will no longer be in the susceptible group." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Parameters of the model\n", "N = 6.7e7 # Total population\n", "i0 = 1e-4 # 0.5*Proportion of the population infected on day 0\n", "tlast = 365.0 # Consider a year\n", "latent_period = 5.0 # Days between being infected and becoming infectious\n", "infectious_period = 7.0 # Days infectious\n", "R0 = 2.5 # Basic reproduction number in the absence of interventions\n", "Rt = 0.75 # Reproduction number in the presence of interventions\n", "tend = 21.0 # Number of days of interventions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The parameters are correct for the ‘discrete system,’ where the\n", "inectious period is a discrete time, and the numbers are discrete\n", "values. To translate into our continuous differential equation system’s\n", "parameters, we need to do a couple of manipulations. Note the factor of\n", "2 associated with `gamma` and `sigma`. This is a doubling of the rate to\n", "account for the fact that there are two compartments for each of these\n", "states (to fix-up the statistics of the duration models)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "beta0 = R0 / infectious_period\n", "betat = Rt / infectious_period\n", "sigma = 2.0 / latent_period\n", "gamma = 2.0 / infectious_period" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next we solve the system using `scipy`’s initial value problem solver.\n", "The solution method is \"Runge-Kutta-Fehlberg method, as indicated by the\n", "`'RK45'` solver. This is a numerical method for solving differential\n", "equations. The 45 is the order of the method and the error estimator.\n", "\n", "We can view the solver itself as somehow a piece of simulation code, but\n", "here it’s being called as sub routine in the system. It returns a\n", "solution for each time step, stored in a list `sol`.\n", "\n", "This is typical of this type of non-linear differential equation\n", "problem. Whether it’s partial differential equations, ordinary\n", "differential equations, there’s a step where a numerical solver needs to\n", "be called. These are often expensive to run. For climate and weather\n", "models, this would be where we solved the Navier-Stokes equations. For\n", "this simple model, the solution is relatively quick." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "t0ran = np.array([-100, 40, 52.5, 65])\n", "sol=[]\n", "for tt in range(0,len(t0ran)):\n", " sol.append(integrate.solve_ivp(lambda t,x: odefun(t,x,beta0,betat,t0ran[tt],t0ran[tt]+tend,sigma,gamma),\n", " (0.0,tlast),\n", " np.array([1.0-2.0*i0, 0.0, 0.0, i0, i0, 0.0]),\n", " 'RK45',\n", " atol=1e-8,\n", " rtol=1e-9))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import teaching_plots as plot\n", "import mlai" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def mylab(t):\n", " if t>0:\n", " return \"Start at \" + str(t) + \" days\"\n", " else:\n", " return \"Baseline\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(1, 2, figsize=plot.big_wide_figsize)\n", "for tt in range(0,len(t0ran)):\n", " ax[0].plot(sol[tt].t,N*(sol[tt].y[3] + sol[tt].y[4]).T, label=mylab(t0ran[tt]))\n", "ax[0].set_xlim([30,70])\n", "ax[0].set_ylim([0,7e6])\n", "ax[0].set_xlabel('Time (days)')\n", "ax[0].set_ylabel('Number of infectious cases')\n", "ax[0].legend()\n", "ax[1].subplot(1,2,2)\n", "for tt in range(0,len(t0ran)):\n", " ax[1].plot(sol[tt].t,N*sol[tt].y[5].T, label=mylab(t0ran[tt]))\n", "ax[1].set_xlim([30,70])\n", "ax[1].set_ylim([0,1e7])\n", "ax[1].set_xlabel('Time (days)')\n", "ax[1].set_ylabel('Cumulative infections')\n", "ax[1].legend()\n", "\n", "mlai.write_figure('house-model-zoom.svg', directory='./simulation')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "Figure: A zoomed in version of Thomas House’s variation on the SEIR\n", "model for evaluating the effect of early interventions." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(1, 2, figsize=plot.big_wide_size)\n", "for tt in range(0,len(t0ran)):\n", " ax[0].plot(sol[tt].t,N*(sol[tt].y[3] + sol[tt].y[4]).T, label=mylab(t0ran[tt]))\n", "ax[0].xlim([0,tlast])\n", "ax[0].ylim([0,1.2e7])\n", "ax[0].xlabel('Time (days)')\n", "ax[0].ylabel('Number of infectious cases')\n", "ax[0].legend()\n", "for tt in range(0,len(t0ran)):\n", " ax[1].plot(sol[tt].t,N*sol[tt].y[5].T, label=mylab(t0ran[tt]))\n", "ax[1].set_xlabel('Time (days)')\n", "ax[1].set_ylabel('Cumulative infections')\n", "ax[1].legend()\n", "ax[1].set_xlim([0,tlast])\n", "ax[1].set_ylim([0,6.2e7])\n", "\n", "mlai.write_figure('house-model-full.svg', directory='./simulation/')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "Figure: The full progress of the disease in Thomas House’s variation\n", "on the SEIR model for evaluating the effect of early interventions.\n", "\n", "In practice, immunity for Covid19 may only last around 6 months. As an\n", "exercise, try to extend Thomas’s model for the case where immunity is\n", "temporary. You’ll need to account for deaths as well in your new model.\n", "\n", "\n", "\n", "Figure: The sets of different models. There are all the models in the\n", "Universe we might like to work with. Then there are those models that\n", "are computable e.g. by a Turing machine. Then there are those which are\n", "analytical tractable. I.e. where the solution might be found\n", "analytically. Finally, there are Gaussian processes, where the joint\n", "distribution of the states in the model is Gaussian.\n", "\n", "\n", "\n", "Figure: Data driven and mechanistic models have separated since the\n", "origin of the field. Accelerate science is about bringing these two\n", "modalities back together.\n", "\n", "INTRODUCE THE SOLUTIONS\n", "\n", "\n", "\n", "Figure: Experiment, analyze and design is a flywheel of knowledge\n", "that is the dual of the model, data and compute. By running through this\n", "spiral, we refine our hypothesis/model and develop new experiments which\n", "can be analyzed to further refine our hypothesis.\n", "\n", "- Step Change in Science through Machine Learning\n", "- You!\n", "- [ML and the Physical World](http://mlatcl.github.io/mlphysical)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Thanks!\n", "-------\n", "\n", "For more information on these subjects and more you might want to check\n", "the following resources.\n", "\n", "- twitter: [@lawrennd](https://twitter.com/lawrennd)\n", "- podcast: [The Talking Machines](http://thetalkingmachines.com)\n", "- newspaper: [Guardian Profile\n", " Page](http://www.theguardian.com/profile/neil-lawrence)\n", "- blog:\n", " [http://inverseprobability.com](http://inverseprobability.com/blog.html)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "References\n", "----------" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Elsner, F., Leistedt, B., Peiris, H.V., 2016. Unbiased pseudo-$C_\\ell$\n", "power spectrum estimation with mode projection. Monthly Notices of the\n", "Royal Astronomical Society 465, 1847–1855.\n", "\n", "\n", "Elsner, F., Leistedt, B., Peiris, H.V., 2015. Unbiased methods for\n", "removing systematics from galaxy clustering measurements. Monthly\n", "Notices of the Royal Astronomical Society 456, 2095–2104.\n", "\n", "\n", "Jaffe, A.H., Bond, J.R., Ferreira, P.G., Knox, L.E., 1998. CMB\n", "likelihood functions for beginners and experts, in: AIP Conf. Proc.\n", "\n", "\n", "Laplace, P.S., 1814. Essai philosophique sur les probabilités, 2nd ed.\n", "Courcier, Paris.\n", "\n", "Maxwell, J.C., 1871. Theory of heat. Longmans, Green; Co, London.\n", "\n", "Mishra-Sharma, S., Cranmer, K., 2020. Semi-parametric $\\gamma$-ray\n", "modeling with Gaussian processes and variational inference.\n", "\n", "Pontzen, A., Peiris, H.V., 2010. The cut-sky cosmic microwave background\n", "is not anomalous. Phys. Rev. D 81, 103008.\n", "\n", "\n", "Seba, A., Tanjé, P., 1734. Thesaurus rerum naturalium. Apud J.\n", "Wetstenium & Gul. Smith, & Janssonio-Waesbergios.\n", "\n", "Taigman, Y., Yang, M., Ranzato, M., Wolf, L., 2014. DeepFace: Closing\n", "the gap to human-level performance in face verification, in: Proceedings\n", "of the IEEE Computer Society Conference on Computer Vision and Pattern\n", "Recognition. \n", "\n", "Vogelsberger, M., Marinacci, F., Torrey, P., Puchwei, E., 2020.\n", "Cosmological simulations of galaxy formation. Nature Reviews Physics\n", "42–66. " ] } ], "nbformat": 4, "nbformat_minor": 5, "metadata": {} }