{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Settings and GPflow config\n",
    "GPflow has a config file, `gpflowrc` which allows the user to change the default behavious in GPflow. GPflow searches for the file in the following order:\n",
    "1. In the working directory\n",
    "2. In the user's home directory\n",
    "3. In the GPflow directory (revert to default)\n",
    "\n",
    "You can also make `gpflowrc` a hidden file, if you don't want it clutting your home directory, by renaming as `.gpflowrc`.\n",
    "\n",
    "Usually, one wants to change the settings for an entire project. We recommend adding a `gpflowrc` file in the working directory to achieve this effect. It is helpful to add assertions / warnings in code to ensure that the correct settings are used. GPflow also allows settings to be adjusted in code, although this comes with some additional difficulties (see below). This is why we recommend using a settings file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Common uses\n",
    "The most common use cases for changing settings are:\n",
    "- **Jitter levels**: GPflow has to perform Cholesky decompositions to learn GPs, which require positive definite (PSD) matrices. Due to the numerical behaviour of floating point numbers, kernel matrices may behave as though they are not exactly PSD. To avoid errors regarding to positive definiteness or invertibility, we add *jitter* to a matrix that is to be inverted, i.e. we perform $\\mathrm{Cholesky}\\left(K + jI\\right)$.\n",
    "- **Floating point precision**: We can adjust the precision at which computations get performed to take advantage of e.g. faster `float32` computations on the GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Default settings\n",
    "By default, the configuration looks like this:\n",
    "```\n",
    "[logging]\n",
    "# possible levels: CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET\n",
    "level = WARNING\n",
    "\n",
    "[verbosity]\n",
    "tf_compile_verb = False\n",
    "\n",
    "[dtypes]\n",
    "float_type = float64\n",
    "int_type = int32\n",
    "\n",
    "[numerics]\n",
    "jitter_level = 1e-6\n",
    "# quadrature can be set to: allow, warn, error\n",
    "ekern_quadrature = warn\n",
    "\n",
    "[profiling]\n",
    "dump_timeline = False\n",
    "dump_tensorboard = False\n",
    "output_file_name = timeline\n",
    "output_directory = ./\n",
    "each_time = False\n",
    "\n",
    "[session]\n",
    "intra_op_parallelism_threads = 0\n",
    "inter_op_parallelism_threads = 0\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing settings\n",
    "You can access the settings as `gpflow.settings`, and the different options are nested under the headings in the file. For example, to see how much jitter is added before attempting Cholesky decomposition:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0813 20:55:52.087115 4544173504 deprecation_wrapper.py:119] From /Users/artemav/code/GPflow/gpflow/session_manager.py:31: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
      "\n",
      "W0813 20:55:52.090715 4544173504 deprecation_wrapper.py:119] From /Users/artemav/code/GPflow/gpflow/misc.py:27: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.\n",
      "\n",
      "W0813 20:55:52.241827 4544173504 deprecation_wrapper.py:119] From /Users/artemav/code/GPflow/gpflow/training/tensorflow_optimizer.py:169: The name tf.train.AdadeltaOptimizer is deprecated. Please use tf.compat.v1.train.AdadeltaOptimizer instead.\n",
      "\n",
      "W0813 20:55:52.242599 4544173504 deprecation_wrapper.py:119] From /Users/artemav/code/GPflow/gpflow/training/tensorflow_optimizer.py:156: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
      "\n",
      "W0813 20:55:52.243191 4544173504 deprecation_wrapper.py:119] From /Users/artemav/code/GPflow/gpflow/training/tensorflow_optimizer.py:169: The name tf.train.AdagradDAOptimizer is deprecated. Please use tf.compat.v1.train.AdagradDAOptimizer instead.\n",
      "\n",
      "W0813 20:55:52.243880 4544173504 deprecation_wrapper.py:119] From /Users/artemav/code/GPflow/gpflow/training/tensorflow_optimizer.py:169: The name tf.train.AdagradOptimizer is deprecated. Please use tf.compat.v1.train.AdagradOptimizer instead.\n",
      "\n",
      "W0813 20:55:52.244505 4544173504 deprecation_wrapper.py:119] From /Users/artemav/code/GPflow/gpflow/training/tensorflow_optimizer.py:169: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
      "\n",
      "W0813 20:55:52.256444 4544173504 deprecation_wrapper.py:119] From /Users/artemav/code/GPflow/gpflow/saver/coders.py:80: The name tf.data.Iterator is deprecated. Please use tf.compat.v1.data.Iterator instead.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1e-05\n"
     ]
    }
   ],
   "source": [
    "import gpflow\n",
    "print(gpflow.settings.jitter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If settings are updated during programme execution (see later), you need to always access the setting through the `settings` module, to ensure you obtain the latest value. If you make a module level copy of the variable, you may end up using stale values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modifying settings\n",
    "Settings can be modified for an entire session, or for a limited set of statements, using a Python context manager. It is recommeded to use the context manager, as this prevents the change of state unintentionally spilling into other parts of the program. The global settings can be set with `gpflow.settings.push(custom_config)`. This pushes the old config to a stack, which can be popped with `gpflow.settings.pop()`. Again, we recommend using the context manager.\n",
    "\n",
    "### Example: Jitter levels\n",
    "We first train a model without any jitter. We start by making a copy of the current settings using `gpflow.settings.get_settings()`. We can edit the values of settings in this object by simple assignment. We then use the context manager `gpflow.settings.temp_settings()` to use those settings for the duration of the model creation.\n",
    "\n",
    "We will expect this to fail with an `InvalidArgumentError` due to the Cholesky failing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Default jitter level: 1.00e-05\n",
      "Current jitter level: 0.00e+00\n",
      "\n",
      "As expected, we encountered the error: `Cholesky decomposition was not successful. The input might not be valid.`\n",
      "\n",
      "Reverted jitter level: 1.00e-05\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import numpy.random as rnd\n",
    "import tensorflow as tf\n",
    "np.random.seed(0)\n",
    "X = np.random.randn(100, 1)\n",
    "Y = np.sin(X) + np.sin(1.5*X) + 0.3 * rnd.randn(*X.shape)\n",
    "\n",
    "custom_config = gpflow.settings.get_settings()\n",
    "custom_config.numerics.jitter_level = 0.0\n",
    "print(\"Default jitter level: %.2e\" % gpflow.settings.jitter)\n",
    "with gpflow.settings.temp_settings(custom_config):\n",
    "    print(\"Current jitter level: %.2e\" % gpflow.settings.jitter)\n",
    "    m = gpflow.models.SGPR(X, Y, gpflow.kernels.RBF(1), Z=X.copy())\n",
    "\n",
    "try:\n",
    "    opt = gpflow.train.ScipyOptimizer()\n",
    "    opt.minimize(m)\n",
    "except tf.errors.InvalidArgumentError as e:\n",
    "    print(\"\\nAs expected, we encountered the error: `%s`\\n\" % e.message.split(\"\\n\")[0])\n",
    "print(\"Reverted jitter level: %.2e\" % gpflow.settings.jitter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the default jitter levels reverted, we can optimise the model just fine:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = gpflow.models.SGPR(X, Y, gpflow.kernels.RBF(1), Z=X.copy())\n",
    "gpflow.train.ScipyOptimizer().minimize(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example: Debug messages\n",
    "GPflow also logs a wide variety of events that happen under the hood. One complex piece of code, is how Gaussian conditioning is handled. Many different cases are handled using multiple dispatch, and sometimes it is complicated to keep track of what code is being run. This is logged with the logger module. To display these debug messages, we need to set the logging level to `DEBUG`. This needs to be done in `gpflowrc` for it to have effect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### WARNING\n",
    "Because of the static graph compilation of TensorFlow, models defined inside of the context manager will still retain the settings outside the context manager. The zero jitter example above demonstrates this. We define the model inside a context manager with a zero jitter setting. When we try to optimise the model *outside*, the model will use zero jitter, since that was the setting's value at the time of model creation. Hence the Cholesky will fail. If we re-define the model with some jitter, the model will optimise just fine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change TensorFlow session settings\n",
    "GPflow may create multiple tensorflow sessions for a single model. To control the session parameters change the [session] section of the settings. This section may contain any valid [TensorFlow ConfigProto](https://www.tensorflow.org/api_docs/python/tf/ConfigProto) setting. \n",
    "\n",
    "For instance to ensure all tensorflow graphs are run serially set\n",
    "```\n",
    "[session]\n",
    "intra_op_parallelism_threads = 1\n",
    "inter_op_parallelism_threads = 1\n",
    "```\n",
    "As per the TensorFlow documentation, a setting of 0 means the system picks an appropriate number of cores to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}