{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![book](https://raw.githubusercontent.com/ageron/tensorflow-safari-course/master/images/intro_to_tf_course.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Try not to peek at the solutions when you go through the exercises. ;-)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's make sure this notebook works well in both Python 2 and Python 3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*__From previous notebooks__*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 0.01\n",
    "momentum = 0.8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorBoard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Exercise](https://c1.staticflickr.com/9/8101/8553474140_c50cf08708_b.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise, you will learn to use TensorBoard. It is a great visualization tool that comes with TensorFlow. It works by parsing special TensorFlow logs, called _summaries_, and displaying them nicely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.1) Starting the TensorBoard server. Open a Terminal and type the following commands."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Move to the `tensorflow-safari-course` directory:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`~$` **`cd tensorflow-safari-course`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the `tf_logs` directory that will hold the TensorFlow data that we will want TensorBoard to display:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`~/tensorflow-safari-course$` **`mkdir tf_logs`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Activate the virtual environment:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`~/tensorflow-safari-course$` **`source env/bin/activate`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start the TensorBoard server:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`(env) ~/tensorflow-safari-course$` **`tensorboard --logdir=tf_logs`**\n",
    "\n",
    "`Starting TensorBoard b'41' on port 6006\n",
    "(You can navigate to` http://127.0.1.1:6006 `)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now visit the URL given by TensorBoard. You should see the TensorBoard interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.2) Now create a `tf.summary.FileWriter`, with the parameters: `logdir=\"tf_logs/run_number_1/\"` and `graph=graph` where `graph` is the one we built just before this exercise. This will automatically:\n",
    "* create the `run_number_1` directory inside the `tf_logs` directory,\n",
    "* create an `events.out.tfevents.*` file in that subdirectory that will contain the data that TensorBoard will display,\n",
    "* write the graph's definition to this file.\n",
    "\n",
    "Next, try refreshing the TensorBoard page in your browser (you may need to wait a couple minutes for it to detect the change, or else you can just restart the TensorBoard server). Visit the Graph tab: you should be able to visualize the graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.3) As you can see, the graph looks really messy in TensorBoard. We need to organize it a bit. For this, name scopes come in handy. An operation can be placed inside a name scope in one of two ways:\n",
    "\n",
    "* Add the scope as a prefix to the operation's name, for example:\n",
    "\n",
    "```python\n",
    "a = tf.constant(0.0, name=\"my_name_scope/a\")\n",
    "```\n",
    "\n",
    "* Or (generally clearer) use a `tf.name_scope()` block, for example:\n",
    "\n",
    "```python\n",
    "with tf.name_scope(\"my_name_scope\"):\n",
    "    a = tf.constant(0.0, name=\"a\")\n",
    "```\n",
    "\n",
    "Add name scopes to the following graph, then write it to TensorBoard (using a different run number for the log directory name) and see how much better it looks, and how much easier it is to explore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filenames = [\"data/life_satisfaction.csv\"]\n",
    "n_epochs = 500\n",
    "batch_size = 5\n",
    "\n",
    "graph = tf.Graph()\n",
    "with graph.as_default():\n",
    "    reader = tf.TextLineReader(skip_header_lines=1)\n",
    "\n",
    "    filename_queue = tf.train.string_input_producer(filenames, num_epochs=n_epochs)\n",
    "    record_id, record = reader.read(filename_queue)\n",
    "\n",
    "    record_defaults = [[''], [0.0], [0.0]]\n",
    "    country, gdp_per_capita, life_satisfaction = tf.decode_csv(record, record_defaults=record_defaults)\n",
    "\n",
    "    X_batch, y_batch = tf.train.batch([gdp_per_capita, life_satisfaction], batch_size=batch_size)\n",
    "    X_batch_reshaped = tf.reshape(X_batch, [-1, 1])\n",
    "    y_batch_reshaped = tf.reshape(y_batch, [-1, 1])\n",
    "\n",
    "    X = tf.placeholder_with_default(X_batch_reshaped, shape=[None, 1], name=\"X\")\n",
    "    y = tf.placeholder_with_default(y_batch_reshaped, shape=[None, 1], name=\"y\")\n",
    "\n",
    "    b = tf.Variable(0.0, name=\"b\")\n",
    "    w = tf.Variable(tf.zeros([1, 1]), name=\"w\")\n",
    "    y_pred = tf.add(tf.matmul(X / 10000, w), b, name=\"y_pred\")  # X @ w + b\n",
    "    \n",
    "    mse = tf.reduce_mean(tf.square(y_pred - y), name=\"mse\")\n",
    "    global_step = tf.Variable(0, trainable=False, name='global_step')\n",
    "    optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)\n",
    "    training_op = optimizer.minimize(mse, global_step=global_step)\n",
    "        \n",
    "    init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())\n",
    "    saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.4) Print out the name of a few operations. Notice how the names now have the scope as a prefix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.5) TensorBoard is capable of displaying data from multiple TensorFlow runs (for example multiple training sessions). For this, we need to place the data from each run in a different subdirectory of the `tf_logs` directory. We can name these subdirectories however we want, but a simple option is to name them using a timestamp. The following `logdir()` function returns the path of a subdirectory whose name is based on the current date and time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "def logdir():\n",
    "    root_logdir = \"tf_logs\"\n",
    "    now = datetime.utcnow().strftime(\"%Y%m%d%H%M%S\")\n",
    "    return \"{}/run_{}/\".format(root_logdir, now)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logdir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a few different graphs and instantiate a different FileWriter for each one, using a different log directory every time (with the help of the `logdir()` function). Refresh TensorBoard and notice that you can browse any graph you want by selecting the appropriate run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.6) Now we will use TensorBoard to visualize the learning curve, that is the evolution of the cost function during training.\n",
    "\n",
    "* First add a scalar summary operation in the graph, using `tf.summary.scalar(\"MSE\", mse)`.\n",
    "* Next, update the training code to evaluate this scalar summary and write the result to the events file using the `FileWriter`'s `add_summary()` method (also specifying the training step). For performance reasons, you probably want to do this only every 10 training iterations or so.\n",
    "* Next, train the model.\n",
    "* Refresh TensorBoard, and visit the Scalars tab. Select the appropriate run and visualize the learning curve. Try zooming in and out, and play around with the options, in particular the smoothing option."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try not to peek at the solution below before you have done the exercise! :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![thinking](https://upload.wikimedia.org/wikipedia/commons/0/06/Filos_segundo_logo_%28flipped%29.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 6 - Solution\n",
    "6.1)\n",
    "\n",
    "N/A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_writer = tf.summary.FileWriter(\"tf_logs/run_number_1_solution/\", graph=graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filenames = [\"data/life_satisfaction.csv\"]\n",
    "n_epochs = 500\n",
    "batch_size = 5\n",
    "\n",
    "graph = tf.Graph()\n",
    "with graph.as_default():\n",
    "    with tf.name_scope(\"reader\"):\n",
    "        reader = tf.TextLineReader(skip_header_lines=1)\n",
    "\n",
    "        filename_queue = tf.train.string_input_producer(filenames, num_epochs=n_epochs)\n",
    "        record_id, record = reader.read(filename_queue)\n",
    "\n",
    "        record_defaults = [[''], [0.0], [0.0]]\n",
    "        country, gdp_per_capita, life_satisfaction = tf.decode_csv(record, record_defaults=record_defaults)\n",
    "\n",
    "        X_batch, y_batch = tf.train.batch([gdp_per_capita, life_satisfaction], batch_size=batch_size)\n",
    "        X_batch_reshaped = tf.reshape(X_batch, [-1, 1])\n",
    "        y_batch_reshaped = tf.reshape(y_batch, [-1, 1])\n",
    "\n",
    "    with tf.name_scope(\"linear_model\"):\n",
    "        X = tf.placeholder_with_default(X_batch_reshaped, shape=[None, 1], name=\"X\")\n",
    "        y = tf.placeholder_with_default(y_batch_reshaped, shape=[None, 1], name=\"y\")\n",
    "\n",
    "        b = tf.Variable(0.0, name=\"b\")\n",
    "        w = tf.Variable(tf.zeros([1, 1]), name=\"w\")\n",
    "        y_pred = tf.add(tf.matmul(X / 10000, w), b, name=\"y_pred\")  # X @ w + b\n",
    "    \n",
    "    with tf.name_scope(\"train\"):\n",
    "        mse = tf.reduce_mean(tf.square(y_pred - y), name=\"mse\")\n",
    "        global_step = tf.Variable(0, trainable=False, name='global_step')\n",
    "        optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)\n",
    "        training_op = optimizer.minimize(mse, global_step=global_step)\n",
    "        \n",
    "    init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())\n",
    "    saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_writer = tf.summary.FileWriter(\"tf_logs/run_number_2_solution/\", graph=graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "country.name, gdp_per_capita.name, X_batch.name, y_batch.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.name, y.name, b.name, w.name, y_pred.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mse.name, global_step.name, training_op.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "graph1 = tf.Graph()\n",
    "with graph1.as_default():\n",
    "    a = tf.constant(1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_writer = tf.summary.FileWriter(logdir(), graph=graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "graph2 = tf.Graph()\n",
    "with graph2.as_default():\n",
    "    a = tf.constant(1.0, name=\"a\")\n",
    "    b = tf.Variable(2.0, name=\"b\")\n",
    "    c = a * b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we run `logdir()` twice within the same second, we will get the same directory name twice. To avoid this, let's wait a bit over 1 second here. In real life, this is quite unlikely to happen since training a model typically takes much longer than 1 second."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "time.sleep(1.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_writer = tf.summary.FileWriter(logdir(), graph=graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time.sleep(1.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filenames = [\"data/life_satisfaction.csv\"]\n",
    "n_epochs = 500\n",
    "batch_size = 5\n",
    "\n",
    "graph = tf.Graph()\n",
    "with graph.as_default():\n",
    "    with tf.name_scope(\"reader\"):\n",
    "        reader = tf.TextLineReader(skip_header_lines=1)\n",
    "\n",
    "        filename_queue = tf.train.string_input_producer(filenames, num_epochs=n_epochs)\n",
    "        record_id, record = reader.read(filename_queue)\n",
    "\n",
    "        record_defaults = [[''], [0.0], [0.0]]\n",
    "        country, gdp_per_capita, life_satisfaction = tf.decode_csv(record, record_defaults=record_defaults)\n",
    "\n",
    "        X_batch, y_batch = tf.train.batch([gdp_per_capita, life_satisfaction], batch_size=batch_size)\n",
    "        X_batch_reshaped = tf.reshape(X_batch, [-1, 1])\n",
    "        y_batch_reshaped = tf.reshape(y_batch, [-1, 1])\n",
    "\n",
    "    with tf.name_scope(\"linear_model\"):\n",
    "        X = tf.placeholder_with_default(X_batch_reshaped, shape=[None, 1], name=\"X\")\n",
    "        y = tf.placeholder_with_default(y_batch_reshaped, shape=[None, 1], name=\"y\")\n",
    "\n",
    "        b = tf.Variable(0.0, name=\"b\")\n",
    "        w = tf.Variable(tf.zeros([1, 1]), name=\"w\")\n",
    "        y_pred = tf.add(tf.matmul(X / 10000, w), b, name=\"y_pred\")\n",
    "    \n",
    "    with tf.name_scope(\"train\"):\n",
    "        mse = tf.reduce_mean(tf.square(y_pred - y), name=\"mse\")\n",
    "        mse_summary = tf.summary.scalar('MSE', mse)                     # <= ADDED\n",
    "        global_step = tf.Variable(0, trainable=False, name='global_step')\n",
    "        optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)\n",
    "        training_op = optimizer.minimize(mse, global_step=global_step)\n",
    "        \n",
    "    init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())\n",
    "    saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_writer = tf.summary.FileWriter(logdir(), graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.Session(graph=graph) as sess:\n",
    "    init.run()\n",
    "    coord = tf.train.Coordinator()\n",
    "    threads = tf.train.start_queue_runners(coord=coord)\n",
    "    try:\n",
    "        while not coord.should_stop():\n",
    "            _, mse_summary_val, global_step_val = sess.run([training_op, mse_summary, global_step])\n",
    "            if global_step_val % 10 == 0:\n",
    "                summary_writer.add_summary(mse_summary_val, global_step_val)\n",
    "    except tf.errors.OutOfRangeError:\n",
    "        print(\"End of training\")\n",
    "    coord.request_stop()\n",
    "    coord.join(threads)\n",
    "    saver.save(sess, \"./my_life_satisfaction_model\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}