{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Log output\n",
    "\n",
    "We use the same example as in the quickstart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "using MIPVerify\n",
    "using Gurobi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1×28×28×1 Array{Float64,4}:\n",
       "[:, :, 1, 1] =\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0  0.0\n",
       "\n",
       "[:, :, 2, 1] =\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0  0.0\n",
       "\n",
       "[:, :, 3, 1] =\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0  0.0\n",
       "\n",
       "...\n",
       "\n",
       "[:, :, 26, 1] =\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0  0.0\n",
       "\n",
       "[:, :, 27, 1] =\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0  0.0\n",
       "\n",
       "[:, :, 28, 1] =\n",
       " 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  …  0.0  0.0  0.0  0.0  0.0  0.0  0.0"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnist = MIPVerify.read_datasets(\"MNIST\")\n",
    "n1 = MIPVerify.get_example_network_params(\"MNIST.WK17a_linf0.1_authors\")\n",
    "sample_image = MIPVerify.get_image(mnist.test.images, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Viewing Log Output\n",
    "\n",
    "Our package logs to STDOUT at the `debug`, `info`, `notice` and `warn` levels. \n",
    "The default output level is to show logs only at the `notice` level and above, which is also the recommended level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Academic license - for non-commercial use only\n",
      "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n",
      "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m  Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Academic license - for non-commercial use only\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m  Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n",
      "\u001b[32m  Imposing relu constraint: 100%|███████████████████████| Time: 0:00:01\u001b[39m\n",
      "\u001b[32m  Calculating upper bounds:   1%|▏                      |  ETA: 0:47:34\u001b[39m"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Academic license - for non-commercial use only\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m  Calculating upper bounds: 100%|███████████████████████| Time: 0:00:21\u001b[39m\n",
      "\u001b[32m  Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n",
      "\u001b[32m  Imposing relu constraint: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n",
      "\u001b[32m  Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n",
      "\u001b[32m  Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n",
      "\u001b[32m  Imposing relu constraint: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Academic license - for non-commercial use only\n",
      "Academic license - for non-commercial use only\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Dict{Any,Any} with 11 entries:\n",
       "  :TargetIndexes      => [10]\n",
       "  :SolveTime          => 0.00462294\n",
       "  :TotalTime          => 50.902\n",
       "  :Perturbation       => JuMP.VariableRef[noname noname … noname noname]…\n",
       "  :PerturbedInput     => JuMP.VariableRef[noname noname … noname noname]…\n",
       "  :TighteningApproach => \"lp\"\n",
       "  :PerturbationFamily => linf-norm-bounded-0.05\n",
       "  :SolveStatus        => INFEASIBLE_OR_UNBOUNDED\n",
       "  :Model              => A JuMP Model…\n",
       "  :Output             => JuMP.GenericAffExpr{Float64,JuMP.VariableRef}[0.000383…\n",
       "  :PredictedIndex     => 8"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MIPVerify.find_adversarial_example(\n",
    "    n1, \n",
    "    sample_image, \n",
    "    10, \n",
    "    Gurobi.Optimizer,\n",
    "    # OutputFlag=0 prevents any output from being printed out\n",
    "    Dict(\"OutputFlag\" => 0),\n",
    "    pp = MIPVerify.LInfNormBoundedPerturbationFamily(0.05),\n",
    "    norm_order = Inf,\n",
    "    tightening_algorithm = lp,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information is available if we set the log level to `debug`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m[debug | Memento]: Logger(root) is already registered.\u001b[39m\n",
      "Academic license - for non-commercial use only\n",
      "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n",
      "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n",
      "Academic license - for non-commercial use only\n",
      "\u001b[32m[info | MIPVerify]: Applying Conv2d(1, 16, kernel_size=(4, 4), stride=(2, 2), padding=same) ... \u001b[39m\n",
      "\u001b[32m[info | MIPVerify]: Applying ReLU() ...\u001b[39m\n",
      "\u001b[32m[info | MIPVerify]: Applying Conv2d(16, 32, kernel_size=(4, 4), stride=(2, 2), padding=same) ... \u001b[39m\n",
      "\u001b[32m[info | MIPVerify]: Applying ReLU() ...\u001b[39m\n",
      "Academic license - for non-commercial use only\n",
      "\u001b[32m[info | MIPVerify]: Applying Flatten() ... \u001b[39m\n",
      "\u001b[32m[info | MIPVerify]: Applying Linear(1568 -> 100) ... \u001b[39m\n",
      "\u001b[32m[info | MIPVerify]: Applying ReLU() ...\u001b[39m\n",
      "\u001b[32m[info | MIPVerify]: Applying Linear(100 -> 10) ... \u001b[39m\n",
      "Academic license - for non-commercial use only\n",
      "Academic license - for non-commercial use only\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Dict{Any,Any} with 11 entries:\n",
       "  :TargetIndexes      => [10]\n",
       "  :SolveTime          => 0.00461507\n",
       "  :TotalTime          => 4.68894\n",
       "  :Perturbation       => JuMP.VariableRef[noname noname … noname noname]…\n",
       "  :PerturbedInput     => JuMP.VariableRef[noname noname … noname noname]…\n",
       "  :TighteningApproach => \"lp\"\n",
       "  :PerturbationFamily => linf-norm-bounded-0.05\n",
       "  :SolveStatus        => INFEASIBLE_OR_UNBOUNDED\n",
       "  :Model              => A JuMP Model…\n",
       "  :Output             => JuMP.GenericAffExpr{Float64,JuMP.VariableRef}[0.000383…\n",
       "  :PredictedIndex     => 8"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MIPVerify.set_log_level!(\"debug\")\n",
    "\n",
    "MIPVerify.find_adversarial_example(\n",
    "    n1, \n",
    "    sample_image, \n",
    "    10, \n",
    "    Gurobi.Optimizer,\n",
    "    # OutputFlag=0 prevents any output from being printed out\n",
    "    Dict(\"OutputFlag\" => 0),\n",
    "    pp = MIPVerify.LInfNormBoundedPerturbationFamily(0.05),\n",
    "    norm_order = Inf,\n",
    "    tightening_algorithm = lp,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handling log output\n",
    "\n",
    "We use the logger provided by the [`Memento`](https://github.com/invenia/Memento.jl) package. Rather than just having logs written to STDOUT, you can choose to capture the output (for example, to write logs to file) by adding your own handlers to `MIPVerify.LOGGER`. See the [Memento documentation](https://invenia.github.io/Memento.jl/latest/) for more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Memento.Logger"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "typeof(MIPVerify.LOGGER)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.4.2",
   "language": "julia",
   "name": "julia-1.4"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}