{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Log output\n", "\n", "We use the same example as in the quickstart." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "using MIPVerify\n", "using HiGHS" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "mnist = MIPVerify.read_datasets(\"MNIST\")\n", "n1 = MIPVerify.get_example_network_params(\"MNIST.n1\")\n", "sample_image = MIPVerify.get_image(mnist.test.images, 1);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Viewing Log Output\n", "\n", "Our package logs to STDOUT at the `debug`, `info`, `notice` and `warn` levels. \n", "The default output level is to show logs only at the `notice` level and above, which is also the recommended level." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n", "Running HiGHS 1.4.2 [date: 1970-01-01, git hash: f797c1ab6]\n", "Copyright (c) 2022 ERGO-Code under MIT licence terms\n", "Presolving model\n", "1779 rows, 1627 cols, 56716 nonzeros\n", "1779 rows, 1627 cols, 56716 nonzeros\n", "\n", "Solving MIP model with:\n", " 1779 rows\n", " 1627 cols (29 binary, 0 integer, 0 implied int., 1598 continuous)\n", " 56716 nonzeros\n", "\n", " Nodes | B&B Tree | Objective Bounds | Dynamic Constraints | Work \n", " Proc. InQueue | Leaves Expl. | BestBound BestSol Gap | Cuts InLp Confl. | LpIters Time\n", "\n", " 0 0 0 0.00% 0 inf inf 0 0 0 0 0.0s\n", " 0 0 0 0.00% 0 inf inf 0 0 13 1616 0.1s\n", " 0 0 0 0.00% 0 inf inf 3942 12 99 30330 14.6s\n", " T 15 0 3 1.61% 0 0.0498354344 100.00% 3951 12 109 39558 16.3s\n", " T 27 1 10 3.42% 0 0.0474230913 100.00% 3974 12 131 45590 17.4s\n", " T 45 4 20 77.27% 0.0381825316 0.0460846816 17.15% 4563 15 162 58011 20.1s\n", "\n", "Solving report\n", " Status Optimal\n", " Primal bound 0.0460846815889\n", " Dual bound 0.0460846815889\n", " Gap 0% (tolerance: 0.01%)\n", " Solution status feasible\n", " 0.0460846815889 (objective)\n", " 0 (bound viol.)\n", " 0 (int. viol.)\n", " 0 (row viol.)\n", " Timing 21.37 (total)\n", " 0.02 (presolve)\n", " 0.00 (postsolve)\n", " Nodes 53\n", " LP iterations 60163 (total)\n", " 19015 (strong br.)\n", " 707 (separation)\n", " 28049 (heuristics)\n" ] }, { "data": { "text/plain": [ "Dict{Any, Any} with 11 entries:\n", " :TargetIndexes => [10]\n", " :SolveTime => 21.369\n", " :TotalTime => 39.5667\n", " :Perturbation => [_[1] _[2] … _[27] _[28];;; _[29] _[30] … _[55] _[56];…\n", " :PerturbedInput => [_[785] _[786] … _[811] _[812];;; _[813] _[814] … _[83…\n", " :TighteningApproach => \"lp\"\n", " :PerturbationFamily => linf-norm-bounded-0.05\n", " :SolveStatus => OPTIMAL\n", " :Model => A JuMP Model…\n", " :Output => JuMP.AffExpr[-0.012063867412507534 _[1601] + 0.6606525…\n", " :PredictedIndex => 8" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10, \n", " HiGHS.Optimizer,\n", " Dict(),\n", " pp = MIPVerify.LInfNormBoundedPerturbationFamily(0.05),\n", " norm_order = Inf,\n", " tightening_algorithm = lp,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "More information is available if we set the log level to `debug`." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[34m[debug | Memento]: Logger(root) is already registered and force=false.\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n", "\u001b[32m[info | MIPVerify]: Applying Flatten() ... \u001b[39m\n", "\u001b[32m[info | MIPVerify]: Applying Linear(784 -> 40) ... \u001b[39m\n", "\u001b[32m[info | MIPVerify]: Applying ReLU() ...\u001b[39m\n", "\u001b[32m[info | MIPVerify]: Applying Linear(40 -> 20) ... \u001b[39m\n", "\u001b[32m[info | MIPVerify]: Applying ReLU() ...\u001b[39m\n", "\u001b[32m[info | MIPVerify]: Applying Linear(20 -> 10) ... \u001b[39m\n", "Running HiGHS 1.4.2 [date: 1970-01-01, git hash: f797c1ab6]\n", "Copyright (c) 2022 ERGO-Code under MIT licence terms\n", "Presolving model\n", "1779 rows, 1627 cols, 56716 nonzeros\n", "1779 rows, 1627 cols, 56716 nonzeros\n", "\n", "Solving MIP model with:\n", " 1779 rows\n", " 1627 cols (29 binary, 0 integer, 0 implied int., 1598 continuous)\n", " 56716 nonzeros\n", "\n", " Nodes | B&B Tree | Objective Bounds | Dynamic Constraints | Work \n", " Proc. InQueue | Leaves Expl. | BestBound BestSol Gap | Cuts InLp Confl. | LpIters Time\n", "\n", " 0 0 0 0.00% 0 inf inf 0 0 0 0 0.0s\n", " 0 0 0 0.00% 0 inf inf 0 0 13 1616 0.1s\n", " 0 0 0 0.00% 0 inf inf 3942 12 99 30330 14.3s\n", " T 15 0 3 1.61% 0 0.0498354344 100.00% 3951 12 109 39558 16.1s\n", " T 27 1 10 3.42% 0 0.0474230913 100.00% 3974 12 131 45590 17.1s\n", " T 45 4 20 77.27% 0.0381825316 0.0460846816 17.15% 4563 15 162 58011 20.2s\n", "\n", "Solving report\n", " Status Optimal\n", " Primal bound 0.0460846815889\n", " Dual bound 0.0460846815889\n", " Gap 0% (tolerance: 0.01%)\n", " Solution status feasible\n", " 0.0460846815889 (objective)\n", " 0 (bound viol.)\n", " 0 (int. viol.)\n", " 0 (row viol.)\n", " Timing 21.52 (total)\n", " 0.02 (presolve)\n", " 0.00 (postsolve)\n", " Nodes 53\n", " LP iterations 60163 (total)\n", " 19015 (strong br.)\n", " 707 (separation)\n", " 28049 (heuristics)\n" ] }, { "data": { "text/plain": [ "Dict{Any, Any} with 11 entries:\n", " :TargetIndexes => [10]\n", " :SolveTime => 21.5182\n", " :TotalTime => 22.2792\n", " :Perturbation => [_[1] _[2] … _[27] _[28];;; _[29] _[30] … _[55] _[56];…\n", " :PerturbedInput => [_[785] _[786] … _[811] _[812];;; _[813] _[814] … _[83…\n", " :TighteningApproach => \"lp\"\n", " :PerturbationFamily => linf-norm-bounded-0.05\n", " :SolveStatus => OPTIMAL\n", " :Model => A JuMP Model…\n", " :Output => JuMP.AffExpr[-0.012063867412507534 _[1601] + 0.6606525…\n", " :PredictedIndex => 8" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "MIPVerify.set_log_level!(\"debug\")\n", "\n", "MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10, \n", " HiGHS.Optimizer,\n", " Dict(),\n", " pp = MIPVerify.LInfNormBoundedPerturbationFamily(0.05),\n", " norm_order = Inf,\n", " tightening_algorithm = lp,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Handling log output\n", "\n", "We use the logger provided by the [`Memento`](https://github.com/invenia/Memento.jl) package. Rather than just having logs written to STDOUT, you can choose to capture the output (for example, to write logs to file) by adding your own handlers to `MIPVerify.LOGGER`. See the [Memento documentation](https://invenia.github.io/Memento.jl/latest/) for more." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Memento.Logger" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "typeof(MIPVerify.LOGGER)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Julia 1.7.1", "language": "julia", "name": "julia-1.7" }, "language_info": { "file_extension": ".jl", "mimetype": "application/julia", "name": "julia", "version": "1.7.1" } }, "nbformat": 4, "nbformat_minor": 2 }