{ "cells": [ { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "# Log output\n", "\n", "We use the same example as in the quickstart." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "using MIPVerify\n", "using Gurobi" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "data": { "text/plain": [ "10" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mnist = MIPVerify.read_datasets(\"MNIST\")\n", "n1 = MIPVerify.get_example_network_params(\"MNIST.n1\")\n", "sample_image = MIPVerify.get_image(mnist.test.images, 1)\n", "target_label_index = 10" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "## Viewing Log Output\n", "\n", "Our package logs to STDOUT at the `debug`, `info`, `notice` and `warn` levels. \n", "The default output level is to show logs only at the `notice` level and above, which is also the recommended level." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Rebuilding model from scratch. This may take some time as we determine upper and lower bounds for the input to each non-linear unit.\n", "\u001b[39m" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 2%|█ | ETA: 0:00:06\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\r\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:02\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Imposing relu constraint: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:01:48\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:02:20\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: The model built will be cached and re-used for future solves, unless you explicitly set rebuild=true.\n", "\u001b[39m" ] } ], "source": [ "d = MIPVerify.find_adversarial_example(n1, sample_image, target_label_index, GurobiSolver(), rebuild=true);" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "More information is available if we set the log level to `debug`." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Optimize a model with 3385 rows, 3256 columns and 71132 nonzeros\n", "Variable types: 3196 continuous, 60 integer (60 binary)\n", "Coefficient statistics:\n", " Matrix range [2e-05, 7e+02]\n", " Objective range [1e+00, 1e+00]\n", " Bounds range [1e+00, 1e+02]\n", " RHS range [4e-03, 7e+02]\n", "\n", "MIP start did not produce a new incumbent solution\n", "MIP start violates constraint R1024 by 1.000000000\n", "\n", "Presolve removed 2956 rows and 2227 columns\n", "Presolve time: 0.16s\n", "Presolved: 429 rows, 1029 columns, 61365 nonzeros\n", "Variable types: 969 continuous, 60 integer (60 binary)\n", "\n", "Root relaxation: objective 0.000000e+00, 223 iterations, 0.00 seconds\n", "\n", " Nodes | Current Node | Objective Bounds | Work\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\n", "\n", " 0 0 0.00000 0 4 - 0.00000 - - 0s\n", "Another try with MIP start\n", "H 0 0 36.9304736 0.00000 100% - 0s\n", " 0 0 0.00000 0 6 36.93047 0.00000 100% - 0s\n", "H 0 0 32.4680084 0.00000 100% - 0s\n", " 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s\n", " 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s\n", " 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s\n", " 0 2 0.00000 0 7 32.46801 0.00000 100% - 0s\n", "H 27 26 26.4992057 0.00000 100% 95.6 0s\n", "H 117 100 19.5583736 0.00000 100% 73.5 1s\n", "* 145 111 43 18.7741931 0.00000 100% 72.1 1s\n", "H 237 97 7.2802489 0.00000 100% 57.9 1s\n", "* 266 117 40 6.9633438 0.00000 100% 55.8 1s\n", "* 463 194 43 6.5385430 0.00000 100% 53.7 2s\n", " 901 439 4.46206 35 10 6.53854 0.00000 100% 58.2 5s\n", "H 1975 402 5.9198759 0.00000 100% 59.3 8s\n", " 2353 459 0.36395 32 9 5.91988 0.02050 100% 59.5 10s\n", "* 3375 661 49 5.7487698 0.42240 92.7% 56.1 12s\n", "* 3510 669 51 5.4974367 0.42240 92.3% 55.6 12s\n", " 4522 811 5.06223 33 10 5.49744 0.86529 84.3% 53.4 15s\n", "* 4731 846 46 5.2900951 1.22249 76.9% 52.3 15s\n", "* 5433 904 51 5.1129105 1.53829 69.9% 48.9 16s\n", "* 5719 936 45 4.9728818 1.65195 66.8% 47.7 17s\n", "* 5785 902 48 4.6601251 1.67856 64.0% 47.4 17s\n", " 7053 1025 3.63314 27 6 4.66013 2.04395 56.1% 44.8 20s\n", "* 8453 827 44 4.6418593 2.74366 40.9% 43.0 22s\n", " 9569 411 cutoff 25 4.64186 3.67506 20.8% 41.3 25s\n", "\n", "Cutting planes:\n", " Gomory: 3\n", " Projected implied bound: 6\n", " MIR: 2\n", " Flow cover: 1\n", "\n", "Explored 10523 nodes (420298 simplex iterations) in 26.66 seconds\n", "Thread count was 4 (of 4 available processors)\n", "\n", "Solution count 10: 4.64186 4.66013 4.97288 ... 6.96334\n", "\n", "Optimal solution found (tolerance 1.00e-04)\n", "Best objective 4.641859316319e+00, best bound 4.641859316319e+00, gap 0.0000%\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Rebuilding model from scratch. This may take some time as we determine upper and lower bounds for the input to each non-linear unit.\n", "\u001b[39m\u001b[32m[info | MIPVerify]: Applying Flatten() ... \n", "\u001b[39m\u001b[32m[info | MIPVerify]: Applying Linear(784 -> 40) ... \n", "\u001b[39m\u001b[32m[info | MIPVerify]: Applying ReLU() ...\n", "\u001b[39mAcademic license - for non-commercial use only\n", "\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 0.0\n", "\u001b[39m\u001b[32m[info | MIPVerify]: Applying Linear(40 -> 20) ... \n", "\u001b[39m\u001b[32m[info | MIPVerify]: Applying ReLU() ...\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 236.19499039114618\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 347.3507384582369\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 111.66031695101941\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 110.09320929282381\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 263.9686250131348\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 170.58623081063135\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 342.81111856390635\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 322.1609214959067\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 190.94181084213062\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 120.56886946816377\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 333.3978216299831\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 156.25265560651098\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 306.0828261286914\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 176.38206258926618\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 301.9881744933099\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 325.9390651275078\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 146.18208942669833\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 292.4320734889739\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 184.93657306973148\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 256.17035614083665\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 329.4706219653086\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 490.33543193659597\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 230.8574409869865\n", "\u001b[39m\u001b[32m[info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.7451049966147174.\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 240.98736242290283\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 443.7377089444059\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 546.9967283877565\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 561.8499789605797\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 252.88861567468402\n", "\u001b[39m\u001b[32m[info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 1.258999125666667.\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 368.8231767436639\n", "\u001b[39m\u001b[32m[info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.5343960501301519.\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 306.69327050758636\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 395.6692979013308\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 263.6311034209684\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 259.3597215144569\n", "\u001b[39m\u001b[32m[info | MIPVerify]: Hit user limit during solve to determine bounds. Multiplicative gap was 0.1117666858312083.\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 272.3179195538885\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 363.8405178338219\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 223.20192786473305\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 397.6660964044127\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 354.40977058244283\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 347.3454124031803\n", "\u001b[39m\u001b[34m[debug | MIPVerify]: Δu = 540.5128944047548\n", "\u001b[39m\u001b[32m[info | MIPVerify]: Applying Linear(20 -> 10) ... \n", "\u001b[39m\u001b[36m[notice | MIPVerify]: The model built will be cached and re-used for future solves, unless you explicitly set rebuild=true.\n", "\u001b[39m" ] }, { "data": { "text/plain": [ "Dict{Any,Any} with 10 entries:\n", " :PerturbedInput => JuMP.Variable[__anon__ __anon__ __anon__ __anon__ __an…\n", " :TargetIndexes => [10]\n", " :SolveStatus => :Optimal\n", " :TotalTime => 342.126\n", " :TighteningApproach => \"mip\"\n", " :Output => JuMP.GenericAffExpr{Float64,JuMP.Variable}[-0.01206386…\n", " :PredictedIndex => 8\n", " :Model => Minimization problem with:…\n", " :Perturbation => JuMP.Variable[__anon__ __anon__ __anon__ __anon__ __an…\n", " :PerturbationFamily => unrestricted" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\r\n", "Optimize a model with 3385 rows, 3256 columns and 71132 nonzeros\r\n", "Variable types: 3196 continuous, 60 integer (60 binary)\r\n", "Coefficient statistics:\r\n", " Matrix range [2e-05, 7e+02]\r\n", " Objective range [1e+00, 1e+00]\r\n", " Bounds range [1e+00, 1e+02]\r\n", " RHS range [4e-03, 7e+02]\r\n", "\r\n", "MIP start did not produce a new incumbent solution\r\n", "MIP start violates constraint R1024 by 1.000000000\r\n", "\r\n", "Presolve removed 2956 rows and 2227 columns\r\n", "Presolve time: 0.17s\r\n", "Presolved: 429 rows, 1029 columns, 61365 nonzeros\r\n", "Variable types: 969 continuous, 60 integer (60 binary)\r\n", "\r\n", "Root relaxation: objective 0.000000e+00, 212 iterations, 0.02 seconds\r\n", "\r\n", " Nodes | Current Node | Objective Bounds | Work\r\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\r\n", "\r\n", " 0 0 0.00000 0 3 - 0.00000 - - 0s\r\n", "Another try with MIP start\r\n", "H 0 0 36.9304736 0.00000 100% - 0s\r\n", "H 0 0 32.4680084 0.00000 100% - 0s\r\n", " 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s\r\n", " 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s\r\n", " 0 2 0.00000 0 7 32.46801 0.00000 100% - 0s\r\n", "H 106 61 16.8778596 0.00000 100% 61.7 1s\r\n", "H 150 81 6.3285068 0.00000 100% 64.0 1s\r\n", "* 772 401 41 6.2748878 0.00000 100% 51.5 2s\r\n", " 875 468 0.00000 28 9 6.27489 0.00000 100% 56.5 5s\r\n", "H 888 444 6.1453585 0.00000 100% 56.7 5s\r\n", " 2318 623 0.02050 30 12 6.14536 0.02050 100% 56.2 10s\r\n", "* 2577 683 52 6.0729847 0.02050 100% 54.6 10s\r\n", "* 3003 803 49 5.7567275 0.02050 100% 53.3 11s\r\n", " 4503 1259 0.72742 27 12 5.75673 0.33754 94.1% 50.0 15s\r\n", " 6702 1834 cutoff 36 5.75673 0.50354 91.3% 46.6 20s\r\n", " 8849 2358 4.79543 37 11 5.75673 0.66488 88.5% 46.4 25s\r\n", "* 9568 2095 50 4.6839523 0.66565 85.8% 46.4 26s\r\n", " 10924 2206 4.42918 31 4 4.68395 0.88975 81.0% 45.9 30s\r\n", " 12870 2458 3.07862 32 10 4.68395 1.11657 76.2% 46.1 35s\r\n", "*13768 2488 53 4.6601251 1.27564 72.6% 45.6 36s\r\n", " 14627 2653 cutoff 31 4.66013 1.35669 70.9% 45.2 41s\r\n", "H14661 2643 4.6418593 1.35669 70.8% 45.2 41s\r\n", " 16232 2662 3.78233 36 17 4.64186 1.66017 64.2% 44.4 45s\r\n", " 18398 2845 1.98944 23 8 4.64186 1.98944 57.1% 43.1 50s\r\n", " 20003 2806 cutoff 44 4.64186 2.28758 50.7% 42.6 55s\r\n", " 21951 2650 3.90135 27 15 4.64186 2.55568 44.9% 41.9 60s\r\n", " 24164 2427 3.11569 39 10 4.64186 2.98694 35.7% 40.6 65s\r\n", " 26303 2060 cutoff 36 4.64186 3.37603 27.3% 39.6 70s\r\n", " 28161 1541 cutoff 30 4.64186 3.69591 20.4% 39.1 75s\r\n", " 30006 846 3.98690 31 12 4.64186 3.97502 14.4% 38.6 80s\r\n", "\r\n", "Cutting planes:\r\n", " Gomory: 1\r\n", " Projected implied bound: 41\r\n", " MIR: 21\r\n", " Flow cover: 19\r\n", "\r\n", "Explored 31449 nodes (1197329 simplex iterations) in 84.54 seconds\r\n", "Thread count was 4 (of 4 available processors)\r\n", "\r\n", "Solution count 10: 4.64186 4.66013 4.68395 ... 32.468\r\n", "\r\n", "Optimal solution found (tolerance 1.00e-04)\r\n", "Best objective 4.641859316319e+00, best bound 4.641859316319e+00, gap 0.0000%\r\n" ] } ], "source": [ "MIPVerify.setloglevel!(\"debug\")\n", "d = MIPVerify.find_adversarial_example(n1, sample_image, target_label_index, GurobiSolver(), rebuild=true)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "When we select `rebuild=false`, we do not do any of the work in determining upper and lower bounds, and none of the `info` and `debug` logs are produced." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "Optimize a model with 3385 rows, 3256 columns and 71132 nonzeros\n", "Variable types: 3196 continuous, 60 integer (60 binary)\n", "Coefficient statistics:\n", " Matrix range [2e-05, 7e+02]\n", " Objective range [1e+00, 1e+00]\n", " Bounds range [1e+00, 1e+02]\n", " RHS range [4e-03, 7e+02]\n", "\n", "MIP start did not produce a new incumbent solution\n", "MIP start violates constraint R1024 by 1.000000000\n", "\n", "Presolve removed 2956 rows and 2227 columns\n", "Presolve time: 0.16s\n", "Presolved: 429 rows, 1029 columns, 61365 nonzeros\n", "Variable types: 969 continuous, 60 integer (60 binary)\n", "\n", "Root relaxation: objective 0.000000e+00, 212 iterations, 0.02 seconds\n", "\n", " Nodes | Current Node | Objective Bounds | Work\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\n", "\n", " 0 0 0.00000 0 3 - 0.00000 - - 0s\n", "Another try with MIP start\n", "H 0 0 36.9304736 0.00000 100% - 0s\n", "H 0 0 32.4680084 0.00000 100% - 0s\n", " 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s\n", " 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s\n", " 0 2 0.00000 0 7 32.46801 0.00000 100% - 0s\n", "H 106 61 16.8778596 0.00000 100% 61.7 1s\n", "H 150 81 6.3285068 0.00000 100% 64.0 1s\n", "* 772 401 41 6.2748878 0.00000 100% 51.5 2s\n", "H 888 444 6.1453585 0.00000 100% 56.7 4s\n", " 953 466 0.00000 16 5 6.14536 0.00000 100% 58.6 5s\n", " 2465 667 5.66918 34 5 6.14536 0.02050 100% 55.1 10s\n", "* 2577 683 52 6.0729847 0.02050 100% 54.6 10s\n", "* 3003 803 49 5.7567275 0.02050 100% 53.3 10s\n", " 4740 1353 0.96158 29 8 5.75673 0.33754 94.1% 49.1 15s\n", " 7176 1971 3.40887 37 11 5.75673 0.55686 90.3% 46.5 20s\n", " 9473 2472 4.86502 32 5 5.75673 0.66565 88.4% 46.5 25s\n", "* 9568 2095 50 4.6839523 0.66565 85.8% 46.4 25s\n", " 11702 2322 2.86913 33 9 4.68395 0.97990 79.1% 46.1 30s\n", "*13768 2488 53 4.6601251 1.27564 72.6% 45.6 34s\n", " 13993 2555 cutoff 29 4.66013 1.33004 71.5% 45.3 35s\n", "H14661 2643 4.6418593 1.35669 70.8% 45.2 39s\n", " 14760 2643 cutoff 38 4.64186 1.37322 70.4% 45.1 40s\n", " 16838 2691 1.69505 24 8 4.64186 1.69505 63.5% 44.0 45s\n", " 18962 2868 cutoff 27 4.64186 2.07242 55.4% 43.0 50s\n", " 20514 2814 4.08882 31 5 4.64186 2.37951 48.7% 42.3 55s\n", " 22468 2601 cutoff 35 4.64186 2.71056 41.6% 41.6 60s\n", " 24747 2361 cutoff 23 4.64186 3.08461 33.5% 40.3 65s\n", " 26908 1883 3.51087 27 3 4.64186 3.49281 24.8% 39.4 70s\n", " 28747 1429 cutoff 36 4.64186 3.70938 20.1% 39.0 75s\n", " 30599 580 cutoff 40 4.64186 4.16933 10.2% 38.3 80s\n", "\n", "Cutting planes:\n", " Gomory: 1\n", " Projected implied bound: 41\n", " MIR: 21\n", " Flow cover: 19\n", "\n", "Explored 31449 nodes (1197329 simplex iterations) in 82.56 seconds\n", "Thread count was 4 (of 4 available processors)\n", "\n", "Solution count 10: 4.64186 4.66013 4.68395 ... 32.468\n", "\n", "Optimal solution found (tolerance 1.00e-04)\n", "Best objective 4.641859316319e+00, best bound 4.641859316319e+00, gap 0.0000%\n" ] } ], "source": [ "d = MIPVerify.find_adversarial_example(n1, sample_image, target_label_index, GurobiSolver(), rebuild=false);" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "## Handling log output\n", "\n", "We use the logger provided by the [`Memento`](https://github.com/invenia/Memento.jl) package. Rather than just having logs written to STDOUT, you can choose to capture the output (for example, to write logs to file) by adding your own handlers to `MIPVerify.LOGGER`. See the [Memento documentation](https://invenia.github.io/Memento.jl/latest/) for more." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "data": { "text/plain": [ "Memento.Logger" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "typeof(MIPVerify.LOGGER)" ] } ], "metadata": { "kernelspec": { "display_name": "Julia 0.6.4", "language": "julia", "name": "julia-0.6" }, "language_info": { "file_extension": ".jl", "mimetype": "application/julia", "name": "julia", "version": "0.6.4" } }, "nbformat": 4, "nbformat_minor": 2 }