{ "cells": [ { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "# Finding adversarial examples, in depth\n", "\n", "In the quickstart, we used the default parameters for `find_adversarial_example`. Using the same example from the quickstart, we explore how to get more out of the function." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "using MIPVerify\n", "using Gurobi\n", "using JuMP\n", "using Images\n", "\n", "mnist = MIPVerify.read_datasets(\"MNIST\")\n", "n1 = MIPVerify.get_example_network_params(\"MNIST.n1\")\n", "sample_image = MIPVerify.get_image(mnist.test.images, 1);\n", "\n", "function print_summary(d::Dict)\n", " # Helper function to print out output\n", " obj_val = getobjectivevalue(d[:Model])\n", " solve_time = getsolvetime(d[:Model])\n", " println(\"Objective Value: $(@sprintf(\"%.6f\", obj_val)), Solve Time: $(@sprintf(\"%.2f\", solve_time))\")\n", "end\n", "\n", "function view_diff(diff::Array{<:Real, 2})\n", " n = 1001\n", " colormap(\"RdBu\", n)[ceil.(Int, (diff+1)/2*n)]\n", "end;" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "## `find_adversarial_example`\n", "\n", "`find_adversarial_example` takes four positional arguments\n", "\n", "```\n", "find_adversarial_example(nn, input, target_selection, main_solver)\n", "```\n", "\n", "It also takes named arguments, each with the default value specified.\n", "\n", "```\n", "norm_order = 1\n", "tolerance = 0\n", "rebuild = false\n", "invert_target_selection = false\n", "pp = MIPVerify.UnrestrictedPerturbationFamily()\n", "tightening_algorithm = mip\n", "tightening_solver: same as main solver, but with output suppressed and a time limit of 20s per solve.\n", "```\n", "\n", "We explore what each of these options allow us to do." ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "# Basic Options" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "## Specifying target categories for the adversarial example\n", "\n", "`target_selection` and `invert_target_selection` control what the category we want the adversarial example to be classified in.\n", "\n", "**Specification**: `target_selection` accepts either a single integer or a list of integers.\n", "\n", "For example, if I wanted the original image (which is the digit 7) to be classified as the digit 8 or 9, I could run two separate solves with `target_selection=9` and `target_selection=10` (Julia is 1-indexed), finding closest adversarial examples at distance `13.763` and `4.642` ..." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "Objective Value: 13.762581, Solve Time: 177.25\n" ] } ], "source": [ "d = MIPVerify.find_adversarial_example(n1, sample_image, 9, GurobiSolver(OutputFlag=0))\n", "# OutputFlag=0 prevents any output from being printed out\n", "print_summary(d)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "Objective Value: 4.641859, Solve Time: 55.27\n" ] } ], "source": [ "d = MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver(OutputFlag=0));\n", "print_summary(d)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "Or I can can pass the targets as `target_selection = [9, 10]`, where the same optimal value of `4.642` is found.\n", "\n", "Solve times for multiple target labels are typically on par with or faster than the aggregate solve times when solving with each target label in sequence." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9, 10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39m" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 50%|████████████ | ETA: 0:00:00\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\r\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:19\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\r\n", "Objective Value: 4.641859, Solve Time: 77.31\n" ] } ], "source": [ "d = MIPVerify.find_adversarial_example(n1, sample_image, [9, 10], GurobiSolver(OutputFlag=0));\n", "print_summary(d)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "A common use case is to have the adversarial example being in any category but the original:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [1, 2, 3, 4, 5, 6, 7, 9, 10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:02:37\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:01:47\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\r\n", "Objective Value: 4.641859, Solve Time: 325.92\n" ] } ], "source": [ "d = MIPVerify.find_adversarial_example(n1, sample_image, [1, 2, 3, 4, 5, 6, 7, 9, 10], GurobiSolver(OutputFlag=0))\n", "print_summary(d)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "Rather than typing the full list of other categories, we can set `target_selection = 8`, and `invert_target_selection = true`." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [1, 2, 3, 4, 5, 6, 7, 9, 10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:02:36\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:01:47\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\r\n", "Objective Value: 4.641859, Solve Time: 278.47\n" ] } ], "source": [ "d = MIPVerify.find_adversarial_example(n1, sample_image, 8, GurobiSolver(OutputFlag=0), \n", " invert_target_selection=true)\n", "print_summary(d)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "## More precise control over activations in output layer\n", "### Default Behavior\n", "The default option for `tolerance` is 0. This means that the activations in the softmax layer will take its maximum value for the target label (or at least one of them if more than one is specified), but that this maximum may not be unique.\n", "\n", "In the example below, notice how the activations `y[8]` and `y[10]` (corresponding to labels 7 and 9) are equal, at 4.56707." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "Objective Value: 4.641859, Solve Time: 52.90\n" ] }, { "data": { "text/plain": [ "10-element Array{Float64,1}:\n", " 1.05198 \n", " 0.785344\n", " 0.313807\n", " 0.427011\n", " 0.369729\n", " 0.256145\n", " 0.66623 \n", " 4.56707 \n", " -0.147984\n", " 4.56707 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver(OutputFlag=0))\n", "print_summary(d)\n", "perturbed_sample_image = getvalue(d[:PerturbedInput])\n", "y = perturbed_sample_image |> n1" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "### Ensuring difference in activation between target labels and non-target labels\n", "\n", "If we want to ensure that the maximum is unique, we can add a small tolerance. This is the minimum difference between the activation of the target label (or the maximum activation of any of the target labels) and any non-target label. In the example below, with a tolerance of 0.001, notice how `y[8] = 4.5683 = 0.0010 + 4.5673 = tolerance + y[10]`.\n", "\n", "_This increases the objective value slightly (since the closest adversarial example with the required tolerance must be slightly further away.)_" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "Objective Value: 4.643283, Solve Time: 63.39\n" ] }, { "data": { "text/plain": [ "10-element Array{Float64,1}:\n", " 1.05292 \n", " 0.785758\n", " 0.313776\n", " 0.427317\n", " 0.370021\n", " 0.256382\n", " 0.666551\n", " 4.5673 \n", " -0.14813 \n", " 4.5683 " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver(OutputFlag=0), \n", " tolerance = 0.001)\n", "print_summary(d)\n", "perturbed_sample_image = getvalue(d[:PerturbedInput])\n", "y = perturbed_sample_image |> n1" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "## Restricting the Family of Perturbations\n", "\n", "### Unrestricted Perturbations\n", "\n", "The standard threat model is to allow each pixel to be perturbed independently, which is what happens by default:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", " 53.041781 seconds (331.52 k allocations: 17.399 MiB, 0.03% gc time)\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAAAJiS0dEAP+Hj8y/AAAB1klEQVRo3u3Zu2sUURzF8Y8mKQQbDQoWPioLLSSIICSCj0ZjYWH+haRRy4B/glhaWNjbCEIgIiIoRHwVaSS+MRYqIghaqJBCiMWMsJHsuuOE3Ts/72nm7swwX86cPZd7GbKysvqtdVVuPo8LNYHre+0wPjCr+erYwwcYXWNg/FrkHmZl/YdqO5dOYBIfsYSr+IQ3NYHxp7aeA9tm+Ba7/jj3Dc/+8sAPuIj5VBzGBw62uzCJfXiOPRjBYRzEe2xvufcnPmNb+fudnGEPtaKHyzovVDcpspzHgZbzS3iNF9iMs7icisP4wEp7/E46jWt4iiP4korD+MA1yXArFsrjBK6n5DA+cLD+IziDLfiKV6k5jA+s3cNR3MWQYs1zLzWH8YG1eziuyO8OHq1yfQpX+ukwPrBWDzfgPvbiKB6m6DA+sFYPpxV7jVu6y2+5Hw7jA/+5hycxgx84YfV59Lda953xX2kzejiMSxjATZ3zY+UfJf4rTb+HA3iM/VjE8fKYrMP4wMoZ7sbLcnwKs6k7jA+sNJfuxO1yPI0bTXAYH1gpwynsKMdzirVK8g7jA7vO8BDONdFhfGDXGY5hYzlexPemOIwPrLy3eIJj2n9bSs5hfGBW8/ULlPE/E/nN6ocAAAAASUVORK5CYII=", "text/plain": [ "28×28 Array{Gray{Float64},2}:\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " ⋮ ⋱ \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(1.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = @time MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver(OutputFlag=0));\n", "perturbed_sample_image = getvalue(d[:PerturbedInput])\n", "colorview(Gray, perturbed_sample_image[1, :, :, 1])" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "### $L_\\infty$-norm Bounded Perturbations\n", "\n", "We can bound the $L_\\infty$-norm of the perturbation." ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "As long as the size of the $L_\\infty$-norm bound we choose is larger than the actual ($L_\\infty$-)minimal perturbation, we will find the same result, and often more quickly." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", " 55.975023 seconds (370.78 k allocations: 19.352 MiB, 0.02% gc time)\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAAAJiS0dEAP+Hj8y/AAACx0lEQVRo3u3av69MQRQH8M/y4lceBYkIeToSGhERBYWOUCiUGhoVjdBIFP4MnUgkGp2IUIgfQSUR8TNoUNC+LSQkFPdeb3Z25u42u/dlsifZ7Mzcc2Zyv9+cMzPn3N48+gZlvv7vB+1cP7brJ+zDsRWmLOUvOBfyEPIZ8xOPz0f/WmzD5+VDOn0Ow07jL38M+1Kb/8X+2cwxF/QbnfIhnfqCvZCrRtpiYk5SMTelXz6k3cTSMC7mfKw/YqJcTO78DctfsBfiPconR/EomkfCpnxIp88hg5iTPk+GPPcTNiK9WBq78iGdfizNcXIIV/ASv3CzHv+UmSj25XBM8Kx8SLvxQ4ax/4k1dX89Fuvfm5bJtuAbLuC9tM+WD2l359JQ+jiAPXiL3diLw9iGr1gI9P9gJTbU/Wu4KJ0TKB/S5cEhafwXVFzewX6VE/9VxdqPeIeNOIcbtV28n5YPafextO280jyPbeAkruMLdmGV9Hm2fEi747CRcfiLdTbjNdbhNG632JcPabe5NvJ+mOOvj6sq/n7jQ8auszcsf8HeOPm0ttrFQdyr28fxWD7edvKG5S84ULcY5x4fjx2r/1/geYve7EwzMRnYD3Pc5fhci6fYiSN4ltCZ3Q8nLnMM141ScTXF4yUVf08s8ZfL081qwBOTgbvFODnthovjuIXvOIMH0rWq2X44cRnyw5TE9f3VuFz376r4Y9hvRf1ZvnQikrzjj5KH2IfPOIofI/Rn++FEpZerM+Ri61ZVPhtOqPiMbdrORuVD2m0dP5QUf5twv25fssRfw1nb91TNNznlQ9rNfpi6k8d3RTiL7XX7UWAzKscazlk+pMsj19ZIGBN34LylWqKMHu2clg9pdxzmvqtpxg/V7UXVWSZ1d8zlVDt9w/IX/M9hLk8a+9QrnFLV62O9tnbTLx/S5ZXzbuMjJePYlw/p1Bf8B4PsuR5pg3hSAAAAAElFTkSuQmCC", "text/plain": [ "28×28 Array{Gray{Float64},2}:\n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " ⋮ ⋱ \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) … Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) … Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0) " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = @time MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver(OutputFlag=0),\n", " norm_order=Inf);\n", "perturbed_sample_image = getvalue(d[:PerturbedInput])\n", "colorview(Gray, perturbed_sample_image[1, :, :, 1])" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", " 3.161656 seconds (304.00 k allocations: 17.236 MiB)\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAAAJiS0dEAP+Hj8y/AAACx0lEQVRo3u3av69MQRQH8M/y4lceBYkIeToSGhERBYWOUCiUGhoVjdBIFP4MnUgkGp2IUIgfQSUR8TNoUNC+LSQkFPdeb3Z25u42u/dlsifZ7Mzcc2Zyv9+cMzPn3N48+gZlvv7vB+1cP7brJ+zDsRWmLOUvOBfyEPIZ8xOPz0f/WmzD5+VDOn0Ow07jL38M+1Kb/8X+2cwxF/QbnfIhnfqCvZCrRtpiYk5SMTelXz6k3cTSMC7mfKw/YqJcTO78DctfsBfiPconR/EomkfCpnxIp88hg5iTPk+GPPcTNiK9WBq78iGdfizNcXIIV/ASv3CzHv+UmSj25XBM8Kx8SLvxQ4ax/4k1dX89Fuvfm5bJtuAbLuC9tM+WD2l359JQ+jiAPXiL3diLw9iGr1gI9P9gJTbU/Wu4KJ0TKB/S5cEhafwXVFzewX6VE/9VxdqPeIeNOIcbtV28n5YPafextO280jyPbeAkruMLdmGV9Hm2fEi747CRcfiLdTbjNdbhNG632JcPabe5NvJ+mOOvj6sq/n7jQ8auszcsf8HeOPm0ttrFQdyr28fxWD7edvKG5S84ULcY5x4fjx2r/1/geYve7EwzMRnYD3Pc5fhci6fYiSN4ltCZ3Q8nLnMM141ScTXF4yUVf08s8ZfL081qwBOTgbvFODnthovjuIXvOIMH0rWq2X44cRnyw5TE9f3VuFz376r4Y9hvRf1ZvnQikrzjj5KH2IfPOIofI/Rn++FEpZerM+Ri61ZVPhtOqPiMbdrORuVD2m0dP5QUf5twv25fssRfw1nb91TNNznlQ9rNfpi6k8d3RTiL7XX7UWAzKscazlk+pMsj19ZIGBN34LylWqKMHu2clg9pdxzmvqtpxg/V7UXVWSZ1d8zlVDt9w/IX/M9hLk8a+9QrnFLV62O9tnbTLx/S5ZXzbuMjJePYlw/p1Bf8B4PsuR5pg3hSAAAAAElFTkSuQmCC", "text/plain": [ "28×28 Array{Gray{Float64},2}:\n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " ⋮ ⋱ \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) … Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) … Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0) " ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = @time MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver(OutputFlag=0),\n", " pp = MIPVerify.LInfNormBoundedPerturbationFamily(0.05), norm_order=Inf);\n", "perturbed_sample_image = getvalue(d[:PerturbedInput])\n", "colorview(Gray, perturbed_sample_image[1, :, :, 1])" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "If the $L_\\infty$-norm bound you choose is smaller than the actual minimal perturbation, the problem is Infeasible, with a status code of either `:Infeasible` or `:InfeasibleOrUnbounded`. (Make sure you cover both cases!)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", " 0.401018 seconds (287.74 k allocations: 16.295 MiB, 3.33% gc time)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1m\u001b[33mWARNING: \u001b[39m\u001b[22m\u001b[33mGurobi reported infeasible or unbounded. Set InfUnbdInfo=1 for more specific status.\u001b[39m\n", "\u001b[1m\u001b[33mWARNING: \u001b[39m\u001b[22m\u001b[33mNot solved to optimality, status: InfeasibleOrUnbounded\u001b[39m\n" ] } ], "source": [ "d = @time MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver(OutputFlag=0),\n", " pp = MIPVerify.LInfNormBoundedPerturbationFamily(0.03), norm_order=Inf);" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "data": { "text/plain": [ ":InfeasibleOrUnbounded" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d[:SolveStatus]" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "### Blurring Perturbations\n", "\n", "We can restrict the perturbations to a blur with a 5x5 kernel instead. (We are still minimizing over the norm of the perturbation.)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", " 1.801206 seconds (479.47 k allocations: 27.147 MiB, 1.31% gc time)\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAAAJiS0dEAP+Hj8y/AAACSUlEQVRo3u3avWsUURQF8N8mIYjGjxijGPwiCLZ+FVYWlopoo2CnraD/hI2t2FnZpNJKhKggglgoViKKlZBCjJKgMSYxCZtY3BEXycbZBJK3jznwYHbmzh7OHM68+2aGChUqVKhQoUKF9kdtuYN7MIgZLGAEv/BzFYQda60wf8JlPTyHXiwWYx5z+PafP53BhyZ1+V/SNSfsanagH52o47vI30b0YD+mi99/sIhZbBCZnVV5uEZo6uEgJsR9sy78nMYUtmMcOxrq6/iBU/gi8pqEwvwJl/Rwn5gLOzCJMeFhp8jZp+LEsaJ+UXh2AAMityOpKMyfcEkPjwivoK8Y8yKDPcLXzQ31C8W4Lnx9rXnfk/8lTcPDV6KXmcDWYnsXtgkfextqF4S/p3G0OPedyGYSCvMnrJUt7PZ3Huxr2F8X2Xwp+qDLGCr2J6Ewf8KusoVzGC22R/85dkn0N+/xWHP/1kVh/oSlc9gMu/FCrDfO42FqCvMnLJ3DZrgi+ti3Yi5MTmH+hKvK4WEMix7nDJ6mqDB/whXnsBvXRL/6CM9TVZg/4Yo8rOECzornaTfE+iJJhfkTrsjDXlwVveht5ebBdVOYP2HLHm7CTZwQ/egtsc5PVmH+hC17eBwXxf10CB9TV5g/YUse7sUdbMEbPGkHhfkTtuThSRwsth+I9xPJK8yfsLSHA2I9X8Mz3BXPxJNXmD9haQ+PifeG93AfX7XWy6ybwvwJS3m4E4fE9xjjYj0/1S4K8ycs5WE/PhdjUnzz1DYK8yf8DZfnacuCLWDTAAAAAElFTkSuQmCC", "text/plain": [ "28×28 Array{Gray{Float64},2}:\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) … Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) … Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) … Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " ⋮ ⋱ \n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) … Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) … Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)\n", " Gray{Float64}(-0.0) Gray{Float64}(-0.0) Gray{Float64}(-0.0)" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = @time MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver(OutputFlag=0),\n", " pp = MIPVerify.BlurringPerturbationFamily((5, 5)));\n", "perturbed_sample_image = getvalue(d[:PerturbedInput])\n", "colorview(Gray, perturbed_sample_image[1, :, :, 1])" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAMAAADxPgR5AAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAABTVBMVEX5+Pna7/u63/fD5PnF5fnh8/zo9v2p1fSXy/GZzPGf0POl0/S23fe/4vjk9P3/yr3/ppX/x7r/wbPr9/3N6fqdz/KWyvGg0POYy/GYzPHG5fnqXk7WRTnZST3XRjrnWUr/i3n/noz/iXf/no3/m4n/jHr/4Ni73/fL6PrY7vvl9P3/49z/9vP/ybv/n47nWUvSQjbsYFHpXE3uZFT/saD/6uPn9f2i0fO+4fjg8vy84Pjh8vzc8Pz/4Nf/w7X/3NPx+v7e8fzP6vqn1fSo1fSez/Km1PSv2fb/7+rt+P7u+P7p9v3w+f7/6uT/5d7/6eL/5d3/9fLy+v7/9PHQ6vrX7vv/6+b/5+D/6eP/1cr/9fP/5t/d8fz/7uj/8u6bzvLW7fv/2tCr1/X/0cX/yLv/2c//sJ//t6f/hnT/8/D/qJfM6fqq1vXm9f3////eL8B3AAAAAWJLR0RuIg9RFwAAAbVJREFUaN7t2VdP3FAURWEglFACIcDQIYROaKH3EnoNHUJvoef/v2ZtaUayECII3StjdL4XXyQ0y5ZsH884KckYY4wx5p1LRgo+IBVpSIcFoxPMwEdkxmUhGzn/8Qm5sGC4wTx8Rj6+oACFKEIMxYgFlEA7pHUpymDBcIPlqEAlqqBtNb6iBt9QG1CHejSgEU2wYHjBZrTgO1rRhnZ0oBM/oG1FnHamC93oQS/6YMHwgrppFz3SjwEMYgjDASPQoB7FGMYxAQuGF5zET+iDtFVoCgpPI7gjM5jFHHSCzWMBL45Z0EvwKfoQ7cgiJgN0Ei1B/6Og1suw4NsLPmcFiq3iF7zGLOicHprWsA4N3Q1YMFpB3aw3oUGsBy+vMQs6twUNaa31hdX70VnQKQ3kbST+1mC2YHSC+qKii3wHuug1fL0enQWd07DdhdZ78P7ga0GndJP+jX0cQFuvR2dB5w5xBK0Tw9eC0Qnqgj+G1ic4hQWjFTxDYtie41VvQS0YWlA/oF9A60voxyCvg9eCzv3BFa5xA70I8xazoHP6If0Wd9AJ4/3brgWdu0fi5dYD/sKCbyb4D3kRP8+xeXrHAAAAAElFTkSuQmCC", "text/plain": [ "28×28 Array{RGB{Float64},2}:\n", " RGB{Float64}(0.974598,0.972607,0.975359) … RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) … RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) … RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " ⋮ ⋱ \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) … RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) … RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "diff = getvalue(d[:Perturbation])\n", "view_diff(diff[1, :, :, 1])" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "## Minimizing Over Different Norms\n", "### $l_1$\n", "By default, we minimize the $l_1$ norm of the perturbation. This generally encourages sparsity in the perturbations. \n", "\n", "In this case, the minimum $l_1$ norm perturbation required for the image to be classified as a `9` is `4.641859.`" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "Objective Value: 4.641859, Solve Time: 53.11\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAAAJiS0dEAP+Hj8y/AAAB1klEQVRo3u3Zu2sUURzF8Y8mKQQbDQoWPioLLSSIICSCj0ZjYWH+haRRy4B/glhaWNjbCEIgIiIoRHwVaSS+MRYqIghaqJBCiMWMsJHsuuOE3Ts/72nm7swwX86cPZd7GbKysvqtdVVuPo8LNYHre+0wPjCr+erYwwcYXWNg/FrkHmZl/YdqO5dOYBIfsYSr+IQ3NYHxp7aeA9tm+Ba7/jj3Dc/+8sAPuIj5VBzGBw62uzCJfXiOPRjBYRzEe2xvufcnPmNb+fudnGEPtaKHyzovVDcpspzHgZbzS3iNF9iMs7icisP4wEp7/E46jWt4iiP4korD+MA1yXArFsrjBK6n5DA+cLD+IziDLfiKV6k5jA+s3cNR3MWQYs1zLzWH8YG1eziuyO8OHq1yfQpX+ukwPrBWDzfgPvbiKB6m6DA+sFYPpxV7jVu6y2+5Hw7jA/+5hycxgx84YfV59Lda953xX2kzejiMSxjATZ3zY+UfJf4rTb+HA3iM/VjE8fKYrMP4wMoZ7sbLcnwKs6k7jA+sNJfuxO1yPI0bTXAYH1gpwynsKMdzirVK8g7jA7vO8BDONdFhfGDXGY5hYzlexPemOIwPrLy3eIJj2n9bSs5hfGBW8/ULlPE/E/nN6ocAAAAASUVORK5CYII=", "text/plain": [ "28×28 Array{Gray{Float64},2}:\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " ⋮ ⋱ \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(1.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver(OutputFlag=0));\n", "print_summary(d)\n", "perturbed_sample_image = getvalue(d[:PerturbedInput])\n", "colorview(Gray, perturbed_sample_image[1, :, :, 1])" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "We also show the difference between the perturbed image and the original image. Red is areas of decreased brightness and blue is areas of increased brightness." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwBAMAAAA0zul4AAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAElBMVEX5+Pl+uuozcbYJLFiRx+////9ShjGxAAAAAWJLR0QF+G/pxwAAAENJREFUWMPt1UEJADAMBMFYiIVaiIX411QPLRSy3fkv3O8iBJbJD8nW4oeS9EIVP5Q0UTc//MegyzgKh8y8CiVJEsAGBXEIgdrWYGoAAAAASUVORK5CYII=", "text/plain": [ "28×28 Array{RGB{Float64},2}:\n", " RGB{Float64}(0.974598,0.972607,0.975359) … RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) … RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) … RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " ⋮ ⋱ \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.0358504,0.171831,0.345409)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) … RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) … RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359) " ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "diff = getvalue(d[:Perturbation])\n", "view_diff(diff[1, :, :, 1])" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "### $l_\\infty$\n", "\n", "We can also minimize over the $l_\\infty$ norm. This generally results in large patches of the image being changed. \n", "\n", "In this case, the minimum $l_\\infty$ norm perturbation required for the image to be classified as a `9` is `0.046085.`" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "Objective Value: 0.046085, Solve Time: 56.44\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAAAJiS0dEAP+Hj8y/AAACx0lEQVRo3u3av69MQRQH8M/y4lceBYkIeToSGhERBYWOUCiUGhoVjdBIFP4MnUgkGp2IUIgfQSUR8TNoUNC+LSQkFPdeb3Z25u42u/dlsifZ7Mzcc2Zyv9+cMzPn3N48+gZlvv7vB+1cP7brJ+zDsRWmLOUvOBfyEPIZ8xOPz0f/WmzD5+VDOn0Ow07jL38M+1Kb/8X+2cwxF/QbnfIhnfqCvZCrRtpiYk5SMTelXz6k3cTSMC7mfKw/YqJcTO78DctfsBfiPconR/EomkfCpnxIp88hg5iTPk+GPPcTNiK9WBq78iGdfizNcXIIV/ASv3CzHv+UmSj25XBM8Kx8SLvxQ4ax/4k1dX89Fuvfm5bJtuAbLuC9tM+WD2l359JQ+jiAPXiL3diLw9iGr1gI9P9gJTbU/Wu4KJ0TKB/S5cEhafwXVFzewX6VE/9VxdqPeIeNOIcbtV28n5YPafextO280jyPbeAkruMLdmGV9Hm2fEi747CRcfiLdTbjNdbhNG632JcPabe5NvJ+mOOvj6sq/n7jQ8auszcsf8HeOPm0ttrFQdyr28fxWD7edvKG5S84ULcY5x4fjx2r/1/geYve7EwzMRnYD3Pc5fhci6fYiSN4ltCZ3Q8nLnMM141ScTXF4yUVf08s8ZfL081qwBOTgbvFODnthovjuIXvOIMH0rWq2X44cRnyw5TE9f3VuFz376r4Y9hvRf1ZvnQikrzjj5KH2IfPOIofI/Rn++FEpZerM+Ri61ZVPhtOqPiMbdrORuVD2m0dP5QUf5twv25fssRfw1nb91TNNznlQ9rNfpi6k8d3RTiL7XX7UWAzKscazlk+pMsj19ZIGBN34LylWqKMHu2clg9pdxzmvqtpxg/V7UXVWSZ1d8zlVDt9w/IX/M9hLk8a+9QrnFLV62O9tnbTLx/S5ZXzbuMjJePYlw/p1Bf8B4PsuR5pg3hSAAAAAElFTkSuQmCC", "text/plain": [ "28×28 Array{Gray{Float64},2}:\n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " ⋮ ⋱ \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) … Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) … Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0) " ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver(OutputFlag=0),\n", " norm_order=Inf);\n", "print_summary(d)\n", "perturbed_sample_image = getvalue(d[:PerturbedInput])\n", "colorview(Gray, perturbed_sample_image[1, :, :, 1])" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwBAMAAAA0zul4AAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAJ1BMVEXo9v35+Pnr9/3/7+rx+v7/8Ovw+f7t+P7s9/3v+f7/9PH/9fP////ogXYvAAAAAWJLR0QMgbNRYwAAAV5JREFUWMPVl1EZwjAMhGMBC1iohVmYhVmYBSzMwixgDkoXcsmVdy4PsDX5y/ddkjaY2e0ys/5slr/dF3736YDoCDyHx0pE6oH95X5HWUKI2HJEXDFCYM1lBKIgvK4EZglmGKZeE6yl/RultpcBuah9q/DkVv42sgyIB1JrZu1jtaWzNPBLEiAuDmxZGln3t0Z3lQiIKW4/zGx5WxZSE6zLHVlX3ABbXgmMl5pV3Hfb+kZ4kKmBpBLpxn5FEF341p9GgaNoWqA7+QL10H33Jp52hQBI1yWZS1MuITmwDvFoj0cVEsQRAXGJCx6l8Qg9cD7YjvDj8GMYh0Q9MMogiqGbD0u8IYy5IiCj3c4TRwaWTQms7eyWsZr+yTnwx+BcGseeT76Q0gkgAuKSh1VpqpXJSgTERHsL+/jOo7AeiO2c/5xh8U/EEQIxzS7OuGwQi8/vgSwChiTszoWR/ULgCzIyXaB8LVeXAAAAAElFTkSuQmCC", "text/plain": [ "28×28 Array{RGB{Float64},2}:\n", " RGB{Float64}(0.911214,0.963766,0.992437) … RGB{Float64}(0.911214,0.963766,0.992437)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.911214,0.963766,0.992437)\n", " RGB{Float64}(0.911214,0.963766,0.992437) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.911214,0.963766,0.992437) … RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.911214,0.963766,0.992437) RGB{Float64}(0.911214,0.963766,0.992437)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.911214,0.963766,0.992437)\n", " RGB{Float64}(0.911214,0.963766,0.992437) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.911214,0.963766,0.992437) … RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " ⋮ ⋱ \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.911214,0.963766,0.992437)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.911214,0.963766,0.992437)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.911214,0.963766,0.992437)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.911214,0.963766,0.992437)\n", " RGB{Float64}(0.974598,0.972607,0.975359) … RGB{Float64}(0.911214,0.963766,0.992437)\n", " RGB{Float64}(0.911214,0.963766,0.992437) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.911214,0.963766,0.992437)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) … RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.911214,0.963766,0.992437)\n", " RGB{Float64}(0.911214,0.963766,0.992437) RGB{Float64}(0.974598,0.972607,0.975359)" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "diff = getvalue(d[:Perturbation])\n", "view_diff(diff[1, :, :, 1])" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "### $l_2$\n", "With solvers that can handle MIQPs (like Gurobi), we can minimize over the $l_2$ norm. This generally takes a bit more time. \n", "\n", "In this case, the minimum $l_2$ norm perturbation required for the image to be classified as a `9` is `0.705367 = sqrt(0.497542).`" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "Objective Value: 0.497542, Solve Time: 197.97\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAAAJiS0dEAP+Hj8y/AAAER0lEQVRo3u2az4scRRTHP7XTvTO7bnY20QQVPXkS9CAeRIRATopgID/wH/DkTfTgUfAv8J8QPIogehAPG/cQT4KBJIrgQUFFkk12s8n09vR4+L63VTM72TnNtBTzoJmp6qqurvet76v3XnUIwIhxSeu6QGV1DVAApZUPHtOvY21HwIr9D9ZmhQVL/gMWKX6u7wI4tLqOXbWVa7tKK28Bu9bH2wynPNPXQP4qXTyGacE5dIh0vg48QDg5RgXwLxHnh8mbOw+du6v2PwD37Dd/lS58wLCK+FEnlWsImy1gn8i7muN2N31zx9zXwpBxu9zKDPMfsPC9DqAHPEreYgPZyTPAnRkPahDeENfA/2KG+Q8Y0sIpxLVU/45HSdwjZ4lz9izwT9szzH/AALJ7ATgN7CEuui10n6ZKOqS2sWPlEY+3sxDXQv4qXTyGPuIqwq5E2L0DfAT8iPT/OfA38NvE2wa76qSuIa4Bl02WPs2c5MiWrgMD5EceIN/zDMKhQPysgJ+Tjit2OV8D8CfwKXCDuL+WQN/a5a/Sxfs0awibEfJhBnbjMvAywuxV4BXgvF23gecR3s6/O8CzCMc/gA8RfgGtgXtoP81fpe3E+BsIwy3grpW3gR/sjXYQh54GXgKuI1w9lhwAt4Bf7Bk3kgFKe7b7Q/mrtD1b2kV8eQA8iXDxvEqKQZV07iC894GrwBfAT8AFFJPA8Tgjf5W2G1uAeOQ2sLY3qq08JPqePeAJhPVTCLs+8C7wpfUbJG0DS790TlJMVuwifnWJmLl/6YCfsroK+T/vI/wOgN8R92qib/qozRnmP2DoIL//LoovqimN/KxiiLDZRJzqon3xe2v3NvCdtWus7hzx3GO3jRnmP2AxRPhBxM9zMiXCaZDc81x3Y9ebVr8NXEO8bdB6GFrfHootW5lh/gMWad5lHdnDBtnDUVIOCIsG2cYRwumi1X1C9HvWkB0t0fpYs7bLs6e5yNj5YUX0I2vG86Opb1pY2w+AF5H93LF7Hu8/tF+Av5Cv229jhvkPOBZbDGY09HP5gPa+r4D79n9nor37M320f+6y5OGcpADFCNWMhs7XBvmln1n5a47jB9GXdXu8h2x1/iptj4cF8bxwlmwDrwG/Am+h3NpJ0kfYL3Ntc5FQIv32EF+cb9O+lQJ4AbiJ9ssrwDcnPLxAvkxAftAy5z0XKQ6RbZw8/5uG3zPAt3bvY07GD+L3Vfv2u9wP5yIFaK9y2US+jfs3KR/fA56zumsTD+okbT0+HKD4YoPI8fxV2m6urUB+Zvo2Hhu+gc6TesQY38VzqV2E3R7ykzxPemD1nn9dqOQ/4BGGntuG+O3oITH/9joRv1uMc9fjfedvgc4TnX/+fWIrM8x/wCMMU155Hi1w/Dua68AldM7vkn5vk66FCmGenh3mr9J2YgvHZ9r3Th7neR7uJDlNzL2CfCXnq9vl/FW68AH/A9C4ISleHrHbAAAAAElFTkSuQmCC", "text/plain": [ "28×28 Array{Gray{Float64},2}:\n", " Gray{Float64}(0.00272082) … Gray{Float64}(0.00424106)\n", " Gray{Float64}(0.00071994) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.00289914)\n", " Gray{Float64}(0.0241623) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(3.3605e-5) … Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.00875976) Gray{Float64}(0.0577916) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0668495) \n", " Gray{Float64}(0.0345795) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0307308) … Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " ⋮ ⋱ \n", " Gray{Float64}(0.00392799) Gray{Float64}(0.0165317) \n", " Gray{Float64}(0.00120835) Gray{Float64}(0.131568) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0413101) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0405618) \n", " Gray{Float64}(0.0) … Gray{Float64}(0.0330876) \n", " Gray{Float64}(0.0195081) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.00496392)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) … Gray{Float64}(0.0) \n", " Gray{Float64}(0.0) Gray{Float64}(0.00827445)\n", " Gray{Float64}(0.00157579) Gray{Float64}(0.0) " ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver(OutputFlag=0),\n", " norm_order=2);\n", "print_summary(d)\n", "perturbed_sample_image = getvalue(d[:PerturbedInput])\n", "colorview(Gray, perturbed_sample_image[1, :, :, 1])" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAMAAADxPgR5AAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAhFBMVEXy+v75+Pnx+f7s9/3x+v7w+f7v+f7r9/3q9v3u+P7q9/3l9P3p9v3k9P3h8/zt+P7n9f3o9v3a7/vj8/zg8vz/8+//6uT/9fH/9fLm9f3/9PH/7+r/8u7/9PD/9vP/8e3/8Ov/9fPY7vvh8vz/8u/i8/ze8fzV7fvZ7/v/8Oz/8/D///+swsOBAAAAAWJLR0QrJLnkCAAAA1ZJREFUaN7tmtlyFDEMRWcCSQhbCPueAGH9/w9EZ6rPlFDNAoX9pvsQu922rlJ9Lcv2rFaBdcFJwPY7gbuB0wDPZ4HzwL1AHecY+jJGW5a0rZpwOGEeoNGLgG33A5BU4/Z5ENCR9Q7YT+ImnEuYxUGHhwHqjwKPA5a0IRxEgqh4vgwoFIVG+SSAUyvRhMMJ6Vg/OMYUhO/4+BqswA6OXgUUh47Xf6oJxxMqkEykWJ4GKJ8F1v8A7SjEJpxPKJikuXMenAP6MSiw54H6rgnnEDJJmeB0MBg7cSkJDtsBaSI74YHC2wcdb8LxhA5wQeWD8/wiQNvLwKvA6wXZiGOzoLKjuS+CfBNowvGENrA5QTgkvnbAgbcBJvG7AE68X0D9QwAnqFN+DDD2OpCFgkMmYU04npBGRcNHtZPGbwIKBvF8CtwsgBDjlLQzLifTBgdsKqwmHE9IsCbhRSSfA5B+CTh5EZIG6EuQllzHENZtgJJnRQMI/HlRb8LxhFYwZgL8NcDGhGBASfvlghq8cTA7VoN8TcqacB6hQEDnC5isOHK2ICdQvM+O5eTLcbnvH1lbE04lBBjHiGLYfvCFkCDBQk1JIMgHQRim3fHVdhOOJ6SR5IaHOrGzAxjiPQQIizqT3oAAIQHCRNqxbJDoD8fmkL0JhxPuIjDhoWRiawRijGuYd16K0G6Q0HnqOPUtcFClTfhfhPnwzQssE2OTZCc9zxjNm1iDvOKhThvOKEYIdaAJxxMakCmd4Fk4NQAAg7WHeBcJ+fIkB34W6++BJhxPaMO+wC28oT5NwGDdgArbmPg/AizKBIwmnEPoZmX9l2DzCblOHOoLEYKhTmBpwjmEAAEYbA+Bg4OfgV8BD/qOjfHQYnMI3ITDCU2QEE6+7Nh38QEhZB7qHSLywBfb24W8CYcT8ocPStJz7OMz4Tm8UzAeyu4DTntw5IanCecQCgIx5DmQ50QZoUAEYRWMB0ImuyZSBG5ITaiacC5h/SGPkxVSLrIgoe6henaU0h9dUecHIwRtLrnyJVoTziPMyawbE+oKyIstBMMCXBfeLDQd90CC4G1gacJ5hFUsJwts43ILQi6f6+YlP+f6rouVJhxP6KZk80HLxiQP3vVD8oqaSBsEsq0mHE74GwFMW8Cjgp5lAAAAAElFTkSuQmCC", "text/plain": [ "28×28 Array{RGB{Float64},2}:\n", " RGB{Float64}(0.947578,0.979616,0.996444) … RGB{Float64}(0.945958,0.978922,0.996267)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.947578,0.979616,0.996444)\n", " RGB{Float64}(0.929583,0.971845,0.994467) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) … RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.942708,0.977527,0.99591) RGB{Float64}(0.901037,0.959226,0.991304)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.89419,0.956146,0.990537) \n", " RGB{Float64}(0.92128,0.968212,0.993551) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.924611,0.969673,0.993919) … RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " ⋮ ⋱ \n", " RGB{Float64}(0.945958,0.978922,0.996267) RGB{Float64}(0.93617,0.974706,0.995192) \n", " RGB{Float64}(0.947578,0.979616,0.996444) RGB{Float64}(0.835849,0.929008,0.983789)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.914582,0.965259,0.99281) \n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.916261,0.966001,0.992996)\n", " RGB{Float64}(0.974598,0.972607,0.975359) … RGB{Float64}(0.92128,0.968212,0.993551) \n", " RGB{Float64}(0.932883,0.973281,0.99483) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.945958,0.978922,0.996267)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) … RGB{Float64}(0.974598,0.972607,0.975359)\n", " RGB{Float64}(0.974598,0.972607,0.975359) RGB{Float64}(0.942708,0.977527,0.99591) \n", " RGB{Float64}(0.947578,0.979616,0.996444) RGB{Float64}(0.974598,0.972607,0.975359)" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "diff = getvalue(d[:Perturbation])\n", "view_diff(diff[1, :, :, 1])" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "# Advanced Options" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "## Models\n", "Finding adversarial examples consists of two steps:\n", "\n", " 1. Building a model that expresses the input-output constraints of the neural net, which involves determining upper and lower bounds on each intermediate value.\n", " 2. Setting a target category (or multiple target categories) and solving to determine the minimal adversarial example that is classified in that category.\n", "\n", "The first time we verify a neural net, we must do step 1, which can take a long time." ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [1]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Rebuilding model from scratch. This may take some time as we determine upper and lower bounds for the input to each non-linear unit.\n", "\u001b[39mAcademic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Imposing relu constraint: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:01:45\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:02:24\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: The model built will be cached and re-used for future solves, unless you explicitly set rebuild=true.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "487.657528 seconds (1.95 M allocations: 148.187 MiB, 0.02% gc time)\n", "Objective Value: 13.821871, Solve Time: 226.85\n" ] } ], "source": [ "d = @time MIPVerify.find_adversarial_example(n1, sample_image, 1, GurobiSolver(OutputFlag=0), rebuild=true);\n", "print_summary(d)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "When re-running the solve (with the `rebuild` parameter defaulting to `false`), we load the model from cache and skip ahead straight to step 2; finding the adversarial example takes much less time." ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [1]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "211.359899 seconds (331.46 k allocations: 17.398 MiB, 0.01% gc time)\n", "Objective Value: 13.821871, Solve Time: 211.27\n" ] } ], "source": [ "d = @time MIPVerify.find_adversarial_example(n1, sample_image, 1, GurobiSolver(OutputFlag=0));\n", "print_summary(d)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "By setting `rebuild=true`, you force the MIP model to be rebuilt even if one exists; this is useful if you want to use a different `tightening_algorithm` (or use different parameters for the `tightening solver`)." ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "### Removing Cached Models\n", "\n", "If you find that your cached models are taking up too much space, you can remove them with `remove_cached_models()`." ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "## Advanced Solver-fu\n", "\n", "We use two different solvers for the two steps of finding adversarial examples:\n", "\n", " 1. the `tightening_solver`, used only to determine upper and lower bounds on intermediate values when expressing the input-output constraints in the neural net (i.e. if no cached model exists, or if `rebuild=true`), and \n", " 2. the `main_solver`, used to determine the minimal adversarial example for the loaded model." ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "### `tightening_algorithm`\n", "\n", "By default, we tighten the bounds on each intermediate value by solving an MIP using the `tightening_solver`. Compare total solve times for three different tightening algorithms:" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Rebuilding model from scratch. This may take some time as we determine upper and lower bounds for the input to each non-linear unit.\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: The model built will be cached and re-used for future solves, unless you explicitly set rebuild=true.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "172.541745 seconds (465.05 k allocations: 39.319 MiB, 0.02% gc time)\n" ] } ], "source": [ "@time MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver(OutputFlag=0), rebuild=true,\n", " tightening_algorithm=interval_arithmetic);" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\r\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Rebuilding model from scratch. This may take some time as we determine upper and lower bounds for the input to each non-linear unit.\n", "\u001b[39m" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 75%|█████████████████ | ETA: 0:00:00\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: The model built will be cached and re-used for future solves, unless you explicitly set rebuild=true.\n", "\u001b[39m" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\r\n", " 47.955957 seconds (496.37 k allocations: 72.376 MiB, 0.06% gc time)\n" ] } ], "source": [ "@time MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver(OutputFlag=0), rebuild=true,\n", " tightening_algorithm=lp, tightening_solver = GurobiSolver(Gurobi.Env(), OutputFlag=0));" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Rebuilding model from scratch. This may take some time as we determine upper and lower bounds for the input to each non-linear unit.\n", "\u001b[39mAcademic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:01:46\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:02:21\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: The model built will be cached and re-used for future solves, unless you explicitly set rebuild=true.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "327.380706 seconds (517.02 k allocations: 72.765 MiB, 0.01% gc time)\n" ] } ], "source": [ "MIPVerify.setloglevel!(\"notice\")\n", "@time MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver(OutputFlag=0), rebuild=true,\n", " tightening_algorithm=mip);" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "We can also modify many of the parameters of the solver to change behavior:\n", "\n", "We will be focusing on the parameters available via Gurobi (http://www.gurobi.com/documentation/7.5/refman/parameters.html), but other solvers often have similar options." ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ " \n", "### `main_solver`\n", "\n", "#### Muting Output\n", "To mute the output from the `GurobiSolver`, set `OutputFlag=0`.\n" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "collapsed": false, "deletable": true, "editable": true, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "Optimize a model with 3385 rows, 3256 columns and 71132 nonzeros\n", "Variable types: 3196 continuous, 60 integer (60 binary)\n", "Coefficient statistics:\n", " Matrix range [2e-05, 7e+02]\n", " Objective range [1e+00, 1e+00]\n", " Bounds range [1e+00, 1e+02]\n", " RHS range [4e-03, 7e+02]\n", "\n", "MIP start did not produce a new incumbent solution\n", "MIP start violates constraint R1024 by 1.000000000\n", "\n", "Presolve removed 2956 rows and 2227 columns\n", "Presolve time: 0.18s\n", "Presolved: 429 rows, 1029 columns, 61365 nonzeros\n", "Variable types: 969 continuous, 60 integer (60 binary)\n", "\n", "Root relaxation: objective 0.000000e+00, 212 iterations, 0.02 seconds\n", "\n", " Nodes | Current Node | Objective Bounds | Work\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\n", "\n", " 0 0 0.00000 0 3 - 0.00000 - - 0s\n", "Another try with MIP start\n", "H 0 0 36.9304736 0.00000 100% - 0s\n", "H 0 0 32.4680084 0.00000 100% - 0s\n", " 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s\n", " 0 0 0.00000 0 7 32.46801 0.00000 100% - 0s\n", " 0 2 0.00000 0 7 32.46801 0.00000 100% - 0s\n", "H 106 61 16.8778596 0.00000 100% 61.7 1s\n", "H 150 81 6.3285068 0.00000 100% 64.0 1s\n", "* 339 202 50 6.0319541 0.00000 100% 52.1 1s\n", " 932 456 0.00000 19 4 6.03195 0.00000 100% 61.5 5s\n", "* 2259 525 48 6.0319390 0.00000 100% 57.9 9s\n", " 2395 584 3.94423 24 13 6.03194 0.02050 100% 57.8 10s\n", "* 2576 643 49 6.0194767 0.02050 100% 55.8 10s\n", "* 2869 720 54 5.7675028 0.02050 100% 56.0 11s\n", "* 2870 634 54 5.3566208 0.02050 100% 56.0 11s\n", " 4121 990 2.27182 33 13 5.35662 0.33754 93.7% 56.4 15s\n", "* 4325 895 51 4.6601251 0.33754 92.8% 56.6 15s\n", " 6063 1290 4.28290 32 12 4.66013 0.55686 88.1% 55.6 20s\n", " 7212 1539 2.82894 35 3 4.66013 0.66630 85.7% 59.4 25s\n", " 8732 1771 2.74110 30 15 4.66013 0.97990 79.0% 60.5 30s\n", " 10883 1869 cutoff 40 4.66013 1.71949 63.1% 55.8 35s\n", "*12282 1982 48 4.6418593 1.94632 58.1% 55.3 39s\n", " 12699 2020 3.65375 29 6 4.64186 1.98370 57.3% 55.0 40s\n", " 14316 1989 2.45456 34 8 4.64186 2.25731 51.4% 55.3 45s\n", " 15785 1845 2.53334 33 9 4.64186 2.53334 45.4% 55.0 50s\n", " 16512 1851 2.61060 26 8 4.64186 2.61060 43.8% 55.1 55s\n", " 18081 1492 cutoff 20 4.64186 3.21152 30.8% 54.3 60s\n", " 19746 1183 3.71451 44 7 4.64186 3.63004 21.8% 53.3 65s\n", " 21069 522 cutoff 27 4.64186 3.95299 14.8% 53.6 70s\n", "\n", "Cutting planes:\n", " Gomory: 1\n", " Projected implied bound: 33\n", " MIR: 17\n", " Flow cover: 10\n", "\n", "Explored 22057 nodes (1174039 simplex iterations) in 73.00 seconds\n", "Thread count was 4 (of 4 available processors)\n", "\n", "Solution count 10: 4.64186 4.66013 5.35662 ... 32.468\n", "\n", "Optimal solution found (tolerance 1.00e-04)\n", "Best objective 4.641859316319e+00, best bound 4.641859316319e+00, gap 0.0000%\n", "Objective Value: 4.641859, Solve Time: 73.00\n" ] } ], "source": [ "d = MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver());\n", "print_summary(d)" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "Objective Value: 4.641859, Solve Time: 73.00\n" ] } ], "source": [ "MIPVerify.find_adversarial_example(n1, sample_image, 10, GurobiSolver(OutputFlag=0));\n", "print_summary(d)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "#### Terminating early if a conditon is satisfied\n", "\n", "Sometimes, finding an adversarial example takes a long time:" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "Optimize a model with 4169 rows, 3257 columns and 72700 nonzeros\n", "Variable types: 3197 continuous, 60 integer (60 binary)\n", "Coefficient statistics:\n", " Matrix range [2e-05, 7e+02]\n", " Objective range [1e+00, 1e+00]\n", " Bounds range [1e+00, 1e+02]\n", " RHS range [4e-03, 7e+02]\n", "\n", "MIP start did not produce a new incumbent solution\n", "MIP start violates constraint R1024 by 1.000000000\n", "\n", "Presolve removed 2956 rows and 2227 columns\n", "Presolve time: 0.25s\n", "Presolved: 1213 rows, 1030 columns, 62933 nonzeros\n", "Variable types: 970 continuous, 60 integer (60 binary)\n", "\n", "Root relaxation: objective 0.000000e+00, 879 iterations, 0.08 seconds\n", "\n", " Nodes | Current Node | Objective Bounds | Work\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\n", "\n", " 0 0 0.00000 0 3 - 0.00000 - - 0s\n", "Another try with MIP start\n", "H 0 0 0.2251479 0.00000 100% - 0s\n", " 0 0 0.00000 0 8 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 6 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 1 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 1s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 1s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 1s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 1s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 1s\n", " 0 2 0.00000 0 2 0.22515 0.00000 100% - 1s\n", "* 91 66 27 0.1185735 0.00000 100% 231 4s\n", " 98 70 0.00138 7 13 0.11857 0.00000 100% 242 5s\n", "H 136 72 0.1097069 0.00000 100% 255 6s\n", "* 155 74 22 0.1059174 0.00000 100% 241 6s\n", "* 156 74 22 0.1054467 0.00000 100% 239 6s\n", "* 171 83 21 0.1053747 0.00000 100% 225 7s\n", "* 205 84 18 0.1042007 0.00000 100% 221 7s\n", "* 206 84 18 0.1040731 0.00000 100% 220 7s\n", " 269 116 0.00762 8 11 0.10407 0.00000 100% 241 10s\n", " 436 148 infeasible 11 0.10407 0.00000 100% 253 15s\n", " 548 168 0.02436 8 7 0.10407 0.00000 100% 263 20s\n", " 728 183 0.08459 15 7 0.10407 0.00000 100% 264 25s\n", " 825 190 0.02433 8 7 0.10407 0.00000 100% 276 30s\n", " 1001 237 0.03296 9 11 0.10407 0.00021 100% 272 36s\n", " 1131 251 infeasible 9 0.10407 0.01205 88.4% 269 40s\n", " 1316 250 0.02321 5 8 0.10407 0.02321 77.7% 262 45s\n", " 1499 277 cutoff 10 0.10407 0.02321 77.7% 259 50s\n", " 1713 268 0.07814 14 8 0.10407 0.02830 72.8% 257 56s\n", " 1868 272 0.09631 14 8 0.10407 0.03441 66.9% 256 61s\n", " 2042 269 infeasible 14 0.10407 0.04175 59.9% 255 65s\n", " 2210 264 0.07882 14 8 0.10407 0.04673 55.1% 255 70s\n", " 2454 288 0.07419 14 6 0.10407 0.05023 51.7% 251 76s\n", " 2725 296 0.05622 13 9 0.10407 0.05375 48.4% 246 81s\n", " 2859 292 infeasible 14 0.10407 0.05566 46.5% 244 85s\n", " 3097 292 infeasible 12 0.10407 0.06116 41.2% 244 91s\n", " 3225 328 0.10222 11 4 0.10407 0.06116 41.2% 243 95s\n", " 3466 320 infeasible 13 0.10407 0.06118 41.2% 245 103s\n", " 3599 312 0.09837 14 8 0.10407 0.06409 38.4% 245 107s\n", " 3745 295 infeasible 15 0.10407 0.06503 37.5% 245 111s\n", " 3908 280 0.09088 14 9 0.10407 0.06830 34.4% 244 115s\n", " 4235 234 infeasible 14 0.10407 0.07122 31.6% 245 124s\n", " 4483 254 0.07122 12 3 0.10407 0.07122 31.6% 241 129s\n", " 4714 226 0.10284 13 8 0.10407 0.07472 28.2% 239 133s\n", " 4911 191 0.07480 9 10 0.10407 0.07480 28.1% 236 137s\n", " 5070 151 infeasible 13 0.10407 0.08313 20.1% 236 141s\n", " 5368 81 0.09204 8 7 0.10407 0.09204 11.6% 235 147s\n", " 5637 10 0.10347 15 6 0.10407 0.09880 5.06% 232 150s\n", "\n", "Cutting planes:\n", " Gomory: 1\n", " Cover: 7\n", " Implied bound: 28\n", " MIR: 9\n", " Flow cover: 13\n", " Inf proof: 11\n", "\n", "Explored 5734 nodes (1323607 simplex iterations) in 150.87 seconds\n", "Thread count was 4 (of 4 available processors)\n", "\n", "Solution count 8: 0.104073 0.104201 0.105375 ... 0.225148\n", "\n", "Optimal solution found (tolerance 1.00e-04)\n", "Best objective 1.040731274903e-01, best bound 1.040731274903e-01, gap 0.0000%\n" ] } ], "source": [ "MIPVerify.find_adversarial_example(n1, sample_image, 9, GurobiSolver(),\n", " norm_order=Inf);" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "You may want to terminate early when a particular condition is satisfied. Common reasons are:\n", "\n", " 1. Solve exceeding time limit\n", " 2. Lower bound on robustness proved (i.e. `BestBd` increases above a pre-determined threshold)\n", " 3. Counter-example found (i.e. `Incumbent` adversarial image found that is closer to the original image than expected).\n", " 4. Difference between `Incumbent` and `BestBd` falls below a pre-determined threshold.\n", " \n", "Fortunately, Gurobi has a parameter for all of these cases." ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "##### Terminate if time limit is reached\n", "Set `TimeLimit`:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "Optimize a model with 4169 rows, 3257 columns and 72700 nonzeros\n", "Variable types: 3197 continuous, 60 integer (60 binary)\n", "Coefficient statistics:\n", " Matrix range [2e-05, 7e+02]\n", " Objective range [1e+00, 1e+00]\n", " Bounds range [1e+00, 1e+02]\n", " RHS range [4e-03, 7e+02]\n", "\n", "MIP start did not produce a new incumbent solution\n", "MIP start violates constraint R1024 by 1.000000000\n", "\n", "Presolve removed 2956 rows and 2227 columns\n", "Presolve time: 0.17s\n", "Presolved: 1213 rows, 1030 columns, 62933 nonzeros\n", "Variable types: 970 continuous, 60 integer (60 binary)\n", "\n", "Root relaxation: objective 0.000000e+00, 879 iterations, 0.06 seconds\n", "\n", " Nodes | Current Node | Objective Bounds | Work\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\n", "\n", " 0 0 0.00000 0 3 - 0.00000 - - 0s\n", "Another try with MIP start\n", "H 0 0 0.2251479 0.00000 100% - 0s\n", " 0 0 0.00000 0 8 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 6 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 1 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 1s\n", " 0 2 0.00000 0 2 0.22515 0.00000 100% - 1s\n", "* 91 66 27 0.1185735 0.00000 100% 231 4s\n", " 102 65 infeasible 8 0.11857 0.00000 100% 255 5s\n", "H 136 72 0.1097069 0.00000 100% 255 6s\n", "* 155 74 22 0.1059174 0.00000 100% 241 6s\n", "* 156 74 22 0.1054467 0.00000 100% 239 6s\n", "* 171 83 21 0.1053747 0.00000 100% 225 6s\n", "* 205 84 18 0.1042007 0.00000 100% 221 7s\n", "* 206 84 18 0.1040731 0.00000 100% 220 7s\n", " 269 116 0.00762 8 11 0.10407 0.00000 100% 241 10s\n", " 436 148 infeasible 11 0.10407 0.00000 100% 253 15s\n", " 583 172 0.08712 10 5 0.10407 0.00000 100% 262 20s\n", " 728 183 0.08459 15 7 0.10407 0.00000 100% 264 25s\n", " 855 194 0.01180 10 7 0.10407 0.00021 100% 274 30s\n", "\n", "Cutting planes:\n", " Gomory: 1\n", " Implied bound: 21\n", " MIR: 9\n", " Flow cover: 11\n", " Inf proof: 2\n", "\n", "Explored 856 nodes (236345 simplex iterations) in 30.01 seconds\n", "Thread count was 4 (of 4 available processors)\n", "\n", "Solution count 8: 0.104073 0.104201 0.105375 ... 0.225148\n", "\n", "Time limit reached\n", "Best objective 1.040731274903e-01, best bound 2.118627167182e-04, gap 99.7964%\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1m\u001b[33mWARNING: \u001b[39m\u001b[22m\u001b[33mNot solved to optimality, status: UserLimit\u001b[39m\n" ] } ], "source": [ "MIPVerify.find_adversarial_example(n1, sample_image, 9, GurobiSolver(TimeLimit=30),\n", " norm_order=Inf);" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "##### Terminate if lower bound on robustness proved\n", "\n", "Set `BestBdStop` or `Cutoff`.\n", "\n", "(`Cutoff` gives a different error message that is not currently processed correctly by the latest release of `Gurobi.jl`)." ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "Optimize a model with 4169 rows, 3257 columns and 72700 nonzeros\n", "Variable types: 3197 continuous, 60 integer (60 binary)\n", "Coefficient statistics:\n", " Matrix range [2e-05, 7e+02]\n", " Objective range [1e+00, 1e+00]\n", " Bounds range [1e+00, 1e+02]\n", " RHS range [4e-03, 7e+02]\n", "\n", "MIP start did not produce a new incumbent solution\n", "MIP start violates constraint R1024 by 1.000000000\n", "\n", "Presolve removed 2956 rows and 2227 columns\n", "Presolve time: 0.17s\n", "Presolved: 1213 rows, 1030 columns, 62933 nonzeros\n", "Variable types: 970 continuous, 60 integer (60 binary)\n", "\n", "Root relaxation: objective 0.000000e+00, 879 iterations, 0.05 seconds\n", "\n", " Nodes | Current Node | Objective Bounds | Work\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\n", "\n", " 0 0 0.00000 0 3 - 0.00000 - - 0s\n", "Another try with MIP start\n", "H 0 0 0.2251479 0.00000 100% - 0s\n", " 0 0 0.00000 0 8 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 6 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 1 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 0s\n", " 0 2 0.00000 0 2 0.22515 0.00000 100% - 1s\n", "* 91 66 27 0.1185735 0.00000 100% 231 4s\n", " 102 65 infeasible 8 0.11857 0.00000 100% 255 5s\n", "H 136 72 0.1097069 0.00000 100% 255 6s\n", "* 155 74 22 0.1059174 0.00000 100% 241 6s\n", "* 156 74 22 0.1054467 0.00000 100% 239 6s\n", "* 171 83 21 0.1053747 0.00000 100% 225 7s\n", "* 205 84 18 0.1042007 0.00000 100% 221 7s\n", "* 206 84 18 0.1040731 0.00000 100% 220 7s\n", " 269 116 0.00762 8 11 0.10407 0.00000 100% 241 10s\n", " 436 148 infeasible 11 0.10407 0.00000 100% 253 15s\n", " 548 168 0.02436 8 7 0.10407 0.00000 100% 263 20s\n", " 728 183 0.08459 15 7 0.10407 0.00000 100% 264 25s\n", " 825 190 0.02433 8 7 0.10407 0.00000 100% 276 30s\n", " 967 230 cutoff 16 0.10407 0.00021 100% 270 35s\n", " 1131 251 infeasible 9 0.10407 0.01205 88.4% 269 40s\n", " 1316 250 0.02321 5 8 0.10407 0.02321 77.7% 262 45s\n", " 1499 277 cutoff 10 0.10407 0.02321 77.7% 259 50s\n", " 1713 268 0.07814 14 8 0.10407 0.02830 72.8% 257 56s\n", " 1868 272 0.09631 14 8 0.10407 0.03441 66.9% 256 61s\n", " 2042 269 infeasible 14 0.10407 0.04175 59.9% 255 66s\n", " 2210 264 0.07882 14 8 0.10407 0.04673 55.1% 255 71s\n", "\n", "Cutting planes:\n", " Gomory: 1\n", " Cover: 3\n", " Implied bound: 25\n", " MIR: 9\n", " Flow cover: 13\n", " Inf proof: 2\n", "\n", "Explored 2454 nodes (616806 simplex iterations) in 74.00 seconds\n", "Thread count was 4 (of 4 available processors)\n", "\n", "Solution count 8: 0.104073 0.104201 0.105375 ... 0.225148\n", "\n", "Optimization achieved user objective limit\n", "Best objective 1.040731274903e-01, best bound 5.023143684607e-02, gap 51.7345%\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1m\u001b[33mWARNING: \u001b[39m\u001b[22m\u001b[33mNot solved to optimality, status: UserObjLimit\u001b[39m\n" ] } ], "source": [ "MIPVerify.find_adversarial_example(n1, sample_image, 9, GurobiSolver(BestBdStop=0.05),\n", " norm_order=Inf);" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "##### Terminate if adversarial example found closer than expected robustness\n", "\n", "Set `BestObjStop`." ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "Optimize a model with 4169 rows, 3257 columns and 72700 nonzeros\n", "Variable types: 3197 continuous, 60 integer (60 binary)\n", "Coefficient statistics:\n", " Matrix range [2e-05, 7e+02]\n", " Objective range [1e+00, 1e+00]\n", " Bounds range [1e+00, 1e+02]\n", " RHS range [4e-03, 7e+02]\n", "\n", "MIP start did not produce a new incumbent solution\n", "MIP start violates constraint R1024 by 1.000000000\n", "\n", "Presolve removed 2956 rows and 2227 columns\n", "Presolve time: 0.17s\n", "Presolved: 1213 rows, 1030 columns, 62933 nonzeros\n", "Variable types: 970 continuous, 60 integer (60 binary)\n", "\n", "Root relaxation: objective 0.000000e+00, 879 iterations, 0.05 seconds\n", "\n", " Nodes | Current Node | Objective Bounds | Work\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\n", "\n", " 0 0 0.00000 0 3 - 0.00000 - - 0s\n", "Another try with MIP start\n", "H 0 0 0.2251479 0.00000 100% - 0s\n", " 0 0 0.00000 0 8 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 6 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 1 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 1s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 1s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 1s\n", " 0 2 0.00000 0 2 0.22515 0.00000 100% - 1s\n", "* 91 66 27 0.1185735 0.00000 100% 231 4s\n", "\n", "Cutting planes:\n", " Gomory: 1\n", " Implied bound: 3\n", " MIR: 6\n", " Flow cover: 6\n", "\n", "Explored 98 nodes (24534 simplex iterations) in 4.10 seconds\n", "Thread count was 4 (of 4 available processors)\n", "\n", "Solution count 2: 0.118574 0.225148 \n", "\n", "Optimization achieved user objective limit\n", "Best objective 1.185735336641e-01, best bound 0.000000000000e+00, gap 100.0000%\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[1m\u001b[33mWARNING: \u001b[39m\u001b[22m\u001b[33mNot solved to optimality, status: UserObjLimit\u001b[39m\n" ] } ], "source": [ "MIPVerify.find_adversarial_example(n1, sample_image, 9, GurobiSolver(BestObjStop=0.2),\n", " norm_order=Inf);" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "##### Terminate if gap between `Incumbent` and `BestBd` is below threshold\n", "\n", "Set `MIPGap`." ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9]\n", "\u001b[39m\u001b[36m[notice | MIPVerify]: Loading model from cache.\n", "\u001b[39mAcademic license - for non-commercial use only\n", "Optimize a model with 4169 rows, 3257 columns and 72700 nonzeros\n", "Variable types: 3197 continuous, 60 integer (60 binary)\n", "Coefficient statistics:\n", " Matrix range [2e-05, 7e+02]\n", " Objective range [1e+00, 1e+00]\n", " Bounds range [1e+00, 1e+02]\n", " RHS range [4e-03, 7e+02]\n", "\n", "MIP start did not produce a new incumbent solution\n", "MIP start violates constraint R1024 by 1.000000000\n", "\n", "Presolve removed 2956 rows and 2227 columns\n", "Presolve time: 0.17s\n", "Presolved: 1213 rows, 1030 columns, 62933 nonzeros\n", "Variable types: 970 continuous, 60 integer (60 binary)\n", "\n", "Root relaxation: objective 0.000000e+00, 879 iterations, 0.05 seconds\n", "\n", " Nodes | Current Node | Objective Bounds | Work\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\n", "\n", " 0 0 0.00000 0 3 - 0.00000 - - 0s\n", "Another try with MIP start\n", "H 0 0 0.2251479 0.00000 100% - 0s\n", " 0 0 0.00000 0 8 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 6 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 1 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.22515 0.00000 100% - 1s\n", " 0 2 0.00000 0 2 0.22515 0.00000 100% - 1s\n", "* 91 66 27 0.1185735 0.00000 100% 231 4s\n", " 102 65 infeasible 8 0.11857 0.00000 100% 255 5s\n", "H 136 72 0.1097069 0.00000 100% 255 6s\n", "* 155 74 22 0.1059174 0.00000 100% 241 6s\n", "* 156 74 22 0.1054467 0.00000 100% 239 6s\n", "* 171 83 21 0.1053747 0.00000 100% 225 6s\n", "* 205 84 18 0.1042007 0.00000 100% 221 7s\n", "* 206 84 18 0.1040731 0.00000 100% 220 7s\n", " 297 127 infeasible 13 0.10407 0.00000 100% 237 10s\n", " 436 148 infeasible 11 0.10407 0.00000 100% 253 15s\n", " 583 172 0.08712 10 5 0.10407 0.00000 100% 262 20s\n", " 728 183 0.08459 15 7 0.10407 0.00000 100% 264 25s\n", " 855 192 0.01125 8 14 0.10407 0.00021 100% 275 30s\n", " 1001 237 0.03296 9 11 0.10407 0.00021 100% 272 35s\n", " 1180 257 infeasible 13 0.10407 0.01832 82.4% 269 41s\n", " 1373 282 0.03916 9 12 0.10407 0.02321 77.7% 261 46s\n", " 1499 277 cutoff 10 0.10407 0.02321 77.7% 259 50s\n", " 1713 268 0.07814 14 8 0.10407 0.02830 72.8% 257 55s\n", " 1868 272 0.09631 14 8 0.10407 0.03441 66.9% 256 60s\n", " 2042 269 infeasible 14 0.10407 0.04175 59.9% 255 65s\n", " 2210 264 0.07882 14 8 0.10407 0.04673 55.1% 255 70s\n", " 2454 288 0.07419 14 6 0.10407 0.05023 51.7% 251 75s\n", " 2725 296 0.05622 13 9 0.10407 0.05375 48.4% 246 81s\n", " 2980 286 cutoff 14 0.10407 0.05968 42.7% 244 88s\n", " 3097 292 infeasible 12 0.10407 0.06116 41.2% 244 91s\n", " 3225 328 0.10222 11 4 0.10407 0.06116 41.2% 243 95s\n", " 3466 320 infeasible 13 0.10407 0.06118 41.2% 245 103s\n", "\n", "Cutting planes:\n", " Gomory: 1\n", " Cover: 4\n", " Implied bound: 26\n", " MIR: 9\n", " Flow cover: 13\n", " Inf proof: 6\n", "\n", "Explored 3599 nodes (883040 simplex iterations) in 103.15 seconds\n", "Thread count was 4 (of 4 available processors)\n", "\n", "Solution count 8: 0.104073 0.104201 0.105375 ... 0.225148\n", "\n", "Optimal solution found (tolerance 4.00e-01)\n", "Best objective 1.040731274903e-01, best bound 6.261060501743e-02, gap 39.8398%\n" ] } ], "source": [ "MIPVerify.find_adversarial_example(n1, sample_image, 9, GurobiSolver(MIPGap=0.4),\n", " norm_order=Inf);" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "### `tightening_solver`\n", "The default model build solver has the same type as the `main_solver`, but uses the default settings for that solver type other than 1) muting the output and 2) setting a time limit of 20s per solve (i.e. per upper/lower bound per intermediate value).\n", "\n", "The most common reason to pass in your own `tightening_solver` to modify the time limit per solve." ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "Whew! That was a lot. The next tutorial will introduce you to everything you can extract from the results dictionary." ] } ], "metadata": { "kernelspec": { "display_name": "Julia 0.6.4", "language": "julia", "name": "julia-0.6" }, "language_info": { "file_extension": ".jl", "mimetype": "application/julia", "name": "julia", "version": "0.6.4" } }, "nbformat": 4, "nbformat_minor": 2 }