{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Finding adversarial examples, in depth\n", "\n", "Using the same example from the quickstart, we explore how to get more out of the function `find_adversarial_example`" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "view_diff (generic function with 1 method)" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "using MIPVerify\n", "using Gurobi\n", "using JuMP\n", "using Images\n", "using Printf\n", "\n", "mnist = MIPVerify.read_datasets(\"MNIST\")\n", "n1 = MIPVerify.get_example_network_params(\"MNIST.n1\")\n", "sample_image = MIPVerify.get_image(mnist.test.images, 1);\n", "\n", "function print_summary(d::Dict)\n", " # Helper function to print out output\n", " obj_val = JuMP.objective_value(d[:Model])\n", " solve_time = JuMP.solve_time(d[:Model])\n", " println(\"Objective Value: $(@sprintf(\"%.6f\", obj_val)), Solve Time: $(@sprintf(\"%.2f\", solve_time))\")\n", "end\n", "\n", "function view_diff(diff::Array{<:Real, 2})\n", " n = 1001\n", " colormap(\"RdBu\", n)[ceil.(Int, (diff .+ 1) ./ 2 .* n)]\n", "end" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## `find_adversarial_example`\n", "\n", "`find_adversarial_example` takes five positional arguments\n", "\n", "```\n", "find_adversarial_example(nn, input, target_selection, optimizer, main_solve_options)\n", "```\n", "\n", "It also takes named arguments, each with the default value specified.\n", "\n", "```\n", "norm_order = 1\n", "invert_target_selection = false\n", "pp = MIPVerify.UnrestrictedPerturbationFamily()\n", "tightening_algorithm = mip\n", "tightening_options: same as main solver, but with output suppressed and a time limit of 20s per solve.\n", "```\n", "\n", "See [full documentation here](https://vtjeng.github.io/MIPVerify.jl/dev/finding_adversarial_examples/single_image/#MIPVerify.find_adversarial_example-Tuple{NeuralNet,Array{#s165,N}%20where%20N%20where%20#s165%3C:Real,Union{Integer,%20Array{#s164,1}%20where%20#s164%3C:Integer},Any,Dict}).\n", "\n", "We explore what each of these options allow us to do." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Basic Options" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Specifying target categories for the adversarial example\n", "\n", "`target_selection` and `invert_target_selection` control what the category we want the adversarial example to be classified in.\n", "\n", "`target_selection` accepts either a single integer or a list of integers.\n", "\n", "For example, if I wanted the original image (which is the digit 7) to be classified as the digit 8 or 9, I could run two separate solves with `target_selection=9` and `target_selection=10` (Julia is 1-indexed), finding closest adversarial examples at an $L_\\infty$ distance `0.104073` and `0.046085` ..." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Imposing relu constraint: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating upper bounds: 10%|██▎ | ETA: 0:02:50\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:19\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Imposing relu constraint: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Objective Value: 0.104073, Solve Time: 531.95\n" ] } ], "source": [ "d = MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 9, \n", " Gurobi.Optimizer,\n", " # OutputFlag=0 prevents any output from being printed out\n", " Dict(\"OutputFlag\" => 0),\n", " pp = MIPVerify.LInfNormBoundedPerturbationFamily(0.15),\n", " norm_order = Inf,\n", " tightening_algorithm = lp,\n", ")\n", "print_summary(d)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 40%|█████████▎ | ETA: 0:00:00\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Objective Value: 0.046085, Solve Time: 44.76\n" ] } ], "source": [ "d = MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10, \n", " Gurobi.Optimizer,\n", " Dict(\"OutputFlag\" => 0),\n", " pp = MIPVerify.LInfNormBoundedPerturbationFamily(0.15),\n", " norm_order = Inf,\n", " tightening_algorithm = lp,\n", ")\n", "print_summary(d)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or I can can pass the targets as `target_selection = [9, 10]`, where the same optimal value of `0.046085` is found.\n", "\n", "Solve times for multiple target labels are typically on par with or faster than the aggregate solve times when solving with each target label in sequence." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [9, 10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 10%|██▎ | ETA: 0:00:01\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Objective Value: 0.046085, Solve Time: 24.28\n" ] } ], "source": [ "d = MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " [9, 10], \n", " Gurobi.Optimizer,\n", " Dict(\"OutputFlag\" => 0),\n", " pp = MIPVerify.LInfNormBoundedPerturbationFamily(0.15),\n", " norm_order = Inf,\n", " tightening_algorithm = lp,\n", ")\n", "print_summary(d)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A common use case is to have the adversarial example being in any category but the original:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [1, 2, 3, 4, 5, 6, 7, 9, 10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 25%|█████▊ | ETA: 0:00:00\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Objective Value: 0.046085, Solve Time: 102.45\n" ] } ], "source": [ "d = MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " [1, 2, 3, 4, 5, 6, 7, 9, 10], \n", " Gurobi.Optimizer,\n", " Dict(\"OutputFlag\" => 0),\n", " pp = MIPVerify.LInfNormBoundedPerturbationFamily(0.15),\n", " norm_order = Inf,\n", " tightening_algorithm = lp,\n", ")\n", "print_summary(d)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Rather than typing the full list of other categories, we can set `target_selection = 8`, and `invert_target_selection = true`." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [1, 2, 3, 4, 5, 6, 7, 9, 10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 25%|█████▊ | ETA: 0:00:00\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Objective Value: 0.046085, Solve Time: 107.32\n" ] } ], "source": [ "d = MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 8, \n", " Gurobi.Optimizer,\n", " Dict(\"OutputFlag\" => 0),\n", " pp = MIPVerify.LInfNormBoundedPerturbationFamily(0.15),\n", " norm_order = Inf,\n", " tightening_algorithm = lp,\n", " invert_target_selection = true,\n", ")\n", "print_summary(d)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Restricting the Family of Perturbations\n", "\n", "### Unrestricted Perturbations\n", "\n", "By default, every pixel can be modified without bound." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 15%|███▌ | ETA: 0:00:01\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", " 46.896601 seconds (4.36 M allocations: 177.896 MiB)\n", "Objective Value: 0.046085, Solve Time: 44.07\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAALHSURBVGje7dq/r0xBFAfwz/LiVx4FiQh5OhIaEREFhY5QKJQaGhWN0EgU/gydSCQanYhQiB9BJRHxM2hQ0L4tJCQU915vdnbm7ja792WyJ9nszNxzZnK/35wzM+fc3jz6BmW+/u8H7Vw/tusn7MOxFaYs5S84F/IQ8hnzE4/PR/9abMPn5UM6fQ7DTuMvfwz7Upv/xf7ZzDEX9Bud8iGd+oK9kKtG2mJiTlIxN6VfPqTdxNIwLuZ8rD9iolxM7vwNy1+wF+I9yidH8SiaR8KmfEinzyGDmJM+T4Y89xM2Ir1YGrvyIZ1+LM1xcghX8BK/cLMe/5SZKPblcEzwrHxIu/FDhrH/iTV1fz0W69+blsm24Bsu4L20z5YPaXfn0lD6OIA9eIvd2IvD2IavWAj0/2AlNtT9a7gonRMoH9LlwSFp/BdUXN7BfpUT/1XF2o94h404hxu1Xbyflg9p97G07bzSPI9t4CSu4wt2YZX0ebZ8SLvjsJFx+It1NuM11uE0brfYlw9pt7k28n6Y46+Pqyr+fuNDxq6zNyx/wd44+bS22sVB3Kvbx/FYPt528oblLzhQtxjnHh+PHav/X+B5i97sTDMxGdgPc9zl+FyLp9iJI3iW0JndDycucwzXjVJxNcXjJRV/Tyzxl8vTzWrAE5OBu8U4Oe2Gi+O4he84gwfStarZfjhxGfLDlMT1/dW4XPfvqvhj2G9F/Vm+dCKSvOOPkofYh884ih8j9Gf74USll6sz5GLrVlU+G06o+Ixt2s5G5UPabR0/lBR/m3C/bl+yxF/DWdv3VM03OeVD2s1+mLqTx3dFOIvtdftRYDMqxxrOWT6kyyPX1kgYE3fgvKVaoowe7ZyWD2l3HOa+q2nGD9XtRdVZJnV3zOVUO33D8hf8z2EuTxr71CucUtXrY722dtMvH9LllfNu4yMl49iXD+nUF/wHg+y5HmmDeFIAAAAASUVORK5CYII=", "text/plain": [ "28×28 reinterpret(Gray{Float64}, ::Array{Float64,2}):\n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " ⋮ ⋱ \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) … Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = @time MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10, \n", " Gurobi.Optimizer,\n", " Dict(\"OutputFlag\" => 0),\n", " tightening_algorithm = lp,\n", " norm_order = Inf,\n", ")\n", "print_summary(d)\n", "perturbed_sample_image = value.(d[:PerturbedInput])\n", "colorview(Gray, perturbed_sample_image[1, :, :, 1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### $L_\\infty$-norm Bounded Perturbations\n", "\n", "We can bound the $L_\\infty$-norm of the perturbation." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As long as the size of the $L_\\infty$-norm bound we choose is larger than the actual ($L_\\infty$-)minimal perturbation, we will find the same result, and often more quickly." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 25%|█████▊ | ETA: 0:00:00\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", " 41.363017 seconds (2.07 M allocations: 68.034 MiB)\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAALHSURBVGje7dq/r0xBFAfwz/LiVx4FiQh5OhIaEREFhY5QKJQaGhWN0EgU/gydSCQanYhQiB9BJRHxM2hQ0L4tJCQU915vdnbm7ja792WyJ9nszNxzZnK/35wzM+fc3jz6BmW+/u8H7Vw/tusn7MOxFaYs5S84F/IQ8hnzE4/PR/9abMPn5UM6fQ7DTuMvfwz7Upv/xf7ZzDEX9Bud8iGd+oK9kKtG2mJiTlIxN6VfPqTdxNIwLuZ8rD9iolxM7vwNy1+wF+I9yidH8SiaR8KmfEinzyGDmJM+T4Y89xM2Ir1YGrvyIZ1+LM1xcghX8BK/cLMe/5SZKPblcEzwrHxIu/FDhrH/iTV1fz0W69+blsm24Bsu4L20z5YPaXfn0lD6OIA9eIvd2IvD2IavWAj0/2AlNtT9a7gonRMoH9LlwSFp/BdUXN7BfpUT/1XF2o94h404hxu1Xbyflg9p97G07bzSPI9t4CSu4wt2YZX0ebZ8SLvjsJFx+It1NuM11uE0brfYlw9pt7k28n6Y46+Pqyr+fuNDxq6zNyx/wd44+bS22sVB3Kvbx/FYPt528oblLzhQtxjnHh+PHav/X+B5i97sTDMxGdgPc9zl+FyLp9iJI3iW0JndDycucwzXjVJxNcXjJRV/Tyzxl8vTzWrAE5OBu8U4Oe2Gi+O4he84gwfStarZfjhxGfLDlMT1/dW4XPfvqvhj2G9F/Vm+dCKSvOOPkofYh884ih8j9Gf74USll6sz5GLrVlU+G06o+Ixt2s5G5UPabR0/lBR/m3C/bl+yxF/DWdv3VM03OeVD2s1+mLqTx3dFOIvtdftRYDMqxxrOWT6kyyPX1kgYE3fgvKVaoowe7ZyWD2l3HOa+q2nGD9XtRdVZJnV3zOVUO33D8hf8z2EuTxr71CucUtXrY722dtMvH9LllfNu4yMl49iXD+nUF/wHg+y5HmmDeFIAAAAASUVORK5CYII=", "text/plain": [ "28×28 reinterpret(Gray{Float64}, ::Array{Float64,2}):\n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " ⋮ ⋱ \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) … Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = @time MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10,\n", " Gurobi.Optimizer,\n", " Dict(\"OutputFlag\" => 0),\n", " tightening_algorithm = lp,\n", " norm_order=Inf,\n", ")\n", "perturbed_sample_image = value.(d[:PerturbedInput])\n", "colorview(Gray, perturbed_sample_image[1, :, :, 1])" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", " 14.070512 seconds (2.93 M allocations: 95.636 MiB)\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAALHSURBVGje7dq/r0xBFAfwz/LiVx4FiQh5OhIaEREFhY5QKJQaGhWN0EgU/gydSCQanYhQiB9BJRHxM2hQ0L4tJCQU915vdnbm7ja792WyJ9nszNxzZnK/35wzM+fc3jz6BmW+/u8H7Vw/tusn7MOxFaYs5S84F/IQ8hnzE4/PR/9abMPn5UM6fQ7DTuMvfwz7Upv/xf7ZzDEX9Bud8iGd+oK9kKtG2mJiTlIxN6VfPqTdxNIwLuZ8rD9iolxM7vwNy1+wF+I9yidH8SiaR8KmfEinzyGDmJM+T4Y89xM2Ir1YGrvyIZ1+LM1xcghX8BK/cLMe/5SZKPblcEzwrHxIu/FDhrH/iTV1fz0W69+blsm24Bsu4L20z5YPaXfn0lD6OIA9eIvd2IvD2IavWAj0/2AlNtT9a7gonRMoH9LlwSFp/BdUXN7BfpUT/1XF2o94h404hxu1Xbyflg9p97G07bzSPI9t4CSu4wt2YZX0ebZ8SLvjsJFx+It1NuM11uE0brfYlw9pt7k28n6Y46+Pqyr+fuNDxq6zNyx/wd44+bS22sVB3Kvbx/FYPt528oblLzhQtxjnHh+PHav/X+B5i97sTDMxGdgPc9zl+FyLp9iJI3iW0JndDycucwzXjVJxNcXjJRV/Tyzxl8vTzWrAE5OBu8U4Oe2Gi+O4he84gwfStarZfjhxGfLDlMT1/dW4XPfvqvhj2G9F/Vm+dCKSvOOPkofYh884ih8j9Gf74USll6sz5GLrVlU+G06o+Ixt2s5G5UPabR0/lBR/m3C/bl+yxF/DWdv3VM03OeVD2s1+mLqTx3dFOIvtdftRYDMqxxrOWT6kyyPX1kgYE3fgvKVaoowe7ZyWD2l3HOa+q2nGD9XtRdVZJnV3zOVUO33D8hf8z2EuTxr71CucUtXrY722dtMvH9LllfNu4yMl49iXD+nUF/wHg+y5HmmDeFIAAAAASUVORK5CYII=", "text/plain": [ "28×28 reinterpret(Gray{Float64}, ::Array{Float64,2}):\n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " ⋮ ⋱ \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) … Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = @time MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10,\n", " Gurobi.Optimizer,\n", " Dict(\"OutputFlag\" => 0),\n", " tightening_algorithm = lp,\n", " norm_order=Inf,\n", " pp = MIPVerify.LInfNormBoundedPerturbationFamily(0.05)\n", ")\n", "perturbed_sample_image = value.(d[:PerturbedInput])\n", "colorview(Gray, perturbed_sample_image[1, :, :, 1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If the $L_\\infty$-norm bound you choose is smaller than the actual minimal perturbation, the problem is infeasible. If you observe the following solve status, there is provably no perturbation within the selected $L_\\infty$-norm bound (in this case `0.03`) that is an adversarial example." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", " 0.631356 seconds (2.27 M allocations: 75.066 MiB)\n" ] }, { "data": { "text/plain": [ "Dict{Any,Any} with 11 entries:\n", " :TargetIndexes => [10]\n", " :SolveTime => 0.233993\n", " :TotalTime => 0.609287\n", " :Perturbation => VariableRef[noname noname … noname noname]…\n", " :PerturbedInput => VariableRef[noname noname … noname noname]…\n", " :TighteningApproach => \"lp\"\n", " :PerturbationFamily => linf-norm-bounded-0.03\n", " :SolveStatus => INFEASIBLE_OR_UNBOUNDED\n", " :Model => A JuMP Model…\n", " :Output => GenericAffExpr{Float64,VariableRef}[0.0014746447222673…\n", " :PredictedIndex => 8" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = @time MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10,\n", " Gurobi.Optimizer,\n", " Dict(\"OutputFlag\" => 0),\n", " tightening_algorithm = lp,\n", " norm_order=Inf,\n", " pp = MIPVerify.LInfNormBoundedPerturbationFamily(0.03)\n", ")" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "INFEASIBLE_OR_UNBOUNDED::TerminationStatusCode = 6" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d[:SolveStatus]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Blurring Perturbations\n", "\n", "We can restrict the perturbations to a blur; in this case, we select a 5x5 kernel. (Note that we are still minimizing over the norm of the perturbation.)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 10%|██▎ | ETA: 0:00:01\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", " 48.396935 seconds (4.98 M allocations: 216.330 MiB, 0.18% gc time)\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAAKQSURBVGje7drbSlRRHMfxj3lI7ahlRQeyA4V0EUFFEEh40TN401XP0Xt00xv0AkEQ3hRBN0JdVKAlJZiV1ZiHcpou/ks0p3BroHsW84cNe+1ZzG9++8t/rf9aa2hGM5rRjGY0oxnNaPxoWfvgME5gIbWnsIg5VNOz2n8I7thqh/kL1jEcRFe6b08dqphNz2r+znAB4/iWPq/+o1/+r3TLBdvWPniBPdiNC9iJffgs+KyOGn6kPkS+VgT3ZY7b7jB/wTqGn9J1wwqzL4LrElqtMFpmeFXkbHe6xgTzUjjMX7COYTvOiDmxDR/xQXDtQKcYNztT/370CmaLOCLmzumyOMxfsI5hJ65gV2ofw1Erdc1efBXjaweupy8ZxzW8xGiZHOYvWMdwHiOC1Sx60Cfya7/IwQOp72kxF86l+1Mi/ybL5DB/wTqGS5hY1X4j5r8uKznYK9guipwbxR0xPz4Tc2RpHOYv2FakU03k2lxqvxe1y6Soe84LxuN4XjaH+QsWYrg2WnEw3VcxLHLynqh9SuUwf8FNMewX42oFF0UOvsLbMjrMX3DDDPvE2mN5DT8kap+7mCmjw/wFN8SwHQOixpnBJbGf80TkYSkd5i9YmGELLotxc17k4KCoae7jV1kd5i9YmGG3mAOJuvRWaj9SPAe3xWH+goUYdokcJNYPN8VezjuRg6V2mL9gIYbHxR5ci8jHITGWPvDnfkApHeYvuC7DHpxM94dwW3AcwcNGcJi/YCGGremXnRXn/NMiByuN4DB/wUJj6Q5xdjEg9mGeivX8Zv6Tkf8rLR/DMbEvOizOmmbxGj8bxWH+gusybBVniOdSu4LH+N4oDvMXXJdhm6hlJsQ51JRY4zeMw/wFfwP9nHz9VPH8aQAAAABJRU5ErkJggg==", "text/plain": [ "28×28 reinterpret(Gray{Float64}, ::Array{Float64,2}):\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " ⋮ ⋱ \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = @time MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10,\n", " Gurobi.Optimizer,\n", " Dict(\"OutputFlag\" => 0),\n", " tightening_algorithm = lp,\n", " norm_order=Inf,\n", " pp = MIPVerify.BlurringPerturbationFamily((5, 5))\n", ")\n", "perturbed_sample_image = value.(d[:PerturbedInput])\n", "colorview(Gray, perturbed_sample_image[1, :, :, 1])" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAMAAADxPgR5AAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAABv1BMVEX5+Pne8fzE5PnL6Prg8vzn9f3v+f7x+f7x+v663/ey2vas1/Wr1/Wu2fW73/e74Pe84PfE5Pji8/zu+P7X7vv/3NP/q5r/yLv/2tHY7vvO6fqs2PWy2/bL6Pnt+P3xaVndTUDiU0XuZFTqXU7/k4H/lIL/kYD/hnT/wrP/xbj/no3/moj/4trw+f7P6vrc8Pv/7Ob/zsL/ppXvZVXwZ1fyaln/opH/pJPkVkfkVUf/rp7y+v7o9f2bzfKPxu+Qxu+Xy/G03PbV7fvW7fvQ6vqVyfCJwe2g0PP/qZnzbFvZ7vvH5vmZzPGRx++azfFpquJIi85Sltb/vrD/r5//7+vq9v3r9/3b7/vM6Pr/yr7/vK3/6uPS6/q/4vj/2c//wbP/sqL/5uD/9PD/v7H/u6z/7unU7Pu+4fj/w7X/2tDp9v3h8vzK5/n/3tX/vq//6+XC4/j/6OH/wLL/zcHA4vj/5+D/tqb/wLH/6uTl9Pz/sKD/08f/wrT/8/Cx2vb/uqrk9Py13Pa53/f/0sav2fX/9fKw2fXR6/r/va7+hHL/39bs9/3/tKT9gW//5N3/0cXu+P2Rx/Cp1fTm9f3///8TIqe1AAAAAWJLR0SUf2dKFQAAAidJREFUaN7t2utTTVEch/GkokRSSEVFKbnkmuTalSJyq1xKFApFSi5FJNdu6B/2fGf2mvHivGjG2rM38/u8OXOaOfPsPbNaZ621T1KSMcYYY8x/aAWSsTKQglSkYRVWw4LxDqYjI7AGmViLdYEsrE8gGxuQA3dxFowmqA/lYiM2YTPysAX5cBeh1wIUIjOwFdsCilowmqBThOJACbZjB0pRBr3uRDkqsAuV2I092AsLRhPUP+4+VGE/9P4ADuIQdAGH4S7mCKpxFDU4hlochwWjCZ7ASZz6w2mcQR3qoYm9AY1oQjPO4hxaoEG07NFpQe9BTbqt0AJKk/h5aIK+gDZchLuQS2jHZehL+Aqu4hosGF0wES1+r0OL5A50ogsaSBpg+vsN3MQt/FXMgqEEE7kNTQDd6MEd9OIuLBj/oBZICil4D33ox32EcncW9O4BFHsILXoHMIhHsGD8g49RGHiCIQzjKUK5Owt6pY2NDoc0YJ5hBM8ximUfBFkwsqAWUmNQTNEXGMdLvIL3u7Og96AmZndw+xpvoAEzAe8xC3qPaQPjDt/15auD20no8PYtLBj/oNu4vIM2plPQ4leTufeYBb3H3uMDFJvGR2jAaEOjg0ALxj/oDmRn8AlaMGnzqY2o9wfQFgwt+BlfoI3nV3zDd3iPWTCUoMxCh3baeM5hHguw4L8R1IMTLZz0Ix4FF/EDod2dBb3TgNFhrB5u/cQvLMGCsQn+BvCtkF4ovvyqAAAAAElFTkSuQmCC", "text/plain": [ "28×28 Array{RGB{Float64},2} with eltype RGB{Float64}:\n", " RGB{Float64}(0.974944,0.972996,0.976035) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " ⋮ ⋱ \n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "diff = value.(d[:Perturbation])\n", "view_diff(diff[1, :, :, 1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Minimizing Over Different Norms\n", "### $L_1$\n", "By default, we minimize the $L_1$ norm of the perturbation. This generally encourages sparsity in the perturbations. \n", "\n", "In this case, the minimum $L_1$ norm perturbation required for the image to be classified as a `9` is `4.641859.`" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 10%|██▎ | ETA: 0:00:01\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Objective Value: 4.641859, Solve Time: 55.99\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAAHWSURBVGje7dm7axRRHMXxjyYpBBsNChY+KgstJIggJIKPRmNhYf6FpFHLgH+CWFpY2NsIQiAiIihEfBVpJL4xFioiCFqokEKIxYywkey644TdOz/vaebuzDBfzpw9l3sZsrKy+q11VW4+jws1get77TA+MKv56tjDBxhdY2D8WuQeZmX9h2o7l05gEh+xhKv4hDc1gfGntp4D22b4Frv+OPcNz/7ywA+4iPlUHMYHDra7MIl9eI49GMFhHMR7bG+59yc+Y1v5+52cYQ+1oofLOi9UNymynMeBlvNLeI0X2IyzuJyKw/jASnv8TjqNa3iKI/iSisP4wDXJcCsWyuMErqfkMD5wsP4jOIMt+IpXqTmMD6zdw1HcxZBizXMvNYfxgbV7OK7I7w4erXJ9Clf66TA+sFYPN+A+9uIoHqboMD6wVg+nFXuNW7rLb7kfDuMD/7mHJzGDHzhh9Xn0t1r3nfFfaTN6OIxLGMBNnfNj5R8l/itNv4cDeIz9WMTx8pisw/jAyhnuxstyfAqzqTuMD6w0l+7E7XI8jRtNcBgfWCnDKewox3OKtUryDuMDu87wEM410WF8YNcZjmFjOV7E96Y4jA+svLd4gmPaf1tKzmF8YFbz9QuU8T8T+c3qhwAAAABJRU5ErkJggg==", "text/plain": [ "28×28 reinterpret(Gray{Float64}, ::Array{Float64,2}):\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " ⋮ ⋱ \n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(1.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0) Gray{Float64}(0.0)" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10,\n", " Gurobi.Optimizer,\n", " Dict(\"OutputFlag\" => 0),\n", " tightening_algorithm = lp,\n", " norm_order=1\n", ")\n", "print_summary(d)\n", "perturbed_sample_image = value.(d[:PerturbedInput])\n", "colorview(Gray, perturbed_sample_image[1, :, :, 1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We also show the difference between the perturbed image and the original image. Red is areas of decreased brightness and blue is areas of increased brightness." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwBAMAAAA0zul4AAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAElBMVEX5+Pl+uuozcbYILFmRx+/////jii5GAAAAAWJLR0QF+G/pxwAAAENJREFUWMPt1UEJADAMBMFYiIVaiIX411QPLRSy3fkv3O8iBJbJD8nW4oeS9EIVP5Q0UTc//MegyzgKh8y8CiVJEsAGBXEIgdrWYGoAAAAASUVORK5CYII=", "text/plain": [ "28×28 Array{RGB{Float64},2} with eltype RGB{Float64}:\n", " RGB{Float64}(0.974944,0.972996,0.976035) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " ⋮ ⋱ \n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.032796,0.17167,0.347312)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "diff = value.(d[:Perturbation])\n", "view_diff(diff[1, :, :, 1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### $L_\\infty$\n", "\n", "We can also minimize over the $L_\\infty$ norm. This generally results in large patches of the image being changed. \n", "\n", "In this case, the minimum $L_\\infty$ norm perturbation required for the image to be classified as a `9` is `0.046085.`" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 25%|█████▊ | ETA: 0:00:00\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Objective Value: 0.046085, Solve Time: 42.74\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAALHSURBVGje7dq/r0xBFAfwz/LiVx4FiQh5OhIaEREFhY5QKJQaGhWN0EgU/gydSCQanYhQiB9BJRHxM2hQ0L4tJCQU915vdnbm7ja792WyJ9nszNxzZnK/35wzM+fc3jz6BmW+/u8H7Vw/tusn7MOxFaYs5S84F/IQ8hnzE4/PR/9abMPn5UM6fQ7DTuMvfwz7Upv/xf7ZzDEX9Bud8iGd+oK9kKtG2mJiTlIxN6VfPqTdxNIwLuZ8rD9iolxM7vwNy1+wF+I9yidH8SiaR8KmfEinzyGDmJM+T4Y89xM2Ir1YGrvyIZ1+LM1xcghX8BK/cLMe/5SZKPblcEzwrHxIu/FDhrH/iTV1fz0W69+blsm24Bsu4L20z5YPaXfn0lD6OIA9eIvd2IvD2IavWAj0/2AlNtT9a7gonRMoH9LlwSFp/BdUXN7BfpUT/1XF2o94h404hxu1Xbyflg9p97G07bzSPI9t4CSu4wt2YZX0ebZ8SLvjsJFx+It1NuM11uE0brfYlw9pt7k28n6Y46+Pqyr+fuNDxq6zNyx/wd44+bS22sVB3Kvbx/FYPt528oblLzhQtxjnHh+PHav/X+B5i97sTDMxGdgPc9zl+FyLp9iJI3iW0JndDycucwzXjVJxNcXjJRV/Tyzxl8vTzWrAE5OBu8U4Oe2Gi+O4he84gwfStarZfjhxGfLDlMT1/dW4XPfvqvhj2G9F/Vm+dCKSvOOPkofYh884ih8j9Gf74USll6sz5GLrVlU+G06o+Ixt2s5G5UPabR0/lBR/m3C/bl+yxF/DWdv3VM03OeVD2s1+mLqTx3dFOIvtdftRYDMqxxrOWT6kyyPX1kgYE3fgvKVaoowe7ZyWD2l3HOa+q2nGD9XtRdVZJnV3zOVUO33D8hf8z2EuTxr71CucUtXrY722dtMvH9LllfNu4yMl49iXD+nUF/wHg+y5HmmDeFIAAAAASUVORK5CYII=", "text/plain": [ "28×28 reinterpret(Gray{Float64}, ::Array{Float64,2}):\n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0460847) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " ⋮ ⋱ \n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) … Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0460847)\n", " Gray{Float64}(0.0460847) Gray{Float64}(0.0)" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10,\n", " Gurobi.Optimizer,\n", " Dict(\"OutputFlag\" => 0),\n", " tightening_algorithm = lp,\n", " norm_order=Inf\n", ")\n", "print_summary(d)\n", "perturbed_sample_image = value.(d[:PerturbedInput])\n", "colorview(Gray, perturbed_sample_image[1, :, :, 1])" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwBAMAAAA0zul4AAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAJ1BMVEXp9v35+Pnr9/3s9/3/7+vx+v7/8Ovx+f7t+P3v+f7/9PH/9vP///9TFX8KAAAAAWJLR0QMgbNRYwAAAVxJREFUWMPVl0EVAjEMRGMBC1jAQi2sBSxgAQtYwALmoCxDJplyZ3KA3Sa/vDdJ2hARh49FzOeI+g1f+uHzAdmReA3PlYz0A+fL8ciypBC55elE0hmBPZcZyILouhNYJVhhnHpPsJf2b1Ta3gbUosZW6amt/G1kG5APpDEixtt6S1dp6JcsQF7csW0bYtM/htxVJiCnePywiO1lVUhPsC9P5HzmDbjlncB86VnlfS+XuREfZG6gqCS6qd8RZBe/zae9wFk0LxBOvUARer2iiZddYQDKdSkGadolZAf2IZ4NI3zzWoG8pAXP0iDCD1wPtnv47YZjmIdEPzDLIIthGoYl3ZDGXBNQ0Wn3O48MKpsT2NsZVrGe/sU58MfgWhpgj4deSOUEMAF5CWFdmm5tsjIBOdFoYYzvOgr7gdzO9c8ZF/9CHCOQ0wxx9suGsfz8HsgmYEqi7loY1W8EPgFw1YaIp86glAAAAABJRU5ErkJggg==", "text/plain": [ "28×28 Array{RGB{Float64},2} with eltype RGB{Float64}:\n", " RGB{Float64}(0.911923,0.96363,0.991867) … RGB{Float64}(0.911923,0.96363,0.991867)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.911923,0.96363,0.991867)\n", " RGB{Float64}(0.911923,0.96363,0.991867) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.911923,0.96363,0.991867) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.911923,0.96363,0.991867) RGB{Float64}(0.911923,0.96363,0.991867)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.911923,0.96363,0.991867)\n", " RGB{Float64}(0.911923,0.96363,0.991867) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.911923,0.96363,0.991867) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " ⋮ ⋱ \n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.911923,0.96363,0.991867)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.911923,0.96363,0.991867)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.911923,0.96363,0.991867)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.911923,0.96363,0.991867)\n", " RGB{Float64}(0.974944,0.972996,0.976035) … RGB{Float64}(0.911923,0.96363,0.991867)\n", " RGB{Float64}(0.911923,0.96363,0.991867) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.911923,0.96363,0.991867)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.911923,0.96363,0.991867)\n", " RGB{Float64}(0.911923,0.96363,0.991867) RGB{Float64}(0.974944,0.972996,0.976035)" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "diff = value.(d[:Perturbation])\n", "view_diff(diff[1, :, :, 1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### $L_2$\n", "With solvers that can handle MIQPs (like Gurobi), we can minimize over the $L_2$ norm. This generally takes a bit more time. \n", "\n", "In this case, the minimum $L_2$ norm perturbation required for the image to be classified as a `9` is `0.705367 = sqrt(0.497542).`" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 40%|█████████▎ | ETA: 0:00:00\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Objective Value: 0.497542, Solve Time: 815.09\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAAAAADji6uXAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAARHSURBVGje7ZrPixxFFMc/tdO9M7tudjbRBBU9eRL0IB5EhEBOimAgP/Af8ORN9OBR8C/wnxA8iiB6EA8b9xBPgoEkiuBBQUWSTXazyfT29Hj4vrdVMzvZOc20FPOgmanqqq6u963vq/dedQjAiHFJ67pAZXUNUACllQ8e069jbUfAiv0P1maFBUv+AxYpfq7vAji0uo5dtZVru0orbwG71sfbDKc809dA/ipdPIZpwTl0iHS+DjxAODlGBfAvEeeHyZs7D527q/Y/APfsN3+VLnzAsIr4USeVawibLWCfyLua43Y3fXPH3NfCkHG73MoM8x+w8L0OoAc8St5iA9nJM8CdGQ9qEN4Q18D/Yob5DxjSwinEtVT/jkdJ3CNniXP2LPBP2zPMf8AAsnsBOA3sIS66LXSfpko6pLaxY+URj7ezENdC/ipdPIY+4irCrkTYvQN8BPyI9P858Dfw28TbBrvqpK4hrgGXTZY+zZzkyJauAwPkRx4g3/MMwqFA/KyAn5OOK3Y5XwPwJ/ApcIO4v5ZA39rlr9LF+zRrCJsR8mEGduMy8DLC7FXgFeC8XbeB5xHezr87wLMIxz+ADxF+Aa2Be2g/zV+l7cT4GwjDLeCulbeBH+yNdhCHngZeAq4jXD2WHAC3gF/sGTeSAUp7tvtD+au0PVvaRXx5ADyJcPG8SopBlXTuILz3gavAF8BPwAUUk8DxOCN/lbYbW4B45DawtjeqrTwk+p494AmE9VMIuz7wLvCl9RskbQNLv3ROUkxW7CJ+dYmYuX/pgJ+yugr5P+8j/A6A3xH3aqJv+qjNGeY/YOggv/8uii+qKY38rGKIsNlEnOqiffF7a/c28J21a6zuHPHcY7eNGeY/YDFE+EHEz3MyJcJpkNzzXHdj15tWvw1cQ7xt0HoYWt8eii1bmWH+AxZp3mUd2cMG2cNRUg4IiwbZxhHC6aLVfUL0e9aQHS3R+liztsuzp7nI2PlhRfQja8bzo6lvWljbD4AXkf3csXse7z+0X4C/kK/bb2OG+Q84FlsMZjT0c/mA9r6vgPv2f2eivfszfbR/7rLk4ZykAMUI1YyGztcG+aWfWflrjuMH0Zd1e7yHbHX+Km2PhwXxvHCWbAOvAb8Cb6Hc2knSR9gvc21zkVAi/fYQX5xv076VAngBuIn2yyvANyc8vEC+TEB+0DLnPRcpDpFtnDz/m4bfM8C3du9jTsYP4vdV+/a73A/nIgVor3LZRL6N+zcpH98DnrO6axMP6iRtPT4coPhig8jx/FXabq6tQH5m+jYeG76BzpN6xBjfxXOpXYTdHvKTPE96YPWef12o5D/gEYae24b47eghMf/2OhG/W4xz1+N952+BzhOdf/59YiszzH/AIwxTXnkeLXD8O5rrwCV0zu+Sfm+TroUKYZ6eHeav0nZiC8dn2vdOHud5Hu4kOU3MvYJ8Jeer2+X8VbrwAf8D0LghKV4esdsAAAAASUVORK5CYII=", "text/plain": [ "28×28 reinterpret(Gray{Float64}, ::Array{Float64,2}):\n", " Gray{Float64}(0.00272082) … Gray{Float64}(0.00424106)\n", " Gray{Float64}(0.00071994) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.00289914)\n", " Gray{Float64}(0.0241623) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(3.3605e-5) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.00875976) Gray{Float64}(0.0577916)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0668495)\n", " Gray{Float64}(0.0345795) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0307308) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " ⋮ ⋱ \n", " Gray{Float64}(0.00392799) Gray{Float64}(0.0165317)\n", " Gray{Float64}(0.00120835) Gray{Float64}(0.131568)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0413101)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0405618)\n", " Gray{Float64}(0.0) … Gray{Float64}(0.0330876)\n", " Gray{Float64}(0.0195081) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.00496392)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) … Gray{Float64}(0.0)\n", " Gray{Float64}(0.0) Gray{Float64}(0.00827445)\n", " Gray{Float64}(0.00157579) Gray{Float64}(0.0)" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "d = MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10,\n", " Gurobi.Optimizer,\n", " Dict(\"OutputFlag\" => 0),\n", " tightening_algorithm = lp,\n", " norm_order=2\n", ")\n", "print_summary(d)\n", "perturbed_sample_image = value.(d[:PerturbedInput])\n", "colorview(Gray, perturbed_sample_image[1, :, :, 1])" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHAAAABwCAMAAADxPgR5AAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAilBMVEXy+v75+Pnx+f7s9/3x+v7w+f7v+f7r9/3q9v3u+P7l9P3p9v3l9Pzi8/zu+P3n9f3m9P3t+P3o9v3a7/vj8/zo9f3g8vzh8vz/8/D/6+X/9fL/9fPm9f3/9PH/7+v/8u7/8+//9vP/8e3k9Pz/8Oz/8OvY7vve8fzV7fvZ7/v/9PD/8u//7+r///81RR+xAAAAAWJLR0QtzdpBPQAAA2VJREFUaN7tmtlyFDEMRacDSQiBQNghLGFf///70JnymRKuSQ8U9pvuQ9ztsX2Uallty73ZhJZORyHrb4Vuh45D3J+ETkN3Qn0/+9CWPo5lSd2mgMOBuYODnoWsY2Ag/eC2uRvSkGWPbCe4gHOB2TlocB7i+l7ofsiSOhwHJ8GpuL8I6Sg6GuWD0MPQRhVwOPAy1D9wBtMh/I2H74C9GAdDH4V0Dg3v/6kCjgfqID5sSp3lcYjySWj5B2mwjljA+UD1NJQb5845oB+SDvYs1P9WwDlAJikT/HlIx3HiUhKYdx3SRHbCHzWtGaHhBRwPtDEgBuCBc/8iRKOXoVdNV6E8CO0AZofqg7fiRfw6VMDxQCtc8LpJoQEGvAkxid+GMOJd03WI3zCEesr3Ifp+CFHqgBjE4utjqIDjgTgGD5iSh2ojBmTgT6GrJgf93ASQNpTCMC4HAYxmbB2rgOOBANhs4CRfQkAJ4k5iHMlJTFvqhFOHwwDUiXQsgQR+/4kCzgF6QaULYCYpGxOCwHnTRVMOyBiCgb7ANSYH+X5RVsB5QMUm9LSJyYohJ015AUU9hmEMhuXFl/1y2z9WbQWcCkS+iJ34uwfegAQJfqd0AWYiiIH9vV9IFXAOkEoWN9z0EzsbwEAkkQDgWFwz6S+bAOJALqTty6aU9jC2SfYCDgfuA7jgoXQQAzSDW0eA91CEeoOExnONUV9Dq15awP8C5uSbGxkXxmh3wNEWtAx63JSDvM7DNXUYpTMCNBFcwPFAAzKlEzw7Th8AkMHaJN5ZUj48yYGfl/W3UAHHA63goS4r8oTaZJAv534Dqnz5MvG/h0j+4kgFnAN0s7L8pdh8AteItbYsqDzkJLAUcA4Q4QAcKh+CkTggWfQjRPLgZ+hQH5MW24PqAg4HukDCcfIEvungA6AwrtdAbooYe3cIXcDhQP7wQE0Krem6CSDOYlL2JmG0iSM/DingHKAiEAPPL+O8UOYABGcB2DuMCSEXuy6kCNxAXVAVcC6w/5DHyQoUB/Hgy6R6NpTSj6645oMRgjbAfIhWwHnAHKzdmHCtAwHBaXAYXsD5MCS3y4abfDBRX8C5wN5Z+g86ONwC+CvUJ17zfe7TL663HzsXcDjQTcn2gXYbkzzQvg/Je5kIUmxCe0MKOBz4GzljYSA9W5Y7AAAAAElFTkSuQmCC", "text/plain": [ "28×28 Array{RGB{Float64},2} with eltype RGB{Float64}:\n", " RGB{Float64}(0.948272,0.979477,0.995882) … RGB{Float64}(0.946653,0.978783,0.995703)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.948272,0.979477,0.995882)\n", " RGB{Float64}(0.930285,0.971707,0.9939) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.943404,0.977388,0.995346) RGB{Float64}(0.901748,0.959091,0.990733)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.894904,0.956012,0.989966)\n", " RGB{Float64}(0.921985,0.968075,0.992983) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.925315,0.969535,0.993351) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " ⋮ ⋱ \n", " RGB{Float64}(0.946653,0.978783,0.995703) RGB{Float64}(0.93687,0.974567,0.994626)\n", " RGB{Float64}(0.948272,0.979477,0.995882) RGB{Float64}(0.836567,0.928883,0.983228)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.915289,0.965122,0.992241)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.916968,0.965864,0.992427)\n", " RGB{Float64}(0.974944,0.972996,0.976035) … RGB{Float64}(0.921985,0.968075,0.992983)\n", " RGB{Float64}(0.933584,0.973142,0.994264) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.946653,0.978783,0.995703)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) … RGB{Float64}(0.974944,0.972996,0.976035)\n", " RGB{Float64}(0.974944,0.972996,0.976035) RGB{Float64}(0.943404,0.977388,0.995346)\n", " RGB{Float64}(0.948272,0.979477,0.995882) RGB{Float64}(0.974944,0.972996,0.976035)" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "diff = value.(d[:Perturbation])\n", "view_diff(diff[1, :, :, 1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Advanced Options" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `tightening_algorithm`\n", "\n", "By default, we tighten the bounds on each intermediate value by solving an MIP using the `optimize` with the `tightening_options` specified. Compare total solve times for three different tightening algorithms. As the tightening algorithm gets more complex (`interval_arithmetic -> lp -> mip`), the time spent on tightening bounds increases, but generally (not in this case with `mip`) with a reduction in the amount of time for the main solve. \n", "\n", "You'll have to find the sweet spot for your own application." ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n", "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Gurobi Optimizer version 9.0.2 build v9.0.2rc0 (linux64)\n", "Optimize a model with 1263 rows, 1020 columns and 66888 nonzeros\n", "Model fingerprint: 0xdc39e74c\n", "Variable types: 960 continuous, 60 integer (60 binary)\n", "Coefficient statistics:\n", " Matrix range [2e-05, 8e+02]\n", " Objective range [1e+00, 1e+00]\n", " Bounds range [5e-01, 4e+02]\n", " RHS range [4e-03, 8e+02]\n", "Presolve added 0 rows and 10 columns\n", "Presolve removed 50 rows and 0 columns\n", "Presolve time: 0.07s\n", "Presolved: 1213 rows, 1030 columns, 62933 nonzeros\n", "Variable types: 970 continuous, 60 integer (60 binary)\n", "\n", "Root relaxation: objective 0.000000e+00, 938 iterations, 0.05 seconds\n", "\n", " Nodes | Current Node | Objective Bounds | Work\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\n", "\n", " 0 0 0.00000 0 2 - 0.00000 - - 0s\n", "H 0 0 0.4967121 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 2 0.00000 0 2 0.49671 0.00000 100% - 0s\n", "H 29 31 0.0710671 0.00000 100% 250 1s\n", " 151 93 0.00448 7 10 0.07107 0.00000 100% 259 5s\n", " 406 136 cutoff 14 0.07107 0.00000 100% 202 10s\n", "H 420 131 0.0635998 0.00000 100% 200 10s\n", "H 546 154 0.0539995 0.00000 100% 189 12s\n", "H 567 144 0.0539995 0.00000 100% 188 12s\n", " 683 160 0.04687 8 4 0.05400 0.00448 91.7% 185 15s\n", " 973 183 infeasible 15 0.05400 0.00600 88.9% 171 20s\n", " 1308 212 cutoff 24 0.05400 0.00940 82.6% 166 25s\n", " 1801 234 cutoff 20 0.05400 0.02198 59.3% 146 30s\n", "* 1816 209 31 0.0502775 0.02198 56.3% 145 31s\n", " 2254 230 0.03211 13 10 0.05028 0.02944 41.4% 132 35s\n", "* 2352 191 31 0.0462748 0.02981 35.6% 129 35s\n", "H 2586 164 0.0460847 0.03250 29.5% 126 38s\n", " 2679 151 0.04096 25 3 0.04608 0.03325 27.9% 126 42s\n", " 2831 119 cutoff 19 0.04608 0.03599 21.9% 129 46s\n", " 3001 57 cutoff 23 0.04608 0.03950 14.3% 133 50s\n", "\n", "Cutting planes:\n", " Cover: 5\n", " Implied bound: 35\n", " MIR: 3\n", " Flow cover: 7\n", " Inf proof: 6\n", "\n", "Explored 3276 nodes (429716 simplex iterations) in 52.10 seconds\n", "Thread count was 4 (of 4 available processors)\n", "\n", "Solution count 7: 0.0460847 0.0462748 0.0502775 ... 0.496712\n", "\n", "Optimal solution found (tolerance 1.00e-04)\n", "Best objective 4.608468158892e-02, best bound 4.608468158892e-02, gap 0.0000%\n", " 52.207113 seconds (1.03 M allocations: 36.524 MiB)\n" ] }, { "data": { "text/plain": [ "Dict{Any,Any} with 11 entries:\n", " :TargetIndexes => [10]\n", " :SolveTime => 52.0963\n", " :TotalTime => 52.1851\n", " :Perturbation => GenericAffExpr{Float64,VariableRef}[noname noname … no…\n", " :PerturbedInput => VariableRef[noname noname … noname noname]…\n", " :TighteningApproach => \"interval_arithmetic\"\n", " :PerturbationFamily => unrestricted\n", " :SolveStatus => OPTIMAL\n", " :Model => A JuMP Model…\n", " :Output => GenericAffExpr{Float64,VariableRef}[-0.012063867412507…\n", " :PredictedIndex => 8" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "@time d = MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10,\n", " Gurobi.Optimizer,\n", " Dict(),\n", " tightening_algorithm = interval_arithmetic,\n", " norm_order=Inf\n", ")" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 35%|████████ | ETA: 0:00:00\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Gurobi Optimizer version 9.0.2 build v9.0.2rc0 (linux64)\n", "Optimize a model with 1263 rows, 1020 columns and 66888 nonzeros\n", "Model fingerprint: 0x0202e6a7\n", "Variable types: 960 continuous, 60 integer (60 binary)\n", "Coefficient statistics:\n", " Matrix range [2e-05, 7e+02]\n", " Objective range [1e+00, 1e+00]\n", " Bounds range [5e-01, 3e+02]\n", " RHS range [4e-03, 7e+02]\n", "Presolve added 0 rows and 10 columns\n", "Presolve removed 50 rows and 0 columns\n", "Presolve time: 0.09s\n", "Presolved: 1213 rows, 1030 columns, 62933 nonzeros\n", "Variable types: 970 continuous, 60 integer (60 binary)\n", "\n", "Root relaxation: objective 0.000000e+00, 837 iterations, 0.08 seconds\n", "\n", " Nodes | Current Node | Objective Bounds | Work\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\n", "\n", " 0 0 0.00000 0 1 - 0.00000 - - 0s\n", "H 0 0 0.4967121 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 2 0.00000 0 2 0.49671 0.00000 100% - 0s\n", "H 29 31 0.3535137 0.00000 100% 429 2s\n", " 67 89 0.01071 20 16 0.35351 0.00000 100% 423 5s\n", "H 95 102 0.2051025 0.00000 100% 369 5s\n", "* 145 139 42 0.2051025 0.00000 100% 320 6s\n", " 254 233 0.15085 15 8 0.20510 0.00000 100% 273 10s\n", "H 315 227 0.1476387 0.00000 100% 241 11s\n", "H 360 216 0.1309385 0.00000 100% 226 12s\n", " 496 245 infeasible 30 0.13094 0.00000 100% 196 15s\n", "* 736 285 40 0.1084155 0.00000 100% 178 19s\n", "H 764 249 0.0958362 0.00000 100% 178 21s\n", "H 810 268 0.0696157 0.00000 100% 176 24s\n", " 816 270 0.02997 25 5 0.06962 0.00000 100% 177 25s\n", "* 892 259 42 0.0574457 0.00447 92.2% 172 27s\n", "* 893 246 42 0.0574449 0.00447 92.2% 172 27s\n", " 985 240 0.03383 24 2 0.05744 0.00808 85.9% 168 30s\n", "* 1148 201 41 0.0564840 0.02084 63.1% 156 32s\n", "* 1149 191 41 0.0537551 0.02084 61.2% 156 32s\n", "* 1252 168 40 0.0502853 0.02545 49.4% 149 34s\n", "* 1255 145 40 0.0460847 0.02545 44.8% 149 34s\n", " 1334 124 0.03878 25 10 0.04608 0.02679 41.9% 144 35s\n", " 1734 111 cutoff 26 0.04608 0.03239 29.7% 127 40s\n", "\n", "Cutting planes:\n", " Gomory: 5\n", " Projected implied bound: 3\n", " MIR: 1\n", " Flow cover: 3\n", "\n", "Explored 2020 nodes (244073 simplex iterations) in 42.99 seconds\n", "Thread count was 4 (of 4 available processors)\n", "\n", "Solution count 10: 0.0460847 0.0502853 0.0537551 ... 0.130938\n", "\n", "Optimal solution found (tolerance 1.00e-04)\n", "Best objective 4.608468158892e-02, best bound 4.608468158892e-02, gap 0.0000%\n", " 43.813121 seconds (2.18 M allocations: 73.722 MiB)\n" ] }, { "data": { "text/plain": [ "Dict{Any,Any} with 11 entries:\n", " :TargetIndexes => [10]\n", " :SolveTime => 42.9934\n", " :TotalTime => 43.6904\n", " :Perturbation => GenericAffExpr{Float64,VariableRef}[noname noname … no…\n", " :PerturbedInput => VariableRef[noname noname … noname noname]…\n", " :TighteningApproach => \"lp\"\n", " :PerturbationFamily => unrestricted\n", " :SolveStatus => OPTIMAL\n", " :Model => A JuMP Model…\n", " :Output => GenericAffExpr{Float64,VariableRef}[-0.012063867412507…\n", " :PredictedIndex => 8" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "@time d = MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10,\n", " Gurobi.Optimizer,\n", " Dict(),\n", " tightening_algorithm = lp,\n", " norm_order=Inf\n", ")" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 10%|██▎ | ETA: 0:00:42\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:02:17\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:02:35\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Gurobi Optimizer version 9.0.2 build v9.0.2rc0 (linux64)\n", "Optimize a model with 1263 rows, 1020 columns and 66888 nonzeros\n", "Model fingerprint: 0x04b67a3b\n", "Variable types: 960 continuous, 60 integer (60 binary)\n", "Coefficient statistics:\n", " Matrix range [2e-05, 7e+02]\n", " Objective range [1e+00, 1e+00]\n", " Bounds range [5e-01, 4e+02]\n", " RHS range [4e-03, 7e+02]\n", "Presolve added 0 rows and 10 columns\n", "Presolve removed 50 rows and 0 columns\n", "Presolve time: 0.06s\n", "Presolved: 1213 rows, 1030 columns, 62933 nonzeros\n", "Variable types: 970 continuous, 60 integer (60 binary)\n", "\n", "Root relaxation: objective 0.000000e+00, 792 iterations, 0.04 seconds\n", "\n", " Nodes | Current Node | Objective Bounds | Work\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\n", "\n", " 0 0 0.00000 0 4 - 0.00000 - - 0s\n", "H 0 0 0.5984254 0.00000 100% - 0s\n", " 0 0 0.00000 0 6 0.59843 0.00000 100% - 0s\n", " 0 0 0.00000 0 8 0.59843 0.00000 100% - 0s\n", " 0 0 0.00000 0 8 0.59843 0.00000 100% - 0s\n", " 0 0 0.00000 0 6 0.59843 0.00000 100% - 0s\n", " 0 0 0.00000 0 6 0.59843 0.00000 100% - 0s\n", " 0 0 0.00000 0 3 0.59843 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.59843 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.59843 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.59843 0.00000 100% - 0s\n", " 0 2 0.00000 0 2 0.59843 0.00000 100% - 0s\n", "H 29 31 0.3535137 0.00000 100% 315 1s\n", "H 43 47 0.0976870 0.00000 100% 352 2s\n", " 140 106 0.00078 13 10 0.09769 0.00000 100% 289 5s\n", "H 226 109 0.0719725 0.00000 100% 234 6s\n", "* 265 107 30 0.0712215 0.00000 100% 235 7s\n", "H 336 119 0.0613848 0.00000 100% 227 9s\n", " 361 119 0.03817 23 5 0.06138 0.00000 100% 221 10s\n", "H 392 118 0.0584827 0.00000 100% 220 10s\n", "* 396 118 30 0.0584544 0.00000 100% 221 10s\n", " 546 144 0.03002 8 6 0.05845 0.00000 100% 226 15s\n", " 753 183 cutoff 17 0.05845 0.01146 80.4% 210 20s\n", " 928 239 cutoff 14 0.05845 0.02208 62.2% 208 26s\n", " 1138 266 0.03425 14 4 0.05845 0.02460 57.9% 203 30s\n", "* 1194 265 26 0.0565325 0.02491 55.9% 205 31s\n", "H 1318 197 0.0498354 0.02712 45.6% 203 34s\n", " 1367 203 0.02847 15 5 0.04984 0.02838 43.1% 202 35s\n", "* 1431 181 24 0.0460847 0.02863 37.9% 203 36s\n", " 1651 190 infeasible 25 0.04608 0.03074 33.3% 201 41s\n", " 1868 170 0.04118 29 2 0.04608 0.03473 24.6% 203 46s\n", " 2170 130 0.04593 22 2 0.04608 0.03748 18.7% 198 51s\n", " 2370 63 0.04523 12 4 0.04608 0.03884 15.7% 198 56s\n", "\n", "Cutting planes:\n", " Cover: 3\n", " Implied bound: 15\n", " MIR: 7\n", " Flow cover: 15\n", " Inf proof: 1\n", " RLT: 7\n", " Relax-and-lift: 1\n", "\n", "Explored 2616 nodes (505401 simplex iterations) in 59.17 seconds\n", "Thread count was 4 (of 4 available processors)\n", "\n", "Solution count 10: 0.0460847 0.0498354 0.0565325 ... 0.353514\n", "\n", "Optimal solution found (tolerance 1.00e-04)\n", "Best objective 4.608468158892e-02, best bound 4.608468158892e-02, gap 0.0000%\n", "359.844930 seconds (1.54 M allocations: 53.731 MiB)\n" ] }, { "data": { "text/plain": [ "Dict{Any,Any} with 11 entries:\n", " :TargetIndexes => [10]\n", " :SolveTime => 59.1741\n", " :TotalTime => 359.826\n", " :Perturbation => GenericAffExpr{Float64,VariableRef}[noname noname … no…\n", " :PerturbedInput => VariableRef[noname noname … noname noname]…\n", " :TighteningApproach => \"mip\"\n", " :PerturbationFamily => unrestricted\n", " :SolveStatus => OPTIMAL\n", " :Model => A JuMP Model…\n", " :Output => GenericAffExpr{Float64,VariableRef}[-0.012063867412507…\n", " :PredictedIndex => 8" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "@time d = MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10,\n", " Gurobi.Optimizer,\n", " Dict(),\n", " tightening_algorithm = mip,\n", " norm_order=Inf\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also modify many of the parameters of the solver to change behavior:\n", "\n", "We will be focusing on the parameters available via Gurobi (http://www.gurobi.com/documentation/9.0/refman/parameters.html), but other solvers often have similar options." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " \n", "### `main_solve_options`\n", "\n", "#### Muting Output\n", "To mute the output from the `GurobiSolver`, set `OutputFlag=0`.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Terminating early if a conditon is satisfied\n", "\n", "Sometimes, finding an adversarial example takes a long time:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You may want to terminate early when a particular condition is satisfied. Common reasons are:\n", "\n", " 1. Solve exceeding time limit\n", " 2. Lower bound on robustness proved (i.e. `BestBd` increases above a pre-determined threshold)\n", " 3. Counter-example found (i.e. `Incumbent` adversarial image found that is closer to the original image than expected).\n", " 4. Difference between `Incumbent` and `BestBd` falls below a pre-determined threshold.\n", " \n", "Fortunately, Gurobi has a parameter for all of these cases." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Terminate if time limit is reached\n", "Set `TimeLimit`:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 10%|██▎ | ETA: 0:00:01\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Gurobi Optimizer version 9.0.2 build v9.0.2rc0 (linux64)\n", "Optimize a model with 1263 rows, 1020 columns and 66888 nonzeros\n", "Model fingerprint: 0x0202e6a7\n", "Variable types: 960 continuous, 60 integer (60 binary)\n", "Coefficient statistics:\n", " Matrix range [2e-05, 7e+02]\n", " Objective range [1e+00, 1e+00]\n", " Bounds range [5e-01, 3e+02]\n", " RHS range [4e-03, 7e+02]\n", "Presolve added 0 rows and 10 columns\n", "Presolve removed 50 rows and 0 columns\n", "Presolve time: 0.11s\n", "Presolved: 1213 rows, 1030 columns, 62933 nonzeros\n", "Variable types: 970 continuous, 60 integer (60 binary)\n", "\n", "Root relaxation: objective 0.000000e+00, 837 iterations, 0.07 seconds\n", "\n", " Nodes | Current Node | Objective Bounds | Work\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\n", "\n", " 0 0 0.00000 0 1 - 0.00000 - - 0s\n", "H 0 0 0.4967121 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 2 0.00000 0 2 0.49671 0.00000 100% - 0s\n", "H 29 31 0.3535137 0.00000 100% 429 2s\n", " 88 102 0.02818 25 9 0.35351 0.00000 100% 379 5s\n", "H 95 102 0.2051025 0.00000 100% 369 5s\n", "* 145 139 42 0.2051025 0.00000 100% 320 6s\n", " 254 228 0.15085 15 8 0.20510 0.00000 100% 273 10s\n", "\n", "Cutting planes:\n", " MIR: 3\n", " Flow cover: 4\n", "\n", "Explored 300 nodes (75240 simplex iterations) in 10.00 seconds\n", "Thread count was 4 (of 4 available processors)\n", "\n", "Solution count 3: 0.205102 0.353514 0.496712 \n", "\n", "Time limit reached\n", "Best objective 2.051024945226e-01, best bound 0.000000000000e+00, gap 100.0000%\n", " 10.854215 seconds (2.07 M allocations: 68.026 MiB)\n" ] }, { "data": { "text/plain": [ "Dict{Any,Any} with 11 entries:\n", " :TargetIndexes => [10]\n", " :SolveTime => 10.0028\n", " :TotalTime => 10.8295\n", " :Perturbation => GenericAffExpr{Float64,VariableRef}[noname noname … no…\n", " :PerturbedInput => VariableRef[noname noname … noname noname]…\n", " :TighteningApproach => \"lp\"\n", " :PerturbationFamily => unrestricted\n", " :SolveStatus => TIME_LIMIT\n", " :Model => A JuMP Model…\n", " :Output => GenericAffExpr{Float64,VariableRef}[-0.012063867412507…\n", " :PredictedIndex => 8" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "@time d = MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10,\n", " Gurobi.Optimizer,\n", " Dict(\"TimeLimit\" => 10),\n", " tightening_algorithm = lp,\n", " norm_order=Inf\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Terminate if lower bound on robustness proved\n", "\n", "Set `BestBdStop`." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 40%|█████████▎ | ETA: 0:00:00\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Gurobi Optimizer version 9.0.2 build v9.0.2rc0 (linux64)\n", "Optimize a model with 1263 rows, 1020 columns and 66888 nonzeros\n", "Model fingerprint: 0x0202e6a7\n", "Variable types: 960 continuous, 60 integer (60 binary)\n", "Coefficient statistics:\n", " Matrix range [2e-05, 7e+02]\n", " Objective range [1e+00, 1e+00]\n", " Bounds range [5e-01, 3e+02]\n", " RHS range [4e-03, 7e+02]\n", "Presolve added 0 rows and 10 columns\n", "Presolve removed 50 rows and 0 columns\n", "Presolve time: 0.09s\n", "Presolved: 1213 rows, 1030 columns, 62933 nonzeros\n", "Variable types: 970 continuous, 60 integer (60 binary)\n", "\n", "Root relaxation: objective 0.000000e+00, 837 iterations, 0.07 seconds\n", "\n", " Nodes | Current Node | Objective Bounds | Work\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\n", "\n", " 0 0 0.00000 0 1 - 0.00000 - - 0s\n", "H 0 0 0.4967121 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 2 0.00000 0 2 0.49671 0.00000 100% - 0s\n", "H 29 31 0.3535137 0.00000 100% 429 2s\n", " 88 102 0.02818 25 9 0.35351 0.00000 100% 379 5s\n", "H 95 102 0.2051025 0.00000 100% 369 5s\n", "* 145 139 42 0.2051025 0.00000 100% 320 6s\n", " 313 263 0.10418 23 10 0.20510 0.00000 100% 242 10s\n", "H 315 227 0.1476387 0.00000 100% 241 10s\n", "H 360 216 0.1309385 0.00000 100% 226 11s\n", " 599 286 0.12051 22 11 0.13094 0.00000 100% 186 15s\n", "* 736 285 40 0.1084155 0.00000 100% 178 18s\n", "H 764 249 0.0958362 0.00000 100% 178 19s\n", " 765 248 0.09385 21 2 0.09584 0.00000 100% 178 20s\n", "H 810 268 0.0696157 0.00000 100% 176 22s\n", " 871 273 0.03477 19 9 0.06962 0.00447 93.6% 175 25s\n", "* 892 259 42 0.0574457 0.00447 92.2% 172 25s\n", "* 893 246 42 0.0574449 0.00447 92.2% 172 25s\n", "* 1148 201 41 0.0564840 0.02084 63.1% 156 29s\n", "* 1149 191 41 0.0537551 0.02084 61.2% 156 29s\n", "\n", "Cutting planes:\n", " Gomory: 5\n", " Projected implied bound: 3\n", " MIR: 1\n", " Flow cover: 3\n", "\n", "Explored 1153 nodes (180859 simplex iterations) in 29.40 seconds\n", "Thread count was 4 (of 4 available processors)\n", "\n", "Solution count 10: 0.0537551 0.056484 0.0574449 ... 0.205102\n", "\n", "Optimization achieved user objective limit\n", "Best objective 5.375506126689e-02, best bound 2.084402587199e-02, gap 61.2241%\n", " 30.399972 seconds (2.31 M allocations: 80.203 MiB, 0.22% gc time)\n" ] }, { "data": { "text/plain": [ "Dict{Any,Any} with 11 entries:\n", " :TargetIndexes => [10]\n", " :SolveTime => 29.3971\n", " :TotalTime => 30.2711\n", " :Perturbation => GenericAffExpr{Float64,VariableRef}[noname noname … no…\n", " :PerturbedInput => VariableRef[noname noname … noname noname]…\n", " :TighteningApproach => \"lp\"\n", " :PerturbationFamily => unrestricted\n", " :SolveStatus => OBJECTIVE_LIMIT\n", " :Model => A JuMP Model…\n", " :Output => GenericAffExpr{Float64,VariableRef}[-0.012063867412507…\n", " :PredictedIndex => 8" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "@time d = MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10,\n", " Gurobi.Optimizer,\n", " Dict(\"BestBdStop\" => 0.02),\n", " tightening_algorithm = lp,\n", " norm_order=Inf\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Terminate if adversarial example found closer than expected robustness\n", "\n", "Set `BestObjStop`." ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 30%|██████▉ | ETA: 0:00:00\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Gurobi Optimizer version 9.0.2 build v9.0.2rc0 (linux64)\n", "Optimize a model with 1263 rows, 1020 columns and 66888 nonzeros\n", "Model fingerprint: 0x0202e6a7\n", "Variable types: 960 continuous, 60 integer (60 binary)\n", "Coefficient statistics:\n", " Matrix range [2e-05, 7e+02]\n", " Objective range [1e+00, 1e+00]\n", " Bounds range [5e-01, 3e+02]\n", " RHS range [4e-03, 7e+02]\n", "Presolve added 0 rows and 10 columns\n", "Presolve removed 50 rows and 0 columns\n", "Presolve time: 0.07s\n", "Presolved: 1213 rows, 1030 columns, 62933 nonzeros\n", "Variable types: 970 continuous, 60 integer (60 binary)\n", "\n", "Root relaxation: objective 0.000000e+00, 837 iterations, 0.05 seconds\n", "\n", " Nodes | Current Node | Objective Bounds | Work\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\n", "\n", " 0 0 0.00000 0 1 - 0.00000 - - 0s\n", "H 0 0 0.4967121 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 2 0.00000 0 2 0.49671 0.00000 100% - 0s\n", "H 29 31 0.3535137 0.00000 100% 429 1s\n", " 67 89 0.01071 20 16 0.35351 0.00000 100% 423 5s\n", "H 95 102 0.2051025 0.00000 100% 369 5s\n", "* 145 139 42 0.2051025 0.00000 100% 320 7s\n", " 254 233 0.15085 15 8 0.20510 0.00000 100% 273 10s\n", "H 315 227 0.1476387 0.00000 100% 241 11s\n", "\n", "Cutting planes:\n", " Implied bound: 2\n", " MIR: 3\n", " Flow cover: 4\n", "\n", "Explored 349 nodes (80689 simplex iterations) in 11.58 seconds\n", "Thread count was 4 (of 4 available processors)\n", "\n", "Solution count 4: 0.147639 0.205102 0.353514 0.496712 \n", "\n", "Optimization achieved user objective limit\n", "Best objective 1.476386863211e-01, best bound 0.000000000000e+00, gap 100.0000%\n", " 12.117183 seconds (2.07 M allocations: 68.014 MiB)\n" ] }, { "data": { "text/plain": [ "Dict{Any,Any} with 11 entries:\n", " :TargetIndexes => [10]\n", " :SolveTime => 11.5785\n", " :TotalTime => 12.0981\n", " :Perturbation => GenericAffExpr{Float64,VariableRef}[noname noname … no…\n", " :PerturbedInput => VariableRef[noname noname … noname noname]…\n", " :TighteningApproach => \"lp\"\n", " :PerturbationFamily => unrestricted\n", " :SolveStatus => OBJECTIVE_LIMIT\n", " :Model => A JuMP Model…\n", " :Output => GenericAffExpr{Float64,VariableRef}[-0.012063867412507…\n", " :PredictedIndex => 8" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "@time d = MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10,\n", " Gurobi.Optimizer,\n", " Dict(\"BestObjStop\" => 0.2),\n", " tightening_algorithm = lp,\n", " norm_order=Inf\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Terminate if gap between `Incumbent` and `BestBd` is below threshold\n", "\n", "Set `MIPGap`." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n", "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\r", "\u001b[32m Calculating upper bounds: 20%|████▋ | ETA: 0:00:00\u001b[39m" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m Calculating upper bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n", "\u001b[32m Calculating lower bounds: 100%|███████████████████████| Time: 0:00:00\u001b[39m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Academic license - for non-commercial use only\n", "Academic license - for non-commercial use only\n", "Gurobi Optimizer version 9.0.2 build v9.0.2rc0 (linux64)\n", "Optimize a model with 1263 rows, 1020 columns and 66888 nonzeros\n", "Model fingerprint: 0x0202e6a7\n", "Variable types: 960 continuous, 60 integer (60 binary)\n", "Coefficient statistics:\n", " Matrix range [2e-05, 7e+02]\n", " Objective range [1e+00, 1e+00]\n", " Bounds range [5e-01, 3e+02]\n", " RHS range [4e-03, 7e+02]\n", "Presolve added 0 rows and 10 columns\n", "Presolve removed 50 rows and 0 columns\n", "Presolve time: 0.09s\n", "Presolved: 1213 rows, 1030 columns, 62933 nonzeros\n", "Variable types: 970 continuous, 60 integer (60 binary)\n", "\n", "Root relaxation: objective 0.000000e+00, 837 iterations, 0.07 seconds\n", "\n", " Nodes | Current Node | Objective Bounds | Work\n", " Expl Unexpl | Obj Depth IntInf | Incumbent BestBd Gap | It/Node Time\n", "\n", " 0 0 0.00000 0 1 - 0.00000 - - 0s\n", "H 0 0 0.4967121 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 0 0.00000 0 2 0.49671 0.00000 100% - 0s\n", " 0 2 0.00000 0 2 0.49671 0.00000 100% - 0s\n", "H 29 31 0.3535137 0.00000 100% 429 2s\n", "H 95 102 0.2051025 0.00000 100% 369 4s\n", " 101 117 0.05359 29 9 0.20510 0.00000 100% 364 5s\n", "* 145 139 42 0.2051025 0.00000 100% 320 6s\n", " 313 263 0.10418 23 10 0.20510 0.00000 100% 242 10s\n", "H 315 227 0.1476387 0.00000 100% 241 10s\n", "H 360 216 0.1309385 0.00000 100% 226 10s\n", " 599 286 0.12051 22 11 0.13094 0.00000 100% 186 16s\n", "* 736 285 40 0.1084155 0.00000 100% 178 18s\n", "H 764 249 0.0958362 0.00000 100% 178 20s\n", "H 810 268 0.0696157 0.00000 100% 176 23s\n", " 865 267 infeasible 38 0.06962 0.00447 93.6% 175 25s\n", "* 892 259 42 0.0574457 0.00447 92.2% 172 25s\n", "* 893 246 42 0.0574449 0.00447 92.2% 172 25s\n", "* 1148 201 41 0.0564840 0.02084 63.1% 156 29s\n", "* 1149 191 41 0.0537551 0.02084 61.2% 156 29s\n", " 1199 182 cutoff 26 0.05376 0.02293 57.3% 153 30s\n", "* 1252 168 40 0.0502853 0.02545 49.4% 149 30s\n", "* 1255 145 40 0.0460847 0.02545 44.8% 149 30s\n", "\n", "Cutting planes:\n", " Gomory: 5\n", " Projected implied bound: 3\n", " MIR: 1\n", " Flow cover: 3\n", "\n", "Explored 1401 nodes (197770 simplex iterations) in 31.95 seconds\n", "Thread count was 4 (of 4 available processors)\n", "\n", "Solution count 10: 0.0460847 0.0502853 0.0537551 ... 0.130938\n", "\n", "Optimal solution found (tolerance 4.00e-01)\n", "Best objective 4.608468158892e-02, best bound 2.895634473127e-02, gap 37.1671%\n", " 32.665103 seconds (2.07 M allocations: 68.027 MiB)\n" ] }, { "data": { "text/plain": [ "Dict{Any,Any} with 11 entries:\n", " :TargetIndexes => [10]\n", " :SolveTime => 31.9488\n", " :TotalTime => 32.6446\n", " :Perturbation => GenericAffExpr{Float64,VariableRef}[noname noname … no…\n", " :PerturbedInput => VariableRef[noname noname … noname noname]…\n", " :TighteningApproach => \"lp\"\n", " :PerturbationFamily => unrestricted\n", " :SolveStatus => OPTIMAL\n", " :Model => A JuMP Model…\n", " :Output => GenericAffExpr{Float64,VariableRef}[-0.012063867412507…\n", " :PredictedIndex => 8" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "@time d = MIPVerify.find_adversarial_example(\n", " n1, \n", " sample_image, \n", " 10,\n", " Gurobi.Optimizer,\n", " Dict(\"MIPGap\" => 0.4),\n", " tightening_algorithm = lp,\n", " norm_order=Inf\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Whew! That was a lot. The next tutorial will introduce you to everything you can extract from the results dictionary." ] } ], "metadata": { "kernelspec": { "display_name": "Julia 1.4.2", "language": "julia", "name": "julia-1.4" }, "language_info": { "file_extension": ".jl", "mimetype": "application/julia", "name": "julia", "version": "1.4.2" } }, "nbformat": 4, "nbformat_minor": 2 }