{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Multi-label classification\n", "Previously, we classified things into 2 classes: dogs and cats. But what if our image belongs to multiple classes?\n", "\n", "The process is largely the same in terms of fast.ai, but there are a few differences.\n", "\n", "## 1. Softmax --> Sigmoid\n", "Instead of using a `softmax` activation function to evaluate our classes, we'll use a __sigmoid__ function. Softmax wants to pick one thing -- remember, the $e^{activation}$ means differences between numbers will be greatly accentuated. Sigmoid works better than softmax because it will normalize between 0-1, but it will still allow us to normalize towards multiple classes." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl8XHW9//HXJ5ksTZN0TdPSLS3dWUrpAshPaKVIUSzXBQGvXBC1XhXQi8jiVfTi/alX3FDxIheQiygVUaCyyNoCytaW0kIXSppu6U7apM06mZnP/WOmJdS0mSaTnpnJ+/l4zCNzZr6TvL/NzDunZ86cY+6OiIhkl5ygA4iISOqp3EVEspDKXUQkC6ncRUSykMpdRCQLqdxFRLKQyl1EJAup3EVEspDKXUQkC4WC+sEDBw70ioqKTj22oaGB3r17pzZQQDSX9JQtc8mWeYDmst/SpUvfcfeyjsYFVu4VFRUsWbKkU49dtGgRM2fOTG2ggGgu6Slb5pIt8wDNZT8z25jMOG2WERHJQip3EZEspHIXEclCKncRkSzUYbmb2V1mttPM3jzE/WZmPzezSjNbYWYnpz6miIgciWTW3O8G5hzm/nOBsYnLPOC/ux5LRES6osNyd/fngd2HGXI+cI/HvQz0NbMhqQooIiJHLhX7uQ8FNrdZrk7cti0F31tE5KiLRGM0tERpCEdobo3S1BqluTVGS2uU5kiUpnCM5gPXo7RE4sstkRjhSIzW6P6LE47GaI3E4l+jMVojzpDcMN29y34qyt3aua3dE7Oa2Tzim24oLy9n0aJFnfqB9fX1nX5sutFc0lO2zCVb5gFHPpdw1KlvdfaFnfow7Gt16sPx5YZWpykCzVGnKRK/3tTqNEWhKeKEo903D4CTB3q3/15SUe7VwPA2y8OAre0NdPfbgdsBpk2b5p39hJY+qZaeNJf0ky3zgHfnEonG2L63me11ze9+TVzfsTf+dXd9mIYuNHSOQXFBiKL8EL3ycykI5VCYl0thXg698nIT1+PLBaFceuXnUhjKpSAvh/zcHPJCOeTnGnm5OQcu+aF3l6tWLe/230sqyn0BcIWZzQdOAercXZtkRKTTojFnY00D63Y1sLGmgU27G1n2djPfWbyQ6j1NRGLtbhx4j7xco19RPv1757/7tXce/Yvy6VOUT0lhiJKCECWFeRQXhiguCFGS+FqUn4tZexslUqN5U/fvhd5huZvZfcBMYKCZVQPfBvIA3P024DHgQ0Al0Ah8prvCikj2eae+hTe31LF2xz7WbN/H2h37eHtHPS2RWDujGwEoLy1gSJ9eDC4tZHCfQspLCxncpyD+tbSQspICigtC3VrQ6a7Dcnf3izu434EvpyyRiGSt5tYob26p4/XNtSzbXMvyzbVU72lqd+wxfQo5dlAxFQN6M3JAEfu2VfHhM09hRP8iCvNyj3LyzBPYUSFFJPuFIzGWV9fyYmUNL657h2WbaglH37tGXpSfy/HH9GHCkBLGDy5hfHkJ4waXUFqY955xixZtYlx5ydGMn9FU7iKSUjv3NvPMmp08vWoHL66roan13Tc2zWDC4BJOGt6Xk4b3ZfLwvowrLyE3p+duPukuKncR6bKNNQ38ZflWnlq9k+Wba99z39hBxZx27ADed+wAThk1gH698wNK2bOo3EWkU2rqW3j0jW08uGwLyza9W+gFoRzeP3YgsyeWM2vCIMpLCwNM2XOp3EUkae7O3ytr+O3LG3hm9c4DuyQW5ecy57jBzDl+MO8fW0avfL3hGTSVu4h0aG9zKw8sqebeVzZStasBgNwcY9b4Mv5pylDOnlROUb7qJJ3otyEih/ROfQt3/W09v31pI/taIgAMLi3kU6eM4KLpwxmkTS5pS+UuIv9ge10ztz23jvte3XTgw0Snju7PZe+rYPbEckK5Os9PulO5i8gBdU2t3PbcOu762/oDpT57YjlfmnUsJ4/oF3A6ORIqdxEhHIlxz0sb+OXCSmobWwH40AmDueqssUwYXBpsOOkUlbtID/fiunf41kNvsi7xRukpo/pzw4cmctLwvgEnk65QuYv0UDv3NfO9R1fz0OvxI3SPHtibb543kVnjB/XoA25lC5W7SA+0YPlWvvXQm9Q1tVIQyuHKD4zh82eMpiCk/dOzhcpdpAfZ0xDmmw+/yaMr4qdcOHNcGd89/3hGDCgKOJmkmspdpId44e1dXH3/cnbta6F3fi7fPG8SF00frk0wWUrlLpLlYjHnF89W8rNn1uIOMyr686MLJmttPcup3EWy2L6wc9ndi3l+7S7M4Kuzx3LlB8bqELs9gMpdJEut2rqXb7/YxO7mRvoV5XHLRVM4Y1xZ0LHkKFG5i2ShZ9fs4MrfL6Mh7Jw0vC+/+ueTOaZvr6BjyVGkchfJMnf/fT03PbKKmMOpQ3K5e96pOudoD6RyF8kS7s73H1/D7c9XAfHt65Nzt6jYeygd2k0kC0Rjzg1/foPbn68ilGP87MKT+OrscdrNsQfTmrtIhgtHYvzb/a/z6IptFIRyuO3TU5k1YVDQsSRgKneRDBaOxPjivUt5Zs1OSgpC3HnZdGaM6h90LEkDKneRDNUajXHlfa/xzJqd9C3K497PnsLxQ/sEHUvShLa5i2SgSDTGV//wOk+s3EFpYUjFLv9A5S6SYWIx59oHVvDoim2UFIT4rYpd2qFyF8kw3398NX9etoWi/Fzuvnw6k3VSDWmHyl0kg9z5t/X8zwvrCeUYt18yjakj9eaptE/lLpIhHlmxlf98dBUAN19wIv9v7MCAE0k6U7mLZIBXqmq4+g/LcYfrz53AR6cMCzqSpLmkyt3M5pjZW2ZWaWbXt3P/CDNbaGbLzGyFmX0o9VFFeqbNuxv54u9eIxyNcelpI/nCGaODjiQZoMNyN7Nc4FbgXGAScLGZTTpo2DeB+919CnAR8KtUBxXpiRpaInz+niXsbghz5rgybvzIcTqkgCQlmTX3GUClu1e5exiYD5x/0BgHShPX+wBbUxdRpGdyd67543LWbN/H6IG9+fnFU3SSDUmaufvhB5h9Apjj7p9LLF8CnOLuV7QZMwR4EugH9AZmu/vSdr7XPGAeQHl5+dT58+d3KnR9fT3FxcWdemy60VzSUzrM5eHKMA9WttIrBN86tRfHFB/5W2TpMI9U0VziZs2atdTdp3U40N0PewEuAO5os3wJ8IuDxlwNfC1x/TRgFZBzuO87depU76yFCxd2+rHpRnNJT0HPZeGaHT7yuke84vpH/NnVOzr/ffQ7SUtdmQuwxDvobXdParNMNTC8zfIw/nGzy2eB+xN/LF4CCgHtpyXSCdvqmrj6/uUAXD17nI7wKJ2STLkvBsaa2Sgzyyf+humCg8ZsAs4CMLOJxMt9VyqDivQEkWiMq+5bxu6GMO8fO5AvzxoTdCTJUB2Wu7tHgCuAJ4DVxPeKWWlmN5nZ3MSwrwGfN7PlwH3AZYn/PojIEfjJU2tZvGEP5aUF/PTCk8jRG6jSSUkd8tfdHwMeO+i2G9tcXwWcntpoIj3Lord28qtF68gx+PlFUxhYXBB0JMlg+oSqSBqoqW/hmj+uAODqs8dxyugBASeSTKdyFwmYe/z8p+/Ut3Dq6P58aaa2s0vXqdxFAvbA0mqeXLWDkoIQP7pgsrazS0qo3EUCtHl3I//xl/iRHr8z9ziG9SsKOJFkC5W7SECiMedr9y+nviXCuccP5mMnDw06kmQRlbtIQH7z9/W8umE3ZSUF/P+PnqADgklKqdxFArCxpoEfPfkWAD/42An0750fcCLJNip3kaNs/94xza0xzj/pGM6aWB50JMlCKneRo+yPS6p5cV0N/YryuPG8g0+NIJIaKneRo2jn3uYD50H9ztzjGKBPoUo3UbmLHEU3PrySvc0RZo0vY+7kY4KOI1lM5S5ylDy5cjt/Xbmd3vm5/Kf2jpFupnIXOQqawtEDH1a65pzxDO3bK+BEku1U7iJHwa0LK9lS28SkIaVccurIoONID6ByF+lmVbvquf35KgC++0/HE8rVy066n55lIt3I3fn2gpWEozE+OW0YU0f2CzqS9BAqd5Fu9Pib23nh7XcoLQxx3ZwJQceRHkTlLtJNGloifPeR+JuoX58zQfu0y1GlchfpJr9cWMm2umZOGNqHT80YEXQc6WFU7iLdYPPuRu58YT0AN51/HLk6AYccZSp3kW7wg8fXEI7G+OiUoUwZoTdR5ehTuYuk2Kvrd/PoG9sozMvh2jnjg44jPZTKXSSFYjE/8CbqvDOOZUgffRJVgqFyF0mhB5dt4Y0tdZSXFvCvZ44OOo70YCp3kRRpDEf44RNrALj2nAkU5YcCTiQ9mcpdJEVue66KHXtbOHFYHz46RSe7lmCp3EVSYHtdM7c/vw6Ab354Ejna9VECpnIXSYFfPPs2za0x5hw3mBmj+gcdR0TlLtJVm2oa+cPizeQYXHPOuKDjiAAqd5Eu+9nTa4nEnI9OGcaYQSVBxxEBVO4iXbJ2xz4efH0LebnGV2ePDTqOyAFJlbuZzTGzt8ys0syuP8SYT5rZKjNbaWa/T21MkfT0kyfX4g4XTR/B8P5FQccROaDDHXHNLBe4FTgbqAYWm9kCd1/VZsxY4AbgdHffY2aDuiuwSLpYUV3LX1dupyCUwxUfGBN0HJH3SGbNfQZQ6e5V7h4G5gPnHzTm88Ct7r4HwN13pjamSPr50ZNrAbj0fRWUlxYGnEbkvczdDz/A7BPAHHf/XGL5EuAUd7+izZiHgLXA6UAu8B13/2s732seMA+gvLx86vz58zsVur6+nuLi4k49Nt1oLumpo7m8tTvK919tpjAXbj6ziJL89NyvvSf9TjJJV+Yya9aspe4+raNxyXw+ur1n7cF/EULAWGAmMAx4wcyOd/fa9zzI/XbgdoBp06b5zJkzk/jx/2jRokV09rHpRnNJT4ebi7tz669fApr5wsyxfOTs9N39saf8TjLN0ZhLMptlqoHhbZaHAVvbGfOwu7e6+3rgLeJlL5J1nlu7i8Ub9tC3KI/PvX9U0HFE2pVMuS8GxprZKDPLBy4CFhw05iFgFoCZDQTGAVWpDCqSDtydHye2tX/xzGMpKcwLOJFI+zosd3ePAFcATwCrgfvdfaWZ3WRmcxPDngBqzGwVsBD4urvXdFdokaA8sXI7b2ypo6ykgH85rSLoOCKHlNQxSd39MeCxg267sc11B65OXESyUjTmB/aQueoDY+iVnxtwIpFD0ydURZL08OtbqNxZz7B+vbhw+oig44gclspdJAnhSIyfPh1fa//q7HHkh/TSkfSmZ6hIEu5fspnNu5s4tqy3TsQhGUHlLtKB5tYov3j2bQCuPns8uToRh2QAlbtIB3770kZ27G3huGNKOff4wUHHEUmKyl3kMPY1t/KrRZUAXPPB8Tp9nmQMlbvIYdz1tw3saWxl2sh+zBxfFnQckaSp3EUOobYxzB0vxD9ofc054zHTWrtkDpW7yCHc9lwV+1oivH/sQE4dPSDoOCJHROUu0o7a5hh3v7geiG9rF8k0KneRdvylqpXm1hgfnFTO5OF9g44jcsRU7iIH2by7kUWbI5jB17TWLhlK5S5ykJ8/8zZRh/MnH8P4wSVBxxHpFJW7SBvrdtXzp9eqybH4MWREMpXKXaSNnzy1lpjDGUNDVAzsHXQckU5TuYskrNxax6MrtpEfymHuGJ1hSTKbyl0k4SeJE3FccupI+hfqpSGZTc9gEWDpxj08s2YnRfm5fHHmsUHHEekylbv0eO7OzU+sAeDy00cxsLgg4EQiXadylx7v75U1vFy1m9LCEJ8/Y3TQcURSQuUuPZq7c/OTbwHwhTOPpU8vvZEq2UHlLj3a06t3snxzLQOL8/nM6RVBxxFJGZW79FixmPPjxFr7l2eNoSg/FHAikdRRuUuP9ZcVW1mzfR/H9CnkU6eMCDqOSEqp3KVHao3G+OlT8f3avzJ7LAWh3IATiaSWyl16pD8trWZDTSOjBvbm4ycPCzqOSMqp3KXHaW6NcsszbwPwb2ePI5Srl4FkHz2rpce59+WNbKtrZtKQUs47YUjQcUS6hcpdepR9za3curASgK+fM56cHJ30WrKTyl16lP95YT17GluZXtGPmePLgo4j0m2SKnczm2Nmb5lZpZldf5hxnzAzN7NpqYsokho19S3c+UIVANfOmYCZ1tole3VY7maWC9wKnAtMAi42s0ntjCsBrgJeSXVIkVS4deE6GsJRZo0vY3pF/6DjiHSrZNbcZwCV7l7l7mFgPnB+O+O+C/wQaE5hPpGU2FLbxL0vbwTg6+dMCDiNSPdLptyHApvbLFcnbjvAzKYAw939kRRmE0mZW55eSzgaY+7kY5h0TGnQcUS6XTIH02hvw6QfuNMsB/gpcFmH38hsHjAPoLy8nEWLFiUV8mD19fWdfmy60Vy639b6GH9c0kSuwemle5LKmK5zOVLZMg/QXI6Yux/2ApwGPNFm+QbghjbLfYB3gA2JSzOwFZh2uO87depU76yFCxd2+rHpRnPpfv/62yU+8rpH/IY/r0j6Mek6lyOVLfNw11z2A5Z4B73t7kltllkMjDWzUWaWD1wELGjzx6HO3Qe6e4W7VwAvA3PdfUkq/viIdMXyzbU8/uZ2CkI5XPWBsUHHETlqOix3d48AVwBPAKuB+919pZndZGZzuzugSGe5O997bDUAl51eweA+hQEnEjl6kjqAtbs/Bjx20G03HmLszK7HEum6p1fv5JX1u+lXlMeXZo4JOo7IUaVPqEpWao3G+P7j8bX2q84aq9PnSY+jcpesNH/xZqp2NVAxoIh/PmVk0HFEjjqVu2Sdfc2t3PJ0/EQc182ZQH5IT3PpefSsl6zz6+eqeKc+zNSR/Zhz/OCg44gEQuUuWWVbXRP/kzg42Dc+NFEHB5MeS+UuWeXHT66lJRLjwycMYerIfkHHEQmMyl2yxptb6vjTa9Xk5RrXzhkfdByRQKncJSu4O99esBJ3uPS0CkYO6B10JJFAqdwlKzz8+laWbtzDwOICvjJbhxkQUblLxqtviRw4zMB1c8ZTUqgPLImo3CXj3bqwkp37Wpg8vC8fP3lY0HFE0oLKXTLa+ncauPOF9QD8x9zjyMnRro8ioHKXDPfdR1YRjsa4YOowThreN+g4ImlD5S4Z69k1O3h2zU5KCkJcO0fnRRVpS+UuGakpHOXGh1cC8JXZYykrKQg4kUh6UblLRrrlmbep3tPExCGlXPa+iqDjiKQdlbtknDXb93LHC1WYwfc+ejyhXD2NRQ6mV4VklFjM+caf3yAScz59ykimjNDxY0Tao3KXjHLf4k28tqmWspICvq7jx4gckspdMsbOfc381+NrAPj2RyZRqk+iihySyl0ygrvz7w++yd7mCGeOK+PDJwwJOpJIWlO5S0ZYsHwrT63aQXFBiO997ASdhEOkAyp3SXs79zXz7QXxfdq/+eGJDO3bK+BEIulP5S5pbf/mmNrGVs4YV8aF04cHHUkkI6jcJa09/Hp8c0xJQYgfaHOMSNJU7pK2tte12Rxz3kSO0eYYkaSp3CUtRWPOv/3hdeqaWpk5voxPTtPmGJEjoXKXtHT781W8VFXDwOJ8bv7EZG2OETlCKndJO8s31/LjJ98C4OZPTNYRH0U6QeUuaaW+JcJX5i8jEnM+c3oFsyYMCjqSSEZSuUvacHe+9dCbbKhpZMLgEq7TCThEOi2pcjezOWb2lplVmtn17dx/tZmtMrMVZvaMmY1MfVTJdve+vJEHl22hV14uv7h4CoV5uUFHEslYHZa7meUCtwLnApOAi81s0kHDlgHT3P1E4AHgh6kOKtnttU17uOmRVQD84OMnMLa8JOBEIpktmTX3GUClu1e5exiYD5zfdoC7L3T3xsTiy8Cw1MaUbFZT38KXf/carVHnsvdVcP5JQ4OOJJLxkin3ocDmNsvVidsO5bPA410JJT1HJBrjqvnL2FbXzMkj+vKND00MOpJIVjB3P/wAswuAc9z9c4nlS4AZ7n5lO2M/DVwBnOnuLe3cPw+YB1BeXj51/vz5nQpdX19PcXFxpx6bbnr6XH63uoWnNkYozYf/eF8v+hWmx3v82fJ7yZZ5gOay36xZs5a6+7QOB7r7YS/AacATbZZvAG5oZ9xsYDUwqKPv6e5MnTrVO2vhwoWdfmy66clzuefF9T7yukd8zDce9VeqaronVCdly+8lW+bhrrnsByzxJDo2mdWkxcBYMxtlZvnARcCCtgPMbArwa2Cuu+9M9i+Q9FzPr93Fd/6SeAP1YycyY1T/gBOJZJcOy93dI8Q3tTxBfM38fndfaWY3mdncxLCbgWLgj2b2upktOMS3E+HtHfv48u9fIxpzvjTzWD4+Ve+/i6RaKJlB7v4Y8NhBt93Y5vrsFOeSLLWltol/uetV9jVHmHPcYK75oE5yLdId0uPdK+kRdjeEueTOV9hW18z0in789MKTyMnRAcFEuoPKXY6K+pYIn/nNq1TtamDC4BLuuHQ6vfL1CVSR7qJyl27XFI4y754lLK+uY3j/Xtxz+Qz69MoLOpZIVlO5S7dqCkf57P8u5sV1NZSVFPDby09hUGlh0LFEsl5Sb6iKdEZTOMrldy/mpap4sd/3+VOpGNg76FgiPYLW3KVb1LdE/qHYxwzKjk8XimQCrblLyr1T38JnfrOYN7bUUVZSwPx5p3JsmYpd5GhSuUtKbd7dyCV3vsKGmkZGDijinstnMHKANsWIHG0qd0mZDXVRvv7fL7JrXwuThpTyv5fP0PlPRQKicpeUeGTFVr73SjPhGJw2egC3/8tUSgq1u6NIUFTu0iWxmPOzp9fy82crAfjktGF895+OpyCkDyiJBEnlLp22pyHM1x9YztOrd5JjcOH4fL738RMx0yEFRIKmcpdOWbJhN1fdt4ytdc2UFob4xadOxreuVLGLpAmVuxyRaMz59fPr+PGTa4nGnCkj+vKLi6cwrF8Ri7YGnU5E9lO5S9LW7arn2gdWsHTjHgC+cMZorjlnPHm5+iycSLpRuUuHojHnrr+t50dPvkVLJMagkgL+6+MnMmvCoKCjicghqNzlsF7fXMuND7/Jiuo6AD5+8jBuPG8SfYq0m6NIOlO5S7veqW/hh39dw/1LqgEYXFrI9z52PB+YUB5wMhFJhspd3qMxHOE3f9/Abc+tY19zhLxc43PvH80Vs8bQu0BPF5FMoVerANASiXLfK5v45cJK3qkPAzBzfBk3njeJ0Trol0jGUbn3cPUtEea/uom7/raerXXNAEwe3pdrzxnP6WMGBpxORDpL5d5D7dzXzN1/38C9L29kb3MEgHHlxVzzwfGcPalcH0YSyXAq9x4kFnNeqqrh969u4smV22mNOgDTK/ox74xjOWvCIHJyVOoi2UDl3gNU72lkwfKt/GHxZjbWNAKQY3DOceXMO+NYpo7sF3BCEUk1lXuW2rG3mUdXbOORFVt5bVPtgduP6VPIhdNH8MnpwxjSp1eACUWkO6ncs0Qs5qzatpdn1+xk4Vs7eX1zLR7f6kKvvFzOmjiIj508lDPHDSJXm15Esp7KPYNtqW3ilaoaXlpXw6K1u9i1r+XAffm5OcwcX8ZHJh/DWRMHUZSvX7VIT6JXfIZojcZ4e0c9y6trWbx+N6+s382W2qb3jBlcWsisCYOYNb6M08cM1IeORHowvfrTUGM4QtWuBlZureONLXW8sWUvq7ftJRyJvWdcaWGIGaP6M72iP2eMK2PC4BLtwigigMo9MNGYs31vM2/tjrLt1U1U7qw/cDl4jXy/kQOKOH5oH6aP7MeMUQMYP7hE289FpF0q924QjTk1DS3s2vfuZVtdM9V7Gqne00T1nia21jYRiSXe8Xz1jfc8Pi/XqBjQm/GDSzhhaB9OGNqH44b2oU8vHYlRRJKTVLmb2RzgFiAXuMPdf3DQ/QXAPcBUoAa40N03pDZqMFoiUeqaWtnb1Epd4lLb+O71/Zea+nC8yOtbqKlvYX9vH86gkgJKclqZPHoIxw4qZkziMqJ/kU6AISJd0mG5m1kucCtwNlANLDazBe6+qs2wzwJ73H2MmV0E/BdwYXcErm+JUNsSo3pPI+FIjNaoE47ECEejtLRdTtzWGnFaorHE2PjXptYojS0RGsPRxCVCQzhKUzhKQzgS/9oSoak1euBTnEdqQO98ykoK4pfiAgaVFjK8fy+G9StiWL9eDO3bi8K8XBYtWsTMmSel+F9JRHq6ZNbcZwCV7l4FYGbzgfOBtuV+PvCdxPUHgF+ambl755rxML5471JeeLsJFi5M9bduVyjH6NMrL34pynv3+kGXAcX5lBUXUlZSwIDifK15i0igkin3ocDmNsvVwCmHGuPuETOrAwYA77QdZGbzgHkA5eXlLFq06IgDt9Y3U5Ln5OXmkJcDuTmQl2OEDEKJ6/Hb3l0OJa6HEtcLcqAgZBTkQkHuu18LQ/+4HHrPG5atiUsbUaA+fqkhfjkS9fX1nfp3SEeaS/rJlnmA5nKkkin39nbHOHiNPJkxuPvtwO0A06ZN85kzZybx499r5kwSmzKO/LHpSHNJT9kyl2yZB2guRyqZbQfVwPA2y8OArYcaY2YhoA+wOxUBRUTkyCVT7ouBsWY2yszygYuABQeNWQBcmrj+CeDZ7tjeLiIiyelws0xiG/oVwBPEd4W8y91XmtlNwBJ3XwDcCfzWzCqJr7Ff1J2hRUTk8JLaz93dHwMeO+i2G9tcbwYuSG00ERHpLO2vJyKShVTuIiJZSOUuIpKFVO4iIlnIgtpj0cx2ARs7+fCBHPTp1wymuaSnbJlLtswDNJf9Rrp7WUeDAiv3rjCzJe4+LegcqaC5pKdsmUu2zAM0lyOlzTIiIllI5S4ikoUytdxvDzpACmku6Slb5pIt8wDN5Yhk5DZ3ERE5vExdcxcRkcPI6HI3syvN7C0zW2lmPww6T1eZ2TVm5mY2MOgsnWVmN5vZGjNbYWYPmlnfoDMdCTObk3hOVZrZ9UHn6SwzG25mC81sdeL18ZWgM3WFmeWa2TIzeyToLF1hZn3N7IHEa2S1mZ3WXT8rY8vdzGYRP73fie5+HPCjgCN1iZkNJ36e2k1BZ+mip4Dj3f1EYC1wQ8B5ktbmfMHnApOAi80b8MUvAAACm0lEQVRsUrCpOi0CfM3dJwKnAl/O4LkAfAVYHXSIFLgF+Ku7TwAm041zythyB74I/MDdWwDcfWfAebrqp8C1tHMGq0zi7k+6eySx+DLxk7tkigPnC3b3MLD/fMEZx923uftriev7iJfI0GBTdY6ZDQM+DNwRdJauMLNS4Azih0jH3cPuXttdPy+Ty30c8H4ze8XMnjOz6UEH6iwzmwtscfflQWdJscuBx4MOcQTaO19wRhZiW2ZWAUwBXgk2Saf9jPiKTyzoIF00GtgF/CaxiekOM+vdXT8sqeO5B8XMngYGt3PXvxPP3o/4fzmnA/eb2eh0PQNUB3P5BvDBo5uo8w43F3d/ODHm34lvGvjd0czWRUmdCziTmFkx8Cfgq+6+N+g8R8rMzgN2uvtSM5sZdJ4uCgEnA1e6+ytmdgtwPfCt7vphacvdZx/qPjP7IvDnRJm/amYx4sdr2HW08h2JQ83FzE4ARgHLzQzimzFeM7MZ7r79KEZM2uF+LwBmdilwHnBWuv6xPYRkzhecMcwsj3ix/87d/xx0nk46HZhrZh8CCoFSM7vX3T8dcK7OqAaq3X3//6AeIF7u3SKTN8s8BHwAwMzGAflk4EGF3P0Ndx/k7hXuXkH8CXByuhZ7R8xsDnAdMNfdG4POc4SSOV9wRrD4msKdwGp3/0nQeTrL3W9w92GJ18ZFxM/PnInFTuI1vdnMxiduOgtY1V0/L63X3DtwF3CXmb0JhIFLM2wtMVv9EigAnkr8T+Rld//XYCMl51DnCw44VmedDlwCvGFmrydu+0bilJkSnCuB3yVWHqqAz3TXD9InVEVEslAmb5YREZFDULmLiGQhlbuISBZSuYuIZCGVu4hIFlK5i4hkIZW7iEgWUrmLiGSh/wMu/HNySa51rQAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# The sigmoid function aka logistic function\n", "# Sigmoid will be close to 0 when -ve, and close to 1 when +ve\n", "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "\n", "def sigmoid(z):\n", " # Apply sigmoid activation function\n", " return 1/(1+np.exp(-z))\n", "\n", "test_input = np.arange(-6, 6, 0.01)\n", "plt.plot(test_input, sigmoid(test_input), linewidth=2)\n", "plt.grid(1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Read from CSV\n", "In our previous function, we were able to use a keras-style set up -- put each class in a folder with that class name. Since each image can belong to multiple classes, we'd have to copy each image to multiple folders, which isn't really practical or optimal. So instead we must read our data from a .csv file.\n", "\n", "## Note on one-hot encoding\n", "In single classification, our data might be one-hot encoded. Imagine we have a bunch of data like this:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([['Dog', 'Cat', 'Frog', 'Bird'],\n", " ['0', '0', '1', '0']], dtype='" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Let's take a look at the first image\n", "# Note, because the image is just a matrix of numbers, we can *1.4 to make it brighter\n", "plt.imshow(data.val_ds.denorm(to_np(images))[0]*1.4);" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[('agriculture', 1.0),\n", " ('artisinal_mine', 0.0),\n", " ('bare_ground', 0.0),\n", " ('blooming', 0.0),\n", " ('blow_down', 0.0),\n", " ('clear', 1.0),\n", " ('cloudy', 0.0),\n", " ('conventional_mine', 0.0),\n", " ('cultivation', 0.0),\n", " ('habitation', 0.0),\n", " ('haze', 0.0),\n", " ('partly_cloudy', 0.0),\n", " ('primary', 1.0),\n", " ('road', 0.0),\n", " ('selective_logging', 0.0),\n", " ('slash_burn', 0.0),\n", " ('water', 1.0)]" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# And to see which labels this has\n", "list(zip(data.classes, labels[0]))" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Start off training on small images\n", "image_size=64" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": true }, "outputs": [], "source": [ "data = get_data(image_size)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "31148b41e48344949d8ce54289e7534e", "version_major": 2, "version_minor": 0 }, "text/html": [ "

Failed to display Jupyter Widget of type HBox.

\n", "

\n", " If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n", " that the widgets JavaScript is still loading. If this message persists, it\n", " likely means that the widgets JavaScript library is either not installed or\n", " not enabled. See the Jupyter\n", " Widgets Documentation for setup instructions.\n", "

\n", "

\n", " If you're reading this message in another frontend (for example, a static\n", " rendering on GitHub or NBViewer),\n", " it may mean that your frontend doesn't currently support widgets.\n", "

\n" ], "text/plain": [ "HBox(children=(IntProgress(value=0, max=6), HTML(value='')))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "# This goes through and resizes all of our images for efficiencies sake?\n", "data = data.resize(int(image_size*1.3), \"tmp\")" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Now we want to find the learning rate\n", "# First set up our model\n", "learn = ConvLearner.pretrained(f_model, data, metrics=metrics)" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d10efd9527314edaa84cd57252fe5ae9", "version_major": 2, "version_minor": 0 }, "text/html": [ "

Failed to display Jupyter Widget of type HBox.

\n", "

\n", " If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n", " that the widgets JavaScript is still loading. If this message persists, it\n", " likely means that the widgets JavaScript library is either not installed or\n", " not enabled. See the Jupyter\n", " Widgets Documentation for setup instructions.\n", "

\n", "

\n", " If you're reading this message in another frontend (for example, a static\n", " rendering on GitHub or NBViewer),\n", " it may mean that your frontend doesn't currently support widgets.\n", "

\n" ], "text/plain": [ "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "epoch trn_loss val_loss f2 \n", " 0 0.220537 0.422463 0.789931 \n", "\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEOCAYAAABmVAtTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl8VfWd//HX52aHhLCFLewIIoqCBMSliri2WrDVUbC2Wm2t7Vhb7TI67cxY7fabtuPUfWundrFo1VFUWrW21h0JgmyKsglhDQRC9vXz++NeMzEkISH3cHJv3s/H4z4459zvPffzzQ33nbN9j7k7IiIiAJGwCxARke5DoSAiIk0UCiIi0kShICIiTRQKIiLSRKEgIiJNFAoiItJEoSAiIk0UCiIi0kShICIiTVLDLqCzBg4c6KNHjw67DBGRhLJ06dLd7p53sHYJFwqjR4+msLAw7DJERBKKmX3YkXbafSQiIk0UCiIi0kShICIiTRQKIiLSRKEgIiJNFAoiItIk4U5JDVp1XQO7y2sY3q9Xq883NjpFe6uoa2xk+75qhvbNJL9vFjV1jRSXVzMkN4vsjI//WN2dfZV11DU2kp2RSl2Dk54SITMtgpk1tVlRVEpdQyN9e6Xj7mSlp5CblUZ2RmpTOxGRIPWoUGhodFIibX+57iqrZt59b7JpTwUnjBnA6IG96ZOVyrZ91Rw1NIf3tpfx9ua9FO2tavd98nIyyEiNkJWWQlVdA7v211Db0HhAu/TUCH2z0shKT2Hn/mqq6w5s01zv9BSG9c0iv18WQ3OzGNA7nX6908nJSCUzPYWGxkb69UonJzONuoZG+mSmMSQ3k3690j4WKo2Nzr6qOkqr6thbWcvwvlns3F/DhyUV9MlMY2huJjv2V5OTmcbRw/rgDlWxsBzQO7r+2vpGstJTDvITF5FE02NC4anlW7n3Hxt49Csz2VxSiTvkZqXRr3c6v3vjQzaXVPKXVdupqGngtAl57K2s4/nVOyitqiM7M5Wn39lGv15pHD0sl6/OGkev9BQGZmewbV8VW/dVk5ORSl5OBttLq9lQXE5tQyM1dY1kpkUYnJvJ4JxM0lIjlFfXk5Zi1NQ3sq+ylt3ltZRV13PWUYOZMCSHvOwMymrqMaCytp7SqjoqahpwoLy6nq37KinaW8WqraWUVNTS6Afve+/0FNJTI6REItQ3NlJZ09BqSHVWasQY3CeTsXm9SUuJMKJfFpGIkZ4aIScjlQ/3VFJZ20BlbT0pESMvJ4OxA7OZMrIv00b2I9JOQItIOHpMKAzKyWTtjv0c94Pnm75IM1Ij9OuVzo791WSmRaipb+T2eVP59HHDml7XGGu8s6yavOwMUlO6z2GYxkantKqO/dV17K2sIzsjlZKKWipq6gGorG1ge2kVW/dVUd/g1Dc66SlGZnoKedkZ9MlKo1+vdLaXVjEoJ5MR/bMoqailpKKWvJwM9lbUsXZnGRmpkaaf1b6qOvZX1ZGeGqGsup4dpVVs2F1BbX0jSzaVgENNfSO1DY0MyskgOzOVXukpVNc18vbmfZRUbAGiQTWoTyZ52RkMzEknLzuDQX0ymTGmP8P7ZTGkT6Z2mYmEwNw78KdmN1JQUOCHOszFC2t28pdVO5g4JIey6jre21FGTX0jX/7EWE4aN4Dd5TUM6pMZ54p7nvqGRuoavNXdS7vLa3j1g90s37KP4vIaistq2B37t6y6vqldXk4G500eyuyJg5g2qh+9M3rM3y8igTCzpe5ecNB2PSkUpHvbW1HL0g/3sr20itfX7+GFNTupb3RSI8bk4bnMHDuAmWMHUKCQEOk0hYIkvIqaepZ+uJc3N+zhzQ17WFFUSn3sZIFjFRIinaJQkKTTPCQWbyzhnS37mkLipHEDuHTGSM6cNJi0bnTcR6S7UChI0qusjYbE6+v38NSyrWwrrWZgdjrnHjOE62aP1/EhkWa6RSiY2bnAL4EU4EF3/2mL50cCDwF9Y21udPdF7a1ToSCtaWh0Xlq7iyeWbeWF1TtJiRizjxrEuUcPYcqIvozo3/rFiCI9ReihYGYpwPvAWUARsASY7+5rmrW5H1jm7veY2SRgkbuPbm+9CgU5mPXF5Tz4ykZeWLOD3eW1AJwxcRDXnzWBY/JzQ65OJBwdDYUgj87NANa5+4ZYQQuAucCaZm0c6BObzgW2BViP9BDj8rL5yWcn88MLjmH5lr288sFufv3qRs6/41XOmjSYb545nqOHKRxEWhNkKOQDW5rNFwEntGhzM/C8mX0d6A2cGWA90sOkRIxpo/ozbVR/rjxlDL9+dSO/enUj592+kxPHDuB75x2lLQeRFoI8TaO1y1Fb7quaD/zG3YcDnwJ+Z2YH1GRmV5tZoZkVFhcXB1CqJLs+mWl888wJvPrd2Xz77Al8sKucT9/5Kp+9+zUKN5WEXZ5ItxFkKBQBI5rND+fA3UNXAY8CuPsbQCYwsOWK3P1+dy9w94K8vLyAypWeILdXGtfOHs+LN5zGN84Yz/bSai669w2u/m0hb20sIdHOxhOJtyBDYQkw3szGmFk6MA9Y2KLNZuAMADM7imgoaFNAApfbK7rl8OK3TuP6MyfwxoY9XHzfG8y96zWWbd4bdnkioQn6lNRPAf9N9HTTX7v7j8zsFqDQ3RfGzjh6AMgmumvpu+7+fHvr1NlHEoTK2noWLt/GbX99n537azj/2KF8bdYRTBrW5+AvFkkAoZ+SGhSFggSpvKaee15ax0Ovf0h5TT1nHjWYH15wDENydSGcJLaOhoLGAxBpJjsjle+cM5HXbpzNt86awGvrdnPOf7/Mn1duD7s0kcNCoSDSitysNL5+xnieve4URvbvxVf/8DbXP7KckorasEsTCZRCQaQdY/OyefyrJ3Hd7CN4+p1tnHf7K7y5YU/YZYkERqEgchDpqRFuOPtInvznk4mYcekDb/LsCu1OkuSkUBDpoGPyc3n++lOZNqof31iwjEeXbDn4i0QSjEJBpBN6Z6TyqyumM3PsAL77+ApuemIlNfUNYZclEjcKBZFO6pOZxkNXzuBrs8bxx7c2c+E9r7O9tCrsskTiQqEgcghSIsZ3z53IA18oYNPuSr70UCG7y2vCLkukyxQKIl1w1qTB3DF/Kut2lTP3ztd4b8f+sEsS6RKFgkgXnT5xEI9+5UTqGhq58O7XeWO9TlmVxKVQEImD40b0ZeG1pzCsbxbX/H4pH+6pCLskkUOiUBCJkyG5mfzq8umYwaUPLGbznsqwSxLpNIWCSByNHNCL3115AhW19Vz+P2+xdZ/OSpLEolAQibPJw3P51eUF7C6v4aJ7XueDnWVhlyTSYQoFkQBMG9WfR64+kfpGZ/4Di9mmLQZJEAoFkYBMGtaHh790AjV1DVz1UCEVNfVhlyRyUAoFkQCNH5zDHZdOZe2O/Vz/yHIaGxPrplbS8ygURAI268hB/Nv5k3h+zU5+9vzasMsRaVdq2AWI9ARXnDSaD3aVc89L6zkiL5sLpw0PuySRVmlLQeQwMDN+MOdoTho3gJueWEnhppKwSxJplUJB5DBJS4lw9+eOZ1jfTL7yu6VsKdHFbdL9KBREDqO+vdL51RXTqW1o5EsPFVJVq3sxSPeiUBA5zMblZXPnpcezdmcZtz67JuxyRD4m0FAws3PNbK2ZrTOzG1t5/jYzWx57vG9m+4KsR6S7OG1CHl85dSwPL97MX1btCLsckSaBhYKZpQB3AZ8EJgHzzWxS8zbufr27T3H3KcAdwBNB1SPS3Xzr7COZnJ/LjU+s0J3bpNsIckthBrDO3Te4ey2wAJjbTvv5wB8DrEekW0lPjfDLeVOorW/k+keW06AL26QbCDIU8oEtzeaLYssOYGajgDHA39p4/mozKzSzwuLi4rgXKhKWsXnZ3DznaN7cUMK9/1gfdjkigYaCtbKsrT+F5gGPuXurp2K4+/3uXuDuBXl5eXErUKQ7+Kdpw/nU5CH88sUP2LhbN+eRcAUZCkXAiGbzw4FtbbSdh3YdSQ9lZtz86aPJSInw70+twl27kSQ8QYbCEmC8mY0xs3SiX/wLWzYysyOBfsAbAdYi0q0N6pPJt885klc+2M3TK7aHXY70YIGFgrvXA9cCzwHvAo+6+2ozu8XM5jRrOh9Y4PrzSHq4y2aOYnJ+Lrc+s4b91XVhlyM9lCXad3FBQYEXFhaGXYZIIFYU7WPuXa/xhZmj+MHcY8IuR5KImS1194KDtdMVzSLdyLHD+zJ/xkj+sHizrl2QUCgURLqZr542Dgfu/rtOUZXDT6Eg0s2M6N+Ly04YyR8Wf8h7O/aHXY70MAoFkW7om2dOICczjVueXqNTVOWwUiiIdEP9eqdzw1kTeH39Hp5fszPscqQHUSiIdFOfO2EkEwZn86Nn36WmXvddkMNDoSDSTaWmRPi38yexuaSS37y2KexypIdQKIh0Y58Yn8dpE/K47+UNVNbWh12O9AAKBZFu7rozjqCkopY/vLk57FKkB1AoiHRz00b156RxA7jv5Q1U1+nYggRLoSCSAL4+ezy7y2tY8Ja2FiRYCgWRBDBzbH+mj+7Hvf/YoDORJFAKBZEEYGZcd8Z4duyv5rGlRWGXI0lMoSCSIE45YiBTRvTlnpfWU9fQGHY5kqQUCiIJIrq1cARFe6t4anlbNzEU6RqFgkgCOf3IQUwcksP9L6/XmEgSCIWCSAIxM64+dSzv7yznpbXFYZcjSUihIJJgPn3cMIbmZnLvP3S/BYk/hYJIgklLiXDVKWNYvLGE5Vv2hV2OJBmFgkgCmjdjJDmZqdz/srYWJL4UCiIJKDsjlctmjuIvq3awaXdF2OVIElEoiCSoL540mtRIhF+/tjHsUiSJBBoKZnauma01s3VmdmMbbS42szVmttrMHg6yHpFkMqhPJnOmDONPhUXsq6wNuxxJEoGFgpmlAHcBnwQmAfPNbFKLNuOBm4CT3f1o4JtB1SOSjK46ZQxVdQ08rIHyJE6C3FKYAaxz9w3uXgssAOa2aPNl4C533wvg7rsCrEck6Rw1tA+nHDGQh17fRG29hr6QrgsyFPKBLc3mi2LLmpsATDCz18zsTTM7N8B6RJLSVaeMYef+Gv6yekfYpUgSCDIUrJVlLa/LTwXGA7OA+cCDZtb3gBWZXW1mhWZWWFysqzhFmjttQh75fbN4ZIl2IUnXBRkKRcCIZvPDgZajeBUBT7l7nbtvBNYSDYmPcff73b3A3Qvy8vICK1gkEUUixiXTR/Dauj1sKakMuxxJcEGGwhJgvJmNMbN0YB6wsEWbJ4HTAcxsINHdSRsCrEkkKV00bTgRg0cLtxy8sUg7AgsFd68HrgWeA94FHnX31WZ2i5nNiTV7DthjZmuAvwPfcfc9QdUkkqyG9c3itAl5/KmwiHrda0G6INDrFNx9kbtPcPdx7v6j2LJ/d/eFsWl39xvcfZK7T3b3BUHWI5LMLpk+gh37q3n5Ax13k0OnK5pFksTsiYMZmJ3Ogre0C0kOnUJBJEmkp0a48PjhvPjeLnaVVYddjiQohYJIErl4+ggaGp3Hl24NuxRJUAoFkSQyLi+bGaP788iSzbpdpxwShYJIkrlk+gg27alk8caSsEuRBKRQEEkyn5o8lJyMVB5ZogPO0nkKBZEkk5Wewtypw1i0cjulVXVhlyMJRqEgkoTmTR9JTX0jC5frgLN0jkJBJAkdk5/L0cP6sEC7kKSTFAoiSWre9BGs3rafVVtLwy5FEohCQSRJzZmST0ZqhAUaUls6QaEgkqRys9I4b/JQnlq2jarahrDLkQTRoVAws2+YWR+L+pWZvW1mZwddnIh0zSXTR1BWU8+ildvDLkUSREe3FK509/3A2UAe8EXgp4FVJSJxMWNMf8YM7M0jus+CdFBHQ+GjW2t+Cvgfd3+H1m+3KSLdiJlx4fH5vLWxRHdlkw7paCgsNbPniYbCc2aWA+hOHiIJYO6UfAAWvtPybrgiB+poKFwF3AhMd/dKII3oLiQR6eZG9O/FjNH9eeLtIg2SJwfV0VA4EVjr7vvM7DLg+4BOfhZJEJ85Pp/1xRWs1DULchAdDYV7gEozOw74LvAh8NvAqhKRuPrU5KGkp0R44m0NeyHt62go1Ht0u3Mu8Et3/yWQE1xZIhJPuVlpnHHUIJ5+Zxt1DTocKG3raCiUmdlNwOeBZ80shehxBRFJEJ+Zms+eilpe/WB32KVIN9bRULgEqCF6vcIOIB/4WWBViUjczTpyEP16pfHEMu1CkrZ1KBRiQfAHINfMzgeq3f2gxxTM7FwzW2tm68zsxlaev8LMis1seezxpU73QEQ6JD01wvnHDuP51Tsoq9Z9FqR1HR3m4mLgLeCfgIuBxWZ20UFekwLcBXwSmATMN7NJrTR9xN2nxB4Pdqp6EemUC6bmU1PfyJ9X7Qi7FOmmOrr76HtEr1G43N2/AMwA/u0gr5kBrHP3De5eCywgeqBaREJy/Mi+jB7Qiye1C0na0NFQiLj7rmbzezrw2nyg+YArRbFlLV1oZivM7DEzG9HBekTkEJgZF0zN540Ne9heWhV2OdINdTQU/mJmz8WOAVwBPAssOshrWhsbqeXllE8Do939WOCvwEOtrsjsajMrNLPC4uLiDpYsIq35zNR83OHJZRr2Qg7U0QPN3wHuB44FjgPud/d/OcjLioDmf/kPBz72W+jue9y9Jjb7ADCtjfe/390L3L0gLy+vIyWLSBtGDejN8SP78r/LNOyFHKjDN9lx98fd/QZ3v97d/7cDL1kCjDezMWaWDswDFjZvYGZDm83OAd7taD0icug+c/xw3t9Zzupt+8MuRbqZdkPBzMrMbH8rjzIza/e3yd3rgWuB54h+2T/q7qvN7BYzmxNrdp2ZrTazd4DrgCu63iUROZjzJw8lLcV0wFkOYIm2+VhQUOCFhYVhlyGS8K7+bSHLtuzjjRtnk5qiO/MmOzNb6u4FB2un3wSRHuqzxw+nuKyGVzTshTSjUBDpoWZPjA578djbRWGXIt2IQkGkh0pPjTB3Sj4vrN5JaaWGvZAohYJID3bRtOHUNjSycIWuWZAohYJID3b0sD5MHJLDY4VbDt5YegSFgkgPZmZcXDCCd4pKWb1Nt+oUhYJIj/fZ4/PJSI3wx7c2h12KdAMKBZEerm+vdM6bPJQnl22jvKY+7HIkZAoFEeHyk0ZTXlPPAm0t9HgKBRHhuBF9mTm2Pw++spHa+sawy5EQKRREBIBrThvHjv3VPLtSp6f2ZAoFEQHg1PF5jB3Ym9++8WHYpUiIFAoiAkAkYnxu5iiWbd7Hqq06PbWnUiiISJOLpg0nKy2F32lrocdSKIhIk9ysNC6YOoyn3tmq8ZB6KIWCiHzMZTNHUV3XyJ+WauiLnkihICIfc/SwXKaN6sfv3/yQxsbEugmXdJ1CQUQO8IUTR7FpTyWvrNMNeHoahYKIHODcY4YwMDtdB5x7IIWCiBwgIzWFS6aP4G/v7aRob2XY5chhpFAQkVZdesIoAP6wWOMh9SQKBRFpVX7fLM48ajCPLNlCTX1D2OXIYRJoKJjZuWa21szWmdmN7bS7yMzczAqCrEdEOufzJ46ipKKWRSu3h12KHCaBhYKZpQB3AZ8EJgHzzWxSK+1ygOuAxUHVIiKH5uRxAzUeUg8T5JbCDGCdu29w91pgATC3lXa3Av8JVAdYi4gcgkjEuEzjIfUoQYZCPtD8ksii2LImZjYVGOHuzwRYh4h0wYUaD6lHCTIUrJVlTZdHmlkEuA341kFXZHa1mRWaWWFxcXEcSxSRg9F4SD1LkKFQBIxoNj8caH73jhzgGOAlM9sEzAQWtnaw2d3vd/cCdy/Iy8sLsGQRac3nZ47WeEg9RJChsAQYb2ZjzCwdmAcs/OhJdy9194HuPtrdRwNvAnPcvTDAmkTkEEwa1ocCjYfUIwQWCu5eD1wLPAe8Czzq7qvN7BYzmxPU+4pIMD6v8ZB6hNQgV+7ui4BFLZb9exttZwVZi4h0zf+Nh7SJ0yZoN26y0hXNItIhGakpzJ8xkhff28W6XeVhlyMBUSiISIddcdJoMlNTuPvv68IuRQKiUBCRDhuQncFlM0fy5PKtbNpdEXY5EgCFgoh0ypdPHUtaSoQ7tbWQlBQKItIpg3Iy+dwJo3ji7SI+2FkWdjkSZwoFEem0a2cfQe/0VH765/fCLkXiTKEgIp3Wv3c6Xzv9CF58bxdvrN8TdjkSRwoFETkkXzx5NMNyM/nxond1lXMSUSiIyCHJTEvh2+ccycqtpfz2jU1hlyNxolAQkUN2wZR8Zk8cxI8XvceWksqwy5E4UCiIyCGLRIwff2YyZvCL59eGXY7EgUJBRLpkSG4mV54yhieXb9Pd2ZKAQkFEuuya08bRt1catz6zRgedE5xCQUS6LDcrjZs+OZHFG0v41asbwy5HukChICJxcXHBCM6YOIjb/vo+2/ZVhV2OHCKFgojEhZlx85yjaXTn1mfWhF2OHCKFgojEzYj+vfj67PH8edUO/r52V9jlyCFQKIhIXH3pE2MYm9eb/3hqNdV1DWGXI52kUBCRuMpITeGHc49hc0kl339yFQ06GymhKBREJO5OOmIg180+gseWFnHbC++HXY50QmrYBYhIcrr+rAns2F/NXS+t4xPjB3LC2AFhlyQdoC0FEQmEmfEfnz6akf17ccOj71BaVRd2SdIBgYaCmZ1rZmvNbJ2Z3djK89eY2UozW25mr5rZpCDrEZHDq3dGKrddMoWd+6u56YkVuOv4QncXWCiYWQpwF/BJYBIwv5Uv/YfdfbK7TwH+E/ivoOoRkXAcP7If3znnSBat3MHvF28Ouxw5iCC3FGYA69x9g7vXAguAuc0buPv+ZrO9Af0ZIZKEvvyJscw6Mo9bn1nDiqJ9YZcj7QgyFPKBLc3mi2LLPsbM/tnM1hPdUrguwHpEJCSRiPGLfzqOgb3TufSBxXywsyzskqQNQYaCtbLsgC0Bd7/L3ccB/wJ8v9UVmV1tZoVmVlhcXBznMkXkcBiQncFjXz2JjNQIX//jMl3Y1k0FGQpFwIhm88OBbe20XwBc0NoT7n6/uxe4e0FeXl4cSxSRw2lY3yx+fvFxvLejjG8sWKZhtruhIENhCTDezMaYWTowD1jYvIGZjW82ex7wQYD1iEg3cPqRg/j+eUfx3Oqd/GHxh2GXIy0EFgruXg9cCzwHvAs86u6rzewWM5sTa3atma02s+XADcDlQdUjIt3HVaeM4dQJefzkz++xvrg87HKkGUu084YLCgq8sLAw7DJEpIu27avi03e8Sk5mKk987WT6904Pu6SkZmZL3b3gYO10RbOIhGJY3yzu/0IB20qrueb3S6mtbwy7JEGhICIhmjaqHz+76Fje2ljC959cqSueuwENiCcioZo7JZ/1u8q5/W/rOGJQNlefOi7skno0hYKIhO6bZ05gfXEFP170HimRCFedMibsknoshYKIhC4SMf7rkuOa7u9cXFbDd885kkiktWtgJUg6piAi3UJGagp3zJ/KZTNHcu8/1nPLM2vCLqlH0paCiHQbqSkRbp17DJmpKTz46kZGDejFF0/WrqTDSaEgIt2KmfGvnzqKLXsrueWZNQzv14uzJg0Ou6weQ7uPRKTbiUSM/75kKpPzc7n24bd5ZkV7w6ZJPCkURKRbykpP4TdfnMFRQ/tw7cPL+Plza3Udw2GgUBCRbqt/73T+dM2JzJs+gjv/vo5/eXyFhtwOmI4piEi3lpYS4SefncygnAxu/9s6lm3exx2XTmXikD5hl5aUtKUgIt2emXHD2Ufy2ytnsK+qjvNuf5XbXnifBt2PIe4UCiKSME6dkMei6z7Bp48dyi9f/IDz73iVv6/dpWMNcaRQEJGEkpeTwW2XTOHOS6dSVl3HF/9nCRfd+wavr9sddmlJQaEgIgnHzDj/2GH87Vuz+NFnjmHr3ioufXAxX//jMipq6sMuL6EpFEQkYaWnRvjcCaN46TuzuOGsCTy7Yhufvft11u4oC7u0hKVQEJGEl5mWwnVnjOehK2ewq6ya825/hf96fi019Tp9tbMUCiKSND4xPo8XvzWLOccN4/a/rWP2z//BbS+8T1l1XdilJQzdo1lEktLL7xdz59/XsWRTCUP6ZHL6xEFMG9mP844dSmZaStjlHXYdvUezQkFEktryLfv44TNrWLG1lNr6RrIzUjn76MGMy8tmX2UtfXulM3viIAb3yWTj7nIG98lkUE4m6anJtSNFoSAi0oy78/r6Pfzvsq389d2d7KusIyM1Qk194wFt+/dO57jhuZw4bgCnHzmImvpGVm4t5aihfZg0tM9hD4xt+6p4Y/0epozsy7i87ENah0JBRKQN7s7+qnr6ZKVSUlHLn1ftoGhvFUcNzWHX/hqWbdnLO1tK2bqv6oDXDsxO59IZIzl2eF9OGT+wy7ui6hoaWbRyO6VVdfTrlc5bG0volZFCXb2zZW8llbX1vLu9jJKKWm6dezSfP3H0Ib1PtwgFMzsX+CWQAjzo7j9t8fwNwJeAeqAYuNLdP2xvnQoFETkc3J3NJZX89d1d9EpPYfro/ry3Yz//89om3t68F3fISI0wfXR/RvTvxcj+vZgxph/FZbUU7a1k5tgBHDkkh7SU6FZFXUMjq7aWsqKolJ37q9lVVsOushrW7SxjW2l10/tmpEaob3TSUyKM6J9F74xUUsz49jlHctzwvmSlH1oIhR4KZpYCvA+cBRQBS4D57r6mWZvTgcXuXmlmXwVmufsl7a1XoSAiYSuvqWfph3t5bvUOXlizk4ZGp6Si9oB2g3IyGNY3i8y0CCuLSqmojZ4imxox8nIyyMvJYFBOJhdNy2fKiH5sK61i/KBsMlJTSEsxzOJ3j+ruEAonAje7+zmx+ZsA3P0nbbSfCtzp7ie3t16Fgoh0R8VlNby9eS99s9IYmJPBko0lvPxBMbvLa6moqWfaqH7MHDuAaaP6kZedQSQSvy/8juhoKAQ5dHY+sKXZfBFwQjvtrwL+HGA9IiKBycvJ4JyjhzTNj8vLZt6MkSFWdGiCDIXWYrDVzRIzuwwoAE5r4/mrgasBRo5MvB+yiEiiCPK8qiJgRLP54cABN1o1szOB7wFz3L2mtRW5+/3uXuDuBXl5eYEUKyIiwYbY0zinAAAJZUlEQVTCEmC8mY0xs3RgHrCweYPYcYT7iAbCrgBrERGRDggsFNy9HrgWeA54F3jU3Veb2S1mNifW7GdANvAnM1tuZgvbWJ2IiBwGgd6j2d0XAYtaLPv3ZtNnBvn+IiLSOck1uIeIiHSJQkFERJooFEREpEnCDYhnZsXAPqA0tii3lemBQFfu4t18nYfSprXnWi5rre62phOtPy3nP5puvizMPnWkPy2X6TPqnI70p7128epP8+lE+4zi3Z9R7n7wc/rdPeEewP3tTQOF8Vr/obRp7bmWyw7Wh0TuTzv9aL4stD51pD/6jLrWp470p6ufUWf/TyXaZxR0f9p6JOruo6c7MB2v9R9Km9aea7msI31I1P60nH+6jTZd0ZU+daQ/LZfpM+qcjq6nK5/Rofyf6opk+15oVcLtPuoIMyv0Dgz8lCiSrT+QfH1Ktv5A8vVJ/emYRN1SOJj7wy4gzpKtP5B8fUq2/kDy9Un96YCk3FIQEZFDk6xbCiIicggUCiIi0kShICIiTXpcKJjZLDN7xczuNbNZYdcTD2bW28yWmtn5YdcSD2Z2VOzzeSx27+6EZmYXmNkDZvaUmZ0ddj1dZWZjzexXZvZY2LV0Rez/zUOxz+ZzYdfTVfH6XBIqFMzs12a2y8xWtVh+rpmtNbN1ZnbjQVbjQDmQSfRGQKGJU38A/gV4NJgqOycefXL3d939GuBionfkC02c+vOku38ZuAK4JMByDypO/dng7lcFW+mh6WT/Pgs8Fvts5hywsm6gM/2J2+cSxBVxQT2AU4HjgVXNlqUA64GxQDrwDjAJmAw80+IxCIjEXjcY+EMS9OdMojcwugI4Pxk+o9hr5gCvA5cmQ39ir/sFcHwS9eexsH/futi/m4ApsTYPh117V/sTr88l0PspxJu7v2xmo1ssngGsc/cNAGa2AJjr7j8B2tudshfICKLOjopHf8zsdKA30V/yKjNb5O6NgRbejnh9Ru6+EFhoZs8CDwdXcfvi9BkZ8FPgz+7+drAVty/O/4e6nc70j+ieguHAcrrpXpNO9mdNPN6zW/4gOikf2NJsvii2rFVm9lkzuw/4HXBnwLUdik71x92/5+7fJPrF+UCYgdCOzn5Gs8zs9tjntKitdiHqVH+ArxPdorvIzK4JsrBD1NnPZ4CZ3QtMNbObgi4uDtrq3xPAhWZ2DwEPHRFnrfYnXp9LQm0ptMFaWdbmFXnu/gTRX4buqlP9aWrg/pv4lxI3nf2MXgJeCqqYOOhsf24Hbg+unC7rbH/2AN0x3NrSav/cvQL44uEuJg7a6k9cPpdk2FIoAkY0mx8ObAuplnhItv5A8vVJ/Uksyda/QPuTDKGwBBhvZmPMLJ3oQdeFIdfUFcnWH0i+Pqk/iSXZ+hdsf8I+ut7JI/F/BLYDdUTT8qrY8k8B7xM9Iv+9sOvsqf1Jxj6pP4n1SLb+hdEfDYgnIiJNkmH3kYiIxIlCQUREmigURESkiUJBRESaKBRERKSJQkFERJooFCRwZlZ+GN5jTgeHGY/ne84ys5MO4XVTzezB2PQVZtYtxuAys9Eth2hupU2emf3lcNUkh59CQRKGmaW09Zy7L3T3nwbwnu2NDzYL6HQoAP8K3HFIBYXM3YuB7WZ2cti1SDAUCnJYmdl3zGyJma0wsx80W/6kRe8et9rMrm62vNzMbjGzxcCJZrbJzH5gZm+b2Uozmxhr1/QXt5n9JjbK6utmtsHMLootj5jZ3bH3eMbMFn30XIsaXzKzH5vZP4BvmNmnzWyxmS0zs7+a2eDYcMbXANeb2XIz+0Tsr+jHY/1b0toXp5nlAMe6+zutPDfKzF6M/WxeNLORseXjzOzN2DpvaW3Ly6J3EXvWzN4xs1Vmdkls+fTYz+EdM3vLzHJiWwSvxH6Gb7e2tWNmKWb2s2af1VeaPf0kkPB3KpM2hH0Ztx7J/wDKY/+eDdxPdJTHCNGbtpwae65/7N8sYBUwIDbvwMXN1rUJ+Hps+mvAg7HpK4A7Y9O/Af4Ue49JRMeeB7iI6FDcEWAI0XtqXNRKvS8Bdzeb7wdNV/9/CfhFbPpm4NvN2j0MnBKbHgm828q6TwcebzbfvO6ngctj01cCT8amnwHmx6av+ejn2WK9FxIdOv2j+VyiN2DZAEyPLetDdGTkXkBmbNl4oDA2PZrYzVyAq4Hvx6YzgEJgTGw+H1gZ9u+VHsE8kmHobEkcZ8cey2Lz2US/lF4GrjOzz8SWj4gt3wM0AI+3WM9HQ58vJXpLxdY86dF7S6wxs8GxZacAf4ot32Fmf2+n1keaTQ8HHjGzoUS/aDe28ZozgUlmTSMb9zGzHHcva9ZmKFDcxutPbNaf3wH/2Wz5BbHph4Gft/LalcDPzez/Ac+4+ytmNhnY7u5LANx9P0S3KoA7zWwK0Z/vhFbWdzZwbLMtqVyin8lGYBcwrI0+SIJTKMjhZMBP3P2+jy00m0X0C/VEd680s5eI3kMboNrdG1qspyb2bwNt/w7XNJu2Fv92REWz6TuA/3L3hbFab27jNRGifahqZ71V/F/fDqbDA5O5+/tmNo3oQGk/MbPnie7maW0d1wM7geNiNVe30saIbpE918pzmUT7IUlIxxTkcHoOuNLMsgHMLN/MBhH9K3RvLBAmAjMDev9Xid5pKxLbepjVwdflAltj05c3W14G5DSbfx649qOZ2F/iLb0LHNHG+7xOdBhkiO6zfzU2/SbR3UM0e/5jzGwYUOnuvye6JXE88B4wzMymx9rkxA6c5xLdgmgEPk/0nr8tPQd81czSYq+dENvCgOiWRbtnKUniUijIYePuzxPd/fGGma0EHiP6pfoXINXMVgC3Ev0SDMLjRIcfXgXcBywGSjvwupuBP5nZK8DuZsufBj7z0YFm4DqgIHZgdg2t3AXL3d8DcmMHnFu6Dvhi7OfweeAbseXfBG4ws7eI7n5qrebJwFtmthz4HvBDd68FLgHuMLN3gBeI/pV/N3C5mb1J9Au+opX1PUj0nr9vx05TvY//2yo7HXi2lddIEtDQ2dKjmFm2u5eb2QDgLeBkd99xmGu4Hihz9wc72L4XUOXubmbziB50nhtoke3X8zIw1933hlWDBEfHFKSnecbM+hI9YHzr4Q6EmHuAf+pE+2lEDwwbsI/omUmhMLM8osdXFAhJSlsKIiLSRMcURESkiUJBRESaKBRERKSJQkFERJooFEREpIlCQUREmvx/y8AYLSFkuFgAAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "learning_rate_finder = learn.lr_find()\n", "learn.sched.plot()" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Note that the learning rate is decreasing fastest at around 0.2\n", "learning_rate = 0.2" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2beac4b5ac8c4f17b50484ebc403462d", "version_major": 2, "version_minor": 0 }, "text/html": [ "

Failed to display Jupyter Widget of type HBox.

\n", "

\n", " If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n", " that the widgets JavaScript is still loading. If this message persists, it\n", " likely means that the widgets JavaScript library is either not installed or\n", " not enabled. See the Jupyter\n", " Widgets Documentation for setup instructions.\n", "

\n", "

\n", " If you're reading this message in another frontend (for example, a static\n", " rendering on GitHub or NBViewer),\n", " it may mean that your frontend doesn't currently support widgets.\n", "

\n" ], "text/plain": [ "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "epoch trn_loss val_loss f2 \n", " 0 0.14736 0.13469 0.879414 \n", " 1 0.142267 0.129391 0.885923 \n", " 2 0.138263 0.127576 0.887258 \n", " 3 0.140281 0.126938 0.888762 \n", " 4 0.137914 0.124001 0.890997 \n", " 5 0.131529 0.123408 0.893647 \n", " 6 0.130181 0.12329 0.891949 \n", "\n" ] }, { "data": { "text/plain": [ "[0.12329047, 0.8919489753927757]" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Do some training for the final layers\n", "# We now know that when we train, what we are doing is setting the kernel (image feature filter) values\n", "# and the numbers of the weight matrix in the fully connected layer\n", "learn.fit(learning_rate, 3, cycle_len=1, cycle_mult=2)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# The satellite images are quite different from the image-net images that this network was trained on\n", "# Therefore, we use slightly higher learning rates than our cats/dogs examples to train the network\n", "differential_learning_rates = np.array([learning_rate/9, learning_rate/3, learning_rate])" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2da22cfecd9144f2880ebf591ef70933", "version_major": 2, "version_minor": 0 }, "text/html": [ "

Failed to display Jupyter Widget of type HBox.

\n", "

\n", " If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n", " that the widgets JavaScript is still loading. If this message persists, it\n", " likely means that the widgets JavaScript library is either not installed or\n", " not enabled. See the Jupyter\n", " Widgets Documentation for setup instructions.\n", "

\n", "

\n", " If you're reading this message in another frontend (for example, a static\n", " rendering on GitHub or NBViewer),\n", " it may mean that your frontend doesn't currently support widgets.\n", "

\n" ], "text/plain": [ "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "epoch trn_loss val_loss f2 \n", " 0 0.120932 0.108382 0.909156 \n", " 1 0.115432 0.105932 0.909191 \n", " 2 0.105167 0.10103 0.915758 \n", " 3 0.111772 0.102488 0.91311 \n", " 4 0.105998 0.100651 0.915267 \n", " 5 0.101719 0.09802 0.916691 \n", " 6 0.098998 0.097133 0.917914 \n", "\n" ] }, { "data": { "text/plain": [ "[0.09713308, 0.91791370766977]" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Unfreeze the previous layers so we can train them with our differential learning rates\n", "learn.unfreeze()\n", "learn.fit(differential_learning_rates, 3, cycle_len=1, cycle_mult=2)" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Save the weights\n", "learn.save(f'{image_size}')" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl8VNX9//HXJzt7WGU3YVEBRdAACoIbstlKbbWC+qu2Wtqqba21/QZXSqti/bZqW+vWr9VqEZeiYgER11pFJcgiOwGRfRMhQMh+fn/MncnMZJIMkGTCzPv5eOTBveeee+cz1/EzZ8699xxzziEiIokhKdYBiIhIw1HSFxFJIEr6IiIJRElfRCSBKOmLiCQQJX0RkQSipC8ikkCU9EVEEoiSvohIAkmJdQDh2rVr57KysmIdhojIcWXRokV7nHPta6vX6JJ+VlYWeXl5sQ5DROS4YmZfRlNP3TsiIglESV9EJIEo6YuIJBAlfRGRBKKkLyKSQJT0RUQSiJK+iEgCiaukv3DjXtbsOBDrMEREGq1G93DWsbj8sQUAbJx2cYwjERFpnOKqpS8iIjWLm6RfUeFiHYKISKMXN0m/tKIi1iGIiDR6cZP0S8oqk35RaXkMIxERabziJumXlrugZbX6RUQiiZukn5JsgeXgVr+IiFSKm6TfMiM1sFyilr6ISERRJX0zG2Nma8ws38xyI2y/xcxWmtkyM3vbzE70ygeY2QIzW+Ftu6Ku30AkaumLiERWa9I3s2TgEWAs0BeYaGZ9w6otBnKcc/2Bl4Hfe+WFwPecc/2AMcBDZpZZV8FXp1hJX0Qkomha+oOBfOfcBudcCTADGB9cwTn3rnOu0Fv9GOjqla91zq3zlrcBu4Ba53A8Wg9dMQBQS19EpDrRJP0uwOag9S1eWXWuA+aGF5rZYCANWB9h2yQzyzOzvN27d0cRUmSZTX39+mrpi4hEFk3StwhlER9/NbOrgRzggbDyTsCzwPedc1UysnPuCedcjnMup337o/8hkJbieztq6YuIRBbNgGtbgG5B612BbeGVzGwkcDtwrnOuOKi8JTAbuMM59/GxhVuzdH/S1907IiIRRdPSXwj0NrNsM0sDJgCzgiuY2UDgceAS59yuoPI04BXgH865l+ou7MjSU5IBKNYTuSIiEdWa9J1zZcBNwDxgFfCic26FmU01s0u8ag8AzYGXzGyJmfm/FL4LjACu9cqXmNmAun8bPv6Wvvr0RUQii2o8fefcHGBOWNldQcsjq9nvOeC5YwnwSGSkei19JX0RkYji5olcqGzpa8A1EZHI4izpq6UvIlKT+Er6qf4+fbX0RUQiia+kH+jeUUtfRCSSuEr6ZkZaSpJa+iIi1YirpA+QkZJEsVr6IiIRxV3ST09NVktfRKQa8Zf01dIXEalW3CX9jNRk3bIpIlKNuEv66SlJejhLRKQacZn01dIXEYks7pJ+hi7kiohUK+6Svq97Ry19EZFI4jDpq6UvIlKd+Ev6qerTFxGpTtwl/YyUZN29IyJSjbhL+mrpi4hUL/6Svp7IFRGpVtwlff8tm865WIciItLoxF3SLymvoML5/hURkVBRJX0zG2Nma8ws38xyI2y/xcxWmtkyM3vbzE4M2naNma3z/q6py+Aj2V9YCsBnX+6r75cSETnu1Jr0zSwZeAQYC/QFJppZ37Bqi4Ec51x/4GXg996+bYC7gSHAYOBuM2tdd+FXdXlOV0CTo4uIRBJNS38wkO+c2+CcKwFmAOODKzjn3nXOFXqrHwNdveXRwHzn3F7n3NfAfGBM3YQeWcuMVAAOFpfV58uIiByXokn6XYDNQetbvLLqXAfMPcp9j1mz9BQACkuU9EVEwqVEUccilEW8NcbMrgZygHOPZF8zmwRMAujevXsUIVWvWZrvLR0sVveOiEi4aFr6W4BuQetdgW3hlcxsJHA7cIlzrvhI9nXOPeGcy3HO5bRv3z7a2CNqlp6MGewvLDmm44iIxKNokv5CoLeZZZtZGjABmBVcwcwGAo/jS/i7gjbNA0aZWWvvAu4or6zepCQn0SWzCV/uLay9sohIgqm1e8c5V2ZmN+FL1snAU865FWY2Fchzzs0CHgCaAy+ZGcAm59wlzrm9ZvZbfF8cAFOdc3vr5Z0EaZmRysEi9emLiISLpk8f59wcYE5Y2V1ByyNr2Pcp4KmjDfBoNE1LprBEffoiIuHi7olcgCZpyRzWffoiIlXEZ9JPTeawWvoiIlXEZdJvqpa+iEhEcZn0m6hPX0QkovhM+qkpHNYTuSIiVcRl0vd372hMfRGRUHGZ9JukJVPh0LSJIiJh4jPppyYDGl5ZRCRcfCb9NF/S18VcEZFQcZn0m3pJX7dtioiEisuk7+/e0QNaIiKh4jPpq6UvIhJRXCb9purTFxGJKC6TfpNU3+ChekBLRCRUfCZ9de+IiEQUl0lf3TsiIpHFZdLP0N07IiIRxWXSD9ynr6QvIhIiLpN+anISqcmmPn0RkTBxmfTB18WjPn0RkVBxm/Sbp6dwsFi3bIqIBIsq6ZvZGDNbY2b5ZpYbYfsIM/vMzMrM7LKwbb83sxVmtsrM/mRmVlfB16Rt8zS+OljcEC8lInLcqDXpm1ky8AgwFugLTDSzvmHVNgHXAtPD9h0KDAP6A6cCg4BzjznqKKzfdYh31+ymokITqYiI+EXT0h8M5DvnNjjnSoAZwPjgCs65jc65ZUD4rCUOyADSgHQgFdh5zFFHwX8Rd9/h0oZ4ORGR40I0Sb8LsDlofYtXVivn3ALgXWC79zfPObfqSIM8GjkntgagUEMxiIgERJP0I/XBR9VnYma9gD5AV3xfFBeY2YgI9SaZWZ6Z5e3evTuaQ9fq2mFZABwq1h08IiJ+0ST9LUC3oPWuwLYoj38p8LFz7qBz7iAwFzgrvJJz7gnnXI5zLqd9+/ZRHrpmzdJ9g67pDh4RkUrRJP2FQG8zyzazNGACMCvK428CzjWzFDNLxXcRt0G6d5ql+ZK+undERCrVmvSdc2XATcA8fAn7RefcCjObamaXAJjZIDPbAlwOPG5mK7zdXwbWA58DS4GlzrnX6+F9VNEiw5f0DxQp6YuI+KVEU8k5NweYE1Z2V9DyQnzdPuH7lQM/OsYYj0qrJqkAfLBuD+NO6xSLEEREGp24fSLXn/Sf/3RTjCMREWk84jbp+y/kDs5uE+NIREQaj7hN+gCDslqT1CCDPoiIHB/iOum3bprGvkI9kSsi4hfXSb9NszT2HiqJdRgiIo1GXCf9zKZpfF1YgnMadE1EBOI86bdtlkZpueNrdfGIiABxnvSz2zUDYP3ugzGORESkcYjrpO9/Kvd3sxtk5AcRkUYvrpP+oCzfPfp9O7WIcSQiIo1DXCf9pCSjU6sMysp1IVdEBOI86YPvts0dBUWxDkNEpFGI+6TfJbMJuwo0QbqICCRA0m/t3asvIiIJkPQzm6Wyr7BUD2iJiJAASb910zRKyisoLNFcuSIiCZD0fePqq4tHRCQBkn5m0zQAjbYpIkICJP3WXtJXS19EJCGSvq97R0Msi4gkQtJv5mvpT/9Ec+WKiESV9M1sjJmtMbN8M8uNsH2EmX1mZmVmdlnYtu5m9qaZrTKzlWaWVTehR6etl/QrdMumiEjtSd/MkoFHgLFAX2CimfUNq7YJuBaYHuEQ/wAecM71AQYDu44l4CNlZvRo34wOLTMa8mVFRBqllCjqDAbynXMbAMxsBjAeWOmv4Jzb6G2rCN7R+3JIcc7N9+rFZGD7zCap7NfdOyIiUXXvdAE2B61v8cqicRKwz8xmmtliM3vA++XQoDKbprHvsC7kiohEk/QtQlm0HeQpwHDgVmAQ0ANfN1DoC5hNMrM8M8vbvXt3lIeOXmaTVN2nLyJCdEl/C9AtaL0rsC3K428BFjvnNjjnyoBXgTPCKznnnnDO5Tjnctq3bx/loaPXqqm6d0REILqkvxDobWbZZpYGTABmRXn8hUBrM/Nn8gsIuhbQUDKbpHGguIzS8oraK4uIxLFak77XQr8JmAesAl50zq0ws6lmdgmAmQ0ysy3A5cDjZrbC27ccX9fO22b2Ob6uoifr561UL9N7QKvgsFr7IpLYorl7B+fcHGBOWNldQcsL8XX7RNp3PtD/GGI8Zv6kv+9wKW2bp8cyFBGRmIr7J3IBWjXxkr7G3xGRBJdQSb+gqCzGkYiIxFZCJP2WXtLXHTwikugSI+ln+JL+zS8siXEkIiKxlRhJv0lU16tFROJeQmTD9JRkurVpwmHNkysiCS4hWvoAY/p15EBRGU5DLItIAkuYpN+xVROKyyrYrwe0RCSBJUzS79TKN57+3OU7YhyJiEjsJEzS792hOQCTZ35O9uTZMY5GRCQ2EibpZ7VrFlh2Tvfsi0hiSpikn5oc+lb/MH9NjCIREYmdhEn6AEOy2wSWn/90UwwjERGJjYRK+tN/eBb594wF4MwTW8c4GhGRhpdQST85yUhJTuKcXu0oKdOEKiKSeBIq6fu1bZ7GnoMaZllEEk9CJv3OmU3Yuu8wxWUalkFEEktCJv3sds0or3CcfMcb6uYRkYSSkEm/c6smgeVnPtoYu0BERBpYQib9Lq0rk/7ry7bFMBIRkYaVkEk/q21TTmzbFICOLTNiHI2ISMOJKumb2RgzW2Nm+WaWG2H7CDP7zMzKzOyyCNtbmtlWM/tLXQR9rMyM9249DzN4c+VODbcsIgmj1qRvZsnAI8BYoC8w0cz6hlXbBFwLTK/mML8F3j/6MOuemdGnY0sA/vXZ1hhHIyLSMKJp6Q8G8p1zG5xzJcAMYHxwBefcRufcMqDKrTBmdiZwAvBmHcRbp2beMBSAxZu+jnEkIiINI5qk3wXYHLS+xSurlZklAX8AflVLvUlmlmdmebt3747m0HUiIzWZUzq2YGdBcYO9pohILEWT9C1CWbSd4DcAc5xzm2uq5Jx7wjmX45zLad++fZSHrhs7C4p4a9XOBn1NEZFYiWZi9C1At6D1rkC09zmeDQw3sxuA5kCamR10zlW5GBwrX3vj6s9bsYPR/TrGOBoRkfoVTUt/IdDbzLLNLA2YAMyK5uDOuaucc92dc1nArcA/GlPCB/jzxIEATJm1IsaRiIjUv1qTvnOuDLgJmAesAl50zq0ws6lmdgmAmQ0ysy3A5cDjZnbcZNCLT+sEwPb9RZSWa0gGEYlv0XTv4JybA8wJK7sraHkhvm6fmo7xNPD0EUdYz5KSKi9ZvL9mNyP7nhDDaERE6ldCPpEb7sPcCwDYsOdgjCMREalfSvpAl0zfWDz3zlkd40hEROqXkr6ISAJR0vf8+NyepCabxuERkbimpO85oWU6peWOvYc0jaKIxC8lfU9nr19/wYavYhyJiEj9UdL39GzfHICbpi+OcSQiIvVHSd/Tq0PzwHJW7mz17YtIXFLSD/LD4dmB5ezJc5i3YkcMoxERqXtK+kEmj+0Tsv6jZxcx/ZNNMYpGRKTuKekHSUoy/nLlwJCy2175nKzc2WTlzubXLy+NUWQiInVDST+M/y6eSF7M28KBotIGjEZEpG4p6YcZ0DWTyWNPCQzNEG7T3sIGjkhEpO5YY7tLJScnx+Xl5cU6DPYXlrJgwx5WbCvgP+v28LMLenHdM764Vv92DBmpyTGOUESkkpktcs7l1FpPST86zjmyJ1eOLj3lm335w5tr+c34fnz7jBpHlRYRqXfRJn1170TJzPjlRScF1qe8vpIDxWXc8uJSdhYUxTAyEZHoKekfgZ9e2Jt2zdOqlK/cXgBARYVjf6Eu9IpI46Wkf4QuHdglsPzN0zsDcP0zeTz01lp63DaH06e+yXMff0lRaXmsQhQRqZaS/hHq3qYpAJed2TUwqXp5heOht9YF6tzx6nLGPvzBMb/W/JU7uf6ZPP789joKS8oYcu9bjH/kQ17K23zMxxaRxBTVHLlSaeLg7pSWO646q3uN9b7Yc6jabXsPlWBA62ZVu4r8/vpePr9/Yw0Ab63ayVurdrKzoJidBcUs3byPX728jM6tMri4fyduG9cHM6v2WCIiflG19M1sjJmtMbN8M8uNsH2EmX1mZmVmdllQ+QAzW2BmK8xsmZldUZfBx0JKchI/OCeb9BTfLZtv3XJuYNvZPdry2NVnBNZ73jaHNTsOVDnGGb+dz8Dfzq/2NYrLygMJ32/plv1V6m3bX8STH3zBPbNXVdm2r7BEg8aJSBW1Jn0zSwYeAcYCfYGJZtY3rNom4Fpgelh5IfA951w/YAzwkJllHmvQjUnw6Jy/HHUSY07txB8uPx3wdfsEd8U457hv7qqQ9Uie/nDjEcWwZmfoF8umrwoZMHU+2ZPnKPGLSIhoWvqDgXzn3AbnXAkwAxgfXME5t9E5twyoCCtf65xb5y1vA3YB7esk8kZk/b3jeOYHgznzxNYAZLVrFthWWl55St5atYvH398QWN9z0DdL18ptBVz61w/ZuOcQC9Z/hb+n5tZRJ7Fx2sWMH+C7YDygWyZf3DeOK3K68djVZ7Bx2sWc3aMth4rLKCotp6LCl+BHPPBu4DWGTXtHiV9EAqLp0+8CBF853AIMOdIXMrPBQBqw/kj3beySk4xzT6r8LssOSvqvL9vOxq8KeX/t7ir7vbVqJyNOas+4P/ku+p73v++FbL/hvF4APHDZ6XRokc5PzuuFmXH/Zf0DdXYUFPHFnkOccucbANz1jdAfYdv2F5E9eQ7/+snZnHlim2N7oyJy3IumpR/pCuERNR3NrBPwLPB951xFhO2TzCzPzPJ2766aHI83rZumkpbiO7V7D5VUSfjXDs0CYPLMzxk27Z1qj5OU5Dv1aSlJ3H5xX9pEuPAbfsF46r9XRjzWdx5dQEWF4/43VrNt32EAnl2wkVcWb4nqPYlIfIgm6W8BugWtdwW2RfsCZtYSmA3c4Zz7OFId59wTzrkc51xO+/bHf++PmfHGz4fTtXXVQdvSkpOYckm/iPt9PmUUAM3TUwLLtXn8/50ZsXzsqR1Z+7uxIWXn/u+7PPreeoZ6XT53vraCX7ywlI/W74nqtUTk+BdN0l8I9DazbDNLAyYAs6I5uFf/FeAfzrmXjj7M40+P9s1DEvKjV53BySe0YMXU0QBc4j3Y5Tckuw0tMlLJv2csy+4eRYuM1KheZ3S/jvzxu74Lxw9PGMDEwd0Ykt2GP00cSFpKEvdcemqg7ua9hwPLv3p5WWD5yic/Ye3OAyze9PWRv1EROa5ENeCamY0DHgKSgaecc/eY2VQgzzk3y8wG4UvurYEiYIdzrp+ZXQ38HVgRdLhrnXNLqnutxjrg2tHauOcQbZunVUnizjnKKxyb9hYyb8VOxg/oXONY/sdi+db9fOPP/42q7sZpFweW/7N2N6nJSZzds229xCUidUejbErAoeIy+t09jx8My2ZA90x+9vziwLabR/YOeZp41dQxJCXB+2t2M+nZRUDoF4EcvdLyCqa+vpJJI3rQzXuyW6SuRJv09URuAmiWnsKyKaNonpZCUpIxsFsmw3/vu63zm6d3Dkn6ryzeym2vfB6y/1srd7Jm5wFuOK9nXD/5u6+whMym1T8lfaxWbS/g2Y+/ZPWOAl768dB6ex2RmijpJ4iWQd1L3do0DWm9z/35cJ7+cCMv5G3m/jdWV9n3+n/4fnmVlTt+PrJ3/QcbA/9eto2bpi9m2rdPY8LgmofYOFoFh8sA3222IrGiAdeEPp1akjv2FAD2H64cGnrOz4ZzerfKB6gffGtt4HZPv4oKR9+73iArdzaHissaJuB6sHbnQQByZ37O1n2HKSuvCDkXuf9aRlbubLJyZ0d1vIKiUrJyZ3Pnq8t5cP5asnJnc/X/fQL4Lqj/P29ZpKGppS8AZDat/CWQlpIUuN1z4qBuLN28L7Bt6LR3+ON3T+cb/TuTlpLEHa8tp7DEN4z0i3mb+f6wbM574F02flVI304tue6cbL5zZuOfWewv71R2cQU/O7HojpGc+bu3Ququ2l5An04tqz3W5r2Fge6zZz/+MmKdD9bt4cP8PQzt2Tauu8yk8VFLXwDfswUfT76QrLZN+e//nB8o/25ON1768dm8/OOzA2W3vLiUk+6Yi3OOect3BMpnLd3G5r2FbPzKN3n8yu0F/PKlpSxY/1XDvZGjUFZeQUU19zOEJ3yAsQ9/wMEaftX84oVqb04jxxuqA+Cqv33CzTXUFakPSvoS0LFVBu/96nw6tMgIlCUlGYOy2tC/a9Vx8n43exVfHfKNH5SeksTiTfuYPPPzKvX+/M46snJn8+kXe+sv+KM0+sH/0Ov2uQCMO61jtfWuPyebv39/UGD9W498GLFe/q4D5H1Z+bzDyD4dAsvr7x3Hyz8Zynu3nhcoe23JNj7Z0Li/FCW+6JZNidoby3cwbe6qQEu+JkvvGkXzjBR63jYnpDz89s+i0nIyUpPrNM5oFZeVc/IdbwTWrz8nm/2HS9lzsJhfjT4lMCbSazcOC1zbKCotD4xzlH/PWFKSK9tNew+VcEbQkNkv//hscrIij3d0x6uf89zHmwLrG+4dFxh242iUV/ie+/AP/wG+W0RLyipolq5e3ESgWzalzo05tSNjTu3I9v2HOfu+yn7vWTcN44N1e3hgXuUcAC2bpGBmfH9YFn8PGir60r9+yE3n9+LR99aT3a4ZLy3aQrO0ZD6fMrrGpOd/wOxbAzrz0ISBRxW/c44f/iOP687pwdk92zLj08pxBKeO78dVQ04kOSiGSM8nZKQm0yIjhQNFZfS6fS6pyUbLjFQWTL4wJOED1SZ8gCnf7Mek4T0DI6LOWb6db/TvXG39muRt3Mtljy2gWVoyK6aOCZTf9dpynv90s56zkBDq3pEj1qlV5ZPDI/t0oH/XTG48v1egbGD3zMDFyQmDQm9/XLxpH9c9k0fel1/z0iLfYG+HSsrpcdsccv+1rEr//+xl28nKnR14ovjVJdsCQ0gfqezJc3hr1S4mPukbAmrmZ77XXzZlFN87Oysk4ddk4e0jA8ul5Y6vDpVw0h1zA2Vv//Jc1t0zNtKuASnJSXRv25Tnf3gWADdNX8zJd8wNuWMo2CPv5rNyW0HEbZc9tgDwncfyCodzjtU7Cnje+1L76mBxVO9LEoOSvhyVhycMYEh2G578XuWvyek/HMJvx/djxqSzAmUnd2zBunvGcuc3wufdqWrGws1MfPJjbn1paWD00Bunf1al3uoIs5HVpjzsi+Ke2SsDs5G1jHKcI7+auqNuHXUSPds3JzU5uv+1ep9QOQlPcVkF59z/DjfPWBwyB8IzH23kgXlrAt1N+bsOsnpHAe+t2QWEDuW9aW8hry/bzpiHKudofnmRRlKVSurTlwb15oodbPzqEP27ZtKnY0tSko0P1u3mvrmr+TLsWsELk87iiicqB2ad87PhgcT3lysH0qdTS7q3aVprgr1n9koyUpP58zv5pCYbpeWVn/lhvdryz+vPqmHvmj04fy0Hisp46sMvgKPrm3/24y+589XlIWX9u7ZiwqDufLBuN3OD7pD64r5xZE+uvE6SO/YU7n9jNS3SUygoKuNnF/Ri/+FSnlkQeqvoF/eNC/z6OlhcRpJB0zT17sYTjb0jxxXnHJOeXcT8lTsDZV1bN2HL14e5qO8J/M+Yk+nZvnlIwvN765ZzQ6atDPaftbv53lOfBtbP6dWO/+ZXDiUdnAyPRUFRKXsPloTMmnYktnxdyDn3v1trvQ9zL4g4B8OjV53Bna8tD8zGFu7p7w/ivJM78PKiLdz60lLaNU8j746LjipWaZyiTfrq3pFGwcx48ns5TL9+CFltfYORbfna9/Rv304t6dWhBWbGxAhDJIz84/vVHved1btC1h+eMCCwPKZfxzp7MKplRupRJ3yArq2bsnLqaH52Qa+I26/I8U1pcW3QF1iwAd0zKQvrwnr0qjMCXW2zlmzjYHEZt760FPBN1fnIu/mBuhUVjqzc2Zx179tH/R7k+KCkL43K0F7teO9X54eUXT88O7B837dPCxkawi8rdzZD7n0rpC/8QFEpT3+0kQ4t0rnx/J6sv3ccbZunM2lED3q2b8Zj1UxAEytN01K4ZdTJIQ/CATxwWX9u/0YfANbt8g0XMfOGobRr7hscrl/nlnRq1YRHrwp9P2NP68RZPdoyvHc7Zi7eyql3zws97rw1FJb4HjLbW+j7hbCjoIg3lm+v+zcnjYaSvjRK9156GgB/mjiwylwEPxiWBYQ++ASws6CY657xdQ0eKCrltClvAr5hJX41+pTA3Tm3jevD2788rx6jPzandW0VWH7j5uFcntOtysXmM7q35pPbRrJx2sXM/tlwAM7u2ZZ3bz2P8QM6M+umYYG64RexH7qi8tdO37vmMfKP75MT9OTxj5+revHcr7S8gqPpEv7qYDEFRZHvTJKGpT59Oe4cLC7j5hlLyB17Mv9etj1kaOhIPp8S/UxkjYV/YLfP7rwoMDfyjE83kTvz8xqvYUQyeeaywO2bVw7pzr2XnkZBUSn9vS/F6vzz+iEM69UusF5aXkFv7+nlpmnJrPjN6Ki7x/re9QYpScayKaOjjluOjPr0JW41T0/hb9fk0KtDC35+YW/+/dNzAr8Mwj129ZnHXcIPltmkMvYJg7uzcdrFR5TwAe7+ZuWczP7z1DIjlbk/Hx5S73tnn8i0b1eex6v+5hsJtLzCMfbhDxh0T+WvgcKScm74p+8XwUfr9/DJhq9YE+FW2kPFZWTlzqawpJyCojLum7Mq0KUksaGWvsSN15Zs5eczKgcwm/7DIQzt2a6GPRqv3QeK+WLPIQZnV/9U75H4+4df0LV1Uy7qe0JIuX9YiWuHZjHlkn4s3LiXy72Hvfxm3jCUb//1o4jHvWpId/75SeVwEh/8+vzArGCRjgVwQst0PrltZEjZ3kMlJCcZrZocv1/QsaZbNiUhvb1qJ88s+JKvDhbz6o3Don5ISiot2byPRV9+zW//vTLi9o3TLmb0g/9hzc7ID8n5h3147P31TJtbdVIeCB23qP+UeRQU+Vr/4V1KfhUVjpmLt5JzYutjuksqnmnsHUlIF/Y5gQv7nFB7RanWgG6ZDOiWSbO0ZHLDRk3t1sY3BMffrskJzBkAcMnpnZm1dBvgm1fhuzndWOENG/HTC3rxrYHwzU1ZAAALnUlEQVRdyGrbLDAA33ce/YgXfnQ2ry7eGkj4AP/6bAv3zlkV2NcvyQgZ/vqG83ry6zGnhNR5ZfEWUpKS+ObpRzeGUV3auu8wbZulxWwwwZqopS8i1Rpy71vsLPCN3TPv5hF0ad2E5t6onf6Lzf4Hvx6cv5aH3/ZdVP/ivnGcdd/bDMluy58mVg6QN2/FDn707KI6i2/p3aMCXUL+ePzdeh/m72HPwWJ+PmMJJ5/Qgnm/GFFnrxvu3dW7mLl4K9cOzaJDi/TAF+KrNw7j8sc+4p1fnhfo9qovdXoh18zGmNkaM8s3s9wI20eY2WdmVmZml4Vtu8bM1nl/10T/FkQk1j65bSTv3noe6+8dx8kdWwQSPsCDV5zOyD4ncN7Jvltnr/Oep+jYMoNpc1ezs6CYfp1DZxg7q0fbal7nwlpjuX1cnyplp//GdwdScOP10ffWA74L0f5rPNV1RdWV7z+9kNeXbuM7j34U8gvoW498SGm5CymLtVqTvpklA48AY4G+wEQzCx89axNwLTA9bN82wN3AEGAwcLeZtUZEjhvZ7ZpFHIH00oFd+ds1lQ3LlhmpTBzcjR0FRTz+nw0AVaaVbNUklek/HMLF/TsFyh6eMIATWmZw/sntA2WfTxnFxmkXB64PpCQZPxzRg6V3jeKRK8+oEkvwxeRPNuyN+CzB3kORh6g4Vtv3H669ElQ7gmpDi6ZPfzCQ75zbAGBmM4DxQOAqj3Nuo7etImzf0cB859xeb/t8YAzw/DFHLiKNzqldWgGV8xSMOKl9lTpDe7ZjaM92XH7mLtbvPsQlXh/8X648g0fezWfSiB4ht9nO+dlwMlJ97dNWTVO5uH8nBmVdyGtLtnHPnFWcc/87gSE7/HMd+Mdo6pLZhOx2zfhvvm9O4vro7w+eW6ImT/33C35x0UkRt53/v+/xxZ5DDTL3QTTdO10I/q8IW7yyaBzLviJynJkYNH9C8DhHkZx3cgeuOyc78IBXs/QUfj3mFDKbpoXU69u5JT3ahz6b0KFlRiCB+xM+wKi+oVNezph0VmCay5rmLq7J26t2cv8bq7n0rx+SlTs78Ldg/VfMW1E5AurrN53DF/eN47Ubh/HgFacHyj/4tW9YkYffXsef317HaXfPwzlHUWk5ry/dxqHissBQ4g0xdWY0Lf1Ij9xFe/U3qn3NbBIwCaB796oDaonI8SEpyZh+/RC6tG7CiW3r99bKjq0y+PbALsxcvBWAX4w8ieuHZ1NYUsbc5Tt49cZhgYun/l8AH63fw9Ce7dh/uJSWGSm1PlFcVFoeGNojnH8yHoCstk0Dw2ec3i2T07tlcnaPdrRvkU5yknFKxxas3nGAP8xfCxBxtFjwzTv9+k/PObITcYSiaelvAboFrXcFtkV5/Kj2dc494ZzLcc7ltG9f9eegiBw/hvZqV+8J32/8wMqOg+uHZ9MsPYVHrz6TjdMuZkDQwHxZXjxXPvkJry3Zyum/eZPsyXM4XFJepf9/f2Ept7ywhFteXMJzH4fOS1Cdl348tEpZx1YZgWshN1Uzemq41Tsiz45Wl2q9ZdPMUoC1wIXAVmAhcKVzbkWEuk8D/3bOveyttwEWAf4rL58BZ/r7+CPRLZsiEi3nHFP/vZKxp3aq8enlVxdv5eYaundW/GZ0YAL5q/72MR/mh3azdG6Vwbb9Rdz/ndMY0K012e2aBabIXHznRbRullblmOFx/s+/lvFiXuRZzJbeNYrDpeVkNk096nv76/SJXDMbBzwEJANPOefuMbOpQJ5zbpaZDQJeAVoDRcAO51w/b98fALd5h7rHOff3ml5LSV9E6sPrS7fx0+cXR9w266Zh9O/q+2Xgv9/fr3l6Cm//8lw+WLeHy87sGijffaCY1GSrcg2iNkWl5Wzdd5gL//A+15+TzR1RTCUaDQ3DICISxp/QR/U9gTeDZmn7y5UD+Ub/zny0fg9XPukbaO62cafw+PsbmPeLEbRrnh6TeI+ERtkUEQlzp9eqvrBPBxZMvqCy/NXl/OS5RYGE37tDcyaN6MmiOy86LhL+kVBLX0QSRnmFY9bSrVxyepfARdbw7hyAVVPH0CSt8Y2bUxO19EVEwiQnGZcO7BryhPHofqED9KUk2XGX8I+ERtkUkYT26FVn8osXlzCqb0c27D7IRf3ie5RWJX0RSWhJScbDEwbWXjFOqHtHRCSBKOmLiCQQJX0RkQSipC8ikkCU9EVEEoiSvohIAlHSFxFJIEr6IiIJpNGNvWNmu4HoZi6oqh2wpw7DqU+KtX4o1vqhWOtHXcZ6onOu1lmoGl3SPxZmlhfNgEONgWKtH4q1fijW+hGLWNW9IyKSQJT0RUQSSLwl/SdiHcARUKz1Q7HWD8VaPxo81rjq0xcRkZrFW0tfRERqEBdJ38zGmNkaM8s3s9xYxwNgZhvN7HMzW2JmeV5ZGzObb2brvH9be+VmZn/y4l9mZmfUc2xPmdkuM1seVHbEsZnZNV79dWZ2TQPHO8XMtnrnd4mZjQvaNtmLd42ZjQ4qr9fPiZl1M7N3zWyVma0ws5975Y3u3NYQa6M7r95rZJjZp2a21Iv3N155tpl94p2nF8wszStP99bzve1Ztb2PBoj1aTP7IujcDvDKG/Zz4Jw7rv+AZGA90ANIA5YCfRtBXBuBdmFlvwdyveVc4H5veRwwFzDgLOCTeo5tBHAGsPxoYwPaABu8f1t7y60bMN4pwK0R6vb1PgPpQLb32UhuiM8J0Ak4w1tuAaz14ml057aGWBvdefVe34Dm3nIq8Il3zl4EJnjljwE/8ZZvAB7zlicAL9T0Phoo1qeByyLUb9DPQTy09AcD+c65Dc65EmAGMD7GMVVnPPCMt/wM8K2g8n84n4+BTDPrVF9BOOf+A+w9xthGA/Odc3udc18D84ExDRhvdcYDM5xzxc65L4B8fJ+Rev+cOOe2O+c+85YPAKuALjTCc1tDrNWJ2Xn1YnTOuYPeaqr354ALgJe98vBz6z/nLwMXmpnV8D4aItbqNOjnIB6Sfhdgc9D6Fmr+8DYUB7xpZovMbJJXdoJzbjv4/qcDOnjljeE9HGlsjSHmm7yfw0/5u0xqiKtB4/W6Ewbia+U16nMbFis00vNqZslmtgTYhS8Brgf2OefKIrx2IC5v+36gbUPFGx6rc85/bu/xzu2DZpYeHmtYTPUSazwkfYtQ1hhuSRrmnDsDGAvcaGYjaqjbWN8DVB9brGN+FOgJDAC2A3/wymMer5k1B/4F3OycK6ipajUxxTLWRntenXPlzrkBQFd8rfM+Nbx2TOMNj9XMTgUmA6cAg/B12fxPLGKNh6S/BegWtN4V2BajWAKcc9u8f3cBr+D7kO70d9t4/+7yqjeG93CkscU0ZufcTu9/rArgSSp/osc0XjNLxZdE/+mcm+kVN8pzGynWxnpegznn9gHv4ev/zjSzlAivHYjL294KXxdhg8YbFOsYr0vNOeeKgb8To3MbD0l/IdDbu4qfhu+izaxYBmRmzcyshX8ZGAUs9+LyX4G/BnjNW54FfM+7in8WsN/fHdCAjjS2ecAoM2vtdQGM8soaRNg1j0vxnV9/vBO8uzeygd7ApzTA58TrM/4/YJVz7o9Bmxrdua0u1sZ4Xr242ptZprfcBBiJ7zrEu8BlXrXwc+s/55cB7zjf1dHq3kd9x7o66Ivf8F17CD63Dfc5ONYrwY3hD9/V77X4+vhubwTx9MB3h8BSYIU/Jnx9im8D67x/27jKq/2PePF/DuTUc3zP4/vpXoqvNXHd0cQG/ADfhbB84PsNHO+zXjzLvP9pOgXVv92Ldw0wtqE+J8A5+H5+LwOWeH/jGuO5rSHWRndevdfoDyz24loO3BX0/9qn3nl6CUj3yjO89Xxve4/a3kcDxPqOd26XA89ReYdPg34O9ESuiEgCiYfuHRERiZKSvohIAlHSFxFJIEr6IiIJRElfRCSBKOmLiCQQJX0RkQSipC8ikkD+Pxs2rOJm24D6AAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Take a look -- we can see the loss decreasing as we train\n", "learn.sched.plot_loss()" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Now we train again with slightly larger images\n", "# Still not 100% sure what the reason for this is -- just simulating different data?\n", "image_size = 128" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "edcf55e4065249f0b9f96d90c1f671a1", "version_major": 2, "version_minor": 0 }, "text/html": [ "

Failed to display Jupyter Widget of type HBox.

\n", "

\n", " If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n", " that the widgets JavaScript is still loading. If this message persists, it\n", " likely means that the widgets JavaScript library is either not installed or\n", " not enabled. See the Jupyter\n", " Widgets Documentation for setup instructions.\n", "

\n", "

\n", " If you're reading this message in another frontend (for example, a static\n", " rendering on GitHub or NBViewer),\n", " it may mean that your frontend doesn't currently support widgets.\n", "

\n" ], "text/plain": [ "HBox(children=(IntProgress(value=0, description='Epoch', max=13), HTML(value='')))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "epoch trn_loss val_loss f2 \n", " 0 0.098714 0.096537 0.917149 \n", " 1 0.100112 0.095297 0.919537 \n", " 2 0.095763 0.094022 0.920467 \n", " 3 0.095169 0.093924 0.919777 \n", " 4 0.098707 0.09404 0.921207 \n", " 5 0.097631 0.093178 0.921026 \n", " 6 0.095902 0.093244 0.920144 \n", " 7 0.093888 0.092771 0.921678 \n", " 8 0.092554 0.092677 0.921674 \n", " 9 0.09418 0.091928 0.922913 \n", " 10 0.092501 0.091704 0.922293 \n", " 11 0.091666 0.091564 0.922321 \n", " 12 0.090078 0.091956 0.921488 \n", "\n" ] }, { "data": { "text/plain": [ "[0.09195557, 0.9214877960864891]" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = get_data(image_size)\n", "learn.set_data(data)\n", "learn.freeze()\n", "learn.fit(learning_rate, 3, cycle_len=1, cycle_mult=3)" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "54cb7bc1cf5044f2b38a4b0a6f3fb2e6", "version_major": 2, "version_minor": 0 }, "text/html": [ "

Failed to display Jupyter Widget of type HBox.

\n", "

\n", " If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n", " that the widgets JavaScript is still loading. If this message persists, it\n", " likely means that the widgets JavaScript library is either not installed or\n", " not enabled. See the Jupyter\n", " Widgets Documentation for setup instructions.\n", "

\n", "

\n", " If you're reading this message in another frontend (for example, a static\n", " rendering on GitHub or NBViewer),\n", " it may mean that your frontend doesn't currently support widgets.\n", "

\n" ], "text/plain": [ "HBox(children=(IntProgress(value=0, description='Epoch', max=13), HTML(value='')))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "epoch trn_loss val_loss f2 \n", " 0 0.092793 0.087805 0.927363 \n", " 1 0.09623 0.088546 0.925771 \n", " 2 0.091078 0.086233 0.92801 \n", " 3 0.087241 0.085489 0.9284 \n", " 4 0.095144 0.089547 0.926124 \n", " 5 0.090785 0.087691 0.926577 \n", " 6 0.087424 0.086907 0.926908 \n", " 7 0.086251 0.086164 0.926724 \n", " 8 0.082321 0.085769 0.92639 \n", " 9 0.080621 0.084496 0.929468 \n", " 10 0.080943 0.084145 0.929588 \n", " 11 0.075996 0.084212 0.929369 \n", " 12 0.077381 0.084321 0.9293 \n", "\n" ] }, { "data": { "text/plain": [ "[0.08432135, 0.9292995924747179]" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Now tweak the initial layers again\n", "learn.unfreeze()\n", "learn.fit(differential_learning_rates, 3, cycle_len=1, cycle_mult=3)" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "collapsed": true }, "outputs": [], "source": [ "learn.save(f'{image_size}')" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Now do it AGAIN, with even bigger image sizes\n", "image_size = 256" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2de642c9cf5848e383a4ec40af258159", "version_major": 2, "version_minor": 0 }, "text/html": [ "

Failed to display Jupyter Widget of type HBox.

\n", "

\n", " If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n", " that the widgets JavaScript is still loading. If this message persists, it\n", " likely means that the widgets JavaScript library is either not installed or\n", " not enabled. See the Jupyter\n", " Widgets Documentation for setup instructions.\n", "

\n", "

\n", " If you're reading this message in another frontend (for example, a static\n", " rendering on GitHub or NBViewer),\n", " it may mean that your frontend doesn't currently support widgets.\n", "

\n" ], "text/plain": [ "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "epoch trn_loss val_loss f2 \n", " 0 0.091138 0.090089 0.924436 \n", " 1 0.089965 0.089359 0.925116 \n", " 2 0.08823 0.089175 0.926036 \n", " 3 0.088119 0.088418 0.926243 \n", " 4 0.085408 0.088381 0.927063 \n", " 5 0.085668 0.088337 0.926008 \n", " 6 0.086202 0.087519 0.927309 \n", "\n" ] }, { "data": { "text/plain": [ "[0.0875194, 0.9273087637207408]" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = get_data(image_size)\n", "learn.set_data(data)\n", "learn.freeze()\n", "learn.fit(learning_rate, 3, cycle_len=1, cycle_mult=2)" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "4ab85ff4d87247f28d6ebc2922ef7309", "version_major": 2, "version_minor": 0 }, "text/html": [ "

Failed to display Jupyter Widget of type HBox.

\n", "

\n", " If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n", " that the widgets JavaScript is still loading. If this message persists, it\n", " likely means that the widgets JavaScript library is either not installed or\n", " not enabled. See the Jupyter\n", " Widgets Documentation for setup instructions.\n", "

\n", "

\n", " If you're reading this message in another frontend (for example, a static\n", " rendering on GitHub or NBViewer),\n", " it may mean that your frontend doesn't currently support widgets.\n", "

\n" ], "text/plain": [ "HBox(children=(IntProgress(value=0, description='Epoch', max=21), HTML(value='')))" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "epoch trn_loss val_loss f2 \n", " 0 0.085767 0.083354 0.932566 \n", " 1 0.08356 0.081909 0.93292 \n", " 2 0.078167 0.080999 0.932661 \n", " 3 0.086357 0.083911 0.930388 \n", " 4 0.083617 0.083095 0.93072 \n", " 5 0.081474 0.083832 0.927965 \n", " 6 0.076187 0.08217 0.93121 \n", " 7 0.075701 0.081892 0.932569 \n", " 34%|███▍ | 174/506 [03:31<06:43, 1.22s/it, loss=0.0751]" ] }, { "ename": "KeyboardInterrupt", "evalue": "", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------\u001b[0m", "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0mlearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munfreeze\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mlearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdifferential_learning_rates\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcycle_len\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcycle_mult\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m~/fastai/courses/dl1/fastai/learner.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, lrs, n_cycle, wds, **kwargs)\u001b[0m\n\u001b[1;32m 207\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msched\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 208\u001b[0m \u001b[0mlayer_opt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_layer_opt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlrs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 209\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit_gen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlayer_opt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_cycle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 210\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 211\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mwarm_up\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwds\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/fastai/courses/dl1/fastai/learner.py\u001b[0m in \u001b[0;36mfit_gen\u001b[0;34m(self, model, data, layer_opt, n_cycle, cycle_len, cycle_mult, cycle_save_name, use_clr, metrics, callbacks, use_wd_sched, norm_wds, wds_sched_mult, **kwargs)\u001b[0m\n\u001b[1;32m 154\u001b[0m \u001b[0mn_epoch\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msum_geom\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcycle_len\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mcycle_len\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcycle_mult\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_cycle\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 155\u001b[0m return fit(model, data, n_epoch, layer_opt.opt, self.crit,\n\u001b[0;32m--> 156\u001b[0;31m metrics=metrics, callbacks=callbacks, reg_fn=self.reg_fn, clip=self.clip, **kwargs)\n\u001b[0m\u001b[1;32m 157\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 158\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mget_layer_groups\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodels\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_layer_groups\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/fastai/courses/dl1/fastai/model.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(model, data, epochs, opt, crit, metrics, callbacks, **kwargs)\u001b[0m\n\u001b[1;32m 94\u001b[0m \u001b[0mbatch_num\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 95\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mcb\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mcb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_batch_begin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 96\u001b[0;31m \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstepper\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mV\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mV\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 97\u001b[0m \u001b[0mavg_loss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mavg_loss\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mavg_mom\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mloss\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mavg_mom\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 98\u001b[0m \u001b[0mdebias_loss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mavg_loss\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mavg_mom\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mbatch_num\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/fastai/courses/dl1/fastai/model.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, xs, y)\u001b[0m\n\u001b[1;32m 47\u001b[0m \u001b[0mnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclip_grad_norm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrainable_params_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mm\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclip\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 48\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 49\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mraw_loss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 50\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 51\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mevaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mxs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mKeyboardInterrupt\u001b[0m: " ] } ], "source": [ "# (Interrupted the kernel here as it started running for 21 cycles for some reason. fasi.ai bug?)\n", "learn.unfreeze()\n", "learn.fit(differential_learning_rates, 3, cycle_len=3, cycle_mult=2)" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "collapsed": true }, "outputs": [], "source": [ "learn.save(f'{image_size}')" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " \r" ] } ], "source": [ "# Finallly, get our predictions using test time augmentation\n", "# Reminder: TTA = Test Time Augmentation, applying transforms to the test data and taking an average\n", "multi_label_classification_predictions,target_values = learn.TTA()" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "collapsed": true }, "outputs": [], "source": [ "predictions = np.mean(multi_label_classification_predictions, 0)" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[0.76759, 0.0016 , 0.02022, 0.00089, 0.00083, 0.99982, 0.00006, 0.00021, 0.43698, 0.02853, 0.00011,\n", " 0.00003, 0.99958, 0.22834, 0.00342, 0.01352, 0.73973],\n", " [0.18095, 0.00006, 0.00077, 0.00014, 0.00077, 0.00035, 0.00022, 0.00009, 0.02922, 0.00091, 0.00005,\n", " 0.99785, 0.99984, 0.01146, 0.0006 , 0.001 , 0.05649],\n", " [0.99405, 0.00013, 0.01334, 0.00012, 0.00006, 0.99976, 0.00001, 0.00005, 0.09454, 0.1095 , 0.00048,\n", " 0.00003, 0.99309, 0.9827 , 0.00023, 0.00075, 0.07534],\n", " [0.01294, 0.9995 , 0.04651, 0.00125, 0.0003 , 0.99966, 0.00012, 0.03368, 0.00482, 0.02961, 0.00009,\n", " 0.00026, 0.96083, 0.30619, 0.00502, 0.00017, 0.99083],\n", " [0.99758, 0.00004, 0.00231, 0.00024, 0.00009, 0.99609, 0.00003, 0.00016, 0.11167, 0.79664, 0.00222,\n", " 0.00135, 0.97242, 0.99619, 0.00044, 0.00086, 0.0222 ]], dtype=float32)" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Predictions contains the likelihood for each of our 17 classes\n", "predictions[:5]" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1.],\n", " [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0., 0.],\n", " [1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0.],\n", " [0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1.],\n", " [1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0.]], dtype=float32)" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Target values contains the actual correct classes for each image\n", "target_values[:5]" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.9308049768554749" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "f2(predictions, target_values)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Note on structured/unstructured data\n", "In the next lesson, we will look at processing structured and unstructed data.\n", "\n", "These are not 100% standardized terms, but generally we think of it as:\n", "\n", "1. __Unstructured data__: Every \"piece\" of the data objects we're processing is the same. For example, in an image, everything is a pixel.\n", "\n", "2. __Structured data__: The parts of the data objects are different. For example, if we are processing sales reports, one column could contain profit, another weather data, another quarterly dates, etc." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.3" } }, "nbformat": 4, "nbformat_minor": 2 }