{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Smooth Bilevel Programming for Sparse Regularization\n", "\n", "This tour is a tutorial reviewing the method developped in the paper:\n", "\n", "> _[Smooth Bilevel Programming for Sparse Regularization](https://arxiv.org/abs/2106.01429)_\n", "Clarice Poon, Gabriel Peyré, 2021\n", "\n", "We present a surprisingly simple reformulation of the Lasso as a bilevel program, which can be solved using efficient method such as L-BFGS. This gives an algorithm which is simple to implement and is in the same ballpark as state of the art approach when the regularization parameter $\\lambda$ is small (it can in particular cope in a seamless way with the constraint formulation when $\\lambda=0$). For large $\\lambda$, and when the solution is very sparse, the method is not as good as for instance coordinate methods with safe screening/support pruning such as the [Celer](https://github.com/mathurinm/celer) solver which we warmly recommend." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "from numpy.linalg import norm\n", "import matplotlib.pyplot as plt\n", "import time\n", "import warnings\n", "import progressbar" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Lasso problem\n", "\n", "We aim at solving the [Lasso](https://en.wikipedia.org/wiki/Lasso_(statistics)) problem\n", "$$\n", " \\min_x \\frac{1}{2\\lambda}\\| A x -y \\|^2 + \\|x\\|_1. \n", "$$\n", "We generate here a synthetic example using a random matrix $A \\in \\mathbb{R}^{n \\times p}$ and $y \\in \\mathbb{R}^p$." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "n = 10\n", "p = n*2\n", "np.random.seed(0)\n", "A = np.random.randn(n, p)\n", "y = np.random.randn(n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The simplest (but arguably not the fastest) algorithm to solve this problem is the iterative soft thresholding\n", "\n", "$$\n", " x_{k+1} = \\text{Thresh}_{\\tau \\lambda}( y - \\tau A^\\top (Ax_k-y) )\n", "$$\n", "\n", "where the step size should satisfies $\\tau < 2/\\|A\\|^2$, and Thresh is the soft thresholding\n", "\n", "$$\n", " \\text{Thresh}_\\sigma( x ) = \\text{sign}(x) \\max(|x|-\\sigma,0)\n", "$$" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAhtklEQVR4nO3deXyU5bn/8c9lTAiEVXYIqyKCKCFEFBcUF9xFORZFOHpaj0jQurRasVb9aV1qT2utilB71NYSARURLLgvdVdICPtqZIkgCQHCGkKS6/cHqQc1gZCZzJOZ+b5fr7yY5Znnvp5Bv7m555nrMXdHRERi32FBFyAiIpGhwBcRiRMKfBGROKHAFxGJEwp8EZE4cXjQBRxIq1atvGvXrkGXISISNbKzsze5e+uqnqvXgd+1a1fmzp0bdBkiIlHDzNZU95yWdERE4oQCX0QkTijwRUTiRL1ew6/K3r17yc/Pp6SkJOhS4k5ycjKpqakkJiYGXYqI1ELUBX5+fj5NmjSha9eumFnQ5cQNd6eoqIj8/Hy6desWdDkiUgthWdIxs2fNrMDMFlXz/BlmVmxmuZU/99R2rJKSElq2bKmwjzAzo2XLlvqXlUgUC9cM/2/Ak8DzB9jmI3e/KByDKeyDofddJLqFZYbv7h8Cm8OxLxGRePbl15v534/yqIvW9ZE8S2egmc03s9fN7NjqNjKz0WY218zmFhYWRrA8EZFgFWwv4YYXcsj6Yi2795aHff+RCvwcoIu79wWeAF6tbkN3f9rdM9w9o3XrKr8dLCISc8rKK/j5C/PYXrKXCaPSaZQU/nNqIhL47r7N3XdU3p4NJJpZq0iMHW5FRUWkpaWRlpZGu3bt6Nix43f3V6xYQZ8+fcI+5urVq8Oy3927d3P66adTXl79zKG0tJRBgwZRVlYW8ngiUnN/eGsFX3y9mYcuO45j2jWtkzEiEvhm1s4qP/EzswGV4xZFYuxwa9myJbm5ueTm5jJmzBhuvfXW7+4nJSUd9PXuTkVFRQQq/bFnn32WYcOGkZCQUO02SUlJnHXWWUydOjWClYnEt7eXbGTiv77iqhM7Myw9tc7GCddpmZOBz4CeZpZvZtea2RgzG1O5yeXAIjObDzwOXOkxejHd8vJyrrvuOo499liGDBnC7t27Wb16Nb169WLs2LGkp6ezbt06Jk2axIABA0hLS+P666+nvLycnTt3cuGFF9K3b1/69OnzvdCtar+HKisri6FDh353f/Dgwbz99tsA/OY3v+Gmm24C4NJLLyUrKyvEd0JEamJN0U5+8WIux3Vsxj0X9a7TscKySOTuIw7y/JPsO20zrO57bTFL1m8L6z57d2jKvRdX+5nyQa1cuZLJkyfz17/+leHDhzNt2jROPfVUli9fznPPPcdTTz3F0qVLmTp1Kp988gmJiYmMHTuWrKwsUlJS6NChA7NmzQKguLj4gPsdNWpUjesqLS0lLy+P/dtN33fffdxzzz0UFBQwb948Zs6cCUCfPn2YM2dOrd8DEamZkr3lZE7K4TAznhqZTnJi9f/6Dgf10gmzbt26kZaWBkD//v1ZvXo1AF26dOGkk04C4N133yU7O5sTTjiBtLQ03n33XfLy8jjuuON45513uOOOO/joo49o1qzZAfebl5fHtddey+WXX/6jOj744ANOO+00xowZwwcffMCmTZto3rz597YZNGgQ7s6jjz7KlClTvlvqSUhIICkpie3bt4f3zRGR77l3xmKWbNjGn67oS6cjGtX5eFHXWmF/oczE60qDBg2+u52QkPDd0ktKSsp3j7s711xzDQ8//PCPXp+dnc3s2bO58847GTJkCPfcc0+1++3evTvPPPNMlYFvZjRu3JiSkhJSU1Np2LDhj74lu3DhQjZs2ECrVq1o0qTJ957bs2cPycnJtXgHRKQmXpyzjqlz13Hj4KM485i2ERlTM/wAnHXWWbz88ssUFBQAsHnzZtasWcP69etp1KgRo0aN4rbbbiMnJ6fWY5x22mm8/vrrPPLII9x77720aNGC8vLy70J/w4YNjBw5khkzZpCSksKbb7753WuLiopo3bq1mqSJ1JHF64u5e8YiTjmqJbeec3TExo3qGX606t27Nw888ABDhgyhoqKCxMRExo8fT3FxMbfffjuHHXYYiYmJTJgwodZjHHbYvt/lLVq0YM+ePQAMGTKEjz/+mJNPPplhw4bxxz/+kV69enH33Xdzxx13cO655wLw/vvvc8EFF4R+oCLyI8W795I5KYcWjZL485X9SDgsgi1L3L3e/vTv399/aMmSJT96LF5t2rTJr7/+eu/evbs/9NBD/umnn/r48ePd3X3atGk+evRoHz58uL///vvu7p6Tk+OjRo066H4vu+wyX7ZsWZXP6f0Xqb2Kigr/77/P8SPvnOVzVxfVyRjAXK8mUzXDj2ItW7Zk4sSJ33ts4MCBAAwbNoxhw4Z977l+/foxePBgysvLqz0Xv7S0lEsvvZSePXvWTdEicewvH+bx9pKN3HNRb/p3OSLi4yvw48zPfvazAz6flJTE1VdfHaFqROLH53lF/P6NZVx4fHt+ekrXQGrQh7YiInWsYFsJN74wj66tUnjkP44PrNW4ZvgiInWorLyCGyfPY+eeMl647kQaNwgudqNyhu+x2ZWh3tP7LnLo/ufN5Xz59WYeHnYcR7dtcvAX1KGoC/zk5GSKiooUPhHmlde01ZexRGruzcXf8pcP8xh1Umcu7dcx6HKib0knNTWV/Px8dHGUyEtOTiY1te46+YnEktWbdnLbi/Ppm9qMu+u4KVpNRV3gJyYm0q1bt6DLEBGpVsnecjKzckhIMMaPTKfB4XXbFK2moi7wRUTqM3fnN68uYtm323j2v04gtUXdN0WrqahbwxcRqc+mzlnHy9n5/HzwUQzu2Sbocr5HgS8iEiaLvinmnpmLOa1HK24+O3JN0WpKgS8iEgbFu/aSmZVNy5QkHrsiLbJN0WpIa/giIiGqqHB++VIu3xaXMPX6gbRs3ODgLwqAZvgiIiGa+OFXvLO0gLsu6EV65xZBl1MtBb6ISAg+/WoTf3hzORf37cA1J3cNupwDUuCLiNTSxm0l3DR5Ht1apfC7YccF1hStprSGLyJSC3vLK7jxhRx2lZYz+bqTSAmwKVpN1f8KRUTqod+/sYw5q7fw5yvT6BFwU7Sa0pKOiMghemPRBv760ddcPbALQ9OCb4pWU2EJfDN71swKzGxRNc+bmT1uZqvMbIGZpYdjXBGRSMsr3MFtLy2gb6fm3HVhr6DLOSThmuH/DTjvAM+fD/So/BkNTAjTuCIiEbO7tJyxWTkkJhhP1aOmaDUVlsB39w+BzQfYZCjwfOVF1T8HmptZ+3CMLSISCe7OXa8uZPnG7Tx2ZT86Nm8YdEmHLFJr+B2Bdfvdz6987EfMbLSZzTWzuep5LyL1xeQv1/FKzjfcdGYPTj+6ddDl1EqkAr+qk1OrvGSVuz/t7hnuntG6dXS+qSISWxbmF/P/Kpui3XRWj6DLqbVIBX4+0Gm/+6nA+giNLSJSa1t3lZKZlU2rxkn8+cp+9bIpWk1FKvBnAldXnq1zElDs7hsiNLaISK1UVDi/eHE+G7eV8NSo/hyRkhR0SSEJyxevzGwycAbQyszygXuBRAB3nwjMBi4AVgG7gJ+GY1wRkbo04V9f8d6yAu4feixpnZoHXU7IwhL47j7iIM87cEM4xhIRiYRPVm3ij28t55K+HfjPk7oEXU5Y6Ju2IiI/8G3xvqZo3Vs35uEoaIpWU+qlIyKyn383Rdu9t5ypo9KjoilaTcXOkYiIhMHvXl/G3DVbeGJEP45qEx1N0WpKSzoiIpVmL9zAMx9/zX+d3JWL+3YIupywU+CLiABfFe7g9pfm069zc359QXQ1RaspBb6IxL1dpWVkTsqmQWIC469KJ+nw2IxGreGLSFxzd+6avoiVBTt4/mcD6BCFTdFqKjZ/jYmI1FDWF2uZPu8bbjnraE7rEdv9uxT4IhK3FuRv5f7XlnD60a35+ZlHBV1OnVPgi0hc2rKzlMxJObRu0oDHrkjjsChuilZTWsMXkbhTUeHc+mIuhdv38NKYgbSI8qZoNaUZvojEnfHvr+KD5YXcfXFv+sZAU7SaUuCLSFz5eOUmHn1nBZemdWDUiZ2DLieiFPgiEjc2FO/mpinz6NGmMQ/FUFO0mlLgi0hcKC2r4IasHPbsLWfCqP40Soq/jzDj74hFJC49/PpSctZuZfxV6RzZunHQ5QRCM3wRiXmvzV/Pc5+s5qendOXC49sHXU5gFPgiEtNWFexg3LQFpHduzp3nx2ZTtJpS4ItIzNq5Z7+maCNjtylaTWkNX0Rikrvz6+kLWVW4g3/87ETaN4vdpmg1Fd+/7kQkZk36fA0zctfzi7OP5tQerYIup15Q4ItIzMldt5X7/7mEwT1bc8Pg2G+KVlMKfBGJKVt2lnJDVg5tmiTzpzhpilZTWsMXkZhRUeHcMnVfU7SXMwfSvFF8NEWrqbDM8M3sPDNbbmarzGxcFc+fYWbFZpZb+XNPOMYVEdnfE++t4l8rCrn3kt4cn9o86HLqnZBn+GaWAIwHzgHygTlmNtPdl/xg04/c/aJQxxMRqcqHKwp57N0VDOvXkasGxFdTtJoKxwx/ALDK3fPcvRSYAgwNw35FRGpk/dbd3DxlHke3acKDl8VfU7SaCkfgdwTW7Xc/v/KxHxpoZvPN7HUzO7a6nZnZaDOba2ZzCwsLw1CeiMSy0rIKxmblsLfcmTAqnYZJCUGXVG+FI/Cr+lXqP7ifA3Rx977AE8Cr1e3M3Z929wx3z2jdOrYvKCwioXtw1hJy123l95cfT/c4bYpWU+EI/Hyg0373U4H1+2/g7tvcfUfl7dlAopnpmxAiEpKZ89fz98/WcO2p3bjguPhtilZT4Qj8OUAPM+tmZknAlcDM/Tcws3ZWuahmZgMqxy0Kw9giEqdWbtzOuGkLyOjSgnHnHxN0OVEh5LN03L3MzG4E3gQSgGfdfbGZjal8fiJwOZBpZmXAbuBKd//hso+ISI3s3FNGZlYOjZISePKqdBIT9B3SmgjLF68ql2lm/+CxifvdfhJ4MhxjiUh8c3fGvbKQvMIdTLr2RNo1Sw66pKihX4siElWe/2wNr81fzy+H9OTko/RR4KFQ4ItI1MhZu4UHZi3hrGPakHn6kUGXE3UU+CISFTbvLOXGrBzaNk3m0eFqilYbap4mIvVeeYVz85R5bNpZyiuZJ9OsUWLQJUUlzfBFpN57/N2VfLRyE/ddcix9OjYLupyopcAXkXrtg+UFPP7eSv4jPZUrT+h08BdItRT4IlJv5W/ZxS1Tc+nZtgkPXNpHTdFCpMAXkXppT1k5N2TlUF7uTBjVX03RwkAf2opIvfTAP5cyP7+YiaPS6dYqJehyYoJm+CJS78zI/YZ/fL6G607rxnl91BQtXBT4IlKvrNi4nXHTFnJC1xb86jw1RQsnBb6I1Bs79pQxZlI2KQ0OV1O0OqB3U0TqBXfnjmkLWL1pJ0+M6EfbpmqKFm4KfBGpF/726WpmLdjAbef2ZOCRLYMuJyYp8EUkcNlrtvDgrKWc3astYwapKVpdUeCLSKCKduzhxhdy6NC8IX8c3ldN0eqQzsMXkcDsa4qWS9G/m6I1VFO0uqQZvogE5s/vrODjVZv47VA1RYsEBb6IBOL9ZQU8/t4qftI/lStO6Bx0OXFBgS8iEbdu876maL3aN+W3l/YJupy4ocAXkYjaU1bODS/kUFHhTBiZTnKimqJFij60FZGIuv+1JSzIL+Yv/9mfrmqKFlGa4YtIxEyfl0/WF2u5flB3zj22XdDlxB0FvohExPJvt3PnKwsZ0O0Ibj+3Z9DlxKWwBL6ZnWdmy81slZmNq+J5M7PHK59fYGbp4RhXRKLD9pK9ZE7KpnGDRJ4c0Y/D1RQtECG/62aWAIwHzgd6AyPMrPcPNjsf6FH5MxqYEOq4IhId3J1x0xayZvMunryqH23UFC0w4fjQdgCwyt3zAMxsCjAUWLLfNkOB593dgc/NrLmZtXf3DWEYX0I0+cu1fLiiMOgyJEZt3bWXz/KKGHf+MZzUXU3RghSOwO8IrNvvfj5wYg226Qj8KPDNbDT7/hVA5876MkZde2PRBu58ZSEdmzckpYFOj5O6Meb0I7l+UPegy4h74Qj8qjodeS222feg+9PA0wAZGRlVbiPh8fWmndz+0gL6dmrOi9efRIPDFfgisSwcn5zkA532u58KrK/FNhJBu0vLyZyUzeEJxlMj0xX2InEgHIE/B+hhZt3MLAm4Epj5g21mAldXnq1zElCs9fvguDu/eXURyzdu57Er+9GxecOgSxKRCAh5Scfdy8zsRuBNIAF41t0Xm9mYyucnArOBC4BVwC7gp6GOK7U3Zc46puXkc8vZPTj96NZBlyMiERKW1gruPpt9ob7/YxP3u+3ADeEYS0KzML+Ye2csZtDRrbnpzB5BlyMiEaRvP8SRrbtKyczKplXjJB67Ik1XFhKJM2qeFicqKpxfvDifjdtKeGnMyRyRkhR0SSISYZrhx4mnPljFe8sKuOei3qR1ah50OSISAAV+HPhk1SYefXsFQ9M6MOqkLkGXIyIBUeDHuG+LS7hp8jyObN2Yh4cdh5nW7UXilQI/hpWWVTA2K5uSveVMGNWfRkn6yEYknikBYtjDry8lZ+1WnryqH0e1aRx0OSISMM3wY9Q/F6znuU9W89NTunLR8R2CLkdE6gEFfgxaVbCDO15eQHrn5tx5fq+gyxGRekKBH2N27ikjc1I2yYkJjB+ZTtLh+isWkX20hh9D3J1fT1/IV4U7+Me1J9K+mZqiicj/0fQvhkz6fA0zctfzi3OO5pSjWgVdjojUMwr8GJG7biv3/3MJZx7ThrFnHBV0OSJSDynwY8DmnaWMnZRN26bJPDq8r5qiiUiVtIYf5cornFum5rJpRynTMk+meSM1RRORqinwo9wT763kwxWFPHTZcRyX2izockSkHtOSThT7YHkBf353JcPSOzJiQKeDv0BE4poCP0p9s3U3t0zNpWfbJjx4qZqiicjBKfCj0J6ycsZm5VBe7kwY1Z+GSQlBlyQiUUBr+FHowVlLmb9uKxNHpdOtVUrQ5YhIlNAMP8rMyP2G5z9bw+hB3TmvT/ugyxGRKKLAjyIrNm5n3LSFDOh6BL86t2fQ5YhIlFHgR4kde8oYMymblAaH8+RV/Tg8QX91InJolBpRwN254+UFrN60kydG9KNN0+SgSxKRKBTSh7ZmdgQwFegKrAaGu/uWKrZbDWwHyoEyd88IZdx489wnq5m1cAPjzj+GgUe2DLocEYlSoc7wxwHvunsP4N3K+9UZ7O5pCvtDk71mMw/NXso5vdty/aDuQZcjIlEs1MAfCvy98vbfgUtD3J/sZ9OOPdyQNY+OLRryh5/01ZerRCQkoQZ+W3ffAFD5Z5tqtnPgLTPLNrPRB9qhmY02s7lmNrewsDDE8qJXeYVz85R5bNlVylMj02nWMDHokkQkyh10Dd/M3gHaVfHUXYcwzinuvt7M2gBvm9kyd/+wqg3d/WngaYCMjAw/hDFiyp/eXsEnq4r4/eXHc2wHNUUTkdAdNPDd/ezqnjOzjWbW3t03mFl7oKCafayv/LPAzKYDA4AqA1/gvWUbefL9VVyR0YnhGWqKJiLhEeqSzkzgmsrb1wAzfriBmaWYWZN/3waGAItCHDdmrdu8i1um5NK7fVPuG3ps0OWISAwJNfB/B5xjZiuBcyrvY2YdzGx25TZtgY/NbD7wJTDL3d8IcdyYVLK3nMysbAAmjupPcqKaoolI+IR0Hr67FwFnVfH4euCCytt5QN9QxokX9722hEXfbON/r86gc8tGQZcjIjFG37StJ17Ozmfyl2vJPONIzu7dNuhyRCQGKfDrgaUbtnHX9IUM7N6SX55zdNDliEiMUuAHbFvJXjInZdOsYSKPj1BTNBGpO7oASoDcndtfms+6LbuZMvokWjdpEHRJIhLDNJ0M0F8/yuPNxRu58/xjOKHrEUGXIyIxToEfkC/yinjkjeWc36cd157aLehyRCQOKPADULC9hBsnz6PLEY34/eXHqymaiESE1vAjrKy8gp+/MI/tJXv5x7UDaJKspmgiEhkK/Aj7n7eW88XXm3l0eF+Oadc06HJEJI5oSSeC3lr8LX/5Vx4jT+zMsPTUoMsRkTijwI+Q1Zt28suX5nN8ajPuubh30OWISBxS4EfAvqZoORxmxvir0mlwuJqiiUjkaQ0/Au5+dRFLN2zjuf86gU5HqCmaiARDM/w6NnXOWl7KzuemM49i8DHVXQFSRKTuKfDr0KJvirl7xmJO69GKm89WUzQRCZYCv44U79pLZlY2LVOSeOyKNBIO05erRCRYWsOvAxUVzi9fyuXb4hKmXj+Qlo3VFE1EgqcZfh2Y+OFXvLO0gLsu6EV65xZBlyMiAijww+7TrzbxhzeXc3HfDlxzctegyxER+Y4CP4y+LS7hpsnz6NYqhd8NO05N0USkXtEafpjsLa/gxhdy2FVazuTrTiKlgd5aEalflEph8sjry5i7ZguPj+hHj7ZNgi5HRORHtKQTBrMXbuB/P/6aawZ24ZK+HYIuR0SkSgr8EOUV7uBXLy8grVNz7rpQTdFEpP4KKfDN7CdmttjMKsws4wDbnWdmy81slZmNC2XM+mRXaRmZk3JITDCeGplO0uH6/Ski9VeoCbUIGAZ8WN0GZpYAjAfOB3oDI8ws6qfC7s5d0xexomA7j4/oR4fmDYMuSUTkgEL60NbdlwIHO/1wALDK3fMqt50CDAWWhDJ20LK+WMv0ed9w69lHc1qP1kGXIyJyUJFYg+gIrNvvfn7lY1Uys9FmNtfM5hYWFtZ5cbWxIH8r97+2hDN6tubnZx4VdDkiIjVy0Bm+mb0DtKviqbvcfUYNxqhq+u/VbezuTwNPA2RkZFS7XVC27Cwlc1IOrZs04E/D0zhMTdFEJEocNPDd/ewQx8gHOu13PxVYH+I+A1FR4dz6Yi6F2/fw0piBtEhJCrokEZEai8SSzhygh5l1M7Mk4EpgZgTGDbsn31/FB8sLufvi3vTt1DzockREDkmop2VeZmb5wEBglpm9Wfl4BzObDeDuZcCNwJvAUuBFd18cWtmR99HKQv70zgou69eRUSd2DrocEZFDZu71bpn8OxkZGT537tygy2D91t1c9MTHtGqcxKs3nEKjJHWkEJH6ycyy3b3K70Xpm0IHUVpWwdisHErLKpgwqr/CXkSiltLrIB6avZTcdVt5amQ6R7ZuHHQ5IiK1phn+Acycv56/fbqaa0/txgXHtQ+6HBGRkCjwq7Fy43bGTVtARpcWjDv/mKDLEREJmQK/Cjv3lJGZlUOjpASevCqdxAS9TSIS/bSG/wPuzrhXFpJXuINJ155Iu2bJQZckIhIWmrr+wPOfreG1+ev55ZCenHxUq6DLEREJGwX+fnLWbuGBWUs465g2ZJ5+ZNDliIiElQK/UtGOPdyQlUO7Zsk8qqZoIhKDtIYPlFc4t0zNpWhnKa9knkyzRolBlyQiEnaa4QN/fmcFH63cxP2XHEufjs2CLkdEpE7EfeC/v7yAx99bxeX9U7nihE4Hf4GISJSK68DP37KLW6fm0qt9U347tM/BLtUoIhLV4jbw95SVMzYrh/JyZ8LIdBomJQRdkohInYrbD23vf20JC/KL+ct/9qdrq5SgyxERqXNxOcOfPi+frC/Wcv2g7px7bFWX6xURiT1xF/jLvt3Gna8sZEC3I7j93J5BlyMiEjFxFfjbS/aSOSmHJsmJPHlVPw5XUzQRiSNxs4bv7vzq5QWs3byLF/77RNo0UVM0EYkvcTPFfebjr3l90bf86tyenNi9ZdDliIhEXFwE/pzVm/nd68sY0rstowd1D7ocEZFAxHzgF27f1xQttUVD/jC8r75cJSJxK6bX8MvKK7hp8jyKd+/lbz8dQNNkNUUTkfgV04H/6Nsr+CyviD/8pC+9OzQNuhwRkUCFtKRjZj8xs8VmVmFmGQfYbrWZLTSzXDObG8qYNfX2ko089cFXjBjQicv7p0ZiSBGRei3UGf4iYBjwlxpsO9jdN4U4Xo2sLdrFL17MpU/Hptx78bGRGFJEpN4LKfDdfSlQrz4ILdlbTmZWNgZMGNmf5EQ1RRMRgcidpePAW2aWbWajD7ShmY02s7lmNrewsPDQB3Lo2bYJf7oijU5HNKptvSIiMeegM3wzeweoqsPYXe4+o4bjnOLu682sDfC2mS1z9w+r2tDdnwaeBsjIyPAa7v87DZMSePSKtEN9mYhIzDto4Lv72aEO4u7rK/8sMLPpwACgysAXEZG6UedLOmaWYmZN/n0bGMK+D3tFRCSCQj0t8zIzywcGArPM7M3KxzuY2ezKzdoCH5vZfOBLYJa7vxHKuCIicuhCPUtnOjC9isfXAxdU3s4D+oYyjoiIhC7me+mIiMg+CnwRkTihwBcRiRMKfBGROGHuh/zdpogxs0JgTS1f3gqISO+eCIiVY4mV4wAdS30UK8cBoR1LF3dvXdUT9TrwQ2Fmc9292g6e0SRWjiVWjgN0LPVRrBwH1N2xaElHRCROKPBFROJELAf+00EXEEaxciyxchygY6mPYuU4oI6OJWbX8EVE5PtieYYvIiL7UeCLiMSJmA58M/utmS2ovHj6W2bWIeiaasPM/sfMllUey3Qzax50TbVV0wvf11dmdp6ZLTezVWY2Luh6QmFmz5pZgZlFdbtyM+tkZu+b2dLK/7ZuDrqm2jKzZDP70szmVx7LfWHdfyyv4ZtZU3ffVnn7JqC3u48JuKxDZmZDgPfcvczMHgFw9zsCLqtWzKwXUMG+C9/f5u5zAy6pxswsAVgBnAPkA3OAEe6+JNDCasnMBgE7gOfdvU/Q9dSWmbUH2rt7TuW1N7KBS6Px78X2XSA8xd13mFki8DFws7t/Ho79x/QM/99hXymFfdfWjTru/pa7l1Xe/RxIDbKeULj7UndfHnQdtTQAWOXuee5eCkwBhgZcU61VXmZ0c9B1hMrdN7h7TuXt7cBSoGOwVdWO77Oj8m5i5U/YciumAx/AzB40s3XASOCeoOsJg58BrwddRJzqCKzb734+URosscrMugL9gC8CLqXWzCzBzHKBAuBtdw/bsUR94JvZO2a2qIqfoQDufpe7dwKygBuDrbZ6BzuOym3uAsrYdyz1Vk2OJUpZFY9F5b8aY5GZNQamAbf84F/3UcXdy909jX3/kh9gZmFbbgvpilf1wSFcZP0FYBZwbx2WU2sHOw4zuwa4CDjL6/kHL+G48H09lQ902u9+KrA+oFpkP5Xr3dOALHd/Jeh6wsHdt5rZB8B5hOk64FE/wz8QM+ux391LgGVB1RIKMzsPuAO4xN13BV1PHJsD9DCzbmaWBFwJzAy4prhX+UHnM8BSd3806HpCYWat/30Wnpk1BM4mjLkV62fpTAN6su+skDXAGHf/JtiqDp2ZrQIaAEWVD30ejWcbwb4L3wNPAK2BrUCuu58baFGHwMwuAB4DEoBn3f3BYCuqPTObDJzBvla8G4F73f2ZQIuqBTM7FfgIWMi+/9cBfu3us4OrqnbM7Hjg7+z77+sw4EV3vz9s+4/lwBcRkf8T00s6IiLyfxT4IiJxQoEvIhInFPgiInFCgS8iEicU+CIicUKBLyISJ/4/Xb74Yrbv17sAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "def Thresh(x,sigma): \n", " return np.sign(x) * np.maximum(np.abs(x) - sigma, 0)\n", "x = np.linspace(-3, 3, 1000)\n", "\n", "plt.plot(x, Thresh(x, 1.5), label='Thresh$_{1.5}(x)$')\n", "plt.legend();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The critical $\\lambda$ for which the solution of the Lasso is 0 is $\\|A^\\top y\\|_\\infty$. We select the $\\lambda$ as a fraction of this value." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "la = .5 * np.max(A.T @ y)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABTb0lEQVR4nO29d5wkR333/64Ok8PmeFkXdHe6JJ0iQUJCSIggjCwRjCXARmDAGAwYDD+Ms3l4MBgwthFCPAIMCESQLINAEggJFC9Il0+Xw97mMDt5prvr90f3zM7u7V7YcHu7W++XSlVdXd1TPbfz/VR9K7SQUqJQKBSKuYs23RVQKBQKxfSihEChUCjmOEoIFAqFYo6jhEChUCjmOEoIFAqFYo5jTHcFxkNdXZ1ctGjRdFdDoVAoZhSbN2/ukVLWj8yfkUKwaNEiNm3aNN3VUCgUihmFEOLIaPnKNaRQKBRzHCUECoVCMcdRQqBQKBRzHCUECoVCMcdRQqBQKBRzHCUECoVCMcdRQqBQKBRznBm5jkAxs7Edm4yVIVPMkLWyZK0seTtfjvN2noJdoOgUKdpFLGlhORa2tHGkg+VYSCmRSBzpDLu3EAINDQRoaOiajiY0dKFjaAaGMNA1HVMzMTQDUzPx6T58mg9TN/HrfvdY9xHQA/h1PwEjQEAPoGv6NH1jCsXUooRAMW6KdpHeXC+9uV76sn0M5Afoz/UzkB8gkU8wWBgkkU+QLCRJFVMkC0nSxTQ5Ozep9RCIcloyde/XMDWTgBEgqAcJmkECeoCgESRkhtzYCBEy3RA2woTNoRAxI4TMEBEzQsQXKR9rQnXKFdOPEgLFqBTtIu3pdtpSbbSn2+lId9CR7qAr00VXtovuTDcD+YFRr9WFTtwfJ+aLEfPFiPvjzIvOKxvEsC9M2AgTNIMEDS/oQfyGn4AeKLfIS630UmteF/qwFr5AIIQ46fOldHsKDg6OdLAdG1va2I5d7l0UnWI5LtpFCk6Bgj0UcnauHOetPDk7R87KleOMNdSbSRVSdGW6yBQzZKwM6WKaolM87XcsEGVhiPqiRMwIMV+MqC86LMR8MWL+2LDvM+aLETSCoz6/QnG2KCGYw1iOxYnUCQ4mDnIocYgjg0c4ljzG0eRROtOdw1rXAkFtsJbGUCPzIvO4uOFi6oJ11AZrqQ3UUhOooSZQQ1WgiqgZnVYDJYRwRQPPlTMNHp2iXSRjZUgVU6SLaTLFjNsjstKkCilShRTJottDShaS5V5TR6aDfQP7GCwMkiqkTtnDMTWTuD9O3Bd3xcEfK6er/FXuOS9dDoEq/Lr/HH4TipmAEoI5Qm+2lz19e9jTt4f9A/vZP7CfgwMHKTiFcpmaQA3zo/O5tPFSWqOttEbc0BxupjHUiKmb0/gEMwtTN4nrriEeL450SBVTDOYHSRaSJAoJBvODZZfbyLg91c7u/G4GC4NkreyY9w0aQar91VQFqobH/iqqA9VUB9x0TaCmLCBqfGR2o4RgFpIuptnWvY0dPTvKoSvbVT7fFG5iadVSrmy+kiVVS1gcX8yi2KIJGS3F5KMJrewOOlvydp5EPlEer6kcu+nP95fP9ef6OZo8ykBugGQxOeq9BIK4P+6KhL+a2mAt1f5qaoI15biyVxj3x9XYxwxDCcEsoCfbw6aOTWzq3MQLXS+wb2BfeTbNwthCNjZtZFXtKlbWrGRFzQpl8OcAft1PQ6iBhlDDGV9TtIsM5Afoy/XRn++nP9fvpnNuuj/vHh8YOFAWltFcV7rQqfJXURt0xaHkPiwd1wXrqA3UUhesozpQjaEpMzTdqH+BGchgYZDn25/n6fanebb9WQ4PHgYgZIRYW7+W9659L+vq13FR3UXK6CvOGFM3qQ/VUx86abv6UbEcq9yr6Mv10ZvtpT/fT2+21z3OufGxrmP05fpGdVcJRFk06oJ1w0I5L1BHfaiemC+mBseniEkRAiHEjcCXcYfl7pZSfm7EeeGdvwnIAO+UUm7xzh0GkoANWFLKjZNRp9mElJK9/Xt58viTPNn2JC92v4gjHYJGkEsaL+HNy97MxsaNrKxdqVpXinOGoRllo30mZIoZerPudOPebC892R56c25cSm/t2kpPtoe8nT/pelMzh4lEfbC+fFxK14fqqQ3WYmpqPOtsmLDVEELowNeA64HjwPNCiAellLsqir0WWOaFy4H/9OISr5JS9ky0LrMJy7HY3LmZXx/9Nb8+9ms60h0ArKpdxZ+u+VOuarmKtXVr1QCuYsZQWmMxPzb/lOWklKSKqSGByPbSne2mO9tdFpC2VBvburfRl+sb9R6lsYz6oNvDKYtFyI1LwhEyQ1PxqDOOyWg+Xgbsl1IeBBBC/AC4GagUgpuBb0spJfCMEKJKCNEspWyfhM+fNdiOzZauLTx86GEeOfII/fl+/Lqfq1qu4v3r3s8r5r3ijFtfCsVMRQhRXkOxOL74lGWLTpG+bB892Z6yWPRkesrHPdkeDnUcoifbg+VYJ10fNsMnicVJghGqm/Yp0VPNZAhBK3Cs4vg4w1v7Y5VpBdoBCfxKCCGBr0sp75qEOs0oDgwc4MEDD/LQwYfoynQRNIJcPe9qblh0A1e1XKVaLQrFGJiaSWO4kcZw4ynLOdIhkU+UhaIsGtkeujNuvKNnBz3ZnlHHMvy6f0goKnsYnjuqlK4OVM/IGVOTIQSjyeTIqQSnKvMyKeUJIUQD8IgQYo+U8omTPkSIO4E7ARYsWDCR+p4XZIoZfnHoF9z/0v3s6N2BLnRe3vpyPr7x47xy3iuV8VcoJhFNaOU1Esurl49ZTkpJupgeJhLd2e5y3JPtYf/Afp458cyo020NYVATrBlVJCqFpDZQe165dSdDCI4DlU6/ecCJMy0jpSzFXUKIn+K6mk4SAq+ncBfAxo0bp25DmSnmwMABvrf7ezx08CEyVoalVUv5xKWf4LWLX0ttsHa6q6dQzGmEEO5eUL7Iad1SOSt3kmCUxjW6s910pDvY3rOd/lz/qNNsq/xVJw12DwshNz4XbqnJEILngWVCiMVAG/BW4O0jyjwIfNAbP7gcSEgp24UQYUCTUia99GuAv5+EOp1XSCl5su1Jvrvruzzd/jQ+zceNi2/k1uW3sq5+3az2PSoUs5WAEWB+dD7zo6ce/K4cx6gcy+jN9rpuqVwPmzs3053tHnWPqpJbqjT4/Z6172F17epJfZYJC4GU0hJCfBD4Je700XuklDuFEO/zzv8X8HPcqaP7caePvsu7vBH4qWcIDeB7UsqHJ1qn84WiU+ThQw9zz4572D+wn4ZQAx/a8CFuWX4LNYGa6a6eQqE4B5zpOIaUksHC4LCB7pJYlKbZHhk8QtE+/YaGZ4twJ/LMLDZu3Cg3bdo03dUYk6Jd5GcHfsbd2+7mRPoES6uW8q6L3sVrF79WzW9WKBTThhBi82hrtdTqo0mk6BT56b6fcvf2u2lPt7O2bi2fuvxTvGLeK2bkTAKFQjE3UEIwCTjS4VeHf8VXt36Vo8mjrK1fy2ev/CxXtVyl/P8KheK8RwnBBNnUsYnPP/95dvftZln1Mr523dd4ResrlAAoFIoZgxKCcdKWauOLm77Ir478iqZwE//88n/mpsU3qX3bFQrFjEMJwVlStIt8a+e3uGvbXQgE71//ft65+p0EjeB0V02hUCjGhRKCs2BTxyb+4Zl/4GDiIK9Z+Bo+funHaQo3TXe1FAqFYkIoITgDMsUMX9z8Re7bex+tkVb+47r/4BXzXjHd1VIoFIpJQQnBaXi+43k+8/vPcCJ1gttX3c4HN3xQuYEUCsWsQgnBGBTtIl/d+lW+tfNbLIgu4N7X3suGhg3TXS2FQqGYdJQQjMKxwWP81RN/xY7eHdy2/DY+dunHVC9AMSuRUlK5u0Dl8cj80dKnuu/p4pHpU+VJKXEcZ1iZyuOxyo2VV3k8Vnoqyk3GfV73utdN+g7MSghG8MiRR/jM7z+DJjS+eM0XuX7h9dNdJcU5REqJbdtYloVlWeV0ZTwyVOY7jjPsXOWx4zjl49HSlaH04x8rr9I4jGUQxzKgI42u4uwRQpSDpmmjpk937nRlDMMYM3+yUULgYTs2//7Cv3P39rtZU7eGL1z9BVoiLdNdrTlNySgXi8Uxg2VZox6PFp9JsG170p+j9EPWdX1YPDI9WjAMo5wu3acyfSrjA4xqfEp1Gi3vVPHI9KnyRp4bLR6rPpX1HusZxgqjlT0b43w2ZWcTSgiARD7BJ574BL8/8XtuWXYLn7r8U/h033RX67xmpJEuFArDDPLI49PljxXG02ottZpM0xwWl0IgECjn6bo+LF0qU0qfKh4rlAx8pZFXKM5n5rwQHE8e5/2PvZ9jyWP8zZV/w63Lb53uKk2IkQZ6rBbyqVrVpzPcpbyzNdJCCEzTHDWEw+GyQfb5fKOmR7tuZH7JSM+2FptCMZXMaSHY0bODDzz2AYpOkbuuv4tLmy6d7iph2zbZbJZMJkMmkyGbzZZDLpcrh3w+Xw6FQqEcxtuKBsqGt9IAm6ZJIBAgGo2WjyvPjXY8VhlloBWK85M5KwRPHH+Cj/32Y9QEavjWdd9iSdWSKfssKSW5XI7BwUGSyWQ5pFKpckin06TTaXK53Jj3EUIQCATw+/3lOBaL4fP5ymG0FnNlq3lkurLFrYy0QjE3mZNC8MvDv+STT3yS5TXL+dp1X6MuWDfuezmOQyqVKhv3krEfHBwcForFk98qFAgEiEQihMNhmpqaCIfDhEKhYSEYDBIMBsuGXxlrhUIx2cw5IXhg/wP8zVN/w/r69fz7df9O1Bcds6zjOCSTSRKJRDlUGvdSq36kK0YIQTQaJRqN0tjYyLJly4jFYsRisXJ+ydWiUCgU082cEoL79tzHPz77j1zRfAVfftWXCZkhAPL5PJ2dnXR3d9Pd3U1PTw99fX0MDAzgOM6we5TcMdFolIaGBqLR6DADH4vFCIfDaqaIQqGYMUyKEAghbgS+jPvy+rullJ8bcV5452/CfXn9O6WUW87k2snE0AyumX8Nf3fJ37Fv1z4OHz5MW1sb3d3d5Va9aZrU1tbS1NTEypUrqa6upqqqing8TiwWw+/3T1X1FAqFYlqYsBAIIXTga8D1wHHgeSHEg1LKXRXFXgss88LlwH8Cl5/htZPGouQiVr+0mq888RUAgsEgra2trFy5kpaWFhoaGojH46o1r1Ao5hST0SO4DNgvpTwIIIT4AXAzUGnMbwa+Ld1m9zNCiCohRDOw6AyunTT+ZsduuprnIxpaEY4EKRGaRPQPwEA/Ys8OhJCgxmMVCsV5Sl1nmgf+9I5JvedkCEErcKzi+Dhuq/90ZVrP8FoAhBB3AncC495wKRLpIzP22HAZiYCZuA3LTBOwmfgdKxTTjG6ePANxokyGEIxmfkb+xMcqcybXuplS3gXcBbBx48ZxmZCvrm4k17eT8LJrkcLBsSGX8pMZMEl0QMeBAh3781gFgaYLWpdXsWhtPUvW1xOpPv3YQLGtjd5772Xgvh8ii0Xib3gD0RtvwLdgAWZrK1ogMJ5qj8lL6Rz3dfTxw44+ugsWDT6DV1eFuVJ3WJUfRPb3kurrJdXfR2agn/RAP+lEP5lEgmIuO+Z9dcPAH47gD4XwBUP4AkHMYNCN/X5MfwDD78fw+TB8fnTDxDBNdNNE81b2arqBpkEgeQR//y4CfTvw927HyHQC4OgBCrWrKdStIV+7BivcjO2LY+shJCBLm6s5Do5jDz+27XLs2Fb5nHtsl69xLKsi36q4ZkRwHBzbGpFv4dgj72PhWLabV1FG2ja2d//pRNN1NE1H6DqarrlpTUPTh2JN0xCajqbrgEA6AscGxwbbBrsIVlFiW6WfpwYIhKZh+k18ARNfcCj4gz5Mv4Fh6uim4X5O+TO9z9Pdz9N1r27e558cjFGOtWH5+mnKiIo9ihRnzmQIwXFgfsXxPODEGZbxncG1k8bOR3/Iq3oepff5X1F78xcQi14OjcPL2JZD56EEh7b1cnhbD0/e9xJP3vcSTUviLL2kgQsuridSPdygZ3fupO+ebzH48MMgBPE3vIG6970X38KFk/4MnYNJfnjoOD8dyLDL0dGkZM1AB687sJ2W3VsoplMcBY5WXBMIRwhX1xCuqqJ56QpCsTiheBXBWIxgNEYwEiMQjRKIRPGHw5i+cQ6IZ/rg+CY4/hwcexbatkAh5Z6LNMGiK2D+FbDgCrSmNQR0k8mVxulFSol0HFcUbNsVkpLAOCOOhwmRjWNVCFi5jHOygJUFyRomjG7axrbc8qV6FHMW+XSefLZAPlukkC1STBcp5i1sywLp4La9XDepYYLhE/iDoOkSTQehgUAipftZ+bRDLlkhyt5zD4mw+xxM0+6mowmiNkyERuadLJrDhVNDCG0oXdr4r3SNpiE0UT4nhFYWpKFywsv3YiHKZSrTlMogymURDIsXrl1PtGb8a59GQ0x0K1ohhAG8BFwHtAHPA2+XUu6sKPM64IO4s4YuB74ipbzsTK4djY0bN8pNmzaddV2//1cfY9sFJh/ou595+S6yy24ieN2noemiMa/p70hzYEsX+7d003vcNWrNF8S5YG019T1byT/wQ3LbtqGFQlTddhs1d9yO2dx81nWrJJtKMtB+gv6OE/S3n6C7s4OnHZ1nqlvY17IERzeo6+3gor1bWXfsJVqjYaL1DcTqGojW1hGtqydaU0ukpo5wdfX4DfupsIvQucMz/Jvg+PPQd8A9J3T3O51/Ocy7DBZcDvH5oFpqk45jOyT7ciS6swx2Z0l4YbAnR6Ini5Wv2E1VQKTaT7w+SLwuSKw+SLw+RLzeTfuDkzubvFIUhgnjqL2ril7WiPySsNq2PdT7sp1h+a4wji2elb22k3qOlT3Myt5nWWCdYT3S4Wl7aKtvx0HKUr4cKuflTRZv/uu/Y/H6S8Z1rRBis5Ry40n5k7EnuRDiJuDfcKeA3iOl/CchxPsApJT/5U0f/XfgRtzpo++SUm4a69rTfd54hcDJ5Tj4r//Kv+UyLFqU5L0nfkTUzlBYcj2+az8J8076fobRd7iXXQ9s4dCeNIMyBtKhunCCJSvCrPyjq4nPP3OVLmQz9HvGfqD9BP3tba7h72gnlxzE0g0Oz1vKnqVrOLhoJXnTR9wqcI2d4fUxH5c21lPV2EwgEjnr7+GscRzo3Q8ntsKJLdC2Gdq3gZ13z0caoXUjzL8U5l0KzevBfw7qNUco5m0GeyqMfHeWhHec7M0hnaHfsG5oxOoCZeMerw8SqxuKdUPNiJsu3HdAlASDIYGoeG9EKS29d0hQ+e4ILw7F45j+8fWlp1QIzjXjFYLvPvwCB5/azMduu5xf/OsX+X9XXMErfLt4z/H7qbaTpP3zYc0fEbjyXdhZC7u3l2J7B9lt28i+8AK5HTuQhQJ6dTXymjfQu+RqjrTr9J1IA9C4OMaS9fUsXldHrM5PeqCPwe4uEl2dXuhgoKOdgc52MomBYXWL1Nbhn7eQI4tXsb22lc1mmCyCKkPndfVx3thQzcuqIhjaFLeqbQt697mGvv1FaH/BTReS7nkzBC0boPViaLnYNfzxeaq1PwGklGSTxZOMfek4M1gYVt4fMoaMu2fsSyEc9yOm+m9EMWNRQgD8/l1/TuylIxz8zGd546vXkX7qKR564nd8Y8EiLtf2cHvbA1yYPUze0Wnri9HRFSPRGUB3dIItrYSWLCGwZg3m4sVI3FZ9PpNmoHOArsMdDHT0kEsnkE4KZJqR496RmlqqmpqpanRDtLmVzuoGthhBfpvIsimRxgEafAY31sW5sS7OK6qjmFP1w073QtdO6Nzlunk6d0DXbrC8je+MoOviaV7vGv+WDVC3HPQ5tSB9UrAth2Rvrmzch2I3r5gf/kKcSLWfWN3Jhj5WFyQQVluTKMaHEgKg654t5HZ08ftt3+L6n95LXdTtXjn5PI9v38M3ugdI547wls6HeV33b6myU+Qdg2PpGEfTVRxJV9NXCHLSZCchCEZjhKuq8YdiSBkilwmQ6jeRRDF9VbRcOJ/6VQ0MLAhywC95djDNswMpkrbrO1wbDXJdTYzramNcHAuhTVYL27FhsA169rnunZ590L3HNfiZnqFyoVpoXA1Na72wRhn9s6CyVT/Ym2WwO+fGPW461Z8bNnY6zIVTFxzWwo/VBTBMffoeRjFrGUsI5tSv/L+F5FZfmEvNGr73f+/lQ3//XgA0v59rN67jWqAtV+BHHa/hbV29xI49xU09T/Cq/k1cGz0IQM4XJ1W7GtmwFrN5DYF56/A3XYgwTIqOJOs49BQs2vMF2tJ5XjyeYEd/hkN2kW6tD3nCNfCNeXiF6eeapjg3LKmlMTjOAV3HgVSna+wTx2DgqBv6j0D/ITdtV7gW/DGoXwErboT6ldCwEhovgkiDcu+chnymyGBvjmSPZ+R7cyR7siR6ciR7s1iF4QOCobiPeF2Q5mVxYnVN7gBtXYBYXYhw3KdcOIrzhjnVI/j1zj0Y3zvEvHSBE4//C/y/7/Py9YvHLH88V+DXvYNsHszQ3vES89uf5pLBXVw8uJvlmcNonusnp/k4EmjhaKCZo4Em2v31dPtq6PLVMGDEiEVqaIzX0eqL05TUiR/KktybYLDbnctv+DQaFsZoXBSmaYGP+kaNSNhC5Ach2w/ZAcj0QroH0t2Q7oJkx1BwRiwwCVRB1QKoWQzVi924dinULlMGfwxKLfpkX45UX45kX8419OWQpZAb7r4xAzqxWs+41waJ1Qfc1n1tkGhdANOnWvWK8wvlGgJ27vo4/Vs6WLTjTtLP/Re/joV5xw++RjRwZj7XtGVzLF/gWLZAZ2oAX89eon0vUdW/j5rUcarTbcRTx/GXBlbHQveBZiKRSAekY6PJIoIzmGIWrIFwPUSbINbixa3u9Mx4qysAgfgZPc9cQUpJLl0kPZAn1V8KOdL9eZL9eVJ9OVL9eWxr+PfvGvoA0ZoA0dogUS/ttuqD+EPqZT6KmYUSAqC397e88MKfUPvYfxIuZrH+5xP84tYP85G/v3Nyf9CFjNtqT3W5i6xyCcgNQCENVh6srOu7LyE0MPzYGGTSGoODOgMDgr4+jd5eH1krRM6JkJVxorVhqpvDVNWHiDcEiTd4LdCaALo5t6YG2pZDNlkkmyyQGSyFPOlEgUwiTyZRIJ3Ikx4onGTkhYBwlZ9wld818NUBIjV+ojUBIjWuwVeGXjHbUGMEQHX1VRhmnCMtz3HJ8VfSveoarvnJf/CT9Rdyyy1XT94H+ULgWwTVi87qMh2IeqHVy3Nsh4GuLL1tKfo7MvR3pOnvyND20sBJi4XCMR+RmgCRKj/haj/huJ9QzEco5iMQMQmETQIRE9M//e8Olo6kWLAp5m2sgk0x71DM2xTzFsWcTSFnUcja5DNF8hmLfMYilymSS7khmypSyFqj3tsX0Al5z964OO5+H1V+ItX+svEPx31o+twSToViLOaUEGiaSWPjTRSX/4jB4y+n6tp30XNkK1X/8v+x86LvsXrF/NPf5Byj6Ro1zWFqmsPD8qWUZAYLJLrcWSrJXtenne7P0dee5ujuPoojfNolhABf0MAM6Jh+A9OnYfh0DFNDNzV0Q0MzBJoQCN1b6u5dKLzPlgCOxJEgbYnjSBzbwbYkji2xLQe76Lix5WAV3WOrYGMVnJNa6KfCF9Dxh0xPzAxitQECUR+hqEkg4iMU9RGK+whGXdEz/co3r1CcDXNKCAAaG15PW9v3eD56jGuPL6T1H79E4KPvYfcHPkzrj79DVTw03VU8I4QQhONuq79lWdWoZYp5u+wyyaUK5NJDLelCzqaQtbDydrllnksXsS3XiLsbrknXyEvK+8ZI6Y01C4EQoGnuBn1CCDRDQ9MFui7QDVdUfEEDw6eVRcbw6Zg+Dd3UMX06pt/L87vB8Ov4Aga+YCk20NTsGoViSplzQlBVdSl+fxORtY+R+/27CHTHsP78Y6z+8v/h12//U266/x4CwdmxFZrp18sLkRQKhWIs5pyTVAiNxobXUR15ioeNLHJXH2tuewtd7/wAKw9s5Te3vRsrm5vuaioUCsU5Y84JAUBj4+uR0qL2ioMUkRx66ABXf/KDvPT297No31Z+97Z3Y6dS011NhUKhOCfMSSGIRtcQDC5gWfNTPGo6+Hb3U+zL8sbPfJCnb3kfdXteYOtNbyL30kvTXVWFQqGYcuakEAghaGx8A4OJZ6h/VRgHya6fvYQQgjv+4UP8/F3/H/nEIPv+8DYGHnhguqurUCgUU8qcFAKAluZbAYe1C5/i9wGIvZQg3ZNB1wQf/fjb+NWHP8/uaCvtn/gkx97/fgrH26a7ygqFQjElzFkhCAbnU1PzCjo6fsiSN7j7DT373+6L0XRN8Nk7rubFj/wT31z9OvqffIqDr389Pf/1XzjZsd/1q1AoFDOROTd9tJLWlreyfccHWLdiP881hFjVnuWFbR2sX9uEpgn+8ZZ1fCke4t3/s4G/PvQwq/7ty/R9+zvUvPOdVL/9bejn4u1gCsUUIh0HLAtZCrb7Fntp226+bSMtG7zXQg47b9tI2/HOOeC95hEp3fNSgnOKLWyEcN/Dq2leWhs69oKbdt/5i/c+YXTdPWcY5WOh68Pj0jnDcNO67qbVliGjMqf2GhqJ4xT5/VMvJxZbz+KWL9H9hU1s9kve8OmXE6zYOfInW47zyR9v5+r8cT7a9RTyuWfQYjHib3wjVbe8mcDKlROui2J2IaVEFovIQgGZz5djp1BA5gvIYkW+V84pFCqOi0PXV8Yjg+XGlK6xLC+/IraKUBwy9NKy3PK27W5jPpfQNFcUTBNREo4KsRCGAaaB0L28sY4Ns3wsTMO7x/A8YZpuvmkOnTPN0c9V5A+VGwpuWR/C5+WNU9DUpnNjsP/AFzhy5Ou87GVPcuAnfURf7OFn6+N86K3rhpV77lAff/bdzWQKNv93nY8NTz1E8pFHkcUi/pUrid1wA5FXvQr/8mWq1XGeIS0LJ5dD5nIVcR6Zz43IzyML+RHnKvNGpMuGvZTOu0Y+l0MWCqev2JlSaRh8vpONxTDDYbhGrtL4VBorfYxjwzOImu6e84ykMAzQDYSulVvarjH18jTdvVbTy8doYqhVjzjpPU7uP4r3v9K7eB3HzZMOOEPv8sW23XcyO/ZQr8MTMGnZQ/mVvZLKno1lI23LTRet4WnHBqtU3nJ7QMME0xNQTzyHCWo5jJbnCu1UMf+urxN55SvHde2UCIEQoga4D1gEHAZuk1L2j1LuRuDLuPuq3S2l/JyX/7fAe4Bur+inpJQ/P93nTqYQZLNHeerpV7Fk8YdZ2Pg+Dv/TszxjF2i4fTWvXtU4rGznYI6/+MFWnjnYx5svbuWz18zHfuRhEg88SG7bNgDM1lZCV15B6JKNhDZegjlvnhKGMZCO4xrQbBYnk0Xmsm46m0VmszjZHE4u6xrqbM477+VlS4Y6653zjvM5r8xQHtbom9OdFiEQgQCa348IBBB+H5rPP5Tn87npgN9trQUCbp7PRPMHEKUyft9QedNXzhOmiebzjitDyeD7fENuDcWMQko55HIr99DcNFZFj23EuWE9vFKZQmFIoIpFYje9Ft/88e2LNlVC8HmgT0r5OSHEJ4FqKeUnRpTRgZeA64HjwPPA26SUuzwhSEkpv3A2nzuZQgCwdesdpDMHeNlVv6X/0WNkHjvGX5o5/umDV7CsMTqsrO1IvvLYPr7y6320xIN87pY1vGJZPcXOLlK/fZzU478ls2kTzuAgAHp1NYGVF+JfuRL/0mX4Fi7At3Ahek3NjBAIaVlDhjqbKRtqJ5PFyWZcg50pGfBRjjNDxn3kscyNYwW3aaL5/WjBICIYLBtp9zjgGuCAHy0QHB77A8PPB4MIvx8tEED4PWNeYfQ1v99tWc+AfyOF4kyZKiHYC1wjpWwXQjQDj0spV4wocyXwt1LKG7zjvwaQUv7L+SIEXd2/ZPv293PRRV+lvuoGTnz+efZm8vx9lcMDH3wZVSHfSddsPtLHX92/jQPdaW7bOI9P3bSyXE46Dvl9+8ls3kRu507yu/eQ37fPVXoPEQhgNDRgNNRj1Nahx+Po8ThaLIoWDKGFQmjBwFAL0TTdwTTN3eBNSul1peVQd9eyyy0KWfBaEuUw5NJw8p7Lw2s9D2tFZzI4uVy5VS7PsosrTBMRCqEFg+UgQkH3mSqPA975UBBRmfYH0EJeuYCbpwUCXss74H4PCoViXEyVEAxIKasqjvullNUjyvwhcKOU8k+94z8GLpdSftATgncCg8Am4KOjuZa86+4E7gRYsGDBJUeOHBl3vUcipc3Tz1yPacTZuPEnZDZ30n//Pv5B5EhfEONb77oUc5S963NFm688to+vP3GQWMDgL1+zgrddOh9jlLKyWKRw/DjFo0cpHDlCsb0Dq6sLq7MTq68PO5HATiTG78Y4EzTtpFavCJYMrb9snEUw4BnugHu+bMQrjkvGOhj0RMs7VoZaoThvGbcQCCEeBZpGOfVp4N4zEIJbgRtGCMFlUso/F0I0Aj24bdt/AJqllO8+3cNMdo8A4Hjb99i79zNcvOF7VMUvo+urW0kmctyU6eeGdS3821vWo4+xHfLu9kH+7n928szBPi5sivLXN63klcvqztqtIKX0fOPZYS3yUnBb/447oObuBe36sc3SNDpjaFaBYXi+aNd/rXk+aYVCMXcZ9xvKpJSvPsVNO4UQzRWuoa5Rih0HKkc25gEnvHt3VtzrG8BDp6vPVNHc9GYOHvwSR45+g+p1lxN/3RKKd2/na8tbec+Lbfh0jf/7h2tH3Rt/ZXOM77/nCh7e0cE//Xw3d9zzHJctquFjN6zgssU1Z1wHIYTrVgnNjHciKBSK2cFEVxY/CNzhpe8ARtuY53lgmRBisRDCB7zVuw5PPEr8AbBjgvUZN7oeYN682+nt/Q2p9D4CS6sIrKxh1ZEMf331Bfx4y3E+9dPtOGMskBFC8No1zTz20av5+5tXc6g3zW1ff5q3f+MZnnipm5k4TVehUMwNJioEnwOuF0Lsw50VVJoW2iKE+DmAlNICPgj8EtgN/FBKudO7/vNCiO1CiG3Aq4CPTLA+E2Je6x+haQGOHr0bgPhNi5GWw60pjQ++aik/eP4YH77vBfLW6K+ABPAbOrdfuYgnPv4qPn3TSg50p7j9nud4/Vd/x8+2tp3yWoVCoZgO5vyCspHs3fu3tJ34AVdd+RsCgWYSvzhE8rfHqXvfWr51uIf/8/AeLltUw123XzLqbKKR5C2bB7ae4OtPHOBAd5rasI+3Xjaft122gHnVygWkUCjOHWpl8RmSzbbx9DPX0dx8Cysv/CecvE3nFzehhUwa/nwD/7O9nY/98EXm1QS5+/aNLKk/s/2GHEfy+wM9fPvpIzy2uxMJXHVBLX94yTxuWN1EyDent31SKBTnACUEZ4HbK/g+V1z+K0KhhWS2ddP3vT1U3XwBkStbeO5QH+/9ziYKlsM/v3kNN69vPav7H+/P8JMtbdy/+ThH+zIETZ3rVjbwhnUtXL28noCpVpIqFIrJRwnBWZDPd/HU06+iof5GVq/+V6SU9HxzB4XjSZo+thE94qM9keXPv7eVTUf6edtl8/mb168etlHdmSCl5PnD/TzwQhu/2NFBX7pA2Kdz9Yp6rl/VyLUrGomH1JRPhUIxOSghOEv27/8/HDn6DS6/7H+JRFZQ7MrQ+eUtBC+qo/ZtFwJg2Q5ffOQl/uPxAyyqDfG5W9ZyxZLacX2eZTs8daCXh3d28OiuTrqSeTQBFy+o5url9Vy9op7VLfEx1zIoFArF6VBCcJYUi/38/qlrqKm5irVr/hOAwUePMPjoUWrfuZrghUPrA5460MNf/2Q7R3ozvP3yBXzytRcSC4y/Je84khePD/DrPV08vreb7W0JAOJBkyuX1PKypbVcvqSWpfWRUdc1KBQKxWgoIRgHBw99hUOHvszGS+4nHt+AtBw6v7oVmbNp/MuL0fxDA7zZgs0XH9nLN393iOqQj798zXLesnH07SbOlp5Unt/t6+H3+3t46kAvbQPuW9KqQyYbF9WwcWE1lyys5qLWuBpfUCgUY6KEYBxYVoqnn7kev7+RSzf+BCE08kcG6f6vF4lc2ULVGy846ZodbQn+/qFdPHfI3W7iE6+9kGuW10/aLpZSSo71ZXn2UC/PHerjucN9HOnNAGDqgpXNMdbPr2LdvCrWzItzQX1EuZMUCgWghGDcdHQ8wM5df8mFF/4zrS1vAWDgwQOknj5B/fvW4V8YO+kaKSUP7+jgX36xh6N9GdbPr+Ij1y8f1/5DZ0JPKs+WI/1sOTrAC8f62X48QbrgLlwLmjqrWmKsbomxqjnGqpYYyxujquegUEwBRdshV7TJFR3y1lCct9z8vOWQLw4/zhVt8kWHnOXGJ52rvNZy+Mc3reaShWe+dU0lSgjGiZSSzVveSiZzkCuveBTTjOPkLTr/bQtCEzT8xcVoY8wWKlgOP95ynH//9X7aBrKsnRfn3S9bzE1rmvEZE3cZjYXtSPZ3pdjRlmB7W4IdbQl2tw+WxUETsLA2zIrGKMsbIyxrjLKsMcKi2rASCMWswXEkOcsmW7DJWY4bF+2ygS4Z7FzRJleRly+65d3zQ4Z4qOwYxtxysE/1jubToAkImDoBU8dvaOXYb2j4K/I+dO0y1syLj+szlBBMgGRyN889/0bmzfsjViz/WwByBwbo+cZ2wlc2U33z0lNeX7AcfrT5GN/83SEOdqdpiPp522ULuOXieSyoPTerix1Hcqw/w64Tg+zpSLK3I8neziRHetPl94trAubXhLigPsKSujCL68MsrnXjxmhADUwrJhXHkWSKNpm8Rbpgk85bZAo2mUIptsl66WzRNejldNEmV3FcMtqlcjnLoWCN/33MAVPDb+gETNf4Brx0pUEeMtgaAUPH78UBcyjt9+4z/Hjk9W482lb3k40SggmyZ+9naWv7Hpdd+gDR6CoABv7nAKnfn6DuTy8isLT6NHdw//Cf2NfNPb8/zJP7upESLltcwx9saOX6VY3URfxT/RgnkSvaHOpJ81JnkgPdaQ50pzjQleJQT5p8xQ/Jb2gsqAmxsDbEgpowC2tDzK8JMr86xLzq0FmvoVDMbKSUpAs2iWyRRKbIQLbAYNZiMFtkMFf0YotkziKZK5LKW27IWeV0pnB2+24FTI2gqRPyGWUDHfLpZaMa8g0Z7KDPIGhWGHJzyPi6hl0jWLp2hJH3G9qsfTOdEoIJUiwO8MyzN+DzNXDpxp+gaSayaNP5la3IgkPjRy5GC5z5NhEnBrL8dKu7uvhQTxpNwMaFNVy/qpFXLq9neWNkWv8YHUfSMZjjUE+aw71pjvRmONST5mhvhqN9GbLF4T/iuoif1uog86qDzKsK0lIVpNWLW6oCxIPqtY/nG7YjSeaKDGYtEp4BT2RdI54YJZTyB3OuwbdO4QYRAiJ+g1jAJBowiAYMIn6DsH8odtOuYQ/7dcI+g5DPIOjTCft1QqabDvl0gqaueqSTgBKCSaC7+1ds2/5nLF78YZYs/nMACseSdP3nCwTX1lPzlhXjehnNrvZBfrmzk1/u6GBvZxKAhqifly2tY+Oiai5dVHNerRmQUtKdynOsL8OxvizH+924bcBNnxjIUbCHd8uDpk5zVYCWeJCmeICWeIDGeICmWICmeIDGWICakO+8ecaZgO1IUjnLbYF7Bj2ZGzLUgyOM/GCFER/MFknmT/02PF0TxIMm8aBJzIvdYAzlB8yTy4RMIj5D/VuehyghmCR27PwIXV0/59KNPyMaXQnA4GNHGXzkCNW3Lid8SeOE7t82kOV3+7p5cl8PzxzspSdVACAWMLioNc6a1jirWmJc2BRjcV14Sgedx4vjSHrSeU4M5Gjrz9KeyHJiIEd7Ikt7IkdHIkdXMsfIBqWpCxqiARpjfhpjARqifhoq4vqIn4aYn+qQb8ZPibUdSSo/5DYpuVCSZXdK5XHJzeIelwx66jSGHCDqN4gF3VZ5yVBHA0bZiMfKBt04yeCHfLrqxc0ylBBMEsViP08/c0N5bYGmmUhH0v2NbRTb0jR+aANGXXBSPktKyZHeDM8f7mPrsQF2tCXY054st7YNTbCoLsyi2jCL60IsqgszrzpEa1WA1qrz229v2Q7dqTwdnjB0DuboGMzTOeiKRKeXTuZONna6JqgJ+6iP+KmL+qmLuOnaiI+6iJ/aiJ/asI/aiI+asA+/MTnfg2U75UHNdMUAZ8o7TlWmcxapvE0qXyz7xpMVPvIz8Y8bmvDcKiaxoEHUb5aPS4Y9FjDKBj0WNMot9FjAJBIwZrxgKiYXJQSTSFf3L9m+/f0sXPg+ll7wcQCsgTydX96CURug4X3rEFPUUi9YDvu7UuzrSvJSZ5KXOlMc8Xz4+RGzJGIBg8aY63apj5aMo5/qkElVyEd1yO3GRwOuQQmfh935bMGmO5mnK5mjO5mnO5WnazBPT8oNpbyeVGHMWSJBUycacH3SpQFEv6Fj6gJdE2heq9cBbNudbZK1HDIVs1jSBfuMZ6Fonn88GjAJ+3UifoNIwCTq+cejAYOI5zMv+dAjnoF389x0wJy9g5aK6WHc7yxWnExD/Q20NN/GkSP/RXXVZdTWXo1R5afmlmX0fnc3iYcPU/X6JVPy2T5DY1WLuzCsktLgbttAlrZ+11/fOTjU0j7cm6YnlSdXPLUxC/l0wn6jPEAXqDCcfkPDNDR8uoahCQwv1rWSQcU1qgIEQwZMIvH+w3EktpTl2HbcYNkSy5FYjkPRlli2Gxdth6LtULAdipak4BnqUpy3XAN9qunbpemGJPNn/D0bmsDnTQ0M+XTqo/6yca8KmlSHXSGti/ppiPrLghtTBlwxA1E9gnFi2zk2bXoz+UI3l132PwT8TQD0P7Cf9NPt1L5jJcGL6qa1jqORKVj0Z4r0pwsMZIYGEZO5IbdGumCR9eZnZwreakfbXURTsByKjoNlS4q2xJGu0bYd6Rp6KYcbZYknDO5MEl24LXBNExiaG+tCYOgCU9fQvXxT1zB0gU/X8Bla2TD7vJa83xMkN29o/nbJeJcX41RMD6ycdhgwNaSEZM6iP1OgL12gP1OgP11kIFOgL1OgP+NOjezPuN/VQKZQXpQ3GkJALGBSFRo+kDp8oLWUN9xPHwsYk7IvlUJxKqbENSSEqAHuAxYBh4HbpJT9o5S7B3g90CWlvOhsrx/J+SAEAOn0AZ7f9CaikdVs2PBdNM1AWg5dX9+G1ZWh4c83YE7SeIHi/CBv2d5MnCGhGMi6IlGaXjkwxrTLon3q31rYp3ui4I0JeC67kXnRwJBLqTwQHDBn9fx3xeQwVULweaBPSvk5IcQngWop5SdGKfdKIAV8e4QQnNH1IzlfhACG9iJaMP9PWLbsUwBY/Tm6vroVPean4QPrEGrbhjmPlJJs0R4+Z7+iR5bIWhVpb3ZQbiidzBVP6f4Cd9ZVSTyiFfP3h9LuuZKLqzQ2EasYn1AzhWY3UzVGcDNwjZe+F3gcOMmQSymfEEIsGu/15zNNTTeTGHyBo8e+STi8lJaW2zCqA9S8ZQU9/28n/T/dT/Wty9WPa44jhCDkLZhqigfO+vrSSt7SArDBXPGk6aQl8aicetrTky5PRz2T6aaVA92VC8Gi3iykqH8oL+KJx9Cxez4ScAfl1d/8zGGiQtAopWwHkFK2CyEapup6IcSdwJ0ACxYsGG99p4RlSz9NJnOIPXs/QzC4gOrqKwisqCF23QIGHz2K2RIh+vKze6+xQlGJEMI1tn6D5vHtN4bjSFIFTxQq1ynkh9Kl6a2DFcc9qQKHetLl9Q4jZ6eNhq6JYSIRG6OHUsorjZNUngur3sk547RCIIR4FGga5dSnJ786YyOlvAu4C1zX0Ln87NOhaQZrLvoqz2/6Q7Zt/wCXbvwxodAiotcuoHAiTeLnBzGbQme0H5FCMVVomnDHGibw9jxwpzCn80M9kNIailKvY9j+QjmrvBiuYzDHvq6hBXKn26lzZO/kdGLiio45THzCfl0Nwp8BpxUCKeWrxzonhOgUQjR7rflmoOssP3+i1583GEaUdWvvYtPmW9j6wjvZeMl9+P2N1LxlOV3/8SJ939tDwwfWY9SqwWPFzMadqeWjOuwb9z1KYybDV05bZddXcoSLqyQm7YkcL3UNrcA+k22fS+tIKl1Xbu/KJOLXveOhdNg35Oqq3BspNIv3O5qoa+hB4A7gc178wDm+/rwiFFrI+nX3sGXrO9j6wh1ccvH3Mf3V1P3xKjq/9gI99+6i4f3rzmpzOoViNlI5ZtIYO/sxE3DFJFd0SOaHhGE0l1epxzKYG1r13ZPMlFeCn8lAfImwt84m4jcIeRvluWmjfC7s08vHpQ31SnHQ9GLv3PkiLhOdNVQL/BBYABwFbpVS9gkhWoC7pZQ3eeW+jzsoXAd0Ap+VUn5zrOtP97nn06yh0ejrf5oXX3w3kfCFbNjwHQwjQm7/AD337MB/QZy6d65GqO6qQnFeUBKU1MjtQnLumpqh9Ojbi2QKFpm8Xd46JF2wOBuzWlrjEvS20g75hoQi6C3sLOUFTZ2b17eyuC48rmdVW0ycY7p7HmP79j8jFtvA+nV3YxhR0ps66L9/H+HLmqj6g6VqIEyhmIWU3F7pvPuSnHTBqnjpjvcCnqL70p10vrRw03sBT8Em7b2Qp7SgM1vxYp685fDtd1/GK5fXj6tuaouJc0x93XWsXv1v7Nz5EbZuvZ31679FeGMTVm+O5G+OYdQGiF49f7qrqVAoJplKt9dkM5FXYZ4K5Z+YQhobbmLNRV8jmdrDlq3voFDoJXb9QoLr6kn84jDpLZ3TXUWFQjGDKO3rNdkoIZhi6utfzbq1XyeTOcimzbeRzR2h5tbl+C+I03//PrJ7TzskolAoFFOKEoJzQG3tK9mw/ttYVoJNm/+QRHortX+8CrMpRN93d5M/OjjdVVQoFHMYJQTniKqqjWy85EcYRoytW99BV+IX1L3rIrSYj55v7aTQnp7uKioUijmKEoJzSCi0mI2X3E80upadOz/MgY7PU/uuC9F8Gj13b6fYlZnuKioUijmIEoJzjM9Xw8UbvsO8ebdz7Ni32HbkT4jd3gICuu/ejtWbne4qKhSKOYYSgmlA03ysWP5ZVq/6EsnkLjYffDP6LSmwHLq/ocRAoVCcW5QQTCNNTW/k0kt/it/fxI7295K89nGcvEX3XduwepQYKBSKc4MSgmkmEl7GpRt/zMKF7+N44Zu0XfYV7EKe7ru2UexWYwYKhWLqUUJwHqBpPpZe8HEuufgH2NU9HFr/GYr5Qbq//gLFDjWbSKFQTC1KCM4jqqo2ctml/0vrmj/g6MZ/plDso+M/nyd35LSvcVYoFIpxo4TgPEPX/SxZ8mE2XPsNEq9+mKLWS9c3NtO55XFm4gaBCoXi/EcJwXlKOLyUtS//KtE7qrGDCfL3O+x48NP09z873VVTKBSzDCUE5zFCCOoXXcv8v7gRrdGh6ukbOPjAN9my9Y/p739G9RAUCsWkoIRgBmBEwrS+/1UEVtbQuOeP8W1aypYt72DT5j+ku/tXSGlPdxUVCsUMRgnBDEGYOnV/fBHhK5qpOnAtKw59jWI2wbbtf8bTT7+ao0e/SbGoNq9TKGYTUkocJ49lJSkU+sjlO7Dt/KR/jnpD2QxDSknqyTYSvziEOS+CfG07x/vvJZHYhKYFaWx8PS0ttxKPXazegKZQTBApbWw7h+PkcJy8l866aSePY+dwnMLQeSePU87P48iCd5x3y8mCVz4/Ii4gS8flMgWkLJ5Up/XrvkVt7SvH9TzqDWWzBCEE0VfOw6gJ0HffXrQf1rH2jnvILz/M8ePfpbPrIdrbf0QotJTmpj+gsfENBIOt011thWLSkdLBtjPYdhbbTruxk8G2MjhO1svPYjtZHDvrlnVy2LZ37OTKsW1ncZxc2dDbniEfzRCfOQJNC6BpfjTNVxGX0n4MIzqUL9xzolzGhybMoTwvHQ4vnbTvsFzTCb68vga4D1gEHAZuk1KeNOldCHEP8HqgS0p5UUX+3wLvAbq9rE9JKX9+us+dyz2CSgrHk/R8excya1F92wpCa+qwrBRdXT/nRPuPSCS2ABCPb6Sh4Ubq616jREExbbhujiyWlXKDncS2Ulh2yovTFXHazbcz2FbaM/QZLDtTTjtO7ixroKHrIXQ9gKYF0fUguhZA0wPoetDNKx2PiDXNS2t+NN3vpQOewQ549/QPC0IY512vfEpeXi+E+DzQJ6X8nBDik0C1lPITo5R7JZACvj2KEKSklF84m89VQjCEPVig97u7KBxNEr12PrFXL0R4r7LLZo/R0fkgnZ3/Qzq9D4BoZDW1da+itvaVxKLr0DTVKVScOY5ToGgNYhUTWFaCYjGBZQ1StBJYxUTFuUGK1iC2lXLzrCS2nTyjiQ1CGOh6GEMPoxvhobQecoPhpbUQuhFC18PoWtA7Pzx2jbRr/IXwnXeG+VwzVUKwF7hGStkuhGgGHpdSrhij7CLgISUEk48sOvT/bD+ZzZ0ELqyh5rblaCFzWJlM5hDd3b+iu/sREoMvAg6GEaO66nKqqi+nuupyIpEVCKFPz0MozhmOU/SM+KAbWwmsYqUxr4itQYrFAS9O4Din3gxR10MYRqwcTCOGYUS94wiGEUU3ohh6BMOIVKRdg6/rEdcVMscN9lQxVUIwIKWsqjjul1JWj1F2EaMLwTuBQWAT8NHRXEte2TuBOwEWLFhwyZEjR8Zd79mIlJL0M+0MPHQQPe6n9h0r8bVERi1bLA7Q1/d7evueZKD/WbK5owDoeoRYbC3x2HpisbVEIqsIBFrUj/I8xLbzWNbohrzUGq805pVp2z71ZoYlY24acQyzyjXmZtw9rkybXhkjjmm6hl/TfOfoG1CMh3ELgRDiUaBplFOfBu6doBA0Aj2ABP4BaJZSvvt0D6N6BGOTPzJI73/vxslYVL/pAkKXNJ7WkOdyJ+gfeI5EYiuDg1tJpfaUu/CGUUUksoJweBnh8FLCoSWEQovx+5sQQs0+Hi+2nceyk1jFQc+gJ714KF0s55Va70PBcU49hVDXw64xN11DbRhRz3jHKwx7lTLmc4zz0jV0NucrUUJwauxkgb4f7CF/IEFoQwNVb1qK5j9zl49tZ0ml9pBM7iKZ2kk69RKp9D5sO1Uuo2l+AoH5BIOtBAKtBPwt+P2N+P2N+PwN+H11GEZ8VonF0CyVNJY3gOkOdHppK4VtlwZCvQFQK4lle3E5pJCycMrPEkL33ClR10iX3C1mpcul1CqPeYY8XnbJaJp5yvsr5iZTNX30QeAO4HNe/MBZVqpZStnuHf4BsGOC9VEAetRH3Z+sIfnroww+dpTC8SQ1b7twTFfRSdfrQeLxDcTjG8p5Ukry+Q4ymYNkskfIZg6TzR0jl2sjkXgRyxo46T5C6JhmNYZRhVl2LUTRjQiGHhka0NOD6OXZFj6EMBGaiSYMEDpCaAj0ClERuJ1IkEiQEomNlA5IG0daSGkhHTcuzccuzc12SnO9vTnc7vTC3ElTC4emJma8WSpn+rIgzTPg3nMaEXy+OkKhxZ4/vGTgoyenzRiGHkXXQ8olpzhnTLRHUAv8EFgAHAVulVL2CSFagLullDd55b4PXAPUAZ3AZ6WU3xRCfAdYj/urPgy8t0IYxkT1CM6c3IEB+n6wFydTJH7jYiIvaynPKppMbDtDPt9JPt9FPt9JsdhHodBLodjr+a8HKBYT5emClpWc4BztyUDzpv0FTp5GqI+cheLNTtGD3iwWT8iMiDejJVI2/poWUEZccV4yJa6h6UIJwdlhp4v03/8Sud19+JdVUXPrcvSYf7qrheMUvUU8mYrWeW6oJS+LIB2ktL0xC1nuASAEAoHbO3B7H0LoIDQ0YXpzuPVyD0PTzPIiHfc4oKbOKuYcamXxHEYPm9Tevor0cx0kHjpIx5e2UH3zBQTX1U9ry9U1ziaGEZ22OigUCrXp3JxBCEHk8mYaPrQBsz5I3w/20ve9PdipUw9aKhSK2Y8SgjmGWR+i/n3riN24iOyuXjq/tJnMC13q3QYKxRxGCcEcRGiC2DXzafzQBowat3fQe+8urIHJ395WoVCc/yghmMOYjWHq/2wd8dctIX9ggM4vbib5uzakrXoHCsVcQgnBHEdogugrWmn8yCX4F8dIPHSQrq9tpXAsOd1VUygU5wglBAoAjJoAte9cTc3bL8ROFun6jxfo/8k+7PR0z/VXKBRTjZo+qigjhCC0tp7A8moGHz1K6qk2sjt6iL1mIeFLmxG6WiSlUMxGVI9AcRJawKDq9Uto/NDFmE1hBn52gM6vbCG3f9SNYRUKxQxHCYFiTMymMHXvWUPNH61EFmx67t5Bz7d3Uew+9TbGCoViZqFcQ4pTIoQgtKaO4IU1JH/XRvLxY3R+aQvhy5uIXbcAPaK2LFYoZjpKCBRnhDA1Yq+aT3hjI4OPHSX9bDuZLV1EXzmPyMtbz2qba4VCcX6hhEBxVuhRH9VvWkrkqhYSDx9m8JEjpJ4+Qey6BYQvbUIYytuoUMw0lBAoxoXZEKLu9lXkjwySePgQAw8cIPnb48RevYDQhkY1w0ihmEGo5ptiQvgXxqi/cy11774ILWzSf/8+d/+irV1IR61QVihmAqpHoJgwQggCy6vxL6sit7OXxCNH6LtvL8avjxK7bgHBtfVT8jIchUIxOSghUEwaQgiCF9URWFVLdmcPg48epe8HezEeO0r0mvmE1jcol5FCcR6ihEAx6QhNEFpTT3B1HdmdPSR/fYz+H73E4GNHiV49j/AljWpQWaE4j5jQr1EIUSOEeEQIsc+Lq0cpM18I8RshxG4hxE4hxF+czfWKmUtJEBo+tIHa21ehhQwGfrqf9s8/T/LJ4zh5a7qrqFAomPhg8SeBx6SUy4DHvOORWMBHpZQrgSuADwghVp3F9YoZjhCC4KpaGj6wnro/uQizLkjifw/R/rnnSfzysHpLmkIxzUzo5fVCiL3ANVLKdiFEM/C4lHLFaa55APh3KeUj47ke1MvrZwP5o4Okfnuc7K5e0DXClzQQeXkrZn1ouqumUMxapurl9Y1SynYAz5g3nKYSi4ANwLPjuV4xe/AviOH/41UUuzOknmwjvbmT9HMdBFbWEn15K77FMYRQA8sKxbngtEIghHgUaBrl1KfP5oOEEBHgx8CHpZSDZ3Otd/2dwJ0ACxYsONvLFecpZn2I6jcvI3b9QlJPnyD9TDvdu3oxWyNEXtZCaG29GlhWKKaYc+IaEkKYwEPAL6WUXzzb60eiXEOzF6dgk9naRer3bVhdWbSISfiyJiJXNKPH/NNdPYViRjOWa2iiTa0HgTu89B3AA6N8sAC+CeyuFIEzvV4xt9B8OpHLm2n88CXUvfsifPOiJH9zjPbPPU/vf+8md2CAiTReFArFyUy0R1AL/BBYABwFbpVS9gkhWoC7pZQ3CSFeDjwJbAcc79JPSSl/Ptb1p/tc1SOYW1i9WVLPtJPZ3ImTsTDqg4Qvbya0oQE9bE539RSKGcNYPYIJCcF0oYRgbiKLNpltPaSfbadwNAmGILi6jvBlTfgXx9U2FgrFaZiqWUMKxTlDmDrhSxoJX9JIoT1N+rl2Mlu7yL7YjV4TILyxkdAljRhxNZagUJwNqkegmNE4BZvszl4yz3eQP5gAAf6lVYQvaSS4uhZhqhfmKBQlVI9AMSvRfDrhDQ2ENzRg9WZJb+kis7mTvh/sRfh1gmvqCF/cgG+Rch0pFGOhhEAxazBqg8SvX0jsugXkDyZct9G2HjKbOtGr/ITW1xNa34DZFJ7uqioU5xVKCBSzDqEJAkurCCytwrn5AnK7esls7SL5xHGSjx/HbAoTXFdPaF09Rk1guqurUEw7SggUsxrNpxNa30BofQN2quD2EF7oYvCXhxn85WF886ME19YTXFunBpkVcxYlBIo5gx7xEbmqhchVLVh9ObLbu8m80E3ifw+S+N+D+BbGCK6pI7hGiYJibqFmDSnmPMXuDNltPWS3d1PsyADgWxAleFEdwYvqlPtIMWtQC8oUijOg2J0hu72H7PYeiu1pAMzWCMHVtQQvqsNsUNtkK2YuSggUirPE6s2S3dFLdkcPhWNJAIz6IMFVtQRW1+KbF1VTUhUzCrWOQKE4S4zaINGr5xG9eh52Ik92Vy/Znb0kn2wj+dvjaFGT4MpaAqtqCVwQV4vXFDMWJQQKxRmgx/1ErmwhcmULTtYit7fPXdH8Qjfp5zoQpoZ/WTXBlTUELqxBj/qmu8oKxRmjhEChOEu0oFGekioth/zBBNldveR295Hb1QuAOS9C8MIaAitrMVvC6m1rivMaJQQKxQQQhkZgeTWB5dXImyXF9rQrCHv7GHzsKIOPHkWL+gisqCZ4YQ3+pVVoAfWzU5xfqL9IhWKSEELga4nga4kQu24BdqpAbm8/uT19ZHe4W12gCfyLYgRWVBNYUYPRGFK9BcW0o4RAoZgi9IivvG22tB0KRwZdYdjbT+IXh0n84jB6zId/ebUrDEur0YLqJ6k496i/OoXiHCB0Df+SKvxLqoi/djFWIk/+pX530Hl7qbcAvvkxAsuq8C+vVtNTFecMJQQKxTRgxP0YlzYRvrTJ7S0cS5J7qZ/cS/0MPnYUHj2KCBoEllbhX1ZFYGm1WuGsmDKUECgU04zQNfyL4vgXxYm/ZhF2ukh+/wC5ff3kX+onu70HAKM2gH9ZtSsOF1QpN5Ji0pjQX5IQoga4D1gEHAZuk1L2jygzH/g20IT78vq7pJRf9s79LfAeoNsr/ikp5c8nUieFYqajh01C3jbZUkqs7qwrCvsGyGzpJP1MOwgw50VdUVhahX9BDGFq0111xQxlQltMCCE+D/RJKT8nhPgkUC2l/MSIMs1As5RyixAiCmwG3iSl3OUJQUpK+YWz+Vy1xYRiriItz420r5/8gQSFY4Nu88rQ8C+K4V9aReCCKszWiBpfUJzEVG0xcTNwjZe+F3gcGCYEUsp2oN1LJ4UQu4FWYNcEP1uhmHMIQ8O/OI5/cRwAJ2eRP5RwXUn7Bxh8+DCDgAjobrkL3Bf0GA0hJQyKMZmoEDR6hh4pZbsQouFUhYUQi4ANwLMV2R8UQtwObAI+OtK1VHHtncCdAAsWLJhgtRWK2YEWMAiurCW4shYAO1kgf3CA/IEEuf0D5Hb3kQC0sIl/SRz/BXH8S6ow6oNq/YKizGldQ0KIR3H9+yP5NHCvlLKqomy/lLJ6jPtEgN8C/ySl/ImX1wj0ABL4B1wX0rtPV2nlGlIozgyrP0f+QIL8gQHyBwewEwUAtKjPFQYvGHVKGOYC43YNSSlffYqbdgohmr3eQDPQNUY5E/gx8N8lEfDu3VlR5hvAQ6erj0KhOHOM6gDGxgDhjY1IKbF7c+QODpA/mCB/IEH2RXeexjBhWBxXPYY5xkRdQw8CdwCf8+IHRhYQ7l/TN4HdUsovjjjXXHItAX8A7JhgfRQKxRgIITDqgkTqgkQua3ZnJPVkXVE4NEIYIqY7xlASBjXGMKuZ6KyhWuCHwALgKHCrlLJPCNEC3C2lvEkI8XLgSWA77vwG8KaJCiG+A6zHdQ0dBt5bIQxjolxDCsXkI6XE6s1ROJhwxxkOJYZcSSED36J4WRzM5rAShhmIekOZQqE4K6SU2P35oR7D4QR2bw4A4dfxLYx5M5hi7nYYhlrHcL6j3lCmUCjOCiEERk0Ao8YdYwCwE3lPFAbJH0ow+MvDbmFD4JsXxb84jm9RDP+CmFr5PINQ/1IKheKM0eP+8kt5AOx0kcLhQfKHXXFI/vY4/EaCAKMhhH9hDN/CGL4FUTUz6TxGCYFCoRg3etgkuLqW4Gp3HYNTsCkcS7ricGSQzDb3VZ7gjTMsiOGbH8W3IIpvXlT1Gs4T1L+CQqGYNDSfTuACd5sLAOlIrK4M+aODFI4kKRwdJLenr1zeqA/imxfFNz+KOS+Crzmi9kyaBpQQKBSKKUNoArMpjNkUhsuaAXCyFoXjSQpHkxSOu/smZbZ6S5A0gdkYwmyN4JsXwdcaxWwKK3GYYpQQKBSKc4oWNAgsqyawzN2EQEqJnShQPJ6k0JZyxWFXr/uyHgANzIYQZkvEDc1hfC0R5VaaRNQ3qVAophUhBEaVH6PKT/CiOsATh4E8xbYUhbYUxRMpt+ewZWjzAr3Kj9kc9oIrEEZNQK1vGAdKCBQKxXmHEMLdHqM6UBYHcDfVK55IUTiRptieotiedsccvOVQwtQwGkOuO6oxjNnkprWIqWYsnQIlBAqFYsagR33oK2oIrKgp58miTbEzQ7EjTbE9TbEjTW5335BrCXfGktEYcl1MjeFyWgmEixIChUIxoxGm7s48mhcdlm+nCq44dGawOjMUOzNkXuxG5jqGrg0amA0hjPqgGzeEMOuD6NVzy8WkhEChUMxK9IgPfamPwNKhnfGllDjJgtuD6MpgdbnxyB4EurtBn1kXxKh3hcKoc4MeNqfhaaYWJQQKhWLOIIRAj/nRY/7yrKUSdrqI1Z3B6s5S7M5idbu9iOzuPnCG9mQTQWNIJGoDrkDUumktNDNFQgmBQqFQ4K6S1sNx/Iviw/Kl7WD15bB6slg9OayeDFZvjvyhBJkXusoD1eCOReg1gbIwuHs1BdFrA+hR33nrblJCoFAoFKdA6BpmfQizPnTSOVm0PZHIYfVm3dCXo3AsSXZ799DG+wCGNxOqJoBeHShv6KdXBzCq/YigMW0D10oIFAqFYpwIU3enqTaGTzonbQd7II/Vm3PFoi+H3ecKRf5IEpmzht/Lr2NU+z1hCKBX+dGr/eW0Fp66GU5KCBQKhWIKELrmuYiCo553spYrDv05rP4cdn++HOcPJpB5e/j9TA29yk/Vm5aW93KaLJQQKBQKxTSgBQ18rRFojZx0TkqJzFpYA3lXIAZy2AN57IH8lMxaUkKgUCgU5xlCCETIxBcyoeVkoZhs1JZ+CoVCMceZkBAIIWqEEI8IIfZ5cfUoZQJCiOeEEC8KIXYKIf7ubK5XKBQKxdQy0R7BJ4HHpJTLgMe845HkgWullOuA9cCNQogrzuJ6hUKhUEwhExWCm4F7vfS9wJtGFpAuKe/Q9EJpCcZpr1coFArF1DJRIWiUUrYDeHHDaIWEELoQ4gWgC3hESvns2Vzv3eNOIcQmIcSm7u7uCVZboVAoFCVOO2tICPEo0DTKqU+f6YdIKW1gvRCiCvipEOIiKeWOM66le4+7gLsANm7cKE9TXKFQKBRnyGmFQEr56rHOCSE6hRDNUsp2IUQzbov/VPcaEEI8DtwI7ADO6nqFQqFQTD4TdQ09CNzhpe8AHhhZQAhR7/UEEEIEgVcDe870eoVCoVBMLULK8XtZhBC1wA+BBcBR4FYpZZ8QogW4W0p5kxBiLe5AsI4rPD+UUv79qa4/g8/tBo6Ms9p1QM84r53JzMXnnovPDHPzuefiM8PZP/dCKWX9yMwJCcFMRAixSUq5cbrrca6Zi889F58Z5uZzz8Vnhsl7brWyWKFQKOY4SggUCoVijjMXheCu6a7ANDEXn3suPjPMzeeei88Mk/Tcc26MQKFQKBTDmYs9AoVCoVBUoIRAoVAo5jhzSgiEEDcKIfYKIfYLIWblTqdCiPlCiN8IIXZ7237/hZc/67f89va02iqEeMg7ngvPXCWEuF8Iscf7N79ytj+3EOIj3t/2DiHE972t7mfdMwsh7hFCdAkhdlTkjfmcQoi/9mzbXiHEDWfzWXNGCIQQOvA14LXAKuBtQohV01urKcECPiqlXAlcAXzAe865sOX3XwC7K47nwjN/GXhYSnkhsA73+WftcwshWoEPARullBfhLlR9K7Pzmf8f7nY8lYz6nN5v/K3Aau+a//Bs3hkxZ4QAuAzYL6U8KKUsAD/A3QZ7ViGlbJdSbvHSSVzD0Mos3/JbCDEPeB1wd0X2bH/mGPBK4JsAUsqClHKAWf7cuHukBYUQBhACTjALn1lK+QQwcqeFsZ7zZuAHUsq8lPIQsB/X5p0Rc0kIWoFjFcfHvbxZixBiEbABeJaz2PJ7hvJvwF8BTkXebH/mJUA38C3PJXa3ECLMLH5uKWUb8AXcLWnagYSU8lfM4mcewVjPOSH7NpeEQIySN2vnzgohIsCPgQ9LKQenuz5TiRDi9UCXlHLzdNflHGMAFwP/KaXcAKSZHS6RMfF84jcDi4EWICyEeMf01uq8YEL2bS4JwXFgfsXxPNwu5axDCGHiisB/Syl/4mV3elt9Mwu3/H4Z8EYhxGFcl9+1QojvMrufGdy/6eMVL3q6H1cYZvNzvxo4JKXsllIWgZ8AVzG7n7mSsZ5zQvZtLgnB88AyIcRiIYQPd2DlwWmu06QjhBC4PuPdUsovVpyatVt+Syn/Wko5T0q5CPff9ddSyncwi58ZQErZARwTQqzwsq4DdjG7n/socIUQIuT9rV+HOw42m5+5krGe80HgrUIIvxBiMbAMeO6M7yqlnDMBuAl4CTgAfHq66zNFz/hy3C7hNuAFL9wE1OLOMtjnxTXTXdcpev5rgIe89Kx/ZmA9sMn79/4ZUD3bnxv4O9x3muwAvgP4Z+MzA9/HHQcp4rb4/+RUz4n71sgDwF7gtWfzWWqLCYVCoZjjzCXXkEKhUChGQQmBQqFQzHGUECgUCsUcRwmBQqFQzHGUECgUCsUcRwmBQqFQzHGUECgUCsUc5/8HPoFsvc72laYAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "tau = 1 / np.linalg.norm(A) ** 2\n", "niter = 100\n", "x = np.zeros(p)\n", "x_ista = np.zeros((p, niter))\n", "for it in range(niter):\n", " x = Thresh(x - tau*A.T @ (A@x - y), tau*la)\n", " x_ista[:, it] = x\n", "plt.plot(x_ista.T);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Non-convex Variational Projection\n", "\n", "The following variational formulation is pivotal for the developpement of our approach\n", "$$\n", " \\|x\\|_1 = \\min_{u \\odot v = x} \\frac{\\|u\\|_2^2 + \\|v\\|_2^2 }{2}\n", "$$\n", "where $u \\odot v = (u_i v_i)_{i=1}^p$.\n", "\n", "Thanks to this formula, we can minimize the Lasso problem on $(u,v)$ instead of $x$ (which corresponds to an over-parameterization of the problem). The crux of the method is that instead of doing an alternative minimization over $u$ and $v$ (which is a classical method), we rather consider a bilevel formulation\n", "\n", "$$\n", " \\min_{v} f(v) \n", " \\quad\n", " \\text{where}\n", " \\quad\n", " f(v) = \\min_{u} F(u,v) \\triangleq \\frac{1}{2 \\lambda}\\| A(u \\odot v) - y \\|^2\n", " + \\frac{\\|u\\|_2^2 + \\|v\\|_2^2 }{2} \n", "$$\n", "The effect of this bilevel formulation is to make the function $f$ better conditionned (and in particular smooth) than the original Lasso functional. \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For a given $v$, the optimal $u$ solving the inner problem $\\min_u F(u,v)$ is unique and can be found by solving a linear system\n", "\n", "$$\n", " u^\\star(v) = ( \\text{diag}(v) A^\\top A \\text{diag}(v) + \\lambda \\text{Id}_p )^{-1} ( v \\odot A^\\top y ) \n", "$$" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def u_opt1(v): \n", " T = np.diag(v) @ A.T @A @np.diag(v) + la * np.eye(p)\n", " return np.linalg.solve(T, v * (A.T@y))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using [Sherman–Morrison](https://en.wikipedia.org/wiki/Sherman%E2%80%93Morrison_formula), this equivalently reads\n", "$$\n", " u^\\star(v) = v \\odot A^\\top ( A \\text{diag}(v^2) A^\\top + \\lambda \\text{Id}_n )^{-1} y.\n", "$$\n", "This formula is more efficient when $p>n$ (which is the case here)." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "def u_opt(v): \n", " S = A @ np.diag(v**2) @ A.T + la * np.eye(n)\n", " return v * (A.T @ np.linalg.solve(S, y))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Compare the two formula." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Should be 0: 2.27e-15\n" ] } ], "source": [ "v = np.random.randn(p)\n", "print(f\"Should be 0: {norm(u_opt1(v) - u_opt(v)):.2e}\") " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "According to the [envelope theorem](https://en.wikipedia.org/wiki/Envelope_theorem), the function $f$ is thus differentiable and its gradient reads\n", "$$\n", " \\nabla f(v) = \\nabla_v F(u^\\star(v),v)\n", " = \n", " \\frac{1}{\\lambda} u^\\star \\odot A^\\top ( A (u^\\star \\odot v) - y ) + v\n", "$$" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "def nabla_f(v):\n", " u = u_opt(v)\n", " function = 1/(2*la) * norm( A@(u*v) - y )**2 + (norm(u)**2 + norm(v)**2)/2\n", " gradient = u * ( A.T@( A@(u*v)-y ) ) / la + v\n", " return function, gradient" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Test using finite difference that the formula is correct\n", "$$\n", " \\frac{f(v+t \\delta) - f(v)}{t} \\approx \\langle \\nabla f(v), \\delta \\rangle\n", "$$" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Should be small: 1.127928221549982e-05\n" ] } ], "source": [ "t = 1e-6\n", "delta = np.random.randn(p)\n", "f,g = nabla_f(v)\n", "f1,g1 = nabla_f(v+t*delta)\n", "d1 = ( f1-f ) / t\n", "d2 = np.sum( delta*g )\n", "print( \"Should be small: \" + str( (d1-d2)/d1 ) )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also let scipy do the work for us using `scipy.optimize.check_grad()`:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "3.4094813520188184e-07" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from scipy import optimize\n", "\n", "optimize.check_grad(\n", " lambda v: nabla_f(v)[0], lambda v: nabla_f(v)[1], np.random.randn(p))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We implement a simple [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent)\n", "$$\n", " v_{k+1} = v_k - \\tau \\nabla f(v_k)\n", "$$\n", "and display the evolution of $x_k = u^\\star(v_k) \\odot v_k$." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABXpElEQVR4nO29eXwdR52v/VQvZ5eO9l2ybHnf4zhxjBNCAiELWwLZgcAkEMIyzAwzMMxwL7PPAMPcAQbeAQIBAoQQQkICJJCQEJzNThzvu2VZtvZ9Ozp7d71/9JF0JEuWZUmWLNWTT6Wqq6qrf+rT/lZ1VXWVkFKiUCgUirmPNtMGKBQKheL8oARfoVAo5glK8BUKhWKeoARfoVAo5glK8BUKhWKeYMy0AWciLy9PVlZWzrQZCoVCccHwxhtvtEsp80dLm9WCX1lZyY4dO2baDIVCobhgEEKcHCtNdekoFArFPEEJvkKhUMwTlOArFArFPEEJvkKhUMwTlOArFArFPEEJvkKhUMwTlOArFArFPGHOCb6Ukq1bt9LU1DTTpigUCsWsYs4JfiQSYceOHTz44IM0NzfPtDkKhUIxa5hzgu/z+fjwhz+MaZr86Ec/UqKvUCgUKeac4APk5OQo0VcoFIoRTIngCyGuE0IcEUJUCyE+P0r6e4QQe4UQu4UQO4QQl0/Fdc9ETk4OH/rQhzAMgx//+Md0dnZO9yUVCoViVjNpwRdC6MC3gOuBlcAdQoiVI7I9B6yTUq4H7ga+N9nrng25ubncdddd2LbNj3/8Y0Kh0Pm4rEKhUMxKpqKFfylQLaWskVLGgYeB96RnkFKG5NBu6X7gvO2cnp+fz5133kkoFOInP/kJ0Wj0fF1aoVAoZhVTIfilQF3acX0qbhhCiJuEEIeB3+K08kdFCHFvqttnR1tb2xSYB+Xl5dx66620trby8MMPk0gkpqRchUKhuJCYCsEXo8Sd1oKXUj4upVwO3Aj8y1iFSSm/K6XcKKXcmJ8/6hr+58SSJUu48cYbqa2t5bHHHsO27SkrW6FQKC4EpkLw64HytOMyoHGszFLKrUCVECJvCq49IdauXcu1117LoUOHeOaZZ8735RUKhWJGmQrBfx1YIoRYKIRwAbcDT6ZnEEIsFkKIVHgD4AI6puDaE2bz5s1s2rSJbdu28dprr82ECQqFQjEjTHqLQyllUgjxKeD3gA48IKU8IIS4L5X+beB9wF1CiAQQAW5LG8Q971x77bV0dXXx9NNPk52dzZIlS2bKFIVCoThviBnU3XHZuHGjnK49bWOxGD/4wQ/o7OzknnvuobCwcFquo1AoFOcTIcQbUsqNo6XNyS9tzwa3280dd9yB2+3moYceoq+vb6ZNUigUimll3go+QDAY5I477iAcDvOzn/2MeDw+0yYpFArFtDGvBR+gpKSE973vfTQ2NvL444+r6ZoKhWLOMu8FH2D58uWD0zWfe+65mTZHoVAopoVJz9KZK1x22WV0dHTw8ssvk5OTw8UXXzzTJikUCsWUogQ/hRCC66+/nq6uLn7zm98QDAZZvHjxTJulUCgUU4bq0klD13VuueUWCgoKeOSRR9Q6+gqFYk6hBH8EHo+HO++8E7fbzU9/+lN6enpm2iSFQqGYEpTgj0IwGOT9738/sViMhx56SC2prFAo5gRK8MegqKiIW2+9lba2Nh555BEsy5ppkxQKhWJSKME/A4sXL+Zd73oXNTU1PPHEE2qOvkKhuKBRs3TG4aKLLqKvr4/nn38ej8fD9ddfT2rhT4VCobigUIJ/FlxxxRVEo1FeeeUVPB4PV1999UybpFAoFBNGCf5ZIITgmmuuIRqNsnXrVtxuN1u2bJlpsxQKhWJCKME/S4QQvPOd7yQajfLss8/i8XjU17gKheKCQgn+BNA0jfe+973E43F+/etfI4Rgw4YNM22WQqFQnBVqls4EMQyD2267jaqqKp588kmma4MWhUKhmGqU4J8Dpmly++23s2TJEn7zm9+wffv2mTZJoVAoxkUJ/jlimia33XYby5Yt4+mnn+ZPf/oTs3m7SIVCoVCCPwkMw+DWW29l3bp1/PGPf+Spp55SH2cpFIpZy5QIvhDiOiHEESFEtRDi86Okv18IsTflXhFCrJuK684GdF3nxhtvZMuWLbz++uv84he/IJFIzLRZCoVCcRqTFnwhhA58C7geWAncIYRYOSLbCeBKKeVa4F+A7072urOJgXn6A7tm/eAHP6C3t3emzVIoFIphTEUL/1KgWkpZI6WMAw8D70nPIKV8RUrZlTrcBpRNwXVnHZs3b+b222+nra2N+++/n8bGxpk2SaFQKAaZCsEvBerSjutTcWNxD/D0FFx3VrJ8+XLuueceNE3jgQceYPfu3TNtkkKhUABTI/ijrSQ26nQVIcRVOIL/t2MWJsS9QogdQogdbW1tU2De+aeoqIiPfvSjlJWV8atf/Ypf/epXxOPxmTZLoVDMc6ZC8OuB8rTjMuC0vgwhxFrge8B7pJQdYxUmpfyulHKjlHJjfn7+FJg3MwQCAe666y6uvPJKdu/ezf33309ra+tMm6VQKOYxUyH4rwNLhBALhRAu4HbgyfQMQogK4DHgg1LKo1NwzQsCTdO46qqr+OAHP0h/fz/3338/u3btUvP1FQrFjDBpwZdSJoFPAb8HDgGPSCkPCCHuE0Lcl8r2RSAX+P+EELuFEPNqPYKqqiruu+8+SkpKeOKJJ/jZz35GX1/fTJulUCjmGWI2tzY3btwo59JaNbZts337dp577jkMw+CGG25gzZo1akMVhUIxZQgh3pBSbhwtTX1pex7RNI3Nmzdz3333kZeXx2OPPcbPf/5z1dpXKBTnBSX4M0BeXh53330311xzDceOHeOb3/wm27ZtUxulKxSKaUUJ/gyhaRpbtmzhE5/4BGVlZfzud7/j/vvvp66ubvyTFQqF4hxQffizACklhw4d4ne/+x29vb2sX7+eq6++mszMzJk2TaFQXGCcqQ9f7Xg1CxBCsHLlSqqqqti6dSuvvvoqBw4cYPPmzWzZsgW32z3TJioUijmAauHPQjo7O3n++efZv38/fr+ft7zlLWzYsAFd12faNIVCMcs5UwtfCf4spr6+nmeffZaTJ0+SlZXFFVdcwbp16zAM9WKmUChGRwn+BYyUkmPHjvHCCy/Q2NhIMBjk8ssv56KLLlLCr1AoTkMJ/hxASkl1dTUvvPACDQ0NZGRksGnTJi6++GK8Xu9Mm6dQKGYJSvDnEFJKampqePHFF6mtrcU0TdavX8+mTZvIy8ubafMUCsUMo2bpzCGEEFRVVVFVVUVzczPbtm1j586dvP7661RVVbFhwwaWLVumunsUCsVpqBb+HCAUCrFjxw527txJb28vPp+PtWvXsmHDBgoKCmbaPIVCcR5RXTrzBNu2OX78ODt37uTIkSPYtk1RURGrVq1i9erVZGdnz7SJCoVimlGCPw/p7+9n7969HDhwgPr6egBKSkpYtWoVS5cuJS8vT63SqVDMQZTgz3O6u7s5cOAA+/fvp6mpCYDs7GyWLl3K0qVLWbBggerzVyjmCErwFYN0d3dz7Ngxjh49Sk1NDZZlYRgGZWVlLFy4kMrKSkpLS1UFoFBcoCjBV4xKPB7nxIkTnDhxgtraWpqbmwEwDIOKigrKy8spLS2ltLQUv98/w9YqFIqzQU3LVIyKy+Vi2bJlLFu2DIBwOMzJkyepra2ltraWrVu3Du6/m5WVNSj+xcXFFBQUqEpAobjAUIKvGMTn87FixQpWrFgBQCwWo6mpiYaGBhoaGqivr+fAgQOD+QOBAAUFBRQWFlJQUEB+fj65ubnqy1+FYpaiBF8xJm63m8rKSiorKwfjQqEQzc3NtLa20traSktLC6+//jrJZHIwj9frJTc3l5ycnGF+VlYWXq9XzQ5SKGYIJfiKCREIBFi8eDGLFy8ejLNtm87OTtrb2+no6KCzs5OOjg5qa2vZu3fvsPNdLhfBYHBMl5GRoQaMFYppYkr+ZQkhrgO+DujA96SUXxqRvhz4AbAB+IKU8qtTcV3F7EDTNPLy8kZdyycej9PV1UVHRwc9PT3DXFNTE/39/aed4/F4yMjIIBAIEAgEhoXTjz0ej3pbUCgmwKQFXwihA98CrgHqgdeFEE9KKQ+mZesEPg3cONnrnQ2PtXSxMdNHhVftFDXTuFwuCgsLKSwsHDU9kUjQ29tLT08P3d3d9PX1EQqFCIVC9PX1UVdXRygUGtZlNICmafh8Pvx+Pz6fb1h4tDifz4emqW2cFfOXqWjhXwpUSylrAIQQDwPvAQYFX0rZCrQKId4xBdc7Ix3xBP/3pf9LRuZ6fv+WDxM0VffAbMY0TXJzc8nNzR0zj5SSWCx2WmUQDofp7+8nHA4TDodpamoiHA4TjUbHLMvr9Z5VxTAQNk1zOv5shWJGmAo1LAXq0o7rgU1TUO45YcgwC/Rm6hq/zjVPPMP33/KPrMlbOVPmKKYAIQQejwePx0N+fv64+S3LOq0yGAin+x0dHdTV1REOhxnrexTTNMetFNLDqptJMZuZCsEf7ek+56+5hBD3AvcCVFRUTPj8oDvIr9/9c/7+jZ/wm8Pf4c7f3s7NS97Hn2/4c3I8OedqluICQtd1MjIyyMjIOKv8tm0TjUZPqyRGqzTa2toIh8MkEolRyxroZkp341USaq9ixfliKgS/HihPOy4DGs+1MCnld4HvgvOl7bmUoWs6X77kQ3gyN/OTA9/ll9WP8fuTv+eT6z/JrctuxdTUa7piiHSRPttNZOLx+JiVQnq4paWFcDhMJBIZsyyPx4Pf7x90A5XBaM7r9apxCMU5MxWC/zqwRAixEGgAbgfunIJyz5n93/gYJUUZ/EN+DvU51/JM4C0s6/0+X3rtS/xi/w/5u2UfZFPZFeDPB5cf1Cu4YoK4XC5cLhdZWVlnld+yLCKRyKiVQno3U3t7++DxaAghzlghjKww3G636mJSDDIla+kIIW4AvoYzLfMBKeW/CSHuA5BSflsIUQTsADIBGwgBK6WUvWcq91zW0omFw3znnptJ2IJFgU5W5rXz0cu/TLsrm384+HF+kCFoMA2u6Q/zN51dlOByhD+jCLIXQNaCIb9gBQTUBiKK849t28MqhJEVw0gXi8VGLUfX9TNWDiMrCjVIfeEz7xZP6+/uYs+zT7H7maeI9PYQLCvn8WWXEl+5lp8X9fHIqd/w/frnAMnd/sX8GUE8fS3QfRJ6GkBaQ4X5C6BwleOK18OCzRAsm7K/UaGYCpLJ5KgVwWiVRSgUwrKsUctxuVxnXUF4vV41/jALmXeCP0AiHuPQiy+w86kn6Kg/RVdmDomrbuDfb72Z1mgbX93xVZ45+QylgVI+u/GzXF1xNcK2oLcBuk5A6yFo3g8t+6D1MFipVlRWBSzYAgveBIuugqzyM9qhUMwmpJTE4/FxK4j0imIsnUjvPhqvq0nNYDo/zFvBH0BKyfE3XuOJh34EDacQuQW8/dY7WXnFVexofYP/eO0/qO6uZnPxZj5/6edZlLXo9EKsJLQegJOvwMmXHT/c4aQVrYXl74Bl1zth9VAr5hADs5jOtoIY6zsITdPOatxh4NjlcqkK4hyY94I/gG3b/NWvfoP7ud9Q2N5IsLCIy266jaWXv5lfVD/Kt3Z/i0giwp0r7uTj6z5OwBUYuzApoe0wHP09HHka6rYDEoIVsOZmWHc75C+bMtsViguFZDJ52jcPZ3JnmuI68P2F1+udkD+fB6uV4KcRSlpcv+MIvmP7uW3fS3SdrCG7uITNt7yfgotW8T+7v8ljxx4jx5PDJ9Z/gpuW3HR20zhDbXDs93DgV3D8eWccoOQiWHcHrL4Z/GN/SapQzGdG616KRCJEIhGi0eiY/pm0a+BjvYEKYKASSHejxY10F+IUWCX4I6gOR7lux1EWe938t+zk9V/8lLZTteSVL+BNt32AWGUGX9nxFXa17qIio4JPrv8k1y28Dk2c5Y/f1wL7H4U9P4PmfaC7YfV74ZKPQOnFqstHoZgkA8ttnKlCSPdjsdhp7mwwTXPUimFgWq7L5cI0zWHHZ4ozTXPa3zyU4I/CU23d3L2/lg8U5/KfS0s5su0lXnnkp3Q1NVC4aAlbbvsAp3JDfGPXNzjadZRl2cv49IZPc0XpFRP7wZr3wxs/dMQ/HnJm+lzyEafbx1QbhSgUM4Ft2yQSCaLR6KiVQbobLU88Hh90Y814GouzqSh8Ph9XX331Of1tSvDH4N+PN/KNU63817Jy3l+Si21ZHNz6PK/+8mf0trVSunwVb7rt/ezz1PHNXd+kPlTP8pzl3LP6Hq5ZcA26NoEpabE+2PMwvP59aDsEnizY8EG45KPOvH+FQnFBYlkWiURisAJID48Vd6Y8iUQCt9vNpz/96XOyRwn+GFhScueeGl7tDvHEhiVclOlz4pMJ9j33DNse/zn9XZ0sWHsRl91yBzu0ozyw/wFO9JygIqOCP1v9Z7yr6l249QkswyylM8vntfvh0K8BCctugE33QeXlqrtHoVBMCiX4Z6AjnuTtO44A8PuNy8hzDa02kYhF2fPMU2x/4lGifb1UbbyMN916J3up4Xv7vseBjgNku7O5ccmN3LL0FsozJjgfv6cBdnwfdvwAIp1QuBo2fQzW3KK6exQKxTmhBH8c9vSFeffOY1yS6efhdVUY2vBWdjwS5o2nnmDHrx8nHo2wbPMVvOnmOzmmNfDw4Yf5Y90fsaXNltIt3Lr0Vi4vu3xiC7QlIrDvUdj+bWjZD94cuPjDTl9/sHRq/1iFQjGnUYJ/FvysqYO/OlzHJysK+L9VJaPmiYT62PHrx9j59JNYiQSrrnwrm266jVhA8Mtjv+TRo4/SFmkj253NdQuv4x2L3sHavLVnP8grJdS+5Aj/kacAASvfDZs+DuWXqu4ehUIxLkrwz5LPHanjwcYO7l9VybsKssbM19/dxWtPPMqeZ5/CTlosuWwLl7zrveQurOSl+pf47Ynf8kLdC8SsGOUZ5VxXeR1XV1zNqtxVZy/+XSfh9fth54MQ7XFm91z2cVh1Exhq60aFQjE6SvDPkphtc9Ouao70R3nq4qUs83vOmD/U2cHO3/2aPc88RTwSpnzVWi5513upXLeB/mSYP5z6A7+t+S2vN7+OJS0KfAVcVX4VV5VfxcaijWc32Bvvd2b3bP8OtB9xFnPbeLfjMkbfJ1ahUMxflOBPgMZonLfvOErQ0PndxqVkGONPvYyFw+x77ne88dQThDo7yCosZu3brmPVW96GLzNId7SbrQ1b+eOpP/Jy48tEkhHcupsNBRvYXLKZy4ovY1nOsjN/2CUl1PwRtn3b+aJXM2DpdbD+TljydtDVsrYKhUIJ/oR5uauPW/cc58rsDB5cs+i0QdyxsJIJjm5/hT3PPEXD4QPopsnSyy5n7dXXUrp8JULTiCajvNb8Gq82vsq2pm1Ud1cDkOPJYVPRJi4uvJj1BetZnLV47Hn+HcdhxwOw9xHobwVfHqy91RH/ojVTdRsUCsUFiBL8c+DBhnY+d7Seu0vz+PelE1//vv1ULXv+8DQHtz5PPBIhIzef5VvezPItV5K/YOFgX35ruJXtTdt5tfFVtjdtpzXSCoDf9LM2by3rC9azPn89q/JWEXQHh1/ESsLx52D3T50F3Kw4FK6B1TfByhsht2qyt0GhUFxgKME/R/6xuoFv17Xxr0tK+UhZ/jmVkYhGqd6xjcMv/4naPTuxLYvcsgoWX7KZRRsuoWjxErRUS15KSUOogd1tu9nd6rhj3cewpQ1AaaCUlbkrWZGzghW5K1iRs4Jcb2pRtnAn7P8l7P051L/uxBWuhpXvgeXvdHbvUrN8FIo5z7wT/JMf/jOMgny8a9fhXbsG9/LlaC7XhMuxpOQj+2v5fXsPP1izkGvzguOfdAbCvT0c3fYyR17ZSsORg0jbxpsZZOH6i1m4/mLKVq4hkJ0z7JxQPMS+9n0c6DjAoY5DHOo8RF1f3WB6rieXqqwqFgUXsShrEVXBKhZpXnJrXkQcehJObQMkZJbB4rfCkmtg4ZXgyZzU36JQKGYn80rw7Xichs98huievSTb2gAQpol7xQq8a9fiXbsG79q1mAsWnNUUyX7L4uZdxznYH+GhtYvYkp1xTn/LSCKhPmr37OTEztc5sWsH0f4QANnFJZQuX03ZilUUL1lOdlExYsQSrb3xXo50HuFgx0Gqu6up6amhpruGUCI0mCfoDrIouIhyTx5l0X7KuuopbzxAWaSHXKkhSi5yduyqeBNUbAJv9pT8XQqFYmaZV4I/gJSSZHMzkb37iOzdQ3TvPiIHDiDDYQC0YBDvGkf8PalKwMjJGbWszkSSG3dW0xiL8+j6xaxPrbkzVdiWReuJ49Qf2k/94QM0HDowWAGYHi8FlYsoXFhFwcIqckrKyC4pxeMfvjmLlJLWcCvHe45zoucEx7uPU9NTQ31fPS3hlmF5vUKn1BaUREIUJBMUWDaFvgIKcpZQULCWwvI3kVl6KcKllndQKC405qXgj4a0LGLVx4cqgL17iR07BrbTR26WlzuVwLq1eNauxbNiBZrHmYvfFIvz7p3V9FsWj1+0ZNw5+pOy07ZprztJS001LSeO03KimrbaEyTjQ2t4+4JZZBeXklNSSnZxKZn5BWTk5pORl4c/K3twXAAgZsVoDDVS31dPfaie+r566vrqaAo10BpqpDPtzWAAj22Tj06B7iPHk022v5DsjDKyc6rI8ReR7ckm251NjieHLE/WxJaSUCgU08a0C74Q4jrg64AOfE9K+aUR6SKVfgMQBj4spdw5XrnnKvh2LIkwdIQ+fpeN3d9P9OBBInv3Etmzl8i+fSSbmpxEw8C9ZAnuJYtxL1lC8+Kl3KFnY2mCR9ZVsTpjalv6oyGlJGknCcVDtDWcoqPxFD3NzYRaWgk1txJqaSHWN1ywNV3Hn51LZl4evmAWvswsvJlBfMEgvswsfJmZ+IJZeDMycfn82JqkLdJGa38LLe2HaG3ZS2tXNa3hFlrjvXTZcbp0jW5NQ47RDebTXARMP5muTDI8WWS4g2S4MsgwM8hwZTjxrgz8Lj8+w4fX8Dq+6R12bKrvCRSKSTGtgi+E0IGjwDVAPfA6cIeU8mBanhuAP8cR/E3A16WUm8Yr+1wFv+GLLyPjNugCYeoIl4bm0hGm5riBcMrXRhzbsX6STQ0k6k8Sr6slUVdLsrURacWozwrymU/+JRGvj6+98Fs2+ExcpaXoeXkYuXkYebnoubnoWVljDhSHE2Ga+pto6m+iub+ZjkgHXbEuOqOddEY76Yp20R3tpj/ZTzQZxZJn3mDBlRD4Igb+qE4gYuCLGgQiOoGoiTeu447ruOIgGF2spS4QbgPhdqF5XRgeN4bXi8vrxZXaTNpDEpfVD/Eu7Hg3yWQ38UQPYbuXPt0irEPIgJAJfbqg19DpNXX6dEGfJjnbLSIMoeM1vHhN32BFMOBM3cStu3FpLly649y6G1MzB8MD8S4tlZZ2jqmb6ELH0Ax0oWNqJro2dGxoBoYwBuMMYWBoxrzdG1VxYTLdgr8Z+Ecp5bWp478DkFL+R1qe7wAvSCl/ljo+ArxFStl0prLPVfBf/NmzWIkkWBJp2UhLOuHkQDgtzhqKwz678ts9Jl/ZUEG3y+DTu06wqq0TO5kE20LaFtK2HV9aSCQ2EksDS0iSSGxsJGALiS0FEgmaQAgdNB1N6OiaBmgIQGgCITRE6r+R0ytl2qEtJc5/pK4inTjbwrIT2FYMy4pj2zFsK4G0k0g75cuksxcvCZAWQiQR0gKSziU16fgChJDD/cG0tOdJ4FQxA/8TqQgBCIEUApmKlhpIBFJI0CR2Kt+ALwVIIYfyCIkUzj0cSB+45kBYpgwY9oSP0G458njkjy0GIp2yhBQg5MAvMaJAMfzvTosTI9JHS9UG/4ABS1KpMr3MUcobg8FUOUq8GDo6++psKO/Ic+QZyplMdXlaqWcsTJzhaAqZyl7w0X4bQKLzyc8+ck5FnknwjdEiJ0gpUJd2XI/Tih8vTylwmuALIe4F7gWoqKg4J4O+tKsXaQt0BLp0ZNMJC3Q0NKkPpukItJSvS4EGw3yd1Pkj8r7/N31oCDSyOSqz0cd7vMarTAabwDZCT6AZcYQeRzNiaHockfKd+BiaER+M11P5xEC6lkw5Ky2cBM1C0xMIzYLB9ETKtxzxViguYCRgYWChpXx9FJee7vg22qhOjhF/ZqdPKL8c5Vx3MsInp+H+TIXgj6Z0I5XjbPI4kVJ+F/guOC38czHoulA22lm21iVga6RaiyA1gdQG4sRQmgZJaRElSUwmiMk4MZkgYQSwdA2X0YnX6MTtSuJyJ3EbFh5XAreZxNRj6HoUQ49iaFF0PeKE9QiG5sTrWgxdi2Ho8Qn/vZZtkLRcJC0Xlm2StHVs28CSKd82sCw3VkJHsyWGbWPaFqa0MFK+SyYxZRI3cTwygYkNUpJqTAPC8VNxIEjaOkmpk8QgaetY0iBJKk7qThgn3ULDRmBJHYnARjgPunB8iUi9jzjhgadjoJWuSQZzaZBq8jstfnBslHLovMGGuUyFpeMGfvOh338gZ/o101KHZx4MDAQHW9pSDsWmv+TI4SeLYdccraU8GqP98xGpv2+0M9La/8Oa+WKUosRQWcNCjp22ppPUdJJCkNQMLE13nNBIpvyEbmAJJz6pDcU7+XSSuk4y/Tjlpx/bQmBpOjYCW9OwhBNnC6esgbAtROrYcZaWFhYa8kzrUc0AQkqEdGRdyNTTLW00mXqWx0gPJKLTYs9UCH49kL7VUxnQeA55pozEZbnousDnNfF7DQI+kwyfSYbfJOhzEQy4CPpdmC4NTR/+gEgpicc7qevez/GOnTR0H6Q9VEMk2oyLBLkaZOgaOYYLnw4mCQRnIdISpO0C28ROGsikjhXXseMaVtwgHtdJxrwkoxI7IbCTAjupOS6RFk4K7ERaOKkN65cQSIJmlBxvlGJPiFxXhKArQoYeIaBH0Ua04mPCpN2VTbuZRYeZRZ0ri3Yzmx49QEx4kdKAlLOlCSknbAPdEujRBEQsZDRJMmaTSEIsKYgnIW4LLDSSQscSgoTQiQudpG6QNAySmk4i5eJCI45GDI241EiO88bk0yUBEwIGZJgQdAuy3ZDtEmS7BdkeQa5bkOXWCLoFup56B9MGusk0p29eiDTfiRPD4px4p0dKc3qlhOZ0wSEQmiOYg91uw84bXvZgGWOcO3SNofMAEmjEkERtiCOJ2RBDEpOCmG0TkxCTkqgtiUuI2jZRyyZqS2K27RzbMhU/FBezJZGUH7Uc3ylPkrAliSmewScAtyYwhcClabg0gUsIzFScWwiMAaeBKQS6cNKcOJGKG5GmOenpacaIcwaPBaPECTQBunDe6HUh0FJl6SL1tj/qsROnkZYmUr0FA2mzbPxnKgT/dWCJEGIh0ADcDtw5Is+TwKeEEA/jdPf0jNd/Pxkeqmkg1B/GkElMO4khk7hljCxXN0F3L0FXH5meXrI8/WR6+/F7Qvi8fXjcUTyeBJrmPOgGsABYYIDtFSSjOlZUx4ppWFFBX0zHirmxYjotviJ2LN5MjzubBcerWbFvDyIqsRMayaRBwnaR1FxYmklSMxwnTBKaSVI4x3FhkkQnLkwSwiAuDOJCJ4FJQjNI6CZJw8Dy6thCw0OcpXo9K/VTrNJPslSrZ6FowS0Sg/eiVcvihLuUOl8x9f4iGtyF1HsKaRB5NFvZ9Mfc6DEbI2o5LiIxkzaGdERH18AlBdmWICcBOXHIjEMgAd7k8Pue0CHu1kj4BQm3RtKtYXt0bI8Gbg3cOsKjoRsahqaha84/yCFfw9CdYyQkbZu4ZRNP2sSSNrGETTRhEUlYRBMW4bgTDsWSNIaT9LQniFs2I9vJmoBsn4uCTDdFmR6Kg15KsjwUZ3kpz/ZSlu0jP+DC0LUJD9BKKYnZkn7LJmLbhC3HjRm2bMK2TdiyzpA2FI5Y9qS6jF1C4NYEHl3DrQm8moZb0/BoAo+mETBNPLoT9mhDQuxOCbFb01ICnXJCGwybYijdrTn5h+VJnWdqArfQ0AVqAHyGmappmTcAX8OZlvmAlPLfhBD3AUgpv52alvlN4DqcaZl/JqUcdzT2XAZtpbR5+Gtb0N0xTF8S05fE8CUxvNaoS8kkIzqJiEE8ohOL6ESjJrGYh1jUSyyWSTSeTTSeSVx6sHQXtmZi6SaWZiANF1J3gWEiDRcJl4cDFfmcys3Ak7BZ0p1gQQRM3UDTxGALYCCsa0OiqouB8EA8g3m0VIsjK95Eae8uSnt2UdSzm6xw7eAreLe7kJrAYo74K9jtqeCAr4JjvgX0mhnko1EsNAqlThEaebbAtB2xsqR0xqulxLIlVtJG705gdMUxepIYfUnMsDXYNWHpEPPpRD0aUbcg7BZE3IKQKYgLR6QtW5K05QjfxrLGiLclCev8jR9IcGoBUyB1DQyBNASkwpgawtDQTOFM7zVTeXQNaQikLrB1ga0JbJ0xp6qOhS7BkGDiNCpcCMcJgUuAW2h4BoRaOGLsTgm0V9fw6BreVNira/gMDb+h49M1fIaO39QJGE7YbWi4dG2wYlWCO/eZ7kFbpJRPAU+NiPt2WljCtIxBnIYQGoVr+wEDjRwStpeQpdHZn6QhEqIu0ku3LeiW4A+UsbRwFasr17A6bzXLc5bjMyc/t/6VrhBfPtHEdrOfVpfBR8ryuaUohyL3BOeYhzvh+PNQ/RzUvgg9zrh33JXJifwN/K7i7TznqmKbfwntrhxMIVju97Amw8v7MnysDXhZEfDi08fu10wmLJqqe6g/0kXz8R5aanuxEs4ASCDbTV5VNnllAceVB8jM9aa6IqYee5SKYODYGlZR2CQsm+6ERXfSoidp0Zu06bWS9CVtei2LkGUTsm36bccPS+m0nqUkwllOE5USkhJh2WgW6LbEtCWmJXGT6oKQoFkSzR7IK5HWgG87M8OSEpI2dtLGTqb9LZbzN0ZtSZ9lj94dP8W4dOctytAErtSblqGLtHgN09Bw6U66qTsVhstw3EAFMnBsDoT1VNqA03XMVBlD5+lD6YaGqQvcaXH6ND1XiiHm3Je2CSvBv237V/Z3HKC6u3pwDnuBt4BVeatYk7eGVXmrWJU7ynLDU4iUkpe7Q/x3bQsvd4cQwOqAly3ZAd6UFWCJz0Ox28Qzcgyh5SDxQ7/BPvYsnsYdCGnT7wryRu7FPBdYw9bMdRz2L8Sj66wMeFmTEvbVGV6W+T24tTMPWkkp6Wzqp+5gJ3UHO2k41o2VsNE0QV55gOLFWRRXBSmqCuIPnr+tFJO2pD2RpD2eoDNh0ZFIOi6epDMV7kxYg+GuRJLkGR5dQ0CmoZNlGGQaOkFDJ2im/JTLMHQyDZ2AruHXNQKGToau49cE0ahFS1eY2o4wJ9r7qW3v50R7Pyc7wkQSQ9VFYaabpYUZLCvMYGlhBkuLMlhSEMDvnnhbynnTsUnakkTSJmHbJC0nLmE5lWAiKQfjk5bT5ZVMpcVTcUlLpuKdsgbzWGl57IFyU9dIXXOgnERyqDstkdatNhA3EJ+0p04/dE0MViZmegWip1ckToU0kOYekeZULsMrlpF53OZolZg+FE454wJ9I5p3Syvc9MRNFPgKWJW7itV5q1mdt5oCX8E0WHh2HA9HeaK1m5e6Quzo6Seeds/zTINlkZNc0/w81zQ/R1X/CQD2BJbxXM4m/pB7GS15a1ji97PM72F1hpc1GV4Wez1nvTGLtCUtJ3up2dVGza42etoiAGQX+ShfmUP5ihxKlmTh8kzJC98wkrakNZ6gOZ6gNZakJZ6gOZZw4mJJWuMJWuIJ2uPJUWeuCiDb1Mk1DXJNg5yUy3UZ5Jg62aZBVkq408Xcdw798WeDlJLGnihHW/o42tzHkZY+jrb0cawlRCw59BeU53hZVRxkdWkmq0qDrC4Jkp8x9/YiHqikYqlKIG7ZwyqLseLjSZvYiMojPsp5I+OHn2cNix84b6oqISFIq0QGKgv9tApoWMUxylvQaJXJyDyDlVQqzmNqlGWfW2/DvBP82UzEstndF6alq5msg4+y9NgvKek+io2gJn8DRyuvp7XqevxZJSzyulni95B5FtssjkRKSXtdiCPbmqne2Up/dwxNE5Quz2bR+nwWrM4lI2fy6wElbEljLE5d1HH10fRwgsZYnJHd8wLIdxkUukwK3SaFLiPlm+S7UoKeEvYsU0e/AFpZli2p6ww7FUBzH4eb+zjQ2ENtR3gwT2Gmm9UlwVQFkMnq0iDFQc8F2Yqczdipt5r0yiaWGKVyGFkpjahEYmmVyOkVy/DyYqNUVgPXts6hAsoLuNjxf645p79fCf5swbbg+B9h14Nw+CmwE1B6May51dmoJLN40pcIdUU5+loLh7c109XUj2YIFqzKpWpDAQtW5+LxT3ytGiklTbEEx8MxqiMxjoejTjgcoyEaH9YyF0Cx26TM46Lc46LM46LUbVLkNilwOX6eaZz128mFTm80waHGXvY39nKgoYf9jT1Ut4YY0IBsn8nq0iAXlWexLuXyAnPvTWA+Y9lyqJIYrQKxnBlocWsoXdcE71xbck7XU4I/00S6YddP4LXvQPcp8ObAutvhog9C4cpJF2/bklP7O9j/YgMn93eAhOKqIEs3FbH44oIJiXxXIsmBUISDoQj7QxEOh6JUR2KErSFZ9+kai71uqnxuKr1uyr0uKlICX+w2cY0zjjDficQtDjWnKoCGXvY29HCkuXewEijL9rKuPIv1ZVmsr8hidUkQr2vib3mK+YkS/JmivRq2fxt2PwSJfqjYDJd+1Nly0Jh8Ky7cG+fgy40cfLGRvs4ovkwXKy8vYfnmIoL54/f/hZIWu3rD7OjtZ1dvmAOhCA2xoTn8+S6DlX4vS/xuqnweFnvdLPa7KXKZqhtiignHk+xv6GVPXTe767vZfaqbhm5nrEXXBEsLM1hfnsX68iDryrNYUpChZrUoRkUJ/vmmcRds/Soc/g3oLlj9Pth0H5Ssn5LiO5v62f2HUxzZ3oydlJQuy2b1m0tZuD4P/QxTMBuicV7tDvF6Tz9v9IY5GIpg43TDLPa5WZPhY2VqYHhVwEu+Sy1VPJO09cXYW9/N7jrH7anrpjfqfO3mc+msKQ2yPtUNtLYsSGmWV1XECiX4542TrzhCf/w5cAed1vyl90JG4aSLllLSVN3NrmfrqN3bjm5qrNhczNqry8gu8o96Tm/S4pWuEH/q6mNrZx/HI84GKhm6xoZMPxuDPjZm+tmQ6SNoTv0MHcXUYtuS2o5+9tR3s6euh1113Rxq7E19XewM9K0ry2JtWRbryoOsK8si2z/xvZwVFzZK8KcTKZ0Po178Kpx6FXx5sPmTcMlHpmSjcGlLana3sfOZU7TW9uIJmKx5SxlrrizFm3H6P+bmWIKn23t4uq2bV7pDJCV4NY03ZQW4MifA5dkZLPN7LoiZL4rxiSUtDjf1pd4EethT383xttDgR1wVOT5nMLjM6QpS4wFzHyX404FtO102L/4XNO2GzFJ406dhw13gmvzXutKWHN/Vxuu/PUFnYz/BfC/rr6lg2WVFmCP+wbbFEzzW0sWTrd280etMA6zyurkuP8hbczLZGPSpgdR5RF80wb6GHvbU9bC33ukKauxxVl/UNcGSgsCwrqBlhRkYZ+gKVFxYKMGfSqwE7P8lvPj/oP0I5CyCy/8K1t4OxuRfn0cKfXaRj43vqGTxxYVoaYN0CVvyXEcvDzd38IeOXpIS1gS8vCM/yPX5WSz1uVV/rmKQ1r4oe1NvAHvqe9hT101PxBmg95gaq0uCrC4NsrIkk5XFmSwtzMBlqErgQkQJ/lSQiMLun8DLX3emVhasgis+AytvBH3y/d9SSo7vHF/oW2IJftjQzo8bO2hPJMl3GdxcmM3txbnTurG6Ym4hpeRkR3hwPGBPfTeHmnoJx51lI0xdsKQgg5UlmaxKVQIrSzLJ8KiB/NmOEvzJEO6EnQ/Ctv8PQi1QuhHe/Dew9DpGXX7zHGg40sUrj1XTerJvTKE/3B/hW6da+VVLN0kpuSY3kw+U5HJVTiammp6nmAKs1KDwwcZeDjT2crCpl4ONPbSHhvZ7WJDrG3wDcFyAyjw/puoSmjVM+2qZc5KmvfD6/bD3F5CMwMIr4X3fg8orpkzoOxpCvPr4cU7u7yCQ7eatH1rB0k1Fpwn9f9e28GRrN15d466SXD5Sls9Cn/oaUzG16JqgKj9AVX6Ad61zvvKUUtLWF+NAYy8HGntSlUAvvzvQPDgwbOqCRXkBlhQGBiuBpYUZLMj1q28FZhlK8NOJ9cHh38IbP3Rm3BheWHurM72yaM2UXaavM8prv67h8LZm3F6Dze+tYu1VZRjm0GBsUyzOv9c08WhzFz5d49MLCvlYeT45avqk4jwihKAg00NBpoerlg8tQBhNWFS3hjjW2seR5hDHWvrYU9/Nb/YO7WvkMjQW5flZmHKVeX4Wpfxcv0uNMc0ASj1ifVDzAhx43FnfJhmB7IXw9n+Di94P3uypu1Q4wc7fn2TP8/VIKVn/1nIuvr5y2NIHEcvmf+ta+Z+TrVhS8vHyAj5ZUUCuS/1UitmDx9RZXeoM9KYTjiepbg1xpLmPY62hwfCzB1uGrWKZ4TGGKoJcP4vy/VTk+CjL9pEXUJXBdDH/VCQRdaZRntoGNX+E2pedRcy8ObD+TqdFX75pyrptAKyEzb4/1bPj6Vpi4STLLi3i0ncvJDPXOyzfK10h/vrIKU5E4rwjP8gXq0pY4FVdN4oLB5/LYG3q4690kpZNfVeEEx39nGhz9hao7ehnR20XT+5pHLb5i9vQKEttPTnc91Ke41NvB5Ngbgq+lBDthp56x7Ufg7bD0HoIWvaDlRqEylsGl90HS66FistAn9oZCNKWHH29he1P1NDXGaViZQ6b31tFXlnGsHx9SYt/Od7Ig40dLPC4+MW6Kq7IyRijVIXiwsPQNSpT3TlXLRueFk1YnOoMU9cZpr4rQn3XgB9hb303XeHEsPweU6Mo00NhpoeioGdYeMAvyHCrgeRRmHuCb9vwlUqI9gyPDxRC/nLY9DEov8xpxQfyp82MuoOdvPJ4Ne11IfLKA1z1wfWUr8g5Ld+evjAfO1DLyUicj5Xl87lFRfh19SWkYv7gMfXBWT+jEYolaRhWEYRp6onS0htl56kuWnpig8tLDCAE5PrdFGa6yQsMOBd5ATe5aX5+wE223zVvKoe5J/iaBps+Du4MCJZCZhnkVoHvdLGdDtpO9fHq49XUHeoiI9fDNXevZMnGwtP2gZVS8v2Gdv65upE8l8GvLlrMpqzAebFRobiQCLgNlhVlsKxo9ApBSklXOEFzT5Tm3gjNPTGae6O09ERp6YvSEYpzrKWP9lD8tIphgGyfSW7ATa7fRbbPRbbfJOh1keUzyfKaZPlSYZ9JVireY154DbO5J/gAV/3deb9kb3uE7U/WcPS1Ftx+gy03L2bNlWXo5ukth4hl85eHT/FEazfX5Gby9RUVavaNQnGOCCHI8bvI8btYWTL2+lVSSvpiSdr7YnT0x2nvi9Ge8jv6Y3SE4rSHYhxvC9F9KkF3OE5i5HZtaXhMbVD8g16TDI9Bhsck4DYIeAzn2D1anJM34DHO+5vFpFRGCJED/ByoBGqBW6WUXaPkewB4J9AqpVw9mWvONiKhOG88dZJ9W+sRQrDh2gVsuLYCt2/08YC2eIIP7TvBrt4wX1hUzKcqCtQAlEJxHhBCkOkxyfSYLDqL3lwpJZGERVfYEf+ecILuSIKucJzucIKeiBPfnYpv7I4SioXoiyboiybPam9dt6GR4THxu3V8LoNAys/PcPPVW9ZNwV89nMk2Kz8PPCel/JIQ4vOp478dJd8PgW8CD07yerOGRMxiz3On2PnMKZIxi+VvKubSdy4kkD328gaH+yN8YG8NHXGLB1ZXcn1+1vkzWKFQTAghBD6Xgc9lUJrlHTe/tCxkLIYdi2HH48T6I4RCEcL9EcKhCJH+CJFwlEh/hHg4SiwSJR6JEe+NYQ2clzoXtxtmoeC/B3hLKvwj4AVGEXwp5VYhROUkrzUrsCybQy818tpva4n0xlm4Lo/Lbqwip3j0NekH2NMX5rbdx3Frgl9tWMy6jMmvqKlQKIYjpUTGYoPCO2Y4GkPGxwjHYtjxiYbjkEiMaZcBZKTcuBgGZuHk99AYy47JUCilbAKQUjYJIQrGO2E8hBD3AvcCVFRUTLa4KUPakuqdrWx/ooaetgjFi4PccN8aihYFxz33jZ5+7th7nExD55frF6u59Yp5h5QSmUggIxHsaNTxIxHsSBQZdeLsSAQZjWJHotiRMDISdfJGnXx2NOLERSJD4YGyolFH0OPx8Y05E4aB5nIhPB6E231aWM/IROS50dwuhNuDcLvQ3O7hYZcb4XI5zu342sDxoHMjXObweLcbYZqIaZylN67gCyH+ABSNkvSFqTcHpJTfBb4LzuJp03GNiVJ3qJNXHz9O26k+ckv9vOOTa1mwOves+t63d4e4c28N+S6DR9cvpsyjdiBSzC6klI7QponnaeI7Mi5NfE8T5AGRDkeGlYk9+gyZMyE8HjSPB+H1onm9g2E9kIHIz0fzeNG8HoTbg+YZEN50QU6FPR6Ea4zwgLC73Qhjbk+eGPevk1K+baw0IUSLEKI41bovBlqn1LoZJn2KZSDHzVs/vIKllw5f3OxM7OsL8/69NRS5TB69qIpitxJ7xcSQicRpAjxMYAeEerS0kS3kSGRI0Ee0kCeMEGheryPEHg/C63HE1+NBz85C8xQ7QuwZke7zOiLuTfkDgp1KGxT3lBALtXHPlDLZ6uxJ4EPAl1L+E5O2aBbQ3Rpm+5M1VO9oxeM32XLzYlZfWTpscbPxOBmJcefeGoKGzi/WK7G/0JCW5XRBjHTx9HA8LS0+Rv60+Fj8dEGORUfpukhrFVvWxI3XdUcwfd6UoHoHBVfPzx8u0l7fcEEeEF/vgCgPxHmGtbCFSy1vcCEyWcH/EvCIEOIe4BRwC4AQogT4npTyhtTxz3AGd/OEEPXAP0gpvz/Ja085/T0xdjxVy8EXG9EMwcYbKll/TQVu78RuU1s8we17jpOwJb/csJgS1Y0ziLTtMQQ0Pjw8QjhJJLDTxJMxxTh+ujCfSYTHcOcktOMxIMQp4UxvAevZWWjuolGFd6AlLDzu0dMGhHig+8NUm5QoRmdSgi+l7ADeOkp8I3BD2vEdk7nOdBOPJNn17Cl2P1eHnbBZeXkJG99RiT848cHVqGVz194TNMcSPLJ+MUsvgF2oZDyO1duL3d+PHQ6fwQ87sxnicae1Go8PDpTZ8ZgjsqljGYthJ1L50kR2WoQUnBanaZ7uRsRrXg9kZpwxz5BLi3eNOB52/njlOA5dV61ixYwyt0coxsFK2Ozf2sCOp2uJhhIsvriATe9eRFbhuU2ZlFLyt0fr2dUX5oHVlVwSPPNUzenEjkZJNjeTaG4m0dRMsqWZZGcnVlc3Vnc3VleX47q7sfv7z65Qw0jNQkgNcLlczsCXmXbs96VmJbiHzzxIF8MzCaMrTWxdY4tnejlKSBWKs2NeCr5l2Rx+pYkdT9US6opRtjybzTdVUbBg7M+yz4YHGtr5eXMnf7WgkBvOw0dVdjRKvLaWeE0NsZoTxGtqiNfWkmhqwuo67YNnNL8fPTt70LkWLcTIzkbPykILBtH9fjS/H83nc9xAOOULl+qaUiguZOaV4Nu25Nhrzbz2mxP0tkcpXJjJ1XetGHUVy4nyaneIf6hu4JrcTD67cLRZrJNDJhJEDx8mun8/kf37ie4/QKy6eqiLRAjM0lJclZV4Vq/GLC7CKCrGLC7CLCrCKCpC88z+7iWFQjF9zAvBl7bk+K42Xvt1DV3NYfLKAxOaSz8ebfEE9x6oZYHHzbdWLkCbgjKlbRM7fJj+V7fRv30b4R1vIMNhAPSsLDyrVxO46i14li3DtWgRrgULlKArFIozMqcF37Jsjr3Wws7fn6SrOUx2kY9rP7qaqovyT1uu+FyRUvKXh+roTVo8sq6KTOPcv5Kz43HCr75K77PPEnru+cFuGdeiRWTd+B58l1yCd+1ajJIS1WetUCgmzJwU/ETc4tDLjex69hShzhi5pX6uuXslizcWnvVHU2fLAw3tPNfZy78uKWVFYPwFlkYik0lCW1+k96mnCL3wAnYohBYIELjySgJXvhnfpk3Ttq6GQqGYX8w5wY9Fkvz0i68S6UtQXBXkyjuWTVnXzUgOhSL88/FG3pqTyT2leROzs6aGnsceo/uJJ7Da2tGzssi47loyr7kG3+bNaGqAVKFQTDFzTvDdXoN1by2neHEWJYuzpu06cdvmEwdPkqHrfG1F+VlVKDKRoPeZZ+j6yU+J7NoFuk7gyivJet97Cbz5zeqDGYVCMa3MOcEHuPi6ymm/xjdOtnKoP8qDaxaS7zqzUFvd3XT94hd0/fQhks3NmAsqKPjs3xB897sx8qdvX12FQjG7kFIiZQLbjmPb8VQ4MRQnE0g7AQiCwfVTfv05KfjTzeH+CF8/2cJ7C7N5e97YyyPHamrofPBBen71BDIaxbf5Mor+4YsErrxSLQqlUMwgtp3EtiNYdgzbimBZEWw7imVFsOwItjUibEdSIh1zhDotbMv4sLTT8snhaTD+IsAuVx5XXL59yv9uJfgTxJKSzxyuI8PQ+OfFpaelSynpf/kVOh/8Ef1bX0S4XGS+653k3PUhPMuWzoDFCsWFi5QS245jWSEsq59ksh/LclzS6sdK9pO0QljJUeKsMJYVdgTdjmBZUUfkrShSjr1ZyZnQNDea5kIIF5rmGjxODxtGAE0MHYuR+YQLTXejCRdCMwfjhGaiCRNNM9G0iU8AORuU4E+Q79e3sbM3zP+uXECea+j22eEwPU/+ms6f/Jh49XH0vDzyPv3nZN92G0Zu7gxarFDMHI5gR0kke0gmekkme0km+5zjZO9gXCI5kObEDRxbVj9SJs/qWprmQtcD6Lofw/Cn/Ax0dwG65kXTPSnfi655Bo913ZOKS8/jQdc86LoXTfOi626EuPBXCFWCPwHqo3H+o6aZa3IzubEgC4B4fQNdDz1E96OPYvf24l65guIv/QeZN9ygZtoo5hy2nSSR6CKe6CAR7yAe7yCR6CSe6BwKp/xEoodksm/c1rSu+zCMTAwjA8MI4nYX4vcvccTaCGDofnTD7/h6IC3sQ9cDg+KuaWrSw3gowZ8A/1TdCEj+bUkp4e2v0fmTHxN6/o8gBBlvv4acD34Q70UXXfCtAMX8wbaTJJLdw8U73kE80enEJTpJxDuJJzqIxztJJrvHKEnDNLNxuXJxmTkEAiswzayUkGdipnzDyMQ0gylxd0Re01TD6HyhBP8sebGzj1+3dfPnHfUkbvo8p2pr0bOyyP3oR8m+43bMoqlfP0ehmChSWiQS3WOK91B8J4lEB4lEN6MPIopBATfNHAKB5bjMXExXDi4zBzMl7APpppmFEGoiwmxHCf442NEoXX/ayuciBiXxBO/8579HX7+W4nvvJfOG69X6NYppRUqbRKJ7HPFO97sYW8CzMM1cXK4cAv4lmK5NQyKeEnBzUMSzEGL6NtNWzAxK8EfBjkYJbd1K3+9+T+iFF3h481WcuPkD/M/xvSz/9RO4Fy6caRMVFyi2HXda4KmuEqc/3PEHuk4cP13AR9/82zCyBoXa56siK+uSQcEe2Qo3jCw0Tf1zn++oJyBFoqWV/he3Etr6IqGXXkKGw+jZ2SRueh8PXn49b8vJ4Jar7pppMxWzBCktksl+ksk+klaf4yd7UyLeSTwl4EOC7nSjWFZozDIdAc/GNHPx+SrJCl6c6kIZ6kpxufJSXSjZSsAVE2bePjF2OEx41y7C27YTevFFYocPA2AUFhJ817vIvO5afJdcwueONxFr6uCflpw+515xYSKlTTIZGibWVnJAtIfEezBsjUwLnVG4wZkiaJopsTaz8XornD5xM2eoH9zMTvWT56gWuOK8MG+esGRHB5F9+4i88Qbh114ncuAAJJOg6/g2bCD/rz9D4M1X4l66ZHCWzeH+CD9t7ODusjyqfKqvfjZwdmKdJtjnINYAQjgf0DizSRzn8+Vh6BnDZpiMdE6XSg6a5lWztRSzjjkp+FZPD9EDB4js2z+4Q1SyqclJNAy8q1eTe/fdzvryF12EHhh979l/rm4kYGh8plLNwJkqhgS7l2Syx/nAJjH00U36BznDPsY5D2LtzPvOwNAz0PWJb2CvUMx2JiX4Qogc4OdAJVAL3Cql7BqRpxx4ECjCGX36rpTy65O57pmQ8TjHLr8CmXA+9jAXVOC76CI8d92Fd/UqPKtWofnG36T8T519PN/ZxxerSsgx52S9OGXYdoxYrIVYrDU1Xzv1UU4qfHazSAYQqXnbQQzTEWefb6ESa4ViChBSjr+Qz5gnC/EVoFNK+SUhxOeBbCnl347IUwwUSyl3CiEygDeAG6WUB8crf+PGjXLHjh0Ttqv70UcxS0rwrFqFHhx7cbOxsKTkmtePELJsXty0HPc8X+gskeglEqklHDlJLNpENNY05MeaicfbRz3PMIKD87RdrtxUOBvTSH2QY2Y4wm5kYhhBTDMTXfer+dwKxSQQQrwhpdw4Wtpkm67vAd6SCv8IeAEYJvhSyiagKRXuE0IcAkqBcQX/XMm6+eZJnf94SxcH+6N8e+WCeSP2UlpEIqcIhY7Q33+McKSWcPgkkchJEonOYXl1PYDHU4zbXURGYCVuTwkedzFudwEuV96gsKsvKBWK2cVkBb8wJehIKZuEEAVnyiyEqAQuAsZc91MIcS9wL0BFRcUkzZs4cdvmP080sybg5d2p9XLmGpYVobdvP329+wj1HxkUeduODuZxu4vxeReQn/92fL5KfN4FeL0L8HhKMIyMGbReoVCcK+MKvhDiDzj97yP5wkQuJIQIAL8E/lJK2TtWPinld4HvgtOlM5FrTAU/a+rkZDTOT9cuQpsDsyyklEQiJ+np3U1vz256encSCh1GSgsA08wlI7Cc0tI7CfiXEQgsxe9fgq5Pz/KsCoVi5hhX8KWUbxsrTQjRIoQoTrXui4HWMfKZOGL/UynlY+ds7TQTsWz+u7aFS4N+rs65cFux0WgTXV2v0Nn5Cp1drxCPOz+LrvvJzFzLgop7CQY3kJG5FrdrYnvxKhSKC5fJduk8CXwI+FLKf2JkBuFMRv4+cEhK+f8meb1p5YcN7TTHE/zvqgXnNIda2hKrO4bVHcOOJsGWCJeOFjAxcr1o7ulZm0RKm97ePbS1PUNb+/OEw9UAmGYOOdlvIjv7MoLBDfj9i9X6KArFPGaygv8l4BEhxD3AKeAWACFECfA9KeUNwBbgg8A+IcTu1Hl/L6V8apLXnlL6khb/c6qFq3Iy2JwVOOvzkj0xIvvaiR7pJH6qDxmzxsxr5HtxLwriXZWHuyqI0M99QFhKm66uV2lt+x1tbX8gHm9FCIPsrE2UlNxCTvYWAoFlasaLQqEYZFKCL6XsAN46SnwjcEMq/BIw6zvDv1PXRmfC4vOLis8qf+xkL33PnyJ6tAskGAVefOvzMUsDGNkeNK8BmkDGLazeOMm2CPH6PsK72ujf3ozwGPjW5hHYUoJZOPqHX6MRidTR1PRLmpofIxptQNO85OZeSUH+28nNvQrTzDzXW6BQKOY46osioCOe5Nt1rbwjP8i6jDN/lBU71UvvH04RO9qF5jfIuKoc/4ZCjLyzG+SUCZvosS4i+9rp39lK/2vNeJbnkHFlGe6Fo38zYNtJ2tqfob7+J3R3bwcEOTmXU1X1WfLzrkHX1bIPCoVifJTgA9881ULYsvncwrFb93YsSc/TtfRva0LzGwSvX4h/czGaa2J94sLU8K7Mxbsyl+A7FtK/rYnQq020fWcvnhU5BK9fiFngVDrJZIjGxkeoq/8h0WgDHk85ixZ9huKim/B4Sib1NysUivnHvBf85liCHzS0c3NRNsv8o7eUo8e66PrlMayeGIEtJWS+vXJKBmD1gIvMty0g48oy+l5upO+PdbR87Q18l+bTu/JFTrV8h0Sii6ysS1m65P+Ql/dWNeiqUCjOmXkv+P9d24wl4a9HWSBN2pLeP5yk7/k6jHwv+fetw71g6vvIhamT+ZZyvBtyaX7yefq32chdheRd8k5Kr7iRYHD9lF9ToVDMP+a14J+MxPhpUwcfKMljgXf4glt2JEnnw4eJHunCd3Eh2TdWIczpa113dPyJo8f+nXBJNflF7yZv782YL72VZJcL66Y4ekAtU6BQKCbHvBb8/zzRjCkEf7WgcFh8oj1Cxw/2k+yOkXVjFf5NxdO2tnk01syRI/9Ae/sf8HoXsHbNd8jLeytcJenb2kDvH04S+9pOsm9eind5zrTYoFAo5gfzVvAP90f4ZUsXn6gooNBtDsbH6/po/+F+API/ugZ35cRX2zwbpLRpaHyY6uovI2WSqqrPUVH+YTQt9aahCzKvKse7IofOhw/T8cMD+DcVEXzHogkPFCsUCgXMY8H/Sk0zAV3jkxVD671Fj3bR8ZODaAEXeXevxjzLqZYTJRpr5uCBv6arexvZ2ZtZvuzf8PkWjJrXLPJT8KmL6HmmltCLDcSO95Bz2zJc5Rfu0g8KhWJmmJeCv6s3zFPtPXxuYdHg5iaR/e10PHQYs9BH3t2r0TOmp8+8re0ZDh76O6SMs3zZv1FSctu43UXC0Mi6YRGeZTl0/eIorf+725nd85ZyhDbrv2lTKBSzhHkp+F+uaSLH1Lm3LB8YEntXeQZ5f7YKzTP1t8WyIhyr/ncaGh4iI2MVq1d9HZ9v4YTK8FRlUfgXG+j6VTW9z5wkeqyLnNuWYWSpD68UCsX4zLuFVl7u6uOFrj7+YkEhAUMfEvuywLSJfX9/Na/vuImGhoeoqPgIGy9+dMJiP4DmNci5fRnZtywl0dBPy9d2Ed7bNsUWKxSKuci8auFLKfnX402UuE3uKskjciBN7O9ePS1i39b+HAcOfAZNc7F+3Q/Jzb1i0mUKIfBfXIi7MpPOnx+h86HDRA93kvWeKjT3vPpJFQrFBJhXLfyn2nvY1RfmbxYWweEuOn56GFfp9Ii9lDYnTvwPe/fei89XyaWXPDElYp+Okesl/2Nrybi6nPCuVlq+sYvYqTH3llEoFPOceSP4SVvyHzVNLPG5eVenpOOhQ47Y3zP1Yp9Mhti3/1PUnPgaRYU3cvGGn0/b2jdC1wi+vZL8e9eCJWn79h56nz+FtM/7ZmEKhWKWM2/e/3/e3El1OMZ3grn0PHQYs2R6xD4cPsnefR+jv/84Sxb/PeXld0/bR1vpuBcGhw/oHuki++YlmPlnXv1ToVDMH+ZFCz9i2Xy1tpkNposNvzyJWewnfxq6cTo6X+L1HTcRi7Vy0fofUlFxz3kR+wEGB3RvW0aiJUzL13fS+0Id0lKtfYVCMU9a+N+vb6MpluCfdoRxFfvJv2eNs0HJFCGl5FTd96iu/gp+/2LWrf0OXm/FlJU/EYQQ+C8qwFOVRdcT1fT+rpbIvnayb16Kq/jsN1pRKBRzjznfwu9OJPnGiWa2tCfZ5PE6LfspFHvLinLw4F9TXf0l8vPfzsaLH50xsU9Hz3SR+4EV5Ny5HKs7Ruv/7KL76RPYseRMm6ZQKGaIOd/C/9obJ+mzbf6iWyf/ntVoPnP8k86SaLSRvfvuo6/vIIsWfYbKBZ84r1044yGEwLc2H3dVFj1PnSD0p3rCO1sIXr8Q3/oC9ZWuQjHPmNOCf2x3Ez8I9XJDn+DyD0xty76raxv79v85th1n3drvkpd39ZSVPdXofpOcW5bi31RE95PH6XrkKP3bmsh6dxWuMrUmj0IxX5izXTrhPa38y6F6hBD849VLp0zsnf76H7Br912YZhaXbHxsVot9Ou6KTAo+sZ7sm5eS7IzS+s3dtP/4IInm/pk2TaFQnAcmpYJCiBzg50AlUAvcKqXsGpHHA2wF3KnrPSql/IfJXHc8+ne18sLvj/HMJj9/VVZAeXBqpiZaVoTDh79Ac8sT5Oddw8qV/4lhXFgtZKEJ/BsL8a7OJfRSA30vNtByoAPv2jwy31qBWagGdhWKucpkm72fB56TUn5JCPH51PHfjsgTA66WUoaEECbwkhDiaSnltklee1Ss/gSdv6rmvy/zU+Qy+NSiwvFPOgsikXr27vs4odAhFi38KyorP4EQF+4LkuYxyHzbAgJvKqHvxQZCLzcS2deOd1UugS2luCozZ9V4hEKhmDyTFfz3AG9JhX8EvMAIwZdSSiCUOjRTbtomhut+k5dur2R/cyv/U1WCX5/8ZiEdnS+xf/9fABbr1t5PXt5Vkzd0lqD5TILXVhK4vJTQi/WEtjcT2d+BWeInsKUU37p8hHHhVmwKhWII4ejxOZ4sRLeUMivtuEtKmT1KPh14A1gMfEtKOfItID3vvcC9ABUVFRefPHlyQjb1Jy22bD9MkdvkqYuXoE2ilSqlxYkT3+RE7Tfx+xezds3/4vNVnnN5FwJ23CK8q5XQy40kW8NofhPf+nx8FxVglgZUq1+hmOUIId6QUm4cNW08wRdC/AEoGiXpC8CPzkbw09KzgMeBP5dS7h/P8I0bN8odO3aMl20YCVvyg4Y2NmT62Rg89/7oaKyZAwc+Q3f3doqKbmTZ0n/GMOZP/7aUklh1N/3bmogc7gRLYhR48V1UiG9tHkbu9OwGplAoJseZBH/cLh0p5dvOUHCLEKJYStkkhCgGWscpq1sI8QJwHTCu4J8Lpia4t7xg/IxnoL3jBQ4e/CyWFWHliq9QXPy+KbLuwkEIgWdJNp4l2djhBOF97YR3tdL7+1p6f1+LUejDuzwHz4ocXBWZak6/QnEBMNk+/CeBDwFfSvlPjMwghMgHEimx9wJvA748yetOC5YV5XjNf1FX9wCBwHJWr/oGfn/VTJs142g+k8CmYgKbikl2Rokc7CB6qIO+Fxvo+1M9mt/AvSgL98IgroVBzEKfqgAUilnIZAX/S8AjQoh7gFPALQBCiBLge1LKG4Bi4EepfnwNeERK+ZtJXnfK6enZzcFDnyUcrqG09AMsWfx36LraOnAkRo6HjMtLybi8FDuSJHq0i+iRTmI1PUT2tQOg+QxcFZm4ygKYpQFcpRnomdOzR7BCoTh7JjVoO92cSx/+RLHtGDUnvsHJk9/F7S5k5Yovk5OzZVqvOVdJdkWJ1fQQO9FD/FQvybbI4HwsLcOFWeTDzPdhFHgx8p2wlmGqgWCFYgqZVB/+XKanZyeHDn+B/v6jFBffwtIlX7jgPqSaTRjZHoyLPfgvdr59sGMWiaYQ8foQiYYQibYw/TtakHFr8Bzh1jFyPehZHowsN3q6y3Sh+10IU00LVSimgnkp+PF4B9XHv0JT06O43UWsW/u9OTW3fraguXXclUHclcHBOCkldm+cRFuYZFuERGsYqyuG1RkhVtONjFqnlSPcOlrARPebaAEXesBE85toXgPh0dE8Bprn9LAwNfX2oFCkMa8EX0qLhoaHOV7zVSwrzIKKe6ms/NS8mm450wgh0INu9KAbFp8+g9eOJrG6YyS7Y9i9caz+OHYogdWfcPzOKPG6Xuz+BNjjXEwTCJeOcGlopuZUAGaqInCl/PR4l4YwHIcuELqGMBwfXThpuoCUL3QtLSyGzhvIp2mgoSodxaxhXgi+lJLOzhepPv6fhEIHyc66jGXL/gm/f/FMm6YYgeYx0IoMzKIzV8JSSmTcwo5ayGgSO2phR5JD4WgSGbWQcQuZsLETji8TtnNeXxyZsJDxVFwqfVq+AddEqgJI8zUxajy6Nmba2Odqp6fro+QTAoRjj3OcCou08Mh4DRAp24SzFtPA8VC+EfED5wiGyiYtDlL+0LEYcTxaHsSFUXkOGxcd7XmSIwJjPHPT8YX7nBf8np7dHD/+n3R1b8PjKWPVqq9RWPDOC+LBUYyNEALhNtDcBgTdU1KmlBIsibQkMmmnwnZaeCDeRiadNFJxMpWX5MA5EmzpbCZvDfi249tOWaP6g+k2MpEqc2S6Jccsh/myef0YFcdQhZFW0YxAjhTcEcHTj88gzNN0u7WAScn/uWzKy52Tgi+lpKt7Gydr/5fOrpcxzRyWLvkipaW3o2lTIw6KuYcQAgyBMAD35NdgmgmklE5Xl20PVQ6WBCkdoRuoKAbCUiJtHBW0JUic8+TIfAxWMsiR+UaE5cjwgHGp/404liPTB7zBeJnuDQXGKFeOjEurBBgRHFYnjMw3WoUxatzpkaNWNqNFnlZROQfCNT0TFeac4CeTfeza/Wf09u7C5cpjcdXfUlp6J4YRmGnTFIppRwgBOqDro2qTYn4z5wTfMDLwesspLrqJ4uKb0XXVolcoFAqYg4IPsHrVf8+0CQqFQjHrUF+0KBQKxTxBCb5CoVDME5TgKxQKxTxBCb5CoVDME5TgKxQKxTxBCb5CoVDME5TgKxQKxTxBCb5CoVDME2b1jldCiDbg5Dmenge0T6E5U4Wya+LMVtuUXRND2TVxzsW2BVLK/NESZrXgTwYhxI6xtvmaSZRdE2e22qbsmhjKrokz1bapLh2FQqGYJyjBVygUinnCXBb87860AWOg7Jo4s9U2ZdfEUHZNnCm1bc724SsUCoViOHO5ha9QKBSKNJTgKxQKxTxhzgm+EOI6IcQRIUS1EOLzM2hHuRDij0KIQ0KIA0KIv0jF/6MQokEIsTvlbpgh+2qFEPtSNuxIxeUIIZ4VQhxL+dnn2aZlafdltxCiVwjxlzNxz4QQDwghWoUQ+9Pixrw/Qoi/Sz1zR4QQ186Abf8phDgshNgrhHhcCJGViq8UQkTS7t23z7NdY/525+uejWHXz9NsqhVC7E7Fn8/7NZZGTN9zJqWcMw5nN8/jwCLABewBVs6QLcXAhlQ4AzgKrAT+EfibWXCvaoG8EXFfAT6fCn8e+PIM/5bNwIKZuGfAm4ENwP7x7k/qd90DuIGFqWdQP8+2vR0wUuEvp9lWmZ5vBu7ZqL/d+bxno9k1Iv2/gC/OwP0aSyOm7Tmbay38S4FqKWWNlDIOPAy8ZyYMkVI2SSl3psJ9wCGgdCZsmQDvAX6UCv8IuHHmTOGtwHEp5bl+aT0ppJRbgc4R0WPdn/cAD0spY1LKE0A1zrN43myTUj4jpUymDrcBZdN1/YnYdQbO2z07k11CCAHcCvxsOq59Js6gEdP2nM01wS8F6tKO65kFIiuEqAQuAranoj6VevV+4Hx3m6QhgWeEEG8IIe5NxRVKKZvAeRiBghmyDeB2hv8jnA33bKz7M9ueu7uBp9OOFwohdgkh/iSEuGIG7Bntt5st9+wKoEVKeSwt7rzfrxEaMW3P2VwTfDFK3IzOOxVCBIBfAn8ppewF/heoAtYDTTivkzPBFinlBuB64JNCiDfPkB2nIYRwAe8GfpGKmi33bCxmzXMnhPgCkAR+mopqAiqklBcBnwEeEkJknkeTxvrtZss9u4PhDYvzfr9G0Ygxs44SN6F7NtcEvx4oTzsuAxpnyBaEECbOD/lTKeVjAFLKFimlJaW0gfuZxlf/MyGlbEz5rcDjKTtahBDFKduLgdaZsA2nEtoppWxJ2Tgr7hlj359Z8dwJIT4EvBN4v0x1+qZe/ztS4Tdw+n2Xni+bzvDbzfg9E0IYwHuBnw/Ene/7NZpGMI3P2VwT/NeBJUKIhalW4u3AkzNhSKpv8PvAISnl/0uLL07LdhOwf+S558E2vxAiYyCMM+C3H+defSiV7UPAE+fbthTDWl2z4Z6lGOv+PAncLoRwCyEWAkuA186nYUKI64C/Bd4tpQynxecLIfRUeFHKtprzaNdYv92M3zPgbcBhKWX9QMT5vF9jaQTT+Zydj9Ho8+mAG3BGu48DX5hBOy7Hed3aC+xOuRuAHwP7UvFPAsUzYNsinNH+PcCBgfsE5ALPAcdSfs4M2OYDOoBgWtx5v2c4FU4TkMBpWd1zpvsDfCH1zB0Brp8B26px+ncHnrVvp/K+L/Ub7wF2Au86z3aN+dudr3s2ml2p+B8C943Iez7v11gaMW3PmVpaQaFQKOYJc61LR6FQKBRjoARfoVAo5glK8BUKhWKeoARfoVAo5glK8BUKhWKeoARfoVAo5glK8BUKhWKe8P8DOQktgGLbm7MAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "tau = .05\n", "niter = 200\n", "x_pro = np.zeros((p, niter))\n", "v = np.random.randn(p)\n", "for it in range(niter):\n", " f, g = nabla_f(v)\n", " v -= tau*g\n", " x_pro[:, it] = v * u_opt(v)\n", "plt.plot(x_pro.T);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "While the trajectory of ISTA are piecewise smooth, and are not differentiable when crossing 0, the trajectory of this bilevel method can smoothly cross or approach 0. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Non-convex Pro method: using BFGS\n", "\n", "One can show that $f$ is a smooth function. Although it is non-convex, it has no spurious local minima and its saddle points (such as $v=0$) are \"ridable\" (aka strict saddle), so that trajectory of descent method are not blocked as such saddle points." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Loading a dataset from LibSVM using https://github.com/mathurinm/libsvmdata. \n", "In some cases, the $A$ matrix is sparse." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset shape: n=60000, p=780\n" ] } ], "source": [ "name = 'w8a'\n", "name = 'connect-4'\n", "name = 'mnist'\n", "from libsvmdata import fetch_libsvm\n", "\n", "A, y = fetch_libsvm(name)\n", "n, p = A.shape\n", "print(f'Dataset shape: n={n}, p={p}')" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "import scipy\n", "\n", "if scipy.sparse.issparse(A):\n", " A = A.toarray()\n", "\n", "from sklearn.preprocessing import StandardScaler\n", "A = StandardScaler().fit_transform(A)\n", "y -= y.mean()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We select $\\lambda$ as a fraction of $\\|A^\\top y\\|_\\infty$." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "la = .1 * norm(A.T @ y, ord=np.inf)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then re-define the function computing $\\nabla f$, select the most efficient formula to compute $u^\\star(v)$." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "def u_opt(v): \n", " S = A @ np.diag(v**2) @ A.T + la * np.eye(n)\n", " return v * (A.T @ np.linalg.solve(S, y))\n", "\n", "def nabla_f(v):\n", " u = u_opt(v)\n", " f = 1/(2*la) * norm(A@(u*v) - y)**2 + (norm(u)**2 + norm(v)**2) / 2\n", " g = u * (A.T @(A@(u*v) - y)) / la + v\n", " return f, g" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Denoting $C \\triangleq A^\\top A \\in \\mathbb{R}^{p \\times p}$ the correlation matrix and $h \\triangleq A^\\top y \\in \\mathbb{R}^p$, since\n", "$$\n", " \\|Ax-y\\|^2 = \\langle Cx,x \\rangle + \\langle y,y \\rangle - 2 \\langle x,h \\rangle\n", "$$\n", "the Lasso problem only depends on $C$ and $h$.\n", "When $n>p$, it is faster to implement the solver in these variables." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "C = A.T@A\n", "ATy = A.T@y\n", "y2 = y @ y \n", "\n", "if scipy.sparse.issparse(C):\n", " C = C.toarray()\n", "\n", "def u_opt_cov(v): \n", " T = np.outer(v, v) * C + la * np.eye(p)\n", " return np.linalg.solve( T, v * ATy )\n", "\n", "def nabla_f_cov(v):\n", " u = u_opt(v)\n", " x = u * v\n", " E = (C@x) @ x + y2 - 2* x @ ATy\n", " f = 1/(2*la) * E + (norm(u)**2 + norm(v)**2 )/2\n", " g = u * ( C@x-ATy ) / la + v\n", " return f, g" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "if n < 200 and p < 200:\n", " v = np.random.randn(p)\n", " f1,g1 = nabla_f(v)\n", " f2,g2 = nabla_f_cov(v)\n", " print(f'Function difference, should be 0: {(f1-f2):.2e}')\n", " print(f'Gradient difference, should be 0: {norm(g1-g2):.2e}')" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "pn, using full mode')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In order to obtain an efficient numerical scheme, one should use a quasi-Newton [L-BFGS](https://en.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%80%93Goldfarb%E2%80%93Shanno_algorithm).\n", "The option `maxcor` controls the size of the memory." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "# callback to store intermediate results\n", "flist = []\n", "tlist = []\n", "t0 = time.time()\n", "def callback(v):\n", " f, g = nabla_f(v)\n", " flist.append(f)\n", " tlist.append(time.time() - t0)\n", " return f, g\n", "\n", "# run L-BFGS\n", "import scipy.optimize\n", "v0 = np.random.randn(p)\n", "opts = { 'gtol': 1e-30, 'maxiter':1000, 'maxcor': 10, 'ftol': 1e-30, 'maxfun':10000 }\n", "result = scipy.optimize.minimize(\n", " callback, v0, method='L-BFGS-B', jac=True, tol=1e-30, options=opts)\n", "# retrieve optimal solution \n", "v = result.x\n", "x_pro = v * u_opt(v)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEICAYAAACavRnhAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmz0lEQVR4nO3dd3iV9f3/8ec7mxkICQQy2CB7iEFBESsoWBS3oLWto1atfrW17c/u8f06Wju01lHrblXqqqKiqIgCskGmDFmSsBKIhBkSkvfvj8Q2pcycnHOf5Lwe15WL3HfOeJ3Ly7zy+dz3/bnN3REREQlFXNABRESk/lOZiIhIyFQmIiISMpWJiIiETGUiIiIhU5mIiEjIVCYiIhKyhKADBCE9Pd07dOgQdAwRkXplwYIF290943A/q/dlYmY9gNuAdGCKuz9yrOd06NCB+fPnhz2biEhDYmafH+lnUTnNZWZPmlmhmS07ZP8oM1tlZmvM7E4Ad1/h7jcClwODgsgrIhLrorJMgKeBUTV3mFk88BAwGugJjDezntU/uwCYAUyJbEwREYEoLRN3nwYUH7I7D1jj7uvcvQyYAIytfvxEdx8CXBXZpCIiAvXrmEkWkF9juwAYbGbDgYuBZGDSkZ5sZjcANwDk5uaGLaSISCyqT2Vih9nn7v4h8OGxnuzujwGPAQwaNEhLJYuI1KGonOY6ggIgp8Z2NrA5oCwiIlJDfSqTeUBXM+toZknAOGBiwJlERIQoLRMzewGYBXQ3swIzu87dDwK3AJOBFcCL7r48krnmri/moalrWJy/E91UTETk36LymIm7jz/C/kkc5SB7uM1au4M/vr+a+yav4rKTs/ntpX0xO9yhHBGR2BKVI5NodduIrsz7yQiuO70jLy0o4Pm5G4OOJCISFVQmJyijWTI/Pq8Hp3ZK4w/vrg46johIVFCZ1EJ8nDGkczo79pZRdrAy6DgiIoFTmdRSWpMkAL7YVxZwEhGR4KlMaqlVdZns2KMyERFRmdTSlyOT4r0qExERlUkttWpaPTLZeyDgJCIiwVOZ1FJak2RA01wiIqAyqbUWjRKJM01ziYiAyqTW4uKMlo2T2KEyERFRmYSiVdMkinXMREREZRKKtCZJFO8tY39ZBbtLy4OOIyISGJVJCFo1SWbH3jJ+9OoSrnlqXtBxREQCE5WrBtcXaU2S2LGnjBlrtlNeoSXpRSR2qUxCkNYkiZL9/57eKi2vICUxPsBEIiLB0DRXCL68cPFLW0tKA0oiIhIslUkIvlxS5Utbd6lMRCQ2qUxC0Kr6KvjM5imARiYiErtUJiH4cpprdJ9MQCMTEYldKpMQdGjVhIsHZPG1U9vTNDlBIxMRiVk6mysESQlx/OGK/gC0aZ7MNo1MRCRGaWRSRzJTUzTNJSIxS2VSRzKbN2KbprlEJEapTOpIZmoy23YfoKJSV8KLSOxRmdSRzOYpVFQ6hbs1OhGR2KMyqSMnt08D4O2lWwNOIiISeSqTOtKzXXP65bTg+bkbcddUl4jElnpfJmbWxMyeMbO/mtlVQWa5anAuawr3MHd9cZAxREQiLirLxMyeNLNCM1t2yP5RZrbKzNaY2Z3Vuy8GXnb3bwEXRDxsDef3bUeTpHgmLt4cZAwRkYiLyjIBngZG1dxhZvHAQ8BooCcw3sx6AtlAfvXDKiKY8b80Soonr2Mas9btCDKGiEjERWWZuPs04NC5ojxgjbuvc/cyYAIwFiigqlAgCj7PaZ1bsa5or66GF5GYEvgv3xOQxb9HIFBVIlnAq8AlZvYI8MaRnmxmN5jZfDObX1RUFLaQp3VKB2C2RiciEkPqU5nYYfa5u+9192vc/SZ3f+5IT3b3x9x9kLsPysjICFvInu2a0zwlgVlrVSYiEjvqU5kUADk1trOBqDvSHR9n5HVsxfTPtusUYRGJGfWpTOYBXc2so5klAeOAiQFnOqxze7Vh0879LNy4M+goIiIREZVlYmYvALOA7mZWYGbXuftB4BZgMrACeNHdlweZ80hG9c4kOSGO1xdtCjqKiEhEROX9TNx9/BH2TwImRTjOCWuWksjInm14c8kWfjamJ4nxUdnZIiJ1Rr/lwuSiAVkU7y3jw1XhO3NMRCRaqEzCZFi3DDKaJfOPeRuDjiIiEnYqkzBJjI/jspOz+WBloe4NLyINnsokjK44JYdKh5fm5x/7wSIi9ZjKJIzat2rCkM6teH7uRnbsORB0HBGRsFGZhNl3R3ajeG8ZVzw2W3dhFJEGS2USZqd0SOPZa/PYvHM/1z8zn31lB4OOJCJS51QmETC4UyseHD+AZZtKuG3CIioqtcyKiDQsKpMIObtHG34+pifvfbqNuyetCDqOiEidisor4Buqbw7tyIYd+3hixnp6tG3OpSdnH/tJIiL1gEYmEfbTr/bgtE6t+Mk/l7JsU0nQcURE6oTKJMIS4uN48MoBpDVJ4sa/L+CLvWVBRxIRCZnKJADpTZN55GsnU7jrALf9QwfkRaT+U5kEpH9OC349thfTVhfxh/dWBR1HRCQkKpMAjcvLZXxeDg9NXcs7y7YGHUdEpNZUJgH75QW96JfTgu+/tJg1hXuCjiMiUisqk4AlJ8TzyFUDSU6I444XF1Gp4yciUg+pTKJAuxaN+OmYHiwuKOHVT3SrXxGpf1QmUWJsvyz657TgN++sZOc+nS4sIvWLyiRKxMUZ/3dhb77YW8ZPXluGu6a7RKT+UJlEkd5ZqXx3ZDfeWrKFJ2asDzqOiMhx09pcUebGMzuztKCE/3trBeUVzo1ndsLMgo4lInJUGplEmfg448ErBzCmb1t+885Kvv/SEl0hLyJRTyOTKJQYH8efxg2gU0ZT/jTlM7q1acq3z+wcdCwRkSPSyCRKxcUZ3x3RldG9M/ndu6u0wrCIRDWVSRQzM+6+qA/pTZP59t8WsH3PgaAjiYgcVr0vEzPrYWaPmtnLZnZT0HnqWssmSTx29SC27znAzX9fSNnByqAjiYj8l0DLxMyeNLNCM1t2yP5RZrbKzNaY2Z1Hew13X+HuNwKXA4PCmTcofbJT+e2lfZm7oZhfTNQ1KCISfYIemTwNjKq5w8zigYeA0UBPYLyZ9TSzPmb25iFfraufcwEwA5gS2fiRM7Z/FjcP78wLc/O57pn5ukpeRKJKoGXi7tOA4kN25wFr3H2du5cBE4Cx7r7U3ccc8lVY/ToT3X0IcFVkP0Fk/eDc7vzqgl7M+Gw7Fz88k8937A06kogIEPzI5HCygPwa2wXV+w7LzIab2Z/M7C/ApKM87gYzm29m84uKiuoubQSZGd8Y0oHnvjWY4n1lXPnXOezQQXkRiQLRWCaHu9z7iAcJ3P1Dd/8fd/+2uz90lMc95u6D3H1QRkZGnQQNyikd0njmmjyK9hzg5ucWUlpeEXQkEYlx0VgmBUBOje1sYHNAWaJWv5wW3HdpX+asL+amvy9QoYhIoKKxTOYBXc2so5klAeOAiQFnikpj+2dx78V9mLqqiF9OXB50HBGJYUGfGvwCMAvobmYFZnadux8EbgEmAyuAF91dvymPYFxeLjcP78yEefm8vXRL0HFEJEYFujaXu48/wv5JHOVguvyn20d04+M127n1hU/47va93HRmZ+LitNKwiERONE5zyQlKSojj2esGM6p3JvdNXsWtEz6hZH950LFEJIZo1eAGIrVRIg+OH0Df7FTueXslk5Zu4azurXn4qoGkJMYHHU9EGjiNTBoQM+OGYZ157eahfHtYZ6auKuTWFz5h267SoKOJSAOnkUkD1C+nBf1yWpDZPJlfvvEp76/Yxnl92nLHyG50ymgadDwRaYBUJg3YN4d25IxuGbw0v4BnZm5g0tItnH1Sa248szODOqQFHU9EGpDjmuYys8xwB5Hw6JzRlDtHn8S0H57FrWd14ZONO7n8L7N4eUFB0NFEpAE5ZpmYWRbw/LGWgpfoltEsme+d051pPzyLIZ3T+f5Li3l21oagY4lIA3HMMnH3TVRd87Ey/HEk3JokJ/D4NwYxsmcbfv76cu6bvJKDFbrhloiE5rimudz9d+7+WpizSISkJMbz8FUDGXdKDg9NXculj87ScvYiEhKdGhyjEuPjuPeSvvz5ygGsK9rDeQ9M56/T1um2wCJSK8dzzOSB6n8bhT+ORNqYvu145/Zh5HVM465JKzj3/mlMXr6VykrdGlhEjt/xjEzOrv53RjiDSHDatWjEU9fk8dQ1p2AG3/7bAs69fxpz1u0IOpqI1BPHUybvmNksINPMrjWzk80sJdzBJPLO6t6aybcP4/4r+lNeUcn4v87m3rdXsrtU63yJyNGZ+7GnM8ysE/Ah8AzQB+gFlAHL3P2KcAYMh0GDBvn8+fODjhHV9hw4yC8nLuflBQWkN03i9hHduDIvV6sRi8QwM1vg7oMO+7PjKZPqF+nm7qtrbDcFerv77LqJGTkqk+O3OH8nd721grkbirl4YBb3XdqPeBWKSEyqkzJpSFQmJ8bdefCDNfzhvdVkt2xEtzbN6JjehI7pTeiU0YRO6U1p0zwZM5WMSEN2tDLR2lxyTGbG/5zdlawWjfhgZSHrtu9l5trtlJb/+zTiXu2a85PzejCkS3qASUUkKCc8MjGz8939jTDliQiNTEJXWels3VXKuqK9rNy6i6c+3sCmnfs5+6TW/Oi8HnRprdWJRRqaOp3mMrMl7t63TpIFRGVS90rLK3jq4w08PHUN+8oruDIvl9tHdKVV0+Sgo4lIHanraS5NjMt/SUmM56bhnbl8UDb3v/8Zz8/dyKsLCxjSJZ3BHdMY3LEVPds118F7kQaqNmUSe0fs5bi1aprM/17Ym28M6cDj09cxa90O3vt0GwDNkhM4s3sGv7ygF+kasYg0KDoAL2HRpXVT7r2kajZ0a0kpc9bvYM76Yl5ZUMD8DV9wzyV9OKt764BTikhd0UKPEnaZqSmM7Z/F3Rf14dWbh9A4KZ5rnprH1U/MYeXWXUHHE5E6UJsy2VbnKSRm9GqXyju3D+NnY3qypKCE8x6Yzt2TVlCue6qI1GsnXCbuPjIcQSR2JCXEcd3pHfnoB8MZl5fLY9PWcfHDM3lXqxWL1Fua5pLAtGicxN0X9eGhKwfyxb4ybvjbAkb88SP+MW8jBw5WBB1PRE6AllORqHCwopJJy7by6Idr+XTLLjKaJXPxgCyuGtye3FaNg44nItT9RYtNgFJ3j4o/Hc0sDvhfoDkw392fOdZzVCbRy92ZsWY7z8zcwIerinDg/L5tGZeXyykd0nSdikiAQrposfqX9TjgKuAU4ACQbGZFwCTgMXf/rJbBngTGAIXu3rvG/lHAA0A88Li733uUlxkLZAHFQEFtckj0MDPO6JrBGV0z2LarlEc/WstL8wt4bdFm0pokMbZ/O752ans6Z2i5FpFocsyRiZl9BLwPvE7V/Usqq/enAWcBVwL/dPe/n/Cbmw0D9gDPflkmZhYPrAZGUlUO84DxVBXLPYe8xLXVX1+4+1/M7GV3v/RY76uRSf2yr+wgU1YUMnn5ViYv30p5hTOsWwa/OL+nSkUkgkJdTmWEux/uVnt73P0V4BUzS6xNMHefZmYdDtmdB6xx93UAZjYBGOvu91A1ivkPZlZA1Y26AKJi6k3qVuOkBM7v147z+7WjaPcB/jFvI3+dvp7RD0znkoFZXHpyNgNzW2oJfJEAHfNsriMUCcCvanyfVzdxgKopq/wa2wXV+47kVeBcM3sQmHakB5nZDWY238zmFxUV1U1SibiMZsnc8pWuvPe9YVzYvx2vfbKZSx6Zxcg/TmO27lkvEphan81lZsOBbsBeoIe7/7SWr9MBeLPGNNdlwLnufn319tVAnrvfWqugh6FproZjz4GDTFq6hYenrmHDjn2M7p3Jt4Z1YmBuy6CjiTQ4dX5zLDN7AigB+gOz3f3HtY/3XwqAnBrb2cDmOnx9aUCaJidw+aAcxvRty58/WMNzczby9rKtnNOzDVecksMZXTNIStDlVCLhFsrIpBEwkKozvHq4+7dr+Tod+M+RSQJVB+DPBjZRdQD+SndfXqugh6GRScO1r+wgT0xfz2PT17G79CAZzZIZ2bMNLRolckrHNE7r1IqUxPigY4rUS3VynYmZzQR+4u5T6zDYC8BwIJ2qNb9+4e5PmNl5wP1UncH1pLvfVVfvCSqTWFB2sJLpnxXxt9mfszh/J7tLD3Kw0klrksSF/bMY2qUVg9qnkdq4VueOiMSkuiqT3lQddG8B/NTdZ9VZwghTmcSe0vIKZq3dwYR5G/lgZSHlFY4ZdG/TjCGd0zm/X1v657TQGWEiR1HXV8APBH5dvflTd18UWrzIU5nEttLyChbl72Tu+mLmbShmzvpiyg5WkpvWmLH92zG2fzu6tG4WdEyRqFPXZdIc6AFcA1zv7vXuBlsqE6lpV2k5k5dt5fVFm5m5djuVDj3bNmds/3ac3L4lHdOb6F72ItTdNNcHQFegFPi0+mt5ba58D5rKRI6kcFcpby7ZwuuLN7M4fycAifHG+LxcvtqnLSdlNtdxFolZdVUmA4EV7r6/LsMFQWUixyO/eB9ri/Ywefk2Xpqfz8HKquMsPTKbc16fTJokJ1BeUclZ3VvTpXVTHW+RBi+kMjEz82M86HgeE01UJnKiinYfYPnmEpYUlPDR6iIWfP7Ff/y8Q6vGjOzZhpE9MxmY24KEeF3bIg1PqGUylaolS15394019icBpwPfAKa6+9N1ljjMVCYSqm27SjGg0uH9Fdt499NtzFq7nfIKp1lKAgNyW9I3K5XeWan0zU6lbWqKRi5S74V6BfxnVC2g+E8zawvsBFKougbkXeCP9fGMLpFQtGme8q/vv3Zqe752ant2lZYz47PtTP9sO4vyd/LIR2upqL4NcasmSQxs35Ir83I5o2u6Ri7S4BzPyGSZu/c2s4VULeiYAex3950RyBcWGplIJJSWV7Biyy6Wbfr39Fjh7gOkNkrk9K7pnNk1g8Gd0shNa6xRi9QLoY5MJpvZLKAN8HVgMVBnS5uINFQpifEMyG3JgOpFJ8sOVjJlxTY+WFnIR6uLeGvJFgBaN0smr2Magzu1YkyftrRskhRkbJFaOa6zucysE/Ah8AzQB+hF1T1Elrn7FeEMGA4amUjQ3J3V2/Ywd0Mx89YXM3d9MVt3lZKUEMcZXdIZ2iWdM7qm07WNLp6U6BHyqsHuvs7MRrj76hov2hTofZSnicgRmBndM5vRPbMZV5/aHndn1bbdTJibz0eri5iyshCAvI5pfOesLgzrmq6pMIlqtV41uD7TyESiXcEX+3hn2VaemLGeLSWl9MlK5bazu3J2j9YqFQlMnS6n0hCoTKS+KDtYyasLC3jowzXkF++nb3Yqt4/oylndVSoSeUcrE52fKBLFkhLiGJeXywd3DOe3l/SleG8Z1z49n4sfmcmyTSVBxxP5F5WJSD2QGB/H5afkMPX7w7n34j7kF+/ngj/P4H/f/JS9Bw4GHU9EZSJSnyTGV41UptxxJuPzcnlixnpG/uEj3l66hYMVlUHHkximMhGph1IbJXLXRX145abTaJaSyE3PLSTv7ik89fF6Kitj7zioBE8H4EXqufKKqoshn5uzkemfbad/TguGdU3n0pNzyG3VOOh40oDobK5DqEykIXJ3Xl5QwKMfrWX99r00Tkrg5+f35IJ+7UhJjA86njQAKpNDqEykoSv4Yh+3PP8Ji/J30iw5gdF9MrloQDa7Ssv5zTsrue3sroztnxV0TKlnVCaHUJlILKiodGat3cE/P9nEO8u2sLesAoCUxDjKK5xfXdCLcafkaAVjOW4qk0OoTCTW7C+r4L0V2yjec4Cx/bO46bkFzF5XTHbLRlzYP4vz+7Wje6bWAZOjU5kcQmUisc7dee/TbTwzawOz1u6g0iEnrREDc1syMLcleR3TOCmzma6yl/8Q8kKPItKwmBnn9MrknF6ZFO0+wDvLtvDxmh3MWruD1xdtBuCkzGYM796a/jmp9MtpQWZz3S1SjkwjExH5F3dnc0kpH6ws5NWFBSzbVEJ5RdXviNbNkumX0+JftyM+tVMrGiXpLLFYommuQ6hMRI7Pl3eLXJy/k8UFJSzO38m67XsBaJQYT17HNHq2a06Pts3p2bY5HdObEB+n0UtDpWkuEamVQ+8WCbC7tJzF+SW8vWwLCzfuZOb0df8avTRLSeDUTq3ok5VKdstG9M1OpVN6U+JUMA1evS8TM+sJ/BLYAUxx95eDTSTSsDVLqbqH/eld04GqZfLXFO5h+eYSFm78go/X7OC9T7f96/FNkxMY1KEl5/Vuyzm92tCisW5L3BAFOs1lZk8CY4BCd+9dY/8o4AEgHnjc3e89ymvcAcx19+lmNtHdLzjW+2qaSyS8Sssr2Fi8j8X5O1mUv5Ppn21nY/E+EuKMoV3S+WrftpzbM5PUxolBR5UTELXHTMxsGLAHePbLMjGzeGA1MBIoAOYB46kqlnsOeYlrq//9BbAPGOLuQ4/1vioTkchyd5Zt2sWbSzfz1pItFHyxn4Q44/Su6Xy1T1vOUbHUC1FbJgBm1gF4s0aZnAb80t3Prd7+EYC7H1okh75OPPCqu4891nuqTESC4+4sKShh0tItvLlkC5t27icpPo5RvTO5anAueR3TdApylKpvB+CzgPwa2wXA4CM9uLqMfgw0Ae47yuNuAG4AyM3NrYucIlILZka/nBb0y2nBnaNPYnFBCa99solXFhYwcfFmurZuylWDcxmXl6sFKuuRaByZXAac6+7XV29fDeS5+6119Z4amYhEn/1lFbyxZDPPzdnI4vyd5KQ14q4L+zCsW0bQ0aRafbsHfAGQU2M7G9gcUBYRiZBGSfFcPiiH178zlOevH0xKQjzfeGouf3xvNSX7y4OOJ8cQjWUyD+hqZh3NLAkYB0wMOJOIRNCQLum8fstQxvRtxwNTPuPUu6dw5ytLWLapJOhocgRBn831AjAcSAe2Ab9w9yfM7DzgfqrO4HrS3e+qy/fVNJdI/bG0oIS/z/6c1xdvorS8kgG5Lfj6ae0Z3butjqlEWFSfzRUElYlI/VOyr5yXFxbw99mfs377XtKbJnPn6JO4eECWrrCPEJXJIVQmIvVXZaXz8drt/OG91XyycSfZLRtx7dCOfP209rrRV5jVtwPwIiJHFBdnnNE1g1duHMKD4weQ1aIRv37zUy5+ZCafbt4VdLyYpTIRkXopLs44v187JtxwKg9dOZDNO/dz/p9ncN/klZSWVwQdL+aoTESkXjMzvtq3Le9/70wuGpDFQ1PX8tU/TWfB518EHS2mqExEpEFo0TiJ313Wj2euzaO0vJJLH53Jr9/4VNeoRIgOwItIg7PnwEF+8/ZK/jb7cxLijNM6t+KcXpmc37etlsAPgc7mOoTKRCQ2LNtUwhtLNvPu8m2s376XRonxjM/L5bYRXUltpFWKT5TK5BAqE5HY4u58umUXT328gVcXFtCycRJ3jj6JSwZm6xqVE6BTg0UkppkZvdql8rvL+vHGrafTvlVjfvDyEi58+GOmf1YUdLwGQWUiIjGlV7tUXr5xCL+/rB879pRx9RNz+dlry3Q6cYhUJiISc+LijEtOzuaD75/JDcM68bfZn3PeA9OZuWZ70NHqLZWJiMSs5IR4fnxeD569No+Dlc6Vj8/hu/9YRMk+nU58olQmIhLzhnXL4N3vDuPWr3ThjcWbGfPn6cxZtyPoWPWKykREBEhJjOeOc7rzj2+fxsEK54rHZnPV47NZV7Qn6Gj1gspERKSGk9u35IM7hvOzMT1ZWlDC6Aem8/t3V7GrVFNfR6MyERE5RKOkeK47vSPvfe9MRvZsw4MfrGHYb6fyl4/W6qyvI1CZiIgcQZvmKfz5yoG8eevp9M9pwT1vr+TM+6YycfHmoKNFHV0BLyJynOauL+auSStYnL+Ts09qTa+sVC4ZmEX7Vk2CjhYRWk7lECoTEamtgxWVPPjBGv4xL5/C3aXEmXHT8M58d0S3Br80i8rkECoTEakLhbtLufftlby6cBPn9GzDfZf2I7Vxw11AUmtziYiEQetmKfz+sn78bExPpqws5Jz7P2LqqsKgYwVCZSIiEgIz47rTO/LazUNJbZTINU/N4/+9vITdMXYqscpERKQO9MlO5Y1bT+fGMzvz0oJ8Rt0/nbeWbCFWDiWoTERE6khyQjx3jj6Jl28aQrOUBL7z/EK++dQ8inYfCDpa2KlMRETq2MDclrz1P2fwqwt6MXvdDs7540c8M3NDg77gUWUiIhIG8XHGN4Z04I1bT+ekzOb8YuJyBt89hV+/8Sl7DhwMOl6dq3dlYmadzOwJM3u5xr4LzeyvZva6mZ0TZD4RkZq6tWnG898azPPXD+bMbhk8PXM9Fz30MXPW7aCisuEcT4nodSZm9iQwBih099419o8CHgDigcfd/d7jeK2X3f3SQ/a1BH7n7tcd7bm6zkREgjJzzXZueeETiveW0apJEpeenM3NZ3UhtVH0X58SNRctmtkwYA/w7JdlYmbxwGpgJFAAzAPGU1Us9xzyEte6e2H18w5XJr8HnnP3hUfLoTIRkSDtLi3nw1VFvL1sC28v20rLxkncMKwTV5/anibJCUHHO6KjlUlEU7v7NDPrcMjuPGCNu68DMLMJwFh3v4eqUcwxmZkB9wJvH6tIRESC1iwlkfP7teP8fu1YtqmE305exb1vr+Sxaeu4/oyOfP20DjSN4lI5nGg4ZpIF5NfYLqjed1hm1srMHgUGmNmPqnffCowALjWzG4/wvBvMbL6ZzS8qKqqj6CIioemdlcqz1+bx6s1D6Judym/fWcVXfvchyzeXBB3thER8ba7qkcmbNaa5LgPOdffrq7evBvLc/dZwZdA0l4hEq4Ubv+A7zy1kd+lBHvv6yQzpnB50pH+J9rW5CoCcGtvZgG4WICIxaWBuS165aQhtU1P45pPzeKOe3DslGspkHtDVzDqaWRIwDpgYcCYRkcC0a9GIl248jT7Zqdz6wif84KXF7CuL7mtTIlomZvYCMAvobmYFZnadux8EbgEmAyuAF919eSRziYhEmxaNk3jhW6dyy1ldeGVhAVc/MZeS/dG7eKTuZyIiEuUmLd3CbRM+oWvrZjx7XR7pTZMDyRHtx0xEROQozuvTlr9+fRDrtu/h8r/MYkvJ/qAj/ReViYhIPTC8e2uevXYwhbsOcOkjs1i2KbpOHVaZiIjUE3kd03jhW6dy4GAFYx6cwW0TPmHjjn1BxwJUJiIi9Uqf7FSm3DGcm4d3ZvLyrYz440cs+Lw46FgqExGR+ia1USI/HHUSH37/LNqmpnDzcwsDvwGXykREpJ7KTE3h0a+dTMn+ci586GM+XrM9sCwqExGReqxH2+Y8/61TSU6I4+on5rA4f2cgOVQmIiL13MDclrx+y1BaNU3m5xOXUxnATbdUJiIiDUCzlER+NPokFufv5MmP10f8/VUmIiINxIX9sxjRow3/99YKJszdGNH3VpmIiDQQcXHGQ1cN4MxuGfz0tWUs3PhF5N47Yu8kIiJhl5wQz5/GD6BtixRueW4hO/eVReR9VSYiIg1MaqNEHrpyIEV7DnDHi4sjckBeZSIi0gD1zW7BT87rwZSVhUyYl3/sJ4RIZSIi0kB9Y0gHUhslsmLLrrC/l8pERKSBMjPi4ywi76UyERGRkKlMREQkZCoTEREJmcpERERCpjIREZGQqUxERCRkKhMREQmZykREREJm7pG/iUrQzKwI+Pw4HpoOBHcfzGDE2meOtc8LsfeZY+3zQvg+c3t3zzjcD2KyTI6Xmc1390FB54ikWPvMsfZ5IfY+c6x9XgjmM2uaS0REQqYyERGRkKlMju6xoAMEINY+c6x9Xoi9zxxrnxcC+Mw6ZiIiIiHTyEREREKmMjkCMxtlZqvMbI2Z3Rl0nnAzsyfNrNDMlgWdJRLMLMfMpprZCjNbbma3BZ0pnMwsxczmmtni6s/7q6AzRYKZxZvZJ2b2ZtBZIsHMNpjZUjNbZGbzI/remub6b2YWD6wGRgIFwDxgvLt/GmiwMDKzYcAe4Fl37x10nnAzs7ZAW3dfaGbNgAXAhQ31v7GZGdDE3feYWSIwA7jN3WcHHC2szOx7wCCgubuPCTpPuJnZBmCQu0f8uhqNTA4vD1jj7uvcvQyYAIwNOFNYufs0oDjoHJHi7lvcfWH197uBFUBWsKnCx6vsqd5MrP5q0H9Jmlk28FXg8aCzxAKVyeFlAfk1tgtowL9oYp2ZdQAGAHMCjhJW1VM+i4BC4D13b9CfF7gf+CFQGXCOSHLgXTNbYGY3RPKNVSaHd7ibJjfov+JilZk1BV4Bbnf3XUHnCSd3r3D3/kA2kGdmDXY608zGAIXuviDoLBE21N0HAqOB71RPX0eEyuTwCoCcGtvZwOaAskiYVB87eAV4zt1fDTpPpLj7TuBDYFSwScJqKHBB9TGECcBXzOzvwUYKP3ffXP1vIfBPqqbsI0JlcnjzgK5m1tHMkoBxwMSAM0kdqj4g/QSwwt3/EHSecDOzDDNrUf19I2AEsDLQUGHk7j9y92x370DV/78fuPvXAo4VVmbWpPpkEsysCXAOELGzM1Umh+HuB4FbgMlUHZh90d2XB5sqvMzsBWAW0N3MCszsuqAzhdlQ4Gqq/mJdVP11XtChwqgtMNXMllD1x9J77h4Tp8vGkDbADDNbDMwF3nL3dyL15jo1WEREQqaRiYiIhExlIiIiIVOZiIhIyFQmIiISMpWJiIiETGUiIiIhU5mIhMjMWpjZzTW225nZy2F6rwvN7OdH+XkfM3s6HO8tcjS6zkQkRNULRb4ZiaX7zWwmcMHRlhg3s/eBa919Y7jziHxJIxOR0N0LdK6+iv4+M+vw5U3GzOybZvaamb1hZuvN7BYz+171DZtmm1la9eM6m9k71au9Tjezkw59EzPrBhz4skjM7DIzW1Z9w6tpNR76BlVLiIhEjMpEJHR3Amvdvb+7/+AwP+8NXEnVont3AfvcfQBVy9d8vfoxjwG3uvvJwPeBhw/zOkOBhTW2fw6c6+79gAtq7J8PnBHC5xE5YQlBBxCJAVOrb8C128xKqBo5ACwF+lYvgz8EeKlq/UkAkg/zOm2BohrbHwNPm9mLQM1VjwuBdnWYX+SYVCYi4XegxveVNbYrqfp/MA7YWX2vkaPZD6R+ueHuN5rZYKruJrjIzPq7+w4gpfqxIhGjaS6R0O0GmtX2ydU35VpvZpdB1fL4ZtbvMA9dAXT5csPMOrv7HHf/ObCdf9+DpxsRXHpcBFQmIiGrHg18XH0w/L5avsxVwHXVy4cvB8Ye5jHTgAH277mw+8xsafXB/mnA4ur9ZwFv1TKHSK3o1GCResTMHgDecPf3j/DzZOAj4PTq+/KIRIRGJiL1y91A46P8PBe4U0UikaaRiYiIhEwjExERCZnKREREQqYyERGRkKlMREQkZCoTEREJ2f8HIEUQcGnhp28AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "tlist = np.array(tlist)\n", "flist = np.array(flist)\n", "fstar = np.min(flist)\n", "plt.semilogy(tlist, flist - fstar)\n", "plt.xlabel(\"time (s)\")\n", "plt.ylabel(\"$f(v_k) - f^\\star$\");" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Sparsity: 10 %\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD4CAYAAAAEhuazAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAb80lEQVR4nO3df3Ac93nf8feDI0hBEi2QIu2QEBXCHg1cKrIDh6MfZSeNZSuU5EjEJONGmjpVOnZlzzjTKE7hkGNPRKXphC2TTDqtG5njKHVrl5ZNszBlqUM7kjOdOpZkyCBF0TIjRVRIHmkTlAiLliASBJ/+cXvQ4bh3uL3bvdu9/bxmMLzbW9w+xO0++73vPt/vmrsjIiL50tPpAEREpP2U/EVEckjJX0Qkh5T8RURySMlfRCSHFnU6gHpWrFjha9eu7XQYIiKZ8cwzz5xy95ULrZfq5L927VrGx8c7HYaISGaY2T82sp66fUREckjJX0Qkh5T8RURySMlfRCSHlPxFRHIo1dU+ImkwNlFk+95DHJ+aZnV/H6MbhxgZHuh0WCItUfIXqWNsosiW3QeYnpkFoDg1zZbdBwB0ApBMU7ePSB3b9x6aS/xl0zOzbN97qEMRicRDyV+kjuNT05GWi2SFkr9IHav7+yItF8kKJX+ROkY3DtHXW5i3rK+3wOjGoQ5FJBIPXfAVqaN8UffTu57l3OwFBlTtI11CyV9kASPDA+x8+ggAD3/8pg5HIxIPdfuIiOSQkr+ISA4p+YuI5JCSv4hIDin5i4jkkJK/iEgOKfmLiOSQkr+ISA4p+YuI5JCSv4hIDin5i4jkkJK/iEgOKfmLiOSQkr+ISA7FkvzN7FYzO2RmL5rZ5pDX/6WZPRv8/J2ZvTeO7YqISHNaTv5mVgA+B9wGrAPuNrN1VasdBv65u78H+PfAjla3KyIizYuj5X898KK7v+Tu54CvAJsqV3D3v3P308HTJ4GrYtiuiIg0KY7kPwAcrXh+LFhWy0eB/1PrRTO718zGzWx8cnIyhvBERKRaHMnfQpZ56Ipm76eU/P+g1pu5+w53X+/u61euXBlDeCIiUi2Oe/geA9ZUPL8KOF69kpm9B/gCcJu7vxLDdkVEpElxtPy/D1xjZoNmthi4C9hTuYKZXQ3sBn7L3f8+hm2KiEgLWm75u/t5M/sdYC9QAB5y94Nm9ong9QeBPwSuBP6bmQGcd/f1rW5bRESaE0e3D+7+GPBY1bIHKx5/DPhYHNsSEZHWxZL8RSSasYki2/ce4vjUNKv7+xjdOMTIcL0iOZF4KfmLtNnYRJEtuw8wPTMLQHFqmi27DwDoBCBto+QvbaPWbsn2vYfmEn/Z9Mws2/ceyuXfQzpDyV/aQq3dtxyfmo60XCdNSYJm9ZS2qNfazZvV/X0NLy+fNItT0zhvnTTHJooJRyndTslf2iJqa7ebjW4coq+3MG9ZX2+B0Y1DF62rk6YkRclf2iJKa7fbjQwP8Ce/fh2LC6XDb6C/jz/59etCu3J00pSkqM9f2mJ049C8Pn+o3dpNi8q+9t5CD2uWxXeiGhkeYOfTRwB4+OM31VxvdX8fxZBEn8eTpsRLLX9piyit3TSo7ms/N3uBw6+83va+9ihdRCJRqOUvbdNoazcNwvraLzh1yzGTqMop//6ndz3LudkLDKjaR2Ki5C8SoplyzKRKWbN00pTsULePSIioF6hVlSNZo+QvEiKsr73HqNnXrqocyRolf5EQ1ReoFxd6GLzysppdOCpllaxR8hepYWR4gOGr+7lhcDnDV/ezYumSmuuqKkeyRhd8RWLQ6aoczf8jUSn5i8SkU1U5mjRPmqFuH5GMU6WRNEPJXyTjVGkkzVC3j+RKN/aNa/4faYZa/pIb3To3viqNpBlK/pIb3do3nrVJ8yQd1O0jmRalGyfJvvFyHMWpaRYXehibKLY1+Wr+H4lKyV9SJUoyj1rimFTfeHUc52YvqNSyy3TjtSJ1+0hqRO2Tj9qNk1TfeLd2J0lJ2H5538P7GP6jb2X6epGSv6RG1CQatRsnqb5xlVp2t7D9EuD0GzOZLhhQ8pfUiJpEm5lMrXK+nu9uvjmWr+6a1K271TuJZ/kbnpK/pEbUJJqWEse0xCHJWOgkntVveLEkfzO71cwOmdmLZrY55PV3m9n3zOysmf27OLYp3SdqEk1LiWPY9M8qteweYftlpax+w2s5+ZtZAfgccBuwDrjbzNZVrfYq8G+BP211e9K9mknmSXTjNKMcx9JLFjF8db8SfxcZGR7gN34p/PMs9Fhmv+HFUep5PfCiu78EYGZfATYBPyyv4O4ngZNm9qEYtiddTPXqkkaPPnsidPnsBW9zJPGJo9tnADha8fxYsKwpZnavmY2b2fjk5GTLwUn2nDpzlokjUwxufpQN257IbDWFdIexiSKn35ip+frWPQfbGE184mj5W8iypk+H7r4D2AGwfv367J5WpSljE0UOv/I65QZVUnPTnzpzlg3bnuiqQTuSjIWqeaama58Y0iyOlv8xYE3F86uA4zG8r+TQ9r2HqP4mHXc53akzZzn8yutdN8GbJCOr1TwLiSP5fx+4xswGzWwxcBewJ4b3lRxqx4Cpo6enEz/BSPfov7S37uvLFng9rVru9nH382b2O8BeoAA85O4HzewTwesPmtnPAePA24ALZnYfsM7dX2t1+9Jd2jE3/bnZC6HLu7WFJ80bmyjyszfP13y9t2Dcf8e1bYwoPrFM7ObujwGPVS17sOLxjyl1B4nUNbpxiE99dd+8lnncA6YWF3pCTwBpq9eunEyst9DDmmXpii8Ptu89xEyNip6BjF8r0ghfSZWR4QEGr7xsroogiYFba5b10VNVppC2EbnVk4mdm73A4Vde13WJNqv3bbCT40rioCmdJXVWLF3CyZ+dZd2qtyVS679i6RKg1Pd/bvZC3RbcqTNn59Zr5zz9YZOJXfDS8iwnnKyp1Q3ZDdTyl1xasXTJgiODy2Wn5S6i8jz97Wh9a6bQdBjdOERv9dfEQNa/hSn5S+qcOnOWn715nqcOv9rRQV7tKDutRTOFpsPI8ACXXxLeQXLfw/syPQhRyV9SZWyiyEunXp8bJVicmmZ01/6OHGCdbH2HTSbWY6TqukReTNUZ3ZvlMSJK/hK7sYkiG7Y90dT0DA88cvCi4eEzs84Dj7R/CH2tVvYVfcnXdYfNFDp45WXq7++Ahb5tZXWMiJK/xCrqrRir1ZpDpd7cKkmp1cp+/dz5trT0KmcsHb66f+5CtbTX6Mahi6rDqmXxWoySv8SqE/ezTWoiuJHhARaFHPUzs57Jlp40b4Hcn8lrMUr+EqtW+8n7a3Sp1FpeWZGTxDw952sM8MliS0+iK3+Tna0zxWTaxog0SslfYtVqlcrWOy8eKt/bY6HLIfmKnHKfe7UstvQkulo3by8rmPEbvzSQyWsxSv4Sq1bvZzsyPMC7Vlw293ygv4/tH35vzYMr6YqcsCkVstrSk+gW2o9m3fn6M0VV+4jEcV/d8oXNgi08hD7pevgVS5ewZNFbPb6duk+wdEYj+5GqfUQC7byvblglRuwTwS0qUDBYesmizM/nItG8/90rF7zYC9m8BqTkL5nW7ERw5QqhTo8ilvQamyjy9WeKDd2WMIvXgDSxm2Re1IngynfySvpWkZJtC13sLQv7plk5HXdabxOqlr/kju7kJY1opCsn7JtmqwMd20XJX3JHd/KSRizUlVPrGlAnBjo2Q8lfcke1+9KIsLLlRmRlOm4lf8mdLNzJSzqvumy5UVmZjlvJX3JnxdIlid8qUrpDuWy50Ei9Z6DVgY7tomofaYty9UNxarqtt0OspVwhBKWBZCJxKe/Xn9717IK3Ce0kJf82y0IJWNzK1Q/li2Dl2yGCSiulc1o9Fs+8eZ4N254I/b2R4QF2Pn0EIJH7UMdB3T5tlJUSsLjVqn7Yuqf9N2gRgdKxOLpr/7xjMeyOcWMTRSaOTNWc1TPLx7CSfxtlpQQsbrWqHKamZzJ50Ej2PfDIQWaqMnr1HePKjbVapcFlWT2GlfzbqFgjCdZa3i3qVTlk8aCR7GvkjnGNjvCF9JVxNkLJXxJXr8ohiwdNPefOzyZyVzFpvyj7ZtrKOBuh5B+DVm5YngcjwwMsuzT8TlxZPGhqOXXmLGfPe2J3FZP4NHLHuEb3zeoyznI+eOrwq0wcmUrt56/k36LPjh3g9x7e19BF3IKFFwvXWt5N7r/j2q4fWHX09MUtxaz2B3e7rXdeS2/VDll9x7hGRvhWjxGpLOqAtyrb0ngCUPJvwdhEkS89eeSiKV9rHfB337Am9H1qLe8m5amXy7pxYJXmDMqOkeEBtn/4vfMG+lXfMa48wrdW0yzsZkNZKuow90Zmq17gTcxuBf4zUAC+4O7bql634PXbgTeA33b3Hyz0vuvXr/fx8fFIsXx27ABfevJIpN8REUmjj9x4NX88cl2k3zGzZ9x9/ULrtdzyN7MC8DngNmAdcLeZrata7TbgmuDnXuAvW91uGCV+EekmX3ryCJ8dO5DIe8cxwvd64EV3fwnAzL4CbAJ+WLHOJuB/eOlrxpNm1m9mq9z9RAzbn7PzqaMAfPzZb/DOn6avj01EpBEvXTHA59+zCSjltait/0bE0ec/AByteH4sWBZ1HQDM7F4zGzez8cnJyUiBzMbQhSUikiZJ5bU4Wv5h10Oqo21kndJC9x3ADij1+UcJpGDGrPvcGVNEJOuSqgaMo+V/DKgsV7kKON7EOi3LQ9WMiHROocfY8K7lbd1mUnktjuT/feAaMxs0s8XAXcCeqnX2AP/KSm4Efhp3fz/AH49cx0duvDrutxUR4bLFBf7sw+/ly//mprblmWaqfRoVV6nn7cBfUCr1fMjd/4OZfQLA3R8MSj3/K3ArpVLPf+3uC9ZwNlPq2S1+8/PfAxqbDvZdWx5l1uGGweWxTR87uPnR0H45Aw5v+1Do70SJuVFJvGezGp0CuN7f7vrBUqtxof/Pb37+e/zwxGusW/W2eetet3UvAAe2bpy37I2z51m/duHPP+z3q9WqmisnorDPpJHPaWyiyKe+uo8LFX+cvt5C6HiP6mnA660r8zVa6hnLfP7u/hjwWNWyByseO/DJOLYl7bG6vy90wrlumo4hqpHhgYYST9b/dt/5UXihRa3ljRoZHuC/PP4CR09Pzw2Iq5XMq2+IsrjQo8QfM43w7QLlltRTh1+NbW6hrNyKLo2S+tuNTRT52Zvn524iktSUAUnegHzF0iVzt0UsWP2b+ZRvobj0kkUMX92vxB8zJf8UKt9AopFkPjZRnNfFENdkYuWh7QNBa1Utr8ZV/u2MeKayKHeDlD/rJCeNq3cD8ij7pqSbbuOYMtU3kCgf5BDeSgqbM6Q8l0iribr6q3d5WzoBLKzRLqJGtXPOmNGNQ6H97e9/98qa+2YSyieac7MX5mbH1L4XH7X8UybqQZ7kV/RaJyK19tovyc+5Wvmby+JCKT2Uv7l850eTbTsBVe97aZ4dM6uU/FMm6kFe7yt6q7I0Q2G3S/JzDlPub79hcPnczJXtPAFp30uekn/KRD3Iwy4ixnVhtp0Hu9SXhgvwtfbBK/p6G7oOUHm9YNaZV/JZTfte8pT8UybqQT4yPDA3d0ZcFxfL2t3alNpqdcW0sw88bN/s7TFeP3d+wa7BsJuhe7A8jPa95Cn5p0wzlSI9Qdnc4W0fuujmEq1IQ2tT4tNqqWjYvnn5JYuYmZ3fhA/rnql1M/Ra3Tja95Knap8UirtSpJU4gIZGtUp8wqpcgJYqbWqVikK06q3qfXNw86Oh61V3z0Ttxqnc94pT0yo1ToCSv9SVlhNRXtSqcrmkt6elC6D1LqCGfb6VJ6AN256oedJvdDRzM6Oey/teeeoI7YfxUrdPxo1NFJl1mHU6OuhGg3/iUStJn35jJnT9Ri+ARml5RynxbbR7ptbN0NWN0zlK/hk2NlFk9Gv7554Xp6YZ/dr+tidejQdoTCMnyKjVLFf09XLmzfPMeml6j+E/+lbo+0a5gBqlzLLRa1TVI8Yrt6X9pDPU7ZNhW/ccZKaqXm7mgrN1z8G2fkWO2qWQR42O3K7VPdLf18vZ8xfm/Z17e4zX3pz/jeD0GzOM7tp/0fvWGrUb1vJupn++kc+5vM7orv1zF4mLU9Oh8Ury1PLPsKnp8K6AWsuToprshTXamq7VjbL1zmtDK23CauVnZv2i941SKppkmeUDjxy8qDpoZtZ54JGDLb+3RKOWv7Qs61MYt0OjJ8iFqlwaqbSptb2R4QF2Pl2ap7/evPtRviVEVevaRa3lkhy1/DNs2aW9kZYnRTXZC4vSmh4ZHuC7m2/mhsHldacyrndybeXEm8SspJI+Sv4Zdv8d19JbmH9z596Ccf8d17Y1DiWLhSVxghzdOERvz8U39+4tWMsn3vIJKO6Bg/194Q2TWsslOer2ybDqKZcHOjgIS+MB6ktiwFz5d3//q/sod6Mvu7SX+++4NrWfxdY7r2X0a/vnFSr09hhb7wxvsDQ63kCiU/IXaZMkTpCN9uOnRZQGS9R7W0g0Sv4ZpoNDsqjRE5ZKiJOlPv8M05znEkXWRmGrhDhZSv4ZpoNDGk3oWRyFrWmdk6Xkn2E6OPItSkLP4rdElRAnS8k/w3RwdK9GWvRREnoWvyWqhDhZuuCbYZpvvzs1eiE/SkLP6ihslRAnR8k/43RwdJ9Gq1yiJPQkp2yQbFK3j0jKNNqij9Ltpy4UqaaWv0jKNNqij9rtl5ZviRq1mw4tJX8zWw48DKwFXgb+hbufDlnvIeDXgJPu/gutbFOk20XpoklLQm+UBiamR6vdPpuBx939GuDx4HmY/w7c2uK2RHKhm7toslhy2q1a7fbZBPxK8PiLwN8Cf1C9krv/XzNb2+K2RHIjay36RmWx5LRbtdryf4e7nwAI/n17qwGZ2b1mNm5m45OTk62+nYikiAYmpseCyd/M/sbMngv52ZREQO6+w93Xu/v6lStXJrEJEekQDUxMjwW7fdz9g7VeM7OfmNkqdz9hZquAk7FGJyJdRQMT06PVPv89wD3AtuDfb7QckYh0tW69npE1rfb5bwNuMbMXgFuC55jZajN7rLySme0EvgcMmdkxM/toi9sVEZEWtNTyd/dXgA+ELD8O3F7x/O5WtiMiIvHS9A4iIjmk5C8ikkNK/iIiOaTkLyKSQ0r+IiI5pOQvIpJDSv4iIjmk5C8ikkNK/iIiOaTkLyKSQ0r+IiI5pOQvIpJDSv4iIjmk5C8ikkNK/iIiOaTkLyKSQ0r+IiI5pOQvIpJDSv4iIjmk5C8ikkNK/iIiOaTkLyKSQ0r+IiI5pOQvIpJDSv4iIjmk5C8ikkNK/iIiOaTkLyKSQy0lfzNbbmbfNrMXgn+Xhayzxsy+Y2bPm9lBM/vdVrYpIiKta7Xlvxl43N2vAR4Pnlc7D/y+u/8T4Ebgk2a2rsXtiohIC1pN/puALwaPvwiMVK/g7ifc/QfB4zPA88BAi9sVEZEWtJr83+HuJ6CU5IG311vZzNYCw8BTdda518zGzWx8cnKyxfBERCTMooVWMLO/AX4u5KXPRNmQmV0OfB24z91fq7Weu+8AdgCsX7/eo2xDREQas2Dyd/cP1nrNzH5iZqvc/YSZrQJO1livl1Li/7K77246WhERiUWr3T57gHuCx/cA36hewcwM+CvgeXf/8xa3JyIiMWg1+W8DbjGzF4BbgueY2WozeyxYZwPwW8DNZrYv+Lm9xe2KiEgLFuz2qcfdXwE+ELL8OHB78Pj/AdbKdkREJF4a4SsikkNK/iIiOaTkLyKSQ0r+IiI5pOQvIpJDSv4iIjmk5C8ikkNK/iIiOaTkLyKSQ0r+IiI5pOQvIpJDSv4iIjmk5C8ikkNK/iIiOaTkLyKSQ0r+IiI5pOQvIpJDSv4iIjmk5C+pNTZRZOLIFE8dfpUN255gbKLY6ZBEuoaSv6TS2ESRLbsPcG72AgDFqWm27D6gE4BITJT8JZW27z3E9MzsvGXTM7Ns33uoQxGJdBclf0ml41PTkZaLSDRK/pJKq/v7Ii0XkWiU/CWVRjcO0ddbmLesr7fA6MahDkUk0l0WdToAkTAjwwNAqe//+NQ0q/v7GN04NLdcRFqj5C+pNTI8oGQvkhB1+4iI5JCSv4hIDin5i4jkkJK/iEgOKfmLiOSQuXunY6jJzCaBf2zy11cAp2IMJ06KrTmKrTmKrTlZje3n3X3lQm+Q6uTfCjMbd/f1nY4jjGJrjmJrjmJrTrfHpm4fEZEcUvIXEcmhbk7+OzodQB2KrTmKrTmKrTldHVvX9vmLiEht3dzyFxGRGpT8RURyqOuSv5ndamaHzOxFM9vcge0/ZGYnzey5imXLzezbZvZC8O+yite2BLEeMrONCce2xsy+Y2bPm9lBM/vdtMRnZpeY2dNmtj+I7YG0xFaxvYKZTZjZN9MUm5m9bGYHzGyfmY2nLLZ+M9tlZj8K9rubUhTbUPA3K/+8Zmb3pSE+M/u94Dh4zsx2BsdHvHG5e9f8AAXgH4B3AouB/cC6Nsfwy8D7gOcqlv0nYHPweDPwH4PH64IYlwCDQeyFBGNbBbwveLwU+Psgho7HBxhwefC4F3gKuDENsVXE+CngfwHfTNnn+jKwompZWmL7IvCx4PFioD8tsVXFWQB+DPx8p+MDBoDDQF/w/KvAb8cdV+J/1Hb+ADcBeyuebwG2dCCOtcxP/oeAVcHjVcChsPiAvcBNbYzzG8AtaYsPuBT4AXBDWmIDrgIeB27mreSflthe5uLk3/HYgLcFSczSFltIrL8KfDcN8VFK/keB5ZTuufLNIL5Y4+q2bp/yH63sWLCs097h7icAgn/fHizvWLxmthYYptTCTkV8QbfKPuAk8G13T01swF8AnwYuVCxLS2wOfMvMnjGze1MU2zuBSeCvg+6yL5jZZSmJrdpdwM7gcUfjc/ci8KfAEeAE8FN3/1bccXVb8reQZWmuZe1IvGZ2OfB14D53f63eqiHLEovP3Wfd/RcptbKvN7NfqLN622Izs18DTrr7M43+SsiyJD/XDe7+PuA24JNm9st11m1nbIsodYH+pbsPA69T6q6opVPHw2LgTuBrC60asiz2+IK+/E2UunBWA5eZ2Ufijqvbkv8xYE3F86uA4x2KpdJPzGwVQPDvyWB52+M1s15Kif/L7r47bfEBuPsU8LfArSmJbQNwp5m9DHwFuNnMvpSS2HD348G/J4H/DVyfktiOAceCb3AAuyidDNIQW6XbgB+4+0+C552O74PAYXefdPcZYDfwT+OOq9uS//eBa8xsMDib3wXs6XBMUIrhnuDxPZT62svL7zKzJWY2CFwDPJ1UEGZmwF8Bz7v7n6cpPjNbaWb9weM+SgfAj9IQm7tvcfer3H0tpX3qCXf/SBpiM7PLzGxp+TGlvuHn0hCbu/8YOGpmQ8GiDwA/TENsVe7mrS6fchydjO8IcKOZXRocsx8Ano89rnZcTGnnD3A7pSqWfwA+04Ht76TUTzdD6Yz8UeBKShcLXwj+XV6x/meCWA8BtyUc2z+j9HXwWWBf8HN7GuID3gNMBLE9B/xhsLzjsVXF+Su8dcG347FR6lffH/wcLO/zaYgt2NYvAuPB5zoGLEtLbMH2LgVeAa6oWNbx+IAHKDV+ngP+J6VKnljj0vQOIiI51G3dPiIi0gAlfxGRHFLyFxHJISV/EZEcUvIXEckhJX8RkRxS8hcRyaH/D89/6bhYoTfJAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.stem(x_pro)\n", "s = np.sum(np.abs(x_pro) > 1e-3)\n", "print(f'Sparsity: {(s/p*100).astype(int)} %')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We compare agains [Celer](https://github.com/mathurinm/celer)'s algorithm. Note that this algorithm is very efficient (more efficient than the bilevel method) for large $\\lambda$. " ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "def lasso_func(x): \n", " return 1/(2*la) * norm(A@x-y)**2 + norm(x,1)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100% (10 of 10) |########################| Elapsed Time: 0:00:15 Time: 0:00:15\n" ] } ], "source": [ "import celer\n", "\n", "warnings.filterwarnings(\"ignore\")\n", "m = 10\n", "niter_list = np.arange(0, 20, 20 // m)\n", "tlist_celer = np.zeros(m)\n", "flist_celer = np.zeros(m)\n", "\n", "for i in progressbar.progressbar(range(m)):\n", " niter = niter_list[i]\n", " t0 = time.time()\n", " x_celer = celer.Lasso(\n", " alpha = la/n, verbose=False,fit_intercept=False, max_iter=niter, tol=1e-20).fit(\n", " A,y).coef_.T \n", " tlist_celer[i] = time.time()-t0\n", " flist_celer[i] = lasso_func(x_celer)" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEGCAYAAACgt3iRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2OUlEQVR4nO3dd1yVdf/H8deHrcgQUFBQcCsiLly5UnNk5vhlZcuGlQ1v62573827u3FXd2V3S0tbWs5K01yVaZYL3HsjiAoOUFRE4Pv745ARAQ7OOdc58Hk+HjziXOc613kfKt5c6/sVYwxKKaVUeXhYHUAppZT70zJRSilVblomSimlyk3LRCmlVLlpmSillCo3L6sDWCEsLMzExMRYHUMppdxKUlLSEWNMjZKeq5RlEhMTQ2JiotUxlFLKrYhIcmnP6WEupZRS5aZlopRSqty0TJRSSpVbpTxnopRyH+fOnSM1NZWcnByro1Qafn5+REVF4e3tfdGv0TJRSrm01NRUAgICiImJQUSsjlPhGWM4evQoqamp1KtX76Jfp4e5lFIuLScnh9DQUC0SJxERQkNDL3lPUMtEKeXytEic63J+3loml+DU2TxemruFH7YcJvtsntVxlFLKZWiZXIKd6dl8tjyZuz9PpO9bS8k4edbqSEopJxARHn300fOP33jjDZ5//vnL3t6OHTvo378/DRs2pFmzZtxwww0cPny43DljYmJo0aIFLVu2pE+fPhw6dKjc27xYWiaXoFWdYDY814ePhydw9NRZ7p+UxLn8AqtjKaUczNfXl6+//pojR46Ue1s5OTlcc8013H///ezatYutW7dy//33k5GRYYeksHjxYtavX09CQgIvv/zyn54zxlBQ4JjfWVoml8jP25OrYsN5/trmJCYfJ3HfcasjKaUczMvLi3vvvZe33nrrL88lJyfTq1cv4uPj6dWrF/v37wfgjjvuYPTo0VxxxRXUr1+fGTNmAPDll1/SqVMnrr322vPb6NGjB3FxcXTo0IHNmzefX37llVeSlJTE6NGj+de//gXAggUL6Nat2wVLoVu3buzatYt9+/bRrFkzHnjgAdq0aUNKSgqPP/44cXFxtGjRgqlTp5b75wN6afBlS4ipDkBGth7qUspZXvhuM1vSTth1m7G1A3nu2uYXXO/BBx8kPj6eJ5544k/LR40axfDhw7n99tuZOHEio0eP5ttvvwXg4MGDLFu2jG3btjFw4ECGDh3Kpk2baNu2bYnvMWzYMKZNm8YLL7zAwYMHSUtLo23btjRr1ox27drRtWtXRo8ezffff4+HR9n7AnPmzKFFixYAbN++nU8++YT333+fmTNnsm7dOtavX8+RI0do164d3bp1o1atWhfx0ypdhdgzEZHBIvKRiMwSkT7OeM9Qf18AjmqZKFUpBAYGMnz4cN55550/LV++fDk333wzALfddhvLli07/9zgwYPx8PAgNjb2os6J3HDDDUyfPh2AadOmcf311wNQtWpVPvroI3r37s2oUaNo0KBBqdvo0aMHrVq14sSJE4wZMwaA6OhoOnbsCMCyZcu46aab8PT0JDw8nO7du7N69epL+EmUzGX3TERkIjAASDfGxBVZ3g8YC3gCHxtjXjXGfAt8KyLVgTeAhY7OF1TFG08P4YiWiVJOczF7EI708MMP06ZNG+68885S1yl6Wa2vr+/5740xADRv3pwlS5aU+NrIyEhCQ0PZsGEDU6dOZdy4ceef27hxI6GhoaSlpQGQn59/fg9n4MCB5w+DLV68mLCwsPOvy8zMxN/f/y857M2V90w+BfoVXSAinsB7wNVALHCTiMQWWeXpwucdzsNDCPX34Wh2rjPeTinlAkJCQrjhhhuYMGHC+WVXXHEFU6ZMAWDy5Ml06dKlzG3cfPPN/Pbbb8ydO/f8svnz57Nx40bAdqjrtddeIysr6/xhquTkZP773/+ydu1a5s2bx8qVK/H09GTdunWsW7fufJFcjG7dujF16lTy8/PJyMhg6dKltG/f/qJfXxqXLRNjzFLgWLHF7YFdxpg9xphcYAowSGz+A8wzxqwpaXsicq+IJIpIor2umgit5ssRLROlKpVHH330T1d1vfPOO3zyySfEx8fzxRdfMHbs2DJfX6VKFebMmcP//vc/GjVqRGxsLJ9++ik1a9YEYOjQoUyZMoUbbrgBsO1JjBgxgjfeeIPatWszYcIE7r777sseq2zIkCHEx8fTsmVLevbsyWuvvUZERMRlbasocdQujz2ISAww5/fDXCIyFOhnjLm78PFtQAdgB3A7sBpYZ4z5sKztJiQkGHtMjnXbhJVkn83jmwc6l3tbSqmSbd26lWbNmlkdo9Ip6ecuIknGmISS1nfZcyalKOkef2OMeQd4p4TnHCrU34d9R085+22VUsrluOxhrlKkAnWKPI4C0izKQlg1Xz1nopRSuF+ZrAYaiUg9EfEBhgGzrQoTWs2X07n5nM7VcbqUUpWby5aJiHwFLAeaiEiqiIwwxuQBo4AFwFZgmjFmc1nbcaTQaj4A/LrrKLd+vFJLRSlVabnsORNjzE2lLP8e+N7JcUpUo5rtGvKJy/ayfM9RdhzOplWdYGtDKaWUBVx2z8Qd/L5nsnLvUQAOZZ2xMo5SSllGy6QcQgv3TAoKr65Oy9Q5qpWqqA4dOsSwYcNo0KABsbGx9O/fnx07dpS6frVq1ZyYznpaJuUQ6m/bM6lBJu96v8PJI6kWJ1JKOYIxhiFDhnDllVeye/dutmzZwssvv2yXOUh+376jhoZ3Fi2TcvDz9iTA14tgyaaX51r6b/8n5OtJeKUqmsWLF+Pt7c199913flmrVq3o2rUrr7/+Ou3atSM+Pp7nnnuuxNeXtE5JQ8O7M5c9Ae8uQqv5UODRhImeo3kw83VY/G+46nmrYylVMc17Cg5ttO82I1rA1a+WuUppw8YvXLiQnTt3smrVKowxDBw4kKVLl9KtW7cLrlO3bt0/DQ3v7rRMyumWDtEEVfFmxZ5gvj25nsHL3oI6HaDJ1VZHU0o52MKFC1m4cCGtW7cGIDs7m507d/6lTEpap27dun8aGt7daZmU0z3d6gOQfOwU/zhzC4OiDyPfjISRS6F6jLXhlKpoLrAH4SjNmzc/P1NiUcYYxowZw8iRI0t9bWnr7Nu3709Dw7s7PWdiJxFBVThd4M2R/h/ZFkwbDuf06i6lKoKePXty9uxZPvroo/PLVq9eTWBgIBMnTiQ7OxuAAwcOkJ6e/qfX9u3b94LrVAS6Z2IntYP8ADhAODUGfwhTboIFY2DAX+eMVkq5FxHhm2++4eGHH+bVV1/Fz8+PmJgY3n77bYKDg+nUqRNguxx40qRJ54eTB+jTpw9bt279yzqenp6WfBZHcekh6B3FXkPQF7U5LYtr3lnGB7e04eoWtWDRc/Dr2zBkPLS80a7vpVRlokPQW+NSh6DXw1x2EhlcBYBd6bZdWXo+A9GdYc7DkL7VumBKKeUEWiZ2ElzVhw71QpiamEJ+gQFPLxg6EXyqwdTbIOeE1RGVUsphtEzs6PYrYkg9foafthWeXAuIsBXKsT3wzUhw8ztclbJKZTwcb6XL+XlrmdhRn9hwagX58fnyfX8srNcV+r0C27+HJdZc1qiUO/Pz8+Po0aNaKE5ijOHo0aP4+fld0uv0ai478vL0YFCrSCYs28Ops3n4+xb+eNvfC4c2wJL/QHhziB1kbVCl3EhUVBSpqalkZGRYHaXS8PPzIyoq6pJeo2ViZ10bhfHhkt2s2nuMHk0LLw8UgWvehIzt8M39ENIAIuKsDaqUm/D29qZevXpWx1AXoIe57KxtdHV8vTz4ZeeRPz/h5Qs3TgK/QNs9KKeOWhNQKaUcQMvEzvy8PWkXE8Kvu4789cmACLhxMpw8DDPu0BGGlVIVhpaJA3RpFMb2wyc5fKKE4VSi2sK1Y2HvUlj4tPPDKaWUA2iZOMBVzWznSr5bn1byCq1ugo4PwsoPYO0kJyZTSinH0DJxgIY1A2hZJ5jpiamlX87Y+19Q/0qY83dIWe3UfEopZW9aJg5yfdsoth8+yaYDpdz57ukFQz+BwNow9VY4cdC5AZVSyo60TBzk2pa18fHyYGri/tJXqhoCw76CsydthaJD1iul3JSWiYMEVfFmQHwtvl2bRvbZMq7aCo+F/xsHBxJh7iPOC6iUUnZUIcpEROqLyAQR+etUaBa6tWM02Wfz+HbtgbJXbHYtdHsC1k2G7fOdE04ppezI8jIRkYkiki4im4ot7yci20Vkl4g8VdY2jDF7jDEjHJv00rWuE0zz2oFMWpF84XGFuj0ONZrCvMch97RzAiqllJ1YXibAp0C/ogtExBN4D7gaiAVuEpFYEWkhInOKfdX86yZdg4hwW8doth06yfSk1LJX9vKBa/4Lmfth2ZvOCaiUUnZieZkYY5YCx4otbg/sKtzjyAWmAIOMMRuNMQOKfbn0ZMqDW0fSqX4oT8zYwGe/7St75ZguED8Mlr0NR3Y6I55SStmF5WVSikggpcjj1MJlJRKRUBH5EGgtImNKWedeEUkUkURnjj7q5+3Jp3e1o2/zcJ6bvfnC50/6vAjeVWHuo6BDbiul3ISrlomUsKzU36zGmKPGmPuMMQ2MMa+Uss54Y0yCMSahRo0adgt6MXy9PBk7rDUd64fw2PT1LN1RRplVqwm9noG9S2DTTOeFVEqpcnDVMkkF6hR5HAWUMjaJe/Dz9mT88AQahQdw36QkNqZmlb5ywl1QqxUs+AfklLGeUkq5CFctk9VAIxGpJyI+wDBgtsWZyi3Qz5vP7mxH9ao+3PN5IuklDQQJ4OEJA96C7HRYXOKOllJKuRTLy0REvgKWA01EJFVERhhj8oBRwAJgKzDNGLPZypz2UjPQj49vT+BEzjnu+SKJnHP5Ja8Y2QbajYBV4+DgeueGVEqpSySVcV7lhIQEk5iYaGmGhZsPMXJSEgNb1ubtG1shUsJpojPH4d12EBwNIxaBh+Xdr5SqxEQkyRiTUNJz+tvJIn2aR/BYnybMWpfG+z/vLnmlKtWhz79tQ62s/dy5AZVS6hJomVjogSsbMKhVbV5fsJ0Fmw+VvFL8jRDdGRY9B6dKmL1RKaVcgJaJhUSE/1wXT8s6wfx96jq2pJUwXL2I7c743Gz44Tnnh1RKqYugZWIxP29PPrqtLUFVvLnn88SSRxiu2Qw6PWiblXH/CueHVEqpC9AycQE1A/149+bWHMg8wzs/ljKMSvcnITDKNjNj/jnnBlRKqQvQMnERbaNDuDGhDhOX7WXrwRIOd/n4w9X/gfQtsHKc8wMqpVQZtExcyJNXNyW4qg+jv1rLmdwS7j9peg006gs/vwJZFxjjSymlnEjLxIWE+Pvw9o2t2JWRzWPT13Muv+DPK4jY9k4K8mxDrSillIvQMnExXRqF8Y+rmzF340Hun5T01xPyIfWg62Ow5VtI+sySjEopVZyWiQu6p1t9XhzUnJ+2pTPw3WWkHCs282KXh6HhVbaT8TsWWpJRKaWK0jJxUbd1imHy3R05cvIsD0xew9m8IudQPL3h+s8gIg6m3w4H1lgXVCml0DJxaZ0ahPL69S3ZeCCLl+du/fOTvtXg5ungHwZf3gDH9loTUiml0DJxeX2bR3B3l3p8tjyZz5fv+/OTAeFwy0zbfSeTh8Lp4rMfK6WUc2iZuIEx/ZtxVbOaPD97M4u3FZvyvkZjuGkKZKbAV8Pg3BlrQiqlKjUtEzfg6SGMHdaa2NqBPPjlGpbvPvrnFaI7wXUfQcoqmHk3FJQyR4pSSjmIlomb8Pf1YsLt7YgI9OPmj1fwxoLtFBQUmYsmdhD0ewW2zYH5Y6ASzlOjlLKOlokbCQ/047u/deH6tlG8u3gXIyclkZtX5MbGjvdDp1G22RmXv2tdUKVUpaNl4mb8fb34z3XxPHdtLIu2HOaf32zkT7Nl9n4Rmg+BhU/DxhnWBVVKVSpeVgdQl05EuLNzPTJPn2PsjzsJrurNP/o3s0396+EBgz+Ek4fh2/shIAJiulgdWSlVwemeiRt7+KpGDO8UzUe/7OXfRe9D8faDYZOhej2YcjOkby19I0opZQdaJm5MRHhhYHPuuCKGCcv28t36tD+erBoCt84ALz+YNBROHLQuqFKqwtMycXMiwj+vaUbb6Oo8OXMD36xN/eMcSnBduGU65GTC5Oshp4R5UpRSyg60TCoAb08P3r+lDbG1Avn71PU8On09OecK7zWp1RJu+BwytsK02yAv19qwSqkKScukgggP9GPqyE48fFUjvl5zgB5v/Mzj09eTeToXGvaCa9+BPT/Dd6P1HhSllN3p1VwViKeH8PBVjWlZJ5hpq1OYtS6NnenZfHpnO4Jb3wInDsDilyAoCno+bXVcpVQFUmH2TETEX0SSRGSA1Vms1qNJTT64tS3/u7k1Gw9k0e6lH7h/UhJbG42ENsNh6euQ+InVMZVSFYjlZSIiE0UkXUQ2FVveT0S2i8guEXnqIjb1JDDNMSndU9/mEcwe1ZnbO8WwbOcRrn5nGbceGkZ6RHfM3Edg+3yrIyqlKggxFh8/F5FuQDbwuTEmrnCZJ7AD6A2kAquBmwBP4JVim7gLiAfCAD/giDFmTlnvmZCQYBITE+35MVxe1ulzTFqZzJcr93M88zgLg/9DZH4KcscciGxrdTyllBsQkSRjTEKJz1ldJgAiEgPMKVImnYDnjTF9Cx+PATDGFC+S31//EuAPxAJngCHGmIJi69wL3AtQt27dtsnJyY75MC4uv8Aw9ocdfPlTEvOqvUCYTx5y9yIIqW91NKWUiyurTCw/zFWKSCClyOPUwmUlMsb80xjzMPAl8FHxIilcZ7wxJsEYk1CjRg1753Ubnh7CI32a8PDgzgw7/Rgnz5wl97P/g1NHL/xipZQqhauWiZSw7IK7UMaYTy90iEvZ3NoxmqduHcgo8yQFmamkjx9MwdlTVsdSSrmpiyoTEYlwdJBiUoE6RR5HAWmlrKsuU+/YcF5/5B4+rvkPwjI3suq/17FyV/qFX6iUUsVcsExEJBL48iKvqLKX1UAjEaknIj7AMGC2E9+/0ggP9OPBB/7OxvgxdMxdzrZPH2DEJ6tISj6GK5xPU0q5hwuWiTHmAPA9sM0RAUTkK2A50EREUkVkhDEmDxgFLAC2AtOMMZsd8f7KNr5Xy+ueJK/jKG73WkTz5M+47oPl3P7Jag6fyLE6nlLKDbjE1VzOVhkvDb4oBQUwcwRs/pqfmr/MAxvq4+3pwX3dG3Bn5xiq+uiACUpVZu54NZeygocHDPkQorvQc+tz/PR/HnSoF8LrC7bT/fWfmbdRh7FXSpXsYs6ZjC38ZxXHx1GW8/KFYZMgtAG159/Dx/38mXl/J2oH+fHAl2t4b/Eu1qVkknX6nNVJlVIu5IKHuURkkzEmrnD3pkLcKq2HuS5CZgp8fBV4eMLdP5BTJZwHJ6/hx21/XO0VXNWbmFB/YkKr0iIqmGHt6uDvq4fClKqoynUHvIi8AXQG6gLPAOuBzcYYtz0zq2VykQ5ugE/6Q/VouPN7jG8gu9Kz2Xf0NPuOnGLv0VMkHz3FviOnOZB5hrBqvozu1ZBh7eri46VHUJWqaMo9nIqI1Ad+Bj4DWgDNgVxgkzHmRvtFdQ4tk0uw+yfbLI2N+trmlZeS7ieFpOTj/Gf+NlbtPUadkCo80rsxA1tG4ulR8vpKKfdjl7G5RKSxMWZHkcfVgDhjzAr7xHQeLZNL9Os7sOgZ24yNsYNKXc0Yw5IdGby+YDub007QNCKAmzvUpW10dZpGBGqxKOXmXH6gR2fTMrlE+Xkw/ko4lQGjVoFfUJmrFxQY5m48yFs/7GBPhm2Ilmq+XrSuG0z7mBCGd4ohqKq3E4IrpexJy6QYLZPLkJoEH/eC9vdA/9cv6iXGGFKPnyEp+TiJycdI3Hec7YdPUjuoCm/e0JIO9UMdHFopZU92LRMRudYY851dkllEy+Qyff8ErBoPd/8AUSX+93RB61MyGfXVGlKOneGqZuE83rcJTSIC7BxUKeUI9r5p8aVy5lHuqufTEFALvnsI8i/vPpOWdYJZ8HA3Hu/bhJV7jtJv7FKenLGB7LN5dg6rlHKmyykTPYtaWfkFQv/X4PAmWPH+ZW+mqo8XD/ZoyNInenB3l3pMT0qh/9hfmJ6Ywtm8fDsGVko5y+WUSeU7yaL+0HQANOkPi1+B4+WbrbK6vw//vCaWqSM7UcXbk8dnbKDzq4sZ+8NO0k+67W1MSlVKl3POZIMxJt5BeZxCz5mUU2YKvNcBYjrDzdNKvffkUhhjWLbrCBOW7eXn7Rl4ewr94moxvFM0CdHVETu8h1KqfMo6Z6JjX6hLF1zHdv5kwRjY/A3E/V+5NykidG1Ug66NarAnI5tJK/YzPSmF79an0TQigBsS6tAvLoLawTpEnFKu6HL2TBYZY3o7KI9T6J6JHeTnwcc94eQheHAVVAm2+1uczs1j1ro0vly5n40HsgBoERnEsPZ1GNiyNgF+eq+KUs6k95kUo2ViJ2lr4aOe0PYOGPCWQ99qV3o2P207zNdrDrDt0Emq+ngyslsD7ruyPr5eng59b6WUjZZJMVomdjR/jO3KrhGLoE57h7+dMYb1qVl8tHQPczceJKyaDwPiazOkdSTxUUF6bkUpB9IyKUbLxI7OnrSdjPcLgpFLwdN5h55+23WEL1Yk8+O2dHLzCujWuAb/HhRH3dCqTsugVGVi7zvg/YEcY4zb3hCgZWJn276HKTdBr+eg6yNOf/usM+eYnpjCm4t2kHMun55Nw3moVyNaRJU9hphS6tKUdz4TD2AYcAvQDjgL+AIZwPfAeGPMTrsmdjAtEweYcgvs+gEeWAEh9SyJcDDrDJNWJDN55X4yT5+jbXR1+reoxdA2UTqwpFJ2UN4yWQL8AMzCNn9JQeHyEKAHcDPwjTFmkl1TO5CWiQOcSIN320OddnDr13a59+Syo+Sc44vlyczdcJAtB08A4OPlQbOIAK5sUpOhbaOoE6KHwpS6VOUtE29jzF8GYhIRH2NMblnruCotEwdZOQ7mPQHXTYAWQ61OA8CWtBP8sPUwJ3POsXZ/Jkn7j2MM1K/hT9u61UmIqU7b6BAa1PDXk/dKXYBDTsCLyCvGmDGF33c2xvxajoxOpWXiIAX5tnnjs1Js955UDbE60V8cyDzD7HVpJO47RtL+42Setv0NVD/Mn2via9G/RS2aRgRosShVAkeVyZVAY+AU0MwY8/TlBnQ2LRMHOrjBNpFW61th4DtWpymTMYY9R06xfPdR5m06yPLdRyko3Gu5pkUtromvRZNwLRalfmf3MhGRCUAW0ApYYYz5R7kSOpmWiYMt+CcsfxfunA/RnaxOc9GOZJ9l/qZDfL/xICv22IqlZoAvkdWrMKxdHYa0jsLH63LGRlWqYnDUnkkVoA22K7yaGWNGXn7E8hGRrtiuNvMCYo0xV5S1vpaJg+West174uMPI38BLx+rE12yjJNnmb/5EOv2Z7Ll4Am2HjyBn7cH9cOq0Ta6Oj2a1iC2VhDpJ3NoHB6An7feha8qPruUiYj8BvzTGLPYzuEmAgOAdGNMXJHl/YCxgCfwsTHm1YvY1mAg3Bgzrqz1tEycYPt8+OpG24CQ3R63Ok25GGNYsiODZTuPsP3wSdYkH+dU7h+3WVXx9qRzwzB6NatJjyY1iQjyszCtUo5jrzKJA14AgoGnjTHL7RSuG5ANfP57mYiIJ7AD6A2kAquBm7AVyyvFNnGXMSa98HXTgLuNMSfKek8tEyeZNtxWKg8sh9AGVqexm9y8An7ddYT9x04TWs2HlXuO8dO2dA5kngGgcXg12tcLIa52EM1rB9E4opqOH6YqBHvfAd8G+Ffhw6eNMevKFw9EJAaYU6RMOgHPG2P6Fj4eA2CMKV4kRbdRF3jGGHNPKc/fC9wLULdu3bbJyeWb2EldhBMH4b32ULs1DJ9l6b0njmaMYcfhbBZvT+fXXUdYuz/z/FTEXh5Cw5rViIsM4pr4WnRvVAMPj4r7s1AVl73nM9kFvAjcCSRe5jYuJBJIKfI4FehwgdeMAD4p7UljzHhgPNj2TMobUF2EwFrQ61n4/jHYMA1a3mh1IocREZpEBNAkIoD7ujegoMCw/9hpNqedYHNaFpsL73eZkZRKWDUfrmgQRpdGYXSoF0LdkKp6xZhyexddBCLyE9AIyAG2FH7d4ZhYJc4zX2YBGGOec1AWVR4JI2D9FNtEWo16u+S9J47g4SHEhPkTU3j/CtgOjy3ccogfthxm2a6jzF6fBkBYNV/aRgfTNro6vZqF06BGNSujK3VZLmWv4jFgqzHmjKPCFJEK1CnyOApIc8L7Knvz8IBrx8K4brDoGRj0ntWJLOPj5cGA+NoMiK99/rBYYvIxkvYdJzH5OAs2H+bl77cRHxXEFQ3C6Fg/hI71Q/VKMeUWLmY4FTEXWOli1rnA62P48zkTL2wn4HsBB7CdgL/ZGLP5ct+jKD0Bb4FFz8KvY+GOuRDTxeo0LulQVg7frD3AD1sPsz4lk7wCQ1g1H4Z3iuHWjtGE+LvfJdaqYinv2FyLga+BWcaY/UWW+wBdgNuBxcaYTy8z3FfAlUAYcBh4zhgzQUT6A29ju4JrojHmpcvZfkm0TCyQexre7wCevnD/r+Dla3Uil3Y6N48Ve47yxfJkFm/PwM/bg2Ht6jKye31qBVWxOp6qpMpbJuOBddhOcNcCMgE/bL/kFwLv2eOKLmfSMrHIzh9g8nVw5Ri48imr07iNnYdPMm7pHr5dewARuK5NFPd1b0BMmL/V0VQlU94y2WSMiRORNUB7oAZwxhiTafekTqJlYqEZd8HW7+D+3yCskdVp3Erq8dOMX7qHKatTyMsv4JYO0TzapzHBVfXwl3KOssrkYgYaWiAiy4FwYDhQG9sVXUpdur6vgFcVmPN3qIRTRpdHVPWq/GtQHMue7MHwTjFMXplMjzd+5qtV+8kv0J+lstYFy8QY8yi2ca/ygXrAM8BGEdksIlMdnE9VNAHh0Pt52PcLrPvS6jRuqWaAH88PbM7c0V1pFB7AmK83Mvi9X/l6TSonctxmWiFVwVzKcCqNjTE7ijyuBsQZY1Y4Kpyj6GEuixUUwCf94MhOGJUI/qFWJ3Jbxhhmr0/jtfnbOZB5hlB/H569NpZr42vrXfbK7hwyarA70zJxAYe3wLiu0OJ6GPKh1WncXkGBYc3+47w4dyvrUzLx8/aga6MavDQkjpoBOvCksg8tk2K0TFzEj/+CX/4Lw2dD/e5Wp6kQ8gsMczaksXZ/JlNW78fP25MhrSO5tmVtWtcJRkRYsecoYdV8aFgzwOq4ys1omRSjZeIizp2B9zuCeNqu7vLWv6Dtaefhk7yxcDuLt2eQm1dAZHAVmkYE8OO2dAJ8vfhsRHva1K1udUzlRsp7NZdSjuFdBQa8Bcd22/ZQlF01Cg9g3G0JJD59Ff+9viWNwquxau8x7uwcQ2g1H24ct5ynZm5gzf7jFOjVYKqcdM9EWW/mPbD5G9ud8TWaWJ2mUsg4eZb//bSTKatTyM0rICLQj6tia9I2ujqt6lQnJlRHMlZ/pYe5itEycTHZGfBuAtSMtY3d5aE7zM6SdeYcP207zPxNh1i64whnztlmkAyq4k3v2HD6NY8gPiqImoF6CFJpmfyFlokLWvM5zP4bXP0adBhpdZpKKS+/gF0Z2axPyWTV3uMs2Hzo/ARfNQN8aREZRFxkEC0ig2haK4DI4Cq691LJaJkUo2XigoyBydfDvmVw3zIIa2h1okov51w+Gw9ksTE1i00HsthwIIvdGdnnBy6Iql6Fnk1rElsrkKa1AmkcXo2qPo6YK0+5Ci2TYrRMXNSJg7aru8IawZ3zwVN/MbmaU2fz2HrwBFsOnmDxtnRW7Dl2/tCYCMSE+tMupjpt6lYnsnoVmoQH6CGyCkTLpBgtExe2YTp8fbdtut+uj1qdRl1AQYEh5fhpth06ybaDJ9l4IIvV+46RdeaPYV3CA32JjwqmV9Oa9I4NJ7SaTj/grrRMitEycWHGwPTbYdv3cO9iiGhhdSJ1iX4vmLTMHLYcPMGmwoJJPX4GD4H29ULo1zyCvnEROjeLm9EyKUbLxMWdOmo73FWtJtyzGLx0iHV3Z4xhy8ETLNh0iHmbDrEzPRuAVnWC6RcXQb/mETo/ixvQMilGy8QNbJ8HXw2zHerq9azVaZSd7UrPZsHmQ8zfdIiNB7IAaBoRQL+4CK5pUYtG4TrUiyvSMilGy8RNfPsgrP8S7loIddpZnUY5SOrx08zfdIgFmw+RmHwcY+CqZuE82qcxzWoFWh1PFaFlUoyWiZvIyYIPOtvmix/5C/hUtTqRcrD0EzlMXZ3C+F/2kH02jyGtI3lhYHMC/LytjqbQsbmUu/ILgkHvwdFd8OMLVqdRTlAz0I+/9WrEL0/0YGS3Bsxal8bAd39lZlIqZ3LzrY6nyqBlolxb/e7QfiSs/BD2LLE6jXKS4Ko+PHV1Uybf3QEBHp2+nvYv/8CzszaxOS3L6niqBHqYS7m+3NPwYRfIz7UNBukXZHUi5UTGGFbuPcaUVfv5ftMhcvMKiI8K4sZ2dbiuTRR+3p5WR6w09JxJMVombihlNUzsA61uth36UpVS1ulzfLM2lSmrU9h26CSRwVV4qFcjromvhb+vjpjgaHrORLm/Ou2g88OwdpLtsmFVKQVV9eaOzvWY91BXvry7A4FVvHli5gY6vvwjE5ftJS+/wOqIlZbumSj3kXcWPuoJ2enwwArwD7U6kbKYMYY1+4/zzo+7WLIjg2a1Avn34DjaRusMko5QofZMRKS+iEwQkRlFlvmLyGci8pGI3GJlPuVAXr4wZBycOQ7f67hdCkSEttEhfHpnOz68tQ2Zp3O57oPfGPP1RrJOn7vwBpTdOLVMRGSiiKSLyKZiy/uJyHYR2SUiT5W1DWPMHmPMiGKL/w+YYYy5Bxho59jKlUTEQY8xtpkZN8648PqqUhAR+sXV4odHunN3l3pMXb2fXm/+zKx1B6iMR1+s4Ow9k0+BfkUXiIgn8B5wNRAL3CQisSLSQkTmFPuqWcp2o4CUwu/1YvSK7oqHIKodzH3UNmy9UoX8fb14ekAss0d1sZ2cn7KOwe/9ygc/72bn4ZNaLA7k1DIxxiwFjhVb3B7YVbjHkQtMAQYZYzYaYwYU+0ovZdOp2AoFSvlMInKviCSKSGJGRoY9Po6yiqcXDP7Qdg5l9t9Af0GoYuIig/j6gc68ODiOvALDf+Zvo/dbS+nz1lK+WL6Ps3n6N6e9ucI5k0j+2KsAWzFElrayiISKyIdAaxEZU7j4a+A6EfkA+K6k1xljxhtjEowxCTVq1LBTdGWZsIbQ+wXYtcg25a9SxXh6CLd1jGbu6K4sH9OTfw+Oo4qPJ8/M2kyft5Yyf9Mh3VOxI1e4MLukSaRL/TdsjDkK3Fds2SngTjvnUq6u3T2wbQ4s+IftTvnqMVYnUi6qVlAVbu0Yza0do1myI4N/z9nCfZOSaF8vhL9f1ZiO9UN0PvtycoU9k1SgTpHHUUCaRVmUO/HwKLyBUWwjDBfoPQbqwro3rsG8h7ry4uA49h05xU0frWDUV2v16q9ycoUyWQ00EpF6IuIDDANmW5xJuYvgunD1q5C8zDZ+l1IXwcvTg9s6RrP0iR481qcxCzYd4qq3ljAzKZWCAj30dTmcfWnwV8ByoImIpIrICGNMHjAKWABsBaYZYzY7M5dyc61ugcZX20YWzthhdRrlRvy8PRnVsxHfPNCZyOAqPDp9Pdd9+BtbD56wOprb0TvgVcVw8rBtqt/qMTBike2KL6UuQUGB4eu1B3h13lZOnMnj0T6NGd4phio+OpDk7yrUHfBKlSggHAa8CWlrYNmbVqdRbsjDQxjaNooFD3ejW+MwXpm3jSte/ZEvlu/TQ18XQctEVRzNh0DcUFjyHzi43uo0yk2FVvPlo+EJTL+vE81qBfLMrM0M+N8yFmzWS4nLomWiKpb+r0PVMPjmPttNjUpdBhGhXUwIk+/uwNhhrTidm8fIL5Lo/84y5m08qHsqJdAyURVL1RAY9C6kb4HFL1mdRrk5EWFQq0h+eKQ7b97QkrPn8rl/8hoGvrdMT9IXoyfgVcX03UOQ9BnctQDqdrA6jaog8gsMs9cf4KW5Wzl2KpdaQVXo1CCUv/VsSHSov9XxHE5nWixGy6QSOHsSPugM4gG3zoTQBlYnUhXIsVO5fPrrXvYcOcWiLYcxwMtDWjC0bdQFX+vOtEyK0TKpJPavgMnX2+aO7/Y4XDEavHysTqUqmMMncvj71HX8tvsoA+Jr8eyAWGoG+lkdyyH00mBVOdXtCA+ugsZ94acXYXx3SFlldSpVwYQH+vH5Xe15tHdjFm45TK//LuGLFcmV7iS9lomq2AJrwQ2fw7CvICcLJvSxzYOSk2V1MlWBeHl68LdejVjwcDfi6wTxzLeb+L8PfmNdSqbV0ZxGD3OpyuPsSfjpJdsYXgERtsuIm11rdSpVwRhjmLUujX/P3cKR7Fz6NY/g+YHNiQhy/0Nfes6kGC2TSi41yXa11+GN0OQaW6kElTqFjlKXJftsHhOX7eW9xbvw8fRgZPf63NYphqAq3lZHu2xaJsVomSjyz8GK92HxK+DhBb2egXZ3g4eOw6Tsa9+RU7w4Zws/bkvHx8uDKxvXYHSvRsRFBlkd7ZJpmRSjZaLOO7YX5j4Cu3+CyLZw7TsQEWd1KlUBbTqQxcw1qcxel0bWmXOM6FKPvnERxEcG4eXpHqevtUyK0TJRf2IMbJwB85+CnEy44m/Q/UnwrmJ1MlUBZZ7O5ZlZm5mzIQ1jIKyaD3d1qcetHaMJ9HPtQ2BaJsVomagSnT4GC5+BdZNsQ9kPeAsa9LQ6laqgjp3K5bfdR5iemMqSHRlU8fZkYMva3NoxmhZRrnkITMukGC0TVaa9S+G7h+HYbogfBn1fAv8wq1OpCmzTgSwmrUhm1ro0zpzLp2VUEM8NbE6butWtjvYnWibFaJmoCzqXA7/8F5a9Bb4BtkJpeROIWJ1MVWAncs7xzZoDjF+6h0MnchjVoyGjejbE20XOqWiZFKNloi5a+lbbZcQpK6FeNxjwto7zpRzuRM45np+1ma/XHqBlVBBj+jejY/1Qq2PpcCpKXbaazeDO+bbzJ2nr4P1OsPQNyMu1OpmqwAL9vHnzxla8e3Nr0rJyGDZ+BbdNWMnujGyro5VK90yUulgnDsL8J2HLLKgZC9eOhTrtrU6lKricc/lMWpHM2B92kpOXz4gu9flbz4b4+3o5PYse5ipGy0SVy/Z5MPcxOHEA2o2AXs+Cn2tefaMqjoyTZ3l13jZmrkklPNCXB65sSM+mNakTUtVpGbRMitEyUeVWfJyvq1+zjfOlJ+iVgyUlH+OF77awIdU2WGm9MH9u7xTNLR2jHX6iXsukGC0TZTcHkmC2jvOlnMsYw54jp1i6I4N5Gw+xat8xwgN96dKwBrd0rOuwS4q1TIrRMlF29adxvjxth710nC/lJMYYftqWzoykVH7ddYQTOXm0rxdCx/qhJERXp3XdYALsdGe9lkkxWibKIY7vgzmPwO4fdZwvZYlTZ/P49Ld9zNt0kC1pJygw4CGQEB3C/T0aUDekKv4+Xpc9HH6FKhMRqQ/8EwgyxgwtXNYMeAgIA340xnxQ1ja0TJTDGAObZsK8J+HM8T/G+fJx3klSpcA2BP7a/cdZve84MxJTSMvKAaBv83DG3VZiH1yQy5SJiEwEBgDpxpi4Isv7AWMBT+BjY8yrF7GtGb+XSZFlHsBHxpgRZb1Wy0Q53OljsOgZWKvjfCnrnc3LZ8n2DM6cy6dWUBXa1wu5rO240k2LnwL9ii4QEU/gPeBqIBa4SURiRaSFiMwp9lWztA2LyEBgGfCj4+IrdZGqhsCg9+D2Obb5Ur4YAl/fC6eOWJ1MVUK+Xp70aR7BoFaRl10kF+LUMjHGLAWOFVvcHthljNljjMkFpgCDjDEbjTEDin2ll7Ht2caYK4BbSnpeRO4VkUQRSczIyLDXR1KqbPW6wn2/QrcnYNPX8G4CrJ1sOxymVAXiCsOpRAIpRR6nFi4rkYiEisiHQGsRGVO47EoReUdExgHfl/Q6Y8x4Y0yCMSahRo0adoyv1AV4+0HPf8J9yyCsCcx6AD67Fo7utjqZUnbj/Pvx/6qku7xK/bPNGHMUuK/Ysp+Bn+2aSil7q9kU7pwHaz6FRc/bxvnq/jhc8RB4+VidTqlycYU9k1SgTpHHUUCaRVmUciwPD0i4C0atgiZXw0//hnHdYP9Kq5MpVS6uUCargUYiUk9EfIBhwGyLMynlWAERcMNncNNU29AsE/va7lHJybI6mVKXxallIiJfAcuBJiKSKiIjjDF5wChgAbAVmGaM2ezMXEpZpkk/eHAldLwfkj6Bd9vbRiXWE/TKzbjdTYv2oPeZKJd0YA18NxoObYQm/QvH+YqyOpVS57nSfSZKqdJEtoF7fobeL8LuxfBeB1g5DgryrU6m1AVpmSjlSjy9oPNoeHAF1O0I856ACb1teytKuTAtE6VcUfUYuGUGXDcBjifDuO6w6DnIPW11MqVKpGWilKsSgRZDYdRqaHUz/Po2vN8Rkn+zOplSf6FlopSrqxpim28+bihkJsPSN6xOpNRfuMId8EqpshzbA98+APuX22ZzHPCW1YmU+gstE6VcVUEBJE6ARc+ChzcMGQfxN+o888olaZko5YoyU2DWg7B3CTToBQP/p3PLK5emZaKUKzHGNqHW/DGAsZ0raXO77o0ol6dlopQrmf8UrPwQYrraJteqHm11IqUuipaJUq5kzxKo1w1um2UbYVgpN6H/tSrlSkTAL0iLRLkd/S9WKZciOmKwcktaJkq5Ej3RrtyUlolSLkX3TJR70jJRypUIgJaJcj9aJkq5FN0zUe5Jy0QpVyKC7pkod6RlopRL0T0T5Z60TJRyJbpnotyUmEr4V5CIZADJF7l6GHDEgXFcjX7eiq+yfebK9nnBcZ852hhTo6QnKmWZXAoRSTTGJFidw1n081Z8le0zV7bPC9Z8Zj3MpZRSqty0TJRSSpWblsmFjbc6gJPp5634KttnrmyfFyz4zHrORCmlVLnpnolSSqly0zJRSilVblompRCRfiKyXUR2ichTVudxNBGZKCLpIrLJ6izOICJ1RGSxiGwVkc0i8pDVmRxJRPxEZJWIrC/8vC9YnclZRMRTRNaKyByrsziaiOwTkY0isk5EEp363nrO5K9ExBPYAfQGUoHVwE3GmC2WBnMgEekGZAOfG2PirM7jaCJSC6hljFkjIgFAEjC4ov47FhEB/I0x2SLiDSwDHjLGrLA4msOJyCNAAhBojBlgdR5HEpF9QIIxxuk3aeqeScnaA7uMMXuMMbnAFGCQxZkcyhizFDhmdQ5nMcYcNMasKfz+JLAViLQ2leMYm+zCh96FXxX+L0kRiQKuAT62OktFp2VSskggpcjjVCrwL5rKTkRigNbASoujOFTh4Z51QDqwyBhToT9vobeBJ4ACi3M4iwEWikiSiNzrzDfWMilZSXOnVvi/4iojEakGzAQeNsacsDqPIxlj8o0xrYAooL2IVOjDmSIyAEg3xiRZncWJOhtj2gBXAw8WHr52Ci2TkqUCdYo8jgLSLMqiHKTw3MFMYLIx5mur8ziLMSYT+BnoZ20Sh+sMDCw8jzAF6Ckik6yN5FjGmLTCf6YD32A7ZO8UWiYlWw00EpF6IuIDDANmW5xJ2VHhCekJwFZjzJtW53E0EakhIsGF31cBrgK2WRrKwYwxY4wxUcaYGGz/D/9kjLnV4lgOIyL+hReTICL+QB/AaVdnapmUwBiTB4wCFmA7MTvNGLPZ2lSOJSJfAcuBJiKSKiIjrM7kYJ2B27D9tbqu8Ku/1aEcqBawWEQ2YPtjaZExpsJfKlvJhAPLRGQ9sAqYa4yZ76w310uDlVJKlZvumSillCo3LROllFLlpmWilFKq3LRMlFJKlZuWiVJKqXLTMlGqnEQkWEQeKPK4tojMcNB7DRaRZ8t4voWIfOqI91aqLHppsFLlVDi21xxnjLYsIr8BA8saFVZEfgDuMsbsd3QepX6neyZKld+rQIPCGx9fF5GY3+eFEZE7RORbEflORPaKyCgReaRwfo0VIhJSuF4DEZlfOEDfLyLStPibiEhj4OzvRSIi14vIpsI5SpYWWfU7bHd8K+U0WiZKld9TwG5jTCtjzOMlPB8H3IxtnKSXgNPGmNbYRhwYXrjOeOBvxpi2wGPA+yVspzOwpsjjZ4G+xpiWwMAiyxOBruX4PEpdMi+rAyhVCSwunDPlpIhkYdtzANgIxBeOXHwFMN02ZBgAviVspxaQUeTxr8CnIjINKDpQZTpQ2475lbogLROlHO9ske8LijwuwPb/oAeQWTg8fFnOAEG/PzDG3CciHbBN/rRORFoZY44CfoXrKuU0ephLqfI7CQRc7osL51HZKyLXg21EYxFpWcKqW4GGvz8QkQbGmJXGmGeBI/wxbUJjnDharFKgZaJUuRXuDfxaeDL89cvczC3AiMIRXzdT8jTRS4HW8sexsNdFZGPhyf6lwPrC5T2AuZeZQ6nLopcGK+VGRGQs8J0x5odSnvcFlgBdCqdSUMopdM9EKffyMlC1jOfrAk9pkShn0z0TpZRS5aZ7JkoppcpNy0QppVS5aZkopZQqNy0TpZRS5aZlopRSqtz+H4e1mfJxKLXRAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "fstar = min(np.min(flist_celer), np.min(flist))\n", "tmax = np.inf\n", "plt.semilogy(tlist[tlist