{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第十五讲：子空间投影\n",
    "\n",
    "从$\\mathbb{R}^2$空间讲起，有向量$a, b$，做$b$在$a$上的投影$p$，如图：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW8AAAD7CAYAAAClvBX1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VPXZPvD7mSRgWEIgvgRkS3KB5VXQCEiRNWwCBREK\nxfoCElAsosUKohKrBGsVW7hARbvJpgUFXFBpEOgPBm1f0ADyFlFACQkuLJWGNQhJ5vn9MZmQPbN/\n55zcn+uaS2fJOXdOp4/f3Dk5I6oKIiKyFofpAERE5DsObyIiC+LwJiKyIA5vIiIL4vAmIrIgDm8i\nIguKDteORITnJBIR+UFVpeJjYV15q2rIbnPnzg3p9kN9Y35mZ35r3kKdvzq2qU1yc3NNRwgI85tj\n5ewA85tmKr9thjcRUV1im+Gdnp5uOkJAmN8cK2cHmN80U/mlpk4lqDsS0XDti4jILkQEavoXlqHk\ndDpNRwgI85tj5ewA85tmKr9thjcRUV3C2oSIKILZvjYhIqpLbDO82ZuZZeX8Vs4OML9plu+8RcQh\nIntE5L1gbZOIiKoWtM5bRB4C0BVAnKqOrOJ5dt5ERD4KaectIq0B/ATAK8HYHhER1SxYtckiALMB\nGFtaszczy8r5rZwdYH7TLNt5i8hwACdUdS8AKbkRURg89Ow+zFzzQo1XnyN7CrjzFpFnAEwAUAQg\nFkBjAG+r6l0VXqeTJk1CUlISACA+Ph6pqalIS0sDcOW/XrzP+7zv3f033wRe+sdRYPQkzG07F2nJ\naRGVj/f9u+90OrFixQoAQFJSEubNm1dl5x3UP9IRkX4AZvEXlkShtXIlcN99QGHqy3ANmYGmsfHI\n+1UeGtZraDoaBZnt/0jH818uq2J+c6yW/e233YP74kXAEXseyFUUFBZg3vZ5pqP5xWrHvyJT+YM6\nvFV1e1WrbiIKjs2bgQkT3IMbAArlLFzqwsWii1jyyRJ8eepLswEpbHhtEyKL+N//BQYPBgoKrjzW\naNwDOH/dSwAAhzjQu01vONOdEOF5A3Zh+9qEyM727gWGDCk/uAGgSeLp0n93qQu7ju3C+wffD3M6\nMsE2w5u9mVlWzh/p2Q8eBNLSgPPnyz8eHQ00TDgDHLnyWEFhAe7dcC9+KPohrBkDEenHvza26LyJ\nKLjy8oBevYCzZys/16ABENOo8hPnLp3Dbz/8bRjSkUnsvIki2G23AVlZgMtV+bkGDYDk+V2w/z+f\nVnouNjoWn9//OZLik0IfkkKKnTeRBS1aBIweDVx1FRAVVf654mLgMi5U+XWXiy/jFxt+EYaEZIpt\nhjd7M7OsnD+Ss7dvD7z5JpCd7R7WwJUhnpICFBReKNd5A0Bc/Tg4xIHLxZfDG9ZPkXz8vcHOm4iq\nNWuW+58HD7pX4g4H0KULcLHoIq6KvgoxjpjS1/519F9xcvZJbJu0zVBaCgd23kQR7tIld20yZw7w\nzDPux44cARITAec3WSjSIvRp2weH8w/j5r/cjMInChHtiDYbmoKmus6bw5sowg0Z4v7LSpcLqO1v\nb2Se4LXRr2HCDRPCE45Czva/sGRvZpaV80dy9kuX3IN7zpzqB3fF/JnOzJDnCqZIPv7eYOdNRJWM\nLLlS0G+9PG17xLUjcDj/cOgCUcRgbUIUoarqumuz67td7L1txva1CZHd+LrqBoBu13QDALzx2Rsh\nSESRxDbDm72ZWVbKn5eXB4fDgSlTpgCIzOzedN0eVeW3Uu8dicffF+y8iaiUP6tuD/bedQM7b6pz\n8vLykJycjPT0dCxbtsx0nEr86brLYu9tL+y8iSwikFU3wN67rrDN8GZvZpZV8x88eBC9e/dGQkIC\nGjVqhD59+mDLli3G8vjSdXtUd+yt0ntb9b3jwc6bKMxycnJwyy234Pz585g2bRrGjRuHPXv2YNiw\nYVi3bp2RTIGuuj3Ye9sfO2+qczydt4hg9uzZmD9/fulze/bsQY8ePdC4cWPk5eWhUaNGYcsVaNdd\nFntv+2DnTVRBkyZN8MQTT5R7rEuXLhg/fjxOnz6Nd955J6x5grXqBth71wW2Gd7szcyyYv4uXbqg\nYcOGlbKnpaVBVfHpp5U/oSZU/Om6PWo69lbova343imLnTdRmCUmJlb5eIsWLQAAZ86cCVuWYK66\nPdh725tthndaWprpCAFh/vA7ceIEgMrZjx8/DsBdq4RDIKtuoPpjP7ffXABAkasogHShZ8X3Tlmm\n8ttmeBP5as+ePbhwofJnQG7btg0igptuuiksOUKx6gbYe9udbYY3ezOzrJj/zJkzmDdvXrnsu3bt\nwurVqxEfH4/Ro0eHPEOgq26g9mMf6b23Fd87ZZnKz3OIqM7q27cvli5dik2bNmH48OH47rvvsHbt\nWqgq/vSnP4XlNMFQrbo9Rlw7AhsObQjNxskonudNdU5eXh5SUlKQnp6ORx55BI899hg+/PBDXLp0\nCV26dMGTTz6JQYMGhTxHMM/rrg7P97Y+foYlUYTx5bMpAyHzBK+OehUTb5wYup1EMIfDgbS0NGzd\nutV0FL/Y/o902JuZZeX8JrIHo+v28CZ/5vbMwHYSQlZ+7wA8z5so4mVnZwftmieh7rrLGnHtCOTk\n54R+RxRWrE2IvJSTk4NBgwbhrrvuwty5cyEVlsybN2/G4MGDKz1eUTi67rLqeu/N2oSojktJScGO\nHTuQlZWFSZMm4fLly+Wev//++5GdnV3rdsK56gaunO/9+r7Xw7NDL3z88ccYO3YsWrZsifr166Nt\n27aYNm0ajh07FrJ9Hjt2DBMnTkRiYiIaNGiAbt264fXXfT8mDocDAwYMCNr2/GWb4c3ezCwr5/cl\ne2JiIpxOJ86ePYuhQ4ciPz+/9LmpU6fi5ZdfrvHrg9l1e3ibP1J672XLlqF3797YtGkTBgwYgDFj\nxuDmm2/G0qVL0a1bN3zzzTdB32d+fj569uyJ/fv3Y8qUKZg0aRKOHDmC8ePHY+HChQFtb9CgQQFv\nzx+2qU2cTqel/8yW+c3xJ3txcTFmzZqFzZs3IysrC0lJSfj+++/Rvn17HD58GAkJCVV+XSjOMPEm\n/22v34YNhzZA55qtLr/88kt06tQJSUlJ2L59O1q0aFGaf9u2bRg8eDBuv/12vPXWWwCA7du3+7ww\nmDt3brn7DocDIoJx48aVWxnn5eWhS5cuuHDhAg4cOICkpCSvtl9xe578/m6vNtXVJlDVsNzcuyKy\nl8WLF+s111yj2dnZqqo6ceJEXbBgQZWv/eEHVUB1zpxwJnTL/jZbkQktLC4M/87L+NWvfqUOh0Oz\nsrKqfH706NEaExOj58+fV1XVzMxMdTgcXt+ioqIqbVNENCYmRnNzcys959n+U0895fX3EOzt1aZk\ndlaaqbZZeROZ8u677+Kee+7B0qVL0bx5c0yYMAGHDh2Cw1G+lQzXed3V8fV878LCQkRFRVX6PgJx\nyy234JNPPsHDDz+MBg0aVHp+y5Yt2LFjB3bt2hW0a8s4HA6kpKTgq6++qvTc9u3b0b9/f4waNQpv\nv/02zpw5g0WLFlX6pfNDDz2EuLg4n7cXDLZfeW/bti2k2w815jfH1+wXLlzQd955R48dO1b62Cef\nfKItW7bU559/Xm+66Sb94IMPyn1NKFfd3uZHJjTl+ZRaX+dyuXTFihWamJioe/bsCTBdeR06dPBq\n9fzhhx8GbZ8ioj179qzyuQMHDqiI6IABA1RVNTc3V0WkUqa8vLxqt1f2+FfcXjCgmpV3wOcNiUhr\nAK8CSATgAvAXVX0h0O0SRaqCggKsXLkSU6ZMQcuWLdG/f38MGDAAGzZswIQJE9CqVSu89NJLGDJk\nSOnXhPsMk6p4c52TL774Avfddx/Onz+PDRs2BP3Kip7L7J49exYNGzYEUHNnH4zOG7hy+d+KKl7+\nt127dnC5XLXuw9vthVRVE92XG4AWAFJL/r0RgIMAOlbxuqD9l4goEhQVFemuXbv0d7/7nQ4bNkwb\nN26snTt31pYtW6qI6BdffKGqZrvusmrqvQsKCvTxxx/Xq6++Wl988UUtKiryersul0vz8/P1wIED\nun37dl2zZo0uW7ZMCwsr7+eBBx6osfOuKJidd9nVs8fcuXNVRPQ3v/mN199vsLdXG4Sr8xaR9QBe\nVNX/V+FxDfa+iCJJYWEhsrOz8fe//x2vvPIKHnvsMUyfPt14111WVb33pk2bMH36dHTr1g2LFi3C\nNddcA1VFfn4+Tpw4UePt+PHjOHnyJOrVq4cWLVogMTERiYmJaNu2LZ599lnUr1+/3P4PHjyIG264\nAcnJyXj//ffRoUOHcs8XFhbi448/Ru/evYP2PZc9O2T16tWlffaRI0fQtWvX0rNDkpOTjWyvNmG5\nMJWIJAFwAuikqucrPBfS4W3lU9UA5jcplNnD8deUvuSXeYKUpik4POMw/v3vf2PIkCHIyclB9+7d\nERMTUzqUT548idjYWCQmJpYbytXdYmNjvc67atUq3H333XC5XBg6dChiY2PRokULHD16FB999BGa\nN2+Ozz//3M+jUZnD4cCNN96IM2fOID4+Hrfeeivy8/Oxbt06nDlzBr///e8xc+ZMv7fXsWNHNG7c\n2O/t1aa64R20v5UVkUYA3gTwYMXB7ZGenl567mN8fDxSU1NL33SeXsvf+3v37g3o603fZ3573n/2\nWff9wYOdcDrN5/H03k6nE0ePHsWpU6dQWFiI//znP+jduzcyMzORmJiIAwcOoF69ejVu7+LFi6X/\nf/Ylz/jx43H58mWsXbsW+/btw3fffYerrroKV199NX72s5/hjjvuCOr3LyIQESxYsADr16/HihUr\ncPr0aSQlJeEPf/iDX/sru70NGzagoKAgoO2Vve90OrFixQoAqPFc8aCsvEUkGsAGABtV9flqXsPa\nhOqUcF/DxBtVXeckJycHy5Ytw/Lly9G2bVtMnToV48aNC8uHUVhRuK+VEuprmywD8Hl1g5uoLoqE\nM0wqquo6JykpKXj66aeRl5eHjIwMvPvuu2jTpg3uvfde7N6921RUqkXAw1tEegEYD2CAiHwqIntE\nZGjg0Xzj+bHDqpjfnFBkD8U1TKrjT/6qrnMSHR2N2267De+++y7279+P5ORkPPbYY16dOhcIK793\nAAtfz1tV/6mqUaqaqqo3qWoXVf0gGOGIrCoSV90e3lzf+5prrsGcOXOwZcuWoP6FpR14OnTT+Ofx\nREHm6bozMiJzeNf163tbDa/nTRQmnlX300+bzVGdSLy+N/nONsObvZlZVs4fzOyerjsjI3x/kONv\n/ki5vreV3zuAhTtvIroi0lfdHvxcS+tj500UJJHedZfF3ts62HkThZhVVt3Ald579b7VhpOQv2wz\nvNmbmWXl/MHIbqLr9ggkf6YzM2g5/GXl9w7AzpvI0qy06vYYce0IHDl9xHQM8hM7b6IAWanrLou9\ntzWw8yYKESuuugH23lZnm+HN3swsK+cPJLvJrtsj0GNvuve28nsHYOdNZElWXXV7sPe2LnbeRH6y\natddFnvvyMfOmyjIrL7qBth7W5lthjd7M7OsnN+f7JHQdXsE49ib7L2t/N4B2HkTWYodVt0e7L2t\niZ03kY/s0HWXxd47srHzJgoSO626AfbeVmWblbfT6URaWlrIth9qzG+OL9kjcdUdjGMv8wTJ8cnI\neTDAy8ROmAB89hnQqBHQuDHQpIn71rSp+5+NGrlvDRuW/ruzsBBpAwcGtl+DQv3er27lzZ+RiHxg\nt1W3x4hrR2DDoQ2Bb2j3buDAgaqfcziAmBggOtr97+fPux9fuTLw/dZBtll5E4VaJK66gyVovfes\nWcCiRUBt/19v2BC49lpg+XLgxhv9318dwM6bKEB2XXUDQey9W7Rwr66rc9VVQHw88Mc/ulfpHNx+\ns83w5rmiZlk5vzfZI+m87oqCeex9Pt/744+Bn/zEfVBEgEceAS5frvw6ESA2Fpg6FcjLc3fjJQfS\nyu8dgOd5E0U0O6+6Pbw637visO7RA9i4EejQAVi1CigqAjp2LP81DRsC3bu7V9ovvADExYXum6hL\nVDUsN/euiKznhx9UAdWMDNNJQiv722xFJrSwuPDKgzt3qg4b5j4AZW8dOqiuWqVaVFR5QzNnqjoc\nqrGxqgkJqmvXqrpc4ftGbKZkdlaaqfyFJVEthgxxVyYuV+RVJjVxuVxYs2YNCgoKUL9+/dJbbGws\n6tevj3r16lW6Ja9MxsrDN+Cu1/5VfmMdOgCZmcAddwBRUTXv2OkEhg4FZswA5s51r7zJb7Y/VdDK\n5xkDzG9STdkjuev2qC6/iCA3NxeXq+qgAURFRcEBQC5dghQV4VL9+uinvdH+wx2+DeuK0tKAc+dq\n/sWlF/mtwlR+2wxvolCwctctImjdujVycqr+w5vi4mIUA0B0NGJU0TY2Fm9NfgsJ85oHvnMvBzf5\nj7UJUTXscF73xuXLsTs3F8WOqs9NiImORr369TFixAj86Ec/gkTqjxd1WHW1CYc3UTWs2HV/u2kT\nnB98gK/i42t8nYggKioKPXr0QN++fRHDlXLEsv0f6fBcUbOsnL+q7FbougH3sH587FjMmzcP8+bN\nwys7d+Kr+Hg0O3sWP01MxK/nzKk0mGNiYpCUlITp06dj4MCBxge3ld87gLn87LyJqhCpXXdVK+tv\nGzVC17NnkdahA66/+244KgzjNm3aICcnBzExMahfvz5GjhyJDh06hDs6BRlrE6IKIqnrrq4GaVbD\nsK5o9+7dyMrKQt++fdGrVy9ER3PNZiXsvIm8ZLLr9mdYHzlyBG3atKlxKBcWFhqvR8g/th/ePFfU\nLCvnL5s93KvuYKys4+Pj0bNnT2RlZYUyashY+b0D8HreRBEh1F13lcO65BeM3g7rsk6dOoWCggLs\n2LEDq1atwvjx40OQmiKRbVbeRIEKxao7GCvrmqxduxYLFiyAiCAnJwfbtm1Dp06dAo1NEcT2tQlR\noILRdYd6WFd0zz33IDk5GQsXLsTixYvx9NNPY9euXYjjlftsw/a1CXszs6yc3+l04pZb0vw6rzvY\nNYgvVBWbN2/G008/jQYNGqB3794YNGgQJk+ejDfffNMyfy1p5fcOYPFrm4jIUACL4f6jn6Wq+lww\ntksULt523SaHdUUHDx4E4D6Pu2vXrti9ezcWLVqEvn37YsGCBZg9e3ZYcpAZAdcmIuIAcAjAQADf\nAcgG8HNVPVDhdaxNKCLV1HWHuwbxxQsvvIB//etfeOWVV/DUU0+hoKAA8+fPx9GjR9G9e3e88cYb\nll7Rklsoa5PuAL5U1bySHb0B4HYA1XyENFFkKbvqjqSVdW02b96Mu+66CwDQtWtXLF68GADQtm1b\nrFq1Ch988AGHt40FY+U9BsAQVb235P4EAN1VdUaF1/E87xowvxmX//EJes90Yvjwi+Uej8RhXVZx\ncTGaNm2KI0eOYN++fejYsSOuu+46nDp1yjJdt4dV3zsedeI87/T0dCQlJQFw/2FBampq6TftubiL\nv/f37t0b0Nebvs/8YbofGwvMmwfnxo0AgB5R9fCjoeNxav95pLZujfTnnoMjJgZOpxOngNLBHTH5\nS+5/+OGHmDt3LhISEgAABw4cQFRUFHJzc5GcnGw8H+/7f9/pdGLFihUAUDovqxKMlXcPAJmqOrTk\n/mNwf+bacxVex86bwu/jj4F589wfkltWIJ8UE6Fuv/12TJw4EWPHjjUdhYIolJeEzQbQXkTaiUg9\nAD8H8F4QtkvkO28+3VwVOHQI+J//sc3gBlB6xgnVDQEPb1UtBvAAgM0A9gN4Q1W/CHS7vvL82GFV\nzO+nIAxruxz7rl27YteuXWbD+MEuxz/cgtJ5q+oHAH4UjG0R1agO1SC+8qy8VdVyv7Qk3/HP4ymy\ncVj7pFWrVvjHP/6B5ORk01EoSGz/MWhkE3W4sw4G9t51h22GN3szs/zOHwHD2k7H3orD207HP5xs\nM7zJIvwc1qqKhg0b4s4778T+/fsNfxORq1u3bvjss89Mx6AwYOdNoRXEzjolJQXHjx9HXFwcOnbs\niPvvvx+jRo3ix3uV4XK5kJeXx87bRth5U3iEsAYZM2YMevTogVatWmHKlClYsmQJ2rVrh8zMTHz7\n7bch/Kasw+FwcHDXEbYZ3uzNDCkZ1s4wdNb9+/eHqqJz58546623sHXrVmzevBn//ve/0blzZ4wb\nNw5nz571ebuWPfYlmN8sdt5kDdWtrFu1CvkvGPv06YPs7Gy88MILOHfuHGbNmoVOnTrhpZdeQm5u\nLkaOHIkonn1CdQQ7b6pZhJ1nfcstt+CZZ55BamoqevXqhWnTpmHGjBm1fyGRRbHzJu9EwKl7Nenf\nvz+2bt2Kpk2bIisrC/Pnz8d77/FSOlT32GZ4szfzU5CGdbjy9+/fH9u2bQPgvlzm+vXrcffddwd0\nbjPfO2Yxv39sM7zJSxG+sq5Nr169sHfvXly4cAEA0L17d/zlL3/ByJEjkZeXZzgdUfiw87a7COus\ng+GZZ57B9OnTEV/mo8oWL16MV155Bf/85z/RpEkTg+mIgoudd11h8ZW1NzIyMsoNbgB48MEHMWDA\nAIwZMwYul8tQMt/k5eXB4XBgypQpOHjwIEaNGoWEhAQ0atQIffr0wZYtW4K6v08++QT169dH+/bt\nce7cuXLPHT9+HImJiYiLi8OhQ4dKH9+zZw8efPBBpKamIiEhAbGxsbj22mvx8MMP4/Tp00HNRz5S\n1bDc3LsKnW3btoV0+6Hmd/6dO1WHDVN1j+Qrtw4dVFetUi0qCmrO6kTC8S8qKtLnnntOf/jhB5++\nzlT23NxcFRHt16+fNm3aVPv166cZGRk6efJkbdCggUZFRenatWtr3Y4v+RcuXKgionfccUfpYy6X\nS/v3768Oh0Nfe+21cq+fNm2atmjRQu+44w59+OGHdebMmdqvXz8VEb3++uv1/PnzXu87GPkjUajz\nl8zOSjM1rJ9hSUFgwxokWKKiovDII4+YjuGzjz76CLNnz8b8+fNLH3vggQfQo0cPTJs2DcOGDUOj\nRo1w5swZLFq0qNK1unNzc7F9+/Yqtz169GjccMMNpfdnzpyJrVu3Yt26dRg4cCCmTp2Kp556Ck6n\nE+np6ZgwYUK5r8/IyMDLL79caZ/Lly/H3XffjZdffhmzZ88O9BCQP6qa6KG4IcQrb9uKkJU1BZ9n\n5d20adMqV7Dp6enqcDj01VdfLfd6h8Ph9W3lypWVtvv9999r69attUGDBvrSSy9pdHS0XnfddVpQ\nUOB1dpfLpU2aNNGBAwf6fwDIK6hm5c3OO9LUgc6ayuvSpQsaNmxY6fG0tDSoKj799FMAQLt27eBy\nuVBcXOz17a677qq03YSEBKxevRqXL1/GAw88gHr16mHNmjWIjY2t9NqioiIsWbIEffr0QUJCAqKj\no+FwOBAVFYWzZ8/ymjIG2WZ4W/pc0fXrw3JtkFCy8vE3nT0xMbHKx1u0aAEAOHPmTI1f70/+7t27\no23bthAR9O/fH506darydePGjcOMGTNw/PhxjBo1Co8++igyMzORmZmJJk2a4NKlSz7vuyLTxz9Q\nlv4MSwrQqVNAu3bAM8/U6c66rjpx4kSVjx8/fhwASk99DEbn7TFjxgwcOXIE//Vf/4WNGzfi9ddf\nx5133lnuNbt378b69etx6623IisrCw7HlbWequK5557z/puk4KuqSwnFDey8icqprfOeNGlSSDrv\nNWvWqIho//799ZtvvtHmzZtrXFycfvXVV+Ve98Ybb6iI6JIlSyptY8eOHSoimpycHKSjQdVBNZ03\nhzfVGcuXL9cxY8ZoSkqKxsbGalxcnPbq1Uv/+te/GslTdhjPnj273HPZ2dkaExOjzZo103PnzgVt\nn4cPH9a4uDht3ry5Hjt2TFVVN27cqA6HQ7t27aqXL18ufe3OnTtVRHTs2LHltnHixAnt2rUrh3eY\n2H5481xRs6yQPzY2Vm+++WadPHmyZmRk6C9+8Qtt06aNiog++eSTYc/jGd5paWnarFkz7du3r86Z\nM0cnTZqksbGxGh0drevWrat1O94e+8LCQr355pvV4XDo3/72t3LPzZ49W0VEf/nLX5Y+VlxcrL17\n91aHw6E9e/bURx55RCdNmqTNmzfXvn37aqtWrYIyvK3w3qmJqfO8ObwjBPOHXk5OTqXHCgsLtUuX\nLlqvXj397rvvwprHM7wnT56sBw4c0FGjRmmzZs20YcOG2qdPH92yZYtX2/H22M+aNUsdDoc+9NBD\nlZ4rLCzUH//4x+pwOHT9+vWlj+fn5+v999+vycnJGhsbq+3bt9df//rXevHiRU1KStKUlBSv9h2M\n/JHK1PDmtU2oznvnnXcwduxYrFy5stIfqYSS57Mm09PTsWzZsrDtl6ylumub8GwTqjO+/vprzJ8/\nH1u3bsXRo0dx8eLF0udEhOcsk6XwPO8IwfyhdeTIEdx0003485//jJYtW2Lq1Kl44oknkJmZiSFD\nhgBAUM5ZNiHSj31tmN8/XHlTnbBw4ULk5+djxYoVmDhxYrnnnnzySWzatMlILhGpdN42kTdsM7zT\n0tJMRwgI84fW4cOHAQA//elPKz138uRJIwO0Xbt2KC4uDng7kX7sa8P8/rFNbUJUk6SkJACVf8Td\ntGkTli5dGv5ARAGyzfBmb2ZWpOefPn06YmJiMHbsWEycOBGPPvoohg8fjuHDh6Nv376w8plQkX7s\na8P8/rHN8CaqSefOneF0OtGrVy9kZWXhj3/8I86dO4d33nkHt912G7tnshye501EFMH4GZZERDZi\nm+HN3swsK+e3cnaA+U1j501ERF5j501EFMHYeRMR2Yhthjd7M7OsnN/K2QHmN82SnbeI/E5EvhCR\nvSLylojEBSsYERFVL6DOW0QGAdiqqi4RmQ/3RcPnVPNadt5ERD4KSeetqn9XVVfJ3Z0AWgeyPSIi\n8k4wO+8pADYGcXs+YW9mlpXzWzk7wPymRez1vEVkC4DEsg8BUACPq+r7Ja95HEChqq6uaVvp6eml\nV3eLj49Hampq6eUUPQfA3/t79+4N6OtN32d+3ud93k9LS4PT6cSKFSsAXLkaZlUCPs9bRNIBTAUw\nQFWr/SidlzGPAAAGhklEQVQSdt5ERL4LyWdYishQALMB9K1pcBMRUXAF2nm/CKARgC0iskdEXg5C\nJr94fuywKuY3x8rZAeY3zVT+gFbeqtohWEGIiMh7vLYJEVEE47VNiIhsxDbDm72ZWVbOb+XsAPOb\nZiq/bYY3EVFdws6biCiCsfMmIrIR2wxv9mZmWTm/lbMDzG8aO28iIvIaO28iogjGzpuIyEZsM7zZ\nm5ll5fxWzg4wv2nsvImIyGvsvImIIhg7byIiG7HN8GZvZpaV81s5O8D8prHzJiIir7HzJiKKYOy8\niYhsxDbDm72ZWVbOb+XsAPObxs6biIi8xs6biCiCsfMmIrIR2wxv9mZmWTm/lbMDzG8aO28iIvIa\nO28iogjGzpuIyEZsM7zZm5ll5fxWzg4wv2nsvImIyGvsvImIIhg7byIiG7HN8GZvZpaV81s5O8D8\nprHzJiIir7HzJiKKYOy8iYhsxDbDm72ZWVbOb+XsAPObxs6biIi8xs6biCiCsfMmIrKRoAxvEZkl\nIi4RaRaM7fmDvZlZVs5v5ewA85tm2c5bRFoDGAwgL/A4RETkjYA7bxFZB+ApAO8B6Kqq/6nmdey8\niYh8FJLOW0RGAvhaVfcFsh0iIvJNrcNbRLaIyL/K3PaV/HMkgAwAc8u+PGRJa8HezCwr57dydoD5\nTTOVP7q2F6jq4KoeF5FOAJIA/J+ICIDWAHaLSHdVPVnV16SnpyMpKQkAEB8fj9TUVKSlpQG4cgD8\nvb93796Avt70febnfd7n/bS0NDidTqxYsQIASudlVYJ2nreIHAHQRVXzq3menTcRkY/CcZ63wmBt\nQkRUlwRteKtqSnVnmoSD58cOq2J+c6ycHWB+00zl519YEhFZEK9tQkQUwXhtEyIiG7HN8GZvZpaV\n81s5O8D8prHzJiIir7HzJiKKYOy8iYhsxDbDm72ZWVbOb+XsAPObxs47QJ5ra1gV85tj5ewA85tm\nKr9thvfp06dNRwgI85tj5ewA85tmKr9thjcRUV1im+Gdm5trOkJAmN8cK2cHmN80U/nDeqpgWHZE\nRGQzVZ0qGLbhTUREwWOb2oSIqC7h8CYisiDbDW8R+aWIfFHyQcnzTefxlYjMEhGXiDQzncUXIvK7\nkuO+V0TeEpE405m8ISJDReSAiBwSkUdN5/GFiLQWka0isr/k/T7DdCZfiYhDRPaIyHums/hKRJqI\nyLqS9/1+EflxOPdvq+EtImkAbgPQWVU7A1hgNpFvRKQ1gMEA8kxn8cNmANeraiqALwHMMZynViLi\nALAEwBAA1wO4U0Q6mk3lkyIAM1X1egC3ALjfYvkB4EEAn5sO4afnAWSp6n8DuBHAF+Hcua2GN4D7\nAMxX1SIAUNXvDefx1SIAs02H8Ieq/l1VXSV3dwJobTKPl7oD+FJV81S1EMAbAG43nMlrqnpcVfeW\n/Pt5uIdHK7OpvFeyWPkJgFdMZ/FVyU+WfVR1OQCoapGqng1nBrsN72sB9BWRnSKyTUS6mQ7kLREZ\nCeBrVd1nOksQTAGw0XQIL7QC8HWZ+9/AQsOvLBFJApAK4GOzSXziWaxY8ZS3ZADfi8jyktrnzyIS\nG84A0eHcWTCIyBYAiWUfgvt//F/D/f00VdUeInIzgLUAUsKfsmq1ZM+AuzIp+1xEqSH/46r6fslr\nHgdQqKqrDUSsk0SkEYA3ATxYsgKPeCIyHMAJVd1bUndG3Pu9FtEAugC4X1V3ichiAI8BmBvOAJai\nqoOre05EpgF4u+R12SW/+EtQ1VNhC1iD6rKLSCcASQD+T0QE7spht4h0V9WTYYxYo5qOPQCISDrc\nPwYPCEugwH0LoG2Z+61LHrMMEYmGe3C/pqrvms7jg14ARorITwDEAmgsIq+q6l2Gc3nrG7h/Ut5V\ncv9NAGH9hbfdapP1KBkcInItgJhIGdw1UdXPVLWFqqaoajLcb4ybImlw10ZEhsL9I/BIVb1kOo+X\nsgG0F5F2IlIPwM8BWO2sh2UAPlfV500H8YWqZqhqW1VNgfu4b7XQ4IaqngDwdcmcAYCBCPMvXi23\n8q7FcgDLRGQfgEsALPNmqEBhvR8jXwRQD8AW9w8P2Kmq081GqpmqFovIA3CfKeMAsFRVw3rGQCBE\npBeA8QD2icincL9vMlT1A7PJ6owZAFaJSAyAHACTw7lz/nk8EZEF2a02ISKqEzi8iYgsiMObiMiC\nOLyJiCyIw5uIyII4vImILIjDm4jIgji8iYgs6P8DKEoXb/fywMMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1044a9b38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "plt.style.use(\"seaborn-dark-palette\")\n",
    "\n",
    "fig = plt.figure()\n",
    "plt.axis('equal')\n",
    "plt.axis([-7, 7, -6, 6])\n",
    "plt.arrow(-4, -1, 8, 2, head_width=0.3, head_length=0.5, color='r', length_includes_head=True)\n",
    "plt.arrow(0, 0, 2, 4, head_width=0.3, head_length=0.5, color='b', length_includes_head=True)\n",
    "plt.arrow(0, 0, 48/17, 12/17, head_width=0.3, head_length=0.5, color='gray', length_includes_head=True)\n",
    "plt.arrow(48/17, 12/17, 2-48/17, 4-12/17, head_width=0.3, head_length=0.5, color='g', length_includes_head=True)\n",
    "# plt.plot([48/17], [12/17], 'o')\n",
    "# y=1/4x\n",
    "# y=-4x+12\n",
    "# x=48/17\n",
    "# y=12/17\n",
    "plt.annotate('b', xy=(1, 2), xytext=(-30, 15), textcoords='offset points', size=20, arrowprops=dict(arrowstyle=\"->\"))\n",
    "plt.annotate('a', xy=(-1, -0.25), xytext=(15, -30), textcoords='offset points', size=20, arrowprops=dict(arrowstyle=\"->\"))\n",
    "plt.annotate('e=b-p', xy=(2.5, 2), xytext=(30, 0), textcoords='offset points', size=20, arrowprops=dict(arrowstyle=\"->\"))\n",
    "plt.annotate('p=xa', xy=(2, 0.5), xytext=(-20, -40), textcoords='offset points', size=20, arrowprops=dict(arrowstyle=\"->\"))\n",
    "plt.grid()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.close(fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从图中我们知道，向量$e$就像是向量$b, p$之间的误差，$e=b-p, e \\bot p$。$p$在$a$上，有$\\underline{p=ax}$。\n",
    "\n",
    "所以有$a^Te=a^T(b-p)=a^T(b-ax)=0$。关于正交的最重要的方程：\n",
    "\n",
    "$$\n",
    "a^T(b-xa)=0 \\\\\n",
    "\\underline{xa^Ta=a^Tb} \\\\\n",
    "\\underline{x=\\frac{a^Tb}{a^Ta}} \\\\\n",
    "p=a\\frac{a^Tb}{a^Ta}\n",
    "$$\n",
    "\n",
    "从上面的式子可以看出，如果将$b$变为$2b$则$p$也会翻倍，如果将$a$变为$2a$则$p$不变。\n",
    "\n",
    "设投影矩阵为$P$，则可以说投影矩阵作用与某个向量后，得到其投影向量，$projection_p=Pb$。\n",
    "\n",
    "易看出$\\underline{P=\\frac{aa^T}{a^Ta}}$，若$a$是$n$维列向量，则$P$是一个$n \\times n$矩阵。\n",
    "\n",
    "观察投影矩阵$P$的列空间，$C(P)$是一条通过$a$的直线，而$rank(P)=1$（一列乘以一行：$aa^T$，而这一列向量$a$是该矩阵的基）。\n",
    "\n",
    "投影矩阵的性质：\n",
    "\n",
    "* $\\underline{P=P^T}$，投影矩阵是一个对称矩阵。\n",
    "* 如果对一个向量做两次投影，即$PPb$，则其结果仍然与$Pb$相同，也就是$\\underline{P^2=P}$。\n",
    "\n",
    "为什么我们需要投影？因为就像上一讲中提到的，有些时候$Ax=b$无解，我们只能求出最接近的那个解。\n",
    "\n",
    "$Ax$总是在$A$的列空间中，而$b$却不一定，这是问题所在，所以我们可以将$b$变为$A$的列空间中最接近的那个向量，即将无解的$Ax=b$变为求有解的$A\\hat{x}=p$（$p$是$b$在$A$的列空间中的投影，$\\hat{x}$不再是那个不存在的$x$，而是最接近的解）。\n",
    "\n",
    "现在来看$\\mathbb{R}^3$中的情形，将向量$b$投影在平面$A$上。同样的，$p$是向量$b$在平面$A$上的投影，$e$是垂直于平面$A$的向量，即$b$在平面$A$法方向的分量。\n",
    "设平面$A$的一组基为$a_1, a_2$，则投影向量$p=\\hat{x_1}a_1+\\hat{x_2}a_2$，我们更倾向于写作$p=A\\hat{x}$，这里如果我们求出$\\hat{x}$，则该解就是无解方程组最近似的解。\n",
    "\n",
    "现在问题的关键在于找$e=b-A\\hat{x}$，使它垂直于平面，因此我们得到两个方程\n",
    "$\n",
    "\\begin{cases}a_1^T(b-A\\hat{x})=0\\\\\n",
    "a_2^T(b-A\\hat{x})=0\\end{cases}\n",
    "$，将方程组写成矩阵形式\n",
    "$\n",
    "\\begin{bmatrix}a_1^T\\\\a_2^T\\end{bmatrix}\n",
    "(b-A\\hat{x})=\n",
    "\\begin{bmatrix}0\\\\0\\end{bmatrix}\n",
    "$，即$A^T(b-A\\hat{x})=0$。\n",
    "\n",
    "比较该方程与$\\mathbb{R}^2$中的投影方程，发现只是向量$a$变为矩阵$A$而已，本质上就是$A^Te=0$。所以，$e$在$A^T$的零空间中（$e\\in N(A^T)$），从前面几讲我们知道，左零空间$\\bot$列空间，则有$e\\bot C(A)$，与我们设想的一致。\n",
    "\n",
    "再化简方程得$A^TAx=A^Tb$，比较在$\\mathbb{R}^2$中的情形，$a^Ta$是一个数字而$A^TA$是一个$n$阶方阵，而解出的$x$可以看做两个数字的比值。现在在$\\mathbb{R}^3$中，我们需要再次考虑：什么是$\\hat{x}$？投影是什么？投影矩阵又是什么？\n",
    "\n",
    "* 第一个问题：$\\hat x=(A^TA)^{-1}A^Tb$；\n",
    "* 第二个问题：$p=A\\hat x=\\underline{A(A^TA)^{-1}A^T}b$，回忆在$\\mathbb{R}^2$中的情形，下划线部分就是原来的$\\frac{aa^T}{a^Ta}$；\n",
    "* 第三个问题：易看出投影矩阵就是下划线部分$P=A(A^TA)^{-1}A^T$。\n",
    "\n",
    "这里还需要注意一个问题，$P=A(A^TA)^{-1}A^T$是不能继续化简为$P=AA^{-1}(A^T)^{-1}A^T=I$的，因为这里的$A$并不是一个可逆方阵。\n",
    "也可以换一种思路，如果$A$是一个$n$阶可逆方阵，则$A$的列空间是整个$\\mathbb{R}^n$空间，于是$b$在$\\mathbb{R}^n$上的投影矩阵确实变为了$I$，因为$b$已经在空间中了，其投影不再改变。\n",
    "\n",
    "再来看投影矩阵$P$的性质：\n",
    "* $P=P^T$：有\n",
    "$\n",
    "\\left[A(A^TA)^{-1}A^T\\right]^T=A\\left[(A^TA)^{-1}\\right]^TA^T\n",
    "$，而$(A^TA)$是对称的，所以其逆也是对称的，所以有$A((A^TA)^{-1})^TA^T=A(A^TA)^{-1}A^T$，得证。\n",
    "* $P^2=P$：有\n",
    "$\n",
    "\\left[A(A^TA)^{-1}A^T\\right]\\left[A(A^TA)^{-1}A^T\\right]=A(A^TA)^{-1}\\left[(A^TA)(A^TA)^{-1}\\right]A^T=A(A^TA)^{-1}A^T\n",
    "$，得证。\n",
    "\n",
    "## 最小二乘法\n",
    "\n",
    "接下看看投影的经典应用案例：最小二乘法拟合直线（least squares fitting by a line）。\n",
    "\n",
    "我们需要找到距离图中三个点 $(1, 1), (2, 2), (3, 2)$ 偏差最小的直线：$b=C+Dt$。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD7CAYAAABpJS8eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHglJREFUeJzt3Xt0VPX57/H3ExSJCEKLcgnFAPXW2hIuIpTTZY63KPAT\nta5qwcWKXRy8IMqxP9HWUrwutNUWRWllHSpSfhU1p8fSUlBbCRCk2h8alRAESUIFKoqAFI0Y4Tl/\nZMjOwITMMGP2TubzWmuWs/f+Zu9vHsmTnc/s2WPujoiItH05YU9ARERahhq+iEiWUMMXEckSavgi\nIllCDV9EJEuo4YuIZIljwp7AocxM14mKiKTI3a25MZE8w3f3rH+oFsFj+vTpoc8hKg/VQrVI9EhW\nJBu+SGM1NTVhTyEyVIuAapE6NXwRkSyhhi+RV1xcHPYUIkO1CKgWqbNU8p+WYGYetTmFwaz+9RfV\nQkSaY2Z4S7xoa2a9zexlM6sws7fN7OYEY841s91m9nrs8dN0jyvZo7S0NOwpRIZqEVAtUpeJyzK/\nAG5193IzOwFYY2Yvuvv6Q8atcPdLM3A8ERE5ChmPdMzseWCWu/+t0bpzgf909/9I4usV6aBIR0SS\n12KRziEHzQcKgFcTbB5uZuVmttjMvpHJ44qISPMy1vBjcU4JcIu77z1k8xqgj7sXAI8Bz2fquNL2\nKasNqBYB1SJ1Gbm1gpkdQ32z/527//HQ7Y1/Abj7EjObbWZfcfedifZXXFxMfn4+AF26dKGgoIDC\nwkIg+J/c1pcPisp8wlwuLy+P1HzCXC4vL4/UfLQczvLB56m++SwjGb6ZzQd2uPutTWzv7u7bY8+H\nAs+6e34TY5XhowxfRJKXbIaf9hm+mY0AxgFvm9kbgAM/AU4B3N3nAFea2Q1AHVALXJXucUVEJDVp\nZ/juvsrd27l7gbsPdPdB7r7U3Z+INXvc/XF3Pyu2/TvunuhFXZGEGv8Zm+3aUi12795Njx49qK6u\nPqqvT6cWs2fP5tJLs+8qcd1aQURCcf/99zNq1Cj69u0LwI4dO7j44ovJy8ujQ4cO9OnTh5tuuok9\ne/aktN/ly5dz2WWX0atXLzp27MiAAQN48skn48ZMmDCBNWvWsGrVqox9P62Bbq0QUcrwpS2rra0l\nLy+PxYsXM3z4cAB27txJSUkJZ599Nt26dePdd9/lxhtvZMCAASxcuDDpfc+YMYPa2louueQSevbs\nydKlS5k8eTK/+93vuPrqqxvG3XbbbWzevJlnn302499fS0s2w1fDjyg1fGnLSkpKuP7669mxY8cR\nx82aNYsHHniArVu3pnW8q666igMHDvDcc881rFu5ciUXXXQRu3btokOHDmntP2yhvPFK5MvQlnLr\ndLWVWpSVlTF48OAjjtm2bRt/+MMfGi5JPFQqtdizZw9du3aNWzdkyBDq6upYvXp10vtp7dTwRaTF\nbd68mV69eiXcNnbsWDp27Ejv3r3p3Lkzv/3tb9M61p///Gdefvllrrvuurj1ubm5nHjiiVn1QSpq\n+BJ5TZ3hZaO2Uova2tomY5SZM2fyxhtvsGjRIqqqqrjlllsSjkumFqtWrWLcuHHMmjUr4V8Uubm5\n1NbWpjT31ixyH2IuIm1ft27d2LVrV8JtJ598MieffDKnnXYaXbt25bvf/S7Tpk0jLy8vpWOUlZUx\natQo7rvvPiZOnJhwzM6dOznppJNSnn9rpTN8iby2kltnQlupxcCBA1m3bl2z4/bv34+ZsW/fvsO2\nHakWK1asYOTIkdxzzz1Mnjw54Ziqqir27dvHoEGDkp53a6czfBFpcUVFRdxxxx3s2rWr4cXUxYsX\n89FHHzF48GBOOOEE1q5dy9SpUxk+fDj9+vVLet+lpaWMHj2aSZMmcfXVV7N9+3YA2rVrR7du3RrG\nrVy5kn79+tG/f//MfnNR5u6RetRPSai/RUXY0xD50nznO9/x2bNnNyz/9a9/9eHDh3vXrl39+OOP\n99NPP91//OMf++7du+O+zsz87rvvbnK/xcXFnpOTc9ijb9++ceOKior85z//eWa/qZDEekWz/VXX\n4UeUrsOXtu6FF15gypQprFu3ruHfe3Oqq6s59dRTKSsrY9iwYUd97IqKCi644AI2bNhAp06djno/\nUaHr8KXNaCu5dSa0pVoUFRUxadIktmzZkvTXLFmyhPHjxzNs2LC0arFt2zbmz5/fJpp9KpThi0ho\nbrrpppTG33jjjRk57oUXXpiR/bQ2inQiSpGOiCRLkY6IiMRRw5fIa0u5dbpUi4BqkTo1fBFplRYt\nWsRbb70V9jRaFTV8iby2cv+YTFAtAps2bTrqT8vKVmr4ItIqffbZZ63+PvYtTQ1fIk9ZbSDba3Hg\nwIGGK9fef/99cnNzQ55R66KGLyKtxgMPPMADDzwAwL59++jQoQO7du1izJgxIc+sdVDDl8hTbh3I\n9lpcfvnlzJw5k08//ZRjjz2W3NxcZsyYQffu3cOeWqugd9qKSKtx5plncs455/DUU09RW1vLrl27\nmDt3Lm+//XbYU2sVdIYvkZftuXVjqgVMnTqVhx9+mN27dzNr1ixuuOGGJj8uUeKp4UubsHv3bnr0\n6BHKZXqzZ8/m0ksvbfHjZqsRI0Zw0kkn8e9//5vly5czderUsKfUaqTd8M2st5m9bGYVZva2md3c\nxLhHzWyjmZWbWUG6x5XskUxuff/99zNq1Cj69u0LwFtvvcXYsWPp06cPxx9/PGeccQa/+MUvUr43\n0fLly7nsssvo1asXHTt2ZMCAATz55JNxYyZMmMCaNWtYtWpVSvs+Gtme4UP9fWOmTp3K559/ztSp\nU+ncuXPYU2o1MpHhfwHc6u7lZnYCsMbMXnT39QcHmNklQH93P9XMzgF+Axz9zaxFGqmtrWXu3Lks\nXry4Yd2aNWs4+eSTWbBgAX369OG1115jwoQJ7N+/nzvuuCPpfb/yyit8+9vf5vbbb6dnz54sXbqU\niRMnkpuby9VXXw1A+/btGTt2LI888ggjRow46u9jypQpdOrUiZtvvvmwz1ndXF3NvGnTOLB1Kzl5\neRTfey+nxH65ZZvN1dWUP/ccX//KV/j4v/+bzdXVWVuLlCXzKSmpPIDngfMPWfcb4KpGy5VA9ya+\nPoOfA9N6oU+8arBs2bIjbn/uuef8q1/9arP7mTp1qg8ZMiTt+Xz/+9/3K6+8Mm7dihUrvEOHDl5b\nW3vU+920aZNPnDjRu3bt6pMnT/aamhp3d6+pqvIf9e/ve8GXge8F/1H//l5TVZXW99EaqRaJkeQn\nXmU0wzezfKAAePWQTXnAe42Wt8bWiaStrKyMwYMHNztuz549DZ+fmo5E+xkyZAh1dXWsXr065f25\nO3V1dXTv3p0ZM2awbNkyPv/8cwoKChgzZgz3XHcdd2/aRMfY+I7A3Zs2MW/atLS/l9Zm3rRpqkUa\nMnZZZizOKQFucfe96eyruLiY/Px8ALp06UJBQUFDdnnwKoW2vnxQVOYT9vJBibavWbOGr3/960f8\n+s6dOzNv3jzuvPNOSktLj3o+M2bM4G9/+1tDY2+8/cQTT2Tp0qWY2WFfP2fOHNasWcPHH3/MF198\nAdS/cWjfvn3U1dXRrl07jjvuOHJycjj22GPp2LEj7du3Z9GiRXQ+7jjmNq4BUAgc2LYtMv9/Wmq5\nau1a/hH7/guprwXU1yIK82vJn4fS0lJqampISTJ/BjT3oP4Xx1Lqm32i7YdGOutRpHNEKNJJWlFR\nkV9//fVNbl+/fr337NnTf/KTn6R1nLKyMu/cubM/8cQTCbfn5eX5448/nnDbtm3b/J133vHq6mrf\nunWrf/jhh75nzx7/7LPPfP/+/XFjX3nlFT/33HP99NNP95KSEp8+dqzvBfdGj73gd40bl9b30xrd\nNW6capEASUY6mWr484FfHmH7SGBx7Pkw4O9HGPvlVaUVUcMPNJfhjxs3zq+66qqE2yorK71nz57+\nox/9KK05rFy50jt37uyPPvpok2Nyc3P92WefPepjbNiwwceMGeNf+9rXfO7cuV5XV+fuyq0bUy0S\na7GGD4wA9gPlwBvA68DFwHXAxEbjHgPeBd4EBh1hf192bVoFNfxAcw3/oYce8m9961uHra+oqPAe\nPXr4lClT0jr+8uXLvVOnTj5z5swmx2zatMlzcnL83XffPerj/PrXv/aHHnoo4Qu/NVVVfte4cT6+\noMDvGjcuqxucanG4ZBu+PtM2ovSZtslbu3YtAwcO5IMPPmh4MbWiooLzzjuP888/n1/96ldx41O5\n70ppaSmjR49m0qRJ3HrrrQ3r27VrR7du3RqWn3rqKe677z42btyY5ncjkjp9pq1kjbPOOouhQ4ey\ncOHChnUlJSXs2LGDZ555hl69etGrVy969ux52Fvwc3JyuOeee5rc98F7tjz00EMN++nVqxdDhw6N\nG/f0008zceLEzH5jIpmWzJ8BLflAMYa7K9JprLlIx9196dKlfsYZZ/iBAweS3m9VVZW3a9fOV69e\nncbs3NeuXes9evTwPXv2pLWfZCRTi2yhWgQI4zp8kbAUFRUxadIktmzZkvTXLFmyhPHjxzNsWHpv\n+t62bRvz58+nU6dOae1H5MumDD+ilOGLSLKU4YuISBw1fIm8Q99tm81Ui4BqkTo1fBGRLKEMP6KU\n4YtIspThi4hIHDV8iTxltQHVIqBapE4NX0QkSyjDjyhl+CKSLGX4IiISRw1fIk9ZbUC1CKgWqVPD\nFxHJEsrwI0oZvogkSxm+iIjEUcOXyFNWG1AtAqpF6tTwRUSyhDL8iFKGLyLJUoYvIiJx1PAl8pTV\nBlSLgGqROjV8EZEsoQw/opThi0iyWjTDN7O5ZrbdzN5qYvu5ZrbbzF6PPX6aieOKiEjyMhXpPAkU\nNTNmhbsPij3uy9BxJQsoqw2oFgHVInUZafjuXgbsamZYs39uiIjIlydjGb6ZnQL8yd2/nWDbucD/\nBbYAW4Hb3H1dE/tRho8yfBFJXrIZ/jEtMRlgDdDH3T81s0uA54HTWujYIiJCCzV8d9/b6PkSM5tt\nZl9x952JxhcXF5Ofnw9Aly5dKCgooLCwEAhyu7a+fFBU5hPmcnl5OVOmTInMfMJcnjlzZlb+PCRa\nbvyzEoX5tOTywec1NTWkIpORTj71kc63Emzr7u7bY8+HAs+6e34T+1GkgyKdxkpLSxv+wWc71SKg\nWgSSjXQy0vDN7PdAIfBVYDswHWgPuLvPMbNJwA1AHVAL/G93f7WJfanho4YvIslr0YafSWr49cJo\n+Lt37+aMM85g9erV9O3bt8WOCzB79myWLl3KokWLWvS4Im2Bbp4mKbv//vsZNWpUXLOfMmUKZ599\nNrm5ufTr1++o9rtv3z6uvfZaBgwYQPv27TnvvPMOGzNhwgTWrFnDqlWrDtvWOLfMdqpFQLVInRq+\nAFBbW8vcuXOZMGFC3Hp3p7i4mPHjxx/1vvfv309ubi6TJ09m9OjRCce0b9+esWPH8sgjjxz1cUTk\nyBTpRFRLRzolJSVcf/317NixI+H2hx9+mMcff5yqqqq0jjN58mQqKip4+eWXD9u2cuVKLrroInbt\n2kWHDh3SOo5INlGkIykpKytj8ODBoc5hyJAh1NXVsXr16lDnIdJWqeELAJs3b6ZXr16hziE3N5cT\nTzzxsGuLldUGVIuAapE6NXwB6jP8KMQoubm51NbWhj0NkTZJDV8A6NatG7t2NXf/uy/fzp07Oemk\nk+LW6c01AdUioFqkTg1fABg4cCDr1iW8n12LqaqqYt++fQwaNCjUeYi0VWr4AkBRURGVlZWHneVv\n2rSJN998k61bt/L555/z5ptv8uabb/LFF1+ktP/KykrKy8vZsWMHe/fubdhPYytXrqRfv370798/\nbr2y2oBqEVAtUtdSd8uUiDvrrLMYOnQoCxcu5IYbbmhYP2HCBFasWNGwfPDsu7q6mj59+gCQk5PD\nXXfdxc9+9rMm9z9y5Ej++c9/NiwPHDgQM2P//v0N655++mkmTpyYse9JROLpOvyICuPWCi+88AJT\npkxh3bp1DcdvTnV1NaeeeiplZWUMGzbsqI9dUVHBBRdcwIYNG+jUqdNR70ckG+k6fElZUVERkyZN\nYsuWLUl/zZIlSxg/fnxazR5g27ZtzJ8/X81e5EukM/yI0t0yA7oNbkC1CKgWAZ3hi4hIHJ3hR5TO\n8EUkWTrDl8j66KOP+OCDD8KehkjWUcOXFvf3v/+d4cOHs3379qTG63rrgGoRUC1Sp4YvLW7UqFFc\nc801jB49mk8++STs6YhkDWX4EdXWM3x354c//CEffvghzz//PMcco/cAihwtZfgSaWbGnDlzqKur\nY9KkSW32F5tIlKjhS2iOPfZYSkpKeO2115gxY0bD+vnz51NZWdmwrKw2oFoEVIvUqeFLqDp16sRf\n/vIX5syZw/z58wHYuHEjCxYsCHlmIm2PMvyIausZ/qEqKyspLCxkwYIFtG/fnqlTp/Lqq6+GPS2R\nVkEZvrQqZ555JiUlJYwbN47jjz8+4a2aRSQ9avgSuiuuuILvfe97fPbZZ8yaNYsrrriCQYMGsWzZ\nMkBZbWOqRUC1SF1GGr6ZzTWz7Wb21hHGPGpmG82s3MwKMnFcaRvmzZvHBRdcwG233cadd95JQUEB\na9euZfHixWFPTaRNyUiGb2b/A9gLzHf3byfYfglwk7uPMrNzgEfcPeH9dJXh18u2DB/qv9dXX32V\nJ554ggULFnBc+/bcdvnlHNi6lZy8PIrvvZdT+vYNe5oikZNshp+xF23N7BTgT000/N8Ay9z9mdhy\nJVDo7oe9t14Nv142NvzGyl9/nduLivjDjh10BD4Bpvfvz+SXXlLTFzlE1F60zQPea7S8NbZOJKE/\n/vKXDc2+FOgI3L1pE/OmTQt3YiFTbh1QLVIXyfezFxcXk5+fD0CXLl0oKCho+KCDg/+T2/ryQVGZ\nT0svH9i6taHZlwOF1Df9qoqKuA++iMp8W2q5vLw8UvPRcjjLB5/X1NSQirAinfXAuYp0mpbtkc7d\n11zDf/7Xf9Gx0bpPgIfGjWO63pQlEieMSMdij0QWAeNjExsG7E7U7EUOKr73Xqb378/Be2kezPCL\n7703zGmJtGqZukrn99T/1f1VYDswHWgPuLvPiY15DLiY+p/da9399Sb2pTN8dIYPsLm6mnnTplFV\nUUG/b35TV+mgz3FtTLUIJHuGn5EM393HJjHmpkwcS7LHKX37Mn3BAv1gi2SI7qUTUTrDF5FkRe2y\nTBERCZkavkRe40vRsp1qEVAtUqeGLyKSJZThR5QyfBFJljJ8ERGJo4YvkaesNqBaBFSL1Knhi4hk\nCWX4EaUMX0SSpQxfRETiqOFL5CmrDagWAdUidWr4IiJZQhl+RCnDF5FkKcMXEZE4avgSecpqA6pF\nQLVInRq+iEiWUIYfUcrwRSRZyvBFRCSOGr5EnrLagGoRUC1Sp4YvIpIllOFHlDJ8EUmWMnwREYmj\nhi+Rp6w2oFoEVIvUqeGLiGSJjGT4ZnYxMJP6XyBz3f3BQ7afC/wRqIqt+oO739fEvpThowxfRJKX\nbIZ/TAYOlAM8BpwPbAP+YWZ/dPf1hwxd4e6Xpns8ERE5OpmIdIYCG919s7vXAQuBMQnGNfvbRyQR\nZbUB1SKgWqQuEw0/D3iv0fKW2LpDDTezcjNbbGbfyMBxRUQkBWln+Gb2PaDI3SfGlq8Bhrr7zY3G\nnAAccPdPzewS4BF3P62J/SnDRxm+iCSvxTJ8YCvQp9Fy79i6Bu6+t9HzJWY228y+4u47E+2wuLiY\n/Px8ALp06UJBQQGFhYVA8GdcW18+KCrz0bKWtRyd5YPPa2pqSEUmzvDbAe9Q/6Ltv4DXgB+4e2Wj\nMd3dfXvs+VDgWXfPb2J/OsNHZ/iNlZaWNvyDz3aqRUC1CLTYGb677zezm4AXCS7LrDSz6+o3+xzg\nSjO7AagDaoGr0j2uiIikRvfSiSid4YtIsnQvHRERiaOGL5HX+IWqbKdaBFSL1Knhi4hkCWX4EaUM\nX0SSpQxfRETiqOFL5CmrDagWAdUidWr4IiJZQhl+RCnDF5FkKcMXEZE4avgSecpqA6pFQLVInRq+\niEiWUIYfUcrwRSRZyvBFRCSOGr5EnrLagGoRUC1Sp4YvIpIllOFHlDJ8EUmWMnwREYmjhi+Rp6w2\noFoEVIvUqeGLiGQJZfgRpQxfRJKlDF9EROKo4UvkKasNqBYB1SJ1avgiIllCGX5EKcMXkWQpwxcR\nkTgZafhmdrGZrTezDWZ2exNjHjWzjWZWbmYFmTiuZAdltQHVIqBapC7thm9mOcBjQBHwTeAHZnbG\nIWMuAfq7+6nAdcBv0j2uiIikJu0M38yGAdPd/ZLY8h2Au/uDjcb8Bljm7s/EliuBQnffnmB/yvBR\nhi8iyUs2wz8mA8fKA95rtLwFGNrMmK2xdYc1fAianagWIpI5etFWRCRLZOIMfyvQp9Fy79i6Q8d8\nrZkxDRRjKNJprLS0lMLCwrCnEQmqRUC1CCSbBGQiw28HvAOcD/wLeA34gbtXNhozEpjk7qNimf9M\ndx/WxP6U4aOGLyLJa7EM3933m9lNwIvUR0Rz3b3SzK6r3+xz3P0vZjbSzN4FPgGuTfe4IiKSGr3T\nNqJ0hh/Qn+4B1SKgWgT0TlsREYmjM/yI0hm+iCRLZ/giIhJHDV8iT/dMCagWAdUidWr4IiJZQhl+\nRCnDF5FkKcMXEZE4avgSecpqA6pFQLVInRq+iEiWUIYfUcrwRSRZyvBFRCSOGr5EnrLagGoRUC1S\np4YvIpIllOFHlDJ8EUmWMnwREYmjhi+Rp6w2oFoEVIvUqeGLiGQJZfgRpQxfRJKlDF9EROKo4Uvk\nKasNqBYB1SJ1avgiIllCGX5EKcMXkWQpwxcRkThq+BJ5ymoDqkVAtUjdMel8sZl1BZ4BTgFqgO+7\n+8cJxtUAHwMHgDp3H5rOcUVEJHVpZfhm9iDwkbv/3MxuB7q6+x0JxlUBg919VxL7VIaPMnwRSV5L\nZfhjgKdiz58CLmtqPhk4loiIpCHdJnyyu28HcPf3gZObGOfAS2b2DzP7X2keU7KMstqAahFQLVLX\nbIZvZi8B3Ruvor6B/zTB8KbyhxHu/i8zO4n6xl/p7mUpz1ZERI5asw3f3S9sapuZbTez7u6+3cx6\nAB80sY9/xf77oZn9P2Ao0GTDLy4uJj8/H4AuXbpQUFBAYWEhEPxWb+vLB0VlPmEvHxSV+YS1fHBd\nVOYT5nJhYWGk5tOSywef19TUkIpMvGi7090fbOpFWzM7Hshx971m1hF4Ebjb3V9sYp960Ra9aCsi\nyWupF20fBC40s3eA84EHYgfvaWZ/jo3pDpSZ2RvA34E/NdXsRRI59Cw/m6kWAdUidWldh+/uO4EL\nEqz/FzA69rwaKEjnOCIikj7dSyeiFOmISLJ0Lx0REYmjhi+Rp6w2oFoEVIvUqeFL5JWXl4c9hchQ\nLQKqRerU8CXydu/eHfYUIkO1CKgWqVPDFxHJEmr4EnmpvpuwLVMtAqpF6iJ5WWbYcxARaW2SuSwz\ncg1fRES+HIp0RESyhBq+iEiWiFzDN7MrzWytme03s0FhzycMZnaxma03sw2xu5BmJTObG7sF91th\nzyVsZtbbzF42swoze9vMbg57TmExs+PM7FUzeyNWi+lhzylsZpZjZq+b2aIjjYtcwwfeBi4Hloc9\nkTCYWQ7wGFAEfBP4gZmdEe6sQvMk9XUQ+AK41d2/CQwHJmXrvwt33wf8T3cfSP2NGS8xs6EhTyts\ntwDrmhsUuYbv7u+4+0bqP1krGw0FNrr7ZnevAxZS/9nBWSf2qWjNfvB9NnD39929PPZ8L1AJ5IU7\nq/C4+6exp8dRf9ffrL36xMx6AyOB/9Pc2Mg1fCEPeK/R8hay+AdbDmdm+dSf2b4a7kzCE4sw3gDe\nB15y93+EPacQ/Qq4jSR+6YXS8M3sJTN7q9Hj7dh//yOM+Yi0FmZ2AlAC3BI7089K7n4gFun0Bs4x\ns2+EPacwmNkoYHvsrz+jmWQkrQ9AOVpH+pxcYSvQp9Fy79g6yXJmdgz1zf537v7HsOcTBe6+x8yW\nAReTRIbdBo0ALjWzkUAu0MnM5rv7+ESDox7pZGOO/w/g62Z2ipm1B64GjvjKexvX7FlLFvktsM7d\nHwl7ImEys25mdmLseS5wIbA+3FmFw91/4u593L0f9b3i5aaaPUSw4ZvZZWb2HjAM+LOZLQl7Ti3J\n3fcDN1H/Ye8VwEJ3rwx3VuEws98DrwCnmdk/zezasOcUFjMbAYwDzotdjvi6mV0c9rxC0hNYZmbl\n1L+O8YK7/yXkObUKurWCiEiWiNwZvoiIfDnU8EVEsoQavohIllDDFxHJEmr4IiJZQg1fRCRLqOGL\niGQJNXwRkSzx/wF1KKmlk+PdHwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1175bd2b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.style.use(\"seaborn-dark-palette\")\n",
    "\n",
    "fig = plt.figure()\n",
    "plt.axis('equal')\n",
    "plt.axis([-1, 4, -1, 3])\n",
    "plt.axhline(y=0, c='black', lw='2')\n",
    "plt.axvline(x=0, c='black', lw='2')\n",
    "\n",
    "plt.plot(1, 1, 'o', c='r')\n",
    "plt.plot(2, 2, 'o', c='r')\n",
    "plt.plot(3, 2, 'o', c='r')\n",
    "\n",
    "plt.annotate('(1, 1)', xy=(1, 1), xytext=(-40, 20), textcoords='offset points', size=14, arrowprops=dict(arrowstyle=\"->\"))\n",
    "plt.annotate('(2, 2)', xy=(2, 2), xytext=(-60, -5), textcoords='offset points', size=14, arrowprops=dict(arrowstyle=\"->\"))\n",
    "plt.annotate('(3, 2)', xy=(3, 2), xytext=(-18, 20), textcoords='offset points', size=14, arrowprops=dict(arrowstyle=\"->\"))\n",
    "\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.close(fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "根据条件可以得到方程组 \n",
    "$\n",
    "\\begin{cases}\n",
    "C+D&=1 \\\\\n",
    "C+2D&=2 \\\\\n",
    "C+3D&=2 \\\\\n",
    "\\end{cases}\n",
    "$，写作矩阵形式\n",
    "$\\begin{bmatrix}1&1 \\\\1&2 \\\\1&3\\\\\\end{bmatrix}\\begin{bmatrix}C\\\\D\\\\\\end{bmatrix}=\\begin{bmatrix}1\\\\2\\\\2\\\\\\end{bmatrix}$，也就是我们的$Ax=b$，很明显方程组无解。但是$A^TA\\hat x=A^Tb$有解，于是我们将原是两边同时乘以$A^T$后得到的新方程组是有解的，$A^TA\\hat x=A^Tb$也是最小二乘法的核心方程。\n",
    "\n",
    "下一讲将进行最小二乘法的验算。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}