{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Week 5: Introduction to neural Networks"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Solutions
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Perceptron learning rule"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This week, we will start working with neural networks. For each of the exercises below you can use the method of your choice but you should display the final boundary of your classifier."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Exercise 1. \n",
"As a first exercise, load the binary dataset below and code a few steps of the perceptron learning rule. "
]
},
{
"cell_type": "code",
"execution_count": 84,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"2\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"3\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"4\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"5\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"6\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"7\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"8\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"9\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"10\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"11\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"12\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"13\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"14\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"15\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"16\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"17\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"18\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"19\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"20\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"21\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"22\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"23\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"24\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"25\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"26\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"27\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"28\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"29\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"30\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"31\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"32\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"33\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"34\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"35\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"36\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"37\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"38\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"39\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"40\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"41\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"42\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"43\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"44\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"45\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"46\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"47\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"48\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"49\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"50\n",
"gradient\n",
"[ 22. 372.75345622 793.64963504]\n",
"norm of gradient\n",
"877.1025494880358\n",
"[372.75345622 793.64963504]\n",
"51\n",
"gradient\n",
"[ 21. 345.01728111 790.69343066]\n",
"norm of gradient\n",
"862.9449724900206\n",
"[345.01728111 790.69343066]\n",
"52\n",
"gradient\n",
"[ 20. 327.07373272 787.88321168]\n",
"norm of gradient\n",
"853.3095463429563\n",
"[327.07373272 787.88321168]\n",
"53\n",
"gradient\n",
"[ 20. 327.07373272 787.88321168]\n",
"norm of gradient\n",
"853.3095463429563\n",
"[327.07373272 787.88321168]\n",
"54\n",
"gradient\n",
"[ 19. 293.23156682 783.46715328]\n",
"norm of gradient\n",
"836.7595425544075\n",
"[293.23156682 783.46715328]\n",
"55\n",
"gradient\n",
"[ 19. 293.23156682 783.46715328]\n",
"norm of gradient\n",
"836.7595425544075\n",
"[293.23156682 783.46715328]\n",
"56\n",
"gradient\n",
"[ 19. 293.23156682 783.46715328]\n",
"norm of gradient\n",
"836.7595425544075\n",
"[293.23156682 783.46715328]\n",
"57\n",
"gradient\n",
"[ 18. 253.28341014 776.86131387]\n",
"norm of gradient\n",
"817.3065439826896\n",
"[253.28341014 776.86131387]\n",
"58\n",
"gradient\n",
"[ 18. 253.28341014 776.86131387]\n",
"norm of gradient\n",
"817.3065439826896\n",
"[253.28341014 776.86131387]\n",
"59\n",
"gradient\n",
"[ 18. 253.28341014 776.86131387]\n",
"norm of gradient\n",
"817.3065439826896\n",
"[253.28341014 776.86131387]\n",
"60\n",
"gradient\n",
"[ 17. 214.25691244 768.50364964]\n",
"norm of gradient\n",
"797.9930350771989\n",
"[214.25691244 768.50364964]\n",
"61\n",
"gradient\n",
"[ 16. 192.62672811 761.89781022]\n",
"norm of gradient\n",
"786.0339239492583\n",
"[192.62672811 761.89781022]\n",
"62\n",
"gradient\n",
"[ 16. 192.62672811 761.89781022]\n",
"norm of gradient\n",
"786.0339239492583\n",
"[192.62672811 761.89781022]\n",
"63\n",
"gradient\n",
"[ 14. 110.65668203 739.34306569]\n",
"norm of gradient\n",
"747.7092149133439\n",
"[110.65668203 739.34306569]\n",
"64\n",
"gradient\n",
"[ 12. 41.359447 717.08029197]\n",
"norm of gradient\n",
"718.372291356967\n",
"[ 41.359447 717.08029197]\n",
"65\n",
"gradient\n",
"[ 10. -15.72580645 696.71532847]\n",
"norm of gradient\n",
"696.9645255747577\n",
"[-15.72580645 696.71532847]\n",
"66\n",
"gradient\n",
"[ 10. -15.72580645 696.71532847]\n",
"norm of gradient\n",
"696.9645255747577\n",
"[-15.72580645 696.71532847]\n",
"67\n",
"gradient\n",
"[ 10. -15.72580645 696.71532847]\n",
"norm of gradient\n",
"696.9645255747577\n",
"[-15.72580645 696.71532847]\n",
"68\n",
"gradient\n",
"[ 10. -15.72580645 696.71532847]\n",
"norm of gradient\n",
"696.9645255747577\n",
"[-15.72580645 696.71532847]\n",
"69\n",
"gradient\n",
"[ 9. -28.25460829 688.94160584]\n",
"norm of gradient\n",
"689.5794799343245\n",
"[-28.25460829 688.94160584]\n",
"70\n",
"gradient\n",
"[ 9. -28.25460829 688.94160584]\n",
"norm of gradient\n",
"689.5794799343245\n",
"[-28.25460829 688.94160584]\n",
"71\n",
"gradient\n",
"[ 8. -71.42857143 671.09489051]\n",
"norm of gradient\n",
"674.9328802823502\n",
"[-71.42857143 671.09489051]\n",
"72\n",
"gradient\n",
"[ 6. -130.5875576 641.38686131]\n",
"norm of gradient\n",
"654.5733083978846\n",
"[-130.5875576 641.38686131]\n",
"73\n",
"gradient\n",
"[ 3. -228.54262673 592.66423358]\n",
"norm of gradient\n",
"635.209907032922\n",
"[-228.54262673 592.66423358]\n",
"74\n",
"gradient\n",
"[ 3. -228.54262673 592.66423358]\n",
"norm of gradient\n",
"635.209907032922\n",
"[-228.54262673 592.66423358]\n",
"75\n",
"gradient\n",
"[ 3. -228.54262673 592.66423358]\n",
"norm of gradient\n",
"635.209907032922\n",
"[-228.54262673 592.66423358]\n",
"76\n",
"gradient\n",
"[ 3. -228.54262673 592.66423358]\n",
"norm of gradient\n",
"635.209907032922\n",
"[-228.54262673 592.66423358]\n",
"77\n",
"gradient\n",
"[ 3. -228.54262673 592.66423358]\n",
"norm of gradient\n",
"635.209907032922\n",
"[-228.54262673 592.66423358]\n",
"78\n",
"gradient\n",
"[ 2. -257.89170507 574.81751825]\n",
"norm of gradient\n",
"630.0216748878181\n",
"[-257.89170507 574.81751825]\n",
"79\n",
"gradient\n",
"[ 2. -257.89170507 574.81751825]\n",
"norm of gradient\n",
"630.0216748878181\n",
"[-257.89170507 574.81751825]\n",
"80\n",
"gradient\n",
"[ 2. -257.89170507 574.81751825]\n",
"norm of gradient\n",
"630.0216748878181\n",
"[-257.89170507 574.81751825]\n",
"81\n",
"gradient\n",
"[ 1. -263.39285714 560.91240876]\n",
"norm of gradient\n",
"619.6769541413013\n",
"[-263.39285714 560.91240876]\n",
"82\n",
"gradient\n",
"[ 0. -287.32718894 541.60583942]\n",
"norm of gradient\n",
"613.1017850192484\n",
"[-287.32718894 541.60583942]\n",
"83\n",
"gradient\n",
"[ -2. -300.63364055 505.62043796]\n",
"norm of gradient\n",
"588.2487680490048\n",
"[-300.63364055 505.62043796]\n",
"84\n",
"gradient\n",
"[ -4. -359.10138249 459.41605839]\n",
"norm of gradient\n",
"583.1234154238397\n",
"[-359.10138249 459.41605839]\n",
"85\n",
"gradient\n",
"[ -6. -390.61059908 410.87591241]\n",
"norm of gradient\n",
"566.94942941154\n",
"[-390.61059908 410.87591241]\n",
"86\n",
"gradient\n",
"[ -4. -250.69124424 425.10948905]\n",
"norm of gradient\n",
"493.5384256769653\n",
"[-250.69124424 425.10948905]\n",
"87\n",
"gradient\n",
"[ -8. -215.43778802 191.97080292]\n",
"norm of gradient\n",
"288.6697588593456\n",
"[-215.43778802 191.97080292]\n",
"88\n",
"gradient\n",
"[ -4. -34.33179724 150.3649635 ]\n",
"norm of gradient\n",
"154.28640429681616\n",
"[-34.33179724 150.3649635 ]\n",
"89\n",
"gradient\n",
"[ -13. -316.33064516 -163.39416058]\n",
"norm of gradient\n",
"356.2747939176852\n",
"[-316.33064516 -163.39416058]\n",
"90\n",
"gradient\n",
"[ 1. 98.58870968 219.30656934]\n",
"norm of gradient\n",
"240.44979732343998\n",
"[ 98.58870968 219.30656934]\n",
"91\n",
"gradient\n",
"[ -11. -255.78917051 -144.19708029]\n",
"norm of gradient\n",
"293.8399185157514\n",
"[-255.78917051 -144.19708029]\n",
"92\n",
"gradient\n",
"[ 3. 141.96428571 238.21167883]\n",
"norm of gradient\n",
"277.32230770419983\n",
"[141.96428571 238.21167883]\n",
"93\n",
"gradient\n",
"[ -10. -228.05299539 -134.52554745]\n",
"norm of gradient\n",
"264.96281177285\n",
"[-228.05299539 -134.52554745]\n",
"94\n",
"gradient\n",
"[ 3. 141.96428571 238.21167883]\n",
"norm of gradient\n",
"277.32230770419983\n",
"[141.96428571 238.21167883]\n",
"95\n",
"gradient\n",
"[ -9. -198.70391705 -116.67883212]\n",
"norm of gradient\n",
"230.60398200253536\n",
"[-198.70391705 -116.67883212]\n",
"96\n",
"gradient\n",
"[ 4. 159.9078341 241.02189781]\n",
"norm of gradient\n",
"289.27162085319554\n",
"[159.9078341 241.02189781]\n",
"97\n",
"gradient\n",
"[ -8. -168.20276498 -93.43065693]\n",
"norm of gradient\n",
"192.57584947513544\n",
"[-168.20276498 -93.43065693]\n",
"98\n",
"gradient\n",
"[ 3. 132.40207373 197.48175182]\n",
"norm of gradient\n",
"237.77794563946242\n",
"[132.40207373 197.48175182]\n",
"99\n",
"gradient\n",
"[ -6. -121.02534562 -72.77372263]\n",
"norm of gradient\n",
"141.34761755350667\n",
"[-121.02534562 -72.77372263]\n",
"100\n",
"gradient\n",
"[ 2. 102.24654378 151.16788321]\n",
"norm of gradient\n",
"182.51050553162744\n",
"[102.24654378 151.16788321]\n",
"101\n",
"gradient\n",
"[ -6. -121.02534562 -72.77372263]\n",
"norm of gradient\n",
"141.34761755350667\n",
"[-121.02534562 -72.77372263]\n",
"102\n",
"gradient\n",
"[ 1. 74.04953917 112.15328467]\n",
"norm of gradient\n",
"134.3975204904403\n",
"[ 74.04953917 112.15328467]\n",
"103\n",
"gradient\n",
"[ -5. -103.08179724 -69.96350365]\n",
"norm of gradient\n",
"124.68259206535166\n",
"[-103.08179724 -69.96350365]\n",
"104\n",
"gradient\n",
"[ 2. 90.9562212 119.7810219]\n",
"norm of gradient\n",
"150.41451851976154\n",
"[ 90.9562212 119.7810219]\n",
"105\n",
"gradient\n",
"[ -5. -103.08179724 -69.96350365]\n",
"norm of gradient\n",
"124.68259206535166\n",
"[-103.08179724 -69.96350365]\n",
"106\n",
"gradient\n",
"[ 2. 90.9562212 119.7810219]\n",
"norm of gradient\n",
"150.41451851976154\n",
"[ 90.9562212 119.7810219]\n",
"107\n",
"gradient\n",
"[ -5. -103.08179724 -69.96350365]\n",
"norm of gradient\n",
"124.68259206535166\n",
"[-103.08179724 -69.96350365]\n",
"108\n",
"gradient\n",
"[ 2. 90.9562212 119.7810219]\n",
"norm of gradient\n",
"150.41451851976154\n",
"[ 90.9562212 119.7810219]\n",
"109\n",
"gradient\n",
"[ -5. -103.08179724 -69.96350365]\n",
"norm of gradient\n",
"124.68259206535166\n",
"[-103.08179724 -69.96350365]\n",
"110\n",
"gradient\n",
"[ 2. 90.9562212 119.7810219]\n",
"norm of gradient\n",
"150.41451851976154\n",
"[ 90.9562212 119.7810219]\n",
"111\n",
"gradient\n",
"[ -4. -81.33640553 -57.66423358]\n",
"norm of gradient\n",
"99.78363943309886\n",
"[-81.33640553 -57.66423358]\n",
"112\n",
"gradient\n",
"[ 1. 58.03571429 75.51094891]\n",
"norm of gradient\n",
"95.2420471073733\n",
"[58.03571429 75.51094891]\n",
"113\n",
"gradient\n",
"[ -4. -81.33640553 -57.66423358]\n",
"norm of gradient\n",
"99.78363943309886\n",
"[-81.33640553 -57.66423358]\n",
"114\n",
"gradient\n",
"[ 1. 58.03571429 75.51094891]\n",
"norm of gradient\n",
"95.2420471073733\n",
"[58.03571429 75.51094891]\n",
"115\n",
"gradient\n",
"[ -4. -81.33640553 -57.66423358]\n",
"norm of gradient\n",
"99.78363943309886\n",
"[-81.33640553 -57.66423358]\n",
"116\n",
"gradient\n",
"[ 1. 58.03571429 75.51094891]\n",
"norm of gradient\n",
"95.2420471073733\n",
"[58.03571429 75.51094891]\n",
"117\n",
"gradient\n",
"[ -3. -53.36981567 -34.7080292 ]\n",
"norm of gradient\n",
"63.73369999618191\n",
"[-53.36981567 -34.7080292 ]\n",
"118\n",
"gradient\n",
"[ 1. 58.03571429 75.51094891]\n",
"norm of gradient\n",
"95.2420471073733\n",
"[58.03571429 75.51094891]\n",
"119\n",
"gradient\n",
"[ -3. -53.36981567 -34.7080292 ]\n",
"norm of gradient\n",
"63.73369999618191\n",
"[-53.36981567 -34.7080292 ]\n",
"120\n",
"gradient\n",
"[ 1. 58.03571429 75.51094891]\n",
"norm of gradient\n",
"95.2420471073733\n",
"[58.03571429 75.51094891]\n",
"121\n",
"gradient\n",
"[ -2. -36.46313364 -27.08029197]\n",
"norm of gradient\n",
"45.4631975130738\n",
"[-36.46313364 -27.08029197]\n",
"122\n",
"gradient\n",
"[ 1. 58.03571429 75.51094891]\n",
"norm of gradient\n",
"95.2420471073733\n",
"[58.03571429 75.51094891]\n",
"123\n",
"gradient\n",
"[ -2. -36.46313364 -27.08029197]\n",
"norm of gradient\n",
"45.4631975130738\n",
"[-36.46313364 -27.08029197]\n",
"124\n",
"gradient\n",
"[ 1. 58.03571429 75.51094891]\n",
"norm of gradient\n",
"95.2420471073733\n",
"[58.03571429 75.51094891]\n",
"125\n",
"gradient\n",
"[ -2. -36.46313364 -27.08029197]\n",
"norm of gradient\n",
"45.4631975130738\n",
"[-36.46313364 -27.08029197]\n",
"126\n",
"gradient\n",
"[ 1. 58.03571429 75.51094891]\n",
"norm of gradient\n",
"95.2420471073733\n",
"[58.03571429 75.51094891]\n",
"127\n",
"gradient\n",
"[ -2. -36.46313364 -27.08029197]\n",
"norm of gradient\n",
"45.4631975130738\n",
"[-36.46313364 -27.08029197]\n",
"128\n",
"gradient\n",
"[ 1. 58.03571429 75.51094891]\n",
"norm of gradient\n",
"95.2420471073733\n",
"[58.03571429 75.51094891]\n",
"129\n",
"gradient\n",
"[ -2. -36.46313364 -27.08029197]\n",
"norm of gradient\n",
"45.4631975130738\n",
"[-36.46313364 -27.08029197]\n",
"130\n",
"gradient\n",
"[ 1. 58.03571429 75.51094891]\n",
"norm of gradient\n",
"95.2420471073733\n",
"[58.03571429 75.51094891]\n",
"131\n",
"gradient\n",
"[ -2. -36.46313364 -27.08029197]\n",
"norm of gradient\n",
"45.4631975130738\n",
"[-36.46313364 -27.08029197]\n",
"132\n",
"gradient\n",
"[ 1. 58.03571429 75.51094891]\n",
"norm of gradient\n",
"95.2420471073733\n",
"[58.03571429 75.51094891]\n",
"133\n",
"gradient\n",
"[ -2. -36.46313364 -27.08029197]\n",
"norm of gradient\n",
"45.4631975130738\n",
"[-36.46313364 -27.08029197]\n",
"134\n",
"gradient\n",
"[ 1. 58.03571429 75.51094891]\n",
"norm of gradient\n",
"95.2420471073733\n",
"[58.03571429 75.51094891]\n",
"135\n",
"gradient\n",
"[ -2. -36.46313364 -27.08029197]\n",
"norm of gradient\n",
"45.4631975130738\n",
"[-36.46313364 -27.08029197]\n",
"136\n",
"gradient\n",
"[ 0. 24.07834101 35.18248175]\n",
"norm of gradient\n",
"42.633009842087475\n",
"[24.07834101 35.18248175]\n",
"137\n",
"gradient\n",
"[ -2. -36.46313364 -27.08029197]\n",
"norm of gradient\n",
"45.4631975130738\n",
"[-36.46313364 -27.08029197]\n",
"138\n",
"gradient\n",
"[ 0. 24.07834101 35.18248175]\n",
"norm of gradient\n",
"42.633009842087475\n",
"[24.07834101 35.18248175]\n",
"139\n",
"gradient\n",
"[ -2. -36.46313364 -27.08029197]\n",
"norm of gradient\n",
"45.4631975130738\n",
"[-36.46313364 -27.08029197]\n",
"140\n",
"gradient\n",
"[ 0. 24.07834101 35.18248175]\n",
"norm of gradient\n",
"42.633009842087475\n",
"[24.07834101 35.18248175]\n",
"141\n",
"gradient\n",
"[ -1. -12.52880184 -7.77372263]\n",
"norm of gradient\n",
"14.778418018249504\n",
"[-12.52880184 -7.77372263]\n",
"142\n",
"gradient\n",
"[ 0. 24.07834101 35.18248175]\n",
"norm of gradient\n",
"42.633009842087475\n",
"[24.07834101 35.18248175]\n",
"143\n",
"gradient\n",
"[ -1. -12.52880184 -7.77372263]\n",
"norm of gradient\n",
"14.778418018249504\n",
"[-12.52880184 -7.77372263]\n",
"144\n",
"gradient\n",
"[ 0. 24.07834101 35.18248175]\n",
"norm of gradient\n",
"42.633009842087475\n",
"[24.07834101 35.18248175]\n",
"145\n",
"gradient\n",
"[ -1. -12.52880184 -7.77372263]\n",
"norm of gradient\n",
"14.778418018249504\n",
"[-12.52880184 -7.77372263]\n",
"146\n",
"gradient\n",
"[ -1. -12.52880184 -7.77372263]\n",
"norm of gradient\n",
"14.778418018249504\n",
"[-12.52880184 -7.77372263]\n",
"147\n",
"gradient\n",
"[ 3. 103.48502304 127.55474453]\n",
"norm of gradient\n",
"164.28135269978821\n",
"[103.48502304 127.55474453]\n",
"148\n",
"gradient\n",
"[ -1. -12.52880184 -7.77372263]\n",
"norm of gradient\n",
"14.778418018249504\n",
"[-12.52880184 -7.77372263]\n",
"149\n",
"gradient\n",
"[ 2. 70.56451613 83.28467153]\n",
"norm of gradient\n",
"109.17732112878855\n",
"[70.56451613 83.28467153]\n",
"150\n",
"gradient\n",
"[ -1. -12.52880184 -7.77372263]\n",
"norm of gradient\n",
"14.778418018249504\n",
"[-12.52880184 -7.77372263]\n",
"151\n",
"gradient\n",
"[ 2. 70.56451613 83.28467153]\n",
"norm of gradient\n",
"109.17732112878855\n",
"[70.56451613 83.28467153]\n",
"152\n",
"gradient\n",
"[ -1. -12.52880184 -7.77372263]\n",
"norm of gradient\n",
"14.778418018249504\n",
"[-12.52880184 -7.77372263]\n",
"153\n",
"gradient\n",
"[ 2. 70.56451613 83.28467153]\n",
"norm of gradient\n",
"109.17732112878855\n",
"[70.56451613 83.28467153]\n",
"154\n",
"gradient\n",
"[ -1. -12.52880184 -7.77372263]\n",
"norm of gradient\n",
"14.778418018249504\n",
"[-12.52880184 -7.77372263]\n",
"155\n",
"gradient\n",
"[ 2. 70.56451613 83.28467153]\n",
"norm of gradient\n",
"109.17732112878855\n",
"[70.56451613 83.28467153]\n",
"156\n",
"gradient\n",
"[ -1. -12.52880184 -7.77372263]\n",
"norm of gradient\n",
"14.778418018249504\n",
"[-12.52880184 -7.77372263]\n",
"157\n",
"gradient\n",
"[ 2. 70.56451613 83.28467153]\n",
"norm of gradient\n",
"109.17732112878855\n",
"[70.56451613 83.28467153]\n",
"158\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"159\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"160\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"161\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"162\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"163\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"164\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"165\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"166\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"167\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"168\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"169\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"170\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"171\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"172\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"173\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"174\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"175\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"176\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"177\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"178\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"179\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"180\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"181\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"182\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"183\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"184\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"185\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"186\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"187\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"188\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"189\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"190\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"191\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"192\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"193\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"194\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"195\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"196\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"197\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"198\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"199\n",
"gradient\n",
"[0. 0. 0.]\n",
"norm of gradient\n",
"0.0\n",
"[0. 0.]\n",
"200\n"
]
}
],
"source": [
"# \n",
"\n",
"\n",
"import scipy.io as sio\n",
"data1 = sio.loadmat('perceptron_data_class1.mat')\n",
"data2 = sio.loadmat('perceptron_data_class2.mat')\n",
"\n",
"from numpy import linalg as LA\n",
"\n",
"data1 = data1['perceptron_data_class1']\n",
"data2 = data2['perceptron_data_class2']\n",
"\n",
"# We first build the matrix of features (here [1, x, y])\n",
"\n",
"sz1 = np.shape(data1)\n",
"sz2 = np.shape(data2)\n",
"\n",
"targetsClass1 = np.ones((sz1[0],))\n",
"targetsClass2 = -1 * np.ones((sz2[0],))\n",
"\n",
"total_targets = np.hstack((targetsClass1, targetsClass2))\n",
"total_data = np.vstack((data1, data2))\n",
"\n",
"# precomputing the product y(x_i)t_i\n",
"product_yiti = np.multiply(np.vstack((data1, data2)),(np.ones((2,1))* total_targets).T)\n",
"\n",
"\n",
"total_Xtilde = np.hstack((np.ones((np.shape(total_data)[0],1)), total_data))\n",
"\n",
"# Then we initialize the beta_tilde (here I chose to take beta_tilde random Gaussian but any other choice is also possible)\n",
"\n",
"sigma = 1\n",
"beta = np.random.normal(0, 1, (2,))\n",
"beta0 = np.random.normal(0, 1, 1)\n",
"\n",
"beta_tilde_int = np.hstack((beta0, beta))\n",
"betaTotal = beta_tilde_int\n",
"\n",
"# Initialization of the max number of iter and learning rate\n",
"eta = .01\n",
"iter_num = 1\n",
"max_iter = 100\n",
"\n",
"\n",
"while iter_num < 200:\n",
" \n",
" \n",
" # We start by looking for the misclassified points (in the case of the perceptron, \n",
" # the misclassified points are the points for which the sign of the product y(x_i)t_i is negative)\n",
" \n",
" sign = np.sign(np.multiply(np.matmul(betaTotal,total_Xtilde.T),total_targets))\n",
" ind_misclassified = np.where(sign < 0)\n",
" misclassified_targets = total_targets[sign < 0]\n",
" # we then extract the misclassified products $x_it_i$\n",
" misclassified_yiti = product_yiti[sign < 0,:]\n",
" # now summing each of the misclassified vectors to get the gradient, we get \n",
" gradient_beta0 = np.sum(misclassified_targets, axis=0)\n",
" gradient_beta = np.sum(misclassified_yiti, axis=0)\n",
" gradient_betaTotal = np.hstack((gradient_beta0, gradient_beta))\n",
" print('gradient')\n",
" print(gradient_betaTotal)\n",
" print('norm of gradient')\n",
" print(LA.norm(gradient_betaTotal))\n",
" if LA.norm(gradient_betaTotal)!=0:\n",
" gradient_betaTotal = np.true_divide(gradient_betaTotal,LA.norm(gradient_betaTotal))\n",
" \n",
" print(gradient_beta)\n",
" \n",
" # updating Beta with learning rate eta\n",
" betaTotal += gradient_betaTotal * eta\n",
"\n",
" \n",
" iter_num +=1\n",
" print(iter_num)\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 86,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi40LCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcv7US4rQAAHEtJREFUeJzt3W+MXNd53/HvY60qsco2skZLQdy1\nQS9MdtcoawkhtxZtEhsRSZREFtegITQIKRogwDcqYKM2JDl5YbRoAEsv5Bio0JSwDamiE1mxQkvI\ni7aCqgWbivBQihzTCdmJsqFSLmVxtZGcjaPQof30xb3DHa52ufPn/jnn3t8HIHZndkZzdOfOc8+c\n85znmLsjIiLV8r6yGyAiItlTcBcRqSAFdxGRClJwFxGpIAV3EZEKUnAXEamgoW4eZGZngSXgp8Al\nd99uZjcB3wI2A2eBe9397XyaKSIiveil5/6L7n6bu29Pbz8EvODuW4AX0tsiIhKAQYZl9gJPpL8/\nAcwM3hwREcmCdbNC1cz+GngbcOC/uvsRM3vH3W9M/27A2+3bK557GDgMcP31N/zC2NhElu0XEbm6\nH/+Yf1z6J4Yu/phL193A9cPX8vM3XCq7VT155bXX3nL3kV6e09WYO/AJd583s43A82Z2pvOP7u5m\ntupVwt2PAEcAtmzZ7o8++nIv7RMR6VvraJPFl84wcd2rLNy2m63To3xy6s2ym9Uzu+ee13t9TlfB\n3d3n058XzOwYMAW8aWa3uvsbZnYrcKHXFxcRyUvraJPFU/PsGD7D6M4x7ts/BMQX2Pu17pi7md1g\nZsPt34FfBn4APAccTB92EHg2r0aKiPSjsW2UucYOpvZvLbspheum534LcCwZVmcI+H13/+9mdhJ4\n2swOAa8D9+bXTBGRHp2fL7sFpVo3uLv7HPDRVe5fBPbk0ah+zc7Ck0/CW2/BzTfDgQMwPV12q0Sk\nSO3hmImlkwwPw9C+Geo0HNPW7YRq8GZn4bHH4OLF5PbCQnIbFOBFaqHZpDU7z+LZJQAmdjbS4Zj6\nBXaoUHB/8snlwN528WJyv4K7SLVd7q1fOM74xg0M7Zthaur9ZTerVJUJ7m+91dv9IlIBK3vre8Zq\n3VvvVJngfvPNyVDMaveLSPV09taZTHLY695b71SZ4H7gwJVj7gDXXZfcLyIVkvbWz51eYvfGM4zu\nqV8OezcqE9zb4+rKlhGptlYLxhdPwuQOhqZnmIpwxWkRKhPcIQnkCuYi1TfX2MGuTXNMTVUqhGVK\nRyYHyreXfui8WV97nB2ATeW2JXQK7hlTvr30Q+fN+lqPHLucFdPYPFzLkgK9UHDPmPLtpR86b9bW\nueJ0PF1xGmNlx6IpuGdM+fbSD503q9CK04EouGdM+fbSD503V1IO++AG2WZPVnHgQJJf3ym2fPvZ\nWTh0CPbuTX7OzpbdouqrwnmTtca2URYmd/P5B4Y0DNMH9dwzFnu+vSb2yhH7eSPhUXDPQcz59prY\nK0/M502W2lvjQZIVI/1RcJcraGJPStNscuKZ5ayY0W0NpvZrnL1fCu5yBU3sxS/GxVCdOewLm3dw\n3wMKTYPSEZQrqABb3GKbM1EOe34U3OUKmtiLW0xzJqv31hXYs6LgLu+hib3ehDQMEsOciXrrxVBw\nFxlAaMMgoc+ZdGbCaMVpviq1iCmrxTdaxCPdutowSBliWAzV2DzMzM4FFf7KWWV67ln1oELriVVN\nSEMYWQhtGKSoOZOqvY9VVJngntVEUkwTUrGp4oUzxGGQvOdM+nof0xz2dy8kW+OxqZFfAwWo0LBM\nVj2o0HpiVRLaEEYWYhgGyVqv72PraJMTj5+BpSXGJocZ2jejIZkCVKbnnlUPKsSeWFVU8cJZx9TR\nrt/HNVecagK1CJUJ7lktvtEinvxU9cJZt9TRbt5HrTgtX2WOeFY9qDr2xIqiC2c1XO19VA57OCoT\n3CG7HlTdemJF0YWzGlZ9Hz/+Gpuap7TiNCCVCu4SPl04q2Hl+9h65BTji+qth0TBXUQyMbotSW+c\nUmAPQmVSIUWkJM0mAPPnS26HXEE9dxHpW2etGDaPM7R1FI2zh0HBXaRGMisbsGLFaa1y2COpvaDg\nLlITWZV/6MxhH5sc5lMP3JFpO4MWUQ2NroO7mV0DvAzMu/vdZvYh4CmgAbwCHHD3n+TTTJFlkXSc\ngjNw3STtcRpV8aleeu6fBU4D/yK9/TDwFXd/ysx+DzgE/JeM2ydyhYg6TsEZtPxDqwWNbaMsnKe+\nK04jqqHRVbaMmY0Bvw58Lb1twJ3At9OHPAHM5NFAkU5VLD5WlLXKPMRe/qFQER3EblMhfxd4APhZ\nersBvOPul9Lb54DR1Z5oZofN7GUze/lHP1qlIIVIDyLqOAVnkAqW7bIClzNj6iqiMqDrfrcys7uB\nC+7+iplN9/oC7n4EOAKwZct277mFIh1iKT4W4rxAX+Ufmk1as/OcO51mxeys4Th7p4hqaHQzcPZx\n4B4z+zXgepIx968CN5rZUNp7HwPm82umSCKG4mMhzwv0Uv6hncM+8e6rMLmboekZrT6FaGporBvc\n3f2LwBcB0p77F9z9N83sD4FPk2TMHASezbGdIkAcHaeIEipW957e+hj37VcRsNgMMuX9IPCUmf0n\n4FXg69k0SeTqQu84xTwv0LnidGxyWL31iPUU3N19FphNf58DprJvkkjcYpkXuIJ2TaqcmiariuQn\nhnmBTp29de2aVB16F0UyFsO8wEqNzcPs2jRX70yYilFwF8lB6PMCUn0K7oEIMS9aKi7NimkXAWNT\nuc2RbCm4ByDkvGippssbWV84DpO72To9ytSUhmSqRME9ANHnRUs8Vuaw71EOe1UpuAcg5rxoiYdW\nnNaLgnsAosyLlnis3DVJK05rQRtkByCiQnMSmdbRJicePwNLS8mK030zTO3fWnazpADquQcgxrxo\nCV/rkWOML2rFaV0puAdCedGSh9FtDQD11mtIwzIiVdRslt0CKZl67iIV05nDPr+4gaF9M2g4pn4U\n3EWqoiOHfcMGmNgzlg7HKLDXkYK7SAVoxamspOAuuVPdnBxpxamsQcFdcqW6OflqtZKf2jVJVlJw\nl1ypbk4xdm2aY2pKH+eeVPwrpc4GyZXq5uSnPc4OqFxvr2rwlVJ57pKrterjqG7OYFqPHEu2xlta\notHQIqWeXe0rZUWo5y65im0/0dBdzopJN7Ie2jfDJzXO3rsafKVUcJdcqW5ORlbsmjSxs6Ec9kHU\noBSrgrvkTnVzBhNlDnvok5U1+Eqp4C4SqlVWnEaRwx7DZGUNvlIquIsEKMreelss+a8V/0qp4C4S\nqEYDJrZF0lvvVIPJyhgoFVJEsqX81yAouIsEpp3DPr54suym9Kcu+0bOzsKhQ7B3b/JzdrbsFl1B\nwzIigVgthz3KWjE1mKyMYdJYwV0kAK1Hjl3OYV/YvIP7HohsnH2lik9WxjBprOAuUqLOrJjxjRu0\n4jQWEUwaK7iLlGHlilPtmhSXCFa4akJVpGCto01OPDPPyOnjNDYPc8dnJlT4KzYRTBqr5y5SoM6x\n9WhWnMp7RTBpvG5wN7PrgePAdenjv+3uXzKzDwFPAQ3gFeCAu/8kz8aKVMHMzuTrvHrrkQt80rib\nYZmLwJ3u/lHgNuAuM/sY8DDwFXf/MPA2cCi/ZoqISC/WDe6e+Pv05rXpPwfuBL6d3v8EMJNLC0Wq\noNnkxIPHGDl7kvlTi2W3RmqgqzF3M7uGZOjlw8BjwF8B77j7pfQh54DRNZ57GDgMMDLywUHbKxKd\n1XPYRfLV1Vnm7j8FbjOzG4FjwES3L+DuR4AjAFu2bPd+GilXF3rp7LrSrklSpp66EO7+jpm9CNwB\n3GhmQ2nvfQyYz6OBMSkjyEawCrp+tGuSBGDdMXczG0l77JjZBuCXgNPAi8Cn04cdBJ7Nq5ExaAfZ\nhQVwXw6yedcSqsE+v9FptWB88SQ7hs8oh11K0022zK3Ai2b2feAk8Ly7/zHwIPDvzew1knTIr+fX\nzPCVFWQjWAVdS3ONHYxua2gYRkqz7rCMu38fuH2V++eAqTwaFaOygmwEq6DrpdmE8/MsLgKbym5M\n5DSZNBCVH8hIWfsTRLAKujZaR5ucePwMi2eXaDTgza27ym5SvMoa56wQ5WRlpKzN1CNYBV19zaRW\nTDsrZnRbg6n970cTqAOIoKRu6BTcM1JmkA18FXSlKYc9J5pMGpjOxAwpyNbImr11yYQmkwam4C7S\nI/XWC1DWOGeF6KwU6ZJWnBZIk0kDU3AX6db5eRoNmNimFaeF0DjnQJQKKdKDXZvmym6CSFcU3EW6\n0DraZPHsksr1SjQ0LCNyNWkRsHOnl9i98UyaFaNaMRI+BXeRNbSONll86QwAY5PDDE3PMKUJVImE\ngrvISlpxKhWg4C7SQTnsUhU6c0V4bw67VpzKqiKqVKngLrWn3rp0JbJtz3QWS701m0nuelp7Xb11\nWVNklSoV3CMV0bdDkWqIrFKlgnuEYvh2GMXFJ81hf/vsIsPpOLvImiKrVKngHqHQvx3GcPFp57BP\nvPsqC5O7ef/0aL1z2EO8GofWpsgqVSq4Ryj0b4dBX3zSHPZ3L6QrTneOcd/+IWqdwx7i1TjENkVW\nqVLBPUKhfzsM9eKjFadrCPFqHGKbIKpKlQruEQr922GIF5/W0Sbjp74Tx4rToocjQrwah9imyKgq\nZISmp+H++2FkBMySn/ffH06H4sCB5GLTKYSLz1xjR/6Fv2Zn4dAh2Ls3+Tk72/vzH3ssuTq6Lw9H\n9Prf6cVaV90yr8Yhtiky6rlHKuRvh8ENTTabcH4+/9fJYpy4jOGIEL8KhtimyCi4Sy5CufhcLitw\n4TgLk7t5c+suchuOySIwlzEcEdzVONA2RUbBXaqpow77hg0wsaeArJgsAnNZExahXI07hdimiGjM\nXSqndTRJdxw5fZyxyWHu+MxEMRtsZDFOHOqEhURHPXepjpW7JhXRW++UxTixhiMkIwruUglB5LBn\nFZg1HCEZUHCXuK1ccVp2DrsCswRCwV2i1pqdp9EAGlpxKtJJwV2it2vTHABTUzqdRdqULSPRah1t\nsrgI86cWy26KSHDU1ZH4pFkx7a3xRnfmXFJAJEIK7hKVzhWnTO5m6/QoU1PaGk9kpXWDu5l9APhv\nwC2AA0fc/atmdhPwLWAzcBa4193fzq+pUmtlrDgViVg3PfdLwOfd/U/NbBh4xcyeBz4DvODuXzaz\nh4CHgAfza6rUlXrrIr1bN7i7+xvAG+nvS2Z2GhgF9gLT6cOeAGZRcJcslb3iVCRiPY25m9lm4Hbg\nu8AtaeAH+CHJsM1qzzkMHAYYGflgv+2Umsl0xWloe3GKFKDr4G5mPwc8A3zO3f/OzC7/zd3dzHy1\n57n7EeAIwJYt21d9jMgV0vrrMzuT6ogDrTgNcS9OkQJ0leduZteSBPZvuvsfpXe/aWa3pn+/FbiQ\nTxOljtoLkwZ2tRrrIhW2bnC3pIv+deC0uz/a8afngIPp7weBZ7NvntROOs4+f2oxm8VJ2otTaqqb\nYZmPAweAU2b2vfS+3wK+DDxtZoeA14F782mi1MUVWTEbNzC0L4NaMSHu1i3vpXmRzHWTLfMngK3x\n5z3ZNkdqacWK04k9Y+mK0wyyYrQXZ/g0L5KLSqxQjeGiH0Mby5B7DnuIm1/oZLhSEZuC1/CYRx/c\nY7jox9DGMrQeOcbi2SV2DOecwx5SjXWdDO+V97xITY959FUhY0iGiKGNZZnZuZBusFGTwl9VPhlm\nZ+HQIdi7N/k5O9vd87LYe/ZqqnzMryL64B5DMkQMbZSCVPVkaPeOFxbAfbl33E2Az3tT8Koe83VE\nH9zzvuhnIYY2FqrZ5MSDxxg5e7J+tdirejIM0juenob774eRETBLft5/f3ZDJlU95uuIfsw9hmSI\nGNpYlPY4O8DC5h3c90D0p2BvqnoyDNo7znNepKrHfB3Rf7JCTIZYKYY25u1yVszSScaHYWjfDJ+s\n436nVT0ZQl5PUNVjvg5zL67cy5Yt2/3RR18u7PUkACty2Gd2LtRn8rQMZaX8rcxIgaR3nOXwSo3Z\nPfe84u7be3lO9D13CVdnDvv45RWnqsOemzJT/mraOw6ZgrtkL88Vp7K2IhYDXU1I6wlEwV2y15qd\nZ3zxJGzeoV2TilTTlD9ZnYK7ZKvZTMr1bmowyhxTUzrFChPypKYULvo8dwnP/PnknxQs78VAEhV1\nqyQzyznsIzQ2D/Op/Tq9CqVJTemgT58MrtnkxDPLOexJrRiNs5dCk5qSUnCXgdR+xalIoPRJlL5o\nxalI2Gob3GtYuz8bK3PYdzaqlcOuE0MqopbBvaa1+wdW+RWnOjGkQmqZClnT2v2DaTbh/DyQrDj9\n1MN3VG8YRieGVEgte+5ayNefXZvmmAfe3LqLygzDdNKJIRVSy557TWv3D6TVItlYY7HCm2voxJAK\nqWVw10K+HrR3TXrpO0CSw1654Zg2nRjh6Hc/1hgU9P9Wy2EZLeTrTu1y2HVihKHbie0YM5sKnLSv\n+Kd1bVrIt7Ygc9iL+iDrxChfN6WLY81sKrAsc22Du6xu9d56AIE9xg+y9Kebie2ya9f3q8BJewV3\neY+ZnUnZ2GDqw8T6QZb+dFO6ONbMpgLLMtdyQlUiE+sHWfrTzcR2rJlNBU7aK7hLotmk9cgxzp1e\nSlIeQxLrB1n6Mz2dbKw9MgJmyc+VG23HmtnUzf9bRjQsI8kE6ktnABibHGZoeoapsidQOx04cOWY\nO8TxQZb+rTexHXNmU0GT9grudZbWYX/3whK7N57pqMMeUGCHuD/Ikh9lNl2VgntNBd9bX0kfZJGe\nKLjXzZq7JgUc2EWkZwruNVK7FaciNbbup9vMvgHcDVxw93+V3ncT8C1gM3AWuNfd386vmTKQdION\n8cWAVpyKSK66SYV8HLhrxX0PAS+4+xbghfS2BGzXpjlGtzWqXfhLVlflIlyypnWDu7sfB/52xd17\ngSfS358AZjJul+Rg/nzZLai4EINou3TDwgK4L5duCKFtkqt+FzHd4u5vpL//ELhlrQea2WEze9nM\nXv7Rj1ZZdiu5ah1tcuLxM5x54RxzjKcbbUjmQg2i2l2qtgaeUXN3NzO/yt+PAEcAtmzZvubjJGPp\nOPu502kO+84x7tsfQBGwqgq1/o1KN9RWv8H9TTO71d3fMLNbgQtZNkoGE10OexWEGkQLLFQlYek3\nuD8HHAS+nP58NrMWSf9iWXFaRaEGUZVuqK1uUiH/AJgGbjazc8CXSIL602Z2CHgduDfPRsr6OnPY\n1VsvQahBNNTSDTHuohSZdYO7u//GGn/ak3FbpB9acRqGUIMohFe6QZuvFEJLFGOWTpqCVpwGIbQg\nGqpQJ58rRtEgQptmv8nkk7/NhoW/4Sf//Of5/tiv8sPpz6HeukQh1MnnilFwj8ym2W/y0ccOM3Tx\nHwC47h/e4fa//EP+7AcTMHV7fi+sMVLJSqiTzxWjnZgiM/nkb18O7G1Dfolf+D//Ob8XDXWBjsQp\n1l2UIqPgHpkNC3+z+h/y/EqrVY6SpQK3mqszDcvEIp08fft9N3HTz1bZ4zTPr7QaI5WsafI5dwru\nEehccXriwwf4lb/+PYb+6R+XH5D3V9oQxkg15l8Pep8zo+Aeg/PzzOxMguvU/jth9n3FfgDKXqCj\nvOh60PucKQX3GBX9lbbsBTrKi64Hvc+ZUnAPWTrOvnh2ifnFRUa3NcprS5ljpBrzrwe9z5lScA9U\n62iTxVPzTFw4zvjGDQztq3GtmBDG/CV/ep8zpVTI0DSbSRGwl87A0hITe8b41MN31HtrPOVF14Pe\n50yp5x6Qzt46k7vZOj3K1NT7y25W+coe85di6H3OlIJ7CFbumrRHuya9h/Ki60Hvc2YU3EvWzmGf\nePdVmNytOuwikgkF97J0ZMLsGNYepyKSLQX3Eu3aNMc8qLcuxdDqz1pRcBepA63+rB0F9xJczopZ\nWmR4uOzWSC1o9WftKLgXqWOcHWBiZ4Op/VvROLvkTqs/a0fBvSCrrzhVDrsURKs/a0fBPW8dOewb\nNsDEnjH11qV4ZVf2lMIpuOdIK05rJuRsFK3+rB0F9zzUbcVpyEGtKDFko2j1Z62ocFjW0sA+cvYk\nY5PDydj6/q1ltyo/2jw7oX1mJTAK7jnYtWmO4c0Ntk6PVr+ao4JaQtkoEhgF94y1WjB/vuxWFEhB\nLbFW1omyUaQkCu4ZatdhP3l2hDnGq99rBwW1NtUil8BoQjUDyytOTzI+DEP7ZuoR2EEpdm3KRpHA\nKLgPQitOFdQ6KRtFAqLg3iflsHdQUBMJjoJ7P5pNOD9PowET2yqewy4iUVJw79OuTXNlN0FEZE0K\n7n1oteDSqUUARrc1Sm6NiMh7Kbj3otnkxDNJVgzDSWCv9OpTEYnWQMHdzO4CvgpcA3zN3b+cSasC\n1Hrk2OWsmIXNO7jvAV0XRSRcfUcoM7sGeAz4JeAccNLMnnP3v8iqcSGodQ67iERrkO7nFPCau88B\nmNlTwF6gMsG9HdhZWurorSuwi0j4Bgnuo8D/67h9Dvg3Kx9kZoeBw+nNi/fcYz8Y4DXLcw6+8CeZ\n/hdvBmpWgGVNOhbLdCyW6Vgs+5e9PiH3gWN3PwIcATCzl919e96vGQMdi2U6Fst0LJbpWCwzs5d7\nfc4ghcPmgQ903B5L7xMRkZINEtxPAlvM7ENm9s+Afws8l02zRERkEH0Py7j7JTP7d8D/IEmF/Ia7\n//k6TzvS7+tVkI7FMh2LZToWy3QslvV8LMzd82iIiIiUSJt1iIhUkIK7iEgFFRLczewuM/u/Zvaa\nmT1UxGuGxMy+YWYXzJZz/M3sJjN73sz+Mv1Z+WLwZvYBM3vRzP7CzP7czD6b3l/HY3G9mTXN7M/S\nY/Ef0vs/ZGbfTT8r30qTFWrBzK4xs1fN7I/T27U8FmZ21sxOmdn32imQ/XxGcg/uHWUKfhX4CPAb\nZvaRvF83MI8Dd6247yHgBXffAryQ3q66S8Dn3f0jwMeA+9NzoY7H4iJwp7t/FLgNuMvMPgY8DHzF\n3T8MvA0cKrGNRfsscLrjdp2PxS+6+20def49f0aK6LlfLlPg7j8B2mUKasPdjwN/u+LuvcAT6e9P\nADOFNqoE7v6Gu/9p+vsSyQd5lHoeC3f3v09vXpv+c+BO4Nvp/bU4FgBmNgb8OvC19LZR02Oxhp4/\nI0UE99XKFIwW8Lqhu8Xd30h//yFwS5mNKZqZbQZuB75LTY9FOgzxPeAC8DzwV8A77n4pfUidPiu/\nCzwA/Cy93aC+x8KB/2lmr6TlW6CPz4jq1gbA3d3MapOTamY/BzwDfM7d/y7ppCXqdCzc/afAbWZ2\nI3AMmCi5SaUws7uBC+7+iplNl92eAHzC3efNbCPwvJmd6fxjt5+RInruKlOwujfN7FaA9OeFkttT\nCDO7liSwf9Pd/yi9u5bHos3d3wFeBO4AbjSzdqerLp+VjwP3mNlZkmHbO0n2iajjscDd59OfF0gu\n+lP08RkpIrirTMHqngMOpr8fBJ4tsS2FSMdRvw6cdvdHO/5Ux2MxkvbYMbMNJPsinCYJ8p9OH1aL\nY+HuX3T3MXffTBIf/pe7/yY1PBZmdoOZDbd/B34Z+AF9fEYKWaFqZr9GMqbWLlPwO7m/aEDM7A+A\naZISpm8CXwK+AzwNfBB4HbjX3VdOulaKmX0C+N/AKZbHVn+LZNy9bsfiX5NMjF1D0sl62t3/o5mN\nk/RebwJeBfa7+8XyWlqsdFjmC+5+dx2PRfr/fCy9OQT8vrv/jpk16PEzovIDIiIVpBWqIiIVpOAu\nIlJBCu4iIhWk4C4iUkEK7iIiFaTgLiJSQQruIiIV9P8BgBc54uN4GBkAAAAASUVORK5CYII=\n",
"text/plain": [
"