{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Fully-Connected Layers Tutorial on Fashion MNIST Data Set "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is a naive tutorial on how to use `FCLayer` (Fully-connected Layer) to train and predict the Fashion MNIST data set"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### First Let's load the Packages"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Plots.GRBackend()"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"using MLDatasets\n",
"using NumNN\n",
"using Plots\n",
"gr()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Temp for ProgressMeter.jl Package"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"### uncomment this line the first time you run this code\n",
"# ] add https://github.com/timholy/ProgressMeter.jl.git ;"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"using ProgressMeter\n",
"ProgressMeter.ijulia_behavior(:clear);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Load the Train/Test Data/Labels"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"X_train, Y_train = FashionMNIST.traindata(Float64);\n",
"X_test, Y_test = FashionMNIST.testdata(Float64);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Let's Prepare The Data/Labels"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"X_train ./= 255\n",
"X_test ./=255\n",
"Y_train = oneHot(Y_train)\n",
"Y_test = oneHot(Y_test);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### It's Time fot The Layers "
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"X_Input = Input(X_train) #or Input(size(X_train))\n",
"X = Flatten()(X_Input)\n",
"X = FCLayer(120, :relu)(X)\n",
"X_Output = FCLayer(10, :softmax)(X);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Another way when there is no side branches is to use the `chain` function as follows:\n",
"\n",
"```julia\n",
"X_Input, X_Ouput = chain(X_train,[Flatten(),FCLayer(120,:relu),FCLayer(10,:softmax)]);\n",
"```\n",
"\n",
"`chain` returns a `Tuple` of two pointers of the Input `Layer` and Output `Layer`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Define the Model "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This will also initialize the `Layer`s' parameters"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"model = Model(X_train,Y_train,X_Input,X_Output, 0.005; optimizer=:adam);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Let's use `predict` to see the current Accuracy"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[32mProgress: 100%|█████████████████████████████████████████| Time: 0:00:05\u001b[39m\n",
"\u001b[34m Instances 10000: 10000\u001b[39m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"The accuracy of Test Data before the training process 0.0634\n",
"The cost of Test Data before the training process 2.3028\n"
]
}
],
"source": [
"TestP = predict(model, X_test, Y_test);\n",
"\n",
"println()\n",
"println(\"The accuracy of Test Data before the training process $(round(TestP[:accuracy], digits=4))\")\n",
"println(\"The cost of Test Data before the training process $(round(TestP[:cost], digits=4))\")"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[32mProgress: 100%|█████████████████████████████████████████| Time: 0:00:06\u001b[39m\n",
"\u001b[34m Instances 60000: 60000\u001b[39m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"The accuracy of Train Data before the training process 0.0612\n",
"The cost of Train Data before the training process 2.3028\n"
]
}
],
"source": [
"TrainP = predict(model, X_train, Y_train);\n",
"\n",
"println()\n",
"println(\"The accuracy of Train Data before the training process $(round(TrainP[:accuracy], digits=4))\")\n",
"println(\"The cost of Train Data before the training process $(round(TrainP[:cost], digits=4))\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Train the model"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[32mProgress: 100%|█████████████████████████████████████████| Time: 0:00:56\u001b[39m\n",
"\u001b[34m Epoch 10: 10\u001b[39m\n",
"\u001b[34m Instances 60000: 60000\u001b[39m\n",
"\u001b[34m Train Cost: 0.3447\u001b[39m\n",
"\u001b[34m Train Accuracy: 0.8757\u001b[39m\n"
]
}
],
"source": [
"TrainD = train(X_train, Y_train, model, 10);# testData = X_test, testLabels = Y_test);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`train` function provides an extra `kwargs` to use test Data/Labels to get the Costs and Accuracies during each training epochs. \n",
"\n",
"**Note** This will take extra time to do the training\n",
"\n",
"Instead it can be used as follows:\n",
"\n",
"```julia\n",
"TrainD = train(X_train, Y_train, model, 10)\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
"<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600\" height=\"400\" viewBox=\"0 0 2400 1600\">\n",
"<defs>\n",
" <clipPath id=\"clip890\">\n",
" <rect x=\"0\" y=\"0\" width=\"2400\" height=\"1600\"/>\n",
" </clipPath>\n",
"</defs>\n",
"<path clip-path=\"url(#clip890)\" d=\"\n",
"M0 1600 L2400 1600 L2400 0 L0 0 Z\n",
" \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
"<defs>\n",
" <clipPath id=\"clip891\">\n",
" <rect x=\"480\" y=\"0\" width=\"1681\" height=\"1600\"/>\n",
" </clipPath>\n",
"</defs>\n",
"<path clip-path=\"url(#clip890)\" d=\"\n",
"M211.602 1486.45 L2352.76 1486.45 L2352.76 47.2441 L211.602 47.2441 Z\n",
" \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
"<defs>\n",
" <clipPath id=\"clip892\">\n",
" <rect x=\"211\" y=\"47\" width=\"2142\" height=\"1440\"/>\n",
" </clipPath>\n",
"</defs>\n",
"<polyline clip-path=\"url(#clip892)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
" 496.641,1486.45 496.641,47.2441 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip892)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
" 945.52,1486.45 945.52,47.2441 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip892)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
" 1394.4,1486.45 1394.4,47.2441 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip892)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
" 1843.28,1486.45 1843.28,47.2441 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip892)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
" 2292.16,1486.45 2292.16,47.2441 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip892)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
" 211.602,1304.36 2352.76,1304.36 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip892)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
" 211.602,1048.65 2352.76,1048.65 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip892)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
" 211.602,792.935 2352.76,792.935 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip892)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
" 211.602,537.222 2352.76,537.222 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip892)\" style=\"stroke:#000000; stroke-width:2; stroke-opacity:0.1; fill:none\" points=\"\n",
" 211.602,281.509 2352.76,281.509 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip890)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 211.602,1486.45 2352.76,1486.45 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip890)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 211.602,1486.45 211.602,47.2441 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip890)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 496.641,1486.45 496.641,1469.18 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip890)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 945.52,1486.45 945.52,1469.18 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip890)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 1394.4,1486.45 1394.4,1469.18 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip890)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 1843.28,1486.45 1843.28,1469.18 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip890)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 2292.16,1486.45 2292.16,1469.18 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip890)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 211.602,1304.36 237.296,1304.36 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip890)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 211.602,1048.65 237.296,1048.65 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip890)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 211.602,792.935 237.296,792.935 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip890)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 211.602,537.222 237.296,537.222 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip890)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 211.602,281.509 237.296,281.509 \n",
" \"/>\n",
"<path clip-path=\"url(#clip890)\" d=\"M 0 0 M491.293 1535.98 L507.613 1535.98 L507.613 1539.92 L485.669 1539.92 L485.669 1535.98 Q488.331 1533.23 492.914 1528.6 Q497.52 1523.95 498.701 1522.61 Q500.946 1520.08 501.826 1518.35 Q502.729 1516.59 502.729 1514.9 Q502.729 1512.14 500.784 1510.41 Q498.863 1508.67 495.761 1508.67 Q493.562 1508.67 491.108 1509.43 Q488.678 1510.2 485.9 1511.75 L485.9 1507.03 Q488.724 1505.89 491.178 1505.31 Q493.631 1504.73 495.668 1504.73 Q501.039 1504.73 504.233 1507.42 Q507.428 1510.11 507.428 1514.6 Q507.428 1516.73 506.617 1518.65 Q505.83 1520.54 503.724 1523.14 Q503.145 1523.81 500.043 1527.03 Q496.942 1530.22 491.293 1535.98 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M948.529 1509.43 L936.724 1527.88 L948.529 1527.88 L948.529 1509.43 M947.302 1505.36 L953.182 1505.36 L953.182 1527.88 L958.112 1527.88 L958.112 1531.77 L953.182 1531.77 L953.182 1539.92 L948.529 1539.92 L948.529 1531.77 L932.927 1531.77 L932.927 1527.26 L947.302 1505.36 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1394.8 1520.78 Q1391.66 1520.78 1389.8 1522.93 Q1387.98 1525.08 1387.98 1528.83 Q1387.98 1532.56 1389.8 1534.73 Q1391.66 1536.89 1394.8 1536.89 Q1397.95 1536.89 1399.78 1534.73 Q1401.63 1532.56 1401.63 1528.83 Q1401.63 1525.08 1399.78 1522.93 Q1397.95 1520.78 1394.8 1520.78 M1404.09 1506.12 L1404.09 1510.38 Q1402.33 1509.55 1400.52 1509.11 Q1398.74 1508.67 1396.98 1508.67 Q1392.35 1508.67 1389.9 1511.8 Q1387.47 1514.92 1387.12 1521.24 Q1388.48 1519.23 1390.54 1518.16 Q1392.6 1517.07 1395.08 1517.07 Q1400.29 1517.07 1403.3 1520.24 Q1406.33 1523.39 1406.33 1528.83 Q1406.33 1534.16 1403.18 1537.37 Q1400.04 1540.59 1394.8 1540.59 Q1388.81 1540.59 1385.64 1536.01 Q1382.47 1531.4 1382.47 1522.67 Q1382.47 1514.48 1386.36 1509.62 Q1390.24 1504.73 1396.79 1504.73 Q1398.55 1504.73 1400.34 1505.08 Q1402.14 1505.43 1404.09 1506.12 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1843.28 1523.51 Q1839.94 1523.51 1838.02 1525.29 Q1836.13 1527.07 1836.13 1530.2 Q1836.13 1533.32 1838.02 1535.11 Q1839.94 1536.89 1843.28 1536.89 Q1846.61 1536.89 1848.53 1535.11 Q1850.45 1533.3 1850.45 1530.2 Q1850.45 1527.07 1848.53 1525.29 Q1846.63 1523.51 1843.28 1523.51 M1838.6 1521.52 Q1835.59 1520.78 1833.9 1518.72 Q1832.24 1516.66 1832.24 1513.69 Q1832.24 1509.55 1835.18 1507.14 Q1838.14 1504.73 1843.28 1504.73 Q1848.44 1504.73 1851.38 1507.14 Q1854.32 1509.55 1854.32 1513.69 Q1854.32 1516.66 1852.63 1518.72 Q1850.96 1520.78 1847.98 1521.52 Q1851.36 1522.3 1853.23 1524.6 Q1855.13 1526.89 1855.13 1530.2 Q1855.13 1535.22 1852.05 1537.91 Q1849 1540.59 1843.28 1540.59 Q1837.56 1540.59 1834.48 1537.91 Q1831.43 1535.22 1831.43 1530.2 Q1831.43 1526.89 1833.32 1524.6 Q1835.22 1522.3 1838.6 1521.52 M1836.89 1514.13 Q1836.89 1516.82 1838.56 1518.32 Q1840.25 1519.83 1843.28 1519.83 Q1846.29 1519.83 1847.98 1518.32 Q1849.69 1516.82 1849.69 1514.13 Q1849.69 1511.45 1847.98 1509.94 Q1846.29 1508.44 1843.28 1508.44 Q1840.25 1508.44 1838.56 1509.94 Q1836.89 1511.45 1836.89 1514.13 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2269.03 1535.98 L2276.67 1535.98 L2276.67 1509.62 L2268.36 1511.29 L2268.36 1507.03 L2276.62 1505.36 L2281.3 1505.36 L2281.3 1535.98 L2288.94 1535.98 L2288.94 1539.92 L2269.03 1539.92 L2269.03 1535.98 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2304.01 1508.44 Q2300.4 1508.44 2298.57 1512 Q2296.76 1515.55 2296.76 1522.67 Q2296.76 1529.78 2298.57 1533.35 Q2300.4 1536.89 2304.01 1536.89 Q2307.64 1536.89 2309.45 1533.35 Q2311.28 1529.78 2311.28 1522.67 Q2311.28 1515.55 2309.45 1512 Q2307.64 1508.44 2304.01 1508.44 M2304.01 1504.73 Q2309.82 1504.73 2312.87 1509.34 Q2315.95 1513.92 2315.95 1522.67 Q2315.95 1531.4 2312.87 1536.01 Q2309.82 1540.59 2304.01 1540.59 Q2298.2 1540.59 2295.12 1536.01 Q2292.06 1531.4 2292.06 1522.67 Q2292.06 1513.92 2295.12 1509.34 Q2298.2 1504.73 2304.01 1504.73 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M138.205 1290.16 Q134.593 1290.16 132.765 1293.73 Q130.959 1297.27 130.959 1304.4 Q130.959 1311.5 132.765 1315.07 Q134.593 1318.61 138.205 1318.61 Q141.839 1318.61 143.644 1315.07 Q145.473 1311.5 145.473 1304.4 Q145.473 1297.27 143.644 1293.73 Q141.839 1290.16 138.205 1290.16 M138.205 1286.46 Q144.015 1286.46 147.07 1291.06 Q150.149 1295.65 150.149 1304.4 Q150.149 1313.12 147.07 1317.73 Q144.015 1322.31 138.205 1322.31 Q132.394 1322.31 129.316 1317.73 Q126.26 1313.12 126.26 1304.4 Q126.26 1295.65 129.316 1291.06 Q132.394 1286.46 138.205 1286.46 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M155.218 1315.76 L160.103 1315.76 L160.103 1321.64 L155.218 1321.64 L155.218 1315.76 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M178.019 1291.16 L166.214 1309.6 L178.019 1309.6 L178.019 1291.16 M176.792 1287.08 L182.672 1287.08 L182.672 1309.6 L187.602 1309.6 L187.602 1313.49 L182.672 1313.49 L182.672 1321.64 L178.019 1321.64 L178.019 1313.49 L162.417 1313.49 L162.417 1308.98 L176.792 1287.08 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M139.686 1034.45 Q136.075 1034.45 134.246 1038.01 Q132.441 1041.55 132.441 1048.68 Q132.441 1055.79 134.246 1059.35 Q136.075 1062.9 139.686 1062.9 Q143.32 1062.9 145.126 1059.35 Q146.954 1055.79 146.954 1048.68 Q146.954 1041.55 145.126 1038.01 Q143.32 1034.45 139.686 1034.45 M139.686 1030.74 Q145.496 1030.74 148.552 1035.35 Q151.63 1039.93 151.63 1048.68 Q151.63 1057.41 148.552 1062.02 Q145.496 1066.6 139.686 1066.6 Q133.876 1066.6 130.797 1062.02 Q127.742 1057.41 127.742 1048.68 Q127.742 1039.93 130.797 1035.35 Q133.876 1030.74 139.686 1030.74 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M156.7 1060.05 L161.584 1060.05 L161.584 1065.93 L156.7 1065.93 L156.7 1060.05 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M166.7 1031.37 L185.056 1031.37 L185.056 1035.3 L170.982 1035.3 L170.982 1043.78 Q172.001 1043.43 173.019 1043.27 Q174.038 1043.08 175.056 1043.08 Q180.843 1043.08 184.223 1046.25 Q187.602 1049.42 187.602 1054.84 Q187.602 1060.42 184.13 1063.52 Q180.658 1066.6 174.339 1066.6 Q172.163 1066.6 169.894 1066.23 Q167.649 1065.86 165.241 1065.12 L165.241 1060.42 Q167.325 1061.55 169.547 1062.11 Q171.769 1062.66 174.246 1062.66 Q178.251 1062.66 180.589 1060.56 Q182.927 1058.45 182.927 1054.84 Q182.927 1051.23 180.589 1049.12 Q178.251 1047.02 174.246 1047.02 Q172.371 1047.02 170.496 1047.43 Q168.644 1047.85 166.7 1048.73 L166.7 1031.37 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M138.529 778.734 Q134.918 778.734 133.089 782.299 Q131.283 785.84 131.283 792.97 Q131.283 800.076 133.089 803.641 Q134.918 807.183 138.529 807.183 Q142.163 807.183 143.968 803.641 Q145.797 800.076 145.797 792.97 Q145.797 785.84 143.968 782.299 Q142.163 778.734 138.529 778.734 M138.529 775.03 Q144.339 775.03 147.394 779.637 Q150.473 784.22 150.473 792.97 Q150.473 801.697 147.394 806.303 Q144.339 810.886 138.529 810.886 Q132.718 810.886 129.64 806.303 Q126.584 801.697 126.584 792.97 Q126.584 784.22 129.64 779.637 Q132.718 775.03 138.529 775.03 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M155.542 804.336 L160.427 804.336 L160.427 810.215 L155.542 810.215 L155.542 804.336 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M176.075 791.072 Q172.927 791.072 171.075 793.225 Q169.246 795.377 169.246 799.127 Q169.246 802.854 171.075 805.03 Q172.927 807.183 176.075 807.183 Q179.223 807.183 181.052 805.03 Q182.903 802.854 182.903 799.127 Q182.903 795.377 181.052 793.225 Q179.223 791.072 176.075 791.072 M185.357 776.419 L185.357 780.678 Q183.598 779.845 181.792 779.405 Q180.01 778.965 178.251 778.965 Q173.621 778.965 171.167 782.09 Q168.737 785.215 168.39 791.535 Q169.755 789.521 171.815 788.456 Q173.876 787.368 176.352 787.368 Q181.561 787.368 184.57 790.539 Q187.602 793.688 187.602 799.127 Q187.602 804.451 184.454 807.669 Q181.306 810.886 176.075 810.886 Q170.079 810.886 166.908 806.303 Q163.737 801.697 163.737 792.97 Q163.737 784.776 167.626 779.914 Q171.515 775.03 178.065 775.03 Q179.825 775.03 181.607 775.377 Q183.413 775.725 185.357 776.419 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M139.593 523.021 Q135.982 523.021 134.154 526.585 Q132.348 530.127 132.348 537.257 Q132.348 544.363 134.154 547.928 Q135.982 551.47 139.593 551.47 Q143.228 551.47 145.033 547.928 Q146.862 544.363 146.862 537.257 Q146.862 530.127 145.033 526.585 Q143.228 523.021 139.593 523.021 M139.593 519.317 Q145.404 519.317 148.459 523.923 Q151.538 528.507 151.538 537.257 Q151.538 545.983 148.459 550.59 Q145.404 555.173 139.593 555.173 Q133.783 555.173 130.705 550.59 Q127.649 545.983 127.649 537.257 Q127.649 528.507 130.705 523.923 Q133.783 519.317 139.593 519.317 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M156.607 548.622 L161.491 548.622 L161.491 554.502 L156.607 554.502 L156.607 548.622 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M165.38 519.942 L187.602 519.942 L187.602 521.933 L175.056 554.502 L170.172 554.502 L181.977 523.877 L165.38 523.877 L165.38 519.942 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M138.783 267.307 Q135.172 267.307 133.343 270.872 Q131.538 274.414 131.538 281.543 Q131.538 288.65 133.343 292.215 Q135.172 295.756 138.783 295.756 Q142.417 295.756 144.223 292.215 Q146.052 288.65 146.052 281.543 Q146.052 274.414 144.223 270.872 Q142.417 267.307 138.783 267.307 M138.783 263.604 Q144.593 263.604 147.649 268.21 Q150.728 272.793 150.728 281.543 Q150.728 290.27 147.649 294.877 Q144.593 299.46 138.783 299.46 Q132.973 299.46 129.894 294.877 Q126.839 290.27 126.839 281.543 Q126.839 272.793 129.894 268.21 Q132.973 263.604 138.783 263.604 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M155.797 292.909 L160.681 292.909 L160.681 298.789 L155.797 298.789 L155.797 292.909 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M175.751 282.377 Q172.417 282.377 170.496 284.159 Q168.598 285.941 168.598 289.066 Q168.598 292.191 170.496 293.974 Q172.417 295.756 175.751 295.756 Q179.084 295.756 181.005 293.974 Q182.927 292.168 182.927 289.066 Q182.927 285.941 181.005 284.159 Q179.107 282.377 175.751 282.377 M171.075 280.386 Q168.065 279.645 166.376 277.585 Q164.709 275.525 164.709 272.562 Q164.709 268.418 167.649 266.011 Q170.612 263.604 175.751 263.604 Q180.913 263.604 183.852 266.011 Q186.792 268.418 186.792 272.562 Q186.792 275.525 185.102 277.585 Q183.436 279.645 180.45 280.386 Q183.829 281.173 185.704 283.465 Q187.602 285.756 187.602 289.066 Q187.602 294.09 184.524 296.775 Q181.468 299.46 175.751 299.46 Q170.033 299.46 166.954 296.775 Q163.899 294.09 163.899 289.066 Q163.899 285.756 165.797 283.465 Q167.695 281.173 171.075 280.386 M169.362 273.002 Q169.362 275.687 171.028 277.192 Q172.718 278.696 175.751 278.696 Q178.76 278.696 180.45 277.192 Q182.163 275.687 182.163 273.002 Q182.163 270.317 180.45 268.812 Q178.76 267.307 175.751 267.307 Q172.718 267.307 171.028 268.812 Q169.362 270.317 169.362 273.002 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M40.4842 868.395 L40.4842 838.349 L45.895 838.349 L45.895 861.966 L59.9632 861.966 L59.9632 839.335 L65.3741 839.335 L65.3741 861.966 L82.5933 861.966 L82.5933 837.776 L88.0042 837.776 L88.0042 868.395 L40.4842 868.395 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M82.657 825.967 L101.563 825.967 L101.563 831.856 L52.3562 831.856 L52.3562 825.967 L57.7671 825.967 Q54.5842 824.121 53.0564 821.321 Q51.4968 818.488 51.4968 814.573 Q51.4968 808.08 56.6531 804.038 Q61.8093 799.964 70.212 799.964 Q78.6147 799.964 83.771 804.038 Q88.9272 808.08 88.9272 814.573 Q88.9272 818.488 87.3994 821.321 Q85.8398 824.121 82.657 825.967 M70.212 806.043 Q63.7508 806.043 60.0905 808.716 Q56.3984 811.358 56.3984 816.005 Q56.3984 820.652 60.0905 823.326 Q63.7508 825.967 70.212 825.967 Q76.6732 825.967 80.3653 823.326 Q84.0256 820.652 84.0256 816.005 Q84.0256 811.358 80.3653 808.716 Q76.6732 806.043 70.212 806.043 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M56.4621 780.007 Q56.4621 784.718 60.1542 787.455 Q63.8145 790.192 70.212 790.192 Q76.6095 790.192 80.3017 787.487 Q83.9619 784.75 83.9619 780.007 Q83.9619 775.328 80.2698 772.591 Q76.5777 769.854 70.212 769.854 Q63.8781 769.854 60.186 772.591 Q56.4621 775.328 56.4621 780.007 M51.4968 780.007 Q51.4968 772.368 56.4621 768.008 Q61.4273 763.647 70.212 763.647 Q78.9649 763.647 83.9619 768.008 Q88.9272 772.368 88.9272 780.007 Q88.9272 787.678 83.9619 792.038 Q78.9649 796.367 70.212 796.367 Q61.4273 796.367 56.4621 792.038 Q51.4968 787.678 51.4968 780.007 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M53.7248 731.851 L59.1993 731.851 Q57.8307 734.333 57.1623 736.848 Q56.4621 739.33 56.4621 741.876 Q56.4621 747.574 60.0905 750.725 Q63.6872 753.876 70.212 753.876 Q76.7369 753.876 80.3653 750.725 Q83.9619 747.574 83.9619 741.876 Q83.9619 739.33 83.2935 736.848 Q82.5933 734.333 81.2247 731.851 L86.6355 731.851 Q87.7814 734.301 88.3543 736.943 Q88.9272 739.553 88.9272 742.513 Q88.9272 750.566 83.8664 755.308 Q78.8057 760.051 70.212 760.051 Q61.491 760.051 56.4939 755.276 Q51.4968 750.47 51.4968 742.131 Q51.4968 739.426 52.0697 736.848 Q52.6108 734.269 53.7248 731.851 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M66.4881 696.075 L88.0042 696.075 L88.0042 701.932 L66.679 701.932 Q61.6183 701.932 59.1038 703.905 Q56.5894 705.878 56.5894 709.825 Q56.5894 714.568 59.6131 717.305 Q62.6368 720.042 67.8567 720.042 L88.0042 720.042 L88.0042 725.93 L38.479 725.93 L38.479 720.042 L57.8944 720.042 Q54.6797 717.941 53.0883 715.109 Q51.4968 712.244 51.4968 708.52 Q51.4968 702.377 55.3163 699.226 Q59.1038 696.075 66.4881 696.075 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M53.4065 667.207 L58.9447 667.207 Q57.6716 669.689 57.035 672.363 Q56.3984 675.037 56.3984 677.901 Q56.3984 682.262 57.7352 684.458 Q59.072 686.622 61.7456 686.622 Q63.7826 686.622 64.9603 685.063 Q66.1061 683.503 67.1565 678.792 L67.6021 676.787 Q68.9389 670.549 71.3897 667.939 Q73.8086 665.297 78.1691 665.297 Q83.1344 665.297 86.0308 669.244 Q88.9272 673.159 88.9272 680.034 Q88.9272 682.898 88.3543 686.017 Q87.8132 689.105 86.6992 692.542 L80.6518 692.542 Q82.3387 689.296 83.198 686.145 Q84.0256 682.994 84.0256 679.906 Q84.0256 675.769 82.6251 673.541 Q81.1929 671.313 78.6147 671.313 Q76.2276 671.313 74.9545 672.936 Q73.6813 674.527 72.5037 679.97 L72.0262 682.007 Q70.8804 687.45 68.5251 689.869 Q66.138 692.288 62.0002 692.288 Q56.9713 692.288 54.2341 688.723 Q51.4968 685.158 51.4968 678.601 Q51.4968 675.355 51.9743 672.49 Q52.4517 669.626 53.4065 667.207 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip892)\" style=\"stroke:#009af9; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 272.201,420.617 496.641,209.696 721.08,170.359 945.52,146.023 1169.96,132.854 1394.4,123.734 1618.84,112.141 1843.28,104.129 2067.72,94.0708 2292.16,87.9763 \n",
" \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip892)\" style=\"stroke:#e26f46; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 272.201,464.57 496.641,1092.13 721.08,1204.56 945.52,1268.45 1169.96,1310.64 1394.4,1343.04 1618.84,1373.53 1843.28,1398.11 2067.72,1421.8 2292.16,1445.72 \n",
" \n",
" \"/>\n",
"<path clip-path=\"url(#clip890)\" d=\"\n",
"M1635.6 312.204 L2280.76 312.204 L2280.76 130.764 L1635.6 130.764 Z\n",
" \" fill=\"#ffffff\" fill-rule=\"evenodd\" fill-opacity=\"1\"/>\n",
"<polyline clip-path=\"url(#clip890)\" style=\"stroke:#000000; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 1635.6,312.204 2280.76,312.204 2280.76,130.764 1635.6,130.764 1635.6,312.204 \n",
" \"/>\n",
"<polyline clip-path=\"url(#clip890)\" style=\"stroke:#009af9; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 1659.6,191.244 1803.6,191.244 \n",
" \"/>\n",
"<path clip-path=\"url(#clip890)\" d=\"M 0 0 M1827.6 173.964 L1856.83 173.964 L1856.83 177.899 L1844.56 177.899 L1844.56 208.524 L1839.86 208.524 L1839.86 177.899 L1827.6 177.899 L1827.6 173.964 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1869.36 186.58 Q1868.64 186.163 1867.78 185.978 Q1866.95 185.77 1865.93 185.77 Q1862.32 185.77 1860.37 188.131 Q1858.45 190.469 1858.45 194.867 L1858.45 208.524 L1854.17 208.524 L1854.17 182.598 L1858.45 182.598 L1858.45 186.626 Q1859.79 184.265 1861.95 183.131 Q1864.1 181.973 1867.18 181.973 Q1867.62 181.973 1868.15 182.043 Q1868.68 182.089 1869.33 182.205 L1869.36 186.58 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1885.6 195.492 Q1880.44 195.492 1878.45 196.672 Q1876.46 197.853 1876.46 200.7 Q1876.46 202.969 1877.94 204.311 Q1879.45 205.631 1882.02 205.631 Q1885.56 205.631 1887.69 203.131 Q1889.84 200.607 1889.84 196.441 L1889.84 195.492 L1885.6 195.492 M1894.1 193.733 L1894.1 208.524 L1889.84 208.524 L1889.84 204.589 Q1888.38 206.95 1886.21 208.084 Q1884.03 209.195 1880.88 209.195 Q1876.9 209.195 1874.54 206.973 Q1872.2 204.728 1872.2 200.978 Q1872.2 196.603 1875.12 194.381 Q1878.06 192.158 1883.87 192.158 L1889.84 192.158 L1889.84 191.742 Q1889.84 188.802 1887.9 187.205 Q1885.98 185.584 1882.48 185.584 Q1880.26 185.584 1878.15 186.117 Q1876.04 186.649 1874.1 187.714 L1874.1 183.779 Q1876.44 182.876 1878.64 182.436 Q1880.84 181.973 1882.92 181.973 Q1888.54 181.973 1891.32 184.89 Q1894.1 187.807 1894.1 193.733 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1898.57 182.598 L1902.83 182.598 L1902.83 208.524 L1898.57 208.524 L1898.57 182.598 M1898.57 172.506 L1902.83 172.506 L1902.83 177.899 L1898.57 177.899 L1898.57 172.506 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1928.85 192.876 L1928.85 208.524 L1924.59 208.524 L1924.59 193.015 Q1924.59 189.334 1923.15 187.506 Q1921.72 185.677 1918.85 185.677 Q1915.4 185.677 1913.41 187.876 Q1911.42 190.075 1911.42 193.871 L1911.42 208.524 L1907.13 208.524 L1907.13 182.598 L1911.42 182.598 L1911.42 186.626 Q1912.94 184.288 1915 183.131 Q1917.09 181.973 1919.79 181.973 Q1924.26 181.973 1926.55 184.751 Q1928.85 187.506 1928.85 192.876 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1933.31 182.598 L1937.57 182.598 L1937.57 208.524 L1933.31 208.524 L1933.31 182.598 M1933.31 172.506 L1937.57 172.506 L1937.57 177.899 L1933.31 177.899 L1933.31 172.506 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1963.59 192.876 L1963.59 208.524 L1959.33 208.524 L1959.33 193.015 Q1959.33 189.334 1957.9 187.506 Q1956.46 185.677 1953.59 185.677 Q1950.14 185.677 1948.15 187.876 Q1946.16 190.075 1946.16 193.871 L1946.16 208.524 L1941.88 208.524 L1941.88 182.598 L1946.16 182.598 L1946.16 186.626 Q1947.69 184.288 1949.75 183.131 Q1951.83 181.973 1954.54 181.973 Q1959.01 181.973 1961.3 184.751 Q1963.59 187.506 1963.59 192.876 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1985.12 195.26 Q1985.12 190.631 1983.2 188.084 Q1981.3 185.538 1977.85 185.538 Q1974.42 185.538 1972.5 188.084 Q1970.6 190.631 1970.6 195.26 Q1970.6 199.867 1972.5 202.413 Q1974.42 204.959 1977.85 204.959 Q1981.3 204.959 1983.2 202.413 Q1985.12 199.867 1985.12 195.26 M1989.38 205.307 Q1989.38 211.927 1986.44 215.144 Q1983.5 218.385 1977.43 218.385 Q1975.19 218.385 1973.2 218.038 Q1971.21 217.714 1969.33 217.019 L1969.33 212.876 Q1971.21 213.894 1973.04 214.381 Q1974.86 214.867 1976.76 214.867 Q1980.95 214.867 1983.04 212.668 Q1985.12 210.492 1985.12 206.07 L1985.12 203.964 Q1983.8 206.256 1981.74 207.39 Q1979.68 208.524 1976.81 208.524 Q1972.04 208.524 1969.12 204.89 Q1966.21 201.256 1966.21 195.26 Q1966.21 189.242 1969.12 185.608 Q1972.04 181.973 1976.81 181.973 Q1979.68 181.973 1981.74 183.108 Q1983.8 184.242 1985.12 186.533 L1985.12 182.598 L1989.38 182.598 L1989.38 205.307 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2020.65 178.571 L2014.31 195.77 L2027.02 195.77 L2020.65 178.571 M2018.01 173.964 L2023.31 173.964 L2036.48 208.524 L2031.62 208.524 L2028.47 199.658 L2012.9 199.658 L2009.75 208.524 L2004.82 208.524 L2018.01 173.964 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2058.78 183.594 L2058.78 187.575 Q2056.97 186.58 2055.14 186.094 Q2053.34 185.584 2051.48 185.584 Q2047.34 185.584 2045.05 188.223 Q2042.76 190.839 2042.76 195.584 Q2042.76 200.33 2045.05 202.969 Q2047.34 205.584 2051.48 205.584 Q2053.34 205.584 2055.14 205.098 Q2056.97 204.589 2058.78 203.594 L2058.78 207.529 Q2056.99 208.362 2055.07 208.779 Q2053.17 209.195 2051.02 209.195 Q2045.16 209.195 2041.72 205.515 Q2038.27 201.834 2038.27 195.584 Q2038.27 189.242 2041.74 185.608 Q2045.23 181.973 2051.3 181.973 Q2053.27 181.973 2055.14 182.39 Q2057.02 182.783 2058.78 183.594 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2081.9 183.594 L2081.9 187.575 Q2080.09 186.58 2078.27 186.094 Q2076.46 185.584 2074.61 185.584 Q2070.47 185.584 2068.17 188.223 Q2065.88 190.839 2065.88 195.584 Q2065.88 200.33 2068.17 202.969 Q2070.47 205.584 2074.61 205.584 Q2076.46 205.584 2078.27 205.098 Q2080.09 204.589 2081.9 203.594 L2081.9 207.529 Q2080.12 208.362 2078.2 208.779 Q2076.3 209.195 2074.15 209.195 Q2068.29 209.195 2064.84 205.515 Q2061.39 201.834 2061.39 195.584 Q2061.39 189.242 2064.86 185.608 Q2068.36 181.973 2074.42 181.973 Q2076.39 181.973 2078.27 182.39 Q2080.14 182.783 2081.9 183.594 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2085.93 198.293 L2085.93 182.598 L2090.19 182.598 L2090.19 198.131 Q2090.19 201.811 2091.62 203.663 Q2093.06 205.492 2095.93 205.492 Q2099.38 205.492 2101.37 203.293 Q2103.38 201.094 2103.38 197.297 L2103.38 182.598 L2107.64 182.598 L2107.64 208.524 L2103.38 208.524 L2103.38 204.543 Q2101.83 206.904 2099.77 208.061 Q2097.73 209.195 2095.03 209.195 Q2090.56 209.195 2088.24 206.418 Q2085.93 203.64 2085.93 198.293 M2096.65 181.973 L2096.65 181.973 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2127.13 186.58 Q2126.41 186.163 2125.56 185.978 Q2124.72 185.77 2123.71 185.77 Q2120.09 185.77 2118.15 188.131 Q2116.23 190.469 2116.23 194.867 L2116.23 208.524 L2111.95 208.524 L2111.95 182.598 L2116.23 182.598 L2116.23 186.626 Q2117.57 184.265 2119.72 183.131 Q2121.88 181.973 2124.96 181.973 Q2125.4 181.973 2125.93 182.043 Q2126.46 182.089 2127.11 182.205 L2127.13 186.58 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2143.38 195.492 Q2138.22 195.492 2136.23 196.672 Q2134.24 197.853 2134.24 200.7 Q2134.24 202.969 2135.72 204.311 Q2137.22 205.631 2139.79 205.631 Q2143.34 205.631 2145.46 203.131 Q2147.62 200.607 2147.62 196.441 L2147.62 195.492 L2143.38 195.492 M2151.88 193.733 L2151.88 208.524 L2147.62 208.524 L2147.62 204.589 Q2146.16 206.95 2143.98 208.084 Q2141.81 209.195 2138.66 209.195 Q2134.68 209.195 2132.32 206.973 Q2129.98 204.728 2129.98 200.978 Q2129.98 196.603 2132.9 194.381 Q2135.84 192.158 2141.65 192.158 L2147.62 192.158 L2147.62 191.742 Q2147.62 188.802 2145.67 187.205 Q2143.75 185.584 2140.26 185.584 Q2138.03 185.584 2135.93 186.117 Q2133.82 186.649 2131.88 187.714 L2131.88 183.779 Q2134.21 182.876 2136.41 182.436 Q2138.61 181.973 2140.7 181.973 Q2146.32 181.973 2149.1 184.89 Q2151.88 187.807 2151.88 193.733 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2175 183.594 L2175 187.575 Q2173.2 186.58 2171.37 186.094 Q2169.56 185.584 2167.71 185.584 Q2163.57 185.584 2161.27 188.223 Q2158.98 190.839 2158.98 195.584 Q2158.98 200.33 2161.27 202.969 Q2163.57 205.584 2167.71 205.584 Q2169.56 205.584 2171.37 205.098 Q2173.2 204.589 2175 203.594 L2175 207.529 Q2173.22 208.362 2171.3 208.779 Q2169.4 209.195 2167.25 209.195 Q2161.39 209.195 2157.94 205.515 Q2154.49 201.834 2154.49 195.584 Q2154.49 189.242 2157.96 185.608 Q2161.46 181.973 2167.52 181.973 Q2169.49 181.973 2171.37 182.39 Q2173.24 182.783 2175 183.594 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2179.47 182.598 L2183.73 182.598 L2183.73 208.524 L2179.47 208.524 L2179.47 182.598 M2179.47 172.506 L2183.73 172.506 L2183.73 177.899 L2179.47 177.899 L2179.47 172.506 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2210.37 194.496 L2210.37 196.58 L2190.79 196.58 Q2191.07 200.978 2193.43 203.293 Q2195.81 205.584 2200.05 205.584 Q2202.5 205.584 2204.79 204.982 Q2207.11 204.381 2209.38 203.177 L2209.38 207.205 Q2207.08 208.177 2204.68 208.686 Q2202.27 209.195 2199.79 209.195 Q2193.59 209.195 2189.96 205.584 Q2186.34 201.973 2186.34 195.816 Q2186.34 189.45 2189.77 185.723 Q2193.22 181.973 2199.05 181.973 Q2204.28 181.973 2207.32 185.353 Q2210.37 188.709 2210.37 194.496 M2206.11 193.246 Q2206.07 189.751 2204.14 187.668 Q2202.25 185.584 2199.1 185.584 Q2195.53 185.584 2193.38 187.598 Q2191.25 189.612 2190.93 193.27 L2206.11 193.246 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2231.37 183.362 L2231.37 187.39 Q2229.56 186.464 2227.62 186.001 Q2225.67 185.538 2223.59 185.538 Q2220.42 185.538 2218.82 186.51 Q2217.25 187.483 2217.25 189.427 Q2217.25 190.908 2218.38 191.765 Q2219.52 192.598 2222.94 193.362 L2224.4 193.686 Q2228.94 194.658 2230.83 196.441 Q2232.76 198.2 2232.76 201.371 Q2232.76 204.982 2229.89 207.089 Q2227.04 209.195 2222.04 209.195 Q2219.96 209.195 2217.69 208.779 Q2215.44 208.385 2212.94 207.575 L2212.94 203.177 Q2215.3 204.404 2217.59 205.029 Q2219.89 205.631 2222.13 205.631 Q2225.14 205.631 2226.76 204.612 Q2228.38 203.57 2228.38 201.695 Q2228.38 199.959 2227.2 199.033 Q2226.04 198.107 2222.08 197.251 L2220.6 196.904 Q2216.64 196.07 2214.89 194.358 Q2213.13 192.621 2213.13 189.612 Q2213.13 185.955 2215.72 183.964 Q2218.31 181.973 2223.08 181.973 Q2225.44 181.973 2227.52 182.321 Q2229.61 182.668 2231.37 183.362 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><polyline clip-path=\"url(#clip890)\" style=\"stroke:#e26f46; stroke-width:4; stroke-opacity:1; fill:none\" points=\"\n",
" 1659.6,251.724 1803.6,251.724 \n",
" \"/>\n",
"<path clip-path=\"url(#clip890)\" d=\"M 0 0 M1827.6 234.444 L1856.83 234.444 L1856.83 238.379 L1844.56 238.379 L1844.56 269.004 L1839.86 269.004 L1839.86 238.379 L1827.6 238.379 L1827.6 234.444 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1869.36 247.06 Q1868.64 246.643 1867.78 246.458 Q1866.95 246.25 1865.93 246.25 Q1862.32 246.25 1860.37 248.611 Q1858.45 250.949 1858.45 255.347 L1858.45 269.004 L1854.17 269.004 L1854.17 243.078 L1858.45 243.078 L1858.45 247.106 Q1859.79 244.745 1861.95 243.611 Q1864.1 242.453 1867.18 242.453 Q1867.62 242.453 1868.15 242.523 Q1868.68 242.569 1869.33 242.685 L1869.36 247.06 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1885.6 255.972 Q1880.44 255.972 1878.45 257.152 Q1876.46 258.333 1876.46 261.18 Q1876.46 263.449 1877.94 264.791 Q1879.45 266.111 1882.02 266.111 Q1885.56 266.111 1887.69 263.611 Q1889.84 261.087 1889.84 256.921 L1889.84 255.972 L1885.6 255.972 M1894.1 254.213 L1894.1 269.004 L1889.84 269.004 L1889.84 265.069 Q1888.38 267.43 1886.21 268.564 Q1884.03 269.675 1880.88 269.675 Q1876.9 269.675 1874.54 267.453 Q1872.2 265.208 1872.2 261.458 Q1872.2 257.083 1875.12 254.861 Q1878.06 252.638 1883.87 252.638 L1889.84 252.638 L1889.84 252.222 Q1889.84 249.282 1887.9 247.685 Q1885.98 246.064 1882.48 246.064 Q1880.26 246.064 1878.15 246.597 Q1876.04 247.129 1874.1 248.194 L1874.1 244.259 Q1876.44 243.356 1878.64 242.916 Q1880.84 242.453 1882.92 242.453 Q1888.54 242.453 1891.32 245.37 Q1894.1 248.287 1894.1 254.213 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1898.57 243.078 L1902.83 243.078 L1902.83 269.004 L1898.57 269.004 L1898.57 243.078 M1898.57 232.986 L1902.83 232.986 L1902.83 238.379 L1898.57 238.379 L1898.57 232.986 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1928.85 253.356 L1928.85 269.004 L1924.59 269.004 L1924.59 253.495 Q1924.59 249.814 1923.15 247.986 Q1921.72 246.157 1918.85 246.157 Q1915.4 246.157 1913.41 248.356 Q1911.42 250.555 1911.42 254.351 L1911.42 269.004 L1907.13 269.004 L1907.13 243.078 L1911.42 243.078 L1911.42 247.106 Q1912.94 244.768 1915 243.611 Q1917.09 242.453 1919.79 242.453 Q1924.26 242.453 1926.55 245.231 Q1928.85 247.986 1928.85 253.356 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1933.31 243.078 L1937.57 243.078 L1937.57 269.004 L1933.31 269.004 L1933.31 243.078 M1933.31 232.986 L1937.57 232.986 L1937.57 238.379 L1933.31 238.379 L1933.31 232.986 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1963.59 253.356 L1963.59 269.004 L1959.33 269.004 L1959.33 253.495 Q1959.33 249.814 1957.9 247.986 Q1956.46 246.157 1953.59 246.157 Q1950.14 246.157 1948.15 248.356 Q1946.16 250.555 1946.16 254.351 L1946.16 269.004 L1941.88 269.004 L1941.88 243.078 L1946.16 243.078 L1946.16 247.106 Q1947.69 244.768 1949.75 243.611 Q1951.83 242.453 1954.54 242.453 Q1959.01 242.453 1961.3 245.231 Q1963.59 247.986 1963.59 253.356 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M1985.12 255.74 Q1985.12 251.111 1983.2 248.564 Q1981.3 246.018 1977.85 246.018 Q1974.42 246.018 1972.5 248.564 Q1970.6 251.111 1970.6 255.74 Q1970.6 260.347 1972.5 262.893 Q1974.42 265.439 1977.85 265.439 Q1981.3 265.439 1983.2 262.893 Q1985.12 260.347 1985.12 255.74 M1989.38 265.787 Q1989.38 272.407 1986.44 275.624 Q1983.5 278.865 1977.43 278.865 Q1975.19 278.865 1973.2 278.518 Q1971.21 278.194 1969.33 277.499 L1969.33 273.356 Q1971.21 274.374 1973.04 274.861 Q1974.86 275.347 1976.76 275.347 Q1980.95 275.347 1983.04 273.148 Q1985.12 270.972 1985.12 266.55 L1985.12 264.444 Q1983.8 266.736 1981.74 267.87 Q1979.68 269.004 1976.81 269.004 Q1972.04 269.004 1969.12 265.37 Q1966.21 261.736 1966.21 255.74 Q1966.21 249.722 1969.12 246.088 Q1972.04 242.453 1976.81 242.453 Q1979.68 242.453 1981.74 243.588 Q1983.8 244.722 1985.12 247.013 L1985.12 243.078 L1989.38 243.078 L1989.38 265.787 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2034.98 237.106 L2034.98 242.037 Q2032.62 239.838 2029.93 238.75 Q2027.27 237.662 2024.26 237.662 Q2018.34 237.662 2015.19 241.296 Q2012.04 244.907 2012.04 251.759 Q2012.04 258.587 2015.19 262.222 Q2018.34 265.833 2024.26 265.833 Q2027.27 265.833 2029.93 264.745 Q2032.62 263.657 2034.98 261.458 L2034.98 266.342 Q2032.53 268.009 2029.77 268.842 Q2027.04 269.675 2023.98 269.675 Q2016.14 269.675 2011.62 264.884 Q2007.11 260.069 2007.11 251.759 Q2007.11 243.426 2011.62 238.634 Q2016.14 233.819 2023.98 233.819 Q2027.09 233.819 2029.82 234.652 Q2032.57 235.463 2034.98 237.106 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2049.49 246.064 Q2046.07 246.064 2044.08 248.75 Q2042.09 251.412 2042.09 256.064 Q2042.09 260.717 2044.05 263.402 Q2046.04 266.064 2049.49 266.064 Q2052.9 266.064 2054.89 263.379 Q2056.88 260.694 2056.88 256.064 Q2056.88 251.458 2054.89 248.773 Q2052.9 246.064 2049.49 246.064 M2049.49 242.453 Q2055.05 242.453 2058.22 246.064 Q2061.39 249.675 2061.39 256.064 Q2061.39 262.43 2058.22 266.064 Q2055.05 269.675 2049.49 269.675 Q2043.91 269.675 2040.74 266.064 Q2037.59 262.43 2037.59 256.064 Q2037.59 249.675 2040.74 246.064 Q2043.91 242.453 2049.49 242.453 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2082.39 243.842 L2082.39 247.87 Q2080.58 246.944 2078.64 246.481 Q2076.69 246.018 2074.61 246.018 Q2071.44 246.018 2069.84 246.99 Q2068.27 247.963 2068.27 249.907 Q2068.27 251.388 2069.4 252.245 Q2070.53 253.078 2073.96 253.842 L2075.42 254.166 Q2079.96 255.138 2081.85 256.921 Q2083.78 258.68 2083.78 261.851 Q2083.78 265.462 2080.9 267.569 Q2078.06 269.675 2073.06 269.675 Q2070.97 269.675 2068.71 269.259 Q2066.46 268.865 2063.96 268.055 L2063.96 263.657 Q2066.32 264.884 2068.61 265.509 Q2070.9 266.111 2073.15 266.111 Q2076.16 266.111 2077.78 265.092 Q2079.4 264.05 2079.4 262.175 Q2079.4 260.439 2078.22 259.513 Q2077.06 258.587 2073.1 257.731 L2071.62 257.384 Q2067.66 256.55 2065.9 254.838 Q2064.15 253.101 2064.15 250.092 Q2064.15 246.435 2066.74 244.444 Q2069.33 242.453 2074.1 242.453 Q2076.46 242.453 2078.54 242.801 Q2080.63 243.148 2082.39 243.842 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2092.46 235.717 L2092.46 243.078 L2101.23 243.078 L2101.23 246.388 L2092.46 246.388 L2092.46 260.462 Q2092.46 263.634 2093.31 264.537 Q2094.19 265.439 2096.85 265.439 L2101.23 265.439 L2101.23 269.004 L2096.85 269.004 Q2091.92 269.004 2090.05 267.175 Q2088.17 265.324 2088.17 260.462 L2088.17 246.388 L2085.05 246.388 L2085.05 243.078 L2088.17 243.078 L2088.17 235.717 L2092.46 235.717 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /><path clip-path=\"url(#clip890)\" d=\"M 0 0 M2122.22 243.842 L2122.22 247.87 Q2120.42 246.944 2118.47 246.481 Q2116.53 246.018 2114.45 246.018 Q2111.28 246.018 2109.68 246.99 Q2108.1 247.963 2108.1 249.907 Q2108.1 251.388 2109.24 252.245 Q2110.37 253.078 2113.8 253.842 L2115.26 254.166 Q2119.79 255.138 2121.69 256.921 Q2123.61 258.68 2123.61 261.851 Q2123.61 265.462 2120.74 267.569 Q2117.9 269.675 2112.9 269.675 Q2110.81 269.675 2108.54 269.259 Q2106.3 268.865 2103.8 268.055 L2103.8 263.657 Q2106.16 264.884 2108.45 265.509 Q2110.74 266.111 2112.99 266.111 Q2116 266.111 2117.62 265.092 Q2119.24 264.05 2119.24 262.175 Q2119.24 260.439 2118.06 259.513 Q2116.9 258.587 2112.94 257.731 L2111.46 257.384 Q2107.5 256.55 2105.74 254.838 Q2103.98 253.101 2103.98 250.092 Q2103.98 246.435 2106.58 244.444 Q2109.17 242.453 2113.94 242.453 Q2116.3 242.453 2118.38 242.801 Q2120.46 243.148 2122.22 243.842 Z\" fill=\"#000000\" fill-rule=\"evenodd\" fill-opacity=\"1\" /></svg>\n"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"plot(1:10, TrainD[:trainAccuracies], label=\"Training Accuracies\")\n",
"plot!(1:10, TrainD[:trainCosts], label=\"Training Costs\")\n",
"# plot!(1:10, TrainD[:testAccuracies], label=\"Test Accuracies\")\n",
"# plot!(1:10, TrainD[:testCosts], label=\"Test Costs\")\n",
"ylabel!(\"Epochs\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Predict After Training"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[32mProgress: 100%|█████████████████████████████████████████| Time: 0:00:06\u001b[39m\n",
"\u001b[34m Instances 60000: 60000\u001b[39m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"The accuracy of Train Data before the training process 0.8812\n",
"The cost of Train Data before the training process 0.3314\n"
]
}
],
"source": [
"TrainP = predict(model, X_train, Y_train);\n",
"\n",
"println()\n",
"println(\"The accuracy of Train Data before the training process $(round(TrainP[:accuracy], digits=4))\")\n",
"println(\"The cost of Train Data before the training process $(round(TrainP[:cost], digits=4))\")"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[32mProgress: 100%|█████████████████████████████████████████| Time: 0:00:01\u001b[39m\n",
"\u001b[34m Instances 10000: 10000\u001b[39m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"The accuracy of Test Data before the training process 0.8629\n",
"The cost of Test Data before the training process 0.3845\n"
]
}
],
"source": [
"TestP = predict(model, X_test, Y_test);\n",
"\n",
"println()\n",
"println(\"The accuracy of Test Data before the training process $(round(TestP[:accuracy], digits=4))\")\n",
"println(\"The cost of Test Data before the training process $(round(TestP[:cost], digits=4))\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Julia 1.4.1",
"language": "julia",
"name": "julia-1.4"
},
"language_info": {
"file_extension": ".jl",
"mimetype": "application/julia",
"name": "julia",
"version": "1.4.1"
}
},
"nbformat": 4,
"nbformat_minor": 4
}