{ "cells": [ { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "# Modeling timeseries data in MXNet: using LSTM to predict AWS spot instance price" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "# Problem\n", "\n", "Amazon EC2 Spot instances allow you to bid on spare Amazon EC2 computing capacity. Since Spot instances are often available at a discount compared to On-Demand pricing, you can significantly reduce the cost of running your applications. In this tutorial we will build a neural network that models time series data to predict the price of an instance in the Amazon EC2 Spot market. By predicting the price, you may be able to make smart decisions on when to launch your spot jobs.\n", "\n", "To simplify the prediction of spot prices, we'll assume the cost at any previous timestep in time to be dependent on previous values. This may not be entirely true but for simplicity we'll assume so and convert the data in to a timeseries sequence. Ultimately we will train a variant of Recurrent Neural Network called [LSTM](https://en.wikipedia.org/wiki/Long_short-term_memory) on AWS Spot dataset to build a neural network that can predict the price of an instance in the next timestep.\n", "\n", "Click to learn more about [AWS Spot](https://aws.amazon.com/ec2/spot/pricing/)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "# Recurrent Neural Networks (RNN)\n", "\n", "Certain problems require you to model information dependency, for example take the construction of a sentence. Now these dependencies can be external or could be based on previous context. We'll explore the latter in this notebook. Recurrent neural networks (RNN) are designed to address this issue. They are networks with loops in them, thus allowing information to persist, making them ideal to model sequences. Speech recognition, image captioning, video analysis, language modeling and many other use case can be addressed using RNNs. Check out Andrej Karapthy's [blog](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) for more details. Lets visualize a few examples of sequence in input or oputput: \n", "\n", "![rnn](http://karpathy.github.io/assets/rnn/diags.jpeg)\n", " img src: http://karpathy.github.io\n", "\n", "A RNN can be imagined as a repeated copy of the same network, each passing a message to the next step, in this case it may be to the same layer. But RNNs have problems with long range dependencies - interactions between sequences that are many steps apart, this is explained in detail in this [blog](http://colah.github.io/posts/2015-08-Understanding-LSTMs/). Also RNNs tend to be deep, that is in the temporal direction and hence more vulnerable to the [vanishing gradient problem](https://en.wikipedia.org/wiki/Vanishing_gradient_problem) \n", "\n", "To address these issues [Hochreiter & Schmidhuber (1997)](http://www.bioinf.jku.at/publications/older/2604.pdf) introduced LSTMs, which now are widely used to model the problems described above. \n" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "# What is Long Short-Term Memory (LSTM) ?\n", "\n", "The Long Short-Term Memory network (LSTM), is a Recurrent Neural Network (RNN) capable of learning long-term dependencies. Instead of neurons, LSTM networks have memory blocks thus making it \"smarter\" given that it can store recent sequences. These LSTM blocks are are connected through layers. Each block contains three multiplicative gates that govern what information enters the cell, what information exits the cell, and whether or not to discard information discard information stored at previous time steps. The gates are illustriated below: \n", "\n", "![memory cell](http://deeplearning.net/tutorial/_images/lstm_memorycell.png) \n", " img src: http://deeplearning.net/tutorial/_images/lstm_memorycell.png\n", "\n", "\n", "### Training LSTM's\n", "\n", "These networks are trained using backpropagation by unrolling in the time dimension. This is necessary given that LSTMs still have a self loop. The unrolling reveals a chain like nature as shown below. This infact makes them very suitable for learning sequences.\n", "\n", "![lstm](http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-unrolled.png)\n", " img src: http://colah.github.io/posts/2015-08-Understanding-LSTMs\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "# Prerequisites\n", "\n", "Familiarity with MXNet, Python, Pandas, Numpy, basics of LSTM networks. It's recommended that you know the basics of RNNs and LSTMs to proceed.\n", "\n", "To complete this tutorial, you need:\n", "- [MXNet](http://mxnet.io)\n", "- Python 2.7 or greater and the following libraries for Python:\n", " - numpy - ndarrays \n", " - pandas - to easily get our dataset\n", " - Matplotlib - for visualization\n", " - mpld3 - add zoom capability to matplotlib\n", "- [recommended] AWS ec2 p2 instance with Deep Learning AMI.\n", "\n", "For setting up an deep learning environment on AWS using Deep Learning AMI, please read [this post on AWS AI Blog](https://aws.amazon.com/blogs/ai/the-aws-deep-learning-ami-now-with-ubuntu/) for detailed instructions. " ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "# Prepare the Data\n", "\n", "Lets start by loading the dataset using pandas. The data has been recorded at almost constant intervals, for simplicity we assume constant intervals. Hence we only extract the time (column 1) and the cost of instance (current bid price) per hour (coulmn 5) which we'll later model as a sequence. As mentioned earlier, we'll ignore the fact that the spot prices may be dependent on other variables and external factors.\n", "\n", "The dataset is a CSV file with over 7000 data points collected over a period of 11 days containing for P2.XLarge in US-EAST-1 and has the following columns:\n", "Time, Instance type, OS, Availability Zone, Current Bid Price\n" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "## Load and Visualize the dataset" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
datecost ($)
02017-05-08 21:42:56+00:000.2500
12017-05-08 21:41:01+00:000.2402
22017-05-08 21:40:05+00:000.2414
32017-05-08 21:42:56+00:000.2500
42017-05-08 21:41:01+00:000.2402
\n", "
" ], "text/plain": [ " date cost ($)\n", "0 2017-05-08 21:42:56+00:00 0.2500\n", "1 2017-05-08 21:41:01+00:00 0.2402\n", "2 2017-05-08 21:40:05+00:00 0.2414\n", "3 2017-05-08 21:42:56+00:00 0.2500\n", "4 2017-05-08 21:41:01+00:00 0.2402" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "(7183,) \n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAikAAAF5CAYAAABa9bp0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzsnXe4FOX1x7/nFnoRREEEFBUVFaXYwF6xRKPRRK+oCImA\nmmiuxJIoscUWFWxRwUb4qVcsCWqMohJRDE1pFsAogmikqhTlApd7398fZ9/Mu7MzuzO7Mzszd8/n\nefaZ2Wl7dnfmne+c97znkFIKgiAIgiAIcaMsagMEQRAEQRCcEJEiCIIgCEIsEZEiCIIgCEIsEZEi\nCIIgCEIsEZEiCIIgCEIsEZEiCIIgCEIsEZEiCIIgCEIsEZEiCIIgCEIsEZEiCIIgCEIsEZEiCIIg\nCEIsiY1IIaLLiGgpEdUS0UwiOijLtkcRUYPtVU9EO9q2+zkRLUodcwERnRz+NxEEQRAEIQhiIVKI\n6BwA9wC4AUAfAAsATCaiDll2UwB6AOiUeu2klFptHHMAgGcAPAqgN4CXAEwion1C+RKCIAiCIAQK\nxaHAIBHNBDBLKXVF6j0B+ArA/UqpPztsfxSAfwFop5Ta4HLMZwG0UEqdbiybAWCeUurSEL6GIAiC\nIAgBErknhYgqAfQDMEUvU6yc3gLQP9uuAOYT0TdE9EbKc2LSP3UMk8k5jikIgiAIQkyIXKQA6ACg\nHMAq2/JV4G4cJ1YAGA7gLAA/A3tdphJRb2ObTj6PKQiCIAhCjKiI2oB8UEr9B8B/jEUziWh3ANUA\nBud7XCLaHsBAAMsAbC7ERkEQBEEoMZoB2BXAZKXUt0EcMA4iZS2AegAdbcs7Aljp4zizARxmvF+Z\nxzEHAnjax2cKgiAIgpDOIPDAlYKJXKQopeqIaA6A4wC8DPwvcPY4APf7OFRvcDeQZobDMU5ILXdj\nGQA89dRT6Nmzp4+PjifV1dUYM2ZM1GYkBvm9/CG/lz+i/L0eeQR49FHggQeAAfbovRgi55Y/4vJ7\nLVq0COeffz6QupcGQeQiJcVoAONTYmU2uNumBYDxAEBEtwPorJQanHp/BYClAD4Bu5cuBnAMWIRo\n7gPHqVwJ4FUAVeAA3Yuz2LEZAHr27Im+ffsG9d0io23bto3iexQL+b38Ib+XP6L8vbp352nXrkAS\n/jI5t/wRw98rsHCJWIgUpdRzqZwoN4O7ZOYDGKiUWpPapBOArsYuTcB5VToD2ATgQwDHKaXeNY45\ng4jOA3Br6vUZgJ8qpRaG/X0EQRDiRLNmPN0skXZCwoiFSAEApdRDAB5yWTfE9v4uAHd5OOaLAF4M\nxEBBEISEUl7O0/r6aO0QBL/EYQiyIAiCECJlqZa+oSFaOwTBLyJSGjFVVVVRm5Ao5Pfyh/xe/ojy\n90qaJ0XOLX805t9LREojpjGfuGEgv5c/5PfyRxxESlI8KXJu+aMx/14iUgRBEBo5ursnKZ4UQdCI\nSBEEQWjkJM2TIggaESmCIAiNHPGkCElFRIogCEIjRzwpQlIRkSIIgtDIEU+KkFREpAiCIDRykjYE\nWRA0IlIEQRAaOZLMTUgqIlIEQRAaOeJJEZKKiBRBEIRGjgTOCklFRIogCEIjRwJnhaQiIkUQBKGR\nI54UIamISBEEQWjkEPFUPClC0hCRIgiCUCKIJ0VIGiJSBEEQSgTxpAhJQ0SKIAhCiSAiRUgaIlIE\nQRAKgMiK+Yg70t1TutTXA1dcAaxeHbUl/qiI2gBBEAShOIgnpXT58EPg/vuBrVuBhx+O2hrviCdF\nEAShRBBPipA0RKQIgiCUCOJJEZKGiBRBEIQSQTwpQtIQkSIIgpAndXXW/Pr10dnhFfGkCEpFbYE/\nRKQIgiDkyebN1vzvfx+dHV4RkVK6JGUEmh0RKYIgCHlidp/U1kZnh1eku0dIGiJSBEEQ8sS86SfB\njS6eFCFpiEgRBEHIk6Td9MWTIiSN2IgUIrqMiJYSUS0RzSSigzzudxgR1RHRXNvywUTUQET1qWkD\nEW0Kx3pBEEqRbdus+SR4UiZMiNoCQfBHLEQKEZ0D4B4ANwDoA2ABgMlE1CHHfm0B/BXAWy6brAfQ\nyXjtEpTNgiAIDz1kzS9fHp0dfvjss6gtEKJAB84mQUybxEKkAKgGMFYpNUEptRjACACbAAzNsd8j\nAJ4GMNNlvVJKrVFKrU691gRnsiAIpc6yZdZ8Urp+Nok/WUgQkYsUIqoE0A/AFL1MKaXA3pH+WfYb\nAqA7gJuyHL4VES0jouVENImI9gnIbEEQhLQ8KUkRKUkdiiqUJpGLFAAdAJQDWGVbvgrcRZMBEfUA\ncBuAQUopt1CwT8GemNMBDAJ/1+lE1DkIowVBEEyRYs4LghAMiauCTERl4C6eG5RSS/Ri+3ZKqZkw\nuoGIaAaARQCGg2NfXKmurkbbtm3TllVVVaGqqqow4wVBaFSYwqRZs+js8ENZHB5NhcRTU1ODmpqa\ntGXrQ0i7HAeRshZAPYCOtuUdAax02L41gAMB9Caiv6SWlQEgItoK4ESl1FT7TkqpbUQ0D8AeuQwa\nM2YM+vbt6/0bCIJQkpgi5ZxzorPDD5WVUVsgREHQgbNOD+5z585Fv379gvmAFJFraqVUHYA5AI7T\ny4iIUu+nO+yyAcB+AHoDOCD1egTA4tT8LKfPSXlgegFYEaD5giCUMKZISUqsR1JiZwQBiIcnBQBG\nAxhPRHMAzAaP9mkBYDwAENHtADorpQangmoXmjsT0WoAm5VSi4xlo8DdPZ8D2A7A1QC6AXgs9G8j\nCEJJYOZJSUqiNBEpQpKIhUhRSj2XyolyM7ibZz6AgcaQ4U4Auvo8bDsA41L7fg/21vRPDXEWBEEo\nGNOTMn9+dHb44eyzgU8/jdoKQfBG5N09GqXUQ0qpXZVSzZVS/ZVSHxjrhiiljs2y701Kqb62ZVcq\npbqnjtdZKXWaUurDML+DIAilRZcu1vzjj0dnhx/+85+oLRAE78RGpAiCICSNnj2TE4silDaScVYQ\nBKHE2LoV2EWKbQhCaIhIEQRByJOtW4EmTaK2QhAaLyJSBEEQ8mTr1vS8I0kZ4fPOO8C6dVFbIQi5\nEZEiCIKQJ0pxBteDD+b348ZFa49Xjj4a6O9aGU0Q4oOIFEEQhALZeWeefvVVtHb4YbEkYygpJHBW\nEAShRGnalKdJuwEIQtwRkSIIgpAnWpT07MlTqYsjCMEiIkUQBKEAiIBLLuH5Hj2itUUQGhsiUgRB\nEAqkfXuemmnyBSFOJDXpoIgUQRCEPNHdPeXlfBPYujVaewQhF0mLmxKRIgiCUAD6CbWyUjwpghA0\nIlIEQRACQESKIASPiBRBEIQ8MV3nTZqISBGEoBGRIgiCUABmd4/EpAhxRQJnBUEQSpjVq4FRo6K2\nQhCyI4GzgiAIgiAIASAiRRAEIU+S9lQqCElDRIogCEIBJLWvXxCSgIgUQRAEQWjkJFVMi0gRBEHI\nE+nuEZJG0s5ZESmCIAgFkKQn1L/9LWoLBMEfIlIEQRBKhIMPTn9fXx+NHYLgFREpgiAIAXDccVFb\n4J/XX4/aAkHIjogUQRCEPDH79087DWjRIjpb8kHS+JcOSeqWNBGRIgiCUABJbfyB5AVRCoWTtP9c\nRIogCEIAJFGsbNsWtQWCkB0RKYIgCHmStKdSO1IQUYg7sREpRHQZES0loloimklEB3nc7zAiqiOi\nuQ7rfk5Ei1LHXEBEJwdvuSAIpUzSPCgTJ1rzW7ZEZ4cgeCEWIoWIzgFwD4AbAPQBsADAZCLqkGO/\ntgD+CuAth3UDADwD4FEAvQG8BGASEe0TrPWCIAhMEjwre+5pzW/eHJ0dQnFJmpjWxEKkAKgGMFYp\nNUEptRjACACbAAzNsd8jAJ4GMNNh3eUAXlNKjVZKfaqU+iOAuQB+HaDdgiAIAJJzEygzWn0Z3VN6\nJEFIm0QuUoioEkA/AFP0MqWUAntH+mfZbwiA7gBuctmkPzI9LJOzHVMQBMEPSWvwgXQxJTEpQtyp\niNoAAB0AlANYZVu+CsBeTjsQUQ8AtwE4XCnVQM6PMJ1cjtmpIGsFQRAMkuJB0Zj2rlgRnR2C4IU4\niBRfEFEZuIvnBqXUEr04yM+orq5G27Zt05ZVVVWhqqoqyI8RBKGRkQTPitndM2kSMHp0dLYIyaWm\npgY1NTVpy9avXx/458RBpKwFUA+go215RwArHbZvDeBAAL2J6C+pZWUAiIi2AjhRKTU1ta/XY6Yx\nZswY9O3b1/MXEAShNEmCKLHTrZs1X14enR1CcQna4+f04D537lz069cv0M+JPCZFKVUHYA6A/1W+\nIO6/OQ7AdIddNgDYDzxi54DU6xEAi1Pzs1LbzTCPmeKE1HJBEIRA0I1/Urp9WrUC/vEPa14oLZIm\nrOPgSQGA0QDGE9EcALPBo31aABgPAER0O4DOSqnBqaDahebORLQawGal1CJj8X0AphLRlQBeBVAF\nDtC9OOTvIgiCEGtOOomnw4ZFa4cg5CIWIkUp9VwqJ8rN4C6Z+QAGKqXWpDbpBKCrz2POIKLzANya\nen0G4KdKqYXZ9xQEQfBG0p5KNeXlHJtSFrkvXRCyEwuRAgBKqYcAPOSybkiOfW+Cw1BkpdSLAF4M\nxEBBEAQHzG6eJImWigqgvj5qK4RikZTuSDuiowVBEEqQigopMCjEHxEpgiAIAZC0J9XychEppUiS\nvH2AiBRBEIS8SVqDbyKeFCEJiEgRBEEogCTHpIhIEeKOiBRBEIQSRAJnS4ukdUdqRKQIgiDkSZI8\nJ3YkJkVIAiJSBEEQCiBpGWc10t1TmiRNWItIEQRBKEFEpAhJQESKIAhCQCTpKVVEipAERKQIgiDk\nSZJEiZ3ycgmcLSWS1h2pEZEiCIJQABKTIgjhISJFEAShBBGRUppo79/YscAPP0RrixdEpAiCIOSJ\nvbsnSd0/IlJKlyVLgBEjgGuvjdqS3IhIEQRBKICkdfNoJE9K6aL/9x9/jNYOL4hIEQRBKEHEk1Ja\nJFVMi0gRBEEIgKTdBMrKgIaGqK0QhOyISBEEQciTJMWg2CFKtv1CfiTtPxeRIgiCUABJrYIsIkVI\nAiJSBEEQSpCyMhEpQvwRkSIIgpAnSb7JiyeltHCKmZo2rfh2+EVEiiAIQgEkNeMskQTOlipanC5Z\nEq0dXhCRIgiCEBC68V+wAPjww2htyYV4UkqTpP3nFVEbIAiCkFTcGvzevbOvjwMSkyIkAfGkCIIg\nFEDSunk04kkRkoCIFEEQhBJEREppkVQxLSJFEAQhAJJ2ExCRIiQBESmCIAh5koQqyEoBkyZlLpe0\n+EISiI1IIaLLiGgpEdUS0UwiOijLtocR0XtEtJaINhHRIiKqtm0zmIgaiKg+NW0gok3hfxNBEEqJ\nuHtQ3n4bGDcuc7l4UoQkEIvRPUR0DoB7AAwDMBtANYDJRLSnUmqtwy4/AngAwIep+cMBjCOiH5VS\n5uW4HsCeAHQzIpekIAglxZo1zsvFk1JaJLV8Q1w8KdUAxiqlJiilFgMYAWATgKFOGyul5iulJiql\nFimlliulngEwGcBhmZuqNUqp1amXy+UqCILgH7Oxj6tHZeRI5+UVFcC2bcW1RRD8ErlIIaJKAP0A\nTNHLlFIKwFsA+ns8Rp/Utm/YVrUiomVEtJyIJhHRPgGZLQiCACD7E2ocnlj/+1/n5U2aAHV1xbVF\niAdxFdRORC5SAHQAUA5glW35KgCdsu1IRF8R0WYA74M9MU8bqz8Fe2JOBzAI/F2nE1HnoAwXBEHI\nRpxFQGUlsHVr1FYIxcYunL/8Mho7vBIHkVIIh4O9MMMB/JaIhukVSqmZSqmnlFIfKqWmAfgZgDWp\nbQVBEEJn8+aoLXBHRIoAANddF7UF2YlD4OxaAPUAOtqWdwSwMtuOSimtAT8hok4AfgfAIY4dUEpt\nI6J5APbIZVB1dTXatm2btqyqqgpVVVW5dhUEoYTI1Z2zeTPQpk1xbPGLdPeUFm5dPE8/DTz1lP/j\n1dTUoKamJm3Z+vXr87AsO5GLFKVUHRHNAXAcgJcBgIgo9f5+H4cqT70cIaIyAL0AvJrrQGPGjEHf\nvn19fLQgCKVKtirIcQ5M/egjLoQolB6bAkjG4fTgPnfuXPTr16/wgxtELlJSjAYwPiVW9BDkFgDG\nAwAR3Q6gs1JqcOr9pQCWA1ic2v8oACNTx0Fqm1EAZgL4HMB2AK4G0A3AY+F/HUEQhHh7Kt59N2oL\nhCiYOBH48ceorfBOLGJSlFLPgbtqbgYwD8D+AAYaQ4Y7Aehq7FIG4PbUtu8DuATAVUqpm4xt2oG7\nfhaCvSetAPRPDXEWBEEoGLfunorU41+cRYpQumzcmLns0UfZGzhxYvHtyUZcPClQSj0E4CGXdUNs\n7x8E8GCO410J4MrADBQEQXDAqZtH5yCJs0g54ghg2rSorRDiQG0tMCw17OTcc4HPPgOuvz5amzSx\n8KQIgiA0JrQnJc4xKT/5CdCuXdRWCMUiW24Uu0dw1KhwbfGDb08KER0K4AgAPQC0Aaee/xzAe0qp\nGcGaJwiCkAz0TUCpZHT3lJcD9fVRWyHEgbIYuys8mUZETYnod0T0BYB3AZwHHiJcB44XGQTgXSL6\nIrVds9AsFgRBiAluMSmVlTwVkSLEEft5261bNHZ4wasn5XMAcwBcBeCfSqla+wZE1BzAqQAuAHAF\n0gNdBUEQGiVuMSmAiBQhGbgVoYwDXkXKSUqpT7JtkBIuLwB4gYj2LdgyQRCEhJKEmBQRKaVLo6vd\nk0ugFLq9IAhCEknyEOTychZRcSiCKISPKUw2bIjODr8UHC5DRNsT0TFEZE9rLwiC0OixZ5xVKhkx\nKdtvz7Z+913UlgjFZt68qC3wji+RQkTDiWi48b43OF5lCoAviGhgwPYJgiAkjjh5Ulq0cF7epQtP\nv/66eLYIgl/8elIuBhcE1NwErrfTBsA9AG4NyC5BEITYk6u7Jw5udTcbdeHDJKVIF0oPr0OQjySi\nowDsBqCt8f4YADMA9AWP/umZWndkaBYLgiDEiGyjey66qKimONLQ4Ly8aVOebtlSPFsEwS9eR/d0\nT03LAOwEoB6czK0ewKbU+gpwFeJdARA4n4ogCEJJYMak6ORYbgKhmLh5Upqlsllt3lw8W4ToSNKI\nHhNPIkUp9VcAIKKLwflPngBwHIC3lFITUuv2BPCNfi8IglCKNDQA33wTtRUWbiJFe1JuvRU4+eTi\n2SMIfvCbFn8UgEmwYlOONdZVAfhXQHYJgiDEHicBMHp0skTKv/9dPFsEwS++AmeVUm8D6AbgYAC7\n2/KhvAwgJnUTBUEQioPdjb5oUfr7Dz4ATjopunwkubp7BCHO+C4wqJRaDw6StS9P0MhrQRCEcNhh\nB2v+pz8FrroKmDqVA1SjEAa5RiAJQpzxOrrnMD8HJaIB+ZkjCIKQHEwBoD0qPXtay+rqok+R7yZS\nkhpIKeRHUv9vr909TxLRVCK6kIjaOW1ARO2IaDARvQPgyeBMFARBiC/2xt9MnrZ1K/DWWzwfN5Ei\nCEnAq0jpCaAGwJUA1hLR50T0LyJ6OTX9HBxIeyWAZwBIgUEh1jz+ON9cJEeEEDRVVdb8qlXWvJxr\nguAfr0OQ6wGMBTCWiPYBcAQ4T0obAOsBTAQwTSm1MCxDBSFInn+ep1u2WKMcBCFIBg0Cnn7aet+p\nE7BiBU+LhRcvygEHhG+HIORLPoGzCwGIGBESjQ5g3LDBSg8uCH7JJgJatcpcVl0N1NSEZ4+dXCLl\ntNOSG6sg+COp/3PBVZAFIYnsvz9Pe/SI1g4heXz7bXo3jr0KsqZJk8x9i93lk0ukEMUjK64guCGD\n0ISSZMcdeSopwQW/dO0K1Na6C4ADDgC++85ZpHz/fbi22bHbaH9fVgbU1xfPHkHwi3hShJJE4lCE\nfKmt5enWrc5CpUUL4MgjnUVKsSsO2+378sv092Vl3j0pRMntMhCSi4gUoSSRYZlCoWzZkl5MULNt\nG+dGcRIpxfZa2M9z7UHUSHePEHeku0coSXTD3LVrtHYIyWXLFj6P7CKlrg6orHQWKXV1xbFNo0XK\nE08AHTtmxmCVlYlgF+JNXiKFiI4DV0HeETZvjFJqaAB2CUKoaJGydWu0dgjJZevWdJGiu0I2bGCR\nUlmZuU9U3T0VFcApp2Su99PdIwhR4Lu7h4huAPAGWKR0ANDO9hKE2DN9Ok+/+04CB4X86NfP2ZPy\nxReZnpSdd+bpDz8Uzz7AEilusSTS3SPEnXxiUkYAuEgpdYhS6gyl1JnmK2gDBSEIfvgBeOUV671O\nslVXByxbFolJQkLZfnuerlyZLlLMtPf2mBSdPPC004pjoyaXSMmnu2fJksJsEgQ/5CNSmgCYHrQh\nRHQZES0loloimklEB2XZ9jAieo+I1hLRJiJaRETVDtv9PLWulogWENHJQdstJIORI4HTT3ceArp+\nffHtEZLLTjvxtGXLdJHyy19a29g9Kf37A926FTfbLOBNpHjxpGzYYM2vXVu4XUJ8+Pvfrfk99wQ6\ndAAGDozOHjv5iJTHAJwXpBFEdA6AewDcAKAPgAUAJhNRB5ddfgTwADg9/94AbgFwCxENM445AFxH\n6FEAvQG8BGBSKq2/UGLoRtapgRV3t+AH7TFp25aDZ7VIMbt9nAJnoxi+G1R3z4oV1rxZQFFIPmec\nASxYwG3kp58CffrEKwt3PiKlGYAriegdInqAiEabrzztqAYwVik1QSm1GNyltAmAYxCuUmq+Umqi\nUmqRUmq5UuoZAJMBHGZsdjmA15RSo5VSnyql/ghgLoBf52mjkGD2SUnTTz7h6YEHAnvvzfMyukHw\ngx6h8803wDvvWOLkPOPRzW10T7EJypPy2mvWfLET0gnhs//+QOvWUVvhTD4iZX8A8wE0ANgP7PnQ\nr95+D0ZElQD6AZiilymlFIC3APT3eIw+qW3fMBb3Tx3DZLLXYwqNC+1mP/NMfhKurwd22IGXmbEE\ngpAL+zDiZ5/laXm5tayiwhrdc/rpxbHLiaBiUvbbz5qfM6dwu4T4QhSvB7d8CgweE7ANHQCUA1hl\nW74KwF7ZdiSirwDsAP4etyiljJqj6ORyzCL3ChdObS3XCunYkYc9rlnDQXsDBmSOLBCcMYd+Pv00\n/6Z77MHvR460RvsIQi5WrACuuAK477705WZXoulJ6dbNWl7sxj8oT0r79ta8PWutIIRJ0pO5HQ6g\nFYBDAdxNRCuUUuMKPWh1dTXatm2btqyqqgpVVVWFHjovdB9wy5bpN9uWLYs/pDGpVBth1ePH8w1F\nJ3KbNy8Sk4QEohR7UqZMAY45Bnj7beDUU3nd5MnWdpWV3M8P8LZANDEpWoC4Pcx4jUnRYqddu/T4\nFKHx4dWTUlNTgxpbSe/1IYxC8CRSiOhv4GHHG1LzriilfubThrUA6gF0tC3vCGBljs/Smv4TIuoE\n4HcAtEhZmc8xAWDMmDHo27dvrs2Kjj0RVLETQzUWpk7lafv2QLNmwFVXRWqOkCD0NTdsGI+AuPtu\n4OGHedmllwL338/zFRXA0qU8v3hx8e3UaAFidkWZeO3u0dt07iwiRWCcHtznzp2Lfv36Bfo5XjsL\n1gNQxny2ly+UUnUA5oCTwwEAiIhS7/044ctTL80M85gpTkgtbzRIIrLcfP658/LmzbkLLU79r0K8\n+e47nu61Fw/XHDfOEgDdu1vbVVYCRxzB82Y8R7HPtVyeFK/dPdruTp2461loHHS0P8YjoTEpSqkh\nTvMBMhrAeCKaA2A2eLRPCwDjAYCIbgfQWSk1OPX+UgDLAehnlKMAjEwdR3MfgKlEdCWAVwFUgQN0\nLw7B/sj48cd4DReLI2Yj/M9/WunBmzbliHZ94xGEXOhzxYzR0Fx+udWtWFkJHH44z++/P0+j6O7R\nDzFBdfe0aAFs3hyMbUL0fPFF1BbkJhYxKUqp51I5UW4Gd8nMBzBQKbUmtUknAGYpuDIAtwPYFcA2\nAEsAXGXGoyilZhDReQBuTb0+A/BTpdTCkL9OUfnwQ6sxFJwxh4KefDInK1q7loNnd9vNcssLQi6y\niRRTCGzdCnTpAvTtC1xzTXFsc0LnB3Lr7vH61Ky3adq0+EUShfBwynmTSE9KMVBKPQTgIZd1Q2zv\nHwTwoIdjvgjgxUAMjCmSsyA3WqToTKETJrA3ZeNGTsglafEFr2QTKSZ1dXxDj3q47l6p8ZHZPCle\n0N6Wpk2lKKdQXGQAa8KRmJTc6KfIBx7gqe4eq6/nJwkJQBay8frrfDP/4QcrCDZXF6vbzT+qJ9RC\nsypru5s0EZHS2ImbJ0VESsKRnAXe0R6V/v2Bm28Grr46c1i3INi5916erl8PjBrF87nyEzmJlChi\nUjSFpirQN63KSnkwEoqLb5FCRBcSUVOH5U2I6MJgzBK88tvfRm1B8igr45tNmzbsSdm0KWqLhDij\n858kucbTxo2F7a9FSkWFiJTGTmPwpDwJoK3D8tapdUIRiUN9kCQjnhQhG//8pzVfV8dlFU48Mfd+\nbnVQit34V6SiDs0qxvkgIkWIinxECsHKmWLSBXnkSREK45BDorYg2YgnRcjGDTdY81u2AN9+C+y4\no/v2v/sdTwcNylwXRXePHr1RkWWIhJ/RPZWVUuuqsZNYTwoRzSOiuWCBMoWI5hqvBQCmIbOgnxAy\nX30VtQXJpmVLzvsgT4eCEx98YM0//jjw7rvAU0+5b3/XXdzAuw35LTY6Zf/w4c7rvQon05MCJLvr\nS0gWfoYgT0pNe4OrCZuhWFsBLEMjH+4bR2pro7Yg2bRsydNNm+JbqlyIB/fcE7UF/ikrA448svBu\nYS1Slizh6YcfAr1917wXkkDcPCmeRYpS6iYAIKJlAJ5VSm0JyyjBG23bynDAQtHu8KuvtmqwCIIT\nuqDgZZflf4woGv8gupm03brExGWXAf/+d+HHFeJJnERKPjEp/wKwg35DRAcT0b1ENCw4swQvHHUU\nJ3MLofBkyaCfDB95JFo7hHhSXs6FAwErvb2uauyXKIcgF4ru3hk7lqfT/VRVExJF3M7TfETKMwCO\nAYBU5eG3ABwM4FYi+mOAtgk50ImlZs2K1o4kc+ihUVsgxBWlOFapb19ghx0sr2VlZbR2RYF+st5h\nh+zbCY0WdxLOAAAgAElEQVSDpHtS9gMXAQSAXwD4SCk1AMAgABcFZJfgAT08cuBA4LPPorUlqfTv\nz7V8AGDmzGhtEeKFrlFTWZleiO+OO/I/Zpwaf42bTZs3AzfdxKN59DZxe8oWgidu/3E+IqUSgI5H\nOR7Ay6n5xQB2CsIowRs7Gb/2nntGZ0fS0TkkbrwxUjOEmGGKFMAaets0I5WlN+LW+Ofi5pv5mqis\nFJFSasRJTOcjUj4BMIKIjgBwAoDXU8s7A/g2KMMEi/POc17uVMFS8I9240tiPEGzejWwahXPa0+K\nFilxGV4cBNlEx5o11ryIlNIhbv9xPiLlGgDDAUwFUKOUWpBafjqsbiAhQOTmGS66m6dPn2jtEOJD\nx47A7rvzvBYpWsxmS4yWi2I/oRbyefvuy9MLLhCRUmok2pOilJoKoAOADkqpocaqcQBGBGSX4BFd\nil3In0MOAY4/Pj1xlyBodHePzknUvHl+x4nqBp/v5+oYnPbt070q0i3auImbEM3rmUApVU9EFUR0\neGrRp0qpZcGZJXjlo4+A7t2BXXeN2pJks9tuEjgrOKM9KZs38/t8RUrS0E/TSgEjR/L8smXA9tuL\nd7exk2hPChG1JKInAKwA8G7q9Q0RPU5EEiUREocd5pyOu7ISOO64+KnfpDFuHGfRjNPFKcSDuXN5\nqj0pzZpFZ0sYmInaHnwwc7lSQHU1zx94YPwykgrB8ve/A6+/nnu7YpFPTMpoAEcBOA3AdqnXT1PL\nEpg4Oh5s2cIX/4ABPDVf48cD8+ZximsnWrcGvvuuqOY2OrTI00/LgqDp2ZPPD12IshBPSpxv7qed\nBvzmN9Z7U6TooopNm8oDkVBc8hEpZwH4pVLqNaXUhtTrnwAuBnB2sOaVDlpkzJjhvH7TpvRRBXff\nbc3vvju7YYX8eTFVdarQkvZC4+OEE9K7e/L1pMTx5m7aZC8auCWVaEIpHtlEZD0oxVlsCYXRt2/U\nFqSTT0xKCwCrHJavTq0T8sBLFV5TpJiNRNOmnNNBqXg2hEmgbVuefvUVj+wQBI0WJY21u0ej40zq\n67mt0UOwdeZd3f5IG9O4GTDAyhEUB/LxpMwAcBMR/e9SJaLmAG5IrRPywItIMYPV3n/fmu/UiU+q\n1auDt6tUaNWKpwcdFK0dQvzQ3a66uF4hIiXOHgh9DfyQqm+fTaTE+XsIhdG8eby6vfMRKVcAOAzA\n10Q0hYimAPgKwIDUOiEPdKKobOy8szV/yy3WfJcuPF3l5N8SPLHddta8DEUW7Hz9tTWfrychCg+E\nHzGh4050O+ImUvweV4gnY8Y4L2/e3PIaxoF88qR8DKAHgN8DmJ96XQugh1Lqk2DNKx2cPClr16Z7\nR7p1s+bNLol27XgqwbP50727Nf/qq9HZIUTPf/9rzedb8ThO5BJHWnDoNuWhh3j644/WeunuSTa3\n3sqjQE1++1vnbSsqvHn2i0W+eVI2AXg0YFtKGvtJcffdnI/AxHzfurU13749T7//PhzbSoHKSmD+\nfKB3b+DEE6O2RogS02ty7LHp60xvZmNDi5T77gPuvTd9dI94UpLN9denv0/S/5dPnpTfE9EQh+VD\nieiaYMwqPewi5eij099vv336E4w5HFl3VYgnpTB0ZlFBcGP69ML2j9vNwWxTzC5PIFOk6HIAEpOS\nbCZMyL1NnP7ffGJShgNY6LD8E0ha/LyxD/+zv89WL6S8nG+wuraIUBhxukCF4pOtO6OQHClx7yYx\nH3yuvda6DhoaOJBSV3+O+/cQsmP30NuJ2/+bT3dPJ/BwYztrAOxUmDmli92TYhcchRQ1c+LFF7nP\n+cILgz2uICSdsERKkrjzTmtYvlIcG6cDa4XGT5we1PLxpHwFHt1j5zAA3xRmTuliFyk6kRLAjeZ1\n1wX7eWefDQweHOwxk46+OcXpAhXiRaEiJUnn1vr1PN26lUf66JiVuD1pC8ESt/83H5HyKIB7iWgI\nEe2Seg0FMAYFBNMS0WVEtJSIaoloJhG5ZqwgojOJ6A0iWk1E64loOhGdaNtmMBE1EFF9atpARJvy\ntS9s7CLlqKOs+YYG4JJLeH7sWGDqVOdj5NMAehn6LAgCYwaP+iWuQ5BzbbNpU7onRcR84ydO/20+\nIuUuAI8DeAjAF6nXAwDuV0rdno8RRHQOuO7PDQD6AFgAYDIRdXDZ5UgAbwA4GUBfAG8DeIWIDrBt\ntx7cPaVfu+RjXzEwRcohh7g3hsOGpQsYTb4NoB5mKMTvCUKIhsZ2Hvj5Ps2aAStXAiefbC2zixRN\nnG5kQnDE7fz3HemglFIAriGiWwD0BFAL4DOl1Jbse2alGsBYpdQEACCiEQBOBTAUwJ8dbKi2LbqO\niH4KLnq4wGbumgLsKhqmSHGr3xMGP/5o9T0LjDS+QqlSUcHdOvrhpXVrFinS3dN4+MZDUEac2sB8\nPCkAAKXUD0qp95VSHxciUIioEkA/AP9Lm5QSQm8B6O/xGASgNQD7INxWRLSMiJYT0SQi2idfO8PG\nrJUQdiNgfpZ4Uiyk8RWAcM+DODX+gPt33W8/nh56KKc22LBBunsaC7myycatHcxbpARIBwDlyCxa\nuArcReOFqwC0BPCcsexTsCfmdACDwN91OhF1Lshan6xcyQnCiCxx8OSTVj0Q/Ro0qPDPUgqYMwe4\n6y7glVd42fLlnOZ9/Hjgs8942dy51j66Tke2Y06alDkkOkj0Z+hG75VXoo2VkcZXCIMoGv9nnwUW\nOiWMyMGYMcAbbwD77AN8/DEvsxfejOo6+e9/gZkzo/nspLByJfDoo8BPfpKZgDCs83DVKuBXvwr+\nuAEPbC0+RHQegFEATldKrdXLlVIzAcw0tpsBYBE4z8sN2Y5ZXV2NtrY+kKqqKlRVVfm2bydjUPaI\nEcDjjwNDh2Zut3Il0K8f8Le/+f4IANaJd+CB1rKPP+ay2+Zw5okTgXPOSf/cbLz5JnDmmcATTwBD\nMlL4BcPf/w6cdRbw3HPAnnsCp58O3HEHcE2RUwPG7QlCiIZnn43agmDJdY2bSds0TZoAJ5zAqQo0\nuks66uukZ09g40Z5mMjGqFHAY485r9u4Mff+Xn7bmpoa1NTU/O/9woXAkiXrPVronTiIlLUA6gHY\ndDo6Ash6eRHRuQDGAThbKfV2tm2VUtuIaB6APXIZNGbMGPTt2zfXZhk0awbccw9w2WXO6594Anjk\nEed1774LHHGE74/MSm1tZr4VU6AQpacAd0IPQxw6lF9hNAw6nf+6dZa9114LnHqq5XYuJtL4lTYt\nWoR37GKeW369n0plCpB166z5Xr0yt48CLzfZUqdTlj6IXJXevYpQ+4P7RRcBS5bMBUdvBEfk3T1K\nqToAcwD8r/xRKsbkOACuSaiJqAo8yuhcpdTruT6HiMoA9AKwolCb3diyBbghq48GuCJVJ/rYY7n0\n+4QJwE03BS9QgOwn26mnAh06cNT+xx+7d/sU+6nJvEH06mWJpGIQ9ROiEA/CuvkW+/zy22XqJFLM\nJJJdu/JUrpP4Y5b4sJf78PL/5XMNbN7sfx8vRC5SUowGcDERXUhEewN4BEALAOMBgIhuJ6K/6o1T\nXTx/BTASwPtE1DH1amNsM4qITiCi7kTUB8DTALoBcHGCBcO332Zf//DDPH32WWD33YELLgD++Mdg\nPtt+YmU7GS+8EGjZkqu89uqVXrAw2zG//LIwG53IFohnVn4uFuJJKW3CjL8qJtor2cEtkYMNJ5HS\npAlPn3jCWiaBs/FHKS5j0K0bx4oMHOh9XyK+j/n9fydO9Le9V2IhUpRSzwH4HYCbAcwDsD+Agcbw\n4U4Auhq7XAwOtv0LOMutft1rbNMO3BW0EMCrAFoB6K+UWhzeN3HHXqF4hx2CPb5uOEyPjL3BMcNs\nKiuBZcuAt41Osi0exmh17563iRl8/TUH+ZoXg/3pb8OG4D4vF/KEKADh3nyLeWPX1/O4ce7bmOe8\nk0jRT+F9+mTuKyIlvijF4vTLL4F27YDJk73vu2wZT91iWopNLEQKACilHlJK7aqUaq6U6q+U+sBY\nN0Qpdazx/hilVLnDa6ixzZVKqe6p43VWSp2mlPowaLv32YdvtCZOT2ItW7p7K4LEbDjMocZA+ol6\n+unA3nunr8/lBbIfv1DOPRe4+mrgk0+sZXaR0rJl8RtDaXyjY8oUvqai/A8OPzy6zw4S7X5v1szb\n9tm6e8zrshhi/pln0kdECv5w+i81rVpl3/e7VCKPefOCtSlfYiNSksqiRXyjNRvV227j6fjx1rKK\nCmvY3LvvFse2Qw5Jf9+kCbuAN2/mjLamFwUAlixhj8+tt7q7vE85JTj7dAM4Zoy1TDeGH30EPPQQ\n53EZNiy4z8yGeFKi59pr+ZqKkrKQWsVin19apOjqxW6Yo3vs3123A+byYnT3PP44T597Lvt2gjPZ\nRIr9vmBHe8/iUjJFREpAmEFDo0bx9HajSACR9YQYRpAswMfO1nCUl/MJqBste96D664DRo8Grr8e\nmDbN+Rj//GcwtgLAeeelv1fKenKqqODh00Dx3Y7iSYkOHUcRt//guuuAl16K2gp/6O4er56UhobM\nG5t+KCl2bNjiVKf88OHF/dzGQjaRkgsRKY0U+7A4pazuk1zKNQi8nJD2ekD2fb78Epg9m+efeip9\nnQ6g88vXX2fG42icvDX6wqioAPbf31oeVuS4iXhSoufDwDtk/eN0rv3pT9xFWijFFF9ePSkapxvb\naadZ8Q2aYnhSdOp2yYidH05eMcDy8mfDqYvPC3fc4W97r4hICQi7SOnQwRIpZmR8lDgVLfziC2t+\n+XLOMglwpeXPP7cauh12YE+QX7p2Bdq3d17n1MiZIqV5c070BgD33+//s/Mlbk/xpUiU/0G7djx9\n4YVgj1tsEazTnzdv7m17r0/fYYuUYse/NEacvGKAt5QO+XpSWrTIHO4cBCJSAkKPQjn3XJ5+Z1QR\nyufmng9KZR8+6SRSdD2O445LX/7FF0CPHjxUGQC22w44/3zvwxntTJqUucxu69dfp4sUADjjDE4i\nd8stwNq1CBVpEAXAOg+KlUjw+++Be+8N/qavRYrX5HSFdBEEiVlbplhtZ2PD7b/0I0KD+sxCEZES\nEL/4BU/tOU+uuqo4n69PjmwixanLpmVLbiTffDP9BLMfp0ULfiLLVZzKjTPPzFxmb5T/9CdLpJiK\nfJddONlc0MO23RBPSvRE+R/ozw6jwXX6XtdfD1RXs+eyEJYtY0Gv09d78aTkGoKcbZ+w/iPTK33W\nWeF8RmPH7b+0P4xmw+/5LyIl5ugGplWr9IDQsPrp3NANlBNmHSGT7bbjk6tLF/d9mzfnPs58G6b+\n/Tmz7qBBHLm/YYOzoNKlIEyvTxguREFww2lESxC4NeD6/C7UU9i9Oxcv1cVKvXb3mKN7/NxkwhIp\nZkE8eWDID/t/qTOhH3988T4zKESk+OCHHzh7XzbateOuiSFDePuwhjO6kU2k5Ap+3X57nuqqpyZD\nhrBwqKvLLyNnx47AjTdy/oNf/YoTyzk1QDpgznRRb9rk//PyQTJpxoco/4OwRIobOmbL7CIuBJ35\nM4g8KU7EoUtIyI79v7zxRv/XVFz+ZxEpPujfP7Nwk70BaNUK2G03DpZt2bJ4tgFWTIrTEGcvDdVL\nLwGvvgrsu2/mupNO4u9VV5e7KKET9fWZSYQWLMjc7swz2VZTpKwwqi2JgBDCJszuHid0kke3+ll+\n0QVOa2tZaHn1RMalu8dk6dLwP6MxEkV8UVjng4gUH2gPw7Jl1oiduNT50Cdkfb1zo3TllbmP0a2b\nlRfh+ec5eHXECH7frJnlaXEbUuxE58483boVOPTQ9HVPPpm5fX19ZoCvma5/RWjlIcWTEjWjR1vz\njdWT4vS9gr6Z6ONt3szXrdfjxyVwVnPSSe75moTsBPFfSkxKguneHfjlL9k9m617JQrq69Mrl2qu\nvdbfcc4+m70md98NvPYax63ofAs64ZYXtC3r1nGXz5FHZt/eSaSY36fQ4EIhvowcWZzP2biRG1O3\nJIE33sjTsMSD3ZZXXuH5QoSZ2SVqBs567erRnx8nT8rJJ1vdv07Iw4Q7UXlSRKTEjI8+4gZBj+j5\n85+jtQfgp0AnT4rXXAl2WrbkJxogvyQ/+qlUj88nyp5av6Ehu0gJM6mbeFLiQ5j/gY79uPhi5/XT\np/M03wSGfvj1r4F//YvnC/nOS5ZY87pbdfPm3Nd9HEf3ACyuWrfmByW3B8G4PSDGCREpJYq9AvDR\nR/N01iwWLMV6EnRDqczuHj1818m74hd9DK+Nw/r1HL/SvDmnuf7Pf3j5q68CM2Zkbt+nDx/b7mY3\nbY9LqmbBmR9+iH8cQTaha55f9vizILDf2CdMcF/nB7NKsT6OV09KvqN7wqRXL8tzW1HBKRLsxKWr\nPY6ISClRtEfBzg8/cOKnYo/kMXGKSWnXjpMhBRXA69eT0r8/T7U9779v2WmPT9E4dfeceqo1H6ZI\nEU9K4Zx1FgdYr1njbz/7DSfM/0B3jfTsmblu9WqehlGnx2zAv/462AbdfHDQv6UXT4pJXDwplZXA\n4MHp7ZZT1m7xpLhTiGCIi1DViEjxgf2iHDiQp2bAX9TU11s38u+/D/aE0+LBq1DQ1Wx1Nl47V1zB\n08suAy65hOedRMpPfmK56MWTEm90WQWdydgry5alvw/jBqgUd63owpVOXgadYsDM1RE0K1dyuQgn\n+4LAryfF3C8OIkXbceyx1rLTTsvcTjwp7kQVOBsGIlJ8YP8TJk/mobIHHxyNPU40NKRnZg1SpORb\neGroUGvedLWPGcP2Pvggv9fdVU7p+/P9bD+IJ6VwLr/cml+50vt+OkBSl5UIg+HD0zNumsPclQLG\nj7dsbtMmPDvWrXNeHrRICcuTEjbbtvGDTevW3MYCzoUGRaS4I909JUrcb176Jm8+BYYhUry6WXXt\nk+22s24+pkghsuwz0/o7iRS/XhwhGvQwdQD47DPv++kRY7oLJoxr7dFH09+btr73HicsvPNOft+2\nbfCfD/D3sgfk6pF3fob2249pMm0aX0dJ9KTodAM6Q+qJJ3K3j1MOGenucUeGIJcoxchxkC9mTIoZ\nGxOlJ0UHHn77rWXT/PnO265ezeueeAL48svCPzsfxJMSLH5y2tTV8VQHSxYDMyD70095qs/TMBIx\n6vPLHrumA/BNL5Qf7NfEnDnsMUqiJ0WfB2YXeqtW4knxi3hSShSlMrt2nC6eKNGeiFGj+H2UMSm7\n7cbTSy4BevfOvu0LL/DU7cZWDJEiBIPuKvHzpKtvTtrLEIZQPOQQni5bxsPgzZucfoLXAiJMsWT/\nbvr3yjcO5oMPMpcRsScl1/cg8j+6J0wxr6/vjh2tZS1bOrez4klxRzwpJUocnjRyoWM69JNgGCLl\ntde8bd+0KQ8lPOSQ3NU33RJracrK+LuIJyX+6PPEz01Ed/eEmZtEKU7CuMsufD6Z9un5bdt4XRBD\n9t1ssJ9f7dtzOQjdPeqXF1/MXNapE/+mSevu0de3+fu7dfeIJ8WdJNyrvCIixQdx/+PNwFN9owhy\nWLR+Knv8ceDtt73Zo8k12uOoo6x5t6HeFRXiSUkC+pzzcxP55BOe6mDWMG6ADQ2WbeXl6fZpkVJX\nF54XxX5zv+46/t577cXlI7JlV82G6YG5+Waedu7MIsWP6ItD+6YfgMxcT9Ld4x+p3VOixOEidsMM\nPC0rs0SKnyepXDRtClxwAc+bwwO92NWlC6fYnz3beTuzUdJVYe2ELVLi+t8mjXw8KTprc76Zkb1g\njhwrK3MWKZs3hx8Xoxvz447jPEZAYSLFrMs1aBA/ELRrx4LLj0jZts2bBylMT8qFF/LUfLiSwFn/\nFHKvyvd/le6eGLBtG99M27WL2hJ3dEOsM1nW1gZ7fDNDZi7sJ/vIkcBBBzlv68XjUyxPinT3FEY+\nnhSNnxvgpElWfh0vmCKlvNw5AdrcueH+/0pZOX/M36dzZw4wdxuenI3ttrPmKystAbZ1q/cKyAC3\nFV5EYhg1jcaMSV+m8+0ALBqffx646qr0bcST4k4hgkF3vfoV6yJSYsAzz3D6+3vvBXbdNWprnNEN\nsR5FE8ZJc8wxQFWVt229fr457NhtH/GkJAM9tNzPTaRLF383VIDjOB55xPv2ZnePmycFsOpMhYUO\n4H32WWuZfpg47zz/x+vRw5qvqOBr6T//4artfm40XkWKJkgxd8cdPNUeWrMQ6auv8vTuu9M/U0SK\nO0GIFL9eeBEpMaC+nnMZXHgh1yd59dX8hw2GgVJWQ6yH8P3978F/TkWFNRojG368OOJJaTwQZQam\n5qKigjMQ59OVMGWKt+2yeVLMxlUPCQ4aewNeXW3Na3Hy2mv+vENAuqhq0oR/+6ee4vd33ZXbJjND\nrR9PSpDXyerVfHPs14+/w5lnWuvMmJtZs6x56e5xpxDBoEe6iUhJKKZr9ZRTgPvui84WEzNPSnk5\n8NOfhvdZlZW5Rcrs2ZzzZMECb8eMgydFCA67CLDz1VeWx+XHH3lYsJnczw+XXuptu2wxKXV1HPuw\neLG3oPB80Wn3ASseBUhPHvftt/6OWVcH7Lsv31Tat+ffVvN//+f9OH49KUGz1178n9gLuZrVqufO\ntebFk+JOIYJBJ9zMR6SEgYgUj+hS6HvsEa0d2dDDG8vLrRwlYdhbWZlbLOi6PV4xPSlujY9fkZKt\nzLsTMgQ5OHI1kLqQH8CjN4D0p34//4Gurp0LM5uxfXTP1q18g99rL++fmw/6ezllVdaYZS28sG0b\nF3asrc383c8/3/txamvTSwW4EdZ1smwZcM89VmI9jZlYz4zZEU+KO1F4UgDxpETKxIk8NZ+E4sba\ntTx9/32e3nef1Z8bJBUVVr+lG2bKcS+YJ7e2385XXwG33GJdRLno3j1edZVKAa83Lqf4k48+Cjcu\nyMzGbO+OqqvzHxPjFyLrM19+2X07t4Kcbmzbll30eGXDhui6e7JhPvCY1bXFk+JOEJ6UfAJnwyA2\nIoWILiOipURUS0QzichlHAhARGcS0RtEtJqI1hPRdCI60WG7nxPRotQxFxDRyfna16sXT2fOzPcI\n4UJkDWHcuJGnl18O7Lln8J/VpInV3TN7Nn+2vYqt2zBiN8yhj+ZTthPjxuU+nlLAf/+b7h7OhXhS\ngiXb72j3iFVWpicz8/If7LADP/l7rbicq7snzERyGv2Z2WKw/JQTALwPHc7G559zOv0o2rdctpsx\nKdLd441CRIpu27VoX78eGDAA+P3vgREjrG5Z89W7N8dpNlpPChGdA+AeADcA6ANgAYDJRNTBZZcj\nAbwB4GQAfQG8DeAVIjrAOOYAAM8AeBRAbwAvAZhERPtkHi43X3/NF1MHN4tigD6p/Abe+cUUKXqo\noE7GpdE3mV//2tsx27QBnnyS553KsgNWEHAuEQOw21jjp9CdUBwmTeKpTj725ps89dPINTTweeOU\nQ8Nte7fAWb/DdfNFizMnkfLEE+xi9/p9NPX1zjd6PwHAuvrzwoW5tw1azDdvbnVPO6Gf6Hv1ApYv\nt5ZLd487QeRJ0fsfeywwYwaPwBo71nmfBQs4/qnRihQA1QDGKqUmKKUWAxgBYBOAoU4bK6WqlVJ3\nK6XmKKWWKKWuA/AZAPP2djmA15RSo5VSnyql/ghgLgCPt810Fi5kr0QxnrbyRTeAYTe2TZpY3T26\nobA3utoWP6OfLroI+OKLzGq1mjPO4KHfXp6gzJwKXp8OxZMSHLkaK/0bX345B86aGYfN9dloaGBP\nillZOxv27h67JyXs62bOHEuMOQntIUOAoUOBTZv8HdfuSRk2jKdmwKkbenSPn6rPYdyITGG2j+0x\nUndBlZend/WKJ8WdIGv3OHWv20XlL36Rvk+QhFShwjtEVAmgH4Db9DKllCKitwD093gMAtAawHfG\n4v5g74zJZAB5jXvZuNF/F0axcap7EQbl5dYIhKVLeWr2FRdiiz2y306rVv6fNHMNwxTCIZvQ2Gkn\nPjfsN0c/QrGhgW9gDQ3eujzsQ5DNm9y6deGLU/Pz3GpZPf20/zwt9piUXXbhqZ9Kzvph4x//8L5P\nkL+Xaetbb6WvGz6cu27btUu/lsWT4k4hIsW+n91LDvBAkjPOYM/fpZdm/mdBEgdPSgcA5QDsIamr\nAHTyeIyrALQE8JyxrFOBx0xj0aJ0V2PcMJNnBVmvx4lHHmGPx5NPAn/9Ky8bPDh9m7AEk1tFVBP7\nE5aX7iFAPCnFZMuW7IF5Bx9sZWZ1Q4sUwJs3xezumTkTmDfP6lOfOJHfF4uddnJergPOf/tb4MEH\ncx/nscfY82L+lnq0lI5N84K+Xr1UYg7jOrnkEk7oB2T+Ni1bcvftwQenBxWfdZY136OHJGM0CcqT\nku2BcNIkTkh45JHsJQSsbsMgidyTUihEdB6AUQBOV0qtDeKY1dXVaGt7xJs9uwqAxzSrEVEskbLf\nfpzNcqjRGXfEEdY8kdVQBi1SWrXK3vhecIGVyMqOPHkVBy/5TtxEit5vyRK+YWe7EeruHn08fc65\nYXb3OAVUP/RQ9v2DQnsfnZg4kUtH6PxLuWK6dJdOt27WshNTQwjcSlA4oa8NL6OEghYDSnE7YeZ3\nceKYY9LfL1/ON8VOnTjwF2CPixeh1dgJKrHanXe6r6upqUFNTQ0AM29W8Oma4yBS1gKoB9DRtrwj\ngKy6jIjOBTAOwNlKKXsKppX5HBMAxowZg759+6Yta9IE+MlPcu0ZLVqkhP1EMXNm5g1B1zPSTzpa\ngQfdzz99OudzGDEis9ECOCOw5uWX2U6davull9hF6YZ4UoIl2++4ZYtzHgb7uWt20dgxPSlehqWb\nx7rtNuAPf7DWrVrlfZRQoWQrqXHggcDee3NSOYC7ftziRcwRUmZ3yd57+z+H/YgUTZDXiZc2y6kt\nufFG9uz26sVD2F94gTMXlzpBxqS4UVVVhSqjPspRRwHvvjsXHL0RHJF39yil6gDMAfC/XtpUjMlx\nAI8ox4kAACAASURBVKa77UdEVQAeB3CuUup1h01mmMdMcUJquW+6dOGLP8789788DduTYjaIWqzo\nJEtmzQ0geE9Knz48deonBdKFy7778oWjh2Hbi5gJ0fHYY85Vf+3L9BOyE2aGVL/dPdXVwKhRfM2s\nXl08geKFhx+25m+4wZq3e6gOP9yaL/RhQIuUqKsg+2XpUrZd2z99Ont5Sx2zTlUh+BE677xjdfsE\nSeQiJcVoABcT0YVEtDeARwC0ADAeAIjodiL6q9441cXzVwAjAbxPRB1TrzbGMe8DcBIRXUlEexHR\njWCJ56GnN5Og/vSwIOJhYno+bM49l6faY6K7YOxp8IMWKa+n5KhbsjjzSXDXXfk/+/RTblDNkgZO\nxKnxbex8/bXz8o8+Sn8/cqSzINXnug7Y8+pJ0ddws2Y8/LlzZ/8ZXsPGHDr85ZeZ67dt44cCs45N\nIaMOlbICZuPcxgHWg+IBB7DX6Y03uI1ZuJCF2nPPsVeFiAs5+onLaUwE1d0Th/MhBiYASqnnAPwO\nwM0A5gHYH8BApZQeM9IJQFdjl4vBwbZ/AfCN8brXOOYMAOcBGAZgPoCfAfipUspDJoBMsrmd40Yx\nbrL2ETPz5nHjae9DD1qk6DgGt9pB+rt/9FE8LrBSxSxcZ0d7Pcw4Jo1Omqh59dX0JG8aHTyth+vm\nEikNDex50d7GKPjww8ykh27orhydT8a+7ppr0pflK1L0jUx/jltAr9M+UYj5999n8TFtmhVoq7n9\n9vT3s2dH+39HSVDdPU7HaN26sOP6JTbNuFLqIaXUrkqp5kqp/kqpD4x1Q5RSxxrvj1FKlTu8htqO\n+aJSau/UMfdXSk3O1764e1JMinESdemSWefjhRcy3X1BixTt1s6Wlr9VK+cbWy7Ek1IcdBeNUxFM\nr4kI9ROyDuzLNeJLZ3GNcjh6r17W8OBclJcDp55qnZPm96uvz8y6XMh1tmED1z9q29bbsGV9fZh1\ndIpFq1YcE9e6NfDnP1tt3aBBwJVXsm0bNgBTp/LyXOU7GitBJHMDMo9xzTX+MyIXSkJuu8VHKQ6u\n033kSfCk6Ah/rw1hodiTTq1YYbnhNUH/ZkR8zHffzSxEBojAiJK5czmQMZdXQ5+n552Xuc5sFGtr\neWpP9DZxIvCzn/F87948NeMznNBBun/6U/bt4sT++1vXsuklcBIHXhPaOaGD3U84wdv2ukvXjJeJ\ngh49+LeYNInjePS507q1FWwsIiV/7J6Uzz8Hbr3VX/6dIBCR4kJtLXDddVYmPbM/O47oPCl+i0IF\ngR7ZU1GRGfcRRnxMkyacgtkcBRHEZ4onpTD6pYL6daFLt4yge+zBIsOta2HkSPYiNGsGVFXxdbdq\nFf8/V13F/70mV5yRHXs20zhj5gS69VZr+axZVtxKWRlw2GH5B/Vv2GCJyksv9baPfvDQIjJKysrY\nI2f3HuvuLxEphWHe83bfPZoH9RjfdqNF36j+/W+emiMD4kpQJ6Zf6uq4hsrWrXzzOfTQcD/PHMnQ\nsycP09SJ9qL6DUqZhgYOhO1kpEnctInd79OmWctqa3m46L/+ld3bd/fdViDnihXA229bx7777vTK\n3kTA+efzOdHY8uC0aWN5OcycJ2edxd0ZhxzC3/m99zK7Xr1ijiRyGhLuhJcu16jRD2teK6ZHhfZW\n3HdfsOdvmDEpxUZEigvmU2D79sno7in2DVrX2DnoIG4Utm61aqCEmZrf7trebju+6RXaIIknJT/+\n/Gega1d+2TnySL6WlOIbqY45sQc9uqFjC7IxZAifd++959nkRNCuHZ/TtbXA8cdnlowwvSv5Yl6n\nXq8fPcw/7IeRQtAegDhfy+Y95re/5f8iqPZbREoJYJ5A33+fjMDZhobinlQXXcQu/jvvtIoOapEy\nbpyVnyRo3J7gNm4UT0qQNDR4K3an08mvWsVD0+37tG6dWV3XSdA44RS3YkIE6LyL2cof2Cu7JgEt\nSh59lGNSli61sskC7vV//LB0KQ/DBqzuulzstx8HP3sZCQRw9XKn+DFNnIVEmLiN9Jo1iz2Ghfwu\nhbSDPXoUfowgifltNzo++MCa33XX+HtSdExKMU+qigr+nQ46KFOkDBmSvWEKir/8xZoPKolTqTaa\ndnbdleMicsUeaMGxfDl3GejRO5pNm6zRVtOmcW6Ts8/2ZsPTT3N3jsbuLWnVyvq8devSs7AmnX79\n+BozM6hqD5TXINdcdOnCJQjee8/fqMDmzb3FpNTWchfwSSdl3y4ON8Ni8uc/c4wHwCMizevh0EM5\n9ipb+YRcFCIwdHkIong8mMfAhHgy2RisvGwZN7RxH3MfZXDv0qU8Gurf/w6/H/jSSznHxtln87wW\nKsccE2z1z7C4807Odhp3dC0Vp8ywJnvsYc3bM5/aA7kPP5w9AH5+6//7P/5fleIg0Zdesta1b88C\nmQgYNgw4+WTvx407zZpZXg6Njj0J0kvZrBn/rn738TKi6LPPeGqvkl4s4vrAYea56dMHeP75TK9h\nIQ9dhbSDZnxTHMSjiBQXnIKYosyx4IWNGznYLkpWrsxdvbZQ/vIXHoL8/PP8fvhwngZVWynshu3a\na9OHws6YwV0lceWLL7KvN2vL2EWKWabgyiuDsef00635du24IdXelDBLxkdBR6P62G23WXFgujxE\nVDRv7k2k6ID2oGt45SION1c3dPty7LHcDaptffrp9O2c8gj5+Yw4/wZ+EJHiglNG01deKb4dXiGy\nulqixnTPF4PychYoZWXJ8KRolOK05wMGpI+MiQv6ad0p66kbds/J1Kmc1+Thh4F77gnMtP+lstdd\nsF5iZ4DkNdzvv2/NH3ig5aX0GtMTFs2aeevuOeccnkaR+C1fdEr9sNDxjuefz6MTTbLVqvLDxx9z\nN16hxOF6EZHigr1ve4894l8FWal4iJSgLjQ/TJ3KVY+DoFgu4rKy9Iq4bkUTo2DaNM5CCqTXiMmF\nDqL99lvOAvrtt5xraMSIYO17/fX04c25iKvbPxePPGLNEwFPPsnzXoNcw8Jrd48pHh97LDx73Mj3\nf589O1g7TLJVnN59d7b5llvyL3o5axZ3v9uD1ZOKiBQX7CIlDgFEXohKpJhF0XQ20GKiCx0mwZMy\nYIDzch0DEgfMatZ+KptOT9Utb98eeOqp/PN35KJv3/QssxdcYM2//npyRYmd4cOB66/neaWAwYNZ\nHLgV2CwWDQ3pSfWcsP8HF18cnj1B4ZaAMEiyiRRN5848Ws2tRpkb06dbQ8ODELLiSYkxOvJaN9Zx\n+LO8EJVImTLFms+VojwM9E0qiEYm7BuceeM2C+rFOS7FS5fP0Ue7VzcOmwkTrCGxJ5+cnIcKL+hC\njN27czsURVZpO/PmsZfkiCPSvT0mOlWAmxdt7Fj+PrnqLhWTYiQE9CJS9Cguvw8upndrwgR/+zoR\nh/teI7qUg0WPFT/llGjt8Io+mcJMopYN86ZQ7NoOgDXEcfPm+HtSGhq41PwTTwDvvMO1UJo358y5\nccBJLJ15Zu79/vEPK1YkCuKcAbUQTjyR4z/MUVRx4b33OEHft99mrtM3zGOOYQ+Q3asWdBegSb7X\nclxEii5zsPvuQIcO3o+tt/3FLworAREHcaIRkeKCfpp+802eJiXwK8qYlMGD03M6FBP9dBmHeiK5\nqK/nvCFDhvDolP33Z7uvuCIe3RQHH2zN6zo8SUDH0Gi0aPnmGyvgNE6Nrx+8pqyPig4dMrtz9LXY\nvDkXgrT/9maMX1jeFL/XUzFEig4l8OJJAVgA+i0gOXKkf7vsxKEtAkSkuPKHP/BUDwOLsyveZObM\n6D57/Hjg3nuj+WzdiNfWxrvA4E47sffEbah4lP+fRg8bPf54jn145BFuUONeB6V9e27MdRft+PH8\nfuedOe24EBxVVZnL7IGxOq9U06b8sp8/uj4TEJ+Rk2GJlPff54eSjRuB6mpelk2YlZUBzzxjeZ+u\nuy4cu3Kh20SzW7rYiEhxQT+VDR4crR2CN7RIKaS7pxisXMlT+4gZnV3SLai2WPz+9zytrATeeIPn\ne/TgxvvCC533icsTF8A3Q2338OFW/pQXX4zOpsbIM8/k3ua223j65ZeccG/bNvZI//ijFfOnMdP9\nR0lYIuXBB1k0t2ljBRznGkZeVWUVmPTq0Qy6/IP2zNuTChYTESkOmH3bbdrwUK7XXovOHi/op5So\nYlLiwuLF+e8btifFTHJn9ziZQ5Gj5I47eFpXZ/0eOqHYc89xVWK31PNxEYdNm0afR6SUOPdcnu6/\nf/ryv/2Np+edZwXWn3kmD7XXCQL32os9LkF7CuIWk2KWWdl5Z76GzBGRbpSXsxfQa/HMoEWKThp4\nww3BHC8fRKQ4oIezanr2zF17Imp00GVjql3iB12Mbf78+Nws7Xz/PU+nTLFGbJjcfXf0GYOdMJ96\nO3e2nq7s10mc0I2rrhmkieu5kWRqanj64YfO61u2tDxaFRXptZ2WLOFzqkmTcGyLS0yKzlmiFI+A\n81MHbu+92RvldzhyEOgRQl6LSYaBiBQH4tz45iLMTIlxZrfduKHzUyTNTtieFJ2yfa+9nNe3acN9\n1sXI1eCE+b3NxHLNmqUPMQeAM87g39qL2z8Kxo/n0gkffZSs4N/GxMEHW7WUHnuMxUqrVlYsxu23\n8/8TJ4IWKe++a3mYjj8+v2Po7tZc5SmA4DwpSnF36bhx/L5bt8KOVwgiUhzYuJGnL7wQrR35cPfd\nUVsQHVu3cndEXAM89ZDLnXd2Xt+mDTcOUeWN0IGPbdpkDl889lirRhJgFfkbNKg4tvmlXTvLW7X9\n9lZK/rgM824MTJkCjBmTufyVV4CjjuKsrbqbXOdDmTQJ6N+fl519tjXUNi4ELVKOOsoSYmYdJj9o\nT+ayZbm3DeMBa++9o807JCLFgdWredq3b7R25EOUUdhxId8bUZCelEcfBV59lb0i77zDWVBzobt6\nokqItmgRT936v3/1q+LZEjS6+0d/R6Fwjj02c9SUUlwA8t13+b05lLZdu/Rtw7zx5XstBylSVqyw\n5nv1Au67L7/j6LxT557Lv5kXT2uQ3Zpu3XjFQkSKA7/7HU+32y5aO/LBrEhbauyyS9QWWAwbxnkg\n9t6bA+S02ztbALaOoPeShKnQxrS+nrtqdCM+d671VOz2+QcemJnFct99C7OjWBx1FAdnBlWJWUjn\nzjt5qkf0aB5+2Jq/8cb0daaAiQtBihTdvXPHHXyjz7eUgX54WbeOr9e77nLfNihPysCBnBDu5Zej\nrwcnIiULpXzDTyJe3KHZCMqTYu7/2Wfp68yaOHYOOMCaz/bEv24dByDqgMV8GDuWu2qmTuX3ZimD\nbEF9F1zAT3JK8Q0/KYHaZWXAn/4Ufc2bxoo+j3SdIY3Z9frrX1vzK1aEFyxbCKZIIco/PuyHH6wE\nk2Y3aT7Yf6drrwWefz77PoV6Ul5/nauXn3ZaYccJAhEpLsyb17jqfwjFQ3cXam6/Hbj/fhYsuQru\n6SHU2bwpOrD72Wfzt1EH4embyG67ed9XN4CVlRwHFNe4FKF46MKSdk491ZrX7en55wOdOoVvUz7Y\nPSlealY5UVXF95DHHgvGI3/ttTzV8Wy/+AV7PzV/+AN7kg88kN83pgdsuQ07cN99nMZZKE0K9aSc\nfz5PzzyTh65ffTXwm994q72ia0YBwB//6LyNbkgLeVrSgaTNm/PQ6MWLgcsu4zpCXqmosJLQCaWN\nmSOleXMW2bfckpnOX6nc1ZODIKiYFC8japz44AOOgxoyJL/97dx+uzV8WceG9evHQcpEvF5nigbi\nk3cpCESkOBBFFV8hGI49FjjrrPz2DSrYTIuRmhqOQfHjkSsrY1EDcCPvFAQchEjR7Lgjjz6orwd+\n+cvMhFzZiKO7XogGsyuntpaHsNu7fqLgsce4jo3ZbfPVV+ndlBUVfC0RAXPmpO9/1VWca2fNGn+f\nu20bj14KwxuvhwUDHKSs6dcPeOopzlLbmHoBGtFXEQQeFlno0PFCPSm77MJ1ZHSftF/MxIHbbZdZ\nXEw3sEGIlOuv56BSwH8iuWuuSX/fmBpGwR+mYDVvolHz4ovA6NEcZ9WiBV8z3bpxV+Xll3PXrOk9\nuegiaz+dBfeTTzjbsld0+v+wBl4QObdRs2Zx12shuaLiSGyaFSK6jIiWElEtEc0kooOybNuJiJ4m\nok+JqJ6IRjtsM5iIGlLrG1KvTeF+CyHJON30leKAS3sAbDaefjo9Bb5f7NloX345/X2QIkWnLgcy\n66nkomnT9NFKca/UK4SHHuI9e3ZmNeQocMqVZK+Q/sADVu6SWbOsPEE//znws5/xdX/TTbxsk487\nx9df8zXqJ84rH+rquIvnvfeAUaP8ZbFNErEQKUR0DoB7ANwAoA+ABQAmE1EHl12aAlgN4BYA87Mc\nej2ATsYrRoNUhbhiz7w6ahSw557eR7J8/HFhn19RYeVGAIBzzklfH6RI0Vx2WX77xb1chFAcdtmF\nr5uDXB8ti0vXrpzd9owz2BPZ0MBxWHfcwde0mU/qiis4O+4vf8nbmV4THRd29dU8/D6Xl/XHH60S\nHWEX5auo4O952GHAzTeH+1lREguRAqAawFil1ASl1GIAIwBsAjDUaWOl1JdKqWql1FMANmQ5rlJK\nrVFKrU69fPYsCqWE003ffPoyvQ7ZOPxw4JRTCrPl++/dczYEGZOiefDB/Pddty6auiKC4EaTJpw5\n/O9/Z48fEQ+Zv+YaDur98EMWHEqlF/t0uqYeeICngwdb8Slvvsnb2hMc/uMf1vwOOwT/vUqRyEUK\nEVUC6Afgf9VBlFIKwFsA+hd4+FZEtIyIlhPRJCLykCZLKHXMpyVTKNiTUbmxbp3/rhM7lZUc43H2\n2Znr8vGkzJjBT3Y6KNdk1qz8bNS0bSvVt4XGy69/zd6V1q25UOCOOwInnsjrHn+cY1w2b+ZuFzN+\nbMcdo7G3sRG5SAHQAUA5gFW25avAXTT58inYE3M6gEHg7zqdiEJ2wglJxemmb4oUL2USVq/m7p49\n9wzGJh3UunAh8O9/c3/3kiW8zI9IGTCAE2jddRfw6afp6w4+OBhbBaGx8vOf86gZM4maHsb/1Vcs\n1I84wgq8HTSo8caIFJtG+/yjlJoJYKZ+T0QzACwCMBwc+yIIjjh5Ujp18iYK/vAHnuZb8dTOOedw\njhWn9PN+0lUfcICVA8WstxJVnSBBSCJnn83Bqi1b8gi+3/yGM7MuX26NBgJ4KLAQDHEQKWsB1AOw\n14jsCGBlUB+ilNpGRPMA5EypVV1djba2lH1VVVWoqqoKyhwhhmTzpDRt6i3u4vHHebrXXsHYlK1f\ne+FC78fZc09+Ely61Cp22Lq1e0VmQRCc6drVmq+stJI3apESVAK3uFNTU4MaW22O9SGUGY9cpCil\n6ohoDoDjALwMAEREqff3B/U5RFQGoBeAV3NtO2bMGPRNYAlkSUIXDE6elKZN3Uf3PPssp8E2CTKo\ntaEhMwfJKacA//wnJ1/zUqW0oYFHO5hcemlwNgpCqfPGG9zdWyplIpwe3OfOnYt+/foF+jmRi5QU\nowGMT4mV2eDRPi0AjAcAIrodQGel1GC9AxEdAIAAtAKwQ+r9VqXUotT6UeDuns8BbAfgagDdADxW\npO9UVL74QqLJC8UuLGbMsGrpZBMpunqwRseMBGnXmjXsYi4vZ8ExcyaLlI8+Yg+JTsSmFNf06Ns3\n/fs0NKQPawaAW28N1k5BKGVOOCFqCxoncQichVLqOQC/A3AzgHkA9gcw0Bgy3AlAV9tu8wDMAdAX\nwHkA5iLdS9IOwDgAC1PLWwHonxri3Ojo3j3zSVnID6U4GdSAAdznDHCiMt3dM3o0J63S7LuvJRB3\n3DGcJE4dOnBNlCZN2Jajj+b6IAAH7T31lOVxOfBA4LbbMr+TKVKGDpXAPkEQ4g+pQnOANyKIqC+A\nOXPmzElkd49QONr7cM45HBCnOeEE9lq8/rpVfHLpUi7kdeKJnMRp2jReXsz08Nm6lV55hYP75s/P\nTNb2n/+kFzMUBEEoFKO7p59Sam6u7b0Ql+4eQYgVpkAB2HuxcmV6dezu3TlB1JtvWnlNis369RwU\nu8oYwD9uHDBsGHDaae77FZrHRRAEoRj8f3t3Hi5HVeZx/PtLAkHWicQh4GDCIpFdiQturAIDiqKg\nCIbJmERgRHFEVkdFHuKDIIIDI4pEwjLAzOMMIGgCKuDIJgiExRDCYmQLYQsGSALR5Mwf7+mkUunu\n231z+3aF+/s8Tz+5XXXq1Kk31dVv1TnVVYnuHrMqmjx5+U9qFx+DDst/M+Ws/NSo007r37bVrL9+\nDNirmTo1np3ywgsrl3300bgD4eab/TBAM1s9+FBlVjJ6dPxM/IQJy39S+4474Ec/igGsS5bEeJCd\nd47yp5wSj4Pvlh12WP73vvvGvxtuGD/e9pOfRKKVUlw9mTTJd4GZ2erD3T1mBRdcAB/9KGy88YrT\nhw6FI49ccdqtt8Zg2qFD+699jTzyyPI7fGpGjKjGE2nNzHrLSYpZQfmBYc0MGlSNBAVgyx5/otDM\nbPXj7h4zMzOrJCcpZmZmVklOUszMzKySnKSYmZlZJTlJMTMzs0pykmJmZmaV5CTFzMzMKslJipmZ\nmVWSkxQzMzOrJCcpZmZmVklOUszMzKySnKSYmZlZJTlJMTMzs0pykmJmZmaV5CTFzMzMKslJipmZ\nmVWSkxQzMzOrJCcpZmZmVklOUszMzKySnKSYmZlZJTlJMTMzs0pykmJmZmaV5CTFzMzMKqkySYqk\noyTNlrRI0u8lvadJ2RGSLpM0S9ISSWc1KPdpSTNznfdJ2rdzW1A9V1xxRbebsFpxvNrjeLXH8Wqd\nY9WeN3K8KpGkSDoY+D5wMvAu4D7geknDGywyFHgOOBW4t0GdHwAuBy4A3gn8HLha0jZ92/rqeiPv\nuJ3geLXH8WqP49U6x6o9b+R4VSJJAb4KnJ9SuiSl9BBwJLAQGF+vcErp8ZTSV1NK/wm83KDOo4Fp\nKaWzUkqzUkrfAu4BvtSB9puZmVkf63qSImkNYAxwQ21aSikBvwHevwpVvz/XUXT9KtZpZmZm/aTr\nSQowHBgMPFua/iwwYhXqHdGBOs3MzKyfDOl2AypmLYCZM2d2ux19Yv78+dxzzz3dbsZqw/Fqj+PV\nHserdY5Ve6oSr8J351p9VWcVkpQXgCXARqXpGwFzV6Heub2ocxTA2LFjV2G11TJmzJhuN2G14ni1\nx/Fqj+PVOseqPRWL1yjgtr6oqOtJSkrpr5LuBvYErgGQpPz+nFWo+vY6deyVpzdyPfA54M/Aa6uw\nbjMzs4FmLSJBub6vKux6kpKdBVyUk5U7ibt91gYuApB0GrBJSmlcbQFJOwIC1gXekt8vTinVrjf9\nO/BbSccAvwQOIQbofqFRI1JKLxK3LZuZmVn7+uQKSo3iRpruk/RF4HiiS+Ze4MsppbvyvCnAyJTS\nHoXyS4Fy4x9PKW1eKHMg8B1gJPAIcFxKqc8yPDMzM+ucyiQpZmZmZkVVuAXZzMzMbCVOUszMzKyS\nnKSsAkk7S1oq6doOr2cHSZdLekLSQkkzJB1dp9x++eGMr0h6TtL/SBrZpN6RkiZL+lOu9xFJ386/\nAlwst6mkX0paIGmupDMkDSrMHyppiqT7Jf1V0pV11jUlxyrl19L8emBV41NnXW+WNE3S05Jey3E7\nV9J6pXI7SPpdfgDl45KO66HejsVL0kU5HuflMrV4LSnELHUiXqW27ybp7hy3hyWNK82fmGM2L79+\n3exhoHmZXSVdLWmOpFclTZd0aC/WvU3ep2cX9qHzSmVma8V41fazc1clLk22rWmbS2U/m9uy0uej\nVG5Ax0vSBpJ+mLf/NUkPSfrHJnUO9Hh9TfGw3YWKY91ZkoY2qbMT8Vqq+t9JswsxKr5aj1dKya9e\nvoiHF/4eWAyM6IP6BpHHCZWmfx44G/gwcXvXocAC4IuFMlsArxMPXdyMeKjib4G7mqxvH+CnxK3a\no4CPEb8jc0apTQ8Qt5Rtn5d5DphUKLM28ENgAjAVuLLOutYDLgXuyvHakfiNnG92IF5/BxwB7ARs\nCuwOzAQuL7XnGeBiYGvgMzmmE7sRL2AKcev7POIBmusBfw/8A/AS8ARxW3yv49VCPEcBrwJnAKOB\no4C/AnsVylxKPFtrB2Ar4MLcvo2b1HsScAqwc943jwb+BuzX5rrfDZxe+L96sRavQpmNc3seJ+7U\n25P4HaYPdyNepbJPEp/JlT4fjteyMmsAfwCuzdv/NuK4t73jVbfN/wQsAg7OsfoI8BRwZj/H62ng\n6Drr2pA4jtVebcerIwe7gfAC1iEebrgHcDNwYmn+rsBSYD/iqc6LiN9o2bZQZlze4fcHZhBf3m9r\ncf3/Afym8P5A4PVSmY/lnW9wG9t1LPBo4f2+eeccXph2RG73kDrLT6F+klKO1yW5bZv2U7y+TNz9\nVXv/L0SSNKQw7TTgwTb3gz6JV/73qrzthxTmHwJMz7FYWojXPjmOL+XtuBbYvLDcDcC5pXUNJxLZ\n3Rtsy+nA/aVpVwBTm2z/IGA+MLbNuP0CmNzbdQOv5Fg1iteVRAL1A+DhbsYrx+gW4mSj7ufD8Vr2\n/kjiTsyWj1kDPF7nAr8ulTkT+F0/x2s2dZKUOuV+ADzcTtvc3dN7BwPPppRuBCYTZ8X1nEH87su7\ngeeBayQNLsxfm7j1egKwLXHW3YoNiCy/5jZgoaTxkgZJ2gA4jNiBl7RYJ8RViGK9OwMPpJReKEy7\nPq9/2zbqLcfrU0SS9WSpXJ/HS9ImeX2/Km3X71JKfytt1+gcu1b1ZbwSceArPv17PPHFNgqYU4jX\nOsD3iatFexBnJ1cVlpsMHKIVu6IOA55KKd3UYP070/5DOdchzn7nNSlTT3n/7c26oXG8lF+fI65+\ndTNeJxP7/pQetqWZgRKv/YmE/DxFV+kDkk5Sobu0RQMlXtcB71LucpW0OXGi98setqusr+LVQw8W\nIwAACV5JREFUUN72WrxatyrZ6kB+EWdGJ+a/30Rk0LsU5teuDBxUmDaMuIx4UH4/jtiZt2tz3R8g\nMvA960x/njiTX5rbuH4b9W4J/AUYX5h2PjCtVO5Nuf596tTR6EpKMV6bEV/I3+hkvIjLsQtyvVOB\nNQrzrgd+VCq/da5/dH/Hq/DvcOIq0qbE7/ssyO1aCtzQpC3Dc5lt8vuhxOXqYjzvLca8Th2zgBNK\n0/bNMRnaYJnziDPfNdvYzz6Tt3Hr3q6b5We69eL1ZuIL4kYadMX2V7yADxFddcOafT4cr2Xxmpnb\ndwHwLuDTtNktPJDilafVumIW53k/7ML+1eOVlLyetodG+EpKL0gaDbyX/Iu4KaVFwH+x8tWURIxZ\nIZd7ifjP37pQZnFK6Y9trHs74Grg2ymlGwrTN8vTJxNXIXYhdtz/bbHetwLTgP9OKV3YantarHuF\neAGfJT4UW5WK9nW8/pU40H2c6K+9uN22N9KpeKW4AvMLomvgn4kzogOI/8snCuvfUjGY+jFJ84mD\nRCK2k5TS68T4kfG5/E7ElZy+jMGJxIHngJTS4haX2Z04O52Ylv86dK/Vi1dKqXZGuBWRMM7tRrwk\nrUt0a34h78u9qWPAxCsbRDyt/vCU0vSU0s+IH+Q8spWFB1q8JH2SiM/hxLHuU8DHJH2jxeX7NF49\nGE+OVzsLVeVn8Vc3E4DBwJOSitNfl/SllNIrbdS1qNWCkrYhLsH9OKV0Wmn2EcCfU0onFcqPzW18\nb0rpzib1bkKcFdySUjqiNHsuUL57Y6PCvFaU4zWEOMv4lKSjOhWvlNJzRHfQw5JeAm6WdGxKaQ6N\nH0AJPWxXP8RrCjHmKAFfJPqdn2DFX1j+BXEgnAjMIQ7uM4A1C2UmA9Nzez8P3JhW7l4rt71eTF7O\nB9llJB1LdLvtmVKa0cpGSdqVeD7XV1JKl/V23XWU4wVx9WoTln+59Xu8JL2DOPu+VssPFIMAJC0m\nrtjNblT5QItXfv8McSJS3NdnAiMkDUkrds+uYIDG6wTgJ2l5V+KMnByfD0xqtlEdjFe9ddUG9R7Q\n7rK+ktKmPD7iMOAY4g6V4msOMbBqWXGib6+27DAi+36wF+vdlvhinJJS+ladIoOIgahFSwvzGtX7\nVuAmYkT9+DpFbge2lzS8MG1vYrBkj9tRJ14Tc7s+QQfjVcdg4iBTG99yO7BLabzL3sCslNL8RpV0\nOl7ZdcTBbQjRrbcFcedPrQ1vJuIyKaV0U0ppFjGKfgX5itNdxFnWIfTcF1x7KGfR3pQeyinpeODf\niO6r6a1skKTdiAP5cSmleu1oad0NFONVG3c0kkhop3YxXg8Rd3i9k+XHiGuIz/GOxN0+dQ3QeAHc\nSnSjFo0GnukhQdmNgRmvhsf9QmK8kg7Hq57xxBWyqW0v2U7fkF8JIhNcBKxXZ953gTvz37UxFvcT\nA6m2A35OZNtDcplxwLwW1rkdcUXgYiKbrb2Kd5B8iNhZv0l8yHciPlyP0Xg8wSbEeIJf5b+X1V0o\nM4jom51G3Ha6T97ZTi3VtTVxMP45MZK9dlBeIV7EJc/bOhyvfYlLs9sSB5OPEmc1NxXKrE8kSRcD\n2xADe18FJjSpt5PxuprCmBPiwZnr1uJF9IFfmOeJGHt0MZHA7AHcQfQXf7y0nonErcsv0MO4EWJw\n7ivEyP7RxFnjYuAjhTIn5Po+yYr74jpN6t09x3ZSaZlhba57jbxPvZMYGzArv9+iFq9CfBYA93Y7\nXnWW6XFMykCOF3HL/V+Ip9e/nfjszqV096TjtazM13O8Ds7l9yKOUZf3c7yezmV3BLYorU/ESdZ3\nejp2121vbxYayC/iTOiaBvPek3fM7Ygv3SXESOsHiC/q2ygM+qT1L92Tc13l159K5Q4kzvBfzh/s\nq4CtmtQ7rk6dS4ElpXKbEln3q8QX7unAoFKZ2fXqKcaLSAxeJQ807WC8diPOyOYRB5OHiH7b9Uvl\ntgP+D1hIdKcc20O9nYxXAlKpzLJ4UUhS8rw9gD/mtk8nfkui3kFxnVzHOS3u37sAd+f4PwIc1kO7\na69vNalzSoNlbmxz3SML+1XtAaP16tkrz/tZt+PVIBY9JSkDOl7A+8h3K+YyJ1Dn95Acr2UnRV8n\nEqoFRDJwDk1umOhQvJrVs1eevmUr21x++QGDHZL7+24kstOXu92eqnO8OkPSKOBRYExK6b7utqb6\nHK/2OF7tcbza54GzndWwT9Dqcrz6iKQhxG2Qk4DbfUBszvFqj+PVHser9zxwtrN8mao9jlff+SAx\n5mYnWrx9c4BzvNrjeLXH8eold/eYmZlZJflKipmZmVWSkxQzMzOrJCcpZmZmVklOUszMzKySnKSY\nmZlZJTlJMTMzs0pykmJmlSDpJklndbsdZlYdTlLMbLUjaVdJSyWt3+22mFnnOEkxs9WRiF8o9qMU\nzN7AnKSYWb+TtLakSyS9IulpSceU5o+V9AdJL0t6RtJlkt6S540kHkYJ8JKkJZIuzPMk6SRJf5K0\nUNJ0SQf279aZWV9xkmJm3XAm8Tj7/YG9gd2I55rUDAG+AewAfIJ4LPyUPO9JoJZ4vB3YGPhKfv91\nYCxwOLANcDZwqaQPd2g7zKyD/OweM+tXktYBXgQOTSldmacNA54Czk8pHVNnmXcDdwDrpZQWStqV\nuJoyLKX0ci6zJjAP2DOldEdh2QuAN6WUxnZ408ysjw3pdgPMbMDZAlgDuLM2IaX0kqRZtfeSxgAn\nAzsCw1h+1fdtwEMN6t0SWBv4taTiWJU1gOl91noz6zdOUsysUiStDVwHTAMOBZ4nunuuA9Zssui6\n+d/9gDmlea/3cTPNrB84STGz/vYY8DfgfUQXT627Zyvgt8A7gA2Bk1JKT+f57y3VsTj/O7gw7UEi\nGRmZUrqlU403s/7jJMXM+lVKaYGknwLfkzSPuFIyCViSizxBJCFHS/oxsD0xiLboceIW5P0lTQUW\npZRelXQmcLakwcAtwAbAB4H5KaVLO71tZta3fHePmXXDccDNwDXAr/LfdwOklF4AxgEHATOA44Gv\nFRdOKc0hxqx8F5gLnJunfxM4FTiRuLIyjej+md3pDTKzvue7e8zMzKySfCXFzMzMKslJipmZmVWS\nkxQzMzOrJCcpZmZmVklOUszMzKySnKSYmZlZJTlJMTMzs0pykmJmZmaV5CTFzMzMKslJipmZmVWS\nkxQzMzOrpP8HVCfP6MTfxdEAAAAASUVORK5CYII=\n", "text/html": [ "\n", "\n", "\n", "\n", "
\n", "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import dateutil.parser\n", "import pandas\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import mpld3\n", "\n", "%matplotlib inline\n", "mpld3.enable_notebook()\n", "\n", "from IPython.display import display, HTML\n", "\n", "# Data format: Time, Instance type, OS, Availability Zone, Current Bid Price\n", "# 2017-05-08 20:14:04+00:00,p2.xlarge,Linux/UNIX,us-west-1b,0.482\n", "dataframe = pandas.read_csv('data/p2-east-1b.csv', usecols=[0, 4], engine='python', names=['date','cost ($)'])\n", "display(dataframe[:5])\n", "values = dataframe.values[::-1] # reverse to order the dataset chronologically\n", "ticks = map(dateutil.parser.parse, values[:,0]) # get time ticks\n", "dataset = values[:,1] # get spot instance prices\n", "print dataset.shape, type(dataset)\n", "plt.plot(ticks, dataset)\n", "plt.ylabel(\"cost in ($)\")\n", "plt.xlabel(\"date\")\n", "plt.show()\n", "dataset = dataset.astype(np.float32)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "## Data Preprocessing: Convert to Sequence, Normalize" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "import numpy as np\n", "import mxnet as mx\n", "import logging\n", "import sys\n", "from sklearn.preprocessing import MinMaxScaler\n", "\n", "# Config the logging\n", "logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)\n", "\n", "# define sequence length\n", "seq_len = 2\n", "timesteps = seq_len\n", "batch_size = 32\n", "data_dim = 5\n", "\n", "# normalize the dataset\n", "dataset = np.reshape(dataset, (len(dataset), 1))\n", "scaler = MinMaxScaler(feature_range=(0, 1))\n", "dataset = scaler.fit_transform(dataset)\n", "\n", "# select the closing price as the label\n", "x = dataset\n", "y = dataset[:, [-1]]\n", "\n", "dataX = []\n", "dataY = []\n", "\n", "# convert an array of values into a dataset matrix\n", "# reshape into X=t and Y=t+seq_len\n", "for i in range(0, len(y) - seq_len):\n", " xx = x[i:i + seq_len]\n", " yy = y[i + seq_len]\n", " dataX.append(xx)\n", " dataY.append(yy)\n", " \n", "print dataX[0], dataY[0] # note that it has seq_len number of elements" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "## Split Dataset and Define Iterators" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "# divide the data in to training set and test set\n", "train_size = int(len(dataY) * 0.7)\n", "test_size = len(dataY) - train_size\n", "trainX, testX = np.array(dataX[0:train_size]), np.array(\n", " dataX[train_size:len(dataX)]) # continous sequence of datapoints\n", "trainY, testY = np.array(dataY[0:train_size]), np.array(\n", " dataY[train_size:len(dataY)]) \n", "\n", "# create train and test iterators\n", "train_iter = mx.io.NDArrayIter(data=trainX, label=trainY,\n", " data_name=\"data\", label_name=\"target\",\n", " batch_size=batch_size,\n", " shuffle=True)\n", "test_iter = mx.io.NDArrayIter(data=testX, label=testY,\n", " data_name=\"data\", label_name=\"target\",\n", " batch_size=batch_size)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "# Model Definition and Training" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "deletable": true, "editable": true, "scrolled": true }, "outputs": [], "source": [ "# Define the LSTM Neural Network\n", "num_epochs = 20\n", "\n", "# Note that when unrolling, if 'merge_outputs' is set to True, \n", "# the 'outputs' is merged into a single symbol\n", "# In the layout, 'N' represents batch size, 'T' represents sequence length, \n", "# and 'C' represents the number of dimensions in hidden states.\n", "\n", "data = mx.sym.var(\"data\") # Shape: (N, T, C)\n", "target = mx.sym.var(\"target\") # Shape: (N, T, C)\n", "data = mx.sym.transpose(data, axes=(1, 0, 2)) # Shape: (T, N, C)\n", "lstm1 = mx.rnn.FusedRNNCell(num_hidden=5, mode=\"lstm\", prefix=\"lstm1_\")\n", "lstm2 = mx.rnn.FusedRNNCell(num_hidden=10, mode=\"lstm\", prefix=\"lstm2_\",\n", " get_next_state=True)\n", "\n", "L1, L1_states = lstm1.unroll(length=seq_len, inputs=data, \n", " merge_outputs=True,\n", " layout=\"TNC\") # Shape: (T, N, 5)\n", "\n", "L1 = mx.sym.Dropout(L1, p=0.2) # Shape: (T, N, 5)\n", "\n", "L2, L2_states = lstm2.unroll(length=seq_len, inputs=L1, \n", " merge_outputs=True,\n", " layout=\"TNC\") # Shape: (T, N, 10)\n", "\n", "L2 = mx.sym.reshape(L2_states[0], shape=(-1, 0), reverse=True) # Shape: (T * N, 10)\n", "pred = mx.sym.FullyConnected(L2, num_hidden=1, name=\"pred\")\n", "pred = mx.sym.LinearRegressionOutput(data=pred, label=target)\n", "\n", "model = mx.mod.Module(symbol=pred, data_names=['data'], \n", " label_names=['target'], context=[mx.gpu(i) for i in range(1)])\n", "\n", "model.fit(train_data=train_iter, eval_data=test_iter,\n", " initializer=mx.init.Xavier(rnd_type=\"gaussian\", magnitude=1),\n", " optimizer=\"adam\",\n", " optimizer_params={\"learning_rate\": 1E-3},\n", " eval_metric=\"mse\", num_epoch=num_epochs)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "## Model Prediction and Visualization" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "test_predict = model.predict(test_iter).asnumpy()\n", "mse = np.mean((test_predict - testY)**2)\n", "print \"Mean Square Loss:\", mse\n", "\n", "# shift the plot\n", "t_plot = np.empty_like(dataset)\n", "t_plot[:] = np.nan\n", "\n", "# lets convert the prediction back to real scale\n", "t_plot[len(trainY):-seq_len] = scaler.inverse_transform(test_predict)\n", "\n", "plt.close('all')\n", "fig = plt.figure()\n", "plt.plot(ticks, scaler.inverse_transform(dataset), label='Ground Truth')\n", "plt.plot(ticks, t_plot, label='Predicted')\n", "\n", "plt.ylabel(\"cost in ($)\")\n", "plt.xlabel(\"date\")\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "deletable": true, "editable": true }, "source": [ "# Summary\n", "\n", "With just 20 epochs, our two layer LSTM seems to have learnt to predict the next value in the sequence given the previous values (equal to the sequence length). The mean square loss for the validation set is very small incidating high prediction accuracy, which is observed in the resulting graph as well." ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "deletable": true, "editable": true }, "source": [ "# References\n", "\n", "Basic LSTM example by hunkim [Github](https://github.com/hunkim/DeepLearningZeroToAll/blob/master/mxnet/mxlab-12-4-rnn_deep_prediction.py)\n", "\n", "Understanding [LSTM's](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.11" } }, "nbformat": 4, "nbformat_minor": 2 }