{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "83708667-4fdc-1563-7b3a-06b6575d2865" }, "source": [ "# Reinforcement Learning for Portfolio Allocation\n", "\n", "In this case study, similar to Case Study 1 of this chapter, we will use the Reinforcement Learning models to come up with a policy for optimal portfolio allocation among a set of cryptocurrencies.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Content" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* [1. Problem Definition](#0)\n", "* [2. Getting Started - Load Libraries and Dataset](#1)\n", " * [2.1. Load Libraries](#1.1) \n", " * [2.2. Load Dataset](#1.2)\n", "* [3. Exploratory Data Analysis](#2)\n", " * [3.1 Descriptive Statistics](#2.1) \n", " * [3.2. Data Visualisation](#2.2)\n", "* [4.Evaluate Algorithms and Models](#4) \n", " * [4.1. Defining the Environment](#4.1)\n", " * [4.2. Agent Script](#4.2)\n", " * [4.3. Training the model](#4.3) \n", "* [5.Testing the Model](#5) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "# 1. Problem Definition" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the reinforcement learning based framework defined for this problem, the algorithm determines the optimal portfolio allocation depending upon the current state of the portfolio of instruments. \n", "\n", "The algorithm is trained using Deep QLearning framework and the components of the reinforcement learning environment are:\n", "\n", "* Agent: Portfolio manager, robo advisor or an individual.\n", "* Action: Assignment and rebalancing the portfolio weights. The DQN model provides the Q-values which is further converted into portfolio weights.\n", "\n", "* Reward function: Sharpe ratio, which consists of the standard deviation as the risk assessment measure is used reward function. \n", "\n", "* State: The state is the correlation matrix of the instruments based on a specific time window. The correlation matrix is a suitable state variable for the portfolio allocation, as it contains the information about the relationships between different instruments and can be useful in performing portfolio allocation.\n", "\n", "* Environment: Cryptocurrency exchange.\n", "\n", "\n", "The data of cryptocurrencies that we will be using for this case study is obtained from\n", "the Kaggle platform and contains the daily prices of the cryptocurrencies during the\n", "period of 2018. The data contains some of the most liquid cryptocurrencies such as\n", "Bitcoin, Ethereum, Ripple, Litecoin and Dash." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "# 2. Getting Started- Loading the data and python packages" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 2.1. Loading the python packages" ] }, { "cell_type": "code", "execution_count": 328, "metadata": { "_cell_guid": "5d8fee34-f454-2642-8b06-ed719f0317e1" }, "outputs": [], "source": [ "# Load libraries\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "from pandas import read_csv, set_option\n", "from pandas.plotting import scatter_matrix\n", "import seaborn as sns\n", "from sklearn.preprocessing import StandardScaler\n", "import datetime\n", "import math\n", "from numpy.random import choice\n", "import random\n", "\n", "from keras.layers import Input, Dense, Flatten, Dropout\n", "from keras.models import Model\n", "from keras.regularizers import l2\n", "\n", "import numpy as np\n", "import pandas as pd\n", "\n", "import random\n", "from collections import deque\n", "import matplotlib.pylab as plt" ] }, { "cell_type": "code", "execution_count": 329, "metadata": {}, "outputs": [], "source": [ "#Diable the warnings\n", "import warnings\n", "warnings.filterwarnings('ignore')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 2.2. Loading the Data" ] }, { "cell_type": "code", "execution_count": 330, "metadata": {}, "outputs": [], "source": [ "#The data already obtained from yahoo finance is imported.\n", "dataset = read_csv('data/crypto_portfolio.csv',index_col=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "# 3. Exploratory Data Analysis" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 3.1. Descriptive Statistics" ] }, { "cell_type": "code", "execution_count": 331, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(375, 15)" ] }, "execution_count": 331, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# shape\n", "dataset.shape" ] }, { "cell_type": "code", "execution_count": 332, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | ADA | \n", "BCH | \n", "BNB | \n", "BTC | \n", "DASH | \n", "EOS | \n", "ETH | \n", "IOT | \n", "LINK | \n", "LTC | \n", "TRX | \n", "USDT | \n", "XLM | \n", "XMR | \n", "XRP | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Date | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
2018-01-01 | \n", "0.7022 | \n", "2319.120117 | \n", "8.480 | \n", "13444.879883 | \n", "1019.419983 | \n", "7.64 | \n", "756.200012 | \n", "3.90 | \n", "0.7199 | \n", "224.339996 | \n", "0.05078 | \n", "1.01 | \n", "0.4840 | \n", "338.170013 | \n", "2.05 | \n", "
2018-01-02 | \n", "0.7620 | \n", "2555.489990 | \n", "8.749 | \n", "14754.129883 | \n", "1162.469971 | \n", "8.30 | \n", "861.969971 | \n", "3.98 | \n", "0.6650 | \n", "251.809998 | \n", "0.07834 | \n", "1.02 | \n", "0.5560 | \n", "364.440002 | \n", "2.19 | \n", "
2018-01-03 | \n", "1.1000 | \n", "2557.520020 | \n", "9.488 | \n", "15156.620117 | \n", "1129.890015 | \n", "9.43 | \n", "941.099976 | \n", "4.13 | \n", "0.6790 | \n", "244.630005 | \n", "0.09430 | \n", "1.01 | \n", "0.8848 | \n", "385.820007 | \n", "2.73 | \n", "
2018-01-04 | \n", "1.1300 | \n", "2355.780029 | \n", "9.143 | \n", "15180.080078 | \n", "1120.119995 | \n", "9.47 | \n", "944.830017 | \n", "4.10 | \n", "0.9694 | \n", "238.300003 | \n", "0.21010 | \n", "1.02 | \n", "0.6950 | \n", "372.230011 | \n", "2.73 | \n", "
2018-01-05 | \n", "1.0100 | \n", "2390.040039 | \n", "14.850 | \n", "16954.779297 | \n", "1080.880005 | \n", "9.29 | \n", "967.130005 | \n", "3.76 | \n", "0.9669 | \n", "244.509995 | \n", "0.22400 | \n", "1.01 | \n", "0.6400 | \n", "357.299988 | \n", "2.51 | \n", "