{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Correlation and Autocorrelation\n",
"> A Summary of lecture \"Time Series Analysis in Python\", via datacamp\n",
"\n",
"- toc: true \n",
"- badges: true\n",
"- comments: true\n",
"- author: Chanseok Kang\n",
"- categories: [Python, Datacamp, Time_Series_Analysis]\n",
"- image: images/dji_ufo.png"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"\n",
"plt.rcParams['figure.figsize'] = (10, 5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Introduction to Course\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### A \"Thin\" Application of Time Series\n",
"[Google Trends](https://trends.google.com/trends/) allows users to see how often a term is searched for. We downloaded a file from Google Trends containing the frequency over time for the search word \"diet\". A first step when analyzing a time series is to visualize the data with a plot. You should be able to clearly see a gradual decrease in searches for \"diet\" throughout the calendar year, hitting a low around the December holidays, followed by a spike in searches around the new year as people make New Year's resolutions to lose weight.\n",
"\n",
"Like many time series datasets you will be working with, the index of dates are strings and should be converted to a datetime index before plotting."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Preprocess"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# Convert the date index to datetime\n",
"diet.index = pd.to_datetime(diet.index)\n",
"\n",
"# Plot the entire time series diet and show gridlines\n",
"diet.plot(grid=True);\n",
"plt.title('Seasonal trend of \"Diet\" keywords');\n",
"\n",
"# Slice the dataset to keep only 2012\n",
"diet2012 = diet[diet.index.year == 2012]\n",
"\n",
"# Plot 2012 data\n",
"diet2012.plot(grid=True);\n",
"plt.title('2012 trend of \"Diet\" keywords');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Merging Time Series With Different Dates\n",
"Stock and bond markets in the U.S. are closed on different days. For example, although the bond market is closed on Columbus Day (around Oct 12) and Veterans Day (around Nov 11), the stock market is open on those days. One way to see the dates that the stock market is open and the bond market is closed is to convert both indexes of dates into sets and take the difference in sets.\n",
"\n",
"The pandas ```.join()``` method is a convenient tool to merge the stock and bond DataFrames on dates when both markets are open.\n",
"Stock prices and 10-year US Government bond yields is downloaded from [FRED](https://fred.stlouisfed.org/)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Preprocess"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"stocks = pd.read_csv('./dataset/stocks.csv', index_col=0)\n",
"bonds = pd.read_csv('./dataset/bonds.csv', index_col=0)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"stocks.index = pd.to_datetime(stocks.index)\n",
"bonds.index = pd.to_datetime(bonds.index)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{Timestamp('2016-11-11 00:00:00'), Timestamp('2013-11-11 00:00:00'), Timestamp('2010-11-11 00:00:00'), Timestamp('2007-11-12 00:00:00'), Timestamp('2011-10-10 00:00:00'), Timestamp('2009-10-12 00:00:00'), Timestamp('2009-11-11 00:00:00'), Timestamp('2017-06-09 00:00:00'), Timestamp('2010-10-11 00:00:00'), Timestamp('2012-10-08 00:00:00'), Timestamp('2011-11-11 00:00:00'), Timestamp('2007-10-08 00:00:00'), Timestamp('2014-10-13 00:00:00'), Timestamp('2008-11-11 00:00:00'), Timestamp('2015-11-11 00:00:00'), Timestamp('2012-11-12 00:00:00'), Timestamp('2008-10-13 00:00:00'), Timestamp('2013-10-14 00:00:00'), Timestamp('2016-10-10 00:00:00'), Timestamp('2015-10-12 00:00:00'), Timestamp('2014-11-11 00:00:00')}\n"
]
}
],
"source": [
"# Convert the stock index and bond index into sets\n",
"set_stock_dates = set(stocks.index)\n",
"set_bond_dates = set(bonds.index)\n",
"\n",
"# Take the difference between the sets and print\n",
"print(set_stock_dates - set_bond_dates)\n",
"\n",
"# Merge stocks and bonds DataFrame using join()\n",
"stocks_and_bonds = stocks.join(bonds, how='inner')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Correlation of Two Time Series\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Correlation of Stocks and Bonds\n",
"Investors are often interested in the correlation between the returns of two different assets for asset allocation and hedging purposes. In this exercise, you'll try to answer the question of whether stocks are positively or negatively correlated with bonds. Scatter plots are also useful for visualizing the correlation between the two variables.\n",
"\n",
"Keep in mind that you should compute the correlations on the percentage changes rather than the levels."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Correlation of stocks and interest rates: 0.4119448886249272\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmIAAAEvCAYAAADmeK3JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3df5Ac5X3n8c93R4OYxY5WsmUb1siSHSLKKoJ0bIBElVTABjnBhj3AET7iIndOqFRd/oA4W1nKXBCEhHVtpXCq4tQVcZwjB7FlwLcWJhUZI3zl0kWYlVcyUYIOAbbQiLNlpCU2WsNq9dwfO72ane2f0z3TPTPvV5VKu/Oj59nununvPM/3+T7mnBMAAADary/vBgAAAPQqAjEAAICcEIgBAADkhEAMAAAgJwRiAAAAOSEQAwAAyMmyvBvQjHe+851u7dq1eTcDAAAg0t69e3/snFvtd19HBmJr167V5ORk3s0AAACIZGY/CLqPoUkAAICcEIgBAADkhEAMAAAgJwRiAAAAOSEQAwAAyAmBGAAAQE4IxAAAAHJCIAYAAJATAjEAAICcEIgBAADkhEAMAAAgJwRiAAAAOSEQAwAAyAmBGAAAQE4IxAAAAHJCIAYAAJATAjEAAICcZBKImdlHzOygmR0ys1Gf+3/NzL5rZqfM7MaG+24xsxdq/27Joj0A0KyJqao2j+3SutEntHlslyamqnk3CUAXW5Z2A2ZWkvR5SVdJOiLpWTPb4Zz717qHHZb0O5L+qOG5qyTdJWlIkpO0t/bcE2nbBQBJTUxVdcdXn9PM7JwkqTo9ozu++pwkaXjTYJ5NA9ClsugRu1TSIefcS865tyR9WdJ19Q9wzn3fOfc9SacbnrtF0pPOueO14OtJSR/JoE0AkNj4zoMLQZhnZnZO4zsP5tQiAN0ui0BsUNIrdb8fqd3W6ucCQKaOTs8kuh0A0soiEDOf21zWzzWzW81s0swmjx07FrtxABDXeQOVRLcDQFpZBGJHJJ1f9/t7JR3N+rnOuQecc0POuaHVq1c31VAACDOyZb0q5dKi2yrlkka2rM+pRQC6XepkfUnPSrrAzNZJqkq6SdJ/ivncnZL+3MxW1n6/WtIdGbQJABLzEvLHdx7U0ekZnTdQ0ciW9R2XqD8xVe34vwHoFakDMefcKTP7A80HVSVJX3TOHTCzeyRNOud2mNkvSfpfklZK+piZ3e2c2+CcO25mf6r5YE6S7nHOHU/bJgBo1vCmwY4OWpj5CXQWcy5uOldxDA0NucnJybybAQCFs3lsl6o+kwsGByraPXplDi0CYGZ7nXNDfvdlMTQJAPCRxxAhMz+BzsISRwDQAt4QYXV6Rk5nhghbXamfmZ9AZyEQA4AWyKs4LDM/gc7C0CQAtEBeQ4TdMvMT6BUEYgDQAucNVHyT5tsxRBg285PSFkCxMDQJAC1QxCHCvPLWAASjRwwAlH1PURGHCMPy1ugVA/JBIAag57WqCGrRisNS2gIoHoYmAfS8vGY4thulLYDiIRAD0PN6paeoiHlrQK9jaBJAz8tzhmM75ZW3xkxNIBiBGICeN7Jl/aIcMal7e4ranbfGIuRAOIYmAfS84U2Duu/6izQ4UJFpfoHs+66/iEAhA72Sfwc0ix4xAF0jzRBY0WY4doteyb8DmkWPGICuQLHSYmKmJhCOQAxAV8hiCGxiqqrNY7u0bvQJbR7bRRCXAWZqAuEYmgTQFdIOgZFU3hpFXGEAKBICMQCFlDTfK6gExUB/WRvv/oamZ2YXblvZX9ZdH9uwaHss/9M65N8BwQjEAATKq/5TM71TfiUoyiXT6ydndbrhsSdOzmrk0f2LtlfEpHLqbwHdj0AMgK88h+qCeqfufvxAYGDiNwT2xpunFvWE1Zudc4t6u1pV1LXZYKpoQ6UEhUBrEIgB8JXnUF1QL9SJk7M6cXI+sPILTBqHwNaNPhH7deIUda0PRgb6y3JOen1mNjAwSRNMJdn/rQ6S4vwd7QrUCAjRbQjEAPhq51Bd48V1RaUc2JNVLyowDOrlqr/fE5ZUPjFV1d2PH1gIAiUt+jkowEoTzMbd/+3oOYv6O9rVe1e0XkIgC5SvAOCrXfWf/Op/vfHWKZX7LNbzwwLDkS3rA7dTLtmSEgrDmwa1e/RKvTx2jXaPXrkoyKgPvPzMzM7ptu37FpW9SBPMxt3/7ahcH/V3tKt6PlX60Y0IxAD4iqr/lFXNLb+L6+yc09vOXrZoyaFK2f/jKiwwHN40qPGPX6yBSnnR7Sv7yxq/8eJYvSh+7QtTX0g2TTAbt/5WO3ouo/6OdvWeFnFCBZAWQ5MAfA1vGtTkD47rS8+8ojnnVDLTDZcMZj4UFXQRnT45q6k/uVrSfNDnzXKsV+5b2qvl93ekGbZq5iLv9dKkWUw8bv2tVk0yqBf1d7SjDe18HaCd6BEDekzcnqyJqaoe21vVnHOSpDnn9Nje6kI+V1ZDRHF6jcZ3HtTsnFvyGK9trdTsRb46PaPxnQd1wyWDTS8m7jdU2qgdleujFkVvV/V8qvSjG5lrwwdZ1oaGhtzk5GTezQASKcJsr8aeLGn+QuYXHGwe2+Xb+7Cyv6zpk7Py++QwSS+PXZN5m9aNPuH7et5r3nz5Gt07fFGi103TviRMktN88NLNMwmZNQkEM7O9zrkh3/sIxIDWSxIAtVJQcDU4UNHu0SsX3RYW/AwEzGr0204cfhdX6cywXJ9ZaO+XSbp/68bY9bmSXsi95wTNwOyTlhSN9ZPHMQeQv7BAjBwxoA2KsnxOkmTnsNIPZvNBRTO5T34a87gaA9eoIUgnxdqXzea2ee0LCmRPKzg4rVc/fFsfDF5x4Wo9/fwxenmAHkSOGNAGRZntlWQWX1hQNX1yNjRnKEpUnlrSmYpSvH2ZNrct7DXOWT4/yzOKF/zVl+t4aM/hRb97sy4BdD8CMaAN2lWTK0qSZOfhTYNLyj54VlTKsRLJ/fjVDWsMPMICnqDqYnH2ZdqAOOw1jk7P+O7fRiWzyCCT2lhA7yAQA9qgKLO9oma/Ndp27QbfgqhvvHVqIXBKWk8sTq9UUMAzOFDRzZevWRKM+e1Lv3YFbbfPLHT26OaxXVo7+oRefT28Sn/9/pWWBo2Vcin2TE9qYwG9gRwxoA3i1oRqV1uSlE9oXNpHOrNgtqTEOVdxeqXC6lYNbxrU0PtWhe7LoFywGy4Z1GN7q0sCwTnnfNvduJ3TATFUfSBYv3/9JgaEJf3X84JDcsWA7pbJrEkz+4ikv5RUkvQF59xYw/3LJf29pEskvSZpq3Pu+2a2VtK/SfK+Cu9xzv1+1OsxaxJon6DZk6bghP6V/eWFYqyN4s7cTFOmIOw1Rras16e/sj+wZ6q+zETQdoIeH0eSchjMsgS6Q0tnTZpZSdLnJV0l6YikZ81sh3PuX+se9ilJJ5xzP29mN0n6rKSttftedM5tTNsOAM0LC3rCqpkH9W6dODkb2JsTt9p8sxXxJ6aqgcHT0ekZDW8a1O3b9wU+v75XL2p40KRE5Trqi+GWaiU5BmuzJr0VDOrlMbMWQHtlMTR5qaRDzrmXJMnMvizpOkn1gdh1krbVfn5U0l+ZWbwVfQG0VFRJh7DAKWyYLSiAaGaYNqrOWP1tXtv9eMN9YaU5pPkA6O7HD0Q+LijnLKi9jSU56odbH95z2Hdb5IoB0Tq50G8WgdigpFfqfj8i6bKgxzjnTpnZ65LeUbtvnZlNSfp3SXc6576dQZsAxBRV4ywqcLotoHcpLIBI0tvlFyiOPLJfMi0se+QFj8uX9YUO+Xm5YP9hzYrIIccTJ2f1wXPfruNvvOW7zUq5pCsuXK3NY7tCAy6vbWeXl7atfj/nuY5iJ1/EgCzXvs1DFoGYX89WY/JF0GNelbTGOfeamV0iacLMNjjn/n3Ji5jdKulWSVqzZk3KJgPwxEmeDwqchjcNatuOA/6FTE1aO/qEpPlip9uu3ZD4Q3FiquqbzzXrkzU/MzsXK+9qZnZOe146Eev1/8+Lx/UrH1ilPS+dWNQGbzixPvE/KuAKapu3n9MsEB5HULDV6RexViAw7SxFKZjdrCzKVxyRdH7d7++VdDToMWa2TNIKScedc286516TJOfcXkkvSvoFvxdxzj3gnBtyzg2tXr06g2YDkNLXONt27Qbf2ln1sdP0zKz+cPu+WEVK68tF3L59X0sW9o67TSdp94vHFz3eC46efv6Y74d/4wzTKN5+biwtsrK/rOXL+nT79n2xyoKECavdluUC7t0gTp07FEtRCmY3K4tA7FlJF5jZOjM7S9JNknY0PGaHpFtqP98oaZdzzpnZ6lqyv8zs/ZIukPRSBm0CEFPaGmeNAUQpIP3ztKRtOw6Ebqv+Iigt7VrPSlAb4/CClKQf8iZF7mevSO7Nl6/R9MlZTc/MRgYDceq4hQVbnX4RyxqBaecpSsHsZqUOxJxzpyT9gaSdmi9F8RXn3AEzu8fMrq097G8lvcPMDkn6Q0mjtdt/TdL3zGy/5pP4f985dzxtmwDEl7TIa9A2vCr7p0N6m6LWYrz78QOxhhd9aszGVimX9InLzo+sgB+mOj2jFQGrDgRxUqz9PDFV1cN7Di8JQv2Cgbi9N2HBVqdfxLJGYNp5ilIwu1mZFHR1zv2jpH9suO1P6n7+maSP+zzvMUmPZdEGAM3zcsC8oarbt+/T+M6DTeXGRM00DDIxVY09rBdUWDVKyWwh+KkvClsp9+nk7OlE2/rJm6d8b+8P2NZgrfJ+1P4c33kwsCfQCwa84+S3n/1yY8ImAqTJTevGXKo8J02gOUUqmN0MKusDkBQ+80jSoqT8PpsPhvyKmY5sWR84k3Jlfznw4t2OoZ/6XK/6oGjz2C6dTBg8zgVEg0EB3RUXxsttDet5OW+gEqsgbP02JqaqOvnW0qCxvnSGpIXAzlsL0zseQRczv/Pl9u37dNv2fYmL3BZJqydNoDWarTtYBARiACQF58bc/fgB/fRnpxbNVPR+9JthN7xpUJM/OK6HGupilUuma37x3MBgr11DP34zAtvx2tu/84qG3rfK92JRH5z21Qq9NjJpoXZb1PDtQH95Ybt+QVvjLFbv/ySzJ/3a4bW6k2dednrvCjoPi34DkBQcjJw4OetbLsLjl7t07/BF+tzWjYvyocZvvDhwpuH4zoNtG/rxa287Xnv2tPPt9ZuYqmrkkf0LeV5BQdjNl6/R8KbBWEGjt4mwoG1858FFCf5Jk9Sj2tHJCe71OY+7R68kCENLEYgBHSrObLkk0gQjfhdlv4tZUO5YdXpGI1vW+xYcbIXq9Iw+cMc/6s6J+V6bdg07+e2nbTsOhAa6gwMV3b91o+4dvkhSvOPkDSEHBUvTM7NLEvzDjo3f+RWnHc3kCgK9hkAMaJGsA6XGbWdd6yho5tFAjNmBURfliamqNt79jdDH3LZ9nyrl9n0kzTmnh/Yc1s1/889t6/Hw209hM0kb87gk/+PUyKSFpZzi8Na+DOJ3fsVpR5oyIUCvIBADWqDVRSH9yjykHQoKKmOx7doNKofUizAF95pIZ/ZFVOkKaT7RvdxnbesZk+YLtt458ZxaHTOU+yxxz5vfMR3eNKgbLhkM3UdOWpj1GrdMx5xzodtsbEv9+RK2zU7Ryi9OQBiS9YEWaOWSG2FlHtImnQfNPPJLvvdEJWjHSS6vFzZM1ypBf1tWTNLWS89fKA1SnwB+zlklvfFW8P7xgtwrLlytp58/tpDQH7WXjk7P+Caen3zrVOD5E2eb9bzzZfPYLt9hSK9nrug5VizzhDwRiHWRbqzp06laWRQyrNerVUnnTz9/LNbj/IJNCmHOBzh+61JO/uC43joVXb+sOj2zKFiM09NUv3RS/fGIU/4iapuNRras1+3b9y0J5LyeuaJ/DoX1MBe97eh8DE12CdZHK5ZWVisPC2xalXSeJJhqfCyFMOf5Xei/9MwrLekBDKt7FWdIMUjYNqOK0BZVK3uYgTgIxLoE66MVSyuX3AgKbAYq5Uy/vdfnzPQlSKBqbF+SPKVe04ocqjhLVHkzWpMEYyv7w8+voG0VPRDPo4cZqEcg1iVYH61Ysli/MUhQkLft2g2pt+1p7GH1CxjKJVuSxO8XbHr7ghl0SwXtk2b31eBAJVHdK79zqdxnKpf8Xz+sh71T1/vLo4cZqEeOWJdgfbTiadWSG+2o/B2UYF8y02nnFl4zbjuGNw3q9oBlj3pVpVzSDZcMLsod8yxfZpo9Lc3Oxe8xayboCTqXpMVLWknzhX3DEtg7tSJ90Gdn1j3MQBACsS7B+mi9pdXrqgX1Epx2Ti+PXbOkLXGsqJRjlbDoBY1rMT78zGHVdzqenD2tPpPM5qvkl8x0+ftX6vuvzSwEOfWzKNMEPUHn0vjOg0uOV1QCeyeu9xf02ZllDzOy1W0T0wjEukSnfhtFMQ30l30TmL01DJOamKrqDZ+Fp3vdxFRVj+2tyi9VrD6Hf845fffw66HD215OX1bv/15Jd+Czs7N0Y6kRArEu0onfRlFMbwaUNvALGIK+ncZZyLpXVadndNv2fTJF1+7yhPVGJbk4xe1N6KV0hzSfnd3WO1N0razRmBcCMQCLTExVdXLWv7aVN1TlXXyq0zOLggkvAHhk8rB2v3h84Xm9GoQt6zO9++fODlxzMeleCeqNintxShKwke4QrRt7Z4quG3tqmTUJYJGw6fwm6c6J5xYtEt0YTMzMzi0KwnrZ25Yv0+7RKzPbXmNvlDccGRToNV6ckpS5aeXM325B2aD2a2WNxrzQIwZgkbBvlk7Sw3sOJ+7J6VWvz8zqzonnEj+vv9yn2Tm3qNhrY2/UnRPPRR6LxotT0t4E0h3CdWPvTNF1Y08tPWIAFon6ZkkQFt9Af1kPN7GO5Z9f/4sa//jFgb1RE1PVyCDM7+LUjb0JeWJ/tl839tSa68DcjaGhITc5OZl3M4CulGYtQiw20GTJDq9sRaOV/WXd9bENC/l5QRrLY3j8jm2lXOr4C1le2J+Iy8z2OueGfO8jEEOnYHZS+0xMVZcU9EQyfba4BEVWyiULLfTqBWFesFaqzVj1bpco1ZAlPpcQB4EYOh7fPNNr5oLR+JyjtSWPEE+pRWU7grZrkm6+fI1vtX6J9wyQl7BAjBwxdARmJzVvYqqqTfd8Q7dt37ewdmR1eka3b9+ntaNPaPPYrtA1BE++dWrhOcv4xEhkzjm1YoXNoODuVz6wSk8/fyxwWJn3DFA8zJpERyja7KSiDUeEFVUNyvdqrP0laUnNqZFH9y8aBgsoL4YQ7exB9JZACsOMPqBY+H6LjlCk2UlecFPfu3THV58L7VXKqz1Bi3c3auwpmZiq6tNf2Z9o0WnEVyn7f/T2pew+8wLxMMzoA4qFQAwdYWTLelXKpUW35VU7pmjDpGHtSdL74T3WC+x6tRp+O5xdLqnsE3WlTe73ekMb3yueTq+31Am8IrvrYgz7AxJDk+gQRVqYt2jDpGHtCVov0M95A5WFnjCCsNY6cXJW5VJ099dgrffK7xg2rlPpBVn17xW/WZMk6rcOSx6hGQRi6BhFqfKdxWLIWeaYhbXHrwq1n0q5pCsuXK2RRwnC2qFk4SUoPEenZ3T/1o2+M4ZvuGRQTz9/zPccKsp7pdd044LUaD0CMSChtEtsZP2tOaw93vbufvyATpycrwk2UCnroxefu+QifvfjB8gJa4OoOmD1zhuoFKY3uGgTVIqoaL3l6AwEYshVJ364p70wZv2tOU57fvqzUws/T8/Mavt3XtH4xy9e9Jjbtu9L/NpI7tK1K7XnpROxeh6vuHC1pPx7uBhyiyeL3nL0HgIx5CbLD/d2B3RpLoyt+NYc1p5tOw4sWjxakmZPO23bcYCLaA52v3g89mO/vv9V3Tt8Uehj2nHuM+QWTzcuSI3WY9YkcpPV7MOilZOIEvTtuM+sJTOtgpYparx9oFLO7DWRjemZ2dBzoV3nPkNu8XTjgtRoPQIx5CarD/eilZOIElReYM653ALJiamqrBUl4JFa2HncrnO/SHX8im5406B2j16pl8eu0e7RKwnCECmTQMzMPmJmB83skJmN+ty/3My21+5/xszW1t13R+32g2a2JYv2oDNk9eHead/WG781l3wioCwvpkFFQr3bvV4VL5kfxRJWfqRd536R6vgB3SZ1jpiZlSR9XtJVko5IetbMdjjn/rXuYZ+SdMI59/NmdpOkz0raamYflHSTpA2SzpP0TTP7BedcdClwdLys8imKmiAblrtTn9O1bvQJ3+c3czGdmKpq244DC8OOK/vLgUVCT7szr81cyeIK66hs17lflJmbQDfKIln/UkmHnHMvSZKZfVnSdZLqA7HrJG2r/fyopL8yM6vd/mXn3JuSXjazQ7Xt/XMG7ULBZfXhXsQE2SQTEbK6mE5MVTXyyP5FiflRvVwEYMXnNH9s/d4XV1y4Wg/tOex7e9bynrkJdKssArFBSa/U/X5E0mVBj3HOnTKz1yW9o3b7nobn8k7vIVl8uBfx23qSWWZZBZLjOw8umR2J7hA0O/Hp54/5Pj7odgDFk0Ug5tdz3ng1CHpMnOfOb8DsVkm3StKaNWuStA89oGjf1pPk7mQVSBY1Jw7pVadntHls15LzotPyIwEslUUgdkTS+XW/v1fS0YDHHDGzZZJWSDoe87mSJOfcA5IekKShoSG+9qPQkg43ZhFIJllXEsVikgb6y6FDyX7D20XNjwQQXxazJp+VdIGZrTOzszSffL+j4TE7JN1S+/lGSbucc652+021WZXrJF0g6TsZtAnIVR6zzEa2rFc5aIokCs1pvmZY1Ady42zapOfZxFRVm8d2taReHYDmpO4Rq+V8/YGknZJKkr7onDtgZvdImnTO7ZD0t5L+Zy0Z/7jmgzXVHvcVzSf2n5L0X5kxiW7Qzry1+tmZlXIfeWIFUzLpPSsqOjo9E9rr5Zy0rGRa3meamT0duL36Ycck5xnLFAHFZC7GemdFMzQ05CYnJ/NuBpCrxlIVKK7Pbd24EOxsHtsVOoQ8OFDR7tErAx/n3Z9U1tsDEJ+Z7XXODfndR2V9oEDiDh15vRsEYZ2hfqWEoJUVPF5ifnV6ZslspjTD2yT2A8XEot8ZaPeC0+gcSc6NOyee08N7Di9MG/YbOvK2R1J+Z6kvXeIdy09/Zb/mfEYkTGeq6XtTy53me67SfLaQ2A8UEz1iKXXagtNonyTnxsRUdVEQ5qlPzq7fXhJ9ks45K7gHBu3RmNv1F791sW/PWOM54AVhadctZJkioJgIxFLqtAWn0T5Jzo3xnQcDq9xXp2c0MVXVp7+yf8n24iiVTH/2Hy/S57ZuXFjfkrmV7dfY89S45ujK/nLgc7MYPmx8vcGBiu67/qLUvffMxATSYWgyJfIuOlsrh5WTnBtR58vII/7DWHHMzjndtn3fwhAX2s8k356n+qHKzWO7AmdUZjV8mHXhY2ZiAunRI5ZS0AckeRfF1+ph5STnRtT5kkVJijhbqJT5SMiaSbr58jWRgUlYMJ738GFQrxcjAkB6fOqmRN5F52r1RSTJuVGU8+XNU8H1q5Dcyv6y7t+6UfcOXxT52KBgfKBSzrV3KewLCyMCQHoMTaZUxAWnEU87LiJnl/sWgr2BSlnbrt3ge24MbxrUI5OHtfvF47G3XTLTz1WWhS6LkxS1YLOz+QOr9PDv/XKsx05MVXXyrVNLbq+US9p27Yasm5ZI2BeWZmZiMsscWIxALANFW3A6CB+Ai7VyOn9j7ox0prepsRDrOWeV5JzTyZBq6o36pMyDMGSjZKZPXHZ+rF4wyf9ckcID93YK+8Jy/9aNS9oetcQSOWXAYgRiPaKXPwCDAtCRLesTXUTCttUoqBfh7scP6Kc/O7Uo5+uNt5LPhDwtEYQVUDNV6v3OFUk6Z/myQrw3g76w9Nn83Nv7rr8o9he8sN61IvytQB4IxHpEt3wAJu3VixOAxt1ekmA2qBeB4Km7RQ1r+52/Rc+z8vvCIklzzumOrz6n+66/KHbwWfS/FcgDgViP6IYPwGZ69aIC0CTDykmC2aBeBHS3FZXgWmBB5++KStl3qaqizLwOWwkg6Zc5qvsDSzFrskd0Q5mNZmY5ZhmAJtlW0IzJgZALNTqfhVTKDTp/zVT4mdfDmwZ1OqCOXZL3ErPMgaUIxHpEN3wANhNUZRmAJtlWUBXzbdduULmPuvbdajpk6DnoPJ0+OduSivdZy+K91Krq/kAnY2iyR3RDmY1mhjWaScjPalthw571sybjVrynMn7xhZ2LYedvJ8y8zuq91Al/K9BOBGI9pNM/AJu5EGQZgGa1Le84BJUtCEIQVmxR52KWXwry0A1f5oAiMtfk+nV5GhoacpOTk3k3Aznoplpom8d2+faQlMyaXlcyroFKWR+9+Fw9tOdwS1+nV3jLGEXVDuum8xdAfGa21zk35HcfPWJoqawvPHn06rXq4hk0q/K0cxps8axLr7gsw53xlftMWy893zd4dZKe+N6revr5Y6HnSaf3SgPIHsn6aJlWL6rdDq36G8KeP9Bf9p1ckaWZ2Tk9tOcwQVhMlXKfxj9+cehjTpyczew8CVpkG0D3IRBDy7R6Ue1W8i6Et23f15K/Iez5zi2dXVYKq4uAllrZX9a//elvaHjToL70zCuxn9fsedINX2AAxEcghpbp1CKy9RfCINXpmVS9FWH74PXabMrhTYPaPXqlXh67Rn/xWxeLUKw1VvaXZZrPmyuXFu/lSrmkuz52ZtHtpLl7zZzrnfwFBkBy5IihZTqtiraXCxY3N6u+t0JKtmZnWOV9b/805qYxjJi9Ppuv43XeQEVXXLhaX9//6kJZkZX9Zd31scWLbiedSNHMud6pX2AANIceMbRMJxWRjdMLFqSZ3oqRLeuX9L5I8wnhI1vW+w5PIXun3ZmA+qE9hxctNfSz2dNLHv+Jy8733c7mD6zK7FzvhlUwAMRHIIaWKVIV7ajkZ7/hoCSS9lYMbxrU+I0Xa2X/mSWPBiplbb30fI3vPOibm4b28guw7x2+SL99+ZqFnL2SmX778jV6+Pd+ObNzvZO+wABIj6FJtFQRpuvHWSw8KpCqlEu67/qLAocum+mtaNw3SQu8ovXqz4uoMiZZnetFL5xKLTQgWwRi6FInHXEAABUZSURBVHphyc/eBSQsZ2uw4WLTTHX0oItX/e19bSjkimTq8/WigvksteMLTDMBVbv3A9ALCMTQ9eIkPwctP9M4vNRMb0XQxWvyB8f12N7qwu1FD8L6JMnm86p6QX2AHSeY7yTNBlTdth+AIiAQQ9cL6u1aUSlr89iuhYDqhksGIyujS0uDMS+PKOhCFHTx+tIzr8QOvopQAf+0CtCIGPrLfVp5zvJUExz6bHGOWLfNZGw2oOq2/QAUAYEYWqYouSR+vV3lPtMbb51amCVXnZ7RY3ursRKsk/YmBF2k4gRhXq+cpESlNXrZWcvme7JGHt2v2bkz+7jUZ1peMp30mQ3ZyOv1847tQH9ZJ07OLnlcp85kbDag6rSSNEAnYNYkWqJI1cH9Zm++7exliy7SUvwyFEkLbgZdpIKq5ZfMlsy884q7hhV1rZT7ltzv/e691uBARb99+ZqWLp+UN68gbmPvXZ/mg7SkZmbn5Jy6aiZjsyUymNEJZI8eMbRE0XJJGpOf140+4fu4OEMsSXsTgvLPbrhkcFGOmDTfU/e2s5dp2qf3RQqfVDBT6+nxio42TjKoN/S+Vfr0V/a3JS9tZX9Zr8/Mti237LyBisZ3HtRswwvOnnaL6oQl8frMrO7furEQPbxZCDonowKqos/oBDoRgRhaoui5JGmGWJI+N+ziNfS+VQu3r6iU9cZbpxaGwPyGPEe2rNft2/eFpmrNObdwUQ27QLY6CKsfVh15ZL9O171en6QVAcN9aZjO7KMsnTdQyXQmY97D9mkCqiKUpAG6CUOTaImiVwdPM8TSzHPr140c2bJe4zsPat3oExrfeVAjW9br5bFrdM7y6OHS4U2DuvnyNZHrToYNlXrDxq3gN6zq1zt1WtKJk7O+6zs2Dqcm4TS/j4LOs5X95cTDslkPvRVl2L7+nNw9eiXBFZATAjG0RNFzSdJU/fd77g2XDC4EV2ELgYddhIN6C6vTM4tWBRh63yrdv3XjwusHCdpe2lUEglTKJX3isvN13kBlYTZp2N8laX6o0J1ZeHtwoKL7t27U98eu0Yv3/aYGEwbu3uODzr+7PrZh4dhJ8t1/lXLfovZkvRoEi3oDqGcuxfCEma2StF3SWknfl/RbzrkTPo+7RdKdtV/vdc49WLv9W5LOleR9Ul/tnPtR1OsODQ25ycnJptuN9sh7+KVd/Cri+9Ugk6TNY7t8hzW9wMDvvsbSFY3bDtvm7tErF9oYZ9blyv6ypv7kam28+xuJ8qlKZvrEZecvyXmrlEtavqwvclv1ba3nt2/LJdOyPlvIiat/rfr9Euf8y+McXTf6hO/Qskl6eeyalr42gHyY2V7n3JDffWlzxEYlPeWcGzOz0drvf9zw4qsk3SVpSPPXk71mtqMuYLvZOUdU1YV6JZckycSEsNy5+7duXBJ0+NUPa9x2VOJ1kqWTvEkC267doJFH9i8ZUgxy2jk9/fwx3/1wdrlPlXIp9PWD9kt9LlN1ekYlM83OOb3r7WfrigtXh9Z9i3P+5XGOUgICQL20Q5PXSXqw9vODkoZ9HrNF0pPOueO14OtJSR9J+bpAYSSZmBCWO+c35BkUBtVvO2qYNclQpNe+4U2DGv/4xQvb9Mvlanxe0H6YPjm7aDgw7HX9DG8aXBhq9CYYeHXfvPy6TspxKvqwPYD2Stsj9m7n3KuS5Jx71cze5fOYQUmv1P1+pHab5+/MbE7SY5oftuyA2t3AGUl6OKJ6rxp7aIKGHRu3HdazE3emamMwUL/NzWO7AocXvefd/fiBwKKn3raChnGjgpCilUNJwm/401tAvtuH7QFEiwzEzOybkt7jc9dnYr6G39doL9i62TlXNbO3az4Q+6Skvw9ox62SbpWkNWvWxHxpoPWS1GRKWjag2XpP0pkAIO43m7Ck9LDcMq9ExU9/dmrJfeWSLRoi9QKqOLXO6hWhHEqWi2Tfd/1FvjlxAHpPZCDmnPtw0H1m9kMzO7fWG3auJL9E+yOSfr3u9/dK+lZt29Xa/z8xs3+QdKkCAjHn3AOSHpDmk/Wj2o3elEfyddLgKkleUrP1npLkhUnzw5lB25yYqgaudek9b/PYLt98snPOWubbE+ZX6yzs2OWdV8Ui2QBaJe3Q5A5Jt0gaq/3/NZ/H7JT052a2svb71ZLuMLNlkgaccz82s7Kkj0r6Zsr2oIc1e7HMQlBwlUVgGBW4+b1GkrywqB62oF41r3iqFNwz5S03FBWQRB27ND2DWWCRbACtkjZZf0zSVWb2gqSrar/LzIbM7AuS5Jw7LulPJT1b+3dP7bblknaa2fck7ZNUlfQ3KduDHla0+kztKNwZ9BpBQ4km6XN1Ncji1MkKChq84qlSdAHfqIAk6tilqfuWhTSLZCe5HUDvSdUj5px7TdKHfG6flPS7db9/UdIXGx7zhqRL0rw+UK9ovQ/tGJYKeg0vB6tR0qV6Jqaq6gvY1sr+sjaP7dLR6RkN9JdV7rNFw5P1PVZRQ4txjl2e5VCaHRrNuycPQPFRWR9dI4veh4mp6qIq9ml6r1oZGHrtDOr58nKw6iUNALzeNr8grFwy/fRnpxZ64k6cnJVsvsyFX49VVMmGovYc1e/nxllHcRfJzrMnD0Dxseg3ukba3oesc8xalWAeJxF/sC5XrNn8tKA8s5KZzjlr2ZJyFrNzTucsX6Z9d13tu72zy30L2xuolLXt2g2xi9LmoXE/O50psBt3tqfUO4WNATSHQKyFemWJn6JodoahJ+uhxCxKT/j9HVGJ+PWzEdOcb0E9d6edW0jCj/Mcv8DxzVOLlydKe+xawW8/e0EYpScAZIVArEXynMHXy9IEH1kPJWZVeqLx3AlrT5KemihRPXpxe/viBrhF6zkqWs5hL+FLLHoJgViLUD8of0k/zFsxlNhMcBF17gS106+nJs0F7YoLV+uhPYd9bx9636rYvX2dGtCsqJR9VxNYUSnn0JrewZdY9BqS9VukUy8+3aKZ0hFFWQMw6tyJ28605TOefv5Y4O1JktCLmogfxQKW1gy6HdkoWhkaoNXoEWuRvCuB97pmeiSLkqcUde7EbWfaXtmogDBub18RE/HjmPZZNzPsdmSDL7HoNQRiLdKpF59u0eyHeRHylOKcO3HamfaCltWXiaIEuEnxZSof7Hf0GgKxFunUi0+QTkue7eQP86zOnah9EHVMs/wyUYQANym+TOWD/Y5eY86nWGPRDQ0NucnJybyb0TP8yg9UyqVCF6bsxDZnLWwfSIq1fzotAM9ar//9eWG/o9uY2V7n3JDvfQRi2erGD5CgCu5Fr6fkHYvq9MzCkj9ZlnfoBEHnYyuOaZHP/SK3DUD3CwvEGJrMULdOu+7U5Flvn3fjMZHiBRdBQ4JZH9Min/tFbhsAUL4iQ9067bpTyw9InX1Mwta9TFuaIutjWuT9XOS2AQCBWIY6tecoSlHqazWjU49JVKCVNrjI+pgWeT8XuW0AQCCWoU7uOQqTpHhn0XTqMYkKtPzyu8Jub5T1MS3yfi5y2wCAHLEMdfO0604sPyB17jGJ6sXxJh80KiUo+57lMS3yfi5y2wCAQCxD3VY7rBt06jGJqgHmF4SF3d5qRd7PRW4bAFC+AiigqDponVpSBAB6UVj5CnLEgAKKyuHq5AkUAIAzGJpEJIph5iMsh4vhNgDoDgRiCEUxzOLq1AkU6F18qQOWIhBDqLAyCnyANo8LEnoNX+oAfwRiCNWNxTDzDoK4IKEX8aUO8EeyPkJ1WzHMtEsDZYEld9CLuvFLHZAFAjGE6rbZeUUIgrggoRd125c6ICsEYgjVycsb+SlCEMQFCb2o277UAVkhRwyRuml2XlTF+nZgyR30IkquAP4IxNBTihAEcUFCr+qmL3VAVgjE0FOKEgRxQQIASARi6EEEQQCAoiBZHwAAICcEYgAAADlhaBLIUN5V+wEAnSVVj5iZrTKzJ83shdr/KwMe909mNm1mX2+4fZ2ZPVN7/nYzOytNe4A8FaFqPwCgs6QdmhyV9JRz7gJJT9V+9zMu6ZM+t39W0v2155+Q9KmU7QFyU4Sq/QCAzpI2ELtO0oO1nx+UNOz3IOfcU5J+Un+bmZmkKyU9GvV8oBMUoWo/AKCzpA3E3u2ce1WSav+/K8Fz3yFp2jl3qvb7EUkk06BjsXQRACCpyEDMzL5pZv/i8++6lK9tPre5kHbcamaTZjZ57NixlC8NZI+19AAASUXOmnTOfTjoPjP7oZmd65x71czOlfSjBK/9Y0kDZras1iv2XklHQ9rxgKQHJGloaCgwYAPyUpSq/QCAzpG2fMUOSbdIGqv9/7W4T3TOOTN7WtKNkr6c9PlAEVG1HwCQRNocsTFJV5nZC5Kuqv0uMxsysy94DzKzb0t6RNKHzOyImW2p3fXHkv7QzA5pPmfsb1O2BwAAoGOk6hFzzr0m6UM+t09K+t2633814PkvSbo0TRsAAAA6FUscAQAA5IRADAAAICcEYgAAADkhEAMAAMgJgRgAAEBOCMQAAAByQiAGAACQEwIxAACAnBCIAQAA5IRADAAAICcEYgAAADkhEAMAAMgJgRgAAEBOCMQAAAByQiAGAACQEwIxAACAnBCIAQAA5IRADAAAICcEYgAAADkhEAMAAMgJgRgAAEBOCMQAAAByQiAGAACQEwIxAACAnBCIAQAA5IRADAAAICcEYgAAADkhEAMAAMgJgRgAAEBOCMQAAAByQiAGAACQEwIxAACAnBCIAQAA5CRVIGZmq8zsSTN7ofb/yoDH/ZOZTZvZ1xtu/x9m9rKZ7av925imPQAAAJ0kbY/YqKSnnHMXSHqq9rufcUmfDLhvxDm3sfZvX8r2AAAAdIy0gdh1kh6s/fygpGG/BznnnpL0k5SvBQAA0FXSBmLvds69Kkm1/9/VxDb+zMy+Z2b3m9nylO0BAADoGMuiHmBm35T0Hp+7PpPB698h6f9JOkvSA5L+WNI9Ae24VdKtkrRmzZoMXhoAACBfkYGYc+7DQfeZ2Q/N7Fzn3Ktmdq6kHyV5ca83TdKbZvZ3kv4o5LEPaD5Y09DQkEvyOgAAAEWUdmhyh6Rbaj/fIulrSZ5cC95kZqb5/LJ/SdkeAACAjpE2EBuTdJWZvSDpqtrvMrMhM/uC9yAz+7akRyR9yMyOmNmW2l0Pm9lzkp6T9E5J96ZsDwAAQMeIHJoM45x7TdKHfG6flPS7db//asDzr0zz+gAAAJ2MyvoAAAA5IRADAADICYEYAABATgjEAAAAckIgBgAAkBMCMQAAgJwQiAEAAOSEQAwAACAnqQq6AgCSmZiqanznQR2dntF5AxWNbFmv4U2DeTcLQE4IxACgTSamqrrjq89pZnZOklSdntEdX31OkgjGgB7F0CQAtMn4zoMLQZhnZnZO4zsP5tQiAHkjEAOANjk6PZPodgDdj0AMANrkvIFKotsBdD8CMQBok5Et61UplxbdVimXNLJlfU4tApA3kvUBoE28hHxmTQLwEIgBQBsNbxok8AKwgKFJAACAnBCIAQAA5IRADAAAICcEYgAAADkhEAMAAMgJgRgAAEBOCMQAAAByQiAGAACQEwIxAACAnBCIAQAA5IRADAAAICcEYgAAADkhEAMAAMgJgRgAAEBOCMQAAAByQiAGAACQEwIxAACAnJhzLu82JGZmxyT9oMmnv1PSjzNsDoqF49vdOL7dj2Pc3Xr1+L7PObfa746ODMTSMLNJ59xQ3u1Aa3B8uxvHt/txjLsbx3cphiYBAAByQiAGAACQk14MxB7IuwFoKY5vd+P4dj+OcXfj+DbouRwxAACAoujFHjEAAIBC6MpAzMxWmdmTZvZC7f+VAY/7JzObNrOvN9y+zsyeqT1/u5md1Z6WI44Ex/eW2mNeMLNb6m7/lpkdNLN9tX/val/rEcTMPlI7LofMbNTn/uW19+Oh2vtzbd19d9RuP2hmW9rZbsTT7PE1s7VmNlP3fv3v7W47osU4vr9mZt81s1NmdmPDfb6f1b2iKwMxSaOSnnLOXSDpqdrvfsYlfdLn9s9Kur/2/BOSPtWSVqJZkcfXzFZJukvSZZIulXRXQ8B2s3NuY+3fj9rRaAQzs5Kkz0v6DUkflPQJM/tgw8M+JemEc+7nJd2v+fepao+7SdIGSR+R9Ne17aEg0hzfmhfr3q+/35ZGI7aYx/ewpN+R9A8Nz436rO563RqIXSfpwdrPD0oa9nuQc+4pST+pv83MTNKVkh6Nej5yE+f4bpH0pHPuuHPuhKQnNX+RRjFdKumQc+4l59xbkr6s+eNcr/64PyrpQ7X363WSvuyce9M597KkQ7XtoTjSHF8UX+Txdc593zn3PUmnG57b85/V3RqIvds596ok1f5PMvT0DknTzrlTtd+PSBrMuH1IJ87xHZT0St3vjcfx72rDHP+ND/tCiDpeix5Te3++rvn3a5znIl9pjq8krTOzKTP732b2q61uLBJL8x7s+ffvsrwb0Cwz+6ak9/jc9Zm0m/a5jamlbZbB8Q07jjc756pm9nZJj2l+ePrvk7cSGYrzvgt6DO/Z4ktzfF+VtMY595qZXSJpwsw2OOf+PetGomlp3oM9//7t2EDMOffhoPvM7Idmdq5z7lUzO1dSkhygH0saMLNltW9l75V0NGVzkVAGx/eIpF+v+/29kr5V23a19v9PzOwfNN+tTiCWryOSzq/73e995z3miJktk7RC0vGYz0W+mj6+br7G0puS5Jzba2YvSvoFSZMtbzXiSvMeDPys7hXdOjS5Q5I38+IWSV+L+8Tam/5pSd6sjkTPR1vEOb47JV1tZitriZ9XS9ppZsvM7J2SZGZlSR+V9C9taDPCPSvpgtqM5bM0n3y/o+Ex9cf9Rkm7au/XHZJuqs26WyfpAknfaVO7EU/Tx9fMVnuTL8zs/Zo/vi+1qd2IJ87xDeL7Wd2idhaTc67r/mk+r+ApSS/U/l9Vu31I0hfqHvdtScckzWg+Kt9Su/39mv8gPyTpEUnL8/6b+NfU8f0vtWN4SNJ/rt12jqS9kr4n6YCkv5RUyvtv4p+TpN+U9H8lvSjpM7Xb7pF0be3ns2vvx0O19+f76577mdrzDkr6jbz/Fv5ld3wl3VB7r+6X9F1JH8v7b+FfU8f3l2rX2TckvSbpQN1zl3xW99I/KusDAADkpFuHJgEAAAqPQAwAACAnBGIAAAA5IRADAADICYEYAABATgjEAAAAckIgBgAAkBMCMQAAgJz8f0nMj23zqwo1AAAAAElFTkSuQmCC\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# Compute percent change using pct_change()\n",
"returns = stocks_and_bonds.pct_change()\n",
"\n",
"# Compute correlation using corr()\n",
"correlation = returns['SP500'].corr(returns['US10Y'])\n",
"print(\"Correlation of stocks and interest rates: \", correlation)\n",
"\n",
"# Make scatter plot\n",
"plt.scatter(returns['SP500'], returns['US10Y']);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The positive correlation means that when interest rates go down, stock prices go down. For example, during crises like 9/11, investors sold stocks and moved their money to less risky bonds (this is sometimes referred to as a 'flight to quality'). During these periods, stocks drop and interest rates drop as well. Of course, there are times when the opposite relationship holds too."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Flying Saucers Aren't Correlated to Flying Markets\n",
"Two trending series may show a strong correlation even if they are completely unrelated. This is referred to as \"spurious correlation\". That's why when you look at the correlation of say, two stocks, you should look at the correlation of their returns and not their levels.\n",
"\n",
"To illustrate this point, calculate the correlation between the levels of the stock market and the annual sightings of UFOs. Both of those time series have trended up over the last several decades, and the correlation of their levels is very high. Then calculate the correlation of their percent changes. This will be close to zero, since there is no relationship between those two series.\n",
"\n",
"UFO data was downloaded from [www.nuforc.org](www.nuforc.org)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Preprocess"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"levels.plot(grid=True);\n",
"plt.xlabel('Year');\n",
"plt.ylabel('Dow Jones Average/Number of Sightings')\n",
"plt.savefig('../images/dji_ufo.png')"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Correlation of levels: 0.9399762210726428\n",
"Correlation of changes: 0.06026935462405373\n"
]
}
],
"source": [
"# Compute correlation o f levels\n",
"correlation1 = levels['DJI'].corr(levels['UFO'])\n",
"print(\"Correlation of levels: \", correlation1)\n",
"\n",
"# Compute correlation fo percent changes\n",
"changes = levels.pct_change()\n",
"correlation2 = changes['DJI'].corr(changes['UFO'])\n",
"print(\"Correlation of changes: \", correlation2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Simple Linear Regression\n",
"- What is a Regression?\n",
" - Simple linear regression: $y_t = \\alpha + \\beta x_t + \\epsilon_t$\n",
"- Relationship between R-Squared and Correlation\n",
" - $[corr(x, y)]^2 = R^2$\n",
" - $sign(corr) = sign(\\text{regression slope})$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Looking at a Regression's R-Squared\n",
"R-squared measures how closely the data fit the regression line, so the R-squared in a simple regression is related to the correlation between the two variables. In particular, the magnitude of the correlation is the square root of the R-squared and the sign of the correlation is the sign of the regression coefficient.\n",
"\n",
"In this exercise, you will start using the statistical package ```statsmodels```, which performs much of the statistical modeling and testing that is found in R and software packages like SAS and MATLAB.\n",
"\n",
"You will take two series, ```x``` and ```y```, compute their correlation, and then regress ```y``` on ```x``` using the function ```OLS(y,x)``` in the ```statsmodels.api``` library (note that the dependent, or right-hand side variable y is the first argument). Most linear regressions contain a constant term which is the intercept (the $\\alpha$ in the regression $y_t = \\alpha + \\beta x_t + \\epsilon_t$). To include a constant using the function ```OLS()```, you need to add a column of 1's to the right hand side of the regression."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Preprocess"
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {},
"outputs": [],
"source": [
"df_x = pd.read_csv('./dataset/x.csv', index_col=0, header=None)\n",
"df_y = pd.read_csv('./dataset/y.csv', index_col=0, header=None)\n",
"\n",
"df_x.columns = ['x']\n",
"df_y.columns = ['y']\n",
"\n",
"x = df_x.reset_index(drop=True)['x']\n",
"y = df_y.reset_index(drop=True)['y']"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0 -0.835129\n",
"1 -0.061004\n",
"2 -0.194677\n",
"3 -2.461142\n",
"4 1.040073\n",
" ... \n",
"995 -1.017080\n",
"996 -0.430943\n",
"997 1.989779\n",
"998 -1.171907\n",
"999 -1.565902\n",
"Name: y, Length: 1000, dtype: float64"
]
},
"execution_count": 62,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y"
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The correlation between x and y is -0.90\n",
" OLS Regression Results \n",
"==============================================================================\n",
"Dep. Variable: y R-squared: 0.818\n",
"Model: OLS Adj. R-squared: 0.817\n",
"Method: Least Squares F-statistic: 4471.\n",
"Date: Sun, 07 Jun 2020 Prob (F-statistic): 0.00\n",
"Time: 20:39:54 Log-Likelihood: -560.10\n",
"No. Observations: 1000 AIC: 1124.\n",
"Df Residuals: 998 BIC: 1134.\n",
"Df Model: 1 \n",
"Covariance Type: nonrobust \n",
"==============================================================================\n",
" coef std err t P>|t| [0.025 0.975]\n",
"------------------------------------------------------------------------------\n",
"const -0.0052 0.013 -0.391 0.696 -0.032 0.021\n",
"x -0.9080 0.014 -66.869 0.000 -0.935 -0.881\n",
"==============================================================================\n",
"Omnibus: 0.048 Durbin-Watson: 2.066\n",
"Prob(Omnibus): 0.976 Jarque-Bera (JB): 0.103\n",
"Skew: -0.003 Prob(JB): 0.950\n",
"Kurtosis: 2.951 Cond. No. 1.03\n",
"==============================================================================\n",
"\n",
"Warnings:\n",
"[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n"
]
}
],
"source": [
"import statsmodels.api as sm\n",
"\n",
"# Compute correlation of x and y\n",
"correlation = x.corr(y)\n",
"print(\"The correlation between x and y is %4.2f\" % (correlation))\n",
"\n",
"# Convert the Series x to a DataFrame and name the column x\n",
"dfx = pd.DataFrame(x, columns=['x'])\n",
"\n",
"# Add a constant to the DataFrame dfx\n",
"dfx1 = sm.add_constant(dfx)\n",
"\n",
"# Regress y on dfx1\n",
"result = sm.OLS(y, dfx1).fit()\n",
"\n",
"# Print out the results and look at the relationship between R-squared and the correlation above\n",
"print(result.summary())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Autocorrelation\n",
"- Correlation of a time series with a lagged copy of itself\n",
"- Lag-one autocorrelation\n",
"- Also called **serial correlation**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### A Popular Strategy Using Autocorrelation\n",
"One puzzling anomaly with stocks is that investors tend to overreact to news. Following large jumps, either up or down, stock prices tend to reverse. This is described as mean reversion in stock prices: prices tend to bounce back, or revert, towards previous levels after large moves, which are observed over time horizons of about a week. A more mathematical way to describe mean reversion is to say that stock returns are negatively autocorrelated.\n",
"\n",
"This simple idea is actually the basis for a popular hedge fund strategy. If you're curious to learn more about this hedge fund strategy (although it's not necessary reading for anything else later in the course), see [here](https://www.quantopian.com/posts/enhancing-short-term-mean-reversion-strategies-1).\n",
"\n",
"You'll look at the autocorrelation of weekly returns of MSFT stock from 2012 to 2017. You'll start with a DataFrame ```MSFT``` of daily prices. You should use the ```.resample()``` method to get weekly prices and then compute returns from prices. Use the pandas method ```.autocorr()``` to get the autocorrelation and show that the autocorrelation is negative. Note that the ```.autocorr()``` method only works on Series, not DataFrames (even DataFrames with one column), so you will have to select the column in the DataFrame."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Preprocess"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
Adj Close
\n",
"
\n",
"
\n",
"
Date
\n",
"
\n",
"
\n",
" \n",
" \n",
"
\n",
"
2012-08-06
\n",
"
26.107651
\n",
"
\n",
"
\n",
"
2012-08-07
\n",
"
26.377876
\n",
"
\n",
"
\n",
"
2012-08-08
\n",
"
26.438896
\n",
"
\n",
"
\n",
"
2012-08-09
\n",
"
26.587088
\n",
"
\n",
"
\n",
"
2012-08-10
\n",
"
26.517351
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Adj Close\n",
"Date \n",
"2012-08-06 26.107651\n",
"2012-08-07 26.377876\n",
"2012-08-08 26.438896\n",
"2012-08-09 26.587088\n",
"2012-08-10 26.517351"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"MSFT = pd.read_csv('./dataset/MSFT.csv', index_col=0)\n",
"MSFT.index = pd.to_datetime(MSFT.index, format=\"%m/%d/%Y\")\n",
"MSFT.head()"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The auto correlation of weekly returns is -0.16\n"
]
}
],
"source": [
"# Convert the daily data to weekly data\n",
"MSFT = MSFT.resample(rule='W').last()\n",
"\n",
"# Compute the percentage change of prices\n",
"returns = MSFT.pct_change()\n",
"\n",
"# Compute and print the autocorrelation of returns\n",
"autocorrelation = returns['Adj Close'].autocorr()\n",
"print('The auto correlation of weekly returns is %4.2f' % (autocorrelation))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Are Interest Rates Autocorrelated?\n",
"When you look at daily changes in interest rates, the autocorrelation is close to zero. However, if you resample the data and look at annual changes, the autocorrelation is negative. This implies that while short term changes in interest rates may be uncorrelated, long term changes in interest rates are negatively autocorrelated. A daily move up or down in interest rates is unlikely to tell you anything about interest rates tomorrow, but a move in interest rates over a year can tell you something about where interest rates are going over the next year. And this makes some economic sense: over long horizons, when interest rates go up, the economy tends to slow down, which consequently causes interest rates to fall, and vice versa."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Preprocess"
]
},
{
"cell_type": "code",
"execution_count": 72,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
US10Y
\n",
"
\n",
"
\n",
"
DATE
\n",
"
\n",
"
\n",
" \n",
" \n",
"
\n",
"
1962-01-02
\n",
"
4.06
\n",
"
\n",
"
\n",
"
1962-01-03
\n",
"
4.03
\n",
"
\n",
"
\n",
"
1962-01-04
\n",
"
3.99
\n",
"
\n",
"
\n",
"
1962-01-05
\n",
"
4.02
\n",
"
\n",
"
\n",
"
1962-01-08
\n",
"
4.03
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" US10Y\n",
"DATE \n",
"1962-01-02 4.06\n",
"1962-01-03 4.03\n",
"1962-01-04 3.99\n",
"1962-01-05 4.02\n",
"1962-01-08 4.03"
]
},
"execution_count": 72,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"daily_rates = pd.read_csv('./dataset/daily_rates.csv', index_col=0, parse_dates=['DATE'])\n",
"daily_rates.head()"
]
},
{
"cell_type": "code",
"execution_count": 75,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The autocorrelation of daily interest rate changes is 0.07\n",
"The autocorrelation of annual interest rate changes is -0.22\n"
]
}
],
"source": [
"# Compute the daily change in interest rates\n",
"daily_diff = daily_rates.diff()\n",
"\n",
"# Compute and print the autocorrelation of daily changes\n",
"autocorrelation_daily = daily_diff['US10Y'].autocorr()\n",
"print(\"The autocorrelation of daily interest rate changes is %4.2f\" %(autocorrelation_daily))\n",
"\n",
"# Convert the daily data to annual data\n",
"yearly_rates = daily_rates.resample(rule='A').last()\n",
"\n",
"# Repeat above for annual data\n",
"yearly_diff = yearly_rates.diff()\n",
"autocorrelation_yearly = yearly_diff['US10Y'].autocorr()\n",
"print(\"The autocorrelation of annual interest rate changes is %4.2f\" %(autocorrelation_yearly))"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 4
}