{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Factor Analysis\n",
"\n",
"by Maxwell Margenot, Gil Wassermann, James Christopher Hall, and Delaney Granizo-Mackenzie.\n",
"\n",
"Part of the Quantopian Lecture Series:\n",
"\n",
"* [www.quantopian.com/lectures](https://www.quantopian.com/lectures)\n",
"* [https://github.com/quantopian/research_public](https://github.com/quantopian/research_public)\n",
"\n",
"Notebook released under the Creative Commons Attribution 4.0 License. Please do not remove this attribution.\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"How can we tell whether an alpha factor is good or not? Unfortunately, there is no cut-off that tells you whether a factor is objectively useful. Instead, we need to compare a particular factor to other options before deciding whether to use it. Our end goal in defining and selecting the best factors is to use them to rank stocks in a long-short equity strategy, covered elsewhere in the [lecture series](https://www.quantopian.com/lectures/long-short-equity). The more independent predictive the factors we use, the better our ranking scheme and our overall strategy will be.\n",
"\n",
"\n",
"What we want when comparing factors is to make sure the chosen signal is actually predictive of **relative price movements**. We do not want to predict the absolute amount the assets in our universe will move up or down. We only care that we can choose assets to long that will do better than the assets we short. In a long-short equity strategy, we hold a long basket and a short basket of assets, determined by the factor values associated with each asset in our universe. If our ranking scheme is predictive, this means that assets in the top basket will tend to outperform assets in the bottom basket. As long this spread is consistent over time our strategy will have a positive return.\n",
"\n",
"\n",
"An individual factor can have a lot of moving parts to assess, but ideally it should be independent of other factors that you are already trading on in order to keep your portfolio diverse. We discuss the reasoning for this in [this lecture](https://www.quantopian.com/lectures/position-concentration-risk). \n",
"\n",
"\n",
"In this lecture, we detail and explain relevant statistics to evaluate your alpha factor before attempting to implement it in an algorithm. What's important to keep in mind is that all the metrics provided here are relative to other factors you may be trading or evaluating. \n",
"\n",
"\n",
"Let's have a look at a factor and try to assess its viability. We will calculate the factor values using Pipeline, so make sure you check out the [tutorial](https://www.quantopian.com/tutorials/pipeline) if you are unfamiliar with how Pipeline works."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"from quantopian.research import run_pipeline\n",
"from quantopian.pipeline import Pipeline\n",
"from quantopian.pipeline.data.builtin import USEquityPricing\n",
"from quantopian.pipeline.factors import CustomFactor, Returns\n",
"from quantopian.pipeline.classifiers.morningstar import Sector\n",
"from quantopian.pipeline.filters import Q1500US\n",
"from time import time"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Momentum\n",
"\n",
"Here we will be using a **momentum** factor as our example. Momentum factors are a very common form of alpha factor and they come in many shapes and sizes. They all try to get at the same idea, however, that securities in motion will stay in motion. Momentum factors try to quantify trends in financial markets and to \"ride the wave\", so to speak.\n",
"\n",
"Let's say that we suspect that a momentum factor could potentially be predictive of stock returns. We define it as a `CustomFactor` so that we can easily pull its values when we run our Pipeline. We should get a factor value for every security in our universe."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"class MyFactor(CustomFactor):\n",
" \"\"\" Momentum factor \"\"\"\n",
" inputs = [USEquityPricing.close,\n",
" Returns(window_length=126)]\n",
" window_length = 252\n",
"\n",
" def compute(self, today, assets, out, prices, returns):\n",
" out[:] = ((prices[-21] - prices[-252])/prices[-252] -\n",
" (prices[-1] - prices[-21])/prices[-21]) / np.nanstd(returns, axis=0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This momentum factor takes the change in price over the past year, up until a month ago, and standardizes it over the the change in price over the last month. This allows it to account for any changes over the past month and use them to temper its expectations of future movement."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Judging a Factor with Alphalens\n",
"\n",
"In order to judge whether a factor is viable, we have created a package called Alphalens. Its source code is available on [github](http://github.com/quantopian/alphalens) if you want to get into the nitty-gritty of how it works. We use Alphalens to create a \"tear sheet\" of a factor, similar to how we use [pyfolio](http://github.com/quantopian/pyfolio) to create a tear sheet for analyzing backtests."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import alphalens as al"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alphalens takes your factor and examines how useful it is for predicting relative value through a collection of different metrics. It breaks all the stocks in your chosen universe into different quantiles based on their ranking according to your factor and analyzes the returns, information coefficient (IC), the turnover of each quantile, and provides a breakdown of returns and IC by sector.\n",
"\n",
"Throughout the course of this lecture we will detail how to interpret the various individual plots generated by an `Alphalens` tear sheet and include the proper call to generate the whole tear sheet at once at the end."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sector Codes\n",
"\n",
"These are the possible sector codes for each security, as given by Morningstar. We will use this dictionary to help categorize our results as we walk through a factor analysis so that we can break out our information by sector."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"MORNINGSTAR_SECTOR_CODES = {\n",
" -1: 'Misc',\n",
" 101: 'Basic Materials',\n",
" 102: 'Consumer Cyclical',\n",
" 103: 'Financial Services',\n",
" 104: 'Real Estate',\n",
" 205: 'Consumer Defensive',\n",
" 206: 'Healthcare',\n",
" 207: 'Utilities',\n",
" 308: 'Communication Services',\n",
" 309: 'Energy',\n",
" 310: 'Industrials',\n",
" 311: 'Technology' , \n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Defining a universe\n",
"\n",
"As always, we need to define our universe. In this case we use the Q1500US, as seen in the forums [here](https://www.quantopian.com/posts/the-q500us-and-q1500us)."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"universe = Q1500US()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Getting Data\n",
"\n",
"Now we will pull values for our factor for all stocks in our universe by using Pipeline. We also want to make sure that we have the sector code for each individual equity, so we add `Sector` as another factor for our Pipeline. Note that running the Pipeline may take a while."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:1147: RuntimeWarning: Degrees of freedom <= 0 for slice.\n",
" warnings.warn(\"Degrees of freedom <= 0 for slice.\", RuntimeWarning)\n"
]
}
],
"source": [
"pipe = Pipeline(\n",
" columns = {\n",
" 'MyFactor' : MyFactor(mask=universe),\n",
" 'Sector' : Sector()\n",
" },\n",
" screen=universe\n",
")\n",
"\n",
"start_timer = time()\n",
"results = run_pipeline(pipe, '2015-01-01', '2016-01-01')\n",
"end_timer = time()\n",
"results.fillna(value=0);"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Time to run pipeline 71.34 secs\n"
]
}
],
"source": [
"print \"Time to run pipeline %.2f secs\" % (end_timer - start_timer)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's take a look at the data to get a quick sense of what we have."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2015-01-02 00:00:00+00:00 Equity(2 [ARNC]) 5.239913\n",
" Equity(24 [AAPL]) 5.451305\n",
" Equity(41 [ARCB]) 1.155913\n",
" Equity(62 [ABT]) 4.818421\n",
" Equity(67 [ADSK]) 1.724293\n",
"Name: MyFactor, dtype: float64\n"
]
}
],
"source": [
"my_factor = results['MyFactor']\n",
"print my_factor.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our `my_factor` variable contains a pandas `Series` with a factor value for each equity in our universe for each point in time.\n",
"\n",
"Here we create another `Series` that contains sector codes for each equity instead of factor values. This is categorical data that we will use as a parameter for `Alphalens` later."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"sectors = results['Sector']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"While our universe is defined to consist of 1500 stocks, the actual number of unique stocks that we end up ranking will likely be greater than this due to stocks passing in and out of our filters. For this reason, we grab pricing data for any stock that may have been in our Pipeline at some point to make sure that we have all of the data that we might need."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"asset_list = results.index.levels[1].unique()"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"prices = get_pricing(asset_list, start_date='2015-01-01', end_date='2016-02-01', fields='price')"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false,
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Equity(2 [ARNC]) | \n",
" Equity(24 [AAPL]) | \n",
" Equity(41 [ARCB]) | \n",
" Equity(53 [ABMD]) | \n",
" Equity(62 [ABT]) | \n",
" Equity(67 [ADSK]) | \n",
" Equity(76 [TAP]) | \n",
" Equity(110 [ACXM]) | \n",
" Equity(114 [ADBE]) | \n",
" Equity(122 [ADI]) | \n",
" ... | \n",
" Equity(48925 [ADRO]) | \n",
" Equity(48934 [ETSY]) | \n",
" Equity(48962 [UNIT]) | \n",
" Equity(49003 [TLN]) | \n",
" Equity(49051 [APLE]) | \n",
" Equity(49060 [SHOP]) | \n",
" Equity(49141 [CPGX]) | \n",
" Equity(49176 [TRU]) | \n",
" Equity(49229 [KHC]) | \n",
" Equity(49496 [FDC]) | \n",
"
\n",
" \n",
" \n",
" \n",
" | 2015-01-02 00:00:00+00:00 | \n",
" 15.717 | \n",
" 107.469 | \n",
" 45.513 | \n",
" 37.30 | \n",
" 43.701 | \n",
" 59.53 | \n",
" 72.300 | \n",
" 19.605 | \n",
" 72.33 | \n",
" 54.041 | \n",
" ... | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
"
\n",
" \n",
" | 2015-01-05 00:00:00+00:00 | \n",
" 14.817 | \n",
" 104.470 | \n",
" 44.839 | \n",
" 37.09 | \n",
" 43.721 | \n",
" 58.66 | \n",
" 71.889 | \n",
" 19.425 | \n",
" 71.99 | \n",
" 53.058 | \n",
" ... | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
"
\n",
" \n",
" | 2015-01-06 00:00:00+00:00 | \n",
" 14.906 | \n",
" 104.451 | \n",
" 42.805 | \n",
" 36.13 | \n",
" 43.205 | \n",
" 57.50 | \n",
" 71.527 | \n",
" 19.080 | \n",
" 70.52 | \n",
" 51.812 | \n",
" ... | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
"
\n",
" \n",
" | 2015-01-07 00:00:00+00:00 | \n",
" 15.302 | \n",
" 105.945 | \n",
" 41.734 | \n",
" 37.28 | \n",
" 43.565 | \n",
" 57.37 | \n",
" 73.807 | \n",
" 19.330 | \n",
" 71.12 | \n",
" 52.357 | \n",
" ... | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
"
\n",
" \n",
" | 2015-01-08 00:00:00+00:00 | \n",
" 15.757 | \n",
" 109.996 | \n",
" 42.716 | \n",
" 38.96 | \n",
" 44.451 | \n",
" 58.80 | \n",
" 76.088 | \n",
" 19.790 | \n",
" 72.91 | \n",
" 53.281 | \n",
" ... | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
" NaN | \n",
"
\n",
" \n",
"
\n",
"
5 rows × 1735 columns
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" | \n",
" 1 | \n",
" 5 | \n",
" 10 | \n",
" factor | \n",
" group | \n",
" factor_quantile | \n",
"
\n",
" \n",
" | date | \n",
" asset | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 2015-01-02 00:00:00+00:00 | \n",
" Equity(2 [ARNC]) | \n",
" -0.057263 | \n",
" 0.014507 | \n",
" -0.038366 | \n",
" 5.239913 | \n",
" Basic Materials | \n",
" 5 | \n",
"
\n",
" \n",
" | Equity(24 [AAPL]) | \n",
" -0.027906 | \n",
" 0.024705 | \n",
" -0.030372 | \n",
" 5.451305 | \n",
" Technology | \n",
" 5 | \n",
"
\n",
" \n",
" | Equity(41 [ARCB]) | \n",
" -0.014809 | \n",
" -0.095028 | \n",
" -0.084350 | \n",
" 1.155913 | \n",
" Industrials | \n",
" 3 | \n",
"
\n",
" \n",
" | Equity(62 [ABT]) | \n",
" 0.000458 | \n",
" 0.006682 | \n",
" -0.004325 | \n",
" 4.818421 | \n",
" Healthcare | \n",
" 5 | \n",
"
\n",
" \n",
" | Equity(67 [ADSK]) | \n",
" -0.014614 | \n",
" -0.020998 | \n",
" -0.045187 | \n",
" 1.724293 | \n",
" Technology | \n",
" 4 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" | \n",
" 1 | \n",
" 5 | \n",
" 10 | \n",
"
\n",
" \n",
" | factor_quantile | \n",
" date | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 1 | \n",
" 2015-01-02 00:00:00+00:00 | \n",
" -0.010211 | \n",
" -0.021515 | \n",
" -0.034468 | \n",
"
\n",
" \n",
" | 2015-01-05 00:00:00+00:00 | \n",
" -0.005019 | \n",
" -0.015829 | \n",
" -0.030863 | \n",
"
\n",
" \n",
" | 2015-01-06 00:00:00+00:00 | \n",
" -0.002690 | \n",
" -0.014040 | \n",
" -0.015068 | \n",
"
\n",
" \n",
" | 2015-01-07 00:00:00+00:00 | \n",
" 0.001770 | \n",
" -0.009864 | \n",
" -0.015686 | \n",
"
\n",
" \n",
" | 2015-01-08 00:00:00+00:00 | \n",
" -0.000807 | \n",
" -0.021366 | \n",
" -0.018793 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" 1 | \n",
" 5 | \n",
" 10 | \n",
"
\n",
" \n",
" | factor_quantile | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 1 | \n",
" -0.000564 | \n",
" -0.002738 | \n",
" -0.005741 | \n",
"
\n",
" \n",
" | 2 | \n",
" -0.000163 | \n",
" -0.000484 | \n",
" -0.000488 | \n",
"
\n",
" \n",
" | 3 | \n",
" 0.000137 | \n",
" 0.000673 | \n",
" 0.001225 | \n",
"
\n",
" \n",
" | 4 | \n",
" 0.000247 | \n",
" 0.000962 | \n",
" 0.001714 | \n",
"
\n",
" \n",
" | 5 | \n",
" 0.000343 | \n",
" 0.001591 | \n",
" 0.003296 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" 1 | \n",
" 5 | \n",
" 10 | \n",
"
\n",
" \n",
" | date | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 2015-01-02 00:00:00+00:00 | \n",
" 0.007513 | \n",
" 0.016455 | \n",
" 0.029473 | \n",
"
\n",
" \n",
" | 2015-01-05 00:00:00+00:00 | \n",
" 0.004336 | \n",
" 0.011862 | \n",
" 0.024210 | \n",
"
\n",
" \n",
" | 2015-01-06 00:00:00+00:00 | \n",
" 0.002294 | \n",
" 0.011022 | \n",
" 0.015213 | \n",
"
\n",
" \n",
" | 2015-01-07 00:00:00+00:00 | \n",
" 0.000154 | \n",
" 0.007648 | \n",
" 0.013604 | \n",
"
\n",
" \n",
" | 2015-01-08 00:00:00+00:00 | \n",
" 0.001380 | \n",
" 0.016175 | \n",
" 0.014856 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" 1 | \n",
" 5 | \n",
" 10 | \n",
"
\n",
" \n",
" \n",
" \n",
" | Ann. alpha | \n",
" 0.099923 | \n",
" 0.082827 | \n",
" 0.077657 | \n",
"
\n",
" \n",
" | beta | \n",
" -0.070473 | \n",
" -0.189530 | \n",
" -0.200049 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" 1 | \n",
" 5 | \n",
" 10 | \n",
"
\n",
" \n",
" | date | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 2015-01-02 00:00:00+00:00 | \n",
" 0.305250 | \n",
" 0.345014 | \n",
" 0.375594 | \n",
"
\n",
" \n",
" | 2015-01-05 00:00:00+00:00 | \n",
" 0.253169 | \n",
" 0.265510 | \n",
" 0.328463 | \n",
"
\n",
" \n",
" | 2015-01-06 00:00:00+00:00 | \n",
" 0.175803 | \n",
" 0.205640 | \n",
" 0.230546 | \n",
"
\n",
" \n",
" | 2015-01-07 00:00:00+00:00 | \n",
" -0.018505 | \n",
" 0.144226 | \n",
" 0.196403 | \n",
"
\n",
" \n",
" | 2015-01-08 00:00:00+00:00 | \n",
" 0.062513 | \n",
" 0.287120 | \n",
" 0.214292 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" 1 | \n",
" 5 | \n",
" 10 | \n",
"
\n",
" \n",
" | group | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | Basic Materials | \n",
" 0.031098 | \n",
" 0.047993 | \n",
" 0.061131 | \n",
"
\n",
" \n",
" | Communication Services | \n",
" 0.005562 | \n",
" 0.035027 | \n",
" 0.038822 | \n",
"
\n",
" \n",
" | Consumer Cyclical | \n",
" 0.023081 | \n",
" 0.037877 | \n",
" 0.043151 | \n",
"
\n",
" \n",
" | Consumer Defensive | \n",
" 0.023285 | \n",
" 0.044498 | \n",
" 0.053996 | \n",
"
\n",
" \n",
" | Energy | \n",
" 0.031482 | \n",
" 0.068066 | \n",
" 0.090729 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" | \n",
" 1 | \n",
" 5 | \n",
" 10 | \n",
"
\n",
" \n",
" | factor_quantile | \n",
" group | \n",
" | \n",
" | \n",
" | \n",
"
\n",
" \n",
" \n",
" \n",
" | 1 | \n",
" Basic Materials | \n",
" -0.001323 | \n",
" -0.007757 | \n",
" -0.015815 | \n",
"
\n",
" \n",
" | Communication Services | \n",
" -0.000023 | \n",
" -0.001349 | \n",
" -0.002273 | \n",
"
\n",
" \n",
" | Consumer Cyclical | \n",
" -0.000266 | \n",
" -0.001632 | \n",
" -0.003109 | \n",
"
\n",
" \n",
" | Consumer Defensive | \n",
" 0.000534 | \n",
" 0.002763 | \n",
" 0.005328 | \n",
"
\n",
" \n",
" | Energy | \n",
" -0.001601 | \n",
" -0.007050 | \n",
" -0.015111 | \n",
"
\n",
" \n",
"
\n",
"