{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Regulome Explorer 2.X\n", "Comparison of the average gene expression of two groups based on clinical data (catageorical)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Authenticate with Google (IMPORTANT)\n", "Our first step is to authenticate with Google -- you will need to be a member of a Google Cloud Platform (GCP) project, with authorization to run BigQuery jobs in order to run this notebook. If you don't have access to a GCP project, please contact the ISB-CGC team for help (www.isb-cgc.org)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Import Python libraries" ] }, { "cell_type": "code", "execution_count": 160, "metadata": {}, "outputs": [], "source": [ "from google.cloud import bigquery\n", "import numpy as np\n", "import pandas as pd\n", "from scipy import stats" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Python convenience functions\n", "\n", "- **`runQuery`**: a relatively generic BigQuery query-execution wrapper function which can be used to run a query in \"dry-run\" mode or not: the call to the `query()` function itself is inside a `try/except` block and if it fails we return `None`; otherwise a \"dry\" will return an empty dataframe, and a \"live\" run will return the query results as a dataframe. This function was modify from previous notebooks to handle user-defined parameteres necessary for the purpose of this notbeook.\n", "\n" ] }, { "cell_type": "code", "execution_count": 161, "metadata": {}, "outputs": [], "source": [ "def runQuery ( client, qString, ParameterList1, ParameterList2, ParameterList3, dryRun=False ):\n", " \n", " print ( \"\\n in runQuery ... \" )\n", " if ( dryRun ):\n", " print ( \" dry-run only \" )\n", " \n", " ## set up QueryJobConfig object\n", " job_config = bigquery.QueryJobConfig()\n", " \n", " query_params = [\n", " bigquery.ArrayQueryParameter('PARAMETERLIST1', 'STRING', ParameterList1 ),\n", " bigquery.ArrayQueryParameter('PARAMETERLIST2', 'STRING', ParameterList2 ),\n", " bigquery.ArrayQueryParameter('PARAMETERLIST3', 'STRING', ParameterList3 )\n", " ]\n", " job_config.query_parameters = query_params \n", " \n", " job_config.dry_run = dryRun\n", " job_config.use_query_cache = True\n", " job_config.use_legacy_sql = False\n", " \n", " ## run the query\n", " try:\n", " query_job = client.query ( qString, job_config=job_config )\n", " ## print ( \" query job state: \", query_job.state )\n", " except:\n", " print ( \" FATAL ERROR: query execution failed \" )\n", " return ( None )\n", " \n", " ## return results as a dataframe (or an empty dataframe for a dry-run) \n", " if ( not dryRun ):\n", " try:\n", " df = query_job.to_dataframe()\n", " if ( query_job.total_bytes_processed==0 ):\n", " print ( \" the results for this query were previously cached \" )\n", " else:\n", " print ( \" this query processed {} bytes \".format(query_job.total_bytes_processed) )\n", " if ( len(df) < 1 ):\n", " print ( \" WARNING: this query returned NO results \")\n", " return ( df )\n", " except:\n", " print ( \" FATAL ERROR: query execution failed \" )\n", " return ( None )\n", " \n", " else:\n", " print ( \" if not cached, this query will process {} bytes \".format(query_job.total_bytes_processed) )\n", " ## return an empty dataframe\n", " return ( pd.DataFrame() )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### SQL Building Functions\n", "\n", "\n", "**`build_expdata`**. This function outputs a query string to create a table with three columns: patient ID, gene expresion, and gene name, for a user defined list of genes (PARAMETERLIST1) or all genes with gene expression data. Since a patient can have multiple samples, the gene expression value for each patient is averaged over the respective samples. " ] }, { "cell_type": "code", "execution_count": 162, "metadata": {}, "outputs": [], "source": [ "def build_expdata ( study ):\n", " qString = \"\"\"\n", " expdata AS (\n", " SELECT \n", " participantBarcode as patientid,\n", " Symbol,\n", " AVG( LOG10( normalized_count +1 ) ) as gene_exp\n", " FROM\n", " `pancancer-atlas.Filtered.EBpp_AdjustPANCAN_IlluminaHiSeq_RNASeqV2_genExp_filtered` \n", " WHERE\n", " #study = '__study__' AND Symbol IN UNNEST(@PARAMETERLIST1)\n", " study = '__study__' AND Symbol IS NOT NULL\n", " GROUP BY ParticipantBarcode, Symbol\n", " )\"\"\".replace('__study__',study)\n", " return qString\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**`build_statdata`**. This function outputs a query string to create a table with four columns: gene_name, category, n, sumx, and sumx2. n is the number of patients with the same clinical category, sumx (sumx2) is the sum of the gene expression (squared) data over all patients with the same category. " ] }, { "cell_type": "code", "execution_count": 163, "metadata": {}, "outputs": [], "source": [ "def build_statdata( category ):\n", " qString =\"\"\"\n", " statdata AS (\n", " SELECT\n", " A.Symbol,\n", " B.__category__ as category,\n", " COUNT( A.patientid ) as n,\n", " SUM( A.gene_exp ) as sumx,\n", " SUM( A.gene_exp * A.gene_exp ) as sumx2\n", " FROM\n", " expdata A\n", " INNER JOIN `pancancer-atlas.Filtered.clinical_PANCAN_patient_with_followup_filtered` B\n", " ON A.patientid = B.bcr_patient_barcode\n", " GROUP BY\n", " B.__category__, A.Symbol \n", " )\"\"\".replace('__category__', category) \n", " return qString " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**`build_groupdate`**. Outputs a query string that summarize the statistical data for the desired catageory values that are specified for group1 and group 2. This catagory values are defined by a couple parameter lists, PARAMETERLIST2 and PARAMETERLIST3 for group 1 and group 2 respectively." ] }, { "cell_type": "code", "execution_count": 164, "metadata": {}, "outputs": [], "source": [ "def build_groupdate():\n", " qString =\"\"\"\n", " group1 AS (\n", " SELECT Symbol, SUM( n ) as n, SUM( sumx ) as sumx, SUM( sumx2 ) as sumx2\n", " FROM statdata \n", " WHERE category IN UNNEST(@PARAMETERLIST2)\n", " GROUP BY Symbol \n", " ),\n", " group2 AS (\n", " SELECT Symbol, SUM( n ) as n, SUM( sumx ) as sumx, SUM( sumx2 ) as sumx2\n", " FROM statdata \n", " WHERE category IN UNNEST(@PARAMETERLIST3)\n", " GROUP BY Symbol \n", " )\n", " \"\"\" \n", " return qString" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Start the analysis" ] }, { "cell_type": "code", "execution_count": 165, "metadata": {}, "outputs": [], "source": [ "# Start by getting authorized.\n", "bqclient = bigquery.Client()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Specify Parameters" ] }, { "cell_type": "code", "execution_count": 166, "metadata": {}, "outputs": [], "source": [ "study = 'ACC' # Select Tumor type \n", "ClinicFeature1 = 'pathologic_stage' # Clinical feature \n", "#gene_list =['IGF2','ADAM6','GRWD1','PATE3','CASP8'] #gene nems, irrelevant if all genes are evaluated\n", "group1 = [ 'Stage I', 'Stage II'] # Category values to define group 1\n", "group2 = ['Stage III', 'Stage IV'] # Category values to define group 1\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Obtain data from Bigquery\n", "We retreive data for t test evaluation. The ouput of the code is a dataframe with 7 columns corresponding to gene name (Symbol), the number of samples, average expression and variance of the two groups.\n", "This computation can take several seconds." ] }, { "cell_type": "code", "execution_count": 167, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", " in runQuery ... \n", " the results for this query were previously cached \n" ] } ], "source": [ "# Build the sql code\n", "sql = (\n", "\"\"\"WITH\n", "\"\"\"+\n", "build_expdata( study ) + ',\\n' + \n", "build_statdata( ClinicFeature1 ) + ',\\n' +\n", "build_groupdate() + '\\n' + \n", "\"\"\"\n", "SELECT \n", " A.Symbol,\n", " A.n as n1, (A.sumx/A.n) as avg1 , (A.sumx2 - A.sumx*A.sumx/A.n)/(A.n - 1) as var1 ,\n", " B.n as n2, (B.sumx/B.n) as avg2 , (B.sumx2 - B.sumx*B.sumx/B.n)/(B.n - 1) as var2\n", "FROM \n", " group1 A\n", "INNER JOIN\n", " group2 B\n", "ON \n", " A.Symbol = B.Symbol \n", "\"\"\"\n", ")\n", "\n", "# bigquery \n", "res0 = runQuery ( bqclient, sql, gene_list, group1, group2, dryRun=False )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now take a look the computed statistics. We have to make sure that there is not NUL values which may indicate that a group has no elements." ] }, { "cell_type": "code", "execution_count": 168, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Symboln1avg1var1n2avg2var2
0TNIP2462.7210170.032786302.7240200.023377
1AP1G1463.3709780.014340303.3468230.029890
2C14orf178460.1620270.030331300.1766040.034447
3UBASH3B461.5666460.122094301.5337230.125481
4PFKFB4462.5333250.232271302.6996640.241280
5POU3F2460.6490670.366423300.8088810.471779
6LOC100132831460.1018040.019569300.1215660.025672
7NKAP462.8064410.020916302.7894370.032605
8C6orf162462.1640800.031893302.1459190.029427
9CXCL10461.3362280.318461301.3425210.510212
\n", "
" ], "text/plain": [ " Symbol n1 avg1 var1 n2 avg2 var2\n", "0 TNIP2 46 2.721017 0.032786 30 2.724020 0.023377\n", "1 AP1G1 46 3.370978 0.014340 30 3.346823 0.029890\n", "2 C14orf178 46 0.162027 0.030331 30 0.176604 0.034447\n", "3 UBASH3B 46 1.566646 0.122094 30 1.533723 0.125481\n", "4 PFKFB4 46 2.533325 0.232271 30 2.699664 0.241280\n", "5 POU3F2 46 0.649067 0.366423 30 0.808881 0.471779\n", "6 LOC100132831 46 0.101804 0.019569 30 0.121566 0.025672\n", "7 NKAP 46 2.806441 0.020916 30 2.789437 0.032605\n", "8 C6orf162 46 2.164080 0.031893 30 2.145919 0.029427\n", "9 CXCL10 46 1.336228 0.318461 30 1.342521 0.510212" ] }, "execution_count": 168, "metadata": {}, "output_type": "execute_result" } ], "source": [ "res0[0:10]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Analyze the results\n", "We can now use python function to the data in the resulting dataframe (res0). For example we can compute t-scores and p-values to determine if the difference between the mean of the two groups is statistically significant" ] }, { "cell_type": "code", "execution_count": 169, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/borisaguilar/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:2: RuntimeWarning: invalid value encountered in double_scalars\n", " \n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Symboln1avg1var1n2avg2var2tscorepvalue
6740DCAF15462.7916520.021481303.0018010.021450-6.1125624.211413e-08
2810C14orf80461.9722420.065065302.3273490.060538-6.0612335.213308e-08
5810HAUS8462.0228370.103465302.3989310.062825-5.7066522.243011e-07
11548CENPJ461.9936530.035947302.2184510.026015-5.5364444.472172e-07
9086DPF1460.7179710.338007301.3416530.161269-5.5291464.605700e-07
17069MCM10461.3904640.201234301.9402070.175986-5.4323956.793837e-07
13006PTCH2461.2379710.119995301.7345550.181423-5.3374409.924677e-07
17115POLD1462.5969900.053311302.8573530.037465-5.3060791.124178e-06
11638TRIP13461.9051450.174519302.4041350.152228-5.2987681.157266e-06
1901UBE2S462.7035920.117995303.0696380.069413-5.2405401.457236e-06
\n", "
" ], "text/plain": [ " Symbol n1 avg1 var1 n2 avg2 var2 tscore \\\n", "6740 DCAF15 46 2.791652 0.021481 30 3.001801 0.021450 -6.112562 \n", "2810 C14orf80 46 1.972242 0.065065 30 2.327349 0.060538 -6.061233 \n", "5810 HAUS8 46 2.022837 0.103465 30 2.398931 0.062825 -5.706652 \n", "11548 CENPJ 46 1.993653 0.035947 30 2.218451 0.026015 -5.536444 \n", "9086 DPF1 46 0.717971 0.338007 30 1.341653 0.161269 -5.529146 \n", "17069 MCM10 46 1.390464 0.201234 30 1.940207 0.175986 -5.432395 \n", "13006 PTCH2 46 1.237971 0.119995 30 1.734555 0.181423 -5.337440 \n", "17115 POLD1 46 2.596990 0.053311 30 2.857353 0.037465 -5.306079 \n", "11638 TRIP13 46 1.905145 0.174519 30 2.404135 0.152228 -5.298768 \n", "1901 UBE2S 46 2.703592 0.117995 30 3.069638 0.069413 -5.240540 \n", "\n", " pvalue \n", "6740 4.211413e-08 \n", "2810 5.213308e-08 \n", "5810 2.243011e-07 \n", "11548 4.472172e-07 \n", "9086 4.605700e-07 \n", "17069 6.793837e-07 \n", "13006 9.924677e-07 \n", "17115 1.124178e-06 \n", "11638 1.157266e-06 \n", "1901 1.457236e-06 " ] }, "execution_count": 169, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# t statistics\n", "f = lambda n1,n2,avg1,avg2,var1,var2 : (avg1 - avg2) / np.sqrt(var1/float(n1) + var2/float(n2) ) \n", "res0['tscore'] = res0.apply(lambda x: f(x.n1,x.n2,x.avg1,x.avg2,x.var1,x.var2), axis=1)\n", "\n", "# p value \n", "f = lambda n1,n2,tscore : (1.0 - stats.t.cdf(abs(tscore), n1 + n2 - 2 )) * 2.0 \n", "res0['pvalue'] = res0.apply(lambda x: f(x.n1,x.n2,x.tscore), axis=1)\n", "\n", "res0.sort_values( by= 'pvalue')[0:10]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can plot the data" ] }, { "cell_type": "code", "execution_count": 170, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 170, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAD8CAYAAACGsIhGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl4XHd97/H3V7tkSbZkS7Jlybsc20nsOCgOEEigZGVJKBBIIJfQ5mmgNL1t6e1zU2iBG9p7We7lLhAgAfI00EIWltRtEkIISYCQUDteEq+xvEiWV21etFiypO/9Y2bMWJE9I2mOzmjm83qeeTRz5pyZbyZjffQ7v+WYuyMiInI+OWEXICIi6U9hISIiCSksREQkIYWFiIgkpLAQEZGEFBYiIpKQwkJERBJSWIiISEIKCxERSSgv7AJSZdasWb5gwYKwyxARmVJefvnldnevSrRfxoTFggULWL9+fdhliIhMKWbWnMx+Og0lIiIJKSxERCQhhYWIiCSksBARkYQUFiIikpDCQkREElJYiIhIQoGGhZldb2Y7zazJzO4e5flPmdk2M3vFzJ4xs/lxzw2Z2abobW2QdYpMBnfn2R1H+dozu/jZlkMMDeuSxjJ1BDYpz8xygXuBa4BWYJ2ZrXX3bXG7bQQa3b3XzP4U+DLwoehzfe5+SVD1iUym3oFB/vqRzTy55fCZbRfNLefbH21kzvTiECsTSU6QLYs1QJO773H3AeAh4Kb4Hdz9WXfvjT58CagLsB6RwB09cYof/K6FTfuPndm2v7OX93/zRZ7aepi7b1jGtnuu4//dupp97b188L4X6ewZCLFikeQEudzHXGB/3ONW4PLz7H8H8GTc4yIzWw8MAl9098dSX6JI6nR09/Pee1/g4PFTAFy+sJJL51fwzy9FVlN44GOX8bYLqgG4cVUt9RXFfOj+l/iLhzbyvT9eg5mFVrtIIkG2LEb75o96ktbMbgMaga/EbZ7n7o3Ah4H/Y2aLRznuTjNbb2br29raUlGzyLh99zd7OXTiFA/+8Ro+++4VNHf08s3ndrOqbgb//udvORMUMavnVfB371rOr3e1s3bzwZCqFklOkC2LVqA+7nEd8Lp/EWZ2NfAZ4Cp3749td/eD0Z97zOw5YDWwO/5Yd78fuB+gsbFRvYUSmuFh55H1rVyzvIarllZx1dIqPvbmBQwMDVOUn3vO4z5y+XweXd/KV57ayTsvnkN+rgYoSnoK8pu5Dmgws4VmVgDcApw1qsnMVgP3ATe6+9G47RVmVhi9Pwu4AojvGBdJK1sOHqe9u58bLp59ZltOjp03KAByc4y/eEcDrV19rN2k1oWkr8DCwt0HgbuAp4DtwCPuvtXM7jGzG6O7fQUoBR4dMUR2ObDezDYDzxLps1BYSNr6TVM7AG9tSHhZgNd5x/JqGqpL+effJbVStEgoAr2ehbs/ATwxYttn4+5ffY7jfgtcHGRtIqm0ef8xFswsYVZp4ZiPNTM+dFk9//D4dpqOnmRJdVkAFYpMjE6QiqTAK63HWVk3Y9zHv3f1XHJzjJ9sOJDCqkRSR2EhMkEd3f0cOn6KlXXTx/0as0oLuXxhJU9vO5LCykRSR2EhMkFNR7sBaKiZ2Omjq5fXsOtoN/vae1JRlkhKKSxEJmh3W+SX++KqaRN6nWtW1ADwi+1qXUj6UViITFDT0W6K83OpneAaT/WVJSybXcYz248m3llkkiksRCZod1s3i6qmkZMz8eU6rlxaxcvNXfQNDKWgMpHUUViITFDT0W6WVJem5LXevHgmA0PDrNvXmZLXE0kVhYXIBPQPDnHweB8LZ02svyJmzcJK8nONF3a3p+T1RFJFYSEyAQePncId6itKUvJ6JQV5rJ5XwQtNCgtJLwoLkQlo7YpcjqWuInUXMLpi8Sy2HjxBl65zIWlEYSEyAa1dfQDUVaamZQHwloaZuMOLezpS9poiE6WwEJmA1q5e8nKMmrKxrwl1LivrZjCtIFenoiStKCxEJqC1q485M4rIS+F1KPJzc3jjoplnVrIVSQcKC5EJaO3qo25G6k5Bxby1YRbNHb00d2jpD0kPCguRCWjt6k1p53bMlUsj18X41Wu6XLCkB4WFyDj1Dw5x5EQ/dSkaNhtv4axp1FcW8/xrOhUl6UFhITJOB4+dAlI7bDbGzLiyoYoXd7czMDic8tcXGSuFhcg47e+MzLGoT+Gw2XhXLa2iZ2CI9c1a+kPCp7AQGaczcywCaFkAvHnJLArycvj5Vi1ZLuFTWIiM05k5FuVFgbx+aWEeb7+giidePcTwsAfyHiLJUliIjFNrVx+1M4rJTcHS5OfyrpW1HD3Zr1VoJXQKC5Fxau3qZe6MYE5BxbxjWTVF+Tk8tulgoO8jkojCQmScWrv6AuuviJlWmMd7Vtby2MYDHO87Heh7iZyPwkJkHPoHhzh6sp+5AYcFwO1vXkDf6SEeXb8/8PcSOReFhcg4HIrOsQj6NBTARXOn84b5FTz44j4GhzTnQsKhsBAZhwPHYsNmg5ljMdKdVy5if2cfj796aFLeT2QkhYXIOARx0aPzuWZ5DQ3VpXzj2d0aRiuhUFiIjMOBrj5yDGZPD2aOxUg5OcYnrlrMziMn+eWOo5PyniLxFBYi49B6rI+a8iLyU3gdi0RuvKSWuTOK+cZzTbirdSGTS2EhMg4HuvompXM7Xn5uDh+/ahEbWo6xoeXYpL63SKBhYWbXm9lOM2sys7tHef5TZrbNzF4xs2fMbH7cc7eb2a7o7fYg6xQZqwPHgp9jMZr3XVpHYV4OazcdmPT3luwWWFiYWS5wL3ADsAK41cxWjNhtI9Do7iuBHwFfjh5bCXwOuBxYA3zOzCqCqlVkLAaHhjl0/NSkzLEYqbQwjz9YVs3jrx5mSB3dMomCbFmsAZrcfY+7DwAPATfF7+Duz7p7b/ThS0Bd9P51wNPu3unuXcDTwPUB1iqStCMn+xkaduYGcDnVZLxr5Rzau/vZ2NIVyvtLdgoyLOYC8VNOW6PbzuUO4MlxHisyaQ5ElyYPo2UB8JYlszCDF5o6Qnl/yU5BhsVoS3GO2m42s9uARuArYznWzO40s/Vmtr6tTdcqlslx4FikMTzZHdwxM0oKuKh2Oi806ZKrMnmCDItWoD7ucR3wuqUzzexq4DPAje7eP5Zj3f1+d29098aqqqqUFS5yPgcCvuhRMt68ZCYb93fROzAYWg2SXYIMi3VAg5ktNLMC4BZgbfwOZrYauI9IUMTPNHoKuNbMKqId29dGt4mErrWrj1mlBRTl54ZWw2XzKzk95Gw9eCK0GiS7BBYW7j4I3EXkl/x24BF332pm95jZjdHdvgKUAo+a2SYzWxs9thP4ApHAWQfcE90mEroDxyZ/jsVIK+unA7B5v+ZbyOTIC/LF3f0J4IkR2z4bd//q8xz7APBAcNWJjM+Brj6WzSkLtYbqsiJqpxexufV4qHVI9tAMbpExcPe0aFkArKybwSutalnI5FBYiIxBe/cA/YPDk7Y0+fmsrJ9Oc0cvx3oHwi5FsoDCQmQMYkuTp0PLYlXdDABePaBTURI8hYXIGOyPDputrwy/ZbF8TjkAOw6dDLkSyQYKC5ExaOnoAWBeGoRF5bQCqsoK2XFYYSHBU1iIjEFzRy9VZYUUF4Q3xyLestll7DyiuRYSPIWFyBi0dPYyPw1aFTEX1JSx60i3VqCVwCksRMagpbOXeTPTJyyWzi6jf3CY5ujpMZGgKCxEknTq9BCHT5xKi/6KmGWzI5MDd6rfQgKmsBBJUmtXH+4wP41aFg3VZZjBziMKCwmWwkIkSS2d6TMSKqa4IJf5lSVqWUjgFBYiSWrpiEzIm1c5LeRKznbB7DKFhQROYSGSpObOXkoKcplVWhB2KWe5YHY5+zp6OHV6KOxSJIMpLESStL+zl3mVJZiNdiHH8FxQU8awQ9PR7rBLkQymsBBJUnNHb1r1V8Q01JQCCgsJlsJCJAnDwx6ZkJdGI6FiFsycRm6Oseuo+i0kOAoLkSS0dffTPzicli2LgrwcFswsYdcRtSwkOAoLkSQ0x0ZCzUyvkVAxDdVlOg0lgVJYiCShpTM2bDb9WhYAS2tK2dfRQ/+gRkRJMBQWIklo6eghx9LjokejWRIdEbW3XWtESTAUFiJJaO7spXZGMQV56flPpqE6MiJK/RYSlPT85oukmZbO9Bw2G7Nw1jRyDHap30ICorAQSUJLR3oOm40pys9l/sxpNGn4rAREYSGSQHf/IB09A2m3JtRIS6pLeU2noSQgCguRBH6/gGD6tiwg0m+xr72HgcHhsEuRDKSwEEkgtjR5Op+GgsiyH4PDrqvmSSAUFiIJxOZY1Kd9yyJy1Tx1cksQFBYiCTR39DKjJJ/pxflhl3Jei6tKMdPwWQmGwkIkgeaOXuan6TIf8YoLcqmvKNGCghIIhYVIAnvbe1iY5v0VMQ3VpVojSgIRaFiY2fVmttPMmszs7lGev9LMNpjZoJl9YMRzQ2a2KXpbG2SdIudy6vQQB4/3TYmWBcCSmlL2tPUwOKQRUZJaeUG9sJnlAvcC1wCtwDozW+vu2+J2awE+BvyXUV6iz90vCao+kWS0dvXiHpkhPRU0VJcxMDRMS2cvi6pKwy5HMkiQLYs1QJO773H3AeAh4Kb4Hdx9n7u/AujPIElLe9sjI6HSfdhszJk1onQqSlIsyLCYC+yPe9wa3ZasIjNbb2Yvmdl7U1uaSHJicxamSsti8ZkFBdXJLakV2GkoYLSr2vsYjp/n7gfNbBHwSzN71d13n/UGZncCdwLMmzdv/JWKnMPe9h5mlOQzo6Qg7FKSUlqYx9wZxWpZSMol1bIwsx+b2bvMbCwtkVagPu5xHXAw2YPd/WD05x7gOWD1KPvc7+6N7t5YVVU1htJEkrOvo2fKdG7HLKku1VwLSblkf/l/E/gwsMvMvmhmy5I4Zh3QYGYLzawAuAVIalSTmVWYWWH0/izgCmDb+Y8SSb197b1TZthsTEN1KbvbuhkaHktDXuT8kgoLd/+Fu38EuBTYBzxtZr81sz8ys1Gntbr7IHAX8BSwHXjE3bea2T1mdiOAmV1mZq3AzcB9ZrY1evhyYL2ZbQaeBb44YhSVSOCm2rDZmIaaUvoHh2nt6g27FMkgSfdZmNlM4DbgPwEbgX8B3gLcDrxttGPc/QngiRHbPht3fx2R01Mjj/stcHGytYkEYaoNm41pqImuEXWke8oFnaSvZPssfgL8GigB3uPuN7r7w+7+54AGc0tGmmrDZmOWaPisBCDZlsV3oq2EM8ys0N373b0xgLpEQrevfWoNm40pL8pndnmR1oiSlEq2g/sfRtn2YioLEUk3+zqm1rDZeA01WiNKUuu8LQszm01kIl2xma3m93MnyomckhLJWFNx2GzMkupSHl63n+FhJydntClPImOT6DTUdUTWbqoDvhq3/STw6YBqEkkL+9p7uWxBRdhljEtDdRm9A5HRXHUV+rtOJu68YeHuDwIPmtn73f3Hk1STSOh+P2z2dYP1poSGmtiyH90KC0mJRKehbnP3fwYWmNmnRj7v7l8d5TCRKW+qDpuNiS0o+NqRk7x9WXXI1UgmSHQaKvYvRcNjJatM1WGzMTNKCqguK+Q1LfshKZLoNNR90Z//bXLKEUkPe9oiv2Sn8jUhLphdxmtafVZSJNlJeV82s3IzyzezZ8ys3cxuC7o4kbA0He2mqqyQ6cWjrmYzJTRUl9F0tJthrRElKZDsPItr3f0E8G4iq8kuBf4msKpEQra7rZvFVVOzvyJmaU0pfaeHaO3qC7sUyQDJhkXsz6t3Aj90986A6hEJnbuzu62HxVP4FBTA0tmRNaJ26lSUpECyYfFvZrYDaASeMbMq4FRwZYmEp6NngON9p6d8WMSPiBKZqGSXKL8beBPQ6O6ngR5GXE9bJFPsji6TEbtE6VRVVpRP7fQihYWkxFguq7qcyHyL+GO+l+J6REK3uy2ygOCSKR4WEDkVpeGzkgpJhYWZfR9YDGwChqKbHYWFZKDdbd0U5+cyp7wo7FImbGlNGb/d3cHg0DB5uWO5KrLI2ZJtWTQCK9xdY/Ak4+1u62ZR1bSMWICvobqUgcFhmjt7p3wfjIQr2T81tgCzgyxEJF00He3OmF+sF8yOXTVP/RYyMcmGxSxgm5k9ZWZrY7cgCxMJQ9/AEAeO9WVMWCw5MyJK/RYyMcmehvp8kEWIpIvdbd24Z0bnNkBJQR71lcWaayETllRYuPvzZjYfaHD3X5hZCZAbbGkik2/n4cgv1WVzykKuJHWWzS5nx6ETYZchU1yya0P9CfAj4L7oprnAY0EVJRKWHYdPUJiXw4IpeoW80ayYU86e9h56BwbDLkWmsGT7LP4MuAI4AeDuuwAtki8ZZ8fhk1wwu4zcDBgJFXNhbTnusP2QTkXJ+CUbFv3uPhB7EJ2Yp2G0knG2HzrJstmZcwoK4MK50wHYdvB4yJXIVJZsWDxvZp8Gis3sGuBR4N+CK0tk8rWd7Ke9u59ls8vDLiWlaqcXMaMkn60H1W8h45dsWNwNtAGvAh8HngD+LqiiRMKw43Dkl2kmdW4DmBkX1pYrLGRCkh0NNWxmjwGPuXtbwDWJhGJH9Jx+prUsINLJ/eCLzZweGiZfy37IOJz3W2MRnzezdmAHsNPM2szss5NTnsjk2X7oBDXlhVROKwi7lJS7sHY6A4PD7G7T5DwZn0R/YvwlkVFQl7n7THevBC4HrjCzvwq8OpFJtLn1GBdHO4MzzYW1kdbS1gM6FSXjkygsPgrc6u57YxvcfQ9wW/Q5kYxwrHeA3W09rJ5XEXYpgVhUVUpRfo76LWTcEoVFvru3j9wY7bdIeCV7M7vezHaaWZOZ3T3K81ea2QYzGzSzD4x47nYz2xW93Z7ovUQmYuP+YwCsnjcj5EqCkZtjLJtdzlYNn5VxShQWA+N8DjPLBe4FbgBWALea2YoRu7UAHwN+MOLYSuBzRE55rQE+Z2aZ+SefpIWNzV3kGKyqy8ywgMipqG2HTqArDch4JAqLVWZ2YpTbSeDiBMeuAZrcfU90Qt9DjLgUq7vvc/dXgOERx14HPO3une7eBTwNXJ/0f5XIGG1oOcay2eVMKxzLxSOnlhW15Zw8Ncj+zr6wS5Ep6Lxh4e657l4+yq3M3ROdhpoL7I973BrdloyJHCsyJkPDzqb9x7h0fua2KiAyIgpg2yGdipKxC3LA9WiL6yTb/k3qWDO708zWm9n6tjZN/5Dx2XX0JN39g6yuz+wznRfUlJFjsE2d3DIOQYZFK1Af97gOOJjKY939fndvdPfGqqqqcRcq2W1jS6Rz+9L5mR0WxQW5LKoqZZuWK5dxCDIs1gENZrbQzAqAW4Bkr673FHCtmVVEO7avjW4TSbkNzV1UTitgwcySsEsJ3IW15WpZyLgEFhbuPgjcReSX/HbgEXffamb3mNmNAGZ2mZm1AjcD95nZ1uixncAXiATOOuCe6DaRlNvQ0sXq+hmYZc6y5OeyYk45B4+foqvnvIMZRV4n0KEf7v4EkUUH47d9Nu7+OiKnmEY79gHggSDrE4lNxnvfpaN+DTPOiuhM7u2HTvDmJbNCrkamEq0oJlntzGS8+sweCRWzfE502Q+dipIxUlhIVjszGS9LwmJWaSE15YXq5JYxU1hIVtvQcowLMnwy3kgr5qiTW8ZOYSFZ68xkvAxdD+pcVtSW09TWzanTQ2GXIlOIwkKyVtPRbrr7B7k0Q1eaPZcLa6czNOzsOqJrW0jyFBaStTa0dAGZu9LsuayIdnJr2Q8ZC4WFZK2NLV1UlOSzcNa0sEuZVPMqS5hWkKsRUTImCgvJWpv3H2dVlkzGi5eTYyxXJ7eMkcJCslLvwCC7jp5kZYZeRjWRFbXlbD90guFhXdtCkqOwkKy05cAJhh1WZvDFjs5nxZxyegaGaOnsDbsUmSIUFpKVXmmNzNxeWZ+9LQtAk/MkaQoLyUqbW48zZ3oR1WVFYZcSiqU1ZeTmmK7JLUlTWEhWeqX1GCvrsrNVAVCUn0tDdSlbDqhlIclRWEjWOdY7QHNHb9b2V8RcWDudLQeO465ObklMYSFZ55XWyKmXVVkeFhfPLaejZ4AjJ/rDLkWmAIWFZJ1Y5/bFWXwaCuCi6LDhLQfUbyGJKSwk62xuPc7CWdOYXpwfdimhWj6nHDN4VWEhSVBYSNbJ9s7tmGmFeSyuKtWIKEmKwkKyypETpzhyoj/rO7djLqot14goSYrCQrLK5uhlVFepZQFE+i0OnzhF20l1csv5KSwkq7zSepzcHOPCWoUFxHVy61SUJKCwkKyyufUYDdWlFBfkhl1KWogt+7FVndySgMJCsoa780rr8ay72NH5lBfls2BmCZtbFRZyfgoLyRr7Ono53nc66yfjjXTp/Ao2NHdpJrecl8JCssaZzu16hUW8yxZU0tEzwN72nrBLkTSmsJCssWn/MUoKcllaUxZ2KWmlcX4FAOubu0KuRNKZwkKyxubWY1w0dzq5Odl1GdVEFleVMqMkn5f3KSzk3BQWkhUGBofZevAEl+gU1Ovk5BhvmFfBuubOsEuRNKawkKyw/dAJBgaH1bl9Do0LKtnT1kNHtybnyegUFpIVfre3A4DLFlSEXEl6WrOwEoAX93SEXImkq0DDwsyuN7OdZtZkZneP8nyhmT0cff53ZrYgun2BmfWZ2abo7VtB1imZ76U9nSyqmkZ1eXZeRjWRVXXTKSvK49evtYddiqSpvKBe2MxygXuBa4BWYJ2ZrXX3bXG73QF0ufsSM7sF+BLwoehzu939kqDqk+wxNOys29vJey6pDbuUtJWXm8NblsziV7vacHfMNAhAzhZky2IN0OTue9x9AHgIuGnEPjcBD0bv/wh4h+lbKim27eAJTvYPcnn0VIuM7sqlVRw6fordbd1hlyJpKMiwmAvsj3vcGt026j7uPggcB2ZGn1toZhvN7Hkze2uAdUqGe2F35NTKGxfNTLBndntrwywAntepKBlFkGExWgth5HoC59rnEDDP3VcDnwJ+YGblr3sDszvNbL2ZrW9ra5twwZKZntl+hAtry6lRf8V51VWUsKhqGs/tPBp2KZKGggyLVqA+7nEdcPBc+5hZHjAd6HT3fnfvAHD3l4HdwNKRb+Du97t7o7s3VlVVBfCfIFNdR3c/Lzd3cfXymrBLmRKuWV7DS3s6OHHqdNilSJoJMizWAQ1mttDMCoBbgLUj9lkL3B69/wHgl+7uZlYV7SDHzBYBDcCeAGuVDPXszjaGHa5ZobBIxrUXzub0kPPsDrUu5GyBhUW0D+Iu4ClgO/CIu281s3vM7Mbobt8FZppZE5HTTbHhtVcCr5jZZiId359wd00vlTH72ZZDzC4v4sLa153FlFGsrp9BdVkhP9tyOOxSJM0ENnQWwN2fAJ4Yse2zcfdPATePctyPgR8HWZtkvvbufp7b2cYdb1mooaBJyskxrllRw082HODU6SGK8nWRKInQDG7JWP+66SCDw87731AXdilTyvUXzabv9BC/3qVRUfJ7CgvJSO7Oo+v3s6puupYkH6M3LprJ9OJ8nnj1UNilSBpRWEhGenF3BzsOn+TDl88Lu5QpJz83hxsums3Ptx6mb2Ao7HIkTSgsJCN95zd7mTmtgJsuGTkPVJJx4yW19AwM8cyOI2GXImlCYSEZZ3dbN7/ccZTb3jhfHbTjdPnCmdSUF/Kvm0ZOjZJspbCQjPPN53ZTmJfDbW+cH3YpU1ZujvGelbU8t/Mox3s1QU8UFpJhmjt6+OnGA3zk8vlUlRWGXc6UdtMlczk95Dy5RR3dorCQDPP1XzaRl2N84qpFYZcy5V00t5xFVdP48YbWsEuRNKCwkIzR0tHLTzYe4NY183SRoxQwMz7YWM+6fV00HdWy5dlOYSEZ4+vP7iI3x/jTty0Ou5SM8f5L68jLMR5e1xJ2KRIyhYVkhH3tPfxkwwE+vGaeliJPoaqyQq5ZUcOPNxygf1BzLrKZwkIywpd+toOCvBw+qVZFyt2yZh6dPQM8vU1zLrKZwkKmvHX7Onlyy2E+fuVi9VUE4C1LZjGvsoR/emFf2KVIiBQWMqUNDzv/8Ph2asoL+ZMrF4ZdTkbKzTH+6IoFrG/uYmNLV9jlSEgUFjKlfe/FfWzef4y7b1hGSUGgK+5ntZsb6ykryuO7v9kbdikSEoWFTFn7O3v58lM7uWppFe/VGlCBKi3M48Nr5vHklsPs7+wNuxwJgcJCpqTTQ8P85cObMOC/v+9iXdxoEnzsigXkGNz3q91hlyIhUFjIlPSlJ3fwcnMXX3z/SubOKA67nKwwZ3oxH2ys5+F1+zl4rC/scmSSKSxkyvnZlkN85zd7+eib5vOeVbVhl5NVPvn2JQB847mmkCuRyaawkClly4Hj/NXDm1lVN53PvGt52OVknbkzirlZrYuspLCQKePQ8T7ueHAdFSX5fPujjRTm6VoVYYhNfLz3WbUusonCQqaEnv5B7vin9fT0D/HAH12myXchqqso4ZbL5vHQuv00HT0ZdjkySRQWkvb6Boa448F17Dxykq99eDXLZpeHXVLW+8urGyjJz+UfH98edikySRQWktZ6+ge58/vr+d3eTr76wVW8/YLqsEsSYGZpIXf9wRKe3dnGr15rC7scmQQKC0lbu46c5KZ7X+A3Te186f0ruUkT79LKx65YwIKZJXzmsVfp7h8MuxwJmMJC0tJjGw9w49df4FjvAP9yx+V8sLE+7JJkhMK8XP7nzato7erjHx/fFnY5EjAtpiNp5dTpIb7w79v4l9+1sGZBJV/78GpdnyKNNS6o5ONXLuZbz+/mkvoZfOiyeWGXJAFRWEjaaOno5ZM/eJktB07w8asW8TfXXkBerhq/6e6vr13KtkMn+PRPt1CUn6vThRlK/xIlLfx862He9bVf09LRy7c/2sjf3rBcQTFF5Ofm8I2PXMob5lfwFw9t4otP7uDUaV1VL9PoX6OEqrt/kP/6o1e48/svs2DmNB7/z2/lmhU1YZclY1RamMf371jDrWvq+dbzu3nn//01/7b5IEPDHnZpkiKBhoWZXW9mO82syczuHuX5QjN7OPr878xsQdxzfxvdvtPMrguyTpl8Pf2DfO/FfbztK8+AuiV4AAAGpElEQVTx6Mv7+eTbFvOjP30T9ZUlYZcm41SYl8v/eN9Kvn/HGnJzjD//4Uau+erzfOfXe+jqGQi7PJkgcw8m+c0sF3gNuAZoBdYBt7r7trh9PgmsdPdPmNktwB+6+4fMbAXwQ2ANUAv8Aljq7uds2zY2Nvr69esD+W+RiRsYHKals4eXm7t4cXcHP992hN6BIdYsqOTT71rOJfUzwi5RUmh42HliyyEe+M1eNrQcoyA3hzctnskfLKvmDfMrWFJdSlG+lmtJB2b2srs3JtovyA7uNUCTu++JFvQQcBMQP8buJuDz0fs/Ar5ukQsT3AQ85O79wF4za4q+3osB1itJGhgcpm9giL7TQ/QODNJ75v4QfQOD9PQPcfjEKVo6emnpjNwOHe8jdkaicloBN66q5f1vqKNxfoWuRZGBcnKMd6+s5d0ra9lx+ASPrm/llzuO8rm1W4HIpVrrKoojtxklkZ+VxdRVlFBVWsj04nzKi/PJzdF3I10EGRZzgf1xj1uBy8+1j7sPmtlxYGZ0+0sjjg1kiMWx3gE+8K1IBsW3svx1d866e2bfs7fF7+uv3zZKI27U94zb1+O2+ii1nP2ar993tJrP+V7neJ641zp1eojBJM9DV5UVMq+yhDULK6mvLGF+ZQmXzJvBolnTFBBZZNnscv7+3Sv4+3evoLmjh1cPHGf7oRM0d/TS2tXHMzuO0t7dP+qxZYV5FBXkkpdj5OYYeTlGTo6Ro+/PGQYsm1PO125dHej7BBkWo/3fHPlb5lz7JHMsZnYncCfAvHnjG9+dm2NcUFM2akWxu/G/2OILi20+e9so+571mnHPj3r86/c9a9tZn8wEX+scdZ/v+KL8HIrzcykuyKWkII+Sgtj9yK04P7KturxQ18SW15k/cxrzZ07j3SvPvg5J38AQB4710drVS3v3AMf7TnO87zQn+k7TPzjE4JAz5M7QsEf+WFG/OfD7PybrK4K/AFiQ/5pbgfhpt3XAwXPs02pmecB0oDPJY3H3+4H7IdJnMZ4iy4ryufcjl47nUBFJkeKCXJZUl7KkujTsUuQcghwNtQ5oMLOFZlYA3AKsHbHPWuD26P0PAL/0yLmQtcAt0dFSC4EG4D8CrFVERM4jsJZFtA/iLuApIBd4wN23mtk9wHp3Xwt8F/h+tAO7k0igEN3vESKd4YPAn51vJJSIiAQrsKGzk01DZ0VExi7ZobOawS0iIgkpLEREJCGFhYiIJKSwEBGRhBQWIiKSUMaMhjKzNqA57DrOYRbQHnYRaUSfx9n0eZxNn8fZgv485rt7VaKdMiYs0pmZrU9maFq20OdxNn0eZ9PncbZ0+Tx0GkpERBJSWIiISEIKi8lxf9gFpBl9HmfT53E2fR5nS4vPQ30WIiKSkFoWIiKSkMIiIGZ2s5ltNbNhM2sc8dzfmlmTme00s+vCqjFMZvZ5MztgZpuit3eGXVMYzOz66PegyczuDruesJnZPjN7NfqdyLqVQc3sATM7amZb4rZVmtnTZrYr+rMijNoUFsHZArwP+FX8RjNbQWQp9guB64FvmFm2Xrn+f7v7JdHbE2EXM9mi/9/vBW4AVgC3Rr8f2e7t0e9E6MNFQ/BPRH4vxLsbeMbdG4Bnoo8nncIiIO6+3d13jvLUTcBD7t7v7nuBJmDN5FYnaWIN0OTue9x9AHiIyPdDspS7/4rItX3i3QQ8GL3/IPDeSS0qSmEx+eYC++Met0a3ZaO7zOyVaNM7lKZ1yPRdeD0Hfm5mL5vZnWEXkyZq3P0QQPRndRhFBHkN7oxnZr8AZo/y1Gfc/V/Pddgo2zJySNr5Ph/gm8AXiPy3fwH4X8AfT151aSFrvgtjcIW7HzSzauBpM9sR/WtbQqawmAB3v3och7UC9XGP64CDqakovST7+ZjZt4F/D7icdJQ134VkufvB6M+jZvZTIqfqsj0sjpjZHHc/ZGZzgKNhFKHTUJNvLXCLmRWa2UKgAfiPkGuadNEvfcwfEhkQkG3WAQ1mttDMCogMfFgbck2hMbNpZlYWuw9cS3Z+L0ZaC9wevX87cK6zFoFSyyIgZvaHwNeAKuBxM9vk7te5+1YzewTYBgwCf+buQ2HWGpIvm9klRE677AM+Hm45k8/dB83sLuApIBd4wN23hlxWmGqAn5oZRH43/cDdfxZuSZPLzH4IvA2YZWatwOeALwKPmNkdQAtwcyi1aQa3iIgkotNQIiKSkMJCREQSUliIiEhCCgsREUlIYSEiIgkpLEREJCGFhYiIJKSwEBGRhP4/tBFms496lDYAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "res0['tscore'].plot.kde()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1" } }, "nbformat": 4, "nbformat_minor": 2 }