{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Regulome Explorer Kruskal-Wallis test for numerical and categorical data\n",
"CCheck out more notebooks at our ['Regulome Explorer Repository'](https://github.com/isb-cgc/Community-Notebooks/tree/master/RegulomeExplorer)!\n",
"\n",
"In this notebook we describe how Regulome Explorer uses Kruskal-Wallis test to compute the significance of associations between a numerical feature (Gene expression, Somatic copy number, etc.) and a categorical feature. Details of the Kruskal-Wallist test can be found in the following link: https://en.wikipedia.org/wiki/Kruskal%E2%80%93Wallis_one-way_analysis_of_variance \n",
"\n",
"To describe the implementation of the test using BigQuery, we will use Gene expresion data of a user defined gene and a user defined clinical feature. This data is read from a BigQuery table in the pancancer-atlas dataset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Authenticate with Google (IMPORTANT)\n",
"The first step is to authorize access to BigQuery and the Google Cloud. For more information see ['Quick Start Guide to ISB-CGC'](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/HowToGetStartedonISB-CGC.html) and alternative authentication methods can be found [here](https://googleapis.github.io/google-cloud-python/latest/core/auth.html)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Import Python libraries"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"from google.cloud import bigquery\n",
"import numpy as np\n",
"import pandas as pd\n",
"from scipy import stats\n",
"from scipy.stats import mstats\n",
"import seaborn as sns\n",
"import re_module.bq_functions as regulome"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## User defined Parameters\n",
"The parameters for this experiment are the cancer type, the name of gene for which gene expression data will be obtained, and the clinical feature name. Categorical groups with number of samples smaller than 'MinSampleSize' will be ignored in the test. "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cancer_type = 'LGG'\n",
"gene_name = 'IGF2'\n",
"clinical_feature = 'icd_o_3_histology'\n",
"MinSampleSize = 26\n",
"\n",
"bqclient = bigquery.Client()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data from BigQuery tables"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Gene expression data from the BigQuery:** The following query string retrieves the gene expression data of the user specified gene ('gene_name') from the 'Filtered.EBpp_AdjustPANCAN_IlluminaHiSeq_RNASeqV2_genExp_filtered' table available in pancancer-atlas dataset."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"query_table1 = \"\"\"table1 AS (\n",
"SELECT symbol, data, ParticipantBarcode\n",
"FROM ( \n",
" SELECT \n",
" Symbol AS symbol, AVG( LOG10( normalized_count + 1 )) AS data, ParticipantBarcode\n",
" FROM `pancancer-atlas.Filtered.EBpp_AdjustPANCAN_IlluminaHiSeq_RNASeqV2_genExp_filtered` \n",
" WHERE Study = '{0}' AND Symbol ='{1}' AND normalized_count IS NOT NULL\n",
" GROUP BY \n",
" ParticipantBarcode, symbol\n",
" )\n",
")\n",
"\"\"\".format(cancer_type, gene_name )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Clinical data from the BigQuery:** The following string query will retrieve clinical data fromthe 'pancancer-atlas.Filtered.clinical_PANCAN_patient_with_followup_filtered' table available in pancancer-atlas dataset. It is worth noting that some of the values of the clinical feature may be 'indetermined' or 'not-evaluated'; typically these values are inside square brackets. The 'REGEXP_CONTAINS' command is used to avoid using those values in the test."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"query_table2 = \"\"\"table2 AS (\n",
"SELECT\n",
" symbol,\n",
" avgdata AS data,\n",
" ParticipantBarcode\n",
"FROM (\n",
" SELECT\n",
" '{0}' AS symbol, \n",
" {0} AS avgdata,\n",
" bcr_patient_barcode AS ParticipantBarcode\n",
" FROM `pancancer-atlas.Filtered.clinical_PANCAN_patient_with_followup_filtered`\n",
" WHERE acronym = '{1}' AND {0} IS NOT NULL \n",
" AND NOT REGEXP_CONTAINS({0},r\"^(\\[.*\\]$)\") \n",
" )\n",
")\n",
"\"\"\".format(clinical_feature, cancer_type)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following query combines the two tables based on Participant barcodes. T"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"table_data = \"\"\"table_data AS (\n",
"SELECT \n",
" n1.data as data1,\n",
" n2.data as data2,\n",
" n1.ParticipantBarcode\n",
"FROM\n",
" table1 AS n1\n",
"INNER JOIN\n",
" table2 AS n2\n",
"ON\n",
" n1.ParticipantBarcode = n2.ParticipantBarcode\n",
") \n",
"\"\"\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"At this point we can take a look at output table"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
" in runQuery ... \n",
" the results for this query were previously cached \n"
]
},
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" data1 | \n",
" data2 | \n",
" ParticipantBarcode | \n",
"
\n",
" \n",
" \n",
" \n",
" | 1 | \n",
" 2.359532 | \n",
" 9382/3 | \n",
" TCGA-E1-A7YW | \n",
"
\n",
" \n",
" | 2 | \n",
" 2.868692 | \n",
" 9382/3 | \n",
" TCGA-FG-7637 | \n",
"
\n",
" \n",
" | 3 | \n",
" 2.713119 | \n",
" 9382/3 | \n",
" TCGA-TQ-A7RG | \n",
"
\n",
" \n",
" | 4 | \n",
" 3.064997 | \n",
" 9382/3 | \n",
" TCGA-DB-5280 | \n",
"
\n",
" \n",
" | 5 | \n",
" 2.554518 | \n",
" 9382/3 | \n",
" TCGA-IK-8125 | \n",
"
\n",
" \n",
" | 6 | \n",
" 2.799724 | \n",
" 9382/3 | \n",
" TCGA-DU-8163 | \n",
"
\n",
" \n",
" | 7 | \n",
" 2.800062 | \n",
" 9382/3 | \n",
" TCGA-HT-8018 | \n",
"
\n",
" \n",
" | 8 | \n",
" 2.558207 | \n",
" 9382/3 | \n",
" TCGA-DU-7019 | \n",
"
\n",
" \n",
" | 9 | \n",
" 2.793514 | \n",
" 9382/3 | \n",
" TCGA-DU-5852 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" Ngroups | \n",
" Nsamples | \n",
" Hscore | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 5 | \n",
" 513 | \n",
" 7.857342 | \n",
"
\n",
" \n",
"
\n",
"