{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "name": "Update of Quick Start Guide to ISB-CGC.ipynb", "provenance": [], "collapsed_sections": [], "include_colab_link": true }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "view-in-github", "colab_type": "text" }, "source": [ "\"Open" ] }, { "cell_type": "markdown", "metadata": { "id": "V_oh9DspLamP" }, "source": [ "# ISB-CGC Community Notebooks\n", "\n", "Check out more notebooks at our [Community Notebooks Repository](https://github.com/isb-cgc/Community-Notebooks)!\n", "\n", "```\n", "Title: Quick Start Guide to ISB-CGC\n", "Author: Lauren Hagen\n", "Created: 2019-06-20\n", "Updated: 2021-07-27\n", "Purpose: Painless intro to working in the cloud\n", "URL: https://github.com/isb-cgc/Community-Notebooks/blob/master/Notebooks/Quick_Start_Guide_to_ISB_CGC.ipynb\n", "Notes: \n", "```\n", "***" ] }, { "cell_type": "markdown", "metadata": { "id": "35Ew4aYCspKn" }, "source": [ "# Quick Start Guide to ISB-CGC\n", "[ISB-CGC](https://isb-cgc.appspot.com/)\n", "\n", "This Quick Start Guide gives an overview of the data available, account set-up overview, and getting started with a basic example in python. If you have read the R version, you can skip to the Example section.\n", "\n", "## Access Requirements\n", "* Google Account to access ISB-CGC\n", "* [Google Cloud Account](https://console.cloud.google.com)\n", "\n", "## Access Suggestions\n", "* Favored Programming Language (R or Python)\n", "* Favored IDE (RStudio or Jupyter)\n", "* Some knowledge of SQL\n", "\n", "## Outline for this Notebook\n", "* Libraries Needed for this Notebook\n", "* Overview of ISB-CGC\n", "* Overview How to Access Data\n", "* Example of Accessing Data with Python\n", "* Where to go next" ] }, { "cell_type": "markdown", "metadata": { "id": "WJs8FZBJ8x1f" }, "source": [ "## Libraries needed for the Notebook\n", "This notebook requires the BigQuery API to be loaded [(click here for more information)](https://googleapis.github.io/google-cloud-python/latest/bigquery/usage/client.html). This library will allow you to access BigQuery programmatically." ] }, { "cell_type": "code", "metadata": { "id": "usHb94n-L0a5" }, "source": [ "# Load BigQuery API\n", "from google.cloud import bigquery" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "OAgOtUGpvR5h" }, "source": [ "## Overview of ISB-CGC\n", "The ISB-CGC provides interactive and programmatic access to data hosted by institutes such as the [Genomic Data Commons (GDC)](https://gdc.cancer.gov/) and [Proteomic Data Commons (PDC)](https://proteomic.datacommons.cancer.gov/pdc/) from the [National Cancer Institute (NCI)](https://www.cancer.gov/) while leveraging many aspects of the Google Cloud Platform. You can also import your data, analyze it side by side with the datasets, and share your data when you see fit." ] }, { "cell_type": "markdown", "metadata": { "id": "Ue2bsjIWu7o3" }, "source": [ "### About the ISB-CGC Data in the Cloud\n", "ISB-CGC hosts carefully curated, high-level clinical, biospecimen, and molecular datasets and tables in Google BigQuery, including data from programs such as [The Cancer Genome Atlas (TCGA)](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga), [Therapeutically Applicable Research to Generate Effective Treatments (TARGET)](https://ocg.cancer.gov/programs/target), and [Clinical Proteomic Tumor Analysis Consortium (CPTAC)](https://proteomics.cancer.gov/programs/cptac). For more information about hosted data, please visit: [Programs and DataSets](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/Hosted-Data.html)" ] }, { "cell_type": "markdown", "metadata": { "id": "4rbvEDj6xfMT" }, "source": [ "## Overview of How to Access Data\n", "There are several ways to access and explore the data hosted by ISB-CGC. Though in this notebook, we will cover using Python and SQL to access the data.\n", "\n", "* [ISB-CGC WebApp](https://isb-cgc.appspot.com/)\n", " * Provides a graphical interface to file and case data\n", " * Easy cohort creation\n", " * Doesn't require knowledge of programming languages\n", "* [ISB-CGC BigQuery Table Search](https://isb-cgc.appspot.com/bq_meta_search/)\n", " * Provides a table search for available ISB-CGC BigQuery Tables\n", "* [ISB-CGC APIs](https://api-dot-isb-cgc.appspot.com/v4/swagger/)\n", " * Provides programmatic access to metadata\n", "* [Google Cloud Platform](https://cloud.google.com/)\n", " * Access and store data in [Google Cloud Storage](https://cloud.google.com/storage) and [BigQuery](https://cloud.google.com/bigquery) via User Interfaces or programmatically\n", "* Suggested Programming Languages and Programs to use\n", " * SQL\n", " * Can be used directly in [BigQuery Console](https://console.cloud.google.com/bigquery)\n", " * Or via API in Python or R\n", " * [Python](https://www.python.org/)\n", " * [gsutil tool](https://cloud.google.com/storage/docs/gsutil)\n", " * [Jupyter Notebooks](https://jupyter.org/)\n", " * [Google Colabratory](https://colab.research.google.com/)\n", " * [Cloud Datalab](https://cloud.google.com/datalab/)\n", " * [R](https://www.r-project.org/)\n", " * [RStudio](https://rstudio.com/)\n", " * [RStudio.Cloud](https://rstudio.cloud/)\n", "* Command Line Interfaces\n", " * Cloud Shell via Project Console\n", " * [CLOUD SDK](https://cloud.google.com/sdk/)" ] }, { "cell_type": "markdown", "metadata": { "id": "Xlx2svEHfLfN" }, "source": [ "### Account Set-up\n", "To run this notebook, you will need to have your Google Cloud Account set up. If you need to set up a Google Cloud Account, follow the \"Obtain a Google identity\" and \"Set up a Google Cloud Project\" steps on our [Quick-Start Guide documentation](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/HowToGetStartedonISB-CGC.html) page.\n", " " ] }, { "cell_type": "markdown", "metadata": { "id": "UI5UpdyaeS2v" }, "source": [ "### ISB-CGC Web Interface\n", "The [ISB-CGC Web Interface](https://isb-cgc.appspot.com/) is an interactive web-based application to access and explore the rich TCGA, TARGET, and CCLE datasets with more datasets regularly added. Through WebApp, you can create Cohorts, lists of Favorite Genes, miRNA, and Variables. The Cohorts and Variables can be used in Workbooks to allow you to quickly analyze and export datasets by mixing and matching the selections." ] }, { "cell_type": "markdown", "metadata": { "id": "c1bk7F_-uuYp" }, "source": [ "### Google Cloud Platform and BigQuery Overview\n", "\n", "The [Google Cloud Platform Console](https://console.cloud.google.com/) is the web-based interface to your GCP Project. From the Console, you can check the overall status of your project, create and delete Cloud Storage buckets, upload and download files, spin up and shut down VMs, add members to your project, access the [Cloud Shell command line](https://cloud.google.com/shell/docs/), etc. You'll want to remember that any costs that you incur are charged under your *current* project, so you will want to make sure you are on the correct one if you are part of multiple projects.\n", "\n", "ISB-CGC has uploaded multiple cancer genomic and proteomic datasets into BigQuery tables that are open-source such as TCGA and TARGET Clinical, Biospecimen, and Molecular Data, along with case and file data. This data can be accessed from the Google Cloud Platform Console User Interface (UI), programmatically with R and python, or explored with our [BigQuery Table Search tool](https://isb-cgc.appspot.com/bq_meta_search/)." ] }, { "cell_type": "markdown", "metadata": { "id": "FLkFocDF265b" }, "source": [ "## Example of Accessing BigQuery Data with Python\n" ] }, { "cell_type": "markdown", "metadata": { "id": "uwUePpm5uLNS" }, "source": [ "### Log into Google Cloud Storage and Authenticate ourselves\n", "1. Authenticate yourself with your Google Cloud Login\n", "2. A second tab will open or follow the link provided\n", "3. Follow prompts to Authorize your account to use Google Cloud SDK\n", "4. Copy code provided and paste into the box under the Command\n", "5. Press Enter\n", "\n", "[Alternative authentication methods](https://googleapis.github.io/google-cloud-python/latest/core/auth.html)" ] }, { "cell_type": "code", "metadata": { "id": "CzjAwn9w3Ghc" }, "source": [ "!gcloud auth application-default login" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "lqtslyr2zNTA" }, "source": [ "### View ISB-CGC Datasets and Tables in BigQuery\n", "Let us look at the datasets available through ISB-CGC that are in BigQuery. " ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "usDiQye0PWPF", "outputId": "10cf4e42-2ceb-4505-bebd-483c5f167c08" }, "source": [ "# Create a client to access the data within BigQuery\n", "# Note: you cannot use the project below as a billing project,\n", "# it can only be used to view the tables and table schema\n", "client = bigquery.Client('isb-cgc-bq')\n", "\n", "# Create a variable of datasets \n", "datasets = list(client.list_datasets())\n", "# Create a variable for the name of the project\n", "project = client.project\n", "\n", "# If there are datasets available then print their names,\n", "# else print that there are no datasets available\n", "if datasets:\n", " print(\"Datasets in project {}:\".format(project))\n", " for dataset in datasets: # API request(s)\n", " print(\"\\t{}\".format(dataset.dataset_id))\n", "else:\n", " print(\"{} project does not contain any datasets.\".format(project))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "/usr/local/lib/python3.7/dist-packages/google/auth/_default.py:70: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a \"quota exceeded\" or \"API not enabled\" error. We recommend you rerun `gcloud auth application-default login` and make sure a quota project is added. Or you can use service accounts instead. For more information about service accounts, see https://cloud.google.com/docs/authentication/\n", " warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Datasets in project isb-cgc-bq:\n", "\t0_README\n", "\tBEATAML1_0\n", "\tBEATAML1_0_versioned\n", "\tCBTTC\n", "\tCBTTC_versioned\n", "\tCCLE\n", "\tCCLE_versioned\n", "\tCGCI\n", "\tCGCI_versioned\n", "\tCMI\n", "\tCMI_versioned\n", "\tCPTAC\n", "\tCPTAC_versioned\n", "\tCTSP\n", "\tCTSP_versioned\n", "\tFM\n", "\tFM_versioned\n", "\tGDC_case_file_metadata\n", "\tGDC_case_file_metadata_versioned\n", "\tGENCODE\n", "\tGENCODE_versioned\n", "\tGENIE\n", "\tGENIE_versioned\n", "\tGPRP\n", "\tGPRP_versioned\n", "\tHCMI\n", "\tHCMI_versioned\n", "\tICPC\n", "\tICPC_versioned\n", "\tMMRF\n", "\tMMRF_versioned\n", "\tNCICCR\n", "\tNCICCR_versioned\n", "\tOHSU\n", "\tOHSU_versioned\n", "\tORGANOID\n", "\tORGANOID_versioned\n", "\tPDC_metadata\n", "\tPDC_metadata_versioned\n", "\tQuant_Maps_Tissue_Biopsies\n", "\tQuant_Maps_Tissue_Biopsies_versioned\n", "\tTARGET\n", "\tTARGET_versioned\n", "\tTCGA\n", "\tTCGA_versioned\n", "\tVAREPOP\n", "\tVAREPOP_versioned\n", "\tWCDT\n", "\tWCDT_versioned\n", "\tannotations\n", "\tannotations_versioned\n", "\tfunctions\n", "\tpancancer_atlas\n", "\tsupplementary_tables\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "dcKkXP0H8W6I" }, "source": [ "The ISB-CGC has two datasets for each Program. One dataset contains the most current data, and the other contains versioned tables, which serve as an archive for reproducibility. The current tables are labeled with \"_current\" and are updated when new data is released. For more information, visit our [ISB-CGC BigQuery Projects](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/BigQuery/ISBCGC-BQ-Projects.html) page.\n", "\n", "Now, let us see which tables are under the TCGA dataset." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "IZ_Odw0z1fOn", "outputId": "251648c2-1443-436a-f313-8434d73c2ef0" }, "source": [ "print(\"Tables:\")\n", "# Create a variable with the list of tables in the dataset\n", "tables = list(client.list_tables('isb-cgc-bq.TCGA'))\n", "\n", "# If there are tables then print their names,\n", "# else print that there are no tables\n", "if tables:\n", " for table in tables:\n", " print(\"\\t{}\".format(table.table_id))\n", "else:\n", " print(\"\\tThis dataset does not contain any tables.\")" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Tables:\n", "\tDNA_methylation_chr10_hg19_gdc_current\n", "\tDNA_methylation_chr10_hg38_gdc_current\n", "\tDNA_methylation_chr11_hg19_gdc_current\n", "\tDNA_methylation_chr11_hg38_gdc_current\n", "\tDNA_methylation_chr12_hg19_gdc_current\n", "\tDNA_methylation_chr12_hg38_gdc_current\n", "\tDNA_methylation_chr13_hg19_gdc_current\n", "\tDNA_methylation_chr13_hg38_gdc_current\n", "\tDNA_methylation_chr14_hg19_gdc_current\n", "\tDNA_methylation_chr14_hg38_gdc_current\n", "\tDNA_methylation_chr15_hg19_gdc_current\n", "\tDNA_methylation_chr15_hg38_gdc_current\n", "\tDNA_methylation_chr16_hg19_gdc_current\n", "\tDNA_methylation_chr16_hg38_gdc_current\n", "\tDNA_methylation_chr17_hg19_gdc_current\n", "\tDNA_methylation_chr17_hg38_gdc_current\n", "\tDNA_methylation_chr18_hg19_gdc_current\n", "\tDNA_methylation_chr18_hg38_gdc_current\n", "\tDNA_methylation_chr19_hg19_gdc_current\n", "\tDNA_methylation_chr19_hg38_gdc_current\n", "\tDNA_methylation_chr1_hg19_gdc_current\n", "\tDNA_methylation_chr1_hg38_gdc_current\n", "\tDNA_methylation_chr20_hg19_gdc_current\n", "\tDNA_methylation_chr20_hg38_gdc_current\n", "\tDNA_methylation_chr21_hg19_gdc_current\n", "\tDNA_methylation_chr21_hg38_gdc_current\n", "\tDNA_methylation_chr22_hg19_gdc_current\n", "\tDNA_methylation_chr22_hg38_gdc_current\n", "\tDNA_methylation_chr2_hg19_gdc_current\n", "\tDNA_methylation_chr2_hg38_gdc_current\n", "\tDNA_methylation_chr3_hg19_gdc_current\n", "\tDNA_methylation_chr3_hg38_gdc_current\n", "\tDNA_methylation_chr4_hg19_gdc_current\n", "\tDNA_methylation_chr4_hg38_gdc_current\n", "\tDNA_methylation_chr5_hg19_gdc_current\n", "\tDNA_methylation_chr5_hg38_gdc_current\n", "\tDNA_methylation_chr6_hg19_gdc_current\n", "\tDNA_methylation_chr6_hg38_gdc_current\n", "\tDNA_methylation_chr7_hg19_gdc_current\n", "\tDNA_methylation_chr7_hg38_gdc_current\n", "\tDNA_methylation_chr8_hg19_gdc_current\n", "\tDNA_methylation_chr8_hg38_gdc_current\n", "\tDNA_methylation_chr9_hg19_gdc_current\n", "\tDNA_methylation_chr9_hg38_gdc_current\n", "\tDNA_methylation_chrX_hg19_gdc_current\n", "\tDNA_methylation_chrX_hg38_gdc_current\n", "\tDNA_methylation_chrY_hg19_gdc_current\n", "\tDNA_methylation_chrY_hg38_gdc_current\n", "\tDNA_methylation_hg19_gdc_current\n", "\tDNA_methylation_hg38_gdc_current\n", "\tRNAseq_hg19_gdc_current\n", "\tRNAseq_hg38_gdc_current\n", "\tannotations_gdc_current\n", "\tbiospecimen_gdc_current\n", "\tclinical_CPTAC_TCGA_pdc_current\n", "\tclinical_diagnoses_treatments_gdc_current\n", "\tclinical_gdc_current\n", "\tcopy_number_segment_masked_hg19_gdc_current\n", "\tcopy_number_segment_masked_hg38_gdc_current\n", "\tmiRNAseq_hg19_gdc_current\n", "\tmiRNAseq_hg38_gdc_current\n", "\tmiRNAseq_isoform_hg19_gdc_current\n", "\tmiRNAseq_isoform_hg38_gdc_current\n", "\tper_sample_file_metadata_CPTAC_TCGA_pdc_current\n", "\tper_sample_file_metadata_hg19_gdc_current\n", "\tper_sample_file_metadata_hg38_gdc_current\n", "\tprotein_expression_hg19_gdc_current\n", "\tprotein_expression_hg38_gdc_current\n", "\tquant_phosphoproteome_TCGA_breast_cancer_pdc_current\n", "\tquant_phosphoproteome_TCGA_ovarian_PNNL_velos_qexactive_pdc_current\n", "\tquant_proteome_TCGA_breast_cancer_pdc_current\n", "\tquant_proteome_TCGA_ovarian_JHU_pdc_current\n", "\tquant_proteome_TCGA_ovarian_PNNL_pdc_current\n", "\tradiology_images_tcia_current\n", "\tslide_images_gdc_current\n", "\tsomatic_mutation_hg38_gdc_current\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "mjXF-Vg4z5mv" }, "source": [ "### Query ISB-CGC BigQuery Tables\n" ] }, { "cell_type": "markdown", "metadata": { "id": "GjCNNGweyyCE" }, "source": [ "First, use a magic command to call to BigQuery. Then we can use Standard SQL to write your query. Click [here](https://googleapis.github.io/google-cloud-python/latest/bigquery/magics.html) for more on IPython Magic Commands for BigQuery. The result will be a [Pandas Dataframe](https://pandas.pydata.org/).\n", "\n", "> Note: you will need to update PROJECT_ID in the next cell to your Google Cloud Project ID." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "TYf4eBOKnKEz", "outputId": "0e024253-161b-4220-a426-85515b96d860" }, "source": [ "# Call to BigQuery with a magic command\n", "# and replace PROJECT_ID with your project ID Number\n", "%%bigquery --project PROJECT_ID\n", "SELECT # Select a few columns to view\n", " proj__project_id, # GDC project\n", " submitter_id, # case barcode\n", " proj__name # GDC project name\n", "FROM # From the GDC TCGA Clinical Dataset\n", " `isb-cgc-bq.TCGA.clinical_gdc_current`\n", "LIMIT # Limit to 5 rows as the dataset is very large and we only want to see a few results\n", " 5\n", "\n", "# Syntax for the above query\n", "# SELECT * \n", "# FROM `project_name.dataset_name.INFORMATION_SCHEMA.COLUMNS`\n", "# Limit to the first 5 fields" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
proj__project_idsubmitter_idproj__name
0TCGA-HNSCTCGA-CN-5363Head and Neck Squamous Cell Carcinoma
1TCGA-HNSCTCGA-CN-5365Head and Neck Squamous Cell Carcinoma
2TCGA-HNSCTCGA-CN-A642Head and Neck Squamous Cell Carcinoma
3TCGA-HNSCTCGA-CR-7380Head and Neck Squamous Cell Carcinoma
4TCGA-HNSCTCGA-CV-5978Head and Neck Squamous Cell Carcinoma
\n", "
" ], "text/plain": [ " proj__project_id submitter_id proj__name\n", "0 TCGA-HNSC TCGA-CN-5363 Head and Neck Squamous Cell Carcinoma\n", "1 TCGA-HNSC TCGA-CN-5365 Head and Neck Squamous Cell Carcinoma\n", "2 TCGA-HNSC TCGA-CN-A642 Head and Neck Squamous Cell Carcinoma\n", "3 TCGA-HNSC TCGA-CR-7380 Head and Neck Squamous Cell Carcinoma\n", "4 TCGA-HNSC TCGA-CV-5978 Head and Neck Squamous Cell Carcinoma" ] }, "metadata": { "tags": [] }, "execution_count": 6 } ] }, { "cell_type": "markdown", "metadata": { "id": "XBHrxYK_1v6f" }, "source": [ "Now that wasn't so difficult! Have fun exploring and analyzing the ISB-CGC Data!" ] }, { "cell_type": "markdown", "metadata": { "id": "vcRchKFAuiz5" }, "source": [ "## Where to Go Next\n", "\n", "Access, Explore and Analyze Large-Scale Cancer Data Through the Google Cloud! :)\n", "\n", "Getting Started for Free:\n", "* [Free Cloud Credits from ISB-CGC for Cancer Research](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/HowtoRequestCloudCredits.html)\n", "* [Google Free Tier with up to 1TB of free queries a month](https://cloud.google.com/free)\n", "\n", "ISB-CGC Links:\n", "\n", "* [ISB-CGC Landing Page](https://isb-cgc.appspot.com/)\n", "* [ISB-CGC Documentation](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/)\n", "* [How to Get Started on ISB-CGC](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/HowToGetStartedonISB-CGC.html)\n", "* [How to access Google BigQuery](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/progapi/bigqueryGUI/HowToAccessBigQueryFromTheGoogleCloudPlatform.html)\n", "* [Community Notebook Repository](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/HowTos.html)\n", "* [Query of the Month](https://isb-cancer-genomics-cloud.readthedocs.io/en/latest/sections/QueryOfTheMonthClub.html)\n", "\n", "Google Tutorials:\n", "\n", "* [Google's What is BigQuery?](https://cloud.google.com/bigquery/docs/introduction)\n", "* [Google Cloud Client Library for Python](https://googleapis.github.io/google-cloud-python/latest/index.html)" ] } ] }