{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jcyB6NFmQReL"
   },
   "source": [
    "# Tutorial: Describe Neuroscience Dataset using MINDS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize and configure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install nexusforge==0.7.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install allensdk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install neurom[plotly]==3.0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade nest-asyncio==1.5.4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade jinja2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get an authentication token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Nexus sandbox application](https://sandbox.bluebrainnexus.io) can be used to get a token:\n",
    "\n",
    "- Step 1: From the [web page](https://sandbox.bluebrainnexus.io), click on the login button in the top right corner and follow the instructions on screen.\n",
    "\n",
    "- Step 2: Once logged in, click on the button on the top right that displays your GitHub username. From the dropdown select `Copy token` option. This will copy the token to your clipboard.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once a token is obtained, proceed to paste it as the value of the `TOKEN` variable below.\n",
    "\n",
    "__Important__: A Nexus token is valid for 8 hours, if your working session is open for more than 8 hours, you may need to refresh the value of the token and reintialize the forge client in the _'Configure a forge client to store, manage and access datasets'_ section below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TOKEN = getpass.getpass()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure a forge client to store, manage and access datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import uuid\n",
    "import base64\n",
    "import requests\n",
    "import json\n",
    "from pathlib import Path\n",
    "\n",
    "from kgforge.core import KnowledgeGraphForge\n",
    "from kgforge.specializations.mappings import DictionaryMapping\n",
    "\n",
    "from allensdk.api.queries.cell_types_api import CellTypesApi\n",
    "from allensdk.core.cell_types_cache import CellTypesCache"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.get('https://raw.githubusercontent.com/BlueBrain/nexus/d9f6cb83a27149c29bc604f3c34ea00c9ad64e67/docs/src/main/paradox/docs/getting-started/notebooks/rdfmodel/jsonldcontext.json')\n",
    "dirpath = './rdfmodel'\n",
    "Path(dirpath).mkdir(parents=True, exist_ok=True)\n",
    "with open(f'{dirpath}/jsonldcontext.json', 'w') as outfile:\n",
    "    json.dump(r.json(), outfile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ORG = \"github-users\"\n",
    "PROJECT = \"\"  # Provide here the automatically created project name created when you logged into the Nexus sandbox instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge = KnowledgeGraphForge(\"https://raw.githubusercontent.com/BlueBrain/nexus/d9f6cb83a27149c29bc604f3c34ea00c9ad64e67/docs/src/main/paradox/docs/getting-started/notebooks/forge.yml\",\n",
    "                            bucket=f\"{ORG}/{PROJECT}\",\n",
    "                            endpoint=\"https://sandbox.bluebrainnexus.io/v1\",\n",
    "                            token=TOKEN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9W5M5Ck9Tq7q"
   },
   "source": [
    "## Download datasets from Allen Cell Types Database"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download mouse neuron morphology from the Allen Cell Types Database"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be downloading mouse neuron morphology data from the [Allen Cell Types Database](https://celltypes.brain-map.org/). The [AllenSDK](https://allensdk.readthedocs.io/en/latest/) can be used for data download."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ALLEN_DIR = \"allen_cell_types_database\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ctc = CellTypesCache(manifest_file=f\"{ALLEN_DIR}/manifest.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_CELLS = 1\n",
    "SPECIES = CellTypesApi.MOUSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nm_allen_identifiers = [cell[\"id\"] for cell in ctc.get_cells(species=[SPECIES], require_reconstruction = True)][:MAX_CELLS]\n",
    "print(f\"Selected a mouse neuron with identifier: {nm_allen_identifiers}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(f\"{ALLEN_DIR}/cells.json\") as f:\n",
    "    allen_cell_types_metadata = json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nm_allen_metadata = [neuron for neuron in allen_cell_types_metadata if neuron[\"specimen__id\"] in nm_allen_identifiers]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Metadata of the neuron {nm_allen_identifiers}:\")\n",
    "nm_allen_metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true,
    "toc-nb-collapsed": true
   },
   "source": [
    "### Download one mouse neuron morphology reconstructed from the selected neuron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be downloading one mouse neuron morphology from the [Allen Cell Types Database](https://celltypes.brain-map.org/) using the [AllenSDK](https://allensdk.readthedocs.io/en/latest/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for identifier in nm_allen_identifiers:\n",
    "    ctc.get_reconstruction(identifier)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true,
    "toc-nb-collapsed": true
   },
   "source": [
    "### Download one mouse neuron electrophysiology recording from the selected neuron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be downloading one mouse neuron electrophysiology from the [Allen Cell Types Database](https://celltypes.brain-map.org/) using the [AllenSDK](https://allensdk.readthedocs.io/en/latest/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for identifier in nm_allen_identifiers:\n",
    "    ctc.get_ephys_data(identifier)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transform Allen Cell Types Database Metadata to [Neuroshapes' MINDS](https://bbp-nexus.epfl.ch/datamodels/class-schemadataset.html) metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Map the Allen Cell Types Database neuron morphologies metadata to Neuroshapes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "allen_nm_mapping = DictionaryMapping.load(\"https://raw.githubusercontent.com/BlueBrain/nexus/d9f6cb83a27149c29bc604f3c34ea00c9ad64e67/docs/src/main/paradox/docs/getting-started/notebooks/mappings/allen_morphology_dataset.hjson\")\n",
    "nm_allen_resources = forge.map(nm_allen_metadata, allen_nm_mapping, na='')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Map the Allen Cell Types Database neuron electrophysiology recording to Neuroshapes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "allen_ephys_mapping = DictionaryMapping.load(\"https://raw.githubusercontent.com/BlueBrain/nexus/d9f6cb83a27149c29bc604f3c34ea00c9ad64e67/docs/src/main/paradox/docs/getting-started/notebooks/mappings/allen_ephys_dataset.hjson\")\n",
    "nephys_allen_resources = forge.map(nm_allen_metadata, allen_ephys_mapping, na='')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Register\n",
    "\n",
    "If the registration fails, try refreshing the access token and reinitializing the forge client in the _'Configure a forge client to store, manage and access datasets'_ section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Register the Allen Cell Types Database neuron morphology"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nm_allen_resources.id = forge.format(\"identifier\", \"neuronmorphologies\", str(uuid.uuid4()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.register(nm_allen_resources)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Register the Allen Cell Types Database neuron electrophysiology recording"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nephys_allen_resources.id = forge.format(\"identifier\", \"traces\", str(uuid.uuid4()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.register(nephys_allen_resources)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Access"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set filters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_type = \"NeuronMorphology\"\n",
    "\n",
    "filters = {\"type\": _type}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_results = 10  # You can limit the number of results, pass `None` to fetch all the results\n",
    "\n",
    "data = forge.search(filters, limit=number_of_results)\n",
    "\n",
    "print(f\"{str(len(data))} dataset(s) of type {_type} found\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display the results as pandas dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "property_to_display = [\"id\",\"name\",\"subject\",\"brainLocation.brainRegion.id\",\"brainLocation.brainRegion.label\",\"brainLocation.layer.id\",\"brainLocation.layer.label\", \"contribution\",\"brainLocation.layer.id\",\"brainLocation.layer.label\",\"distribution.name\",\"distribution.contentUrl\",\"distribution.encodingFormat\"]\n",
    "reshaped_data = forge.reshape(data, keep=property_to_display)\n",
    "\n",
    "forge.as_dataframe(reshaped_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dirpath = \"./downloaded/\"\n",
    "forge.download(data, \"distribution.contentUrl\", dirpath, overwrite=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls ./downloaded/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display a result as 3D Neuron Morphology"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from neurom import load_morphology\n",
    "from neurom.view.plotly_impl import plot_morph3d\n",
    "import IPython"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "neuron = load_morphology(f\"{dirpath}/{data[0].distribution.name}\")\n",
    "plot_morph3d(neuron, inline=False)\n",
    "IPython.display.HTML(filename='./morphology-3D.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Version the dataset\n",
    "Tagging a dataset is equivalent to `git tag`. It allows to version a dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.tag(data, value=\"releaseV112\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The version argument can be specified to retrieve the dataset at a given tag.\n",
    "\n",
    "tagged_data = forge.retrieve(id=data[0].id, version=\"releaseV112\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.as_dataframe(tagged_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[0].description=\"Neuron Morphology from Allen\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.update(data[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "non_tagged_data = forge.retrieve(id=data[0].id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.as_dataframe(non_tagged_data)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Step by step jupyter notebook for bringing data to Nexus v1.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3.7 (nexusforgelatest)",
   "language": "python",
   "name": "nexusforgelatest"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}