{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-23T18:50:19.036357Z",
     "start_time": "2019-09-23T18:50:19.031896Z"
    }
   },
   "source": [
    "# Querying\n",
    "\n",
    "This notebook demonstrates Nexus Forge data [querying features](https://nexus-forge.readthedocs.io/en/latest/interaction.html#querying)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-23T18:50:20.068658Z",
     "start_time": "2019-09-23T18:50:19.054054Z"
    }
   },
   "outputs": [],
   "source": [
    "from kgforge.core import KnowledgeGraphForge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A configuration file is needed in order to create a KnowledgeGraphForge session. A configuration can be generated using the notebook [00-Initialization.ipynb](00%20-%20Initialization.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge = KnowledgeGraphForge(\"../../configurations/forge.yml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from kgforge.core import Resource\n",
    "from kgforge.specializations.resources import Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### latest version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jane = Resource(type=\"Person\", name=\"Jane Doe\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.register(jane)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resource = forge.retrieve(jane.id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resource == jane"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### specific version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jane = Resource(type=\"Person\", name=\"Jane Doe\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.register(jane)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.tag(jane, \"v1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jane.email = \"jane.doe@epfl.ch\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.update(jane)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-23T18:50:21.317601Z",
     "start_time": "2019-09-23T18:50:21.310418Z"
    }
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    # DemoStore\n",
    "    print(jane._store_metadata.version)\n",
    "except:\n",
    "    # BlueBrainNexus\n",
    "    print(jane._store_metadata._rev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-23T18:50:21.332678Z",
     "start_time": "2019-09-23T18:50:21.322025Z"
    }
   },
   "outputs": [],
   "source": [
    "jane_v1 = forge.retrieve(jane.id, version=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-23T18:50:21.370051Z",
     "start_time": "2019-09-23T18:50:21.363782Z"
    }
   },
   "outputs": [],
   "source": [
    "jane_v1_tag = forge.retrieve(jane.id, version=\"v1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-23T18:50:21.379911Z",
     "start_time": "2019-09-23T18:50:21.373539Z"
    }
   },
   "outputs": [],
   "source": [
    "jane_v1 == jane_v1_tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### crossbucket retrieval\n",
    "It is possible to retrieve resources stored in buckets different then the configured one. The configured store should of course support it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resource = forge.retrieve(jane.id, cross_bucket=True) # cross_bucket defaults to False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### error handling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resource = forge.retrieve(\"123\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resource is None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: DemoModel and RdfModel schemas have not been synchronized yet. This section is to be run with RdfModel. Commented lines are for DemoModel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jane = Resource(type=\"Person\", name=\"Jane Doe\")\n",
    "contribution_jane = Resource(type=\"Contribution\", agent=jane)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "john = Resource(type=\"Person\", name=\"John Smith\")\n",
    "contribution_john = Resource(type=\"Contribution\", agent=john)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = Dataset(forge, type=\"Dataset\", contribution=[contribution_jane, contribution_john])\n",
    "dataset.add_distribution(\"../../data/associations.tsv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.register(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.as_json(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Paths as filters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `paths` method load the data structure for the given type.\n",
    "\n",
    "Please refer to the [Modeling.ipynb](11%20-%20Modeling.ipynb) notebook to learn about modeling and types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p = forge.paths(\"Dataset\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have autocompletion on `p` and this can be used to create search filters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: There is a known issue for RdfModel which requires using `p.type.id` instead of `p.type`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resources = forge.search(p.type.id==\"Dataset\", limit=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(resources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(resources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.as_dataframe(resources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.as_json(resources[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.as_dataframe(resources, store_metadata=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Nested property querying"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Property autocompletion is available on a path `p` even for nested properties like `p.contribution`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Search for resources of type Dataset and with a attached files of content type text/tab-separated-values\n",
    "resources = forge.search(p.type.id == \"Dataset\", p.distribution.encodingFormat == \"text/tab-separated-values\", limit=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(resources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "forge.as_dataframe(resources)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dict as filters\n",
    "A dictionary can be provided for filters:\n",
    "* {'type': {'id':'Dataset'}} is equivalent to p.type.id==\"Dataset\"\n",
    "* only the '==' operator is supported\n",
    "* nested dict are supported\n",
    "* it is not mandatory for the provided properties and values to be defined in the forge model. Results will be retrieved if there are corresponding data in the store.\n",
    "\n",
    "This feature is not supported when using the DemoStore\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Search for resources of type Dataset and with conribution from agent named \"Jane Doe\"\n",
    "filters = {\"type\": \"Dataset\", \"contribution\":{\"type\":\"Contribution\", \"agent\":{\"name\":\"Jane Doe\"}}}\n",
    "resources = forge.search(filters, limit=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(resources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(resources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.as_dataframe(resources, store_metadata=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Crossbucket search\n",
    "It is possible to search for resources stored in buckets different then the configured one. The configured store should of course support it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resources = forge.search(p.type.id == \"Dataset\", limit=3, cross_bucket=True)  # cross_bucket defaults to False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(resources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(resources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.as_dataframe(resources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Furthermore it is possible to filter by bucket when cross_bucket is set to True. Setting a bucket value when cross_bucket is False will trigger a not_supported exception.\n",
    "resources = forge.search(p.type.id == \"Dataset\", limit=3, cross_bucket=True, bucket=<str>)  # add a bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(resources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(resources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.as_dataframe(resources)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graph traversing\n",
    "\n",
    "SPARQL is used as a query language.\n",
    "\n",
    "A SPARQL query rewriting strategy lets users write simplified queries without using prefix declarations, prefix names or long IRIs. With this strategy, the user could only provides type and property names. For a given entity type, these names could be seen in its template.\n",
    "\n",
    "Please refer to the [Modeling.ipynb](11%20-%20Modeling.ipynb) notebook to learn about templates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: DemoStore doesn't implement SPARQL operations yet. Please use another store for this section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: DemoModel and RdfModel schemas have not been synchronized yet. This section is to be run with RdfModel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jane = Resource(type=\"Person\", name=\"Jane Doe\")\n",
    "contribution_jane = Resource(type=\"Contribution\", agent=jane)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "john = Resource(type=\"Person\", name=\"John Smith\")\n",
    "contribution_john = Resource(type=\"Contribution\", agent=john)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "association = Resource(type=\"Dataset\", contribution=[contribution_jane, contribution_john])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.register(association)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "forge.template(\"Dataset\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prefix and namespace free SPARQL query\n",
    "\n",
    "When a forge RDFModel is configured, then there is no need to provide prefixes and namespaces when writing a SPARQL query. Prefixes and namespaces will be automatically inferred from the provided schemas and/or JSON-LD context and the query rewritten accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"\"\"\n",
    "    SELECT ?id ?name\n",
    "    WHERE {\n",
    "        ?id a Dataset ;\n",
    "        contribution/agent ?contributor.\n",
    "        ?contributor name ?name.\n",
    "    }\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resources = forge.sparql(query, limit=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(resources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(resources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(resources[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.as_dataframe(resources)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### rewritten query display\n",
    "The prefix free SPARQL query provided above is rewritten as the ouput of cell when a forge Model is configured. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resources = forge.sparql(query, limit=3, debug=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SPARQL query\n",
    "\n",
    "regular SPARQL query can also be provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"\"\"\n",
    "PREFIX dc: <http://purl.org/dc/elements/1.1/>\n",
    "   PREFIX dcat: <http://www.w3.org/ns/dcat#>\n",
    "   PREFIX dcterms: <http://purl.org/dc/terms/>\n",
    "   PREFIX mba: <http://api.brain-map.org/api/v2/data/Structure/>\n",
    "   PREFIX nsg: <https://neuroshapes.org/>\n",
    "   PREFIX owl: <http://www.w3.org/2002/07/owl#>\n",
    "   PREFIX prov: <http://www.w3.org/ns/prov#>\n",
    "   PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n",
    "   PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n",
    "   PREFIX schema: <http://schema.org/>\n",
    "   PREFIX sh: <http://www.w3.org/ns/shacl#>\n",
    "   PREFIX shsh: <http://www.w3.org/ns/shacl-shacl#>\n",
    "   PREFIX skos: <http://www.w3.org/2004/02/skos/core#>\n",
    "   PREFIX vann: <http://purl.org/vocab/vann/>\n",
    "   PREFIX void: <http://rdfs.org/ns/void#>\n",
    "   PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>\n",
    "   PREFIX : <https://neuroshapes.org/>\n",
    "   SELECT ?id ?name\n",
    "   WHERE {\n",
    "       ?id a schema:Dataset ;\n",
    "       nsg:contribution/prov:agent ?contributor.\n",
    "       ?contributor schema:name ?name.\n",
    "   }\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resources = forge.sparql(query, limit=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(resources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(resources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(resources[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.as_dataframe(resources)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: DemoStore doesn't implement file operations yet. Please use another store for this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jane = Resource(type=\"Person\", name=\"Jane Doe\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! ls -p ../../data | egrep -v /$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "distribution = forge.attach(\"../../data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "association = Resource(type=\"Association\", agent=jane, distribution=distribution)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.register(association)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forge.download(association, \"distribution.contentUrl\", \"./downloaded/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! ls -l ./downloaded/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ! rm -R ./downloaded/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (nexusforgelatest)",
   "language": "python",
   "name": "nexusforgelatest"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}