{ "metadata": { "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5-final" }, "orig_nbformat": 2, "kernelspec": { "name": "Python 3.8.5 64-bit", "display_name": "Python 3.8.5 64-bit", "metadata": { "interpreter": { "hash": "1ee38ef4a5a9feb55287fd749643f13d043cb0a7addaab2a9c224cbe137c0062" } } } }, "nbformat": 4, "nbformat_minor": 2, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Machine Actionable Data Management Plan connections\n", "\n", "Data management plans (DMPs) are documents accompanying research proposals and project outputs. DMPs are created as textual narratives and describe the data and tools employed in scientific investigations.They are sometimes seen as an administrative exercise and not as an integral part of research practice. Machine Actionable DMPs (maDMPs) take the DMP concept further by using PIDs and PIDs services to connect all resources associated with a DMP.\n", "\n", "This notebook displays in a human-friendly way all of the connections embedded in a maDMP. By the end of this notebook, you will be able to succinctly display the essential components of the maDMP vision using persistent identifiers (PIDs): Open Researcher and Contributor IDs (ORCIDs), funders IDs, organizations Org IDs, and Dataset IDs (DOIs). To demonstrate this we use an example DMP, viz. https://doi.org/10.4124/test/.879w8. The notebook fetches all the PIDs associated with this DMP and displays it in a Tree Diagram. See below. The diagram puts the DMP at center and there are four main branches: datasets, funders, organisations, and people. Each branch gives birth to individual entities of those branches. For example, the name of all the people that contributed to the DMP.\n", "\n", "\n", "\n", "\n", "\n", "The process of displaying the DMP visulisation is very simple. First, and after a initial setup, we fetch all the we need from the DataCite GraphQL API. Then, we transform this data into a data structure that can be use for visulisation. Finally, we take the data tranformation and supply it to a Vega visulisation specification to generate the Chart you can see above. \n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "output_type": "display_data", "data": { "text/plain": "Dropdown(description='Choose DOI:', options=('https://doi.org/10.48321/D1H59R', 'https://doi.org/10.1575/1912/…", "application/vnd.jupyter.widget-view+json": { "version_major": 2, "version_minor": 0, "model_id": "f0ab1d32a7ce46d88860e269b3bd7e63" } }, "metadata": {} } ], "source": [ "import ipywidgets as widgets\n", "f = widgets.Dropdown(\n", " options=['https://doi.org/10.48321/D1H59R', 'https://doi.org/10.1575/1912/bco-dmo.775500.1', 'https://doi.org/10.48321/D1G59F'],\n", " value='https://doi.org/10.48321/D1H59R',\n", " description='Choose DOI:',\n", " disabled=False,\n", ")\n", "f" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "%%capture\n", "# Install required Python packages\n", "!pip install dfply altair altair_saver vega altair_viewer" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import json\n", "import pandas as pd\n", "import numpy as np\n", "from dfply import *\n", "import altair.vega.v5 as alt\n", "from altair_saver import save\n", "# alt.renderers.enable('notebook')\n", "\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# Prepare the GraphQL client\n", "import requests\n", "from IPython.display import display, Markdown\n", "from gql import gql, Client\n", "from gql.transport.requests import RequestsHTTPTransport\n", "\n", "_transport = RequestsHTTPTransport(\n", " url='https://api.datacite.org/graphql',\n", " use_json=True,\n", ")\n", "\n", "client = Client(\n", " transport=_transport,\n", " fetch_schema_from_transport=True,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fetching Data\n", "\n", "We obtain all the data from the DataCite GraphQL API.\n" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "# Generate the GraphQL query to retrieve up to 100 outputs of University of Oxford, with at least 100 views each.\n", "query_params = {\n", " \"id\" : f.value,\n", " \"maxOutputs\": 100,\n", " \"minViews\" : 100\n", "}\n", "\n", "query = gql(\"\"\"query getOutputs($id: ID!)\n", "{\n", " dataManagementPlan(id: $id) {\n", " id\n", " name: titles(first:1) {\n", " title\n", " }\n", " datasets: citations(query:\"types.resourceTypeGeneral:Dataset\") {\n", " totalCount\n", " nodes {\n", " id: doi\n", " name: titles(first:1) {\n", " title\n", " }\n", " }\n", " }\n", " publications: citations(query:\"types.resourceTypeGeneral:Text\") {\n", " totalCount\n", " nodes {\n", " id: doi\n", " name: titles(first:1) {\n", " title\n", " }\n", " }\n", " }\n", " hostingInstitution: contributors(contributorType: \"HostingInstitution\") {\n", " id\n", " name\n", " contributorType\n", " }\n", " producer: contributors(contributorType: \"Producer\") {\n", " id\n", " name\n", " contributorType \n", " } \n", " fundingReferences {\n", " id: funderIdentifier\n", " name: funderName\n", " }\n", " creators {\n", " id\n", " name\n", " type\n", " affiliation{\n", " id\n", " name\n", " }\n", " } \n", " contributors {\n", " id\n", " name\n", " contributorType\n", " affiliation{\n", " id\n", " name\n", " }\n", " } \n", " }\n", "}\n", "\"\"\")" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "def get_data():\n", " return client.execute(query, variable_values=json.dumps(query_params))[\"dataManagementPlan\"]\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data Transformation\n", "\n", "Simple transformations are performed to convert the graphql response into an array that can be take by Vega." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "def get_affiliation(series_element):\n", " if len(series_element) == 0:\n", " return \"None\"\n", " return series_element[0]['name']" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "def add_node_attributes(dataframe, parent=2):\n", " \"\"\"Modifies each item to include attributes needed for the node visulisation\n", "\n", " Parameters:\n", " dataframe (dataframe): A dataframe with all the itemss\n", " parent (int): The id of the parent node\n", "\n", " Returns:\n", " dataframe:Returning vthe same dataframe with new attributes\n", "\n", " \"\"\"\n", " if (dataframe) is None:\n", " return pd.DataFrame() \n", " else: \n", " # print(dataframe) \n", " return (dataframe >>\n", " mutate(\n", " id = X.id,\n", " tooltip = X.id,\n", " parent = parent,\n", " ))\n", " " ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "def create_node(array=[], parent=2):\n", " \"\"\"creates a node for the chart and formats it\n", "\n", " Parameters:\n", " array (array): An array with all the itemss\n", " parent (int): The id of the parent node\n", "\n", " Returns:\n", " dict:Dict with all the nodes\n", "\n", " \"\"\"\n", " # print(array)\n", " if len(array) == 0:\n", " return {} \n", " else:\n", " # return {} if (array) is None else array\n", " df = add_node_attributes(pd.DataFrame(array,columns=array[0].keys()), parent)\n", " return df.to_dict(orient='records')\n", " " ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": [ "def merge_nodes(dmpTitle,id,dataset=[],references=[],funders=[],orgs=[],people=[]):\n", " \"\"\"Merges all the nodes lists\n", "\n", " Parameters:\n", " datasets (array): dataset nodes\n", " funders (array): funders nodes\n", " orgs (array): orgs nodes\n", " people (array): people nodes\n", "\n", " Returns:\n", " array:Array with all the nodes\n", "\n", " \"\"\"\n", " dataset = [] if len(dataset) == 0 else dataset\n", " references = [] if len(references) == 0 else references\n", " funders = [] if len(funders) == 0 else funders\n", " orgs = [] if len(orgs) == 0 else orgs\n", " people = [] if len(people) == 0 else people\n", "\n", " dmp = {\"id\":id, \"name\": dmpTitle}\n", " datasets_node = {\"id\":2, \"name\": \"Datasets\", \"parent\":id}\n", " references_node = {\"id\":6, \"name\": \"Publications\", \"parent\":id}\n", " funders_node = {\"id\":3, \"name\": \"Funders\", \"parent\":id}\n", " organisations_node = {\"id\":4, \"name\": \"Organisations\", \"parent\":id}\n", " people_node = {\"id\":5, \"name\": \"People\", \"parent\":id}\n", " nodes_list = [dmp, datasets_node, references_node, funders_node,organisations_node,people_node] + dataset + references + funders + orgs + people,\n", " # return np.array(nodes_list, dtype=object) \n", " return nodes_list[0]" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "def get_title(series_element):\n", " if len(series_element) == 0:\n", " return \"None\"\n", " return series_element[0]['title']" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "def extract_titles(list):\n", " if len(list) == 0:\n", " return []\n", " return (pd.DataFrame(list) >> \n", " mutate(\n", " name = X.name.apply(get_title)\n", " )).to_dict('records')\n" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [], "source": [ "def filter_people(list):\n", " return (pd.DataFrame(data[\"creators\"] + data[\"contributors\"]) >> \n", " filter_by(\n", " X.type == \"Person\"\n", " )).to_dict('records')" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "tags": [] }, "outputs": [], "source": [ "data = get_data()\n", "\n", "datasets = create_node(extract_titles(data[\"datasets\"][\"nodes\"]),2)\n", "references = create_node(extract_titles(data[\"publications\"][\"nodes\"]),6)\n", "orgs = create_node(data[\"hostingInstitution\"]+data[\"producer\"],4)\n", "people = create_node(filter_people(data[\"creators\"] + data[\"contributors\"]),5)\n", "dmp_title = str('\"' + data[\"name\"][0][\"title\"] + '\"') \n", "funders = create_node(data[\"fundingReferences\"],3)\n", "id = data[\"id\"]\n", "nodes = merge_nodes(\" \",id, datasets, references, funders, orgs, people)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Visulization\n", "\n", "All transofrmed data is then feeded into a Vega specification for display.\n", "\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [], "source": [ "def vega_template(data):\n", " \"\"\"Injects data into the vega specification\n", "\n", " Parameters:\n", " data (array): Array of nodes\n", "\n", " Returns:\n", " VegaSpec:Specification with data\n", "\n", " \"\"\"\n", " return \"\"\"\n", "{\n", " \"$schema\": \"https://vega.github.io/schema/vega/v5.json\",\n", " \"description\": \"An example of a radial layout for a node-link diagram of hierarchical data.\",\n", " \"width\": 1024,\n", " \"height\": 720,\n", " \"padding\": 5,\n", " \"autosize\": \"none\",\n", " \"signals\": [\n", " {\"name\": \"Chart\", \"value\": \"\"\" + dmp_title + \"\"\", \"bind\": {\"input\": \"url\", \"size\":100}},\n", " {\"name\": \"labels\", \"value\": true, \"bind\": {\"input\": \"checkbox\"}},\n", " {\n", " \"name\": \"radius\",\n", " \"value\": 280,\n", " \"bind\": {\"input\": \"range\", \"min\": 20, \"max\": 600}\n", " },\n", " {\n", " \"name\": \"extent\",\n", " \"value\": 360,\n", " \"bind\": {\"input\": \"range\", \"min\": 0, \"max\": 360, \"step\": 1}\n", " },\n", " {\n", " \"name\": \"rotate\",\n", " \"value\": 0,\n", " \"bind\": {\"input\": \"range\", \"min\": 0, \"max\": 360, \"step\": 1}\n", " },\n", " {\n", " \"name\": \"layout\",\n", " \"value\": \"cluster\",\n", " \"bind\": {\"input\": \"radio\", \"options\": [\"tidy\", \"cluster\"]}\n", " },\n", " {\n", " \"name\": \"links\",\n", " \"value\": \"orthogonal\",\n", " \"bind\": {\n", " \"input\": \"select\",\n", " \"options\": [\"line\", \"curve\", \"diagonal\", \"orthogonal\"]\n", " }\n", " },\n", " {\"name\": \"originX\", \"update\": \"width / 2\"},\n", " {\"name\": \"originY\", \"update\": \"height / 2\"}\n", " ],\n", " \"data\": [\n", " {\n", " \"name\": \"tree\",\n", " \"values\": \"\"\" + data + \"\"\",\n", " \"transform\": [\n", " {\"type\": \"stratify\", \"key\": \"id\", \"parentKey\": \"parent\"},\n", " {\n", " \"type\": \"tree\",\n", " \"method\": {\"signal\": \"layout\"},\n", " \"size\": [1, {\"signal\": \"radius\"}],\n", " \"as\": [\"alpha\", \"radius\", \"depth\", \"children\"]\n", " },\n", " {\n", " \"type\": \"formula\",\n", " \"expr\": \"(rotate + extent * datum.alpha + 270) % 360\",\n", " \"as\": \"angle\"\n", " },\n", " {\"type\": \"formula\", \"expr\": \"PI * datum.angle / 180\", \"as\": \"radians\"},\n", " {\n", " \"type\": \"formula\",\n", " \"expr\": \"inrange(datum.angle, [90, 270])\",\n", " \"as\": \"leftside\"\n", " },\n", " {\n", " \"type\": \"formula\",\n", " \"expr\": \"originX + datum.radius * cos(datum.radians)\",\n", " \"as\": \"x\"\n", " },\n", " {\n", " \"type\": \"formula\",\n", " \"expr\": \"originY + datum.radius * sin(datum.radians)\",\n", " \"as\": \"y\"\n", " }\n", " ]\n", " },\n", " {\n", " \"name\": \"links\",\n", " \"source\": \"tree\",\n", " \"transform\": [\n", " {\"type\": \"treelinks\"},\n", " {\n", " \"type\": \"linkpath\",\n", " \"shape\": {\"signal\": \"links\"},\n", " \"orient\": \"radial\",\n", " \"sourceX\": \"source.radians\",\n", " \"sourceY\": \"source.radius\",\n", " \"targetX\": \"target.radians\",\n", " \"targetY\": \"target.radius\"\n", " }\n", " ]\n", " }\n", " ],\n", " \"scales\": [\n", " {\n", " \"name\": \"color\",\n", " \"type\": \"linear\",\n", " \"range\": {\"scheme\": \"viridis\"},\n", " \"domain\": {\"data\": \"tree\", \"field\": \"depth\"},\n", " \"zero\": true\n", " }\n", " ],\n", " \"marks\": [\n", " {\n", " \"type\": \"path\",\n", " \"from\": {\"data\": \"links\"},\n", " \"encode\": {\n", " \"update\": {\n", " \"x\": {\"signal\": \"originX\"},\n", " \"y\": {\"signal\": \"originY\"},\n", " \"path\": {\"field\": \"path\"},\n", " \"stroke\": {\"value\": \"#ccc\"}\n", " }\n", " }\n", " },\n", " {\n", " \"type\": \"symbol\",\n", " \"from\": {\"data\": \"tree\"},\n", " \"encode\": {\n", " \"enter\": {\n", " \"size\": {\"value\": 300}, \"stroke\": {\"value\": \"#fff\"}\n", " },\n", " \"update\": {\n", " \"x\": {\"field\": \"x\"},\n", " \"y\": {\"field\": \"y\"},\n", " \"fill\": {\"scale\": \"color\", \"field\": \"depth\"}\n", " }\n", " }\n", " },\n", " {\n", " \"type\": \"text\",\n", " \"from\": {\"data\": \"tree\"},\n", " \"encode\": {\n", " \"enter\": {\n", " \"text\": {\"field\": \"name\"},\n", " \"fontSize\": {\"value\": 12},\n", " \"baseline\": {\"value\": \"middle\"},\n", " \"tooltip\": {\"signal\": \n", " \"{'Identifier': datum.tooltip, 'Affiliation': datum.affiliation, 'Contribution': datum.contributorType}\"}\n", " },\n", " \"update\": {\n", " \"x\": {\"field\": \"x\"},\n", " \"y\": {\"field\": \"y\"},\n", " \"dx\": {\"signal\": \"(datum.leftside ? -1 : 1) * 12\"},\n", " \"align\": {\"signal\": \"datum.leftside ? 'right' : 'left'\"},\n", " \"opacity\": {\"signal\": \"labels ? 1 : 0\"}\n", " }\n", " }\n", " }\n", " ]\n", "}\n", "\n", "\"\"\"" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "output_type": "display_data", "data": { "application/vnd.vega.v5+json": { "$schema": "https://vega.github.io/schema/vega/v5.json", "description": "An example of a radial layout for a node-link diagram of hierarchical data.", "width": 1024, "height": 720, "padding": 5, "autosize": "none", "signals": [ { "name": "Chart", "value": "LTREB: Drivers of temperate forest carbon storage from canopy closure through successional time", "bind": { "input": "url", "size": 100 } }, { "name": "labels", "value": true, "bind": { "input": "checkbox" } }, { "name": "radius", "value": 280, "bind": { "input": "range", "min": 20, "max": 600 } }, { "name": "extent", "value": 360, "bind": { "input": "range", "min": 0, "max": 360, "step": 1 } }, { "name": "rotate", "value": 0, "bind": { "input": "range", "min": 0, "max": 360, "step": 1 } }, { "name": "layout", "value": "cluster", "bind": { "input": "radio", "options": [ "tidy", "cluster" ] } }, { "name": "links", "value": "orthogonal", "bind": { "input": "select", "options": [ "line", "curve", "diagonal", "orthogonal" ] } }, { "name": "originX", "update": "width / 2" }, { "name": "originY", "update": "height / 2" } ], "data": [ { "name": "tree", "values": [ { "id": "https://doi.org/10.48321/d1h59r", "name": " " }, { "id": 2, "name": "Datasets", "parent": "https://doi.org/10.48321/d1h59r" }, { "id": 6, "name": "Publications", "parent": "https://doi.org/10.48321/d1h59r" }, { "id": 3, "name": "Funders", "parent": "https://doi.org/10.48321/d1h59r" }, { "id": 4, "name": "Organisations", "parent": "https://doi.org/10.48321/d1h59r" }, { "id": 5, "name": "People", "parent": "https://doi.org/10.48321/d1h59r" }, { "id": "10.6073/pasta/e43651bde3564e402cd0595e3ef26994", "name": "Forest tree, woody debris, and soil inventory data from long-term research plots for LTREB at the University of Michigan Biological Station", "tooltip": "10.6073/pasta/e43651bde3564e402cd0595e3ef26994", "parent": 2 }, { "id": "10.1007/s00442-018-4236-x", "name": "Effects of canopy structure and species diversity on primary production in upper Great Lakes forests", "tooltip": "10.1007/s00442-018-4236-x", "parent": 6 }, { "id": "10.1111/nph.15227", "name": "Forest aging, disturbance and the carbon cycle", "tooltip": "10.1111/nph.15227", "parent": 6 }, { "id": "10.1139/cjfr-2018-0211", "name": "Multidecadal trajectories of soil chemistry and nutrient availability following cutting vs. burning disturbances in Upper Great Lakes forests", "tooltip": "10.1139/cjfr-2018-0211", "parent": 6 }, { "id": "https://api.crossref.org/funders/100000001", "name": "National Science Foundation (NSF)", "tooltip": "https://api.crossref.org/funders/100000001", "parent": 3 }, { "id": "https://ror.org/03yrm5c26", "name": "California Digital Library", "contributorType": "HostingInstitution", "tooltip": "https://ror.org/03yrm5c26", "parent": 4 }, { "id": "https://ror.org/00jmfr291", "name": "University Of Michigan", "contributorType": "Producer", "tooltip": "https://ror.org/00jmfr291", "parent": 4 }, { "id": "https://orcid.org/0000-0001-8258-8335", "name": "Nave, Luke", "type": "Person", "affiliation": [ { "id": null, "name": "University of Michigan" } ], "contributorType": "nan", "tooltip": "https://orcid.org/0000-0001-8258-8335", "parent": 5 } ], "transform": [ { "type": "stratify", "key": "id", "parentKey": "parent" }, { "type": "tree", "method": { "signal": "layout" }, "size": [ 1, { "signal": "radius" } ], "as": [ "alpha", "radius", "depth", "children" ] }, { "type": "formula", "expr": "(rotate + extent * datum.alpha + 270) % 360", "as": "angle" }, { "type": "formula", "expr": "PI * datum.angle / 180", "as": "radians" }, { "type": "formula", "expr": "inrange(datum.angle, [90, 270])", "as": "leftside" }, { "type": "formula", "expr": "originX + datum.radius * cos(datum.radians)", "as": "x" }, { "type": "formula", "expr": "originY + datum.radius * sin(datum.radians)", "as": "y" } ] }, { "name": "links", "source": "tree", "transform": [ { "type": "treelinks" }, { "type": "linkpath", "shape": { "signal": "links" }, "orient": "radial", "sourceX": "source.radians", "sourceY": "source.radius", "targetX": "target.radians", "targetY": "target.radius" } ] } ], "scales": [ { "name": "color", "type": "linear", "range": { "scheme": "viridis" }, "domain": { "data": "tree", "field": "depth" }, "zero": true } ], "marks": [ { "type": "path", "from": { "data": "links" }, "encode": { "update": { "x": { "signal": "originX" }, "y": { "signal": "originY" }, "path": { "field": "path" }, "stroke": { "value": "#ccc" } } } }, { "type": "symbol", "from": { "data": "tree" }, "encode": { "enter": { "size": { "value": 300 }, "stroke": { "value": "#fff" } }, "update": { "x": { "field": "x" }, "y": { "field": "y" }, "fill": { "scale": "color", "field": "depth" } } } }, { "type": "text", "from": { "data": "tree" }, "encode": { "enter": { "text": { "field": "name" }, "fontSize": { "value": 12 }, "baseline": { "value": "middle" }, "tooltip": { "signal": "{'Identifier': datum.tooltip, 'Affiliation': datum.affiliation, 'Contribution': datum.contributorType}" } }, "update": { "x": { "field": "x" }, "y": { "field": "y" }, "dx": { "signal": "(datum.leftside ? -1 : 1) * 12" }, "align": { "signal": "datum.leftside ? 'right' : 'left'" }, "opacity": { "signal": "labels ? 1 : 0" } } } } ] }, "text/plain": "\n\nIf you see this message, it means the renderer has not been properly enabled\nfor the frontend that you are using. For more information, see\nhttps://altair-viz.github.io/user_guide/troubleshooting.html\n" }, "metadata": {} } ], "source": [ "chart = alt.vega(json.loads(vega_template(json.dumps(nodes))))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A series of sliders and option are included to interact with the visulisation is displayed. One ca nremove the labels, rotate the nodes, zoom in/out, and adjust the layout." ] } ] }