{ "cells": [ { "cell_type": "raw", "id": "c68c6d49", "metadata": {}, "source": [ "---\n", "title: FHIR for Research Workshop - Bulk Data\n", "\n", "execute:\n", " cache: true\n", "---" ] }, { "cell_type": "markdown", "id": "a25355e4", "metadata": {}, "source": [ "## Introduction" ] }, { "cell_type": "markdown", "id": "e265b460", "metadata": {}, "source": [ "### Learning Objectives and Key Concepts\n", "\n", "**The goal of this workshop is to connect to the SMART Bulk Data Server and fetch a set of sample patient data.**\n", "\n", "In this exercise, you will:\n", "\n", " - Connect to an authorization server using a provided key, and retrieve an access token\n", " - Make a Bulk Data Export Request with that access token\n", " - Download the exported Bulk Data\n", " - Convert the downloaded data into DataFrames\n", "\n", "While libraries like [FHIR-PYrate](https://github.com/UMEssen/FHIR-PYrate) allow you to fetch data from a server and parse it directly into a DataFrame, these libraries generally do not support FHIR Bulk Data. This workshop will step through the process of building up a tool to fetch Bulk Data and convert it into DataFrames.\n", "\n", "This notebook is best experienced interactively. If the notebook already has output in it, you may clear that prior to starting via the menu: Cell -> All Output -> Clear." ] }, { "cell_type": "markdown", "id": "33b39e6f", "metadata": {}, "source": [ "### Setup\n", "\n", "If you are not using a JupyterHub instance with dependencies already installed, you will need to:\n", "\n", "1. Clone this repository\n", "2. Install dependencies with `pip install -r requirements.txt`\n", "3. Run `jupyter notebook workshops/fhir-bulk-data`\n", "\n", "This should open the Jupyter environment in your browser window. You should see `notebook.ipynb` listed in the interface. Open this notebook in Jupyter, and you should be able to run the code." ] }, { "cell_type": "markdown", "id": "4bb7145a", "metadata": {}, "source": [ "### Background\n", "\n", "The [Bulk Data Access standard](http://hl7.org/fhir/uv/bulkdata/) enables researchers to retrieve large volumes of data from a patient population in an EHR. The Bulk Data Access standard is part of the [SMART ecosystem](https://smarthealthit.org/), and SMART on FHIR can be used to authenticate and authorize applications that retrieve bulk data automatically\n", "\n", "Clients of FHIR Bulk Data servers use [SMART Backend Authorization](http://www.hl7.org/fhir/smart-app-launch/backend-services.html) to connect to the server. With SMART Backend Authorization, registered clients make a signed request to a token endpoint to receive a Bearer token, which they use for subsequent calls to the FHIR server.\n", "\n", "Client registration often happens manually as a separate one-time event. The SMART Backend Authorization specification does not define an API for registration.\n", "\n", "For this workshop, we connect to the [SMART Bulk Data Server](https://bulk-data.smarthealthit.org/) (). This is a developer tool provided by SMART Health IT to facilitate development with Bulk Data Access. This test server allows clients to \"register\" on the launch page by providing either a URL for a [JSON Web Key Set(JWKS)](https://auth0.com/docs/secure/tokens/json-web-tokens/json-web-key-sets) or a raw JWKS. In this case, \"registration\" is not stored on the server. Instead, the FHIR Server URL contains the \"registration\" information stored as state in the URL and clientID. Production servers will usually have a more standard registration process rather than taking this approach.\n", "\n", "For convenience, the SMART Bulk Data Server launch page allows users to generate a one-off JWKS to use for testing. For production usage, clients must generate their own certificates and JWKS and keep the private key private.\n", "In this workshop, we will use a JWKS generated by the launch page.\n", "\n", "**IMPORTANT**: this workshop is not meant to be a formal documentation of the specification, and largely skips error handling and stays on the \"happy path\" for brevity and readability. We strongly recommend reviewing the specifications and adding error handling before using any of this code in a production environment.\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "0d3b3991", "metadata": {}, "outputs": [], "source": [ "# The default style for rendering JSON parsed as Python dicts isn't the best.\n", "# Use this import and call `print(json)` when we want a cleaner view.\n", "\n", "from rich import print\n", "\n", "# Status bars for long-running cels\n", "from tqdm.notebook import trange, tqdm" ] }, { "cell_type": "markdown", "id": "579c4426", "metadata": {}, "source": [ "## Getting our Access Token\n", "\n", "The first step in obtaining data from a FHIR server that supports Bulk Data Access is to obtain an access token. That access token identifies and authorizes the client on requests made to the FHIR resource server.\n", "\n", "Obtaining an access token is itself a two-step process:\n", "1. Make a discovery request to the FHIR resource server to get the address of the authorization server.\n", "2. Post a token request, signed by the client's private key, to the authorization server\n", "\n", "To keep the focus of this workshop on the Bulk Data process rather than the details of generating keys, we will use a JWKS pre-generated by the SMART Bulk Data server launch page.\n", "\n", "For reference, the steps followed to generate the keys used here were:\n", "\n", "- Visit the SMART Bulk Data Server launch page\n", "- In the upper left, click the `JWKS` button for Authentication\n", "- Click the `Generate` button and choose `Generate RS384`\n", "- Choose `R4` for the FHIR Version\n", "- The associated text box now contains a JWKS with both a public and private key, and the Launch Configuration contains a FHIR Server URL and Client ID\n", "- Convert the private key from the JWKS to \"PEM\" format so it can be used by Python (this is not easy to do natively in Python, so we have done it with JavaScript out of band)\n", "\n", "Let's start by defining our credentials. **In practice, real credentials must always be stored and loaded securely**, but for simplicity in this workshop we will define them as local variables." ] }, { "cell_type": "code", "execution_count": 2, "id": "932a50fe", "metadata": {}, "outputs": [], "source": [ "client_id = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6InJlZ2lzdHJhdGlvbi10b2tlbiJ9.eyJqd2tzIjp7ImtleXMiOlt7Imt0eSI6IlJTQSIsImFsZyI6IlJTMzg0IiwibiI6IngzMDc2RTJNaUpMR3JPbXJXRjZXSWZ1RjFSZDBlTjBSdEhUSVRuMlNGVWhMYTFQWE5Ia0xBR2xSSmtJWk1QMUk5SEhxdTRERy02d2JraFMweU9GbEZhZE1iaGgzcHkySHoybDctRmg1M3Y3bmpwb3dxUGV2eEpqMlpEQU5BanFWeHRLOGdvMm1BZmZFSnJ2ZkVHbm5oUGkzdGE1U2U5UTBkS29la2hJRVRCaVJTa0ozN0pobEZGSDh3S2hFLXVwaXBQU3VycTBrQ0JkNlNaS3NOVHpHNzJmLVJoNENiREZWTVdfRm5zcTh5LWRJMTdMSDJZcHBBLWc0eGlUZnMwMGZOUG9FUEdoWFU2bHFKMHMwclp4Um9zYnVuV0NTYi1UaEtWV0RyeUFudE83S3dWN1BxVG1NMmVrVS1yenZFaWprVjZfUUlnVTJxRTd6X1k1N1l4aW8zUSIsImUiOiJBUUFCIiwia2V5X29wcyI6WyJ2ZXJpZnkiXSwiZXh0Ijp0cnVlLCJraWQiOiI0ZDc3OTJjZTQyMDU0ZDVkZjhkZDg1ZjhiNTI3ZGQ4OCJ9LHsia3R5IjoiUlNBIiwiYWxnIjoiUlMzODQiLCJuIjoieDMwNzZFMk1pSkxHck9tcldGNldJZnVGMVJkMGVOMFJ0SFRJVG4yU0ZVaExhMVBYTkhrTEFHbFJKa0laTVAxSTlISHF1NERHLTZ3YmtoUzB5T0ZsRmFkTWJoaDNweTJIejJsNy1GaDUzdjduanBvd3FQZXZ4SmoyWkRBTkFqcVZ4dEs4Z28ybUFmZkVKcnZmRUdubmhQaTN0YTVTZTlRMGRLb2VraElFVEJpUlNrSjM3SmhsRkZIOHdLaEUtdXBpcFBTdXJxMGtDQmQ2U1pLc05Uekc3MmYtUmg0Q2JERlZNV19GbnNxOHktZEkxN0xIMllwcEEtZzR4aVRmczAwZk5Qb0VQR2hYVTZscUowczByWnhSb3NidW5XQ1NiLVRoS1ZXRHJ5QW50TzdLd1Y3UHFUbU0yZWtVLXJ6dkVpamtWNl9RSWdVMnFFN3pfWTU3WXhpbzNRIiwiZSI6IkFRQUIiLCJkIjoiUnptQWRTMlMtb1FsS1VGNHF1R0Npdm1KekE1R3lJeHRzTmR0V1JEZVluamdiSjZQbksyRzd3dXJMSlMyOTlYSEFYZld6a0ZwU2h3bDc5OHl1UEk0ckNXQ1ZXQ29fLWh5ci14Q2xlWEpCWVJQV292VXljODlVMTBsdzVtZ1cyWmRhWkotT2NLblBkYWZreERLME1wdkhmdkxZN09zd1lkX2Z4UHFQRTd3ZDlaQU5XLUIyWmNURUVmd2taNWdlcmtDdnFHQ1lEUTdVcVJqR3k1dWRjTkRiQ01ITFdGaEZZMTVqMDVMMFpJV0RwUDY2cmN6UWZEdnduR0pIbWxJbnJMbTl5WkowUTNkVlpHSmo2Y2dMeWI4WHhkNHpWRjZGSy1NX2VKbnFzZFRveHRPMDNUOVotSWlrN1BfbFBheWRvMWRycXRZdUxmZXpvU1lnUGp0V0NnV0JRIiwicCI6IjZwNlV5aGZiQ0JjQlEzcGttMHZEb1lqSDZsc1FCeS1PTzlEYlpfZnFfSHpzZl96UWhENDdua0dZZngxbGVTUFlQU0ZSeDlRTUR3cTlvYWxjYmEwNmE3QTVmMUxQNVpaRnNvSDVCTElHTUcxNmhDbW1mTEdRMURkZ3pMb2s3Q3RldDRnNGhUTlpseFZOYV9uYVNmZGJSdmQycF8zNTM1RGpaOXoyMEpSNllDYyIsInEiOiIyYXNhQ0RCTmY3NTQ1ajdOcXI2TTZiUW8wVGZEWGNlb2FxcGVtNGhpNE1pYUtBOEcydVFvdXNTOGcyUTlZOFZiZmxjX3I2WmxPVjIxSmJhYW5WN253MDRxbVpqMG5Xdkk0a19yX2lKWTVuSDNUMHk0Y0lGV21tLUhPY1dzazJXWl9QQ1NSc1piOU1qOUs4UXh6b1h5WEo0ck9aLUw4OTNZbDZ5bVdKa2xqVnMiLCJkcCI6Ik9LeWI5b0Z5dUc2T01KV2xMZHBNWkgzZEJPQ0FhNnZ5S01MWDdUSjNBZ3pQT0UtQ3N4OHhXWll3MXl2cnNpcVZkcGJRNFh0NGVqMjI5eEVwTVpreHpvZWdMQUItRmRDSl80Zmo5bDFtbjFZaXpVQWVabXFpT0pFMEFlQkpRUDlzX3RxYUJKc1YzaWdZTHFnSk1lcmRrclAtWnJBMEp1d2g4cG51eVEzRXplcyIsImRxIjoib2I2R0FvMjZHUEcxcnduLUZDR3lYanMwbFhzRlhwdHRaNDJmN1owa05IcDhLc1kzeHRJQl9mOFJRZVZyeE1hem5TZENPTWpCc1NZVDVLbFRMUnVIeHRZX3k1RWdQQllLMlRpZ1dXQzJoTTh0QWEwMTVNd0hTWTBVZ19hQ3JhaXpDNFRNZlhFS2hkUVFaTVJPYW5PWVRBQndpRW9wV2hhQXl2eE5ROHJSWDc4IiwicWkiOiJLSjhJU0RKaHVyUmEyTVRHdG4zWjR3NU9ob3o2N29OcE10MG1TakxGUEt0QjFWbjRaZ3VkTUxfWTZ4V2lWTnBOR1hQa3hoMEJjRmNKakNKcC0yeUZLV0d4Si14M2JMWVllbkVUaGRFSGRRR0xuUUszMHlEdHFTY2NDUVY5U2xGc281NUdnUmxhODNaY2NBZTdBMXBWN2sxRGE4dFVFNkE4TXNlQ1ZXamRLbFUiLCJrZXlfb3BzIjpbInNpZ24iXSwiZXh0Ijp0cnVlLCJraWQiOiI0ZDc3OTJjZTQyMDU0ZDVkZjhkZDg1ZjhiNTI3ZGQ4OCJ9XX0sImFjY2Vzc1Rva2Vuc0V4cGlyZUluIjoxNSwiaWF0IjoxNjg2NjUyNzM4fQ.j1urst068-21CxiH0Nqml7XoE9v6hWJ_vfqAK4W22vg'\n", "\n", "\n", "# Don't worry! This is not anybody's real private key. It was generated specifically and only for this exercise.\n", "private_key = \"\"\"-----BEGIN RSA PRIVATE KEY-----\n", "MIIEowIBAAKCAQEAx3076E2MiJLGrOmrWF6WIfuF1Rd0eN0RtHTITn2SFUhLa1PX\n", "NHkLAGlRJkIZMP1I9HHqu4DG+6wbkhS0yOFlFadMbhh3py2Hz2l7+Fh53v7njpow\n", "qPevxJj2ZDANAjqVxtK8go2mAffEJrvfEGnnhPi3ta5Se9Q0dKoekhIETBiRSkJ3\n", "7JhlFFH8wKhE+upipPSurq0kCBd6SZKsNTzG72f+Rh4CbDFVMW/Fnsq8y+dI17LH\n", "2YppA+g4xiTfs00fNPoEPGhXU6lqJ0s0rZxRosbunWCSb+ThKVWDryAntO7KwV7P\n", "qTmM2ekU+rzvEijkV6/QIgU2qE7z/Y57Yxio3QIDAQABAoIBAEc5gHUtkvqEJSlB\n", "eKrhgor5icwORsiMbbDXbVkQ3mJ44Gyej5ythu8LqyyUtvfVxwF31s5BaUocJe/f\n", "MrjyOKwlglVgqP/ocq/sQpXlyQWET1qL1MnPPVNdJcOZoFtmXWmSfjnCpz3Wn5MQ\n", "ytDKbx37y2OzrMGHf38T6jxO8HfWQDVvgdmXExBH8JGeYHq5Ar6hgmA0O1KkYxsu\n", "bnXDQ2wjBy1hYRWNeY9OS9GSFg6T+uq3M0Hw78JxiR5pSJ6y5vcmSdEN3VWRiY+n\n", "IC8m/F8XeM1RehSvjP3iZ6rHU6MbTtN0/WfiIpOz/5T2snaNXa6rWLi33s6EmID4\n", "7VgoFgUCgYEA6p6UyhfbCBcBQ3pkm0vDoYjH6lsQBy+OO9DbZ/fq/Hzsf/zQhD47\n", "nkGYfx1leSPYPSFRx9QMDwq9oalcba06a7A5f1LP5ZZFsoH5BLIGMG16hCmmfLGQ\n", "1DdgzLok7Ctet4g4hTNZlxVNa/naSfdbRvd2p/3535DjZ9z20JR6YCcCgYEA2asa\n", "CDBNf7545j7Nqr6M6bQo0TfDXceoaqpem4hi4MiaKA8G2uQousS8g2Q9Y8Vbflc/\n", "r6ZlOV21JbaanV7nw04qmZj0nWvI4k/r/iJY5nH3T0y4cIFWmm+HOcWsk2WZ/PCS\n", "RsZb9Mj9K8QxzoXyXJ4rOZ+L893Yl6ymWJkljVsCgYA4rJv2gXK4bo4wlaUt2kxk\n", "fd0E4IBrq/IowtftMncCDM84T4KzHzFZljDXK+uyKpV2ltDhe3h6Pbb3ESkxmTHO\n", "h6AsAH4V0In/h+P2XWafViLNQB5maqI4kTQB4ElA/2z+2poEmxXeKBguqAkx6t2S\n", "s/5msDQm7CHyme7JDcTN6wKBgQChvoYCjboY8bWvCf4UIbJeOzSVewVem21njZ/t\n", "nSQ0enwqxjfG0gH9/xFB5WvExrOdJ0I4yMGxJhPkqVMtG4fG1j/LkSA8FgrZOKBZ\n", "YLaEzy0BrTXkzAdJjRSD9oKtqLMLhMx9cQqF1BBkxE5qc5hMAHCISilaFoDK/E1D\n", "ytFfvwKBgCifCEgyYbq0WtjExrZ92eMOToaM+u6DaTLdJkoyxTyrQdVZ+GYLnTC/\n", "2OsVolTaTRlz5MYdAXBXCYwiaftshSlhsSfsd2y2GHpxE4XRB3UBi50Ct9Mg7akn\n", "HAkFfUpRbKOeRoEZWvN2XHAHuwNaVe5NQ2vLVBOgPDLHglVo3SpV\n", "-----END RSA PRIVATE KEY-----\"\"\"\n", "\n", "# note key id is the \"kid\" field from the JWKS -- it's same for both values of `keys`\n", "key_id = \"4d7792ce42054d5df8dd85f8b527dd88\"\n", "\n", "server_url = 'https://bulk-data.smarthealthit.org/eyJlcnIiOiIiLCJwYWdlIjoxMDAwMCwiZHVyIjoxMCwidGx0IjoxNSwibSI6MSwic3R1Ijo0LCJkZWwiOjB9/fhir'\n" ] }, { "cell_type": "markdown", "id": "100781c4", "metadata": {}, "source": [ "We will use the [Requests](https://requests.readthedocs.io/en/latest/) library for making all HTTP requests, and use a `Session`, in case we need to persist common settings such as proxy or SSL configuration." ] }, { "cell_type": "code", "execution_count": 3, "id": "4d75e66a", "metadata": {}, "outputs": [], "source": [ "import requests\n", "\n", "session = requests.Session()\n", "\n", "# Optional: Turn off SSL verification. Useful when dealing with a corporate proxy with self-signed certificates.\n", "from urllib3.exceptions import InsecureRequestWarning\n", "requests.packages.urllib3.disable_warnings(category=InsecureRequestWarning)\n", "session.verify = False\n" ] }, { "cell_type": "markdown", "id": "f7c419f5", "metadata": {}, "source": [ "Let's start by confirming we can hit the server via the `/metadata` endpoint. When connecting to a server for the first time it is generally a good idea to review the metadata to see what the server supports, and that it matches your expectations. In this case, expect to see the name \"SMART Sample Bulk Data Server\", and references to \"export\" operations." ] }, { "cell_type": "code", "execution_count": 4, "id": "00ba67df", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    'resourceType': 'CapabilityStatement',\n",
       "    'status': 'active',\n",
       "    'date': '2023-06-13T01:26:34+00:00',\n",
       "    'publisher': \"Boston Children's Hospital\",\n",
       "    'kind': 'instance',\n",
       "    'instantiates': ['http://hl7.org/fhir/uv/bulkdata/CapabilityStatement/bulk-data'],\n",
       "    'software': {'name': 'SMART Sample Bulk Data Server', 'version': '2.1.1'},\n",
       "    'implementation': {'description': 'SMART Sample Bulk Data Server'},\n",
       "    'fhirVersion': '4.0.1',\n",
       "    'acceptUnknown': 'extensions',\n",
       "    'format': ['json'],\n",
       "    'rest': [\n",
       "        {\n",
       "            'mode': 'server',\n",
       "            'security': {\n",
       "                'extension': [\n",
       "                    {\n",
       "                        'url': 'http://fhir-registry.smarthealthit.org/StructureDefinition/oauth-uris',\n",
       "                        'extension': [\n",
       "                            {'url': 'token', 'valueUri': 'https://bulk-data.smarthealthit.org/auth/token'},\n",
       "                            {'url': 'register', 'valueUri': 'https://bulk-data.smarthealthit.org/auth/register'}\n",
       "                        ]\n",
       "                    }\n",
       "                ],\n",
       "                'service': [\n",
       "                    {\n",
       "                        'coding': [\n",
       "                            {\n",
       "                                'system': 'http://hl7.org/fhir/restful-security-service',\n",
       "                                'code': 'SMART-on-FHIR',\n",
       "                                'display': 'SMART-on-FHIR'\n",
       "                            }\n",
       "                        ],\n",
       "                        'text': 'OAuth2 using SMART-on-FHIR profile (see http://docs.smarthealthit.org)'\n",
       "                    }\n",
       "                ]\n",
       "            },\n",
       "            'resource': [\n",
       "                {\n",
       "                    'type': 'Patient',\n",
       "                    'operation': [\n",
       "                        {\n",
       "                            'extension': [\n",
       "                                {\n",
       "                                    'url': \n",
       "'http://hl7.org/fhir/StructureDefinition/capabilitystatement-expectation',\n",
       "                                    'valueCode': 'SHOULD'\n",
       "                                }\n",
       "                            ],\n",
       "                            'name': 'patient-export',\n",
       "                            'definition': 'http://hl7.org/fhir/uv/bulkdata/OperationDefinition/patient-export'\n",
       "                        }\n",
       "                    ]\n",
       "                },\n",
       "                {\n",
       "                    'type': 'Group',\n",
       "                    'operation': [\n",
       "                        {\n",
       "                            'extension': [\n",
       "                                {\n",
       "                                    'url': \n",
       "'http://hl7.org/fhir/StructureDefinition/capabilitystatement-expectation',\n",
       "                                    'valueCode': 'SHOULD'\n",
       "                                }\n",
       "                            ],\n",
       "                            'name': 'group-export',\n",
       "                            'definition': 'http://hl7.org/fhir/uv/bulkdata/OperationDefinition/group-export'\n",
       "                        }\n",
       "                    ]\n",
       "                },\n",
       "                {\n",
       "                    'type': 'OperationDefinition',\n",
       "                    'profile': {'reference': 'http://hl7.org/fhir/Profile/OperationDefinition'},\n",
       "                    'interaction': [{'code': 'read'}],\n",
       "                    'searchParam': []\n",
       "                }\n",
       "            ],\n",
       "            'operation': [\n",
       "                {'name': 'get-resource-counts', 'definition': 'OperationDefinition/-s-get-resource-counts'},\n",
       "                {\n",
       "                    'extension': [\n",
       "                        {\n",
       "                            'url': 'http://hl7.org/fhir/StructureDefinition/capabilitystatement-expectation',\n",
       "                            'valueCode': 'SHOULD'\n",
       "                        }\n",
       "                    ],\n",
       "                    'name': 'export',\n",
       "                    'definition': 'http://hl7.org/fhir/uv/bulkdata/OperationDefinition/export'\n",
       "                }\n",
       "            ]\n",
       "        }\n",
       "    ]\n",
       "}\n",
       "
\n" ], "text/plain": [ "\u001B[1m{\u001B[0m\n", " \u001B[32m'resourceType'\u001B[0m: \u001B[32m'CapabilityStatement'\u001B[0m,\n", " \u001B[32m'status'\u001B[0m: \u001B[32m'active'\u001B[0m,\n", " \u001B[32m'date'\u001B[0m: \u001B[32m'2023-06-13T01:26:34+00:00'\u001B[0m,\n", " \u001B[32m'publisher'\u001B[0m: \u001B[32m\"Boston Children's Hospital\"\u001B[0m,\n", " \u001B[32m'kind'\u001B[0m: \u001B[32m'instance'\u001B[0m,\n", " \u001B[32m'instantiates'\u001B[0m: \u001B[1m[\u001B[0m\u001B[32m'http://hl7.org/fhir/uv/bulkdata/CapabilityStatement/bulk-data'\u001B[0m\u001B[1m]\u001B[0m,\n", " \u001B[32m'software'\u001B[0m: \u001B[1m{\u001B[0m\u001B[32m'name'\u001B[0m: \u001B[32m'SMART Sample Bulk Data Server'\u001B[0m, \u001B[32m'version'\u001B[0m: \u001B[32m'2.1.1'\u001B[0m\u001B[1m}\u001B[0m,\n", " \u001B[32m'implementation'\u001B[0m: \u001B[1m{\u001B[0m\u001B[32m'description'\u001B[0m: \u001B[32m'SMART Sample Bulk Data Server'\u001B[0m\u001B[1m}\u001B[0m,\n", " \u001B[32m'fhirVersion'\u001B[0m: \u001B[32m'4.0.1'\u001B[0m,\n", " \u001B[32m'acceptUnknown'\u001B[0m: \u001B[32m'extensions'\u001B[0m,\n", " \u001B[32m'format'\u001B[0m: \u001B[1m[\u001B[0m\u001B[32m'json'\u001B[0m\u001B[1m]\u001B[0m,\n", " \u001B[32m'rest'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'mode'\u001B[0m: \u001B[32m'server'\u001B[0m,\n", " \u001B[32m'security'\u001B[0m: \u001B[1m{\u001B[0m\n", " \u001B[32m'extension'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'url'\u001B[0m: \u001B[32m'http://fhir-registry.smarthealthit.org/StructureDefinition/oauth-uris'\u001B[0m,\n", " \u001B[32m'extension'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\u001B[32m'url'\u001B[0m: \u001B[32m'token'\u001B[0m, \u001B[32m'valueUri'\u001B[0m: \u001B[32m'https://bulk-data.smarthealthit.org/auth/token'\u001B[0m\u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\u001B[32m'url'\u001B[0m: \u001B[32m'register'\u001B[0m, \u001B[32m'valueUri'\u001B[0m: \u001B[32m'https://bulk-data.smarthealthit.org/auth/register'\u001B[0m\u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'service'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'coding'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'system'\u001B[0m: \u001B[32m'http://hl7.org/fhir/restful-security-service'\u001B[0m,\n", " \u001B[32m'code'\u001B[0m: \u001B[32m'SMART-on-FHIR'\u001B[0m,\n", " \u001B[32m'display'\u001B[0m: \u001B[32m'SMART-on-FHIR'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'text'\u001B[0m: \u001B[32m'OAuth2 using SMART-on-FHIR profile \u001B[0m\u001B[32m(\u001B[0m\u001B[32msee http://docs.smarthealthit.org\u001B[0m\u001B[32m)\u001B[0m\u001B[32m'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[32m'resource'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'type'\u001B[0m: \u001B[32m'Patient'\u001B[0m,\n", " \u001B[32m'operation'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'extension'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'url'\u001B[0m: \n", "\u001B[32m'http://hl7.org/fhir/StructureDefinition/capabilitystatement-expectation'\u001B[0m,\n", " \u001B[32m'valueCode'\u001B[0m: \u001B[32m'SHOULD'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'name'\u001B[0m: \u001B[32m'patient-export'\u001B[0m,\n", " \u001B[32m'definition'\u001B[0m: \u001B[32m'http://hl7.org/fhir/uv/bulkdata/OperationDefinition/patient-export'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'type'\u001B[0m: \u001B[32m'Group'\u001B[0m,\n", " \u001B[32m'operation'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'extension'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'url'\u001B[0m: \n", "\u001B[32m'http://hl7.org/fhir/StructureDefinition/capabilitystatement-expectation'\u001B[0m,\n", " \u001B[32m'valueCode'\u001B[0m: \u001B[32m'SHOULD'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'name'\u001B[0m: \u001B[32m'group-export'\u001B[0m,\n", " \u001B[32m'definition'\u001B[0m: \u001B[32m'http://hl7.org/fhir/uv/bulkdata/OperationDefinition/group-export'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'type'\u001B[0m: \u001B[32m'OperationDefinition'\u001B[0m,\n", " \u001B[32m'profile'\u001B[0m: \u001B[1m{\u001B[0m\u001B[32m'reference'\u001B[0m: \u001B[32m'http://hl7.org/fhir/Profile/OperationDefinition'\u001B[0m\u001B[1m}\u001B[0m,\n", " \u001B[32m'interaction'\u001B[0m: \u001B[1m[\u001B[0m\u001B[1m{\u001B[0m\u001B[32m'code'\u001B[0m: \u001B[32m'read'\u001B[0m\u001B[1m}\u001B[0m\u001B[1m]\u001B[0m,\n", " \u001B[32m'searchParam'\u001B[0m: \u001B[1m[\u001B[0m\u001B[1m]\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'operation'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\u001B[32m'name'\u001B[0m: \u001B[32m'get-resource-counts'\u001B[0m, \u001B[32m'definition'\u001B[0m: \u001B[32m'OperationDefinition/-s-get-resource-counts'\u001B[0m\u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'extension'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'url'\u001B[0m: \u001B[32m'http://hl7.org/fhir/StructureDefinition/capabilitystatement-expectation'\u001B[0m,\n", " \u001B[32m'valueCode'\u001B[0m: \u001B[32m'SHOULD'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'name'\u001B[0m: \u001B[32m'export'\u001B[0m,\n", " \u001B[32m'definition'\u001B[0m: \u001B[32m'http://hl7.org/fhir/uv/bulkdata/OperationDefinition/export'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m\n", "\u001B[1m}\u001B[0m\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "r = session.get(f'{server_url}/metadata')\n", "metadata = r.json()\n", "\n", "print(metadata)" ] }, { "cell_type": "markdown", "id": "b163cf8e", "metadata": {}, "source": [ "The SMART Backend Authorization specification defines that the token endpoint will be published as part of the FHIR resource server's SMART metadata, at `.well-known/smart-configuration`. Let's fetch that endpoint and review the contents." ] }, { "cell_type": "code", "execution_count": 5, "id": "d39d1a9b", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    'token_endpoint': 'https://bulk-data.smarthealthit.org/auth/token',\n",
       "    'registration_endpoint': 'https://bulk-data.smarthealthit.org/auth/register',\n",
       "    'token_endpoint_auth_methods_supported': ['private_key_jwt'],\n",
       "    'token_endpoint_auth_signing_alg_values_supported': [\n",
       "        'HS256',\n",
       "        'HS384',\n",
       "        'HS512',\n",
       "        'RS256',\n",
       "        'RS384',\n",
       "        'RS512',\n",
       "        'ES256',\n",
       "        'ES384',\n",
       "        'ES512',\n",
       "        'PS256',\n",
       "        'PS384',\n",
       "        'PS512'\n",
       "    ],\n",
       "    'scopes_supported': [\n",
       "        'system/*.rs',\n",
       "        'system/Patient.rs',\n",
       "        'system/Encounter.rs',\n",
       "        'system/Condition.rs',\n",
       "        'system/Claim.rs',\n",
       "        'system/ExplanationOfBenefit.rs',\n",
       "        'system/Observation.rs',\n",
       "        'system/Immunization.rs',\n",
       "        'system/DiagnosticReport.rs',\n",
       "        'system/Procedure.rs',\n",
       "        'system/CareTeam.rs',\n",
       "        'system/CarePlan.rs',\n",
       "        'system/MedicationRequest.rs',\n",
       "        'system/AllergyIntolerance.rs',\n",
       "        'system/Device.rs',\n",
       "        'system/ImagingStudy.rs',\n",
       "        'system/Organization.rs',\n",
       "        'system/Practitioner.rs',\n",
       "        'system/DocumentReference.rs',\n",
       "        'system/Group.rs',\n",
       "        'system/*.read',\n",
       "        'system/Patient.read',\n",
       "        'system/Encounter.read',\n",
       "        'system/Condition.read',\n",
       "        'system/Claim.read',\n",
       "        'system/ExplanationOfBenefit.read',\n",
       "        'system/Observation.read',\n",
       "        'system/Immunization.read',\n",
       "        'system/DiagnosticReport.read',\n",
       "        'system/Procedure.read',\n",
       "        'system/CareTeam.read',\n",
       "        'system/CarePlan.read',\n",
       "        'system/MedicationRequest.read',\n",
       "        'system/AllergyIntolerance.read',\n",
       "        'system/Device.read',\n",
       "        'system/ImagingStudy.read',\n",
       "        'system/Organization.read',\n",
       "        'system/Practitioner.read',\n",
       "        'system/DocumentReference.read',\n",
       "        'system/Group.read'\n",
       "    ],\n",
       "    'capabilities': ['permission-v2', 'permission-v1', 'client-confidential-asymmetric']\n",
       "}\n",
       "
\n" ], "text/plain": [ "\u001B[1m{\u001B[0m\n", " \u001B[32m'token_endpoint'\u001B[0m: \u001B[32m'https://bulk-data.smarthealthit.org/auth/token'\u001B[0m,\n", " \u001B[32m'registration_endpoint'\u001B[0m: \u001B[32m'https://bulk-data.smarthealthit.org/auth/register'\u001B[0m,\n", " \u001B[32m'token_endpoint_auth_methods_supported'\u001B[0m: \u001B[1m[\u001B[0m\u001B[32m'private_key_jwt'\u001B[0m\u001B[1m]\u001B[0m,\n", " \u001B[32m'token_endpoint_auth_signing_alg_values_supported'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[32m'HS256'\u001B[0m,\n", " \u001B[32m'HS384'\u001B[0m,\n", " \u001B[32m'HS512'\u001B[0m,\n", " \u001B[32m'RS256'\u001B[0m,\n", " \u001B[32m'RS384'\u001B[0m,\n", " \u001B[32m'RS512'\u001B[0m,\n", " \u001B[32m'ES256'\u001B[0m,\n", " \u001B[32m'ES384'\u001B[0m,\n", " \u001B[32m'ES512'\u001B[0m,\n", " \u001B[32m'PS256'\u001B[0m,\n", " \u001B[32m'PS384'\u001B[0m,\n", " \u001B[32m'PS512'\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'scopes_supported'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[32m'system/*.rs'\u001B[0m,\n", " \u001B[32m'system/Patient.rs'\u001B[0m,\n", " \u001B[32m'system/Encounter.rs'\u001B[0m,\n", " \u001B[32m'system/Condition.rs'\u001B[0m,\n", " \u001B[32m'system/Claim.rs'\u001B[0m,\n", " \u001B[32m'system/ExplanationOfBenefit.rs'\u001B[0m,\n", " \u001B[32m'system/Observation.rs'\u001B[0m,\n", " \u001B[32m'system/Immunization.rs'\u001B[0m,\n", " \u001B[32m'system/DiagnosticReport.rs'\u001B[0m,\n", " \u001B[32m'system/Procedure.rs'\u001B[0m,\n", " \u001B[32m'system/CareTeam.rs'\u001B[0m,\n", " \u001B[32m'system/CarePlan.rs'\u001B[0m,\n", " \u001B[32m'system/MedicationRequest.rs'\u001B[0m,\n", " \u001B[32m'system/AllergyIntolerance.rs'\u001B[0m,\n", " \u001B[32m'system/Device.rs'\u001B[0m,\n", " \u001B[32m'system/ImagingStudy.rs'\u001B[0m,\n", " \u001B[32m'system/Organization.rs'\u001B[0m,\n", " \u001B[32m'system/Practitioner.rs'\u001B[0m,\n", " \u001B[32m'system/DocumentReference.rs'\u001B[0m,\n", " \u001B[32m'system/Group.rs'\u001B[0m,\n", " \u001B[32m'system/*.read'\u001B[0m,\n", " \u001B[32m'system/Patient.read'\u001B[0m,\n", " \u001B[32m'system/Encounter.read'\u001B[0m,\n", " \u001B[32m'system/Condition.read'\u001B[0m,\n", " \u001B[32m'system/Claim.read'\u001B[0m,\n", " \u001B[32m'system/ExplanationOfBenefit.read'\u001B[0m,\n", " \u001B[32m'system/Observation.read'\u001B[0m,\n", " \u001B[32m'system/Immunization.read'\u001B[0m,\n", " \u001B[32m'system/DiagnosticReport.read'\u001B[0m,\n", " \u001B[32m'system/Procedure.read'\u001B[0m,\n", " \u001B[32m'system/CareTeam.read'\u001B[0m,\n", " \u001B[32m'system/CarePlan.read'\u001B[0m,\n", " \u001B[32m'system/MedicationRequest.read'\u001B[0m,\n", " \u001B[32m'system/AllergyIntolerance.read'\u001B[0m,\n", " \u001B[32m'system/Device.read'\u001B[0m,\n", " \u001B[32m'system/ImagingStudy.read'\u001B[0m,\n", " \u001B[32m'system/Organization.read'\u001B[0m,\n", " \u001B[32m'system/Practitioner.read'\u001B[0m,\n", " \u001B[32m'system/DocumentReference.read'\u001B[0m,\n", " \u001B[32m'system/Group.read'\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'capabilities'\u001B[0m: \u001B[1m[\u001B[0m\u001B[32m'permission-v2'\u001B[0m, \u001B[32m'permission-v1'\u001B[0m, \u001B[32m'client-confidential-asymmetric'\u001B[0m\u001B[1m]\u001B[0m\n", "\u001B[1m}\u001B[0m\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "r = session.get(f'{server_url}/.well-known/smart-configuration')\n", "smart_config = r.json()\n", "\n", "print(smart_config)" ] }, { "cell_type": "markdown", "id": "72e61789", "metadata": {}, "source": [ "We care most about the `token_endpoint` field, which we need to request our JWT. For more information about the other fields, see [here](http://www.hl7.org/fhir/smart-app-launch/scopes-and-launch-context.html)." ] }, { "cell_type": "code", "execution_count": 6, "id": "c0dcf45e", "metadata": {}, "outputs": [], "source": [ "token_endpoint = smart_config['token_endpoint']" ] }, { "cell_type": "markdown", "id": "c16cb0df", "metadata": {}, "source": [ "Now we have our token endpoint, so we can make a request to it to get a token.\n", "The request follows the [OAuth 2.0 \"Client Credentials\" flow](https://datatracker.ietf.org/doc/html/rfc6749#section-4.4), using a [JSON Web Token (JWT) assertion](https://datatracker.ietf.org/doc/html/rfc7523) containing our client ID and signed with our private key.\n", "\n", "\n", "📘 [Read more about the access token request specification](http://www.hl7.org/fhir/smart-app-launch/backend-services.html#obtain-access-token)" ] }, { "cell_type": "code", "execution_count": 7, "id": "e08062b3", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'token_type': 'bearer',\n", " 'scope': 'system/*.read',\n", " 'expires_in': 300,\n", " 'access_token': 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0b2tlbl90eXBlIjoiYmVhcmVyIiwic2NvcGUiOiJzeXN0ZW0vKi5yZWFkIiwiZXhwaXJlc19pbiI6MzAwLCJpYXQiOjE2ODY2NTQzNTIsImV4cCI6MTY4NjY1NDY1Mn0.6VcZfI7YBkrGV7IoIBKmQo2usjrpCkIgmJHx8jFir3g'}" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a JWT client assertion as follows:\n", "import jwt\n", "import datetime\n", "\n", "assertion = jwt.encode({\n", " 'iss': client_id, # \"iss\" == \"issuer\", the client that created this JWT\n", " 'sub': client_id, # \"sub\" == \"subject\", the client that will use the access token\n", " 'aud': token_endpoint, # \"aud\" == \"audience\", the receiver of this request\n", " 'exp': int((datetime.datetime.now() + datetime.timedelta(minutes=5)).timestamp())\n", " },\n", " private_key, # signed with the private key\n", " algorithm='RS384', # algorithm for the key\n", " headers={\"kid\": key_id}) # kid is required for smart bulk data server\n", "\n", "\n", "# And then POST it to the token endpont\n", "r = session.post(token_endpoint, data={\n", " 'scope': 'system/*.read',\n", " 'grant_type': 'client_credentials',\n", " 'client_assertion_type': 'urn:ietf:params:oauth:client-assertion-type:jwt-bearer',\n", " 'client_assertion': assertion\n", "})\n", "\n", "token_response = r.json()\n", "\n", "# And inspect the response:\n", "token_response" ] }, { "cell_type": "markdown", "id": "26be8546", "metadata": {}, "source": [ "Two important fields we need to keep track of are the token itself, and the expire time.\n", "Tokens are only valid for a certain amount of time, and once they expire we will need to fetch a new one via the same process as above.\n", "`expires_in` is in seconds from the current time, so we'll add that to the current time to get a timestamp we can compare against.\n", "\n", "Note that for this example we requested and received `'scope': 'system/*.read'` which allows access to all resource types. In practice, requesting access to all resource types is generally not recommended, and servers do not always support asking for `*` scopes. Generally it is recommended to request only the minimal level of access necessary." ] }, { "cell_type": "code", "execution_count": 8, "id": "4f06a599", "metadata": {}, "outputs": [], "source": [ "token = token_response['access_token']\n", "expire_time = datetime.datetime.now() + datetime.timedelta(seconds=token_response['expires_in'])" ] }, { "cell_type": "markdown", "id": "decd8b4d", "metadata": {}, "source": [ "To make this easier for ourselves, let's package this up into a `get_token()` function that we can call anytime we need to use a token. If the current token is still valid, use that, or if it has expired, fetch a new one. The logic is exactly the same as the previous steps we just ran:" ] }, { "cell_type": "code", "execution_count": 9, "id": "59b5fc2c", "metadata": {}, "outputs": [], "source": [ "def get_token():\n", " global token, expire_time\n", " if datetime.datetime.now() < expire_time:\n", " # the existing token is still valid so return it\n", " return token\n", "\n", " assertion = jwt.encode({\n", " 'iss': client_id,\n", " 'sub': client_id,\n", " 'aud': token_endpoint,\n", " 'exp': int((datetime.datetime.now() + datetime.timedelta(minutes=5)).timestamp())\n", " }, private_key, algorithm='RS384',\n", " headers={\"kid\": key_id})\n", "\n", " r = session.post(token_endpoint, data={\n", " 'scope': 'system/*.read',\n", " 'grant_type': 'client_credentials',\n", " 'client_assertion_type': 'urn:ietf:params:oauth:client-assertion-type:jwt-bearer',\n", " 'client_assertion': assertion\n", " })\n", "\n", " token_response = r.json()\n", " token = token_response['access_token']\n", " expire_time = datetime.datetime.now() + datetime.timedelta(seconds=token_response['expires_in'])\n", "\n", " return token\n" ] }, { "cell_type": "markdown", "id": "176cd056", "metadata": {}, "source": [ "## Starting, Checking, and Downloading the Export\n", "\n", "Now that we have an access token, the next step in using Bulk Data is to request the export of data, via a \"kick-off request\". This is an asynchronous request -- once the request is accepted, instead of returning the results directly, the server response will point to a URL where the client can check the status.\n", "\n", "There are three levels of export:\n", " - **Patient**, to obtain resources related to all Patients\n", " - **Group**, to obtain resources associated with a particular [Group](https://www.hl7.org/fhir/group.html)\n", " - **System**, to obtain all resources, whether or not they are associated with a patient\n", "\n", "For this exercise we will initially only request Patient-level data, but the general process for Groups and System-level data is exactly the same - there is just a different endpoint to hit, and a different set of data will be returned.\n", "\n", "There are also a number of parameters that may be set, but to keep things simple we will only use the `_type` parameter, to request only `Patient` and `Condition` resource types.\n", "\n", "📘 [Read more about the Bulk Data Kick-off Request](http://hl7.org/fhir/uv/bulkdata/export.html#bulk-data-kick-off-request)\n", "\n", "Let's make the export request and inspect the response headers. For \"Patient\" level data, the URL we want to hit is `{server}/Patient/$export`. Our token is used in the \"Authorization\" header in the format `\"Bearer {token}\"`." ] }, { "cell_type": "code", "execution_count": 10, "id": "c12edfda", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{'Server': 'Cowboy', 'Connection': 'keep-alive', 'X-Powered-By': 'Express', 'Content-Location': \n",
       "'https://bulk-data.smarthealthit.org/fhir/bulkstatus/55e2770a4b9c3f861f002634bd44ac62', 'Content-Type': \n",
       "'application/json; charset=utf-8', 'Content-Length': '644', 'Etag': 'W/\"284-G8JHR+JPFTg+y5JhfrRbOzc4ZMI\"', 'Date': \n",
       "'Tue, 13 Jun 2023 11:05:52 GMT', 'Via': '1.1 vegur'}\n",
       "
\n" ], "text/plain": [ "\u001B[1m{\u001B[0m\u001B[32m'Server'\u001B[0m: \u001B[32m'Cowboy'\u001B[0m, \u001B[32m'Connection'\u001B[0m: \u001B[32m'keep-alive'\u001B[0m, \u001B[32m'X-Powered-By'\u001B[0m: \u001B[32m'Express'\u001B[0m, \u001B[32m'Content-Location'\u001B[0m: \n", "\u001B[32m'https://bulk-data.smarthealthit.org/fhir/bulkstatus/55e2770a4b9c3f861f002634bd44ac62'\u001B[0m, \u001B[32m'Content-Type'\u001B[0m: \n", "\u001B[32m'application/json; \u001B[0m\u001B[32mcharset\u001B[0m\u001B[32m=\u001B[0m\u001B[32mutf\u001B[0m\u001B[32m-8'\u001B[0m, \u001B[32m'Content-Length'\u001B[0m: \u001B[32m'644'\u001B[0m, \u001B[32m'Etag'\u001B[0m: \u001B[32m'W/\"284-G8JHR+JPFTg+y5JhfrRbOzc4ZMI\"'\u001B[0m, \u001B[32m'Date'\u001B[0m: \n", "\u001B[32m'Tue, 13 Jun 2023 11:05:52 GMT'\u001B[0m, \u001B[32m'Via'\u001B[0m: \u001B[32m'1.1 vegur'\u001B[0m\u001B[1m}\u001B[0m\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "r = session.get(f'{server_url}/Patient/$export?_type=Patient,Condition',\n", " headers={'Authorization': f'Bearer {get_token()}',\n", " 'Accept': 'application/fhir+json',\n", " 'Prefer': 'respond-async'})\n", "\n", "print(r.headers)\n" ] }, { "cell_type": "markdown", "id": "4e64bcab", "metadata": {}, "source": [ "We see the status URL in the `Content-Location` header, so let's save that into a variable." ] }, { "cell_type": "code", "execution_count": 11, "id": "a7bdf84f", "metadata": {}, "outputs": [], "source": [ "check_url = r.headers['Content-Location']" ] }, { "cell_type": "markdown", "id": "b3d0a8c4", "metadata": {}, "source": [ "We can now check the status by getting that URL, and the HTTP status code of the response will indicate the exort status.\n", " - Code **200** means the export is complete, and the response body will indicate the location\n", " - Code **202** means the export is still in progress\n", " - Codes in the range **4xx-5xx** indicate an error has occurred. 4xx codes generally indicate an error in the request, and 5xx codes generally indicate a server error.\n", "\n", "Note that in production environments it is recommended to check the status as infrequently as possible, to minimize the load on the server. In this case we expect the export to complete in just a few seconds so the impact of checking every two seconds is minimal. The server will also include a \"Retry-After\" header which will give us a hint on how long to wait before trying again.\n", "We'll check that status in a loop, and break out of the loop when we get a complete or error response. We'll print status each time through the loop, and the response body when the export is complete." ] }, { "cell_type": "code", "execution_count": 12, "id": "50ab1b5b", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{'Server': 'Cowboy', 'Connection': 'keep-alive', 'X-Powered-By': 'Express', 'X-Progress': '1% complete, currenly \n",
       "processing Patient resources', 'Retry-After': '2', 'Date': 'Tue, 13 Jun 2023 11:05:52 GMT', 'Content-Length': '0', \n",
       "'Via': '1.1 vegur'}\n",
       "
\n" ], "text/plain": [ "\u001B[1m{\u001B[0m\u001B[32m'Server'\u001B[0m: \u001B[32m'Cowboy'\u001B[0m, \u001B[32m'Connection'\u001B[0m: \u001B[32m'keep-alive'\u001B[0m, \u001B[32m'X-Powered-By'\u001B[0m: \u001B[32m'Express'\u001B[0m, \u001B[32m'X-Progress'\u001B[0m: \u001B[32m'1% complete, currenly \u001B[0m\n", "\u001B[32mprocessing Patient resources'\u001B[0m, \u001B[32m'Retry-After'\u001B[0m: \u001B[32m'2'\u001B[0m, \u001B[32m'Date'\u001B[0m: \u001B[32m'Tue, 13 Jun 2023 11:05:52 GMT'\u001B[0m, \u001B[32m'Content-Length'\u001B[0m: \u001B[32m'0'\u001B[0m, \n", "\u001B[32m'Via'\u001B[0m: \u001B[32m'1.1 vegur'\u001B[0m\u001B[1m}\u001B[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
Sleeping 2 seconds before retrying\n",
       "
\n" ], "text/plain": [ "Sleeping \u001B[1;36m2\u001B[0m seconds before retrying\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
{'Server': 'Cowboy', 'Connection': 'keep-alive', 'X-Powered-By': 'Express', 'X-Progress': '21% complete, currenly \n",
       "processing Patient resources', 'Retry-After': '2', 'Date': 'Tue, 13 Jun 2023 11:05:54 GMT', 'Content-Length': '0', \n",
       "'Via': '1.1 vegur'}\n",
       "
\n" ], "text/plain": [ "\u001B[1m{\u001B[0m\u001B[32m'Server'\u001B[0m: \u001B[32m'Cowboy'\u001B[0m, \u001B[32m'Connection'\u001B[0m: \u001B[32m'keep-alive'\u001B[0m, \u001B[32m'X-Powered-By'\u001B[0m: \u001B[32m'Express'\u001B[0m, \u001B[32m'X-Progress'\u001B[0m: \u001B[32m'21% complete, currenly \u001B[0m\n", "\u001B[32mprocessing Patient resources'\u001B[0m, \u001B[32m'Retry-After'\u001B[0m: \u001B[32m'2'\u001B[0m, \u001B[32m'Date'\u001B[0m: \u001B[32m'Tue, 13 Jun 2023 11:05:54 GMT'\u001B[0m, \u001B[32m'Content-Length'\u001B[0m: \u001B[32m'0'\u001B[0m, \n", "\u001B[32m'Via'\u001B[0m: \u001B[32m'1.1 vegur'\u001B[0m\u001B[1m}\u001B[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
Sleeping 2 seconds before retrying\n",
       "
\n" ], "text/plain": [ "Sleeping \u001B[1;36m2\u001B[0m seconds before retrying\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
{'Server': 'Cowboy', 'Connection': 'keep-alive', 'X-Powered-By': 'Express', 'X-Progress': '42% complete, currenly \n",
       "processing Patient resources', 'Retry-After': '2', 'Date': 'Tue, 13 Jun 2023 11:05:56 GMT', 'Content-Length': '0', \n",
       "'Via': '1.1 vegur'}\n",
       "
\n" ], "text/plain": [ "\u001B[1m{\u001B[0m\u001B[32m'Server'\u001B[0m: \u001B[32m'Cowboy'\u001B[0m, \u001B[32m'Connection'\u001B[0m: \u001B[32m'keep-alive'\u001B[0m, \u001B[32m'X-Powered-By'\u001B[0m: \u001B[32m'Express'\u001B[0m, \u001B[32m'X-Progress'\u001B[0m: \u001B[32m'42% complete, currenly \u001B[0m\n", "\u001B[32mprocessing Patient resources'\u001B[0m, \u001B[32m'Retry-After'\u001B[0m: \u001B[32m'2'\u001B[0m, \u001B[32m'Date'\u001B[0m: \u001B[32m'Tue, 13 Jun 2023 11:05:56 GMT'\u001B[0m, \u001B[32m'Content-Length'\u001B[0m: \u001B[32m'0'\u001B[0m, \n", "\u001B[32m'Via'\u001B[0m: \u001B[32m'1.1 vegur'\u001B[0m\u001B[1m}\u001B[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
Sleeping 2 seconds before retrying\n",
       "
\n" ], "text/plain": [ "Sleeping \u001B[1;36m2\u001B[0m seconds before retrying\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
{'Server': 'Cowboy', 'Connection': 'keep-alive', 'X-Powered-By': 'Express', 'X-Progress': '63% complete, currenly \n",
       "processing Patient resources', 'Retry-After': '2', 'Date': 'Tue, 13 Jun 2023 11:05:58 GMT', 'Content-Length': '0', \n",
       "'Via': '1.1 vegur'}\n",
       "
\n" ], "text/plain": [ "\u001B[1m{\u001B[0m\u001B[32m'Server'\u001B[0m: \u001B[32m'Cowboy'\u001B[0m, \u001B[32m'Connection'\u001B[0m: \u001B[32m'keep-alive'\u001B[0m, \u001B[32m'X-Powered-By'\u001B[0m: \u001B[32m'Express'\u001B[0m, \u001B[32m'X-Progress'\u001B[0m: \u001B[32m'63% complete, currenly \u001B[0m\n", "\u001B[32mprocessing Patient resources'\u001B[0m, \u001B[32m'Retry-After'\u001B[0m: \u001B[32m'2'\u001B[0m, \u001B[32m'Date'\u001B[0m: \u001B[32m'Tue, 13 Jun 2023 11:05:58 GMT'\u001B[0m, \u001B[32m'Content-Length'\u001B[0m: \u001B[32m'0'\u001B[0m, \n", "\u001B[32m'Via'\u001B[0m: \u001B[32m'1.1 vegur'\u001B[0m\u001B[1m}\u001B[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
Sleeping 2 seconds before retrying\n",
       "
\n" ], "text/plain": [ "Sleeping \u001B[1;36m2\u001B[0m seconds before retrying\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
{'Server': 'Cowboy', 'Connection': 'keep-alive', 'X-Powered-By': 'Express', 'X-Progress': '83% complete, currenly \n",
       "processing Patient resources', 'Retry-After': '2', 'Date': 'Tue, 13 Jun 2023 11:06:00 GMT', 'Content-Length': '0', \n",
       "'Via': '1.1 vegur'}\n",
       "
\n" ], "text/plain": [ "\u001B[1m{\u001B[0m\u001B[32m'Server'\u001B[0m: \u001B[32m'Cowboy'\u001B[0m, \u001B[32m'Connection'\u001B[0m: \u001B[32m'keep-alive'\u001B[0m, \u001B[32m'X-Powered-By'\u001B[0m: \u001B[32m'Express'\u001B[0m, \u001B[32m'X-Progress'\u001B[0m: \u001B[32m'83% complete, currenly \u001B[0m\n", "\u001B[32mprocessing Patient resources'\u001B[0m, \u001B[32m'Retry-After'\u001B[0m: \u001B[32m'2'\u001B[0m, \u001B[32m'Date'\u001B[0m: \u001B[32m'Tue, 13 Jun 2023 11:06:00 GMT'\u001B[0m, \u001B[32m'Content-Length'\u001B[0m: \u001B[32m'0'\u001B[0m, \n", "\u001B[32m'Via'\u001B[0m: \u001B[32m'1.1 vegur'\u001B[0m\u001B[1m}\u001B[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
Sleeping 2 seconds before retrying\n",
       "
\n" ], "text/plain": [ "Sleeping \u001B[1;36m2\u001B[0m seconds before retrying\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
{\n",
       "    'transactionTime': '1686654352606',\n",
       "    'request': \n",
       "'https://bulk-data.smarthealthit.org/eyJlcnIiOiIiLCJwYWdlIjoxMDAwMCwiZHVyIjoxMCwidGx0IjoxNSwibSI6MSwic3R1Ijo0LCJkZW\n",
       "wiOjB9/fhir/Patient/$export?_type=Patient,Condition',\n",
       "    'requiresAccessToken': True,\n",
       "    'output': [\n",
       "        {\n",
       "            'type': 'Condition',\n",
       "            'count': 639,\n",
       "            'url': \n",
       "'https://bulk-data.smarthealthit.org/eyJpZCI6IjU1ZTI3NzBhNGI5YzNmODYxZjAwMjYzNGJkNDRhYzYyIiwib2Zmc2V0IjowLCJsaW1pdC\n",
       "I6NjM5LCJzZWN1cmUiOnRydWV9/fhir/bulkfiles/1.Condition.ndjson'\n",
       "        },\n",
       "        {\n",
       "            'type': 'Patient',\n",
       "            'count': 100,\n",
       "            'url': \n",
       "'https://bulk-data.smarthealthit.org/eyJpZCI6IjU1ZTI3NzBhNGI5YzNmODYxZjAwMjYzNGJkNDRhYzYyIiwib2Zmc2V0IjowLCJsaW1pdC\n",
       "I6MTAwLCJzZWN1cmUiOnRydWV9/fhir/bulkfiles/1.Patient.ndjson'\n",
       "        }\n",
       "    ],\n",
       "    'deleted': [],\n",
       "    'error': []\n",
       "}\n",
       "
\n" ], "text/plain": [ "\u001B[1m{\u001B[0m\n", " \u001B[32m'transactionTime'\u001B[0m: \u001B[32m'1686654352606'\u001B[0m,\n", " \u001B[32m'request'\u001B[0m: \n", "\u001B[32m'https://bulk-data.smarthealthit.org/eyJlcnIiOiIiLCJwYWdlIjoxMDAwMCwiZHVyIjoxMCwidGx0IjoxNSwibSI6MSwic3R1Ijo0LCJkZW\u001B[0m\n", "\u001B[32mwiOjB9/fhir/Patient/$export?\u001B[0m\u001B[32m_type\u001B[0m\u001B[32m=\u001B[0m\u001B[32mPatient\u001B[0m\u001B[32m,Condition'\u001B[0m,\n", " \u001B[32m'requiresAccessToken'\u001B[0m: \u001B[3;92mTrue\u001B[0m,\n", " \u001B[32m'output'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'type'\u001B[0m: \u001B[32m'Condition'\u001B[0m,\n", " \u001B[32m'count'\u001B[0m: \u001B[1;36m639\u001B[0m,\n", " \u001B[32m'url'\u001B[0m: \n", "\u001B[32m'https://bulk-data.smarthealthit.org/eyJpZCI6IjU1ZTI3NzBhNGI5YzNmODYxZjAwMjYzNGJkNDRhYzYyIiwib2Zmc2V0IjowLCJsaW1pdC\u001B[0m\n", "\u001B[32mI6NjM5LCJzZWN1cmUiOnRydWV9/fhir/bulkfiles/1.Condition.ndjson'\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'type'\u001B[0m: \u001B[32m'Patient'\u001B[0m,\n", " \u001B[32m'count'\u001B[0m: \u001B[1;36m100\u001B[0m,\n", " \u001B[32m'url'\u001B[0m: \n", "\u001B[32m'https://bulk-data.smarthealthit.org/eyJpZCI6IjU1ZTI3NzBhNGI5YzNmODYxZjAwMjYzNGJkNDRhYzYyIiwib2Zmc2V0IjowLCJsaW1pdC\u001B[0m\n", "\u001B[32mI6MTAwLCJzZWN1cmUiOnRydWV9/fhir/bulkfiles/1.Patient.ndjson'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'deleted'\u001B[0m: \u001B[1m[\u001B[0m\u001B[1m]\u001B[0m,\n", " \u001B[32m'error'\u001B[0m: \u001B[1m[\u001B[0m\u001B[1m]\u001B[0m\n", "\u001B[1m}\u001B[0m\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Now we check the status in a loop\n", "\n", "from time import sleep\n", "\n", "while True:\n", " r = session.get(check_url, headers={'Authorization': f'Bearer {get_token()}', 'Accept': 'application/fhir+json'})\n", "\n", " if r.status_code == 200:\n", " # complete\n", " response = r.json()\n", " print(response)\n", " break\n", "\n", " elif r.status_code == 202:\n", " # in progress\n", " print(r.headers)\n", "\n", " delay = r.headers['Retry-After']\n", "\n", " print(f\"Sleeping {delay} seconds before retrying\")\n", " sleep(int(delay))\n", "\n", " else:\n", " # error\n", " print(r.text)\n", "\n", " break" ] }, { "cell_type": "markdown", "id": "401fbd3c", "metadata": {}, "source": [ "We can see that the response points us to one or more NDJSON (Newline Delimited JSON) files per resource type, in the `output` field of the response.\n", "\n", "Note that in this case the volume of data is relatively small, and there is only one entry in the list per resource type, but for large datasets it is possible that there could be multiple files (and therefore multiple entries in this list) per resource type.\n", "\n", "Let's save that list to a variable." ] }, { "cell_type": "code", "execution_count": 13, "id": "4cfae17a", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'type': 'Condition',\n", " 'count': 639,\n", " 'url': 'https://bulk-data.smarthealthit.org/eyJpZCI6IjU1ZTI3NzBhNGI5YzNmODYxZjAwMjYzNGJkNDRhYzYyIiwib2Zmc2V0IjowLCJsaW1pdCI6NjM5LCJzZWN1cmUiOnRydWV9/fhir/bulkfiles/1.Condition.ndjson'},\n", " {'type': 'Patient',\n", " 'count': 100,\n", " 'url': 'https://bulk-data.smarthealthit.org/eyJpZCI6IjU1ZTI3NzBhNGI5YzNmODYxZjAwMjYzNGJkNDRhYzYyIiwib2Zmc2V0IjowLCJsaW1pdCI6MTAwLCJzZWN1cmUiOnRydWV9/fhir/bulkfiles/1.Patient.ndjson'}]" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "output_files = response['output']\n", "output_files" ] }, { "cell_type": "markdown", "id": "3c10b626", "metadata": {}, "source": [ "Now we can loop through the list and download each one. Each file is an NDJSON, so that means we'll see one resource per line.\n", "\n", "To make each step clear and distinct, we'll keep a dict of `{ resourceType: [resources,...]}` which we can process later.\n", "\n", "Note: for this exercise we are only reading the NDJSON files into a dict in memory, but in practice you may want to save the file locally first in case there are errors in processing, especially if the files are large." ] }, { "cell_type": "code", "execution_count": 14, "id": "68d72300", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "6016387320b344acb502b84f079daa46", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/2 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
resourceTypeidextensionidentifiernametelecomgenderbirthDateaddressmultipleBirthBooleancommunicationtext.statustext.divmaritalStatus.codingmaritalStatus.textmultipleBirthInteger
0Patient6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2[{'url': 'http://hl7.org/fhir/StructureDefinit...[{'system': 'https://github.com/synthetichealt...[{'use': 'official', 'family': 'Lemke', 'given...[{'system': 'phone', 'value': '555-532-1156', ...male1965-01-13[{'extension': [{'url': 'http://hl7.org/fhir/S...False[{'language': {'coding': [{'system': 'urn:ietf...generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...[{'system': 'http://terminology.hl7.org/CodeSy...MNaN
1Patient58c297c4-d684-4677-8024-01131d93835e[{'url': 'http://hl7.org/fhir/StructureDefinit...[{'system': 'https://github.com/synthetichealt...[{'use': 'official', 'family': 'Wintheiser', '...[{'system': 'phone', 'value': '555-712-4709', ...female1971-04-05[{'extension': [{'url': 'http://hl7.org/fhir/S...False[{'language': {'coding': [{'system': 'urn:ietf...generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...[{'system': 'http://terminology.hl7.org/CodeSy...MNaN
2Patient538a9a4e-8437-47d3-8c01-1a17dca8f0be[{'url': 'http://hl7.org/fhir/StructureDefinit...[{'system': 'https://github.com/synthetichealt...[{'use': 'official', 'family': 'Alaniz', 'give...[{'system': 'phone', 'value': '555-446-6900', ...male1923-03-24[{'extension': [{'url': 'http://hl7.org/fhir/S...False[{'language': {'coding': [{'system': 'urn:ietf...generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...[{'system': 'http://terminology.hl7.org/CodeSy...MNaN
3Patientc6c60742-8694-46e4-bb42-b00bf6d8b536[{'url': 'http://hl7.org/fhir/StructureDefinit...[{'system': 'https://github.com/synthetichealt...[{'use': 'official', 'family': 'Walsh', 'given...[{'system': 'phone', 'value': '555-436-4287', ...female1965-10-27[{'extension': [{'url': 'http://hl7.org/fhir/S...False[{'language': {'coding': [{'system': 'urn:ietf...generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...[{'system': 'http://terminology.hl7.org/CodeSy...MNaN
4Patientfbfec681-d357-4b28-b1d2-5db6434c7846[{'url': 'http://hl7.org/fhir/StructureDefinit...[{'system': 'https://github.com/synthetichealt...[{'use': 'official', 'family': 'Bednar', 'give...[{'system': 'phone', 'value': '555-405-4909', ...female1942-07-04[{'extension': [{'url': 'http://hl7.org/fhir/S...False[{'language': {'coding': [{'system': 'urn:ietf...generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...[{'system': 'http://terminology.hl7.org/CodeSy...MNaN
...................................................
95Patient5efb1ac1-d29b-40a5-a3d1-2d682f10bfa7[{'url': 'http://hl7.org/fhir/StructureDefinit...[{'system': 'https://github.com/synthetichealt...[{'use': 'official', 'family': 'Schmeler', 'gi...[{'system': 'phone', 'value': '555-971-6300', ...male1995-10-19[{'extension': [{'url': 'http://hl7.org/fhir/S...False[{'language': {'coding': [{'system': 'urn:ietf...generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...[{'system': 'http://terminology.hl7.org/CodeSy...Never MarriedNaN
96Patientc1981741-f90e-4077-9156-429a3c4c5ded[{'url': 'http://hl7.org/fhir/StructureDefinit...[{'system': 'https://github.com/synthetichealt...[{'use': 'official', 'family': 'Lubowitz', 'gi...[{'system': 'phone', 'value': '555-328-5229', ...male1956-05-06[{'extension': [{'url': 'http://hl7.org/fhir/S...False[{'language': {'coding': [{'system': 'urn:ietf...generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...[{'system': 'http://terminology.hl7.org/CodeSy...MNaN
97Patientf98b23bf-4443-46d0-9eaf-563e767cf948[{'url': 'http://hl7.org/fhir/StructureDefinit...[{'system': 'https://github.com/synthetichealt...[{'use': 'official', 'family': 'Funk', 'given'...[{'system': 'phone', 'value': '555-497-7639', ...male1966-02-07[{'extension': [{'url': 'http://hl7.org/fhir/S...False[{'language': {'coding': [{'system': 'urn:ietf...generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...[{'system': 'http://terminology.hl7.org/CodeSy...MNaN
98Patientc536dee9-9ef6-4807-ae20-9f1045c9c7d6[{'url': 'http://hl7.org/fhir/StructureDefinit...[{'system': 'https://github.com/synthetichealt...[{'use': 'official', 'family': 'Bergstrom', 'g...[{'system': 'phone', 'value': '555-845-1730', ...male1990-11-18[{'extension': [{'url': 'http://hl7.org/fhir/S...False[{'language': {'coding': [{'system': 'urn:ietf...generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...[{'system': 'http://terminology.hl7.org/CodeSy...SNaN
99Patienta845ead4-d9de-42eb-b4b5-eb21a8963578[{'url': 'http://hl7.org/fhir/StructureDefinit...[{'system': 'https://github.com/synthetichealt...[{'use': 'official', 'family': 'Pagac', 'given...[{'system': 'phone', 'value': '555-504-1379', ...female1968-04-20[{'extension': [{'url': 'http://hl7.org/fhir/S...False[{'language': {'coding': [{'system': 'urn:ietf...generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...[{'system': 'http://terminology.hl7.org/CodeSy...SNaN
\n", "

100 rows × 16 columns

\n", "" ], "text/plain": [ " resourceType id \\\n", "0 Patient 6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2 \n", "1 Patient 58c297c4-d684-4677-8024-01131d93835e \n", "2 Patient 538a9a4e-8437-47d3-8c01-1a17dca8f0be \n", "3 Patient c6c60742-8694-46e4-bb42-b00bf6d8b536 \n", "4 Patient fbfec681-d357-4b28-b1d2-5db6434c7846 \n", ".. ... ... \n", "95 Patient 5efb1ac1-d29b-40a5-a3d1-2d682f10bfa7 \n", "96 Patient c1981741-f90e-4077-9156-429a3c4c5ded \n", "97 Patient f98b23bf-4443-46d0-9eaf-563e767cf948 \n", "98 Patient c536dee9-9ef6-4807-ae20-9f1045c9c7d6 \n", "99 Patient a845ead4-d9de-42eb-b4b5-eb21a8963578 \n", "\n", " extension \\\n", "0 [{'url': 'http://hl7.org/fhir/StructureDefinit... \n", "1 [{'url': 'http://hl7.org/fhir/StructureDefinit... \n", "2 [{'url': 'http://hl7.org/fhir/StructureDefinit... \n", "3 [{'url': 'http://hl7.org/fhir/StructureDefinit... \n", "4 [{'url': 'http://hl7.org/fhir/StructureDefinit... \n", ".. ... \n", "95 [{'url': 'http://hl7.org/fhir/StructureDefinit... \n", "96 [{'url': 'http://hl7.org/fhir/StructureDefinit... \n", "97 [{'url': 'http://hl7.org/fhir/StructureDefinit... \n", "98 [{'url': 'http://hl7.org/fhir/StructureDefinit... \n", "99 [{'url': 'http://hl7.org/fhir/StructureDefinit... \n", "\n", " identifier \\\n", "0 [{'system': 'https://github.com/synthetichealt... \n", "1 [{'system': 'https://github.com/synthetichealt... \n", "2 [{'system': 'https://github.com/synthetichealt... \n", "3 [{'system': 'https://github.com/synthetichealt... \n", "4 [{'system': 'https://github.com/synthetichealt... \n", ".. ... \n", "95 [{'system': 'https://github.com/synthetichealt... \n", "96 [{'system': 'https://github.com/synthetichealt... \n", "97 [{'system': 'https://github.com/synthetichealt... \n", "98 [{'system': 'https://github.com/synthetichealt... \n", "99 [{'system': 'https://github.com/synthetichealt... \n", "\n", " name \\\n", "0 [{'use': 'official', 'family': 'Lemke', 'given... \n", "1 [{'use': 'official', 'family': 'Wintheiser', '... \n", "2 [{'use': 'official', 'family': 'Alaniz', 'give... \n", "3 [{'use': 'official', 'family': 'Walsh', 'given... \n", "4 [{'use': 'official', 'family': 'Bednar', 'give... \n", ".. ... \n", "95 [{'use': 'official', 'family': 'Schmeler', 'gi... \n", "96 [{'use': 'official', 'family': 'Lubowitz', 'gi... \n", "97 [{'use': 'official', 'family': 'Funk', 'given'... \n", "98 [{'use': 'official', 'family': 'Bergstrom', 'g... \n", "99 [{'use': 'official', 'family': 'Pagac', 'given... \n", "\n", " telecom gender birthDate \\\n", "0 [{'system': 'phone', 'value': '555-532-1156', ... male 1965-01-13 \n", "1 [{'system': 'phone', 'value': '555-712-4709', ... female 1971-04-05 \n", "2 [{'system': 'phone', 'value': '555-446-6900', ... male 1923-03-24 \n", "3 [{'system': 'phone', 'value': '555-436-4287', ... female 1965-10-27 \n", "4 [{'system': 'phone', 'value': '555-405-4909', ... female 1942-07-04 \n", ".. ... ... ... \n", "95 [{'system': 'phone', 'value': '555-971-6300', ... male 1995-10-19 \n", "96 [{'system': 'phone', 'value': '555-328-5229', ... male 1956-05-06 \n", "97 [{'system': 'phone', 'value': '555-497-7639', ... male 1966-02-07 \n", "98 [{'system': 'phone', 'value': '555-845-1730', ... male 1990-11-18 \n", "99 [{'system': 'phone', 'value': '555-504-1379', ... female 1968-04-20 \n", "\n", " address multipleBirthBoolean \\\n", "0 [{'extension': [{'url': 'http://hl7.org/fhir/S... False \n", "1 [{'extension': [{'url': 'http://hl7.org/fhir/S... False \n", "2 [{'extension': [{'url': 'http://hl7.org/fhir/S... False \n", "3 [{'extension': [{'url': 'http://hl7.org/fhir/S... False \n", "4 [{'extension': [{'url': 'http://hl7.org/fhir/S... False \n", ".. ... ... \n", "95 [{'extension': [{'url': 'http://hl7.org/fhir/S... False \n", "96 [{'extension': [{'url': 'http://hl7.org/fhir/S... False \n", "97 [{'extension': [{'url': 'http://hl7.org/fhir/S... False \n", "98 [{'extension': [{'url': 'http://hl7.org/fhir/S... False \n", "99 [{'extension': [{'url': 'http://hl7.org/fhir/S... False \n", "\n", " communication text.status \\\n", "0 [{'language': {'coding': [{'system': 'urn:ietf... generated \n", "1 [{'language': {'coding': [{'system': 'urn:ietf... generated \n", "2 [{'language': {'coding': [{'system': 'urn:ietf... generated \n", "3 [{'language': {'coding': [{'system': 'urn:ietf... generated \n", "4 [{'language': {'coding': [{'system': 'urn:ietf... generated \n", ".. ... ... \n", "95 [{'language': {'coding': [{'system': 'urn:ietf... generated \n", "96 [{'language': {'coding': [{'system': 'urn:ietf... generated \n", "97 [{'language': {'coding': [{'system': 'urn:ietf... generated \n", "98 [{'language': {'coding': [{'system': 'urn:ietf... generated \n", "99 [{'language': {'coding': [{'system': 'urn:ietf... generated \n", "\n", " text.div \\\n", "0
Gene... \n", "1
Gene... \n", "2
Gene... \n", "3
Gene... \n", "4
Gene... \n", ".. ... \n", "95
Gene... \n", "96
Gene... \n", "97
Gene... \n", "98
Gene... \n", "99
Gene... \n", "\n", " maritalStatus.coding maritalStatus.text \\\n", "0 [{'system': 'http://terminology.hl7.org/CodeSy... M \n", "1 [{'system': 'http://terminology.hl7.org/CodeSy... M \n", "2 [{'system': 'http://terminology.hl7.org/CodeSy... M \n", "3 [{'system': 'http://terminology.hl7.org/CodeSy... M \n", "4 [{'system': 'http://terminology.hl7.org/CodeSy... M \n", ".. ... ... \n", "95 [{'system': 'http://terminology.hl7.org/CodeSy... Never Married \n", "96 [{'system': 'http://terminology.hl7.org/CodeSy... M \n", "97 [{'system': 'http://terminology.hl7.org/CodeSy... M \n", "98 [{'system': 'http://terminology.hl7.org/CodeSy... S \n", "99 [{'system': 'http://terminology.hl7.org/CodeSy... S \n", "\n", " multipleBirthInteger \n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 NaN \n", "4 NaN \n", ".. ... \n", "95 NaN \n", "96 NaN \n", "97 NaN \n", "98 NaN \n", "99 NaN \n", "\n", "[100 rows x 16 columns]" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "\n", "resource_dfs = {}\n", "\n", "for resource_type, resources in resources_by_type.items():\n", " resource_dfs[resource_type] = pd.json_normalize(resources)\n", "\n", "# Now we can work with them by type:\n", "\n", "resource_dfs['Patient']\n" ] }, { "cell_type": "markdown", "id": "aac28904", "metadata": {}, "source": [ "This works, but it's clearly not ideal in how it handles nested fields, such as the nested lists of the `name` field. One way we can do a little better is with the flatten_json library: \n" ] }, { "cell_type": "code", "execution_count": 16, "id": "160fcda7", "metadata": { "scrolled": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
resourceTypeidtext_statustext_divextension_0_urlextension_0_valueStringextension_1_urlextension_1_valueAddress_cityextension_1_valueAddress_stateextension_1_valueAddress_country...multipleBirthBooleancommunication_0_language_coding_0_systemcommunication_0_language_coding_0_codecommunication_0_language_coding_0_displaycommunication_0_language_textname_1_usename_1_familyname_1_given_0name_1_prefix_0multipleBirthInteger
0Patient6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...http://hl7.org/fhir/StructureDefinition/patien...Lettie Boylehttp://hl7.org/fhir/StructureDefinition/patien...BostonMassachusettsUS...Falseurn:ietf:bcp:47en-USEnglishEnglishNaNNaNNaNNaNNaN
1Patient58c297c4-d684-4677-8024-01131d93835egenerated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...http://hl7.org/fhir/StructureDefinition/patien...Marquetta Schambergerhttp://hl7.org/fhir/StructureDefinition/patien...MacauMacao Special Administrative Region of the Peo...CN...Falseurn:ietf:bcp:47zhChineseChinesemaidenHeathcoteAletaMrs.NaN
2Patient538a9a4e-8437-47d3-8c01-1a17dca8f0begenerated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...http://hl7.org/fhir/StructureDefinition/patien...Pilar Ortahttp://hl7.org/fhir/StructureDefinition/patien...San JoseSan JoseCR...Falseurn:ietf:bcp:47esSpanishSpanishNaNNaNNaNNaNNaN
3Patientc6c60742-8694-46e4-bb42-b00bf6d8b536generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...http://hl7.org/fhir/StructureDefinition/patien...Arvilla Haaghttp://hl7.org/fhir/StructureDefinition/patien...NortonMassachusettsUS...Falseurn:ietf:bcp:47en-USEnglishEnglishmaidenKuphalAlyceMrs.NaN
4Patientfbfec681-d357-4b28-b1d2-5db6434c7846generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...http://hl7.org/fhir/StructureDefinition/patien...Marcelina Harberhttp://hl7.org/fhir/StructureDefinition/patien...BrocktonMassachusettsUS...Falseurn:ietf:bcp:47en-USEnglishEnglishmaidenRunolfssonArnetteMrs.NaN
..................................................................
95Patient5efb1ac1-d29b-40a5-a3d1-2d682f10bfa7generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...http://hl7.org/fhir/StructureDefinition/patien...Allison Daughertyhttp://hl7.org/fhir/StructureDefinition/patien...BostonMassachusettsUS...Falseurn:ietf:bcp:47en-USEnglishEnglishNaNNaNNaNNaNNaN
96Patientc1981741-f90e-4077-9156-429a3c4c5dedgenerated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...http://hl7.org/fhir/StructureDefinition/patien...Antoinette Parkerhttp://hl7.org/fhir/StructureDefinition/patien...MansfieldMassachusettsUS...Falseurn:ietf:bcp:47en-USEnglishEnglishNaNNaNNaNNaNNaN
97Patientf98b23bf-4443-46d0-9eaf-563e767cf948generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...http://hl7.org/fhir/StructureDefinition/patien...Barbar Windlerhttp://hl7.org/fhir/StructureDefinition/patien...RandolphMassachusettsUS...Falseurn:ietf:bcp:47en-USEnglishEnglishNaNNaNNaNNaNNaN
98Patientc536dee9-9ef6-4807-ae20-9f1045c9c7d6generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...http://hl7.org/fhir/StructureDefinition/patien...Juli Johnshttp://hl7.org/fhir/StructureDefinition/patien...HolyokeMassachusettsUS...Falseurn:ietf:bcp:47en-USEnglishEnglishNaNNaNNaNNaNNaN
99Patienta845ead4-d9de-42eb-b4b5-eb21a8963578generated<div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...http://hl7.org/fhir/StructureDefinition/patien...Lanie Hyatthttp://hl7.org/fhir/StructureDefinition/patien...MillisMassachusettsUS...Falseurn:ietf:bcp:47en-USEnglishEnglishNaNNaNNaNNaNNaN
\n", "

100 rows × 73 columns

\n", "
" ], "text/plain": [ " resourceType id text_status \\\n", "0 Patient 6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2 generated \n", "1 Patient 58c297c4-d684-4677-8024-01131d93835e generated \n", "2 Patient 538a9a4e-8437-47d3-8c01-1a17dca8f0be generated \n", "3 Patient c6c60742-8694-46e4-bb42-b00bf6d8b536 generated \n", "4 Patient fbfec681-d357-4b28-b1d2-5db6434c7846 generated \n", ".. ... ... ... \n", "95 Patient 5efb1ac1-d29b-40a5-a3d1-2d682f10bfa7 generated \n", "96 Patient c1981741-f90e-4077-9156-429a3c4c5ded generated \n", "97 Patient f98b23bf-4443-46d0-9eaf-563e767cf948 generated \n", "98 Patient c536dee9-9ef6-4807-ae20-9f1045c9c7d6 generated \n", "99 Patient a845ead4-d9de-42eb-b4b5-eb21a8963578 generated \n", "\n", " text_div \\\n", "0
Gene... \n", "1
Gene... \n", "2
Gene... \n", "3
Gene... \n", "4
Gene... \n", ".. ... \n", "95
Gene... \n", "96
Gene... \n", "97
Gene... \n", "98
Gene... \n", "99
Gene... \n", "\n", " extension_0_url extension_0_valueString \\\n", "0 http://hl7.org/fhir/StructureDefinition/patien... Lettie Boyle \n", "1 http://hl7.org/fhir/StructureDefinition/patien... Marquetta Schamberger \n", "2 http://hl7.org/fhir/StructureDefinition/patien... Pilar Orta \n", "3 http://hl7.org/fhir/StructureDefinition/patien... Arvilla Haag \n", "4 http://hl7.org/fhir/StructureDefinition/patien... Marcelina Harber \n", ".. ... ... \n", "95 http://hl7.org/fhir/StructureDefinition/patien... Allison Daugherty \n", "96 http://hl7.org/fhir/StructureDefinition/patien... Antoinette Parker \n", "97 http://hl7.org/fhir/StructureDefinition/patien... Barbar Windler \n", "98 http://hl7.org/fhir/StructureDefinition/patien... Juli Johns \n", "99 http://hl7.org/fhir/StructureDefinition/patien... Lanie Hyatt \n", "\n", " extension_1_url \\\n", "0 http://hl7.org/fhir/StructureDefinition/patien... \n", "1 http://hl7.org/fhir/StructureDefinition/patien... \n", "2 http://hl7.org/fhir/StructureDefinition/patien... \n", "3 http://hl7.org/fhir/StructureDefinition/patien... \n", "4 http://hl7.org/fhir/StructureDefinition/patien... \n", ".. ... \n", "95 http://hl7.org/fhir/StructureDefinition/patien... \n", "96 http://hl7.org/fhir/StructureDefinition/patien... \n", "97 http://hl7.org/fhir/StructureDefinition/patien... \n", "98 http://hl7.org/fhir/StructureDefinition/patien... \n", "99 http://hl7.org/fhir/StructureDefinition/patien... \n", "\n", " extension_1_valueAddress_city \\\n", "0 Boston \n", "1 Macau \n", "2 San Jose \n", "3 Norton \n", "4 Brockton \n", ".. ... \n", "95 Boston \n", "96 Mansfield \n", "97 Randolph \n", "98 Holyoke \n", "99 Millis \n", "\n", " extension_1_valueAddress_state \\\n", "0 Massachusetts \n", "1 Macao Special Administrative Region of the Peo... \n", "2 San Jose \n", "3 Massachusetts \n", "4 Massachusetts \n", ".. ... \n", "95 Massachusetts \n", "96 Massachusetts \n", "97 Massachusetts \n", "98 Massachusetts \n", "99 Massachusetts \n", "\n", " extension_1_valueAddress_country ... multipleBirthBoolean \\\n", "0 US ... False \n", "1 CN ... False \n", "2 CR ... False \n", "3 US ... False \n", "4 US ... False \n", ".. ... ... ... \n", "95 US ... False \n", "96 US ... False \n", "97 US ... False \n", "98 US ... False \n", "99 US ... False \n", "\n", " communication_0_language_coding_0_system \\\n", "0 urn:ietf:bcp:47 \n", "1 urn:ietf:bcp:47 \n", "2 urn:ietf:bcp:47 \n", "3 urn:ietf:bcp:47 \n", "4 urn:ietf:bcp:47 \n", ".. ... \n", "95 urn:ietf:bcp:47 \n", "96 urn:ietf:bcp:47 \n", "97 urn:ietf:bcp:47 \n", "98 urn:ietf:bcp:47 \n", "99 urn:ietf:bcp:47 \n", "\n", " communication_0_language_coding_0_code \\\n", "0 en-US \n", "1 zh \n", "2 es \n", "3 en-US \n", "4 en-US \n", ".. ... \n", "95 en-US \n", "96 en-US \n", "97 en-US \n", "98 en-US \n", "99 en-US \n", "\n", " communication_0_language_coding_0_display communication_0_language_text \\\n", "0 English English \n", "1 Chinese Chinese \n", "2 Spanish Spanish \n", "3 English English \n", "4 English English \n", ".. ... ... \n", "95 English English \n", "96 English English \n", "97 English English \n", "98 English English \n", "99 English English \n", "\n", " name_1_use name_1_family name_1_given_0 name_1_prefix_0 \\\n", "0 NaN NaN NaN NaN \n", "1 maiden Heathcote Aleta Mrs. \n", "2 NaN NaN NaN NaN \n", "3 maiden Kuphal Alyce Mrs. \n", "4 maiden Runolfsson Arnette Mrs. \n", ".. ... ... ... ... \n", "95 NaN NaN NaN NaN \n", "96 NaN NaN NaN NaN \n", "97 NaN NaN NaN NaN \n", "98 NaN NaN NaN NaN \n", "99 NaN NaN NaN NaN \n", "\n", " multipleBirthInteger \n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 NaN \n", "4 NaN \n", ".. ... \n", "95 NaN \n", "96 NaN \n", "97 NaN \n", "98 NaN \n", "99 NaN \n", "\n", "[100 rows x 73 columns]" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from flatten_json import flatten\n", "\n", "for resource_type, resources in resources_by_type.items():\n", " resource_dfs[resource_type] = pd.json_normalize(list(map(lambda r: flatten(r), resources)))\n", "\n", "# Now let's take another look\n", "resource_dfs['Patient']" ] }, { "cell_type": "markdown", "id": "3354c92b", "metadata": {}, "source": [ "Let's look at just one row so it's easier to see all the columns and an example value:" ] }, { "cell_type": "code", "execution_count": 17, "id": "5aa89719", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
resourceType                                                                                Patient\n",
       "id                                                             6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2\n",
       "text_status                                                                               generated\n",
       "text_div                                          <div xmlns=\"http://www.w3.org/1999/xhtml\">Gene...\n",
       "extension_0_url                                   http://hl7.org/fhir/StructureDefinition/patien...\n",
       "extension_0_valueString                                                                Lettie Boyle\n",
       "extension_1_url                                   http://hl7.org/fhir/StructureDefinition/patien...\n",
       "extension_1_valueAddress_city                                                                Boston\n",
       "extension_1_valueAddress_state                                                        Massachusetts\n",
       "extension_1_valueAddress_country                                                                 US\n",
       "extension_2_url                                   http://synthetichealth.github.io/synthea/disab...\n",
       "extension_2_valueDecimal                                                                   0.305628\n",
       "extension_3_url                                   http://synthetichealth.github.io/synthea/quali...\n",
       "extension_3_valueDecimal                                                                  53.694372\n",
       "identifier_0_system                                      https://github.com/synthetichealth/synthea\n",
       "identifier_0_value                                             6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2\n",
       "identifier_1_type_coding_0_system                     http://terminology.hl7.org/CodeSystem/v2-0203\n",
       "identifier_1_type_coding_0_code                                                                  MR\n",
       "identifier_1_type_coding_0_display                                            Medical Record Number\n",
       "identifier_1_type_text                                                        Medical Record Number\n",
       "identifier_1_system                                               http://hospital.smarthealthit.org\n",
       "identifier_1_value                                             6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2\n",
       "identifier_2_type_coding_0_system                     http://terminology.hl7.org/CodeSystem/v2-0203\n",
       "identifier_2_type_coding_0_code                                                                  SS\n",
       "identifier_2_type_coding_0_display                                           Social Security Number\n",
       "identifier_2_type_text                                                       Social Security Number\n",
       "identifier_2_system                                                  http://hl7.org/fhir/sid/us-ssn\n",
       "identifier_2_value                                                                      999-18-8203\n",
       "identifier_3_type_coding_0_system                     http://terminology.hl7.org/CodeSystem/v2-0203\n",
       "identifier_3_type_coding_0_code                                                                  DL\n",
       "identifier_3_type_coding_0_display                                                 Driver's License\n",
       "identifier_3_type_text                                                             Driver's License\n",
       "identifier_3_system                                                urn:oid:2.16.840.1.113883.4.3.25\n",
       "identifier_3_value                                                                        S99914534\n",
       "identifier_4_type_coding_0_system                     http://terminology.hl7.org/CodeSystem/v2-0203\n",
       "identifier_4_type_coding_0_code                                                                 PPN\n",
       "identifier_4_type_coding_0_display                                                  Passport Number\n",
       "identifier_4_type_text                                                              Passport Number\n",
       "identifier_4_system                               http://standardhealthrecord.org/fhir/Structure...\n",
       "identifier_4_value                                                                       X41457228X\n",
       "name_0_use                                                                                 official\n",
       "name_0_family                                                                                 Lemke\n",
       "name_0_given_0                                                                                Abram\n",
       "name_0_prefix_0                                                                                 Mr.\n",
       "telecom_0_system                                                                              phone\n",
       "telecom_0_value                                                                        555-532-1156\n",
       "telecom_0_use                                                                                  home\n",
       "gender                                                                                         male\n",
       "birthDate                                                                                1965-01-13\n",
       "address_0_extension_0_url                         http://hl7.org/fhir/StructureDefinition/geoloc...\n",
       "address_0_extension_0_extension_0_url                                                      latitude\n",
       "address_0_extension_0_extension_0_valueDecimal                                            42.264144\n",
       "address_0_extension_0_extension_1_url                                                     longitude\n",
       "address_0_extension_0_extension_1_valueDecimal                                           -72.642902\n",
       "address_0_line_0                                                                  167 Nikolaus Gate\n",
       "address_0_city                                                                          Easthampton\n",
       "address_0_state                                                                       Massachusetts\n",
       "address_0_postalCode                                                                          01027\n",
       "address_0_country                                                                                US\n",
       "maritalStatus_coding_0_system                     http://terminology.hl7.org/CodeSystem/v3-Marit...\n",
       "maritalStatus_coding_0_code                                                                       M\n",
       "maritalStatus_coding_0_display                                                                    M\n",
       "maritalStatus_text                                                                                M\n",
       "multipleBirthBoolean                                                                          False\n",
       "communication_0_language_coding_0_system                                            urn:ietf:bcp:47\n",
       "communication_0_language_coding_0_code                                                        en-US\n",
       "communication_0_language_coding_0_display                                                   English\n",
       "communication_0_language_text                                                               English\n",
       "name_1_use                                                                                      NaN\n",
       "name_1_family                                                                                   NaN\n",
       "name_1_given_0                                                                                  NaN\n",
       "name_1_prefix_0                                                                                 NaN\n",
       "multipleBirthInteger                                                                            NaN\n",
       "Name: 0, dtype: object\n",
       "
\n" ], "text/plain": [ "resourceType Patient\n", "id \u001B[93m6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2\u001B[0m\n", "text_status generated\n", "text_div \u001B[1m<\u001B[0m\u001B[1;95mdiv\u001B[0m\u001B[39m \u001B[0m\u001B[33mxmlns\u001B[0m\u001B[39m=\u001B[0m\u001B[32m\"http\u001B[0m\u001B[32m://www.w3.org/1999/xhtml\"\u001B[0m\u001B[1m>\u001B[0mGene\u001B[33m...\u001B[0m\n", "extension_0_url \u001B[4;94mhttp://hl7.org/fhir/StructureDefinition/patien...\u001B[0m\n", "extension_0_valueString Lettie Boyle\n", "extension_1_url \u001B[4;94mhttp://hl7.org/fhir/StructureDefinition/patien...\u001B[0m\n", "extension_1_valueAddress_city Boston\n", "extension_1_valueAddress_state Massachusetts\n", "extension_1_valueAddress_country US\n", "extension_2_url \u001B[4;94mhttp://synthetichealth.github.io/synthea/disab...\u001B[0m\n", "extension_2_valueDecimal \u001B[1;36m0.305628\u001B[0m\n", "extension_3_url \u001B[4;94mhttp://synthetichealth.github.io/synthea/quali...\u001B[0m\n", "extension_3_valueDecimal \u001B[1;36m53.694372\u001B[0m\n", "identifier_0_system \u001B[4;94mhttps://github.com/synthetichealth/synthea\u001B[0m\n", "identifier_0_value \u001B[93m6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2\u001B[0m\n", "identifier_1_type_coding_0_system \u001B[4;94mhttp://terminology.hl7.org/CodeSystem/v2-0203\u001B[0m\n", "identifier_1_type_coding_0_code MR\n", "identifier_1_type_coding_0_display Medical Record Number\n", "identifier_1_type_text Medical Record Number\n", "identifier_1_system \u001B[4;94mhttp://hospital.smarthealthit.org\u001B[0m\n", "identifier_1_value \u001B[93m6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2\u001B[0m\n", "identifier_2_type_coding_0_system \u001B[4;94mhttp://terminology.hl7.org/CodeSystem/v2-0203\u001B[0m\n", "identifier_2_type_coding_0_code SS\n", "identifier_2_type_coding_0_display Social Security Number\n", "identifier_2_type_text Social Security Number\n", "identifier_2_system \u001B[4;94mhttp://hl7.org/fhir/sid/us-ssn\u001B[0m\n", "identifier_2_value \u001B[1;36m999\u001B[0m-\u001B[1;36m18\u001B[0m-\u001B[1;36m8203\u001B[0m\n", "identifier_3_type_coding_0_system \u001B[4;94mhttp://terminology.hl7.org/CodeSystem/v2-0203\u001B[0m\n", "identifier_3_type_coding_0_code DL\n", "identifier_3_type_coding_0_display Driver's License\n", "identifier_3_type_text Driver's License\n", "identifier_3_system urn:oi\u001B[1;92md:2\u001B[0m.\u001B[1;92m16.840.1.113\u001B[0m\u001B[1;92m883.4.3.25\u001B[0m\n", "identifier_3_value S99914534\n", "identifier_4_type_coding_0_system \u001B[4;94mhttp://terminology.hl7.org/CodeSystem/v2-0203\u001B[0m\n", "identifier_4_type_coding_0_code PPN\n", "identifier_4_type_coding_0_display Passport Number\n", "identifier_4_type_text Passport Number\n", "identifier_4_system \u001B[4;94mhttp://standardhealthrecord.org/fhir/Structure...\u001B[0m\n", "identifier_4_value X41457228X\n", "name_0_use official\n", "name_0_family Lemke\n", "name_0_given_0 Abram\n", "name_0_prefix_0 Mr.\n", "telecom_0_system phone\n", "telecom_0_value \u001B[1;36m555\u001B[0m-\u001B[1;36m532\u001B[0m-\u001B[1;36m1156\u001B[0m\n", "telecom_0_use home\n", "gender male\n", "birthDate \u001B[1;36m1965\u001B[0m-\u001B[1;36m01\u001B[0m-\u001B[1;36m13\u001B[0m\n", "address_0_extension_0_url \u001B[4;94mhttp://hl7.org/fhir/StructureDefinition/geoloc...\u001B[0m\n", "address_0_extension_0_extension_0_url latitude\n", "address_0_extension_0_extension_0_valueDecimal \u001B[1;36m42.264144\u001B[0m\n", "address_0_extension_0_extension_1_url longitude\n", "address_0_extension_0_extension_1_valueDecimal \u001B[1;36m-72.642902\u001B[0m\n", "address_0_line_0 \u001B[1;36m167\u001B[0m Nikolaus Gate\n", "address_0_city Easthampton\n", "address_0_state Massachusetts\n", "address_0_postalCode \u001B[1;36m01027\u001B[0m\n", "address_0_country US\n", "maritalStatus_coding_0_system \u001B[4;94mhttp://terminology.hl7.org/CodeSystem/v3-Marit...\u001B[0m\n", "maritalStatus_coding_0_code M\n", "maritalStatus_coding_0_display M\n", "maritalStatus_text M\n", "multipleBirthBoolean \u001B[3;91mFalse\u001B[0m\n", "communication_0_language_coding_0_system urn:iet\u001B[1;92mf:bc\u001B[0mp:\u001B[1;36m47\u001B[0m\n", "communication_0_language_coding_0_code en-US\n", "communication_0_language_coding_0_display English\n", "communication_0_language_text English\n", "name_1_use NaN\n", "name_1_family NaN\n", "name_1_given_0 NaN\n", "name_1_prefix_0 NaN\n", "multipleBirthInteger NaN\n", "Name: \u001B[1;36m0\u001B[0m, dtype: object\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "with pd.option_context('display.max_rows', 1000, 'display.max_columns', 10):\n", " print(resource_dfs['Patient'].loc[0].T)" ] }, { "cell_type": "markdown", "id": "755858bd", "metadata": {}, "source": [ "Next, what if we know in advance we will only want certain fields?\n", "\n", "Let's follow the same pattern the FHIR-PYrate library uses, and use [FHIRPath](https://hl7.org/fhir/fhirpath.html) to define the fields we want to extract, along with a friendly name.\n", "For this we'll use the [fhirpathpy](https://github.com/beda-software/fhirpath-py) library.\n", "\n", "[FHIRPath is](http://hl7.org/fhirpath/N1/):\n", "\n", "> a path based navigation and extraction language, somewhat like XPath. Operations are expressed in terms of the logical content of hierarchical data models, and support traversal, selection and filtering of data." ] }, { "cell_type": "markdown", "id": "0b6f1587", "metadata": {}, "source": [ "If you are not familiar with FHIRPath, [Section 3 of the FHIRPath spec](http://hl7.org/fhirpath/N1/#path-selection) describes some of the basics." ] }, { "cell_type": "code", "execution_count": 18, "id": "0e16d9c8", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idgenderdate_of_birthmarital_status
06c5d9ca9-54d7-42f5-bfae-a7c19cd217f2male1965-01-13M
158c297c4-d684-4677-8024-01131d93835efemale1971-04-05M
2538a9a4e-8437-47d3-8c01-1a17dca8f0bemale1923-03-24M
3c6c60742-8694-46e4-bb42-b00bf6d8b536female1965-10-27M
4fbfec681-d357-4b28-b1d2-5db6434c7846female1942-07-04M
...............
955efb1ac1-d29b-40a5-a3d1-2d682f10bfa7male1995-10-19S
96c1981741-f90e-4077-9156-429a3c4c5dedmale1956-05-06M
97f98b23bf-4443-46d0-9eaf-563e767cf948male1966-02-07M
98c536dee9-9ef6-4807-ae20-9f1045c9c7d6male1990-11-18S
99a845ead4-d9de-42eb-b4b5-eb21a8963578female1968-04-20S
\n", "

100 rows × 4 columns

\n", "
" ], "text/plain": [ " id gender date_of_birth marital_status\n", "0 6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2 male 1965-01-13 M\n", "1 58c297c4-d684-4677-8024-01131d93835e female 1971-04-05 M\n", "2 538a9a4e-8437-47d3-8c01-1a17dca8f0be male 1923-03-24 M\n", "3 c6c60742-8694-46e4-bb42-b00bf6d8b536 female 1965-10-27 M\n", "4 fbfec681-d357-4b28-b1d2-5db6434c7846 female 1942-07-04 M\n", ".. ... ... ... ...\n", "95 5efb1ac1-d29b-40a5-a3d1-2d682f10bfa7 male 1995-10-19 S\n", "96 c1981741-f90e-4077-9156-429a3c4c5ded male 1956-05-06 M\n", "97 f98b23bf-4443-46d0-9eaf-563e767cf948 male 1966-02-07 M\n", "98 c536dee9-9ef6-4807-ae20-9f1045c9c7d6 male 1990-11-18 S\n", "99 a845ead4-d9de-42eb-b4b5-eb21a8963578 female 1968-04-20 S\n", "\n", "[100 rows x 4 columns]" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import fhirpathpy\n", "\n", "fhir_paths = [\n", " [\"id\", \"identifier[0].value\"],\n", " [\"gender\", \"gender\"],\n", " [\"date_of_birth\", \"birthDate\"],\n", " [\"marital_status\", \"maritalStatus.coding.first().code\"]\n", " ]\n", "\n", "# compile the fhirpath so they can be reused. this will result in better performance on large datasets\n", "for f in fhir_paths:\n", " f[1] = fhirpathpy.compile(f[1])\n", "\n", "for resource_type, resources in resources_by_type.items():\n", " filtered_resources = []\n", "\n", " for resource in resources:\n", " filtered_resource = {}\n", " for f in fhir_paths:\n", " fieldname = f[0]\n", " func = f[1]\n", " filtered_resource[fieldname] = func(resource)\n", "\n", " # fhirpathpy always returns a list, which can make the DataFrame messy\n", " # if it's a list with only one item, extract the item from the list\n", " if isinstance(filtered_resource[fieldname], list) and len(filtered_resource[fieldname]) == 1:\n", " filtered_resource[fieldname] = filtered_resource[fieldname][0]\n", "\n", " filtered_resources.append(filtered_resource)\n", "\n", " resource_dfs[resource_type] = pd.json_normalize(list(map(lambda r: flatten(r), filtered_resources)))\n", "\n", "\n", "resource_dfs['Patient']" ] }, { "cell_type": "markdown", "id": "828fbf49", "metadata": {}, "source": [ "## Bringing it all together\n", "\n", "Now we have everything we need to connect to a FHIR server that supports Bulk Data, request and download exported data, and convert it into a DataFrame. Let's bring everything together from the previous steps into one class with a clear entrypoint." ] }, { "cell_type": "code", "execution_count": 19, "id": "561dee3b", "metadata": { "pycharm": { "is_executing": true } }, "outputs": [], "source": [ "import requests\n", "import jwt\n", "import datetime\n", "import json\n", "import fhirpathpy\n", "from flatten_json import flatten\n", "from typing import Optional\n", "from collections import defaultdict\n", "\n", "class BulkDataFetcher:\n", " def __init__(\n", " self,\n", " base_url: str,\n", " client_id: str,\n", " private_key: str,\n", " key_id: str,\n", " endpoint: Optional[str] = None,\n", " session: Optional[str] = None\n", " ):\n", " self.base_url = base_url\n", " self.client_id = client_id\n", " self.private_key = private_key\n", " self.key_id = key_id\n", "\n", " self.token = None\n", " self.token_expire_time = None\n", "\n", " if endpoint is None:\n", " self.endpoint = \"Patient\"\n", " else:\n", " self.endpoint = endpoint\n", "\n", "\n", " if session is None:\n", " self.session = requests.Session()\n", " else:\n", " self.session = session\n", "\n", " r = self.session.get(f'{base_url}/.well-known/smart-configuration')\n", " smart_config = r.json()\n", " self.token_endpoint = smart_config['token_endpoint']\n", "\n", " self.resource_types = []\n", " self.fhir_paths = {}\n", "\n", " # Store raw FHIR resource instances; populated as part of get_dataframes()\n", " self.resources_by_type = {}\n", "\n", "\n", " def get_token(self):\n", " if self.token and datetime.datetime.now() < self.expire_time:\n", " # the existing token is still valid so use it\n", " return self.token\n", "\n", " assertion = jwt.encode({\n", " 'iss': self.client_id,\n", " 'sub': self.client_id,\n", " 'aud': self.token_endpoint,\n", " 'exp': int((datetime.datetime.now() + datetime.timedelta(minutes=5)).timestamp())\n", " }, self.private_key, algorithm='RS384',\n", " headers={\"kid\": key_id})\n", "\n", " r = self.session.post(self.token_endpoint, data={\n", " 'scope': 'system/*.read',\n", " 'grant_type': 'client_credentials',\n", " 'client_assertion_type': 'urn:ietf:params:oauth:client-assertion-type:jwt-bearer',\n", " 'client_assertion': assertion\n", " })\n", "\n", " token_response = r.json()\n", " self.token = token_response['access_token']\n", " self.expire_time = datetime.datetime.now() + datetime.timedelta(seconds=token_response['expires_in'])\n", "\n", " return self.token\n", "\n", " def add_resource_type(self, resource_type: str, fhir_paths = None):\n", " self.resource_types.append(resource_type)\n", " if fhir_paths:\n", " # fhir_paths=[\n", " # (\"id\", \"identifier[0].value\"),\n", " # (\"marital_status\", \"maritalStatus.coding[0].code\")\n", " # ]\n", " compiled_fhir_paths = [(f[0], fhirpathpy.compile(f[1])) for f in fhir_paths]\n", " self.fhir_paths[resource_type] = compiled_fhir_paths\n", "\n", " def _invoke_request(self):\n", " types = ','.join(self.resource_types)\n", " url = f'{self.base_url}/{self.endpoint}/$export?_type={types}'\n", " print(f'Fetching from {url}')\n", " r = self.session.get(url, headers={'Authorization': f'Bearer {self.get_token()}', 'Accept': 'application/fhir+json', 'Prefer': 'respond-async'})\n", "\n", " self.check_url = r.headers['Content-Location']\n", " return self.check_url\n", "\n", " def _wait_until_ready(self):\n", " while True:\n", " r = self.session.get(self.check_url, headers={'Authorization': f'Bearer {self.get_token()}', 'Accept': 'application/fhir+json'})\n", "\n", " # There are three possible options here: http://hl7.org/fhir/uv/bulkdata/export.html#bulk-data-status-request\n", " # Error = 4xx or 5xx status code\n", " # In-Progress = 202\n", " # Complete = 200\n", "\n", " if r.status_code == 200:\n", " # complete\n", " response = r.json()\n", " self.output_files = response['output']\n", " return self.output_files\n", "\n", " elif r.status_code == 202:\n", " # in progress\n", " delay = r.headers['Retry-After']\n", "\n", " sleep(int(delay))\n", "\n", " else:\n", " raise RuntimeError(r.text)\n", "\n", " def get_dataframes(self):\n", " self._invoke_request()\n", " self._wait_until_ready()\n", "\n", " resources_by_type = {}\n", " self.resources_by_type = {} # Reset store of raw FHIR resources each time this is run\n", "\n", " for output_file in self.output_files:\n", " download_url = output_file['url']\n", " resource_type = output_file['type']\n", "\n", " r = self.session.get(download_url, headers={'Authorization': f'Bearer {get_token()}', 'Accept': 'application/fhir+json'})\n", "\n", " ndjson = r.text.strip()\n", "\n", " if resource_type not in resources_by_type:\n", " resources_by_type[resource_type] = []\n", " self.resources_by_type[resource_type] = []\n", "\n", " for line in ndjson.split('\\n'):\n", " resource = json.loads(line)\n", "\n", " # Make raw resource instances available for future use\n", " self.resources_by_type[resource_type].append(resource)\n", "\n", " if resource_type in self.fhir_paths:\n", " fhir_paths = self.fhir_paths[resource_type]\n", " filtered_resource = {}\n", " for f in fhir_paths:\n", " fieldname = f[0]\n", " func = f[1]\n", " filtered_resource[fieldname] = func(resource)\n", "\n", " if isinstance(filtered_resource[fieldname], list) and len(filtered_resource[fieldname]) == 1:\n", " filtered_resource[fieldname] = filtered_resource[fieldname][0]\n", " resource = filtered_resource\n", "\n", " resources_by_type[resource_type].append(resource)\n", "\n", " dfs = {}\n", "\n", " for resource_type, resources in resources_by_type.items():\n", " dfs[resource_type] = pd.json_normalize(list(map(lambda r: flatten(r), resources)))\n", "\n", " return dfs\n", "\n", " def get_example_resource(self, resource_type: str, resource_id: Optional[str] = None):\n", " if self.resources_by_type is None:\n", " print(\"You need to run get_dataframes() first\")\n", " return None\n", "\n", " if resource_type not in self.resources_by_type:\n", " print(f\"{resource_type} not available. Try one of these: {', '.join(self.resources_by_type.keys())}\")\n", " return None\n", "\n", " if resource_id is None:\n", " return self.resources_by_type[resource_type][0]\n", "\n", " resource = [r for r in self.resources_by_type[resource_type] if r['id'] == resource_id]\n", "\n", " if len(resource) > 0:\n", " return resource[0]\n", "\n", " print(f\"No {resource_type} with id={resource_id} was found.\")\n", " return None\n", "\n", " def reprocess_dataframes(self, fhir_paths):\n", " return BulkDataFetcher._reprocess_dataframes(self.resources_by_type, fhir_paths)\n", "\n", " @classmethod\n", " def _reprocess_dataframes(cls, obj_resources_by_type, user_fhir_paths):\n", " parsed_resources_by_type = defaultdict(list)\n", "\n", " for this_resource_type in obj_resources_by_type.keys():\n", " if this_resource_type in user_fhir_paths:\n", " user_fhir_paths[this_resource_type] = [(f[0], fhirpathpy.compile(f[1])) for f in user_fhir_paths[this_resource_type]]\n", " for resource in obj_resources_by_type[this_resource_type]:\n", " if this_resource_type in user_fhir_paths:\n", " filtered_resource = {}\n", " for f in user_fhir_paths[this_resource_type]:\n", " fieldname = f[0]\n", " func = f[1]\n", " filtered_resource[fieldname] = func(resource)\n", "\n", " if isinstance(filtered_resource[fieldname], list) and len(filtered_resource[fieldname]) == 1:\n", " filtered_resource[fieldname] = filtered_resource[fieldname][0]\n", " parsed_resources_by_type[this_resource_type].append(filtered_resource)\n", " else:\n", " parsed_resources_by_type[this_resource_type].append(resource)\n", "\n", " dfs = {}\n", "\n", " for t, res in parsed_resources_by_type.items():\n", " dfs[t] = pd.json_normalize(list(map(lambda r: flatten(r), res)))\n", "\n", " return dfs" ] }, { "cell_type": "code", "execution_count": 20, "id": "8079a34b", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Fetching from \n",
       "https://bulk-data.smarthealthit.org/eyJlcnIiOiIiLCJwYWdlIjoxMDAwMCwiZHVyIjoxMCwidGx0IjoxNSwibSI6MSwic3R1Ijo0LCJkZWw\n",
       "iOjB9/fhir/Patient/$export?_type=Patient,Condition\n",
       "
\n" ], "text/plain": [ "Fetching from \n", "\u001B[4;94mhttps://bulk-data.smarthealthit.org/eyJlcnIiOiIiLCJwYWdlIjoxMDAwMCwiZHVyIjoxMCwidGx0IjoxNSwibSI6MSwic3R1Ijo0LCJkZWw\u001B[0m\n", "\u001B[4;94miOjB9/fhir/Patient/$export?\u001B[0m\u001B[4;94m_type\u001B[0m\u001B[4;94m=\u001B[0m\u001B[4;94mPatient\u001B[0m\u001B[4;94m,Condition\u001B[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idgenderdate_of_birthmarital_status
06c5d9ca9-54d7-42f5-bfae-a7c19cd217f2male1965-01-13M
158c297c4-d684-4677-8024-01131d93835efemale1971-04-05M
2538a9a4e-8437-47d3-8c01-1a17dca8f0bemale1923-03-24M
3c6c60742-8694-46e4-bb42-b00bf6d8b536female1965-10-27M
4fbfec681-d357-4b28-b1d2-5db6434c7846female1942-07-04M
...............
955efb1ac1-d29b-40a5-a3d1-2d682f10bfa7male1995-10-19S
96c1981741-f90e-4077-9156-429a3c4c5dedmale1956-05-06M
97f98b23bf-4443-46d0-9eaf-563e767cf948male1966-02-07M
98c536dee9-9ef6-4807-ae20-9f1045c9c7d6male1990-11-18S
99a845ead4-d9de-42eb-b4b5-eb21a8963578female1968-04-20S
\n", "

100 rows × 4 columns

\n", "
" ], "text/plain": [ " id gender date_of_birth marital_status\n", "0 6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2 male 1965-01-13 M\n", "1 58c297c4-d684-4677-8024-01131d93835e female 1971-04-05 M\n", "2 538a9a4e-8437-47d3-8c01-1a17dca8f0be male 1923-03-24 M\n", "3 c6c60742-8694-46e4-bb42-b00bf6d8b536 female 1965-10-27 M\n", "4 fbfec681-d357-4b28-b1d2-5db6434c7846 female 1942-07-04 M\n", ".. ... ... ... ...\n", "95 5efb1ac1-d29b-40a5-a3d1-2d682f10bfa7 male 1995-10-19 S\n", "96 c1981741-f90e-4077-9156-429a3c4c5ded male 1956-05-06 M\n", "97 f98b23bf-4443-46d0-9eaf-563e767cf948 male 1966-02-07 M\n", "98 c536dee9-9ef6-4807-ae20-9f1045c9c7d6 male 1990-11-18 S\n", "99 a845ead4-d9de-42eb-b4b5-eb21a8963578 female 1968-04-20 S\n", "\n", "[100 rows x 4 columns]" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# And then to invoke it:\n", "\n", "# create a BulkDataFetcher with our credentials\n", "fetcher = BulkDataFetcher(\n", " base_url=server_url, client_id=client_id, private_key=private_key, key_id=key_id, session=session\n", ")\n", "\n", "# add a resource type of interest, with some FHIRPath field mappings\n", "fetcher.add_resource_type('Patient', [\n", " (\"id\", \"identifier[0].value\"),\n", " (\"gender\", \"gender\"),\n", " (\"date_of_birth\", \"birthDate\"),\n", " (\"marital_status\", \"maritalStatus.coding.first().code\")\n", "])\n", "\n", "# add another resource type, with no FHIRPath mappings (load the entire resource)\n", "fetcher.add_resource_type('Condition')\n", "\n", "dfs = fetcher.get_dataframes()\n", "\n", "dfs['Patient']" ] }, { "cell_type": "code", "execution_count": 21, "id": "64342e0e", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
resourceTypeidclinicalStatus_coding_0_systemclinicalStatus_coding_0_codeverificationStatus_coding_0_systemverificationStatus_coding_0_codecode_coding_0_systemcode_coding_0_codecode_coding_0_displaycode_textsubject_referenceencounter_referenceonsetDateTimerecordedDateabatementDateTime
0Conditiona5a38601-b6fe-46b4-a67e-cde9d5957ddehttp://terminology.hl7.org/CodeSystem/conditio...activehttp://terminology.hl7.org/CodeSystem/conditio...confirmedhttp://snomed.info/sct40055000Chronic sinusitis (disorder)Chronic sinusitis (disorder)Patient/6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2Encounter/17b801ac-58e3-4f6b-8b48-8e33f3a360861985-06-18T17:30:49-04:001985-06-18T17:30:49-04:00NaN
1Condition8f818ad4-c292-47e8-8d99-c4c54174b671http://terminology.hl7.org/CodeSystem/conditio...activehttp://terminology.hl7.org/CodeSystem/conditio...confirmedhttp://snomed.info/sct162864005Body mass index 30+ - obesity (finding)Body mass index 30+ - obesity (finding)Patient/6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2Encounter/0953dd44-90bb-4805-badd-169a761a6ab32005-01-19T16:30:49-05:002005-01-19T16:30:49-05:00NaN
2Condition65d9d5f2-a772-4586-932f-df1f2ce1a863http://terminology.hl7.org/CodeSystem/conditio...activehttp://terminology.hl7.org/CodeSystem/conditio...confirmedhttp://snomed.info/sct15777000PrediabetesPrediabetesPatient/6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2Encounter/d4e1370a-a679-4570-a3dc-e4f7ac8475122013-02-06T16:30:49-05:002013-02-06T16:30:49-05:00NaN
3Condition77ac8342-6950-4302-a303-efba12e06785http://terminology.hl7.org/CodeSystem/conditio...resolvedhttp://terminology.hl7.org/CodeSystem/conditio...confirmedhttp://snomed.info/sct68496003Polyp of colonPolyp of colonPatient/6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2Encounter/58ad433b-3707-4d40-9b63-2a803b4913bd2015-01-14T16:30:49-05:002015-01-14T16:30:49-05:002017-05-03T17:30:49-04:00
4Condition6514ab0c-bc64-4e1b-aa61-b97d27d72bc7http://terminology.hl7.org/CodeSystem/conditio...activehttp://terminology.hl7.org/CodeSystem/conditio...confirmedhttp://snomed.info/sct271737000Anemia (disorder)Anemia (disorder)Patient/6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2Encounter/58ad433b-3707-4d40-9b63-2a803b4913bd2015-01-14T16:30:49-05:002015-01-14T16:30:49-05:00NaN
................................................
634Conditionab051f6c-4298-407b-9315-2322ce913539http://terminology.hl7.org/CodeSystem/conditio...activehttp://terminology.hl7.org/CodeSystem/conditio...confirmedhttp://snomed.info/sct162864005Body mass index 30+ - obesity (finding)Body mass index 30+ - obesity (finding)Patient/a845ead4-d9de-42eb-b4b5-eb21a8963578Encounter/c5ed8aed-2b7e-4630-bd1d-ac5090967edc2014-11-22T15:43:42-05:002014-11-22T15:43:42-05:00NaN
635Condition76c1f07a-f8f2-4705-aa80-5f7a25d7c651http://terminology.hl7.org/CodeSystem/conditio...resolvedhttp://terminology.hl7.org/CodeSystem/conditio...confirmedhttp://snomed.info/sct39848009Whiplash injury to neckWhiplash injury to neckPatient/a845ead4-d9de-42eb-b4b5-eb21a8963578Encounter/9c8b41dd-d6fd-4691-ae46-01b47992dd8d2015-07-13T16:43:42-04:002015-07-13T16:43:42-04:002015-08-10T16:43:42-04:00
636Conditionb9a078eb-bb83-49ed-b4ed-633d1445356dhttp://terminology.hl7.org/CodeSystem/conditio...resolvedhttp://terminology.hl7.org/CodeSystem/conditio...confirmedhttp://snomed.info/sct70704007Sprain of wristSprain of wristPatient/a845ead4-d9de-42eb-b4b5-eb21a8963578Encounter/f044f05a-8433-4952-926d-dd8e2b4ee44e2018-07-25T16:43:42-04:002018-07-25T16:43:42-04:002018-08-15T16:43:42-04:00
637Condition0fe427ce-7ea1-4409-8de1-3879f9dc56bbhttp://terminology.hl7.org/CodeSystem/conditio...resolvedhttp://terminology.hl7.org/CodeSystem/conditio...confirmedhttp://snomed.info/sct444814009Viral sinusitis (disorder)Viral sinusitis (disorder)Patient/a845ead4-d9de-42eb-b4b5-eb21a8963578Encounter/9100e9aa-1206-403b-b2bf-b75ac23991bd2018-09-26T16:43:42-04:002018-09-26T16:43:42-04:002018-10-17T16:43:42-04:00
638Condition88f3f41a-68ec-46dd-8d44-9178d3872220http://terminology.hl7.org/CodeSystem/conditio...resolvedhttp://terminology.hl7.org/CodeSystem/conditio...confirmedhttp://snomed.info/sct72892002Normal pregnancyNormal pregnancyPatient/a845ead4-d9de-42eb-b4b5-eb21a8963578Encounter/2d975caf-e6bf-43c2-8778-ea293df1f2552018-12-22T15:43:42-05:002018-12-22T15:43:42-05:002019-07-27T16:43:42-04:00
\n", "

639 rows × 15 columns

\n", "
" ], "text/plain": [ " resourceType id \\\n", "0 Condition a5a38601-b6fe-46b4-a67e-cde9d5957dde \n", "1 Condition 8f818ad4-c292-47e8-8d99-c4c54174b671 \n", "2 Condition 65d9d5f2-a772-4586-932f-df1f2ce1a863 \n", "3 Condition 77ac8342-6950-4302-a303-efba12e06785 \n", "4 Condition 6514ab0c-bc64-4e1b-aa61-b97d27d72bc7 \n", ".. ... ... \n", "634 Condition ab051f6c-4298-407b-9315-2322ce913539 \n", "635 Condition 76c1f07a-f8f2-4705-aa80-5f7a25d7c651 \n", "636 Condition b9a078eb-bb83-49ed-b4ed-633d1445356d \n", "637 Condition 0fe427ce-7ea1-4409-8de1-3879f9dc56bb \n", "638 Condition 88f3f41a-68ec-46dd-8d44-9178d3872220 \n", "\n", " clinicalStatus_coding_0_system \\\n", "0 http://terminology.hl7.org/CodeSystem/conditio... \n", "1 http://terminology.hl7.org/CodeSystem/conditio... \n", "2 http://terminology.hl7.org/CodeSystem/conditio... \n", "3 http://terminology.hl7.org/CodeSystem/conditio... \n", "4 http://terminology.hl7.org/CodeSystem/conditio... \n", ".. ... \n", "634 http://terminology.hl7.org/CodeSystem/conditio... \n", "635 http://terminology.hl7.org/CodeSystem/conditio... \n", "636 http://terminology.hl7.org/CodeSystem/conditio... \n", "637 http://terminology.hl7.org/CodeSystem/conditio... \n", "638 http://terminology.hl7.org/CodeSystem/conditio... \n", "\n", " clinicalStatus_coding_0_code \\\n", "0 active \n", "1 active \n", "2 active \n", "3 resolved \n", "4 active \n", ".. ... \n", "634 active \n", "635 resolved \n", "636 resolved \n", "637 resolved \n", "638 resolved \n", "\n", " verificationStatus_coding_0_system \\\n", "0 http://terminology.hl7.org/CodeSystem/conditio... \n", "1 http://terminology.hl7.org/CodeSystem/conditio... \n", "2 http://terminology.hl7.org/CodeSystem/conditio... \n", "3 http://terminology.hl7.org/CodeSystem/conditio... \n", "4 http://terminology.hl7.org/CodeSystem/conditio... \n", ".. ... \n", "634 http://terminology.hl7.org/CodeSystem/conditio... \n", "635 http://terminology.hl7.org/CodeSystem/conditio... \n", "636 http://terminology.hl7.org/CodeSystem/conditio... \n", "637 http://terminology.hl7.org/CodeSystem/conditio... \n", "638 http://terminology.hl7.org/CodeSystem/conditio... \n", "\n", " verificationStatus_coding_0_code code_coding_0_system \\\n", "0 confirmed http://snomed.info/sct \n", "1 confirmed http://snomed.info/sct \n", "2 confirmed http://snomed.info/sct \n", "3 confirmed http://snomed.info/sct \n", "4 confirmed http://snomed.info/sct \n", ".. ... ... \n", "634 confirmed http://snomed.info/sct \n", "635 confirmed http://snomed.info/sct \n", "636 confirmed http://snomed.info/sct \n", "637 confirmed http://snomed.info/sct \n", "638 confirmed http://snomed.info/sct \n", "\n", " code_coding_0_code code_coding_0_display \\\n", "0 40055000 Chronic sinusitis (disorder) \n", "1 162864005 Body mass index 30+ - obesity (finding) \n", "2 15777000 Prediabetes \n", "3 68496003 Polyp of colon \n", "4 271737000 Anemia (disorder) \n", ".. ... ... \n", "634 162864005 Body mass index 30+ - obesity (finding) \n", "635 39848009 Whiplash injury to neck \n", "636 70704007 Sprain of wrist \n", "637 444814009 Viral sinusitis (disorder) \n", "638 72892002 Normal pregnancy \n", "\n", " code_text \\\n", "0 Chronic sinusitis (disorder) \n", "1 Body mass index 30+ - obesity (finding) \n", "2 Prediabetes \n", "3 Polyp of colon \n", "4 Anemia (disorder) \n", ".. ... \n", "634 Body mass index 30+ - obesity (finding) \n", "635 Whiplash injury to neck \n", "636 Sprain of wrist \n", "637 Viral sinusitis (disorder) \n", "638 Normal pregnancy \n", "\n", " subject_reference \\\n", "0 Patient/6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2 \n", "1 Patient/6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2 \n", "2 Patient/6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2 \n", "3 Patient/6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2 \n", "4 Patient/6c5d9ca9-54d7-42f5-bfae-a7c19cd217f2 \n", ".. ... \n", "634 Patient/a845ead4-d9de-42eb-b4b5-eb21a8963578 \n", "635 Patient/a845ead4-d9de-42eb-b4b5-eb21a8963578 \n", "636 Patient/a845ead4-d9de-42eb-b4b5-eb21a8963578 \n", "637 Patient/a845ead4-d9de-42eb-b4b5-eb21a8963578 \n", "638 Patient/a845ead4-d9de-42eb-b4b5-eb21a8963578 \n", "\n", " encounter_reference \\\n", "0 Encounter/17b801ac-58e3-4f6b-8b48-8e33f3a36086 \n", "1 Encounter/0953dd44-90bb-4805-badd-169a761a6ab3 \n", "2 Encounter/d4e1370a-a679-4570-a3dc-e4f7ac847512 \n", "3 Encounter/58ad433b-3707-4d40-9b63-2a803b4913bd \n", "4 Encounter/58ad433b-3707-4d40-9b63-2a803b4913bd \n", ".. ... \n", "634 Encounter/c5ed8aed-2b7e-4630-bd1d-ac5090967edc \n", "635 Encounter/9c8b41dd-d6fd-4691-ae46-01b47992dd8d \n", "636 Encounter/f044f05a-8433-4952-926d-dd8e2b4ee44e \n", "637 Encounter/9100e9aa-1206-403b-b2bf-b75ac23991bd \n", "638 Encounter/2d975caf-e6bf-43c2-8778-ea293df1f255 \n", "\n", " onsetDateTime recordedDate \\\n", "0 1985-06-18T17:30:49-04:00 1985-06-18T17:30:49-04:00 \n", "1 2005-01-19T16:30:49-05:00 2005-01-19T16:30:49-05:00 \n", "2 2013-02-06T16:30:49-05:00 2013-02-06T16:30:49-05:00 \n", "3 2015-01-14T16:30:49-05:00 2015-01-14T16:30:49-05:00 \n", "4 2015-01-14T16:30:49-05:00 2015-01-14T16:30:49-05:00 \n", ".. ... ... \n", "634 2014-11-22T15:43:42-05:00 2014-11-22T15:43:42-05:00 \n", "635 2015-07-13T16:43:42-04:00 2015-07-13T16:43:42-04:00 \n", "636 2018-07-25T16:43:42-04:00 2018-07-25T16:43:42-04:00 \n", "637 2018-09-26T16:43:42-04:00 2018-09-26T16:43:42-04:00 \n", "638 2018-12-22T15:43:42-05:00 2018-12-22T15:43:42-05:00 \n", "\n", " abatementDateTime \n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 2017-05-03T17:30:49-04:00 \n", "4 NaN \n", ".. ... \n", "634 NaN \n", "635 2015-08-10T16:43:42-04:00 \n", "636 2018-08-15T16:43:42-04:00 \n", "637 2018-10-17T16:43:42-04:00 \n", "638 2019-07-27T16:43:42-04:00 \n", "\n", "[639 rows x 15 columns]" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dfs['Condition']" ] }, { "cell_type": "markdown", "id": "79856cfe", "metadata": {}, "source": [ "## Group export\n", "\n", "[§170.315(g)(10) Standardized API for patient and population services](https://www.healthit.gov/test-method/standardized-api-patient-and-population-services) requires [`group-export`](https://hl7.org/fhir/uv/bulkdata/OperationDefinition-group-export.html) as of December 2022.\n", "\n", "This is therefore the FHIR Bulk Data endpoint you are likely to find in EHRs.\n", "\n", "To use this endpoint, you will need the ID of the group of patients you want to export. In a production setting, this would typically be provided by the administrators of the EHR.\n", "\n", "For the `bulk-data.smarthealthit.org` testing server, we can ask it for a list of groups via the FHIR API:" ] }, { "cell_type": "code", "execution_count": 22, "id": "3c69b5bb", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'resourceType': 'Bundle',\n", " 'id': 'e21c6a557591d81d35d3f3bae22e6b490c71ad36b1b75b163393ea102e47eae8',\n", " 'meta': {'lastUpdated': '2023-06-13 01:26:34'},\n", " 'type': 'searchset',\n", " 'total': 8,\n", " 'link': [{'relation': 'self',\n", " 'url': 'https://bulk-data.smarthealthit.org/fhir/Group'}],\n", " 'entry': [{'fullUrl': 'https://bulk-data.smarthealthit.org/fhir/Group/1f76e2b7-a222-4765-9097-a71b86e90d07',\n", " 'resource': {'resourceType': 'Group',\n", " 'id': '1f76e2b7-a222-4765-9097-a71b86e90d07',\n", " 'identifier': [{'system': 'https://bulk-data/db-id',\n", " 'value': '1f76e2b7-a222-4765-9097-a71b86e90d07'}],\n", " 'quantity': 25,\n", " 'name': 'Health New England',\n", " 'text': {'status': 'generated',\n", " 'div': '
Health New England
'},\n", " 'type': 'person',\n", " 'actual': True}},\n", " {'fullUrl': 'https://bulk-data.smarthealthit.org/fhir/Group/84e2fc85-2b9b-4680-b7df-cfbc2ea7a12b',\n", " 'resource': {'resourceType': 'Group',\n", " 'id': '84e2fc85-2b9b-4680-b7df-cfbc2ea7a12b',\n", " 'identifier': [{'system': 'https://bulk-data/db-id',\n", " 'value': '84e2fc85-2b9b-4680-b7df-cfbc2ea7a12b'}],\n", " 'quantity': 3,\n", " 'name': 'Minuteman Health',\n", " 'text': {'status': 'generated',\n", " 'div': '
Minuteman Health
'},\n", " 'type': 'person',\n", " 'actual': True}},\n", " {'fullUrl': 'https://bulk-data.smarthealthit.org/fhir/Group/a1f090cb-ffd1-436d-a815-fb047d9a1903',\n", " 'resource': {'resourceType': 'Group',\n", " 'id': 'a1f090cb-ffd1-436d-a815-fb047d9a1903',\n", " 'identifier': [{'system': 'https://bulk-data/db-id',\n", " 'value': 'a1f090cb-ffd1-436d-a815-fb047d9a1903'}],\n", " 'quantity': 10,\n", " 'name': 'BMC HealthNet',\n", " 'text': {'status': 'generated',\n", " 'div': '
BMC HealthNet
'},\n", " 'type': 'person',\n", " 'actual': True}},\n", " {'fullUrl': 'https://bulk-data.smarthealthit.org/fhir/Group/a95907b4-0c41-462a-bfcf-cb822075eb39',\n", " 'resource': {'resourceType': 'Group',\n", " 'id': 'a95907b4-0c41-462a-bfcf-cb822075eb39',\n", " 'identifier': [{'system': 'https://bulk-data/db-id',\n", " 'value': 'a95907b4-0c41-462a-bfcf-cb822075eb39'}],\n", " 'quantity': 3,\n", " 'name': 'Harvard Pilgrim Health Care',\n", " 'text': {'status': 'generated',\n", " 'div': '
Harvard Pilgrim Health Care
'},\n", " 'type': 'person',\n", " 'actual': True}},\n", " {'fullUrl': 'https://bulk-data.smarthealthit.org/fhir/Group/ae6ad3d7-f19d-44d7-9e70-fd0b7cf915e7',\n", " 'resource': {'resourceType': 'Group',\n", " 'id': 'ae6ad3d7-f19d-44d7-9e70-fd0b7cf915e7',\n", " 'identifier': [{'system': 'https://bulk-data/db-id',\n", " 'value': 'ae6ad3d7-f19d-44d7-9e70-fd0b7cf915e7'}],\n", " 'quantity': 22,\n", " 'name': 'Tufts Health Plan',\n", " 'text': {'status': 'generated',\n", " 'div': '
Tufts Health Plan
'},\n", " 'type': 'person',\n", " 'actual': True}},\n", " {'fullUrl': 'https://bulk-data.smarthealthit.org/fhir/Group/b058c5e7-209c-4162-9289-0ff703347c0f',\n", " 'resource': {'resourceType': 'Group',\n", " 'id': 'b058c5e7-209c-4162-9289-0ff703347c0f',\n", " 'identifier': [{'system': 'https://bulk-data/db-id',\n", " 'value': 'b058c5e7-209c-4162-9289-0ff703347c0f'}],\n", " 'quantity': 3,\n", " 'name': 'Fallon Health',\n", " 'text': {'status': 'generated',\n", " 'div': '
Fallon Health
'},\n", " 'type': 'person',\n", " 'actual': True}},\n", " {'fullUrl': 'https://bulk-data.smarthealthit.org/fhir/Group/cf04e363-eef4-4653-9650-846bca43f357',\n", " 'resource': {'resourceType': 'Group',\n", " 'id': 'cf04e363-eef4-4653-9650-846bca43f357',\n", " 'identifier': [{'system': 'https://bulk-data/db-id',\n", " 'value': 'cf04e363-eef4-4653-9650-846bca43f357'}],\n", " 'quantity': 7,\n", " 'name': 'Neighborhood Health Plan',\n", " 'text': {'status': 'generated',\n", " 'div': '
Neighborhood Health Plan
'},\n", " 'type': 'person',\n", " 'actual': True}},\n", " {'fullUrl': 'https://bulk-data.smarthealthit.org/fhir/Group/ff7dc35f-79e9-47a0-af22-475cf301a085',\n", " 'resource': {'resourceType': 'Group',\n", " 'id': 'ff7dc35f-79e9-47a0-af22-475cf301a085',\n", " 'identifier': [{'system': 'https://bulk-data/db-id',\n", " 'value': 'ff7dc35f-79e9-47a0-af22-475cf301a085'}],\n", " 'quantity': 27,\n", " 'name': 'Blue Cross Blue Shield',\n", " 'text': {'status': 'generated',\n", " 'div': '
Blue Cross Blue Shield
'},\n", " 'type': 'person',\n", " 'actual': True}}]}" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "r = session.get(f'{server_url}/Group', headers={'Authorization': f'Bearer {get_token()}', 'Accept': 'application/fhir+json'})\n", "r.json()" ] }, { "cell_type": "markdown", "id": "8ff95bca", "metadata": {}, "source": [ "Let's quickly pull this into a Pandas DataFrame to make it easier to read:" ] }, { "cell_type": "code", "execution_count": 23, "id": "7d9ab479", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
resource.idresource.nameresource.quantity
01f76e2b7-a222-4765-9097-a71b86e90d07Health New England25
184e2fc85-2b9b-4680-b7df-cfbc2ea7a12bMinuteman Health3
2a1f090cb-ffd1-436d-a815-fb047d9a1903BMC HealthNet10
3a95907b4-0c41-462a-bfcf-cb822075eb39Harvard Pilgrim Health Care3
4ae6ad3d7-f19d-44d7-9e70-fd0b7cf915e7Tufts Health Plan22
5b058c5e7-209c-4162-9289-0ff703347c0fFallon Health3
6cf04e363-eef4-4653-9650-846bca43f357Neighborhood Health Plan7
7ff7dc35f-79e9-47a0-af22-475cf301a085Blue Cross Blue Shield27
\n", "
" ], "text/plain": [ " resource.id resource.name \\\n", "0 1f76e2b7-a222-4765-9097-a71b86e90d07 Health New England \n", "1 84e2fc85-2b9b-4680-b7df-cfbc2ea7a12b Minuteman Health \n", "2 a1f090cb-ffd1-436d-a815-fb047d9a1903 BMC HealthNet \n", "3 a95907b4-0c41-462a-bfcf-cb822075eb39 Harvard Pilgrim Health Care \n", "4 ae6ad3d7-f19d-44d7-9e70-fd0b7cf915e7 Tufts Health Plan \n", "5 b058c5e7-209c-4162-9289-0ff703347c0f Fallon Health \n", "6 cf04e363-eef4-4653-9650-846bca43f357 Neighborhood Health Plan \n", "7 ff7dc35f-79e9-47a0-af22-475cf301a085 Blue Cross Blue Shield \n", "\n", " resource.quantity \n", "0 25 \n", "1 3 \n", "2 10 \n", "3 3 \n", "4 22 \n", "5 3 \n", "6 7 \n", "7 27 " ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "groups = pd.json_normalize(r.json()['entry'])[['resource.id', 'resource.name', 'resource.quantity']]\n", "groups" ] }, { "cell_type": "markdown", "id": "f482a614", "metadata": {}, "source": [ "Now we can request the patients and associated data for a specific group:" ] }, { "cell_type": "code", "execution_count": 24, "id": "01c27e11", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Fetching from \n",
       "https://bulk-data.smarthealthit.org/eyJlcnIiOiIiLCJwYWdlIjoxMDAwMCwiZHVyIjoxMCwidGx0IjoxNSwibSI6MSwic3R1Ijo0LCJkZWw\n",
       "iOjB9/fhir/Group/1f76e2b7-a222-4765-9097-a71b86e90d07/$export?_type=Patient,Condition\n",
       "
\n" ], "text/plain": [ "Fetching from \n", "\u001B[4;94mhttps://bulk-data.smarthealthit.org/eyJlcnIiOiIiLCJwYWdlIjoxMDAwMCwiZHVyIjoxMCwidGx0IjoxNSwibSI6MSwic3R1Ijo0LCJkZWw\u001B[0m\n", "\u001B[4;94miOjB9/fhir/Group/1f76e2b7-a222-4765-9097-a71b86e90d07/$export?\u001B[0m\u001B[4;94m_type\u001B[0m\u001B[4;94m=\u001B[0m\u001B[4;94mPatient\u001B[0m\u001B[4;94m,Condition\u001B[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idgenderdate_of_birthmarital_status
0fbfec681-d357-4b28-b1d2-5db6434c7846female1942-07-04M
10b8a6ef0-07c8-48ca-804d-1e64f6e44b95female1967-10-24S
262e03ae7-079c-4eda-9b5a-29440d3a015amale2013-05-25S
37cdaae04-4ce7-4a6d-b6a3-5cddba9bc888female1959-07-23M
484d4bafe-1891-4e6c-b8aa-38d2eafd8193female2000-09-12S
54bc3ef6a-65c5-470d-8911-f26194b2a0e3male2004-08-20S
63ca9f003-e6dd-4110-b4c2-12c056b880f4female1991-12-27M
7daf4e787-0ea5-45ff-a9a1-c68308e9f6a3male1995-06-21S
81ad52ff0-428a-4048-aff8-f7196a2da649female1996-09-11S
9644d85af-aaf9-4068-ad23-1e55aedd5205male2003-09-12S
10687eb477-32ae-44ac-a0ef-2912623a14fffemale1960-10-18M
1170ac5078-22ef-471d-bed7-cb694775b4bafemale2001-08-25S
127646ecba-4812-452e-88e7-6235f77dabb2female1997-11-20S
1369071541-e760-4d0c-bf8a-961a061cb0d5male1952-02-23M
14221fe1ec-a258-4fc4-8cc8-b7c960a8a0a9female2008-01-04S
1532e46528-35d1-4ed7-9aaa-09ae00f9681cmale2007-02-12S
16b20c7c80-49ac-4926-8b03-e9c69b40e1f5male1951-04-26S
17733abdda-2bfa-485f-9c83-ed9b206889b2male1972-02-27M
1855940999-fd98-4922-b9bc-a6bf0c1855edmale1931-04-18M
197ba8d35f-3f70-48b9-b711-104374136ac7male2002-05-10S
20ff9d23d8-f3c8-4eee-a5f9-e05e843675b5female1993-02-09S
218d3e1155-278a-4824-a7e0-fddb24c7c179male1991-10-27S
228c9fea57-6ded-47b0-88c9-75518430b572female1962-08-01M
23d2524ab6-4db9-440d-b588-6dcfcab89270male1979-11-11S
24f98b23bf-4443-46d0-9eaf-563e767cf948male1966-02-07M
\n", "
" ], "text/plain": [ " id gender date_of_birth marital_status\n", "0 fbfec681-d357-4b28-b1d2-5db6434c7846 female 1942-07-04 M\n", "1 0b8a6ef0-07c8-48ca-804d-1e64f6e44b95 female 1967-10-24 S\n", "2 62e03ae7-079c-4eda-9b5a-29440d3a015a male 2013-05-25 S\n", "3 7cdaae04-4ce7-4a6d-b6a3-5cddba9bc888 female 1959-07-23 M\n", "4 84d4bafe-1891-4e6c-b8aa-38d2eafd8193 female 2000-09-12 S\n", "5 4bc3ef6a-65c5-470d-8911-f26194b2a0e3 male 2004-08-20 S\n", "6 3ca9f003-e6dd-4110-b4c2-12c056b880f4 female 1991-12-27 M\n", "7 daf4e787-0ea5-45ff-a9a1-c68308e9f6a3 male 1995-06-21 S\n", "8 1ad52ff0-428a-4048-aff8-f7196a2da649 female 1996-09-11 S\n", "9 644d85af-aaf9-4068-ad23-1e55aedd5205 male 2003-09-12 S\n", "10 687eb477-32ae-44ac-a0ef-2912623a14ff female 1960-10-18 M\n", "11 70ac5078-22ef-471d-bed7-cb694775b4ba female 2001-08-25 S\n", "12 7646ecba-4812-452e-88e7-6235f77dabb2 female 1997-11-20 S\n", "13 69071541-e760-4d0c-bf8a-961a061cb0d5 male 1952-02-23 M\n", "14 221fe1ec-a258-4fc4-8cc8-b7c960a8a0a9 female 2008-01-04 S\n", "15 32e46528-35d1-4ed7-9aaa-09ae00f9681c male 2007-02-12 S\n", "16 b20c7c80-49ac-4926-8b03-e9c69b40e1f5 male 1951-04-26 S\n", "17 733abdda-2bfa-485f-9c83-ed9b206889b2 male 1972-02-27 M\n", "18 55940999-fd98-4922-b9bc-a6bf0c1855ed male 1931-04-18 M\n", "19 7ba8d35f-3f70-48b9-b711-104374136ac7 male 2002-05-10 S\n", "20 ff9d23d8-f3c8-4eee-a5f9-e05e843675b5 female 1993-02-09 S\n", "21 8d3e1155-278a-4824-a7e0-fddb24c7c179 male 1991-10-27 S\n", "22 8c9fea57-6ded-47b0-88c9-75518430b572 female 1962-08-01 M\n", "23 d2524ab6-4db9-440d-b588-6dcfcab89270 male 1979-11-11 S\n", "24 f98b23bf-4443-46d0-9eaf-563e767cf948 male 1966-02-07 M" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "group_id = groups.loc[0, 'resource.id']\n", "\n", "fetcher = BulkDataFetcher(\n", " base_url=server_url, client_id=client_id, private_key=private_key, key_id=key_id, session=session,\n", "\n", " # Tell the BulkDataFetcher to request data from the specified group rather than all patients\n", " endpoint=f'Group/{group_id}'\n", ")\n", "\n", "# add a resource type of interest, with some FHIRPath field mappings\n", "fetcher.add_resource_type('Patient', [\n", " (\"id\", \"identifier[0].value\"),\n", " (\"gender\", \"gender\"),\n", " (\"date_of_birth\", \"birthDate\"),\n", " (\"marital_status\", \"maritalStatus.coding.first().code\")\n", "])\n", "\n", "# add another resource type, with no FHIRPath mappings (load the entire resource)\n", "fetcher.add_resource_type('Condition')\n", "\n", "dfs = fetcher.get_dataframes()\n", "\n", "dfs['Patient']" ] }, { "cell_type": "markdown", "id": "e5a1a09f", "metadata": {}, "source": [ "A number of different FHIR resources are available from the test server:\n", "\n", "- [AllergyIntolerance](https://hl7.org/fhir/R4/allergyintolerance.html)\n", "- [CarePlan](https://hl7.org/fhir/R4/careplan.html)\n", "- [CareTeam](https://hl7.org/fhir/R4/careteam.html)\n", "- [Claim](https://hl7.org/fhir/R4/claim.html)\n", "- [Condition](https://hl7.org/fhir/R4/condition.html)\n", "- [Device](https://hl7.org/fhir/R4/device.html)\n", "- [DiagnosticReport](https://hl7.org/fhir/R4/diagnosticreport.html)\n", "- [DocumentReference](https://hl7.org/fhir/R4/documentreference.html)\n", "- [Encounter](https://hl7.org/fhir/R4/encounter.html)\n", "- [ExplanationOfBenefit](https://hl7.org/fhir/R4/explanationofbenefit.html)\n", "- [ImagingStudy](https://hl7.org/fhir/R4/imagingstudy.html)\n", "- [Immunization](https://hl7.org/fhir/R4/immunization.html)\n", "- [MedicationRequest](https://hl7.org/fhir/R4/medicationrequest.html)\n", "- [Observation](https://hl7.org/fhir/R4/observation.html)\n", "- [Patient](https://hl7.org/fhir/R4/patient.html)\n", "- [Procedure](https://hl7.org/fhir/R4/procedure.html)\n", "\n", "Try modifying the request above to pull in resource types other than `Patient` and `Condition`. The links above go to the FHIR documentation for each resource type, which can help with constructing FHIRPaths." ] }, { "cell_type": "code", "execution_count": 25, "id": "a130f585", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Fetching from \n",
       "https://bulk-data.smarthealthit.org/eyJlcnIiOiIiLCJwYWdlIjoxMDAwMCwiZHVyIjoxMCwidGx0IjoxNSwibSI6MSwic3R1Ijo0LCJkZWw\n",
       "iOjB9/fhir/Group/1f76e2b7-a222-4765-9097-a71b86e90d07/$export?_type=Patient,Condition,Observation\n",
       "
\n" ], "text/plain": [ "Fetching from \n", "\u001B[4;94mhttps://bulk-data.smarthealthit.org/eyJlcnIiOiIiLCJwYWdlIjoxMDAwMCwiZHVyIjoxMCwidGx0IjoxNSwibSI6MSwic3R1Ijo0LCJkZWw\u001B[0m\n", "\u001B[4;94miOjB9/fhir/Group/1f76e2b7-a222-4765-9097-a71b86e90d07/$export?\u001B[0m\u001B[4;94m_type\u001B[0m\u001B[4;94m=\u001B[0m\u001B[4;94mPatient\u001B[0m\u001B[4;94m,Condition,Observation\u001B[0m\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
resourceTypeidstatuscategory_0_coding_0_systemcategory_0_coding_0_codecategory_0_coding_0_displaycode_coding_0_systemcode_coding_0_codecode_coding_0_displaycode_text...component_1_valueQuantity_systemcomponent_1_valueQuantity_codevalueCodeableConcept_coding_0_systemvalueCodeableConcept_coding_0_codevalueCodeableConcept_coding_0_displayvalueCodeableConcept_textcode_coding_1_systemcode_coding_1_codecode_coding_1_displayvalueString
0Observation7a10d1ef-97f2-468b-b3ab-78dc99c65cf6finalhttp://terminology.hl7.org/CodeSystem/observat...vital-signsvital-signshttp://loinc.org8302-2Body HeightBody Height...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
1Observation168c33d9-b4c8-4783-b97a-82fee9625f35finalhttp://terminology.hl7.org/CodeSystem/observat...vital-signsvital-signshttp://loinc.org72514-3Pain severity - 0-10 verbal numeric rating [Sc...Pain severity - 0-10 verbal numeric rating [Sc......NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
2Observation7a6ee07e-72d0-4a74-896f-13c1b2e9b254finalhttp://terminology.hl7.org/CodeSystem/observat...vital-signsvital-signshttp://loinc.org29463-7Body WeightBody Weight...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
3Observationfe4863c9-ac4e-439d-9d8f-834f1aebd3d8finalhttp://terminology.hl7.org/CodeSystem/observat...vital-signsvital-signshttp://loinc.org39156-5Body Mass IndexBody Mass Index...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
4Observationa7e89580-6be8-4b0a-a528-dfb91f2508bffinalhttp://terminology.hl7.org/CodeSystem/observat...vital-signsvital-signshttp://loinc.org85354-9Blood PressureBlood Pressure...http://unitsofmeasure.orgmm[Hg]NaNNaNNaNNaNNaNNaNNaNNaN
..................................................................
4024Observation6f4249dd-abeb-43dd-b388-407fff8ea4f8finalhttp://terminology.hl7.org/CodeSystem/observat...laboratorylaboratoryhttp://loinc.org1920-8Aspartate aminotransferase [Enzymatic activity...Aspartate aminotransferase [Enzymatic activity......NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
4025Observationa04bc225-ad1c-4235-b792-1abde0cb5055finalhttp://terminology.hl7.org/CodeSystem/observat...laboratorylaboratoryhttp://loinc.org2093-3Total CholesterolTotal Cholesterol...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
4026Observation1b26f91e-68d2-4630-a1e0-d2aba21ed4f6finalhttp://terminology.hl7.org/CodeSystem/observat...laboratorylaboratoryhttp://loinc.org2571-8TriglyceridesTriglycerides...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
4027Observatione43ed8c1-0535-407a-abd0-41dd09be6d4afinalhttp://terminology.hl7.org/CodeSystem/observat...laboratorylaboratoryhttp://loinc.org18262-6Low Density Lipoprotein CholesterolLow Density Lipoprotein Cholesterol...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
4028Observationb8c6b918-95b6-43f9-9524-78371a931384finalhttp://terminology.hl7.org/CodeSystem/observat...laboratorylaboratoryhttp://loinc.org2085-9High Density Lipoprotein CholesterolHigh Density Lipoprotein Cholesterol...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
\n", "

4029 rows × 42 columns

\n", "
" ], "text/plain": [ " resourceType id status \\\n", "0 Observation 7a10d1ef-97f2-468b-b3ab-78dc99c65cf6 final \n", "1 Observation 168c33d9-b4c8-4783-b97a-82fee9625f35 final \n", "2 Observation 7a6ee07e-72d0-4a74-896f-13c1b2e9b254 final \n", "3 Observation fe4863c9-ac4e-439d-9d8f-834f1aebd3d8 final \n", "4 Observation a7e89580-6be8-4b0a-a528-dfb91f2508bf final \n", "... ... ... ... \n", "4024 Observation 6f4249dd-abeb-43dd-b388-407fff8ea4f8 final \n", "4025 Observation a04bc225-ad1c-4235-b792-1abde0cb5055 final \n", "4026 Observation 1b26f91e-68d2-4630-a1e0-d2aba21ed4f6 final \n", "4027 Observation e43ed8c1-0535-407a-abd0-41dd09be6d4a final \n", "4028 Observation b8c6b918-95b6-43f9-9524-78371a931384 final \n", "\n", " category_0_coding_0_system \\\n", "0 http://terminology.hl7.org/CodeSystem/observat... \n", "1 http://terminology.hl7.org/CodeSystem/observat... \n", "2 http://terminology.hl7.org/CodeSystem/observat... \n", "3 http://terminology.hl7.org/CodeSystem/observat... \n", "4 http://terminology.hl7.org/CodeSystem/observat... \n", "... ... \n", "4024 http://terminology.hl7.org/CodeSystem/observat... \n", "4025 http://terminology.hl7.org/CodeSystem/observat... \n", "4026 http://terminology.hl7.org/CodeSystem/observat... \n", "4027 http://terminology.hl7.org/CodeSystem/observat... \n", "4028 http://terminology.hl7.org/CodeSystem/observat... \n", "\n", " category_0_coding_0_code category_0_coding_0_display \\\n", "0 vital-signs vital-signs \n", "1 vital-signs vital-signs \n", "2 vital-signs vital-signs \n", "3 vital-signs vital-signs \n", "4 vital-signs vital-signs \n", "... ... ... \n", "4024 laboratory laboratory \n", "4025 laboratory laboratory \n", "4026 laboratory laboratory \n", "4027 laboratory laboratory \n", "4028 laboratory laboratory \n", "\n", " code_coding_0_system code_coding_0_code \\\n", "0 http://loinc.org 8302-2 \n", "1 http://loinc.org 72514-3 \n", "2 http://loinc.org 29463-7 \n", "3 http://loinc.org 39156-5 \n", "4 http://loinc.org 85354-9 \n", "... ... ... \n", "4024 http://loinc.org 1920-8 \n", "4025 http://loinc.org 2093-3 \n", "4026 http://loinc.org 2571-8 \n", "4027 http://loinc.org 18262-6 \n", "4028 http://loinc.org 2085-9 \n", "\n", " code_coding_0_display \\\n", "0 Body Height \n", "1 Pain severity - 0-10 verbal numeric rating [Sc... \n", "2 Body Weight \n", "3 Body Mass Index \n", "4 Blood Pressure \n", "... ... \n", "4024 Aspartate aminotransferase [Enzymatic activity... \n", "4025 Total Cholesterol \n", "4026 Triglycerides \n", "4027 Low Density Lipoprotein Cholesterol \n", "4028 High Density Lipoprotein Cholesterol \n", "\n", " code_text ... \\\n", "0 Body Height ... \n", "1 Pain severity - 0-10 verbal numeric rating [Sc... ... \n", "2 Body Weight ... \n", "3 Body Mass Index ... \n", "4 Blood Pressure ... \n", "... ... ... \n", "4024 Aspartate aminotransferase [Enzymatic activity... ... \n", "4025 Total Cholesterol ... \n", "4026 Triglycerides ... \n", "4027 Low Density Lipoprotein Cholesterol ... \n", "4028 High Density Lipoprotein Cholesterol ... \n", "\n", " component_1_valueQuantity_system component_1_valueQuantity_code \\\n", "0 NaN NaN \n", "1 NaN NaN \n", "2 NaN NaN \n", "3 NaN NaN \n", "4 http://unitsofmeasure.org mm[Hg] \n", "... ... ... \n", "4024 NaN NaN \n", "4025 NaN NaN \n", "4026 NaN NaN \n", "4027 NaN NaN \n", "4028 NaN NaN \n", "\n", " valueCodeableConcept_coding_0_system valueCodeableConcept_coding_0_code \\\n", "0 NaN NaN \n", "1 NaN NaN \n", "2 NaN NaN \n", "3 NaN NaN \n", "4 NaN NaN \n", "... ... ... \n", "4024 NaN NaN \n", "4025 NaN NaN \n", "4026 NaN NaN \n", "4027 NaN NaN \n", "4028 NaN NaN \n", "\n", " valueCodeableConcept_coding_0_display valueCodeableConcept_text \\\n", "0 NaN NaN \n", "1 NaN NaN \n", "2 NaN NaN \n", "3 NaN NaN \n", "4 NaN NaN \n", "... ... ... \n", "4024 NaN NaN \n", "4025 NaN NaN \n", "4026 NaN NaN \n", "4027 NaN NaN \n", "4028 NaN NaN \n", "\n", " code_coding_1_system code_coding_1_code code_coding_1_display valueString \n", "0 NaN NaN NaN NaN \n", "1 NaN NaN NaN NaN \n", "2 NaN NaN NaN NaN \n", "3 NaN NaN NaN NaN \n", "4 NaN NaN NaN NaN \n", "... ... ... ... ... \n", "4024 NaN NaN NaN NaN \n", "4025 NaN NaN NaN NaN \n", "4026 NaN NaN NaN NaN \n", "4027 NaN NaN NaN NaN \n", "4028 NaN NaN NaN NaN \n", "\n", "[4029 rows x 42 columns]" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Try adding an additional resources\n", "fetcher.add_resource_type('Observation')\n", "\n", "dfs = fetcher.get_dataframes()\n", "\n", "dfs['Observation']" ] }, { "cell_type": "code", "execution_count": 26, "id": "81707814", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idpatienttypetype_displaycodecode_display
1551202a40-c62e-4727-bfef-b643869d0951Patient/fbfec681-d357-4b28-b1d2-5db6434c784672166-2Tobacco smoking status NHIS8517006Former smoker
3204cc91b0-4fd0-4244-9738-32c563116308Patient/fbfec681-d357-4b28-b1d2-5db6434c784672166-2Tobacco smoking status NHIS8517006Former smoker
5387cc17d0-3914-4e95-9344-b76e394dfae6Patient/fbfec681-d357-4b28-b1d2-5db6434c784672166-2Tobacco smoking status NHIS8517006Former smoker
81b10d3da1-249e-406d-90cb-9ac19314f823Patient/fbfec681-d357-4b28-b1d2-5db6434c784672166-2Tobacco smoking status NHIS8517006Former smoker
98a0e5b7d8-30a2-4952-ae7a-c0a8d49bc9f3Patient/fbfec681-d357-4b28-b1d2-5db6434c784672166-2Tobacco smoking status NHIS8517006Former smoker
119f843b2e7-6caf-4284-aa90-51c2d55c3c51Patient/fbfec681-d357-4b28-b1d2-5db6434c784672166-2Tobacco smoking status NHIS8517006Former smoker
136d070c5c5-2a0a-4fe9-ba57-0b7fde240f3bPatient/fbfec681-d357-4b28-b1d2-5db6434c784672166-2Tobacco smoking status NHIS8517006Former smoker
15364beef69-dd3c-46a1-b970-8a5dc95be927Patient/fbfec681-d357-4b28-b1d2-5db6434c784672166-2Tobacco smoking status NHIS8517006Former smoker
185b120c744-e1ec-49b9-b8b0-2a54455f410cPatient/fbfec681-d357-4b28-b1d2-5db6434c784672166-2Tobacco smoking status NHIS8517006Former smoker
202984b970a-b027-4778-a820-7f6c56761125Patient/fbfec681-d357-4b28-b1d2-5db6434c784672166-2Tobacco smoking status NHIS8517006Former smoker
21317378ae3-f3b7-418b-b8e1-7af2856a5eacPatient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b9572166-2Tobacco smoking status NHIS266919005Never smoker
221ec100afb-6dad-47e9-8a58-2ea28c1a5fb6Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b9572166-2Tobacco smoking status NHIS266919005Never smoker
233bdf0d1b0-d405-4dbe-acc3-855d6c38d0d2Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b9572166-2Tobacco smoking status NHIS266919005Never smoker
2528b4c3ebb-0f3b-4a9f-aca6-894df17a652dPatient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b9572166-2Tobacco smoking status NHIS266919005Never smoker
260db972319-bfa5-4d88-a562-3e71e94bafc1Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b9572166-2Tobacco smoking status NHIS266919005Never smoker
27388515b19-a75f-486e-af88-96222bd4ca18Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b9572166-2Tobacco smoking status NHIS266919005Never smoker
2812da712dc-da92-4f89-a36e-3aa8fc714d73Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b9572166-2Tobacco smoking status NHIS266919005Never smoker
289219df89c-b801-4d3d-bfc0-7b270ad22542Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b9572166-2Tobacco smoking status NHIS266919005Never smoker
312b99b0288-8c94-4126-a7e0-d1c8717e51a3Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b9572166-2Tobacco smoking status NHIS266919005Never smoker
33218e82af6-9bb3-4657-8ee6-b02bdb288476Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
34145e97461-0051-4bf5-82df-a8c4a429f099Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
350e84acd03-8ae3-4b82-95cf-25799c7c94b8Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
359f3d3151e-9a5b-4227-9b28-89f9cb61ca90Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
368c3364946-7957-4a88-8912-7dc8a9a1cfcdPatient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
377cb400c7f-485e-4674-a3d8-4700543235aePatient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
398450e1f51-8765-431c-a795-40206ac5c9faPatient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
4254ab3823f-27c4-466c-ae57-16e914454522Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
43423c9d6e1-71bf-4766-96df-65a3b0d25d05Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
445187e50b4-ac94-48a1-b052-02b5549be600Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
456a0e8bef3-8317-4930-858d-8dff3830ff13Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
46830c32d11-4e9a-4feb-b352-2196792bb2cfPatient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
4790145b24c-12c5-4ef2-a437-3efa9006f498Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
4882b444657-c019-4434-a9d4-3e0f2f7cdd2fPatient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
50973b8dc8c-91f8-41dd-a6ba-981c6ed12316Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
518c83226df-6f61-4849-bcce-1eec0f56be74Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a72166-2Tobacco smoking status NHIS266919005Never smoker
534683a4889-8ee5-4f5b-befb-35ec22b13107Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc88872166-2Tobacco smoking status NHIS266919005Never smoker
5622735a667-d2cd-4491-9a32-0758ce0f48e9Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc88872166-2Tobacco smoking status NHIS266919005Never smoker
583f86585e1-ec12-4125-a254-2533ea3b9bbaPatient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc88872166-2Tobacco smoking status NHIS266919005Never smoker
600495b7f39-53b4-4d98-884d-fe14e464a13aPatient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc88872166-2Tobacco smoking status NHIS266919005Never smoker
63712ea697f-c2b3-4539-80c7-2ed09c84da2bPatient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc88872166-2Tobacco smoking status NHIS266919005Never smoker
6583a4f53d6-3c82-4efb-b891-d6a866949e58Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc88872166-2Tobacco smoking status NHIS266919005Never smoker
675fb5ed249-16e5-4379-85e7-6b8300ee9548Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc88872166-2Tobacco smoking status NHIS266919005Never smoker
723ee6a13b1-b6d1-49ab-8eff-128dc7f40dafPatient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc88872166-2Tobacco smoking status NHIS266919005Never smoker
760ec72955c-b5ba-40b8-a67e-13b137d95cb7Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc88872166-2Tobacco smoking status NHIS266919005Never smoker
82271d62e29-789d-4417-95fa-b71292d2e685Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc88872166-2Tobacco smoking status NHIS266919005Never smoker
8596d191cde-5731-41e5-9197-5827fad6608cPatient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc88872166-2Tobacco smoking status NHIS266919005Never smoker
871566c6a71-d7ee-49fa-88a4-2c5e6e1d1476Patient/84d4bafe-1891-4e6c-b8aa-38d2eafd819372166-2Tobacco smoking status NHIS266919005Never smoker
88055ffbd45-6042-4240-8f32-4cd10c1c6d48Patient/84d4bafe-1891-4e6c-b8aa-38d2eafd819372166-2Tobacco smoking status NHIS266919005Never smoker
900abd678ea-30d0-48b1-9f8e-4b96c952e55ePatient/84d4bafe-1891-4e6c-b8aa-38d2eafd819372166-2Tobacco smoking status NHIS266919005Never smoker
910c00b5394-c828-46d3-b81f-498b768c5c81Patient/84d4bafe-1891-4e6c-b8aa-38d2eafd819372166-2Tobacco smoking status NHIS266919005Never smoker
.....................
30941448ef79-0efb-49fd-a32e-8e4501d63654Patient/7ba8d35f-3f70-48b9-b711-104374136ac772166-2Tobacco smoking status NHIS266919005Never smoker
310370b64209-a5f0-4d9b-9b92-ca8d4e37ad20Patient/7ba8d35f-3f70-48b9-b711-104374136ac772166-2Tobacco smoking status NHIS266919005Never smoker
3128518b9a78-3ed3-401e-8792-b1f5653da244Patient/7ba8d35f-3f70-48b9-b711-104374136ac772166-2Tobacco smoking status NHIS266919005Never smoker
31374a491920-5523-4e12-a178-ad7692aae555Patient/7ba8d35f-3f70-48b9-b711-104374136ac772166-2Tobacco smoking status NHIS266919005Never smoker
3146eec65c1b-e59a-4e65-a311-2c0cd097b078Patient/7ba8d35f-3f70-48b9-b711-104374136ac772166-2Tobacco smoking status NHIS266919005Never smoker
31665559cf2a-1386-4a48-b889-29798870a3f4Patient/7ba8d35f-3f70-48b9-b711-104374136ac772166-2Tobacco smoking status NHIS266919005Never smoker
319062abe2b1-4b46-41e4-9784-05509fe2439bPatient/7ba8d35f-3f70-48b9-b711-104374136ac772166-2Tobacco smoking status NHIS266919005Never smoker
31999cc9a329-78d7-4b4c-9734-624576028f5fPatient/ff9d23d8-f3c8-4eee-a5f9-e05e843675b572166-2Tobacco smoking status NHIS266919005Never smoker
3208aab937d5-dc7c-44f2-9328-c58039baeedbPatient/ff9d23d8-f3c8-4eee-a5f9-e05e843675b572166-2Tobacco smoking status NHIS266919005Never smoker
3228c8321dcc-cba2-454e-a862-b692bf75455ePatient/ff9d23d8-f3c8-4eee-a5f9-e05e843675b572166-2Tobacco smoking status NHIS266919005Never smoker
3236df310c44-a40d-4596-a16d-22ae9454e2d7Patient/ff9d23d8-f3c8-4eee-a5f9-e05e843675b572166-2Tobacco smoking status NHIS266919005Never smoker
3245ebb48186-7b0c-4ea2-a4ee-1bc77bd40926Patient/8d3e1155-278a-4824-a7e0-fddb24c7c17972166-2Tobacco smoking status NHIS266919005Never smoker
3253837fd4cb-10c3-4d89-b2c1-23be8a8572b6Patient/8d3e1155-278a-4824-a7e0-fddb24c7c17972166-2Tobacco smoking status NHIS266919005Never smoker
3262e9eabd4c-3d45-45e8-b21e-0a479cb0530fPatient/8d3e1155-278a-4824-a7e0-fddb24c7c17972166-2Tobacco smoking status NHIS266919005Never smoker
32826a594967-b815-4650-82f6-eb404bfc7aafPatient/8d3e1155-278a-4824-a7e0-fddb24c7c17972166-2Tobacco smoking status NHIS266919005Never smoker
3299fe884b8b-04d9-4c3b-a710-f77d89d87befPatient/8d3e1155-278a-4824-a7e0-fddb24c7c17972166-2Tobacco smoking status NHIS266919005Never smoker
331722d56cc5-7a0e-49cf-8483-97de08415e1cPatient/8d3e1155-278a-4824-a7e0-fddb24c7c17972166-2Tobacco smoking status NHIS266919005Never smoker
33346bcef563-1082-4398-85d1-86a8f6603141Patient/8d3e1155-278a-4824-a7e0-fddb24c7c17972166-2Tobacco smoking status NHIS266919005Never smoker
33516d9a0171-6e80-464f-8022-5b740af8a932Patient/8d3e1155-278a-4824-a7e0-fddb24c7c17972166-2Tobacco smoking status NHIS266919005Never smoker
3368e5447fff-dee3-4e8a-9756-e11304fa7f82Patient/8d3e1155-278a-4824-a7e0-fddb24c7c17972166-2Tobacco smoking status NHIS266919005Never smoker
3396df8ba6a0-a4e8-4983-b1fb-85a6e68d5d39Patient/8d3e1155-278a-4824-a7e0-fddb24c7c17972166-2Tobacco smoking status NHIS266919005Never smoker
34241c7ca98b-18a6-401a-b5f9-218d21455658Patient/8d3e1155-278a-4824-a7e0-fddb24c7c17972166-2Tobacco smoking status NHIS266919005Never smoker
344171029c58-cca3-45d6-8f2e-cfa1b225883cPatient/8d3e1155-278a-4824-a7e0-fddb24c7c17972166-2Tobacco smoking status NHIS266919005Never smoker
3458756e618d-9983-4da5-9023-6103c1d4f3c0Patient/8c9fea57-6ded-47b0-88c9-75518430b57272166-2Tobacco smoking status NHIS8517006Former smoker
3479f70de159-6f5c-40fb-a3fd-90c722a51aa6Patient/8c9fea57-6ded-47b0-88c9-75518430b57272166-2Tobacco smoking status NHIS8517006Former smoker
34964ea513ae-7aaf-4e61-bfa1-4e82333e6788Patient/8c9fea57-6ded-47b0-88c9-75518430b57272166-2Tobacco smoking status NHIS8517006Former smoker
352421baef36-0caf-452e-a049-d624974a0739Patient/8c9fea57-6ded-47b0-88c9-75518430b57272166-2Tobacco smoking status NHIS8517006Former smoker
354574b3fc01-da0c-4043-b630-38bb679fbd42Patient/8c9fea57-6ded-47b0-88c9-75518430b57272166-2Tobacco smoking status NHIS8517006Former smoker
35622f7f598b-fe86-44c4-88c1-f4951b5f9560Patient/8c9fea57-6ded-47b0-88c9-75518430b57272166-2Tobacco smoking status NHIS8517006Former smoker
3580938dcefa-4ffb-4757-b13a-7cd715bcd599Patient/8c9fea57-6ded-47b0-88c9-75518430b57272166-2Tobacco smoking status NHIS8517006Former smoker
359811f8c8fc-ecee-4ec3-9201-8c192f3f8449Patient/8c9fea57-6ded-47b0-88c9-75518430b57272166-2Tobacco smoking status NHIS8517006Former smoker
36195d0dabae-f225-4ac7-a9e5-ae1103720cdePatient/8c9fea57-6ded-47b0-88c9-75518430b57272166-2Tobacco smoking status NHIS8517006Former smoker
3647437d29cc-02b8-48a3-bc00-4e895bcdddaePatient/8c9fea57-6ded-47b0-88c9-75518430b57272166-2Tobacco smoking status NHIS8517006Former smoker
36670d4d8892-536d-4e3d-adea-690b9f6582edPatient/d2524ab6-4db9-440d-b588-6dcfcab8927072166-2Tobacco smoking status NHIS266919005Never smoker
3675a563d054-0081-4432-8977-a5cde30e13dePatient/d2524ab6-4db9-440d-b588-6dcfcab8927072166-2Tobacco smoking status NHIS266919005Never smoker
36870f7039d9-1ef2-4aac-8617-96e33f827a10Patient/d2524ab6-4db9-440d-b588-6dcfcab8927072166-2Tobacco smoking status NHIS266919005Never smoker
36956e0600dc-f43a-48f8-87f6-45a0983c6d97Patient/d2524ab6-4db9-440d-b588-6dcfcab8927072166-2Tobacco smoking status NHIS266919005Never smoker
3703be1d1942-ae59-42db-914a-04cbc61cdcf3Patient/d2524ab6-4db9-440d-b588-6dcfcab8927072166-2Tobacco smoking status NHIS266919005Never smoker
372682de3db7-2921-453d-bbc5-8c56d70d630fPatient/d2524ab6-4db9-440d-b588-6dcfcab8927072166-2Tobacco smoking status NHIS266919005Never smoker
373407f06643-6409-404c-89f6-fe1eb0e8c8ccPatient/d2524ab6-4db9-440d-b588-6dcfcab8927072166-2Tobacco smoking status NHIS266919005Never smoker
37425ec5a8fd-43e8-4835-bef4-c0a429745a59Patient/d2524ab6-4db9-440d-b588-6dcfcab8927072166-2Tobacco smoking status NHIS266919005Never smoker
37542151a281-9eac-4f98-9a1d-3b30b169e2f2Patient/d2524ab6-4db9-440d-b588-6dcfcab8927072166-2Tobacco smoking status NHIS266919005Never smoker
3762889b8074-f24d-4b7d-aba3-474fed898bdaPatient/d2524ab6-4db9-440d-b588-6dcfcab8927072166-2Tobacco smoking status NHIS266919005Never smoker
3801f596e621-6fe7-4acf-a0a7-2a68a2a77b35Patient/f98b23bf-4443-46d0-9eaf-563e767cf94872166-2Tobacco smoking status NHIS8517006Former smoker
38333e0ed4d8-7e00-42e0-8a7c-d9d6dd57e05dPatient/f98b23bf-4443-46d0-9eaf-563e767cf94872166-2Tobacco smoking status NHIS8517006Former smoker
38814a32ee10-0c5c-4f1c-820a-9155ee04d060Patient/f98b23bf-4443-46d0-9eaf-563e767cf94872166-2Tobacco smoking status NHIS8517006Former smoker
39098e5f48c0-651c-4f02-9a75-8268fa1c0b1bPatient/f98b23bf-4443-46d0-9eaf-563e767cf94872166-2Tobacco smoking status NHIS8517006Former smoker
39523f033821-1193-4ecf-95eb-9cfc2fa21cc0Patient/f98b23bf-4443-46d0-9eaf-563e767cf94872166-2Tobacco smoking status NHIS8517006Former smoker
39805368a223-6972-410f-a253-6be399479303Patient/f98b23bf-4443-46d0-9eaf-563e767cf94872166-2Tobacco smoking status NHIS8517006Former smoker
40086d9ff554-d1cf-48bb-9487-7d132c3575afPatient/f98b23bf-4443-46d0-9eaf-563e767cf94872166-2Tobacco smoking status NHIS8517006Former smoker
\n", "

258 rows × 6 columns

\n", "
" ], "text/plain": [ " id \\\n", "15 51202a40-c62e-4727-bfef-b643869d0951 \n", "32 04cc91b0-4fd0-4244-9738-32c563116308 \n", "53 87cc17d0-3914-4e95-9344-b76e394dfae6 \n", "81 b10d3da1-249e-406d-90cb-9ac19314f823 \n", "98 a0e5b7d8-30a2-4952-ae7a-c0a8d49bc9f3 \n", "119 f843b2e7-6caf-4284-aa90-51c2d55c3c51 \n", "136 d070c5c5-2a0a-4fe9-ba57-0b7fde240f3b \n", "153 64beef69-dd3c-46a1-b970-8a5dc95be927 \n", "185 b120c744-e1ec-49b9-b8b0-2a54455f410c \n", "202 984b970a-b027-4778-a820-7f6c56761125 \n", "213 17378ae3-f3b7-418b-b8e1-7af2856a5eac \n", "221 ec100afb-6dad-47e9-8a58-2ea28c1a5fb6 \n", "233 bdf0d1b0-d405-4dbe-acc3-855d6c38d0d2 \n", "252 8b4c3ebb-0f3b-4a9f-aca6-894df17a652d \n", "260 db972319-bfa5-4d88-a562-3e71e94bafc1 \n", "273 88515b19-a75f-486e-af88-96222bd4ca18 \n", "281 2da712dc-da92-4f89-a36e-3aa8fc714d73 \n", "289 219df89c-b801-4d3d-bfc0-7b270ad22542 \n", "312 b99b0288-8c94-4126-a7e0-d1c8717e51a3 \n", "332 18e82af6-9bb3-4657-8ee6-b02bdb288476 \n", "341 45e97461-0051-4bf5-82df-a8c4a429f099 \n", "350 e84acd03-8ae3-4b82-95cf-25799c7c94b8 \n", "359 f3d3151e-9a5b-4227-9b28-89f9cb61ca90 \n", "368 c3364946-7957-4a88-8912-7dc8a9a1cfcd \n", "377 cb400c7f-485e-4674-a3d8-4700543235ae \n", "398 450e1f51-8765-431c-a795-40206ac5c9fa \n", "425 4ab3823f-27c4-466c-ae57-16e914454522 \n", "434 23c9d6e1-71bf-4766-96df-65a3b0d25d05 \n", "445 187e50b4-ac94-48a1-b052-02b5549be600 \n", "456 a0e8bef3-8317-4930-858d-8dff3830ff13 \n", "468 30c32d11-4e9a-4feb-b352-2196792bb2cf \n", "479 0145b24c-12c5-4ef2-a437-3efa9006f498 \n", "488 2b444657-c019-4434-a9d4-3e0f2f7cdd2f \n", "509 73b8dc8c-91f8-41dd-a6ba-981c6ed12316 \n", "518 c83226df-6f61-4849-bcce-1eec0f56be74 \n", "534 683a4889-8ee5-4f5b-befb-35ec22b13107 \n", "562 2735a667-d2cd-4491-9a32-0758ce0f48e9 \n", "583 f86585e1-ec12-4125-a254-2533ea3b9bba \n", "600 495b7f39-53b4-4d98-884d-fe14e464a13a \n", "637 12ea697f-c2b3-4539-80c7-2ed09c84da2b \n", "658 3a4f53d6-3c82-4efb-b891-d6a866949e58 \n", "675 fb5ed249-16e5-4379-85e7-6b8300ee9548 \n", "723 ee6a13b1-b6d1-49ab-8eff-128dc7f40daf \n", "760 ec72955c-b5ba-40b8-a67e-13b137d95cb7 \n", "822 71d62e29-789d-4417-95fa-b71292d2e685 \n", "859 6d191cde-5731-41e5-9197-5827fad6608c \n", "871 566c6a71-d7ee-49fa-88a4-2c5e6e1d1476 \n", "880 55ffbd45-6042-4240-8f32-4cd10c1c6d48 \n", "900 abd678ea-30d0-48b1-9f8e-4b96c952e55e \n", "910 c00b5394-c828-46d3-b81f-498b768c5c81 \n", "... ... \n", "3094 1448ef79-0efb-49fd-a32e-8e4501d63654 \n", "3103 70b64209-a5f0-4d9b-9b92-ca8d4e37ad20 \n", "3128 518b9a78-3ed3-401e-8792-b1f5653da244 \n", "3137 4a491920-5523-4e12-a178-ad7692aae555 \n", "3146 eec65c1b-e59a-4e65-a311-2c0cd097b078 \n", "3166 5559cf2a-1386-4a48-b889-29798870a3f4 \n", "3190 62abe2b1-4b46-41e4-9784-05509fe2439b \n", "3199 9cc9a329-78d7-4b4c-9734-624576028f5f \n", "3208 aab937d5-dc7c-44f2-9328-c58039baeedb \n", "3228 c8321dcc-cba2-454e-a862-b692bf75455e \n", "3236 df310c44-a40d-4596-a16d-22ae9454e2d7 \n", "3245 ebb48186-7b0c-4ea2-a4ee-1bc77bd40926 \n", "3253 837fd4cb-10c3-4d89-b2c1-23be8a8572b6 \n", "3262 e9eabd4c-3d45-45e8-b21e-0a479cb0530f \n", "3282 6a594967-b815-4650-82f6-eb404bfc7aaf \n", "3299 fe884b8b-04d9-4c3b-a710-f77d89d87bef \n", "3317 22d56cc5-7a0e-49cf-8483-97de08415e1c \n", "3334 6bcef563-1082-4398-85d1-86a8f6603141 \n", "3351 6d9a0171-6e80-464f-8022-5b740af8a932 \n", "3368 e5447fff-dee3-4e8a-9756-e11304fa7f82 \n", "3396 df8ba6a0-a4e8-4983-b1fb-85a6e68d5d39 \n", "3424 1c7ca98b-18a6-401a-b5f9-218d21455658 \n", "3441 71029c58-cca3-45d6-8f2e-cfa1b225883c \n", "3458 756e618d-9983-4da5-9023-6103c1d4f3c0 \n", "3479 f70de159-6f5c-40fb-a3fd-90c722a51aa6 \n", "3496 4ea513ae-7aaf-4e61-bfa1-4e82333e6788 \n", "3524 21baef36-0caf-452e-a049-d624974a0739 \n", "3545 74b3fc01-da0c-4043-b630-38bb679fbd42 \n", "3562 2f7f598b-fe86-44c4-88c1-f4951b5f9560 \n", "3580 938dcefa-4ffb-4757-b13a-7cd715bcd599 \n", "3598 11f8c8fc-ecee-4ec3-9201-8c192f3f8449 \n", "3619 5d0dabae-f225-4ac7-a9e5-ae1103720cde \n", "3647 437d29cc-02b8-48a3-bc00-4e895bcdddae \n", "3667 0d4d8892-536d-4e3d-adea-690b9f6582ed \n", "3675 a563d054-0081-4432-8977-a5cde30e13de \n", "3687 0f7039d9-1ef2-4aac-8617-96e33f827a10 \n", "3695 6e0600dc-f43a-48f8-87f6-45a0983c6d97 \n", "3703 be1d1942-ae59-42db-914a-04cbc61cdcf3 \n", "3726 82de3db7-2921-453d-bbc5-8c56d70d630f \n", "3734 07f06643-6409-404c-89f6-fe1eb0e8c8cc \n", "3742 5ec5a8fd-43e8-4835-bef4-c0a429745a59 \n", "3754 2151a281-9eac-4f98-9a1d-3b30b169e2f2 \n", "3762 889b8074-f24d-4b7d-aba3-474fed898bda \n", "3801 f596e621-6fe7-4acf-a0a7-2a68a2a77b35 \n", "3833 3e0ed4d8-7e00-42e0-8a7c-d9d6dd57e05d \n", "3881 4a32ee10-0c5c-4f1c-820a-9155ee04d060 \n", "3909 8e5f48c0-651c-4f02-9a75-8268fa1c0b1b \n", "3952 3f033821-1193-4ecf-95eb-9cfc2fa21cc0 \n", "3980 5368a223-6972-410f-a253-6be399479303 \n", "4008 6d9ff554-d1cf-48bb-9487-7d132c3575af \n", "\n", " patient type \\\n", "15 Patient/fbfec681-d357-4b28-b1d2-5db6434c7846 72166-2 \n", "32 Patient/fbfec681-d357-4b28-b1d2-5db6434c7846 72166-2 \n", "53 Patient/fbfec681-d357-4b28-b1d2-5db6434c7846 72166-2 \n", "81 Patient/fbfec681-d357-4b28-b1d2-5db6434c7846 72166-2 \n", "98 Patient/fbfec681-d357-4b28-b1d2-5db6434c7846 72166-2 \n", "119 Patient/fbfec681-d357-4b28-b1d2-5db6434c7846 72166-2 \n", "136 Patient/fbfec681-d357-4b28-b1d2-5db6434c7846 72166-2 \n", "153 Patient/fbfec681-d357-4b28-b1d2-5db6434c7846 72166-2 \n", "185 Patient/fbfec681-d357-4b28-b1d2-5db6434c7846 72166-2 \n", "202 Patient/fbfec681-d357-4b28-b1d2-5db6434c7846 72166-2 \n", "213 Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b95 72166-2 \n", "221 Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b95 72166-2 \n", "233 Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b95 72166-2 \n", "252 Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b95 72166-2 \n", "260 Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b95 72166-2 \n", "273 Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b95 72166-2 \n", "281 Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b95 72166-2 \n", "289 Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b95 72166-2 \n", "312 Patient/0b8a6ef0-07c8-48ca-804d-1e64f6e44b95 72166-2 \n", "332 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "341 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "350 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "359 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "368 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "377 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "398 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "425 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "434 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "445 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "456 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "468 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "479 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "488 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "509 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "518 Patient/62e03ae7-079c-4eda-9b5a-29440d3a015a 72166-2 \n", "534 Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc888 72166-2 \n", "562 Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc888 72166-2 \n", "583 Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc888 72166-2 \n", "600 Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc888 72166-2 \n", "637 Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc888 72166-2 \n", "658 Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc888 72166-2 \n", "675 Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc888 72166-2 \n", "723 Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc888 72166-2 \n", "760 Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc888 72166-2 \n", "822 Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc888 72166-2 \n", "859 Patient/7cdaae04-4ce7-4a6d-b6a3-5cddba9bc888 72166-2 \n", "871 Patient/84d4bafe-1891-4e6c-b8aa-38d2eafd8193 72166-2 \n", "880 Patient/84d4bafe-1891-4e6c-b8aa-38d2eafd8193 72166-2 \n", "900 Patient/84d4bafe-1891-4e6c-b8aa-38d2eafd8193 72166-2 \n", "910 Patient/84d4bafe-1891-4e6c-b8aa-38d2eafd8193 72166-2 \n", "... ... ... \n", "3094 Patient/7ba8d35f-3f70-48b9-b711-104374136ac7 72166-2 \n", "3103 Patient/7ba8d35f-3f70-48b9-b711-104374136ac7 72166-2 \n", "3128 Patient/7ba8d35f-3f70-48b9-b711-104374136ac7 72166-2 \n", "3137 Patient/7ba8d35f-3f70-48b9-b711-104374136ac7 72166-2 \n", "3146 Patient/7ba8d35f-3f70-48b9-b711-104374136ac7 72166-2 \n", "3166 Patient/7ba8d35f-3f70-48b9-b711-104374136ac7 72166-2 \n", "3190 Patient/7ba8d35f-3f70-48b9-b711-104374136ac7 72166-2 \n", "3199 Patient/ff9d23d8-f3c8-4eee-a5f9-e05e843675b5 72166-2 \n", "3208 Patient/ff9d23d8-f3c8-4eee-a5f9-e05e843675b5 72166-2 \n", "3228 Patient/ff9d23d8-f3c8-4eee-a5f9-e05e843675b5 72166-2 \n", "3236 Patient/ff9d23d8-f3c8-4eee-a5f9-e05e843675b5 72166-2 \n", "3245 Patient/8d3e1155-278a-4824-a7e0-fddb24c7c179 72166-2 \n", "3253 Patient/8d3e1155-278a-4824-a7e0-fddb24c7c179 72166-2 \n", "3262 Patient/8d3e1155-278a-4824-a7e0-fddb24c7c179 72166-2 \n", "3282 Patient/8d3e1155-278a-4824-a7e0-fddb24c7c179 72166-2 \n", "3299 Patient/8d3e1155-278a-4824-a7e0-fddb24c7c179 72166-2 \n", "3317 Patient/8d3e1155-278a-4824-a7e0-fddb24c7c179 72166-2 \n", "3334 Patient/8d3e1155-278a-4824-a7e0-fddb24c7c179 72166-2 \n", "3351 Patient/8d3e1155-278a-4824-a7e0-fddb24c7c179 72166-2 \n", "3368 Patient/8d3e1155-278a-4824-a7e0-fddb24c7c179 72166-2 \n", "3396 Patient/8d3e1155-278a-4824-a7e0-fddb24c7c179 72166-2 \n", "3424 Patient/8d3e1155-278a-4824-a7e0-fddb24c7c179 72166-2 \n", "3441 Patient/8d3e1155-278a-4824-a7e0-fddb24c7c179 72166-2 \n", "3458 Patient/8c9fea57-6ded-47b0-88c9-75518430b572 72166-2 \n", "3479 Patient/8c9fea57-6ded-47b0-88c9-75518430b572 72166-2 \n", "3496 Patient/8c9fea57-6ded-47b0-88c9-75518430b572 72166-2 \n", "3524 Patient/8c9fea57-6ded-47b0-88c9-75518430b572 72166-2 \n", "3545 Patient/8c9fea57-6ded-47b0-88c9-75518430b572 72166-2 \n", "3562 Patient/8c9fea57-6ded-47b0-88c9-75518430b572 72166-2 \n", "3580 Patient/8c9fea57-6ded-47b0-88c9-75518430b572 72166-2 \n", "3598 Patient/8c9fea57-6ded-47b0-88c9-75518430b572 72166-2 \n", "3619 Patient/8c9fea57-6ded-47b0-88c9-75518430b572 72166-2 \n", "3647 Patient/8c9fea57-6ded-47b0-88c9-75518430b572 72166-2 \n", "3667 Patient/d2524ab6-4db9-440d-b588-6dcfcab89270 72166-2 \n", "3675 Patient/d2524ab6-4db9-440d-b588-6dcfcab89270 72166-2 \n", "3687 Patient/d2524ab6-4db9-440d-b588-6dcfcab89270 72166-2 \n", "3695 Patient/d2524ab6-4db9-440d-b588-6dcfcab89270 72166-2 \n", "3703 Patient/d2524ab6-4db9-440d-b588-6dcfcab89270 72166-2 \n", "3726 Patient/d2524ab6-4db9-440d-b588-6dcfcab89270 72166-2 \n", "3734 Patient/d2524ab6-4db9-440d-b588-6dcfcab89270 72166-2 \n", "3742 Patient/d2524ab6-4db9-440d-b588-6dcfcab89270 72166-2 \n", "3754 Patient/d2524ab6-4db9-440d-b588-6dcfcab89270 72166-2 \n", "3762 Patient/d2524ab6-4db9-440d-b588-6dcfcab89270 72166-2 \n", "3801 Patient/f98b23bf-4443-46d0-9eaf-563e767cf948 72166-2 \n", "3833 Patient/f98b23bf-4443-46d0-9eaf-563e767cf948 72166-2 \n", "3881 Patient/f98b23bf-4443-46d0-9eaf-563e767cf948 72166-2 \n", "3909 Patient/f98b23bf-4443-46d0-9eaf-563e767cf948 72166-2 \n", "3952 Patient/f98b23bf-4443-46d0-9eaf-563e767cf948 72166-2 \n", "3980 Patient/f98b23bf-4443-46d0-9eaf-563e767cf948 72166-2 \n", "4008 Patient/f98b23bf-4443-46d0-9eaf-563e767cf948 72166-2 \n", "\n", " type_display code code_display \n", "15 Tobacco smoking status NHIS 8517006 Former smoker \n", "32 Tobacco smoking status NHIS 8517006 Former smoker \n", "53 Tobacco smoking status NHIS 8517006 Former smoker \n", "81 Tobacco smoking status NHIS 8517006 Former smoker \n", "98 Tobacco smoking status NHIS 8517006 Former smoker \n", "119 Tobacco smoking status NHIS 8517006 Former smoker \n", "136 Tobacco smoking status NHIS 8517006 Former smoker \n", "153 Tobacco smoking status NHIS 8517006 Former smoker \n", "185 Tobacco smoking status NHIS 8517006 Former smoker \n", "202 Tobacco smoking status NHIS 8517006 Former smoker \n", "213 Tobacco smoking status NHIS 266919005 Never smoker \n", "221 Tobacco smoking status NHIS 266919005 Never smoker \n", "233 Tobacco smoking status NHIS 266919005 Never smoker \n", "252 Tobacco smoking status NHIS 266919005 Never smoker \n", "260 Tobacco smoking status NHIS 266919005 Never smoker \n", "273 Tobacco smoking status NHIS 266919005 Never smoker \n", "281 Tobacco smoking status NHIS 266919005 Never smoker \n", "289 Tobacco smoking status NHIS 266919005 Never smoker \n", "312 Tobacco smoking status NHIS 266919005 Never smoker \n", "332 Tobacco smoking status NHIS 266919005 Never smoker \n", "341 Tobacco smoking status NHIS 266919005 Never smoker \n", "350 Tobacco smoking status NHIS 266919005 Never smoker \n", "359 Tobacco smoking status NHIS 266919005 Never smoker \n", "368 Tobacco smoking status NHIS 266919005 Never smoker \n", "377 Tobacco smoking status NHIS 266919005 Never smoker \n", "398 Tobacco smoking status NHIS 266919005 Never smoker \n", "425 Tobacco smoking status NHIS 266919005 Never smoker \n", "434 Tobacco smoking status NHIS 266919005 Never smoker \n", "445 Tobacco smoking status NHIS 266919005 Never smoker \n", "456 Tobacco smoking status NHIS 266919005 Never smoker \n", "468 Tobacco smoking status NHIS 266919005 Never smoker \n", "479 Tobacco smoking status NHIS 266919005 Never smoker \n", "488 Tobacco smoking status NHIS 266919005 Never smoker \n", "509 Tobacco smoking status NHIS 266919005 Never smoker \n", "518 Tobacco smoking status NHIS 266919005 Never smoker \n", "534 Tobacco smoking status NHIS 266919005 Never smoker \n", "562 Tobacco smoking status NHIS 266919005 Never smoker \n", "583 Tobacco smoking status NHIS 266919005 Never smoker \n", "600 Tobacco smoking status NHIS 266919005 Never smoker \n", "637 Tobacco smoking status NHIS 266919005 Never smoker \n", "658 Tobacco smoking status NHIS 266919005 Never smoker \n", "675 Tobacco smoking status NHIS 266919005 Never smoker \n", "723 Tobacco smoking status NHIS 266919005 Never smoker \n", "760 Tobacco smoking status NHIS 266919005 Never smoker \n", "822 Tobacco smoking status NHIS 266919005 Never smoker \n", "859 Tobacco smoking status NHIS 266919005 Never smoker \n", "871 Tobacco smoking status NHIS 266919005 Never smoker \n", "880 Tobacco smoking status NHIS 266919005 Never smoker \n", "900 Tobacco smoking status NHIS 266919005 Never smoker \n", "910 Tobacco smoking status NHIS 266919005 Never smoker \n", "... ... ... ... \n", "3094 Tobacco smoking status NHIS 266919005 Never smoker \n", "3103 Tobacco smoking status NHIS 266919005 Never smoker \n", "3128 Tobacco smoking status NHIS 266919005 Never smoker \n", "3137 Tobacco smoking status NHIS 266919005 Never smoker \n", "3146 Tobacco smoking status NHIS 266919005 Never smoker \n", "3166 Tobacco smoking status NHIS 266919005 Never smoker \n", "3190 Tobacco smoking status NHIS 266919005 Never smoker \n", "3199 Tobacco smoking status NHIS 266919005 Never smoker \n", "3208 Tobacco smoking status NHIS 266919005 Never smoker \n", "3228 Tobacco smoking status NHIS 266919005 Never smoker \n", "3236 Tobacco smoking status NHIS 266919005 Never smoker \n", "3245 Tobacco smoking status NHIS 266919005 Never smoker \n", "3253 Tobacco smoking status NHIS 266919005 Never smoker \n", "3262 Tobacco smoking status NHIS 266919005 Never smoker \n", "3282 Tobacco smoking status NHIS 266919005 Never smoker \n", "3299 Tobacco smoking status NHIS 266919005 Never smoker \n", "3317 Tobacco smoking status NHIS 266919005 Never smoker \n", "3334 Tobacco smoking status NHIS 266919005 Never smoker \n", "3351 Tobacco smoking status NHIS 266919005 Never smoker \n", "3368 Tobacco smoking status NHIS 266919005 Never smoker \n", "3396 Tobacco smoking status NHIS 266919005 Never smoker \n", "3424 Tobacco smoking status NHIS 266919005 Never smoker \n", "3441 Tobacco smoking status NHIS 266919005 Never smoker \n", "3458 Tobacco smoking status NHIS 8517006 Former smoker \n", "3479 Tobacco smoking status NHIS 8517006 Former smoker \n", "3496 Tobacco smoking status NHIS 8517006 Former smoker \n", "3524 Tobacco smoking status NHIS 8517006 Former smoker \n", "3545 Tobacco smoking status NHIS 8517006 Former smoker \n", "3562 Tobacco smoking status NHIS 8517006 Former smoker \n", "3580 Tobacco smoking status NHIS 8517006 Former smoker \n", "3598 Tobacco smoking status NHIS 8517006 Former smoker \n", "3619 Tobacco smoking status NHIS 8517006 Former smoker \n", "3647 Tobacco smoking status NHIS 8517006 Former smoker \n", "3667 Tobacco smoking status NHIS 266919005 Never smoker \n", "3675 Tobacco smoking status NHIS 266919005 Never smoker \n", "3687 Tobacco smoking status NHIS 266919005 Never smoker \n", "3695 Tobacco smoking status NHIS 266919005 Never smoker \n", "3703 Tobacco smoking status NHIS 266919005 Never smoker \n", "3726 Tobacco smoking status NHIS 266919005 Never smoker \n", "3734 Tobacco smoking status NHIS 266919005 Never smoker \n", "3742 Tobacco smoking status NHIS 266919005 Never smoker \n", "3754 Tobacco smoking status NHIS 266919005 Never smoker \n", "3762 Tobacco smoking status NHIS 266919005 Never smoker \n", "3801 Tobacco smoking status NHIS 8517006 Former smoker \n", "3833 Tobacco smoking status NHIS 8517006 Former smoker \n", "3881 Tobacco smoking status NHIS 8517006 Former smoker \n", "3909 Tobacco smoking status NHIS 8517006 Former smoker \n", "3952 Tobacco smoking status NHIS 8517006 Former smoker \n", "3980 Tobacco smoking status NHIS 8517006 Former smoker \n", "4008 Tobacco smoking status NHIS 8517006 Former smoker \n", "\n", "[258 rows x 6 columns]" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Try filtering to just Observations of smoking status\n", "\n", "# `fetcher.reprocess_dataframes()` does the same thing as `get_dataframes()`,\n", "# but with FHIRPaths and without re-downloading everything\n", "dfs = fetcher.reprocess_dataframes({\n", " 'Patient': [\n", " (\"id\", \"identifier[0].value\"),\n", " (\"gender\", \"gender\"),\n", " (\"date_of_birth\", \"birthDate\"),\n", " (\"marital_status\", \"maritalStatus.coding.first().code\")\n", " ],\n", " 'Observation': [\n", " (\"id\", \"id\"),\n", " (\"patient\", \"subject.reference\"),\n", " (\"type\", \"code.coding.first().code\"),\n", " (\"type_display\", \"code.coding.first().display\"),\n", " (\"code\", \"valueCodeableConcept.coding.first().code\"),\n", " (\"code_display\", \"valueCodeableConcept.coding.first().display\"),\n", " ]\n", "})\n", "\n", "with pd.option_context('display.max_rows', 100, 'display.min_rows', 100):\n", " display(dfs['Observation'][dfs['Observation']['type'] == '72166-2'])" ] }, { "cell_type": "markdown", "id": "8f55bf22", "metadata": {}, "source": [ "## Creating FHIRPaths\n", "\n", "It may be helpful to use an online tool like to assist with creating FHIRPaths for filtering the FHIR resources down for creating DataFrames. (Note that you should not use tools like this with identified patient data.)\n", "\n", "We have a convenience method to get an example resource in JSON format from the `fetcher` object:" ] }, { "cell_type": "code", "execution_count": 27, "id": "a2e3e866", "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    \"resourceType\": \"Observation\",\n",
       "    \"id\": \"7a10d1ef-97f2-468b-b3ab-78dc99c65cf6\",\n",
       "    \"status\": \"final\",\n",
       "    \"category\": [\n",
       "        {\n",
       "            \"coding\": [\n",
       "                {\n",
       "                    \"system\": \"http://terminology.hl7.org/CodeSystem/observation-category\",\n",
       "                    \"code\": \"vital-signs\",\n",
       "                    \"display\": \"vital-signs\"\n",
       "                }\n",
       "            ]\n",
       "        }\n",
       "    ],\n",
       "    \"code\": {\n",
       "        \"coding\": [\n",
       "            {\n",
       "                \"system\": \"http://loinc.org\",\n",
       "                \"code\": \"8302-2\",\n",
       "                \"display\": \"Body Height\"\n",
       "            }\n",
       "        ],\n",
       "        \"text\": \"Body Height\"\n",
       "    },\n",
       "    \"subject\": {\n",
       "        \"reference\": \"Patient/fbfec681-d357-4b28-b1d2-5db6434c7846\"\n",
       "    },\n",
       "    \"encounter\": {\n",
       "        \"reference\": \"Encounter/a65fee02-b183-4ae5-a9e2-5edf89a6f327\"\n",
       "    },\n",
       "    \"effectiveDateTime\": \"2010-11-20T02:52:44-05:00\",\n",
       "    \"issued\": \"2010-11-20T02:52:44.074-05:00\",\n",
       "    \"valueQuantity\": {\n",
       "        \"value\": 162.4,\n",
       "        \"unit\": \"cm\",\n",
       "        \"system\": \"http://unitsofmeasure.org\",\n",
       "        \"code\": \"cm\"\n",
       "    }\n",
       "}\n",
       "
\n" ], "text/plain": [ "\u001B[1m{\u001B[0m\n", " \u001B[32m\"resourceType\"\u001B[0m: \u001B[32m\"Observation\"\u001B[0m,\n", " \u001B[32m\"id\"\u001B[0m: \u001B[32m\"7a10d1ef-97f2-468b-b3ab-78dc99c65cf6\"\u001B[0m,\n", " \u001B[32m\"status\"\u001B[0m: \u001B[32m\"final\"\u001B[0m,\n", " \u001B[32m\"category\"\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m\"coding\"\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m\"system\"\u001B[0m: \u001B[32m\"http://terminology.hl7.org/CodeSystem/observation-category\"\u001B[0m,\n", " \u001B[32m\"code\"\u001B[0m: \u001B[32m\"vital-signs\"\u001B[0m,\n", " \u001B[32m\"display\"\u001B[0m: \u001B[32m\"vital-signs\"\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m\"code\"\u001B[0m: \u001B[1m{\u001B[0m\n", " \u001B[32m\"coding\"\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m\"system\"\u001B[0m: \u001B[32m\"http://loinc.org\"\u001B[0m,\n", " \u001B[32m\"code\"\u001B[0m: \u001B[32m\"8302-2\"\u001B[0m,\n", " \u001B[32m\"display\"\u001B[0m: \u001B[32m\"Body Height\"\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m\"text\"\u001B[0m: \u001B[32m\"Body Height\"\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[32m\"subject\"\u001B[0m: \u001B[1m{\u001B[0m\n", " \u001B[32m\"reference\"\u001B[0m: \u001B[32m\"Patient/fbfec681-d357-4b28-b1d2-5db6434c7846\"\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[32m\"encounter\"\u001B[0m: \u001B[1m{\u001B[0m\n", " \u001B[32m\"reference\"\u001B[0m: \u001B[32m\"Encounter/a65fee02-b183-4ae5-a9e2-5edf89a6f327\"\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[32m\"effectiveDateTime\"\u001B[0m: \u001B[32m\"2010-11-20T02:52:44-05:00\"\u001B[0m,\n", " \u001B[32m\"issued\"\u001B[0m: \u001B[32m\"2010-11-20T02:52:44.074-05:00\"\u001B[0m,\n", " \u001B[32m\"valueQuantity\"\u001B[0m: \u001B[1m{\u001B[0m\n", " \u001B[32m\"value\"\u001B[0m: \u001B[1;36m162.4\u001B[0m,\n", " \u001B[32m\"unit\"\u001B[0m: \u001B[32m\"cm\"\u001B[0m,\n", " \u001B[32m\"system\"\u001B[0m: \u001B[32m\"http://unitsofmeasure.org\"\u001B[0m,\n", " \u001B[32m\"code\"\u001B[0m: \u001B[32m\"cm\"\u001B[0m\n", " \u001B[1m}\u001B[0m\n", "\u001B[1m}\u001B[0m\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "print(json.dumps(fetcher.get_example_resource('Observation'), indent=4))" ] }, { "cell_type": "markdown", "id": "a3a6c677", "metadata": {}, "source": [ "This can be copied and pasted into to experiment with FHIRPaths. Note that the JavaScript library used on this testing website is not the same as the Python library used in this notebook, so there may be some implementation differences." ] }, { "cell_type": "markdown", "id": "4076594e", "metadata": {}, "source": [ "## Testing with Synthea data\n", "\n", "Having test data is very helpful when developing code that uses FHIR Bulk Data. The test data from may not have all the data elements you need for a specific research use case. [Synthea](https://mitre.github.io/fhir-for-research/modules/synthea-overview) can be used for generating customized synthetic data in FHIR format. Below we'll look at how to load `.ndjson` from Synthea into this notebook and use `reprocess_dataframes()` with FHIRPaths to convert into Pandas DataFrames.\n", "\n", "First, we'll create a short class to mimic the functionality of `BulkDataFetcher` but with loading the `.ndjson` directly from disk rather than via a bulk data export." ] }, { "cell_type": "code", "execution_count": 28, "id": "0b9964a4", "metadata": { "pycharm": { "is_executing": true } }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "edc6f44ada914f9e92e65db30a7c4dc1", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/37919 [00:00Resources available: \n", "\n" ], "text/plain": [ "Resources available: \n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
- Patient\n",
       "- Organization\n",
       "- Practitioner\n",
       "- Encounter\n",
       "- Condition\n",
       "- Device\n",
       "- Claim\n",
       "- ExplanationOfBenefit\n",
       "- CareTeam\n",
       "- Goal\n",
       "- CarePlan\n",
       "- Observation\n",
       "- Immunization\n",
       "- DiagnosticReport\n",
       "- Procedure\n",
       "- MedicationRequest\n",
       "- ImagingStudy\n",
       "- AllergyIntolerance\n",
       "- MedicationAdministration\n",
       "
\n" ], "text/plain": [ "- Patient\n", "- Organization\n", "- Practitioner\n", "- Encounter\n", "- Condition\n", "- Device\n", "- Claim\n", "- ExplanationOfBenefit\n", "- CareTeam\n", "- Goal\n", "- CarePlan\n", "- Observation\n", "- Immunization\n", "- DiagnosticReport\n", "- Procedure\n", "- MedicationRequest\n", "- ImagingStudy\n", "- AllergyIntolerance\n", "- MedicationAdministration\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "class SyntheaDataFetcher:\n", " def __init__(self, ndjson_file_path):\n", " self.resources_by_type = {}\n", "\n", " num_lines = sum(1 for line in open(ndjson_file_path,'r'))\n", " with open(ndjson_file_path, 'r') as file:\n", " for line in tqdm(file, total=num_lines):\n", " json_obj = json.loads(line)\n", " this_resource_type = json_obj['resourceType']\n", " if this_resource_type not in self.resources_by_type:\n", " self.resources_by_type[this_resource_type] = []\n", " self.resources_by_type[this_resource_type].append(json_obj)\n", "\n", " print(\"Resources available: \")\n", " print('\\n'.join(['- '+ x for x in self.resources_by_type.keys()]))\n", "\n", " def get_example_resource(self, resource_type: str, resource_id: Optional[str] = None):\n", " if self.resources_by_type is None:\n", " print(\"You need to run get_dataframes() first\")\n", " return None\n", "\n", " if resource_type not in self.resources_by_type:\n", " print(f\"{resource_type} not available. Try one of these: {', '.join(self.resources_by_type.keys())}\")\n", " return None\n", "\n", " if resource_id is None:\n", " return self.resources_by_type[resource_type][0]\n", "\n", " resource = [r for r in self.resources_by_type[resource_type] if r['id'] == resource_id]\n", "\n", " if len(resource) > 0:\n", " return resource[0]\n", "\n", " print(f\"No {resource_type} with id={resource_id} was found.\")\n", " return None\n", "\n", " def reprocess_dataframes(self, user_fhir_paths):\n", " return BulkDataFetcher._reprocess_dataframes(self.resources_by_type, user_fhir_paths)\n", "\n", "\n", "# Load in 40 patients of Synthea data.\n", "# The original data come from > 1K Sample Synthetic Patient Records, FHIR R4\n", "synthea_fetcher = SyntheaDataFetcher('synthea_100.ndjson')" ] }, { "cell_type": "markdown", "id": "567d2b65", "metadata": {}, "source": [ "Here is how to apply FHIRPaths to filter the Synthea data:" ] }, { "cell_type": "code", "execution_count": 29, "id": "ebf1bbde", "metadata": { "pycharm": { "is_executing": true } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
id
05cbc121b-cd71-4428-b8b7-31e53eba8184
1adccf2c3-9dc4-4067-ba23-98982c4875da
231191928-6acb-4d73-931c-e601cc3a13fa
367816396-e325-496d-a6ec-c047756b7ce4
4b426b062-8273-4b93-a907-de3176c0567d
......
95ae4c5b55-c704-4406-b353-285f9166a489
96edb1ebc5-d629-4c43-acf5-b8d1c38d9bd2
972d75e3a4-f0f6-45dd-8b57-75fb2f303c9e
98ea95f498-7929-4d50-be55-9bf7baee3a8d
9957ca2c16-7008-41e5-b338-4758b2fc46f0
\n", "

100 rows × 1 columns

\n", "
" ], "text/plain": [ " id\n", "0 5cbc121b-cd71-4428-b8b7-31e53eba8184\n", "1 adccf2c3-9dc4-4067-ba23-98982c4875da\n", "2 31191928-6acb-4d73-931c-e601cc3a13fa\n", "3 67816396-e325-496d-a6ec-c047756b7ce4\n", "4 b426b062-8273-4b93-a907-de3176c0567d\n", ".. ...\n", "95 ae4c5b55-c704-4406-b353-285f9166a489\n", "96 edb1ebc5-d629-4c43-acf5-b8d1c38d9bd2\n", "97 2d75e3a4-f0f6-45dd-8b57-75fb2f303c9e\n", "98 ea95f498-7929-4d50-be55-9bf7baee3a8d\n", "99 57ca2c16-7008-41e5-b338-4758b2fc46f0\n", "\n", "[100 rows x 1 columns]" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dfs = synthea_fetcher.reprocess_dataframes({'Patient': [('id', 'id')]})\n", "\n", "dfs['Patient']" ] }, { "cell_type": "markdown", "id": "dca3731f", "metadata": {}, "source": [ "You can also get a sample resource to look at the raw JSON:" ] }, { "cell_type": "code", "execution_count": 30, "id": "66f1ad8a", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
{\n",
       "    'resourceType': 'Patient',\n",
       "    'id': '5cbc121b-cd71-4428-b8b7-31e53eba8184',\n",
       "    'text': {\n",
       "        'status': 'generated',\n",
       "        'div': '<div xmlns=\"http://www.w3.org/1999/xhtml\">Generated by <a \n",
       "href=\"https://github.com/synthetichealth/synthea\">Synthea</a>.Version identifier: v2.4.0-404-ge7ce2295\\n .   Person\n",
       "seed: 6457100290386878904  Population seed: 0</div>'\n",
       "    },\n",
       "    'extension': [\n",
       "        {\n",
       "            'url': 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-race',\n",
       "            'extension': [\n",
       "                {\n",
       "                    'url': 'ombCategory',\n",
       "                    'valueCoding': {\n",
       "                        'system': 'urn:oid:2.16.840.1.113883.6.238',\n",
       "                        'code': '2106-3',\n",
       "                        'display': 'White'\n",
       "                    }\n",
       "                },\n",
       "                {'url': 'text', 'valueString': 'White'}\n",
       "            ]\n",
       "        },\n",
       "        {\n",
       "            'url': 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-ethnicity',\n",
       "            'extension': [\n",
       "                {\n",
       "                    'url': 'ombCategory',\n",
       "                    'valueCoding': {\n",
       "                        'system': 'urn:oid:2.16.840.1.113883.6.238',\n",
       "                        'code': '2186-5',\n",
       "                        'display': 'Not Hispanic or Latino'\n",
       "                    }\n",
       "                },\n",
       "                {'url': 'text', 'valueString': 'Not Hispanic or Latino'}\n",
       "            ]\n",
       "        },\n",
       "        {\n",
       "            'url': 'http://hl7.org/fhir/StructureDefinition/patient-mothersMaidenName',\n",
       "            'valueString': 'Deadra347 Borer986'\n",
       "        },\n",
       "        {'url': 'http://hl7.org/fhir/us/core/StructureDefinition/us-core-birthsex', 'valueCode': 'M'},\n",
       "        {\n",
       "            'url': 'http://hl7.org/fhir/StructureDefinition/patient-birthPlace',\n",
       "            'valueAddress': {'city': 'Billerica', 'state': 'Massachusetts', 'country': 'US'}\n",
       "        },\n",
       "        {\n",
       "            'url': 'http://synthetichealth.github.io/synthea/disability-adjusted-life-years',\n",
       "            'valueDecimal': 14.062655945052095\n",
       "        },\n",
       "        {\n",
       "            'url': 'http://synthetichealth.github.io/synthea/quality-adjusted-life-years',\n",
       "            'valueDecimal': 58.93734405494791\n",
       "        }\n",
       "    ],\n",
       "    'identifier': [\n",
       "        {'system': 'https://github.com/synthetichealth/synthea', 'value': '2fa15bc7-8866-461a-9000-f739e425860a'},\n",
       "        {\n",
       "            'type': {\n",
       "                'coding': [\n",
       "                    {\n",
       "                        'system': 'http://terminology.hl7.org/CodeSystem/v2-0203',\n",
       "                        'code': 'MR',\n",
       "                        'display': 'Medical Record Number'\n",
       "                    }\n",
       "                ],\n",
       "                'text': 'Medical Record Number'\n",
       "            },\n",
       "            'system': 'http://hospital.smarthealthit.org',\n",
       "            'value': '2fa15bc7-8866-461a-9000-f739e425860a'\n",
       "        },\n",
       "        {\n",
       "            'type': {\n",
       "                'coding': [\n",
       "                    {\n",
       "                        'system': 'http://terminology.hl7.org/CodeSystem/v2-0203',\n",
       "                        'code': 'SS',\n",
       "                        'display': 'Social Security Number'\n",
       "                    }\n",
       "                ],\n",
       "                'text': 'Social Security Number'\n",
       "            },\n",
       "            'system': 'http://hl7.org/fhir/sid/us-ssn',\n",
       "            'value': '999-93-7537'\n",
       "        },\n",
       "        {\n",
       "            'type': {\n",
       "                'coding': [\n",
       "                    {\n",
       "                        'system': 'http://terminology.hl7.org/CodeSystem/v2-0203',\n",
       "                        'code': 'DL',\n",
       "                        'display': \"Driver's License\"\n",
       "                    }\n",
       "                ],\n",
       "                'text': \"Driver's License\"\n",
       "            },\n",
       "            'system': 'urn:oid:2.16.840.1.113883.4.3.25',\n",
       "            'value': 'S99948707'\n",
       "        },\n",
       "        {\n",
       "            'type': {\n",
       "                'coding': [\n",
       "                    {\n",
       "                        'system': 'http://terminology.hl7.org/CodeSystem/v2-0203',\n",
       "                        'code': 'PPN',\n",
       "                        'display': 'Passport Number'\n",
       "                    }\n",
       "                ],\n",
       "                'text': 'Passport Number'\n",
       "            },\n",
       "            'system': 'http://standardhealthrecord.org/fhir/StructureDefinition/passportNumber',\n",
       "            'value': 'X14078167X'\n",
       "        }\n",
       "    ],\n",
       "    'name': [{'use': 'official', 'family': 'Brekke496', 'given': ['Aaron697'], 'prefix': ['Mr.']}],\n",
       "    'telecom': [{'system': 'phone', 'value': '555-677-3119', 'use': 'home'}],\n",
       "    'gender': 'male',\n",
       "    'birthDate': '1945-12-10',\n",
       "    'address': [\n",
       "        {\n",
       "            'extension': [\n",
       "                {\n",
       "                    'url': 'http://hl7.org/fhir/StructureDefinition/geolocation',\n",
       "                    'extension': [\n",
       "                        {'url': 'latitude', 'valueDecimal': 41.93879298871088},\n",
       "                        {'url': 'longitude', 'valueDecimal': -71.06682353144593}\n",
       "                    ]\n",
       "                }\n",
       "            ],\n",
       "            'line': ['894 Brakus Bypass'],\n",
       "            'city': 'Taunton',\n",
       "            'state': 'Massachusetts',\n",
       "            'postalCode': '02718',\n",
       "            'country': 'US'\n",
       "        }\n",
       "    ],\n",
       "    'maritalStatus': {\n",
       "        'coding': [\n",
       "            {'system': 'http://terminology.hl7.org/CodeSystem/v3-MaritalStatus', 'code': 'S', 'display': 'S'}\n",
       "        ],\n",
       "        'text': 'S'\n",
       "    },\n",
       "    'multipleBirthBoolean': False,\n",
       "    'communication': [\n",
       "        {\n",
       "            'language': {\n",
       "                'coding': [{'system': 'urn:ietf:bcp:47', 'code': 'en-US', 'display': 'English'}],\n",
       "                'text': 'English'\n",
       "            }\n",
       "        }\n",
       "    ]\n",
       "}\n",
       "
\n" ], "text/plain": [ "\u001B[1m{\u001B[0m\n", " \u001B[32m'resourceType'\u001B[0m: \u001B[32m'Patient'\u001B[0m,\n", " \u001B[32m'id'\u001B[0m: \u001B[32m'5cbc121b-cd71-4428-b8b7-31e53eba8184'\u001B[0m,\n", " \u001B[32m'text'\u001B[0m: \u001B[1m{\u001B[0m\n", " \u001B[32m'status'\u001B[0m: \u001B[32m'generated'\u001B[0m,\n", " \u001B[32m'div'\u001B[0m: \u001B[32m'\u001B[0m\u001B[32m<\u001B[0m\u001B[32mdiv\u001B[0m\u001B[32m \u001B[0m\u001B[32mxmlns\u001B[0m\u001B[32m=\u001B[0m\u001B[32m\"http\u001B[0m\u001B[32m://www.w3.org/1999/xhtml\">Generated by Synthea.Version identifier: v2.4.0-404-ge7ce2295\\n . Person\u001B[0m\n", "\u001B[32mseed: 6457100290386878904 Population seed: 0\u001B[0m\u001B[32m'\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[32m'extension'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'url'\u001B[0m: \u001B[32m'http://hl7.org/fhir/us/core/StructureDefinition/us-core-race'\u001B[0m,\n", " \u001B[32m'extension'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'url'\u001B[0m: \u001B[32m'ombCategory'\u001B[0m,\n", " \u001B[32m'valueCoding'\u001B[0m: \u001B[1m{\u001B[0m\n", " \u001B[32m'system'\u001B[0m: \u001B[32m'urn:oid:2.16.840.1.113883.6.238'\u001B[0m,\n", " \u001B[32m'code'\u001B[0m: \u001B[32m'2106-3'\u001B[0m,\n", " \u001B[32m'display'\u001B[0m: \u001B[32m'White'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\u001B[32m'url'\u001B[0m: \u001B[32m'text'\u001B[0m, \u001B[32m'valueString'\u001B[0m: \u001B[32m'White'\u001B[0m\u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'url'\u001B[0m: \u001B[32m'http://hl7.org/fhir/us/core/StructureDefinition/us-core-ethnicity'\u001B[0m,\n", " \u001B[32m'extension'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'url'\u001B[0m: \u001B[32m'ombCategory'\u001B[0m,\n", " \u001B[32m'valueCoding'\u001B[0m: \u001B[1m{\u001B[0m\n", " \u001B[32m'system'\u001B[0m: \u001B[32m'urn:oid:2.16.840.1.113883.6.238'\u001B[0m,\n", " \u001B[32m'code'\u001B[0m: \u001B[32m'2186-5'\u001B[0m,\n", " \u001B[32m'display'\u001B[0m: \u001B[32m'Not Hispanic or Latino'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\u001B[32m'url'\u001B[0m: \u001B[32m'text'\u001B[0m, \u001B[32m'valueString'\u001B[0m: \u001B[32m'Not Hispanic or Latino'\u001B[0m\u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'url'\u001B[0m: \u001B[32m'http://hl7.org/fhir/StructureDefinition/patient-mothersMaidenName'\u001B[0m,\n", " \u001B[32m'valueString'\u001B[0m: \u001B[32m'Deadra347 Borer986'\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\u001B[32m'url'\u001B[0m: \u001B[32m'http://hl7.org/fhir/us/core/StructureDefinition/us-core-birthsex'\u001B[0m, \u001B[32m'valueCode'\u001B[0m: \u001B[32m'M'\u001B[0m\u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'url'\u001B[0m: \u001B[32m'http://hl7.org/fhir/StructureDefinition/patient-birthPlace'\u001B[0m,\n", " \u001B[32m'valueAddress'\u001B[0m: \u001B[1m{\u001B[0m\u001B[32m'city'\u001B[0m: \u001B[32m'Billerica'\u001B[0m, \u001B[32m'state'\u001B[0m: \u001B[32m'Massachusetts'\u001B[0m, \u001B[32m'country'\u001B[0m: \u001B[32m'US'\u001B[0m\u001B[1m}\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'url'\u001B[0m: \u001B[32m'http://synthetichealth.github.io/synthea/disability-adjusted-life-years'\u001B[0m,\n", " \u001B[32m'valueDecimal'\u001B[0m: \u001B[1;36m14.062655945052095\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'url'\u001B[0m: \u001B[32m'http://synthetichealth.github.io/synthea/quality-adjusted-life-years'\u001B[0m,\n", " \u001B[32m'valueDecimal'\u001B[0m: \u001B[1;36m58.93734405494791\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'identifier'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\u001B[32m'system'\u001B[0m: \u001B[32m'https://github.com/synthetichealth/synthea'\u001B[0m, \u001B[32m'value'\u001B[0m: \u001B[32m'2fa15bc7-8866-461a-9000-f739e425860a'\u001B[0m\u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'type'\u001B[0m: \u001B[1m{\u001B[0m\n", " \u001B[32m'coding'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'system'\u001B[0m: \u001B[32m'http://terminology.hl7.org/CodeSystem/v2-0203'\u001B[0m,\n", " \u001B[32m'code'\u001B[0m: \u001B[32m'MR'\u001B[0m,\n", " \u001B[32m'display'\u001B[0m: \u001B[32m'Medical Record Number'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'text'\u001B[0m: \u001B[32m'Medical Record Number'\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[32m'system'\u001B[0m: \u001B[32m'http://hospital.smarthealthit.org'\u001B[0m,\n", " \u001B[32m'value'\u001B[0m: \u001B[32m'2fa15bc7-8866-461a-9000-f739e425860a'\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'type'\u001B[0m: \u001B[1m{\u001B[0m\n", " \u001B[32m'coding'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'system'\u001B[0m: \u001B[32m'http://terminology.hl7.org/CodeSystem/v2-0203'\u001B[0m,\n", " \u001B[32m'code'\u001B[0m: \u001B[32m'SS'\u001B[0m,\n", " \u001B[32m'display'\u001B[0m: \u001B[32m'Social Security Number'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'text'\u001B[0m: \u001B[32m'Social Security Number'\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[32m'system'\u001B[0m: \u001B[32m'http://hl7.org/fhir/sid/us-ssn'\u001B[0m,\n", " \u001B[32m'value'\u001B[0m: \u001B[32m'999-93-7537'\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'type'\u001B[0m: \u001B[1m{\u001B[0m\n", " \u001B[32m'coding'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'system'\u001B[0m: \u001B[32m'http://terminology.hl7.org/CodeSystem/v2-0203'\u001B[0m,\n", " \u001B[32m'code'\u001B[0m: \u001B[32m'DL'\u001B[0m,\n", " \u001B[32m'display'\u001B[0m: \u001B[32m\"Driver's License\"\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'text'\u001B[0m: \u001B[32m\"Driver's License\"\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[32m'system'\u001B[0m: \u001B[32m'urn:oid:2.16.840.1.113883.4.3.25'\u001B[0m,\n", " \u001B[32m'value'\u001B[0m: \u001B[32m'S99948707'\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'type'\u001B[0m: \u001B[1m{\u001B[0m\n", " \u001B[32m'coding'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'system'\u001B[0m: \u001B[32m'http://terminology.hl7.org/CodeSystem/v2-0203'\u001B[0m,\n", " \u001B[32m'code'\u001B[0m: \u001B[32m'PPN'\u001B[0m,\n", " \u001B[32m'display'\u001B[0m: \u001B[32m'Passport Number'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'text'\u001B[0m: \u001B[32m'Passport Number'\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[32m'system'\u001B[0m: \u001B[32m'http://standardhealthrecord.org/fhir/StructureDefinition/passportNumber'\u001B[0m,\n", " \u001B[32m'value'\u001B[0m: \u001B[32m'X14078167X'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'name'\u001B[0m: \u001B[1m[\u001B[0m\u001B[1m{\u001B[0m\u001B[32m'use'\u001B[0m: \u001B[32m'official'\u001B[0m, \u001B[32m'family'\u001B[0m: \u001B[32m'Brekke496'\u001B[0m, \u001B[32m'given'\u001B[0m: \u001B[1m[\u001B[0m\u001B[32m'Aaron697'\u001B[0m\u001B[1m]\u001B[0m, \u001B[32m'prefix'\u001B[0m: \u001B[1m[\u001B[0m\u001B[32m'Mr.'\u001B[0m\u001B[1m]\u001B[0m\u001B[1m}\u001B[0m\u001B[1m]\u001B[0m,\n", " \u001B[32m'telecom'\u001B[0m: \u001B[1m[\u001B[0m\u001B[1m{\u001B[0m\u001B[32m'system'\u001B[0m: \u001B[32m'phone'\u001B[0m, \u001B[32m'value'\u001B[0m: \u001B[32m'555-677-3119'\u001B[0m, \u001B[32m'use'\u001B[0m: \u001B[32m'home'\u001B[0m\u001B[1m}\u001B[0m\u001B[1m]\u001B[0m,\n", " \u001B[32m'gender'\u001B[0m: \u001B[32m'male'\u001B[0m,\n", " \u001B[32m'birthDate'\u001B[0m: \u001B[32m'1945-12-10'\u001B[0m,\n", " \u001B[32m'address'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'extension'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'url'\u001B[0m: \u001B[32m'http://hl7.org/fhir/StructureDefinition/geolocation'\u001B[0m,\n", " \u001B[32m'extension'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\u001B[32m'url'\u001B[0m: \u001B[32m'latitude'\u001B[0m, \u001B[32m'valueDecimal'\u001B[0m: \u001B[1;36m41.93879298871088\u001B[0m\u001B[1m}\u001B[0m,\n", " \u001B[1m{\u001B[0m\u001B[32m'url'\u001B[0m: \u001B[32m'longitude'\u001B[0m, \u001B[32m'valueDecimal'\u001B[0m: \u001B[1;36m-71.06682353144593\u001B[0m\u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'line'\u001B[0m: \u001B[1m[\u001B[0m\u001B[32m'894 Brakus Bypass'\u001B[0m\u001B[1m]\u001B[0m,\n", " \u001B[32m'city'\u001B[0m: \u001B[32m'Taunton'\u001B[0m,\n", " \u001B[32m'state'\u001B[0m: \u001B[32m'Massachusetts'\u001B[0m,\n", " \u001B[32m'postalCode'\u001B[0m: \u001B[32m'02718'\u001B[0m,\n", " \u001B[32m'country'\u001B[0m: \u001B[32m'US'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'maritalStatus'\u001B[0m: \u001B[1m{\u001B[0m\n", " \u001B[32m'coding'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\u001B[32m'system'\u001B[0m: \u001B[32m'http://terminology.hl7.org/CodeSystem/v3-MaritalStatus'\u001B[0m, \u001B[32m'code'\u001B[0m: \u001B[32m'S'\u001B[0m, \u001B[32m'display'\u001B[0m: \u001B[32m'S'\u001B[0m\u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m,\n", " \u001B[32m'text'\u001B[0m: \u001B[32m'S'\u001B[0m\n", " \u001B[1m}\u001B[0m,\n", " \u001B[32m'multipleBirthBoolean'\u001B[0m: \u001B[3;91mFalse\u001B[0m,\n", " \u001B[32m'communication'\u001B[0m: \u001B[1m[\u001B[0m\n", " \u001B[1m{\u001B[0m\n", " \u001B[32m'language'\u001B[0m: \u001B[1m{\u001B[0m\n", " \u001B[32m'coding'\u001B[0m: \u001B[1m[\u001B[0m\u001B[1m{\u001B[0m\u001B[32m'system'\u001B[0m: \u001B[32m'urn:ietf:bcp:47'\u001B[0m, \u001B[32m'code'\u001B[0m: \u001B[32m'en-US'\u001B[0m, \u001B[32m'display'\u001B[0m: \u001B[32m'English'\u001B[0m\u001B[1m}\u001B[0m\u001B[1m]\u001B[0m,\n", " \u001B[32m'text'\u001B[0m: \u001B[32m'English'\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m}\u001B[0m\n", " \u001B[1m]\u001B[0m\n", "\u001B[1m}\u001B[0m\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "print(synthea_fetcher.get_example_resource('Patient'))" ] }, { "cell_type": "markdown", "id": "b7e2678c", "metadata": {}, "source": [ "### Try it yourself\n", "\n", "Using FHIRPath, create the necessary dataframes to answer the following questions:\n", "\n", "1. How many patients in the dataset have ever received a flu vaccine?\n", "2. What are the five most common conditions that patients have been diagnosed with? (Use only the first diagnosis of a given condition for each patient.)\n", "3. What is the most common medication (in MedicationRequest), and what are the top 5 encounter types associated with these medications?\n", "\n", "Remember that you can look at the [FHIR resource documentation](https://hl7.org/fhir/R4/resourcelist.html) to see what data elements are in each resource. You can also use `synthea_fetcher.get_example_resource('ResourceTypeHere')` with for testing out FHIRPaths if needed." ] }, { "cell_type": "markdown", "id": "4ffa5d01", "metadata": {}, "source": [ "## Summary\n", "\n", "Through this exercise we built a reusable tool to connect to a FHIR server with Bulk Data capabilities, export a set of resource types, and convert that data into DataFrames for analysis.\n", "\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.3" } }, "nbformat": 4, "nbformat_minor": 5 }