{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Module 6\n", "\n", "## Video 26: Inspecting Cargo Movements\n", "**Python for the Energy Industry**\n", "\n", "In this lesson, we take a closer look at the structure of a Cargo Movement. \n", "\n", "[Cargo movements documentation.](https://vortechsa.github.io/python-sdk/endpoints/cargo_movements/)" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# imports\n", "from datetime import datetime\n", "from dateutil.relativedelta import relativedelta\n", "import pandas as pd\n", "import vortexasdk as v\n", "now = datetime.utcnow()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's start with a basic query to the SDK, for all Cargo Movements that are currently loading. We then look at one of these:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'cargo_movement_id': '00a8b809a62f1b844d9586da3bad219e00a8b809a62f1b844d9586da3bad219e',\n", " 'quantity': 263385,\n", " 'status': 'loading_state',\n", " 'vessels': [{'id': '17fa9b4bcddf35051ba9aa2d1b04c6b4bba7eeefaa522a45dd07ec8e894416bb',\n", " 'mmsi': 248159000,\n", " 'imo': 9391957,\n", " 'name': 'ZAPPHIRE',\n", " 'dwt': 47329,\n", " 'cubic_capacity': 52466,\n", " 'vessel_class': 'handymax',\n", " 'corporate_entities': [{'id': 'bd551ce360a4175771b740a4efcce179d37992bf532c84358a115f78a2354198',\n", " 'label': 'So.Co.Mar',\n", " 'layer': 'commercial_owner',\n", " 'probability': 1,\n", " 'source': 'external'}],\n", " 'start_timestamp': '2021-02-08T04:19:05+0000',\n", " 'fixture_fulfilled': False,\n", " 'voyage_id': '9bd55e30686409e92e0aee35598af38614c0053da7efe4afb9b8d950c297227d',\n", " 'tags': [{'tag': 'vessel_coated_tag'}],\n", " 'status': 'vessel_status_laden_known',\n", " 'year': 2010,\n", " 'scrubber': [],\n", " 'flag': [{'tag': 'vessel_flag_tag',\n", " 'flag': 'MT',\n", " 'flag_country': '80dd61da7ce1edccaa43d2d60207c482e397bdd7f7efe7ad2a222d35dde2bc8c'}]}],\n", " 'product': [{'id': '5de0b00094e0fd7542c10f9f8a71b4008d55750f21dc905cda9b0f7f5f76bc08',\n", " 'layer': 'group',\n", " 'probability': 0.8644994,\n", " 'source': 'model',\n", " 'label': 'Dirty Petroleum Products'},\n", " {'id': '1c107b4317bc2c85fb6c13cd7b28e8e0a02ec7fecc68afc2b68ca0545c835e1c',\n", " 'layer': 'group_product',\n", " 'probability': 0.8644994,\n", " 'source': 'model',\n", " 'label': 'Fuel Oil'},\n", " {'id': '3fe831fb60183af885bc789b0224adfecfd3d911e9d2549981c1d7c092d30154',\n", " 'layer': 'category',\n", " 'probability': 0.8644994,\n", " 'source': 'model',\n", " 'label': 'High Sulphur Fuel Oil'}],\n", " 'events': [{'event_type': 'cargo_port_load_event',\n", " 'location': [{'id': 'ee1de4914cc26e8f1326b49793b089131870d478714c07e0c99c56cb307704c5',\n", " 'layer': 'country',\n", " 'label': 'Italy',\n", " 'source': 'model',\n", " 'probability': 1},\n", " {'id': '29005fdc7273f96d4598adc2a8619b796875bd652b4c5a2b97b3283b92ca8dc8',\n", " 'layer': 'port',\n", " 'label': 'Augusta [IT]',\n", " 'source': 'model',\n", " 'probability': 1},\n", " {'id': 'f39d455f5d38907394d6da3a91da4e391f9a34bd6a17e826d6042761067e88f4',\n", " 'layer': 'region',\n", " 'label': 'Europe',\n", " 'source': 'model',\n", " 'probability': 1},\n", " {'id': '676c3cff1dffd971ae51cd350f231394eeb0ab94eff981432f6ee55c5219b913',\n", " 'layer': 'shipping_region',\n", " 'label': 'UKC-Med region',\n", " 'source': 'model',\n", " 'probability': 1},\n", " {'id': 'a0a9eba4c7b8c1e853c207f32d67c4959b6197c39785e037abad42e90824c088',\n", " 'layer': 'shipping_region',\n", " 'label': 'West Mediterranean',\n", " 'source': 'model',\n", " 'probability': 1},\n", " {'id': '6aa9a4fb76ca378e75eff85f458c90f7fd42778c26c957c417490c097161a3c5',\n", " 'layer': 'trading_region',\n", " 'label': 'Mediterranean (incl. North Africa)',\n", " 'source': 'model',\n", " 'probability': 1},\n", " {'id': 'ee1de4914cc26e8f1326b49793b089131870d478714c07e0c99c56cb307704c5',\n", " 'layer': 'trading_subregion',\n", " 'label': 'Italy',\n", " 'source': 'model',\n", " 'probability': 1},\n", " {'id': '5e5deb2ca3810bf061e7cc540bb6a352651ab1f8b2feced92d285320a43b1468',\n", " 'layer': 'terminal',\n", " 'label': 'Augusta Refinery ESSO',\n", " 'source': 'model',\n", " 'probability': 1}],\n", " 'probability': 1,\n", " 'pos': [15.193458283160899, 37.20789046681706],\n", " 'start_timestamp': '2021-02-08T04:19:05+0000',\n", " 'end_timestamp': '2021-02-09T16:15:06+0000'}],\n", " 'parent_ids': []}" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# basic query\n", "cm_query = v.CargoMovements().search(\n", " filter_activity=\"loading_state\",\n", " filter_time_min=now,\n", " filter_time_max=now)\n", "\n", "# remember - cm_query is a *list* of cargo movements\n", "\n", "# taking a look at a single Cargo Movement\n", "cm_query[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There's a lot of information here. Note that this is a 'dictionary' structure, so we can put out the top level keys, and their type:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dict_keys(['cargo_movement_id', 'quantity', 'status', 'vessels', 'product', 'events', 'parent_ids'])" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cm_query[0].keys()" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[, , , , , , ]\n" ] } ], "source": [ "print([type(cm_query[0][cmk]) for cmk in cm_query[0]])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Three of these keys correspond to individual vales which we can print out:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cargo_movement_id: 00a8b809a62f1b844d9586da3bad219e00a8b809a62f1b844d9586da3bad219e\n", "quantity: 263385\n", "status: loading_state\n" ] } ], "source": [ "print('cargo_movement_id:', cm_query[0]['cargo_movement_id'])\n", "print('quantity:', cm_query[0]['quantity'])\n", "print('status:', cm_query[0]['status'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The remaning keys are lists, which we now now describe in turn.\n", "\n", "`vessels` contains a `VesselEntity` for each vessel involved in the Cargo Movement. In general, there can be multiple vessels, but we shall touch on this later. The Cargo Movement we're looking only contains one `VesselEntity`:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vessels = cm_query[1]['vessels']\n", "len(vessels)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'id': 'c101bba621fa1a4f3a1d1f57c3b800f6976def9448a31e42677ddbb0ea705604',\n", " 'mmsi': 538003436,\n", " 'imo': 9515436,\n", " 'name': 'M.STAR',\n", " 'dwt': 314016,\n", " 'cubic_capacity': 344553,\n", " 'vessel_class': 'vlcc_plus',\n", " 'corporate_entities': [{'id': 'd4a185c32e0e38bb5fe45b23f6d880f35af53a6e4337e74648515832875f24f4',\n", " 'label': 'SK',\n", " 'layer': 'charterer',\n", " 'probability': 1,\n", " 'source': 'external'},\n", " {'id': '1ed67e6d2c516dc0867162777e1b2e042ed8f4eed6602865402930d853cf1f66',\n", " 'label': 'SK GROUP',\n", " 'layer': 'commercial_owner',\n", " 'probability': 1,\n", " 'source': 'external'}],\n", " 'start_timestamp': '2021-02-07T23:02:19+0000',\n", " 'fixture_id': '29e6c45ded5d41ee57a6dc46881e2998477e99d329edd8b9d0f867ef343ea431',\n", " 'fixture_fulfilled': True,\n", " 'voyage_id': '2192d7a78c8ccf617212a1856e4bd43b97992ddd9053ff0dcd35872b9fa68219',\n", " 'tags': [],\n", " 'status': 'vessel_status_laden_known',\n", " 'year': 2008,\n", " 'scrubber': [{'tag': 'vessel_scrubber_tag',\n", " 'scrubber': '478fca39000c49d6',\n", " 'planned': False}],\n", " 'flag': [{'tag': 'vessel_flag_tag',\n", " 'flag': 'MH',\n", " 'flag_country': 'e084d7507dab6604894c203b3834cd7ce8f16385daeae56e202cd9e930f788d2'}]}" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "vessels[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`products` contains Product Entries, which each describe one layer of the product tree (Group, Group Product, Category, and Grade). Not all products specify a Grade. We can view the Product Entries in a DataFrame:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idlayerprobabilitysourcelabel
05de0b00094e0fd7542c10f9f8a71b4008d55750f21dc90...group0.864499modelDirty Petroleum Products
11c107b4317bc2c85fb6c13cd7b28e8e0a02ec7fecc68af...group_product0.864499modelFuel Oil
23fe831fb60183af885bc789b0224adfecfd3d911e9d254...category0.864499modelHigh Sulphur Fuel Oil
\n", "
" ], "text/plain": [ " id layer \\\n", "0 5de0b00094e0fd7542c10f9f8a71b4008d55750f21dc90... group \n", "1 1c107b4317bc2c85fb6c13cd7b28e8e0a02ec7fecc68af... group_product \n", "2 3fe831fb60183af885bc789b0224adfecfd3d911e9d254... category \n", "\n", " probability source label \n", "0 0.864499 model Dirty Petroleum Products \n", "1 0.864499 model Fuel Oil \n", "2 0.864499 model High Sulphur Fuel Oil " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "products = cm_query[0]['product']\n", "pd.DataFrame(products)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`events` contains Cargo Events, which each describe a specific event happening to the cargo at a specific time. These can be:\n", "- cargo_port_load_event\n", "- cargo_port_unload_event\n", "- cargo_fso_load_event\n", "- cargo_fso_unload_event\n", "- cargo_sts_event\n", "- cargo_fixture_event\n", "- cargo_storage_event\n", "\n", "In our example, the two events are:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['cargo_port_load_event']" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "events = cm_query[1]['events']\n", "[e['event_type'] for e in events]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see some high level information about these events by putting them into a DataFrame:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
event_typelocationprobabilityposstart_timestampend_timestamp
0cargo_port_load_event[{'id': '6253047d839a51684f2ab6e3a4fa9956f582c...1[48.32317414902807, 29.142766459317272]2021-02-07T23:02:19+00002021-02-09T14:00:10+0000
\n", "
" ], "text/plain": [ " event_type location \\\n", "0 cargo_port_load_event [{'id': '6253047d839a51684f2ab6e3a4fa9956f582c... \n", "\n", " probability pos \\\n", "0 1 [48.32317414902807, 29.142766459317272] \n", "\n", " start_timestamp end_timestamp \n", "0 2021-02-07T23:02:19+0000 2021-02-09T14:00:10+0000 " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.DataFrame(events)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The 'location' entries are themselves dictionaries, which we can expand into a DataFrame to see the different layers:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idlayerlabelsourceprobability
06253047d839a51684f2ab6e3a4fa9956f582c73dfbbb27...countryKuwaitmodel1
1f8c1ff7397acf5d2e353a369fa399b310591680c7a862c...portMina Al Ahmadi [KW]model1
280aa9e4f3014c3d96559c8e642157edbb2b684ea0144ed...regionMiddle Eastmodel1
30899599f74faadb7ba7eb65205ee5c20cb434367a6e720...shipping_regionMEG/AGmodel1
45057dafe08229da478858da705209d61d88db8dcaadf83...trading_blockOPECmodel1
5a7536c48714140c7ba8e8895cdcccc12ebb2c4813720d6...trading_blockOPEC + Russiamodel1
60899599f74faadb7ba7eb65205ee5c20cb434367a6e720...trading_regionMEG/AGmodel1
76253047d839a51684f2ab6e3a4fa9956f582c73dfbbb27...trading_subregionKuwaitmodel1
81d4d3d8565ea23172f9ea72d3daebc642e835171bccf26...terminalKNPC SBMsmodel1
\n", "
" ], "text/plain": [ " id layer \\\n", "0 6253047d839a51684f2ab6e3a4fa9956f582c73dfbbb27... country \n", "1 f8c1ff7397acf5d2e353a369fa399b310591680c7a862c... port \n", "2 80aa9e4f3014c3d96559c8e642157edbb2b684ea0144ed... region \n", "3 0899599f74faadb7ba7eb65205ee5c20cb434367a6e720... shipping_region \n", "4 5057dafe08229da478858da705209d61d88db8dcaadf83... trading_block \n", "5 a7536c48714140c7ba8e8895cdcccc12ebb2c4813720d6... trading_block \n", "6 0899599f74faadb7ba7eb65205ee5c20cb434367a6e720... trading_region \n", "7 6253047d839a51684f2ab6e3a4fa9956f582c73dfbbb27... trading_subregion \n", "8 1d4d3d8565ea23172f9ea72d3daebc642e835171bccf26... terminal \n", "\n", " label source probability \n", "0 Kuwait model 1 \n", "1 Mina Al Ahmadi [KW] model 1 \n", "2 Middle East model 1 \n", "3 MEG/AG model 1 \n", "4 OPEC model 1 \n", "5 OPEC + Russia model 1 \n", "6 MEG/AG model 1 \n", "7 Kuwait model 1 \n", "8 KNPC SBMs model 1 " ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.DataFrame(events[0]['location'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A point to keep in mind**\n", "\n", "In this lesson we have dived into the structure of a Cargo Movement. While important for understanding, in practice it is often simpler to just use cm_query.to_df('all') to convert all these records to a familiar tabular form, without paying attention to the structure. This is true of other endpoints as well." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise\n", "\n", "The Vortexa SDK also offers a `VesselMovements` endpoint. Vessel Movements can be searched for in a similar way to Cargo Movements, but they have some differences. Do a query for Vessel Movements, and inspect the structure of a Vessel Movement to identify the differences.\n", "\n", "[Vessel Movements documentation.](https://vortechsa.github.io/python-sdk/endpoints/vessel_movements/)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.9" } }, "nbformat": 4, "nbformat_minor": 4 }