{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "<< 04-ECMWF Copernicus Climate Change Service (C3S)           Index >>" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "##

Additional Open Data Ressources

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- [Laminar Data Hub API](#laminar-aero)\n", "- [Reading Buses](#reading-buses)\n", "- [Natural Earth](#natural-earth) \n", "- [European Union Open Data Portal](#eu-open-data) \n", "- [European Data Portal](#eu-data-portal)\n", "- [European Environment Agency (EEA)](#eea)\n", "- [UK Office for National Statistics](#uk-statistics)\n", "- [FAOSTAT](#faostat)\n", "- [World Bank Open Data](#worldbank)\n", "- [DIVA-GIS](#diva-gis)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Laminar Data Hub API" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- [1. Service Description](#service-description)\n", "- [2. Available datasets](#Available Datasets)\n", "- [3. How to access the service](#data-access)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1. Service Description" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Laminar Data Hub enables App Developers to build operational applications using developer-friendly APIs to access harmonised flight, weather, aeronautical, NOTAM and value-added data in a securely hosted cloud platform.
Many of the APIs offer XML and GeoJSON response types, and all are accessed via a RESTful HTTP interface. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "User key: 79b5a363698c6d5ab3e94c266555c8cb (NOTE: for #OpenDataHack weekend)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. Available Datasets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Flight Data\n", "Live flight data is available from four different APIs, each offering a different view of the global commercial airspace." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "| Dataset | Description |\n", "|-----------------------|----------------------------------|\n", "|[Flights by Airline](https://developer.laminardata.aero/documentation/flightdata#/)| Retrieves a list of flight summaries for a given airline using an extended version of FIXM 3.0, including scheduled and airborne flights. Summaries include:
- Departure and arrival aerodromes
- Departure and arrival times (estimated and/or actual)
- Positional information (including heading and altitude)
- GUFI (Globally Unique Flight Identifier)
For a list of possible airlines refer to the Reference Data APIs [here](https://developer.laminardata.aero/documentation/referencedata).
The optional flight status parameter allows you to filter flights according to whether they are airborne, cancelled, completed, filed or scheduled.|\n", "|[Flights by GUFI](https://developer.laminardata.aero/documentation/flightdata#/)| Retrieves the most complete picture of a single flight, available in both GeoJSON and an extended version of FIXM 3.0. Completed flights will remain accessible in the API for 6 hours after the flight has landed. Note that the example GUFI below will not work; you will need a current one from one of the other API calls (e.g. Flights by Airline). Flights by GUFI includes:
- Departure and arrival aerodromes
- Departure and arrival times (estimated and/or actual)
- Positional information (including heading, altitude and speed).
To request GeoJSON use the HTTP header \"Accept : application/json\" using the \"Accepts\" dropdown or add the query string \"format=json\" to the request URL.
Note: that full representation of the data model (including units of measure) can be found in the schema repository on GitHub |\n", "|[Flights by Tile](https://developer.laminardata.aero/documentation/flightdata#/)|Retrieves a list of flight summaries for a given tile using an extended version of FIXM 3.0. For more information on tiling see below. The API only returns airborne flights. Summaries include:
- Departure and arrival aerodromes
- Departure and arrival times (estimated and/or actual)
- Positional information (including heading and altitude)
- GUFI (Globally Unique Flight Identifier)
The Flights by Tile API has a GeoJSON response option available. To request GeoJSON use the HTTP header \"Accept : application/json\" using the \"Accepts\" dropdown or add the query string \"format=json\" to the request URL.
Note: that full representation of the data model (including units of measure) can be found in the schema repository on GitHub. |\n", "|[Flights by Aerodrom Pair](https://developer.laminardata.aero/documentation/flightdata#/)|Retrieves a list of flight summaries between a pair of aerodromes using an extended version of FIXM 3.0, including scheduled and airborne flights. Summaries include:
- Departure and arrival aerodromes,
- Departure and arrival times (estimated and/or actual),
- Positional information (including heading and altitude)
- GUFI (Globally Unique Flight Identifier).
The optional flight status parameter allows you to filter flights according to whether they are airborne, cancelled, completed, filed or scheduled. |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Aeronautical Data\n", "All responses for Aeronautical Data use the [AIXM standard version 5.1](http://www.aixm.aero/).\n", "APIs providing access to Route Segments (Europe and USA coverage), Aerodromes (Global Coverage) Airspaces (Europe Coverage), Designated Points (USA and Europe Coverage) and Navaids (USA and Europe Coverage) are all available to develop against. Find more detail on the [Aeronautical Data Documentation](https://developer.laminardata.aero/documentation/aeronauticaldata). " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Regulation Data\n", "Airspace regulations for the European region can be accessed by ICAO prefix or by a specific Regulation ID to retrieve the full Regulation detail. Check out the [Regulation Documentation](https://developer.laminardata.aero/documentation/regulationdata#/) for more information." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Weather Data\n", "Laminar provides access to up to date aviation specific weather data via it’s METAR (Meteorological Aerodrome Report), TAF (Terminal Aerodrome Forecast) and SIGMET (Significant Meteorological Information) APIs. All responses for Weather Data use the [IWXXM standard version 1.1](http://www.wmo.int/pages/prog/www/WIS/wiswiki/tiki-index.php?page=AvXML-1.1-Release#Online_version).\n", "Full detail on the Weather APIs can be found on the [Weather Documentation](https://developer.laminardata.aero/documentation/weatherdata#/) pages. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### NOTAM Data\n", "Laminar provides access to global coverage of NOTAM information via it’s NOTAM by FIR and NOTAM by ID APIs. All responses for NOTAM Data use the [AIXM 5.1 Digital NOTAM Event Specification](http://www.aixm.aero/).\n", "Full detail on the NOTAM APIs can be found on the [NOTAM Data documentation](https://developer.laminardata.aero/documentation/notamdata) pages." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3. How to access the service?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "All APIs are accessed via a standard RESTful HTTPs interface, with a user key required to gain access to the data. \n", "\n", "The basic structure of API requests is as follows: https://api.laminardata.aero/{api_version}/some_dataset/{some_parameter}?user_key={access_key}\n", "\n", "* The request structure for each individual API can be found on the [Laminar Hub Documentation](https://developer.laminardata.aero/) page for each respective API.\n", "\n", "* Visit Laminar Data's [GitHub Repository](https://github.com/laminardata) for a complete list of schemas used to serve the XML data found on the Laminar Data APIs.\n", "\n", "* Check out the [Getting Started](https://developer.laminardata.aero/documentation/gettingstarted) guide and [Data Structure overview](https://developer.laminardata.aero/documentation) for hints on how to get started using Laminar Data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reading Buses" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Available Datasets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Nimbus schedule adherence\n", "APIs giving:
- Bus stop list,
- Live vehicle positions,
- live journey details,
- bus stop predictions,
- list of services,
- line patterns,
- timetabled journeys\n", "\n", "Access: http://rtl2.ods-live.co.uk/cms (Key generation is on board the CMS interface) \n", "User: opendata@reading-buses.co.uk \n", "Password: Reading!!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "NOTE: Time frequency - as soon as we receive an update from a vehicle via SIRI or our on board unit, the new position will be reflected in the feed. \n", "The service provides urls to each service, users make requests and regular intervals if necessary depending on feed. For example user, might make single request to get list of bus stops, or list of services ( these are things that don’t change often). Then user can make regular requests to get list of vehicles or live journeys. Mashing up these data sets and new datasets can then make compelling apps!\n", "The API isn’t rate limited at the moment, however we are putting in functionality over the next couple of months to limit services to x calls per user per minute\n", "\n", "In order to access the API, users first register, then admin user approves the registration, then users can login, go to the API page and press “Generate API Key\", they get to see their key and they can then use it to access data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Mix Telematics \n", "Allows graphic view of live vehicle location, historic trips etc plus export reporting of “events”, trips, mileage, fuel consumption and driver metrics. \n", "\n", "Access: https://uk.mixtelematics.com/#/login \n", "User: jbickerton@reading-buses.co.uk \n", "Password: opendata123\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "NOTE: Graphic displays are pretty, but reports are accessed under “Measure > Reports”, then select a category from the dropdown. “Events” are heavy braking, heavy acceleration, excess idling, as well as others custom-configured such as low coolant, over speeding in the depot etc. The reporting tool can show detail or will aggregate to eg monthly or totals as required, but will timeout if too much data is requested." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Collision reports\n", "Export of insurance claims (fault and non-fault) showing crude locations, vehicle details and anonymised driver data. This report has been redacted to remove personal details and is now open data.\n", "\n", "Access: Reading Claims Summary Report \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Vehicle roadcall export \n", "\n", "List of every vehicle road call, whenever we’ve attended a vehicle at the roadside.\n", "\n", "Access: 201703 Vehicle Defect Road Call Log" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Natural Earth" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Natural Earth is a public domain map dataset available at 1:10m, 1:50m, and 1:110 million scales. Featuring tightly integrated vector and raster data, with Natural Earth you can make a variety of visually pleasing, well-crafted maps with cartography or GIS software. \n", "* http://www.naturalearthdata.com/downloads/ \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## European Union Open Data Portal" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The EU Open Data Portal is your single point of access to a growing range of data produced by the institutions and other bodies of the European Union. Data are free to use, reuse, link and redistribute for commercial or non-commercial purposes. \n", "\n", "* https://data.europa.eu/euodp/en/data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## European Data Portal" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The European Data Portal harvests the metadata of Public Sector Information available on public data portals across European countries. Information regarding the provision of data and the benefits of re-using data is also included. \n", "\n", "* https://www.europeandataportal.eu/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## European Environment Agency (EEA)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* http://www.eea.europa.eu/data-and-maps" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## UK Office for National Statistics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The UK's largest independent producer of official statistics and the recognised national statistical institute of the UK.\n", "\n", "* https://www.ons.gov.uk/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Food and Agricultural Organisation (FAO) - FAOSTAT" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "FAOSTAT provides free access to food and agriculture data for over 245 countries and territories and covers all FAO regional groupings \n", "from 1961 to the most recent year available.\n", "\n", "* http://www.fao.org/faostat/en/#home" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## World Bank Open Data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Free and open access to global development data - http://datacatalog.worldbank.org/. \n", "* [Data Catalog API](https://datahelpdesk.worldbank.org/knowledgebase/articles/902049-data-catalog-api) \n", "* [Climate Data API](https://datahelpdesk.worldbank.org/knowledgebase/articles/902061-climate-data-api)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Data access from Climate Data API" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[ 145.2147442 152.1000659 132.2469297 95.78192808 65.68873722\n", " 46.7608082 47.37453042 42.63793169 49.62307584 58.67578397\n", " 86.57668973 136.4244906 ]\n", " [ 62.75995045 66.69719343 66.16181581 39.0572022 37.75383592\n", " 34.66583859 29.49465083 29.05384564 25.57699883 21.25628535\n", " 23.16091196 30.46213946]\n", " [ 125.1694013 120.7918709 95.38714592 76.13402103 55.29549028\n", " 42.11043359 32.27587123 26.72897891 23.70789177 32.0979657\n", " 54.95476521 99.66923158]\n", " [ 90.60182995 99.41776593 72.44025636 50.50446155 36.17323923\n", " 24.47993501 20.10282582 12.30734975 11.39511207 12.63417895\n", " 23.78926725 53.76818727]\n", " [ 72.19420139 69.37458183 51.2219884 24.60559168 26.2954302\n", " 30.27406113 27.9851704 26.03990667 19.42956093 15.67589149\n", " 25.05198119 53.15280293]\n", " [ 110.70115751 130.25147978 81.51701366 41.13893758 24.35977013\n", " 29.58668308 24.68106955 19.91794492 14.09036337 19.37600472\n", " 38.33939815 77.84636611]\n", " [ 156.25305328 147.69428309 121.47358039 52.28086881 30.67290293\n", " 32.49736857 36.02608389 30.22172125 25.92128851 34.34684787\n", " 79.30232858 132.64596802]\n", " [ 69.73203261 60.14235423 49.25748765 45.33664723 39.84309673\n", " 30.76229954 28.73984734 30.24678104 26.21868093 32.45583474\n", " 43.49425144 61.12152046]\n", " [ 28.42667264 30.07046372 34.34868024 26.48537367 21.19086709\n", " 28.3150413 25.7101327 19.54393007 13.2807603 11.0003176\n", " 9.76089314 11.278295 ]\n", " [ 124.65992508 103.02928855 114.76784193 67.23532498 46.13047096\n", " 46.83001003 46.68017798 43.1850116 44.05344712 58.29094346\n", " 76.41233465 109.89653953]\n", " [ 103.72972447 109.02453875 92.66296046 45.16087545 37.3367949\n", " 37.69562374 30.31441395 25.74929062 26.01021124 27.67369076\n", " 50.03091165 92.91707203]\n", " [ 79.75841159 84.74859562 66.19650085 37.20875454 40.40065218\n", " 32.39479708 24.29867471 20.04312777 11.44024818 12.47644358\n", " 25.00874602 58.79720462]\n", " [ 50.01958102 60.06018988 42.9568403 19.20301046 19.49039753\n", " 21.50563722 15.5427347 12.85522672 8.23939318 8.23687064\n", " 14.25651598 25.80648332]\n", " [ 112.21491722 103.9517652 82.14369948 51.89847249 38.39832576\n", " 34.04840952 36.84725611 33.47028715 24.37672043 26.04432369\n", " 42.27317052 85.0390786 ]\n", " [ 104.09164574 92.05418331 100.04284676 65.49338068 43.77720391\n", " 29.95268492 30.90813913 26.4695272 24.05359523 27.50542281\n", " 34.26905133 58.58654718]]\n", "[[ 152.1000659 ]\n", " [ 66.69719343]\n", " [ 120.7918709 ]\n", " [ 99.41776593]\n", " [ 69.37458183]\n", " [ 130.25147978]\n", " [ 147.69428309]\n", " [ 60.14235423]\n", " [ 30.07046372]\n", " [ 103.02928855]\n", " [ 109.02453875]\n", " [ 84.74859562]\n", " [ 60.06018988]\n", " [ 103.9517652 ]\n", " [ 92.05418331]]\n" ] } ], "source": [ "import requests\n", "import fastkml.kml as kml\n", "import json\n", "import numpy as np\n", "\n", "climateApi2 = requests.get(\"http://climatedataapi.worldbank.org/climateweb/rest/v1/country/kml/AUS\")\n", "# Put loaded json kml information to a Python KML object\n", "k = kml.KML()\n", "k.from_string(climateApi2.text)\n", "\n", "# Write the KML object to disc\n", "outputFile = file(\"test.kml\",\"w\")\n", "outputFile.write(k.to_string())\n", "outputFile.close()\n", "\n", "# Function loadDataFromClimateApi that puts together the url needed for the Worldbank's Climate Data API. It returns a list of python dictionaries.\n", "def loadDataFromClimateApi(url,level,statType,var,start,end,code):\n", " climateApi = requests.get(url+level+\"/\"+statType+\"/\"+var+\"/\"+start+\"/\"+end+\"/\"+code)\n", " data = json.loads(climateApi.text)\n", " return data\n", "\n", "# Load monthly mean precipitation time series for Australia between 1980 and 1999 --> test is a list of python dictionaries\n", "test = loadDataFromClimateApi(url=\"http://climatedataapi.worldbank.org/climateweb/rest/v1/\",\n", " level=\"country\",\n", " statType=\"mavg\",\n", " var=\"pr\",\n", " start=\"1980\",\n", " end=\"1999\",\n", " code=\"AUS\")\n", "\n", "# Data from a list of python dictionaries are retrieved by iterating through the list and then calling the required dictionary entry\n", "testList = []\n", "for item in test:\n", " testList.append(item[u'monthVals'])\n", "\n", "# Convert the list into a numpy array\n", "test2 = np.matrix(testList)\n", "print (test2)\n", "print(test2[:,1])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## DIVA-GIS" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "DIVA-GIS provides free spatial data of the whole world. \n", "\n", "* http://www.diva-gis.org/Data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "© 2017 ECMWF\n", "\n", "This software is licensed under the terms of the Apache Licence Version 2.0 which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. In applying this licence, ECMWF does not waive the privileges and immunities granted to it by virtue of its status as an intergovernmental organisation nor\n", "does it submit to any jurisdiction." ] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.8" } }, "nbformat": 4, "nbformat_minor": 0 }