{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", " \n", " \n", " \n", "
\n", " National generation capacity: Check notebook\n", " \n", "
This Notebook is part of the National Generation Capacity Datapackage of Open Power System Data.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Table of Contents\n", "* [1. Introductory notes](#1.-Introductory-notes)\n", "* [2. Script setup](#2.-Script-setup)\n", "* [3. Import of processed data](#3.-Import-of-processed-data)\n", "* [4. Visualisation of results for different energy source levels](#4.-Visualisation-of-results-for-different-energy-source-levels)\n", "\t* [4.1 Energy source level 1](#4.1-Energy-source-level-1)\n", "\t\t* [4.1.1 Table](#4.1.1-Table)\n", "\t\t* [4.1.2 Bokeh chart](#4.1.2-Bokeh-chart)\n", "\t* [4.2 Energy source level 2](#4.2-Energy-source-level-2)\n", "\t\t* [4.2.1 Table](#4.2.1-Table)\n", "\t\t* [4.2.2 Bokeh chart](#4.2.2-Bokeh-chart)\n", "\t* [4.3 Energy source level 3](#4.3-Energy-source-level-3)\n", "\t\t* [4.3.1 Table](#4.3.1-Table)\n", "\t\t* [4.3.2 Bokeh chart](#4.3.2-Bokeh-chart)\n", "\t* [4.4 Technology level](#4.4-Technology-level)\n", "\t\t* [4.4.1 Table](#4.4.1-Table)\n", "\t\t* [4.4.2 Bokeh chart](#4.4.2-Bokeh-chart)\n", "* [5. Comparison of total capacity for energy source levels](#5.-Comparison-of-total-capacity-for-energy-source-levels)\n", "\t* [5.1 Calculation of total capacity for energy source levels](#5.1-Calculation-of-total-capacity-for-energy-source-levels)\n", "\t* [5.2 Identifcation of capacity differences for energy source levels](#5.2-Identifcation-of-capacity-differences-for-energy-source-levels)\n" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "# 1. Introductory notes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The notebook extends the [processing notebook](processing.ipynb) to make visualisations and perform consistency checks." ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "# 2. Script setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import os.path\n", "import logging\n", "\n", "from bokeh.charts import Bar, output_file, show\n", "from bokeh.io import output_notebook\n", "from bokeh.models import HoverTool, NumeralTickFormatter\n", "from bokeh.charts.attributes import color\n", "\n", "output_notebook()\n", "\n", "%matplotlib inline\n", "\n", "logging.basicConfig(\n", " level=logging.INFO,\n", " format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',\n", " datefmt='%d %b %Y %H:%M:%S'\n", " )\n", "\n", "logger = logging.getLogger()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3. Import of processed data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The processed data is imported at this stage. This requires that the [processing notebook](processing.ipynb) is compiled prior to this step. Otherwise, no or an old dataset is imported." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data_file = 'national_generation_capacity_stacked.csv'\n", "filepath = os.path.join('output', data_file)\n", "data = pd.read_csv(filepath, index_col=0)\n", "\n", "data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4. Visualisation of results for different energy source levels" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the following, national generation capacity is compared for the different energy source levels. Due to the size of the dataset regarding countries and compiled datasources, the following tables and figures tend to be rather confusing. Therefore, we provide the following option to limit the visualisation of the results to a selection of countries and years. If the corresponding subset is empty, all values will be considered. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "country_subset = ['DE', 'FR', 'BE', 'NL', 'IT']\n", "year_subset = [2013, 2014, 2015, 2016]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data_selection = pd.DataFrame()\n", "\n", "if len(country_subset) == 0:\n", " data_selection = data\n", "else:\n", " for country in country_subset:\n", " for year in year_subset:\n", " if len(data_selection) == 0:\n", " data_selection = data[(data.country == country) &\n", " (data.year == year)]\n", " else:\n", " data_selection = data_selection.append(data[(data.country == country) &\n", " (data.year == year)])\n", "\n", "data_selection" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To improve the data visualisation in Bokeh the colors of the defined technologies is specified explicitly. The user is free to adjust or refine the color definition using the following parameter. The color names are defined [here](http://www.w3schools.com/colors/colors_names.asp)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "colormap = {\n", " 'Fossil fuels': 'Black',\n", " 'Lignite': 'SaddleBrown',\n", " 'Hard coal': 'Black',\n", " 'Oil': 'Violet',\n", " 'Natural gas': 'IndianRed',\n", " 'Combined cycle': '#d57676',\n", " 'Gas turbine': '#e19d9d',\n", " 'Other and unknown natural gas': '#c33c3c',\n", " 'Differently categorized natural gas': 'IndianRed',\n", " 'Non-renewable waste': 'SandyBrown',\n", " 'Mixed fossil fuels': 'LightGray',\n", " 'Other fossil fuels': 'DarkGray',\n", " 'Differently categorized fossil fuels': 'Gray',\n", " 'Nuclear': 'Red',\n", " 'Renewable energy sources': 'Green',\n", " 'Hydro': 'Navy',\n", " 'Run-of-river': '#0000b3',\n", " 'Reservoir': '#0000e6',\n", " 'Reservoir including pumped storage': '#0000e6',\n", " 'Pumped storage': '#1a1aff',\n", " 'Pumped storage with natural inflow': '#1a1aff',\n", " 'Differently categorized hydro': 'Navy',\n", " 'Wind': 'SkyBlue',\n", " 'Onshore': 'LightSkyBlue',\n", " 'Offshore': 'DeepSkyBlue',\n", " 'Differently categorized wind': 'SkyBlue',\n", " 'Solar': 'Yellow',\n", " 'Photovoltaics': '#ffff33',\n", " 'Concentrated solar power': '#ffff66',\n", " 'Differently categorized solar': 'Yellow',\n", " 'Geothermal': 'DarkRed',\n", " 'Marine': 'Blue',\n", " 'Bioenergy and renewable waste': 'Green',\n", " 'Biomass and biogas': '#00b300',\n", " 'Sewage and landfill gas': '#00e600',\n", " 'Other bioenergy and renewable waste': 'Green',\n", " 'Differently categorized renewable energy sources': 'Green',\n", " 'Other or unspecified energy sources': 'Orange',\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.1 Energy source level 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.1.1 Table" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pivot_capacity_level1 = pd.pivot_table(data_selection[data_selection.energy_source_level_1 == True],\n", " index=('country','year','source'),\n", " columns='technology',\n", " values='capacity',\n", " aggfunc=sum,\n", " margins=False)\n", "\n", "pivot_capacity_level1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.1.2 Bokeh chart" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Please use the zoom and hover option to inspect the data graphically." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "data_energy_level_1 = data_selection[data_selection.energy_source_level_1 == True].copy()\n", "data_energy_level_1['color'] = 'White'\n", "data_energy_level_1['color'] = data_energy_level_1['technology'].map(colormap)\n", "\n", "bar = Bar(data_energy_level_1, values='capacity', label=['country', 'year', 'source'], stack='technology',\n", " title=\"National capacity by type of energy source\", tools=\"pan,wheel_zoom,box_zoom,reset,hover,save\",\n", " legend='top_right', \n", " plot_width=1600, plot_height=800,\n", "# color=color(columns='technology', palette=['Black', 'Red', 'Green', 'Orange'], sort=False))\n", " color='color')\n", "bar._yaxis.formatter = NumeralTickFormatter(format=\"00,000 MW\")\n", "\n", "hover = bar.select_one(HoverTool)\n", "hover.point_policy = \"follow_mouse\"\n", "hover.tooltips = [(\"Country\", \"@country\"),\n", " (\"Year\", \"@year\"),\n", " (\"Source\", \"@source\"),\n", " (\"Category\", \"@technology\"),\n", " (\"Capacity\", \"@height{00,000.00} MW\"),\n", "]\n", "\n", "show(bar)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## 4.2 Energy source level 2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.2.1 Table" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pivot_capacity_level2 = pd.pivot_table(data_selection[data_selection.energy_source_level_2 == True],\n", " index=('country','year','source'),\n", " columns='technology',\n", " values='capacity',\n", " aggfunc=sum,\n", " margins=False)\n", "\n", "pivot_capacity_level2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.2.2 Bokeh chart" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Please use the zoom and hover option to inspect the data graphically." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": false }, "outputs": [], "source": [ "data_energy_level_2 = data_selection[data_selection.energy_source_level_2 == True].copy()\n", "data_energy_level_2['color'] = 'White'\n", "data_energy_level_2['color'] = data_energy_level_2['technology'].map(colormap)\n", "\n", "bar = Bar(data_energy_level_2, values='capacity', label=['country', 'year', 'source'], stack='technology',\n", " title=\"National capacity by energy source\", tools=\"pan,wheel_zoom,box_zoom,reset,hover,save\",\n", " legend='top_right', \n", " plot_width=1600, plot_height=800,\n", " color='color'\n", " )\n", "bar._yaxis.formatter = NumeralTickFormatter(format=\"00,000 MW\")\n", "\n", "hover = bar.select_one(HoverTool)\n", "hover.point_policy = \"follow_mouse\"\n", "hover.tooltips = [(\"Country\", \"@country\"),\n", " (\"Year\", \"@year\"),\n", " (\"Source\", \"@source\"),\n", " (\"Category\", \"@technology\"),\n", " (\"Capacity\", \"@height{00,000.00} MW\"),\n", "]\n", "\n", "show(bar)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.3 Energy source level 3" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.3.1 Table" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pivot_capacity_level3 = pd.pivot_table(data_selection[data_selection.energy_source_level_3 == True],\n", " index=('country', 'year', 'source'),\n", " columns='technology',\n", " values='capacity',\n", " aggfunc=sum,\n", " margins=False)\n", "\n", "pivot_capacity_level3" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.3.2 Bokeh chart" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Please use the zoom and hover option to inspect the data graphically." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data_energy_level_3 = data_selection[data_selection.energy_source_level_3 == True].copy()\n", "data_energy_level_3['color'] = 'White'\n", "data_energy_level_3['color'] = data_energy_level_3['technology'].map(colormap)\n", "\n", "bar = Bar(data_energy_level_3, values='capacity', label=['country', 'year', 'source'], stack='technology',\n", " title=\"National capacity by energy source\", tools=\"pan,wheel_zoom,box_zoom,reset,hover,save\",\n", "# legend='top_right', \n", " plot_width=1600, plot_height=800,\n", " color='color'\n", " )\n", "bar._yaxis.formatter = NumeralTickFormatter(format=\"00,000 MW\")\n", "\n", "hover = bar.select_one(HoverTool)\n", "hover.point_policy = \"follow_mouse\"\n", "hover.tooltips = [(\"Country\", \"@country\"),\n", " (\"Year\", \"@year\"),\n", " (\"Source\", \"@source\"),\n", " (\"Category\", \"@technology\"),\n", " (\"Capacity\", \"@height{00,000.00} MW\"),\n", "]\n", "\n", "show(bar)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.4 Technology level" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.4.1 Table" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pivot_capacity_techlevel = pd.pivot_table(data_selection[data_selection.technology_level == True],\n", " index=('country', 'year', 'source'),\n", " columns='technology',\n", " values='capacity',\n", " aggfunc=sum,\n", " margins=False)\n", "\n", "pivot_capacity_techlevel" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.4.2 Bokeh chart" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Please use the zoom and hover option to inspect the data graphically." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data_technology_level = data_selection[data_selection.technology_level == True].copy()\n", "data_technology_level['color'] = 'White'\n", "data_technology_level['color'] = data_technology_level['technology'].map(colormap)\n", "\n", "bar = Bar(data_technology_level, values='capacity', label=['country', 'year', 'source'], stack='technology',\n", " title=\"National capacity by energy source and technology\", tools=\"pan,wheel_zoom,box_zoom,reset,hover,save\",\n", "# legend='top_right', \n", " plot_width=1600, plot_height=800,\n", " color='color'\n", " )\n", "bar._yaxis.formatter = NumeralTickFormatter(format=\"00,000 MW\")\n", "\n", "hover = bar.select_one(HoverTool)\n", "hover.point_policy = \"follow_mouse\"\n", "hover.tooltips = [(\"Country\", \"@country\"),\n", " (\"Year\", \"@year\"),\n", " (\"Source\", \"@source\"),\n", " (\"Category\", \"@technology\"),\n", " (\"Capacity\", \"@height{00,000.00} MW\"),\n", "]\n", "\n", "show(bar)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 5. Comparison of total capacity for energy source levels" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the following, the installed capacities at the different technology levels are compared to each other. In any case, the total sum of all technologies within a certain technology level should match with other energy source levels. Otherwise the classification of categories to the levels is flawed or the specific data entries are wrong.\n", "\n", "Again, the comparison can be done for specific countries, or, if the selection is empty, for all countries." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "country_subset = []\n", "#country_subset = ['DE', 'FR', 'IT', 'ES']" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "data_selection = pd.DataFrame()\n", "\n", "if len(country_subset) == 0:\n", " data_selection = data\n", "else:\n", " for country in country_subset:\n", " if len(data_selection) == 0:\n", " data_selection = data[data.country == country]\n", " else:\n", " data_selection = data_selection.append(data[data.country == country])\n", "\n", "#data_selection" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5.1 Calculation of total capacity for energy source levels" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Define the columns for grouping\n", "groupby_selection = ['capacity_definition', 'source', 'year', 'type', 'country']\n", "\n", "# Calculate the total capacity of all categories within a certain technology level\n", "capacity_total_0 = pd.DataFrame(data_selection[data_selection['energy_source_level_0'] == True]\n", " .groupby(groupby_selection)['capacity'].sum())\n", "capacity_total_1 = pd.DataFrame(data_selection[data_selection['energy_source_level_1'] == True]\n", " .groupby(groupby_selection)['capacity'].sum())\n", "capacity_total_2 = pd.DataFrame(data_selection[data_selection['energy_source_level_2'] == True]\n", " .groupby(groupby_selection)['capacity'].sum())\n", "capacity_total_3 = pd.DataFrame(data_selection[data_selection['energy_source_level_3'] == True]\n", " .groupby(groupby_selection)['capacity'].sum())\n", "capacity_total_tech = pd.DataFrame(data_selection[data_selection['technology_level'] == True]\n", " .groupby(groupby_selection)['capacity'].sum())\n", "\n", "# Merge calculated capacity for different technology levels\n", "capacity_total_comparison = pd.DataFrame(capacity_total_0)\n", "capacity_total_comparison = pd.merge(capacity_total_0, capacity_total_1, \n", " left_index=True, right_index=True, how='left')\n", "capacity_total_comparison = capacity_total_comparison.rename(columns={'capacity_x': 'energy source level 0',\n", " 'capacity_y': 'energy source level 1'})\n", "\n", "capacity_total_comparison = pd.merge(capacity_total_comparison, capacity_total_2, \n", " left_index=True, right_index=True, how='left')\n", "capacity_total_comparison = pd.merge(capacity_total_comparison, capacity_total_3, \n", " left_index=True, right_index=True, how='left')\n", "capacity_total_comparison = capacity_total_comparison.rename(columns={'capacity_x': 'energy source level 2',\n", " 'capacity_y': 'energy source level 3'})\n", "\n", "capacity_total_comparison = pd.merge(capacity_total_comparison, capacity_total_tech, \n", " left_index=True, right_index=True, how='left')\n", "capacity_total_comparison = capacity_total_comparison.rename(columns={'capacity': 'technology level'})\n", "\n", "# Define sorting preferences\n", "capacity_total_comparison = capacity_total_comparison.sortlevel(['country', 'year'])\n", "\n", "capacity_total_comparison" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5.2 Identifcation of capacity differences for energy source levels" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Identification of differences between energy source levels for each country, source, and year. The difference is relative to the previous energy source level. Generally, differences between the energy source levels should be zero, but could differ in particular for ENTSO-E data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "capacity_total_difference = capacity_total_comparison.diff(periods=1, axis=1)\n", "capacity_total_difference = capacity_total_difference[(capacity_total_difference['energy source level 1'] > 0.01) | \n", " (capacity_total_difference['energy source level 1'] < -0.01) |\n", " (capacity_total_difference['energy source level 2'] > 0.01) | \n", " (capacity_total_difference['energy source level 2'] < -0.01) |\n", " (capacity_total_difference['energy source level 3'] > 0.01) | \n", " (capacity_total_difference['energy source level 3'] < -0.01)|\n", " (capacity_total_difference['technology level'] > 0.01) | \n", " (capacity_total_difference['technology level'] < -0.01)]\n", "capacity_total_difference" ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.1" } }, "nbformat": 4, "nbformat_minor": 1 }