{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", " \n", " \n", " \n", "
\n", " National generation capacity: Processing notebook\n", " \n", "
This Notebook is part of the National Generation Capacity Datapackage of Open Power System Data.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Table of Contents\n", "* [1. Introductory notes](#1.-Introductory-notes)\n", "* [2. Script setup](#2.-Script-setup)\n", "* [3. Data download and processing](#3.-Data-download-and-processing)\n", "\t* [3.1 Manually compiled dataset](#3.1-Manually-compiled-dataset)\n", "\t* [3.2 EUROSTAT data](#3.2-EUROSTAT-data)\n", "\t* [3.3 ENTSO-E data](#3.3-ENTSO-E-data)\n", "\t\t* [3.3.1 ENTSO-E statistical data](#3.3.1-ENTSO-E-statistical-data)\n", "\t\t* [3.3.2 ENTSO-E SO&AF data](#3.3.2-ENTSO-E-SO&AF-data)\n", " * [3.3.3 ENTSO-E Transparency Plaftform](#3.3.1-ENTSO-E-Transparency-Platform)\n", "\t\t* [3.3.4 ENTSO-E Power Statistics](#3.3.4-ENTSO-E-Power-Statistics)\n", "\t* [3.4 Merge data sources](#3.4-Merge-data-sources)\n", "* [4. Convert stacked data to crosstable format](#4.-Convert-stacked-data-to-crosstable-format)\n", "* [5. Output](#5.-Output)\n", "\t* [5.1 Write results to file](#5.1-Write-results-to-file)\n", "\t* [5.2 Formatting of Excel tables](#5.2-Formatting-of-Excel-tables)\n", "\t* [5.3 Write checksums](#5.3-Write-checksums)\n", "* [6. Documentation of the data package](#6.-Documentation-of-the-data-package)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 1. Introductory notes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The script processes the compiled nationally aggregated generation capacity for European countries. Due to varying formats and data specifications of references for national generation capacity, the script firstly focuses on rearranging the manually compiled data. Thus, the script itself does not collect, select, download or manage data from original sources. Secondly, international data sources, such as EUROSTAT and ENTSO-E, are directly downloaded from original web sources and complement the initial data set." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 2. Script setup" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# some functions and classes that are defined in seperate files\n", "import functions.helper_functions as func\n", "import functions.soaf as soaf\n", "\n", "# core packages\n", "import os\n", "import pandas as pd\n", "import numpy as np\n", "\n", "# packages to copy files, write sqllite data bases and manipulate excel files\n", "import shutil\n", "import sqlite3\n", "import openpyxl\n", "from openpyxl.styles import PatternFill, colors, Font, Alignment\n", "from openpyxl.utils import get_column_letter\n", "\n", "import yaml\n", "import json" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3. Data download and processing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We compile data from different national and international sources. Firstly, national data sources are manually compiled due to varying data formats and specifications. Secondly, international sources are compiled directly and appended to the compiled data set. The international data sources comprise:\n", "- [EUROSTAT](http://ec.europa.eu/eurostat/product?code=nrg_113a&mode=view)\n", "- [ENTSO-E Statistical data](https://www.entsoe.eu/data/data-portal/miscellaneous/Pages/default.aspx)\n", "- [ENTSO-E System Outlook and Adequacy Forecast](https://www.entsoe.eu/outlooks/maf/Pages/default.aspx)\n", "- [ENTSO-E Transparency Platform](https://transparency.entsoe.eu/)\n", "- [ENTSO-E Power Statistics](https://www.entsoe.eu/data/power-stats/)\n", "\n", "In the following section, the data sets are downloaded and uploaded to Python." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.1 Manually compiled dataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The manually compiled dataset is imported and rearranged to a DataFrame for further processing. The dataset comprises for each European country and specified generation technology different data entries, which are based on different sources. As these sources differ by country and year, information on the corresponding reference are directly given with the data entry. " ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
12345
technology
NaNEnergy source levels and technologyEnergy source levels and technologyEnergy source levels and technologyEnergy source levels and technologyEnergy source levels and technology
NaNTotal generation capacityGeneration capacity by energy type (fossil, nu...Generation capacity by fuel or energy source (...Generation capacity by fuel or energy source a...Generation capacity by fuel and technology (e....
NaNNaNNaNNaNNaNNaN
NaNNaNNaNNaNNaNNaN
NaNNaNNaNNaNNaNNaN
\n", "
" ], "text/plain": [ " 1 \\\n", "technology \n", "NaN Energy source levels and technology \n", "NaN Total generation capacity \n", "NaN NaN \n", "NaN NaN \n", "NaN NaN \n", "\n", " 2 \\\n", "technology \n", "NaN Energy source levels and technology \n", "NaN Generation capacity by energy type (fossil, nu... \n", "NaN NaN \n", "NaN NaN \n", "NaN NaN \n", "\n", " 3 \\\n", "technology \n", "NaN Energy source levels and technology \n", "NaN Generation capacity by fuel or energy source (... \n", "NaN NaN \n", "NaN NaN \n", "NaN NaN \n", "\n", " 4 \\\n", "technology \n", "NaN Energy source levels and technology \n", "NaN Generation capacity by fuel or energy source a... \n", "NaN NaN \n", "NaN NaN \n", "NaN NaN \n", "\n", " 5 \n", "technology \n", "NaN Energy source levels and technology \n", "NaN Generation capacity by fuel and technology (e.... \n", "NaN NaN \n", "NaN NaN \n", "NaN NaN " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_file = 'National_Generation_Capacities.xlsx'\n", "filepath = os.path.join('input', data_file)\n", "\n", "# Read data into pandas\n", "data_raw = pd.read_excel(filepath,\n", " sheet_name='Summary',\n", " header=None,\n", " na_values=['-'],\n", " skiprows=0)\n", "\n", "\n", "\n", "\n", "# Deal with merged cells from Excel: fill first three rows with information\n", "data_raw.iloc[0:2] = data_raw.iloc[0:2].fillna(method='ffill', axis=1)\n", "\n", "# Set index for rows\n", "data_raw = data_raw.set_index([0])\n", "data_raw.index.name = 'technology'\n", "\n", "# Extract energylevels from raw data for later use\n", "energylevels_raw = data_raw.iloc[:, 0:5]\n", "\n", "energylevels_raw.head()" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countryATBE...SISK
typeInstalled capacity in MWInstalled capacity in MW...Installed capacity in MWInstalled capacity in MW
year2014.02015.02016.02017.02018.02019.02015.02016.02017.02018.0...2015.02016.02017.02013.02014.02015.02016.02017.02018.0NaN
sourcee-controle-controle-controle-controle-controle-controlELIAELIAELIAELIA...Agencija za energijoAgencija za energijoAgencija za energijoStatistical Office of SlovakiaStatistical Office of SlovakiaStatistical Office of SlovakiaStatistical Office of SlovakiaStatistical Office of SlovakiaStatistical Office of Slovakia
source_typeRegulatory AuthorityRegulatory AuthorityRegulatory AuthorityRegulatory AuthorityRegulatory AuthorityRegulatory AuthorityTransmission System OperatorTransmission System OperatorTransmission System OperatorTransmission System Operator...MinistryMinistryMinistryStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeNaN
weblinkhttps://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538https://www.e-control.at/documents/1785851/1811756/BeStGes-2019_KW2EPLTyp.xlsx/e9993ee3-838f-ed2e-2031-7503418fac2d?t=1596094915390https://www.elia.be/en/grid-data/power-generation/generating-facilitieshttps://www.elia.be/en/grid-data/power-generation/generating-facilitieshttps://www.elia.be/en/grid-data/power-generation/generating-facilitieshttps://www.elia.be/en/grid-data/power-generation/generating-facilities...https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2015/f1302ae0-7267-4ae7-b74d-7ce8c4323043https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2016/de8cc94f-b3f6-4d32-8e79-ce3b0ec386f5https://www.agen-rs.si/documents/54870/68629/a/78f74b68-dbfc-415e-ab88-882652558d94Link unavailablehttps://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRZJIHHEGAOxQwwYEuvSASM31AE7jWsnb1_Z8aRtGEM7LQcNsP8Hu6t_hQS6R6LJ99WXfFdtmvxRPS-E-TliAR8OdRuAGSMIrtNR5CWWDhTQ3UngeLZP2RSATz0DAttPYysiBGyChArbURQl0ywDL8NjCIjuQZimADE783DhsuHP-Gw6e0vQY3MdgjBM0oxH5GZOT_x7_nziOCpszEdGFpKZS898h6CLt6GHP-f_M2wwUOFE5Z4lkECbbwnE5fLcELrrPwo6-DTo6Z87Njp5h3bzxx_8E-986P-tY7nKf75pU-bqlDOUIYHEstltdw9osSme8wdNNtr2e3EFanms1vmyklewf5a79euvbwCXukGkoVFJLLWsQLO4WWiSyWVBVyujMI3jt7fLqkQLHUtSyJxpK4xXGjUtS-OlBUpdMpaTEuu8_LiX7WER3Vb_wLeL_b0XsZq3YIh1nzoTwjPW4lsC0TfL4iTpmua-Gt4E42SGbYt6zii-VQM7dzBPpiYG0M-CLsP1WV70nQndgrmJQn9TS5TIBi1Uy9jFnkwwuttX8oDSZvOtVsdh8peO8AFdI1EV9eCwrAcw0DFnJsVqMy1M1P3xfK2-Pj0JW5l40-zkyw7d_3cXb-v0_ao5ea0qbR27sX-4cQkVi08_AHhGIRI!/dz/d5/L2dBISEvZ0FBIS9nQSEh/https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsVZIgDhijIHYIQYMiXXpCAwNdcBOQ-3k7Ss7nrQNY2in1YEB9v9gd_WvEEf3iDdiX30RbbVtxKN8XnH9c2j4bDzGFoChTcC_TiahG5sYKKC7k8B2LY8acwA2dzXwLS-JzJAQsAjiMmyFYRjP0xTcVJ2CT7ALQZIARMaZhwvLgj_j0_niLUHXWGLwgyBOUhaSmyU98e_5s5lty7C2nGhpQBYOPfM9gj7eggH-nP_PsGaADMcy9zSGGLp8R8Avl-cE0F__UdDDJ_5A_5yp1svbtJ8__uCfePtD_29t05H-83SLGg6mzEAp4ojnTbtrH9Bqmz2LB6VolN337Ark5bHaiLwqrmD_XLSb11_fiLUotVKjSi4IVijOiMKKfK0wprFCFQwyio_f3uXVGq0wlEKYTFd0zRAKZSJXmF6aikl1BkIzszIvP-5ld1h4v9U_8N1if-9FJOfNH6vYo_aMsNTo8B0BH5plfpL0TfNQDW-CabxQLZO69iS6lQO7tFUWz3UVAJ8FfYYbsjwfOhP6BUsdBd62LlBcNGglW2Zc7MlMRXf7qjigpNl-q-VxGP-lIzxA14hXWT065PUIRlhlhk5VuZmmSuT98Xytvj49cUuaeNu0xUuL7v-7i3d18r5qRl6rStlETuQdbhxC-erTD_ffAqA!/dz/d5/L2dBISEvZ0FBIS9nQSEh/https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRYkIXHEGAOxQwwYGnTpAMYNdcBOQ-3m7Ss7nrQ1Y2in1YEB9v9gd_WvkEAPSDTZvvqctdW2yZ7kcyr0TwHz-HismgCMTsC7jSeBExkqEEAfTwLLMV3C5gB87lDwTDcOjQBjMDESMmwGQRDNkwScRJuCh1UH_DgGCNmZhyvLhD_jk_niLUGHLVXwfD-KEx7guyU58e_585llyTBdTmji44VNznyPoI83YYA_5_8zTBnIcCRzTyKIoMt3BOJ6ebYP_fUfBT187A30z57SXt4i_fzxB__EWxf9v7cMW_rP1U3CbJVwhhIkkCiadtc-onSbv2SPStkou2_5DcjLU7XJiqq8gf1L2W5ef32TGWrGVsZaWfFMV4hOqMKpvCsxLwzAJIeCHr-9K6oVSiHT8hUjupIX60Ihqk4VyZfKeo1xSTnOoNAu97I7LKLf6hd8t9jfexHKefPGmuoSa4Z5wjp8RyCGZlmcJH3TPFTDm2AaLTTTII41Ce_lwC4tjUdzXQNQz4I-ww1ZXgydCf2CpY58d1uXKCoblMqWsas9mck93VflAcXN9mstj8PoLx3hArpFosrr0aGoRzBSNc50osnNNDQs74_na_Xl-VmY0sTbpi2_t-jhv7t4V8fvq-b4taqUTWiH7uHOxkSkH34AcI_n7A!!/dz/d5/L2dBISEvZ0FBIS9nQSEh/https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZRRd5owFMc_yx76iLmEQJJHRASqpQLCal52AOPKLGhXp-u3X7SebpMjbGfLAwe4_x_ce_O_QQI9INHk--pzvqs2Tf6knhfC-hTRgA2Hug1AzREEt-ko8hKuAwH08SRwPNsndArApp4Jge2nMY8MA2wDCRW2oyhKplkGXobHEBi6B2GaAsT0zMOVZcOf8dl09pagR-c6BGGYpBmLjLs5OfHv-bOJ46iwOR-ZWWjMXHLmOwRdvA09_Dn_n2GTggonKvcsgQTafEsgrpfnhtBd_1HQwadBT__csdnJO6SbP_7gn3jnov_3DneV_3zLJtTVCaMoQwKJstltd49osSle8kdNNtr2W3ED6vJUrfOykjewf5G79euvbzDnJTcLoi1zk2gEl1IruGFpRcFXJsF0xTAcv70tqyVa6IwA6HylSWZxjchlqTFpgsYMbEnMTGtF8eVetodFdFv9gm8X-3svYjVvwRDrPnEmBstoi28JRN8si5Oka5r7angTjJMZtjnxnFF8rwZ27mCWTC2sWngWdBmuz_Ki70zoFswtFPqbWqJENmihWkav9mSi9nRfyQNKm83XWh2HyV86wgd0i0RV1INDWQ9goGNGLYLVZnJsqPvj-Vp9eX4WtjLxptnJ7zv08N9dvK3T91Uz47WqtHXsxv7hzjWIWHz4AZRUN9Q!/dz/d5/L2dBISEvZ0FBIS9nQSEh/https://slovak.statistics.sk/PortalTraffic/fileServlet?Dokument=da19bcb8-8c9c-4757-aba5-2d0720266efa
capacity_definitionGross capacityGross capacityGross capacityGross capacityGross capacityGross capacityUnknownUnknownUnknownUnknown...Net capacityNet capacityNet capacityUnknownUnknownUnknownUnknownUnknownUnknownNaN
technology
Fossil fuels7243.637059.077323646964926041NaNNaNNaNNaN...1442.971450.512233348290725863164.513266.972581.47NaN
- Lignite00NaNNaNNaNNaNNaNNaNNaNNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
- Hard coal11711171NaN10121012598NaNNaNNaNNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
- Oil333.736173.636NaN168168164NaNNaNNaNNaN...NaNNaNNaN140138138NaNNaNNaNNaN
- Natural gas4888.664820.19NaN485348764429NaNNaNNaNNaN...NaNNaNNaN10161022959NaNNaNNaNNaN
\n", "

5 rows × 157 columns

\n", "
" ], "text/plain": [ "country AT \\\n", "type Installed capacity in MW \n", "year 2014.0 \n", "source e-control \n", "source_type Regulatory Authority \n", "weblink https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538 \n", "capacity_definition Gross capacity \n", "technology \n", "Fossil fuels 7243.63 \n", "- Lignite 0 \n", "- Hard coal 1171 \n", "- Oil 333.736 \n", "- Natural gas 4888.66 \n", "\n", "country \\\n", "type \n", "year 2015.0 \n", "source e-control \n", "source_type Regulatory Authority \n", "weblink https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538 \n", "capacity_definition Gross capacity \n", "technology \n", "Fossil fuels 7059.07 \n", "- Lignite 0 \n", "- Hard coal 1171 \n", "- Oil 173.636 \n", "- Natural gas 4820.19 \n", "\n", "country \\\n", "type \n", "year 2016.0 \n", "source e-control \n", "source_type Regulatory Authority \n", "weblink https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538 \n", "capacity_definition Gross capacity \n", "technology \n", "Fossil fuels 7323 \n", "- Lignite NaN \n", "- Hard coal NaN \n", "- Oil NaN \n", "- Natural gas NaN \n", "\n", "country \\\n", "type \n", "year 2017.0 \n", "source e-control \n", "source_type Regulatory Authority \n", "weblink https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538 \n", "capacity_definition Gross capacity \n", "technology \n", "Fossil fuels 6469 \n", "- Lignite NaN \n", "- Hard coal 1012 \n", "- Oil 168 \n", "- Natural gas 4853 \n", "\n", "country \\\n", "type \n", "year 2018.0 \n", "source e-control \n", "source_type Regulatory Authority \n", "weblink https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538 \n", "capacity_definition Gross capacity \n", "technology \n", "Fossil fuels 6492 \n", "- Lignite NaN \n", "- Hard coal 1012 \n", "- Oil 168 \n", "- Natural gas 4876 \n", "\n", "country \\\n", "type \n", "year 2019.0 \n", "source e-control \n", "source_type Regulatory Authority \n", "weblink https://www.e-control.at/documents/1785851/1811756/BeStGes-2019_KW2EPLTyp.xlsx/e9993ee3-838f-ed2e-2031-7503418fac2d?t=1596094915390 \n", "capacity_definition Gross capacity \n", "technology \n", "Fossil fuels 6041 \n", "- Lignite NaN \n", "- Hard coal 598 \n", "- Oil 164 \n", "- Natural gas 4429 \n", "\n", "country BE \\\n", "type Installed capacity in MW \n", "year 2015.0 \n", "source ELIA \n", "source_type Transmission System Operator \n", "weblink https://www.elia.be/en/grid-data/power-generation/generating-facilities \n", "capacity_definition Unknown \n", "technology \n", "Fossil fuels NaN \n", "- Lignite NaN \n", "- Hard coal NaN \n", "- Oil NaN \n", "- Natural gas NaN \n", "\n", "country \\\n", "type \n", "year 2016.0 \n", "source ELIA \n", "source_type Transmission System Operator \n", "weblink https://www.elia.be/en/grid-data/power-generation/generating-facilities \n", "capacity_definition Unknown \n", "technology \n", "Fossil fuels NaN \n", "- Lignite NaN \n", "- Hard coal NaN \n", "- Oil NaN \n", "- Natural gas NaN \n", "\n", "country \\\n", "type \n", "year 2017.0 \n", "source ELIA \n", "source_type Transmission System Operator \n", "weblink https://www.elia.be/en/grid-data/power-generation/generating-facilities \n", "capacity_definition Unknown \n", "technology \n", "Fossil fuels NaN \n", "- Lignite NaN \n", "- Hard coal NaN \n", "- Oil NaN \n", "- Natural gas NaN \n", "\n", "country \\\n", "type \n", "year 2018.0 \n", "source ELIA \n", "source_type Transmission System Operator \n", "weblink https://www.elia.be/en/grid-data/power-generation/generating-facilities \n", "capacity_definition Unknown \n", "technology \n", "Fossil fuels NaN \n", "- Lignite NaN \n", "- Hard coal NaN \n", "- Oil NaN \n", "- Natural gas NaN \n", "\n", "country ... \\\n", "type ... \n", "year ... \n", "source ... \n", "source_type ... \n", "weblink ... \n", "capacity_definition ... \n", "technology ... \n", "Fossil fuels ... \n", "- Lignite ... \n", "- Hard coal ... \n", "- Oil ... \n", "- Natural gas ... \n", "\n", "country SI \\\n", "type Installed capacity in MW \n", "year 2015.0 \n", "source Agencija za energijo \n", "source_type Ministry \n", "weblink https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2015/f1302ae0-7267-4ae7-b74d-7ce8c4323043 \n", "capacity_definition Net capacity \n", "technology \n", "Fossil fuels 1442.97 \n", "- Lignite NaN \n", "- Hard coal NaN \n", "- Oil NaN \n", "- Natural gas NaN \n", "\n", "country \\\n", "type \n", "year 2016.0 \n", "source Agencija za energijo \n", "source_type Ministry \n", "weblink https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2016/de8cc94f-b3f6-4d32-8e79-ce3b0ec386f5 \n", "capacity_definition Net capacity \n", "technology \n", "Fossil fuels 1450.5 \n", "- Lignite NaN \n", "- Hard coal NaN \n", "- Oil NaN \n", "- Natural gas NaN \n", "\n", "country \\\n", "type \n", "year 2017.0 \n", "source Agencija za energijo \n", "source_type Ministry \n", "weblink https://www.agen-rs.si/documents/54870/68629/a/78f74b68-dbfc-415e-ab88-882652558d94 \n", "capacity_definition Net capacity \n", "technology \n", "Fossil fuels 1223 \n", "- Lignite NaN \n", "- Hard coal NaN \n", "- Oil NaN \n", "- Natural gas NaN \n", "\n", "country SK \\\n", "type Installed capacity in MW \n", "year 2013.0 \n", "source Statistical Office of Slovakia \n", "source_type Statistical Office \n", "weblink Link unavailable \n", "capacity_definition Unknown \n", "technology \n", "Fossil fuels 3348 \n", "- Lignite NaN \n", "- Hard coal NaN \n", "- Oil 140 \n", "- Natural gas 1016 \n", "\n", "country \\\n", "type \n", "year 2014.0 \n", "source Statistical Office of Slovakia \n", "source_type Statistical Office \n", "weblink https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRZJIHHEGAOxQwwYEuvSASM31AE7jWsnb1_Z8aRtGEM7LQcNsP8Hu6t_hQS6R6LJ99WXfFdtmvxRPS-E-TliAR8OdRuAGSMIrtNR5CWWDhTQ3UngeLZP2RSATz0DAttPYysiBGyChArbURQl0ywDL8NjCIjuQZimADE783DhsuHP-Gw6e0vQY3MdgjBM0oxH5GZOT_x7_nziOCpszEdGFpKZS898h6CLt6GHP-f_M2wwUOFE5Z4lkECbbwnE5fLcELrrPwo6-DTo6Z87Njp5h3bzxx_8E-986P-tY7nKf75pU-bqlDOUIYHEstltdw9osSme8wdNNtr2e3EFanms1vmyklewf5a79euvbwCXukGkoVFJLLWsQLO4WWiSyWVBVyujMI3jt7fLqkQLHUtSyJxpK4xXGjUtS-OlBUpdMpaTEuu8_LiX7WER3Vb_wLeL_b0XsZq3YIh1nzoTwjPW4lsC0TfL4iTpmua-Gt4E42SGbYt6zii-VQM7dzBPpiYG0M-CLsP1WV70nQndgrmJQn9TS5TIBi1Uy9jFnkwwuttX8oDSZvOtVsdh8peO8AFdI1EV9eCwrAcw0DFnJsVqMy1M1P3xfK2-Pj0JW5l40-zkyw7d_3cXb-v0_ao5ea0qbR27sX-4cQkVi08_AHhGIRI!/dz/d5/L2dBISEvZ0FBIS9nQSEh/ \n", "capacity_definition Unknown \n", "technology \n", "Fossil fuels 2907 \n", "- Lignite NaN \n", "- Hard coal NaN \n", "- Oil 138 \n", "- Natural gas 1022 \n", "\n", "country \\\n", "type \n", "year 2015.0 \n", "source Statistical Office of Slovakia \n", "source_type Statistical Office \n", "weblink https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsVZIgDhijIHYIQYMiXXpCAwNdcBOQ-3k7Ss7nrQNY2in1YEB9v9gd_WvEEf3iDdiX30RbbVtxKN8XnH9c2j4bDzGFoChTcC_TiahG5sYKKC7k8B2LY8acwA2dzXwLS-JzJAQsAjiMmyFYRjP0xTcVJ2CT7ALQZIARMaZhwvLgj_j0_niLUHXWGLwgyBOUhaSmyU98e_5s5lty7C2nGhpQBYOPfM9gj7eggH-nP_PsGaADMcy9zSGGLp8R8Avl-cE0F__UdDDJ_5A_5yp1svbtJ8__uCfePtD_29t05H-83SLGg6mzEAp4ojnTbtrH9Bqmz2LB6VolN337Ark5bHaiLwqrmD_XLSb11_fiLUotVKjSi4IVijOiMKKfK0wprFCFQwyio_f3uXVGq0wlEKYTFd0zRAKZSJXmF6aikl1BkIzszIvP-5ld1h4v9U_8N1if-9FJOfNH6vYo_aMsNTo8B0BH5plfpL0TfNQDW-CabxQLZO69iS6lQO7tFUWz3UVAJ8FfYYbsjwfOhP6BUsdBd62LlBcNGglW2Zc7MlMRXf7qjigpNl-q-VxGP-lIzxA14hXWT065PUIRlhlhk5VuZmmSuT98Xytvj49cUuaeNu0xUuL7v-7i3d18r5qRl6rStlETuQdbhxC-erTD_ffAqA!/dz/d5/L2dBISEvZ0FBIS9nQSEh/ \n", "capacity_definition Unknown \n", "technology \n", "Fossil fuels 2586 \n", "- Lignite NaN \n", "- Hard coal NaN \n", "- Oil 138 \n", "- Natural gas 959 \n", "\n", "country \\\n", "type \n", "year 2016.0 \n", "source Statistical Office of Slovakia \n", "source_type Statistical Office \n", "weblink https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRYkIXHEGAOxQwwYGnTpAMYNdcBOQ-3m7Ss7nrQ1Y2in1YEB9v9gd_WvkEAPSDTZvvqctdW2yZ7kcyr0TwHz-HismgCMTsC7jSeBExkqEEAfTwLLMV3C5gB87lDwTDcOjQBjMDESMmwGQRDNkwScRJuCh1UH_DgGCNmZhyvLhD_jk_niLUGHLVXwfD-KEx7guyU58e_585llyTBdTmji44VNznyPoI83YYA_5_8zTBnIcCRzTyKIoMt3BOJ6ebYP_fUfBT187A30z57SXt4i_fzxB__EWxf9v7cMW_rP1U3CbJVwhhIkkCiadtc-onSbv2SPStkou2_5DcjLU7XJiqq8gf1L2W5ef32TGWrGVsZaWfFMV4hOqMKpvCsxLwzAJIeCHr-9K6oVSiHT8hUjupIX60Ihqk4VyZfKeo1xSTnOoNAu97I7LKLf6hd8t9jfexHKefPGmuoSa4Z5wjp8RyCGZlmcJH3TPFTDm2AaLTTTII41Ce_lwC4tjUdzXQNQz4I-ww1ZXgydCf2CpY58d1uXKCoblMqWsas9mck93VflAcXN9mstj8PoLx3hArpFosrr0aGoRzBSNc50osnNNDQs74_na_Xl-VmY0sTbpi2_t-jhv7t4V8fvq-b4taqUTWiH7uHOxkSkH34AcI_n7A!!/dz/d5/L2dBISEvZ0FBIS9nQSEh/ \n", "capacity_definition Unknown \n", "technology \n", "Fossil fuels 3164.51 \n", "- Lignite NaN \n", "- Hard coal NaN \n", "- Oil NaN \n", "- Natural gas NaN \n", "\n", "country \\\n", "type \n", "year 2017.0 \n", "source Statistical Office of Slovakia \n", "source_type Statistical Office \n", "weblink https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZRRd5owFMc_yx76iLmEQJJHRASqpQLCal52AOPKLGhXp-u3X7SebpMjbGfLAwe4_x_ce_O_QQI9INHk--pzvqs2Tf6knhfC-hTRgA2Hug1AzREEt-ko8hKuAwH08SRwPNsndArApp4Jge2nMY8MA2wDCRW2oyhKplkGXobHEBi6B2GaAsT0zMOVZcOf8dl09pagR-c6BGGYpBmLjLs5OfHv-bOJ46iwOR-ZWWjMXHLmOwRdvA09_Dn_n2GTggonKvcsgQTafEsgrpfnhtBd_1HQwadBT__csdnJO6SbP_7gn3jnov_3DneV_3zLJtTVCaMoQwKJstltd49osSle8kdNNtr2W3ED6vJUrfOykjewf5G79euvbzDnJTcLoi1zk2gEl1IruGFpRcFXJsF0xTAcv70tqyVa6IwA6HylSWZxjchlqTFpgsYMbEnMTGtF8eVetodFdFv9gm8X-3svYjVvwRDrPnEmBstoi28JRN8si5Oka5r7angTjJMZtjnxnFF8rwZ27mCWTC2sWngWdBmuz_Ki70zoFswtFPqbWqJENmihWkav9mSi9nRfyQNKm83XWh2HyV86wgd0i0RV1INDWQ9goGNGLYLVZnJsqPvj-Vp9eX4WtjLxptnJ7zv08N9dvK3T91Uz47WqtHXsxv7hzjWIWHz4AZRUN9Q!/dz/d5/L2dBISEvZ0FBIS9nQSEh/ \n", "capacity_definition Unknown \n", "technology \n", "Fossil fuels 3266.97 \n", "- Lignite NaN \n", "- Hard coal NaN \n", "- Oil NaN \n", "- Natural gas NaN \n", "\n", "country \\\n", "type \n", "year 2018.0 \n", "source Statistical Office of Slovakia \n", "source_type Statistical Office \n", "weblink https://slovak.statistics.sk/PortalTraffic/fileServlet?Dokument=da19bcb8-8c9c-4757-aba5-2d0720266efa \n", "capacity_definition Unknown \n", "technology \n", "Fossil fuels 2581.47 \n", "- Lignite NaN \n", "- Hard coal NaN \n", "- Oil NaN \n", "- Natural gas NaN \n", "\n", "country \n", "type \n", "year NaN \n", "source \n", "source_type NaN \n", "weblink \n", "capacity_definition NaN \n", "technology \n", "Fossil fuels NaN \n", "- Lignite NaN \n", "- Hard coal NaN \n", "- Oil NaN \n", "- Natural gas NaN \n", "\n", "[5 rows x 157 columns]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Delete definition of energy levels from raw data\n", "data_raw.drop(data_raw.columns[[0, 1, 2, 3, 4, 5]], axis=1, inplace=True)\n", "\n", "level_names = ['country', 'type', 'year', 'source',\n", " 'source_type', 'weblink', 'capacity_definition']\n", "# Set multiindex column names\n", "data_raw.columns = pd.MultiIndex.from_arrays(data_raw[:7].values,\n", " names=level_names)\n", "\n", "# Remove 3 rows which are already used as column names\n", "data_raw = data_raw[pd.notnull(data_raw.index)]\n", "\n", "# Extract the ordering of technologies\n", "technology_order = data_raw.index.str.replace('- ', '').values.tolist()\n", "\n", "data_raw.head()" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "MultiIndex([], names=['country', 'type', 'year', 'source', 'source_type', 'weblink', 'capacity_definition'])" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_raw.columns[data_raw.columns.duplicated()]" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countryATBE...SISK
typeInstalled capacity in MWInstalled capacity in MW...Installed capacity in MWInstalled capacity in MW
year2014.02015.02016.02017.02018.02019.02015.02016.02017.02018.0...2015.02016.02017.02013.02014.02015.02016.02017.02018.0NaN
sourcee-controle-controle-controle-controle-controle-controlELIAELIAELIAELIA...Agencija za energijoAgencija za energijoAgencija za energijoStatistical Office of SlovakiaStatistical Office of SlovakiaStatistical Office of SlovakiaStatistical Office of SlovakiaStatistical Office of SlovakiaStatistical Office of Slovakia
source_typeRegulatory AuthorityRegulatory AuthorityRegulatory AuthorityRegulatory AuthorityRegulatory AuthorityRegulatory AuthorityTransmission System OperatorTransmission System OperatorTransmission System OperatorTransmission System Operator...MinistryMinistryMinistryStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeNaN
weblinkhttps://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538https://www.e-control.at/documents/1785851/1811756/BeStGes-2019_KW2EPLTyp.xlsx/e9993ee3-838f-ed2e-2031-7503418fac2d?t=1596094915390https://www.elia.be/en/grid-data/power-generation/generating-facilitieshttps://www.elia.be/en/grid-data/power-generation/generating-facilitieshttps://www.elia.be/en/grid-data/power-generation/generating-facilitieshttps://www.elia.be/en/grid-data/power-generation/generating-facilities...https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2015/f1302ae0-7267-4ae7-b74d-7ce8c4323043https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2016/de8cc94f-b3f6-4d32-8e79-ce3b0ec386f5https://www.agen-rs.si/documents/54870/68629/a/78f74b68-dbfc-415e-ab88-882652558d94Link unavailablehttps://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRZJIHHEGAOxQwwYEuvSASM31AE7jWsnb1_Z8aRtGEM7LQcNsP8Hu6t_hQS6R6LJ99WXfFdtmvxRPS-E-TliAR8OdRuAGSMIrtNR5CWWDhTQ3UngeLZP2RSATz0DAttPYysiBGyChArbURQl0ywDL8NjCIjuQZimADE783DhsuHP-Gw6e0vQY3MdgjBM0oxH5GZOT_x7_nziOCpszEdGFpKZS898h6CLt6GHP-f_M2wwUOFE5Z4lkECbbwnE5fLcELrrPwo6-DTo6Z87Njp5h3bzxx_8E-986P-tY7nKf75pU-bqlDOUIYHEstltdw9osSme8wdNNtr2e3EFanms1vmyklewf5a79euvbwCXukGkoVFJLLWsQLO4WWiSyWVBVyujMI3jt7fLqkQLHUtSyJxpK4xXGjUtS-OlBUpdMpaTEuu8_LiX7WER3Vb_wLeL_b0XsZq3YIh1nzoTwjPW4lsC0TfL4iTpmua-Gt4E42SGbYt6zii-VQM7dzBPpiYG0M-CLsP1WV70nQndgrmJQn9TS5TIBi1Uy9jFnkwwuttX8oDSZvOtVsdh8peO8AFdI1EV9eCwrAcw0DFnJsVqMy1M1P3xfK2-Pj0JW5l40-zkyw7d_3cXb-v0_ao5ea0qbR27sX-4cQkVi08_AHhGIRI!/dz/d5/L2dBISEvZ0FBIS9nQSEh/https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsVZIgDhijIHYIQYMiXXpCAwNdcBOQ-3k7Ss7nrQNY2in1YEB9v9gd_WvEEf3iDdiX30RbbVtxKN8XnH9c2j4bDzGFoChTcC_TiahG5sYKKC7k8B2LY8acwA2dzXwLS-JzJAQsAjiMmyFYRjP0xTcVJ2CT7ALQZIARMaZhwvLgj_j0_niLUHXWGLwgyBOUhaSmyU98e_5s5lty7C2nGhpQBYOPfM9gj7eggH-nP_PsGaADMcy9zSGGLp8R8Avl-cE0F__UdDDJ_5A_5yp1svbtJ8__uCfePtD_29t05H-83SLGg6mzEAp4ojnTbtrH9Bqmz2LB6VolN337Ark5bHaiLwqrmD_XLSb11_fiLUotVKjSi4IVijOiMKKfK0wprFCFQwyio_f3uXVGq0wlEKYTFd0zRAKZSJXmF6aikl1BkIzszIvP-5ld1h4v9U_8N1if-9FJOfNH6vYo_aMsNTo8B0BH5plfpL0TfNQDW-CabxQLZO69iS6lQO7tFUWz3UVAJ8FfYYbsjwfOhP6BUsdBd62LlBcNGglW2Zc7MlMRXf7qjigpNl-q-VxGP-lIzxA14hXWT065PUIRlhlhk5VuZmmSuT98Xytvj49cUuaeNu0xUuL7v-7i3d18r5qRl6rStlETuQdbhxC-erTD_ffAqA!/dz/d5/L2dBISEvZ0FBIS9nQSEh/https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRYkIXHEGAOxQwwYGnTpAMYNdcBOQ-3m7Ss7nrQ1Y2in1YEB9v9gd_WvkEAPSDTZvvqctdW2yZ7kcyr0TwHz-HismgCMTsC7jSeBExkqEEAfTwLLMV3C5gB87lDwTDcOjQBjMDESMmwGQRDNkwScRJuCh1UH_DgGCNmZhyvLhD_jk_niLUGHLVXwfD-KEx7guyU58e_585llyTBdTmji44VNznyPoI83YYA_5_8zTBnIcCRzTyKIoMt3BOJ6ebYP_fUfBT187A30z57SXt4i_fzxB__EWxf9v7cMW_rP1U3CbJVwhhIkkCiadtc-onSbv2SPStkou2_5DcjLU7XJiqq8gf1L2W5ef32TGWrGVsZaWfFMV4hOqMKpvCsxLwzAJIeCHr-9K6oVSiHT8hUjupIX60Ihqk4VyZfKeo1xSTnOoNAu97I7LKLf6hd8t9jfexHKefPGmuoSa4Z5wjp8RyCGZlmcJH3TPFTDm2AaLTTTII41Ce_lwC4tjUdzXQNQz4I-ww1ZXgydCf2CpY58d1uXKCoblMqWsas9mck93VflAcXN9mstj8PoLx3hArpFosrr0aGoRzBSNc50osnNNDQs74_na_Xl-VmY0sTbpi2_t-jhv7t4V8fvq-b4taqUTWiH7uHOxkSkH34AcI_n7A!!/dz/d5/L2dBISEvZ0FBIS9nQSEh/https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZRRd5owFMc_yx76iLmEQJJHRASqpQLCal52AOPKLGhXp-u3X7SebpMjbGfLAwe4_x_ce_O_QQI9INHk--pzvqs2Tf6knhfC-hTRgA2Hug1AzREEt-ko8hKuAwH08SRwPNsndArApp4Jge2nMY8MA2wDCRW2oyhKplkGXobHEBi6B2GaAsT0zMOVZcOf8dl09pagR-c6BGGYpBmLjLs5OfHv-bOJ46iwOR-ZWWjMXHLmOwRdvA09_Dn_n2GTggonKvcsgQTafEsgrpfnhtBd_1HQwadBT__csdnJO6SbP_7gn3jnov_3DneV_3zLJtTVCaMoQwKJstltd49osSle8kdNNtr2W3ED6vJUrfOykjewf5G79euvbzDnJTcLoi1zk2gEl1IruGFpRcFXJsF0xTAcv70tqyVa6IwA6HylSWZxjchlqTFpgsYMbEnMTGtF8eVetodFdFv9gm8X-3svYjVvwRDrPnEmBstoi28JRN8si5Oka5r7angTjJMZtjnxnFF8rwZ27mCWTC2sWngWdBmuz_Ki70zoFswtFPqbWqJENmihWkav9mSi9nRfyQNKm83XWh2HyV86wgd0i0RV1INDWQ9goGNGLYLVZnJsqPvj-Vp9eX4WtjLxptnJ7zv08N9dvK3T91Uz47WqtHXsxv7hzjWIWHz4AZRUN9Q!/dz/d5/L2dBISEvZ0FBIS9nQSEh/https://slovak.statistics.sk/PortalTraffic/fileServlet?Dokument=da19bcb8-8c9c-4757-aba5-2d0720266efa
capacity_definitionGross capacityGross capacityGross capacityGross capacityGross capacityGross capacityUnknownUnknownUnknownUnknown...Net capacityNet capacityNet capacityUnknownUnknownUnknownUnknownUnknownUnknownNaN
technology
\n", "

0 rows × 157 columns

\n", "
" ], "text/plain": [ "Empty DataFrame\n", "Columns: [(AT, Installed capacity in MW, 2014.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538, Gross capacity), (AT, Installed capacity in MW, 2015.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538, Gross capacity), (AT, Installed capacity in MW, 2016.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538, Gross capacity), (AT, Installed capacity in MW, 2017.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538, Gross capacity), (AT, Installed capacity in MW, 2018.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538, Gross capacity), (AT, Installed capacity in MW, 2019.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-2019_KW2EPLTyp.xlsx/e9993ee3-838f-ed2e-2031-7503418fac2d?t=1596094915390, Gross capacity), (BE, Installed capacity in MW, 2015.0, ELIA, Transmission System Operator, https://www.elia.be/en/grid-data/power-generation/generating-facilities, Unknown), (BE, Installed capacity in MW, 2016.0, ELIA, Transmission System Operator, https://www.elia.be/en/grid-data/power-generation/generating-facilities, Unknown), (BE, Installed capacity in MW, 2017.0, ELIA, Transmission System Operator, https://www.elia.be/en/grid-data/power-generation/generating-facilities, Unknown), (BE, Installed capacity in MW, 2018.0, ELIA, Transmission System Operator, https://www.elia.be/en/grid-data/power-generation/generating-facilities, Unknown), (BE, Installed capacity in MW, 2019.0, ELIA, Transmission System Operator, https://www.elia.be/en/grid-data/power-generation/generating-facilities, Unknown), (BG, Installed capacity in MW, 2012.0, TSO Bulgaria, Transmission System Operator, http://www.eso.bg/fileObj.php?oid=99, Unknown), (BG, Installed capacity in MW, 2019.0, TSO Bulgaria, Transmission System Operator, http://www.eso.bg/?did=380, Unknown), (CH, Installed capacity in MW, 2012.0, UN Statistical Office, Statistical Office, http://data.un.org/Data.aspx?d=EDATA&f=cmID%3AEC, Net capacity), (CH, Installed capacity in MW, 2012.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvNzA3Ni5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2014.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvNzkyNS5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2015.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvODM0My5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2016.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvODY5My5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2017.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvOTM5Ni5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2018.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvOTc0OC5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2019.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik.exturl.html/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvMTAxMTI=.html, Unknown), (CZ, Installed capacity in MW, 2013.0, ERU, Regulatory Authority, https://www.eru.cz/documents/10540/462820/Annual_report_electricity_2013.pdf/34a35d27-9c58-4c79-99d1-f0fbc5eac06a, Unknown), (CZ, Installed capacity in MW, 2014.0, ERU, Regulatory Authority, https://www.eru.cz/documents/10540/462820/Annual_report_electricity_2014.pdf/f23d80b5-668a-42c0-9d04-1a19556c9c58, Unknown), (CZ, Installed capacity in MW, 2015.0, ERU, Regulatory Authority, https://www.eru.cz/documents/10540/462820/Annual_report_electricity_2015.pdf/2548f33b-7964-4537-8b97-3943c7d83854, Unknown), (CZ, Installed capacity in MW, 2016.0, ERU, Regulatory Authority, https://www.eru.cz/documents/10540/462820/Annual_report_electricity_2016.pdf/bfbc88b7-a814-498f-99bf-98b50148c805, Unknown), (CZ, Installed capacity in MW, 2017.0, ERU, Regulatory Authority, https://www.eru.cz/documents/10540/4580209/Yearly_report_electricity_2017.pdf/a4042a98-8750-4524-bb3c-f172ecba2f3b, Unknown), (CZ, Installed capacity in MW, 2018.0, ERU, Regulatory Authority, http://www.eru.cz/documents/10540/4580207/Yearly_report_electricity_2018.pdf/f25a55d8-6730-4521-8e40-96d8e5f00c70, Unknown), (DE, Installed capacity in MW, 2013.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2014.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2015.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2016.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2017.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2018.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2019.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DK, Installed capacity in MW, 2013.0, DEA, Ministry, Link unavailable, Unknown), (DK, Installed capacity in MW, 2014.0, DEA, Ministry, https://ens.dk/sites/ens.dk/files/Statistik/energystatistics2014.pdf, Unknown), (DK, Installed capacity in MW, 2015.0, DEA, Ministry, https://ens.dk/sites/ens.dk/files/Statistik/energy_statistics_2015.pdf, Unknown), (DK, Installed capacity in MW, 2016.0, DEA, Ministry, https://ens.dk/sites/ens.dk/files/Statistik/energy_statistics_2016.pdf, Unknown), (DK, Installed capacity in MW, 2017.0, DEA, Ministry, https://ens.dk/sites/ens.dk/files/Statistik/energystatistics2017.pdf, Unknown), (DK, Installed capacity in MW, 2018.0, DEA, Ministry, https://ens.dk/sites/ens.dk/files/Statistik/energy_statistics_2018.pdf, Unknown), (EE, Installed capacity in MW, 2013.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2014.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2015.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2016.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2017.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2018.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2019.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (ES, Installed capacity in MW, 2013.0, REE, Transmission System Operator, Link unavailable, Unknown), (ES, Installed capacity in MW, 2014.0, REE, Transmission System Operator, nan, Unknown), (ES, Installed capacity in MW, 2015.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/septiembre_2019/6_Potencia_instalada_09_2019.xlsm, Unknown), (ES, Installed capacity in MW, 2016.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/septiembre_2019/6_Potencia_instalada_09_2019.xlsm, Unknown), (ES, Installed capacity in MW, 2017.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/septiembre_2019/6_Potencia_instalada_09_2019.xlsm, Unknown), (ES, Installed capacity in MW, 2018.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/septiembre_2019/6_Potencia_instalada_09_2019.xlsm, Unknown), (ES, Installed capacity in MW, 2019.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/septiembre_2019/6_Potencia_instalada_09_2019.xlsm, Unknown), (ES, Installed capacity in MW, 2020.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/agosto_2020/6_Potencia_instalada_08_2020.xlsm, Unknown), (FR, Installed capacity in MW, 2014.0, RTE, Transmission System Operator, https://www.rte-france.com/sites/default/files/bilan_electrique_2014_en.pdf, Unknown), (FR, Installed capacity in MW, 2015.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FR, Installed capacity in MW, 2016.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FR, Installed capacity in MW, 2017.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FR, Installed capacity in MW, 2018.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FR, Installed capacity in MW, 2019.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FR, Installed capacity in MW, 2020.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FI, Installed capacity in MW, 2012.0, Statistics Finland, Statistical Office, http://pxnet2.stat.fi/PXWeb/pxweb/en/StatFin_Passiivi/StatFin_Passiivi__ene__ehk/statfinpas_ehk_pxt_904_201500_en.px/, Unknown), (FI, Installed capacity in MW, 2013.0, Statistics Finland, Statistical Office, http://pxnet2.stat.fi/PXWeb/pxweb/en/StatFin_Passiivi/StatFin_Passiivi__ene__ehk/statfinpas_ehk_pxt_904_201500_en.px/, Unknown), (FI, Installed capacity in MW, 2014.0, Statistics Finland, Statistical Office, http://pxnet2.stat.fi/PXWeb/pxweb/en/StatFin_Passiivi/StatFin_Passiivi__ene__ehk/statfinpas_ehk_pxt_904_201500_en.px/, Unknown), (FI, Installed capacity in MW, 2015.0, Statistics Finland, Statistical Office, http://pxnet2.stat.fi/PXWeb/pxweb/en/StatFin_Passiivi/StatFin_Passiivi__ene__ehk/statfinpas_ehk_pxt_904_201500_en.px/, Unknown), (GB, Installed capacity in MW, 2015.0, Department of Energy & Climate Change, Ministry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/826556/DUKES_5.7.xls, Unknown), (GB, Installed capacity in MW, 2016.0, Department of Energy & Climate Change, Ministry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/826556/DUKES_5.7.xls, Unknown), (GB, Installed capacity in MW, 2017.0, Department of Energy & Climate Change, Ministry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/826556/DUKES_5.7.xls, Unknown), (GB, Installed capacity in MW, 2018.0, Department of Energy & Climate Change, Ministry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/826556/DUKES_5.7.xls, Unknown), (GB, Installed capacity in MW, 2019.0, Department of Energy & Climate Change, Ministry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/904812/DUKES_5.7.xls, Unknown), (GR, Installed capacity in MW, 2012.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2015, Unknown), (GR, Installed capacity in MW, 2013.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2016, Unknown), (GR, Installed capacity in MW, 2014.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2017, Unknown), (GR, Installed capacity in MW, 2015.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2018, Unknown), (GR, Installed capacity in MW, 2016.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2018, Unknown), (GR, Installed capacity in MW, 2017.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2018, Unknown), (GR, Installed capacity in MW, 2018.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2019, Unknown), (HR, Installed capacity in MW, 2013.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/qyxRItZNXQL84ygVm4NbnE/publications/Temeljni_podaci_2013.pdf, Unknown), (HR, Installed capacity in MW, 2014.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/vHpFOrmCvaEk2wfzW8lBZ6/publications/Temeljni_podaci_za_2014.pdf, Unknown), (HR, Installed capacity in MW, 2015.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/PsIM6mazLoTDVjKzm4NbnE/publications/Temeljni_podaci_2015.pdf, Unknown), (HR, Installed capacity in MW, 2016.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/wEfz2PYHVQLQIygVm4NbnE/publications/Temeljni_podaci_2016.pdf, Unknown), (HR, Installed capacity in MW, 2017.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/KzqT4wOZhIFGvbKzm4t0IF/annual-report-hops/HOPS_GI_2017_-__za_web.pdf, Unknown), (HR, Installed capacity in MW, 2018.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/x2VUqipcFKEnEgg50EV391/annual-report-hops/GI%202018%20unutarnje%20-%209-7-19%20web.pdf, Unknown), (HU, Installed capacity in MW, 2013.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/160362/Planned+Installed+generation+capacity+for+2010_2011_2012_2013+++4.3.1.pdf/84e86e8b-3b7c-4377-8020-c4beec92c914, Unknown), (HU, Installed capacity in MW, 2014.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/188569160/BT_terv_2014/9946a7a2-38ec-4794-9d7f-96a7a927d1b9, Unknown), (HU, Installed capacity in MW, 2015.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/188569160/BT_terv_2014/9946a7a2-38ec-4794-9d7f-96a7a927d1b9, Unknown), (HU, Installed capacity in MW, 2016.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/188569160/BT_terv_2014/9946a7a2-38ec-4794-9d7f-96a7a927d1b9, Unknown), (HU, Installed capacity in MW, 2017.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/188569160/BT_terv_2014/9946a7a2-38ec-4794-9d7f-96a7a927d1b9, Unknown), (HU, Installed capacity in MW, 2018.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/230112823/VER+BT_20181231.pdf/20c127e3-bb04-8b84-925f-2548761a41d4, Unknown), (IE, Installed capacity in MW, 2014.0, Eirgrid, Transmission System Operator, http://www.eirgridgroup.com/site-files/library/EirGrid/Eirgrid_Generation_Capacity_Statement_2015.-2024.pdf, Unknown), (IE, Installed capacity in MW, 2015.0, Eirgrid, Transmission System Operator, http://www.eirgridgroup.com/site-files/library/EirGrid/Eirgrid_Generation_Capacity_Statement_2015.-2024.pdf, Unknown), (IE, Installed capacity in MW, 2019.0, Eirgrid, Transmission System Operator, http://www.eirgridgroup.com/site-files/library/EirGrid/EirGrid-Group-All-Island-Generation-Capacity-Statement-2019-2028.pdf, Unknown), (IE, Installed capacity in MW, 2020.0, Eirgrid, Transmission System Operator, http://www.eirgridgroup.com/site-files/library/EirGrid/All-Island-Generation-Capacity-Statement-2020-2029.pdf, Unknown), (IT, Installed capacity in MW, 2014.0, Terna, Transmission System Operator, https://download.terna.it/terna/0000/0775/97.PDF, Gross capacity), (IT, Installed capacity in MW, 2015.0, Terna, Transmission System Operator, https://download.terna.it/terna/0000/0995/02.PDF, Gross capacity), (IT, Installed capacity in MW, 2016.0, Terna, Transmission System Operator, https://download.terna.it/terna/0000/1112/78.PDF, Gross capacity), (IT, Installed capacity in MW, 2017.0, Terna, Transmission System Operator, https://download.terna.it/terna/2017_Eng_3-POWER_PLANTS_8d73131c70720ed.pdf, Gross capacity), (IT, Installed capacity in MW, 2018.0, Terna, Transmission System Operator, https://www.terna.it/en/electric-system/transparency-report/installed-capacity, Gross capacity), (IT, Installed capacity in MW, 2019.0, Terna, Transmission System Operator, https://www.terna.it/en/electric-system/transparency-report/installed-capacity, Gross capacity), ...]\n", "Index: []\n", "\n", "[0 rows x 157 columns]" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_raw[data_raw.index.duplicated()]" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
technologycountrytypeyearsourcesource_typeweblinkcapacity_definitioncapacitycomment
0Fossil fuelsATInstalled capacity in MW2014e-controlRegulatory Authorityhttps://www.e-control.at/documents/1785851/181...Gross capacity7243.634NaN
1Fossil fuelsATInstalled capacity in MW2015e-controlRegulatory Authorityhttps://www.e-control.at/documents/1785851/181...Gross capacity7059.065NaN
2Fossil fuelsATInstalled capacity in MW2016e-controlRegulatory Authorityhttps://www.e-control.at/documents/1785851/181...Gross capacity7323.000NaN
3Fossil fuelsATInstalled capacity in MW2017e-controlRegulatory Authorityhttps://www.e-control.at/documents/1785851/181...Gross capacity6469.000NaN
4Fossil fuelsATInstalled capacity in MW2018e-controlRegulatory Authorityhttps://www.e-control.at/documents/1785851/181...Gross capacity6492.000NaN
\n", "
" ], "text/plain": [ " technology country type year source \\\n", "0 Fossil fuels AT Installed capacity in MW 2014 e-control \n", "1 Fossil fuels AT Installed capacity in MW 2015 e-control \n", "2 Fossil fuels AT Installed capacity in MW 2016 e-control \n", "3 Fossil fuels AT Installed capacity in MW 2017 e-control \n", "4 Fossil fuels AT Installed capacity in MW 2018 e-control \n", "\n", " source_type weblink \\\n", "0 Regulatory Authority https://www.e-control.at/documents/1785851/181... \n", "1 Regulatory Authority https://www.e-control.at/documents/1785851/181... \n", "2 Regulatory Authority https://www.e-control.at/documents/1785851/181... \n", "3 Regulatory Authority https://www.e-control.at/documents/1785851/181... \n", "4 Regulatory Authority https://www.e-control.at/documents/1785851/181... \n", "\n", " capacity_definition capacity comment \n", "0 Gross capacity 7243.634 NaN \n", "1 Gross capacity 7059.065 NaN \n", "2 Gross capacity 7323.000 NaN \n", "3 Gross capacity 6469.000 NaN \n", "4 Gross capacity 6492.000 NaN " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Reshape dataframe to list\n", "data_opsd = pd.DataFrame(data_raw.stack(level=level_names))\n", "\n", "# Reset index for dataframe\n", "data_opsd.reset_index(inplace=True)\n", "data_opsd['technology'] = data_opsd['technology'].str.replace('- ', '')\n", "\n", "data_opsd.rename(columns={0: 'capacity'}, inplace=True)\n", "data_opsd['capacity'] = pd.to_numeric(data_opsd['capacity'], errors='coerce')\n", "\n", "# For some source, permission to publish data\n", "banlist = ['ELIA', 'BMWi', 'Mavir']\n", "davail = 'data available, but cannot be provided'\n", "data_opsd.loc[data_opsd['source'].isin(banlist), 'comment'] = davail\n", "\n", "data_opsd.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.2 EUROSTAT data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "EUROSTAT publishes annual structural data on national electricity generation capacities for European countries. The dataset is available in the EUROSTAT database within the category 'Environment and Energy' ([nrg_113a](http://ec.europa.eu/eurostat/product?code=nrg_113a&mode=view))." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
unitproductindic_nrggeo\\time201620152014201320122011...1999199819971996199519941993199219911990
0MW900712_1176011AL98989898970...0000000000
1MW900712_1176011AT556159646136632663476339...5089491751995031502850795061513347504743
2MW900712_1176011BA206517651765000...0000000000
3MW900712_1176011BE718271997202755981829128...7746765069457126721172986433639464456490
4MW900712_1176011BG363538544410463648494495...5172497700000000
\n", "

5 rows × 31 columns

\n", "
" ], "text/plain": [ " unit product indic_nrg geo\\time 2016 2015 2014 2013 2012 \\\n", "0 MW 9007 12_1176011 AL 98 98 98 98 97 \n", "1 MW 9007 12_1176011 AT 5561 5964 6136 6326 6347 \n", "2 MW 9007 12_1176011 BA 2065 1765 1765 0 0 \n", "3 MW 9007 12_1176011 BE 7182 7199 7202 7559 8182 \n", "4 MW 9007 12_1176011 BG 3635 3854 4410 4636 4849 \n", "\n", " 2011 ... 1999 1998 1997 1996 1995 1994 1993 1992 1991 \\\n", "0 0 ... 0 0 0 0 0 0 0 0 0 \n", "1 6339 ... 5089 4917 5199 5031 5028 5079 5061 5133 4750 \n", "2 0 ... 0 0 0 0 0 0 0 0 0 \n", "3 9128 ... 7746 7650 6945 7126 7211 7298 6433 6394 6445 \n", "4 4495 ... 5172 4977 0 0 0 0 0 0 0 \n", "\n", " 1990 \n", "0 0 \n", "1 4743 \n", "2 0 \n", "3 6490 \n", "4 0 \n", "\n", "[5 rows x 31 columns]" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "url_eurostat = 'Link unavailable'\n", "\n", "filepath_eurostat = os.path.join('input', 'Eurostat', 'Eurostat.tsv.gz')\n", "\n", "data_eurostat = pd.read_csv(filepath_eurostat,\n", " compression='gzip',\n", " sep='\\t|,',\n", " engine='python'\n", " )\n", "data_eurostat.head()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
unitproductindic_nrggeo\\timeyearvalue
0MW900712_1176011AL201698
1MW900712_1176011AT20165561
2MW900712_1176011BA20162065
3MW900712_1176011BE20167182
4MW900712_1176011BG20163635
\n", "
" ], "text/plain": [ " unit product indic_nrg geo\\time year value\n", "0 MW 9007 12_1176011 AL 2016 98 \n", "1 MW 9007 12_1176011 AT 2016 5561 \n", "2 MW 9007 12_1176011 BA 2016 2065 \n", "3 MW 9007 12_1176011 BE 2016 7182 \n", "4 MW 9007 12_1176011 BG 2016 3635 " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "id_vars = ['unit', 'product','indic_nrg', 'geo\\\\time']\n", "data_eurostat = pd.melt(data_eurostat, id_vars=id_vars,\n", " var_name='year', value_name='value')\n", "\n", "data_eurostat.head()" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "data_definition = pd.read_csv(os.path.join('input', 'definition_EUROSTAT_indic.txt'),\n", " header=None,\n", " names=['indic', 'description',\n", " 'energy source'],\n", " sep='\\t')\n", "\n", "data_eurostat = data_eurostat.merge(data_definition,\n", " how='left',\n", " left_on='indic_nrg',\n", " right_on='indic')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The classification of generation capacities in the EUROSTAT dataset is specified in [Regulation (EC) No 1099/2008](http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32008R1099&from=EN) (Annex B, 3.3). The available EUROSTAT dataset [nrg_113a](http://ec.europa.eu/eurostat/product?code=nrg_113a&mode=view) covers the following indicators:\n", "\n", "|indic_nrg | Description | Technology in OPSD |\n", "|---|---|---|\n", "|**12_1176011**|\t**Electrical capacity, main activity producers - Combustible Fuels**|\tFossil fuels & bioenergy|\n", "|**12_1176012**|\t**Electrical capacity, autoproducers - Combustible Fuels**|\tFossil fuels & bioenergy|\n", "|*12_1176061*|\t*Electrical capacity, main activity producers - Mixed plants*|\t|\n", "|*12_1176101*|\t*Electrical capacity, main activity producers - Other Sources*|\t|\n", "|*12_1176102*|\t*Electrical capacity, autoproducers - Other Sources*|\t|\n", "|*12_1176111*|\t*Electrical capacity, main activity producers - Steam*||\n", "|*12_1176112*|\t*Electrical capacity, autoproducers - Steam*||\n", "|*12_1176121*|\t*Electrical capacity, main activity producers - Gas Turbine*||\n", "|*12_1176122*|\t*Electrical capacity, autoproducers - Gas Turbine*||\n", "|*12_1176131*|\t*Electrical capacity, main activity producers - Combined Cycle*||\n", "|*12_1176132*|\t*Electrical capacity, autoproducers - Combined Cycle*||\n", "|*12_1176141*|\t*Electrical capacity, main activity producers - Internal Combustion*||\n", "|*12_1176142*|\t*Electrical capacity, autoproducers - Internal Combustion*||\n", "|*12_1176401*|\t*Electrical capacity, main activity producers - Other Type of Generation*|\t|\n", "|*12_1176402*|\t*Electrical capacity, autoproducers - Other Type of Generation*|\t|\n", "|12_1176253|\tNet maximum capacity - Municipal Wastes|\tNon-renewable waste|\n", "|12_1176263|\tNet maximum capacity - Wood/Wood Wastes/Other Solid Wastes|\tOther bioenergy and renewable waste|\n", "|12_1176273|\tNet maximum capacity - Biogases|\tBiomass and biogas|\n", "|12_1176283|\tNet maximum capacity - Industrial Wastes (non-renewable)|\tNon-renewable waste|\n", "|12_1176343|\tNet maximum capacity - Liquid Biofuels|\tBiomass and biogas|\n", "|**12_1176031**|\t**Electrical capacity, main activity producers - Nuclear**|\tNuclear|\n", "|**12_1176032**|\t**Electrical capacity, autoproducers - Nuclear**|\tNuclear|\n", "|**12_1176051**|\t**Electrical capacity, main activity producers - Hydro**|\tHydro|\n", "|**12_1176052**|\t**Electrical capacity, autoproducers - Hydro**|\tHydro|\n", "|12_1176071|\tNet electrical capacity, main activity producers - Pure Pumped Hydro|\tPumped storage|\n", "|12_1176072|\tNet electrical capacity, autoproducers - Pure Pumped Hydro|\tPumped storage|\n", "|*12_117615*|\t*Net maximum capacity - Hydro <1 MW*|\t|\n", "|*12_117616*|\t*Net maximum capacity - Hydro >= 1 MW and <= 10 MW*|\t|\n", "|*12_117617*|\t*Net maximum capacity - Hydro 10 MW and over*|\t|\n", "|**12_1176301**|\t**Electrical capacity, main activity producers - Tide, wave and ocean**|\tMarine|\n", "|**12_1176302**|\t**Electrical capacity, autoproducers - Tide, wave and ocean**|\tMarine|\n", "|*12_1176303*|\t*Net maximum capacity - Tide, Wave, Ocean*||\n", "|**12_1176081**|\t**Electrical capacity, main activity producers - Geothermal**|\tGeothermal|\n", "|**12_1176082**|\t**Electrical capacity, autoproducers - Geothermal**|\tGeothermal|\n", "|*12_1176083*|\t*Net maximum capacity - Geothermal*|\t|\n", "|**12_1176091**|\t**Electrical capacity, main activity producers - Wind**|\tWind|\n", "|**12_1176092**|\t**Electrical capacity, autoproducers - Wind**|\tWind|\n", "|**12_1176233**|\t**Net maximum capacity - Solar Photovoltaic**|\tPhotovoltaics|\n", "|**12_1176243**|\t**Net maximum capacity - Solar Thermal Electric**|\tConcentrated solar power|\n", "\n", "**Bold** rows indicate top level classes within the EUROSTAT classification, whereas normal and *italic* rows cover different kinds of subclassifications. Especially within the top level 'Combustible fuels' different kinds of subcategorizations based on fuel or technology are available. Simarily, 'Hydro' is differentiated by type (e.g. pumped-hydro storage) or capacity classes. *Italic* rows are not further considered within the OPSD dataset due to the mismatch with existing technology classes." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
unitproductindic_nrggeo\\timeyearvalueindicdescriptionenergy source
0MW900712_1176011AL201698.012_1176011Electrical capacity, main activity producers -...Fossil fuels
1MW900712_1176011AT20165561.012_1176011Electrical capacity, main activity producers -...Fossil fuels
2MW900712_1176011BA20162065.012_1176011Electrical capacity, main activity producers -...Fossil fuels
3MW900712_1176011BE20167182.012_1176011Electrical capacity, main activity producers -...Fossil fuels
4MW900712_1176011BG20163635.012_1176011Electrical capacity, main activity producers -...Fossil fuels
\n", "
" ], "text/plain": [ " unit product indic_nrg geo\\time year value indic \\\n", "0 MW 9007 12_1176011 AL 2016 98.0 12_1176011 \n", "1 MW 9007 12_1176011 AT 2016 5561.0 12_1176011 \n", "2 MW 9007 12_1176011 BA 2016 2065.0 12_1176011 \n", "3 MW 9007 12_1176011 BE 2016 7182.0 12_1176011 \n", "4 MW 9007 12_1176011 BG 2016 3635.0 12_1176011 \n", "\n", " description energy source \n", "0 Electrical capacity, main activity producers -... Fossil fuels \n", "1 Electrical capacity, main activity producers -... Fossil fuels \n", "2 Electrical capacity, main activity producers -... Fossil fuels \n", "3 Electrical capacity, main activity producers -... Fossil fuels \n", "4 Electrical capacity, main activity producers -... Fossil fuels " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_eurostat = data_eurostat[data_eurostat['energy source'].isnull() == False]\n", "\n", "values_as_string = data_eurostat['value'].astype(str)\n", "string_values = values_as_string.str.split(' ', 1).str[0]\n", "string_values.replace(':', np.nan, inplace=True)\n", "subset_nan = string_values.isnull()\n", "\n", "data_eurostat['value'] = string_values\n", "data_eurostat['year'] = data_eurostat['year'].astype(int)\n", "data_eurostat['value'] = data_eurostat['value'].astype(float)\n", "\n", "data_eurostat.head()" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "data_eurostat = data_eurostat.drop(['unit', 'product', 'indic_nrg',\n", " 'indic', 'description'], axis=1)\n", "\n", "data_eurostat = data_eurostat.rename(columns={'geo\\\\time': 'country',\n", " 'energy source': 'technology',\n", " 'value': 'capacity'})\n" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
technologyyearcountrycapacity
0Biomass and biogas1990AL0.0
1Biomass and biogas1990AT0.0
2Biomass and biogas1990BA0.0
3Biomass and biogas1990BE1.0
4Biomass and biogas1990BG0.0
\n", "
" ], "text/plain": [ " technology year country capacity\n", "0 Biomass and biogas 1990 AL 0.0\n", "1 Biomass and biogas 1990 AT 0.0\n", "2 Biomass and biogas 1990 BA 0.0\n", "3 Biomass and biogas 1990 BE 1.0\n", "4 Biomass and biogas 1990 BG 0.0" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_eurostat['country'].replace({'UK': 'GB', 'EL': 'GR'}, inplace=True)\n", "drop_list = data_eurostat[data_eurostat['country'].isin(['EU28','EA19'])].index\n", "data_eurostat.drop(drop_list, inplace=True)\n", "\n", "by_columns = ['technology', 'year', 'country']\n", "data_eurostat = pd.DataFrame(data_eurostat.groupby(by_columns)['capacity'].sum())\n", "data_eurostat_isnull = data_eurostat['capacity'].isnull() == True\n", "data_eurostat.reset_index(inplace=True)\n", "\n", "data_eurostat.head()" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
technologyBiomass and biogasConcentrated solar powerFossil fuelsGeothermalHydroMarineNon-renewable wasteNuclearOther bioenergy and renewable wastePhotovoltaicsPumped storageWind
countryyear
AL19900.00.00.00.00.00.00.00.00.00.00.00.0
19910.00.00.00.00.00.00.00.00.00.00.00.0
19920.00.00.00.01453.00.00.00.00.00.00.00.0
19930.00.00.00.01453.00.00.00.00.00.00.00.0
19940.00.00.00.01453.00.00.00.00.00.00.00.0
\n", "
" ], "text/plain": [ "technology Biomass and biogas Concentrated solar power Fossil fuels \\\n", "country year \n", "AL 1990 0.0 0.0 0.0 \n", " 1991 0.0 0.0 0.0 \n", " 1992 0.0 0.0 0.0 \n", " 1993 0.0 0.0 0.0 \n", " 1994 0.0 0.0 0.0 \n", "\n", "technology Geothermal Hydro Marine Non-renewable waste Nuclear \\\n", "country year \n", "AL 1990 0.0 0.0 0.0 0.0 0.0 \n", " 1991 0.0 0.0 0.0 0.0 0.0 \n", " 1992 0.0 1453.0 0.0 0.0 0.0 \n", " 1993 0.0 1453.0 0.0 0.0 0.0 \n", " 1994 0.0 1453.0 0.0 0.0 0.0 \n", "\n", "technology Other bioenergy and renewable waste Photovoltaics \\\n", "country year \n", "AL 1990 0.0 0.0 \n", " 1991 0.0 0.0 \n", " 1992 0.0 0.0 \n", " 1993 0.0 0.0 \n", " 1994 0.0 0.0 \n", "\n", "technology Pumped storage Wind \n", "country year \n", "AL 1990 0.0 0.0 \n", " 1991 0.0 0.0 \n", " 1992 0.0 0.0 \n", " 1993 0.0 0.0 \n", " 1994 0.0 0.0 " ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "eurostat_pivot = data_eurostat.pivot_table(values='capacity',\n", " index=['country','year'],\n", " columns='technology')\n", "\n", "eurostat_pivot.head()" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "01 Oct 2020 16:07:59 - numexpr.utils - INFO - NumExpr defaulting to 4 threads.\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
technologyBiomass and biogasConcentrated solar powerFossil fuelsGeothermalHydroMarineNon-renewable wasteNuclearOther bioenergy and renewable wastePhotovoltaicsPumped storageWindDifferently categorized solarSolarDifferently categorized windBioenergy and renewable wasteRenewable energy sourcesDifferently categorized fossil fuelsTotal
countryyear
AL19900.00.00.00.00.00.00.00.00.00.00.00.000.00.00.00.00.00.0
19910.00.00.00.00.00.00.00.00.00.00.00.000.00.00.00.00.00.0
19920.00.00.00.01453.00.00.00.00.00.00.00.000.00.00.01453.00.01453.0
19930.00.00.00.01453.00.00.00.00.00.00.00.000.00.00.01453.00.01453.0
19940.00.00.00.01453.00.00.00.00.00.00.00.000.00.00.01453.00.01453.0
\n", "
" ], "text/plain": [ "technology Biomass and biogas Concentrated solar power Fossil fuels \\\n", "country year \n", "AL 1990 0.0 0.0 0.0 \n", " 1991 0.0 0.0 0.0 \n", " 1992 0.0 0.0 0.0 \n", " 1993 0.0 0.0 0.0 \n", " 1994 0.0 0.0 0.0 \n", "\n", "technology Geothermal Hydro Marine Non-renewable waste Nuclear \\\n", "country year \n", "AL 1990 0.0 0.0 0.0 0.0 0.0 \n", " 1991 0.0 0.0 0.0 0.0 0.0 \n", " 1992 0.0 1453.0 0.0 0.0 0.0 \n", " 1993 0.0 1453.0 0.0 0.0 0.0 \n", " 1994 0.0 1453.0 0.0 0.0 0.0 \n", "\n", "technology Other bioenergy and renewable waste Photovoltaics \\\n", "country year \n", "AL 1990 0.0 0.0 \n", " 1991 0.0 0.0 \n", " 1992 0.0 0.0 \n", " 1993 0.0 0.0 \n", " 1994 0.0 0.0 \n", "\n", "technology Pumped storage Wind Differently categorized solar Solar \\\n", "country year \n", "AL 1990 0.0 0.0 0 0.0 \n", " 1991 0.0 0.0 0 0.0 \n", " 1992 0.0 0.0 0 0.0 \n", " 1993 0.0 0.0 0 0.0 \n", " 1994 0.0 0.0 0 0.0 \n", "\n", "technology Differently categorized wind Bioenergy and renewable waste \\\n", "country year \n", "AL 1990 0.0 0.0 \n", " 1991 0.0 0.0 \n", " 1992 0.0 0.0 \n", " 1993 0.0 0.0 \n", " 1994 0.0 0.0 \n", "\n", "technology Renewable energy sources Differently categorized fossil fuels \\\n", "country year \n", "AL 1990 0.0 0.0 \n", " 1991 0.0 0.0 \n", " 1992 1453.0 0.0 \n", " 1993 1453.0 0.0 \n", " 1994 1453.0 0.0 \n", "\n", "technology Total \n", "country year \n", "AL 1990 0.0 \n", " 1991 0.0 \n", " 1992 1453.0 \n", " 1993 1453.0 \n", " 1994 1453.0 " ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "eurostat_pivot['Differently categorized solar'] = 0\n", "eurostat_pivot['Solar'] = eurostat_pivot[['Photovoltaics', 'Concentrated solar power']].sum(axis=1)\n", "\n", "eurostat_pivot['Differently categorized wind'] = eurostat_pivot['Wind']\n", "\n", "bio_arr = ['Biomass and biogas', 'Other bioenergy and renewable waste']\n", "eurostat_pivot['Bioenergy and renewable waste'] = eurostat_pivot[bio_arr].sum(axis=1)\n", "\n", "res_arr = ['Hydro', 'Wind', 'Solar', 'Geothermal', 'Marine', 'Bioenergy and renewable waste']\n", "eurostat_pivot['Renewable energy sources'] = eurostat_pivot[res_arr].sum(axis=1)\n", "\n", "\n", "eurostat_pivot['Fossil fuels'] = eurostat_pivot['Fossil fuels'] - eurostat_pivot['Bioenergy and renewable waste']\n", "eurostat_pivot['Differently categorized fossil fuels'] = eurostat_pivot['Fossil fuels']\\\n", " - eurostat_pivot['Non-renewable waste']\n", "\n", "total_arr = ['Fossil fuels','Nuclear','Renewable energy sources']\n", "eurostat_pivot['Total'] = eurostat_pivot[total_arr].sum(axis=1)\n", "\n", "eurostat_pivot.head()" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countryyeartechnologycapacitysourcesource_typecapacity_definitiontypeweblink
0AL1990Biomass and biogas0.0EUROSTATStatistical OfficeUnknownInstalled capacity in MWLink unavailable
1AL1990Concentrated solar power0.0EUROSTATStatistical OfficeUnknownInstalled capacity in MWLink unavailable
2AL1990Fossil fuels0.0EUROSTATStatistical OfficeUnknownInstalled capacity in MWLink unavailable
3AL1990Geothermal0.0EUROSTATStatistical OfficeUnknownInstalled capacity in MWLink unavailable
4AL1990Hydro0.0EUROSTATStatistical OfficeUnknownInstalled capacity in MWLink unavailable
\n", "
" ], "text/plain": [ " country year technology capacity source \\\n", "0 AL 1990 Biomass and biogas 0.0 EUROSTAT \n", "1 AL 1990 Concentrated solar power 0.0 EUROSTAT \n", "2 AL 1990 Fossil fuels 0.0 EUROSTAT \n", "3 AL 1990 Geothermal 0.0 EUROSTAT \n", "4 AL 1990 Hydro 0.0 EUROSTAT \n", "\n", " source_type capacity_definition type \\\n", "0 Statistical Office Unknown Installed capacity in MW \n", "1 Statistical Office Unknown Installed capacity in MW \n", "2 Statistical Office Unknown Installed capacity in MW \n", "3 Statistical Office Unknown Installed capacity in MW \n", "4 Statistical Office Unknown Installed capacity in MW \n", "\n", " weblink \n", "0 Link unavailable \n", "1 Link unavailable \n", "2 Link unavailable \n", "3 Link unavailable \n", "4 Link unavailable " ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_eurostat = eurostat_pivot.stack().reset_index().rename(columns={0: 'capacity'})\n", "\n", "data_eurostat['source'] = 'EUROSTAT'\n", "data_eurostat['source_type'] = 'Statistical Office'\n", "data_eurostat['capacity_definition'] = 'Unknown'\n", "data_eurostat['type'] = 'Installed capacity in MW'\n", "data_eurostat['weblink'] = url_eurostat\n", "\n", "data_eurostat.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.3 ENTSO-E data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The ENTSO-E publishes annual data on national generation capacites in different specifications and formats. We use two relevant data sources from the ENTSOE-E, which comprises firstly statistical data within the [Data Portal (up to 2015)](https://www.entsoe.eu/data/data-portal/Pages/default.aspx) or [ENTSO-E Transparency Platform](https://transparency.entsoe.eu/), and secondly datasets compiled within the [ENTSO-E System Outlook & Adequacy Forecast (SO&AF)](https://www.entsoe.eu/outlooks/maf/Pages/default.aspx). The ENTSO-E Transparency Platform is currently not implemented as a data source for national generation capacities.\n", "\n", "The advantage of the ENTSO-E SO&AF is the higher granularity of the data with respect to the main fuel or technology. However, as the SO&AF provides a forecast on future system conditions in particular peak hours, the dataset also accounts for expected capacity changes throughout the years. Therefore, we only consider years which are closest to the publication year of the respective SO&AF." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.3.1 ENTSO-E Statistical Data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the following, we use the statistical data available in the [Data Portal (up to 2015)](https://www.entsoe.eu/data/data-portal/Pages/default.aspx)." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "01 Oct 2020 16:07:59 - root - INFO - Downloading file Statistics.xls\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Countryyearhydroof which storageof which run of riverof which pumped storagenuclearrenewableof which windof which solarof which biomassfossil_fuealsotherrepresentativity
0AT201012665.07205.05241.00.001031.01002.00.00.07389.00.0100.0
1AT201113200.00.00.00.001542.01056.055.00.07818.068.0100.0
2AT201213350.00.00.00.001906.01307.0172.0427.07834.074.0100.0
3AT201313427.00.00.00.002305.01555.0324.0426.07847.0244.0100.0
4AT201413569.00.00.00.003413.02110.0589.0512.07243.00.0100.0
\n", "
" ], "text/plain": [ " Country year hydro of which storage of which run of river \\\n", "0 AT 2010 12665.0 7205.0 5241.0 \n", "1 AT 2011 13200.0 0.0 0.0 \n", "2 AT 2012 13350.0 0.0 0.0 \n", "3 AT 2013 13427.0 0.0 0.0 \n", "4 AT 2014 13569.0 0.0 0.0 \n", "\n", " of which pumped storage nuclear renewable of which wind of which solar \\\n", "0 0.0 0 1031.0 1002.0 0.0 \n", "1 0.0 0 1542.0 1056.0 55.0 \n", "2 0.0 0 1906.0 1307.0 172.0 \n", "3 0.0 0 2305.0 1555.0 324.0 \n", "4 0.0 0 3413.0 2110.0 589.0 \n", "\n", " of which biomass fossil_fueals other representativity \n", "0 0.0 7389.0 0.0 100.0 \n", "1 0.0 7818.0 68.0 100.0 \n", "2 427.0 7834.0 74.0 100.0 \n", "3 426.0 7847.0 244.0 100.0 \n", "4 512.0 7243.0 0.0 100.0 " ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "url_entsoe = 'https://docstore.entsoe.eu/Documents/Publications/Statistics/NGC_2010-2015.xlsx'\n", "\n", "filepath_entsoe = func.downloadandcache(url_entsoe, 'Statistics.xls',\n", " os.path.join('ENTSO-E','Data Portal 2010-2015')\n", " )\n", "\n", "data_entsoe_raw = pd.read_excel(filepath_entsoe)\n", "\n", "data_entsoe_raw.head()" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countryyearHydroReservoirRun-of-riverPumped storageNuclearrenewableWindSolarBiomass and biogasFossil fuelsOther or unspecified energy sources
0AT201012665.07205.05241.00.001031.01002.00.00.07389.00.0
1AT201113200.00.00.00.001542.01056.055.00.07818.068.0
2AT201213350.00.00.00.001906.01307.0172.0427.07834.074.0
3AT201313427.00.00.00.002305.01555.0324.0426.07847.0244.0
4AT201413569.00.00.00.003413.02110.0589.0512.07243.00.0
\n", "
" ], "text/plain": [ " country year Hydro Reservoir Run-of-river Pumped storage Nuclear \\\n", "0 AT 2010 12665.0 7205.0 5241.0 0.0 0 \n", "1 AT 2011 13200.0 0.0 0.0 0.0 0 \n", "2 AT 2012 13350.0 0.0 0.0 0.0 0 \n", "3 AT 2013 13427.0 0.0 0.0 0.0 0 \n", "4 AT 2014 13569.0 0.0 0.0 0.0 0 \n", "\n", " renewable Wind Solar Biomass and biogas Fossil fuels \\\n", "0 1031.0 1002.0 0.0 0.0 7389.0 \n", "1 1542.0 1056.0 55.0 0.0 7818.0 \n", "2 1906.0 1307.0 172.0 427.0 7834.0 \n", "3 2305.0 1555.0 324.0 426.0 7847.0 \n", "4 3413.0 2110.0 589.0 512.0 7243.0 \n", "\n", " Other or unspecified energy sources \n", "0 0.0 \n", "1 68.0 \n", "2 74.0 \n", "3 244.0 \n", "4 0.0 " ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dict_energy_source = {'hydro': 'Hydro',\n", " 'of which storage': 'Reservoir',\n", " 'of which run of river': 'Run-of-river',\n", " 'of which pumped storage': 'Pumped storage',\n", " 'nuclear': 'Nuclear',\n", " 'of which wind': 'Wind',\n", " 'of which solar': 'Solar',\n", " 'of which biomass': 'Biomass and biogas',\n", " 'fossil_fuels': 'Fossil fuels',\n", " 'other': 'Other or unspecified energy sources',\n", " \"Country\": \"country\",\n", " 'fossil_fueals': 'Fossil fuels'}\n", "\n", "data_entsoe_raw.rename(columns=dict_energy_source,\n", " inplace=True)\n", "\n", "data_entsoe_raw.drop(columns='representativity', inplace=True)\n", "\n", "data_entsoe_raw.head()" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countryyeartechnologycapacity
0AT2010Hydro12665.0
1AT2011Hydro13200.0
2AT2012Hydro13350.0
3AT2013Hydro13427.0
4AT2014Hydro13569.0
\n", "
" ], "text/plain": [ " country year technology capacity\n", "0 AT 2010 Hydro 12665.0\n", "1 AT 2011 Hydro 13200.0\n", "2 AT 2012 Hydro 13350.0\n", "3 AT 2013 Hydro 13427.0\n", "4 AT 2014 Hydro 13569.0" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "\n", "data_entsoe_raw['Differently categorized solar'] = data_entsoe_raw['Solar']\n", "data_entsoe_raw['Differently categorized wind'] = data_entsoe_raw['Wind']\n", "data_entsoe_raw['Bioenergy and renewable waste'] = data_entsoe_raw['Biomass and biogas']\n", "data_entsoe_raw['Differently categorized fossil fuels'] = data_entsoe_raw['Fossil fuels']\n", "\n", "\n", "data_entsoe_raw['Differently categorized hydro'] = (\n", " data_entsoe_raw['Hydro']\n", " - data_entsoe_raw['Run-of-river']\n", " - data_entsoe_raw['Reservoir']\n", " - data_entsoe_raw['Pumped storage'])\n", "\n", "data_entsoe_raw['Differently categorized renewable energy sources'] = (\n", " data_entsoe_raw['renewable']\n", " - data_entsoe_raw['Wind']\n", " - data_entsoe_raw['Solar']\n", " - data_entsoe_raw['Biomass and biogas'])\n", "\n", "data_entsoe_raw.drop(columns='renewable', inplace=True)\n", "\n", "data_entsoe_raw['Renewable energy sources'] = (\n", " data_entsoe_raw['Hydro']\n", " + data_entsoe_raw['Wind']\n", " + data_entsoe_raw['Solar']\n", " + data_entsoe_raw['Bioenergy and renewable waste']\n", " + data_entsoe_raw['Differently categorized renewable energy sources'])\n", "\n", "data_entsoe_raw['Total'] = (\n", " data_entsoe_raw['Renewable energy sources']\n", " + data_entsoe_raw['Nuclear']\n", " + data_entsoe_raw['Fossil fuels']\n", " + data_entsoe_raw['Other or unspecified energy sources'])\n", "\n", "data_entsoe = pd.melt(data_entsoe_raw,\n", " id_vars=['country', 'year'],\n", " var_name='technology',\n", " value_name='capacity')\n", "\n", "data_entsoe.head()" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countryyeartechnologycapacitysourcesource_typecapacity_definitiontype
0AT2010Hydro12665.0ENTSO-E Data PortalOther associationNet capacityInstalled capacity in MW
1AT2011Hydro13200.0ENTSO-E Data PortalOther associationNet capacityInstalled capacity in MW
2AT2012Hydro13350.0ENTSO-E Data PortalOther associationNet capacityInstalled capacity in MW
3AT2013Hydro13427.0ENTSO-E Data PortalOther associationNet capacityInstalled capacity in MW
4AT2014Hydro13569.0ENTSO-E Data PortalOther associationNet capacityInstalled capacity in MW
\n", "
" ], "text/plain": [ " country year technology capacity source source_type \\\n", "0 AT 2010 Hydro 12665.0 ENTSO-E Data Portal Other association \n", "1 AT 2011 Hydro 13200.0 ENTSO-E Data Portal Other association \n", "2 AT 2012 Hydro 13350.0 ENTSO-E Data Portal Other association \n", "3 AT 2013 Hydro 13427.0 ENTSO-E Data Portal Other association \n", "4 AT 2014 Hydro 13569.0 ENTSO-E Data Portal Other association \n", "\n", " capacity_definition type \n", "0 Net capacity Installed capacity in MW \n", "1 Net capacity Installed capacity in MW \n", "2 Net capacity Installed capacity in MW \n", "3 Net capacity Installed capacity in MW \n", "4 Net capacity Installed capacity in MW " ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_entsoe['country'].replace('NI', 'GB', inplace=True)\n", "# set negative capacities to zero\n", "data_entsoe.loc[data_entsoe['capacity'] < 0, 'capacity'] = 0\n", "\n", "data_entsoe['source'] = 'ENTSO-E Data Portal'\n", "data_entsoe['source_type'] = 'Other association'\n", "data_entsoe['capacity_definition'] = 'Net capacity'\n", "data_entsoe['type'] = 'Installed capacity in MW'\n", "\n", "data_entsoe.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.3.2 ENTSO-E SO&AF data" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "01 Oct 2020 16:08:00 - root - INFO - Downloading file SO_AF_2011_-_2025_.zip\n", "01 Oct 2020 16:08:01 - root - INFO - Extracting SO_AF_2011_-_2025_.zip into the directory download\\ENTSO-E\\SO&AF\\2011\n", "01 Oct 2020 16:08:01 - root - INFO - Reading download\\ENTSO-E\\SO&AF\\2011\\SO&AF 2011 - 2025 Scenario B.xls and transforming into a DataFrame\n", "01 Oct 2020 16:08:02 - root - INFO - Downloading file 120705_SOAF_2012_Dataset.zip\n", "01 Oct 2020 16:08:02 - root - INFO - Extracting 120705_SOAF_2012_Dataset.zip into the directory download\\ENTSO-E\\SO&AF\\2012\n", "01 Oct 2020 16:08:02 - root - INFO - Reading download\\ENTSO-E\\SO&AF\\2012\\SOAF 2012 Scenario B.xls and transforming into a DataFrame\n", "01 Oct 2020 16:08:03 - root - INFO - Downloading file 130403_SOAF_2013-2030_dataset.zip\n", "01 Oct 2020 16:08:04 - root - INFO - Extracting 130403_SOAF_2013-2030_dataset.zip into the directory download\\ENTSO-E\\SO&AF\\2013\n", "01 Oct 2020 16:08:04 - root - INFO - Reading download\\ENTSO-E\\SO&AF\\2013\\ScB.xls and transforming into a DataFrame\n", "01 Oct 2020 16:08:05 - root - INFO - Downloading file 140602_SOAF%202014_dataset.zip\n", "01 Oct 2020 16:08:05 - root - INFO - Extracting 140602_SOAF%202014_dataset.zip into the directory download\\ENTSO-E\\SO&AF\\2014\n", "01 Oct 2020 16:08:05 - root - INFO - Reading download\\ENTSO-E\\SO&AF\\2014\\ScB.xlsx and transforming into a DataFrame\n", "01 Oct 2020 16:08:06 - root - INFO - Downloading file SO_AF_2015_dataset.zip\n", "01 Oct 2020 16:08:07 - root - INFO - Extracting SO_AF_2015_dataset.zip into the directory download\\ENTSO-E\\SO&AF\\2016\n", "01 Oct 2020 16:08:07 - root - INFO - Reading download\\ENTSO-E\\SO&AF\\2016\\SO&AF 2015 dataset\\ScB_publication.xlsx and transforming into a DataFrame\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
technologycapacitycountryyear
1Nuclear0.0AL2011
2Fossil fuels0.0AL2011
3Lignite0.0AL2011
4Hard coal0.0AL2011
5Natural gas0.0AL2011
\n", "
" ], "text/plain": [ " technology capacity country year\n", "1 Nuclear 0.0 AL 2011\n", "2 Fossil fuels 0.0 AL 2011\n", "3 Lignite 0.0 AL 2011\n", "4 Hard coal 0.0 AL 2011\n", "5 Natural gas 0.0 AL 2011" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "soafs = [soaf.SoafDataRaw('https://www.entsoe.eu/fileadmin/user_upload/_library/SDC/SOAF/SO_AF_2011_-_2025_.zip',\n", " 'SO_AF_2011_-_2025_.zip',\n", " 'SO&AF 2011 - 2025 Scenario B.xls',\n", " 2011),\n", "\n", " soaf.SoafDataRaw('https://www.entsoe.eu/fileadmin/user_upload/_library/SDC/SOAF/120705_SOAF_2012_Dataset.zip',\n", " '120705_SOAF_2012_Dataset.zip',\n", " 'SOAF 2012 Scenario B.xls',\n", " 2012),\n", " \n", " soaf.SoafDataRaw('https://www.entsoe.eu/fileadmin/user_upload/_library/publications/entsoe/So_AF_2013-2030/130403_SOAF_2013-2030_dataset.zip',\n", " '130403_SOAF_2013-2030_dataset.zip',\n", " 'ScB.xls',\n", " 2013),\n", " \n", " soaf.SoafDataRaw('https://www.entsoe.eu/Documents/SDC%20documents/SOAF/140602_SOAF%202014_dataset.zip',\n", " '140602_SOAF%202014_dataset.zip',\n", " 'ScB.xlsx',\n", " 2014),\n", " \n", " soaf.SoafDataRaw('https://www.entsoe.eu/Documents/Publications/SDC/data/SO_AF_2015_dataset.zip',\n", " 'SO_AF_2015_dataset.zip',\n", " os.path.join('SO&AF 2015 dataset', 'ScB_publication.xlsx'),\n", " 2016)]\n", "\n", "\n", "data_soaf = pd.concat([s.transformed_df for s in soafs])\n", "\n", "# Correct that in the Soaf2015 datatset the year column is 2016 instead of 2015\n", "data_soaf['year'].replace({2016 : 2015}, inplace=True)\n", "\n", "data_soaf.head()" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
technologyBiomass and biogasFossil fuelsHard coalHydroLigniteMixed fossil fuelsNatural gasNuclearOffshoreOil...SolarWindDifferently categorized solarDifferently categorized windDifferently categorized hydroBioenergy and renewable wasteDifferently categorized renewable energy sourcesDifferently categorized fossil fuelsRenewable energy sourcesTotal
countryyear
AL20110.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
20120.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.00.0
20130.00.00.00.00.00.00.00.00.00.0...0.00.00.00.0NaN0.00.00.00.00.0
20140.00.00.00.00.00.00.00.00.00.0...0.00.00.00.0NaN0.00.00.00.00.0
20150.0100.00.01860.00.00.0100.00.00.00.0...0.00.00.00.0NaN0.00.00.01860.01960.0
\n", "

5 rows × 24 columns

\n", "
" ], "text/plain": [ "technology Biomass and biogas Fossil fuels Hard coal Hydro Lignite \\\n", "country year \n", "AL 2011 0.0 0.0 0.0 0.0 0.0 \n", " 2012 0.0 0.0 0.0 0.0 0.0 \n", " 2013 0.0 0.0 0.0 0.0 0.0 \n", " 2014 0.0 0.0 0.0 0.0 0.0 \n", " 2015 0.0 100.0 0.0 1860.0 0.0 \n", "\n", "technology Mixed fossil fuels Natural gas Nuclear Offshore Oil ... \\\n", "country year ... \n", "AL 2011 0.0 0.0 0.0 0.0 0.0 ... \n", " 2012 0.0 0.0 0.0 0.0 0.0 ... \n", " 2013 0.0 0.0 0.0 0.0 0.0 ... \n", " 2014 0.0 0.0 0.0 0.0 0.0 ... \n", " 2015 0.0 100.0 0.0 0.0 0.0 ... \n", "\n", "technology Solar Wind Differently categorized solar \\\n", "country year \n", "AL 2011 0.0 0.0 0.0 \n", " 2012 0.0 0.0 0.0 \n", " 2013 0.0 0.0 0.0 \n", " 2014 0.0 0.0 0.0 \n", " 2015 0.0 0.0 0.0 \n", "\n", "technology Differently categorized wind Differently categorized hydro \\\n", "country year \n", "AL 2011 0.0 0.0 \n", " 2012 0.0 0.0 \n", " 2013 0.0 NaN \n", " 2014 0.0 NaN \n", " 2015 0.0 NaN \n", "\n", "technology Bioenergy and renewable waste \\\n", "country year \n", "AL 2011 0.0 \n", " 2012 0.0 \n", " 2013 0.0 \n", " 2014 0.0 \n", " 2015 0.0 \n", "\n", "technology Differently categorized renewable energy sources \\\n", "country year \n", "AL 2011 0.0 \n", " 2012 0.0 \n", " 2013 0.0 \n", " 2014 0.0 \n", " 2015 0.0 \n", "\n", "technology Differently categorized fossil fuels Renewable energy sources \\\n", "country year \n", "AL 2011 0.0 0.0 \n", " 2012 0.0 0.0 \n", " 2013 0.0 0.0 \n", " 2014 0.0 0.0 \n", " 2015 0.0 1860.0 \n", "\n", "technology Total \n", "country year \n", "AL 2011 0.0 \n", " 2012 0.0 \n", " 2013 0.0 \n", " 2014 0.0 \n", " 2015 1960.0 \n", "\n", "[5 rows x 24 columns]" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "soaf_unstacked = func.unstackData(data_soaf)\n", "\n", "soaf_unstacked['Differently categorized solar'] = soaf_unstacked['Solar']\n", "\n", "soaf_unstacked['Differently categorized wind'] = soaf_unstacked['Wind']\\\n", " - soaf_unstacked['Offshore']\\\n", " - soaf_unstacked['Onshore']\n", "\n", "soaf_unstacked['Differently categorized hydro'] = soaf_unstacked['Hydro']\\\n", " - soaf_unstacked['Run-of-river']\\\n", " - soaf_unstacked['Reservoir including pumped storage']\n", " \n", "\n", "soaf_unstacked['Bioenergy and renewable waste'] = soaf_unstacked['Biomass and biogas']\n", " \n", "soaf_unstacked['Differently categorized renewable energy sources'] = (\n", " soaf_unstacked['renewable']\n", " - soaf_unstacked['Wind']\n", " - soaf_unstacked['Solar']\n", " - soaf_unstacked['Biomass and biogas'])\n", "\n", "soaf_unstacked.drop(columns='renewable', inplace=True)\n", "\n", "subtract_fossils_arr = ['Lignite','Hard coal','Oil','Natural gas','Mixed fossil fuels']\n", "\n", "soaf_unstacked['Differently categorized fossil fuels'] = soaf_unstacked['Fossil fuels']\\\n", " - soaf_unstacked[subtract_fossils_arr].sum(axis=1)\n", "\n", "\n", "res_arr = ['Solar','Wind','Bioenergy and renewable waste','Hydro',\n", " 'Differently categorized renewable energy sources']\n", "\n", "soaf_unstacked['Renewable energy sources'] = soaf_unstacked[res_arr].sum(axis=1)\n", "\n", "total_arr = ['Renewable energy sources','Fossil fuels','Nuclear',\n", " 'Other or unspecified energy sources']\n", "\n", "soaf_unstacked['Total'] = soaf_unstacked[total_arr].sum(axis=1)\n", "\n", "soaf_unstacked.head()" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countryyeartechnologycapacitysourcetypecapacity_definitionsource_typeweblink
0AL2011Biomass and biogas0.0ENTSO-E SOAFInstalled capacity in MWNet capacityOther associationhttps://docstore.entsoe.eu/Documents/Publicati...
1AL2011Fossil fuels0.0ENTSO-E SOAFInstalled capacity in MWNet capacityOther associationhttps://docstore.entsoe.eu/Documents/Publicati...
2AL2011Hard coal0.0ENTSO-E SOAFInstalled capacity in MWNet capacityOther associationhttps://docstore.entsoe.eu/Documents/Publicati...
3AL2011Hydro0.0ENTSO-E SOAFInstalled capacity in MWNet capacityOther associationhttps://docstore.entsoe.eu/Documents/Publicati...
4AL2011Lignite0.0ENTSO-E SOAFInstalled capacity in MWNet capacityOther associationhttps://docstore.entsoe.eu/Documents/Publicati...
\n", "
" ], "text/plain": [ " country year technology capacity source \\\n", "0 AL 2011 Biomass and biogas 0.0 ENTSO-E SOAF \n", "1 AL 2011 Fossil fuels 0.0 ENTSO-E SOAF \n", "2 AL 2011 Hard coal 0.0 ENTSO-E SOAF \n", "3 AL 2011 Hydro 0.0 ENTSO-E SOAF \n", "4 AL 2011 Lignite 0.0 ENTSO-E SOAF \n", "\n", " type capacity_definition source_type \\\n", "0 Installed capacity in MW Net capacity Other association \n", "1 Installed capacity in MW Net capacity Other association \n", "2 Installed capacity in MW Net capacity Other association \n", "3 Installed capacity in MW Net capacity Other association \n", "4 Installed capacity in MW Net capacity Other association \n", "\n", " weblink \n", "0 https://docstore.entsoe.eu/Documents/Publicati... \n", "1 https://docstore.entsoe.eu/Documents/Publicati... \n", "2 https://docstore.entsoe.eu/Documents/Publicati... \n", "3 https://docstore.entsoe.eu/Documents/Publicati... \n", "4 https://docstore.entsoe.eu/Documents/Publicati... " ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_soaf = func.restackData(soaf_unstacked)\n", "\n", "data_soaf.loc[data_soaf['capacity'] < 0, 'capacity'] = 0\n", "\n", "data_soaf['source'] = 'ENTSO-E SOAF'\n", "data_soaf['type'] = 'Installed capacity in MW'\n", "data_soaf['capacity_definition'] = 'Net capacity'\n", "data_soaf['source_type'] = 'Other association'\n", "data_soaf['weblink'] = url_entsoe\n", "\n", "data_soaf.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.3.3 ENTSO-E Transparency Platform" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
YearMonthDayDateTimeResolutionCodeareacodeAreaTypeCodeAreaNameMapCodeProductionTypeAggregatedInstalledCapacityDeletedFlagUpdateTime
02015112015-01-01 00:00:00.000P1Y10YAT-APG------LCTAAPG CAATOther22.802017-09-13 15:14:19
12015112015-01-01 00:00:00.000P1Y10YAT-APG------LCTAAPG CAATWind Offshore0.002017-09-13 15:14:19
22015112015-01-01 00:00:00.000P1Y10YAT-APG------LCTAAPG CAATFossil Brown coal/Lignite0.002017-09-13 15:14:19
32015112015-01-01 00:00:00.000P1Y10YAT-APG------LCTAAPG CAATNuclear0.002017-09-13 15:14:19
42015112015-01-01 00:00:00.000P1Y10YAT-APG------LCTAAPG CAATFossil Hard coal1171.002017-09-13 15:14:19
\n", "
" ], "text/plain": [ " Year Month Day DateTime ResolutionCode areacode \\\n", "0 2015 1 1 2015-01-01 00:00:00.000 P1Y 10YAT-APG------L \n", "1 2015 1 1 2015-01-01 00:00:00.000 P1Y 10YAT-APG------L \n", "2 2015 1 1 2015-01-01 00:00:00.000 P1Y 10YAT-APG------L \n", "3 2015 1 1 2015-01-01 00:00:00.000 P1Y 10YAT-APG------L \n", "4 2015 1 1 2015-01-01 00:00:00.000 P1Y 10YAT-APG------L \n", "\n", " AreaTypeCode AreaName MapCode ProductionType \\\n", "0 CTA APG CA AT Other \n", "1 CTA APG CA AT Wind Offshore \n", "2 CTA APG CA AT Fossil Brown coal/Lignite \n", "3 CTA APG CA AT Nuclear \n", "4 CTA APG CA AT Fossil Hard coal \n", "\n", " AggregatedInstalledCapacity DeletedFlag UpdateTime \n", "0 22.8 0 2017-09-13 15:14:19 \n", "1 0.0 0 2017-09-13 15:14:19 \n", "2 0.0 0 2017-09-13 15:14:19 \n", "3 0.0 0 2017-09-13 15:14:19 \n", "4 1171.0 0 2017-09-13 15:14:19 " ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# file pattern for the single years\n", "filenamepattern = '_1_InstalledGenerationCapacityAggregated.csv'\n", "list_of_data_tables = [] # list to append\n", "\n", "# iterate over the years from 2015 to 2020\n", "for i in range(2015,2021):\n", " filepath = os.path.join('input',\n", " 'ENTSO-E',\n", " 'Transparency',\n", " 'InstalledGenerationCapacityAggregated',\n", " str(i) + filenamepattern)\n", " \n", " list_of_data_tables.append(pd.read_csv(filepath, delimiter=\"\\t\", encoding = \"UTF-16\"))\n", "\n", "# merge the datasets of the single of files into one pandas dataframe\n", "data_transparency = pd.concat(list_of_data_tables, ignore_index=True)\n", "\n", "data_transparency.head()" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
technologycapacitycountryyear
0Other22.8AT2015
1Wind Offshore0.0AT2015
2Fossil Brown coal/Lignite0.0AT2015
3Nuclear0.0AT2015
4Fossil Hard coal1171.0AT2015
\n", "
" ], "text/plain": [ " technology capacity country year\n", "0 Other 22.8 AT 2015\n", "1 Wind Offshore 0.0 AT 2015\n", "2 Fossil Brown coal/Lignite 0.0 AT 2015\n", "3 Nuclear 0.0 AT 2015\n", "4 Fossil Hard coal 1171.0 AT 2015" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# rename columns according to the opsd standards\n", "data_transparency.rename(columns={'ProductionType': 'technology',\n", " 'AggregatedInstalledCapacity': 'capacity',\n", " 'MapCode': 'country',\n", " 'Year': 'year'}, inplace=True)\n", "\n", "# drop non relevant columns\n", "data_transparency = data_transparency.filter(items=['technology','capacity','country','year'], axis=1)\n", "# drop countries that are not part of opsd\n", "data_transparency = data_transparency[data_transparency['country'].isin(data_opsd.country.unique())]\n", "\n", "data_transparency.head()" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
technologycapacitycountryyear
0Other or unspecified energy sources22.8AT2015
1Offshore0.0AT2015
2Lignite0.0AT2015
3Nuclear0.0AT2015
4Hard coal1171.0AT2015
\n", "
" ], "text/plain": [ " technology capacity country year\n", "0 Other or unspecified energy sources 22.8 AT 2015\n", "1 Offshore 0.0 AT 2015\n", "2 Lignite 0.0 AT 2015\n", "3 Nuclear 0.0 AT 2015\n", "4 Hard coal 1171.0 AT 2015" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# adapt energy source notation\n", "dict_energy_source = {'Biomass': 'Biomass and biogas',\n", " 'Fossil Brown coal/Lignite': 'Lignite',\n", " 'Fossil Coal-derived gas': 'Mixed fossil fuels',\n", " 'Fossil Gas': 'Natural gas',\n", " 'Fossil Hard coal': 'Hard coal',\n", " 'Fossil Oil': 'Oil',\n", " 'Fossil Oil shale': 'Oil',\n", " 'Fossil Peat': 'Other fossil fuels',\n", " 'Hydro Pumped Storage': 'Pumped storage',\n", " 'Hydro Run-of-river and poundage': 'Run-of-river',\n", " 'Hydro Water Reservoir': 'Reservoir',\n", " 'Other': 'Other or unspecified energy sources',\n", " 'Other renewable': 'Differently categorized renewable energy sources',\n", " 'Waste': 'Other bioenergy and renewable waste',\n", " 'Wind Offshore': 'Offshore',\n", " 'Wind Onshore': 'Onshore',\n", " ' ': np.nan}\n", "\n", "data_transparency['technology'].replace(dict_energy_source, inplace=True)\n", "\n", "\n", "data_transparency.head()" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
technologycountryyearBiomass and biogasDifferently categorized renewable energy sourcesGeothermalHard coalLigniteMarineMixed fossil fuelsNatural gas...Differently categorized solarDifferently categorized natural gasNon-renewable wasteDifferently categorized fossil fuelsHydroWindBioenergy and renewable wasteRenewable energy sourcesFossil fuelsTotal
0AT2015464.231.30.91171.00.00.00.04501.2...587.04501.20011478.72120.9608.314827.15761.1520611.05
1AT2016473.632.50.9766.00.00.00.04465.8...723.04465.80011510.72497.4617.715382.25320.7520725.75
2AT2017473.832.50.9598.00.00.00.04465.8...1031.04465.80011946.82696.0617.916325.15152.7521500.65
3AT2018491.042.30.9598.00.00.00.04467.7...1193.04467.70011990.42887.0640.816754.45154.6521931.85
4AT2019499.842.30.9598.00.00.00.04463.2...1192.74463.20011117.53034.6649.816037.85150.1521210.75
\n", "

5 rows × 31 columns

\n", "
" ], "text/plain": [ "technology country year Biomass and biogas \\\n", "0 AT 2015 464.2 \n", "1 AT 2016 473.6 \n", "2 AT 2017 473.8 \n", "3 AT 2018 491.0 \n", "4 AT 2019 499.8 \n", "\n", "technology Differently categorized renewable energy sources Geothermal \\\n", "0 31.3 0.9 \n", "1 32.5 0.9 \n", "2 32.5 0.9 \n", "3 42.3 0.9 \n", "4 42.3 0.9 \n", "\n", "technology Hard coal Lignite Marine Mixed fossil fuels Natural gas ... \\\n", "0 1171.0 0.0 0.0 0.0 4501.2 ... \n", "1 766.0 0.0 0.0 0.0 4465.8 ... \n", "2 598.0 0.0 0.0 0.0 4465.8 ... \n", "3 598.0 0.0 0.0 0.0 4467.7 ... \n", "4 598.0 0.0 0.0 0.0 4463.2 ... \n", "\n", "technology Differently categorized solar \\\n", "0 587.0 \n", "1 723.0 \n", "2 1031.0 \n", "3 1193.0 \n", "4 1192.7 \n", "\n", "technology Differently categorized natural gas Non-renewable waste \\\n", "0 4501.2 0 \n", "1 4465.8 0 \n", "2 4465.8 0 \n", "3 4467.7 0 \n", "4 4463.2 0 \n", "\n", "technology Differently categorized fossil fuels Hydro Wind \\\n", "0 0 11478.7 2120.9 \n", "1 0 11510.7 2497.4 \n", "2 0 11946.8 2696.0 \n", "3 0 11990.4 2887.0 \n", "4 0 11117.5 3034.6 \n", "\n", "technology Bioenergy and renewable waste Renewable energy sources \\\n", "0 608.3 14827.1 \n", "1 617.7 15382.2 \n", "2 617.9 16325.1 \n", "3 640.8 16754.4 \n", "4 649.8 16037.8 \n", "\n", "technology Fossil fuels Total \n", "0 5761.15 20611.05 \n", "1 5320.75 20725.75 \n", "2 5152.75 21500.65 \n", "3 5154.65 21931.85 \n", "4 5150.15 21210.75 \n", "\n", "[5 rows x 31 columns]" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# add missing categories\n", "transparency_pivot = data_transparency.pivot_table(values='capacity',\n", " index=['country','year'],\n", " columns='technology')\n", "\n", "# technology level\n", "transparency_pivot['Differently categorized solar'] = transparency_pivot['Solar']\n", "transparency_pivot['Differently categorized natural gas'] = transparency_pivot['Natural gas']\n", "transparency_pivot['Non-renewable waste'] = 0\n", "transparency_pivot['Differently categorized fossil fuels'] = 0\n", "\n", "# level 3\n", "hydro_arr = ['Pumped storage', 'Reservoir', 'Run-of-river']\n", "transparency_pivot['Hydro'] = transparency_pivot[hydro_arr].sum(axis=1)\n", "\n", "wind_arr = ['Onshore', 'Offshore']\n", "transparency_pivot['Wind'] = transparency_pivot[wind_arr].sum(axis=1)\n", "\n", "# level 2\n", "bio_arr = ['Biomass and biogas', 'Other bioenergy and renewable waste']\n", "transparency_pivot['Bioenergy and renewable waste'] = transparency_pivot[bio_arr].sum(axis=1)\n", "\n", "#level 1\n", "res_arr = ['Hydro', 'Wind', 'Solar', 'Geothermal', 'Marine',\n", " 'Bioenergy and renewable waste', 'Differently categorized renewable energy sources']\n", "transparency_pivot['Renewable energy sources'] = transparency_pivot[res_arr].sum(axis=1)\n", "\n", "fossil_arr = ['Lignite', 'Hard coal', 'Oil', 'Natural gas', 'Mixed fossil fuels', 'Other fossil fuels']\n", "transparency_pivot['Fossil fuels'] = transparency_pivot[fossil_arr].sum(axis=1)\n", "\n", "# level 0\n", "total_arr = ['Fossil fuels','Nuclear','Renewable energy sources', 'Other or unspecified energy sources']\n", "transparency_pivot['Total'] = transparency_pivot[total_arr].sum(axis=1)\n", "\n", "transparency_pivot.reset_index(inplace=True)\n", "\n", "transparency_pivot.head()" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countryyeartechnologycapacitysourcesource_typecapacity_definitiontypeweblink
0AT2015Biomass and biogas464.2ENTSO-E Transparency PlatformOther associationNet capacityInstalled capacity in MWhttps://transparency.entsoe.eu/generation/r2/i...
1AT2016Biomass and biogas473.6ENTSO-E Transparency PlatformOther associationNet capacityInstalled capacity in MWhttps://transparency.entsoe.eu/generation/r2/i...
2AT2017Biomass and biogas473.8ENTSO-E Transparency PlatformOther associationNet capacityInstalled capacity in MWhttps://transparency.entsoe.eu/generation/r2/i...
3AT2018Biomass and biogas491.0ENTSO-E Transparency PlatformOther associationNet capacityInstalled capacity in MWhttps://transparency.entsoe.eu/generation/r2/i...
4AT2019Biomass and biogas499.8ENTSO-E Transparency PlatformOther associationNet capacityInstalled capacity in MWhttps://transparency.entsoe.eu/generation/r2/i...
\n", "
" ], "text/plain": [ " country year technology capacity source \\\n", "0 AT 2015 Biomass and biogas 464.2 ENTSO-E Transparency Platform \n", "1 AT 2016 Biomass and biogas 473.6 ENTSO-E Transparency Platform \n", "2 AT 2017 Biomass and biogas 473.8 ENTSO-E Transparency Platform \n", "3 AT 2018 Biomass and biogas 491.0 ENTSO-E Transparency Platform \n", "4 AT 2019 Biomass and biogas 499.8 ENTSO-E Transparency Platform \n", "\n", " source_type capacity_definition type \\\n", "0 Other association Net capacity Installed capacity in MW \n", "1 Other association Net capacity Installed capacity in MW \n", "2 Other association Net capacity Installed capacity in MW \n", "3 Other association Net capacity Installed capacity in MW \n", "4 Other association Net capacity Installed capacity in MW \n", "\n", " weblink \n", "0 https://transparency.entsoe.eu/generation/r2/i... \n", "1 https://transparency.entsoe.eu/generation/r2/i... \n", "2 https://transparency.entsoe.eu/generation/r2/i... \n", "3 https://transparency.entsoe.eu/generation/r2/i... \n", "4 https://transparency.entsoe.eu/generation/r2/i... " ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_transparency = pd.melt(transparency_pivot,\n", " id_vars=['country', 'year'],\n", " var_name='technology',\n", " value_name='capacity')\n", "\n", "data_transparency = data_transparency.loc[data_transparency[\"year\"] < 2020, :]\n", "\n", "data_transparency['source'] = 'ENTSO-E Transparency Platform'\n", "data_transparency['source_type'] = 'Other association'\n", "data_transparency['capacity_definition'] = 'Net capacity'\n", "data_transparency['type'] = 'Installed capacity in MW'\n", "data_transparency['weblink'] = ('https://transparency.entsoe.eu/generation'\n", " '/r2/installedGenerationCapacityAggregation/show')\n", "\n", "data_transparency.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.3.4 ENTSO-E Power Statistics" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
technologycountrycapacityyear
0Non-RenewableAL02014
1NuclearALNaN2014
2Non-renwable hydroAL02014
3Of which hydro pure pumped storageALNaN2014
4Of which Hydro mixed pumped storage (non renew...ALNaN2014
\n", "
" ], "text/plain": [ " technology country capacity year\n", "0 Non-Renewable AL 0 2014\n", "1 Nuclear AL NaN 2014\n", "2 Non-renwable hydro AL 0 2014\n", "3 Of which hydro pure pumped storage AL NaN 2014\n", "4 Of which Hydro mixed pumped storage (non renew... AL NaN 2014" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "row_of_year = {2014: 9,\n", " 2015: 53,\n", " 2016: 97,\n", " 2017: 141,\n", " 2018: 185}\n", "\n", "dataframes = []\n", "\n", "for year, row in row_of_year.items():\n", " # read the dataframe for each year\n", " power_statistics_raw = pd.read_excel(os.path.join('input',\n", " 'ENTSO-E',\n", " 'Power Statistics',\n", " 'NGC.xlsx'),\n", " header=[0,1], \n", " sheet_name='NGC',\n", " skiprows=row,\n", " nrows=42)\n", " \n", " # drop non relevant columns\n", " power_statistics_raw.drop(columns='Coverage ratio in %', level=1, inplace=True)\n", " power_statistics_raw.drop(columns=['Unnamed: 1_level_1','Unnamed: 2_level_1','Unnamed: 3_level_1','Unnamed: 4_level_1'], level=1, inplace=True)\n", "\n", " # get rid of multi index\n", " df = power_statistics_raw.set_index(year).stack().reset_index().drop('level_1', axis=1)\n", " \n", " # remove leftovers of multi index in the index column\n", " df[\"technology\"] = df[year].apply(lambda x: x[0])\n", " df.drop(columns=year, inplace=True)\n", " \n", " # stack df to the opsd standard format\n", " stacked_df = df.melt(id_vars='technology', var_name='country', value_name='capacity')\n", " \n", " # add information about the year\n", " stacked_df['year'] = year\n", " \n", " # append to the main list of dataframes\n", " dataframes.append(stacked_df)\n", " \n", "power_statistics = pd.concat(dataframes)\n", "power_statistics.head()" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
technologycountrycapacityyear
41NuclearAT0.02014
43Of which hydro pure pumped storageAT0.02014
44Of which Hydro mixed pumped storage (non renew...AT0.02014
46Of which Fossil Brown coal/LigniteAT0.02014
47Of which Fossil Coal-derived gasAT0.02014
\n", "
" ], "text/plain": [ " technology country capacity year\n", "41 Nuclear AT 0.0 2014\n", "43 Of which hydro pure pumped storage AT 0.0 2014\n", "44 Of which Hydro mixed pumped storage (non renew... AT 0.0 2014\n", "46 Of which Fossil Brown coal/Lignite AT 0.0 2014\n", "47 Of which Fossil Coal-derived gas AT 0.0 2014" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# drop countries that are not covered in opsd\n", "opsd_countries = data_opsd.country.unique()\n", "drop_list_country = power_statistics.loc[~power_statistics['country'].isin(opsd_countries)].index.to_list()\n", "power_statistics.drop(drop_list_country, inplace=True)\n", "\n", "# technology classes to be dropped\n", "tech_to_drop = ['Non-Renewable', 'Fossil fuels', 'Renewable','Non-renwable hydro',\n", " 'Total Waste', 'Bio', 'Renewable Hydro', 'Comments', 'Total NGC']\n", "\n", "drop_list_tech = power_statistics.loc[power_statistics['technology'].isin(tech_to_drop)].index.to_list()\n", "power_statistics.drop(drop_list_tech, inplace=True)\n", "\n", "# replace string with values that can be used in math operations \n", "power_statistics['capacity'].replace(to_replace='Not Expected', value=0, inplace=True)\n", "power_statistics['capacity'].replace(to_replace='Not Available', value=np.nan, inplace=True)\n", "\n", "power_statistics.head()" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
technologycountrycapacityyear
41NuclearAT0.02014
43Pumped storageAT0.02014
44Pumped storageAT0.02014
46LigniteAT0.02014
47Differently categorized fossil fuelsAT0.02014
\n", "
" ], "text/plain": [ " technology country capacity year\n", "41 Nuclear AT 0.0 2014\n", "43 Pumped storage AT 0.0 2014\n", "44 Pumped storage AT 0.0 2014\n", "46 Lignite AT 0.0 2014\n", "47 Differently categorized fossil fuels AT 0.0 2014" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Not included because already categorized in OPSD standard:\n", "# Nuclear, Solar, Geothermal, Wind\n", "dict_energy_source = {'Of which hydro pure pumped storage':'Pumped storage',\n", " 'Of which Hydro mixed pumped storage (non renewable part)':'Pumped storage',\n", " 'Of which Fossil Brown coal/Lignite':'Lignite',\n", " 'Of which Fossil Coal-derived gas':'Differently categorized fossil fuels',\n", " 'Of which Fossil Gas':'Natural gas',\n", " 'Of which Fossil Hard coal':'Hard coal', \n", " 'Of which Fossil Oil':'Oil',\n", " 'Of which Fossil Oil shale':'Oil', \n", " 'Of which Fossil Peat':'Differently categorized fossil fuels',\n", " 'Of which Mixed fuels':'Mixed fossil fuels', \n", " 'Of which Other fossil fuels':'Other fossil fuels',\n", " 'Non-renewable Waste':'Non-renewable waste', \n", " 'Other non-renewable':'Differently categorized fossil fuels', \n", " 'Of which Wind offshore':'Offshore', \n", " 'Of which Wind onshore':'Onshore', \n", " 'Of which Solar PV':'Photovoltaics', \n", " 'Of which Solar Thermal':'Differently categorized solar', \n", " 'Of which Biomass':'Biomass and biogas', \n", " 'Of which Biogas':'Biomass and biogas', \n", " 'Renewable Waste':'Other bioenergy and renewable waste', \n", " 'Of which Hydro Pure storage':'Reservoir',\n", " 'Of which Hydro Run-of-river and pondage':'Run-of-river',\n", " 'Of which Hydro mixed pumped storage (renewable part)':'Pumped storage',\n", " 'Of which Hydro Marine (tidal/wave)':'Marine',\n", " 'Other renewable (not listed)':'Differently categorized renewable energy sources',\n", " 'Non identified (other not listed)':'Other or unspecified energy sources', \n", " 'Total Hydro':'Hydro'}\n", "\n", "power_statistics[\"technology\"].replace(dict_energy_source, inplace=True)\n", "power_statistics.head()" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
technologycountryyearBiomass and biogasDifferently categorized fossil fuelsDifferently categorized renewable energy sourcesDifferently categorized solarGeothermalHard coalHydroLignite...OnshoreOther bioenergy and renewable wasteOther fossil fuelsOther or unspecified energy sourcesPhotovoltaicsPumped storageReservoirRun-of-riverSolarWind
0AT2014302.68529.00.00.01.01171.013568.00.0...2110.021.96414.00.0586.122656.3333330.05599.0586.122110.0
1AT2015298.00030.0NaN0.01.01171.013656.00.0...2489.023.00414.0NaN732.003997.000000NaN5662.0732.002489.0
2AT2016286.00035.0NaN0.01.0598.014116.00.0...2730.022.00414.0NaN1031.004212.000000NaN5692.01031.002730.0
3AT2017286.00035.0NaN0.01.0598.014116.00.0...2730.022.00414.0NaN1031.004212.000000NaN5692.01031.002730.0
4AT2018293.00035.0NaN0.01.0598.014150.00.0...2887.022.00414.0NaN1193.004218.000000NaN5714.01193.002887.0
\n", "

5 rows × 27 columns

\n", "
" ], "text/plain": [ "technology country year Biomass and biogas \\\n", "0 AT 2014 302.685 \n", "1 AT 2015 298.000 \n", "2 AT 2016 286.000 \n", "3 AT 2017 286.000 \n", "4 AT 2018 293.000 \n", "\n", "technology Differently categorized fossil fuels \\\n", "0 29.0 \n", "1 30.0 \n", "2 35.0 \n", "3 35.0 \n", "4 35.0 \n", "\n", "technology Differently categorized renewable energy sources \\\n", "0 0.0 \n", "1 NaN \n", "2 NaN \n", "3 NaN \n", "4 NaN \n", "\n", "technology Differently categorized solar Geothermal Hard coal Hydro \\\n", "0 0.0 1.0 1171.0 13568.0 \n", "1 0.0 1.0 1171.0 13656.0 \n", "2 0.0 1.0 598.0 14116.0 \n", "3 0.0 1.0 598.0 14116.0 \n", "4 0.0 1.0 598.0 14150.0 \n", "\n", "technology Lignite ... Onshore Other bioenergy and renewable waste \\\n", "0 0.0 ... 2110.0 21.96 \n", "1 0.0 ... 2489.0 23.00 \n", "2 0.0 ... 2730.0 22.00 \n", "3 0.0 ... 2730.0 22.00 \n", "4 0.0 ... 2887.0 22.00 \n", "\n", "technology Other fossil fuels Other or unspecified energy sources \\\n", "0 414.0 0.0 \n", "1 414.0 NaN \n", "2 414.0 NaN \n", "3 414.0 NaN \n", "4 414.0 NaN \n", "\n", "technology Photovoltaics Pumped storage Reservoir Run-of-river Solar \\\n", "0 586.12 2656.333333 0.0 5599.0 586.12 \n", "1 732.00 3997.000000 NaN 5662.0 732.00 \n", "2 1031.00 4212.000000 NaN 5692.0 1031.00 \n", "3 1031.00 4212.000000 NaN 5692.0 1031.00 \n", "4 1193.00 4218.000000 NaN 5714.0 1193.00 \n", "\n", "technology Wind \n", "0 2110.0 \n", "1 2489.0 \n", "2 2730.0 \n", "3 2730.0 \n", "4 2887.0 \n", "\n", "[5 rows x 27 columns]" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "powerstats_pivot = power_statistics.pivot_table(values='capacity',\n", " index=['country','year'],\n", " columns='technology').reset_index()\n", "powerstats_pivot.head()" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
technologycountryyearBiomass and biogasDifferently categorized fossil fuelsDifferently categorized renewable energy sourcesDifferently categorized solarGeothermalHard coalHydroLignite...Pumped storageReservoirRun-of-riverSolarWindDifferently categorized natural gasBioenergy and renewable wasteFossil fuelsRenewable energy sourcesTotal
0AT2014302.68529.00.00.01.01171.013568.00.0...2656.3333330.05599.0586.122110.04888.0324.6457105.016589.76523694.765
1AT2015298.00030.0NaN0.01.01171.013656.00.0...3997.000000NaN5662.0732.002489.04820.0321.0007002.017199.00024201.000
2AT2016286.00035.0NaN0.01.0598.014116.00.0...4212.000000NaN5692.01031.002730.04841.0308.0006407.018186.00024593.000
3AT2017286.00035.0NaN0.01.0598.014116.00.0...4212.000000NaN5692.01031.002730.04841.0308.0006407.018186.00024593.000
4AT2018293.00035.0NaN0.01.0598.014150.00.0...4218.000000NaN5714.01193.002887.04853.0315.0006420.018546.00024966.000
\n", "

5 rows × 32 columns

\n", "
" ], "text/plain": [ "technology country year Biomass and biogas \\\n", "0 AT 2014 302.685 \n", "1 AT 2015 298.000 \n", "2 AT 2016 286.000 \n", "3 AT 2017 286.000 \n", "4 AT 2018 293.000 \n", "\n", "technology Differently categorized fossil fuels \\\n", "0 29.0 \n", "1 30.0 \n", "2 35.0 \n", "3 35.0 \n", "4 35.0 \n", "\n", "technology Differently categorized renewable energy sources \\\n", "0 0.0 \n", "1 NaN \n", "2 NaN \n", "3 NaN \n", "4 NaN \n", "\n", "technology Differently categorized solar Geothermal Hard coal Hydro \\\n", "0 0.0 1.0 1171.0 13568.0 \n", "1 0.0 1.0 1171.0 13656.0 \n", "2 0.0 1.0 598.0 14116.0 \n", "3 0.0 1.0 598.0 14116.0 \n", "4 0.0 1.0 598.0 14150.0 \n", "\n", "technology Lignite ... Pumped storage Reservoir Run-of-river Solar \\\n", "0 0.0 ... 2656.333333 0.0 5599.0 586.12 \n", "1 0.0 ... 3997.000000 NaN 5662.0 732.00 \n", "2 0.0 ... 4212.000000 NaN 5692.0 1031.00 \n", "3 0.0 ... 4212.000000 NaN 5692.0 1031.00 \n", "4 0.0 ... 4218.000000 NaN 5714.0 1193.00 \n", "\n", "technology Wind Differently categorized natural gas \\\n", "0 2110.0 4888.0 \n", "1 2489.0 4820.0 \n", "2 2730.0 4841.0 \n", "3 2730.0 4841.0 \n", "4 2887.0 4853.0 \n", "\n", "technology Bioenergy and renewable waste Fossil fuels \\\n", "0 324.645 7105.0 \n", "1 321.000 7002.0 \n", "2 308.000 6407.0 \n", "3 308.000 6407.0 \n", "4 315.000 6420.0 \n", "\n", "technology Renewable energy sources Total \n", "0 16589.765 23694.765 \n", "1 17199.000 24201.000 \n", "2 18186.000 24593.000 \n", "3 18186.000 24593.000 \n", "4 18546.000 24966.000 \n", "\n", "[5 rows x 32 columns]" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# technology level\n", "powerstats_pivot['Differently categorized natural gas'] = powerstats_pivot['Natural gas']\n", "\n", "# level 2\n", "powerstats_pivot['Bioenergy and renewable waste'] = (\n", " powerstats_pivot['Biomass and biogas'] +\n", " powerstats_pivot['Other bioenergy and renewable waste'])\n", "\n", "#level 1\n", "fossil_techs = ['Lignite', 'Hard coal', 'Oil', 'Natural gas', 'Non-renewable waste',\n", " 'Mixed fossil fuels', 'Other fossil fuels', 'Differently categorized fossil fuels']\n", "\n", "powerstats_pivot['Fossil fuels'] = powerstats_pivot[fossil_techs].sum(axis=1)\n", "\n", "res_tech = ['Hydro', 'Wind', 'Solar', 'Geothermal', 'Marine', 'Bioenergy and renewable waste',\n", " 'Differently categorized renewable energy sources']\n", "\n", "powerstats_pivot['Renewable energy sources'] = powerstats_pivot[res_tech].sum(axis=1)\n", "\n", "total_arr = ['Fossil fuels','Nuclear','Renewable energy sources']\n", "powerstats_pivot['Total'] = powerstats_pivot[total_arr].sum(axis=1)\n", "\n", "powerstats_pivot.head()" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countryyeartechnologycapacitysourcesource_typecapacity_definitiontypeweblink
0AT2014Biomass and biogas302.685ENTSO-E Power StatisticsOther associationNet capacityInstalled capacity in MWhttps://www.entsoe.eu/data/power-stats/net-gen...
1AT2015Biomass and biogas298.000ENTSO-E Power StatisticsOther associationNet capacityInstalled capacity in MWhttps://www.entsoe.eu/data/power-stats/net-gen...
2AT2016Biomass and biogas286.000ENTSO-E Power StatisticsOther associationNet capacityInstalled capacity in MWhttps://www.entsoe.eu/data/power-stats/net-gen...
3AT2017Biomass and biogas286.000ENTSO-E Power StatisticsOther associationNet capacityInstalled capacity in MWhttps://www.entsoe.eu/data/power-stats/net-gen...
4AT2018Biomass and biogas293.000ENTSO-E Power StatisticsOther associationNet capacityInstalled capacity in MWhttps://www.entsoe.eu/data/power-stats/net-gen...
\n", "
" ], "text/plain": [ " country year technology capacity source \\\n", "0 AT 2014 Biomass and biogas 302.685 ENTSO-E Power Statistics \n", "1 AT 2015 Biomass and biogas 298.000 ENTSO-E Power Statistics \n", "2 AT 2016 Biomass and biogas 286.000 ENTSO-E Power Statistics \n", "3 AT 2017 Biomass and biogas 286.000 ENTSO-E Power Statistics \n", "4 AT 2018 Biomass and biogas 293.000 ENTSO-E Power Statistics \n", "\n", " source_type capacity_definition type \\\n", "0 Other association Net capacity Installed capacity in MW \n", "1 Other association Net capacity Installed capacity in MW \n", "2 Other association Net capacity Installed capacity in MW \n", "3 Other association Net capacity Installed capacity in MW \n", "4 Other association Net capacity Installed capacity in MW \n", "\n", " weblink \n", "0 https://www.entsoe.eu/data/power-stats/net-gen... \n", "1 https://www.entsoe.eu/data/power-stats/net-gen... \n", "2 https://www.entsoe.eu/data/power-stats/net-gen... \n", "3 https://www.entsoe.eu/data/power-stats/net-gen... \n", "4 https://www.entsoe.eu/data/power-stats/net-gen... " ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_power_statistics = powerstats_pivot.melt(id_vars=['country', 'year'],\n", " var_name='technology',\n", " value_name='capacity')\n", "\n", "data_power_statistics['source'] = 'ENTSO-E Power Statistics'\n", "data_power_statistics['source_type'] = 'Other association'\n", "data_power_statistics['capacity_definition'] = 'Net capacity'\n", "data_power_statistics['type'] = 'Installed capacity in MW'\n", "data_power_statistics['weblink'] = ('https://www.entsoe.eu/data/'\n", " 'power-stats/net-gen-capacity/')\n", "\n", "data_power_statistics.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.4 Merge data sources" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
technologysourcesource_typeweblinkyeartypecountrycapacity_definitioncapacitycommentenergy_source_level_0energy_source_level_1energy_source_level_2energy_source_level_3technology_level
0Fossil fuelse-controlRegulatory Authorityhttps://www.e-control.at/documents/1785851/181...2014Installed capacity in MWATGross capacity7243.634FalseTrueFalseFalseFalse
1Fossil fuelse-controlRegulatory Authorityhttps://www.e-control.at/documents/1785851/181...2015Installed capacity in MWATGross capacity7059.065FalseTrueFalseFalseFalse
2Fossil fuelse-controlRegulatory Authorityhttps://www.e-control.at/documents/1785851/181...2016Installed capacity in MWATGross capacity7323.000FalseTrueFalseFalseFalse
3Fossil fuelse-controlRegulatory Authorityhttps://www.e-control.at/documents/1785851/181...2017Installed capacity in MWATGross capacity6469.000FalseTrueFalseFalseFalse
4Fossil fuelse-controlRegulatory Authorityhttps://www.e-control.at/documents/1785851/181...2018Installed capacity in MWATGross capacity6492.000FalseTrueFalseFalseFalse
\n", "
" ], "text/plain": [ " technology source source_type \\\n", "0 Fossil fuels e-control Regulatory Authority \n", "1 Fossil fuels e-control Regulatory Authority \n", "2 Fossil fuels e-control Regulatory Authority \n", "3 Fossil fuels e-control Regulatory Authority \n", "4 Fossil fuels e-control Regulatory Authority \n", "\n", " weblink year \\\n", "0 https://www.e-control.at/documents/1785851/181... 2014 \n", "1 https://www.e-control.at/documents/1785851/181... 2015 \n", "2 https://www.e-control.at/documents/1785851/181... 2016 \n", "3 https://www.e-control.at/documents/1785851/181... 2017 \n", "4 https://www.e-control.at/documents/1785851/181... 2018 \n", "\n", " type country capacity_definition capacity comment \\\n", "0 Installed capacity in MW AT Gross capacity 7243.634 \n", "1 Installed capacity in MW AT Gross capacity 7059.065 \n", "2 Installed capacity in MW AT Gross capacity 7323.000 \n", "3 Installed capacity in MW AT Gross capacity 6469.000 \n", "4 Installed capacity in MW AT Gross capacity 6492.000 \n", "\n", " energy_source_level_0 energy_source_level_1 energy_source_level_2 \\\n", "0 False True False \n", "1 False True False \n", "2 False True False \n", "3 False True False \n", "4 False True False \n", "\n", " energy_source_level_3 technology_level \n", "0 False False \n", "1 False False \n", "2 False False \n", "3 False False \n", "4 False False " ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataframes = [data_opsd, data_eurostat, data_soaf, data_entsoe, data_transparency, data_power_statistics]\n", "data = pd.concat(dataframes, sort=False)\n", "data['comment'] = data['comment'].fillna('').astype(str)\n", "\n", "col_order = ['technology', 'source', 'source_type', 'weblink', 'year', 'type',\n", " 'country', 'capacity_definition', 'capacity', 'comment']\n", "\n", "data = data[col_order]\n", "\n", "\n", "energy_source_mapping = pd.read_csv(os.path.join('input','energy_source_mapping.csv'),\n", " index_col ='name')\n", "energy_source_mapping.replace({0: False, 1: True}, inplace=True)\n", "\n", "data = data.merge(energy_source_mapping,\n", " left_on='technology',\n", " right_index=True,\n", " how='left')\n", "\n", "new_level_names = {\"Level 0\": \"energy_source_level_0\",\n", " \"Level 1\": \"energy_source_level_1\",\n", " \"Level 2\": \"energy_source_level_2\",\n", " \"Level 3\": \"energy_source_level_3\",\n", " \"Technology level\": \"technology_level\"}\n", "\n", "data.rename(columns=new_level_names, inplace=True)\n", "\n", "data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4. Convert stacked data to crosstable format" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Country (ISO code)AL...XK
Type of dataInstalled capacity in MW...Installed capacity in MW
Year1990199119921993199419951996199719981999...2007200820092010201120122013201420152016
SourceEUROSTATEUROSTATEUROSTATEUROSTATEUROSTATEUROSTATEUROSTATEUROSTATEUROSTATEUROSTAT...EUROSTATEUROSTATEUROSTATEUROSTATEUROSTATEUROSTATEUROSTATEUROSTATEUROSTATEUROSTAT
Type of sourceStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical Office...Statistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical OfficeStatistical Office
WeblinkLink unavailableLink unavailableLink unavailableLink unavailableLink unavailableLink unavailableLink unavailableLink unavailableLink unavailableLink unavailable...Link unavailableLink unavailableLink unavailableLink unavailableLink unavailableLink unavailableLink unavailableLink unavailableLink unavailableLink unavailable
Capacity definition (net, gross, unknown)UnknownUnknownUnknownUnknownUnknownUnknownUnknownUnknownUnknownUnknown...UnknownUnknownUnknownUnknownUnknownUnknownUnknownUnknownUnknownUnknown
Fossil fuels0.00.00.00.00.00.00.00.00.00.0...915.0915.0915.0915.0915.0915.0915.0915.0915.0915.0
LigniteNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
Hard coalNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
OilNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
Natural gasNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
\n", "

5 rows × 1633 columns

\n", "
" ], "text/plain": [ "Country (ISO code) AL \\\n", "Type of data Installed capacity in MW \n", "Year 1990 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 0.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 1991 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 0.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 1992 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 0.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 1993 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 0.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 1994 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 0.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 1995 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 0.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 1996 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 0.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 1997 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 0.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 1998 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 0.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) ... \\\n", "Type of data ... \n", "Year 1999 ... \n", "Source EUROSTAT ... \n", "Type of source Statistical Office ... \n", "Weblink Link unavailable ... \n", "Capacity definition (net, gross, unknown) Unknown ... \n", "Fossil fuels 0.0 ... \n", "Lignite NaN ... \n", "Hard coal NaN ... \n", "Oil NaN ... \n", "Natural gas NaN ... \n", "\n", "Country (ISO code) XK \\\n", "Type of data Installed capacity in MW \n", "Year 2007 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 915.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 2008 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 915.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 2009 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 915.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 2010 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 915.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 2011 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 915.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 2012 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 915.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 2013 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 915.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 2014 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 915.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \\\n", "Type of data \n", "Year 2015 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 915.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "Country (ISO code) \n", "Type of data \n", "Year 2016 \n", "Source EUROSTAT \n", "Type of source Statistical Office \n", "Weblink Link unavailable \n", "Capacity definition (net, gross, unknown) Unknown \n", "Fossil fuels 915.0 \n", "Lignite NaN \n", "Hard coal NaN \n", "Oil NaN \n", "Natural gas NaN \n", "\n", "[5 rows x 1633 columns]" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cols = ['technology', 'source', 'source_type', 'weblink','year',\n", " 'type', 'country', 'capacity_definition', 'capacity']\n", "\n", "data_crosstable = pd.pivot_table(data[cols],\n", " index=['technology'],\n", " columns=['country', 'type', 'year',\n", " 'source', 'source_type',\n", " 'weblink', 'capacity_definition'],\n", " values='capacity')\n", "\n", "# Apply initial ordering of technologies\n", "data_crosstable = data_crosstable.reindex(technology_order)\n", "\n", "# Delete index naming\n", "data_crosstable.index.name = None\n", "data_crosstable.columns.names = ('Country (ISO code)',\n", " 'Type of data', 'Year',\n", " 'Source', 'Type of source',\n", " 'Weblink',\n", " 'Capacity definition (net, gross, unknown)')\n", "\n", "data_crosstable.head()" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Country (ISO code)Energy source levels and technology
DescriptionTotal generation capacityGeneration capacity by energy type (fossil, nuclear, renewable, other)Generation capacity by fuel or energy source (e.g. coal, lignite, hard coal, natural gas, wind)Generation capacity by fuel or energy source and detailed fuel for bioenergyGeneration capacity by fuel and technology (e.g. combined cycle natural gas, gas turbine, onshore wind, offshore wind)
NaNNaNNaNNaNNaN
NaNNaNNaNNaNNaN
NaNNaNNaNNaNNaN
LevelNaNNaNNaNNaNNaN
NaNLevel 0Level 1Level 2Level 3Technology level
Fossil fuels01000
Lignite00111
Hard coal00111
Oil00111
\n", "
" ], "text/plain": [ "Country (ISO code) Energy source levels and technology \\\n", "Description Total generation capacity \n", " NaN \n", " NaN \n", " NaN \n", "Level NaN \n", "NaN Level 0 \n", "Fossil fuels 0 \n", "Lignite 0 \n", "Hard coal 0 \n", "Oil 0 \n", "\n", "Country (ISO code) \\\n", "Description Generation capacity by energy type (fossil, nuclear, renewable, other) \n", " NaN \n", " NaN \n", " NaN \n", "Level NaN \n", "NaN Level 1 \n", "Fossil fuels 1 \n", "Lignite 0 \n", "Hard coal 0 \n", "Oil 0 \n", "\n", "Country (ISO code) \\\n", "Description Generation capacity by fuel or energy source (e.g. coal, lignite, hard coal, natural gas, wind) \n", " NaN \n", " NaN \n", " NaN \n", "Level NaN \n", "NaN Level 2 \n", "Fossil fuels 0 \n", "Lignite 1 \n", "Hard coal 1 \n", "Oil 1 \n", "\n", "Country (ISO code) \\\n", "Description Generation capacity by fuel or energy source and detailed fuel for bioenergy \n", " NaN \n", " NaN \n", " NaN \n", "Level NaN \n", "NaN Level 3 \n", "Fossil fuels 0 \n", "Lignite 1 \n", "Hard coal 1 \n", "Oil 1 \n", "\n", "Country (ISO code) \n", "Description Generation capacity by fuel and technology (e.g. combined cycle natural gas, gas turbine, onshore wind, offshore wind) \n", " NaN \n", " NaN \n", " NaN \n", "Level NaN \n", "NaN Technology level \n", "Fossil fuels 0 \n", "Lignite 1 \n", "Hard coal 1 \n", "Oil 1 " ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "energylevels_table = energylevels_raw[6:]\n", "energylevels_table.columns = pd.MultiIndex.from_arrays(energylevels_raw[:6].values,\n", " names=['country', 'type', 'year',\n", " 'source', 'source_type',\n", " 'capacity_definition'\n", " ])\n", "\n", "energylevels_table = energylevels_table.reset_index()\n", "energylevels_table['technology'] = energylevels_table['technology'].str.replace('- ', '')\n", "energylevels_table = energylevels_table.set_index('technology')\n", "\n", "# Delete index naming\n", "energylevels_table.index.name = None\n", "energylevels_table.columns.names = ('Country (ISO code)',\n", " 'Description', None,\n", " None, None,\n", " 'Level')\n", "\n", "energylevels_table.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 5. Output" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Delete downloaded zip files" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Deleted: download\\ENTSO-E\\SO&AF\\2011\\SO_AF_2011_-_2025_.zip\n", "Deleted: download\\ENTSO-E\\SO&AF\\2012\\120705_SOAF_2012_Dataset.zip\n", "Deleted: download\\ENTSO-E\\SO&AF\\2013\\130403_SOAF_2013-2030_dataset.zip\n", "Deleted: download\\ENTSO-E\\SO&AF\\2014\\140602_SOAF%202014_dataset.zip\n", "Deleted: download\\ENTSO-E\\SO&AF\\2016\\SO_AF_2015_dataset.zip\n" ] } ], "source": [ "for root, dirs, files in os.walk(\"download\"):\n", " for file in files:\n", " item = os.path.join(root, file)\n", " if item.endswith(\".zip\"):\n", " os.remove(item)\n", " print(\"Deleted: \" + item)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Copy input files" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "scrolled": true }, "outputs": [], "source": [ "orig_data_path = os.path.join('output', 'original_data')\n", "shutil.rmtree(orig_data_path, ignore_errors=True)\n", "func.copydir(os.path.join('input'), orig_data_path)\n", "func.copydir(os.path.join('download'), orig_data_path)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5.1 Write results to file" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Write stacked data to formats: csv, xls and sql." ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [], "source": [ "# Write the result to file\n", "data.to_csv(os.path.join('output', 'national_generation_capacity_stacked.csv'),\n", " encoding='utf-8', index_label='ID')\n", "\n", "# Write the results to excel file\n", "data.to_excel(os.path.join('output', 'national_generation_capacity_stacked.xlsx'),\n", " sheet_name='output', index_label='ID')\n", "\n", "# Write the results to sql database\n", "data.to_sql('national_generation_capacity_stacked',\n", " sqlite3.connect(os.path.join('output',\n", " 'national_generation_capacity.sqlite')),\n", " if_exists=\"replace\", index_label='ID')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Write data in human readable form to excel." ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "# Write crosstable data to excel file\n", "writer = pd.ExcelWriter(os.path.join('output', 'national_generation_capacity.xlsx'))\n", "data_crosstable.to_excel(writer, sheet_name='output')\n", "energylevels_table.to_excel(writer, sheet_name='technology levels')\n", "writer.save()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5.2 Formatting of Excel tables" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [], "source": [ "outputxls = openpyxl.load_workbook(os.path.join('output',\n", " 'national_generation_capacity.xlsx'))\n", "\n", "ws1 = outputxls['output']\n", "ws2 = outputxls['technology levels']\n", "ws1_rows, ws1_cols = data_crosstable.shape\n", "amount_cols = ws1_cols + 1 # correct 0 index\n", "\n", "ws1.column_dimensions['A'].width = 50\n", "ws2.column_dimensions['A'].width = 50" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [], "source": [ "blackfont = Font(color=colors.BLACK, italic=False, bold=False)\n", "blackfontitalic = Font(color=colors.BLACK, italic=True, bold=False)\n", "blackfontbold = Font(color=colors.BLACK, italic=False, bold=True)\n", "\n", "align0 = Alignment(horizontal='left', indent=0)\n", "align1 = Alignment(horizontal='left', indent=1)\n", "align2 = Alignment(horizontal='left', indent=2)\n", "\n", "# darkest grey\n", "colour = \"{0:02X}{1:02X}{2:02X}\".format(166, 166, 166)\n", "grey166 = PatternFill(fgColor=colour, bgColor=colour, patternType=\"solid\")\n", "\n", "# darker grey\n", "colour = \"{0:02X}{1:02X}{2:02X}\".format(191, 191, 191)\n", "grey191 = PatternFill(fgColor=colour, bgColor=colour, patternType=\"solid\")\n", "\n", "# lighter grey\n", "colour = \"{0:02X}{1:02X}{2:02X}\".format(217, 217, 217)\n", "grey217 = PatternFill(fgColor=colour, bgColor=colour, patternType=\"solid\")\n", "\n", "# lightest grey\n", "colour = \"{0:02X}{1:02X}{2:02X}\".format(242, 242, 242)\n", "grey242 = PatternFill(fgColor=colour, bgColor=colour, patternType=\"solid\")" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [], "source": [ "for col in range(2, amount_cols+1):\n", " colname = openpyxl.utils.cell.get_column_letter(col)\n", " ws1.column_dimensions[colname].width = 16\n", "\n", "\n", "for col in range(1, amount_cols+1):\n", " # format column name block\n", " for row in range(2,8):\n", " \n", " ws1.cell(row=row, column=col).font = blackfont\n", " \n", "\n", " # format cells that contain the values\n", " for row in range(9, 48):\n", " ws1.cell(row=row, column=col).fill = grey242\n", " ws1.cell(row=row, column=1).font = blackfontitalic\n", " ws1.cell(row=row, column=1).alignment = align2\n", " \n", " \n", " # format row 'Total' with dark grey\n", " ws1.cell(row=47, column=col).fill = grey166\n", " ws1.cell(row=47, column=col).font = blackfontbold\n", " \n", " \n", " # format level 1\n", " for row in [9, 22, 23, 46]:\n", " ws1.cell(row=row, column=col).fill = grey191\n", " ws1.cell(row=row, column=col).font = blackfontbold\n", " ws1.cell(row=row, column=1).alignment = align0\n", " \n", " \n", " # format level 2\n", " for row in [10, 11, 12, 13, 18, 19, 20, 21, 24, 31, 35, 39, 40, 41, 45]:\n", " ws1.cell(row=row, column=col).fill = grey217\n", " ws1.cell(row=row, column=1).alignment = align1\n", "\n", " \n", "ws1.cell(row=47, column=1).alignment = align0\n", "ws1.freeze_panes = ws1['B8'] #freeze first column and header rows" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [], "source": [ "# do the same for the second worksheet 'technology levels' \n", "for col in range(1, 7):\n", " colname = get_column_letter(col + 1)\n", " ws2.column_dimensions[colname].width = 25\n", " \n", " for row in range(2, 8):\n", " ws2.cell(row=row, column=col).font = blackfont\n", " \n", " \n", " for row in range(9, 48): \n", " ws2.cell(row=row, column=col).fill = grey242\n", " ws2.cell(row=row, column=1).font = blackfontitalic\n", " ws2.cell(row=row, column=1).alignment = align2\n", " \n", " \n", " # format row 'Total' with dark grey\n", " ws2.cell(row=46, column=col).fill = grey166\n", " \n", " \n", " # format level 1\n", " for row in [8, 21, 22, 45]:\n", " ws2.cell(row=row, column=col).fill = grey191\n", " ws2.cell(row=row, column=col).font = blackfontbold\n", " ws2.cell(row=row, column=1).font = blackfontbold\n", " ws2.cell(row=row, column=1).alignment = align0\n", " \n", " \n", " # format level 2\n", " for row in [9, 10, 11, 12, 17, 18, 19, 20, 23, 30, 34, 38, 39, 40, 44]:\n", " ws2.cell(row=row, column=col).fill = grey217\n", " ws2.cell(row=row, column=1).alignment = align1\n", "\n", " \n", "ws2.cell(row=46, column=1).alignment = align0" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [], "source": [ "additional_notes = openpyxl.load_workbook(os.path.join('input',\n", " 'National_Generation_Capacities.xlsx'))['Additional notes']\n", "# copy additional notes to output file\n", "for col in range(1, 3):\n", " for row in range(1, 10):\n", " add_notes_value = additional_notes.cell(row=row, column=col).value\n", " ws1.cell(row=row + 50, column=col).value = add_notes_value\n", " ws1.cell(row=51, column=1).font = blackfontbold\n", " ws1.cell(row=row + 51, column=1).font = blackfontitalic" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [], "source": [ "outputxls.save(os.path.join('output', 'national_generation_capacity.xlsx'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5.3 Write checksums" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [], "source": [ "files = ['national_generation_capacity.xlsx',\n", " 'national_generation_capacity_stacked.csv',\n", " 'national_generation_capacity_stacked.xlsx',\n", " 'national_generation_capacity.sqlite']\n", "\n", "hash_dict = {}\n", "filesize_dict = {}\n", "\n", "with open('checksums.txt', 'w') as f:\n", " for file_name in files:\n", " path = os.path.join('output', file_name)\n", " file_hash = func.get_sha_hash(path)\n", " hash_dict[file_name] = file_hash\n", " filesize_dict[file_name] = os.path.getsize(path)\n", " f.write('{},{}\\n'.format(file_name, file_hash))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 6. Documentation of the data package" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We document the data packages meta data in the specific format JSON as proposed by the Open Knowledge Foundation. See the Frictionless Data project by OKFN (http://data.okfn.org/) and the Data Package specifications (http://dataprotocols.org/data-packages/) for more details.\n", "\n", "In order to keep the notebook more readable, we first formulate the metadata in the human-readable YAML format using a multi-line string. We then parse the string into a Python dictionary and save that to disk as a JSON file." ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [], "source": [ "with open(os.path.join('input', 'metadata.yml'), 'r') as f:\n", " metadata = yaml.load(f.read(), Loader=yaml.BaseLoader)\n", " \n", "metadata['resources'][0]['hash'] = hash_dict['national_generation_capacity.xlsx']\n", "metadata['resources'][1]['hash'] = hash_dict['national_generation_capacity_stacked.csv']\n", "metadata['resources'][0]['bytes'] = filesize_dict['national_generation_capacity.xlsx']\n", "metadata['resources'][1]['bytes'] = filesize_dict['national_generation_capacity_stacked.csv']" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [], "source": [ "datapackage_json = json.dumps(metadata, indent=4, separators=(',', ': '))\n", " \n", "# Write the information of the metadata\n", "with open(os.path.join('output', 'datapackage.json'), 'w') as f:\n", " f.write(datapackage_json)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "End of script." ] } ], "metadata": { "@webio": { "lastCommId": null, "lastKernelId": null }, "anaconda-cloud": {}, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 1 }