{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<table style=\"width:100%\">\n",
" <tr>\n",
" <td style=\"background-color:#EBF5FB; border: 1px solid #CFCFCF\">\n",
" <b>National generation capacity: Processing notebook</b>\n",
" <ul>\n",
" <li><a href=\"main.ipynb\">Main notebook</a></li>\n",
" <li>Processing notebook (this)</li>\n",
" <li><a href=\"tests.ipynb\">Check notebook</a></li>\n",
" </ul>\n",
" <br>This Notebook is part of the <a href=\"http://data.open-power-system-data.org/national_generation_capacity\">National Generation Capacity Datapackage</a> of <a href=\"http://open-power-system-data.org\">Open Power System Data</a>.\n",
" </td>\n",
" </tr>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Table of Contents\n",
"* [1. Introductory notes](#1.-Introductory-notes)\n",
"* [2. Script setup](#2.-Script-setup)\n",
"* [3. Data download and processing](#3.-Data-download-and-processing)\n",
"\t* [3.1 Manually compiled dataset](#3.1-Manually-compiled-dataset)\n",
"\t* [3.2 EUROSTAT data](#3.2-EUROSTAT-data)\n",
"\t* [3.3 ENTSO-E data](#3.3-ENTSO-E-data)\n",
"\t\t* [3.3.1 ENTSO-E statistical data](#3.3.1-ENTSO-E-statistical-data)\n",
"\t\t* [3.3.2 ENTSO-E SO&AF data](#3.3.2-ENTSO-E-SO&AF-data)\n",
" * [3.3.3 ENTSO-E Transparency Plaftform](#3.3.1-ENTSO-E-Transparency-Platform)\n",
"\t\t* [3.3.4 ENTSO-E Power Statistics](#3.3.4-ENTSO-E-Power-Statistics)\n",
"\t* [3.4 Merge data sources](#3.4-Merge-data-sources)\n",
"* [4. Convert stacked data to crosstable format](#4.-Convert-stacked-data-to-crosstable-format)\n",
"* [5. Output](#5.-Output)\n",
"\t* [5.1 Write results to file](#5.1-Write-results-to-file)\n",
"\t* [5.2 Formatting of Excel tables](#5.2-Formatting-of-Excel-tables)\n",
"\t* [5.3 Write checksums](#5.3-Write-checksums)\n",
"* [6. Documentation of the data package](#6.-Documentation-of-the-data-package)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 1. Introductory notes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The script processes the compiled nationally aggregated generation capacity for European countries. Due to varying formats and data specifications of references for national generation capacity, the script firstly focuses on rearranging the manually compiled data. Thus, the script itself does not collect, select, download or manage data from original sources. Secondly, international data sources, such as EUROSTAT and ENTSO-E, are directly downloaded from original web sources and complement the initial data set."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 2. Script setup"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# some functions and classes that are defined in seperate files\n",
"import functions.helper_functions as func\n",
"import functions.soaf as soaf\n",
"\n",
"# core packages\n",
"import os\n",
"import pandas as pd\n",
"import numpy as np\n",
"\n",
"# packages to copy files, write sqllite data bases and manipulate excel files\n",
"import shutil\n",
"import sqlite3\n",
"import openpyxl\n",
"from openpyxl.styles import PatternFill, colors, Font, Alignment\n",
"from openpyxl.utils import get_column_letter\n",
"\n",
"import yaml\n",
"import json"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 3. Data download and processing"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We compile data from different national and international sources. Firstly, national data sources are manually compiled due to varying data formats and specifications. Secondly, international sources are compiled directly and appended to the compiled data set. The international data sources comprise:\n",
"- [EUROSTAT](http://ec.europa.eu/eurostat/product?code=nrg_113a&mode=view)\n",
"- [ENTSO-E Statistical data](https://www.entsoe.eu/data/data-portal/miscellaneous/Pages/default.aspx)\n",
"- [ENTSO-E System Outlook and Adequacy Forecast](https://www.entsoe.eu/outlooks/maf/Pages/default.aspx)\n",
"- [ENTSO-E Transparency Platform](https://transparency.entsoe.eu/)\n",
"- [ENTSO-E Power Statistics](https://www.entsoe.eu/data/power-stats/)\n",
"\n",
"In the following section, the data sets are downloaded and uploaded to Python."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3.1 Manually compiled dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The manually compiled dataset is imported and rearranged to a DataFrame for further processing. The dataset comprises for each European country and specified generation technology different data entries, which are based on different sources. As these sources differ by country and year, information on the corresponding reference are directly given with the data entry. "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>1</th>\n",
" <th>2</th>\n",
" <th>3</th>\n",
" <th>4</th>\n",
" <th>5</th>\n",
" </tr>\n",
" <tr>\n",
" <th>technology</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>NaN</th>\n",
" <td>Energy source levels and technology</td>\n",
" <td>Energy source levels and technology</td>\n",
" <td>Energy source levels and technology</td>\n",
" <td>Energy source levels and technology</td>\n",
" <td>Energy source levels and technology</td>\n",
" </tr>\n",
" <tr>\n",
" <th>NaN</th>\n",
" <td>Total generation capacity</td>\n",
" <td>Generation capacity by energy type (fossil, nu...</td>\n",
" <td>Generation capacity by fuel or energy source (...</td>\n",
" <td>Generation capacity by fuel or energy source a...</td>\n",
" <td>Generation capacity by fuel and technology (e....</td>\n",
" </tr>\n",
" <tr>\n",
" <th>NaN</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>NaN</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>NaN</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" 1 \\\n",
"technology \n",
"NaN Energy source levels and technology \n",
"NaN Total generation capacity \n",
"NaN NaN \n",
"NaN NaN \n",
"NaN NaN \n",
"\n",
" 2 \\\n",
"technology \n",
"NaN Energy source levels and technology \n",
"NaN Generation capacity by energy type (fossil, nu... \n",
"NaN NaN \n",
"NaN NaN \n",
"NaN NaN \n",
"\n",
" 3 \\\n",
"technology \n",
"NaN Energy source levels and technology \n",
"NaN Generation capacity by fuel or energy source (... \n",
"NaN NaN \n",
"NaN NaN \n",
"NaN NaN \n",
"\n",
" 4 \\\n",
"technology \n",
"NaN Energy source levels and technology \n",
"NaN Generation capacity by fuel or energy source a... \n",
"NaN NaN \n",
"NaN NaN \n",
"NaN NaN \n",
"\n",
" 5 \n",
"technology \n",
"NaN Energy source levels and technology \n",
"NaN Generation capacity by fuel and technology (e.... \n",
"NaN NaN \n",
"NaN NaN \n",
"NaN NaN "
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_file = 'National_Generation_Capacities.xlsx'\n",
"filepath = os.path.join('input', data_file)\n",
"\n",
"# Read data into pandas\n",
"data_raw = pd.read_excel(filepath,\n",
" sheet_name='Summary',\n",
" header=None,\n",
" na_values=['-'],\n",
" skiprows=0)\n",
"\n",
"\n",
"\n",
"\n",
"# Deal with merged cells from Excel: fill first three rows with information\n",
"data_raw.iloc[0:2] = data_raw.iloc[0:2].fillna(method='ffill', axis=1)\n",
"\n",
"# Set index for rows\n",
"data_raw = data_raw.set_index([0])\n",
"data_raw.index.name = 'technology'\n",
"\n",
"# Extract energylevels from raw data for later use\n",
"energylevels_raw = data_raw.iloc[:, 0:5]\n",
"\n",
"energylevels_raw.head()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th>country</th>\n",
" <th colspan=\"6\" halign=\"left\">AT</th>\n",
" <th colspan=\"4\" halign=\"left\">BE</th>\n",
" <th>...</th>\n",
" <th colspan=\"3\" halign=\"left\">SI</th>\n",
" <th colspan=\"7\" halign=\"left\">SK</th>\n",
" </tr>\n",
" <tr>\n",
" <th>type</th>\n",
" <th colspan=\"6\" halign=\"left\">Installed capacity in MW</th>\n",
" <th colspan=\"4\" halign=\"left\">Installed capacity in MW</th>\n",
" <th>...</th>\n",
" <th colspan=\"3\" halign=\"left\">Installed capacity in MW</th>\n",
" <th colspan=\"7\" halign=\"left\">Installed capacity in MW</th>\n",
" </tr>\n",
" <tr>\n",
" <th>year</th>\n",
" <th>2014.0</th>\n",
" <th>2015.0</th>\n",
" <th>2016.0</th>\n",
" <th>2017.0</th>\n",
" <th>2018.0</th>\n",
" <th>2019.0</th>\n",
" <th>2015.0</th>\n",
" <th>2016.0</th>\n",
" <th>2017.0</th>\n",
" <th>2018.0</th>\n",
" <th>...</th>\n",
" <th>2015.0</th>\n",
" <th>2016.0</th>\n",
" <th>2017.0</th>\n",
" <th>2013.0</th>\n",
" <th>2014.0</th>\n",
" <th>2015.0</th>\n",
" <th>2016.0</th>\n",
" <th>2017.0</th>\n",
" <th>2018.0</th>\n",
" <th>NaN</th>\n",
" </tr>\n",
" <tr>\n",
" <th>source</th>\n",
" <th>e-control</th>\n",
" <th>e-control</th>\n",
" <th>e-control</th>\n",
" <th>e-control</th>\n",
" <th>e-control</th>\n",
" <th>e-control</th>\n",
" <th>ELIA</th>\n",
" <th>ELIA</th>\n",
" <th>ELIA</th>\n",
" <th>ELIA</th>\n",
" <th>...</th>\n",
" <th>Agencija za energijo</th>\n",
" <th>Agencija za energijo</th>\n",
" <th>Agencija za energijo</th>\n",
" <th>Statistical Office of Slovakia</th>\n",
" <th>Statistical Office of Slovakia</th>\n",
" <th>Statistical Office of Slovakia</th>\n",
" <th>Statistical Office of Slovakia</th>\n",
" <th>Statistical Office of Slovakia</th>\n",
" <th>Statistical Office of Slovakia</th>\n",
" <th></th>\n",
" </tr>\n",
" <tr>\n",
" <th>source_type</th>\n",
" <th>Regulatory Authority</th>\n",
" <th>Regulatory Authority</th>\n",
" <th>Regulatory Authority</th>\n",
" <th>Regulatory Authority</th>\n",
" <th>Regulatory Authority</th>\n",
" <th>Regulatory Authority</th>\n",
" <th>Transmission System Operator</th>\n",
" <th>Transmission System Operator</th>\n",
" <th>Transmission System Operator</th>\n",
" <th>Transmission System Operator</th>\n",
" <th>...</th>\n",
" <th>Ministry</th>\n",
" <th>Ministry</th>\n",
" <th>Ministry</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>NaN</th>\n",
" </tr>\n",
" <tr>\n",
" <th>weblink</th>\n",
" <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
" <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
" <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
" <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
" <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
" <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-2019_KW2EPLTyp.xlsx/e9993ee3-838f-ed2e-2031-7503418fac2d?t=1596094915390</th>\n",
" <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
" <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
" <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
" <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
" <th>...</th>\n",
" <th>https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2015/f1302ae0-7267-4ae7-b74d-7ce8c4323043</th>\n",
" <th>https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2016/de8cc94f-b3f6-4d32-8e79-ce3b0ec386f5</th>\n",
" <th>https://www.agen-rs.si/documents/54870/68629/a/78f74b68-dbfc-415e-ab88-882652558d94</th>\n",
" <th>Link unavailable</th>\n",
" <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRZJIHHEGAOxQwwYEuvSASM31AE7jWsnb1_Z8aRtGEM7LQcNsP8Hu6t_hQS6R6LJ99WXfFdtmvxRPS-E-TliAR8OdRuAGSMIrtNR5CWWDhTQ3UngeLZP2RSATz0DAttPYysiBGyChArbURQl0ywDL8NjCIjuQZimADE783DhsuHP-Gw6e0vQY3MdgjBM0oxH5GZOT_x7_nziOCpszEdGFpKZS898h6CLt6GHP-f_M2wwUOFE5Z4lkECbbwnE5fLcELrrPwo6-DTo6Z87Njp5h3bzxx_8E-986P-tY7nKf75pU-bqlDOUIYHEstltdw9osSme8wdNNtr2e3EFanms1vmyklewf5a79euvbwCXukGkoVFJLLWsQLO4WWiSyWVBVyujMI3jt7fLqkQLHUtSyJxpK4xXGjUtS-OlBUpdMpaTEuu8_LiX7WER3Vb_wLeL_b0XsZq3YIh1nzoTwjPW4lsC0TfL4iTpmua-Gt4E42SGbYt6zii-VQM7dzBPpiYG0M-CLsP1WV70nQndgrmJQn9TS5TIBi1Uy9jFnkwwuttX8oDSZvOtVsdh8peO8AFdI1EV9eCwrAcw0DFnJsVqMy1M1P3xfK2-Pj0JW5l40-zkyw7d_3cXb-v0_ao5ea0qbR27sX-4cQkVi08_AHhGIRI!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
" <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsVZIgDhijIHYIQYMiXXpCAwNdcBOQ-3k7Ss7nrQNY2in1YEB9v9gd_WvEEf3iDdiX30RbbVtxKN8XnH9c2j4bDzGFoChTcC_TiahG5sYKKC7k8B2LY8acwA2dzXwLS-JzJAQsAjiMmyFYRjP0xTcVJ2CT7ALQZIARMaZhwvLgj_j0_niLUHXWGLwgyBOUhaSmyU98e_5s5lty7C2nGhpQBYOPfM9gj7eggH-nP_PsGaADMcy9zSGGLp8R8Avl-cE0F__UdDDJ_5A_5yp1svbtJ8__uCfePtD_29t05H-83SLGg6mzEAp4ojnTbtrH9Bqmz2LB6VolN337Ark5bHaiLwqrmD_XLSb11_fiLUotVKjSi4IVijOiMKKfK0wprFCFQwyio_f3uXVGq0wlEKYTFd0zRAKZSJXmF6aikl1BkIzszIvP-5ld1h4v9U_8N1if-9FJOfNH6vYo_aMsNTo8B0BH5plfpL0TfNQDW-CabxQLZO69iS6lQO7tFUWz3UVAJ8FfYYbsjwfOhP6BUsdBd62LlBcNGglW2Zc7MlMRXf7qjigpNl-q-VxGP-lIzxA14hXWT065PUIRlhlhk5VuZmmSuT98Xytvj49cUuaeNu0xUuL7v-7i3d18r5qRl6rStlETuQdbhxC-erTD_ffAqA!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
" <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRYkIXHEGAOxQwwYGnTpAMYNdcBOQ-3m7Ss7nrQ1Y2in1YEB9v9gd_WvkEAPSDTZvvqctdW2yZ7kcyr0TwHz-HismgCMTsC7jSeBExkqEEAfTwLLMV3C5gB87lDwTDcOjQBjMDESMmwGQRDNkwScRJuCh1UH_DgGCNmZhyvLhD_jk_niLUGHLVXwfD-KEx7guyU58e_585llyTBdTmji44VNznyPoI83YYA_5_8zTBnIcCRzTyKIoMt3BOJ6ebYP_fUfBT187A30z57SXt4i_fzxB__EWxf9v7cMW_rP1U3CbJVwhhIkkCiadtc-onSbv2SPStkou2_5DcjLU7XJiqq8gf1L2W5ef32TGWrGVsZaWfFMV4hOqMKpvCsxLwzAJIeCHr-9K6oVSiHT8hUjupIX60Ihqk4VyZfKeo1xSTnOoNAu97I7LKLf6hd8t9jfexHKefPGmuoSa4Z5wjp8RyCGZlmcJH3TPFTDm2AaLTTTII41Ce_lwC4tjUdzXQNQz4I-ww1ZXgydCf2CpY58d1uXKCoblMqWsas9mck93VflAcXN9mstj8PoLx3hArpFosrr0aGoRzBSNc50osnNNDQs74_na_Xl-VmY0sTbpi2_t-jhv7t4V8fvq-b4taqUTWiH7uHOxkSkH34AcI_n7A!!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
" <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZRRd5owFMc_yx76iLmEQJJHRASqpQLCal52AOPKLGhXp-u3X7SebpMjbGfLAwe4_x_ce_O_QQI9INHk--pzvqs2Tf6knhfC-hTRgA2Hug1AzREEt-ko8hKuAwH08SRwPNsndArApp4Jge2nMY8MA2wDCRW2oyhKplkGXobHEBi6B2GaAsT0zMOVZcOf8dl09pagR-c6BGGYpBmLjLs5OfHv-bOJ46iwOR-ZWWjMXHLmOwRdvA09_Dn_n2GTggonKvcsgQTafEsgrpfnhtBd_1HQwadBT__csdnJO6SbP_7gn3jnov_3DneV_3zLJtTVCaMoQwKJstltd49osSle8kdNNtr2W3ED6vJUrfOykjewf5G79euvbzDnJTcLoi1zk2gEl1IruGFpRcFXJsF0xTAcv70tqyVa6IwA6HylSWZxjchlqTFpgsYMbEnMTGtF8eVetodFdFv9gm8X-3svYjVvwRDrPnEmBstoi28JRN8si5Oka5r7angTjJMZtjnxnFF8rwZ27mCWTC2sWngWdBmuz_Ki70zoFswtFPqbWqJENmihWkav9mSi9nRfyQNKm83XWh2HyV86wgd0i0RV1INDWQ9goGNGLYLVZnJsqPvj-Vp9eX4WtjLxptnJ7zv08N9dvK3T91Uz47WqtHXsxv7hzjWIWHz4AZRUN9Q!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
" <th>https://slovak.statistics.sk/PortalTraffic/fileServlet?Dokument=da19bcb8-8c9c-4757-aba5-2d0720266efa</th>\n",
" <th></th>\n",
" </tr>\n",
" <tr>\n",
" <th>capacity_definition</th>\n",
" <th>Gross capacity</th>\n",
" <th>Gross capacity</th>\n",
" <th>Gross capacity</th>\n",
" <th>Gross capacity</th>\n",
" <th>Gross capacity</th>\n",
" <th>Gross capacity</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>...</th>\n",
" <th>Net capacity</th>\n",
" <th>Net capacity</th>\n",
" <th>Net capacity</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>NaN</th>\n",
" </tr>\n",
" <tr>\n",
" <th>technology</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Fossil fuels</th>\n",
" <td>7243.63</td>\n",
" <td>7059.07</td>\n",
" <td>7323</td>\n",
" <td>6469</td>\n",
" <td>6492</td>\n",
" <td>6041</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>1442.97</td>\n",
" <td>1450.5</td>\n",
" <td>1223</td>\n",
" <td>3348</td>\n",
" <td>2907</td>\n",
" <td>2586</td>\n",
" <td>3164.51</td>\n",
" <td>3266.97</td>\n",
" <td>2581.47</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>- Lignite</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>- Hard coal</th>\n",
" <td>1171</td>\n",
" <td>1171</td>\n",
" <td>NaN</td>\n",
" <td>1012</td>\n",
" <td>1012</td>\n",
" <td>598</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>- Oil</th>\n",
" <td>333.736</td>\n",
" <td>173.636</td>\n",
" <td>NaN</td>\n",
" <td>168</td>\n",
" <td>168</td>\n",
" <td>164</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>140</td>\n",
" <td>138</td>\n",
" <td>138</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>- Natural gas</th>\n",
" <td>4888.66</td>\n",
" <td>4820.19</td>\n",
" <td>NaN</td>\n",
" <td>4853</td>\n",
" <td>4876</td>\n",
" <td>4429</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>1016</td>\n",
" <td>1022</td>\n",
" <td>959</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 157 columns</p>\n",
"</div>"
],
"text/plain": [
"country AT \\\n",
"type Installed capacity in MW \n",
"year 2014.0 \n",
"source e-control \n",
"source_type Regulatory Authority \n",
"weblink https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538 \n",
"capacity_definition Gross capacity \n",
"technology \n",
"Fossil fuels 7243.63 \n",
"- Lignite 0 \n",
"- Hard coal 1171 \n",
"- Oil 333.736 \n",
"- Natural gas 4888.66 \n",
"\n",
"country \\\n",
"type \n",
"year 2015.0 \n",
"source e-control \n",
"source_type Regulatory Authority \n",
"weblink https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538 \n",
"capacity_definition Gross capacity \n",
"technology \n",
"Fossil fuels 7059.07 \n",
"- Lignite 0 \n",
"- Hard coal 1171 \n",
"- Oil 173.636 \n",
"- Natural gas 4820.19 \n",
"\n",
"country \\\n",
"type \n",
"year 2016.0 \n",
"source e-control \n",
"source_type Regulatory Authority \n",
"weblink https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538 \n",
"capacity_definition Gross capacity \n",
"technology \n",
"Fossil fuels 7323 \n",
"- Lignite NaN \n",
"- Hard coal NaN \n",
"- Oil NaN \n",
"- Natural gas NaN \n",
"\n",
"country \\\n",
"type \n",
"year 2017.0 \n",
"source e-control \n",
"source_type Regulatory Authority \n",
"weblink https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538 \n",
"capacity_definition Gross capacity \n",
"technology \n",
"Fossil fuels 6469 \n",
"- Lignite NaN \n",
"- Hard coal 1012 \n",
"- Oil 168 \n",
"- Natural gas 4853 \n",
"\n",
"country \\\n",
"type \n",
"year 2018.0 \n",
"source e-control \n",
"source_type Regulatory Authority \n",
"weblink https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538 \n",
"capacity_definition Gross capacity \n",
"technology \n",
"Fossil fuels 6492 \n",
"- Lignite NaN \n",
"- Hard coal 1012 \n",
"- Oil 168 \n",
"- Natural gas 4876 \n",
"\n",
"country \\\n",
"type \n",
"year 2019.0 \n",
"source e-control \n",
"source_type Regulatory Authority \n",
"weblink https://www.e-control.at/documents/1785851/1811756/BeStGes-2019_KW2EPLTyp.xlsx/e9993ee3-838f-ed2e-2031-7503418fac2d?t=1596094915390 \n",
"capacity_definition Gross capacity \n",
"technology \n",
"Fossil fuels 6041 \n",
"- Lignite NaN \n",
"- Hard coal 598 \n",
"- Oil 164 \n",
"- Natural gas 4429 \n",
"\n",
"country BE \\\n",
"type Installed capacity in MW \n",
"year 2015.0 \n",
"source ELIA \n",
"source_type Transmission System Operator \n",
"weblink https://www.elia.be/en/grid-data/power-generation/generating-facilities \n",
"capacity_definition Unknown \n",
"technology \n",
"Fossil fuels NaN \n",
"- Lignite NaN \n",
"- Hard coal NaN \n",
"- Oil NaN \n",
"- Natural gas NaN \n",
"\n",
"country \\\n",
"type \n",
"year 2016.0 \n",
"source ELIA \n",
"source_type Transmission System Operator \n",
"weblink https://www.elia.be/en/grid-data/power-generation/generating-facilities \n",
"capacity_definition Unknown \n",
"technology \n",
"Fossil fuels NaN \n",
"- Lignite NaN \n",
"- Hard coal NaN \n",
"- Oil NaN \n",
"- Natural gas NaN \n",
"\n",
"country \\\n",
"type \n",
"year 2017.0 \n",
"source ELIA \n",
"source_type Transmission System Operator \n",
"weblink https://www.elia.be/en/grid-data/power-generation/generating-facilities \n",
"capacity_definition Unknown \n",
"technology \n",
"Fossil fuels NaN \n",
"- Lignite NaN \n",
"- Hard coal NaN \n",
"- Oil NaN \n",
"- Natural gas NaN \n",
"\n",
"country \\\n",
"type \n",
"year 2018.0 \n",
"source ELIA \n",
"source_type Transmission System Operator \n",
"weblink https://www.elia.be/en/grid-data/power-generation/generating-facilities \n",
"capacity_definition Unknown \n",
"technology \n",
"Fossil fuels NaN \n",
"- Lignite NaN \n",
"- Hard coal NaN \n",
"- Oil NaN \n",
"- Natural gas NaN \n",
"\n",
"country ... \\\n",
"type ... \n",
"year ... \n",
"source ... \n",
"source_type ... \n",
"weblink ... \n",
"capacity_definition ... \n",
"technology ... \n",
"Fossil fuels ... \n",
"- Lignite ... \n",
"- Hard coal ... \n",
"- Oil ... \n",
"- Natural gas ... \n",
"\n",
"country SI \\\n",
"type Installed capacity in MW \n",
"year 2015.0 \n",
"source Agencija za energijo \n",
"source_type Ministry \n",
"weblink https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2015/f1302ae0-7267-4ae7-b74d-7ce8c4323043 \n",
"capacity_definition Net capacity \n",
"technology \n",
"Fossil fuels 1442.97 \n",
"- Lignite NaN \n",
"- Hard coal NaN \n",
"- Oil NaN \n",
"- Natural gas NaN \n",
"\n",
"country \\\n",
"type \n",
"year 2016.0 \n",
"source Agencija za energijo \n",
"source_type Ministry \n",
"weblink https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2016/de8cc94f-b3f6-4d32-8e79-ce3b0ec386f5 \n",
"capacity_definition Net capacity \n",
"technology \n",
"Fossil fuels 1450.5 \n",
"- Lignite NaN \n",
"- Hard coal NaN \n",
"- Oil NaN \n",
"- Natural gas NaN \n",
"\n",
"country \\\n",
"type \n",
"year 2017.0 \n",
"source Agencija za energijo \n",
"source_type Ministry \n",
"weblink https://www.agen-rs.si/documents/54870/68629/a/78f74b68-dbfc-415e-ab88-882652558d94 \n",
"capacity_definition Net capacity \n",
"technology \n",
"Fossil fuels 1223 \n",
"- Lignite NaN \n",
"- Hard coal NaN \n",
"- Oil NaN \n",
"- Natural gas NaN \n",
"\n",
"country SK \\\n",
"type Installed capacity in MW \n",
"year 2013.0 \n",
"source Statistical Office of Slovakia \n",
"source_type Statistical Office \n",
"weblink Link unavailable \n",
"capacity_definition Unknown \n",
"technology \n",
"Fossil fuels 3348 \n",
"- Lignite NaN \n",
"- Hard coal NaN \n",
"- Oil 140 \n",
"- Natural gas 1016 \n",
"\n",
"country \\\n",
"type \n",
"year 2014.0 \n",
"source Statistical Office of Slovakia \n",
"source_type Statistical Office \n",
"weblink https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRZJIHHEGAOxQwwYEuvSASM31AE7jWsnb1_Z8aRtGEM7LQcNsP8Hu6t_hQS6R6LJ99WXfFdtmvxRPS-E-TliAR8OdRuAGSMIrtNR5CWWDhTQ3UngeLZP2RSATz0DAttPYysiBGyChArbURQl0ywDL8NjCIjuQZimADE783DhsuHP-Gw6e0vQY3MdgjBM0oxH5GZOT_x7_nziOCpszEdGFpKZS898h6CLt6GHP-f_M2wwUOFE5Z4lkECbbwnE5fLcELrrPwo6-DTo6Z87Njp5h3bzxx_8E-986P-tY7nKf75pU-bqlDOUIYHEstltdw9osSme8wdNNtr2e3EFanms1vmyklewf5a79euvbwCXukGkoVFJLLWsQLO4WWiSyWVBVyujMI3jt7fLqkQLHUtSyJxpK4xXGjUtS-OlBUpdMpaTEuu8_LiX7WER3Vb_wLeL_b0XsZq3YIh1nzoTwjPW4lsC0TfL4iTpmua-Gt4E42SGbYt6zii-VQM7dzBPpiYG0M-CLsP1WV70nQndgrmJQn9TS5TIBi1Uy9jFnkwwuttX8oDSZvOtVsdh8peO8AFdI1EV9eCwrAcw0DFnJsVqMy1M1P3xfK2-Pj0JW5l40-zkyw7d_3cXb-v0_ao5ea0qbR27sX-4cQkVi08_AHhGIRI!/dz/d5/L2dBISEvZ0FBIS9nQSEh/ \n",
"capacity_definition Unknown \n",
"technology \n",
"Fossil fuels 2907 \n",
"- Lignite NaN \n",
"- Hard coal NaN \n",
"- Oil 138 \n",
"- Natural gas 1022 \n",
"\n",
"country \\\n",
"type \n",
"year 2015.0 \n",
"source Statistical Office of Slovakia \n",
"source_type Statistical Office \n",
"weblink https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsVZIgDhijIHYIQYMiXXpCAwNdcBOQ-3k7Ss7nrQNY2in1YEB9v9gd_WvEEf3iDdiX30RbbVtxKN8XnH9c2j4bDzGFoChTcC_TiahG5sYKKC7k8B2LY8acwA2dzXwLS-JzJAQsAjiMmyFYRjP0xTcVJ2CT7ALQZIARMaZhwvLgj_j0_niLUHXWGLwgyBOUhaSmyU98e_5s5lty7C2nGhpQBYOPfM9gj7eggH-nP_PsGaADMcy9zSGGLp8R8Avl-cE0F__UdDDJ_5A_5yp1svbtJ8__uCfePtD_29t05H-83SLGg6mzEAp4ojnTbtrH9Bqmz2LB6VolN337Ark5bHaiLwqrmD_XLSb11_fiLUotVKjSi4IVijOiMKKfK0wprFCFQwyio_f3uXVGq0wlEKYTFd0zRAKZSJXmF6aikl1BkIzszIvP-5ld1h4v9U_8N1if-9FJOfNH6vYo_aMsNTo8B0BH5plfpL0TfNQDW-CabxQLZO69iS6lQO7tFUWz3UVAJ8FfYYbsjwfOhP6BUsdBd62LlBcNGglW2Zc7MlMRXf7qjigpNl-q-VxGP-lIzxA14hXWT065PUIRlhlhk5VuZmmSuT98Xytvj49cUuaeNu0xUuL7v-7i3d18r5qRl6rStlETuQdbhxC-erTD_ffAqA!/dz/d5/L2dBISEvZ0FBIS9nQSEh/ \n",
"capacity_definition Unknown \n",
"technology \n",
"Fossil fuels 2586 \n",
"- Lignite NaN \n",
"- Hard coal NaN \n",
"- Oil 138 \n",
"- Natural gas 959 \n",
"\n",
"country \\\n",
"type \n",
"year 2016.0 \n",
"source Statistical Office of Slovakia \n",
"source_type Statistical Office \n",
"weblink https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRYkIXHEGAOxQwwYGnTpAMYNdcBOQ-3m7Ss7nrQ1Y2in1YEB9v9gd_WvkEAPSDTZvvqctdW2yZ7kcyr0TwHz-HismgCMTsC7jSeBExkqEEAfTwLLMV3C5gB87lDwTDcOjQBjMDESMmwGQRDNkwScRJuCh1UH_DgGCNmZhyvLhD_jk_niLUGHLVXwfD-KEx7guyU58e_585llyTBdTmji44VNznyPoI83YYA_5_8zTBnIcCRzTyKIoMt3BOJ6ebYP_fUfBT187A30z57SXt4i_fzxB__EWxf9v7cMW_rP1U3CbJVwhhIkkCiadtc-onSbv2SPStkou2_5DcjLU7XJiqq8gf1L2W5ef32TGWrGVsZaWfFMV4hOqMKpvCsxLwzAJIeCHr-9K6oVSiHT8hUjupIX60Ihqk4VyZfKeo1xSTnOoNAu97I7LKLf6hd8t9jfexHKefPGmuoSa4Z5wjp8RyCGZlmcJH3TPFTDm2AaLTTTII41Ce_lwC4tjUdzXQNQz4I-ww1ZXgydCf2CpY58d1uXKCoblMqWsas9mck93VflAcXN9mstj8PoLx3hArpFosrr0aGoRzBSNc50osnNNDQs74_na_Xl-VmY0sTbpi2_t-jhv7t4V8fvq-b4taqUTWiH7uHOxkSkH34AcI_n7A!!/dz/d5/L2dBISEvZ0FBIS9nQSEh/ \n",
"capacity_definition Unknown \n",
"technology \n",
"Fossil fuels 3164.51 \n",
"- Lignite NaN \n",
"- Hard coal NaN \n",
"- Oil NaN \n",
"- Natural gas NaN \n",
"\n",
"country \\\n",
"type \n",
"year 2017.0 \n",
"source Statistical Office of Slovakia \n",
"source_type Statistical Office \n",
"weblink https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZRRd5owFMc_yx76iLmEQJJHRASqpQLCal52AOPKLGhXp-u3X7SebpMjbGfLAwe4_x_ce_O_QQI9INHk--pzvqs2Tf6knhfC-hTRgA2Hug1AzREEt-ko8hKuAwH08SRwPNsndArApp4Jge2nMY8MA2wDCRW2oyhKplkGXobHEBi6B2GaAsT0zMOVZcOf8dl09pagR-c6BGGYpBmLjLs5OfHv-bOJ46iwOR-ZWWjMXHLmOwRdvA09_Dn_n2GTggonKvcsgQTafEsgrpfnhtBd_1HQwadBT__csdnJO6SbP_7gn3jnov_3DneV_3zLJtTVCaMoQwKJstltd49osSle8kdNNtr2W3ED6vJUrfOykjewf5G79euvbzDnJTcLoi1zk2gEl1IruGFpRcFXJsF0xTAcv70tqyVa6IwA6HylSWZxjchlqTFpgsYMbEnMTGtF8eVetodFdFv9gm8X-3svYjVvwRDrPnEmBstoi28JRN8si5Oka5r7angTjJMZtjnxnFF8rwZ27mCWTC2sWngWdBmuz_Ki70zoFswtFPqbWqJENmihWkav9mSi9nRfyQNKm83XWh2HyV86wgd0i0RV1INDWQ9goGNGLYLVZnJsqPvj-Vp9eX4WtjLxptnJ7zv08N9dvK3T91Uz47WqtHXsxv7hzjWIWHz4AZRUN9Q!/dz/d5/L2dBISEvZ0FBIS9nQSEh/ \n",
"capacity_definition Unknown \n",
"technology \n",
"Fossil fuels 3266.97 \n",
"- Lignite NaN \n",
"- Hard coal NaN \n",
"- Oil NaN \n",
"- Natural gas NaN \n",
"\n",
"country \\\n",
"type \n",
"year 2018.0 \n",
"source Statistical Office of Slovakia \n",
"source_type Statistical Office \n",
"weblink https://slovak.statistics.sk/PortalTraffic/fileServlet?Dokument=da19bcb8-8c9c-4757-aba5-2d0720266efa \n",
"capacity_definition Unknown \n",
"technology \n",
"Fossil fuels 2581.47 \n",
"- Lignite NaN \n",
"- Hard coal NaN \n",
"- Oil NaN \n",
"- Natural gas NaN \n",
"\n",
"country \n",
"type \n",
"year NaN \n",
"source \n",
"source_type NaN \n",
"weblink \n",
"capacity_definition NaN \n",
"technology \n",
"Fossil fuels NaN \n",
"- Lignite NaN \n",
"- Hard coal NaN \n",
"- Oil NaN \n",
"- Natural gas NaN \n",
"\n",
"[5 rows x 157 columns]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Delete definition of energy levels from raw data\n",
"data_raw.drop(data_raw.columns[[0, 1, 2, 3, 4, 5]], axis=1, inplace=True)\n",
"\n",
"level_names = ['country', 'type', 'year', 'source',\n",
" 'source_type', 'weblink', 'capacity_definition']\n",
"# Set multiindex column names\n",
"data_raw.columns = pd.MultiIndex.from_arrays(data_raw[:7].values,\n",
" names=level_names)\n",
"\n",
"# Remove 3 rows which are already used as column names\n",
"data_raw = data_raw[pd.notnull(data_raw.index)]\n",
"\n",
"# Extract the ordering of technologies\n",
"technology_order = data_raw.index.str.replace('- ', '').values.tolist()\n",
"\n",
"data_raw.head()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"MultiIndex([], names=['country', 'type', 'year', 'source', 'source_type', 'weblink', 'capacity_definition'])"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_raw.columns[data_raw.columns.duplicated()]"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th>country</th>\n",
" <th colspan=\"6\" halign=\"left\">AT</th>\n",
" <th colspan=\"4\" halign=\"left\">BE</th>\n",
" <th>...</th>\n",
" <th colspan=\"3\" halign=\"left\">SI</th>\n",
" <th colspan=\"7\" halign=\"left\">SK</th>\n",
" </tr>\n",
" <tr>\n",
" <th>type</th>\n",
" <th colspan=\"6\" halign=\"left\">Installed capacity in MW</th>\n",
" <th colspan=\"4\" halign=\"left\">Installed capacity in MW</th>\n",
" <th>...</th>\n",
" <th colspan=\"3\" halign=\"left\">Installed capacity in MW</th>\n",
" <th colspan=\"7\" halign=\"left\">Installed capacity in MW</th>\n",
" </tr>\n",
" <tr>\n",
" <th>year</th>\n",
" <th>2014.0</th>\n",
" <th>2015.0</th>\n",
" <th>2016.0</th>\n",
" <th>2017.0</th>\n",
" <th>2018.0</th>\n",
" <th>2019.0</th>\n",
" <th>2015.0</th>\n",
" <th>2016.0</th>\n",
" <th>2017.0</th>\n",
" <th>2018.0</th>\n",
" <th>...</th>\n",
" <th>2015.0</th>\n",
" <th>2016.0</th>\n",
" <th>2017.0</th>\n",
" <th>2013.0</th>\n",
" <th>2014.0</th>\n",
" <th>2015.0</th>\n",
" <th>2016.0</th>\n",
" <th>2017.0</th>\n",
" <th>2018.0</th>\n",
" <th>NaN</th>\n",
" </tr>\n",
" <tr>\n",
" <th>source</th>\n",
" <th>e-control</th>\n",
" <th>e-control</th>\n",
" <th>e-control</th>\n",
" <th>e-control</th>\n",
" <th>e-control</th>\n",
" <th>e-control</th>\n",
" <th>ELIA</th>\n",
" <th>ELIA</th>\n",
" <th>ELIA</th>\n",
" <th>ELIA</th>\n",
" <th>...</th>\n",
" <th>Agencija za energijo</th>\n",
" <th>Agencija za energijo</th>\n",
" <th>Agencija za energijo</th>\n",
" <th>Statistical Office of Slovakia</th>\n",
" <th>Statistical Office of Slovakia</th>\n",
" <th>Statistical Office of Slovakia</th>\n",
" <th>Statistical Office of Slovakia</th>\n",
" <th>Statistical Office of Slovakia</th>\n",
" <th>Statistical Office of Slovakia</th>\n",
" <th></th>\n",
" </tr>\n",
" <tr>\n",
" <th>source_type</th>\n",
" <th>Regulatory Authority</th>\n",
" <th>Regulatory Authority</th>\n",
" <th>Regulatory Authority</th>\n",
" <th>Regulatory Authority</th>\n",
" <th>Regulatory Authority</th>\n",
" <th>Regulatory Authority</th>\n",
" <th>Transmission System Operator</th>\n",
" <th>Transmission System Operator</th>\n",
" <th>Transmission System Operator</th>\n",
" <th>Transmission System Operator</th>\n",
" <th>...</th>\n",
" <th>Ministry</th>\n",
" <th>Ministry</th>\n",
" <th>Ministry</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>NaN</th>\n",
" </tr>\n",
" <tr>\n",
" <th>weblink</th>\n",
" <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
" <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
" <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
" <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
" <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538</th>\n",
" <th>https://www.e-control.at/documents/1785851/1811756/BeStGes-2019_KW2EPLTyp.xlsx/e9993ee3-838f-ed2e-2031-7503418fac2d?t=1596094915390</th>\n",
" <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
" <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
" <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
" <th>https://www.elia.be/en/grid-data/power-generation/generating-facilities</th>\n",
" <th>...</th>\n",
" <th>https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2015/f1302ae0-7267-4ae7-b74d-7ce8c4323043</th>\n",
" <th>https://www.agen-rs.si/documents/54870/68629/Report-on-the-energy-sector-in-Slovenia-for-2016/de8cc94f-b3f6-4d32-8e79-ce3b0ec386f5</th>\n",
" <th>https://www.agen-rs.si/documents/54870/68629/a/78f74b68-dbfc-415e-ab88-882652558d94</th>\n",
" <th>Link unavailable</th>\n",
" <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRZJIHHEGAOxQwwYEuvSASM31AE7jWsnb1_Z8aRtGEM7LQcNsP8Hu6t_hQS6R6LJ99WXfFdtmvxRPS-E-TliAR8OdRuAGSMIrtNR5CWWDhTQ3UngeLZP2RSATz0DAttPYysiBGyChArbURQl0ywDL8NjCIjuQZimADE783DhsuHP-Gw6e0vQY3MdgjBM0oxH5GZOT_x7_nziOCpszEdGFpKZS898h6CLt6GHP-f_M2wwUOFE5Z4lkECbbwnE5fLcELrrPwo6-DTo6Z87Njp5h3bzxx_8E-986P-tY7nKf75pU-bqlDOUIYHEstltdw9osSme8wdNNtr2e3EFanms1vmyklewf5a79euvbwCXukGkoVFJLLWsQLO4WWiSyWVBVyujMI3jt7fLqkQLHUtSyJxpK4xXGjUtS-OlBUpdMpaTEuu8_LiX7WER3Vb_wLeL_b0XsZq3YIh1nzoTwjPW4lsC0TfL4iTpmua-Gt4E42SGbYt6zii-VQM7dzBPpiYG0M-CLsP1WV70nQndgrmJQn9TS5TIBi1Uy9jFnkwwuttX8oDSZvOtVsdh8peO8AFdI1EV9eCwrAcw0DFnJsVqMy1M1P3xfK2-Pj0JW5l40-zkyw7d_3cXb-v0_ao5ea0qbR27sX-4cQkVi08_AHhGIRI!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
" <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsVZIgDhijIHYIQYMiXXpCAwNdcBOQ-3k7Ss7nrQNY2in1YEB9v9gd_WvEEf3iDdiX30RbbVtxKN8XnH9c2j4bDzGFoChTcC_TiahG5sYKKC7k8B2LY8acwA2dzXwLS-JzJAQsAjiMmyFYRjP0xTcVJ2CT7ALQZIARMaZhwvLgj_j0_niLUHXWGLwgyBOUhaSmyU98e_5s5lty7C2nGhpQBYOPfM9gj7eggH-nP_PsGaADMcy9zSGGLp8R8Avl-cE0F__UdDDJ_5A_5yp1svbtJ8__uCfePtD_29t05H-83SLGg6mzEAp4ojnTbtrH9Bqmz2LB6VolN337Ark5bHaiLwqrmD_XLSb11_fiLUotVKjSi4IVijOiMKKfK0wprFCFQwyio_f3uXVGq0wlEKYTFd0zRAKZSJXmF6aikl1BkIzszIvP-5ld1h4v9U_8N1if-9FJOfNH6vYo_aMsNTo8B0BH5plfpL0TfNQDW-CabxQLZO69iS6lQO7tFUWz3UVAJ8FfYYbsjwfOhP6BUsdBd62LlBcNGglW2Zc7MlMRXf7qjigpNl-q-VxGP-lIzxA14hXWT065PUIRlhlhk5VuZmmSuT98Xytvj49cUuaeNu0xUuL7v-7i3d18r5qRl6rStlETuQdbhxC-erTD_ffAqA!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
" <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZTBcpswEIafpYccsRYkIXHEGAOxQwwYGnTpAMYNdcBOQ-3m7Ss7nrQ1Y2in1YEB9v9gd_WvkEAPSDTZvvqctdW2yZ7kcyr0TwHz-HismgCMTsC7jSeBExkqEEAfTwLLMV3C5gB87lDwTDcOjQBjMDESMmwGQRDNkwScRJuCh1UH_DgGCNmZhyvLhD_jk_niLUGHLVXwfD-KEx7guyU58e_585llyTBdTmji44VNznyPoI83YYA_5_8zTBnIcCRzTyKIoMt3BOJ6ebYP_fUfBT187A30z57SXt4i_fzxB__EWxf9v7cMW_rP1U3CbJVwhhIkkCiadtc-onSbv2SPStkou2_5DcjLU7XJiqq8gf1L2W5ef32TGWrGVsZaWfFMV4hOqMKpvCsxLwzAJIeCHr-9K6oVSiHT8hUjupIX60Ihqk4VyZfKeo1xSTnOoNAu97I7LKLf6hd8t9jfexHKefPGmuoSa4Z5wjp8RyCGZlmcJH3TPFTDm2AaLTTTII41Ce_lwC4tjUdzXQNQz4I-ww1ZXgydCf2CpY58d1uXKCoblMqWsas9mck93VflAcXN9mstj8PoLx3hArpFosrr0aGoRzBSNc50osnNNDQs74_na_Xl-VmY0sTbpi2_t-jhv7t4V8fvq-b4taqUTWiH7uHOxkSkH34AcI_n7A!!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
" <th>https://slovak.statistics.sk/wps/portal/!ut/p/z1/rZRRd5owFMc_yx76iLmEQJJHRASqpQLCal52AOPKLGhXp-u3X7SebpMjbGfLAwe4_x_ce_O_QQI9INHk--pzvqs2Tf6knhfC-hTRgA2Hug1AzREEt-ko8hKuAwH08SRwPNsndArApp4Jge2nMY8MA2wDCRW2oyhKplkGXobHEBi6B2GaAsT0zMOVZcOf8dl09pagR-c6BGGYpBmLjLs5OfHv-bOJ46iwOR-ZWWjMXHLmOwRdvA09_Dn_n2GTggonKvcsgQTafEsgrpfnhtBd_1HQwadBT__csdnJO6SbP_7gn3jnov_3DneV_3zLJtTVCaMoQwKJstltd49osSle8kdNNtr2W3ED6vJUrfOykjewf5G79euvbzDnJTcLoi1zk2gEl1IruGFpRcFXJsF0xTAcv70tqyVa6IwA6HylSWZxjchlqTFpgsYMbEnMTGtF8eVetodFdFv9gm8X-3svYjVvwRDrPnEmBstoi28JRN8si5Oka5r7angTjJMZtjnxnFF8rwZ27mCWTC2sWngWdBmuz_Ki70zoFswtFPqbWqJENmihWkav9mSi9nRfyQNKm83XWh2HyV86wgd0i0RV1INDWQ9goGNGLYLVZnJsqPvj-Vp9eX4WtjLxptnJ7zv08N9dvK3T91Uz47WqtHXsxv7hzjWIWHz4AZRUN9Q!/dz/d5/L2dBISEvZ0FBIS9nQSEh/</th>\n",
" <th>https://slovak.statistics.sk/PortalTraffic/fileServlet?Dokument=da19bcb8-8c9c-4757-aba5-2d0720266efa</th>\n",
" <th></th>\n",
" </tr>\n",
" <tr>\n",
" <th>capacity_definition</th>\n",
" <th>Gross capacity</th>\n",
" <th>Gross capacity</th>\n",
" <th>Gross capacity</th>\n",
" <th>Gross capacity</th>\n",
" <th>Gross capacity</th>\n",
" <th>Gross capacity</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>...</th>\n",
" <th>Net capacity</th>\n",
" <th>Net capacity</th>\n",
" <th>Net capacity</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>NaN</th>\n",
" </tr>\n",
" <tr>\n",
" <th>technology</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" </tbody>\n",
"</table>\n",
"<p>0 rows × 157 columns</p>\n",
"</div>"
],
"text/plain": [
"Empty DataFrame\n",
"Columns: [(AT, Installed capacity in MW, 2014.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538, Gross capacity), (AT, Installed capacity in MW, 2015.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538, Gross capacity), (AT, Installed capacity in MW, 2016.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538, Gross capacity), (AT, Installed capacity in MW, 2017.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538, Gross capacity), (AT, Installed capacity in MW, 2018.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-JR_KWEPL.xlsx/d4357de2-9d0e-a3ae-4349-17d10306677e?t=1568280083538, Gross capacity), (AT, Installed capacity in MW, 2019.0, e-control, Regulatory Authority, https://www.e-control.at/documents/1785851/1811756/BeStGes-2019_KW2EPLTyp.xlsx/e9993ee3-838f-ed2e-2031-7503418fac2d?t=1596094915390, Gross capacity), (BE, Installed capacity in MW, 2015.0, ELIA, Transmission System Operator, https://www.elia.be/en/grid-data/power-generation/generating-facilities, Unknown), (BE, Installed capacity in MW, 2016.0, ELIA, Transmission System Operator, https://www.elia.be/en/grid-data/power-generation/generating-facilities, Unknown), (BE, Installed capacity in MW, 2017.0, ELIA, Transmission System Operator, https://www.elia.be/en/grid-data/power-generation/generating-facilities, Unknown), (BE, Installed capacity in MW, 2018.0, ELIA, Transmission System Operator, https://www.elia.be/en/grid-data/power-generation/generating-facilities, Unknown), (BE, Installed capacity in MW, 2019.0, ELIA, Transmission System Operator, https://www.elia.be/en/grid-data/power-generation/generating-facilities, Unknown), (BG, Installed capacity in MW, 2012.0, TSO Bulgaria, Transmission System Operator, http://www.eso.bg/fileObj.php?oid=99, Unknown), (BG, Installed capacity in MW, 2019.0, TSO Bulgaria, Transmission System Operator, http://www.eso.bg/?did=380, Unknown), (CH, Installed capacity in MW, 2012.0, UN Statistical Office, Statistical Office, http://data.un.org/Data.aspx?d=EDATA&f=cmID%3AEC, Net capacity), (CH, Installed capacity in MW, 2012.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvNzA3Ni5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2014.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvNzkyNS5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2015.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvODM0My5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2016.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvODY5My5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2017.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvOTM5Ni5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2018.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik/_jcr_content/par/tabs/items/tab/tabpar/externalcontent.external.exturl.pdf/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvOTc0OC5wZGY=.pdf, Unknown), (CH, Installed capacity in MW, 2019.0, BFE, Regulatory Authority, https://www.bfe.admin.ch/bfe/de/home/versorgung/statistik-und-geodaten/energiestatistiken/elektrizitaetsstatistik.exturl.html/aHR0cHM6Ly9wdWJkYi5iZmUuYWRtaW4uY2gvZGUvcHVibGljYX/Rpb24vZG93bmxvYWQvMTAxMTI=.html, Unknown), (CZ, Installed capacity in MW, 2013.0, ERU, Regulatory Authority, https://www.eru.cz/documents/10540/462820/Annual_report_electricity_2013.pdf/34a35d27-9c58-4c79-99d1-f0fbc5eac06a, Unknown), (CZ, Installed capacity in MW, 2014.0, ERU, Regulatory Authority, https://www.eru.cz/documents/10540/462820/Annual_report_electricity_2014.pdf/f23d80b5-668a-42c0-9d04-1a19556c9c58, Unknown), (CZ, Installed capacity in MW, 2015.0, ERU, Regulatory Authority, https://www.eru.cz/documents/10540/462820/Annual_report_electricity_2015.pdf/2548f33b-7964-4537-8b97-3943c7d83854, Unknown), (CZ, Installed capacity in MW, 2016.0, ERU, Regulatory Authority, https://www.eru.cz/documents/10540/462820/Annual_report_electricity_2016.pdf/bfbc88b7-a814-498f-99bf-98b50148c805, Unknown), (CZ, Installed capacity in MW, 2017.0, ERU, Regulatory Authority, https://www.eru.cz/documents/10540/4580209/Yearly_report_electricity_2017.pdf/a4042a98-8750-4524-bb3c-f172ecba2f3b, Unknown), (CZ, Installed capacity in MW, 2018.0, ERU, Regulatory Authority, http://www.eru.cz/documents/10540/4580207/Yearly_report_electricity_2018.pdf/f25a55d8-6730-4521-8e40-96d8e5f00c70, Unknown), (DE, Installed capacity in MW, 2013.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2014.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2015.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2016.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2017.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2018.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DE, Installed capacity in MW, 2019.0, BMWi, Ministry, https://www.bmwi.de/Redaktion/DE/Binaer/Energiedaten/energiedaten-gesamt-xls.xlsx?__blob=publicationFile&v=117, Gross capacity), (DK, Installed capacity in MW, 2013.0, DEA, Ministry, Link unavailable, Unknown), (DK, Installed capacity in MW, 2014.0, DEA, Ministry, https://ens.dk/sites/ens.dk/files/Statistik/energystatistics2014.pdf, Unknown), (DK, Installed capacity in MW, 2015.0, DEA, Ministry, https://ens.dk/sites/ens.dk/files/Statistik/energy_statistics_2015.pdf, Unknown), (DK, Installed capacity in MW, 2016.0, DEA, Ministry, https://ens.dk/sites/ens.dk/files/Statistik/energy_statistics_2016.pdf, Unknown), (DK, Installed capacity in MW, 2017.0, DEA, Ministry, https://ens.dk/sites/ens.dk/files/Statistik/energystatistics2017.pdf, Unknown), (DK, Installed capacity in MW, 2018.0, DEA, Ministry, https://ens.dk/sites/ens.dk/files/Statistik/energy_statistics_2018.pdf, Unknown), (EE, Installed capacity in MW, 2013.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2014.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2015.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2016.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2017.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2018.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (EE, Installed capacity in MW, 2019.0, Statistics Estonia, Statistical Office, http://pub.stat.ee/px-web.2001/Dialog/varval.asp?ma=FE032&ti=CAPACITY+AND+PRODUCTION+OF+POWER+PLANTS&path=../I_Databas/Economy/07Energy/02Energy_consumption_and_production/01Annual_statistics/&lang=1, Gross capacity), (ES, Installed capacity in MW, 2013.0, REE, Transmission System Operator, Link unavailable, Unknown), (ES, Installed capacity in MW, 2014.0, REE, Transmission System Operator, nan, Unknown), (ES, Installed capacity in MW, 2015.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/septiembre_2019/6_Potencia_instalada_09_2019.xlsm, Unknown), (ES, Installed capacity in MW, 2016.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/septiembre_2019/6_Potencia_instalada_09_2019.xlsm, Unknown), (ES, Installed capacity in MW, 2017.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/septiembre_2019/6_Potencia_instalada_09_2019.xlsm, Unknown), (ES, Installed capacity in MW, 2018.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/septiembre_2019/6_Potencia_instalada_09_2019.xlsm, Unknown), (ES, Installed capacity in MW, 2019.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/septiembre_2019/6_Potencia_instalada_09_2019.xlsm, Unknown), (ES, Installed capacity in MW, 2020.0, REE, Transmission System Operator, https://www.ree.es/sites/default/files/11_PUBLICACIONES/Documentos/SeriesEstadisticas/agosto_2020/6_Potencia_instalada_08_2020.xlsm, Unknown), (FR, Installed capacity in MW, 2014.0, RTE, Transmission System Operator, https://www.rte-france.com/sites/default/files/bilan_electrique_2014_en.pdf, Unknown), (FR, Installed capacity in MW, 2015.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FR, Installed capacity in MW, 2016.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FR, Installed capacity in MW, 2017.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FR, Installed capacity in MW, 2018.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FR, Installed capacity in MW, 2019.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FR, Installed capacity in MW, 2020.0, RTE, Transmission System Operator, https://www.services-rte.com/en/view-data-published-by-rte/production-installed-capacity.html, Unknown), (FI, Installed capacity in MW, 2012.0, Statistics Finland, Statistical Office, http://pxnet2.stat.fi/PXWeb/pxweb/en/StatFin_Passiivi/StatFin_Passiivi__ene__ehk/statfinpas_ehk_pxt_904_201500_en.px/, Unknown), (FI, Installed capacity in MW, 2013.0, Statistics Finland, Statistical Office, http://pxnet2.stat.fi/PXWeb/pxweb/en/StatFin_Passiivi/StatFin_Passiivi__ene__ehk/statfinpas_ehk_pxt_904_201500_en.px/, Unknown), (FI, Installed capacity in MW, 2014.0, Statistics Finland, Statistical Office, http://pxnet2.stat.fi/PXWeb/pxweb/en/StatFin_Passiivi/StatFin_Passiivi__ene__ehk/statfinpas_ehk_pxt_904_201500_en.px/, Unknown), (FI, Installed capacity in MW, 2015.0, Statistics Finland, Statistical Office, http://pxnet2.stat.fi/PXWeb/pxweb/en/StatFin_Passiivi/StatFin_Passiivi__ene__ehk/statfinpas_ehk_pxt_904_201500_en.px/, Unknown), (GB, Installed capacity in MW, 2015.0, Department of Energy & Climate Change, Ministry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/826556/DUKES_5.7.xls, Unknown), (GB, Installed capacity in MW, 2016.0, Department of Energy & Climate Change, Ministry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/826556/DUKES_5.7.xls, Unknown), (GB, Installed capacity in MW, 2017.0, Department of Energy & Climate Change, Ministry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/826556/DUKES_5.7.xls, Unknown), (GB, Installed capacity in MW, 2018.0, Department of Energy & Climate Change, Ministry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/826556/DUKES_5.7.xls, Unknown), (GB, Installed capacity in MW, 2019.0, Department of Energy & Climate Change, Ministry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/904812/DUKES_5.7.xls, Unknown), (GR, Installed capacity in MW, 2012.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2015, Unknown), (GR, Installed capacity in MW, 2013.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2016, Unknown), (GR, Installed capacity in MW, 2014.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2017, Unknown), (GR, Installed capacity in MW, 2015.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2018, Unknown), (GR, Installed capacity in MW, 2016.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2018, Unknown), (GR, Installed capacity in MW, 2017.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2018, Unknown), (GR, Installed capacity in MW, 2018.0, Regulatory Authority for Energy, Regulatory Authority, http://www.rae.gr/site/file/system/docs/ActionReports/national_2019, Unknown), (HR, Installed capacity in MW, 2013.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/qyxRItZNXQL84ygVm4NbnE/publications/Temeljni_podaci_2013.pdf, Unknown), (HR, Installed capacity in MW, 2014.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/vHpFOrmCvaEk2wfzW8lBZ6/publications/Temeljni_podaci_za_2014.pdf, Unknown), (HR, Installed capacity in MW, 2015.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/PsIM6mazLoTDVjKzm4NbnE/publications/Temeljni_podaci_2015.pdf, Unknown), (HR, Installed capacity in MW, 2016.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/wEfz2PYHVQLQIygVm4NbnE/publications/Temeljni_podaci_2016.pdf, Unknown), (HR, Installed capacity in MW, 2017.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/KzqT4wOZhIFGvbKzm4t0IF/annual-report-hops/HOPS_GI_2017_-__za_web.pdf, Unknown), (HR, Installed capacity in MW, 2018.0, Croation Transmission System operator (HOPS), Transmission System Operator, https://www.hops.hr/page-file/x2VUqipcFKEnEgg50EV391/annual-report-hops/GI%202018%20unutarnje%20-%209-7-19%20web.pdf, Unknown), (HU, Installed capacity in MW, 2013.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/160362/Planned+Installed+generation+capacity+for+2010_2011_2012_2013+++4.3.1.pdf/84e86e8b-3b7c-4377-8020-c4beec92c914, Unknown), (HU, Installed capacity in MW, 2014.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/188569160/BT_terv_2014/9946a7a2-38ec-4794-9d7f-96a7a927d1b9, Unknown), (HU, Installed capacity in MW, 2015.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/188569160/BT_terv_2014/9946a7a2-38ec-4794-9d7f-96a7a927d1b9, Unknown), (HU, Installed capacity in MW, 2016.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/188569160/BT_terv_2014/9946a7a2-38ec-4794-9d7f-96a7a927d1b9, Unknown), (HU, Installed capacity in MW, 2017.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/188569160/BT_terv_2014/9946a7a2-38ec-4794-9d7f-96a7a927d1b9, Unknown), (HU, Installed capacity in MW, 2018.0, Mavir, Transmission System Operator, https://www.mavir.hu/documents/10262/230112823/VER+BT_20181231.pdf/20c127e3-bb04-8b84-925f-2548761a41d4, Unknown), (IE, Installed capacity in MW, 2014.0, Eirgrid, Transmission System Operator, http://www.eirgridgroup.com/site-files/library/EirGrid/Eirgrid_Generation_Capacity_Statement_2015.-2024.pdf, Unknown), (IE, Installed capacity in MW, 2015.0, Eirgrid, Transmission System Operator, http://www.eirgridgroup.com/site-files/library/EirGrid/Eirgrid_Generation_Capacity_Statement_2015.-2024.pdf, Unknown), (IE, Installed capacity in MW, 2019.0, Eirgrid, Transmission System Operator, http://www.eirgridgroup.com/site-files/library/EirGrid/EirGrid-Group-All-Island-Generation-Capacity-Statement-2019-2028.pdf, Unknown), (IE, Installed capacity in MW, 2020.0, Eirgrid, Transmission System Operator, http://www.eirgridgroup.com/site-files/library/EirGrid/All-Island-Generation-Capacity-Statement-2020-2029.pdf, Unknown), (IT, Installed capacity in MW, 2014.0, Terna, Transmission System Operator, https://download.terna.it/terna/0000/0775/97.PDF, Gross capacity), (IT, Installed capacity in MW, 2015.0, Terna, Transmission System Operator, https://download.terna.it/terna/0000/0995/02.PDF, Gross capacity), (IT, Installed capacity in MW, 2016.0, Terna, Transmission System Operator, https://download.terna.it/terna/0000/1112/78.PDF, Gross capacity), (IT, Installed capacity in MW, 2017.0, Terna, Transmission System Operator, https://download.terna.it/terna/2017_Eng_3-POWER_PLANTS_8d73131c70720ed.pdf, Gross capacity), (IT, Installed capacity in MW, 2018.0, Terna, Transmission System Operator, https://www.terna.it/en/electric-system/transparency-report/installed-capacity, Gross capacity), (IT, Installed capacity in MW, 2019.0, Terna, Transmission System Operator, https://www.terna.it/en/electric-system/transparency-report/installed-capacity, Gross capacity), ...]\n",
"Index: []\n",
"\n",
"[0 rows x 157 columns]"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_raw[data_raw.index.duplicated()]"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>technology</th>\n",
" <th>country</th>\n",
" <th>type</th>\n",
" <th>year</th>\n",
" <th>source</th>\n",
" <th>source_type</th>\n",
" <th>weblink</th>\n",
" <th>capacity_definition</th>\n",
" <th>capacity</th>\n",
" <th>comment</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Fossil fuels</td>\n",
" <td>AT</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>2014</td>\n",
" <td>e-control</td>\n",
" <td>Regulatory Authority</td>\n",
" <td>https://www.e-control.at/documents/1785851/181...</td>\n",
" <td>Gross capacity</td>\n",
" <td>7243.634</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Fossil fuels</td>\n",
" <td>AT</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>2015</td>\n",
" <td>e-control</td>\n",
" <td>Regulatory Authority</td>\n",
" <td>https://www.e-control.at/documents/1785851/181...</td>\n",
" <td>Gross capacity</td>\n",
" <td>7059.065</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Fossil fuels</td>\n",
" <td>AT</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>2016</td>\n",
" <td>e-control</td>\n",
" <td>Regulatory Authority</td>\n",
" <td>https://www.e-control.at/documents/1785851/181...</td>\n",
" <td>Gross capacity</td>\n",
" <td>7323.000</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Fossil fuels</td>\n",
" <td>AT</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>2017</td>\n",
" <td>e-control</td>\n",
" <td>Regulatory Authority</td>\n",
" <td>https://www.e-control.at/documents/1785851/181...</td>\n",
" <td>Gross capacity</td>\n",
" <td>6469.000</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Fossil fuels</td>\n",
" <td>AT</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>2018</td>\n",
" <td>e-control</td>\n",
" <td>Regulatory Authority</td>\n",
" <td>https://www.e-control.at/documents/1785851/181...</td>\n",
" <td>Gross capacity</td>\n",
" <td>6492.000</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" technology country type year source \\\n",
"0 Fossil fuels AT Installed capacity in MW 2014 e-control \n",
"1 Fossil fuels AT Installed capacity in MW 2015 e-control \n",
"2 Fossil fuels AT Installed capacity in MW 2016 e-control \n",
"3 Fossil fuels AT Installed capacity in MW 2017 e-control \n",
"4 Fossil fuels AT Installed capacity in MW 2018 e-control \n",
"\n",
" source_type weblink \\\n",
"0 Regulatory Authority https://www.e-control.at/documents/1785851/181... \n",
"1 Regulatory Authority https://www.e-control.at/documents/1785851/181... \n",
"2 Regulatory Authority https://www.e-control.at/documents/1785851/181... \n",
"3 Regulatory Authority https://www.e-control.at/documents/1785851/181... \n",
"4 Regulatory Authority https://www.e-control.at/documents/1785851/181... \n",
"\n",
" capacity_definition capacity comment \n",
"0 Gross capacity 7243.634 NaN \n",
"1 Gross capacity 7059.065 NaN \n",
"2 Gross capacity 7323.000 NaN \n",
"3 Gross capacity 6469.000 NaN \n",
"4 Gross capacity 6492.000 NaN "
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Reshape dataframe to list\n",
"data_opsd = pd.DataFrame(data_raw.stack(level=level_names))\n",
"\n",
"# Reset index for dataframe\n",
"data_opsd.reset_index(inplace=True)\n",
"data_opsd['technology'] = data_opsd['technology'].str.replace('- ', '')\n",
"\n",
"data_opsd.rename(columns={0: 'capacity'}, inplace=True)\n",
"data_opsd['capacity'] = pd.to_numeric(data_opsd['capacity'], errors='coerce')\n",
"\n",
"# For some source, permission to publish data\n",
"banlist = ['ELIA', 'BMWi', 'Mavir']\n",
"davail = 'data available, but cannot be provided'\n",
"data_opsd.loc[data_opsd['source'].isin(banlist), 'comment'] = davail\n",
"\n",
"data_opsd.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3.2 EUROSTAT data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"EUROSTAT publishes annual structural data on national electricity generation capacities for European countries. The dataset is available in the EUROSTAT database within the category 'Environment and Energy' ([nrg_113a](http://ec.europa.eu/eurostat/product?code=nrg_113a&mode=view))."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>unit</th>\n",
" <th>product</th>\n",
" <th>indic_nrg</th>\n",
" <th>geo\\time</th>\n",
" <th>2016</th>\n",
" <th>2015</th>\n",
" <th>2014</th>\n",
" <th>2013</th>\n",
" <th>2012</th>\n",
" <th>2011</th>\n",
" <th>...</th>\n",
" <th>1999</th>\n",
" <th>1998</th>\n",
" <th>1997</th>\n",
" <th>1996</th>\n",
" <th>1995</th>\n",
" <th>1994</th>\n",
" <th>1993</th>\n",
" <th>1992</th>\n",
" <th>1991</th>\n",
" <th>1990</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>MW</td>\n",
" <td>9007</td>\n",
" <td>12_1176011</td>\n",
" <td>AL</td>\n",
" <td>98</td>\n",
" <td>98</td>\n",
" <td>98</td>\n",
" <td>98</td>\n",
" <td>97</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>MW</td>\n",
" <td>9007</td>\n",
" <td>12_1176011</td>\n",
" <td>AT</td>\n",
" <td>5561</td>\n",
" <td>5964</td>\n",
" <td>6136</td>\n",
" <td>6326</td>\n",
" <td>6347</td>\n",
" <td>6339</td>\n",
" <td>...</td>\n",
" <td>5089</td>\n",
" <td>4917</td>\n",
" <td>5199</td>\n",
" <td>5031</td>\n",
" <td>5028</td>\n",
" <td>5079</td>\n",
" <td>5061</td>\n",
" <td>5133</td>\n",
" <td>4750</td>\n",
" <td>4743</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>MW</td>\n",
" <td>9007</td>\n",
" <td>12_1176011</td>\n",
" <td>BA</td>\n",
" <td>2065</td>\n",
" <td>1765</td>\n",
" <td>1765</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>...</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>MW</td>\n",
" <td>9007</td>\n",
" <td>12_1176011</td>\n",
" <td>BE</td>\n",
" <td>7182</td>\n",
" <td>7199</td>\n",
" <td>7202</td>\n",
" <td>7559</td>\n",
" <td>8182</td>\n",
" <td>9128</td>\n",
" <td>...</td>\n",
" <td>7746</td>\n",
" <td>7650</td>\n",
" <td>6945</td>\n",
" <td>7126</td>\n",
" <td>7211</td>\n",
" <td>7298</td>\n",
" <td>6433</td>\n",
" <td>6394</td>\n",
" <td>6445</td>\n",
" <td>6490</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>MW</td>\n",
" <td>9007</td>\n",
" <td>12_1176011</td>\n",
" <td>BG</td>\n",
" <td>3635</td>\n",
" <td>3854</td>\n",
" <td>4410</td>\n",
" <td>4636</td>\n",
" <td>4849</td>\n",
" <td>4495</td>\n",
" <td>...</td>\n",
" <td>5172</td>\n",
" <td>4977</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 31 columns</p>\n",
"</div>"
],
"text/plain": [
" unit product indic_nrg geo\\time 2016 2015 2014 2013 2012 \\\n",
"0 MW 9007 12_1176011 AL 98 98 98 98 97 \n",
"1 MW 9007 12_1176011 AT 5561 5964 6136 6326 6347 \n",
"2 MW 9007 12_1176011 BA 2065 1765 1765 0 0 \n",
"3 MW 9007 12_1176011 BE 7182 7199 7202 7559 8182 \n",
"4 MW 9007 12_1176011 BG 3635 3854 4410 4636 4849 \n",
"\n",
" 2011 ... 1999 1998 1997 1996 1995 1994 1993 1992 1991 \\\n",
"0 0 ... 0 0 0 0 0 0 0 0 0 \n",
"1 6339 ... 5089 4917 5199 5031 5028 5079 5061 5133 4750 \n",
"2 0 ... 0 0 0 0 0 0 0 0 0 \n",
"3 9128 ... 7746 7650 6945 7126 7211 7298 6433 6394 6445 \n",
"4 4495 ... 5172 4977 0 0 0 0 0 0 0 \n",
"\n",
" 1990 \n",
"0 0 \n",
"1 4743 \n",
"2 0 \n",
"3 6490 \n",
"4 0 \n",
"\n",
"[5 rows x 31 columns]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"url_eurostat = 'Link unavailable'\n",
"\n",
"filepath_eurostat = os.path.join('input', 'Eurostat', 'Eurostat.tsv.gz')\n",
"\n",
"data_eurostat = pd.read_csv(filepath_eurostat,\n",
" compression='gzip',\n",
" sep='\\t|,',\n",
" engine='python'\n",
" )\n",
"data_eurostat.head()"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>unit</th>\n",
" <th>product</th>\n",
" <th>indic_nrg</th>\n",
" <th>geo\\time</th>\n",
" <th>year</th>\n",
" <th>value</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>MW</td>\n",
" <td>9007</td>\n",
" <td>12_1176011</td>\n",
" <td>AL</td>\n",
" <td>2016</td>\n",
" <td>98</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>MW</td>\n",
" <td>9007</td>\n",
" <td>12_1176011</td>\n",
" <td>AT</td>\n",
" <td>2016</td>\n",
" <td>5561</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>MW</td>\n",
" <td>9007</td>\n",
" <td>12_1176011</td>\n",
" <td>BA</td>\n",
" <td>2016</td>\n",
" <td>2065</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>MW</td>\n",
" <td>9007</td>\n",
" <td>12_1176011</td>\n",
" <td>BE</td>\n",
" <td>2016</td>\n",
" <td>7182</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>MW</td>\n",
" <td>9007</td>\n",
" <td>12_1176011</td>\n",
" <td>BG</td>\n",
" <td>2016</td>\n",
" <td>3635</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" unit product indic_nrg geo\\time year value\n",
"0 MW 9007 12_1176011 AL 2016 98 \n",
"1 MW 9007 12_1176011 AT 2016 5561 \n",
"2 MW 9007 12_1176011 BA 2016 2065 \n",
"3 MW 9007 12_1176011 BE 2016 7182 \n",
"4 MW 9007 12_1176011 BG 2016 3635 "
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"id_vars = ['unit', 'product','indic_nrg', 'geo\\\\time']\n",
"data_eurostat = pd.melt(data_eurostat, id_vars=id_vars,\n",
" var_name='year', value_name='value')\n",
"\n",
"data_eurostat.head()"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"data_definition = pd.read_csv(os.path.join('input', 'definition_EUROSTAT_indic.txt'),\n",
" header=None,\n",
" names=['indic', 'description',\n",
" 'energy source'],\n",
" sep='\\t')\n",
"\n",
"data_eurostat = data_eurostat.merge(data_definition,\n",
" how='left',\n",
" left_on='indic_nrg',\n",
" right_on='indic')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The classification of generation capacities in the EUROSTAT dataset is specified in [Regulation (EC) No 1099/2008](http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32008R1099&from=EN) (Annex B, 3.3). The available EUROSTAT dataset [nrg_113a](http://ec.europa.eu/eurostat/product?code=nrg_113a&mode=view) covers the following indicators:\n",
"\n",
"|indic_nrg | Description | Technology in OPSD |\n",
"|---|---|---|\n",
"|**12_1176011**|\t**Electrical capacity, main activity producers - Combustible Fuels**|\tFossil fuels & bioenergy|\n",
"|**12_1176012**|\t**Electrical capacity, autoproducers - Combustible Fuels**|\tFossil fuels & bioenergy|\n",
"|*12_1176061*|\t*Electrical capacity, main activity producers - Mixed plants*|\t|\n",
"|*12_1176101*|\t*Electrical capacity, main activity producers - Other Sources*|\t|\n",
"|*12_1176102*|\t*Electrical capacity, autoproducers - Other Sources*|\t|\n",
"|*12_1176111*|\t*Electrical capacity, main activity producers - Steam*||\n",
"|*12_1176112*|\t*Electrical capacity, autoproducers - Steam*||\n",
"|*12_1176121*|\t*Electrical capacity, main activity producers - Gas Turbine*||\n",
"|*12_1176122*|\t*Electrical capacity, autoproducers - Gas Turbine*||\n",
"|*12_1176131*|\t*Electrical capacity, main activity producers - Combined Cycle*||\n",
"|*12_1176132*|\t*Electrical capacity, autoproducers - Combined Cycle*||\n",
"|*12_1176141*|\t*Electrical capacity, main activity producers - Internal Combustion*||\n",
"|*12_1176142*|\t*Electrical capacity, autoproducers - Internal Combustion*||\n",
"|*12_1176401*|\t*Electrical capacity, main activity producers - Other Type of Generation*|\t|\n",
"|*12_1176402*|\t*Electrical capacity, autoproducers - Other Type of Generation*|\t|\n",
"|12_1176253|\tNet maximum capacity - Municipal Wastes|\tNon-renewable waste|\n",
"|12_1176263|\tNet maximum capacity - Wood/Wood Wastes/Other Solid Wastes|\tOther bioenergy and renewable waste|\n",
"|12_1176273|\tNet maximum capacity - Biogases|\tBiomass and biogas|\n",
"|12_1176283|\tNet maximum capacity - Industrial Wastes (non-renewable)|\tNon-renewable waste|\n",
"|12_1176343|\tNet maximum capacity - Liquid Biofuels|\tBiomass and biogas|\n",
"|**12_1176031**|\t**Electrical capacity, main activity producers - Nuclear**|\tNuclear|\n",
"|**12_1176032**|\t**Electrical capacity, autoproducers - Nuclear**|\tNuclear|\n",
"|**12_1176051**|\t**Electrical capacity, main activity producers - Hydro**|\tHydro|\n",
"|**12_1176052**|\t**Electrical capacity, autoproducers - Hydro**|\tHydro|\n",
"|12_1176071|\tNet electrical capacity, main activity producers - Pure Pumped Hydro|\tPumped storage|\n",
"|12_1176072|\tNet electrical capacity, autoproducers - Pure Pumped Hydro|\tPumped storage|\n",
"|*12_117615*|\t*Net maximum capacity - Hydro <1 MW*|\t|\n",
"|*12_117616*|\t*Net maximum capacity - Hydro >= 1 MW and <= 10 MW*|\t|\n",
"|*12_117617*|\t*Net maximum capacity - Hydro 10 MW and over*|\t|\n",
"|**12_1176301**|\t**Electrical capacity, main activity producers - Tide, wave and ocean**|\tMarine|\n",
"|**12_1176302**|\t**Electrical capacity, autoproducers - Tide, wave and ocean**|\tMarine|\n",
"|*12_1176303*|\t*Net maximum capacity - Tide, Wave, Ocean*||\n",
"|**12_1176081**|\t**Electrical capacity, main activity producers - Geothermal**|\tGeothermal|\n",
"|**12_1176082**|\t**Electrical capacity, autoproducers - Geothermal**|\tGeothermal|\n",
"|*12_1176083*|\t*Net maximum capacity - Geothermal*|\t|\n",
"|**12_1176091**|\t**Electrical capacity, main activity producers - Wind**|\tWind|\n",
"|**12_1176092**|\t**Electrical capacity, autoproducers - Wind**|\tWind|\n",
"|**12_1176233**|\t**Net maximum capacity - Solar Photovoltaic**|\tPhotovoltaics|\n",
"|**12_1176243**|\t**Net maximum capacity - Solar Thermal Electric**|\tConcentrated solar power|\n",
"\n",
"**Bold** rows indicate top level classes within the EUROSTAT classification, whereas normal and *italic* rows cover different kinds of subclassifications. Especially within the top level 'Combustible fuels' different kinds of subcategorizations based on fuel or technology are available. Simarily, 'Hydro' is differentiated by type (e.g. pumped-hydro storage) or capacity classes. *Italic* rows are not further considered within the OPSD dataset due to the mismatch with existing technology classes."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>unit</th>\n",
" <th>product</th>\n",
" <th>indic_nrg</th>\n",
" <th>geo\\time</th>\n",
" <th>year</th>\n",
" <th>value</th>\n",
" <th>indic</th>\n",
" <th>description</th>\n",
" <th>energy source</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>MW</td>\n",
" <td>9007</td>\n",
" <td>12_1176011</td>\n",
" <td>AL</td>\n",
" <td>2016</td>\n",
" <td>98.0</td>\n",
" <td>12_1176011</td>\n",
" <td>Electrical capacity, main activity producers -...</td>\n",
" <td>Fossil fuels</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>MW</td>\n",
" <td>9007</td>\n",
" <td>12_1176011</td>\n",
" <td>AT</td>\n",
" <td>2016</td>\n",
" <td>5561.0</td>\n",
" <td>12_1176011</td>\n",
" <td>Electrical capacity, main activity producers -...</td>\n",
" <td>Fossil fuels</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>MW</td>\n",
" <td>9007</td>\n",
" <td>12_1176011</td>\n",
" <td>BA</td>\n",
" <td>2016</td>\n",
" <td>2065.0</td>\n",
" <td>12_1176011</td>\n",
" <td>Electrical capacity, main activity producers -...</td>\n",
" <td>Fossil fuels</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>MW</td>\n",
" <td>9007</td>\n",
" <td>12_1176011</td>\n",
" <td>BE</td>\n",
" <td>2016</td>\n",
" <td>7182.0</td>\n",
" <td>12_1176011</td>\n",
" <td>Electrical capacity, main activity producers -...</td>\n",
" <td>Fossil fuels</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>MW</td>\n",
" <td>9007</td>\n",
" <td>12_1176011</td>\n",
" <td>BG</td>\n",
" <td>2016</td>\n",
" <td>3635.0</td>\n",
" <td>12_1176011</td>\n",
" <td>Electrical capacity, main activity producers -...</td>\n",
" <td>Fossil fuels</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" unit product indic_nrg geo\\time year value indic \\\n",
"0 MW 9007 12_1176011 AL 2016 98.0 12_1176011 \n",
"1 MW 9007 12_1176011 AT 2016 5561.0 12_1176011 \n",
"2 MW 9007 12_1176011 BA 2016 2065.0 12_1176011 \n",
"3 MW 9007 12_1176011 BE 2016 7182.0 12_1176011 \n",
"4 MW 9007 12_1176011 BG 2016 3635.0 12_1176011 \n",
"\n",
" description energy source \n",
"0 Electrical capacity, main activity producers -... Fossil fuels \n",
"1 Electrical capacity, main activity producers -... Fossil fuels \n",
"2 Electrical capacity, main activity producers -... Fossil fuels \n",
"3 Electrical capacity, main activity producers -... Fossil fuels \n",
"4 Electrical capacity, main activity producers -... Fossil fuels "
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_eurostat = data_eurostat[data_eurostat['energy source'].isnull() == False]\n",
"\n",
"values_as_string = data_eurostat['value'].astype(str)\n",
"string_values = values_as_string.str.split(' ', 1).str[0]\n",
"string_values.replace(':', np.nan, inplace=True)\n",
"subset_nan = string_values.isnull()\n",
"\n",
"data_eurostat['value'] = string_values\n",
"data_eurostat['year'] = data_eurostat['year'].astype(int)\n",
"data_eurostat['value'] = data_eurostat['value'].astype(float)\n",
"\n",
"data_eurostat.head()"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"data_eurostat = data_eurostat.drop(['unit', 'product', 'indic_nrg',\n",
" 'indic', 'description'], axis=1)\n",
"\n",
"data_eurostat = data_eurostat.rename(columns={'geo\\\\time': 'country',\n",
" 'energy source': 'technology',\n",
" 'value': 'capacity'})\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>technology</th>\n",
" <th>year</th>\n",
" <th>country</th>\n",
" <th>capacity</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Biomass and biogas</td>\n",
" <td>1990</td>\n",
" <td>AL</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Biomass and biogas</td>\n",
" <td>1990</td>\n",
" <td>AT</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Biomass and biogas</td>\n",
" <td>1990</td>\n",
" <td>BA</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Biomass and biogas</td>\n",
" <td>1990</td>\n",
" <td>BE</td>\n",
" <td>1.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Biomass and biogas</td>\n",
" <td>1990</td>\n",
" <td>BG</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" technology year country capacity\n",
"0 Biomass and biogas 1990 AL 0.0\n",
"1 Biomass and biogas 1990 AT 0.0\n",
"2 Biomass and biogas 1990 BA 0.0\n",
"3 Biomass and biogas 1990 BE 1.0\n",
"4 Biomass and biogas 1990 BG 0.0"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_eurostat['country'].replace({'UK': 'GB', 'EL': 'GR'}, inplace=True)\n",
"drop_list = data_eurostat[data_eurostat['country'].isin(['EU28','EA19'])].index\n",
"data_eurostat.drop(drop_list, inplace=True)\n",
"\n",
"by_columns = ['technology', 'year', 'country']\n",
"data_eurostat = pd.DataFrame(data_eurostat.groupby(by_columns)['capacity'].sum())\n",
"data_eurostat_isnull = data_eurostat['capacity'].isnull() == True\n",
"data_eurostat.reset_index(inplace=True)\n",
"\n",
"data_eurostat.head()"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>technology</th>\n",
" <th>Biomass and biogas</th>\n",
" <th>Concentrated solar power</th>\n",
" <th>Fossil fuels</th>\n",
" <th>Geothermal</th>\n",
" <th>Hydro</th>\n",
" <th>Marine</th>\n",
" <th>Non-renewable waste</th>\n",
" <th>Nuclear</th>\n",
" <th>Other bioenergy and renewable waste</th>\n",
" <th>Photovoltaics</th>\n",
" <th>Pumped storage</th>\n",
" <th>Wind</th>\n",
" </tr>\n",
" <tr>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th rowspan=\"5\" valign=\"top\">AL</th>\n",
" <th>1990</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1991</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1992</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1453.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1993</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1453.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1994</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1453.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"technology Biomass and biogas Concentrated solar power Fossil fuels \\\n",
"country year \n",
"AL 1990 0.0 0.0 0.0 \n",
" 1991 0.0 0.0 0.0 \n",
" 1992 0.0 0.0 0.0 \n",
" 1993 0.0 0.0 0.0 \n",
" 1994 0.0 0.0 0.0 \n",
"\n",
"technology Geothermal Hydro Marine Non-renewable waste Nuclear \\\n",
"country year \n",
"AL 1990 0.0 0.0 0.0 0.0 0.0 \n",
" 1991 0.0 0.0 0.0 0.0 0.0 \n",
" 1992 0.0 1453.0 0.0 0.0 0.0 \n",
" 1993 0.0 1453.0 0.0 0.0 0.0 \n",
" 1994 0.0 1453.0 0.0 0.0 0.0 \n",
"\n",
"technology Other bioenergy and renewable waste Photovoltaics \\\n",
"country year \n",
"AL 1990 0.0 0.0 \n",
" 1991 0.0 0.0 \n",
" 1992 0.0 0.0 \n",
" 1993 0.0 0.0 \n",
" 1994 0.0 0.0 \n",
"\n",
"technology Pumped storage Wind \n",
"country year \n",
"AL 1990 0.0 0.0 \n",
" 1991 0.0 0.0 \n",
" 1992 0.0 0.0 \n",
" 1993 0.0 0.0 \n",
" 1994 0.0 0.0 "
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"eurostat_pivot = data_eurostat.pivot_table(values='capacity',\n",
" index=['country','year'],\n",
" columns='technology')\n",
"\n",
"eurostat_pivot.head()"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"01 Oct 2020 16:07:59 - numexpr.utils - INFO - NumExpr defaulting to 4 threads.\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>technology</th>\n",
" <th>Biomass and biogas</th>\n",
" <th>Concentrated solar power</th>\n",
" <th>Fossil fuels</th>\n",
" <th>Geothermal</th>\n",
" <th>Hydro</th>\n",
" <th>Marine</th>\n",
" <th>Non-renewable waste</th>\n",
" <th>Nuclear</th>\n",
" <th>Other bioenergy and renewable waste</th>\n",
" <th>Photovoltaics</th>\n",
" <th>Pumped storage</th>\n",
" <th>Wind</th>\n",
" <th>Differently categorized solar</th>\n",
" <th>Solar</th>\n",
" <th>Differently categorized wind</th>\n",
" <th>Bioenergy and renewable waste</th>\n",
" <th>Renewable energy sources</th>\n",
" <th>Differently categorized fossil fuels</th>\n",
" <th>Total</th>\n",
" </tr>\n",
" <tr>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th rowspan=\"5\" valign=\"top\">AL</th>\n",
" <th>1990</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1991</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1992</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1453.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1453.0</td>\n",
" <td>0.0</td>\n",
" <td>1453.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1993</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1453.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1453.0</td>\n",
" <td>0.0</td>\n",
" <td>1453.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1994</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1453.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1453.0</td>\n",
" <td>0.0</td>\n",
" <td>1453.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"technology Biomass and biogas Concentrated solar power Fossil fuels \\\n",
"country year \n",
"AL 1990 0.0 0.0 0.0 \n",
" 1991 0.0 0.0 0.0 \n",
" 1992 0.0 0.0 0.0 \n",
" 1993 0.0 0.0 0.0 \n",
" 1994 0.0 0.0 0.0 \n",
"\n",
"technology Geothermal Hydro Marine Non-renewable waste Nuclear \\\n",
"country year \n",
"AL 1990 0.0 0.0 0.0 0.0 0.0 \n",
" 1991 0.0 0.0 0.0 0.0 0.0 \n",
" 1992 0.0 1453.0 0.0 0.0 0.0 \n",
" 1993 0.0 1453.0 0.0 0.0 0.0 \n",
" 1994 0.0 1453.0 0.0 0.0 0.0 \n",
"\n",
"technology Other bioenergy and renewable waste Photovoltaics \\\n",
"country year \n",
"AL 1990 0.0 0.0 \n",
" 1991 0.0 0.0 \n",
" 1992 0.0 0.0 \n",
" 1993 0.0 0.0 \n",
" 1994 0.0 0.0 \n",
"\n",
"technology Pumped storage Wind Differently categorized solar Solar \\\n",
"country year \n",
"AL 1990 0.0 0.0 0 0.0 \n",
" 1991 0.0 0.0 0 0.0 \n",
" 1992 0.0 0.0 0 0.0 \n",
" 1993 0.0 0.0 0 0.0 \n",
" 1994 0.0 0.0 0 0.0 \n",
"\n",
"technology Differently categorized wind Bioenergy and renewable waste \\\n",
"country year \n",
"AL 1990 0.0 0.0 \n",
" 1991 0.0 0.0 \n",
" 1992 0.0 0.0 \n",
" 1993 0.0 0.0 \n",
" 1994 0.0 0.0 \n",
"\n",
"technology Renewable energy sources Differently categorized fossil fuels \\\n",
"country year \n",
"AL 1990 0.0 0.0 \n",
" 1991 0.0 0.0 \n",
" 1992 1453.0 0.0 \n",
" 1993 1453.0 0.0 \n",
" 1994 1453.0 0.0 \n",
"\n",
"technology Total \n",
"country year \n",
"AL 1990 0.0 \n",
" 1991 0.0 \n",
" 1992 1453.0 \n",
" 1993 1453.0 \n",
" 1994 1453.0 "
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"eurostat_pivot['Differently categorized solar'] = 0\n",
"eurostat_pivot['Solar'] = eurostat_pivot[['Photovoltaics', 'Concentrated solar power']].sum(axis=1)\n",
"\n",
"eurostat_pivot['Differently categorized wind'] = eurostat_pivot['Wind']\n",
"\n",
"bio_arr = ['Biomass and biogas', 'Other bioenergy and renewable waste']\n",
"eurostat_pivot['Bioenergy and renewable waste'] = eurostat_pivot[bio_arr].sum(axis=1)\n",
"\n",
"res_arr = ['Hydro', 'Wind', 'Solar', 'Geothermal', 'Marine', 'Bioenergy and renewable waste']\n",
"eurostat_pivot['Renewable energy sources'] = eurostat_pivot[res_arr].sum(axis=1)\n",
"\n",
"\n",
"eurostat_pivot['Fossil fuels'] = eurostat_pivot['Fossil fuels'] - eurostat_pivot['Bioenergy and renewable waste']\n",
"eurostat_pivot['Differently categorized fossil fuels'] = eurostat_pivot['Fossil fuels']\\\n",
" - eurostat_pivot['Non-renewable waste']\n",
"\n",
"total_arr = ['Fossil fuels','Nuclear','Renewable energy sources']\n",
"eurostat_pivot['Total'] = eurostat_pivot[total_arr].sum(axis=1)\n",
"\n",
"eurostat_pivot.head()"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" <th>technology</th>\n",
" <th>capacity</th>\n",
" <th>source</th>\n",
" <th>source_type</th>\n",
" <th>capacity_definition</th>\n",
" <th>type</th>\n",
" <th>weblink</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AL</td>\n",
" <td>1990</td>\n",
" <td>Biomass and biogas</td>\n",
" <td>0.0</td>\n",
" <td>EUROSTAT</td>\n",
" <td>Statistical Office</td>\n",
" <td>Unknown</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>Link unavailable</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>AL</td>\n",
" <td>1990</td>\n",
" <td>Concentrated solar power</td>\n",
" <td>0.0</td>\n",
" <td>EUROSTAT</td>\n",
" <td>Statistical Office</td>\n",
" <td>Unknown</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>Link unavailable</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>AL</td>\n",
" <td>1990</td>\n",
" <td>Fossil fuels</td>\n",
" <td>0.0</td>\n",
" <td>EUROSTAT</td>\n",
" <td>Statistical Office</td>\n",
" <td>Unknown</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>Link unavailable</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AL</td>\n",
" <td>1990</td>\n",
" <td>Geothermal</td>\n",
" <td>0.0</td>\n",
" <td>EUROSTAT</td>\n",
" <td>Statistical Office</td>\n",
" <td>Unknown</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>Link unavailable</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AL</td>\n",
" <td>1990</td>\n",
" <td>Hydro</td>\n",
" <td>0.0</td>\n",
" <td>EUROSTAT</td>\n",
" <td>Statistical Office</td>\n",
" <td>Unknown</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>Link unavailable</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" country year technology capacity source \\\n",
"0 AL 1990 Biomass and biogas 0.0 EUROSTAT \n",
"1 AL 1990 Concentrated solar power 0.0 EUROSTAT \n",
"2 AL 1990 Fossil fuels 0.0 EUROSTAT \n",
"3 AL 1990 Geothermal 0.0 EUROSTAT \n",
"4 AL 1990 Hydro 0.0 EUROSTAT \n",
"\n",
" source_type capacity_definition type \\\n",
"0 Statistical Office Unknown Installed capacity in MW \n",
"1 Statistical Office Unknown Installed capacity in MW \n",
"2 Statistical Office Unknown Installed capacity in MW \n",
"3 Statistical Office Unknown Installed capacity in MW \n",
"4 Statistical Office Unknown Installed capacity in MW \n",
"\n",
" weblink \n",
"0 Link unavailable \n",
"1 Link unavailable \n",
"2 Link unavailable \n",
"3 Link unavailable \n",
"4 Link unavailable "
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_eurostat = eurostat_pivot.stack().reset_index().rename(columns={0: 'capacity'})\n",
"\n",
"data_eurostat['source'] = 'EUROSTAT'\n",
"data_eurostat['source_type'] = 'Statistical Office'\n",
"data_eurostat['capacity_definition'] = 'Unknown'\n",
"data_eurostat['type'] = 'Installed capacity in MW'\n",
"data_eurostat['weblink'] = url_eurostat\n",
"\n",
"data_eurostat.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3.3 ENTSO-E data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The ENTSO-E publishes annual data on national generation capacites in different specifications and formats. We use two relevant data sources from the ENTSOE-E, which comprises firstly statistical data within the [Data Portal (up to 2015)](https://www.entsoe.eu/data/data-portal/Pages/default.aspx) or [ENTSO-E Transparency Platform](https://transparency.entsoe.eu/), and secondly datasets compiled within the [ENTSO-E System Outlook & Adequacy Forecast (SO&AF)](https://www.entsoe.eu/outlooks/maf/Pages/default.aspx). The ENTSO-E Transparency Platform is currently not implemented as a data source for national generation capacities.\n",
"\n",
"The advantage of the ENTSO-E SO&AF is the higher granularity of the data with respect to the main fuel or technology. However, as the SO&AF provides a forecast on future system conditions in particular peak hours, the dataset also accounts for expected capacity changes throughout the years. Therefore, we only consider years which are closest to the publication year of the respective SO&AF."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3.3.1 ENTSO-E Statistical Data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the following, we use the statistical data available in the [Data Portal (up to 2015)](https://www.entsoe.eu/data/data-portal/Pages/default.aspx)."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"01 Oct 2020 16:07:59 - root - INFO - Downloading file Statistics.xls\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Country</th>\n",
" <th>year</th>\n",
" <th>hydro</th>\n",
" <th>of which storage</th>\n",
" <th>of which run of river</th>\n",
" <th>of which pumped storage</th>\n",
" <th>nuclear</th>\n",
" <th>renewable</th>\n",
" <th>of which wind</th>\n",
" <th>of which solar</th>\n",
" <th>of which biomass</th>\n",
" <th>fossil_fueals</th>\n",
" <th>other</th>\n",
" <th>representativity</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AT</td>\n",
" <td>2010</td>\n",
" <td>12665.0</td>\n",
" <td>7205.0</td>\n",
" <td>5241.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>1031.0</td>\n",
" <td>1002.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>7389.0</td>\n",
" <td>0.0</td>\n",
" <td>100.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>AT</td>\n",
" <td>2011</td>\n",
" <td>13200.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>1542.0</td>\n",
" <td>1056.0</td>\n",
" <td>55.0</td>\n",
" <td>0.0</td>\n",
" <td>7818.0</td>\n",
" <td>68.0</td>\n",
" <td>100.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>AT</td>\n",
" <td>2012</td>\n",
" <td>13350.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>1906.0</td>\n",
" <td>1307.0</td>\n",
" <td>172.0</td>\n",
" <td>427.0</td>\n",
" <td>7834.0</td>\n",
" <td>74.0</td>\n",
" <td>100.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AT</td>\n",
" <td>2013</td>\n",
" <td>13427.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>2305.0</td>\n",
" <td>1555.0</td>\n",
" <td>324.0</td>\n",
" <td>426.0</td>\n",
" <td>7847.0</td>\n",
" <td>244.0</td>\n",
" <td>100.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AT</td>\n",
" <td>2014</td>\n",
" <td>13569.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>3413.0</td>\n",
" <td>2110.0</td>\n",
" <td>589.0</td>\n",
" <td>512.0</td>\n",
" <td>7243.0</td>\n",
" <td>0.0</td>\n",
" <td>100.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Country year hydro of which storage of which run of river \\\n",
"0 AT 2010 12665.0 7205.0 5241.0 \n",
"1 AT 2011 13200.0 0.0 0.0 \n",
"2 AT 2012 13350.0 0.0 0.0 \n",
"3 AT 2013 13427.0 0.0 0.0 \n",
"4 AT 2014 13569.0 0.0 0.0 \n",
"\n",
" of which pumped storage nuclear renewable of which wind of which solar \\\n",
"0 0.0 0 1031.0 1002.0 0.0 \n",
"1 0.0 0 1542.0 1056.0 55.0 \n",
"2 0.0 0 1906.0 1307.0 172.0 \n",
"3 0.0 0 2305.0 1555.0 324.0 \n",
"4 0.0 0 3413.0 2110.0 589.0 \n",
"\n",
" of which biomass fossil_fueals other representativity \n",
"0 0.0 7389.0 0.0 100.0 \n",
"1 0.0 7818.0 68.0 100.0 \n",
"2 427.0 7834.0 74.0 100.0 \n",
"3 426.0 7847.0 244.0 100.0 \n",
"4 512.0 7243.0 0.0 100.0 "
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"url_entsoe = 'https://docstore.entsoe.eu/Documents/Publications/Statistics/NGC_2010-2015.xlsx'\n",
"\n",
"filepath_entsoe = func.downloadandcache(url_entsoe, 'Statistics.xls',\n",
" os.path.join('ENTSO-E','Data Portal 2010-2015')\n",
" )\n",
"\n",
"data_entsoe_raw = pd.read_excel(filepath_entsoe)\n",
"\n",
"data_entsoe_raw.head()"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" <th>Hydro</th>\n",
" <th>Reservoir</th>\n",
" <th>Run-of-river</th>\n",
" <th>Pumped storage</th>\n",
" <th>Nuclear</th>\n",
" <th>renewable</th>\n",
" <th>Wind</th>\n",
" <th>Solar</th>\n",
" <th>Biomass and biogas</th>\n",
" <th>Fossil fuels</th>\n",
" <th>Other or unspecified energy sources</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AT</td>\n",
" <td>2010</td>\n",
" <td>12665.0</td>\n",
" <td>7205.0</td>\n",
" <td>5241.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>1031.0</td>\n",
" <td>1002.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>7389.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>AT</td>\n",
" <td>2011</td>\n",
" <td>13200.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>1542.0</td>\n",
" <td>1056.0</td>\n",
" <td>55.0</td>\n",
" <td>0.0</td>\n",
" <td>7818.0</td>\n",
" <td>68.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>AT</td>\n",
" <td>2012</td>\n",
" <td>13350.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>1906.0</td>\n",
" <td>1307.0</td>\n",
" <td>172.0</td>\n",
" <td>427.0</td>\n",
" <td>7834.0</td>\n",
" <td>74.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AT</td>\n",
" <td>2013</td>\n",
" <td>13427.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>2305.0</td>\n",
" <td>1555.0</td>\n",
" <td>324.0</td>\n",
" <td>426.0</td>\n",
" <td>7847.0</td>\n",
" <td>244.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AT</td>\n",
" <td>2014</td>\n",
" <td>13569.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>3413.0</td>\n",
" <td>2110.0</td>\n",
" <td>589.0</td>\n",
" <td>512.0</td>\n",
" <td>7243.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" country year Hydro Reservoir Run-of-river Pumped storage Nuclear \\\n",
"0 AT 2010 12665.0 7205.0 5241.0 0.0 0 \n",
"1 AT 2011 13200.0 0.0 0.0 0.0 0 \n",
"2 AT 2012 13350.0 0.0 0.0 0.0 0 \n",
"3 AT 2013 13427.0 0.0 0.0 0.0 0 \n",
"4 AT 2014 13569.0 0.0 0.0 0.0 0 \n",
"\n",
" renewable Wind Solar Biomass and biogas Fossil fuels \\\n",
"0 1031.0 1002.0 0.0 0.0 7389.0 \n",
"1 1542.0 1056.0 55.0 0.0 7818.0 \n",
"2 1906.0 1307.0 172.0 427.0 7834.0 \n",
"3 2305.0 1555.0 324.0 426.0 7847.0 \n",
"4 3413.0 2110.0 589.0 512.0 7243.0 \n",
"\n",
" Other or unspecified energy sources \n",
"0 0.0 \n",
"1 68.0 \n",
"2 74.0 \n",
"3 244.0 \n",
"4 0.0 "
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dict_energy_source = {'hydro': 'Hydro',\n",
" 'of which storage': 'Reservoir',\n",
" 'of which run of river': 'Run-of-river',\n",
" 'of which pumped storage': 'Pumped storage',\n",
" 'nuclear': 'Nuclear',\n",
" 'of which wind': 'Wind',\n",
" 'of which solar': 'Solar',\n",
" 'of which biomass': 'Biomass and biogas',\n",
" 'fossil_fuels': 'Fossil fuels',\n",
" 'other': 'Other or unspecified energy sources',\n",
" \"Country\": \"country\",\n",
" 'fossil_fueals': 'Fossil fuels'}\n",
"\n",
"data_entsoe_raw.rename(columns=dict_energy_source,\n",
" inplace=True)\n",
"\n",
"data_entsoe_raw.drop(columns='representativity', inplace=True)\n",
"\n",
"data_entsoe_raw.head()"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" <th>technology</th>\n",
" <th>capacity</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AT</td>\n",
" <td>2010</td>\n",
" <td>Hydro</td>\n",
" <td>12665.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>AT</td>\n",
" <td>2011</td>\n",
" <td>Hydro</td>\n",
" <td>13200.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>AT</td>\n",
" <td>2012</td>\n",
" <td>Hydro</td>\n",
" <td>13350.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AT</td>\n",
" <td>2013</td>\n",
" <td>Hydro</td>\n",
" <td>13427.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AT</td>\n",
" <td>2014</td>\n",
" <td>Hydro</td>\n",
" <td>13569.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" country year technology capacity\n",
"0 AT 2010 Hydro 12665.0\n",
"1 AT 2011 Hydro 13200.0\n",
"2 AT 2012 Hydro 13350.0\n",
"3 AT 2013 Hydro 13427.0\n",
"4 AT 2014 Hydro 13569.0"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"\n",
"data_entsoe_raw['Differently categorized solar'] = data_entsoe_raw['Solar']\n",
"data_entsoe_raw['Differently categorized wind'] = data_entsoe_raw['Wind']\n",
"data_entsoe_raw['Bioenergy and renewable waste'] = data_entsoe_raw['Biomass and biogas']\n",
"data_entsoe_raw['Differently categorized fossil fuels'] = data_entsoe_raw['Fossil fuels']\n",
"\n",
"\n",
"data_entsoe_raw['Differently categorized hydro'] = (\n",
" data_entsoe_raw['Hydro']\n",
" - data_entsoe_raw['Run-of-river']\n",
" - data_entsoe_raw['Reservoir']\n",
" - data_entsoe_raw['Pumped storage'])\n",
"\n",
"data_entsoe_raw['Differently categorized renewable energy sources'] = (\n",
" data_entsoe_raw['renewable']\n",
" - data_entsoe_raw['Wind']\n",
" - data_entsoe_raw['Solar']\n",
" - data_entsoe_raw['Biomass and biogas'])\n",
"\n",
"data_entsoe_raw.drop(columns='renewable', inplace=True)\n",
"\n",
"data_entsoe_raw['Renewable energy sources'] = (\n",
" data_entsoe_raw['Hydro']\n",
" + data_entsoe_raw['Wind']\n",
" + data_entsoe_raw['Solar']\n",
" + data_entsoe_raw['Bioenergy and renewable waste']\n",
" + data_entsoe_raw['Differently categorized renewable energy sources'])\n",
"\n",
"data_entsoe_raw['Total'] = (\n",
" data_entsoe_raw['Renewable energy sources']\n",
" + data_entsoe_raw['Nuclear']\n",
" + data_entsoe_raw['Fossil fuels']\n",
" + data_entsoe_raw['Other or unspecified energy sources'])\n",
"\n",
"data_entsoe = pd.melt(data_entsoe_raw,\n",
" id_vars=['country', 'year'],\n",
" var_name='technology',\n",
" value_name='capacity')\n",
"\n",
"data_entsoe.head()"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" <th>technology</th>\n",
" <th>capacity</th>\n",
" <th>source</th>\n",
" <th>source_type</th>\n",
" <th>capacity_definition</th>\n",
" <th>type</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AT</td>\n",
" <td>2010</td>\n",
" <td>Hydro</td>\n",
" <td>12665.0</td>\n",
" <td>ENTSO-E Data Portal</td>\n",
" <td>Other association</td>\n",
" <td>Net capacity</td>\n",
" <td>Installed capacity in MW</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>AT</td>\n",
" <td>2011</td>\n",
" <td>Hydro</td>\n",
" <td>13200.0</td>\n",
" <td>ENTSO-E Data Portal</td>\n",
" <td>Other association</td>\n",
" <td>Net capacity</td>\n",
" <td>Installed capacity in MW</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>AT</td>\n",
" <td>2012</td>\n",
" <td>Hydro</td>\n",
" <td>13350.0</td>\n",
" <td>ENTSO-E Data Portal</td>\n",
" <td>Other association</td>\n",
" <td>Net capacity</td>\n",
" <td>Installed capacity in MW</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AT</td>\n",
" <td>2013</td>\n",
" <td>Hydro</td>\n",
" <td>13427.0</td>\n",
" <td>ENTSO-E Data Portal</td>\n",
" <td>Other association</td>\n",
" <td>Net capacity</td>\n",
" <td>Installed capacity in MW</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AT</td>\n",
" <td>2014</td>\n",
" <td>Hydro</td>\n",
" <td>13569.0</td>\n",
" <td>ENTSO-E Data Portal</td>\n",
" <td>Other association</td>\n",
" <td>Net capacity</td>\n",
" <td>Installed capacity in MW</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" country year technology capacity source source_type \\\n",
"0 AT 2010 Hydro 12665.0 ENTSO-E Data Portal Other association \n",
"1 AT 2011 Hydro 13200.0 ENTSO-E Data Portal Other association \n",
"2 AT 2012 Hydro 13350.0 ENTSO-E Data Portal Other association \n",
"3 AT 2013 Hydro 13427.0 ENTSO-E Data Portal Other association \n",
"4 AT 2014 Hydro 13569.0 ENTSO-E Data Portal Other association \n",
"\n",
" capacity_definition type \n",
"0 Net capacity Installed capacity in MW \n",
"1 Net capacity Installed capacity in MW \n",
"2 Net capacity Installed capacity in MW \n",
"3 Net capacity Installed capacity in MW \n",
"4 Net capacity Installed capacity in MW "
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_entsoe['country'].replace('NI', 'GB', inplace=True)\n",
"# set negative capacities to zero\n",
"data_entsoe.loc[data_entsoe['capacity'] < 0, 'capacity'] = 0\n",
"\n",
"data_entsoe['source'] = 'ENTSO-E Data Portal'\n",
"data_entsoe['source_type'] = 'Other association'\n",
"data_entsoe['capacity_definition'] = 'Net capacity'\n",
"data_entsoe['type'] = 'Installed capacity in MW'\n",
"\n",
"data_entsoe.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3.3.2 ENTSO-E SO&AF data"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"01 Oct 2020 16:08:00 - root - INFO - Downloading file SO_AF_2011_-_2025_.zip\n",
"01 Oct 2020 16:08:01 - root - INFO - Extracting SO_AF_2011_-_2025_.zip into the directory download\\ENTSO-E\\SO&AF\\2011\n",
"01 Oct 2020 16:08:01 - root - INFO - Reading download\\ENTSO-E\\SO&AF\\2011\\SO&AF 2011 - 2025 Scenario B.xls and transforming into a DataFrame\n",
"01 Oct 2020 16:08:02 - root - INFO - Downloading file 120705_SOAF_2012_Dataset.zip\n",
"01 Oct 2020 16:08:02 - root - INFO - Extracting 120705_SOAF_2012_Dataset.zip into the directory download\\ENTSO-E\\SO&AF\\2012\n",
"01 Oct 2020 16:08:02 - root - INFO - Reading download\\ENTSO-E\\SO&AF\\2012\\SOAF 2012 Scenario B.xls and transforming into a DataFrame\n",
"01 Oct 2020 16:08:03 - root - INFO - Downloading file 130403_SOAF_2013-2030_dataset.zip\n",
"01 Oct 2020 16:08:04 - root - INFO - Extracting 130403_SOAF_2013-2030_dataset.zip into the directory download\\ENTSO-E\\SO&AF\\2013\n",
"01 Oct 2020 16:08:04 - root - INFO - Reading download\\ENTSO-E\\SO&AF\\2013\\ScB.xls and transforming into a DataFrame\n",
"01 Oct 2020 16:08:05 - root - INFO - Downloading file 140602_SOAF%202014_dataset.zip\n",
"01 Oct 2020 16:08:05 - root - INFO - Extracting 140602_SOAF%202014_dataset.zip into the directory download\\ENTSO-E\\SO&AF\\2014\n",
"01 Oct 2020 16:08:05 - root - INFO - Reading download\\ENTSO-E\\SO&AF\\2014\\ScB.xlsx and transforming into a DataFrame\n",
"01 Oct 2020 16:08:06 - root - INFO - Downloading file SO_AF_2015_dataset.zip\n",
"01 Oct 2020 16:08:07 - root - INFO - Extracting SO_AF_2015_dataset.zip into the directory download\\ENTSO-E\\SO&AF\\2016\n",
"01 Oct 2020 16:08:07 - root - INFO - Reading download\\ENTSO-E\\SO&AF\\2016\\SO&AF 2015 dataset\\ScB_publication.xlsx and transforming into a DataFrame\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>technology</th>\n",
" <th>capacity</th>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Nuclear</td>\n",
" <td>0.0</td>\n",
" <td>AL</td>\n",
" <td>2011</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Fossil fuels</td>\n",
" <td>0.0</td>\n",
" <td>AL</td>\n",
" <td>2011</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Lignite</td>\n",
" <td>0.0</td>\n",
" <td>AL</td>\n",
" <td>2011</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Hard coal</td>\n",
" <td>0.0</td>\n",
" <td>AL</td>\n",
" <td>2011</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Natural gas</td>\n",
" <td>0.0</td>\n",
" <td>AL</td>\n",
" <td>2011</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" technology capacity country year\n",
"1 Nuclear 0.0 AL 2011\n",
"2 Fossil fuels 0.0 AL 2011\n",
"3 Lignite 0.0 AL 2011\n",
"4 Hard coal 0.0 AL 2011\n",
"5 Natural gas 0.0 AL 2011"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"soafs = [soaf.SoafDataRaw('https://www.entsoe.eu/fileadmin/user_upload/_library/SDC/SOAF/SO_AF_2011_-_2025_.zip',\n",
" 'SO_AF_2011_-_2025_.zip',\n",
" 'SO&AF 2011 - 2025 Scenario B.xls',\n",
" 2011),\n",
"\n",
" soaf.SoafDataRaw('https://www.entsoe.eu/fileadmin/user_upload/_library/SDC/SOAF/120705_SOAF_2012_Dataset.zip',\n",
" '120705_SOAF_2012_Dataset.zip',\n",
" 'SOAF 2012 Scenario B.xls',\n",
" 2012),\n",
" \n",
" soaf.SoafDataRaw('https://www.entsoe.eu/fileadmin/user_upload/_library/publications/entsoe/So_AF_2013-2030/130403_SOAF_2013-2030_dataset.zip',\n",
" '130403_SOAF_2013-2030_dataset.zip',\n",
" 'ScB.xls',\n",
" 2013),\n",
" \n",
" soaf.SoafDataRaw('https://www.entsoe.eu/Documents/SDC%20documents/SOAF/140602_SOAF%202014_dataset.zip',\n",
" '140602_SOAF%202014_dataset.zip',\n",
" 'ScB.xlsx',\n",
" 2014),\n",
" \n",
" soaf.SoafDataRaw('https://www.entsoe.eu/Documents/Publications/SDC/data/SO_AF_2015_dataset.zip',\n",
" 'SO_AF_2015_dataset.zip',\n",
" os.path.join('SO&AF 2015 dataset', 'ScB_publication.xlsx'),\n",
" 2016)]\n",
"\n",
"\n",
"data_soaf = pd.concat([s.transformed_df for s in soafs])\n",
"\n",
"# Correct that in the Soaf2015 datatset the year column is 2016 instead of 2015\n",
"data_soaf['year'].replace({2016 : 2015}, inplace=True)\n",
"\n",
"data_soaf.head()"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>technology</th>\n",
" <th>Biomass and biogas</th>\n",
" <th>Fossil fuels</th>\n",
" <th>Hard coal</th>\n",
" <th>Hydro</th>\n",
" <th>Lignite</th>\n",
" <th>Mixed fossil fuels</th>\n",
" <th>Natural gas</th>\n",
" <th>Nuclear</th>\n",
" <th>Offshore</th>\n",
" <th>Oil</th>\n",
" <th>...</th>\n",
" <th>Solar</th>\n",
" <th>Wind</th>\n",
" <th>Differently categorized solar</th>\n",
" <th>Differently categorized wind</th>\n",
" <th>Differently categorized hydro</th>\n",
" <th>Bioenergy and renewable waste</th>\n",
" <th>Differently categorized renewable energy sources</th>\n",
" <th>Differently categorized fossil fuels</th>\n",
" <th>Renewable energy sources</th>\n",
" <th>Total</th>\n",
" </tr>\n",
" <tr>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th rowspan=\"5\" valign=\"top\">AL</th>\n",
" <th>2011</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2012</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2013</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>NaN</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2014</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>NaN</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2015</th>\n",
" <td>0.0</td>\n",
" <td>100.0</td>\n",
" <td>0.0</td>\n",
" <td>1860.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>100.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>NaN</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1860.0</td>\n",
" <td>1960.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 24 columns</p>\n",
"</div>"
],
"text/plain": [
"technology Biomass and biogas Fossil fuels Hard coal Hydro Lignite \\\n",
"country year \n",
"AL 2011 0.0 0.0 0.0 0.0 0.0 \n",
" 2012 0.0 0.0 0.0 0.0 0.0 \n",
" 2013 0.0 0.0 0.0 0.0 0.0 \n",
" 2014 0.0 0.0 0.0 0.0 0.0 \n",
" 2015 0.0 100.0 0.0 1860.0 0.0 \n",
"\n",
"technology Mixed fossil fuels Natural gas Nuclear Offshore Oil ... \\\n",
"country year ... \n",
"AL 2011 0.0 0.0 0.0 0.0 0.0 ... \n",
" 2012 0.0 0.0 0.0 0.0 0.0 ... \n",
" 2013 0.0 0.0 0.0 0.0 0.0 ... \n",
" 2014 0.0 0.0 0.0 0.0 0.0 ... \n",
" 2015 0.0 100.0 0.0 0.0 0.0 ... \n",
"\n",
"technology Solar Wind Differently categorized solar \\\n",
"country year \n",
"AL 2011 0.0 0.0 0.0 \n",
" 2012 0.0 0.0 0.0 \n",
" 2013 0.0 0.0 0.0 \n",
" 2014 0.0 0.0 0.0 \n",
" 2015 0.0 0.0 0.0 \n",
"\n",
"technology Differently categorized wind Differently categorized hydro \\\n",
"country year \n",
"AL 2011 0.0 0.0 \n",
" 2012 0.0 0.0 \n",
" 2013 0.0 NaN \n",
" 2014 0.0 NaN \n",
" 2015 0.0 NaN \n",
"\n",
"technology Bioenergy and renewable waste \\\n",
"country year \n",
"AL 2011 0.0 \n",
" 2012 0.0 \n",
" 2013 0.0 \n",
" 2014 0.0 \n",
" 2015 0.0 \n",
"\n",
"technology Differently categorized renewable energy sources \\\n",
"country year \n",
"AL 2011 0.0 \n",
" 2012 0.0 \n",
" 2013 0.0 \n",
" 2014 0.0 \n",
" 2015 0.0 \n",
"\n",
"technology Differently categorized fossil fuels Renewable energy sources \\\n",
"country year \n",
"AL 2011 0.0 0.0 \n",
" 2012 0.0 0.0 \n",
" 2013 0.0 0.0 \n",
" 2014 0.0 0.0 \n",
" 2015 0.0 1860.0 \n",
"\n",
"technology Total \n",
"country year \n",
"AL 2011 0.0 \n",
" 2012 0.0 \n",
" 2013 0.0 \n",
" 2014 0.0 \n",
" 2015 1960.0 \n",
"\n",
"[5 rows x 24 columns]"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"soaf_unstacked = func.unstackData(data_soaf)\n",
"\n",
"soaf_unstacked['Differently categorized solar'] = soaf_unstacked['Solar']\n",
"\n",
"soaf_unstacked['Differently categorized wind'] = soaf_unstacked['Wind']\\\n",
" - soaf_unstacked['Offshore']\\\n",
" - soaf_unstacked['Onshore']\n",
"\n",
"soaf_unstacked['Differently categorized hydro'] = soaf_unstacked['Hydro']\\\n",
" - soaf_unstacked['Run-of-river']\\\n",
" - soaf_unstacked['Reservoir including pumped storage']\n",
" \n",
"\n",
"soaf_unstacked['Bioenergy and renewable waste'] = soaf_unstacked['Biomass and biogas']\n",
" \n",
"soaf_unstacked['Differently categorized renewable energy sources'] = (\n",
" soaf_unstacked['renewable']\n",
" - soaf_unstacked['Wind']\n",
" - soaf_unstacked['Solar']\n",
" - soaf_unstacked['Biomass and biogas'])\n",
"\n",
"soaf_unstacked.drop(columns='renewable', inplace=True)\n",
"\n",
"subtract_fossils_arr = ['Lignite','Hard coal','Oil','Natural gas','Mixed fossil fuels']\n",
"\n",
"soaf_unstacked['Differently categorized fossil fuels'] = soaf_unstacked['Fossil fuels']\\\n",
" - soaf_unstacked[subtract_fossils_arr].sum(axis=1)\n",
"\n",
"\n",
"res_arr = ['Solar','Wind','Bioenergy and renewable waste','Hydro',\n",
" 'Differently categorized renewable energy sources']\n",
"\n",
"soaf_unstacked['Renewable energy sources'] = soaf_unstacked[res_arr].sum(axis=1)\n",
"\n",
"total_arr = ['Renewable energy sources','Fossil fuels','Nuclear',\n",
" 'Other or unspecified energy sources']\n",
"\n",
"soaf_unstacked['Total'] = soaf_unstacked[total_arr].sum(axis=1)\n",
"\n",
"soaf_unstacked.head()"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" <th>technology</th>\n",
" <th>capacity</th>\n",
" <th>source</th>\n",
" <th>type</th>\n",
" <th>capacity_definition</th>\n",
" <th>source_type</th>\n",
" <th>weblink</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AL</td>\n",
" <td>2011</td>\n",
" <td>Biomass and biogas</td>\n",
" <td>0.0</td>\n",
" <td>ENTSO-E SOAF</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>Net capacity</td>\n",
" <td>Other association</td>\n",
" <td>https://docstore.entsoe.eu/Documents/Publicati...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>AL</td>\n",
" <td>2011</td>\n",
" <td>Fossil fuels</td>\n",
" <td>0.0</td>\n",
" <td>ENTSO-E SOAF</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>Net capacity</td>\n",
" <td>Other association</td>\n",
" <td>https://docstore.entsoe.eu/Documents/Publicati...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>AL</td>\n",
" <td>2011</td>\n",
" <td>Hard coal</td>\n",
" <td>0.0</td>\n",
" <td>ENTSO-E SOAF</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>Net capacity</td>\n",
" <td>Other association</td>\n",
" <td>https://docstore.entsoe.eu/Documents/Publicati...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AL</td>\n",
" <td>2011</td>\n",
" <td>Hydro</td>\n",
" <td>0.0</td>\n",
" <td>ENTSO-E SOAF</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>Net capacity</td>\n",
" <td>Other association</td>\n",
" <td>https://docstore.entsoe.eu/Documents/Publicati...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AL</td>\n",
" <td>2011</td>\n",
" <td>Lignite</td>\n",
" <td>0.0</td>\n",
" <td>ENTSO-E SOAF</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>Net capacity</td>\n",
" <td>Other association</td>\n",
" <td>https://docstore.entsoe.eu/Documents/Publicati...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" country year technology capacity source \\\n",
"0 AL 2011 Biomass and biogas 0.0 ENTSO-E SOAF \n",
"1 AL 2011 Fossil fuels 0.0 ENTSO-E SOAF \n",
"2 AL 2011 Hard coal 0.0 ENTSO-E SOAF \n",
"3 AL 2011 Hydro 0.0 ENTSO-E SOAF \n",
"4 AL 2011 Lignite 0.0 ENTSO-E SOAF \n",
"\n",
" type capacity_definition source_type \\\n",
"0 Installed capacity in MW Net capacity Other association \n",
"1 Installed capacity in MW Net capacity Other association \n",
"2 Installed capacity in MW Net capacity Other association \n",
"3 Installed capacity in MW Net capacity Other association \n",
"4 Installed capacity in MW Net capacity Other association \n",
"\n",
" weblink \n",
"0 https://docstore.entsoe.eu/Documents/Publicati... \n",
"1 https://docstore.entsoe.eu/Documents/Publicati... \n",
"2 https://docstore.entsoe.eu/Documents/Publicati... \n",
"3 https://docstore.entsoe.eu/Documents/Publicati... \n",
"4 https://docstore.entsoe.eu/Documents/Publicati... "
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_soaf = func.restackData(soaf_unstacked)\n",
"\n",
"data_soaf.loc[data_soaf['capacity'] < 0, 'capacity'] = 0\n",
"\n",
"data_soaf['source'] = 'ENTSO-E SOAF'\n",
"data_soaf['type'] = 'Installed capacity in MW'\n",
"data_soaf['capacity_definition'] = 'Net capacity'\n",
"data_soaf['source_type'] = 'Other association'\n",
"data_soaf['weblink'] = url_entsoe\n",
"\n",
"data_soaf.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3.3.3 ENTSO-E Transparency Platform"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Year</th>\n",
" <th>Month</th>\n",
" <th>Day</th>\n",
" <th>DateTime</th>\n",
" <th>ResolutionCode</th>\n",
" <th>areacode</th>\n",
" <th>AreaTypeCode</th>\n",
" <th>AreaName</th>\n",
" <th>MapCode</th>\n",
" <th>ProductionType</th>\n",
" <th>AggregatedInstalledCapacity</th>\n",
" <th>DeletedFlag</th>\n",
" <th>UpdateTime</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2015</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>2015-01-01 00:00:00.000</td>\n",
" <td>P1Y</td>\n",
" <td>10YAT-APG------L</td>\n",
" <td>CTA</td>\n",
" <td>APG CA</td>\n",
" <td>AT</td>\n",
" <td>Other</td>\n",
" <td>22.8</td>\n",
" <td>0</td>\n",
" <td>2017-09-13 15:14:19</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2015</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>2015-01-01 00:00:00.000</td>\n",
" <td>P1Y</td>\n",
" <td>10YAT-APG------L</td>\n",
" <td>CTA</td>\n",
" <td>APG CA</td>\n",
" <td>AT</td>\n",
" <td>Wind Offshore</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>2017-09-13 15:14:19</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2015</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>2015-01-01 00:00:00.000</td>\n",
" <td>P1Y</td>\n",
" <td>10YAT-APG------L</td>\n",
" <td>CTA</td>\n",
" <td>APG CA</td>\n",
" <td>AT</td>\n",
" <td>Fossil Brown coal/Lignite</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>2017-09-13 15:14:19</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2015</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>2015-01-01 00:00:00.000</td>\n",
" <td>P1Y</td>\n",
" <td>10YAT-APG------L</td>\n",
" <td>CTA</td>\n",
" <td>APG CA</td>\n",
" <td>AT</td>\n",
" <td>Nuclear</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>2017-09-13 15:14:19</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2015</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>2015-01-01 00:00:00.000</td>\n",
" <td>P1Y</td>\n",
" <td>10YAT-APG------L</td>\n",
" <td>CTA</td>\n",
" <td>APG CA</td>\n",
" <td>AT</td>\n",
" <td>Fossil Hard coal</td>\n",
" <td>1171.0</td>\n",
" <td>0</td>\n",
" <td>2017-09-13 15:14:19</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Year Month Day DateTime ResolutionCode areacode \\\n",
"0 2015 1 1 2015-01-01 00:00:00.000 P1Y 10YAT-APG------L \n",
"1 2015 1 1 2015-01-01 00:00:00.000 P1Y 10YAT-APG------L \n",
"2 2015 1 1 2015-01-01 00:00:00.000 P1Y 10YAT-APG------L \n",
"3 2015 1 1 2015-01-01 00:00:00.000 P1Y 10YAT-APG------L \n",
"4 2015 1 1 2015-01-01 00:00:00.000 P1Y 10YAT-APG------L \n",
"\n",
" AreaTypeCode AreaName MapCode ProductionType \\\n",
"0 CTA APG CA AT Other \n",
"1 CTA APG CA AT Wind Offshore \n",
"2 CTA APG CA AT Fossil Brown coal/Lignite \n",
"3 CTA APG CA AT Nuclear \n",
"4 CTA APG CA AT Fossil Hard coal \n",
"\n",
" AggregatedInstalledCapacity DeletedFlag UpdateTime \n",
"0 22.8 0 2017-09-13 15:14:19 \n",
"1 0.0 0 2017-09-13 15:14:19 \n",
"2 0.0 0 2017-09-13 15:14:19 \n",
"3 0.0 0 2017-09-13 15:14:19 \n",
"4 1171.0 0 2017-09-13 15:14:19 "
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# file pattern for the single years\n",
"filenamepattern = '_1_InstalledGenerationCapacityAggregated.csv'\n",
"list_of_data_tables = [] # list to append\n",
"\n",
"# iterate over the years from 2015 to 2020\n",
"for i in range(2015,2021):\n",
" filepath = os.path.join('input',\n",
" 'ENTSO-E',\n",
" 'Transparency',\n",
" 'InstalledGenerationCapacityAggregated',\n",
" str(i) + filenamepattern)\n",
" \n",
" list_of_data_tables.append(pd.read_csv(filepath, delimiter=\"\\t\", encoding = \"UTF-16\"))\n",
"\n",
"# merge the datasets of the single of files into one pandas dataframe\n",
"data_transparency = pd.concat(list_of_data_tables, ignore_index=True)\n",
"\n",
"data_transparency.head()"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>technology</th>\n",
" <th>capacity</th>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Other</td>\n",
" <td>22.8</td>\n",
" <td>AT</td>\n",
" <td>2015</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Wind Offshore</td>\n",
" <td>0.0</td>\n",
" <td>AT</td>\n",
" <td>2015</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Fossil Brown coal/Lignite</td>\n",
" <td>0.0</td>\n",
" <td>AT</td>\n",
" <td>2015</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Nuclear</td>\n",
" <td>0.0</td>\n",
" <td>AT</td>\n",
" <td>2015</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Fossil Hard coal</td>\n",
" <td>1171.0</td>\n",
" <td>AT</td>\n",
" <td>2015</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" technology capacity country year\n",
"0 Other 22.8 AT 2015\n",
"1 Wind Offshore 0.0 AT 2015\n",
"2 Fossil Brown coal/Lignite 0.0 AT 2015\n",
"3 Nuclear 0.0 AT 2015\n",
"4 Fossil Hard coal 1171.0 AT 2015"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# rename columns according to the opsd standards\n",
"data_transparency.rename(columns={'ProductionType': 'technology',\n",
" 'AggregatedInstalledCapacity': 'capacity',\n",
" 'MapCode': 'country',\n",
" 'Year': 'year'}, inplace=True)\n",
"\n",
"# drop non relevant columns\n",
"data_transparency = data_transparency.filter(items=['technology','capacity','country','year'], axis=1)\n",
"# drop countries that are not part of opsd\n",
"data_transparency = data_transparency[data_transparency['country'].isin(data_opsd.country.unique())]\n",
"\n",
"data_transparency.head()"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>technology</th>\n",
" <th>capacity</th>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Other or unspecified energy sources</td>\n",
" <td>22.8</td>\n",
" <td>AT</td>\n",
" <td>2015</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Offshore</td>\n",
" <td>0.0</td>\n",
" <td>AT</td>\n",
" <td>2015</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Lignite</td>\n",
" <td>0.0</td>\n",
" <td>AT</td>\n",
" <td>2015</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Nuclear</td>\n",
" <td>0.0</td>\n",
" <td>AT</td>\n",
" <td>2015</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Hard coal</td>\n",
" <td>1171.0</td>\n",
" <td>AT</td>\n",
" <td>2015</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" technology capacity country year\n",
"0 Other or unspecified energy sources 22.8 AT 2015\n",
"1 Offshore 0.0 AT 2015\n",
"2 Lignite 0.0 AT 2015\n",
"3 Nuclear 0.0 AT 2015\n",
"4 Hard coal 1171.0 AT 2015"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# adapt energy source notation\n",
"dict_energy_source = {'Biomass': 'Biomass and biogas',\n",
" 'Fossil Brown coal/Lignite': 'Lignite',\n",
" 'Fossil Coal-derived gas': 'Mixed fossil fuels',\n",
" 'Fossil Gas': 'Natural gas',\n",
" 'Fossil Hard coal': 'Hard coal',\n",
" 'Fossil Oil': 'Oil',\n",
" 'Fossil Oil shale': 'Oil',\n",
" 'Fossil Peat': 'Other fossil fuels',\n",
" 'Hydro Pumped Storage': 'Pumped storage',\n",
" 'Hydro Run-of-river and poundage': 'Run-of-river',\n",
" 'Hydro Water Reservoir': 'Reservoir',\n",
" 'Other': 'Other or unspecified energy sources',\n",
" 'Other renewable': 'Differently categorized renewable energy sources',\n",
" 'Waste': 'Other bioenergy and renewable waste',\n",
" 'Wind Offshore': 'Offshore',\n",
" 'Wind Onshore': 'Onshore',\n",
" ' ': np.nan}\n",
"\n",
"data_transparency['technology'].replace(dict_energy_source, inplace=True)\n",
"\n",
"\n",
"data_transparency.head()"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>technology</th>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" <th>Biomass and biogas</th>\n",
" <th>Differently categorized renewable energy sources</th>\n",
" <th>Geothermal</th>\n",
" <th>Hard coal</th>\n",
" <th>Lignite</th>\n",
" <th>Marine</th>\n",
" <th>Mixed fossil fuels</th>\n",
" <th>Natural gas</th>\n",
" <th>...</th>\n",
" <th>Differently categorized solar</th>\n",
" <th>Differently categorized natural gas</th>\n",
" <th>Non-renewable waste</th>\n",
" <th>Differently categorized fossil fuels</th>\n",
" <th>Hydro</th>\n",
" <th>Wind</th>\n",
" <th>Bioenergy and renewable waste</th>\n",
" <th>Renewable energy sources</th>\n",
" <th>Fossil fuels</th>\n",
" <th>Total</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AT</td>\n",
" <td>2015</td>\n",
" <td>464.2</td>\n",
" <td>31.3</td>\n",
" <td>0.9</td>\n",
" <td>1171.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>4501.2</td>\n",
" <td>...</td>\n",
" <td>587.0</td>\n",
" <td>4501.2</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>11478.7</td>\n",
" <td>2120.9</td>\n",
" <td>608.3</td>\n",
" <td>14827.1</td>\n",
" <td>5761.15</td>\n",
" <td>20611.05</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>AT</td>\n",
" <td>2016</td>\n",
" <td>473.6</td>\n",
" <td>32.5</td>\n",
" <td>0.9</td>\n",
" <td>766.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>4465.8</td>\n",
" <td>...</td>\n",
" <td>723.0</td>\n",
" <td>4465.8</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>11510.7</td>\n",
" <td>2497.4</td>\n",
" <td>617.7</td>\n",
" <td>15382.2</td>\n",
" <td>5320.75</td>\n",
" <td>20725.75</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>AT</td>\n",
" <td>2017</td>\n",
" <td>473.8</td>\n",
" <td>32.5</td>\n",
" <td>0.9</td>\n",
" <td>598.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>4465.8</td>\n",
" <td>...</td>\n",
" <td>1031.0</td>\n",
" <td>4465.8</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>11946.8</td>\n",
" <td>2696.0</td>\n",
" <td>617.9</td>\n",
" <td>16325.1</td>\n",
" <td>5152.75</td>\n",
" <td>21500.65</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AT</td>\n",
" <td>2018</td>\n",
" <td>491.0</td>\n",
" <td>42.3</td>\n",
" <td>0.9</td>\n",
" <td>598.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>4467.7</td>\n",
" <td>...</td>\n",
" <td>1193.0</td>\n",
" <td>4467.7</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>11990.4</td>\n",
" <td>2887.0</td>\n",
" <td>640.8</td>\n",
" <td>16754.4</td>\n",
" <td>5154.65</td>\n",
" <td>21931.85</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AT</td>\n",
" <td>2019</td>\n",
" <td>499.8</td>\n",
" <td>42.3</td>\n",
" <td>0.9</td>\n",
" <td>598.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>4463.2</td>\n",
" <td>...</td>\n",
" <td>1192.7</td>\n",
" <td>4463.2</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>11117.5</td>\n",
" <td>3034.6</td>\n",
" <td>649.8</td>\n",
" <td>16037.8</td>\n",
" <td>5150.15</td>\n",
" <td>21210.75</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 31 columns</p>\n",
"</div>"
],
"text/plain": [
"technology country year Biomass and biogas \\\n",
"0 AT 2015 464.2 \n",
"1 AT 2016 473.6 \n",
"2 AT 2017 473.8 \n",
"3 AT 2018 491.0 \n",
"4 AT 2019 499.8 \n",
"\n",
"technology Differently categorized renewable energy sources Geothermal \\\n",
"0 31.3 0.9 \n",
"1 32.5 0.9 \n",
"2 32.5 0.9 \n",
"3 42.3 0.9 \n",
"4 42.3 0.9 \n",
"\n",
"technology Hard coal Lignite Marine Mixed fossil fuels Natural gas ... \\\n",
"0 1171.0 0.0 0.0 0.0 4501.2 ... \n",
"1 766.0 0.0 0.0 0.0 4465.8 ... \n",
"2 598.0 0.0 0.0 0.0 4465.8 ... \n",
"3 598.0 0.0 0.0 0.0 4467.7 ... \n",
"4 598.0 0.0 0.0 0.0 4463.2 ... \n",
"\n",
"technology Differently categorized solar \\\n",
"0 587.0 \n",
"1 723.0 \n",
"2 1031.0 \n",
"3 1193.0 \n",
"4 1192.7 \n",
"\n",
"technology Differently categorized natural gas Non-renewable waste \\\n",
"0 4501.2 0 \n",
"1 4465.8 0 \n",
"2 4465.8 0 \n",
"3 4467.7 0 \n",
"4 4463.2 0 \n",
"\n",
"technology Differently categorized fossil fuels Hydro Wind \\\n",
"0 0 11478.7 2120.9 \n",
"1 0 11510.7 2497.4 \n",
"2 0 11946.8 2696.0 \n",
"3 0 11990.4 2887.0 \n",
"4 0 11117.5 3034.6 \n",
"\n",
"technology Bioenergy and renewable waste Renewable energy sources \\\n",
"0 608.3 14827.1 \n",
"1 617.7 15382.2 \n",
"2 617.9 16325.1 \n",
"3 640.8 16754.4 \n",
"4 649.8 16037.8 \n",
"\n",
"technology Fossil fuels Total \n",
"0 5761.15 20611.05 \n",
"1 5320.75 20725.75 \n",
"2 5152.75 21500.65 \n",
"3 5154.65 21931.85 \n",
"4 5150.15 21210.75 \n",
"\n",
"[5 rows x 31 columns]"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# add missing categories\n",
"transparency_pivot = data_transparency.pivot_table(values='capacity',\n",
" index=['country','year'],\n",
" columns='technology')\n",
"\n",
"# technology level\n",
"transparency_pivot['Differently categorized solar'] = transparency_pivot['Solar']\n",
"transparency_pivot['Differently categorized natural gas'] = transparency_pivot['Natural gas']\n",
"transparency_pivot['Non-renewable waste'] = 0\n",
"transparency_pivot['Differently categorized fossil fuels'] = 0\n",
"\n",
"# level 3\n",
"hydro_arr = ['Pumped storage', 'Reservoir', 'Run-of-river']\n",
"transparency_pivot['Hydro'] = transparency_pivot[hydro_arr].sum(axis=1)\n",
"\n",
"wind_arr = ['Onshore', 'Offshore']\n",
"transparency_pivot['Wind'] = transparency_pivot[wind_arr].sum(axis=1)\n",
"\n",
"# level 2\n",
"bio_arr = ['Biomass and biogas', 'Other bioenergy and renewable waste']\n",
"transparency_pivot['Bioenergy and renewable waste'] = transparency_pivot[bio_arr].sum(axis=1)\n",
"\n",
"#level 1\n",
"res_arr = ['Hydro', 'Wind', 'Solar', 'Geothermal', 'Marine',\n",
" 'Bioenergy and renewable waste', 'Differently categorized renewable energy sources']\n",
"transparency_pivot['Renewable energy sources'] = transparency_pivot[res_arr].sum(axis=1)\n",
"\n",
"fossil_arr = ['Lignite', 'Hard coal', 'Oil', 'Natural gas', 'Mixed fossil fuels', 'Other fossil fuels']\n",
"transparency_pivot['Fossil fuels'] = transparency_pivot[fossil_arr].sum(axis=1)\n",
"\n",
"# level 0\n",
"total_arr = ['Fossil fuels','Nuclear','Renewable energy sources', 'Other or unspecified energy sources']\n",
"transparency_pivot['Total'] = transparency_pivot[total_arr].sum(axis=1)\n",
"\n",
"transparency_pivot.reset_index(inplace=True)\n",
"\n",
"transparency_pivot.head()"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" <th>technology</th>\n",
" <th>capacity</th>\n",
" <th>source</th>\n",
" <th>source_type</th>\n",
" <th>capacity_definition</th>\n",
" <th>type</th>\n",
" <th>weblink</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AT</td>\n",
" <td>2015</td>\n",
" <td>Biomass and biogas</td>\n",
" <td>464.2</td>\n",
" <td>ENTSO-E Transparency Platform</td>\n",
" <td>Other association</td>\n",
" <td>Net capacity</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>https://transparency.entsoe.eu/generation/r2/i...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>AT</td>\n",
" <td>2016</td>\n",
" <td>Biomass and biogas</td>\n",
" <td>473.6</td>\n",
" <td>ENTSO-E Transparency Platform</td>\n",
" <td>Other association</td>\n",
" <td>Net capacity</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>https://transparency.entsoe.eu/generation/r2/i...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>AT</td>\n",
" <td>2017</td>\n",
" <td>Biomass and biogas</td>\n",
" <td>473.8</td>\n",
" <td>ENTSO-E Transparency Platform</td>\n",
" <td>Other association</td>\n",
" <td>Net capacity</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>https://transparency.entsoe.eu/generation/r2/i...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AT</td>\n",
" <td>2018</td>\n",
" <td>Biomass and biogas</td>\n",
" <td>491.0</td>\n",
" <td>ENTSO-E Transparency Platform</td>\n",
" <td>Other association</td>\n",
" <td>Net capacity</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>https://transparency.entsoe.eu/generation/r2/i...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AT</td>\n",
" <td>2019</td>\n",
" <td>Biomass and biogas</td>\n",
" <td>499.8</td>\n",
" <td>ENTSO-E Transparency Platform</td>\n",
" <td>Other association</td>\n",
" <td>Net capacity</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>https://transparency.entsoe.eu/generation/r2/i...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" country year technology capacity source \\\n",
"0 AT 2015 Biomass and biogas 464.2 ENTSO-E Transparency Platform \n",
"1 AT 2016 Biomass and biogas 473.6 ENTSO-E Transparency Platform \n",
"2 AT 2017 Biomass and biogas 473.8 ENTSO-E Transparency Platform \n",
"3 AT 2018 Biomass and biogas 491.0 ENTSO-E Transparency Platform \n",
"4 AT 2019 Biomass and biogas 499.8 ENTSO-E Transparency Platform \n",
"\n",
" source_type capacity_definition type \\\n",
"0 Other association Net capacity Installed capacity in MW \n",
"1 Other association Net capacity Installed capacity in MW \n",
"2 Other association Net capacity Installed capacity in MW \n",
"3 Other association Net capacity Installed capacity in MW \n",
"4 Other association Net capacity Installed capacity in MW \n",
"\n",
" weblink \n",
"0 https://transparency.entsoe.eu/generation/r2/i... \n",
"1 https://transparency.entsoe.eu/generation/r2/i... \n",
"2 https://transparency.entsoe.eu/generation/r2/i... \n",
"3 https://transparency.entsoe.eu/generation/r2/i... \n",
"4 https://transparency.entsoe.eu/generation/r2/i... "
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_transparency = pd.melt(transparency_pivot,\n",
" id_vars=['country', 'year'],\n",
" var_name='technology',\n",
" value_name='capacity')\n",
"\n",
"data_transparency = data_transparency.loc[data_transparency[\"year\"] < 2020, :]\n",
"\n",
"data_transparency['source'] = 'ENTSO-E Transparency Platform'\n",
"data_transparency['source_type'] = 'Other association'\n",
"data_transparency['capacity_definition'] = 'Net capacity'\n",
"data_transparency['type'] = 'Installed capacity in MW'\n",
"data_transparency['weblink'] = ('https://transparency.entsoe.eu/generation'\n",
" '/r2/installedGenerationCapacityAggregation/show')\n",
"\n",
"data_transparency.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3.3.4 ENTSO-E Power Statistics"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>technology</th>\n",
" <th>country</th>\n",
" <th>capacity</th>\n",
" <th>year</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Non-Renewable</td>\n",
" <td>AL</td>\n",
" <td>0</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Nuclear</td>\n",
" <td>AL</td>\n",
" <td>NaN</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Non-renwable hydro</td>\n",
" <td>AL</td>\n",
" <td>0</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Of which hydro pure pumped storage</td>\n",
" <td>AL</td>\n",
" <td>NaN</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Of which Hydro mixed pumped storage (non renew...</td>\n",
" <td>AL</td>\n",
" <td>NaN</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" technology country capacity year\n",
"0 Non-Renewable AL 0 2014\n",
"1 Nuclear AL NaN 2014\n",
"2 Non-renwable hydro AL 0 2014\n",
"3 Of which hydro pure pumped storage AL NaN 2014\n",
"4 Of which Hydro mixed pumped storage (non renew... AL NaN 2014"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"row_of_year = {2014: 9,\n",
" 2015: 53,\n",
" 2016: 97,\n",
" 2017: 141,\n",
" 2018: 185}\n",
"\n",
"dataframes = []\n",
"\n",
"for year, row in row_of_year.items():\n",
" # read the dataframe for each year\n",
" power_statistics_raw = pd.read_excel(os.path.join('input',\n",
" 'ENTSO-E',\n",
" 'Power Statistics',\n",
" 'NGC.xlsx'),\n",
" header=[0,1], \n",
" sheet_name='NGC',\n",
" skiprows=row,\n",
" nrows=42)\n",
" \n",
" # drop non relevant columns\n",
" power_statistics_raw.drop(columns='Coverage ratio in %', level=1, inplace=True)\n",
" power_statistics_raw.drop(columns=['Unnamed: 1_level_1','Unnamed: 2_level_1','Unnamed: 3_level_1','Unnamed: 4_level_1'], level=1, inplace=True)\n",
"\n",
" # get rid of multi index\n",
" df = power_statistics_raw.set_index(year).stack().reset_index().drop('level_1', axis=1)\n",
" \n",
" # remove leftovers of multi index in the index column\n",
" df[\"technology\"] = df[year].apply(lambda x: x[0])\n",
" df.drop(columns=year, inplace=True)\n",
" \n",
" # stack df to the opsd standard format\n",
" stacked_df = df.melt(id_vars='technology', var_name='country', value_name='capacity')\n",
" \n",
" # add information about the year\n",
" stacked_df['year'] = year\n",
" \n",
" # append to the main list of dataframes\n",
" dataframes.append(stacked_df)\n",
" \n",
"power_statistics = pd.concat(dataframes)\n",
"power_statistics.head()"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>technology</th>\n",
" <th>country</th>\n",
" <th>capacity</th>\n",
" <th>year</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>41</th>\n",
" <td>Nuclear</td>\n",
" <td>AT</td>\n",
" <td>0.0</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" <tr>\n",
" <th>43</th>\n",
" <td>Of which hydro pure pumped storage</td>\n",
" <td>AT</td>\n",
" <td>0.0</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" <tr>\n",
" <th>44</th>\n",
" <td>Of which Hydro mixed pumped storage (non renew...</td>\n",
" <td>AT</td>\n",
" <td>0.0</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" <tr>\n",
" <th>46</th>\n",
" <td>Of which Fossil Brown coal/Lignite</td>\n",
" <td>AT</td>\n",
" <td>0.0</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" <tr>\n",
" <th>47</th>\n",
" <td>Of which Fossil Coal-derived gas</td>\n",
" <td>AT</td>\n",
" <td>0.0</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" technology country capacity year\n",
"41 Nuclear AT 0.0 2014\n",
"43 Of which hydro pure pumped storage AT 0.0 2014\n",
"44 Of which Hydro mixed pumped storage (non renew... AT 0.0 2014\n",
"46 Of which Fossil Brown coal/Lignite AT 0.0 2014\n",
"47 Of which Fossil Coal-derived gas AT 0.0 2014"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# drop countries that are not covered in opsd\n",
"opsd_countries = data_opsd.country.unique()\n",
"drop_list_country = power_statistics.loc[~power_statistics['country'].isin(opsd_countries)].index.to_list()\n",
"power_statistics.drop(drop_list_country, inplace=True)\n",
"\n",
"# technology classes to be dropped\n",
"tech_to_drop = ['Non-Renewable', 'Fossil fuels', 'Renewable','Non-renwable hydro',\n",
" 'Total Waste', 'Bio', 'Renewable Hydro', 'Comments', 'Total NGC']\n",
"\n",
"drop_list_tech = power_statistics.loc[power_statistics['technology'].isin(tech_to_drop)].index.to_list()\n",
"power_statistics.drop(drop_list_tech, inplace=True)\n",
"\n",
"# replace string with values that can be used in math operations \n",
"power_statistics['capacity'].replace(to_replace='Not Expected', value=0, inplace=True)\n",
"power_statistics['capacity'].replace(to_replace='Not Available', value=np.nan, inplace=True)\n",
"\n",
"power_statistics.head()"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>technology</th>\n",
" <th>country</th>\n",
" <th>capacity</th>\n",
" <th>year</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>41</th>\n",
" <td>Nuclear</td>\n",
" <td>AT</td>\n",
" <td>0.0</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" <tr>\n",
" <th>43</th>\n",
" <td>Pumped storage</td>\n",
" <td>AT</td>\n",
" <td>0.0</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" <tr>\n",
" <th>44</th>\n",
" <td>Pumped storage</td>\n",
" <td>AT</td>\n",
" <td>0.0</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" <tr>\n",
" <th>46</th>\n",
" <td>Lignite</td>\n",
" <td>AT</td>\n",
" <td>0.0</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" <tr>\n",
" <th>47</th>\n",
" <td>Differently categorized fossil fuels</td>\n",
" <td>AT</td>\n",
" <td>0.0</td>\n",
" <td>2014</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" technology country capacity year\n",
"41 Nuclear AT 0.0 2014\n",
"43 Pumped storage AT 0.0 2014\n",
"44 Pumped storage AT 0.0 2014\n",
"46 Lignite AT 0.0 2014\n",
"47 Differently categorized fossil fuels AT 0.0 2014"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Not included because already categorized in OPSD standard:\n",
"# Nuclear, Solar, Geothermal, Wind\n",
"dict_energy_source = {'Of which hydro pure pumped storage':'Pumped storage',\n",
" 'Of which Hydro mixed pumped storage (non renewable part)':'Pumped storage',\n",
" 'Of which Fossil Brown coal/Lignite':'Lignite',\n",
" 'Of which Fossil Coal-derived gas':'Differently categorized fossil fuels',\n",
" 'Of which Fossil Gas':'Natural gas',\n",
" 'Of which Fossil Hard coal':'Hard coal', \n",
" 'Of which Fossil Oil':'Oil',\n",
" 'Of which Fossil Oil shale':'Oil', \n",
" 'Of which Fossil Peat':'Differently categorized fossil fuels',\n",
" 'Of which Mixed fuels':'Mixed fossil fuels', \n",
" 'Of which Other fossil fuels':'Other fossil fuels',\n",
" 'Non-renewable Waste':'Non-renewable waste', \n",
" 'Other non-renewable':'Differently categorized fossil fuels', \n",
" 'Of which Wind offshore':'Offshore', \n",
" 'Of which Wind onshore':'Onshore', \n",
" 'Of which Solar PV':'Photovoltaics', \n",
" 'Of which Solar Thermal':'Differently categorized solar', \n",
" 'Of which Biomass':'Biomass and biogas', \n",
" 'Of which Biogas':'Biomass and biogas', \n",
" 'Renewable Waste':'Other bioenergy and renewable waste', \n",
" 'Of which Hydro Pure storage':'Reservoir',\n",
" 'Of which Hydro Run-of-river and pondage':'Run-of-river',\n",
" 'Of which Hydro mixed pumped storage (renewable part)':'Pumped storage',\n",
" 'Of which Hydro Marine (tidal/wave)':'Marine',\n",
" 'Other renewable (not listed)':'Differently categorized renewable energy sources',\n",
" 'Non identified (other not listed)':'Other or unspecified energy sources', \n",
" 'Total Hydro':'Hydro'}\n",
"\n",
"power_statistics[\"technology\"].replace(dict_energy_source, inplace=True)\n",
"power_statistics.head()"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>technology</th>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" <th>Biomass and biogas</th>\n",
" <th>Differently categorized fossil fuels</th>\n",
" <th>Differently categorized renewable energy sources</th>\n",
" <th>Differently categorized solar</th>\n",
" <th>Geothermal</th>\n",
" <th>Hard coal</th>\n",
" <th>Hydro</th>\n",
" <th>Lignite</th>\n",
" <th>...</th>\n",
" <th>Onshore</th>\n",
" <th>Other bioenergy and renewable waste</th>\n",
" <th>Other fossil fuels</th>\n",
" <th>Other or unspecified energy sources</th>\n",
" <th>Photovoltaics</th>\n",
" <th>Pumped storage</th>\n",
" <th>Reservoir</th>\n",
" <th>Run-of-river</th>\n",
" <th>Solar</th>\n",
" <th>Wind</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AT</td>\n",
" <td>2014</td>\n",
" <td>302.685</td>\n",
" <td>29.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>1171.0</td>\n",
" <td>13568.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>2110.0</td>\n",
" <td>21.96</td>\n",
" <td>414.0</td>\n",
" <td>0.0</td>\n",
" <td>586.12</td>\n",
" <td>2656.333333</td>\n",
" <td>0.0</td>\n",
" <td>5599.0</td>\n",
" <td>586.12</td>\n",
" <td>2110.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>AT</td>\n",
" <td>2015</td>\n",
" <td>298.000</td>\n",
" <td>30.0</td>\n",
" <td>NaN</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>1171.0</td>\n",
" <td>13656.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>2489.0</td>\n",
" <td>23.00</td>\n",
" <td>414.0</td>\n",
" <td>NaN</td>\n",
" <td>732.00</td>\n",
" <td>3997.000000</td>\n",
" <td>NaN</td>\n",
" <td>5662.0</td>\n",
" <td>732.00</td>\n",
" <td>2489.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>AT</td>\n",
" <td>2016</td>\n",
" <td>286.000</td>\n",
" <td>35.0</td>\n",
" <td>NaN</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>598.0</td>\n",
" <td>14116.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>2730.0</td>\n",
" <td>22.00</td>\n",
" <td>414.0</td>\n",
" <td>NaN</td>\n",
" <td>1031.00</td>\n",
" <td>4212.000000</td>\n",
" <td>NaN</td>\n",
" <td>5692.0</td>\n",
" <td>1031.00</td>\n",
" <td>2730.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AT</td>\n",
" <td>2017</td>\n",
" <td>286.000</td>\n",
" <td>35.0</td>\n",
" <td>NaN</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>598.0</td>\n",
" <td>14116.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>2730.0</td>\n",
" <td>22.00</td>\n",
" <td>414.0</td>\n",
" <td>NaN</td>\n",
" <td>1031.00</td>\n",
" <td>4212.000000</td>\n",
" <td>NaN</td>\n",
" <td>5692.0</td>\n",
" <td>1031.00</td>\n",
" <td>2730.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AT</td>\n",
" <td>2018</td>\n",
" <td>293.000</td>\n",
" <td>35.0</td>\n",
" <td>NaN</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>598.0</td>\n",
" <td>14150.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>2887.0</td>\n",
" <td>22.00</td>\n",
" <td>414.0</td>\n",
" <td>NaN</td>\n",
" <td>1193.00</td>\n",
" <td>4218.000000</td>\n",
" <td>NaN</td>\n",
" <td>5714.0</td>\n",
" <td>1193.00</td>\n",
" <td>2887.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 27 columns</p>\n",
"</div>"
],
"text/plain": [
"technology country year Biomass and biogas \\\n",
"0 AT 2014 302.685 \n",
"1 AT 2015 298.000 \n",
"2 AT 2016 286.000 \n",
"3 AT 2017 286.000 \n",
"4 AT 2018 293.000 \n",
"\n",
"technology Differently categorized fossil fuels \\\n",
"0 29.0 \n",
"1 30.0 \n",
"2 35.0 \n",
"3 35.0 \n",
"4 35.0 \n",
"\n",
"technology Differently categorized renewable energy sources \\\n",
"0 0.0 \n",
"1 NaN \n",
"2 NaN \n",
"3 NaN \n",
"4 NaN \n",
"\n",
"technology Differently categorized solar Geothermal Hard coal Hydro \\\n",
"0 0.0 1.0 1171.0 13568.0 \n",
"1 0.0 1.0 1171.0 13656.0 \n",
"2 0.0 1.0 598.0 14116.0 \n",
"3 0.0 1.0 598.0 14116.0 \n",
"4 0.0 1.0 598.0 14150.0 \n",
"\n",
"technology Lignite ... Onshore Other bioenergy and renewable waste \\\n",
"0 0.0 ... 2110.0 21.96 \n",
"1 0.0 ... 2489.0 23.00 \n",
"2 0.0 ... 2730.0 22.00 \n",
"3 0.0 ... 2730.0 22.00 \n",
"4 0.0 ... 2887.0 22.00 \n",
"\n",
"technology Other fossil fuels Other or unspecified energy sources \\\n",
"0 414.0 0.0 \n",
"1 414.0 NaN \n",
"2 414.0 NaN \n",
"3 414.0 NaN \n",
"4 414.0 NaN \n",
"\n",
"technology Photovoltaics Pumped storage Reservoir Run-of-river Solar \\\n",
"0 586.12 2656.333333 0.0 5599.0 586.12 \n",
"1 732.00 3997.000000 NaN 5662.0 732.00 \n",
"2 1031.00 4212.000000 NaN 5692.0 1031.00 \n",
"3 1031.00 4212.000000 NaN 5692.0 1031.00 \n",
"4 1193.00 4218.000000 NaN 5714.0 1193.00 \n",
"\n",
"technology Wind \n",
"0 2110.0 \n",
"1 2489.0 \n",
"2 2730.0 \n",
"3 2730.0 \n",
"4 2887.0 \n",
"\n",
"[5 rows x 27 columns]"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"powerstats_pivot = power_statistics.pivot_table(values='capacity',\n",
" index=['country','year'],\n",
" columns='technology').reset_index()\n",
"powerstats_pivot.head()"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th>technology</th>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" <th>Biomass and biogas</th>\n",
" <th>Differently categorized fossil fuels</th>\n",
" <th>Differently categorized renewable energy sources</th>\n",
" <th>Differently categorized solar</th>\n",
" <th>Geothermal</th>\n",
" <th>Hard coal</th>\n",
" <th>Hydro</th>\n",
" <th>Lignite</th>\n",
" <th>...</th>\n",
" <th>Pumped storage</th>\n",
" <th>Reservoir</th>\n",
" <th>Run-of-river</th>\n",
" <th>Solar</th>\n",
" <th>Wind</th>\n",
" <th>Differently categorized natural gas</th>\n",
" <th>Bioenergy and renewable waste</th>\n",
" <th>Fossil fuels</th>\n",
" <th>Renewable energy sources</th>\n",
" <th>Total</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AT</td>\n",
" <td>2014</td>\n",
" <td>302.685</td>\n",
" <td>29.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>1171.0</td>\n",
" <td>13568.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>2656.333333</td>\n",
" <td>0.0</td>\n",
" <td>5599.0</td>\n",
" <td>586.12</td>\n",
" <td>2110.0</td>\n",
" <td>4888.0</td>\n",
" <td>324.645</td>\n",
" <td>7105.0</td>\n",
" <td>16589.765</td>\n",
" <td>23694.765</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>AT</td>\n",
" <td>2015</td>\n",
" <td>298.000</td>\n",
" <td>30.0</td>\n",
" <td>NaN</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>1171.0</td>\n",
" <td>13656.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>3997.000000</td>\n",
" <td>NaN</td>\n",
" <td>5662.0</td>\n",
" <td>732.00</td>\n",
" <td>2489.0</td>\n",
" <td>4820.0</td>\n",
" <td>321.000</td>\n",
" <td>7002.0</td>\n",
" <td>17199.000</td>\n",
" <td>24201.000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>AT</td>\n",
" <td>2016</td>\n",
" <td>286.000</td>\n",
" <td>35.0</td>\n",
" <td>NaN</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>598.0</td>\n",
" <td>14116.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>4212.000000</td>\n",
" <td>NaN</td>\n",
" <td>5692.0</td>\n",
" <td>1031.00</td>\n",
" <td>2730.0</td>\n",
" <td>4841.0</td>\n",
" <td>308.000</td>\n",
" <td>6407.0</td>\n",
" <td>18186.000</td>\n",
" <td>24593.000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AT</td>\n",
" <td>2017</td>\n",
" <td>286.000</td>\n",
" <td>35.0</td>\n",
" <td>NaN</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>598.0</td>\n",
" <td>14116.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>4212.000000</td>\n",
" <td>NaN</td>\n",
" <td>5692.0</td>\n",
" <td>1031.00</td>\n",
" <td>2730.0</td>\n",
" <td>4841.0</td>\n",
" <td>308.000</td>\n",
" <td>6407.0</td>\n",
" <td>18186.000</td>\n",
" <td>24593.000</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AT</td>\n",
" <td>2018</td>\n",
" <td>293.000</td>\n",
" <td>35.0</td>\n",
" <td>NaN</td>\n",
" <td>0.0</td>\n",
" <td>1.0</td>\n",
" <td>598.0</td>\n",
" <td>14150.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>4218.000000</td>\n",
" <td>NaN</td>\n",
" <td>5714.0</td>\n",
" <td>1193.00</td>\n",
" <td>2887.0</td>\n",
" <td>4853.0</td>\n",
" <td>315.000</td>\n",
" <td>6420.0</td>\n",
" <td>18546.000</td>\n",
" <td>24966.000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 32 columns</p>\n",
"</div>"
],
"text/plain": [
"technology country year Biomass and biogas \\\n",
"0 AT 2014 302.685 \n",
"1 AT 2015 298.000 \n",
"2 AT 2016 286.000 \n",
"3 AT 2017 286.000 \n",
"4 AT 2018 293.000 \n",
"\n",
"technology Differently categorized fossil fuels \\\n",
"0 29.0 \n",
"1 30.0 \n",
"2 35.0 \n",
"3 35.0 \n",
"4 35.0 \n",
"\n",
"technology Differently categorized renewable energy sources \\\n",
"0 0.0 \n",
"1 NaN \n",
"2 NaN \n",
"3 NaN \n",
"4 NaN \n",
"\n",
"technology Differently categorized solar Geothermal Hard coal Hydro \\\n",
"0 0.0 1.0 1171.0 13568.0 \n",
"1 0.0 1.0 1171.0 13656.0 \n",
"2 0.0 1.0 598.0 14116.0 \n",
"3 0.0 1.0 598.0 14116.0 \n",
"4 0.0 1.0 598.0 14150.0 \n",
"\n",
"technology Lignite ... Pumped storage Reservoir Run-of-river Solar \\\n",
"0 0.0 ... 2656.333333 0.0 5599.0 586.12 \n",
"1 0.0 ... 3997.000000 NaN 5662.0 732.00 \n",
"2 0.0 ... 4212.000000 NaN 5692.0 1031.00 \n",
"3 0.0 ... 4212.000000 NaN 5692.0 1031.00 \n",
"4 0.0 ... 4218.000000 NaN 5714.0 1193.00 \n",
"\n",
"technology Wind Differently categorized natural gas \\\n",
"0 2110.0 4888.0 \n",
"1 2489.0 4820.0 \n",
"2 2730.0 4841.0 \n",
"3 2730.0 4841.0 \n",
"4 2887.0 4853.0 \n",
"\n",
"technology Bioenergy and renewable waste Fossil fuels \\\n",
"0 324.645 7105.0 \n",
"1 321.000 7002.0 \n",
"2 308.000 6407.0 \n",
"3 308.000 6407.0 \n",
"4 315.000 6420.0 \n",
"\n",
"technology Renewable energy sources Total \n",
"0 16589.765 23694.765 \n",
"1 17199.000 24201.000 \n",
"2 18186.000 24593.000 \n",
"3 18186.000 24593.000 \n",
"4 18546.000 24966.000 \n",
"\n",
"[5 rows x 32 columns]"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# technology level\n",
"powerstats_pivot['Differently categorized natural gas'] = powerstats_pivot['Natural gas']\n",
"\n",
"# level 2\n",
"powerstats_pivot['Bioenergy and renewable waste'] = (\n",
" powerstats_pivot['Biomass and biogas'] +\n",
" powerstats_pivot['Other bioenergy and renewable waste'])\n",
"\n",
"#level 1\n",
"fossil_techs = ['Lignite', 'Hard coal', 'Oil', 'Natural gas', 'Non-renewable waste',\n",
" 'Mixed fossil fuels', 'Other fossil fuels', 'Differently categorized fossil fuels']\n",
"\n",
"powerstats_pivot['Fossil fuels'] = powerstats_pivot[fossil_techs].sum(axis=1)\n",
"\n",
"res_tech = ['Hydro', 'Wind', 'Solar', 'Geothermal', 'Marine', 'Bioenergy and renewable waste',\n",
" 'Differently categorized renewable energy sources']\n",
"\n",
"powerstats_pivot['Renewable energy sources'] = powerstats_pivot[res_tech].sum(axis=1)\n",
"\n",
"total_arr = ['Fossil fuels','Nuclear','Renewable energy sources']\n",
"powerstats_pivot['Total'] = powerstats_pivot[total_arr].sum(axis=1)\n",
"\n",
"powerstats_pivot.head()"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>country</th>\n",
" <th>year</th>\n",
" <th>technology</th>\n",
" <th>capacity</th>\n",
" <th>source</th>\n",
" <th>source_type</th>\n",
" <th>capacity_definition</th>\n",
" <th>type</th>\n",
" <th>weblink</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>AT</td>\n",
" <td>2014</td>\n",
" <td>Biomass and biogas</td>\n",
" <td>302.685</td>\n",
" <td>ENTSO-E Power Statistics</td>\n",
" <td>Other association</td>\n",
" <td>Net capacity</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>https://www.entsoe.eu/data/power-stats/net-gen...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>AT</td>\n",
" <td>2015</td>\n",
" <td>Biomass and biogas</td>\n",
" <td>298.000</td>\n",
" <td>ENTSO-E Power Statistics</td>\n",
" <td>Other association</td>\n",
" <td>Net capacity</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>https://www.entsoe.eu/data/power-stats/net-gen...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>AT</td>\n",
" <td>2016</td>\n",
" <td>Biomass and biogas</td>\n",
" <td>286.000</td>\n",
" <td>ENTSO-E Power Statistics</td>\n",
" <td>Other association</td>\n",
" <td>Net capacity</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>https://www.entsoe.eu/data/power-stats/net-gen...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>AT</td>\n",
" <td>2017</td>\n",
" <td>Biomass and biogas</td>\n",
" <td>286.000</td>\n",
" <td>ENTSO-E Power Statistics</td>\n",
" <td>Other association</td>\n",
" <td>Net capacity</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>https://www.entsoe.eu/data/power-stats/net-gen...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>AT</td>\n",
" <td>2018</td>\n",
" <td>Biomass and biogas</td>\n",
" <td>293.000</td>\n",
" <td>ENTSO-E Power Statistics</td>\n",
" <td>Other association</td>\n",
" <td>Net capacity</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>https://www.entsoe.eu/data/power-stats/net-gen...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" country year technology capacity source \\\n",
"0 AT 2014 Biomass and biogas 302.685 ENTSO-E Power Statistics \n",
"1 AT 2015 Biomass and biogas 298.000 ENTSO-E Power Statistics \n",
"2 AT 2016 Biomass and biogas 286.000 ENTSO-E Power Statistics \n",
"3 AT 2017 Biomass and biogas 286.000 ENTSO-E Power Statistics \n",
"4 AT 2018 Biomass and biogas 293.000 ENTSO-E Power Statistics \n",
"\n",
" source_type capacity_definition type \\\n",
"0 Other association Net capacity Installed capacity in MW \n",
"1 Other association Net capacity Installed capacity in MW \n",
"2 Other association Net capacity Installed capacity in MW \n",
"3 Other association Net capacity Installed capacity in MW \n",
"4 Other association Net capacity Installed capacity in MW \n",
"\n",
" weblink \n",
"0 https://www.entsoe.eu/data/power-stats/net-gen... \n",
"1 https://www.entsoe.eu/data/power-stats/net-gen... \n",
"2 https://www.entsoe.eu/data/power-stats/net-gen... \n",
"3 https://www.entsoe.eu/data/power-stats/net-gen... \n",
"4 https://www.entsoe.eu/data/power-stats/net-gen... "
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_power_statistics = powerstats_pivot.melt(id_vars=['country', 'year'],\n",
" var_name='technology',\n",
" value_name='capacity')\n",
"\n",
"data_power_statistics['source'] = 'ENTSO-E Power Statistics'\n",
"data_power_statistics['source_type'] = 'Other association'\n",
"data_power_statistics['capacity_definition'] = 'Net capacity'\n",
"data_power_statistics['type'] = 'Installed capacity in MW'\n",
"data_power_statistics['weblink'] = ('https://www.entsoe.eu/data/'\n",
" 'power-stats/net-gen-capacity/')\n",
"\n",
"data_power_statistics.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3.4 Merge data sources"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>technology</th>\n",
" <th>source</th>\n",
" <th>source_type</th>\n",
" <th>weblink</th>\n",
" <th>year</th>\n",
" <th>type</th>\n",
" <th>country</th>\n",
" <th>capacity_definition</th>\n",
" <th>capacity</th>\n",
" <th>comment</th>\n",
" <th>energy_source_level_0</th>\n",
" <th>energy_source_level_1</th>\n",
" <th>energy_source_level_2</th>\n",
" <th>energy_source_level_3</th>\n",
" <th>technology_level</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Fossil fuels</td>\n",
" <td>e-control</td>\n",
" <td>Regulatory Authority</td>\n",
" <td>https://www.e-control.at/documents/1785851/181...</td>\n",
" <td>2014</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>AT</td>\n",
" <td>Gross capacity</td>\n",
" <td>7243.634</td>\n",
" <td></td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Fossil fuels</td>\n",
" <td>e-control</td>\n",
" <td>Regulatory Authority</td>\n",
" <td>https://www.e-control.at/documents/1785851/181...</td>\n",
" <td>2015</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>AT</td>\n",
" <td>Gross capacity</td>\n",
" <td>7059.065</td>\n",
" <td></td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Fossil fuels</td>\n",
" <td>e-control</td>\n",
" <td>Regulatory Authority</td>\n",
" <td>https://www.e-control.at/documents/1785851/181...</td>\n",
" <td>2016</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>AT</td>\n",
" <td>Gross capacity</td>\n",
" <td>7323.000</td>\n",
" <td></td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Fossil fuels</td>\n",
" <td>e-control</td>\n",
" <td>Regulatory Authority</td>\n",
" <td>https://www.e-control.at/documents/1785851/181...</td>\n",
" <td>2017</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>AT</td>\n",
" <td>Gross capacity</td>\n",
" <td>6469.000</td>\n",
" <td></td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Fossil fuels</td>\n",
" <td>e-control</td>\n",
" <td>Regulatory Authority</td>\n",
" <td>https://www.e-control.at/documents/1785851/181...</td>\n",
" <td>2018</td>\n",
" <td>Installed capacity in MW</td>\n",
" <td>AT</td>\n",
" <td>Gross capacity</td>\n",
" <td>6492.000</td>\n",
" <td></td>\n",
" <td>False</td>\n",
" <td>True</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" <td>False</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" technology source source_type \\\n",
"0 Fossil fuels e-control Regulatory Authority \n",
"1 Fossil fuels e-control Regulatory Authority \n",
"2 Fossil fuels e-control Regulatory Authority \n",
"3 Fossil fuels e-control Regulatory Authority \n",
"4 Fossil fuels e-control Regulatory Authority \n",
"\n",
" weblink year \\\n",
"0 https://www.e-control.at/documents/1785851/181... 2014 \n",
"1 https://www.e-control.at/documents/1785851/181... 2015 \n",
"2 https://www.e-control.at/documents/1785851/181... 2016 \n",
"3 https://www.e-control.at/documents/1785851/181... 2017 \n",
"4 https://www.e-control.at/documents/1785851/181... 2018 \n",
"\n",
" type country capacity_definition capacity comment \\\n",
"0 Installed capacity in MW AT Gross capacity 7243.634 \n",
"1 Installed capacity in MW AT Gross capacity 7059.065 \n",
"2 Installed capacity in MW AT Gross capacity 7323.000 \n",
"3 Installed capacity in MW AT Gross capacity 6469.000 \n",
"4 Installed capacity in MW AT Gross capacity 6492.000 \n",
"\n",
" energy_source_level_0 energy_source_level_1 energy_source_level_2 \\\n",
"0 False True False \n",
"1 False True False \n",
"2 False True False \n",
"3 False True False \n",
"4 False True False \n",
"\n",
" energy_source_level_3 technology_level \n",
"0 False False \n",
"1 False False \n",
"2 False False \n",
"3 False False \n",
"4 False False "
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataframes = [data_opsd, data_eurostat, data_soaf, data_entsoe, data_transparency, data_power_statistics]\n",
"data = pd.concat(dataframes, sort=False)\n",
"data['comment'] = data['comment'].fillna('').astype(str)\n",
"\n",
"col_order = ['technology', 'source', 'source_type', 'weblink', 'year', 'type',\n",
" 'country', 'capacity_definition', 'capacity', 'comment']\n",
"\n",
"data = data[col_order]\n",
"\n",
"\n",
"energy_source_mapping = pd.read_csv(os.path.join('input','energy_source_mapping.csv'),\n",
" index_col ='name')\n",
"energy_source_mapping.replace({0: False, 1: True}, inplace=True)\n",
"\n",
"data = data.merge(energy_source_mapping,\n",
" left_on='technology',\n",
" right_index=True,\n",
" how='left')\n",
"\n",
"new_level_names = {\"Level 0\": \"energy_source_level_0\",\n",
" \"Level 1\": \"energy_source_level_1\",\n",
" \"Level 2\": \"energy_source_level_2\",\n",
" \"Level 3\": \"energy_source_level_3\",\n",
" \"Technology level\": \"technology_level\"}\n",
"\n",
"data.rename(columns=new_level_names, inplace=True)\n",
"\n",
"data.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 4. Convert stacked data to crosstable format"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th>Country (ISO code)</th>\n",
" <th colspan=\"10\" halign=\"left\">AL</th>\n",
" <th>...</th>\n",
" <th colspan=\"10\" halign=\"left\">XK</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Type of data</th>\n",
" <th colspan=\"10\" halign=\"left\">Installed capacity in MW</th>\n",
" <th>...</th>\n",
" <th colspan=\"10\" halign=\"left\">Installed capacity in MW</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Year</th>\n",
" <th>1990</th>\n",
" <th>1991</th>\n",
" <th>1992</th>\n",
" <th>1993</th>\n",
" <th>1994</th>\n",
" <th>1995</th>\n",
" <th>1996</th>\n",
" <th>1997</th>\n",
" <th>1998</th>\n",
" <th>1999</th>\n",
" <th>...</th>\n",
" <th>2007</th>\n",
" <th>2008</th>\n",
" <th>2009</th>\n",
" <th>2010</th>\n",
" <th>2011</th>\n",
" <th>2012</th>\n",
" <th>2013</th>\n",
" <th>2014</th>\n",
" <th>2015</th>\n",
" <th>2016</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Source</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>...</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" <th>EUROSTAT</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Type of source</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>...</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" <th>Statistical Office</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Weblink</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>...</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" <th>Link unavailable</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Capacity definition (net, gross, unknown)</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>...</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" <th>Unknown</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Fossil fuels</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>...</td>\n",
" <td>915.0</td>\n",
" <td>915.0</td>\n",
" <td>915.0</td>\n",
" <td>915.0</td>\n",
" <td>915.0</td>\n",
" <td>915.0</td>\n",
" <td>915.0</td>\n",
" <td>915.0</td>\n",
" <td>915.0</td>\n",
" <td>915.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Lignite</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Hard coal</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Oil</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Natural gas</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 1633 columns</p>\n",
"</div>"
],
"text/plain": [
"Country (ISO code) AL \\\n",
"Type of data Installed capacity in MW \n",
"Year 1990 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 0.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 1991 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 0.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 1992 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 0.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 1993 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 0.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 1994 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 0.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 1995 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 0.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 1996 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 0.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 1997 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 0.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 1998 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 0.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) ... \\\n",
"Type of data ... \n",
"Year 1999 ... \n",
"Source EUROSTAT ... \n",
"Type of source Statistical Office ... \n",
"Weblink Link unavailable ... \n",
"Capacity definition (net, gross, unknown) Unknown ... \n",
"Fossil fuels 0.0 ... \n",
"Lignite NaN ... \n",
"Hard coal NaN ... \n",
"Oil NaN ... \n",
"Natural gas NaN ... \n",
"\n",
"Country (ISO code) XK \\\n",
"Type of data Installed capacity in MW \n",
"Year 2007 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 915.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 2008 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 915.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 2009 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 915.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 2010 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 915.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 2011 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 915.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 2012 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 915.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 2013 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 915.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 2014 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 915.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \\\n",
"Type of data \n",
"Year 2015 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 915.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"Country (ISO code) \n",
"Type of data \n",
"Year 2016 \n",
"Source EUROSTAT \n",
"Type of source Statistical Office \n",
"Weblink Link unavailable \n",
"Capacity definition (net, gross, unknown) Unknown \n",
"Fossil fuels 915.0 \n",
"Lignite NaN \n",
"Hard coal NaN \n",
"Oil NaN \n",
"Natural gas NaN \n",
"\n",
"[5 rows x 1633 columns]"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cols = ['technology', 'source', 'source_type', 'weblink','year',\n",
" 'type', 'country', 'capacity_definition', 'capacity']\n",
"\n",
"data_crosstable = pd.pivot_table(data[cols],\n",
" index=['technology'],\n",
" columns=['country', 'type', 'year',\n",
" 'source', 'source_type',\n",
" 'weblink', 'capacity_definition'],\n",
" values='capacity')\n",
"\n",
"# Apply initial ordering of technologies\n",
"data_crosstable = data_crosstable.reindex(technology_order)\n",
"\n",
"# Delete index naming\n",
"data_crosstable.index.name = None\n",
"data_crosstable.columns.names = ('Country (ISO code)',\n",
" 'Type of data', 'Year',\n",
" 'Source', 'Type of source',\n",
" 'Weblink',\n",
" 'Capacity definition (net, gross, unknown)')\n",
"\n",
"data_crosstable.head()"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th>Country (ISO code)</th>\n",
" <th colspan=\"5\" halign=\"left\">Energy source levels and technology</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Description</th>\n",
" <th>Total generation capacity</th>\n",
" <th>Generation capacity by energy type (fossil, nuclear, renewable, other)</th>\n",
" <th>Generation capacity by fuel or energy source (e.g. coal, lignite, hard coal, natural gas, wind)</th>\n",
" <th>Generation capacity by fuel or energy source and detailed fuel for bioenergy</th>\n",
" <th>Generation capacity by fuel and technology (e.g. combined cycle natural gas, gas turbine, onshore wind, offshore wind)</th>\n",
" </tr>\n",
" <tr>\n",
" <th></th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" </tr>\n",
" <tr>\n",
" <th></th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" </tr>\n",
" <tr>\n",
" <th></th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" </tr>\n",
" <tr>\n",
" <th>Level</th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" <th>NaN</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>NaN</th>\n",
" <td>Level 0</td>\n",
" <td>Level 1</td>\n",
" <td>Level 2</td>\n",
" <td>Level 3</td>\n",
" <td>Technology level</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Fossil fuels</th>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Lignite</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Hard coal</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>Oil</th>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
"Country (ISO code) Energy source levels and technology \\\n",
"Description Total generation capacity \n",
" NaN \n",
" NaN \n",
" NaN \n",
"Level NaN \n",
"NaN Level 0 \n",
"Fossil fuels 0 \n",
"Lignite 0 \n",
"Hard coal 0 \n",
"Oil 0 \n",
"\n",
"Country (ISO code) \\\n",
"Description Generation capacity by energy type (fossil, nuclear, renewable, other) \n",
" NaN \n",
" NaN \n",
" NaN \n",
"Level NaN \n",
"NaN Level 1 \n",
"Fossil fuels 1 \n",
"Lignite 0 \n",
"Hard coal 0 \n",
"Oil 0 \n",
"\n",
"Country (ISO code) \\\n",
"Description Generation capacity by fuel or energy source (e.g. coal, lignite, hard coal, natural gas, wind) \n",
" NaN \n",
" NaN \n",
" NaN \n",
"Level NaN \n",
"NaN Level 2 \n",
"Fossil fuels 0 \n",
"Lignite 1 \n",
"Hard coal 1 \n",
"Oil 1 \n",
"\n",
"Country (ISO code) \\\n",
"Description Generation capacity by fuel or energy source and detailed fuel for bioenergy \n",
" NaN \n",
" NaN \n",
" NaN \n",
"Level NaN \n",
"NaN Level 3 \n",
"Fossil fuels 0 \n",
"Lignite 1 \n",
"Hard coal 1 \n",
"Oil 1 \n",
"\n",
"Country (ISO code) \n",
"Description Generation capacity by fuel and technology (e.g. combined cycle natural gas, gas turbine, onshore wind, offshore wind) \n",
" NaN \n",
" NaN \n",
" NaN \n",
"Level NaN \n",
"NaN Technology level \n",
"Fossil fuels 0 \n",
"Lignite 1 \n",
"Hard coal 1 \n",
"Oil 1 "
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"energylevels_table = energylevels_raw[6:]\n",
"energylevels_table.columns = pd.MultiIndex.from_arrays(energylevels_raw[:6].values,\n",
" names=['country', 'type', 'year',\n",
" 'source', 'source_type',\n",
" 'capacity_definition'\n",
" ])\n",
"\n",
"energylevels_table = energylevels_table.reset_index()\n",
"energylevels_table['technology'] = energylevels_table['technology'].str.replace('- ', '')\n",
"energylevels_table = energylevels_table.set_index('technology')\n",
"\n",
"# Delete index naming\n",
"energylevels_table.index.name = None\n",
"energylevels_table.columns.names = ('Country (ISO code)',\n",
" 'Description', None,\n",
" None, None,\n",
" 'Level')\n",
"\n",
"energylevels_table.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 5. Output"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Delete downloaded zip files"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Deleted: download\\ENTSO-E\\SO&AF\\2011\\SO_AF_2011_-_2025_.zip\n",
"Deleted: download\\ENTSO-E\\SO&AF\\2012\\120705_SOAF_2012_Dataset.zip\n",
"Deleted: download\\ENTSO-E\\SO&AF\\2013\\130403_SOAF_2013-2030_dataset.zip\n",
"Deleted: download\\ENTSO-E\\SO&AF\\2014\\140602_SOAF%202014_dataset.zip\n",
"Deleted: download\\ENTSO-E\\SO&AF\\2016\\SO_AF_2015_dataset.zip\n"
]
}
],
"source": [
"for root, dirs, files in os.walk(\"download\"):\n",
" for file in files:\n",
" item = os.path.join(root, file)\n",
" if item.endswith(\".zip\"):\n",
" os.remove(item)\n",
" print(\"Deleted: \" + item)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Copy input files"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"orig_data_path = os.path.join('output', 'original_data')\n",
"shutil.rmtree(orig_data_path, ignore_errors=True)\n",
"func.copydir(os.path.join('input'), orig_data_path)\n",
"func.copydir(os.path.join('download'), orig_data_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5.1 Write results to file"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Write stacked data to formats: csv, xls and sql."
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [],
"source": [
"# Write the result to file\n",
"data.to_csv(os.path.join('output', 'national_generation_capacity_stacked.csv'),\n",
" encoding='utf-8', index_label='ID')\n",
"\n",
"# Write the results to excel file\n",
"data.to_excel(os.path.join('output', 'national_generation_capacity_stacked.xlsx'),\n",
" sheet_name='output', index_label='ID')\n",
"\n",
"# Write the results to sql database\n",
"data.to_sql('national_generation_capacity_stacked',\n",
" sqlite3.connect(os.path.join('output',\n",
" 'national_generation_capacity.sqlite')),\n",
" if_exists=\"replace\", index_label='ID')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Write data in human readable form to excel."
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
"# Write crosstable data to excel file\n",
"writer = pd.ExcelWriter(os.path.join('output', 'national_generation_capacity.xlsx'))\n",
"data_crosstable.to_excel(writer, sheet_name='output')\n",
"energylevels_table.to_excel(writer, sheet_name='technology levels')\n",
"writer.save()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5.2 Formatting of Excel tables"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [],
"source": [
"outputxls = openpyxl.load_workbook(os.path.join('output',\n",
" 'national_generation_capacity.xlsx'))\n",
"\n",
"ws1 = outputxls['output']\n",
"ws2 = outputxls['technology levels']\n",
"ws1_rows, ws1_cols = data_crosstable.shape\n",
"amount_cols = ws1_cols + 1 # correct 0 index\n",
"\n",
"ws1.column_dimensions['A'].width = 50\n",
"ws2.column_dimensions['A'].width = 50"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [],
"source": [
"blackfont = Font(color=colors.BLACK, italic=False, bold=False)\n",
"blackfontitalic = Font(color=colors.BLACK, italic=True, bold=False)\n",
"blackfontbold = Font(color=colors.BLACK, italic=False, bold=True)\n",
"\n",
"align0 = Alignment(horizontal='left', indent=0)\n",
"align1 = Alignment(horizontal='left', indent=1)\n",
"align2 = Alignment(horizontal='left', indent=2)\n",
"\n",
"# darkest grey\n",
"colour = \"{0:02X}{1:02X}{2:02X}\".format(166, 166, 166)\n",
"grey166 = PatternFill(fgColor=colour, bgColor=colour, patternType=\"solid\")\n",
"\n",
"# darker grey\n",
"colour = \"{0:02X}{1:02X}{2:02X}\".format(191, 191, 191)\n",
"grey191 = PatternFill(fgColor=colour, bgColor=colour, patternType=\"solid\")\n",
"\n",
"# lighter grey\n",
"colour = \"{0:02X}{1:02X}{2:02X}\".format(217, 217, 217)\n",
"grey217 = PatternFill(fgColor=colour, bgColor=colour, patternType=\"solid\")\n",
"\n",
"# lightest grey\n",
"colour = \"{0:02X}{1:02X}{2:02X}\".format(242, 242, 242)\n",
"grey242 = PatternFill(fgColor=colour, bgColor=colour, patternType=\"solid\")"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [],
"source": [
"for col in range(2, amount_cols+1):\n",
" colname = openpyxl.utils.cell.get_column_letter(col)\n",
" ws1.column_dimensions[colname].width = 16\n",
"\n",
"\n",
"for col in range(1, amount_cols+1):\n",
" # format column name block\n",
" for row in range(2,8):\n",
" \n",
" ws1.cell(row=row, column=col).font = blackfont\n",
" \n",
"\n",
" # format cells that contain the values\n",
" for row in range(9, 48):\n",
" ws1.cell(row=row, column=col).fill = grey242\n",
" ws1.cell(row=row, column=1).font = blackfontitalic\n",
" ws1.cell(row=row, column=1).alignment = align2\n",
" \n",
" \n",
" # format row 'Total' with dark grey\n",
" ws1.cell(row=47, column=col).fill = grey166\n",
" ws1.cell(row=47, column=col).font = blackfontbold\n",
" \n",
" \n",
" # format level 1\n",
" for row in [9, 22, 23, 46]:\n",
" ws1.cell(row=row, column=col).fill = grey191\n",
" ws1.cell(row=row, column=col).font = blackfontbold\n",
" ws1.cell(row=row, column=1).alignment = align0\n",
" \n",
" \n",
" # format level 2\n",
" for row in [10, 11, 12, 13, 18, 19, 20, 21, 24, 31, 35, 39, 40, 41, 45]:\n",
" ws1.cell(row=row, column=col).fill = grey217\n",
" ws1.cell(row=row, column=1).alignment = align1\n",
"\n",
" \n",
"ws1.cell(row=47, column=1).alignment = align0\n",
"ws1.freeze_panes = ws1['B8'] #freeze first column and header rows"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [],
"source": [
"# do the same for the second worksheet 'technology levels' \n",
"for col in range(1, 7):\n",
" colname = get_column_letter(col + 1)\n",
" ws2.column_dimensions[colname].width = 25\n",
" \n",
" for row in range(2, 8):\n",
" ws2.cell(row=row, column=col).font = blackfont\n",
" \n",
" \n",
" for row in range(9, 48): \n",
" ws2.cell(row=row, column=col).fill = grey242\n",
" ws2.cell(row=row, column=1).font = blackfontitalic\n",
" ws2.cell(row=row, column=1).alignment = align2\n",
" \n",
" \n",
" # format row 'Total' with dark grey\n",
" ws2.cell(row=46, column=col).fill = grey166\n",
" \n",
" \n",
" # format level 1\n",
" for row in [8, 21, 22, 45]:\n",
" ws2.cell(row=row, column=col).fill = grey191\n",
" ws2.cell(row=row, column=col).font = blackfontbold\n",
" ws2.cell(row=row, column=1).font = blackfontbold\n",
" ws2.cell(row=row, column=1).alignment = align0\n",
" \n",
" \n",
" # format level 2\n",
" for row in [9, 10, 11, 12, 17, 18, 19, 20, 23, 30, 34, 38, 39, 40, 44]:\n",
" ws2.cell(row=row, column=col).fill = grey217\n",
" ws2.cell(row=row, column=1).alignment = align1\n",
"\n",
" \n",
"ws2.cell(row=46, column=1).alignment = align0"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [],
"source": [
"additional_notes = openpyxl.load_workbook(os.path.join('input',\n",
" 'National_Generation_Capacities.xlsx'))['Additional notes']\n",
"# copy additional notes to output file\n",
"for col in range(1, 3):\n",
" for row in range(1, 10):\n",
" add_notes_value = additional_notes.cell(row=row, column=col).value\n",
" ws1.cell(row=row + 50, column=col).value = add_notes_value\n",
" ws1.cell(row=51, column=1).font = blackfontbold\n",
" ws1.cell(row=row + 51, column=1).font = blackfontitalic"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [],
"source": [
"outputxls.save(os.path.join('output', 'national_generation_capacity.xlsx'))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5.3 Write checksums"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [],
"source": [
"files = ['national_generation_capacity.xlsx',\n",
" 'national_generation_capacity_stacked.csv',\n",
" 'national_generation_capacity_stacked.xlsx',\n",
" 'national_generation_capacity.sqlite']\n",
"\n",
"hash_dict = {}\n",
"filesize_dict = {}\n",
"\n",
"with open('checksums.txt', 'w') as f:\n",
" for file_name in files:\n",
" path = os.path.join('output', file_name)\n",
" file_hash = func.get_sha_hash(path)\n",
" hash_dict[file_name] = file_hash\n",
" filesize_dict[file_name] = os.path.getsize(path)\n",
" f.write('{},{}\\n'.format(file_name, file_hash))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 6. Documentation of the data package"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We document the data packages meta data in the specific format JSON as proposed by the Open Knowledge Foundation. See the Frictionless Data project by OKFN (http://data.okfn.org/) and the Data Package specifications (http://dataprotocols.org/data-packages/) for more details.\n",
"\n",
"In order to keep the notebook more readable, we first formulate the metadata in the human-readable YAML format using a multi-line string. We then parse the string into a Python dictionary and save that to disk as a JSON file."
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [],
"source": [
"with open(os.path.join('input', 'metadata.yml'), 'r') as f:\n",
" metadata = yaml.load(f.read(), Loader=yaml.BaseLoader)\n",
" \n",
"metadata['resources'][0]['hash'] = hash_dict['national_generation_capacity.xlsx']\n",
"metadata['resources'][1]['hash'] = hash_dict['national_generation_capacity_stacked.csv']\n",
"metadata['resources'][0]['bytes'] = filesize_dict['national_generation_capacity.xlsx']\n",
"metadata['resources'][1]['bytes'] = filesize_dict['national_generation_capacity_stacked.csv']"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [],
"source": [
"datapackage_json = json.dumps(metadata, indent=4, separators=(',', ': '))\n",
" \n",
"# Write the information of the metadata\n",
"with open(os.path.join('output', 'datapackage.json'), 'w') as f:\n",
" f.write(datapackage_json)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"End of script."
]
}
],
"metadata": {
"@webio": {
"lastCommId": null,
"lastKernelId": null
},
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
}
},
"nbformat": 4,
"nbformat_minor": 1
}