{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Calculate generation capacity by month\n", "This notebook uses the december 2017 EIA-860m file to determine operable generating capacity by fuel category in every month from 2001-2017.\n", "\n", "Because this method uses EIA-860 data on individual plants and generators it does not include the capacity of small scale or distributed solar. EIA does provide an estimate of small scale solar as part of the [Electric Power Monthly (Table 6.1.A)](https://www.eia.gov/electricity/monthly/), although I am not sure if it is supposed to represent all installed non-utility solar in the US. \n", "\n", "## Instructions\n", "The most recent EIA-860m file should be downloaded to the `EIA downloads` folder, and the correct file name should be used for loading data. Otherwise the code below can be run straight through as-is." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "import pandas as pd\n", "import os\n", "import pathlib\n", "from pathlib import Path\n", "import sys\n", "from os.path import join\n", "import json\n", "import calendar\n", "sns.set(style='white')\n", "\n", "idx = pd.IndexSlice" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Type of capacity\n", "The default is net summer capacity, which is what EIA uses for their capacity factor calculations. eGRID uses nameplate capacity. Valid parameter values are:\n", "- `nameplate capacity (mw)`\n", "- `net summer capacity (mw)`\n", "- `net winter capacity (mw)`\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "capacity_type = 'net summer capacity (mw)'\n", "# capacity_type = 'nameplate capacity (mw)'" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "seaborn 0.8.1\n", "pandas 0.22.0\n", "json 2.0.9\n", "CPython 3.6.4\n", "IPython 6.2.1\n" ] } ], "source": [ "%load_ext watermark\n", "%watermark -iv -v" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# Load the \"autoreload\" extension\n", "%load_ext autoreload\n", "\n", "# always reload modules marked with \"%aimport\"\n", "%autoreload 1" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "# add the 'src' directory as one where we can import modules\n", "cwd = os.getcwd()\n", "src_dir = join(cwd, os.pardir, 'src')\n", "sys.path.append(src_dir)\n", "\n", "data_path = Path(cwd, '..', 'Data storage')" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "%aimport Analysis.index\n", "from Analysis.index import group_fuel_cats\n", "\n", "%aimport Analysis.capacity\n", "from Analysis.capacity import monthly_capacity_all, monthly_capacity_year\n", "from Analysis.capacity import monthly_ng_type_all, monthly_ng_type_year" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load monthly EIA-860 data" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "file_path = data_path / 'EIA downloads' / 'december_generator2017.xlsx'\n", "op = pd.read_excel(file_path, sheet_name='Operating', skiprows=1, skip_footer=1,\n", " parse_dates={'op datetime': [14, 15]},\n", " na_values=' ')" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# need to make some helper functions for the retired generators sheet\n", "def bad_month_values(month):\n", " 'Change value to 1 if outside 1-12'\n", "\n", " if month > 12 or month < 1:\n", " new_month = 1\n", " else:\n", " new_month = month\n", " return new_month\n", "\n", "def make_dt_col(df, month_col, year_col):\n", " months = df[month_col].astype(str)\n", " years = df[year_col].astype(str)\n", " dt_string = years + '-' + months + '-' + '01'\n", " dt = pd.to_datetime(dt_string)\n", " return dt\n", "\n", "ret = pd.read_excel(file_path, sheet_name='Retired', skiprows=1, skip_footer=1,\n", " converters={'Operating Month': bad_month_values},\n", " # parse_dates={'op datetime': [16, 17],\n", " # 'ret datetime': [14, 15]},\n", " na_values=' ')\n", "\n", "ret['op datetime'] = make_dt_col(ret, 'Operating Month', 'Operating Year')\n", "ret['ret datetime'] = make_dt_col(ret, 'Retirement Month', 'Retirement Year')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Clean up column names and only keep desired columns" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "op.columns = op.columns.str.strip()\n", "ret.columns = ret.columns.str.strip()" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "op_cols = [\n", " 'Plant ID', 'Nameplate Capacity (MW)', 'Net Summer Capacity (MW)',\n", " 'Energy Source Code', 'Prime Mover Code', 'op datetime'\n", "]\n", "\n", "ret_cols = [\n", " 'Plant ID', 'Nameplate Capacity (MW)', 'Net Summer Capacity (MW)',\n", " 'Energy Source Code', 'Prime Mover Code', 'Retirement Month',\n", " 'Retirement Year', 'Operating Month', 'Operating Year',\n", " 'op datetime', 'ret datetime'\n", "]" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "op = op.loc[:, op_cols]\n", "ret = ret.loc[:, ret_cols]\n", "\n", "op.columns = op.columns.str.lower()\n", "ret.columns = ret.columns.str.lower()" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
plant idnameplate capacity (mw)net summer capacity (mw)energy source codeprime mover codeop datetime
0253.956.0WATHY1963-07-01
13153.155.0NGST1954-02-01
23153.155.0NGST1954-07-01
33403.7362.0BITST1969-12-01
43788.8738.5BITST1971-10-01
\n", "
" ], "text/plain": [ " plant id nameplate capacity (mw) net summer capacity (mw) \\\n", "0 2 53.9 56.0 \n", "1 3 153.1 55.0 \n", "2 3 153.1 55.0 \n", "3 3 403.7 362.0 \n", "4 3 788.8 738.5 \n", "\n", " energy source code prime mover code op datetime \n", "0 WAT HY 1963-07-01 \n", "1 NG ST 1954-02-01 \n", "2 NG ST 1954-07-01 \n", "3 BIT ST 1969-12-01 \n", "4 BIT ST 1971-10-01 " ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "op.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Read fuel category definitions and apply to the generators" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "state_cat_path = data_path / 'Fuel categories' / 'State_facility.json'\n", "custom_cat_path = data_path / 'Fuel categories' / 'Custom_results.json'\n", "with open(state_cat_path) as json_data:\n", " state_cats = json.load(json_data)\n", "with open(custom_cat_path) as json_data:\n", " custom_cats = json.load(json_data)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "def reverse_cats(cat_file):\n", " 'Reverse a dict of lists so each item in the list is a key'\n", " cat_map = {}\n", " for key, vals in cat_file.items():\n", " for val in vals:\n", " cat_map[val] = key\n", " return cat_map" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "# Aggregate EIA fuel codes to my final definitions\n", "op['fuel'] = op.loc[:, 'energy source code'].map(reverse_cats(state_cats))\n", "op['fuel category'] = op.loc[:, 'fuel'].map(reverse_cats(custom_cats))\n", "\n", "ret['fuel'] = ret.loc[:, 'energy source code'].map(reverse_cats(state_cats))\n", "ret['fuel category'] = ret.loc[:, 'fuel'].map(reverse_cats(custom_cats))" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
plant idnameplate capacity (mw)net summer capacity (mw)energy source codeprime mover codeop datetimefuelfuel category
0253.956.0WATHY1963-07-01HYCHydro
13153.155.0NGST1954-02-01NGNatural Gas
23153.155.0NGST1954-07-01NGNatural Gas
33403.7362.0BITST1969-12-01COWCoal
43788.8738.5BITST1971-10-01COWCoal
\n", "
" ], "text/plain": [ " plant id nameplate capacity (mw) net summer capacity (mw) \\\n", "0 2 53.9 56.0 \n", "1 3 153.1 55.0 \n", "2 3 153.1 55.0 \n", "3 3 403.7 362.0 \n", "4 3 788.8 738.5 \n", "\n", " energy source code prime mover code op datetime fuel fuel category \n", "0 WAT HY 1963-07-01 HYC Hydro \n", "1 NG ST 1954-02-01 NG Natural Gas \n", "2 NG ST 1954-07-01 NG Natural Gas \n", "3 BIT ST 1969-12-01 COW Coal \n", "4 BIT ST 1971-10-01 COW Coal " ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "op.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load the NERC region each power plant is in and add to dataframes" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "nercs_path = data_path / 'Facility labels' / 'Facility locations_RF.csv'\n", "facility_nerc = pd.read_csv(nercs_path, index_col=['nerc', 'year'])\n", "facility_nerc.sort_index(inplace=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Need to make this into a dictionary of dictionaries of lists (year, nerc, plant ids)**" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": [ "nerc_dict = {}\n", "for year in facility_nerc.index.get_level_values('year').unique():\n", " nerc_dict[year] = {}\n", " \n", " for nerc in facility_nerc.index.get_level_values('nerc').unique():\n", " nerc_dict[year][nerc] = facility_nerc.loc[idx[nerc, year], 'plant id'].tolist()" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "set()" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Make sure there aren't lots of plants in 2016 that disappear in 2017\n", "set(nerc_dict[2016]['MRO']) - set(nerc_dict[2017]['MRO'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Determine operable capacity in every month" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "# Define iterables to loop over\n", "years = range(2001,2018)\n", "months = range(1,13)\n", "\n", "fuels = list(custom_cats.keys())\n", "\n", "# capacity_type is defined at the top of this notebook\n", "op_df_capacity = monthly_capacity_all(op=op, ret=ret, years=years,\n", " nerc_plant_list=nerc_dict, fuels=fuels,\n", " cap_type=capacity_type, n_jobs=-1,\n", " print_year=False)" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
active capacitypossible gendatetime
nercfuel categoryyearmonth
WECCWind2017819611.41.45909e+072017-08-01
919611.41.41202e+072017-09-01
1019611.41.45909e+072017-10-01
1119611.41.41202e+072017-11-01
1219611.41.45909e+072017-12-01
\n", "
" ], "text/plain": [ " active capacity possible gen datetime\n", "nerc fuel category year month \n", "WECC Wind 2017 8 19611.4 1.45909e+07 2017-08-01\n", " 9 19611.4 1.41202e+07 2017-09-01\n", " 10 19611.4 1.45909e+07 2017-10-01\n", " 11 19611.4 1.41202e+07 2017-11-01\n", " 12 19611.4 1.45909e+07 2017-12-01" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "op_df_capacity.tail()" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [], "source": [ "# Write data to file\n", "out_path = data_path / 'Derived data' / 'Plant capacity' / 'monthly capacity by fuel.csv'\n", "op_df_capacity.to_csv(out_path)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Determine natural gas capacity by prime mover type in each month" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [], "source": [ "# Define iterables to loop over\n", "years = range(2001,2018)\n", "months = range(1,13)\n", "\n", "op_ng_type = monthly_ng_type_all(op=op, ret=ret, years=years,\n", " nerc_plant_list=nerc_dict, fuels=fuels,\n", " cap_type=capacity_type, n_jobs=-1,\n", " print_year=False)" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ngccturbineothertotalngcc fractionturbine fractionother fractiondatetime
nercyearmonth
ASCC20011214.4592.43.0809.80.2647570.7315390.0037052001-01-01
2214.4592.43.0809.80.2647570.7315390.0037052001-02-01
3214.4592.43.0809.80.2647570.7315390.0037052001-03-01
4214.4592.43.0809.80.2647570.7315390.0037052001-04-01
5214.4592.43.0809.80.2647570.7315390.0037052001-05-01
\n", "
" ], "text/plain": [ " ngcc turbine other total ngcc fraction \\\n", "nerc year month \n", "ASCC 2001 1 214.4 592.4 3.0 809.8 0.264757 \n", " 2 214.4 592.4 3.0 809.8 0.264757 \n", " 3 214.4 592.4 3.0 809.8 0.264757 \n", " 4 214.4 592.4 3.0 809.8 0.264757 \n", " 5 214.4 592.4 3.0 809.8 0.264757 \n", "\n", " turbine fraction other fraction datetime \n", "nerc year month \n", "ASCC 2001 1 0.731539 0.003705 2001-01-01 \n", " 2 0.731539 0.003705 2001-02-01 \n", " 3 0.731539 0.003705 2001-03-01 \n", " 4 0.731539 0.003705 2001-04-01 \n", " 5 0.731539 0.003705 2001-05-01 " ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "op_ng_type.head()" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [], "source": [ "out_path = data_path / 'Derived data' / 'Plant capacity' / 'monthly natural gas split.csv'\n", "op_ng_type.to_csv(out_path)" ] } ], "metadata": { "kernelspec": { "display_name": "psci", "language": "python", "name": "psci" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.4" } }, "nbformat": 4, "nbformat_minor": 2 }