Merge branch 'documentation' into documentation

README.md

@@ -34,167 +34,3 @@ The dataset consists of:
   [OPSD project](https://open-power-system-data.org/).
 - Renewable time series based on ERA5 and SARAH, assembled using the [atlite tool](https://github.com/FRESNA/atlite).
 - Geographical potentials for wind and solar generators based on land use (CORINE) and excluding nature reserves (Natura2000) are computed with the [vresutils library](https://github.com/FRESNA/vresutils).
-
-Building the model with the scripts in this repository uses up to 20GB of
-memory. Computing optimal investment and operation scenarios requires a strong
-interior-point solver compatible with the modelling library
-[PYOMO](https://github.com/Pyomo/pyomo) like Gurobi or CPLEX with up to 100GB of
-memory (for the 356-bus approximation).
-
-This project is maintained by the [Energy System Modelling
-group](https://www.iai.kit.edu/english/2338.php) at the [Institute for
-Automation and Applied
-Informatics](https://www.iai.kit.edu/english/index.php) at the
-[Karlsruhe Institute of
-Technology](http://www.kit.edu/english/index.php). It is currently
-funded by the [Helmholtz
-Association](https://www.helmholtz.de/en/). Previous versions were
-developed by the [Renewable Energy
-Group](https://fias.uni-frankfurt.de/physics/schramm/renewable-energy-system-and-network-analysis/)
-at [FIAS](https://fias.uni-frankfurt.de/) to carry out simulations for
-the [CoNDyNet project](http://condynet.de/), financed by the [German
-Federal Ministry for Education and Research
-(BMBF)](https://www.bmbf.de/en/index.html) as part of the [Stromnetze
-Research
-Initiative](http://forschung-stromnetze.info/projekte/grundlagen-und-konzepte-fuer-effiziente-dezentrale-stromnetze/).
-
-# Installation
-
-The steps are demonstrated as shell commands, where the path before the `%` sign denotes the
-directory in which the commands following the `%` should be entered.
-
-Clone the repository using `git` (**to a directory without any spaces in the path**)
-```shell
-/some/other/path % cd /some/path/without/spaces
-/some/path/without/spaces % git clone https://github.com/PyPSA/pypsa-eur.git
-```
-
-## Python dependencies
-The python package requirements are curated in the conda [environment.yaml](environment.yaml) file.
-The environment can be installed and activated using
-```shell
-.../pypsa-eur % conda env create -f environment.yaml
-.../pypsa-eur % conda activate pypsa-eur   # or source activate pypsa-eur on older linux installations
-```
-
-**Note that activation is local to the currently open shell! After opening a new terminal window, one needs to reissue the second command!**
-
-## Data dependencies
-Not all data dependencies are shipped with the git repository (since git is not suited for handling large changing files). Instead we provide two separate data bundles:
-1. [pypsa-eur-data-bundle.tar.xz](https://vfs.fias.science/d/0a0ca1e2fb/files/?p=/pypsa-eur-data-bundle.tar.xz) contains common GIS datasets like NUTS3 shapes, EEZ shapes, CORINE Landcover, Natura 2000 and also electricity specific summary statistics like historic per country yearly totals of hydro generation, GDP and POP on NUTS3 levels and per-country load time-series. It should be extracted in the `data` subdirectory (so that all files are in the `data/bundle` subdirectory)
-```shell
-.../pypsa-eur/data % curl -OL "https://vfs.fias.science/d/0a0ca1e2fb/files/?dl=1&p=/pypsa-eur-data-bundle.tar.xz"
-.../pypsa-eur/data % tar xJf pypsa-eur-data-bundle.tar.xz
-```
-2. [pypsa-eur-cutouts.tar.xz](https://vfs.fias.science/d/0a0ca1e2fb/files/?p=/pypsa-eur-cutouts.tar.xz) are spatiotemporal subsets of the European weather data from the [ECMWF ERA5](https://software.ecmwf.int/wiki/display/CKB/ERA5+data+documentation) reanalysis dataset and the [CMSAF SARAH-2](https://wui.cmsaf.eu/safira/action/viewDoiDetails?acronym=SARAH_V002) solar surface radiation dataset for the year 2013. They have been prepared by and are for use with the [atlite](https://github.com/FRESNA/atlite) tool. You can either generate them yourself using the `build_cutouts` snakemake rule or extract them directly in the `pypsa-eur` directory (extracting the bundle is recommended, since procuring the source weather data files for atlite is not properly documented at the moment):
-```shell
-.../pypsa-eur % curl -OL "https://vfs.fias.science/d/0a0ca1e2fb/files/?dl=1&p=/pypsa-eur-cutouts.tar.xz"
-.../pypsa-eur % tar xJf pypsa-eur-cutouts.tar.xz
-```
-
-3. Optionally, you can download a rasterized version of the NATURA dataset [natura.tiff](https://vfs.fias.science/d/0a0ca1e2fb/files/?p=/natura.tiff&dl=1) and put it into the `resources` sub-directory. If you don't, it will be generated automatically, which takes several hours.
-
-```shell
-.../pypsa-eur % curl -L "https://vfs.fias.science/d/0a0ca1e2fb/files/?p=/natura.tiff&dl=1" -o "resources/natura.tiff"
-```
-
-4. Optionally, if you want to save disk space, you can delete `data/pypsa-eur-data-bundle.tar.xz` and `pypsa-eur-cutouts.tar.xz` once extracting the bundles is complete. E.g.
-
-```shell
-.../pypsa-eur % rm -rf data/pypsa-eur-data-bundle.tar.xz pypsa-eur-cutouts.tar.xz
-```
-
-# Script overview
-
-The model has several configuration options collected in the [config.yaml](config.yaml) file
-located in the root directory.
-
-## Model workflow
-The generation of the model is controlled by the workflow management system
-[Snakemake](https://snakemake.bitbucket.io/). In a nutshell, one declares in the
-`Snakefile` for each python script in the `scripts` directory a rule which
-describes which files the scripts consume and produce. `snakemake` then runs the
-scripts in the correct order and is able to track, what parts of the workflow
-have to be regenerated, when a data file or script is updated. For instance,
-with the [Snakefile of pypsa-eur](Snakefile), an invocation to
-```shell
-snakemake networks/elec_s_128.nc
-```
-follows the dependency graph
-![Dependency graph for network elec_s_128](doc/img/dependencies-elec_s_128.png)
-
-## Building the network
-In detail this means it has to run the independent scripts,
-- `build_shapes` to generate GeoJSON files with country, exclusive economic zones and nuts3 shapes
-- `build_cutout` to prepare smaller weather data portions from ERA5 for cutout `europe-2013-era5` and SARAH for cutout `europe-2013-sarah`.
-
-With these and the externally extracted `ENTSO-E online map topology`, it can build the PyPSA basis model
-- `base_network` stored at `networks/base.nc` with all `buses`, HVAC `lines` and HVDC `links`, and in
-- `build_bus_regions` determine the Voronoi cell of each substation.
-
-Then it hands these over to the scripts for generating renewable and hydro feedin data,
-- `build_hydro_profile` for the hourly hydro energy availability,
-- `build_renewable_potentials` for the landuse/natura2000 constrained installation potentials for PV and wind,
-- `build_renewable_profiles` for the PV and wind hourly capacity factors in each Voronoi cell.
-- `build_powerplants` uses [powerplantmatching](https://github.com/FRESNA/powerplantmatching) to determine today's thermal power plant capacities and then locates the closest substation for each powerplant.
-
-The central rule `add_electricity` then ties all the different data inputs together to a detailed PyPSA model stored in `networks/elec.nc`, containing:
-
-- Today's transmission topology and capacities (optionally including lines which are under construction according to the config settings `lines: under_construction` and `links: under_construction`)
-- Today's thermal and hydro generation capacities (for the technologies listed in the config setting `electricity: conventional_carriers`)
-- Today's load time-series (upsampled according to population and gross domestic product)
-
-It further adds extendable `generators` and `storage_units` with *zero* capacity for
-- wind and pv installations with today's locational, hourly wind and solar pv capacity factors (but **no** capacities)
-- long-term hydrogen and short-term battery storage units (if listed in `electricity: extendable_carriers`)
-- additional open-cycle gas turbines (if `OCGT` is listed in `electricity: extendable_carriers`)
-
-The additional rules prepare approximations of the full model, in which generation, storage and transmission capacities can be co-optimized
-- `simplify_network` transforms the transmission grid to a 380 kV-only equivalent network, while
-- `cluster_network` uses a kmeans based clustering technique to partition the network into a certain number of zones and then reduce the network to a representation with one bus per zone.
-
-The simplification and clustering steps are described in detail in the paper
-[The role of spatial scale in joint optimisations of generation and transmission for European highly renewable scenarios](https://arxiv.org/abs/1705.07617), 2017, [arXiv:1705.07617](https://arxiv.org/abs/1705.07617), [doi:10.1109/EEM.2017.7982024](https://doi.org/10.1109/EEM.2017.7982024).
-
-## Solving the network
-After generating the network it can be solved by using 'solve_all_elec_networks'. This runs the following rules:
-- 'cluster_network'
-- 'prepare_network'
-- 'solve_all_elec_networks'
-- 'solve_network'
-
-## Summarising the results and making plots
-The following rule can be used to summarize the results in seperate .csv files:
-```
-snakemake results/summaries/elec_s_all_lall_Co2L-3H_all
-                                      ^ clusters
-                                          ^ line volume or cost cap
-                                               ^- options
-                                                       ^- all countries
-```
-the line volume/cost cap field can be set to one of the following:
-* `lv1.25` for a particular line volume extension by 25%
-* `lc1.25` for a line cost extension by 25 %
-* `lall` for all evalutated caps
-* `lvall` for all line volume caps
-* `lcall` for all line cost caps
-
-Replacing '/summaries/' with '/plots/' creates nice colored maps of the results.
-
-# Solver choice
-Default choice for the solver is Gurobi (freely available under academic license) or CPLEX. If you want to go fully opensource the CBC solver (https://projects.coin-or.org/Cbc) can be used. To install CBC run 'conda install -c conda-forge coincbc'.
-
-# Hints
-
-For the use of `snakemake`, it makes sense to familiarize oneself quickly with its [basic tutorial](https://snakemake.readthedocs.io/en/stable/tutorial/basics.html) and then read carefully through the section [Executing Snakemake](https://snakemake.readthedocs.io/en/stable/executable.html), noting the arguments `-n`, `-r`, but also `--dag`, `-R` and `-t`.
-
-The dependency graph shown above was generated using
-```shell
-snakemake --dag networks/elec_s_128.nc | dot -Tpng > dependency-graph-elec_s_128.png
-```
-
-# License
-
-The code in PyPSA-Eur is released as free software under the
-[GPLv3](http://www.gnu.org/licenses/gpl-3.0.en.html), see
-[LICENSE](LICENSE.txt).

doc/conf.py

@@ -61,17 +61,17 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'PyPSA-Eur'
-copyright = u'copyright statement'
-author = u'authors'
+copyright = u'2017-2019 Jonas Hoersch (KIT, FIAS), Fabian Hofmann (FIAS), David Schlachtberger (FIAS), Tom Brown (KIT, FIAS); 2019 Fabian Neumann (KIT)'
+author = u'Jonas Hoersch (KIT, FIAS), Fabian Hofmann (FIAS), David Schlachtberger (FIAS), Tom Brown (KIT, FIAS), Fabian Neumann (KIT)'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 #
 # The short X.Y version.
-version = u'0.0.1'
+version = u'0.1.0'
 # The full version, including alpha/beta/rc tags.
-release = u'0.0.1'
+release = u'0.1.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

doc/configtables/atlite.csv

@@ -1,13 +1,8 @@
 ,Unit,Values,Description
 nprocesses,--,int,"Number of parallel processes in cutout preparation"
-cutouts,--,E,TODO
--- europe-2013-era5,--,,"Directory to write cutout data to."
--- -- module,--,"Must be 'era5'","Source of the reanalysis weather dataset. `ERA5 <https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5>`_"
+cutouts,,,
+-- {name},--,"Convention is to name cutouts like ``<region>-<year>-<source>`` (e.g. ``europe-2013-era5``).","Directory to write cutout data to. The user may specify multiple cutouts under configuration ``atlite: cutouts:``. Reference is used in configuration ``renewable: {technology}: cutout:``"
+-- -- module,--,"One of {'era5','sarah'}","Source of the reanalysis weather dataset (e.g. `ERA5 <https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5>`_ or `SARAH-2 <https://wui.cmsaf.eu/safira/action/viewDoiDetails?acronym=SARAH_V002>`_)"
 -- -- xs,°,"Float interval within [-180, 180]","Range of longitudes to download weather data for."
 -- -- ys,°,"Float interval within [-90, 90]","Range of latitudes to download weather data for."
--- -- years,--,"Integer interval within [1979,2018]","Range of years to download weather data for."
--- europe-2013-sarah,--,,"Directory to write cutout data to."
--- -- module,--,"Must be 'sarah'","Source of the surface solar radiation dataset. `SARAH-2 <https://wui.cmsaf.eu/safira/action/viewDoiDetails?acronym=SARAH_V002>`_"
--- -- xs,°,"Float interval within [-180, 180]","Range of longitudes to download weather data for."
--- -- ys,°,"Float interval within [-90, 90]","Range of latitudes to download weather data for."
--- -- years,--,"Integer interval within [1983,2015]","Range of years to download weather data for."
+-- -- years,--,"Integer interval within [1979,2018]","Range of years to download weather data for."

doc/configtables/costs.csv

@@ -4,4 +4,5 @@ discountrate,--,float,"Default discount rate if not specified for a technology i
 USD2013_to_EUR2013,--,float,"Exchange rate from USD :math:`_{2013}` to EUR :math:`_{2013}` from `ECB <https://www.ecb.europa.eu/stats/exchange/eurofxref/html/eurofxref-graph-usd.en.html>`_"
 capital_cost,EUR/MW,"Keys should be in the 'technology' column of ``data/costs.csv``. Values can be any float.","For the given technologies, assumptions about their capital investment costs are set to the corresponding value. Optional; overwrites cost assumptions from ``data/costs.csv``."
 marginal_cost,EUR/MWh,"Keys should be in the 'technology' column of ``data/costs.csv``. Values can be any float.","For the given technologies, assumptions about their marginal operating costs are set to the corresponding value. Optional; overwrites cost assumptions from ``data/costs.csv``."
-emission_prices,--,E,TODO
+emission_prices,,,
+-- co2,EUR/t,float,"Exogenous price of carbon-dioxide added to the marginal costs of fossil-fuelled generators according to their carbon intensity."

doc/configtables/hydro.csv

@@ -3,4 +3,4 @@ cutout,--,"Must be 'europe-2013-era5'","Specifies the directory where the releva
 carriers,--,"Any subset of {'ror', 'PHS', 'hydro'}","Specifies the types of hydro power plants to build per-unit availability time series for. 'ror' stands for run-of-river plants, 'PHS' represents pumped-hydro storage, and 'hydro' stands for hydroelectric dams."
 PHS_max_hours,h,float,"Maximum state of charge capacity of the pumped-hydro storage (PHS) in terms of hours at full output capacity ``p_nom``. Cf. `PyPSA documentation <https://pypsa.readthedocs.io/en/latest/components.html#storage-unit>`_."
 hydro_max_hours,h,"Any of {float, 'energy_capacity_totals_by_country', 'estimate_by_large_installations'}","Maximum state of charge capacity of the pumped-hydro storage (PHS) in terms of hours at full output capacity ``p_nom`` or heuristically determined. Cf. `PyPSA documentation <https://pypsa.readthedocs.io/en/latest/components.html#storage-unit>`_."
-clip_min_inflow,??,float,"To avoid too small values in the inflow time series, values below this threshold are set to zero."
+clip_min_inflow,MW,float,"To avoid too small values in the inflow time series, values below this threshold are set to zero."

doc/configtables/offwind-ac.csv

@@ -1,5 +1,5 @@
 ,Unit,Values,Description
-cutout,--,"Must be 'europe-2013-era5'","Specifies the directory where the relevant weather data ist stored."
+cutout,--,"Should be a folder listed in the configuration ``atlite: cutouts:`` (e.g. 'europe-2013-era5') or reference an existing folder in the directory ``cutouts``. Source module must be ERA5.","Specifies the directory where the relevant weather data ist stored."
 resource,,,
 -- method,--,"Must be 'wind'","A superordinate technology type."
 -- turbine,--,"One of turbine types included in `atlite <https://github.com/PyPSA/atlite/tree/master/atlite/resources/windturbine>`_","Specifies the turbine type and its characteristic power curve."
@@ -8,5 +8,5 @@ corine,--,"Any *realistic* subset of the `CORINE Land Cover code list <http://ww
 natura,bool,"{true, false}","Switch to exclude `Natura 2000 <https://en.wikipedia.org/wiki/Natura_2000>`_ natural protection areas. Area is excluded if ``true``."
 max_depth,m,float,"Maximum sea water depth at which wind turbines can be build. Maritime areas with deeper waters are excluded in the process of calculating the AC-connected offshore wind potential."
 min_shore_distance,m,float,"Minimum distance to the shore below which wind turbines cannot be build. Such areas close to the shore are excluded in the process of calculating the AC-connected offshore wind potential."
-potential,--,"One of {'simple', 'conservative'}",TODO
+potential,--,"One of {'simple', 'conservative'}","Method to compute the maximal installable potential for a node; confer :ref:`renewableprofiles`"
 clip_p_max_pu,p.u.,float,"To avoid too small values in the renewables` per-unit availability time series values below this threshold are set to zero."

doc/configtables/offwind-dc.csv

@@ -1,5 +1,5 @@
 ,Unit,Values,Description
-cutout,--,"Must be 'europe-2013-era5'","Specifies the directory where the relevant weather data ist stored."
+cutout,--,"Should be a folder listed in the configuration ``atlite: cutouts:`` (e.g. 'europe-2013-era5') or reference an existing folder in the directory ``cutouts``. Source module must be ERA5.","Specifies the directory where the relevant weather data ist stored."
 resource,,,
 -- method,--,"Must be 'wind'","A superordinate technology type."
 -- turbine,--,"One of turbine types included in `atlite <https://github.com/PyPSA/atlite/tree/master/atlite/resources/windturbine>`_","Specifies the turbine type and its characteristic power curve."
@@ -8,5 +8,5 @@ corine,--,"Any *realistic* subset of the `CORINE Land Cover code list <http://ww
 natura,bool,"{true, false}","Switch to exclude `Natura 2000 <https://en.wikipedia.org/wiki/Natura_2000>`_ natural protection areas. Area is excluded if ``true``."
 max_depth,m,float,"Maximum sea water depth at which wind turbines can be build. Maritime areas with deeper waters are excluded in the process of calculating the AC-connected offshore wind potential."
 min_shore_distance,m,float,"Minimum distance to the shore below which wind turbines cannot be build. Such areas close to the shore are excluded in the process of calculating the AC-connected offshore wind potential."
-potential,--,"One of {'simple', 'conservative'}",TODO
+potential,--,"One of {'simple', 'conservative'}","Method to compute the maximal installable potential for a node; confer :ref:`renewableprofiles`"
 clip_p_max_pu,p.u.,float,"To avoid too small values in the renewables` per-unit availability time series values below this threshold are set to zero."

doc/configtables/onwind.csv

@@ -1,13 +1,13 @@
 ,Unit,Values,Description
-cutout,--,"Must be 'europe-2013-era5'","Specifies the directory where the relevant weather data ist stored."
+cutout,--,"Should be a folder listed in the configuration ``atlite: cutouts:`` (e.g. 'europe-2013-era5') or reference an existing folder in the directory ``cutouts``. Source module must be ERA5.","Specifies the directory where the relevant weather data ist stored."
 resource,,,
 -- method,--,"Must be 'wind'","A superordinate technology type."
 -- turbine,--,"One of turbine types included in `atlite <https://github.com/PyPSA/atlite/tree/master/atlite/resources/windturbine>`_","Specifies the turbine type and its characteristic power curve."
-capacity_per_sqkm,MW/km2,float,"Allowable density of wind turbine placement."
+capacity_per_sqkm,:math:`MW/km^2`,float,"Allowable density of wind turbine placement."
 corine,,,
 -- grid_codes,--,"Any subset of the `CORINE Land Cover code list <http://www.eea.europa.eu/data-and-maps/data/corine-land-cover-2006-raster-1/corine-land-cover-classes-and/clc_legend.csv/at_download/file>`_","Specifies areas according to CORINE Land Cover codes which are generally eligible for wind turbine placement."
 -- distance,m,float,"Distance to keep from areas specified in ``distance_grid_codes``"
 -- distance_grid_codes,--,"Any subset of the `CORINE Land Cover code list <http://www.eea.europa.eu/data-and-maps/data/corine-land-cover-2006-raster-1/corine-land-cover-classes-and/clc_legend.csv/at_download/file>`_","Specifies areas according to CORINE Land Cover codes to which wind turbines must maintain a distance specified in the setting ``distance``."
 natura,bool,"{true, false}","Switch to exclude `Natura 2000 <https://en.wikipedia.org/wiki/Natura_2000>`_ natural protection areas. Area is excluded if ``true``."
-potential,--,"One of {'simple', 'conservative'}","TODO"
+potential,--,"One of {'simple', 'conservative'}","Method to compute the maximal installable potential for a node; confer :ref:`renewableprofiles`"
 clip_p_max_pu,p.u.,float,"To avoid too small values in the renewables` per-unit availability time series values below this threshold are set to zero."

doc/configtables/plotting.csv

@@ -10,15 +10,6 @@ costs_threshold,bn Euro,float,"Threshold below which technologies will not be sh
 energy_max,TWh,float,"Upper y-axis limit in energy bar plots."
 energy_min,TWh,float,"Lower y-axis limit in energy bar plots."
 energy_threshold,TWh,float,"Threshold below which technologies will not be shown in energy bar plots."
-vre_techs,--,E,TODO
-conv_techs,--,E,TODO
-storage_techs,--,E,TODO
-store_techs,--,E,TODO
-load_carriers,--,E,TODO
-AC_carriers,--,E,TODO
-link_carriers,--,E,TODO
-heat_links,--,E,TODO
-heat_generators,--,E,TODO
-tech_colors,--,E,TODO
-nice_names,--,E,TODO
-nice_names_n,--,E,TODO
+tech_colors,--,"carrier -> HEX colour code","Mapping from network ``carrier`` to a colour (`HEX colour code <https://en.wikipedia.org/wiki/Web_colors#Hex_triplet>`_)."
+nice_names,--,"str -> str","Mapping from network ``carrier`` to a more readable name."
+nice_names_n,--,"str -> str","Same as nice_names, but with linebreaks."

doc/configtables/scenario.csv

@@ -1,6 +1,6 @@
 ,Unit,Values,Description
-sectors,--,{'E'},TODO
-simpl,--,E,TODO
-ll,--,E,TODO
-clusters,--,E,TODO
-opts,--,E,TODO
+sectors,--,"Must be 'elec'","Placeholder for integration of other energy sectors."
+simpl,--,cf. :ref:`simpl`,"List of ``{simpl}`` wildcards to run."
+ll,--,cf. :ref:`ll`,"List of ``{ll}`` wildcards to run."
+clusters,--,cf. :ref:`clusters`,"List of ``{clusters}`` wildcards to run."
+opts,--,cf. :ref:`opts`,"List of ``{opts}`` wildcards to run."

doc/configtables/solar.csv

@@ -1,5 +1,5 @@
 ,Unit,Values,Description
-cutout,--,"One of {'europe-2013-era5', 'europe-2013-sarah'}","Specifies the directory where the relevant weather data ist stored that is specified at ``atlite/cutouts`` configuration. Both ``sarah`` and ``era5`` work."
+cutout,--,"Should be a folder listed in the configuration ``atlite: cutouts:`` (e.g. 'europe-2013-era5') or reference an existing folder in the directory ``cutouts``. Source module can be ERA5 or SARAH-2.","Specifies the directory where the relevant weather data ist stored that is specified at ``atlite/cutouts`` configuration. Both ``sarah`` and ``era5`` work."
 resource,,,
 -- method,--,"Must be 'pv'","A superordinate technology type."
 -- panel,--,"One of {'Csi', 'CdTe', 'KANENA'} as defined in `atlite <https://github.com/PyPSA/atlite/tree/master/atlite/resources/solarpanel>`_","Specifies the solar panel technology and its characteristic attributes."
@@ -10,5 +10,5 @@ capacity_per_sqkm,:math:`MW/km^2`,float,"Allowable density of solar panel placem
 correction_factor,--,float,"A correction factor for the capacity factor (availability) time series."
 corine,--,"Any subset of the `CORINE Land Cover code list <http://www.eea.europa.eu/data-and-maps/data/corine-land-cover-2006-raster-1/corine-land-cover-classes-and/clc_legend.csv/at_download/file>`_","Specifies areas according to CORINE Land Cover codes which are generally eligible for solar panel placement."
 natura,bool,"{true, false}","Switch to exclude `Natura 2000 <https://en.wikipedia.org/wiki/Natura_2000>`_ natural protection areas. Area is excluded if ``true``."
-potential,--,"One of {'simple', 'conservative'}",TODO
+potential,--,"One of {'simple', 'conservative'}","Method to compute the maximal installable potential for a node; confer :ref:`renewableprofiles`"
 clip_p_max_pu,p.u.,float,"To avoid too small values in the renewables` per-unit availability time series values below this threshold are set to zero."

doc/configtables/solving-options.csv

@@ -1,6 +1,6 @@
 ,Unit,Values,Description
 formulation,--,"Any of {'angles', 'kirchhoff', 'cycles', 'ptdf'}","Specifies which variant of linearized power flow formulations to use in the optimisation problem. Recommended is 'kirchhoff'. Explained in `this article <https://arxiv.org/abs/1704.01881>`_."
-load_shedding,--,E,TODO
+load_shedding,bool,"{'true','false'}","Add generators with a prohibitively high marginal cost to simulate load shedding and avoid problem infeasibilities."
 noisy_costs,bool,"{'true','false'}","Add random noise to marginal cost of generators by :math:`\mathcal{U}(0.009,0,011)` and capital cost of lines and links by :math:`\mathcal{U}(0.09,0,11)`."
 min_iterations,--,int,"Minimum number of solving iterations in between which resistance and reactence (``x/r``) are updated for branches according to ``s_nom_opt`` of the previous run."
 max_iterations,--,int,"Maximum number of solving iterations in between which resistance and reactence (``x/r``) are updated for branches according to ``s_nom_opt`` of the previous run."

doc/configtables/solving-solver.csv

@@ -1,3 +1,3 @@
 ,Unit,Values,Description
-name,--,{'E'},TODO
-opts,--,E,TODO
+name,--,"One of {'gurobi', 'cplex', 'cbc', 'glpk', 'ipopt'}; potentially more possible","Solver to use for optimisation problems in the workflow; e.g. clustering and linear optimal power flow."
+opts,--,"Parameter list for `Gurobi <https://www.gurobi.com/documentation/8.1/refman/parameters.html>`_ and `CPLEX <https://www.ibm.com/support/knowledgecenter/SSSA5P_12.5.1/ilog.odms.cplex.help/CPLEX/Parameters/topics/introListAlpha.html>`_","Solver specific parameter settings."

doc/configuration.rst

@@ -1,9 +1,13 @@
+.. _config:
+
 ##########################################
 Configuration
 ##########################################
 
 PyPSA-Eur has several configuration options which are documented in this section and are collected in a ``config.yaml`` file located in the root directory. Users can amend their own modifications and assumptions by changing the default configuration provided in the configuration file (``config.yaml``).
 
+.. _toplevel_cf:
+
 Top-level configuration
 =======================
 
@@ -16,9 +20,28 @@ Top-level configuration
    :widths: 25,7,22,30
    :file: configtables/toplevel.csv
 
+.. _scenario:
 
-Wildcards and ``scenario``
-==========================
+``scenario``
+============
+
+It is common conduct to analyse energy system optimisation models for **multiple scenarios** for a variety of reasons,
+e.g. assessing their sensitivity towards changing the temporal and/or geographical resolution or investigating how
+investment changes as more ambitious greenhouse-gas emission reduction targets are applied.
+
+The ``scenario`` section is an extraordinary section of the config file
+that is strongly connected to the :ref:`wildcards` and is designed to
+facilitate running multiple scenarios through a single command 
+
+.. code:: bash
+    
+    snakemake solve_all_elec_networks
+
+For each wildcard, a **list of values** is provided. The rule ``solve_all_elec_networks`` will trigger the rules for creating ``results/networks/elec_s{simpl}_{clusters}_l{ll}_{opts}.nc`` for **all combinations** of the provided wildcard values as defined by Python's `itertools.product(...) <https://docs.python.org/2/library/itertools.html#itertools.product>`_ function that snakemake's `expand(...) function <https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#targets>`_ uses.
+
+An exemplary dependency graph (starting from the simplification rules) then looks like this:
+
+.. image:: img/scenarios.png
 
 .. literalinclude:: ../config.yaml
    :language: yaml
@@ -28,13 +51,13 @@ Wildcards and ``scenario``
    :header-rows: 1
    :widths: 25,7,22,30
    :file: configtables/scenario.csv
-   
-.. image:: img/scenarios.png
+
+.. _snapshots_cf:
 
 ``snapshots``
 =============
 
-Arguments to `pandas.date_range <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.date_range.html>`_
+Specifies the temporal range to build an energy system model for as arguments to `pandas.date_range <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.date_range.html>`_
 
 .. literalinclude:: ../config.yaml
    :language: yaml
@@ -45,6 +68,7 @@ Arguments to `pandas.date_range <https://pandas.pydata.org/pandas-docs/stable/re
    :widths: 25,7,22,30
    :file: configtables/snapshots.csv
 
+.. _electricity_cf:
 
 ``electricity``
 ===============
@@ -61,6 +85,8 @@ Arguments to `pandas.date_range <https://pandas.pydata.org/pandas-docs/stable/re
 .. warning::
     Carriers in ``conventional_carriers`` must not also be in ``extendable_carriers``.
 
+.. _atlite_cf:
+
 ``atlite``
 =============
 
@@ -73,6 +99,8 @@ Arguments to `pandas.date_range <https://pandas.pydata.org/pandas-docs/stable/re
    :widths: 25,7,22,30
    :file: configtables/atlite.csv
 
+.. _renewable_cf:
+
 ``renewable``
 =============
 
@@ -116,6 +144,8 @@ Arguments to `pandas.date_range <https://pandas.pydata.org/pandas-docs/stable/re
    :widths: 25,7,22,30
    :file: configtables/hydro.csv
 
+.. _lines_cf:
+
 ``lines``
 =============
 
@@ -124,6 +154,8 @@ Arguments to `pandas.date_range <https://pandas.pydata.org/pandas-docs/stable/re
    :widths: 25,7,22,30
    :file: configtables/lines.csv
 
+.. _links_cf:
+
 ``links``
 =============
 
@@ -132,6 +164,8 @@ Arguments to `pandas.date_range <https://pandas.pydata.org/pandas-docs/stable/re
    :widths: 25,7,22,30
    :file: configtables/links.csv
 
+.. _transformers_cf:
+
 ``transformers``
 ================
 
@@ -140,6 +174,8 @@ Arguments to `pandas.date_range <https://pandas.pydata.org/pandas-docs/stable/re
    :widths: 25,7,22,30
    :file: configtables/transformers.csv
 
+.. _load_cf:
+
 ``load``
 =============
 
@@ -148,6 +184,8 @@ Arguments to `pandas.date_range <https://pandas.pydata.org/pandas-docs/stable/re
    :widths: 25,7,22,30
    :file: configtables/load.csv
 
+.. _costs_cf:
+
 ``costs``
 =============
 
@@ -159,6 +197,10 @@ Arguments to `pandas.date_range <https://pandas.pydata.org/pandas-docs/stable/re
 .. note::
     To change cost assumptions in more detail (i.e. other than ``marginal_cost`` and ``capital_cost``), consider modifying cost assumptions directly in ``data/costs.csv`` as this is not yet supported through the config file.
 
+    You can also build multiple different cost databases. Make a renamed copy of ``data/costs.csv`` (e.g. ``data/costs-optimistic.csv``) and set the variable ``COSTS=data/costs-optimistic.csv`` in the ``Snakefile``.
+
+.. _solving_cf:
+
 ``solving``
 =============
 
@@ -178,6 +220,8 @@ Arguments to `pandas.date_range <https://pandas.pydata.org/pandas-docs/stable/re
    :widths: 25,7,22,30
    :file: configtables/solving-solver.csv
 
+.. _plotting_cf:
+
 ``plotting``
 =============
 

doc/contributing.rst

@@ -2,7 +2,9 @@
 Contributing
 #######################
 
+We strongly welcome anyone interested in contributing to this project,
+be it with new ideas, suggestions, by filing bug reports or contributing code.
 
-We strongly welcome anyone interested in contributing to this project.
+You are invited to submit pull requests and file issues to the `GitHub repository <https://github.com/PyPSA/PyPSA-Eur>`_.
 
-Submit pull requests / issues to the `github repository <https://github.com/PyPSA/PyPSA-Eur>`_.
+If you are unfamiliar with pull requests, the GitHub help pages have a nice `guide <https://help.github.com/en/articles/about-pull-requests>`_.

doc/examples.rst

@@ -0,0 +1,11 @@
+.. _examples:
+
+########
+Examples
+########
+
+Building an Energy System Model with PyPSA-Eur
+==============================================
+
+Analysing a Solved Energy System Model
+======================================

doc/index.rst

@@ -1,7 +1,25 @@
 PyPSA-Eur: An Open Optimisation Model of the European Transmission System
 =========================================================================
 
-TODO: insert badges
+.. image:: https://img.shields.io/github/tag-date/pypsa/pypsa-eur
+    :alt: GitHub tag
+
+.. image:: https://readthedocs.org/projects/pypsa-eur/badge/?version=latest
+    :target: https://pypsa-eur.readthedocs.io/en/latest/?badge=latest
+    :alt: Documentation Status
+
+.. image:: https://img.shields.io/github/license/pypsa/pypsa-eur
+    :alt: GitHub
+
+.. image:: https://img.shields.io/github/repo-size/pypsa/pypsa-eur
+    :alt: GitHub repo size
+
+.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.1246852.svg
+    :target: https://doi.org/10.5281/zenodo.1246852
+
+.. image:: https://badges.gitter.im/PyPSA/community.svg
+    :target: https://gitter.im/PyPSA/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge
+    :alt: Chat on Gitter
 
 PyPSA-Eur is an open model dataset of the European power system at the
 transmission network level that covers the full ENTSO-E area.
@@ -10,11 +28,13 @@ It contains alternating current lines at and above 220 kV voltage level and all
 
 The model is suitable both for operational studies and generation and transmission expansion planning studies. The continental scope and highly resolved spatial scale enables a proper description of the long-range smoothing effects for renewable power generation and their varying resource availability.
 
+.. image:: img/base.png
+
 The restriction to freely available and open data encourages the open exchange of model data developments and eases the comparison of model results. It provides a full, automated software pipeline to assemble the load-flow-ready model from the original datasets, which enables easy replacement and improvement of the individual parts.
 
-PyPSA-Eur is designed to be imported into the open toolbox `PyPSA <https://www.pypsa.org>`_ for which `documentation <>`_ is available as well.
+PyPSA-Eur is designed to be imported into the open toolbox `PyPSA <https://www.pypsa.org>`_ for which `documentation <https://pypsa.org/doc>`_ is available as well.
 
-This project is maintained by the `Energy System Modelling group <https://www.iai.kit.edu/english/2338.php>`_ at the `Institute for Automation and Applied Informatics <https://www.iai.kit.edu/english/index.php>`_ at the `Karlsruhe Institute of Technology <http://www.kit.edu/english/index.php>`_. The group is funded by the `Helmholtz Association <https://www.helmholtz.de/en/>`_ until 2024. Previous versions were developed by the `Renewable Energy Group <https://fias.uni-frankfurt.de/physics/schramm/renewable-energy-system-and-network-analysis/>`_ at `FIAS <https://fias.uni-frankfurt.de/>`_ to carry out simulations for the `CoNDyNet project <http://condynet.de/>`_, financed by the `German Federal Ministry for Education and Research (BMBF) <https://www.bmbf.de/en/index.html>` as part of the `Stromnetze Research Initiative <http://forschung-stromnetze.info/projekte/grundlagen-und-konzepte-fuer-effiziente-dezentrale-stromnetze/>`_.
+This project is maintained by the `Energy System Modelling group <https://www.iai.kit.edu/english/2338.php>`_ at the `Institute for Automation and Applied Informatics <https://www.iai.kit.edu/english/index.php>`_ at the `Karlsruhe Institute of Technology <http://www.kit.edu/english/index.php>`_. The group is funded by the `Helmholtz Association <https://www.helmholtz.de/en/>`_ until 2024. Previous versions were developed by the `Renewable Energy Group <https://fias.uni-frankfurt.de/physics/schramm/renewable-energy-system-and-network-analysis/>`_ at `FIAS <https://fias.uni-frankfurt.de/>`_ to carry out simulations for the `CoNDyNet project <http://condynet.de/>`_, financed by the `German Federal Ministry for Education and Research (BMBF) <https://www.bmbf.de/en/index.html>`_ as part of the `Stromnetze Research Initiative <http://forschung-stromnetze.info/projekte/grundlagen-und-konzepte-fuer-effiziente-dezentrale-stromnetze/>`_.
 
 Documentation
 =============
@@ -23,9 +43,7 @@ Documentation
 
 * :doc:`introduction`
 * :doc:`installation`
-* :doc:`wildcards`
-* :doc:`configuration`
-* :doc:`example`
+* :doc:`examples`
 
 .. toctree::
    :hidden:
@@ -34,6 +52,18 @@ Documentation
 
    introduction
    installation
+   examples
+
+**Configuration**
+
+* :doc:`wildcards`
+* :doc:`configuration`
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+   :caption: Configuration
+
    wildcards
    configuration
    example
@@ -74,12 +104,12 @@ Citing PyPSA-Eur
 
 If you use PyPSA-Eur for your research, we would appreciate it if you would cite the following paper:
 
-- J. Hörsch, F. Hofmann, D. Schlachtberger, T. Brown, `PyPSA-Eur: An Open Optimisation Model of the European Transmission System <https://arxiv.org/abs/1806.01613>`_, 2018, Energy Strategy Reviews, Volume 22, November 2018, Pages 207-215, `10.1016/j.esr.2018.08.012 <https://doi.org/10.1016/j.esr.2018.08.012>`_, `arXiv:1806.01613 <https://arxiv.org/abs/1806.01613>`_
+- Jonas Hörsch, Fabian Hofmann, David Schlachtberger, and Tom Brown. `PyPSA-Eur: An open optimisation model of the European transmission system <https://arxiv.org/abs/1806.01613>`_. Energy Strategy Reviews, 22:207-215, 2018. `arXiv:1806.01613 <https://arxiv.org/abs/1806.01613>`_, `doi:10.1016/j.esr.2018.08.012 <https://doi.org/10.1016/j.esr.2018.08.012>`_.
 
 Please use the following BibTeX: ::
 
     @article{PyPSAEur,
-        author = "Jonas H\"orsch and Fabian Hofmann and David Schlachtberger and Tom Brown",
+        author = "Jonas Hoersch and Fabian Hofmann and David Schlachtberger and Tom Brown",
         title = "PyPSA-Eur: An open optimisation model of the European transmission system",
         journal = "Energy Strategy Reviews",
         volume = "22",
@@ -87,12 +117,11 @@ Please use the following BibTeX: ::
         year = "2018",
         issn = "2211-467X",
         doi = "10.1016/j.esr.2018.08.012",
-        url = "https://doi.org/10.1016/j.esr.2018.08.012",
         eprint = "1806.01613"
     }
 
 
-TODO: actually add versions to zenodo
+.. todo:: Add versions to zenodo
 
 If you want to cite a specific PyPSA-Eur version, each release of PyPSA-Eur is stored on Zenodo with a release-specific DOI.
 This can be found linked from the overall PyPSA-Eur Zenodo DOI:
@@ -100,6 +129,7 @@ This can be found linked from the overall PyPSA-Eur Zenodo DOI:
 .. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.1246852.svg
    :target: https://doi.org/10.5281/zenodo.1246852
 
+
 Licence
 =======
 

doc/installation.rst

@@ -4,16 +4,14 @@
 Installation
 ##########################################
 
-TODO: install via bash installation script
-
-.. note:: 
-    The steps are demonstrated as shell commands, where the path before the ``%`` sign denotes the
-    directory in which the commands following the ``%`` should be entered.
+The subsequently described installation steps are demonstrated as shell commands, where the path before the ``%`` sign denotes the
+directory in which the commands following the ``%`` should be entered.
 
 Clone the Repository
 ====================
 
-Clone the PyPSA-Eur repository using ``git``
+First of all, clone the `PyPSA-Eur repository <https://github.com/PyPSA/pypsa-eur>`_ using the version control system ``git``.
+The path to the directory into which the ``git repository`` is cloned, must **not** have any spaces!
 
 .. code:: bash
 
@@ -22,17 +20,19 @@ Clone the PyPSA-Eur repository using ``git``
     /some/path/without/spaces % git clone https://github.com/PyPSA/pypsa-eur.git
 
 .. note::
-    The path to the directory into which the ``git repository`` is cloned, 
-    must not have any spaces.
+    If you do not have ``git`` installed, follow installation instructions `here <https://git-scm.com/book/en/v2/Getting-Started-Installing-Git>`_.
 
 .. _deps:
 
 Install Python Dependencies
 ===============================
 
-TODO: conda link
+PyPSA-Eur relies on a set of other Python packages to function.
+We recommend using the package manager and environment management system ``conda`` to install them.
+Install `miniconda <https://docs.conda.io/en/latest/miniconda.html>`_, which is a mini version of `Anaconda <https://www.anaconda.com/>`_ that includes only ``conda`` and its dependencies or make sure ``conda`` is already installed on your system.
+For instructions for your operating system follow the ``conda`` `installation guide <https://docs.conda.io/projects/conda/en/latest/user-guide/install/>`_.
 
-The python package requirements are curated in the conda `environment.yaml <https://github.com/PyPSA/pypsa-eur/blob/master/environment.yaml>`_ file.
+The python package requirements are curated in the `environment.yaml <https://github.com/PyPSA/pypsa-eur/blob/master/environment.yaml>`_ file.
 The environment can be installed and activated using
 
 .. code:: bash
@@ -43,15 +43,18 @@ The environment can be installed and activated using
 
 .. note::
     Note that activation is local to the currently open shell!
-    After opening a new terminal window,
-    one needs to reissue the second command! 
+    After opening a new terminal window, one needs to reissue the second command! 
 
 .. _data:
 
 Download Data Dependencies
 ==============================
 
-Not all data dependencies are shipped with the git repository, since git is not suited for handling large changing files. Instead we provide separate data bundles:
+Not all data dependencies are shipped with the git repository,
+since git is not suited for handling large changing files.
+Instead we provide separate data bundles which can be obtained
+using the described shell commands or by downloading and
+extracting them manually in the locations outlined below.
 
 1. **Data Bundle:** `pypsa-eur-data-bundle.tar.xz <https://vfs.fias.science/d/0a0ca1e2fb/files/?p=/pypsa-eur-data-bundle.tar.xz>`_ contains common GIS datasets like NUTS3 shapes, EEZ shapes, CORINE Landcover, Natura 2000 and also electricity specific summary statistics like historic per country yearly totals of hydro generation, GDP and POP on NUTS3 levels and per-country load time-series. It should be extracted in the ``data`` sub-directory, such that all files of the bundle are stored in the ``data/bundle`` subdirectory)
 
@@ -86,6 +89,22 @@ Not all data dependencies are shipped with the git repository, since git is not
 Install a Solver
 ================
 
-Default choice for the solver is Gurobi (freely available under academic license) or CPLEX. If you want to go fully opensource the CBC solver (https://projects.coin-or.org/Cbc) can be used. To install CBC run ::
+PyPSA passes the PyPSA-Eur network model to an external solver for performing a total annual system cost minimization with optimal power flow.
+PyPSA is known to work with the free software
 
-    conda install -c conda-forge coincbc
+- `Cbc <https://projects.coin-or.org/Cbc#DownloadandInstall>`_
+- `GLPK <https://www.gnu.org/software/glpk/>`_ (`WinGLKP <http://winglpk.sourceforge.net/>`_)
+
+and the non-free, commercial software (for which free academic licenses are available)
+
+- `Gurobi <https://www.gurobi.com/documentation/8.1/remoteservices/installation.html>`_
+- `CPLEX <https://www.ibm.com/products/ilog-cplex-optimization-studio>`_
+
+and any other solver that works with the underlying modelling framework `Pyomo <http://www.pyomo.org/>`_. For installation instructions of these solvers for your operating system, follow the links above.
+
+.. seealso::
+    `Getting a solver in the PyPSA documentation <https://pypsa.readthedocs.io/en/latest/installation.html#getting-a-solver-for-linear-optimisation>`_
+
+.. note::
+    Commercial solvers such as Gurobi and CPLEX currently significantly outperform open-source solvers for large-scale problems.
+    It might be the case that you can only retrieve solutions by using a commercial solver.

doc/introduction.rst

@@ -31,13 +31,24 @@ The **blocks** represent the individual rules which are required to create the f
 
 For the use of ``snakemake``, it makes sense to familiarize oneself quickly with its `basic tutorial <https://snakemake.readthedocs.io/en/stable/tutorial/basics.html>`_ and then read carefully through the section `Executing Snakemake <https://snakemake.readthedocs.io/en/stable/executable.html>`_, noting the arguments ``-n``, ``-r``, but also ``--dag``, ``-R`` and ``-t``.
 
-Modification
-============
+Scenarios, Configuration and Modification
+=========================================
 
-TODO: wildcards modification
+It is easy to run PyPSA-Eur for multiple scenarios using the `wildcards feature <https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#wildcards>`_ of ``snakemake``. Wildcards allow to generalise a rule to produce all files that follow a `regular expression <https://en.wikipedia.org/wiki/Regular_expression>`_ pattern, which e.g. defines one particular scenario. One can think of a wildcard as a parameter that shows up in the input/output file names and thereby determines which rules to run, what data to retrieve and what files to produce. **Details are explained in** :ref:`wildcards` **and** :ref:`scenario`.
 
-The model has several configuration options collected in the ``config.yaml`` file
-located in the root directory.
+The model also has several further configuration options collected in the ``config.yaml`` file
+located in the root directory, which that are not part of the scenarios. **All options are explained in detail in** :ref:`config`.
+
+Folder Structure
+================
+
+- ``data``: Includes input data that is not produced by any ``snakemake`` rule.
+- ``scripts``: Includes all the Python scripts executed by the ``snakemake`` rules.
+- ``resources``: Stores intermediate results of the workflow which can be picked up again by subsequent rules.
+- ``networks``: Stores intermediate, unsolved stages of the PyPSA network that describes the energy system model.
+- ``results``: Stores the solved PyPSA network data, summary files and plots.
+- ``benchmarks``: Stores ``snakemake`` benchmarks.
+- ``logs``: Stores log files about solving, including the solver output, console output and the output of a memory logger.
 
 System Requirements
 ===================

doc/plotting.rst

@@ -2,17 +2,35 @@
 Plotting and Summary
 ##########################################
 
+.. warning:: The corresponding code is currently under revision and has only minimal documentation.
+
+.. _plot_potentials:
+
+Plot Potentials
+===============
+
+.. automodule:: plot_p_nom_max
+
+.. _summary:
+
 Make Summary
 ============
 
 .. automodule:: make_summary
 
+.. _summary_plot:
+
 Plot Summary
 ============
 
+.. automodule:: plot_summary
+
+.. _map_plot:
+
 Plot Network
 ============
 
+.. automodule:: plot_network
 
 .. image:: img/tech-colors.png
     :align: center

doc/preparation.rst

@@ -2,84 +2,100 @@
 Preparing Networks
 ##########################################
 
-In detail this means it has to run the independent scripts,
-- `build_shapes` to generate GeoJSON files with country, exclusive economic zones and nuts3 shapes
-- `build_cutout` to prepare smaller weather data portions from ERA5 for cutout `europe-2013-era5` and SARAH for cutout `europe-2013-sarah`.
+The preparation process of the PyPSA-Eur energy system model consists of a group of ``snakemake``
+rules which are briefly outlined and explained in detail in the sections below:
 
-With these and the externally extracted `ENTSO-E online map topology`, it can build the PyPSA basis model
-- `base_network` stored at `networks/base.nc` with all `buses`, HVAC `lines` and HVDC `links`, and in
-- `build_bus_regions` determine the Voronoi cell of each substation.
+- ``build_shapes`` generates GeoJSON files with shapes of the countries, exclusive economic zones and `NUTS3 <https://en.wikipedia.org/wiki/Nomenclature_of_Territorial_Units_for_Statistics>`_ areas.
+- ``build_cutout`` prepares smaller weather data portions from `ERA5 <https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5>`_ for cutout ``europe-2013-era5`` and SARAH for cutout ``europe-2013-sarah``.
 
-Then it hands these over to the scripts for generating renewable and hydro feedin data,
-- `build_hydro_profile` for the hourly hydro energy availability,
-- `build_renewable_potentials` for the landuse/natura2000 constrained installation potentials for PV and wind,
-- `build_renewable_profiles` for the PV and wind hourly capacity factors in each Voronoi cell.
-- `build_powerplants` uses [powerplantmatching](https://github.com/FRESNA/powerplantmatching) to determine today's thermal power plant capacities and then locates the closest substation for each powerplant.
+With these and the externally extracted ENTSO-E online map topology
+(``data/entsoegridkit``), it can build a base PyPSA network with the following rules:
 
-The central rule `add_electricity` then ties all the different data inputs together to a detailed PyPSA model stored in `networks/elec.nc`, containing:
+- ``base_network`` builds and stores the base network with all buses, HVAC lines and HVDC links, while
+- ``build_bus_regions`` determines `Voronoi cells <https://en.wikipedia.org/wiki/Voronoi_diagram>`_ for all substations.
 
-- Today's transmission topology and capacities (optionally including lines which are under construction according to the config settings `lines: under_construction` and `links: under_construction`)
-- Today's thermal and hydro generation capacities (for the technologies listed in the config setting `electricity: conventional_carriers`)
-- Today's load time-series (upsampled according to population and gross domestic product)
+Then the process continues by calculating conventional power plant capacities, potentials, and per-unit availability time series for variable renewable energy carriers and hydro power plants with the following rules:
 
-It further adds extendable `generators` and `storage_units` with *zero* capacity for
-- wind and pv installations with today's locational, hourly wind and solar pv capacity factors (but **no** capacities)
-- long-term hydrogen and short-term battery storage units (if listed in `electricity: extendable_carriers`)
-- additional open-cycle gas turbines (if `OCGT` is listed in `electricity: extendable_carriers`)
+- ``build_powerplants`` for today's thermal power plant capacities using `powerplantmatching <https://github.com/FRESNA/powerplantmatching>`_ allocating these to the closest substation for each powerplant,
+- ``build_renewable_potentials`` for the installation potentials for solar panels, onshore and offshore wind turbines constrained by landuse restrictions and natural protection areas,
+- ``build_renewable_profiles`` for the hourly capacity factors in each substation's Voronoi cell for PV, onshore and offshore wind, and
+- ``build_hydro_profile`` for the hourly per-unit hydro power availability time series.
 
-.. each rule description should have a list of parameters
-.. from the config.yaml that affect this rule.
+The central rule ``add_electricity`` then ties all the different data inputs
+together into a detailed PyPSA network stored in ``networks/elec.nc``.
+
+.. _shapes:
 
 Build Shapes
 =============================
 
 .. automodule:: build_shapes
 
+.. _cutout:
+
 Build Cutout
 =============================
 
 .. automodule:: build_cutout
 
+.. _links:
+
 Prepare HVDC Links
 =============================
 
 .. automodule:: prepare_links_p_nom
 
+.. _base:
+
 Base Network
 =============================
 
 .. automodule:: base_network
 
+.. _busregions:
+
 Build Bus Regions
 =============================
 
 .. automodule:: build_bus_regions
 
-Build Country Full Load Hours
-=============================
-
-.. automodule:: build_country_flh
-
-Build Hydro Profile
-=============================
-
-.. automodule:: build_hydro_profile
+.. _natura:
 
 Build Natura Raster
 =============================
 
 .. automodule:: build_natura_raster
 
-Build Renewable Profiles
-========================
+.. _flh:
 
-.. automodule:: build_renewable_profiles
+Build Country Full Load Hours
+=============================
+
+.. automodule:: build_country_flh
+
+.. _powerplants:
 
 Build Power Plants
 =============================
 
 .. automodule:: build_powerplants
 
+.. _renewableprofiles:
+
+Build Renewable Profiles
+========================
+
+.. automodule:: build_renewable_profiles
+
+.. _hydroprofiles:
+
+Build Hydro Profile
+=============================
+
+.. automodule:: build_hydro_profile
+
+.. _electricity:
+
 Add Electricity
 =============================
 

doc/references.bib

@@ -0,0 +1,115 @@
+@article{PyPSAEur,
+        author = "Jonas Hoersch and Fabian Hofmann and David Schlachtberger and Tom Brown",
+        title = "PyPSA-Eur: An open optimisation model of the European transmission system",
+        journal = "Energy Strategy Reviews",
+        volume = "22",
+        pages = "207 - 215",
+        year = "2018",
+        issn = "2211-467X",
+        doi = "10.1016/j.esr.2018.08.012",
+        eprint = "1806.01613"
+    }
+
+@article{brown2019sectoral,
+  title={Sectoral Interactions as Carbon Dioxide Emissions Approach Zero in a Highly-Renewable European Energy System},
+  author={Brown, Tom and Sch{\"a}fer, Mirko and Greiner, Martin},
+  journal={Energies},
+  volume={12},
+  number={6},
+  pages={1032},
+  year={2019},
+  publisher={Multidisciplinary Digital Publishing Institute}
+}
+
+@article{neumann2019heuristics,
+  title={Heuristics for Transmission Expansion Planning in Low-Carbon Energy System Models},
+  author={Neumann, Fabian and Brown, Tom},
+  journal={arXiv preprint arXiv:1907.10548},
+  year={2019}
+}
+
+@article{weber2019counter,
+  title={Counter-intuitive behaviour of energy system models under CO2 caps and prices},
+  author={Weber, Juliane and Heinrichs, Heidi Ursula and Gillessen, Bastian and Schumann, Diana and H{\"o}rsch, Jonas and Brown, Tom and Witthaut, Dirk},
+  journal={Energy},
+  volume={170},
+  pages={22--30},
+  year={2019},
+  publisher={Elsevier}
+}
+
+@phdthesis{horsch2018spatial,
+  title={Spatial Scaling in Renewable Energy Networks},
+  author={H{\"o}rsch, Jonas},
+  year={2018},
+  school={Johann Wolfgang Goethe-Universit{\"a}t Frankfurt am Main}
+}
+
+@inproceedings{tranberg2018flow,
+  title={Flow-based analysis of storage usage in a low-carbon European electricity scenario},
+  author={Tranberg, Bo and Sch{\"a}fer, Mirko and Brown, Tom and H{\"o}rsch, Jonas and Greiner, Martin},
+  booktitle={2018 15th International Conference on the European Energy Market (EEM)},
+  pages={1--5},
+  year={2018},
+  organization={IEEE}
+}
+
+@article{pagnier2018disturbance,
+  title={Disturbance propagation, inertia location and slow modes in large-scale high voltage power grids},
+  author={Pagnier, Laurent and Jacquod, Philippe},
+  journal={arXiv preprint arXiv:1810.04982},
+  year={2018}
+}
+
+@article{victoria2019role,
+  title={The role of storage technologies throughout the decarbonisation of the sector-coupled European energy system},
+  author={Victoria, Marta and Zhu, Kun and Brown, Tom and Andresen, Gorm B and Greiner, Martin},
+  journal={arXiv preprint arXiv:1906.06936},
+  year={2019}
+}
+
+@article{brownasynergies,
+  title={Synergies of sector coupling and transmission extension in a cost-optimised, highly renewable European energy system},
+  author={Browna, T and Schlachtbergera, D and Kiesa, A and Schramma, S and Greinerb, M}
+}
+
+@article{gardumi2019representation,
+  title={Representation of Balancing Options for Variable Renewables in Long-Term Energy System Models: An Application to OSeMOSYS},
+  author={Gardumi, Francesco and Welsch, Manuel and Howells, Mark and Colombo, Emanuela},
+  journal={Energies},
+  volume={12},
+  number={12},
+  pages={2366},
+  year={2019},
+  publisher={Multidisciplinary Digital Publishing Institute}
+}
+
+@article{zhu2019impact,
+  title={Impact of CO2 prices on the design of a highly decarbonised coupled electricity and heating system in Europe},
+  author={Zhu, Kun and Victoria, Marta and Brown, Tom and Andresen, Gorm B and Greiner, Martin},
+  journal={Applied energy},
+  volume={236},
+  pages={622--634},
+  year={2019},
+  publisher={Elsevier}
+}
+
+@article{schlott2018impact,
+  title={The impact of climate change on a cost-optimal highly renewable European electricity network},
+  author={Schlott, Markus and Kies, Alexander and Brown, Tom and Schramm, Stefan and Greiner, Martin},
+  journal={Applied energy},
+  volume={230},
+  pages={1645--1659},
+  year={2018},
+  publisher={Elsevier}
+}
+
+@article{gotzens2019performing,
+  title={Performing energy modelling exercises in a transparent way-The issue of data quality in power plant databases},
+  author={Gotzens, Fabian and Heinrichs, Heidi and H{\"o}rsch, Jonas and Hofmann, Fabian},
+  journal={Energy Strategy Reviews},
+  volume={23},
+  pages={1--12},
+  year={2019},
+  publisher={Elsevier}
+}

doc/release_notes.rst

@@ -2,8 +2,9 @@
 Release Notes
 ##########################################
 
-Section A
-=========
+PyPSA-Eur 0.1.0 (DATE)
+======================
 
-Section B
-=========
+This is the first release of PyPSA-Eur. It now features:
+
+* Documentation on installation, workflows and configuration settings.

doc/simplification.rst

@@ -4,56 +4,32 @@
 Simplifying Networks
 ##########################################
 
-The additional rules prepare approximations of the full model, in which generation, storage and transmission capacities can be co-optimized
-- `simplify_network` transforms the transmission grid to a 380 kV-only equivalent network, while
-- `cluster_network` uses a kmeans based clustering technique to partition the network into a certain number of zones and then reduce the network to a representation with one bus per zone.
+The simplification ``snakemake`` rules prepare **approximations** of the full model, for which it is computationally viable to co-optimize generation, storage and transmission capacities.
+
+- ``simplify_network`` transforms the transmission grid to a 380 kV only equivalent network, while
+- ``cluster_network`` uses a `k-means <https://en.wikipedia.org/wiki/K-means_clustering>`_ based clustering technique to partition the network into a given number of zones and then reduce the network to a representation with one bus per zone.
 
 The simplification and clustering steps are described in detail in the paper
-[The role of spatial scale in joint optimisations of generation and transmission for European highly renewable scenarios](https://arxiv.org/abs/1705.07617), 2017, [arXiv:1705.07617](https://arxiv.org/abs/1705.07617), [doi:10.1109/EEM.2017.7982024](https://doi.org/10.1109/EEM.2017.7982024).
 
+- Jonas Hörsch and Tom Brown. `The role of spatial scale in joint optimisations of generation and transmission for European highly renewable scenarios <https://arxiv.org/abs/1705.07617>`_), *14th International Conference on the European Energy Market*, 2017. `arXiv:1705.07617 <https://arxiv.org/abs/1705.07617>`_, `doi:10.1109/EEM.2017.7982024 <https://doi.org/10.1109/EEM.2017.7982024>`_.
+
+.. _simplify:
 
 Simplify Network
 ================
 
-The rule simplify_network does up to four things:
-
-1. Create an equivalent transmission network in which all voltage levels are mapped to the 380 kV level by the function ``simplify_network(...)``.
-
-2. DC only sub-networks that are connected at only two buses to the AC network are reduced to a single representative link by the function ``simplify_links(...)``. The components attached to buses in between are moved to the nearest endpoint. The grid connection cost of offshore wind generators are added to the captial costs of the generator.
-
-3. Stub lines and links, i.e. dead-ends of the network, are sequentially removed from the network by the function ``remove_stubs(...)``. Components are moved along.
-
-4. If a number was provided after the s (as in elec_s500_...), the network is clustered to this number of clusters with the routines from the cluster_network rule by the function cluster. This step is usually skipped.
+.. automodule:: simplify_network
 
+.. _cluster:
 
 Cluster Network
 ===============
 
-The rule cluster_network instead clusters the network to a given number of buses.
-
-    -Why is this cluster function used?
-    -Why the user can set a number behind the elec_sXXX for simplification?
-
-As you found out for yourself, elec_s100_50.nc for example is a network in which simplify_network clusters the network to 100 buses and in a second step cluster_network reduces it down to 50 buses.
-
-Well, let me provide a use-case where this makes sense:
-
-In preliminary tests, it turns out, that the principal effect of changing spatial resolution is actually only partially due to the transmission network. It is more important to differentiate between wind generators with higher capacity factors from those with lower capacity factors, ie to have a higher spatial resolution in the renewable generation than in the number of buses.
-
-This two-step clustering can take advantage of that fact (and allows to study it)
-by looking at networks like networks/elec_s100_50m.nc (note the additional m in the cluster wildcard). For this example simplify_network clusters to 100 buses and then cluster_network clusters to 50m buses, which means 50 buses for the network topology but only moving instead of aggregating the generators to the clustered buses. So in this network you still have up to 100 different wind generators, 2 at each bus on average.
-
-In combination these two features allow you to study the spatial resolution of the transmission network separately from the spatial resolution of renewable generators. Beware: There is no clear evidence telling you what is a good representation of the full model. These options are under active study.
-
-    Why we have a cluster function inside of the simplification method?
-
-Why are you asking three times the same question?
-
-    Is it possible to run the model without the simplification method / rule?
-    I tryed to run the snakemake without the s for simplification.
-
-No, the network clustering methods in PyPSA's networkclustering module don't work reliably with multiple voltage levels and transformers. If it is somehow necessary for you we could include switches to make Step 2 and 3 optional as well. But that's about it.
+.. automodule:: cluster_network
 
+.. _prepare:
 
 Prepare Network
 ===============
+
+.. automodule:: prepare_network

doc/solving.rst

@@ -2,18 +2,25 @@
 Solving Networks
 ##########################################
 
-After generating the network it can be solved by using 'solve_all_elec_networks'. This runs the following rules:
-- 'cluster_network'
-- 'prepare_network'
-- 'solve_all_elec_networks'
-- 'solve_network'
+After generating and simplifying the networks they can be solved through the rule ``solve_network``  by using the collection rule ``solve_all_elec_networks``. Moreover, networks can be solved for another focus with the derivative rules ``trace_solve_network`` to log changes during iterations and ``solve_operations_network`` for dispatch-only analyses on an already solved network.
 
+.. _solve:
 
 Solve Network
 =============
 
+.. automodule:: solve_network
+
+.. _trace_solve:
+
 Trace Solve Network
 ===================
 
+.. automodule:: trace_solve_network
+
+.. _solve_operations:
+
 Solve Operations Network
 ========================
+
+.. automodule:: solve_operations_network

doc/wildcards.rst

@@ -1,26 +1,67 @@
+.. _wildcards:
+
 #########
 Wildcards
 #########
 
 Detailed explanations of how wildcards work in ``snakemake`` can be found in the `relevant section of the documentation <https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#wildcards>`_.
 
-The ``simpl`` wildcard
-======================
+.. _network:
 
-The ``ll`` wildcard
-===================
+The ``{network}`` wildcard
+==========================
 
-The ``clusters`` wildcard
-=========================
+The ``{network}`` wildcard specifies the considered energy sector(s)
+and, as currently only ``elec`` (for electricity) is included,
+it currently represents rather a placeholder wildcard to facilitate
+future extensions including multiple energy sectors at once.
 
-The ``network`` wildcard
+.. _simpl:
+
+The ``{simpl}`` wildcard
 ========================
 
+The ``{simpl}`` wildcard specifies number of buses a detailed network model should be pre-clustered to in the rule ``simplify_network`` (before ``cluster_network``).
 
-The ``opts`` wildcard
+.. seealso::
+    :mod:`simplify_network`
+
+.. _clusters:
+
+The ``{clusters}`` wildcard
+===========================
+
+The ``{clusters}`` wildcard specifies the number of buses a detailed network model should be reduced to in the rule ``cluster_network``.
+The number of clusters must be lower than the total number of nodes
+and higher than the number of countries. However, a country counts twice if
+it has two asynchronous subnetworks (e.g. Denmark or Italy).
+
+If an `m` is placed behind the number of clusters (e.g. ``100m``), generators are only moved to the clustered buses but not aggregated by carrier; i.e. the clustered bus may have more than one e.g. wind generator.
+
+.. seealso::
+    :mod:`cluster_network`
+
+.. _ll:
+
+The ``{ll}`` wildcard
 =====================
 
-The ``opts`` wildcard triggers optional constraints, which are activated in either ``prepare_network`` or the ``solve_network`` step. It may hold multiple triggers separated by ``-``, i.e. ``Co2L-3H`` contains the ``Co2L`` trigger and the ``3H`` switch. There are currently:
+``v`` (volume) or ``c`` (cost)
+
+``opt`` or a float bigger than one (e.g. 1.25)
+
+.. seealso::
+    :mod:`prepare_network`
+
+.. _opts:
+
+The ``{opts}`` wildcard
+=======================
+
+The ``{opts}`` wildcard triggers optional constraints, which are activated in either
+``prepare_network`` or the ``solve_network`` step.
+It may hold multiple triggers separated by ``-``, i.e. ``Co2L-3H`` contains the
+``Co2L`` trigger and the ``3H`` switch. There are currently:
 
 
 .. csv-table::
@@ -28,27 +69,69 @@ The ``opts`` wildcard triggers optional constraints, which are activated in eith
    :widths: 10,20,10,10
    :file: configtables/opts.csv
 
+.. seealso::
+    :mod:`prepare_network`, :mod:`solve_network`
 
-The ``country`` wildcard
-========================
+.. _country:
 
-The rules ``make_summary`` (generating summaries of all or a subselection of the solved networks) and ``plot_p_nom_max`` (for plotting the cumulative generation potentials for renewable technologies) can be narrowed to individual countries using the ``country`` wildcard.
+The ``{country}`` wildcard
+==========================
 
-If ``country = all``, then the rule acts on the network for all countries defined in ``config.yaml``. If otherwise ``country = DE`` or another country code, then the network is narrowed to buses of this country for the rule. For example to get a summary of the energy generated in Germany (in the solution for Europe) use:
+The rules ``make_summary`` and ``plot_summary`` (generating summaries of all or a subselection
+of the solved networks) as well as ``plot_p_nom_max`` (for plotting the cumulative
+generation potentials for renewable technologies) can be narrowed to
+individual countries using the ``{country}`` wildcard.
 
+If ``country = all``, then the rule acts on the network for all countries
+defined in ``config.yaml``. If otherwise ``country = DE`` or another 2-letter
+country code, then the network is narrowed to buses of this country
+for the rule. For example to get a summary of the energy generated
+in Germany (in the solution for Europe) use:
 
 .. code:: bash
 
     snakemake results/summaries/elec_s_all_lall_Co2L-3H_DE
 
-The ``cutout`` wildcard
+.. seealso::
+    :mod:`make_summary`, :mod:`plot_summary`, :mod:`plot_p_nom_max`
+
+.. _cutout_wc:
+
+The ``{cutout}`` wildcard
+=========================
+
+.. seealso::
+    :mod:`build_cutout`
+
+.. _technology:
+
+The ``{technology}`` wildcard
+=============================
+
+.. seealso::
+    :mod:`build_renewable_profiles`, :mod:`plot_p_nom_max`, :mod:`build_country_flh`
+
+.. _attr:
+
+The ``{attr}`` wildcard
 =======================
 
-The ``technology`` wildcard
-===========================
+The ``{attr}`` wildcard specifies which attribute are used for size representations of network components on a map plot produced by the rule ``plot_network``. While it might be extended in the future, ``{attr}`` currently only supports plotting of ``p_nom``.
 
-The ``attr`` wildcard
-=====================
+.. seealso::
+    :mod:`plot_network`
 
-The ``ext`` wildcard
-====================
+.. _ext:
+
+The ``{ext}`` wildcard
+======================
+
+The ``{ext}`` wildcard specifies the file type of the figures the rule ``plot_network``, ``plot_summary``, and ``plot_p_nom_max`` produce. Typical examples are ``pdf`` and ``png``. The list of supported file formats depends on the used backend. To query the supported file types on your system, issue:
+
+.. code:: python
+
+    import matplotlib.pyplot as plt
+    plt.gcf().canvas.get_supported_filetypes()
+
+.. seealso::
+    :mod:`plot_network`, :mod:`plot_summary`, :mod:`plot_p_nom_max`

environment_docs.yaml

@@ -44,4 +44,4 @@ dependencies:
     - git+https://github.com/PyPSA/geokit.git#egg=geokit
     - git+https://github.com/FRESNA/powerplantmatching.git#egg=powerplantmatching
     - sphinx
-    - sphinx_rtd_theme
+    - sphinx_rtd_theme

scripts/add_electricity.py

@@ -1,6 +1,90 @@
 # coding: utf-8
 """
-Adds electrical generators and storage units to base network
+Adds electrical generators and storage units to a base network.
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    costs:
+        year:
+        USD2013_to_EUR2013:
+        dicountrate:
+        emission_prices:
+
+    electricity:
+        max_hours:
+        marginal_cost:
+        capital_cost:
+        conventional_carriers:
+        co2limit:
+        extendable_carriers:
+            Generator:
+            StorageUnit:
+        estimate_renewable_capacities_from_capacity_stats:
+
+    load:
+        scaling_factor:
+
+    renewable: (keys)
+        hydro:
+            carriers:
+            hydro_max_hours:
+            hydro_capital_cost:
+
+    lines:
+        length_factor:
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at :ref:`costs_cf`, :ref:`electricity_cf`, :ref:`load_cf`, :ref:`renewable_cf`, :ref:`lines_cf`
+
+Inputs
+------
+
+- ``data/costs.csv``: The database of cost assumptions for all included technologies for specific years from various sources; e.g. discount rate, lifetime, investment (CAPEX), fixed operation and maintenance (FOM), variable operation and maintenance (VOM), fuel costs, efficiency, carbon-dioxide intensity.
+- ``data/bundle/hydro_capacities.csv``: Hydropower plant store/discharge power capacities, energy storage capacity, and average hourly inflow by country.
+
+    .. image:: img/hydrocapacities.png
+        :scale: 34 %
+
+- ``data/geth2015_hydro_capacities.csv``: alternative to capacities above; NOT CURRENTLY USED!
+- ``data/bundle/time_series_60min_singleindex_filtered.csv``: Hourly per-country load profiles since 2010 from the `ENTSO-E statistical database <https://www.entsoe.eu/data/power-stats/hourly_load/>`_
+
+    .. image:: img/load-box.png
+        :scale: 33 %
+
+    .. image:: img/load-ts.png
+        :scale: 33 %
+
+- ``resources/regions_onshore.geojson``: confer :ref:`busregions`
+- ``resources/nuts3_shapes.geojson``: confer :ref:`shapes`
+- ``resources/powerplants.csv``: confer :ref:`powerplants`
+- ``resources/profile_{}.nc``: all technologies in ``config["renewables"].keys()``, confer :ref:`renewableprofiles`.
+- ``networks/base.nc``: confer :ref:`base`
+
+Outputs
+-------
+
+- ``networks/elec.nc``:
+
+    .. image:: img/elec.png
+            :scale: 33 %
+
+Description
+-----------
+
+The rule ``add_electricity`` ties all the different data inputs from the preceding rules together into a detailed PyPSA network that is stored in ``networks/elec.nc``. It includes:
+
+- today's transmission topology and transfer capacities (optionally including lines which are under construction according to the config settings ``lines: under_construction`` and ``links: under_construction``),
+- today's thermal and hydro power generation capacities (for the technologies listed in the config setting ``electricity: conventional_carriers``), and
+- today's load time-series (upsampled in a top-down approach according to population and gross domestic product)
+
+It further adds extendable ``generators`` and ``storage_units`` with **zero** capacity for
+
+- photovoltaic, onshore and AC- as well as DC-connected offshore wind installations with today's locational, hourly wind and solar capacity factors (but **no** current capacities),
+- long-term hydrogen and short-term battery storage units (if listed in the config setting ``electricity: extendable_carriers``), and
+- additional open- and combined-cycle gas turbines (if ``OCGT`` and/or ``CCGT`` is listed in the config setting ``electricity: extendable_carriers``)
 """
 
 import logging

scripts/base_network.py

@@ -1,6 +1,61 @@
 # coding: utf-8
 """
-Creates the network topology from ENTSO-E map extracts as a PyPSA network
+Creates the network topology from the ENTSO-E map extracts as a PyPSA network.
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    snapshots:
+
+    countries:
+
+    electricity:
+        voltages:
+
+    lines:
+        types:
+        s_max_pu:
+        under_construction:
+        
+    links:
+        p_max_pu:
+        under_construction:
+        include_tyndp:
+        
+    transformers:
+        x:
+        s_nom:
+        type:
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`snapshots_cf`, :ref:`toplevel_cf`, :ref:`electricity_cf`, :ref:`load_cf`,
+    :ref:`lines_cf`, :ref:`links_cf`, :ref:`transformers_cf`
+        
+Inputs
+------
+
+- ``data/entsoegridkit``:  Extract from the geographical vector data of the online `ENTSO-E Interactive Map <https://www.entsoe.eu/data/map/>`_ by the `GridKit <https://github.com/pypsa/gridkit>`_ toolkit. 
+- ``data/parameter_corrections.yaml``: Corrections for ``data/entsoegridkit``
+- ``data/links_p_nom.csv``: confer :ref:`links`
+- ``data/links_tyndp.csv``: List of projects in the `TYNDP 2018 <https://tyndp.entsoe.eu/tyndp2018/>`_ that are at least *in permitting* with fields for start- and endpoint (names and coordinates), length, capacity, construction status, and project reference ID.
+- ``resources/country_shapes.geojson``: confer :ref:`shapes`
+- ``resources/offshore_shapes.geojson``: confer :ref:`shapes`
+- ``resources/europe_shape.geojson``: confer :ref:`shapes`
+
+Outputs
+-------
+
+- ``networks/base.nc``
+
+    .. image:: img/base.png
+        :scale: 33 %
+
+Description
+-----------
+
 """
 
 import yaml

scripts/build_bus_regions.py

@@ -1,5 +1,40 @@
 """
-Creates onshore and offshore Voronoi shapes for each bus
+Creates Voronoi shapes for each bus representing both onshore and offshore regions.
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    countries:
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`toplevel_cf`
+
+Inputs
+------
+
+- ``resources/country_shapes.geojson``: confer :ref:`shapes`
+- ``resources/offshore_shapes.geojson``: confer :ref:`shapes`
+- ``networks/base.nc``: confer :ref:`base`
+
+Outputs
+-------
+
+- ``resources/regions_onshore.geojson``:
+
+    .. image:: img/regions_onshore.png
+        :scale: 33 %
+
+- ``resources/regions_offshore.geojson``:
+
+    .. image:: img/regions_offshore.png
+        :scale: 33 %
+
+Description
+-----------
+
 """
 
 import os

scripts/build_country_flh.py

@@ -1,6 +1,60 @@
 #!/usr/bin/env python
 """
-Create csv files and plots for comparing per country full load hours of renewable time-series
+Create ``.csv`` files and plots for comparing per country full load hours of renewable time series.
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    snapshots:
+
+    renewable:
+        {technology}:
+            cutout:
+            resource:
+            correction_factor:
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`snapshots_cf`, :ref:`renewable_cf`
+
+Inputs
+------
+
+- ``data/bundle/corine/g250_clc06_V18_5.tif``: `CORINE Land Cover (CLC) <https://land.copernicus.eu/pan-european/corine-land-cover>`_ inventory on `44 classes <https://wiki.openstreetmap.org/wiki/Corine_Land_Cover#Tagging>`_ of land use (e.g. forests, arable land, industrial, urban areas).
+
+    .. image:: img/corine.png
+        :scale: 33 %
+
+- ``data/bundle/GEBCO_2014_2D.nc``:  A `bathymetric <https://en.wikipedia.org/wiki/Bathymetry>`_ data set with a global terrain model for ocean and land at 15 arc-second intervals by the `General Bathymetric Chart of the Oceans (GEBCO) <https://www.gebco.net/data_and_products/gridded_bathymetry_data/>`_.
+
+    .. image:: img/gebco_2019_grid_image.jpg
+        :scale: 50 %
+
+    **Source:** `GEBCO <https://www.gebco.net/data_and_products/images/gebco_2019_grid_image.jpg>`_
+
+- ``data/pietzker2014.xlsx``: NOT IN DATA BUNDLE!
+- ``resources/natura.tiff``: confer :ref:`natura`
+- ``resources/country_shapes.geojson``: confer :ref:`shapes`
+- ``resources/offshore_shapes.geojson``: confer :ref:`shapes`
+- ``resources/regions_onshore.geojson``: (if not offshore wind), confer :ref:`busregions`
+- ``resources/regions_offshore.geojson``: (if offshore wind), :ref:`busregions`
+- ``"cutouts/" + config["renewable"][{technology}]['cutout']``: :ref:`cutout`
+- ``networks/base.nc``: :ref:`base`
+
+Outputs
+-------
+
+- ``resources/country_flh_area_{technology}.csv``:
+- ``resources/country_flh_aggregated_{technology}.csv``:
+- ``resources/country_flh_uncorrected_{technology}.csv``:
+- ``resources/country_flh_{technology}.pdf``:
+- ``resources/country_exclusion_{technology}``:
+
+Description
+-----------
+
 """
 
 import os

scripts/build_cutout.py

@@ -1,5 +1,84 @@
 """
-Create cutouts configured in `atlite` config section
+Create cutouts with `atlite <https://atlite.readthedocs.io/en/latest/>`_.
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    atlite:
+        nprocesses:
+        cutouts:
+            {cutout}:
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`atlite_cf`
+
+Inputs
+------
+
+*None*
+
+Outputs
+-------
+
+- ``cutouts/{cutout}``: weather data from either the `ERA5 <https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5>`_
+  reanalysis weather dataset or `SARAH-2 <https://wui.cmsaf.eu/safira/action/viewProduktSearch>`_
+  satellite-based historic weather data with the following structure:
+
+**ERA5 cutout:**
+
+    ===================  ==========  ==========  =========================================================
+    Field                Dimensions  Unit        Description
+    ===================  ==========  ==========  =========================================================
+    pressure             time, y, x  Pa          Surface pressure
+    -------------------  ----------  ----------  ---------------------------------------------------------
+    temperature          time, y, x  K           Air temperature 2 meters above the surface.
+    -------------------  ----------  ----------  ---------------------------------------------------------
+    soil temperature     time, y, x  K           Soil temperature between 1 meters and 3 meters
+                                                 depth (layer 4).
+    -------------------  ----------  ----------  ---------------------------------------------------------
+    influx_toa           time, y, x  Wm**-2      Top of Earth's atmosphere TOA incident solar radiation
+    -------------------  ----------  ----------  ---------------------------------------------------------
+    influx_direct        time, y, x  Wm**-2      Total sky direct solar radiation at surface
+    -------------------  ----------  ----------  ---------------------------------------------------------
+    runoff               time, y, x  m           `Runoff <https://en.wikipedia.org/wiki/Surface_runoff>`_
+                                                 (volume per area)
+    -------------------  ----------  ----------  ---------------------------------------------------------
+    roughness            y, x        m           Forecast surface roughness
+                                                 (`roughness length <https://en.wikipedia.org/wiki/Roughness_length>`_)
+    -------------------  ----------  ----------  ---------------------------------------------------------
+    height               y, x        m           Surface elevation above sea level
+    -------------------  ----------  ----------  ---------------------------------------------------------
+    albedo               time, y, x  --          `Albedo <https://en.wikipedia.org/wiki/Albedo>`_
+                                                 measure of diffuse reflection of solar radiation.
+                                                 Calculated from relation between surface solar radiation
+                                                 downwards (Jm**-2) and surface net solar radiation
+                                                 (Jm**-2). Takes values between 0 and 1.
+    -------------------  ----------  ----------  ---------------------------------------------------------
+    influx_diffuse       time, y, x  Wm**-2      Diffuse solar radiation at surface.
+                                                 Surface solar radiation downwards minus
+                                                 direct solar radiation.
+    -------------------  ----------  ----------  ---------------------------------------------------------
+    wnd100m              time, y, x  ms**-1      Wind speeds at 100 meters (regardless of direction)
+    ===================  ==========  ==========  =========================================================
+
+    .. image:: img/era5.png
+        :scale: 40 %    
+    
+A **SARAH-2 cutout** can be used to amend the fields ``temperature``, ``influx_toa``, ``influx_direct``, ``albedo``,
+``influx_diffuse`` of ERA5 using satellite-based radiation observations.
+    
+    .. image:: img/sarah.png
+        :scale: 40 %
+
+.. seealso::
+    For details on the weather data read the `atlite documentation <https://atlite.readthedocs.io/en/latest/>`_.
+
+Description
+-----------
+
 """
 import os
 import atlite

scripts/build_hydro_profile.py

@@ -1,10 +1,57 @@
 #!/usr/bin/env python
 """
-Build hydroelectric inflow time-series for each country
+Build hydroelectric inflow time-series for each country.
 
-See also
---------
-build_renewable_profiles
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    countries:
+
+    renewable:
+        hydro:
+            cutout:
+            clip_min_inflow:
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`toplevel_cf`, :ref:`renewable_cf`
+
+Inputs
+------
+
+- ``data/bundle/EIA_hydro_generation_2000_2014.csv``: Hydroelectricity net generation per country and year (`EIA <https://www.eia.gov/beta/international/data/browser/#/?pa=000000000000000000000000000000g&c=1028i008006gg6168g80a4k000e0ag00gg0004g800ho00g8&ct=0&ug=8&tl_id=2-A&vs=INTL.33-12-ALB-BKWH.A&cy=2014&vo=0&v=H&start=2000&end=2016>`_)
+
+    .. image:: img/hydrogeneration.png
+        :scale: 33 %
+
+- ``resources/country_shapes.geojson``: confer :ref:`shapes`
+- ``"cutouts/" + config["renewable"]['hydro']['cutout']``: confer :ref:`cutout`
+
+Outputs
+-------
+
+- ``resources/profile_hydro.nc``:
+
+    ===================  ================  =========================================================
+    Field                Dimensions        Description
+    ===================  ================  =========================================================
+    inflow               countries, time   Inflow to the state of charge (in MW),
+                                           e.g. due to river inflow in hydro reservoir.
+    ===================  ================  =========================================================
+
+    .. image:: img/inflow-ts.png
+        :scale: 33 %
+    
+    .. image:: img/inflow-box.png
+        :scale: 33 %
+
+Description
+-----------
+
+.. seealso::
+    :mod:`build_renewable_profiles`
 """
 
 import os

scripts/build_natura_raster.py

@@ -1,5 +1,38 @@
 """
-Rasters the vector data of the NATURA2000 data onto all cutout regions
+Rasters the vector data of the `Natura 2000 <https://en.wikipedia.org/wiki/Natura_2000>`_ natural protection areas onto all cutout regions.
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    renewable:
+        {technology}:
+            cutout:
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`renewable_cf`
+
+Inputs
+------
+
+- ``data/bundle/natura/Natura2000_end2015.shp``: `Natura 2000 <https://en.wikipedia.org/wiki/Natura_2000>`_ natural protection areas.
+
+    .. image:: img/natura.png
+        :scale: 33 %
+
+Outputs
+-------
+
+- ``resources/natura.tiff``: Rasterized version of `Natura 2000 <https://en.wikipedia.org/wiki/Natura_2000>`_ natural protection areas to reduce computation times.
+
+    .. image:: img/natura.png
+        :scale: 33 %
+
+Description
+-----------
+
 """
 
 import numpy as np

scripts/build_powerplants.py

@@ -1,6 +1,37 @@
 # coding: utf-8
 """
-Get conventional powerplants from `powerplantmatching`, assign to buses and create csv file
+Retrieves conventional powerplant capacities and locations from `powerplantmatching <https://github.com/FRESNA/powerplantmatching>`_, assigns these to buses and creates a ``.csv`` file.
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    enable:
+        powerplantmatching:
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`toplevel_cf`
+
+Inputs
+------
+
+- ``networks/base.nc``: confer :ref:`base`.
+
+Outputs
+-------
+
+- ``resource/powerplants.csv``: A list of conventional power plants (i.e. neither wind nor solar) with fields for name, fuel type, technology, country, capacity in MW, duration, commissioning year, retrofit year, latitude, longitude, and dam information as documented in the `powerplantmatching README <https://github.com/FRESNA/powerplantmatching/blob/master/README.md>`_; additionally it includes information on the closest substation/bus in ``networks/base.nc``.
+
+    .. image:: img/powerplantmatching.png
+        :scale: 30 %
+
+    **Source:** `powerplantmatching on GitHub <https://github.com/FRESNA/powerplantmatching>`_
+
+Description
+-----------
+
 """
 
 import logging

scripts/build_renewable_profiles.py

@@ -1,77 +1,153 @@
 #!/usr/bin/env python
-"""
-Summary
--------
-The script ``build_renewable_profiles.py`` calculates for each node several geographical properties:
-
-  1. the installable capacity (based on land-use)
-  2. the available generation time series (based on weather data) and
-  3. the average distance from the node for onshore wind, AC-connected offshore wind, DC-connected offshore wind and solar PV generators.
-  4. In addition for offshore wind it calculates the fraction of the grid connection which is under water.
+"""Calculates for each network node the
+(i) installable capacity (based on land-use), (ii) the available generation time
+series (based on weather data), and (iii) the average distance from the node for
+onshore wind, AC-connected offshore wind, DC-connected offshore wind and solar
+PV generators. In addition for offshore wind it calculates the fraction of the
+grid connection which is under water.
 
 .. note:: Hydroelectric profiles are built in script :mod:`build_hydro_profiles`.
 
 Relevant settings
 -----------------
 
-config.renewable (describes the parameters for onwind, offwind-ac, offwind-dc
-and solar)
-config.snapshots (describes the time dimensions of the selection of snapshots)
+.. code:: yaml
+
+    snapshots:
+
+    atlite:
+        nprocesses:
+
+    renewable:
+        {technology}:
+            cutout:
+            corine:
+            grid_codes:
+            distance:
+            natura:
+            max_depth:
+            max_shore_distance:
+            min_shore_distance:
+            capacity_per_sqkm:
+            correction_factor:
+            potential:
+            min_p_max_pu:
+            clip_p_max_pu:
+            resource:
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`snapshots_cf`, :ref:`atlite_cf`, :ref:`renewable_cf`
 
 Inputs
 ------
 
-base_network
-land-use shapes
-region shapes for onshore, offshore and countries
-cutout
+- ``data/bundle/corine/g250_clc06_V18_5.tif``: `CORINE Land Cover (CLC) <https://land.copernicus.eu/pan-european/corine-land-cover>`_ inventory on `44 classes <https://wiki.openstreetmap.org/wiki/Corine_Land_Cover#Tagging>`_ of land use (e.g. forests, arable land, industrial, urban areas).
+
+    .. image:: img/corine.png
+        :scale: 33 %
+
+- ``data/bundle/GEBCO_2014_2D.nc``: A `bathymetric <https://en.wikipedia.org/wiki/Bathymetry>`_ data set with a global terrain model for ocean and land at 15 arc-second intervals by the `General Bathymetric Chart of the Oceans (GEBCO) <https://www.gebco.net/data_and_products/gridded_bathymetry_data/>`_.
+
+    .. image:: img/gebco_2019_grid_image.jpg
+        :scale: 50 %
+
+    **Source:** `GEBCO <https://www.gebco.net/data_and_products/images/gebco_2019_grid_image.jpg>`_
+
+- ``resources/natura.tiff``: confer :ref:`natura`
+- ``resources/country_shapes.geojson``: confer :ref:`shapes`
+- ``resources/offshore_shapes.geojson``: confer :ref:`shapes`
+- ``resources/regions_onshore.geojson``: (if not offshore wind), confer :ref:`busregions`
+- ``resources/regions_offshore.geojson``: (if offshore wind), :ref:`busregions`
+- ``"cutouts/" + config["renewable"][{technology}]['cutout']``: :ref:`cutout`
+- ``networks/base.nc``: :ref:`base`
 
 Outputs
 -------
 
-profile_{tech}.nc for tech in [onwind,offwind-ac,offwind-dc,solar]
+- ``resources/profile_{technology}.nc`` with the following structure
 
-profile_{tech}.nc contains five common fields:
+    ===================  ==========  =========================================================
+    Field                Dimensions  Description
+    ===================  ==========  =========================================================
+    profile              bus, time   the per unit hourly availability factors for each node
+    -------------------  ----------  ---------------------------------------------------------
+    weight               bus         sum of the layout weighting for each node
+    -------------------  ----------  ---------------------------------------------------------
+    p_nom_max            bus         maximal installable capacity at the node (in MW)
+    -------------------  ----------  ---------------------------------------------------------
+    potential            y, x        layout of generator units at cutout grid cells inside the
+                                     Voronoi cell (maximal installable capacity at each grid
+                                     cell multiplied by capacity factor)
+    -------------------  ----------  ---------------------------------------------------------
+    average_distance     bus         average distance of units in the Voronoi cell to the
+                                     grid node (in km)
+    -------------------  ----------  ---------------------------------------------------------
+    underwater_fraction  bus         fraction of the average connection distance which is
+                                     under water (only for offshore)
+    ===================  ==========  =========================================================
 
-profile (bus x time) - the per unit hourly availability factors for each node
-weight (bus) - the sum of the layout weighting for each node
-p_nom_max (bus) - the maximal installable capacity at the node (in MW)
-potential (y,x) - the layout of generator units at cutout grid cells inside the
-voronoi cell (maximal installable capacity at each grid cell multiplied by the
-capacity factor)
-average_distance (bus) - the average distance of units in the voronoi cell to
-the grid node (in km)
+    - **profile**
 
-for offshore we also have:
+    .. image:: img/profile_ts.png
+        :scale: 33 %
+    
+    - **p_nom_max**
 
-underwater_fraction (bus) - the fraction of the average connection distance
-which is under water
+    .. image:: img/p_nom_max_hist.png
+        :scale: 33 %
+    
+    - **potential**
 
-Long description:
+    .. image:: img/potential_heatmap.png
+        :scale: 33 %
+    
+    - **average_distance**
+    
+    .. image:: img/distance_hist.png
+        :scale: 33 %
+    
+    - **underwater_fraction**
+    
+    .. image:: img/underwater_hist.png
+        :scale: 33 %
+
+Description
+-----------
+
+This script functions at two main spatial resolutions: the resolution of the
+network nodes and their `Voronoi cells
+<https://en.wikipedia.org/wiki/Voronoi_diagram>`_, and the resolution of the
+cutout grid cells for the weather data. Typically the weather data grid is
+finer than the network nodes, so we have to work out the distribution of
+generators across the grid cells within each Voronoi cell. This is done by
+taking account of a combination of the available land at each grid cell and the
+capacity factor there.
 
 First the script computes how much of the technology can be installed at each
-cutout grid cell and each node using the library `GLAES
-<https://github.com/FZJ-IEK3-VSA/glaes>`_. This uses the CORINE land use data,
-Natura2000 nature reserves and GEBCO for bathymetry.
+cutout grid cell and each node using the `GLAES
+<https://github.com/FZJ-IEK3-VSA/glaes>`_ library. This uses the CORINE land use data,
+Natura2000 nature reserves and GEBCO bathymetry data.
 
-To compute the layout of generators in each node's voronoi cell, the installable
-potential in each grid cell is multiplied with the capacity factor at each grid
-cell (since we assume more generators are installed at cells with a higher
-capacity factor).
+To compute the layout of generators in each node's Voronoi cell, the
+installable potential in each grid cell is multiplied with the capacity factor
+at each grid cell. This is done since we assume more generators are installed
+at cells with a higher capacity factor.
 
-This layout is then used to compute the generation availability time series from
-the atlite cutout.
+This layout is then used to compute the generation availability time series
+from the weather data cutout from ``atlite``.
 
 Two methods are available to compute the maximal installable potential for the
-node (`p_nom_max`): `simple` and `conservative`:
+node (`p_nom_max`): ``simple`` and ``conservative``:
 
-`simple` adds up the installable potentials of the individual grid cells (if the
-model comes close to this limit, then the time series may slightly overestimate
-production since we assumed the geographical distribution is proportional to
-capacity factor).
+- ``simple`` adds up the installable potentials of the individual grid cells.
+  If the model comes close to this limit, then the time series may slightly
+  overestimate production since it is assumed the geographical distribution is
+  proportional to capacity factor.
 
-`conservative` assertains the nodal limit by increasing capacities proportional
-to the layout until the limit of an individual grid cell is reached.
+- ``conservative`` assertains the nodal limit by increasing capacities
+  proportional to the layout until the limit of an individual grid cell is
+  reached.
 
 """
 

scripts/build_shapes.py

@@ -1,5 +1,66 @@
 """
-Create GIS shape files for countries on-shore and off-shore, europe and nuts3 regions
+Creates GIS shape files of the countries, exclusive economic zones and `NUTS3 <https://en.wikipedia.org/wiki/Nomenclature_of_Territorial_Units_for_Statistics>`_ areas.
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    countries:
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`toplevel_cf`
+
+Inputs
+------
+
+- ``data/bundle/naturalearth/ne_10m_admin_0_countries.shp``: World country shapes
+
+    .. image:: img/countries.png
+        :scale: 33 %
+
+- ``data/bundle/eez/World_EEZ_v8_2014.shp``: World `exclusive economic zones <https://en.wikipedia.org/wiki/Exclusive_economic_zone>`_ (EEZ)
+
+    .. image:: img/eez.png
+        :scale: 33 %
+
+- ``data/bundle/NUTS_2013_60M_SH/data/NUTS_RG_60M_2013.shp``: Europe NUTS3 regions
+
+    .. image:: img/nuts3.png
+        :scale: 33 %
+
+- ``data/bundle/nama_10r_3popgdp.tsv.gz``: Average annual population by NUTS3 region (`eurostat <http://appsso.eurostat.ec.europa.eu/nui/show.do?dataset=nama_10r_3popgdp&lang=en>`_)
+- ``data/bundle/nama_10r_3gdp.tsv.gz``: Gross domestic product (GDP) by NUTS 3 regions (`eurostat <http://appsso.eurostat.ec.europa.eu/nui/show.do?dataset=nama_10r_3gdp&lang=en>`_)
+- ``data/bundle/ch_cantons.csv``: Mapping between Swiss Cantons and NUTS3 regions
+- ``data/bundle/je-e-21.03.02.xls``: Population and GDP data per Canton (`BFS - Swiss Federal Statistical Office <https://www.bfs.admin.ch/bfs/en/home/news/whats-new.assetdetail.7786557.html>`_ )
+
+Outputs
+-------
+
+- ``resources/country_shapes.geojson``: country shapes out of country selection
+
+    .. image:: img/country_shapes.png
+        :scale: 33 %
+
+- ``resources/offshore_shapes.geojson``: EEZ shapes out of country selection
+
+    .. image:: img/offshore_shapes.png
+        :scale: 33 %
+
+- ``resources/europe_shape.geojson``: Shape of Europe including countries and EEZ
+
+    .. image:: img/europe_shape.png
+        :scale: 33 %
+        
+- ``resources/nuts3_shapes.geojson``: NUTS3 shapes out of country selection including population and GDP data.
+        
+    .. image:: img/nuts3_shapes.png
+        :scale: 33 %
+
+Description
+-----------
+
 """
 
 import os

scripts/cluster_network.py

@@ -1,6 +1,92 @@
 # coding: utf-8
 """
-Create networks clustered to `cluster` number of zones with aggregated buses, generators and transmission corridors
+Creates networks clustered to ``{cluster}`` number of zones with aggregated buses, generators and transmission corridors.
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    renewable: (keys)
+        {technology}:
+            potential:
+
+    solving:
+        solver:
+            name:
+
+    lines:
+        length_factor:
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`renewable_cf`, :ref:`solving_cf`, :ref:`lines_cf`
+
+Inputs
+------
+
+- ``resources/regions_onshore_{network}_s{simpl}.geojson``: confer :ref:`simplify`
+- ``resources/regions_offshore_{network}_s{simpl}.geojson``: confer :ref:`simplify`
+- ``resources/clustermaps_{network}_s{simpl}.h5``: confer :ref:`simplify`
+- ``networks/{network}_s{simpl}.nc``: confer :ref:`simplify`
+
+Outputs
+-------
+
+- ``resources/regions_onshore_{network}_s{simpl}_{clusters}.geojson``:
+
+    .. image:: img/regions_onshore_elec_s_X.png
+        :scale: 33 %
+
+- ``resources/regions_offshore_{network}_s{simpl}_{clusters}.geojson``:
+
+    .. image:: img/regions_offshore_elec_s_X.png
+        :scale: 33 %
+
+- ``resources/clustermaps_{network}_s{simpl}_{clusters}.h5``: Mapping of buses and lines from ``networks/elec_s{simpl}.nc`` to ``networks/elec_s{simpl}_{clusters}.nc``; has keys ['/busmap', '/busmap_s', '/linemap', '/linemap_negative', '/linemap_positive']
+- ``networks/{network}_s{simpl}_{clusters}.nc``: 
+
+    .. image:: img/elec_s_X.png
+        :scale: 40  %
+
+Description
+-----------
+
+.. note::
+
+    **Why is clustering used both in** ``simplify_network`` **and** ``cluster_network`` **?**
+    
+        Consider for example a network ``networks/elec_s100_50.nc`` in which
+        ``simplify_network`` clusters the network to 100 buses and in a second
+        step ``cluster_network``` reduces it down to 50 buses.
+
+        In preliminary tests, it turns out, that the principal effect of
+        changing spatial resolution is actually only partially due to the
+        transmission network. It is more important to differentiate between
+        wind generators with higher capacity factors from those with lower
+        capacity factors, i.e. to have a higher spatial resolution in the
+        renewable generation than in the number of buses.
+
+        The two-step clustering allows to study this effect by looking at
+        networks like ``networks/elec_s100_50m.nc``. Note the additional
+        ``m`` in the ``{cluster}`` wildcard. So in the example network
+        there are still up to 100 different wind generators.
+
+        In combination these two features allow you to study the spatial
+        resolution of the transmission network separately from the
+        spatial resolution of renewable generators.
+
+    **Is it possible to run the model without the** ``simplify_network`` **rule?**
+
+        No, the network clustering methods in the PyPSA module
+        `pypsa.networkclustering <https://github.com/PyPSA/PyPSA/blob/master/pypsa/networkclustering.py>`_
+        do not work reliably with multiple voltage levels and transformers.
+
+.. tip::
+    The rule ``cluster_all_networks`` runs
+    for all ``scenario`` s in the configuration file 
+    the rule ``cluster_network``.
+
 """
 
 import pandas as pd
@@ -65,7 +151,7 @@ def plot_weighting(n, country, country_shape=None):
 
 def distribute_clusters(n, n_clusters, solver_name=None):
     if solver_name is None:
-        solver_name = snakemake.config['solver']['solver']['name']
+        solver_name = snakemake.config['solving']['solver']['name']
 
     L = (n.loads_t.p_set.mean()
          .groupby(n.loads.bus).sum()

scripts/make_summary.py

@@ -1,5 +1,52 @@
 """
-Create summaries of aggregated energy and costs as csv files
+Creates summaries of aggregated energy and costs as ``.csv`` files.
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    costs:
+        USD2013_to_EUR2013:
+        discountrate:
+        marginal_cost:
+        capital_cost:
+
+    electricity:
+        max_hours:
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`costs_cf`, :ref:`electricity_cf`
+
+Inputs
+------
+
+Outputs
+-------
+
+Description
+-----------
+
+The following rule can be used to summarize the results in seperate .csv files:
+
+.. code::
+
+    snakemake results/summaries/elec_s_all_lall_Co2L-3H_all
+                                         clusters
+                                             line volume or cost cap
+                                                - options
+                                                        - all countries
+
+the line volume/cost cap field can be set to one of the following:
+* ``lv1.25`` for a particular line volume extension by 25%
+* ``lc1.25`` for a line cost extension by 25 %
+* ``lall`` for all evalutated caps
+* ``lvall`` for all line volume caps
+* ``lcall`` for all line cost caps
+
+Replacing '/summaries/' with '/plots/' creates nice colored maps of the results.
+
 """
 
 import os

scripts/plot_network.py

@@ -1,28 +1,20 @@
 """
-Plot map with pie charts and cost box plots
+Plots map with pie charts and cost box bar charts.
+
+Relevant Settings
+-----------------
+
+Inputs
+------
+
+Outputs
+-------
+
+Description
+-----------
+
 """
 
-# Dirty work-around so that sphinx can import this module and get the
-# doc-string should be refactored in the style of the other scripts, ideally
-# several functions for the different plots
-if __name__ != "__main__":
-    import sys
-    sys.exit(0)
-
-if 'snakemake' not in globals():
-    from vresutils.snakemake import MockSnakemake, Dict
-    from snakemake.rules import expand
-    import yaml
-    snakemake = Dict()
-    snakemake = MockSnakemake(
-        path='..',
-        wildcards=dict(network='elec', simpl='', clusters='90', lv='1.25', opts='Co2L-3H', attr='p_nom', ext="pdf"),
-        input=dict(network="results/networks/{network}_s{simpl}_{clusters}_lv{lv}_{opts}.nc",
-                   tech_costs="data/costs.csv"),
-        output=dict(only_map="results/plots/{network}_s{simpl}_{clusters}_lv{lv}_{opts}_{attr}.{ext}",
-                    ext="results/plots/{network}_s{simpl}_{clusters}_lv{lv}_{opts}_{attr}_ext.{ext}")
-    )
-
 import pypsa
 
 from _helpers import load_network, aggregate_p, aggregate_costs
@@ -37,7 +29,9 @@ from itertools import product, chain
 from six.moves import map, zip
 from six import itervalues, iterkeys
 from collections import OrderedDict as odict
+import logging
 
+import cartopy.crs as ccrs
 import matplotlib.pyplot as plt
 import matplotlib as mpl
 from matplotlib.patches import Circle, Ellipse
@@ -70,203 +64,242 @@ def make_handler_map_to_scale_circles_as_in(ax, dont_resize_actively=False):
 def make_legend_circles_for(sizes, scale=1.0, **kw):
     return [Circle((0,0), radius=(s/scale)**0.5, **kw) for s in sizes]
 
-plt.style.use(['classic', 'seaborn-white',
-               {'axes.grid': False, 'grid.linestyle': '--', 'grid.color': u'0.6',
-                'hatch.color': 'white',
-                'patch.linewidth': 0.5,
-                'font.size': 12,
-                'legend.fontsize': 'medium',
-                'lines.linewidth': 1.5,
-                'pdf.fonttype': 42,
-                # 'font.family': 'Times New Roman'
-               }])
+def set_plot_style():
+    plt.style.use(['classic', 'seaborn-white',
+                {'axes.grid': False, 'grid.linestyle': '--', 'grid.color': u'0.6',
+                    'hatch.color': 'white',
+                    'patch.linewidth': 0.5,
+                    'font.size': 12,
+                    'legend.fontsize': 'medium',
+                    'lines.linewidth': 1.5,
+                    'pdf.fonttype': 42,
+                    # 'font.family': 'Times New Roman'
+                }])
 
-opts = snakemake.config['plotting']
-map_figsize = opts['map']['figsize']
-map_boundaries = opts['map']['boundaries']
+def plot_map(n, ax=None, attribute='p_nom', opts={}):
+    if ax is None:
+        ax = plt.gca()
 
-n = load_network(snakemake.input.network, snakemake.input.tech_costs, snakemake.config)
+    ## DATA
+    line_colors = {'cur': "purple",
+                   'exp': to_rgba("red", 0.7)}
+    tech_colors = opts['tech_colors']
 
-scenario_opts = snakemake.wildcards.opts.split('-')
-
-## DATA
-line_colors = {'cur': "purple",
-               'exp': to_rgba("red", 0.7)}
-tech_colors = opts['tech_colors']
-
-if snakemake.wildcards.attr == 'p_nom':
-    # bus_sizes = n.generators_t.p.sum().loc[n.generators.carrier == "load"].groupby(n.generators.bus).sum()
-    bus_sizes = pd.concat((n.generators.query('carrier != "load"').groupby(['bus', 'carrier']).p_nom_opt.sum(),
-                           n.storage_units.groupby(['bus', 'carrier']).p_nom_opt.sum()))
-    line_widths_exp = dict(Line=n.lines.s_nom_opt, Link=n.links.p_nom_opt)
-    line_widths_cur = dict(Line=n.lines.s_nom_min, Link=n.links.p_nom_min)
-else:
-    raise 'plotting of {} has not been implemented yet'.format(plot)
+    if attribute == 'p_nom':
+        # bus_sizes = n.generators_t.p.sum().loc[n.generators.carrier == "load"].groupby(n.generators.bus).sum()
+        bus_sizes = pd.concat((n.generators.query('carrier != "load"').groupby(['bus', 'carrier']).p_nom_opt.sum(),
+                               n.storage_units.groupby(['bus', 'carrier']).p_nom_opt.sum()))
+        line_widths_exp = dict(Line=n.lines.s_nom_opt, Link=n.links.p_nom_opt)
+        line_widths_cur = dict(Line=n.lines.s_nom_min, Link=n.links.p_nom_min)
+    else:
+        raise 'plotting of {} has not been implemented yet'.format(plot)
 
 
-line_colors_with_alpha = \
-dict(Line=(line_widths_cur['Line'] / n.lines.s_nom > 1e-3)
-     .map({True: line_colors['cur'], False: to_rgba(line_colors['cur'], 0.)}),
-     Link=(line_widths_cur['Link'] / n.links.p_nom > 1e-3)
-     .map({True: line_colors['cur'], False: to_rgba(line_colors['cur'], 0.)}))
+    line_colors_with_alpha = \
+    dict(Line=(line_widths_cur['Line'] / n.lines.s_nom > 1e-3)
+        .map({True: line_colors['cur'], False: to_rgba(line_colors['cur'], 0.)}),
+        Link=(line_widths_cur['Link'] / n.links.p_nom > 1e-3)
+        .map({True: line_colors['cur'], False: to_rgba(line_colors['cur'], 0.)}))
 
-## FORMAT
-linewidth_factor = opts['map'][snakemake.wildcards.attr]['linewidth_factor']
-bus_size_factor  = opts['map'][snakemake.wildcards.attr]['bus_size_factor']
+    ## FORMAT
+    linewidth_factor = opts['map'][attribute]['linewidth_factor']
+    bus_size_factor  = opts['map'][attribute]['bus_size_factor']
 
-## PLOT
-fig, ax = plt.subplots(figsize=map_figsize)
-n.plot(line_widths=pd.concat(line_widths_exp)/linewidth_factor,
-       line_colors=dict(Line=line_colors['exp'], Link=line_colors['exp']),
-       bus_sizes=bus_sizes/bus_size_factor,
-       bus_colors=tech_colors,
-       boundaries=map_boundaries,
-       basemap=True,
-       ax=ax)
-n.plot(line_widths=pd.concat(line_widths_cur)/linewidth_factor,
-       line_colors=pd.concat(line_colors_with_alpha),
-       bus_sizes=0,
-       bus_colors=tech_colors,
-       boundaries=map_boundaries,
-       basemap=False,
-       ax=ax)
-ax.set_aspect('equal')
-ax.axis('off')
+    ## PLOT
+    n.plot(line_widths=pd.concat(line_widths_exp)/linewidth_factor,
+           line_colors=dict(Line=line_colors['exp'], Link=line_colors['exp']),
+           bus_sizes=bus_sizes/bus_size_factor,
+           bus_colors=tech_colors,
+           boundaries=map_boundaries,
+           geomap=True,
+           ax=ax)
+    n.plot(line_widths=pd.concat(line_widths_cur)/linewidth_factor,
+           line_colors=pd.concat(line_colors_with_alpha),
+           bus_sizes=0,
+           bus_colors=tech_colors,
+           boundaries=map_boundaries,
+           geomap=True,   # TODO : Turn to False, after the release of PyPSA 0.14.2 (refer to https://github.com/PyPSA/PyPSA/issues/75)
+           ax=ax)
+    ax.set_aspect('equal')
+    ax.axis('off')
 
-# x1, y1, x2, y2 = map_boundaries
-# ax.set_xlim(x1, x2)
-# ax.set_ylim(y1, y2)
+    # x1, y1, x2, y2 = map_boundaries
+    # ax.set_xlim(x1, x2)
+    # ax.set_ylim(y1, y2)
 
 
-# Rasterize basemap
-for c in ax.collections[:2]: c.set_rasterized(True)
+    # Rasterize basemap
+    # TODO : Check if this also works with cartopy
+    for c in ax.collections[:2]: c.set_rasterized(True)
 
-# LEGEND
-handles = []
-labels = []
+    # LEGEND
+    handles = []
+    labels = []
 
-for s in (10, 1):
-    handles.append(plt.Line2D([0],[0],color=line_colors['exp'],
-                              linewidth=s*1e3/linewidth_factor))
-    labels.append("{} GW".format(s))
-l1 = l1_1 = ax.legend(handles, labels,
-               loc="upper left", bbox_to_anchor=(0.24, 1.01),
-               frameon=False,
-               labelspacing=0.8, handletextpad=1.5,
-               title='Transmission Exist./Exp.             ')
-ax.add_artist(l1_1)
+    for s in (10, 1):
+        handles.append(plt.Line2D([0],[0],color=line_colors['exp'],
+                                linewidth=s*1e3/linewidth_factor))
+        labels.append("{} GW".format(s))
+    l1 = l1_1 = ax.legend(handles, labels,
+                loc="upper left", bbox_to_anchor=(0.24, 1.01),
+                frameon=False,
+                labelspacing=0.8, handletextpad=1.5,
+                title='Transmission Exist./Exp.             ')
+    ax.add_artist(l1_1)
 
-handles = []
-labels = []
-for s in (10, 5):
-    handles.append(plt.Line2D([0],[0],color=line_colors['cur'],
-                              linewidth=s*1e3/linewidth_factor))
-    labels.append("/")
-l1_2 = ax.legend(handles, labels,
-               loc="upper left", bbox_to_anchor=(0.26, 1.01),
-               frameon=False,
-               labelspacing=0.8, handletextpad=0.5,
-               title=' ')
-ax.add_artist(l1_2)
+    handles = []
+    labels = []
+    for s in (10, 5):
+        handles.append(plt.Line2D([0],[0],color=line_colors['cur'],
+                                linewidth=s*1e3/linewidth_factor))
+        labels.append("/")
+    l1_2 = ax.legend(handles, labels,
+                loc="upper left", bbox_to_anchor=(0.26, 1.01),
+                frameon=False,
+                labelspacing=0.8, handletextpad=0.5,
+                title=' ')
+    ax.add_artist(l1_2)
 
-handles = make_legend_circles_for([10e3, 5e3, 1e3], scale=bus_size_factor, facecolor="w")
-labels = ["{} GW".format(s) for s in (10, 5, 3)]
-l2 = ax.legend(handles, labels,
-               loc="upper left", bbox_to_anchor=(0.01, 1.01),
-               frameon=False, labelspacing=1.0,
-               title='Generation',
-               handler_map=make_handler_map_to_scale_circles_as_in(ax))
-ax.add_artist(l2)
+    handles = make_legend_circles_for([10e3, 5e3, 1e3], scale=bus_size_factor, facecolor="w")
+    labels = ["{} GW".format(s) for s in (10, 5, 3)]
+    l2 = ax.legend(handles, labels,
+                loc="upper left", bbox_to_anchor=(0.01, 1.01),
+                frameon=False, labelspacing=1.0,
+                title='Generation',
+                handler_map=make_handler_map_to_scale_circles_as_in(ax))
+    ax.add_artist(l2)
 
-techs =  (bus_sizes.index.levels[1]) & pd.Index(opts['vre_techs'] + opts['conv_techs'] + opts['storage_techs'])
-handles = []
-labels = []
-for t in techs:
-    handles.append(plt.Line2D([0], [0], color=tech_colors[t], marker='o', markersize=8, linewidth=0))
-    labels.append(opts['nice_names'].get(t, t))
-l3 = ax.legend(handles, labels, loc="upper center",  bbox_to_anchor=(0.5, -0.), # bbox_to_anchor=(0.72, -0.05),
-               handletextpad=0., columnspacing=0.5, ncol=4, title='Technology')
+    techs =  (bus_sizes.index.levels[1]) & pd.Index(opts['vre_techs'] + opts['conv_techs'] + opts['storage_techs'])
+    handles = []
+    labels = []
+    for t in techs:
+        handles.append(plt.Line2D([0], [0], color=tech_colors[t], marker='o', markersize=8, linewidth=0))
+        labels.append(opts['nice_names'].get(t, t))
+    l3 = ax.legend(handles, labels, loc="upper center",  bbox_to_anchor=(0.5, -0.), # bbox_to_anchor=(0.72, -0.05),
+                handletextpad=0., columnspacing=0.5, ncol=4, title='Technology')
 
-
-fig.savefig(snakemake.output.only_map, dpi=150,
-            bbox_inches='tight', bbox_extra_artists=[l1,l2,l3])
+    return fig
 
 #n = load_network(snakemake.input.network, opts, combine_hydro_ps=False)
 
-## Add total energy p
 
-ax1 = ax = fig.add_axes([-0.115, 0.625, 0.2, 0.2])
-ax.set_title('Energy per technology', fontdict=dict(fontsize="medium"))
+def plot_total_energy_pie(n, ax=None):
+    """Add total energy pie plot"""
+    if ax is None:
+        ax = plt.gca()
 
-e_primary = aggregate_p(n).drop('load', errors='ignore').loc[lambda s: s>0]
+    ax.set_title('Energy per technology', fontdict=dict(fontsize="medium"))
 
-patches, texts, autotexts = ax.pie(e_primary,
-       startangle=90,
-       labels = e_primary.rename(opts['nice_names_n']).index,
-      autopct='%.0f%%',
-      shadow=False,
-          colors = [tech_colors[tech] for tech in e_primary.index])
-for t1, t2, i in zip(texts, autotexts, e_primary.index):
-    if e_primary.at[i] < 0.04 * e_primary.sum():
-        t1.remove()
-        t2.remove()
+    e_primary = aggregate_p(n).drop('load', errors='ignore').loc[lambda s: s>0]
 
-## Add average system cost bar plot
-# ax2 = ax = fig.add_axes([-0.1, 0.2, 0.1, 0.33])
-# ax2 = ax = fig.add_axes([-0.1, 0.15, 0.1, 0.37])
-ax2 = ax = fig.add_axes([-0.075, 0.1, 0.1, 0.45])
-total_load = (n.snapshot_weightings * n.loads_t.p.sum(axis=1)).sum()
+    patches, texts, autotexts = ax.pie(e_primary,
+        startangle=90,
+        labels = e_primary.rename(opts['nice_names_n']).index,
+        autopct='%.0f%%',
+        shadow=False,
+            colors = [tech_colors[tech] for tech in e_primary.index])
+    for t1, t2, i in zip(texts, autotexts, e_primary.index):
+        if e_primary.at[i] < 0.04 * e_primary.sum():
+            t1.remove()
+            t2.remove()
 
-def split_costs(n):
-    costs = aggregate_costs(n).reset_index(level=0, drop=True)
-    costs_ex = aggregate_costs(n, existing_only=True).reset_index(level=0, drop=True)
-    return (costs['capital'].add(costs['marginal'], fill_value=0.),
-            costs_ex['capital'], costs['capital'] - costs_ex['capital'], costs['marginal'])
+def plot_total_cost_bar(n, ax=None):
+    """Add average system cost bar plot"""
+    if ax is None:
+        ax = plt.gca()
 
-costs, costs_cap_ex, costs_cap_new, costs_marg = split_costs(n)
+    total_load = (n.snapshot_weightings * n.loads_t.p.sum(axis=1)).sum()
 
-costs_graph = pd.DataFrame(dict(a=costs.drop('load', errors='ignore')),
-                           index=['AC-AC', 'AC line', 'onwind', 'offwind-ac', 'offwind-dc', 'solar', 'OCGT','CCGT', 'battery', 'H2']).dropna()
-bottom = np.array([0., 0.])
-texts = []
+    def split_costs(n):
+        costs = aggregate_costs(n).reset_index(level=0, drop=True)
+        costs_ex = aggregate_costs(n, existing_only=True).reset_index(level=0, drop=True)
+        return (costs['capital'].add(costs['marginal'], fill_value=0.),
+                costs_ex['capital'], costs['capital'] - costs_ex['capital'], costs['marginal'])
 
-for i,ind in enumerate(costs_graph.index):
-    data = np.asarray(costs_graph.loc[ind])/total_load
-    ax.bar([0.5], data, bottom=bottom, color=tech_colors[ind], width=0.7, zorder=-1)
-    bottom_sub = bottom
-    bottom = bottom+data
+    costs, costs_cap_ex, costs_cap_new, costs_marg = split_costs(n)
 
-    if ind in opts['conv_techs'] + ['AC line']:
-        for c in [costs_cap_ex, costs_marg]:
-            if ind in c:
-                data_sub = np.asarray([c.loc[ind]])/total_load
-                ax.bar([0.5], data_sub, linewidth=0,
-                       bottom=bottom_sub, color=tech_colors[ind],
-                       width=0.7, zorder=-1, alpha=0.8)
-                bottom_sub += data_sub
+    costs_graph = pd.DataFrame(dict(a=costs.drop('load', errors='ignore')),
+                            index=['AC-AC', 'AC line', 'onwind', 'offwind-ac', 'offwind-dc', 'solar', 'OCGT','CCGT', 'battery', 'H2']).dropna()
+    bottom = np.array([0., 0.])
+    texts = []
 
-    if abs(data[-1]) < 5:
-        continue
+    for i,ind in enumerate(costs_graph.index):
+        data = np.asarray(costs_graph.loc[ind])/total_load
+        ax.bar([0.5], data, bottom=bottom, color=tech_colors[ind], width=0.7, zorder=-1)
+        bottom_sub = bottom
+        bottom = bottom+data
 
-    text = ax.text(1.1,(bottom-0.5*data)[-1]-3,opts['nice_names_n'].get(ind,ind))
-    texts.append(text)
+        if ind in opts['conv_techs'] + ['AC line']:
+            for c in [costs_cap_ex, costs_marg]:
+                if ind in c:
+                    data_sub = np.asarray([c.loc[ind]])/total_load
+                    ax.bar([0.5], data_sub, linewidth=0,
+                        bottom=bottom_sub, color=tech_colors[ind],
+                        width=0.7, zorder=-1, alpha=0.8)
+                    bottom_sub += data_sub
 
-ax.set_ylabel("Average system cost [Eur/MWh]")
-ax.set_ylim([0,80]) # opts['costs_max']])
-ax.set_xlim([0,1])
-#ax.set_xticks([0.5])
-ax.set_xticklabels([]) #["w/o\nEp", "w/\nEp"])
-ax.grid(True, axis="y", color='k', linestyle='dotted')
+        if abs(data[-1]) < 5:
+            continue
 
-#fig.tight_layout()
+        text = ax.text(1.1,(bottom-0.5*data)[-1]-3,opts['nice_names_n'].get(ind,ind))
+        texts.append(text)
 
-ll = snakemake.wildcards.ll
-ll_type = ll[0]
-ll_factor = ll[1:]
-lbl = dict(c='line cost', v='line volume')[ll_type]
-amnt = '{ll} x today\'s'.format(ll=ll_factor) if ll_factor != 'opt' else 'optimal'
-fig.suptitle('Expansion to {amount} {label} at {clusters} clusters'
-             .format(amount=amnt, label=lbl, clusters=snakemake.wildcards.clusters))
+    ax.set_ylabel("Average system cost [Eur/MWh]")
+    ax.set_ylim([0,80]) # opts['costs_max']])
+    ax.set_xlim([0,1])
+    #ax.set_xticks([0.5])
+    ax.set_xticklabels([]) #["w/o\nEp", "w/\nEp"])
+    ax.grid(True, axis="y", color='k', linestyle='dotted')
 
-fig.savefig(snakemake.output.ext, transparent=True,
-            bbox_inches='tight', bbox_extra_artists=[l1, l2, l3, ax1, ax2])
+
+if __name__ == "__main__":
+    if 'snakemake' not in globals():
+        from vresutils.snakemake import MockSnakemake, Dict
+        from snakemake.rules import expand
+
+        snakemake = Dict()
+        snakemake = MockSnakemake(
+            path='..',
+            wildcards=dict(network='elec', simpl='', clusters='90', lv='1.25', opts='Co2L-3H', attr='p_nom', ext="pdf"),
+            input=dict(network="results/networks/{network}_s{simpl}_{clusters}_lv{lv}_{opts}.nc",
+                    tech_costs="data/costs.csv"),
+            output=dict(only_map="results/plots/{network}_s{simpl}_{clusters}_lv{lv}_{opts}_{attr}.{ext}",
+                        ext="results/plots/{network}_s{simpl}_{clusters}_lv{lv}_{opts}_{attr}_ext.{ext}")
+        )
+
+    logging.basicConfig(level=snakemake.config['logging_level'])
+
+    set_plot_style()
+
+    opts = snakemake.config['plotting']
+    map_figsize = opts['map']['figsize']
+    map_boundaries = opts['map']['boundaries']
+
+    n = load_network(snakemake.input.network, snakemake.input.tech_costs, snakemake.config)
+
+    scenario_opts = snakemake.wildcards.opts.split('-')
+
+    fig, ax = plt.subplots(figsize=map_figsize, subplot_kw={"projection": ccrs.PlateCarree()})
+    plot_map(n, ax, snakemake.wildcards.attr, opts)
+
+    fig.savefig(snakemake.output.only_map, dpi=150,
+                bbox_inches='tight', bbox_extra_artists=[l1,l2,l3])
+
+    ax1 = fig.add_axes([-0.115, 0.625, 0.2, 0.2])
+    plot_total_energy_pie(n, ax1)
+
+    ax2 = fig.add_axes([-0.075, 0.1, 0.1, 0.45])
+    plot_total_cost_bar(n, ax2)
+
+    #fig.tight_layout()
+
+    ll = snakemake.wildcards.ll
+    ll_type = ll[0]
+    ll_factor = ll[1:]
+    lbl = dict(c='line cost', v='line volume')[ll_type]
+    amnt = '{ll} x today\'s'.format(ll=ll_factor) if ll_factor != 'opt' else 'optimal'
+    fig.suptitle('Expansion to {amount} {label} at {clusters} clusters'
+                .format(amount=amnt, label=lbl, clusters=snakemake.wildcards.clusters))
+
+    fig.savefig(snakemake.output.ext, transparent=True,
+                bbox_inches='tight', bbox_extra_artists=[l1, l2, l3, ax1, ax2])

scripts/plot_p_nom_max.py

@@ -1,5 +1,18 @@
 """
-Plot renewable installation potentials per capacity factor
+Plots renewable installation potentials per capacity factor.
+
+Relevant Settings
+-----------------
+
+Inputs
+------
+
+Outputs
+-------
+
+Description
+-----------
+
 """
 
 import pypsa
@@ -24,7 +37,7 @@ def cum_p_nom_max(net, tech, country=None):
     return generators
 
 
-if __name__ == __main__:
+if __name__ == "__main__":
     # Detect running outside of snakemake and mock snakemake for testing
     if 'snakemake' not in globals():
         from vresutils.snakemake import MockSnakemake, Dict

scripts/plot_summary.py

@@ -1,5 +1,18 @@
 """
-Plot energy and cost summaries for several solved networks
+Plots energy and cost summaries for solved networks.
+
+Relevant Settings
+-----------------
+
+Inputs
+------
+
+Outputs
+-------
+
+Description
+-----------
+
 """
 
 import os

scripts/prepare_links_p_nom.py

@@ -1,6 +1,34 @@
 #!/usr/bin/env python
 """
-Extract capacities for HVDC links from wikipedia
+Extracts capacities of HVDC links from `Wikipedia <https://en.wikipedia.org/wiki/List_of_HVDC_projects>`_.
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    enable:
+        prepare_links_p_nom:
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`toplevel_cf`
+
+Inputs
+------
+
+*None*
+
+Outputs
+-------
+
+- ``data/links_p_nom.csv``: A plain download of https://en.wikipedia.org/wiki/List_of_HVDC_projects#Europe plus extracted coordinates.
+
+Description
+-----------
+
+*None*
+
 """
 
 import pandas as pd

scripts/prepare_network.py

@@ -1,6 +1,46 @@
 # coding: utf-8
 """
-Prepare PyPSA network for solving according to `opts`-wildcard
+Prepare PyPSA network for solving according to the ``{opts}`` wildcard (:ref:`opts`).
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    costs:
+        emission_prices:
+        USD2013_to_EUR2013:
+        discountrate:
+        marginal_cost:
+        capital_cost:
+
+    electricity:
+        co2limit:
+        max_hours:
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`costs_cf`, :ref:`electricity_cf`
+
+Inputs
+------
+
+- ``data/costs.csv``: The database of cost assumptions for all included technologies for specific years from various sources; e.g. discount rate, lifetime, investment (CAPEX), fixed operation and maintenance (FOM), variable operation and maintenance (VOM), fuel costs, efficiency, carbon-dioxide intensity.
+- ``networks/{network}_s{simpl}_{clusters}.nc``: confer :ref:`cluster`
+
+Outputs
+-------
+
+- ``networks/{network}_s{simpl}_{clusters}_l{ll}_{opts}.nc``: Complete PyPSA network that will be handed to the ``solve_network`` rule.
+
+Description
+-----------
+
+.. tip::
+    The rule ``prepare_all_networks`` runs
+    for all ``scenario`` s in the configuration file
+    the rule ``prepare_network``.
+
 """
 
 import logging

scripts/simplify_network.py

@@ -1,7 +1,81 @@
 # coding: utf-8
-"""Bring electrical transmission network to a single 380kV voltage layer,
-remove network dead-ends, and reduce multi-hop linear HVDC connections to a
-single link
+"""
+Lifts electrical transmission network to a single 380 kV voltage layer,
+removes dead-ends of the network,
+and reduces multi-hop HVDC connections to a single link.
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    costs:
+        USD2013_to_EUR2013:
+        discountrate:
+        marginal_cost:
+        capital_cost:
+
+    electricity:
+        max_hours:
+
+    renewables: (keys)
+        {technology}:
+            potential:
+
+    lines:
+        length_factor:
+
+    links:
+        p_max_pu:
+
+    solving:
+        solver:
+            name:
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`costs_cf`, :ref:`electricity_cf`, :ref:`renewable_cf`,
+    :ref:`lines_cf`, :ref:`links_cf`, :ref:`solving_cf`
+
+Inputs
+------
+
+- ``data/costs.csv``: The database of cost assumptions for all included technologies for specific years from various sources; e.g. discount rate, lifetime, investment (CAPEX), fixed operation and maintenance (FOM), variable operation and maintenance (VOM), fuel costs, efficiency, carbon-dioxide intensity.
+- ``resources/regions_onshore.geojson``: confer :ref:`busregions`
+- ``resources/regions_offshore.geojson``: confer :ref:`busregions`
+- ``networks/{network}.nc``: confer :ref:`electricity`
+
+Outputs
+-------
+
+- ``resources/regions_onshore_{network}_s{simpl}.geojson``:
+
+    .. image:: img/regions_onshore_elec_s.png
+            :scale: 33 %
+
+- ``resources/regions_offshore_{network}_s{simpl}.geojson``:
+
+    .. image:: img/regions_offshore_elec_s  .png
+            :scale: 33 %
+
+- ``resources/clustermaps_{network}_s{simpl}.h5``: Mapping of buses from ``networks/elec.nc`` to ``networks/elec_s{simpl}.nc``; has keys ['/busmap_s']
+- ``networks/{network}_s{simpl}.nc``:
+
+    .. image:: img/elec_s.png
+        :scale: 33 %
+
+Description
+-----------
+
+The rule ``simplify_network`` does up to four things:
+
+1. Create an equivalent transmission network in which all voltage levels are mapped to the 380 kV level by the function ``simplify_network(...)``.
+
+2. DC only sub-networks that are connected at only two buses to the AC network are reduced to a single representative link in the function ``simplify_links(...)``. The components attached to buses in between are moved to the nearest endpoint. The grid connection cost of offshore wind generators are added to the captial costs of the generator.
+
+3. Stub lines and links, i.e. dead-ends of the network, are sequentially removed from the network in the function ``remove_stubs(...)``. Components are moved along.
+
+4. Optionally, if an integer were provided for the wildcard ``{simpl}`` (e.g. ``networks/elec_s500.nc``), the network is clustered to this number of clusters with the routines from the ``cluster_network`` rule with the function ``cluster_network.cluster(...)``. This step is usually skipped!
 """
 
 import pandas as pd

scripts/solve_network.py

@@ -1,5 +1,54 @@
 """
-Solve networks iteratively linear optimal power flow, while updating reactances
+Solves linear optimal power flow for a network iteratively while updating reactances.
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    (electricity:)
+        (BAU_mincapacities:)
+        (SAFE_reservemargin:)
+
+    solving:
+        tmpdir:
+        options:
+            formulation:
+            clip_p_max_pu:
+            load_shedding:
+            noisy_costs:
+            nhours:
+            min_iterations:
+            max_iterations:
+        solver:
+            name:
+            (solveroptions):
+
+    (plotting:)
+        (conv_techs:)
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`electricity_cf`, :ref:`solving_cf`, :ref:`plotting_cf`
+
+Inputs
+------
+
+- ``networks/{network}_s{simpl}_{clusters}_l{ll}_{opts}.nc``: confer :ref:`prepare`
+
+Outputs
+-------
+
+- ``results/networks/{network}_s{simpl}_{clusters}_l{ll}_{opts}.nc``: Solved PyPSA network including optimisation results
+
+Description
+-----------
+
+.. tip::
+    The rule ``solve_all_networks`` runs
+    for all ``scenario`` s in the configuration file 
+    the rule ``solve_network``.
+
 """
 
 import numpy as np

scripts/solve_operations_network.py

@@ -1,6 +1,44 @@
 """
-Solve linear optimal dispatch in hourly resolution with capacities of previous
-capacity expansion
+Solves linear optimal dispatch in hourly resolution
+using the capacities of previous capacity expansion in rule ``solve_network``.
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    solving:
+        tmpdir:
+        options:
+            formulation:
+            clip_p_max_pu:
+            load_shedding:
+            noisy_costs:
+            nhours:
+            min_iterations:
+            max_iterations:
+        solver:
+            name:
+            (solveroptions):
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`solving_cf`
+
+Inputs
+------
+
+- ``networks/{network}_s{simpl}_{clusters}.nc``: confer :ref:`cluster`
+- ``results/networks/{network}_s{simpl}_{clusters}_l{ll}_{opts}.nc``: confer :ref:`solve`
+
+Outputs
+-------
+
+- ``results/networks/{network}_s{simpl}_{clusters}_l{ll}_{opts}_op.nc``: Solved PyPSA network for optimal dispatch including optimisation results
+
+Description
+-----------
+
 """
 
 import pypsa

scripts/trace_solve_network.py

@@ -1,6 +1,43 @@
 """
-Iteratively solves expansion problem like solve_network, but additionally
-records intermediate branch capacity steps and values of the objective
+Iteratively solves expansion problem like the rule ``solve_network``, but additionally
+records intermediate branch capacity steps and values of the objective function.
+
+Relevant Settings
+-----------------
+
+.. code:: yaml
+
+    solving:
+        tmpdir:
+        options:
+            formulation:
+            clip_p_max_pu:
+            load_shedding:
+            noisy_costs:
+            nhours:
+            min_iterations:
+            max_iterations:
+        solver:
+            name:
+            (solveroptions):
+
+.. seealso:: 
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`solving_cf`
+
+Inputs
+------
+
+- ``networks/{network}_s{simpl}_{clusters}_l{ll}_{opts}.nc``: confer :ref:`prepare`
+
+Outputs
+-------
+
+- ``results/networks/{network}_s{simpl}_{clusters}_l{ll}_{opts}_trace.nc``: Solved PyPSA network including optimisation results (with trace)
+
+Description
+-----------
+
 """
 
 import numpy as np