add release notes

doc/release_notes.rst

@@ -10,12 +10,71 @@ Release Notes
 Upcoming Release
 ================
 
-* PyPSA-EUR now supports the simultaneous execution of multiple scenarios. For
+* Added new scenario management that supports the simultaneous execution of
-  this purpose, a scenarios.yaml file has been introduced which contains
+  multiple scenarios with a single ``snakemake`` call. For this purpose, a
-  customizable scenario names with corresponding configuration overrides. To
+  ``scenarios.yaml`` file is introduced which contains customizable scenario
-  enable it, set the ``run: scenarios:`` key to ``True`` and define the scenario
+  names with configuration overrides. To enable it, set the ``run: scenarios:
-  names to run under ``run: name:`` in the configuration file. The latter must
+  true`` and define the list of scenario names to run under ``run: name:`` in
-  be a subset of toplevel keys in the scenario file.
+  the configuration file. The latter must be a subset of toplevel keys in the
+  scenario file.
+
+  - To get started, a scenarios template file ``config/scenarios.template.yaml``
+    is included in the repository, which is copied to ``config/scenarios.yaml``
+    on first use.
+
+  - The scenario file can be changed via ``run: scenarios: file:``.
+
+  - If scenario management is activated with ``run: scenarios: enable: true``, a
+    new wildcard ``{run}`` is introduced. This means that the configuration
+    settings may depend on the new ``{run}`` wildcard. Therefore, a new
+    ``config_provider()`` function is used in the ``Snakefile`` and ``.smk``
+    files, which takes wildcard values into account. The calls to the ``config``
+    object have been reduced in ``.smk`` files since there is no awareness of
+    wildcard values outside rule definitions.
+
+  - The scenario files can also be programmatically created using the template
+    script ``config/create_scenarios.py``. This script can be run with
+    ``snakemake -j1 create_scenarios`` and creates the scenarios file referenced
+    under ``run: scenarios: file:``.
+
+  - The setting ``run: name: all`` will run all scenarios in
+    ``config/scenarios.yaml``. Otherwise, it will run those passed as list in
+    ``run: name:`` as long as ``run: scenarios: enable: true``.
+
+  - The setting ``run: shared_resources:`` indicates via a boolean whether the
+    resources should be encapsulated by the ``run: name:``. The special setting
+    ``run: shared_resources: base`` shares resources until ``add_electricity``
+    that do not contain wildcards other than ``{"technology", "year",
+    "scope"}``.
+
+  - Added new configuration options for all ``{opts}`` and ``{sector_opts}``
+    wildcard values to create a unique configuration file (``config.yaml``) per
+    PyPSA network file. This is done with the help of a new function
+    ``update_config_from_wildcards()`` which parses configuration settings from
+    wildcards and updates the ``snakemake.config`` object. These updated
+    configuration settings are used in the scripts rather than directly parsed
+    values from ``snakemake.wildcards``.
+
+  - The cost data was moved from ``data/costs_{year}.csv`` to
+    ``resources/costs_{year}.csv`` since it depends on configuration settings.
+    The ``retrieve_cost_data`` rule was changed to calling a Python script.
+
+  - Moved time clustering settings to ``clustering: temporal:`` from
+    ``snapshots:`` so that the latter is only used to define the
+    ``pandas.DatetimeIndex`` which simplifies the scenario management.
+
+  - Collection rules get a new wildcard ``run=config["run"]["name"]`` so they
+    can collect outputs across different scenarios.
+
+  - **Warning:** One caveat remains for the scenario management with myopic or
+    perfect foresight pathway optimisation. The first investment period must be
+    shared across all scenarios. The reason is that the ``wildcard_constraints``
+    defined for the rule ``add_existing_baseyear`` do not accept wildcard-aware
+    input functions (cf.
+    `https://github.com/snakemake/snakemake/issues/2703`_).
+
+* The outputs of the rule ``retrieve_gas_infrastructure_data`` no longer
+  marked as ``protected()`` as the download size is small.
 
 * Improved representation of industry transition pathways. A new script was
   added to interpolate industry sector ratios from today's status quo to future

rules/validate.smk

@@ -17,10 +17,7 @@ rule build_electricity_production:
     The data is used for validation of the optimization results.
     """
     params:
-        snapshots=lambda w: {
+        snapshots=config_provider("snapshots"),
-            k: config_provider("snapshots", k)(w)
-            for k in ["start", "end", "inclusive"]
-        },
         countries=config_provider("countries"),
     output:
         resources("historical_electricity_production.csv"),