bjoern/pypsa-eur
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Merge pull request #413 from PyPSA/scenario-management

Browse Source 
Enable scneario management for different configs
This commit is contained in:
Fabian Neumann 2022-09-13 15:16:22 +02:00 committed by GitHub
commit aac7f6681b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 157 additions and 128 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

257
Snakefile
View File

@ -13,9 +13,14 @@ if not exists("config.yaml"):
configfile: "config.yaml" configfile: "config.yaml"
COSTS="resources/costs.csv" run = config.get("run", {})
RDIR = run["name"] + "/" if run.get("name") else ""
CDIR = RDIR if not run.get("shared_cutouts") else ""
COSTS = "resources/" + RDIR + "costs.csv"
ATLITE_NPROCESSES = config['atlite'].get('nprocesses', 4) ATLITE_NPROCESSES = config['atlite'].get('nprocesses', 4)
wildcard_constraints: wildcard_constraints:
simpl="[a-zA-Z0-9]*|all", simpl="[a-zA-Z0-9]*|all",
clusters="[0-9]+m?|all", clusters="[0-9]+m?|all",
@ -24,25 +29,25 @@ wildcard_constraints:
rule cluster_all_networks: rule cluster_all_networks:
input: expand("networks/elec_s{simpl}_{clusters}.nc", **config['scenario']) input: expand("networks/" + RDIR + "elec_s{simpl}_{clusters}.nc", **config['scenario'])
rule extra_components_all_networks: rule extra_components_all_networks:
input: expand("networks/elec_s{simpl}_{clusters}_ec.nc", **config['scenario']) input: expand("networks/" + RDIR + "elec_s{simpl}_{clusters}_ec.nc", **config['scenario'])
rule prepare_all_networks: rule prepare_all_networks:
input: expand("networks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc", **config['scenario']) input: expand("networks/" + RDIR + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc", **config['scenario'])
rule solve_all_networks: rule solve_all_networks:
input: expand("results/networks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc", **config['scenario']) input: expand("results/networks/" + RDIR + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc", **config['scenario'])
if config['enable'].get('prepare_links_p_nom', False): if config['enable'].get('prepare_links_p_nom', False):
rule prepare_links_p_nom: rule prepare_links_p_nom:
output: 'data/links_p_nom.csv' output: 'data/links_p_nom.csv'
log: 'logs/prepare_links_p_nom.log' log: "logs/" + RDIR + "prepare_links_p_nom.log"
threads: 1 threads: 1
resources: mem_mb=500 resources: mem_mb=500
script: 'scripts/prepare_links_p_nom.py' script: 'scripts/prepare_links_p_nom.py'
@ -62,7 +67,7 @@ if not config.get('tutorial', False):
if config['enable'].get('retrieve_databundle', True): if config['enable'].get('retrieve_databundle', True):
rule retrieve_databundle: rule retrieve_databundle:
output: expand('data/bundle/{file}', file=datafiles) output: expand('data/bundle/{file}', file=datafiles)
log: "logs/retrieve_databundle.log" log: "logs/" + RDIR + "retrieve_databundle.log"
resources: mem_mb=1000 resources: mem_mb=1000
script: 'scripts/retrieve_databundle.py' script: 'scripts/retrieve_databundle.py'
@ -76,17 +81,17 @@ rule retrieve_load_data:
rule build_load_data: rule build_load_data:
input: "data/load_raw.csv" input: "data/load_raw.csv"
output: "resources/load.csv" output: "resources/" + RDIR + "load.csv"
log: "logs/build_load_data.log" log: "logs/" + RDIR + "build_load_data.log"
resources: mem_mb=5000 resources: mem_mb=5000
script: 'scripts/build_load_data.py' script: 'scripts/build_load_data.py'
rule build_powerplants: rule build_powerplants:
input: input:
base_network="networks/base.nc", base_network="networks/" + RDIR + "base.nc",
custom_powerplants="data/custom_powerplants.csv" custom_powerplants="data/custom_powerplants.csv"
output: "resources/powerplants.csv" output: "resources/" + RDIR + "powerplants.csv"
log: "logs/build_powerplants.log" log: "logs/" + RDIR + "build_powerplants.log"
threads: 1 threads: 1
resources: mem_mb=5000 resources: mem_mb=5000
script: "scripts/build_powerplants.py" script: "scripts/build_powerplants.py"
@ -102,12 +107,12 @@ rule base_network:
parameter_corrections='data/parameter_corrections.yaml', parameter_corrections='data/parameter_corrections.yaml',
links_p_nom='data/links_p_nom.csv', links_p_nom='data/links_p_nom.csv',
links_tyndp='data/links_tyndp.csv', links_tyndp='data/links_tyndp.csv',
country_shapes='resources/country_shapes.geojson', country_shapes="resources/" + RDIR + "country_shapes.geojson",
offshore_shapes='resources/offshore_shapes.geojson', offshore_shapes="resources/" + RDIR + "offshore_shapes.geojson",
europe_shape='resources/europe_shape.geojson' europe_shape="resources/" + RDIR + "europe_shape.geojson"
output: "networks/base.nc" output: "networks/" + RDIR + "base.nc"
log: "logs/base_network.log" log: "logs/" + RDIR + "base_network.log"
benchmark: "benchmarks/base_network" benchmark: "benchmarks/" + RDIR + "base_network"
threads: 1 threads: 1
resources: mem_mb=500 resources: mem_mb=500
script: "scripts/base_network.py" script: "scripts/base_network.py"
@ -123,11 +128,11 @@ rule build_shapes:
ch_cantons='data/bundle/ch_cantons.csv', ch_cantons='data/bundle/ch_cantons.csv',
ch_popgdp='data/bundle/je-e-21.03.02.xls' ch_popgdp='data/bundle/je-e-21.03.02.xls'
output: output:
country_shapes='resources/country_shapes.geojson', country_shapes="resources/" + RDIR + "country_shapes.geojson",
offshore_shapes='resources/offshore_shapes.geojson', offshore_shapes="resources/" + RDIR + "offshore_shapes.geojson",
europe_shape='resources/europe_shape.geojson', europe_shape="resources/" + RDIR + "europe_shape.geojson",
nuts3_shapes='resources/nuts3_shapes.geojson' nuts3_shapes="resources/" + RDIR + "nuts3_shapes.geojson"
log: "logs/build_shapes.log" log: "logs/" + RDIR + "build_shapes.log"
threads: 1 threads: 1
resources: mem_mb=500 resources: mem_mb=500
script: "scripts/build_shapes.py" script: "scripts/build_shapes.py"
@ -135,13 +140,13 @@ rule build_shapes:
rule build_bus_regions: rule build_bus_regions:
input: input:
country_shapes='resources/country_shapes.geojson', country_shapes="resources/" + RDIR + "country_shapes.geojson",
offshore_shapes='resources/offshore_shapes.geojson', offshore_shapes="resources/" + RDIR + "offshore_shapes.geojson",
base_network="networks/base.nc" base_network="networks/" + RDIR + "base.nc"
output: output:
regions_onshore="resources/regions_onshore.geojson", regions_onshore="resources/" + RDIR + "regions_onshore.geojson",
regions_offshore="resources/regions_offshore.geojson" regions_offshore="resources/" + RDIR + "regions_offshore.geojson"
log: "logs/build_bus_regions.log" log: "logs/" + RDIR + "build_bus_regions.log"
threads: 1 threads: 1
resources: mem_mb=1000 resources: mem_mb=1000
script: "scripts/build_bus_regions.py" script: "scripts/build_bus_regions.py"
@ -149,11 +154,11 @@ rule build_bus_regions:
if config['enable'].get('build_cutout', False): if config['enable'].get('build_cutout', False):
rule build_cutout: rule build_cutout:
input: input:
regions_onshore="resources/regions_onshore.geojson", regions_onshore="resources/" + RDIR + "regions_onshore.geojson",
regions_offshore="resources/regions_offshore.geojson" regions_offshore="resources/" + RDIR + "regions_offshore.geojson"
output: "cutouts/{cutout}.nc" output: "cutouts/" + CDIR + "{cutout}.nc"
log: "logs/build_cutout/{cutout}.log" log: "logs/" + CDIR + "build_cutout/{cutout}.log"
benchmark: "benchmarks/build_cutout_{cutout}" benchmark: "benchmarks/" + CDIR + "build_cutout_{cutout}"
threads: ATLITE_NPROCESSES threads: ATLITE_NPROCESSES
resources: mem_mb=ATLITE_NPROCESSES * 1000 resources: mem_mb=ATLITE_NPROCESSES * 1000
script: "scripts/build_cutout.py" script: "scripts/build_cutout.py"
@ -162,8 +167,8 @@ if config['enable'].get('build_cutout', False):
if config['enable'].get('retrieve_cutout', True): if config['enable'].get('retrieve_cutout', True):
rule retrieve_cutout: rule retrieve_cutout:
input: HTTP.remote("zenodo.org/record/6382570/files/{cutout}.nc", keep_local=True, static=True) input: HTTP.remote("zenodo.org/record/6382570/files/{cutout}.nc", keep_local=True, static=True)
output: "cutouts/{cutout}.nc" output: "cutouts/" + CDIR + "{cutout}.nc"
log: "logs/retrieve_cutout_{cutout}.log" log: "logs/" + CDIR + "retrieve_cutout_{cutout}.log"
resources: mem_mb=5000 resources: mem_mb=5000
run: move(input[0], output[0]) run: move(input[0], output[0])
@ -171,7 +176,7 @@ if config['enable'].get('retrieve_cost_data', True):
rule retrieve_cost_data: rule retrieve_cost_data:
input: HTTP.remote(f"raw.githubusercontent.com/PyPSA/technology-data/{config['costs']['version']}/outputs/costs_{config['costs']['year']}.csv", keep_local=True) input: HTTP.remote(f"raw.githubusercontent.com/PyPSA/technology-data/{config['costs']['version']}/outputs/costs_{config['costs']['year']}.csv", keep_local=True)
output: COSTS output: COSTS
log: "logs/retrieve_cost_data.log" log: "logs/" + RDIR + "retrieve_cost_data.log"
resources: mem_mb=5000 resources: mem_mb=5000
run: move(input[0], output[0]) run: move(input[0], output[0])
@ -179,17 +184,17 @@ if config['enable'].get('build_natura_raster', False):
rule build_natura_raster: rule build_natura_raster:
input: input:
natura="data/bundle/natura/Natura2000_end2015.shp", natura="data/bundle/natura/Natura2000_end2015.shp",
cutouts=expand("cutouts/{cutouts}.nc", **config['atlite']) cutouts=expand("cutouts/" + CDIR + "{cutouts}.nc", **config['atlite'])
output: "resources/natura.tiff" output: "resources/" + RDIR + "natura.tiff"
resources: mem_mb=5000 resources: mem_mb=5000
log: "logs/build_natura_raster.log" log: "logs/" + RDIR + "build_natura_raster.log"
script: "scripts/build_natura_raster.py" script: "scripts/build_natura_raster.py"
if config['enable'].get('retrieve_natura_raster', True): if config['enable'].get('retrieve_natura_raster', True):
rule retrieve_natura_raster: rule retrieve_natura_raster:
input: HTTP.remote("zenodo.org/record/4706686/files/natura.tiff", keep_local=True, static=True) input: HTTP.remote("zenodo.org/record/4706686/files/natura.tiff", keep_local=True, static=True)
output: "resources/natura.tiff" output: "resources/" + RDIR + "natura.tiff"
resources: mem_mb=5000 resources: mem_mb=5000
run: move(input[0], output[0]) run: move(input[0], output[0])
@ -204,36 +209,36 @@ rule retrieve_ship_raster:
rule build_ship_raster: rule build_ship_raster:
input: input:
ship_density="data/shipdensity_global.zip", ship_density="data/shipdensity_global.zip",
cutouts=expand("cutouts/{cutouts}.nc", **config['atlite']) cutouts=expand("cutouts/" + CDIR + "{cutouts}.nc", **config['atlite'])
output: "resources/shipdensity_raster.nc" output: "resources/" + RDIR + "shipdensity_raster.nc"
log: "logs/build_ship_raster.log" log: "logs/" + RDIR + "build_ship_raster.log"
resources: mem_mb=5000 resources: mem_mb=5000
benchmark: "benchmarks/build_ship_raster" benchmark: "benchmarks/" + RDIR + "build_ship_raster"
script: "scripts/build_ship_raster.py" script: "scripts/build_ship_raster.py"
rule build_renewable_profiles: rule build_renewable_profiles:
input: input:
base_network="networks/base.nc", base_network="networks/" + RDIR + "base.nc",
corine="data/bundle/corine/g250_clc06_V18_5.tif", corine="data/bundle/corine/g250_clc06_V18_5.tif",
natura=lambda w: ("resources/natura.tiff" natura=lambda w: ("resources/" + RDIR + "natura.tiff"
if config["renewable"][w.technology]["natura"] if config["renewable"][w.technology]["natura"]
else []), else []),
gebco=lambda w: ("data/bundle/GEBCO_2014_2D.nc" gebco=lambda w: ("data/bundle/GEBCO_2014_2D.nc"
if "max_depth" in config["renewable"][w.technology].keys() if "max_depth" in config["renewable"][w.technology].keys()
else []), else []),
ship_density= lambda w: ("resources/shipdensity_raster.nc" ship_density= lambda w: ("resources/" + RDIR + "shipdensity_raster.nc"
if "ship_threshold" in config["renewable"][w.technology].keys() if "ship_threshold" in config["renewable"][w.technology].keys()
else []), else []),
country_shapes='resources/country_shapes.geojson', country_shapes="resources/" + RDIR + "country_shapes.geojson",
offshore_shapes='resources/offshore_shapes.geojson', offshore_shapes="resources/" + RDIR + "offshore_shapes.geojson",
regions=lambda w: ("resources/regions_onshore.geojson" regions=lambda w: ("resources/" + RDIR + "regions_onshore.geojson"
if w.technology in ('onwind', 'solar') if w.technology in ('onwind', 'solar')
else "resources/regions_offshore.geojson"), else "resources/" + RDIR + "regions_offshore.geojson"),
cutout=lambda w: "cutouts/" + config["renewable"][w.technology]['cutout'] + ".nc" cutout=lambda w: "cutouts/" + CDIR + config["renewable"][w.technology]['cutout'] + ".nc"
output: profile="resources/profile_{technology}.nc", output: profile="resources/" + RDIR + "profile_{technology}.nc",
log: "logs/build_renewable_profile_{technology}.log" log: "logs/" + RDIR + "build_renewable_profile_{technology}.log"
benchmark: "benchmarks/build_renewable_profiles_{technology}" benchmark: "benchmarks/" + RDIR + "build_renewable_profiles_{technology}"
threads: ATLITE_NPROCESSES threads: ATLITE_NPROCESSES
resources: mem_mb=ATLITE_NPROCESSES * 5000 resources: mem_mb=ATLITE_NPROCESSES * 5000
wildcard_constraints: technology="(?!hydro).*" # Any technology other than hydro wildcard_constraints: technology="(?!hydro).*" # Any technology other than hydro
@ -242,31 +247,33 @@ rule build_renewable_profiles:
rule build_hydro_profile: rule build_hydro_profile:
input: input:
country_shapes='resources/country_shapes.geojson', country_shapes="resources/" + RDIR + "country_shapes.geojson",
eia_hydro_generation='data/eia_hydro_annual_generation.csv', eia_hydro_generation='data/eia_hydro_annual_generation.csv',
cutout=f"cutouts/{config['renewable']['hydro']['cutout']}.nc" if "hydro" in config["renewable"] else "config['renewable']['hydro']['cutout'] not configured", cutout=f"cutouts/" + CDIR + "{config['renewable']['hydro']['cutout']}.nc" if "hydro" in config["renewable"] else "config['renewable']['hydro']['cutout'] not configured",
output: 'resources/profile_hydro.nc' output: "resources/" + RDIR + "profile_hydro.nc"
log: "logs/build_hydro_profile.log" log: "logs/" + RDIR + "build_hydro_profile.log"
resources: mem_mb=5000 resources: mem_mb=5000
script: 'scripts/build_hydro_profile.py' script: 'scripts/build_hydro_profile.py'
rule add_electricity: rule add_electricity:
input: input:
base_network='networks/base.nc', base_network="networks/" + RDIR + "base.nc",
tech_costs=COSTS, tech_costs=COSTS,
regions="resources/regions_onshore.geojson", regions="resources/" + RDIR + "regions_onshore.geojson",
powerplants='resources/powerplants.csv', powerplants="resources/" + RDIR + "powerplants.csv",
hydro_capacities='data/bundle/hydro_capacities.csv', hydro_capacities='data/bundle/hydro_capacities.csv',
geth_hydro_capacities='data/geth2015_hydro_capacities.csv', geth_hydro_capacities='data/geth2015_hydro_capacities.csv',
load='resources/load.csv', load="resources/" + RDIR + "load.csv",
nuts3_shapes='resources/nuts3_shapes.geojson', nuts3_shapes="resources/" + RDIR + "nuts3_shapes.geojson",
**{f"profile_{tech}": f"resources/profile_{tech}.nc" **{f"profile_{tech}": "resources/" + RDIR + f"profile_{tech}.nc"
for tech in config['renewable']}, for tech in config['renewable']},
**{f"conventional_{carrier}_{attr}": fn for carrier, d in config.get('conventional', {None: {}}).items() for attr, fn in d.items() if str(fn).startswith("data/")}, **{f"conventional_{carrier}_{attr}": fn
output: "networks/elec.nc" for carrier, d in config.get('conventional', {None: {}}).items()
log: "logs/add_electricity.log" for attr, fn in d.items() if str(fn).startswith("data/")},
benchmark: "benchmarks/add_electricity" output: "networks/" + RDIR + "elec.nc"
log: "logs/" + RDIR + "add_electricity.log"
benchmark: "benchmarks/" + RDIR + "add_electricity"
threads: 1 threads: 1
resources: mem_mb=5000 resources: mem_mb=5000
script: "scripts/add_electricity.py" script: "scripts/add_electricity.py"
@ -274,18 +281,18 @@ rule add_electricity:
rule simplify_network: rule simplify_network:
input: input:
network='networks/elec.nc', network="networks/" + RDIR + "elec.nc",
tech_costs=COSTS, tech_costs=COSTS,
regions_onshore="resources/regions_onshore.geojson", regions_onshore="resources/" + RDIR + "regions_onshore.geojson",
regions_offshore="resources/regions_offshore.geojson" regions_offshore="resources/" + RDIR + "regions_offshore.geojson"
output: output:
network='networks/elec_s{simpl}.nc', network="networks/" + RDIR + "elec_s{simpl}.nc",
regions_onshore="resources/regions_onshore_elec_s{simpl}.geojson", regions_onshore="resources/" + RDIR + "regions_onshore_elec_s{simpl}.geojson",
regions_offshore="resources/regions_offshore_elec_s{simpl}.geojson", regions_offshore="resources/" + RDIR + "regions_offshore_elec_s{simpl}.geojson",
busmap='resources/busmap_elec_s{simpl}.csv', busmap="resources/" + RDIR + "busmap_elec_s{simpl}.csv",
connection_costs='resources/connection_costs_s{simpl}.csv' connection_costs="resources/" + RDIR + "connection_costs_s{simpl}.csv"
log: "logs/simplify_network/elec_s{simpl}.log" log: "logs/" + RDIR + "simplify_network/elec_s{simpl}.log"
benchmark: "benchmarks/simplify_network/elec_s{simpl}" benchmark: "benchmarks/" + RDIR + "simplify_network/elec_s{simpl}"
threads: 1 threads: 1
resources: mem_mb=4000 resources: mem_mb=4000
script: "scripts/simplify_network.py" script: "scripts/simplify_network.py"
@ -293,21 +300,21 @@ rule simplify_network:
rule cluster_network: rule cluster_network:
input: input:
network='networks/elec_s{simpl}.nc', network="networks/" + RDIR + "elec_s{simpl}.nc",
regions_onshore="resources/regions_onshore_elec_s{simpl}.geojson", regions_onshore="resources/" + RDIR + "regions_onshore_elec_s{simpl}.geojson",
regions_offshore="resources/regions_offshore_elec_s{simpl}.geojson", regions_offshore="resources/" + RDIR + "regions_offshore_elec_s{simpl}.geojson",
busmap=ancient('resources/busmap_elec_s{simpl}.csv'), busmap=ancient("resources/" + RDIR + "busmap_elec_s{simpl}.csv"),
custom_busmap=("data/custom_busmap_elec_s{simpl}_{clusters}.csv" custom_busmap=("data/custom_busmap_elec_s{simpl}_{clusters}.csv"
if config["enable"].get("custom_busmap", False) else []), if config["enable"].get("custom_busmap", False) else []),
tech_costs=COSTS tech_costs=COSTS
output: output:
network='networks/elec_s{simpl}_{clusters}.nc', network="networks/" + RDIR + "elec_s{simpl}_{clusters}.nc",
regions_onshore="resources/regions_onshore_elec_s{simpl}_{clusters}.geojson", regions_onshore="resources/" + RDIR + "regions_onshore_elec_s{simpl}_{clusters}.geojson",
regions_offshore="resources/regions_offshore_elec_s{simpl}_{clusters}.geojson", regions_offshore="resources/" + RDIR + "regions_offshore_elec_s{simpl}_{clusters}.geojson",
busmap="resources/busmap_elec_s{simpl}_{clusters}.csv", busmap="resources/" + RDIR + "busmap_elec_s{simpl}_{clusters}.csv",
linemap="resources/linemap_elec_s{simpl}_{clusters}.csv" linemap="resources/" + RDIR + "linemap_elec_s{simpl}_{clusters}.csv"
log: "logs/cluster_network/elec_s{simpl}_{clusters}.log" log: "logs/" + RDIR + "cluster_network/elec_s{simpl}_{clusters}.log"
benchmark: "benchmarks/cluster_network/elec_s{simpl}_{clusters}" benchmark: "benchmarks/" + RDIR + "cluster_network/elec_s{simpl}_{clusters}"
threads: 1 threads: 1
resources: mem_mb=6000 resources: mem_mb=6000
script: "scripts/cluster_network.py" script: "scripts/cluster_network.py"
@ -315,21 +322,21 @@ rule cluster_network:
rule add_extra_components: rule add_extra_components:
input: input:
network='networks/elec_s{simpl}_{clusters}.nc', network="networks/" + RDIR + "elec_s{simpl}_{clusters}.nc",
tech_costs=COSTS, tech_costs=COSTS,
output: 'networks/elec_s{simpl}_{clusters}_ec.nc' output: "networks/" + RDIR + "elec_s{simpl}_{clusters}_ec.nc"
log: "logs/add_extra_components/elec_s{simpl}_{clusters}.log" log: "logs/" + RDIR + "add_extra_components/elec_s{simpl}_{clusters}.log"
benchmark: "benchmarks/add_extra_components/elec_s{simpl}_{clusters}_ec" benchmark: "benchmarks/" + RDIR + "add_extra_components/elec_s{simpl}_{clusters}_ec"
threads: 1 threads: 1
resources: mem_mb=3000 resources: mem_mb=3000
script: "scripts/add_extra_components.py" script: "scripts/add_extra_components.py"
rule prepare_network: rule prepare_network:
input: 'networks/elec_s{simpl}_{clusters}_ec.nc', tech_costs=COSTS input: "networks/" + RDIR + "elec_s{simpl}_{clusters}_ec.nc", tech_costs=COSTS,
output: 'networks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc' output: "networks/" + RDIR + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc"
log: "logs/prepare_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.log" log: "logs/" + RDIR + "prepare_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.log"
benchmark: "benchmarks/prepare_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}" benchmark: "benchmarks/" + RDIR + "prepare_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}"
threads: 1 threads: 1
resources: mem_mb=4000 resources: mem_mb=4000
script: "scripts/prepare_network.py" script: "scripts/prepare_network.py"
@ -356,13 +363,13 @@ def memory(w):
rule solve_network: rule solve_network:
input: "networks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc" input: "networks/" + RDIR + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc"
output: "results/networks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc" output: "results/networks/" + RDIR + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc"
log: log:
solver=normpath("logs/solve_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_solver.log"), solver=normpath("logs/" + RDIR + "solve_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_solver.log"),
python="logs/solve_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_python.log", python="logs/" + RDIR + "solve_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_python.log",
memory="logs/solve_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_memory.log" memory="logs/" + RDIR + "solve_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_memory.log"
benchmark: "benchmarks/solve_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}" benchmark: "benchmarks/" + RDIR + "solve_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}"
threads: 4 threads: 4
resources: mem_mb=memory resources: mem_mb=memory
shadow: "minimal" shadow: "minimal"
@ -371,14 +378,14 @@ rule solve_network:
rule solve_operations_network: rule solve_operations_network:
input: input:
unprepared="networks/elec_s{simpl}_{clusters}_ec.nc", unprepared="networks/" + RDIR + "elec_s{simpl}_{clusters}_ec.nc",
optimized="results/networks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc" optimized="results/networks/" + RDIR + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc"
output: "results/networks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_op.nc" output: "results/networks/" + RDIR + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_op.nc"
log: log:
solver=normpath("logs/solve_operations_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_op_solver.log"), solver=normpath("logs/" + RDIR + "solve_operations_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_op_solver.log"),
python="logs/solve_operations_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_op_python.log", python="logs/" + RDIR + "solve_operations_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_op_python.log",
memory="logs/solve_operations_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_op_memory.log" memory="logs/" + RDIR + "solve_operations_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_op_memory.log"
benchmark: "benchmarks/solve_operations_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}" benchmark: "benchmarks/" + RDIR + "solve_operations_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}"
threads: 4 threads: 4
resources: mem_mb=(lambda w: 5000 + 372 * int(w.clusters)) resources: mem_mb=(lambda w: 5000 + 372 * int(w.clusters))
shadow: "minimal" shadow: "minimal"
@ -387,12 +394,12 @@ rule solve_operations_network:
rule plot_network: rule plot_network:
input: input:
network="results/networks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc", network="results/networks/" + RDIR + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc",
tech_costs=COSTS tech_costs=COSTS
output: output:
only_map="results/plots/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{attr}.{ext}", only_map="results/plots/" + RDIR + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{attr}.{ext}",
ext="results/plots/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{attr}_ext.{ext}" ext="results/plots/" + RDIR + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{attr}_ext.{ext}"
log: "logs/plot_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{attr}_{ext}.log" log: "logs/" + RDIR + "plot_network/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{attr}_{ext}.log"
script: "scripts/plot_network.py" script: "scripts/plot_network.py"
@ -405,7 +412,7 @@ def input_make_summary(w):
else: else:
ll = w.ll ll = w.ll
return ([COSTS] + return ([COSTS] +
expand("results/networks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc", expand("results/networks/" + RDIR + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}.nc",
ll=ll, ll=ll,
**{k: config["scenario"][k] if getattr(w, k) == "all" else getattr(w, k) **{k: config["scenario"][k] if getattr(w, k) == "all" else getattr(w, k)
for k in ["simpl", "clusters", "opts"]})) for k in ["simpl", "clusters", "opts"]}))
@ -413,30 +420,30 @@ def input_make_summary(w):
rule make_summary: rule make_summary:
input: input_make_summary input: input_make_summary
output: directory("results/summaries/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{country}") output: directory("results/summaries/" + RDIR + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{country}")
log: "logs/make_summary/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{country}.log", log: "logs/" + RDIR + "make_summary/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{country}.log",
resources: mem_mb=500 resources: mem_mb=500
script: "scripts/make_summary.py" script: "scripts/make_summary.py"
rule plot_summary: rule plot_summary:
input: "results/summaries/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{country}" input: "results/summaries/" + RDIR + "elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{country}"
output: "results/plots/summary_{summary}_elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{country}.{ext}" output: "results/plots/" + RDIR + "summary_{summary}_elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{country}.{ext}"
log: "logs/plot_summary/{summary}_elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{country}_{ext}.log" log: "logs/" + RDIR + "plot_summary/{summary}_elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{country}_{ext}.log"
resources: mem_mb=500 resources: mem_mb=500
script: "scripts/plot_summary.py" script: "scripts/plot_summary.py"
def input_plot_p_nom_max(w): def input_plot_p_nom_max(w):
return [("results/networks/elec_s{simpl}{maybe_cluster}.nc" return [("results/networks/" + RDIR + "elec_s{simpl}{maybe_cluster}.nc"
.format(maybe_cluster=('' if c == 'full' else ('_' + c)), **w)) .format(maybe_cluster=('' if c == 'full' else ('_' + c)), **w))
for c in w.clusts.split(",")] for c in w.clusts.split(",")]
rule plot_p_nom_max: rule plot_p_nom_max:
input: input_plot_p_nom_max input: input_plot_p_nom_max
output: "results/plots/elec_s{simpl}_cum_p_nom_max_{clusts}_{techs}_{country}.{ext}" output: "results/plots/" + RDIR + "elec_s{simpl}_cum_p_nom_max_{clusts}_{techs}_{country}.{ext}"
log: "logs/plot_p_nom_max/elec_s{simpl}_{clusts}_{techs}_{country}_{ext}.log" log: "logs/" + RDIR + "plot_p_nom_max/elec_s{simpl}_{clusts}_{techs}_{country}_{ext}.log"
resources: mem_mb=500 resources: mem_mb=500
script: "scripts/plot_p_nom_max.py" script: "scripts/plot_p_nom_max.py"

5
config.default.yaml
View File

@ -9,6 +9,11 @@ logging:
level: INFO level: INFO
format: '%(levelname)s:%(name)s:%(message)s' format: '%(levelname)s:%(name)s:%(message)s'
run:
name: "" # use this to keep track of runs with different settings
shared_cutouts: false # set to true to share the default cutout(s) across runs
scenario: scenario:
simpl: [''] simpl: ['']
ll: ['copt'] ll: ['copt']

2
config.tutorial.yaml
View File

@ -9,6 +9,8 @@ logging:
level: INFO level: INFO
format: '%(levelname)s:%(name)s:%(message)s' format: '%(levelname)s:%(name)s:%(message)s'
run:
name: ""
scenario: scenario:
simpl: [''] simpl: ['']

15
doc/configuration.rst
View File

@ -28,13 +28,24 @@ Top-level configuration
.. _scenario: .. _scenario:
``scenario`` ``run``
============ =======
It is common conduct to analyse energy system optimisation models for **multiple scenarios** for a variety of reasons, It is common conduct to analyse energy system optimisation models for **multiple scenarios** for a variety of reasons,
e.g. assessing their sensitivity towards changing the temporal and/or geographical resolution or investigating how e.g. assessing their sensitivity towards changing the temporal and/or geographical resolution or investigating how
investment changes as more ambitious greenhouse-gas emission reduction targets are applied. investment changes as more ambitious greenhouse-gas emission reduction targets are applied.
The ``run`` section is used for running and storing scenarios with different configurations which are not covered by :ref:`wildcards`. It determines the path at which resources, networks and results are stored. Therefore the user can run different configurations within the same directory. If a run with a non-empty name should use cutouts shared across runs, set ``shared_cutouts`` to `true`.
.. literalinclude:: ../config.default.yaml
:language: yaml
:start-at: run:
:end-before: scenario:
``scenario``
============
The ``scenario`` section is an extraordinary section of the config file The ``scenario`` section is an extraordinary section of the config file
that is strongly connected to the :ref:`wildcards` and is designed to that is strongly connected to the :ref:`wildcards` and is designed to
facilitate running multiple scenarios through a single command facilitate running multiple scenarios through a single command

2
doc/release_notes.rst
View File

@ -34,6 +34,8 @@ PyPSA-Eur 0.6.0 (10th September 2022)
setting ``excluder_resolution``. setting ``excluder_resolution``.
* The software now supports running the workflow with different settings within the same directory. A new config section ``run`` was created that specifies under which scenario ``name`` the created resources, networks and results should be stored. If ``name`` is not specified, the workflow uses the default paths. The entry ``shared_cutouts`` specifies whether the run should use cutouts from the default root directory or use run-specific cutouts.
PyPSA-Eur 0.5.0 (27th July 2022) PyPSA-Eur 0.5.0 (27th July 2022)
===================================== =====================================

2
test/config.test1.yaml
View File

@ -8,6 +8,8 @@ logging:
level: INFO level: INFO
format: '%(levelname)s:%(name)s:%(message)s' format: '%(levelname)s:%(name)s:%(message)s'
run:
name: ""
scenario: scenario:
simpl: [''] simpl: ['']