Commit Graph

400 Commits

Author SHA1 Message Date
Fabian Neumann
013b705ee4
Clustering: build renewable profiles and add all assets after clustering (#1201)
* Cluster first: build renewable profiles and add all assets after clustering

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* correction: pass landfall_lengths through functions

* assign landfall_lenghts correctly

* remove parameter add_land_use_constraint

* fix network_dict

* calculate distance to shoreline, remove underwater_fraction

* adjust simplification parameter to exclude Crete from offshore wind connections

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove unused geth2015 hydro capacities

* removing remaining traces of {simpl} wildcard

* add release notes and update workflow graphics

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lisazeyen <lisa.zeyen@web.de>
2024-09-13 15:37:01 +02:00
Bobby Xiong
0c36de9bf8
Introducing OpenStreetMap high-voltage grid to PyPSA-Eur (#1079)
* Implemented  which uses the overpass API to download power features for individual countries.

* Extended  rule by input.

* Bug fixes and improvements to clean_osm_data.py. Added  in retrieve_osm_data.py.

* Updated clean_osm_data and retrieve_osm_data to create clean substations.

* Finished clean_osm_data function.

* Added check whether line is a circle. If so, drop it.

* Extended build_electricity.smk by build_osm_network.py

* Added build_osm_network

* Working osm-network-fast

* Bug fixes.

* Finalised and cleaned  including docstrings.

* Added try catch to retrieve_osm_data. Allows for parallelisation of downloads.

* Updated cleaning process.

* Set maximum number of threads for retrieving to 4, wrt. fair usage policy and potential request errors.

* Intermediate update on clean_osm_data.py. Added docstrings.

* Bug fix.

* Bug fix.

* Bug fixes in data types out of clean_osm_data

* Significant improvements to retrieve_osm_data, clean_osm_data. Cleaned code. Speed improvements

* Cleaned config.

* Fixes.

* Bug fixes.

* Updated default config

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Removed overpass from required packages. Not needed anymore.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added links_relations (route = power, frequency = 0) to retrieval. This will change how HVDC links are extracted in the near future.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Work-in-progress clean_osm_data

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added clean links output to clean_osm_data. Script uses OSM relations to retrieve clean HVDC links.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* New code for integrating HVDC links. Using relations. Base network implementation functioning.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed manual line dropping.

* Updated clean script

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* reverted Snakefile to default: sync settings

* added prebuilt functionality.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated build_electricity.smk to work with scenario management.

* removed commented-out code.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed commented-out code.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fixed bug in pdf export by substituting pdf export with svg.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Bug-fix Snakefile

* dropped not needed columns from build_osm_network.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated build_shapes, config.default and clean_osm_data.

* pre-commit changes.

* test

* Added initial prepare_osm_network_release.py script

* Finalised prepare_osm_network_release script to build clean and stable OSM base_network input files.

* Added new rules/development.smk

* Updated clean_osm_data to add substation_centroid to linestrings

* Updated clean_osm_data to add substation_centroid to linestrings

* Updated clean_osm_data to add substation_centroid to linestrings

* Updated clean_osm_data to add substation_centroid to linestrings

* Added osm-prebuilt functionality and zenodo sandbox repository.

* Updated clean_osm_data to geopandas v.1.01

* Made base_network and build_osm_network function more robust for empty links.

* Made base_network and build_osm_network function more robust for empty links.

* Bug fix in base_network. Voltage level null is now kept (relevant e.g. for Corsica)

* Merge with hcanges in upstream PR 1146. Fixing UA and MD.

* Updated Zenodo and fixed prepare_osm_network_release

* Updated osm network release.

* Updated prepare osm network release.

* Updated MD, UA scripts.

* Cleaned determine_availability_matrix_MD_UA.py, removed redundant code

* Bug fixes.

* Bug fixes for UA MD scripts.

* Rename of build script.

* Bug fix: only distribute load to buses with substation.

* Updated zenodo sandbox repository.

* Updated config.default

* Cleaned config.default.yaml: Related settings grouped together and redundant voltage settings aggregated.

* Cleaned config.default.yaml: Related settings grouped together and redundant voltage settings aggregated. Added release notes.

* Updated Zenodo repositories for OSM-prebuilt to offcial publication.

* Updated configtables

* Updated links.csv: Under_construction lines to in commission.

* Updated link 8394 and parameter_corrections: Continuation of North-Sea-Link.

* Major update: fix simplify_network, fix Corsica, updated build_osm_network to include lines overpassing nodes.

* remove config backup

* Bug fix: Carrier type of all supernodes corrected to 'AC'

* Bug fix: Carrier type of all supernodes corrected to 'AC'

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated rules and base_network for compatibility with TYNDP projects.

* Updated Zenodo repository and prebuilt network to include 150 kV HVDC connections.

* Removed outdated config backup.

* Implemented all comments from PR #1079. Cleaned up OSM implementation.

* Bug fix: Added all voltages, 200 kV-750 kV, to default config.

* Cleaning and bugfixes.

* Updated Zenodo repository to https://zenodo.org/records/13358976. Added converter voltages, 'underground' property for DC lines/cables, and included Konti-Skan HVDC (DK-SE). Added compatibility with https://github.com/PyPSA/pypsa-eur/pull/1079 and https://github.com/PyPSA/pypsa-eur/pull/1085

* Apply suggestions from code review

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* simplify_network: handle complicated transformer topologies

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* syntax fix

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
2024-08-22 15:01:20 +02:00
Bobby Xiong
1b0ef5ebd5 Bug fix Snakefile: rule all input updated to 'graphs/costs.svg'. 2024-06-11 15:36:51 +02:00
Fabian Neumann
3aaea4699b fix missed renaming of mode to policy 2024-05-13 14:50:38 +02:00
Fabian Neumann
6321a69870 rearrange config settings under 'run: shared_resources:' 2024-05-13 14:23:09 +02:00
toniseibold
e75e3bd8d7 allow excluding files from shared_resources 2024-05-13 14:21:43 +02:00
Fabian Neumann
edc1d4991a reenable windows ci with snakemake 8.11 2024-04-26 09:35:54 +02:00
Fabian Hofmann
7611b28d16
Merge branch 'master' into refactor-copy-config-get-rdir 2024-03-20 09:57:31 +01:00
Koen van Greevenbroek
651e28775e Use raw strings to avoid illegal backslash warnings python 12 2024-03-19 14:22:16 +01:00
Fabian
8190fde9ce Snakefile: reinsert shutil and os functions 2024-03-19 09:48:52 +01:00
Fabian
23e1139c21 snakefile + helpers: separate scenario and rdir getter 2024-03-19 09:39:35 +01:00
Fabian
2565a7db4f snakefile: move copy_default_files and process_run_config to helpers 2024-03-19 08:30:20 +01:00
Fabian Neumann
ed71978b92 use humanfriendly walltime specifications 2024-03-09 14:47:01 +01:00
Fabian Neumann
d983d79ed2 Merge branch 'master' into snakemake-8 2024-03-08 17:50:08 +01:00
Fabian Neumann
66d078e407 reorder release notes 2024-02-20 14:24:23 +01:00
Fabian Neumann
719d4dad23 Merge branch 'master' into scenario-management 2024-02-20 12:08:54 +01:00
pre-commit-ci[bot]
f4422f1361 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2024-02-19 16:17:34 +00:00
Fabian Neumann
8e04b7a157 test.sh: add scenarios dry-run to test cases 2024-02-18 11:03:37 +01:00
Fabian Neumann
1c0daf826c Merge branch 'scenario-management' of github.com:PyPSA/pypsa-eur into scenario-management 2024-02-18 10:41:55 +01:00
Fabian Neumann
83f1b5e41d Add rule create_scenarios that runs config/create_scenarios.py on request 2024-02-18 10:41:31 +01:00
Fabian Neumann
67acbbda8a run: name: all will collect all keys from scenarios.yaml if run: scenarios: enable: true 2024-02-18 10:24:26 +01:00
pre-commit-ci[bot]
831f0779b3 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2024-02-18 08:57:26 +00:00
Fabian Neumann
20e37c232f copy a scenarios.yaml from scenarios.template.yaml so it is not git tracked 2024-02-18 09:55:45 +01:00
Fabian Neumann
f710923320 Merge branch 'master' into scenario-management 2024-02-18 09:48:12 +01:00
Fabian Neumann
3e612950c2 revert part of #925 so that config.default.yaml is still copied and config.yaml remains untracked 2024-02-18 09:46:54 +01:00
Fabian Neumann
16793d1335 Snakefile: don't use {run} wildcard if just specifying a run name 2024-02-17 20:42:37 +01:00
Fabian Neumann
db7c983950 mock_snakemake: remove dangeours empty list as default argument 2024-02-17 18:14:18 +01:00
Fabian Neumann
709b831e06 further adjustments where config_provider needs to be called 2024-02-17 17:51:26 +01:00
Fabian Neumann
16e42c8fe6 further progress and bugfixes 2024-02-16 11:17:00 +01:00
Fabian Neumann
dbbfbaf1f3 Merge branch 'master' into scenario-management 2024-02-14 10:45:43 +01:00
Fabian Neumann
c3feb0e283 handle complicated config-dependent inputs using unpack() with input functions 2024-02-12 16:56:00 +01:00
Fabian Neumann
c83db4e84e add retrieve_cost_data.py script to handle config_provider and move to resources
since dependent on config setting
2024-02-12 16:54:33 +01:00
pre-commit-ci[bot]
abe06a68fb [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2024-02-12 11:11:04 +00:00
Fabian Neumann
fe143ff8fe use default config as base configuration 2024-02-12 12:10:05 +01:00
Fabian Neumann
a9dad3f34e add new resources(), logs(), benchmarks(), config_providers() 2024-02-10 18:09:23 +01:00
Fabian Neumann
6de08bd523 Merge branch 'master' into scenario-management 2024-02-10 17:22:01 +01:00
Fabian
d32789bd02 snakefile: allow to share resources in directory
ci: use test.sh script which can be executed locally
2024-01-31 16:53:14 +01:00
Koen van Greevenbroek
e4a130353f Make copying default config file compatible with snakemake module 2024-01-24 14:47:20 +01:00
Fabian
2d027e80c3 fix capital costs of co2 tanks 2024-01-04 19:54:25 +01:00
Fabian
254d50b1b4 prepare sectors: allow for updating co2 network costs 2024-01-04 19:49:59 +01:00
Fabian Neumann
1a6031f318 only copy config.default.yaml if it exists 2024-01-03 15:25:27 +01:00
Fabian Neumann
ea95f2b38c bumb minimum snakemake version to v8 2023-12-31 13:01:17 +01:00
Fabian Neumann
48850431fb Snakefile: switch to new storage providers 2023-12-30 14:37:45 +01:00
pre-commit-ci[bot]
9765b24c1c [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2023-09-29 06:40:25 +00:00
lisazeyen
21d1109853
Merge branch 'master' into multi-horizon 2023-09-29 08:40:05 +02:00
Lukas Franken
834f7ea144
Add dialogue to rule purge to prevent accidents (#745) 2023-09-28 17:19:13 +02:00
chrstphtrs
138584ecf7 Move all rule down to other rules 2023-09-22 09:01:12 +02:00
chrstphtrs
8b5476e273 Move all rule to main Snakefile 2023-09-20 19:33:44 +02:00
Fabian
04a2bea176 scenario-management: use list for including wildcards with shared resource 2023-08-24 13:20:11 +02:00
Fabian
62c6143863 scenario-management: reenable shared resources, make shared resources wildcards dependent 2023-08-23 17:14:57 +02:00