Commit Graph

150 Commits

Author SHA1 Message Date
Philipp Glaum
d4bad07cf4
build_transmission_projects: set s_max_pu to the value according to config (#1323) 2024-09-25 16:58:35 +02:00
Fabian Neumann
013b705ee4
Clustering: build renewable profiles and add all assets after clustering (#1201)
* Cluster first: build renewable profiles and add all assets after clustering

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* correction: pass landfall_lengths through functions

* assign landfall_lenghts correctly

* remove parameter add_land_use_constraint

* fix network_dict

* calculate distance to shoreline, remove underwater_fraction

* adjust simplification parameter to exclude Crete from offshore wind connections

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove unused geth2015 hydro capacities

* removing remaining traces of {simpl} wildcard

* add release notes and update workflow graphics

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lisazeyen <lisa.zeyen@web.de>
2024-09-13 15:37:01 +02:00
Bobby Xiong
42f2c37ebf
Updated osm-prebuilt network to v0.4 and added version control to config (#1293)
* Updated base_network script and rules.

* Updated osm-prebuilt to version 0.4 and added version control.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-09-13 14:55:52 +02:00
cpschau
e35f651518
Update NUTS3 shapes (#1286)
* increase nuts3 shape resolution

* retrieve and use shapes with EPSG:4326

* co-retrieve nuts2 shapes; add retries

* added release note

* changed doc/data_sources
2024-09-12 19:20:07 +02:00
Fabian Neumann
b207b4ebe8
remove unused geth2015 hydro capacities (#1246) 2024-08-30 10:26:48 +02:00
Tom Brown
66051e04d4
build_electricity: raise memory for build_tranmission_projects (#1237)
2 GB RAM was not enough, to avoid out-of-memory problems raised to 4 GB.
2024-08-26 16:51:57 +02:00
Bobby Xiong
ef0bbd5f37
Custom busmap (#1231)
* Added functionality to set custom_busmaps for different base_networks.

* Updated configtables and location for busmaps.

* Added release nots.

* removed run parameter in mock_snakemake (only used for debugging).

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-08-23 17:35:46 +02:00
Bobby Xiong
0c36de9bf8
Introducing OpenStreetMap high-voltage grid to PyPSA-Eur (#1079)
* Implemented  which uses the overpass API to download power features for individual countries.

* Extended  rule by input.

* Bug fixes and improvements to clean_osm_data.py. Added  in retrieve_osm_data.py.

* Updated clean_osm_data and retrieve_osm_data to create clean substations.

* Finished clean_osm_data function.

* Added check whether line is a circle. If so, drop it.

* Extended build_electricity.smk by build_osm_network.py

* Added build_osm_network

* Working osm-network-fast

* Bug fixes.

* Finalised and cleaned  including docstrings.

* Added try catch to retrieve_osm_data. Allows for parallelisation of downloads.

* Updated cleaning process.

* Set maximum number of threads for retrieving to 4, wrt. fair usage policy and potential request errors.

* Intermediate update on clean_osm_data.py. Added docstrings.

* Bug fix.

* Bug fix.

* Bug fixes in data types out of clean_osm_data

* Significant improvements to retrieve_osm_data, clean_osm_data. Cleaned code. Speed improvements

* Cleaned config.

* Fixes.

* Bug fixes.

* Updated default config

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Removed overpass from required packages. Not needed anymore.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added links_relations (route = power, frequency = 0) to retrieval. This will change how HVDC links are extracted in the near future.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Work-in-progress clean_osm_data

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added clean links output to clean_osm_data. Script uses OSM relations to retrieve clean HVDC links.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* New code for integrating HVDC links. Using relations. Base network implementation functioning.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed manual line dropping.

* Updated clean script

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* reverted Snakefile to default: sync settings

* added prebuilt functionality.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated build_electricity.smk to work with scenario management.

* removed commented-out code.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed commented-out code.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fixed bug in pdf export by substituting pdf export with svg.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Bug-fix Snakefile

* dropped not needed columns from build_osm_network.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated build_shapes, config.default and clean_osm_data.

* pre-commit changes.

* test

* Added initial prepare_osm_network_release.py script

* Finalised prepare_osm_network_release script to build clean and stable OSM base_network input files.

* Added new rules/development.smk

* Updated clean_osm_data to add substation_centroid to linestrings

* Updated clean_osm_data to add substation_centroid to linestrings

* Updated clean_osm_data to add substation_centroid to linestrings

* Updated clean_osm_data to add substation_centroid to linestrings

* Added osm-prebuilt functionality and zenodo sandbox repository.

* Updated clean_osm_data to geopandas v.1.01

* Made base_network and build_osm_network function more robust for empty links.

* Made base_network and build_osm_network function more robust for empty links.

* Bug fix in base_network. Voltage level null is now kept (relevant e.g. for Corsica)

* Merge with hcanges in upstream PR 1146. Fixing UA and MD.

* Updated Zenodo and fixed prepare_osm_network_release

* Updated osm network release.

* Updated prepare osm network release.

* Updated MD, UA scripts.

* Cleaned determine_availability_matrix_MD_UA.py, removed redundant code

* Bug fixes.

* Bug fixes for UA MD scripts.

* Rename of build script.

* Bug fix: only distribute load to buses with substation.

* Updated zenodo sandbox repository.

* Updated config.default

* Cleaned config.default.yaml: Related settings grouped together and redundant voltage settings aggregated.

* Cleaned config.default.yaml: Related settings grouped together and redundant voltage settings aggregated. Added release notes.

* Updated Zenodo repositories for OSM-prebuilt to offcial publication.

* Updated configtables

* Updated links.csv: Under_construction lines to in commission.

* Updated link 8394 and parameter_corrections: Continuation of North-Sea-Link.

* Major update: fix simplify_network, fix Corsica, updated build_osm_network to include lines overpassing nodes.

* remove config backup

* Bug fix: Carrier type of all supernodes corrected to 'AC'

* Bug fix: Carrier type of all supernodes corrected to 'AC'

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated rules and base_network for compatibility with TYNDP projects.

* Updated Zenodo repository and prebuilt network to include 150 kV HVDC connections.

* Removed outdated config backup.

* Implemented all comments from PR #1079. Cleaned up OSM implementation.

* Bug fix: Added all voltages, 200 kV-750 kV, to default config.

* Cleaning and bugfixes.

* Updated Zenodo repository to https://zenodo.org/records/13358976. Added converter voltages, 'underground' property for DC lines/cables, and included Konti-Skan HVDC (DK-SE). Added compatibility with https://github.com/PyPSA/pypsa-eur/pull/1079 and https://github.com/PyPSA/pypsa-eur/pull/1085

* Apply suggestions from code review

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* simplify_network: handle complicated transformer topologies

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* syntax fix

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
2024-08-22 15:01:20 +02:00
Philipp Glaum
a9f67b313f
handle new and upgraded TYNDP&NEP lines/links in base network (OSM compatible) (#1085)
* add general implementation to add tyndp and nep. TODO: Remove duplicate?

* pre_osm_merge

* clean and update base_network.py
update default config settings

* clean and update base_network.py
update default config settings

* base_network.py:remove adding of transmission projects
add_transmission_project.py: add new script for creating lines and link csv from transmission projects
add_electricity.py: add new projects from created csv files

* cluster_network.py: do not allow to group lines with different build years together-> requires pypsa update
simplify_network.py: fix bug of simplify links

* remove legacies of removing transmission projects from base_network

* restructure folders:new folder for tranmission project csv
add_transmission_projects: improve logging, cleanup

* update lines csvs and add default line type for upgraded and new lines

* remove duplicate lines which were already in gridkit

* allow to connect ac lines only to ac buses

* add manual links csv (adjusted links_tyndp.csv) and update default config

* add realease note

* remove links_tyndp.csv file, references in build_elec.smk and function in base_network.py to add them

* add configuration options for transmission projects to documentation and add template folder for transmission projects

* update pypsa version in environments

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* integrate Fabian's review comments 2

* add electricity:add import pathlib and duplicate printing out of adding line projects

* update NEP line csv

* address Fabian's comments

* build_transmission_projects: address Fabian's final comments
simplify_network: use modus to get line type which appears most often

* build_transmission_project: change from .geometry to ["geometry"]

* build_transmission_projects: remove redundanty line

* build_transmission_projects: remove buffer for europe shape because of higher resolution
default config: fix wrong key for skip argument in transmission_projects

* update configtables and default config

* update manual links csv and delete undetected duplicate links in tyndp2020

* final adjustments

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
2024-08-15 11:42:21 +02:00
Fabian Neumann
f263862455
use lower resolution EEZ shapes to reduce excessive RAM use (#1210) 2024-08-13 21:03:07 +02:00
Fabian Neumann
fb41016c60
EEZ: Update EEZ to v12, auto-download and remove from databundle (#1188)
* eez: update to version 12, autodownload, remove pycountry

* eez: do not simplify as it distorts topology

* remove missed merge conflicts
2024-08-07 15:28:55 +02:00
Fabian Neumann
c907d59253
remove unused rule prepare_links_p_nom (#1203)
* remove rule

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-08-07 15:20:08 +02:00
Fabian Neumann
311c82d65e naturalearth: automatically download and remove from data bundle 2024-08-02 09:53:34 +02:00
Fabian Neumann
b49895a041
determine_availability_matrix_MD_UA: enable parallelism & remove plots (#1170)
* determine_availability_matrix_MD_UA: enable parallelism through temp files and remove plots

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-07-19 17:43:11 +00:00
Iegor Riepin
ba55971c23
Compatibility of data processing for Ukraine (#1146)
* data: retrieve gdp and pop raw data for UA,MD

* Updated retrieval code.

* add_electricity script for UA and MD using updated, endogenised GDP and PPP data.

* Outsourced building of gdp and ppp. Only called when UA and/or MD are in country list.

* Accepted suggestion by @fneum in retrieve rule.

* Cleaned determine_availability_matrix_MD_UA.py, removed redundant code

* Updated build_gdp_pop_non_nuts3 script to use the mean to distribute GDP p.c. Added release notes. Bug fixes.

* Bug fixes --> 'ppp' to 'pop'

* Bug fix: only distribute load to buses with substation.

* Updated to download GDP and population raster via PyPSA-Eur zenodo databundle. Removal of dedicated retrieve function. Updated release notes.

* Updated release notes.

* correct input file paths

* update zenodo url for databundle release v0.3.0

* doc: update release_notes

* minor adjustments: benchmark, increase mem_mb, remove plots, handling of optional inputs

---------

Co-authored-by: bobbyxng <bobbyxng@gmail.com>
Co-authored-by: Bobby Xiong <36541459+bobbyxng@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
2024-07-19 19:23:35 +02:00
Fabian Neumann
54009e7af8 increase mem_mb for build_powerplants rule 2024-07-01 13:33:29 +01:00
Fabian Neumann
5397f3fe85 avoid duplicate existing RES capacities (closes #1016) 2024-05-24 09:47:23 +02:00
lisazeyen
2223e011a0 merge master 2024-05-15 16:14:25 +02:00
pre-commit-ci[bot]
985aecb0c4 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2024-05-13 19:39:02 +00:00
Fabian Neumann
84d6634af0
Merge branch 'master' into implement-floating 2024-05-13 10:04:42 +02:00
Fabian Neumann
0d9e0f34fd Merge branch 'master' into reduced-data-bundle 2024-05-12 15:53:08 +02:00
Philipp Glaum
da617be276 Merge branch 'master' of github.com:PyPSA/pypsa-eur into implement-floating 2024-05-10 13:29:58 +02:00
Koen van Greevenbroek
76c201ef7d Integrate build_bus_regions into base_network
Fixes a problem with the `build_bus_regions` writing to base.nc
without declaring base.nc as an output.
2024-05-03 14:53:18 +02:00
Parisra
17b3e1677d
Merge branch 'PyPSA:master' into adding_solar_tracking_single_axis 2024-04-29 14:27:26 +02:00
pre-commit-ci[bot]
105cadaf7b [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2024-04-25 16:08:48 +00:00
Parisra
b9cb82cc3b
Update build_renewable_profiles
Add solar-hsat to Onshore technologies
2024-04-25 11:30:42 +02:00
Parisra
91de7a891d
Update rule: determine_availability_matrix
Add solar-hsat as an Onshore technology
2024-04-25 11:28:33 +02:00
martacki
feb564506a remove connection_costs as snakemake output from simplify_network 2024-04-16 15:41:01 +02:00
Fabian Neumann
991eef6d97 update and reduce databundle size 2024-04-15 14:48:34 +02:00
toniseibold
3b31b99dd2 removing the output file eia_hydro_stats.csv 2024-04-12 10:01:09 +02:00
Fabian Neumann
e459c3c89d add release notes; rename artificial to synthetic 2024-03-15 14:28:01 +01:00
Fabian Neumann
57b487f163 handle leap days in temporal clustering 2024-03-14 15:38:36 +01:00
Fabian Neumann
0d6ba9fbd1 move leap year handling into helper function 2024-03-14 15:15:56 +01:00
Fabian Neumann
f84707ba38 drop_leap_days already in renewable profiles 2024-03-14 14:30:50 +01:00
Fabian Neumann
cbeb9834ac update hydro capacity and generation data 2024-03-14 14:08:58 +01:00
pre-commit-ci[bot]
8aa6d98f56 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2024-03-04 18:38:31 +00:00
Fabian Neumann
8904943a87 integrate artificial load data supplement into build_electricity_demand 2024-03-04 19:37:52 +01:00
Fabian Neumann
9d913bcb38 smaller fixes and reduction of diff 2024-03-04 19:36:41 +01:00
Fabian Neumann
030e5b15ad remove rule build_cutout_year 2024-03-04 17:51:32 +01:00
Fabian Neumann
c2011bce49 remove {weather_year} wildcard 2024-03-04 17:48:56 +01:00
Koen van Greevenbroek
1ff8321c5e Remove erroneous automatic snakefmt formatting 2024-03-01 15:56:09 +01:00
Koen van Greevenbroek
61d565ec09 Correct formatting in build_electricity.smk 2024-03-01 15:26:16 +01:00
Koen van Greevenbroek
c2e7c125c7 Automatic formatting of snakemake rules 2024-03-01 13:24:02 +01:00
Koen van Greevenbroek
f6f98c102c Use default cutout for ship and nature raster extents
This avoids the need to either collect all cutouts for all weather
years or arbitrarily pick all cutouts for a certain weather year.
2024-03-01 11:45:39 +01:00
Koen van Greevenbroek
de0c0cd1f4 Merge remote-tracking branch 'upstream/master' into multiyear 2024-03-01 11:45:31 +01:00
Koen van Greevenbroek
90fb0b13da Various bugfixes to make multiyear branch run 2024-02-29 12:48:16 +01:00
Fabian Neumann
719d4dad23 Merge branch 'master' into scenario-management 2024-02-20 12:08:54 +01:00
Fabian Neumann
16e2286188 update copyright years to 2024 2024-02-19 16:21:48 +01:00
Fabian Neumann
d335275e0f build_electricity: add missing wildcard call 2024-02-17 20:43:11 +01:00
pre-commit-ci[bot]
cc3d7a4cbb [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2024-02-17 17:35:54 +00:00