Commit Graph

150 Commits

Author SHA1 Message Date
Fabian Neumann
bbff4297f9
temporarily disable n.shapes until memory issues resolved (#1238) 2024-08-26 15:16:06 +00:00
Bobby Xiong
0c36de9bf8
Introducing OpenStreetMap high-voltage grid to PyPSA-Eur (#1079)
* Implemented  which uses the overpass API to download power features for individual countries.

* Extended  rule by input.

* Bug fixes and improvements to clean_osm_data.py. Added  in retrieve_osm_data.py.

* Updated clean_osm_data and retrieve_osm_data to create clean substations.

* Finished clean_osm_data function.

* Added check whether line is a circle. If so, drop it.

* Extended build_electricity.smk by build_osm_network.py

* Added build_osm_network

* Working osm-network-fast

* Bug fixes.

* Finalised and cleaned  including docstrings.

* Added try catch to retrieve_osm_data. Allows for parallelisation of downloads.

* Updated cleaning process.

* Set maximum number of threads for retrieving to 4, wrt. fair usage policy and potential request errors.

* Intermediate update on clean_osm_data.py. Added docstrings.

* Bug fix.

* Bug fix.

* Bug fixes in data types out of clean_osm_data

* Significant improvements to retrieve_osm_data, clean_osm_data. Cleaned code. Speed improvements

* Cleaned config.

* Fixes.

* Bug fixes.

* Updated default config

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Removed overpass from required packages. Not needed anymore.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added links_relations (route = power, frequency = 0) to retrieval. This will change how HVDC links are extracted in the near future.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Work-in-progress clean_osm_data

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added clean links output to clean_osm_data. Script uses OSM relations to retrieve clean HVDC links.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* New code for integrating HVDC links. Using relations. Base network implementation functioning.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed manual line dropping.

* Updated clean script

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* reverted Snakefile to default: sync settings

* added prebuilt functionality.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated build_electricity.smk to work with scenario management.

* removed commented-out code.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed commented-out code.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fixed bug in pdf export by substituting pdf export with svg.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Bug-fix Snakefile

* dropped not needed columns from build_osm_network.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated build_shapes, config.default and clean_osm_data.

* pre-commit changes.

* test

* Added initial prepare_osm_network_release.py script

* Finalised prepare_osm_network_release script to build clean and stable OSM base_network input files.

* Added new rules/development.smk

* Updated clean_osm_data to add substation_centroid to linestrings

* Updated clean_osm_data to add substation_centroid to linestrings

* Updated clean_osm_data to add substation_centroid to linestrings

* Updated clean_osm_data to add substation_centroid to linestrings

* Added osm-prebuilt functionality and zenodo sandbox repository.

* Updated clean_osm_data to geopandas v.1.01

* Made base_network and build_osm_network function more robust for empty links.

* Made base_network and build_osm_network function more robust for empty links.

* Bug fix in base_network. Voltage level null is now kept (relevant e.g. for Corsica)

* Merge with hcanges in upstream PR 1146. Fixing UA and MD.

* Updated Zenodo and fixed prepare_osm_network_release

* Updated osm network release.

* Updated prepare osm network release.

* Updated MD, UA scripts.

* Cleaned determine_availability_matrix_MD_UA.py, removed redundant code

* Bug fixes.

* Bug fixes for UA MD scripts.

* Rename of build script.

* Bug fix: only distribute load to buses with substation.

* Updated zenodo sandbox repository.

* Updated config.default

* Cleaned config.default.yaml: Related settings grouped together and redundant voltage settings aggregated.

* Cleaned config.default.yaml: Related settings grouped together and redundant voltage settings aggregated. Added release notes.

* Updated Zenodo repositories for OSM-prebuilt to offcial publication.

* Updated configtables

* Updated links.csv: Under_construction lines to in commission.

* Updated link 8394 and parameter_corrections: Continuation of North-Sea-Link.

* Major update: fix simplify_network, fix Corsica, updated build_osm_network to include lines overpassing nodes.

* remove config backup

* Bug fix: Carrier type of all supernodes corrected to 'AC'

* Bug fix: Carrier type of all supernodes corrected to 'AC'

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated rules and base_network for compatibility with TYNDP projects.

* Updated Zenodo repository and prebuilt network to include 150 kV HVDC connections.

* Removed outdated config backup.

* Implemented all comments from PR #1079. Cleaned up OSM implementation.

* Bug fix: Added all voltages, 200 kV-750 kV, to default config.

* Cleaning and bugfixes.

* Updated Zenodo repository to https://zenodo.org/records/13358976. Added converter voltages, 'underground' property for DC lines/cables, and included Konti-Skan HVDC (DK-SE). Added compatibility with https://github.com/PyPSA/pypsa-eur/pull/1079 and https://github.com/PyPSA/pypsa-eur/pull/1085

* Apply suggestions from code review

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* simplify_network: handle complicated transformer topologies

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* syntax fix

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
2024-08-22 15:01:20 +02:00
Fabian
ca47becc87 build_transport_data: select swiss years according to data selection
use index.unique instead of index.levels where needed
2024-08-19 13:33:28 +02:00
Philipp Glaum
a9f67b313f
handle new and upgraded TYNDP&NEP lines/links in base network (OSM compatible) (#1085)
* add general implementation to add tyndp and nep. TODO: Remove duplicate?

* pre_osm_merge

* clean and update base_network.py
update default config settings

* clean and update base_network.py
update default config settings

* base_network.py:remove adding of transmission projects
add_transmission_project.py: add new script for creating lines and link csv from transmission projects
add_electricity.py: add new projects from created csv files

* cluster_network.py: do not allow to group lines with different build years together-> requires pypsa update
simplify_network.py: fix bug of simplify links

* remove legacies of removing transmission projects from base_network

* restructure folders:new folder for tranmission project csv
add_transmission_projects: improve logging, cleanup

* update lines csvs and add default line type for upgraded and new lines

* remove duplicate lines which were already in gridkit

* allow to connect ac lines only to ac buses

* add manual links csv (adjusted links_tyndp.csv) and update default config

* add realease note

* remove links_tyndp.csv file, references in build_elec.smk and function in base_network.py to add them

* add configuration options for transmission projects to documentation and add template folder for transmission projects

* update pypsa version in environments

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* integrate Fabian's review comments 2

* add electricity:add import pathlib and duplicate printing out of adding line projects

* update NEP line csv

* address Fabian's comments

* build_transmission_projects: address Fabian's final comments
simplify_network: use modus to get line type which appears most often

* build_transmission_project: change from .geometry to ["geometry"]

* build_transmission_projects: remove redundanty line

* build_transmission_projects: remove buffer for europe shape because of higher resolution
default config: fix wrong key for skip argument in transmission_projects

* update configtables and default config

* update manual links csv and delete undetected duplicate links in tyndp2020

* final adjustments

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
2024-08-15 11:42:21 +02:00
Fabian Neumann
ddf0da2687
base_network: bugfix - bring voronoi cells in correct order 2024-08-04 16:01:51 +02:00
Fabian Neumann
0470f119cf
base_network: use GeoSeries.voronoi_polygons instead of custom solution (#1172)
* base_network: use GeoSeries.voronoi_polygons instead of custom solution

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* base_network: move voronoi calculations into separate function

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* base_network: fix typo in function voronoi()

* base_network: refine voronoi function

* add release note [no ci]

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-07-24 15:31:46 +02:00
Fabian Neumann
c3151902f6
Compatibility with geopandas version 1 (#1136)
* address geopandas and pandas deprecations/warnings

* compatibility with geopandas v1

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-07-08 08:29:16 +02:00
Fabian Neumann
da10374158 apply ruff linting 2024-05-21 15:28:24 +02:00
Philipp Glaum
ac48fdaaf4 base_network.py: connect new tyndp links only to closest AC bus 2024-05-17 14:46:20 +02:00
Fabian Neumann
788e07a0d1 bugfix to avoid duplicated offshore regions (closes #396) 2024-05-13 17:56:28 +02:00
Koen van Greevenbroek
303b526bf1 Fix formatting of link in base_network documentation 2024-05-06 09:09:23 +02:00
Koen van Greevenbroek
ca6f803e28 Update documentation to reflect removal of build_bus_regions rule 2024-05-06 08:52:42 +02:00
Koen van Greevenbroek
76c201ef7d Integrate build_bus_regions into base_network
Fixes a problem with the `build_bus_regions` writing to base.nc
without declaring base.nc as an output.
2024-05-03 14:53:18 +02:00
cpschau
6ea8d52a81 no directory change before mock_snakemake 2024-04-16 10:43:07 +02:00
cpschau
3cf2d0c059 no addition of straight lines 2024-04-16 10:24:16 +02:00
cpschau
1867e510e2 remove redundant line 2024-04-11 17:14:17 +02:00
cpschau
c68e3420c0 clean-up after rebase 2024-04-11 16:42:19 +02:00
cpschau
f2db3c6327 add AC & DC lines 2024-04-11 16:32:35 +02:00
cpschau
4d4b8ea266 n.madd instead of pd.concat 2024-04-11 16:32:25 +02:00
Fabian
47134a88d8 regions to n.shapes: smooth out remaining issues 2024-04-10 18:35:50 +02:00
pre-commit-ci[bot]
7d0b775ca9 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2024-04-10 15:48:48 +00:00
cpschau
0ae016a7d0 write shapes to base network 2024-04-10 17:47:03 +02:00
Fabian Neumann
0d6ba9fbd1 move leap year handling into helper function 2024-03-14 15:15:56 +01:00
Koen van Greevenbroek
de0c0cd1f4 Merge remote-tracking branch 'upstream/master' into multiyear 2024-03-01 11:45:31 +01:00
Koen van Greevenbroek
90fb0b13da Various bugfixes to make multiyear branch run 2024-02-29 12:48:16 +01:00
Fabian Neumann
719d4dad23 Merge branch 'master' into scenario-management 2024-02-20 12:08:54 +01:00
Fabian Neumann
16e2286188 update copyright years to 2024 2024-02-19 16:21:48 +01:00
Fabian Neumann
945180f153 allow_missing in collection rules, fixes for myopic, wildcard constraints without function 2024-02-17 23:37:03 +01:00
Fabian Neumann
d2a66879de Merge branch 'master' into scenario-management 2024-02-17 17:04:36 +01:00
Martha Frysztacki
234f2f247c
Merge branch 'master' into bus-regions 2024-02-14 17:05:43 +01:00
Fabian Neumann
6de08bd523 Merge branch 'master' into scenario-management 2024-02-10 17:22:01 +01:00
Fabian Neumann
4b6199de42 create a bus for every unique coordinate, not only substations (closes #699) 2024-02-09 15:43:27 +01:00
Fabian Neumann
3691f9f4c1 update version compatibility handling 2024-02-09 13:59:15 +01:00
Fabian
46d8ce8f1f address deprecation warnings 2024-01-31 17:10:08 +01:00
Fabian Neumann
498f3bffad
Update scripts/base_network.py
Co-authored-by: Fabian Hofmann <fab.hof@gmx.de>
2024-01-18 18:22:21 +01:00
Philipp Glaum
8565c124c0 fix bus filter for buses without countries 2024-01-12 16:50:17 +01:00
Fabian Neumann
c3bb41c900 Merge branch 'master' into multiyear 2024-01-03 15:14:27 +01:00
Fabian Neumann
83ecaac4de
Merge branch 'master' into wildcard-opts-config 2024-01-03 13:08:21 +01:00
pre-commit-ci[bot]
ea3fdb6c15 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2023-12-18 08:58:15 +00:00
martacki
7e543ad0cc base_network: fix pandas 2.1 issue 2023-12-18 09:57:05 +01:00
pre-commit-ci[bot]
326aeb682b [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2023-10-09 14:28:08 +00:00
Fabian Neumann
ce609b6ebe
Merge branch 'master' into ukraine_hackathon 2023-10-09 16:25:49 +02:00
Fabian Neumann
a15157c788 add optional function to restore transmission connections to Crimea 2023-10-09 16:16:53 +02:00
Fabian Neumann
b94a797fed do not remove large unconnected subnetworks 2023-10-09 10:00:10 +02:00
Fabian Neumann
b8fee80919 Merge branch 'master' into multiyear 2023-10-08 11:55:11 +02:00
Sourcery AI
3c36c60bdd 'Refactored by Sourcery' 2023-10-08 09:20:36 +00:00
virio-andreyana
01cd1bbb74 add Fabians suggestion 2023-09-23 18:39:11 +02:00
pre-commit-ci[bot]
b2c19eda40 [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
2023-09-15 10:18:03 +00:00
virio-andreyana
a14e751ed8 fix snapshot bugs 2023-09-15 12:16:53 +02:00
Fabian
91eff472a7 scenario-management: fix set_scenario_config function; apply config_provider to some direct inputs 2023-08-17 12:31:07 +02:00