pypsa-eur/scripts
Bobby Xiong 0c36de9bf8
Introducing OpenStreetMap high-voltage grid to PyPSA-Eur (#1079)
* Implemented  which uses the overpass API to download power features for individual countries.

* Extended  rule by input.

* Bug fixes and improvements to clean_osm_data.py. Added  in retrieve_osm_data.py.

* Updated clean_osm_data and retrieve_osm_data to create clean substations.

* Finished clean_osm_data function.

* Added check whether line is a circle. If so, drop it.

* Extended build_electricity.smk by build_osm_network.py

* Added build_osm_network

* Working osm-network-fast

* Bug fixes.

* Finalised and cleaned  including docstrings.

* Added try catch to retrieve_osm_data. Allows for parallelisation of downloads.

* Updated cleaning process.

* Set maximum number of threads for retrieving to 4, wrt. fair usage policy and potential request errors.

* Intermediate update on clean_osm_data.py. Added docstrings.

* Bug fix.

* Bug fix.

* Bug fixes in data types out of clean_osm_data

* Significant improvements to retrieve_osm_data, clean_osm_data. Cleaned code. Speed improvements

* Cleaned config.

* Fixes.

* Bug fixes.

* Updated default config

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Removed overpass from required packages. Not needed anymore.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added links_relations (route = power, frequency = 0) to retrieval. This will change how HVDC links are extracted in the near future.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Work-in-progress clean_osm_data

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added clean links output to clean_osm_data. Script uses OSM relations to retrieve clean HVDC links.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* New code for integrating HVDC links. Using relations. Base network implementation functioning.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed manual line dropping.

* Updated clean script

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* reverted Snakefile to default: sync settings

* added prebuilt functionality.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated build_electricity.smk to work with scenario management.

* removed commented-out code.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed commented-out code.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fixed bug in pdf export by substituting pdf export with svg.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Bug-fix Snakefile

* dropped not needed columns from build_osm_network.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated build_shapes, config.default and clean_osm_data.

* pre-commit changes.

* test

* Added initial prepare_osm_network_release.py script

* Finalised prepare_osm_network_release script to build clean and stable OSM base_network input files.

* Added new rules/development.smk

* Updated clean_osm_data to add substation_centroid to linestrings

* Updated clean_osm_data to add substation_centroid to linestrings

* Updated clean_osm_data to add substation_centroid to linestrings

* Updated clean_osm_data to add substation_centroid to linestrings

* Added osm-prebuilt functionality and zenodo sandbox repository.

* Updated clean_osm_data to geopandas v.1.01

* Made base_network and build_osm_network function more robust for empty links.

* Made base_network and build_osm_network function more robust for empty links.

* Bug fix in base_network. Voltage level null is now kept (relevant e.g. for Corsica)

* Merge with hcanges in upstream PR 1146. Fixing UA and MD.

* Updated Zenodo and fixed prepare_osm_network_release

* Updated osm network release.

* Updated prepare osm network release.

* Updated MD, UA scripts.

* Cleaned determine_availability_matrix_MD_UA.py, removed redundant code

* Bug fixes.

* Bug fixes for UA MD scripts.

* Rename of build script.

* Bug fix: only distribute load to buses with substation.

* Updated zenodo sandbox repository.

* Updated config.default

* Cleaned config.default.yaml: Related settings grouped together and redundant voltage settings aggregated.

* Cleaned config.default.yaml: Related settings grouped together and redundant voltage settings aggregated. Added release notes.

* Updated Zenodo repositories for OSM-prebuilt to offcial publication.

* Updated configtables

* Updated links.csv: Under_construction lines to in commission.

* Updated link 8394 and parameter_corrections: Continuation of North-Sea-Link.

* Major update: fix simplify_network, fix Corsica, updated build_osm_network to include lines overpassing nodes.

* remove config backup

* Bug fix: Carrier type of all supernodes corrected to 'AC'

* Bug fix: Carrier type of all supernodes corrected to 'AC'

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated rules and base_network for compatibility with TYNDP projects.

* Updated Zenodo repository and prebuilt network to include 150 kV HVDC connections.

* Removed outdated config backup.

* Implemented all comments from PR #1079. Cleaned up OSM implementation.

* Bug fix: Added all voltages, 200 kV-750 kV, to default config.

* Cleaning and bugfixes.

* Updated Zenodo repository to https://zenodo.org/records/13358976. Added converter voltages, 'underground' property for DC lines/cables, and included Konti-Skan HVDC (DK-SE). Added compatibility with https://github.com/PyPSA/pypsa-eur/pull/1079 and https://github.com/PyPSA/pypsa-eur/pull/1085

* Apply suggestions from code review

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* simplify_network: handle complicated transformer topologies

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* syntax fix

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
2024-08-22 15:01:20 +02:00
..
build_cop_profiles [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-08-13 15:08:49 +00:00
definitions clean up, improve docs 2024-08-05 16:11:18 +02:00
__init__.py scenario-management: reenable shared resources, make shared resources wildcards dependent 2023-08-23 17:14:57 +02:00
_benchmark.py update copyright years to 2024 2024-02-19 16:21:48 +01:00
_helpers.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-08-13 15:08:49 +00:00
add_brownfield.py Lifetime of Gas Pipelines (#1162) 2024-07-29 09:47:08 +02:00
add_electricity.py handle new and upgraded TYNDP&NEP lines/links in base network (OSM compatible) (#1085) 2024-08-15 11:42:21 +02:00
add_existing_baseyear.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-08-08 14:46:09 +00:00
add_extra_components.py sanitize_locations: avoid attribute error for electricity-only runs (closes #939) 2024-05-13 17:59:21 +02:00
base_network.py Introducing OpenStreetMap high-voltage grid to PyPSA-Eur (#1079) 2024-08-22 15:01:20 +02:00
build_ammonia_production.py update ammonia demand data to 2022 2024-07-29 16:09:38 +02:00
build_biomass_potentials.py build_transport_data: select swiss years according to data selection 2024-08-19 13:33:28 +02:00
build_biomass_transport_costs.py modified environments to include tabula-py update and fixed PDF econding (#1219) 2024-08-15 14:52:33 +02:00
build_clustered_population_layouts.py remove remaining {weather_year} wildcard usages 2024-03-04 18:24:01 +01:00
build_cross_border_flows.py Merge branch 'master' into scenario-management 2024-02-20 12:08:54 +01:00
build_cutout.py cutouts: update zenodo repository version (#1176) 2024-07-24 11:01:00 +02:00
build_daily_heat_demand.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-06-17 09:44:05 +00:00
build_district_heat_share.py Fix negative district heating progress (#1168) 2024-07-19 12:30:26 +02:00
build_egs_potentials.py changed EGS paper link to doi-containing version 2024-06-01 18:21:49 +01:00
build_electricity_demand.py skip MD for synthetic load data 2024-06-04 13:56:54 +02:00
build_electricity_prices.py Merge branch 'master' into scenario-management 2024-02-20 12:08:54 +01:00
build_electricity_production.py Merge branch 'master' into scenario-management 2024-02-20 12:08:54 +01:00
build_energy_totals.py build_transport_data: select swiss years according to data selection 2024-08-19 13:33:28 +02:00
build_existing_heating_distribution.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-06-17 09:44:05 +00:00
build_gas_input_locations.py Compatibility with geopandas version 1 (#1136) 2024-07-08 08:29:16 +02:00
build_gas_network.py Compatibility with geopandas version 1 (#1136) 2024-07-08 08:29:16 +02:00
build_gdp_pop_non_nuts3.py determine_availability_matrix_MD_UA: enable parallelism & remove plots (#1170) 2024-07-19 17:43:11 +00:00
build_heat_totals.py correct typos in heating rule docs 2024-06-17 11:47:55 +02:00
build_hourly_heat_demand.py remove {scope} wildcard (#1171) 2024-07-24 13:19:57 +02:00
build_hydro_profile.py Update scripts/build_hydro_profile.py 2024-05-07 10:53:07 +02:00
build_industrial_distribution_key.py Compatibility with geopandas version 1 (#1136) 2024-07-08 08:29:16 +02:00
build_industrial_energy_demand_per_country_today.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-08-09 12:24:39 +00:00
build_industrial_energy_demand_per_node_today.py build_transport_data: select swiss years according to data selection 2024-08-19 13:33:28 +02:00
build_industrial_energy_demand_per_node.py documentation for industry rules (#1086) 2024-06-05 15:02:44 +02:00
build_industrial_production_per_country_tomorrow.py documentation for industry rules (#1086) 2024-06-05 15:02:44 +02:00
build_industrial_production_per_country.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-08-07 09:59:09 +00:00
build_industrial_production_per_node.py documentation for industry rules (#1086) 2024-06-05 15:02:44 +02:00
build_industry_sector_ratios_intermediate.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-08-20 08:16:00 +00:00
build_industry_sector_ratios.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-08-19 11:39:36 +00:00
build_line_rating.py ruff: safe fixes 2024-05-21 14:55:58 +02:00
build_monthly_prices.py Merge branch 'master' into scenario-management 2024-02-20 12:08:54 +01:00
build_osm_network.py Introducing OpenStreetMap high-voltage grid to PyPSA-Eur (#1079) 2024-08-22 15:01:20 +02:00
build_population_layouts.py Compatibility with geopandas version 1 (#1136) 2024-07-08 08:29:16 +02:00
build_population_weighted_energy_totals.py pop_weighted_energy_totals: feasible for sub-annual temporal scope 2024-03-14 18:52:59 +01:00
build_powerplants.py change FRESNA references to PyPSA 2024-08-02 11:42:30 +02:00
build_renewable_profiles.py Compatibility with geopandas version 1 (#1136) 2024-07-08 08:29:16 +02:00
build_retro_cost.py build_transport_data: select swiss years according to data selection 2024-08-19 13:33:28 +02:00
build_salt_cavern_potentials.py Merge branch 'master' into scenario-management 2024-02-20 12:08:54 +01:00
build_sequestration_potentials.py allow sequestration potential attributes to be string 2024-04-10 17:13:58 +02:00
build_shapes.py bugfix: make sure to include Ukraine offshore shapes with new EEZ files 2024-08-14 16:27:06 +02:00
build_ship_raster.py add missing atlite import 2024-04-15 15:28:02 +02:00
build_shipping_demand.py Compatibility with geopandas version 1 (#1136) 2024-07-08 08:29:16 +02:00
build_solar_thermal_profiles.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-06-17 09:44:05 +00:00
build_temperature_profiles.py remove {scope} wildcard (#1171) 2024-07-24 13:19:57 +02:00
build_transmission_projects.py handle new and upgraded TYNDP&NEP lines/links in base network (OSM compatible) (#1085) 2024-08-15 11:42:21 +02:00
build_transport_demand.py take eurostat year for transport demand 2024-07-23 15:10:05 +02:00
clean_osm_data.py Introducing OpenStreetMap high-voltage grid to PyPSA-Eur (#1079) 2024-08-22 15:01:20 +02:00
cluster_gas_network.py some small adjustments to run as single node model (#1183) 2024-07-31 11:53:20 +02:00
cluster_network.py handle new and upgraded TYNDP&NEP lines/links in base network (OSM compatible) (#1085) 2024-08-15 11:42:21 +02:00
determine_availability_matrix_MD_UA.py determine_availability_matrix_MD_UA: enable parallelism & remove plots (#1170) 2024-07-19 17:43:11 +00:00
make_summary_perfect.py build_transport_data: select swiss years according to data selection 2024-08-19 13:33:28 +02:00
make_summary.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-05-13 15:33:35 +00:00
plot_gas_network.py use set_scenario_config everywhere 2024-02-12 11:53:20 +01:00
plot_hydrogen_network.py Merge branch 'master' into scenario-management 2024-02-17 17:04:36 +01:00
plot_power_network_clustered.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-02-12 10:54:15 +00:00
plot_power_network_perfect.py use set_scenario_config everywhere 2024-02-12 11:53:20 +01:00
plot_power_network.py use set_scenario_config everywhere 2024-02-12 11:53:20 +01:00
plot_statistics.py Merge branch 'master' into scenario-management 2024-02-20 12:08:54 +01:00
plot_summary.py Fixed bug which generated empty pdfs when calling plot_summary from CI or snakemake. Using .svg exports instead. 2024-06-11 15:10:12 +02:00
plot_validation_cross_border_flows.py Merge branch 'master' into scenario-management 2024-02-20 12:08:54 +01:00
plot_validation_electricity_prices.py Merge branch 'master' into scenario-management 2024-02-20 12:08:54 +01:00
plot_validation_electricity_production.py address groupby(axis=...) deprecation (#1182) 2024-07-29 09:51:20 +00:00
prepare_network.py address fillna(method='{b|f}fill') deprecation (#1181) 2024-07-29 11:56:14 +02:00
prepare_osm_network_release.py Introducing OpenStreetMap high-voltage grid to PyPSA-Eur (#1079) 2024-08-22 15:01:20 +02:00
prepare_perfect_foresight.py Rename ev battery master (#1116) 2024-07-29 09:50:02 +02:00
prepare_sector_network.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-08-16 11:42:20 +00:00
retrieve_cost_data.py Fix typos 2024-03-19 08:54:58 +01:00
retrieve_databundle.py Compatibility of data processing for Ukraine (#1146) 2024-07-19 19:23:35 +02:00
retrieve_electricity_demand.py apply pre-commit 2024-05-21 15:30:56 +02:00
retrieve_eurostat_data.py drop-in mirror for broken eurostat energy balance link (#1147) 2024-07-12 09:34:49 +02:00
retrieve_eurostat_household_data.py tidy up update_residential_from_eurostat 2024-05-21 20:16:57 +02:00
retrieve_gas_infrastructure_data.py change in zenodo urls from record to records 2024-04-15 15:14:02 +02:00
retrieve_jrc_idees.py [pre-commit.ci] auto fixes from pre-commit.com hooks 2024-07-19 08:21:09 +00:00
retrieve_monthly_fuel_prices.py Merge branch 'master' into scenario-management 2024-02-20 12:08:54 +01:00
retrieve_osm_data.py Introducing OpenStreetMap high-voltage grid to PyPSA-Eur (#1079) 2024-08-22 15:01:20 +02:00
simplify_network.py Introducing OpenStreetMap high-voltage grid to PyPSA-Eur (#1079) 2024-08-22 15:01:20 +02:00
solve_network.py update release notes and doc 2024-08-20 15:06:02 +02:00
solve_operations_network.py remove {weather_year} wildcard 2024-03-04 17:48:56 +01:00
time_aggregation.py time_aggregation: handle case without any temporal aggregation 2024-05-25 15:48:48 +02:00