pypsa-eur/scripts/prepare_osm_network_release.py
Bobby Xiong 0c36de9bf8
Introducing OpenStreetMap high-voltage grid to PyPSA-Eur (#1079)
* Implemented  which uses the overpass API to download power features for individual countries.

* Extended  rule by input.

* Bug fixes and improvements to clean_osm_data.py. Added  in retrieve_osm_data.py.

* Updated clean_osm_data and retrieve_osm_data to create clean substations.

* Finished clean_osm_data function.

* Added check whether line is a circle. If so, drop it.

* Extended build_electricity.smk by build_osm_network.py

* Added build_osm_network

* Working osm-network-fast

* Bug fixes.

* Finalised and cleaned  including docstrings.

* Added try catch to retrieve_osm_data. Allows for parallelisation of downloads.

* Updated cleaning process.

* Set maximum number of threads for retrieving to 4, wrt. fair usage policy and potential request errors.

* Intermediate update on clean_osm_data.py. Added docstrings.

* Bug fix.

* Bug fix.

* Bug fixes in data types out of clean_osm_data

* Significant improvements to retrieve_osm_data, clean_osm_data. Cleaned code. Speed improvements

* Cleaned config.

* Fixes.

* Bug fixes.

* Updated default config

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Removed overpass from required packages. Not needed anymore.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added links_relations (route = power, frequency = 0) to retrieval. This will change how HVDC links are extracted in the near future.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Work-in-progress clean_osm_data

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added clean links output to clean_osm_data. Script uses OSM relations to retrieve clean HVDC links.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* New code for integrating HVDC links. Using relations. Base network implementation functioning.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed manual line dropping.

* Updated clean script

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* reverted Snakefile to default: sync settings

* added prebuilt functionality.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated build_electricity.smk to work with scenario management.

* removed commented-out code.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* removed commented-out code.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fixed bug in pdf export by substituting pdf export with svg.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Bug-fix Snakefile

* dropped not needed columns from build_osm_network.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated build_shapes, config.default and clean_osm_data.

* pre-commit changes.

* test

* Added initial prepare_osm_network_release.py script

* Finalised prepare_osm_network_release script to build clean and stable OSM base_network input files.

* Added new rules/development.smk

* Updated clean_osm_data to add substation_centroid to linestrings

* Updated clean_osm_data to add substation_centroid to linestrings

* Updated clean_osm_data to add substation_centroid to linestrings

* Updated clean_osm_data to add substation_centroid to linestrings

* Added osm-prebuilt functionality and zenodo sandbox repository.

* Updated clean_osm_data to geopandas v.1.01

* Made base_network and build_osm_network function more robust for empty links.

* Made base_network and build_osm_network function more robust for empty links.

* Bug fix in base_network. Voltage level null is now kept (relevant e.g. for Corsica)

* Merge with hcanges in upstream PR 1146. Fixing UA and MD.

* Updated Zenodo and fixed prepare_osm_network_release

* Updated osm network release.

* Updated prepare osm network release.

* Updated MD, UA scripts.

* Cleaned determine_availability_matrix_MD_UA.py, removed redundant code

* Bug fixes.

* Bug fixes for UA MD scripts.

* Rename of build script.

* Bug fix: only distribute load to buses with substation.

* Updated zenodo sandbox repository.

* Updated config.default

* Cleaned config.default.yaml: Related settings grouped together and redundant voltage settings aggregated.

* Cleaned config.default.yaml: Related settings grouped together and redundant voltage settings aggregated. Added release notes.

* Updated Zenodo repositories for OSM-prebuilt to offcial publication.

* Updated configtables

* Updated links.csv: Under_construction lines to in commission.

* Updated link 8394 and parameter_corrections: Continuation of North-Sea-Link.

* Major update: fix simplify_network, fix Corsica, updated build_osm_network to include lines overpassing nodes.

* remove config backup

* Bug fix: Carrier type of all supernodes corrected to 'AC'

* Bug fix: Carrier type of all supernodes corrected to 'AC'

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated rules and base_network for compatibility with TYNDP projects.

* Updated Zenodo repository and prebuilt network to include 150 kV HVDC connections.

* Removed outdated config backup.

* Implemented all comments from PR #1079. Cleaned up OSM implementation.

* Bug fix: Added all voltages, 200 kV-750 kV, to default config.

* Cleaning and bugfixes.

* Updated Zenodo repository to https://zenodo.org/records/13358976. Added converter voltages, 'underground' property for DC lines/cables, and included Konti-Skan HVDC (DK-SE). Added compatibility with https://github.com/PyPSA/pypsa-eur/pull/1079 and https://github.com/PyPSA/pypsa-eur/pull/1085

* Apply suggestions from code review

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* simplify_network: handle complicated transformer topologies

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* syntax fix

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
2024-08-22 15:01:20 +02:00

147 lines
3.5 KiB
Python

# -*- coding: utf-8 -*-
# SPDX-FileCopyrightText: : 2020-2024 The PyPSA-Eur Authors
#
# SPDX-License-Identifier: MIT
import logging
import os
import pandas as pd
import pypsa
from _helpers import configure_logging, set_scenario_config
logger = logging.getLogger(__name__)
BUSES_COLUMNS = [
"bus_id",
"voltage",
"dc",
"symbol",
"under_construction",
"x",
"y",
"country",
"geometry",
]
LINES_COLUMNS = [
"line_id",
"bus0",
"bus1",
"voltage",
"circuits",
"length",
"underground",
"under_construction",
"geometry",
]
LINKS_COLUMNS = [
"link_id",
"bus0",
"bus1",
"voltage",
"p_nom",
"length",
"underground",
"under_construction",
"geometry",
]
TRANSFORMERS_COLUMNS = [
"transformer_id",
"bus0",
"bus1",
"voltage_bus0",
"voltage_bus1",
"geometry",
]
CONVERTERS_COLUMNS = [
"converter_id",
"bus0",
"bus1",
"voltage",
"geometry",
]
def export_clean_csv(df, columns, output_file):
"""
Export a cleaned DataFrame to a CSV file.
Args:
df (pandas.DataFrame): The DataFrame to be exported.
columns (list): A list of column names to include in the exported CSV file.
output_file (str): The path to the output CSV file.
Returns:
None
"""
rename_dict = {
"Bus": "bus_id",
"Line": "line_id",
"Link": "link_id",
"Transformer": "transformer_id",
"v_nom": "voltage",
"num_parallel": "circuits",
}
if "converter_id" in columns:
rename_dict["Link"] = "converter_id"
df.reset_index().rename(columns=rename_dict).loc[:, columns].replace(
{True: "t", False: "f"}
).to_csv(output_file, index=False, quotechar="'")
return None
if __name__ == "__main__":
if "snakemake" not in globals():
from _helpers import mock_snakemake
snakemake = mock_snakemake("prepare_osm_network_release")
configure_logging(snakemake)
set_scenario_config(snakemake)
network = pypsa.Network(snakemake.input.base_network)
network.buses["dc"] = network.buses.pop("carrier").map({"DC": "t", "AC": "f"})
network.lines.length = network.lines.length * 1e3
network.links.length = network.links.length * 1e3
# Export to clean csv for release
logger.info(f"Exporting {len(network.buses)} buses to %s", snakemake.output.buses)
export_clean_csv(network.buses, BUSES_COLUMNS, snakemake.output.buses)
logger.info(
f"Exporting {len(network.transformers)} transformers to %s",
snakemake.output.transformers,
)
export_clean_csv(
network.transformers, TRANSFORMERS_COLUMNS, snakemake.output.transformers
)
logger.info(f"Exporting {len(network.lines)} lines to %s", snakemake.output.lines)
export_clean_csv(network.lines, LINES_COLUMNS, snakemake.output.lines)
# Boolean that specifies if link element is a converter
is_converter = network.links.index.str.startswith("conv") == True
logger.info(
f"Exporting {len(network.links[~is_converter])} links to %s",
snakemake.output.links,
)
export_clean_csv(
network.links[~is_converter], LINKS_COLUMNS, snakemake.output.links
)
logger.info(
f"Exporting {len(network.links[is_converter])} converters to %s",
snakemake.output.converters,
)
export_clean_csv(
network.links[is_converter], CONVERTERS_COLUMNS, snakemake.output.converters
)
logger.info("Export of OSM network for release complete.")