When the kmeans algorithm is used to cluster networks, this is not
deterministic by default. The result is that repeated runs of the
`simplify_network` and `cluster_network` rules can and usually do
produce different results that vary somewhat randomly. This makes
results less reproducible when given only a pypsa-eur configuration
file.
The fix is to supply a fixed random state to the k-means algorithm.
Assume you have 10 nodes that need to be distributed between 2 countries.
What can happen with some of the open source solvers is that one country
gets assigned to 9.01 (float) nodes, and the other one to 0.99.
Now using .astype(int) would lead to a node distribution of 0 and 9, as
the `astype(int)` function round down by default (0.99 -> 0). This assigned
zero value breaks the code in case open source solvers are used.
Gurobi somehow does deal with it.
* for now use n.snapshot_weightings.generators
* require pypsa master; use .objective for Nyears
* implement suggestions from code review
* add release note
* Propagate the solver log file name to the solver
Previously, the PyPSA network solving functions were not told about
the solver logfile specified in the Snakemake file.
* Pass solver_logfile on as kwargs
The `solve_network` function passes any additional arguments on to the
pypsa `network_lopf` and `ilopf` functions. Now we also pass
`solver_logfile` on as part of kwargs.
* Add focus_weights to pre-clustering
Hey guys,
another quick fix since I noticed it wasn't implemented yet: When pre-clustering the network, the focus_weights have not yet been considered. This may distort clustering results when pre-clustering to a low resolution.
* Update release_notes.rst
Hey guys,
please correct me if I'm wrong, but I think pricing both the charge and discharge of the battery store component with the inverter's capital costs results in duplicating costs since a bi-directional inverter is considered (I checked the source of the cost assumption, Budischak 2013).
Before working with PyPSA-EUR, I have worked with the toy model WHOBS where only the charging-link of the battery store component is priced with its capital costs (cf. https://github.com/PyPSA/WHOBS/blob/master/run_single_simulation.ipynb). The proposed change in code is equivalent.
* adjust buil_cutout.py and Snakefile
* try adjusting build_renewable_profiles, currently crashing due to weird pyproj error
* build_renewable_profiles: -remove printing gid
* build_renewable_profiles: use dask for paralellization, use dense functions
* build_renewable_profiles:
- revise imports
- add logging for long calculation
- revise explaining comment
- revise distance calculation
* build profiles: adjust to cutout.grid
* * fix area to square km
* rename potmatrix -> capacity_potential
* rename available to availibility
* config.default update cutout params
build_renewable_potentials: major refactoring and simplification
hydro_profiles: update code
* build profiles: fix weight output dimensions
* build profiles: fix typo, fix selection of buses
* build profiles: reinsert paths variable
* follow up
* build profiles: move to dask calculation only
* CI: set build cutout to true (add CDSAPI)
* build profiles: use pyproj, test with gleas and geokit upstream
* environment.yaml fix atlite version
* build profiles: use dask 'processes' for more than 25 regions
* build profiles: specify dask scheduler according to number of regions
* backpedal a bit, only allow scheduler='processes'
* follow up, code style and fixup
* build profiles: add logger info for underwater fraction calc
* config adjust cutout parameters
Snakefile fixup
* config.default.yaml: adjust resolution
* config: use one cutout in total
build_cutout: automatic detetection of geographical boundaries
* env: add python>=3.8 requirement
build_cutout: fixup for region bound
* config: allow base cutout
* folllow up, fix up
* follow up II
* clean up
* clean up II
* build profiles: move back to multiprocessing due to performance issues
* small code style corrections
* move in pool context
* swqitch to ratsterio
* switch to rasterio for availibility calculation
* tiny fixup
* * build continental raster for offshore distance calculation
* adjust Snakefile to new script build_raster
* rename continental raster to onshore raster
add projected_mask function (not yet tested)
add docstrings, modularize
* Snakefile: remove build_onhore_raster rule, build mask directly from geometry instead
build_natura_raster: adjust code, add function for exporting
build_profiles:
* add buffer to shore distance to init_globals function
* update docstrings
* improve handling of nodata grid codes
* add geometry mask if natura raster not activated
(the 255 value is an 'eligible' value for the corine data base,
do this for excluding data outside the shape)
* build_profiles: adjust docstrings
* update environment
* build profiles: fixup reproject woth padding
* follow up, small fixups
* fix resampling method
checkpoint: reproduces solar profile in tut data
* reintegrate plot map
code style
* config: rename cutout into "base"
* build profiles: adjust to new atlite code
* natura raster: small fixup
* build natura raster: compress tiff file
* config: adjust cutout names
* build profiles: cover case if no or partial overlap between natura raster and cutout
* config-tutorial: adjust cutout params
* buid-profifiles: fixup in gebco filter
* follow up
* update config files
* build profiles: select layoutmatrix != 0
* build profiles: speed up average_distance and underwaterfraction
* build profiles: fix typo
* update release notes
build_cutout: only build needed features
* update envs
* config: add temperature to sarah features
* temporary fix for atlite v0.2.1 and new xarray version release
* env: remove xarray specification
* * remove rule build_country_flh
* build profiles: remove sneaked in line
* doc: update configuration.rst (section atlite) and corresponding csv table
* release notes: fix quotes
* build profiles: use 3035 for area calculation
* Update envs/environment.docs.yaml
* Update scripts/build_cutout.py
* Update doc/release_notes.rst
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* Update doc/configuration.rst
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/build_cutout.py
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* update release notes
* release notes: add deprecation of 'keep_all_available_areas'
build profiles: remove warning for 'keep_all_available_areas'
* build cutout: rearrage code, set buffer correctly
* Rename tutorial cutout to remove name clash with real cutout.
* Update release_notes.rst: Rename tutorial cutout.
* retrieve: update cutouts and downloads (alternative) (#237)
* retrieve: update cutouts and downloads
* retrieve: remove unnecessary import
* use snakemake remote file functionality
* Snakefile: update zenodo link
* update natura remote link (closes#234)
* env: update atlite version to 0.2.2
* env: fix dask version due to memory issues
* test: retrieve cutout instead of build
* test: use tutorial cutout for CI
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
In some cases, in networks with DC links, a non-existing column would
be referenced in an empty Dataframe. This commit adds a check for this
case.
Co-authored-by: Koen van Greevenbroek <koen@vesoldo.com>
* Update prepare_network.py
The new ATK wildcard removes all lines + links without further distinction; however, since storage options are now modeled as store components, the links to and from the the storage units for (dis)charge are eliminated as well. Thus, the storage options drop out of the optimisation.
Especially when only allowing renewables as generation sources, optimisation may become infeasible for a high temporal resolution (capacity factors = 0 for certain hours; no further options to serve the load).
This issue does not arise with the ATKc wildcard, since bus0 and bus1 of the (dis)charge links share the same country code.
The proposed change is a very quick fix in the enforce_autarky function, solely removing DC links.
* Update scripts/prepare_network.py
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
* Update prepare_network.py
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
* release_notes: order for release
* doc: fix smaller typos and tidy up
* config: bump version
* doc: fix line references
* doc: bump confpy version
* envs: update fixed versions yaml
* Snakefile: simplify all_elec to all
* fix clustering of offwind-ac and offwind-dc in sites option
* add release nodes
* attach renewable assets by location (lat and lon) from OPSD register to network
* adapt default config to changes
* undo changes from a differen PR in cluster_network.py
* undo changes from a different PR, add release notes for this PR
* correct release notes
* add comments for relevant settings in add_electricity.py
* adjust configtable for electricity to OPSD renewable option and add estimate_renewable_capacities_from_capacitiy_stats
* reset cluster_network to HEAD
* add_electricity: Capacity is float
* config: add GB to OPSD_VRE_countries
* review and modify implementation
* update release notes
* Update envs/environment.yaml
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
Co-authored-by: martha.frysztacki <eb5194@iai-esm003.iai.kit.edu>
Co-authored-by: eb5194 <martha.frysztacki@kit.edu>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
* add option to use custom clustermaps from data folder
* adapt default config to custom busmap
* input file from Snakefile
* adapt input description
* add option to use custom clustermaps from data folder
* adapt default config to custom busmap
* input file from Snakefile
* adapt input description
* Snakefile: custom_busmap in cluster_network input is now csv
* cluster_network: custom_busmap is now read as csv file, adaptions of description
* simplify_network: adapt descriptions
* configfiles: add cutom_clustermaps switch
* unify clustarmap and busmap names
* unify clustermap and busmap names
* test/config: unify clustermap and busmap names
* cluster_network: make clustering_for_n_clusters compatible with simplify_network
* simplify_network: make compatible with changes in cluster_network.py
* Update scripts/cluster_network.py
* Update scripts/simplify_network.py
* Update scripts/simplify_network.py
* Update scripts/cluster_network.py
* Update scripts/cluster_network.py
* cluster_network: move custom_busmap flag to enable; simplify names
* cluster_network: move custom_busmap flag to enable; simplify names
* custom_busmap: add documentation
* cluster_network: add default for custom_busmap for compatibility with old configs
* cluster_network: add default for custom_busmap for compatibility with old configs
Co-authored-by: martha.frysztacki <eb5194@iai-esm003.iai.kit.edu>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
* fix emission prices
I'm not sure if the previous setup was intentional, but regarding that different generators might have different efficiencies and the emissions are carrier specific only, it makes more sense set net emission price.
* small fix
* update release_notes and config
* find all closest links that are in operation
* update function _find_closest_links
* update _set_electrical_parameters_links
* Update scripts/base_network.py
* cleanup code
Co-authored-by: FabianHofmann <hofmann@fias.uni-frankfurt.de>
* environment: Free pyproj from version constraint (fixes#119)
proj was constrained to 1.9.6, since cartopy was incompatible with
recent proj versions.
* build_renewable_profiles: Import gdal and geokit after forking
GDAL sets up a shared context on module import which seems to contain a
handle to the PROJ database for coordinate reference systems. By forking
these handles seem to become invalidated or at least are not usable in parallel
anymore.
Instead importing gdal only after setting up the different processes fixes
the database disk image is malformed proj error.
* rewrite mocksnakemake for parsing real Snakefile
* continue add function to scripts
* going through all scripts, setting new mocksnakemake
* fix plotting scripts
* fix build_country_flh
* fix build_country_flh II
* adjust config files
* fix make_summary for tutorial network
* create dir also for output
* incorporate suggestions
* consistent import of mocksnakemake
* consistent import of mocksnakemake II
* Update scripts/_helpers.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/_helpers.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/_helpers.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/_helpers.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/plot_network.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/plot_network.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/retrieve_databundle.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* use pathlib for mocksnakemake
* rename mocksnakemake into mock_snakemake
* revert change in data
* Update scripts/_helpers.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* remove setting logfile in mock_snakemake, use Path in configure_logging
* fix fallback path and base_dir
fix return type of make_io_accessable
* reformulate mock_snakemake
* incorporate suggestion, fix typos
* mock_snakemake: apply absolute paths again, add assertion error
*.py: make hard coded io path accessable for mock_snakemake
* retrieve_natura_raster: use snakemake.output for fn_out
* include suggestion
* Apply suggestions from code review
Co-Authored-By: Jonas Hörsch <jonas.hoersch@posteo.de>
* linting, add return ad end of file
* Update scripts/plot_p_nom_max.py
Co-Authored-By: Jonas Hörsch <jonas.hoersch@posteo.de>
* Update scripts/plot_p_nom_max.py
fixes#112
Co-Authored-By: Jonas Hörsch <jonas.hoersch@posteo.de>
* plot_p_nom_max: small correction
* config.tutorial.yaml fix snapshots end
* use techs instead of technology
* revert try out from previous commit, complete replacing
* change clusters -> clusts in plot_p_nom_max due to wildcard constraints of clusters
* change clusters -> clusts in plot_p_nom_max due to wildcard constraints of clusters II
