When retrieving a remote file over HTTP, Snakemake uses the
"last-modified" property in HTTP header as a proxy for `mtime` of the
remote file. If this time is more recent than the `mtime` of the
output of the retrieve rule, the rule is triggered and the remote file
is retrieved again (since it was apparently updated).
However, Zenodo periodically updates the "last-modified" property of
records retrieved over HTTP even if those records have not been
updated. This causes Snakemake to false assume that the records have
to downloaded again.
By setting `static=True` for datasets we know don't actually change,
we avoid this problem.
* adjust buil_cutout.py and Snakefile
* try adjusting build_renewable_profiles, currently crashing due to weird pyproj error
* build_renewable_profiles: -remove printing gid
* build_renewable_profiles: use dask for paralellization, use dense functions
* build_renewable_profiles:
- revise imports
- add logging for long calculation
- revise explaining comment
- revise distance calculation
* build profiles: adjust to cutout.grid
* * fix area to square km
* rename potmatrix -> capacity_potential
* rename available to availibility
* config.default update cutout params
build_renewable_potentials: major refactoring and simplification
hydro_profiles: update code
* build profiles: fix weight output dimensions
* build profiles: fix typo, fix selection of buses
* build profiles: reinsert paths variable
* follow up
* build profiles: move to dask calculation only
* CI: set build cutout to true (add CDSAPI)
* build profiles: use pyproj, test with gleas and geokit upstream
* environment.yaml fix atlite version
* build profiles: use dask 'processes' for more than 25 regions
* build profiles: specify dask scheduler according to number of regions
* backpedal a bit, only allow scheduler='processes'
* follow up, code style and fixup
* build profiles: add logger info for underwater fraction calc
* config adjust cutout parameters
Snakefile fixup
* config.default.yaml: adjust resolution
* config: use one cutout in total
build_cutout: automatic detetection of geographical boundaries
* env: add python>=3.8 requirement
build_cutout: fixup for region bound
* config: allow base cutout
* folllow up, fix up
* follow up II
* clean up
* clean up II
* build profiles: move back to multiprocessing due to performance issues
* small code style corrections
* move in pool context
* swqitch to ratsterio
* switch to rasterio for availibility calculation
* tiny fixup
* * build continental raster for offshore distance calculation
* adjust Snakefile to new script build_raster
* rename continental raster to onshore raster
add projected_mask function (not yet tested)
add docstrings, modularize
* Snakefile: remove build_onhore_raster rule, build mask directly from geometry instead
build_natura_raster: adjust code, add function for exporting
build_profiles:
* add buffer to shore distance to init_globals function
* update docstrings
* improve handling of nodata grid codes
* add geometry mask if natura raster not activated
(the 255 value is an 'eligible' value for the corine data base,
do this for excluding data outside the shape)
* build_profiles: adjust docstrings
* update environment
* build profiles: fixup reproject woth padding
* follow up, small fixups
* fix resampling method
checkpoint: reproduces solar profile in tut data
* reintegrate plot map
code style
* config: rename cutout into "base"
* build profiles: adjust to new atlite code
* natura raster: small fixup
* build natura raster: compress tiff file
* config: adjust cutout names
* build profiles: cover case if no or partial overlap between natura raster and cutout
* config-tutorial: adjust cutout params
* buid-profifiles: fixup in gebco filter
* follow up
* update config files
* build profiles: select layoutmatrix != 0
* build profiles: speed up average_distance and underwaterfraction
* build profiles: fix typo
* update release notes
build_cutout: only build needed features
* update envs
* config: add temperature to sarah features
* temporary fix for atlite v0.2.1 and new xarray version release
* env: remove xarray specification
* * remove rule build_country_flh
* build profiles: remove sneaked in line
* doc: update configuration.rst (section atlite) and corresponding csv table
* release notes: fix quotes
* build profiles: use 3035 for area calculation
* Update envs/environment.docs.yaml
* Update scripts/build_cutout.py
* Update doc/release_notes.rst
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* Update doc/configuration.rst
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/build_cutout.py
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* update release notes
* release notes: add deprecation of 'keep_all_available_areas'
build profiles: remove warning for 'keep_all_available_areas'
* build cutout: rearrage code, set buffer correctly
* Rename tutorial cutout to remove name clash with real cutout.
* Update release_notes.rst: Rename tutorial cutout.
* retrieve: update cutouts and downloads (alternative) (#237)
* retrieve: update cutouts and downloads
* retrieve: remove unnecessary import
* use snakemake remote file functionality
* Snakefile: update zenodo link
* update natura remote link (closes#234)
* env: update atlite version to 0.2.2
* env: fix dask version due to memory issues
* test: retrieve cutout instead of build
* test: use tutorial cutout for CI
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
* release_notes: order for release
* doc: fix smaller typos and tidy up
* config: bump version
* doc: fix line references
* doc: bump confpy version
* envs: update fixed versions yaml
* Snakefile: simplify all_elec to all
* add option to use custom clustermaps from data folder
* adapt default config to custom busmap
* input file from Snakefile
* adapt input description
* add option to use custom clustermaps from data folder
* adapt default config to custom busmap
* input file from Snakefile
* adapt input description
* Snakefile: custom_busmap in cluster_network input is now csv
* cluster_network: custom_busmap is now read as csv file, adaptions of description
* simplify_network: adapt descriptions
* configfiles: add cutom_clustermaps switch
* unify clustarmap and busmap names
* unify clustermap and busmap names
* test/config: unify clustermap and busmap names
* cluster_network: make clustering_for_n_clusters compatible with simplify_network
* simplify_network: make compatible with changes in cluster_network.py
* Update scripts/cluster_network.py
* Update scripts/simplify_network.py
* Update scripts/simplify_network.py
* Update scripts/cluster_network.py
* Update scripts/cluster_network.py
* cluster_network: move custom_busmap flag to enable; simplify names
* cluster_network: move custom_busmap flag to enable; simplify names
* custom_busmap: add documentation
* cluster_network: add default for custom_busmap for compatibility with old configs
* cluster_network: add default for custom_busmap for compatibility with old configs
Co-authored-by: martha.frysztacki <eb5194@iai-esm003.iai.kit.edu>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
* Separate config switches for retrieving and building cutout, natura raster.
* Include new switches in tutorial and test configs.
* Update release_notes.rst
* Update toplevel.csv
* Update tutorial.rst
* Update configuration.rst
* rewrite mocksnakemake for parsing real Snakefile
* continue add function to scripts
* going through all scripts, setting new mocksnakemake
* fix plotting scripts
* fix build_country_flh
* fix build_country_flh II
* adjust config files
* fix make_summary for tutorial network
* create dir also for output
* incorporate suggestions
* consistent import of mocksnakemake
* consistent import of mocksnakemake II
* Update scripts/_helpers.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/_helpers.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/_helpers.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/_helpers.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/plot_network.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/plot_network.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/retrieve_databundle.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* use pathlib for mocksnakemake
* rename mocksnakemake into mock_snakemake
* revert change in data
* Update scripts/_helpers.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* remove setting logfile in mock_snakemake, use Path in configure_logging
* fix fallback path and base_dir
fix return type of make_io_accessable
* reformulate mock_snakemake
* incorporate suggestion, fix typos
* mock_snakemake: apply absolute paths again, add assertion error
*.py: make hard coded io path accessable for mock_snakemake
* retrieve_natura_raster: use snakemake.output for fn_out
* include suggestion
* Apply suggestions from code review
Co-Authored-By: Jonas Hörsch <jonas.hoersch@posteo.de>
* linting, add return ad end of file
* Update scripts/plot_p_nom_max.py
Co-Authored-By: Jonas Hörsch <jonas.hoersch@posteo.de>
* Update scripts/plot_p_nom_max.py
fixes#112
Co-Authored-By: Jonas Hörsch <jonas.hoersch@posteo.de>
* plot_p_nom_max: small correction
* config.tutorial.yaml fix snapshots end
* use techs instead of technology
* revert try out from previous commit, complete replacing
* change clusters -> clusts in plot_p_nom_max due to wildcard constraints of clusters
* change clusters -> clusts in plot_p_nom_max due to wildcard constraints of clusters II
* Add logging to logfiles to all snakemake workflow scripts.
* Fix missing quotation marks in Snakefile.
* Apply suggestions from code review
Co-Authored-By: Fabian Neumann <fabian.neumann@outlook.de>
* Apply suggestions from code review
Co-Authored-By: Fabian Neumann <fabian.neumann@outlook.de>
* doc: fix _ec_ filenames in docs
* Allow logging message format to be specified in config.yaml.
* Add logging for Snakemake rule 'retrieve_databundle '.
* Add limited logging to STDERR only for retrieve_*.py scripts.
* Import progressbar module only on demand.
* Fix logging to file and enable concurrent printing to STDERR for most scripts.
* Add new 'logging_format' option to Travis CI test config.yaml.
* Add missing parenthesis (bug fix) and cross-os compatible paths.
* Fix typos in messages.
* Use correct log files for logging (bug fix).
* doc: fix line references
* config: logging_format in all configs
* doc: add doc for logging_format
* environment: update to powerplantmatching 0.4.3
* doc: update line references for tutorial.rst
* Change logging configuration scheme for config.yaml.
* Add helper function for doing basic logging configuration.
* Add logpath for prepare_links_p_nom rule.
* Outsource basic logging configuration for all scripts to _helper submodule.
* Update documentation for changed config.yaml structure.
Instead of 'logging_level' and 'logging_format', now 'logging' with subcategories is used.
* _helpers: Change configure_logging signature.
