When retrieving a remote file over HTTP, Snakemake uses the
"last-modified" property in HTTP header as a proxy for `mtime` of the
remote file. If this time is more recent than the `mtime` of the
output of the retrieve rule, the rule is triggered and the remote file
is retrieved again (since it was apparently updated).
However, Zenodo periodically updates the "last-modified" property of
records retrieved over HTTP even if those records have not been
updated. This causes Snakemake to false assume that the records have
to downloaded again.
By setting `static=True` for datasets we know don't actually change,
we avoid this problem.
* adjust buil_cutout.py and Snakefile
* try adjusting build_renewable_profiles, currently crashing due to weird pyproj error
* build_renewable_profiles: -remove printing gid
* build_renewable_profiles: use dask for paralellization, use dense functions
* build_renewable_profiles:
- revise imports
- add logging for long calculation
- revise explaining comment
- revise distance calculation
* build profiles: adjust to cutout.grid
* * fix area to square km
* rename potmatrix -> capacity_potential
* rename available to availibility
* config.default update cutout params
build_renewable_potentials: major refactoring and simplification
hydro_profiles: update code
* build profiles: fix weight output dimensions
* build profiles: fix typo, fix selection of buses
* build profiles: reinsert paths variable
* follow up
* build profiles: move to dask calculation only
* CI: set build cutout to true (add CDSAPI)
* build profiles: use pyproj, test with gleas and geokit upstream
* environment.yaml fix atlite version
* build profiles: use dask 'processes' for more than 25 regions
* build profiles: specify dask scheduler according to number of regions
* backpedal a bit, only allow scheduler='processes'
* follow up, code style and fixup
* build profiles: add logger info for underwater fraction calc
* config adjust cutout parameters
Snakefile fixup
* config.default.yaml: adjust resolution
* config: use one cutout in total
build_cutout: automatic detetection of geographical boundaries
* env: add python>=3.8 requirement
build_cutout: fixup for region bound
* config: allow base cutout
* folllow up, fix up
* follow up II
* clean up
* clean up II
* build profiles: move back to multiprocessing due to performance issues
* small code style corrections
* move in pool context
* swqitch to ratsterio
* switch to rasterio for availibility calculation
* tiny fixup
* * build continental raster for offshore distance calculation
* adjust Snakefile to new script build_raster
* rename continental raster to onshore raster
add projected_mask function (not yet tested)
add docstrings, modularize
* Snakefile: remove build_onhore_raster rule, build mask directly from geometry instead
build_natura_raster: adjust code, add function for exporting
build_profiles:
* add buffer to shore distance to init_globals function
* update docstrings
* improve handling of nodata grid codes
* add geometry mask if natura raster not activated
(the 255 value is an 'eligible' value for the corine data base,
do this for excluding data outside the shape)
* build_profiles: adjust docstrings
* update environment
* build profiles: fixup reproject woth padding
* follow up, small fixups
* fix resampling method
checkpoint: reproduces solar profile in tut data
* reintegrate plot map
code style
* config: rename cutout into "base"
* build profiles: adjust to new atlite code
* natura raster: small fixup
* build natura raster: compress tiff file
* config: adjust cutout names
* build profiles: cover case if no or partial overlap between natura raster and cutout
* config-tutorial: adjust cutout params
* buid-profifiles: fixup in gebco filter
* follow up
* update config files
* build profiles: select layoutmatrix != 0
* build profiles: speed up average_distance and underwaterfraction
* build profiles: fix typo
* update release notes
build_cutout: only build needed features
* update envs
* config: add temperature to sarah features
* temporary fix for atlite v0.2.1 and new xarray version release
* env: remove xarray specification
* * remove rule build_country_flh
* build profiles: remove sneaked in line
* doc: update configuration.rst (section atlite) and corresponding csv table
* release notes: fix quotes
* build profiles: use 3035 for area calculation
* Update envs/environment.docs.yaml
* Update scripts/build_cutout.py
* Update doc/release_notes.rst
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* Update doc/configuration.rst
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/build_cutout.py
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* update release notes
* release notes: add deprecation of 'keep_all_available_areas'
build profiles: remove warning for 'keep_all_available_areas'
* build cutout: rearrage code, set buffer correctly
* Rename tutorial cutout to remove name clash with real cutout.
* Update release_notes.rst: Rename tutorial cutout.
* retrieve: update cutouts and downloads (alternative) (#237)
* retrieve: update cutouts and downloads
* retrieve: remove unnecessary import
* use snakemake remote file functionality
* Snakefile: update zenodo link
* update natura remote link (closes#234)
* env: update atlite version to 0.2.2
* env: fix dask version due to memory issues
* test: retrieve cutout instead of build
* test: use tutorial cutout for CI
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
* release_notes: order for release
* doc: fix smaller typos and tidy up
* config: bump version
* doc: fix line references
* doc: bump confpy version
* envs: update fixed versions yaml
* Snakefile: simplify all_elec to all
* add option to use custom clustermaps from data folder
* adapt default config to custom busmap
* input file from Snakefile
* adapt input description
* add option to use custom clustermaps from data folder
* adapt default config to custom busmap
* input file from Snakefile
* adapt input description
* Snakefile: custom_busmap in cluster_network input is now csv
* cluster_network: custom_busmap is now read as csv file, adaptions of description
* simplify_network: adapt descriptions
* configfiles: add cutom_clustermaps switch
* unify clustarmap and busmap names
* unify clustermap and busmap names
* test/config: unify clustermap and busmap names
* cluster_network: make clustering_for_n_clusters compatible with simplify_network
* simplify_network: make compatible with changes in cluster_network.py
* Update scripts/cluster_network.py
* Update scripts/simplify_network.py
* Update scripts/simplify_network.py
* Update scripts/cluster_network.py
* Update scripts/cluster_network.py
* cluster_network: move custom_busmap flag to enable; simplify names
* cluster_network: move custom_busmap flag to enable; simplify names
* custom_busmap: add documentation
* cluster_network: add default for custom_busmap for compatibility with old configs
* cluster_network: add default for custom_busmap for compatibility with old configs
Co-authored-by: martha.frysztacki <eb5194@iai-esm003.iai.kit.edu>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
