2022-09-16 13:04:04 +00:00
|
|
|
# -*- coding: utf-8 -*-
|
2024-02-19 15:21:48 +00:00
|
|
|
# SPDX-FileCopyrightText: : 2017-2024 The PyPSA-Eur Authors
|
2021-04-28 07:25:27 +00:00
|
|
|
#
|
2021-09-14 14:37:41 +00:00
|
|
|
# SPDX-License-Identifier: MIT
|
2019-08-08 13:02:28 +00:00
|
|
|
"""
|
2019-08-11 20:34:18 +00:00
|
|
|
Create cutouts with `atlite <https://atlite.readthedocs.io/en/latest/>`_.
|
2019-08-11 09:40:47 +00:00
|
|
|
|
2019-10-03 10:38:59 +00:00
|
|
|
For this rule to work you must have
|
|
|
|
|
|
|
|
- installed the `Copernicus Climate Data Store <https://cds.climate.copernicus.eu>`_ ``cdsapi`` package (`install with `pip``) and
|
|
|
|
- registered and setup your CDS API key as described `on their website <https://cds.climate.copernicus.eu/api-how-to>`_.
|
|
|
|
|
|
|
|
.. seealso::
|
|
|
|
For details on the weather data read the `atlite documentation <https://atlite.readthedocs.io/en/latest/>`_.
|
|
|
|
If you need help specifically for creating cutouts `the corresponding section in the atlite documentation <https://atlite.readthedocs.io/en/latest/examples/create_cutout.html>`_ should be helpful.
|
|
|
|
|
2019-08-11 09:40:47 +00:00
|
|
|
Relevant Settings
|
|
|
|
-----------------
|
|
|
|
|
2019-08-11 11:17:36 +00:00
|
|
|
.. code:: yaml
|
|
|
|
|
|
|
|
atlite:
|
|
|
|
nprocesses:
|
|
|
|
cutouts:
|
|
|
|
{cutout}:
|
|
|
|
|
2019-12-09 20:29:15 +00:00
|
|
|
.. seealso::
|
2023-04-21 08:41:44 +00:00
|
|
|
Documentation of the configuration file ``config/config.yaml`` at
|
2019-08-13 08:03:46 +00:00
|
|
|
:ref:`atlite_cf`
|
|
|
|
|
2019-08-11 09:40:47 +00:00
|
|
|
Inputs
|
|
|
|
------
|
|
|
|
|
2019-08-11 20:34:18 +00:00
|
|
|
*None*
|
|
|
|
|
2019-08-11 09:40:47 +00:00
|
|
|
Outputs
|
|
|
|
-------
|
|
|
|
|
2019-08-13 13:48:21 +00:00
|
|
|
- ``cutouts/{cutout}``: weather data from either the `ERA5 <https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5>`_
|
2024-08-30 16:06:26 +00:00
|
|
|
reanalysis weather dataset or `SARAH-3 <https://wui.cmsaf.eu/safira/action/viewProduktSearch>`_
|
2019-08-13 13:48:21 +00:00
|
|
|
satellite-based historic weather data with the following structure:
|
|
|
|
|
|
|
|
**ERA5 cutout:**
|
|
|
|
|
|
|
|
=================== ========== ========== =========================================================
|
|
|
|
Field Dimensions Unit Description
|
|
|
|
=================== ========== ========== =========================================================
|
|
|
|
pressure time, y, x Pa Surface pressure
|
|
|
|
------------------- ---------- ---------- ---------------------------------------------------------
|
|
|
|
temperature time, y, x K Air temperature 2 meters above the surface.
|
|
|
|
------------------- ---------- ---------- ---------------------------------------------------------
|
|
|
|
soil temperature time, y, x K Soil temperature between 1 meters and 3 meters
|
|
|
|
depth (layer 4).
|
|
|
|
------------------- ---------- ---------- ---------------------------------------------------------
|
|
|
|
influx_toa time, y, x Wm**-2 Top of Earth's atmosphere TOA incident solar radiation
|
|
|
|
------------------- ---------- ---------- ---------------------------------------------------------
|
|
|
|
influx_direct time, y, x Wm**-2 Total sky direct solar radiation at surface
|
|
|
|
------------------- ---------- ---------- ---------------------------------------------------------
|
|
|
|
runoff time, y, x m `Runoff <https://en.wikipedia.org/wiki/Surface_runoff>`_
|
|
|
|
(volume per area)
|
|
|
|
------------------- ---------- ---------- ---------------------------------------------------------
|
|
|
|
roughness y, x m Forecast surface roughness
|
|
|
|
(`roughness length <https://en.wikipedia.org/wiki/Roughness_length>`_)
|
|
|
|
------------------- ---------- ---------- ---------------------------------------------------------
|
|
|
|
height y, x m Surface elevation above sea level
|
|
|
|
------------------- ---------- ---------- ---------------------------------------------------------
|
|
|
|
albedo time, y, x -- `Albedo <https://en.wikipedia.org/wiki/Albedo>`_
|
|
|
|
measure of diffuse reflection of solar radiation.
|
|
|
|
Calculated from relation between surface solar radiation
|
|
|
|
downwards (Jm**-2) and surface net solar radiation
|
|
|
|
(Jm**-2). Takes values between 0 and 1.
|
|
|
|
------------------- ---------- ---------- ---------------------------------------------------------
|
|
|
|
influx_diffuse time, y, x Wm**-2 Diffuse solar radiation at surface.
|
|
|
|
Surface solar radiation downwards minus
|
|
|
|
direct solar radiation.
|
|
|
|
------------------- ---------- ---------- ---------------------------------------------------------
|
|
|
|
wnd100m time, y, x ms**-1 Wind speeds at 100 meters (regardless of direction)
|
|
|
|
=================== ========== ========== =========================================================
|
|
|
|
|
2023-03-09 12:28:42 +00:00
|
|
|
.. image:: img/era5.png
|
2019-12-09 20:29:15 +00:00
|
|
|
:scale: 40 %
|
|
|
|
|
2024-08-30 16:06:26 +00:00
|
|
|
A **SARAH-3 cutout** can be used to amend the fields ``temperature``, ``influx_toa``, ``influx_direct``, ``albedo``,
|
2019-08-13 13:48:21 +00:00
|
|
|
``influx_diffuse`` of ERA5 using satellite-based radiation observations.
|
2019-12-09 20:29:15 +00:00
|
|
|
|
2023-03-09 12:28:42 +00:00
|
|
|
.. image:: img/sarah.png
|
2019-08-13 13:48:21 +00:00
|
|
|
:scale: 40 %
|
|
|
|
|
2019-08-11 09:40:47 +00:00
|
|
|
Description
|
|
|
|
-----------
|
2019-08-08 13:02:28 +00:00
|
|
|
"""
|
2019-11-28 07:22:52 +00:00
|
|
|
|
2018-08-03 09:54:28 +00:00
|
|
|
import logging
|
2022-09-16 13:04:04 +00:00
|
|
|
|
Atlite availability (#224)
* adjust buil_cutout.py and Snakefile
* try adjusting build_renewable_profiles, currently crashing due to weird pyproj error
* build_renewable_profiles: -remove printing gid
* build_renewable_profiles: use dask for paralellization, use dense functions
* build_renewable_profiles:
- revise imports
- add logging for long calculation
- revise explaining comment
- revise distance calculation
* build profiles: adjust to cutout.grid
* * fix area to square km
* rename potmatrix -> capacity_potential
* rename available to availibility
* config.default update cutout params
build_renewable_potentials: major refactoring and simplification
hydro_profiles: update code
* build profiles: fix weight output dimensions
* build profiles: fix typo, fix selection of buses
* build profiles: reinsert paths variable
* follow up
* build profiles: move to dask calculation only
* CI: set build cutout to true (add CDSAPI)
* build profiles: use pyproj, test with gleas and geokit upstream
* environment.yaml fix atlite version
* build profiles: use dask 'processes' for more than 25 regions
* build profiles: specify dask scheduler according to number of regions
* backpedal a bit, only allow scheduler='processes'
* follow up, code style and fixup
* build profiles: add logger info for underwater fraction calc
* config adjust cutout parameters
Snakefile fixup
* config.default.yaml: adjust resolution
* config: use one cutout in total
build_cutout: automatic detetection of geographical boundaries
* env: add python>=3.8 requirement
build_cutout: fixup for region bound
* config: allow base cutout
* folllow up, fix up
* follow up II
* clean up
* clean up II
* build profiles: move back to multiprocessing due to performance issues
* small code style corrections
* move in pool context
* swqitch to ratsterio
* switch to rasterio for availibility calculation
* tiny fixup
* * build continental raster for offshore distance calculation
* adjust Snakefile to new script build_raster
* rename continental raster to onshore raster
add projected_mask function (not yet tested)
add docstrings, modularize
* Snakefile: remove build_onhore_raster rule, build mask directly from geometry instead
build_natura_raster: adjust code, add function for exporting
build_profiles:
* add buffer to shore distance to init_globals function
* update docstrings
* improve handling of nodata grid codes
* add geometry mask if natura raster not activated
(the 255 value is an 'eligible' value for the corine data base,
do this for excluding data outside the shape)
* build_profiles: adjust docstrings
* update environment
* build profiles: fixup reproject woth padding
* follow up, small fixups
* fix resampling method
checkpoint: reproduces solar profile in tut data
* reintegrate plot map
code style
* config: rename cutout into "base"
* build profiles: adjust to new atlite code
* natura raster: small fixup
* build natura raster: compress tiff file
* config: adjust cutout names
* build profiles: cover case if no or partial overlap between natura raster and cutout
* config-tutorial: adjust cutout params
* buid-profifiles: fixup in gebco filter
* follow up
* update config files
* build profiles: select layoutmatrix != 0
* build profiles: speed up average_distance and underwaterfraction
* build profiles: fix typo
* update release notes
build_cutout: only build needed features
* update envs
* config: add temperature to sarah features
* temporary fix for atlite v0.2.1 and new xarray version release
* env: remove xarray specification
* * remove rule build_country_flh
* build profiles: remove sneaked in line
* doc: update configuration.rst (section atlite) and corresponding csv table
* release notes: fix quotes
* build profiles: use 3035 for area calculation
* Update envs/environment.docs.yaml
* Update scripts/build_cutout.py
* Update doc/release_notes.rst
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* Update doc/configuration.rst
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/build_cutout.py
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* update release notes
* release notes: add deprecation of 'keep_all_available_areas'
build profiles: remove warning for 'keep_all_available_areas'
* build cutout: rearrage code, set buffer correctly
* Rename tutorial cutout to remove name clash with real cutout.
* Update release_notes.rst: Rename tutorial cutout.
* retrieve: update cutouts and downloads (alternative) (#237)
* retrieve: update cutouts and downloads
* retrieve: remove unnecessary import
* use snakemake remote file functionality
* Snakefile: update zenodo link
* update natura remote link (closes #234)
* env: update atlite version to 0.2.2
* env: fix dask version due to memory issues
* test: retrieve cutout instead of build
* test: use tutorial cutout for CI
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
2021-04-27 15:58:31 +00:00
|
|
|
import atlite
|
|
|
|
import geopandas as gpd
|
|
|
|
import pandas as pd
|
2023-08-15 13:02:41 +00:00
|
|
|
from _helpers import configure_logging, set_scenario_config
|
2019-11-28 07:22:52 +00:00
|
|
|
|
2020-12-03 18:50:53 +00:00
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
2019-08-08 13:02:28 +00:00
|
|
|
if __name__ == "__main__":
|
2019-12-09 20:29:15 +00:00
|
|
|
if "snakemake" not in globals():
|
|
|
|
from _helpers import mock_snakemake
|
2022-09-16 13:04:04 +00:00
|
|
|
|
2024-07-24 09:01:00 +00:00
|
|
|
snakemake = mock_snakemake("build_cutout", cutout="europe-2013-sarah3-era5")
|
2019-11-28 07:22:52 +00:00
|
|
|
configure_logging(snakemake)
|
2023-08-15 13:02:41 +00:00
|
|
|
set_scenario_config(snakemake)
|
2018-08-03 09:54:28 +00:00
|
|
|
|
2023-06-15 16:52:25 +00:00
|
|
|
cutout_params = snakemake.params.cutouts[snakemake.wildcards.cutout]
|
2020-12-03 18:50:53 +00:00
|
|
|
|
2023-06-15 16:52:25 +00:00
|
|
|
snapshots = pd.date_range(freq="h", **snakemake.params.snapshots)
|
Atlite availability (#224)
* adjust buil_cutout.py and Snakefile
* try adjusting build_renewable_profiles, currently crashing due to weird pyproj error
* build_renewable_profiles: -remove printing gid
* build_renewable_profiles: use dask for paralellization, use dense functions
* build_renewable_profiles:
- revise imports
- add logging for long calculation
- revise explaining comment
- revise distance calculation
* build profiles: adjust to cutout.grid
* * fix area to square km
* rename potmatrix -> capacity_potential
* rename available to availibility
* config.default update cutout params
build_renewable_potentials: major refactoring and simplification
hydro_profiles: update code
* build profiles: fix weight output dimensions
* build profiles: fix typo, fix selection of buses
* build profiles: reinsert paths variable
* follow up
* build profiles: move to dask calculation only
* CI: set build cutout to true (add CDSAPI)
* build profiles: use pyproj, test with gleas and geokit upstream
* environment.yaml fix atlite version
* build profiles: use dask 'processes' for more than 25 regions
* build profiles: specify dask scheduler according to number of regions
* backpedal a bit, only allow scheduler='processes'
* follow up, code style and fixup
* build profiles: add logger info for underwater fraction calc
* config adjust cutout parameters
Snakefile fixup
* config.default.yaml: adjust resolution
* config: use one cutout in total
build_cutout: automatic detetection of geographical boundaries
* env: add python>=3.8 requirement
build_cutout: fixup for region bound
* config: allow base cutout
* folllow up, fix up
* follow up II
* clean up
* clean up II
* build profiles: move back to multiprocessing due to performance issues
* small code style corrections
* move in pool context
* swqitch to ratsterio
* switch to rasterio for availibility calculation
* tiny fixup
* * build continental raster for offshore distance calculation
* adjust Snakefile to new script build_raster
* rename continental raster to onshore raster
add projected_mask function (not yet tested)
add docstrings, modularize
* Snakefile: remove build_onhore_raster rule, build mask directly from geometry instead
build_natura_raster: adjust code, add function for exporting
build_profiles:
* add buffer to shore distance to init_globals function
* update docstrings
* improve handling of nodata grid codes
* add geometry mask if natura raster not activated
(the 255 value is an 'eligible' value for the corine data base,
do this for excluding data outside the shape)
* build_profiles: adjust docstrings
* update environment
* build profiles: fixup reproject woth padding
* follow up, small fixups
* fix resampling method
checkpoint: reproduces solar profile in tut data
* reintegrate plot map
code style
* config: rename cutout into "base"
* build profiles: adjust to new atlite code
* natura raster: small fixup
* build natura raster: compress tiff file
* config: adjust cutout names
* build profiles: cover case if no or partial overlap between natura raster and cutout
* config-tutorial: adjust cutout params
* buid-profifiles: fixup in gebco filter
* follow up
* update config files
* build profiles: select layoutmatrix != 0
* build profiles: speed up average_distance and underwaterfraction
* build profiles: fix typo
* update release notes
build_cutout: only build needed features
* update envs
* config: add temperature to sarah features
* temporary fix for atlite v0.2.1 and new xarray version release
* env: remove xarray specification
* * remove rule build_country_flh
* build profiles: remove sneaked in line
* doc: update configuration.rst (section atlite) and corresponding csv table
* release notes: fix quotes
* build profiles: use 3035 for area calculation
* Update envs/environment.docs.yaml
* Update scripts/build_cutout.py
* Update doc/release_notes.rst
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* Update doc/configuration.rst
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/build_cutout.py
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* update release notes
* release notes: add deprecation of 'keep_all_available_areas'
build profiles: remove warning for 'keep_all_available_areas'
* build cutout: rearrage code, set buffer correctly
* Rename tutorial cutout to remove name clash with real cutout.
* Update release_notes.rst: Rename tutorial cutout.
* retrieve: update cutouts and downloads (alternative) (#237)
* retrieve: update cutouts and downloads
* retrieve: remove unnecessary import
* use snakemake remote file functionality
* Snakefile: update zenodo link
* update natura remote link (closes #234)
* env: update atlite version to 0.2.2
* env: fix dask version due to memory issues
* test: retrieve cutout instead of build
* test: use tutorial cutout for CI
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
2021-04-27 15:58:31 +00:00
|
|
|
time = [snapshots[0], snapshots[-1]]
|
|
|
|
cutout_params["time"] = slice(*cutout_params.get("time", time))
|
|
|
|
|
|
|
|
if {"x", "y", "bounds"}.isdisjoint(cutout_params):
|
|
|
|
# Determine the bounds from bus regions with a buffer of two grid cells
|
|
|
|
onshore = gpd.read_file(snakemake.input.regions_onshore)
|
|
|
|
offshore = gpd.read_file(snakemake.input.regions_offshore)
|
2022-06-23 12:25:30 +00:00
|
|
|
regions = pd.concat([onshore, offshore])
|
Atlite availability (#224)
* adjust buil_cutout.py and Snakefile
* try adjusting build_renewable_profiles, currently crashing due to weird pyproj error
* build_renewable_profiles: -remove printing gid
* build_renewable_profiles: use dask for paralellization, use dense functions
* build_renewable_profiles:
- revise imports
- add logging for long calculation
- revise explaining comment
- revise distance calculation
* build profiles: adjust to cutout.grid
* * fix area to square km
* rename potmatrix -> capacity_potential
* rename available to availibility
* config.default update cutout params
build_renewable_potentials: major refactoring and simplification
hydro_profiles: update code
* build profiles: fix weight output dimensions
* build profiles: fix typo, fix selection of buses
* build profiles: reinsert paths variable
* follow up
* build profiles: move to dask calculation only
* CI: set build cutout to true (add CDSAPI)
* build profiles: use pyproj, test with gleas and geokit upstream
* environment.yaml fix atlite version
* build profiles: use dask 'processes' for more than 25 regions
* build profiles: specify dask scheduler according to number of regions
* backpedal a bit, only allow scheduler='processes'
* follow up, code style and fixup
* build profiles: add logger info for underwater fraction calc
* config adjust cutout parameters
Snakefile fixup
* config.default.yaml: adjust resolution
* config: use one cutout in total
build_cutout: automatic detetection of geographical boundaries
* env: add python>=3.8 requirement
build_cutout: fixup for region bound
* config: allow base cutout
* folllow up, fix up
* follow up II
* clean up
* clean up II
* build profiles: move back to multiprocessing due to performance issues
* small code style corrections
* move in pool context
* swqitch to ratsterio
* switch to rasterio for availibility calculation
* tiny fixup
* * build continental raster for offshore distance calculation
* adjust Snakefile to new script build_raster
* rename continental raster to onshore raster
add projected_mask function (not yet tested)
add docstrings, modularize
* Snakefile: remove build_onhore_raster rule, build mask directly from geometry instead
build_natura_raster: adjust code, add function for exporting
build_profiles:
* add buffer to shore distance to init_globals function
* update docstrings
* improve handling of nodata grid codes
* add geometry mask if natura raster not activated
(the 255 value is an 'eligible' value for the corine data base,
do this for excluding data outside the shape)
* build_profiles: adjust docstrings
* update environment
* build profiles: fixup reproject woth padding
* follow up, small fixups
* fix resampling method
checkpoint: reproduces solar profile in tut data
* reintegrate plot map
code style
* config: rename cutout into "base"
* build profiles: adjust to new atlite code
* natura raster: small fixup
* build natura raster: compress tiff file
* config: adjust cutout names
* build profiles: cover case if no or partial overlap between natura raster and cutout
* config-tutorial: adjust cutout params
* buid-profifiles: fixup in gebco filter
* follow up
* update config files
* build profiles: select layoutmatrix != 0
* build profiles: speed up average_distance and underwaterfraction
* build profiles: fix typo
* update release notes
build_cutout: only build needed features
* update envs
* config: add temperature to sarah features
* temporary fix for atlite v0.2.1 and new xarray version release
* env: remove xarray specification
* * remove rule build_country_flh
* build profiles: remove sneaked in line
* doc: update configuration.rst (section atlite) and corresponding csv table
* release notes: fix quotes
* build profiles: use 3035 for area calculation
* Update envs/environment.docs.yaml
* Update scripts/build_cutout.py
* Update doc/release_notes.rst
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* Update doc/configuration.rst
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/build_cutout.py
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
* update release notes
* release notes: add deprecation of 'keep_all_available_areas'
build profiles: remove warning for 'keep_all_available_areas'
* build cutout: rearrage code, set buffer correctly
* Rename tutorial cutout to remove name clash with real cutout.
* Update release_notes.rst: Rename tutorial cutout.
* retrieve: update cutouts and downloads (alternative) (#237)
* retrieve: update cutouts and downloads
* retrieve: remove unnecessary import
* use snakemake remote file functionality
* Snakefile: update zenodo link
* update natura remote link (closes #234)
* env: update atlite version to 0.2.2
* env: fix dask version due to memory issues
* test: retrieve cutout instead of build
* test: use tutorial cutout for CI
Co-authored-by: euronion <42553970+euronion@users.noreply.github.com>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
2021-04-27 15:58:31 +00:00
|
|
|
d = max(cutout_params.get("dx", 0.25), cutout_params.get("dy", 0.25)) * 2
|
|
|
|
cutout_params["bounds"] = regions.total_bounds + [-d, -d, d, d]
|
|
|
|
elif {"x", "y"}.issubset(cutout_params):
|
|
|
|
cutout_params["x"] = slice(*cutout_params["x"])
|
|
|
|
cutout_params["y"] = slice(*cutout_params["y"])
|
|
|
|
|
|
|
|
logging.info(f"Preparing cutout with parameters {cutout_params}.")
|
|
|
|
features = cutout_params.pop("features", None)
|
|
|
|
cutout = atlite.Cutout(snakemake.output[0], **cutout_params)
|
|
|
|
cutout.prepare(features=features)
|