85c356297a
* Add logging to logfiles to all snakemake workflow scripts. * Fix missing quotation marks in Snakefile. * Apply suggestions from code review Co-Authored-By: Fabian Neumann <fabian.neumann@outlook.de> * Apply suggestions from code review Co-Authored-By: Fabian Neumann <fabian.neumann@outlook.de> * doc: fix _ec_ filenames in docs * Allow logging message format to be specified in config.yaml. * Add logging for Snakemake rule 'retrieve_databundle '. * Add limited logging to STDERR only for retrieve_*.py scripts. * Import progressbar module only on demand. * Fix logging to file and enable concurrent printing to STDERR for most scripts. * Add new 'logging_format' option to Travis CI test config.yaml. * Add missing parenthesis (bug fix) and cross-os compatible paths. * Fix typos in messages. * Use correct log files for logging (bug fix). * doc: fix line references * config: logging_format in all configs * doc: add doc for logging_format * environment: update to powerplantmatching 0.4.3 * doc: update line references for tutorial.rst * Change logging configuration scheme for config.yaml. * Add helper function for doing basic logging configuration. * Add logpath for prepare_links_p_nom rule. * Outsource basic logging configuration for all scripts to _helper submodule. * Update documentation for changed config.yaml structure. Instead of 'logging_level' and 'logging_format', now 'logging' with subcategories is used. * _helpers: Change configure_logging signature.
59 lines
2.0 KiB
Python
59 lines
2.0 KiB
Python
## Copyright 2019 Fabian Hofmann (FIAS)
|
|
"""
|
|
.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.3517935.svg
|
|
:target: https://doi.org/10.5281/zenodo.3517935
|
|
|
|
The data bundle (1.4 GB) contains common GIS datasets like NUTS3 shapes, EEZ shapes, CORINE Landcover, Natura 2000 and also electricity specific summary statistics like historic per country yearly totals of hydro generation, GDP and POP on NUTS3 levels and per-country load time-series.
|
|
|
|
This rule downloads the data bundle from `zenodo <https://doi.org/10.5281/zenodo.3517935>`_ and extracts it in the ``data`` sub-directory, such that all files of the bundle are stored in the ``data/bundle`` subdirectory.
|
|
|
|
The :ref:`tutorial` uses a smaller `data bundle <https://zenodo.org/record/3517921/files/pypsa-eur-tutorial-data-bundle.tar.xz>`_ than required for the full model (19 MB)
|
|
|
|
.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.3517921.svg
|
|
:target: https://doi.org/10.5281/zenodo.3517921
|
|
|
|
**Relevant Settings**
|
|
|
|
.. code:: yaml
|
|
|
|
tutorial:
|
|
|
|
.. seealso::
|
|
Documentation of the configuration file ``config.yaml`` at
|
|
:ref:`toplevel_cf`
|
|
|
|
**Outputs**
|
|
|
|
- ``cutouts/bundle``: input data collected from various sources
|
|
|
|
"""
|
|
|
|
import logging
|
|
logger = logging.getLogger(__name__)
|
|
from _helpers import progress_retrieve, configure_logging
|
|
|
|
from pathlib import Path
|
|
import tarfile
|
|
|
|
if __name__ == "__main__":
|
|
|
|
configure_logging(snakemake) # TODO Make logging compatible with progressbar (see PR #102)
|
|
|
|
if snakemake.config['tutorial']:
|
|
url = "https://zenodo.org/record/3517921/files/pypsa-eur-tutorial-data-bundle.tar.xz"
|
|
else:
|
|
url = "https://zenodo.org/record/3517935/files/pypsa-eur-data-bundle.tar.xz"
|
|
|
|
# Save locations
|
|
tarball_fn = Path("./bundle.tar.xz")
|
|
to_fn = Path("./data")
|
|
|
|
logger.info(f"Downloading databundle from '{url}'.")
|
|
progress_retrieve(url, tarball_fn)
|
|
|
|
logger.info(f"Extracting databundle.")
|
|
tarfile.open(tarball_fn).extractall(to_fn)
|
|
|
|
tarball_fn.unlink()
|
|
|
|
logger.info(f"Databundle available in '{to_fn}'.") |