The heat output from the carbon capture (CC) was being subtracted from
the CHP rather than the heat input.
Since the heat output and heat input are the same in the DEA
technology database (but at different temperatures), this bug has no
consequence, but still better to correct it.
Reasoning: we can also have fossil and biomass liquid hydrocarbons, as
well as production from the Fischer-Tropsch process, particularly for
simulations before 2050.
For the myopic method, based on the carbon budget indicated in the config file (sector_opts), a CO2 limit is calculated for every planning_horizon following an exponential or beta decay. A file with CO2 limit in every planning_horizon and a plot showing historical and planned CO2 emissions
are saved in the results
**This requires updating to UNFCCC_v23.csv in the data bundle**
This enables that the same function is used to read emissions in 2018, which are assumed to remain constant in 2019 and 2020 and subtracted from carbon budget (estimated from 2018 on).
I checked and 1990 emissions calculated with UNFCCC_v23 are very similar to those calculated with UNFCCC_v21 (<1% differences in all values at EU level).
Some countries show higher deviations, mainly in domestic aviation and navigation. I guess because those values started to be reported later and v23 should include more accurate values.
Running the rule solve_network in the university cluster, I was getting a "No space left on device" error. Making solve_dir=tmpdir by default avoids this error and makes it easier to identify any problem with temp files
This was breaking add_existing_baseyear.py and it is now fixed. Column name for commissioning year in powerplantmatching has changed. Now "DataIn" is used as column name, also when renewable capacities per country are added to the power plants dataframe
* Update prepare_network.py
The new ATK wildcard removes all lines + links without further distinction; however, since storage options are now modeled as store components, the links to and from the the storage units for (dis)charge are eliminated as well. Thus, the storage options drop out of the optimisation.
Especially when only allowing renewables as generation sources, optimisation may become infeasible for a high temporal resolution (capacity factors = 0 for certain hours; no further options to serve the load).
This issue does not arise with the ATKc wildcard, since bus0 and bus1 of the (dis)charge links share the same country code.
The proposed change is a very quick fix in the enforce_autarky function, solely removing DC links.
* Update scripts/prepare_network.py
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
* Update prepare_network.py
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
This is almost a direct copy PyPSA-Eur #167https://github.com/PyPSA/pypsa-eur/pull/167
A factor altering the maximum capacity (p_nom_max) can also be specified by e.g. solar+p3
One should be careful when using this for solar because the factor is applied to all the generators whose carrier includes the string 'solar' (i.e., it is applied to both utility and rooftop solar)
I would suggest implementing 'solar utility' and 'solar rooftop' as carriers, since this can be useful for other selecting processes. Is there is any reason for keeping 'solar' as a carrier for 'solar utility'?
The previous way of increasing maximum capacity via the config file (e.g 'solar3') is still present in the code.
For the myopic method, based on the carbon budget indicated in the config file (sector_opts), a CO2 limit is calculated for every planning_horizon following an exponential or beta decay. A file with CO2 limit in every planning_horizon and a plot showing historical and planned CO2 emissions
are saved in the results
**This requires updating to UNFCCC_v23.csv in the data bundle**
This enables that the same function is used to read emissions in 2018, which are assumed to remain constant in 2019 and 2020 and subtracted from carbon budget (estimated from 2018 on).
I checked and 1990 emissions calculated with UNFCCC_v23 are very similar to those calculated with UNFCCC_v21 (<1% differences in all values at EU level).
Some countries show higher deviations, mainly in domestic aviation and navigation. I guess because those values started to be reported later and v23 should include more accurate values.
Running the rule solve_network in the university cluster, I was getting a "No space left on device" error. Making solve_dir=tmpdir by default avoids this error and makes it easier to identify any problem with temp files
This was breaking add_existing_baseyear.py and it is now fixed. Column name for commissioning year in powerplantmatching has changed. Now "DataIn" is used as column name, also when renewable capacities per country are added to the power plants dataframe
Before it just had a fixed marginal cost. Now it uses DEA assumptions
for heat, electricity and capital costs.
This necessitates locating it somewhere concrete. Heat is taken from
urban central or decentral buses.
Use DEA assumptions for post-combustion carbon capture.
Also rename CCS as CC whenever only carbon capture is involved, since
sequestration (or CCU) is a separate step.
Only change was to remove the Store-Link-Bus combinations for
batteries and H2 storage from PyPSA-Eur, since they are implemented
with different names, costs and voltage level in PyPSA-Eur-Sec.
Removals are now done in a more transparent way in the config.yaml.
The assumptions for c_b and c_v and eta were arranged assuming
extraction plants (like the coal CHP in DEA).
However, if you look in DEA assumptions at "09b Wood Pellets Medium"
(used for solid biomass CHP) and "Gas turbine simple cycle (large)"
(used for gas CHP) they are not extraction plants but back pressure
plants.
The back pressure coefficient in DEA c_b is simply
c_b = name plate electricity efficiency / name plate heat efficiency
both measured when both heat and electricity are produced at maximum.
For the extraction plants, the efficiency was measured in condensation
mode, i.e. no heat production.
In almost 99.5% of cases the CHP dispatches along the backpressure
line where heat output is proportional to electricity output.
So we can switch to a single link to avoid the burden of modelling the
full electricity-heat feasibility space of CHPs.
This only applies to large CHPs in district heating networks.
* release_notes: order for release
* doc: fix smaller typos and tidy up
* config: bump version
* doc: fix line references
* doc: bump confpy version
* envs: update fixed versions yaml
* Snakefile: simplify all_elec to all
* fix clustering of offwind-ac and offwind-dc in sites option
* add release nodes
* attach renewable assets by location (lat and lon) from OPSD register to network
* adapt default config to changes
* undo changes from a differen PR in cluster_network.py
* undo changes from a different PR, add release notes for this PR
* correct release notes
* add comments for relevant settings in add_electricity.py
* adjust configtable for electricity to OPSD renewable option and add estimate_renewable_capacities_from_capacitiy_stats
* reset cluster_network to HEAD
* add_electricity: Capacity is float
* config: add GB to OPSD_VRE_countries
* review and modify implementation
* update release notes
* Update envs/environment.yaml
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
Co-authored-by: martha.frysztacki <eb5194@iai-esm003.iai.kit.edu>
Co-authored-by: eb5194 <martha.frysztacki@kit.edu>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
* add option to use custom clustermaps from data folder
* adapt default config to custom busmap
* input file from Snakefile
* adapt input description
* add option to use custom clustermaps from data folder
* adapt default config to custom busmap
* input file from Snakefile
* adapt input description
* Snakefile: custom_busmap in cluster_network input is now csv
* cluster_network: custom_busmap is now read as csv file, adaptions of description
* simplify_network: adapt descriptions
* configfiles: add cutom_clustermaps switch
* unify clustarmap and busmap names
* unify clustermap and busmap names
* test/config: unify clustermap and busmap names
* cluster_network: make clustering_for_n_clusters compatible with simplify_network
* simplify_network: make compatible with changes in cluster_network.py
* Update scripts/cluster_network.py
* Update scripts/simplify_network.py
* Update scripts/simplify_network.py
* Update scripts/cluster_network.py
* Update scripts/cluster_network.py
* cluster_network: move custom_busmap flag to enable; simplify names
* cluster_network: move custom_busmap flag to enable; simplify names
* custom_busmap: add documentation
* cluster_network: add default for custom_busmap for compatibility with old configs
* cluster_network: add default for custom_busmap for compatibility with old configs
Co-authored-by: martha.frysztacki <eb5194@iai-esm003.iai.kit.edu>
Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
Specify as dictionary, use get_parameter to get correct value.
Also remove old parameter "space_heating_fraction" since this is
superceded by the new exogenous retro code.
Strategy is too keep as much of configuration in config.yaml as
possible.
We also aim to allow exogenous investment-year-dependent
configurations to be done in a similar manner (e.g. share of district
heating or FCEV transport).
Since today's industrial electricity demand is distributed by
population and GDP, subtract this from the regular electricity demand
(which already has space/water heating subtracted).
Now regular electricity demand is only non-heating electricity demand
in residential and tertiary sectors.
Add back new industry electricity demand at the correct locations, as
determined using the hotmaps database.
I.e. per sector geographical distribution of industrial facilities
within each country.
Drop facilities outside Europe and with no geocoordinates.
Use ETS emissions as a distribution key; where emissions data is
missing, substitute with an average for that sector and that country
(strong assumption).
I.e. when the generators are clustered to the "simplified" network
resolution, but the grid is clustered further, e.g. by using the
clusters = 37m "m" option.
List classes in config.yaml, rather than integer selection in
build_biomass_potentials.py.
Also output potentials for all years and scenarios for analysis.
I.e. when the generators are clustered to the "simplified" network
resolution, but the grid is clustered further, e.g. by using the
clusters = 37m "m" option.
Don't fix uniform ratios e.g. of 0.3:0.7 primary:secondary for steel
and aluminium, but convert the necessary amount of existing primary in
each country so that the overall ratio applies at European level.
This stops sudden swings from primary to secondary in countries
dominated by primary production.
Remove non-existing biomass from chemicals and cement, since these
need higher temperatures than achievable with residues and waste.
Increase biomass in pulp and paper (since already used extensively
here and T < 500), and replace methane with biomass in food, beverages
and tobacco, since temperatures needed are low (T < 500).
This allows us to control the substitution of natural gas for hydrogen
in NH3 production.
Remaining basic chemicals are olefins, BTX and chlorine.
For 2015 NH3 production, we use the USGS data source.
This was handled before in industry_sector_ratios.csv which was
confusing.
Now industry_sector_ratios.csv represents the genuine energy
consumption per tonne of material for each industrial route
(MWh/tMaterial).
An new file is created with ktMaterial/a in
industrial_production_per_country_tomorrow.csv which contains changes
to the fraction of primary/secondary routes compared to today's
production in industrial_production_per_country.csv.
This is less confusing I think.
* fix emission prices
I'm not sure if the previous setup was intentional, but regarding that different generators might have different efficiencies and the emissions are carrier specific only, it makes more sense set net emission price.
* small fix
* update release_notes and config
Rather than taking a mean of the clustered connection costs.
Apply cost update also for overnight scenarios based on planning year.
Add land costs for onshore wind.
- add_brownfield.py: Have to make sure that for each CHP there is both
a heat and electric link, but they have different p_nom for each
CHP, so have to make sure we don't remove one without the other.
- solve_network.py: Make sure extra_functionality constraints for CHP
power-heat feasibility graph also work for non-extendable CHPs.
This simplifies the structure of add_brownfield.py dramatically.
Some other changes need to be make elsewhere because of name
changes (e.g. battery constraints in solve_network.py).
In order to calculate connection costs, average values for underground_fraction and average_distance are calculated for all the buses in the initial network mapped to the clustered network.
Previously they were distributed only by country to the first node in
the country.
Now conventional power plants are assigned to the correct node using
the bus map from PyPSA-Eur.
Wind and solar are distributed in each country by capacity factor.
The code has been refactored and a bug was fixed whereby total
capacities of wind and solar in each country were not correct.
Now the years in the config.yaml for myopic are integers not strings.
In prepare_costs, you need the min_count=1 in the sum so that it
generates NaNs for missing data (rather than 0) so that NaNs can be
subsituted by .fillna in the next line. Otherwise many values
(discount rates and efficiencies for solar, wind) are set to zero.
Also added carriers, storage and generators for coal, nuclear and
oil. (This needs to be organized better soon so that the carriers are
defined in config.yaml.)
Existing onwind and offwind capacity are now read from IRENA database, similarly to solar capacities. Previously we were using thewindpower.net database which is not open.
These geometries are apparently invalid due to self-crossing or
self-touching polygons.
The geometries are created by pypsa-eur/scripts/cluster_network.py but
appear to be valid before being written to file.
They are only valid after being read back in from file.
This seems to indicate some numerical issue relating to file reading
and writing.
Now the geometries are cleaned after being read in.
This commit now work with PyPSA-Eur 0.1.0 (tested with commit
bb3477 from 14th April 2020).
Changes to line/link_widths/colors for plotting networks in PyPSA
0.17.0.
Other corrections to plotting code so it works with this version.
Include oil boilers in colors in config.yaml.
* prepare_sector_network:
- add link for oil boiler in function add_industry()
- add function for partitioning clusters into different heat node types
* config.yaml: add option for oil_boiler
* costs.csv: add costs for oil boiler
* find all closest links that are in operation
* update function _find_closest_links
* update _set_electrical_parameters_links
* Update scripts/base_network.py
* cleanup code
Co-authored-by: FabianHofmann <hofmann@fias.uni-frankfurt.de>
* changes for retrofitting
* changed Snakefile to work with new pypsa-eur version, change solve_network.py to clip also n.generators_t.p_min_pu
* removed retrofitting data files
* environment: Free pyproj from version constraint (fixes#119)
proj was constrained to 1.9.6, since cartopy was incompatible with
recent proj versions.
* build_renewable_profiles: Import gdal and geokit after forking
GDAL sets up a shared context on module import which seems to contain a
handle to the PROJ database for coordinate reference systems. By forking
these handles seem to become invalidated or at least are not usable in parallel
anymore.
Instead importing gdal only after setting up the different processes fixes
the database disk image is malformed proj error.
Use s_nom_min for the reference point for the transmission
reinforcement not s_nom, since s_nom is now overwritten for LV > 1.0
(the old pyomo code kept the s_nom variable fixed in the pyomo model
rather than via pypsa setting s_nom_extendable = False, like the
nomopyomo code now does).
The legend scale of the H2 network electrolyzers is wrong. I don't yet
understand why. Possibly the cost scale in the other map is also wrong.
df.loc[idx[a,b,some_list],label] does NOT preserve the ordering of
some_list, but sorts it instead. Therefore the pattern:
df.loc[idx[a,b,s.index],label] = s.values
was mismatching the index and values.
Remove emissions from hydrogen production for ammonia (since H2 now
comes from electrolysis).
Allow process emissions from petrochemical production to be captured
(the carbon is not necessarily fossil, but could come from CCU).
* rewrite mocksnakemake for parsing real Snakefile
* continue add function to scripts
* going through all scripts, setting new mocksnakemake
* fix plotting scripts
* fix build_country_flh
* fix build_country_flh II
* adjust config files
* fix make_summary for tutorial network
* create dir also for output
* incorporate suggestions
* consistent import of mocksnakemake
* consistent import of mocksnakemake II
* Update scripts/_helpers.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/_helpers.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/_helpers.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/_helpers.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/plot_network.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/plot_network.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* Update scripts/retrieve_databundle.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* use pathlib for mocksnakemake
* rename mocksnakemake into mock_snakemake
* revert change in data
* Update scripts/_helpers.py
Co-Authored-By: euronion <42553970+euronion@users.noreply.github.com>
* remove setting logfile in mock_snakemake, use Path in configure_logging
* fix fallback path and base_dir
fix return type of make_io_accessable
* reformulate mock_snakemake
* incorporate suggestion, fix typos
* mock_snakemake: apply absolute paths again, add assertion error
*.py: make hard coded io path accessable for mock_snakemake
* retrieve_natura_raster: use snakemake.output for fn_out
* include suggestion
* Apply suggestions from code review
Co-Authored-By: Jonas Hörsch <jonas.hoersch@posteo.de>
* linting, add return ad end of file
* Update scripts/plot_p_nom_max.py
Co-Authored-By: Jonas Hörsch <jonas.hoersch@posteo.de>
* Update scripts/plot_p_nom_max.py
fixes#112
Co-Authored-By: Jonas Hörsch <jonas.hoersch@posteo.de>
* plot_p_nom_max: small correction
* config.tutorial.yaml fix snapshots end
* use techs instead of technology
* revert try out from previous commit, complete replacing
* change clusters -> clusts in plot_p_nom_max due to wildcard constraints of clusters
* change clusters -> clusts in plot_p_nom_max due to wildcard constraints of clusters II
With new pandas:
pd.Index([])|pd.MultiIndex(...)
returns a pd.Index, not a pd.MultiIndex, so just reversed:
pd.MultiIndex(...)|pd.Index([])
This returns a pd.MultiIndex.
* Add logging to logfiles to all snakemake workflow scripts.
* Fix missing quotation marks in Snakefile.
* Apply suggestions from code review
Co-Authored-By: Fabian Neumann <fabian.neumann@outlook.de>
* Apply suggestions from code review
Co-Authored-By: Fabian Neumann <fabian.neumann@outlook.de>
* doc: fix _ec_ filenames in docs
* Allow logging message format to be specified in config.yaml.
* Add logging for Snakemake rule 'retrieve_databundle '.
* Add limited logging to STDERR only for retrieve_*.py scripts.
* Import progressbar module only on demand.
* Fix logging to file and enable concurrent printing to STDERR for most scripts.
* Add new 'logging_format' option to Travis CI test config.yaml.
* Add missing parenthesis (bug fix) and cross-os compatible paths.
* Fix typos in messages.
* Use correct log files for logging (bug fix).
* doc: fix line references
* config: logging_format in all configs
* doc: add doc for logging_format
* environment: update to powerplantmatching 0.4.3
* doc: update line references for tutorial.rst
* Change logging configuration scheme for config.yaml.
* Add helper function for doing basic logging configuration.
* Add logpath for prepare_links_p_nom rule.
* Outsource basic logging configuration for all scripts to _helper submodule.
* Update documentation for changed config.yaml structure.
Instead of 'logging_level' and 'logging_format', now 'logging' with subcategories is used.
* _helpers: Change configure_logging signature.
The pandas Index union operation does no longer produce a MultiIndex but a Index tuple in pandas-0.25.2.
This causes the script to fail. (The problem does not exist with pandas-0.24.0.)
Fix by explicitly checking for this case (only the first loop iteration).
If the user selects a date range of less than a year in `config.yaml` the snapshot weightings should be scaled to add up to represet 8760 hours (1 year) such that operational expenses and investments are aligned in the objective function.
These are different for residential and services demand.
Also include Snakefile in config files copied for each run.
Use gurobi settings from gurobi support for speed.
Commented out settings for testing randomness for noise.
All urban central (i.e. district heating) is aggregated to the same
profile and bus.
The code is now written to cycle over each heating sector to add
demand and supply technologies, only changing what is necessary to
change, rather than just copying chunks of code and modifying
parameters there. This should make it easier to get an overview of
what's going on.
These are specified in data/heat_load_profile.csv.
The resulting heat_demand df has MultiIndex columns, where the first
level is ["sector use"], and the second level level is nodes.
Because there was insufficient solid biomass in 3-4 countries to
supply industry for it locally, and we need to account for transport
of solid biomass.
Should be replaced by transport cost links between countries.
Heat buses renamed to:
rural (for low-density areas where district heating not possible)
urban decentral (for high-density areas without district heating and
individual heating technologies) (used to be called "urban")
urban central (for high-density areas with district heating) (used to
be called "central")
District heating losses applied only to urban central.
* skip_iterating flag: solve network only once without updating impedances
* extra_functionality parameter: add function to modify pyomo model
* extra_functionality args: function arguments for extra_functionality
* extra_postprocessing: add function for postprocessing steps depending on n.model
Before both initial SOC and final SOC were set to be zero, which
prevents synthetic fuel transfer over the year boundary, and prevents
the use of fossil fuels for non-zero CO2 scenarios.
Now done properly with cyclic Store (prevents accumulation of fossil
fuels as a form of sequestration) and Generator (to imitate fossil
fuel extraction).
tech name must only appear, but not be identical to generator
carrier. This allows to use the name "offshore" for both "offshore-ac"
and "offshore-dc", but then "solar" also catches "solar thermal",
which is fine since for solar thermal potentials are np.inf, unless
limit is 0 since np.inf*0 is nan.
Remove non-renewable generator and storage units from electricity-only
base network, since they're added differently here with links.
Remove unncessary cruft from config.yaml which is not used by
PyPSA-Eur-Sec (e.g. renewable configuration parameters).
Rename "naptha" to correct "naphtha".
Split per-country capacity totals reported in entsoe SO&AF 2016 in proportion to
yearly generation potential at each bus, i.e. p_nom_max * mean(p_max_pu)
elec_s_37_lc1.25_... adds a constraint on the total line cost for an extension
by a 25%, compare with elec_s_37_lv1.25_... for the line volume limit.
`ll` is an acronym for line limit.
pandas has to be provoked to give NaNs for missing values in the sum,
so force min_count=1. Then these NaNs are filled with defaults.
OCGT and CCGT have fuel "gas".
- a network name like elec_s1000_ now clusters the network down to 1000 buses in
the simplify step.
- an 'm' after the number of clusters as in elec_s1000_181m_ skips the
aggregation of renewable generators and just moves them to the new clustered
bus in the second clustering step.
- to distribute the number of clusters to countries a small quadratic
optimization is now performed to minimize \sum_i (n_i - L_i/\sum_j L_j N)**2,
where n_i >= 1.
