Merge branch 'master' into add_methanol_techs

data/biomass_transport_costs_supplychain1.csv

@@ -0,0 +1,41 @@
+country,Loading,Loading,Loading,Transport to plant,Transport to plant,Unloading,TOTAL,TOTAL,TOTAL
+country,Time,Machinery costs,Waiting time truck driver,60 average speed km/h,Trailer,Waiting time truck driver,Fix,per km,per km/ton
+country,h,EUR,EUR.1,EUR/km,EUR.2,EUR.3,EUR.4,EUR/km.1,EUR/km/ton
+BE,0.08,3.56,1.31,1.45,6.42,1.31,12.61,0.02,0.47
+BG,0.08,2.06,0.56,1.06,2.61,0.56,5.79,0.02,0.22
+CZ,0.08,2.34,0.62,1.1,4.08,0.62,7.65,0.02,0.29
+DK,0.08,5.14,2.54,1.85,7.91,2.54,18.13,0.03,0.68
+DE,0.08,3.41,1.31,1.37,5.99,1.31,12.01,0.02,0.45
+EE,0.08,2.26,0.5,1.1,4.14,0.5,7.38,0.02,0.28
+IE,0.08,3.29,1.03,1.4,6.29,1.03,11.64,0.02,0.44
+EL,0.08,2.83,0.85,1.25,5.17,0.85,9.7,0.02,0.37
+ES,0.08,3.19,1.22,1.32,5.28,1.22,10.91,0.02,0.41
+FR,0.08,3.52,1.35,1.38,6.49,1.35,12.71,0.02,0.48
+IT,0.08,3.59,1.26,1.56,5.82,1.26,11.94,0.03,0.45
+CY,0.08,2.82,0.85,1.25,5.09,0.85,9.6,0.02,0.36
+LV,0.08,2.11,0.43,1.05,3.87,0.43,6.84,0.02,0.26
+LT,0.08,1.99,0.39,1.02,3.49,0.39,6.25,0.02,0.24
+LU,0.08,3.38,1.21,1.31,6.97,1.21,12.78,0.02,0.48
+HU,0.08,2.12,0.42,1.12,3.33,0.42,6.28,0.02,0.24
+MT,0.08,2.44,0.62,1.16,4.32,0.62,7.99,0.02,0.3
+NL,0.08,3.83,1.57,1.51,6.37,1.57,13.33,0.03,0.5
+AT,0.08,3.33,1.18,1.35,6.37,1.18,12.06,0.02,0.45
+PL,0.08,2.27,0.68,1.08,3.35,0.68,6.98,0.02,0.27
+PT,0.08,2.3,0.37,1.16,4.67,0.37,7.71,0.02,0.29
+RO,0.08,1.88,0.4,0.99,2.8,0.4,5.49,0.02,0.21
+SI,0.08,2.47,0.58,1.18,4.65,0.58,8.29,0.02,0.32
+SK,0.08,2.24,0.48,1.12,3.93,0.48,7.13,0.02,0.27
+FI,0.08,3.59,1.2,1.48,7.0,1.2,12.98,0.02,0.49
+SE,0.08,3.84,1.26,1.57,7.71,1.26,14.07,0.03,0.53
+UK,0.08,3.65,1.21,1.56,6.56,1.21,12.64,0.03,0.48
+HR,0.08,2.12,0.46,1.05,3.79,0.46,6.84,0.02,0.26
+AL,0.08,1.67,0.19,0.96,2.55,0.19,4.61,0.02,0.18
+BA,0.08,1.66,0.23,0.91,2.81,0.23,4.92,0.02,0.19
+MK,0.08,1.5,0.22,0.84,2.32,0.22,4.27,0.01,0.17
+ME,0.08,1.71,0.24,0.94,2.83,0.24,5.02,0.02,0.2
+RS,0.08,1.69,0.22,0.96,2.64,0.22,4.77,0.02,0.19
+XK,0.08,1.62,0.21,0.89,2.81,0.21,4.85,0.01,0.19
+UA,0.08,1.25,0.13,0.65,2.8,0.13,4.32,0.01,0.16
+TR,0.08,2.35,0.52,1.24,3.44,0.52,6.82,0.02,0.26
+MD,0.08,1.33,0.09,0.74,2.8,0.09,4.31,0.01,0.17
+CH,0.08,3.79,1.18,1.47,8.9,1.18,15.06,0.02,0.56

data/biomass_transport_costs_supplychain2.csv

@@ -0,0 +1,41 @@
+country,Loading,Loading,Loading,Transport to plant,Transport to plant,Unloading,TOTAL,TOTAL,TOTAL
+country,Time,Machinery costs,Waiting time truck driver,60 average speed km/h,Trailer,Waiting time truck driver,Fix,per km,per km/ton
+country,h,EUR,EUR.1,EUR/km,EUR.2,EUR.3,EUR.4,EUR/km.1,EUR/km/ton
+BE,0.05,3.06,0.88,1.49,16.57,0.88,21.39,0.02,0.88
+BG,0.05,1.77,0.37,1.07,6.74,0.37,9.26,0.02,0.39
+CZ,0.05,2.05,0.41,1.13,10.52,0.41,13.4,0.02,0.55
+DK,0.05,4.24,1.7,1.89,20.4,1.7,28.04,0.03,1.15
+DE,0.05,2.91,0.87,1.41,15.45,0.87,20.11,0.02,0.83
+EE,0.05,2.01,0.33,1.13,10.67,0.33,13.34,0.02,0.55
+IE,0.05,2.87,0.69,1.44,16.23,0.69,20.48,0.02,0.84
+EL,0.05,2.47,0.57,1.28,13.34,0.57,16.95,0.02,0.7
+ES,0.05,2.71,0.82,1.35,13.63,0.82,17.98,0.02,0.74
+FR,0.05,3.02,0.91,1.42,16.74,0.91,21.56,0.02,0.89
+IT,0.05,3.07,0.85,1.59,15.0,0.85,19.77,0.03,0.82
+CY,0.05,2.46,0.57,1.28,13.12,0.57,16.71,0.02,0.69
+LV,0.05,1.89,0.29,1.08,9.98,0.29,12.44,0.02,0.52
+LT,0.05,1.78,0.26,1.04,9.01,0.26,11.3,0.02,0.47
+LU,0.05,2.94,0.81,1.35,17.98,0.81,22.54,0.02,0.92
+HU,0.05,1.88,0.28,1.14,8.59,0.28,11.03,0.02,0.46
+MT,0.05,2.15,0.41,1.19,11.15,0.41,14.12,0.02,0.58
+NL,0.05,3.23,1.05,1.55,16.42,1.05,21.76,0.03,0.9
+AT,0.05,2.88,0.79,1.38,16.42,0.79,20.88,0.02,0.86
+PL,0.05,1.96,0.46,1.1,8.64,0.46,11.51,0.02,0.48
+PT,0.05,2.09,0.25,1.18,12.04,0.25,14.63,0.02,0.6
+RO,0.05,1.66,0.27,1.01,7.23,0.27,9.43,0.02,0.39
+SI,0.05,2.2,0.39,1.2,12.0,0.39,14.98,0.02,0.62
+SK,0.05,1.99,0.32,1.15,10.13,0.32,12.77,0.02,0.53
+FI,0.05,3.12,0.8,1.52,18.05,0.8,22.78,0.03,0.94
+SE,0.05,3.35,0.85,1.62,19.89,0.85,24.93,0.03,1.02
+UK,0.05,3.16,0.81,1.6,16.93,0.81,21.71,0.03,0.9
+HR,0.05,1.89,0.31,1.07,9.77,0.31,12.28,0.02,0.51
+AL,0.05,1.5,0.13,0.98,6.59,0.13,8.35,0.02,0.35
+BA,0.05,1.5,0.15,0.93,7.25,0.15,9.05,0.02,0.38
+MK,0.05,1.35,0.15,0.85,5.98,0.15,7.62,0.01,0.32
+ME,0.05,1.54,0.16,0.96,7.31,0.16,9.17,0.02,0.38
+RS,0.05,1.52,0.15,0.97,6.8,0.15,8.62,0.02,0.36
+XK,0.05,1.46,0.14,0.91,7.25,0.14,8.99,0.02,0.37
+UA,0.05,1.16,0.09,0.66,7.23,0.09,8.57,0.01,0.35
+TR,0.05,2.07,0.35,1.26,8.88,0.35,11.63,0.02,0.49
+MD,0.05,1.24,0.06,0.75,7.23,0.06,8.59,0.01,0.36
+CH,0.05,3.37,0.79,1.52,22.97,0.79,27.92,0.03,1.14

doc/release_notes.rst

@@ -10,15 +10,23 @@ Release Notes
 
 .. Upcoming Release
 
+
 * Add technology options for methanol, like electricity production from methanol, biomass to methanol, methanol to kerosene, ...
 
+* Change the heating demand from final energy which includes losses in legacy equipment to thermal energy service based on JRC-IDEES. Efficiencies of existing heating capacities are lowered according to the conversion of final energy to thermal energy service. For overnight scenarios or future planning horizon this change leads to a reduction in heat supply.
+
 * Updated district heating supply temperatures based on `Euroheat's DHC Market Outlook 2024<https://api.euroheat.org/uploads/Market_Outlook_2024_beeecd62d4.pdf>`__ and `AGFW-Hauptbericht 2022 <https://www.agfw.de/securedl/sdl-eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE3MjU2MjI2MTUsImV4cCI6MTcyNTcxMjYxNSwidXNlciI6MCwiZ3JvdXBzIjpbMCwtMV0sImZpbGUiOiJmaWxlYWRtaW4vdXNlcl91cGxvYWQvWmFobGVuX3VuZF9TdGF0aXN0aWtlbi9IYXVwdGJlcmljaHRfMjAyMi9BR0ZXX0hhdXB0YmVyaWNodF8yMDIyLnBkZiIsInBhZ2UiOjQzNn0.Bhma3PKg9uJnC57Ixi2p9STW5-II9VXPTDXS544M208/AGFW_Hauptbericht_2022.pdf>`__. `min_forward_temperature` and `return_temperature` (not given by Euroheat) are extrapolated based on German values.
+
 * Made the overdimensioning factor for heating systems specific for central/decentral heating, defaults to no overdimensionining for central heating and no changes to decentral heating compared to previous version.
 
 * bugfix: The carrier of stores was silently overwritten by their bus_carrier as a side effect when building the co2 constraints
 
 * bugfix: The oil generator was incorrectly dropped when the config `oil_refining_emissions` was greater than zero. This was the default behaviour in 0.12.0.
 
+* Uses of Snakemake's ``storage()`` function are integrated into retrieval
+  rules. This simplifies the use of ``mock_snakemake`` and places downloaded
+  data more transparently into the ``data`` directory.
+
 PyPSA-Eur 0.12.0 (30th August 2024)
 ===================================
 

envs/environment.yaml

@@ -24,7 +24,7 @@ dependencies:
 - yaml
 - pytables
 - lxml
-- powerplantmatching>=0.5.15
+- powerplantmatching>=0.5.15,<0.6
 - numpy
 - pandas>=2.1
 - geopandas>=1
@@ -64,4 +64,3 @@ dependencies:
   - snakemake-executor-plugin-slurm
   - snakemake-executor-plugin-cluster-generic
   - highspy
-  - tabula-py

rules/build_sector.smk

@@ -366,6 +366,7 @@ rule build_energy_totals:
         co2_name=resources("co2_totals.csv"),
         transport_name=resources("transport_data.csv"),
         district_heat_share=resources("district_heat_share.csv"),
+        heating_efficiencies=resources("heating_efficiencies.csv"),
     threads: 16
     resources:
         mem_mb=10000,
@@ -402,10 +403,7 @@ rule build_biomass_potentials:
     params:
         biomass=config_provider("biomass"),
     input:
-        enspreso_biomass=storage(
+        enspreso_biomass="data/ENSPRESO_BIOMASS.xlsx",
-            "https://zenodo.org/records/10356004/files/ENSPRESO_BIOMASS.xlsx",
-            keep_local=True,
-        ),
         eurostat="data/eurostat/Balances-April2023",
         nuts2="data/bundle/nuts/NUTS_RG_10M_2013_4326_LEVL_2.geojson",
         regions_onshore=resources("regions_onshore_elec_s{simpl}_{clusters}.geojson"),
@@ -435,10 +433,8 @@ rule build_biomass_potentials:
 
 rule build_biomass_transport_costs:
     input:
-        transport_cost_data=storage(
+        sc1="data/biomass_transport_costs_supplychain1.csv",
-            "https://publications.jrc.ec.europa.eu/repository/bitstream/JRC98626/biomass potentials in europe_web rev.pdf",
+        sc2="data/biomass_transport_costs_supplychain2.csv",
-            keep_local=True,
-        ),
     output:
         biomass_transport_costs=resources("biomass_transport_costs.csv"),
     threads: 1
@@ -460,10 +456,7 @@ rule build_sequestration_potentials:
             "sector", "regional_co2_sequestration_potential"
         ),
     input:
-        sequestration_potential=storage(
+        sequestration_potential="data/complete_map_2020_unit_Mt.geojson",
-            "https://raw.githubusercontent.com/ericzhou571/Co2Storage/main/resources/complete_map_2020_unit_Mt.geojson",
-            keep_local=True,
-        ),
         regions_onshore=resources("regions_onshore_elec_s{simpl}_{clusters}.geojson"),
         regions_offshore=resources("regions_offshore_elec_s{simpl}_{clusters}.geojson"),
     output:
@@ -505,9 +498,7 @@ rule build_salt_cavern_potentials:
 
 rule build_ammonia_production:
     input:
-        usgs=storage(
+        usgs="data/myb1-2022-nitro-ert.xlsx",
-            "https://d9-wret.s3.us-west-2.amazonaws.com/assets/palladium/production/s3fs-public/media/files/myb1-2022-nitro-ert.xlsx"
-        ),
     output:
         ammonia_production=resources("ammonia_production.csv"),
     threads: 1
@@ -636,10 +627,7 @@ rule build_industrial_distribution_key:
     input:
         regions_onshore=resources("regions_onshore_elec_s{simpl}_{clusters}.geojson"),
         clustered_pop_layout=resources("pop_layout_elec_s{simpl}_{clusters}.csv"),
-        hotmaps=storage(
+        hotmaps="data/Industrial_Database.csv",
-            "https://gitlab.com/hotmaps/industrial_sites/industrial_sites_Industrial_Database/-/raw/master/data/Industrial_Database.csv",
-            keep_local=True,
-        ),
         gem_gspt="data/gem/Global-Steel-Plant-Tracker-April-2024-Standard-Copy-V1.xlsx",
         ammonia="data/ammonia_plants.csv",
         cement_supplement="data/cement-plants-noneu.csv",
@@ -1028,6 +1016,7 @@ rule prepare_sector_network:
         RDIR=RDIR,
         heat_pump_sources=config_provider("sector", "heat_pump_sources"),
         heat_systems=config_provider("sector", "heat_systems"),
+        energy_totals_year=config_provider("energy", "energy_totals_year"),
     input:
         unpack(input_profile_offwind),
         **rules.cluster_gas_network.output,
@@ -1105,6 +1094,7 @@ rule prepare_sector_network:
         district_heat_share=resources(
             "district_heat_share_elec_s{simpl}_{clusters}_{planning_horizons}.csv"
         ),
+        heating_efficiencies=resources("heating_efficiencies.csv"),
         temp_soil_total=resources("temp_soil_total_elec_s{simpl}_{clusters}.nc"),
         temp_air_total=resources("temp_air_total_elec_s{simpl}_{clusters}.nc"),
         cop_profiles=resources("cop_profiles_elec_s{simpl}_{clusters}.nc"),

rules/retrieve.smk

@@ -191,6 +191,65 @@ if config["enable"]["retrieve"]:
             validate_checksum(output[0], input[0])
 
 
+if config["enable"]["retrieve"]:
+
+    rule retrieve_jrc_enspreso_biomass:
+        input:
+            storage(
+                "https://zenodo.org/records/10356004/files/ENSPRESO_BIOMASS.xlsx",
+                keep_local=True,
+            ),
+        output:
+            "data/ENSPRESO_BIOMASS.xlsx",
+        retries: 1
+        run:
+            move(input[0], output[0])
+
+
+if config["enable"]["retrieve"]:
+
+    rule retrieve_hotmaps_industrial_sites:
+        input:
+            storage(
+                "https://gitlab.com/hotmaps/industrial_sites/industrial_sites_Industrial_Database/-/raw/master/data/Industrial_Database.csv",
+                keep_local=True,
+            ),
+        output:
+            "data/Industrial_Database.csv",
+        retries: 1
+        run:
+            move(input[0], output[0])
+
+
+if config["enable"]["retrieve"]:
+
+    rule retrieve_usgs_ammonia_production:
+        input:
+            storage(
+                "https://d9-wret.s3.us-west-2.amazonaws.com/assets/palladium/production/s3fs-public/media/files/myb1-2022-nitro-ert.xlsx"
+            ),
+        output:
+            "data/myb1-2022-nitro-ert.xlsx",
+        retries: 1
+        run:
+            move(input[0], output[0])
+
+
+if config["enable"]["retrieve"]:
+
+    rule retrieve_geological_co2_storage_potential:
+        input:
+            storage(
+                "https://raw.githubusercontent.com/ericzhou571/Co2Storage/main/resources/complete_map_2020_unit_Mt.geojson",
+                keep_local=True,
+            ),
+        output:
+            "data/complete_map_2020_unit_Mt.geojson",
+        retries: 1
+        run:
+            move(input[0], output[0])
+
+
 if config["enable"]["retrieve"]:
 
     # Downloading Copernicus Global Land Cover for land cover and land use:

rules/solve_myopic.smk

@@ -10,6 +10,7 @@ rule add_existing_baseyear:
         existing_capacities=config_provider("existing_capacities"),
         costs=config_provider("costs"),
         heat_pump_sources=config_provider("sector", "heat_pump_sources"),
+        energy_totals_year=config_provider("energy", "energy_totals_year"),
     input:
         network=RESULTS
         + "prenetworks/elec_s{simpl}_{clusters}_l{ll}_{opts}_{sector_opts}_{planning_horizons}.nc",
@@ -26,6 +27,7 @@ rule add_existing_baseyear:
         existing_heating_distribution=resources(
             "existing_heating_distribution_elec_s{simpl}_{clusters}_{planning_horizons}.csv"
         ),
+        heating_efficiencies=resources("heating_efficiencies.csv"),
     output:
         RESULTS
         + "prenetworks-brownfield/elec_s{simpl}_{clusters}_l{ll}_{opts}_{sector_opts}_{planning_horizons}.nc",

rules/solve_perfect.smk

@@ -8,6 +8,7 @@ rule add_existing_baseyear:
         existing_capacities=config_provider("existing_capacities"),
         costs=config_provider("costs"),
         heat_pump_sources=config_provider("sector", "heat_pump_sources"),
+        energy_totals_year=config_provider("energy", "energy_totals_year"),
     input:
         network=RESULTS
         + "prenetworks/elec_s{simpl}_{clusters}_l{ll}_{opts}_{sector_opts}_{planning_horizons}.nc",
@@ -25,6 +26,7 @@ rule add_existing_baseyear:
             "existing_heating_distribution_elec_s{simpl}_{clusters}_{planning_horizons}.csv"
         ),
         existing_heating="data/existing_infrastructure/existing_heating_raw.csv",
+        heating_efficiencies=resources("heating_efficiencies.csv"),
     output:
         RESULTS
         + "prenetworks-brownfield/elec_s{simpl}_{clusters}_l{ll}_{opts}_{sector_opts}_{planning_horizons}.nc",

scripts/add_existing_baseyear.py

@@ -418,6 +418,50 @@ def add_power_capacities_installed_before_baseyear(n, grouping_years, costs, bas
             ]
 
 
+def get_efficiency(heat_system, carrier, nodes, heating_efficiencies, costs):
+    """
+    Computes the heating system efficiency based on the sector and carrier
+    type.
+
+    Parameters:
+    -----------
+    heat_system : object
+    carrier : str
+        The type of fuel or energy carrier (e.g., 'gas', 'oil').
+    nodes : pandas.Series
+        A pandas Series containing node information used to match the heating efficiency data.
+    heating_efficiencies : dict
+        A dictionary containing efficiency values for different carriers and sectors.
+    costs : pandas.DataFrame
+        A DataFrame containing boiler cost and efficiency data for different heating systems.
+
+    Returns:
+    --------
+    efficiency : pandas.Series or float
+        A pandas Series mapping the efficiencies based on nodes for residential and services sectors, or a single
+        efficiency value for other heating systems (e.g., urban central).
+
+    Notes:
+    ------
+    - For residential and services sectors, efficiency is mapped based on the nodes.
+    - For other sectors, the default boiler efficiency is retrieved from the `costs` database.
+    """
+
+    if heat_system.value == "urban central":
+        boiler_costs_name = getattr(heat_system, f"{carrier}_boiler_costs_name")
+        efficiency = costs.at[boiler_costs_name, "efficiency"]
+    elif heat_system.sector.value == "residential":
+        key = f"{carrier} residential space efficiency"
+        efficiency = nodes.str[:2].map(heating_efficiencies[key])
+    elif heat_system.sector.value == "services":
+        key = f"{carrier} services space efficiency"
+        efficiency = nodes.str[:2].map(heating_efficiencies[key])
+    else:
+        logger.warning(f"{heat_system} not defined.")
+
+    return efficiency
+
+
 def add_heating_capacities_installed_before_baseyear(
     n: pypsa.Network,
     baseyear: int,
@@ -546,6 +590,10 @@ def add_heating_capacities_installed_before_baseyear(
                 lifetime=costs.at[heat_system.resistive_heater_costs_name, "lifetime"],
             )
 
+            efficiency = get_efficiency(
+                heat_system, "gas", nodes, heating_efficiencies, costs
+            )
+
             n.madd(
                 "Link",
                 nodes,
@@ -554,7 +602,7 @@ def add_heating_capacities_installed_before_baseyear(
                 bus1=nodes + " " + heat_system.value + " heat",
                 bus2="co2 atmosphere",
                 carrier=heat_system.value + " gas boiler",
-                efficiency=costs.at[heat_system.gas_boiler_costs_name, "efficiency"],
+                efficiency=efficiency,
                 efficiency2=costs.at["gas", "CO2 intensity"],
                 capital_cost=(
                     costs.at[heat_system.gas_boiler_costs_name, "efficiency"]
@@ -569,6 +617,10 @@ def add_heating_capacities_installed_before_baseyear(
                 lifetime=costs.at[heat_system.gas_boiler_costs_name, "lifetime"],
             )
 
+            efficiency = get_efficiency(
+                heat_system, "oil", nodes, heating_efficiencies, costs
+            )
+
             n.madd(
                 "Link",
                 nodes,
@@ -577,7 +629,7 @@ def add_heating_capacities_installed_before_baseyear(
                 bus1=nodes + " " + heat_system.value + " heat",
                 bus2="co2 atmosphere",
                 carrier=heat_system.value + " oil boiler",
-                efficiency=costs.at[heat_system.oil_boiler_costs_name, "efficiency"],
+                efficiency=efficiency,
                 efficiency2=costs.at["oil", "CO2 intensity"],
                 capital_cost=costs.at[heat_system.oil_boiler_costs_name, "efficiency"]
                 * costs.at[heat_system.oil_boiler_costs_name, "fixed"],
@@ -621,12 +673,12 @@ if __name__ == "__main__":
 
         snakemake = mock_snakemake(
             "add_existing_baseyear",
-            configfiles="config/config.yaml",
+            configfiles="config/test/config.myopic.yaml",
             simpl="",
-            clusters="20",
+            clusters="5",
             ll="v1.5",
             opts="",
-            sector_opts="none",
+            sector_opts="",
             planning_horizons=2030,
         )
 
@@ -660,6 +712,11 @@ if __name__ == "__main__":
 
     if options["heating"]:
 
+        # one could use baseyear here instead (but dangerous if no data)
+        fn = snakemake.input.heating_efficiencies
+        year = int(snakemake.params["energy_totals_year"])
+        heating_efficiencies = pd.read_csv(fn, index_col=[1, 0]).loc[year]
+
         add_heating_capacities_installed_before_baseyear(
             n=n,
             baseyear=baseyear,

scripts/build_biomass_transport_costs.py

@@ -16,33 +16,51 @@ assuming as an approximation energy content of wood pellets
 
 @author: bw0928
 """
-
-import platform
-
 import pandas as pd
-import tabula as tbl
 
 ENERGY_CONTENT = 4.8  # unit MWh/t (wood pellets)
-system = platform.system()
-encoding = "cp1252" if system == "Windows" else "utf-8"
 
 
-def get_countries():
+def get_cost_per_tkm(pdf, datapage, countrypage):
-    pandas_options = dict(
+    """
+    Extracts the cost tables from the JRC report PDF.
+
+    https://publications.jrc.ec.europa.eu/repository/bitstream/JRC98626/biomass%20potentials%20in%20europe_web%20rev.pdf
+    - pdf (str): The filepath of the PDF file containing the data.
+    - datapage (int): The page number of the data table in the PDF.
+    - countrypage (int): The page number of the table containing the country indices in the PDF.
+
+    Returns:
+    - pandas.DataFrame: The data table with the cost per tkm for biomass transport, indexed by country.
+
+    Raises:
+    - ImportError: If tabula-py and platform are not installed.
+    """
+    try:
+        import platform
+
+        import tabula as tbl
+    except:
+        ImportError("Please install tabula-py and platform")
+
+    system = platform.system()
+    encoding = "cp1252" if system == "Windows" else "utf-8"
+
+    # Obtain countries:
+    pandas_options_country = dict(
         skiprows=range(6), header=None, index_col=0, encoding=encoding
     )
 
-    return tbl.read_pdf(
+    countries = tbl.read_pdf(
-        str(snakemake.input.transport_cost_data),
+        pdf,
-        pages="145",
+        pages=countrypage,
         multiple_tables=False,
-        pandas_options=pandas_options,
+        pandas_options=pandas_options_country,
         encoding=encoding,
     )[0].index
 
-
+    # Obtain data tables
-def get_cost_per_tkm(page, countries):
+    pandas_options_data = dict(
-    pandas_options = dict(
         skiprows=range(6),
         header=0,
         sep=" |,",
@@ -52,10 +70,10 @@ def get_cost_per_tkm(page, countries):
     )
 
     sc = tbl.read_pdf(
-        str(snakemake.input.transport_cost_data),
+        pdf,
-        pages=page,
+        pages=datapage,
         multiple_tables=False,
-        pandas_options=pandas_options,
+        pandas_options=pandas_options_data,
         encoding=encoding,
     )[0]
     sc.index = countries
@@ -65,10 +83,16 @@ def get_cost_per_tkm(page, countries):
 
 
 def build_biomass_transport_costs():
-    countries = get_countries()
+    # Optional build from JRC report pdf, requires tabula and java dependencies.
+    # Update `pdf` path to the JRC report if needed.
+    # sc1 = get_cost_per_tkm(pdf = "report.pdf", datapage=146, countrypage=145)
+    # sc2 = get_cost_per_tkm(pdf = "report.pdf", datapage=147, countrypage=145)
 
-    sc1 = get_cost_per_tkm(146, countries)
+    # Use extracted csv from JRC report
-    sc2 = get_cost_per_tkm(147, countries)
+    # https://publications.jrc.ec.europa.eu/repository/bitstream/JRC98626/biomass%20potentials%20in%20europe_web%20rev.pdf
+    # Pages 146 (144) for supply chain 1 and 147 (145) for supply chain 2
+    sc1 = pd.read_csv(snakemake.input.sc1, index_col=0, skiprows=2)
+    sc2 = pd.read_csv(snakemake.input.sc2, index_col=0, skiprows=2)
 
     # take mean of both supply chains
     to_concat = [sc1["EUR/km/ton"], sc2["EUR/km/ton"]]

scripts/build_energy_totals.py

@@ -367,6 +367,24 @@ def idees_per_country(ct: str, base_dir: str) -> pd.DataFrame:
     assert df.index[43] == "Thermal uses"
     ct_totals["thermal uses residential"] = df.iloc[43]
 
+    df = pd.read_excel(fn_residential, "RES_hh_eff", index_col=0)
+
+    ct_totals["total residential space efficiency"] = df.loc["Space heating"]
+
+    assert df.index[5] == "Diesel oil"
+    ct_totals["oil residential space efficiency"] = df.iloc[5]
+
+    assert df.index[6] == "Natural gas"
+    ct_totals["gas residential space efficiency"] = df.iloc[6]
+
+    ct_totals["total residential water efficiency"] = df.loc["Water heating"]
+
+    assert df.index[18] == "Diesel oil"
+    ct_totals["oil residential water efficiency"] = df.iloc[18]
+
+    assert df.index[19] == "Natural gas"
+    ct_totals["gas residential water efficiency"] = df.iloc[19]
+
     # services
 
     df = pd.read_excel(fn_tertiary, "SER_hh_fec", index_col=0)
@@ -400,6 +418,24 @@ def idees_per_country(ct: str, base_dir: str) -> pd.DataFrame:
     assert df.index[46] == "Thermal uses"
     ct_totals["thermal uses services"] = df.iloc[46]
 
+    df = pd.read_excel(fn_tertiary, "SER_hh_eff", index_col=0)
+
+    ct_totals["total services space efficiency"] = df.loc["Space heating"]
+
+    assert df.index[5] == "Diesel oil"
+    ct_totals["oil services space efficiency"] = df.iloc[5]
+
+    assert df.index[7] == "Conventional gas heaters"
+    ct_totals["gas services space efficiency"] = df.iloc[7]
+
+    ct_totals["total services water efficiency"] = df.loc["Hot water"]
+
+    assert df.index[20] == "Diesel oil"
+    ct_totals["oil services water efficiency"] = df.iloc[20]
+
+    assert df.index[21] == "Natural gas"
+    ct_totals["gas services water efficiency"] = df.iloc[21]
+
     # agriculture, forestry and fishing
 
     start = "Detailed split of energy consumption (ktoe)"
@@ -576,7 +612,8 @@ def build_idees(countries: List[str]) -> pd.DataFrame:
     # efficiency kgoe/100km -> ktoe/100km so that after conversion TWh/100km
     totals.loc[:, "passenger car efficiency"] /= 1e6
     # convert ktoe to TWh
-    exclude = totals.columns.str.fullmatch("passenger cars")
+    patterns = ["passenger cars", ".*space efficiency", ".*water efficiency"]
+    exclude = totals.columns.str.fullmatch("|".join(patterns))
     totals = totals.copy()
     totals.loc[:, ~exclude] *= 11.63 / 1e3
 
@@ -654,11 +691,14 @@ def build_energy_totals(
     eurostat_countries = eurostat.index.unique(0)
     eurostat_years = eurostat.index.unique(1)
 
-    to_drop = ["passenger cars", "passenger car efficiency"]
     new_index = pd.MultiIndex.from_product(
         [countries, eurostat_years], names=["country", "year"]
     )
 
+    efficiency_keywords = ["space efficiency", "water efficiency"]
+    to_drop = idees.columns[idees.columns.str.contains("|".join(efficiency_keywords))]
+    to_drop = to_drop.append(pd.Index(["passenger cars", "passenger car efficiency"]))
+
     df = idees.reindex(new_index).drop(to_drop, axis=1)
 
     in_eurostat = df.index.levels[0].intersection(eurostat_countries)
@@ -1501,6 +1541,59 @@ def build_transformation_output_coke(eurostat, fn):
     df.to_csv(fn)
 
 
+def build_heating_efficiencies(
+    countries: List[str], idees: pd.DataFrame
+) -> pd.DataFrame:
+    """
+    Build heating efficiencies for a set of countries based on IDEES data.
+
+    Parameters
+    ----------
+    countries : List[str]
+        List of country codes.
+    idees : pd.DataFrame
+        DataFrame with IDEES data.
+
+    Returns
+    -------
+    pd.DataFrame
+        DataFrame with heating efficiencies.
+
+
+    Notes
+    -----
+    - It fills missing data with average data.
+    """
+
+    years = np.arange(2000, 2022)
+
+    cols = idees.columns[
+        idees.columns.str.contains("space efficiency")
+        ^ idees.columns.str.contains("water efficiency")
+    ]
+
+    heating_efficiencies = pd.DataFrame(idees[cols])
+
+    new_index = pd.MultiIndex.from_product(
+        [countries, heating_efficiencies.index.unique(1)],
+        names=["country", "year"],
+    )
+
+    heating_efficiencies = heating_efficiencies.reindex(index=new_index)
+
+    for col in cols:
+        unstacked = heating_efficiencies[col].unstack()
+
+        fillvalue = unstacked.mean()
+
+        for ct in unstacked.index:
+            mask = unstacked.loc[ct].isna()
+            unstacked.loc[ct, mask] = fillvalue[mask]
+        heating_efficiencies[col] = unstacked.stack()
+
+    return heating_efficiencies
+
+
 # %%
 if __name__ == "__main__":
     if "snakemake" not in globals():
@@ -1556,3 +1649,6 @@ if __name__ == "__main__":
 
     transport = build_transport_data(countries, population, idees)
     transport.to_csv(snakemake.output.transport_name)
+
+    heating_efficiencies = build_heating_efficiencies(countries, idees)
+    heating_efficiencies.to_csv(snakemake.output.heating_efficiencies)

scripts/plot_hydrogen_network.py

@@ -115,7 +115,9 @@ def plot_h2_map(n, regions):
 
         retro_wo_new_i = h2_retro.index.difference(h2_new.index)
         h2_retro_wo_new = h2_retro.loc[retro_wo_new_i]
-        h2_retro_wo_new.index = h2_retro_wo_new.index_orig
+        h2_retro_wo_new.index = h2_retro_wo_new.index_orig.apply(
+            lambda x: x.split("-2")[0]
+        )
 
         to_concat = [h2_new, h2_retro_w_new, h2_retro_wo_new]
         h2_total = pd.concat(to_concat).p_nom_opt.groupby(level=0).sum()
@@ -126,7 +128,12 @@ def plot_h2_map(n, regions):
     link_widths_total = h2_total / linewidth_factor
 
     n.links.rename(index=lambda x: x.split("-2")[0], inplace=True)
-    n.links = n.links.groupby(level=0).first()
+    # group links by summing up p_nom values and taking the first value of the rest of the columns
+    other_cols = dict.fromkeys(n.links.columns.drop(["p_nom_opt", "p_nom"]), "first")
+    n.links = n.links.groupby(level=0).agg(
+        {"p_nom_opt": "sum", "p_nom": "sum", **other_cols}
+    )
+
     link_widths_total = link_widths_total.reindex(n.links.index).fillna(0.0)
     link_widths_total[n.links.p_nom_opt < line_lower_threshold] = 0.0
 

scripts/prepare_sector_network.py

@@ -2143,9 +2143,14 @@ def build_heat_demand(n):
     for sector, use in product(sectors, uses):
         name = f"{sector} {use}"
 
+        # efficiency for final energy to thermal energy service
+        eff = pop_weighted_energy_totals.index.str[:2].map(
+            heating_efficiencies[f"total {sector} {use} efficiency"]
+        )
+
         heat_demand[name] = (
             heat_demand_shape[name] / heat_demand_shape[name].sum()
-        ).multiply(pop_weighted_energy_totals[f"total {sector} {use}"]) * 1e6
+        ).multiply(pop_weighted_energy_totals[f"total {sector} {use}"] * eff) * 1e6
         electric_heat_supply[name] = (
             heat_demand_shape[name] / heat_demand_shape[name].sum()
         ).multiply(pop_weighted_energy_totals[f"electricity {sector} {use}"]) * 1e6
@@ -4709,6 +4714,10 @@ if __name__ == "__main__":
     )
     pop_weighted_energy_totals.update(pop_weighted_heat_totals)
 
+    fn = snakemake.input.heating_efficiencies
+    year = int(snakemake.params["energy_totals_year"])
+    heating_efficiencies = pd.read_csv(fn, index_col=[1, 0]).loc[year]
+
     patch_electricity_network(n)
 
     spatial = define_spatial(pop_layout.index, options)