1.
Aissa Sehili
Delft3D FM at BAW Hamburg – Current Status and Outlook
Delft Software Days, 14.11.2022

2.
www.baw.de
|
Content
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 2
1. The BAW (Federal Waterways Engineering and Research Institute)
2. Developed D-Flow FM models
3. Current activities
4. Challenges: scalability on the BAW-Cluster
5. Outlook

3.
www.baw.de
|
1 BAW: Federal Waterways Engineering and Research Institute
Hamburg
Karlsruhe
Standorte BAW
The BAW ist central consultancy provider for
- the Federal Ministry for Digital and Transport (BMDV)
and
- the Federal Waterways and Shipping Administration
(WSV)
The BAW is a Research Institute
The BAW provides expertise with regards to
- Operation and maintenance
- Extentions and renewal
- New projects
of all federal waterways infrastructures.
Inland waterways 6545 km
Coastal waterways 761 km
Locks: 326
Ship lifts: 4
Weirs: 337
Barriers: 11
Bridges: 1354
Channel bridges: 10
Hydroelectric power plants: 106
Quelle:
www.WSV.de,
Stand
2007
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 3

4.
www.baw.de
|
Standorte BAW-Dienststellen
Hamburg
Karlsruhe
Page 4
1 BAW: Federal Waterways Engineering and Research Institute

5.
www.baw.de
|
1 Tasks of the BAW
Hydraulic engineering Structural engineering
Geotechnical
engineering
Shipbuilding
R&D
Infrastructure
Environment
Mobility
Page 5

6.
www.baw.de
|
1 Technical Equipment (selection)
Model of the "Jungferngrund" in the River Rhine
Shallow Water Basin for Coastal Investigations
Ship Handling Simulation High-performance computer UnTRIM2
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 6

7.
www.baw.de
|
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 7
2 Developed D-Flow FM models

8.
www.baw.de
|
• December 2014: pre-release agreement
• February 2015:
- First Elbe Flexible Mesh converted from an UnTRIM unstructured grid
- First D-Flow FM simulations using DeltaShell plugin and …first problems:
(no 3D simulation, no salinity, problems with setting boundary condition,
problems with restarts,…)
• Mai 2015: version used to set up the very first model
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 8
2 Elbe Estuary Model (DSD-INT 2015)

9.
www.baw.de
|
2 Elbe Estuary Model (DSD-INT 2015)
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 9

10.
www.baw.de
|
2 Elbe Estuary Model: close up at Cuxhaven
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 10

11.
www.baw.de
|
2 Elbe Km 725 (Cuxhaven)
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 11

12.
www.baw.de
|
2 Elbe Km 623N (St. Pauli)
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 12

13.
www.baw.de
|
▪ Mesh conversion from UnTRIM
▪ 3D model with salinity
▪ Realistic forcing
Bremerhaven Alter Leuchtturm
2 Jade-Weser Model (OTB)
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 13

14.
www.baw.de
|
▪ The 2D model was designed to estimate the large-scale
suspended sediment transport in the southern North Sea.
▪ Research Goal:
▪ We aimed at an estimation of the perennial variability of
sediment transport pathways in the southern North Sea
▪ Model Information:
▪ 342,000 Elements and 540,000 vertices
▪ Hydrodynamics (EasyGSH-DB), Salinity, Heat-Flux,
Bedload, suspended Sediments
▪ Deployed on the cluster
2 The Herkulis Southern North Sea Model*
Model Setup
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 14
* Robert Hagen

15.
www.baw.de
|
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 15
3 Currents activities
3.1 Operational model & model based monitoring
3.2 UGRID-NetCDF Format

16.
www.baw.de
|
Das DCSM FM Modell
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 16

17.
www.baw.de
|
3.1 DCSM FM 100m - GBOM
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 17

18.
www.baw.de
|
German Bight Operational Model
ICON-EU (NETCDF)
weather forecast model DWD
DCSM FM ( UGRID-NETCDF)
BAW Dev. Env.
GBOM:
- Topo 2016
- Topo 2019A
- Topo 2019C
- Astro
…
Water Discharge BFG
Water Temperature BFG: Cumlosen
Wind,
Pres,Temp
Wl
BSHcmod
North Sea forecast model BSH
Sa,
Temp
3.1 GBOM flow chart
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 18
BSH: Federal Maritime and Hydrographic Agency. BFG: Federal Institute of Hydrology

19.
www.baw.de
|
Sa,
Temp
UGRID-NETCDF
Output (HIS or MAP)
DCSM FM
(Delft3D FM)
Forcing
GBOM
(UnTRIM)
Operational
Salinity, Temperature
BSH forecast
ICON-EU METEO ICON-EU METEO
Discharge, Temperature
Measurements BfG
48 hours forecast
Operational data communication between
D-Flow FM (Deltares) and UnTRIM (BAW)
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 19
3.1 GBOM flow chart

20.
www.baw.de
|
28.08.2018
00:00:00
MEZ
29.08.2018
00:00:00
MEZ
30.08.2018
00:00:00
MEZ
31.08.2018
00:00:00
MEZ
01.09.2018
00:00:00
MEZ
02.09.2018
00:00:00
MEZ
03.09.2018
00:00:00
MEZ
04.09.2018
00:00:00
MEZ
05.09.2018
00:00:00
MEZ
06.09.2018
00:00:00
MEZ
07.09.2018
00:00:00
MEZ
08.09.2018
00:00:00
MEZ
-2.
-1.5
-1.
-0.5
0
0.5
1.
1.5
2.
2.5
3.
Wasserstand
mNHN
Wasserstand 10wl Knoten "St.Pauli_m"
Wasserstand mesStPauli Hamburg_St_Pauli_PegelOnline
3.1 Results at St. Pauli with DCSM forcing
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 20

21.
www.baw.de
|
28.08.2018
00:00:00
MEZ
29.08.2018
00:00:00
MEZ
30.08.2018
00:00:00
MEZ
31.08.2018
00:00:00
MEZ
01.09.2018
00:00:00
MEZ
02.09.2018
00:00:00
MEZ
03.09.2018
00:00:00
MEZ
04.09.2018
00:00:00
MEZ
05.09.2018
00:00:00
MEZ
06.09.2018
00:00:00
MEZ
07.09.2018
00:00:00
MEZ
08.09.2018
00:00:00
MEZ
-2.
-1.5
-1.
-0.5
0
0.5
1.
1.5
2.
2.5
3.
Wasserstand
mNHN
Wasserstand 10wl Knoten "Zollenspieker_m"
Wasserstand mesZoll Zollenspieker_Elbe_km_598_PegelOnline
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 21
3.1 Results at Zollenspieker with DCSM forcing

22.
www.baw.de
|
Planned implementation:
with rearrangement of dredged material building underwater structures in the mouth of the Elbe in
order to minimize increase of tidal range
i.e. raise tidal low water LW
and lower tidal high water HW
Plan:
allow container ships with draught of
13,5 m to reach the Port of Hamburg
independent of tide
3.1 Deepening of the Elbe-waterway*
Plan and application: September 2006, start of construction works: July 2019, finished in January 2022
deepening
widening and
passing box for
encounters of
incomming and
outgoing ships
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 22
* Elisabeth Rudolph

23.
www.baw.de
|
▪ Identify the impact of the construction work by
comparing to measurements
… but already questions during the construction
period
➢ Support waterways and shipping administration
(WSV) giving answers with model-based monitoring
▪ Requirement:
• Today’s boundary values (wind, water level, salinity,
fresh water discharge)
• Up-to-date topography of the Elbe (i.e. during
construction works)
Fotodokumentation der Jan de Nul Group
https://www.fahrrinnenanpassung.de/download.html
… not only deepening, also building i.e. the initial dam (NHN-3,5 m)
of the under water structure Medemrinne
Medemgrund
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 23
3.1 Model-based monitoring of the deepening of the Elbe-waterway

24.
www.baw.de
|
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 24
Today:
GBOM (UnTRIM2 )
Step 1:
every day operational forecast using different topographies
representing states of the construction
▪ Generating boundary values
▪ Answer questions during construction phase at once
Step 2:
hindcast using different topographies representing states of the
construction
▪ Wind from analysis (… not forecast)
▪ Discharge from measurements
Step 3:
analyse tidal characteristic numbers for selected periods during the
construction phase
▪ Compare before start of construction with during construction
▪ Compare with predicted values in BAW report (2006)
3.1 Model-based monitoring of the deepening of the Elbe-waterway

25.
www.baw.de
|
3.1 Change in Topography in the mouth of the Elbe
state C – state A
Under water structure UWA Medemrinne
(rearrangement of dredged material)
Under water structure UWA Neufeld
(rearrangement of dredged material)
Holding area
Warteplatz Brunsbüttel
and naturally moving tidal channels,
dunes, sand banks, …
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 25

26.
www.baw.de
|
3.1 Example: mean tidal low water
SNZ23_2021 Q320: LW in the mouth of the Elbe (TgWest)
state A (2019) „before start of work“ state C* (2021) „deepening only“
state C (2021) „partial opening“
state C - state A state C* - state A
Do the under water structures
change the tidal low water LW
already in the construction period?
Yes
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 26

27.
www.baw.de
|
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 27
3 Currents activities
3.2 UGRID-NetCDF Format

28.
www.baw.de
|
UGRID CF NetCDF file mesh, IC, BC
NetCDF is a well established and freely
available cross-platform API
The CF (Climate and Forecast)
conventions metadata provide
standards for the description of
NetCDF data
UGRID can be regarded as an extension
to CF ready made for use with
unstructured grids
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 28
3.2 Towards a unified UGRID-NetCDF file format

29.
www.baw.de
|
Why UGRID CF NetCDF?
Safe use of dimensional data thanks CF canonical units
→ attribute “units”
Safe data transformation (e.g. intensive to extensive variables)
→ attribute “cell_methods”
Filtering based on number of data (easy tidal analysis)
→ attribute “ancillary_variables” + attribute “standard_name = number_of_observations”
Keep track of data processing activities
→ attribute “history”
Visualization of data at their original locations
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 29
3.2 Towards a unified UGRID-NetCDF file format

30.
www.baw.de
|
Tidal Characteristic Numbers of Water Level
float tnw_mean (TimePeriod, nFace) ;
:cell_methods = "TimePeriod: mean nFace: mean" ;
:ancillary_variables = “tnw_number of events" ;
int tnw_number of events(TimePeriod,nFace) ;
:units = “1" ;
:standard_name = number_of_observations" ;
Remark: “number of observations” (NOO) can be used to modify
visualization of results. E. g. results are shown
− only for locations where all events were detected, or
− for locations where at a certain NOO. were detected,
− … and so on.
Mean tidal low water at Elbe Estuary for the period 16.02-03.03.2019 (28 tides)
https://wiki.baw.de/en/index.php/NCANALYSE
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 30
3.2 Towards a unified UGRID-NetCDF file format

31.
www.baw.de
|
Ongoing
Joint project between Deltares and BAW to enhance
D-Flow FM outputs to the UGRID CF NetCDF standard
used at BAW.
D-Flow FM NetCDF results should be processed
without modification by BAW NC-Postprocessors.
Striving to
Make UGRID extensions full part of the CF NetCDF
conventions.
Use of BAW NetCDF data at Deltares as is.
UGRID
NetCDF
UnTRIM
Delft3D FM
DELWAQ
Others
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 31
3.2 Towards a unified UGRID-NetCDF file format

32.
www.baw.de
|
UGRID
NetCDF
UnTRIM
Delft3D FM
DELWAQ
Others
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 32
3.2 Towards a unified UGRID-NetCDF file format
▪ Modification in the map files
▪ Modification in the his files
▪ Split the output into 3D and 2D
▪ Include some BAW intrinsic metadata
(DMQS) for data management and
quality control

33.
www.baw.de
|
• Model runs for a spring-neap cycle
• Mean tidal condition (07.2006)
Original UGRID NetCDF
Elbe D-Flow FM model results
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 33
3.2 Postprocessing: Tidal Characteristic Numbers for Water Level

34.
www.baw.de
|
• Model runs for a spring-neap cycle
• Mean tidal condition (07.2006)
Original UGRID NetCDF
Elbe D-Flow FM model results
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 34
3.2 Postprocessing: Tidal Characteristic Numbers for Water Level

35.
www.baw.de
|
• Model runs for a spring-neap cycle
• Mean tidal condition (07.2006)
Original UGRID NetCDF
Elbe D-Flow FM model results
Without filtering (NOO)
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 35
3.2 Postprocessing: Tidal Characteristic Numbers for Water Level

36.
www.baw.de
|
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 36
4 Challenges: scalability on the BAW cluster

37.
www.baw.de
|
4 Challenges: running on the cluster
Linux Build:
• Numerous prerequisite libraries need to be installed
• MPI library compatibility issues
• OS issues
• Hardware issues
Singularity container image:
• MPI library issues (use host)
• Batch system issues (e.g. communication with SLURM via pmi2)
• Right setting for the cluster interconnect (Omnipath, InfiniBand,…)
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 37

38.
www.baw.de
|
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 38
4 Build benchmarks on BAW cluster

39.
www.baw.de
|
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 39
4 Singularity benchmarks on BAW cluster

40.
www.baw.de
|
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 40
4 Singularity benchmarks on future BAW cluster
Setting along with Intel MPI 2021.5 (host) and InfiniBand interconnect:
export I_MPI_FABRICS=shm:ofi
export FI_PROVIDER=verbs
export I_MPI_PMI=pmi2
export I_MPI_PMI_LIBRARY=/usr/lib64/libpmi2.so.0.0.0
The verbs-provider requires additional libraries which are not in the standard Intel
MPI installation, Therefore, some new singularity bindpath were used.
The communication with SLURM via PMI2 requires also further bindpaths.

41.
www.baw.de
|
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 41
5 Outlook

42.
www.baw.de
|
5 Outlook
Delft3D FM at BAW Hamburg – Current Status and Outlook
Page 42
✓ Able in the next future to process UGRID-NetCDF
results from D-Flow FM as is
▪ Calibration and validation of a model of the
German Bight.
▪ Extend the model for sediment transport and
waves
Offline BC from DCSM FM:
can we do it differently?
Actual extend

43.
Bundesanstalt für Wasserbau
22559 Hamburg, Germany
www.baw.de
Thanks!