The Food Security TEP enhanced with the technologies from the ExtremeEarth project

1. © VISTA 2019 www.vista-geo.de No. 1
Florian Appel, Markus Muerth, Heike Bach
– VISTA Remote Sensing in Geosciences GmbH
Claudia Paris, Lorenzo Buzzone
- Department of Information Engineering and Computer Science, University of Trento (Italy)
The Food Security Use Case
Extreme Data Analytics to Manage an Extremely Dynamic
Planet

2. © VISTA 2019 www.vista-geo.de No. 2
Landwirtschaft
• Erntevorhersage
• Precision Farming
• Bewässerungsplanung
• Bio-Zertifizierung
Data Assimilation
Snow Cover
Snow Water Equivalent
Snow & Water Balance
Modeling
Hydrologie
• Schneemonitoring
• Wasserhaushalt
• Abflussvorhersage
• Wasserkraftproduktion
VISTA Remote Sensing in Geosciences GmbH
Gegründet: 1995 in München Mitarbeiter: 28 Seit Juli 2017: BayWa AG hält 51 %
operational
Services
Satelliten und Modell basierte Anwendungen für die Landwirtschaft und Hydrologie
S N
S S
O W
E EN
YPSILON
Yield Prediction
by Satellite
Roll in „Extreme Earth“: Food Security Use Case & Innovation Management
R&D
Projects

3. © VISTA 2019 www.vista-geo.de No. 3
ExtremeEarth
Objectives & Concept
Use Cases
WP4 and WP5
Copernicus and DIASs
WP1
Impact
WP6
Thematic Exploitation Platforms
Food Security TEP
Polar TEP
WP2 and WP3
Deep Learning &
Linked Open Data
HOPS
WP3 and WP2

4. © VISTA 2019 www.vista-geo.de No. 4
ExtremeEarth Applications
The Food Security Use Case
o FOOD SECURITY IS ONE OF
THE MOST CHALLENGING
ISSUES OF THIS CENTURY
(ESPECIALLY IN A
CHANGING EARTH
ENVIRONMENT)
o POPULATION GROWTH,
INCREASED FOOD
CONSUMPTION AND
CHALLENGES OF CLIMATE
CHANGE AND INCREASED
VARIABILITIES WILL EXPAND
OVER THE NEXT DECADES
• Biomass production and yield will need
to be increased
• Risks of yield loss even under extreme
environmental conditions need to be
minimized
• Irrigation requires reliable water
resources either from ground water or
surface water
• Large portion fresh water is linked to
snowfall, snow/ice storage and seasonal
release of the water
• Water Availability Maps as EO based
product
• Information to support farmers decision
making and irrigation management

5. © VISTA 2019 www.vista-geo.de No. 5
Need of Water
Polar
TEP
EO Processing
Sentinel-1
Snow
Parameters
Medium Resolution
Modelling:
Water Balance
Parameters
High Resolution
Modelling:
Crop Growth
Food Security
TEP
EO Processing
Sentinel-2
Crop
Parameters
… for secure food production
Water Availability for Irrigation
• Surface Water
• Soil Moisture
• Groundwater
Origin of the Water
Water from seasonal snow …
The Food Security Use Case
Tools and Methods

6. © VISTA 2019 www.vista-geo.de No. 6
Polar
TEP
EO Processing
Sentinel-1
Snow
Parameters
Medium Resolution
Modelling:
Water Balance
Parameters
Water from seasonal snow …
Building Blocks of Part 1
„Water from seasonal snow storage“:
• In-Situ Snow Measurements
• methods developed within ESA IAP
SnowSense
• Earth Observation
• methods applied within ESA GSE
Polar View and now applied on the
Polar TEP
• Water Balance Modelling
• methods applied within ESA GSE
Polar View and general tool of
VISTAs hydrological activities
The Food Security Use Case
Tools and Methods
In-Situ Snow
Measurements
Meteo
Data
and
NWP

7. © VISTA 2019 www.vista-geo.de No. 7
Building Blocks of Part 2
„Efficient and intelligent use of water“
• Earth Observation
• Sentinel 2 maximum exploitation
• methods applied in various activities of
VISTAs agricultural services
• Leaf Monitoring / Biomass / Drought
• Plant Dynamics Modelling
• methods applied in various activities of
VISTAs agricultural services
• Soil Moisture Simulations
• Irrigation recommendations
The Food Security Use Case
Tools and Methods
High Resolution
Modelling:
Crop Growth
Soil Moisture
Food Security
TEP
EO Processing
Sentinel-2
Crop
Parameters
… for secure food production
Deep Learning Algorithms and Results
• Field Boundaries / Crop Types / Irrigation

8. © VISTA 2019 www.vista-geo.de No. 8
For the first stage of the project, the Danube catchment was chosen.
The criteria for the selection of test areas for the development and
demonstration of the new application were:
• Variability in water supply due to changing precipitation patterns
leading to more extremes, such as floods and droughts
• Significant portion of irrigated agriculture
• Significant water supply from water storage in the cryosphere
• Interest of Demo Users and Stakeholder
• Strong societal and political linkage
• Future demo area under discussion (Spain! Italy? Germany?)
The Food Security Use Case
Regions of Application

9. © VISTA 2019 www.vista-geo.de No. 9
Regional / Catchment Scale
Local Scale / Field Level
Wasserverfügbarkeit
• Schnee im Gebirge
• Flüsse & (Stau-)Seen
• Trinkwasser
• Bewässerung
Wasserentnahme
Beispiel Donau
817.000 km2
The Food Security Use Case
The Danube Catchment
Basin shared by 20 countries
• Mean annual precipitation:
~750 mm
• Mean outlet discharge (MQ): 6
550 m³/s
• Intensive agricultural
use in lower reaches
• Irrigation (increasing)
• Navigation / Transport
• Hydro Power
• Water Supply

10. © VISTA 2019 www.vista-geo.de No. 10
10
EO Training and Service Development
Danube Catchment
Austrian Crop Type Map

11. © VISTA 2019 www.vista-geo.de No. 11
Tools and Methods
Mapping and Modelling
Existing Crop Type Maps
Long TS of Sentinel 2 images
Large Training
Database Definition
Deep Architecture
for Agricultural
Mapping Crop type map
Crop boundaries
Biophysical
Crop Parameter
Database
Biomass
Plant productivity
Water use
Hydrological
Modelling
‘Water’
Crop Growth
Modelling
‘Plant’
Snow Storage
River / Reservoirs
Water availability

12. © VISTA 2019 www.vista-geo.de No. 12
Large Training Database
Definition - Basics
PreProcessing
Unsupervised Deep
Feature Extraction
Pool of weak
Labeled Samples
PreProcessing
Weak Labeled
Samples Selection
WEAK LABELED SAMPLES EXTRACTION
Unsupervised Deep
Clustering
Time Series of
Sentinel 2 Images
Crop Type Maps
𝑋 𝑅
✓ In computer vision, millions of
annotated images are used for
the training of deep networks
architecture.
✓ In remote sensing such
databases are not available. The
deep architectures proposed for
Sentinel 2 images are typically
trained on a relatively small
number of labeled samples.
𝑋 𝑅
✓ A large database of weak labeled samples will be extracted from uncertain and obsolete crop type maps available
at country level in an unsupervised way. Active learning strategies will be used for optimizing the definition of the
database of labeled samples and integrate it with reliable samples.

13. © VISTA 2019 www.vista-geo.de No. 13
25th March 2018
16th September 201827th August 2018
12nd August 2018
22nd August 2018
07th August 2018
Ground Through
Last Years Map
Forage/Grass
Maize
Oat
Sugarbeet
Potato
Winter Barley
Winter Wheat
Soy
Large Training Database
Definition – Snapshot
Extracting labeled
samples existing
thematic product is
not straightforward:
• they are not
completely reliable
(outdated or not
accurate);
• they are typically
aggregated at polygon
level (polygon labels do
not necessarily
correspond to
spectrally
homogeneous areas)

14. © VISTA 2019 www.vista-geo.de No. 14
14
Map Legend Challenges
Semantic Aggregation
I
D
Extensive Grassland
Intensive Grassland
Maize (Silage)
Forage
Legumes
Maize (Corn)
Oat
Sugarbeet
Potato
Rye
Set Aside
Sugarbeet
Summer Barley
Summer Wheat
Winter Barley
Winter Wheat
Soy
Sunflower
Vegetables (foiled)
Greenhouses
Tree Plantation
Wine Fruits
Flowers/ornamental plants
other crop
useful plant
hemp
20180809
Crop Types
covered for use case & service
(Preliminary)

15. © VISTA 2019 www.vista-geo.de No. 15
15
Map Legend Challenges
Similar Spectral Signature
I
D
Extensive Grassland
Intensive Grassland
Maize (Silage)
Forage
Legumes
Maize (Corn)
Oat
Sugarbeet
Potato
Rye
Set Aside
Sugarbeet
Summer Barley
Summer Wheat
Winter Barley
Winter Wheat
Soy
Sunflower
Vegetables (foiled)
Greenhouses
Tree Plantation
Wine Fruits
Flowers/ornamental plants
other crop
useful plant
hemp
Crop Types
covered for use case & service
(Preliminary)

16. © VISTA 2019 www.vista-geo.de No. 16
Deep Architecture for Agricultural Mapping
✓ High resolution multispectral optical images such as
Sentinel 2 allow for the characterization of the
phenological parameters of different crop types due to
their spectral properties.
✓ Most of the deep architectures for remote sensing focus
on very high resolution (VHR) optical images. The ad-hoc
architectures for Sentinel 2 data are typically trained on
a relatively small number of samples and tested on few
benchmark images without assessing their real
generalization ability.
✓ A deep network architecture tailored to Sentinel 2 images will be defined to:
• take advantage of the spatial, spectral, temporal properties of these data;
• process very dense time series of images to capture the phenological behavior of the different crops;
• take into account possible limitations on the amount of reliable labeled training data.
Example of Sentinel 2 Time Series of Images

17. © VISTA 2019 www.vista-geo.de No. 17
17
Food Security Use Case
Deep Learning Architecture
Driving criteria for the design of the deep architecture that has to be
effective at such large scale for the complex agricultural mapping task:
✓ Focus on the specific spatial, temporal and spectral properties of
Sentinel 2 with the aim of performing agricultural mapping (e.g.,
RNN, ConvLSTM).
✓ Define architecture that may take advantage of:
• Available pre-trained networks;
• Weak labeled data;
• Reliable labeled data.
✓ Optimize the generalization capability of the architectures and the
fine tuning for the analysis of very large heterogeneous areas.
✓ Define a deep architecture being able to work at large scale with
both high accuracy and good generalizations capabilities.
Semi-supervised
Learning
Transfer Learning
Domain Adaptation
Deep Learning
Architectures

18. © VISTA 2019 www.vista-geo.de No. 18
Food Security:
Focus on Water Availability and Irrigation
„ Globally, the major
use of water (70%) is in
the agricultural sector.
In developing countries, whose
income mainly depends on
agricultural products, water
use in agriculture can occupy up to
90 percent of all water withdrawals
(FAO, 2010)“

19. © VISTA 2019 www.vista-geo.de No. 19
Food Security Use Case
User Requirements to Technical Requirements
User Requirement Technical Planning
Water
availability
Basin-wide information layers:
• Soil moisture
• River run-off
• Reservoir conditions
• Current snow storage
Crop
conditions
Basin-wide crop-type specific information layers:
• Crop type map of most widely planted crops
• Leaf area development (LAI)
• Drought stress
• Phenological development (e.g. early ripening)
Irrigation
recommendations
Field-specific delivery for specific demo users:
• Recommendation when and how much to irrigate
• Yield forecast with and without optimized irrigation plan
MAPs
Info
ExtremeEarthUserRequirements
1stUserWorkshopMarch2019inMunich

20. © VISTA 2019 www.vista-geo.de No. 20
Copernicus
Land
Services
Linked
Open
Data
Water Availability Information to Irrigation Recommendation
Modell & EO & DL
PRODUCTS
Soil
Moisture
Snow
Cover
Ground-
water
Run-
Off
ReservoirsLakes
Crop
Type
Crop
Mask
NDVI LAI
Drought
Stress
Phenology
EO
DIAS
Big Data
MODEL
IRR
REC
etc
EO
MAPs
Snow
In-Situ
MODELEO

21. © VISTA 2019 www.vista-geo.de No. 21
Precipitation Soil Moisture
1. Layer
Soil Moisture Soil Moisture
2. Layer 3. Layer
Daily Values
Sep-Oct 2017
Germany
The Food Security Use Case
Water Storage in the soil / high variance
Example of
Water
Availability:
Precipitation
driven soil
moisture
medium
resolution

22. © VISTA 2019 www.vista-geo.de No. 22
Yield (t/ha)
Effect of optimal irrigation
Regional - example Danube
= rainfeed
= Maize
= irrigated
• Irrigation for the catchment area has increased efficiency by 50%
• Corresponds to 30 million tons increase in corn
• Corresponds to 5 billion euros in sales
• Means 5.3 billion m³ of water evaporates
• Excessive impact on the ecology
Great potential to increase the
efficiency of water use through
knowledge and information!!

23. © VISTA 2019 www.vista-geo.de No. 23
Supporting Sustainable Food Production
from Space
New Business Model offer for
private companies
Access to ready-to-use products
or customized services
Access to tools to derive
agricultural and
aquacultural products
Technical support for
platform use
Ability to easily develop new
services, with the ability to
share processors and outputs
only with selected user
groups
Provision on request of high-
accuracy, quality checked
vegetation parameters (LAI,
fAPAR, etc), suitable for use in
operational scenarios.
Interact with a range of users
through a dedicated forum
Access to key satellite products
and ancillary data, backed up by a
scalable processing infrastructure

24. © VISTA 2019 www.vista-geo.de No. 24
Building blocks of data, resources, tools & services

25. © VISTA 2019 www.vista-geo.de No. 25
Processing
power
Mobile visualization of
biophysical parameters in the field
Customized
products & services
Satellite data
(mainly Copernicus)
Open expert interface
User generated results
knowledge &
algorithms
tools
Service
provider
User data
Toolbox
{api}
User input
User data
The Food Security TEP 2019

26. © VISTA 2019 www.vista-geo.de No. 26
Main platform functionality
The Food Security TEP main platform functionalities:
➢ Exploring EO data, products & user data as well as applications &
processing services
➢ Developing your own services as Docker applications
➢ Managing & Sharing your data, services & jobs
➢ Visualization of products in the Analyst
➢ Account management

27. © VISTA 2019 www.vista-geo.de No. 27
Complementary data for analysis
Upcoming:
• HWSD and ESDB Soil maps
• Population data
• Precipitation data documentation

28. © VISTA 2019 www.vista-geo.de No. 28
Apps / toolboxes on Food Security TEP
Run your scripts in
parallel mode!

29. © VISTA 2019 www.vista-geo.de No. 29
Devloping new services

30. © VISTA 2019 www.vista-geo.de No. 30
Food Security TEP Analyst View:
Browser based visualization
Green leaf area of agricultural areas near Berlin, Germany showing the decrease of
plant health during the summer drought 2018

31. © VISTA 2019 www.vista-geo.de No. 31
Specific EO product collections:
Coastal aquaculture example

32. © VISTA 2019 www.vista-geo.de No. 32
fAPAR, calculated from Sent-2 on FS-TEP
Temperature – Rainfall – fAPAR: time profile for a specific potato field
Rainfall, from KNMI
Customized Frontends:
Potato monitoring example

33. © VISTA 2019 www.vista-geo.de No. 33
Food Security Use Case
Messages
✓ EE will strengthen the TEPs by
providing Deep Learning to
Polar TEP and Food Security TEP
✓ Crop type monitoring will be
available as a front end service
after training activities
✓ Customized services for water
availability mapping potentially
applicable globally
✓ The Food Security TEP is open to
additional public or customized
services
✓ Different business model (including
access to ESAs EO Network of
Resources and H2020 OCRE)
✓ Enlarges your customer base and
boost interactions with the
agriculture and aquaculture user
community