Phidias: Steps forward in detection and identification of anomalous atmospheric events

The PHIDIAS project has received funding from the European Union's Connecting Europe Facility under grant agreement n° INEA/CEF/ICT/A2018/1810854.
PHIDIAS: Boosting the use of cloud
services for marine data management,
services and processing
Webinar | October 13, 2020, 15:00 CEST

Objectives
213.10.2020 PHIDIAS Webinar | 13.10.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc
Develop a catalogue that will allow users to discover and access data, open-source software, public
Application Programming Interfaces (APIs) and interactive processing services.
Optimise and industrialize workflows to allow the largest degree of reusability of data as possible.
Implement an end-user web common interactive processing service based on notebook and datacube
technologies allowing new users to easily have open access to HPC capacities and develop new
algorithms.
Improve the FAIRisation of satellite and environmental datasets and preserve FAIR (Findable
Accessible Interoperable Reusable) datasets into a Remote Data Access (RDA) certified repository.
Develop new data pre-processing models coupled with HPC capabilities, by building a new and
innovative on-the-fly computing service for smart processing of data, addressing the problem of
efficient filtering and on-the-fly first diagnostics on incoming in-situ data.
Deploy data post-processing methods as a service for several end-users, including scientific
communities, public authorities, private entities and citizen scientists.
Consortium of 13 committed partners from 5 different EU countries, led by CINES (French national
Computing Centre for Higher Education)

Webinar Series
Next Webinar will be on «Big Data Earth Observation» Use Case –
All recorded Webinars are available on Phidias HPC website under the Webinar page
https://www.phidias-hpc.eu/events/webinars/

Webinar Agenda
15:00 - 15:05 - Introduction of PHIDIAS project
Francesco Osimanti, Trust-IT Services, PHIDIAS WP7 Leader
15:05 - 15:15 - Thematic and objective: proposed approaches and available Sentinel
measurements
Pascal Prunet, SPASCIA & PHIDIAS WP4 Leader
15:15 - 15:25 - HPC/HPDA Infrastrucuture and capacities for implementing intelligent
screening in an operational context
Nicolas Pascal, ICARE & SPASCIA Technical Partner
15:25 - 15:30 - Q&A Session
15:30 - 15:40 - Implementation of processing and products
Dominique Jolivet, HYGEOS & SPASCIA Technical Partner
15:40 - 15:50 - ESCAPE: A dive into a Datalake for Open Science
Xavier Espinal, CERN & ESCAPE project
15:50 - 16:00 - Q&A and Closing Remarks
13.10.2020 PHIDIAS Webinar | 13.10.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc 4

PHIDIAS: Steps forward in detection
and identification of anomalous
atmospheric events from space
Pascal PRUNET, CEO SPASCIA
SPASCIA

6
Earth and environment challenges at local, regional and global scales : continuous increase of
human activities modify the atmospheric composition
Impact on environment (climate; soil, water and air quality; biodiversity)
Impact on health, on economy
Needs to understand and to monitor for acting
European answers for better adressing these challenges : Europe is entering in the era of
operational measurement of the air composition from space, for the analysis and forecasting of chemical
weather and climate monitoring
CONTEXT
First measurements already available : IASI, GOME (Metop) Sentinel 5 precursor (S5P/TROPOMI)
4 Operational sounding instruments from 2022-2023 : Sentinel 4 and 5 onboard Metop-NG and MTG
Copernicus Operational forecasting services for atmospheric chemistry (CAMS) and Climate (C3S)
Development of the European capacity for the measurement of GHG for monitoring human emissions

7
From 2022, European atmospheric sounding missions will deliver each day several TB (terabytes) of raw
datacubes at high spatial/temporal/spectral resolutions. This represents an unprecedented amount of atmospheric
data, with improved quality and coverage.
How to comprehensively deal with all available information ?
➢ Key challenge is to provide the capacity of intelligent screening of large amounts of satellite data for targeting scenes
or events of interest in view of their dedicated processing or exploitation.
PHIDIAS addresses those needs by using HPC and HPDA capacities of intelligent screening approaches in an
operational context, for detection and identification of atmospheric composition events.
.
CONTEXT
Onboard geostationary MTG satellite,
the Sentinel-4 mission comprises an UVN
instrument and data InfraRed Sounder
(IRS), completed by FCI
Onboard polar-orbiting MetOp SG satellite,
the Sentinel-5 mission comprises an UVNS
instrument and data from IASI-NG, MetImage
and 3MI
Each 30 minutes
over Europe
Each day over
the globe

8
PHIDIAS ADDED VALUE
Earth observation from space is underexploited
Huge quantity of data, increasing information content and
complexity
Operational forecasting systems
Efficient, real-time data assimilation in models,
but could not use « extreme » data related with
« extreme events »
Reduced added value of the data
Research groups, scientific institutes
Innovative and well adapted to event studies
but case by case exploitation of the data
Not exhaustive exploitation,
no real-time analysis
Detection and
Filtering in real time
of relevant data
Realtime Detection &
analyse of
extreme events, pollutant
plumes, other targeted
events
Dedicated services
Early warning, monitoring, decision support
Added-value :
Robust detection,
objective filtering
Added-value :
Verification,
improvement
Innovation

9
HOW ?
Earth observation from space
Sentinel 5 Precursor (S5P) L1B, L2
PHIDIAS Pilot users :
On the flow, realtime
detection & analysis of
anomalous events
Complementary dedicated services
Early warning, monitoring, decision
support
PCA-based screening of
L1B data
AI-based Detection of gas plume
and sources from L2 products
On demand processing
of the data for detection
of plumes and sources
over a region/period
specified by the user
Catalogue/Metadata of
selected and
characterised L1B data
Processed L2 products
with additional
characteristics
L1B data L2 product

10
This use case propose to develop, test and prototype the approach with Sentinel 5
Precursor data/products : Level 1 data …
DATA

11
… and Level 2 products
DATA

12
WP4 WORK PLAN
Objective : intelligent screening of large amount of satellite data for detection and identification of anomalous atmospheric
composition events
Two processing prototypes are addressed proposed for development and test :
1. PCA-based screening of L1 data (SWIR) for detection of extreme events. Based on experience and methods developed for IASI,
implementation and consolidation of algorithms and tools for generic processing of atmospheric spectra recorded by S5P.
Real-time, systematic processing of data
2. New AI methods for objective/automatic detection of plumes from L2 products (CO, CH4 NO2, SO2): dedicated AI methods based
on Particle Swarm Optimisation approaches and exploiting similitude with animal behavior modelling -> comprehensive
exploitation of spatial and temporal information for detecting plumes and sources.
On demand processing : historical data or real time processing

13
1. PCA-based screening of L1 data – Example of IASI
Nominal
case:
denoising
filter
Extreme
case:
identification
of the
residual
signal
Reconstruction of
"scores" maps:
rapid detection of
anomalous situations
Analysis of "residuals" :
interpretation of the
anomalous event

14
1. PCA-based screening of L1 data (SWIR) for detection of extreme events.
Status:
➢ 1 year, global scale of S5P Level 1 (L1) data available at ICARE
➢ PCA-screening processing under development (adapted from IASI-PCA developed with CNES)
▪ Processing already tested on IASI real data
▪ Development/adaptation at HYGEOS is ongoing
➢ Analysis and pre-processing of L1 ShortWave InfraRed (SWIR) spectra (bands 7 and 8) is ongoing, in order to be
ingested by the PCA processing
▪ Issue: complex but necessary filtering of bad pixels for exploiting the whole spectrum
▪ Analysis of spectral micro-windows: indicators for detection of pollutants (air quality, greenhouse gases)
Next steps : first tests of PCA processing on small subsets of S5P data; generation of global scale database for the learning
phase; implementation at ICARE for test and prototyping
WORK PROGRESS

15
2. New AI methods for objective/automatic detection of plumes from L2 products (CO, others?: CH4 NO2, SO2)
Status:
➢ Global S5P Level 2 products since 2018 available at ICARE
➢ Preparation and analysis of S5P Level 2 product (NO2)
▪ Analysis of image over pollution sources (see side results in the next slide)
▪ Implementation of dedicated storage of the product, optimised for speedy retrievals given {geolocation, time}
constraints over the whole dataset (2018 - present)
➢ Development of the AI engine for source and plume detection
▪ Neural network agents, stochastic optimization process
▪ Selection of data subsets for the learning phase
➢ Alternative method analyzing significant signal enhancements
Next steps : learning phase and tests on S5P data subsets
WORK PROGRESS

16
Project objectives
ANALYSIS OF S5P PRODUCT OF NO2 FOR DETECTING POLLUTION LEVEL AT CITY SCALE
P. PRUNET, O. LEZEAUX, C CAMY-PEYRET, H. THEVENON, ACCEPTED IN CITY AND ENVIRONMENT INTERACTIONS
➢ From ESA S5P measurements of the air pollutant NO2 from space, we have assessed the impact of the human activity
reduction on air pollution by comparing the first 4-months periods of 2019 and 2020 on a daily, weekly and monthly basis
for 4 major cities in Europe PARIS
2019
monthly
averages
MILAN MADRID ATHENS
2020
Monthly
averages
Reductions in the pollution level (using NO2 tropospheric
column as a proxy) have been observed from Mid March and
for April 2020 (52% +/-9% for Paris; 28% +/-8% for Milan
region; 54% +/-16% for Madrid; not significantly observed for
Athens), as compared to the same periods in 2019
PARIS : Reduction rates of the NO2 plume mass fortnightly mean
between 2019 and 2020 from mid-March to end of April, and
✓ Start dealing with the data
✓ Understand the information
✓ Show the added-value of S5P

17
Space-based Earth observation will provide amazing amount of complex, informative data : How
to extract and use comprehensively this increasing information ?
Operational Models and systems have strong limitation to exploit « extreme » data, i.e., too far from the model
Research face to the necessity to focus their analyses of the underlined information and science
PHIDIAS will provide new tools and approaches to deal with the measurements and the
information, allowing better and more efficient use of the data, and pave the ways for finding
new paradigms
AIM
XCO2 IFS model simulation
provided by ECMWF
(from H2020 ECMWF CHE project)
:

PHIDIAS: Steps forward in detection
and identification of anomalous
atmospheric events
Nicolas Pascal, AERIS/ICARE technical director

AERIS/ICARE ?
A Data and Services Centre mostly dedicated
to satellite data, but also ground-based and
model data
Part of the Data Terra Research Infrastructure
• 6,7 PB of usable storage capacity,
• ~1200 cores dedicated to production,
• 10 Gb bidirectional connection to
RENATER.

Towards a interoperable infrastructure
Credits: R. Moreno
sedoo

Why CINES and AERIS/ICARE ?
Convergence needed for HPDA algorithms: processing of big data
volume with big computing resources
CINES: access to important computing resources, experts in HPC and data
sharing techniques.
AERIS/ICARE: access to big volume satellite database, close to the
atmosphere science community, experienced in science prototypes
industrialization.

High level implementation scheme
S5-P data
~500 GB/day
Data
sharing/caching
Trigger process
Get result data
Users
Trigger process
Get result data
Systematic
processing
On-demand
processing
W
e
b
P
o
r
t
a
l
Users
Data discovery
Get result data
Data users

PHIDIAS WP4 : Technical
implementation
Dominique Jolivet, HYGEOS

OUTLINE
State of the art : IASI-PCA
PCA processing of IASI level 1 data already tested (CNES, SPASCIA,
HYGEOS, LATMOS)
Existing approach and methodology
Promising results
On going work : implementation for Sentinel 5P :
Generation of a reference basis
L1B reader for S5P data
PCA analysis
Challenges with S5P L1 data

PROCESSING SCHEME
13.10.2020 25PHIDIAS Webinar | 13.10.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc

26
CO
NH3
This method has been
successfully applied to IASI on
Metop (SPASCIA, HYGEOS,
LATMOS collaboration, CNES
funding)
More than 60 indicators
(scores) of extreme
atmospheric events (concerning
more than 10 molecules: CO,
NH3, CH4, SO2, HCl, ….).
IASI experience
Detection of fire events : exemple of IASI, 10/2017

S5P - Reference basis
Has to be done once. Can be seen as an ancillary data
Composed of more than 75 000 spectra random – oriented selection
Covering a long temporal period (typically one year): day 1, 6, 11, 16, 21 and 26 of each
month are selected
About 75 spectra per orbit: orientated random selection to extract a representative set of :
Scanline and ground pixel (IFOV) number
Solar and viewing zenith angles (SZA)
Latitude/longitude

S5P-L1B reader
13.10.2020 28
For one scanline
Spectral channel
Ground
pixel
Read L1B data
measurements
(scanline x ground pixel x
spectrum)
And
Associated flags
For one ground pixel into a
scanline line (so for one
spectrum):
Detect and correct
anomalous measurements
Filter if neccessary
(use or not)
And normalization
Temporary
output: filtered
normalized
spectra
Challenging for
S5P
PHIDIAS Webinar | 13.10.2020 | https://www.phidias-hpc.eu/ | @PhidiasHpc

PCA

S5P CHALLENGES (1)
Code written in python (core API)
Processing at the scale of an orbit (divided into chunks – arrays of data)
S5P versus IASI: much more input data (but only SWIR bands BD7 and BD8),
and only daytime data will be used
S5P L1 data versus IASI
Dealing with noise → the noise is radiance dependent hence scene dependent (different from IASI)
Normalization → the instrument covariance matrix is to be established (more complicated than for
IASI)
Spectral indicators → to be defined (as for IASI) searching for the most appropriate spectral
signatures of CH4 (first priority) and CO (second priority, since strongly perturbed by CH4 features).
H2O features are also to be considered because of their ubiquity and large variability. Other species
will be searched for in the discovery or extreme event mode.
Day/night transition → since the recorded signal is the solar radiation reflected by the
surface/atmosphere system, low sun scenes will be filtered out (threshold for high solar zenith angles,
SZA)
Does a huge amount of input data mean a huge amount of output data ?

S5P CHALLENGES (2)
Main output: logbook of atmospheric events detected (netcdf file). How to be
efficient and useful for users and researchers ?
Where is the L1B data: L1B filename (attribute), scanline, ground pixel
Ground pixel quality flag (from L1B)
Spatial geolocation: latitude/longitude
Which indicators will provide an alert (detection flag coded in 8 or 16 bytes depending on the
number of indicators)
Values of the indicators (mean and standard deviation of the residual difference)
Other outputs (to be investigated and validated by ICARE):
Maps
Post-processing for the whole day
Tools to link to L1B or L2 data corresponding to detected events ?
Additional difficulties: bad pixels always present in 2D SWIR focal arrays
Need to establish the map of bad pixels in the spatial (ifov) and spectral (channel) plane using solar
calibration views (onboard diffuser plate) acquired in the polar region for one orbit per day

Xavier Espinal (CERN) - ESCAPE WP2 leader
ESCAPE - A dive into a Datalake for
Open Science
Webinar - Steps forward in detection and identification of anomalous atmospheric events 13 Oct 2020

- Contributes to deliver
Open Access and
FAIR data services:
trustable data
repositories; enable
data management
policies; transparent
data access layer
The ESCAPE Data Infrastructure for Open
Science
- Define, integrate
and commission an
ecosystem of tools
and services to build
a data lake
- Science projects to
drive the services
requirements most
suitable to their needs
PHIDIAS webinar 13/10/20 Steps forward in detection and identification of anomalous atmospheric events

37

38

39

40

41

Thank-you
13.02.2020 PHIDIAS Webinar | 13.02.2020 | www.phidias-hpc.eu | @PhidiasHpc 42

Phidias: Steps forward in detection and identification of anomalous atmospheric events

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Phidias: Steps forward in detection and identification of anomalous atmospheric events

Similar a Phidias: Steps forward in detection and identification of anomalous atmospheric events (20)

Último

Último (20)

Phidias: Steps forward in detection and identification of anomalous atmospheric events