Recommended
Recommended
More Related Content
Similar to presentation_eCSAAP_WorldCIST2021.pptx
Similar to presentation_eCSAAP_WorldCIST2021.pptx (20)
Recently uploaded
Recently uploaded (20)
presentation_eCSAAP_WorldCIST2021.pptx
- 1. Using Expert Crowdsourcing to Annotate Extreme Weather Events WorldCIST 2021 Dennis Paulino, António Correia, João Barroso, Margarida Liberato, Hugo Paredes
- 2. Introduction Background Architecture for Crowd Annotation of Extreme Weather Events Discussion Final Remarks 2
- 3. 3 Introduction • Global warming accentuates the probability of having more unprecedented Extreme Weather Events (EWE) • Such extreme meteorological events can have significant socio-economic impacts, increasing agriculture production shortfalls, economic losses and human mortality • The use of data science approaches to analyze EWE has an utterly importance to detect and prevent potential hazards
- 4. 4 Introduction • Through a human-in-the-loop data integration approach, it is possible to transform heterogeneous datasets generated by automated methods into a unified view by expert crowds • However, gathering labeled data is a difficult process for data science researchers since the evolution of EWE • Crowdsourcing is defined as an online activity where a person or organization proposes the elaboration of a task to a group of individuals • In our work, we propose a full stack architecture for weather annotation using crowdsourcing
- 5. 5 Background • In the last few years, many studies have already reported the usage of crowdsourcing for evaluating weather phenomena • Most studies focused on providing online surveys or mobile apps to collect data on the impacts of the global climate change • Findings of a study conducted by Hennon et al. (2015), offer several interesting insights and implications for supporting the changing stages of cyclones Hennon, C.C., Knapp, K.R., Schreck, C.J., III., Stevens, S.E., Kossin, J.P., Thorne, P.W., Hennon, P.A., Kruk, M.C., Rennie, J., Gadéa, J.-M.: Cyclone center: can citizen scientists improve tropical cyclone intensity records? Bull. Am. Meteor. Soc. 96(4), 591–607 (2015)
- 6. 6 eCSAAP Architecture for Crowd Annotation of Extreme Weather Events • The eCSAAP Architecture for Crowd Annotation of Extreme Weather Events (eCSAAP-ACAEWE) has the purpose of solving the problems of weather annotation • eCSAAP-ACAEWE is built on four vectors: • Crowd • Task • Process • Evaluation
- 7. 7 eCSAAP Architecture for Crowd Annotation of Extreme Weather Events
- 8. 8 eCSAAP Architecture for Crowd Annotation of Extreme Weather Events – Visualization and Annotation
- 9. 9 eCSAAP Architecture for Crowd Annotation of Extreme Weather Events – eCSAAP Runtime • eCSAAP runtime was designed for supporting weather experts in the creation of weather maps • The runtime environment follows a microservices architecture model • The microservices provide the required functionalities for the management of weather maps through Django • The end user layer is ensured by a frontend (developed using React) that consumes the microservices for the creation of a user-friendly web interface for end users’ interaction.
- 10. 10 eCSAAP Architecture for Crowd Annotation of Extreme Weather Events – eCSAAP Runtime
- 11. 11 eCSAAP Architecture for Crowd Annotation of Extreme Weather Events –Task Presenter (Crowdsourcing)
- 12. 12 Discussion • The eCSAAP-ACAEWE defines a set of components and their respective communication flow for collecting the annotations of weather experts on EWE • The eCSAAP runtime allows a user to generate visualization maps about a specific weather event • Harnessing human-in-the-loop features in EWE algorithms constitutes a requirement and an opportunity for evolving current solutions on the study of these phenomena
- 13. 13 Final Remarks • The current trend of unprecedented extreme weather events brings more hazardous damages worldwide • In this article, the feasibility of an architecture supporting the annotation process from crowd workers using weather maps was demonstrated • For future work, the task presenter should be tests with crowd workers and it is necessary to visualize and analyze the crowd workers’ outputs
- 14. Using Expert Crowdsourcing to Annotate Extreme Weather Events WorldCIST 2021 Dennis Paulino, António Correia, João Barroso, Margarida Liberato, Hugo Paredes
Editor's Notes
- Here it is presented the outline of this presentation.
- 1 – (After reading the Bullet Item) Such as heat waves or precipitation extremes. As a result, climate change can be noticed in different regions worldwide suffering the effects of Extreme Weather Events. 2- (After reading the Bullet Item) In recent years, there has been an increasing tendency in the number of extreme events, becoming more unpredictable and bringing hazardous consequences. 3- (Before reading the Bullet Item) There is a rising need to make hazardous weather detection systems and their decisions more accurate when large and complex phenomena are involved. … The use of data science…
- 1 – (After reading the Bullet Item) … who examine and interpret raw sources. 2- (After reading the Bullet Item) … like droughts, cyclones or severe storms is not yet completely understood. Many prediction systems, including state-of-art climate models, face difficulties when dealing with unprecedent weather events. In addition, the current methods for classifying Extreme Weather Events are based on subjective criteria, which makes it difficult to apply feasible detection algorithms. 3- (After reading the Bullet Item) Using crowdsourcing to collect weather annotations from experts could be a reliable approach to classify EWE. 4- (After reading the Bullet Item) This approach provides an integrated environment as a technological solution for allowing the development of automatic methods for Extreme Weather Events prediction.
- 1 – (After reading the Bullet Item) This can be accomplished by asking volunteers to provide or annotate weather data. 2- (After reading the Bullet Item) These approaches focused either on identifying current weather data or on the aftermath of weather events, disregarding the way in which Extreme Weather Events could be detected. 3- (After reading the Bullet Item) However, it does not focus on detecting how is their nature, which is the ultimate goal for identifying these extreme events.
- 1 – (After reading the Bullet Item) … through a service-oriented approach where each functionality is designed in a modular and interoperable Manner 2 – (After reading the Bullet Item) Crowd: the target are weather experts that will perform tasks; • Task: the work unit(s) that will be outsourced to the crowd, derived from the problem statement. The main task relies on annotating a weather map to answering a specific question. To do this, a crowd worker must select in the map the area where it seems that is happening a hurricane; • Process: the flow of information, interactions and control, including collaboration dynamics; • Evaluation: judge the quality of the work done taking into account the objective of the tasks.
- 1 – This figure presents the workflow of the eCSAAP-ACAEWE, which is based on an end-to-end approach. This has in one end the meteorological and oceanographic data that can be gathered from weather centers APIs (e.g., European Centre for Medium-Range Weather Forecasts – ECMWF) and, at the other end, the creation of crowdsourcing tasks based on the weather maps generated. In addition, the eCSAAP-ACAEWE allows users to create weather maps using a vast set of criteria such as weather metrics (e.g., wind, temperature), date range, domain (i.e., area of the weather phenomena), and visualization scale. After generating the weather map, it can be exported to a crowdsourcing platform in the form of microtask where crowd workers are able to annotate a specific phenomenon, like in the case of an EWE. Accordingly, the creator of a microtask can import the results to the eCSAAP-ACAEWE and validate or visualize the annotations added.
- 1 - Activity diagram of the crowdsourcing task presenter on the eCSAAP-ACAEWE. 2- The task presenter starts by loading the weather data from the visualization data collected on the eCSAAP runtime. As this activity relies on external dependencies, the system must warn the crowd worker that the task could not be loaded and end it immediately. After the visualization data is loaded, a dynamic map is displayed to the crowd worker by enabling to choose the weather annotation. Moreover, it is also important to note that the crowd worker may choose to end the task at any time.
- 1 – (After reading the Bullet Item) … creating visualization data extracted from 2 dimensions images 2- (After reading the Bullet Item) … ensuring interoperability, scalability, and manageability 3- (After reading the Bullet Item) … Python based web development framework. Python was selected due to the migration of the existing climate research libraries and the map libraries that facilitate the generation of visualization maps (e.g., Matplotlib). 4- (After reading the Bullet Item)… The frontend was developed using React for enhanced interoperability with the provided APIs. These weather maps ingest data through a publicly available API for weather data reanalysis (it was used the ECMWF).
- 1 – Weather map page showing a storm with the weather metrics of geopotential (contour lines) and the potential vorticity (contour filled). This map helped identifying a cyclone with severe impact approaching the Iberian Peninsula. When we look at the interface, there is also a button (at the top right) for creating a crowdsourcing task associated to the weather map
- 1 – Performing a weather task for weather annotation in PYBOSSA (left: selection of the annotation area; right: filter the weather values’ range). In order to implement the crowdsourcing approach of the proposed architecture, it is necessary that the crowd workers execute the tasks in a web platform. To this end, it was chosen the PYBOSSA6 platform, an open-source crowdsourcing platform for presenting the tasks to the crowd workers. This platform provides flexibility on the development of the task presenter, which is important to enhance the interface showed to the crowd worker. Additionally, PYBOSSA also comes with an API that allows the integration with the eCSAAP runtime for managing the tasks and importing the results. The task presenter is comprised with a map using the Javascript Library Leaflet, which allows the creation of interactive maps. The overlays displayed in the interactive map represent the visualization data obtained from the weather map.
- 1 – (After reading the Bullet Item) This architecture delineates activities that range from collecting data of weather centers to gathering weather experts’ annotations. A proposal for the implementation of this architecture consists on a system that generates visualization data about atmospheric phenomena and a tool for annotating interactive maps 2- (After reading the Bullet Item) During this process, the user only needs to define few parameters such as area, date, or weather metrics. With the generated maps, tasks can be created and the weather experts can collaborate on the identification of EWE. 3- (After reading the Bullet Item) We introduce a methodology for bringing the expertise of collaboration research to climate experts, by allowing a simplified creation of crowd campaigns. In particular, the eCSAAP-ACAEWE extends the existing abilities of EWE tools by allowing the semantic annotation of data into an integrated environment
- 1 – (After reading the Bullet Item) Technology has an important role in detecting these extreme events, which benefits from the annotated weather data. The annotation can be done using crowdsourcing to present crowd workers these extreme weather events 2- (After reading the Bullet Item) Such approach was envisioned taking into consideration an end-to-end approach based on producing weather maps from weather centers data and by collecting the crowd worker opinions from the visualization of that data. 3- (After reading the Bullet Item) … which are necessary to guarantee the quality of the annotations.