Pulmonary drug delivery system M.pharm -2nd sem P'ceutics
SemEval-2018 task 5: Counting events and participants in the long tail
1. Counting Events and Participants within
Highly Ambiguous Data covering a very long tail
SemEval-2018 Task 5
Marten Postma, Filip Ilievski, Piek Vossen
{m.c.postma, f.ilievski, piek.vossen}@vu.nl
5. Subtasks
● Subtask S1: Event Questions with Answer=1
○ Which killing incident happened in 2014 in Columbus, OH?
● Subtask S2 Event Questions with Answer=any number
○ How many killing incidents happened in 2016 in Columbus, MS?
● Subtask S3 Participant Questions with Answer=any number
○ How many people were killed in 2016 in Columbus, MS?
Optionally, participants could also provide the text mentions of events in the
documents.
6. Data
3 Domains: gun violence, fire
disasters, business.
The data consists of local news
articles and reports on small-world
events and participants.
The data is split into trial and
test parts.
7. Evaluation
1. Incident-level evaluation – Did you get the question right?
a. Accuracy (exact answer matching)
b. RMSE
2. Document-level evaluation – How many of the gold documents did you
retrieve?
a. Precision, recall and F1-value
3. Mention-level evaluation – Did you extract the correct coreference chains?
a. BLANC, CEAF_E, CEAF_M, MUC, BCUB
8. Participating systems
Task 5 had four participating systems.
NewsReader and ID-DE built a knowledge graph of incidents, which was then
queried for each question based on its constraints.
NAI-SEA performed clustering on a document level. FEUP applied a supervised
approach to address participants and locations separately.
All systems aided their clustering by extracting temporal expressions, word
senses, participants, locations, semantic roles, ...