The talk discuss about the architecture of a BIG DATA system able to ingest and enrich social media data to quickly perform complex data analysis. The proposed software uses Apache Kafka as distributed messaging system, Apache Spark as general purpose data processor and Elasticsearch/Kibana as data storage/search engine/data analytics platform. This work was presented at "Elastic In a Day", 6 june 2017, Milan, Italy (http://www.seacom.it/elastic-stack-in-a-day-2017/)

5. •
•
•
•
•
•
•
•
•
•

6. •
•
•
•
•
•
•

7. •
•
•
•
•
•

8. Indexing
Social media data sources
Kafka
Indexer
ES
EnricherDP_nDP_2DP_1 ...
Processing Enrichment
Kibana/other GUIs

9. •
•
•
•
•

10. •
•
•
•
•

11. Extract of JSON data package metainformation:
“meta” : {
“type”: “twitter”,
“index”: “tweet_index”,
“row_id”: “434”,
“analysis”: [
{
“module_id”:”language_detection”,
“contracts”: [
{
“inputTextField”: “tweet.text”,
“outputLanguageField”: “tweet.lang”
}
]
}
]
}

12. •
•
•
•
•
•
•
•
•
•
•
•
•
•

13. •
•
•
•
•
•
•
•
•

14. •
•
•
•
•
•
•
•
•
•
•

15. •
•
•
•
•
•
•
•
•

18. https://github.com/marghe943/kibanaChoroplethMap

19. •
•
•
•
•
•
•