SlideShare a Scribd company logo

SFScon14: The OSS-Mailcluster of Raiffeisen OnLine GmbH

•
•
South Tyrol Free Software Conference
South Tyrol Free Software Conference

Urban Lösch talk about "The OSS-Mailcluster of Raiffeisen OnLine GmbH" during the SFScon14

Technology
1 of 17
Download to read offline
18.11.2014 Raiffeisen OnLine‘s OSS mail cluster
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Purpose + requirements •implementation of a new spam filter system •replacement of the present Windows solution (3 servers) •redundant setup •Active/Active preferred •easily scalable and flexible •give our “Service Center” access to •account configuration setting •log data for customer support purposes •should allow further expansion in order to implement new features and improve hit rate
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Which solution should we use? •Evaluation of manufacturers •major differences in price/performance ratio •not all solutions grant actual redundancy (only standby setup) •Test installation by a manufacturer works OK … •… until Friday afternoon, when a sudden deadlock occurs! •After having spent 3 days searching for the error, the manufacturer states that: •…there must be a power outage in the datacenter -> I’m sure there was no outage 
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 We finally decided to try out a self-made solution •But why OSS? •The experience told us it’s preferable to adopt a solution on which we are able to investigate by ourselves in an emergency situation. •We were aiming at a solution that would allow us to meet new requirements on our own in the future. •We wanted to be able to develop own new components if necessary. •Our objective was to reduce dependency on one single manufacturer.
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 We found all components in the open source world very quickly •First implementation in the middle of 2008: •Postfix (MTA) -> on the front line •Spamassassin + ClamAV -> as content filter •AmaVis -> interface between postfix + content filter •MySQL –> configuration + quarantine (Spam/Virus) •MySQL -> central logging for “ServiceCenter” + customers •PHP -> administration front end for “ServiceCenter” + customers
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 First phase of implementation •2x bare metal servers (8GB RAM, QuadCore, SAS) •„extreme“ container-virtualization •Each service had its own container •2 Postfix front end MTAs •2 AmaVis (Spamassassin, ClamAV) content filters •2 MySQL Master/Master Replications (configuration) •2 MySQL Master/Master Replications (quarantine) •2 MySQL Master/Master Replications (logdatabase) •2 DNS Resolvers •2 Postfix Backend MTAs (delivering e-mails to storage back ends)
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Each node could go down at any time •Redundant SMTP paths IN ---> Postfix---> AmaVis---> Postfix---> OUT
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 What are the main problems? •OSS components are available in heaps, but it was difficult to •have all different components work together •process large amounts of log data in order for users to understand them •We had not much experience with high mail traffic. •It took us some time to identify the correct performance and configuration parameters.
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Technical support •Google  •knows about 90% of all the problems we came across •Community •We posted our own questions to mailing lists and forums. •We kept reading newsgroups. •Common sense •logical + analytical approach •“Don’t worry, it’s just numbers and letters.”* * cit.: Thomas Gelf
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Continuous enhancements in the last years •Perl -> small policy daemons we developed in 2012 and 2013 •selective grey listing •performance improvements, storing non important data into the RAM •ClamAv content filter replaced(July 2014) •by ClamAv-Milter: scans viruses directly on front end servers and rejects with 550 in real time instead of storing in quarantine •SPF-Policyd (Juli 2014) •supports Sender Policy Framework in either direction (In/Out) •OpendDKIM (Juli 2014) •supports Domain Key Identified Mail in either direction (In/Out) •OpenDMARC (Juli 2014) •supports Domain-based Message Authentication, Reporting & Conformance in either direction (In/Out)
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Current mail traffic (Postfix 1)
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Current mail traffic (Postfix 2, Content filter)
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Some more highlights: •SSL/TLS availableon all front end servers •IPv6 •active on all front end servers in both directions •(partial) communication between components •now facing IMAP/POP3 + SMTP submission •IMAP + POP3 + SMTP failover •front end displays processed log data for „ServiceCenter“ and customers
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 IMAP + POP3 failover •outward: virtialIP + Multicast MAC •basedon IPTablesCluster-IP Module •[imap,pop3].rolmail.net (Dovecot) •smtp.rolmail.net (Postfix) •2 proxyservers •imap1.rolmail.net •imap2.rolmail.net •morepossible •Mailstore: •manybackend servers(Dovecot) •Load balancingofI/O
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Processed data for „ServiceCenter“ and customers
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Future plans •to force SSL/TLS for IMAP/POP3 + SMTP submission clients •to protect mail traffic with DNS-based Authentication of Named Entities (called DANE) •protection of zones with DNSSEC is needed •to improve quarantine mechanism •rejecting all e-mails tagged as spam directly on the frond end servers with 550 -> no mail will be stored in quarantine •maybe to implement the mailstore backend server pool as an active/active setup
Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 ### It was a pleasure to be here ### Many thanks for your attention! Urban Lösch Raiffeisen OnLine GmbH urban.loesch@raiffeisen.net

Recommended

Fast Online Access to Massive Offline Data - SECR 2016
Fast Online Access to Massive Offline Data - SECR 2016Fast Online Access to Massive Offline Data - SECR 2016
Fast Online Access to Massive Offline Data - SECR 2016Felix GV
 
Introducing Venice - Strata NYC 2017
Introducing Venice - Strata NYC 2017Introducing Venice - Strata NYC 2017
Introducing Venice - Strata NYC 2017Felix GV
 
Kafka Connect by Datio
Kafka Connect by DatioKafka Connect by Datio
Kafka Connect by DatioDatio Big Data
 
Architecture Sustaining LINE Sticker services
Architecture Sustaining LINE Sticker servicesArchitecture Sustaining LINE Sticker services
Architecture Sustaining LINE Sticker servicesLINE Corporation
 
Joe witt may2015_kafka_nyc_apachenifi-overview
Joe witt may2015_kafka_nyc_apachenifi-overviewJoe witt may2015_kafka_nyc_apachenifi-overview
Joe witt may2015_kafka_nyc_apachenifi-overviewJoseph Witt
 
Confluent building a real-time streaming platform using kafka streams and k...
Confluent building a real-time streaming platform using kafka streams and k...Confluent building a real-time streaming platform using kafka streams and k...
Confluent building a real-time streaming platform using kafka streams and k...Thomas Alex
 
Introduction to Kafka connect
Introduction to Kafka connectIntroduction to Kafka connect
Introduction to Kafka connectKnoldus Inc.
 
(ATS6-PLAT09) Deploying Applications on load balanced AEP servers for high av...
(ATS6-PLAT09) Deploying Applications on load balanced AEP servers for high av...(ATS6-PLAT09) Deploying Applications on load balanced AEP servers for high av...
(ATS6-PLAT09) Deploying Applications on load balanced AEP servers for high av...BIOVIA
 

Recommended

Fast Online Access to Massive Offline Data - SECR 2016
Fast Online Access to Massive Offline Data - SECR 2016Fast Online Access to Massive Offline Data - SECR 2016
Fast Online Access to Massive Offline Data - SECR 2016Felix GV
 
Introducing Venice - Strata NYC 2017
Introducing Venice - Strata NYC 2017Introducing Venice - Strata NYC 2017
Introducing Venice - Strata NYC 2017Felix GV
 
Kafka Connect by Datio
Kafka Connect by DatioKafka Connect by Datio
Kafka Connect by DatioDatio Big Data
 
Architecture Sustaining LINE Sticker services
Architecture Sustaining LINE Sticker servicesArchitecture Sustaining LINE Sticker services
Architecture Sustaining LINE Sticker servicesLINE Corporation
 
Joe witt may2015_kafka_nyc_apachenifi-overview
Joe witt may2015_kafka_nyc_apachenifi-overviewJoe witt may2015_kafka_nyc_apachenifi-overview
Joe witt may2015_kafka_nyc_apachenifi-overviewJoseph Witt
 
Confluent building a real-time streaming platform using kafka streams and k...
Confluent building a real-time streaming platform using kafka streams and k...Confluent building a real-time streaming platform using kafka streams and k...
Confluent building a real-time streaming platform using kafka streams and k...Thomas Alex
 
Introduction to Kafka connect
Introduction to Kafka connectIntroduction to Kafka connect
Introduction to Kafka connectKnoldus Inc.
 
(ATS6-PLAT09) Deploying Applications on load balanced AEP servers for high av...
(ATS6-PLAT09) Deploying Applications on load balanced AEP servers for high av...(ATS6-PLAT09) Deploying Applications on load balanced AEP servers for high av...
(ATS6-PLAT09) Deploying Applications on load balanced AEP servers for high av...BIOVIA
 
Nov 2014 webinar Making The Transition From Ftp
Nov 2014 webinar Making The Transition From FtpNov 2014 webinar Making The Transition From Ftp
Nov 2014 webinar Making The Transition From FtpFileCatalyst
 
The Many Faces of Apache Kafka: Leveraging real-time data at scale
The Many Faces of Apache Kafka: Leveraging real-time data at scaleThe Many Faces of Apache Kafka: Leveraging real-time data at scale
The Many Faces of Apache Kafka: Leveraging real-time data at scaleNeha Narkhede
 
Data streaming-systems
Data streaming-systemsData streaming-systems
Data streaming-systemsimcpune
 
Data ingestion and distribution with apache NiFi
Data ingestion and distribution with apache NiFiData ingestion and distribution with apache NiFi
Data ingestion and distribution with apache NiFiLev Brailovskiy
 
Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...
Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...
Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...Nitin Kumar
 
Intel omni path architecture
Intel omni path architectureIntel omni path architecture
Intel omni path architectureAshay Shirwadkar
 
Metrics Are Not Enough: Monitoring Apache Kafka and Streaming Applications
Metrics Are Not Enough: Monitoring Apache Kafka and Streaming ApplicationsMetrics Are Not Enough: Monitoring Apache Kafka and Streaming Applications
Metrics Are Not Enough: Monitoring Apache Kafka and Streaming Applicationsconfluent
 
Database ingest with Apache NiFi and MiNiFi
Database ingest with Apache NiFi and MiNiFiDatabase ingest with Apache NiFi and MiNiFi
Database ingest with Apache NiFi and MiNiFiLucian Neghina
 
[Webinar] AWS Monitoring with Site24x7
[Webinar] AWS Monitoring with Site24x7[Webinar] AWS Monitoring with Site24x7
[Webinar] AWS Monitoring with Site24x7Site24x7
 
Metrics are Not Enough: Monitoring Apache Kafka / Gwen Shapira (Confluent)
Metrics are Not Enough: Monitoring Apache Kafka / Gwen Shapira (Confluent)Metrics are Not Enough: Monitoring Apache Kafka / Gwen Shapira (Confluent)
Metrics are Not Enough: Monitoring Apache Kafka / Gwen Shapira (Confluent)Ontico
 
UDP accelerated file transfer - introducing an FTP replacement and its benefits
UDP accelerated file transfer - introducing an FTP replacement and its benefitsUDP accelerated file transfer - introducing an FTP replacement and its benefits
UDP accelerated file transfer - introducing an FTP replacement and its benefitsFileCatalyst
 
Apache Nifi - Custom Processor
Apache Nifi - Custom Processor Apache Nifi - Custom Processor
Apache Nifi - Custom Processor thotasrinath
 
Common issues with Apache Kafka® Producer
Common issues with Apache Kafka® ProducerCommon issues with Apache Kafka® Producer
Common issues with Apache Kafka® Producerconfluent
 
What to expect with Microsoft Exchange 2016?
What to expect with Microsoft Exchange 2016?What to expect with Microsoft Exchange 2016?
What to expect with Microsoft Exchange 2016?Kemp
 
OSMC 2021 | Handling 250K flows per second with OpenNMS: a case study
OSMC 2021 | Handling 250K flows per second with OpenNMS: a case studyOSMC 2021 | Handling 250K flows per second with OpenNMS: a case study
OSMC 2021 | Handling 250K flows per second with OpenNMS: a case studyNETWAYS
 
Monitoring Apache Kafka
Monitoring Apache KafkaMonitoring Apache Kafka
Monitoring Apache Kafkaconfluent
 
Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?confluent
 
GWAVACon 2015: Microsoft MVP - What's new in Exchange Server 2016?
GWAVACon 2015: Microsoft MVP - What's new in Exchange Server 2016?GWAVACon 2015: Microsoft MVP - What's new in Exchange Server 2016?
GWAVACon 2015: Microsoft MVP - What's new in Exchange Server 2016?GWAVA
 
Apache Kafka from 0.7 to 1.0, History and Lesson Learned
Apache Kafka from 0.7 to 1.0, History and Lesson LearnedApache Kafka from 0.7 to 1.0, History and Lesson Learned
Apache Kafka from 0.7 to 1.0, History and Lesson LearnedGuozhang Wang
 
Nifi
NifiNifi
NifiJulio Castro
 
Building real time data-driven products
Building real time data-driven productsBuilding real time data-driven products
Building real time data-driven productsLars Albertsson
 
Building Scalable Big Data Infrastructure Using Open Source Software Presenta...
Building Scalable Big Data Infrastructure Using Open Source Software Presenta...Building Scalable Big Data Infrastructure Using Open Source Software Presenta...
Building Scalable Big Data Infrastructure Using Open Source Software Presenta...ssuserd3a367
 

More Related Content

What's hot

Nov 2014 webinar Making The Transition From Ftp
Nov 2014 webinar Making The Transition From FtpNov 2014 webinar Making The Transition From Ftp
Nov 2014 webinar Making The Transition From FtpFileCatalyst
 
The Many Faces of Apache Kafka: Leveraging real-time data at scale
The Many Faces of Apache Kafka: Leveraging real-time data at scaleThe Many Faces of Apache Kafka: Leveraging real-time data at scale
The Many Faces of Apache Kafka: Leveraging real-time data at scaleNeha Narkhede
 
Data streaming-systems
Data streaming-systemsData streaming-systems
Data streaming-systemsimcpune
 
Data ingestion and distribution with apache NiFi
Data ingestion and distribution with apache NiFiData ingestion and distribution with apache NiFi
Data ingestion and distribution with apache NiFiLev Brailovskiy
 
Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...
Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...
Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...Nitin Kumar
 
Intel omni path architecture
Intel omni path architectureIntel omni path architecture
Intel omni path architectureAshay Shirwadkar
 
Metrics Are Not Enough: Monitoring Apache Kafka and Streaming Applications
Metrics Are Not Enough: Monitoring Apache Kafka and Streaming ApplicationsMetrics Are Not Enough: Monitoring Apache Kafka and Streaming Applications
Metrics Are Not Enough: Monitoring Apache Kafka and Streaming Applicationsconfluent
 
Database ingest with Apache NiFi and MiNiFi
Database ingest with Apache NiFi and MiNiFiDatabase ingest with Apache NiFi and MiNiFi
Database ingest with Apache NiFi and MiNiFiLucian Neghina
 
[Webinar] AWS Monitoring with Site24x7
[Webinar] AWS Monitoring with Site24x7[Webinar] AWS Monitoring with Site24x7
[Webinar] AWS Monitoring with Site24x7Site24x7
 
Metrics are Not Enough: Monitoring Apache Kafka / Gwen Shapira (Confluent)
Metrics are Not Enough: Monitoring Apache Kafka / Gwen Shapira (Confluent)Metrics are Not Enough: Monitoring Apache Kafka / Gwen Shapira (Confluent)
Metrics are Not Enough: Monitoring Apache Kafka / Gwen Shapira (Confluent)Ontico
 
UDP accelerated file transfer - introducing an FTP replacement and its benefits
UDP accelerated file transfer - introducing an FTP replacement and its benefitsUDP accelerated file transfer - introducing an FTP replacement and its benefits
UDP accelerated file transfer - introducing an FTP replacement and its benefitsFileCatalyst
 
Apache Nifi - Custom Processor
Apache Nifi - Custom Processor Apache Nifi - Custom Processor
Apache Nifi - Custom Processor thotasrinath
 
Common issues with Apache Kafka® Producer
Common issues with Apache Kafka® ProducerCommon issues with Apache Kafka® Producer
Common issues with Apache Kafka® Producerconfluent
 
What to expect with Microsoft Exchange 2016?
What to expect with Microsoft Exchange 2016?What to expect with Microsoft Exchange 2016?
What to expect with Microsoft Exchange 2016?Kemp
 
OSMC 2021 | Handling 250K flows per second with OpenNMS: a case study
OSMC 2021 | Handling 250K flows per second with OpenNMS: a case studyOSMC 2021 | Handling 250K flows per second with OpenNMS: a case study
OSMC 2021 | Handling 250K flows per second with OpenNMS: a case studyNETWAYS
 
Monitoring Apache Kafka
Monitoring Apache KafkaMonitoring Apache Kafka
Monitoring Apache Kafkaconfluent
 
Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?confluent
 
GWAVACon 2015: Microsoft MVP - What's new in Exchange Server 2016?
GWAVACon 2015: Microsoft MVP - What's new in Exchange Server 2016?GWAVACon 2015: Microsoft MVP - What's new in Exchange Server 2016?
GWAVACon 2015: Microsoft MVP - What's new in Exchange Server 2016?GWAVA
 
Apache Kafka from 0.7 to 1.0, History and Lesson Learned
Apache Kafka from 0.7 to 1.0, History and Lesson LearnedApache Kafka from 0.7 to 1.0, History and Lesson Learned
Apache Kafka from 0.7 to 1.0, History and Lesson LearnedGuozhang Wang
 

What's hot (20)

Nov 2014 webinar Making The Transition From Ftp
Nov 2014 webinar Making The Transition From FtpNov 2014 webinar Making The Transition From Ftp
Nov 2014 webinar Making The Transition From Ftp
 
The Many Faces of Apache Kafka: Leveraging real-time data at scale
The Many Faces of Apache Kafka: Leveraging real-time data at scaleThe Many Faces of Apache Kafka: Leveraging real-time data at scale
The Many Faces of Apache Kafka: Leveraging real-time data at scale
 
Data streaming-systems
Data streaming-systemsData streaming-systems
Data streaming-systems
 
Data ingestion and distribution with apache NiFi
Data ingestion and distribution with apache NiFiData ingestion and distribution with apache NiFi
Data ingestion and distribution with apache NiFi
 
Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...
Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...
Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...
 
Intel omni path architecture
Intel omni path architectureIntel omni path architecture
Intel omni path architecture
 
Metrics Are Not Enough: Monitoring Apache Kafka and Streaming Applications
Metrics Are Not Enough: Monitoring Apache Kafka and Streaming ApplicationsMetrics Are Not Enough: Monitoring Apache Kafka and Streaming Applications
Metrics Are Not Enough: Monitoring Apache Kafka and Streaming Applications
 
Database ingest with Apache NiFi and MiNiFi
Database ingest with Apache NiFi and MiNiFiDatabase ingest with Apache NiFi and MiNiFi
Database ingest with Apache NiFi and MiNiFi
 
[Webinar] AWS Monitoring with Site24x7
[Webinar] AWS Monitoring with Site24x7[Webinar] AWS Monitoring with Site24x7
[Webinar] AWS Monitoring with Site24x7
 
Metrics are Not Enough: Monitoring Apache Kafka / Gwen Shapira (Confluent)
Metrics are Not Enough: Monitoring Apache Kafka / Gwen Shapira (Confluent)Metrics are Not Enough: Monitoring Apache Kafka / Gwen Shapira (Confluent)
Metrics are Not Enough: Monitoring Apache Kafka / Gwen Shapira (Confluent)
 
UDP accelerated file transfer - introducing an FTP replacement and its benefits
UDP accelerated file transfer - introducing an FTP replacement and its benefitsUDP accelerated file transfer - introducing an FTP replacement and its benefits
UDP accelerated file transfer - introducing an FTP replacement and its benefits
 
Apache Nifi - Custom Processor
Apache Nifi - Custom Processor Apache Nifi - Custom Processor
Apache Nifi - Custom Processor
 
Common issues with Apache Kafka® Producer
Common issues with Apache Kafka® ProducerCommon issues with Apache Kafka® Producer
Common issues with Apache Kafka® Producer
 
What to expect with Microsoft Exchange 2016?
What to expect with Microsoft Exchange 2016?What to expect with Microsoft Exchange 2016?
What to expect with Microsoft Exchange 2016?
 
OSMC 2021 | Handling 250K flows per second with OpenNMS: a case study
OSMC 2021 | Handling 250K flows per second with OpenNMS: a case studyOSMC 2021 | Handling 250K flows per second with OpenNMS: a case study
OSMC 2021 | Handling 250K flows per second with OpenNMS: a case study
 
Monitoring Apache Kafka
Monitoring Apache KafkaMonitoring Apache Kafka
Monitoring Apache Kafka
 
Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?
 
GWAVACon 2015: Microsoft MVP - What's new in Exchange Server 2016?
GWAVACon 2015: Microsoft MVP - What's new in Exchange Server 2016?GWAVACon 2015: Microsoft MVP - What's new in Exchange Server 2016?
GWAVACon 2015: Microsoft MVP - What's new in Exchange Server 2016?
 
Apache Kafka from 0.7 to 1.0, History and Lesson Learned
Apache Kafka from 0.7 to 1.0, History and Lesson LearnedApache Kafka from 0.7 to 1.0, History and Lesson Learned
Apache Kafka from 0.7 to 1.0, History and Lesson Learned
 
Nifi
NifiNifi
Nifi
 

Similar to SFScon14: The OSS-Mailcluster of Raiffeisen OnLine GmbH

Building real time data-driven products
Building real time data-driven productsBuilding real time data-driven products
Building real time data-driven productsLars Albertsson
 
Building Scalable Big Data Infrastructure Using Open Source Software Presenta...
Building Scalable Big Data Infrastructure Using Open Source Software Presenta...Building Scalable Big Data Infrastructure Using Open Source Software Presenta...
Building Scalable Big Data Infrastructure Using Open Source Software Presenta...ssuserd3a367
 
Mohamed Amine Abdessemed – Real-time Data Integration with Apache Flink & Kafka
Mohamed Amine Abdessemed – Real-time Data Integration with Apache Flink & KafkaMohamed Amine Abdessemed – Real-time Data Integration with Apache Flink & Kafka
Mohamed Amine Abdessemed – Real-time Data Integration with Apache Flink & KafkaFlink Forward
 
Building Modern Digital Services on Scalable Private Government Infrastructur...
Building Modern Digital Services on Scalable Private Government Infrastructur...Building Modern Digital Services on Scalable Private Government Infrastructur...
Building Modern Digital Services on Scalable Private Government Infrastructur...Andrés Colón Pérez
 
Managing Exchange 2016 - Paul Robichaux
Managing Exchange 2016 - Paul RobichauxManaging Exchange 2016 - Paul Robichaux
Managing Exchange 2016 - Paul RobichauxSummit 7 Systems
 
(APP307) Leverage the Cloud with a Blue/Green Deployment Architecture | AWS r...
(APP307) Leverage the Cloud with a Blue/Green Deployment Architecture | AWS r...(APP307) Leverage the Cloud with a Blue/Green Deployment Architecture | AWS r...
(APP307) Leverage the Cloud with a Blue/Green Deployment Architecture | AWS r...Amazon Web Services
 
Adding Real-time Features to PHP Applications
Adding Real-time Features to PHP ApplicationsAdding Real-time Features to PHP Applications
Adding Real-time Features to PHP ApplicationsRonny López
 
Spark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice Machine
Spark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice MachineSpark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice Machine
Spark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice MachineData Con LA
 
How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.Renzo Tomà
 
Realtime traffic analyser
Realtime traffic analyserRealtime traffic analyser
Realtime traffic analyserAlex Moskvin
 
ADV Slides: Trends in Streaming Analytics and Message-oriented Middleware
ADV Slides: Trends in Streaming Analytics and Message-oriented MiddlewareADV Slides: Trends in Streaming Analytics and Message-oriented Middleware
ADV Slides: Trends in Streaming Analytics and Message-oriented MiddlewareDATAVERSITY
 
Business Automation - Cloud Automation Orchestration Service - Nordea
Business Automation - Cloud Automation Orchestration Service - NordeaBusiness Automation - Cloud Automation Orchestration Service - Nordea
Business Automation - Cloud Automation Orchestration Service - NordeaCA | Automic Software
 
Architectural considerations when building an API
Architectural considerations when building an APIArchitectural considerations when building an API
Architectural considerations when building an APIRod Hemphill
 
Cloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark AnalyticsCloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark Analyticsamesar0
 
High throughput data replication over RAFT
High throughput data replication over RAFTHigh throughput data replication over RAFT
High throughput data replication over RAFTDataWorks Summit
 
2014 09-12 lambda-architecture-at-indix
2014 09-12 lambda-architecture-at-indix2014 09-12 lambda-architecture-at-indix
2014 09-12 lambda-architecture-at-indixYu Ishikawa
 
Suning OpenStack Cloud and Heat
Suning OpenStack Cloud and HeatSuning OpenStack Cloud and Heat
Suning OpenStack Cloud and HeatQiming Teng
 
How to run a bank on Apache CloudStack
How to run a bank on Apache CloudStackHow to run a bank on Apache CloudStack
How to run a bank on Apache CloudStackgjdevos
 

Similar to SFScon14: The OSS-Mailcluster of Raiffeisen OnLine GmbH (20)

Building real time data-driven products
Building real time data-driven productsBuilding real time data-driven products
Building real time data-driven products
 
Building Scalable Big Data Infrastructure Using Open Source Software Presenta...
Building Scalable Big Data Infrastructure Using Open Source Software Presenta...Building Scalable Big Data Infrastructure Using Open Source Software Presenta...
Building Scalable Big Data Infrastructure Using Open Source Software Presenta...
 
Mohamed Amine Abdessemed – Real-time Data Integration with Apache Flink & Kafka
Mohamed Amine Abdessemed – Real-time Data Integration with Apache Flink & KafkaMohamed Amine Abdessemed – Real-time Data Integration with Apache Flink & Kafka
Mohamed Amine Abdessemed – Real-time Data Integration with Apache Flink & Kafka
 
Building Modern Digital Services on Scalable Private Government Infrastructur...
Building Modern Digital Services on Scalable Private Government Infrastructur...Building Modern Digital Services on Scalable Private Government Infrastructur...
Building Modern Digital Services on Scalable Private Government Infrastructur...
 
Managing Exchange 2016 - Paul Robichaux
Managing Exchange 2016 - Paul RobichauxManaging Exchange 2016 - Paul Robichaux
Managing Exchange 2016 - Paul Robichaux
 
Spotify: Data center & Backend buildout
Spotify: Data center & Backend buildoutSpotify: Data center & Backend buildout
Spotify: Data center & Backend buildout
 
(APP307) Leverage the Cloud with a Blue/Green Deployment Architecture | AWS r...
(APP307) Leverage the Cloud with a Blue/Green Deployment Architecture | AWS r...(APP307) Leverage the Cloud with a Blue/Green Deployment Architecture | AWS r...
(APP307) Leverage the Cloud with a Blue/Green Deployment Architecture | AWS r...
 
Web Fendamentals
Web FendamentalsWeb Fendamentals
Web Fendamentals
 
Adding Real-time Features to PHP Applications
Adding Real-time Features to PHP ApplicationsAdding Real-time Features to PHP Applications
Adding Real-time Features to PHP Applications
 
Spark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice Machine
Spark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice MachineSpark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice Machine
Spark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice Machine
 
How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.
 
Realtime traffic analyser
Realtime traffic analyserRealtime traffic analyser
Realtime traffic analyser
 
ADV Slides: Trends in Streaming Analytics and Message-oriented Middleware
ADV Slides: Trends in Streaming Analytics and Message-oriented MiddlewareADV Slides: Trends in Streaming Analytics and Message-oriented Middleware
ADV Slides: Trends in Streaming Analytics and Message-oriented Middleware
 
Business Automation - Cloud Automation Orchestration Service - Nordea
Business Automation - Cloud Automation Orchestration Service - NordeaBusiness Automation - Cloud Automation Orchestration Service - Nordea
Business Automation - Cloud Automation Orchestration Service - Nordea
 
Architectural considerations when building an API
Architectural considerations when building an APIArchitectural considerations when building an API
Architectural considerations when building an API
 
Cloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark AnalyticsCloud Security Monitoring and Spark Analytics
Cloud Security Monitoring and Spark Analytics
 
High throughput data replication over RAFT
High throughput data replication over RAFTHigh throughput data replication over RAFT
High throughput data replication over RAFT
 
2014 09-12 lambda-architecture-at-indix
2014 09-12 lambda-architecture-at-indix2014 09-12 lambda-architecture-at-indix
2014 09-12 lambda-architecture-at-indix
 
Suning OpenStack Cloud and Heat
Suning OpenStack Cloud and HeatSuning OpenStack Cloud and Heat
Suning OpenStack Cloud and Heat
 
How to run a bank on Apache CloudStack
How to run a bank on Apache CloudStackHow to run a bank on Apache CloudStack
How to run a bank on Apache CloudStack
 

More from South Tyrol Free Software Conference

SFSCON23 - Rufai Omowunmi Balogun - SMODEX – a Python package for understandi...
SFSCON23 - Rufai Omowunmi Balogun - SMODEX – a Python package for understandi...SFSCON23 - Rufai Omowunmi Balogun - SMODEX – a Python package for understandi...
SFSCON23 - Rufai Omowunmi Balogun - SMODEX – a Python package for understandi...South Tyrol Free Software Conference
 
SFSCON23 - Roberto Innocenti - From the design to reality is here the Communi...
SFSCON23 - Roberto Innocenti - From the design to reality is here the Communi...SFSCON23 - Roberto Innocenti - From the design to reality is here the Communi...
SFSCON23 - Roberto Innocenti - From the design to reality is here the Communi...South Tyrol Free Software Conference
 
SFSCON23 - Martin Rabanser - Real-time aeroplane tracking and the Open Data Hub
SFSCON23 - Martin Rabanser - Real-time aeroplane tracking and the Open Data HubSFSCON23 - Martin Rabanser - Real-time aeroplane tracking and the Open Data Hub
SFSCON23 - Martin Rabanser - Real-time aeroplane tracking and the Open Data HubSouth Tyrol Free Software Conference
 
SFSCON23 - Marianna d'Atri Enrico Zanardo - How can Blockchain technologies i...
SFSCON23 - Marianna d'Atri Enrico Zanardo - How can Blockchain technologies i...SFSCON23 - Marianna d'Atri Enrico Zanardo - How can Blockchain technologies i...
SFSCON23 - Marianna d'Atri Enrico Zanardo - How can Blockchain technologies i...South Tyrol Free Software Conference
 
SFSCON23 - Lucas Lasota - The Future of Connectivity, Open Internet and Human...
SFSCON23 - Lucas Lasota - The Future of Connectivity, Open Internet and Human...SFSCON23 - Lucas Lasota - The Future of Connectivity, Open Internet and Human...
SFSCON23 - Lucas Lasota - The Future of Connectivity, Open Internet and Human...South Tyrol Free Software Conference
 
SFSCON23 - Giovanni Giannotta - Intelligent Decision Support System for trace...
SFSCON23 - Giovanni Giannotta - Intelligent Decision Support System for trace...SFSCON23 - Giovanni Giannotta - Intelligent Decision Support System for trace...
SFSCON23 - Giovanni Giannotta - Intelligent Decision Support System for trace...South Tyrol Free Software Conference
 
SFSCON23 - Elena Maines - Embracing CI/CD workflows for building ETL pipelines
SFSCON23 - Elena Maines - Embracing CI/CD workflows for building ETL pipelinesSFSCON23 - Elena Maines - Embracing CI/CD workflows for building ETL pipelines
SFSCON23 - Elena Maines - Embracing CI/CD workflows for building ETL pipelinesSouth Tyrol Free Software Conference
 
SFSCON23 - Charles H. Schulz - Why open digital infrastructure matters
SFSCON23 - Charles H. Schulz - Why open digital infrastructure mattersSFSCON23 - Charles H. Schulz - Why open digital infrastructure matters
SFSCON23 - Charles H. Schulz - Why open digital infrastructure mattersSouth Tyrol Free Software Conference
 
SFSCON23 - Thomas Aichner - How IoT and AI are revolutionizing Mass Customiza...
SFSCON23 - Thomas Aichner - How IoT and AI are revolutionizing Mass Customiza...SFSCON23 - Thomas Aichner - How IoT and AI are revolutionizing Mass Customiza...
SFSCON23 - Thomas Aichner - How IoT and AI are revolutionizing Mass Customiza...South Tyrol Free Software Conference
 
SFSCON23 - Mirko Boehm - European regulators cast their eyes on maturing OSS ...
SFSCON23 - Mirko Boehm - European regulators cast their eyes on maturing OSS ...SFSCON23 - Mirko Boehm - European regulators cast their eyes on maturing OSS ...
SFSCON23 - Mirko Boehm - European regulators cast their eyes on maturing OSS ...South Tyrol Free Software Conference
 
SFSCON23 - Marco Pavanelli - Monitoring the fleet of Sasa with free software
SFSCON23 - Marco Pavanelli - Monitoring the fleet of Sasa with free softwareSFSCON23 - Marco Pavanelli - Monitoring the fleet of Sasa with free software
SFSCON23 - Marco Pavanelli - Monitoring the fleet of Sasa with free softwareSouth Tyrol Free Software Conference
 
SFSCON23 - Marco Cortella - KNOWAGE and AICS for 2030 agenda SDG goals monito...
SFSCON23 - Marco Cortella - KNOWAGE and AICS for 2030 agenda SDG goals monito...SFSCON23 - Marco Cortella - KNOWAGE and AICS for 2030 agenda SDG goals monito...
SFSCON23 - Marco Cortella - KNOWAGE and AICS for 2030 agenda SDG goals monito...South Tyrol Free Software Conference
 
SFSCON23 - Lina Ceballos - Interoperable Europe Act - A real game changer
SFSCON23 - Lina Ceballos - Interoperable Europe Act - A real game changerSFSCON23 - Lina Ceballos - Interoperable Europe Act - A real game changer
SFSCON23 - Lina Ceballos - Interoperable Europe Act - A real game changerSouth Tyrol Free Software Conference
 
SFSCON23 - Johannes Näder Linus Sehn - Let’s monitor implementation of Free S...
SFSCON23 - Johannes Näder Linus Sehn - Let’s monitor implementation of Free S...SFSCON23 - Johannes Näder Linus Sehn - Let’s monitor implementation of Free S...
SFSCON23 - Johannes Näder Linus Sehn - Let’s monitor implementation of Free S...South Tyrol Free Software Conference
 
SFSCON23 - Gabriel Ku Wei Bin - Why Do We Need A Next Generation Internet
SFSCON23 - Gabriel Ku Wei Bin - Why Do We Need A Next Generation InternetSFSCON23 - Gabriel Ku Wei Bin - Why Do We Need A Next Generation Internet
SFSCON23 - Gabriel Ku Wei Bin - Why Do We Need A Next Generation InternetSouth Tyrol Free Software Conference
 
SFSCON23 - Davide Vernassa - Empowering Insights Unveiling the latest innova...
SFSCON23 - Davide Vernassa - Empowering Insights Unveiling the latest innova...SFSCON23 - Davide Vernassa - Empowering Insights Unveiling the latest innova...
SFSCON23 - Davide Vernassa - Empowering Insights Unveiling the latest innova...South Tyrol Free Software Conference
 

More from South Tyrol Free Software Conference (20)

SFSCON23 - Rufai Omowunmi Balogun - SMODEX – a Python package for understandi...
SFSCON23 - Rufai Omowunmi Balogun - SMODEX – a Python package for understandi...SFSCON23 - Rufai Omowunmi Balogun - SMODEX – a Python package for understandi...
SFSCON23 - Rufai Omowunmi Balogun - SMODEX – a Python package for understandi...
 
SFSCON23 - Roberto Innocenti - From the design to reality is here the Communi...
SFSCON23 - Roberto Innocenti - From the design to reality is here the Communi...SFSCON23 - Roberto Innocenti - From the design to reality is here the Communi...
SFSCON23 - Roberto Innocenti - From the design to reality is here the Communi...
 
SFSCON23 - Martin Rabanser - Real-time aeroplane tracking and the Open Data Hub
SFSCON23 - Martin Rabanser - Real-time aeroplane tracking and the Open Data HubSFSCON23 - Martin Rabanser - Real-time aeroplane tracking and the Open Data Hub
SFSCON23 - Martin Rabanser - Real-time aeroplane tracking and the Open Data Hub
 
SFSCON23 - Marianna d'Atri Enrico Zanardo - How can Blockchain technologies i...
SFSCON23 - Marianna d'Atri Enrico Zanardo - How can Blockchain technologies i...SFSCON23 - Marianna d'Atri Enrico Zanardo - How can Blockchain technologies i...
SFSCON23 - Marianna d'Atri Enrico Zanardo - How can Blockchain technologies i...
 
SFSCON23 - Lucas Lasota - The Future of Connectivity, Open Internet and Human...
SFSCON23 - Lucas Lasota - The Future of Connectivity, Open Internet and Human...SFSCON23 - Lucas Lasota - The Future of Connectivity, Open Internet and Human...
SFSCON23 - Lucas Lasota - The Future of Connectivity, Open Internet and Human...
 
SFSCON23 - Giovanni Giannotta - Intelligent Decision Support System for trace...
SFSCON23 - Giovanni Giannotta - Intelligent Decision Support System for trace...SFSCON23 - Giovanni Giannotta - Intelligent Decision Support System for trace...
SFSCON23 - Giovanni Giannotta - Intelligent Decision Support System for trace...
 
SFSCON23 - Elena Maines - Embracing CI/CD workflows for building ETL pipelines
SFSCON23 - Elena Maines - Embracing CI/CD workflows for building ETL pipelinesSFSCON23 - Elena Maines - Embracing CI/CD workflows for building ETL pipelines
SFSCON23 - Elena Maines - Embracing CI/CD workflows for building ETL pipelines
 
SFSCON23 - Christian Busse - Free Software and Open Science
SFSCON23 - Christian Busse - Free Software and Open ScienceSFSCON23 - Christian Busse - Free Software and Open Science
SFSCON23 - Christian Busse - Free Software and Open Science
 
SFSCON23 - Charles H. Schulz - Why open digital infrastructure matters
SFSCON23 - Charles H. Schulz - Why open digital infrastructure mattersSFSCON23 - Charles H. Schulz - Why open digital infrastructure matters
SFSCON23 - Charles H. Schulz - Why open digital infrastructure matters
 
SFSCON23 - Andrea Vianello - Achieving FAIRness with EDP-portal
SFSCON23 - Andrea Vianello - Achieving FAIRness with EDP-portalSFSCON23 - Andrea Vianello - Achieving FAIRness with EDP-portal
SFSCON23 - Andrea Vianello - Achieving FAIRness with EDP-portal
 
SFSCON23 - Thomas Aichner - How IoT and AI are revolutionizing Mass Customiza...
SFSCON23 - Thomas Aichner - How IoT and AI are revolutionizing Mass Customiza...SFSCON23 - Thomas Aichner - How IoT and AI are revolutionizing Mass Customiza...
SFSCON23 - Thomas Aichner - How IoT and AI are revolutionizing Mass Customiza...
 
SFSCON23 - Stefan Mutschlechner - Smart Werke Meran
SFSCON23 - Stefan Mutschlechner - Smart Werke MeranSFSCON23 - Stefan Mutschlechner - Smart Werke Meran
SFSCON23 - Stefan Mutschlechner - Smart Werke Meran
 
SFSCON23 - Mirko Boehm - European regulators cast their eyes on maturing OSS ...
SFSCON23 - Mirko Boehm - European regulators cast their eyes on maturing OSS ...SFSCON23 - Mirko Boehm - European regulators cast their eyes on maturing OSS ...
SFSCON23 - Mirko Boehm - European regulators cast their eyes on maturing OSS ...
 
SFSCON23 - Marco Pavanelli - Monitoring the fleet of Sasa with free software
SFSCON23 - Marco Pavanelli - Monitoring the fleet of Sasa with free softwareSFSCON23 - Marco Pavanelli - Monitoring the fleet of Sasa with free software
SFSCON23 - Marco Pavanelli - Monitoring the fleet of Sasa with free software
 
SFSCON23 - Marco Cortella - KNOWAGE and AICS for 2030 agenda SDG goals monito...
SFSCON23 - Marco Cortella - KNOWAGE and AICS for 2030 agenda SDG goals monito...SFSCON23 - Marco Cortella - KNOWAGE and AICS for 2030 agenda SDG goals monito...
SFSCON23 - Marco Cortella - KNOWAGE and AICS for 2030 agenda SDG goals monito...
 
SFSCON23 - Lina Ceballos - Interoperable Europe Act - A real game changer
SFSCON23 - Lina Ceballos - Interoperable Europe Act - A real game changerSFSCON23 - Lina Ceballos - Interoperable Europe Act - A real game changer
SFSCON23 - Lina Ceballos - Interoperable Europe Act - A real game changer
 
SFSCON23 - Johannes Näder Linus Sehn - Let’s monitor implementation of Free S...
SFSCON23 - Johannes Näder Linus Sehn - Let’s monitor implementation of Free S...SFSCON23 - Johannes Näder Linus Sehn - Let’s monitor implementation of Free S...
SFSCON23 - Johannes Näder Linus Sehn - Let’s monitor implementation of Free S...
 
SFSCON23 - Gabriel Ku Wei Bin - Why Do We Need A Next Generation Internet
SFSCON23 - Gabriel Ku Wei Bin - Why Do We Need A Next Generation InternetSFSCON23 - Gabriel Ku Wei Bin - Why Do We Need A Next Generation Internet
SFSCON23 - Gabriel Ku Wei Bin - Why Do We Need A Next Generation Internet
 
SFSCON23 - Edoardo Scepi - The Brand-New Version of IGis Maps
SFSCON23 - Edoardo Scepi - The Brand-New Version of IGis MapsSFSCON23 - Edoardo Scepi - The Brand-New Version of IGis Maps
SFSCON23 - Edoardo Scepi - The Brand-New Version of IGis Maps
 
SFSCON23 - Davide Vernassa - Empowering Insights Unveiling the latest innova...
SFSCON23 - Davide Vernassa - Empowering Insights Unveiling the latest innova...SFSCON23 - Davide Vernassa - Empowering Insights Unveiling the latest innova...
SFSCON23 - Davide Vernassa - Empowering Insights Unveiling the latest innova...
 

Recently uploaded

Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slidespraypatel2
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024Results
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsRoshan Dwivedi
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024The Digital Insurer
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxFactors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxKatpro Technologies
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 

Recently uploaded (20)

Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slides
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxFactors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 

SFScon14: The OSS-Mailcluster of Raiffeisen OnLine GmbH

  • 2. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Purpose + requirements •implementation of a new spam filter system •replacement of the present Windows solution (3 servers) •redundant setup •Active/Active preferred •easily scalable and flexible •give our “Service Center” access to •account configuration setting •log data for customer support purposes •should allow further expansion in order to implement new features and improve hit rate
  • 3. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Which solution should we use? •Evaluation of manufacturers •major differences in price/performance ratio •not all solutions grant actual redundancy (only standby setup) •Test installation by a manufacturer works OK … •… until Friday afternoon, when a sudden deadlock occurs! •After having spent 3 days searching for the error, the manufacturer states that: •…there must be a power outage in the datacenter -> I’m sure there was no outage 
  • 4. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 We finally decided to try out a self-made solution •But why OSS? •The experience told us it’s preferable to adopt a solution on which we are able to investigate by ourselves in an emergency situation. •We were aiming at a solution that would allow us to meet new requirements on our own in the future. •We wanted to be able to develop own new components if necessary. •Our objective was to reduce dependency on one single manufacturer.
  • 5. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 We found all components in the open source world very quickly •First implementation in the middle of 2008: •Postfix (MTA) -> on the front line •Spamassassin + ClamAV -> as content filter •AmaVis -> interface between postfix + content filter •MySQL –> configuration + quarantine (Spam/Virus) •MySQL -> central logging for “ServiceCenter” + customers •PHP -> administration front end for “ServiceCenter” + customers
  • 6. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 First phase of implementation •2x bare metal servers (8GB RAM, QuadCore, SAS) •„extreme“ container-virtualization •Each service had its own container •2 Postfix front end MTAs •2 AmaVis (Spamassassin, ClamAV) content filters •2 MySQL Master/Master Replications (configuration) •2 MySQL Master/Master Replications (quarantine) •2 MySQL Master/Master Replications (logdatabase) •2 DNS Resolvers •2 Postfix Backend MTAs (delivering e-mails to storage back ends)
  • 7. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Each node could go down at any time •Redundant SMTP paths IN ---> Postfix---> AmaVis---> Postfix---> OUT
  • 8. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 What are the main problems? •OSS components are available in heaps, but it was difficult to •have all different components work together •process large amounts of log data in order for users to understand them •We had not much experience with high mail traffic. •It took us some time to identify the correct performance and configuration parameters.
  • 9. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Technical support •Google  •knows about 90% of all the problems we came across •Community •We posted our own questions to mailing lists and forums. •We kept reading newsgroups. •Common sense •logical + analytical approach •“Don’t worry, it’s just numbers and letters.”* * cit.: Thomas Gelf
  • 10. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Continuous enhancements in the last years •Perl -> small policy daemons we developed in 2012 and 2013 •selective grey listing •performance improvements, storing non important data into the RAM •ClamAv content filter replaced(July 2014) •by ClamAv-Milter: scans viruses directly on front end servers and rejects with 550 in real time instead of storing in quarantine •SPF-Policyd (Juli 2014) •supports Sender Policy Framework in either direction (In/Out) •OpendDKIM (Juli 2014) •supports Domain Key Identified Mail in either direction (In/Out) •OpenDMARC (Juli 2014) •supports Domain-based Message Authentication, Reporting & Conformance in either direction (In/Out)
  • 11. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Current mail traffic (Postfix 1)
  • 12. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Current mail traffic (Postfix 2, Content filter)
  • 13. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Some more highlights: •SSL/TLS availableon all front end servers •IPv6 •active on all front end servers in both directions •(partial) communication between components •now facing IMAP/POP3 + SMTP submission •IMAP + POP3 + SMTP failover •front end displays processed log data for „ServiceCenter“ and customers
  • 14. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 IMAP + POP3 failover •outward: virtialIP + Multicast MAC •basedon IPTablesCluster-IP Module •[imap,pop3].rolmail.net (Dovecot) •smtp.rolmail.net (Postfix) •2 proxyservers •imap1.rolmail.net •imap2.rolmail.net •morepossible •Mailstore: •manybackend servers(Dovecot) •Load balancingofI/O
  • 15. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Processed data for „ServiceCenter“ and customers
  • 16. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 Future plans •to force SSL/TLS for IMAP/POP3 + SMTP submission clients •to protect mail traffic with DNS-based Authentication of Named Entities (called DANE) •protection of zones with DNSSEC is needed •to improve quarantine mechanism •rejecting all e-mails tagged as spam directly on the frond end servers with 550 -> no mail will be stored in quarantine •maybe to implement the mailstore backend server pool as an active/active setup
  • 17. Raiffeisen OnLine‘sOSS mail cluster urban.loesch@raiffeisen.net 18.11.2014 ### It was a pleasure to be here ### Many thanks for your attention! Urban Lösch Raiffeisen OnLine GmbH urban.loesch@raiffeisen.net