Prezentace z konference Virtualization Forum 2019 Praha, 3.10.2019 Hlavní sál

1. Presentation name:
Author/team:
Location:
Date:
FEG Case Study: from legacy to enterprise – Veeam and ExaGrid
Jiri Suchora
FEG, Prague
3.10. 2019

2. 2
Introduction
• Jiri Suchora
• Role: IT and Cybersecurity Architect
• Mail: suchora.jiri@feg.eu
• Location: FEG HQ – Prague

3. 3
Fortuna Entertainment Group (FEG) History
1990
Fortuna sázková
kancelář a.s.
is established in
Czechoslovakia
1991
the Terno a.s. (later
Fortuna) sister
company is
established in
Slovakia
2005
Penta bought
Fortuna and
Polish company
Professional
2010
Fortuna
Entertainment
Group holding
is established
2015
the acquisition
of Public Bet
In Romania
2018
the acquisition of
Hattrick Group
in Croatia

4. 4
FORTUNA ENTERTAINMENT GROUP
ABOUT FEG
• The largest Central European betting operator.
• Operates on the Czech, Slovak, Polish,
Romanian and Croatian markets
• B2B areas in Colombia, Spain, Montenegro,
Bosnia and Herzegovina
• Constantly investing in new products and
expanding its existing network of branches and
services.
• 6 850 employees in 5 countries
• Nomination on EGR Operation Awards 2019 –
“Gambling Oscars” in category Operator of the
Year.
CZ SK PL RO CRO

5. 5
Infrastructure Numbers
• Sites and Locations across EU
• Presence in 5 countries
• 8 DCs
• 10 HQs/Offices
• ~ 3500 Branch Offices (Outlets)
• ~ 4000 Self Service Betting Terminals
• Hosts and Servers
• ~3500 VMs
• ~ 200 Hypervisors

6. 6
Infrastructure Details
• Network
• Management LAN Networks – 1 Gbit
• Production LAN Networks – 10 Gbit
• Backup/NAS non-routed Networks – 10 Gbit
• SAN Storage Networks – 16 Gbit
• Storage:
• Mainly SAN based (Virtualisation, DBs)
• NAS/NFS for some application workload and Backup
Storages
• S3 for some application workload
• Operation Systems
• Linux – Business workload
• MS Windows – Backoffice + Retail in some countries

7. 7
Application Workload
• Production DB (PostgreSQL) – VMs (Majority)
and Physical Servers (Minority)
• MS World (MSSQL, Exchange, Active Directory,
Fileshares)
• Mongo DB and other NOSQL
• Oracle (BI)
• Application and Infrastructure Logs – long term
archiving (legal point of view)
• Repositories – Code (GIT) and Images (Docker
Registry and RPM repos)

8. 8
Old World to be replaced
• Mix of various technologies across countries and sites (mainly
historical reasons; acquisitions)
• Combinations of old IBM TSMs; VMware vSphere Data Protection;
home-made scripts; Tape Devices; Datadomains; Localdisks; Old
SAN storages…
• Big need for consolidation, standardisation => One technology
stack for Backup across whole Group

9. 9
Requirements for New World
• NFS – to separate main storage network (SAN) from backup
network
• Good TCO
• Scalability – need for adding new countries/sites or easily growing
the current locations
• Central management across multiple sites and countries

10. 10
New World – Backup and Restore
• Veeam Backup & Replication Enterprise
• Hypervisors - VMware and Hyper-V
• Dedicated Non-routed Network based on
Nexus 3K model switches
• Veeam Proxy – Virtual and Physical

11. 11
New World – Backup and Restore
• ExaGrid as backup storage
• ExaGrid Models per DC tier:
• EX63000E modules for Big sized DCs (126 TB per module)
• EX32000E modules for Medium sized DCs (64 TB per module)
• EX13000E modules for Small sized DCs (26 TB per module)
• Together ~800 TB Backup Storage across sites before dedup

12. 12
Veeam & ExaGrid Platform – High Level Architecture
• Backup and Recovery solution scope:
• DCs to focus “On-premise”
• Cloud open readiness (Amazon, Azure)
• Long-term Retention
• Landing Zones for fast backup & recovery
• Dedup for long term/low costs
• Off-site backup and recovery
• Country(ies) -> Nagano DC in CZ
• 3rd location – Cloud

13. 13
Standard Topology of One Site
Description:
BLACK – Routed
networks
RED – Non-Routed
local networks
GREEN – Routed
Networks to WAN

14. 14
Full Topology across FEG
• Local non-routed 10 GB network in every site
is used for transferring of data between
ExaGrid Backup Appliance Veeam Proxies
and Workload we would like to backup
• As HQ sites don't have own backup
appliance, those are being backed up to local
country DC - FEG WAN is used for
interconnection
• Veeam Backup Enterprise server is used for
managing all other Veeam backup servers
and their proxies in all locations of FEG - FEG
WAN is used for interconnection

15. 15
Data Transfer between Sites
• Lease Lines are used between main CZ DC (Nagano) and other
DCs for WAN with Internet connection as backup line
• To separate and also avoid QoSing between production and
„Backup data“ we decided to order another dedicated Internet
lines just for Backup traffic between primary and secondary
lcoations.
• All lines are 1 Gbit per sec

16. 16
Types of Backup Jobs
• Normal Classic Snaphots
• Normal common VMs, stateless without needs of higher consistency
• Application Consistent Snapshots
• Active Directory; Exchange; Fileservers; MSSQL; …
• Production DBs
• PostgreSQL Servers
• Combination of snapshots, Pre and Post Scripts and archiving of WAL
logs directly to backup storage

17. 17
Next Steps and Outlook
• Backup as a Service for internal customers
• Possible by upgrade to Enterprise Plus OR own
service portal based on automation scripts
• Handover responsibility to other teams and reduce
TCO
• DR to public cloud
• For some countries (legal point of view)
• Fast recovery
• Uploaded as object (eg. to AWS S3 or Azure Blob
Object Store)

18. 18
Gains
• Industry leading backup, restore and replications
• Consolidation and Standardization across whole Group
• Cost effectivity and long-term storage
• Scalability in terms of further expansion
• No specific upgrades needed, just Veeam license coverage
plus adding ExaGrid appliance to extend the storage pool
• Rapid recovery of most recent copy
• VM, files, apps (AD, Databases)
• Veeam local development resources
• Detail design verification, best practices
• Veeam architecture design assessment

19. 19
Pains
• Shared backup storage across environments
• dev-test-stg-prod = Shared non-routed network
• fight between cost and security
• Data transfer between sites
• QoS via Shared lines vs. Dedicated lines for backup
• Cost and complexity