SlideShare una empresa de Scribd logo
1 de 13
Descargar para leer sin conexión
Big Data at CallFire

Vijesh Mehta (Co-Founder and CTO)
Agenda

•  A little about CallFire

•  CallFire’s technical challenges

•  How CallFire deals with data

•  Summary
Some background about myself

•  I am one of the founders of CallFire.
  –  Started in 2005 in a small apartment
  –  Now 28 people
  –  Bootstrapped and profitable

•  I’ve been writing software primarily in the
   Java space for 12 years. CallFire is all
   Java.
  –  We use : Wicket, Guice, Hibernate, MySQL,
     Cassandra, ActiveMQ, XEN, Puppet
About CallFire

•  We are a cloud telephony provider.
   –  Outbound Phone calls
   –  Phone Numbers
   –  SMS through long and short codes
   –  IVR – Interactive Voice Response
   –  Power Dialing

•  CallFire’s call volume can get large very quickly.
   –  Hurricane Sandy : 1.9 million emergency calls

•  4 Engineers and 1 System admin managing
   operations and new features.
      •  We just hired 7 more engineers this year, and still hiring!
Technical Challenges by Numbers

•  1.4 billion calls and texts
     –  Growing exponentially
•    Over 50,000 accounts
•    Over 6 million campaigns
•    80 million sound files
•    14 TB in storage (NFS)
•    MySQL : Over 10,000 qps at peak

 Big data isn’t always big company problem!
Growing faster each day

                Campaigns	
  over	
  Time	
  
7000000	
  

6000000	
  

5000000	
  

4000000	
  

3000000	
  

2000000	
  

1000000	
  

        0	
  
The first challenge

•  Problem : We outgrew our datacenter. New
   systems need access to central storage.
   Replication across a 1gb/s interconnect.

•  Needed Solution:
  –  Must work across datacenter
  –  Must scale as demand increases
  –  Must be fault tolerant
  –  Must deal with over 80 million sound files
  –  Cheaper the better
Solutions Considered (2010)
                                  NFS	
                              GLUSTER	
                              HDFS	
                              CASSANDRA	
  
Fault	
  Tolerant	
               Yes,	
  if	
  configured	
          Yes	
                                  Yes	
                               Yes	
  

Datacenter	
                      Maybe.	
  Rsync	
  isn’t	
         Not	
  at	
  the	
  Dme	
              Yes	
                               Yes	
  
Replica>on	
                      fun	
  with	
  lots	
  of	
  
                                  files.	
  
Easy	
  to	
  add	
  storage	
   No	
                                Not	
  at	
  the	
  Dme	
              Yes	
                               Yes	
  

No	
  Single	
  point	
  of	
     No	
                               Yes	
                                  Not	
  exactly,	
                   Yes	
  
failure	
                                                                                                   NameNode.	
  
Data	
  always	
                  No,	
  hard	
  to	
  sort	
        No,	
  same	
  as	
  a	
  file	
        Yes	
                               Yes	
  
accessible	
  easily	
            through	
  file	
                   system	
  
                                  systems.	
  	
  

Notes	
                           Not	
  working	
  for	
  us.	
     Looks	
  good,	
  tried	
  it	
        Didn’t	
  like	
  the	
  name	
     Everything	
  we	
  
                                  Too	
  much	
                      for	
  a	
  while.	
  Easy	
  at	
     node	
  issue.	
  May	
             need,	
  quick	
  to	
  
                                  management	
  and	
                first	
  because	
  it	
  was	
         have	
  been	
  a	
  good	
         learn.	
  We	
  went	
  all	
  
                                  downDme.	
                         a	
  file	
  system.	
                  way	
  to	
  go.	
                  in!	
  



*	
  Only	
  LAN	
  soluDons	
  considered.	
  Calls	
  had	
  too	
  much	
  latency	
  in	
  the	
  cloud,	
  or	
  even	
  
across	
  datacenter.	
  
Cassandra

•  Storage isn’t the best use of Cassandra.

•  Do not exceed 50% of drive space.
  –  Compaction needs the space. Hard lesson learned.

•  Fault Tolerance: Replication factor of 3.

•  Result
      •  1 TB of data = 6 TB of storage needed!
      •  CallFire has a 74TB Cassandra Cluster
Extending the scope

•  We like SQL and Hibernate.
   –  Pros: Easy, Flexible, Ad-Hoc Queries, Locks
   –  Cons: Scaling


•  Solution: Sharding with Cassandra for universal data


         Shard	
  1	
             Shard	
  2	
         Shard	
  3	
  




                            Cassandra	
  Cluster	
  
Sharding + Big Data

•  Cassandra makes sharding easier
  –  Easy to store universal data. (Authentication)
  –  Performs very well


•  Tungsten Replicator (Big Data with SQL)
  –  Sharding makes joins impossible, so fan your
     data into central places.
  –  NoSQL can’t handle ad-hoc queries. No
     worries, you can still have SQL.
Big Data Summary

•  Not Just for big companies, data grows rapidly in
   todays environment.
   –  Nice article about Obama’s Data Crunchers:
   –    http://swampland.time.com/2012/11/07/inside-the-secret-world-of-quants-and-data-crunchers-who-helped-obama-win/




•  NoSQL systems have easier scaling and fault
   tolerance mechanisms.
   –  Not uncommon to see small teams with 10-20 node
      clusters.

•  SQL is still a big part of the equation. (Tungsten)
   –  Fan in information across partitions
   –  Replicate across datacenters
   –  Keep your ad-hoc dreams alive!
Passive / Archived Storage
Backblaze	
  –	
  $5,300	
  for	
  empty	
  case.	
  Holds	
  45	
  Drives	
  (117TB	
  usable	
  space)	
  




hUp://www.protocase.com/products/index.php?e=Backblaze	
  

Más contenido relacionado

Similar a Big data at CallFire

Storage Systems For Scalable systems
Storage Systems For Scalable systemsStorage Systems For Scalable systems
Storage Systems For Scalable systemselliando dias
 
Mary Firme Content Marketing for Demand Creation
Mary Firme Content Marketing for Demand Creation Mary Firme Content Marketing for Demand Creation
Mary Firme Content Marketing for Demand Creation Mary Firme
 
How to overcome mysterious problems caused by large and multi-tenancy Hadoop ...
How to overcome mysterious problems caused by large and multi-tenancy Hadoop ...How to overcome mysterious problems caused by large and multi-tenancy Hadoop ...
How to overcome mysterious problems caused by large and multi-tenancy Hadoop ...DataWorks Summit/Hadoop Summit
 
Big data explanation with real time use case
 Big data explanation with real time use case Big data explanation with real time use case
Big data explanation with real time use caseN.Jagadish Kumar
 
Overview of Redundant Disk Arrays
Overview of Redundant Disk ArraysOverview of Redundant Disk Arrays
Overview of Redundant Disk ArraysAndrew Robinson
 
Ceph Day Santa Clara: Ceph at DreamHost
Ceph Day Santa Clara: Ceph at DreamHost Ceph Day Santa Clara: Ceph at DreamHost
Ceph Day Santa Clara: Ceph at DreamHost Ceph Community
 
Make Life Suck Less (Building Scalable Systems)
Make Life Suck Less (Building Scalable Systems)Make Life Suck Less (Building Scalable Systems)
Make Life Suck Less (Building Scalable Systems)guest0f8e278
 
Improving Hadoop Cluster Performance via Linux Configuration
Improving Hadoop Cluster Performance via Linux ConfigurationImproving Hadoop Cluster Performance via Linux Configuration
Improving Hadoop Cluster Performance via Linux ConfigurationDataWorks Summit
 
Patella railsconf 2012
Patella railsconf 2012Patella railsconf 2012
Patella railsconf 2012Jeff Dwyer
 
Scaling a High Traffic Web Application: Our Journey from Java to PHP
Scaling a High Traffic Web Application: Our Journey from Java to PHPScaling a High Traffic Web Application: Our Journey from Java to PHP
Scaling a High Traffic Web Application: Our Journey from Java to PHP120bi
 
Scaling High Traffic Web Applications
Scaling High Traffic Web ApplicationsScaling High Traffic Web Applications
Scaling High Traffic Web ApplicationsAchievers Tech
 
Nimble Storage Series A presentation 2007
Nimble Storage Series A presentation 2007Nimble Storage Series A presentation 2007
Nimble Storage Series A presentation 2007Wing Venture Capital
 
Web20expo Scalable Web Arch
Web20expo Scalable Web ArchWeb20expo Scalable Web Arch
Web20expo Scalable Web Archroyans
 
Web20expo Scalable Web Arch
Web20expo Scalable Web ArchWeb20expo Scalable Web Arch
Web20expo Scalable Web Archguest18a0f1
 
Web20expo Scalable Web Arch
Web20expo Scalable Web ArchWeb20expo Scalable Web Arch
Web20expo Scalable Web Archmclee
 
Scaling with sync_replication using Galera and EC2
Scaling with sync_replication using Galera and EC2Scaling with sync_replication using Galera and EC2
Scaling with sync_replication using Galera and EC2Marco Tusa
 
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCScalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCCal Henderson
 
High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013Server Density
 
Scaling HDFS for Exabyte Storage@twitter
Scaling HDFS for Exabyte Storage@twitterScaling HDFS for Exabyte Storage@twitter
Scaling HDFS for Exabyte Storage@twitterlohitvijayarenu
 
Handling Massive Writes
Handling Massive WritesHandling Massive Writes
Handling Massive WritesLiran Zelkha
 

Similar a Big data at CallFire (20)

Storage Systems For Scalable systems
Storage Systems For Scalable systemsStorage Systems For Scalable systems
Storage Systems For Scalable systems
 
Mary Firme Content Marketing for Demand Creation
Mary Firme Content Marketing for Demand Creation Mary Firme Content Marketing for Demand Creation
Mary Firme Content Marketing for Demand Creation
 
How to overcome mysterious problems caused by large and multi-tenancy Hadoop ...
How to overcome mysterious problems caused by large and multi-tenancy Hadoop ...How to overcome mysterious problems caused by large and multi-tenancy Hadoop ...
How to overcome mysterious problems caused by large and multi-tenancy Hadoop ...
 
Big data explanation with real time use case
 Big data explanation with real time use case Big data explanation with real time use case
Big data explanation with real time use case
 
Overview of Redundant Disk Arrays
Overview of Redundant Disk ArraysOverview of Redundant Disk Arrays
Overview of Redundant Disk Arrays
 
Ceph Day Santa Clara: Ceph at DreamHost
Ceph Day Santa Clara: Ceph at DreamHost Ceph Day Santa Clara: Ceph at DreamHost
Ceph Day Santa Clara: Ceph at DreamHost
 
Make Life Suck Less (Building Scalable Systems)
Make Life Suck Less (Building Scalable Systems)Make Life Suck Less (Building Scalable Systems)
Make Life Suck Less (Building Scalable Systems)
 
Improving Hadoop Cluster Performance via Linux Configuration
Improving Hadoop Cluster Performance via Linux ConfigurationImproving Hadoop Cluster Performance via Linux Configuration
Improving Hadoop Cluster Performance via Linux Configuration
 
Patella railsconf 2012
Patella railsconf 2012Patella railsconf 2012
Patella railsconf 2012
 
Scaling a High Traffic Web Application: Our Journey from Java to PHP
Scaling a High Traffic Web Application: Our Journey from Java to PHPScaling a High Traffic Web Application: Our Journey from Java to PHP
Scaling a High Traffic Web Application: Our Journey from Java to PHP
 
Scaling High Traffic Web Applications
Scaling High Traffic Web ApplicationsScaling High Traffic Web Applications
Scaling High Traffic Web Applications
 
Nimble Storage Series A presentation 2007
Nimble Storage Series A presentation 2007Nimble Storage Series A presentation 2007
Nimble Storage Series A presentation 2007
 
Web20expo Scalable Web Arch
Web20expo Scalable Web ArchWeb20expo Scalable Web Arch
Web20expo Scalable Web Arch
 
Web20expo Scalable Web Arch
Web20expo Scalable Web ArchWeb20expo Scalable Web Arch
Web20expo Scalable Web Arch
 
Web20expo Scalable Web Arch
Web20expo Scalable Web ArchWeb20expo Scalable Web Arch
Web20expo Scalable Web Arch
 
Scaling with sync_replication using Galera and EC2
Scaling with sync_replication using Galera and EC2Scaling with sync_replication using Galera and EC2
Scaling with sync_replication using Galera and EC2
 
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYCScalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
Scalable Web Architectures: Common Patterns and Approaches - Web 2.0 Expo NYC
 
High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013High performance Infrastructure Oct 2013
High performance Infrastructure Oct 2013
 
Scaling HDFS for Exabyte Storage@twitter
Scaling HDFS for Exabyte Storage@twitterScaling HDFS for Exabyte Storage@twitter
Scaling HDFS for Exabyte Storage@twitter
 
Handling Massive Writes
Handling Massive WritesHandling Massive Writes
Handling Massive Writes
 

Último

Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 

Último (20)

Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 

Big data at CallFire

  • 1. Big Data at CallFire Vijesh Mehta (Co-Founder and CTO)
  • 2. Agenda •  A little about CallFire •  CallFire’s technical challenges •  How CallFire deals with data •  Summary
  • 3. Some background about myself •  I am one of the founders of CallFire. –  Started in 2005 in a small apartment –  Now 28 people –  Bootstrapped and profitable •  I’ve been writing software primarily in the Java space for 12 years. CallFire is all Java. –  We use : Wicket, Guice, Hibernate, MySQL, Cassandra, ActiveMQ, XEN, Puppet
  • 4. About CallFire •  We are a cloud telephony provider. –  Outbound Phone calls –  Phone Numbers –  SMS through long and short codes –  IVR – Interactive Voice Response –  Power Dialing •  CallFire’s call volume can get large very quickly. –  Hurricane Sandy : 1.9 million emergency calls •  4 Engineers and 1 System admin managing operations and new features. •  We just hired 7 more engineers this year, and still hiring!
  • 5. Technical Challenges by Numbers •  1.4 billion calls and texts –  Growing exponentially •  Over 50,000 accounts •  Over 6 million campaigns •  80 million sound files •  14 TB in storage (NFS) •  MySQL : Over 10,000 qps at peak Big data isn’t always big company problem!
  • 6. Growing faster each day Campaigns  over  Time   7000000   6000000   5000000   4000000   3000000   2000000   1000000   0  
  • 7. The first challenge •  Problem : We outgrew our datacenter. New systems need access to central storage. Replication across a 1gb/s interconnect. •  Needed Solution: –  Must work across datacenter –  Must scale as demand increases –  Must be fault tolerant –  Must deal with over 80 million sound files –  Cheaper the better
  • 8. Solutions Considered (2010) NFS   GLUSTER   HDFS   CASSANDRA   Fault  Tolerant   Yes,  if  configured   Yes   Yes   Yes   Datacenter   Maybe.  Rsync  isn’t   Not  at  the  Dme   Yes   Yes   Replica>on   fun  with  lots  of   files.   Easy  to  add  storage   No   Not  at  the  Dme   Yes   Yes   No  Single  point  of   No   Yes   Not  exactly,   Yes   failure   NameNode.   Data  always   No,  hard  to  sort   No,  same  as  a  file   Yes   Yes   accessible  easily   through  file   system   systems.     Notes   Not  working  for  us.   Looks  good,  tried  it   Didn’t  like  the  name   Everything  we   Too  much   for  a  while.  Easy  at   node  issue.  May   need,  quick  to   management  and   first  because  it  was   have  been  a  good   learn.  We  went  all   downDme.   a  file  system.   way  to  go.   in!   *  Only  LAN  soluDons  considered.  Calls  had  too  much  latency  in  the  cloud,  or  even   across  datacenter.  
  • 9. Cassandra •  Storage isn’t the best use of Cassandra. •  Do not exceed 50% of drive space. –  Compaction needs the space. Hard lesson learned. •  Fault Tolerance: Replication factor of 3. •  Result •  1 TB of data = 6 TB of storage needed! •  CallFire has a 74TB Cassandra Cluster
  • 10. Extending the scope •  We like SQL and Hibernate. –  Pros: Easy, Flexible, Ad-Hoc Queries, Locks –  Cons: Scaling •  Solution: Sharding with Cassandra for universal data Shard  1   Shard  2   Shard  3   Cassandra  Cluster  
  • 11. Sharding + Big Data •  Cassandra makes sharding easier –  Easy to store universal data. (Authentication) –  Performs very well •  Tungsten Replicator (Big Data with SQL) –  Sharding makes joins impossible, so fan your data into central places. –  NoSQL can’t handle ad-hoc queries. No worries, you can still have SQL.
  • 12. Big Data Summary •  Not Just for big companies, data grows rapidly in todays environment. –  Nice article about Obama’s Data Crunchers: –  http://swampland.time.com/2012/11/07/inside-the-secret-world-of-quants-and-data-crunchers-who-helped-obama-win/ •  NoSQL systems have easier scaling and fault tolerance mechanisms. –  Not uncommon to see small teams with 10-20 node clusters. •  SQL is still a big part of the equation. (Tungsten) –  Fan in information across partitions –  Replicate across datacenters –  Keep your ad-hoc dreams alive!
  • 13. Passive / Archived Storage Backblaze  –  $5,300  for  empty  case.  Holds  45  Drives  (117TB  usable  space)   hUp://www.protocase.com/products/index.php?e=Backblaze