Adopting Hadoop to manage your Big Data is an important step, but not the end-solution to your Big Data challenges. Here are some of the additional considerations you must face: Choosing the right cloud for the job: The massive computing and storage resources that are needed to support Big Data applications make cloud environments an ideal fit, and more than ever, there is a growing number of choices of cloud infrastructure types and providers. Given the diverse options, and the dynamic environments involved, it becomes ever more important to maintain the flexibility for all your IT needs. Big Data is a complex beast: It involves many and different moving parts, in large clusters, and is continually growing and evolving. Managing such an environment manually is not a viable option. The question is, how can you achieve automation of all this complexity? The world beyond Hadoop: Big Data is not just Hadoop – there is a whole rapidly growing ecosystem to contend with, including NoSQL, data processing, analytics tools… As well as your own application services. How can you manage deployment, configuration, scaling and failover of all the different pieces, in a consistent way? In this session, you'll learn how to deploy and manage your Hadoop cluster on any Cloud, as well as manage the rest of your big data application stack using a new open source framework called Cloudify.

1. Thank you

2. @natishalom

3. About GigaSpaces
3

5. The Reality of Big Data..
2.7 ZB
Global Digital Data
0.5 Petabytes
43% think that data
analytics could be improved in their
Two years tweets organization if data analytics was part of
cloud services
66%
Plan to use Big Data/Cloud
5

6. Large ISV Case Study
• Application
– Call Center surveillance
• Background
– Previously – voice data
• Goal for a new system
– Monitor data & voice
– Multiple data sources
– Advanced correlations

7. Ever Growing Data
Deeper Correlation
Tight Performance

8. A Classic Case for..

9. A Typical Big Data System…

10. Business
Cost
Impact

11. Big Data
in the Cloud

12. Big Data in the Cloud- 3 Reasons
• Skills
– Do you really need/want this all in-
house?
• Huge amounts of external data.
Holger Kisker – Does it make sense to move and
manage all this data behind your
firewall?
• Focus on the value of your data
– Instead of big data management.

13. • Auto start VMs
• Install and configure
app components
• Monitor
• Repair
• (Auto) Scale
• Burst…

14. Running Bare-Metal for
high I/O workloads, Public
cloud for sporadic
workloads ..

15. • Consistent
Management
• Automation Through
the Entire Stack

19. http://code.zynga.com/2012/02/the-evolution-of-zcloud/

20. http://code.mixpanel.com/2010/11/08/amazon-vs-rackspace/

25. Realization:
What You
Really Care
about Is
App
Portability

27. Consistent Management
Recipes consistent description for running any app:
 What middleware services to run
 Dependencies between services
 How to install services
 Where application and service binaries are
 When to spawn or terminate instances
 How to monitor each of the services.
® Copyright 2012 GigaSpaces Ltd. All Rights
27
Reserved

29. Choosing the Right Cloud for the Job
compute {
template "SMALL_LINUX"
}
SMALL_LINUX : template{ SMALL_LINUX : template
imageId "1234" imageId "us-east-1/ami-76f0061f“
machineMemoryMB 3200 remoteDirectory "/home/ec2-user/gs-files“
hardwareId "103" machineMemoryMB 1600
remoteDirectory "/root/gs-files" hardwareId "m1.small"
localDirectory "upload" locationId "us-east-1"
keyFile "gigaPGHP.pem" localDirectory "upload"
options ([ keyFile "myKeyFile.pem"
"openstack.securityGroup" : "default",
"openstack.keyPair" : "gigaPGHP" options ([
]) "securityGroups" : ["default"]as
privileged true String[],
} "keyPair" : "myKeyFile"
])
overrides (["jclouds.ec2.ami-query":"",
"jclouds.ec2.cc-ami-query":""])
privileged true
}
® Copyright 2012 GigaSpaces Ltd. All Rights
29
Reserved

30. Automation across the stack
1 Upload your recipe.
2 Cloudify creates VM’s & installs agents
3 Agents install and manage your app
4 Cloudify automate the scaling

32. Demo Time – Storm on Demand..
32

34. RightScale

35. Amazon Elastic Map Reduce

36. Large ISV Case Study
• Application
– Call Center surveillance system
• Background
– Previously – voice data
• Goal for a new system
Monitor data & voice
Multiple data sources
Advanced correlations Mission
Accomplished

37. Additional Benefits
• True Cloud Economics
• One product -> any
Customer Environment
• Increased Agility

38. Try a simple Big Data Demo Yourself
launch.cloudifysource.org/d

39. http://www.cloudifysource.org
http://github.com/CloudifySource