This presentation is an overview of Big Data concepts and it tries to define a Big Data Tech Stack to meet your business needs.

1. Big Data
Tech Stack
Big Data 2015
by Abdullah Cetin CAVDAR

2. Me :)

3. Graduated from
@HU

4. PhD Student
@METU

5. Ex Entrepreneur
I had 3 start-ups

6. Senior Software
Engineer
@Udemy

7. Founder and Organizer of
meetup.com/ankara-big-data-meetup

8. What's Big Data
Big data is data that exceeds the processing capacity
of conventional database systems.

9. What's Big Data
Big data is when the data itself becomes part of the
problem.

10. 4V's of Big Data

18. Multitude of Data
Types
Structured
Semi-structured
Unstructured

19. Data Data Data

24. What We Need?
Store
Join
Index
Analytics
Aggregate
Visualize

25. Challenge
The challenge in big data analytics is to
dig deeply
quickly (real time?)
and widely

26. "ilities" or NFR?
Availability
Scalability
Security
Performance
...

27. Solution?

28. Big Data Tech
Stack

29. What're essential
components?

30. Data Sources

31. Multiple internal
& external
data sources

32. Creates a
data lake

36. Different
Volume, Variety,
Velocity

37. Aim is to create
a funnel after
proper validation
and cleaning

38. Ingestion Layer

39. Signal-to-Noise
ratio
10:90

40. separate the
noise from
relevant info

41. It has capability to
Validate
Cleanse
Transform
Reduce
Integrate

43. Distributed
Storage Layer

44. Fault tolerance
Parallelization

45. HDFS
massively scalable distributed
file system

46. HDFS

47. HDFS Architecture

48. Non-relational,
distributed data?

49. NoSQL

50. CAP theorem
Consistency, Availability,
Partition Tolerance

53. Ingestion to DFS
Sqoop, Flume, MapReduce, ETL

54. Infrastructure &
Platform Layer

55. Computing &
Scalability

56. Hadoop?

57. Vertical Scaling

58. Vertical Scaling

59. Vertical Scaling

60. Horizontal Scaling

61. Horizontal Scaling

62. Horizontal Scaling

64. MapReduce
is the main computation paradigm

65. MapReduce

67. Hadoop 2

68. What's new?

69. What's new?

70. H1 vs. H2

71. One cluster,
distributed storage,
distributed scheduler,
many types of applications.

72. Blueprints
NoSQL with HBase
Stream Processing with Storm/Spark
Graph Processing with Giraph
SQL on Hadoop with Impala
Columnar Data Formats

73. Security Layer

74. Data need to be protected
Meet compliance requirements
Individual's privacy

75. Proper
authorization and
authentication
needed

76. What can we do?
Authentication protocol like Kerberos
Enable file layer encryption
Use SSL, certificates and trusted keys
Provision with Chef, Puppet or Ansible like tools
Log all the communication for detecting anomalies
Monitor whole system

77. Monitoring Layer

78. Get a complete
picture
of our Big Data tech stack

79. Satisfy SLAs with
min downtime

80. DataDog

81. New Relic (Overview)

82. New Relic (Databases)

83. Analytics Engine

84. Co-Existence
with Traditional
BI
Data warehouse in the traditional way
Distributed MR processing on big data stores

85. Mediate data in either direction
i.e use Hive/HBase with Sqoop

86. Real-time analysis can leverage
low-latency NoSQL stores
i.e Cassandra, Vertica, ...

87. R may be used for complex
statistical algorithms

88. Search Engines

89. Huge volume and
variety of data

90. “needle in a
haystack”

92. Need blazing fast search
mechanism
to index and search for big data
analytics

94. Elastic Search,
Solr, ...

95. Real-time
Processing

96. In memory?

97. Apache Spark

101. Storm, Kinesis,
Flink, ...

102. Visualization
Layer

103. Gain insight faster
Look at different aspects of
data visually

105. Tableau

106. ChartIO

107. Lambda
Architecture

108. Lambda Architecture / MapR

109. Don't forget

110. There is no
"One Size Fits All"
solution

111. We need
Continuous
Development

114. Thank You :)