Make sense of your big data - Pilato

•

1 recomendación•1,913 vistas

Codemotion

Slides from David Pilato talk @ codemotion roma 2014

Tecnología Empresariales

#elasticsearch
MAKE SENSE OF YOUR (BIG) DATA!
David Pilato
Technical advocate

!
elasticsearch.
@dadoonet

StartUp #elasticsearch
Source: http://www.csc.com/insights/flxwd/78931-big_data_just_beginning_to_explode
35.000.000.000.000.000 mb

StartUp #elasticsearch
Source: http://www.thebigdatainsightgroup.com/site/article/big-data-infographic

StartUp #elasticsearch
search = like % ?
SELECT

doc.*, country.*

FROM

doc, country

WHERE

doc.country_code = country.code AND

doc.date_doc > to_date('2011-12', 'yyyy-mm') AND

doc.date_doc < to_date('2012-01', 'yyyy-mm') AND

lower(country.name) = 'france' AND

lower(doc.comment) LIKE ‘%product%' AND
lower(doc.comment) LIKE ‘%david%';

StartUp #elasticsearch
elasticsearch ?
plug & play
REST/JSON
scalable
Apache 2 license
Lucene
elasticsearch

#elasticsearch
Start…
$ wget https://download.elasticsearch.org/elasticsearch/
elasticsearch/elasticsearch-1.1.0.tar.gz!
$ tar -xf elasticsearch-1.1.0.tar.gz!
$ ./elasticsearch-1.1.0/bin/elasticsearch!
[INFO ][node ][Ghost Maker] {1.1.0}[5645]: initializing

$#elasticsearch … and play! $ curl -XPUT localhost:9200/sessions/session/1 -d '{! "title" : "Elasticsearch",! "subtitle" : "Make sense of your (BIG) data !",! "date" : "2014-04-12T14:10:00",! "tags" : [ "elasticsearch", "codemotion", "bigdata" ],! "speakers" : [{! "first_name" : "David", ! "last_name" : "Pilato" ! }]! }'$

$#elasticsearch Search! $ curl http://localhost:9200/sessions/session/_search -d' { "query": { "multi_match": { "query": "elasticsearch codemotion david", "fields": [ "title^3", "tags^2", "speakers.first_name" ] } }, "post_filter": { "range": { "date": { "from": "2014-04-09", "to": "2014-04-13" } } } }'$

$#elasticsearch $ curl http://localhost:9200/sessions/session/_search -d' { "query": { ... }, "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } } }' "by_date": [ { "key_as_string": "03/04/2014", "doc_count": 1 }, { "key_as_string": "12/04/2014", "doc_count": 2 }, { "key_as_string": "16/04/2014", "doc_count": 3 } ] Compute!$

#mstechdays #elasticsearch StartUp #elasticsearch
• logs

• twitter

• github

• marketing data

• ...

• your data

• your big data
Let’s make sense of …

#mstechdays #elasticsearch StartUp #elasticsearch
• logs

• twitter

• github

• marketing data

• ...

• your data

• your big data
Let’s make sense of …
{
"name":"Pilato David",
"dateOfBirth":"1971-12-26",
"gender":"male",
"children":3,
"marketing":{
"fashion":334,
"music":3363,
"hifi":2351
},
"address":{
"country":"France",
"city":"Paris",
"location": [2.332395, 48.861871]
}
}

démo
#mstechdays #elasticsearch StartUp #elasticsearch
MAKE SENSE OF YOUR (BIG) DATA!
let’s inject some marketing documents…

$StartUp #elasticsearch Distributed indices node 1 orders products 1 2 3 4 1 2 $ curl -XPUT localhost:9200/orders -d '{! "settings.index.number_of_shards" : 4,! "settings.index.number_of_replicas" : 1! }' $ curl -XPUT localhost:9200/products -d '{! "settings.index.number_of_shards" : 2,! "settings.index.number_of_replicas" : 0! }'$

StartUp #elasticsearch
Distributed indices
node 1
orders
products
1 2
3 4
1 2
node 2
$ bin/elasticsearch!
[INFO ][cluster.service][Armageddon] detected_master [Ghost Maker]

StartUp #elasticsearch
Distributed indices
node 1
orders
products
1
4
1
node 2
orders
products
2
3
2
2
3
1
4
2
3
2

StartUp #elasticsearch
node 3
Distributed indices
node 1
orders
products
1
4
1
node 2
orders
products
2
3
2
2
3
1
4
$ bin/elasticsearch!
[INFO ][cluster.service][Karnak] detected_master [Ghost Maker]

StartUp #elasticsearch
node 3
products
orders
Distributed indices
node 1
orders
products
1
4
1
node 2
orders
products
2
33
2
2
3
1
4 3
1
4

elasticsearch.
elasticsearch
kibana
logstash
Marvel

elasticsearch.
Training (public and on-site)
Development support
Production support
Marvel

@dadoonet
questions ?
we are hiring! jobs@elasticsearch.com

Más contenido relacionado

Similar a Make sense of your big data - Pilato

David pilato codemotion-telaviv

Codemotion Tel Aviv

If you’re thinking about using a document database, it can be intimidating to start. A flexible data model gives you a lot of choices, but which way is the right way? Is a document database even the right tool? In this session we’ll go over the basics of data modeling using JSON. We’ll compare and contrast with traditional RDBMS modeling. Impact on application code will be discussed, as well as some tooling that could be helpful along the way. The examples use the free, open-source Couchbase Server document database, but the principles from this session can also be applied to CosmosDb, Mongo, RavenDb, etc.

JSON Data Modeling - July 2018 - Tulsa Techfest

Matthew Groves

The Business of Big Data (IA Ventures)

Ben Siscovick

Big Data Analytics - Best of the Worst : Anti-patterns & Antidotes

Krishna Sankar

Data Scientist has been regarded as the sexiest job of the twenty first century. As data in every industry keeps growing the need to organize, explore, analyze, predict and summarize is insatiable. Data Science is creating new paradigms in data driven business decisions. As the field is emerging out of its infancy a wide range of skill sets are becoming an integral part of being a Data Scientist. In this talk I will discuss the different driven roles and the expertise required to be successful in them. I will highlight some of the unique challenges and rewards of working in a young and dynamic field.

From Rocket Science to Data Science

Sanghamitra Deb

KDD 2019 IADSS Workshop - Research Updates from Usama Fayyad & Hamit Hamutcu

IADSS

Test trend analysis: Towards robust reliable and timely tests

Hugh McCamphill

Séminaire Big Data Alter Way - Elasticsearch - octobre 2014

ALTER WAY

Intro to Data Science

TJ Stalcup

The role of the Chief Data Officer (CDO) has become integral to the evolution needed to turn a wisdom-driven company into an analytics-driven company. With Data Governance at the core of your responsibility, moving the innovation meter is a global challenge among CDOs. Specifically the CDO must: • Provide a single point of accountability for data initiatives and issues • Innovate ways to use existing data and evangelize a data vision for the organization • Support & enforce data governance policies via outreach, training & tools • Work with IT to develop/maintain an enterprise data repository • Set standards for analytical reporting and generate data insights through data science In this session, Joe Caserta addresses real-word CDO challenges, shares techniques to overcome them, manage corporate disruption and achieve success.

Integrating the CDO Role Into Your Organization; Managing the Disruption (MIT...

Caserta

iTrain Malaysia: Data Science by Tarun Sukhani

iTrain

Open Data and News Analytics Demo

Ontotext

Embracing data science

Vipul Kalamkar

Chapter 4 marketing research pace

Nicholsb1

Dpo latin strategy 04132018

Ramón Luis Pizarro Romero

Presented virtually at GDG Indy - https://www.meetup.com/indy-gdg/events/269467916/ If you’re thinking about using a document database, it can be intimidating to start. A flexible data model gives you a lot of choices, but which way is the right way? Is a document database even the right tool? In this session we’ll go over the basics of data modeling using JSON. We’ll compare and contrast with traditional RDBMS modeling. Impact on application code will be discussed, as well as some tooling that could be helpful along the way. The examples use the free, open-source Couchbase Server document database, but the principles from this session can also be applied to CosmosDb, Mongo, RavenDb, etc.

JSON Data Modeling - GDG Indy - April 2020

Matthew Groves

Thinkful DC - Intro to Data Science

TJ Stalcup

Industry of Things World - Berlin 19-09-16

Boris Adryan

Why CxOs care about Data Governance; the roadblock to digital mastery

Coert Du Plessis (杜康)

Big Data in Practice.pdf

Tom Tan

Similar a Make sense of your big data - Pilato (20)

David pilato codemotion-telaviv

JSON Data Modeling - July 2018 - Tulsa Techfest

The Business of Big Data (IA Ventures)

Big Data Analytics - Best of the Worst : Anti-patterns & Antidotes

From Rocket Science to Data Science

KDD 2019 IADSS Workshop - Research Updates from Usama Fayyad & Hamit Hamutcu

Test trend analysis: Towards robust reliable and timely tests

Séminaire Big Data Alter Way - Elasticsearch - octobre 2014

Intro to Data Science

Integrating the CDO Role Into Your Organization; Managing the Disruption (MIT...

iTrain Malaysia: Data Science by Tarun Sukhani

Open Data and News Analytics Demo

Embracing data science

Chapter 4 marketing research pace

Dpo latin strategy 04132018

JSON Data Modeling - GDG Indy - April 2020

Thinkful DC - Intro to Data Science

Industry of Things World - Berlin 19-09-16

Why CxOs care about Data Governance; the roadblock to digital mastery

Big Data in Practice.pdf

Más de Codemotion

Fuzz-testing: A hacker's approach to making your code more secure | Pascal Ze...

Codemotion

Pompili - From hero to_zero: The FatalNoise neverending story

Codemotion

Pastore - Commodore 65 - La storia

Codemotion

Rivivere l'ebbrezza di progettare un vecchio computer o una consolle da bar è oggi possibile sfruttando le FPGA, ovvero logiche programmabili che consentono a chiunque di progettare il proprio hardware o di ricrearne uno del passato. In questa sessione si racconta come dal reverse engineering dell'hardware di vecchie glorie come il Commodore 64 e lo ZX Spectrum sia stato possibile farle rivivere attraverso tecnologie oggi alla portata di tutti.

Pennisi - Essere Richard Altwasser

Codemotion

There's a lot of talk about blockchain, but how does the technology behind it actually work? For developers, getting some hands-on experience is the fastest way to get familiair with new technologies. So let's build a blockchain, then! In this session, we're going to build one in plain old Java, and have it working in 40 minutes. We'll cover key concepts of a blockchain: transactions, blocks, mining, proof-of-work, and reaching consensus in the blockchain network. After this session, you'll have a better understanding of core aspects of blockchain technology.

Michel Schudel - Let's build a blockchain... in 40 minutes! - Codemotion Amst...

Codemotion

When was the last time you were truly lost? Thanks to the maps and location technology in our phones, a whole generation has now grown up in a world where getting lost is truly a thing of the past. Location technology goes far beyond maps in the palm of our hand, however. In this talk, we will explore how a ridesharing app works. How do we discover our destination?How do we find the closest driver? How do we display this information on a map? How do we find the best route?To answer these questions,we will be learning about a variety of location APIs, including Maps, Positioning, Geocoding etc.

Richard Süselbeck - Building your own ride share app - Codemotion Amsterdam 2019

Codemotion

Eward Driehuis, SecureLink's research chief, will guide you through the bumpy ride we call the cyber threat landscape. As the industry has over a decade of experience of dealing with increasingly sophisticated attacks, you might be surprised to hear more attacks slip through the cracks than ever. From analyzing 20.000 of them in 2018, backed by a quarter of a million security events and over ten trillion data points, Eward will outline why this happens, how attacks are changing, and why it doesn't matter how neatly or securely you code.

Eward Driehuis - What we learned from 20.000 attacks - Codemotion Amsterdam 2019

Codemotion

IoT revolution is ended. Thanks to hardware improvement, building an intelligent ecosystem is easier than never before for both startups and large-scale enterprises. The real challenge is now to connect, process, store and analyze data: in the cloud, but also, at the edge. We’ll give a quick look on frameworks that aggregate dispersed devices data into a single global optimized system allowing to improve operational efficiency, to predict maintenance, to track asset in real-time, to secure cloud-connected devices and much more.

Francesco Baldassarri - Deliver Data at Scale - Codemotion Amsterdam 2019 -

Codemotion

What if Virtual Reality glasses could transform your environment into a three-dimensional work of art in realtime in the style of a painting from Van Gogh? One of the many interesting developments in the field of Deep Learning is the so called "Style Transfer". It describes a possibility to create a patchwork (or pastiche) from two images. While one of these images defines the the artistic style of the result picture, the other one is used for extracting the image content. A team from TNG Technology Consulting managed to build an AI showcase using OpenCV and Tensorflow to realize such goggles.

Martin Förtsch, Thomas Endres - Stereoscopic Style Transfer AI - Codemotion A...

Codemotion

Blockchain (and Cryptocurrency) is an evolution of 20-year old research from scientists like Chaum, Lamport, and Castro & Liskov. Due to the current hype, it's hard to distinguish beneficial aspects of the technology from a desire for a "silver bullet" for device security, verifiable logistics, or "saving democracy". The problem: blockchain introduces new security challenges - and blind adoption without understanding reduces overall security. In this talk, Melanie Rieback and Klaus Kursawe explain the pitfalls and limits of blockchain, so you can avoid making your applications LESS secure.

Melanie Rieback, Klaus Kursawe - Blockchain Security: Melting the "Silver Bul...

Codemotion

Networking is a core part of computing in the digital world we inhabit. But, how well do you know how it works? Do you understand all the moving parts of the OSI stack inside your computer, and how the network is actually put together? How can this ever work? This guided safari of layers, standards, protocols, and happenstance will bring us close to the copper wire, and up through the layers of CDMA/CD, ARP, routing and HTTP. We will make a few excursions through patchworks that still work forty years later, and cleverly designed mechanisms that show that simplicity is the only way to last.

Angelo van der Sijpt - How well do you know your network stack? - Codemotion ...

Codemotion

Performance tests are not only an important instrument for understanding a system and its runtime environment. It is also essential in order to check stability and scalability – non-functional requirements that might be decisive for success. But won't my cloud hosting service scale for me as long as I can afford it? Yes, but… It only operates and scales resources. It won't automatically make your system fast, stable and scalable. This talk shows how such and comparable questions can be clarified with performance tests and how DevOps teams benefit from regular test practise.

Lars Wolff - Performance Testing for DevOps in the Cloud - Codemotion Amsterd...

Codemotion

Sascha will demonstrate the opportunities and challenges of Conversational AI learned from the practice. Both Technology and User Experience will be covered introducing a process finding micro-moments, writing happy paths, gathering intents, designing the conversational flow, and finally publishing on almost all channels including Voice Services and Chatbots. Valuable for enterprises, developers, and designers. All live on stage in just minutes and with almost no code.

Sascha Wolter - Conversational AI Demystified - Codemotion Amsterdam 2019

Codemotion

A key challenge we face at Pacmed is quickly calibrating and deploying our tools for clinical decision support in different hospitals, where data formats may vary greatly. Using Intensive Care Units as a case study, I’ll delve into our scalable Python pipeline, which leverages Pandas’ split-apply-combine approach to perform complex feature engineering and automatic quality checks on large time-varying data, e.g. vital signs. I’ll show how we use the resulting flexible and interpretable dataframes to quickly (re)train our models to predict mortality, discharge, and medical complications.

Michele Tonutti - Scaling is caring - Codemotion Amsterdam 2019

Codemotion

Coolblue is a proud Dutch company, with a large internal development department; one that truly takes CI/CD to heart. Empowerment through automation is at the heart of these development teams, and with more than 1000 deployments a day, we think it's working out quite well. In this session, Pat Hermens (a Development Managers) will step you through what enables us to move so quickly, which tools we use, and most importantly, the mindset that is required to enable development teams to deliver at such a rapid pace.

Pat Hermens - From 100 to 1,000+ deployments a day - Codemotion Amsterdam 2019

Codemotion

Quantum computers can use all of the possible pathways generated by quantum decisions to solve problems that will forever remain intractable to classical compute power. As the mega players vie for quantum supremacy and Rigetti announces its $1M "quantum advantage" prize, we live in exciting times. IBM-Q and Microsoft Q# are two ways you can learn to program quantum computers so that you're ready when the quantum revolution comes. I'll demonstrate some quantum solutions to problems that will forever be out of reach of classical, including organic chemistry and large number factorisation.

James Birnie - Using Many Worlds of Compute Power with Quantum - Codemotion A...

Codemotion

Chinese food exploded across America in the early 20th century, rapidly adapting to local tastes while also spreading like wildfire. How was it able to spread so fast? The GY6 is a family of scooter engines that has achieved near total ubiquity in Europe. It is reliable and cheap to manufacture, and it's made in factories across China. How are these factories able to remain afloat? Chinese-American food and the GY6 are both riveting studies in product-market fit, and both are the product of a distributed open source-like development model. What lessons can we learn for open source software?

Don Goodman-Wilson - Chinese food, motor scooters, and open source developmen...

Codemotion

Pieter Omvlee - The story behind Sketch - Codemotion Amsterdam 2019

Codemotion

Would you fly in a plane designed by a craftsman or would you prefer your aircraft to be designed by engineers? We are learning that science and empiricism works in software development, maybe now is the time to redefine what “Software Engineering” really means. Software isn't bridge-building, it is not car or aircraft development either, but then neither is Chemical Engineering. Engineering is different in different disciplines. Maybe it is time for us to begin thinking about retrieving the term "Software Engineering" maybe it is time to define what our "Engineering" discipline should be.

Dave Farley - Taking Back “Software Engineering” - Codemotion Amsterdam 2019

Codemotion

What is the job of a CTO and how does it change as a startup grows in size and scale? As a CTO, where should you spend your focus? As an engineer aspiring to be a CTO, what skills should you pursue? In this inspiring and personal talk, I describe my journey from early Red Hat engineer to CTO at Bloomon. I will share my view on what it means to be a CTO, and ultimately answer the question: Should the CTO be coding?

Joshua Hoffman - Should the CTO be Coding? - Codemotion Amsterdam 2019

Codemotion

Más de Codemotion (20)