Content Identification using HBase

•Descargar como PPTX, PDF•

3 recomendaciones•3,819 vistas

Speaker: Daniel Nelson (Nielsen) The motivation behind content identification is to determine the media people are consuming (via TV shows, movies, or streaming). Nielsen collects that data via its Fingerprints system, which generates significant amounts of structured data that is stored in HBase. This presentation will review the options a developer has for HBase querying and retrieval of hash data. Also covered is the use of wire protocols (Protocol Buffers), and how they can improve network efficiency and throughput, especially when combined with an HBase coprocessor.

Software Tecnología

CONTENT IDENTIFICATION
USING HBASE
Daniel Nelson
3-10-2014

2
Copyright©2013TheNielsenCompany.Confidentialandproprietary.
TV MEASUREMENT HISTORY
Traditional TV Video On
Demand
Time Shifted
Viewing
Internet-Based
Content
What are people watching?

3
Copyright©2013TheNielsenCompany.Confidentialandproprietary.
WHY IDENTIFY CONTENT?
Broadcasters:
What are people watching?
Advertisers:
How can I focus my spend?
Movies
Commercials
Programs Streaming

4
Copyright©2013TheNielsenCompany.Confidentialandproprietary.
WHAT ARE AUDIO FINGERPRINTS?
Program Audio
Audio
Fingerprints

5
Copyright©2013TheNielsenCompany.Confidentialandproprietary.
BUILDING A LIBRARY OF CONTENT
Nielsen Remote Media
Monitoring Sites
Audio
Movies
Commercials
Programs Streaming
Fingerprint
Generator Fingerprints
HBase Region
Servers

Copyright©2013TheNielsenCompany.Confidentialandproprietary.
6
NEED FOR HBASE
• Rapidly Growing Content
• Monolithic Limitations
• Storage Scalability
• Distributed Computations

7
Copyright©2013TheNielsenCompany.Confidentialandproprietary.
COLLECTING VIEWING DATA

Copyright©2013TheNielsenCompany.Confidentialandproprietary.
8
CONTENT IDENTIFICATION
Matching Process identifies Content by comparing
Unknown Fingerprints (Left) against Reference Fingerprints (Right).
Match
Unknown Fingerprints
Reference Fingerprints
ESPN
QVC
SyFy
BNN

Copyright©2013TheNielsenCompany.Confidentialandproprietary.
9
FINGERPRINTS AND HBASE
• Think of HBase as One Big Hash Table
• Fingerprints – Fit Nicely into HBase as the Key
• Keys Are Not 100% Unique – Collisions Without Loss?
• Near Constant Time Lookups
• Hash Table Load Factor will impact this ( n/k )

10
Copyright©2013TheNielsenCompany.Confidentialandproprietary.
REFERENCE HOUSE KEEPING
• Maintaining a Moving Window of Relevant Data
• Broadcast Reference Fingerprints Expire after 8 Days.
• Managing of 20+ Billion Hash Keys is no Small Task
• HBase TTL (Time To Live)
• Places an Expiration Date on All Table Data
• Hides Expired Data From Queries
• Purged on Next Compaction Cycle

11
Copyright©2013TheNielsenCompany.Confidentialandproprietary.
OPTIMIZING USE OF HBASE
• Network
• Fastest Network
• Bonded 1Gig Ethernet
• Reduce Data Volume in Network Transfers
• Protobufs – Google Protocol Buffers
• HBase Co-Processors
• Your Code Running on Region Servers
• Computations, Advanced Filtering, Transformations…

12
Copyright©2013TheNielsenCompany.Confidentialandproprietary.
HBase Region Servers
HBASE ENDPOINT CO-PROCESSORS
• Push your Business Logic into the Coprocessor.
• Keeping Co-Processor Code Simple
• Loading your Co-Processor
HBase Client
Application
Co-Processor
Co-Processor
Co-Processor
Co-Processor
Query/Response

13
Copyright©2013TheNielsenCompany.Confidentialandproprietary.
QUERYING HBASE
• Table.scan (…)
• Table.get (…)
• Table.get(List<Get>keyList)
• Query using a Co Processor
• coprocessorExec( yourProtocolClass, null, null, … )
• coprocessorExec( yourProtocolClass, startKey, endKey, … )

14
Copyright©2013TheNielsenCompany.Confidentialandproprietary.
STORING MILLIONS OF FILES
• Have a lot of Files to Store, use SAN/NAS/HDFS Right?
• SAN/NAS
• Costly
• More Hardware to Buy/Maintain
• HDFS
• Limited File Count
• Sequence Files
• Immutable
• Inefficient to Retrieve, Delete, Modify or Add Files...
• There is another way….

15
Copyright©2013TheNielsenCompany.Confidentialandproprietary.
LEVERAGING WHAT ALREADY WORKS
Example File To Store: /foo/bar/myFile.bin
• HBase Key = File Path: /foo/bar/
• Qualifier = File Name: myFile.bin
• Value = Your File Data (Serialized Object, Text, etc…)
• “List” Files - Scan using built in KeyOnlyFilter
• Wrap this with an API and your Application can use “HBase FS”
• File Retrieval, Delete, Modification, Updates, Versions
• Apply TTL to Purge Files

Más contenido relacionado

La actualidad más candente

HBaseCon 2013: Using Coprocessors to Index Columns in an Elasticsearch Cluster

Cloudera, Inc.

Speaker: Varun Sharma (Pinterest) Over the past year, HBase has become an integral component of Pinterest's storage stack. HBase has enabled us to quickly launch and iterate on new products and create amazing pinner experiences. This talk briefly describes some of these applications, the underlying schema, and how our HBase setup stays highly available and performant despite billions of requests every week. It will also include some performance tips for running on SSDs. Finally, we will talk about a homegrown serving technology we built from a mashup of HBase components that has gained wide adoption across Pinterest.

Large-scale Web Apps @ Pinterest

HBaseCon

HBaseCon 2015: Graph Processing of Stock Market Order Flow in HBase on AWS

HBaseCon

YapMap is a new kind of search platform that does multi-quanta search to better understand threaded discussions. This talk will cover how HBase made it possible for two self-funded guys to build a new kind of search platform. We will discuss our data model and how we use row based atomicity to manage parallel data integration problems. We’ll also talk about where we don’t use HBase and instead use a traditional SQL based infrastructure. We’ll cover the benefits of using MapReduce and HBase for index generation. Then we’ll cover our migration of some tasks from a message based queue to the Coprocessor framework as well as our future Coprocessor use cases. Finally, we’ll talk briefly about our operational experience with HBase, our hardware choices and challenges we’ve had.

HBaseCon 2012 | Building a Large Search Platform on a Shoestring Budget

Cloudera, Inc.

Speakers: Jesse Yates (Salesforce.com), Demai Ni, Richard Ding & Jing Chen He (IBM) This talk provides an overview of enterprise-scale backup strategies for HBase: Jesse Yates will describe how Salesforce.com runs backup and recovery on its multi-tenant, enterprise scale HBase deploys; Demai Ni, Songqinq Ding, and Jing Chen of the IBM InfoSphere BigInsights development team will then follow with a description of IBM's recently open-sourced disaster/recovery solution based on HBase snapshots and replication.

HBase Backups

HBaseCon

HBaseCon 2015 General Session: State of HBase

HBaseCon

HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBase

Cloudera, Inc.

HBaseCon 2013:High-Throughput, Transactional Stream Processing on Apache HBase

Cloudera, Inc.

HBase brings interactivity to Hadoop, and allows users to collect, manage and process data in real-time. Lily wraps HBase and Solr in a comprehensive Big Data platform, with HBase-native secondary indexing complementing ad-hoc structured search. Through spare write-cycles during read operations, Lily transforms HBase in an scalable data management engine providing interactive analytics, profile harvesting and real-time recommendations. This talk highlights the architecture of Lily, how it completes HBase, and explains some of its implementation use cases.

HBaseCon 2012 | Getting Real about Interactive Big Data Management with Lily ...

Cloudera, Inc.

HBaseConAsia2018 Track2-1: Kerberos-based Big Data Security Solution and Prac...

Michael Stack

Managing Hadoop, HBase and Storm Clusters at Yahoo Scale

DataWorks Summit/Hadoop Summit

HBaseCon 2015 General Session: The Evolution of HBase @ Bloomberg

HBaseCon

Apache HBase: State of the Union

DataWorks Summit/Hadoop Summit

HBaseCon 2012 | Real-Time and Batch HBase for Healthcare at Explorys

Cloudera, Inc.

The Hadoop Distributed File System is the foundational storage layer in typical Hadoop deployments. Performance and stability of HDFS are crucial to the correct functioning of applications at higher layers in the Hadoop stack. This session is a technical deep dive into recent enhancements committed to HDFS by the entire Apache contributor community. We describe real-world incidents that motivated these changes and how the enhancements prevent those problems from reoccurring. Attendees will leave this session with a deeper understanding of the implementation challenges in a distributed file system and identify helpful new metrics to monitor in their own clusters.

Hdfs 2016-hadoop-summit-san-jose-v4

Chris Nauroth

Empower Data-Driven Organizations

DataWorks Summit/Hadoop Summit

Splice Machine Overview

Kunal Gupta

Data is the fuel for the idea economy, and being data-driven is essential for businesses to be competitive. HPE works with all the Hadoop partners to deliver packaged solutions to become data driven. Join us in this session and you’ll hear about HPE’s Enterprise-grade Hadoop solution which encompasses the following -Infrastructure – Two industrialized solutions optimized for Hadoop; a standard solution with co-located storage and compute and an elastic solution which lets you scale storage and compute independently to enable data sharing and prevent Hadoop cluster sprawl. -Software – A choice of all popular Hadoop distributions, and Hadoop ecosystem components like Spark and more. And a comprehensive utility to manage your Hadoop cluster infrastructure. -Services – HPE’s data center experts have designed some of the largest Hadoop clusters in the world and can help you design the right Hadoop infrastructure to avoid performance issues and future proof you against Hadoop cluster sprawl. -Add-on solutions – Hadoop needs more to fill in the gaps. HPE partners with the right ecosystem partners to bring you solutions such an industrial grade SQL on Hadoop with Vertica, data encryption with SecureData, SAP ecosystem with SAP HANA VORA, Multitenancy with Blue Data, Object storage with Scality and more.

Empower Data-Driven Organizations with HPE and Hadoop

DataWorks Summit/Hadoop Summit

HBaseConAsia2018 Track1-5: Improving HBase reliability at PInterest with geo ...

Michael Stack

HBaseCon 2015 General Session: Zen - A Graph Data Model on HBase

HBaseCon

La actualidad más candente (20)

HBaseCon 2013: Using Coprocessors to Index Columns in an Elasticsearch Cluster

Large-scale Web Apps @ Pinterest

HBaseCon 2015: Graph Processing of Stock Market Order Flow in HBase on AWS

HBaseCon 2012 | Building a Large Search Platform on a Shoestring Budget

HBase Backups

HBaseCon 2015 General Session: State of HBase

HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBase

HBaseCon 2013:High-Throughput, Transactional Stream Processing on Apache HBase

HBaseCon 2012 | Getting Real about Interactive Big Data Management with Lily ...

HBaseConAsia2018 Track2-1: Kerberos-based Big Data Security Solution and Prac...

Managing Hadoop, HBase and Storm Clusters at Yahoo Scale

HBaseCon 2015 General Session: The Evolution of HBase @ Bloomberg

Apache HBase: State of the Union

HBaseCon 2012 | Real-Time and Batch HBase for Healthcare at Explorys

Hdfs 2016-hadoop-summit-san-jose-v4

Empower Data-Driven Organizations

Splice Machine Overview

Empower Data-Driven Organizations with HPE and Hadoop

HBaseConAsia2018 Track1-5: Improving HBase reliability at PInterest with geo ...

HBaseCon 2015 General Session: Zen - A Graph Data Model on HBase

Destacado

Speakers: Enis Soztutar and Devaraj Das (Hortonworks) HBase has ACID semantics within a row that make it a perfect candidate for a lot of real-time serving workloads. However, single homing a region to a server implies some periods of unavailability for the regions after a server crash. Although the mean time to recovery has improved a lot recently, for some use cases, it is still preferable to do possibly stale reads while the region is recovering. In this talk, you will get an overview of our design and implementation of region replicas in HBase, which provide timeline-consistent reads even when the primary region is unavailable or busy.

HBase Read High Availability Using Timeline-Consistent Region Replicas

HBaseCon

Web archiving initiatives around the world capture ephemeral web content to preserve our collective digital memory. However, that requires scalable, responsive tools that support exploration and discovery of captured content. Here you'll learn about why Warcbase, an open-source platform for managing web archives built on HBase, is one such tool. It provides a flexible data model for storing and managing raw content as well as metadata and extracted knowledge, tightly integrates with Hadoop for analytics and data processing, and relies on HBase for storage infrastructure.

HBaseCon 2015: Warcbase - Scaling 'Out' and 'Down' HBase for Web Archiving

HBaseCon

Speaker: Bryan Beaudreault (HubSpot) Running HBase in real time in the cloud provides an interesting and ever-changing set of challenges -- instance types are not ideal, neighbors can degrade your performance, and instances can randomly die in unanticipated ways. This talk will cover what HubSpot has learned about running in production on Amazon EC2, how it handle DR and redundancy, and the tooling the team has found to be the most helpful.

Real-time HBase: Lessons from the Cloud

HBaseCon

Blackbird is a large-scale object store built at Rocket Fuel, which stores 100+ TB of data and provides real time access to 10 billion+ objects in a 2-3 milliseconds at a rate of 1 million+ times per second. In this talk (an update from HBaseCon 2014), we will describe Blackbird's comprehensive collections API and various examples of how it can be used to model collections like sets, maps, and aggregates on these collections like counters, etc. We will also illustrate the flexibility and power of the API by modeling custom collection types that are unique to the Rocket Fuel context.

HBaseCon 2015: Blackbird Collections - In-situ Stream Processing in HBase

HBaseCon

HBaseCon 2015: HBase Operations in a Flurry

HBaseCon

Speakers: Chris Huang and Scott Miao (Trend Micro) Trend Micro collects lots of threat knowledge data for clients containing many different threat (web) entities. Most threat entities will be observed along with relations, such as malicious behaviors or interaction chains among them. So, we built a graph model on HBase to store all the known threat entities and their relationships, allowing clients to query threat relationships via any given threat entity. This presentation covers what problems we try to solve, what and how the design decisions we made, how we design such a graph model, and the graph computation tasks involved.

A Graph Service for Global Web Entities Traversal and Reputation Evaluation B...

HBaseCon

HBaseCon 2013: Apache Drill - A Community-driven Initiative to Deliver ANSI S...

Cloudera, Inc.

Partha Saha and CW Chung (Visa) Visa has embarked on an ambitious multi-year redesign of its entire data platform that powers its business. As part of this plan, the Apache Hadoop ecosystem, including HBase, will now become a staple in many of its solutions. Here, we will describe our journey in rolling out a high-availability NoSQL solution based on HBase behind some of our prominent mobile offerings.

Rolling Out Apache HBase for Mobile Offerings at Visa

HBaseCon

HTrace is a new Apache incubator project which makes it much easier to diagnose and detect performance problems in HBase. It provides a unified view of the performance of requests, following them from their origin in the HBase client, through the HBase region servers, and finally into HDFS. System administrators can use a central web interface to query and view aggregate performance information for the whole cluster. This talk will cover the motivations for creating HTrace, its design, and some examples of how HTrace can help diagnose real-world HBase problems.

HBaseCon 2015: Solving HBase Performance Problems with Apache HTrace

HBaseCon

Update on OpenTSDB and AsyncHBase

HBaseCon

Digital Library Collection Management using HBase

HBaseCon

Speaker: Adam Warrington (Cloudera) The Kite SDK is a set of libraries and tools focused on making it easier to build systems on top of the Hadoop ecosystem. HBase support has recently been added to the Kite SDK Data Module, which allows a developer to model and access data in HBase consistent with how they would model data in HDFS using Kite. This talk will focus on Kite's HBase support by covering Kite basics and moving through the specifics of working with HBase as a data source. This feature overview will be supplemented by specifics of how that feature is being used in production applications at Cloudera.

HBase Data Modeling and Access Patterns with Kite SDK

HBaseCon

Speaker: Sudarshan Kadambi and Matthew Hunt (Bloomberg LP) Bloomberg is a financial data and analytics provider, so data management is core to what we do. There's tremendous diversity in the type of data we manage, and HBase is a natural fit for many of these datasets - from the perspective of the data model as well as in terms of a scalable, distributed database. This talk covers data and analytics use cases at Bloomberg and operational challenges around HA. We'll explore the work currently being done under HBASE-10070, further extensions to it, and how this solution is qualitatively different to how failover is handled by Apache Cassandra.

HBase at Bloomberg: High Availability Needs for the Financial Industry

HBaseCon

Apache HBase in the Enterprise Data Hub at Cerner

HBaseCon

Apache HBase Improvements and Practices at Xiaomi

HBaseCon

HBaseCon 2015: HBase @ CyberAgent

HBaseCon

Speakers: Andrew Purtell and Ramkrishna Vasudevan (Intel) HBase 0.98 introduces several new security features: visibility labels, cell ACLs, transparent encryption, and coprocessor framework changes. This talk will cover the new capabilities available in HBase 0.98+, the threat models and use cases they cover, how these features stack up against other data stores in the Apache big data ecosystem, and how operators and security architects can take advantage of them.

New Security Features in Apache HBase 0.98: An Operator's Guide

HBaseCon

HBaseCon 2013: Full-Text Indexing for Apache HBase

Cloudera, Inc.

Hive/HBase integration extends the familiar analytical tooling of Hive to cover online data stored in HBase. This talk will walk through the architecture for Hive/HBase integration with a focus on some of the latest improvements such as Hive over HBase Snapshots, HBase filter pushdown, and composite keys. These changes improve the performance of the Hive/HBase integration and expose HBase to a larger audience of business users. We'll also discuss our future plans for HBase integration.

HBaseCon 2015: Analyzing HBase Data with Apache Hive

HBaseCon

Speaker: Jonathan Natkins (WibiData) Many companies aspire to have 360-degree views of their data. Whether they're concerned about customers, users, accounts, or more abstract things like sensors, organizations are focused on developing capabilities for analyzing all the data they have about these entities. This talk will introduce the concept of entity-centric storage, discuss what it means, what it enables for businesses, and how to develop an entity-centric system using the open-source Kiji framework and HBase. It will also compare and contrast traditional methods of building a 360-degree view on a relational database versus building against a distributed key-value store, and why HBase is a good choice for implementing an entity-centric system.

Design Patterns for Building 360-degree Views with HBase and Kiji

HBaseCon

Destacado (20)

HBase Read High Availability Using Timeline-Consistent Region Replicas

HBaseCon 2015: Warcbase - Scaling 'Out' and 'Down' HBase for Web Archiving

Real-time HBase: Lessons from the Cloud

HBaseCon 2015: Blackbird Collections - In-situ Stream Processing in HBase

HBaseCon 2015: HBase Operations in a Flurry

A Graph Service for Global Web Entities Traversal and Reputation Evaluation B...

HBaseCon 2013: Apache Drill - A Community-driven Initiative to Deliver ANSI S...

Rolling Out Apache HBase for Mobile Offerings at Visa

HBaseCon 2015: Solving HBase Performance Problems with Apache HTrace

Update on OpenTSDB and AsyncHBase

Digital Library Collection Management using HBase

HBase Data Modeling and Access Patterns with Kite SDK

HBase at Bloomberg: High Availability Needs for the Financial Industry

Apache HBase in the Enterprise Data Hub at Cerner

Apache HBase Improvements and Practices at Xiaomi

HBaseCon 2015: HBase @ CyberAgent

New Security Features in Apache HBase 0.98: An Operator's Guide

HBaseCon 2013: Full-Text Indexing for Apache HBase

HBaseCon 2015: Analyzing HBase Data with Apache Hive

Design Patterns for Building 360-degree Views with HBase and Kiji

Más de HBaseCon

Zhiyong Bai As a high performance and scalable key value database, Zhihu use HBase to provide online data store system along with Mysql and Redis. Zhihu’s platform team had accumulated some experience in technology of container, and this time, based on Kubernetes, we build flexible platform of online HBase system, create multiple logic isolated HBase clusters on the shared physical cluster with fast rapid，and provide customized service for different business needs. Combined with Consul and DNS server, we implement high available access of HBase using client mainly written with Python. This presentation is mainly shared the architecture of online HBase platform in Zhihu and some practical experience in production environment. hbaseconasia2017 hbasecon hbase

hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes

HBaseCon

Jingcheng Du Apache Beam is an open source and unified programming model for defining batch and streaming jobs that run on many execution engines, HBase on Beam is a connector that allows Beam to use HBase as a bounded data source and target data store for both batch and streaming data sets. With this connector HBase can work with many batch and streaming engines directly, for example Spark, Flink, Google Cloud Dataflow, etc. In this session, I will introduce Apache Beam, and the current implementation of HBase on Beam and the future plan on this. hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: HBase on Beam

HBaseCon

Ashish Singhi HBase Disaster recovery solution aims to maintain high availability of HBase service in case of disaster of one HBase cluster with very minimal user intervention. This session will introduce the HBase disaster recovery use cases and the various solutions adopted at Huawei like. a) Cluster Read-Write mode b) DDL operations synchronization with standby cluster c) Mutation and bulk loaded data replication d) Further challenges and pending work hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: HBase Disaster Recovery Solution at Huawei

HBaseCon

Tianying Chang HBase is used to serve online facing traffic in Pinterest. It means no downtime is allowed. However, we were on HBase 94. To upgrade to latest version, we need to figure out a way to live upgrade while keeping Pinterest site live. Recently, we successfully upgrade 94 HBase cluster to 1.2 with no downtime. We made change to both Asynchbase and HBase server side. We will talk about what we did and how we did it. We will also talk about the finding in config and performance tuning we did to achieve low latency. hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest

HBaseCon

Mon-Fong Mike Jiang, Kuan-Yu Hubert Fan-Chiang and Tienyu Rebecca Lin 自2011年起，我們就開始使用HBase作為結構化大數據的儲存工具，主要是做為半導體製造設備參數的分析。為了有效進行數據查詢，我們開發Standard Query Language(SQL)的整合介面，最早的方式是(1)自行開發GUI操作介面及(2)透過自行定義SQL語法的方式進行，但是這樣會衍生出很多額外的工作，特別是SQL Parser與對應的HBase API的連結。為了解決此問題，我們解析了Hive QL Parser作為主要的核心，將此部分的原始碼整合進HareDB HBase Client之中，另外，也整合了HBase Coprocessor，可以加速查詢的進行，這個架構我們實際使用在數個半導體製造廠的大數據系統中，也展現了高查詢效率。除此之外，透過整合Kafka來處理串流數據的匯入，同時對於數據分析的呈現也加上Cube建立工具，這些都是實際開發大數據系統時陸續面對的問題與解決方法，我們將分享這一連串的系統開發過程。 hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: HareQL：快速HBase查詢工具的發展過程

HBaseCon

Xinxin Fan and Hongxiang Jiang First, we will give a brief introduction about the HBase service at Netease，include the basic cluster info and the key HBase service. And then we will talk same tips about the tuning practices for HBase. Last, we will introduce some improvements at the internal HBase version. hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: Apache HBase at Netease

HBaseCon

Qianxi Zhang 1. Hulu是美国最受欢迎的在线视频网站之一，Hulu Beijing是Hulu第二大研发中心。北京大数据基础架构团队负责整个公司的大数据基础架构的研发和运维。 2. HBase在Hulu的概况 3. HBase在Hulu的使用 4. 用户画像系统，存放所有用户的基本信息，用户行为，第三方DMP数据和机器学习结果标签(几十万个Qualifier)，Spark和Spark Streaming读写HBase数据，运行各种机器学习模型，为公司的视频推荐，精准广告和Marketing团队服务 5. HBase在Hulu的优化 hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: HBase在Hulu的使用和实践

HBaseCon

Xinyu Zhang, Xueliang Chen and Zheng Fan 基于HBase的大数据平台已经成为中国人寿新一代综合业务处理系统中非常重要的基础性数据平台。目前基于该平台已经整合了上百TB的数据，并将几亿客户的客户、业务、接触数据整合到一个统一的数据模型中，并基于此形成了上千个客户标签。同时，基于该平台为客户、营销员和内部管理人员提供了销售支持、客户服务、运营支持等多类应用。通过APP、网页等形式提供了多种信息的检索和查询，并通过深度学习模型提供了反欺诈等方面的数据应用。 hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: 基于HBase的企业级大数据平台

HBaseCon

Xingbo Peng, Nan Zhang and Bang Wen 1.规模现状 HBase在京东CTO体系中经历了数年的发展，集群规模已经达到3000+台，支持了京东600+业务系统，京东CTO体系的HBase集群，已经经历了多次618和双11的考验。京东CTO体系是HBase的重要用户。 2.应用的业务场景介绍HBase在京东的典型应用的业务，包括监控、风控、推荐、广告等 3.高可用改进介绍我们在HBase集群高可用方面做的一些工作，包括跨机房容灾、多租户-资源分组、集群安全等 4.运维实践主要介绍我们在HBase集群运维上的一些实践，包括：HBase集群监控系统Mummut、报警系统、HBase集群与大数据平台结合、业务运营及数据迁移等 5.未来展望介绍我们正在基于HBase做的及未来要做的一些工作，包括：kylin、phoenix和容器化部署等 hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: HBase at JD.com

HBaseCon

Shuaifeng Zhou When we do real-time data loading to HBase, we use put/putlist interface. After receiving put request, regionserver will write WAL, write data into memory store, flush memory store to disk-store, then compact files again and again. That precedure occupies too much resource and causing read/write performance decrease. To solve the problem, we provide a kind of near-line loading method and architecture, greatly increase the loading bandwidth, and decrease the influence to read operations. hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: Large scale data near-line loading method and architecture

HBaseCon

Jieshan Bi and Yanhui Zhong 1. CTBase: A light-weight HBase client for structured data. 1). Schematized table, more friendly for structured data storage. 2). Global secondary index for HBase. 3). HBase Query DSL. JSON based light-weight API. 4) Cluster table. Pre-joining with keys, a better solution for cross-table join queries from HBase. 2. Tagram: Distributed bitmap index implementation with HBase. 1). Distributed bitmap index for accelerating AD-HOC queries with low cardinality columns. 2). Powerful and flexible query API. 3). Tagram offers millisecond-level query latency. 3. CloudTable Service Introduction: HBase on Huawei cloud. hbaseconasia2017 hbasecon hbase https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#

hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei

HBaseCon

hbaseconasia2017: HBase Practice At XiaoMi

HBaseCon

hbaseconasia2017: hbase-2.0.0

HBaseCon

As HBase and Hadoop continue to become routine across enterprises, these enterprises inevitably shift priorities from effective deployments to cost-efficient operations. Consolidation of infrastructure, the sum of hardware, software, and system-administrator effort, is the most common strategy to reduce costs. As a company grows, the number of business organizations, development teams, and individuals accessing HBase grows commensurately, creating a not-so-simple requirement: HBase must effectively service many users, each with a variety of use-cases. This is problem is known as multi-tenancy. While multi-tenancy isn’t a new problem, it also isn’t a solved one, in HBase or otherwise. This talk will present a high-level view of the common issues organizations face when multiple users and teams share a single HBase instance and how certain HBase features were designed specifically to mitigate the issues created by the sharing of finite resources.

HBaseCon2017 Democratizing HBase

HBaseCon

HBase is used to serve online facing traffic in Pinterest. It means no downtime is allowed. However, we were on HBase 94. To upgrade to latest version, we need to figure out a way to live upgrade while keeping Pinterest site live. Recently, we successfully upgrade 94 HBase cluster to 1.2 with no downtime. We made change to both Asynchbase and HBase server side. We will talk about what we did and how we did it. We will also talk about the finding in config and performance tuning we did to achieve low latency.

HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest

HBaseCon

Hundreds of millions of people use Quora to find accurate, informative, and trustworthy answers to their questions. As it so happens, counting things at scale is both an important and a difficult problem to solve. In this talk, we will be talking about Quanta, Quora's counting system built on top of HBase that powers our high-volume near-realtime analytics that serves many applications like ads, content views, and many dashboards. In addition to regular counting, Quanta supports count propagation along the edges of an arbitrary DAG. HBase is the underlying data store for both the counting data and the graph data. We will describe the high-level architecture of Quanta and share our design goals, constraints, and choices that enabled us to build Quanta very quickly on top of our existing infrastructure systems.

HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase

HBaseCon

In the age of NoSQL, big data storage engines such as HBase have given up ACID semantics of traditional relational databases, in exchange for high scalability and availability. However, it turns out that in practice, many applications require consistency guarantees to protect data from concurrent modification in a massively parallel environment. In the past few years, several transaction engines have been proposed as add-ons to HBase; three different engines, namely Omid, Tephra, and Trafodion were open-sourced in Apache alone. In this talk, we will introduce and compare the different approaches from various perspectives including scalability, efficiency, operability and portability, and make recommendations pertaining to different use cases.

HBaseCon2017 Transactions in HBase

HBaseCon

HBaseCon2017 Highly-Available HBase

HBaseCon

In DiDi Chuxing Company, which is China’s most popular ride-sharing company. we use HBase to serve when we have a bigdata problem. We run three clusters which serve different business needs. We backported the Region Grouping feature back to our internal HBase version so we could isolate the different use cases. We built the Didi HBase Service platform which is popular amongst engineers at our company. It includes a workflow and project management function as well as a user monitoring view. Internally we recommend users use Phoenix to simplify access.even more,we used row timestamp;multidimensional table schema to slove muti dimension query problems C++, Go, Python, and PHP clients get to HBase via thrift2 proxies and QueryServer. We run many important buisness applications out of our HBase cluster such as ETA/GPS/History Order/API metrics monitoring/ and Traffic in the Cloud. If you are interested in any aspects listed above, please come to our talk. We would like to share our experiences with you.

HBaseCon2017 Apache HBase at Didi

HBaseCon

HBaseCon2017 gohbase: Pure Go HBase Client

HBaseCon

Más de HBaseCon (20)

hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes

hbaseconasia2017: HBase on Beam

hbaseconasia2017: HBase Disaster Recovery Solution at Huawei

hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest

hbaseconasia2017: HareQL：快速HBase查詢工具的發展過程

hbaseconasia2017: Apache HBase at Netease

hbaseconasia2017: HBase在Hulu的使用和实践

hbaseconasia2017: 基于HBase的企业级大数据平台

hbaseconasia2017: HBase at JD.com

hbaseconasia2017: Large scale data near-line loading method and architecture

hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei

hbaseconasia2017: HBase Practice At XiaoMi

hbaseconasia2017: hbase-2.0.0

HBaseCon2017 Democratizing HBase

HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest

HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase

HBaseCon2017 Transactions in HBase

HBaseCon2017 Highly-Available HBase

HBaseCon2017 Apache HBase at Didi

HBaseCon2017 gohbase: Pure Go HBase Client

Último

We specialize in Psychic Readings, Psychic Love Spells, Binding Love Spells, Obsession Spells, Voodoo Spells, Lottery Spells, Marriage Spells, Black Magic Spells, Palm Readings & much more. Are you depressed? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? Do u need to solve any relationship problem? Contact the powerful spells caster chief kule with love spells that work overnight and love spells that really work. Have you found yourself infatuated with a special someone you think could be the one? Are you looking for a spell to provide them with a nudge in the right direction? Or maybe the spell you cast didn’t achieve the results you were hoping for? Whether you’re new or versed in the ways of spell casting, we’re here to help. Today we’re going to provide you with a detailed guide on the types of love spells to cast. Not only that but there’s something for those who wish to find outside advice from more advanced spell casters. We’re also going to provide you with the top sites available to help you with your dilemma. Let’s begin our journey by educating ourselves on love magic and what a real love caster looks like. Love Magic and Love Casters Love magic made its first appearance back in Ancient Egypt and has been an active practice since. This type of magic is a branch of traditional magic and can be practiced in various ways. Typically the more common use of love magic is through the work of spells, but other methods look like Charms Rituals-LOVE Potions-Dolls and even Amulets If you are interested in becoming a love caster, be prepared for what’s to come. A genuine love caster knows that the art of love casting is no easy feat and shouldn’t be done casually. You should know that not only does it require you to be gifted spiritually, but you must be ready to serve others. Someone who is considered a real love caster has experience in all manner of spells, no matter the difficulty. Training yourself in attraction, commitment, and marriage spells is an excellent place to start. But this by no means will make you a professional. Practice your craft and expand your knowledge; understand that you will possess the ability to help others in time truly. Types of Love Spells What better way to start broadening your experiences with love spells than by learning more about them? These spells work like just about any other spell. Simply apply your intention, use a medium (sigils, mantras, candles, or charm bags), and top it off with establishing the belief that you will receive what you want. So what kind of spells are available and which ones suit your needs the best? Let’s take a look at the many options you have at your disposal.

%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...

masabamasaba

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

masabamasaba

SHRMPro HRMS Software Solutions Presentation

Shrmpro

Software Quality Assurance Interview Questions

Arshad QA

Direct Style Effect Systems -The Print[A] Example- A Comprehension Aid

Philip Schwarz

10 Trends Likely to Shape Enterprise Technology in 2024

Mind IT Systems

%in Harare+277-882-255-28 abortion pills for sale in Harare

masabamasaba

Model Call Girl Services in Delhi reach out to us at 🔝 9953056974 🔝✔️✔️ Our agency presents a selection of young, charming call girls available for bookings at Oyo Hotels. Experience high-class escort services at pocket-friendly rates, with our female escorts exuding both beauty and a delightful personality, ready to meet your desires. Whether it's Housewives, College girls, Russian girls, Muslim girls, or any other preference, we offer a diverse range of options to cater to your tastes. We provide both in-call and out-call services for your convenience. Our in-call location in Delhi ensures cleanliness, hygiene, and 100% safety, while our out-call services offer doorstep delivery for added ease. We value your time and money, hence we kindly request pic collectors, time-passers, and bargain hunters to refrain from contacting us. Our services feature various packages at competitive rates: One shot: ₹2000/in-call, ₹5000/out-call Two shots with one girl: ₹3500/in-call, ₹6000/out-call Body to body massage with sex: ₹3000/in-call Full night for one person: ₹7000/in-call, ₹10000/out-call Full night for more than 1 person: Contact us at 🔝 9953056974 🔝. for details Operating 24/7, we serve various locations in Delhi, including Green Park, Lajpat Nagar, Saket, and Hauz Khas near metro stations. For premium call girl services in Delhi 🔝 9953056974 🔝. Thank you for considering us!

CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE

9953056974 Low Rate Call Girls In Saket, Delhi NCR

%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...

masabamasaba

(Vivek)Call Us, 8448380779,Call girls in Delhi NCr – We Offer best in class call girls. escort Service At Affordable Price At low Rate with Space Night 8000 We Are One Of The Oldest Escort and Call girls Agencies in Delhi. You Will Find That Our Female Escorts Are Full Of Fun, Sexy And They Would Love Enjoy Your Company. We Have A Fantastic Selection Of Escort Ladies Available For In-Calls As Well As Out-Calls. Our Escorts Are Not Only Beautiful But All Have Great Personalities Making Them The Perfect Companion For Any Occasion. In-Call:- You Can Come At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call:- You have To Come Pick The Girl From My Place We Are Also Provide Door Step Services (Delhi Ncr, Noida, Gurgaon, Faridabad, Ghaziabad Note:- Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic:- Full Ac room And Clean Rooms Available In Hotel 24 * 7 Hourly In Delhi NCR More Details, With WhatsApp Number, +91-8448380779

call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️

Delhi Call girls

%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...

masabamasaba

Data spaces in distributed environments should be allowed to evolve in agile ways providing data space owners with large flexibility about which data they store. Agility and heterogeneity, however, jeopardize data exchanges because representations may build on varying ontologies and data consumers may not rely on the semantic correctness of their queries in the context of semantically heterogeneous, evolving data spaces. Graph data spaces are one example of a powerful model for representing and querying data whose semantics may change over time. To assert and enforce conditions on individual graph data spaces, shape languages (e.g SHACL) have been developed. We investigate the question of how querying and programming can be guarded by reasoning over SHACL constraints in a distributed setting and we sketch a picture of how a future landscape based on semantically heterogeneous data spaces might look like.

Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...

Steffen Staab

Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...

SelfMade bd

In today's dynamic e-commerce landscape, the payment gateway emerges as a linchpin, ensuring smooth and secure transactions between buyers and sellers. In this discourse, we delve into the meticulous process of devising test cases tailored for scrutinizing payment gateways. Crafting precise test cases for payment gateways is a quintessential responsibility for testers operating within the service industry. This article meticulously explores pivotal scenarios integral to how to test payment gateways, coupled with essential guidelines for drafting effective test cases.

Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf

kalichargn70th171

The title is not connected to what is inside

shinachiaurasa2

Generic or specific? Making sensible software design decisions

Bert Jan Schrijver

VTU technical seminar 8Th Sem on Scikit-learn

AmarnathKambale

%in Durban+277-882-255-28 abortion pills for sale in Durban

masabamasaba

+971565801893 Mtp-Kit (500MG) Prices » Dubai [(+971565801893**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Leen Whatsapp +971565801893 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971565801893''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971565801893' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Clinic in Abu Dhabi, United Arab Emirates.+971565801893

+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...

Health

Abortion Clinic And Abortion Pills For Sale in Oman MEDICAL ABORTION PILLS FOR SALE in Fujairah , Ajman, Al Ain Quality Abortion services provided by specialists. Women's health problems. A same day service, safe, legal & pain free Abortions. From 1 week to 5 months. We use tested and approved pills. It's 100% guaranteed & safe. No side effects at all. We also do free womb cleaning and free pills delivery. We sell pregnancy termination pills {ABORTION PILLS} Our service is ever private with all our clients. Call/WhatsApp:

%+27788225528 love spells in Vancouver Psychic Readings, Attraction spells,Br...

masabamasaba

Content Identification using HBase

1. CONTENT IDENTIFICATION USING HBASE Daniel Nelson 3-10-2014

3. 3 Copyright©2013TheNielsenCompany.Confidentialandproprietary. WHY IDENTIFY CONTENT? Broadcasters: What are people watching? Advertisers: How can I focus my spend? Movies Commercials Programs Streaming

5. 5 Copyright©2013TheNielsenCompany.Confidentialandproprietary. BUILDING A LIBRARY OF CONTENT Nielsen Remote Media Monitoring Sites Audio Movies Commercials Programs Streaming Fingerprint Generator Fingerprints HBase Region Servers

8. Copyright©2013TheNielsenCompany.Confidentialandproprietary. 8 CONTENT IDENTIFICATION Matching Process identifies Content by comparing Unknown Fingerprints (Left) against Reference Fingerprints (Right). Match Unknown Fingerprints Reference Fingerprints ESPN QVC SyFy BNN

9. Copyright©2013TheNielsenCompany.Confidentialandproprietary. 9 FINGERPRINTS AND HBASE • Think of HBase as One Big Hash Table • Fingerprints – Fit Nicely into HBase as the Key • Keys Are Not 100% Unique – Collisions Without Loss? • Near Constant Time Lookups • Hash Table Load Factor will impact this ( n/k )

10. 10 Copyright©2013TheNielsenCompany.Confidentialandproprietary. REFERENCE HOUSE KEEPING • Maintaining a Moving Window of Relevant Data • Broadcast Reference Fingerprints Expire after 8 Days. • Managing of 20+ Billion Hash Keys is no Small Task • HBase TTL (Time To Live) • Places an Expiration Date on All Table Data • Hides Expired Data From Queries • Purged on Next Compaction Cycle

11. 11 Copyright©2013TheNielsenCompany.Confidentialandproprietary. OPTIMIZING USE OF HBASE • Network • Fastest Network • Bonded 1Gig Ethernet • Reduce Data Volume in Network Transfers • Protobufs – Google Protocol Buffers • HBase Co-Processors • Your Code Running on Region Servers • Computations, Advanced Filtering, Transformations…

12. 12 Copyright©2013TheNielsenCompany.Confidentialandproprietary. HBase Region Servers HBASE ENDPOINT CO-PROCESSORS • Push your Business Logic into the Coprocessor. • Keeping Co-Processor Code Simple • Loading your Co-Processor HBase Client Application Co-Processor Co-Processor Co-Processor Co-Processor Query/Response

13. 13 Copyright©2013TheNielsenCompany.Confidentialandproprietary. QUERYING HBASE • Table.scan (…) • Table.get (…) • Table.get(List<Get>keyList) • Query using a Co Processor • coprocessorExec( yourProtocolClass, null, null, … ) • coprocessorExec( yourProtocolClass, startKey, endKey, … )

14. 14 Copyright©2013TheNielsenCompany.Confidentialandproprietary. STORING MILLIONS OF FILES • Have a lot of Files to Store, use SAN/NAS/HDFS Right? • SAN/NAS • Costly • More Hardware to Buy/Maintain • HDFS • Limited File Count • Sequence Files • Immutable • Inefficient to Retrieve, Delete, Modify or Add Files... • There is another way….

15. 15 Copyright©2013TheNielsenCompany.Confidentialandproprietary. LEVERAGING WHAT ALREADY WORKS Example File To Store: /foo/bar/myFile.bin • HBase Key = File Path: /foo/bar/ • Qualifier = File Name: myFile.bin • Value = Your File Data (Serialized Object, Text, etc…) • “List” Files - Scan using built in KeyOnlyFilter • Wrap this with an API and your Application can use “HBase FS” • File Retrieval, Delete, Modification, Updates, Versions • Apply TTL to Purge Files

16. THANK YOU

Content Identification using HBase

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (20)

Similar a Content Identification using HBase

Similar a Content Identification using HBase (20)

Más de HBaseCon

Más de HBaseCon (20)

Último

Último (20)

Content Identification using HBase