Trinity: A Distributed Graph Engine on a Memory Cloud

•Descargar como PPTX, PDF•

3 recomendaciones•948 vistas

Qian Lin

Educación Tecnología

Trinity: A Distributed Graph Engine
on a Memory Cloud
Speaker: LIN Qian
http://www.comp.nus.edu.sg/~linqian/

Graph applications
Online query processing  Low latency
Offline graph analytics  High throughput

Online queries
Random data access
e.g., BFS, sub-graph matching, …

Offline computations
Performed iteratively

Insight: Keeping the
graph in memory
at least the topology

Trinity
Online query + Offline analytics

Random data access problem
in large graph computation
Globally addressable distr. memory
Random access abstraction

Belief
High-speed network is more available
DRAM is cheaper
In-memory solution become practical

“Trinity itself is not a system that comes with
comprehensive built-in graph computation
modules.”

Stack of Trinity system modules
User define:
Graph schema, Communication protocols, Computation paradigms

Memory cloud
Partition memory space into trunks
Hashing

Memory trunks
2p > m
1. Trunk level parallelism
2. Efficient hashing

Hashing
Key-value store
p-bit value  i ∈ [0, 2p – 1]
Inner trunk hash table

Data partitioning and addressing
Benefit:
Scalability
Fault-tolerance

Modeling graph
Cell: value + schema
Represent a node in a cell

TSL
Object-oriented cell manipulation
Data integration
Network communication

Online queries
Traversal based
New paradigm

Vertex centric
offline analytics
Restrictive vertex centric model

Message passing
optimization
Create a bipartite partition of the
local graph
Buffer hub vertices

A new paradigm for
offline analytics
1. Aggregate answers from local
computations
2. Employ probabilistic inference

Circular memory management
• Aim to avoid memory gaps between large
number of key-value pairs

Fault tolerance
Heartbeat-based failure detection
BSP: checkpointing
Async.: “periodical interruption”

Más contenido relacionado

La actualidad más candente

A new class of Data Grid infrastructure is needed to support management, transport, distributed access, and analysis of terabyte and peta byte of data collections by thousands of users. Even though some of the existing data management systems (DMS) of Grid computing infrastructures provides methodologies to handle bulk data transfer. These technologies are not usable in addressing some kind of simultaneous data access requirements. Often, in most of the scientific computing environments, a common data will be needed to access from different locations. Further, most of such computing entities will wait for a common scientific data (such as a data belonging to an astronomical phenomenon) which will be published only when it is available. These kinds of data access needs were not yet addressed in the design of data component Grid Access to Secondary Storage (GASS) or GridFTP. In this paper, we address an application layer content based data transfer scheme for grid computing environments. By using the proposed scheme in a grid computing environment, we can simultaneously move bulk data in an efficient way using simple subscribe and publish mechanism.

CONTENT BASED DATA TRANSFER MECHANISM FOR EFFICIENT BULK DATA TRANSFER IN GRI...

ijgca

Java Abs Peer To Peer Design & Implementation Of A Tuple Space

ncct

In the last decade Peer to Peer technology has been thoroughly explored, becauseit overcomes many limitations compared to the traditional client server paradigm. Despite its advantages over a traditional approach, the ubiquitous availability of high speed, high bandwidth and low latency networks has supported the traditional client-server paradigm. Recently, however, the surge of streaming services has spawned renewed interest in Peer to Peer technologies. In addition, services like geolocation databases and browser technologies like Web-RTC make a hybrid approach attractive. In this paper we present algorithms for the construction and the maintenance of a hybrid P2P overlay multicast tree based on topological distances. The essential idea of these algorithms is to build a multicast tree by choosing neighbours close to each other. The topological distances can be easily obtained by the browser using the geolocation API. Thus the implementation of algorithms can be done web-based in a distributed manner. We present proofs of our algorithms as well as experimental results and evaluations.

Evaluation of a topological distance

IJCNCJournal

In data-intensive applications data transfer is a primary cause of job execution delay. Data access time depends on bandwidth. The major bottleneck to supporting fast data access in Grids is the high latencies of Wide Area Networks and Internet. Effective scheduling can reduce the amount of data transferred across the internet by dispatching a job to where the needed data are present. Another solution is to use a data replication mechanism. Objective of dynamic replica strategies is reducing file access time which leads to reducing job runtime. In this paper we develop a job scheduling policy and a dynamic data replication strategy, called HRS (Hierarchical Replication Strategy), to improve the data access efficiencies. We study our approach and evaluate it through simulation. The results show that our algorithm has improved 12% over the current strategies

The Impact of Data Replication on Job Scheduling Performance in Hierarchical ...

graphhoc

Hpc 3

Yasir Khan

Survey on Synchronizing File Operations Along with Storage Scalable Mechanism

IRJET Journal

Recently graph data rises in many applications and there is need to manage such large amount of data by performing various graph operations over graphs using some graph search queries. Many approaches and algorithms serve this purpose but continuously require improvement over it in terms of stability and performance. Such approaches are less efficient when large and complex data is involved. Applications need to execute faster in order to improve overall performance of the system and need to perform many advanced and complex operations. Shortest path estimation is one of the key search queries in many applications. Here we present a system which will find the shortest path between nodes and contribute to performance of the system with the help of different shortest path algorithms such as bidirectional search and AStar algorithm and takes a relational approach using some new standard SQL queries to solve the problem, utilizing advantages of relational database which solves the problem efficiently.

Shortest path estimation for graph

ijdms

Effective Data Retrieval System with Bloom in a Unstructured p2p Network

Uvaraj Shan

Caching in Information Centric Network (ICN)

Lippo Group Digital

Parallel and Distributed Information Retrieval System

vimalsura

Lately, the Wireless Sensors Networks (WSN) have moved to the concept of the hybrids networks in order to get universal platforms in various types of monitoring and information collecting applications. The work presented in this paper aims in designing a hybrid remote monitoring architecture, largely secured by a high availability and resilience WSN. The modeling approach intends to describe the main operation of polling and dispatching between the communications channels with the purpose of ensuring the information availability and reducing the resilience time. To achieve our goal, we have realized an experimental platform of measuring, processing and routing data through hybrid communications technologies. We have illustrated, via curves, the routing of the data measured by a WSN (ZigBee Technology) to a final user through several communication technologies (HTTPS, SMS, ...).

Heterogeneous Networks of Remote Monitoring with High Availability and Resili...

IJCSIS Research Publications

Classification of Computer Networks

ShohanaakterKakon

A plethora of infinite data is generated from the Internet and other information sources. Analyzing this massive data in real-time and extracting valuable knowledge using different mining applications platforms have been an area for research and industry as well. However, data stream mining has different challenges making it different from traditional data mining. Recently, many studies have addressed the concerns on massive data mining problems and proposed several techniques that produce impressive results. In this paper, we review real time clustering and classification mining techniques for data stream. We analyze the characteristics of data stream mining and discuss the challenges and research issues of data steam mining. Finally, we present some of the platforms for data stream mining.

Data stream mining techniques: a review

TELKOMNIKA JOURNAL

The tremendous growth of digital data has led to a high necessity for compressing applications either to minimize memory usage or transmission speed. Despite of the fact that many techniques already exist, there is still space and need for new techniques in this area of study. With this paper we aim to introduce a new technique for data compression through pixel combinations, used for both lossless and lossy compression. This new technique is also able to be used as a standalone solution, or with some other data compression method as an add-on providing better results. It is here applied only on images but it can be easily modified to work on any other type of data. We are going to present a side-by-side comparison, in terms of compression rate, of our technique with other widely used image compression methods. We will show that the compression ratio achieved by this technique tanks among the best in the literature whilst the actual algorithm remains simple and easily extensible. Finally the case will be made for the ability of our method to intrinsically support and enhance methods used for cryptography, steganography and watermarking.

Image Compression Through Combination Advantages From Existing Techniques

CSCJournals

Object‐oriented software development is an evolutionary process, and hence the opportunities for integration are abundant. Conceptually, classes are encapsulation of data attributes and their associated functions. Software components are amalgamation of logically and/or physically related classes. A complete software system is also an aggregation of software components. All of these various integration levels warrant contemporary integration techniques. Traditional integration techniques towards the end of software development process do not suffice any more. Integration strategies are needed at class level, component level, sub‐system level, and system levels. Classes require integration of methods. Various types of class interaction mechanisms demand different testing strategies. Integration of classes into components presses its own integration requirements. Finally, the system integration demands different types of integration testing strategies.

Svv

Jyothsna Sridhar

La actualidad más candente (15)

CONTENT BASED DATA TRANSFER MECHANISM FOR EFFICIENT BULK DATA TRANSFER IN GRI...

Java Abs Peer To Peer Design & Implementation Of A Tuple Space

Evaluation of a topological distance

The Impact of Data Replication on Job Scheduling Performance in Hierarchical ...

Hpc 3

Survey on Synchronizing File Operations Along with Storage Scalable Mechanism

Shortest path estimation for graph

Effective Data Retrieval System with Bloom in a Unstructured p2p Network

Caching in Information Centric Network (ICN)

Parallel and Distributed Information Retrieval System

Heterogeneous Networks of Remote Monitoring with High Availability and Resili...

Classification of Computer Networks

Data stream mining techniques: a review

Image Compression Through Combination Advantages From Existing Techniques

Svv

Similar a Trinity: A Distributed Graph Engine on a Memory Cloud

Proposed-curricula-MCSEwithSyllabus_24_...

butest

Distributed Computing

Sudarsun Santhiappan

Lecture 05 - Chapter 3 - Models of parallel computers and interconnections

National College of Business Administration & Economics ( NCBA&E)

Par com

tttoracle

In the era of IoTs and A.I., distributed and parallel computing is embracing big data driven and algorithm focused applications and services. With rapid progress and development on parallel frameworks, algorithms and accelerated computing capacities, it still remains challenging on deliver an efficient and scalable data analysis solution. This talk shares a research experience on data pattern discovery in domain applications. In particular, the research scrutinizes key factors in analysis workflow design and data parallelism improvement on cloud.

A TALE of DATA PATTERN DISCOVERY IN PARALLEL

Jenny Liu

01 introduction fundamentals_of_parallelism_and_code_optimization-www.astek.ir

aminnezarat

A Tale of Data Pattern Discovery in Parallel

Jenny Liu

Future prediction-ds

Muhammad Umar Farooq

Distributed systems

Francesc Lordan Gomis

Optimal Chain Matrix Multiplication Big Data Perspective

পল্লব রায়

5th sem

nastysuman009

5th sem

nastysuman009

Infrastructure services (Infrastructure-as-a-service), provided by cloud vendors, allow any user to provision a large number of compute instances fairly easily. Whether leased from public clouds or allocated from private clouds, utilizing these virtual resources to perform data/compute intensive analyses requires employing different parallel runtimes to implement such applications. Among many parallelizable problems, most “pleasingly parallel” applications can be performed using MapReduce technologies such as Hadoop, CGL-MapReduce, and Dryad, in a fairly easy manner. However, many scientific applications, which have complex communication patterns, still require low latency communication mechanisms and rich set of communication constructs offered by runtimes such as MPI. In this paper, we first discuss large scale data analysis using different MapReduce implementations and then, we present a performance analysis of high performance parallel applications on virtualized resources.

High Performance Parallel Computing with Clouds and Cloud Technologies

jaliyae

Parallel architecture

Mr SMAK

Lecture 01 - Chapter 1 (Part 01): This Lecture show the Overview of Course, What is an Operating System, Operating System Functions, Definition of a Distributed System, Properties of Distributed Systems, Software Concepts, Transparency in a Distributed System, Challenges, Approaches, Scalability Problems, Scalability Examples, Web Search, Financial Transactions, Multiplayer Games. Some basic concept of Operating System (OS).

Lecture 01 - Chapter 1 (Part 01): Some basic concept of Operating System (OS)...

National College of Business Administration & Economics ( NCBA&E)

Modern software design in Big data era

Bill GU

Key-Key-Value Stores for Efficiently Processing Graph Data in the Cloud

University of New South Wales

Disambiguating Advanced Computing for Humanities Researchers

Baden Hughes

Thilaganga mphil cs viva presentation ppt

thilaganga

Introduction to parallel iterative deep learning on hadoop’s next generation...

Anh Le

Similar a Trinity: A Distributed Graph Engine on a Memory Cloud (20)

Proposed-curricula-MCSEwithSyllabus_24_...

Distributed Computing

Lecture 05 - Chapter 3 - Models of parallel computers and interconnections

Par com

A TALE of DATA PATTERN DISCOVERY IN PARALLEL

01 introduction fundamentals_of_parallelism_and_code_optimization-www.astek.ir

A Tale of Data Pattern Discovery in Parallel

Future prediction-ds

Distributed systems

Optimal Chain Matrix Multiplication Big Data Perspective

5th sem

High Performance Parallel Computing with Clouds and Cloud Technologies

Parallel architecture

Lecture 01 - Chapter 1 (Part 01): Some basic concept of Operating System (OS)...

Modern software design in Big data era

Key-Key-Value Stores for Efficiently Processing Graph Data in the Cloud

Disambiguating Advanced Computing for Humanities Researchers

Thilaganga mphil cs viva presentation ppt

Introduction to parallel iterative deep learning on hadoop’s next generation...

Más de Qian Lin

Download paper from PVLDB: http://www.vldb.org/pvldb/vol12/p975-ruan.pdf The success of Bitcoin and other cryptocurrencies bring enormous interest to blockchains. A blockchain system implements a tamper-evident ledger for recording transactions that modify some global states. The system captures entire evolution history of the states. The management of that history, also known as data provenance or lineage, has been studied extensively in database systems. However, querying data history in existing blockchains can only be done by replaying all transactions. This approach is applicable to large-scale, offline analysis, but is not suitable for online transaction processing. We present LineageChain, a fine-grained, secure and efficient provenance system for blockchains. LineageChain exposes provenance information to smart contracts via simple and elegant interfaces, thereby enabling a new class of blockchain applications whose execution logics depend on provenance information at runtime. LineageChain captures provenance during contract execution, and efficiently stores it in a Merkle tree. LineageChain provides a novel skip list index designed for supporting efficient provenance query processing. We have implemented LineageChain on top of Hyperledger and a blockchain-optimized storage system called ForkBase. Our extensive evaluation of LineageChain demonstrates its benefits to the new class of blockchain applications, its efficient query, and its small storage overhead.

Fine-Grained, Secure and Efficient Data Provenance on Blockchain Systems

Qian Lin

PaxosStore is a high-availability storage system developed to support the comprehensive business of WeChat. It employs a combinational design in the storage layer to engage multiple storage engines constructed for different storage models. PaxosStore is characteristic of extracting the Paxos-based distributed consensus protocol as a middleware that is universally accessible to the underlying multi-model storage engines. This facilitates tuning, maintaining, scaling and extending the storage engines. According to our experience in engineering practice, to achieve a practical consistent read/write protocol is far more complex than its theory. To tackle such engineering complexity, we propose a layered design of the Paxos-based storage protocol stack, where PaxosLog, the key data structure used in the protocol, is devised to bridge the programming-oriented consistent read/write to the storage-oriented Paxos procedure. Additionally, we present optimizations based on Paxos that made fault-tolerance more efficient. PaxosStore is open source: https://github.com/tencent/paxosstore For details of the design, please refer to our VLDB 2017 paper: http://www.vldb.org/pvldb/vol10/p1730-lin.pdf Video of the presentation is also available: https://youtu.be/5zNRfuaCgBI

PaxosStore: High-availability Storage Made Practical in WeChat

Qian Lin

Presto: Distributed Machine Learning and Graph Processing with Sparse Matrices

Qian Lin

Adaptive Execution Support for Malleable Computation

Qian Lin

C-Cube: Elastic Continuous Clustering in the Cloud

Qian Lin

Kineograph: Taking the Pulse of a Fast-Changing and Connected World

Qian Lin

Optimizing Virtual Machines Using Hybrid Virtualization

Qian Lin

Virtual Machine Performance

Qian Lin

Be an Explorer, Be a Coder, Be a Writer

Qian Lin

SciMATE: A Novel MapReduce-Like Framework for Multiple Scientific Data Formats

Qian Lin

A Survey of Advanced Non-relational Database Systems: Approaches and Applicat...

Qian Lin

In-situ MapReduce for Log Processing

Qian Lin

C-MR: Continuously Executing MapReduce Workflows on Multi-Core Processors

Qian Lin

Más de Qian Lin (13)

Fine-Grained, Secure and Efficient Data Provenance on Blockchain Systems

PaxosStore: High-availability Storage Made Practical in WeChat

Presto: Distributed Machine Learning and Graph Processing with Sparse Matrices

Adaptive Execution Support for Malleable Computation

C-Cube: Elastic Continuous Clustering in the Cloud

Kineograph: Taking the Pulse of a Fast-Changing and Connected World

Optimizing Virtual Machines Using Hybrid Virtualization

Virtual Machine Performance

Be an Explorer, Be a Coder, Be a Writer

SciMATE: A Novel MapReduce-Like Framework for Multiple Scientific Data Formats

A Survey of Advanced Non-relational Database Systems: Approaches and Applicat...

In-situ MapReduce for Log Processing

C-MR: Continuously Executing MapReduce Workflows on Multi-Core Processors

Último

Micro-Scholarship, What it is, How can it help me.pdf

Poh-Sun Goh

How to Give a Domain for a Field in Odoo 17

Celine George

General Principles of Intellectual Property: Concepts of Intellectual Proper...

Poonam Aher Patil

INDIA QUIZ 2024 RLAC DELHI UNIVERSITY.pptx

RAM LAL ANAND COLLEGE, DELHI UNIVERSITY.

Application orientated numerical on hev.ppt

RamjanShidvankar

This PowerPoint helps students to consider the concept of infinity.

christianmathematics

Activity 01 - Artificial Culture (1).pdf

ciinovamais

A Transgenic animal is one that carries a foreign gene that has been deliberately inserted into its genome. The foreign gene are inserted into the germ line of the animal, so it can be transmitted to the progeny. Transgenic animals are animals that are genetically altered to have traits that mimic symptoms of specific human pathologies. They provide genetic model of various human disease which are important in understanding disease and development of new target.

Role Of Transgenic Animal In Target Validation-1.pptx

NikitaBankoti2

microwave assisted reaction. General introduction

Maksud Ahmed

Basic Civil Engineering notes first year Notes Building notes Selection of site for Building Layout of a Building What is Burjis, Mutam Building Bye laws Basic Concept of sunlight ventilation in building National Building Code of India Set back or building line Types of Buildings Floor Space Index (F.S.I) Institutional Vs Educational Building Components & function Sills, Lintels, Cantilever Doors, Windows and Ventilators Types of Foundation AND THEIR USES Plinth Area Shallow and Deep Foundation Super Built-up & carpet area Floor Area Ratio (F.A.R) RCC Reinforced Cement Concrete RCC VS PCC

Basic Civil Engineering first year Notes- Chapter 4 Building.pptx

Denish Jangid

Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...

EduSkills OECD

The basics of sentences session 3pptx.pptx

heathfieldcps1

Web & Social Media Analytics Previous Year Question Paper.pdf

Jayanti Pande

Python Notes for mca i year students osmania university.docx

Ramakrishna Reddy Bijjam

Unit-V; Pricing (Pharma Marketing Management).pptx

VishalSingh1417

Introduction to Nonprofit Accounting: The Basics

TechSoup

Holdier Curriculum Vitae (April 2024).pdf

agholdier

ICT Role in 21st Century Education & its Challenges.pptx

AreebaZafar22

In BC’s nearly-decade-old “new” curriculum, the curricular competencies describe the processes that students are expected to develop in areas of learning such as mathematics. They reflect the “Do” in the “Know-Do-Understand” model. Under the “Communicating” header falls the curricular competency “Explain and justify mathematical ideas and decisions.” Note that it contains two processes: “Explain mathematical ideas” and “Justify mathematical decisions.” I have broken it down into its separate parts in order to understand--or reveal--its meaning. The first part is commonplace in classrooms. By now, BC math teachers—and students—understand that “Explain mathematical ideas” means more than “Show your work.” Teachers consistently ask “What did you do?” and “How do you know?” This process is about retelling, not just of steps but of thinking. The second part happens less frequently. Think back to the last time that you observed a student make—a necessary precursor to justify—a mathematical decision. “Justify” is about defending. Like “explain,” it involves reasoning; unlike “explain,” it also involves opinion and debate. In order to reinterpret the curricular competency “Explain and justify mathematical ideas and decisions,” I will continue to take apart its constituent part “Justify mathematical decisions” and carefully examine the term “mathematical decisions.” What, exactly, is a “mathematical decision”? Below, I will categorize answers to this question. These categories, and the provided examples, may help to suggest new opportunities for students to justify.

Making and Justifying Mathematical Decisions.pdf

Chris Hunter

1029-Danh muc Sach Giao Khoa khoi 6.pdf

QucHHunhnh

Trinity: A Distributed Graph Engine on a Memory Cloud

1. Trinity: A Distributed Graph Engine on a Memory Cloud Speaker: LIN Qian http://www.comp.nus.edu.sg/~linqian/

2. Graph applications Online query processing  Low latency Offline graph analytics  High throughput

3. Online queries Random data access e.g., BFS, sub-graph matching, …

4. Offline computations Performed iteratively

5. Insight: Keeping the graph in memory at least the topology

6. Trinity Online query + Offline analytics

7. Random data access problem in large graph computation Globally addressable distr. memory Random access abstraction

8. Belief High-speed network is more available DRAM is cheaper In-memory solution become practical

9. “Trinity itself is not a system that comes with comprehensive built-in graph computation modules.”

10. Trinity cluster

11. Stack of Trinity system modules User define: Graph schema, Communication protocols, Computation paradigms

12. Memory cloud Partition memory space into trunks Hashing

13. Memory trunks 2p > m 1. Trunk level parallelism 2. Efficient hashing

14. Hashing Key-value store p-bit value  i ∈ [0, 2p – 1] Inner trunk hash table

15. Data partitioning and addressing Benefit: Scalability Fault-tolerance

16. Modeling graph Cell: value + schema Represent a node in a cell

17. TSL Object-oriented cell manipulation Data integration Network communication

18. Online queries Traversal based New paradigm

19. Vertex centric offline analytics Restrictive vertex centric model

20. Message passing optimization Create a bipartite partition of the local graph Buffer hub vertices

21. A new paradigm for offline analytics 1. Aggregate answers from local computations 2. Employ probabilistic inference

22. Circular memory management • Aim to avoid memory gaps between large number of key-value pairs

23. Fault tolerance Heartbeat-based failure detection BSP: checkpointing Async.: “periodical interruption”

24. Performance

25. Performance (cont.)

Trinity: A Distributed Graph Engine on a Memory Cloud

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (15)

Similar a Trinity: A Distributed Graph Engine on a Memory Cloud

Similar a Trinity: A Distributed Graph Engine on a Memory Cloud (20)

Más de Qian Lin

Más de Qian Lin (13)

Último

Último (20)

Trinity: A Distributed Graph Engine on a Memory Cloud