SlideShare a Scribd company logo
1 of 24
GFS: The Google File System Brad Karp UCL Computer Science CS Z03 / 4030 30 th  October, 2006
Motivating Application: Google ,[object Object],[object Object],[object Object],[object Object],[object Object]
Cluster of PCs as Supercomputer ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],GFS: File system for sharing data on clusters, designed with Google’s application workload specifically in mind
Google Platform Characteristics ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Google File System: Design Criteria ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GFS: Architecture ,[object Object],[object Object],[object Object],[object Object],[object Object]
GFS: Architecture (2)
Master Server ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Holds all metadata in RAM; very fast operations on file system metadata
Chunkserver ,[object Object],[object Object],[object Object],[object Object]
Client ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Client API ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Client Read ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Client Write ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Client Write (2) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Client Write (3)
GFS: Consistency Model ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GFS: Data Mutation Consistency ,[object Object],[object Object],inconsistent failure consistent  but  undefined concurrent success defined  interspersed with  inconsistent defined serial success Record Append Write
Applications and Record Append Semantics ,[object Object],[object Object],[object Object],[object Object]
Logging at Master ,[object Object],[object Object],[object Object],[object Object],[object Object]
Chunk Leases and Version Numbers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
What If the Master Reboots? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
What if Chunkserver Fails? ,[object Object],[object Object],[object Object],[object Object]
File Deletion ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GFS: Summary ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

More Related Content

What's hot

advanced Google file System
advanced Google file Systemadvanced Google file System
advanced Google file Systemdiptipan
 
Seminar Report on Google File System
Seminar Report on Google File SystemSeminar Report on Google File System
Seminar Report on Google File SystemVishal Polley
 
Google File System
Google File SystemGoogle File System
Google File Systemguest2cb4689
 
Google file system
Google file systemGoogle file system
Google file systemDhan V Sagar
 
Google File Systems
Google File SystemsGoogle File Systems
Google File SystemsAzeem Mumtaz
 
Cluster based storage - Nasd and Google file system - advanced operating syst...
Cluster based storage - Nasd and Google file system - advanced operating syst...Cluster based storage - Nasd and Google file system - advanced operating syst...
Cluster based storage - Nasd and Google file system - advanced operating syst...Antonio Cesarano
 
Cloud infrastructure. Google File System and MapReduce - Andrii Vozniuk
Cloud infrastructure. Google File System and MapReduce - Andrii VozniukCloud infrastructure. Google File System and MapReduce - Andrii Vozniuk
Cloud infrastructure. Google File System and MapReduce - Andrii VozniukAndrii Vozniuk
 
Google File System
Google File SystemGoogle File System
Google File SystemDreamJobs1
 
Distributed file systems (from Google)
Distributed file systems (from Google)Distributed file systems (from Google)
Distributed file systems (from Google)Sri Prasanna
 
Google file system GFS
Google file system GFSGoogle file system GFS
Google file system GFSzihad164
 
SeaweedFS introduction
SeaweedFS introductionSeaweedFS introduction
SeaweedFS introductionchrislusf
 
Near Real time Indexing Kafka Messages to Apache Blur using Spark Streaming
Near Real time Indexing Kafka Messages to Apache Blur using Spark StreamingNear Real time Indexing Kafka Messages to Apache Blur using Spark Streaming
Near Real time Indexing Kafka Messages to Apache Blur using Spark StreamingDibyendu Bhattacharya
 

What's hot (20)

advanced Google file System
advanced Google file Systemadvanced Google file System
advanced Google file System
 
Google File System
Google File SystemGoogle File System
Google File System
 
Seminar Report on Google File System
Seminar Report on Google File SystemSeminar Report on Google File System
Seminar Report on Google File System
 
Google File System
Google File SystemGoogle File System
Google File System
 
gfs-sosp2003
gfs-sosp2003gfs-sosp2003
gfs-sosp2003
 
Google file system
Google file systemGoogle file system
Google file system
 
Google file system
Google file systemGoogle file system
Google file system
 
Google File Systems
Google File SystemsGoogle File Systems
Google File Systems
 
Cluster based storage - Nasd and Google file system - advanced operating syst...
Cluster based storage - Nasd and Google file system - advanced operating syst...Cluster based storage - Nasd and Google file system - advanced operating syst...
Cluster based storage - Nasd and Google file system - advanced operating syst...
 
Cloud infrastructure. Google File System and MapReduce - Andrii Vozniuk
Cloud infrastructure. Google File System and MapReduce - Andrii VozniukCloud infrastructure. Google File System and MapReduce - Andrii Vozniuk
Cloud infrastructure. Google File System and MapReduce - Andrii Vozniuk
 
Google File System
Google File SystemGoogle File System
Google File System
 
Google file system
Google file systemGoogle file system
Google file system
 
Google
GoogleGoogle
Google
 
Google File System
Google File SystemGoogle File System
Google File System
 
Distributed file systems (from Google)
Distributed file systems (from Google)Distributed file systems (from Google)
Distributed file systems (from Google)
 
Google file system GFS
Google file system GFSGoogle file system GFS
Google file system GFS
 
SeaweedFS introduction
SeaweedFS introductionSeaweedFS introduction
SeaweedFS introduction
 
GFS & HDFS Introduction
GFS & HDFS IntroductionGFS & HDFS Introduction
GFS & HDFS Introduction
 
Near Real time Indexing Kafka Messages to Apache Blur using Spark Streaming
Near Real time Indexing Kafka Messages to Apache Blur using Spark StreamingNear Real time Indexing Kafka Messages to Apache Blur using Spark Streaming
Near Real time Indexing Kafka Messages to Apache Blur using Spark Streaming
 
Anatomy of file write in hadoop
Anatomy of file write in hadoopAnatomy of file write in hadoop
Anatomy of file write in hadoop
 

Similar to Gfs介绍

GFS - Google File System
GFS - Google File SystemGFS - Google File System
GFS - Google File Systemtutchiio
 
Distributed computing seminar lecture 3 - distributed file systems
Distributed computing seminar   lecture 3 - distributed file systemsDistributed computing seminar   lecture 3 - distributed file systems
Distributed computing seminar lecture 3 - distributed file systemstugrulh
 
Distributed file systems
Distributed file systemsDistributed file systems
Distributed file systemsSri Prasanna
 
Teoria efectului defectului hardware: GoogleFS
Teoria efectului defectului hardware: GoogleFSTeoria efectului defectului hardware: GoogleFS
Teoria efectului defectului hardware: GoogleFSAsociatia ProLinux
 
Presentation on nfs,afs,vfs
Presentation on nfs,afs,vfsPresentation on nfs,afs,vfs
Presentation on nfs,afs,vfsPrakriti Dubey
 
Hadoop Distributed File System for Big Data Analytics
Hadoop Distributed File System for Big Data AnalyticsHadoop Distributed File System for Big Data Analytics
Hadoop Distributed File System for Big Data AnalyticsDrPDShebaKeziaMalarc
 
CS9222 ADVANCED OPERATING SYSTEMS
CS9222 ADVANCED OPERATING SYSTEMSCS9222 ADVANCED OPERATING SYSTEMS
CS9222 ADVANCED OPERATING SYSTEMSKathirvel Ayyaswamy
 
storage-systems.pptx
storage-systems.pptxstorage-systems.pptx
storage-systems.pptxShimoFcis
 
Dfs (Distributed computing)
Dfs (Distributed computing)Dfs (Distributed computing)
Dfs (Distributed computing)Sri Prasanna
 
Distributed File Systems
Distributed File SystemsDistributed File Systems
Distributed File Systemsawesomesos
 
Ch16 OS
Ch16 OSCh16 OS
Ch16 OSC.U
 

Similar to Gfs介绍 (20)

tittle
tittletittle
tittle
 
GFS - Google File System
GFS - Google File SystemGFS - Google File System
GFS - Google File System
 
Lec3 Dfs
Lec3 DfsLec3 Dfs
Lec3 Dfs
 
Distributed computing seminar lecture 3 - distributed file systems
Distributed computing seminar   lecture 3 - distributed file systemsDistributed computing seminar   lecture 3 - distributed file systems
Distributed computing seminar lecture 3 - distributed file systems
 
Distributed file systems
Distributed file systemsDistributed file systems
Distributed file systems
 
Cloud storage
Cloud storageCloud storage
Cloud storage
 
Teoria efectului defectului hardware: GoogleFS
Teoria efectului defectului hardware: GoogleFSTeoria efectului defectului hardware: GoogleFS
Teoria efectului defectului hardware: GoogleFS
 
Hadoop
HadoopHadoop
Hadoop
 
Lalit
LalitLalit
Lalit
 
Gfs
GfsGfs
Gfs
 
Gfs final
Gfs finalGfs final
Gfs final
 
Presentation on nfs,afs,vfs
Presentation on nfs,afs,vfsPresentation on nfs,afs,vfs
Presentation on nfs,afs,vfs
 
Hadoop Distributed File System for Big Data Analytics
Hadoop Distributed File System for Big Data AnalyticsHadoop Distributed File System for Big Data Analytics
Hadoop Distributed File System for Big Data Analytics
 
CS9222 ADVANCED OPERATING SYSTEMS
CS9222 ADVANCED OPERATING SYSTEMSCS9222 ADVANCED OPERATING SYSTEMS
CS9222 ADVANCED OPERATING SYSTEMS
 
storage-systems.pptx
storage-systems.pptxstorage-systems.pptx
storage-systems.pptx
 
HDFS Design Principles
HDFS Design PrinciplesHDFS Design Principles
HDFS Design Principles
 
Dfs (Distributed computing)
Dfs (Distributed computing)Dfs (Distributed computing)
Dfs (Distributed computing)
 
Distributed File Systems
Distributed File SystemsDistributed File Systems
Distributed File Systems
 
Ch16 OS
Ch16 OSCh16 OS
Ch16 OS
 
OS_Ch16
OS_Ch16OS_Ch16
OS_Ch16
 

More from yiditushe

Spring入门纲要
Spring入门纲要Spring入门纲要
Spring入门纲要yiditushe
 
J Bpm4 1中文用户手册
J Bpm4 1中文用户手册J Bpm4 1中文用户手册
J Bpm4 1中文用户手册yiditushe
 
性能测试实践2
性能测试实践2性能测试实践2
性能测试实践2yiditushe
 
性能测试实践1
性能测试实践1性能测试实践1
性能测试实践1yiditushe
 
性能测试技术
性能测试技术性能测试技术
性能测试技术yiditushe
 
Load runner测试技术
Load runner测试技术Load runner测试技术
Load runner测试技术yiditushe
 
J2 ee性能测试
J2 ee性能测试J2 ee性能测试
J2 ee性能测试yiditushe
 
面向对象的Js培训
面向对象的Js培训面向对象的Js培训
面向对象的Js培训yiditushe
 
Flex3中文教程
Flex3中文教程Flex3中文教程
Flex3中文教程yiditushe
 
开放源代码的全文检索Lucene
开放源代码的全文检索Lucene开放源代码的全文检索Lucene
开放源代码的全文检索Luceneyiditushe
 
基于分词索引的全文检索技术介绍
基于分词索引的全文检索技术介绍基于分词索引的全文检索技术介绍
基于分词索引的全文检索技术介绍yiditushe
 
Lucene In Action
Lucene In ActionLucene In Action
Lucene In Actionyiditushe
 
Lucene2 4学习笔记1
Lucene2 4学习笔记1Lucene2 4学习笔记1
Lucene2 4学习笔记1yiditushe
 
Lucene2 4 Demo
Lucene2 4 DemoLucene2 4 Demo
Lucene2 4 Demoyiditushe
 
Lucene 全文检索实践
Lucene 全文检索实践Lucene 全文检索实践
Lucene 全文检索实践yiditushe
 
Lucene 3[1] 0 原理与代码分析
Lucene 3[1] 0 原理与代码分析Lucene 3[1] 0 原理与代码分析
Lucene 3[1] 0 原理与代码分析yiditushe
 
7 面向对象设计原则
7 面向对象设计原则7 面向对象设计原则
7 面向对象设计原则yiditushe
 
10 团队开发
10  团队开发10  团队开发
10 团队开发yiditushe
 
9 对象持久化与数据建模
9  对象持久化与数据建模9  对象持久化与数据建模
9 对象持久化与数据建模yiditushe
 
8 Uml构架建模
8  Uml构架建模8  Uml构架建模
8 Uml构架建模yiditushe
 

More from yiditushe (20)

Spring入门纲要
Spring入门纲要Spring入门纲要
Spring入门纲要
 
J Bpm4 1中文用户手册
J Bpm4 1中文用户手册J Bpm4 1中文用户手册
J Bpm4 1中文用户手册
 
性能测试实践2
性能测试实践2性能测试实践2
性能测试实践2
 
性能测试实践1
性能测试实践1性能测试实践1
性能测试实践1
 
性能测试技术
性能测试技术性能测试技术
性能测试技术
 
Load runner测试技术
Load runner测试技术Load runner测试技术
Load runner测试技术
 
J2 ee性能测试
J2 ee性能测试J2 ee性能测试
J2 ee性能测试
 
面向对象的Js培训
面向对象的Js培训面向对象的Js培训
面向对象的Js培训
 
Flex3中文教程
Flex3中文教程Flex3中文教程
Flex3中文教程
 
开放源代码的全文检索Lucene
开放源代码的全文检索Lucene开放源代码的全文检索Lucene
开放源代码的全文检索Lucene
 
基于分词索引的全文检索技术介绍
基于分词索引的全文检索技术介绍基于分词索引的全文检索技术介绍
基于分词索引的全文检索技术介绍
 
Lucene In Action
Lucene In ActionLucene In Action
Lucene In Action
 
Lucene2 4学习笔记1
Lucene2 4学习笔记1Lucene2 4学习笔记1
Lucene2 4学习笔记1
 
Lucene2 4 Demo
Lucene2 4 DemoLucene2 4 Demo
Lucene2 4 Demo
 
Lucene 全文检索实践
Lucene 全文检索实践Lucene 全文检索实践
Lucene 全文检索实践
 
Lucene 3[1] 0 原理与代码分析
Lucene 3[1] 0 原理与代码分析Lucene 3[1] 0 原理与代码分析
Lucene 3[1] 0 原理与代码分析
 
7 面向对象设计原则
7 面向对象设计原则7 面向对象设计原则
7 面向对象设计原则
 
10 团队开发
10  团队开发10  团队开发
10 团队开发
 
9 对象持久化与数据建模
9  对象持久化与数据建模9  对象持久化与数据建模
9 对象持久化与数据建模
 
8 Uml构架建模
8  Uml构架建模8  Uml构架建模
8 Uml构架建模
 

Gfs介绍