Kingso profile

•

0 recomendaciones•468 vistas

智杰付

Latex Beamer Hadoop

Agenda

What is Proﬁle?

henshao Kingso Proﬁle

Agenda

What is Proﬁle?
How to build Proﬁle?

henshao Kingso Proﬁle

Agenda

What is Proﬁle?
How to build Proﬁle?
Let’s talk Kbuild

henshao Kingso Proﬁle

group
segment
encode

henshao Kingso Proﬁle

group

group proﬁle
group group segment segment

proﬁle detail 3500w
proﬁle 12GB

$ ls | grep group
proﬁle group 0.seg 0
proﬁle group 1.seg 0
proﬁle group 2.seg 0

henshao Kingso Proﬁle

segment

group segment (1<<20) doc
segment

$ ls | grep seg
proﬁle group 0.seg 0
proﬁle group 0.seg 1
proﬁle group 0.seg 2

henshao Kingso Proﬁle

encode

provcity

mlr feature prop vid
group 6GB

$ ls | grep encode
cat id path.encode idx
cat id path.encode cnt

henshao Kingso Proﬁle

How to build Proﬁle?

: xml v3

segment
segment oﬀset

henshao Kingso Proﬁle

bitrecord
32 bits proﬁle
32

group varint
4 int32 t 1 byte
varint

henshao Kingso Proﬁle

Kingso Proﬁle

doc doc

isearch 1500w proﬁle 9.5GB kingso
3500w 12GB cat id path
2GB

DocAccessor proﬁle

henshao Kingso Proﬁle

Let’s talk Kbuild

!

henshao Kingso Proﬁle

use Hadoop
builder
Hadoop
xml builder

merge merge
index merge proﬁle merge detail merge

Hadoop streaming
Hadoop streaming Hadoop Hadoop

mapred.cache.ﬁles

henshao Kingso Proﬁle

task 1.6GB*18 7

index
15GB 4

proﬁle
19GB 9

28GB 9

task 21GB 6

henshao Kingso Proﬁle

Hadoop
Happy Hadoop
cpu

henshao Kingso Proﬁle

mapred.map.tasks.speculative.execution=false
data node

tar
index tar

rm
hadoop fs -cat index.tar | tar xf - -C output
tar -c index | hadoop fs -put - index.tar

get/put index proﬁle

detail
detail index proﬁle
job
henshao Kingso Proﬁle

Job 720 map slot

Job

Job 20 task
node

henshao Kingso Proﬁle

The end

Thank you!

henshao Kingso Proﬁle

Recomendados

JBoss Fuse Workshop 101 part 4Christina Lin

Bulk Exporting from Cassandra - Carlo CabanillaDatadog

分布式索引构建智杰付

Nyc hadoop meetup introduction to h base智杰付

Taobao海量图片存储与cdn系统智杰付

单元测试必知必会智杰付

Lightning Talk #9: How UX and Data Storytelling Can Shape Policy by Mika Aldabaux singapore

Succession “Losers”: What Happens to Executives Passed Over for the CEO Job? Stanford GSB Corporate Governance Research Initiative

2024 State of Marketing Report – by HubspotMarius Sescu

Everything You Need To Know About ChatGPTExpeed Software

Product Design Trends in 2024 | Teenage EngineeringsPixeldarts

How Race, Age and Gender Shape Attitudes Towards Mental HealthThinkNow

AI Trends in Creative Operations 2024 by Artwork Flow.pdfmarketingartwork

Skeleton Culture CodeSkeleton Technologies

PEPSICO Presentation to CAGNY Conference Feb 2024Neil Kimberley

Content Methodology: A Best Practices Report (Webinar)contently

How to Prepare For a Successful Job Search for 2024Albert Qian

Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media Age(ncy)

Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal

5 Public speaking tips from TED - Visualized summarySpeakerHub

ChatGPT and the Future of Work - Clark Boyd Clark Boyd

Getting into the tech field. what next Tessa Mero

Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray

How to have difficult conversations Rajiv Jayarajah, MAppComm, ACC

Introduction to Data ScienceChristy Abraham Joy

Time Management & Productivity - Best PracticesVit Horky

The six step guide to practical project managementMindGenius

Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36

Más contenido relacionado

Destacado

2024 State of Marketing Report – by HubspotMarius Sescu

Everything You Need To Know About ChatGPTExpeed Software

Product Design Trends in 2024 | Teenage EngineeringsPixeldarts

How Race, Age and Gender Shape Attitudes Towards Mental HealthThinkNow

AI Trends in Creative Operations 2024 by Artwork Flow.pdfmarketingartwork

Skeleton Culture CodeSkeleton Technologies

PEPSICO Presentation to CAGNY Conference Feb 2024Neil Kimberley

Content Methodology: A Best Practices Report (Webinar)contently

How to Prepare For a Successful Job Search for 2024Albert Qian

Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media Age(ncy)

Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal

5 Public speaking tips from TED - Visualized summarySpeakerHub

ChatGPT and the Future of Work - Clark Boyd Clark Boyd

Getting into the tech field. what next Tessa Mero

Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray

How to have difficult conversations Rajiv Jayarajah, MAppComm, ACC

Introduction to Data ScienceChristy Abraham Joy

Time Management & Productivity - Best PracticesVit Horky

The six step guide to practical project managementMindGenius

Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36

Destacado (20)

2024 State of Marketing Report – by Hubspot

Everything You Need To Know About ChatGPT

Product Design Trends in 2024 | Teenage Engineerings

How Race, Age and Gender Shape Attitudes Towards Mental Health

AI Trends in Creative Operations 2024 by Artwork Flow.pdf

Skeleton Culture Code

PEPSICO Presentation to CAGNY Conference Feb 2024

Content Methodology: A Best Practices Report (Webinar)

How to Prepare For a Successful Job Search for 2024

Social Media Marketing Trends 2024 // The Global Indie Insights

Trends In Paid Search: Navigating The Digital Landscape In 2024

5 Public speaking tips from TED - Visualized summary

ChatGPT and the Future of Work - Clark Boyd

Getting into the tech field. what next

Google's Just Not That Into You: Understanding Core Updates & Search Intent

How to have difficult conversations

Introduction to Data Science

Time Management & Productivity - Best Practices

The six step guide to practical project management

Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...

Kingso profile

1. Kingso Proﬁle henshao October 28, 2011 henshao Kingso Proﬁle

2. Agenda henshao Kingso Proﬁle

3. Agenda What is Proﬁle? henshao Kingso Proﬁle

4. Agenda What is Profile? How to build Profile? henshao Kingso Profile

5. Agenda What is Profile? How to build Profile? Let’s talk Kbuild henshao Kingso Profile

6. group henshao Kingso Proﬁle

7. group segment henshao Kingso Proﬁle

8. group segment encode henshao Kingso Proﬁle

9. group group profile group group segment segment profile detail 3500w profile 12GB $ ls | grep group profile group 0.seg 0 profile group 1.seg 0 profile group 2.seg 0 henshao Kingso Profile

10. segment group segment (1<<20) doc segment $ ls | grep seg profile group 0.seg 0 profile group 0.seg 1 profile group 0.seg 2 henshao Kingso Profile

11. encode provcity mlr feature prop vid group 6GB $ ls | grep encode cat id path.encode idx cat id path.encode cnt henshao Kingso Proﬁle

12. henshao Kingso Proﬁle

13. How to build Profile? : xml v3 segment segment offset henshao Kingso Profile

14. Proﬁle

15. bitrecord 32 bits proﬁle 32 group varint 4 int32 t 1 byte varint henshao Kingso Proﬁle

16. Kingso Profile doc doc isearch 1500w profile 9.5GB kingso 3500w 12GB cat id path 2GB DocAccessor profile henshao Kingso Profile

17. build XML 80 henshao Kingso Proﬁle

18. Let’s talk Kbuild ! henshao Kingso Proﬁle

19. use Hadoop builder Hadoop xml builder merge merge index merge profile merge detail merge Hadoop streaming Hadoop streaming Hadoop Hadoop mapred.cache.files henshao Kingso Profile

20. task 1.6GB*18 7 index 15GB 4 proﬁle 19GB 9 28GB 9 task 21GB 6 henshao Kingso Proﬁle

21. Hadoop Happy Hadoop cpu henshao Kingso Proﬁle

22. mapred.map.tasks.speculative.execution=false data node tar index tar rm hadoop fs -cat index.tar | tar xf - -C output tar -c index | hadoop fs -put - index.tar get/put index profile detail detail index profile job henshao Kingso Profile

23. Job 720 map slot Job Job 20 task node henshao Kingso Proﬁle

24. The end Thank you! henshao Kingso Proﬁle