Buzztterの裏側とその周辺技術

•

18 likes•4,274 views

Yoji Shidara

Technology

w
w TFtgt DFtgt TFref DFref

w TFtgt
DFtgt

w TFref
DFref

>> t = Time.parse(quot;2007-11-3quot;)
=> Sat Nov 03 00:00:00 +0900 2007

>> Status.count(:conditions=>[quot;created_at
BETWEEN ? AND ?quot;, t, t.tomorrow])
=> 125626

Tue Nov 06 15:17:40 +0900 2007 - received 8 / 20, 5793 tuples
Tue Nov 06 15:17:45 +0900 2007 - received 10 / 20, 5794 tuples
Tue Nov 06 15:17:51 +0900 2007 - received 10 / 20, 5798 tuples
Tue Nov 06 15:17:55 +0900 2007 - received 4 / 20, 5797 tuples
Tue Nov 06 15:18:00 +0900 2007 - received 5 / 20, 5797 tuples
Tue Nov 06 15:18:05 +0900 2007 - received 11 / 20, 5797 tuples
Tue Nov 06 15:18:12 +0900 2007 - received 8 / 20, 5802 tuples
Tue Nov 06 15:18:16 +0900 2007 - received 9 / 20, 5807 tuples
Tue Nov 06 15:18:21 +0900 2007 - received 8 / 20, 5809 tuples
Tue Nov 06 15:18:25 +0900 2007 - received 12 / 20, 5810 tuples
Tue Nov 06 15:18:30 +0900 2007 - received 10 / 20, 5812 tuples
Tue Nov 06 15:18:35 +0900 2007 - received 13 / 20, 5817 tuples
Tue Nov 06 15:18:40 +0900 2007 - received 3 / 20, 5811 tuples
Tue Nov 06 15:18:45 +0900 2007 - received 5 / 20, 5811 tuples
Tue Nov 06 15:18:50 +0900 2007 - received 15 / 20, 5820 tuples
Tue Nov 06 15:18:55 +0900 2007 - received 14 / 20, 5826 tuples
Tue Nov 06 15:19:01 +0900 2007 - received 3 / 20, 5823 tuples
Tue Nov 06 15:19:08 +0900 2007 - received 8 / 20, 5814 tuples
Tue Nov 06 15:19:12 +0900 2007 - received 8 / 20, 5822 tuples
Tue Nov 06 15:19:18 +0900 2007 - received 10 / 20, 5818 tuples

i j

i, j
j
Ci,j = P (tk−1 |tk )P (tk+1 |tk )
k=i

Ci,j < 0.75
i..j

$count_by_sql [quot;SELECT COUNT(DISTINCT(user_id)) FROM statuses WHERE #{IGNORE_COND} AND language = ? AND (created_at BETWEEN ? AND ?) AND text @@ ?quot;, language, t.ago(ago), t, add_pragma(word)]$

2007-11-06 13:19:45 ANALYZER-ng(22499) begin for japanese-utf8
2007-11-06 13:19:46 ANALYZER-ng(22499) extracted 3120 sentences
2007-11-06 13:20:12 ANALYZER-ng(22499) 6006 keywords extracted from 3120 sentences
2007-11-06 13:20:12 ANALYZER-ng(22499) deleting stopwords ...
2007-11-06 13:20:19 ANALYZER-ng(22499) odd terms removed (5902 terms)
2007-11-06 13:20:19 ANALYZER-ng(22499) ignore case (5895 terms)
2007-11-06 13:20:19 ANALYZER-ng(22499) trivial terms are removed (1796 terms)
2007-11-06 13:21:38 ANALYZER-ng(22499) occurrence calculated (72.738133 s)
2007-11-06 13:23:35 ANALYZER-ng(22499) modified DDFs calculated
2007-11-06 13:23:35 ANALYZER-ng(22499) scores calculated (1563 terms)
2007-11-06 13:23:40 ANALYZER-ng(22499) redundant terms removed (1151 terms)
2007-11-06 13:23:42 ANALYZER-ng(22499) end for japanese-utf8 (237.531316 s)

2007-11-06 13:23:42 ANALYZER-ng(22499) begin for english
2007-11-06 13:23:43 ANALYZER-ng(22499) extracted 6181 sentences
2007-11-06 13:24:20 ANALYZER-ng(22499) 10168 keywords extracted from 6181 sentences
2007-11-06 13:24:20 ANALYZER-ng(22499) deleting stopwords ...
2007-11-06 13:24:33 ANALYZER-ng(22499) odd terms removed (9808 terms)
2007-11-06 13:24:33 ANALYZER-ng(22499) ignore case (9444 terms)
2007-11-06 13:24:33 ANALYZER-ng(22499) trivial terms are removed (2738 terms)
2007-11-06 13:26:18 ANALYZER-ng(22499) occurrence calculated (96.306258 s)
2007-11-06 13:27:59 ANALYZER-ng(22499) modified DDFs calculated
2007-11-06 13:27:59 ANALYZER-ng(22499) scores calculated (2109 terms)
2007-11-06 13:28:10 ANALYZER-ng(22499) redundant terms removed (1643 terms)
2007-11-06 13:28:13 ANALYZER-ng(22499) end for english (270.044345 s)

Recently uploaded

Real Time Object Detection Using Open CVKhem

[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia

04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG

A Year of the Servo Reboot: Where Are We Now?Igalia

Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun

08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls

How to convert PDF to text with Nanonetsnaman860154

🐬 The future of MySQL is Postgres 🐘RTylerCroy

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung

Scaling API-first – The story of a global engineering organizationRadu Cotescu

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge

The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad

Finology Group – Insurtech Innovation Award 2024The Digital Insurer

Boost PC performance: How more available memory can improve productivityPrincipled Technologies

Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer

CNv6 Instructor Chapter 6 Quality of Servicegiselly40

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo

2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong

Recently uploaded (20)

Real Time Object Detection Using Open CV

[2024]Digital Global Overview Report 2024 Meltwater.pdf

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

A Year of the Servo Reboot: Where Are We Now?

Powerful Google developer tools for immediate impact! (2023-24 C)

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

How to convert PDF to text with Nanonets

🐬 The future of MySQL is Postgres 🐘

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Scaling API-first – The story of a global engineering organization

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

The Codex of Business Writing Software for Real-World Solutions 2.pptx

Finology Group – Insurtech Innovation Award 2024

Boost PC performance: How more available memory can improve productivity

Tata AIG General Insurance Company - Insurer Innovation Award 2024

CNv6 Instructor Chapter 6 Quality of Service

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Buzztterの裏側とその周辺技術

8. w w TFtgt DFtgt TFref DFref w TFtgt DFtgt w TFref DFref

10.

11.

12. >> t = Time.parse(quot;2007-11-3quot;) => Sat Nov 03 00:00:00 +0900 2007 >> Status.count(:conditions=>[quot;created_at BETWEEN ? AND ?quot;, t, t.tomorrow]) => 125626

13.

14.

15.

16.

17. Tue Nov 06 15:17:40 +0900 2007 - received 8 / 20, 5793 tuples Tue Nov 06 15:17:45 +0900 2007 - received 10 / 20, 5794 tuples Tue Nov 06 15:17:51 +0900 2007 - received 10 / 20, 5798 tuples Tue Nov 06 15:17:55 +0900 2007 - received 4 / 20, 5797 tuples Tue Nov 06 15:18:00 +0900 2007 - received 5 / 20, 5797 tuples Tue Nov 06 15:18:05 +0900 2007 - received 11 / 20, 5797 tuples Tue Nov 06 15:18:12 +0900 2007 - received 8 / 20, 5802 tuples Tue Nov 06 15:18:16 +0900 2007 - received 9 / 20, 5807 tuples Tue Nov 06 15:18:21 +0900 2007 - received 8 / 20, 5809 tuples Tue Nov 06 15:18:25 +0900 2007 - received 12 / 20, 5810 tuples Tue Nov 06 15:18:30 +0900 2007 - received 10 / 20, 5812 tuples Tue Nov 06 15:18:35 +0900 2007 - received 13 / 20, 5817 tuples Tue Nov 06 15:18:40 +0900 2007 - received 3 / 20, 5811 tuples Tue Nov 06 15:18:45 +0900 2007 - received 5 / 20, 5811 tuples Tue Nov 06 15:18:50 +0900 2007 - received 15 / 20, 5820 tuples Tue Nov 06 15:18:55 +0900 2007 - received 14 / 20, 5826 tuples Tue Nov 06 15:19:01 +0900 2007 - received 3 / 20, 5823 tuples Tue Nov 06 15:19:08 +0900 2007 - received 8 / 20, 5814 tuples Tue Nov 06 15:19:12 +0900 2007 - received 8 / 20, 5822 tuples Tue Nov 06 15:19:18 +0900 2007 - received 10 / 20, 5818 tuples

18.

19.

20. w w TFtgt DFtgt TFref DFref w TFtgt DFtgt w TFref DFref

21. k

22.

23.

24. i j i, j j Ci,j = P (tk−1 |tk )P (tk+1 |tk ) k=i Ci,j < 0.75 i..j

25.

26.

27. count_by_sql [quot;SELECT COUNT(DISTINCT(user_id)) FROM statuses WHERE #{IGNORE_COND} AND language = ? AND (created_at BETWEEN ? AND ?) AND text @@ ?quot;, language, t.ago(ago), t, add_pragma(word)]

28. 2007-11-06 13:19:45 ANALYZER-ng(22499) begin for japanese-utf8 2007-11-06 13:19:46 ANALYZER-ng(22499) extracted 3120 sentences 2007-11-06 13:20:12 ANALYZER-ng(22499) 6006 keywords extracted from 3120 sentences 2007-11-06 13:20:12 ANALYZER-ng(22499) deleting stopwords ... 2007-11-06 13:20:19 ANALYZER-ng(22499) odd terms removed (5902 terms) 2007-11-06 13:20:19 ANALYZER-ng(22499) ignore case (5895 terms) 2007-11-06 13:20:19 ANALYZER-ng(22499) trivial terms are removed (1796 terms) 2007-11-06 13:21:38 ANALYZER-ng(22499) occurrence calculated (72.738133 s) 2007-11-06 13:23:35 ANALYZER-ng(22499) modified DDFs calculated 2007-11-06 13:23:35 ANALYZER-ng(22499) scores calculated (1563 terms) 2007-11-06 13:23:40 ANALYZER-ng(22499) redundant terms removed (1151 terms) 2007-11-06 13:23:42 ANALYZER-ng(22499) end for japanese-utf8 (237.531316 s) 2007-11-06 13:23:42 ANALYZER-ng(22499) begin for english 2007-11-06 13:23:43 ANALYZER-ng(22499) extracted 6181 sentences 2007-11-06 13:24:20 ANALYZER-ng(22499) 10168 keywords extracted from 6181 sentences 2007-11-06 13:24:20 ANALYZER-ng(22499) deleting stopwords ... 2007-11-06 13:24:33 ANALYZER-ng(22499) odd terms removed (9808 terms) 2007-11-06 13:24:33 ANALYZER-ng(22499) ignore case (9444 terms) 2007-11-06 13:24:33 ANALYZER-ng(22499) trivial terms are removed (2738 terms) 2007-11-06 13:26:18 ANALYZER-ng(22499) occurrence calculated (96.306258 s) 2007-11-06 13:27:59 ANALYZER-ng(22499) modified DDFs calculated 2007-11-06 13:27:59 ANALYZER-ng(22499) scores calculated (2109 terms) 2007-11-06 13:28:10 ANALYZER-ng(22499) redundant terms removed (1643 terms) 2007-11-06 13:28:13 ANALYZER-ng(22499) end for english (270.044345 s)

Buzztterの裏側とその周辺技術

Recommended

Recommended

More Related Content

More from Yoji Shidara

More from Yoji Shidara (12)

Recently uploaded

Recently uploaded (20)

Buzztterの裏側とその周辺技術