Rubyconf 2016 Building Efficient and Reliable Crawler System With Sidekiq Enterprise

1. Photo: http://cliparts.co/clipart/3666251

2. Has anyone ever written crawlers?

3. Has anyone ever used cron?

4. Has anyone ever used Sidekiq?

5. Gary (Chien-Wei Chu)
@icarus4 / @icarus4.chu
Was a C programmer
Fall in love with Ruby since 2013
CTO of Statementdog

7. I Play
Photo: https://static01.nyt.com/images/2016/08/19/sports/19BADMINTONweb3/19BADMINTONweb3-master675.jpg

9. Photo: http://classic.battle.net/images/battle/scc/protoss/pix/units/screenshots/d05.jpg

10. Photo: http://resources.workable.com/wp-content/uploads/2015/08/ruby-560x224.jpg

11. • Introduction to Statementdog

12. • Introduction to Statementdog
• Data behind Statementdog

13. • Introduction to Statementdog
• Data behind Statementdog
• Past practice of Statementdog

14. • Introduction to Statementdog
• Data behind Statementdog
• Past practice of Statementdog
• Problems of the past practice

15. • Introduction to Statementdog
• Data behind Statementdog
• Past practice of Statementdog
• Problems of the past practice
• How we design our system to solve the problems.

16. Focus on:
• More reliable job scheduling
• Dealing with throttling issue

26. (Revenue)

27. (Revenue)
(EPS)

28. (Revenue)
(EPS)
(Gross Margin)
(Net Income)

29. (Revenue)
(EPS)
(Gross Margin)
(Net Income)
(Assets)
(Liabilities)

30. (Revenue)
(EPS)
(Gross Margin)
(Net Income)
(Assets)
(Liabilities)
(Operating Cash Flow)
(Free Cash Flow)
(Investing Cash Flow)

31. (Revenue)
(EPS)
(Gross Margin)
(Net Income)
(Assets)
(Liabilities)
(Operating Cash Flow)
(Free Cash Flow)
(Investing Cash Flow)
(ROE)
(ROA)
(Accounts Receivable)
(Accounts Payable)

32. (Revenue)
(EPS)
(Gross Margin)
(Net Income)
(Assets)
(Liabilities)
(Operating Cash Flow)
(Free Cash Flow)
(Investing Cash Flow)
(ROE)
(ROA)
(Accounts Receivable)
(Accounts Payable)

33. (Revenue)
(EPS)
(Gross Margin)
(Net Income)
(Assets)
(Liabilities)
(Operating Cash Flow)
(Free Cash Flow)
(Investing Cash Flow)
(ROE)
(ROA)
(Accounts Receivable)
(Accounts Payable)
(PMI)

34. (Revenue)
(EPS)
(Gross Margin)
(Net Income)
(Assets)
(Liabilities)
(Operating Cash Flow)
(Free Cash Flow)
(Investing Cash Flow)
(ROE)
(ROA)
(Accounts Receivable)
(Accounts Payable)
(PMI)
GDP

36. Taiwan Market Observation Post System ( )
Taiwan Stock Exchange ( )
Taiwan Depository & Clearing Corporation ( )
Yahoo Stock Feed
…
…

37. Yearly - dividend, remuneration of directors and supervisors
Quarterly - quarterly financial statements
Monthly - Revenue
Weekly -
Daily - closing price
Hourly - stock news from Yahoo stock feed
Minutely - important news from Taiwan Market Observation Post System

39. Something like this,
but written in PHP
A super long running process (1 hour+)
loops from the first stock to the last one
Stock.find_each do |stock|
# download xml financial report data
…
# extract xml data
…
# calculate advanced data
…
end

40. A super long running process
for quarterly report

41. A super long running process
for quarterly report
A super long running process
for monthly revenue

42. A super long running process
for quarterly report
A super long running process
for monthly revenue
A super long running process
for daily price

43. A super long running process
for quarterly report
A super long running process
for monthly revenue
A super long running process
for daily price
A super long running process
for news
.
.
.

44. • Really slow

45. • Really slow
• Inefficient - unable to only retry the failed one

46. • Really slow
• Inefficient - unable to only retry the failed one
• Unpredictable server loading

47. Job 1 Job 2 Job 3
Time
When the server loading is low
Job 4 Job 5
Server
loading

48. When the server loading is HIGH
Time
Server
loading
Other task

49. Job 1
Job 2
Job 3
When the server loading is HIGH
Job 4
Job 5
Time
Server
loading
Other task

50. Job 1
Job 2
Job 3
When the server loading is HIGH
Job 4
Job 5
Time
Server
loading
Other task
Too many crawler processes executed at the same time

51. • Really slow
• Inefficient - unable to only retry the failed one.
• Unpredictable server loading
• Scale out is not easy

52. • Inherent problems of Unix Cron:

53. • Inherent problems of Unix Cron:
• Unreliable scheduling

54. • Inherent problems of Unix Cron:
• Unreliable scheduling
• High availability is not easy

55. • Inherent problems of Unix Cron:
• Unreliable scheduling
• High availability is not easy
• Hard to prioritize job by the popularity

56. • Inherent problems of Unix Cron:
• Unreliable scheduling
• High availability is not easy
• Hard to prioritize job by the popularity
• Not easy to deal with bandwidth throttling issue

58.
Created by Mike Perham

60. Web server
Request
Request
Request
.
.
.
Process

61. Request
Request
Request
.
.
.
Job queue
push to queue
(very fast)
Web server
Process

62. Request
Request
Request
.
.
.
Job queue
push to queue
(very fast)
Worker process
Worker process
.
.
.
Worker server
Worker process
Web server
Process

63. Request
Request
Request
.
.
.
Job queue
push to queue
(very fast)
Worker process
Worker process
.
.
.
Worker server
Worker process
Web server
Process Add extra servers
when needed

64. Request
Request
Request
.
.
.
Job queue
push to queue
(very fast)
Producer
Worker process
Worker process
.
.
.
Worker server
Worker process
Web server
Process

65. Request
Request
Request
.
.
.
Job queue
push to queue
(very fast)
Producer
Consumer
Worker process
Worker process
.
.
.
Worker server
Worker process
Web server
Process

66. Worker process
thread 1
thread 2
thread 3
thread 25
.
.
.
Worker process v.s.
Multi-threadSingle process

67. Worker process
thread 1
thread 2
thread 3
thread 25
.
.
.
Worker process
1 : 25
Multi-threadSingle process

68. Multi-thread
Worker process
thread 1
thread 2
thread 3
thread 25
.
.
.
Single process
Worker process
1 : 25
With the same degree of memory consumption

69. Sidekiq (OSS)
Sidekiq Pro
Sidekiq Enterprise

70. Sidekiq Pro Sidekiq Enterprise
Batches
Enhanced Reliability
Search in Web UI
Worker Metrics
Expiring Jobs
Rate Limiting
Periodic Jobs
Unique Jobs
Historical Metrics
Multi-process
Encryption

72. Parallelism Make Things Faster

73. • Really slow
• Inefficient - unable to only retry the failed one.
• Unpredictable server loading
• Scale out is not easy

74. • Efficient - only retry the failed one
• Predictable server loading
• Easy to scale out

75. • Really slow
• Inefficient - unable to only retry the failed one.
• Unpredictable server loading
• Scale out is not easy

76. • Inherent problem of Unix Cron:
• Unreliable scheduling
• High availability is not easy
• Hard to prioritize job by the popularity
• Not easy to deal with bandwidth throttling issue

79. –Mike Perham, CEO, Contributed Systems,
Creator of Sidekiq

80. Keep states of cron executions in
our robustest part of system - database
All scheduled jobs are invoked by a particular job
executed minutely

81. Keep states of cron executions in
our robustest part of system - database
All scheduled jobs are invoked by a particular job
executed minutely

82. create_table :cron_jobs do |t|
t.string :klass, null: false
t.string :cron_expression, null: false
t.timestamp :next_run_at, null: false, index: true
end
Create table for storing cron settings
table name: cron_jobs

83. create_table :cron_jobs do |t|
t.string :klass, null: false
t.string :cron_expression, null: false
t.timestamp :next_run_at, null: false, index: true
end
Create table for storing cron settings
worker class name

84. create_table :cron_jobs do |t|
t.string :klass, null: false
t.string :cron_expression, null: false
t.timestamp :next_run_at, null: false, index: true
end
Create table for storing cron settings
Something like
0 */2 * * *

85. create_table :cron_jobs do |t|
t.string :klass, null: false
t.string :cron_expression, null: false
t.timestamp :next_run_at, null: false, index: true
end
Create table for storing cron settings
when will a job should be executed

86. klass cron_expression next_run_at
Push2000NewsJobs “0 */2 * * *” …
Push2000DailyPriceJobs “0 2 * * 1-5” …
Push2000MonthlyRevenueJobs “0 0 10 * *” …
…

87. # Add to your Cron setting
every :minute do
runner 'CronJobWorker.perform_async'
end
Cron only schedules one job minutely

88. class CronJobWorker
include Sidekiq::Worker
def perform
CronJob.find_each("next_run_at <= ?", Time.now) do |job|
end
end
end
CronJobWorker to invoke all of your crawlers
Find jobs should be executed

89. class CronJobWorker
include Sidekiq::Worker
def perform
CronJob.find_each("next_run_at <= ?", Time.now) do |job|
Sidekiq::Client.push(
class: job.klass.constantize,
args: ['foo', ‘bar']
)
end
end
end
CronJobWorker to invoke all of your crawlers
Push jobs to job queue

90. class CronJobWorker
include Sidekiq::Worker
def perform
CronJob.find_each("next_run_at <= ?", Time.now) do |job|
Sidekiq::Client.push(
class: job.klass.constantize,
args: ['foo', ‘bar']
)
x = Sidekiq::CronParser.new(job.cron_expression)
job.update!(next_run_at: x.next.to_time)
end
end
end
CronJobWorker to invoke all of your crawlers
Setup the next execution time

91. class CronJobWorker
include Sidekiq::Worker
def perform
CronJob.find_each("next_run_at <= ?", Time.now) do |job|
Sidekiq::Client.push(
class: job.klass.constantize,
args: ['foo', ‘bar']
)
x = Sidekiq::CronParser.new(job.cron_expression)
job.update!(next_run_at: x.next.to_time)
end
end
end
CronJobWorker to invoke all of your crawlers

92. The missed job executions will be
executed at next minute

93. • Inherent problem of Unix Cron:
• Unreliable scheduling
• Hard to prioritize job by the popularity
• High availability is not easy
• Not easy to deal with bandwidth throttling issue

94. Drawbacks solved
• Inherent problem of Unix Cron:
• Unreliable scheduling
• Hard to prioritize job by the popularity
• High availability is not easy
• Not easy to deal with bandwidth throttling issue

95.
table: cron_jobs
klass cron_expression args next_run_at
Push2000NewsJobs “0 */2 * * *” [] …

96.
table: cron_jobs
klass cron_expression args next_run_at
Push2000NewsJobs “0 */2 * * *” [] …
NewsWorker “*/30 * * * *” [popular_stock_id_1] …
NewsWorker “*/30 * * * *” [popular_stock_id_2] …
…

97. Drawbacks solved
• Inherent problem of Unix Cron:
• Unreliable scheduling
• Hard to prioritize job by the popularity
• High availability is not easy
• Not easy to deal with bandwidth throttling issue

98. • Inherent problem of Unix Cron:
• Unreliable scheduling
• Hard to prioritize job by the popularity
• High availability is not easy
• Not easy to deal with bandwidth throttling issue

99. Sidekiq.configure_server do |config|
config.periodic do |mgr|
mgr.register("* * * * * *", CronJobWorker)
end
end

100. • Inherent problem of Unix Cron:
• Unreliable scheduling
• Hard to prioritize job by the popularity
• High availability is not easy
• Not easy to deal with bandwidth throttling issue

101. • Inherent problem of Unix Cron:
• Unreliable scheduling
• Hard to prioritize job by the popularity
• High availability is not easy
• Not easy to deal with bandwidth throttling issue

103. You always want your crawler
as fast as possible

104. However, your target server doesn’t
always allow you to crawl with
unlimited rate

105. Insert 2000 jobs to the queue at the same time
Stock.pluck(:id).each do |stock_id|
SomeWorker.perform_async(stock_id)
end
If you want to craw data for your 2000 stocks

106. Assume a target server accepts request at
maximum rate equals to
1 request / second

107. Time
(second)
1 2 3
job1
job2
job3
.
.
.
job2000
Insert 2000 jobs to the queue at the same time
All of your jobs may be blocked (except the first one)

108. Improvement 1
Schedule jobs with incremental delays
Stock.pluck(:id).each_with_index do |stock_id, index|
SomeWorker.perform_in(index, stock_id)
end

109. Time
(second)
1 2 3
job1 job2 job3
…
job2000
2000

110. Workable, but…
1
job1 job2 job3
…
job2000
If the target server is unreachable
Time
(second)

111. Workable, but…
1 2 3
job1 job2 job3
…
job2000
2000
If the target server is unreachable
job3~2000 will still execute at the same time
Time
(second)

112. • Limit your worker thread to perform specific job
with bounded rate
• Sidekiq Enterprise provides two types of rate
limiting API

113. CONCURRENT_LIMITER = Sidekiq::Limiter.concurrent('price', 10)
def perform(...)
CONCURRENT_LIMITER.within_limit do
# crawl stock data
end
end

114. CONCURRENT_LIMITER = Sidekiq::Limiter.concurrent('price', 10)
def perform(...)
CONCURRENT_LIMITER.within_limit do
# crawl stock data
end
end
Only 10 concurrent operations inside the block
can happen at any given moment

115. BUCKET_LIMITER = Sidekiq::Limiter.bucket('price', 10, :second)
def perform(...)
BUCKET_LIMITER.within_limit do
# crawl stock data
end
end
For every second, you can perform up to 10 operations

116. You must fine tune parameters of your limiter
for each data source for better performance

117. By far, you already got better performance.
However, the throttling control of your target server
may not always be static.
Many websites are dynamically throttling controlled.

118.
If throttling detected, pause your workers for a while

119. Redis (job queue)

120. Redis (job queue)
default
critical
low

121. Redis (job queue)
default
critical
low
Worker thread
Worker thread
Worker thread
Worker thread
Worker thread

122. Redis (job queue)
default
critical
low
Worker thread
Worker thread
Worker thread
Worker thread
Worker thread

123. Redis (job queue)
default
critical
low
Worker thread
Worker thread
Worker thread
Worker thread
Worker thread
yahoo

124. Redis (job queue)
default
critical
low
Worker thread
Worker thread
Worker thread
Worker thread
Worker thread
yahoo
(paused)
Pause this queue when throttled

125. Redis (job queue)
default
critical
low
Worker thread
Worker thread
Worker thread
Worker thread
Worker thread
Schedule a job executed after few seconds
to “unpause" job in another queue
yahoo
(paused)

126. Redis (job queue)
default
critical
low
Worker thread
Worker thread
Worker thread
Worker thread
Worker thread
yahoo
(resumed)
Resumed after the unpause queue job executed

127. class SomeWorker
include Sidekiq::Worker
def perform
# try to crawl something
# ...
if throttled
queue_name = self.class.get_sidekiq_options['queue']
queue = Sidekiq::Queue.new(queue_name)
queue.pause!
ResumeJobQueueWorker.perform_in(30.seconds, queue_name)
end
end
end

128. class SomeWorker
include Sidekiq::Worker
def perform
# try to crawl something
# ...
if throttled
queue_name = self.class.get_sidekiq_options['queue']
queue = Sidekiq::Queue.new(queue_name)
queue.pause!
ResumeJobQueueWorker.perform_in(30.seconds, queue_name)
end
end
end

129. class SomeWorker
include Sidekiq::Worker
def perform
# try to crawl something
# ...
if throttled
queue_name = self.class.get_sidekiq_options['queue']
queue = Sidekiq::Queue.new(queue_name)
queue.pause!
ResumeJobQueueWorker.perform_in(30.seconds, queue_name)
end
end
end
class ResumeJobQueueWorker
include Sidekiq::Worker
sidekiq_options queue: :queue_control, unique: :until_executed
def perform(queue_name)
queue = Sidekiq::Queue.new(queue_name)
queue.unpause! if queue.paused?
end
end

130. The queue for ResumeJobQueueWorker
MUST NOT equal to the paused queue
We have a dedicated queue for
ResumeJobQueueWorker

131. Decrease Sidekiq server poll interval for more
precise timing control

132. Queue pausing alleviates throttling issues
Is it possible for us to do things even better?

133. Most throttling control aim to block requests
from the same IP address

134. We can change our IP address via
proxy service

135. Sidekiq
server
Target
server
a.b.c.d

136. Sidekiq
server
Target
server
a.b.c.d
a.b.c.d

137. Sidekiq
server
Target
server
a.b.c.d
a.b.c.d
a.b.c.d
a.b.c.d
Same IP for each request

138. Sidekiq
server
Target
server
a.b.c.d
Proxy
service
end
point

139. Sidekiq
server
Target
server
a.b.c.d
Proxy
service
end
point
proxy server
e.f.g.h

140. Sidekiq
server
Target
server
a.b.c.d
a.b.c.d
Proxy
service
end
point
proxy server
proxy server
e.f.g.h
i.j.k.l

141. Sidekiq
server
Target
server
a.b.c.d
a.b.c.d
a.b.c.d
a.b.c.d
Proxy
service
end
point
proxy server
proxy server
proxy server
proxy server
e.f.g.h
i.j.k.l
m.n.o.p
q.r.s.t

142. Sidekiq
server
Target
server
a.b.c.d
a.b.c.d
a.b.c.d
a.b.c.d
Proxy
service
end
point
proxy server
proxy server
proxy server
proxy server
e.f.g.h
i.j.k.l
m.n.o.p
q.r.s.t
Different IP for each request

143. • Inherent problem of Unix Cron:
• Unreliable scheduling
• Hard to prioritize job by the popularity
• High availability is not easy
• Not easy to deal with bandwidth throttling issue

145. • With Sidekiq (Enterprise) and a proper design, the following problems
are solved
• Slow crawler
• Inefficient - unable to only retry the failed one
• Unpredictable server loading
• Scale out is not easy
• Inherent problem of Unix Cron
• Not easy to deal with bandwidth throttling issue