Enviar búsqueda
Cargar
Background processing
•
3 recomendaciones
•
1,317 vistas
Kerry Buckley
Seguir
Empresariales
Tecnología
Denunciar
Compartir
Denunciar
Compartir
1 de 21
Descargar ahora
Descargar para leer sin conexión
Recomendados
COOKPADでのHadoop利用
COOKPADでのHadoop利用
Tatsuya Sasaki
R in Minecraft
R in Minecraft
Revolution Analytics
Hadoop and MapReduce
Hadoop and MapReduce
Hemanth Kumar Mantri
Elasticwulf Pycon Talk
Elasticwulf Pycon Talk
Peter Skomoroch
Chef for OpenStack: OpenStack Spring Summit 2013
Chef for OpenStack: OpenStack Spring Summit 2013
Matt Ray
Prototyping Data Intensive Apps: TrendingTopics.org
Prototyping Data Intensive Apps: TrendingTopics.org
Peter Skomoroch
Fact-Based Monitoring - PuppetConf 2014
Fact-Based Monitoring - PuppetConf 2014
Puppet
Migrating Existing Open Source Machine Learning to Azure
Migrating Existing Open Source Machine Learning to Azure
Revolution Analytics
Recomendados
COOKPADでのHadoop利用
COOKPADでのHadoop利用
Tatsuya Sasaki
R in Minecraft
R in Minecraft
Revolution Analytics
Hadoop and MapReduce
Hadoop and MapReduce
Hemanth Kumar Mantri
Elasticwulf Pycon Talk
Elasticwulf Pycon Talk
Peter Skomoroch
Chef for OpenStack: OpenStack Spring Summit 2013
Chef for OpenStack: OpenStack Spring Summit 2013
Matt Ray
Prototyping Data Intensive Apps: TrendingTopics.org
Prototyping Data Intensive Apps: TrendingTopics.org
Peter Skomoroch
Fact-Based Monitoring - PuppetConf 2014
Fact-Based Monitoring - PuppetConf 2014
Puppet
Migrating Existing Open Source Machine Learning to Azure
Migrating Existing Open Source Machine Learning to Azure
Revolution Analytics
Swaggered web apis in Clojure
Swaggered web apis in Clojure
Metosin Oy
(CMP310) Data Processing Pipelines Using Containers & Spot Instances
(CMP310) Data Processing Pipelines Using Containers & Spot Instances
Amazon Web Services
Building a Dev/Test Cloud with Apache CloudStack
Building a Dev/Test Cloud with Apache CloudStack
ke4qqq
Voxxed Days Thesaloniki 2016 - Scaling react.js applications
Voxxed Days Thesaloniki 2016 - Scaling react.js applications
Voxxed Days Thessaloniki
M3D - Metadata Driven Development
M3D - Metadata Driven Development
Henning Kropp
Drupal 8 + Elasticsearch + Docker
Drupal 8 + Elasticsearch + Docker
Roald Umandal
Luigi presentation OA Summit
Luigi presentation OA Summit
Open Analytics
A few questions on MongoDB
A few questions on MongoDB
Tom Crinson
Drupal and Elasticsearch
Drupal and Elasticsearch
Nikolay Ignatov
Storm distributed processing
Storm distributed processing
ducquoc_vn
Configuration management
Configuration management
Luca De Vitis
Resource Scheduling using Apache Mesos in Cloud Native Environments
Resource Scheduling using Apache Mesos in Cloud Native Environments
Sharma Podila
Apache Airflow (incubating) NL HUG Meetup 2016-07-19
Apache Airflow (incubating) NL HUG Meetup 2016-07-19
Bolke de Bruin
Airflow at WePay
Airflow at WePay
Chris Riccomini
Running Airflow Workflows as ETL Processes on Hadoop
Running Airflow Workflows as ETL Processes on Hadoop
clairvoyantllc
Heroku Dockerの使い所
Heroku Dockerの使い所
Yusuke Kon
kumogata-template の紹介
kumogata-template の紹介
Naoya Nakazawa
Building Robust Pipelines with Airflow
Building Robust Pipelines with Airflow
Erin Shellman
Node collaboration - sharing information between your systems
Node collaboration - sharing information between your systems
m_richardson
20150627 bigdatala
20150627 bigdatala
gethue
Leonard Turkel Nonprofit Network: Social Media & Building Attendee Consensus
Leonard Turkel Nonprofit Network: Social Media & Building Attendee Consensus
Alex de Carvalho
AZEC2012 - Social Business in the Enterprise
AZEC2012 - Social Business in the Enterprise
Rawn Shah
Más contenido relacionado
La actualidad más candente
Swaggered web apis in Clojure
Swaggered web apis in Clojure
Metosin Oy
(CMP310) Data Processing Pipelines Using Containers & Spot Instances
(CMP310) Data Processing Pipelines Using Containers & Spot Instances
Amazon Web Services
Building a Dev/Test Cloud with Apache CloudStack
Building a Dev/Test Cloud with Apache CloudStack
ke4qqq
Voxxed Days Thesaloniki 2016 - Scaling react.js applications
Voxxed Days Thesaloniki 2016 - Scaling react.js applications
Voxxed Days Thessaloniki
M3D - Metadata Driven Development
M3D - Metadata Driven Development
Henning Kropp
Drupal 8 + Elasticsearch + Docker
Drupal 8 + Elasticsearch + Docker
Roald Umandal
Luigi presentation OA Summit
Luigi presentation OA Summit
Open Analytics
A few questions on MongoDB
A few questions on MongoDB
Tom Crinson
Drupal and Elasticsearch
Drupal and Elasticsearch
Nikolay Ignatov
Storm distributed processing
Storm distributed processing
ducquoc_vn
Configuration management
Configuration management
Luca De Vitis
Resource Scheduling using Apache Mesos in Cloud Native Environments
Resource Scheduling using Apache Mesos in Cloud Native Environments
Sharma Podila
Apache Airflow (incubating) NL HUG Meetup 2016-07-19
Apache Airflow (incubating) NL HUG Meetup 2016-07-19
Bolke de Bruin
Airflow at WePay
Airflow at WePay
Chris Riccomini
Running Airflow Workflows as ETL Processes on Hadoop
Running Airflow Workflows as ETL Processes on Hadoop
clairvoyantllc
Heroku Dockerの使い所
Heroku Dockerの使い所
Yusuke Kon
kumogata-template の紹介
kumogata-template の紹介
Naoya Nakazawa
Building Robust Pipelines with Airflow
Building Robust Pipelines with Airflow
Erin Shellman
Node collaboration - sharing information between your systems
Node collaboration - sharing information between your systems
m_richardson
20150627 bigdatala
20150627 bigdatala
gethue
La actualidad más candente
(20)
Swaggered web apis in Clojure
Swaggered web apis in Clojure
(CMP310) Data Processing Pipelines Using Containers & Spot Instances
(CMP310) Data Processing Pipelines Using Containers & Spot Instances
Building a Dev/Test Cloud with Apache CloudStack
Building a Dev/Test Cloud with Apache CloudStack
Voxxed Days Thesaloniki 2016 - Scaling react.js applications
Voxxed Days Thesaloniki 2016 - Scaling react.js applications
M3D - Metadata Driven Development
M3D - Metadata Driven Development
Drupal 8 + Elasticsearch + Docker
Drupal 8 + Elasticsearch + Docker
Luigi presentation OA Summit
Luigi presentation OA Summit
A few questions on MongoDB
A few questions on MongoDB
Drupal and Elasticsearch
Drupal and Elasticsearch
Storm distributed processing
Storm distributed processing
Configuration management
Configuration management
Resource Scheduling using Apache Mesos in Cloud Native Environments
Resource Scheduling using Apache Mesos in Cloud Native Environments
Apache Airflow (incubating) NL HUG Meetup 2016-07-19
Apache Airflow (incubating) NL HUG Meetup 2016-07-19
Airflow at WePay
Airflow at WePay
Running Airflow Workflows as ETL Processes on Hadoop
Running Airflow Workflows as ETL Processes on Hadoop
Heroku Dockerの使い所
Heroku Dockerの使い所
kumogata-template の紹介
kumogata-template の紹介
Building Robust Pipelines with Airflow
Building Robust Pipelines with Airflow
Node collaboration - sharing information between your systems
Node collaboration - sharing information between your systems
20150627 bigdatala
20150627 bigdatala
Destacado
Leonard Turkel Nonprofit Network: Social Media & Building Attendee Consensus
Leonard Turkel Nonprofit Network: Social Media & Building Attendee Consensus
Alex de Carvalho
AZEC2012 - Social Business in the Enterprise
AZEC2012 - Social Business in the Enterprise
Rawn Shah
Are web managers still needed when everyone is a web 'expert'?
Are web managers still needed when everyone is a web 'expert'?
IWMW
Search Engines in the fight against Institutional Impecuniousness
Search Engines in the fight against Institutional Impecuniousness
IWMW
TDD refresher
TDD refresher
Kerry Buckley
Blooms Taxonomy
Blooms Taxonomy
Doug Adams
Destacado
(6)
Leonard Turkel Nonprofit Network: Social Media & Building Attendee Consensus
Leonard Turkel Nonprofit Network: Social Media & Building Attendee Consensus
AZEC2012 - Social Business in the Enterprise
AZEC2012 - Social Business in the Enterprise
Are web managers still needed when everyone is a web 'expert'?
Are web managers still needed when everyone is a web 'expert'?
Search Engines in the fight against Institutional Impecuniousness
Search Engines in the fight against Institutional Impecuniousness
TDD refresher
TDD refresher
Blooms Taxonomy
Blooms Taxonomy
Similar a Background processing
Using Riak for Events storage and analysis at Booking.com
Using Riak for Events storage and analysis at Booking.com
Damien Krotkine
Hadoop introduction
Hadoop introduction
musrath mohammad
Real time Analytics with Apache Kafka and Apache Spark
Real time Analytics with Apache Kafka and Apache Spark
Rahul Jain
[262] netflix 빅데이터 플랫폼
[262] netflix 빅데이터 플랫폼
NAVER D2
Celery: The Distributed Task Queue
Celery: The Distributed Task Queue
Richard Leland
Django at Scale
Django at Scale
bretthoerner
Rails scaling
Rails scaling
Sebastian Roth
Databases in the Hosted Cloud
Databases in the Hosted Cloud
Colin Charles
H-Hypermap - Heatmap Analytics at Scale: Presented by David Smiley, D W Smile...
H-Hypermap - Heatmap Analytics at Scale: Presented by David Smiley, D W Smile...
Lucidworks
Icinga 2009 at OSMC
Icinga 2009 at OSMC
Icinga
Tips For Maintaining OSS Projects
Tips For Maintaining OSS Projects
Taro L. Saito
视觉中国的MongoDB应用实践(QConBeijing2011)
视觉中国的MongoDB应用实践(QConBeijing2011)
Night Sailer
Atlanta OpenStack Summit: Technical Deep Dive: Big Data Computations Using El...
Atlanta OpenStack Summit: Technical Deep Dive: Big Data Computations Using El...
Sergey Lukjanov
MongoDB开发应用实践
MongoDB开发应用实践
iammutex
The MySQL Server ecosystem in 2016
The MySQL Server ecosystem in 2016
Colin Charles
Hadoop ppt1
Hadoop ppt1
chariorienit
Hadoop Operations - Best practices from the field
Hadoop Operations - Best practices from the field
Uwe Printz
OSDC 2017 | Lessons from database failures by Colin Charles
OSDC 2017 | Lessons from database failures by Colin Charles
NETWAYS
把鐵路開進視窗裡
把鐵路開進視窗裡
Wei Jen Lu
Impression of Rails 3
Impression of Rails 3
Kosuke Matsuda
Similar a Background processing
(20)
Using Riak for Events storage and analysis at Booking.com
Using Riak for Events storage and analysis at Booking.com
Hadoop introduction
Hadoop introduction
Real time Analytics with Apache Kafka and Apache Spark
Real time Analytics with Apache Kafka and Apache Spark
[262] netflix 빅데이터 플랫폼
[262] netflix 빅데이터 플랫폼
Celery: The Distributed Task Queue
Celery: The Distributed Task Queue
Django at Scale
Django at Scale
Rails scaling
Rails scaling
Databases in the Hosted Cloud
Databases in the Hosted Cloud
H-Hypermap - Heatmap Analytics at Scale: Presented by David Smiley, D W Smile...
H-Hypermap - Heatmap Analytics at Scale: Presented by David Smiley, D W Smile...
Icinga 2009 at OSMC
Icinga 2009 at OSMC
Tips For Maintaining OSS Projects
Tips For Maintaining OSS Projects
视觉中国的MongoDB应用实践(QConBeijing2011)
视觉中国的MongoDB应用实践(QConBeijing2011)
Atlanta OpenStack Summit: Technical Deep Dive: Big Data Computations Using El...
Atlanta OpenStack Summit: Technical Deep Dive: Big Data Computations Using El...
MongoDB开发应用实践
MongoDB开发应用实践
The MySQL Server ecosystem in 2016
The MySQL Server ecosystem in 2016
Hadoop ppt1
Hadoop ppt1
Hadoop Operations - Best practices from the field
Hadoop Operations - Best practices from the field
OSDC 2017 | Lessons from database failures by Colin Charles
OSDC 2017 | Lessons from database failures by Colin Charles
把鐵路開進視窗裡
把鐵路開進視窗裡
Impression of Rails 3
Impression of Rails 3
Más de Kerry Buckley
Jasmine
Jasmine
Kerry Buckley
Testing http calls with Webmock and VCR
Testing http calls with Webmock and VCR
Kerry Buckley
BDD with cucumber
BDD with cucumber
Kerry Buckley
Ruby nooks & crannies
Ruby nooks & crannies
Kerry Buckley
Javasccript MV* frameworks
Javasccript MV* frameworks
Kerry Buckley
Tdd for BT E2E test community
Tdd for BT E2E test community
Kerry Buckley
7li7w devcon5
7li7w devcon5
Kerry Buckley
What I learned from Seven Languages in Seven Weeks (IPRUG)
What I learned from Seven Languages in Seven Weeks (IPRUG)
Kerry Buckley
Functional ruby
Functional ruby
Kerry Buckley
Adastral Park code retreat introduction
Adastral Park code retreat introduction
Kerry Buckley
MongoMapper lightning talk
MongoMapper lightning talk
Kerry Buckley
Ruby
Ruby
Kerry Buckley
Cloud
Cloud
Kerry Buckley
The secret life of bees
The secret life of bees
Kerry Buckley
Katas, Contests and Coding Dojos
Katas, Contests and Coding Dojos
Kerry Buckley
Rack
Rack
Kerry Buckley
Doing REST Right
Doing REST Right
Kerry Buckley
Kanban and Iterationless Working
Kanban and Iterationless Working
Kerry Buckley
Software Development Trends
Software Development Trends
Kerry Buckley
TDD
TDD
Kerry Buckley
Más de Kerry Buckley
(20)
Jasmine
Jasmine
Testing http calls with Webmock and VCR
Testing http calls with Webmock and VCR
BDD with cucumber
BDD with cucumber
Ruby nooks & crannies
Ruby nooks & crannies
Javasccript MV* frameworks
Javasccript MV* frameworks
Tdd for BT E2E test community
Tdd for BT E2E test community
7li7w devcon5
7li7w devcon5
What I learned from Seven Languages in Seven Weeks (IPRUG)
What I learned from Seven Languages in Seven Weeks (IPRUG)
Functional ruby
Functional ruby
Adastral Park code retreat introduction
Adastral Park code retreat introduction
MongoMapper lightning talk
MongoMapper lightning talk
Ruby
Ruby
Cloud
Cloud
The secret life of bees
The secret life of bees
Katas, Contests and Coding Dojos
Katas, Contests and Coding Dojos
Rack
Rack
Doing REST Right
Doing REST Right
Kanban and Iterationless Working
Kanban and Iterationless Working
Software Development Trends
Software Development Trends
TDD
TDD
Último
Darshan Hiranandani [News About Next CEO].pdf
Darshan Hiranandani [News About Next CEO].pdf
Shashank Mehta
Entrepreneurship lessons in Philippines
Entrepreneurship lessons in Philippines
DavidSamuel525586
Independent Call Girls Andheri Nightlaila 9967584737
Independent Call Girls Andheri Nightlaila 9967584737
Riya Pathan
Unlocking the Future: Explore Web 3.0 Workshop to Start Earning Today!
Unlocking the Future: Explore Web 3.0 Workshop to Start Earning Today!
Doge Mining Website
PB Project 1: Exploring Your Personal Brand
PB Project 1: Exploring Your Personal Brand
SharisaBethune
Ten Organizational Design Models to align structure and operations to busines...
Ten Organizational Design Models to align structure and operations to busines...
Seta Wicaksana
International Business Environments and Operations 16th Global Edition test b...
International Business Environments and Operations 16th Global Edition test b...
ssuserf63bd7
8447779800, Low rate Call girls in Tughlakabad Delhi NCR
8447779800, Low rate Call girls in Tughlakabad Delhi NCR
ashishs7044
Financial-Statement-Analysis-of-Coca-cola-Company.pptx
Financial-Statement-Analysis-of-Coca-cola-Company.pptx
saniyaimamuddin
TriStar Gold Corporate Presentation - April 2024
TriStar Gold Corporate Presentation - April 2024
Adnet Communications
Buy gmail accounts.pdf Buy Old Gmail Accounts
Buy gmail accounts.pdf Buy Old Gmail Accounts
Buy Verified Accounts
No-1 Call Girls In Goa 93193 VIP 73153 Escort service In North Goa Panaji, Ca...
No-1 Call Girls In Goa 93193 VIP 73153 Escort service In North Goa Panaji, Ca...
Call girls in Goa, +91 9319373153 Escort Service in North Goa
8447779800, Low rate Call girls in Kotla Mubarakpur Delhi NCR
8447779800, Low rate Call girls in Kotla Mubarakpur Delhi NCR
ashishs7044
Cybersecurity Awareness Training Presentation v2024.03
Cybersecurity Awareness Training Presentation v2024.03
DallasHaselhorst
Call Girls Contact Number Andheri 9920874524
Call Girls Contact Number Andheri 9920874524
najka9823
Church Building Grants To Assist With New Construction, Additions, And Restor...
Church Building Grants To Assist With New Construction, Additions, And Restor...
Americas Got Grants
FULL ENJOY Call girls in Paharganj Delhi | 8377087607
FULL ENJOY Call girls in Paharganj Delhi | 8377087607
dollysharma2066
Traction part 2 - EOS Model JAX Bridges.
Traction part 2 - EOS Model JAX Bridges.
Anamaria Contreras
NewBase 19 April 2024 Energy News issue - 1717 by Khaled Al Awadi.pdf
NewBase 19 April 2024 Energy News issue - 1717 by Khaled Al Awadi.pdf
Khaled Al Awadi
8447779800, Low rate Call girls in Dwarka mor Delhi NCR
8447779800, Low rate Call girls in Dwarka mor Delhi NCR
ashishs7044
Último
(20)
Darshan Hiranandani [News About Next CEO].pdf
Darshan Hiranandani [News About Next CEO].pdf
Entrepreneurship lessons in Philippines
Entrepreneurship lessons in Philippines
Independent Call Girls Andheri Nightlaila 9967584737
Independent Call Girls Andheri Nightlaila 9967584737
Unlocking the Future: Explore Web 3.0 Workshop to Start Earning Today!
Unlocking the Future: Explore Web 3.0 Workshop to Start Earning Today!
PB Project 1: Exploring Your Personal Brand
PB Project 1: Exploring Your Personal Brand
Ten Organizational Design Models to align structure and operations to busines...
Ten Organizational Design Models to align structure and operations to busines...
International Business Environments and Operations 16th Global Edition test b...
International Business Environments and Operations 16th Global Edition test b...
8447779800, Low rate Call girls in Tughlakabad Delhi NCR
8447779800, Low rate Call girls in Tughlakabad Delhi NCR
Financial-Statement-Analysis-of-Coca-cola-Company.pptx
Financial-Statement-Analysis-of-Coca-cola-Company.pptx
TriStar Gold Corporate Presentation - April 2024
TriStar Gold Corporate Presentation - April 2024
Buy gmail accounts.pdf Buy Old Gmail Accounts
Buy gmail accounts.pdf Buy Old Gmail Accounts
No-1 Call Girls In Goa 93193 VIP 73153 Escort service In North Goa Panaji, Ca...
No-1 Call Girls In Goa 93193 VIP 73153 Escort service In North Goa Panaji, Ca...
8447779800, Low rate Call girls in Kotla Mubarakpur Delhi NCR
8447779800, Low rate Call girls in Kotla Mubarakpur Delhi NCR
Cybersecurity Awareness Training Presentation v2024.03
Cybersecurity Awareness Training Presentation v2024.03
Call Girls Contact Number Andheri 9920874524
Call Girls Contact Number Andheri 9920874524
Church Building Grants To Assist With New Construction, Additions, And Restor...
Church Building Grants To Assist With New Construction, Additions, And Restor...
FULL ENJOY Call girls in Paharganj Delhi | 8377087607
FULL ENJOY Call girls in Paharganj Delhi | 8377087607
Traction part 2 - EOS Model JAX Bridges.
Traction part 2 - EOS Model JAX Bridges.
NewBase 19 April 2024 Energy News issue - 1717 by Khaled Al Awadi.pdf
NewBase 19 April 2024 Energy News issue - 1717 by Khaled Al Awadi.pdf
8447779800, Low rate Call girls in Dwarka mor Delhi NCR
8447779800, Low rate Call girls in Dwarka mor Delhi NCR
Background processing
1.
Background Processing Kerry Buckley IPRUG 3
November 2010
2.
Why? • Sending e-mail,
SMS, tweets etc • Resizing images • Generating large PDFs • Updating search indexes • etc
3.
Approaches • Process/thread based •
Spawn • Database-backed • BackgroundDRb, BackgroundJob, DelayedJob, JobFu, Background-Fu • Queue-backed • Sparrow, Starling, Kestrel, RabbitMQ, ActiveMQ, SQS, Beanstalkd, Resque
4.
5.
Delayed::Job
6.
Delayed::Job • Extracted from
Shopify • Daemon runs with Rails environment • Use collectiveidea’s fork • 2.0 for Rails 2.x • 2.1RC (from GitHub) for Rails 3 • ActiveRecord/MongoMapper/DataMapper
7.
Setup Gemfile: gem "delayed_job", :git => "git://github.com/collectiveidea/delayed_job.git" Then
run: $ rails g delayed_job $ rake db:migrate $ script/delayed_job start
8.
Delaying methods Mailer.send_alerts(event)
9.
Delaying methods Mailer.delay.send_alerts(event)
10.
Options Mailer.delay(:priority => 3, run_at
=> 1.hour.from_now). send_alerts(event)
11.
Running workers Foreground $ rake
jobs:work $ script/delayed_job run Daemonised $ script/delayed_job start $ RAILS_ENV=production script/delayed_job -n 4 start
12.
Custom Jobs class NewsletterJob
< Struct.new(:text, :emails) def perform emails.each do |e| NewsletterMailer.deliver_text_to_email(text, e) end end end emails = Customers.all.collect(&:email) Delayed::Job.enqueue NewsletterJob.new(text, emails)
13.
Hooks def enqueue(job) end def before(job) end def
after(job) end def success(job) end def error(job, exception) end def failure end
14.
Resque
15.
Resque • Created by
GitHub • Jobs persisted to Redis as JSON • Multiple queues • Parent/child forking architecture • Web monitoring interface
16.
Setup Redis: $ brew install
redis Gemfile: gem "resque" Rakefile: require "resque/tasks" task "resque:setup" => :environment
17.
Delaying methods class Archive @queue
= :file_serve def self.perform(repo_id, branch = 'master') repo = Repository.find(repo_id) repo.create_archive(branch) end end Resque.enqueue(Archive, repository.id, branch)
18.
Running queues $ QUEUE=queue_name
rake resque:work $ QUEUES=file_serve,warm_cache rake resque:work $ QUEUES=critical,high,low rake resque:work $ QUEUE=* rake resque:work $ COUNT=5 QUEUE=* rake resque:workers
19.
Monitoring $ resque-web
20.
Which to use? Delayed::Job
Resque How much of your site is background work? Less than 50% More than 50%
21.
Demo
Descargar ahora