Submit Search
Upload
Kiwipycon2011 async-with-gevent-redis
•
Download as ODP, PDF
•
11 likes
•
2,478 views
alexdong
Follow
Technology
Education
Slideshow view
Report
Share
Slideshow view
Report
Share
1 of 13
Download now
Recommended
API analytics with Redis and Google Bigquery. NoSQL matters edition
API analytics with Redis and Google Bigquery. NoSQL matters edition
javier ramirez
Building highly reliable data pipeline @datadog par Quentin François
Building highly reliable data pipeline @datadog par Quentin François
Paris Data Engineers !
Clickhouse at Cloudflare. By Marek Vavrusa
Clickhouse at Cloudflare. By Marek Vavrusa
Valery Tkachenko
Spark Summit EU talk by William Benton
Spark Summit EU talk by William Benton
Spark Summit
Speed Up Uber's Presto with Alluxio
Speed Up Uber's Presto with Alluxio
Alluxio, Inc.
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
InfluxData
Let's Compare: A Benchmark review of InfluxDB and Elasticsearch
Let's Compare: A Benchmark review of InfluxDB and Elasticsearch
InfluxData
DOWNSAMPLING DATA
DOWNSAMPLING DATA
InfluxData
Recommended
API analytics with Redis and Google Bigquery. NoSQL matters edition
API analytics with Redis and Google Bigquery. NoSQL matters edition
javier ramirez
Building highly reliable data pipeline @datadog par Quentin François
Building highly reliable data pipeline @datadog par Quentin François
Paris Data Engineers !
Clickhouse at Cloudflare. By Marek Vavrusa
Clickhouse at Cloudflare. By Marek Vavrusa
Valery Tkachenko
Spark Summit EU talk by William Benton
Spark Summit EU talk by William Benton
Spark Summit
Speed Up Uber's Presto with Alluxio
Speed Up Uber's Presto with Alluxio
Alluxio, Inc.
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
InfluxData
Let's Compare: A Benchmark review of InfluxDB and Elasticsearch
Let's Compare: A Benchmark review of InfluxDB and Elasticsearch
InfluxData
DOWNSAMPLING DATA
DOWNSAMPLING DATA
InfluxData
Netflix running Presto in the AWS Cloud
Netflix running Presto in the AWS Cloud
Zhenxiao Luo
Developing high frequency indicators using real time tick data on apache supe...
Developing high frequency indicators using real time tick data on apache supe...
Zekeriya Besiroglu
Introduction to Meteor - revised edition
Introduction to Meteor - revised edition
Stephan Hochhaus
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
Vianney FOUCAULT
The Practice of Alluxio in JD.com
The Practice of Alluxio in JD.com
Alluxio, Inc.
RedisConf17 - Redis Graph
RedisConf17 - Redis Graph
Redis Labs
Taking Your Database Global with Kubernetes
Taking Your Database Global with Kubernetes
Christopher Bradford
Spark Pipelines in the Cloud with Alluxio
Spark Pipelines in the Cloud with Alluxio
Alluxio, Inc.
H2O World - PySparkling Water - Nidhi Mehta
H2O World - PySparkling Water - Nidhi Mehta
Sri Ambati
Best Practices for Using Alluxio with Spark
Best Practices for Using Alluxio with Spark
Alluxio, Inc.
Getting to Know Airflow
Getting to Know Airflow
Rosanne Hoyem
presto-at-netflix-hadoop-summit-15
presto-at-netflix-hadoop-summit-15
Zhenxiao Luo
Setting up InfluxData for IoT
Setting up InfluxData for IoT
InfluxData
Pig on spark
Pig on spark
Sigmoid
Deep Learning to Production with MLflow & RedisAI
Deep Learning to Production with MLflow & RedisAI
Databricks
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Lviv Startup Club
Intro to hadoop
Intro to hadoop
Haden Pereira
BeeGFS Enterprise Deployment
BeeGFS Enterprise Deployment
Dirk Petersen
Rental Cars and Industrialized Learning to Rank with Sean Downes
Rental Cars and Industrialized Learning to Rank with Sean Downes
Databricks
Pachyderm: Building a Big Data Beast On Kubernetes
Pachyderm: Building a Big Data Beast On Kubernetes
KubeAcademy
Clug 2011 March web server optimisation
Clug 2011 March web server optimisation
grooverdan
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
DataWorks Summit
More Related Content
What's hot
Netflix running Presto in the AWS Cloud
Netflix running Presto in the AWS Cloud
Zhenxiao Luo
Developing high frequency indicators using real time tick data on apache supe...
Developing high frequency indicators using real time tick data on apache supe...
Zekeriya Besiroglu
Introduction to Meteor - revised edition
Introduction to Meteor - revised edition
Stephan Hochhaus
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
Vianney FOUCAULT
The Practice of Alluxio in JD.com
The Practice of Alluxio in JD.com
Alluxio, Inc.
RedisConf17 - Redis Graph
RedisConf17 - Redis Graph
Redis Labs
Taking Your Database Global with Kubernetes
Taking Your Database Global with Kubernetes
Christopher Bradford
Spark Pipelines in the Cloud with Alluxio
Spark Pipelines in the Cloud with Alluxio
Alluxio, Inc.
H2O World - PySparkling Water - Nidhi Mehta
H2O World - PySparkling Water - Nidhi Mehta
Sri Ambati
Best Practices for Using Alluxio with Spark
Best Practices for Using Alluxio with Spark
Alluxio, Inc.
Getting to Know Airflow
Getting to Know Airflow
Rosanne Hoyem
presto-at-netflix-hadoop-summit-15
presto-at-netflix-hadoop-summit-15
Zhenxiao Luo
Setting up InfluxData for IoT
Setting up InfluxData for IoT
InfluxData
Pig on spark
Pig on spark
Sigmoid
Deep Learning to Production with MLflow & RedisAI
Deep Learning to Production with MLflow & RedisAI
Databricks
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Lviv Startup Club
Intro to hadoop
Intro to hadoop
Haden Pereira
BeeGFS Enterprise Deployment
BeeGFS Enterprise Deployment
Dirk Petersen
Rental Cars and Industrialized Learning to Rank with Sean Downes
Rental Cars and Industrialized Learning to Rank with Sean Downes
Databricks
Pachyderm: Building a Big Data Beast On Kubernetes
Pachyderm: Building a Big Data Beast On Kubernetes
KubeAcademy
What's hot
(20)
Netflix running Presto in the AWS Cloud
Netflix running Presto in the AWS Cloud
Developing high frequency indicators using real time tick data on apache supe...
Developing high frequency indicators using real time tick data on apache supe...
Introduction to Meteor - revised edition
Introduction to Meteor - revised edition
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
Clickhouse MeetUp@ContentSquare - ContentSquare's Experience Sharing
The Practice of Alluxio in JD.com
The Practice of Alluxio in JD.com
RedisConf17 - Redis Graph
RedisConf17 - Redis Graph
Taking Your Database Global with Kubernetes
Taking Your Database Global with Kubernetes
Spark Pipelines in the Cloud with Alluxio
Spark Pipelines in the Cloud with Alluxio
H2O World - PySparkling Water - Nidhi Mehta
H2O World - PySparkling Water - Nidhi Mehta
Best Practices for Using Alluxio with Spark
Best Practices for Using Alluxio with Spark
Getting to Know Airflow
Getting to Know Airflow
presto-at-netflix-hadoop-summit-15
presto-at-netflix-hadoop-summit-15
Setting up InfluxData for IoT
Setting up InfluxData for IoT
Pig on spark
Pig on spark
Deep Learning to Production with MLflow & RedisAI
Deep Learning to Production with MLflow & RedisAI
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Yaroslav Nedashkovsky "How to manage hundreds of pipelines for processing da...
Intro to hadoop
Intro to hadoop
BeeGFS Enterprise Deployment
BeeGFS Enterprise Deployment
Rental Cars and Industrialized Learning to Rank with Sean Downes
Rental Cars and Industrialized Learning to Rank with Sean Downes
Pachyderm: Building a Big Data Beast On Kubernetes
Pachyderm: Building a Big Data Beast On Kubernetes
Similar to Kiwipycon2011 async-with-gevent-redis
Clug 2011 March web server optimisation
Clug 2011 March web server optimisation
grooverdan
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
DataWorks Summit
WE18_Performance_Up.ppt
WE18_Performance_Up.ppt
webhostingguy
Handout3o
Handout3o
Shahbaz Sidhu
Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon
Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon
DataWorks Summit/Hadoop Summit
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Bhupesh Bansal
Hadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedIn
Hadoop User Group
Pilot Hadoop Towards 2500 Nodes and Cluster Redundancy
Pilot Hadoop Towards 2500 Nodes and Cluster Redundancy
Stuart Pook
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Cloudera, Inc.
Big data at United Airlines
Big data at United Airlines
DataWorks Summit
Eagle from eBay at China Hadoop Summit 2015
Eagle from eBay at China Hadoop Summit 2015
Hao Chen
Inroduction to Big Data
Inroduction to Big Data
Omnia Safaan
Hug syncsort etl hadoop big data
Hug syncsort etl hadoop big data
Stéphane Heckel
Syncsort et le retour d'expérience ComScore
Syncsort et le retour d'expérience ComScore
Modern Data Stack France
Composing re-useable ETL on Hadoop
Composing re-useable ETL on Hadoop
Paul Lam
Hands on Hadoop and pig
Hands on Hadoop and pig
Sudar Muthu
RAPIDS: GPU-Accelerated ETL and Feature Engineering
RAPIDS: GPU-Accelerated ETL and Feature Engineering
Keith Kraus
Arun Rathinasabapathy, Senior Software Engineer, LexisNexis at MLconf ATL 2016
Arun Rathinasabapathy, Senior Software Engineer, LexisNexis at MLconf ATL 2016
MLconf
LLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in Hive
DataWorks Summit/Hadoop Summit
Globo.com & Varnish
Globo.com & Varnish
lokama
Similar to Kiwipycon2011 async-with-gevent-redis
(20)
Clug 2011 March web server optimisation
Clug 2011 March web server optimisation
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
WE18_Performance_Up.ppt
WE18_Performance_Up.ppt
Handout3o
Handout3o
Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon
Improving Hadoop Resiliency and Operational Efficiency with EMC Isilon
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010
Hadoop and Voldemort @ LinkedIn
Hadoop and Voldemort @ LinkedIn
Pilot Hadoop Towards 2500 Nodes and Cluster Redundancy
Pilot Hadoop Towards 2500 Nodes and Cluster Redundancy
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...
Big data at United Airlines
Big data at United Airlines
Eagle from eBay at China Hadoop Summit 2015
Eagle from eBay at China Hadoop Summit 2015
Inroduction to Big Data
Inroduction to Big Data
Hug syncsort etl hadoop big data
Hug syncsort etl hadoop big data
Syncsort et le retour d'expérience ComScore
Syncsort et le retour d'expérience ComScore
Composing re-useable ETL on Hadoop
Composing re-useable ETL on Hadoop
Hands on Hadoop and pig
Hands on Hadoop and pig
RAPIDS: GPU-Accelerated ETL and Feature Engineering
RAPIDS: GPU-Accelerated ETL and Feature Engineering
Arun Rathinasabapathy, Senior Software Engineer, LexisNexis at MLconf ATL 2016
Arun Rathinasabapathy, Senior Software Engineer, LexisNexis at MLconf ATL 2016
LLAP: Sub-Second Analytical Queries in Hive
LLAP: Sub-Second Analytical Queries in Hive
Globo.com & Varnish
Globo.com & Varnish
Recently uploaded
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
Gabriella Davis
Understanding the Laravel MVC Architecture
Understanding the Laravel MVC Architecture
Pixlogix Infotech
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
Delhi Call girls
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Alan Dix
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
Puma Security, LLC
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)
Allon Mureinik
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
HampshireHUG
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
Pooja Nehwal
🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘
RTylerCroy
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
ThousandEyes
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
giselly40
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
Maria Levchenko
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
Rafal Los
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
BookNet Canada
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
2toLead Limited
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
Radu Cotescu
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
Principled Technologies
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Drew Madelung
Recently uploaded
(20)
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
Understanding the Laravel MVC Architecture
Understanding the Laravel MVC Architecture
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Kiwipycon2011 async-with-gevent-redis
1.
Writing a
distributed crawler system using gevent and redis Alex Dong @alexdong
2.
3.
Async 101
4.
Gevent: the monkey
king
5.
Redis: data structure
server
6.
Lessons learned
7.
8.
How many links
today?
9.
What was the
project Google employee #1 working on?
10.
11.
DNS resolution
12.
Request HTTP Header
13.
Download full content
14.
Store to local
file store, database and index
15.
Scheduling, throttling, status
monitoring, scale up by flicking on more servers.
16.
17.
Yield on IO/socket,
but
18.
19.
Context switch overhead
20.
21.
Cooperative multitasking
22.
epollfd =
epoll_create ();
23.
epoll_ctl ( epollfd
, EPOLL_CTL_ADD, listen_sock , &ev)
24.
epoll_wait ( epollfd
, events , MAX_EVENTS, -1)
25.
26.
27.
28.
Working: Hashtable -
HSET, HDEL, HEXISTS
29.
One and only
one: Sets - SADD, SPOP
30.
31.
32.
25% code for
status update and monitoring
33.
34.
DNS is down?
35.
ISP's bandwidth?
36.
Large file download?
37.
Scheduler re-submit tasks?
38.
39.
40.
Carefully profile to
make sure socket.recv becomes the main bottleneck.
41.
The get_title
crisis
42.
Rule of thumb:
load average < 1 to saturate 10M bandwidth
43.
Twitter: @alexdong
trunk.ly/?q= from:alexdong + gevent
Download now