SlideShare a Scribd company logo
1 of 19
Pylons + Tokyo Cabinet
    by Ben Cheng from Oursky.com
About Us
• Founder of Oursky Ltd.
• Web application development services
  (Ruby on Rails, Django, Pylons, PHP,
  Wordpress, etc.);
• FONCEPT.com - a T-Shirt design sharing
  portal;
• Working on a SaaS workflow application.
Pylons
What is Pylons?

• Flexible, Modular, Extensible
• Built on established libraries
• Close to 1.0
• Foundation of TurboGears 2.0
Who is using Pylons?

• Bittorrent
• Reddit (code.reddit.com!)
• and hundreds of sites
WSGI
• WSGI (Web Server Gateway Interface)
 • Wrap a “app” inside another “app”
 • Middleware, e.g.:
   • Authentication
   • Beaker (Cache, Session)
   • Routes
The Stack
• Models: SQLAlchemy, CouchDB...
• Templates: Mako, Genshi, Jinja2, ...
• Helpers: Web Helpers, FormAlchemy, ...
• Request Dispatching: Routes, ...
• Session / Caching: Beaker, ...
• Deployment: Paste, mod_wsgi, Twisted, ...
Take advantage of
      Python Power
• TIOBE: Python is no. 8
• Such as:
 • Want interactive debugger? iPython
 • Logging? Chainsaw
 • Testing? Nose, MiniMock
 • ..... blah blah balh
Power of other Python
  Web Framework
• Tons of lib from Zope
• ToscaWidgets from TurboGears
• Authkit
• (except Django)
Community
• Pylons Book
• Quick response (e.g.):
 • We fired Ticket #620 at Pylons Trac
   • “init_model called twice in nosetests”
   • Bug get fixed in 2 weeks, committed to
      trunk and temp. solution are discussed.
Tokyo Cabinet
Introduction

• By mixi.jp (Facebook in Japan)
• TokyoCabinet (Engine)
• TokyoTyrant (Server)
• TokyoDystopia (Full-Text Search)
• Opensources, standard C
TokyoCabinet
• Key/value DB
• Insert: 0.4 sec / 1M records
• Search: 0.33 sec / 1M records
• hash / B+ Tree structure
• Key Value / Hash / B+ Tree / Fixed Length /
  Table
• Driver: Perl, Ruby, Python, PHP, Erlang...
TokyoTyrant

• Network server
• Binary / Memcache / HTTP protocol
• 17.2 sec / 1M queries
• Hot Backup, Master-to-Master replication
• Extensible by Lua
Lua - Extensions
function incr(key, i)
  i = tonumber(i)
  local old = tonumber(_get(key))
  i = old + i
  _put(key, i)
  return i
end
Lua - Extension
function expire()
  local args={}
  local cdata = string.format(“%d”, _time())
  table.insert(args, “addcond0x0NUMLE”..cdata)
  table.insert(args, “out”)

 local res = _misc(“search”, args)
 if not res then
    _log(“expire failed”)
 end
end
Extensible to...

• Inverted index
• Set operations
• (and we use it to do revision management)
LightCloud
• Distributed and persistent key-value
  database
• Opensource and used by Plurk
• Scale out by just adding nodes
• Nodes are replicated
   Elapsed for 10000 gets: 1.74538516998 seconds [memcache]
   Elapsed for 10000 gets: 3.57339096069 seconds [lightcloud]

   Elapsed for 10000 sets: 1.88236999512 seconds [memcache]
   Elapsed for 10000 sets: 9.23674893379 seconds [lightcloud]
Links

• http://www.pylonshq.com
• http://opensource.plurk.com/LightCloud/
• http://1978th.net/tokyocabinet/

• Reach me at bencheng@oursky.com

More Related Content

What's hot

Centralized Logging with syslog
Centralized Logging with syslogCentralized Logging with syslog
Centralized Logging with syslog
amiable_indian
 
Socket programming with php
Socket programming with phpSocket programming with php
Socket programming with php
Elizabeth Smith
 
Kettunen, miaubiz fuzzing at scale and in style
Kettunen, miaubiz   fuzzing at scale and in styleKettunen, miaubiz   fuzzing at scale and in style
Kettunen, miaubiz fuzzing at scale and in style
DefconRussia
 
Dicas e truques de otimização de websites python
Dicas e truques de otimização de websites pythonDicas e truques de otimização de websites python
Dicas e truques de otimização de websites python
Fabiano Weimar
 

What's hot (20)

Building GUI App with Electron and Lisp
Building GUI App with Electron and LispBuilding GUI App with Electron and Lisp
Building GUI App with Electron and Lisp
 
Using Logstash, elasticsearch & kibana
Using Logstash, elasticsearch & kibanaUsing Logstash, elasticsearch & kibana
Using Logstash, elasticsearch & kibana
 
Application Logging With The ELK Stack
Application Logging With The ELK StackApplication Logging With The ELK Stack
Application Logging With The ELK Stack
 
Large Scale Log collection using LogStash & mongoDB
Large Scale Log collection using LogStash & mongoDB Large Scale Log collection using LogStash & mongoDB
Large Scale Log collection using LogStash & mongoDB
 
ELK stack at weibo.com
ELK stack at weibo.comELK stack at weibo.com
ELK stack at weibo.com
 
Logstash-Elasticsearch-Kibana
Logstash-Elasticsearch-KibanaLogstash-Elasticsearch-Kibana
Logstash-Elasticsearch-Kibana
 
From zero to hero - Easy log centralization with Logstash and Elasticsearch
From zero to hero - Easy log centralization with Logstash and ElasticsearchFrom zero to hero - Easy log centralization with Logstash and Elasticsearch
From zero to hero - Easy log centralization with Logstash and Elasticsearch
 
Dexador Rises
Dexador RisesDexador Rises
Dexador Rises
 
On Centralizing Logs
On Centralizing LogsOn Centralizing Logs
On Centralizing Logs
 
Solving some of the scalability problems at booking.com
Solving some of the scalability problems at booking.comSolving some of the scalability problems at booking.com
Solving some of the scalability problems at booking.com
 
Caching Up and Down the Stack
Caching Up and Down the StackCaching Up and Down the Stack
Caching Up and Down the Stack
 
Centralized Logging with syslog
Centralized Logging with syslogCentralized Logging with syslog
Centralized Logging with syslog
 
Socket programming with php
Socket programming with phpSocket programming with php
Socket programming with php
 
Like loggly using open source
Like loggly using open sourceLike loggly using open source
Like loggly using open source
 
{{more}} Kibana4
{{more}} Kibana4{{more}} Kibana4
{{more}} Kibana4
 
Kettunen, miaubiz fuzzing at scale and in style
Kettunen, miaubiz   fuzzing at scale and in styleKettunen, miaubiz   fuzzing at scale and in style
Kettunen, miaubiz fuzzing at scale and in style
 
XForms and eXist: A Perfect Couple
XForms and eXist: A Perfect CoupleXForms and eXist: A Perfect Couple
XForms and eXist: A Perfect Couple
 
Dicas e truques de otimização de websites python
Dicas e truques de otimização de websites pythonDicas e truques de otimização de websites python
Dicas e truques de otimização de websites python
 
Logstash family introduction
Logstash family introductionLogstash family introduction
Logstash family introduction
 
Web Development: The Next Five Years
Web Development: The Next Five YearsWeb Development: The Next Five Years
Web Development: The Next Five Years
 

Viewers also liked (7)

CU Geek Club 簡介
CU Geek Club 簡介CU Geek Club 簡介
CU Geek Club 簡介
 
Presentation at Hong Kong Start-Up Association Event
Presentation at Hong Kong Start-Up Association EventPresentation at Hong Kong Start-Up Association Event
Presentation at Hong Kong Start-Up Association Event
 
Slides for Teachers
Slides for TeachersSlides for Teachers
Slides for Teachers
 
Barcamp 2008 O Auth
Barcamp 2008 O AuthBarcamp 2008 O Auth
Barcamp 2008 O Auth
 
Hadoop New And Note - December 2010 TriHUG
Hadoop New And Note - December 2010 TriHUGHadoop New And Note - December 2010 TriHUG
Hadoop New And Note - December 2010 TriHUG
 
Fundraising & Crowdfunding - Crowdfunding “home made” - 5 passi per un sistem...
Fundraising & Crowdfunding - Crowdfunding “home made” - 5 passi per un sistem...Fundraising & Crowdfunding - Crowdfunding “home made” - 5 passi per un sistem...
Fundraising & Crowdfunding - Crowdfunding “home made” - 5 passi per un sistem...
 
The Outcome Economy
The Outcome EconomyThe Outcome Economy
The Outcome Economy
 

Similar to Pylons + Tokyo Cabinet

Ajax Tutorial
Ajax TutorialAjax Tutorial
Ajax Tutorial
oscon2007
 
Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...
Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...
Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...
adunne
 
Node js quick tour v2
Node js quick tour v2Node js quick tour v2
Node js quick tour v2
Wyatt Fang
 
Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015
Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015
Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015
curryon
 

Similar to Pylons + Tokyo Cabinet (20)

Ajax Tutorial
Ajax TutorialAjax Tutorial
Ajax Tutorial
 
Docker and-containers-for-development-and-deployment-scale12x
Docker and-containers-for-development-and-deployment-scale12xDocker and-containers-for-development-and-deployment-scale12x
Docker and-containers-for-development-and-deployment-scale12x
 
Node azure
Node azureNode azure
Node azure
 
Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek
 
Docker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic StackDocker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic Stack
 
Titanium Desktop Intro
Titanium Desktop IntroTitanium Desktop Intro
Titanium Desktop Intro
 
FOSDEM2018 Janus Lua plugin presentation
FOSDEM2018 Janus Lua plugin presentationFOSDEM2018 Janus Lua plugin presentation
FOSDEM2018 Janus Lua plugin presentation
 
Realtime Analytics with MongoDB Counters (mongonyc 2012)
Realtime Analytics with MongoDB Counters (mongonyc 2012)Realtime Analytics with MongoDB Counters (mongonyc 2012)
Realtime Analytics with MongoDB Counters (mongonyc 2012)
 
Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...
Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...
Living in a Multi-lingual World: Internationalization in Web and Desktop Appl...
 
Multiprocessing with python
Multiprocessing with pythonMultiprocessing with python
Multiprocessing with python
 
Troubleshooting .net core on linux
Troubleshooting .net core on linuxTroubleshooting .net core on linux
Troubleshooting .net core on linux
 
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
 
Living in a multiligual world: Internationalization for Web 2.0 Applications
Living in a multiligual world: Internationalization for Web 2.0 ApplicationsLiving in a multiligual world: Internationalization for Web 2.0 Applications
Living in a multiligual world: Internationalization for Web 2.0 Applications
 
Node js quick-tour_v2
Node js quick-tour_v2Node js quick-tour_v2
Node js quick-tour_v2
 
Node js quick-tour_v2
Node js quick-tour_v2Node js quick-tour_v2
Node js quick-tour_v2
 
Node js quick tour v2
Node js quick tour v2Node js quick tour v2
Node js quick tour v2
 
Scaling a Rails Application from the Bottom Up
Scaling a Rails Application from the Bottom Up Scaling a Rails Application from the Bottom Up
Scaling a Rails Application from the Bottom Up
 
Streaming huge databases using logical decoding
Streaming huge databases using logical decodingStreaming huge databases using logical decoding
Streaming huge databases using logical decoding
 
Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015
Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015
Bits of Advice for the VM Writer, by Cliff Click @ Curry On 2015
 
Rails Performance
Rails PerformanceRails Performance
Rails Performance
 

Recently uploaded

Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 

Recently uploaded (20)

Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 

Pylons + Tokyo Cabinet

  • 1. Pylons + Tokyo Cabinet by Ben Cheng from Oursky.com
  • 2. About Us • Founder of Oursky Ltd. • Web application development services (Ruby on Rails, Django, Pylons, PHP, Wordpress, etc.); • FONCEPT.com - a T-Shirt design sharing portal; • Working on a SaaS workflow application.
  • 4. What is Pylons? • Flexible, Modular, Extensible • Built on established libraries • Close to 1.0 • Foundation of TurboGears 2.0
  • 5. Who is using Pylons? • Bittorrent • Reddit (code.reddit.com!) • and hundreds of sites
  • 6. WSGI • WSGI (Web Server Gateway Interface) • Wrap a “app” inside another “app” • Middleware, e.g.: • Authentication • Beaker (Cache, Session) • Routes
  • 7. The Stack • Models: SQLAlchemy, CouchDB... • Templates: Mako, Genshi, Jinja2, ... • Helpers: Web Helpers, FormAlchemy, ... • Request Dispatching: Routes, ... • Session / Caching: Beaker, ... • Deployment: Paste, mod_wsgi, Twisted, ...
  • 8. Take advantage of Python Power • TIOBE: Python is no. 8 • Such as: • Want interactive debugger? iPython • Logging? Chainsaw • Testing? Nose, MiniMock • ..... blah blah balh
  • 9. Power of other Python Web Framework • Tons of lib from Zope • ToscaWidgets from TurboGears • Authkit • (except Django)
  • 10. Community • Pylons Book • Quick response (e.g.): • We fired Ticket #620 at Pylons Trac • “init_model called twice in nosetests” • Bug get fixed in 2 weeks, committed to trunk and temp. solution are discussed.
  • 12. Introduction • By mixi.jp (Facebook in Japan) • TokyoCabinet (Engine) • TokyoTyrant (Server) • TokyoDystopia (Full-Text Search) • Opensources, standard C
  • 13. TokyoCabinet • Key/value DB • Insert: 0.4 sec / 1M records • Search: 0.33 sec / 1M records • hash / B+ Tree structure • Key Value / Hash / B+ Tree / Fixed Length / Table • Driver: Perl, Ruby, Python, PHP, Erlang...
  • 14. TokyoTyrant • Network server • Binary / Memcache / HTTP protocol • 17.2 sec / 1M queries • Hot Backup, Master-to-Master replication • Extensible by Lua
  • 15. Lua - Extensions function incr(key, i) i = tonumber(i) local old = tonumber(_get(key)) i = old + i _put(key, i) return i end
  • 16. Lua - Extension function expire() local args={} local cdata = string.format(“%d”, _time()) table.insert(args, “addcond0x0NUMLE”..cdata) table.insert(args, “out”) local res = _misc(“search”, args) if not res then _log(“expire failed”) end end
  • 17. Extensible to... • Inverted index • Set operations • (and we use it to do revision management)
  • 18. LightCloud • Distributed and persistent key-value database • Opensource and used by Plurk • Scale out by just adding nodes • Nodes are replicated Elapsed for 10000 gets: 1.74538516998 seconds [memcache] Elapsed for 10000 gets: 3.57339096069 seconds [lightcloud] Elapsed for 10000 sets: 1.88236999512 seconds [memcache] Elapsed for 10000 sets: 9.23674893379 seconds [lightcloud]
  • 19. Links • http://www.pylonshq.com • http://opensource.plurk.com/LightCloud/ • http://1978th.net/tokyocabinet/ • Reach me at bencheng@oursky.com