Building A Platform From Open Source At Yahoo

Building a platform from open source at Yahoo!

Dustin Whittle – Yahoo! Developer Network
Building a platform from open source at Yahoo! | Dus$n Whi*le

Overview

•  Why symfony?
•  symfony vs ysymfony
•  What does scaling really mean?
•  Social Search: Delicious and Answers
•  Yahoo! Open Strategy
–  What is the Yahoo! Open Stack?
–  Application Platform + Developer Tool
•  Developer Tools
–  YUI, Design Patterns, Tutorials
•  Data (YQL) & Social APIs
–  YQL, Geo, Proﬁles, Connections, Updates, …
–  YOS SDK for PHP
•  Building an open application with symfony and YOS
–  OAuth / YQL / OpenSocial


Who am I?

•  Working with symfony since open source
•  symfony Core Team Member
•  Responsible for the development and support of
symfony at Yahoo!
–  Worked with Y! Answers, Delicious, Y! Widgets, Y!
Bookmarks, Yahoo! Application Platform
•  Consultant
–  Commercial symfony support + training (USA)


EXAMPLES | TUTORIALS | CODE SAMPLES

DEVELOPER.YAHOO.COM


W3C | MICROFORMATS | OAUTH | OPENID | OPENSOCIAL

YAHOO! EMBRACES OPEN STANDARDS


RASMUS LERDORF | DOUG CROCKFORD | DOUG CUTTING | CHRISTIAN HEILMANN

YAHOO! HIRES OPEN SOURCE DEVELOPERS


YUI | BROWSER PLUS | DESIGN PATTERNS | R3 | YSLOW + PERFORMANCE RULES

YAHOO! GIVES BACK TO OPEN SOURCE


Conferences | Hack Days | HackU | Tech Talks | YDN Theater

YAHOO! ENGAGES COMMUNITIES WITH OPEN
HACK EVENTS AROUND THE WORLD


Users

Load Balancers

Frontend
Apache PHP ysymfony / YUI
Custom Modules APC, PEAR, PECL, Custom Extensions

Backend
MySQL/Oracle Web Services Ad API User API

FreeBSD 4.x/6.x, Linux 2.6.x


Y! needs from a frontend platfrom

•  Fit existing environment (RHEL/PHP5/Apache)
•  Development Cycle – How easy to develop, test, and
deploy?
•  Clean separation between data, logic, and display (MVC)
•  Independent model layer to ﬁt service oriented
architecture
•  Extensible and pluggable
•  Internationalization and localization support
•  Detailed documentation and active community of support
•  Open source and ability to contribute back

Why a frontend presentation platform?

•  Rasmus says “frameworks are not well suited for Y!”
–  Build applications to requirements
•  Do exactly what you need: no more, no less
•  Understand that frameworks add a lot of overhead
•  Choosing functional components is a better ﬁt
•  Despite choosing open source or building your own
–  Everyone uses a framework
–  If you use open source, use only the pieces you need


Why a framework at all?

•  Another software layer (ysymfony, yphp, yapache)
•  Factors out common patterns
–  Code Layout
–  Conﬁguration
–  URL Routing
–  Authentication / Security
–  Form Validation / Repopulation
–  Internationalization / Localization
•  Encourages good design
•  Abstraction > Consistency > Maintainability


The choice to adopt symfony?

•  Philosophy
–  Full-stack framework for building complex web applications
–  Adopt best ideas from anywhere, using existing code if
available (Mojavi, Prado, Rails, Django)
•  Design
–  Clean separation between Model, View, and Controller
–  Controller using modules and actions
–  Views using templates in straight PHP with helpers
–  Easy to reuse view modules to compose a page
•  Layouts, Components, Partials, Slots


The choice to use symfony

•  Conﬁgurability / Flexibility
•  Features we do not want are easily disabled
•  Use of factories for easy customization
•  Documentation / Support Community
•  The Deﬁnitive Guide to symfony (free online)
•  Excellent tutorials and example applications
–  Askeet & Jobeet
•  Active community with wiki, mailing lists, forums, irc
channel


Why ysymfony for Yahoo! teams?

•  Eliminate common patterns by adding a layer on PHP
–  Code layout/structure (MVC)
–  Conﬁguration
–  Internationalization
•  ysymfony is just a toolkit
–  Learn one set of tools
•  Shift between multiple projects
•  Consistency
–  Long term maintainability through platform


How Yahoo! uses symfony

•  Full Stack
–  Yahoo! Bookmarks
–  Yahoo! Answers
–  Delicious
–  Yahoo! Widgets
–  Yahoo! Application Platform Developer Tool
–  Corporate Reporting Tools
•  Components
–  Forms
–  Caching
–  Routing


symfony MVC framework


A look at Yahoo! Answers

•  http://answers.yahoo.com
•  Yahoo! Answers is the largest collection of human
knowledge on the Web with more than 135 million
users and 515 million answers worldwide (Yahoo!
Internal Data, March 2008).
•  Yahoo! Answers is the 2nd ranked education &
reference site on the web (comScore)
•  Available in 26 markets and 12 languages


Yahoo! Answers at the beginning

•  Started as a small development team on PHP4 from a
fork of Yahoo! Taiwan Knowledge+
•  Launched December 2005 by December 2006 there
were 60 million users and 65 million answers
•  The code base eventually became difﬁcult to maintain
and iterate new features
•  Large distributed development teams (US / UK)


The big picture

•  A complete platform for building web applications
from frameworks
–  PHP Framework
–  JavaScript Framework
–  CSS Framework
–  UI Design Patterns + Best Practices
–  Development Tools (logger, proﬁler, debugger, docs)
–  Unit + Functional Testing Frameworks (LIME / YUI Test)
–  Deployment Tools (rsync deployment system)


What does Yahoo! change?

•  Minor changes to ﬁt our environment (bsd/php/apache)
–  Most of our changes are easily implemented via factories
•  Added dimensions to conﬁgurations
(ysfDimensionsPlugin)
•  Integrate R3 translation/template management
(ysfR3Plugin)
–  R3 - http://developer.yahoo.com/r3/
•  Dropped the ORM and pushed down the stack (SOA)
–  Added a parallel API Dispatcher (ysfAPIClientPlugin)
•  Created a build and deployment solution (ysfBuildPlugin)
•  Integrate support for Y! User Interface libraries
(ysfYUIPlugin)


Propel or Doctrine or ???

•  No ORM for large projects
•  Propel or Doctrine for large projects
–  Doctrine for internal projects (best supported)
•  Service Oriented Architecture
–  Platforms as services (reusable to all)
–  No heavy lifting, push down the stack
–  Thin Controller/Fat Model (where model == services)
•  Java/C++/Erlang + JSON/XML


Localizing the experience with dimensions

•  Cascading Conﬁguration based on YAML
–  Framework -> Project -> Application -> Module
•  Extending the cascade to be based on dimensions
–  Dimensions can be anything (and can be chained together)
•  Data Center + Environment for customizing conﬁgurations
•  Culture for localizing user interface + data
•  Theme for customizing look and feel
•  User info (is user on corporate intranet?)
•  Caching


symfony I18n + L10n

•  Translation + Template Management
–  Integrated with symfony i18n + view layers
•  <r3:trans>hello</r3:trans> vs __(‘hello’)
•  Dictionary Management (YALA)
–  SQLite/XLIFF Backend
•  Added tasks
–  Importing/Exporting XLIFF dictionary
–  Generating translations for projects
–  Find missing strings


A build and deployment system

•  Aggregate and minify stylesheets and javascripts
•  Rewrite templates, css, js for CDN (Akamai, S3, …)
•  Generate translations for conﬁgurations + templates
•  Generate conﬁguration cache
•  Aggregate core classes + remove debug statements
•  Run lint, unit, functional tests
•  Package applications as .tgz
•  Deployment via rsync


What does it mean to scale?

•  A system whose performance improves after adding
hardware, proportionally to the capacity added, is said to
be a scalable system.
•  High Availability + Scalability + Performance
•  Bigger dataset, more trafﬁc, maintainable
•  Not about performance
–  PHP is slow, but it is not your bottleneck
•  Languages do not scale, architectures do.
•  Planning to grow and planning to fail
–  Capacity Planning
–  Business Continuity Planning


Scaling – Planning

•  Planning hardware purchases and hosting options to
have as much as you need without breaking your
wallet
•  Partitioning and distributing databases to support
large datasets and simultaneous transactions
•  Monitoring your applications to ﬁnd and clear
bottlenecks
•  Providing services APIs and using services from other
providers to increase your site's reach and capabilities
•  Think Minimal, Plan to grow, Plan to fail.


Scaling – The basics in PHP

•  PHP is rarely the bottleneck
•  “Most performance comes not from the language, but from
application design” - Rasmus
•  Share Nothing Architecture
–  Independent, self-sufﬁcient, no single point of contention
–  No local storage = No PHP Sessions
•  Use a database (works for distributed)
•  Use a small signed cookie (ideal)
–  Important data in database
–  Individual expiration on session objects
–  Small data items
–  Use a distributed cache
•  Memcache
•  Forget about small efﬁciencies -- Premature optimization is the
root of all evil.


Scaling Databases – The basics

•  Master/Slave Replication
–  First steps
–  Helps with reads, writes are still bottleneck
•  Partitioning
–  Segmenting data
•  Sharding (horizontal partitioning)
–  Segmenting data onto different physical machines
–  Make problems smaller, easier to grow


Improving latency with Caching

•  Always use PHP opcode cache (APC, Xcache, etc)
–  Use for routing and i18n cache
•  Memcache (distributed cache)
–  Use for view cache
•  Distributed invalidation can be a pain
•  sfViewCacheManager makes this easy!
•  Be intelligent about cache_keys (uri, user, state)
•  There is a ﬁne line to caching
–  At what point do you spend more time managing the cache,
than reading from it?


Tweaking Performance

•  Don’t use features you do not need
–  settings.yml / factories.yml
•  Use core_compile (aggregate classes)
•  Remove debug statements (sfOptimizerPlugin)
•  Do not use .htaccess (move to real apache conﬁg)
•  Set a minimal include path
•  Increase realpath_cache_size + realpath_cache_ttl
•  Use apc.stat=0
•  Use @routeName
•  Do not use components in loop


Do it yourself for cheap

•  Open source software = Free
–  Apache
–  PHP
–  MySQL
–  Memcache / Perlbal / MogileFS / Squid / Gearman
–  symfony / Doctrine / Propel / Swift
–  Nagios
•  Amazon Shared Infrastructure = Cheap
–  EC2 Cloud Computing
–  S3 Distributed Storage
–  SimpleDB


Yahoo! Open Strategy


OPENID | XRDS | OAUTH | PORTABLE CONTACTS | OPEN SOCIAL

OPEN PLATFORMS + COLLABORATION


The Open Web


Y! OS – The Open Stack


What does Yahoo! offer developers?

•  Yahoo! Developer Network
•  Developer Tools (YUI, etc)
•  Social APIs
•  Proﬁles
•  ConnecWons
•  Updates
•  Data APIs
•  OAuth
•  Yahoo! Query Language
•  Yahoo! ApplicaWon PlaYorm
•  OpenSocial


developer.yahoo.com


What is Yahoo! Developer Network?

The Yahoo! Developer Network offers open source tools and
open data APIs to make it easy for developers to build
applications and mashups.

•  50+ APIs / Web Services
–  Developer Dashboard
–  Tutorials + Code Samples
–  API Documentation

•  Yahoo! User Interface libraries + ASTRA
•  Design Patterns Library
•  PHP / Python / Ruby Dev Centers
•  Evangelism: Conferences / Theater / Blogs / Events


Y! Developer Network – YUI Javascript

•  JavaScript Framework
–  Utilities - YAHOO, Dom, Event, Animation, Browser History
Manager, Connection Manager, Cookie, DataSource, Drag and
Drop, Element, Get, ImageLoader, JSON, Resize, Selector,
Loader
–  Controls / Widgets - AutoComplete, Button, Calendar, Charts,
Color Picker, DataTable, ImageCropper, Rich Text Editor,
Slider, Uploader
–  Container (Module, Overlay, Panel, Tooltip, Dialog), Layout
Manager, Menu, TabView, TreeView
–  Debug – Logger, Proﬁler, Test


Y! Developer Network – YUI CSS

•  CSS Foundation
–  Reset - Neutralizes browser CSS styles
–  Base - Applies consistent style foundation
–  Fonts - Foundation for typography and font-sizing
–  Grids - Thousands of wireframe layouts
•  User Interface Design Patterns Library
–  Proven solutions to common interfaces
–  http://developer.yahoo.com/ypatterns/
–  Grade Browser Support / Progressive Enhancement


Y! Developer Network – Documentation

•  More than 275 functional examples
–  http://developer.yahoo.com/yui/examples/
•  YSlow + Performance Rules
–  http://developer.yahoo.com/performance
•  YUI Blog
–  http://yuiblog.com/
•  Mailing List @ Yahoo! Groups
–  http://tech.groups.yahoo.com/group/ydn-javascript/


A Query Language for the Web

SELECT * FROM INTERNET

Before YQL

•  Thousands of Web Services that provide valuable data
•  Require developers to read documentation and form
URLs/queries.
•  Data is isolated
•  Needs combining, tweaking, shaping even after it gets
to the developer.


Y! Open Stack – YQL

•  SQL-Like Language
–  Synonymous with Data access
–  Familiar to developers
–  Expressive enough to get the right data
•  Self Describing - show, desc table
•  Allows you to query, ﬁlter and join data across Web
Services.


YQL – Open Tables

•  Twitter •  Delicious
•  Weather •  Dopplr
•  Wesabe •  Friendfeed
•  Whitepages •  Github
•  Zillow •  New York Times
•  …. •  Shopping

Available on github ‐ h[p://github.com/spullara/yql‐tables/

YQL – Open Tables


YQL - Examples

•  select * from social.connections
•  select * from delicious.feeds.popular
•  select * from ﬂickr.photos.interestingness
•  select * from friendfeed.status
•  select * from github.checkins


YQL – Javascript Execute

•  Allows executing of javascript on the server side to
mashup (join, ﬁlter, etc)
•  Makes combining many web services very simple
•  Support for Oauth


YQL – API End Points

OAuth Endpoint
•  http://query.yahooapis.com/v1/yql?q=...

Public Endpoint
•  http://query.yahooapis.com/v1/public/yql?q=

YQL Console
•  http://developer.yahoo.com/yql/console


Y! Open Stack – Application Platform

•  Allows developers to deploy their own web based
applications on Yahoo!
•  Multiple Views: Small and Canvas
•  Social Context: the new Yahoo! Social Directory
•  OpenSocial 0.8 Javascript APIs


Y! Open Stack – Open Applications


Y! Open Stack – SDKs

PHP SDK Available
Open and OAuth Applica$ons

Ac$onScript 3 SDK Available
Open Applica$ons

Objec$veC SDK Available
Open and OAuth Applica$ons


YOSSDK – Methods

3‐Legged OAuth 2‐Legged OAuth
getSessionedUser (session) setSmallView   (applicaWon)
getOwner                       (session) query          (applicaWon)
getUser                          (session)
query                              (session)
getPresence                   (user)
setPresence                   (user)
listUpdates                     (user)
listConnecWonUpdates   (user)
insertUpdate                  (user)
deleteUpdate                 (user)
loadProﬁle                     (user)
getConnecWons             (user)
getContacts                   (user)
setSmallView                (user)


YOSSDK – 2-Legged OAuth

Used For:
‐  Public user data and open APIs


YOSSDK – 3-Legged OAuth

Used For:
‐  Private data access


Building an Open App : CommonGround

Experience: Basic Web Dev Knowledge – PHP/HTML/CSS/
Javascript

What we are building: Common ground – Find out what
you have in common with your social graph: music,
movies, books, hobbies.

What we will use: YOSSDK, YQL, YAP



h*p://developer.yahoo.com/dashboard

CommonGround available on GitHub

http://github.com/dwhittle/commonground


QUESTIONS?
(I will be around for the whole event, so please do say hello and ask questions)


WANT TO JOIN YAHOO?
WE ARE HIRING AND HAVE INTERNSHIPS!


ENJOY THE REST OF SYMFONYLIVE 2009

EXAMPLES | TUTORIALS | CODE SAMPLES
DEVELOPER.YAHOO.COM

Books worth reading


Building A Platform From Open Source At Yahoo

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (6)

Similar a Building A Platform From Open Source At Yahoo

Similar a Building A Platform From Open Source At Yahoo (20)

Más de elliando dias

Más de elliando dias (20)

Último

Último (20)

Building A Platform From Open Source At Yahoo