2. YQL
Paul Donnelly (@pjdonnelly)
YQL Team

3. When creating an application the
heart of the project is the data.

4. So how do we get data?

5. So how do we get data?
- API’s(flickr, twitter)
- Spreadsheets (yahoo finance csv, google docs)
- RSS (blogs, news)
- HTML pages

6. Many applications require more then
one source of data from different
providers.

7. This can get complicated very quickly.

8. This can get complicated very quickly.
For each web service I need to:
 Read API docs
 Figure out REST endpoints
 Figure out rate limits
 Figure out syntax

9. YQL TO THE RESCUE
YQL is an expressive SQL-like
language that lets you
query, filter, and join data across
Web services.
 SQL = familiar
 Not limited to just Yahoo web
services, but any web service.
 Single access point
 Learn the YQL syntax to filter
data instead of learning an
individual API.

10. YQL TO THE RESCUE
Using YQL, accessing the web and
it’s API’s becomes as simple as
SQL:
select {what} from
{service} where
{condition}

11. RSS AGGREGATION
select * from feed where url in
("http://feeds.feedburner.com/TechCru
nch/","http://allthingsd.com/feed/")

12. RSS AGGREGATION
select title,description from feed where
url in
("http://feeds.feedburner.com/TechCru
nch/","http://allthingsd.com/feed/")

13. RSS AGGREGATION
select title,description from feed where
url in
("http://feeds.feedburner.com/TechCru
nch/","http://allthingsd.com/feed/") |
sort(field="pubDate", descending="true
”)

14. SORT AND OTHER METHODS
Anything right of the | is performed after all select
operations are done.
Valid methods:
• sort
• tail (gets the last count items)
• truncate (gets the first count items)
• reverse
• unique (no duplicates)
• sanitize (emit safe html, prevents XSS)

15. HTML SCRAPING (SCRAY-P-I)
Pinterest doesn’t (currently) have an
API for it’s top 50 pins.

16. HTML SCRAPING (SCRAY-P-I)
select * from html where url
=
"http://pinterest.com/popular/
" and xpath =
"//a[@class='PinImage
ImgLink']"

17. HTML SCRAPING (SCRAY-P-I)

18. FLICKR
First get a Flickr API Key:
flickr.com/services/apps/create/apply

19. FLICKR
Query for pizza:
select * from flickr.photos.search
where text="pizza" and
api_key="5b7b21a3d44ed659e1341
0edee783a3b" limit 10

20. FLICKR
Query for pizza:

21. FLICKR
Query for pizza:
select source from flickr.photos.sizes
where photo_id in (select id from
flickr.photos.search where
text=”pizza" and api_key=“” and
sort="relevance") and api_key=“”
and label="Medium"

22. FLICKR
Query for pizza:

23. HOW DO I ACCESS YQL
Public Endpoint:
http://query.yahooapis.com/v1/public
/yql?q={query}&format={format}

24. HOW DO I ACCESS YQL
Formats
Current outputs are XML or JSON or
JSON-P

25. HOW DO I ACCESS YQL
YQL Console:
developer.yahoo.com/yql/conso
le/

26. YQL CONSOLE

27. CONSOLE PARAMETERS - DIAGNOSTICS

28. CONSOLE PARAMETERS - DEBUG

29. CONSOLE PARAMETERS – DEBUG CONTD.

30. CONSOLE TIPS – REST QUERY

31. CONSOLE TIPS – COMMUNITY TABLES

32. CONSOLE TIPS – PERMALINK & ALIAS

33. TABLE EDITOR
• Rapid YQL table prototyping
• Create YQL tables
• Create YQL Environment files
• Create YQL Javascript execute files
• Sample templates for all three
• Nifty drag & drop support
• All the above stored in the Yahoo! Cloud
• http://developer.yahoo.com/yql/editor

34. TABLE EDITOR – SAMPLE TABLE

35. YQL IN YAHOO PIPES
• http://pipes.yahoo.com
• Easy way to create RSS and create
complex data flows using a visual
editor.
• NEW!! YQL method
• y.pipe(“pipeid”,”<obj params>”)

36. YQL IN YAHOO PIPES

37. RECAP

38. YQL FORMATS
• Accepts JSON, XML, CSV
• Outputs XML, JSON, JSONP
• Provides data type transformers and mutators
• Tidies up data sources for you (html table)
• Lets you spend more time building your app

39. FAST, GLOBALLY AVAILABLE
• Deployed across the globe
• Both front and back facing caches
• Developers can control cache behavior
• _maxage=<seconds> parameter
• _stalewhilerevalidate=<seconds> parameter
• Rate limited for external developers
• 2000 calls per hour per IP on public endpoint
• 20000 on OAuth protected endpoint

40. APP DEVELOPMENT USING YQL
• Pick open tables or base YQL tables to build
your app.
• Build new ones for your API if needed
• Test the tables in the YQL console
• Use the YUI-YQL module to get data
• Or just call the REST query endpoint

41. TIPS
Search github.com/yql for table examples:
Use: http://developer.yahoo.com/yql/guide/
Use: http://christianheilmann.com/tag/yql/ for
awesome examples and inspiration

42. THANKS
Site: http://developer.yahoo.com/yql/
Community:
http://www.yqlblog.net/
http://www.datatables.org/
https://github.com/yql/yql-tables
Email: yql-questions@yahoo-inc.com
Twitter: @pjdonnelly