My presentation from melbourne roro showing my new mess around library "smoke" with concepts stolen from yahoo and yql. Re interpret the web.

1. smoke
Friday, 1 May 2009
Hi!

2. smoke
Ben Schwarz
@benschwarz
http://github.com/benschwarz
Friday, 1 May 2009
Hi!

3. Lunch time
Friday, 1 May 2009
Starting with a story... It was lunch time.

4. 2pm
Friday, 1 May 2009
It was well and truly past eating time

5. 1 Hour
Friday, 1 May 2009
I had one hour. What do you do?

6. Code lunch
Friday, 1 May 2009
No need for eating. What will I do?

7. Sinatra
http://sinatrarb.com
Friday, 1 May 2009
We use Sinatra!

8. Yahoo Pipes
http://pipes.yahoo.com
Friday, 1 May 2009
Annd data from other peoples websites. No RDBMS for me!

9. Yahoo Pipes
• Get data from multiple
web based sources
• Mash it up a bit
• Provide a single output
in either JSON or XML
Friday, 1 May 2009

10. YQL
Friday, 1 May 2009

11. YQL?
Friday, 1 May 2009
Yeah, its kinda SQL for the web. Most people don’t even really like writing SQL

12. @lachlanhardy
Friday, 1 May 2009
Lachlan convinced me to have a second look at YQL after slagging it o

13. Time to start
Friday, 1 May 2009

14. Friday, 1 May 2009
The code isn’t important, but basically it just talks to a yahoo pipe that I made up.

15. Friday, 1 May 2009
I made “munch”. It pulls from various recipe and cooking related web sites like taste.com.au
and friends. It reads RSS and scrapes sites. Pretty simple.

16. 2:46pm
Friday, 1 May 2009
46 six minutes later. I’d been eected.

17. “I need a new
web site”
- Me, probably avoiding real work
Friday, 1 May 2009
I wanted to blog about this, sadly my blog is cached files sitting on a slicehost account.
Mephisto isn’t even running any longer.

18. Friday, 1 May 2009
I quickly threw a grid together.

19. HTML5
Friday, 1 May 2009
I wanted to experiment.

20. Sinatra
Friday, 1 May 2009
a bit more sinatra.

21. Friday, 1 May 2009
I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it
once and be able to keep it updated.

22. Friday, 1 May 2009
I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it
once and be able to keep it updated.

23. Friday, 1 May 2009
I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it
once and be able to keep it updated.

24. Friday, 1 May 2009
I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it
once and be able to keep it updated.

25. Friday, 1 May 2009
I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it
once and be able to keep it updated.

26. Friday, 1 May 2009
I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it
once and be able to keep it updated.

27. ?
Friday, 1 May 2009
I knew that I didn’t want to write everything then link it in to my blog, I wanted to write it
once and be able to keep it updated.

28. Friday, 1 May 2009
of course, I could’ve just got a tumblr account. That didn’t let me use sinatra or mess around
with HTML5.

29. Friday, 1 May 2009
It was getting easier, I had constraints around the use of my data

30. Delicious
Friday, 1 May 2009
It was getting easier, I had constraints around the use of my data

31. All
Delicious
Friday, 1 May 2009
It was getting easier, I had constraints around the use of my data

32. All
Delicious
Twitter
Friday, 1 May 2009
It was getting easier, I had constraints around the use of my data

33. All
Delicious
When I wasn’t talking to @someone
Twitter
Friday, 1 May 2009
It was getting easier, I had constraints around the use of my data

34. All
Delicious
When I wasn’t talking to @someone
Twitter
Flickr
Friday, 1 May 2009
It was getting easier, I had constraints around the use of my data

35. All
Delicious
When I wasn’t talking to @someone
Twitter
Selected (special tag)
Flickr
Friday, 1 May 2009
It was getting easier, I had constraints around the use of my data

36. All
Delicious
When I wasn’t talking to @someone
Twitter
Selected (special tag)
Flickr
?
Friday, 1 May 2009
It was getting easier, I had constraints around the use of my data

37. All
Delicious
When I wasn’t talking to @someone
Twitter
Selected (special tag)
Flickr
?
?
Friday, 1 May 2009
It was getting easier, I had constraints around the use of my data

38. Friday, 1 May 2009
The yahoo pipe looked like this. A bit of a mental leap.

39. Friday, 1 May 2009
Some things stood out. like union, I could join multiple sources of data together and output
them as one

40. Friday, 1 May 2009
or sort them by the publication date

41. Friday, 1 May 2009
and filter out the tweets to people. Sure a regex would’ve been smarter, but this was 90%
good enough

42. Friday, 1 May 2009
I started thinking about my process, the data coming from all over the place.

43. “What if I could
write something
like pipes?”
Friday, 1 May 2009
I wanted to do something myself (again)

44. “What if I could
write a ruby block
based DSL?”
Friday, 1 May 2009
I’d never written one, totally a good time to jump right in head first.

45. “What if you got
plain old Ruby
objects?”
Friday, 1 May 2009
If the data returned could be ruby, irrespective of what format it started as... it would be
awesome

46. Or JSON
Friday, 1 May 2009

47. Friday, 1 May 2009
Enter, “Smoke.”

48. Smoke
Friday, 1 May 2009
Enter, “Smoke.”

49. Friday, 1 May 2009
A search, which I named “ruby” uses yql and does a web based (search engine) search for the
keyword of “ruby”

50. Friday, 1 May 2009
A search, which I named “ruby” uses yql and does a web based (search engine) search for the
keyword of “ruby”

51. Friday, 1 May 2009
A search, which I named “ruby” uses yql and does a web based (search engine) search for the
keyword of “ruby”

52. Friday, 1 May 2009
A search, which I named “ruby” uses yql and does a web based (search engine) search for the
keyword of “ruby”

53. Friday, 1 May 2009
and hey, I’m gonna grab the RSS feed from slashdot.
“emit” will transform the objects that my feed returns.
I’ll rename any keys of link to url, this will make my data mind-map better

54. Friday, 1 May 2009
and hey, I’m gonna grab the RSS feed from slashdot.
“emit” will transform the objects that my feed returns.
I’ll rename any keys of link to url, this will make my data mind-map better

55. Friday, 1 May 2009
and hey, I’m gonna grab the RSS feed from slashdot.
“emit” will transform the objects that my feed returns.
I’ll rename any keys of link to url, this will make my data mind-map better

56. Friday, 1 May 2009
and hey, I’m gonna grab the RSS feed from slashdot.
“emit” will transform the objects that my feed returns.
I’ll rename any keys of link to url, this will make my data mind-map better

57. Friday, 1 May 2009
and hey, I’m gonna grab the RSS feed from slashdot.
“emit” will transform the objects that my feed returns.
I’ll rename any keys of link to url, this will make my data mind-map better

58. Friday, 1 May 2009
and hey, I’m gonna grab the RSS feed from slashdot.
“emit” will transform the objects that my feed returns.
I’ll rename any keys of link to url, this will make my data mind-map better

59. Friday, 1 May 2009
and hey, I’m gonna grab the RSS feed from slashdot.
“emit” will transform the objects that my feed returns.
I’ll rename any keys of link to url, this will make my data mind-map better

60. Friday, 1 May 2009
The whole point of this was to bring these multiple feeds together, re interpret them as one
common feed.

61. Friday, 1 May 2009
This creates a new “source”, it’ll be named “ruby_useless_joined”. It can be refiltered if
required

62. Creates
Friday, 1 May 2009
This creates a new “source”, it’ll be named “ruby_useless_joined”. It can be refiltered if
required

63. Creates
Friday, 1 May 2009
This creates a new “source”, it’ll be named “ruby_useless_joined”. It can be refiltered if
required

64. Friday, 1 May 2009
That can be extended and the items from the two sources can have some transformations as
well.

65. Friday, 1 May 2009
That can be extended and the items from the two sources can have some transformations as
well.

66. Friday, 1 May 2009
That can be extended and the items from the two sources can have some transformations as
well.

67. Friday, 1 May 2009
That can be extended and the items from the two sources can have some transformations as
well.

68. Friday, 1 May 2009
That can be extended and the items from the two sources can have some transformations as
well.

69. Friday, 1 May 2009
That can be extended and the items from the two sources can have some transformations as
well.

70. Friday, 1 May 2009
That can be extended and the items from the two sources can have some transformations as
well.

71. Friday, 1 May 2009
That can be extended and the items from the two sources can have some transformations as
well.

72. Friday, 1 May 2009
That accessor

73. Friday, 1 May 2009
Get the first item output (output is an array of the items from the source)

74. Friday, 1 May 2009
It gives us this

75. What just happened?
Friday, 1 May 2009
Thinking a little more about what actually just happened.
It parsed the data to a ruby object, irrespective of the format it came in as.

76. What just happened?
• Queried the web
Friday, 1 May 2009
Thinking a little more about what actually just happened.
It parsed the data to a ruby object, irrespective of the format it came in as.

77. What just happened?
• Queried the web
• Auto-magically parsed the data
Friday, 1 May 2009
Thinking a little more about what actually just happened.
It parsed the data to a ruby object, irrespective of the format it came in as.

78. What just happened?
• Queried the web
• Auto-magically parsed the data
• Returned a clean object
Friday, 1 May 2009
Thinking a little more about what actually just happened.
It parsed the data to a ruby object, irrespective of the format it came in as.

79. Friday, 1 May 2009
We can also output JSON

80. Friday, 1 May 2009
Its string escaped and not worth reading, but you could pass this through a web based app
straight down to javascript

81. Method
chaining
Friday, 1 May 2009
Methods can also be chained without invoking a new source or joining sources together

82. Friday, 1 May 2009
Here we can get a predefined source, shorten the result set and re sort it.

83. Friday, 1 May 2009
Here we can get a predefined source, shorten the result set and re sort it.

84. Friday, 1 May 2009
Here we can get a predefined source, shorten the result set and re sort it.

85. Transformations
Amount of items
Truncate
By a single key
Sort
One key to another
Rename
Items that match regex
Keep
Items that match regex
Discard
Ruby (default) or JSON
Output
Friday, 1 May 2009
Right now, you can do these things

86. YQL
Definitions
Friday, 1 May 2009
I’m looking at adding support for YQL definitions

87. Friday, 1 May 2009
They look like this, I’ve never written one, but it doesn’t look too hard. You can add search to
your own preexisting site / rss etc

88. http://github.com/spullara/yql-tables
Friday, 1 May 2009
There are a collection of community provided yql definitions hosted on github

89. All
these
fucking
web sites
Friday, 1 May 2009
Need I say more?

90. My message
Friday, 1 May 2009
Forgetting all the stu about pipes, yql and smoke, I want you to try something

91. “Re-interpret
the web”
Friday, 1 May 2009
Reconsider data, re-represent it to your own needs.

92. Thanks
Friday, 1 May 2009

93. Thanks
github.com/benschwarz/smoke
http://github.com/benschwarz/munch
http://sinatrarb.com
http://pipes.yahoo.com/pipes/pipe.edit?_id=fHK7LuUW3hGZHgSfbbsjiw
http://twitter.com/lachlanhardy
http://pipes.yahoo.com/pipes
http://developer.yahoo.com/yql
http://github.com/spullara/yql-tables
http://www.flickr.com/photos/95579828@N00/71240905
http://www.flickr.com/photos/hockadilly/2910737768
http://www.flickr.com/photos/maxblack/2655837607
Friday, 1 May 2009