1. Cannibalising Google App Engine
Kevin Noonan
Calbane Ltd.
IT@Cork, 30th September, 2009

2. Cannibalising Google App Engine
Introduction

3. A working definition of Cloudware
• Outsourced Virtualization
o Virtual server running on outsourced server farm

4. The Cloud Computing Pyramid
•Michael Sheehan of GoGrid devised this classification

5. What's Cloudware good for? (I)
• Guerilla development
o Pay-per-use model is ideal for startups
o Also good for guerillas inside the enterprise...

6. What's Cloudware good for? (II)
• Scaling up
o Clone the virtual server
o pay for extra resources (bandwidth, storage, RAM, etc.)

7. What's Cloudware good for? (III)
• In general, outsourcing IT infrastructure
o pay someone else to run your servers, network, database...

8. Python: The Original of the Species

9. Java: the All New Flavour

10. Languages on App Engine's JVM
• Java
• JRuby
• Jython
• BeanShell
• JavaScript
• Scala
• Groovy
• …
• Will it Play in App Engine?
http://groups.google.com/group/google-appengine-java/web/will-it-play-in-app-engine

11. What is Google App Engine?
• Complete web stack
• Runs on Google's infrastructure
• Bigtable Datastore
o Like a sorted hashtable
o Not a relational database
o Use JDO or JPA (with Java) for datastore work
• Memcache
• Image API
• Mail API
• SDK for development on Windows/Mac/Linux
o Run & test locally before uploading to Cloud

12. Limitations of Google App Engine
• Applications run in “The Sandbox”:
http://code.google.com/appengine/docs/java/runtime.html#The_Sandbox
• Many Java libraries will not work (but many have been patched or will be).
See the JRE whitelist:
http://code.google.com/appengine/docs/java/jrewhitelist.html
No “long-lived” processing (requests must return in 30 seconds).
• HTTPS only on XYZ.appspot.com
o and only with Google's certificate.
o Due to limitation of SSL protocol, as applied to Google's Cloud.

13. Billing on Google App Engine
• True pay-per-use
o free quotas & no bill to pay if quotas not exceeded.
o quite different model to AWS
• Developer can assign budget per resource.
• Prices
o $0.10 per CPU core-hour
o $0.15 per GB-month of storage
o $0.12 per GB outgoing bandwidth
o $0.10 per GB incoming bandwidth
o $0.0001 per email recipient for emails sent by the application

14. Free Quotas on App Engine
•Daily outgoing bandwidth: 10 GB
•Daily incoming bandwidth 10 GB
•Daily requests 1.3 million
• Daily CPU-hours: 46
• More quotas (mail, datastore, images, memcache)
http://code.google.com/appengine/docs/quotas.html

15. Other limitations on App Engine
• 1 GB of free storage per app.
•100 MB Memcache storage limit
• App Engine apps can now receive requests and send responses of up to 10MB.
• Caveat: 1 MB limit on datastore/memcache API call
• You can deploy up to 1,000 code and 1,000 static files with an app.
Workaround: use ZipImport (with Python) to store files in a Zip-archive:
http://code.google.com/appengine/articles/django10_zipimport.html
• 150MB cumulative limit on an app's code and static files
(as opposed to storage in the datastore).

16. Cannibalising Google App Engine
JIQL for JDBC emulation

17. What is JIQL?
o
o Abstraction layer over the Google datastore, Bigtable
o
o Bigtable is non-relational, but JIQL simulates a relational database
o
o JIQL enables the use of JDBC and SQL with Google Bigtable

18. How to use JIQL
o Download Jiql.jar from http://jiql.googlecode.com/files/jiql.jar
o Place Jiql.jar under WEB-INFlib
o Create a connection as follows:
String url = "jdbc:jiql://local";
String password = "";
String user = "";
Properties props = new Properties();
props.put("user",user);
props.put("password",password);
Class clazz = Class.forName("org.jiql.jdbc.Driver");
Driver driver = (Driver) clazz.newInstance();
Connection Conn = driver.connect(url,props);

19. Supported SQL syntax on JIQL
o SUPPORTED SQL SYNTAX
o INSERT
o SELECT
o DROP
o CREATE
o CONSTRAINT
o DISTINCT
o FOREIGN KEY
o PRIMARY KEY
o ALTER
o AND
o OR
o JOINS
o STRING FUNCTIONS
o ORDER

20. Supported JDBC statements in JIQL
o * Connection
o * Statement
o * ResultSet
o * PreparedStatement
o * Driver
o * DatabaseMetaData
o * ResultSetMetaData

21. Castles in the Cloud
GaeVFS: a virtual filesystem on Bigtable

22. What is GaeVFS ?
o Filesystem abstraction over Google BigTable
(To enable writes to the “local filesystem” in Google's Cloud)
o
o Implemented as an Apache Commons VFS plugin
o
o Uses Memcache too under the hood
o
o The URI format is
gae://path
o
o Best to use relative paths

23. Using GaeVFS
import org.apache.commons.vfs.FileSystemManager;
import org.apache.commons.vfs.FileObject;
import com.newatlanta.commons.vfs.provider.gae.GaeVFS;
.
GaeVFS.setRootPath( getServletContext.getRealPath( "/" ) );
.
FileSystemManager fsManager = GaeVFS.getManager();
FileObject tmpDir = fsManager.resolveFile( "gae://WEB-INF/tmp" );
if ( !tmpDir.exists() ) {
tmpDir.createFolder();
}

24. Clear the Cache for GAEVFS
Clear the cache at the end of every servlet request!
public void doGet( HttpServletRequest req, HttpServletResponse res )
throws ServletException, IOException {
try {
// ...process request...
} finally {
GaeVFS.clearFilesCache();
}
}

25. Close the connection to GaeVFS
It's good practice to close the connection when your servlet is destroyed.
public void destroy() {
GaeVFS.close(); // this is not strictly required, but good practice
}

26. Castles in the Cloud
RESTlets
2/27/2009

27. What is the RESTlet project?
o An open source project to make RESTful web services easy to
implement in Java.
o
o There's a special build for App Engine, a snapshot tagged “unstable”
o

28. How to use RESTlets on App Engine
o Create a new project with the Eclipse plugin
o
o Add "org.restlet.gae.jar" to the war/WEB-INF/lib directoy
o
o Add "org.restlet.gae.jar" to the project build path
o
o
o

29. Create a simple RESTful resource
package simpleREST;
import org.restlet.resource.Get;
import org.restlet.resource.ServerResource;
//simple resource
public class simpleResource extends ServerResource {
@Get
public String represent() {
return "A word from the Cloud gods ";
}
}

30. Create a parent app for resource
package simpleREST;
import org.restlet.Application;
import org.restlet.Restlet;
import org.restlet.routing.Router;
public class simpleRESTApplication extends Application {
@Override
public synchronized Restlet createRoot() {
Router router = new Router(getContext());
router.attachDefault(simpleResource.class);
return router;
}
}

31. Castles in the Cloud
Scheduled tasks
2/27/2009

32. Scheduled tasks on App Engine
o Aka “Cron jobs”
o Vital for housekeeping and maintenance tasks, especially on the
database
o Useful for various polling exercises
o Significant in a system that is single-threaded and usually activated by
incoming HTTP request.
o
o “Web hook pattern”:
Cron jobs work by invoking a URL, just like an incoming web-request
o Maximum 20 cron jobs per app
o Max one per minute (use “webcron” service to overcome limit)

33. cron.xml in WEB-INF directory
<?xml version="1.0" encoding="UTF-8"?>
<cronentries>
<cron>
<url>/polling</url>
<description>Poll for some event every couple of minutes</description>
<schedule>every 2 minutes</schedule>
</cron>
<cron>
<url>/stats</url>
<description>Send out statistics every week</description>
<schedule>every monday 07:30</schedule>
<timezone>America/New_York</timezone>
</cron>
</cronentries>

34. English syntax in the cron schedule
every 5 minutes
every 12 hours
2nd,third mon,wed,thu of march 17:00
every monday 09:00
1st monday of sep,oct,nov 17:00
every day 00:00

35. Updating your cron jobs
o Update the whole “app” via Eclipse plugin or on the command-line:
appcfg update
Update just your cron-jobs
appcfg update-cron

36. Cannibalising Google App Engine
JRuby-on-Rails
Photo credit: http://www.flickr.com/photos/geoftheref/

37. JRuby
o One of the popular new dynamic languages on the JVM
o Highest performing Ruby implementation
o Excellent compatibility with “the standard Ruby” (MRI)
o Rails runs without code-changes
o Native Rails plugins will not run on Jruby
o Core developers lately moved from SUN to Engine Yard
o
o The next four slides are a recipe for JRuby-on-Rails...

38. Generate a Rails app
o Install the gems
$ sudo gem install google-appengine
o Assign an app identifier
Simply create an application at appspot.com
o Generate your rails app
$ rails my-app; cd my-app;

39. Configure Rack (middleware for Rails)
o Generate a rackup file: In the same folder, save this as config.ru.
require 'appengine-rack'
AppEngine::Rack.configure_app(
:application => 'my-app',
:version => 1 )
%w{R db/ doc/ log/ script/ test/ tmp/}.each do |x|
AppEngine::Rack.app.resource_files.exclude "/#{x}**"
end
ENV['RAILS_ENV'] = AppEngine::Rack.environment
require 'config/environment'
run ActionController::Dispatcher.new

40. Install gems (libraries) & generate model
o Check gem sources for datamapper
$ gem sources
o Only when missing, add it like this...
$ sudo gem source -a http://gems.datamapper.org
o Install gems into your app
$ appcfg.rb gem install rails dm-appengine rails_datamapper
o #Generate a model
$ ./script/generate dm_model book title:string summary:text image_url:string
available:boolean pages:integer -f --skip-timestamps --skip-migration

41. Run app locally & deploy to Cloud
o Run your app locally
$ dev_appserver.rb .
o Open up the console
$ appcfg.rb run -S irb
> require 'config/environment'
o Deploy to App Engine
$ appcfg.rb update .

42. Cannibalising Google App Engine
Task Queues for background processing
:Image courtesy the SeaWiFS Project, NASA/Goddard Space Flight Center & ORBIMAGE
Visible Earth Project: http://visibleearth.nasa.gov/view_rec.php?id=2600

43. Task Queues: Background processing
o Motivation: App Engine is single-threaded.
o Place small, discrete units of work on a queue.
o App Engine monitors work-queue & executes task in background.
o
o Configure queues in web.xml (in WEB-INF directory)
o Without any configuration, each app gets a queue named “default”
o Default rate: 5 tasks per second
o
o Tasks comprises: data payload and code to process it.
o Web-hook pattern
o Asynchronous, parallel processing in background

44. Add a task to a queue
import com.google.appengine.api.labs.taskqueue.Queue;
import com.google.appengine.api.labs.taskqueue.QueueFactory;
import static com.google.appengine.api.labs.taskqueue.TaskOptions.Builder.*;
// ...
Queue queue = QueueFactory.getDefaultQueue();
queue.add(url("/worker").param("key", key))
queue.add(url("/path/to/my/worker"));
queue.add(url("/path?a=b&c=d").method("GET"));

45. Cannibalising Google App Engine
XMPP: instant messages & peer-to-peer
Photo credit: NOAA Photo Library and NSSL: http://www.photolib.noaa.gov/nssl/nssl0113.htm

46. XMPP – Instant message protocol
o “Jabber” peer-to-peer protocol for instant messaging
o Standardised as “XML Message Passing Protocol“
o
o IM protocol in use by “Google Talk”
o Twitter uses XMPP under-the-hood
o
o An enabling technology of “The realtime Web”
o Makes peer-to-peer architectures possible with App Engine
o
o XMPP is useful for all kinds of chat / IM applications
o Also for interprocess communication.

47. Construct an instant message...
import com.google.appengine.api.xmpp.JID;
import com.google.appengine.api.xmpp.Message;
import com.google.appengine.api.xmpp.MessageBuilder;
import com.google.appengine.api.xmpp.SendResponse;
import com.google.appengine.api.xmpp.XMPPService;
import com.google.appengine.api.xmpp.XMPPServiceFactory;
// ...
JID jid = new JID("somebody@gmail.com");
String msgBody = "Hello from the Cloud!";
Message msg = new MessageBuilder()
.withRecipientJids(jid)
.withBody(msgBody)
.build();

48. Check on “presence” & send message
boolean messageSent = false;
XMPPService xmpp = XMPPServiceFactory.getXMPPService();
if (xmpp.getPresence(jid).isAvailable()) {
SendResponse status = xmpp.sendMessage(msg);
messageSent = (status.getStatusMap().get(jid) ==
SendResponse.Status.SUCCESS);
}
if (!messageSent) {
// Send an email message instead...
}

49. Configuration to receive messages
o Web hook pattern again
o Configuration: edit appengine-web.xml file; include these lines:
<inbound-services>
<service>xmpp_message</service>
</inbound-services>
o when App Engine receives a chat message for the app, it makes an
HTTP POST to:
/_ah/xmpp/message/chat/

50. Code to receive messages
o import java.io.IOException;
o import javax.servlet.http.*;
o import com.google.appengine.api.xmpp.JID;
o import com.google.appengine.api.xmpp.Message;
o import com.google.appengine.api.xmpp.XMPPService;
o import com.google.appengine.api.xmpp.XMPPServiceFactory;
o @SuppressWarnings("serial")
o public class XMPPReceiverServlet extends HttpServlet {
o public void doPost(HttpServletRequest req, HttpServletResponse res)
o throws IOException {
o XMPPService xmpp = XMPPServiceFactory.getXMPPService();
o Message message = xmpp.parseMessage(req);
o JID fromJid = message.getFromJid();
o String body = message.getBody(); // .......
o }
}

51. Map the URL to the servlet
To map the servlet to the XMPP URL path, add the following in the
web.xml file, inside the <web-app> element:
<servlet>
<servlet-name>xmppreceiver</servlet-name>
<servlet-class>XMPPReceiverServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>xmppreceiver</servlet-name>
<url-pattern>/_ah/xmpp/message/chat/</url-pattern>
</servlet-mapping>

53. Clojure
• Elegant
• Functional
• Dynamic
• Fast
• Excellent concurrency (unusable on App Engine, unfortunately)
• Direct access to Java
• Lisp Reloaded
(Feature-list above by Stuart Halloway.)

54. Running Clojure on App Engine
• How to run Servlets on App Engine with Clojure
http://fragmental.tw/2009/04/08/clojure-on-google-app-engine/
(“Getting Cloudy”, by Phillip Calçado of Thoughtworks)
•How to run Compojure (a web-framework for Clojure) on App Engine
http://elhumidor.blogspot.com/2009/04/clojure-on-google-appengine.html
(“Clojure on Google App Engine”, by John Hume of Thoughtworks)

55. Servlets with Clojure (sample-code)
Code from Phillip Calçado's BestMateServlet.clj
(ns BestMateServlet
(:gen-class
:extends javax.servlet.http.HttpServlet)
(:use mate))
(defn -doGet [_ req resp]
(process-request req resp))

56. Resources for Clojure
o Disclojure: http://disclojure.org/where-to-start/
– tutorial
– Book
– main Clojure site
– Plugins for IDEs
– Videos
•
VimClojure: http://kotka.de/projects/clojure/vimclojure.html

57. Cannibalising Google App Engine
Store Static Content on App Engine
Photo credit: Nikolai, http://www.blogger.com/profile/13624955858359692394

58. Getting started with App Engine
To use App Engine, you can begin with your current Gmail address and
password or with a new Google identity you create for the purpose.
Sign up for App Engine: http://appengine.google.com
(You'll need to receive an SMS on your mobile 'phone.)

59. Create a project on App Engine
Log into http://appengine.google.com and create a new app. (Many
project-names will be taken already, since App Engine shares its
namespace with Gmail.)
The project name you choose will determine the default domain-name.
For example, if "blissedbride" is available,
your app's default domain will be:
http://blissedbride.appspot.com

60. Store static web-assets in Google's Cloud
Install the plugin from e.g. (for Galileo)
http://dl.google.com/eclipse/plugin/3.5
Select the File menu > New > Web Application Project
Supposing you picked "blissedbride" as your project-name.
Navigate to this directory: workspace/blissedbride/war"
Dump all of your web-assets in this directory. Your index.html will
overwrite the placeholder file. And the rest of your HTML files can be
here too. You can put other web-assets in subdirectories.

61. Cannibalising Google App Engine
Wrapping up
Photo credit: NOAA Photo Library, NOAA Central Library; OAR/ERL/National Severe Storms Lab

62. Book (I)
Developing with Google App Engine
by Eugene Ciurana
Published:
Feb 2009
http://apress.com/

63. Book (II)
Google App Engine in Action
by Noah Gift & Michael Orr
To be published:
May 2009
http://www.manning.com/gift/

64. Book (III)
Google App Engine
by Dan Sanderson
To be published:
Sept 2009
http://oreilly.com/

65. Resources for Google App Engine
• Google App Engine: http://appengine.google.com
• Videos from conference Google held in 2009:
o http://sites.google.com/site/io/
• Chat with developers from the App Engine team
o first Wednesday of the month: 3am next day (Dublin/London time).
o third Wednesday of the month: 5pm (Dublin/London time).
o IRC channel: #appengine on irc.freenode.net
o http://en.wikipedia.org/wiki/List_of_IRC_clients
• Google Group
o http://groups.google.com/group/google-appengine/
• Paper on Bigtable:
o http://labs.google.com/papers/bigtable.html

66. Credits
• Photos used under Creative Commons license:
• Photo Credits
o http://www.flickr.com/photos/notsogoodphotography/
o http://www.flickr.com/photos/barto/
o http://www.flickr.com/photos/kitoy/
o http://www.flickr.com/photos/wolfgangstaudt/
o http://www.flickr.com/photos/motumboe/
o http://www.flickr.com/photos/ragnar1984/
o http://www.flickr.com/photos/janekm/
o http://www.flickr.com/photos/raylopez/
o http://www.flickr.com/photos/pagedooley/
o http://www.flickr.com/photos/gustty/
o http://www.flickr.com/photos/mike9alive/
o http://www.flickr.com/photos/tambako/
o http://www.flickr.com/photos/nate
o http://www.flickr.com/photos/freewine/

67. Contact Information
Kevin Noonan
kevin_noonan on Twitter
http://www.calbane.com