1.
Google App Engine Development Java, Data Models, and Other Things You Should Know Navin Kumar Socialwok

2.
Introduction to Google App Engine <ul><ul><li>Google App Engine is an on-demand cloud platform that can be used to rapidly develop and scale web applications. </li></ul></ul><ul><li> </li></ul><ul><ul><li>Advantages: </li></ul></ul><ul><ul><ul><li>You are using the same architecture and tools that Google uses to scale their own applications. </li></ul></ul></ul><ul><ul><ul><li>Easy to develop your own applications using Java and Python </li></ul></ul></ul><ul><ul><ul><li>Free Quotas to get you started immediately. </li></ul></ul></ul>

3.
Java Support on Google App Engine <ul><ul><li>Java support was introduced on April 2009 </li></ul></ul><ul><li> </li></ul><ul><ul><li>Remarkable milestone for several reasons: </li></ul></ul><ul><ul><ul><li>Brought the Java Servlet development model to Google App Engine </li></ul></ul></ul><ul><ul><ul><li>You can use your favorite Java IDE to develop your applications now (Eclipse, NetBeans, IntelliJ) </li></ul></ul></ul><ul><ul><ul><li>Database development is easy with JDO and JPA </li></ul></ul></ul><ul><ul><ul><li>Not only limited to the Java Language, but ANY JVM-supported language can be used (JRuby, Groovy, Scala, even JavaScript(Rhino), PHP etc.) </li></ul></ul></ul>

4.
Eclipse Support and GWT <ul><ul><li>Eclipse is the premier open source Java IDE, and with the Google Plugin for Eclipse, developing Google AppEngine apps can be done very easily. </li></ul></ul><ul><li> </li></ul><ul><ul><li>Eclipse will automatically layout your web application for you in addition to providing 1-click deployment. </li></ul></ul><ul><li> </li></ul><ul><ul><li>GWT is also supported by the Eclipse plugin, and can also be used along with your Google AppEngine codebase. </li></ul></ul><ul><ul><ul><li>End-to-end Java development of powerful Java-based web applications. </li></ul></ul></ul>

5.
Google Plugin for Eclipse (GWT and AppEngine)

6.
BigTable: Behind Google's Datastore <ul><ul><li>BigTable: A Distributed Storage System for Structured Data ( http://labs.google.com/papers/bigtable.html ) </li></ul></ul><ul><ul><ul><li>Built on top of GFS (Google File System) ( http://labs.google.com/papers/mapreduce.html ) </li></ul></ul></ul><ul><li> </li></ul><ul><ul><li>Strongly consistent and uses optimistic concurrency control </li></ul></ul><ul><li> </li></ul><ul><ul><li> But it's not a relational database </li></ul></ul><ul><ul><ul><li>No Joins or true OR queries </li></ul></ul></ul><ul><ul><ul><li>&quot;!=&quot; is not implemented </li></ul></ul></ul><ul><ul><ul><li>Limitations on the use of &quot;<&quot; and &quot;>&quot; </li></ul></ul></ul>

7.
Data Models <ul><ul><li>DataNucleus ( http://www.datanucleus.org ) is used to handle the Java persistence frameworks on AppEngine </li></ul></ul><ul><li> </li></ul><ul><ul><li>2 Choices: JDO (Java Data Objects) or JPA (Java Persistence API) (JPA will be very familiar to those who have used Hibernate or EJB persistence frameworks) </li></ul></ul><ul><li> </li></ul><ul><ul><li>Both involve very similar coding styles. </li></ul></ul><ul><li> </li></ul><ul><ul><li>For this talk, we will focus on JDO, but JPA is very similar, so the same concepts can be applied. </li></ul></ul><ul><li> </li></ul><ul><ul><li>There is also a low-level datastore API that we will touch on as well </li></ul></ul>

8.
Defining Your Data Model <ul><li>package com.socialwok.server.data; </li></ul><ul><li>import java.io.Serializable; </li></ul><ul><li>import javax.jdo.annotations.*; </li></ul><ul><li>import com.google.appengine.api.datastore.Text; </li></ul><ul><li>@PersistenceCapable(identityType = IdentityType.APPLICATION) </li></ul><ul><li>public class Post implements Serializable { </li></ul><ul><li> private static final long serialVersionUID = 1L; </li></ul><ul><li> @PrimaryKey </li></ul><ul><li> @Persistent(valueStrategy=IdGeneratorStrategy.IDENTITY) </li></ul><ul><li> @Extension(vendorName=&quot;datanucleus&quot;, key=&quot;gae.encoded-pk&quot;, value=&quot;true&quot;) </li></ul><ul><li> private String id; </li></ul><ul><li> public String getId() { return id; } </li></ul><ul><li> @Persistent </li></ul><ul><li> private String title; </li></ul><ul><li> public String getTitle() { .. } </li></ul><ul><li> public void setTitle(String title) { .. } </li></ul><ul><li> @Persistent </li></ul><ul><li> private Text content; </li></ul><ul><li> public String getContent() { .. } </li></ul><ul><li> public void setContent(String content) { .. } </li></ul><ul><li> .. </li></ul><ul><li>} </li></ul>

9.
Creating, Deleting, and Querying <ul><ul><li>At the heart of everything is the PersistenceManager </li></ul></ul><ul><li> PersistenceManager pm = PMF.get().getPersistenceManager(); </li></ul><ul><li> Post post = new Post(); </li></ul><ul><li> post.setTitle(&quot;Title&quot;); </li></ul><ul><li> post.setContent(&quot;Google AppEngine for Java&quot;); </li></ul><ul><li> try { </li></ul><ul><li> pm.makePersistent(post); </li></ul><ul><li> } </li></ul><ul><li> pm.close(); </li></ul><ul><li> ... </li></ul><ul><li> Post deleteMe = pm.getObjectById(Post.class, deleteId); </li></ul><ul><li> try { pm.deletePersistent(deleteMe); </li></ul><ul><li> } </li></ul><ul><li> ... </li></ul><ul><li> </li></ul><ul><ul><li>Build queries using JDOQL </li></ul></ul><ul><li> Query query = pm.newQuery(Post.class); </li></ul><ul><li> query.setFilter(&quot;title == titleParam&quot;); </li></ul><ul><li> query.declareParameters(&quot;String titleParam&quot;); </li></ul><ul><li> query.setUnique(true); </li></ul><ul><li> Post post = (Post) query.execute(&quot;Title&quot;); </li></ul><ul><li> </li></ul><ul><li> </li></ul>

10.
Relationships <ul><ul><li>Owned one-to-one and one-to-many </li></ul></ul><ul><ul><li>@Persistent(mappedBy=&quot;field&quot;) annotation syntax. </li></ul></ul><ul><ul><li>Unowned relationships (one-to-one, one-to-many, many-to-many) </li></ul></ul><ul><ul><li>@Persistent Key otherEntity ; </li></ul></ul><ul><ul><li>@Persistent List<Key> otherEntities; </li></ul></ul><ul><ul><li>Owned relationships create a parent-child relationship </li></ul></ul><ul><ul><ul><li>Parent and child entities are stored in the same entity group </li></ul></ul></ul><ul><ul><ul><li>Entity group defines a location in the datastore </li></ul></ul></ul><ul><ul><ul><li>This is important because Transactions on the datastore can only be applied over a single entity group </li></ul></ul></ul>

11.
Other APIs you should be aware <ul><ul><li>UsersService </li></ul></ul><ul><ul><ul><li>Don't write a login, use Google's! </li></ul></ul></ul><ul><ul><li>ImagesService </li></ul></ul><ul><ul><ul><li>Picasa image manipulation web services </li></ul></ul></ul><ul><ul><li>Memcache </li></ul></ul><ul><ul><ul><li>Distributed cache for objects </li></ul></ul></ul><ul><ul><ul><li>Very useful! More on this later... </li></ul></ul></ul><ul><ul><li>URL Fetch </li></ul></ul><ul><ul><li>Mail service </li></ul></ul><ul><ul><ul><li>Send outbound emails w/ some restrictions </li></ul></ul></ul><ul><li> </li></ul><ul><ul><li>APIs (except UsersService) subject to quota limitations </li></ul></ul>

12.
And now for the fun stuff...

13.
<ul><ul><li>Enterprise social collaboration application built on Google App Engine. </li></ul></ul><ul><ul><ul><li>Utilizes a social concept of feeds (also referred to as presence and activity streams) </li></ul></ul></ul><ul><ul><ul><li>Combines the querying of reasonable complex data with privacy requirements of social networking. </li></ul></ul></ul><ul><li> </li></ul><ul><ul><li>Uses tons of Google App Engine APIs, Google APIs, and GWT. </li></ul></ul><ul><li> </li></ul><ul><ul><li> As we have built it, we have learned several aspects about Google App Engine that have allowed us to make the app reasonable fast and responsive. </li></ul></ul>

14.
Lesson 1: Utilization of Memcache <ul><ul><li>Data structure of each feed is relatively complex </li></ul></ul><ul><ul><ul><li>At least 3 explicit unowned relationships </li></ul></ul></ul><ul><li> @Persistent Key user </li></ul><ul><li> @Persistent Key network </li></ul><ul><li> @Persistent List<Key> attachments </li></ul><ul><li> </li></ul><ul><ul><ul><ul><li>Requires querying for each these objects explicitly when representing in the feed. </li></ul></ul></ul></ul><ul><ul><li>Feed is fetched repeated by several (hundreds) concurrent users </li></ul></ul><ul><ul><ul><li>There is need for the feed display to be reasonable responsive for all the different users </li></ul></ul></ul>

15.
Lesson 1 (cont.) Solution: Memcache <ul><ul><li>Distributed in-memory cache </li></ul></ul><ul><ul><ul><li>Uses javax.cache.* APIs </li></ul></ul></ul><ul><ul><ul><li>Also, a lowlevel API: com.google.appengine.api.memcache.* </li></ul></ul></ul><ul><ul><li>Basic uses: </li></ul></ul><ul><ul><ul><li>Speed up existing common datastore queries </li></ul></ul></ul><ul><ul><ul><li>Session data, user preferences </li></ul></ul></ul><ul><ul><li>Cache data is retained as long as possible if no expiration is set </li></ul></ul><ul><ul><li>Data is not stored on any persistent storage, so you must be sure your app can handle a &quot;cache miss&quot; </li></ul></ul>

16.
Lesson 1: Memcache conclusions <ul><ul><li> Works really well! </li></ul></ul><ul><ul><ul><li>Responsive requests </li></ul></ul></ul><ul><ul><ul><li>2 s. => ~800 ms. resp. time (60% decrease) </li></ul></ul></ul><ul><ul><li>Cache data is generally retained for a very long time </li></ul></ul><ul><ul><li>Distributed nature of cache provides benefits to every user on the system. </li></ul></ul><ul><ul><ul><li>The more people who use your app, the better your app performs** </li></ul></ul></ul><ul><ul><li>Even free quota for Memcache is quite generous: </li></ul></ul><ul><ul><ul><li>~ 8.6 million API calls. </li></ul></ul></ul>

17.
Lesson 2: Message Delivery Fanout <ul><ul><li>Adapted from Building Scalable, Complex Apps... from Google I/O by Brett Slatkin </li></ul></ul><ul><ul><ul><li>http://code.google.com/events/io/sessions/BuildingScalableComplexApps.html </li></ul></ul></ul><ul><ul><li> Basically deals with a problem of fan-out </li></ul></ul><ul><ul><ul><li>Socialwok has a concept of &quot;following&quot; (which is basically a subscription between users) </li></ul></ul></ul><ul><ul><ul><li>In our case, one user posts a single message that needs to be &quot;delivered&quot; to all his subscribers </li></ul></ul></ul><ul><ul><ul><li>How do we show the message efficiently to all his subscribers? </li></ul></ul></ul><ul><ul><ul><ul><li>We can deliver the message by reference to its recipients. </li></ul></ul></ul></ul>

18.
Lesson 2 (cont.): RDBMS version 2 Primary Tables 2 Join Tables <ul><ul><li>To get Messages to display for the current user </li></ul></ul><ul><ul><li>SELECT * from Messages INNER JOIN UserMessages USING (message_id) WHERE UserMessages.user_id = 'current_user_id' </li></ul></ul><ul><ul><li>But there aren't any joins on AppEngine! </li></ul></ul>User ID Name 1 Navin 2 John 3 Vikram Message ID Message User ID 1 Hello world 1 2 Another message 3 Follower ID Following ID 1 2 1 3 2 1 Recipient ID Message ID 1 34 1 67

19.
Lesson 2: List Properties to the Rescue <ul><ul><li>A list property is property in the datastore that has multiple values: </li></ul></ul><ul><ul><li>@Persistent private Collection<String> values; </li></ul></ul><ul><ul><ul><li>Represented in Java using Collection fields (Set, List, etc.) </li></ul></ul></ul><ul><ul><ul><li>Indexed in the same way that normal fields are </li></ul></ul></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><ul><ul><li>Densely pack information </li></ul></ul></ul><ul><li> </li></ul><ul><ul><ul><li>Query like you query any single-valued property: </li></ul></ul></ul><ul><ul><ul><ul><li>query.setFilter(&quot;values == 2&quot;); </li></ul></ul></ul></ul>values Index key=1,values=1 key=2,values=2 key=2,values=1

20.
Lesson 2: Our new data definition <ul><ul><li>Now we can define a collection field to store the list of recipients </li></ul></ul><ul><ul><li>public class Message { </li></ul></ul><ul><ul><li> @Persistent private String msg; </li></ul></ul><ul><ul><li> @Persistent private List<String> recipients; </li></ul></ul><ul><ul><li> ... </li></ul></ul><ul><ul><li>} </li></ul></ul><ul><ul><li> Query on the collection field: </li></ul></ul><ul><ul><li>Query query = pm.newQuery(Message.class); </li></ul></ul><ul><ul><li>query.setFilter(&quot;recipients == recptParam&quot;); </li></ul></ul><ul><ul><li>List<Message> msgs = </li></ul></ul><ul><ul><li> (List<Message>) query.execute(currentUserId); </li></ul></ul><ul><ul><li>But there is one issue with this: </li></ul></ul><ul><ul><ul><li>Serialization overhead when fetching the messages </li></ul></ul></ul><ul><ul><ul><li>We don't really care about the contents of this field when displaying the messages </li></ul></ul></ul><ul><ul><ul><li>So we will take advantage of another trick </li></ul></ul></ul>

21.
Lesson 3: Keys-only Queries and AppEngine Key Structure <ul><ul><li>We can perform queries whose return values are restricted to the keys of the entity </li></ul></ul><ul><ul><ul><li>Currently only supported in low-level datastore API </li></ul></ul></ul><ul><ul><li>AppEngine keys are structured in a very special way </li></ul></ul><ul><ul><ul><li> Stored in protocol buffers </li></ul></ul></ul><ul><ul><ul><li> Consists of an app ID, and series of type-id_or_name pairs </li></ul></ul></ul><ul><ul><ul><ul><li>pair is entity type name and autogenerated-integer ID or user-provided name </li></ul></ul></ul></ul><ul><ul><ul><li>Root entities have exactly one of these pairs; child entities have one for each parent and their own </li></ul></ul></ul><ul><ul><li>Presents a unique ability to retrieve a parent entity's key from the child entity's key </li></ul></ul>

22.
Lesson 3: A solution to our Serialization Problem <ul><ul><li>Now we can store the irrelevant recipients in a child entity </li></ul></ul><ul><li> </li></ul><ul><ul><li>Here's the process: </li></ul></ul><ul><ul><ul><li>Define a child entity with the recipients field </li></ul></ul></ul><ul><ul><ul><li>Store the recipients of the message in the child entity </li></ul></ul></ul><ul><ul><ul><li>Create a keys-only query on the child entity that filters on the recipients field. </li></ul></ul></ul><ul><ul><ul><li>Get a list of parent keys from the list of child keys </li></ul></ul></ul><ul><ul><ul><li>Bulk-fetch the parents from the datastore </li></ul></ul></ul>

23.
Lesson 3 (contd.): Solution (Data Def.) <ul><li>public class MessageRecipients { </li></ul><ul><li> @PrimaryKey private Key id; </li></ul><ul><li> @Persistent private List<String> recipients; </li></ul><ul><li> @Persistent private Date date; </li></ul><ul><li> @Persistent(mappedBy=&quot;msgRecpt&quot;) private Message msg; </li></ul><ul><li> ... </li></ul><ul><li>} </li></ul><ul><li>public class Message { </li></ul><ul><li> ... </li></ul><ul><li> @Persistent private Date date; </li></ul><ul><li> @Persistent private String msg; </li></ul><ul><li> @Persistent private MessageRecipients msgRecpt; </li></ul><ul><li> ... </li></ul><ul><li>} </li></ul>

24.
Lesson 3 (contd): Solution (Querying) <ul><li>DatastoreService dataSvc = ...; </li></ul><ul><li> </li></ul><ul><li>Query query = new Query(&quot;MessageRecipients&quot;) </li></ul><ul><li> .addFilter(&quot;recipients&quot;),FilterOperator.EQUAL,userid) </li></ul><ul><li> .addSort(&quot;date&quot;, SortDirection.DESCENDING) </li></ul><ul><li> .setKeysOnly(); // <-- Only fetch keys! </li></ul><ul><li> </li></ul><ul><li>List<Entity> msgRecpts = dataSvc.prepare(query).asList(); </li></ul><ul><li>List<Key> parents = new ArrayList<Key>(); </li></ul><ul><li>for (Entity recep : msgRecpts) { parents.add(recep.getParent()); </li></ul><ul><li>} </li></ul><ul><li>// Bulk fetch parents using key list </li></ul><ul><li>Map<Key,Entity> msgs = dataSvc.get(parents); </li></ul>

25.
Cool Trick: Lite Full Text Search <ul><ul><li>Most web applications nowadays need some form of full-text search </li></ul></ul><ul><ul><li>Well we are on Google AppEngine aren't we! </li></ul></ul><ul><li> </li></ul><ul><ul><li>Google actually did really release a basic searchable model implementation </li></ul></ul><ul><ul><ul><li>Limited to Python ( google.appengine.ext.search ) </li></ul></ul></ul><ul><ul><ul><li>More info: http://www.billkatz.com/2008/8/A-SearchableModel-for-App-Engine </li></ul></ul></ul><ul><ul><ul><li>Proper full-text search is in the AppEngine roadmap </li></ul></ul></ul><ul><li> </li></ul><ul><ul><li>Some of our earlier lessons do apply here. </li></ul></ul>

26.
How do we build it <ul><ul><li>First, it helps to understand how a basic full-text search index works </li></ul></ul><ul><ul><ul><li>First, break up the text into terms using lexographical analysis </li></ul></ul></ul><ul><ul><ul><li>Then store the terms in a lookup table based on key of the message </li></ul></ul></ul><ul><ul><ul><ul><li>With List fields, Google AppEngine gives us this one. </li></ul></ul></ul></ul><ul><ul><ul><li>We build queries using the same tricks. </li></ul></ul></ul><ul><ul><li>We also apply the same tricks using child entities and key-only queries to optimize for the serialization overhead. </li></ul></ul>

27.
Live example <ul><ul><li>I have deployed a modified version of Google AppEngine guestbook example: </li></ul></ul><ul><ul><ul><li>http://searchguestbook.appspot.com </li></ul></ul></ul><ul><li> </li></ul><ul><ul><li>If anyone wants to &quot;sign&quot; it right now, please go ahead. </li></ul></ul><ul><li> </li></ul><ul><ul><li>We will now search the data </li></ul></ul><ul><ul><ul><li>Limited to 1-2 word queries </li></ul></ul></ul>

28.
How it works. <ul><ul><li>Applies lessons from list fields and keys-only queries </li></ul></ul><ul><li> @Persistent Set<String> searchTerms; </li></ul><ul><ul><li>Our &quot;lexigraphical analysis&quot;: Java regular expression </li></ul></ul><ul><li> String[] tokens = content.toLowerCase().split(&quot;[^]+&quot;); </li></ul><ul><ul><ul><li>Can use a full-text search library like Lucene to improve this part </li></ul></ul></ul><ul><ul><li>Another cool feature of list properties: merge-join </li></ul></ul><ul><ul><ul><li>Think about organizing your data in a Venn-diagram fashion and finding the intersection of your data. </li></ul></ul></ul><ul><ul><ul><li>Watch your indexes! </li></ul></ul></ul><ul><ul><li>Can improve this implementation by using Memcache to cache common search queries. </li></ul></ul><ul><ul><li>Code will be made available after the talk, so you can take a good look for yourself! </li></ul></ul><ul><li> </li></ul>

29.
Conclusions <ul><ul><li>Google AppEngine for Java provides a standardized way to build applications for Google AppEngine </li></ul></ul><ul><ul><li>In building Socialwok, we have learned several lessons that apply when building a scalable application on Google App Engine </li></ul></ul><ul><ul><li>Get the Searchable Guestbook code here: </li></ul></ul><ul><ul><ul><li>http://searchguestbook.appspot.com/searchguestbook.tar.gz </li></ul></ul></ul><ul><ul><li>In short, Google AppEngine development has never been easier and more interesting! </li></ul></ul><ul><ul><li>Get started by visiting: http://code.google.com/appengine </li></ul></ul>

30.
Q & A