Introduction to Datastore by Assoc.Prof. Dr.Thanachart Numnonda Asst.Prof. Thanisa Kruawaisayawan Mini Master of Java Technology KMITL July 2012

1. Introducion to Datastore
Assoc.Prof. Dr.Thanachart Numnonda
Asst.Prof. Thanisa Kruawaisayawan
Mini Master of Java Technology
KMITL
July 2010

2. Agenda
What is DataStore?
Using DataStore
JPA in DataStore

3. What is DataStore?

4. What is Datastore?
Google App Engine Datastore is a schema-less persistence
system, whose fundamental persistence unit is called Entity, c
omposed by an immutable Key and a collection of mutable pr
operties.
Entities can be created, updated, deleted, loaded by key and
queried for properties values.
DataStore is consistent and transactional, with support to
current transaction.

5. The DataStore
The Datastore is not a relational database nor a
façade.
Relational database technology doesn’t scale
horizontally
– Connection pools, shared caching are a problem
The Datastore is one of many public APIs used for
accessing Google’s

6. The DataStore

7. The DataStore

8. The DataStore : Operations
Transactions and Index are based on MegaTable.
File persistence it's done with Google File System
(GFS).
It's distributed by Chubby, a lock service for loosely-
coupled distributed systems.

9. BigTable
BigTable is a compressed, high performance, and
proprietary database system built on Google File
System (GFS), Chubby Lock Service, and a few other
Google programs
Currently not distributed or used outside of Google.
BigTable development began in 2004. and is now used
by a number of Google application Google Earth,
Google Map, Gmail, Youtube, etc..

10. BigTable : Design
BigTable is a fast and extremely large-scale DBMS.
It is a sparse, distributed multi-dimensional sorted map,
sharing characteristics of both row-oriented and column-
oriented databases.
sparse because only "not null" values are persisted
distributed in Google cloud
persistent on Google File System
multidimensional in columns values
ordered lexicographically by key

11. BigTable : Design
Tables are optimized for GFS by being split into
multiple tablets - segments of the table.
BigTable is designed to scale into the petabyte.
Each table has multiple dimensions (one of which is a
feld for time, allowing for versioning and garbage
collection).
It allows an infnite number of rows and columns.

12. Google File System
GFS is a proprietary distributed fle system developed
by Google.
It is designed to provide effcient, reliable access to
data using large clusters of commodity hardware.
GFS grew out of an earlier Google effort, BigFiles,
developed by Larry Page and Sergey Brin in the early
days of Google, while it was still located in Stanford.

13. Using DataStore

14. DataStore Operations
Datastore operations are defned around entities (data
models) which are objects with one or more properties
Types: string, user, Boolean, and so on
Entities may be recursive or self-referential
Entity relationships are one-to-many or many-to-many.
Entities may be fxed or grow as needed.

15. DataStore Storage Model
Every entity is of a particular kind
Entities in a kind need not have the same properties
One entity may have different “columns” from another in
the same kind!
Unique IDs are automatically assigned unless the user
defnes a key_name

16. Compare DataStore with Others

17. DataStore Storage Model
Basic unit of storage is an Entity consisting of
Kind (table)
Key (primary key)
Entity Group (partition)
0..N typed Properties (columns)

18. Datastore Quotas
Each call to Datastore counts towards the quota
The amount of data cannot exceed the billable
 Includes properties and keys but not the indices
CPU and Datastore CPU time quotas apply

19. Using the Datastore
Applications may access the Datastore using the JDO
or the JPA classes.
The JDO and JPA classes are abstracted using the
DataNucleus API
Open source
 Not very popular
 Support for Java standards
 Poor documentation

20. JPA in DataStore

21. Setting Up JPA
The JPA and datastore JARs must be in the app's
war/WEB-INF/lib/ directory.
A confguration fle named persistence.xml must be in
the app's war/WEB-INF/classes/META-INF/ directory,
A confguration fle tells JPA to use the App Engine
datastore.
The appengine-api.jar must also be in the war/WEB-
INF/lib/ directory.

22. persistence.xml: Example
<?xml version="1.0" encoding="UTF-8"?>
<?xml version="1.0" encoding="UTF-8"?>
<persistence version="1.0" xmlns="http://java.sun.com/xml/ns/persistence"
<persistence version="1.0" xmlns="http://java.sun.com/xml/ns/persistence"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/persistence
xsi:schemaLocation="http://java.sun.com/xml/ns/persistence
http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd">
http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd">
<persistence-unit name="thaijavaappPU" transaction-type="RESOURCE_LOCAL">
<persistence-unit name="thaijavaappPU" transaction-type="RESOURCE_LOCAL">
<provider>org.datanucleus.store.appengine.jpa.DatastorePersistenceProvider
<provider>org.datanucleus.store.appengine.jpa.DatastorePersistenceProvider
</provider>
</provider>
<non-jta-data-source/>
<non-jta-data-source/>
<properties>
<properties>
<property name="datanucleus.ConnectionURL" value="appengine"/>
<property name="datanucleus.ConnectionURL" value="appengine"/>
<property name="datanucleus.NontransactionalRead" value="true"/>
<property name="datanucleus.NontransactionalRead" value="true"/>
<property name="datanucleus.NontransactionalWrite" value="true"/>
<property name="datanucleus.NontransactionalWrite" value="true"/>
</properties>
</properties>
</persistence-unit>
</persistence-unit>
</persistence>
</persistence>

23. Getting an EntityManager Instance
An app interacts with JPA using an instance of the EntityManager.
import javax.persistence.EntityManagerFactory;
import javax.persistence.EntityManagerFactory;
import javax.persistence.Persistence;
import javax.persistence.Persistence;
public class EMF {{
public class EMF
private static final EntityManagerFactory emfInstance ==
private static final EntityManagerFactory emfInstance
Persistence.createEntityManagerFactory("transactions-optional");
Persistence.createEntityManagerFactory("transactions-optional");
public static EntityManagerFactory get() {{
public static EntityManagerFactory get()
return emfInstance;
return emfInstance;
}}
}}

24. Entity Class : Example
@Entity
@Entity
public class GuestList implements Serializable {{
public class GuestList implements Serializable
……
@Id
@Id
private String id;
private String id;
@Basic
@Basic
private User author;
private User author;
private String content;
private String content;
@Temporal(javax.persistence.TemporalType.DATE)
@Temporal(javax.persistence.TemporalType.DATE)
private Date visitDate;
private Date visitDate;
……
// Getter and Setter methods
// Getter and Setter methods
}}

25. Queries and Indices
A query operates on every entity of a given kind.
Specify zero or more sort orders
Specify zero or more flters on property values
Indices are defned in the App Engine confguration fles
Results are fetched directly from these indices; no indices are
created on the fly
WEB-INF/datastore-indexes.xml - non-standard fles
Normalization is not recommended
Optimization techniques for RDBMSs may result in poor
Datastore performance!

26. Query : Example
EntityManager em == EMF.get().createEntityManager();
EntityManager em EMF.get().createEntityManager();
try {{
try
Query query == em.createQuery("SELECT oo FROM GuestList AS o");
Query query em.createQuery("SELECT FROM GuestList AS o");
@SuppressWarnings("unchecked")
@SuppressWarnings("unchecked")
List<GuestList> results == (List<GuestList>) query.getResultList();
List<GuestList> results (List<GuestList>) query.getResultList();
for (Object obj :: results) {{
for (Object obj results)
GuestList guest == (GuestList) obj;
GuestList guest (GuestList) obj;
String nickname == guest.getAuthor().getNickname();
String nickname guest.getAuthor().getNickname();
out.println(nickname ++ "" "" ++ guest.getId());
out.println(nickname guest.getId());
}}
}} catch(Exception ex) {{
catch(Exception ex)
out.println(ex);
out.println(ex);
}}

27. Entity Relationships
Models association between entities.
There are four types of relationship multiplicities:
@OneToOne
@OneToMany
@ManyToOne
Supports unidirectional as well as bidirectional relationships
Unidirectional relationship: Entity A references B, but B doesn't
reference A.

28. Example : ManyToOne Mapping

29. Example : OneToMany Mapping

30. Transactions and Entity Groups
Transaction = Group of Datastore operations that either
succeed or fail
Entity groups are required because all grouped entities are
stored in the same Datastore node
An entity may be either created or modifed once per
transaction
Transactions may fail if a different user or process tries an
update in the same group at the same time
Users decide whether to retry or roll the transaction back

31. Transaction in JPA : Example
Book book == em.find(Book.class, "9780596156732");
Book book em.find(Book.class, "9780596156732");
BookReview bookReview == new BookReview();
BookReview bookReview new BookReview();
bookReview.rating == 5;
bookReview.rating 5;
book.getBookReviews().add(bookReview);
book.getBookReviews().add(bookReview);
Transaction txn == em.getTransaction();
Transaction txn em.getTransaction();
txn.begin();
txn.begin();
try {{
try
book == em.merge(book);
book em.merge(book);
txn.commit();
txn.commit();
}} finally {{
finally
if (txn.isActive()) {{
if (txn.isActive())
txn.rollback();
txn.rollback();
}}
}}

32. Unsupported Features of JPA
Owned many-to-many relationships, and unowned
relationships.
"Join" queries.
Aggregation queries (group by, having, sum, avg, max, min)
Polymorphic queries.

33. Resources
Google App Engine for Java HOWTO, Andrew Lombardi, Mar
2010
The Softer Side Of Schemas, Max Ross, May 2009
Official Google App Engine Tutorial,
http://code.google.com/appengine/docs/java/gettingstarted/
Programming Google App Engine, Don Sanderson, O'Reilly,
2010

34. Thank you
thananum@gmail.com
twitter.com/thanachart
www.facebook.com/thanachart
www.thaijavadev.com