How Google AppEngine deals with digital art? how about music? a few case studies developed by Stinkdigital with Google Creative Lab and how App Engine dealt with a considerable amount of visits

1. PyCon6 Apr2015, Florence, Italy
Art & Music VS AppEngine
Thomas Alisi, Solution Architect
@grudelsud @stinkdigital
photo courtesy of https://www.flickr.com/photos/8064990@N08/

2. See our 2015 showreel
Stinkdigital is an interactive production company,
working with clients and advertising agencies
worldwide.
Our services include creative concepting, design
and high-end execution. We create everything from
live-action films and websites, through to mobile
apps and installations.

3. Does this number look familiar to you?
86,400

4. Does this number look familiar to you?
86,400
= 60’’ x 60’ x 24h[number of seconds in 1 day]

5. What about this one?
31,536,000

6. What about this one?
31,536,000
= 86,400’’ x 365d[number of seconds in 1 year]

7. OK, now try and guess the last one
252,000,000

8. OK, now try and guess the last one
252,000,000
= 31.5M’’ x 8y[number of seconds in 8 years]

9. but also…
252,000,000
= 36M x 7’’[7 seconds saved for each of the 36M visits
we had during the first 6 months on DevArt]

10. How did we do it?
1. Google App Engine (GAE) and AngularJS
2. data structures
3. assets management
4. load testing
5. GAE benchmarking tool and task inspector
6. GAE asynchronous API

11. Revolutions in Sound - RedBull, Google+
http://archive.revolutionsinsound.com/
DevArt - Google CL
https://devart.withgoogle.com/
Inside Abbey Road - Google CL
https://insideabbeyroad.withgoogle.com/

13. 1. Google App Engine (GAE) and AngularJS
2. data structures
3. assets management
4. load testing
5. GAE benchmarking tool and task inspector
6. GAE asynchronous API

14. - Grunt, Gulp, Browserify, Webpack… AAARGH!
- we like vanilla JS and have been fan of Gulp
(at least over the past 10 minutes…)
- we (I) tend to use a super simple Gulp-Browserify-
AngularJS boilerplate
https://github.com/grudelsud/angularjs-gulp-browserify-
boilerplate
- but there are other good examples too
e.g. https://github.com/unit9/coffee-bone
divide et impera

15. - GAE is not opinionated (which is good)
- default webapp2 shipped with GAE is really super simple
(a bit too simple…)
- Flask is OK and does not have preferences for a specific
ORM (which is great)
- don’t use Django on GAE unless you really want to
(e.g. use legacy modules or deploy something really
quick)

16. class Jsonifiable(ndb.Model):
def from_dict(cls, dict):
pass
def to_dict(self, async=False):
pass
def resolve_future_blobs(cls, async_blobs):
pass
class JsonRestHandler(webapp2.RequestHandler):
JSON_MIMETYPE = "application/json"
def write(self, data):
self.response.out.write(data)
def send_success(self, obj=None, cache_expiry=None):
self.response.headers["Content-Type"] = self.JSON_MIMETYPE
self.write(json.dumps(obj, cls=JsonifiableEncoder))
webapp2 bespoke REST micro-framework
asynchronous conversion
bespoke encoder
basic permission check (skipped here)
extend ndb.Model
extend webapp2.RequestHandler

17. kibble_authenticator = ModelAuthenticatior()
admin = kibble.Kibble(
'admin',
__name__,
kibble_authenticator,
label='Admin',
static_url_path='/kibble/static',
default_gcs_bucket=app_identity.get_default_gcs_bucket_name(),
)
security_headers.apply(app)
class CustomJsonEncoder(flask.json.JSONEncoder):
def default(self, obj):
try:
if isinstance(obj, blobstore.BlobKey):
return str(obj)
if isinstance(obj, fields.Vertex):
return [obj.x, obj.y, obj.z]
if isinstance(obj, GCSFile):
return obj.cloudstore_url
if isinstance(obj, ndb.Key):
return obj.flat()
except TypeError:
pass
return super(CustomJsonEncoder, self).default(obj)
app = flask.Flask(__name__, template_folder='../templates')
app.json_encoder = CustomJsonEncoder
admin.autodiscover(paths=['admin'])
app.register_blueprint(admin, url_prefix='/admin')
Flask Blueprints and GAE
https://github.com/xlevus/flask-kibble

18. 1. Google App Engine (GAE) and AngularJS
2. data structures
3. assets management
4. load testing
5. GAE benchmarking tool and task inspector
6. GAE asynchronous API

20. this always seems a good way to start

21. class ClubNight(BaseModel):
name = ndb.StringProperty()
content = ContentProperty(ndb.TextProperty)
blob_key_logo = ContentProperty(ndb.BlobKeyProperty)
genre = ndb.KeyProperty(kind=’Genre’)
website = ContentProperty(ndb.StringProperty)
address = ContentProperty(ndb.TextProperty)
location = ContentProperty(ndb.GeoPtProperty)
class Connection(ndb.Model):
from_key = ndb.KeyProperty()
to_key = ndb.KeyProperty()
abstraction

22. class ClubNight(BaseModel):
name = ndb.StringProperty()
content = ContentProperty(ndb.TextProperty)
blob_key_logo = ContentProperty(ndb.BlobKeyProperty)
genre = ndb.KeyProperty(kind=’Genre’)
website = ContentProperty(ndb.StringProperty)
address = ContentProperty(ndb.TextProperty)
location = ContentProperty(ndb.GeoPtProperty)
class Connection(ndb.Model):
from_key = ndb.KeyProperty()
to_key = ndb.KeyProperty()
to_name = ndb.StringProperty()
to_kind = ndb.StringProperty()
to_slug = ndb.StringProperty()
to_genre_colour = ndb.StringProperty()
to_image = ndb.BlobKeyProperty()
to_popularity = ndb.IntegerProperty()
is_published = ndb.BooleanProperty()
is_public = ndb.BooleanProperty(default=False)
is_featured = ndb.BooleanProperty(default=False)
reality

23. class Vertex(namedtuple('Vertex', ['x', 'y', 'z'])):
def __json__(self):
return json.dumps(self)
def __str__(self):
return 'x={0.x:.2f}, y={0.y:.2f}, z={0.z:.2f}'.format(self)
class VertexProperty(ndb.StringProperty):
# Dummy XYZ field, used to quickly add validation through
# admin.base.ModelConverter
def _validate(self, value):
if not isinstance(value, (tuple, Vertex)):
raise TypeError("Expected Vertex got %r" % repr(value))
if len(value) != 3:
raise TypeError("Expected 3-tuple/Vertex, got %r" % repr(value))
def _to_base_type(self, value):
return "%s,%s,%s" % value
def _from_base_type(self, value):
if value:
return Vertex(*[float(x) for x in value.split(",")])
return None
finding the sweet spot

25. 1. Google App Engine (GAE) and AngularJS
2. data structures
3. assets management
4. load testing
5. GAE benchmarking tool and task inspector
6. GAE asynchronous API

26. The get_serving_url() method allows you to generate a stable, dedicated URL for
serving web-suitable image thumbnails. You simply store a single copy of your
original image in Blobstore, and then request a high-performance per-image URL.
[https://cloud.google.com/appengine/docs/python/images/]
ex.
// Resize the image to 32 pixels (aspect-ratio preserved)
http://your_app_id.appspot.com/randomStringImageId=s32
// Crop the image to 32 pixels
http://your_app_id.appspot.com/randomStringImageId=s32-c
Due diligence
=s500 =s100

28. - success == sleepless_nights (not for me! :P)
- PR scheduled on all major funnels:
- Google homepage
- Youtube homepage
- all major blogs (the verge, wired, etc.)
- London silicon roundabout
- 1.2M unique visits, 10M views during the FIRST 48H
si vis pacem, para bellum

29. from admin.publish_pipeline import Publish
rec = urlmap.PublishRecord.new()
p = Publish('admin.web.app', version=rec.key.id())
p.start()
[…]
class Publish(Pipeline):
"""
Root Pipeline. Creates a :py:class:`common.models.urlmap.PublishRecord` for
the pipeline, and spawns children.
On completion, sets publish record to completed.
"""
def run(self, app, locales=None, version=None):
rec = urlmap.PublishRecord.get_by_id(version)
rec.pipeline_id = self.root_pipeline_id
rec.put()
urlmaps = yield URLMaps(version)
with After(urlmaps):
static = yield WriteStaticFiles.pipeline(app, version)
sitemap = yield PublishSitemap(app, version)
short_urls = yield CreateShortUrls.pipeline(app, version)
with After(static, sitemap, short_urls):
json = yield PublishJSON(app, version)
with After(json):
yield CleanShortUrls(app, version)
yield Cleanup.pipeline()
static publishing pipeline
https://github.com/GoogleCloudPlatform/appengine-mapreduce/
create the pipeline and run process
extend mapreduce pipeline
spawn separate generators

30. 1. Google App Engine (GAE) and AngularJS
2. data structures
3. assets management
4. load testing
5. GAE benchmarking tool and task inspector
6. GAE asynchronous API

31. GAE docs are rubbish! :) i.e. read it, then forget it:
https://cloud.google.com/appengine/articles/load_test
3rd party services are OK, but run your own if you can
create a meaningful simulation of users’ behaviour
hit it as hard as you can, but don’t forget your wallet!

32. class ApiNav(TaskSet):
@task(1)
def api_global(self):
self.client.get('/api/global?locale=%s' % langs[random.randint(0, len(langs)-1)], **kwargs)
@task(1)
def api_user(self):
self.client.get('/api/user', **kwargs)
@task(4)
def api_gallery(self):
self.client.get('/api/gallery?i=0&l=15', **kwargs)
@task(8)
def api_search(self):
self.client.get('/api/gallery?i=0&l=15&q=%s' % terms[random.randint(0, len(terms)-1)], **kwargs)
@task(6)
def api_feeling_lucky(self):
self.client.get('/api/page/feeling_lucky', **kwargs)
@task(2)
def api_big_gallery(self):
self.client.get('/api/gallery?i=0&l=30', **kwargs)
@task(2)
def api_featured(self):
self.client.get('/api/gallery?i=0&l=15&t=featured', **kwargs)
class MyLocust(HttpLocust):
host = 'https://sd-goog-devart.appspot.com'
task_set = ApiNav
min_wait = 5000
max_wait = 15000
locust
init data
gallery
random search
random project page
categorized views

33. - approximately 5000 concurrent user hitting the backend
API with a "casual navigation" simulation from different
location (London, New York, AWS data centre in Ireland)
- 85 running instances (class F2) at peak
- no errors reported other than random https sockets
timeout
- average response times
- < 2s for gallery content navigation
- < 1s for singe project page navitation
- < 3s for static contend (loaded just once)

34. 1. Google App Engine (GAE) and AngularJS
2. data structures
3. assets management
4. load testing
5. GAE benchmarking tool and task inspector
6. GAE asynchronous API

38. 1. Google App Engine (GAE) and AngularJS
2. data structures
3. assets management
4. load testing
5. GAE benchmarking tool and task inspector
6. GAE asynchronous API

39. @classmethod
def fix_dict(cls, dict, async=False):
async_blobs = []
def _fix_dict(k, v):
# blob
if isinstance(v, blobstore.BlobKey):
try:
# create futures and put them apart, we'll resolve these later
output = {'key': str(v), 'url': '', 'rpc': images.get_serving_url_async(v, secure_url=True)}
async_blobs.append(output)
return output
except BaseException as e:
# logging.warn('error while fetching serving url for [%s] maybe using corrupted image?' % (v,))
return None
for k, v in dict.iteritems():
dict[k] = _fix_dict(k, v)
if async is True:
return dict, async_blobs
else:
cls.resolve_future_blobs(async_blobs)
return dict
@classmethod
def resolve_future_blobs(cls, async_blobs):
# resolve futures
for blob in async_blobs:
try:
blob['url'] = blob['rpc'].get_result()
except BaseException:
blob['url'] = ''
del(blob['rpc'])
get_serving_url_async
1. get async url
3. get real url
2. resolve future blobs

40. Thanks!
uh, just few notes:
1. we are hiring! check www.stinkdigital.com/careers and send us your CV -
careers@stinkdigital.com
2. we organise a Meetup in London with UNIT9 and B-Reel, get in touch!
tomalisi@stinkdigital.com
3. www.slideshare.net/grudelsud