Presentation delivered at the OpenStack summit Barcelona 2016.
https://www.openstack.org/videos/video/enabling-aws-s3-lambda-like-functionality-with-openstack-swift-and-openwhisk
Does the concept of server-less architecture intrigue you? OpenWhisk (https://git.io/vKeu3) accelerates innovation through creative chaining of microservices into highly scalable applications. By abstracting away infrastructure, OpenWhisk frees small teams to rapidly work on independent pieces of code simultaneously, keeping development focused solely on creating essential business logic. OpenWhisk allows you to create rules to connect events with actions and compose microservices that get executed independently and in parallel.
With a bit of code, you can have OpenWhisk process events from your Swift Object Storage; similar to what you can do with Lambda functions and AWS S3 storage. As an example, we will demonstrate how you can create an OpenWhisk action to transform an image into a thumbnail whenever a new (larger) image is uploaded into a Swift Container.
Why Teams call analytics are critical to your entire business
Open stack ocata summit enabling aws lambda-like functionality with openstack swift and openwhisk
1. Enabling AWS Lambda-like functionality
with OpenStack Swift and OpenWhisk
Shaun Murakami, Senior Cloud Architect, @stmuraka
Andrew Bodine, Software Engineer, @bodine_andrew
2. Development in the Cloud
Bare
metal
Virtual
machines
Containers
Functions
Decreasing concern (and control) over stack implementation
Increasingfocusonbusinesslogic
3. Serverless can handle many cloud native 12 Factors app
I Codebase Handled by developer (Manage versioning of functions on their own)
II Dependencies Handled by developer, facilitated by serverless platform (Runtimes and packages)
III Config Handled by platform (Environment variables or injected event parameters)
IV Backing services Handled by platform (Connection information injected as event parameters)
V Build, release, run Handled by platform (Deployed resources are immutable and internally versioned)
VI Processes Handled by platform (Single stateless containers often used)
VII Port binding Handled by platform (Actions or functions are automatically discovered)
VIII Concurrency Handled by platform (Process model is hidden and scales in response to demand)
IX Disposability Handled by platform (Lifecycle is hidden from the user, fast startup and elastic scale is prioritized)
X Dev/prod parity Handled by developer (The developer is the deployer. Scope of what differs is narrower)
XI Logs Handled by platform (Developer writes to console.log, platform handles log streaming)
XII Admin processes Handled by developer (No distinction between one off processes and long running)
4. Event driven programming
Execute app logic in response to database triggers
Execute app logic in response to sensor data
Execute app logic in response to cognitive trends
Execute app logic in response to scheduled tasks
Provide easy server-side backend for mobile app
5. Amazon Lambda
• An event-driven compute service where AWS Lambda runs
your code in response to events, such as changes to data or in
response to HTTP requests.
http://docs.aws.amazon.com/lambda/latest/dg/welcome.html
http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction.html
6. OpenWhisk
An open source cloud platform
that executes code
in response to events
https://github.com/openwhisk/openwhisk
7. OpenWhisk supports many growing workloads
OpenWhisk can help power
various mobile, web and IoT app
use cases by simplifying the
programming model of
orchestrating various services
using events without a dedicated
backend.
Digital app workloads Big Data/Analytics pipeline
Complex data pipelines for Big
Data/Analytics tasks can be scripted
using changes in data services or
streams for near real-time analytics
and feedback.
DevOps and infrastructure as code
OpenWhisk can be used to
automate DevOps pipelines
based on events triggered from
successful builds, or completed
staging, or a go-live event.
Microservices builder
OpenWhisk can be used to easily
build microservices given the
footprint and programming model
desired by microservices
9. riggers
A class of events that can occur
T
Social events
Data changes
Device readings Location updates
User input
10. ctions
Code that runs in response to an
event (i.e., an event handler)
A
Supports a variety of languages such as
JavaScript, Python, Java, and Swift ...
function main(params) {
return { message: 'Hello, ' +
params.name + ' from ' + params.place };
};
... even as a Docker container
Actions can be chained together
AA := A1 + A2 + A3
AB := A2 + A1 + A3
AC := A3 + A1 + A2
11. OpenWhisk can implement RESTful microservices
Send HTTP request
HTTP GET
app.com/customers
1
Invoke OpenWhisk
action get-customers
Browser
Mobile App
Web App
2
JS Swift Docker …
OpenWhisk
API
Proxy
12. OpenStack Swift Middleware
• Swift uses middleware to add custom behaviors.
• The majority of Swift middleware is applied to the
Proxy Server.
• Python WSGI Middleware (or just “middleware”)
can be used to “wrap” the request and response
of a Python WSGI application (i.e. a webapp, or
REST/HTTP API), like Swift’s WSGI servers
(proxy-server, account-server, container-server,
object-server)
http://docs.openstack.org/developer/swift/development_middleware.html
http://docs.openstack.org/admin-guide/objectstorage-arch.html
13. OpenStack Swift System Metadata
• Provides a means to store potentially private custom metadata
with associated Swift resources in a safe and secure fashion.
• May be set on accounts and containers by including headers
with a PUT or POST request.
• Maybe set on objects using only PUT requests.
• Takes the form of X-<type>-Sysmeta-<key>: <value>
• <type> depends on the resources type (i.e. Account, Container,
Object)
• <key> and <value> are set by trusted code running in a Swift WSGI
Server
http://docs.openstack.org/developer/swift/development_middleware.html#sysmeta
15. AWS S3 + Lambda
1. A user uploads an object to the source bucket in
Amazon S3 (object-created event).
2. Amazon S3 detects the object-created event.
3. Amazon S3 publishes the s3:ObjectCreated:* event
to AWS Lambda by invoking the Lambda function
and passing event data as a function parameter.
4. AWS Lambda executes the Lambda function by
assuming the execution role that you specified at
the time you created the Lambda function.
5. From the event data it receives, the Lambda
function knows the source bucket name and object
key name. The Lambda function reads the object
and creates a thumbnail using graphics libraries,
and saves it to the target bucket
http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html
16. OpenStack Swift + OpenWhisk
JS Swift Docker …
OpenWhisk
APIProxy
1. A user uploads an object to a OpenStack Swift
Container.
2. OpenStack Swift detects an object has been
modified in the Container.
3. The Swift Webhook middleware triggers an
OpenWhisk Action by invoking the OpenWhisk API
specified as the container’s webhook URL.
4. The OpenWhisk credentials are validated against
the OpenWhisk authorization service.
5. From the request payload, the OpenWhisk action
knows the source container and object name. The
OpenWhisk Action creates a thumbnail using
graphics libraries and saves it to a target container
Swift
Middleware
Source
Container
Target
Container
2
3
4
Whisk
Action
1
5
17. Let’s take a look at how it’s implemented
• OpenStack Swift Middleware Code
• OpenWhisk Action Code
18. OpenStack Swift Middleware (webhook.py)
class WebhookMiddleware(object):
@wsgify
def __call__(self, req):
req.headers[SYSMETA_WEBHOOK] = req.headers['x-webhook']
req.headers[SYSMETA_WEBHOOK_AUTH] = req.headers['x-webhook-auth']
resp = req.get_response(self.app)
if obj and is_success(resp.status_int) and req.method in [’PUT’,’DELETE’]:
webhook_req = urllib2.Request(container_info['sysmeta'].get('webhook'))
webhook_req.add_header('Authorization', base64.b64encode(container_info['sysmeta'].get('webhook-auth')))
webhook_req.add_data(json.dumps({
’url’: swiftUrl, ‘token’: req.headers[‘x-auth-token’], ‘container’: container, ‘object’: obj, ‘method’: req.method
}))
...
try:
webhook_resp = urllib2.urlopen(webhook_req).read()
….
return resp
19. Install Webhook Middleware
1. Place webhook.py in middleware of swift-proxy
…/swift/common/middleware/webhook.py
2. Update proxy-server.conf
…
[pipeline:main]
pipeline = catch_errors gatekeeper healthcheck ... webhook proxy-server
[filter:webhook]
paste.filter_factory = swift.common.middleware.webhook:webhook_factory
...
20. OpenWhisk Action Dockerfile
FROM ubuntu:14.04
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update
&& apt-get install -y curl
# Node.js
RUN curl -sL https://deb.nodesource.com/setup_4.x | bash -
&& apt-get install -y nodejs
WORKDIR /root/server
COPY server .
RUN npm install
EXPOSE 8080
CMD node ./app.js
23. Future work
Swift Asynchronous Event Notification Framework
• Support Multiple Notification Use Cases
• OpenWhisk / Serverless computing
• Cloud inter-service notifications
• Metadata search indexing
• Resource / traffic monitoring
• Common Notification APIs
• Enable / disable notifications on an account or container
• Allow for user / system level policies
• Desire to become a de-facto standard in Swift for notification management
• Deployment Flexibility
• Can be adapted to multiple message queue technologies
• Initial submission will include Kafka and Zaqar
• Patch WIP
• https://review.openstack.org/388393
24. Resources
• AWS Lambda
• Example: http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html
• OpenWhisk
• Code: https://github.com/openwhisk/openwhisk
• On IBM BlueMix: http://www.ibm.com/cloud-computing/bluemix/openwhisk/
• OpenStack Swift Middleware
• http://docs.openstack.org/developer/swift/development_middleware.htm
• Demo Code
• https://github.ibm.com/stmuraka/OpenStackSwift-OpenWhisk
