1. March 2014
Case Study: Using WSO2 API
Manager to deliver dynamic
display advertising
Chuck Megivern
Senior Software Engineer
MyWebGrocer, Inc.
cmegivern@mywebgrocer.com
May 2014

2. March 2014
Chuck Megivern is a Senior Software Engineer for MyWebGrocer, Inc.
He is the company technical lead for open source technologies. Chuck has been
developing advertising applications for MyWebGrocer and before that, Dealer.com,
for the last five years.
He has delivered Display Advertising, SEM, SEO and Digital Video applications and
APIs. He has also delivered multiple cloud based server deployments.
Earlier in his career, Chuck developed software for IBM and also had a number of
engineering assignments there.
Chuck has a BSEE degree from Lehigh University, and an MSEE degree from
Syracuse University.
About the author

3. March 2014
• MyWebGrocer (MWG) was founded in 1999.
• We serve the grocery industry and consumer packaged goods industry.
• We provide an online grocery platform used by leading retailers
encompassing over 10,000 retail locations in the US.
• Expanding in Asia Pacific and Europe.
• We also have extensive eCommerce and eMarketing solutions.
• We provide digital display advertising on our own ad network, as well as on
public internet sites that participate in other online ad networks, such as
Google DoubleClick.
• My area provides APIs for internal company use, and for external business
partners.
Introducing MyWebGrocer

4. March 2014
• Our case study involves using WSO2 API Manager to enrich display advertising served to
consumers via the Google DoubleClick ad network.
• The ad network delivers an Adobe Flash based Rich Media ad to consumer’s desktops while
they are browsing internet sites.
• The ad campaign lasts for a number of weeks. The Rich Media ad, using Action Script,
calls a MWG API and dynamically pulls in data every time the ad is displayed.
• The API delivers product information that updates daily. That data
• comes from the retailer’s store that is closest to the consumer, and
• is currently advertising the products of interest.
• Most consumer advertising application use cases involve responding to a consumers 'click'
expressing interest.
• However, this high traffic use case invokes the API every time that a display ad is
presented to a consumer.
• This presentation discusses our technical approach and application stack used to handle
this high volume.
Our case study

5. March 2014
WSO2 Product Usage
• Case study usage
• API Manager 1.5
• Business Activity Monitor (BAM) 2.4.0
• Current Usage
• API Manager 1.6
• Business Activity Monitor (BAM) 2.4.1
Types of APIs served
• Case study: very high volume consumer browser API invokations.
• At the same time, this distributed API manager installation serves:
• external business partners using MWG API’s
• internal MWG servers invoking MWG API’s
• Heterogeneous API consumers and API implementing servers
• Java
• Java related technologies
• Microsoft .NET
• WSO2 is helping us be a “connected enterprise”
MWG WSO2 Product Usage and the MWG API
environment

6. March 2014
WSO2 API Manager - Composition

7. March 2014
Three APIM Gateways. Two Primaries,
one is a backup without a BAM
connection. The F5 has an API health
check
nginx: preprocesses static content
Load balances application servers
(APIM 1.5)
API sync: rsync between gateways
(not shown)
MySQL DBs with
Circular replication (not shown)
F5 load balancer exposed to public
internet
APIM Store / Publisher not shown,
Not accessible from public internet
Our WSO2 Product Deployment

8. March 2014
• Consumer visits a website that is showing ads.
• Doubleclick network delivers the ad to the
consumer’s browser. There are multiple ads that
can be displayed. Each ad contains a set of
keywords relating to the item advertised, and the
retailer id.
• Rich media ad determines the consumer’s zip code.
• MWG API is called with the zip code, keywords,
retailer id.
• The API returned the nearest store and information
on a relevant item for sale. The consumer sees …
An ad is shown: step by step process

9. March 2014

10. March 2014
• Geolocation query for a particular retailer.
• bounding box by latitude, longitude
• Haversine function to sharpen results
• Returns a list of stores, ordered by increasing distance.
• Store circulars are queried
• keyword search
• Caching: database retrieval results
• geolocation query caching
• store circular data caching
• memcached used
Database loaded via background E-T-L processes
API functionality

11. March 2014
• API Manager requires an Authorization header.
• The Ad Unit, running Adobe ActionScript, will supply this header only with a
POST command.
• ActionScript also supports client security by
demanding a crossdomain.xml file.
GET host:port/crossdomain.xml (no context root)
• nginx is used to intercept all the http calls
• serves the static crossdomain.xml file
• passes all other requests to API Manager.
• also used: load balance application server endpoints (needed for APIM 1.5)
Initial Technical Challenges

12. March 2014
• Received only a total impression count from
marketing. No peak load estimate.
•API Manager / BAM integration vital.
• Essential: Used summarized BAM statistics from the
outset.
•Discovered with BAM: Workload was not random, or average. but very
“bursty”.
• Bottleneck was our serving infrastructure, not API
manager.
• Added a memcached infrastructure on the fly.
•essential to boost for application server response time.
Performance challenges

13. March 2014
• Uncached API service time is 150 – 200ms.
• Object caching brings to 22ms average service time.
• Cache hit ratio: 85 – 90%.
• Record (with fast average service time): 376
responses per second at 5ms average service time.
• Peak responses per second: 402 responses /second
(our throttle).
Performance highlights

14. March 2014
• A number of traffic bursts: 100 to 400 responses /
second.
• Traffic slows in the overnight.
• Traffic peaks in late afternoon, evening.
• Hourly, traffic slows 25 minutes after the hour. It
can increase 50% at 55 minutes after the hour.
• Assume: ad network serving hourly adjustments or TV commercials.
• Overall mean: 4.5 responses per second.
• 90th
percentile: 21 responses per second.
• Note: performance summarized by minute, based
on the BAM analytics toolbox for API manager.
Performance Highlights, cont’d.

15. March 2014
• This plot range:
up to 100/sec
Responses per second distribution

16. March 2014
• This plot range:
100-400 per sec.
• High end bursts
• Assume:
Ad network
• Few instances.
Responses per second

17. March 2014
Can spike if:
• Not enough processing
threads
• Cache entries expired.
• Many cache misses and
high traffic.
• No API Manager
limitation seen.
• Note: uncached service is
150ms – 200ms.
• With caching design
point, database is not
limiting.
Average Service Time

18. March 2014
• WSO2 BAM Cassandra storage
• Cassandra used as a ‘fast write’ repository for the streams of data
from the API manager gateways.
• BAM Hive scripts summarize on a per-minute basis, put summaries in
a relational db for reporting.
• Cassandra data does not seem to expire for us, so the Hive scripts
keep resummarizing.
• Purging old, already summarized Cassandra data.
• WSO2 Api Manager: Publisher
• Provides cool displays of usage right in the publisher from the
summarized statistics.
• Publisher had trouble handling the data once we had a lot of data.
We turned off the display inside Publisher.
Issues Encountered

19. March 2014
• Key Features
• API Design Wizard
• Support for OAuth2 Scopes
• Statistics Dashboard for developers
• Multi tenant workflow support
• Workflow integration for token generation
• URL templating support for API definition
• Customisable Store themes for tenants
• Forum Support - API Store
• Visibility Control for Documentation
• Statistics dashboard for southbound API calls
• Using Windows Authentication for OAuth2 Token generation
API Manager 1.7.0 is coming up...

20. March 2014
• The WSO2 API Manager platform is well suited to
handle high volume traffic. It was very capable
and adaptable.
• Essential: WSO2 Business Activity Monitor, using the
WSO2 provided API Manager BAM Analytics toolbox,
allowed us to
•Understand our traffic patterns
•Tune our server infrastructure
•Produce metrics for reporting
Conclusion

21. March 2014
• Thanks for listening.
• Thanks: Ad unit graphic development: Kurrent
Interactive: http://www.kurrent.com/
• Thanks: WSO2 for a flexible configurable product
set, based on open source standards.
Thanks!