Organizations processing mission critical high-volume data must be able to achieve high levels of throughput and durability in data processing workflows. In this session, we will learn how DataXu is using Amazon Kinesis, Amazon S3, and Amazon EMR for its patented approach to programmatic marketing. Every second, the DataXu Marketing Cloud processes over 1 Million ad requests and makes more than 40 billion decisions to select and bid on ad impressions that are most likely to convert. In addition to addressing the scalability and availability of the platform, we will explore Amazon Kinesis producer and consumer applications that support high levels of scalability and durability in mission-critical record processing.

2. Amazon
Redshift
Amazon EMR
Amazon
EC2
Analyze
Amazon
Glacier
Amazon S3
Amazon
DynamoDB
Store
AWS Import/Export
AWS Direct Connect
Collect
Amazon Kinesis

3. Big data
•Hourly server logs: were your systems misbehaving 1hr ago
•Weekly / Monthly Bill: what you spent this billing cycle
•Daily customer-preferences report from your web site’s click stream: what deal or ad to try next time
•Daily fraud reports: was there fraud yesterdaywhat went wrong now: prevent overspendingnowwhat to offer the current customer nowblock fraudulent use now

4. HTTP Post
AWS SDK
LOG4J
Flume
Fluentd
Get* APIs
Kinesis Client
Library
+
Connector Library
Apache
Storm
Amazon Elastic
MapReduce
Sending Reading

5. Client/Sensor Aggregator Continuous
Processing
Storage Analytics +
Reporting

6. Client/Sensor Aggregator Continuous
Processing
Storage Analytics +
Reporting

7. Client/Sensor Aggregator Continuous
Processing
Storage Analytics +
Reporting

8. Client/Sensor Aggregator Continuous
Processing
Storage Analytics +
Reporting

9. Client/Sensor Aggregator Continuous
Processing
Storage Analytics +
Reporting

10. Client/Sensor Aggregator Continuous
Processing
Storage Analytics +
Reporting

11. Client/Sensor Aggregator Continuous
Processing
Storage Analytics +
Reporting

13. DataXu

15. DataXu Records
tx_id: "AFTfN0uAWZ"
exchange: “APPNEXUS"
request_id:"bb656107-3bf7-47a7-8548-8229563e9dc9”
….
adslot: {slot_id: "2686449714718898993”, uuid: "9d2403f1-fc6c-4d38-b6b1- 839fe4b42455”, price_micro_cpm: 661385, currency: "USD”, seat_id: "12-914”, campaign_id: "C0513n7”, creative_id: “R53a537”}
…
time_stamp: 1415393474434
serviced_by_host: "cr02.us-east-01”
Confirmation Record
[-69.120.26.172--[08/Nov/2014:21:59:54 -0500] "GET /rs?id=fc6f2106175a43df8ae4f3b7e6fa8c37&t=marketing&cbust=1415502000191662 HTTP/1.1" 302 -"http://ads- by.madadsmedia.com/tags/25628/10217/iframe/728x90.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0)" "wfivefivec=c876d00e- 1831-4eba-b78d-cd99188e951a" "OWW=-"
Fraud Record

16. Continuous
Processing
CDN
Real-time
Bidding
Retargeting
Platform
Reporting
Qubole
Real Time
KCL Apps Apps
Archiver
Amazon Kinesis Event Replay
Amazon S3
Producers Aggregator
Storage Analytics
Redshift

19. Client/Sensor Aggregator Continuous
Processing
Storage Analytics +
Reporting

20. Demo

22. https://github.com/awslabs/kinesis-log4j-appender

23. Client/Sensor Aggregator Continuous
Processing
Storage Analytics +
Reporting

24. Amazon Kinesis storage is replicated across
Availability Zones
Amazon Web Services
AZ AZ AZ
Durable, highly consistent storage replicates data
across three data centers (availability zones)
Aggregate and
archive to S3
Millions of
sources producing
100s of terabytes
per hour
Front
End
Authentication
Authorization
Ordered stream
of events supports
multiple readers
Real-time
dashboards
and alarms
Machine learning
algorithms or
sliding window
analytics
Aggregate analysis
in Hadoop or a
data warehouse
Inexpensive: $0.028 per million puts

26. 0
200000
400000
600000
800000
1000000
1200000
0
100
200
300
400
500
600
700
800
900
1000
1100
1KB Messages/sec
Shards

27. Demo

28. Client/Sensor Aggregator Continuous
Processing
Storage Analytics +
Reporting

30. Amazon Kinesis
23 21 18 17 14
Shard-i
10 8 5 3 2
Shard
ID
Lock Seq
num
Shard-i
Host A
Host B
Shard ID Last Archived
Shard-i
0
10
X18 2
3
5
8
10
14
17
18
21
23
0
130
Host BA
{Event 10, …}
1203
14
17
18
21
23

31. Demo

32. Client/Sensor Aggregator Continuous
Processing
Storage Analytics +
Reporting

33. CDN
Real Time
Bidding
Retargetin
g
Platform
Reporting
Qubole
Real Time
KCL Apps Apps
Archiver
Kinesis Event Replay
S3

34. Producers Aggregator
Continuous
Processing
Storage Analytics
CDN
Real-time
Bidding
Retargeting
Platform
Reporting
Qubole
Real Time
KCL Apps Apps
Archiver
Amazon Kinesis Event Replay
Amazon S3
Amazon
Redshift

35. Producers Aggregator
Continuous
Processing
Storage Analytics
CDN
Real-time
Bidding
Retargeting
Platform
Reporting
Qubole
Real Time
KCL Apps Apps
Archiver
Amazon Kinesis Event Replay
Amazon S3
Redshift

36. Client/Sensor Aggregator Continuous
Processing
Storage Analytics +
Reporting

37. •Unordered processing
–Randomize partition key to distribute events over many shards and use multiple workers
•Exact order processing
–Control the partition key to ensure events are grouped onto the same shard and read by the same worker.
•Need both? Get global sequence number
Producer
Get Global Sequence
UnorderedStream
Campaign Centric Stream
Fraud Inspection Stream
Get Event Metadata
Id
event
Stream –partition key
1
confirmation
Campaign-centric stream -UUID
2
fraud
Unordered StreamFraud-inspection stream –sessionid

38. HTTP
Post
AWS SDK
LOG4J
Flume
Fluentd
Get* APIs
Apache
Storm
Amazon
Elastic
MapReduce
Sending Reading
Amazon EMR
Playback
Amazon S3
Archiver

39. Client/Sensor Aggregator Continuous
Processing
Storage Analytics +
Reporting

40. Please give us your feedback on this session.
Complete session evaluations and earn re:Invent swag.
http://bit.ly/awsevals