Avro

Adding Avro to your Kafka
streams to meet your
messaging needs
https://github.com/skeletonkey/Avro
erik_tank

HELLO!
I am Erik Tank
Lead Engineer at Ticketmaster
You can find me at @erik_tank
erik_tank

Let’s Talk Schema
Schemata - Plural of schema
Schema
“representation of a plan or theory in the
form of an outline or model” (Oxford)
@erik_tank

Let’s Talk Schema
Pros
❑Contract
❏Clarity
❏Compatibility
❏Decoupling
@erik_tank

Let’s Talk Schema
Cons
❑No mechanism for evolution
❑No contract enforcement
Avro
Schem
a
Registry
@erik_tank

History of Avro
(Aircraft)
1956
Avro Vulcan B. 1
High-altitude strategic bomber
Operated by the Royal Air Force
(RAF) from 1956 until 1984
1963
Merged into Hawker
Siddeley Aviation
However, the Avro name has
been used for some aircraft
since then
1909
Roe I Triplane
First aircraft build by the Roes.
First flight on June 5th
1909
1910
Founded by Alliott &
Humphrey Verdon Roe
One of the world's first
aircraft builders.
@erik_tank

History of Avro (Aircraft)
2006
Hadoop
Search removed from Nutch and with
only the data processing/sharing parts
Hadoop is born.
2009
Apache Avro
A specification based design to
server the data exchange
needs to a world with multiple
*everything*.
2002
Nutch
Doug Cutting set out to
build an open source full
scale search engine
2003
Writable/SequenceFile
Improving the ability to
MapReduce and other
parallel data computations
@erik_tank

Avro
❑Language neutral serialization system
❑Rich data structure
❑Compact, fast, binary data format
❑Integration to dynamic languages
❑Code generation (Java)
❑RPC
❑Schema evolution
@erik_tank

Why Use
❑ Data compression
❑ Language support
❑ Schema evolution
❑ Flexibility
@erik_tank

Why Not
❑ Initial Lift
❑ Binary format
❑ Not turn key in every language
❑ Prototyping
❑ You’re afraid?
@erik_tank

Avro Workflow
Source: https://docs.confluent.io/current/schema-registry/index.html
@erik_tank

Avro Workflow
Source: https://docs.confluent.io/current/schema-registry/index.html (modified)
server client
@erik_tank

Short Kafka Detour
❑Topic contains messages
❑message is a key-value pair
❑subject – Schema Registry
topic_name-key
topic_name-value
@erik_tank

Picking a Topic Name
❑Avoid names that change over time
❑Settle on a template for your topics
❑ <message type>.<dataset name>.<data name>
❑Topic should reflect its purpose
Source: https://riccomini.name/how-paint-bike-shed-kafka-topic-naming-conventions
@erik_tank

Avro Compatibility
Source: https://docs.confluent.io/current/schema-registry/avro.html
@erik_tank

Best Practices (Do’s)
❑Default Value
❑Document!!!
❑Carefully consider topic name
❑Carefully consider changes to schema
❑Deal with Avro as a DB connection
@erik_tank

Best Practices (Dont’s)
❑Change the purpose of the topic
❑Change field types
❑Delete or Rename fields
@erik_tank

What’s in an extension?
❑.avsc – JSON schema file
❑.avdl – IDL file
❑.avpr – JSON protocol file
@erik_tank

.avsc{
"namespace": "com.jundy.client",
"name": "Rules",
"type": "record",
"fields": [
{
"name": ”channel",
"type": {
"type": "enum",
"name": "Channel",
"symbols": [ "INTERNET", "IVR", "PHONE" ],
"default": "INTERNET",
"doc": "Category that the client is allows to transact on"
}
},
{
"name": "Market",
"type": {
"type": "enum",
"name": "Market",
"symbols": [ "UNITEDSTATES", "AUSTRALIA", "CANADA", "GERMANY", "GREATBRITAIN", "IRELAND", "MEXICO", "NEWZEALAND", "NORTHIRELAND", "SPAIN" ],
"default": "UNITEDSTATES",
"doc": "Market which client is allowed to operate in"
}
},
{
"name": "feature_set",
"type": [ "null", "string" ],
"default": null,
"doc": "The human readable client name/id"
},
{
"name": "RuleSet",
"type": {
"name": "RuleSet",
"type": "record",
"fields": [
{
"name": "enable_insurance",
"type": "boolean",
"default": false,
"doc": "Flag indicating if client is allowed to offer insurance"
},
{
"name": "require_delivery_method_to_complete_order",
"type": "boolean",
"default": false,
"doc": "Flag indicating if client is required to provided a delivery method to complete an order"
},
{
"name": "shopping_cart_ttl",
"type": "int",
"default": 600,
"doc": "Max time that a cart is allowed to exists."
},
{
"name": "configurable_customer_info_obj_fields",
"type": ["null", "string"],
"default": null,
"doc": "What customer information fields are required for customer data to be considered 'compelte'"
}
]
}
}
]
}
Full Specs: https://avro.apache.org/docs/1.9.0/spec.html
{
"name": ”channel",
"type": {
"type": "enum",
"name": "Channel",
“default": "INTERNET",
"doc": "Category that client can transact on”
}
}
{
"namespace": "com.jundy.client",
"name": "Rules",
"type": "record",
"fields": [
{
{
"default": null,
"doc": "The human readable client name/id”
},
{
"name": "enable_insurance",
"type": "boolean",
"default": false,
"doc": "Flag indicating if …"
},
@erik_tank

.avsc vs .avdl
@namespace("com.jundy.client")
protocol Rules {
enum Channel { "INTERNET", "IVR", "PHONE" };
record Rules {
/**
Category that the client is allows to transact
*/
Channel channel = “INTERNET”;
/**
The human readable client name/id
*/
union { null, string } featureSet = null;
}
}
{"namespace": "com.jundy.client",
"name": "Rules",
"type": "record",
"fields": [{
"name": "channel",
"type": { "type": "enum",
"name": "Channel",
"doc": "Category that the client is allows to transact on"
}},{"name": "feature_set",
"default": null,
}]}
"name": "channel",
"type": { "type": "enum",
"name": "Channel",
"doc": "Category that the …"
enum Channel { "INTERNET", "IVR", "PHONE" };
record Rules {
/**
Category that the client is allows to transact
*/
Channel channel = “INTERNET”;
"default": null,
/**
The human readable client name/id
*/
union { null, string } featureSet = null;

Main Takeaways
❑Carefully consider topic name
❑Schema changes with caution
❑Use .avdl
❑Prototypes end up in production!!!!
@erik_tank

THANKS!Any questions?
You can find me at:
❑@erik-tank
You can find this talk at:
❑https://github.com/skeletonkey/Avro

Avro

Recommended

Recommended

More Related Content

Similar to Avro

Similar to Avro (20)

Recently uploaded

Recently uploaded (20)

Avro