An Introduction to Apache Avro, what is it and how is it used ? Plus some code examples to show JSON etc.

1. Apache Avro
● What is it ?
● Language API's
● JSON Example
● Schema Evolution
● Avro Use Example
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

2. Avro – What is it ?
● Data serialization system
● Uses JSON based schemas
● Uses RPC calls to send data
● Schema's sent during data exchange
● Integrated with many languages
● Fast binary data format or encode with JSON
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

3. Avro – Language API's
Avro API's exist for the following languages
● Java
● C#
● C
● C++
● Python
● Ruby
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

4. Avro – JSON Example
This text shows an example JSON schema for a customer
record
{
"namespace": "customer.avro",
"type": "record",
"name": "customer",
"fields": [
{"name": "name", "type": "string"},
{"name": "age", "type": "int"},
{"name": "address", "type": "string"},
{"name": "phone", "type": "string"},
{"name": "email", "type": "string"},
]
}
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

5. Avro – Schema Evolution
In a fast changing environment it is possible to
● Write data to file with a schema
● Change the schema
– Add extra fields
– Delete fields
– Rename fields
● And still read written file with changed schema
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

6. Avro – Use Example
// define schema for customer
{
"namespace": "customer.avro",
"type": "record",
"name": "Customer",
"fields": [
{"name": "name", "type": "string"},
{"name": "age", "type": "int"},
{"name": "address", "type": "string"},
{"name": "phone", "type": "string"},
{"name": "email", "type": "string"},
]
}
// compile the schema
java -jar /....../avro-tools-1.7.4.jar compile schema customer.avsc .
// create a couple of customers
Customer cust1 = new Customer("Dave", 33, "12 redmont street", "021234675", "davef@somedom@co.nz" );
Customer cust2 = new Customer("Emily", 27, "13a lake cresent", "022876234", "esuanders@somenz@co.nz" );
// now serialize customers to disk
File file = new File("customers.avro");
DatumWriter<Customer> userDatumWriter = new SpecificDatumWriter<Customer>(Customer.class);
DataFileWriter<Customer> dataFileWriter = new DataFileWriter<Customer>(customerDatumWriter);
dataFileWriter.create(cust1.getSchema(), new File("customers.avro"));
dataFileWriter.append(cust1);
dataFileWriter.append(cust2);
dataFileWriter.close();
www.semtech-solutions.co.nz info@semtech-solutions.co.nz

7. Contact Us
● Feel free to contact us at
– www.semtech-solutions.co.nz
– info@semtech-solutions.co.nz
● We offer IT project consultancy
● We are happy to hear about your problems
● You can just pay for those hours that you need
● To solve your problems