sbt, history of JSON libraries, microservices, and schema evolution (Tokyo ver)
1. sbt,
history of JSON libraries,
microservices, and
schema evolution
Eugene Yokota (@eed3si9n)
February, 2017
2. • Scala hobbyist since 2010
• scalaxb (XML data binding)
• treehugger.scala
• sbt-assembly, sbt-buildinfo, etc
• “learning Scalaz” / “herding Cats”
• ScalaMatsuri
• Lightbend/Typesafe since 2014
• tech lead of Reactive Platform team
• current maintainer / tech lead of sbt
who is this guy (@eed3si9n)?
6. • How do you scale a technological organization?
• Goal: Sustainable development
development at scale
7. • Bezos mandate (written circa 2002 before AWS) https://plus.google.com/
+RipRowan/posts/eVeouesvaVX
microservice is a social stack
1. All teams will henceforth expose their data and functionality through service interfaces.
2. Teams must communicate with each other through these interfaces.
3. There will be no other form of inter-process communication allowed: no direct linking, no direct
reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only
communication allowed is via service interface calls over the network.
4. It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols -- doesn't
matter. Bezos doesn't care.
5. All service interfaces, without exception, must be designed from the ground up to be externalizable.
That is to say, the team must plan and design to be able to expose the interface to developers in the
outside world. No exceptions.
6. Anyone who doesn't do this will be fired.
16. • https://github.com/jonifreeman/literaljson
• Authored by Joni Freeman
• Custom parser
• Values stored in AST, JValue
• On August 11, 2009, Joni contributed literaljson to
Lift, and became Lift-JSON
jonifreeman/literaljson
20. • Allows adding capability to a class after the fact in a
typesafe manner.
Typeclass
trait Eq[A] {
def equals(x: A, y: A): Boolean
}
• Eq typeclass can enable === operator to compare
only the supported types, and prevent compilation
of "1" === 1
• Scala Implicits : Type Classes Here I Come
21. JsonFormat
trait CanRead[A] {
def reads(json: JsValue): A
}
trait CanWrite[A] {
def writes(a: A): JsValue
}
trait JsonFormat[A] extends CanRead[A]
with CanWrite[A]
implicit val intFormat: JsonFormat[Int] = ...
27. • http://argonaut.io/
• Purely functional JSON library
• Authored by Mark Hibberd, Tony Morris, Sean Parsons
• Uses Scalaz or Cats
• Very feature rich (Lenses, Cursor, History Cursor)
Argonaut
40. • “Serialization” tends to start from a programming language construct, and it
generates String or byte array.
• Data binding starts with a contract or a schema of the wire format, and generates
the binding in a programming language.
• XML Schema / WSDL
• Google Protocol Buffer
• Apache Thrift
• Apache Avro
• Facebook GraphQL
Serialization vs Data binding
42. • sealed traits
• case classes
Representing data in Scala
Cannot evolve in a binary compatible way.
43. Representing data in Scala
class Greeting(name: String) {
def copy(name: String = name): Greeting = ???
def unapply(v: Greeting): Option[String] = ???
}
class Greeting(name: String, x: Int) {
def copy(name: String = name,
x: Int = x): Greeting = ???
def unapply(v: Greeting): Option[(String, Int)] =
???
}
44. • sealed traits
• case classes
• Cannot evolve in a binary compatible way.
• But generating equals, hash, and toString is generally useful.
Representing data in Scala
52. Record type example
// DO NOT EDIT MANUALLY
package com.example
final class Person private (
val name: String,
val age: Option[Int]) extends Serializable {
override def equals(o: Any): Boolean = o match {
case x: Person => (this.name == x.name) && (this.age ==
x.age)
case _ => false
}
override def hashCode: Int = {
37 * (37 * (17 + name.##) + age.##)
}
override def toString: String = {
53. Record type example
override def toString: String = {
"Person(" + name + ", " + age + ")"
}
protected[this] def copy(name: String = name,
age: Option[Int] = age): Person = {
new Person(name, age)
}
def withName(name: String): Person = {
copy(name = name)
}
def withAge(age: Option[Int]): Person = {
copy(age = age)
}
def withAge(age: Int): Person = {
54. Record type example
object Person {
def apply(name: String, age: Option[Int]): Person =
new Person(name, age)
def apply(name: String, age: Int): Person =
new Person(name, Option(age))
}
58. JSON codec generation
package generated
import _root_.sjsonnew.{ deserializationError,
serializationError, Builder, JsonFormat, Unbuilder }
trait PersonFormats { self: sjsonnew.BasicJsonProtocol =>
implicit lazy val personFormat: JsonFormat[_root_.Person] =
JsonFormat[_root_.Person] {
override def read[J](jsOpt: Option[J], unbuilder:
Unbuilder[J]): _root_.Person = {
jsOpt match {
case Some(js) =>
unbuilder.beginObject(js)
val name = unbuilder.readField[String]("name")
val age = unbuilder.readField[Option[Int]]("age")
unbuilder.endObject()
_root_.Person(name)
59. JSON codec generation
scala> import sjsonnew.support.scalajson.unsafe.{ Converter,
CompactPrinter, Parser }
scala> import com.example.codec.CustomJsonProtocol._
scala> import com.example.Person
scala> val p = Person("Bob", 20)
p: com.example.Person = Person(Bob, 20)
scala> val j = Converter.toJsonUnsafe(p)
j: scala.json.ast.unsafe.JValue =
JObject([Lscala.json.ast.unsafe.JField;@6731ad72)
scala> val s = CompactPrinter(j)
s: String = {"name":"Bob","age":20}
scala> val x = Parser.parseUnsafe(s)
60. JSON codec generation
scala> val s = CompactPrinter(j)
s: String = {"name":"Bob","age":20}
scala> val x = Parser.parseUnsafe(s)
x: scala.json.ast.unsafe.JValue =
JObject([Lscala.json.ast.unsafe.JField;@7331f7f8)
scala> val q = Converter.fromJsonUnsafe[Person](x)
q: com.example.Person = Person(Bob, 20)
scala> assert(p == q)
61. • http://www.scala-sbt.org/contraband/
• Contraband is a description language for your datatypes and APIs, currently
targeting Java and Scala.
• Low-tech metaprogramming (code generation)
• Binary compatible evolution of pseudo case class.
• Auto derivation of backend-independent JSON codec.
Contraband
62. • An opportunity for datatype-generic programming.
• For example, one backend is Int
Contraband
63. • An opportunity for datatype-generic programming.
• For example, one backend is Int (Murmurhash support)
• Builder API is used to build one-way hash.
Contraband