The Why and How
of Scala at Twitter
Alex Payne (@al3x)
Twitter Infrastructure
Who’s this guy?
•Working at Twitter since
2007, before it was even a
company.

•Spent a couple years on the
API.

•Co-auth...
Part I:
Why Scala?
So, like, Ruby or Scala?
•Both!
•Ruby mostly on the
frontend.

•Scala mostly on the
backend.

•Thrift and data stores to
c...
Elevator Pitch
•   It’s fun.

•   It’s fast.

•   It’s runs on a great virtual machine (JVM).

•   It’s got an awesome com...
No, seriously, why?
•   A rich static type system that gets out of your
    way when you don’t need it, is awesome when
  ...
I’m not sold yet.
•   How about the choice between immutable and
    mutable variables?

•   Powerful functional programmi...
What about concurrency?
•   You can do Actor concurrency (message
    passing), kinda like Erlang.

•   Or, use threads.

...
Basically,
it’s the future.
Part II:
How do you guys
  do Scala?
We build services.
•   Isolated components, can swap them out.

•   Talk to the rest of the system via Thrift (for
    now...
Scala Services at Twitter
•   Kestrel: queuing.

•   Flock (and Gizzard): social graph store.

•   Hawkwind: people search...
Thrift Example
   struct Results {
     1: list<i64> people
     2: string suggestion
     3: i32 processingTime /* millis...
We build libraries.
•   Reusable chunks of code that we can share
    between projects.

•   We open source them.

•   We ...
Ostrich
•   Gather stats from inside your running process.

•   Counters, gauges, and timings.

•   Share stats many ways:...
Ostrich Examples
Stats.time(“find-users”) {
    users.find(queryString)
}


Stats.incr(“write-errors”)


Stats.makeGauge("...
Configgy
•   Configuration files and logging.

•   Flexible file format offers includes, more.

•   Inheritance, variable ...
Configgy Examples
hostname = “service001”
port = 8080
use_admin_interface = true


log {
    filename = “my-service.log”
 ...
Configgy Examples
Configgy.configure(“/etc/my-service.conf”)
val config = Configgy.config


val port = config.getInt(“port...
xrayspecs
•   A set of extensions to the fantastic Specs BDD
    test framework.

•   Handles testing concurrency, time, c...
xrayspecs Examples
“response arrives from server” in {
    get(“/example”) must eventually(notBe(null))
}


“time should s...
scala-json
•   A cleaned-up version of the official Scala JSON
    codec.

•   Makes clever use of parser combinators – a
...
JSON Examples
val myJsonString = """
{"key": "value", "1": 2, "3": 4,
  "myList": [5, 6, 7]
"""
Json.parse(myJsonString)

...
Other Twitter Libraries
• Naggati: build protocols for Apache Mina.
• Smile: a memcached client that uses Actors.
• Querul...
Stuff We Use
•   There's great open source code in the Scala
    community.

•   We try to use and contribute back to thir...
sbt – the simple build tool
•   Scala’s answer to Ant and Maven.

•   Sets up new projects.

•   Maintains project configu...
specs
•   As aforementioned: great way to do BDD
    testing.

•   Set up pre- and post-conditions for
    individual test...
The IDE Question
•   We've had the best luck with IntelliJ IDEA,
    but it's still pretty rough.

•   Most of us use a pl...
Questions!
 Thanks for listening!
The Why and How of Scala at Twitter
The Why and How of Scala at Twitter
Próxima SlideShare
Cargando en…5
×

The Why and How of Scala at Twitter

49.354 visualizaciones

Publicado el

A presentation at Twitter's official developer conference, Chirp, about why we use the Scala programming language and how we build services in it. Provides a tour of a number of libraries and tools, both developed at Twitter and otherwise.

Publicado en: Tecnología
3 comentarios
155 recomendaciones
Estadísticas
Notas
Sin descargas
Visualizaciones
Visualizaciones totales
49.354
En SlideShare
0
De insertados
0
Número de insertados
1.002
Acciones
Compartido
0
Descargas
739
Comentarios
3
Recomendaciones
155
Insertados 0
No insertados

No hay notas en la diapositiva.































  • The Why and How of Scala at Twitter

    1. The Why and How of Scala at Twitter Alex Payne (@al3x) Twitter Infrastructure
    2. Who’s this guy? •Working at Twitter since 2007, before it was even a company. •Spent a couple years on the API. •Co-authored Programming Scala (O’Reilly, 2009). •Into programming languages, scaling stuff.
    3. Part I: Why Scala?
    4. So, like, Ruby or Scala? •Both! •Ruby mostly on the frontend. •Scala mostly on the backend. •Thrift and data stores to connect the two.
    5. Elevator Pitch • It’s fun. • It’s fast. • It’s runs on a great virtual machine (JVM). • It’s got an awesome community. • It can borrow from Java.
    6. No, seriously, why? • A rich static type system that gets out of your way when you don’t need it, is awesome when you do. • Flexible syntax makes it easy to write readable, maintainable code. • Traits for cross-cutting modularity. • Lots of OOP goodness.
    7. I’m not sold yet. • How about the choice between immutable and mutable variables? • Powerful functional programming: mapping, filtering, folding, currying, so much more. • Lazy values. • Pattern matching. • XML literals, and clean syntax for working with XML documents built right in!
    8. What about concurrency? • You can do Actor concurrency (message passing), kinda like Erlang. • Or, use threads. • Or, use all the java.util.concurrent tools. • Or, use other JVM concurrency frameworks (ex: Netty, Apache Mina).
    9. Basically, it’s the future.
    10. Part II: How do you guys do Scala?
    11. We build services. • Isolated components, can swap them out. • Talk to the rest of the system via Thrift (for now, maybe Avro in the future). • Independently tested for correctness, load. • Can have custom operational properties.
    12. Scala Services at Twitter • Kestrel: queuing. • Flock (and Gizzard): social graph store. • Hawkwind: people search. • Hosebird: streaming API. • more in the works all the time...
    13. Thrift Example struct Results { 1: list<i64> people 2: string suggestion 3: i32 processingTime /* milliseconds */ 4: list<i32> timings 5: i32 totalResults } service NameSearch { Results find(1: string name, 2: i32 maxResults, 3: bool wantSuggestion) Results find_with_ranking(1: string name, 2: i32 maxResults, 3: bool wantSuggestion, 4: Ranker ranking) }
    14. We build libraries. • Reusable chunks of code that we can share between projects. • We open source them. • We try to keep the “NIH” to a minimum, but we also have very particular requirements.
    15. Ostrich • Gather stats from inside your running process. • Counters, gauges, and timings. • Share stats many ways: JMX, JSON-over- HTTP, plain text Telnet-style socket, log files.
    16. Ostrich Examples Stats.time(“find-users”) { users.find(queryString) } Stats.incr(“write-errors”) Stats.makeGauge("database-lag") { dbConnection.getLagInMillis() }
    17. Configgy • Configuration files and logging. • Flexible file format offers includes, more. • Inheritance, variable substitution. • Tunable logging, Scribe support. • Subscription API: push and validate configuration changes. • Ruby support, Emacs mode.
    18. Configgy Examples hostname = “service001” port = 8080 use_admin_interface = true log { filename = “my-service.log” level = “error” roll = “hourly” }
    19. Configgy Examples Configgy.configure(“/etc/my-service.conf”) val config = Configgy.config val port = config.getInt(“port”, 8080) val hostname = config.getString(“hostname”) log.info(“service started”) log.debug(“running on port %d”, port)
    20. xrayspecs • A set of extensions to the fantastic Specs BDD test framework. • Handles testing concurrency, time, creating temporary folders. • The concurrency features have been rolled into the main distribution of Specs!
    21. xrayspecs Examples “response arrives from server” in { get(“/example”) must eventually(notBe(null)) } “time should stop” in { Time.freeze() val nowSnapshot = Time.now Thread.sleep(30) Time.now mustEqual nowSnapshot }
    22. scala-json • A cleaned-up version of the official Scala JSON codec. • Makes clever use of parser combinators – a good way to learn about them. • Heavily tested, fixes a bunch of edge cases.
    23. JSON Examples val myJsonString = """ {"key": "value", "1": 2, "3": 4, "myList": [5, 6, 7] """ Json.parse(myJsonString) Json.build(List("user_id", 666))
    24. Other Twitter Libraries • Naggati: build protocols for Apache Mina. • Smile: a memcached client that uses Actors. • Querulous: a nice database client. • Jackhammer: a load testing framework (coming soon). • probably stuff I'm leaving out...
    25. Stuff We Use • There's great open source code in the Scala community. • We try to use and contribute back to third- party projects. • Follow @implicit_ly for new Scala releases. • Also subscribe to Repopular Scala.
    26. sbt – the simple build tool • Scala’s answer to Ant and Maven. • Sets up new projects. • Maintains project configuration, build tasks, and dependencies in pure Scala. Totally open- ended. • Interactive console. • Will run tasks as soon as files in your project change – automatically compile and run tests!
    27. specs • As aforementioned: great way to do BDD testing. • Set up pre- and post-conditions for individual tests, suites, etc. • Extremely flexible and extensible. • Great, responsive maintainer. • Supports mocking (we like Mockito).
    28. The IDE Question • We've had the best luck with IntelliJ IDEA, but it's still pretty rough. • Most of us use a plain text editor like Emacs, VIM, or TextMate. • Combined with sbt, it's actually a really nice workflow!
    29. Questions! Thanks for listening!

    ×