Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

ZIO Schedule: Conquering Flakiness & Recurrence with Pure Functional Programming

1.485 visualizaciones

Publicado el

As professional software engineers, sometimes messy details of the real world stand in the way of us delivering principled software. Flaky connections, unreliable services, and bulletproof job scheduling in the presence of non-determinism and failure all tricky problems that discourage us from writing principled software. Yet sometimes the shortcuts we take to solve these problems result in downtime for the business and sleepless nights for us.

In this brand-new presentation, created exclusively for Scala in the City, John A. De Goes will show how functional programming can help bring order to even the most chaotic systems. Using ZIO, a new zero-dependency Scala library for building massively scalable asynchronous and concurrent applications, John will demonstrate how functional programming leverages reified effects and algebras to solve the trickiest of reliability and scheduling problems in a principled, composable, flexible way.

Join John for an evening of fun and functional programming as you explore fresh ways of thinking about reliability and scheduling, and come out of the talk with valuable skills for using ZIO to solve the everyday problems you encounter at work.

Publicado en: Tecnología
  • Sé el primero en comentar

ZIO Schedule: Conquering Flakiness & Recurrence with Pure Functional Programming

  1. 1. ZIO Schedule John A. De Goes — @jdegoes - http://degoes.net Scala in the City ZIO Schedule Scala in the City, October 2018 - London, UK John A. De Goes @jdegoes - http://degoes.net
  2. 2. What is ZIO? Retries Repeats Schedule Composition
  3. 3. What is ZIO? Retries Repeats Schedule Composition ZIO ZIO lets you build high-performance, type-safe, concurrent, asynchronous applications that don’t leak resources, and are easy to reason about compositionally, test, and refactor.
  4. 4. github.com/scalaz/scalaz-zio ZIO What is ZIO? Retries Repeats Schedule Composition
  5. 5. def main: Unit = { println("Hello, what is your name?") val name = readLine() println(s"Good morning, $name!") } Second-Class Effects ✗ Pass to functions ✗ Return from functions ✗ Store in data structures ✗ Async/sync/concurrency ✗ Async/sync resource-safety What is ZIO? Retries Repeats Schedule Composition
  6. 6. def main: IO[IOException, Unit] = for { _ <- putStrLn("Hello, what is your name?") name <- getStrLn _ <- putStrLn(s"Good morning, $name!") } yield () First-Class Effects ✓ Pass to functions ✓ Return from functions ✓ Store in data structures ✓ Async/sync/concurrency ✓ Async/sync resource-safety What is ZIO? Retries Repeats Schedule Composition
  7. 7. IO[E, A] IO[E, A] is an immutable value that describes an effectful program, which may fail with a value of type E, or return a value of type A. What is ZIO? Retries Repeats Schedule Composition ZIO IO
  8. 8. What is ZIO? Retries Repeats Schedule Composition IO[Nothing, A] Never fails ZIO IO
  9. 9. What is ZIO? Retries Repeats Schedule Composition IO[E, Nothing] Never returns ZIO IO
  10. 10. What is ZIO? Retries Repeats Schedule Composition IO[Nothing, Nothing] Never fails or returns ZIO IO
  11. 11. for { queue <- Queue.unbounded[String] worker = queue.take.flatMap(putStrLn).forever workers10k = List.fill(10000)(worker) _ <- IO.forkAll(workers10k) _ <- queue.offer("Chocolate!").forever.fork } yield () Lightweight “Threads” ✓ Massive scalability over threads ✓ Non-blocking ✓ Safe concurrency ✓ Automatically interruptible ✓ Always resource-safe ✓ Powerful composition What is ZIO? Retries Repeats Schedule Composition
  12. 12. for { resp <- api.getUser(userId) _ <- db.updateUser(userId, resp.profile) _ <- promise.complete(()) } yield () What is ZIO? Retries Repeats Schedule Composition ✓ Massive scalability over threads ✓ Non-blocking ✓ Safe concurrency ✓ Automatically interruptible ✓ Always resource-safe ✓ Powerful composition Asynchronicity
  13. 13. requests.parTraverse { request => for { product <- findProduct(request.id) update <- applyDiscount(product.id, discount) _ <- db.updateProduct(product.id, update) } yield product.id } What is ZIO? Retries Repeats Schedule Composition ✓ Massive scalability over threads ✓ Non-blocking ✓ Safe concurrency ✓ Automatically interruptible ✓ Always resource-safe ✓ Powerful composition Concurrency
  14. 14. def fib(n: Int): IO[Nothing, Int] = if (n <= 1) IO.now(n) else fib(n-1).parWith(fib(n-2))(_ + _) fib(Int.MaxValue).fork(_.interrupt) What is ZIO? Retries Repeats Schedule Composition ✓ Massive scalability over threads ✓ Non-blocking ✓ Safe concurrency ✓ Automatically interruptible ✓ Always resource-safe ✓ Powerful composition Interruption
  15. 15. openFile.bracket(closeFile(_)) { handle => def loop(ref: Ref[Chunk[Byte]]) = for { chunk <- handle.readChunk <- ref.update(_ ++ chunk) all <- if (handle.hasMore) loop(ref) else ref.get } yield all Ref(Chunk.empty).flatMap(loop) } What is ZIO? Retries Repeats Schedule Composition ✓ Massive scalability over threads ✓ Non-blocking ✓ Safe concurrency ✓ Automatically interruptible ✓ Always resource-safe ✓ Powerful composition Bracket
  16. 16. def webCrawler[E: Monoid, A: Monoid](seeds: Set[URL], router: URL => Set[URL], processor: (URL, String) => IO[E, A]): IO[Nothing, Crawl[E, A]] = { def loop(seeds: Set[URL], ref: Ref[CrawlState[E, A]]): IO[Nothing, Unit] = ref.update(_ |+| CrawlState.visited(seeds)) *> IO.parTraverse(seeds)( seed => getURL(seed).redeem(_ => IO.unit, html => for { visited <- ref.get.map(_.visited) seeds <- IO.now(extractURLs(seed, html).toSet .flatMap(router) -- visited) crawl <- processor(seed, html).redeemPure(Crawl(_, mzero[A]), Crawl(mzero[E], _)) _ <- ref.update(_ |+| CrawlState.crawled(crawl)) _ <- loop(seeds, ref) } yield ())).void Ref(mzero[CrawlState[E, A]]).flatMap(ref => loop(seeds, ref).map(_ => ref.get.map(_.crawl))) } What is ZIO? Retries Repeats Schedule Composition ✓ Massive scalability over threads ✓ Non-blocking ✓ Safe concurrency ✓ Automatically interruptible ✓ Always resource-safe ✓ Powerful composition Composable
  17. 17. Retries What is ZIO? Retries Repeats Schedule Composition
  18. 18. What is ZIO? Retries Repeats Schedule Composition App Web API
  19. 19. What is ZIO? Retries Repeats Schedule Composition App Web API 408
  20. 20. What is ZIO? Retries Repeats Schedule Composition App Web API 408429
  21. 21. What is ZIO? Retries Repeats Schedule Composition App Web API 408429 423
  22. 22. What is ZIO? Retries Repeats Schedule Composition App Web API 408429 423 500
  23. 23. What is ZIO? Retries Repeats Schedule Composition App Web API 408429 423 500 503
  24. 24. What is ZIO? Retries Repeats Schedule Composition App Web API 408429 423 500 503 504
  25. 25. What is ZIO? Retries Repeats Schedule Composition App Web API 408429 423 500 503 504
  26. 26. What is ZIO? Retries Repeats Schedule Composition AppDatabase Cache Local Storage Cloud Storage Streaming Logs Analytics Web APIs
  27. 27. def networkRequest(): A = ??? def networkRequestWithRetries(max: Int, millis: Long): A = { var i = 0 while (i < max) { try { return networkRequest() } catch { case _ : Exception => i = i + 1 Thread.sleep(millis) } } } What is ZIO? Retries Repeats Schedule Composition Retrying Synchronously
  28. 28. Repeats What is ZIO? Retries Repeats Schedule Composition
  29. 29. What is ZIO? Retries Repeats Schedule Composition App Generate Report
  30. 30. What is ZIO? Retries Repeats Schedule Composition App Generate Report Every user
  31. 31. What is ZIO? Retries Repeats Schedule Composition App Generate Report Every user Every day
  32. 32. What is ZIO? Retries Repeats Schedule Composition App Generate Report Every user Every day Every account
  33. 33. What is ZIO? Retries Repeats Schedule Composition App Generate Report Every user Every day Every account Every week
  34. 34. What is ZIO? Retries Repeats Schedule Composition AppService Heartbeat Data Retrieval Remote Uploading Report Generation Log Rotation File Cleanup User Reminders Garbage Collection
  35. 35. What is ZIO? Retries Repeats Schedule Composition def reportGen(): A = ??? def repeatedReportGen(max: Int, millis: Long): List[A] = { var list = List.empty[A] var i = 0 while (i < max) { list = reportGen() :: list Thread.sleep(millis) i = i + 1 } list } Repeating Synchronously
  36. 36. Schedule What is ZIO? Retries Repeats Schedule Composition
  37. 37. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Schedule[A, B] Schedule[A, B] is an immutable value that describes an effectful schedule, which after consuming an A, produces a B and decides to halt or continue after some delay d.
  38. 38. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Schedule[A, B] A B B Continue? Continue after delay d Halt immediately
  39. 39. What is ZIO? Retries Repeats Schedule Composition Should I repeat a program that failed with an E? Should I repeat a program that returned an A? Retries Repeats
  40. 40. What is ZIO? Retries Repeats Schedule Composition Retries Repeats Schedule[E, B] Schedule[A, B] Retry policy Consumes errors of type E Emits values of type B Repeat policy Consumes return values of type A Emits values of type B
  41. 41. What is ZIO? Retries Repeats Schedule Composition Schedule.never : Schedule[Any, Nothing] Zero Recurrences Accepts any input Never recurs, never emitting a value
  42. 42. What is ZIO? Retries Repeats Schedule Composition Schedule.once : Schedule[Any, Unit] One Recurrence Accepts any input Recurs once, emitting unit
  43. 43. What is ZIO? Retries Repeats Schedule Composition Schedule.forever : Schedule[Any, Int] Infinite Recurrences Consumes any input Recurs forever without delay, emitting the number of recurrences so far
  44. 44. What is ZIO? Retries Repeats Schedule Composition Schedule.recurs(n: Int) : Schedule[Any, Int] Fixed Recurrences Accepts any input Recurs the specified number of times, emitting number of recurrences so far
  45. 45. durationtask task What is ZIO? Retries Repeats Schedule Composition Schedule.spaced(d: Duration) : Schedule[Any, Int] Spaced Recurrences Accepts any input Recurs forever with delay, emitting the number of recurrences so far
  46. 46. duration task task What is ZIO? Retries Repeats Schedule Composition Schedule.fixed(d: Duration) : Schedule[Any, Int] Fixed Recurrences Accepts any input Recurs forever with a delay, emitting number of recurrences so far duration
  47. 47. What is ZIO? Retries Repeats Schedule Composition Schedule.exponential(d: Duration, f: Double = 2.0) : Schedule[Any, Duration] Exponential Recurrences Accepts any input Recurs forever with delay, emitting the current delay between steps Scaling factor Starting duration
  48. 48. What is ZIO? Retries Repeats Schedule Composition Schedule.doWhile[A](f: A => Boolean): Schedule[A, A] Conditional Recurrences Accepts an A Recurs while the predicate is true without delay, emitting the same A Recurrence condition
  49. 49. What is ZIO? Retries Repeats Schedule Composition Schedule.doUntil[A](f: A => Boolean): Schedule[A, A] Conditional Recurrences Accepts an A Recurs until the predicate is true without delay, emitting the same A Recurrence condition
  50. 50. What is ZIO? Retries Repeats Schedule Composition Schedule.collect: Schedule[A, List[A]] Collecting Recurrences Recurs forever without delay, emitting a list of all consumed A’s Consumes an A
  51. 51. What is ZIO? Retries Repeats Schedule Composition Schedule.identity[A]: Schedule[A, A] Identity Recurrences Recurs forever without delay, emitting the same A Consumes an A
  52. 52. What is ZIO? Retries Repeats Schedule Composition Schedule.unfold[A](a: => A)(f: A => A): Schedule[Any, A] Unfold Recurrences Recurs forever without delay, emitting an A by unfolding the seed through repeated application Consumes any value
  53. 53. What is ZIO? Retries Repeats Schedule Composition Schedule.point[A](a: => A): Schedule[Any, A] Constant Output Recurs forever without delay, emitting a constant Consumes any value
  54. 54. What is ZIO? Retries Repeats Schedule Composition Schedule.lift[A, B](f: A => B): Schedule[A, B] Function Output Recurs forever without delay, emitting the function applied to the input Consumes an A
  55. 55. What is ZIO? Retries Repeats Schedule Composition Retries Repeats val action: IO[E, A] = ??? val policy: Schedule[E, B] = ??? val retried: IO[E, A] = action retry policy val action: IO[E, A] = ??? val policy: Schedule[A, B] = ??? val repeated: IO[E, B] = action repeat policy
  56. 56. What is ZIO? Retries Repeats Schedule Composition Retries Repeats val action: IO[E, A] = ??? val policy: Schedule[E, B] = ??? val orElse: (E, B) => IO[E2, A] = ??? val retried: IO[E2, A] = action retryOrElse (policy, orElse) val action: IO[E, A] = ??? val policy: Schedule[A, B] = ??? val orElse: (E, Option[B]) => IO[E2, B] = ??? val repeated: IO[E2, B] = action repeatOrElse (policy, onError)
  57. 57. What is ZIO? Retries Repeats Schedule Composition Retries Repeats val action: IO[E, A] = ??? val policy: Schedule[E, B] = ??? val orElse: (E, B) => IO[E2, B] = ??? val retried: IO[E2, Either[B, A]] = action retryOrElse0 (policy, orElse) val action: IO[E, A] = ??? val policy: Schedule[A, B] = ??? val orElse: (E, Option[B]) => IO[E2, C] = ??? val repeated: IO[E2, Either[C, B]] = action repeatOrElse0 (policy, onError)
  58. 58. Composition What is ZIO? Retries Repeats Schedule Composition
  59. 59. What is ZIO? Retries Repeats Schedule Composition def networkRequestWithRetries(factor: Float = 1.5f, init: Int = 1, cur: Int = 0) (implicit as: ActorSystem): Future[String] = { networkRequest().recoverWith { case NetworkException => val next: Int = if (cur == 0) init else Math.ceil(cur * factor).toInt after(next.milliseconds, as.scheduler, global, Future.successful(1)).flatMap { _ => networkRequestWithRetries(factor, init, next) } case t: Throwable => throw t } } Future Retries* * https://hackernoon.com/exponential-back-off-with-scala-futures-7426340d0069
  60. 60. What is ZIO? Retries Repeats Schedule Composition def retry[F[_]: Timer: RaiseThrowable, O]( fo : F[O], delay : FiniteDuration, nextDelay : FiniteDuration => FiniteDuration, maxAttempts : Int, retriable : Throwable => Boolean): Stream[F, O] = ??? FS2 Retry
  61. 61. What is ZIO? Retries Repeats Schedule Composition Monix Retry & Repeat Retries Repeats val task : Task[A] = ??? val retries: Int = ??? val retried1: Task[A] = task onErrorRestart (retries) val pred: Throwable => Boolean = ??? val retried2: Task[A] = task onErrorRestartIf (pred) // onErrorRestartLoop val task: Task[A] = ??? val pred: A => Boolean = ??? val repeated: Task[A] = task restartUntil (pred)
  62. 62. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Monoid append zero
  63. 63. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Monoid append zero Applicative map point ap
  64. 64. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Monoid append zero Applicative map point ap Category id compose
  65. 65. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Monoid append zero Applicative map point ap Category id compose Profunctor lmap rmap dimap
  66. 66. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Monoid append zero Applicative map point ap Category id compose Profunctor lmap rmap dimap Strong first second
  67. 67. What is ZIO? Retries Repeats Schedule Composition ZIO SCHEDULE Monoid append zero Applicative map point ap Category id compose Profunctor lmap rmap dimap Strong first second Choice left right
  68. 68. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]) && (s2 : Schedule[A, C]) : Schedule[A, (B, C)] Intersection s1 s2 s1 && s2 Continue : Boolean b1 b2 b1 && b2 Delay : Duration d1 d2 d1.max(d2) Emit : (A, B) a b (a, b)
  69. 69. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]) || (s2 : Schedule[A, C]) : Schedule[A, (B, C)] Union s1 s2 s1 || s2 Continue : Boolean b1 b2 b1 || b2 Delay : Duration d1 d2 d1.min(d2) Emit : (A, B) a b (a, b)
  70. 70. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]) andThen (s2 : Schedule[A, C]) : Schedule[A, Either[B, C]] Sequence A A B C s1 s2 s1 andThen s2A Either[B, C]
  71. 71. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]) >>> (s2 : Schedule[B, C]) : Schedule[A, C] Compose A B B C s1 s2 s1 >>> s2A C
  72. 72. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]).jittered : Schedule[A, B] Jittering A AB Bs1 s1’ Delays randomly jittered
  73. 73. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]).collect : Schedule[A, List[B]] Collecting A AB List[C]s1 s1’ Emissions collected
  74. 74. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]).whileInput(f: A => Boolean) : Schedule[A, B] Filtering By Input A AB Bs1 s1’ Continues while/until f returns true (s1 : Schedule[A, B]).untilInput(f: A => Boolean) : Schedule[A, B]
  75. 75. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]).whileOutput(f: B => Boolean) : Schedule[A, B] Filtering By Output A AB Bs1 s1’ Continues while/until f returns true (s1 : Schedule[A, B]).untilOutput(f: B => Boolean) : Schedule[A, B]
  76. 76. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]).map(f: B => C) : Schedule[A, C] Mapping A AB Cs1 s1’ B mapped to C
  77. 77. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]) *> (s2 : Schedule[A, C]) : Schedule[A, C] (s1 && s2).map(_._2) Left / Right Ap (s1 : Schedule[A, B]) <* (s2 : Schedule[A, C]) : Schedule[A, B] (s1 && s2).map(_._1)
  78. 78. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]).logInput(f: A => IO[Nothing, Unit]) : Schedule[A, B] Logging Inputs A AB Bs1 s1’ Inputs logged to f
  79. 79. What is ZIO? Retries Repeats Schedule Composition (s1 : Schedule[A, B]).logOutput(f: B => IO[Nothing, Unit]) : Schedule[A, B] Logging Outputs A AB Bs1 s1’ Outputs logged to f
  80. 80. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule Produce a jittered schedule that first does exponential spacing (starting from 10 milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed spacing of 60 seconds between recurrences, but will only do that for up to 100 times, and emits a list of the collected inputs.
  81. 81. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule Produce a jittered schedule that first does exponential spacing (starting from 10 milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed spacing of 60 seconds between recurrences, but will only do that for up to 100 times, and emits a list of the collected inputs.
  82. 82. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule Produce a jittered schedule that first does exponential spacing (starting from 10 milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed spacing of 60 seconds between recurrences, but will only do that for up to 100 times, and emits a list of the collected inputs. Schedule.exponential(10.milliseconds)
  83. 83. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule Produce a jittered schedule that first does exponential spacing (starting from 10 milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed spacing of 60 seconds between recurrences, but will only do that for up to 100 times, and emits a list of the collected inputs. (Schedule.exponential(10.milliseconds) andThen ???)
  84. 84. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule Produce a jittered schedule that first does exponential spacing (starting from 10 milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed spacing of 60 seconds between recurrences, but will only do that for up to 100 times, and emits a list of the collected inputs. (Schedule.exponential(10.milliseconds).whileOutput(_ < 60.seconds) andThen ???)
  85. 85. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule Produce a jittered schedule that first does exponential spacing (starting from 10 milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed spacing of 60 seconds between recurrences, but will only do that for up to 100 times, and emits a list of the collected inputs. (Schedule.exponential(10.milliseconds).whileOutput(_ < 60.seconds) andThen Schedule.fixed(60.seconds))
  86. 86. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule Produce a jittered schedule that first does exponential spacing (starting from 10 milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed spacing of 60 seconds between recurrences, but will only do that for up to 100 times, and emits a list of the collected inputs. (Schedule.exponential(10.milliseconds).whileOutput(_ < 60.seconds) andThen (Schedule.fixed(60.seconds) && Schedule.recurs(100))
  87. 87. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule Produce a jittered schedule that first does exponential spacing (starting from 10 milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed spacing of 60 seconds between recurrences, but will only do that for up to 100 times, and emits a list of the collected inputs. (Schedule.exponential(10.milliseconds).whileOutput(_ < 60.seconds) andThen (Schedule.fixed(60.seconds) && Schedule.recurs(100)).jittered
  88. 88. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule Produce a jittered schedule that first does exponential spacing (starting from 10 milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed spacing of 60 seconds between recurrences, but will only do that for up to 100 times, and emits a list of the collected inputs. (Schedule.exponential(10.milliseconds).whileOutput(_ < 60.seconds) andThen (Schedule.fixed(60.seconds) && Schedule.recurs(100)).jittered *> Schedule.identity[A].collect
  89. 89. What is ZIO? Retries Repeats Schedule Composition Crafting a ZIO Schedule Produce a jittered schedule that first does exponential spacing (starting from 10 milliseconds), but then after the spacing reaches 60 seconds, switches over to fixed spacing of 60 seconds between recurrences, but will only do that for up to 100 times, and emits a list of the collected inputs. def customSchedule[A]: Schedule[A, List[A]] = { import Schedule._ (exponential(10.millis).whileOutput(_ < 60.secs) andThen (fixed(60.secs) && recurs(100)).jittered *> identity[A].collect }
  90. 90. Github https://github.com/scalaz/scalaz-zio Microsite https://scalaz.github.io/scalaz-zio/ ZIO Chat https://gitter.im/scalaz/scalaz-zio/ Scalaz Chat https://gitter.im/scalaz/scalaz/ That’s A Wrap!
  91. 91. Functional Scala by John A. De Goes Local Remote Date London London Time Oct 21 - 23 Paris Paris Time Oct 25 - 27 SF / SV Pacific Coast Time Nov 18 - 21 That’s A Wrap!
  92. 92. THANK YOU! @jdegoes - http://degoes.net That’s A Wrap!

×