The document discusses reactive approaches to collecting and modeling uncertain data from sensors in a distributed system. It presents reactive techniques like using queues and immutable data structures to handle out-of-order and concurrent updates at scale. Distributed databases allow scaling data collection across nodes while handling incomplete data from failures. The overall approach focuses on reactive systems principles of responding to changes instead of shared mutable state.

1. Collecting Uncertain Data
the Reactive Way
Jeff Smith
@jeffksmithjr

2. x.ai is a personal assistant
who schedules meetings for you

4. Reactive Machine Learning

8. Machine Learning Systems

9. Machine Learning Systems

10. Machine Learning Systems

11. Traits of Reactive Systems

12. Traits of Reactive Systems

13. Reactive Strategies

14. Reactive Strategies

15. Reactive Machine Learning

16. Reactive Machine Learning

17. Reactive Machine Learning

18. Collecting Data

22. What’s for
dinner?

29. Reactive Data Collection

30. Modeling Uncertain Data

31. Certain Data Model
case class ZebraReading(sensorId: Int,
locationId: Int,
timestamp: Long,
count: Int)

33. Uncertainty Interval
27 33

34. Uncertain Data Model
case class PreyReading(sensorId: Int,
locationId: Int,
timestamp: Long,
animalsLowerBound: Double,
animalsUpperBound: Double,
percentZebras: Double)

35. Scaling Data Collection

37. Simple Data Architecture

38. Simple Data Architecture

39. Mutable State
case class Region(id: Int)
import collection.mutable.HashMap
var densities = new HashMap[Region, Double]()
densities.put(Region(4), 52.4)

40. Scaling with Queues

41. Scaling with Queues

42. Out of Order Updates

43. Out of Order Updates
densities.put(Region(6), 73.6)
densities.put(Region(6), 0.5)
densities.get(Region(6)).get

44. Out of Order Updates
densities.put(Region(6), 73.6)
densities.put(Region(6), 0.5)
densities.get(Region(6)).get
densities.put(Region(6), 0.5)
densities.put(Region(6), 73.6)
densities.get(Region(6)).get

45. Concurrent Collections
import collection.mutable._
var synchronizedDensities = new LinkedHashMap[Region, Double]()
with SynchronizedMap[Region, Double]

46. Scaling with Locks

47. Scaling with Locks

48. Immutable Facts
case class PreyReading(sensorId: Int,
locationId: Int,
timestamp: Long,
animalsLowerBound: Double,
animalsUpperBound: Double,
percentZebras: Double)
implicit val preyReadingFormatter = Json.format[PreyReading]

49. Immutable Facts
val reading = PreyReading(36,
12,
currentTimeMillis(),
12.0,
18.0,
0.60)
val setDoc = bucket.set[PreyReading](readingId(reading), reading)

50. Scaling with Distributed Databases

51. Scaling with Distributed Databases

52. Handling Incomplete Data

53. Distributed Data Storage

54. Querying Complete Data
(bucket.searchValues[PreyReading]("prey", "by_sensor_id")
(new Query().setIncludeDocs(true)))
.enumerate.apply(Iteratee.foreach { doc =>
println(s"Prey Reading: $doc")})

55. Complete Data

56. Partition Tolerance

57. Partition Tolerance

58. Partition Tolerance

59. Partition Tolerance

60. Querying Incomplete Data
(bucket.searchValues[PreyReading]("prey", "by_sensor_id")
(new Query().setIncludeDocs(true)))
.enumerate.apply(Iteratee.foreach { doc =>
println(s"Prey Reading: $doc")})

61. Incomplete Data

62. Incomplete Data

63. Reactive Data Collection

64. For Later

65. reactivemachinelearning.com
medium.com/data-engineering
M A N N I N G
Jeff Smith

66. x.ai
@xdotai
hello@human.x.ai
New York, New York

67. skillsmatter.com/conferences/
6862-scala-exchange-2015#skillscasts

68. Thank You

69. Collecting Uncertain Data
the Reactive Way
Jeff Smith
@jeffksmithjr