In this InfluxDays NYC 2019 talk, InfluxData Founder & CTO Paul Dix will outline his vision around the platform and its new data scripting and query language Flux, and he will give the latest updates on InfluxDB time series database. This talk will walk through the vision and architecture with demonstrations of working prototypes of the projects.

1. Flux and InfluxDB 2.0
Paul Dix
@pauldix
paul@influxdata.com

3. • Data-scripting language
• Functional
• MIT Licensed
• Language, VM, engine, planner, optimizer

4. Language + Query Engine

7. 2.0

8. Biggest Change Since 0.9

9. Clean Migration Path

10. Compatibility Layer

11. • MIT Licensed
• Multi-tenanted
• Telegraf, InfluxDB, Chronograf, Kapacitor rolled into 1
• OSS single server
• Cloud usage based pricing
• Dedicated Cloud
• Enterprise on-premise

12. Consistent Documented API
Collection, Write/Query, Streaming & Batch Processing, Dashboards

14. Officially Supported Client
Libraries
Go, Node.js, Ruby, Python, PHP, Java, C#, C, Kotlin

15. Visualization Libraries

16. Multi-tenant roles
• Operator
• Organization Administrator
• User

17. Data Model
• Organizations
• Buckets (retention)
• Time series data
• Tasks
• Runs
• Logs
• Dashboards
• Users
• Tokens
• Authorizations
• Protos (templates)
• Scrapers
• Telegrafs
• Labels

19. Ways to run Flux - (interpreter,
InfluxDB 1.7 & 2.0)

21. Flux Basics

22. // get all data from the telegraf db
from(bucket:”telegraf/autogen”)
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")

23. // get all data from the telegraf db
from(bucket:”telegraf/autogen”)
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
Comments

24. // get all data from the telegraf db
from(bucket:”telegraf/autogen”)
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
Named Arguments

25. // get all data from the telegraf db
from(bucket:”telegraf/autogen”)
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
String Literals

26. // get all data from the telegraf db
from(bucket:”telegraf/autogen”)
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
Buckets, not DBs

27. // get all data from the telegraf db
from(bucket:”telegraf/autogen”)
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
Duration Literal

28. // get all data from the telegraf db
from(bucket:”telegraf/autogen”)
// filter that by the last hour
|> range(start:2018-11-07T00:00:00Z)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
Time Literal

29. // get all data from the telegraf db
from(bucket:”telegraf/autogen”)
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
Pipe forward operator

30. // get all data from the telegraf db
from(bucket:”telegraf/autogen”)
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
Anonymous Function

31. // get all data from the telegraf db
from(bucket:”telegraf/autogen”)
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => (r._measurement == "cpu" or r._measurement == “cpu")
and r.host == “serverA")
Predicate Function

32. // variables
some_int = 23

33. // variables
some_int = 23
some_float = 23.2

34. // variables
some_int = 23
some_float = 23.2
some_string = “cpu"

35. // variables
some_int = 23
some_float = 23.2
some_string = “cpu"
some_duration = 1h

36. // variables
some_int = 23
some_float = 23.2
some_string = “cpu"
some_duration = 1h
some_time = 2018-10-10T19:00:00

37. // variables
some_int = 23
some_float = 23.2
some_string = “cpu"
some_duration = 1h
some_time = 2018-10-10T19:00:00
some_array = [1, 6, 20, 22]

38. // variables
some_int = 23
some_float = 23.2
some_string = “cpu"
some_duration = 1h
some_time = 2018-10-10T19:00:00
some_array = [1, 6, 20, 22]
some_object = {foo: "hello" bar: 22}

39. // defining a pipe forwardable function
square = (tables=<-) =>
tables
|> map(fn: (r) => {r with _value: r._value * r._value})

40. // defining a pipe forwardable function
square = (tables=<-) =>
tables
|> map(fn: (r) => {r with _value: r._value * r._value})
This is potentially new

41. // defining a pipe forwardable function
square = (tables=<-) =>
tables
|> map(fn: (r) => {r with _value: r._value * r._value})
from(bucket:"foo")
|> range(start: -1h)
|> filter(fn: (r) => r._measurement == "samples")
|> square()
|> filter(fn: (r) => r._value > 23.2)

42. Data Sources (inputs)

43. Data Sinks (outputs)

44. Tasks

45. option task = {
name: "email alert digest",
cron: "0 5 * * 0"
}
import "smtp"
body = ""
from(bucket: "alerts")
|> range(start: -24h)
|> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message")
|> group(columns: ["alert"])
|> count()
|> group()
|> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn")
smtp.to(
config: loadSecret(name: "smtp_digest"),
to: "alerts@influxdata.com",
title: "Alert digest for {now()}",
body: message)

46. option task = {
name: "email alert digest",
cron: "0 5 * * 0"
}
import "smtp"
body = ""
from(bucket: "alerts")
|> range(start: -24h)
|> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message")
|> group(columns: ["alert"])
|> count()
|> group()
|> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn")
smtp.to(
config: loadSecret(name: "smtp_digest"),
to: "alerts@influxdata.com",
title: "Alert digest for {now()}",
body: message)
tasks

47. option task = {
name: "email alert digest",
cron: "0 5 * * 0"
}
import "smtp"
body = ""
from(bucket: "alerts")
|> range(start: -24h)
|> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message")
|> group(columns: ["alert"])
|> count()
|> group()
|> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn")
smtp.to(
config: loadSecret(name: "smtp_digest"),
to: "alerts@influxdata.com",
title: "Alert digest for {now()}",
body: message)
cron scheduling

48. option task = {
name: "email alert digest",
cron: "0 5 * * 0"
}
import "smtp"
body = ""
from(bucket: "alerts")
|> range(start: -24h)
|> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message")
|> group(columns: ["alert"])
|> count()
|> group()
|> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn")
smtp.to(
config: loadSecret(name: "smtp_digest"),
to: "alerts@influxdata.com",
title: "Alert digest for {now()}",
body: message)
packages & imports

49. option task = {
name: "email alert digest",
cron: "0 5 * * 0"
}
import "smtp"
body = ""
from(bucket: "alerts")
|> range(start: -24h)
|> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message")
|> group(columns: ["alert"])
|> count()
|> group()
|> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn")
smtp.to(
config: loadSecret(name: "smtp_digest"),
to: "alerts@influxdata.com",
title: "Alert digest for {now()}",
body: message)
map

50. option task = {
name: "email alert digest",
cron: "0 5 * * 0"
}
import "smtp"
body = ""
from(bucket: "alerts")
|> range(start: -24h)
|> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message")
|> group(columns: ["alert"])
|> count()
|> group()
|> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn")
smtp.to(
config: loadSecret(name: "smtp_digest"),
to: "alerts@influxdata.com",
title: "Alert digest for {now()}",
body: message) String interpolation

51. option task = {
name: "email alert digest",
cron: "0 5 * * 0"
}
import "smtp"
body = ""
from(bucket: "alerts")
|> range(start: -24h)
|> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message")
|> group(columns: ["alert"])
|> count()
|> group()
|> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn")
smtp.to(
config: loadSecret(name: "smtp_digest"),
to: "alerts@influxdata.com",
title: "Alert digest for {now()}",
body: message)
Ship data elsewhere

52. option task = {
name: "email alert digest",
cron: "0 5 * * 0"
}
import "smtp"
body = ""
from(bucket: "alerts")
|> range(start: -24h)
|> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message")
|> group(columns: ["alert"])
|> count()
|> group()
|> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn")
smtp.to(
config: loadSecret(name: "smtp_digest"),
to: "alerts@influxdata.com",
title: "Alert digest for {now()}",
body: message)
Store secrets in a
store like Vault

53. Open Questions

54. User Packages &
Dependencies

55. // in a file called package.flux
package "paul"
option version = "0.1.1"
// define square here or…

56. // in a file called package.flux
package "paul"
option version = "0.1.1"
// import the other package files
// they must have package "paul" declaration at the top
// only package.flux has the version
import “packages”
packages.load(files: ["square.flux", "utils.flux"])

57. // in a file called package.flux
package "paul"
option version = "0.1.1"
// import the other package files
// they must have package "paul" declaration at the top
// only package.flux has the version
import “packages”
packages.load(files: ["square.flux", "utils.flux"])
// or this
packages.load(glob: "*.flux")

58. import "myorg/paul" // latest, will load package.flux
data |> paul.square()

59. import "myorg/paul", "0.1.0" // specific version
// 1. look in $fluxhome/myorg/paul/package.flux
// 2. look in $fluxhome/myorg/paul/0.1.0/package.flux
// 3. look in cloud2.influxdata.com/api/v1/packages/myorg/paul
data |> paul.square()

60. import "myorg/paul", ">=0.1.0" // at least this version
data |> paul.square()

61. Error Handling?

62. import "slack"
// what if this returns an error?
ret = slack.send(room: "foo", message: "testing this", token: "...")

63. Option Types?

64. match ret {
// on match ret gets mapped as the new type
Error => {
// do something ret
},
Else => {
// do something with ret
}
}

65. Loops?

66. records = [
{name: "foo", value: 23},
{name: "bar", value: 23},
{name: "asdf", value: 56}
]

67. records = [
{name: "foo", value: 23},
{name: "bar", value: 23},
{name: "asdf", value: 56}
]
// simple loop over each
records
|> map(fn: (r) => {name: r.name, value: r.value + 1})

68. records = [
{name: "foo", value: 23},
{name: "bar", value: 23},
{name: "asdf", value: 56}
]
// simple loop over each
records
|> map(fn: (r) => {name: r.name, value: r.value + 1})
// compute the sum
sum = records
|> reduce(
fn: (r, accumulator) => r.value + accumulator,
i: 0
)

69. records = [
{name: "foo", value: 23},
{name: "bar", value: 23},
{name: "asdf", value: 56}
]
// simple loop over each
records
|> map(fn: (r) => {name: r.name, value: r.value + 1})
// compute the sum
sum = records
|> reduce(
fn: (r, accumulator) => r.value + accumulator,
i: 0
)
// get matching records
foos = records
|> filter(fn: (r) => r.name == "foo")

70. while(fn: () => {
// do stuff
})

71. while(fn: () => {
// do stuff
})
while = (fn) =>
if fn()
while(fn)

72. // or loop some number of times
loop(fn: (i) => {
// do stuff here
},
times: 10)

73. // or loop some number of times
loop(fn: (i) => {
// do stuff here
},
times: 10)
loop = (fn, times) =>
loopUntil(fn, 0, times)

74. // or loop some number of times
loop(fn: (i) => {
// do stuff here
},
times: 10)
loop = (fn, times) =>
loopUntil(fn, 0, times)
loopUntil = (fn, iteration, times) =>
if iteration < times {
fn(iteration)
loopUntil(fn, iteration + 1, times)
}

75. Syntactic Sugar

76. // <stream object>[<predicate>,<time>:<time>,<list of strings>]

77. // <stream object>[<predicate>,<time>:<time>,<list of strings>]
// and here's an example
from(bucket:"foo")[_measurement == "cpu" and _field == "usage_user",
2018-11-07:2018-11-08,
["_measurement", "_time", "_value", “_field”]]

78. // <stream object>[<predicate>,<time>:<time>,<list of strings>]
// and here's an example
from(bucket:"foo")[_measurement == "cpu" and _field == "usage_user",
2018-11-07:2018-11-08,
["_measurement", "_time", "_value", “_field”]]
from(bucket:"foo")
|> filter(fn: (row) => row._measurement == "cpu" and row._field == "usage_user")
|> range(start: 2018-11-07, stop: 2018-11-08)
|> keep(columns: ["_measurement", "_time", "_value", “_field"])

79. from(bucket:"foo")[_measurement == "cpu"]
// notice the trailing commas can be left off
from(bucket: "foo")
|> filter(fn: (row) => row._measurement == "cpu")
|> last()

80. from(bucket:"foo")["some tag" == "asdf",,]
from(bucket: "foo")
|> filter(fn: (row) => row["some tag"] == "asdf")
|> last()

81. from(bucket:"foo")[foo=="bar",-1h]
from(bucket: "foo")
|> filter(fn: (row) => row.foo == "bar")
|> range(start: -1h)

82. bucket = "foo"
start = -3
from(bucket: bucket)
|> range(start: start, end: -1)
// shortcut if the variable name is the same as the argument
from(bucket)
|> range(start, end: -1)

83. Flux Office Hours Tomorrow

84. InfluxDB 2.0 Status

85. Thank you
Paul Dix
paul@influxdata.com
@pauldix