How InfluxDB Enables NodeSource to Run Extreme Levels of Node.js Processes

1. Product, Services, Support

2. mnedelko
VP Product
Hi!
nw
Principal Solutions Architect

3. WHAT WE DO

4. © 2020 NodeSource Confidential4
3 4 , 6 7 5 , 5 0 6
D O W N L O A D S
Q1 2020
Principal Distributor of Node.js on Linux

5. © 2020 NodeSource Confidential5
Our two strongest value proposition/ point of differentiation is our expertise in the Node.js
ecosystem and our ability to translate performance data into a product that makes it:
• Accessible
• Interpretable
• Actionable
• Does so in production
Product, Services, Support Product, Services, Support Product, Services, Support
Product, Services, Support Product, Services, SupportProduct, Services, Support
VALUE PROPOSITION

6. 6
PRODUCTS | SUPPORT | SERVICES |
CERTIFICATION
Consulting
24/7 Support
Certification

7. © 2020 NodeSource Confidential7
Enterprise Node.js runtime
Deep metrics with negligible overhead
Perf & Sec Insights - No Instrumentation

8. © 2020 NodeSource Confidential8
HOW IT WORKS
NODE.JS
BINDINGS
Timing
Information
Module
Tracking
Expose Internals Relating to
I/O
Redacted
Snapshots
Event Loop
Metrics
POLICIES
Runtime Console Database

9. © 2020 NodeSource Confidential9
• On Premise Solution
• Drop-in replacement for the Node.js runtime
• Realtime metrics
• Visibility into lifecycle events and poor performance
NODE.JS
BINDINGS
Timing
Information
Module
Tracking
Expose Internals Relating to
I/O
Redacted
Snapshots
Event Loop
Metrics
POLICIES

10. © 2020 NodeSource Confidential10
DIAGNOSTICS SECURITYANALYTICS

11. The Need for a Solution

12. © 2020 NodeSource Confidential12
The NSolid Console is the centralized system for communication with and remote
control of applications running on the NSolid Runtime.
• Custom alerts
• Visual Heap Snapshots & CPU Profile
• Live Security Screening in Prod
• Console API + Stats D
Present
Future
• Vulnerability Exposure
• Memory Leak Detection
• Async tracing

13. Up to 3000 Processes Up to 9000 Processes
[screenshot of 9k processes]
InfluxDB InfluxDB Enterprise

14. The Decision Process

15. © 2020 NodeSource Confidential
Decisions
15
We needed a centralized analytics hub for processes.
!

16. © 2020 NodeSource Confidential
History
16
Launched Late 2013InfluxDB
WORK STARTED IN EARLY 2014N|Solid
We sorta drank the kool-aid….
Initial implementationetcd

17. © 2020 NodeSource Confidential
etcd
17
• High availability
• Trusted
• Cloud native
Pros
• Prioritizing HA so earlier
• limited - key/val store
• Feel too hard for container hype
Cons

18. © 2020 NodeSource Confidential18
• Work to migrate to InfluxDB happened soon after initial release
• Designed for heavy writes
• Complex and feature rich query capabilities
• Robust configuration - retention / indexes
• Self contained - single binary

19. In Action

20. © 2020 NodeSource Confidential
Integration
20

21. © 2020 NodeSource Confidential21
Short Demo

22. The Implementation

23. © 2020 NodeSource Confidential23
Unique Challenges
NodeSource embeds InfluxDB as part of a product offering. With
distributions to multiple environments along with 24/7 support for customer
offered a unique set of issues.

24. © 2020 NodeSource Confidential
Logging
24
• Verbosity
• log rotation
• considered /dev/null for some scenarios
• `INFLUXDB_HTTP_LOG_ENABLED: false`

25. © 2020 NodeSource Confidential
Performance & Availability
25
• Custom implementation of buffering writes internally
• Write batch sizes grew massive under load
• HTTP keep-alive support helped
• Provided relay features - slave DBs (more on this later)
• Fine tuning retention policies for use case
• “live” - 1 week

26. © 2020 NodeSource Confidential26
• Considered UDP for a split second
• Utilized multiple sockets to InfluxDB
• Removed / adjusted delays
• Removed record order insertion constraints
• Preventing DNS lookup
• INFLUXDB_HTTP_UNIX_SOCKET_ENABLED
• INFLUXDB_HTTP_UNIX_BIND_SOCKET
Performance & Availability

27. Challenges with InfluxDB

28. © 2020 NodeSource Confidential
Data Integrity
28
• Enforcement and careful planning of data shape
• Properly escaping
• comma-delimited string converted to array
• info records written with a unique sub millisecond timestamp to
ensure not data collisions
• Understanding cardinality for your data and plan around it

29. © 2020 NodeSource Confidential
Supporting Multiple Platforms
29
• Detect platform and settings (and set as needed before starting
InfluxDB)
• `kern.ipc-maxsockbuf` for example
• limiting ourselves to MTU sizes
• Different characteristics and bugs. Alpine broke for us a long time
ago. Mostly minor but added overhead

30. © 2020 NodeSource Confidential
Queries can be dangerous - protect yourself
30
• Pagination
• More filters
• Validation or restriction on queries to expensive indices
!

31. © 2020 NodeSource Confidential
Growing Pains
31
Schema Versioning

32. Strengths of InfluxDB

33. © 2020 NodeSource Confidential
Continuous Queries FTW
33
• Allows us to down sample old data
• Aggregation
• Realtime dashboards and integrations

34. © 2020 NodeSource Confidential
Made for Writing
34
• Fast
• Scalable
• Aggregation

35. © 2020 NodeSource Confidential
Testing and Debugging
35
• Easy to replay data
• Logs and tools are great if used wisely

36. Try it Free on
NODESOURCE.com

37. THANK YOU

38. November 10 - 11, 2020
North America Virtual Experience
www.influxdays.com/virtual-experience- /
Call for Papers is now open!
We’re looking for great speakers – submit
your speaker application today.