EPAM tech talks, Sofia, 25th April

1. 1
KDB Database
Martin Toshev
25.04.2015

2. 2
Agenda
KDB Database Overview
Q Lang Basics

3. 3
Agenda
KDB vs classical RDBMS
KDB Java API
KDB Use Cases

4. 4
KDB Database Overview

5. 5
KDB database overview
KDB:
• an in-memory column-based database
• can be used to handle large volumes of real-time or historical data
• provides OLTP and OLAP capabilities

6. 6
KDB database overview
KDB:
• provides much of the same functionality of an RDBMS
• supports SQL and KSql (for column-based queries)

7. 7
KDB database overview
Used by almost any large financial institution …
• Goldman Sachs
• Morgan Stanley
• J.P. Morgan
• Deutsche Bank
• Commerzbank AG

8. 8
KDB database overview
KDB+:
• 64bit release of KDB
• OLTP: 100 000 – 1 000 000 records per second per CPU
• OLAP: 1 000 000 – 10 000 000 records per second per CPU
• Backed by the Q general purpose programming language

9. 9
KDB database overview
KDB+:
• multi-threaded and multi-process (scales vertically)
• can be deployed in a distributed environment (scales horizontally)
• provides multiple APIs for C++, Java, Perl, Python …
• and provides support for JDBC …

10. 10
Q Lang Basics

11. 11
Q Lang Basics
Q language:
• vector processing language
• allows for complex calculations across a large volume of data
• merges the K and ksql languages

12. 12
Q Lang Basics
Q language:
• Interpreted (a mixture of APL and Scheme)
• simple rules of precedence (right -> left evaluation of all expressions)
• functions are first-class citizens (can be assigned to variables)

13. 13
Q Lang Basics
Q language data types:
• Integer
• floating point
• char
• symbol
• lists
• dictionaries
• time
• datetime
• date
• others …

14. 14
Q Lang
Demo

15. 15
KDB vs classical RDBMS

16. 16
KDB vs classical RDBMS
KDB is column-oriented while RDBMS are row-oriented
Both support SQL for querying the database

17. 17
KDB vs classical RDBMS
Classical RDBMS are in control of the persistence
In KDB+ the user is in control of the persistence

18. 18
KDB vs classical RDBMS
No concept of transactions in KDB+
Queries in KDB+ are always SERIALIZABLE in terms of
isolation levels

19. 19
KDB vs classical RDBMS
KDB provides utilities for dealing with time-series data:
• time joins (af – asof join, wj – window join)

20. 20
KDB Java API

21. 21
KDB Java API
Extremely simple …
A single class c that represents a KDB+ connection

22. 22
KDB Java API
c.ks() – for executing an operation without waiting for a
response
c.k() – for execution an operation and waiting for a response
c.k() – for waiting for a message from the KDB+ server
(naming is somewhat dummy … )

23. 23
KDB Java API
// getting the 'tbl' table
c connection = new c("localhost", 5000);
Flip tbl = (Flip) connection.k("tbl");
for(String key : tbl.x) {
System.out.println(key + " " +
tbl.at(key).getClass().getSimpleName());
}

24. 24
KDB Java API
// inserting a record in the 'tbl‘ table
connection.ks("`tbl insert (`x;10)");

25. 25
KDB Java API
Demo

26. 26
KDB Use Cases

27. 27
KDB Use Cases
Capturing, storing and analyzing massive volumes of data … (kdb+tick)
Kdb+tick
tickerplant
Market data
feeds
Other feeds
KDB+
log files
Kdb+tick
rdb
Java/C/
Python
Kdb+tick
hdb

28. 28
KDB Use Cases
Analyzing FIX messages for post trade analytics and compliance (kdb+tick)
Kdb+tick
tickerplant
Market data
feeds
Other feeds
KDB+
log files
Kdb+tick
rdb
Java/C/
Python
Kdb+tick
hdb
OMS
(Order Management
System)

29. 29
KDB Use Cases
Importing and analyzing massive volumes of data from an exchange … (kdb+taq)
Kdb+tick
taq
NYSE

30. 30
Thank you

31. 31
Resources
Wikipedia’s entry on KDB
http://en.wikipedia.org/wiki/KDB
KDB wiki
http://code.kx.com/wiki/Main_Page
KDB-tick overview
http://kx.com/papers/Kdb+tickDiagram.pdf
KDB guides
http://www.timestored.com/kdb-guides/

32. 32
Resources
K language
http://www.math.bas.bg/bantchev/place/k.html
KDB for Gods
http://www.firstderivatives.com/lecture_series.asp
KDB+ function reference
http://code.kx.com/wiki/Reference

33. 33
Resources
Kdb+ Database and Language Primer
http://kx.com/q/d/primer.htm
KDB+ Database Reference Manual
http://kx.com/q/d/kdb+1.htm
Q language reference manual
http://kx.com/q/d/q1.htm
Abridged KDB+ Database Manual
http://kx.com/q/d/kdb+.htm
Abridged Q Language Manual
http://www.kx.com/q/d/q.htm

34. 34
Resources
Kdb database tutorial
http://www.thalesians.com/finance/index.php/Knowledge_Base/Databases/Kdb
A shallow introduction to the K language
http://www.kuro5hin.org/?op=displaystory;sid=2002/11/14/22741/791
K as a prototyping language
http://www.cs.nyu.edu/courses/fall02/G22.3033-007/kintro.html