From webinar on December 3, 2019 New users of ClickHouse love the speed but may run into a few surprises when designing applications. Column storage turns classic SQL design precepts on their heads. This talk shares our favorite tricks for building great applications. We'll talk about fact tables and dimensions, materialized views, codecs, arrays, and skip indexes, to name a few of our favorites. We'll show examples of each and also reserve time to handle questions. Join us to take your next step to ClickHouse guruhood! Speaker Bio: Robert Hodges is CEO of Altinity, which offers enterprise support for ClickHouse. He has over three decades of experience in data management spanning 20 different DBMS types. ClickHouse is his current favorite. ;)

2. Introduction to Presenter
www.altinity.com
Leading software and services
provider for ClickHouse
Major committer and community
sponsor in US and Western Europe
Robert Hodges - Altinity CEO
30+ years on DBMS plus
virtualization and security.
ClickHouse is DBMS #20

3. Introduction to ClickHouse
Understands SQL
Runs on bare metal to cloud
Shared nothing architecture
Uses column storage
Parallel and vectorized execution
Scales to many petabytes
Is Open source (Apache 2.0)
a b c d
a b c d
a b c d
a b c d
And it’s really fast!

4. © 2019, Altinity LTD
Send server log
to client to see
query processing

5. Enabling server logs in clickhouse-client
clickhouse-client --send_logs_level=trace
(OR)
my-host :) SET send_logs_level = 'trace'
my-host :) SELECT 1
my-host :) SET send_logs_level = 'none'
Ask for logs at start
of the session
Ask for logs during
the session

6. It works in Python too!
# Configure logging.
import logging
logging.basicConfig(level=logging.DEBUG)
# Execute a query using log settings.
from clickhouse_driver import Client
client = Client('localhost')
settings = {'send_logs_level': 'trace'}
result = client.execute('SELECT 1', settings=settings)
print("RESULT: {0}".format(result))
Sends logs
to stdout

7. © 2019, Altinity LTD
Use encodings
to reduce data
size

8. © 2019, Altinity LTD
CREATE TABLE test_codecs ( a String,
a_lc LowCardinality(String) DEFAULT a,
b UInt32,
b_delta UInt32 DEFAULT b Codec(Delta),
b_delta_lz4 UInt32 DEFAULT b Codec(Delta, LZ4),
b_dd UInt32 DEFAULT b Codec(DoubleDelta),
b_dd_lz4 UInt32 DEFAULT b Codec(DoubleDelta, LZ4)
)
Engine = MergeTree
PARTITION BY tuple() ORDER BY tuple();
Applying encodings to table columns
Values with
dictionary
encoding
Differences
between
values
Differences
between change
of value

9. © 2019, Altinity LTD
INSERT INTO test_codecs (a, b)
SELECT
concat('a string prefix',
toString(rand() % 1000)),
now() + (number * 10)
FROM system.numbers
LIMIT 100000000
SETTINGS max_block_size=1000000
Load lots of data

10. © 2019, Altinity LTD
SELECT name, sum(data_compressed_bytes) comp,
sum(data_uncompressed_bytes) uncomp,
round(comp / uncomp * 100.0, 2) AS percent
FROM system.columns WHERE table = 'test_codecs'
GROUP BY name ORDER BY name
┌─name────────┬──────comp─┬─────uncomp─┬─percent─┐
│ a │ 393712480 │ 1888998884 │ 20.84 │
│ a_lc │ 201150993 │ 200431356 │ 100.36 │
│ b │ 401727287 │ 400000000 │ 100.43 │
│ b_delta │ 400164862 │ 400000000 │ 100.04 │
│ b_delta_lz4 │ 1971602 │ 400000000 │ 0.49 │
│ b_dd │ 12738212 │ 400000000 │ 3.18 │
│ b_dd_lz4 │ 476375 │ 400000000 │ 0.12 │
└─────────────┴───────────┴────────────┴─────────┘
Check data sizes
LowCardinality
total compression
is a_lc comp / a
uncomp = 10.65%

11. © 2019, Altinity LTD
SELECT a AS a, count(*) AS c FROM test_codecs
GROUP BY a ORDER BY c ASC LIMIT 10
. . .
10 rows in set. Elapsed: 0.681 sec. Processed 100.00 million
rows, 2.69 GB (146.81 million rows/s., 3.95 GB/s.)
SELECT a_lc AS a, count(*) AS c FROM test_codecs
GROUP BY a ORDER BY c ASC LIMIT 10
. . .
10 rows in set. Elapsed: 0.148 sec. Processed 100.00 million
rows, 241.16 MB (675.55 million rows/s., 1.63 GB/s.)
But wait, there’s more! Encodings help query speed
Faster

12. © 2019, Altinity LTD
Quick Comparison of Encodings
Name Best for
LowCardinality Strings with fewer than 10K values
Delta Time series
Double Delta Increasing counters
Gorilla Gauge data (bounces around mean)
T64 Integers other than random hashes
Compression may vary across ZSTD and LZ4

13. © 2019, Altinity LTD
Use
materialized
views to find
last point data

14. © 2019, Altinity LTD
Last point problems are common in time series
Host
7023
Host
6522
CPU
Utilization
Host
9601
CPU
Utilization
CPU
UtilizationCPU
Utilization
CPU
UtilizationCPU
Utilization
CPU Table
Problem: Show the
current CPU utilization for
each host
CPU
Utilization
CPU
Utilization

15. © 2019, Altinity LTD
argMaxState links columns with aggregates
CREATE MATERIALIZED VIEW cpu_last_point_idle_mv
ENGINE = AggregatingMergeTree()
PARTITION BY tuple()
ORDER BY tags_id
POPULATE
AS SELECT
argMaxState(created_date, created_at) AS created_date,
maxState(created_at) AS max_created_at,
argMaxState(time, created_at) AS time,
tags_id,
argMaxState(usage_idle, created_at) AS usage_idle
FROM cpu
GROUP BY tags_id
Matching
data
Max value
Don’t partition

16. © 2019, Altinity LTD
Let’s hide the merge details with a view
CREATE VIEW cpu_last_point_idle_v AS
SELECT
argMaxMerge(created_date) AS created_date,
maxMerge(max_created_at) AS created_at,
argMaxMerge(time) AS time,
tags_id,
argMaxMerge(usage_idle) AS usage_idle
FROM cpu_last_point_idle_mv
GROUP BY tags_id
Merge functions roll up
partial aggregate data

17. © 2019, Altinity LTD
...Select from the covering view
SELECT
tags_id,
100 - usage_idle usage
FROM cpu_last_point_idle_v
ORDER BY usage DESC, tags_id ASC
LIMIT 10
...
10 rows in set. Elapsed: 0.005 sec. Processed 14.00
thousand rows, 391.65 KB (2.97 million rows/s., 82.97
MB/s.)

18. © 2019, Altinity LTD
Use arrays to
store key-value
pairs

19. © 2019, Altinity LTD
CREATE TABLE cpu (
created_date Date DEFAULT today(),
created_at DateTime DEFAULT now(),
time String,
tags_id UInt32,
usage_user Float64,
usage_system Float64,
. . .
additional_tags String DEFAULT '')
ENGINE = MergeTree()
PARTITION BY created_date
ORDER BY (tags_id, created_at)
SQL tables typically have a tabular form

20. © 2019, Altinity LTD
CREATE TABLE cpu_dynamic (
created_date Date,
created_at DateTime,
time String,
tags_id UInt32,
metrics_name Array(String),
metrics_value Array(Float64)
)
ENGINE = MergeTree()
PARTITION BY created_date
ORDER BY (tags_id, created_at)
Paired arrays allow flexible values
Measurement
names
Floating point
values

21. © 2019, Altinity LTD
clickhouse-client --database default
--query="INSERT INTO cpu_dynamic FORMAT JSONEachRow"
<<DATA
{"created_date":"2016-01-03",
"created_at":"2016-01-03 00:00:00",
"time":"2016-01-03 00:00:00 +0000",
"tags_id":6220,
"metrics_name":["usage_user","usage_system",
"usage_idle","usage_nice"],
"metrics_value":[35,47,77,21]}
. . .
DATA
Insert some values from JSON

22. © 2019, Altinity LTD
SELECT time, tags_id, metrics_name,metrics_value
FROM cpu_dynamic
Row 1:
──────
time: 2016-01-03 00:00:00 +0000
tags_id: 6220
metrics_name:
['usage_user','usage_system','usage_idle','usage_nice']
metrics_value: [35,47,77,21]
. . .
Now each row can have different key/value pairs

23. © 2019, Altinity LTD
SELECT created_at, tags_id, name, value
FROM cpu_dynamic
ARRAY JOIN metrics_name AS name, metrics_value AS value
┌──────────created_at─┬─tags_id─┬─name─────────┬─value─┐
│ 2016-01-03 00:00:00 │ 6220 │ usage_user │ 35 │
│ 2016-01-03 00:00:00 │ 6220 │ usage_system │ 47 │
│ 2016-01-03 00:00:00 │ 6220 │ usage_idle │ 77 │
│ 2016-01-03 00:00:00 │ 6220 │ usage_nice │ 21 │
│ 2016-01-03 00:00:10 │ 6220 │ usage_nice │ 21 │
│ 2016-01-03 00:00:10 │ 6220 │ usage_iowait │ 84 │
│ 2016-01-03 00:00:10 │ 6220 │ usage_irq │ 22 │
You can pivot back to a tabular form with ARRAY JOIN
Correlated arrays

24. © 2019, Altinity LTD
Use materialized
columns to pre-
compute values

25. © 2019, Altinity LTD
ALTER TABLE cpu_dynamic ADD COLUMN usage_user MATERIALIZED
metrics_value[indexOf(metrics_name, 'usage_user')]
AFTER tags_id
SELECT time, tags_id, usage_user
FROM cpu_dynamic
┌─time──────────────────────┬─tags_id─┬─usage_user─┐
│ 2016-01-03 00:00:00 +0000 │ 6220 │ 35 │
│ 2016-01-03 00:00:10 +0000 │ 6220 │ 0 │
└───────────────────────────┴─────────┴────────────┘
You can create new columns using MATERIALIZED
Computed for
old data
Materialized
for new data

26. © 2019, Altinity LTD
Use dictionaries
instead of joins
for dimensions

27. © 2019, Altinity LTD
Star schemas are common in data warehouse
CPU Table
Tags TableOrgs Table
Budgets Table Projects Table
Facts
Dimension

28. © 2019, Altinity LTD
SELECT tags.rack rack, avg(100 - cpu.usage_idle) usage
FROM cpu
INNER JOIN tags AS t ON cpu.tags_id = t.id
GROUP BY rack
ORDER BY usage DESC
LIMIT 10
Matching data
Joining facts and dimensions is not always convenient
Need a join for
every dimension
Dimension values are often mutable

29. © 2019, Altinity LTD
/etc/clickhouse-server/tags_dictionary.xml:
<yandex><dictionary>
<name>tags</name>
<source> <mysql>
<host>localhost</host> <port>3306</port>
<user>root</user> <password>********</password>
<db>tsbs</db> <table>tags</table>
</mysql> </source>
<layout> <hashed/> </layout>
<structure>
<id> <name>id</name> </id>
<attribute>
<name>hostname</name><type>String</type>
<null_value></null_value>
</attribute> . . .
Dictionaries are an alternative to joins

30. © 2019, Altinity LTD
SELECT
dictGetString('tags', 'rack', toUInt64(cpu.tags_id)) rack,
avg(100 - cpu.usage_idle) usage
FROM cpu
GROUP BY rack
ORDER BY usage DESC
LIMIT 10
Now we can avoid costly joins
Dictionary key must
be UInt64
Hash table stored in memory

31. © 2019, Altinity LTD
External dictionaries live in another location
CPU Table
Tags Table
Orgs Table Budgets Table
Projects Table
Small, mutable dimensionsLarge, immutable facts
ClickHouse
MySQL or PostgreSQL

32. © 2019, Altinity LTD
Use MySQL
Database Engine
instead of
dictionaries

33. © 2019, Altinity LTD
Access a MySQL database from ClickHouse
CREATE DATABASE mysql_repl
ENGINE=MySQL(
'127.0.0.1:3306',
'repl',
'root',
'secret')
use mysql_repl
show tables
Database engine

34. © 2019, Altinity LTD
Selecting data from MySQL
SELECT
t.datetime, t.date, t.request_id,
t.name customer, s.name sku
FROM (
SELECT t.* FROM traffic t
JOIN customer c ON t.customer_id = c.id) AS t
JOIN mysql_repl.sku s ON t.sku_id = s.id
WHERE customer_id = 5
ORDER BY t.request_id LIMIT 10
Predicate pushed
down to MySQL

35. © 2019, Altinity LTD
Use TTLs to
delete obsolete
data

36. © 2019, Altinity LTD
ClickHouse is not optimized for deletes
CREATE TABLE traffic (
datetime DateTime,
date Date,
request_id UInt64,
cust_id UInt32,
sku UInt32
) ENGINE = MergeTree
PARTITION BY toYYYYMM(datetime)
ORDER BY (cust_id, date)

37. © 2019, Altinity LTD
Until 2019 there were two options
ALTER TABLE traffic DROP PARTITION 201801
ALTER TABLE traffic DELETE WHERE
datetime < toDateTime('2018-02-01 00:00:00')
Drop entire
partition; very fast
Drop matching values
asynchronously

38. © 2019, Altinity LTD
You can delete automatically with a TTL
CREATE TABLE traffic (
datetime DateTime,
date Date,
request_id UInt64,
cust_id UInt32,
sku UInt32
) ENGINE = MergeTree
PARTITION BY toYYYYMM(datetime)
ORDER BY (cust_id, date)
TTL datetime + INTERVAL 90 DAY
Drop rows after
90 days
TTL application set by
merge_with_ttl_timeout
in merge_tree_settings

39. © 2019, Altinity LTD
TTL value can be variable based on data
CREATE TABLE traffic_with_ttl_variable (
datetime DateTime,
date Date,
retention_days UInt16,
request_id UInt64,
cust_id UInt32,
sku UInt32
) ENGINE = MergeTree
PARTITION BY toYYYYMM(datetime)
ORDER BY (cust_id, date)
TTL date + INTERVAL (retention_days * 2) DAY
Custom
retention for
each row

40. © 2019, Altinity LTD
CREATE TABLE traffic_with_ttl_variable (
datetime DateTime,
date Date,
retention_days UInt16,
request_id UInt64,
cust_id UInt32,
sku UInt32
) ENGINE = MergeTree
PARTITION BY toYYYYMM(datetime)
ORDER BY (cust_id, date)
TTL date + INTERVAL 7 DAY TO DISK 'ssd',
date + INTERVAL 30 DAY TO DISK 'hdd',
date + INTERVAL 180 DAY DELETE
Prepare your brain for more cool TTL uses!!
TTL-based
migration
across tiered
storage

41. © 2019, Altinity LTD
Use replication
instead of
backups

42. © 2019, Altinity LTD
Replication works on a per-table basis
Node 1
Table: events
Part: 201801_1_1_0
Part: 201801_2_2_0
Part: 201801_1_2_1
Asynchronous multi-master replication
Node 2
Table: events
Part: 201801_1_1_0
Part: 201801_2_2_0
Part: 201801_1_2_1
INSERT
INSERT
Merge Merge
Replicate
Replicate
DDL +
TRUNCATE

43. © 2019, Altinity LTD
Clickhouse and Zookeeper implementation
INSERT
Replicate
ClickHouse Node 1
Table: events
(Parts)
ReplicatedMergeTree
:9009
:9000 ClickHouse Node 2
Table: events
(Parts)
ReplicatedMergeTree
:9009
:9000
zookeeper-1
ZNodes
:2181 zookeeper-2
ZNodes
:2181 zookeeper-3
ZNodes
:2181

44. © 2019, Altinity LTD
Clusters define sharding and replication layouts
/etc/clickhouse-server/config.d/remote_servers.xml:
<yandex>
<remote_servers>
<Replicated>
<shard>
<replica><host>10.0.0.71</host><port>9000</port></replica>
<replica><host>10.0.0.72</host><port>9000</port></replica>
<internal_replication>true</internal_replication>
</shard>
<Replicated>
</remote_servers>
</yandex>
Use ZK-based internal
replication; otherwise,
distributed table sends
INSERTS to all replicas

45. © 2019, Altinity LTD
Macros enable consistent DDL over a cluster
/etc/clickhouse-server/config.d/macros.xml:
<yandex>
<macros>
<cluster>Replicated</cluster>
<shard>01</shard>
<replica>01</replica>
</macros>
</yandex>

46. © 2019, Altinity LTD
Zookeeper tag defines servers and task queue
/etc/clickhouse-server/config.d/zookeeper.xml:
<yandex>
<zookeeper>
<node><host>10.0.0.61</host><port>2181</port></node>
<node><host>10.0.0.62</host><port>2181</port></node>
<node><host>10.0.0.63</host><port>2181</port></node>
</zookeeper>
<distributed_ddl>
<path>/clickhouse/ShardedAndReplicated/task_queue/ddl</path>
</distributed_ddl>
</yandex>
Clickhouse restart required after
Zookeeper config changes

47. © 2019, Altinity LTD
Create tables!
CREATE TABLE events ON CLUSTER '{cluster}' (
EventDate DateTime,
CounterID UInt32,
UserID UInt32)
ENGINE =
ReplicatedMergeTree('/clickhouse/{cluster}/test/tables/events/{
shard}', '{replica}')
PARTITION BY toYYYYMM(EventDate)
ORDER BY (CounterID, EventDate, intHash32(UserID))

48. Thank you!
Special Offer:
Contact us for a
1-hour consultation!
Presenter:
rhodges@altinity.com
Visit us at:
https://www.altinity.com
Free Consultation:
https://blog.altinity.com/offer