3.4
2020-05-20T05:55:39Z
Clickhouse servers
Clickhouse
Clickhouse
Clickhouse servers
Clickhouse
-
delayed insert queries
0
ch_params[DelayedInserts]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
9
-
data directory size
0
ch_params[DiskUsage]
30s
90d
365d
0
3
B
0
0
0
0
0
Clickhouse
-
distributed connection fail after all retries finished
0
ch_params[DistributedConnectionFailAtAll]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
9
-
distributed connection fail with retry
0
ch_params[DistributedConnectionFailTry]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
9
-
distributed pending files
0
ch_params[DistributedFilesToInsert]
30s
90d
365d
0
3
0
0
0
0
Number of pending files to process for asynchronous insertion into Distributed tables. Number of files for every shard is summed.
0
Clickhouse
-
distributed connections
0
ch_params[DistributedSend]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
-
dns errors available from 20.3
0
ch_params[DNSError]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
9
-
current HTTP connections
0
ch_params[HTTPConnection]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
-
inserted bytes per second
0
ch_params[InsertedBytes]
30s
90d
365d
0
3
B
0
0
0
0
0
Clickhouse
10
-
inserted rows per second
0
ch_params[InsertedRows]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
10
-
processed INSERT queries
0
ch_params[InsertQuery]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
9
-
the longest currently running query time
0
ch_params[LongestRunningQuery]
30s
90d
365d
0
0
s
0
0
0
0
0
Clickhouse
-
max count of parts per partition across all tables
0
ch_params[MaxPartCountForPartition]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
-
memory for background merges
0
ch_params[MemoryTrackingForMerges]
30s
90d
365d
0
3
B
0
0
0
0
Total amount of memory (bytes) allocated for background merges. Included in MemoryTrackingInBackgroundProcessingPool. Note that this value may include a drift when the memory was allocated in a context of background processing pool and freed in other context or vice-versa. This happens naturally due to caches for tables indexes and doesn't indicate memory leaks.
0
Clickhouse
-
memory for backround moves
0
ch_params[MemoryTrackingInBackgroundMoveProcessingPool]
30s
90d
365d
0
3
B
0
0
0
0
Total amount of memory (bytes) allocated in background processing pool (that is dedicated for backround moves). Note that this value may include a drift when the memory was allocated in a context of background processing pool and freed in other context or vice-versa. This happens naturally due to caches for tables indexes and doesn't indicate memory leaks.
0
Clickhouse
-
memory for background merges, mutations and fetches.
0
ch_params[MemoryTrackingInBackgroundProcessingPool]
30s
90d
365d
0
3
B
0
0
0
0
Total amount of memory (bytes) allocated in background processing pool (that is dedicated for backround merges, mutations and fetches). Note that this value may include a drift when the memory was allocated in a context of background processing pool and freed in other context or vice-versa. This happens naturally due to caches for tables indexes and doesn't indicate memory leaks.
0
Clickhouse
-
memory for bookkeeping tasks of Replicated tables.
0
ch_params[MemoryTrackingInBackgroundSchedulePool]
30s
90d
365d
0
3
B
0
0
0
0
Total amount of memory (bytes) allocated in background schedule pool (that is dedicated for bookkeeping tasks of Replicated tables).
0
Clickhouse
-
memory used by queries
0
ch_params[MemoryTracking]
30s
90d
365d
0
3
B
0
0
0
0
0
Clickhouse
-
merged rows per second
0
ch_params[MergedRows]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
10
-
merged uncompressed bytes per second
0
ch_params[MergedUncompressedBytes]
30s
90d
365d
0
3
B
0
0
0
0
0
Clickhouse
10
-
current MySQL connections
0
ch_params[MySQLConnection]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
-
network errors, i.e. DNS resolve
0
ch_params[NetworkErrors]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
9
-
current running queries
0
ch_params[Query]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
-
read bytes per second
0
ch_params[ReadCompressedBytes]
30s
90d
365d
0
3
B
0
0
0
0
0
Clickhouse
10
-
Read-Only Replicas
0
ch_params[ReadonlyReplica]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
-
read, pread, io_getevents, etc. syscalls in fly
0
ch_params[Read]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
-
replica partial shutdown
0
ch_params[ReplicaPartialShutdown]
30s
90d
365d
0
3
0
0
0
0
how many times ReplicatedMergreTree table yield in state when Zookeeper session is expired
0
Clickhouse
9
-
replication lag across all tables
0
ch_params[ReplicasMaxAbsoluteDelay]
30s
90d
365d
0
3
s
0
0
0
0
0
Clickhouse
-
replication tasks in queue
0
ch_params[ReplicasSumQueueSize]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
-
revision
0
ch_params[Revision]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
-
parts read from disk per second
0
ch_params[SelectedParts]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
10
-
processed SELECT queries
0
ch_params[SelectQuery]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
9
-
current TCP connections
0
ch_params[TCPConnection]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
-
clickhouse-server uptime
0
ch_params[Uptime]
30s
90d
365d
0
3
s
0
0
0
0
0
Clickhouse
-
write, pwrite, io_getevents, etc. syscalls in fly
0
ch_params[Write]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
-
Hardware exceptions in communication with ZooKeeper server
0
ch_params[ZooKeeperHardwareExceptions]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
9
-
Other exceptions in communication with ZooKeeper server
0
ch_params[ZooKeeperOtherExceptions]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
9
-
User exceptions in communication with ZooKeeper server
0
ch_params[ZooKeeperUserExceptions]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
9
-
number of watches in zookeeper
0
ch_params[ZooKeeperWatch]
30s
90d
365d
0
3
0
0
0
0
0
Clickhouse
{$MAX_CONCURENT_QUERIES}
100
{$MAX_DELAYED_FILES_TO_DISTRIBUTED_INSERT}
50
{$MAX_HTTP_CONNECTIONS}
100
{$MAX_MYSQL_CONNECTIONS}
100
{$MAX_PARTS_PER_PARTITION}
300
{$MAX_QUERY_TIME}
600
{$MAX_REPLICA_DELAY_DISTRIBUTED_QUERIES}
300
{$MAX_TCP_CONNECTIONS}
1024
{$MIN_INSERTED_ROWS_PER_QUERY}
1000
Clickhouse metrics
2
5
0
450
100
0
0
1
1
0
0
0
0
0
Concurrent running queries
Clickhouse
3
0
450
100
1
0
1
1
0
0
0
0
0
Write / Merge bytes/sec
Clickhouse
3
0
450
100
0
1
1
1
0
0
0
0
0
Finished Queries
Clickhouse
3
0
450
100
1
1
1
1
0
0
0
0
0
Insert / Merge rows/sec
Clickhouse
3
0
500
100
0
2
1
1
0
0
0
0
0
Memory Usage
Clickhouse
3
0
500
100
1
2
1
1
0
0
0
0
0
Database size
Clickhouse
3
0
450
100
0
3
1
1
0
0
0
0
0
Connections
Clickhouse
3
0
450
100
1
3
1
1
0
0
0
0
0
Distributed
Clickhouse
3
0
450
100
0
4
1
1
0
0
0
0
0
Replication
Clickhouse
3
0
450
100
1
4
1
1
0
0
0
0
0
Zookeeper
Clickhouse
3
{Clickhouse:ch_params[Uptime].nodata(3m)}
0
{HOST.HOST} clickhouse-server monitoring have no data, possible clickhouse-server is down, check `systemd status clickhouse-server` and check zbx_clickhouse_monitor.sh and `systemd status zabbix-agent`
0
0
4
0
1
{Clickhouse:ch_params[Uptime].last()} <= 600
0
{HOST.HOST} clickhouse-server recently restarted
0
0
2
0
1
{Clickhouse:ch_params[DNSError].last()}>0 or {Clickhouse:ch_params[NetworkErrors].last()}>0
0
{HOST.HOST} clickhouse DNS errors occurred
0
0
2
Please check DNS settings and remote_servers part of /etc/clickhouse-server/
https://clickhouse.tech/docs/en/operations/server-configuration-parameters/settings/#server-settings-remote-servers
https://clickhouse.tech/docs/en/operations/server-configuration-parameters/settings/#server-settings-disable-internal-dns-cache
https://clickhouse.tech/docs/en/query_language/system/#query_language-system-drop-dns-cache
0
1
{Clickhouse:ch_params[Revision].change()}=1
0
{HOST.HOST} clickhouse version changed
0
0
2
0
0
{Clickhouse:ch_params[DistributedConnectionFailAtAll].last()}>=0
or
{Clickhouse:ch_params[DistributedConnectionFailTry].last()}>=0
or
{Clickhouse:ch_params[DistributedFilesToInsert].last()}>={$MAX_DELAYED_FILES_TO_DISTRIBUTED_INSERT}
0
{HOST.HOST} distributed connection exceptions occurred
0
https://clickhouse.tech/docs/en/operations/table_engines/distributed/
0
3
please check communications between Clickhouse servers and <remote_servers> in config.xml
https://clickhouse.tech/docs/en/operations/table_engines/distributed/
https://clickhouse.tech/docs/en/sql-reference/statements/system/#query-language-system-distributed
https://clickhouse.tech/docs/en/operations/server-configuration-parameters/settings/#server-settings-remote-servers
When you insert data to distributed table.
Data is written to target *MergreTree tables asynchronously. When inserted in the table, the data block is just written to the local file system. The data is sent to the remote servers in the background as soon as possible. The period for sending data is managed by the distributed_directory_monitor_sleep_time_ms and distributed_directory_monitor_max_sleep_time_ms settings. The Distributed engine sends each file with inserted data separately, but you can enable batch sending of files with the distributed_directory_monitor_batch_inserts setting.
0
0
{Clickhouse:ch_params[DelayedInserts].last()} > 0
0
{HOST.HOST} have INSERT queries that are throttled due to high number of active data parts for partition in a MergeTree, please decrease INSERT frequency
0
https://clickhouse.tech/docs/en/development/architecture/#merge-tree
0
5
INSERT queries that are throttled due to high number of active data parts for partition in a MergeTree table.
0
0
{Clickhouse:ch_params[LongestRunningQuery].last()} >= {$MAX_QUERY_TIME}
0
{HOST.HOST} have queries which running more than {$MAX_QUERY_TIME} sec
0
0
2
0
1
{Clickhouse:ch_params[ReadonlyReplica].last()} > 0
0
{HOST.HOST} have read-only replicated tables, check Zookeeper state
0
https://clickhouse.tech/docs/en/operations/table_engines/replication/#recovery-after-failures
0
5
Number of Replicated tables that are currently in readonly state due to re-initialization after ZooKeeper session loss or due to startup without ZooKeeper configured.
0
0
{Clickhouse:ch_params[ReplicasMaxAbsoluteDelay].last(3m)} >= {$MAX_REPLICA_DELAY_DISTRIBUTED_QUERIES}
0
{HOST.HOST} have replication lag more {$MAX_REPLICA_DELAY_DISTRIBUTED_QUERIES} sec
0
https://clickhouse.tech/docs/en/operations/settings/settings/#settings-max_replica_delay_for_distributed_queries
0
4
When replica have too much lag, it can be skipped from Distributed SELECT Queries without errors and you will have wrong query results
Check disks and networks on monitored ClickHouse servers
0
1
{Clickhouse:ch_params[HTTPConnection].last()} >= {$MAX_HTTP_CONNECTIONS}
0
{HOST.HOST} HTTP connections >= {$MAX_HTTP_CONNECTIONS}
0
https://clickhouse.tech/docs/en/operations/server_settings/settings/#max-concurrent-queries
0
2
The clickhouse is adapted to run not a very large number of parallel requests, not every HTTP connection means a running sql request, but a large number of open tcp connections can cause a spike in sudden sql requests, resulting in performance degradation.
0
0
{Clickhouse:ch_params[MaxPartCountForPartition].last()} >= {$MAX_PARTS_PER_PARTITION} * 0.9
0
{HOST.HOST} MergeTree parts 90% of {$MAX_PARTS_PER_PARTITION}, please decrease INSERT queries frequency
0
0
4
Clickhouse MergeTree table engine split each INSERT query to partitions (PARTITION BY expression) and add one or more PARTS per INSERT inside each partition, after that background merge process run, and when you have too much unmerged parts inside partition, SELECT queries performance can significate degrade, so clickhouse try delay insert, or abort it
0
0
{Clickhouse:ch_params[MySQLConnection].last()} >= {$MAX_MYSQL_CONNECTIONS}
0
{HOST.HOST} MySQL connections >= {$MAX_MYSQL_CONNECTIONS}
0
https://clickhouse.tech/docs/en/operations/server_settings/settings/#max-concurrent-queries
0
2
The clickhouse is adapted to run not a very large number of parallel requests, not every MySQL connection means a running sql request, but a large number of open tcp connections can cause a spike in sudden sql requests, resulting in performance degradation.
0
0
{Clickhouse:ch_params[InsertQuery].last()}>0
and
( {Clickhouse:ch_params[InsertedRows].last()} / {Clickhouse:ch_params[InsertQuery].last()} ) <= {$MIN_INSERTED_ROWS_PER_QUERY}
0
{HOST.HOST} please increase inserted rows per INSERT query
0
https://clickhouse.tech/docs/en/introduction/performance/#performance-when-inserting-data
0
4
Clickhouse team recommends inserting data in packets of at least 1000 rows or no more than a single request per second.
Please use Buffer table
https://clickhouse.tech/docs/en/operations/table_engines/buffer/
or
https://github.com/nikepan/clickhouse-bulk
or
https://github.com/VKCOM/kittenhouse
0
0
{Clickhouse:ch_params[Query].last(3m)} >= 0.9 * {$MAX_CONCURRENT_QUERIES}
0
{HOST.HOST} running queries 90% of {$MAX_CONCURRENT_QUERIES}
0
https://clickhouse.tech/docs/en/operations/server_settings/settings/#max-concurrent-queries
0
4
Each concurrent SELECT query use memory in JOINs use CPU for running aggregation function and can read lot of data from disk when scan parts in partitions and utilize disk IO.
Each concurrent INSERT query, allocate around 1MB per each column in an inserted table and can utilize disk IO.
Look at following documentation parts
https://clickhouse.tech/docs/en/operations/settings/query_complexity/
https://clickhouse.tech/docs/en/operations/quotas/
0
1
{Clickhouse:ch_params[TCPConnection].last()} >= {$MAX_TCP_CONNECTIONS}
0
{HOST.HOST} TCP connections >= {$MAX_TCP_CONNECTIONS}
0
https://clickhouse.tech/docs/en/operations/server_settings/settings/#max-connections
0
2
The clickhouse is adapted to run not a very large number of parallel requests, not every tcp connection means a running sql request, but a large number of open tcp connections can cause a spike in sudden sql requests, resulting in performance degradation.
0
0
Concurrent running queries
600
200
0.0000
100.0000
0
1
0
1
0
0.0000
0.0000
1
0
0
0
0
0
DDDD00
0
4
0
-
Clickhouse
ch_params[Write]
1
0
00BB00
0
4
0
-
Clickhouse
ch_params[Read]
2
0
BB0000
0
4
0
-
Clickhouse
ch_params[Query]
3
0
A54F10
1
2
0
-
Clickhouse
ch_params[LongestRunningQuery]
Connections
600
200
0.0000
100.0000
1
1
0
1
0
0.0000
0.0000
1
0
0
0
0
0
1A7C11
0
2
0
-
Clickhouse
ch_params[TCPConnection]
1
0
F63100
0
2
0
-
Clickhouse
ch_params[HTTPConnection]
2
0
CCCC00
0
2
0
-
Clickhouse
ch_params[MySQLConnection]
3
0
A54F10
0
2
0
-
Clickhouse
ch_params[DistributedSend]
Database size
600
200
0.0000
100.0000
0
1
0
1
0
0.0000
0.0000
1
0
0
0
0
0
AA0000
0
2
0
-
Clickhouse
ch_params[DiskUsage]
Distributed
900
200
0.0000
100.0000
1
1
0
1
0
0.0000
0.0000
1
0
0
0
0
0
CC0000
0
2
0
-
Clickhouse
ch_params[DistributedConnectionFailAtAll]
1
0
CCCC00
0
2
0
-
Clickhouse
ch_params[DistributedConnectionFailTry]
2
0
00BB00
1
2
0
-
Clickhouse
ch_params[DistributedFilesToInsert]
Finished Queries
600
200
0.0000
100.0000
0
1
1
1
0
0.0000
0.0000
1
0
0
0
0
0
4CAF50
0
2
0
-
Clickhouse
ch_params[SelectQuery]
1
0
DDDD00
0
2
0
-
Clickhouse
ch_params[InsertQuery]
Insert / Merge rows/sec
600
200
0.0000
100.0000
0
1
0
1
0
0.0000
0.0000
1
0
0
0
0
0
DDDD00
0
2
0
-
Clickhouse
ch_params[InsertedRows]
1
0
CC0000
0
2
0
-
Clickhouse
ch_params[MergedRows]
Memory Usage
600
200
0.0000
100.0000
0
1
0
1
0
0.0000
0.0000
1
0
0
0
0
0
F63100
0
2
0
-
Clickhouse
ch_params[MemoryTracking]
1
0
FFFF33
0
2
0
-
Clickhouse
ch_params[MemoryTrackingForMerges]
2
0
AAAA00
0
2
0
-
Clickhouse
ch_params[MemoryTrackingInBackgroundProcessingPool]
3
0
000099
0
2
0
-
Clickhouse
ch_params[MemoryTrackingInBackgroundMoveProcessingPool]
4
0
00DDDD
0
2
0
-
Clickhouse
ch_params[MemoryTrackingInBackgroundSchedulePool]
Replication
600
200
0.0000
100.0000
1
1
0
1
0
0.0000
0.0000
1
0
0
0
0
0
EE0000
0
2
0
-
Clickhouse
ch_params[ReadonlyReplica]
1
0
DDDD00
0
2
0
-
Clickhouse
ch_params[ReplicaPartialShutdown]
2
0
2774A4
1
2
0
-
Clickhouse
ch_params[ReplicasMaxAbsoluteDelay]
3
0
A54F10
0
2
0
-
Clickhouse
ch_params[ReplicasSumQueueSize]
Write / Merge bytes/sec
600
200
0.0000
100.0000
0
1
0
1
0
0.0000
0.0000
1
0
0
0
0
0
CCCC00
0
2
0
-
Clickhouse
ch_params[InsertedBytes]
1
0
BB0000
0
2
0
-
Clickhouse
ch_params[MergedUncompressedBytes]
Zookeeper
600
200
0.0000
100.0000
0
1
0
1
0
0.0000
0.0000
1
0
0
0
0
0
FF3333
0
2
0
-
Clickhouse
ch_params[ZooKeeperHardwareExceptions]
1
0
2774A4
1
2
0
-
Clickhouse
ch_params[ZooKeeperWatch]
2
0
CC0000
0
2
0
-
Clickhouse
ch_params[ZooKeeperOtherExceptions]
3
0
CC0000
0
2
0
-
Clickhouse
ch_params[ZooKeeperUserExceptions]