zabbix_export: version: '6.0' date: '2021-11-21T22:05:13Z' groups: - uuid: a571c0d144b14fd4a87a9d9b2aa9fcd6 name: Templates/Applications templates: - uuid: f4962378af634c5785f463c9a14b9709 template: 'Elasticsearch Cluster by HTTP zbx 4.2' name: 'Elasticsearch Cluster by HTTP zbx 4.2' description: | ## Description The template to monitor Elasticsearch by Zabbix that work without any external scripts. It works with both standalone and cluster instances. The metrics are collected in one pass remotely using an HTTP agent. They are getting values from REST API _cluster/health, _cluster/stats, _nodes/stats requests. You can set {$ELASTICSEARCH.USERNAME} and {$ELASTICSEARCH.PASSWORD} macros in the template for using on the host level. If you use an atypical location ES API, don't forget to change the macros {$ELASTICSEARCH.SCHEME},{$ELASTICSEARCH.PORT}. Parsed manually from https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/templates/app/elasticsearch_http from zabbix version 5.0 to 4.2 and add triggers ## Overview This is fork of official [Zabbix template Elasticsearch Cluster by HTTP (Zabbix 5.0)](https://www.zabbix.com/integrations/elasticsearch) Features: * Compatible with Zabbix 4.2 + * Added trigger of Maximum allocated shards on ES cluster Be default in ES max shards per node is 1000 ({$ELASTICSEARCH.MAX\_SHARDS\_PER\_NODE}) if number of shrads is maximum then elasticshearch will be sillently reject write requests. Tested on Zabbix 4.2 and Elasticsearch 7.4.2 ## Author R N groups: - name: Templates/Applications items: - uuid: 035c2398b4de491d90d6decb2bc981f0 name: 'ES: Delayed unassigned shards' type: DEPENDENT key: es.cluster.delayed_unassigned_shards delay: '0' history: 7d description: 'The number of shards whose allocation has been delayed by the timeout settings.' preprocessing: - type: JSONPATH parameters: - $.delayed_unassigned_shards master_item: key: es.cluster.get_health tags: - tag: Application value: 'ES cluster' - uuid: 1f94eb78a7544b41a2e7ceb4be80c821 name: 'ES: Get cluster health' type: HTTP_AGENT key: es.cluster.get_health history: '0' trends: '0' value_type: TEXT authtype: BASIC username: '{$ELASTICSEARCH.USERNAME}' password: '{$ELASTICSEARCH.PASSWORD}' description: 'Returns the health status of a cluster.' timeout: 15s url: '{$ELASTICSEARCH.SCHEME}://{HOST.CONN}:{$ELASTICSEARCH.PORT}/_cluster/health?timeout=5s' tags: - tag: Application value: 'Zabbix raw items' - uuid: a6f2806487ae4cb287146d178694c8c4 name: 'ES: Get cluster stats' type: HTTP_AGENT key: es.cluster.get_stats history: '0' trends: '0' value_type: TEXT authtype: BASIC username: '{$ELASTICSEARCH.USERNAME}' password: '{$ELASTICSEARCH.PASSWORD}' description: 'Returns cluster statistics.' timeout: 15s url: '{$ELASTICSEARCH.SCHEME}://{HOST.CONN}:{$ELASTICSEARCH.PORT}/_cluster/stats' tags: - tag: Application value: 'Zabbix raw items' - uuid: 86fd982df1b648b8bb380fb595247f09 name: 'ES: Get cluster version' type: HTTP_AGENT key: es.cluster.get_version delay: 1h history: '0' status: DISABLED value_type: FLOAT authtype: BASIC username: '{$ELASTICSEARCH.USERNAME}' password: '{$ELASTICSEARCH.PASSWORD}' description: 'Returns the cluster version' preprocessing: - type: JSONPATH parameters: - $.version.number - type: REGEX parameters: - \d.. - \0 timeout: 15s url: '{$ELASTICSEARCH.SCHEME}://{HOST.CONN}:{$ELASTICSEARCH.PORT}' tags: - tag: Application value: 'ES cluster' - uuid: 8f0a396eb9f9426892215ef370465b53 name: 'ES: Inactive shards percentage' type: DEPENDENT key: es.cluster.inactive_shards_percent_as_number delay: '0' history: 7d units: '%' description: 'The ratio of inactive shards in the cluster expressed as a percentage.' preprocessing: - type: JSONPATH parameters: - $.active_shards_percent_as_number - type: JAVASCRIPT parameters: - 'return (100 - value)' master_item: key: es.cluster.get_health tags: - tag: Application value: 'ES cluster' - uuid: c0314bede3ac45e797b8d538aef717d8 name: 'ES: Number of initializing shards' type: DEPENDENT key: es.cluster.initializing_shards delay: '0' history: 7d description: 'The number of shards that are under initialization.' preprocessing: - type: JSONPATH parameters: - $.initializing_shards master_item: key: es.cluster.get_health tags: - tag: Application value: 'ES cluster' triggers: - uuid: ff472a2f671d43f38b79b5d5d1021abe expression: 'min(/Elasticsearch Cluster by HTTP zbx 4.2/es.cluster.initializing_shards,10m)>0' name: 'ES: Cluster has the initializing shards' priority: AVERAGE description: 'The cluster has the initializing shards longer than 10 minutes.' - uuid: 6bb87d1b34bf46d9a264abd69f2fb1fc name: 'ES: Number of data nodes' type: DEPENDENT key: es.cluster.number_of_data_nodes delay: '0' history: 7d description: 'The number of nodes that are dedicated to data nodes.' preprocessing: - type: JSONPATH parameters: - $.number_of_data_nodes - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.cluster.get_health tags: - tag: Application value: 'ES cluster' - uuid: 3a2d3b8bc9084998a05eea822433abca name: 'ES: Number of nodes' type: DEPENDENT key: es.cluster.number_of_nodes delay: '0' history: 7d description: 'The number of nodes within the cluster.' preprocessing: - type: JSONPATH parameters: - $.number_of_nodes - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.cluster.get_health tags: - tag: Application value: 'ES cluster' triggers: - uuid: f4e44fc90d2f48eebdf85952444ece16 expression: 'change(/Elasticsearch Cluster by HTTP zbx 4.2/es.cluster.number_of_nodes)<0' name: 'ES: The number of nodes within the cluster has decreased' priority: INFO manual_close: 'YES' - uuid: f8db058eb2104a0cabf314c39e0e5c49 expression: 'change(/Elasticsearch Cluster by HTTP zbx 4.2/es.cluster.number_of_nodes)>0' name: 'ES: The number of nodes within the cluster has increased' priority: INFO manual_close: 'YES' - uuid: 284fd743f5824e00bb1f8520b7fd68d4 name: 'ES: Number of pending tasks' type: DEPENDENT key: es.cluster.number_of_pending_tasks delay: '0' history: 7d description: 'The number of cluster-level changes that have not yet been executed.' preprocessing: - type: JSONPATH parameters: - $.number_of_pending_tasks master_item: key: es.cluster.get_health tags: - tag: Application value: 'ES cluster' - uuid: ddde025fe5fd4ef286559ee9971c97f3 name: 'ES: Number of relocating shards' type: DEPENDENT key: es.cluster.relocating_shards delay: '0' history: 7d description: 'The number of shards that are under relocation.' preprocessing: - type: JSONPATH parameters: - $.relocating_shards master_item: key: es.cluster.get_health tags: - tag: Application value: 'ES cluster' - uuid: 057c2c3c0ae8440880a8e337707d8283 name: 'ES: Cluster health status' type: DEPENDENT key: es.cluster.status delay: '0' history: 7d description: | Health status of the cluster, based on the state of its primary and replica shards. Statuses are: green All shards are assigned. yellow All primary shards are assigned, but one or more replica shards are unassigned. If a node in the cluster fails, some data could be unavailable until that node is repaired. red One or more primary shards are unassigned, so some data is unavailable. This can occur briefly during cluster startup as primary shards are assigned. valuemap: name: 'ES cluster state' preprocessing: - type: JSONPATH parameters: - $.status - type: JAVASCRIPT parameters: - | var state = ['green', 'yellow', 'red']; return state.indexOf(value.trim()) === -1 ? 255 : state.indexOf(value.trim()); - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.cluster.get_health tags: - tag: Application value: 'ES cluster' triggers: - uuid: 5a7bd1da899b4bfea52e2ac9531bfb73 expression: 'last(/Elasticsearch Cluster by HTTP zbx 4.2/es.cluster.status)=2' name: 'ES: Health is RED' priority: HIGH description: | One or more primary shards are unassigned, so some data is unavailable. This can occur briefly during cluster startup as primary shards are assigned. - uuid: 00b628ef379c4a4ea8f42b3e01bb7ae7 expression: 'last(/Elasticsearch Cluster by HTTP zbx 4.2/es.cluster.status)=255' name: 'ES: Health is UNKNOWN' priority: HIGH description: 'The health status of the cluster is unknown or cannot be obtained.' - uuid: 0b2e909f922f45fd9728a31d689ccf17 expression: 'last(/Elasticsearch Cluster by HTTP zbx 4.2/es.cluster.status)=1' name: 'ES: Health is YELLOW' priority: AVERAGE description: | All primary shards are assigned, but one or more replica shards are unassigned. If a node in the cluster fails, some data could be unavailable until that node is repaired. - uuid: 683ccf961dc04a2d9d945b5b9a754108 name: 'ES: Task max waiting in queue' type: DEPENDENT key: es.cluster.task_max_waiting_in_queue delay: '0' history: 7d units: s description: 'The time expressed in seconds since the earliest initiated task is waiting for being performed.' preprocessing: - type: JSONPATH parameters: - $.task_max_waiting_in_queue_millis - type: MULTIPLIER parameters: - '0.001' master_item: key: es.cluster.get_health tags: - tag: Application value: 'ES cluster' - uuid: dcd6be883efa44ed851d6f6367033774 name: 'ES: Number of unassigned shards' type: DEPENDENT key: es.cluster.unassigned_shards delay: '0' history: 7d description: 'The number of shards that are not allocated.' preprocessing: - type: JSONPATH parameters: - $.unassigned_shards master_item: key: es.cluster.get_health tags: - tag: Application value: 'ES cluster' triggers: - uuid: 93778534cf51463cad6b13ad6be649f4 expression: 'min(/Elasticsearch Cluster by HTTP zbx 4.2/es.cluster.unassigned_shards,10m)>0' name: 'ES: Cluster has the unassigned shards' priority: AVERAGE description: 'The cluster has the unassigned shards longer than 10 minutes.' - uuid: 73939948173840a590d5328d1222df5e name: 'ES: Indices with shards assigned to nodes' type: DEPENDENT key: es.indices.count delay: '0' history: 7d description: 'The total number of indices with shards assigned to the selected nodes.' preprocessing: - type: JSONPATH parameters: - $.indices.count - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.cluster.get_stats tags: - tag: Application value: 'ES cluster' - uuid: 8abe459422f942de97386a598a9b18d5 name: 'ES: Number of non-deleted documents' type: DEPENDENT key: es.indices.docs.count delay: '0' history: 7d description: | The total number of non-deleted documents across all primary shards assigned to the selected nodes. This number is based on the documents in Lucene segments and may include the documents from nested fields. preprocessing: - type: JSONPATH parameters: - $.indices.docs.count - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.cluster.get_stats tags: - tag: Application value: 'ES cluster' - uuid: eabd11f9e9114078a49f9d77e14e8c50 name: 'ES: Nodes with the data role' type: DEPENDENT key: es.nodes.count.data delay: '0' history: 7d description: 'The number of selected nodes with the data role.' preprocessing: - type: JSONPATH parameters: - $.nodes.count.data - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.cluster.get_stats tags: - tag: Application value: 'ES cluster' - uuid: cf5184903c9e415c9f6fae9719302853 name: 'ES: Nodes with the ingest role' type: DEPENDENT key: es.nodes.count.ingest delay: '0' history: 7d description: 'The number of selected nodes with the ingest role.' preprocessing: - type: JSONPATH parameters: - $.nodes.count.ingest - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.cluster.get_stats tags: - tag: Application value: 'ES cluster' - uuid: 2e4d9f1f977d4140aca9cced1c7e5fd1 name: 'ES: Nodes with the master role' type: DEPENDENT key: es.nodes.count.master delay: '0' history: 7d description: 'The number of selected nodes with the master role.' preprocessing: - type: JSONPATH parameters: - $.nodes.count.master - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.cluster.get_stats tags: - tag: Application value: 'ES cluster' triggers: - uuid: 19f30fca44a745dfa78a67ab1baf5716 expression: 'last(/Elasticsearch Cluster by HTTP zbx 4.2/es.nodes.count.master)=2' name: 'ES: Cluster has only two master nodes' priority: DISASTER description: 'The cluster has only two nodes with a master role and will be unavailable if one of them breaks.' - uuid: 350baaa938774ec287e2ba71b49c63ef name: 'ES: Total available size to JVM in all file stores' type: DEPENDENT key: es.nodes.fs.available_in_bytes delay: '0' history: 7d units: B description: | The total number of bytes available to JVM in the file stores across all selected nodes. Depending on OS or process-level restrictions, this number may be less than nodes.fs.free_in_byes. This is the actual amount of free disk space the selected Elasticsearch nodes can use. preprocessing: - type: JSONPATH parameters: - $.nodes.fs.available_in_bytes - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.cluster.get_stats tags: - tag: Application value: 'ES cluster' - uuid: 80c734c1d5bb49fca3c2e45571826971 name: 'ES: Total size of all file stores' type: DEPENDENT key: es.nodes.fs.total_in_bytes delay: '0' history: 7d units: B description: 'The total size in bytes of all file stores across all selected nodes.' preprocessing: - type: JSONPATH parameters: - $.nodes.fs.total_in_bytes - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.cluster.get_stats tags: - tag: Application value: 'ES cluster' - uuid: 8472fc3bbb684fd0a7075e455a00ddc6 name: 'ES: Get nodes stats' type: HTTP_AGENT key: es.nodes.get_stats history: '0' trends: '0' value_type: TEXT authtype: BASIC username: '{$ELASTICSEARCH.USERNAME}' password: '{$ELASTICSEARCH.PASSWORD}' description: 'Returns cluster nodes statistics.' timeout: 30s url: '{$ELASTICSEARCH.SCHEME}://{HOST.CONN}:{$ELASTICSEARCH.PORT}/_nodes/stats' tags: - tag: Application value: 'Zabbix raw items' - uuid: ca8d2bb52d6943a5b6c9676c245ac25b name: 'ES: Cluster uptime' type: DEPENDENT key: 'es.nodes.jvm.max_uptime[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: s description: 'Uptime duration in seconds since JVM has last started.' preprocessing: - type: JSONPATH parameters: - $.nodes.jvm.max_uptime_in_millis - type: MULTIPLIER parameters: - '0.001' master_item: key: es.cluster.get_stats tags: - tag: Application value: 'ES cluster' triggers: - uuid: 6ba9f3e7b85b42788456b910ebf7117e expression: 'last(/Elasticsearch Cluster by HTTP zbx 4.2/es.nodes.jvm.max_uptime[{#ES.NODE}])<10m' name: 'ES: Cluster has been restarted (uptime < 10m)' priority: INFO description: 'Uptime is less than 10 minutes' manual_close: 'YES' - uuid: ef04c0c596474ba3bd0007c7da74df80 name: 'ES: Total shards' type: DEPENDENT key: es.shards.total delay: '0' history: 7d description: 'Total number of shards' preprocessing: - type: JSONPATH parameters: - $.indices.shards.total - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.cluster.get_stats tags: - tag: Application value: 'ES cluster' - uuid: 651710018f14472f85b656067340fb02 name: 'ES: Service response time' type: SIMPLE key: 'net.tcp.service.perf["{$ELASTICSEARCH.SCHEME}","{HOST.CONN}","{$ELASTICSEARCH.PORT}"]' history: 7d units: s description: 'Checks performance of the TCP service.' tags: - tag: Application value: 'ES cluster' triggers: - uuid: ea700f6fc4c240d1869b15c18e2a34d5 expression: 'min(/Elasticsearch Cluster by HTTP zbx 4.2/net.tcp.service.perf["{$ELASTICSEARCH.SCHEME}","{HOST.CONN}","{$ELASTICSEARCH.PORT}"],5m)>{$ELASTICSEARCH.RESPONSE_TIME.MAX.WARN}' name: 'ES: Service response time is too high (over {$ELASTICSEARCH.RESPONSE_TIME.MAX.WARN} for 5m)' priority: WARNING description: 'The performance of the TCP service is very low.' manual_close: 'YES' dependencies: - name: 'ES: Service is down' expression: 'last(/Elasticsearch Cluster by HTTP zbx 4.2/net.tcp.service["{$ELASTICSEARCH.SCHEME}","{HOST.CONN}","{$ELASTICSEARCH.PORT}"])=0' - uuid: 554e977c218f45d681cc7e5f6ffc857b name: 'ES: Service status' type: SIMPLE key: 'net.tcp.service["{$ELASTICSEARCH.SCHEME}","{HOST.CONN}","{$ELASTICSEARCH.PORT}"]' history: 7d description: 'Checks if the service is running and accepting TCP connections.' preprocessing: - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 10m tags: - tag: Application value: 'ES cluster' triggers: - uuid: 333fc740c8ba49bab29a475d08d1070b expression: 'last(/Elasticsearch Cluster by HTTP zbx 4.2/net.tcp.service["{$ELASTICSEARCH.SCHEME}","{HOST.CONN}","{$ELASTICSEARCH.PORT}"])=0' name: 'ES: Service is down' priority: AVERAGE description: 'The service is unavailable or does not accept TCP connections.' manual_close: 'YES' discovery_rules: - uuid: d2941cd8e58c4f3db3dce725621e3eda name: 'Cluster nodes discovery' type: HTTP_AGENT key: es.nodes.discovery delay: 1h authtype: BASIC username: '{$ELASTICSEARCH.USERNAME}' password: '{$ELASTICSEARCH.PASSWORD}' description: 'Discovery ES cluster nodes.' item_prototypes: - uuid: 9125feb6f13641bcac9c7fb0b96b3340 name: 'ES {#ES.NODE}: Total available size' type: DEPENDENT key: 'es.node.fs.total.available_in_bytes[{#ES.NODE}]' delay: '0' history: 7d units: B description: | The total number of bytes available to this Java virtual machine on all file stores. Depending on OS or process level restrictions, this might appear less than fs.total.free_in_bytes. This is the actual amount of free disk space the Elasticsearch node can utilize. preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].fs.total.available_in_bytes.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 43620c89feed45bca3374fb2f3ab10b4 name: 'ES {#ES.NODE}: Total size' type: DEPENDENT key: 'es.node.fs.total.total_in_bytes[{#ES.NODE}]' delay: '0' history: 7d units: B description: 'Total size (in bytes) of all file stores.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].fs.total.total_in_bytes.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1d master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 4a393b8f68d347f68072d44bdada4626 name: 'ES {#ES.NODE}: Number of open HTTP connections' type: DEPENDENT key: 'es.node.http.current_open[{#ES.NODE}]' delay: '0' history: 7d description: 'The number of currently open HTTP connections for the node.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].http.current_open.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 220435a48f4746f2836868994ae215a6 name: 'ES {#ES.NODE}: Rate of HTTP connections opened' type: DEPENDENT key: 'es.node.http.opened.rate[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: rps description: 'The number of HTTP connections opened for the node per second.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].http.total_opened.first()' - type: CHANGE_PER_SECOND parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: a326c913bbc14c39a211cc84ac1803f0 name: 'ES {#ES.NODE}: Flush latency' type: CALCULATED key: 'es.node.indices.flush.latency[{#ES.NODE}]' history: 7d value_type: FLOAT units: ms params: 'last(//es.node.indices.flush.total_time_in_millis[{#ES.NODE}]) / ( last(//es.node.indices.flush.total[{#ES.NODE}]) + (last(//es.node.indices.flush.total[{#ES.NODE}]) = 0) )' description: 'The average flush latency calculated from the available flush.total and flush.total_time_in_millis metrics.' tags: - tag: Application value: 'ES {#ES.NODE}' trigger_prototypes: - uuid: 9c1c094c51a24249be08dafae9f17fec expression: 'min(/Elasticsearch Cluster by HTTP zbx 4.2/es.node.indices.flush.latency[{#ES.NODE}],5m)>{$ELASTICSEARCH.FLUSH_LATENCY.MAX.WARN}' name: 'ES {#ES.NODE}: Flush latency is too high (over {$ELASTICSEARCH.FLUSH_LATENCY.MAX.WARN}ms for 5m)' priority: WARNING description: | If you see this metric increasing steadily, it may indicate a problem with slow disks; this problem may escalate and eventually prevent you from being able to add new information to your index. - uuid: abdc486baf6c42d9b89fe141aa7df868 name: 'ES {#ES.NODE}: Total number of index flushes to disk' type: DEPENDENT key: 'es.node.indices.flush.total[{#ES.NODE}]' delay: '0' history: 7d description: 'The total number of flush operations.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.flush.total.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.nodes.get_stats tags: - tag: Application value: 'Zabbix raw items' - uuid: dbeddbbfc3754a42882e0732e057a8ea name: 'ES {#ES.NODE}: Total time spent on flushing indices to disk' type: DEPENDENT key: 'es.node.indices.flush.total_time_in_millis[{#ES.NODE}]' delay: '0' history: 7d units: ms description: 'Total time in milliseconds spent performing flush operations.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.flush.total_time_in_millis.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.nodes.get_stats tags: - tag: Application value: 'Zabbix raw items' - uuid: bb181354ac214064b04994dc92ea90bb name: 'ES {#ES.NODE}: Current indexing operations' type: DEPENDENT key: 'es.node.indices.indexing.index_current[{#ES.NODE}]' delay: '0' history: 7d description: 'The number of indexing operations currently running.' preprocessing: - type: XMLPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.indexing.index_current.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: f66060a4e1664b83b26e9558004f385a name: 'ES {#ES.NODE}: Indexing latency' type: CALCULATED key: 'es.node.indices.indexing.index_latency[{#ES.NODE}]' history: 7d value_type: FLOAT units: ms params: 'last(//es.node.indices.indexing.index_time_in_millis[{#ES.NODE}]) / ( last(//es.node.indices.indexing.index_total[{#ES.NODE}]) + (last(//es.node.indices.indexing.index_total[{#ES.NODE}]) = 0) )' description: 'The average indexing latency calculated from the available index_total and index_time_in_millis metrics.' tags: - tag: Application value: 'ES {#ES.NODE}' trigger_prototypes: - uuid: a0aa70233a1f48a895ac9f46ae721a8b expression: 'min(/Elasticsearch Cluster by HTTP zbx 4.2/es.node.indices.indexing.index_latency[{#ES.NODE}],5m)>{$ELASTICSEARCH.INDEXING_LATENCY.MAX.WARN}' name: 'ES {#ES.NODE}: Indexing latency is too high (over {$ELASTICSEARCH.INDEXING_LATENCY.MAX.WARN}ms for 5m)' priority: WARNING description: | If the latency is increasing, it may indicate that you are indexing too many documents at the same time (Elasticsearch’s documentation recommends starting with a bulk indexing size of 5 to 15 megabytes and increasing slowly from there). - uuid: bddf2e0bf98d4131a16eb010cb15bff0 name: 'ES {#ES.NODE}: Total time spent performing indexing' type: DEPENDENT key: 'es.node.indices.indexing.index_time_in_millis[{#ES.NODE}]' delay: '0' history: 7d units: ms description: 'Total time in milliseconds spent performing indexing operations.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.indexing.index_time_in_millis.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.nodes.get_stats tags: - tag: Application value: 'Zabbix raw items' - uuid: 9d9d785ba5bd429e94ddae7b3c71a95b name: 'ES {#ES.NODE}: Total number of indexing' type: DEPENDENT key: 'es.node.indices.indexing.index_total[{#ES.NODE}]' delay: '0' history: 7d description: 'The total number of indexing operations.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.indexing.index_total.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.nodes.get_stats tags: - tag: Application value: 'Zabbix raw items' - uuid: 5efc190d287d43e3b1cc0ce2ab21a1bf name: 'ES {#ES.NODE}: Time spent throttling operations' type: DEPENDENT key: 'es.node.indices.indexing.throttle_time[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: s description: 'Time in seconds spent throttling operations for the last measuring span.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.indexing.throttle_time_in_millis.first()' - type: MULTIPLIER parameters: - '0.001' - type: SIMPLE_CHANGE parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 2ee65bf29f854300a6eda24170fad790 name: 'ES {#ES.NODE}: Time spent throttling merge operations' type: DEPENDENT key: 'es.node.indices.merges.total_throttled_time[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: s description: 'Time in seconds spent throttling merge operations for the last measuring span.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.merges.total_throttled_time_in_millis.first()' - type: MULTIPLIER parameters: - '0.001' - type: SIMPLE_CHANGE parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 3f0f1ba7fe4340a2b3e14493576afc27 name: 'ES {#ES.NODE}: Time spent throttling recovery operations' type: DEPENDENT key: 'es.node.indices.recovery.throttle_time[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: s description: 'Time in seconds spent throttling recovery operations for the last measuring span.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.recovery.throttle_time_in_millis.first()' - type: MULTIPLIER parameters: - '0.001' - type: SIMPLE_CHANGE parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 816a1c11b5e240cc86b0911e3ef8f833 name: 'ES {#ES.NODE}: Rate of index refreshes' type: DEPENDENT key: 'es.node.indices.refresh.rate[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: rps description: 'The number of refresh operations per second.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.refresh.total.first()' - type: CHANGE_PER_SECOND parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 20acb0c46c88407aacee48fa44fe965f name: 'ES {#ES.NODE}: Time spent performing refresh' type: DEPENDENT key: 'es.node.indices.refresh.time[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: s description: 'Time in seconds spent performing refresh operations for the last measuring span.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.refresh.total_time_in_millis.first()' - type: MULTIPLIER parameters: - '0.001' - type: SIMPLE_CHANGE parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: a9b33682f48744cfa807a85214663683 name: 'ES {#ES.NODE}: Rate of fetch' type: DEPENDENT key: 'es.node.indices.search.fetch.rate[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: rps description: 'The number of fetch operations per second.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.search.fetch_total.first()' - type: CHANGE_PER_SECOND parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 2c7e77c397554e6ab3a78a31ccaa001b name: 'ES {#ES.NODE}: Current fetch operations' type: DEPENDENT key: 'es.node.indices.search.fetch_current[{#ES.NODE}]' delay: '0' history: 7d description: 'The number of fetch operations currently running.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.search.fetch_current.first()' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 03a6a0dfcc8f49fca5064e8804abfc8f name: 'ES {#ES.NODE}: Fetch latency' type: CALCULATED key: 'es.node.indices.search.fetch_latency[{#ES.NODE}]' history: 7d value_type: FLOAT units: ms params: 'last(//es.node.indices.search.fetch_time_in_millis[{#ES.NODE}]) / ( last(//es.node.indices.search.fetch_total[{#ES.NODE}]) + (last(//es.node.indices.search.fetch_total[{#ES.NODE}]) = 0) )' description: 'The average fetch latency calculated by sampling the total number of fetches and the total elapsed time at regular intervals.' tags: - tag: Application value: 'ES {#ES.NODE}' trigger_prototypes: - uuid: 29175e15657e490bbd953ba90422e727 expression: 'min(/Elasticsearch Cluster by HTTP zbx 4.2/es.node.indices.search.fetch_latency[{#ES.NODE}],5m)>{$ELASTICSEARCH.FETCH_LATENCY.MAX.WARN}' name: 'ES {#ES.NODE}: Fetch latency is too high (over {$ELASTICSEARCH.FETCH_LATENCY.MAX.WARN}ms for 5m)' priority: WARNING description: | The fetch phase should typically take much less time than the query phase. If you notice this metric consistently increasing, this could indicate a problem with slow disks, enriching of documents (highlighting the relevant text in search results, etc.), or requesting too many results. - uuid: 1fcc8c137dec40049f1c71bb88117232 name: 'ES {#ES.NODE}: Time spent performing fetch' type: DEPENDENT key: 'es.node.indices.search.fetch_time[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: s description: 'Time in seconds spent performing fetch operations for the last measuring span.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.search.fetch_time_in_millis.first()' - type: MULTIPLIER parameters: - '0.001' - type: SIMPLE_CHANGE parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 77b27b0329694be6873670c28d088129 name: 'ES {#ES.NODE}: Total time spent performing fetch' type: DEPENDENT key: 'es.node.indices.search.fetch_time_in_millis[{#ES.NODE}]' delay: '0' history: 7d units: ms description: 'Time in milliseconds spent performing fetch operations.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.search.fetch_time_in_millis.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.nodes.get_stats tags: - tag: Application value: 'Zabbix raw items' - uuid: db5203afa0a6409d9f9af16ffe96842b name: 'ES {#ES.NODE}: Total number of fetch' type: DEPENDENT key: 'es.node.indices.search.fetch_total[{#ES.NODE}]' delay: '0' history: 7d description: 'The total number of fetch operations.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.search.fetch_total.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.nodes.get_stats tags: - tag: Application value: 'Zabbix raw items' - uuid: 8b916b8c84884532958bfa6b14ec3b4d name: 'ES {#ES.NODE}: Rate of queries' type: DEPENDENT key: 'es.node.indices.search.query.rate[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: rps description: 'The number of query operations per second.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.search.query_total.first()' - type: CHANGE_PER_SECOND parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: f265765d87e344c896b37372417e55a2 name: 'ES {#ES.NODE}: Current query operations' type: DEPENDENT key: 'es.node.indices.search.query_current[{#ES.NODE}]' delay: '0' history: 7d description: 'The number of query operations currently running.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.search.query_current.first()' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 217f79e5e23d4135b6e93489d2dfc089 name: 'ES {#ES.NODE}: Query latency' type: CALCULATED key: 'es.node.indices.search.query_latency[{#ES.NODE}]' history: 7d value_type: FLOAT units: ms params: 'last(//es.node.indices.search.query_time_in_millis[{#ES.NODE}]) / ( last(//es.node.indices.search.query_total[{#ES.NODE}]) + (last(//es.node.indices.search.query_total[{#ES.NODE}]) = 0) )' description: 'The average query latency calculated by sampling the total number of queries and the total elapsed time at regular intervals.' tags: - tag: Application value: 'ES {#ES.NODE}' trigger_prototypes: - uuid: 1b66e7a179044ad8abd71a15bea8ee4e expression: 'min(/Elasticsearch Cluster by HTTP zbx 4.2/es.node.indices.search.query_latency[{#ES.NODE}],5m)>{$ELASTICSEARCH.QUERY_LATENCY.MAX.WARN}' name: 'ES {#ES.NODE}: Query latency is too high (over {$ELASTICSEARCH.QUERY_LATENCY.MAX.WARN}ms for 5m)' priority: WARNING description: 'If latency exceeds a threshold, look for potential resource bottlenecks, or investigate whether you need to optimize your queries.' - uuid: 076f33fc6cdd4dd59a9d990ebd63c8ab name: 'ES {#ES.NODE}: Time spent performing query' type: DEPENDENT key: 'es.node.indices.search.query_time[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: s description: 'Time in seconds spent performing query operations for the last measuring span.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.search.query_time_in_millis.first()' - type: MULTIPLIER parameters: - '0.001' - type: SIMPLE_CHANGE parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 543299e451884308be6130207d4cb6fd name: 'ES {#ES.NODE}: Total time spent performing query' type: DEPENDENT key: 'es.node.indices.search.query_time_in_millis[{#ES.NODE}]' delay: '0' history: 7d units: ms description: 'Time in milliseconds spent performing query operations.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.search.query_time_in_millis.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.nodes.get_stats tags: - tag: Application value: 'Zabbix raw items' - uuid: b3f42eacb28a410eb3f2350eb4bdc051 name: 'ES {#ES.NODE}: Total number of query' type: DEPENDENT key: 'es.node.indices.search.query_total[{#ES.NODE}]' delay: '0' history: 7d description: 'The total number of query operations.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].indices.search.query_total.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.nodes.get_stats tags: - tag: Application value: 'Zabbix raw items' - uuid: 0b529734a96142e58a4754c18ae06e28 name: 'ES {#ES.NODE}: Amount of JVM heap committed' type: DEPENDENT key: 'es.node.jvm.mem.heap_committed_in_bytes[{#ES.NODE}]' delay: '0' history: 7d units: B description: 'The amount of memory, in bytes, available for use by the heap.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].jvm.mem.heap_committed_in_bytes.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 425363ac487c4ecf9e8e9b68f55a8294 name: 'ES {#ES.NODE}: Maximum JVM memory available for use' type: DEPENDENT key: 'es.node.jvm.mem.heap_max_in_bytes[{#ES.NODE}]' delay: '0' units: B description: 'The maximum amount of memory, in bytes, available for use by the heap.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].jvm.mem.heap_max_in_bytes.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1d master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 8c7aaf24a65d4da68e330575f8428274 name: 'ES {#ES.NODE}: Amount of JVM heap currently in use' type: DEPENDENT key: 'es.node.jvm.mem.heap_used_in_bytes[{#ES.NODE}]' delay: '0' history: 7d units: B description: 'The memory, in bytes, currently in use by the heap.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].jvm.mem.heap_used_in_bytes.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 2a521448e97a42c68692334e69aad54e name: 'ES {#ES.NODE}: Percent of JVM heap currently in use' type: DEPENDENT key: 'es.node.jvm.mem.heap_used_percent[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: '%' description: 'The percentage of memory currently in use by the heap.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].jvm.mem.heap_used_percent.first()' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1h master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' trigger_prototypes: - uuid: 6db6ed84f6a94a3e973b415416bbf363 expression: 'min(/Elasticsearch Cluster by HTTP zbx 4.2/es.node.jvm.mem.heap_used_percent[{#ES.NODE}],1h)>{$ELASTICSEARCH.HEAP_USED.MAX.CRIT}' name: 'ES {#ES.NODE}: Percent of JVM heap in use is critical (over {$ELASTICSEARCH.HEAP_USED.MAX.CRIT}% for 1h)' priority: HIGH description: | This indicates that the rate of garbage collection isn’t keeping up with the rate of garbage creation. To address this problem, you can either increase your heap size (as long as it remains below the recommended guidelines stated above), or scale out the cluster by adding more nodes. - uuid: cf799a39e5db47e29d0a551ab63f9fc0 expression: 'min(/Elasticsearch Cluster by HTTP zbx 4.2/es.node.jvm.mem.heap_used_percent[{#ES.NODE}],1h)>{$ELASTICSEARCH.HEAP_USED.MAX.WARN}' name: 'ES {#ES.NODE}: Percent of JVM heap in use is high (over {$ELASTICSEARCH.HEAP_USED.MAX.WARN}% for 1h)' priority: WARNING description: | This indicates that the rate of garbage collection isn’t keeping up with the rate of garbage creation. To address this problem, you can either increase your heap size (as long as it remains below the recommended guidelines stated above), or scale out the cluster by adding more nodes. dependencies: - name: 'ES {#ES.NODE}: Percent of JVM heap in use is critical (over {$ELASTICSEARCH.HEAP_USED.MAX.CRIT}% for 1h)' expression: 'min(/Elasticsearch Cluster by HTTP zbx 4.2/es.node.jvm.mem.heap_used_percent[{#ES.NODE}],1h)>{$ELASTICSEARCH.HEAP_USED.MAX.CRIT}' - uuid: 2be5a40ba7d547c8b901b4bc0fbadf36 name: 'ES {#ES.NODE}: Node uptime' type: DEPENDENT key: 'es.node.jvm.uptime[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: s description: 'JVM uptime in seconds.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].jvm.uptime_in_millis.first()' - type: MULTIPLIER parameters: - '0.001' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' trigger_prototypes: - uuid: 7d11e471836c4f988cf4cfef9fa8d656 expression: 'last(/Elasticsearch Cluster by HTTP zbx 4.2/es.node.jvm.uptime[{#ES.NODE}])<10m' name: 'ES {#ES.NODE}: Node {#ES.NODE} has been restarted (uptime < 10m)' priority: INFO description: 'Uptime is less than 10 minutes' - uuid: d7377ea6fee54a9dab8f8d11f1e365f8 name: 'ES {#ES.NODE}: Refresh thread pool active threads' type: DEPENDENT key: 'es.node.thread_pool.refresh.active[{#ES.NODE}]' delay: '0' history: 7d description: 'The number of active threads in the refresh thread pool.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].thread_pool.refresh.active.first()' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 8748f3edacc0495bb62ea0e7598c9cc3 name: 'ES {#ES.NODE}: Refresh thread pool executor tasks completed' type: DEPENDENT key: 'es.node.thread_pool.refresh.completed.rate[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: rps description: 'The number of tasks completed by the refresh thread pool executor.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].thread_pool.refresh.completed.first()' - type: CHANGE_PER_SECOND parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 620a6e552e7e48888228df30b046781c name: 'ES {#ES.NODE}: Refresh thread pool tasks in queue' type: DEPENDENT key: 'es.node.thread_pool.refresh.queue[{#ES.NODE}]' delay: '0' history: 7d description: 'The number of tasks in queue for the refresh thread pool.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].thread_pool.refresh.queue.first()' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: b38af314546741eba07dd27a9cf75d9f name: 'ES {#ES.NODE}: Refresh thread pool executor tasks rejected' type: DEPENDENT key: 'es.node.thread_pool.refresh.rejected.rate[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: rps description: 'The number of tasks rejected by the refresh thread pool executor.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].thread_pool.refresh.rejected.first()' - type: CHANGE_PER_SECOND parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' trigger_prototypes: - uuid: 67946619283c42daae8865075fe4ae67 expression: 'min(/Elasticsearch Cluster by HTTP zbx 4.2/es.node.thread_pool.refresh.rejected.rate[{#ES.NODE}],5m)>0' name: 'ES {#ES.NODE}: Refresh thread pool executor has the rejected tasks (for 5m)' priority: WARNING description: 'The number of tasks rejected by the refresh thread pool executor is over 0 for 5m.' - uuid: 1e04c3503c0e450582b6bb589bec993d name: 'ES {#ES.NODE}: Search thread pool active threads' type: DEPENDENT key: 'es.node.thread_pool.search.active[{#ES.NODE}]' delay: '0' history: 7d description: 'The number of active threads in the search thread pool.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].thread_pool.search.active.first()' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 8777b95a154f421aa2db851f1dec3a59 name: 'ES {#ES.NODE}: Search thread pool executor tasks completed' type: DEPENDENT key: 'es.node.thread_pool.search.completed.rate[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: rps description: 'The number of tasks completed by the search thread pool executor.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].thread_pool.search.completed.first()' - type: CHANGE_PER_SECOND parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: bb0e6003f872435a9ed7dfe1b56fa7d5 name: 'ES {#ES.NODE}: Search thread pool tasks in queue' type: DEPENDENT key: 'es.node.thread_pool.search.queue[{#ES.NODE}]' delay: '0' history: 7d description: 'The number of tasks in queue for the search thread pool.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].thread_pool.search.queue.first()' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: db1417d6c6584b6381c41d0518a82cdd name: 'ES {#ES.NODE}: Search thread pool executor tasks rejected' type: DEPENDENT key: 'es.node.thread_pool.search.rejected.rate[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: rps description: 'The number of tasks rejected by the search thread pool executor.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].thread_pool.search.rejected.first()' - type: CHANGE_PER_SECOND parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' trigger_prototypes: - uuid: a24f78df3805448aa4597a66bce74e73 expression: 'min(/Elasticsearch Cluster by HTTP zbx 4.2/es.node.thread_pool.search.rejected.rate[{#ES.NODE}],5m)>0' name: 'ES {#ES.NODE}: Search thread pool executor has the rejected tasks (for 5m)' priority: WARNING description: 'The number of tasks rejected by the search thread pool executor is over 0 for 5m.' - uuid: e16e7e89fe9a496584206606a14d5124 name: 'ES {#ES.NODE}: Write thread pool active threads' type: DEPENDENT key: 'es.node.thread_pool.write.active[{#ES.NODE}]' delay: '0' history: 7d description: 'The number of active threads in the write thread pool.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].thread_pool.write.active.first()' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: dc5518c9b4594991824c3e2613d8b253 name: 'ES {#ES.NODE}: Write thread pool executor tasks completed' type: DEPENDENT key: 'es.node.thread_pool.write.completed.rate[{#ES.NODE}]' delay: '0' history: 7d units: rps description: 'The number of tasks completed by the write thread pool executor.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].thread_pool.write.completed.first()' - type: CHANGE_PER_SECOND parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: b1e5a1f5f761405cbc03c69c141839b9 name: 'ES {#ES.NODE}: Write thread pool tasks in queue' type: DEPENDENT key: 'es.node.thread_pool.write.queue[{#ES.NODE}]' delay: '0' history: 7d description: 'The number of tasks in queue for the write thread pool.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].thread_pool.write.queue.first()' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' - uuid: 1e25e1e6baef4ab0aaefeea2711863fb name: 'ES {#ES.NODE}: Write thread pool executor tasks rejected' type: DEPENDENT key: 'es.node.thread_pool.write.rejected.rate[{#ES.NODE}]' delay: '0' history: 7d value_type: FLOAT units: rps description: 'The number of tasks rejected by the write thread pool executor.' preprocessing: - type: JSONPATH parameters: - '$..[?(@.name==''{#ES.NODE}'')].thread_pool.write.rejected.first()' - type: CHANGE_PER_SECOND parameters: - '' master_item: key: es.nodes.get_stats tags: - tag: Application value: 'ES {#ES.NODE}' trigger_prototypes: - uuid: a3ed94610168483b825eb2852b1b32b9 expression: 'min(/Elasticsearch Cluster by HTTP zbx 4.2/es.node.thread_pool.write.rejected.rate[{#ES.NODE}],5m)>0' name: 'ES {#ES.NODE}: Write thread pool executor has the rejected tasks (for 5m)' priority: WARNING description: 'The number of tasks rejected by the write thread pool executor is over 0 for 5m.' timeout: 15s url: '{$ELASTICSEARCH.SCHEME}://{HOST.CONN}:{$ELASTICSEARCH.PORT}/_nodes/_all/nodes' lld_macro_paths: - lld_macro: '{#ES.NODE}' path: $..name.first() preprocessing: - type: JSONPATH parameters: - '$.nodes.[*]' - type: DISCARD_UNCHANGED_HEARTBEAT parameters: - 1d macros: - macro: '{$ELASTICSEARCH.FETCH_LATENCY.MAX.WARN}' value: '100' - macro: '{$ELASTICSEARCH.FLUSH_LATENCY.MAX.WARN}' value: '100' - macro: '{$ELASTICSEARCH.HEAP_USED.MAX.CRIT}' value: '95' - macro: '{$ELASTICSEARCH.HEAP_USED.MAX.WARN}' value: '85' - macro: '{$ELASTICSEARCH.INDEXING_LATENCY.MAX.WARN}' value: '100' - macro: '{$ELASTICSEARCH.MAX_SHARDS_PER_NODE}' value: '1000' - macro: '{$ELASTICSEARCH.PASSWORD}' - macro: '{$ELASTICSEARCH.PORT}' value: '9200' - macro: '{$ELASTICSEARCH.QUERY_LATENCY.MAX.WARN}' value: '100' - macro: '{$ELASTICSEARCH.RESPONSE_TIME.MAX.WARN}' value: 10s - macro: '{$ELASTICSEARCH.SCHEME}' value: http - macro: '{$ELASTICSEARCH.USERNAME}' valuemaps: - uuid: 2a1dd399e77e4c15afa5d896ebb628b6 name: 'ES cluster state' mappings: - value: '0' newvalue: green - value: '1' newvalue: yellow - value: '2' newvalue: red - value: '255' newvalue: unknown triggers: - uuid: 6810be8551f14e3c925aaf971e3329ac expression: '(last(/Elasticsearch Cluster by HTTP zbx 4.2/es.nodes.fs.total_in_bytes)-last(/Elasticsearch Cluster by HTTP zbx 4.2/es.nodes.fs.available_in_bytes))/(last(/Elasticsearch Cluster by HTTP zbx 4.2/es.cluster.number_of_data_nodes)-1)>last(/Elasticsearch Cluster by HTTP zbx 4.2/es.nodes.fs.available_in_bytes)' name: 'ES: Cluster does not have enough space for resharding' priority: HIGH description: 'There is not enough disk space for index resharding.' - uuid: e51e74d927d94013afc8d33ee5143396 expression: 'last(/Elasticsearch Cluster by HTTP zbx 4.2/es.shards.total)>((last(/Elasticsearch Cluster by HTTP zbx 4.2/es.nodes.count.data)*{$ELASTICSEARCH.MAX_SHARDS_PER_NODE})-(last(/Elasticsearch Cluster by HTTP zbx 4.2/es.shards.total)/last(/Elasticsearch Cluster by HTTP zbx 4.2/es.indices.count)*5))' name: 'ES Shards left for 5 indices' priority: HIGH description: | Be default in ES max shards per node is 1000 ({$ELASTICSEARCH.MAX_SHARDS_PER_NODE}) if number of shrads is maximum then elasticshearch will be sillently reject write requests. Expression understanding: TOTAL_SARDS > (MAX_SHARDS_ON_CLUSTER - SHARDS_NEEDED_FOR_5_INDICES) Fix example: curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": {"cluster.max_shards_per_node": "2000" } }' graphs: - uuid: 0b6df8c009874d218a711720d3b6c862 name: 'ES: Cluster health' graph_items: - color: 1A7C11 item: host: 'Elasticsearch Cluster by HTTP zbx 4.2' key: es.cluster.inactive_shards_percent_as_number - sortorder: '1' color: 2774A4 item: host: 'Elasticsearch Cluster by HTTP zbx 4.2' key: es.cluster.relocating_shards - sortorder: '2' color: F63100 item: host: 'Elasticsearch Cluster by HTTP zbx 4.2' key: es.cluster.initializing_shards - sortorder: '3' color: A54F10 item: host: 'Elasticsearch Cluster by HTTP zbx 4.2' key: es.cluster.unassigned_shards - sortorder: '4' color: FC6EA3 item: host: 'Elasticsearch Cluster by HTTP zbx 4.2' key: es.cluster.delayed_unassigned_shards - sortorder: '5' color: 6C59DC item: host: 'Elasticsearch Cluster by HTTP zbx 4.2' key: es.cluster.number_of_pending_tasks - sortorder: '6' color: AC8C14 item: host: 'Elasticsearch Cluster by HTTP zbx 4.2' key: es.cluster.task_max_waiting_in_queue