5.02021-11-21T21:57:04ZHalley templateHAProxyHAProxy## Overview
How to install:
1. Add in HAProxy (or make sure that you have) next rules to enable statistics on socket
vi /etc/haproxy/haproxy.cfg
stats socket /var/lib/haproxy/stats mode 666 level admin
stats timeout 30s
2. Install socat and nc: yum install nc socat -yum
3. Make sure that HAProxy user can read from socket :sudo -uhaproxy echo "show info;show stat" | socat stdio unix-connect:/var/lib/haproxy/stats
4. Copy files:
a) `userparameter\_haproxy.conf` in `/etc/zabbix/zabbix\_agentd.d/`
b) haproxy\_discovery.sh in /etc/zabbix/scripts/
c) haproxy\_stats.sh in /etc/zabbix/scripts/
Make b and c scripts executable with chmod +x script\_name
Note: Make sure that /etc/zabbix/scripts/ exist, if not, create it: mkdir -p /etc/zabbix/scripts/
5. Add host for HAProxy in Zabbix, add template, wait some time for get data
(You can change LLD discovery time to get data more faster, but after change to initial)
This template is based on:
a) Solution by Anastas Dancha - https://github.com/anapsix/zabbix-haproxy
b) Official template from Zabbix for Zabbix > 4.4 - https://www.zabbix.com/integrations/haproxy
The reason why I create this template was to have official zabbix template logic in Zabbix under 4.4
Files are there
a) https://cloud.mail.ru/public/D2M5%2F7ZEamjnVF
b) https://drive.google.com/open?id=16xoJyWut9R\_EudcRyAf2Ui8WuPyTxw6D
Write to tudorticau@mail.ru if something is not clear
Have a nice day
## Author
Tudor Ticau
Halley templateHAProxy- HAProxy memory usedproc.mem[haproxy]300FLOATbHAProxy
- HAProxy number of running processesproc.num[haproxy]60HAProxy{last()}<1HAProxy is not running on {HOST.NAME}HIGH
- HAProxy config file checksum ($1)vfs.file.cksum[{$HAPROXY_CONFIG}]600HAProxy{diff(0)}>0HAProxy config file changed on {HOST.NAME}WARNING
HAProxy backend discoveryhaproxy.list.discovery[{$HAPROXY_SOCK},BACK]1h5dHAProxy Backend [{#BACKEND_NAME}] bytes inZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,bin]60FLOATbpsHAProxy Backend bytes inHAProxy Backend [{#BACKEND_NAME}]CHANGE_PER_SECONDHAProxy Backend [{#BACKEND_NAME}] bytes outZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,bout]60FLOATbpsHAProxy Backend bytes outHAProxy Backend [{#BACKEND_NAME}]CHANGE_PER_SECONDHAProxy Backend [{#BACKEND_NAME}]: Responses denied per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,dresp]60DISABLEDResponses denied due to security concerns (ACL-restricted).
In most cases denials will originate in the frontend (e.g., a user is attempting to access an unauthorized URL). However, sometimes a request may be benign, yet the corresponding response contains sensitive information. In that case, you would want to set up an ACL to deny the offending response. Backend responses that are denied due to ACL restrictions will emit a 502 error code. With properly configured access controls on frontend, this metric should stay at or near zero.
Denied responses and an increase in 5xx responses go hand-in-hand. If you are seeing a large number of 5xx responses, you should check your denied responses to shed some light on the increase in error codesHAProxy Backend [{#BACKEND_NAME}]CHANGE_PER_SECONDHAProxy Backend [{#BACKEND_NAME}]: Errors connection per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,econ]60DISABLEDmsNumber of requests that encountered an error attempting to connect to a backend server.
Backend connection failures should be acted upon immediately. Unfortunately, the econ metric not only includes failed backend requests but additionally includes general backend errors, like a backend without an active frontend. Thankfully, correlating this metric with eresp and response codes from both frontend and backend servers will give a better idea of the causes of an increase in backend connection errors.HAProxy Backend [{#BACKEND_NAME}]CHANGE_PER_SECONDHAProxy Backend [{#BACKEND_NAME}] : Response errors per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,eresp]60DISABLEDNumber of requests whose responses yielded an error
This represents the number of response errors generated by your backends. This includes errors caused by data transfers aborted by the servers as well as write errors on the client socket and failures due to ACLs. Combined with other error metrics, the backend error response rate helps diagnose the root cause of response errors. For example, an increase in both the backend error response rate and denied responses could indicate that clients are repeatedly attempting to access ACL-ed resources.HAProxy Backend [{#BACKEND_NAME}]CHANGE_PER_SECOND{min(5m)}>10HAProxy Backend [{#BACKEND_NAME}]: Number of responses with error is more than 10 for 5mDISABLEDAVERAGENumber of requests on backend, whose responses yielded an error, is more than 10.
The backend error response rate represents the number of response errors generated by your backends. This includes errors caused by data transfers aborted by the servers as well as write errors on the client socket and failures due to ACLs. Combined with other error metrics, the backend error response rate helps diagnose the root cause of response errors. For example, an increase in both the backend error response rate and denied responses could indicate that clients are repeatedly attempting to access ACL-ed resources.Value{ITEM.VALUE}HAProxy Backend [{#BACKEND_NAME}]: Unassigned requestsZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,qcur]60DISABLEDCurrent number of requests unassigned in queue.
The qcur metric tracks the current number of connections awaiting assignment to a backend server. If you have enabled cookies and the listed server is unavailable, connections will be queued until the queue timeout is reachedHAProxy Backend [{#BACKEND_NAME}]{min(5m)}>10HAProxy Backend [{#BACKEND_NAME}]: Current number of requests unassigned in queue is more than 10 for 5mDISABLEDAVERAGECurrent number of requests on backend unassigned in queue is more than 10.
If your backend is bombarded with connections to the point you have reached your global maxconn limit, HAProxy will seamlessly queue new connections in system kernel’s socket queue until a backend server becomes available.
Keeping connections out of the queue is ideal, resulting in less latency and a better user experience. You should alert if the size of your queue exceeds the threshold. If you find that connections are consistently enqueueing, configuration changes may be in order, such as increasing global maxconn limit or changing the connection limits on individual backend servers.
Keep in mind: empty queue = happy clientValue{ITEM.VALUE}HAProxy Backend [{#BACKEND_NAME}]: Time in queueZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,qtime]60sAverage time spent in queue (in ms) for the last 1,024 requests
Minimizing time spent in the queue results in lower latency and an overall better client experience. Each use case can tolerate a certain amount of queue time but in general, you should aim to keep this value as low as possibleHAProxy Backend [{#BACKEND_NAME}]MULTIPLIER0.001{min(5m)}>10sHAProxy Backend [{#BACKEND_NAME}]: Average time spent in queue is more than 10 sec for 5mAVERAGEAverage time spent in queue (in ms) for the last 1,024 requests is more than 10 s.
It is obviously that minimizing time spent in the queue results in lower latency and an overall better client experience. Each use case can tolerate a certain amount of queue time but in general, you should aim to keep this value as low as possibleValue{ITEM.VALUE}HAProxy Backend [{#BACKEND_NAME}]: Responses timeZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,rtime]60FLOATsAverage backend response time (in ms) for the last 1,024 requests
Tracking average response times is an effective way to measure the latency of your load-balancing setup. Generally speaking, response times in excess of 500 ms will lead to degradation of application performance and customer experience. Monitoring the average response time can give you the upper hand to respond to latency issues before your customers are substantially impacted.
Keep in mind that this metric will be zero if you are not using HTTP (see #60)HAProxy Backend [{#BACKEND_NAME}]MULTIPLIER0.001{min(5m)}>10sHAProxy Backend [{#BACKEND_NAME}]: Average response time is more than 10 sec for 5mAVERAGEAverage backend response time (in ms) for the last 1,024 requests is more than 10 seconds.
Tracking average response times is an effective way to measure the latency of haproxy load-balancing setup. Generally speaking, response times in excess of 500 ms will lead to degradation of application performance and customer experience. Monitoring the average response time can give you the upper hand to respond to latency issues before your customers are substantially impacted.
Keep in mind that this metric will be zero if you are not using HTTPValue{ITEM.VALUE}HAProxy Backend [{#BACKEND_NAME}]: StatusZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,status]60HAProxy Backend [{#BACKEND_NAME}] status
UP = 1
DOWN = 0HAProxy Backend [{#BACKEND_NAME}]Service stateBOOL_TO_DECIMAL{max(#5)}=0HAProxy Backend [{#BACKEND_NAME}]: Server is DOWNDISASTERHAProxy Backend [{#BACKEND_NAME}] is not available.HAProxy Backend [{#BACKEND_NAME}]: Redispatched requests per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,wredis]60DISABLEDconnNumber of times a request was redispatched to a different backend.
The redispatch rate metric tracks the number of times a client connection was unable to reach its original target, and was subsequently sent to a different server. If a client holds a cookie referencing a backend server that is down, the default action is to respond to the client with a 502 status code. However, if is enabled option redispatch in haproxy.cfg, the request will be sent to any available backend server and the cookie will be ignored.HAProxy Backend [{#BACKEND_NAME}]CHANGE_PER_SECONDHAProxy Backend [{#BACKEND_NAME}]: Retried connections per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,wretr]60DISABLEDNumber of times a connection was retried.
Some dropped or timed-out connections are to be expected when connecting to a backend server. The retry rate represents the number of times a connection to a backend server was retried. This metric is usually non-zero under normal operating conditions. Should you begin to see more retries than usual, it is likely that other metrics will also change, including econ and eresp.
Tracking the retry rate in addition to the above two error metrics can shine some light on the true cause of an increase in errorsHAProxy Backend [{#BACKEND_NAME}]CHANGE_PER_SECONDHAProxy Backend [{#BACKEND_NAME}] Redispatched requests and retried connections per second1A7C11- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,wredis]
1F63100- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,wretr]
HAProxy Backend [{#BACKEND_NAME}] Responses time and time in queue1A7C11- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,rtime]
1F63100- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,qtime]
HAProxy Backend [{#BACKEND_NAME}] Status1A7C11- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,status]
HAProxy Backend [{#BACKEND_NAME}] Traffic1A7C11- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,bin]
1F63100- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},BACKEND,bout]
HAProxy frontend discoveryhaproxy.list.discovery[{$HAPROXY_SOCK},FRONT]1d5dHAProxy Frontend [{#FRONTEND_NAME}]: Incoming trafficZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,bin]60FLOATbpsNumber of bits received by the frontendHAProxy Frontend [{#FRONTEND_NAME}]MULTIPLIER8CHANGE_PER_SECONDHAProxy Frontend [{#FRONTEND_NAME}]: Outgoing trafficZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,bout]60bpsNumber of bits sent by the frontendHAProxy Frontend [{#FRONTEND_NAME}]CHANGE_PER_SECONDMULTIPLIER8HAProxy Frontend [{#FRONTEND_NAME}]: Denied requests per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,dreq]60Requests denied due to security concerns (ACL-restricted) per second.
An increase in denied requests will subsequently cause an increase in 403 Forbidden codes.
- For tcp this is because of a matched tcp-request content rule.
- For http this is because of a matched http-request or tarpit rule.
Correlating the two can help to discern the root cause of an increase in 4xx responses.HAProxy Frontend [{#FRONTEND_NAME}]CHANGE_PER_SECOND{min(5m)}>10HAProxy Frontend [{#FRONTEND_NAME}]: Number of requests denied is more than 10 for 5mAVERAGENumber of requests denied due to security concerns (ACL-restricted) is more than 10 in last 5 minutes.
In the event of a significant increase in denials—a malicious attacker or misconfigured application could be to blame
An increase in denied requests will subsequently cause an increase in 403 Forbidden codes. Correlating the two can help you discern the root cause of an increase in 4xx responses.Value{ITEM.VALUE}HAProxy Frontend[{#FRONTEND_NAME}]: Request errors per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,ereq]60HTTP request errors per second.
The frontend request rate measures the number of requests received over the last second. Keeping an eye on peaks and drops is essential to ensure continuous service availability. In the event of a traffic spike, clients could see increases in latency or even denied connections.
Some of the possible causes are:
- early termination from the client, before the request has been sent.
- read error from the client
- client timeout
- client closed connection
- various bad requests from the client.
- request was tarpitted.HAProxy Frontend [{#FRONTEND_NAME}]CHANGE_PER_SECOND{min(5m)}>10HAProxy Frontend [{#FRONTEND_NAME}]: Average response time is more than 10 sec for 5mAVERAGENumber of request errors in last 5 minutes is more than 10.
Client-side request errors could have a number of causes:
client terminates before sending request
read error from client
client timeout
client terminated connection
request was tarpitted/subject to ACL
Under normal conditions, it is acceptable to (infrequently) receive invalid requests from clients. However, a significant increase in the number of invalid requests received could be a sign of larger, looming issues.
For example, an abnormal number of terminations or timeouts by numerous clients could mean that your application is experiencing excessive latency, causing clients to manually close their connections.Value{ITEM.VALUE}HAProxy Frontend [{#FRONTEND_NAME}]: Number of responses with codes 1xx per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,hrsp_1xx]60Number of informational (1xx) HTTP responses per second.HAProxy Frontend [{#FRONTEND_NAME}]CHANGE_PER_SECONDHAProxy Frontend [{#FRONTEND_NAME}]: Number of responses with codes 2xx per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,hrsp_2xx]60Number of successful HTTP responses per second. ( with 2xx code)HAProxy Frontend [{#FRONTEND_NAME}]CHANGE_PER_SECONDHAProxy Frontend [{#FRONTEND_NAME}]: Number of responses with codes 3xx per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,hrsp_3xx]60Number of HTTP redirections per second.. ( with 3xx code)HAProxy Frontend [{#FRONTEND_NAME}]CHANGE_PER_SECONDHAProxy Frontend [{#FRONTEND_NAME}]: Number of responses with codes 4xx per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,hrsp_4xx]60Number of HTTP client errors per second. ( with 4xx code)
Ideally, all responses forwarded by HAProxy would be class 2xx codes, so an unexpected surge in the number of other code classes could be a sign of trouble.
Correlating the denial metrics with the response code data can shed light on the cause of an increase in error codes. No change in denials coupled with an increase in the number of 404 responses could point to a misconfigured application or unruly client.HAProxy Frontend [{#FRONTEND_NAME}]CHANGE_PER_SECONDHAProxy Frontend [{#FRONTEND_NAME}]: Number of responses with codes 5xx per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,hrsp_5xx]60Number of HTTP server errors per second. ( with 5xx code)
Ideally, all responses forwarded by HAProxy would be class 2xx codes, so an unexpected surge in the number of other code classes could be a sign of trouble.
Correlating the denial metrics with the response code data can shed light on the cause of an increase in error codes.HAProxy Frontend [{#FRONTEND_NAME}]CHANGE_PER_SECONDHAProxy Frontend [{#FRONTEND_NAME}]: Number of responses with other codes per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,hrsp_other]60Number of other HTTP server errors per second. ( all other codes, no 1-5xx)HAProxy Frontend [{#FRONTEND_NAME}]CHANGE_PER_SECONDHAProxy Frontend [{#FRONTEND_NAME}]: Sessions rateZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,rate]60Number of sessions created per second
A significant spike in the number of sessions over a short period could cripple server operations and bring servers downHAProxy Frontend [{#FRONTEND_NAME}]HAProxy Frontend [{#FRONTEND_NAME}]: Requests rateZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,req_rate]60HTTP requests per second.
The frontend request rate measures the number of requests received over the last second. Keeping an eye on peaks and drops is essential to ensure continuous service availability. In the event of a traffic spike, clients could see increases in latency or even denied connections.HAProxy Frontend [{#FRONTEND_NAME}]HAProxy Frontend [{#FRONTEND_NAME}]: Established sessionsZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,scur]60The current number of established sessions.HAProxy Frontend [{#FRONTEND_NAME}]HAProxy Frontend [{#FRONTEND_NAME}]: Session limitsZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,slim]60The most simultaneous sessions that are allowed, as defined by the maxconn setting in the frontend.HAProxy Frontend [{#FRONTEND_NAME}]HAProxy Frontend [{#FRONTEND_NAME}]: Session utilizationCALCULATEDhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,sutil]60FLOAT%last(haproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,scur]) / last(haproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,slim]) * 100Percentage of sessions used (scur / slim * 100).
For every HAProxy session, two connections are consumed—one for the client to HAProxy, and the other for HAProxy to your backend.
Alerting on this metric is essential to ensure your server has sufficient capacity to handle all concurrent sessions. Unlike requests, upon reaching the session limit HAProxy will deny additional clients until resource consumption drops.HAProxy Frontend [{#FRONTEND_NAME}]{min(5m)}>80HAProxy Frontend [{#FRONTEND_NAME}]: Session utilization is more than 80% for 5mAVERAGEFor every HAProxy session, two connections are consumed—one for the client to HAProxy, and the other for HAProxy to your backend. Alerting on this metric is essential to ensure your server has sufficient capacity to handle all concurrent sessions. Unlike requests, upon reaching the session limit HAProxy will deny additional clients until resource consumption drops. Furthermore, if you find your session usage percentage to be hovering above 80%, it could be time to either modify HAProxy’s configuration to allow more sessions, or migrate your HAProxy server to a bigger box.Value{ITEM.VALUE}HAProxy Frontend [{#FRONTEND_NAME}]: Errors and denials per second1A7C11- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,dreq]
1F63100- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,ereq]
HAProxy Frontend [{#FRONTEND_NAME}]: In/Out traffic2774A4- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,bin]
1A54F10- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,bout]
HAProxy Frontend [{#FRONTEND_NAME}]: Requests and sessions per second2774A4- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,req_rate]
1E64A19- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,slim]
200FF00- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,rate]
HAProxy Frontend [{#FRONTEND_NAME}]: Responses by HTTP code607D8B- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,hrsp_1xx]
14CAF50- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,hrsp_2xx]
26C59DC- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,hrsp_3xx]
3AC8C14- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,hrsp_4xx]
4611F27- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,hrsp_5xx]
5BF360C- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#FRONTEND_NAME},FRONTEND,hrsp_other]
HAProxy server discoveryhaproxy.list.discovery[{$HAPROXY_SOCK},SERVER]1hHAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: Responses denied per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},{#SERVER_NAME},dresp]60DISABLEDResponses denied due to security concerns (ACL-restricted).
In most cases denials will originate in the frontend (e.g., a user is attempting to access an unauthorized URL). However, sometimes a request may be benign, yet the corresponding response contains sensitive information. In that case, you would want to set up an ACL to deny the offending response. Backend responses that are denied due to ACL restrictions will emit a 502 error code. With properly configured access controls on frontend, this metric should stay at or near zero.
Denied responses and an increase in 5xx responses go hand-in-hand. If you are seeing a large number of 5xx responses, you should check your denied responses to shed some light on the increase in error codesHAProxy Backend [{#BACKEND_NAME}]SIMPLE_CHANGEHAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: Errors connection per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},{#SERVER_NAME},econ]60DISABLEDNumber of requests that encountered an error attempting to connect to a backend server.
Backend connection failures should be acted upon immediately. Unfortunately, the econ metric not only includes failed backend requests but additionally includes general backend errors, like a backend without an active frontend. Thankfully, correlating this metric with eresp and response codes from both frontend and backend servers will give a better idea of the causes of an increase in backend connection errors.HAProxy Backend [{#BACKEND_NAME}]CHANGE_PER_SECONDHAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: Response errors per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},{#SERVER_NAME},eresp]60DISABLEDNumber of requests whose responses yielded an error
This represents the number of response errors generated by your backends. This includes errors caused by data transfers aborted by the servers as well as write errors on the client socket and failures due to ACLs. Combined with other error metrics, the backend error response rate helps diagnose the root cause of response errors. For example, an increase in both the backend error response rate and denied responses could indicate that clients are repeatedly attempting to access ACL-ed resources.HAProxy Backend [{#BACKEND_NAME}]CHANGE_PER_SECOND{min(5m)}>10HAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: Number of responses with error is more than 10s for 5mAVERAGENumber of requests on server, whose responses yielded an error, is more than 10.
The server error response rate represents the number of response errors generated by your servers. This includes errors caused by data transfers aborted by the servers as well as write errors on the client socket and failures due to ACLs. Combined with other error metrics, the server error response rate helps diagnose the root cause of response errors. For example, an increase in both the server error response rate and denied responses could indicate that clients are repeatedly attempting to access ACL-ed resources.Value{ITEM.VALUE}HAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: Number of responses with codes 4xx per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},{#SERVER_NAME},hrsp_4xx]60DISABLEDNumber of HTTP client errors per second.HAProxy Backend [{#BACKEND_NAME}]CHANGE_PER_SECONDHAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: Number of responses with codes 5xx per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},{#SERVER_NAME},hrsp_5xx]60DISABLEDNumber of HTTP server errors per second.HAProxy Backend [{#BACKEND_NAME}]CHANGE_PER_SECONDHAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: Unassigned requestsZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},{#SERVER_NAME},qcur]60DISABLEDCurrent number of requests unassigned in queue.
The qcur metric tracks the current number of connections awaiting assignment to a backend server. If you have enabled cookies and the listed server is unavailable, connections will be queued until the queue timeout is reachedHAProxy Backend [{#BACKEND_NAME}]{min(5m)}>10HAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: Current number of requests unassigned in queue is more than 10s for 5mAVERAGECurrent number of requests unassigned in queue is more than 10.
If your server is bombarded with connections to the point you have reached your global maxconn limit, HAProxy will seamlessly queue new connections in system kernel’s socket queue until the server becomes available.
Keeping connections out of the queue is ideal, resulting in less latency and a better user experience. You should alert if the size of your queue exceeds the threshold. If you find that connections are consistently enqueueing, configuration changes may be in order, such as increasing global maxconn limit or changing the connection limits on individual backend servers.
Keep in mind: empty queue = happy clientValue{ITEM.VALUE}HAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: Time in queueZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},{#SERVER_NAME},qtime]60DISABLEDFLOATsAverage time spent in queue (in ms) for the last 1,024 requests
Minimizing time spent in the queue results in lower latency and an overall better client experience. Each use case can tolerate a certain amount of queue time but in general, you should aim to keep this value as low as possibleHAProxy Backend [{#BACKEND_NAME}]MULTIPLIER0.001{min(5m)}>10sHAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: Average time spent in queue is more than 10s for 5mAVERAGEAverage time spent in queue (in ms) for the last 1,024 requests is more than 10s.
It is obviously that minimizing time spent in the queue results in lower latency and an overall better client experience. Each use case can tolerate a certain amount of queue time but in general, you should aim to keep this value as low as possibleValue{ITEM.VALUE}HAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: Responses timeZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},{#SERVER_NAME},rtime]60DISABLEDFLOATsAverage backend response time (in ms) for the last 1,024 requests
Tracking average response times is an effective way to measure the latency of your load-balancing setup. Generally speaking, response times in excess of 500 ms will lead to degradation of application performance and customer experience. Monitoring the average response time can give you the upper hand to respond to latency issues before your customers are substantially impacted.
Keep in mind that this metric will be zero if you are not using HTTP (see #60)HAProxy Backend [{#BACKEND_NAME}]MULTIPLIER0.001{min(5m)}>10sHAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: Average response time is more than 10s for 5mAVERAGEAverage server response time (in ms) for the last 1,024 requests is more than 10s.
Tracking average response times is an effective way to measure the latency of haproxy load-balancing setup. Generally speaking, response times in excess of 500 ms will lead to degradation of application performance and customer experience. Monitoring the average response time can give you the upper hand to respond to latency issues before your customers are substantially impacted.
Keep in mind that this metric will be zero if you are not using HTTPValue{ITEM.VALUE}HAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: StatusZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},{#SERVER_NAME},status]60HAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}] status
UP = 1
DOWN = 0HAProxy Backend [{#BACKEND_NAME}]Service stateBOOL_TO_DECIMAL{max(#5)}=0HAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: Server is DOWNDISASTERServer is not available.
The check directive must be enabled in HAProxy server configurationHAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: Redispatched requests per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},{#SERVER_NAME},wredis]60DISABLEDNumber of times a request was redispatched to a different backend.
The redispatch rate metric tracks the number of times a client connection was unable to reach its original target, and was subsequently sent to a different server. If a client holds a cookie referencing a backend server that is down, the default action is to respond to the client with a 502 status code. However, if is enabled option redispatch in haproxy.cfg, the request will be sent to any available backend server and the cookie will be ignored.HAProxy Backend [{#BACKEND_NAME}]CHANGE_PER_SECONDHAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}]: Retried connections per secondZABBIX_ACTIVEhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},{#SERVER_NAME},wretr]60DISABLEDNumber of times a connection was retried.
Some dropped or timed-out connections are to be expected when connecting to a backend server. The retry rate represents the number of times a connection to a backend server was retried. This metric is usually non-zero under normal operating conditions. Should you begin to see more retries than usual, it is likely that other metrics will also change, including econ and eresp.
Tracking the retry rate in addition to the above two error metrics can shine some light on the true cause of an increase in errorsHAProxy Backend [{#BACKEND_NAME}]CHANGE_PER_SECONDHAProxy Server [{#BACKEND_NAME}/{#SERVER_NAME}] Response time and time in queue95388E3CRIGHT- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},{#SERVER_NAME},qtime]
1F63100- HAProxyhaproxy.stats[{$HAPROXY_SOCK},{#BACKEND_NAME},{#SERVER_NAME},rtime]
{$HAPROXY_CONFIG}/etc/haproxy/haproxy.cfg{$HAPROXY_SOCK}/var/lib/haproxy/statsService state0Down1Up