--- title: prometheus keywords: - Apache APISIX - API Gateway - Plugin - Prometheus description: The prometheus Plugin provides the capability to integrate APISIX with Prometheus for metric collection and continuous monitoring. --- ## Description The `prometheus` Plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io). After enabling the Plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs. ## Static Configurations By default, `prometheus` configurations are pre-configured in the [default configuration](https://github.com/apache/apisix/blob/master/apisix/cli/config.lua). To customize these values, add the corresponding configurations to `config.yaml`. For example: ```yaml plugin_attr: prometheus: # Plugin: prometheus attributes export_uri: /apisix/prometheus/metrics # Set the URI for the Prometheus metrics endpoint. metric_prefix: apisix_ # Set the prefix for Prometheus metrics generated by APISIX. enable_export_server: true # Enable the Prometheus export server. export_addr: # Set the address for the Prometheus export server. ip: 127.0.0.1 # Set the IP. port: 9091 # Set the port. # metrics: # Create extra labels for metrics. # http_status: # These metrics will be prefixed with `apisix_`. # extra_labels: # Set the extra labels for http_status metrics. # - upstream_addr: $upstream_addr # - status: $upstream_status # expire: 0 # The expiration time of metrics in seconds. # 0 means the metrics will not expire. # http_latency: # extra_labels: # Set the extra labels for http_latency metrics. # - upstream_addr: $upstream_addr # expire: 0 # The expiration time of metrics in seconds. # 0 means the metrics will not expire. # bandwidth: # extra_labels: # Set the extra labels for bandwidth metrics. # - upstream_addr: $upstream_addr # expire: 0 # The expiration time of metrics in seconds. # 0 means the metrics will not expire. # default_buckets: # Set the default buckets for the `http_latency` metrics histogram. # - 10 # - 50 # - 100 # - 200 # - 500 # - 1000 # - 2000 # - 5000 # - 10000 # - 30000 # - 60000 # - 500 ``` You can use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) to create `extra_labels`. See [add extra labels](#add-extra-labels-for-metrics). Reload APISIX for changes to take effect. ## Attribute | Name | Type | Required | Default | Valid values | Description | | ------------- | ------- | -------- | ------- | ------------ | ------------------------------------------ | | prefer_name | boolean | | False | | If true, export Route/Service name instead of their ID in Prometheus metrics. | ## Metrics There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/). The following metrics are exported by the `prometheus` Plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data. | Name | Type | Description | | ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | apisix_bandwidth | counter | Total amount of traffic flowing through APISIX in bytes. | | apisix_etcd_modify_indexes | gauge | Number of changes to etcd by APISIX keys. | | apisix_batch_process_entries | gauge | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging Plugins. | | apisix_etcd_reachable | gauge | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable. | | apisix_http_status | counter | HTTP status codes returned from upstream Services. | | apisix_http_requests_total | gauge | Number of HTTP requests from clients. | | apisix_nginx_http_current_connections | gauge | Number of current connections with clients. | | apisix_nginx_metric_errors_total | counter | Total number of `nginx-lua-prometheus` errors. | | apisix_http_latency | histogram | HTTP request latency in milliseconds. | | apisix_node_info | gauge | Information of the APISIX node, such as host name and the current APISIX version. | | apisix_shared_dict_capacity_bytes | gauge | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). | | apisix_shared_dict_free_space_bytes | gauge | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). | | apisix_upstream_status | gauge | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy. | | apisix_stream_connection_total | counter | Total number of connections handled per Stream Route. | ## Labels [Labels](https://prometheus.io/docs/practices/naming/#labels) are attributes of metrics that are used to differentiate metrics. For example, the `apisix_http_status` metric can be labeled with `route` information to identify which Route the HTTP status originates from. The following are labels for a non-exhaustive list of APISIX metrics and their descriptions. ### Labels for `apisix_http_status` The following labels are used to differentiate `apisix_http_status` metrics. | Name | Description | | ------------ | ----------------------------------------------------------------------------------------------------------------------------- | | code | HTTP response code returned by the upstream node. | | route | ID of the Route that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. | | matched_uri | URI of the Route that matches the request. Default to an empty string if a request does not match any Route. | | matched_host | Host of the Route that matches the request. Default to an empty string if a request does not match any Route, or host is not configured on the Route. | | service | ID of the Service that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. | | consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. | | node | IP address of the upstream node. | | request_type | traditional_http / ai_chat / ai_stream | | llm_model | For non-traditional_http requests, name of the llm_model | ### Labels for `apisix_bandwidth` The following labels are used to differentiate `apisix_bandwidth` metrics. | Name | Description | | ---------- | ----------------------------------------------------------------------------------------------------------------------------- | | type | Type of traffic, `egress` or `ingress`. | | route | ID of the Route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. | | service | ID of the Service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. | | consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. | | node | IP address of the upstream node. | | request_type | traditional_http / ai_chat / ai_stream | | llm_model | For non-traditional_http requests, name of the llm_model | ### Labels for `apisix_llm_latency` | Name | Description | | ---------- | ----------------------------------------------------------------------------------------------------------------------------- | | | route_id | ID of the Route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. | | service_id | ID of the Service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. | | consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. | | node | IP address of the upstream node. | | request_type | traditional_http / ai_chat / ai_stream | | llm_model | For non-traditional_http requests, name of the llm_model | ### Labels for `apisix_llm_active_connections` | Name | Description | | ---------- | ----------------------------------------------------------------------------------------------------------------------------- | | route | Name of the Route that bandwidth corresponds to. Default to an empty string if a request does not match any Route. | | route_id | ID of the Route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. | | matched_uri | URI of the Route that matches the request. Default to an empty string if a request does not match any Route. | | matched_host | Host of the Route that matches the request. Default to an empty string if a request does not match any Route, or host is not configured on the Route. | | service | Name of the Service that bandwidth corresponds to. Default to the configured value of host on the Route if the matched Route does not belong to any Service. | | service_id | ID of the Service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. | | consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. | | node | IP address of the upstream node. | | request_type | traditional_http / ai_chat / ai_stream | | llm_model | For non-traditional_http requests, name of the llm_model | ### Labels for `apisix_llm_completion_tokens` | Name | Description | | ---------- | ----------------------------------------------------------------------------------------------------------------------------- | | | route_id | ID of the Route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. | | service_id | ID of the Service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. | | consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. | | node | IP address of the upstream node. | | request_type | traditional_http / ai_chat / ai_stream | | llm_model | For non-traditional_http requests, name of the llm_model | ### Labels for `apisix_llm_prompt_tokens` | Name | Description | | ---------- | ----------------------------------------------------------------------------------------------------------------------------- | | | route_id | ID of the Route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. | | service_id | ID of the Service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. | | consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. | | node | IP address of the upstream node. | | request_type | traditional_http / ai_chat / ai_stream | | llm_model | For non-traditional_http requests, name of the llm_model | ### Labels for `apisix_http_latency` The following labels are used to differentiate `apisix_http_latency` metrics. | Name | Description | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------- | | type | Type of latencies. See [latency types](#latency-types) for details. | | route | ID of the Route that latencies correspond to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. | | service | ID of the Service that latencies correspond to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. | | consumer | Name of the Consumer associated with latencies. Default to an empty string if no Consumer is associated with the request. | | node | IP address of the upstream node associated with latencies. | | request_type | traditional_http / ai_chat / ai_stream | | llm_model | For non-traditional_http requests, name of the llm_model | #### Latency Types `apisix_http_latency` can be labeled with one of the three types: * `request` represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client. * `upstream` represents the time elapsed waiting on responses from the upstream Service. * `apisix` represents the difference between the `request` latency and `upstream` latency. In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows: ```text APISIX latency = downstream request time - upstream response time = downstream traffic latency + NGINX latency ``` ### Labels for `apisix_upstream_status` The following labels are used to differentiate `apisix_upstream_status` metrics. | Name | Description | | ---------- | --------------------------------------------------------------------------------------------------- | | name | Resource ID corresponding to the upstream configured with health checks, such as `/apisix/routes/1` and `/apisix/upstreams/1`. | | ip | IP address of the upstream node. | | port | Port number of the node. | ## Examples The examples below demonstrate how you can work with the `prometheus` Plugin for different scenarios. ### Get APISIX Metrics The following example demonstrates how you can get metrics from APISIX. The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the [static configuration](#static-configurations). If you would like to customize these configuration, update `config.yaml` and reload APISIX. If you deploy APISIX in a containerized environment and would like to access the Prometheus metrics endpoint externally, update the configuration file as follows and reload APISIX: ```yaml title="conf/config.yaml" plugin_attr: prometheus: export_addr: ip: 0.0.0.0 ``` Send a request to the APISIX Prometheus metrics endpoint: ```shell curl "http://127.0.0.1:9091/apisix/prometheus/metrics" ``` You should see an output similar to the following: ```text # HELP apisix_bandwidth Total bandwidth in bytes consumed per Service in Apisix # TYPE apisix_bandwidth counter apisix_bandwidth{type="egress",route="",service="",consumer="",node="",request_type="traditional_http",request_llm_model="",llm_model=""} 8417 apisix_bandwidth{type="egress",route="1",service="",consumer="",node="127.0.0.1",request_type="traditional_http",request_llm_model="",llm_model=""} 1420 apisix_bandwidth{type="egress",route="2",service="",consumer="",node="127.0.0.1",request_type="traditional_http",request_llm_model="",llm_model=""} 1420 apisix_bandwidth{type="ingress",route="",service="",consumer="",node="",request_type="traditional_http",request_llm_model="",llm_model=""} 189 apisix_bandwidth{type="ingress",route="1",service="",consumer="",node="127.0.0.1",request_type="traditional_http",request_llm_model="",llm_model=""} 332 apisix_bandwidth{type="ingress",route="2",service="",consumer="",node="127.0.0.1",request_type="traditional_http",request_llm_model="",llm_model=""} 332 # HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys # TYPE apisix_etcd_modify_indexes gauge apisix_etcd_modify_indexes{key="consumers"} 0 apisix_etcd_modify_indexes{key="global_rules"} 0 ... ``` ### Expose APISIX Metrics on Public API Endpoint The following example demonstrates how you can disable the Prometheus export server that, by default, exposes an endpoint on port `9091`, and expose APISIX Prometheus metrics on a new public API endpoint on port `9080`, which APISIX uses to listen to other client requests. Disable the Prometheus export server in the configuration file and reload APISIX for changes to take effect: ```yaml title="conf/config.yaml" plugin_attr: prometheus: enable_export_server: false ``` Next, create a Route with [`public-api`](../../../en/latest/plugins/public-api.md) Plugin and expose a public API endpoint for APISIX metrics: ```shell curl "http://127.0.0.1:9180/apisix/admin/routes/prometheus-metrics" -X PUT \ -H "X-API-KEY: ${admin_key}" \ -d '{ "uri": "/apisix/prometheus/metrics", "plugins": { "public-api": {} } }' ``` Send a request to the new metrics endpoint to verify: ```shell curl "http://127.0.0.1:9080/apisix/prometheus/metrics" ``` You should see an output similar to the following: ```text # HELP apisix_http_requests_total The total number of client requests since APISIX started # TYPE apisix_http_requests_total gauge apisix_http_requests_total 1 # HELP apisix_nginx_http_current_connections Number of HTTP connections # TYPE apisix_nginx_http_current_connections gauge apisix_nginx_http_current_connections{state="accepted"} 1 apisix_nginx_http_current_connections{state="active"} 1 apisix_nginx_http_current_connections{state="handled"} 1 apisix_nginx_http_current_connections{state="reading"} 0 apisix_nginx_http_current_connections{state="waiting"} 0 apisix_nginx_http_current_connections{state="writing"} 1 ... ``` ### Monitor Upstream Health Statuses The following example demonstrates how to monitor the health status of upstream nodes. Create a Route with the `prometheus` Plugin and configure upstream active health checks: ```shell curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ -H "X-API-KEY: ${admin_key}" \ -d '{ "id": "prometheus-route", "uri": "/get", "plugins": { "prometheus": {} }, "upstream": { "type": "roundrobin", "nodes": { "httpbin.org:80": 1, "127.0.0.1:20001": 1 }, "checks": { "active": { "timeout": 5, "http_path": "/status", "healthy": { "interval": 2, "successes": 1 }, "unhealthy": { "interval": 1, "http_failures": 2 } }, "passive": { "healthy": { "http_statuses": [200, 201], "successes": 3 }, "unhealthy": { "http_statuses": [500], "http_failures": 3, "tcp_failures": 3 } } } } }' ``` Send a request to the APISIX Prometheus metrics endpoint: ```shell curl "http://127.0.0.1:9091/apisix/prometheus/metrics" ``` You should see an output similar to the following: ```text # HELP apisix_upstream_status upstream status from health check # TYPE apisix_upstream_status gauge apisix_upstream_status{name="/apisix/routes/1",ip="54.237.103.220",port="80"} 1 apisix_upstream_status{name="/apisix/routes/1",ip="127.0.0.1",port="20001"} 0 ``` This shows that the upstream node `httpbin.org:80` is healthy and the upstream node `127.0.0.1:20001` is unhealthy. ### Add Extra Labels for Metrics The following example demonstrates how to add additional labels to metrics and use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) in label values. Currently, only the following metrics support extra labels: * apisix_http_status * apisix_http_latency * apisix_bandwidth Include the following configurations in the configuration file to add labels for metrics and reload APISIX for changes to take effect: ```yaml title="conf/config.yaml" plugin_attr: prometheus: # Plugin: prometheus metrics: # Create extra labels from the NGINX variables. http_status: extra_labels: # Set the extra labels for http_status metrics. - upstream_addr: $upstream_addr # Add an extra upstream_addr label with value being the NGINX variable $upstream_addr. - route_name: $route_name # Add an extra route_name label with value being the APISIX variable $route_name. ``` Note that if you define a variable in the label value but it does not correspond to any existing [APISIX variables](https://apisix.apache.org/docs/apisix/apisix-variable/) and [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html), the label value will default to an empty string. Create a Route with the `prometheus` Plugin: ```shell curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \ -H "X-API-KEY: ${admin_key}" \ -d '{ "id": "prometheus-route", Include the following configurations in the configuration file to add labels for metrics and reload APISIX for changes to take effect: "name": "extra-label", "plugins": { "prometheus": {} }, "upstream": { "nodes": { "httpbin.org:80": 1 } } }' ``` Send a request to the Route to verify: ```shell curl -i "http://127.0.0.1:9080/get" ``` You should see an `HTTP/1.1 200 OK` response. Send a request to the APISIX Prometheus metrics endpoint: ```shell curl "http://127.0.0.1:9091/apisix/prometheus/metrics" ``` You should see an output similar to the following: ```text # HELP apisix_http_status HTTP status codes per Service in APISIX # TYPE apisix_http_status counter apisix_http_status{code="200",route="1",matched_uri="/get",matched_host="",service="",consumer="",node="54.237.103.220",upstream_addr="54.237.103.220:80",route_name="extra-label"} 1 ``` ### Monitor TCP/UDP Traffic with Prometheus The following example demonstrates how to collect TCP/UDP traffic metrics in APISIX. Include the following configurations in `config.yaml` to enable stream proxy and `prometheus` Plugin for stream proxy. Reload APISIX for changes to take effect: ```yaml title="conf/config.yaml" apisix: proxy_mode: http&stream # Enable both L4 & L7 proxies stream_proxy: # Configure L4 proxy tcp: - 9100 # Set TCP proxy listening port udp: - 9200 # Set UDP proxy listening port stream_plugins: - prometheus # Enable prometheus for stream proxy ``` Create a Stream Route with the `prometheus` Plugin: ```shell curl "http://127.0.0.1:9180/apisix/admin/stream_routes" -X PUT \ -H "X-API-KEY: ${admin_key}" \ -d '{ Include the following configurations in `config.yaml` to enable stream proxy and enable `prometheus` Plugin for stream proxy. Reload APISIX for changes to take effect: "plugins": { "prometheus":{} }, "upstream": { "type": "roundrobin", "nodes": { "httpbin.org:80": 1 } } }' ``` Send a request to the Stream Route to verify: ```shell curl -i "http://127.0.0.1:9100" ``` You should see an `HTTP/1.1 200 OK` response. Send a request to the APISIX Prometheus metrics endpoint: ```shell curl "http://127.0.0.1:9091/apisix/prometheus/metrics" ``` You should see an output similar to the following: ```text # HELP apisix_stream_connection_total Total number of connections handled per Stream Route in APISIX # TYPE apisix_stream_connection_total counter apisix_stream_connection_total{route="1"} 1 ```