# Kubernetes Attributes Processor | Status | | | ------------- |-----------| | Stability | [development]: profiles | | | [beta]: logs, metrics, traces | | Distributions | [contrib], [k8s] | | Issues | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aopen%20label%3Aprocessor%2Fk8sattributes%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aprocessor%2Fk8sattributes) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aclosed%20label%3Aprocessor%2Fk8sattributes%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aprocessor%2Fk8sattributes) | | [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@dmitryax](https://www.github.com/dmitryax), [@fatsheep9146](https://www.github.com/fatsheep9146), [@TylerHelmuth](https://www.github.com/TylerHelmuth), [@ChrsMark](https://www.github.com/ChrsMark) | | Emeritus | [@rmfitzpatrick](https://www.github.com/rmfitzpatrick) | [development]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#development [beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta [contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib [k8s]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-k8s Kubernetes attributes processor allow automatic setting of spans, metrics and logs resource attributes with k8s metadata. The processor automatically discovers k8s resources (pods), extracts metadata from them and adds the extracted metadata to the relevant spans, metrics and logs as resource attributes. The processor uses the kubernetes API to discover all pods running in a cluster, keeps a record of their IP addresses, pod UIDs and interesting metadata. The rules for associating the data passing through the processor (spans, metrics and logs) with specific Pod Metadata are configured via "pod_association" key. It represents a list of associations that are executed in the specified order until the first one is able to do the match. ## Configuration The processor stores the list of running pods and the associated metadata. When it sees a datapoint (log, trace or metric), it will try to associate the datapoint to the pod from where the datapoint originated, so we can add the relevant pod metadata to the datapoint. By default, it associates the incoming connection IP to the Pod IP. But for cases where this approach doesn't work (sending through a proxy, etc.), a custom association rule can be specified. Each association is specified as a list of sources of associations. The maximum number of sources within an association is 4. A source is a rule that matches metadata from the datapoint to pod metadata. In order to get an association applied, all the sources specified need to match. Each sources rule is specified as a pair of `from` (representing the rule type) and `name` (representing the attribute name if `from` is set to `resource_attribute`). The following rule types are available: - `connection`: Takes the IP attribute from connection context (if available). In this case the processor must appear before any batching or tail sampling, which remove this information. - `resource_attribute`: Allows specifying the attribute name to lookup in the list of attributes of the received Resource. Semantic convention should be used for naming. Example for a pod association configuration: ```yaml pod_association: # below association takes a look at the datapoint's k8s.pod.ip resource attribute and tries to match it with # the pod having the same attribute. - sources: - from: resource_attribute name: k8s.pod.ip # below association matches for pair `k8s.pod.name` and `k8s.namespace.name` - sources: - from: resource_attribute name: k8s.pod.name - from: resource_attribute name: k8s.namespace.name ``` If Pod association rules are not configured, resources are associated with metadata only by connection's IP Address. Which metadata to collect is determined by `metadata` configuration that defines list of resource attributes to be added. Items in the list called exactly the same as the resource attributes that will be added. The following attributes are added by default: - k8s.namespace.name - k8s.pod.name - k8s.pod.uid - k8s.pod.start_time - k8s.deployment.name - k8s.node.name These attributes are also available for the use within association rules by default. The `metadata` section can also be extended with additional attributes which, if present in the `metadata` section, are then also available for the use within association rules. Available attributes are: - k8s.namespace.name - k8s.pod.name - k8s.pod.hostname - k8s.pod.ip - k8s.pod.start_time - k8s.pod.uid - k8s.replicaset.uid - k8s.replicaset.name - k8s.deployment.uid - k8s.deployment.name - k8s.daemonset.uid - k8s.daemonset.name - k8s.statefulset.uid - k8s.statefulset.name - k8s.cronjob.uid - k8s.cronjob.name - k8s.job.uid - k8s.job.name - k8s.node.name - k8s.cluster.uid - Any tags extracted from the pod labels and annotations, as described in [extracting attributes from pod labels and annotations](#extracting-attributes-from-pod-labels-and-annotations) Not all the attributes are guaranteed to be added. Only attribute names from `metadata` should be used for pod_association's `resource_attribute`, because empty or non-existing values will be ignored. Additional container level attributes can be extracted. If a pod contains more than one container, either the `container.id`, or the `k8s.container.name` attribute must be provided in the incoming resource attributes to correctly associate the matching container to the resource: 1. If the `container.id` resource attribute is provided, the following additional attributes will be available: - k8s.container.name - container.image.name - container.image.tag - container.image.repo_digests (if k8s CRI populates [repository digest field](https://github.com/open-telemetry/semantic-conventions/blob/v1.26.0/model/registry/container.yaml#L60-L71)) 2. If the `k8s.container.name` resource attribute is provided, the following additional attributes will be available: - container.id (if the `k8s.container.restart_count` resource attribute is not provided, it's not guaranteed to get the right container ID.) - container.image.name - container.image.tag - container.image.repo_digests (if k8s CRI populates [repository digest field](https://github.com/open-telemetry/semantic-conventions/blob/v1.26.0/model/registry/container.yaml#L60-L71)) 3. If the `k8s.container.restart_count` resource attribute is provided, it can be used to associate with a particular container instance. If it's not set, the latest container instance will be used: - container.id (not added by default, has to be specified in `metadata`) Please note, however, that container level attributes can't be used for source rules in the pod_association. Example for extracting container level attributes: ```yaml pod_association: - sources: - from: connection extract: metadata: - k8s.pod.name - k8s.pod.uid - container.image.name - container.image.tag - k8s.container.name ``` The previous configuration attaches the attributes listed in the `metadata` section to all resources received by a matching pod with the `k8s.container.name` attribute being present. For example, when the following trace ```json { "name": "lets-go", "context": { "trace_id": "0x5b8aa5a2d2c872e8321cf37308d69df2", "span_id": "0x051581bf3cb55c13" }, "parent_id": null, "start_time": "2022-04-29T18:52:58.114201Z", "end_time": "2022-04-29T18:52:58.114687Z", "attributes": { "k8s.container.name": "telemetrygen" } } ``` is sent to the collector by the following pod, ```yaml apiVersion: v1 kind: Pod metadata: annotations: workload: deployment name: telemetrygen-pod namespace: e2ek8senrichment uid: 038e2267-b473-489b-b48c-46bafdb852eb spec: containers: - command: - /telemetrygen - traces - --otlp-insecure - --otlp-endpoint=otelcollector.svc.cluster.local:4317 - --duration=10s - --rate=1 - --otlp-attributes=k8s.container.name="telemetrygen" image: ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:0.112.0@sha256:b248ef911f93ae27cbbc85056d1ffacc87fd941bbdc2ffd951b6df8df72b8096 name: telemetrygen status: podIP: 10.244.0.11 ``` the processor associates the received trace to the pod, based on the connection IP, and add those attributes to the resulting span: ```json { "name": "lets-go", "context": { "trace_id": "0x5b8aa5a2d2c872e8321cf37308d69df2", "span_id": "0x051581bf3cb55c13" }, "parent_id": null, "start_time": "2022-04-29T18:52:58.114201Z", "end_time": "2022-04-29T18:52:58.114687Z", "attributes": { "k8s.container.name": "telemetrygen", "k8s.pod.name": "telemetrygen-pod", "k8s.pod.uid": "038e2267-b473-489b-b48c-46bafdb852eb", "container.image.name": "telemetrygen", "container.image.tag": "0.112.0", "container.image.repo_digests": ["ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen@sha256:b248ef911f93ae27cbbc85056d1ffacc87fd941bbdc2ffd951b6df8df72b8096"] } } ``` By default, the processor will be ready as soon as it starts, even if no metadata has been fetched yet. If data is sent to this processor before the metadata is synced, there will be no metadata to enrich the data with. To wait for the metadata to be synced before the processor is ready, set the `wait_for_metadata` option to `true`. Then the processor will not be ready until the metadata is fully synced. As a result, the start-up of the Collector will be blocked. If the metadata cannot be synced, the Collector will ultimately fail to start. If a timeout is reached, the processor will fail to start and return an error, which will cause the collector to exit. The timeout defaults to 10s and can be configured with the `metadata_sync_timeout` option. example for setting the processor to wait for metadata to be synced before it is ready: ```yaml wait_for_metadata: true wait_for_metadata_timeout: 10s ``` ## Extracting attributes from pod labels and annotations The k8sattributesprocessor can also set resource attributes from k8s labels and annotations of pods, namespaces and nodes. The config for associating the data passing through the processor (spans, metrics and logs) with specific Pod/Namespace/Node annotations/labels is configured via "annotations" and "labels" keys. This config represents a list of annotations/labels that are extracted from pods/namespaces/nodes and added to spans, metrics and logs. Each item is specified as a config of tag_name (representing the tag name to tag the spans with), key (representing the key used to extract value) and from (representing the kubernetes object used to extract the value). The "from" field has only three possible values "pod", "namespace" and "node" and defaults to "pod" if none is specified. A few examples to use this config are as follows: ```yaml extract: annotations: - tag_name: a1 # extracts value of annotation from pods with key `annotation-one` and inserts it as a tag with key `a1` key: annotation-one from: pod - tag_name: a2 # extracts value of annotation from namespaces with key `annotation-two` with regexp and inserts it as a tag with key `a2` key: annotation-two regex: field=(?P.+) from: namespace - tag_name: a3 # extracts value of annotation from nodes with key `annotation-three` with regexp and inserts it as a tag with key `a3` key: annotation-three regex: field=(?P.+) from: node labels: - tag_name: l1 # extracts value of label from namespaces with key `label1` and inserts it as a tag with key `l1` key: label1 from: namespace - tag_name: l2 # extracts value of label from pods with key `label2` with regexp and inserts it as a tag with key `l2` key: label2 regex: field=(?P.+) from: pod - tag_name: l3 # extracts value of label from nodes with key `label3` and inserts it as a tag with key `l3` key: label3 from: node ``` ### Config example ```yaml k8sattributes: k8sattributes/2: auth_type: "serviceAccount" passthrough: false filter: # only retrieve pods running on the same node as the collector node_from_env_var: KUBE_NODE_NAME extract: # The attributes provided in 'metadata' will be added to associated resources metadata: - k8s.pod.name - k8s.pod.uid - k8s.deployment.name - k8s.namespace.name - k8s.node.name - k8s.pod.start_time labels: # This label extraction rule takes the value 'app.kubernetes.io/component' label and maps it to the 'app.label.component' attribute which will be added to the associated resources - tag_name: app.label.component key: app.kubernetes.io/component from: pod pod_association: - sources: # This rule associates all resources containing the 'k8s.pod.ip' attribute with the matching pods. If this attribute is not present in the resource, this rule will not be able to find the matching pod. - from: resource_attribute name: k8s.pod.ip - sources: # This rule associates all resources containing the 'k8s.pod.uid' attribute with the matching pods. If this attribute is not present in the resource, this rule will not be able to find the matching pod. - from: resource_attribute name: k8s.pod.uid - sources: # This rule will use the IP from the incoming connection from which the resource is received, and find the matching pod, based on the 'pod.status.podIP' of the observed pods - from: connection ``` ## Role-based access control ## Cluster-scoped RBAC If you'd like to set up the k8sattributesprocessor to receive telemetry from across namespaces, it will need `get`, `watch` and `list` permissions on both `pods` and `namespaces` resources, for all namespaces and pods included in the configured filters. Additionally, when using `k8s.deployment.name` (which is enabled by default) or `k8s.deployment.uid` the processor also needs `get`, `watch` and `list` permissions for `replicasets` resources. When using `k8s.node.uid` or extracting metadata from `node`, the processor needs `get`, `watch` and `list` permissions for `nodes` resources. Here is an example of a `ClusterRole` to give a `ServiceAccount` the necessary permissions for all pods, nodes, and namespaces in the cluster (replace `` with a namespace where collector is deployed): ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: collector namespace: --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otel-collector rules: - apiGroups: [""] resources: ["pods", "namespaces", "nodes"] verbs: ["get", "watch", "list"] - apiGroups: ["apps"] resources: ["replicasets"] verbs: ["get", "list", "watch"] - apiGroups: ["extensions"] resources: ["replicasets"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otel-collector subjects: - kind: ServiceAccount name: collector namespace: roleRef: kind: ClusterRole name: otel-collector apiGroup: rbac.authorization.k8s.io ``` ### Namespace-scoped RBAC When running the k8sattributesprocessor to receive telemetry traffic from pods in a specific namespace, you can use a k8s `Role` and `Rolebinding` to provide collector access to query pods and replicasets in the namespace. This would require setting the `filter::namespace` config as shown below. ```yaml k8sattributes: filter: namespace: ``` With the namespace filter set, the processor will only look up pods and replicasets in the selected namespace. Note that with just a role binding, the processor cannot query metadata such as labels and annotations from k8s `nodes` and `namespaces` which are cluster-scoped objects. This also means that the processor cannot set the value for `k8s.cluster.uid` attribute if enabled, since the `k8s.cluster.uid` attribute is set to the uid of the namespace `kube-system` which is not queryable with namespaced rbac. Example `Role` and `RoleBinding` to create in the namespace being watched. ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: otel-collector namespace: --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: otel-collector namespace: rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "watch", "list"] - apiGroups: ["apps"] resources: ["replicasets"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: otel-collector namespace: subjects: - kind: ServiceAccount name: otel-collector namespace: roleRef: kind: Role name: otel-collector apiGroup: rbac.authorization.k8s.io ``` ## Deployment scenarios The processor can be used in collectors deployed both as an agent (Kubernetes DaemonSet) or as a gateway (Kubernetes Deployment). ### As an agent When running as an agent, the processor detects IP addresses of pods sending spans, metrics or logs to the agent and uses this information to extract metadata from pods. When running as an agent, it is important to apply a discovery filter so that the processor only discovers pods from the same host that it is running on. Not using such a filter can result in unnecessary resource usage especially on very large clusters. Once the filter is applied, each processor will only query the k8s API for pods running on its own node. Node filter can be applied by setting the `filter.node` config option to the name of a k8s node. While this works as expected, it cannot be used to automatically filter pods by the same node that the processor is running on in most cases as it is not know before hand which node a pod will be scheduled on. Luckily, kubernetes has a solution for this called the downward API. To automatically filter pods by the node the processor is running on, you'll need to complete the following steps: 1. Use the downward API to inject the node name as an environment variable. Add the following snippet under the pod env section of the OpenTelemetry container. ```yaml spec: containers: - env: - name: KUBE_NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName ``` This will inject a new environment variable to the OpenTelemetry container with the value as the name of the node the pod was scheduled to run on. 2. Set "filter.node_from_env_var" to the name of the environment variable holding the node name. ```yaml k8sattributes: filter: node_from_env_var: KUBE_NODE_NAME # this should be same as the var name used in previous step ``` This will restrict each OpenTelemetry agent to query pods running on the same node only dramatically reducing resource requirements for very large clusters. ### As a gateway When running as a gateway, the processor cannot correctly detect the IP address of the pods generating the telemetry data without any of the well-known IP attributes, when it receives them from an agent instead of receiving them directly from the pods. To workaround this issue, agents deployed with the k8sattributes processor can be configured to detect the IP addresses and forward them along with the telemetry data resources. Collector can then match this IP address with k8s pods and enrich the records with the metadata. In order to set this up, you'll need to complete the following steps: 1. Setup agents in passthrough mode Configure the agents' k8sattributes processors to run in passthrough mode. ```yaml # k8sattributes config for agent k8sattributes: passthrough: true ``` This will ensure that the agents detect the IP address as add it as an attribute to all telemetry resources. Agents will not make any k8s API calls, do any discovery of pods or extract any metadata. 2. Configure the collector as usual No special configuration changes are needed to be made on the collector. It'll automatically detect the IP address of spans, logs and metrics sent by the agents as well as directly by other services/pods. ## Caveats There are some edge-cases and scenarios where k8sattributes will not work properly. ### Host networking mode The processor cannot correct identify pods running in the host network mode and enriching telemetry data generated by such pods is not supported at the moment, unless the association rule is not based on IP attribute. ### As a sidecar The processor does not support detecting containers from the same pods when running as a sidecar. While this can be done, we think it is simpler to just use the kubernetes downward API to inject environment variables into the pods and directly use their values as tags. ## Timestamp Format By default, the `k8s.pod.start_time` uses [Time.MarshalText()](https://pkg.go.dev/time#Time.MarshalText) to format the timestamp value as an RFC3339 compliant timestamp. ## Feature Gate ### `k8sattr.fieldExtractConfigRegex.disallow` The `k8sattr.fieldExtractConfigRegex.disallow` [feature gate](https://github.com/open-telemetry/opentelemetry-collector/blob/main/featuregate/README.md#collector-feature-gates) disallows the usage of the `extract.annotations.regex` and `extract.labels.regex` fields. The validation performed on the configuration will fail, if at least one of the parameters is set (non-empty) and `k8sattr.fieldExtractConfigRegex.disallow` is set to `true` (default `false`). #### Example Usage The following config with the feature gate set will lead to validation error: `config.yaml`: ```yaml extract: labels: regex: annotations: regex: ``` Run collector: `./otelcol --config config.yaml --feature-gates=k8sattr.fieldExtractConfigRegex.disallow` #### Migration Deprecation of the `extract.annotations.regex` and `extract.labels.regex` fields means that it is recommended to use the `ExtractPatterns` function from the transform processor instead. To convert your current configuration please check the `ExtractPatterns` function [documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/ottlfuncs#extractpatterns). You should use the `pattern` parameter of `ExtractPatterns` instead of using the `extract.annotations.regex` and `extract.labels.regex` fields. ##### Example The following configuration of `k8sattributes processor`: `config.yaml`: ```yaml annotations: - tag_name: a2 # extracts value of annotation with key `annotation2` with regexp and inserts it as a tag with key `a2` key: annotation2 regex: field=(?P.+) from: pod ``` can be converted with the usage of `ExtractPatterns` function: ```yaml - set(cache["annotations"], ExtractPatterns(attributes["k8s.pod.annotations["annotation2"], "field=(?P.+))") - set(k8s.pod.annotations["a2"], cache["annotations"]["value"]) ```