--- title: ResourceSet CRD description: Flux Operator API for generating FluxCD resources --- # ResourceSet CRD **ResourceSet** is a declarative API for generating a group of Kubernetes objects based on a matrix of input values and a set of templated resources. ## Example The following example shows a ResourceSet that generates an application instance consisting of a Flux HelmRelease and OCIRepository for each tenant with a specific version and replica count. ```yaml apiVersion: fluxcd.controlplane.io/v1 kind: ResourceSet metadata: name: podinfo namespace: default annotations: fluxcd.controlplane.io/reconcile: "enabled" fluxcd.controlplane.io/reconcileEvery: "30m" fluxcd.controlplane.io/reconcileTimeout: "5m" spec: commonMetadata: labels: app.kubernetes.io/name: podinfo inputs: - tenant: "team1" app: version: "6.7.x" replicas: 2 - tenant: "team2" app: version: "6.6.x" replicas: 3 resources: - apiVersion: source.toolkit.fluxcd.io/v1 kind: OCIRepository metadata: name: podinfo-<< inputs.tenant >> namespace: default spec: interval: 10m url: oci://ghcr.io/stefanprodan/charts/podinfo ref: semver: << inputs.app.version | quote >> - apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: podinfo-<< inputs.tenant >> namespace: default spec: interval: 1h releaseName: podinfo-<< inputs.tenant >> chartRef: kind: OCIRepository name: podinfo-<< inputs.tenant >> values: replicaCount: << inputs.app.replicas | int >> ``` You can run this example by saving the manifest into `podinfo.yaml`. **1.** Apply the ResourceSet on the cluster: ```shell kubectl apply -f podinfo.yaml ``` **2.** Wait for the ResourceSet to reconcile the generated resources: ```shell kubectl wait resourceset/podinfo --for=condition=ready --timeout=5m ``` **3.** Run `kubectl get resourceset` to see the status of the resource: ```console $ kubectl get resourceset NAME AGE READY STATUS podinfo 59s True Reconciliation finished in 52s ``` **4.** Run `kubectl describe resourceset` to see the reconciliation status conditions and events: ```console $ kubectl describe resourceset podinfo Status: Conditions: Last Transition Time: 2024-09-24T09:58:53Z Message: Reconciliation finished in 52s Observed Generation: 1 Reason: ReconciliationSucceeded Status: True Type: Ready Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ApplySucceeded 72s flux-operator HelmRelease/default/podinfo-team1 created HelmRelease/default/podinfo-team2 created OCIRepository/default/podinfo-team1 created OCIRepository/default/podinfo-team2 created Normal ReconciliationSucceeded 72s flux-operator Reconciliation finished in 52s ``` **5.** Run `kubectl events` to see the events generated by the flux-operator: ```shell kubectl events --for resourceset/podinfo ``` **6.** Run `kubectl delete` to remove the ResourceSet and its generated resources: ```shell kubectl delete resourceset podinfo ``` ## Writing a ResourceSet spec As with all other Kubernetes config, a ResourceSet needs `apiVersion`, `kind`, `metadata.name` and `metadata.namespace` fields. The name of a ResourceSet object must be a valid [DNS subdomain name](https://kubernetes.io/docs/concepts/overview/working-with-objects/names#dns-subdomain-names). A ResourceSet also needs a [`.spec` section](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#spec-and-status). ### Inputs configuration The `.spec.inputs` field is optional and specifies a list of input values to be used in the resources templates. An input value is a key-value pair of strings and structs, where the key is the input name which can be referenced in the resource templates using the `<< inputs.name >>` syntax. Example of static inputs: ```yaml spec: inputs: - tenant: team1 role: restricted - tenant: team2 role: privileged ``` The `.spec.inputsFrom` field is optional and specifies a list of [ResourceSetInputProvider](resourcesetinputprovider.md) references to objects that provide input values to the ResourceSet. It has the following subfields (exactly one of them must be set): - `apiVersion`: The API version of the referenced object. Must be `fluxcd.controlplane.io/v1`. Optional. - `kind`: The kind of the referenced object. Must be `ResourceSetInputProvider`. Optional. - `.name`: The name of a `ResourceSetInputProvider` object in the same namespace as the `ResourceSet`. Optional. - `.selector`: A label selector to select multiple `ResourceSetInputProvider` objects in the same namespace as the `ResourceSet`. Optional. Example of inputs generated from GitHub Pull Requests: ```yaml spec: inputsFrom: - apiVersion: fluxcd.controlplane.io/v1 kind: ResourceSetInputProvider name: podinfo-pull-requests ``` Example of inputs generated from multiple `ResourceSetInputProvider` objects via [Label Selectors](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors): ```yaml spec: inputsFrom: - apiVersion: fluxcd.controlplane.io/v1 kind: ResourceSetInputProvider selector: matchLabels: app: podinfo matchExpressions: - key: environment operator: In values: - dev - staging ``` At runtime, the operator will fetch the input values every time the `ResourceSetInputProvider` reconciler detects a change in the upstream source. When both `.spec.inputs` and `.spec.inputsFrom` are set, the resulting inputs are the flattened concatenation of the `.spec.inputs` input sets with the input sets from each selected `ResourceSetInputProvider` object. #### Input strategy By default, the resulting inputs are the flattened concatenation of the `.spec.inputs` input sets with the input sets from each selected `ResourceSetInputProvider` object. The `.spec.inputStrategy` field can be set to change this default behavior. The `.spec.inputStrategy` field has the following subfields: - `.name`: The name of the input strategy. If `.spec.inputStrategy` is not set, the behavior matches setting `.spec.inputStrategy.name` to `Flatten`. Supported values are `Flatten` and `Permute`. When `.spec.inputStrategy.name` is set to `Permute`, the resulting inputs are the Cartesian product of the input sets from each selected `ResourceSetInputProvider` object, and from the `ResourceSet` object itself if `.spec.inputs` is set. Therefore, the total amount of permutations, i.e. input sets, is the product of the number of input sets from each source, i.e. `A_1 x A_2 x ... x A_n`, where `A_i` is the number of input sets from the `i`-th source. **Note**: The combinatorial explosion of the `Permute` strategy can lead to a very large number of permutations and can impact the performance of the operator. It is recommended to use this strategy together with `ResourceSetInputProvider` filters to export only a single input set per `ResourceSetInputProvider` object. If the number of permutations exceeds `10000`, the operator will fail the reconciliation of the `ResourceSet` with a `Stalled` condition. When merging the input sets from each object, the `Permute` strategy places each input set under a key that is the normalized name of the object providing the input set. The normalization process is as follows: 1. Uppercase letters are converted to lowercase. 2. Spaces and punctuation (including `-`) are converted to underscores (`_`). 3. All characters not in the set [a-z0-9_] are removed. 4. The remaining string is split by underscores and the resulting non-empty words are rejoined with underscores. The name normalization applied by the `Permute` strategy ensures that the resulting key is compatible with the templating engine used in the resources templates. Example of the `Permute` strategy with two `ResourceSetInputProvider` objects, one selecting a single Git tag and the other selecting a single OCI tag: ```yaml apiVersion: fluxcd.controlplane.io/v1 kind: ResourceSet metadata: name: my-rset namespace: default spec: inputStrategy: name: Permute inputs: - id: id1 someField: foo - id: id2 someField: bar inputsFrom: - kind: ResourceSetInputProvider name: git-tag - kind: ResourceSetInputProvider name: oci-tag resources: - apiVersion: v1 kind: ConfigMap metadata: name: my-cm-<< inputs.id >> namespace: default data: rsetID: << inputs.my_rset.id | quote >> rsipGitID: << inputs.git_tag.id | quote >> rsipOCIID: << inputs.oci_tag.id | quote >> someField: << inputs.my_rset.someField | quote >> sha: << inputs.git_tag.sha | quote >> digest: << inputs.oci_tag.digest | quote >> ``` In the example above, the resulting input sets are similar to the following: ```yaml - id: "768965678" my_rset: id: id1 someField: foo git_tag: id: "8765674567" sha: bf5d6e01cf802734853f6f3417b237e3ad0ba35d oci_tag: id: "9876543210" digest: sha256:d4ec9861522d4961b2acac5a070ef4f92d732480dff2062c2f3a1dcf9a5d1e91 - id: "234567654" my_rset: id: id2 someField: bar git_tag: id: "8765674567" sha: bf5d6e01cf802734853f6f3417b237e3ad0ba35d oci_tag: id: "9876543210" digest: sha256:d4ec9861522d4961b2acac5a070ef4f92d732480dff2062c2f3a1dcf9a5d1e91 ``` This will generate two `ConfigMap` resources, one whose name will be `my-cm-768965678`, and the other `my-cm-234567654`, each containing the rendered `.data` according to the input set. ### Resources configuration The `.spec.resources` field is optional and specifies the list of Kubernetes resource to be generated and reconciled on the cluster. The resources can be templated using the `<< inputs.name >>` syntax. The templating engine is based on Go text template. The `<< >>` delimiters are used instead of `{{ }}` to avoid conflicts with Helm templating and allow ResourceSets to be included in Helm charts. Example of templated resources: ```yaml spec: inputs: - tenant: team1 role: admin - tenant: team2 role: cluster-admin resources: - apiVersion: v1 kind: Namespace metadata: name: << inputs.tenant >> - apiVersion: v1 kind: ServiceAccount metadata: name: flux namespace: << inputs.tenant >> - apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: flux namespace: << inputs.tenant >> subjects: - kind: ServiceAccount name: flux namespace: << inputs.tenant >> roleRef: kind: ClusterRole name: << inputs.role >> apiGroup: rbac.authorization.k8s.io ``` The above example will generate a `Namespace`, `ServiceAccount` and `RoleBinding` for each tenant with the specified role. #### Templating functions The templating engine supports [slim-sprig](https://go-task.github.io/slim-sprig/) functions. It is recommended to use the `quote` function when templating strings to avoid issues with special characters e.g. `<< inputs.version | quote >>`. When templating integers, use the `int` function to convert the string to an integer e.g. `<< inputs.replicas | int >>`. When using integer or boolean inputs as metadata label values, use the `quote` function to convert the value to a string e.g. `<< inputs.enabled | quote >>`. When templating nested fields, use the `toYaml` and `nindent` functions to properly format the string e.g.: ```yaml spec: inputs: - tenant: team1 layerSelector: mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip" operation: copy resources: - apiVersion: source.toolkit.fluxcd.io/v1 kind: OCIRepository metadata: name: << inputs.tenant >> spec: layerSelector: << inputs.layerSelector | toYaml | nindent 4 >> ``` To assign a default value when an input field is not specified, use `get` combined with `default`. E.g. using the namespace input as a default value for name: ```yaml spec: inputs: - namespace: team1 resources: - apiVersion: source.toolkit.fluxcd.io/v1 kind: OCIRepository metadata: name: << get inputs "name" | default inputs.namespace >> namespace: << inputs.namespace >> ``` In addition to the slim-sprig functions, a `slugify` function is available to normalize a string for use in a Kubernetes [label value](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) e.g. `<< inputs.tenant | slugify >>`. #### Resource deduplication The flux-operator deduplicates resources based on the `apiVersion`, `kind`, `metadata.name` and `metadata.namespace` fields. This allows defining shared resources that are applied only once, regardless of the number of inputs. Example of a shared Flux source: ```yaml spec: inputs: - tenant: "team1" replicas: "2" - tenant: "team2" replicas: "3" resources: - apiVersion: source.toolkit.fluxcd.io/v1 kind: OCIRepository metadata: name: podinfo namespace: default spec: interval: 10m url: oci://ghcr.io/stefanprodan/charts/podinfo ref: semver: '*' - apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: podinfo-<< inputs.tenant >> namespace: default spec: interval: 1h releaseName: podinfo-<< inputs.tenant >> chartRef: kind: OCIRepository name: podinfo values: replicaCount: << inputs.replicas | int >> ``` In the above example, the `OCIRepository` resource is created only once and referred by all `HelmRelease` resources. #### Copying data from existing ConfigMaps and Secrets To generate resources with data copied from existing ConfigMaps and Secrets, the `fluxcd.controlplane.io/copyFrom: namespace/name` annotation must be set in the ConfigMap or Secret template. Example of copying data from an existing ConfigMap and Secret: ```yaml spec: inputs: - tenant: "team1" - tenant: "team2" resources: - apiVersion: v1 kind: Namespace metadata: name: << inputs.tenant >> - apiVersion: v1 kind: ConfigMap metadata: name: runtime-info namespace: << inputs.tenant >> annotations: fluxcd.controlplane.io/copyFrom: "flux-system/runtime-info" - apiVersion: v1 kind: Secret metadata: name: docker-auth namespace: << inputs.tenant >> annotations: fluxcd.controlplane.io/copyFrom: "flux-system/docker-auth" ``` In the above example, a ConfigMap and a Secret are generated for each tenant with the data copied from the `runtime-info` and `docker-auth` ConfigMap and Secret from the `flux-system` namespace. If the source data changes, the operator will update the generated resources accordingly on the next [reconciliation interval](#reconciliation-configuration). Note that on [multi-tenant clusters](#role-based-access-control), the service account used by the ResourceSet must have the necessary permissions to read the ConfigMaps and Secrets from the source namespace. To trigger a reconciliation of the ResourceSet when changes occur in the source ConfigMap or Secret, you can set the following label on the source ConfigMap or Secret: ```yaml metadata: labels: reconcile.fluxcd.io/watch: Enabled ``` Note that neither this label nor any other metadata is copied to the generated ConfigMaps and Secrets, you must label every ConfigMap or Secret that you want to watch for changes. An alternative to labeling every ConfigMap or Secret is setting the `--watch-configs-label-selector=owner!=helm` flag in flux-operator, which allows watching all ConfigMaps and Secrets except for Helm storage Secrets. #### Conditional resource exclusion To exclude a resource based on input values, the `fluxcd.controlplane.io/reconcile` annotation can be set to `disabled` on the resource metadata. This will prevent the resource from being reconciled by the operator. Example of excluding a resource based on an input value: ```yaml spec: inputs: - tenant: "team1" - tenant: "team2" resources: - apiVersion: v1 kind: Namespace metadata: name: << inputs.tenant >> - apiVersion: v1 kind: ServiceAccount metadata: name: flux namespace: << inputs.tenant >> annotations: fluxcd.controlplane.io/reconcile: << if eq inputs.tenant "team1" >>enabled<< else >>disabled<< end >> ``` In the above example, the `ServiceAccount` resource is generated only for the `team1` tenant. #### Built-in input fields When computing all the input sets for generating the resource matrix, the operator adds a few built-in fields to each input set. Users cannot override these fields. Every input set contains a built-in `inputs.id` field that is a unique identifier for the input set amongst all the input sets generated for the ResourceSet. This field can be used in the resource templates to generate unique resource names. Every input set also contains the reference of the object providing those inputs: - `inputs.provider.apiVersion`: The API version of the object providing the inputs. - `inputs.provider.kind`: The kind of the object providing the inputs. - `inputs.provider.name`: The name of the object providing the inputs. - `inputs.provider.namespace`: The namespace of the object providing the inputs. In the case of inline inputs provided directly in the `ResourceSet` object through the `.spec.inputs` field, `inputs.provider.apiVersion` is `fluxcd.controlplane.io/v1` and `inputs.provider.kind` is `ResourceSet`. In the case of inputs provided through a `ResourceSetInputProvider` referenced in the `.spec.inputsFrom` field of a `ResourceSet`, `inputs.provider.apiVersion` is `fluxcd.controlplane.io/v1` and `inputs.provider.kind` is `ResourceSetInputProvider`. If the `.spec.inputStrategy.name` [field](#input-strategy) is set to `Permute`, the built-in fields above are nested under a key that is the normalized name of the object providing the inputs, i.e.: - `inputs..id` - `inputs..provider.apiVersion` - `inputs..provider.kind` - `inputs..provider.name` - `inputs..provider.namespace` And `inputs.id` will be derived from the permutation configuration, making each permutation have a unique ID. ### Resources template The `.spec.resourcesTemplate` field is optional and offers an alternative to the `.spec.resources`. The `.spec.resourcesTemplate` is a single string that contains the multi-document YAML of the resources definitions. This field can be used for complex templating scenarios with the trade-off of reduced readability. Note that when both `.spec.resources` and `.spec.resourcesTemplate` are set, the resulting resources are the union of the two. If duplicate resources are defined in both fields, the resources from `.spec.resources` take precedence. Example of a template containing conditional and repeated blocks: ```yaml spec: inputs: - bundle: addons decryption: false components: - ingress-nginx - cert-manager - bundle: apps decryption: true components: - frontend - backend resourcesTemplate: | --- apiVersion: source.toolkit.fluxcd.io/v1 kind: OCIRepository metadata: name: << inputs.bundle >> namespace: flux-system spec: interval: 10m url: oci://registry.example.com/<< inputs.bundle >> <<- range $component := inputs.components >> --- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: << $component >> namespace: flux-system spec: interval: 1h prune: true <<- if inputs.decryption >> decryption: provider: sops secretRef: name: << inputs.bundle >>-sops <<- end >> sourceRef: kind: OCIRepository name: << inputs.bundle >> path: ./<< $component >> <<- end >> ``` The above example generates two `OCIRepository` resources (one for each bundle) and four `Kustomization` resources (one for each component in each bundle). ### Common metadata The `.spec.commonMetadata` field is optional and specifies common metadata to be applied to all resources. It has two optional fields: - `labels`: A map used for setting [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) on an object. Any existing label will be overridden if it matches with a key in this map. - `annotations`: A map used for setting [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) on an object. Any existing annotation will be overridden if it matches with a key in this map. Example common metadata: ```yaml spec: commonMetadata: labels: app.kubernetes.io/name: podinfo annotations: fluxcd.controlplane.io/prune: disabled ``` In the above example, all resources generated by the ResourceSet will not be pruned by the [garbage collection](#garbage-collection) process as the `fluxcd.controlplane.io/prune` annotation is set to `disabled`. ### Dependency management `.spec.dependsOn` is an optional list used to refer to Kubernetes objects that the ResourceSet depends on. If specified, then the ResourceSet is reconciled after the referred objects exist in the cluster. A dependency is a reference to a Kubernetes object with the following fields: - `apiVersion`: The API version of the referred object (required). - `kind`: The kind of the referred object (required). - `name`: The name of the referred object (required). - `namespace`: The namespace of the referred object (optional). - `ready`: A boolean indicating if the referred object must have the `Ready` status condition set to `True` (optional, default is `false`). - `readyExpr`: A [CEL expression](#cel-readiness-expressions) that evaluates to a boolean indicating if the referred object is ready (optional). Example of conditional reconciliation based on the existence of CustomResourceDefinitions and the readiness of a ResourceSet: ```yaml spec: dependsOn: - apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition name: helmreleases.helm.toolkit.fluxcd.io ready: true - apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition name: servicemonitors.monitoring.coreos.com - apiVersion: fluxcd.controlplane.io/v1 kind: ResourceSet name: cluster-addons namespace: flux-system ready: true ``` Note that is recommended to define dependencies on CustomResourceDefinitions if the ResourceSet deploys Flux HelmReleases which contain custom resources. When the dependencies are not met, the flux-operator will reevaluate the requirements every five seconds and reconcile the ResourceSet when the dependencies are satisfied. Failed dependencies are reported in the ResourceSet `Ready` [status condition](#ResourceSet-Status), in log messages and Kubernetes events. #### CEL readiness expressions The `readyExpr` field allows for more complex readiness checks and can be used for gating the reconciliation of a ResourceSet based on the evaluation of the [CEL](https://cel.dev) expression. The expression is evaluated in the context of the referred object and has access to all the fields of the object, including the status conditions and the status subfields. The expression must evaluate to a boolean value, any syntax or runtime errors will be reported in the ResourceSet status conditions. Example readiness expression: ```yaml spec: dependsOn: - apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster name: my-cluster namespace: dev ready: true readyExpr: | metadata.generation == status.observedGeneration && status.controlPlaneReady == true - apiVersion: v1 kind: Secret name: my-gate namespace: dev ready: true readyExpr: | string(base64.decode(data.gate)) == 'opened' ``` For testing the CEL expressions, you can use the [CEL playground](https://playcel.undistro.io/). ### Reconciliation configuration The reconciliation behavior of a ResourceSet can be configured using the following annotations: - `fluxcd.controlplane.io/reconcile`: Enable or disable the reconciliation loop. Default is `enabled`, set to `disabled` to pause the reconciliation. - `fluxcd.controlplane.io/reconcileEvery`: Set the reconciliation interval used for drift detection and correction. Default is `1h`. - `fluxcd.controlplane.io/reconcileTimeout`: Set the reconciliation timeout including health checks. Default is `5m`. - `fluxcd.controlplane.io/force`: When set to `enabled`, the controller will replace the generated resources that contain immutable field changes. This annotation can also be used on individual resources to force their reconciliation. ### Health check configuration The `.spec.wait` field is optional and instructs the flux-operator to perform a health check on all applied resources and waits for them to become ready. The health check is disabled by default and can be enabled by setting the `.spec.wait` field to `true`. The health check is performed for the following resources types: - Kubernetes built-in kinds: Deployment, DaemonSet, StatefulSet, PersistentVolumeClaim, Service, Ingress, CustomResourceDefinition. - Flux kinds: HelmRelease, OCIRepository, Kustomization, GitRepository, etc. - Custom resources that are compatible with [kstatus](https://github.com/kubernetes-sigs/cli-utils/tree/master/pkg/kstatus). By default, the wait timeout is `5m` and can be changed with the `fluxcd.controlplane.io/reconcileTimeout` annotation, set on the ResourceSet object. ### Role-based access control The `.spec.serviceAccountName` field is optional and specifies the name of the Kubernetes ServiceAccount used by the flux-operator to reconcile the ResourceSet. The ServiceAccount must exist in the same namespace as the ResourceSet and must have the necessary permissions to create, update and delete the resources defined in the ResourceSet. On multi-tenant clusters, it is recommended to use a dedicated ServiceAccount per tenant namespace with the minimum required permissions. To enforce a ServiceAccount for all ResourceSets, the `--default-service-account=flux-operator` flag can be set in the flux-operator container arguments. With this flag set, only the ResourceSets created in the same namespace as the flux-operator will run with cluster-admin permissions. When installing the Flux Operator with Helm, you can change the default service account name with: ```shell helm install flux-operator oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator \ --namespace flux-system \ --create-namespace \ --set multitenancy.enabled=true \ --set multitenancy.defaultServiceAccount=flux-operator ``` When installing the Flux Operator on OpenShift from OperatorHub, the default service account name can be changed by setting the `DEFAULT_SERVICE_ACCOUNT` environment variable using the OLM [Subscription](https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/design/subscription-config.md) `.spec.config.env` field. ### Garbage collection The operator performs garbage collection of the resources previously generated by a ResourceSet that are no longer present in the current revision. Garbage collection is also performed when a ResourceSet object is deleted, triggering a removal of all Kubernetes objects previously applied on the cluster. The garbage collection process removes stale resources in stages, first it deletes the Flux custom resources and waits for the Flux Kustomizations and HelmReleases to be finalized by the controllers. After the Flux resources are removed (max wait is one minute), the operator proceeds with the deletion of the remaining Kubernetes objects. This ensures that the Flux controllers have a chance to clean up the resources they manage before the operator deletes the Kubernetes ServiceAccount and RoleBinding used by Flux impersonation. After the garbage collection process is completed, the operator issues a Kubernetes event containing the list of removed resources and the duration of the cleanup. The garbage collection is enabled by default and can be disabled for certain resources by setting the `fluxcd.controlplane.io/prune` annotation to `disabled`. To fully disable the garbage collection for a ResourceSet, the annotation must be set on all resources using the [`.spec.commonMetadata`](#common-metadata) field. ## ResourceSet Status ### Conditions A ResourceSet enters various states during its lifecycle, reflected as Kubernetes Conditions. It can be [reconciling](#reconciling-resourceset) while applying the resources on the cluster, it can be [ready](#ready-resourceset), it can [fail during reconciliation](#failed-resourceset), or it can [fail due to misconfiguration](#stalled-resourceset). The ResourceSet API is compatible with the **kstatus** specification, and reports `Reconciling` and `Stalled` conditions where applicable to provide better (timeout) support to solutions polling the ResourceSet to become `Ready`. #### Reconciling ResourceSet The flux-operator marks a ResourceSet as _reconciling_ when it starts the reconciliation of the same. The Condition added to the ResourceSet's `.status.conditions` has the following attributes: - `type: Reconciling` - `status: "True"` - `reason: Progressing` | `reason: ProgressingWithRetry` The Condition `message` is updated during the course of the reconciliation to report the action being performed at any particular moment such as building manifests, detecting drift, etc. The `Ready` Condition's `status` is also marked as `Unknown`. #### Ready ResourceSet The flux-operator marks a ResourceSet as _ready_ when the resources were built and applied on the cluster and all health checks are observed to be passing. When the ResourceSet is "ready", the flux-operator sets a Condition with the following attributes in the ResourceSet’s `.status.conditions`: - `type: Ready` - `status: "True"` - `reason: ReconciliationSucceeded` #### Failed ResourceSet The flux-operator may get stuck trying to reconcile and apply a ResourceSet without completing. This can occur due to some of the following factors: - The dependencies are not ready. - The templating of the resources fails. - The resources are invalid and cannot be applied. - Garbage collection fails. - Running health checks fails. When this happens, the flux-operator sets the `Ready` Condition status to False and adds a Condition with the following attributes to the ResourceSet’s `.status.conditions`: - `type: Ready` - `status: "False"` - `reason: DependencyNotReady | BuildFailed | ReconciliationFailed | HealthCheckFailed` The `message` field of the Condition will contain more information about why the reconciliation failed. While the ResourceSet has one or more of these Conditions, the flux-operator will continue to attempt a reconciliation with an exponential backoff, until it succeeds and the ResourceSet is marked as [ready](#ready-resourceset). #### Stalled ResourceSet The flux-operator may fail the reconciliation of a ResourceSet object terminally due to a misconfiguration. When this happens, the flux-operator adds the `Stalled` Condition to the ResourceSet’s `.status.conditions` with the following attributes: - `type: Stalled` - `status: "True"` - `reason: InvalidCELExpression | BuildFailed` Misconfigurations can include: - The `.spec.dependsOn[].readyExpr` expression is invalid. In this case the condition reason is `InvalidCELExpression`. - The templating of the resources fails. In this case the condition reason is `BuildFailed`. When this happens, the flux-operator will not attempt to reconcile the ResourceSet until the misconfiguration is fixed. The `Ready` Condition status is also set to `False`. ### History With `.status.history` the operator tracks the reconciliation attempts over time, providing insights into the ResourceSet's behavior which can be used for audit, anomaly detection and debugging purposes. The history is stored as a list of snapshots, ordered by last reconciliation time. Each snapshot contains: - `digest`: A SHA256 digest that uniquely identifies the set of generated resources being reconciled - `firstReconciled`: The timestamp when this particular configuration was first reconciled - `lastReconciled`: The timestamp of the most recent reconciliation attempt for this configuration - `lastReconciledDuration`: How long the most recent reconciliation attempt took - `lastReconciledStatus`: The status of the most recent reconciliation (e.g., `ReconciliationSucceeded`, `BuildFailed`, `ReconciliationFailed`) - `totalReconciliations`: The total number of reconciliations for this configuration - `metadata`: Additional information about the reconciliation, including the number of generated resources and inputs processed The operator deduplicates entries based on the digest and status. The history is automatically truncated to keep only the 5 most recent entries. Example: ```yaml status: history: - digest: sha256:43ad78c94b2655429d84f21488f29d7cca9cd45b7f54d2b27e16bbec8eff9228 firstReconciled: "2025-07-15T10:11:00Z" lastReconciled: "2025-07-15T14:30:00Z" lastReconciledDuration: 2.818583s lastReconciledStatus: ReconciliationSucceeded totalReconciliations: 5 metadata: inputs: "2" resources: "4" - digest: sha256:ec8dbfe61777b65001190260cf873ffe454451bd2e464bd6f9a154cffcdcd7e5 firstReconciled: "2025-07-14T13:10:00Z" lastReconciled: "2025-07-15T10:10:00Z" lastReconciledDuration: 4.813292s lastReconciledStatus: ReconciliationFailed totalReconciliations: 120 metadata: inputs: "1" resources: "2" ``` Note that for `BuildFailed` errors, the digest is calculated from the resource templates, as the final resources are not available. ### Inventory In order to perform operations such as drift detection, garbage collection, upgrades, etc., the flux-operator needs to keep track of all Kubernetes objects that are reconciled as part of a ResourceSet. To do this, it maintains an inventory containing the list of Kubernetes resource object references that have been successfully applied and records it in `.status.inventory`. The inventory records are in the format `Id: ___, V: `. Example: ```yaml status: inventory: entries: - id: apps_podinfo_helm.toolkit.fluxcd.io_HelmRelease v: v2 - id: apps_podinfo_source.toolkit.fluxcd.io_OCIRepository v: v1 ``` ## ResourceSet Metrics The Flux Operator exports Prometheus metrics for the ResourceSet objects that can be used to monitor the reconciliation status. Metrics: ```text flux_resourceset_info{uid, kind, name, exported_namespace, ready, suspended, revision} ``` Labels: - `uid`: The Kubernetes unique identifier of the resource. - `kind`: The kind of the resource (e.g. `ResourceSet`). - `name`: The name of the resource (e.g. `podinfo`). - `exported_namespace`: The namespace where the resource is deployed (e.g. `apps`). - `ready`: The readiness status of the resource (e.g. `True`, `False` or `Unkown`). - `reason`: The reason for the readiness status (e.g. `ReconciliationSucceeded`, `BuildFailed`, `HealthCheckFailed`, etc.). - `suspended`: The suspended status of the resource (e.g. `True` or `False`). - `resources`: The number of resources generated by the ResourceSet. - `revision`: The revision last applied on the cluster (e.g. `sha256:75aa209c6a...`).