# Source Hydrator

> [!WARNING]
> **Beta Feature (Since v3.5.0)**
>
> This is a [beta-quality](https://github.com/argoproj/argoproj/blob/main/community/feature-status.md#beta)
> feature that pushes hydrated manifests to git before syncing them to the cluster.

Tools like Helm and Kustomize allow users to express their Kubernetes manifests in a more concise and reusable way
(keeping it DRY - Don't Repeat Yourself). However, these tools can obscure the actual Kubernetes manifests that are
applied to the cluster.

The *rendered manifest pattern* is a feature of Argo CD that allows users to push the hydrated manifests to git before syncing them to the cluster. This
allows users to see the actual Kubernetes manifests that are applied to the cluster.

## Enabling the Source Hydrator

The source hydrator is disabled by default.

To enable the source hydrator, you need to enable the "commit server" component and set the `hydrator.enabled` field in
`argocd-cmd-params-cm` ConfigMap to `"true"`.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
  namespace: argocd
data:
  hydrator.enabled: "true"
```

> [!IMPORTANT]
> After updating the ConfigMap, you must restart the Argo CD controller and API server for the changes to take effect.

If you are using one of the `*-install.yaml` manifests to install Argo CD, you can use the 
`*-install-with-hydrator.yaml` version of that file instead.

For example,

```
Without hydrator: https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
With hydrator:    https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install-with-hydrator.yaml
```

> [!IMPORTANT]
> The `*-install-with-hydrator.yaml` manifests will eventually be removed when the source hydrator is either enabled
> by default or removed. The upgrade guide will note if the `install-with-hydrator.yaml` manifests are no longer
> available.

## Using the Source Hydrator

To use the source hydrator, you must first install a push and a pull secret. This example uses a GitHub App for authentication, but
you can use [any authentication method that Argo CD supports for repository access](../operator-manual/declarative-setup.md#repositories).

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-push-secret
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository-write
type: Opaque
stringData:
  url: "https://github.com/<your org or user>/<your repo>"
  type: "git"
  githubAppID: "<your app ID here>"
  # githubAppInstallationID is optional and will be auto-discovered if omitted
  githubAppInstallationID: "<your installation ID here>" # Optional
  githubAppPrivateKey: |
    <your private key here>
---
apiVersion: v1
kind: Secret
metadata:
  name: my-pull-secret
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
type: Opaque
stringData:
  url: "https://github.com/<your org or user>/<your repo>"
  type: "git"
  githubAppID: "<your app ID here>"
  # githubAppInstallationID is optional and will be auto-discovered if omitted
  githubAppInstallationID: "<your installation ID here>"  # Optional
  githubAppPrivateKey: |
    <your private key here>
```

The only difference between the secrets above, besides the resource name, is that the push secret contains the label
`argocd.argoproj.io/secret-type: repository-write`, which causes the Secret to be used for pushing manifests to git
instead of pulling from Git. Argo CD requires different secrets for pushing and pulling to provide better isolation.

Once your secrets are installed, set the `spec.sourceHydrator` field of the Application. For example:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/argoproj/argocd-example-apps
      path: helm-guestbook
      targetRevision: HEAD
    syncSource:
      targetBranch: environments/dev
      path: helm-guestbook
```

In this example, the hydrated manifests will be pushed to the `environments/dev` branch of the `argocd-example-apps`
repository. The `drySource` field tells Argo CD where your original, unrendered configuration lives.
This can be a Helm chart, a Kustomize directory, or plain manifests. Argo CD reads this source, renders the final Kubernetes
manifests from it, and then writes those hydrated manifests into the location specified by `syncSource.path`.

### Separate destination repository

By default, hydrated manifests are written to the same Git repository as the dry source. Set `syncSource.repoURL` to
hydrate into a different repository. When `syncSource.repoURL` is omitted, it defaults to `drySource.repoURL`.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/my-org/config
      path: helm-guestbook
      targetRevision: HEAD
    syncSource:
      repoURL: https://github.com/my-org/deployments
      targetBranch: environments/dev
      path: helm-guestbook
```

Both repositories must be permitted in the application's AppProject (`spec.sourceRepos`). The hydrator needs read
access to the dry source repository and write access to the destination repository (a `repository-write` secret; see
the push secret example above).

> [!NOTE]
> `hydrateTo` inherits its repository and path from `syncSource`. When `syncSource.repoURL` points to a separate
> repository, staged manifests are pushed to that repository as well.

When using source hydration, the `syncSource.path` field is required and must always point to a non-root
directory in the repository. Setting the path to the repository root (for example `"."` or `""`) is not
supported. This ensures that hydration is always scoped to a dedicated subdirectory, which avoids unintentionally overwriting or removing files that may exist in the repository root.

During each hydration run, Argo CD overwrites the files it generates (such as `manifest.yaml`) in the application's configured path, but it does **not** delete other files already present in that path. Stale files from a previous hydration that are not overwritten will remain in the output directory. 

Because the generated `manifest.yaml` is fully rewritten on every run, resources that were removed from the dry source disappear from it and are pruned on the next sync (when the `prune` sync option is enabled); however, extra leftover files are not removed automatically.

The repository root is never written to, so files such as CI/CD configuration, README files, or other root-level assets remain untouched.

If an application’s path changes, the old directory is not removed automatically. Likewise, if an application is deleted, its output path remains in the repository and must be cleaned up manually by the repository owner if desired. This design is intentional: it prevents accidental deletion of files when applications are restructured or removed, and it protects critical files like CI pipelines that may coexist in the repository.

> [!NOTE]
> The hydrator triggers only when a new commit is detected in the dry source.
> Adding or removing Applications does not on its own cause hydration to run.
> If the set of Applications changes but the dry-source commit does not, hydration will wait until the next commit.
>
> This is by design: the hydrator produces one hydration per dry-source commit and maintains atomicity across the
> Applications that share a destination. If you add Applications that depend on a dry-source commit that has already
> been hydrated (for example, Applications generated by an ApplicationSet), push an empty commit to the dry source so
> the new Applications are hydrated.

> [!IMPORTANT]
> **Project-Scoped Repositories**
>

    Repository Secrets may contain a `project` field, making the secret only usable by Applications in that project.
    The source hydrator only supports project-scoped repositories if all Applications writing to the same repository and
    branch are in the same project. If Applications in different projects write to the same repository and branch, the
    source hydrator will not be able to use a project-scoped repository secret and will require a global repository 
    secret. This behavior may change in the future.

If there are multiple repository-write Secrets available for a repo, the source hydrator will non-deterministically
select one of the matching Secrets and log a warning saying "Found multiple credentials for repoURL".

## Source Configuration Options

The source hydrator supports various source types through inline configuration options in the `drySource` field. This allows you to use Helm charts, Kustomize applications, directories, and plugins with environment-specific configurations.

### Helm Charts

You can use Helm charts by specifying the `helm` field in the `drySource`:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-helm-app
spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/argoproj/argocd-example-apps
      path: helm-guestbook
      targetRevision: HEAD
      helm:
        valueFiles:
          - values-prod.yaml
        parameters:
          - name: image.tag
            value: v1.2.3
        releaseName: my-release
    syncSource:
      targetBranch: environments/prod
      path: helm-guestbook-hydrated
```

### Kustomize Applications

For Kustomize applications, use the `kustomize` field:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-kustomize-app
spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/argoproj/argocd-example-apps
      path: kustomize-guestbook
      targetRevision: HEAD
      kustomize:
        namePrefix: prod-
        nameSuffix: -v1
        images:
          - gcr.io/heptio-images/ks-guestbook-demo:0.2
    syncSource:
      targetBranch: environments/prod
      path: kustomize-guestbook-hydrated
```

### Directory Applications

For plain directory applications with specific options, use the `directory` field:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-directory-app
spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/argoproj/argocd-example-apps
      path: guestbook
      targetRevision: HEAD
      directory:
        recurse: true
    syncSource:
      targetBranch: environments/prod
      path: guestbook-hydrated
```

### Config Management Plugins

You can also use Config Management Plugins by specifying the `plugin` field:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-plugin-app
spec:
  sourceHydrator:
    drySource:
      repoURL: https://github.com/argoproj/argocd-example-apps
      path: my-plugin-app
      targetRevision: HEAD
      plugin:
        name: my-custom-plugin
        env:
          - name: ENV_VAR
            value: prod
    syncSource:
      targetBranch: environments/prod
      path: my-plugin-app-hydrated
```

!!! note "Feature Parity"
    The source hydrator supports the same configuration options as the regular Application source field. You can use any combination of these source types with their respective configuration options to match your application's needs.

## Pushing to a "Staging" Branch

The source hydrator can be used to push hydrated manifests to a "staging" branch instead of the `syncSource` branch.
This provides a way to prevent the hydrated manifests from being applied to the cluster until some prerequisite
conditions are met (in effect, providing a way to handle environment promotion via Pull Requests).

To use the source hydrator to push to a "staging" branch, set the `spec.sourceHydrator.hydrateTo` field of the
Application. For example:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
spec:
  project: my-project
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  sourceHydrator:
    drySource:
      repoURL: https://github.com/argoproj/argocd-example-apps
      path: helm-guestbook
      targetRevision: HEAD
    syncSource:
      targetBranch: environments/dev
      path: helm-guestbook
    hydrateTo:
      targetBranch: environments/dev-next
```

In this example, the hydrated manifests will be pushed to the `environments/dev-next` branch, and Argo CD will not sync
the changes until something moves them to the `environments/dev` branch.

You could use a CI action to move the hydrated manifests from the `hydrateTo` branch to the `syncSource` branch. To
introduce a gating mechanism, you could require a Pull Request to be opened to merge the changes from the `hydrateTo`
branch to the `syncSource` branch.

Argo CD will only push changes to the `hydrateTo` branch, it will not create a PR or otherwise facilitate moving those 
changes to the `syncSource` branch. You will need to use your own tooling to move the changes from the `hydrateTo` 
branch to the `syncSource` branch.

## Commit Tracing

It's common for CI or other tooling to push DRY manifest changes after a code change. It's important for users to be
able to trace the hydrated commits back to the original code change that caused the hydration.

Source Hydrator makes use of some custom git commit trailers to facilitate this tracing. A CI job that builds an image
and pushes an image bump to DRY manifests can use the following commit trailers to link the hydrated commit to the
code commit.

```shell
git commit -m "Bump image to v1.2.3" \
  # Must be an RFC 5322 name
  --trailer "Argocd-reference-commit-author: Author Name <author@example.com>" \
  # Must be a hex string 5-40 characters long
  --trailer "Argocd-reference-commit-sha: <code-commit-sha>" \
  # The subject is the first line of the commit message. It cannot contain newlines.
  --trailer "Argocd-reference-commit-subject: Commit message of the code commit" \
   # The body must be a valid JSON string, including opening and closing quotes
  --trailer 'Argocd-reference-commit-body: "Commit message of the code commit\n\nSigned-off-by: Author Name <author@example.com>"' \
   # The repo URL must be a valid URL
  --trailer "Argocd-reference-commit-repourl: https://git.example.com/owner/repo" \
  # The date must by in ISO 8601 format
  --trailer "Argocd-reference-commit-date: 2025-06-09T13:50:18-04:00" 
```

> [!NOTE]
> The commit trailers must not contain newlines. 

So the full CI script might look something like this:

```shell
# Clone code repo
git clone https://git.example.com/owner/repo.git
cd repo

# Build the image and get the new image tag
# <custom build logic here>

# Get the commit information
author=$(git show -s --format="%an <%ae>")
sha=$(git rev-parse HEAD)
subject=$(git show -s --format='%s')
body=$(git show -s --format='%b')
jsonbody=$(jq -n --arg body "$body" '$body')
repourl=$(git remote get-url origin)
date=$(git show -s --format='%aI')

# Clone the dry source repo
git clone https://git.example.com/owner/deployment-repo.git
cd deployment-repo

# Bump the image in the dry manifests
# <custom bump logic here, e.g. `kustomize edit`>

# Commit the changes with the commit trailers
git commit -m "Bump image to v1.2.3" \
  --trailer "Argocd-reference-commit-author: $author" \
  --trailer "Argocd-reference-commit-sha: $sha" \
  --trailer "Argocd-reference-commit-subject: $subject" \
  --trailer "Argocd-reference-commit-body: $jsonbody" \
  --trailer "Argocd-reference-commit-repourl: $repourl" \
  --trailer "Argocd-reference-commit-date: $date"
```

The commit metadata will appear in the hydrated commit's root `hydrator.metadata` file:

```json
{
  "author": "CI <ci@example.com>",
  "subject": "chore: bump image to b82add2",
  "date": "2025-06-09T13:50:08-04:00",
  "body": "Signed-off-by: CI <ci@example.com>\n",
  "drySha": "6cb951525937865dced818bbdd78c89b2d2b3045",
  "repoURL": "https://git.example.com/owner/manifests-repo",
  "references": [
    {
      "commit": {
        "author": {
          "name": "Author Name",
          "email": "author@example.com"
        },
        "sha": "b82add298aa045d3672880802d5305c5a8aaa46e",
        "subject": "chore: make a change",
        "body": "make a change\n\nSigned-off-by: Author Name <author@example.com>",
        "repoURL": "https://git.example.com/owner/repo",
        "date": "2025-06-09T13:50:18-04:00"
      }
    }
  ]
}
```

The top-level "body" field contains the commit message of the DRY commit minus the subject line and any 
`Argocd-reference-commit-*` trailers that were used in `references`. Unrecognized or invalid trailers are preserved in
the body.

Although `references` is an array, the source hydrator currently only supports a single related commit. If a trailer is
specified more than once, the last one will be used.

All trailers are optional. If a trailer is not specified, the corresponding field in the metadata will be omitted.

## Commit Message Template

The commit message is generated using a [Go text/template](https://pkg.go.dev/text/template), optionally configured by the user via the argocd-cm ConfigMap. The template is rendered using the values from `hydrator.metadata`. The template can be multi-line, allowing users to define a subject line, body and optional trailers. To define the commit message template, you need to set the `sourceHydrator.commitMessageTemplate` field in argocd-cm ConfigMap.

The template can invoke functions from the [Sprig function library](https://github.com/Masterminds/sprig).

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  sourceHydrator.commitMessageTemplate: |
    {{.metadata.drySha | trunc 7}}: {{ .metadata.subject }}
    {{- if .metadata.body }}
    
    {{ .metadata.body }}
    {{- end }}
    {{ range $ref := .metadata.references }}
    {{- if and $ref.commit $ref.commit.author }}
    Co-authored-by: {{ $ref.commit.author }}
    {{- end }}
    {{- end }}
    {{- if .metadata.author }}
    Co-authored-by: {{ .metadata.author }}
    {{- end }}
```

## README Template

The hydration README is generated using a [Go text/template](https://pkg.go.dev/text/template), optionally configured by the user via the `argocd-cm` ConfigMap. The template is rendered using the values from `hydrator.metadata` and can be multi-line to define structured documentation. This allows users to customize how the hydration process and references are documented.

To define the README template, set the `sourceHydrator.readmeMessageTemplate` field in the `argocd-cm` ConfigMap.

The template may also use functions from the [Sprig function library](https://github.com/Masterminds/sprig).

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  sourceHydrator.readmeMessageTemplate: |
    # Manifest Hydration

    To hydrate the manifests in this repository, run the following commands:

    ```shell
    git clone {{ .RepoURL }}
    git checkout {{ .DrySHA }}
    {{ range $command := .Commands }}
    {{ $command }}
    {{ end }}
    ```

    {{ if .References }}
    ## References

    {{ range $ref := .References }}
    {{ if $ref.Commit }}
    * [{{ $ref.Commit.SHA | trunc 7 }}]({{ $ref.Commit.RepoURL }}): {{ $ref.Commit.Subject }} ({{ $ref.Commit.Author }})
    {{ end }}
    {{ end }}
    {{ end }}
```

## Commit Author Configuration

You can customize the git commit author name and email used by the source hydrator when committing hydrated manifests. This is configured via the `argocd-cm` ConfigMap.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  commit.author.name: "GitOps Bot"
  commit.author.email: "gitops@company.com"
```

**Configuration Keys:**
* `commit.author.name`: The git commit author name (defaults to `"Argo CD"` if not set)
* `commit.author.email`: The git commit author email (defaults to `"argo-cd@example.com"` if not set)

Both values are optional. If only one is configured, the configured value will be used and the other will use its default.

### Credential Templates

Credential templates allow a single credential to be used for multiple repositories. The source hydrator supports credential templates. For example, if you setup credential templates for the URL prefix `https://github.com/argoproj`, these credentials will be used for all repositories with this URL as prefix (e.g. `https://github.com/argoproj/argocd-example-apps`) that do not have their own credentials configured.
For more information, please refer to [Credential templates](private-repositories.md#credential-templates).
An example of repo-write-creds secret.

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: private-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repo-write-creds
stringData:
  type: git
  url: https://github.com/argoproj
  password: my-password
  username: my-username
```

## Git Note-based Hydration State Tracking

The Source Hydrator does not create a new hydrated commit for a DRY commit if the commit doesn't affect the hydrated manifests. Instead, the hydration state (the DRY SHA last hydrated) is tracked using a [git note](https://git-scm.com/docs/git-notes) in a dedicated `source-hydrator` namespace.

On each run, the hydrator:
* Checks the git note for the last hydrated DRY SHA.
* If manifests have not changed since that SHA, only updates the note.
* If manifests have changed, commits the new manifests and updates the note as well.

This improves efficiency and reduces commit noise in your repository.

If external tooling reads git notes for promotion or coordination (for example, to gate on which dry SHAs
have been processed), be aware that `manifest-generate-paths` can prevent note advancement when hydration
is skipped. See [Git note attestation and manifest-generate-paths](#git-note-attestation-and-manifest-generate-paths).

## Hydration failures and retries

When hydration fails, the application remains in the `Failed` phase and the error message is kept on
`status.sourceHydrator.currentOperation` so you can diagnose the problem.

> [!NOTE]
> After a failed hydration, the controller waits about **2 minutes** before automatically retrying
> unattended hydration. That interval matches `ARGOCD_RECONCILIATION_TIMEOUT` (the application
> controller `--app-resync` period, default **120s**).
>
> You do not need to wait for that cooldown when:
>
> - You trigger a **manual or API refresh** (or a **dry-source webhook**), which retries hydration
>   immediately.
> - A **new dry commit** is detected while a dry revision baseline is already recorded in
>   `status.sourceHydrator.lastComparedDryRevision` (for example, hydration failed at the commit step
>   after manifests were resolved).
>
> If the first hydration attempt fails before any dry revision is recorded, automatic detection of a
> new commit during the cooldown may be delayed until the cooldown expires or you refresh manually.

## Forcing Hydration with the `hydrate` Annotation

Argo CD uses the `argocd.argoproj.io/hydrate` annotation to request that an Application's dry source be
checked and hydrated (if necessary) outside of the normal commit-detection flow. The application controller
consumes and removes the annotation as soon as it has processed it, so it will not normally persist on the
Application.

The annotation takes one of two values:

| Value    | Effect                                                                                                                    |
|----------|---------------------------------------------------------------------------------------------------------------------------|
| `normal` | Forces a check of whether the dry source has changed since the last hydration.  Hydration only runs if a change is found. |
| `hard`   | Forces hydration unconditionally, even if the dry source revision hasn't changed since the last hydration.                |

> [!NOTE]
> Only the exact value `hard` is special-cased. Any other value is treated as `normal`.

Whenever Argo CD sets this annotation itself (see the triggers below), it also sets the companion
`argocd.argoproj.io/hydrate-timestamp` annotation to a unique timestamp identifying that specific
hydration request. When the controller finishes hydrating, it removes both annotations together, but only
if `hydrate-timestamp` still has the value it had when hydration started. If a new hydration request
arrived while the previous one was being processed, `hydrate-timestamp` will have a newer value, removal is
skipped, and the request is processed instead of being silently dropped.

Argo CD sets these annotations automatically in a few situations:

* **Manual or API refresh.** `argocd app get <app> --refresh` (and the equivalent API call) sets it to `normal`;
  `argocd app get <app> --hard-refresh` sets it to `hard`.
* **Dry-source webhooks.** A webhook event affecting the `drySource` repository sets it to `normal`, so the dry
  source is checked right away instead of waiting for the next periodic reconciliation.

You can also set the annotation yourself to force a re-hydration, for example to retry sooner than the
[failure cooldown](#hydration-failures-and-retries) or after a change that the hydrator wouldn't otherwise detect
(such as a change to a [Config Management Plugin](../operator-manual/config-management-plugins.md) that doesn't
touch the dry source repository):

```shell
kubectl patch application my-app -n argocd --type merge \
  -p '{"metadata": {"annotations": {"argocd.argoproj.io/hydrate": "hard"}}}'
```

> [!NOTE]
> The annotation only has an effect on Applications with `spec.sourceHydrator` configured, it is ignored otherwise.

## Manifest Generate Paths

The source hydrator honors the [`manifest-generate-paths` annotation](../operator-manual/high_availability.md#manifest-paths-annotation)
to avoid unnecessary hydration. When the annotation is set, a new dry-source commit that does not touch the annotated
paths (resolved relative to the `drySource` path) does not trigger re-hydration.

The annotation controls **whether hydration is triggered**, not merely whether hydrated manifests would change.
When hydration is skipped, the hydrator does not run and the `hydrator.metadata` git note on the hydrated branch
is not advanced. See [Git note attestation and manifest-generate-paths](#git-note-attestation-and-manifest-generate-paths)
for implications when external tooling reads git notes.

This applies to both webhook-driven and periodically-reconciled refreshes:

* A webhook for the **dry source** triggers hydration only when the changed files match the annotated paths.
* A webhook for the **sync source** (for example, when an external promotion process merges the `hydrateTo` branch
  into the `syncSource` branch) triggers a normal refresh and sync of the hydrated manifests.

The annotation's path filtering applies to the dry source. The sync source is always watched at its configured
`syncSource.path`; the annotation value is not applied to the sync source.

## Limitations

### Repository support

The source hydrator currently supports only Git repositories. `drySource.repoURL` must reference a Git
repository. The effective sync repository must also be Git: `syncSource.repoURL` may reference a different
Git repository or be omitted to inherit `drySource.repoURL`. 

OCI repositories (`oci://...`) are supported as [regular Application sources](oci.md), but cannot currently
be used as a `drySource` or hydration destination. An OCI dry source may be accepted and resolved to a digest,
but hydration will fail when Argo CD attempts to retrieve Git revision metadata.

### Git note attestation and manifest-generate-paths

The hydrator records which dry SHAs it has processed in a git note (`refs/notes/hydrator.metadata`) on the
hydrated branch. The note is written by commit-server when hydration runs — including the case where manifests
are unchanged and no new hydrated commit is created.

When `manifest-generate-paths` causes hydration to be skipped (because changed files fall outside the annotated
paths), commit-server is never called and the git note is **not** advanced. Application status may still record
that the dry revision was compared, but the note's `drySha` will lag behind. In other words, the note attests only
the dry commits for which the hydrator actually ran; it does not advance for dry commits filtered out by
`manifest-generate-paths`, even though those commits would produce no manifest change.

If you consume `hydrator.metadata` notes, account for this: only dry commits that match an Application's
`manifest-generate-paths` (or all dry commits, when the annotation is unset) are attested on its hydrated branch.

A planned enhancement ([#28556](https://github.com/argoproj/argo-cd/issues/28556)) will add a lightweight
commit-server endpoint to advance the git note without a full hydration run (no repo-server render or manifest
disk compare).

### Signature Verification

*Current Status: Alpha (Since v3.5)*

The hydrator enforces the project's `SourceIntegrity` policy (e.g. GPG signature verification) on the DRY
revision before producing manifests. If the project requires verification and the DRY commit fails it (or was
not verified), hydration is rejected. Verification is opted into per project by configuring `.spec.sourceIntegrity` on
the `AppProject`. See [Source Integrity Verification](source-integrity.md) for how to set it up (for example, using
[Git GnuPG verification](source-integrity-git-gpg.md)).

By default the hydrator **does not** sign the commits it pushes to git, so if signature verification is enabled
for the hydrated branch those commits will fail verification when Argo CD attempts to sync the hydrated manifests.
See [Signing Hydrated Commits](#signing-hydrated-commits) below for how to opt in.

### Signing Hydrated Commits

*Current Status: Alpha (Since v3.6)*

The commit server can optionally GPG-sign every hydrated commit before pushing it. Signing is enabled simply by
configuring a signing key path — there is no separate on/off toggle. When a key is configured, the commit server
fails fast at startup if the signing key is missing or invalid, and refuses to push any commit it cannot locally
verify against the configured key — there is no unsigned fallback.

To enable signing:

1. Create a Kubernetes Secret in the Argo CD namespace named `argocd-commit-server-gpg-signing-key` containing
   the ASCII-armored private key, and (optionally) the passphrase:

    ```yaml
    apiVersion: v1
    kind: Secret
    metadata:
      name: argocd-commit-server-gpg-signing-key
      namespace: argocd
    type: Opaque
    stringData:
      signingKey: |
        -----BEGIN PGP PRIVATE KEY BLOCK-----
        ...
        -----END PGP PRIVATE KEY BLOCK-----
      # Optional. Omit entirely if the key has no passphrase.
      passphrase: "s3cret"
    ```

2. Set the relevant params in `argocd-cmd-params-cm`. Setting the key path is what turns signing on:

    ```yaml
    # Empty by default (signing off). Point this at the mounted Secret entry to enable signing.
    commitserver.signing.key.path: "/app/config/gpg/signing/signingKey"
    # Empty by default. Set this ONLY for a passphrase-protected key, pointing at the
    # mounted `passphrase` Secret entry. Leave unset for an unprotected key.
    commitserver.signing.key.passphrase.file: "/app/config/gpg/signing/passphrase"
    ```

3. Add the corresponding **public** key to Argo CD's trusted GPG keys (via the UI, CLI, or the
   `argocd-gpg-keys-cm` ConfigMap) and configure your project's `SourceIntegrity` policy so the hydrated branch is
   verified at sync time. Without this step Argo CD will accept the signed commit but won't actually enforce that
   it was signed.

    Declaratively, add the public key to `argocd-gpg-keys-cm`, keyed by the signing key's ID with the
    ASCII-armored public key as the value:

    ```yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: argocd-gpg-keys-cm
      namespace: argocd
      labels:
        app.kubernetes.io/name: argocd-gpg-keys-cm
        app.kubernetes.io/part-of: argocd
    data:
      # The signing key's ID, with its ASCII-armored public key as the value.
      D56C4FCA57A46444: |
        -----BEGIN PGP PUBLIC KEY BLOCK-----
        ...
        -----END PGP PUBLIC KEY BLOCK-----
    ```

    Then reference that key ID in the project's `SourceIntegrity` policy so the hydrated (`hydrateTo`) branch is
    verified at sync time:

    ```yaml
    apiVersion: argoproj.io/v1alpha1
    kind: AppProject
    metadata:
      name: my-project
      namespace: argocd
    spec:
      sourceIntegrity:
        git:
          policies:
            - repos:
                - url: "https://github.com/my-org/my-hydrated-repo.git"
              gpg:
                # "strict" verifies the entire branch history; because the commit server signs every hydrated
                # commit, this holds for a branch signed from the start. Use "head" to verify only the tip commit
                # instead — useful when the branch already had unsigned commits before signing was enabled.
                mode: "strict"
                keys:
                  - "D56C4FCA57A46444"
    ```

    See [Git GnuPG verification](./source-integrity-git-gpg.md) for the full policy reference.

> [!NOTE]
> The configured key is the commit server's single, cluster-wide **default** signing key. Finer-grained signing
> keys (for example per-AppProject or per-repository) are not supported yet, but are a planned additive extension —
> the existing `commitserver.signing.key.*` params will remain the default and won't be renamed.

> [!NOTE]
> If your signing key has a dedicated **signing subkey** (common when the primary key is kept certify-only, e.g. a
> key extended with `gpg --quick-add-key <fpr> - sign`), git signs commits with that subkey rather than the primary
> key. You only need to list the **primary** key fingerprint in your `SourceIntegrity` policy — verification
> resolves the signing subkey back to its primary key, so a subkey signature is accepted as long as the primary is
> trusted. The public key you import via `argocd-gpg-keys-cm` already carries its subkeys, so no extra configuration
> is required.

> [!WARNING]
> If you use a tool that auto-merges the hydrated `hydrateTo` branch into the `syncSource` branch — for example
> [GitOps Promoter](https://github.com/argoproj-labs/gitops-promoter) or a GitHub "merge pull request" automation —
> the live branch HEAD becomes a **merge commit signed by the merger**, not the commit server's signed commit.
> Argo CD verifies that merge commit when it picks up the change on the live branch, so verification will fail
> unless the merger's public key is also trusted. Add the merger's public key (e.g. GitHub's web-flow key from
> `https://github.com/web-flow.gpg`) to `argocd-gpg-keys-cm` and your project's `SourceIntegrity` policy alongside
> the commit server's signing key.

### Project-Scoped Push Secrets

If all the Applications for a given destination repo/branch are under the same project, then the hydrator will use any
available project-scoped push secrets. If two Applications for a given repo/branch are in different projects, then the
hydrator will not be able to use a project-scoped push secret and will require a global push secret.

### Multiple Argo CD Instances Hydrating to One Branch

A hydrated repository and branch should be owned by a single Argo CD instance. Two separate Argo CD instances that
share the same `drySource` and write hydrated manifests to the **same** `syncSource` repository and branch are not
supported: the hydrator records hydration state in a per-dry-commit git note on the destination branch, so once one
instance records a commit, the other instance will short-circuit and leave its manifests stale. Hydrate each instance
to its own branch instead.

## Prerequisites

### Handle Secrets on the Destination Cluster

Do not use the source hydrator with any tool that injects secrets into your manifests as part of the hydration process
(for example, Helm with SOPS or the Argo CD Vault Plugin). These secrets would be committed to git. Instead, use a
secrets operator that populates the secret values on the destination cluster.

## Best Practices

### Make Hydration Deterministic

The source hydrator should be deterministic. For a given dry source commit, the hydrator should always produce the same
hydrated manifests. This means that the hydrator should not rely on an external state or configuration that is not stored
in Git.

Examples of non-deterministic hydration:

* A Helm chart using unpinned dependencies
* A Helm chart is using a non-deterministic template function such as `randAlphaNum` or `lookup`
* [Config Management Plugins](../operator-manual/config-management-plugins.md) which retrieve non-git state, such as secrets
* Kustomize manifests referencing unpinned remote bases

### Enable Branch Protection

Argo CD should be the only thing pushing hydrated manifests to the hydrated branches. To prevent other tools or users
from pushing to the hydrated branches, enable branch protection in your SCM.

It is best practice to prefix the hydrated branches with a common prefix, such as `environments/`. This makes it easier
to configure branch protection rules on the destination repository.

> [!NOTE]
> To maintain reproducibility and determinism in the Hydrator’s output,
> Argo CD-specific metadata (such as `argocd.argoproj.io/tracking-id`) is
> not written to Git during hydration. These annotations are added dynamically
> during application sync and comparison.

### Application Path Cleaning Behavior

The Source Hydrator does not clean (remove) files from the application's configured output path before writing new manifests. This means that any files previously generated by hydration (or otherwise present) that are not overwritten by the new hydration run will remain in the output directory.