--- name: k8s-cluster-review description: >- Use when nothing is broken but someone wants to know whether a Kubernetes cluster, or one namespace, is still in good shape: "review the cluster", "check it against best practices", "what drifted since we deployed it?", after taking over a cluster, or before a busy period. Reviews the live objects, not manifests in a repository, in three groups. Reliability: workloads with one replica, no readiness probe or no requests, PodDisruptionBudgets that allow 0 disruptions or match no pod, all replicas on one node or zone. Security: privileged or root containers, hostPath and hostNetwork, namespaces without Pod Security labels, cluster-admin bound to users or ServiceAccounts, namespaces without NetworkPolicies, :latest images. Hygiene and cost: PVCs no pod mounts, Released PVs, Services without endpoints, LoadBalancers, workloads at 0 replicas, failed Jobs, stale CronJobs. Reports each finding once per workload, with why it matters and when it is fine. Read-only: never changes anything itself. license: Apache-2.0 metadata: verified-with: "kubernetes 1.37" verified-on: "2026-11-11" --- # Cluster review Review a running cluster for what drifted in after it was deployed: settings that do not break anything today, but hurt during the next drain, load spike or incident, or cost money for nothing. **Read only.** Run the investigation commands freely. Never change, scale, label or delete anything yourself: every fix here is a proposed fix for the user or the owner of the object. Never read Secrets, and never `exec` into a pod; the review needs neither. Manifest linters (kube-linter, Polaris, kube-score) check what is in a repository. This review checks what ended up in the cluster. If the user already has output from one of those tools, use it as extra evidence; never install them. The commands use placeholders: `my-namespace`, `my-app`, `my-node`. Replace them with the real names. Cluster-wide checks need read access to nodes, PersistentVolumes and the RBAC objects, which the built-in `view` role leaves out. If a command is forbidden, do not treat it as "nothing found": put the check in the report's Skipped section with the command for the user to run. ## When to use - A periodic review: monthly, before a busy period or a launch, after taking over a cluster or a namespace from another team. - Someone asks whether the cluster follows good practice, or what is risky or wasteful in it, and nothing is broken. Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help: - Something is broken today (pods in CrashLoopBackOff or Pending, a node NotReady): the matching troubleshooting skill, or `k8s-triage` for a first look. A review is not the place to debug it. - Deprecated or removed APIs, version skew, anything about the next upgrade: `k8s-upgrade-readiness`. A PDB that allows 0 disruptions is reported here in one line, with a pointer there for what it does to a drain. - Certificates that expire or fail to renew: `k8s-cert-manager-troubleshooting`. - Who can do what in detail, and why a request is forbidden: `k8s-rbac-troubleshooting`. ## Investigate Run every step: the goal is a complete, short list, not the first finding. **0. Scope.** Review the whole cluster unless the user names a namespace; then add `-n my-namespace` instead of `-A` to every namespaced command, and skip the cluster-wide checks that do not concern it (nodes are still read for the spread check). Always skip `kube-system`, `kube-public` and `kube-node-lease`. If other namespaces look like add-ons installed with the cluster (for example `local-path-storage`, `cert-manager`, `monitoring`), name them, say you treat them as system, and let the user say otherwise. ```bash kubectl get namespaces --show-labels kubectl get nodes -L topology.kubernetes.io/zone kubectl get daemonsets -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" "}{.spec.template.spec.containers[*].image}{"\n"}{end}' ``` **1. Reliability**: replicas, probes, requests, PodDisruptionBudgets and spread. ```bash kubectl get deployments,statefulsets -A -o custom-columns='KIND:.kind,NAMESPACE:.metadata.namespace,NAME:.metadata.name,REPLICAS:.spec.replicas' kubectl get deployments,statefulsets,daemonsets -A -o jsonpath='{range .items[*]}{.kind}{" "}{.metadata.namespace}/{.metadata.name}{"\n"}{range .spec.template.spec.containers[*]}{" "}{.name}{" readiness="}{.readinessProbe}{" requests="}{.resources.requests}{"\n"}{end}{end}' kubectl get pdb -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" allowed="}{.status.disruptionsAllowed}{" healthy="}{.status.currentHealthy}/{.status.desiredHealthy}{" expected="}{.status.expectedPods}{" minAvailable="}{.spec.minAvailable}{" maxUnavailable="}{.spec.maxUnavailable}{"\n"}{end}' kubectl get pods -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,OWNER_NAME:.metadata.ownerReferences[0].name,NODE:.spec.nodeName,PHASE:.status.phase' ``` **2. Security**: what pods may do, Pod Security labels, cluster-admin, NetworkPolicies and image tags. ```bash kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" owner="}{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}{" pod.runAsNonRoot="}{.spec.securityContext.runAsNonRoot}{" pod.runAsUser="}{.spec.securityContext.runAsUser}{" hostNetwork="}{.spec.hostNetwork}{" hostPID="}{.spec.hostPID}{" hostIPC="}{.spec.hostIPC}{" hostPath="}{.spec.volumes[*].hostPath.path}{"\n"}{range .spec.containers[*]}{" "}{.name}{" image="}{.image}{" privileged="}{.securityContext.privileged}{" allowPrivilegeEscalation="}{.securityContext.allowPrivilegeEscalation}{" runAsNonRoot="}{.securityContext.runAsNonRoot}{" runAsUser="}{.securityContext.runAsUser}{"\n"}{end}{end}' kubectl get namespaces -o jsonpath='{range .items[*]}{.metadata.name}{" enforce="}{.metadata.labels.pod-security\.kubernetes\.io/enforce}{" warn="}{.metadata.labels.pod-security\.kubernetes\.io/warn}{"\n"}{end}' kubectl get clusterrolebindings -o jsonpath='{range .items[*]}{.metadata.name}{" role="}{.roleRef.name}{" subjects="}{.subjects}{"\n"}{end}' kubectl get rolebindings -A -o jsonpath='{range .items[?(@.roleRef.name=="cluster-admin")]}{.metadata.namespace}/{.metadata.name}{" subjects="}{.subjects}{"\n"}{end}' kubectl get clusterroles -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.rules}{"\n"}{end}' kubectl get networkpolicies -A ``` **3. Hygiene and cost**: volumes, Services, idle workloads, Jobs and CronJobs. ```bash kubectl get pvc -A kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.spec.volumes[*].persistentVolumeClaim.claimName}{"\n"}{end}' kubectl get pv -o custom-columns='NAME:.metadata.name,STATUS:.status.phase,CLAIM:.spec.claimRef.namespace,CLAIM_NAME:.spec.claimRef.name,RECLAIM:.spec.persistentVolumeReclaimPolicy,SIZE:.spec.capacity.storage' kubectl get services -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.spec.type,CLUSTER-IP:.spec.clusterIP,SELECTOR:.spec.selector' kubectl get endpointslices -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.labels.kubernetes\.io/service-name}{" ready="}{.endpoints[?(@.conditions.ready==true)].addresses}{"\n"}{end}' kubectl get jobs -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" owner="}{.metadata.ownerReferences[0].kind}{" failed="}{.status.failed}{" conditions="}{.status.conditions[*].type}{"\n"}{end}' kubectl get cronjobs -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,LAST_SCHEDULE:.status.lastScheduleTime,LAST_SUCCESS:.status.lastSuccessfulTime' ``` **4. Open the matching reference** for the field details and edge cases of a check: | Group | Reference | |---|---| | Replicas, probes, requests, PDBs, spread over nodes and zones | [references/reliability.md](references/reliability.md) | | securityContext, host access, Pod Security labels, RBAC, NetworkPolicies, image tags | [references/security.md](references/security.md) | | PVCs, PVs, Services, LoadBalancers, idle workloads, Jobs, CronJobs | [references/hygiene.md](references/hygiene.md) | ## Read the results Each check, what it means, and when it is fine (Kubernetes 1.37). Report a finding once per workload (Deployment, StatefulSet, DaemonSet, CronJob), not once per pod: group pods by their owner, and for a ReplicaSet name the Deployment it belongs to. A pod with no owner (a debug pod someone started by hand, for example) is reported by its own name, as a bare pod: nothing recreates it, which is a finding of its own. ### Reliability - **One replica** (Deployment or StatefulSet, `replicas: 1`): the app is down while its pod is replaced by a drain, an upgrade or a crash. Fine for batch tools, dev namespaces and singletons by design (a controller with leader election, a database run as one instance on purpose). DaemonSets and Jobs are not counted. - **No readiness probe**: the Service sends traffic to a pod as soon as it starts, and keeps sending it while the app cannot answer. A missing liveness probe is not a finding. Fine for workers that take no traffic through a Service. - **No CPU or memory requests**: the scheduler places the pod as if it needs nothing, and it is the first to be evicted under node pressure (QoS BestEffort). Fine for rare one-off pods. - **A PDB with `allowed=0` while every pod is healthy**: no pod it selects can be evicted, so a drain waits on it forever. Report it in one line and point to `k8s-upgrade-readiness`. **A PDB with `expected=0`**: it matches no pod; stale. - **All replicas of a workload with 2 or more replicas on one node** (or in one zone, when nodes carry `topology.kubernetes.io/zone`): one node or zone failure takes the whole app down. On a one-node cluster this cannot be checked: put it in Skipped, not in the findings or the counts. ### Security - **`privileged=true`**, **`hostNetwork`, `hostPID`, `hostIPC`** or a **`hostPath` volume**: the container can reach the node and everything on it. Fine for node agents that need it (CNI, CSI, log and metrics collectors), usually DaemonSets in an add-on namespace. - **`allowPrivilegeEscalation` not `false`**, or **running as root** (`runAsNonRoot` not `true` at pod or container level and no non-zero `runAsUser`): a process that breaks out of the app gets more rights than it needs. Name it in the evidence even when the same container is privileged. The image may still set a non-root user; say that the field does not show it. - **A namespace without `pod-security.kubernetes.io/enforce`**: nothing stops a privileged pod from being created there. Fine for system namespaces; a label `warn` or `audit` only reports. - **`cluster-admin`, or a ClusterRole with `*` verbs on `*` resources, bound to a user or a group whose name does not start with `system:`, or to a ServiceAccount outside the system and add-on namespaces**: that subject can do anything, including read every Secret. Installer defaults (kubeadm's `kubeadm:cluster-admins` group, a managed cloud's admin group, a ServiceAccount of `kube-system` or of an add-on namespace) are fine if intended: name them as such, not as a problem. - **A namespace with workloads and no NetworkPolicy**: every pod can reach every other pod. A finding only when the CNI enforces NetworkPolicy. Decide from the CNI's DaemonSet (step 0; an operator-installed Calico runs in `calico-system`, Flannel in `kube-flannel`): Cilium, Calico and Antrea enforce, and so does kindnet on current kind releases (seen on the 1.37 node image; on an old kind cluster, treat it as unknown); Flannel alone does not. On managed clusters it depends on an option (EKS `aws-node` only with its network policy agent, AKS only with a policy engine): unless you see that option, treat it as unknown. If you cannot tell, report "no NetworkPolicy; matters only if the CNI enforces them" in Skipped, not as a finding. - **An image with `:latest` or no tag** (a `@sha256:` digest is fine): which version runs depends on when the node pulled it, and a rollback cannot go back to it. ### Hygiene and cost - **A PVC that no pod mounts**: its volume is paid for and holds data nobody reads. Fine if a Job or a CronJob mounts it from time to time; check their templates. - **A PV in phase `Released`**: its claim was deleted and `Retain` kept the volume. It is never bound again by itself. - **A Service (not headless, not `ExternalName`) with no ready endpoints**: its selector matches no ready pod. Traffic to it fails. Fine if its app is scaled to 0 on purpose. - **LoadBalancer Services**: list them for a second look (each one is a cloud load balancer and usually a public address); do not call them wrong. - **A Deployment or StatefulSet at 0 replicas**, **a failed Job** (condition `Failed`, not owned by a CronJob that has since succeeded), **a CronJob that is suspended** or whose `LAST_SUCCESS` is older than several schedule periods: leftovers, or a task that stopped working without anyone noticing. Report the result in this shape, findings ordered by impact within each group. Stay quiet about what is fine: fewer findings with reasons beat a long list. ```text Summary: Reliability 2, Security 3, Hygiene and cost 1 (scope: whole cluster) Security - Deployment my-namespace/my-app: privileged container and a hostPath volume (/var/log). Evidence: privileged=true, hostPath=/var/log. Why it matters: ... Fine when: ... Proposed fix: ... (below) Skipped - kube-system, kube-public, kube-node-lease; local-path-storage (treated as an add-on). - ClusterRoleBindings: forbidden for this user (run: kubectl get clusterrolebindings). ``` ## Proposed fix Show each fix to the user; the agent runs none of them. Every finding belongs to the owner of the object, and a fix made in the cluster is undone by the next deploy unless the source (the manifest, the Helm values, the Git repository) changes too: prefer the change in the source. Write each undo with the values you found. Give a single-replica app a second pod: > Proposed fix: review before running. The agent does not run this. ```bash kubectl scale deployment my-app -n my-namespace --replicas=2 # Undo: kubectl scale deployment my-app -n my-namespace --replicas=1 ``` Add a readiness probe and requests (use the app's real health path, port and usage). A JSON patch `add` replaces the whole field: if the container already has some requests, put them in the value too: > Proposed fix: review before running. The agent does not run this. ```bash kubectl patch deployment my-app -n my-namespace --type=json -p '[{"op": "add", "path": "/spec/template/spec/containers/0/readinessProbe", "value": {"httpGet": {"path": "/healthz", "port": 8080}}}, {"op": "add", "path": "/spec/template/spec/containers/0/resources/requests", "value": {"cpu": "100m", "memory": "128Mi"}}]' # Undo: kubectl rollout undo deployment my-app -n my-namespace ``` Let one pod of a PDB be evicted: > Proposed fix: review before running. The agent does not run this. ```bash kubectl patch pdb my-app -n my-namespace --type=json -p '[{"op": "remove", "path": "/spec/minAvailable"}, {"op": "add", "path": "/spec/maxUnavailable", "value": 1}]' # Undo: kubectl patch pdb my-app -n my-namespace --type=json -p '[{"op": "remove", "path": "/spec/maxUnavailable"}, {"op": "add", "path": "/spec/minAvailable", "value": 1}]' ``` Drop privileges from a container (only once its owner confirms the app runs without them; a container that needs root fails to start). The patch replaces the container's whole `securityContext`: copy the fields it already has into the value. A `hostPath` volume found with it is removed in the source, with its volume mount: > Proposed fix: review before running. The agent does not run this. ```bash kubectl patch deployment my-app -n my-namespace --type=json -p '[{"op": "add", "path": "/spec/template/spec/containers/0/securityContext", "value": {"privileged": false, "allowPrivilegeEscalation": false, "runAsNonRoot": true, "capabilities": {"drop": ["ALL"]}}}]' # Undo: kubectl rollout undo deployment my-app -n my-namespace ``` Enforce a Pod Security level on a namespace. Check first what it would refuse: the server-side dry run below only prints warnings and changes nothing. > Proposed fix: review before running. The agent does not run this. ```bash kubectl label --dry-run=server --overwrite namespace my-namespace pod-security.kubernetes.io/enforce=baseline kubectl label namespace my-namespace pod-security.kubernetes.io/enforce=baseline # Undo: kubectl label namespace my-namespace pod-security.kubernetes.io/enforce- ``` Remove a `cluster-admin` binding that is not needed (save it first; give the subject a narrower role before, if it still needs some access): > Proposed fix: review before running. The agent does not run this. ```bash kubectl get clusterrolebinding my-app-admin -o yaml > my-app-admin.yaml kubectl delete clusterrolebinding my-app-admin # Undo: kubectl create -f my-app-admin.yaml ``` Delete leftovers once their owner confirms nobody needs them: > Proposed fix: review before running. The agent does not run this. ```bash kubectl delete job my-job -n my-namespace kubectl get service my-app -n my-namespace -o yaml > my-app-service.yaml kubectl delete service my-app -n my-namespace # Undo: re-run the Job from its source; kubectl create -f my-app-service.yaml (it asks for the same ClusterIP again, and fails if another Service took it: remove spec.clusterIP and spec.clusterIPs from the file to get a new one) ``` > Proposed fix: review before running. The agent does not run this. ```bash kubectl delete pvc my-data -n my-namespace # Undo: not reversible. With reclaim policy Delete the volume and its data are deleted; # with Retain the PV stays Released and needs an admin to bind it again. ``` Add a default-deny NetworkPolicy, and the allow rules the apps need, through their owners: see [references/security.md](references/security.md). A `Released` PV is deleted or bound again by an admin: see [references/hygiene.md](references/hygiene.md). ## Sources Checked against the Kubernetes 1.37 documentation, and on a Kubernetes 1.37 kind cluster: - [Configuration Good Practices](https://kubernetes.io/docs/concepts/configuration/overview/). - [Configure Liveness, Readiness and Startup Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/). - [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) and [Pod Quality of Service Classes](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/). - [Specifying a Disruption Budget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/). - [Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/). - [Pod Security Standards](https://kubernetes.io/docs/concepts/security/pod-security-standards/) and [Pod Security Admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/). - [Configure a Security Context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/). - [Role Based Access Control Good Practices](https://kubernetes.io/docs/concepts/security/rbac-good-practices/). - [Network Policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/). - [Images](https://kubernetes.io/docs/concepts/containers/images/): tags, digests and `imagePullPolicy`. - [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/): reclaim policies and the `Released` phase. - [Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/) and [CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/). - That kindnet enforces NetworkPolicy was seen on the kind cluster: a default-deny policy stopped traffic that reached the pod before it.