# AKS Engine CLI Overview AKS Engine is designed to be used as a CLI tool (`aks-engine-azurestack`). This document outlines the functionality that `aks-engine-azurestack` provides to create and maintain a Kubernetes cluster on Azure. ## `aks-engine-azurestack` commands To get a quick overview of the commands available via the `aks-engine-azurestack` CLI tool, just run `aks-engine-azurestack` with no arguments (or include the `--help` argument): ```sh $ aks-engine Usage: aks-engine-azurestack [flags] aks-engine-azurestack [command] Available Commands: addpool Add a node pool to an existing AKS Engine-created Kubernetes cluster completion Generates bash completion scripts deploy Deploy an Azure Resource Manager template generate Generate an Azure Resource Manager template get-logs Collect logs and current cluster nodes configuration. get-versions Display info about supported Kubernetes versions help Help about any command rotate-certs (experimental) Rotate certificates on an existing AKS Engine-created Kubernetes cluster scale Scale an existing AKS Engine-created Kubernetes cluster upgrade Upgrade an existing AKS Engine-created Kubernetes cluster version Print the version of aks-engine Flags: --debug enable verbose debug logs -h, --help help for aks-engine --show-default-model Dump the default API model to stdout Use "aks-engine-azurestack [command] --help" for more information about a command. ``` ## Operational Cluster Commands These commands are provided by AKS Engine in order to create and maintain Kubernetes clusters. Note: there is no `aks-engine-azurestack` command to delete a cluster; to delete a Kubernetes cluster created by AKS Engine, you must delete the resource group that contains cluster resources. If the resource group can't be deleted because it contains other, non-Kubernetes-relate Azure resources, then you must manually delete the Virtual Machine and/or Virtual Machine Scale Set (VMSS), Disk, Network Interface, Network Security Group, Public IP Address, Virtual Network, Load Balancer, and all other resources specified in the aks-engine-generated ARM template. Because manually deleting resources is tedious and requires following serial dependencies in the correct order, it is recommended that you dedicate a resource group for the Azure resources that AKS Engine will create to run your Kubernetes cluster. If you're running more than one cluster, we recommend a dedicated resource group per cluster. ### `aks-engine-azurestack deploy` The `aks-engine-azurestack deploy` command will create a new cluster from scratch, using an API model (cluster definition) file as input to define the desired cluster configuration and shape, in the subscription, region, and resource group you provide, using credentials that you provide. Use this command to create a new cluster. ```sh $ aks-engine-azurestack deploy --help Deploy an Azure Resource Manager template, parameters file and other assets for a cluster Usage: aks-engine-azurestack deploy [flags] Flags: -m, --api-model string path to your cluster definition file --auth-method client_secret auth method (default:client_secret, `client_certificate`) --auto-suffix automatically append a compressed timestamp to the dnsPrefix to ensure unique cluster name automatically --azure-env string the target Azure cloud (default "AzurePublicCloud") --ca-certificate-path string path to the CA certificate to use for Kubernetes PKI assets --ca-private-key-path string path to the CA private key to use for Kubernetes PKI assets --certificate-path string path to client certificate (used with --auth-method=client_certificate) --client-id string client id (used with --auth-method=[client_secret|client_certificate]) --client-secret string client secret (used with --auth-method=client_secret) -p, --dns-prefix string dns prefix (unique name for the cluster) -f, --force-overwrite automatically overwrite existing files in the output directory -h, --help help for deploy --identity-system azure_ad identity system (default:azure_ad, `adfs`) (default "azure_ad") --language string language to return error messages in (default "en-us") -l, --location string location to deploy to (required) -o, --output-directory string output directory (derived from FQDN if absent) --private-key-path string path to private key (used with --auth-method=client_certificate) -g, --resource-group string resource group to deploy to (will use the DNS prefix from the apimodel if not specified) --set stringArray set values on the command line (can specify multiple or separate values with commas: key1=val1,key2=val2) -s, --subscription-id string azure subscription id (required) Global Flags: --debug enable verbose debug logs ``` Detailed documentation on `aks-engine-azurestack deploy` can be found [here](../topics/creating_new_clusters.md#deploy). ### `aks-engine-azurestack scale` The `aks-engine-azurestack scale` command will scale (in or out) a specific node pool participating in a Kubernetes cluster created by AKS Engine. Use this command to manually scale a node pool to a specific number of nodes. ```sh $ aks-engine-azurestack scale --help Scale an existing AKS Engine-created Kubernetes cluster by specifying a new desired number of nodes in a node pool Usage: aks-engine-azurestack scale [flags] Flags: -m, --api-model string path to the generated apimodel.json file --apiserver string apiserver endpoint (required to cordon and drain nodes) --auth-method client_secret auth method (default:client_secret, `client_certificate`) --azure-env string the target Azure cloud (default "AzurePublicCloud") --certificate-path string path to client certificate (used with --auth-method=client_certificate) --client-id string client id (used with --auth-method=[client_secret|client_certificate]) --client-secret string client secret (used with --auth-method=client_secret) -h, --help help for scale --identity-system azure_ad identity system (default:azure_ad, `adfs`) (default "azure_ad") --language string language to return error messages in (default "en-us") -l, --location string location the cluster is deployed in -c, --new-node-count int desired number of nodes --node-pool string node pool to scale --private-key-path string path to private key (used with --auth-method=client_certificate) -g, --resource-group string the resource group where the cluster is deployed -s, --subscription-id string azure subscription id (required) Global Flags: --debug enable verbose debug logs ``` The `scale` command has limitations for scaling in (reducing the number of nodes in a node pool): - It accepts a new, desired node count; it does not accept a list of specific nodes to remove from the pool. - For VMSS-backed node pools, the removed nodes will not be cordoned and drained prior to being removed, which means any running workloads on nodes-to-be-removed will be disrupted without warning, and temporary operational impact is to be expected. We generally recommend that you manage node pool scaling dynamically using the `cluster-autoscaler` project. More documentation about `cluster-autoscaler` is [here](../../examples/addons/cluster-autoscaler/README.md), including how to automatically install and configure it at cluster creation time as an AKS Engine addon. Detailed documentation on `aks-engine-azurestack scale` can be found [here](../topics/scale.md). ### `aks-engine-azurestack addpool` The `aks-engine-azurestack addpool` command will add a new node pool to an existing AKS Engine-created cluster. Using a JSON file to define a the new node pool's configuration, and referencing the aks-engine-generated `apimodel.json`, you can add new nodes to your cluster. Use this command to add a specific number of new nodes using a discrete configuration compared to existing nodes participating in your cluster. ```sh $ aks-engine-azurestack addpool --help Add a node pool to an existing AKS Engine-created Kubernetes cluster by referencing a new agentpoolProfile spec Usage: aks-engine-azurestack addpool [flags] Flags: -m, --api-model string path to the generated apimodel.json file --auth-method client_secret auth method (default:client_secret, `client_certificate`) --azure-env string the target Azure cloud (default "AzurePublicCloud") --certificate-path string path to client certificate (used with --auth-method=client_certificate) --client-id string client id (used with --auth-method=[client_secret|client_certificate]) --client-secret string client secret (used with --auth-method=client_secret) -h, --help help for addpool --identity-system azure_ad identity system (default:azure_ad, `adfs`) (default "azure_ad") --language string language to return error messages in (default "en-us") -l, --location string location the cluster is deployed in -p, --node-pool string path to a JSON file that defines the new node pool spec --private-key-path string path to private key (used with --auth-method=client_certificate) -g, --resource-group string the resource group where the cluster is deployed -s, --subscription-id string azure subscription id (required) Global Flags: --debug enable verbose debug logs ``` Detailed documentation on `aks-engine-azurestack addpool` can be found [here](../topics/addpool.md). ### `aks-engine-azurestack upgrade` The `aks-engine-azurestack upgrade` command orchestrates a Kubernetes version upgrade across your existing cluster nodes. Use this command to upgrade the Kubernetes version running your control plane, and optionally on all your nodes as well. ```sh $ aks-engine-azurestack upgrade --help Upgrade an existing AKS Engine-created Kubernetes cluster, one node at a time Usage: aks-engine-azurestack upgrade [flags] Flags: -m, --api-model string path to the generated apimodel.json file --auth-method client_secret auth method (default:client_secret, `client_certificate`) --azure-env string the target Azure cloud (default "AzurePublicCloud") --certificate-path string path to client certificate (used with --auth-method=client_certificate) --client-id string client id (used with --auth-method=[client_secret|client_certificate]) --client-secret string client secret (used with --auth-method=client_secret) --control-plane-only upgrade control plane VMs only, do not upgrade node pools --cordon-drain-timeout int how long to wait for each vm to be cordoned in minutes (default -1) -f, --force force upgrading the cluster to desired version. Allows same version upgrades and downgrades. -h, --help help for upgrade --identity-system azure_ad identity system (default:azure_ad, `adfs`) (default "azure_ad") -b, --kubeconfig string the path of the kubeconfig file --language string language to return error messages in (default "en-us") -l, --location string location the cluster is deployed in (required) --private-key-path string path to private key (used with --auth-method=client_certificate) -g, --resource-group string the resource group where the cluster is deployed (required) -s, --subscription-id string azure subscription id (required) -k, --upgrade-version string desired kubernetes version (required) --upgrade-windows-vhd upgrade image reference of the Windows nodes (default true) --vm-timeout int how long to wait for each vm to be upgraded in minutes (default -1) Global Flags: --debug enable verbose debug logs ``` Detailed documentation on `aks-engine-azurestack upgrade` can be found [here](../topics/upgrade.md). ## Generate an ARM Template AKS Engine also provides a command to generate a reusable ARM template only, without creating any actual Azure resources. ### `aks-engine-azurestack generate` The `aks-engine-azurestack generate` command is similar to `aks-engine-azurestack deploy`: it uses an API model (cluster definition) file as input to define the desired cluster configuration and shape of a new Kubernetes cluster. Unlike `deploy`, `aks-engine-azurestack generate` does not actually submit any operational requests to Azure, but is instead used to generate a reusable ARM template which may be deployed at a later time. Use this command as a part of a workflow that creates one or more Kubernetes clusters via an ARM group deployment that takes an ARM template as input (e.g., `az deployment group create` using the standard `az` Azure CLI). ```sh $ aks-engine-azurestack generate --help Generates an Azure Resource Manager template, parameters file and other assets for a cluster Usage: aks-engine-azurestack generate [flags] Flags: -m, --api-model string path to your cluster definition file --ca-certificate-path string path to the CA certificate to use for Kubernetes PKI assets --ca-private-key-path string path to the CA private key to use for Kubernetes PKI assets --client-id string client id --client-secret string client secret -h, --help help for generate --no-pretty-print skip pretty printing the output -o, --output-directory string output directory (derived from FQDN if absent) --parameters-only only output parameters files --set stringArray set values on the command line (can specify multiple or separate values with commas: key1=val1,key2=val2) Global Flags: --debug enable verbose debug logs ``` Detailed documentation on `aks-engine-azurestack generate` can be found [here](../topics/creating_new_clusters.md#generate). ### `aks-engine-azurestack rotate-certs` Detailed documentation on `aks-engine-azurestack rotate-certs` can be found [here](../topics/rotate-certs.md). ### `aks-engine-azurestack get-logs` The `aks-engine-azurestack get-logs` can conveniently collect host VM logs from your Linux node VMs for local troubleshooting. *This command does not support Windows nodes*. The command assumes that your node VMs have an SSH daemon listening on port 22, that all nodes share a common SSH keypair for interactive login, and that a public endpoint exists on one of the control plane VMs for accommodating SSH agent key forwarding. ```sh $ aks-engine-azurestack get-logs --help Usage: aks-engine-azurestack get-logs [flags] Flags: -m, --api-model string path to the generated apimodel.json file (required) --control-plane-only get logs from control plane VMs only -h, --help help for get-logs --linux-script string path to the log collection script to execute on the cluster's Linux nodes (required) --linux-ssh-private-key string path to a valid private SSH key to access the cluster's Linux nodes (required) -l, --location string Azure location where the cluster is deployed (required) -o, --output-directory string collected logs destination directory, derived from --api-model if missing --ssh-host string FQDN, or IP address, of an SSH listener that can reach all nodes in the cluster (required) Global Flags: --debug enable verbose debug logs ``` The `aks-engine-azurestack` codebase contains a working log retrieval script in `scripts/collect-logs.sh`, so you can use it to quickly gather logs from your node VMs: ```sh $ git clone https://github.com/Azure/aks-engine-azurestack.git && cd aks-engine Cloning into 'aks-engine'... remote: Enumerating objects: 44, done. remote: Counting objects: 100% (44/44), done. remote: Compressing objects: 100% (42/42), done. remote: Total 92107 (delta 13), reused 15 (delta 1), pack-reused 92063 Receiving objects: 100% (92107/92107), 92.86 MiB | 7.27 MiB/s, done. Resolving deltas: 100% (64711/64711), done. $ export LATEST_AKS_ENGINE_RELEASE=v0.56.0 $ git checkout $LATEST_AKS_ENGINE_RELEASE Note: checking out 'v0.56.0'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b HEAD is now at 666073d49 chore: updating Windows VHD with new cached artifacts (#3843) $ bin/aks-engine-azurestack get-logs --api-model _output/$CLUSTER_NAME/apimodel.json --location $CLUSTER_NAME --linux-ssh-private-key _output/$CLUSTER_NAME-ssh --linux-script ./scripts/collect-logs.sh --ssh-host $CLUSTER_NAME.$LOCATION.cloudapp.azure.com ... INFO[0062] Logs downloaded to _output//_logs ``` The following example assumes that the `$CLUSTER_NAME` environment variable is assigned to the value of the cluster name (`properties.masterProfile.dnsPrefix` in the cluster API model), and that `$LOCATION` is assigned to the location string of the resource group that your cluster was created into.