# Command Line Interface for IBM Aspera products ## Introduction ### Overview The IBM Aspera Command Line Interface (`ascli`) is an enterprise-grade tool that provides comprehensive command-line access to IBM Aspera's suite of high-speed file transfer products and services. Designed for automation, scripting, and integration scenarios, `ascli` enables organizations to leverage Aspera's FASP technology within their existing workflows and DevOps pipelines. > [!TIP] > For any question about `ascli` (behavior, options), or to build a command line, give this manual to an AI assistant as a reference and ask it. > See [Leveraging AI Assistance](#leveraging-ai-assistance). ![Architecture: transfer agents, IBM Aspera applications and transfer servers](architecture.svg) ### Key Features **Multi-Product Integration** : Unified interface supporting Aspera on Cloud, Faspex 5, Shares, Node API, Console, Orchestrator, and High-Speed Transfer Server (HSTS). **Flexible Authentication** : Support for OAuth 2.0, JWT, Basic Authentication, and SSH key-based authentication across different Aspera products. **Transfer Automation** : High-speed FASP transfers with multiple agent support (Direct, Connect, Node, HTTP Gateway, Transfer Daemon). **Configuration Management** : Persistent configuration presets, environment variables, and flexible option handling for streamlined operations. **Developer Friendly** : Comprehensive logging, debugging capabilities, and API exploration tools for development and troubleshooting. ### Target Audience This manual is intended for: - **System Administrators** managing Aspera infrastructure - **DevOps Engineers** integrating Aspera into CI/CD pipelines - **Developers** building automation scripts and workflows - **IT Professionals** performing file transfer operations ### Document Structure This manual is organized into the following sections: 1. **Quick Start** - Getting started with basic operations 1. **Installation** - Setup procedures for various platforms 1. **Command Line Interface** - Syntax, options, configuration, transfers, and usage patterns 1. **Plugin sections** (one per product: AoC, ATS, HSTS, Faspex 5, and so on) - Product-specific operations and examples 1. **Operational Utilities** - Sync, hot folders, health checks, email notifications 1. **Common problems** - Common issues and solutions 1. **About** - History and references ### When to use and when not to use > [!NOTE] > If you are scripting or automating transfers from the command line, `ascli` is the right choice. > If you are developing an application, prefer the APIs or SDKs instead. `ascli` is designed for command-line interaction with IBM Aspera products, enabling users to execute remote commands and perform high-performance file transfers. It supports both interactive terminal operations (for example, maintenance tasks on VT100-compatible terminals) and scripting use cases (for example, batch jobs via shell scripts or `cron`). Internally, `ascli` integrates several components: - A configuration file (`config.yaml`) for persistent settings - Advanced command-line options (see [Extended Value](#extended-value-syntax)) - REST API calls, including OAuth (like `curl`) - Aspera's `ascp` for high-speed file transfers For programmatic integration with languages such as C/C++, Go, Python, NodeJS, and others, it is recommended to use the [Aspera APIs](https://ibm.biz/aspera_api) directly. These include: - REST APIs for products like Aspera on Cloud (AoC), Faspex 5, and Node - The Transfer Daemon with gRPC interfaces and language-specific stubs (C/C++, Python, .NET/C#, Java, Go, Ruby, Rust, and so on) These APIs are generally more suitable for long-term development and maintenance. Example implementations can be found at: . For scripting and ad hoc command-line tasks, `ascli` is ideal. It is developer-friendly and well-suited for testing and learning Aspera APIs (see [Logging, Debugging](#logging-debugging)). ### `ascli` or `ascp`? `ascp` is the low-level command-line utility that implements the FASP protocol and is used for actual data transfers. Every Aspera transfer involves an `ascp` process on both the client and server sides. `ascli` is not an alternative transfer engine: it starts `ascp` (directly or through another [transfer agent](#agent-direct)), so transfer performance is identical. Other low-level command line tools are included with Aspera transfer products, and `ascli` is a front end for them too: - `async`: the IBM Aspera Sync utility, to synchronize folders (see [IBM Aspera Sync](#ibm-aspera-sync)). - `ascp4`: an alternate transfer engine, selected with transfer spec parameter `use_ascp4`. `ascp4` will be deprecated: use `ascp` unless one of its specific features is needed. Used directly, `ascp` is limited to sending and receiving files: - Connection parameters (address, ports, user, credentials) must be provided on each command line: there are no connection profiles. - The remote server cannot be browsed: remote paths must be known in advance. - With Aspera applications (AoC, Faspex, Shares, Node API), the transfer address and token must first be obtained through the application's REST API. - Multi-session requires starting and monitoring one `ascp` process per session (option `-C`). - On interruption, `ascp` exits with an error: data already at destination is skipped on the next run (option `-k`), but restarting is up to the caller. `ascli` adds: - Configuration presets, with secrets protected in a vault (see [Option Preset](#option-preset) and [Secret Vault](#secret-vault)) - Detection of the Aspera application from its URL, and automatic retrieval of transfer parameters and tokens (see [Wizard](#wizard)) - Remote file browsing and management: list, create folder, rename, copy, delete, checksum - Built-in multi-session, with an aggregated progress bar (see [Multi-session transfer](#multi-session-transfer)) - Automatic resume on retry-able errors, with increasing delay and token renewal (see [Agent: Direct](#agent-direct)) - Choice of transfer agents: Direct, Node, Connect, HTTP Gateway, Transfer Daemon - Structured output (table, JSON, YAML, CSV) for scripting (see [Output](#output)) All `ascp` options remain available, either through transfer spec parameters (listed with `config ascp spec`), or by passing `ascp` arguments directly when using the `direct` agent (via `ascp_args` in option `transfer`). Using `ascp` directly remains relevant when: - `ascli` cannot be installed: `ascp` is included with every Aspera transfer product. - The need is limited to simple transfers with an HSTS using SSH credentials. To use `ascp` directly as a command line, refer to the IBM Aspera documentation of either [Desktop Client](https://www.ibm.com/docs/en/asdc), [Endpoint](https://www.ibm.com/docs/en/ahte) or [Transfer Server](https://www.ibm.com/docs/en/ahts) - each of which includes [a dedicated section on `ascp`](https://www.ibm.com/docs/en/ahts/4.4?topic=linux-ascp-transferring-from-command-line). ### Notations, Shell, Examples Examples of command-line operations are shown using a shell such as: `bash` (Linux) or `zsh` (macOS). Using [Windows PowerShell or cmd](#shell-parsing-for-windows) is also possible. Command line examples listed in sections titled **Tested commands for `_plugin_name_`** are verified during version validation. Command line arguments formatted as `` in examples represent user-provided values, not literal values. `ascli` is an API **Client** toward the remote Aspera application **Server** (Faspex, HSTS, and so on). Some commands will start an Aspera transfer (for example, `upload`). The transfer is not implemented directly in `ascli`; rather, `ascli` uses one of the external Aspera Transfer Clients called **[Transfer Agents](#transfer-clients-agents)**. > [!NOTE] > A **[Transfer Agent](#transfer-clients-agents)** is a client for the remote Transfer Server (HSTS/HSTE). > It can be local, or remote. > For example, a remote Aspera Transfer Server may be used as a transfer agent through its Node API, using option `--transfer=node`. ## Quick Start This section walks you through your first interaction with `ascli` on Linux. ### Prerequisites - Get the `ascli` binary for Linux (.tgz) in the [release section of the GitHub repository](https://github.com/IBM/aspera-cli/releases). - Decompress the archive to get the executable: ```shell mkdir -p $HOME/bin tar -C $HOME/bin -zxvf ascli-4.29.0.pre-linux-x86_64.tgz export PATH=$PATH:$HOME/bin ``` > [!NOTE] > For other operating systems, complete the [Installation](#installation) section (Ruby, Gem, FASP) to get `ascli` set up on your system. - Once installed, confirm `ascli` is accessible by checking its version: ```shell ascli -v ``` ```text 4.29.0.pre ``` - Install the Aspera transfer runtime (tested version), as it is not included in the `ascli` package: ```shell ascli transferd install ``` ### Option A - Test with the Aspera Demo Server - Run the following command to initialize the demo environment: ```shell ascli config initdemo ``` - Check the configuration: ```shell ascli config preset overview ``` - Browse the remote file system: ```shell ascli server browse / ``` ```text ╭────────────┬──────┬───────────┬───────┬───────────────────────────┬───────────────────────╮ │ zmode │ zuid │ zgid │ size │ mtime │ name │ ╞════════════╪══════╪═══════════╪═══════╪═══════════════════════════╪═══════════════════════╡ │ dr-xr-xr-x │ xfer │ demousers │ 4096 │ 2014-11-05 16:01:56 +0100 │ aspera-test-dir-large │ │ drwxrwxr-x │ xfer │ demousers │ 94208 │ 2025-03-31 11:27:33 +0200 │ Upload │ │ dr-xr-xr-x │ xfer │ demousers │ 4096 │ 2014-11-05 16:01:56 +0100 │ aspera-test-dir-small │ │ dr-xr-xr-x │ xfer │ demousers │ 4096 │ 2014-11-05 16:01:56 +0100 │ aspera-test-dir-tiny │ ╰────────────┴──────┴───────────┴───────┴───────────────────────────┴───────────────────────╯ ``` - Download a file: ```shell ascli server download /aspera-test-dir-small/10MB.1 ``` ```text Time: 00:00:02 ====================================== 100% 100 Mbps Time: 00:00:00 complete ``` ### Option B - Test with Your Own HSTS To use `ascli` with a server of your own, it is recommended to save its connection details as an [Option Preset](#option-preset). This avoids repeating credentials on every command. The steps below create a preset, set it as the default for the server plugin, browse a remote directory, and download a file: - Create a preset with your server's connection details: ```shell ascli config preset update --url=ssh://hsts.example.com:33001 --username= --password= ``` ```text INFO Saving config file: /home/john/.aspera/ascli/config.yaml Updated: ``` - Set the preset as the default for the server plugin: ```shell ascli config preset set default server ``` ```text INFO Updated: default: server <- INFO Saving config file: /home/john/.aspera/ascli/config.yaml ``` - Once your preset is set, follow the same browse and download steps as in [Option A](#option-a---test-with-the-aspera-demo-server). ### Option C - Test with Aspera on Cloud If you'd prefer to test against Aspera on Cloud, skip ahead to the [AoC Wizard](#aoc-configuration-using-wizard) section. ### Leveraging AI Assistance You can significantly accelerate your workflow by using an AI assistant with `ascli`. There are three ways, the first one is the simplest: - **The AI executes commands in a shell** (coding agents): nothing to set up, and no limitation: the AI uses `ascli` as you do, with your configuration. Give it the file [`llms.txt`](https://raw.githubusercontent.com/IBM/aspera-cli/refs/heads/main/llms.txt), located at the root of the gem and of the GitHub repository. It tells the AI what `ascli` does with IBM Aspera products, and how to discover commands, arguments and options (`config commands`, `help`, `config options`, `config documentation`). With an AI client that supports agent skills, install the skill instead: the AI loads it by itself when a request is about IBM Aspera. The skill is the folder `skills/ascli` of the gem (and of the GitHub repository): copy it to the skills folder of the AI client, for example: ```shell cp -R "$(gem contents aspera-cli | grep 'skills/ascli/SKILL.md$' | xargs dirname)" ~/.claude/skills/ ``` - **The AI executes commands through MCP**: for an AI client without a shell, start the [MCP server](#plugin-mcp-model-context-protocol-server) and register it in the AI client. The AI discovers commands and their arguments by itself, and executes them. By default, the server restricts what the AI can do (see [Security](#security)), and the AI never sees secrets. - **The AI generates commands that you execute** (chat assistant that cannot execute commands): it cannot read the documentation through `ascli`, so provide it as context, as described below. In the first two ways, the AI reads the documentation by itself, one section at a time, with `ascli config documentation`: there is no need to provide the manual. Recommended Workflow for a chat assistant: - **Contextualize**: Download the project manual (this manual in Markdown format, [`docs/README.md`](https://raw.githubusercontent.com/IBM/aspera-cli/refs/heads/main/docs/README.md)) from the GitHub repository. - **Upload**: Attach the file to your AI assistant's prompt to ensure it uses specific, up-to-date syntax. - **Inquire**: Use a detailed prompt that specifies your requirements and environment. > [!TIP] > Specify your environment. > Quoting rules and syntax vary significantly between shells. > Always mention if you are using `bash`, **PowerShell** 5 or 7, or Windows `CMD`. Example Prompt: ```text Use only the attached ascli manual (it matches the version I use). Generate a command to send a package via Aspera on Cloud using the Bash shell. Set a custom title and note. Define specific recipients. Cap the transfer speed at 300Mbps. ``` By providing the documentation as a direct reference, you reduce "hallucinations" and receive accurate, executable commands in seconds. ### Next Steps - Learn the CLI: Read [Command Line Interface](#command-line-interface) to understand configuration, options, and commands. - Explore plugins: Jump to the section for the relevant product, for example: [Aspera on Cloud](#plugin-aoc-ibm-aspera-on-cloud), [HSTS](#plugin-server-ibm-aspera-high-speed-transfer-server-ssh), [Faspex 5](#plugin-faspex5-ibm-aspera-faspex-v5). ## Installation There are several ways to install `ascli`: - Using a Ruby environment directly on the host operating system (Linux, macOS, Windows). This is the most general method. It consists of installing: - The [Ruby language](#ruby), - The [aspera-cli](#ruby-gem-aspera-cli) Ruby gem, - [Aspera Transfer Daemon (`ascp`)](#fasp-protocol-ascp). - As a [single file executable](#single-file-executable) This method is simple, but only a limited number of platforms are supported. - On Linux, as a [portable package](#linux-portable-package) This method is simple on Linux: extract and run, without root access. It includes Ruby, gems and `ascp`. - On Windows, as a [portable package](#windows-portable-package) This method is the simplest on Windows: extract and run. It includes Ruby, gems and `ascp`. - On Windows, with the [Chocolatey package](#windows-chocolatey-package) (installs Ruby and the gem). - As a [container](#container) (`docker`, `podman`, `singularity`). The following sections describe the various installation methods. An internet connection is required during installation. If you do not have internet access, see [Installing in an air-gapped environment](#installing-in-an-air-gapped-environment). ### Single file executable > [!WARNING] > Available only on a limited number of platforms. `ascli` is available as a single **platform-dependent executable** in the [Releases](https://github.com/IBM/aspera-cli/releases). This executable includes the Ruby runtime and gems, but not the transfer SDK. #### Installing the single file executable > [!NOTE] > Replace `` and `` with the values of the downloaded archive, for example: `linux-x86_64-glibc2.28-ocran`. > The archive contains the executable `ascli` and its `README.ascli.md`. > Installation of `ascp` is still required separately. > See [Install `ascp`](#installing-ascp-through-transferd). ```shell tar zxvf aspera-cli--.tgz ./ascli transferd install ``` #### Linux: Checking the GLIBC version > [!WARNING] > On Linux, the executable requires a minimum GLIBC version, specified in the archive name on the download site (for example: `glibc2.28`). > If the minimum version is not met, then executables (`ascp`, `transferd`) will exit with error. On Linux, you can check your system's GLIBC version on this site: [repology.org](https://repology.org/project/glibc/versions), or check your GLIBC version with `ldd`: ```shell ldd --version | head -n1 ``` ```text ldd (GNU libc) 2.34 ``` Check an executable's (for example, `/bin/bash`, `ascli`, `ascp`) minimum required GLIBC version: ```shell objdump -p /bin/bash | sed -n 's/^.*GLIBC_//p' | sort -V | tail -n1 ``` ```text 2.34 ``` > [!NOTE] > If `objdump` is not available, then use `strings` or `grep -z 'GLIBC_'|tr \\0 \\n` The required GLIBC version for `ascp` can be found in the [Release Notes of HSTS](https://www.ibm.com/docs/en/ahts) or [on this page](https://eudemo.asperademo.com/download/sdk.html). ### Linux: Portable package A ready-to-use archive for Linux (x86_64) is available in the [Releases](https://github.com/IBM/aspera-cli/releases): `aspera-cli--linux-x86_64-glibc2.28-portable.tgz`. It contains the Ruby runtime with the shared libraries it needs, the aspera-cli gem with its dependencies, and the Aspera Transfer SDK (`ascp`). No installation step, no root access, and no internet access are required. The GLIBC of the system must be at least the version in the archive name, see [Checking the GLIBC version](#linux-checking-the-glibc-version). 1. Extract the archive, for example in `~/.local/share`, and check that `ascli` runs: ```shell tar -xzf aspera-cli--linux-x86_64-glibc2.28-portable.tgz -C ~/.local/share ~/.local/share/aspera-cli--linux-x86_64-glibc2.28-portable/ascli -v ``` 2. Optionally, place a symbolic link to the launcher in a folder of the `PATH`. Then, `ascli` can be used from any folder: ```shell ln -s ~/.local/share/aspera-cli--linux-x86_64-glibc2.28-portable/ascli ~/.local/bin/ascli ascli -v ``` > [!NOTE] > The launcher `ascli` uses the `ascp` located in folder `sdk` of the package, unless environment variable `ASCLI_SDK_FOLDER` is set. > So, `ascli transferd install` is not needed. The configuration is stored in the [main folder](#main-configuration-and-persistency-folder), like for other installation methods. To upgrade, extract the new version, and update the symbolic link. To uninstall, delete the folder, and the symbolic link if it was created. ### Windows: Portable package A ready-to-use ZIP archive for Windows (x64) is available in the [Releases](https://github.com/IBM/aspera-cli/releases): `aspera-cli--windows-x86_64-portable.zip`. It contains the Ruby runtime, the aspera-cli gem with its dependencies, and the Aspera Transfer SDK (`ascp`). Apart from the prerequisite below, no installation step, no administrator rights, and no internet access are required. > [!IMPORTANT] > The package does not include the Microsoft Visual C++ Redistributable (x64), which `ascp` requires. > If it is not already present on the system, install it from [Microsoft Visual C++ Redistributable latest supported downloads](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist) (administrator rights required), otherwise transfers fail. 1. Download the ZIP archive, then right-click on it and select **Extract All...**. Preferably, extract in a folder writable by the user, for example: `%LOCALAPPDATA%\Programs`. 2. In a terminal, in the extracted folder, check that `ascli` runs: ```batchfile .\ascli.cmd -v ``` 3. Optionally, double-click on `add_to_path.cmd` to add the folder to the user's `PATH`. Then, in a new terminal, `ascli` can be used from any folder: ```batchfile ascli -v ``` 4. Optionally, activate the [shell completion](#shell-completion) of PowerShell (the folder must be in the `PATH`). > [!NOTE] > The launcher `ascli.cmd` uses the `ascp` located in folder `sdk` of the package, unless environment variable `ASCLI_SDK_FOLDER` is set. > So, `ascli transferd install` is not needed. The configuration is stored in the [main folder](#main-configuration-and-persistency-folder), like for other installation methods. To upgrade, extract the new version, and update the `PATH` if the folder name changed. To uninstall, delete the folder, and remove it from the `PATH` if it was added. ### Windows: Chocolatey package If you use [Chocolatey](https://chocolatey.org/), `ascli` can be installed with the `aspera-cli` package: it installs Ruby (if not already present) and the `aspera-cli` gem. In a PowerShell as Administrator: ```pwsh choco install aspera-cli -y ``` Then, install the Aspera Transfer Daemon, see [Installing `ascp` through `transferd`](#installing-ascp-through-transferd). ### Ruby A Ruby interpreter is required to run `ascli`. Required Ruby version is version: >= 3.3. > [!WARNING] > The minimum Ruby version will be 3.4 in a future version. **Ruby can be installed using any of the following methods**: `rpm`, `yum`, `dnf`, `rvm`, `rbenv`, `brew`, Windows installer, ... **First**, refer to the official Ruby documentation: - [Official Ruby Installation Guide](https://www.ruby-lang.org/en/documentation/installation/) - [Official Ruby Download](https://www.ruby-lang.org/en/downloads/) For convenience, you may refer to the following sections for a proposed method for specific operating systems. `ascli` supports a Ruby version until 6 months after its [end of official support](https://www.ruby-lang.org/en/downloads/branches/). The minimum version is increased accordingly, and announced in advance. If only an older Ruby version must be used due to system constraints, use an older version of `ascli` that supports it; refer to [rubygems.org](https://rubygems.org/gems/aspera-cli). The following table summarizes Ruby version life cycles and `ascli` support status: | Ruby Version | Official EOL Date | ascli Support End (EOL + 6 mo) | ascli Status | |--------------|-------------------|--------------------------------|-----------------------------------| | 4.0 | 2029-03-31 | 2029-09-30 | **Supported (Tested & Packaged)** | | 3.4 | 2028-03-31 | 2028-09-30 | **Supported (Future minimum)** | | 3.3 | 2027-03-31 | 2027-09-30 | **Supported (Current minimum)** | | 3.2 | 2026-03-31 | 2026-09-30 | End of Support | | 3.1 | 2025-03-26 | 2025-09-26 | End of Support | | 3.0 | 2024-04-23 | 2024-10-23 | End of Support | #### Windows: Installer Manual installation: - Navigate to [https://rubyinstaller.org/](https://rubyinstaller.org/) → **Downloads**. - Download the latest Ruby installer **"with devkit"**. (`Msys2` is needed to install some native extensions, such as `grpc`) - Execute the installer, which installs by default in: `C:\RubyVV-x64` (`VV` is the version number) - At the end of the installation procedure, the `Msys2` installer is automatically executed, select option 3 (`Msys2` and `mingw`) - Then install the `aspera-cli` gem and Aspera Transfer Daemon (see next sections) Automated installation (with internet access): The Ruby installer supports silent installation. To see the options, execute it with `/help`, or refer to the [Ruby Installer FAQ](https://github.com/oneclick/rubyinstaller2/wiki/FAQ). Download the Ruby installer executable from and then install: ```batchfile rubyinstaller-devkit-3.2.2-1-x64.exe /silent /currentuser /noicons /dir=C:\aspera-cli ``` #### Windows: Chocolatey If you use [Chocolatey](https://chocolatey.org/), then install Ruby with: ```powershell choco install ruby ``` #### macOS: `brew` If you use [Homebrew](https://brew.sh/), then install Ruby with: ```shell brew install ruby ``` > [!NOTE] > This is the recommended method. > [!WARNING] > **macOS** comes with Ruby 2.6. > It is an old unsupported version and [Apple has deprecated it](https://developer.apple.com/documentation/macos-release-notes/macos-catalina-10_15-release-notes). > It will be removed from macOS in the future. > Do not use it. > Use the required version of Ruby. This installs a recent Ruby version suitable for `ascli`. After installation, Homebrew's Ruby is not on the `PATH` by default (this is shown at the end of the `brew install ruby` output and by `brew info ruby`). Add it to your shell configuration file: - **zsh** (default shell on macOS; run this once in a terminal): ```shell echo 'PATH="$(brew --prefix ruby)/bin:$($(brew --prefix ruby)/bin/gem env gemdir)/bin:$PATH"' >> ~/.zprofile source ~/.zprofile ``` - **bash**: replace `~/.zprofile` with `~/.bash_profile` in the commands above. #### Linux: Package `ascli` supports Linux distributions that are officially maintained by their vendor (Red Hat for RHEL / UBI, Canonical for Ubuntu LTS). - **GLIBC Requirement:** The Transfer SDK executables (`ascp`, `transferd`) and the Linux portable package require **`glibc >= 2.28`**. - **Installation Compatibility:** - If the OS provides a Ruby package or DNF module stream meeting the required version (≥ 3.3), the `aspera-cli` gem can be installed directly with `gem install`. - If the OS is under maintenance but ships an older default system Ruby (e.g. Ubuntu 22.04 LTS with Ruby 3.0), `ascli` is fully supported via the [Linux portable package](#linux-portable-package), the [container image](#container), or a Ruby version manager (`rbenv`, `rvm`). The following table summarizes supported Linux distributions, vendor end of support dates, GLIBC versions, available Ruby streams, and recommended `ascli` installation modes: | Linux Distribution | OS Support End (EOL) | GLIBC (>= 2.28) | OS Ruby Version(s) | ascli Support / Mode | |----------------------|----------------------|--------------------|--------------------------------------|-------------------------------------------------------| | **RHEL / UBI 10** | 2035-05-31 | 2.39+ (Compatible) | >= 3.3 (default) | Gem, Portable, Container | | **RHEL / UBI 9** | 2032-05-31 | 2.34 (Compatible) | 3.0 (default), 3.1, 3.3 (DNF module) | Gem (`dnf module enable ruby:3.3`), Portable, Container | | **RHEL / UBI 8** | 2029-05-31 | 2.28 (Compatible) | 3.1, 3.3 (DNF module) | Gem (`dnf module enable ruby:3.3`), Portable, Container | | **Ubuntu 26.04 LTS** | 2031-05-29 | 2.43+ (Compatible) | >= 3.3 (APT package) | Gem, Portable, Container | | **Ubuntu 24.04 LTS** | 2029-05-31 | 2.39 (Compatible) | 3.2 (APT package) | Gem (`apt install ruby-full`), Portable, Container | | **Ubuntu 22.04 LTS** | 2027-06-01 | 2.35 (Compatible) | 3.0 (APT package) | Portable, Container, rbenv (OS Ruby < 3.2) | If your Linux distribution provides a standard Ruby package, you can use it provided that the version is supported. **Example:** RHEL 8+, Rocky Linux 8+: with extensions to compile native gems: - Check available Ruby versions: ```shell dnf module list ruby ``` - If Ruby was already installed with an older version, remove it: ```shell dnf module -y reset ruby ``` - Install packages needed to build native gems: ```shell dnf install -y make automake gcc gcc-c++ kernel-devel ``` - Enable the Ruby version you selected: ```shell dnf module -y enable ruby:3.3 dnf install -y ruby-devel ``` **Example:** Ubuntu ```shell apt-get install -y ruby-full ``` **Other examples:** ```shell yum install -y ruby ruby-devel rubygems ruby-json ``` ```shell apt install -y ruby ruby-dev rubygems ruby-json ``` One can remove all installed gems, for example to start fresh: ```shell ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//'|sort -u|xargs gem uninstall -axI rm -f Gemfile.lock ``` #### Unix-like: RVM: Single user installation (not root) Install `rvm`. Follow [https://rvm.io/](https://rvm.io/). Execute the shell/curl command. As regular user, it installs in the user's home: `~/.rvm`. ```shell \curl -sSL https://get.rvm.io | bash -s stable ``` Follow on-screen instructions to install keys, and then re-execute the command. Upon RVM installation, open a new terminal or initialize with: ```shell source ~/.rvm/scripts/rvm ``` It is advised to use one of the precompiled Ruby versions; you can list them with: ```shell rvm list --remote ``` Install the chosen pre-compiled Ruby version: ```shell rvm install 3.2.2 ``` Ruby is now installed for the user, go to [Gem installation](#ruby-gem-aspera-cli). Alternatively RVM can be installed system-wide, for this execute as `root`. It then installs by default in `/usr/local/rvm` for all users and creates `/etc/profile.d/rvm.sh`. One can install in another location with: ```shell curl -sSL https://get.rvm.io | bash -s -- --path /usr/local ``` As root, verify that this will not collide with other applications using Ruby (for example, Faspex). If so, one can rename the environment script so that it is not loaded by default: ```shell mv /etc/profile.d/rvm.sh /etc/profile.d/rvm.sh.ok ``` To activate Ruby (and ascli) later, source it: ```shell source /etc/profile.d/rvm.sh.ok ``` ```shell rvm version ``` On macOS, one way to force use of OpenSSL 3.0 is: ```shell RUBY_CONFIGURE_OPTS="--with-openssl-dir=$(brew --prefix openssl@3.0)" rvm install 3.4.0 ``` #### Unix-like: `rbenv` If you do not have root access, you can install Ruby in your home directory using `rbenv`, see [`rbenv-installer`](https://github.com/rbenv/rbenv-installer#rbenv-installer): ```shell curl -fsSL https://github.com/rbenv/rbenv-installer/raw/HEAD/bin/rbenv-installer | bash ``` Then open a new terminal, or `source` the shell initialization script: ```shell source ~/.bashrc ``` Then install Ruby: ```shell rbenv install 3.2.2 ``` #### Other Unixes (AIX) Ruby is sometimes made available as an installable package through third party providers. For example for AIX, one can look at: If your Unix does not provide a pre-built Ruby, you can get it using one of those [methods](https://www.ruby-lang.org/en/documentation/installation/). For instance, to build from source and install in `/opt/ruby`: ```shell wget https://cache.ruby-lang.org/pub/ruby/x.y/ruby-x.y.z.tar.gz gzip -d ruby-x.y.z.tar.gz tar xvf ruby-x.y.z.tar cd ruby-x.y.z ./configure --prefix=/opt/ruby make ruby.imp make make install ``` #### JRuby `ascli` can also run with the [JRuby](https://www.jruby.org/) interpreter. All that is needed is a JVM (Java Virtual Machine) on your system (`java`). The JRuby package comes pre-compiled and does not require compilation of native extensions. Use a version of JRuby compatible with a Ruby version supported by `ascli`. See [the Wikipedia page](https://en.wikipedia.org/wiki/JRuby) to match JRuby and Ruby versions. Choose the latest version from: > [!WARNING] > The startup time is slightly longer using `jruby` than the native Ruby. > See the [JRuby wiki](https://github.com/jruby/jruby/wiki) for details. > This can be reduced by using the `--dev` option. > The transfer speed is not impacted (executed by `ascp` binary). > [!NOTE] > JRuby can be [installed](https://www.jruby.org/getting-started) using `rvm`. Example: Start `ascli` with JRuby and reduce startup time: ```shell export JRUBY_OPTS=--dev ascli -v ``` Or: ```shell JRUBY_OPTS=--dev ascli -v ``` #### Installing optional gems Some additional gems are required for either development or specific runtime features. For JRuby, some of them have a replacement gem, and others are not available. Those are not installed as part of dependencies because they involve compilation of native code but concern less-used features. See [Gemfile](../Gemfile): | name | version | comment | |------------------------|-----------|----------------------------------------------------------------------------------| | rake | ~> 13.0 | | | debug | ~> 1.11 | (no jruby) | | grpc-tools | ~> 1.67 | | | reek | ~> 6.6.0 | | | rspec | ~> 3.0 | | | rubocop | ~> 1.75 | | | rubocop-ast | ~> 1.4 | | | rubocop-performance | ~> 1.10 | (no jruby) | | rubocop-shopify | ~> 3.1 | | | simplecov | ~> 1.3 | | | solargraph | ~> 0.48 | (no jruby) | | ferrum | ~> 0.20.1 | (no jruby) | | mcp | ~> 1.2 | for ascli mcp server (MCP protocol support) | | rack | ~> 3.0 | for ascli mcp server HTTP transport (required by mcp StreamableHTTPTransport) | | grpc | ~> 1.71 | (no jruby) for Aspera Transfer Daemon | | sqlite3 | ~> 2.7 | (no jruby) for async DB | | jdbc-sqlite3 | ~> 3.46 | (jruby) for async DB | | sequel | ~> 5.96 | (jruby) for async DB | | bcrypt_pbkdf | ~> 1.1 | (no jruby) for passphrase-protected OpenSSH private keys (ed25519 is native in net-ssh 8) | | syslog | ~> 0.3 | (no jruby) for logger=syslog | | zxcvbn | ~> 1.0 | for better estimate of secret strength (faspex5 postprocessing secret_token) | | keeper_secrets_manager | ~> 17.2 | for vault of type keeper (Keeper Secrets Manager) | | rmagick | ~> 7.2 | (no jruby) for terminal view | Install like this: ```shell gem install rake -v '~> 13.0' gem install debug -v '~> 1.11' gem install grpc-tools -v '~> 1.67' gem install reek -v '~> 6.6.0' gem install rspec -v '~> 3.0' gem install rubocop -v '~> 1.75' gem install rubocop-ast -v '~> 1.4' gem install rubocop-performance -v '~> 1.10' gem install rubocop-shopify -v '~> 3.1' gem install simplecov -v '~> 1.3' gem install solargraph -v '~> 0.48' gem install ferrum -v '~> 0.20.1' gem install mcp -v '~> 1.2' gem install rack -v '~> 3.0' gem install grpc -v '~> 1.71' gem install sqlite3 -v '~> 2.7' gem install jdbc-sqlite3 -v '~> 3.46' gem install sequel -v '~> 5.96' gem install bcrypt_pbkdf -v '~> 1.1' gem install syslog -v '~> 0.3' gem install zxcvbn -v '~> 1.0' gem install keeper_secrets_manager -v '~> 17.2' gem install rmagick -v '~> 7.2' ``` ### Ruby gem: `aspera-cli` Once you have Ruby and rights to install gems, install the `aspera-cli` gem and its dependencies: ```shell gem install aspera-cli --pre ``` To upgrade to the latest version: ```shell gem update aspera-cli ``` During execution, `ascli` checks weekly whether a new version is available and notifies the user via a WARN log entry. To deactivate this feature, globally set the option `version_check_days` to `0`, or specify a different period in days. To check if a new version is available (independently of `version_check_days`): ```shell ascli config check_update ``` #### Verifying gem installation signature The gem is signed with a private key, and the public certificate is available in the GitHub repository (`certs/aspera-cli-public-cert.pem`). When installing the gem, the signature may optionally be verified. For [secure installation](https://guides.rubygems.org/security/#using-gems), one can install the gem with the public key: Import the verification certificate: ```shell gem cert --add <(curl -Ls https://raw.githubusercontent.com/IBM/aspera-cli/main/certs/aspera-cli-public-cert.pem) ``` The user installs the gem with `HighSecurity` or `MediumSecurity`: this will succeed only if the gem is trusted: ```shell gem install -P MediumSecurity aspera-cli ``` #### Installing a beta release of the gem A beta version of the gem can be found here: On Linux/macOS, install in a terminal: ```shell curl -sLo aspera-cli-beta.gem https://ibm.biz/aspera-cli-beta gem install aspera-cli-beta.gem ``` On Windows, download the file `aspera-cli-beta.gem` from the link, then install it with `gem install aspera-cli-beta.gem`. ### FASP Protocol: `ascp` File transfers are primarily performed using the **FASP** protocol via the `ascp` utility. To execute an Aspera transfer, only two additional files are required; both are included with the Aspera Transfer Daemon: - `ascp` - the transfer executable - `aspera-license` - the license file (located in the same directory as `ascp` or in `../etc`) These components can be installed either by installing the Aspera transfer software or by running the `ascli` command. #### Installing `ascp` through `transferd` The easiest option to install `ascp` is through the use of the IBM Aspera Transfer Daemon (`transferd`). Install it using `ascli` (for the current platform) with: ```shell ascli transferd install ``` The installation of the transfer binaries follows those steps: - **Select the SDK package to use**. Check the `sdk_url` option: - If the value is **not** the default value (`DEF`), it directly specifies the archive URL to download. - If the value is `DEF`, `ascli` downloads the YAML file from the URLs specified by the `locations_url` option. - URLs are tried in order until one succeeds. All but the last are tried with a short timeout and no retry. By default: , then its target on GitHub. - This YAML file lists supported architectures (OS, CPU) and Aspera Transfer Daemon versions with their associated package URLs. - The SDK version is selected as follows: if an additional **positional parameter** is provided it specifies the version; if the special value `LATEST` is given the latest available version is used; otherwise the version tested with this release of `ascli` is used. - The package URL matching the current system architecture is then used. - **Extract the archive** - By default, the archive is extracted to `$HOME/.aspera/sdk`. - The destination folder can be changed by setting the `sdk_folder` option. | Option | Description | |---------------|--------------------------------------------------------------| | `sdk_url` | URL to download the Aspera Transfer Daemon archive.
Default: `DEF` (select from available archives.) | | `locations_url` | List of URLs to get download URLs of Aspera Transfer Daemon from IBM official repository, tried in order.
A list given in options replaces the default list.
Default:
- `https://ibm.biz/sdk_location`
- `https://raw.githubusercontent.com/IBM/aspera-cli/refs/heads/main/docs/sdk_location.yaml`| | `sdk_folder` | Folder where the SDK archive is extracted.
Default: `$HOME/.aspera/sdk` | > [!NOTE] > Commands `config transferd ...` and `config ascp install` are deprecated aliases of `transferd ...` and `transferd install`. Transfer Daemon versions available from `locations_url` can be listed with: `ascli transferd list` To install a specific version, for example, 1.1.3: ```shell ascli transferd install 1.1.3 ``` To get the download URL for a specific platform and version: ```shell ascli transferd list --select.platform=osx-arm64 --select.version=1.1.3 --fields=url ``` To download it, pipe to `config download`: ```shell ascli transferd list --select.platform=osx-arm64 --select.version=1.1.3 --fields=url | ascli config download @stdin: ``` To install from a local file (air-gapped installation) instead of fetching from the internet, specify the location of the SDK file with option `sdk_url`: ```shell ascli transferd install --sdk-url=file:///macos-arm64-1.1.3-c6c7a2a.zip ``` The format is: `file:///` (canonical form) or `file:` (short form), where `` can be either a relative path (not starting with `/`), or an absolute path (starting with `/`). | Form | Relative example | Absolute example | |-----------|-----------------------------|----------------------------------| | Canonical | `file:///mydir/archive.zip` | `file:////home/user/archive.zip` | | Short | `file:mydir/archive.zip` | `file:/home/user/archive.zip` | Supported platforms are listed in the [Release Notes](https://developer.ibm.com/apis/catalog/aspera--aspera-transfer-sdk/Release+notes) and archives can be downloaded from [Downloads](https://developer.ibm.com/apis/catalog/aspera--aspera-transfer-sdk/downloads/downloads.json). #### Installing `ascp` through another component If the embedded method is not used, the following packages are also suitable: - IBM Aspera Connect Client (Free, to be deprecated in 2027, use IBM Aspera for Desktop) - IBM Aspera Desktop Client (Free) - IBM Aspera for Desktop (Free) - IBM Aspera High Speed Transfer Server (Licensed) - IBM Aspera High Speed Transfer Endpoint (Licensed) For instance, Aspera Connect Client can be installed by visiting the page: [https://www.ibm.com/aspera/connect/](https://www.ibm.com/aspera/connect/). `ascli` detects most Aspera transfer products in standard locations and use the first one found by default. See [FASP](#fasp-configuration) for details on how to select a client or set path to the FASP protocol. Several methods are provided to start a transfer. Use of a local client ([`direct`](#agent-direct) transfer agent) is one of them, but other methods are available. See [Transfer Agents](#transfer-clients-agents). ### Installing in an air-gapped environment For an air-gapped installation, two alternatives are available: 1. Get the installation components from a system with internet access: - Ruby installation packages (depends on system) - Gems Use the gem pack of the release. It includes the `aspera-cli` gem. On the air-gapped system, extract the archive and run: ```shell gem install *.gem ``` - Transfer SDK On a system with internet access, retrieve the SDK archive for the target platform and version: ```shell ascli transferd list --select.platform= --select.version= --fields=url | ascli config download @stdin: ``` On the air-gapped system, install from the downloaded file using option `--sdk-url`: ```shell ascli transferd install --sdk-url=file:/// ``` 2. Retrieve the [single file executable](#single-file-executable) from the release, if available, and the SDK (not included) as described above. #### Packaging gem files and dependencies Releases include gem packs: `gempack.zip` contains the minimal bundle of required gems. Alternatively, the necessary gems can be packaged into a `tar.gz` archive as follows: ```shell mkdir temp_folder gem install aspera-cli:4.29.0.pre --no-document --install-dir temp_folder find temp_folder mv temp_folder/cache aspera-cli-4.29.0.pre-gems rm -fr temp_folder tar zcvf aspera-cli-4.29.0.pre-gems.tgz aspera-cli-4.29.0.pre-gems ``` #### Unix-like: Alternative installation using `rvm` The following procedure applies when using RVM for the Ruby installation: 1. Follow the non-root installation procedure with RVM, including gem installation. 2. Archive the main RVM folder (includes ascli): ```shell cd $HOME && tar zcvf rvm-ascli.tgz .rvm ``` 3. Download the Transfer Daemon archive for the target architecture. See [Install `ascp`](#installing-ascp-through-transferd). 4. Transfer the two archive files to the target system. 5. On the target system, run: ```shell cd $HOME tar zxvf rvm-ascli.tgz source ~/.rvm/scripts/rvm ascli transferd install --sdk-url=file:/// ``` 6. Add the following line to the shell environment file (`.profile`): ```shell source ~/.rvm/scripts/rvm ``` #### Windows: Installing in an air-gapped environment > [!TIP] > The simplest method is to use the [portable package](#windows-portable-package), which requires no internet access on the target system. The procedure is similar to the internet-connected Windows installation. Copy the required files from a system with internet access, then install them on the target system. 1. Download the Ruby installer from : ```shell v=$(curl -s https://rubyinstaller.org/downloads/ | sed -nEe 's|.*(https://.*/releases/download/.*exe).*|\1|p' | head -n 1) curl -o ${v##*/} $v ``` 2. Create a gem archive as described in [Packaging gem files and dependencies](#packaging-gem-files-and-dependencies). 3. Download the Transfer Daemon. See [Install `ascp`](#installing-ascp-through-transferd). 4. Create a Zip archive containing all the files above and transfer it to the target system. On the target system: 1. Extract the archive. 2. Run the Ruby installer: ```batchfile /silent /currentuser /noicons /dir= ``` 3. Install the gems: extract the gem archive, then run: ```batchfile gem install --force --local *.gem ``` 4. Install the Aspera Transfer Daemon SDK: ```shell ascli transferd install --sdk-url=file:/// ``` ### Container The container image is: [`docker.io/martinlaurent/ascli`](https://hub.docker.com/r/docker.io/martinlaurent/ascli). The container contains: Ruby, `ascli` and the Aspera Transfer Daemon. To use the container, ensure that you have `podman` (or `docker`) installed. ```shell podman --version ``` #### Container: Quick start To start an interactive shell, run: ```shell podman run --rm --tty --interactive --entrypoint bash docker.io/martinlaurent/ascli:latest ``` > [!NOTE] > This command changes the entry point to an interactive shell instead of direct execution of `ascli`. Then, execute individual `ascli` commands such as: ```shell ascli config initdemo ascli config preset overview ascli config ascp info ascli server ls / ``` That is simple, but there are limitations: - Everything happens in the container. - Any file generated in the container, including configuration files and downloaded files, is lost when the container (shell) exits. - Files located on the host system cannot be uploaded. #### Container: Details The container image is built from this [Dockerfile](../build/container/Dockerfile.tmpl.erb). The entry point is `ascli` and the default command is `help`. The container can be executed for individual commands like this: (add `ascli` commands and options at the end of the command line, for example, `-v` to display the version) ```shell podman run --rm --tty --interactive docker.io/martinlaurent/ascli:latest ``` For more convenience, you may define a shell alias: ```shell alias ascli='podman run --rm --tty --interactive docker.io/martinlaurent/ascli:latest' ``` Then, you can execute the container like a local command: ```shell ascli -v ``` ```text 4.29.0.pre ``` To persist the configuration on the host, specify your user's configuration folder as a volume for the container. To enable write access, a possibility is to run as `root` in the container (and set the default configuration folder to `/home/cliuser/.aspera/ascli`). Add options: ```shell --user root --env ASCLI_HOME=/home/cliuser/.aspera/ascli --volume $HOME/.aspera/ascli:/home/cliuser/.aspera/ascli ``` > [!NOTE] > If you are using a `podman machine`, for example, on macOS, ensure that the folder is also shared between the VM and the host, so that sharing is: > > container → VM → Host: `podman machine init ... --volume="/Users:/Users"` As shown in the quick start, if you prefer to keep a running container with a shell and `ascli` available, you can change the entry point, add option: ```shell --entrypoint bash ``` > [!WARNING] > `ascli` is run inside the container, so transfers are also executed inside the container and do not have access to host storage by default. You may also want files downloaded in the container to be available on the host. For example, files transferred with `ascli` through folder `/xferfiles` (right-hand side) would be available on the host in `$HOME/xferdir`. In this case, you also need to specify the shared transfer folder as a volume: ```shell --volume $HOME/xferdir:/xferfiles ``` To combine all the above options: ```shell alias asclish="podman run --rm --tty --interactive --user root --env ASCLI_HOME=/home/cliuser/.aspera/ascli --volume $HOME/.aspera/ascli:/home/cliuser/.aspera/ascli --volume $HOME/xferdir:/xferfiles --entrypoint bash docker.io/martinlaurent/ascli:latest" ``` ```shell export xferdir=$HOME/xferdir mkdir -p $xferdir chmod -R 777 $xferdir mkdir -p $HOME/.aspera/ascli asclish ``` #### Container: Sample start script A convenience sample script is also provided: download the script [`dascli`](../build/container/dascli) from [the GitHub repository](https://raw.githubusercontent.com/IBM/aspera-cli/main/build/container/dascli). Some environment variables can be set for this script to adapt its behavior: | env var | Description | Default | Example | |----------------|------------------------------------|--------------------------|--------------------------| | `ASCLI_HOME` | Configuration folder (persistency) | `$HOME/.aspera/ascli` | `$HOME/.ascli_config` | | `docker_args` | Additional options to `podman` | <empty> | `--volume /Users:/Users` | | `image` | Container image name | `docker.io/martinlaurent/ascli` | n/a | | `version` | Container image version | Latest | `4.8.0.pre` | The wrapping script maps the folder `$ASCLI_HOME` on the host to `/home/cliuser/.aspera/ascli` in the container (the configuration folder expected in the container). This keeps the configuration persistent on the host. To add local storage as a volume, you can use the env var `docker_args`: Example of use: ```shell curl -o ascli https://raw.githubusercontent.com/IBM/aspera-cli/main/build/container/dascli chmod a+x ascli export xferdir=$HOME/xferdir mkdir -p $xferdir chmod -R 777 $xferdir export docker_args="--volume $xferdir:/xferfiles" ./ascli config initdemo echo 'Local file to transfer' > $xferdir/samplefile.txt ./ascli server upload /xferfiles/samplefile.txt --to-folder=/Upload ``` > [!NOTE] > The local file path is the one seen from the container (`/xferfiles/samplefile.txt`), where `/xferfiles` is mapped to the host folder `$HOME/xferdir`. > [!WARNING] > Do not use too many volumes, as the legacy `aufs` driver limits their number. > Prefer the `overlay2` driver. #### Container: Installing in an air-gapped environment - First create the image archive: ```shell podman pull docker.io/martinlaurent/ascli podman save docker.io/martinlaurent/ascli|gzip>ascli_image_latest.tar.gz ``` - Then, on the air-gapped system: ```shell podman load -i ascli_image_latest.tar.gz ``` #### Container: `aspera.conf` `ascp`'s configuration file `aspera.conf` is located in the container at: `/ibm_aspera/aspera.conf` (see Dockerfile). As the container is immutable, modifying this file is not recommended. To change the configuration, tell `ascp` to use another file using `ascp` option `-f`, for example, by locating it on the host folder `$HOME/.aspera/ascli` mapped to the container folder `/home/cliuser/.aspera/ascli`: ```shell echo '' > $HOME/.aspera/ascli/aspera.conf ``` Then, tell `ascp` to use that other configuration file: ```shell --transfer=@json:'{"ascp_args":["-f","/home/cliuser/.aspera/ascli/aspera.conf"]}' ``` #### Container: Singularity Singularity is another container runtime. On Linux install: ```shell dnf install singularity-ce ``` Build an image like this: ```shell singularity build ascli.sif docker://docker.io/martinlaurent/ascli ``` Then, start `ascli` like this: ```shell singularity run ascli.sif ``` Or get a shell with access to `ascli` like this: ```shell singularity shell ascli.sif ``` ### Configuring the SSL library `ascli` uses the Ruby `openssl` gem which uses by default the system's `openssl` library and its CA certificate bundle. To display the version of **OpenSSL** used in `ascli`: ```shell ascli config echo @ruby:OpenSSL::OPENSSL_VERSION --format=text ``` To use another SSL library or version, execute: ```shell gem install openssl -- --with-openssl-dir=[openssl library folder] ``` Where `[openssl library folder]` is the path to the folder containing the `lib` and `include` folders of the `openssl` library. For example, on macOS, to use the `openssl@3` library installed with `brew`: ```shell openssl version -e|sed -n 's|ENGINESDIR: "\(.*\)/lib[^/]*/.*|\1|p' ``` ```text /opt/homebrew/Cellar/openssl@3/3.3.0 ``` Then install the `openssl` gem with: ```shell gem install openssl -- --with-openssl-dir=$(openssl version -e|sed -n 's|ENGINESDIR: "\(.*\)/lib[^/]*/.*|\1|p') ``` ### Configuring the SSL CA certificate bundle SSL certificates are validated using a certificate store. By default, it is the one of the system's `openssl` library. To display the default trusted certificate store locations: ```shell ascli config echo '@ruby:[OpenSSL::X509::DEFAULT_CERT_DIR,OpenSSL::X509::DEFAULT_CERT_FILE]' ``` Certificates are checked against the [Ruby default certificate store](https://ruby-doc.org/stdlib-3.0.3/libdoc/openssl/rdoc/OpenSSL/X509/Store.html) `OpenSSL::X509::DEFAULT_CERT_FILE` and `OpenSSL::X509::DEFAULT_CERT_DIR`, which are typically the ones of `openssl` on Unix-like systems (Linux, macOS, and so on). Ruby's default values can be overridden using env vars: `SSL_CERT_FILE` and `SSL_CERT_DIR`. If neither these env vars are set nor the default locations exist (for example, a single executable built on another Linux distribution), `ascli` sets `SSL_CERT_FILE` to the first system CA bundle found: `/etc/ssl/certs/ca-certificates.crt`, `/etc/pki/tls/certs/ca-bundle.crt`, `/etc/ssl/ca-bundle.pem`, `/etc/ssl/cert.pem`. To get certificate validation, the CA certificate bundle must be up-to-date. Check this repository on how to update the system's CA certificate bundle: [https://github.com/millermatt/osca](https://github.com/millermatt/osca). For example, on RHEL/Rocky Linux: ```shell dnf install -y ca-certificates update-ca-trust extract ``` The SSL CA certificate bundle can be specified using the `cert_stores` option, which accepts a list of files or directories. By default, Ruby's system certificate store is used. When `cert_stores` is provided: - It overrides the default locations, which can still be included explicitly using the special value `DEF`. - The option accepts either a `String` (single path) or an `Array` (multiple paths). - Each use of the option appends to the list of search paths incrementally. - If a directory is specified, all files within that directory are automatically included. > [!NOTE] > JRuby uses its own implementation and CA bundles. For example, on Linux, to force the use of the system's certificate store: ```shell --cert-stores=$(openssl version -d|cut -f2 -d'"')/cert.pem ``` `ascp` also needs to validate certificates when using **WSS** for transfer TCP part (instead of **SSH**). By default, `ascp` uses a hard-coded root location `OPENSSLDIR`. Original `ascp`'s hard-coded locations can be found using: ```shell ascli config ascp info --fields=openssldir ``` For example, on macOS: `/Library/Aspera/ssl`. Then trusted certificates are taken from `[OPENSSLDIR]/cert.pem` and files in `[OPENSSLDIR]/certs`. `ascli` overrides the default hard-coded location used by `ascp` for WSS and uses the same locations as specified in `cert_stores` (using the `-i` option of `ascp`). To update trusted root certificates for `ascli`, update the system's root certificate store (see the locations displayed above). An up-to-date version of the certificate bundle can also be retrieved with: ```shell ascli config echo @uri:https://curl.se/ca/cacert.pem --format=text ``` To download that certificate store: ```shell ascli config echo @uri:https://curl.se/ca/cacert.pem --format=text --out.file=/tmp/cacert.pem ``` Then, use this store by setting the option `cert_stores` (or env var `SSL_CERT_FILE`). To trust a specific certificate (for example, self-signed), **provided that the `CN` is correct**, save the certificate chain to a file: ```shell ascli config remote_certificate chain https://localhost:9092 --insecure=yes --out.file=.pem ``` > [!NOTE] > Use command `name` to display the remote common name of the remote certificate. Then, use this file as certificate store (for example, here, Node API): ```shell ascli config echo @uri:https://localhost:9092/ping --cert-stores=.pem ``` ## Command Line Interface The command line tool is: `ascli` The `aspera-cli` gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs): - IBM Aspera High Speed Transfer Server (FASP and Node) - IBM Aspera on Cloud (including ATS) - IBM Aspera Faspex - IBM Aspera Shares - IBM Aspera Console - IBM Aspera Orchestrator - And more... `ascli` provides the following features: - Commands to Aspera server products (on-premise and SaaS) - Any command line **options** (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files, ... - Commands, Options, and Option values shortcuts - FASP [Transfer Agents](#transfer-clients-agents) can be: local `ascp`, or Connect Client, or any transfer node - Transfer parameters can be altered by modification of [**transfer-spec**](#transfer-specification), this includes requiring multi-session - Allows transfers from products to products, essentially at node level (using the node transfer agent) - FASPStream creation (using Node API) - **Watchfolder** creation (using Node API) - Additional command plugins can be written by the user - Download of Faspex and Aspera on Cloud "external" links - **Legacy** SSH based FASP transfers and remote commands (`ascmd`) Basic usage is displayed by executing: ```shell ascli -h ``` See [Usage](#usage). > [!NOTE] > Not all features of `ascli` are documented here: explore commands on the command line, using `-h`. ### Command Line Arguments Command line arguments are the units of command line typically separated by spaces (the `argv` of C). `ascli` handles the following types of command line arguments: - [**Options**](#options): absolute position is not important, but relative order is, as a given option may be provided several times - [**Plugins**](#plugins) for example, `config`, on first position - [**Resource Types**](#resource-types) for example, `users` - [**Verbs**](#verbs) for example, `create`, to act on those resources or plugins. - [**Identifiers**](#identifiers) for example, `123`, uniquely identifying a resource - [**Command Parameters**](#command-parameters) arguments, for example, creation data, can be an [Extended Value](#extended-value-syntax). Command line arguments that are not options are referred to as **Positional Arguments**. For example: ```shell ascli plugin command verb --option-name=VAL1 VAL2 ``` - **Positional Arguments**: `plugin command verb VAL2` - One **Option**: `option_name` and its **value**: `VAL1` Enumeration values (positional arguments and option names/values) support prefix matching. See [Enumerations](#enumerations) for details. For example `ascli config pre ov --for=c` is the same as `ascli config preset overview --format=csv`. The value of **Options** and **Command Parameters** is evaluated with the [Extended Value](#extended-value-syntax) syntax. #### Positional Arguments When options (beginning with `--`) are removed from the command line, the remaining arguments are **Positional Arguments** with a significant, pre-defined order. **Plugins**, **Resource Types** and **Verbs** are [Enumerations](#enumerations). **Identifiers** and **Command Parameters** are open values, but a given type is expected, either a simple `String` or an [Extended Value](#extended-value-syntax). The general structure of positional arguments is: ```text ascli [] [] ``` If a sub-resource is used: ```text ascli [1] [] [] ``` Example: ```shell ascli config ascp info ``` - `ascli` is the executable executed by the shell - `config` is the name of the plugin to be used - `ascp` is the name of the resource (singleton, so no identifier is provided) - `info` is the verb to be performed (action on selected resource) #### Resource types They identify a type of resource accessible in a given context. A resource type can be a singleton, or have multiple instances (identified by their identifier). A plugin is a kind of resource type singleton. A resource type can also be a grouping of other resource types, for example `admin` or `files`, like a singleton. #### Verbs Standard resource **Verbs** are: `create`, `show`, `list`, `modify`, `delete`. Some entities also support additional verbs. When such additional commands relate to a resource also accessible in another context, they are placed under the `do` command. Such subcommands appear after the resource identifier, for example, in `ascli aoc admin node do browse /`, `browse` is a subcommand of `node`. Typically, the `create` verb takes resource creation data as a parameter. `show`, `modify` and `delete` take an identifier, unless manipulating a singleton. `list` typically uses the `query`, `select` options. `list` and `show` typically use the `fields` option. #### Identifiers Identifiers uniquely identify a resource. They are typically located immediately after a verb like `show`, `modify` or `delete`, itself placed after the resource type, for example: `user show foobar`. Some resources can also be selected by a unique field other than the native identifier (typically: `id`), using the [**percent selector**](#percent-selector). ##### Percent selector The **percent selector** is a special syntax that lets you identify a resource by any unique field, instead of its default identifier (typically a numeric `id`). Syntax: `%:` - The leading `%` marks the argument as a percent selector (not a plain identifier). - `` is the name of the field to match on (e.g. `name`, `email`). - `` is the expected value of that field. The tool then looks up the resource whose `` equals `` and uses its native identifier internally. For example, to show the user whose `name` field is `john`: ```shell ascli aoc admin user show %name:john ``` #### Command Parameters **Command Parameters** are typically mandatory values for a command, such as resource creation data. > [!NOTE] > A Command Parameter could also have been designed as an option. > However, since it is mandatory and such data typically do not need to be stored in a configuration file, using a Command Parameter is the better design choice. > The advantages over an option are: a shorter command line (no option name, only position), a clearly mandatory value, and a position that unambiguously indicates its role. > [Extended Value](#extended-value-syntax) syntax is supported, so it is possible to retrieve a value from the configuration file (using `@preset:`) or an environment variable (using `@env:`). If a **Command Parameter** begins with `-`, then either use the `@val:` syntax (see [Extended Value](#extended-value-syntax)), or use the `--` separator (see below). A few **Command Parameters** are optional: they are always located at the end of the command line. #### Enumerations **Enumerations** are arguments that must match a value from a predefined list in a given context. The following are enumerations: - **Plugins**, **Resource Types**, and **Verbs** (positional arguments) - Option names (for example, `--log-level`) - Some option values (for example, `debug` for `--log-level`) **Prefix matching**: You can use a unique prefix instead of the full value, provided it uniquely identifies the value in that context. Examples: - Positional: `ascli config pre ov --for=c` is the same as `ascli config preset overview --format=csv` - Option name: `--log-l=debug` is the same as `--log-level=debug` - Option value: `--format=c` is the same as `--format=csv` > [!NOTE] > While prefix matching works for option names, using full names is recommended for clarity. **Error handling**: If an invalid or missing enumeration value is provided, an error message displays the list of valid values for that context. #### Dot-path Notation `ascli` uses a unified [dot-path](#dot-path-notation) notation in several places on the command line and in output. Understanding this concept once makes it immediately usable everywhere it appears. [Dot-path](#dot-path-notation) notation is used in four distinct places in `ascli`: | Surface | Syntax | in/out | read/write | |----------------|-----------------------|--------|------------| | [Option](#options) names | `--opt.key.sub=value` | Input | write | | [Positional arguments](#positional-arguments) | `@: key.sub=value ... [END]` | Input | write | | [Extended value](#extended-value-syntax) `@preset:` | `@preset:key.sub` | Input | read | | [Output field names](#filtering-results) | `--out.fields=key.sub` | Output | read | A [dot-path](#dot-path-notation) is a `String` where segments are separated by `.` (dot), each segment designating a key in a nested structure: - A **string segment** designates a key in a `Hash` (associative array). - An **integer segment** designates an index in an `Array`. For example, the path `a.b.0` means: key `a`, then key `b`, then the first element of an array. When a **value** is assigned to the path (**write** with `=`), it is automatically converted to the simplest matching type: `Boolean`, `Integer`, `Float`, or `String`. Values `true` and `yes` are converted to `Boolean` `true`, and values `false` and `no` to `false`. For an option expecting a list of values that includes `yes` or `no` (e.g. `--out.table.pivot=no`), the `Boolean` is converted back to that value. > [!NOTE] > A value of `1` will be automatically converted to an `Integer`. > When a specific type is required for the value, the [Extended Value](#extended-value-syntax) syntax modifiers `@json:` or `@ruby:` can be used. > For example: `--opt.x=1` generates `{"x": 1}`. > To get a `String`: `--opt.x=@json:\"1\"` or `--opt.x=@ruby:%q{1}`. > Likewise, to get the `String` `yes`: `--opt.x=@json:\"yes\"`. Example: [dot-path](#dot-path-notation) to JSON output ```shell ascli config echo @: a.b=1 a.c=2 a.d.0=hello a.d.1=world --format=json ``` ```json {"a":{"b":1,"c":2,"d":["hello","world"]}} ``` Example: JSON to [dot-path](#dot-path-notation) output ```shell ascli config echo @json:'{"a":{"b":1,"c":2,"d":["hello","world"]}}' ``` ```text ╭───────┬───────╮ │ field │ value │ ╞═══════╪═══════╡ │ a.b │ 1 │ │ a.c │ 2 │ │ a.d │ hello │ │ │ world │ ╰───────┴───────╯ ``` ##### Positional Arguments with Dot-path The `@:` syntax is a specialized argument type used for **in-place definition** of nested data structures. It allows you to construct complex parameters (`Hash`es and `Array`s) directly from the command line, serving as a readable alternative to providing a single, serialized JSON string. **Usage and Syntax**: The general syntax for this argument is: ```text @:[] = [=] ... [] ``` - `@:[]`: A standalone argument that starts collecting [dot-path](#dot-path-notation) assignments into a single data structure. The optional text after `:` sets the end marker. If it is empty, the marker is `END`. - `=`: An assignment using the standard [dot-path](#dot-path-notation) notation. Multiple assignments can be provided in sequence to build a complex object. The value is converted like any [dot-path](#dot-path-notation) value, and can be an [Extended Value](#extended-value-syntax), for example `public_key=@file:~/.ssh/id_rsa.pub`. Options placed among the assignments are not collected: they are processed as usual, for example `@: a=1 --format=json b=2`. - `` (`END` by default): An optional argument that ends the collection. - Without the marker: **All** remaining positional arguments are collected, and each one must contain `=`. - With the marker: **Only** arguments between `@:` and the marker are used for the structure. Arguments after the marker are passed to the command as normal positional arguments. - Example with a custom marker: `@:STOP a=1 STOP`. > [!IMPORTANT] > Use `END` whenever any positional argument must follow the object built with `@:` (for example, a file list, or any other subsequent positional parameter). > Without `END`, those arguments are collected as [dot-path](#dot-path-notation) assignments instead of being passed to the command: an argument without `=` causes an error, and an argument containing `=` is silently added to the structure. **Example**: Sending a package with a file list using `@:` for package information. ```shell ascli aoc packages send @: name="" recipients.0=user@example.com END file1.dat file2.dat ``` > [!CAUTION] > In the above example, removing `END` causes an error, because `file1.dat` is read as a [dot-path](#dot-path-notation) assignment and does not contain `=`. > [!NOTE] > `@:` can also be used as an option value (for example, `--query=@: a=b`). > In that case, only positional arguments **after the option's position** in the original command line are consumed, so sub-commands before the option are not affected. > Use `END` as usual to stop collection when further positional arguments must follow: `ascli aoc tier_restrictions --query=@: a=b END other_arg`. #### Options Command-line options, such as `--log-level=debug`, follow these conventions: - **Prefix**: All options begin with `--`. - **Naming**: Option names on command line use lowercase letters and hyphens (`-`) as word separators. Option names in the configuration file use underscores (`_`) as word separators. Example: `--log-level=debug` is `log_level` in config file. - **Values**: An option's value is assigned using `=` (for example, `--log-level=debug`). Alternatively, the value can be provided as a separate, space-separated token (for example, `--log-level debug`). Both forms are equivalent. - **Prefix Usage**: Options support prefix matching (see [Enumerations](#enumerations)), though full names are recommended. - **Optionality**: Most options are optional; they either have a default value or do not require one. - **Order**: Options can appear in any position on the command line, but their order may affect processing. Exceptions and Special Cases: - **Short Forms** Some options have short forms. For example, `-Ptoto` and `-P toto` are both equivalent to `--preset=toto` and `--preset toto`. See the manual or `-h` for details. - **Flags** Certain options are flags and do not require a value (for example, `-N`). - **Option Terminator** The special option `--` ends option parsing. All subsequent arguments, including those starting with `-`, are treated as positional arguments. - [**dot-path notation**](#dot-path-notation) - **Cumulative Hashes** When an option of type `Hash` is set, the value is deep-merged with any existing value. Setting to `@json:{}` starts from an empty hash (the existing value is discarded for that merge step). Setting to `@none:` explicitly disables the option: it is set to `nil` and any preset or default value is prevented from restoring it. This is useful, for example, to disable a vault or any Hash option that is configured in a default preset. Example: ```shell ascli config echo -- --sample ``` ```text --sample ``` > [!NOTE] > Here, `--sample` is taken as an argument, and not as an option, due to `--`. Options may have a (hard coded) default value. Options can be placed anywhere on command line and are evaluated in order. Usually the last value evaluated overrides previous values, but some options are cumulative, for example, `ts`. Options are typically optional: to change the default behavior. But some are mandatory, so they can be placed in a configuration file, for example: connection information. The value for **any** options can come from the following locations (in this order, last value evaluated overrides previous value): - [Configuration file](#configuration-file) - Environment variable - Command line Environment variables starting with prefix ASCLI_ are taken as option values, for example, `ASCLI_OPTION_NAME` is for `--option-name`. Option `show_config` dry runs the configuration, and then returns currently set values for options. `ascli --show-config` outputs global options only, and `ascli [plugin] --show-config` outputs global and plugin default options. In addition, option `--show-config` can be added at the end of any full command line, this displays the options that would be used for the command. A command line argument is typically designed as option if: - It is optional, or - It is a mandatory parameter with a default value that would benefit from being set persistently (that is, in a configuration file or environment variable, for example, URL and credentials). ### Interactive Input Some options and **Command Parameters** are mandatory and others are optional. By default, `ascli` raises an error for missing mandatory options or **Command Parameters**. The behavior can be controlled with: - `--interactive=<yes|no>` (default=no) - `yes`: Missing mandatory parameters/arguments are asked to the user. - `no`: Missing mandatory parameters/arguments raise an error message. - `--ask-options=<yes|no>` (default=no) - `yes`: Optional parameters/arguments are asked to the user. ### Output Command execution will result in output (terminal, stdout/stderr). The information displayed depends on the action. To redirect results to a file, use option `--out.file`. #### Types of output data Depending on action, the output will contain: | Result Type | Description | |-----------------|-----------------------------------------------------------------------------------| | `single_object` | Displayed as a two-dimensional table: one line per field, first column is field name, and second is field value. Nested hashes are collapsed. | | `object_list` | Displayed as a two-dimensional table: one line per item, one column per field. | | `value_list` | A table with one column. | | `empty` | nothing | | `status` | A message. | | `other_struct` | A complex structure that cannot be displayed as an array. | With a structured format (`json`, `jsonpp`, `yaml`, `ruby`), the output is always valid in this format: a status is a string, an empty list is `[]`, and no result is `null`. #### Enhanced display of special values Special values are highlighted as follows in `format=table`: | Value | Display | |----------------|------------------| | `nil` | `<null>` | | empty `String` | `<empty string>` | | empty `Array` | `<empty list>` | | empty `Hash` | `<empty dict>` | Example: ```shell ascli config echo @json:'{"ni":null,"es":"","ea":[],"eh":{}}' ``` ```text ╭───────┬────────────────╮ │ field │ value │ ╞═══════╪════════════════╡ │ ni │ <null> │ │ es │ <empty string> │ │ ea │ <empty list> │ │ eh │ <empty dict> │ ╰───────┴────────────────╯ ``` #### Filtering results The result of a command can be filtered on the server side, by the API, and then on the client side, by `ascli`. Filtering on the server side reduces the data transferred, while filtering on the client side accepts any condition. Options are evaluated in this order: ![Order of evaluation of the options that filter results](filtering-order.svg) 1. `--query`: parameters sent to the API. 2. `--out.secrets`: secrets are hidden first, so that an expression cannot reveal them. 3. `--out.query`: JMESPath expression or Ruby lambda, applied to the data as returned by the API. 4. `--out.select`: selection of items (rows). 5. `--out.fields`: selection of fields (columns). 6. Display in the format set by `--out.format`. | To | Use | |------------------------------------------|------------------------------------------| | Limit what the API returns | `--query`, with parameters `max` and `pmax` | | Filter on any condition, reshape, count | `--out.query` | | Select items by the value of fields | `--out.select` | | Choose the fields displayed | `--out.fields` | ##### Option: `--query`: Parameters of the API The `query` option can generally be used to add URL parameters to commands that list resources. It takes a `Hash`, corresponding to key/value pairs that appear in the query part of the request. For example: `--query=@json:'{"p1":"v1","p2":"v2"}'` leads to query: `?p1=v1&p2=v2`. If the same parameter needs to be provided several times, then it is also possible to provide an `Array`. For example: `--query=@json:'{"p":["v1","v2"]}'` leads to query: `?p=v1&p=v2`. If PHP-style arrays are expected in the API, add `[]` to the name of the parameter. For example: `--query=@json:'{"p[]":["v1","v2"]}'` leads to query: `?p[]=v1&p[]=v2`. Commands that read a list by pages (for example, `aoc admin user list`) also accept parameters `max` and `pmax`. They are processed by `ascli`, not sent to the API: they limit the number of items (`max`) and of pages (`pmax`) requested. For example: `--query.max=10` reads the first 10 items. > [!TIP] > Use `--query=help` on any `list` command to display the available query parameters and their types, as documented in the API. > For example: `ascli faspex5 admin accounts list --query=help` ##### Option: `--out.query`: JMESPath expression Option `--out.query` applies a [JMESPath](https://jmespath.org) expression to the result of a command, when it is an object or a list. JMESPath is a standard query language for JSON, also used by option `--query` of the AWS and Azure CLIs. The expression is applied to the data as returned by the API (secrets hidden), before options `--out.select` and `--out.fields`. Its syntax is checked before the command is executed. Results of other types (status, no result) are not changed. Examples: ```shell # Name and email of ATS administrators ascli aoc admin user list --out.query='[?ats_admin].{name: name, email: email}' # Number of users ascli aoc admin user list --out.query='length(@)' # Email of users, one per line, for use in a shell loop ascli aoc admin user list --out.query='[].email' --format=text ``` > [!TIP] > Enclose the expression in single quotes for the shell. > In JMESPath, literal values are enclosed in backticks, for example: ``[?size > `1000`]``. > An expression with string literals in single quotes (`'...'`) is enclosed in double quotes, for example: `"[?name=='Ann']"`. For processing that JMESPath does not provide (regular expressions, string manipulation, grouping), the value can be a Ruby lambda (`@ruby:`): it is called with the data, and returns the new data, converted to JSON data (for example, symbols and dates become strings). It is refused at [Security Level](#security-level) `medium` and `high`. ```shell # Number of users per email domain ascli aoc admin user list --out.query=@ruby:'->(users){users.group_by{|u|u["email"].split("@").last}.transform_values(&:count)}' ``` The result of the query is displayed according to its type: a list of objects as a table, a list of values with one value per line, a single value as is. A list of lists (for example, `[].[name, email]`) is displayed only with a structured format (`json`, `yaml`, ...): to display a table, or CSV, use a multi-select hash instead, for example, `[].{name: name, email: email}`. > [!NOTE] > Option `--query` gives filtering parameters to the API, while option `--out.query` filters and reshapes the result received. > Option `--out.query` is the equivalent of option `--query` of the AWS and Azure CLIs. ##### Option: `--out.select`: Selection of items A list of objects (`object_list`) can be filtered with option `--out.select` (or `--select`). Its value is either a `Hash` or a `Proc`: - `Hash`: an item is selected if each key of the `Hash` is a field of the item with the same value, and the same type: `true` (`Boolean`) is not `"true"` (`String`). - `Proc`: a Ruby lambda called with each item (`Hash`), that returns `true` to select it, or `false` to remove it. Example: ```shell ascli aoc admin user list --out.fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --out.select=@json:'{"ats_admin":true}' ``` ```text ╭────────────────┬─────────────────────┬───────────╮ │ name │ email │ ats_admin │ ╞════════════════╪═════════════════════╪═══════════╡ │ John Curtis │ john@example.com │ true │ │ Laurent Martin │ laurent@example.com │ true │ ╰────────────────┴─────────────────────┴───────────╯ ``` In the above example, the same result is obtained with option: ```shell --out.select=@ruby:'->(i){i["ats_admin"]}' ``` A key can be a [dot-path](#dot-path-notation), for example, `directory_user.type`, with all formats. ##### Option: `--out.fields`: Selection of fields Depending on the command, results may include by default all fields, or only some selected fields. It is possible to define specific columns to be displayed, by setting option `--out.fields` (or `--fields`). The `--out.fields` option is a list that can be either a comma-separated list or an [Extended Value](#extended-value-syntax) `Array`. Individual elements of the list can be: - **property**: add property to the current list - `-`**property**: remove property from the current list - `DEF`: default list of fields (that is the default, when not set) - `ALL`: all fields - A Ruby `RegEx`: using `@ruby:'/.../'`, or `@re:...` add those matching to the list To display a **property** inside a complex structure (`Hash`, `Array`), use [dot-path](#dot-path-notation) notation, only with `--format=table` or `csv` (see option `--out.flat`). With other formats, use a JMESPath multi-select hash with option `--out.query`, for example: `--out.query='[].{name: name, type: directory_user.type}'`. Examples: - `a,b,c`: the list of attributes specified as a comma-separated list (overrides the default) - `@list:,a,b,c`: `Array` [Extended Value](#extended-value-syntax): same as above - `@json:'["a","b","c"]'`: `Array` [Extended Value](#extended-value-syntax): same as above - `b,DEF,-a`: default property list, remove `a` and add `b` in first position - `@ruby:'/^server/'`: Display all fields whose name begins with `server` ##### `--out.select` or `--out.query` Option `--out.select` is shorter for a test of equality, while option `--out.query` accepts any condition. | Condition | `--out.select` | `--out.query` | |--------------------------|----------------------------------------------------|--------------------------------------------------| | Equal to a `Boolean` | `@json:'{"ats_admin":true}'` | `'[?ats_admin]'` | | Equal to a `String` | `@json:'{"name":"Ann"}'` | `"[?name=='Ann']"` | | Field of a nested object | `@json:'{"directory_user.type":"active_directory"}'` | `"[?directory_user.type=='active_directory']"` | | Several conditions (and) | `@json:'{"ats_admin":true,"name":"Ann"}'` | `"[?ats_admin && name=='Ann']"` | | Negation, or | Ruby lambda (`@ruby:`) | `'[?!ats_admin]'`, or: see [Recipes](#recipes) | | Number comparison | Ruby lambda (`@ruby:`) | ``'[?size > `1000000`]'`` | | Substring | Ruby lambda (`@ruby:`) | ``'[?contains(email, `@example`)]'`` | | Choice of fields | With option `--out.fields` | `'[].{name: name, email: email}'` | | Count | Not possible | `'length(@)'` | > [!WARNING] > > - Values are compared with their type. With `--out.select`, `"true"` (`String`) does not match `true` (`Boolean`). In JMESPath, ``[?ats_admin == `true`]`` matches, but `[?ats_admin == 'true']` does not: `'...'` is a string literal. > - After `--out.query`, the default fields of the command no longer apply: all the fields of the result are displayed. Choose fields in the expression (multi-select hash), or with option `--out.fields`. > - Secrets are hidden before `--out.query` is applied, so that the expression cannot reveal them. ##### Recipes ```shell # Users of Active Directory, with user name and email ascli shares admin user all list --out.query="[?directory_user.type=='active_directory'].{username: username, email: email}" # Users without directory (field missing or null) ascli shares admin user all list --out.query='[?directory_user == null].username' # Number of ATS administrators ascli aoc admin user list --out.query='length([?ats_admin])' # Users who are ATS administrators, or have no email ascli aoc admin user list --out.query='[?ats_admin || !email].name' # Emails of a domain, one per line ascli aoc admin user list --out.query='[?ends_with(email, `@example.com`)].email' --format=text # Largest file of a folder ascli node browse / --out.query='max_by([?type==`file`], &size).path' # Files larger than 1 MB ascli node browse / --out.query='[?size > `1000000`].{path: path, size: size}' # Files modified since a date ascli node browse / --out.query='[?mtime >= `"2025-06-01"`].path' ``` > [!NOTE] > The JMESPath implementation of `ascli` (gem `jmespath`) also compares strings with `<`, `<=`, `>` and `>=`, which the JMESPath specification defines only for numbers. > Dates in ISO 8601 format, in the same time zone, are compared in chronological order. The result in JSON (`--format=json`) can also be processed by other tools, such as [`jq`](https://jqlang.org), whose language differs from JMESPath: for example, `jq 'map(select(.directory_user.type == "active_directory"))'`. ##### Other filters Some commands provide their own filters: - [Percent selector](#percent-selector) `%<field>:<value>`: selects a single resource by the value of one of its fields. - `find` of plugin `node` (gen4) and `aoc files find`: recursive search of files on the client side, with a glob or a Ruby lambda (see [Operation `find`](#operation-find-on-gen4access-key)). - `faspex5 packages list`: a Ruby lambda as last argument (see [Faspex 5: List packages](#faspex-5-list-packages)). - `console transfer`: query parameter `filter` (see [Transfer filter](#transfer-filter)). - `preview`: options `--filter` and `--skip-folders` (see [Candidate detection](#candidate-detection-for-creation-or-update-or-deletion)). #### Option: `--out.format` The style of output can be set using option `--out.format` (or `--format`): | `format` | Output formatting | | -------- | -------- | | `table` | Text table (default) | | `text` | Value as `String` | | `ruby` | Ruby code | | `json` | JSON code | | `jsonpp` | JSON pretty printed | | `yaml` | YAML | | `csv` | Comma Separated Values | | `image` | URL or data for a [picture/video](#image-and-video-thumbnails) | | `nagios` | Suitable for Nagios | By default, results of type `single_object` and `object_list` are displayed using format `table`. #### Option: `--out.table` The way `format`: `table` and `csv` are generated can be customized with option `--out.table` which expects a `Hash`. For `format=table`, options are the ones described in gem [`terminal-table`](https://github.com/tj/terminal-table). For example, to display a table with thick Unicode borders: ```shell ascli config preset overview --out.table=@ruby:'{border: :unicode_thick_edge}' ``` > [!NOTE] > Other border styles exist, not limited to: `:unicode`, `:unicode_round`. By default, if the terminal supports Unicode (see [`--out.utf8`](#terminal-rendering-colors-and-utf-8)), then `border=unicode_round` is used. A special parameter is defined: `str_lst_sep` (`String`), default is `\n`. It defines how lists of strings are displayed. Alternatively, set to `,`. When the output is a terminal, the table is fitted to the terminal width: columns with text (e.g. `description`) are wrapped, the widest first. If not enough, other columns (names, identifiers) are wrapped too, preferably after a punctuation (`-`, `_`, `.`, `/`). If the table cannot fit, columns are narrowed to their minimum width (20 characters for text, 10 for others, or heading). Parameter `width` (`Integer`) sets the maximum table width (e.g. `--out.table.width=100`), also when the output is not a terminal. For `format=csv`, options are described in gem [`csv`](https://ruby.github.io/csv/CSV.html#class-CSV-label-Options+for+Generating). For example, to display a CSV with headers and quotes: ```shell ascli config echo @json:'[{"name":"foo","id":1},{"name":"bar","id":8}]' --format=csv --out.table=@json:'{"headers":true,"force_quotes":true}' ``` #### Option: `--out.flat`: Single level `Hash` This option controls how object fields are displayed for complex objects. Effective only when `format` is `table` or `csv` to display `single_object` or `object_list`. If value is `no`, then object's `field` names are only the first level keys of the `Hash` result and values that are `Hash` are displayed as such in Ruby syntax. If value is `yes` (default), then objects are "flattened" using [dot-path](#dot-path-notation) notation with a variation: - Final arrays are displayed as a list of values, one per line (see `str_lst_sep` in `--out.table` above) - `Array` of `Hash` with only `name` keys are displayed as comma separated list of values - `Array` of `Hash` with only `name` and `value` keys are displayed like a `Hash` with value of `name` as key. Example: The result of the command is a single object: ```shell ascli config echo @json:'{"A":"a","B":[{"name":"B1","value":"b1"},{"name":"B2","value":"b2"}],"C":[{"C1":"c1"},{"C2":"c2"}],"D":{"D1":"d1","D2":"d2"}}' ``` ```text ╭────────┬───────╮ │ field │ value │ ╞════════╪═══════╡ │ A │ a │ │ B.B1 │ b1 │ │ B.B2 │ b2 │ │ C.0.C1 │ c1 │ │ C.1.C2 │ c2 │ │ D.D1 │ d1 │ │ D.D2 │ d2 │ ╰────────┴───────╯ ``` For the same command, adding option `--out.flat=no`: ```text ╭───────┬────────────────────────────────────────────────────────────────────────╮ │ field │ value │ ╞═══════╪════════════════════════════════════════════════════════════════════════╡ │ A │ a │ │ B │ [{"name" => "B1", "value" => "b1"}, {"name" => "B2", "value" => "b2"}] │ │ C │ [{"C1" => "c1"}, {"C2" => "c2"}] │ │ D │ {"D1" => "d1", "D2" => "d2"} │ ╰───────┴────────────────────────────────────────────────────────────────────────╯ ``` #### Option: `--out.table.pivot` This option controls how result fields are displayed as columns or lines, when option `format` is set to `table`. Values are `false` (default), `true` or `single`. On command line, `no` and `yes` can be used as well (converted to `Boolean`, see [dot-path](#dot-path-notation)), but a structured value (e.g. `--out=@json:'{"table":{"pivot":true}}'`) expects a `Boolean`. There are two types of results that are affected by this option: | Result | Description | |-----------------|---------------------------------------------| | `single_object` | A single item with multiple fields. | | `object_list` | A list of items, each with multiple fields. | An item (object) is displayed in one of those 2 ways: | Display | rows | columns | |------------|--------|---------------------| | Simple | Fields | `field` and `value` | | Transposed | Items | Fields | The display of result is as follows: | Result | `false` | `true` | `single` | |-----------------|------------|-------------|-------------------------------------| | `single_object` | Simple | Simple | Simple | | `object_list` | Transposed | Simple<br/>(Multiple objects) | Simple if 1 object.<br/>transposed if 2+ objects. | This parameter can be set as a global default with: ```shell ascli config preset set GLOBAL out.table.pivot single ``` Examples: Simulate a result by executing this command: ```shell ascli config echo @json:'<JSON_VALUE>' --out.table.pivot=<no|yes|single> ``` Example 1: A list of one object ```json [{"user":{"id":1,"name":"toto"},"project":"blash"}] ``` Display with `no` (Transposed): ```text ╭─────────┬───────────┬─────────╮ │ user.id │ user.name │ project │ ╞═════════╪═══════════╪═════════╡ │ 1 │ toto │ blash │ ╰─────────┴───────────┴─────────╯ ``` Display with `yes` and `single` (Simple): ```text ╭───────────┬───────╮ │ field │ value │ ╞═══════════╪═══════╡ │ user.id │ 1 │ │ user.name │ toto │ │ project │ blash │ ╰───────────┴───────╯ ``` Example 2: A list of two objects: ```json [{"id":1,"speed":111},{"id":"2","speed":222}] ``` Display with `no` and `single` (Transposed): ```text ╭────┬───────╮ │ id │ speed │ ╞════╪═══════╡ │ 1 │ 111 │ │ 2 │ 222 │ ╰────┴───────╯ ``` Display with `yes` (multiple Simple): ```text ╭───────┬───────╮ │ field │ value │ ╞═══════╪═══════╡ │ id │ 1 │ │ speed │ 111 │ ╰───────┴───────╯ ╭───────┬───────╮ │ field │ value │ ╞═══════╪═══════╡ │ id │ 2 │ │ speed │ 222 │ ╰───────┴───────╯ ``` #### Option: `--out.level`: Verbosity of output Output messages are categorized into three types: - `info` output contains additional information, such as the number of elements in a table - `data` output contains the actual output of the command (object, or list of objects) - `error` output contains error messages The option `--out.level` controls the level of output: - `info` displays all messages: `info`, `data`, and `error` - `data` displays `data` and `error` messages - `error` displays only error messages #### Option: `--out.secrets`: Hide or show secrets in results - If value is `no` (default), then secrets are redacted from command results: replaced with `🔑`, or `***` if the terminal does not support Unicode (see [`--out.utf8`](#terminal-rendering-colors-and-utf-8)). - If value is `yes`, then secrets are shown in clear in results. #### Terminal rendering: Colors and UTF-8 By default, `ascli` detects the capabilities of the terminal: | Option | Effect when `yes` | Auto-detection (default) | |--|--|--| | `--out.colors` | ANSI colors and styles in results, messages and logs | `no` if [`NO_COLOR`](https://no-color.org) is set or results are written to a file (`--out.file`), else `yes` if `CLICOLOR_FORCE` is set and not `0`, else `yes` if both `stdout` and `stderr` are terminals, `TERM` is not `dumb` and `CLICOLOR` is not `0` | | `--out.utf8` | Unicode characters: table borders, check marks, hidden secrets, symbols in messages, half blocks in images (else ASCII equivalents) | `yes` if `stdout` is a terminal and the locale is UTF-8, or on Windows, the terminal is Windows Terminal, VS Code, Git Bash, ConEmu, or sets `TERM` to `xterm*` | Set either option to `yes` or `no` to override detection, for example to keep colors when piping to `less -R`, or to get plain ASCII output in a terminal: ```shell ascli config preset overview --out.colors=no --out.utf8=no ``` > [!NOTE] > Logs issued before the option is processed (e.g. at startup) use the detected value. ### Exit status The exit code of `ascli` indicates the category of error, so that scripts can react to it: | Code | Description | |------|----------------------------------------------------------------------------| | `0` | Success | | `1` | Other error | | `2` | Command line error: unknown command or option, invalid or missing argument | | `3` | Authentication or authorization failure: SSH, HTTP 401 or 403, OAuth grant | | `4` | Resource not found, or several resources match the identifier, HTTP 404 | | `5` | Transfer failure | | `6` | Network or TLS failure: name resolution, connection, timeout, certificate | | `7` | Other error returned by a remote API | | `130` | Interrupted by user (Ctrl-C) | With format `nagios`, the exit code is the status of the check: `0` (OK), `1` (WARNING), `2` (CRITICAL), and `3` (UNKNOWN) if the command fails, following the [Nagios plugin guidelines](https://nagios-plugins.org/doc/guidelines.html). Example: ```bash ascli aoc files download /report.csv case $? in 0) echo 'Downloaded' ;; 6) echo 'Network failure, retry later' ;; *) echo 'Failed' ;; esac ``` ### Extended Value Syntax An **Extended Value** is a structured value - typically a `Hash` (key/value dictionary) or `Array`, potentially nested - that is passed on the command line wherever a simple string is not sufficient. Most options and arguments are specified by a simple string (for example, `username` or `url`). However, some situations require a complex structure: - An option expects a **structured value** (`Hash`, `Array`) rather than a plain string. - The value is too long or complex to type inline (for example, a PEM private key, a JSON body). - The value must be **read from a file, an environment variable, or a URL** at runtime rather than embedded in the command. - The value needs **type coercion**: for example, an integer `1` instead of the string `"1"`, or a boolean `true`. The **Extended Value** syntax addresses all of these needs with a uniform, composable mechanism. It lets you specify **what the value is** and **how to obtain or decode it**, entirely on the command line. #### Syntax and Decoders The syntax is: ```text <0 or more decoders><some text value or nothing> ``` Decoders act like a function with its parameter on right-hand side and are recognized by the prefix: `@` and suffix `:` ```text @<NAME>: ``` ##### Available Decoders The following decoders are supported: | Decoder | Parameter| Returns | Description | |----------|----------|----------|------------------------------------------------------------------------------------------| | `base64` | `String` | `String` | Decode a base64 encoded string. | | `csvt` | `String` | `Array` | Decode a titled CSV value. | | `env` | `String` | `String` | Read from the named environment variable. For example, `--password=@env:MYPASSVAR` | | `file` | `String` | `String` | Read value from specified file (prefix `~/` is replaced with the user's home folder). For example, `--key=@file:~/.ssh/mykey` | | `json` | `String` | Any | Decode JSON values. Convenient to provide complex structures. | | `lines` | `String` | `Array` | Split a string in multiple lines and return an `Array`. | | `list` | `String` | `Array` | Split a string in multiple items taking first character as separator and return an `Array`. | | `none` | None | Nil | A `null` value. | | `path` | `String` | `String` | Performs path expansion on specified path (prefix `~/` is replaced with the user's home folder). For example, `--config-file=@path:~/sample_config.yml` | | `preset` | `String` | Any | Get value from configuration file using [dot-path](#dot-path-notation) notation: a whole preset (`Hash`) or a single value. | | `extend` | `String` | `String` | Evaluates embedded [Extended Value](#extended-value-syntax) syntax in string. | | `re` | `String` | `Regexp` | Ruby Regular Expression (short for `@ruby:/.../`) | | `ruby` | `String` | Any | Execute specified Ruby code. | | `s` | Any | `String` | Converts argument to `String`. | | `secret` | `String` | `String` | Ask password interactively: on terminal (hides input), or on a web page if option `ui` is `graphical`. Argument is the prompt. | | `stdin` | `String` | `String` | Read from stdin in text mode. Argument: `<empty>`, `bin` or `chomp`. | | `uri` | `String` | `String` | Read value from specified URL. Supported schemes: `http:`, `https:`, `data:`, `file:`. For example, `--fpac=@uri:http://serv/f.pac` or `--key=@uri:file:/path/to/key.pem` | | `val` | `String` | `String` | Prevent decoding by the decoders on the right. For example, `--key=@val:@file:foo` sets the option `key` to value `@file:foo`. | | `vault` | `String` | `String` | Read a secret from the [Secret Vault](#secret-vault): `<label>.<field>`, or a [reference](#vault-secret-references) `//<store>/<path>[?<options>][#<field>]`. For example, `--password=@vault:my_label.password` | | `yaml` | `String` | Any | Decode YAML. | | `zlib` | `String` | `String` | Decompress data using zlib. | | `<empty>`| `String` | Any | The **dot-path** modifier: argument `@:` collects the following positional arguments as `key.subkey=value` assignments and builds a `Hash` or `Array` using [dot-path](#dot-path-notation) notation. Shell-friendly alternative to `@json:` for structured values.<br/>Argument is the end marker: `<empty>` for `END`, or a custom marker (e.g. `@:STOP`). Use the end marker to stop collection when further positional arguments must follow.<br/>See [Positional Arguments with Dot-path](#positional-arguments-with-dot-path) for full syntax. | > [!NOTE] > A few commands support a value of type `Proc` (lambda expression). > For example, the [Extended Value](#extended-value-syntax) `@ruby:'->(i){i["attr"]}'` is a lambda expression that returns the value for key `attr` of the `Hash` parameter named `i`. **Chaining rule**: When multiple decoders are combined, they are applied right-to-left: the rightmost decoder runs first on the literal string, and each decoder to its left receives the output of the one to its right. To display the result of an [Extended Value](#extended-value-syntax), use the `config echo` command. The `extend` decoder is useful to evaluate embedded [Extended Value](#extended-value-syntax) syntax in a string. It expects a `@` to close the embedded [Extended Value](#extended-value-syntax) syntax. Option `parser` allows definition of a default parser when the positional parameter or option expects a `Hash` or `Array`. For example, with `--parser=json`, the parameter `{}` will be parsed as an empty JSON Hash, even without prefix `@json:`. #### Syntax Equivalences There are four main ways to express a structured value (for example, a `Hash` with two keys). They are all equivalent and produce the same result: | Syntax | Example | Best for | |--------|---------|----------| | `@json:` | `@json:'{"name":"alice","age":30}'` | Scripts, API payloads, copy-paste from docs | | `@yaml:` | `@yaml:@stdin:` + here-doc | Multi-line or deeply nested structures | | `@ruby:` | `@ruby:'{"name"=>"alice","age"=>30}'` | Advanced use: lambdas, file reads, logic | | dot-path (`key.subkey=value`) via `@:` | `@: name=alice age=30` | Interactive CLI, simple structures, no quoting | All four examples above produce the same `Hash`: `{"name" => "alice", "age" => 30}`. ```shell ascli config echo @json:'{"name":"alice","age":30}' --format=json ascli config echo @ruby:'{"name"=>"alice","age"=>30}' --format=json ascli config echo @: name=alice age=30 --format=json ``` ```json {"name":"alice","age":30} ``` > [!NOTE] > The dot-path form (`key.subkey=value` assignments via `@:`) does **not** need quotes for simple string values and is the most shell-friendly. > The `@json:` form requires enclosing in single quotes to protect `"` from the shell. > The `@ruby:` form accepts both single and double quotes and supports arbitrary Ruby expressions. > The `@yaml:` form is most convenient when combined with a shell **here-document** (`<<EOF`). ##### Equivalences: Positional Parameter vs Option The same `Hash` value `{"a":1,"b":"two"}` can be expressed differently depending on whether it is supplied as a **positional parameter** or as an **option**. **As a positional parameter** (argument to a command): ```shell # JSON form some_command @json:'{"a":1,"b":"two"}' # Dot-path form (END is optional when no positional argument follows) some_command @: a=1 b=two some_command @: a=1 b=two END ``` **As an option** named `opt`: ```shell # JSON form --opt=@json:'{"a":1,"b":"two"}' # Dot-path form via option name (each key as a separate option flag) --opt.a=1 --opt.b=two # Dot-path form via option value (consumes following positional arguments) --opt=@: a=1 b=two END ``` > [!NOTE] > `--opt.a=1 --opt.b=two` and `--opt=@json:'{"a":1,"b":"two"}'` and `--opt=@: a=1 b=two` all produce the same `Hash` assigned to option `opt`. > The `--opt.key=value` form is the most concise when only a few keys need to be set. > Use `END` with `@:` when further positional arguments follow on the command line; it is optional otherwise. #### Which Syntax to Choose? **When in doubt: start with dot-path** (`--opt.key=value` for options, `@: key=value` for positional arguments). It requires no quoting, auto-converts types, and reads naturally on the command line. Switch to `@json:` when you already have a JSON payload from an API doc or external tool. | Context | Recommended syntax | Reason | |---------|--------------------|--------| | Interactive CLI, simple values | dot-path `key.subkey=value` via `@:` or `--opt.key=value` | No quoting needed, auto-typed, readable | | Shell scripts, API payloads | `@json:` | Standard, portable, easy to copy from API docs | | Multi-line or nested structures in scripts | `@yaml:` + `@stdin:` heredoc | Readable, no escaping | | Dynamic values, lambdas, file reads | `@ruby:` | Full Ruby expressiveness | #### Common Usage Examples Example: Create a `Hash` value - for a small structure, the dot-path `key.subkey=value` form (via `@:`) is simpler; for larger or API-sourced structures, use `@json:`: ```shell ascli config echo @: key1=value1 key2=value2 ascli config echo @json:'{"key1":"value1","key2":"value2"}' ``` Example: read the content of the specified file, then, base64 decode, then unzip: ```shell ascli config echo @zlib:@base64:@file:myfile.dat ``` Example: Create a `Hash` value with one key and the value is read from a file: ```shell ascli config echo @ruby:'{"token_verification_key"=>File.read("mykey.txt")}' ``` Example: read a CSV file and create an `Array` of `Hash` for bulk provisioning: ```shell cat test.csv ``` ```text name,email lolo,laurent@example.com toto,titi@tutu.tata ``` ```shell ascli config echo @csvt:@file:test.csv ``` ```text ╭──────┬─────────────────────╮ │ name │ email │ ╞══════╪═════════════════════╡ │ lolo │ laurent@example.com │ │ toto │ titi@tutu.tata │ ╰──────┴─────────────────────╯ ``` Example: create a `Hash` with values coming from a preset named `config` ```shell ascli config echo @json:@extend:'{"hello":true,"version":"@preset:config.version@"}' ``` ```text ╭─────────┬────────╮ │ field │ value │ ╞═════════╪════════╡ │ hello │ true │ │ version │ 4.21.1 │ ╰─────────┴────────╯ ``` Example: Create a `Hash` from YAML provided as shell **Here document**: ```shell ascli config echo @yaml:@stdin: --format=json<<EOF key1: value1 key2: - item1 - item2 key3: key4: value4 key5: value5 EOF ``` ```json {"key1":"value1","key2":["item1","item2"],"key3":{"key4":"value4","key5":"value5"}} ``` #### Schema Discovery with `help` Any option or **Command Parameter** that expects a `Hash` value accepts the special value `help`. Instead of performing the action, `ascli` displays the schema of the expected `Hash` as a table (field name, type, description), then exits with an error. When a mandatory `Hash` argument is missing, `ascli` automatically hints to use `help`: ```shell ascli aoc packages send ``` ```text ERRR Missing: Missing argument: package (Hash) HINT:Give `help` as argument to retrieve the schema of the missing argument. ``` Following the hint and passing `help` as the argument displays the schema: ```shell ascli aoc packages send help ``` ```text INFO Schema: argument: package (Hash) ╭───────────────────────┬────────┬──────────┬────────────────────────────────────────────────────────────────────────────────────────────╮ │ name │ type │ required │ description │ ╞═══════════════════════╪════════╪══════════╪════════════════════════════════════════════════════════════════════════════════════════════╡ │ bcc_recipients │ array │ false │ <empty string> │ │ bcc_recipients[].id │ string │ true │ The ID of the recipient. │ │ bcc_recipients[].type │ enum │ false │ The entity type of the recipient. │ │ │ │ │ Allowed: user, group │ │ name │ string │ true │ Package name. Required for POST. Optional for PUT. │ │ note │ string │ false │ The sender's message to recipients to include with the package. Maximum characters: 65535. │ │ recipients │ array │ false │ <empty string> │ │ recipients[].id │ string │ true │ The ID of the recipient. │ │ recipients[].type │ enum │ false │ The entity type of the recipient. │ │ │ │ │ Allowed: user, group │ ... ╰───────────────────────┴────────┴──────────┴────────────────────────────────────────────────────────────────────────────────────────────╯ ``` The same applies to options: display the schema of the transfer-spec option `ts`: ```shell ascli --ts=help ``` ```text INFO Schema: option: ts ╭─────────────────────┬─────────┬──────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ name │ type │ required │ description │ ╞═════════════════════╪═════════╪══════════╪════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡ │ apply_local_docroot │ boolean │ false │ Apply local docroot to source paths. │ │ cipher │ enum │ false │ In transit encryption algorithms. │ │ │ │ │ Allowed: none, aes-128, aes-192, aes-256, aes-128-cfb, aes-192-cfb, aes-256-cfb, aes-128-gcm, aes-192-gcm, aes-256-gcm │ │ authentication │ string │ false │ Set to `token` for SSH bypass keys, else password asked if not provided. │ ... ╰─────────────────────┴─────────┴──────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ``` This works for any `Hash` option or positional parameter that has a defined schema. #### Schema Validation The value of an option or **Command Parameter** with a schema defined by `ascli` (e.g. options `ts`, `transfer`, `out`, `http_options`, argument of `orchestrator workflows start`) is validated against it before the action. An invalid value is rejected with the path of the invalid element and the reason: ```shell ascli config echo 1 --ts=@json:'{"direction":"sideways"}' ``` ```text ERRR Argument: Option ts: value at `/direction` is not one of: ["send", "receive"] (use --ts=help for schema) Use option -h to get help. ``` Option values are merged from several sources (presets, command line), so mandatory fields are not checked for options. Request bodies of product APIs (e.g. `create` and `modify` commands) are not validated by `ascli`: the API validates them. #### Testing Extended Value Two complementary commands help verify that a value is parsed as expected: | What to check | Tool | How | |---|---|---| | Value of a **positional parameter** | `config echo` | Pass the expression as the sole argument | | Value of an **option** | `--show-config` | Append `--show-config` to any command line | **Checking a positional parameter with `config echo`**: `config echo` takes exactly **one** argument which can use the [Extended Value](#extended-value-syntax) syntax, and displays its parsed value. Unprocessed command line arguments are shown in the error message. ```shell ascli config echo @json:'{"a":1,"b":"two"}' --format=json ascli config echo @: a=1 b=two --format=json ``` ```json {"a":1,"b":"two"} ``` > [!NOTE] > The value is shown after both shell command-line parsing and `ascli` [Extended Value](#extended-value-syntax) parsing. Example: the shell parses three arguments (`1`, `2`, `3`), but `config echo` only processes the first: ```shell ascli config echo 1 2 3 ``` ```text 1 ERRR Argument: unprocessed values: ["2", "3"] ``` **Checking an option value with `--show-config`**: Adding `--show-config` to any command line performs a dry run and displays the resolved value of all options that would be used, without executing the command. To display a specific option, add `--fields=<option_name>`. Add `--out.flat=no` when the option holds a structured value (`Hash`, `Array`) to display it as-is rather than flattened into dot-path keys: ```shell ascli --opt=@json:'{"a":1,"b":"two"}' some_plugin --show-config --fields=opt --out.flat=no ascli --opt.a=1 --opt.b=two some_plugin --show-config --fields=opt --out.flat=no ``` Both lines above display the same resolved value for option `opt`. In the following examples (using a POSIX shell, such as `bash`), several equivalent commands are provided. For all examples, most special character handling is not specific to `ascli`: It depends on the underlying syntax: shell, JSON, and so on. Depending on the case, a different `format` option is used to display the actual value. For example, in the simple string `Hello World`, the space character is special for the shell, so it must be escaped so that a single value is represented. Double quotes are processed by the shell to create a single string argument. For **POSIX shells**, single quotes can also be used in this case, or protect the special character ` ` (space) with a backslash. ```shell ascli config echo "Hello World" --format=text ascli config echo 'Hello World' --format=text ascli config echo Hello\ World --format=text ``` ```text Hello World ``` ### Security Level On the command line, `ascli` can do anything that the user running it is allowed to do. Option `security` restricts the features that access local resources or execute code, for example, when commands come from another program, such as an MCP client (see [MCP Security](#security)). It is a `Hash` with the following keys: | Key | Type | Default | Description | |-----|-----|-----|-----| | `level` | `string` | `low` | `low`, `medium`, or `high` | | `allowed_paths` | `Array<string>` | - | With level `medium`: files and folders that `@file:` can read | | `writable_paths` | `Array<string>` | - | With levels `medium` and `high`: folders where files can be written | | Level | `@file:` | `@ruby:`, `@uri:`, and `@env:` | Secret processed by another extended value | Write files | Features that execute code | |---|---|---|---|---|---| | `low` | Allowed | Allowed | Allowed | Allowed | Allowed | | `medium` | Only files in `allowed_paths` | Refused | Refused | Only in `writable_paths` | Refused | | `high` | Refused | Refused | Refused | Only in `writable_paths` | Refused | With level `medium`, the real path of the file is checked: a symbolic link leading outside of `allowed_paths` is refused. With levels `medium` and `high`, files are written only in `writable_paths` (none by default): downloaded files (`config download`, and transfers that receive files), because a file written where the client chooses could replace a file that is used later, for example a plugin or a shell startup file. The real path of the folder is checked: neither `..` nor a symbolic link leading outside of `writable_paths` is accepted. > [!NOTE] > Reading files is not restricted for transfers: a command that sends files (for example `server upload`) reads the files that it is given, whatever the level. > The security level restricts extended values, not the paths of transfers. The same applies to `@file:<path>` in the parameters of stores of option `vault`, and in option `vault_password` (for example, `token`), although the file is read when used, not when the option is evaluated. A secret (read by `@secret:`, `@vault:`, or `@preset:`) is hidden in results by its value: processed by another extended value, it could be revealed (for example, split by `@list:@vault:...`, or with a short value in a text by `@extend:`). With levels `medium` and `high`, a secret must be the whole value of an option or argument, for example `@preset:<PRESET_NAME>.password` for option `password`, and it cannot be a parameter name of `config preset set`. Features that execute code (own, or downloaded), load code, or write a file chosen by the client, are available only at level `low`: | Feature | Why | |---|---| | Option `notify_template` | ERB template: Ruby code | | Option `fpac` | Proxy auto configuration: JavaScript code | | Option `ui`, key `command` | Command executed to open a URL | | Options `sdk_url` and `locations_url` | The SDK, including `ascp`, is downloaded from this location, then executed | | Option `out`, key `file` (and `output`), option `pid_file` | Write (or truncate, or delete) a file chosen by the client | | Option `home` | Configuration folder: its presets, and its folder `plugins` (Ruby code) are used | | Option `plugin_folder` | Plugins are Ruby code | | Plugins of the folder `plugins` of the configuration folder, and of option `plugin_folder` | Ruby code: not loaded at other levels, also if the file was there before | | Command `config plugins create` | Creates Ruby code, loaded as a plugin | | Command `faspex5 postprocessing` | Executes scripts, or Ruby code | At other levels, an option is refused whatever the way it is set (command line, or preset), and a command is not listed by `config commands`. Values from the environment of the process (for example `ASCLI_HOME`) are not restricted: they are set by the one who starts the process, not by a client. A sub-option of an option is restricted in the same way: it is marked `x-security: low` in the schema of the option (for example, `file` of option `out`), and refused when given, by dot notation (`--out.file=...`) or in a `Hash`. A preset is used later, maybe at level `low`: at level `medium` and `high`, `config preset set`, `update` and `initialize` refuse such an option, and a value that contains an extended value that the level refuses (for example `@val:@ruby:...`). Options are evaluated in the order of the command line: place `--security` before options that use extended values. ```shell ascli config echo @file:/etc/hosts --security.level=high ``` ```text ERRR Argument: Evaluation of @file: for argument: value: @file: extended value is disabled by security level high: use level medium with the file in security.allowed_paths (option security), or give the value itself ``` ### Main, configuration and Persistency Folder `ascli` looks for configuration and persistency files (token cache, file lists, persistency files) in the folder specified using option `home`. The default value is `[User's home folder]/.aspera/ascli`. > [!NOTE] > The `[User's home folder]` is determined using Ruby's `Dir.home` method. > Primary source: The HOME environment variable. > On Windows: Ruby also checks `%HOMEDRIVE%%HOMEPATH%` and `%USERPROFILE%` (via `rb_w32_home_dir`). > Additionally, `ascli` sets the `%HOME%` environment variable to the value of `%USERPROFILE%` if it exists and is valid. > Therefore, on Windows, `%USERPROFILE%` is preferred because it is generally more reliable than `%HOMEDRIVE%%HOMEPATH%`. The configuration folder can be displayed using: ```shell ascli config folder ``` ```shell /Users/kenji/.aspera/ascli ``` > [!NOTE] > This is equivalent to displaying the value of the `home` option. ```shell ascli --show-config --fields=home ``` Example (Windows): ```batchfile set ASCLI_HOME=C:\Users\Kenji\.aspera\ascli ascli config folder C:\Users\Kenji\.aspera\ascli ``` When OAuth is used (AoC, Faspex 5), `ascli` keeps a cache of generated bearer tokens in folder `persist_store` located in the configuration folder by default. Option `cache_tokens` (**yes**/no) allows controlling if OAuth tokens are cached on file system, or generated for each request. The command `config tokens flush` clears that cache. Tokens are kept on disk for a maximum of 30 minutes and garbage collected after that. When a token has expired, then a new token is generated, either using a `refresh_token` if it is available, or by the default method. ### Configuration file On the first execution of `ascli`, an empty configuration file is created in the configuration folder (`ascli config folder`). There is no mandatory information required in this file. Its use is optional, as any option can be provided on the command line. Although the file is a standard `YAML` file, `ascli` provides commands to read and modify it using the `config` command. All options for `ascli` can be set on command line, or by env vars, or using [Option Preset](#option-preset) in the configuration file. A configuration file provides a way to define default values, especially for authentication options, thus avoiding having to always specify those options on the command line. The default configuration file is: `$HOME/.aspera/ascli/config.yaml` (this can be overridden with option `config_file`). The configuration file is a catalog of named lists of options, called: [Option Preset](#option-preset). Then, instead of specifying some common options on the command line (for example, address, credentials), it is possible to invoke the ones of an [Option Preset](#option-preset) (for example, `<PRESET_NAME>`) using the option `preset`: `--preset=<PRESET_NAME>` or its shortcut: `-P<PRESET_NAME>`. #### Format of configuration file The configuration file is a `Hash` in a YAML file. **Example**: ```yaml config: version: 0.3.7 default: config: cli_default server: demo_server cli_default: interactive: no demo_server: url: ssh://demo.asperasoft.com:33001 username: asperaweb password: <PASSWORD> ``` We can see here: - The configuration was created with `ascli` version 0.3.7 - The default [Option Preset](#option-preset) to load for `server` plugin is: `demo_server` - The [Option Preset](#option-preset) `demo_server` defines some options: the URL and credentials - The default [Option Preset](#option-preset) to load in any case is: `cli_default` Two [Option Presets](#option-preset) are reserved: - `config` contains a single value: `version` showing the version used to create the configuration file. It is used to check compatibility. - `default` is reserved to define the default [Option Preset](#option-preset) name used for known plugins. The user may create as many [Option Presets](#option-preset) as needed. For instance, a particular [Option Preset](#option-preset) can be created for a particular application instance and contain URL and credentials. Values in the configuration also follow the [Extended Value](#extended-value-syntax) syntax. > [!NOTE] > If the user wants to keep the [Extended Value](#extended-value-syntax) syntax inside the configuration file, the user shall use the `@val:` prefix. Example: ```shell ascli config preset set <AOC_ORG> private_key @val:@file:"$HOME/.aspera/ascli/<PKEY_NAME>" ``` This creates the [Option Preset](#option-preset): ```yaml <AOC_ORG>: private_key: "@file:/Users/laurent/.aspera/ascli/<PKEY_NAME>" ``` So, the key file is read only at execution time and is not embedded in the configuration file. > [!NOTE] > The main use of the configuration file is to store collections of options. > Nevertheless, it can be used to store any type of information, not only option values. > For example, the special preset: [`default`](#special-option-preset-default). #### Option Preset An [Option Preset](#option-preset) is a collection of options and their associated values in a named section in the configuration file. A named [Option Preset](#option-preset) can be modified directly using `ascli`, which will update the configuration file: ```text ascli config preset <show|delete|get|unset|set|initialize|update|ask> <PRESET_NAME> ... ``` The syntax of all `config preset` commands, including `list` and `overview`, is displayed with `ascli config commands config preset`. The command `initialize` allows setting several options at once, but it deletes an existing configuration instead of updating it, and expects a `Hash` [Extended Value](#extended-value-syntax). ```shell ascli config preset initialize demo_server @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"<PASSWORD>","ts":{"precalculate_job_size":true}}' ``` One can also use [positional arguments with dot-path](#positional-arguments-with-dot-path): ```shell ascli config preset initialize demo_server @: url=ssh://demo.asperasoft.com:33001 username=asperaweb password=<PASSWORD> ts.precalculate_job_size=true ``` The command `update` creates an [Option Preset](#option-preset) from options provided on the command line, for example: ```shell ascli config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=<PASSWORD> --ts=@json:'{"precalculate_job_size":true}' ``` This creates an [Option Preset](#option-preset) `demo_server` with all provided options. > [!NOTE] > `update` takes **ALL** options provided on the command line (starting with `--` with a value). The command `set` allows setting individual options in an [Option Preset](#option-preset): ```text ascli config preset set <PRESET_NAME> <PARAMETER_NAME> <PARAMETER_VALUE> ``` Example: ```shell ascli config preset set demo_server password <PASSWORD> ``` The parameter name supports **dot-notation** to set nested hash options. If an intermediate hash already exists, the new value is **deep-merged** into it (existing sibling keys are preserved). ```shell ascli config preset set GLOBAL out.table.pivot single ascli config preset set GLOBAL out.level data ``` The parameter value is **automatically coerced** to its natural type: integers, floats and booleans (`true`/`yes`/`false`/`no`) are stored as native YAML types rather than strings. To **delete** a key from a preset, pass `@none:` as the value (evaluates to `nil`): ```shell ascli config preset set GLOBAL out.table.pivot @none: ``` A full terminal based overview of the configuration can be displayed using: ```shell ascli config preset overview ``` The list of [Option Presets](#option-preset) can be displayed using: ```shell ascli config preset list ``` Any key starting with `_` (underscore) in a preset is treated as a comment and is ignored when the preset is applied. This is useful to annotate presets directly in the configuration file: ```yaml demo_server: _comment: "Demo server - public credentials" url: ssh://demo.asperasoft.com:33001 username: asperaweb password: <PASSWORD> ``` > [!NOTE] > Comment keys are preserved and visible in `config preset overview` for inspection, but are never injected as CLI options. By convention, `_comment` holds a short description of the preset. The [shell completion](#shell-completion) of preset names (e.g. `-P <tab>`) shows it next to each name. The wizard sets it for the presets it creates: `_comment: "wizard: <application>"`. A good practice is to not manually edit the configuration file and use modification commands instead. If necessary, the configuration file can be opened in a text editor with: ```shell ascli config open ``` > [!NOTE] > This starts the editor specified by env var `EDITOR` if defined. It is possible to load an [Option Preset](#option-preset) from within another [Option Preset](#option-preset) using the `preset` option. For example if `pcommon` is a preset with common options, and `pspecific` is a preset with specific options, then `pspecific` can load `pcommon` using: ```shell ascli config preset set pspecific preset pcommon ``` When `pspecific` is loaded, then cumulative option `preset` will be set, and it will also load `pcommon`. #### Special Option Preset: `config` This preset name is reserved and contains a single key: `version`. This is the version of `ascli` which created the file. #### Special Option Preset: `default` This preset name is reserved and contains a `Hash`, where the key is the name of a plugin, and the value is the name of another preset. Usually, [Option presets](#option-preset) are used to contain pre-defined options and values, but this preset contains names of presets to be used by default for plugins. When a plugin is invoked, the preset associated with the name of the plugin is loaded, unless the option `--no-default` (or `-N`) is used. > [!NOTE] > The special plugin name `config` in the `default` preset points to a preset containing **global default options** > (such as `--log-level`, `--vault`, `--interactive`), loaded before any plugin is invoked. > To set global default options, use the `GLOBAL` keyword (see [Plugin: `config`](#plugin-config-configuration)). Operations on this preset are done using regular `config` operations: ```shell ascli config preset set default _plugin_name_ _default_preset_for_plugin_ ``` > [!NOTE] > `default` is not a command, it is the name of the special preset. ```shell ascli config preset get default _plugin_name_ ``` ```json "_default_preset_for_plugin_" ``` ### Handling invalid filename characters Some commands of `ascli` may create files or folders based on input that may contain invalid characters for the local file system. The option `invalid_characters` allows specifying a replacement character for a list of characters that are invalid in filenames on the local file system and replaces them with the specified character. The first character specifies the replacement character, and the following characters are the invalid ones. This is used when a folder or file is created from a value that potentially contains invalid characters. For example, using the option `package_folder`, a package name may contain characters not allowed, such as `/`. The default value is `_<>:"/\|?*`, corresponding to replacement character `_` and characters not allowed on Windows. > [!NOTE] > This option is different from the `replace_illegal_chars` parameter in `aspera.conf`, which applies to transfers only. ### Managing temporary files Some temporary files may be needed during runtime, for example the file list of `ascp`. The composite option `temp` (use `--temp=help` to display this schema) controls them: | Field | Type | Description | |--------|----------------------|----------------------------------------------------------------------------------| | `clean` | `Bool` | Delete temporary files created by the command when it ends. `false` keeps them, e.g. for debugging. Files left by previous commands are still deleted according to `expiry`.<br/>Default: `true`. | | `expiry` | `String, Number, Bool` | Age after which file lists of `ascp` left by previous commands (e.g. interrupted) are deleted, when a command starts. They are in folder `filelists` of the main folder (option `home`). A duration, e.g. `5d`, `12h`, `1w`, `1h 30m`, `1d 02:00:00`, a plain number is seconds. `false` or `null` (`@none:`): never delete them. `0`: delete all of them, including file lists of transfers in progress in other executions of ascli. Such a transfer fails if `ascp` reads its file list again: next session of a multi-session transfer, or resume after an error. Use `0` only when no other transfer is in progress, not in a permanent configuration.<br/>Default: `5d`. | | `folder` | `String` | Folder for temporary files, e.g. certificate for WSS transfers, logs of `transferd`.<br/>- `@sys`: temporary folder of the system for the current user (Ruby `Etc.systmpdir`).<br/>- `@env`: from environment variables, e.g. `TMPDIR` (Ruby `Dir.tmpdir`).<br/>Default: `@sys`. | A plain value is the `folder`, for example: `--temp=/data/tmp`. Temporary files of a command are deleted when it ends, unless `clean` is `false`. If a command is interrupted (for example, killed), its `ascp` file lists remain: a later command deletes them when they are older than `expiry`. > [!WARNING] > `--temp.expiry=0` also deletes the file lists of transfers in progress in other executions of `ascli`, which may then fail: use it only when no other transfer is in progress. > [!TIP] > Options `temp_folder` and `clean_temp` are deprecated aliases for `temp.folder` and `temp.clean`. ### Plugin: `config`: Configuration Plugin `config` provides general commands for `ascli`: - Option Preset operations (configuration file) - `wizard` - `vault` - `ascp` - `transferd` (deprecated, use plugin [`transferd`](#plugin-transferd-transfer-daemon)) The default preset for `config` is read for any plugin invocation: this allows setting global options, such as `--log-level` or `--interactive`. When `ascli` starts, it looks for the `default` Option Preset and checks the value for `config`. If set, it loads the options independently of the plugin used. > [!NOTE] > If `default.config` is not set, `ascli` automatically uses `global_common_defaults` as the global preset name > and saves that mapping in the configuration file the first time the `GLOBAL` keyword is used. > [!TIP] > If you do not know the name of the global preset, you can use `GLOBAL` to refer to it. Show current default (global) Option Preset (`config` plugin): ```shell ascli config preset get default config ``` ```text global_common_defaults ``` Set a global parameter: ```shell ascli config preset set GLOBAL version_check_days 0 ``` If you want to use a custom name instead of `global_common_defaults`: ```shell ascli config preset set default config <GLOBAL_OPTIONS_NAME> ascli config preset set GLOBAL version_check_days 0 ``` #### Tested commands for `config` > [!NOTE] > Add `ascli config` in front of the following commands: ```shell agents jobs cleanup agents jobs list agents jobs status <CONF_ASYNC_NODE_UPLOAD> agents list agents parameters connect agents parameters direct agents parameters node agents show connect agents show direct ascp errors ascp info --sdk-folder=sdk_test_dir ascp products list ascp schema --format=jsonpp ascp show ascp spec check_update coffee coffee --log-level=trace2 --log-format=caller coffee --ui=text coffee --ui=text --out.img.text=true coffee --ui=text --out.img=@json:'{"text":true,"double":false}' commands commands aoc --expand-mounts=yes commands aoc files commands server detect @: url=app.example.com detect @: url=https://f5.example.com/path detect @: url=https://f5.example.com/path plugin=faspex5 detect @: url=https://node.example.com/path detect @: url=https://server1.example.com/path detect @: url=https://shares.example.com/path plugin=shares detect @: url=https://tst.example.com/path plugin=faspio detect @: url=https://tst.example.com/path plugin=httpgw detect @: url=my_org plugin=aoc doc doc github transfer-parameters doc local quick-start --ui=text doc toc echo '<svg viewBox="0 0 100 100" xmlns="http://www.w3.org/2000/svg"><circle cx="50" cy="50" r="50" fill="#069"/></svg>' --format=image --out.img.text=true echo -- --special-string echo @base64:SGVsbG8gV29ybGQK echo @csvt:@stdin: echo @env:USER echo @json:'[{"email":"a@x.com"},{"email":"b@x.com"},{"email":"c@y.org"}]' --out.query=@ruby:'->(users){users.group_by{|u|u["email"].split("@").last}.transform_values(&:count)}' --format=json echo @json:'[{"name":"alice","size":10},{"name":"bob","size":2000}]' --out.query='[?size > `100`].name' --format=text echo @json:'[{"user":{"id":1,"name":"foo"},"project":"bar"}]' --out.table.pivot=single echo @json:'[{"user":{"id":1,"name":"foo"},"project":"bar"}]' --out.table.pivot=yes echo @json:'{"empty":"","bool":true}' --out.colors=yes --out.utf8=yes echo @lines:@stdin: echo @list:,1,2,3 echo @secret: --ui=text --out.secrets=yes echo @stdin:bin echo @uri:/etc/hosts echo @uri:file:/etc/hosts echo @uri:http://ifconfig.me echo @uri:https://ifconfig.me echo @uri:https://ifconfig.me/ip --cert-stores=@ruby:[OpenSSL::X509::DEFAULT_CERT_DIR,OpenSSL::X509::DEFAULT_CERT_FILE] --http-options.ssl_options=@list:,CIPHER_SERVER_PREFERENCE,-NETSCAPE_CA_DN_BUG --insecure=yes echo @vault:my_preset.password echo @zlib:@stdin: echo hello email_test --notify-to=my_email_external file file --log-level=debug --log-format=@ruby:'->(s, d, p, m){"#{d.strftime("%Y-%m-%d %H:%M:%S")} - #{p} - #{s} - #{m}\n"}' file --log-level=debug --log-format=default file --log-level=debug --log-format=standard file --log-level=debug --logger=syslog folder gem name gem path gem version genkey my_key genkey my_key 4096 image https://eudemo.asperademo.com/wallpaper.jpg initdemo open options aoc options server options server --select=@json:'{"option":"--display"}' --fields=replacement plugins create my_command . plugins list preset delete conf_name preset initialize conf_name @json:'{"p1":"v1","p2":"v2"}' preset list preset overview preset set GLOBAL sdk_folder 'product:IBM Aspera Connect' preset set GLOBAL vault @: default=local stores.local.type=file stores.local.file=/secure/vault_file preset set GLOBAL vault_password <PASSWORD> preset set GLOBAL version_check_days 0 preset set conf_name param value preset set default shares conf_name preset show conf_name preset unset conf_name param preset update conf_name --p1=v1 --p2=v2 proxy_check --fpac=@file:proxy.pac https://eudemo.asperademo.com --proxy-credentials=@list:,user,pass pubkey @file:my_key remote_certificate chain https://node.example.com/path remote_certificate name https://node.example.com/path remote_certificate only https://node.example.com/path smtp_settings sync spec tokens flush tokens list vault check @vault://local/my_label vault create @: label=my_label password=<PASSWORD> 'description=my secret' vault delete foo --vault.type=system vault delete my_label vault field my_label vault ids vault info vault list vault overview vault show my_label wizard @: url=https://console.example.com/path plugin=console override=yes --username=my_username --password=<PASSWORD> wizard @: url=https://f5.example.com/path plugin=faspex5 override=yes --username=my_username --private-key=@file:<PRIVATE_KEY_FILE> --client-id=<CLIENT_ID> --client-secret=<CLIENT_SECRET> wizard @: url=https://node.example.com/path plugin=node --username=test --password=<PASSWORD> wizard @: url=https://orch.example.com/path plugin=orchestrator --username=test --password=<PASSWORD> wizard @: url=https://server1.example.com/path plugin=server --username=my_username --password=<PASSWORD> wizard @: url=https://shares.example.com/path plugin=shares --username=test --password=<PASSWORD> wizard @: url=https://tst.example.com/path plugin=faspio override=yes --username=my_username --password=<PASSWORD> wizard @: url=https://tst.example.com/path plugin=httpgw wizard @: url=my_org plugin=aoc --username=my_user_email --private-key=@file:<PRIVATE_KEY_FILE> --use-generic-client=yes wizard @: url=my_org plugin=aoc preset=my_preset --username=my_user_email --private-key=@file:<PRIVATE_KEY_FILE> ``` #### Evaluation order of options Some options are global, others are available only for some plugins (the plugin is the first-level command). Options are loaded using this algorithm: - If option `--no-default` (or `-N`) is specified, then no default value is loaded for the plugin - Else it looks for the name of the plugin as key in section `default`, the value is the name of the default [Option Preset](#option-preset) for it, and loads it. - If option `--preset=<NAME>` is specified (or `-P<NAME>`), this reads the [Option Preset](#option-preset) specified from the configuration file by name. - If option `--preset=<EXTENDED_VALUE_HASH>` is specified, it is used as option values (`Hash` of option/value pairs). - Environment variables are evaluated. - Command line options are evaluated. Options are evaluated in the order of command line. To avoid loading the default [Option Preset](#option-preset) for a plugin, use: `-N` On command line, words in option names are separated by a dash (`-`). In the configuration file, the separator is an underscore. For example, `--xxx-yyy` on command line gives `xxx_yyy` in configuration file. The main plugin name is `config`, so it is possible to define a default [Option Preset](#option-preset) for the main plugin with: ```shell ascli config preset set cli_default interactive no ``` ```shell ascli config preset set default config cli_default ``` An [Option Preset](#option-preset) value can be removed with `unset`: ```shell ascli config preset unset cli_default interactive ``` Example: Define options using command line: ```shell ascli -N --url=<URL> --password=<PASSWORD> --username=<USERNAME> node --show-config ``` Example: Define options using a `Hash`: ```shell ascli -N --preset=@json:'{"url":"<URL>","password":"<PASSWORD>","username":"<USERNAME>"}' node --show-config ``` #### Shell Completion `ascli` supports shell tab-completion for **Bash**, **Zsh**, **Fish**, and **PowerShell** (Windows PowerShell 5.1 and PowerShell 7, on any OS). The command `ascli config completion <shell>` displays the completion script for the given shell. The default shell is Zsh on macOS, Bash on most Linux distributions, and PowerShell on Windows (its prompt starts with `PS`). In Bash and Zsh, `echo $0` displays the name of the running shell. To activate completion, add the line for your shell to its startup file: ```bash # Bash, in ~/.bashrc eval "$(ascli config completion bash)" ``` ```zsh # Zsh, in ~/.zshrc, after compinit eval "$(ascli config completion zsh)" ``` ```fish # Fish, in ~/.config/fish/config.fish ascli config completion fish | source ``` ```powershell # PowerShell, in $PROFILE (edit with notepad $PROFILE) ascli config completion powershell | Out-String | Invoke-Expression ``` The startup file then executes `ascli` each time a shell starts. Alternatively, save the script once in the completion folder of the shell (and save it again after an upgrade of `ascli`): ```bash # Bash, with package bash-completion ascli config completion bash > ~/.local/share/bash-completion/completions/ascli # Zsh (the folder must be in $fpath before compinit) ascli config completion zsh > ~/.zsh/completions/_ascli # Fish ascli config completion fish > ~/.config/fish/completions/ascli.fish ``` PowerShell has no completion folder: save the script, and dot-source it in `$PROFILE`: ```powershell ascli config completion powershell | Set-Content "$HOME\ascli_completion.ps1" Add-Content -Path $PROFILE -Value ". '$HOME\ascli_completion.ps1'" ``` In PowerShell, the words proposed are kept for the session, so only the first `Tab` at a given position executes `ascli`. Once active, press `Tab` to complete: ```bash ascli <Tab> # plugins and global options: aoc, server, node, ..., --format, ... ascli server <Tab> # sub-commands and options of plugin: upload, download, ls, ..., --url, ... ascli aoc admin <Tab> # sub-commands at any depth: user, node, ... ascli server ls --u<Tab> # options, after any command: --url, --username, ... ascli --format=<Tab> # values of option: --format=json, --format=table, ... ascli --ts.<Tab> # keys of option, from its schema: --ts.direction=, --ts.target_rate_kbps=, ... ascli --out.t<Tab> # a sub-structure is followed by a dot: --out.table. ascli --ts.direction=<Tab> # values of key: --ts.direction=send, --ts.direction=receive ascli --transfer=<Tab> # plain value of option, key agent: --transfer=direct, --transfer=node, ... ascli faspex5 admin accounts list --query.<Tab> # keys of --query, from the command: --query.limit=, ... ascli server upload <Tab> # local files ascli server download --to-folder <Tab> # local folders ascli config genkey <Tab> # local files, where a local path is expected ascli --temp.folder=<Tab> # local folders, for a key of option ascli --ts=@<Tab> # extended value modifiers: @json:, @file:, @preset:, @:, ... ascli --private-key=@file:<Tab> # local files, after @file: or @path: ascli -P <Tab> # presets: also for config preset show, @preset: ascli config preset get myconf <Tab> # parameters of preset: url, username, ... ascli config agents jobs status <Tab> # asynchronous transfer jobs ascli node access_keys do <Tab> # type of expected value, not inserted: <identifier> ascli aoc packages send <Tab> # type, and keys of structure (required ones with *): <Hash> name*, workspace_id*, ... ascli aoc packages send @: <Tab> # keys of structure built with @:, from its schema: name=, workspace_id=, recipients.0., ... ascli aoc packages send @: archived=<Tab> # values of key: archived=true, archived=false ascli aoc packages send @: name=x <Tab> # end marker of structure built with @: END, and the other keys ``` No space is inserted after a word that continues: `key=`, `key.`, `@file:`, or a folder (Bash 4 and later). Zsh, Fish and PowerShell also display the description of commands, options, keys and extended value modifiers. In PowerShell, a word starting with `@` is inserted with quotes (e.g. `'@file:README.md'`), as required by PowerShell. Zsh lists options once `-` is typed (zsh style `prefix-needed`). The type of an expected value is displayed by Bash and Zsh, and by PowerShell in the list of `MenuComplete` (`Ctrl+Space`). The scripts call `ascli config completion candidates -- [<word>...] <current word>`, which lists the candidates for the word at the cursor, with their description, followed by a directive for the shell: `:files` or `:folders` (local paths, followed by the beginning of the word that is not part of the path, if any, e.g. `:files --private-key=@file:`), `:hint <text>` or `:`. The words are not evaluated (e.g. `@env:` is not replaced). The command `ascli config completion words [<word>...]` lists the words that can follow the given ones. It can be used directly to inspect available commands: ```bash ascli config completion words aoc admin ``` #### Wizard The wizard is a command that asks the user for information and creates an [Option Preset](#option-preset) with the provided information for a given application. It takes one optional argument: a `Hash`, usually given with [`@:`](#positional-arguments-with-dot-path) as `key=value` items, in any order: | Key | Description | If not provided, `ascli` | |------------|---------------------------------------------------------------------------------|--------------------------------------------| | `url` | **URL** or hostname of the application | asks for it. | | `plugin` | The **plugin name** to limit search to | tries to detect known plugins from the URL | | `preset` | The **preset name** to save in config file | generates a unique preset name | | `default` | Set the preset as default configuration for the plugin (`yes`/`no`) | `yes` | | `override` | Override the existing default preset of the plugin, if it exists (`yes`/`no`) | `no` | Options of the plugin (for example `--username`, `--password`, `--client-id`, `--private-key`), see the plugin, are given as usual on the command line: they are added to the [Option Preset](#option-preset) created by the wizard. Options that the wizard needs, but that are not provided, are asked. An unknown key in the `Hash` is refused. For example: ```shell ascli config wizard @: url=<URL> plugin=<PLUGIN> preset=<PRESET> ``` The command `config detect` takes the same argument, with keys `url` and `plugin`, and shows the detected applications, without creating a preset. Use `ascli config wizard help` to show the schema of the argument. The simplest invocation (without argument: the URL is asked) is: ```shell ascli config wizard ``` If the application requires a private key, the user can provide it with option `--private-key` (the key, e.g. `@file:<path>`, not only its path), or let the wizard generate one. The user is told where to place the associated public key PEM in the application. #### Example of configuration for a plugin For Faspex 5, Shares, Node (including ATS, Aspera Transfer Service), Console, only username/password and URL are required (either on command line, or from configuration file). Those can be provided on the command line: ```shell ascli shares files browse / --url=https://10.25.0.6 --username=john --password=<PASSWORD> ``` This can also be provisioned in a configuration file: - Build [Option Preset](#option-preset) ```shell ascli config preset set shares06 url https://10.25.0.6 ascli config preset set shares06 username john ascli config preset set shares06 password <PASSWORD> ``` This can also be done with one single command: ```shell ascli config preset initialize shares06 @json:'{"url":"https://10.25.0.6","username":"john","password":"<PASSWORD>"}' ``` Or: ```shell ascli config preset update shares06 --url=https://10.25.0.6 --username=john --password=<PASSWORD> ``` - Define this [Option Preset](#option-preset) as the default [Option Preset](#option-preset) for the specified plugin (`shares`) ```shell ascli config preset set default shares shares06 ``` - Display the content of the configuration file in table format ```shell ascli config preset overview ``` - Execute a command on the **Shares** application using default options ```shell ascli shares files browse / ``` ### Secret Vault Secrets, for example, passwords, keys, are needed when connecting to applications. Those secrets are usually provided as command options: on the command line, in env vars, in files, and so on. For security reasons, those secrets shall not be exposed in clear, either: - On terminal during input - In logs - In command output Instead, they shall be hidden (logs, output) or encrypted (configuration). Terminal output (command result) secret removal is controlled by option `--out.secrets` (default: `no`). Log secret removal is controlled by option `log_secrets` (default: `no`). Secrets are identified by the name of their field (for example, `password`, `token`), and by their value: values of secret options and of presets, of [Extended Values](#extended-value-syntax) `@secret:` and `@vault:`, tokens obtained or generated, and secrets of transfer specs are hidden wherever they appear, also in field names. In presets, a secret made of digits only (for example, a password not quoted in the YAML file) is converted to a string, so that it is identified as a secret. A value shorter than 8 characters is hidden only when it is the whole value, not inside a text, except in responses of `mcp server`, where it is also hidden as a word (not next to a letter or digit). Mandatory command line options can be requested interactively (for example, password) using option `interactive`. It is possible to use [Extended Value](#extended-value-syntax) `@secret:[name]` to ask for a secret interactively. If option `ui` is `graphical` (see [Graphical Interactions](#graphical-interactions-browser-and-text-editor)), the secret is entered on a web page opened in the browser, served by a local web server on `127.0.0.1`, else on terminal. In MCP server mode, the secret is entered on a web page, see [Secrets entered in a web page](#secrets-entered-in-a-web-page). It is also possible to enter an option as an environment variable, for example, `ASCLI_PASSWORD` for option `password` and read the env var like this: ```shell read -s ASCLI_PASSWORD export ASCLI_PASSWORD ``` Another possibility is to retrieve values from a secret vault. Option `vault` declares named secret stores (`stores`), and the name of the default store (`default`): - `@vault:<label>.<field>` reads field `<field>` of secret `<label>` of the default store, managed with command `config vault` (see [Vault: Operations](#vault-operations)). - `@vault://<store>/<path>[?<options>][#<field>]` reads a secret of any store (see [Vault: Secret references](#vault-secret-references)). | Field | Type | Description | |-----------|--------|----------------------------------------------------------------------------------| | `cache_ttl` | `Number` | Time in seconds a value read by reference is kept in memory, `0` for no cache.<br/>Default: `60`. | | `default` | `String` | Name of the default store, in `stores`: read by `@vault:<label>.<field>`, managed by `config vault`, unlocked with option `vault_password`. No default store if not set. | | `stores` | `Hash` | Name of store (letters, digits, `_`, `.`, `-`) → parameters of store, with `type`. Secrets are read by reference with `@vault://<store>/<path>[?<options>][#<field>]` from any store.<br/>Required. | For example, to use as default store an encrypted file `vault.bin` in the configuration folder (see [Vault: Encrypted file](#vault-encrypted-file)): ```shell ascli config preset set GLOBAL vault @json:'{"default":"local","stores":{"local":{"type":"file","file":"vault.bin"}}}' ``` Parameters of a store, with `type` (see sections below): | Field | Type | Description | |-------------------|---------------|----------------------------------------------------------------------------------| | **type** | `String` | `vault`: IBM HashiCorp Vault secret store. Labels of `config vault` and `@vault:<label>.<field>` are in mount `secret` (KV v2). Reference `@vault://<store>/<mount>/<path>#<field>` (field required) in any KV v2 mount, options `version=<n>`, `mount=<mount>` (when the mount contains `/`). | | `allowed_prefixes` | `Array[String]` | Allowed prefixes of `<mount>/<path>` of references: a prefix ending with `/` is a folder, else a path or a folder. All paths are allowed if not set. | | `approle` | `Hash` | AppRole authentication: the token is obtained by login, renewed, and obtained again when refused. | | `approle.mount` | `String` | Mount path of the AppRole auth method.<br/>Default: `approle`. | | `approle.role_id` | `String` | Role ID.<br/>Required. | | `approle.secret_id` | `String` | Secret ID (if the role requires one): the value, or `@file:<path>`. | | `ca_file` | `String` | File with the CA certificates that verify the certificate of the server (HTTPS). | | `namespace` | `String` | Namespace (Vault Enterprise). | | `token` | `String` | The authentication token. Its value, or `@file:<path>` (e.g. written and renewed by Vault Agent), read again when modified. One of `token` or `approle` is required. For the default store, defaults to the value of `vault_password`, unless `approle` is set.<br/>Example: `<TOKEN>`. | | `type` | `Enum[String]` | Vault type selector.<br/>Required.<br/>Allowed values: `vault`. | | `url` | `String` | URL of the HashiCorp Vault server.<br/>Required.<br/>Example: `http://127.0.0.1:8200`. | | **type** | `String` | `system`: macOS system keychain secret store. | | `name` | `String` | Name of the keychain to use (file `<name>.keychain-db` in the keychain search list).<br/>Default: `ascli`. | | `type` | `Enum[String]` | Vault type selector.<br/>Required.<br/>Allowed values: `system`. | | **type** | `String` | `secret_service`: Linux Secret Service secret store (GNOME Keyring, KWallet, KeePassXC...), using `secret-tool`. | | `name` | `String` | Name of the group of secrets (attribute `service` of the items).<br/>Default: `ascli`. | | `type` | `Enum[String]` | Vault type selector.<br/>Required.<br/>Allowed values: `secret_service`. | | **type** | `String` | `file`: Encrypted file secret store. Reference `@vault://<store>/<label>[#<field>]`, field `password` by default. | | `file` | `String` | Path to the encrypted vault file, created when the first secret is added. Relative paths are resolved from the configuration folder. Read when used, and again when modified.<br/>Default: `vault.bin`. | | `name` | `String` | Same as `file` (compatibility). | | `password` | `String` | Password of the vault file: the value, or `@file:<path>`, read again when modified. For the default store, defaults to the value of `vault_password`. | | `type` | `Enum[String]` | Vault type selector.<br/>Required.<br/>Allowed values: `file`. | | **type** | `String` | `1password`: 1Password secret store (Connect REST API or CLI). | | `account` | `String` | Account shorthand for `op` (used when `source` is `cli`). Uses the default account if omitted. | | `source` | `Enum[String]` | Backend source. Use `api` for the Connect REST API, `cli` for the `op` CLI.<br/>Required.<br/>Allowed values: `api`, `cli`.<br/>Default: `api`. | | `token` | `String` | Bearer token for the Connect API (required when `source` is `api`). For the default store, defaults to the value of `vault_password`. | | `type` | `Enum[String]` | Vault type selector.<br/>Required.<br/>Allowed values: `1password`. | | `url` | `String` | Base URL of the 1Password Connect server (required when `source` is `api`).<br/>Example: `http://localhost:8080`. | | `vault` | `String` | Name or ID of the 1Password vault to use with the CLI (used when `source` is `cli`). Uses the default vault if omitted. | | `vault_id` | `String` | ID of the 1Password vault to use (required when `source` is `api`). | | **type** | `String` | `keeper`: Keeper Secrets Manager secret store (Keeper Vault), using gem `keeper_secrets_manager`. A secret is a record shared with the application: its label is its title or its UID. Reference `@vault://<store>/<title or uid>[#<field>]`, field `password` by default: standard field by type (`login`, `password`, `url`...), custom field by label, or `notes`. | | `config` | `String` | Configuration of the client device (base64 or JSON, e.g. from `ksm profile export`): the value, or `@file:<path>`. For the default store, defaults to the value of `vault_password`. | | `folder_uid` | `String` | UID of the shared folder where `config vault create` adds records (not needed to read). | | `type` | `Enum[String]` | Vault type selector.<br/>Required.<br/>Allowed values: `keeper`. | > [!NOTE] > The parameters of one store, without `stores` (for example, `{"type":"file","name":"vault.bin"}`), are deprecated: they define a store named after its type (for example, `file`), which is the default store. > Set the store in `stores`, and its name in `default` instead: `{"default":"local","stores":{"local":{"type":"file","file":"vault.bin"}}}`. Secrets used to access a store (`token`, `approle.secret_id`, `password`) are the value, or `@file:<path>` (for example, a credential of `systemd`, or a token written by Vault Agent): the file is read when used, and again when it is modified. As for extended value `@file:`, reading the file is restricted by option `security` (see [Security Level](#security-level)). In a YAML file, quote the value: `@` cannot start a plain scalar. The secret of the default store (`password` of type `file`, `token` of type `vault` without `approle`, of type `1password` with `source` `api`, and `config` of type `keeper`), if not in its parameters, is the value of option `vault_password`. To avoid exposing the vault password in the shell history, provide it via an environment variable: ```shell export ASCLI_VAULT_PASSWORD=<YOUR_PASSWORD> ``` > [!TIP] > Set it interactively using `read` so that it is not recorded in the shell history: > > ```shell > read -rs ASCLI_VAULT_PASSWORD && export ASCLI_VAULT_PASSWORD > ``` #### Vault: IBM HashiCorp Vault <https://developer.hashicorp.com/vault> Quick start macOS: ```shell brew tap hashicorp/tap brew install hashicorp/tap/vault vault server -dev -dev-root-token-id=<TOKEN> ``` ```shell --vault=@json:'{"default":"corp","stores":{"corp":{"type":"vault","url":"http://127.0.0.1:8200"}}}' --vault-password=<TOKEN> ``` With a token in a file (for example, written and renewed by Vault Agent), read again when the file is modified: ```shell --vault=@json:'{"default":"corp","stores":{"corp":{"type":"vault","url":"https://vault.example.com:8200","token":"@file:/run/vault/token"}}}' ``` With AppRole authentication, the token is obtained by login, and renewed: ```shell --vault=@json:'{"default":"corp","stores":{"corp":{"type":"vault","url":"https://vault.example.com:8200","approle":{"role_id":"<ROLE_ID>","secret_id":"@file:/etc/vault/secret_id"}}}}' ``` #### Vault: System keychain > [!WARNING] > **macOS only** It is possible to manage secrets in macOS keychain. Commands `list`, `ids` and `delete` are not supported: use the **Keychain Access** app. ```shell --vault=@json:'{"default":"keychain","stores":{"keychain":{"type":"system","name":"ascli"}}}' ``` #### Vault: Linux Secret Service > [!WARNING] > **Linux only** It is possible to manage secrets in the Linux Secret Service (GNOME Keyring, KWallet, KeePassXC...), using the `secret-tool` command of `libsecret` (for example, package `libsecret-tools` on Debian and Ubuntu). A session with an unlocked keyring is required. Secrets are grouped with attribute `service`, which is parameter `name` (default: `ascli`). ```shell --vault=@json:'{"default":"keyring","stores":{"keyring":{"type":"secret_service","name":"ascli"}}}' ``` #### Vault: Encrypted file It is possible to store and use secrets encrypted in a file, with a store of type `file`: ```json {"default":"local","stores":{"local":{"type":"file","file":"vault.bin"}}} ``` To configure the encrypted file vault as the default store, set the `vault` option in the global preset: ```shell ascli config preset set GLOBAL vault @json:'{"default":"local","stores":{"local":{"type":"file","file":"vault.bin"}}}' ``` The vault file is created automatically when the first secret is added - no explicit initialization is needed. It is read when used, and again when it is modified. > [!WARNING] > The `vault_password` option should **not** be stored in the config file: doing so would protect secrets with a password that is itself stored in plain text, defeating the purpose of the vault. > Use an environment variable as described above, or parameter `password` with `@file:<path>` (for example, a credential of `systemd`). #### Vault: 1Password Use `type: 1password` with a `source` parameter to select the backend. ##### Source: API (Connect REST API) <https://developer.1password.com/docs/connect/> [1Password Connect](https://developer.1password.com/docs/connect/) exposes a local REST API backed by a 1Password vault. Deploy the Connect server with Docker, then point `ascli` at it: ```shell docker run -d --name op-connect \ -p 8080:8080 \ -v /path/to/1password-credentials.json:/home/opuser/.op/1password-credentials.json \ 1password/connect-api:latest ``` ```shell --vault=@json:'{"default":"op","stores":{"op":{"type":"1password","source":"api","url":"http://localhost:8080","vault_id":"<VAULT_ID>"}}}' \ --vault-password=<CONNECT_TOKEN> ``` > [!TIP] > The `vault_id` is the alphanumeric ID shown in the 1Password web app under **Vault Settings**. > The `vault_password` holds the Connect bearer token generated when creating the Connect server credentials. > [!NOTE] > The [1Password Service Accounts API](https://developer.1password.com/docs/service-accounts/) is also supported: use `url=https://api.1password.com` with a service account token as `vault_password`. > No local server is required in that case. ##### Source: Cli (`op` CLI) <https://developer.1password.com/docs/cli/> Requires the `op` CLI to be installed and signed in. No server to deploy: authentication is handled by the 1Password desktop app (biometric unlock) or by `op signin`. ```shell --vault=@json:'{"default":"op","stores":{"op":{"type":"1password","source":"cli"}}}' # or, to target a specific vault and account: --vault=@json:'{"default":"op","stores":{"op":{"type":"1password","source":"cli","vault":"<VAULT_NAME>","account":"<ACCOUNT_SHORTHAND>"}}}' ``` > [!TIP] > Install `op` on macOS with `brew install 1password-cli`, then connect it to the desktop app: > `op signin` ##### Referencing items An item can be referenced by its title (label) or by its 1Password ID, in `@vault:<item>.<field>`: ```shell --password=@vault:<ITEM_ID>.password --password=@vault:<ITEM_TITLE>.password ``` > [!NOTE] > If several items share the same title, use the item ID (or the `vault` parameter to restrict the search). #### Vault: Keeper <https://docs.keeper.io/keeperpam/secrets-manager/overview> Keeper Secrets Manager (KSM) gives an application access to the records of shared folders of a Keeper vault, without the credentials of a user. It requires the optional gem `keeper_secrets_manager`: ```shell gem install keeper_secrets_manager ``` Create an application and a client device in Keeper (see the [Quick Start Guide](https://docs.keeper.io/keeperpam/secrets-manager/quick-start-guide)), share the folders of secrets with the application, and get the configuration of the device (base64 or JSON, for example with `ksm profile export`). Parameter `config` is the value, or `@file:<path>`: ```shell --vault=@json:'{"default":"keeper","stores":{"keeper":{"type":"keeper","config":"@file:<KSM_CONFIG_FILE>"}}}' ``` If `config` is not set in the default store, it is the value of option `vault_password`. A secret is a record of the vault: its label is its title, or its UID. The field of a reference is `password` by default: standard field of the record (`login`, `password`, `url`...), custom field by its label, or `notes`. Field `username` is an alias of `login`. ```shell --password=@vault:<RECORD_TITLE>.password --password=@vault://keeper/<RECORD_TITLE>#<FIELD> ``` To create records with `config vault create` (type `login`), set parameter `folder_uid`: UID of the shared folder where records are added. Reading does not need it. > [!NOTE] > If several records share the same title, use the UID. > Deleting a record with `config vault delete` requires that the application has edit permission. #### Vault: Operations Secrets of the default store can be manipulated using the `config vault` command: - `info` : Show vault information - `ids` : List secret labels in the vault - `list` : List all secrets with full details - `overview` : List labels of all secrets with their field names, without values - `field` : List field names of a secret by label, without values - `show` : Show a secret by label (or id) - `create` : Add a new secret to the vault - `delete` : Delete a secret by label (or id) - `password` : Set a new password for the vault - `import` : Import secrets from a JSON array (supports `--bulk`) - `check` : Check that a [Secret Reference](#vault-secret-references) can be read, without its value To add a new password entry in the vault for label `<NAME>`: ```shell ascli config vault create @: label=<NAME> password=@secret:password description='for this account' ``` #### Vault: Migration between vaults To migrate all secrets from one vault backend to another (for example, from the encrypted file vault to 1Password), use `vault list` piped into `vault import`. > [!NOTE] > Use `list` (not `ids`) as the source: `ids` returns only labels, while `list` returns the full secret details needed for import. ```shell ascli config vault list --format=json --out.level=data --out.secrets=yes \ --vault=@json:'{"default":"local","stores":{"local":{"type":"file","file":"<SOURCE_VAULT_FILE>"}}}' \ --vault-password=<SOURCE_PASSWORD> | \ ascli config vault import @json:@stdin: --bulk=yes \ --vault=@json:'{"default":"op","stores":{"op":{"type":"1password","url":"<CONNECT_URL>","vault_id":"<VAULT_ID>"}}}' \ --vault-password=<CONNECT_TOKEN> ``` > [!TIP] > Use `--out.level=data` on the `list` command so that only the raw JSON array is written to stdout, with no table headers or status lines, and `--out.secrets=yes` so that secrets are not replaced by a placeholder. The `import` command accepts a JSON array where each element is a vault secret object (same schema as `create`). `--bulk` makes each entry reported individually in the result table; omit it to get a single-line summary. On error (for example a duplicate label), the default behavior is to stop (`--bfail=yes`). Pass `--bfail=no` to skip failures and continue with the remaining entries. #### Vault: Secret references Secrets can also be read by reference, from any store of option `vault`: ```text @vault://<store>/<path>[?<options>][#<field>] ``` - `<store>`: name of a store in `stores` of option `vault`. - `<path>`: path of the secret in the store, and `<field>`: field of the secret. Both are percent-decoded (for example, `%20` for a space). For most types, the path is the label, and the field is `password` by default. For type `vault`, the path is `<mount>/<path>` in a KV v2 secrets engine, and the field is required. - `<options>`: options of the store, as a URL query (for example, `version=2`). References only read secrets: they are managed with the tools of the store (for example, `vault kv put` for HashiCorp Vault), or with `config vault` for the default store. Compared to `@vault:<label>.<field>`, several stores can be used in the same command, and a HashiCorp Vault store reads any KV v2 mount, a given version, and limits the paths that can be read by reference (`allowed_prefixes`). Values read by reference are kept in memory for `cache_ttl` seconds. The configuration of stores is the same as in applications that use the library with references `secret://<store>/<path>` (for example, IBM Aspera Orchestrator). For example, a HashiCorp Vault server with AppRole authentication, limited to secrets under `kv/aspera/`, and the encrypted file of `config vault` as default store, unlocked with option `vault_password`: ```shell --vault=@json:'{"default":"local","stores":{"corp":{"type":"vault","url":"https://vault.example.com:8200","approle":{"role_id":"<ROLE_ID>","secret_id":"@file:/etc/vault/secret_id"},"allowed_prefixes":["kv/aspera/"]},"local":{"type":"file","file":"vault.bin"}}}' ``` References in a preset: to field `password` of secret `aspera/faspex` in mount `kv`, and to field `password` (default field of type `file`) of label `<VAULT_LABEL>` in the encrypted file: ```shell ascli config preset update <PRESET_NAME> --url=... --username=... --password=@val:@vault://corp/kv/aspera/faspex#password ascli config preset update <PRESET_NAME> --url=... --username=... --password=@val:@vault://local/<VAULT_LABEL> ``` To check that a reference can be read, without its value (store, version, time of update): ```shell ascli config vault check @vault://corp/kv/aspera/faspex#password ``` #### Configuration Finder When a secret is needed by a sub command, the command can search for existing configurations in the configuration file by setting option `secret` to the special value `PRESET`. The lookup is done by comparing the service URL and username (or access key). #### Securing passwords and secrets A password can be saved in clear in an [Option Preset](#option-preset) together with other account information (URL, username, and so on). Example: ```shell ascli config preset update <PRESET_NAME> --url=... --username=... --password=... ``` For more secure storage, first create a vault entry (see [Vault: Operations](#vault-operations)), then refer to it in the preset: ```shell ascli config preset update <PRESET_NAME> --url=... --username=... --password=@val:@vault:<VAULT_LABEL>.password ``` > [!NOTE] > Use `@val:` in front of `@vault:` so that the [Extended Value](#extended-value-syntax) is not evaluated. To migrate existing clear-text passwords from presets to the vault in one step, use the `secure` command. The vault must already be configured (option `vault`, see [Secret Vault](#secret-vault)). ```shell ascli config preset secure ``` This scans all presets (or a single one if a name is provided) and, for every option whose name ends with `password` or `secret`, moves the value into the vault and replaces it with a `@vault:` reference. ```shell ascli config preset secure <PRESET_NAME> ``` When a vault is configured, the `update`, `set`, and `ask` preset commands automatically move secret values (options ending with `password` or `secret`) into the vault as they are written, so no manual migration is needed. To disable this behavior for a single command, pass `--vault=@none:`. ### Private Key Some Aspera applications allow the user to be authenticated using [Public Key Cryptography](https://en.wikipedia.org/wiki/Public-key_cryptography): - For SSH: Server - For OAuth JWT: AoC, Faspex 5, faspio Gateway It consists of using a pair of associated keys: a private key and a public key. The same pair can be used for multiple applications. The file containing the private key (key pair) can optionally be protected by a passphrase. If the key is protected by a passphrase, then the passphrase is prompted when the key is used. Some plugins support option `passphrase`. Keys of type `ed25519` and OpenSSH encoded keys (default on modern Linux) are supported. See section: [Private key type ed25519](#private-key-type-ed25519). For OpenSSH encoded keys protected by a passphrase, install gem `bcrypt_pbkdf` (not available for JRuby): ```shell gem install bcrypt_pbkdf ``` The following sample commands use the shell variable `KEY_PAIR_PATH`. Set it to the desired safe location of the private key. Typically, located in folder `$HOME/.ssh` or `$HOME/.aspera/ascli`. For example: ```shell KEY_PAIR_PATH=~/.aspera/ascli/<PKEY_NAME> ``` Several methods can be used to generate a key pair. The format expected for keys is [PEM](https://en.wikipedia.org/wiki/Privacy-Enhanced_Mail). If another format is used, such as `DER`, it can be converted to `PEM`, for example, using `openssl`. #### `ascli` for key generation The generated key is of type `RSA`, by default: **4096** bit. For convenience, the public key is also extracted with extension `.pub`. Files are PEM encoded. The key is not passphrase protected. ```shell ascli config genkey ${KEY_PAIR_PATH} 4096 ``` To display the public key of a private key: ```shell ascli config pubkey @file:${KEY_PAIR_PATH} ``` #### `ssh-keygen` Both private and public keys are generated, option `-N` is for passphrase. ```shell ssh-keygen -t rsa -b 4096 -m PEM -N '' -f ${KEY_PAIR_PATH} ``` #### `openssl` To generate a key pair with a passphrase, the following can be used on any system: ```shell openssl genrsa -passout pass:<PASSPHRASE> -out ${KEY_PAIR_PATH} 4096 openssl rsa -pubout -in ${KEY_PAIR_PATH} -out ${KEY_PAIR_PATH}.pub ``` `openssl` is sometimes compiled to support option `-nodes` (no DES, that is, no passphrase, for example, on macOS). In that case, add option `-nodes` instead of `-passout pass:<PASSPHRASE>` to generate a key without passphrase. If option `-nodes` is not available, the passphrase can be removed using this method: ```shell openssl rsa -passin pass:<PASSPHRASE> -in ${KEY_PAIR_PATH} -out ${KEY_PAIR_PATH}.no_des mv ${KEY_PAIR_PATH}.no_des ${KEY_PAIR_PATH} ``` To change (or add) the passphrase for a key: ```shell openssl rsa -des3 -in ${KEY_PAIR_PATH} -out ${KEY_PAIR_PATH}.with_des mv ${KEY_PAIR_PATH}.with_des ${KEY_PAIR_PATH} ``` #### Using an application to generate a key pair Many applications are available, including on the internet, to generate key pairs. For example: <https://cryptotools.net/rsagen> > [!WARNING] > Private keys are sensitive information, and shall be kept secret (like a password), so using online tools is risky. ### Web service `ascli` is a command-line tool: it runs in a shell, with the credentials and presets of the user who starts it, and it assumes that only this user uses it. It is not intended to be a public-facing web server. When commands come from another program, for example an MCP client, see [Security Level](#security-level). Some commands start a web server, for a specific client: | Command | Client | Protection required when `url` is not loopback | |---|---|---| | `mcp server`, `http` transport | MCP client | `auth_token`, see [MCP Security](#security) | | `faspex5 postprocessing` | Faspex 5 web hooks | `secret_token` or `allowed_ips`, see [Faspex 4-style post-processing](#faspex-5-faspex-4-style-post-processing) | | `faspex5 gateway`, `aoc gateway` | Faspex 4 client applications | `username` and `password`, see [Faspex 4 Gateway](#faspex-5-faspex-4-gateway) | | `node emulator` | Node API clients, for tests | `username` and `password`, see [Node emulator](#node-emulator) | By default, the server listens on a loopback address: `localhost`, an address of `127.0.0.0/8`, or `::1`. It can be reached only from the same host. > [!WARNING] > Loopback restricts the host, not the user: on a host shared with other users, they can also reach the server. > On such a host, also set the protection of the command. When the host of `url` is not a loopback address, for example `0.0.0.0` (all interfaces), the server can be reached from the network: the command refuses to start without its protection. On a network, use HTTPS (`https://` in `url`) or a reverse proxy with TLS: with HTTP, passwords and tokens are sent in clear. Other features listen on a loopback address only, to receive input from the browser of the user: - [Web login flow](#web-login-flow): the result of the login, on `redirect_uri`. - `@secret:`, with option `ui` set to `graphical` (see [Secret Vault](#secret-vault)), or with `mcp server` and the `stdio` transport (see [Secrets entered in a web page](#secrets-entered-in-a-web-page)): the secret entered by the user. The server serves HTTP, or HTTPS with a certificate. The following parameters are supported: | Parameter | Type | Default | Description | |-------------------|----------|-------------------------|------------------------------------------------------------------| | `url` | `String` | `http://localhost:8080` | Base URL on which requests are received; a path can be included. | <!-- markdownlint-disable-line --> | `cert` | `String` | - | (HTTPS) Path to certificate file (with ext. `.pfx` or `.p12` for `PKCS12`). | | `key` | `String` | - | (HTTPS) Path to private key file (PEM), or passphrase for `PKCS12`. | | `chain` | `String` | - | (HTTPS) Path to certificate chain (PEM only). | Parameter `url` (base URL) defines: - If `http` or `https` is used - The local port number (default 443 for HTTPS, 80 for HTTP) - The **base path**, that is, the path under which requests are received (useful for routing when a reverse proxy is used). A web page open in a browser can send requests to a local web server, also on `localhost`. To refuse them, the server checks each request: - When `url` is a loopback address (`localhost`, `127.0.0.0/8`, `::1`), header `Host` shall be a loopback name or address, else HTTP 403: a host name that resolves to `127.0.0.1` is refused (DNS rebinding). - Header `Origin`, sent by browsers, shall be the origin of the server, else HTTP 403. - A JSON body shall have `Content-Type: application/json`, else HTTP 415: a web page cannot send it to another site without the agreement of that site. Clients that are not browsers send no `Origin`, and the `Host` of the URL they call. They shall send the type of a JSON body: for example, with `curl`, add `-H 'Content-Type: application/json'`. ### Image and video thumbnails `ascli` can display thumbnails for images and videos in the **terminal**, using [iTerm2 inline image protocol](https://iterm2.com/documentation-images.html) or through colorized text. This feature can be used: - In the `thumbnail` command of `node` when using **gen4/access key** API. - When using the `show` command of `preview` plugin. - `coffee` and `image` commands of `config` plugin. - Any displayed value which is a URL to image can be displayed with option `format` set to `image` The following options can be specified in option `--out.img`: | Field | Type | Description | |------------|---------|----------------------------------------------------------------------------------| | `double` | `Bool` | Use double vertical resolution via half-block characters (Unicode). Default is `true` if the terminal supports Unicode (see `utf8`). | | `font_ratio` | `Number` | Terminal font aspect ratio (height divided by width).<br/>Default: `2.3`. | | `reserve` | `Integer` | Number of terminal lines reserved for non-image output.<br/>Default: `3`. | | `text` | `Bool` | Display the image as colorized text instead of using the iTerm2 inline protocol.<br/>Default: `false`. | Examples: - Display image as colorized text (requires `rmagick`) ```shell ascli config image https://eudemo.asperademo.com/wallpaper.jpg --ui=text --out.img.text=true ``` - Display image from byte stream as image in terminal (requires iTerm2-compatible terminal) ```shell curl -so - https://eudemo.asperademo.com/wallpaper.jpg | ascli config image @stdin:bin ``` - Display image from file (requires iTerm2-compatible terminal) ```shell ascli config image @stdin:bin < A-team.jpg ``` ### Graphical Interactions: Browser and Text Editor Some actions may require the use of a graphical tool: - A browser for Aspera on Cloud authentication (web auth method) - A browser to enter a secret requested with `@secret:` - A text editor for editing the configuration file By default, `ascli` assumes that a graphical environment is available on Windows and macOS; on other systems, it relies on the presence of the `DISPLAY` environment variable. The composite option `ui` (use `--ui=help` to display this schema) selects the mode, the way a web login is received, and the browser command: | Field | Type | Description | |----------|---------------|----------------------------------------------------------------------------------| | `command` | `Array[String]` | Command that opens a URL, instead of the system default. First item is the executable (full path, or name of an executable found in `PATH`), then its arguments. The URL is added as last argument. When set, it is used to open URLs, whatever the `mode`. | | `mode` | `Enum[String]` | - `auto`: `browser` on Windows and macOS, or if env var `DISPLAY` is set, else `terminal`.<br/>- `terminal`: URLs to open are displayed on terminal, and secrets are entered on terminal.<br/>- `browser`: a browser opens URLs, and secrets are entered on a web page.<br/>`text` is an alias for `terminal`, and `graphical` is an alias for `browser`.<br/>Files to edit are always opened with a text editor, whatever the `mode`.<br/>Allowed values: `auto`, `terminal`, `browser`, `text`, `graphical`.<br/>Default: `auto`. | | `web_auth` | `Enum[String]` | How the result of a web login (`--auth=web`, SSO) is received.<br/>- `port`: `ascli` listens on the loopback port of `redirect_uri`, the browser is redirected to it.<br/>- `paste`: no port is opened. After login, the browser shows a connection error on `redirect_uri`: copy the full URL from the address bar of the browser and paste it in the terminal. Use it when the browser cannot reach the host that runs `ascli` and a SSH tunnel is not possible. For `shares`, Shares displays a code (a long text of letters, digits, `-` and `_`) instead: paste it in the terminal.<br/>Allowed values: `port`, `paste`.<br/>Default: `port`. | A plain value is the `mode`: - `--ui=browser` (or `graphical`) forces a graphical environment: a browser is opened for URLs or secrets. - `--ui=terminal` (or `text`) forces a text environment: the URL to open is displayed on terminal, and secrets are entered on terminal. - `--ui=auto` selects the mode from the environment (default). The browser is opened, by priority, with: `--ui.command`, or the system default. The system default is opened with the following command: | OS | Command | |---------|------------------------| | macOS | `open <url>` | | Windows | `start explorer "<url>"` | | Linux | `xdg-open <url>` | To edit a file (for example `config open`), whatever the `mode`, the editor is, by priority: the command in the environment variable `EDITOR`, else `notepad.exe` on Windows, else the system default above (or `--ui.command` if defined). When a command is defined, it is used to open URLs whatever the `mode`. The command is an array: the URL is appended as last argument. The executable is either a full path, or the name of an executable found in the `PATH`. For example, to use Firefox: ```bash ascli config echo @secret: --ui=browser --ui.command=@json:'["firefox","--private-window"]' ``` ### Web login flow Authentication methods that need a user login in a browser (`--auth=web`, including for Shares) all work the same way: `ascli` listens on a local port, the browser is redirected to it after login. ![Web login flow: `ascli` listens on a loopback port and receives the result of the login from the browser](login-flow.svg) 1. `ascli` generates a random `state` and starts a local HTTP server on the loopback address of `redirect_uri` (default: `http://localhost:12345`). 2. `ascli` opens the login URL in the browser (see [Graphical Interactions](#graphical-interactions-browser-and-text-editor)): the authorization endpoint with `redirect_uri` and `state` for `web`, or, for Shares, the SAML login page with `RelayState`. If no graphical environment is available, the URL is displayed, to be opened on any browser of the same host. 3. The user logs in on the server, possibly redirected to an identity provider. 4. The server redirects the browser to `redirect_uri`, with the OAuth code (`web`), or, for Shares, the tokens, and the `state`. 5. The browser sends this request to `ascli`, which checks that `state` is the one it generated, and stops listening. 6. `ascli` requests the token to the server: exchange of the code (`web`) or, for Shares, of the refresh token, and saves it in the cache. Next calls use the cached token, renewed if needed without a browser. The code or tokens never go to another host than the local one: the host of `redirect_uri` shall be loopback. #### Option `ui.web_auth` Option `ui.web_auth` (part of the composite option `ui`, see [Graphical Interactions](#graphical-interactions-browser-and-text-editor)) selects how the result of a web login is received: | Value | Local port | Use when | |------------------|------------|----------| | `port` (default) | Yes: `ascli` listens on the loopback address of `redirect_uri` | The browser runs on the same system as `ascli`, or an SSH tunnel is possible. | | `paste` | No: `redirect_uri` is only used in the login URL | The browser cannot reach the system that runs `ascli`, and no tunnel is possible. | With `paste`, after login the user gives the result to `ascli` through the terminal (nothing is displayed, as it contains a code or tokens): - the code displayed by Shares (a long text of letters, digits, `-` and `_`), or the code shown by `ascli config web_login` on the workstation, - or the full URL (or only the query) from the address bar of the browser, which shows a connection error on `redirect_uri`: this is expected. In all cases the `state` is checked. Details are in the following sections. Example: ```shell ascli <PLUGIN> <COMMAND> --ui=terminal --ui.web_auth=paste ``` #### Web login when the browser is on another system The browser sends the result of the login to `redirect_uri`, on the loopback address of the system where the browser runs. If `ascli` runs on another system than the browser (for example, `ascli` is installed on a server, and the user is connected with SSH from a workstation), the redirection does not reach `ascli`. To solve this, forward the port of `redirect_uri` from the workstation to the server with an SSH tunnel, and open the login URL in the browser of the workstation: 1. On the workstation, open the SSH session with a tunnel (same port on both sides, default `12345`): ```shell ssh -L 127.0.0.1:12345:127.0.0.1:12345 <SERVER_USER>@<SERVER_HOST> ``` 2. In this session, run the command with `--ui=terminal`, so that the login URL is displayed instead of opening a browser on the server (this is the default if the server has no graphical environment): ```shell ascli <PLUGIN> <COMMAND> --ui=terminal ``` 3. Copy the displayed URL in the browser of the workstation, and log in. 4. The browser is redirected to `http://localhost:12345`: the tunnel sends the request to `ascli` on the server, and the flow continues as described above. The tunnel only exists as long as the SSH session is open. If no tunnel is possible (for example, a jump server between the workstation and the server), use `--ui.web_auth=paste`: `ascli` then opens no local port, and the result of the login is given through the terminal. There are two ways, depending on whether `ascli` is also installed on the workstation. With `--ui.web_auth=port` (default), in an SSH session with `--ui=terminal`, a warning reminds this option. ##### With `ascli` on the workstation `ascli` on the workstation opens the login page, receives the redirection of the browser, and shows a code to paste in the terminal of the server. Nothing else is needed on the workstation: no configuration, no preset, no access to the server of the application. The exchange of the code for a token is done by `ascli` on the server: the secret of the client does not leave it. 1. In the SSH session, run the command with `--ui=terminal --ui.web_auth=paste`: ```shell ascli <PLUGIN> <COMMAND> --ui=terminal --ui.web_auth=paste ``` 2. `ascli` displays a message that proposes the two ways to log in, with the login URL for the first one, and a command to run on the workstation for the second one: it contains a code made of letters, digits, `-` and `_`. Then `ascli` waits: ```text USER ACTION: To continue, choose one of these two ways: 1. With any browser (ascli is not needed on the workstation): - Open this URL in a browser and log in: https://... - The browser shows a connection error on http://localhost:12345: this is expected. - Copy the full URL from the address bar of the browser, and paste it in the prompt below. 2. If ascli is installed on the workstation that has the browser: - Run there: ascli config web_login <CODE> - Log in in the page that opens. - Copy the code shown by the browser, and paste it in the prompt below. ``` The command is displayed only if `ascli` does not start the browser itself (`--ui=terminal` without `--ui.command`). If the browser is started by `ascli` (`--ui=graphical`, or `--ui.command`), it is on the same host as `ascli`, and the redirection may be immediate (user already logged in): only the steps to log in, copy the URL from the browser and paste it are displayed. 3. On the workstation, run this command. The first time for a login host, `ascli` displays the host of the login page, and asks to trust it (`yes`): it opens a URL that comes from another system. Trusted hosts are saved; without a terminal, an unknown host is refused. `ascli` then opens the browser on the login page and listens on `redirect_uri`. 4. Log in. The browser shows a page "Login Complete" with the code to paste, hidden by default: - The button **Copy** puts the code in the clipboard. - The **eye** button shows or hides the code, to select it manually. 5. Paste the code in the terminal of the server (nothing is displayed). `ascli` on the server checks the `state`, and the flow continues as described above, from step 6. The command on the workstation: - accepts only a login page in `https`, and a `redirect_uri` on a loopback address; the host shown for confirmation is the one of the URL, not a label given in the code; - is available only at security level `low`; - ends once the browser has received the page with the code; the port of `redirect_uri` shall be free on the workstation (the URL is registered for the login and cannot be changed); - gives to the server only the parameters it needs (the `code` and the `state`). The codes are compressed and encrypted with a key that is not secret (derived from a constant and the current time, in slots of 15 minutes): they are hard to read and are valid for 15 to 30 minutes (the clocks of both computers must be correct), but they are **not confidential**: do not keep them, nor send them. The code shown by Shares also uses the random `state` of the login in its key. The codes are authenticated: a code that is incomplete, mistyped or modified is detected. Spaces and line breaks are ignored (letter case matters): a code displayed on several lines can be pasted. ##### Without `ascli` on the workstation If `ascli` is not installed on the workstation, the URL of the error page is used: 1. In the SSH session, run the command with `--ui=terminal --ui.web_auth=paste`. 2. Copy the displayed login URL in the browser of the workstation, and log in. 3. The browser is redirected to `redirect_uri` (for example `http://localhost:12345/?code=...&state=...`) and shows a connection error: this is expected. 4. Copy the full URL from the address bar of the browser, and paste it in the terminal (nothing is displayed, as it contains a code or tokens). The query alone (`code=...&state=...`) is also accepted. The `state` is mandatory: it is checked like in the standard flow, then the flow continues from step 6. Both ways accept, at the prompt, the code, the URL or the query. If standard input is not a terminal, the line is read from it. The size of a pasted line may be limited by the terminal: the code of the reply contains only what is needed. Notes: - For Shares with `--auth=web`, `paste` is simpler: Shares itself displays the code, so neither the tunnel, nor `ascli config web_login`, nor the URL of the error page is used (see [Authentication of SAML users](#authentication-of-saml-users)). - The port must be free on the workstation. To use another port, change it in the tunnel and in `--redirect-uri`, for example `--redirect-uri=http://localhost:23456` and `-L 127.0.0.1:23456:127.0.0.1:23456`. For Aspera on Cloud, this `redirect_uri` shall also be declared in the API client (see [API Client Registration](#api-client-registration)). - The tunnel listens on `127.0.0.1`: if the browser resolves `localhost` to `::1`, use `--redirect-uri=http://127.0.0.1:12345`. - On Windows, `ssh` is the OpenSSH client provided with the system; with PuTTY, add a local port forwarding (Connection → SSH → Tunnels) from `127.0.0.1:12345` to `127.0.0.1:12345`. - Another solution is X11 forwarding: run an X server on the workstation (for example `VcXsrv` or X410 on Windows) and connect with `ssh -X`. The browser then runs on the server and is displayed on the workstation, so no tunnel is needed. It is heavier (X server to install, slower display), and the browser is the one of the server. - Without interaction (automation), prefer a method that needs no browser, such as `--auth=jwt` for Aspera on Cloud. ### Logging, Debugging The gem is equipped with traces, mainly for debugging and learning APIs. The composite option `log` groups all logging parameters (use `--log=help` to display this schema): | Field | Type | Description | |---------|--------------|---------------------------------------------------------------------------| | `format` | `Enum[String]` | Log line formatter. Use `@ruby:` for a custom lambda.<br/>Allowed values: `default`, `caller`, `standard`.<br/>Default: `default`. | | `level` | `Enum[String]` | Minimum log level displayed.<br/>Allowed values: `trace2`, `trace1`, `debug`, `info`, `warn`, `error`, `fatal`, `unknown`.<br/>Default: `warn`. | | `secrets` | `Bool` | Show passwords and secrets in logs.<br/>Default: `false`. | | `type` | `Enum[String]` | Log output destination.<br/>Allowed values: `stderr`, `stdout`, `syslog`.<br/>Default: `stderr`. | > [!TIP] > Individual options `log_level`, `log_format`, `logger`, and `log_secrets` are also available as aliases for `log.level`, `log.format`, `log.type`, and `log.secrets` respectively. #### `log_level` and `log_secrets` To increase debug level, use option `log_level` (for example, using command line `--log-level=xx`, env var `ASCLI_LOG_LEVEL`, or an [Option Preset](#option-preset)). > [!NOTE] > When using the `direct` agent (`ascp`), additional transfer logs from `ascp` can be activated using `ascp` options and `ascp_args`, see [`direct`](#agent-direct). By default, passwords and secrets are redacted from logs. Set option `log_secrets` to `yes` to include secrets in logs. #### `log_format` Option `log_format` (alias for `log.format`) supports a few pre-defined formatters or a custom one using `@ruby:`. A custom formatter is a lambda that takes 4 arguments; see [Ruby Formatter](https://github.com/ruby/logger/blob/master/lib/logger/formatter.rb): `severity`, `time`, `progname`, `msg`. The default formatter is: ```ruby ->(s, _d, _p, m){"#{s[0..2]}#{s[-1]} #{m}\n"} ``` #### Logging examples - Display debugging log on `stdout`: ```shell ascli config preset overview --log-level=debug --logger=stdout ``` Or equivalently using dot-path notation: ```shell ascli config preset overview --log.level=debug --log.type=stdout ``` - Log errors to `syslog`: ```shell ascli config preset overview --log-level=error --logger=syslog ``` Or using the composite option in a preset: ```yaml log: level: error type: syslog ``` > [!NOTE] > When `ascli` is used interactively in a shell, the shell itself will usually log executed commands in the history file (`history | grep ascli`). ### Learning Aspera Product APIs (REST) `ascli` uses mainly REST APIs to interact with Aspera applications. To get traces of execution, with dump of API calls, use argument: `--log-level=debug`. To display HTTP/S traffic set option `log_level` to `trace2`: `--log-level=trace2`. It will display the exact content of HTTP requests and responses. ### HTTP socket parameters To ignore SSL certificate for **any** address/port, use option: `insecure`, that is, `--insecure=yes`. To ignore SSL certificate for a list of specific address/port, use option `ignore_certificate`, set to a URL or an `Array` of URLs for which the certificate is ignored (only the address and port are matched), for example, `--ignore-certificate=https://127.0.0.1:9092` or `--ignore-certificate=@list:,https://127.0.0.1:9092,https://127.0.0.1:9093` > [!NOTE] > Ignoring certificate also applies to `ascp` WSS. Ignoring a certificate is **not recommended**. It is preferable to add the certificate to the trusted store. When certificate validation is skipped, a warning is displayed. To disable this warning, set the option `warn_insecure` to `no`. HTTP connection parameters (not `ascp` WSS) can be adjusted using option `http_options`: | Field | Type | Description | |-------------------------|---------------|----------------------------------------------------------------------------------| | `download_partial_suffix` | `String` | File suffix appended to the target filename while a download is in progress. (Rest)<br/>Default: `.http_partial`. | | `keep_alive_timeout` | `Number` | Maximum time in seconds to keep an idle persistent connection open. (Net::HTTP)<br/>Default: `2`. | | `open_timeout` | `Number` | Maximum time in seconds to wait for a connection to be established. (Net::HTTP)<br/>Default: `60`. | | `read_timeout` | `Number` | Maximum time in seconds to wait for data to be read from the server. (Net::HTTP)<br/>Default: `60`. | | `retry_max` | `Integer` | Maximum number of retry attempts. (Rest)<br/>Default: `1`. | | `retry_on_error` | `Bool` | Retry the request on any HTTP error response. (Rest)<br/>Default: `false`. | | `retry_on_timeout` | `Bool` | Retry the request on connection timeout. (Rest)<br/>Default: `true`. | | `retry_on_unavailable` | `Bool` | Retry the request on HTTP 503 Service Unavailable. (Rest)<br/>Default: `true`. | | `retry_sleep` | `Integer` | Delay in seconds between retry attempts. (Rest)<br/>Default: `4`. | | `ssl_options` | `Array[String]` | List of OpenSSL option flags to set on the HTTP session (e.g. `IGNORE_UNEXPECTED_EOF`). (SSL) | | `token_cache_max_age` | `Integer` | Maximum age in seconds of a cached OAuth token before it is discarded. (OAuth)<br/>Default: `1800`. | | `token_refresh_threshold` | `Integer` | Remaining token lifetime in seconds below which the token is proactively refreshed. (OAuth)<br/>Default: `120`. | | `user_agent` | `String` | Value sent in the `User-Agent` HTTP request header. (Rest) | | `write_timeout` | `Number` | Maximum time in seconds to wait for data to be written to the server. (Net::HTTP)<br/>Default: `60`. | Time values are set in **seconds** and can be of type either `Integer` or `Float`. Any additional key is forwarded as a setter to the `Net::HTTP` session. For a full list, refer to the Ruby library: [`Net::HTTP`](https://github.com/ruby/net-http/blob/master/lib/net/http.rb). Like any other option, those can be set either on command line, or in configuration file, either in a global preset or server-specific one. Example: ```shell ascli aoc admin package list --http-options=@json:'{"read_timeout":10.0}' ``` The option `ssl_options` corresponds to a list of options as listed in `man SSL_CTX_set_options`. The default initial value is the default of Ruby as specified in `openssl/ssl.rb`. Each option can be specified as a `String` with the same name as in the OpenSSL library by removing the prefix: `SSL_OP_`, or an `Integer` (for example, `0` resets to no option). If the name appears in the list, the option is set. If the name appears in the list prefixed with a hyphen (`-`), the option is cleared. For example to enable option `SSL_OP_CIPHER_SERVER_PREFERENCE`, add it to the list as `CIPHER_SERVER_PREFERENCE`. To disable option `SSL_OP_SAFARI_ECDHE_ECDSA_BUG`, add it as `-SAFARI_ECDHE_ECDSA_BUG`. Example: ```json {"ssl_options":["CIPHER_SERVER_PREFERENCE","-SAFARI_ECDHE_ECDSA_BUG"]} ``` ### Proxy There are several types of network connections, each of them uses a different mechanism to define a (forward) **proxy**: - REST calls (APIs) and HTTP Gateway - `ascp` WSS and Legacy Aspera HTTP/S Fallback - `ascp` SSH and UDP (Aspera FASP) See the following sections. #### Proxy for REST and HTTP Gateway REST API calls and transfers based on HTTP Gateway both use Ruby's `Net::HTTP` class. See [Ruby find proxy](https://rubyapi.org/4.0/o/uri/generic#method-i-find_proxy). When Ruby HTTP is used, there are two possibilities to define an HTTP proxy to be used. The `http_proxy` environment variable (**lower case**) can be set to the **URL** of the proxy (with optional credentials). Syntax is: `(http|https)://[user:password@]host:port`. For example, `http://myproxy.org.net:3128`. > [!NOTE] > Ruby expects a URL and `myproxy.org.net:3128` alone is **not** valid. Credentials for proxy are optional but can also be specified: ```shell export http_proxy=http://<USERNAME>:<PASSWORD>@proxy.example.com:3128 ``` Option `http_proxy` does the same (set env var) but on command line: ```shell ascli --http-proxy=http://<USERNAME>:<PASSWORD>@host:port ... ``` Alternatively, the `fpac` option (function for proxy auto config) can be set to a [Proxy Auto Configuration (PAC)](https://en.wikipedia.org/wiki/Proxy_auto-config) JavaScript value. > [!NOTE] > Proxy credentials are not supported in PAC files. To read the script from a URL (`http:`, `https:` and `file:`), use prefix: `@uri:`. A minimal script can be specified to define the use of a local proxy: ```shell ascli --fpac='function FindProxyForURL(url, host){return "PROXY localhost:3128"}' ... ``` The result of a PAC file can be tested with command: `config proxy_check`. Example, using command line option: ```shell ascli config proxy_check --fpac='function FindProxyForURL(url, host) {return "PROXY proxy.example.com:3128;DIRECT";}' http://example.com ``` ```text proxy://proxy.example.com:3128 ``` ```shell ascli config proxy_check --fpac=@file:./proxy.pac http://www.example.com ``` ```text proxy://proxy.example.com:8080 ``` ```shell ascli config proxy_check --fpac=@uri:http://server/proxy.pac http://www.example.com ``` ```text proxy://proxy.example.com:8080 ``` If the proxy found with the PAC requires credentials, then use option `proxy_credentials` with username and password provided as an `Array`: ```shell ascli --proxy-credentials=@json:'["<USERNAME>","<PASSWORD>"]' ... ``` ```shell ascli --proxy-credentials=@list::<USERNAME>:<PASSWORD> ... ``` #### Proxy for Legacy Aspera HTTP/S Fallback Only supported with the `direct` agent: To specify a proxy for legacy HTTP fallback, use `ascp` native option `-x` and `ascp_args`: `--transfer=@json:'{"ascp_args":["-x","<PROXY_URL>"]}'`. #### FASP proxy (forward) for transfers To specify a FASP proxy (forward), set the [**transfer-spec**](#transfer-specification) parameter: `proxy` (only supported with the `direct` agent). For example, for an Aspera forward proxy not encrypted (HTTP) without authentication running on port 9091, the option would be: ```shell --ts=@json:'{"proxy":"dnat://proxy.example.org:9091"}' ``` Or, alternatively, (prefer transfer spec like above, generally): ```shell --transfer=@json:'{"ascp_args":["--proxy","dnat://proxy.example.org:9091"]}' ``` ### FASP configuration `ascli` uses one of the transfer agents to execute transfers. By default, it uses the `direct` agent, which is a local `ascp`. Nevertheless, `ascli` does not come with `ascp` installed. This is the reason why it is advised to install the Aspera Transfer Daemon during installation (`ascli transferd install`). By default, `ascli` uses the `ascp` binary found in **well known locations**, that is, typical Aspera product installation paths. The `config` plugin allows finding and specifying the location of `ascp`. It provides the following commands for `ascp` sub-command: - `show` : Show the path of `ascp` used - `info` : Show information on `ascp` and its environment - `install` : Install the Transfer SDK (deprecated, same as `transferd install`) - `spec` : List transfer spec parameters supported by `ascp` - `schema` : Show the JSON schema of the transfer spec - `errors` : List known `ascp` errors and whether they are retry-able - `products list` : List Aspera transfer products available locally #### Selection of `ascp` location for [`direct`](#agent-direct) agent Option: `sdk_folder` is used to specify the location of `ascp`. By default, `ascli` uses `ascp` from the Transfer SDK installed in its configuration folder (see `config ascp install`). To override and use an alternate `ascp` path, use option `sdk_folder` (`--sdk-folder=`). For a permanent change, set a global default. For example, `<INSTALL_DIR>` could be `~/my_install_dir` on Linux, or `C:\Users\admin\.aspera\ascli\sdk` on Windows. Using a POSIX shell: ```shell ascli config preset set GLOBAL sdk_folder <INSTALL_DIR> ``` ```text INFO Updated: global_common_defaults: sdk_folder <- <INSTALL_DIR> INFO Saving config file: /home/john/.aspera/ascli/config.yaml ``` If the path has spaces, read section: [Shell and Command line parsing](#command-line-parsing-special-characters). A special value `product:<PRODUCT_NAME>` can be used for option `sdk_folder`. It specifies to use `ascp` from the given product name. A special value for product name is `FIRST`, which means: use the first product found in the internal list. In that case, other files (SSH keys, `aspera.conf`, `transferd`) are still taken from the default SDK folder. Locally installed Aspera products can be listed with: ```shell ascli config ascp products list ``` ```text ╭───────────────────────────────────────┬──────────────────────────────────────╮ │ name │ app_root │ ╞═══════════════════════════════════════╪══════════════════════════════════════╡ │ IBM Aspera Transfer SDK │ /Users/laurent/.aspera/sdk │ │ IBM Aspera for Desktop │ /Applications/IBM Aspera.app │ │ IBM Aspera Connect │ /Applications/IBM Aspera Connect.app │ │ IBM Aspera High-Speed Transfer Server │ /Library/Aspera │ ╰───────────────────────────────────────┴──────────────────────────────────────╯ ``` To permanently use the `ascp` of a product: ```shell ascli config preset set GLOBAL sdk_folder 'product:IBM Aspera Connect' ``` ```text INFO Updated: default: config <- global_common_defaults INFO Updated: global_common_defaults: sdk_folder <- product:IBM Aspera Connect INFO Saving config file: /home/john/.aspera/ascli/config.yaml ``` To show the path of currently used `ascp`: ```shell ascli config ascp show ``` ```text /Users/laurent/.aspera/sdk/ascp ``` ```shell ascli config ascp info ``` ```text ╭─────────┬──────────────────────────────────────────────────╮ │ field │ value │ ╞═════════╪══════════════════════════════════════════════════╡ │ ascp │ /Users/john/.aspera/sdk/ascp │ ... ╰─────────┴──────────────────────────────────────────────────╯ ``` ### Transfer Clients: Agents Some actions on Aspera Applications lead to file transfers (upload and download) using the FASP protocol (`ascp`). Transfers will be executed by a transfer client, here called **Transfer Agent**. The following agents are supported and selected with option `transfer`: | `transfer` | location | Description of agent | |----------|----------|-------------------------------| | [`direct`](#agent-direct) | local | direct execution of `ascp` | | [`transferd`](#agent-transfer-daemon) | local | Aspera Transfer Daemon | | [`connect`](#agent-connect-client) | local | Aspera Connect Client (to be deprecated in 2027, use `desktop`) | | [`desktop`](#agent-desktop-client) | local | Aspera for Desktop | | [`node`](#agent-node-api) | remote | Aspera Transfer Node | | [`httpgw`](#agent-http-gateway) | remote | Aspera HTTP Gateway | The list of agents and their short identifier can be displayed with: ```shell ascli config agents list ``` To display the configurable parameters of a specific agent (with required fields, default values, enumerated values): ```shell ascli config agents parameters <agent name> ``` For example: ```shell ascli config agents parameters direct ascli config agents parameters node ``` To display the description of an agent: ```shell ascli config agents show <agent name> ``` > [!NOTE] > All transfer operations are seen from the point of view of the agent. > For example, an agent executing an **upload**, or **package send** operation will effectively push files to the related server from the system where the agent runs. All above agents (including `direct`) receive transfer parameters as a [**transfer-spec**](#transfer-specification). Parameters in transfer-spec can be modified with option `ts`. **Specific** options for agents are provided with option `transfer` (as a `Hash`, optionally with an `agent` key). > [!NOTE] > Parameters in `transfer` are specific for each agent type and are described in the agents respective sections. #### Asynchronous Transfer Mode Adding `asynchronous: true` to the `transfer` option makes `ascli` return immediately after submitting the transfer, without waiting for completion. The returned `job_id` can be used at any time to check progress: ```shell # Start a download and return immediately ascli server download /remote/big-file.bin --to-folder=/tmp \ '--transfer=@json:{"agent":"desktop","asynchronous":true}' # Poll status later ascli config agents jobs status <job_id> # List all pending/completed async transfers ascli config agents jobs list # Remove completed/failed entries ascli config agents jobs cleanup ``` > [!NOTE] > `config transfer` (`list`, `status`, `cleanup`) is deprecated after 4.28.0: use `config agents jobs`. It still works, with a warning. All transfer agents support asynchronous mode: | Agent | Status source | Persisted across restart | | ----------- | ----------------------------------------------------------------------------- | ------------------------ | | `desktop` | JSON-RPC `get_transfer` (auto-discovered URL) | Yes | | `node` | REST `ops/transfers/{id}` | Yes | | `connect` | REST `transfers/info/{id}` (auto-discovered URL) | Yes | | `transferd` | gRPC `monitor_transfers` | Yes | | `direct` | In-process thread state (re-queryable while process lives, e.g. MCP mode) | No: returns `unknown` after restart | | `httpgw` | In-process thread state (re-queryable while process lives, e.g. MCP mode) | No: returns `unknown` after restart | For `direct` and `httpgw`, the transfer runs as a Ruby thread inside the `ascli` process. The `job_id` is persisted on disk but the live thread state is only available as long as the same process is running. If the process is restarted, `config agents jobs status` returns `unknown` for those jobs. Parameters of the agent are persisted, except secrets (e.g. `password` of agent `node`): `config agents jobs status` takes them from the current `transfer` option, if it is for the same agent (e.g. same option on command line, or in a preset). > [!NOTE] > **`asynchronous` and MCP** - When `ascli` is used as an MCP server, an AI assistant calling > a transfer command may time out or cancel the request and retry, causing duplicate transfers. > Using `asynchronous: true` eliminates this risk: the transfer is submitted immediately, > the `job_id` is returned, and the AI can poll > `config agents jobs status <job_id>` to track progress without any risk of duplication. > This works with all agents, including `direct` and `httpgw`. #### Agent: Direct The `direct` agent directly executes a local `ascp` in `ascli`. This is the default agent for `ascli` (option `--transfer=direct`). `ascli` will locally search installed Aspera products, including SDK, and use `ascp` from that component. See [FASP](#fasp-configuration). ##### Agent: Direct: `transfer` The `transfer` option accepts the following optional parameters to control multi-session, Web Socket Session, Resume policy and add any argument to `ascp`: | Field | Type | Description | |----------------------|---------------|----------------------------------------------------------------------------------| | `ascp_args` | `Array[String]` | List of native `ascp` command-line arguments.<br/>Default: `[]`. | | `client_ssh_key` | `Enum[String]` | SSH key type to use for token-based transfers.<br/>Allowed values: `rsa`, `dsa_rsa`, `per_client`.<br/>Default: `rsa`. | | `file_list` | `Bool` | If `true`, source paths are written to a temp file passed to `ascp` via `--file-list` or `--file-pair-list`.<br/>If `false`, source paths are placed directly on the `ascp` command line.<br/>Default: `true`. | | `monitor` | `Bool` | Enable use of the `ascp` management port for transfer monitoring.<br/>Default: `true`. | | `multi_incr_udp` | `Bool` | Multi session - Increment UDP port for each session.<br/>If `true`, each session uses a different UDP port starting at `fasp_port` (default: 33001).<br/>If `false`, all sessions use the same `fasp_port` (or `ascp` default).<br/>Default: `true` on Windows, `false` on other platforms. | | `quiet` | `Bool` | Suppress the `ascp` progress bar display.<br/>If `false`, the progress bar of option `progress_bar` is not displayed, unless that option is set.<br/>Default: `true`. | | `resume` | `Hash` | Configuration for automatic transfer resume on interruption. | | `resume.iter_max` | `Integer` | Maximum number of retry attempts on error.<br/>Default: `7`. | | `resume.sleep_factor` | `Integer` | Multiplier applied to sleep duration between consecutive retry attempts.<br/>Default: `2`. | | `resume.sleep_initial` | `Integer` | Initial sleep duration (in seconds) before first retry.<br/>Default: `2`. | | `resume.sleep_max` | `Integer` | Maximum sleep duration (in seconds) between retry attempts.<br/>Default: `60`. | | `spawn_delay_sec` | `Number` | Multi session - Delay (in seconds) between starting each `ascp` session.<br/>Default: `2`. | | `spawn_timeout_sec` | `Number` | Multi session - Maximum time (in seconds) to verify that `ascp` is running.<br/>Default: `2`. | | `trusted_certs` | `Array[String]` | List of trusted certificate repositories. | | `wss` | `Bool` | Enable Web Socket Session when available.<br/>Default: `true`. | In case of transfer interruption, the agent will **resume** a transfer up to `iter_max` times. Sleep between iterations is given by the following formula where `iter_index` is the current iteration index, starting at 0: ```text min( sleep_max, sleep_initial * sleep_factor ^ iter_index ) ``` By default, the progress bar of `ascli` is displayed (see [Transfer progress bar](#transfer-progress-bar)): it aggregates all sessions of a multi-session transfer. To display the native progress bar of `ascp` instead, set parameter `quiet` to `false`: ```shell --transfer.quiet=false ``` In that case, the progress bar of `ascli` is not displayed, unless option `progress_bar` is set. For multi-session transfers, each `ascp` process displays its own progress bar. To skip usage of management port (which disables the progress bar of `ascli`), set option `monitor` to `false`. By default, Ruby's root CA store is used to validate any HTTPS endpoint used by `ascp` (for example, WSS). To use a custom certificate store, use the `trusted_certs` option of direct agent's option `transfer`. To use `ascp`'s default, use option: ```shell --transfer.trusted_certs=@none: ``` Some transfer errors are considered **retry-able** (for example, timeout) and others are not (for example, wrong password). The list of known protocol errors and retry level can be listed: ```shell ascli config ascp errors ``` Error codes of sessions (`errcode` in `ascp` logs) are a different numbering, listed with their category and the corresponding error above: ```shell ascli config ascp errors session ``` Examples: ```shell ascli ... --transfer.wss=true --transfer.resume.iter_max=20 ascli ... --transfer.spawn_delay_sec=2.5 --transfer.multi_incr_udp=false ``` Parameter `ascp_args` can also be used to activate logging using option `-L` of `ascp`. For example, to activate debug level 2 for `ascp` (`DD`), and display those logs on the terminal (`-`): ```shell --transfer=@json:'{"ascp_args":["-DDL-"]}' ``` This is useful to debug if a transfer fails. To store `ascp` logs in file `aspera-scp-transfer.log` in a folder, use `--transfer=@json:'{"ascp_args":["-L","/path/to/folder"]}'`. To analyze `ascp` log files (`ascp` 4.0 and later), give log files, or folders with log files, to `config ascp logs`: ```shell ascli config ascp logs summary /path/to/folder ascli config ascp logs sessions /path/to/folder/aspera-scp-transfer.log ``` | Command | Result | |---------------------|-----------------------------------------------------------------| | `summary` | Statistics of sessions, errors by category, maximum throughput, findings. | | `health` | Health check for monitoring in Nagios format (failed sessions and low efficiency over last N minutes). | | `sessions` | One row per session, with diagnostics. | | `diagnostics` | One row per diagnostic of a session: `id`, `severity` (`error`, `warning`, `info`), `message` and `recommendation` (action to take, when the diagnostic has one). | | `multi_sessions` | Multi-session transfers: sessions grouped by transfer, with aggregate rate. | | `snapshots` | Aggregate rates per period. | | `snapshot_sessions` | Rates of each session per period. | | `peers` | Sessions and errors by peer address. | | `transfers` | Transferred files. | | `licenses` | Licenses of local and peer systems. | | `overdrives` | Sessions with loss due to overdriving of network or storage. | | `records` | Log lines parsed into records: `samples`, `sessions` or `transfers` (see below). | | `fields` | Description of the fields of records: `samples`, `sessions` or `transfers`. | Log files compressed with gzip (e.g. rotations `aspera.log-20261101.gz`, `aspera-scp-transfer.1.log.gz`) are read directly, in a folder or given as argument; gzip is detected from the first bytes of the file, not from its name. Argument `-` reads the logs from the standard input, plain or compressed with gzip: ```shell ascli config ascp logs summary /var/log/aspera.log-20261101.gz zcat /var/log/aspera.log-2026*.gz | ascli config ascp logs summary - ``` To analyze the logs of an HSTS read over SSH, instead of files of the local machine, use command `logs` of the plugin [`server`](#analyze-ascp-logs-of-the-server). A file must contain `ascp` log lines (`FASP` messages, or lines of process `ascp` in syslog): in a folder, other files are ignored; a file given as argument that is not an `ascp` log is refused. When logs of both sides of a session are analyzed (client and server), each session includes the status, error and diagnostics of the other side (`peer_side`, `peer_diagnostics`). For example, to list only errors: ```shell ascli config ascp logs diagnostics /path/to/folder --select=@json:'{"severity":"error"}' ``` Besides the default columns, `sessions` rows have context columns to filter sessions (show them with `--fields=ALL`, or select with `--fields=session,product_name,transport,authentication`, or filter with `--select`/`--query`); a column is absent when the log does not give it: | Column | Value | |------------------|--------------------------------------------------------------------------------------------------------------------| | `product_name` | Product that wrote the log, from the line `<product> version <version>` (e.g. `Aspera High-Speed Transfer Server`, `Aspera Transfer SDK`, `Aspera Connect`), without `IBM`. | | `transport` | `ssh` if `Session Start` has `tcp_port`, `wss` if it has `wss_port`. | | `authentication` | `token` if `ascp` logs that `ASPERA_AUTH_TOKEN` is set, else `node` if it logs the storage of a node access key. Never the token, the access key or the storage root. | | `proxy` | Address of the proxy (`proxy_ip` of `Session Params`), absent if there is none. | | `fips` | `true` or `false` (`fips_enabled` of `Session Params`). | Diagnostics of a session include: category of error (e.g. `udp_setup`, `storage`, `authentication`) with a hint, retry of a failed session, parameters set by the server (cipher, resume policy, virtual link), efficiency and loss of control messages, decline of the rate during the transfer (`rate_decline`: second half of the samples below 70% of the first half, not evaluated if the peer changes the rate, if another transfer session starts meanwhile, or if the session failed), and for a sender, which limit of the receiver (network, disk, virtual link) or reading from storage limits the rate. Other diagnostics: no packet received for some seconds, by the receiver (`reception_interruption`) or by the sender, which gets no feedback of the receiver (`feedback_interruption`); overflow of the retransmission timer (`rex_timer_overflow`: the retransmission RTT is very large); maximum rate of the license of `ascp` below the target rate (`license_rate_limit`); license that expires within `license_expiry_days` (default `30`) of the start of the session, or has expired (`license_expiry`); and many small files transferred (`small_files`): at least `small_files_min_count` (default 100) files with a mean size below `small_files_max_mean` (default 1 MB); path MTU measured by `ascp` below `small_mtu_max` (default 1400, `ascp` measures 1492, 1006, 508 or 296), which suggests a VPN or a tunnel (`small_mtu`). Two configuration messages, which have no effect on the transfer, are not diagnostics but only appear in `findings` of `summary`: OpenSSL modules folder not found (`ossl-modules not found`), and a schedule of virtual link capacity that could not be registered (`unable to submit change event`). The `recommendation` of a diagnostic is only in the list `diagnostics` (not in `sessions`, where `diagnostics` is a list of messages, and not in `summary`): it tells what to check or change, with the `ascp` options and `aspera.conf` parameters concerned (e.g. UDP port and firewalls, target and minimum rate, storage write, SSH). Diagnostics without action (e.g. information, or cancelled by the user) have no `recommendation`; the diagnostic of the category of a failed session gives its hint in `message`. Option `ascp_logs` (dot-notation) sets `period` (snapshot period in minutes, default `1`), `zero_bytes` (include sessions without transferred bytes), `mtu` (when not found in logs), `resolve` (host names of peers), `timezone`, `health_minutes` (window in minutes for health check command, default `60`), `remote_max_size` (maximum total size in bytes of the log files read from a server with [`server logs`](#analyze-ascp-logs-of-the-server), default `268435456`), and `thresholds` (diagnostic thresholds, including `rate_decline_ratio`, `rate_decline_min_samples`, `rate_decline_delay`, `rate_decline_end_fraction`, `license_expiry_days`, `small_files_min_count`, `small_files_max_mean` and `small_mtu_max`), e.g. `--ascp-logs.period=5` or `--ascp-logs.thresholds.efficiency=95`. Times of log lines are read in the time zone of the analyst, and displayed in it. To read the log of another system without shifting it, set the time zone of the log with `timezone`: `UTC` or an offset `+HH:MM`, e.g. `--ascp-logs.timezone=+02:00`. Syslog lines (`Oct 25 00:02:20`) have no year: it is deduced from the modification date of the log file (the current date for the standard input), the log being before this date. So a log from December to January, analyzed in March, has its December lines in the previous year. Keep the modification date of the file when copying it (e.g. `cp -p`, `scp -p`). Command `records` gives the log lines parsed into records, with the field names of the log lines, to filter them with `--out.query` (JMESPath): | Record | Log lines | |-------------|-------------------------------------------------------------------------------------------------| | `samples` | `Receiver bl` or `Sender bl` line, with the `DS Qs` line that follows it in field `DS`. | | `sessions` | `Session Start`, `Session Params`, `Session Stop` and `Session Statistics` lines of a session, and lines of the process about it (see below). | | `transfers` | `Transfer Start` and `Transfer Stop` lines of a file. | A group of a log line becomes an object: for example, `rate t/m/c/n/vl/vr/r=...` becomes `rate.t`, `rate.m`, `rate.c`, etc. Values are numbers when the log value is a number: rates in bps, durations in ms, s or µs, and percentages without `%`. `Session Stop` and `Transfer Stop` have the fields of `Session Start` and `Transfer Start`, with their final values. When several log files are read, field `log_file` gives the log file of each record. Field `cookie` of `Session Params` is never output. A session record also has the fields of these lines of the process (`ascp`), when the log has them: | Field | Log line | |----------------------------------------------------------------|-------------------------------------------------------------------------------------------------------| | `fasp_version`, `license_max_rate`, `license_expires` | `Initializing FASP version <version>, license expires on <date>, max rate=<N> Mbps, ...`: `license_max_rate` (bps) and `license_expires` are absent if the license is unlimited or has no expiry date. `account no.` and `license no.` are never output. | | `pMTU`, `start_brtt` | `Measured pMTU: <N> Bytes, start_brtt: <N> ms`. | | `datagram_size`, `block_size`, `path_mtu` | `datagram size <N>B, block size <N>B, path MTU <N>B` (no `path_mtu` in old versions). | | `peer_rate_params` | List of `Peer requested rate params adopted` lines, in the order of the log: `time`, `target_rate`, `min_rate` and `transfer_policy` of each change. | For example, samples where the calculated rate is lower than the target rate, and sessions with loss of control messages from the receiver: ```shell ascli config ascp logs records samples /path/to/aspera-scp-transfer.log --out.query='[?rate.c < rate.t].{time: time, session: uuid, calculated: rate.c, network: rate.n, lost: bl2.l}' ascli config ascp logs records sessions /path/to/folder --out.query='[?rcvr_ctl.lost_pct > `1`]' ``` Command `fields` gives the type and description (with unit) of each field of a kind of record (`samples`, `sessions` or `transfers`), with its path as used in `--out.query` (e.g. `rate.c`): ```shell ascli config ascp logs fields samples ascli config ascp logs fields sessions ascli config ascp logs fields transfers ``` > [!NOTE] > When transfer agent [`direct`](#agent-direct) is used, the list of files to transfer is provided to `ascp` using either `--file-list` or `--file-pair-list` and a temp file list, unless `--file-list` or `--file-pair-list` is already provided via `transfer` parameter `ascp_args`. > To place source paths directly on the `ascp` command line instead of using a temp file, set `file_list` to `false` in `transfer`: `--transfer.file_list=false`. In addition to standard methods described in section [File List](#list-of-files-for-transfers), it is also possible to specify the list of files using `transfer` parameter `ascp_args`: ```shell --sources=@ts --transfer=@json:'{"ascp_args":["--file-list","myfilelist"]}' ``` > [!NOTE] > File lists are shown here; similar options exist for file pair lists. > [!NOTE] > This method avoids the creation of a copy of the file list: if the standard options `--sources=@lines:@file:... --src-type=...` are used, then the file list is read and parsed, and a new file list is created in a temporary folder. > [!NOTE] > This method has limitations: it applies **only** to the [`direct`](#agent-direct) transfer agent (that is, local `ascp`) and not to Aspera on Cloud. ##### Agent: Direct: Management messages By default, `ascli` gets notifications from `ascp` on its management port. This can be disabled with parameter: `monitor=false` of `transfer`. It is also possible to send messages to `ascp` using this management port. A typical use is to change the target rate of a running transfer. The communication is done through a JSON file that shall be created in `ascli`'s config folder as displayed with: ```shell ascli config folder ``` The name of the file shall be: `send_<PID>`, where `<PID>` is the process ID of the running `ascli`. If there is only one `ascli` running, one can get the PID like this: ```shell ps -axo pid,command|grep ascli|grep -v grep|cut -f1 -d' ' ``` Example to change the target rate: ```shell echo '{"type":"RATE","rate":300000}' > ~/.aspera/ascli/send_67470 ``` When `ascli` detects this file, it uses it during a transfer and then deletes it. > [!NOTE] > The JSON's keys use **snake case**, that is, lower case with `_` as word separator. > The list of message `type` values can be found in `aspera/ascp/management.rb`: `OPERATIONS`. > The list of parameters is `PARAMETERS` (native names are capitalized, keys in the JSON file are in snake case). ##### Agent: Direct: `aspera.conf`: Virtual Links This agent supports a local configuration file, `aspera.conf`, where Virtual links can be configured. On a server (HSTS), the following commands can be used to set a global virtual link: ```shell asconfigurator -x 'set_trunk_data;id,1;trunk_name,in;trunk_capacity,100000;trunk_on,true' asconfigurator -x 'set_trunk_data;id,2;trunk_name,out;trunk_capacity,100000;trunk_on,true' asconfigurator -x 'set_node_data;transfer_in_bandwidth_aggregate_trunk_id,1' asconfigurator -x 'set_node_data;transfer_out_bandwidth_aggregate_trunk_id,2' ``` This command is not available on clients: instead, edit the file `aspera.conf` (its location is shown by `ascli config ascp info --fields=aspera_conf`) and modify the sections `default` and `trunks` like this for a global 100 Mbps virtual link (capacity is in kbps): ```xml <?xml version='1.0' encoding='UTF-8'?> <CONF version="2"> <default> <transfer> <in> <bandwidth> <aggregate> <trunk_id>1</trunk_id> </aggregate> </bandwidth> </in> <out> <bandwidth> <aggregate> <trunk_id>2</trunk_id> </aggregate> </bandwidth> </out> </transfer> </default> <trunks> <trunk> <id>1</id> <name>in</name> <on>true</on> <capacity> <schedule format="ranges">100000</schedule> </capacity> </trunk> <trunk> <id>2</id> <name>out</name> <capacity> <schedule format="ranges">100000</schedule> </capacity> <on>true</on> </trunk> </trunks> </CONF> ``` It is also possible to set a schedule with different time and days, for example for the value of `schedule`: ```text start=08 end=19 days=mon,tue,wed,thu capacity=900000;1000000 ``` ##### Agent: Direct: `aspera.conf`: File name with special characters By default, `ascp` replaces file system disallowed characters in transferred file names with `_`. On Windows, it concerns: `* : | < > " ' ?`. Replacing illegal characters in transferred file names is a built-in feature of Aspera transfers. This behavior can be customized with the `aspera.conf` configuration file. For example, to replace illegal character `|` with an underscore `_`: 1. First, locate the configuration file with: ```shell ascli config ascp info --fields=aspera_conf ``` Typically, it is located at `$HOME/.aspera/sdk/aspera.conf` 1. Edit this file, and add the following line inside the XML section `CONF.default.file_system`: ```xml <replace_illegal_chars>_|</replace_illegal_chars> ``` The result should look like this: ```xml <?xml version='1.0' encoding='UTF-8'?> <CONF version="2"> <default> <file_system> <resume_suffix>.aspera-ckpt</resume_suffix> <partial_file_suffix>.partial</partial_file_suffix> <replace_illegal_chars>_|</replace_illegal_chars> </file_system> </default> </CONF> ``` According to the [documentation](https://www.ibm.com/docs/en/ahts/4.4.x?topic=reference-user-group-default-configurations), the parameter works as follows: - The first character in the value is the replacement character. - All other characters listed after it are the **illegal** ones to be replaced. So in this example: `|` will be replaced with `_`. If the mounted target file system is **Windows** but `ascli` runs on Linux, there are several additional illegal characters to handle. In that case, set the parameter as shown below. XML special characters such as `<`, `>`, `"` and `'` must be escaped properly: ```xml <replace_illegal_chars>_*:|<>"'?</replace_illegal_chars> ``` In this example: - `_` is the replacement character. - The list of characters that will be replaced is: `*:|<>"'?` So, for example: - `report|final?.txt` becomes `report_final_.txt` - `data*backup"2025".csv` becomes `data_backup_2025_.csv` #### Agent: Connect Client > **Note:** IBM Aspera Connect will be deprecated in 2027. > Use [IBM Aspera for Desktop](#agent-desktop-client) instead: `--transfer=desktop`. > `ascli` displays a warning when the `connect` agent is used. By specifying option: `--transfer=connect`, `ascli` will start transfers using the locally installed **IBM Aspera Connect Client**. There are no specific parameters for option `transfer`. #### Agent: Desktop Client By specifying option: `--transfer=desktop`, `ascli` will start transfers using the locally installed **IBM Aspera Desktop Client**. There are no specific parameters for option `transfer`. #### Agent: Node API By specifying option: `--transfer=node`, `ascli` starts transfers in an Aspera Transfer Server using the Node API, either on a local or remote node. This is especially useful for direct node-to-node transfers. Parameters provided in option `transfer` are: | Field | Type | Description | |----------|--------|---------------------------------------------------------------| | `password` | `String` | Password, secret, or bearer token.<br/>Required. | | `root_id` | `String` | Root file ID. Mandatory only when `password` is a bearer token. | | `url` | `String` | URL of the Node API.<br/>Required. | | `username` | `String` | Node API user or access key.<br/>Required. | Like any other option, `transfer` can get its value from a pre-configured [Option Preset](#option-preset): ```shell --transfer=@preset:<PRESET_NAME> --transfer.agent=node ``` It can also directly use the [Extended Value](#extended-value-syntax) syntax: ```shell --transfer=@json:'{"agent":"node","url":"https://...","username":"<USERNAME>","password":"<PASSWORD>"}' ``` If `transfer` is not specified and a default node has been configured (name in `node` for section `default`) then this node is used by default. If the `password` value begins with `Bearer` then the `username` is expected to be an access key and the parameter `root_id` is mandatory and specifies the file ID of the top folder to use on the node using this access key. It can be either the access key's root file ID, or any authorized file ID underneath it. #### Agent: HTTP Gateway The Aspera HTTP Gateway is a service that allows sending and receiving files using HTTPS. By specifying option: `--transfer=httpgw`, `ascli` will start transfers using the Aspera HTTP Gateway. Parameters provided in option `transfer` are: | Field | Type | Description | |-------------------|--------------|---------------------------------------------| | `api_version` | `Enum[String]` | Force use of a specific API version.<br/>Allowed values: `v1`, `v2`.<br/>Default: `v2`. | | `synchronous` | `Bool` | Wait for acknowledgment after each message.<br/>Default: `false`. | | `upload_chunk_size` | `Integer` | Size in bytes of each upload chunk.<br/>Default: `64000`. | | `url` | `String` | URL of the HTTP Gateway.<br/>Required. | Example: ```shell ascli faspex5 packages receive 323 --transfer=httpgw --transfer.url=https://asperagw.example.com:9443/aspera/http-gwy ``` > [!NOTE] > The gateway only supports transfers authorized with a token. If the application, for example, AoC or Faspex 5, is configured to use the HTTP Gateway, then `ascli` will automatically use the gateway URL if `--transfer=httpgw` is specified, so `transfer` URL becomes optional. #### Agent: Transfer Daemon Another possibility is to use the Transfer Daemon (`transferd`). Set option `transfer` to `transferd`. Options for `transfer` are: | Field | Type | Description | |---------|--------|----------------------------------------------------------------------------------| | `api_key` | `String` | API key of an existing daemon (`start` is `false`) that has `authentication` enabled in its configuration. It is sent in the metadata `api_key` of each call. | | `ca_file` | `String` | Path to the certificate authority file (PEM) that signed the certificate of an existing daemon (`start` is `false`) that has `tls` enabled in its configuration. A self-signed certificate is its own authority. | | `start` | `Bool` | Start a new daemon instance.<br/>Default: `true`. | | `stop` | `Bool` | Stop the daemon when `ascli` exits.<br/>Default: `true`. | | `url` | `String` | Address and port listened by the daemon (e.g. `:55002`). If port is `0`, the daemon listens on a random available port. If no address is given, `127.0.0.1` is used.<br/>Default: `:0`. | > [!NOTE] > If port zero is specified in the URL, then the daemon will listen on a random available port. > If no address is specified, then `127.0.0.1` is used. For example, to use an external, already running `transferd`, use option: ```shell --transfer=@json:'{"agent":"transferd","url":":55002","start":false,"stop":false}' ``` The gem `grpc` is not part of default dependencies, as it requires compilation of a native part. So, to use the Transfer Daemon you should install this gem: ```shell gem install grpc ``` If the execution complains about incompatible libraries, then force recompilation of the native part: ```shell gem uninstall grpc gem install grpc --platform ruby ``` On Windows the compilation may fail for various reasons (3.1.1): <!-- spellchecker: disable --> - `cannot find -lx64-ucrt-ruby310` → copy the file `[Ruby main dir]\lib\libx64-ucrt-ruby310.dll.a` to (remove the `dll` extension) `[Ruby main dir]\lib\libx64-ucrt-ruby310.a` - `conflicting types for 'gettimeofday'` → edit the file `[Ruby main dir]/include/ruby-[version]/ruby/win32.h` and change the signature of `gettimeofday` to `gettimeofday(struct timeval *, void *)`, that is, change `struct timezone` to `void` <!-- spellchecker: enable --> #### Transfer Daemon as a system service The Transfer Daemon (`transferd`) can run permanently as a `systemd` service, for applications that use an existing daemon (see [Agent: Transfer Daemon](#agent-transfer-daemon), `start` set to `false`). `ascli` installs the SDK, writes the configuration file of the daemon (`/etc/transferd/transferd.json`) and the `systemd` unit, then enables and starts the service. Commands are in `transferd service` (plugin [`transferd`](#plugin-transferd-transfer-daemon)): | Command | Action | |-------------|--------| | `spec` | Show the schema of the configuration file of the daemon (`transferd.json`). Works on any system. | | `install` | Install the SDK if it is missing, then do `configure`. | | `configure` | Merge the given configuration into the existing file, validate it, write the file and the unit, reload `systemd`, enable and restart the service, and wait until the daemon listens. | | `status` | Show the version, the state of the service, the endpoint and whether it listens. The API key is never shown. | | `uninstall` | Stop and remove the service. With `purge` set to `true` in `service`, also remove the configuration file and the log folder. | | `use` | Switch to an installed version of the SDK and restart the service (see below). | | `versions` | List the installed versions of the SDK (see below). | Except `spec` and `versions`, commands take the optional **last** positional argument `CONFIG`: a `Hash` ([Extended Value](#extended-value-syntax)) with two optional sections: - `service`: how the service is installed. - `config`: content of `transferd.json`. ```json { "service": { "user": "asperadaemon" }, "config": { "address": "127.0.0.1", "port": 55002, "log_directory": "/var/log/aspera-transferd", "workers": {"max_concurrent": 10}, "authentication": {"enabled": true} } } ``` Parameters of section `service`: | Field | Type | Description | |-------------|---------------|----------------------------------------------------------------------------------| | `after` | `Array[String]` | Units that shall be started before the daemon (`After=` in the unit).<br/>Default: `["network.target"]`. | | `environment` | `Hash` | Environment variables set in the unit. None is set unless given here. `TRANSFERD_ADDRESS`, `TRANSFERD_PORT`, `TRANSFERD_MGMT_PORT` and `TRANSFERD_MGMT_ADDRESS` override the configuration file. | | `group` | `String` | Group that runs the daemon. Defaults to the primary group of `user`. | | `name` | `String` | Name of the service. The unit file is `<name>.service`.<br/>Default: `aspera-transferd`. | | `purge` | `Bool` | Used by `uninstall`. Also remove the configuration file and the log folder.<br/>Default: `false`. | | `unit_dir` | `String` | Folder where the unit file is created.<br/>Default: `/etc/systemd/system`. | | `user` | `String` | Account that runs the daemon. It must exist, it is not created.<br/>Default: `asperadaemon`. | | `version` | `String` | Version of the SDK to install, used by `install` when the SDK folder is named `current` (versions side by side). Defaults to the SDK version of this tool. | Section `config` follows the JSON schema of the daemon, displayed with: ```shell ascli transferd service spec ``` | Field | Type | Description | |------------------------------------------|---------------|----------------------------------------------------------------------------------| | `$schema` | `String` | Optional URI or path of this schema, used by editors for completion and validation. Ignored by transferd. | | `address` | `String` | IP address or host name on which the gRPC API listens. Use 0.0.0.0 to listen on all IPv4 interfaces (for example in a container); when not listening on a loopback address, consider enabling tls and authentication. Overridden by environment variable TRANSFERD_ADDRESS, and by option --address (-a), which takes precedence.<br/>Default: `127.0.0.1`.<br/>Example: `127.0.0.1`. | | `api_time_settings` | `Hash` | Keepalive and connection lifetime settings of the gRPC API server, in seconds. | | `api_time_settings.max_conn_age` | `Integer` | Maximum lifetime of a client connection, with a random variation of +/-10%; the client must then reconnect. 0 or omitted: no limit.<br/>Default: `0`.<br/>Example: `3600`. | | `api_time_settings.max_conn_age_grace` | `Integer` | Additional time given to ongoing calls once max_conn_age is reached, before the connection is forcibly closed. 0 or omitted: no limit.<br/>Default: `0`.<br/>Example: `10`. | | `api_time_settings.max_conn_idle` | `Integer` | Time after which a client connection with no ongoing call is closed. 0 or omitted: 86400 (24 hours).<br/>Default: `86400`.<br/>Example: `120`. | | `api_time_settings.time` | `Integer` | Time without activity from a client after which the server pings it to check that the connection is still alive. 0 or omitted: 7200 (2 hours).<br/>Default: `7200`.<br/>Example: `60`. | | `api_time_settings.timeout` | `Integer` | Time the server waits for activity after a keepalive ping before closing the connection. 0 or omitted: 20.<br/>Default: `20`.<br/>Example: `3`. | | `authentication` | `Hash` | API key authentication of clients of the gRPC API. | | `authentication.api_key` | `String` | Value that clients must send in gRPC request metadata entry api_key. It is stored in clear text: restrict read access to this file.<br/>Example: `change-me-to-a-long-random-value`. | | `authentication.enabled` | `Bool` | Requires every call to carry the API key; other calls are rejected with status UNAUTHENTICATED. api_key is then required.<br/>Default: `false`. | | `fasp_management` | `Hash` | Management port on which the ascp processes started by transferd report transfer progress. | | `fasp_management.address` | `String` | IP address on which the management server listens. Change it only when ascp must reach transferd through an interface other than loopback. Overridden by environment variable TRANSFERD_MGMT_ADDRESS, and by option --mgmtaddress (-A), which takes precedence.<br/>Default: `127.0.0.1`.<br/>Example: `127.0.0.1`. | | `fasp_management.port` | `Integer` | TCP port of the management server. 0 or omitted: a free port is selected at startup. Overridden by environment variable TRANSFERD_MGMT_PORT, and by option --mgmtport (-P), which takes precedence.<br/>Default: `0`.<br/>Example: `23456`. | | `fasp_management.promiscuous_mode_enabled` | `Bool` | When true, transferd also reports transfers of ascp processes that it did not start. So that those processes find the management port, transferd writes it in file transferd.<port>.optport of directory ../var/run/aspera, relative to the FASP runtime bin directory. Those ascp processes must use the same FASP runtime as transferd (see fasp_runtime).<br/>Default: `false`. | | `fasp_runtime` | `Hash` | FASP runtime (ascp, async, license, aspera.conf) used for transfers. Omitted: the runtime embedded in the SDK is used. When this section is present, set use_embedded explicitly: if it is not true, user_defined is required. | | `fasp_runtime.extra_config` | `Array[Hash]` | Extra command line options added to the executables started for transfers. Does not apply to async. | | `fasp_runtime.extra_config[].command` | `String` | End of the path of the executable to which options are added, for example ascp (ascp.exe on Windows). Only the first matching entry is used.<br/>Required.<br/>Example: `ascp`. | | `fasp_runtime.extra_config[].options` | `Array[String]` | Options added to the command line, one argument per item, before the transfer arguments.<br/>Example: `["--save-before-overwrite"]`. | | `fasp_runtime.force_version` | `String` | Accepted for backward compatibility. Has no effect. | | `fasp_runtime.log` | `Hash` | Default logging of ascp, for transfers whose request configuration does not set localLog or logLevel. Does not apply to async. | | `fasp_runtime.log.dir` | `String` | Directory where ascp writes its log (ascp option -L). Empty or omitted: default location of ascp.<br/>Example: `/var/log/aspera`. | | `fasp_runtime.log.level` | `Integer` | Level of detail of the ascp log: 0 for basic information (files transferred, transfer rate), 1 to 4 for increasingly detailed debug messages (ascp options -D to -DDDD).<br/>Default: `0`. | | `fasp_runtime.use_embedded` | `Bool` | true: use the FASP runtime embedded in the SDK, and ignore user_defined. false: use the runtime located by user_defined. Omitted while the fasp_runtime section is present: false.<br/>Example: `true`. | | `fasp_runtime.user_defined` | `Hash` | Location of a separately installed FASP runtime. Required when use_embedded is not true, ignored otherwise. Both directories must exist at startup. | | `fasp_runtime.user_defined.bin` | `String` | Directory containing the ascp and async executables. A relative path is resolved from the working directory of transferd.<br/>Required.<br/>Example: `/opt/aspera/bin`. | | `fasp_runtime.user_defined.etc` | `String` | Directory containing the license file aspera-license and the configuration file aspera.conf. A relative path is resolved from the working directory of transferd.<br/>Required.<br/>Example: `/opt/aspera/etc`. | | `log_directory` | `String` | Directory of the transferd log file transferd.log (rotated at 10 MiB, 10 files kept). A relative path is resolved from the directory of the transferd executable. The directory is created if missing. Empty or omitted: logs are written to standard output.<br/>Example: `/var/log/aspera`. | | `log_format` | `Enum[String]` | Format of the transferd log: text for human-readable lines, json for one JSON object per line. Any value other than text gives json.<br/>Allowed values: `text`, `json`.<br/>Default: `text`. | | `log_level` | `Enum[String]` | Level of detail of the transferd log, from most (trace) to least (panic) verbose. Omitted or unrecognized value: info.<br/>Allowed values: `trace`, `debug`, `info`, `warning`, `error`, `fatal`, `panic`.<br/>Default: `info`. | | `port` | `Integer` | TCP port on which the gRPC API listens. 0 or omitted: a free port is selected at startup; it is written in the log and in file transferd.<port>.optport of runtime_dir. The configuration file shipped with the SDK sets 55002. Overridden by environment variable TRANSFERD_PORT, and by option --port (-p), which takes precedence.<br/>Default: `0`.<br/>Example: `55002`. | | `reflection` | `Bool` | Enables the gRPC server reflection service, so that tools such as `grpcurl` or `grpcui` can list and call the API without the transferd.proto file. Useful for debugging; leave disabled otherwise.<br/>Default: `false`. | | `runtime_dir` | `String` | Directory for runtime files. When port is 0, transferd writes the selected port in file transferd.<port>.optport there. Created with owner-only permissions if missing. Omitted: ../var/run/aspera relative to the directory of the transferd executable.<br/>Example: `/var/run/aspera`. | | `tls` | `Hash` | TLS for the gRPC API. When enabled, clients must connect using TLS. | | `tls.certificate` | `String` | Path of the PEM server certificate file, which may include intermediate certificates. A relative path is resolved from the working directory of transferd.<br/>Example: `/etc/transferd/server.crt`. | | `tls.enabled` | `Bool` | Enables TLS. certificate and key are then required.<br/>Default: `false`. | | `tls.key` | `String` | Path of the PEM private key file matching certificate. A relative path is resolved from the working directory of transferd.<br/>Example: `/etc/transferd/server.key`. | | `tls.min_tls_version` | `Enum[String]` | Minimum TLS version accepted from clients. Omitted or other value: TLSv12.<br/>Allowed values: `TLSv12`, `TLSv13`.<br/>Default: `TLSv12`. | | `workers` | `Hash` | Transfer execution and retention. | | `workers.max_concurrent` | `Integer` | Maximum number of transfers running at the same time. Additional transfers wait in a queue until a running one ends. 0 or omitted: 100.<br/>Default: `100`.<br/>Example: `10`. | | `workers.transfer_cleaning_period` | `Integer` | Interval, in seconds, at which transferd removes ended transfers older than transfer_max_age. 0 or omitted: 3600 (1 hour).<br/>Default: `3600`. | | `workers.transfer_max_age` | `Integer` | Time, in seconds, during which an ended transfer remains known to transferd (status queries, events) before it can be removed. Removal happens at the next cleaning, so a transfer may remain up to transfer_max_age + transfer_cleaning_period. 0 or omitted: 7200 (2 hours).<br/>Default: `7200`.<br/>Example: `86400`. | The configuration is checked before anything is written: - Keys not given keep their value in the existing `transferd.json` (deep merge), and a key set to `null` is removed. - An unknown key, a value of a wrong type, or an inconsistent configuration (for example `tls.enabled` without `certificate` and `key`) is refused, with the name of the key and of the file if the faulty key is in the existing file. The daemon itself ignores unknown keys silently. - The section `fasp_runtime` is computed from the SDK folder, and can be overridden. - If `authentication.enabled` is `true` and no `api_key` is given, the key of the existing file is kept, else a random one is generated. The file is written in clear text with mode `0640`, owner `user:group`. To get the key from the [vault](#secret-vault), write `{"config":{"authentication":{"api_key":"@vault:transferd.api_key@"}}}` in an [Extended Value](#extended-value-syntax) `@json:@extend:'...'`. Requirements: - `systemd` on Linux only. On other systems, or without `systemctl`, commands fail with a message (only `spec` and `--dry-run` work everywhere). - Run as `root`, except with `--dry-run=yes` (option is a boolean), which shows the actions and the files without any change, and `spec` and `versions`. - The account (`asperadaemon` by default) is **not** created by `ascli`. If it does not exist, the command fails and shows the command to create it: `useradd --system --no-create-home --shell /usr/sbin/nologin asperadaemon`. This is the account used by other Aspera products on the same host, so that files have the same owner. - If the version of the SDK installed is different from the version of the schema, then a warning is displayed (it is not an error). Example: ```shell sudo ascli transferd service install --sdk-folder=/opt/aspera/transferd/current @json:'{"config":{"port":55002,"authentication":{"enabled":true}}}' ``` ```shell ascli transferd service configure --dry-run=yes @json:'{"config":{"log_level":"debug"}}' ``` ```shell sudo ascli transferd service status ``` ```shell sudo ascli transferd service uninstall @json:'{"service":{"purge":true}}' ``` ##### Versions side by side and rollback If the SDK folder (option `sdk_folder`) is named `current`, e.g. `--sdk-folder=/opt/aspera/transferd/current`, then `ascli` uses the layout `<prefix>/<version>` with a link `current` that points to the version in use: ```text /opt/aspera/transferd/1.1.9 /opt/aspera/transferd/1.1.10 /opt/aspera/transferd/current -> 1.1.10 ``` - `service install` installs the version given in `service.version` (default: version tested with this release of `ascli`) in `<prefix>/<version>`, and sets the link `current`. If the installation fails, then the link is set back to the previous version. - The unit and `transferd.json` refer to `current`: switching the link and restarting the service is enough to change the version. - `service versions` lists the installed versions and the one in use. - `service use <VERSION>` switches the link and restarts the service. If the service does not start, then the link is set back to the previous version and the service is restarted. Any other folder name is used as is, with no link. ### Transfer Specification Some commands lead to file transfer (upload/download). All parameters necessary for this transfer are described in a [**transfer-spec**](#transfer-specification) (Transfer Specification), such as: - Server address - Transfer username - Credentials - File list - And so on `ascli` builds the [**transfer-spec**](#transfer-specification) internally as a `Hash`. It is not necessary to provide additional parameters on the command line for a transfer. It is possible to modify or add any of the supported [**transfer-spec**](#transfer-specification) parameters using the `ts` option. The `ts` option accepts a `Hash` [Extended Value](#extended-value-syntax) containing one or several [**transfer-spec**](#transfer-specification) parameters. Multiple `ts` options on command line are cumulative, and the `Hash` value is deeply merged. To remove a (deep) key from transfer spec, set the value to `null`. > [!NOTE] > Default transfer spec values can be displayed with command: ```shell ascli config ascp info --fields=ts --out.flat=no ``` It is possible to specify `ascp` options when the `transfer` option is set to [`direct`](#agent-direct) using `transfer` option parameter: `ascp_args`. Example: ```shell --transfer=@json:'{"ascp_args":["-l","100m"]}' ``` Or an equivalent (using dotted expression): ```shell --transfer.ascp_args=@list:' -l 100m' ``` This is especially useful for `ascp` command line parameters not supported in the transfer spec. The use of a [**transfer-spec**](#transfer-specification) instead of `ascp` command line arguments has the following advantages: - It is common to all [Transfer Agents](#transfer-clients-agents) - It does not depend on command line limitations (special characters, and so on) #### Transfer Parameters All standard [**transfer-spec**](#transfer-specification) parameters can be specified. A [**transfer-spec**](#transfer-specification) can also be saved/overridden in the configuration file. References: - [Aspera Node API Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20node%20api%22) → /opt/transfers - [Aspera Transfer Daemon Documentation](https://developer.ibm.com/apis/catalog?search=%22aspera%20transfer%20sdk%22) → Guides → API Ref → Transfer Spec V1 - [Aspera Connect SDK](https://d3gcli72yxqn2z.cloudfront.net/connect/v4/asperaweb-4.js) → search `The parameters for starting a transfer.` Parameters can be displayed with commands: ```shell ascli config ascp spec ``` An optional parameter can be specified to display only the parameters supported by a specific transfer agent, or `overview` for one column per agent indicating which agents support each parameter: ```shell ascli config ascp spec direct ascli config ascp spec overview ascli config ascp spec overview --fields=name,a,c,d,h,n,t ``` A JSON Schema can be generated with command: ```shell ascli config ascp schema --format=jsonpp ``` An optional parameter can be specified to display the schema for a specific transfer agent: ```shell ascli config ascp schema transferd --format=jsonpp ``` The description gives the supported agents, if not all, and the corresponding `ascp` argument or environment variable. #### Transfer Specification Reference | ID | Name | |----|-----------| | A | Direct | | C | Connect | | D | Desktop | | H | Httpgw | | N | Node | | T | Transferd | | Field | Type | Description | |--------------------------------|-----------------|----------------------------------------------------------------------------------| | `apply_local_docroot` | `Bool` | Apply local docroot to source paths.<br/>(A, T)<br/>(`--apply-local-docroot`) | | `authentication` | `String` | Set to `token` for SSH bypass keys, else password asked if not provided.<br/>(C) | | `cipher` | `Enum[String]` | In transit encryption algorithms.<br/>Allowed values: `none`, `aes-128`, `aes-192`, `aes-256`, `aes-128-cfb`, `aes-192-cfb`, `aes-256-cfb`, `aes-128-gcm`, `aes-192-gcm`, `aes-256-gcm`.<br/>Default: `none`.<br/>(`-c (conversion){enum}`) | | `cipher_allowed` | `Enum[String]` | Returned by node API. Valid literals include `aes-128` and `none`.<br/>Allowed values: `none`, `aes-128`, `aes-192`, `aes-256`, `aes-128-cfb`, `aes-192-cfb`, `aes-256-cfb`, `aes-128-gcm`, `aes-192-gcm`, `aes-256-gcm`.<br/>(C) | | `content_protection` | `Enum[String]` | Enable client-side content protection (CSEAR, encryption-at-rest).<br/>For uploads, set to `encrypt` to transfer encrypted files and store them on the server with the extension `.aspera-env`. (`aspera.conf` parameter `transfer_encryption_content_protection_extension`). To download and decrypt encrypted files, set to `decrypt`<br/>`content_protection_password` must be specified if this option is set.<br/>Allowed values: `encrypt`, `decrypt`.<br/>(`--file-crypt={enum}`) | | `content_protection_password` | `String` | Password for encryption/decryption of transferred assets.<br/>(env:`ASPERA_SCP_FILEPASS`) | | `cookie` | `String` | Metadata for transfer specified by application.<br/>(env:`ASPERA_SCP_COOKIE`) | | `create_dir` | `Bool` | Create target directory if it doesn't already exist.<br/>If **all** the following conditions are met, then the `destination_root` specifies a filename instead of destination folder:<br/>- `create_dir` is `false`<br/>- A single source file is given on **command line**<br/>- The target folder specified by `destination_root` does not exist<br/>In all other cases, `destination_root` specifies a folder, and it is created if it does not already exist. I.e. if **any** of those conditions is met:<br/>- `create_dir` is `true`<br/>- Multiple source files are provided<br/>- List of source files are provided in a file (list or pair), default for Node API and `ascli`.<br/>- The target folder exists<br/>(`-d`) | | `delete_before_transfer` | `Bool` | Before transfer, delete files that exist at the destination but not at the source.<br/>The source and destination arguments must be directories that have matching names.<br/>Objects on the destination that have the same name but different type or size as objects on the source are not deleted.<br/>(`--delete-before-transfer`) | | `delete_source` | `Bool` | Remove transferred source files after transfer success. Equivalent to `remove_after_transfer` + `remove_empty_directories` + `remove_empty_source_directory`. Take precedence over those.<br/>(A, N, T) | | `destination_root` | `String` | Destination root directory.<br/>Required. | | `destination_root_id` | `String` | The file ID of the destination root directory.<br/>Required when using Bearer token auth for the destination node.<br/>(T) | | `dgram_size` | `Integer` | UDP datagram size in bytes.<br/>(`-Z {integer}`) | | `direction` | `Enum[String]` | Direction of transfer (on client side).<br/>Required.<br/>Allowed values: `send`, `receive`.<br/>(`--mode=(conversion){enum}`) | | `exclude_newer_than` | `String` | Exclude files, but not directories, from the transfer if they are newer than the specified number of seconds added to the source computer's epoch.<br/>e.g. `-86400` for newer than a day back.<br/>(`--exclude-newer-than={string}`) | | `exclude_older_than` | `String` | Exclude files, but not directories, from the transfer if they are older than the specified number of seconds added to the source computer's epoch.<br/>e.g. `-86400` for older than a day back.<br/>(`--exclude-older-than={string}`) | | `fail_bad_filepass` | `Bool` | Fail on bad file decryption passphrase.<br/>(A, T)<br/>(`--fail-bad-filepass`) | | `fasp_port` | `Integer` | Specifies fasp (UDP) port.<br/>Default: `33001`.<br/>(`-O {integer}`) | | `fasp_proxy` | `Hash` | Proxy for communications between the remote server and the (local) client.<br/>(T) | | `file_checksum` | `Enum[String]` | Enable checksum reporting for transferred files by specifying the hash to use.<br/>Allowed values: `sha-512`, `sha-384`, `sha-256`, `sha1`, `md5`, `none`.<br/>Default: `none`.<br/>(A, N)<br/>(`--file-checksum={enum}`) | | `http_fallback` | `Bool, String` | When true(1), attempts to perform an HTTP transfer if a FASP transfer cannot be performed.<br/>(`-y (conversion){boolean\|string}`) | | `http_fallback_port` | `Integer` | Specifies HTTP port when no cipher is used.<br/>(`-t {integer}`) | | `https_fallback_port` | `Integer` | Specifies HTTPS port when cipher is used.<br/>(`-t {integer}`) | | `icos` | `Hash` | Configuration parameters for IBM Cloud Object Storage (ICOS).<br/>(T) | | `keepalive` | `Bool` | The session is running in persistent session mode.<br/>(A, T)<br/>(`--keepalive`) | | `lock_min_rate` | `Bool` | n/a<br/>(C) | | `lock_min_rate_kbps` | `Bool` | If `true`, lock the minimum transfer rate to the value set for min_rate_kbps.<br/>If `false`, users can adjust the transfer rate up to the value set for target_rate_cap_kbps.<br/>(C, T) | | `lock_rate_policy` | `Bool` | If `true`, lock the rate policy to the default value.<br/>(C, T) | | `lock_target_rate` | `Bool` | n/a<br/>(C) | | `lock_target_rate_kbps` | `Bool` | If `true`, lock the target transfer rate to the default value set for `target_rate_kbps`.<br/>If `false`, users can adjust the transfer rate up to the value set for `target_rate_cap_kbps`.<br/>(C, T) | | `min_rate_cap_kbps` | `Integer` | The highest minimum rate that an incoming transfer can request, in kilobits per second.<br/>Client minimum rate requests that exceed the minimum rate cap are ignored.<br/>The default value of unlimited applies no cap to the minimum rate. (Default: 0)<br/>(C, T) | | `min_rate_kbps` | `Integer` | Set the minimum transfer rate in kilobits per second.<br/>Default: `0`.<br/>(`-m {integer}`) | | `move_after_transfer` | `String` | Move source files to the specified `archive-dir` directory after they are transferred correctly.<br/>Available as of 3.8.0. Details in `ascp` manual.<br/>Requires write permissions on the source.<br/>If `src_base` is specified, files are moved to `archive-dir`/`path-relative-to-srcbase`.<br/>`archive-dir` must be in the same file system (or cloud storage account) as the source files being transferred.<br/>`archive-dir` is subject to the same docroot restrictions as source files.<br/>`move_after_transfer` and `remove_after_transfer` are mutually exclusive options.<br/>After files have been moved to the archive, the original source directory structure is left in place. Empty directories are not saved to `archive-dir`. To remove empty source directories after a successful move operation, also set `remove_empty_directories` to `true`. When using `remove_empty_directories`, empty directory removal examination starts at the `srcbase` and proceeds down any subdirectories. If no `srcbase` is used and a file path (as opposed to a directory path) is specified, then only the immediate parent directory is examined and removed if it is empty following the move of the source file.<br/>(A, N, T)<br/>(`--move-after-transfer={string}`) | | `multi_session` | `Integer` | Use multi-session transfer. max 128.<br/>Each participant on one host needs an independent UDP (-O) port.<br/>Large files are split between sessions only when transferring with `resume_policy`=`none`.<br/>(special:`-C {integer}`) | | `multi_session_threshold` | `Integer` | Split files across multiple `ascp` sessions if their size in bytes is greater than or equal to the specified value.<br/>(0=no file is split)<br/>(A, N, T)<br/>(`--multi-session-threshold={integer}`) | | `obfuscate_file_names` | `Bool` | HTTP Gateway obfuscates file names when set to `true`.<br/>(H) | | `overwrite` | `Enum[String]` | Overwrite files at the destination with source files of the same name based on the policy:<br/>- `always` : Always overwrite the file.<br/>- `never` : Never overwrite the file. If the destination contains partial files that are older or the same as the source files and resume is enabled, the partial files resume transfer. Partial files with checksums or sizes that differ from the source files are not overwritten.<br/>- `diff` : (default) Overwrite the file if it is different from the source, depending on the compare method (default is size). If the destination is object storage, `diff` has the same effect as always. If resume is not enabled, partial files are overwritten if they are different from the source, otherwise they are skipped. If resume is enabled, only partial files with different sizes or checksums from the source are overwritten; otherwise, files resume.<br/>- `diff+older` : Overwrite the file if it is older and different from the source, depending on the compare method (default is size). If resume is not enabled, partial files are overwritten if they are older and different from the source, otherwise they are skipped. If resume is enabled, only partial files that are different and older than the source are overwritten, otherwise they are resumed.<br/>- `older` : Overwrite the file if its timestamp is older than the source timestamp.<br/>If you set an overwrite policy of `diff` or `diff+older`, difference is determined by the value set for `resume_policy`:<br/>`none` : The source and destination files are always considered different and the destination file is always overwritten<br/>`attributes` : The source and destination files are compared based on file attributes <br/>`sparse_checksum` : The source and destination files are compared based on sparse checksums, (currently file size)<br/>`full_checksum` : The source and destination files are compared based on full checksums<br/>Allowed values: `never`, `always`, `diff`, `older`, `diff+older`.<br/>Default: `diff`.<br/>(`--overwrite={enum}`) | | `password` | `String` | Password for local Windows user when transfer user associated with node API user is not the same as the one running `asperanoded`.<br/>Allows impersonating the transfer user and have access to resources (e.g. network shares).<br/>Windows only, node API only.<br/>(N) | | `paths` | `Array` | Array of path to the source (required) and a path to the destination (optional).<br/>Required. | | `precalculate_job_size` | `Bool` | Specifies whether to precalculate the job size.<br/>(`--precalculate-job-size / --precalculate-disable`) | | `preserve_access_time` | `Bool` | Preserve the source-file access timestamps at the destination.<br/>Because source access times are updated by the transfer operation, the timestamp that is preserved is the one just before to the transfer.<br/>(`--preserve-access-time`) | | `preserve_acls` | `Enum[String]` | Preserve access control lists.<br/>Allowed values: `none`, `native`, `metafile`.<br/>Default: `none`.<br/>(A, T)<br/>(`--preserve-acls={enum}`) | | `preserve_creation_time` | `Bool` | Preserve source-file creation timestamps at the destination.<br/>Only Windows systems retain information about creation time. If the destination is not a Windows computer, this option is ignored.<br/>(`--preserve-creation-time`) | | `preserve_extended_attrs` | `Enum[String]` | Preserve the extended attributes.<br/>Allowed values: `none`, `native`, `metafile`.<br/>Default: `none`.<br/>(A, T)<br/>(`--preserve-xattrs={enum}`) | | `preserve_file_owner_gid` | `Bool` | Preserve the group ID for a file owner.<br/>(A, T)<br/>(`--preserve-file-owner-gid`) | | `preserve_file_owner_uid` | `Bool` | Preserve the user ID for a file owner.<br/>(A, T)<br/>(`--preserve-file-owner-uid`) | | `preserve_modification_time` | `Bool` | Set the modification time, the last time a file or directory was modified (written), of a transferred file to the modification of the source file or directory.<br/>Preserve source-file modification timestamps at the destination.<br/>(`--preserve-modification-time`) | | `preserve_remote_acls` | `Enum[String]` | Preserve remote access control lists.<br/>Allowed values: `none`, `native`, `metafile`.<br/>Default: `none`.<br/>(A, T)<br/>(`--remote-preserve-acls={enum}`) | | `preserve_remote_extended_attrs` | `Enum[String]` | Preserve remote extended attributes.<br/>Allowed values: `none`, `native`, `metafile`.<br/>Default: `none`.<br/>(A, T)<br/>(`--remote-preserve-xattrs={enum}`) | | `preserve_source_access_time` | `Bool` | Preserve the time logged for when the source file was accessed.<br/>(A, T)<br/>(`--preserve-source-access-time`) | | `preserve_times` | `Bool` | Preserve file timestamps.<br/>(A, N, T)<br/>(`-p`) | | `proxy` | `String` | Specify the address of the Aspera high-speed proxy server.<br/>`dnat(s)://[user[:password]@]server:port`<br/>Default ports for DNAT and DNATS protocols are 9091 and 9092.<br/>Password, if specified here, overrides the value of environment variable `ASPERA_PROXY_PASS`.<br/>(A)<br/>(`--proxy={string}`) | | `rate_policy` | `Enum[String]` | The transfer rate policy to use when sharing bandwidth. Allowable values:<br/>- `high` : When sharing bandwidth, transfer at twice the rate of a transfer using a fair policy.<br/>- `fair` : (Default) Share bandwidth equally with other traffic.<br/>- `low` : Use only unused bandwidth.<br/>- `fixed` : Transfer at the target rate, regardless of the actual network capacity. Do not share bandwidth. Aspera recommends that you do not use this setting except under special circumstances, otherwise the destination storage can be damaged.<br/>Allowed values: `low`, `fair`, `high`, `fixed`.<br/>(`--policy={enum}`) | | `rate_policy_allowed` | `Enum[String]` | Specifies most aggressive rate policy that is allowed. Returned by node API.<br/>Allowed values: `low`, `fair`, `high`, `fixed`.<br/>(C) | | `remote_access_key` | `String` | The access key ID of the access key that was used to construct the bearer token that is used to authenticate to the remote node.<br/>(T) | | `remote_host` | `String` | IP or fully qualified domain name (FQDN) of the remote server.<br/>Required.<br/>(`--host={string}`) | | `remote_password` | `String` | SSH session password.<br/>(env:`ASPERA_SCP_PASS`) | | `remote_user` | `String` | Remote user. Default value is `xfer` on node or connect.<br/>Required.<br/>(`--user={string}`) | | `remove_after_transfer` | `Bool` | Remove SRC files after transfer success.<br/>(A, N, T)<br/>(`--remove-after-transfer`) | | `remove_empty_directories` | `Bool` | Specifies whether to remove empty directories.<br/>(A, N, T)<br/>(`--remove-empty-directories`) | | `remove_empty_source_dir` | `Bool` | Remove empty source subdirectories and remove the source directory itself, if empty.<br/>(T) | | `remove_empty_source_directory` | `Bool` | Remove empty source subdirectories and remove the source directory itself, if empty.<br/>(A)<br/>(`--remove-empty-source-directory`) | | `remove_skipped` | `Bool` | Must also have `remove_after_transfer` set to `true`. Defaults to `false`. If `true`, skipped files will be removed as well.<br/>(A, C, N)<br/>(`--remove-skipped`) | | `resume_policy` | `Enum[String]` | If a transfer is interrupted or fails to finish, this policy directs the transfer to resume without retransferring the files. Allowable values:<br/>- `none` : Always re-transfer the entire file.<br/>- `attrs` : Compare file attributes and resume if they match, and re-transfer if they do not.<br/>- `sparse_csum` : Compare file attributes and the sparse file checksums; resume if they match, and re-transfer if they do not.<br/>- `full_csum` : Compare file attributes and the full file checksums; resume if they match, and re-transfer if they do not.<br/>Note: transferd uses values: `attributes`, `sparse_checksum`, `full_checksum`.<br/>Allowed values: `none`, `attrs`, `sparse_csum`, `full_csum`.<br/>Default: `faspmgr:none;other:sparse_csum`.<br/>(`-k (conversion){enum}`) | | `retry_duration` | `Integer, String` | Specifies how long to wait before retrying transfer (e.g. `5min`).<br/>(T) | | `save_before_overwrite` | `Bool` | If a transfer would result in an existing file <filename>.<ext> being overwritten, move that file to <filename>.yyyy.mm.dd.hh.mm.ss.index.<ext> (where index is set to 1 at the beginning of each new second and incremented for each file saved in this manner during the same second) in the same directory before writing the new file.<br/>File attributes are maintained in the renamed file.<br/>(A, N, T)<br/>(`--save-before-overwrite`) | | `skip_duplicate_check` | `Bool` | Don't check for duplicate files at the destination.<br/>(A, T)<br/>(`--skip-dir-traversal-dupes`) | | `skip_special_files` | `Bool` | All assets other than files, directories and symbolic links are considered special. A transfer will fail if the user attempts to transfer special assets. If `true`, `ascp` skips special assets and proceeds with the transfer of all other assets.<br/>(A, T)<br/>(`--skip-special-files`) | | `source_root` | `String` | Path to be prepended to each source path.<br/>This is either a conventional path or it can be a URI but only if there is no root defined.<br/>(`--source-prefix64=(conversion){string}`) | | `source_root_id` | `String` | The file ID of the source root directory. Required when using Bearer token auth for the source node.<br/>(N, T) | | `src_base` | `String` | Specify the prefix to be stripped off from each source object.<br/>The remaining portion of the source path is kept intact at the destination.<br/>Special care must be taken when used with cloud storage.<br/>(A, N, T)<br/>(`--src-base64=(conversion){string}`) | | `src_base64` | `String` | The folder name below which the directory structure is preserved (base64 encoded).<br/>(A, T)<br/>(`--src-base64={string}`) | | `ssh_args` | `Array` | Add arguments to the command-line arguments passed to the external ssh program (implies -SSH). The arguments are inserted before any key file(s) supplied to `ascp` and before the user/host arguments.<br/>(A, T)<br/>(special:`--ssh-arg={array}`) | | `ssh_port` | `Integer` | Specifies SSH (TCP) port.<br/>Default: `direct:22, other:33001`.<br/>(`-P {integer}`) | | `ssh_private_key` | `String` | Private key used for SSH authentication.<br/>Shall look like: -----BEGIN RSA PRIV4TE KEY-----\nMII...<br/>Note the JSON encoding: \n for newlines.<br/>(A, T)<br/>(env:`ASPERA_SCP_KEY`) | | `ssh_private_key_passphrase` | `String` | The passphrase associated with the transfer user's SSH private key. Available as of 3.7.2.<br/>(A, T)<br/>(env:`ASPERA_SCP_PASS`) | | `ssh_private_key_path` | `String` | Path to private key for SSH.<br/>(A, T)<br/>(`-i {string}`) | | `sshfp` | `String` | Check it against server SSH host key fingerprint.<br/>(`--check-sshfp={string}`) | | `symlink_policy` | `Enum[String]` | Handle source side symbolic links. Refer to HSTS manual for more details.<br/>- `follow`: Follow a symbolic link and transfer the contents of the linked file or directory when the link target is in the user's docroot.<br/>- `copy`: Copy only the symbolic link. If a file with the same name exists at the destination, the symbolic link does not replace the file.<br/>- `skip`: Skip-symbolic links. The link or the file to which it points are not transferred.<br/>- `copy+force`: Copy only the symbolic link. If a file with the same name exists at the destination, the symbolic link replaces the file. If the file of the same name at the destination is a symbolic link to a directory, it is not replaced.<br/>Allowed values: `follow`, `copy`, `copy+force`, `skip`.<br/>Default: `follow`.<br/>(`--symbolic-links={enum}`) | | `tags` | `Hash` | Metadata for transfer as JSON. Key `aspera` is reserved. Key `aspera.xfer_retry` specifies a "retry timeout" but is not actually used for node API initiated transfers.<br/>(`--tags64=(conversion){object}`) | | `tags64` | `String` | Base64 string for `tags`.<br/>(A, T)<br/>(`--tags64={string}`) | | `target_rate` | `String, Integer` | Pseudo-parameter: desired transfer rate in bps, with an optional unit suffix. Converted to `target_rate_kbps` (rounded down) before the transfer is started, overriding it if both are present. Also applies to `async` (sync). Accepted formats: integer (bps), or integer followed by `k`/`K` (x1000), `m`/`M` (x1000000), `g`/`G` (x1000000000). Examples: `100000000`, `100000k`, `100m`, `1g`. | | `target_rate_cap_kbps` | `Integer` | Maximum target rate for incoming transfers, in kilobits per second. Returned by upload/download_setup node API.<br/>(C, T) | | `target_rate_kbps` | `Integer` | Specifies desired speed for the transfer.<br/>(`-l {integer}`) | | `title` | `String` | Title of the transfer.<br/>(C, N, T) | | `token` | `String` | Authorization token. Type is Bearer, Basic or ATM.<br/>(env:`ASPERA_SCP_TOKEN`) | | `use_ascp4` | `Bool` | Specify version of protocol. `ascp4` will be deprecated.<br/>Default: `false`.<br/>(A, N, T) | | `use_system_ssh` | `Bool` | Use an external `ssh` program instead of the built-in `libssh2` implementation to establish the connection to the remote host. The desired `ssh` program must be in the environment's `PATH`.<br/>To enable debugging of the `ssh` process, supply `-DD` and `--ssh-arg=-vv` arguments to `ascp`.<br/>(A, T)<br/>(`-SSH`) | | `wss_enabled` | `Bool` | Server has Web Socket service enabled.<br/>(special:`--ws-connect`) | | `wss_port` | `Integer` | TCP port used for Web Socket service feed. | | `xfer_max_retries` | `Integer` | Maximum number of retries, for node API initiated transfers. Shall not exceed `aspera.conf` parameter `transfer_manager_max_retries` (default 3).<br/>(N) | #### Destination folder for transfers The destination folder is set by `ascli` by default to: - `.` for downloads - `/` for uploads It is specified by the [**transfer-spec**](#transfer-specification) parameter `destination_root`. As such, it can be modified with option: `--ts=@json:'{"destination_root":"<PATH>"}'`. The option `to_folder` provides an equivalent and convenient way to change this parameter: `--to-folder=<PATH>`. #### List of files for transfers When uploading, downloading or sending files, the user must specify the list of files to transfer. By default, the list of files to transfer is provided on the command line. The list of (source) files to transfer is specified by ([Extended Value](#extended-value-syntax)) option `sources` (default: `@args`). The list is either the list of source files, or a combined source/destination list (see below) depending on value of option `src_type` (default: `list`). In `ascli`, all transfer parameters, including file list, are provided to the transfer agent in a [**transfer-spec**](#transfer-specification) so that execution of a transfer is independent of the transfer agent (direct, connect, node, transfer daemon...). So, eventually, the list of files to transfer is provided to the transfer agent using the [**transfer-spec**](#transfer-specification) field: `"paths"` which is a list (array) of pairs of `"source"` (mandatory) and `"destination"` (optional). The `sources` and `src_type` options provide convenient ways to populate the transfer spec with the source file list. Possible values for option `sources` are: - `@args` : (default) the list of files (or file pair) is directly provided on the command line (after commands): unused arguments (not starting with `-`) are considered as source files. > [!IMPORTANT] > When using `@:` to build a command parameter and `--sources=@args` (default), > the `END` marker is required between the `@:` block and the file list. > See [Positional Arguments with Dot-path](#positional-arguments-with-dot-path). **Example**: ```shell ascli server upload ~/mysample.file secondfile ``` This is the same as (with default values): ```shell ascli server upload --sources=@args --src-type=list ~/mysample.file secondfile ``` - An [Extended Value](#extended-value-syntax) with type **`Array` of `String`** > [!TIP] > An [Extended Value](#extended-value-syntax) can be tested with the command `config echo` Examples: - Using [Extended Value](#extended-value-syntax) Create the file list: ```shell echo ~/mysample.file > myfilelist.txt echo secondfile >> myfilelist.txt ``` Use the file list: one path per line: ```shell --sources=@lines:@file:myfilelist.txt ``` - Using JSON array ```shell --sources=@json:'["file1","file2"]' ``` - Using STDIN, one path per line ```shell --sources=@lines:@stdin: ``` - Using Ruby code (one path per line in file) ```ruby --sources=@ruby:'File.read("myfilelist.txt").split("\n")' ``` - `@ts` : the user provides the list of files directly in the `paths` field of transfer spec (option `ts`). Examples: - Using transfer spec ```shell --sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}' ``` The option `src_type` allows specifying if the list specified in option `sources` is a simple file list or if it is a file pair list. > [!NOTE] > Option `src_type` is not used if option `sources` is set to `@ts` Supported values for `src_type` are: - `list` : (default) the path of destination is the same as source and each entry is a source file path - `pair` : the first element is the first source, the second element is the first destination, and so on. Example: Source file `200KB.1` is renamed `sample1` on destination: ```shell ascli server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1 ``` #### Source directory structure on destination This section is not specific to `ascli`: it describes `ascp` behavior. The transfer destination is normally expected to designate a destination folder. But there is one exception: The destination specifies the new item name when the following are met: - There is a single source item (file or folder) - Transfer spec `create_dir` is not set to `true` (`ascp` option `-d` not provided) - Destination is not an existing folder - The `dirname` of destination is an existing folder For this reason, it is recommended to set `create_dir` to `true` for consistent behavior between single and multiple item transfers; this is the default in `ascli`. If a simple source file list is provided (no `destination` in `paths`, that is, no `file_pair_list` provided), the destination folder is used as destination folder for each source file, and source file folder names are not preserved. The inner structure of source items that are folders is preserved on destination. A leading `/` on destination is ignored (relative to docroot) unless docroot is not set (relative to home). In the following table source folder `d3` contains 2 files: `f1` and `d4/f2`. | Source files | Destination | Folders on Dest. | `create_dir` | Destination Files | |-----------------|-----|-----|------|-----------------------------------| | `f1` | `d/f` | - | false | Error: `d` does not exist. | | `f1` | `d/f` | `d` | false | `d/f` (renamed) | | `f1` | `d/f/.` | `d` | false | `d/f` (renamed) | | `f1` | `d/f` | `d/f` | false | `d/f/f1` | | `f1` `f2` | `d` | `d` | false | `d/f1` `d/f2` | | `d3` | `d` | - | false | `d/f1` `d/f2` (renamed) | | `f1` | `d` | - | true | `d/f1` | | `f1` `f2` | `d` | - | true | `d/f1` `d/f2` | | `d1/f1` `d2/f2` | `d` | - | true | `d/f1` `d/f2` | | `d3` | `d` | - | true | `d/d3/f1` `d/d3/d4/f2` | If a file pair list is provided, then it is possible to rename or specify a different destination folder for each source (relative to the destination). If transfer spec has a `src_base`, it has the side effect that the simple source file list is considered as a file pair list, and so the lower structure of source folders is preserved on destination. | Source files | Destination | `src_base` | Destination Files | |-------------------|-------------|------------|-----------------------------| | `d1/d2/f2` `d1/d3/f3` | `d` | `d1` | `d/d2/f2` `d/d3/f3` | Advanced Example: Send files `./file1` and `./folder2/files2` to server (for example, `/Upload`) and keep the original file names and folders, that is, send `file1` to `/Upload/file1` and `files2` to `/Upload/folder2/files2`. - If files are specified as `./file1 ./folder2/files2`, then destination will be: `/Upload/file1 /Upload/files2` - One possibility is to specify a file pair list: `--src-type=pair file1 file1 folder2/files2 folder2/files2` - Another possibility is to specify a source base (transfer spec parameter `src_base`): `--ts.src_base=$PWD $PWD/file1 $PWD/folder2/files2` The `.` path cannot be used as a source base. - Similarly, create a temporary soft link (Linux): `ln -s . tmp_base` and use `--ts.src_base=tmp_base tmp_base/file1 tmp_base/folder2/files2` - One can also similarly use `--sources=@ts` and specify the list of files in the `paths` field of transfer spec with both `source` and `destination` for each file. #### Multi-session transfer Multi session, that is, starting a transfer of a file set using multiple sessions (one `ascp` process per session) is supported on `direct` and `node` agents, not yet on connect. - `--transfer=node` ```shell --ts=@json:'{"multi_session":10,"multi_session_threshold":1}' ``` Multi-session is directly supported by the node daemon. - `--transfer=direct` ```shell --ts=@json:'{"multi_session":5,"multi_session_threshold":1,"resume_policy":"none"}' ``` > [!NOTE] > `resume_policy` set to `attr` may cause problems: `none` or `sparse_csum` shall be preferred. `ascli` starts multiple `ascp` for Multi-session using `direct` agent. When multi-session is used, one separate UDP port is used per session (refer to `ascp` manual page). #### Content protection Content protection (Client-Side Encryption at Rest, CSEAR) ensures that files remain encrypted while stored on the server. With CSEAR, the client encrypts files during upload and decrypts files during download, using a passphrase known only to the users sharing the files. - Upload: Files are encrypted on the client side before being sent to the server. - Download: Files are decrypted on the client side as they are retrieved from the server. At all times, files remain encrypted on the server; encryption and decryption occur exclusively on the client side. Activating CSEAR consists of setting transfer spec parameters: - `content_protection` : activate encryption (`encrypt` for upload) or decryption (`decrypt` for download) - `content_protection_password` : the passphrase to be used. Example: parameter to download a Faspex package and decrypt on the fly ```shell --ts=@json:'{"content_protection":"decrypt","content_protection_password":"<PASSWORD>"}' ``` > [!NOTE] > Faspex 5 requires package parameter `ear_enabled` set to `true` for CSEAR. > In that case the transfer spec parameter `content_protection` is automatically set. > `content_protection_password` is then required in all cases. #### Transfer Spec Examples - Change target rate ```shell --ts=@json:'{"target_rate_kbps":500000}' ``` - Override the FASP SSH port to a specific TCP port: ```shell --ts=@json:'{"ssh_port":33002}' ``` - Force HTTP fallback mode: ```shell --ts=@json:'{"http_fallback":"force"}' ``` - Activate progress when not activated by default on server ```shell --ts=@json:'{"precalculate_job_size":true}' ``` ### Transfer progress bar File transfer operations are monitored, and a progress bar is displayed on the standard error if option `progress_bar` (`Bool`) is set to `yes` (default if the standard error is a terminal). So, the progress bar is displayed even if the output of the command is redirected to a file. The same progress bar is used for any type of transfer: using `ascp`, server to server, using HTTPS, and so on. It shows the elapsed time, the percentage, the rate in megabits per second (`Mbps`, 1,000,000 bits per second, averaged over the last 5 seconds) and the estimated remaining time. As long as the total size is not known (for example, when the job size is not pre-calculated), the transferred size is shown instead of the percentage. For multi-session transfers, sessions are aggregated, and the number of running sessions is shown in brackets. Files already at destination (resumed transfer) count in the progress, but not in the rate. If the transfer fails, the progress bar stops at the reached progress, and shows `failed`. Log lines are displayed above the progress bar. Agent `direct` can display the native progress bar of `ascp` instead (see [`direct`](#agent-direct)). ### Scheduler `ascli` does not include a built-in scheduler. Automated execution should therefore rely on operating system facilities. Two common execution modes are supported: - Scheduled execution: run `ascli` commands periodically. - Daemon/service mode: run `ascli` continuously as a server. #### Creating a wrapping script Before configuring scheduling or services, it is often useful to create a wrapper script that prepares the environment and optionally enforces execution limits. This script may: - Configure environment variables (PATH, Ruby, and so on) - Enforce a maximum runtime - Centralize command options **Example** (Linux) ```shell #!/bin/bash # Load the Ruby environment if required # Forward arguments to ascli and ensure it does not exceed 30 min execution exec timeout 30m ascli "${@}" ``` Save as: ```shell /home/xfer/bin/ascli_tool ``` Then, make it executable. #### Windows: Scheduler Windows provides the [Task Scheduler](https://docs.microsoft.com/en-us/windows/win32/taskschd/task-scheduler-start-page). Tasks can be configured using: - [`schtasks.exe`](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/schtasks-create) - PowerShell module [`ScheduledTasks`](https://learn.microsoft.com/en-us/powershell/module/scheduledtasks) - `taskschd.msc` (UI) By default, Windows Task Scheduler prevents overlapping executions. #### Linux: `systemd` Timer Most modern Linux distributions use `systemd` which provides scheduling via [`timer`](https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html) units. Define a name for the job, for example: `ascli_job` as `<NAME>` below. 1. Create the service `/etc/systemd/system/<NAME>.service` ```ini [Unit] Description=Run ascli preview scan [Service] Type=oneshot ExecStart=/home/xfer/bin/ascli_tool preview scan --logger=syslog --out.level=error User=xfer Group=xfer ``` 1. Create the timer (same base name) `/etc/systemd/system/<NAME>.timer` ```ini [Unit] Description=Hourly execution of <NAME> [Timer] OnCalendar=hourly AccuracySec=1min Persistent=true [Install] WantedBy=timers.target ``` 1. Enable the timer ```shell sudo systemctl daemon-reload sudo systemctl enable --now <NAME>.timer ``` Check status: ```shell systemctl status <NAME>.timer ``` ```shell systemctl list-timers | grep <NAME> ``` Run service once to test: ```shell sudo systemctl start <NAME>.service ``` #### Linux: `cron` Scheduler Unix-like systems also provide [`cron`](https://www.man7.org/linux/man-pages/man5/crontab.5.html). Example of `crontab` for user `xfer`. ```shell crontab<<EOF 0 * * * * /home/xfer/bin/ascli_tool preview scan --logger=syslog --out.level=error 2-59 * * * * /home/xfer/bin/ascli_tool preview trevents --logger=syslog --out.level=error EOF ``` Linux also provides `anacron` for daily or hourly jobs that must run even if the system was previously offline. > [!NOTE] > Logging options are specified directly in the `cron` entry so command output remains visible when running the command manually. #### Running as system service (Daemon mode) Some commands run continuously (for example, listening on a network port). In this case, it is recommended to run `ascli` as a system service. On Linux this is typically done using [`systemd`](https://systemd.io/). For example, below `<NAME>` is `ascli_svc`. A [wrapping script](#creating-a-wrapping-script), again, is convenient: `/usr/local/bin/start_<NAME>.sh`: Service definition at `/etc/systemd/system/<NAME>.service`: ```ini [Unit] Description=My ascli daemon After=network.target [Service] ExecStart=/usr/local/bin/start_<NAME>.sh Restart=on-failure RestartSec=15 User=xfer # Optional, specify a working directory # WorkingDirectory=/path/to/working/dir # Optional, redirect output StandardOutput=journal StandardError=journal [Install] WantedBy=multi-user.target ``` Enable and start: ```shell sudo systemctl daemon-reload sudo systemctl enable --now <NAME>.service systemctl status <NAME>.service journalctl -u <NAME>.service ``` #### Preventing Concurrent Execution When running scheduled jobs, it may be necessary to ensure that only one instance of `ascli` runs at a time. This avoids situations such as: - Multiple instances accumulating and overloading the system - Duplicate transfers - Duplicate `preview` outputs ##### Option: `lock_port` `ascli` provides a built-in locking mechanism via the `lock_port` option. - `lock_port` prevents concurrent executions of the same command. - If another instance is already running and holding the specified port, the new execution exits immediately. Under the hood, this works by opening a local TCP server port. If the port is already in use, the process fails, effectively acting as a local lock. The lock is released automatically when the process exits. > [!NOTE] > When using `ascli` in containers, `lock_port` only works within the same container. > Each container has its own network namespace, so the lock does not apply across containers. Example: Test `ascli` locking by running this command in two separate terminals within 30 seconds: ```shell ascli config echo @ruby:'sleep 30' --lock-port=12345 ``` - The first instance will sleep for 30 seconds. - The second instance will exit immediately with: ```text WARN -- : Another instance is already running (Address already in use - bind(2) for "127.0.0.1" port 12345). ``` ##### OS-based concurrency protection Operating system schedulers also provide protection against overlapping runs. - Windows Task Scheduler prevents concurrent runs by default. - Linux `cron` can leverage [`flock`](https://man.cx/flock%281%29) to achieve the same effect: ```shell /usr/bin/flock -w 0 /var/cron.lock ascli ... ``` - `systemd` avoids overlaps when scheduling a `Type=oneshot` service. ```ini [Unit] Description=My ascli daemon ... [Service] ExecStart=/usr/local/bin/start_<NAME>.sh Type=oneshot RemainAfterExit=yes ... ``` > [!NOTE] > Systemd timers can schedule this service and ensure no overlapping runs occur. ### "Provençal" `ascp`, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's `open`,`read`,`write`,`close` primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is called **PVCL** (from **Provençal**, a restaurant located in Sophia Antipolis). Several **PVCL** adapters are available, one is embedded in `ascp`, the others are provided in shared libraries and must be activated. The list of supported **PVCL** adapters can be retrieved with command: ```shell ascli config ascp info --fields=@re:'^pvcl' ``` ```text process v1 shares v1 noded v1 faux v1 file v1 stdio v1 stdio-tar v1 ``` The output shows the adapters: `process`, `shares`, `noded`, `faux`, `file`, `stdio`, `stdio-tar`. Those adapters can be used wherever a file path is used in `ascp` including configuration. They act as a **pseudo drive**. The simplified format is: ```shell <ADAPTER>:///<SUB_FILE_PATH>?<ARG1>=<VAL1>&... ``` One of the adapters, used in this manual, for testing, is `faux`. It is a pseudo file system allowing generation of file data without actual storage (on source or destination). Other PVCL modules exist and are implemented in shared libraries. Those libraries are available only on HSTS. To activate a PVCL library, place the corresponding shared library in the same folder as `ascp`. Example: ```shell cp /opt/aspera/lib/pvcl/libpvcl_cloud.so $(ascli config ascp info --fields=root) ``` Then check available modules as shown previously (`ascp info`). ### `faux:` for testing This adapter can be used to simulate a file or a directory. This is a feature of `ascp`, not `ascli`. The following is an extract of the man page of `ascp`. To discard data at the destination, the destination argument is set to `faux://`. To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form: ```shell faux:///filename?filesize ``` Where: - `filename` is the name that will be assigned to the file on the destination - `filesize` is the number of bytes that will be sent (in decimal). > [!NOTE] > Characters `?` and `&` are shell special characters (wildcard and background), so `faux` file specification on command line should be protected (using quotes or `\`). > If not, the shell may give error: `no matches found` or equivalent. For all sizes, a suffix can be added (case-insensitive) to the size: k, m, g, t, p, e (values are powers of 2, for example, 1M is 2<sup>20</sup>, that is, 1 mebibyte, not megabyte). The maximum allowed value is 8\*2<sup>60</sup>. Extremely large `faux` file sizes (petabyte range and above) will likely fail due to lack of destination storage unless destination is `faux://`. To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form: ```shell faux:///dirname?<ARG1>=<VAL1>&... ``` Where: - `dirname` is the folder name and can contain `/` to specify a subfolder. - Supported arguments are: | Name | Type | Description | |----------|--------|----------------------------------------------------------------| | count | int | Number of files<br/>Mandatory | | file | string | Basename for files<br/>Default: `file` | | size | int | Size of first file.<br/>Default: 0 | | inc | int | Increment applied to determine next file size<br/>Default: 0 | | seq | enum | Sequence in determining next file size<br/>Values: `random`, `sequential`<br/>Default: `sequential` | | buf_init | enum | How source data is initialized<br/>Option `none` is not allowed for downloads.<br/>Values:`none`, `zero`, `random`<br/>Default:`zero` | The sequence parameter is applied as follows: - If `seq` is `random` then each file size is: - `size +/- (inc * rand())` - Where rand is a random number between 0 and 1 - File size must not be negative; `inc` is set to `size` if it is greater than `size`. - Similarly, overall file size must be less than 8\*2<sup>60</sup>. If size + inc is greater, inc will be reduced to limit size + inc to 7\*2<sup>60</sup>. - If `seq` is `sequential` then each file size is: - `size + ((file_index - 1) * inc)` - Where first file is index 1 - So file1 is `size` bytes, file2 is `size + inc` bytes, file3 is `size + inc * 2` bytes, and so on - As with `random`, `inc` will be adjusted if `size + (count * inc)` is not less than 8\*2<sup>60</sup>. Filenames generated are of the form: `<FILE>_<00000 ... count>_<FILESIZE>` Examples: - Upload 20 gibibytes of generated data to file `myfile` in directory `/Upload` ```shell ascli server upload faux:///myfile\?20g --to-folder=/Upload ``` - Upload a file /tmp/sample but do not save results to disk (no docroot on destination) ```shell ascli server upload /tmp/sample --to-folder=faux:// ``` - Upload a faux directory `mydir` containing 1 million files, sequentially with sizes ranging from 0 to 2 Mebibyte - 2 bytes, with the base name of each file being `testfile` to /Upload ```shell ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload ``` - Upload a faux directory `mydir` containing 1000 files, of size 1 byte, with the base name of each file being `testfile` to /Upload ```shell ascli server upload "faux:///mydir?file=testfile&count=1000&size=1" --to-folder=/Upload ``` ### `file:` for growing files A growing file is a file that another program is still writing during the transfer, for example a live recording, a log file, or a stream capture. The built-in `file` PVCL adapter of `ascp` transfers such a file: `ascp` sends the data as the file grows, and completes the transfer when the file stops changing. This is a feature of `ascp`, not `ascli`. For more information, see the [HSTS `ascp` command reference](https://www.ibm.com/docs/en/ahts/4.4.x?topic=line-ascp-command-reference). #### Syntax of the `file:` URI To transfer a growing file, specify the source file as a `file:` URI that contains the `grow` parameter: ```shell file:///<PATH>?grow=<WAIT_TIME>[&<PARAMETER>=<VALUE>...] ``` The path of the file follows the prefix `file:///`: | Path | Example URI | File | |----------|------------------------------------|----------------------------------| | Relative | `file:///live.log?grow=60` | `live.log` in the current folder | | Absolute | `file:////var/log/app.log?grow=60` | `/var/log/app.log` | An absolute path starts with `/`, so the URI of an absolute path contains four slashes. The URI supports the following parameters: | Parameter | Description | |-----------------|----------------------------------------------------------------| | `grow` | Wait time, in seconds: the transfer completes when the file does not change during this time.<br/>Required.<br/>If the value is not a number, for example `grow=default`, the wait time is 10 seconds. | | `wait_start` | Start of the wait time:<br/>- `mtime`: the last modification time of the file.<br/>- `null_read`: the time when `ascp` reaches the end of the file, that is, when a read of the file returns no data.<br/>Default: `mtime` | | `confirm_stop` | If `true`, the program that writes the file confirms the end of the file. See [Confirming the end of a growing file](#confirming-the-end-of-a-growing-file).<br/>Ignored if `wait_start` is `null_read`.<br/>Default: `false` | | `max_grow_size` | Maximum size of the file, in bytes.<br/>Accepts the suffixes `k`, `m`, `g`, `t`, `p`, and `e` (powers of 1024).<br/>Default: 2<sup>63</sup> - 1 bytes, or the maximum file size of the file system if lower. | > [!IMPORTANT] > The growing file must be on a native file system, that is, a file system that the operating system accesses directly. > [!TIP] > On the command line, enclose the URI in single quotes: the characters `?` and `&` have a special meaning for the shell. #### Completion of the transfer `ascp` sends the data as the program writes it. When `ascp` has sent all the data that the file contains, it starts to count the wait time. If the file grows during the wait time, `ascp` sends the new data and starts to count the wait time again. When the wait time ends without a change to the file, the transfer completes. As a result: - The command completes after the wait time that follows the last write. For example, with `grow=120`, the command completes about 2 minutes after the program stops writing. - If the program pauses for longer than the wait time, the transfer completes before the end of the file, and the destination file is incomplete. Set a wait time that is longer than the longest pause of the program. To complete the transfer without waiting, use `confirm_stop`. #### Confirming the end of a growing file With `confirm_stop=true`, the program that writes the file signals the end of the file, so the transfer does not wait. 1. Start the transfer with `confirm_stop=true` and a wait time that is longer than the longest pause of the program: ```shell ascli server upload live.log --ts.source_root='file:///?grow=600&confirm_stop=true' --to-folder=/Upload ``` 2. When the program finishes writing the file, set the modification time of the file to a time earlier than the wait time. For example: ```shell touch -t 200001010000 live.log ``` The transfer completes immediately. > [!IMPORTANT] > With `confirm_stop=true`, if the wait time ends before the program sets the modification time, `ascp` considers the file invalid and the transfer fails. To report an error in the file, with or without `confirm_stop`, the program sets the modification time of the file to 0, that is, `1970-01-01 00:00:00 UTC`. `ascp` then fails the transfer of the file. For example: ```shell TZ=UTC touch -t 197001010000 live.log ``` #### Uploading a growing file `ascp` activates the growing file mode only if it finds the `grow` parameter in one of the following locations: - The first source on its command line. - The source prefix: `ascp` option `--source-prefix`, set by transfer spec parameter `source_root`. - The docroot of the transfer user, on the server. `ascp` does not look for the `grow` parameter in a file list. By default, the [`direct`](#agent-direct) agent provides the sources to `ascp` in a file list. As a result, if you specify a `file:` URI as a source with the default options, the transfer fails with the error `Session initiation failed, fail to set pvcl`. Use one of the following methods. The examples use the [`direct`](#agent-direct) agent. - **Set the URI as source prefix** (recommended) Set transfer spec parameter `source_root` to a `file:` URI that contains the folder of the files and the parameters. Specify the sources as paths relative to this folder. `ascp` inserts each source path before the parameters: for example, `file:////var/log?grow=120` and `app.log` give `file:////var/log/app.log?grow=120`. All the files of the transfer use the same parameters. To upload `live.log` from the current folder: ```shell ascli server upload live.log --ts.source_root='file:///?grow=120' --to-folder=/Upload ``` To upload `/var/log/app.log`: ```shell ascli server upload app.log --ts.source_root='file:////var/log?grow=120' --to-folder=/Upload ``` - **Set the URI as source on the command line** Set transfer parameter `file_list` to `false`, so that `ascli` places the source on the `ascp` command line instead of in a file list. Use this method for a single file. ```shell ascli server upload 'file:///live.log?grow=120' --to-folder=/Upload --transfer.file_list=false ``` > [!TIP] > To display the native progress bar of `ascp` instead of the progress bar of `ascli`, add `--transfer.quiet=false`. #### Downloading a growing file For a download, the growing file is on the server. The transfer user on the server usually has a docroot, and the server does not resolve a `file:` URI from the client in this docroot: the transfer fails with the error `No such file or directory`. Instead, the server administrator sets the parameters in the docroot of the transfer user. For example, on the server: ```shell asconfigurator -x 'set_user_data;user_name,xfer_live;absolute,file:////data/live?grow=120' ``` Then, the client downloads the file with a path relative to the docroot, as for any other file: ```shell ascli server download live.log --to-folder=. ``` All the transfers of this user are then growing file transfers, so use a dedicated transfer user. This method also applies to transfers that use a token, for example with the Node API or Aspera on Cloud: for a download with a token, `ascp` does not accept a `file:` URI as source prefix. If the transfer user has no docroot and the transfer does not use a token, you can also use the upload methods, with absolute paths on the server. ### Usage ```text ascli -h NAME ascli -- a command line tool for Aspera Applications (v4.29.0.pre) SYNOPSIS ascli [GLOBAL_OPTIONS] <command> [OPTIONS] [ARGS] DESCRIPTION Use Aspera application to perform operations on command line. Documentation and examples: https://rubygems.org/gems/aspera-cli execute: ascli conf doc or visit: https://ibm.biz/ascli-doc source repo: https://github.com/IBM/aspera-cli ENVIRONMENT VARIABLES Any option can be set as an environment variable, refer to the manual COMMANDS To list first level commands, execute: ascli Note that commands can be written shortened (provided it is unique). OPTIONS Options begin with a '-' (minus), and value is provided on command line. Special values are supported beginning with special prefix @pfx:, where pfx is one of: val, base64, csvt, env, file, uri, json, lines, list, none, path, re, ruby, s, secret, stdin, yaml, zlib, extend, preset, vault Dates format is 'YYYY-MM-DD HH:MM:SS', or 'now' or '-<num>h' ARGS Some commands require mandatory arguments, e.g. a path. EXIT STATUS 0 Success 1 Other error 2 Command line error: unknown command or option, invalid or missing argument 3 Authentication or authorization failure: SSH, HTTP 401 or 403, OAuth grant 4 Resource not found, or several resources match the identifier, HTTP 404 5 Transfer failure 6 Network or TLS failure: name resolution, connection, timeout, certificate 7 Other error returned by a remote API 130 Interrupted by user (Ctrl-C) OPTIONS: global --interactive=yes|no Use interactive input of missing params --ask-options=yes|no Ask even optional options --out=HASH Output rendering options (dot-notation: format, level, file, fields, select, query, table[.pivot], flat, secrets, colors, utf8, img) --display=info|data|error Output only some information (deprecated after 4.27.0: use --out.level) --format=ENUM Output format (also: --out.format) --output=VALUE Destination for results (deprecated after 4.27.0: use --out.file) --fields=LIST Comma separated list of: fields, or ALL, or DEF (also: --out.fields) --select=HASH Select only some items in lists: column, value (also: --out.select) --table-style=HASH (Table) Display style (deprecated after 4.27.0: use --out.table) --flat-hash=yes|no (Table) Display deep values as additional keys (deprecated after 4.27.0: use --out.flat) --multi-single=no|yes|single (Table) Control how object list is displayed as single table, or multiple objects (deprecated after 4.27.0: use --out.table.pivot) --show-secrets=yes|no Show secrets on command output (deprecated after 4.27.0: use --out.secrets) --image=HASH Options for displaying images and thumbnails in the terminal (deprecated after 4.27.0: use --out.img) -h, --help Show this message --show-config Display parameters used for the provided action -v, --version Display version --ui=HASH User interface (dot-notation: mode, web_auth, command) --invalid-characters=VALUE Replacement character and invalid filename characters --log-level=ENUM Log level --log-format=VALUE Log formatter --logger=stderr|stdout|syslog Logging method --log=HASH Logging options (dot-notation: level, type, format, secrets) --security=HASH Security restrictions (dot-notation: level, allowed_paths) --lock-port=INT Prevent dual execution of a command, e.g. in cron --once-only=yes|no Process only new items (some commands) --dry-run=yes|no Do not perform changes, only show what would be done --log-secrets=yes|no Show passwords in logs --temp=HASH Temporary files (dot-notation: folder, clean, expiry) --clean-temp=yes|no Cleanup temporary files on exit (deprecated after 4.27.5: use --temp.clean) --temp-folder=VALUE Temporary folder (deprecated after 4.27.5: use --temp.folder) --pid-file=VALUE Write process identifier to file, delete on exit --parser=none|json|ruby|yaml Default parser for structured parameters and options --home=VALUE Home folder for tool --config-file=VALUE Path to YAML file with preset configuration --secret=VALUE Secret for access keys --vault=HASH Secret vault configuration --vault-password=VALUE Vault password --progress-bar=yes|no Display progress bar --fpac=VALUE Proxy auto configuration script --proxy-credentials=LIST HTTP proxy credentials for fpac: user, password --sql=VALUE SQL suffix appended to sqlite3 queries for admin subcommands (e.g. WHERE clause) --ascp-logs=HASH Ascp: log analysis (dot-notation: period, zero_bytes, mtu, resolve, thresholds, timezone, health_minutes, remote_max_size) -P, --preset=VALUE Load the named option preset from current config file --version-check-days=INT Period in days to check new version (zero to disable) --plugin-folder=VALUE Folder where to find additional plugins --sdk-url=VALUE Ascp: URL to get Aspera Transfer Executables --locations-url=LIST Ascp: URLs to get download locations of Aspera Transfer Daemon --sdk-folder=VALUE Ascp: Path to folder with ascp (or product with "product:") --smtp=HASH SMTP email server configuration --notify-to=VALUE Email: Recipient for notification of transfers --notify-template=VALUE Email: ERB template for notification of transfers --cache-tokens=yes|no Save and reuse OAuth tokens --expand-mounts=yes|no Commands: list commands of sub-trees provided by another plugin -N, --no-default Do not load default configuration for plugin --insecure=yes|no HTTP/S: Do not validate any certificate --ignore-certificate=LIST HTTP/S: Do not validate certificate for these URLs --warn-insecure=yes|no HTTP/S: Issue a warning if certificate is ignored --cert-stores=LIST HTTP/S: List of folder with trusted certificates --http-options=HASH HTTP/S connection parameters for REST calls (not `ascp` WSS) --http-proxy=VALUE HTTP/S: URL for proxy with optional credentials --query=HASH Additional filter for for some commands (list/delete) --bulk=yes|no Bulk operation (only some) --bfail=yes|no Bulk operation error handling --ts=HASH Override transfer spec values --to-folder=VALUE Destination folder for transferred files --sources=VALUE How list of transferred files is provided (@args,@ts,Array) --src-type=list|pair Type of file list --transfer=HASH Transfer agent type, or agent parameters with optional agent key --transfer-info=HASH Parameters for transfer agent (deprecated after 4.26.2: use --transfer instead) PLUGINS alee Aspera License Entitlement Engine aoc Aspera on Cloud ats Aspera Transfer Service console Console cos IBM Cloud Object Storage faspex5 Faspex v5 faspio faspio Gateway httpgw HTTP Gateway mcp Model Context Protocol Server node HSTS Node API orchestrator Orchestrator preview Preview server HSTS Fasp/SSH shares Shares transferd Transfer Daemon ``` ### Bulk creation and deletion of resources Bulk creation and deletion of resources are possible using option `bulk` (`yes`,`no`(default)). In that case, the operation expects an `Array` of `Hash` instead of a simple `Hash` using the [Extended Value](#extended-value-syntax) syntax. This option is available only for some resources: if you need it, try and see if the entities you create or delete support this option. ### Plugins `ascli` uses a plugin mechanism. The first level command (just after `ascli` on the command line) is the name of the concerned plugin which will execute the command. Each plugin usually represents commands sent to a specific application. Available plugins can be found using command: ```shell ascli config plugins list ``` ```text ╭────────┬────────┬────────┬─────────────────────────────────────────────────╮ │ plugin │ detect │ wizard │ path │ ╞════════╪════════╪════════╪═════════════════════════════════════════════════╡ │ shares │ ✓ │ ✓ │ .../aspera-cli/lib/aspera/cli/plugins/shares.rb │ │ node │ ✓ │ ✓ │ .../aspera-cli/lib/aspera/cli/plugins/node.rb │ ... ╰────────┴────────┴────────┴─────────────────────────────────────────────────╯ ``` Most plugins will take the URL option: `url` to identify their location. REST APIs of Aspera legacy applications (Aspera Node, Shares, Console, Orchestrator) use simple username/password authentication: HTTP Basic Authentication using options: `username` and `password`. Aspera on Cloud and Faspex 5 rely on OAuth. By default, plugins are looked up in folders specified by (multi-value) option `plugin_folder`: ```shell ascli --show-config --fields=plugin_folder ``` You can create the skeleton of a new plugin like this: ```shell ascli config plugins create foo . ``` ```text Created ./foo.rb ``` ```shell ascli --plugin-folder=. foo ``` ### Plugins vs Transfer Agents Plugins typically represent a specific remote application on which `ascli` can operate, while transfer agents are the underlying components that handle the actual data transfer. A given remote application can sometimes be both a plugin and a transfer agent. For example: `node` and `httpgw` are both plugins and transfer agents. A plugin is invoked as the first positional argument in a command line. A Transfer Agent is used by setting the option `transfer` (for example, `--transfer=node`). ### Command line parsing, Special Characters `ascli` is typically executed in a shell, either interactively or in a script. `ascli` receives its arguments on the command line. The way arguments are parsed and provided to `ascli` depends on the operating system and shell. #### Shell parsing for Unix-like systems: Linux, macOS, AIX Linux command line parsing is well-defined: It is fully documented in the shell's documentation. On Unix-like environments, this is typically a POSIX-like shell (`bash`, `zsh`, `ksh`, `sh`). A c-shell (`csh`, `tcsh`) or other shell can also be used. In this environment, the shell parses the command line, possibly replacing variables, and so on. See [bash shell operation](https://www.gnu.org/software/bash/manual/bash.html#Shell-Operation). The shell builds the list of arguments and then `fork`/`exec` Ruby with that list. Ruby receives the list of command line arguments from the shell and gives it to `ascli`. Special character handling (quotes, spaces, env vars, ...) is handled by the shell for any command executed. #### Shell parsing for Windows On Windows, command line parsing first depends on the shell used (see next sections). MS Windows command line parsing differs from Unix-like systems because Windows does not provide a list of arguments to the executable (Ruby): it provides the whole command line as a single string, but the shell may interpret some special characters. So command line parsing is not handled by the shell (`cmd.exe`), not handled by the operating system, but it is handled by the executable (Ruby). Typically, Windows executables use the [Microsoft library for this parsing](https://learn.microsoft.com/en-us/cpp/cpp/main-function-command-line-args). As far as `ascli` is concerned: the executable is Ruby. It has its own parsing algorithm, close to a Linux shell parsing. Thankfully, `ascli` provides a command to check the value of an argument after parsing: `config echo`. One can also run `ascli` with option `--log-level=debug` to display the command line after parsing. It is also possible to display arguments received by Ruby using this command: ```batchfile C:> ruby -e 'puts ARGV' "Hello World" 1 2 ``` ```text Hello World 1 2 ``` > [!NOTE] > Use `pp` instead of `puts` to display as Ruby `Array`. Once the shell has dealt with the command line "special" characters for it, the shell calls Windows' `CreateProcess` with the whole command line as a single string. (Unlike Unix-like systems where the command line is split into arguments by the shell.) It's up to the program to split arguments: - [Windows: How Command Line Parameters Are Parsed](https://daviddeley.com/autohotkey/parameters/parameters.htm#RUBY) - [Understand Quoting and Escaping of Windows Command Line Arguments](https://web.archive.org/web/20190316094059/http://www.windowsinspired.com/understanding-the-command-line-string-and-arguments-received-by-a-windows-program/) `ascli` is a Ruby program, so Ruby parses the command line (received with `GetCommandLineW`) into arguments and provides them to the Ruby code (`$0` and `ARGV`). Ruby vaguely follows the Microsoft C/C++ parameter parsing rules. (See `w32_cmdvector` in Ruby source [`win32.c`](https://github.com/ruby/ruby/blob/master/win32/win32.c#L1766)): - Space characters: split arguments (space, tab, newline) - Backslash: `\` escapes a single special character - Globbing characters: `*?[]{}` for file globbing - Double quotes: `"` - Single quotes: `'` #### Shell parsing for Windows: `cmd.exe` The following examples give the same result on Windows using `cmd.exe`: - Single quote protects the double quote ```batchfile ascli config echo @json:'{"url":"https://..."}' ``` - Triple double quotes are replaced with a single double quote ```batchfile ascli config echo @json:{"""url""":"""https://..."""} ``` - Double quote is escaped with backslash within double quotes ```batchfile ascli config echo @json:"{\"url\":\"https://...\"}" ``` `cmd.exe` handles some special characters: `^"<>|%&`. It handles I/O redirection (`<>|`), shell variables (`%`), and multiple commands (`&`). Eventually, all those special characters are removed from the command line unless escaped with `^` or `"`. `"` are kept and given to the program. #### Shell parsing for Windows: PowerShell For PowerShell, the behavior depends on the version (5.1, 7.3+). A difficulty is that PowerShell parses the command line for its own use and manages special characters, but then it passes the command line to the program (Ruby) as a single string, possibly without the special characters. If not using PowerShell features (for example, variables), one can use the "stop-parsing" token `--%`. Details can be found here: - [Passing arguments with quotes](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_parsing#passing-arguments-that-contain-quote-characters) - [Quoting rules](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_quoting_rules) ##### PowerShell 5 - Check your PowerShell version: ```powershell $psversiontable.psversion.Major ``` ```text 5 ``` The following examples give the same result on Windows using PowerShell 5: ```text ╭───────┬───────╮ │ field │ value │ ╞═══════╪═══════╡ │ x │ true │ │ k │ v │ ╰───────┴───────╯ ``` - Use PowerShell argument `--%` to place PowerShell in "stop-parsing" mode. ```powershell ascli config echo --% @json:'{"k":"v","x":true}' ``` - Triple double quotes are replaced with a single double quote in normal mode: ```powershell ascli config echo @json:'{"""k""":"""v""","""x""":true}' ``` - To insert PowerShell variables in the JSON string, one can do: ```powershell $var="v" ascli config echo $('@json:{"""k""":"""' + $var + '""","""x""":true}') ``` ##### PowerShell 7 - Check your PowerShell version: ```powershell $psversiontable.psversion.Major ``` ```text 7 ``` The following examples give the same result on Windows using PowerShell 7: - Use PowerShell argument `--%` to place PowerShell in "stop-parsing" mode. ```powershell ascli config echo --% @json:{"k":"v","x":true} ``` - Single quote protects double quote in normal mode: ```powershell ascli config echo @json:'{"k":"v","x":true}' ``` - To insert PowerShell variables in the JSON string, one can do: ```powershell $var="v" ascli config echo $('@json:{"k":"' + $var + '","x":true}') ``` - Use PowerShell structure and then convert to JSON string: ```powershell $var="v" ascli config echo "@json:$(@{ k = $var; x = $true } | ConvertTo-Json -Compress)" ``` #### Extended Value (JSON, Ruby, ...) Some values provided to `ascli` (options, **Command Parameters**) are expected to be [Extended Values](#extended-value-syntax), that is, not a simple `String`, but a composite structure (`Hash`, `Array`). For small structures with a few keys, the [dot-path](#dot-path-notation) `key.subkey=value` notation (using `@:` to collect positional arguments, or `--opt.key=value` directly for options) is often the most convenient: it requires no quoting and reads naturally on the command line. For larger or more complex structures (deep nesting, arrays of hashes, values copied from API documentation), the `@json:` modifier is typically a better fit. `@json:` expects a [JSON](https://www.json.org/) value; note that `"` is used to enclose a `String` in JSON, which may be difficult to specify in shells: enclose the whole argument in single quotes to avoid this. Any option or **Command Parameter** expecting a `Hash` value accepts the special value `help` to display its schema. See [Schema Discovery with `help`](#schema-discovery-with-help). #### Using a shell variable, parsed by shell, in an Extended Value To be evaluated by the shell, the shell variable must not be in single quotes. Enclose the variable in double quotes so that it results in a single argument for `ascli` even if its value contains spaces: `bash` splits the result of an unquoted variable expansion into several words (`zsh` does not, by default). > [!NOTE] > We use a simple shell variable in this example. > It does not need to be exported as an environment variable. ```shell MYVAR="Hello World" ascli config echo @json:'{"title":"'"$MYVAR"'"}' --format=json ascli config echo "@json:{\"title\":\"$MYVAR\"}" --format=json ``` ```json {"title":"Hello World"} ``` #### Double quote in strings in command line Double quote is a shell special character. Like any shell special character, it can be protected either by preceding with a backslash or by enclosing in a single quote. ```shell ascli config echo \" ascli config echo '"' ``` ```text " ``` Double quote in JSON is a little tricky because `"` is special both for the shell and JSON. Both shell and JSON syntax allow protecting `"`, but only the shell allows protection using single quote. ```shell ascli config echo @json:'"\""' --format=text ascli config echo @json:\"\\\"\" --format=text ascli config echo @ruby:\'\"\' --format=text ``` ```text " ``` Here a single quote or a backslash protects the double quote to avoid shell processing, and then an additional `\` is added to protect the `"` for JSON. But as `\` is also shell special, then it is protected by another `\`. #### Shell and JSON or Ruby special characters in Extended Value Construction of values with special characters is done like this: - First select a syntax to represent the [Extended Value](#extended-value-syntax), for example, JSON or Ruby - Write the expression using this syntax, for example, using JSON: ```json {"title":"Test \" ' & \\"} ``` Or, using Ruby: ```ruby {"title"=>"Test \" ' & \\"} {'title'=>%q{Test " ' & \\}} ``` Both `"` and `\` are special characters for JSON and Ruby and can be protected with `\` (unless Ruby's extended single quote notation `%q` is used). - Then, since the value will be evaluated by shell, any shell special characters must be protected, either using preceding `\` for each character to protect, or by enclosing in single quote: ```shell ascli config echo @json:{\"title\":\"Test\ \\\"\ \'\ \&\ \\\\\"} --format=json ascli config echo @json:'{"title":"Test \" '\'' & \\"}' --format=json ascli config echo @ruby:"{'title'=>%q{Test \" ' & \\\\}}" --format=json ``` ```json {"title":"Test \" ' & \\"} ``` #### Reading special characters interactively If `ascli` is used interactively (a user typing on terminal), values can be entered by the user: ```shell ascli config echo @ruby:"{'title'=>gets.chomp}" --format=json ``` `gets` is Ruby's method of terminal input (terminated by `\n`), and `chomp` removes the trailing `\n`. #### Command line arguments from a file If you need to provide a list of command line arguments from lines that are in a file, on Linux you can use the `xargs` command: ```shell xargs -a lines.txt -d \\n ascli config echo ``` This is equivalent to execution of: ```shell ascli config echo [line1] [line2] [line3] ... ``` If there are spaces in the lines, those are not taken as separator, as we provide option `-d \\n` to `xargs`. #### Extended value using special characters read from environment variables or files Using a text editor or shell: create a file `title.txt` (and env var) that contains exactly the text required: `Test " ' & \` : ```shell export MYTITLE='Test " '\'' & \' echo -n $MYTITLE > title.txt ``` Using those values will not require any escaping of characters since values do not go through shell or JSON parsing. If the value is to be assigned directly to an option of ascli, then you can directly use the content of the file or env var using the `@file:` or `@env:` readers: ```shell ascli config echo @file:title.txt --format=text ascli config echo @env:MYTITLE --format=text ``` ```text Test " ' & \ ``` If the value to be used is in a more complex structure, then the `@ruby:` modifier can be used: it allows any Ruby code in expression, including reading from file or env var. In those cases, there is no character to protect because values are not parsed by the shell, or JSON or even Ruby. ```shell ascli config echo @ruby:"{'title'=>File.read('title.txt')}" --format=json ascli config echo @ruby:"{'title'=>ENV['MYTITLE']}" --format=json ``` ```json {"title":"Test \" ' & \\"} ``` ## Plugin: `aoc`: IBM Aspera on Cloud The `aoc` plugin enables integration with IBM Aspera on Cloud. Because the Aspera on Cloud API requires **OAuth 2.0** for authentication, HTTP Basic authentication is **not** supported. > [!NOTE] > This plugin is also used for Aspera Enterprise Web Apps (AEWA). ### AoC configuration: Using Wizard `ascli` includes an interactive configuration [wizard](#wizard) to simplify the OAuth setup. While manual configuration is possible, using the wizard is the recommended method for most deployments. #### Prerequisites To complete the setup, you must have: - **The URL or organization name** of your Aspera on Cloud instance: - `_your_organization_` - `_your_organization_.ibmaspera.com` - `https://_your_organization_.ibmaspera.com` - **User Credentials**: Access to the Aspera on Cloud web interface to update your Public Key. #### Wizard Invocation The wizard accepts an optional argument, a `Hash` with keys: - `url`: The organization name or the full FQDN. - `plugin`: Specify `aoc` to bypass automatic application detection. - `preset`: Name of the preset to create. If the URL is omitted, the wizard will prompt for it interactively. #### Example: Standard Configuration During the process, the wizard generates or identifies a PEM-formatted Public Key. You must copy this key (including the `BEGIN` and `END` headers) and paste it into the **Public Key** field in the Aspera on Cloud UI. Once saved in the UI, return to the terminal and press Enter. > [!IMPORTANT] > In the following example, replace `https://<ORG_NAME>.ibmaspera.com` with your actual organization URL. > `(User)` represents the user menu on the top right of the page. ```console $ ascli config wizard Using: Aspera on Cloud at https://<ORG_NAME>.ibmaspera.com Private RSA key, e.g. @file:<path> (leave empty to generate): option: private_key> Using existing key: /home/john/.aspera/ascli/<PKEY_NAME>.pem Please Log in as user john@example.com at: https://<ORG_NAME>.ibmaspera.com Navigate to: (User) → Account Settings → Profile → Public Key Check or update the value to (including BEGIN/END lines): -----BEGIN PUBLIC KEY----- MIICIjANBg.... .... ....kCAwEAAQ== -----END PUBLIC KEY----- Once updated or validated, press [Enter]. Using global client_id. Preparing preset: aoc<ORG_NAME>ibmaspera_com_john_example_com Setting config preset as default for aoc You can test with: ascli aoc user profile show Saving config file. ``` #### Advanced Options - **Custom Client Identification**: By default, `ascli` uses a pre-registered global client ID. To create and use an organization-specific integration, give the option `--use-generic-client=no` of plugin `aoc` to the wizard command. - **Direct Plugin Targeting**: To limit detection to the `aoc` plugin and skip discovery, provide both the URL and plugin name: ```shell ascli config wizard @: url=<ORG_NAME> plugin=aoc ``` #### Verifying the Configuration Upon successful completion, the wizard creates a configuration preset and assigns it as the default for the `aoc` plugin. To verify the current configuration presets, execute: ```shell ascli config preset list ``` To test the connection and display your profile information: ```shell ascli aoc user profile show ``` ### AoC configuration: Using manual setup > [!NOTE] > If you used the wizard (recommended): skip this section. #### AoC manual configuration: Details IBM Aspera on Cloud (AoC) supports several OAuth 2.0 authentication mechanisms to secure API interactions. The method used is determined by the `auth` option. Supported Authentication Methods: | Method | `auth` | Description | |----------------|--------|-------------| | JSON Web Token | `jwt` | **Recommended**. Uses a private RSA key for secure, headless authentication.<br/>Ideal for automation and scripts. | | Web-based | `web` | Redirects the user to a browser for interactive login.<br/>Suitable for manual operations on local workstations. | | URL Token | - | Facilitates authentication for external users via public link tokens. | `ascli` automatically manages the lifecycle of Bearer tokens. Once a token is generated, it is cached in the local configuration folder (option `cache_tokens`). The tool reuses valid tokens and automatically triggers a refresh or regeneration when they expire, according to the expiration policy defined by the AoC administrator. #### API Client Registration > [!NOTE] > If you use the pre-registered `client_id` and `client_secret`, skip this and do not set them in next section. You can use a specific OAuth API `client_id`. The first step is to declare `ascli` in Aspera on Cloud using the admin interface. ([AoC documentation: Registering an API Client](https://ibmaspera.com/help/admin/organization/registering_an_api_client)). To register with web-based authentication (auth=web): - Open a web browser, log in to your instance: for example, `https://<ORG_NAME>.ibmaspera.com/` (use your actual AoC instance URL) - Go to (Apps) → Admin → Organization → Integrations - Click **Create New** - **Client Name**: `ascli` - **Redirect URIs**: `http://localhost:12345` - **Origins**: `localhost` - Uncheck **Prompt users to allow client to access** - Leave the JWT part for now - **Save** > [!NOTE] > For web based authentication, `ascli` listens on a local port (for example, specified by the `redirect_uri` parameter, in this example: `12345`), and the browser will provide the OAuth code there. > See [Web login flow](#web-login-flow). > For `ascli`, HTTP is required, and `12345` is the default port. > The host of `redirect_uri` shall be loopback: `localhost`, an address of `127.0.0.0/8`, or `[::1]`. Once the client is registered, a **Client ID** and **Secret** are created; these values are used in the next step. #### Configuration for Aspera on Cloud If you did not use the wizard, you can also manually create an [Option Preset](#option-preset) for `ascli` in its configuration file. Create an [Option Preset](#option-preset) called `<AOC_ORG>` using `ask` for interactive input (client info from previous step): ```shell ascli config preset ask <AOC_ORG> url client_id client_secret option: url> https://<ORG_NAME>.ibmaspera.com/ option: client_id> <CLIENT_ID> option: client_secret> <CLIENT_SECRET> updated: <AOC_ORG> ``` > [!NOTE] > In above example, replace `https://<ORG_NAME>.ibmaspera.com` with your actual AoC URL. (This can also be done in one line using the command `config preset update <AOC_ORG> --url=...`) Define this [Option Preset](#option-preset) as default configuration for the `aoc` plugin: ```shell ascli config preset set default aoc <AOC_ORG> ``` > [!NOTE] > Default `auth` method is `web` and default `redirect_uri` is `http://localhost:12345`. > Leave those default values. #### Authentication with private key For browser-less, private key-based authentication, use the following steps. To use JSON Web Token (JWT) for Aspera on Cloud API client authentication, a [private/public key pair](#private-key) must be used. ##### API Client JWT activation If you are not using the built-in client_id and secret, JWT needs to be authorized in Aspera on Cloud. This can be done in two ways: - Graphically - Open a web browser, log in to your instance: `https://<ORG_NAME>.ibmaspera.com/` (Use your actual AoC instance URL) - Go to Apps → Admin → Organization → Integrations - Click the previously created application - Select tab: **JSON Web Token Auth** - Modify options if necessary, for instance: activate both options in section **Settings** - **Save** - Using command line ```shell ascli aoc admin client list ``` ```text Items: 4/4 ╭────────────┬─────────────────────╮ │ id │ name │ ╞════════════╪═════════════════════╡ │ oXPUyJ7JpQ │ ALM │ │ <ID> │ web_client │ │ TDNl2bLZqw │ jwt_client │ │ VTh92i5OfQ │ iPaaS │ ╰────────────┴─────────────────────╯ ``` ```shell ascli aoc admin client modify <ID> @json:'{"jwt_grant_enabled":true,"explicit_authorization_required":false}' ``` ```text modified ``` #### User key registration The public key must be assigned to your user. This can be done in two ways as follows. ##### Graphically Open the previously generated public key located here: `$HOME/.aspera/ascli/<PKEY_NAME>.pub` - Open a web browser, log in to your instance: `https://<ORG_NAME>.ibmaspera.com/` (Use your actual AoC instance URL) - Click the user icon (top right) - Select **Account Settings** - Paste the Public Key PEM value in the **Public Key** section - Click **Submit** ##### Using command line ```shell ascli aoc admin user list ``` ```text ╭─────────┬────────────────┬────────────────────╮ │ id │ name │ email │ ╞═════════╪════════════════╪════════════════════╡ │ 1234567 │ John Doe │ john@example.com │ │ 7654321 │ Alice Saprich │ alice@example.com │ │ 1234321 │ Sponge Bob │ bob@example.com │ ╰─────────┴────────────────┴────────────────────╯ ``` ```shell ascli aoc user profile modify @: public_key=@file:~/.aspera/ascli/<PKEY_NAME>.pub ``` ```text modified ``` > [!TIP] > The `ascli aoc user profile show` command can be used to verify modifications. #### [Option Preset](#option-preset) modification for JWT To activate default use of JWT authentication for `ascli` using the [Option Preset](#option-preset), do the following: - Change auth method to JWT - Provide location of private key - Provide username to login as (OAuth **subject**) Execute: ```shell ascli config preset update <AOC_ORG> --auth=jwt --private-key=@val:@file:~/.aspera/ascli/<PKEY_NAME> --username=someuser@example.com ``` > [!NOTE] > The private key argument represents the actual PEM string. > To read the content from a file, use the `@file:` prefix. > But if the `@file:` argument is used as is, it will read the file and set in the configuration file. > So, to keep the `@file:` tag in the configuration file, the `@val:` prefix is added. After this last step, commands do not require web login anymore. #### AoC bootstrap authentication For quick access using an existing browser session, use `--auth=boot`. - Open a Web browser and log in to your AoC instance - Open the browser developer tools - In **Network**, select any request to `api.ibmaspera.com`. - In **Headers**, right-click the `Cookie` Request Header and select **Copy Value** Alternatively: - Go to the **Application** tab → **Cookies** - Copy the full cookie string, all values on one line, separated by `"; "` (semi-colon + space). For a simpler use, configure a preset with the `url` option, and optionally `username`. (If the username is not provided, then the subject from the token is used, else both must match.) Use the cookie string as the value of option `password`; an environment variable is convenient, as the value is temporary anyway: ```shell export ASCLI_PASSWORD="...; aoc.token=...; aoc.refresh=...; ..." ascli aoc user profile show --auth=boot ``` > [!NOTE] > The cookie string contains `aoc.token` (bearer JWT, mandatory) and `aoc.refresh` (refresh token, optional). > Only those two are used. > On first use, the tokens are cached locally. > Subsequent calls reuse the cache and refresh automatically: the `password` option is used only to get the `username` unless the option `username` is already provided. #### Public and private links AoC gives the possibility to generate public links for both the `Files` and `Packages` modules. Public links embed the authorization of access. Provide the public link using option `url` alone. In addition, the `Files` application supports private links. Private links require the user to authenticate. So, provide the same options as for regular authentication, and provide the private link using option `url`. A user may not be part of any workspace, but still have access to shared folders (using private links). In that case, it is possible to list those shared folders by using a value for option `workspace` equal to `@none:` or `@json:null` or `@ruby:nil`. #### AoC: First Use Once client has been registered and [Option Preset](#option-preset) created: `ascli` can be used: ```shell ascli aoc files browse / ``` ```text Current Workspace: Default Workspace (default) empty ``` ### Calling AoC APIs from command line The command `ascli aoc bearer_token` can be used to generate an OAuth token suitable to call any AoC API. This can be useful when a command is not yet available. The token is a secret: it is hidden in the output, unless option `--out.secrets=yes` is used. Example: ```shell curl -s -H "Authorization: $(ascli aoc bearer_token --out.secrets=yes)" 'https://api.ibmaspera.com/api/v1/group_memberships?embed[]=dropbox&embed[]=workspace'|jq -r '.[]|(.workspace.name + " -> " + .dropbox.name)' ``` It is also possible to get the bearer token for node, as user or as admin using: ```shell ascli aoc files bearer_token_node / ``` ```shell ascli aoc admin node bearer_token <NODE_ID> ``` ### Administration The `admin` command allows several administrative tasks (and requires admin privilege). It allows actions (create, update, delete) on **resources**: users, groups, nodes, workspaces, and so on, with the `admin <RESOURCE_TYPE>` commands. #### Listing resources The command `aoc admin <TYPE> list` lists all entities of given type. It uses paging and multiple requests if necessary. The option `query` can be optionally used. It expects a `Hash` using [Extended Value](#extended-value-syntax) syntax. For a single filter key, the dot-path option form is the most direct (for example, `--query.sort=name`). For multiple keys or values copied from API documentation, `--query=@json:{...}` is more suitable. Values are directly sent to the API call and used as a filter on server side. The following parameters are supported: | Parameter | Source | Description | |-----------|--------|----------------------------------------------------| | `q` | Native | A filter on name of resource.<br/>(case-insensitive, matches if value is contained in name) | | `sort` | Native | Name of fields to sort results, prefix with `-` for reverse order. | | `page` | Native | Do not use.<br/>Managed by `ascli`. | | `per_page`| Native | Do not use.<br/>Managed by `ascli`. | | `max` | `ascli` | Maximum number of items to retrieve. | | `pmax` | `ascli` | Maximum number of pages to request. | | other | Native | Other specific parameters depending on resource type. | > [!NOTE] > Both `max` and `pmax` are processed by `ascli` (see [option `--query`](#option---query-parameters-of-the-api)). > `ascli` will return all values using paging if not provided. > `page` and `per_page` are normally added by `ascli` to build successive API calls to get all values if there are more than 1000 > (AoC allows a maximum page size of 1000). > Other parameters depend on the type of resource (refer to AoC API) and are directly sent as parameters to the `GET` request on API. > [!TIP] > Use `--query=help` to display the full list of supported query parameters for any `list` command. > For example: `ascli aoc admin user list --query=help` > [!TIP] > Options `--out.query` and `--out.select` can also be used to further refine selection, refer to [Filtering results](#filtering-results). Examples: - List users with `laurent` in name: ```shell ascli aoc admin user list --query.q=laurent ``` - List users who logged in before a date: ```shell ascli aoc admin user list --query.q='last_login_at:<2018-05-28' ``` - List external users and sort in reverse alphabetical order using name: ```shell ascli aoc admin user list --query=@json:'{"member_of_any_workspace":false,"sort":"-name"}' ``` #### Selecting a resource Resources are identified by a unique `id` and a unique `name` (case-insensitive). To execute an action on a specific resource, select it using one of those methods: - **recommended**: give the ID directly on the command line **after the action**: `aoc admin node show 123` - Give another unique field, such as the name, using the [percent selector](#percent-selector) **after the action**: `aoc admin node show %name:abc` #### Creating a resource New resources (users, groups, workspaces, and so on) can be created using a command like: ```shell ascli aoc admin <RESOURCE_TYPE> create @json:'{<...parameters...>}' ``` Some API endpoints are described in [IBM API Hub](https://developer.ibm.com/apis/catalog?search=%22aspera%20on%20cloud%20api%22). Not all endpoints are documented there. Nevertheless, it is possible to infer the structure of the creation payload by dumping an existing resource and using the same parameters for creation. ```shell ascli aoc admin group show 12345 --format=json ``` ```json {"created_at":"2018-07-24T21:46:39.000Z","description":null,"id":"12345","manager":false,"name":"A8Demo WS1","owner":false,"queued_operation_count":0,"running_operation_count":0,"stopped_operation_count":0,"updated_at":"2018-07-24T21:46:39.000Z","saml_group":false,"saml_group_dn":null,"system_group":true,"system_group_type":"workspace_members"} ``` Remove the parameters that are set by the system (`id`, `created_at`, `updated_at`), and optional ones. And then craft your command: ```shell ascli aoc admin group create @json:'{"wrong":"param"}' ``` If the command returns an error, for example: ```text ERRR Rest: found unpermitted parameter: :wrong code: unpermitted_parameters request_id: 2a487dbc-bc5c-41ab-86c8-3b9972dfd4c4 api.ibmaspera.com 422 Unprocessable Entity ``` Remove the offending parameters and try again. > [!NOTE] > Some properties that are shown in the web UI, such as membership, are not listed directly in the resource, but instead another resource is created to link a user and its group: `group_membership` #### Access Key secrets To access some administrative actions on **nodes** (in fact, access keys), the associated secret may be required. The secret is provided using the `secret` option. For example, in a command like: ```shell ascli aoc admin node do <NODE_ID> v3 info ``` It is also possible to store secrets in the [secret vault](#secret-vault) and then automatically find the related secret using the [config finder](#configuration-finder). #### Activity The activity app can be queried with: ```shell ascli aoc admin analytics transfers ``` It supports filters and can send notifications using option `notify_to`. A template is defined using option `notify_template`: `mytemplate.erb`: ```yaml From: <%=from_name%> <<%=from_email%>> To: <<%=ev['user_email']%>> Subject: <%=ev['files_completed']%> files received Dear <%=ev[:user_email.to_s]%>, We received <%=ev['files_completed']%> files for a total of <%=ev['transferred_bytes']%> bytes, starting with file: <%=ev['content']%> Thank you. ``` The environment provided contains the following additional variable: - `ev` : all details on the transfer event Example: ```shell ascli aoc admin analytics transfers --once-only=yes --lock-port=12345 --query=@json:'{"status":"completed","direction":"receive"}' --notify-to=active --notify-template=@file:mytemplate.erb ``` Options: - `once_only`: keep track of the last date it was called, so that the next call gets only new events - `query`: filter (on API call) - `notify_to`, `notify_template`: send an email as specified by the template; the template can be placed in a file with the `@file:` modifier. > [!NOTE] > This must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. > The period is `[date of previous execution]..[now]`. #### Using ATS See the section **Examples** of [ATS](#plugin-ats-ibm-aspera-transfer-service) and substitute command `ats` with `aoc admin ats`. #### Files with type `link` Aspera on Cloud Shared folders are implemented through a special type of file: `link`. A `link` is the equivalent of a symbolic link on a file system: it points to another folder (not file). Listing a link (as the last element of the path) shows information on the link itself, not the content of the folder it points to. To list the target folder content, add a `/` at the end of the path. Example: ```shell ascli aoc files browse the_link ``` ```text Current Workspace: Default (default) ╭──────────┬──────┬────────────────┬──────┬──────────────────────┬──────────────╮ │ name │ type │ recursive_size │ size │ modified_time │ access_level │ ╞══════════╪══════╪════════════════╪══════╪══════════════════════╪══════════════╡ │ the_link │ link │ │ │ 2021-04-28T09:17:14Z │ edit │ ╰──────────┴──────┴────────────────┴──────┴──────────────────────┴──────────────╯ ``` ```shell ascli aoc files browse the_link/ ``` ```text Current Workspace: Default (default) ╭─────────────┬──────┬────────────────┬──────┬──────────────────────┬──────────────╮ │ name │ type │ recursive_size │ size │ modified_time │ access_level │ ╞═════════════╪══════╪════════════════╪══════╪══════════════════════╪══════════════╡ │ file_inside │ file │ │ │ 2021-04-26T09:00:00Z │ edit │ ╰─────────────┴──────┴────────────────┴──────┴──────────────────────┴──────────────╯ ``` #### Example: Bulk creation of users ```shell ascli aoc admin user create --bulk=yes @json:'[{"email":"dummyuser1@example.com"},{"email":"dummyuser2@example.com"}]' ``` ```text ╭───────┬─────────╮ │ id │ status │ ╞═══════╪═════════╡ │ 98398 │ created │ │ 98399 │ created │ ╰───────┴─────────╯ ``` #### Example: Find with filter and delete ```shell ascli aoc admin user list --query.q=dummyuser --fields=id,email ``` ```text ╭───────┬────────────────────────╮ │ id │ email │ ╞═══════╪════════════════════════╡ │ 98398 │ dummyuser1@example.com │ │ 98399 │ dummyuser2@example.com │ ╰───────┴────────────────────────╯ ``` ```shell ascli aoc admin user list --query.q=dummyuser --fields=id --out.level=data --format=csv | ascli aoc admin user delete @lines:@stdin: --bulk=yes ``` ```text ╭───────┬─────────╮ │ id │ status │ ╞═══════╪═════════╡ │ 98398 │ deleted │ │ 98399 │ deleted │ ╰───────┴─────────╯ ``` #### Example: Find deactivated users for more than 2 years ```shell ascli aoc admin user list --query=@ruby:'{"deactivated"=>true,"q"=>"last_login_at:<#{(DateTime.now.to_time.utc-2*365*86400).iso8601}"}' ``` To delete them use the same method as before. #### Example: Display current user's workspaces ```shell ascli aoc user workspaces list ``` ```text ╭────────┬─────────────────────────╮ │ id │ name │ ╞════════╪═════════════════════════╡ │ 45073 │ Engineering │ │ 145222 │ Marketing │ │ 149207 │ Sales │ │ 149699 │ Secret Project │ ╰────────┴─────────────────────────╯ ``` #### User client settings The `aoc user settings` sub-command manages persistent client-side settings stored server-side in AoC. ```shell ascli aoc user settings list ascli aoc user settings show <ID> ascli aoc user settings modify <ID> @json:'{"value":"..."}' ``` > [!NOTE] > The AoC API endpoint `client_settings` filters entries by two implicit criteria: > > - **User identity**: derived from the OAuth bearer token (the authenticated user). > - **Application identity**: derived from the `client_id` used during authentication. > > This means that settings stored by one client application (for example, the AoC web UI) are not visible to another (for example, `ascli` using a different `client_id`). > By default, `ascli` uses the pre-registered global `client_id` (see option `--use-generic-client` of plugin `aoc`). #### Example: Create a sub access key in a `node` Creation of a sub-access key is like creation of an access key, with the following differences: authentication to the Node API is made with an access key (the master access key), and only the path parameter is provided, relative to the storage root of the master key (`id` and `secret` are optional). ```shell ascli aoc admin node do %name:'<NODE_NAME>' v3 access_keys create @: storage.path=/folder1 ``` #### Example: Display transfer events (ops/transfer) ```shell ascli aoc admin node do <NODE_ID> v3 transfer list --query.q='*' --query.max=5 ``` Examples of query: ```json {"q":"type(file_upload OR file_delete OR file_download OR file_rename OR folder_create OR folder_delete OR folder_share OR folder_share_via_public_link)","sort":"-date"} ``` ```json {"tag":"aspera.files.package_id=LA8OU3p8w"} ``` #### Example: Display node events (events) ```shell ascli aoc admin node do <NODE_ID> v3 events ``` #### Example: Display members of a workspace ```shell ascli aoc admin workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}' ``` ```text ╭─────────────┬─────────┬──────────────────────────╮ │ member_type │ manager │ member.email │ ╞═════════════╪═════════╪══════════════════════════╡ │ user │ true │ john.curtis@email.com │ │ user │ false │ someuser@example.com │ │ user │ false │ jean.dupont@me.com │ │ user │ false │ another.user@example.com │ │ group │ false │ │ │ user │ false │ aspera.user@gmail.com │ ╰─────────────┴─────────┴──────────────────────────╯ ``` Other query parameters: ```json {"workspace_membership_through":true,"include_indirect":true} ``` #### Example: Add all members of a workspace to another workspace a- Get ID of first workspace ```shell WS1='First Workspace' WS1ID=$(ascli aoc admin workspace list --query=@json:'{"q":"'"$WS1"'"}' --select=@json:'{"name":"'"$WS1"'"}' --fields=id --format=csv) ``` b- Get ID of second workspace ```shell WS2='Second Workspace' WS2ID=$(ascli aoc admin workspace list --query=@json:'{"q":"'"$WS2"'"}' --select=@json:'{"name":"'"$WS2"'"}' --fields=id --format=csv) ``` c- Extract membership information ```shell ascli aoc admin workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"workspace_id":'"$WS1ID"'}' --format=jsonpp --out.file=ws1_members.json ``` d- Convert to creation data for second workspace: ```shell grep -Eve '(direct|effective_manager|_count|storage|"id")' ws1_members.json|sed '/workspace_id/ s/"'"$WS1ID"'"/"'"$WS2ID"'"/g' > ws2_members.json ``` Or, using `jq`: ```shell jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"}]' ws1_members.json > ws2_members.json ``` e- Add members to second workspace ```shell ascli aoc admin workspace_membership create --bulk=yes @json:@file:ws2_members.json ``` #### Example: Get users who did not log in since a date ```shell ascli aoc admin user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}' ``` ```text ╭──────────────────────────╮ │ email │ ╞══════════════════════════╡ │ John.curtis@acme.com │ │ Jean.Dupont@tropfort.com │ ╰──────────────────────────╯ ``` #### Example: List **Limited** users ```shell ascli aoc admin user list --fields=email --select=@json:'{"member_of_any_workspace":false}' ``` #### Example: Create a group, add to workspace and add user to group - Create the group and take note of `id` ```shell ascli aoc admin group create @json:'{"name":"group 1","description":"<DESCRIPTION>"}' ``` Group: `<GROUP_ID>` - Get the workspace ID ```shell ascli aoc admin workspace show %name:"<WORKSPACE_NAME>" --fields=id ``` Or (similar, but may return several entries, as it is a simple search): ```shell ascli aoc admin workspace list --query.q="<WORKSPACE_NAME>" --fields=id ``` Workspace: <WORKSPACE_ID> - Add group to workspace ```shell ascli aoc admin workspace_membership create @json:'{"workspace_id":<WORKSPACE_ID>,"member_type":"group","member_id":<GROUP_ID>}' ``` - Get a user's ID ```shell ascli aoc admin user list --query=@json:'{"q":"manu.macron@example.com"}' --fields=id --format=csv --out.level=data ``` User: <USER_ID> - Add user to group ```shell ascli aoc admin group_membership create @json:'{"group_id":<GROUP_ID>,"member_type":"user","member_id":<USER_ID>}' ``` #### Example: Perform a multi Gbps transfer between two remote shared folders In this example, a user has access to a workspace where two shared folders are located on different sites, for example, different cloud regions. First, set up the environment (skip if already done), see [AoC configuration: Using Wizard](#aoc-configuration-using-wizard): ```shell ascli config wizard @: url=https://sedemo.ibmaspera.com plugin=aoc preset=aoc_sedemo --username=someuser@example.com ``` This creates the option preset `aoc_sedemo` to allow seamless command line access and sets it as default for Aspera on Cloud. Then, create two shared folders located in two regions, in your files home, in a workspace. Then, transfer between those: ```shell ascli -Paoc_sedemo aoc files transfer push 'IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":1000000,"multi_session":10,"multi_session_threshold":1}' ``` #### Example: Delete all registration keys ```shell ascli aoc admin client_registration_token list --fields=id --format=csv|ascli aoc admin client_registration_token delete @lines:@stdin: --bulk=yes ``` ```text ╭─────┬─────────╮ │ id │ status │ ╞═════╪═════════╡ │ 99 │ deleted │ │ 100 │ deleted │ │ 101 │ deleted │ │ 102 │ deleted │ ╰─────┴─────────╯ ``` #### Example: Create a tethered Node Follow these steps to configure a new HSTS and link it to your existing Aspera on Cloud (AoC) organization. - Retrieve the Organization Public Key First, obtain the organization's public key. This key is used to verify bearer tokens generated by your organization. This key remains constant for the lifetime of your Organization. ```shell ascli aoc admin organization show --fields=oauth_token_verification_key --out.secrets=yes --out.file=mypubkey.pem ``` It can also be retrieved from an existing node: ```shell ascli aoc admin node do %name:'<NODE_NAME>' v3 access_keys show self --fields=token_verification_key --out.secrets=yes --out.file=mypubkey.pem ``` - Create an Access Key on the Self-Managed HSTS Create the local credentials on your High-Speed Transfer Server (HSTS). > [!TIP] > In this command, "master node credentials" allowing access key creation are used. > The `id` and `secret` parameters are optional. > If you omit them, you must add `--out.secrets=yes` to the command. > Record the generated secret immediately; it cannot be retrieved later, only reset. ```shell ascli node access_keys create @: id=<ACCESS_KEY_ID> secret=<SECRET> storage.type=local storage.path=/data/aoc token_verification_key=@file:mypubkey.pem ``` - Register the Node in AoC Now, register the physical server as a node object within the Aspera on Cloud administration interface. ```shell ascli aoc admin node create @: url=https://aspera.example.com access_key=<ACCESS_KEY_ID> name=testnode1 ``` - Configure Access Key Permissions Assign the necessary permissions to the root file ID for your new access key. This ensures the system and node owner have the correct authorization. > [!TIP] > The following two commands are executed using access key credentials. > If the node is configured for admin user, then add options: `--username=<ACCESS_KEY_ID> --password=<SECRET>`. ```shell ascli node access_keys do self permission / create @: access_type=user access_id='F4 System' ``` ```shell ascli node access_keys do self permission / create @: access_type=user access_id=NODE_OWNER ``` - Optional next steps To register an Aspera Event Journal (AEJ) as described in the HSTS manual, refer to: - [Create a registration key for a tethered node](#example-create-registration-key-to-register-a-tethered-node) - [The Aspera HSTS Operations GitHub repository](https://github.com/laurent-martin/aspera-hsts-operations/blob/main/README.md#create-a-node-registration-token). #### Example: Create registration key to register a tethered node The following command will create and display a secret token to register a self-managed Aspera Transfer Server: ```shell ascli aoc admin client_registration_token create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --out.secrets=yes ``` #### Example: Create an ATS Node AoC nodes are composed of two related entities: - An access key created on the Transfer Server (HSTS/ATS) - A `node` resource in the AoC application. The web UI allows creation of both entities in one shot. For more flexibility, `ascli` allows this in two separate steps. > [!NOTE] > When selecting **Use existing access key** in the web UI, this skips access key creation (first step). So, for example, the creation of a node using ATS in IBM Cloud looks like (see other example in this manual): - Create the access key on ATS The creation options are the ones of ATS API, refer to the [section on ATS](#ats-access-key-creation-parameters) for more details and examples. ```shell ascli aoc admin ats access_key create @json:'{"storage":{"type":"ibm-s3","bucket":"mybucket","credentials":{"access_key_id":"<ACCESS_KEY_ID>","secret_access_key":"<SECRET>"},"path":"/"}}' --cloud=softlayer --region=eu-de ``` Once executed, the access key `id` and `secret`, randomly generated by the Node API, are displayed. > [!NOTE] > Once returned by the API, the secret will not be available anymore, so store it securely. > ATS secrets can only be reset by asking IBM support. - Create the AoC node resource First, retrieve the ATS node address: ```shell ascli aoc admin ats cluster show --cloud=softlayer --region=eu-de --fields=transfer_setup_url --format=csv ``` Then use the returned address for the `url` key to create the AoC Node resource: ```shell ascli aoc admin node create @json:'{"name":"myname","access_key":"<ACCESS_KEY>","ats_access_key":true,"ats_storage_type":"ibm-s3","url":"https://ats-sl-fra-all.aspera.io"}' ``` Creation of a node with a self-managed node is similar, but the command `aoc admin ats access_key create` is replaced with `node access_keys create` on the private node itself. #### Example: Deactivate an application in a workspace This is a two-step procedure: 1. Find the application ID in the workspace: ```shell ascli aoc admin workspace show <WORKSPACE_ID> --query.include_aspera_apps=true --fields=packages_app.id ``` Or, alternatively: ```shell ascli aoc admin application instance list --query.aspera_app_type=packages --query.workspace_id=<WORKSPACE_ID> --fields=id --out.level=data ``` This displays the <APP_ID>. 2. Deactivate the application: ```shell ascli aoc admin application instance packages modify <APP_ID> @: enabled=false inherit_organization_app_settings=false ``` ### List of files to transfer Source files are provided as a list with the `sources` option. By default, the list of files is provided on the command line. See [File list](#list-of-files-for-transfers). ### Packages app The Aspera on Cloud **Packages** app lets you assemble copies of any number of files and folders into a digital "package" and send the package to others, pretty much like a web-mail application. > [!NOTE] > All `aoc packages` commands operate in a workspace context. The workspace is inferred > automatically when a default workspace is configured in the application. If no default is > set, provide one explicitly with `--workspace=NAME`. List available workspaces with > `ascli aoc user workspaces list`. #### Send a Package General syntax: ```shell ascli aoc packages send <PACKAGE_EXTENDED_VALUE> <OPTIONS> <FILE_LIST> ``` Package creation parameters are sent as a **Command Parameter**. See the AoC package creation API, or display an existing package in JSON to list attributes. List allowed shared inbox destinations with: ```shell ascli aoc packages shared_inboxes list ``` Use fields: `recipients` and/or `bcc_recipients` to provide the list of recipients: **user** or **shared inbox**: - Provide either IDs as expected by API: `"recipients":[{"type":"dropbox","id":"<SHBOX_ID>"}]` - or names only: `"recipients":["The Dest"]`. ascli will resolve the list of email addresses and dropbox names to the expected type/ID list, based on case-insensitive partial match. If a user recipient (email) is not already registered and the workspace allows external users, then the package is sent to an external user, and: - If the option `new_user_option` is `--new-user-option.package_contact=true` (**default**), then a public link is sent and the external user does not need to create an account. - If the option `new_user_option` is `@json:{}`, then external users are invited to join the workspace. ##### Example: Send a package with one file to two users, using their email ```shell ascli aoc packages send @json:'{"name":"<TITLE>","note":"<NOTE>","recipients":["someuser@example.com","other@example.com"]}' <FILE_PATH> ``` ##### Example: Send a package to a shared inbox with metadata ```shell ascli aoc packages send --workspace="<WORKSPACE_NAME>" @json:'{"name":"<TITLE>","recipients":["Shared Inbox With Meta"],"metadata":{"Project Id":"123","Type":"Opt2","CheckThose":["Check1","Check2"],"Optional Date":"2021-01-13T15:02:00.000Z"}}' ~/Documents/Samples/200KB.1 ``` It is also possible to use identifiers and API parameters. For example, sending to a shared inbox by ID with metadata: ```shell ascli aoc packages send --workspace="<WORKSPACE_NAME>" @json:'{"name":"<TITLE>","recipients":[{"type":"dropbox","id":"12345"}],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1 ``` ##### Example: Send a package with files from the Files app Find files in Files app: ```shell ascli aoc files browse /src_folder ``` ```text ╭──────────────┬──────┬────────────────┬──────────────┬──────────────────────┬──────────────╮ │ name │ type │ recursive_size │ size │ modified_time │ access_level │ ╞══════════════╪══════╪════════════════╪══════════════╪══════════════════════╪══════════════╡ │ sample_video │ link │ │ │ 2020-11-29T22:49:09Z │ edit │ │ 100G │ file │ │ 107374182400 │ 2021-04-21T18:19:25Z │ edit │ │ 10M.dat │ file │ │ 10485760 │ 2021-05-18T08:22:39Z │ edit │ │ Test.pdf │ file │ │ 1265103 │ 2022-06-16T12:49:55Z │ edit │ ╰──────────────┴──────┴────────────────┴──────────────┴──────────────────────┴──────────────╯ ``` To send a package with the file `10M.dat` from subfolder /src_folder: ```shell ascli aoc files node_info /src_folder --format=json --out.level=data --out.secrets=yes | ascli aoc packages send @json:'{"name":"test","recipients":["someuser@example.com"]}' 10M.dat --transfer=@json:@stdin: --transfer.agent=node ``` #### Receive packages The command to receive one or multiple packages is: ```shell ascli aoc packages recv <PACKAGE_ID> [<FILE> ...] ``` Where `<PACKAGE_ID>` is the identifier of the package to receive or `ALL` to receive all packages matching the query. Option `once_only` is supported, see below. To download only some files from the package, add the path of the files on the command line: `[<FILE> ...]`, see option `sources`. By default, all files in the package are downloaded, that is, `.` is used as the file list. ##### Option `package_folder` The `package_folder` option (`Hash`) controls how downloaded packages are organized into subfolders under the path specified by `to_folder`. - If `package_folder` is not specified or is an empty `Hash`, packages are downloaded directly into the folder defined by `to_folder`. - If specified, packages are stored in dynamically generated subfolders. **Supported Fields**: | Field | Type | Description | |-------|---------------|----------------------------------------------------------------------------------| | `fld` | `Array[String]` | List of package fields (up to 2) used to build the subfolder name. Field values are joined with `.`.<br/>Example: `["name", "id"]`. | | `inf` | `Bool` | If `true`, creates a `<ID>.info.json` sidecar file inside the folder containing package metadata.<br/>Default: `false`. | | `opt` | `Bool` | If `false` (default), all fields in `fld` are always used. If `true`, the first field is used unless the folder already exists.<br/>Default: `false`. | | `seq` | `Bool` | If `true`, appends an incrementing number to the folder name when a conflict occurs.<br/>Default: `false`. | **Examples**: - `{"fld":["id"]}` - The subfolder name is the unique package ID. - Downloading the same package (same ID) multiple times places it in the same folder. - `{"fld":["name"]}` - The subfolder name is the package name. - If multiple packages share the same name, their contents are downloaded into the same folder. - `{"fld":["name","id"]}` - The subfolder name is `<NAME>.<ID>`. - This guarantees uniqueness when package names are identical. - `{"fld":["name","id"],"opt":true}` - The subfolder name is initially the package name. - If a folder with that name already exists, the name and ID (`<NAME>.<ID>`) are used instead. - `{"fld":["name"],"seq":true,"opt":true}` - The subfolder name is the package name if it does not already exist. - If it exists, a sequence number is appended (for example, `<NAME>.1`, `<NAME>.2`, and so on). - Downloading the same package multiple times results in different folders when a conflict occurs. ##### Example: Receive all packages from a given shared inbox ```shell ascli aoc packages recv ALL --workspace=_workspace_ --once-only=yes --lock-port=12345 --query=@json:'{"dropbox_name":"_shared_inbox_name_","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false}' --ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000}' ``` To list packages that would be downloaded, without downloading them, replace `recv ALL` with `list` (keep options `once_only` and `query`). ##### Receive new packages only (Cargo) It is possible to automatically download new packages, like using Aspera Cargo: ```shell ascli aoc packages recv ALL --once-only=yes --lock-port=12345 ``` - `ALL` (case-sensitive) will download all packages - `--once-only=yes` keeps memory of any downloaded package in persistency files located in the configuration folder - `--lock-port=12345` ensures that only one instance is started at the same time, to avoid running two downloads in parallel Typically, one would execute this command on a regular basis, using the method of your choice: see [Scheduler](#scheduler). ##### Example: Content of a received Package Some `node` operations are available for a package, such as `browse` and `find`. To list the content of a package, use command `packages browse <PACKAGE_ID> <FOLDER>`. Example: ```shell ascli aoc packages browse xx5CnbeWng / ``` Use command `find` to list recursively. For advanced users, it is also possible to pipe node information for the package and use node operations: ```shell ascli aoc packages node_info <PACKAGE_ID> / --format=json --out.secrets=yes --out.level=data | ascli node -N --preset=@json:@stdin: access_keys do self browse / ``` #### List packages By default, when using `aoc packages list` or `aoc packages receive ALL`, the following `query` is performed: | Query parameter | Value | |----------------------------|-----------------------------------------------| | `archived` | `false` | | `has_content` | `true` | | `received` | `true` | | `completed` | `true` | | `workspace_id` | Set based on current workspace. | | `dropbox_id` | Set according to `dropbox_name`, if provided. | | `exclude_dropbox_packages` | `true` unless `dropbox_id` is provided. | Parameters provided using option `query` override this query. To remove a parameter, set it to `null`. ##### Example: List packages in a given shared inbox When user packages are listed, the following query is used: ```json {"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true} ``` To list packages in a shared inbox, the query has to be specified with the shared inbox by name or its identifier. Additional parameters can be specified, as supported by the API (to find out available filters, consult the API definition, or use the web interface in developer mode). The current workspace is added unless specified in the query. > [!NOTE] > By default, `exclude_dropbox_packages` is set to `true` for user packages, and to `false` for shared inbox packages. > This can be overridden in the query. Using shared inbox name: ```shell ascli aoc packages list --query=@json:'{"dropbox_name":"My Shared Inbox","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}' ``` Using shared inbox identifier: first retrieve the ID of the shared inbox, and then list packages with the appropriate filter. ```shell shared_box_id=$(ascli aoc packages shared_inboxes show %name:'My Shared Inbox' --format=csv --out.level=data --fields=id) ``` ```shell ascli aoc packages list --query=@json:'{"dropbox_id":"'$shared_box_id'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}' ``` ### Files app The Files application presents a **Home** folder to users in a given workspace. Files located here are either user's files, or shared folders. > [!NOTE] > All `aoc files` commands operate in a workspace context. The workspace is inferred > automatically when a default workspace is configured in the application. If no default is > set, provide one explicitly with `--workspace=NAME`. List available workspaces with > `ascli aoc user workspaces list`. > All commands under `files` are the same as under `access_keys do self` for plugin `node`, that is, **gen4/access key** operations. #### Download Files The general download command is: ```shell ascli aoc files download <SOURCE_FOLDER_PATH> <SOURCE_FILENAME_1> ... ``` That is, the first argument is the source folder, and the following arguments are the source file names in this folder. If a single file or folder is to be downloaded, then a single argument can be provided. ```shell ascli aoc files download <SINGLE_FILE_PATH> ``` #### Shared folders In Aspera on Cloud (AoC), Shared Folders allow users to collaborate on content via Private or Public links. - **Private links**: Require the collaborator to log in to access the folder. - **Public links**: Include a passcode that allows access without an account or login. ##### Management Scopes Shared folders can be managed through two primary administrative scopes: - User Shared Folders: Users share personal folders with others in the same workspace using `aoc files permission`. - Admin Shared Folders: Administrators share folders from any node in a workspace using `aoc admin node do <NODE_ID> permission`. Technically, shared folders are managed as [permissions](https://developer.ibm.com/apis/catalog/aspera--aspera-node-api/Introduction) on a specific node. When a permission is created, AoC generates a link in the recipient's home folder to that shared folder. ##### Permission Payloads When creating a Shared Folder, `ascli` expects a `Hash` payload (typically passed via `@json:` or `@:`) similar to the one used at API level. **API-level Payload Example**: ```json { "file_id": "50", "access_levels": ["list","read","write","delete","mkdir","rename","preview"], "access_type": "user", "access_id": "john@example.com", "tags": {...} } ``` **Field Mapping & Helper Fields**: `ascli` automatically populates (Auto) several native fields and provides helper fields to simplify the process: | Field | Type | Description | |-----------------|----------|----------------------------------------------------------------------| | `file_id` | Native<br/>Auto | The ID of the folder, automatically resolved from the path provided in the command. | | `access_levels` | Native<br/>Optional | List of access levels to set for the shared folder.<br/>Defaults to full access. | | `tags` | Native<br/>Auto | Set with expected values for AoC: username who creates, and workspace in which the shared folder is created. | | `access_type` | Native<br/>Required | Type of access, such as `user`, `group`, or `workspace`.<br/>Can be set with parameter `with`. | | `access_id` | Native<br/>Required | ID of the user, group, or workspace (see `with`) | | `with` | `ascli` | Recipient of shared folder. Can be a username, a group name, or a workspace name.<br/>`ascli` will resolve the name to the proper type and ID in fields `access_type` and `access_id`.<br/>If the value is the empty string, then it declares the shared folder in the workspace (first action to do, see below). | | `link_name` | `ascli` | Name of the link file created in the user's home folder for private links. | | `as` | `ascli` | Name of the link file created in the user's home folder for admin shared folders. | To declare or create the shared folder in the workspace, a special value for `access_id` is used: `ASPERA_ACCESS_KEY_ADMIN_WS_[workspace ID]`, with an `access_type` of `user`. This is conveniently set by `ascli` using an **empty string** for field `with`. To share a folder with a different user, special tags are set, but this is conveniently done by `ascli` using the `as` field. ##### User Shared Folders Personal shared folders, created by users in a workspace, follow the syntax: ```shell ascli aoc files permission --workspace=<WORKSPACE_NAME> <PATH_TO_FOLDER> ... ``` > [!NOTE] > The workspace is identified by name, and folder by path, relative to the user's home. > To use an identifier instead, one can use the [percent selector](#percent-selector), like `%id:<WS_ID>`. ##### Admin Shared Folders Admin shared folders, created by administrators in a workspace, follow the syntax: ```shell ascli aoc admin node do <NODE_ID> permission --workspace=<WORKSPACE_NAME> <PATH_TO_FOLDER> ``` > [!TIP] > The node is identified by identifier. > To use a name instead, one can use the [percent selector](#percent-selector), like `%name:"<NODE_NAME>"`. > The folder is identified by its path; a file ID can be specified instead, with `%id:123`. > If the ID is left blank: `%id:`, then it means `*`, that is, "all". ##### Example: List permissions on a user shared folder ```shell ascli aoc files permission /shared_folder_test1 list ``` ##### Example: Share a personal folder with other users ```shell ascli aoc files permission /shared_folder_test1 create @json:'{"with":"laurent"}' ``` ##### Example: Revoke shared access ```shell ascli aoc files permission /shared_folder_test1 delete 6161 ``` ##### Example: Public and Private short links They can be managed with commands: ```shell ascli aoc files short_link <PATH_TO_FOLDER> private create ascli aoc files short_link <PATH_TO_FOLDER> private list ascli aoc files short_link <PATH_TO_FOLDER> public create @json:'{...}' ascli aoc files short_link <PATH_TO_FOLDER> public list ascli aoc files short_link <PATH_TO_FOLDER> public delete <ID> ascli aoc files short_link <PATH_TO_FOLDER> public modify <ID> @json:'{...}' ``` Only `public` short links can be modified. An optional payload can be provided at creation, for example to protect with a password, set an expiry date or set the access level. A password can be provided on `create` and `modify` for `public` links: ```json {"password":"<PASSWORD>"} ``` To remove a password: ```json {"password_enabled":false} ``` By default, access level is set to `edit`. Change the default access level by providing the parameter: `access_levels` in payload. `access_levels` can be: - A single `String`: one of the shortcuts: `edit`, `preview`, `download`, `upload`. - An Array: any custom combination of: `delete`, `list`, `mkdir`, `preview`, `read`, `rename`, `write`. For example: ```json {"access_levels":"upload"} ``` ```json {"access_levels":["list","rename"]} ``` An expiration date can be set with parameter `expires_at`, using [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) format. For example, `2025-08-29T08:10:31.000Z`. If only a date is provided, for example: `2025-08-29`, it will be set to midnight UTC of that date, that is, `2025-08-29T00:00:00.000Z`. ##### Example: Create a workspace admin shared folder First, identify the node ID where the shared folder will be created. To get the node ID of the default node for workspace `<WORKSPACE_NAME>`, use the command: ```shell ascli aoc admin workspace show %name:'<WORKSPACE_NAME>' --fields=node_id ``` > [!TIP] > This syntax first lists workspaces using API selector `q`, > and then performs an exact match on the specified field: `name`. Alternatively: ```shell ascli aoc admin workspace list --select.name="<WORKSPACE_NAME>" --fields=node_id ``` > [!TIP] > This syntax lists all workspaces, > and then performs an exact match on the specified field: `name`. Alternatively: ```shell ascli aoc admin workspace list --query.q="<WORKSPACE_NAME>" --fields=node_id ``` > [!TIP] > This syntax first lists workspaces using API selector `q`. > It may return several items as the query is case insensitive and made on several fields as substring. Or select a node identifier manually from the list of nodes: ```shell ascli aoc admin node list --fields=id,name ``` In the following commands, replace: - `<NODE_ID>` with the node ID, or with `%name:<NODE_NAME>`. - `<WORKSPACE_NAME>` with the workspace name, or with `%id:<WORKSPACE_ID>`. - `<FOLDER_PATH>` with the path of the folder to share on the node (for example, `/my_folder` or `my_folder`). It can also be a folder deeper than level 1. If the shared folder does not exist, then create it: ```shell ascli aoc admin node do <NODE_ID> mkdir <FOLDER_PATH> ``` Create the shared folder in workspace `<WORKSPACE_NAME>` (set `with` to empty string, or do not specify it). **Optionally**, use `as` to set the name of the shared folder if different from the folder name on the node. For other options, refer to the previous section on shared folders. ```shell ascli aoc admin node do <NODE_ID> permission <FOLDER_PATH> create @json:'{"with":"","as":"folder_for_users"}' --workspace="<WORKSPACE_NAME>" ``` > [!NOTE] > The previous command only declares the shared folder in the workspace, but does not share it with anybody. To share with a user, group, or workspace, use the `with` parameter with the name of a resource to share with (non-empty value). The `"with"` parameter will perform a lookup, and set fields `access_type` and `access_id` accordingly. The native fields `access_type` and `access_id` can also be used, instead of `with`. ```shell ascli aoc admin node do <NODE_ID> permission <FOLDER_PATH> create @json:'{"with":"john@example.com","as":"folder_for_one_user"}' --workspace="<WORKSPACE_NAME>" ``` ```shell ascli aoc admin node do <NODE_ID> permission <FOLDER_PATH> create @json:'{"with":"group 1","as":"folder_for_a_group"}' --workspace="<WORKSPACE_NAME>" ``` ```shell ascli aoc admin node do <NODE_ID> permission <FOLDER_PATH> create @json:'{"with":"<WORKSPACE_NAME>","as":"folder_for_all_workspace"}' --workspace="<WORKSPACE_NAME>" ``` > [!NOTE] > In the previous commands, field `as` is optional. ##### Example: List all workspace admin shared folders in a workspace ```shell ascli aoc admin workspace shared_folder %name:'<WORKSPACE_NAME>' list ``` ```text ╭───────┬───────────┬─────────┬─────────┬───────────┬──────────────────────────────────────╮ │ id │ node_name │ node_id │ file_id │ file.path │ tags.aspera.files.workspace.share_as │ ╞═══════╪═══════════╪═════════╪═════════╪═══════════╪══════════════════════════════════════╡ │ 198 │ eudemo │ 8666 │ 2465 │ /project1 │ │ │ 785 │ eudemo │ 8666 │ 9 │ /folder2 │ project2 │ │ 4788 │ eudemo │ 8666 │ 3691 │ /backup │ │ ╰───────┴───────────┴─────────┴─────────┴───────────┴──────────────────────────────────────╯ ``` To list members: ```shell ascli aoc admin workspace shared_folder %name:'<WORKSPACE_NAME>' member 198 list ``` ```text ╭─────────────┬──────────────────────────────────┬──────────────┬──────────────────────╮ │ access_type │ access_id │ access_level │ last_updated_at │ ╞═════════════╪══════════════════════════════════╪══════════════╪══════════════════════╡ │ user │ ASPERA_ACCESS_KEY_ADMIN_WS_45071 │ edit │ 2020-11-29T22:48:49Z │ │ group │ 160270 │ edit │ 2024-05-13T15:58:02Z │ ╰─────────────┴──────────────────────────────────┴──────────────┴──────────────────────╯ ``` If you have the node ID of the shared folder, then it is equivalent to: ```shell ascli aoc admin node do 8666 permission /project1 list --query=@json:'{"tag":"aspera.files.workspace.id=<WORKSPACE_ID>"}' ``` ##### Example: List all workspace admin shared folders on a node First get the workspace identifier: ```shell ascli aoc admin workspace list --select=@json:'{"name":"<WORKSPACE_NAME>"}' --fields=id ``` ```text <WORKSPACE_ID> ``` Then, identify the node ID on which to list, see previous section. Finally, list all shared folders, as permissions: ```shell ascli aoc admin node do <NODE_ID> permission %id: list --query=@json:'{"access_type":"user","access_id":"ASPERA_ACCESS_KEY_ADMIN_WS_<WORKSPACE_ID>"}' ``` > [!NOTE] > See Node API: `GET /permissions` for all `query` options. > The folder identifier is left empty `%id:`, to apply to all folders. #### Cross Organization transfers It is possible to transfer files directly between organizations without having to first download locally and then upload... Although optional, the creation of [Option Preset](#option-preset) is recommended to avoid placing all parameters in the command line. Procedure to send a file from org1 to org2: - Get access to Organization 1 and create an [Option Preset](#option-preset), for example `org1` (for instance, using the [Wizard](#wizard)) - Check that access works and locate the source folder `<SOURCE_FOLDER>` and the file `<SOURCE_FILE>` in it, for example, using command `files browse` - Get access to Organization 2 and create an [Option Preset](#option-preset), for example `org2` - Check that access works and locate the destination folder `<DEST_FOLDER>` - Execute the following: ```shell ascli -Porg1 aoc files node_info <SOURCE_FOLDER> --format=json --out.level=data --out.secrets=yes | ascli -Porg2 aoc files upload <SOURCE_FILE> --to-folder=<DEST_FOLDER> --transfer=@json:@stdin: --transfer.agent=node ``` Explanation: - `ascli` is the command executed by the shell - `-Porg1` loads options for preset `org1` (URL and credentials) - `aoc` uses the Aspera on Cloud plugin - `files node_info <SOURCE_FOLDER>` generates transfer information for the source folder: Node API URL, credentials and root file ID, suitable for the next command - `--format=json` formats the output as JSON (instead of the default text table) - `--out.level=data` displays only the result, removing other information such as workspace name - `|` pipes the standard output of the first command into the second one - `-Porg2 aoc` uses the Aspera on Cloud plugin and loads credentials for `org2` - `files upload <SOURCE_FILE> --to-folder=<DEST_FOLDER>` uploads the file `<SOURCE_FILE>` (located in `<SOURCE_FOLDER>` of `org1`) to `<DEST_FOLDER>` in `org2` - `--transfer=@json:@stdin:` reads the Node API information from standard input (JSON) and uses it as parameters of the transfer agent - `--transfer.agent=node` selects the `node` transfer agent: the source node of `org1` pushes the file to `org2` #### Find Files The command `aoc files find` allows searching for files in a given workspace. It works also on `node` resource: ```shell ascli aoc admin node do %name:'<NODE_NAME>' find ... ``` For instructions, refer to section `find` for plugin `node`. ### Tested commands for `aoc` > [!NOTE] > Add `ascli aoc` in front of the following commands: ```shell admin analytics application_events admin analytics files organization '' <ID> admin analytics transfers organization --query=@json:'{"status":"completed","direction":"receive","limit":2}' --notify-to=my_email_external --notify-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: <%=ev["files_completed"]%> files received\n\n<%=ev.to_yaml%>}' admin analytics transfers users --once-only=yes admin application instance list admin application instance list --query.aspera_app_type=files admin application instance list --query.organization_apps=true admin application membership list admin application types admin ats access_key create --cloud=aws --region=my_region @json:'{"id":"ak_aws_aoc","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_bucket","credentials":{"access_key_id":"<ACCESS_KEY>","secret_access_key":"<SECRET_KEY>"},"path":"/"}}' admin ats access_key create --cloud=softlayer --region=my_region @: id=ak1ibmcloud secret=<SECRET> 'name="my test key"' storage.type=ibm-s3 storage.bucket=my_bucket storage.credentials.access_key_id=<ACCESS_KEY> storage.credentials.secret_access_key=<SECRET_KEY> storage.path=/ admin ats access_key delete ak1ibmcloud admin ats access_key list --fields=name,id admin ats access_key node ak1ibmcloud --secret=<SECRET> browse / admin ats cluster clouds admin ats cluster list admin ats cluster show --cloud=aws --region=eu-west-1 admin ats cluster show 1f412ae7-869a-445c-9c05-02ad16813be2 admin auth_providers list admin bearer_token --out.level=data --out.secrets=yes admin client list admin client_access_key list admin client_registration_token create @json:'{"data":{"name":"test_client_reg1","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' admin client_registration_token delete <ID> admin client_registration_token list admin configuration_policy list admin configuration_policy show <ID> admin contact list admin dropbox list admin dropbox_membership list admin group list admin group_membership list --fields=ALL --query=@json:'{"page":1,"per_page":50,"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}' admin kms_profile list admin network_policy list admin network_policy show <ID> admin node bearer_token %name:my_node_name user:all admin node do %name:my_node_name --secret=<AK_SECRET> browse / admin node do %name:my_node_name --secret=<AK_SECRET> browse /folder_sub --node-api.cache=false admin node do %name:my_node_name --secret=<AK_SECRET> delete /folder1 admin node do %name:my_node_name --secret=<AK_SECRET> delete /folder_sub admin node do %name:my_node_name --secret=<AK_SECRET> mkdir /folder1 admin node do %name:my_node_name --secret=<AK_SECRET> mkdir /folder_sub admin node do %name:my_node_name --secret=<AK_SECRET> v3 access_key create @json:'{"id":"test-sub-1","storage":{"path":"/folder_sub"}}' admin node do %name:my_node_name --secret=<AK_SECRET> v3 access_key delete test-sub-1 admin node do %name:my_node_name --secret=<AK_SECRET> v3 events admin node do %name:my_node_name delete test_shared_folder admin node do %name:my_node_name mkdir test_shared_folder admin node do %name:my_node_name perm test_shared_folder create @json:'{"with":"","as":"other_name_shared"}' --workspace=my_workspace_shared_inbox admin node do %name:my_node_name perm test_shared_folder create @json:'{"with":"my_user_email","as":"other_name_shared"}' --workspace=my_workspace_shared_inbox admin node do %name:my_node_name perm test_shared_folder create @json:'{"with":"my_user_group","as":"other_name_shared"}' --workspace=my_workspace_shared_inbox admin node list admin operation list admin organization show admin package list --http-options.read_timeout=120.0 admin saml_configuration list admin self show admin short_link list admin subscription account admin subscription usage admin subscription usage MONTH admin user list admin user list --out.query='[?ats_admin].{name: name, email: email}' admin user modify %name:my_user_email @: deactivated=false admin user notifications %name:my_user_email show admin user preferences %name:my_user_email show admin workspace dropbox %name:my_other_workspace list admin workspace list admin workspace shared_folder %name:my_other_workspace list admin workspace shared_folder %name:my_other_workspace member <ID> list admin workspace_membership list admin workspace_membership list --fields=ALL --query=@json:'{"page":1,"per_page":50,"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}' automation workflow action <ID> create @: name=toto automation workflow create @: name=test_workflow automation workflow delete <ID> automation workflow details <ID> automation workflow export <ID> automation workflow list automation workflow list --query.show_org_workflows=@val:true automation workflow list --select.name=test_workflow --fields=id bearer_token --out.level=data --out.secrets=yes files bearer / files bearer_token_node / --cache-tokens=no files browse / files browse / --url=<PRIVATE_LINK_SHARED_FOLDER> files browse / --url=<PUBLIC_LINK_FOLDER_NO_PASS> files browse / --url=<PUBLIC_LINK_FOLDER_WITH_PASS> --password=<PUBLIC_LINK_FOLDER_PASSWORD> files browse my_remote_file files browse my_remote_folder files browse my_remote_folder/ files cat test-dst/test_file.bin files delete /test-src files download --to-folder=. test-dst/test_file.bin test-dst/test_file.bin files download --transfer.agent=connect test-dst/test_file.bin files download --transfer.agent=desktop test-dst/test_file.bin files find /sample_video files find /sample_video '\.partial$' files find /sample_video @ruby:'->(f){f["type"].eql?("file")}' files mkdir /test-src files modify /some_folder @: mount_point=false files permission my_test_folder list files rename /some_folder test-dst files short_link /test-dst private create files short_link /test-dst private list files short_link /test-dst public create @: access_levels.0=mkdir access_levels.1=write --fields=id files short_link /test-dst public modify <AOC_SHORT_LINK_PUB_CREATE> @: access_levels=edit files show %id:<ID> files show / files show test-dst/test_file.bin files sync admin status /data/local_sync files sync pull /test-dst --to-folder=/data/local_sync @json:'{"reset":true,"transport":{"target_rate":my_bps}}' files thumbnail my_test_folder/video_file.mpg files thumbnail my_test_folder/video_file.mpg --query.text=true --query.double=true files transfer push /test-src --to-folder=/test-dst test_file.bin files upload --to-folder=/ test_file.bin --url=<PUBLIC_LINK_FOLDER_NO_PASS> files upload --to-folder=/test-src test_file.bin files upload --to-folder=/test-src test_file.bin test_file.bin files v3 info gateway @: url=https://localhost:12347/aspera/faspex organization organization --format=image --fields=background_image_url --ui=text organization --url=<PUBLIC_LINK_RECV_FROM_AOC_USER> packages browse <ID> / packages list packages list --query=@json:'{"dropbox_name":"my_shared_inbox_name","sort":"-received_at","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false}' packages receive <ID> --to-folder=. packages receive <ID> --to-folder=. / packages receive ALL --once-only=yes --to-folder=. --query.max=5 --lock-port=50101 --package-folder.fld.0=name --package-folder.fld.1=id --package-folder.opt=true packages receive ALL --once-only=yes --to-folder=. --query.max=5 --lock-port=50101 --query=@json:'{"dropbox_name":"my_shared_inbox_name","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false}' --ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000}' packages receive INIT --once-only=yes --query.dropbox_name=my_shared_inbox_name packages send --workspace=my_workspace_shared_inbox --validate-metadata=yes @json:'{"name":"package title","recipients":["my_shared_inbox_meta"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' test_file.bin packages send --workspace=my_workspace_shared_inbox --validate-metadata=yes @json:'{"name":"package title","recipients":["my_shared_inbox_meta"],"metadata":{"Project Id":"456","Type":"Opt2","CheckThose":["Check1","Check2"],"Optional Date":"2021-01-13T15:02:00.000Z"}}' test_file.bin packages send --workspace=my_workspace_shared_inbox @json:'{"name":"package title","recipients":["my_shared_inbox_name"]}' test_file.bin packages send @: 'name=package title' END test_file.bin --url=<PUBLIC_LINK_SEND_AOC_USER> --password=<PUBLIC_LINK_SEND_USE_PASS> packages send @: 'name=package title' END test_file.bin --url=<PUBLIC_LINK_SEND_SHARED_INBOX> packages send @: 'name=package title' recipients.0=my_username 'note=some notes' END test_file.bin packages send @json:'{"name":"package title","recipients":["my_email_external"]}' --new-user-option.package_contact=true test_file.bin packages shared_inboxes list packages shared_inboxes short_link public %name:my_shared_inbox_name create --fields=id packages shared_inboxes short_link public %name:my_shared_inbox_name delete <AOC_SHARED_INBOX_SHORT_LINK_CREATE> packages shared_inboxes short_link public %name:my_shared_inbox_name list packages shared_inboxes short_link public %name:my_shared_inbox_name modify <AOC_SHARED_INBOX_SHORT_LINK_CREATE> @: password=<PUBLIC_LINK_FOLDER_PASSWORD> packages shared_inboxes short_link public %name:my_shared_inbox_name show <AOC_SHARED_INBOX_SHORT_LINK_CREATE> packages shared_inboxes show %name:my_shared_inbox_name packages show '%name:package title' remind --username=my_user_email --url=https://aoc.example.com/path servers --url=https://aoc.example.com/path tier_restrictions user contacts list user pref modify @: default_language=en-us user pref show user profile modify @: 'name=dummy change' user profile show user workspaces current user workspaces list ``` ## Plugin: `ats`: IBM Aspera Transfer Service ATS is usable either: - From an AoC subscription: `ascli aoc admin ats`: use AoC authentication - Or from an IBM Cloud subscription: `ascli ats`: use IBM Cloud API key authentication ### IBM Cloud ATS: Creation of API key > [!NOTE] > If you are using ATS as part of AoC, then authentication is through AoC, not IBM Cloud. > See the AoC section instead. This section is about using ATS with an IBM Cloud subscription. First get your IBM Cloud API key. For instance, it can be created using the IBM Cloud web interface, or using command line: ```shell ibmcloud iam api-key-create <KEY_NAME> -d '<DESCRIPTION>' ``` ```text OK API key <KEY_NAME> was created Please preserve the API key! It cannot be retrieved after it's created. Name <KEY_NAME> Description <DESCRIPTION> Created At 2019-09-30T12:17+0000 API Key <API_KEY> Locked false UUID ApiKey-05b8fadf-e7fe-abcd-93a9-6fd348c5ab1f ``` References: <!-- spellchecker: disable --> - [IBM Cloud Managing user API keys](https://cloud.ibm.com/docs/account?topic=account-userapikey&interface=cli) - [IBM Aspera on Cloud regions](https://ibmaspera.com/help/cloud_platforms_and_regions_supported) <!-- spellchecker: enable --> Then, to register the key by default for the `ats` plugin, create a preset. Execute: ```shell ascli config preset update <PRESET_NAME> --ibm-api-key=<API_KEY> ``` ```shell ascli config preset set default ats <PRESET_NAME> ``` ```shell ascli ats api_key instances ``` ```text aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee ``` ```shell ascli config preset update <PRESET_NAME> --instance=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee ``` ```shell ascli ats api_key create --out.secrets=yes ``` ```text ╭────────┬──────────────╮ │ field │ value │ ╞════════╪══════════════╡ │ id │ <ATS_KEY> │ │ secret │ <ATS_SECRET> │ ╰────────┴──────────────╯ ``` ```shell ascli config preset update <PRESET_NAME> --ats-key=<ATS_KEY> --ats-secret=<ATS_SECRET> ``` ### ATS Access key creation parameters When creating an ATS access key, the creation parameters are provided as a positional argument of `access_key create`, as a `Hash` [Extended Value](#extended-value-syntax). If key `transfer_server_id` is not provided, the transfer server is selected with options `cloud` and `region`. Those are directly the parameters expected by the [ATS API](https://developer.ibm.com/apis/catalog?search=%22Aspera%20ATS%20API%22). ### Misc. Examples Example: create access key on IBM Cloud (Softlayer): ```shell ascli ats access_key create @json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"<SECRET>","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}' --cloud=softlayer --region=ams ``` Example: create access key on AWS: ```shell ascli ats access_key create @json:'{"id":"<ACCESS_KEY>","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"my-bucket","credentials":{"access_key_id":"<ACCESS_KEY_ID>","secret_access_key":"<SECRET>"},"path":"/laurent"}}' --cloud=aws --region=eu-west-1 ``` Example: create access key on Azure SAS: ```shell ascli ats access_key create @json:'{"id":"<ACCESS_KEY>","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}' --cloud=azure --region=eastus ``` > [!NOTE] > The blob name is mandatory after the server address and before the parameters, and parameter `sr=c` is mandatory. Example: create access key on Azure: ```shell ascli ats access_key create @json:'{"id":"<ACCESS_KEY>","name":"laurent key azure","storage":{"type":"azure","credentials":{"account":"myaccount","key":"<ACCESS_KEY>","storage_endpoint":"myblob"},"path":"/"}}' --cloud=azure --region=eastus ``` Delete all access keys: ```shell ascli ats access_key list --fields=id --format=csv | ascli ats access_key delete @lines:@stdin: --bulk=yes ``` The parameters provided to ATS for access key creation are the ones of [ATS API](https://developer.ibm.com/apis/catalog?search=%22aspera%20ats%22) for the `POST /access_keys` endpoint. ### Tested commands for `ats` > [!NOTE] > Add `ascli ats` in front of the following commands: ```shell access_key cluster ak2ibmcloud --secret=<SECRET> access_key create --cloud=aws --region=my_region @json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_bucket","credentials":{"access_key_id":"<ACCESS_KEY>","secret_access_key":"<SECRET_KEY>"},"path":"/"}}' access_key create --cloud=softlayer --region=my_region @json:'{"id":"ak2ibmcloud","secret":"<SECRET>","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_bucket","credentials":{"access_key_id":"<ACCESS_KEY>","secret_access_key":"<SECRET_KEY>"},"path":"/"}}' access_key delete ak2ibmcloud access_key entitlement ak2ibmcloud access_key list --fields=name,id access_key node ak2ibmcloud browse / --secret=<SECRET> access_key show ak2ibmcloud api_key create api_key instances api_key list aws_trust_policy --region=us-east-1 cluster clouds cluster list cluster show --cloud=aws --region=eu-west-1 cluster show 1f412ae7-869a-445c-9c05-02ad16813be2 ``` ## Plugin: `server`: IBM Aspera High Speed Transfer Server (SSH) The `server` plugin is used for operations on Aspera HSTS using SSH authentication. It is the original way of accessing an Aspera Server, often used for **server to server** transfers. An SSH session is established, authenticated with either a password or an SSH private key. Then, commands `ascp` (for transfers) and `ascmd` (for file operations) are executed. > [!IMPORTANT] > If you have Node API credentials, or Access Key/Secret, use the plugin: `node`. > This plugin is exclusively using SSH credentials. The URL to be provided with option `url` shall be like `ssh://<SERVER_ADDRESS>:33001`, then option `username` is used to specify the transfer user, and finally either option `password` or `ssh_keys` (with one or several paths) for the authentication. Typically: ```shell ascli server --url=ssh://hsts.example.com:33001 --username=john --password=<PASSWORD> ... ``` ```shell ascli server --url=ssh://hsts.example.com:33001 --username=john --ssh-keys=~/.ssh/id_rsa ... ``` ### Tested commands for `server` > [!NOTE] > Add `ascli server` in front of the following commands: ```shell browse / browse / --password @none: --ssh-options.number_of_password_prompts=0 --ssh-keys=<SERV_KEY_PATH> browse my_inside_folder/test_file.bin browse my_upload_folder/target_hot cp my_inside_folder/test_file.bin my_upload_folder/200KB.2 delete my_inside_folder delete my_upload_folder/to.delete df download my_inside_folder/test_file.bin --to-folder=. --transfer=@json:'{"wss":false,"resume":{"iter_max":1}}' download my_large_file --to-folder=my_upload_folder --transfer.agent=node --ts.resume_policy=none du / health transfer --to-folder=my_upload_folder health transfer --to-folder=my_upload_folder --format=nagios info md5sum my_inside_folder/test_file.bin mkdir my_inside_folder --logger=stdout mkdir my_upload_folder/target_hot mv my_upload_folder/200KB.2 my_upload_folder/to.delete sync admin file_info --sql='WHERE state=19' /data/local_sync sync admin file_info /data/local_sync sync admin overview /data/local_sync sync admin query --sql='SELECT name FROM sqlite_master WHERE type='table'' /data/local_sync sync admin status /data/local_sync sync pull my_inside_folder --to-folder=/data/local_sync @: name=serv_sync_pull_conf sync pull my_inside_folder --to-folder=/data/local_sync @json:'{"name":"serv_sync_pull_conf","reset":true,"transport":{"target_rate":my_bps}}' upload 'faux:///test.bin?1k' --to-folder=my_upload_folder upload --sources=@ts --transfer.ascp_args=@list:,--file-list,file_list.txt --to-folder=my_inside_folder upload --sources=@ts --transfer.ascp_args=@list:,--file-pair-list,file_pair_list.txt upload --sources=@ts --ts=@json:'{"paths":[{"source":"test_file.bin","destination":"my_inside_folder/other_name_4"}]}' --transfer.agent=transferd upload --src-type=pair --sources=@json:'["test_file.bin","my_inside_folder/other_name_3"]' --transfer.quiet=false upload --src-type=pair test_file.bin my_inside_folder/other_name_2 --notify-to=my_email_external '--transfer.ascp_args=@list: -l 100m' upload --src-type=pair test_file.bin my_upload_folder/other_name_5 --ts=@json:'{"cipher":"aes-192-gcm","content_protection":"encrypt","content_protection_password":"<PASSWORD>","cookie":"biscuit","create_dir":true,"delete_before_transfer":false,"delete_source":false,"exclude_newer_than":"-1","exclude_older_than":"-10000","fasp_port":33001,"http_fallback":false,"multi_session":0,"overwrite":"diff+older","precalculate_job_size":true,"preserve_access_time":true,"preserve_creation_time":true,"rate_policy":"fair","resume_policy":"sparse_csum"}' upload --to-folder=my_upload_folder/target_hot --lock-port=50101 --transfer.ascp_args=@list:,--remove-after-transfer,--remove-empty-directories,--exclude-newer-than=-8,--src-base,hot_folder hot_folder upload /test_file.bin --to-folder=my_upload_folder --transfer.agent=node --transfer.url=http://localhost:12348 --transfer.username=sim --transfer.password=<PASSWORD> upload test_file.bin --to-folder=my_inside_folder --ts=@json:'{"multi_session":3,"multi_session_threshold":1,"resume_policy":"none","target_rate_kbps":100000}' --transfer=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}' --progress-bar=yes ``` ### Authentication on Server with SSH session If SSH is the session protocol (by default, that is, not WSS), then the following session authentication methods are supported: - `password`: SSH password - `ssh_keys`: SSH keys (Multiple SSH key paths can be provided.) If `username` is not provided then the default transfer user `xfer` is used. If neither SSH password nor key is provided and a transfer token is provided in transfer spec (option `ts`), then the standard SSH bypass keys are used. Example: ```shell ascli server --url=ssh://<SERVER_ADDRESS>:33001 ... --ts=@json:'{"token":"Basic <TOKEN>"}' ``` > [!NOTE] > If you need to use the Aspera public keys, then specify an empty token: > `--ts=@json:'{"token":""}'` : Aspera public SSH keys will be used, but the protocol will ignore the empty token. The value of the `ssh_keys` option can be a single value or an `Array`. Each value is a **path** to a private key and is expanded (`~` is replaced with the user's home folder). Examples: ```shell ascli server --ssh-keys=~/.ssh/id_rsa ascli server --ssh-keys=@list:,~/.ssh/id_rsa ascli server --ssh-keys=@json:'["~/.ssh/id_rsa"]' ``` For file operation commands (browse, delete), the Ruby SSH client library `Net::SSH` is used and provides several options settable using option `ssh_options` (additive option like `ts`). For a list of SSH client options, refer to the Ruby documentation of [Net::SSH](http://net-ssh.github.io/net-ssh/Net/SSH.html#method-c-start). Among the 50 available SSH options: - `verbose` - `use_agent` - `passphrase` By default, the SSH library will check if a local `ssh-agent` is running. On Linux, if you get an error message such as: ```text ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: Agent not configured ``` Or on Windows: ```text ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant process not running ``` This means that your environment suggests using an agent, but no SSH agent is running. In that case: - Check env var: `SSH_AUTH_SOCK` - Check your file: `$HOME/.ssh/config` - Check if the SSH key is protected with a passphrase (then, use the `passphrase` SSH option) - [Check the Ruby SSH options in start method](https://github.com/net-ssh/net-ssh/blob/master/lib/net/ssh.rb) - To disable the use of `ssh-agent`, use the option `ssh_options` like this: ```shell ascli server --ssh-options=@json:'{"use_agent": false}' ... ``` > [!TIP] > This can also be set using a preset. If one of the SSH private keys is passphrase-protected, then option `passphrase` can be used. It is equivalent to setting both options `ssh_options.passphrase` and `ts.ssh_private_key_passphrase`. ### Analyze `ascp` logs of the server Command `logs` analyzes the `ascp` log files of the server (`ascp` 4.0 and later), instead of those of the local machine: the files are read over the SSH connection of the plugin (same `url`, `username`, `password` or `ssh_keys`, and `ssh_options`), then analyzed like with [`config ascp logs`](#agent-direct-transfer). It has the same sub-commands (`summary`, `sessions`, `diagnostics`, `health`, `records`, ...) and the same option `ascp_logs`. Paths of the files on the server are mandatory: there is no default location, as it depends on the installation (e.g. a syslog file such as `/var/log/aspera.log` or `/var/log/messages`, or a file of the folder given with `ascp -L`). Files compressed with gzip (rotations) are supported. A folder is not read, give its files. ```shell ascli server --url=ssh://<SERVER_ADDRESS>:33001 --username=root --ssh-keys=~/.ssh/id_rsa logs summary /var/log/aspera.log ascli server ... logs sessions /var/log/aspera.log /var/log/aspera.log-20261101.gz --select=@json:'{"status":"failed"}' ``` The user must be allowed to read the files (log files are often readable by `root` only), and the server shall provide a shell with `head` (and `stat`). The files are copied to a temporary folder, removed after the analysis. The size of the files is limited (total of 256 MB by default), to avoid reading huge logs in memory: the limit is the key `remote_max_size` (in bytes) of option `ascp_logs`, e.g. `--ascp-logs.remote_max_size=1000000000`. The modification date of each file is kept, as the year of syslog lines is deduced from it. Not available with `--url=local:` and `https`. ### Other session channels for `server` URL schemes `local` and `https` are also supported (mainly for testing purposes). (`--url=local:`, `--url=https://...`) - `local` will execute `ascmd` locally, instead of using an SSH connection. - `https` will use Web Socket Session: This requires the use of a transfer token. For example, a `Basic` token can be used. As SSH is used most of the time, if an `http` scheme is provided without a token, the plugin falls back to SSH and port 33001. ### Examples: `server` One can test the `server` application using the well-known demo server: ```shell ascli config initdemo ascli server browse /aspera-test-dir-large ascli server download /aspera-test-dir-large/200MB ``` `initdemo` creates an [Option Preset](#option-preset) `demoserver` and sets it as default for plugin `server`. If an SSH private key is used for authentication with a passphrase, the passphrase needs to be provided to both options: `ssh_options` (for browsing) and `ts` (for transfers): ```shell ascli server --url=ssh://<SERVER_ADDRESS>:33001 --username=<USERNAME> --ssh-keys=<PRIVATE_KEY_PATH> --passphrase=<PASSPHRASE> ``` ## Plugin: `node`: IBM Aspera High Speed Transfer Server Node This plugin gives access to capabilities provided by the HSTS Node API. Authentication uses either a Node API username and password, or an access key and secret, provided with options `username` and `password`. > [!NOTE] > Capabilities of this plugin are used in other plugins that access the Node API, such as `aoc`, `ats`, `shares`. > [!NOTE] > This plugin can be used with any type of **Aspera Node**, either on-premises or ATS, provided that you have Node API credentials. > Those credentials can be either Node API user or Access Key (for example, on ATS). ### File Operations It is possible to do **gen3/node user** operations: - `browse` - Transfer (`upload` / `download` / `sync`) - `delete` - ... When using an access key, the so-called **gen4/access key** API is also supported through subcommands using `access_keys do self`. Example: - `ascli node browse /` : list files with **gen3/node user** API - `ascli node access_keys do self browse /` : list files with **gen4/access key** API #### Browse ##### Gen3 This is when executing `browse` in `node` plugin or `v3`. Native API parameters can be placed in option `query`. Special parameters can be placed in option `query` for "gen3" browse: | Parameter | Description | |-------------|------------------------------------| | `recursive` | Recursively list files. | | `max` | Maximum number of files to list. | | `self` | Show folder metadata, not content. | | `skip` | Single API call. | ##### Gen4 This is when executing `browse` in `aoc files` or in `node access_keys do`. Option `node_api` (`Hash`) controls some options of API used, with the following parameters: | Parameter| Default | Description | |----------|---------|-------------| | `cache` | `true` | `true` Folder content is retrieved from the Redis database (faster).<br/>`false` Folder content retrieved from storage. | | `standard_ports` | `true` | `true` Use hard coded standard ports (`33001`)<br/>`false` Retrieve server ports from an API call (`download_setup`) which reads the information from `aspera.conf` on the server. | | `accept_v4` | `true` | `true` (default) uses API header: `Accept-Version: 4.0`.<br/>`false` uses legacy method (refer to node API). | | `per_page` | - | Requested number of items per API call.<br/>Set to `nil` (`@none:`) to deactivate paging.<br/>Default: `1000` when `accept_v4` is `true`. | ### Operation `find` on **gen4/access key** The command `find <FOLDER> [filter_expr]` is available for **gen4/access key**, under `access_keys do self`. The argument `<FOLDER>` is mandatory and is the root from which the search is performed. The argument `[filter_expr]` is optional and represents the matching criteria. It recursively scans storage to find files and folders matching the criteria and returns a list of matching entries. `[filter_expr]` is either: - Optional (default): All files and folders are selected - Type `String`: The expression is similar to shell globbing; refer to **Ruby** function: [`File.fnmatch`](https://ruby-doc.org/3.2.2/File.html#method-c-fnmatch) - Type `Proc` : The expression is a Ruby lambda that takes one argument: a `Hash` that contains the current folder entry to test. See the following examples. Examples of expressions: - Find all files and folders under `/` ```shell ascli node access_keys do self find / ``` - Find all text files in `/Documents` ```shell ascli node access_keys do self find /Documents '*.txt' ``` The following examples show Ruby lambda code you can use in the template command below: ```shell ascli node access_keys do self find / @ruby:'->(f){[code here]}' ``` > [!TIP] > Single quotes are used above to protect the whole **Ruby** expression from the shell. > Then double quotes are used for strings in the **Ruby** expression to not mix with the shell. - Find files modified in the last 100 days ```ruby ->(f){f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100} ``` - Find files older than 1 year ```ruby ->(f){f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))>365} ``` - Find files larger than 1 MB ```ruby ->(f){f["type"].eql?("file") and f["size"].to_i>1000000} ``` - Filter out files beginning with `._` or named `.DS_Store`: ```ruby ->(f){!(f["name"].start_with?("._") or f["name"].eql?(".DS_Store"))} ``` - Match files using a [Ruby Regexp](https://ruby-doc.org/core/Regexp.html): `\.gif$` ```ruby ->(f){f["name"].match?(/\.gif$/)} ``` `ascli` commands can be piped to combine operations, such as **find and delete**: ```shell ascli node access_keys do self find / @ruby:'->(f){f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))>365}' --fields=path --format=csv | ascli node --bulk=yes delete @lines:@stdin: ``` > [!NOTE] > The pipe character `|` on the last line chains the two commands together. ### Listing transfer events When you run a transfer, its information is stored (typically for 1 day) in the HSTS database (Redis). To view this information, use the command `transfer list`. When you have many transfers, the list is built from multiple API calls. To list only new events since the last run, use the option `once_only`: ```shell ascli node transfer list --once-only=yes ``` `ascli` stores an `iteration_token` in its persistence repository to track the latest event. To reset it, add the option `--query=@json:'{"reset": true}'`. To limit how many events are listed, use the `max` parameter in the query. Other query parameters are passed through to the underlying API (`GET /ops/transfers`). ### Central The `central` sub-command uses the **reliable query** API (session and file). Use it to list transfer sessions and transferred files. To list transferred files: ```shell ascli node central file list ``` To validate transfers offline, use the `validator` option. > [!NOTE] > See the HSTS documentation for more details. ### Sync The `node` plugin provides three commands for file synchronization: | Command | `node` | `shares` | `aoc` | `server` | Description | |-----|-----|-----|-----|-----|-----------------------------------------------| | `sync` | Yes | Yes | Yes | Yes | Perform a local sync by executing `async` locally. | | `async` | Yes | | | | Uses API `/async`.<br/>Get status on sync operations on the server side, as in Aspera Console. | | `ssync` | Yes | | | | Uses API `/asyncs`.<br/>Start a sync on the server side and monitor only those operations. | For details on the `sync` action, see [IBM Aspera Sync](#ibm-aspera-sync). For the `async` subcommands `show` and `delete`, you can use the special identifier `ALL`. ### FASP Stream You can start a FASP Stream session from the Node API. Run the following command: `ascli node stream create @json:<VALUE>` with the following [**transfer-spec**](#transfer-specification): ```json {"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"<PASSWORD>"} ``` ### Watchfolder For watch folder creation, see [Aspera Server documentation](https://www.ibm.com/docs/en/ahte/4.4.x?topic=wffcl-watch-folder-json-configuration-file-reference) or [Aspera Watchfolder API Documentation](https://developer.ibm.com/apis/catalog/aspera--aspera-watch-folders-api/api/API--aspera--ibm-aspera-watch-folders-api). You can run watchfolder operations remotely through the Node API: - Start the `watchd` and `watchfolderd` services as a system user that has access to the files - Configure a **Watchfolder** to define automated transfers ```shell ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"<USERNAME>"}}' ascli node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"<USERNAME>"}}' ascli node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"<USERNAME>","pass":"<PASSWORD>"}}' ``` ### Out of Transfer File Validation To activate this feature, follow the Aspera Transfer Server configuration. The following command lists one file that requires validation and assigns it to the unique validator identifier you provide: ```shell ascli node central file list --validator=ascli @json:'{"file_transfer_filter":{"max_result":1}}' ``` ```text ╭──────────────┬──────────────┬────────────┬─────────────────────────────────────╮ │ session_uuid │ file_id │ status │ path │ ╞══════════════╪══════════════╪════════════╪═════════════════════════════════════╡ │ 1a74444c-... │ 084fb181-... │ validating │ /home/xfer.../PKG - <TITLE>/200KB.1 │ ╰──────────────┴──────────────┴────────────┴─────────────────────────────────────╯ ``` To update the status of the file, use the following command: ```shell ascli node central file modify --validator=ascli @json:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}' ``` ```text updated ``` ### Example: SHOD to ATS Scenario: Access to a **Shares on Demand** (SHOD) server on AWS is provided by a partner. We need to transfer files from this third party SHOD instance into our Azure BLOB storage. Create an **Aspera Transfer Service** instance, which provides access to the Node API. Then create an [Option Preset](#option-preset) with the Node API URL and credentials of the **SHOD** instance, named `aws_shod`. Create another [Option Preset](#option-preset) for the Azure ATS instance, named `azure_ats`. Then execute the following command: ```shell ascli node download /share/sourcefile --to-folder=/destination_folder --preset=aws_shod --transfer=@preset:azure_ats --transfer.agent=node ``` This will get transfer information from the SHOD instance and tell the Azure ATS instance to download files. ### Node file information When Node API is used with an **Access key**, extra information can be retrieved, such as preview. > [!NOTE] > Display of preview on terminal requires installation of extra gem: `rmagick` ```shell dnf install -y ImageMagick-devel gem install rmagick rainbow ``` For example, it is possible to display the preview of a file, if it exists, using an access key on node: ```shell ascli node access_keys do self thumbnail /preview_samples/Aspera.mpg ``` Previews are mainly used in AoC; this also works with AoC: ```shell ascli aoc files thumbnail /preview_samples/Aspera.mpg ``` > [!TIP] > To specify the file by its file ID, use the selector syntax: `%id:<FILE_ID>` > [!NOTE] > To force textual display of the preview on **iTerm**, prefix command with: `env -u TERM_PROGRAM -u LC_TERMINAL` ### Creating an access key ```shell ascli node access_keys create @json:'{"id":"<ACCESS_KEY>","secret":"<SECRET>","storage":{"type":"local","path":"/data/mydir"}}' ``` > [!TIP] > The `id` and `secret` fields are optional. > If not provided, they will be generated and returned in the result. > In that case, provide option `--out.secrets=yes` to get the generated secret. Access keys support extra overriding parameters using parameter: `configuration` and sub keys `transfer` and `server`. For example, an access key can be modified or created with the following options: ```json {"configuration":{"transfer":{"target_rate_cap_kbps":500000}}} ``` The list of supported options can be displayed using command: ```shell ascli node info --fields=@ruby:'/^access_key_configuration_capabilities.*/' ``` ### Generating and using a bearer token Bearer tokens are part of the **gen4/access key** API. It follows the model of OAuth 2. For example, they are used in Aspera on Cloud. This is also available for developers for any application integrating Aspera. In this API, files, users and groups are identified by an ID (a `String`, for example, `"125"`, not necessarily numerical). Bearer tokens are typically generated by the authenticating application and then recognized by the Node API. A bearer token is authorized on the node by creating `permissions` on a **folder**. Bearer tokens can be generated using `ascli` command `bearer_token`: it takes two arguments: - The private key used to sign the token. - The token information, which is a `Hash` containing the following elements: | Parameter | Default | Type | Description | |----------------------|---------------------------|---------|---------------------------------------| | `_scope` | `user:all` | Special | Either `user:all` or `admin:all` | | `_validity` | 86400 | Special | Validity in seconds from now. | | `user_id` | - | Mandatory | Identifier of user | | `scope` | `node.<ACCESS_KEY>:_scope` | Mandatory | API scope<br/>for example, `node.<ACCESS_KEY>:<NODE_SCOPE>` | | `expires_at` | `now+_validity` | Mandatory | Format: `%Y-%m-%dT%H:%M:%SZ`<br/>for example, `2021-12-31T23:59:59Z` | | `auth_type` | `access_key` | Optional | `access_key`, `node_user` | | `group_ids` | - | Optional | List of group IDs | | `organization_id` | - | Optional | Organization ID | | `watermarking_json_base64` | - | Optional | Watermarking information (not used) | > [!NOTE] > For convenience, `ascli` provides additional parameters `_scope` and `_validity`. > They are not part of the API and are removed from the final payload. > They provide a convenient way to set `scope` and `expires_at`. #### Bearer token: Environment An access key shall be created to grant access for transfers to its storage. The access key and its secret represent administrative access to the storage as it has access rights to the whole storage of the access key. The way to create access keys depends slightly on the type of HSTS: - If a self-managed Aspera node is used, then a **node user admin** must be created: It has no `docroot` but has at least one file restriction (for testing, one can use `*` to accept creation of an access key with any storage root path). See the Aspera HSTS documentation. - If Cloud Pak for Integration is used, then the node admin is created automatically. - If Aspera on Cloud or ATS is used, then the SaaS API for access key creation is used. > [!NOTE] > See [HSTS manual](https://www.ibm.com/docs/en/ahts): `Access key authentication` section for more details on access key creation. The following sections assume that an access key has been created and that `ascli` is configured to use this access key by default using `node`. #### Bearer token: Preparation Assume that the access key was created, and a default configuration is set to use this **access key**. Using `ascli`, an access key can be created using the `access_keys create` command on the node (using main node credentials) or on ATS. Create a private key (organization key) that will be used to sign bearer tokens: ```shell my_private_pem=./myorgkey.pem ascli config genkey $my_private_pem ``` > [!NOTE] > This private key is not used for authentication. > It is used to sign bearer tokens. > See [private key](#private-key) for more details on generation. The corresponding public key shall be placed as an attribute of the **access key** (done with `PUT /access_keys/<ID>`): ```shell ascli node access_keys set_bearer_key self @file:$my_private_pem ``` > [!NOTE] > Either the public or private key can be provided, and only the public key is used. > This enables checking the signature of bearer tokens. > Above command is executed with access key credentials. Alternatively, use the following equivalent command, as `config genkey` also saves the public key with extension `.pub`: ```shell ascli node access_keys modify %id:self @ruby:'{token_verification_key: File.read("'$my_private_pem'.pub")}' ``` #### Bearer token: Configuration for user - Select a folder for which to grant access to a user, and get its identifier: ```shell my_folder_id=$(ascli node access_keys do self show / --fields=id) ``` > [!NOTE] > This example uses `/`, but any folder can be selected in the access key storage. - Designate a user by ID: ```shell my_user_id=777 ``` > [!NOTE] > This is an arbitrary identifier, typically managed by the web application. > Not related to Linux user IDs or anything else. - Grant this user access to the selected folder: ```shell ascli node access_keys do self permission %id:$my_folder_id create @json:'{"access_type":"user","access_id":"'$my_user_id'"}' ``` - Create a Bearer token for the user: ```shell ascli node bearer_token @file:./myorgkey.pem @json:'{"user_id":"'$my_user_id'","_validity":3600}' --out.secrets=yes --out.file=bearer.txt ``` > [!NOTE] > The Bearer token can also be created using command `asnodeadmin` on HSTS. > See the [HSTS manual](https://www.ibm.com/docs/en/ahts): `Bearer tokens` section. > Code for token generation is provided in [`lib/aspera/api/node.rb`](../lib/aspera/api/node.rb) #### Bearer token: User side Assume the role of the user, with the following information: - The URL of the Node API - A Bearer token - A file ID for which we have access To use this information: ```shell ascli node -N --url=https://... --password="Bearer $(cat bearer.txt)" --root-id=$my_folder_id access_keys do self browse / ``` ### Tested commands for `node` > [!NOTE] > Add `ascli node` in front of the following commands: ```shell --url=http://localhost:12348 --username=sim --password=<PASSWORD> browse / --fields=path --format=csv --select=@json:'{"basename":"test_file.bin"}' --url=http://localhost:12348 --username=sim --password=<PASSWORD> transfer list --fields=status --url=https://node.example.com:9092 --username=user1 --password=<PASSWORD> --ui=text asperabrowser --url=https://tst.example.com/path --password 'Bearer <ND_BEARER_TOKEN>' --root-id=<ID> access_key do self browse / access_key create @json:'{"id":"my_username","secret":"<PASSWORD>","storage":{"type":"local","path":"/"}}' access_key delete my_username access_key do my_ak_name browse / --secret=<AK_SECRET> access_key do my_ak_name delete /test_nd_ak2 --secret=<AK_SECRET> access_key do my_ak_name delete test_nd_ak3 --secret=<AK_SECRET> access_key do my_ak_name download test_nd_ak3 --to-folder=. --secret=<AK_SECRET> access_key do my_ak_name find my_test_folder --secret=<AK_SECRET> access_key do my_ak_name find my_test_folder @re:'\.jpg$' --secret=<AK_SECRET> access_key do my_ak_name find my_test_folder @ruby:'->(f){f["name"].end_with?(".jpg")}' --secret=<AK_SECRET> access_key do my_ak_name mkdir /tst_nd_ak --secret=<AK_SECRET> access_key do my_ak_name mkfile /mkfile.txt 'hello world' --secret=<AK_SECRET> access_key do my_ak_name mklink /mklink.txt --query=@json:'{"target":"/mkfile.txt","target_node_id":"123"}' --secret=<AK_SECRET> access_key do my_ak_name node_info / --secret=<AK_SECRET> access_key do my_ak_name rename /tst_nd_ak test_nd_ak2 --secret=<AK_SECRET> access_key do my_ak_name show %id:1 --secret=<AK_SECRET> access_key do my_ak_name upload 'faux:///test_nd_ak3?100k' --node-api.standard_ports=false --secret=<AK_SECRET> access_key do self permission %id:<ID> create @json:'{"access_type":"user","access_id":"666"}' access_key do self permission / delete 1 access_key do self permission / modify 1 @: 'access_levels=@list: read list' access_key do self permission / show 1 access_key list access_key set_bearer_key self @file:<PRIVATE_KEY_FILE> access_key show %id:self api_details async bandwidth %name:my_sync_session_name async counters %name:my_sync_session_name async delete ALL async files %name:my_sync_session_name async files %name:my_sync_session_name --once-only=yes async list async show %name:my_sync_session_name async show ALL basic_token browse / --log-level=trace2 cat my_upload_folder/test_file.bin central file list central file modify --validator=1 @: files=@list:. central session list delete @list:,my_upload_folder/a_folder,my_upload_folder/tdlink,my_upload_folder/a_file delete my_upload_folder/test_file.bin download my_upload_folder/test_file.bin --to-folder=. emulator @json:'{"url":"http://localhost:12348","username":"sim","password":"<PASSWORD>","password_min_bits":0,"docroot":"/data"}' health info --fpac='function FindProxyForURL(url,host){return "DIRECT"}' license mkdir my_upload_folder/a_folder mkfile my_upload_folder/a_file1 'hello world' mklink my_upload_folder/a_folder my_upload_folder/tdlink rename my_upload_folder a_file1 a_file search / --query.sort=mtime service create @json:'{"id":"service1","type":"WATCHD","run_as":{"user":"user1"}}' service delete service1 service list slash space / spec ssync bandwidth %name:my_node_sync ssync counters %name:my_node_sync ssync create @json:'{"configuration":{"name":"my_node_sync","local":{"path":"my_local_path_real"},"remote":{"host":"my_host","port":my_port,"user":"my_username","pass":"<PASSWORD>","path":"my_remote_path"}}}' ssync delete %name:my_node_sync ssync files %name:my_node_sync ssync list ssync show %name:my_node_sync ssync start %name:my_node_sync ssync state %name:my_node_sync ssync stop %name:my_node_sync ssync summary %name:my_node_sync stream list sync admin status /data/local_sync sync pull /aspera-test-dir-tiny --to-folder=/data/local_sync @: reset=true sync pull /aspera-test-dir-tiny --to-folder=/data/local_sync @json:'{"name":"my_sync_session_name","reset":true}' transfer bandwidth_average transfer cancel <ID> transfer list --query.active_only=true transfer list --query.reset=true --once-only=yes transfer modify <ID> @: target_rate_kbps=10000 transfer sessions transfer show <ID> transport upload 'faux:///testfile1?1m' --to-folder=my_local_path upload --to-folder=my_upload_folder --sources=@ts --ts=@json:'{"paths":[{"source":"/aspera-test-dir-small/10MB.2"}],"precalculate_job_size":true}' --transfer.agent=node --transfer.url=https://node.example.com/path --transfer.username=my_username --transfer.password=<PASSWORD> upload --username=my_ak_name --password=<AK_SECRET> test_file.bin upload my_mxf my_docx --ts=@json:'{"target_rate_kbps":1000000,"resume_policy":"none"}' watch_folder list ``` ### Open Telemetry The `node` plugin supports Open Telemetry (OTel) for monitoring and tracing. > [!NOTE] > This is an experimental feature and currently only available for the `node` plugin and Instana backend. `ascli` polls the Node API for transfer events and sends them to an OTel collector. The command expects the following parameters provided as a `Hash` positional parameter: | Parameter | Type | Default | Description | |-------------|----------|---------|----------------------------------------------| | `url` | `String` | - | URL of the Instana HTTPS backend for OTel. | | `key` | `String` | - | Agent key for the backend. | | `interval` | `Float` | 10 | Polling interval in seconds.<br/>`0` for single shot. | To retrieve OTel backend information: Go to the Instana web interface, **More** → **Agents** → **Docker** and identify the agent endpoint and key, for example, `endpoint=ingress-blue-saas.instana.io`. Identify the region and the endpoint URL will be `https://otlp-[region]-saas.instana.io`, that is, replace `ingress` with `otlp`. For convenience, those parameters can be provided in a preset, for example, named `otel_default`. ```shell ascli config preset initialize otel_default @json:'{"url":"https://otlp-orange-saas.instana.io:4318","key":"<KEY>","interval":1.1}' ``` Then it is invoked like this (assuming a default node is configured): ```shell ascli node telemetry @preset:otel_default ``` In Instana, create a custom Dashboard to visualize the OTel data: - Add Widget: Histogram - Data Source: Infrastructure and Platforms - Metric: search `transfer` ### Node emulator > [!NOTE] > This is not a feature for production. > It's provided for testing only. The command `emulator` starts a local web server that answers a subset of the Node API, and executes transfers with the Transfer Daemon (`transferd`). It allows testing, without HSTS, the [Node API agent](#agent-node-api) (`--transfer.agent=node`) and the `node` commands `info`, `browse`, `transfer list|show|modify|cancel`. The Transfer Daemon and the gem `grpc` must be installed (see [Agent: Transfer Daemon](#agent-transfer-daemon)). The emulator starts its own `transferd`, and stops it on exit. It takes an optional `Hash` argument with the following parameters: | Field | Type | Description | |-------------------|---------|----------------------------------------------------------------------------------| | `cert` | `String` | Path to the TLS certificate file. Accepted formats: PEM (`.pem`) or PKCS12 (`.p12` / `.pfx`).<br/>Example: `/path/to/cert.pem`. | | `chain` | `String` | Path to the PEM certificate chain file (appended as extra chain certificates).<br/>Example: `/path/to/chain.pem`. | | `docroot` | `String` | Local folder of the node files. Paths of `/files/browse` and local paths of transfers (sources of `send`, destination of `receive`) are relative to it, and confined in it. Defaults to the current working directory.<br/>Example: `/data/aspera`. | | `key` | `String` | Path to the PEM private key file, or the PKCS12 passphrase when `cert` is a `.p12`/`.pfx` file.<br/>Example: `/path/to/key.pem`. | | `password` | `String` | Password expected from clients in HTTP Basic authentication, for the above `username`. Generate it with `openssl rand -hex 32`: a weak password is refused (see `password_min_bits`).<br/>Example: `<PASSWORD>`. | | `password_min_bits` | `Integer` | Minimum estimated strength of `password`, in bits (log2 of the number of guesses to find it). The strength is estimated with gem `zxcvbn` if installed, else with a built-in estimate that detects repeats, sequences, keyboard rows, words and years. `0` disables the check. A random password of 32 bytes (`openssl rand -hex 32`) is far above the default.<br/>Default: `80`. | | `retention_sec` | `Integer` | Time in seconds a transfer stays in the list of transfers after it ended (completed, failed or canceled, and no more retried).<br/>Default: `86400`. | | `url` | `String` | Address and port the emulator listens on. Use `https://` with `cert`/`key` for TLS.<br/>Default: `http://localhost:8080`. | | `username` | `String` | Username expected from clients in HTTP Basic authentication. Set together with `password`. When not set, requests are accepted without authentication. Mandatory when `url` is not a loopback address (`localhost`, `127.0.0.0/8`, `::1`).<br/>Example: `node_user`. | For details on `url` and HTTPS, see [Web service](#web-service). Like on a real node, the files of the emulator are in its `docroot`: paths of `browse`, and local paths of transfers, are relative to it. Local paths are the sources of an upload (`send`), and the destination of a download (`receive`). Paths leading out of the `docroot`, also through symbolic links, are rejected. Start the emulator, it runs until interrupted: ```shell ascli node emulator @json:'{"url":"http://localhost:12348","username":"sim","password":"<PASSWORD>","docroot":"/data"}' ``` Generate the password with `openssl rand -hex 32`. A weak password is refused at startup: its strength is estimated in bits, and shall be at least `password_min_bits`. `username` and `password` are mandatory when `url` is not a loopback address (see [Web service](#web-service)): anyone reaching the emulator could send the files of the `docroot` to any host. Then, in another terminal, use it as a node, for example, to list the files in `/data`: ```shell ascli node --url=http://localhost:12348 --username=sim --password=<PASSWORD> browse / ``` Or as the transfer agent, for example, to upload the file `/data/my_file.dat` to a transfer server: ```shell ascli server upload /my_file.dat --transfer.agent=node --transfer.url=http://localhost:12348 --transfer.username=sim --transfer.password=<PASSWORD> ``` The transfer is executed by `transferd`, and its status is available on the emulator: ```shell ascli node --url=http://localhost:12348 --username=sim --password=<PASSWORD> transfer list ``` Supported endpoints: | Verb | Path | Action | |----------|-----------------------|---------------------------------------------------------------------------------------------------| | `GET` | `/info` | Node information, version and license from `transferd`. | | `GET` | `/ops/transfers` | List transfers. Query parameters: `active_only`, `direction`, `count`. | | `POST` | `/ops/transfers` | Start a transfer with `transferd`. | | `GET` | `/ops/transfers/{id}` | Transfer information, with sessions and files. | | `PUT` | `/ops/transfers/{id}` | Modify `target_rate_kbps`, `min_rate_kbps` or `rate_policy`, or cancel with `status`: `canceled`. | | `CANCEL` | `/ops/transfers/{id}` | Cancel a transfer. | | `POST` | `/files/browse` | List a folder of the `docroot`. | Limitations: - Only transfers started through the emulator are known. They are kept in memory, and lost when the emulator stops. Ended transfers are removed from the list after `retention_sec` (default: one day). - Only Basic authentication is supported: when `username` and `password` are set, bearer tokens and access keys are rejected. When they are not set (loopback `url` only), all requests are accepted. - Not supported: `files/upload_setup` and `files/download_setup` (so, `node upload|download` on the emulator), `ops/transfers/bandwidth`, pause and resume of transfers, query parameters `iteration_token` and `tag`. ## Plugin: `faspex5`: IBM Aspera Faspex v5 Faspex 5 is IBM Aspera's newer self-managed application. The following authentication methods are supported (option `auth`): | Method | Description | |---------------|---------------------------------------------------------------------| | `jwt` | General purpose, private-key based authentication | | `web` | Requires authentication with web browser | | `public_link` | Public link authentication (set when option `url` is a public link) | | `boot` | Use authentication token copied from browser (experimental) | > [!TIP] > If you have a Faspex 5 public link, provide it, as-is, through the option `url`. ### Faspex 5 quick start with wizard For a quick start, one can use the wizard, which will help to create an [Option Preset](#option-preset): ```shell ascli config wizard ``` Then, answer questions interactively: ```text argument: url> faspex5.example.com ``` If multiple applications are detected, the wizard asks which one to use (this step is skipped if only Faspex is detected): ```text Multiple applications detected: ╭─────────┬───────────────────────────────────────────┬─────────────╮ │ product │ url │ version │ ╞═════════╪═══════════════════════════════════════════╪═════════════╡ │ faspex5 │ https://faspex5.example.com/aspera/faspex │ F5.0.6 │ │ server │ ssh://faspex5.example.com:22 │ OpenSSH_8.3 │ ╰─────────┴───────────────────────────────────────────┴─────────────╯ product> faspex5 ``` When Faspex is detected, the wizard asks for a private key (a value like `@file:<path>`). If you do not have a private key, leave that field blank, and one will be generated or a previously generated key will be used. ```text Using: Faspex at https://faspex5.example.com/aspera/faspex Private RSA key, e.g. @file:<path> (leave empty to generate): option: private_key> Using existing key: /Users/someuser/.aspera/ascli/my_key ``` Then, the email of the user shall be provided: ```text option: username> someuser@example.com ``` The administrator of Faspex shall provide you with a `client_id`, as specified below. ```text Ask the ascli client ID and secret to your Administrator. Admin should login to: https://faspex5.example.com/aspera/faspex Navigate to: :: → Admin → Configurations → API clients Create an API client with: - name: ascli - JWT: enabled Then, logged in as someuser@example.com go to your profile: (User) → Account Settings → Preferences → Public Key in PEM: -----BEGIN PUBLIC KEY----- redacted -----END PUBLIC KEY----- Once set, fill in the parameters: option: client_id> <CLIENT_ID> option: client_secret> **** Preparing preset: faspex5_example_com_user Setting config preset as default for faspex5 Done. You can test with: ascli faspex5 user profile show Saving configuration file. ``` > [!NOTE] > Paste the entire public key, including the BEGIN and END lines, into the user's profile. For more information on the JWT method, refer to the section below. If you have generated a private key with the wizard and lost the public key, you can retrieve the public key like this: ```shell ascli faspex5 --show-config --out.secrets=yes --fields=private_key | ascli config pubkey @stdin: --out.secrets=yes ``` ### Faspex 5 JWT authentication This is the general purpose and **recommended** method to use. Activation is in two steps: - The administrator must create an API client in Faspex with JWT support This operation is generally done only once: - As Admin, Navigate to the web UI: Admin → Configurations → API Clients → Create - Give a name, like `ascli` - Activate JWT - There is an option to set a global public key allowing the owner of the private key to impersonate any user. Unless you want to do this, leave this field empty. - Click **Create** - Take note of Client ID (and Client Secret, but not used in current version) - The user will authenticate with a private key and set the public key in their Faspex 5 profile. This operation is done by each user using the CLI. - As a user, click the user logo located to the left of the app switcher in the top-right corner. - Select `Account Settings`. - At the bottom, in the `Public key in PEM format` field, paste the **public key** that corresponds to the private key assigned to your account. > [!TIP] > If you don't have a private key, see [Private Key](#private-key) to generate one. Then use these options: ```text --auth=jwt --client-id=<CLIENT_ID> --client-secret=<SECRET> --username=<USERNAME> --private-key=@file:.../path/to/key.pem ``` > [!NOTE] > Use the `private_key` option to provide the PEM content (not the file path). > To load from a file, prefix the path with `@file:`, for example, `@file:/path/to/key.pem`. Typically, users create a preset so they don't have to enter these options each time. Example: ```shell ascli config preset update <F5_PRESET> --auth=jwt --client-id=<CLIENT_ID> --client-secret=<SECRET> --username=<USERNAME> --private-key=@file:.../path/to/key.pem ascli config preset set default faspex5 <F5_PRESET> ascli faspex5 user profile show ``` ### Faspex 5 web authentication For web-based authentication, the administrator must create an **API client** in Faspex to support an external web app: - As Admin, Navigate to the web UI: Admin → Configurations → API Clients → Create - Do not Activate JWT - Set **Redirect URI** to `https://127.0.0.1:8888` - Click **Create** - Take note of the Client Id (and Client Secret, but not used in current version) The user will use the following options: ```text --auth=web --client-id=<CLIENT_ID> --client-secret=<SECRET> --redirect-uri=https://127.0.0.1:8888 ``` ### Faspex 5 public link authentication If all you have is a public link received by email or other, you can still do authorized actions with it. For example, for a public link to post a package: ```shell ascli faspex5 packages send --url='https://faspex5.example.com/?context=<CONTEXT>' ``` ### Faspex 5 bootstrap authentication For `boot` method: (will be removed in future) - As user: Open a Web Browser - Start developer mode - Login to Faspex 5 - Find the first API call with `Authorization` header, and copy the value of the token (series of base64 values with dots) Use this token as password and use `--auth=boot`. ```shell ascli config preset update f5boot --url=https://localhost/aspera/faspex --auth=boot --password=<TOKEN> ``` ### Tested commands for `faspex5` > [!NOTE] > Add `ascli faspex5` in front of the following commands: ```shell admin accounts list admin alternate_addresses list admin clean_deleted admin configuration modify @: mfa_required=false admin configuration show admin contacts list admin distribution_lists create @json:'{"name":"test4","contacts":[{"name":"john@example.com"}]}' admin distribution_lists delete %name:test4 admin distribution_lists list --query.type=global admin email_notifications list admin email_notifications show welcome_email admin event application --query=@ruby:'{"event_type[]"=>["login_success"],"created_at_start"=>(Time.now.utc-60).strftime("%Y-%m-%dT%H:%M:%S.%LZ")}' admin event webhook admin jobs list --query=@json:'{"job_type":"email","status":"failed"}' --fields=id,error_desc admin metadata_profiles list admin node browse %name:Local admin node list admin node shared_folders %name:Local list admin node shared_folders %name:Local show %name:Main admin node shared_folders %name:Local user %name:Main list admin node show %name:Local admin oauth_clients list admin registrations list admin saml_configs list admin shared_inboxes invite %name:my_shared_box_name @: email_address=johnny@example.com admin shared_inboxes list admin shared_inboxes list --query.all=true admin shared_inboxes members %name:my_shared_box_name create %name:john@example.com submit_only admin shared_inboxes members %name:my_shared_box_name delete %name:john@example.com admin shared_inboxes members %name:my_shared_box_name delete %name:johnny@example.com admin shared_inboxes members %name:my_shared_box_name list admin smtp create @json:'{"auth_type":"open","server_address":"smtp.gmail.com","server_port":587,"domain":"gmail.com","tls_enabled":true,"packages_recipient_from":"sender"}' admin smtp modify @: default_time_zone_offset=@val:0 admin smtp show admin smtp test my_email_external admin workgroups list bearer_token gateway @: url=https://localhost:12346/aspera/faspex health --url=https://f5.example.com/path invitation list invitations create @: email_address=aspera.user1+u@gmail.com packages browse --url=<PUBLIC_LINK_RECV_FR_USER> / packages browse <ID> --query.recursive=true packages delete <ID> packages list --box=ALL packages list --box=my_shared_box_name packages list --box=my_workgroup --group-type=workgroups packages list --box=outbox --fields=DEF,sender.email,recipients.0.recipient_type packages list --query=@json:'{"mailbox":"inbox","status":"completed"}' packages receive --box=my_shared_box_name <ID> --to-folder=. packages receive --box=my_workgroup --group-type=workgroups <ID> --to-folder=. packages receive --url=<PUBLIC_LINK_RECV_FR_USER> --to-folder=. packages receive --url=<PUBLIC_LINK_RECV_FR_USER> ALL --to-folder=. packages receive <ID> --to-folder=. --ts.content_protection_password=<PASSWORD> packages receive <ID> <F5_PACK_FIRST_FILE> --to-folder=. --ts.content_protection_password=<PASSWORD> packages receive ALL --once-only=yes --to-folder=. --query.max=5 packages receive INIT --once-only=yes packages send --url=<PUBLIC_LINK_SEND_SHARED_BOX> @json:'{"title":"test title"}' test_file.bin packages send --url=<PUBLIC_LINK_SEND_TO_USER> @json:'{"title":"test title"}' test_file.bin packages send @: 'title=for shared inbox' recipients.0=my_shared_box_name metadata.Options=Opt1 'metadata.TextInput=example text' END test_file.bin packages send @: 'title=test title' recipients.0.name=my_username END test_file.bin --ts.content_protection_password=<PASSWORD> packages send @json:'{"title":"test title","recipients":["my_workgroup"]}' test_file.bin packages send @json:'{"title":"test_webhook_ascli","recipients":["my_shared_box_name"]}' 'faux:///test1?1m' packages show --box=my_shared_box_name <ID> packages show --box=my_workgroup --group-type=workgroups <ID> packages show <ID> packages status <F5_PACK_SEND_SHARED> @list:,failed,completed packages status <ID> postprocessing @: url=https://localhost:8553/asclihook script_folder=/path/to/scripts cert=.../localhost.p12 key=<PASSWORD> shared browse %name:my_shared_folder_name shared list shared_folders browse %name:my_shared_folder_name shared_folders list user account user account --query.expand=true user profile modify @: preference.rows_per_page=10 user profile show version ``` Most commands correspond directly to REST API calls. Parameters to commands are carried through option `query`, as [Extended Value](#extended-value-syntax), for `list`, or through **Command Parameter** for creation. For a single parameter, the dot-path option form is the most direct (for example, `--query.status=completed`). For multiple parameters or when copying directly from API documentation, `@json:` is more suitable. > [!TIP] > The API is listed in [Faspex 5 API Reference](https://developer.ibm.com/apis/catalog?search="faspex+5") under **IBM Aspera Faspex API**. ### Faspex 5: Inbox selection By default, package operations `receive` and `list` are performed on all the user's inboxes, not archived (`inbox_all`). To select another inbox, use option `box` with one of the following values: | `box` | Comment | |---------------------|------------------------------------------| | `inbox_all` | All packages (not archived) **Default**. | | `inbox_all_history` | All archived packages. | | `inbox` | My packages. (not archived) | | `inbox_history` | My archived packages. | | `outbox` | Sent packages. | | `outbox_history` | Archived sent packages. | | `pending` | Pending packages. | | `pending_history` | Archived pending packages. | | `all` | All boxes accessible by current user. | | `ALL` | All boxes of all users. **admin only**. | | `<NAME>` | Name of shared inbox or workgroup.<br/>If option `group_type` is `shared_inboxes`: name of a shared inbox (default).<br/>If `group_type` is `workgroups`: name of workgroup. | > [!NOTE] > In case the name of the `box` is an open value, use option `group_type` set to either `shared_inboxes` or `workgroups`. > A `submit_only` user cannot list packages in a shared inbox. ### Faspex 5: Send a package Use the following command to send a package: ```shell ascli faspex5 packages send <PACKAGE_DATA> <FILE_LIST> ... ``` The `Hash` passed as a command parameter corresponds to the Faspex 5 API endpoint [`POST /packages`](https://developer.ibm.com/apis/catalog/aspera--ibm-aspera-faspex-5-0-api/api/API--aspera--ibm-aspera-faspex-api#createPackageRecord). See the API reference for a full list of supported fields, or inspect such a request in the browser developer tools. The following fields are required: - `title` - the package title - `recipients` - the intended recipients > [!NOTE] > The `box` option does not select a recipient shared inbox. > Use parameter `recipients` to send a package to a specific shared inbox. Basic example (assuming a default preset has been configured for connection information): ```shell ascli faspex5 packages send @json:'{"title":"some title","recipients":["user@example.com"]}' <FILE_PATH> ``` #### Specifying Recipients The Faspex 5 API expects recipients to be an `Array` of `Hash`, each containing a `name` field and an optional `recipient_type` field: ```json {"title":"some title","recipients":[{"recipient_type":"user","name":"user@example.com"}]} ``` Valid values for `recipient_type` are (API): - `user` - `workgroup` - `external_user` - `distribution_list` - `shared_inbox` Other "recipient" fields are also defined in the API and can be used: - `private_recipients` - `notified_on_upload` - `notified_on_download` - `notified_on_receipt` If the provided list is only an `Array` of `String`, then fields `name` and `recipient_type` are resolved automatically using the user's `contacts`. #### Simplified recipient format As a convenience, `ascli` also accepts plain `String`s in the recipients `Array` rather than `Hash`es: ```json {"title":"some title","recipients":["user@example.com"]} ``` When `Strings` are provided, `ascli` automatically looks up the contact across all recipient types. If exactly one match is found, the `name` and `recipient_type` fields are resolved automatically. If no match or multiple matches are found, an exception is raised. > [!NOTE] > The lookup is case-insensitive and on partial matches. #### Sending to a shared inbox To address a shared inbox, set `recipient_type` to `shared_inbox`: ```json {"title":"some title","recipients":[{"recipient_type":"shared_inbox","name":"box name"}]} ``` #### Restricting the lookup to specific recipient types To limit automatic contact lookup to one or more specific types, include the `recipient_types` field with either a single value or an `Array` of values: ```json {"title":"test title","recipient_types":"user","recipients":["user1@example.com","user2@example.com"]} ``` #### Content Protection (Encryption at Rest) To enable content protection (CSEAR), set parameter `ear_enabled` to `true` in the package creation payload. See the Faspex package creation API for full details. The following error is returned by Faspex if CSEAR was not specified in the package creation and if it is configured as mandatory on the server: ```text the provided encryption value (no) does not match the expected server side encryption value (yes) ``` #### Package with metadata To attach metadata to a package, include the `metadata` field in the package creation payload. See the API documentation. Each key corresponds to a metadata field name, and its value is the metadata value: ```json {"title":"test title","recipients":["<SH_INBOX_NAME>"],"metadata":{"Confidential":"Yes","Drop menu":"Option 1"}} ``` ### Faspex 5: List packages Option `box` can be used to list packages from a specific box (see [Inbox Selection](#faspex-5-inbox-selection) above). Option `query` can be used to filter the list of packages, based on native API parameters, directly sent to [Faspex 5 API `GET /packages`](https://developer.ibm.com/apis/catalog/aspera--ibm-aspera-faspex-5-0-api/api/API--aspera--ibm-aspera-faspex-api#getAllPackages). | Parameter | Type | Description | |---------|---------|---------------------------------------------------| | `offset`| Native | Managed by `ascli`: Offset of first package.<br/>Default: `0` | | `limit` | Native | Managed by `ascli`: # of packages per API call.<br/>Default: `100` | | `q` | Native | General search string, case-insensitive.<br/>Matches if value is contained in one of several fields. | | ... | Native | Other native parameters are supported.<br/>See API documentation. | | `max` | Special | Maximum number of **items** to retrieve<br/>Stop pages when the maximum is passed. | | `pmax` | Special | Maximum number of **pages** to request.<br/>Stop pages when the maximum is passed. | A **Command Parameter** in last position, of type `Proc`, can be used to filter the list of packages. The advantage of this method is that the expression can be any test, even complex, as it is Ruby code. But the disadvantage is that the filtering is done in `ascli` and not in Faspex 5, so it is less efficient. Examples: - List only available packages: (filtering is done in Faspex) ```shell ascli faspex5 packages list --query=@json:'{"status":"completed"}' ``` - Similar, using filtering in `ascli`: ```shell ascli faspex5 packages list @ruby:'->(p){p["state"].eql?("released")}' ``` ### Faspex 5: Browsing folder content Several entities support folder browsing: Packages, Nodes, Shared Folders. All support two modes: paging and legacy API. By default, paging is used. Option `query` is available with parameters supported by the API and `ascli`: | Parameter | Evaluation | Default | Description | |-----------|--------------|-------------------| ----------------------------------------| | `paging` | `ascli` | `true` | Use paging API. | | `recursive`| `ascli` | `false` | List inside folders. | | `max` | `ascli` | - | Maximum number of items. | | `filter` | API | `{"basenames":[]}`| See API doc. | | `offset` | API (legacy) | `0` | Index of first item. | | `limit` | API (legacy) | `500` | Number of items in one API call result. | | `per_page`| API (paging) | `500` | Number of items in one API call result. | ### Faspex 5: Content of a received Package > [!NOTE] > Listing content also applies to sent packages using `--box=outbox`. To list the content of a received package, use command `faspex5 packages browse <PACKAGE_ID>`. Optionally, provide a folder path. ### Faspex 5: Receive a package To receive one, or several packages at once, use command `faspex5 packages receive`. Provide either a single package ID, or an [Extended Value](#extended-value-syntax) `Array` of package IDs, for example, `@list:,1,2,3` as argument. The same options as for `faspex5 packages list` can be used to select the box and filter the packages to download. That is, options `box` and `query`, and the last **Command Parameter** `Proc` (filter). Option `--once-only=yes` can be used, for "cargo-like" behavior. Special package ID `INIT` initializes the persistency of already received packages when option `--once-only=yes` is used. Special package ID `ALL` selects all packages (of the selected box). In this case, typically, only `completed` packages should be downloaded, so use option `--query.status=completed` (or equivalently `--query=@json:'{"status":"completed"}'`). If a package is password protected, then the content protection password is asked interactively. To keep the content encrypted, use option: `--ts=@json:'{"content_protection":null}'`, or provide the password instead of `null`. > [!TIP] > If you use option `query` and/or positional `filter`, you can use the `list` command for a dry run. ### Faspex 5: List all shared inboxes and work groups > [!NOTE] > **Shared inboxes** (`faspex5 admin shared_inboxes`) are package recipients (dropboxes), accessible under the **Packages** app. > They are distinct from **shared folders** (`faspex5 shared_folders`), which are node-based storage access points accessible under the **Files** app. If you are a regular user, to list work groups you belong to: ```shell ascli faspex5 admin workgroups list ``` If you are admin or manager, add option `--query=@json:'{"all":true}'`: this lists items you manage, even if you do not belong to them. Example: ```shell ascli faspex5 admin shared_inboxes list --query=@json:'{"all":true}' --fields=id,name ``` Shared inbox members can also be listed, added, removed, and external users can be invited to a shared inbox. ```shell ascli faspex5 admin shared_inboxes invite_external_collaborator '%name:the shared inbox' @: email_address=john@example.com ``` It is equivalent to: ```shell ascli faspex5 admin shared_inboxes invite_external_collaborator '%name:the shared inbox' @json:'{"email_address":"john@example.com"}' ``` Other payload parameters are possible for `invite_external_collaborator` in this last `Hash` **Command Parameter**: ```json {"description":"blah","prevent_http_upload":true,"custom_link_expiration_policy":false,"invitation_expires_after_upload":false,"set_invitation_link_expiration":false,"invitation_expiration_days":3} ``` ### Faspex 5: Create Metadata profile ```shell ascli faspex5 admin metadata_profiles create @json:'{"name":"the profile","default":false,"title":{"max_length":200,"illegal_chars":[]},"note":{"max_length":400,"illegal_chars":[],"enabled":false},"fields":[{"ordering":0,"name":"field1","type":"text_area","require":true,"illegal_chars":[],"max_length":100},{"ordering":1,"name":"fff2","type":"option_list","require":false,"choices":["opt1","opt2"]}]}' ``` ### Faspex 5: Create a Shared inbox with specific metadata profile ```shell ascli faspex5 admin shared_inboxes create @json:'{"name":"the shared inbox","metadata_profile_id":1}' ``` ### Faspex 5: List content in Shared folder and send package from remote source > [!NOTE] > **Shared folders** (`faspex5 shared_folders`) provide node-based storage access, accessible under the **Files** app. > They are distinct from **shared inboxes** (`faspex5 admin shared_inboxes`), which are package recipients (dropboxes) accessible under the **Packages** app. ```shell ascli faspex5 shared_folders list --fields=id,name ``` ```text ╭────┬──────────────╮ │ id │ name │ ╞════╪══════════════╡ │ 2 │ Server Files │ ╰────┴──────────────╯ ``` ```shell ascli faspex5 shared_folders browse %name:'Server Files' /folder ``` ```shell ascli faspex5 packages send @json:'{"title":"hello","recipients":[{"name":"<RECIPIENT>"}]}' --shared-folder=%name:partages /folder/file ``` > [!TIP] > The shared folder can be identified by its numerical `id` or by name using the [percent selector](#percent-selector): `%<FIELD>:<VALUE>`. For example: `--shared-folder=3` or `--shared-folder=%name:partages`. ### Faspex 5: Receive all packages (cargo) To receive all packages, only once, through persistency of already received packages: ```shell ascli faspex5 packages receive ALL --once-only=yes --query=@json:'{"status":"completed"}' ``` To initialize, that is, skip all current packages, so that next time `ALL` is used, only newer packages are downloaded: ```shell ascli faspex5 packages receive INIT --once-only=yes ``` ### Faspex 5: Invitations There are two types of invitations for package submission: public or private. Public invitations are for external users; provide only the email address. ```shell ascli faspex5 invitations create @json:'{"email_address":"john@example.com"}' --fields=access_url ``` Private invitations are for internal users: provide the user or shared inbox identifier through field `recipient_name`. ### Faspex 5: Cleanup packages > [!WARNING] > Operation requires admin level. The default automated cleanup period can be displayed with: ```shell ascli faspex5 admin configuration show --fields=days_before_deleting_package_records ``` This parameter can be modified with: ```shell ascli faspex5 admin configuration modify @json:'{"days_before_deleting_package_records":30}' ``` To start package purge, that is, permanently remove packages marked for deletion older than `days_before_deleting_package_records`, use command: ```shell ascli faspex5 admin clean_deleted ``` > [!NOTE] > The expiration period taken by default is the one from `admin configuration show`. > To use a different period than the default, specify it on command line with: `@json:'{"days_before_deleting_package_records":15}'` To delete all packages, one can use the following command: ```shell ascli faspex5 packages list --box=ALL --format=yaml --fields=id | ascli faspex5 packages delete @yaml:@stdin: ``` > [!NOTE] > Above command will mark all packages for deletion, and will be permanently removed after the configured period (`clean_deleted` command). > It is possible to add a filter to the list command to only delete packages matching some criteria, for example, using `--select=@ruby:'->(p){...}'` on `packages list`. ### Faspex 5: Admin: Unlock user To unlock a user, you can deactivate and then re-activate the user: ```shell ascli faspex5 admin accounts modify %name:some.user@example.com @json:'{"account_activated":false}' ``` ```shell ascli faspex5 admin accounts modify %name:some.user@example.com @json:'{"account_activated":true}' ``` > [!TIP] > This example uses the [percent selector](#percent-selector), but the numerical ID can be used as well. To send a password reset link to a user, use command `faspex5 admin accounts reset_password <ACCOUNT_ID>`. ### Faspex 5: Faspex 4-style post-processing The command `ascli faspex5 postprocessing` emulates Faspex 4 post-processing script execution in Faspex 5. It implements a web hook for Faspex 5 and calls a script with the same environment variables as set by Faspex 4. Environment variables are set to the values provided by the web hook, which are the same as Faspex 4 post-processing. Only values whose key starts with `faspex_` are passed: other keys are ignored, as they could change the behavior of the script (e.g. `PATH`). A structured value (object or array) is passed as JSON. It allows migrating workflows from Faspex 4 to Faspex 5 while preserving scripts. Nevertheless, in the long term, a native approach shall be considered, such as using Aspera Orchestrator or other workflow engine, using Faspex 5 native web hooks or File Processing. It is invoked like this: ```shell ascli faspex5 postprocessing ``` An optional positional parameter can be provided as [Extended Value](#extended-value-syntax) `Hash`: | Field | Type | Description | |-----------------------|-----------------------|----------------------------------------------------------------------------------| | `allowed_ips` | `String, Array[String]` | IP address or network (CIDR, e.g. `10.89.0.0/24`), or list of them, allowed to call the listener. Requests from any other source IP are rejected with HTTP 403. When not set, no IP restriction is applied. This parameter or `secret_token` is required when `url` is not a local address. | | `cert` | `String` | Path to the TLS certificate file. Accepted formats: PEM (`.pem`) or PKCS12 (`.p12` / `.pfx`).<br/>Example: `/path/to/cert.pem`. | | `chain` | `String` | Path to the PEM certificate chain file (appended as extra chain certificates).<br/>Example: `/path/to/chain.pem`. | | `fail_on_error` | `Bool` | When `true`, return HTTP 500 if the script exits with a non-zero status code. When `false` (default), the exit code is reported in the response body but HTTP 200 is returned.<br/>Default: `false`. | | `key` | `String` | Path to the PEM private key file, or the PKCS12 passphrase when `cert` is a `.p12`/`.pfx` file.<br/>Example: `/path/to/key.pem`. | | `script_folder` | `String` | Folder containing the post-processing scripts. The URL path (after the base path) is appended to this folder to locate the script to execute.<br/>Default: `.`.<br/>Example: `/opt/faspex/scripts`. | | `secret_token` | `String` | Shared secret for webhook authentication. When set, every incoming request must include `?token=<value>` in the URL; requests without the correct token are rejected with HTTP 401. Configure the Faspex webhook URL as `http://host:port/path/script.sh?token=<value>`. This parameter or `allowed_ips` is required when `url` is not a local address. Generate it with `openssl rand -hex 32`: a weak token is refused (see `secret_token_min_bits`). | | `secret_token_min_bits` | `Integer` | Minimum estimated strength of `secret_token`, in bits (log2 of the number of guesses to find it). The strength is estimated with gem `zxcvbn` if installed, else with a built-in estimate that detects repeats, sequences, keyboard rows, words and years. `0` disables the check. A random token of 32 bytes (`openssl rand -hex 32`) is far above the default.<br/>Default: `80`. | | `timeout_seconds` | `Integer` | Maximum time in seconds to wait for the script to complete before killing it.<br/>Default: `60`. | | `url` | `String` | Address and port the server listens on. Use `https://` with `cert`/`key` for TLS.<br/>Default: `http://localhost:8080`. | For details on `url` and HTTPS (`cert`, `key`, `chain`), see [Web service](#web-service). When a request on `ascli` is received the following happens: - If `allowed_ips` is set, `ascli` checks the source IP address of the request (else HTTP 403). - If `secret_token` is set, `ascli` checks the `token` parameter of the URL (else HTTP 401). - `ascli` gets the path of the URL called. - It removes the **base path** of base URL. - It prepends it with the value of `script_folder`. - It executes the script at that path, if it exists (else HTTP 404) and is inside `script_folder` (else HTTP 403). - The script is killed if it does not finish within `timeout_seconds`. - Upon success, a success code is returned. For example: ```shell ascli faspex5 postprocessing @json:'{"url":"http://0.0.0.0:8080/processing","script_folder":"/opt/scripts","secret_token":"<TOKEN>"}' ``` Generate the token with `openssl rand -hex 32`. A weak token is refused at startup: its strength is estimated in bits, and shall be at least `secret_token_min_bits`. The estimate uses the optional gem `zxcvbn` if installed, else a built-in estimate. In Faspex 5, the URL of the webhook endpoint shall be reachable from within Faspex containers. For example, if `ascli` is running on the host, the URL hostname shall not be `localhost`, as this refers to the local address inside the Faspex container. Instead, one can specify the **IP address of the host** or `host.containers.internal` (Check `podman` manual). So, `ascli` shall listen on an address reachable from the containers: the IP address of the host, or `0.0.0.0` for all interfaces. > [!NOTE] > **Security**: by default the listener binds to loopback (`localhost`) and is not reachable from outside the host (see [Web service](#web-service)). > When `url` is not a loopback address, at least one of the following protections is required: > > - **`secret_token`**: include the token in the Faspex webhook URL: `http://host:8080/processing/script1.sh?token=<TOKEN>`. Requests without the correct token are rejected with HTTP 401. > - **`allowed_ips`**: restrict the listener to the known Faspex container IPs or network, e.g. `@json:'{"allowed_ips":["192.168.1.10","10.89.0.0/24"]}'`. Requests from other IPs are rejected with HTTP 403. Define the web hook as follows: **Webhook endpoint URI**: `http://host.containers.internal:8080/processing/script1.sh?token=<TOKEN>` Then the post-processing script executed will be `/opt/scripts/script1.sh`. A Ruby script can also be evaluated in the process of `ascli`, instead of being executed in a new process. For this, add `lambda` to the query of the URL, e.g. `http://host.containers.internal:8080/processing/notify.rb?token=<TOKEN>&lambda`. The script shall have extension `.rb`, and return a lambda, which is called with all the webhook values (`Hash`). For example: ```ruby ->(params) { File.write('/tmp/last_package.txt', params['faspex_pkg_name']) } ``` The lambda runs with the privileges and configuration of `ascli`, and is stopped after `timeout_seconds`. ### Faspex 5: Faspex 4 Gateway > [!NOTE] > This is not a feature for production. > It's provided for testing only. For legacy Faspex client applications that use the `send` API (only) of Faspex v4, the command `gateway` provides the capability to present an API compatible with Faspex 4, and it will call the Faspex 5 API. It takes an optional `Hash` argument: | Field | Type | Description | |-------------------|---------|----------------------------------------------------------------------------------| | `cert` | `String` | Path to the TLS certificate file. Accepted formats: PEM (`.pem`) or PKCS12 (`.p12` / `.pfx`).<br/>Example: `/path/to/cert.pem`. | | `chain` | `String` | Path to the PEM certificate chain file (appended as extra chain certificates).<br/>Example: `/path/to/chain.pem`. | | `key` | `String` | Path to the PEM private key file, or the PKCS12 passphrase when `cert` is a `.p12`/`.pfx` file.<br/>Example: `/path/to/key.pem`. | | `password` | `String` | Password expected from clients in HTTP Basic authentication, for the above `username`. Generate it with `openssl rand -hex 32`: a weak password is refused (see `password_min_bits`).<br/>Example: `<PASSWORD>`. | | `password_min_bits` | `Integer` | Minimum estimated strength of `password`, in bits (log2 of the number of guesses to find it). The strength is estimated with gem `zxcvbn` if installed, else with a built-in estimate that detects repeats, sequences, keyboard rows, words and years. `0` disables the check. A random password of 32 bytes (`openssl rand -hex 32`) is far above the default.<br/>Default: `80`. | | `url` | `String` | Address and port the server listens on. Use `https://` with `cert`/`key` for TLS.<br/>Default: `http://localhost:8080`. | | `username` | `String` | Username expected from clients in HTTP Basic authentication. Set together with `password`. When not set, requests are accepted without authentication. Mandatory when `url` is not a loopback address (`localhost`, `127.0.0.0/8`, `::1`).<br/>Example: `fx4_user`. | For details on `url` and HTTPS (`cert`, `key`, `chain`), see [Web service](#web-service). Example: ```shell ascli faspex5 gateway @json:'{"url":"https://localhost:12345/aspera/faspex","username":"fx4","password":"<PASSWORD>"}' ``` Clients authenticate with HTTP Basic, as on Faspex 4, with the `username` and `password` of the gateway. Generate the password with `openssl rand -hex 32`. A weak password is refused at startup: its strength is estimated in bits, and shall be at least `password_min_bits`. `username` and `password` are mandatory when `url` is not a loopback address (see [Web service](#web-service)): anyone reaching the gateway could send packages to anyone, as the Faspex 5 user. When they are not set, requests are accepted without authentication. There are many limitations: - It's only to emulate the Faspex 4 `send` API (send package). - No support for remote sources, only for an actual file transfer by the client. - The client must use the transfer spec returned by the API (not `faspe:` URL). - Tags returned in transfer spec must be used in transfer. - Only a single authentication is possible (per gateway) on Faspex 5: all packages are sent by the same Faspex 5 user. Behavior: The API client calls the Faspex 4 API on the gateway, then the gateway transforms this into a Faspex 5 API call, which returns a transfer spec, which is returned to the calling client. The calling client uses this to start a transfer to HSTS, which is managed by Faspex 5. ### Faspex 5: Get Bearer token to use API If a command is missing, then it is still possible to call the API directly on the command line using `curl`: ```shell curl -H "Authorization: $(ascli faspex5 bearer_token --out.secrets=yes)" https://faspex5.example.com/aspera/faspex/api/v5/<API_ENDPOINT> ``` ## Plugin: `shares`: IBM Aspera Shares v1 Aspera Shares supports the **Node API** for the file transfer part. Supported commands are listed in the Shares API documentation: <https://developer.ibm.com/apis/catalog/aspera--aspera-shares-api/Introduction> The payload for creation is the same as for the API: parameters are provided as a positional `Hash`. Example: Create a Node: Attributes are like API: | Attribute | Required | Default | |--------------|----------|---------| | `name` | Yes | | | `host` | Yes | | | `api_username` | Yes | | | `api_password` | Yes | | | `port` | | `9092` | | `ssl` | | `true` | | `verify_ssl` | | `false` | | `timeout` | | `30s` | | `open_timeout` | | `10s` | Example: Create a share, grant access to a user and list user permissions on it. ```shell ascli shares admin share create @json:'{"node_id":1,"name":"test1","directory":"test1","create_directory":true}' user_id=$(ascli shares admin user all show %username:john@example.com --fields=id --out.level=data) ascli shares admin share user_permissions %name:test1 create @json:'{"user_id":'$user_id',"browse_permission":true,"download_permission":true,"upload_permission":true}' ascli shares admin share user_permissions %name:test1 list ``` Share permissions are granted, modified and revoked on the share, with `share user_permissions` and `share group_permissions` (`create`, `list`, `show`, `modify`, `delete`). They are identified by the user or group identifier: e.g. `show %username:john@example.com`. Permissions not specified on creation take their default value. Available permissions: `browse_permission`, `download_permission`, `upload_permission`, `mkdir_permission`, `delete_permission`, `rename_permission`, `content_availability_permission`, `manage_permission`. > [!NOTE] > The permissions of a user or group on all shares are listed with `user all share_permissions` and `group all share_permissions`. > These are read-only (`list` and `show`): to change them, use `share user_permissions` and `share group_permissions`. Example: Give a user (`jane@example.com`) the same share permissions as another user (`john@example.com`), using `jq`. ```shell new_user_id=$(ascli shares admin user all show %username:jane@example.com --fields=id --out.level=data) ascli shares admin user all share_permissions %username:john@example.com list --format=json --out.level=data | jq -c '.[]' | while read -r permission; do ascli shares admin share user_permissions "$(jq .share_id <<< "$permission")" create @json:"$(jq -c --argjson id "$new_user_id" '{user_id: $id} + with_entries(select(.key | endswith("_permission")))' <<< "$permission")" done ``` ### Authentication of SAML users By default, `shares` authenticates with `username` and `password`. A user that has only a SAML identity (no local password) can authenticate with a web browser: `--auth=web`. ```shell ascli shares --url=https://shares.example.com --auth=web files browse / ``` Option `ui.web_auth` selects how the result of the login is received (see [Web login flow](#web-login-flow)): - `port` (default): automatic when the browser runs on the same system as `ascli`. `ascli` listens on a local port (`--redirect-uri`, default: `http://localhost:12345`), opens the SAML login page of Shares (`/auth/saml`) in the browser with this URL in `RelayState`, and receives the OAuth tokens there. The host of `redirect_uri` shall be loopback: `localhost`, an address of `127.0.0.0/8`, or `[::1]`. - `paste`: the browser may run on another system than `ascli` (for example, `ascli` runs on a server accessed with SSH), without tunnel. No local port is opened and `--redirect-uri` is not used. `ascli` generates a random `state` and opens the SAML login page with `RelayState` `aspera://device-code?state=<state>`. After login, Shares displays a code (a long text of letters, digits, `-` and `_`). The user copies it and pastes it in the terminal (nothing is displayed, as the code contains a token). If standard input is not a terminal, the code is read from it. `ascli` checks that the `state` of the code is the one it generated. The code is valid for **15 to 30 minutes** after its display: it is encrypted with a key derived from the `state` and from the current time in slots of 15 minutes (the previous and the next slot are also accepted). So the clocks of the Shares server and of the system that runs `ascli` shall be correct (less than 15 minutes apart), and the code shall be pasted soon after the login, otherwise `ascli` reports an expired code or a wrong clock, and the login shall be done again. The code is also protected against typing errors and modification. The token is saved, and renewed with its refresh token: the browser is opened again only if this fails. Option `username` is optional: with it, the token is saved for this user. > [!IMPORTANT] > Shares does not have the OAuth authorization code grant: both ways require a specific patch of Shares, which returns the tokens when `RelayState` is a loopback URL (`port`) or `aspera://device-code?state=<state>` (`paste`). > The patch is the file [`docs/shares/zz_saml_cli_relay_state.rb`](shares/zz_saml_cli_relay_state.rb) of the GitHub repository. > It is temporary code, provided until an equivalent function is officially released in Shares. > It modifies no file of Shares: to install it, copy it to `<shares_rails_root>/config/initializers/` on the Shares server and restart Shares; to remove it, delete the file and restart Shares. > With a Shares without the patch, `ascli` reports that no token was received. > This feature works only with this patch: it is not a standard function of Shares, and it has not been validated on a released version of Shares yet. > It may change with the final implementation in Shares: the `RelayState` values, the format of the code (and so its validity and its encryption) and the returned fields can differ, and `ascli` will then be adapted. Do not rely on the current format outside of `ascli`. > The tokens given by Shares go through the browser (in the URL, or in the code): `ascli` immediately exchanges the refresh token for a new token, so that the tokens that were visible are not valid any more. ### Tested commands for `shares` > [!NOTE] > Add `ascli shares` in front of the following commands: ```shell admin authenticate @: username=my_username password=<PASSWORD> admin directory list admin group all list admin group search list admin node list admin share list --fields=DEF,-status,status_message admin share user_permissions %name:my_share list admin share user_permissions %name:my_share modify %username:my_username @json:'{"browse_permission":true}' admin transfer_settings modify @: min_connect_version=3.6.1 admin transfer_settings show --format=json admin user all app_authorizations %username:my_username modify @: app_login=true admin user all app_authorizations %username:my_username show admin user all list admin user all share_permissions %username:my_username list admin user all share_permissions %username:my_username show %name:my_share admin user ldap add the_name admin user local list admin user saml import @json:'{"id":"the_id","name_id":"the_name"}' admin user search list files browse / files delete my_share_folder/new_folder files delete my_share_folder/test_file.bin files download --to-folder=. my_share_folder/test_file.bin files download --to-folder=. my_share_folder/test_file.bin my_share_folder/test_file.bin --transfer.agent=httpgw --transfer.url=https://tst.example.com/path files mkdir my_share_folder/new_folder files sync push /data/local_sync --to-folder=my_share_folder/synctst files sync push /data/local_sync --to-folder=my_share_folder/synctst @: reset=true files upload 'faux:///test-file?1m' --to-folder=my_share_folder --transfer.agent=httpgw --transfer.url=https://tst.example.com/path --transfer.synchronous=true --transfer.api_version=v1 --transfer.upload_chunk_size=100000 files upload --to-folder=my_share_folder test_file.bin files upload --to-folder=my_share_folder test_file.bin --transfer.agent=httpgw --transfer.url=https://tst.example.com/path files upload send_folder --to-folder=my_share_folder --transfer.agent=httpgw --transfer.url=https://tst.example.com/path --transfer.synchronous=true --transfer.api_version=v1 --transfer.upload_chunk_size=100000 health --url=https://shares.example.com/path info share list ``` ## Plugin: `console`: IBM Aspera Console ### Transfer filter Listing transfers supports the API syntax. In addition, it is possible to place a single `query` parameter in the request to filter the results: `filter`, following the syntax: ```text (field operator value)and(field operator value)... ``` ### Tested commands for `console` > [!NOTE] > Add `ascli console` in front of the following commands: ```shell endpoint list health ssh_key list transfer current files <ID> transfer current list --query.filter='(transfer_name contain aoc)' transfer current list --query=@json:'{"filter1":"transfer_name","comp1":"contain","val1":"aoc"}' transfer current show <ID> transfer smart list transfer smart sub <SMART_ID> @: source.paths.0=my_smart_file source_type=user_selected ``` ## Plugin: `orchestrator`: IBM Aspera Orchestrator To write workflow files without the designer, refer to the [Aspera Orchestrator Development Guide](https://github.com/laurent-martin/aspera-orchestrator-plugins-doc/blob/main/docs/orchestrator-development-guide.md): file format, graph, execution, examples and REST API. Command `workflows development` gives access to the documentation for workflow development, read from GitHub. Without argument, it lists the documents: the guide (`guide`), the [Action Template Reference](https://github.com/laurent-martin/aspera-orchestrator-plugins-doc/blob/main/docs/action-template-reference.md) (`templates`) and the Plugin Manual (`manual`). After the name of a document: `toc` lists its sections, `full` displays the whole document, and `section <anchor>` displays only one section: the guide and the manual are large, use `toc` first. It does not require access to Orchestrator. The [MCP server](#plugin-mcp-model-context-protocol-server) instructs AI assistants to read it before writing or modifying a workflow. ```shell ascli orchestrator workflows development guide toc ascli orchestrator workflows development templates section customruby ``` Commands `plugins spec` display the specifications of action plugins, read from GitHub: [plugins.yml](https://github.com/laurent-martin/aspera-orchestrator-plugins-doc/blob/main/docs/plugins.yml). For each plugin: template attributes (label, allowed values, defaults), and inputs and outputs of its steps, with the template settings they depend on. The fields are described in section [Plugin specifications](https://github.com/laurent-martin/aspera-orchestrator-plugins-doc/blob/main/docs/orchestrator-development-guide.md#plugin-specifications) of the guide. `toc` lists the plugins, with the anchor of their section in the Plugin Manual (`manual_anchor`, for `workflows development manual section`). `show` with a plugin name (class name, case insensitive) displays its specification, or one section: `attributes`, `inputs` or `outputs`. It does not require access to Orchestrator. Option `plugin_specs` gives another URL or path of the file, for example for another version of Orchestrator. ```shell ascli orchestrator plugins spec toc --select=@json:'{"category":"Triggers"}' ascli orchestrator plugins spec show AmazonS3Operation --format=yaml ascli orchestrator plugins spec show ascpclient inputs ``` ### Authentication The Orchestrator plugin supports different credentials and authentication styles configured via `--auth_style`: - **Username / Password** (`--username` and `--password`): - `--auth_style=token` (default): Exchanges credentials for a JWT Bearer token via `/api/login`. - `--auth_style=basic`: Standard HTTP Basic Authentication. - `--auth_style=query`: Passes credentials in URL query parameters (`?login=...&password=...`). - **Username / API Key** (`--username` and `--apikey`): the API key is the one of the user, so the user name is required. - `--auth_style=token` (default): Exchanges the API key for a JWT Bearer token via `/api/login`. - `--auth_style=query`: Passes the API key in URL query parameters (`?login=...&api_key=...`). ### Start a workflow Command `workflows start` creates a work order: ```shell ascli orchestrator workflows start <WORKFLOW_ID> [<PARAMETERS>] [<EXECUTION>] ``` - `parameters`: `Hash` of external parameters of the workflow (optional). - `execution`: `Hash` controlling the execution of the work order (optional): | Key | Type | Description | |---------------|-----------|-------------| | `synchronous` | `Boolean` | Wait for completion of the work order (default: `false`). | | `step` | `String` | Name of the work step providing the result. | | `variable` | `String` | Name of the output variable of `step` returned as result. | `step` and `variable` must be provided together, and imply `synchronous`. By default, the call is asynchronous and returns the work order information. Example: Start workflow `1234` with parameter `Param`, wait for completion and display the value of output `Complete_status_message` of step `ResultStep`: ```shell ascli orchestrator workflows start 1234 @json:'{"Param":"world !"}' @json:'{"step":"ResultStep","variable":"Complete_status_message"}' ``` > [!NOTE] > Options `synchronous` and `result` (`--result=<WORK_STEP>:<VARIABLE>`) are deprecated: use `execution` instead. ### Import a workflow Command `workflows export` displays a workflow in Orchestrator's export format (YAML). With option `dependencies`, it saves instead a package (`.wkf`) that also contains the sub-workflows and remote nodes used by the workflow, in the folder given by option `to_folder` (default: current folder). Command `workflows import` creates a new workflow from such a file, and displays it, including its new `id`. The imported workflow is a draft: publish it with `workflows publish` before starting it. Example: Copy workflow `1234` to another Orchestrator, where the imported workflow gets `id` `5678`: ```shell ascli orchestrator workflows export 1234 --out.file=my_workflow.yml ascli orchestrator workflows import my_workflow.yml --preset=other_orchestrator --out.fields=id ascli orchestrator workflows publish 5678 --preset=other_orchestrator ``` Example: Save workflow `1234` with its dependencies in folder `/tmp`, and import the package: ```shell ascli orchestrator workflows export 1234 @json:'{"dependencies":true}' --to-folder=/tmp ascli orchestrator workflows import /tmp/Workflow_my_workflow_v100.wkf ``` Command `workflows import` always creates a new workflow. Command `workflows import_with_constraints` imports a workflow file already present on the Orchestrator host, either as a new workflow or as a new revision of an existing workflow (`add_as_revision`), and resolves conflicts with existing global templates, sub-workflows and remote nodes. Command `workflows import` stores each uploaded file in folder `<run_dir>/workflows/import/` of the Orchestrator host. Example: Add the file uploaded by a previous `workflows import` as a new revision of workflow `1234`, and publish it: ```shell ascli orchestrator workflows import_with_constraints @json:'{"filename":"/opt/aspera/orchestrator/var/run/orchestrator/workflows/import/my_workflow.yml","add_as_revision":1234}' ascli orchestrator workflows publish 1234 ``` #### Check a workflow before import Orchestrator does not validate the graph of a workflow on import: an invalid workflow is imported, then fails to publish, or at run time. Command `workflows check_import` checks a workflow file (`.yml`) or package (`.wkf`) with the rules of the workflow designer, and does not need access to Orchestrator: ```shell ascli orchestrator workflows check_import my_workflow.yml ``` It displays one line per issue, or `no problem found`. An issue has these fields: | Field | Description | |------------|-------------| | `severity` | `error`: the workflow will not import, publish or run correctly. `warning`: to check. | | `code` | Stable identifier of the rule, for programs, for example `graph.step_orphan`. | | `file` | File of the package where the issue is (only for a `.wkf`). | | `path` | Where in the file, for example `graph/Step[Say hello]`. | | `message` | Description. | Checked: format of the file (YAML, number of elements, plain format layout), attributes, `error_free`, action template keys, XML of the graph, `Start`, `End`, `Fail` and `Error` nodes, unique step names, step template, prerequisites, `Synch_factor`, steps that have no output or no prerequisite, references to outputs and parameters. With the specifications of plugins (see `plugins spec show`), it also checks action templates and the inputs of steps, for the settings of each template: - plugin class name, attributes of the template, and their values (type, allowed values), - required inputs not mapped (`step.input_missing`), - inputs that the plugin does not have with this template, which Orchestrator ignores (`step.input_unknown`), - mapped outputs that the source step does not produce (`step.output_not_produced`). If the specifications cannot be read (option `plugin_specs`), these checks are skipped, with a warning (`plugins.specs_unavailable`). Not checked: the plugins enabled on the server (see `workflows find_constraints`). Command `workflows import` runs the same check, and logs the issues as warnings; it does not stop the import. Use `--log-level=warn` to see them. The check is done by `ascli`, behind one method that has the shape of a REST operation: the file goes in, a structure with `valid`, `file_name` and the list of `issues` comes out. It can be replaced by a call to Orchestrator if such an API becomes available. #### Resolve conflicts before import The web interface of Orchestrator lists the conflicts with existing sub-workflows, global templates and remote nodes, and the plugins to enable, and asks for a choice for each of them. Command `workflows find_constraints` does the same for a file present on the Orchestrator host (see `workflows import_with_constraints`): ```shell ascli orchestrator workflows find_constraints /opt/aspera/orchestrator/var/run/orchestrator/workflows/import/my_workflow.yml ascli orchestrator workflows find_constraints /opt/aspera/orchestrator/var/run/orchestrator/workflows/import/my_workflow.yml 1234 ``` The optional second argument is the id of the workflow that receives the file as a new revision. Each line is a conflict (`type`: `subwf_constraints`, `action_template_constraints`, `remote_node_constraints`), with its `name` and the possible `choices`, or a plugin to enable (`missing_plugins`). The `type` is the key to use in the argument of `import_with_constraints`. Orchestrator does not check that the file exists: a wrong path gives the same answer as a file without conflict. Orchestrator does not refuse an incomplete resolution: `workflows import_with_constraints` checks with `find_constraints` first, and refuses to import in these cases: - Plugins are not enabled and `auto_enable_missing_plugins` is not `true` (Orchestrator fails with an internal error). - A resolution is given for some conflicts of a type, but not all (Orchestrator imports the others as copies, without error). `auto_enable_missing_plugins` must be a boolean (`true` or `false`): Orchestrator takes the string `"false"` as true, and `ascli` refuses a string. ### Tested commands for `orchestrator` > [!NOTE] > Add `ascli orchestrator` in front of the following commands: ```shell --auth_style=query workflow list --auth_style=token workflow list alerts list health info journal files journal --query=@json:'{"details":"false"}' monitors list node monitors snapshot plugins list plugins spec show AmazonS3Operation --format=yaml plugins spec show ascpclient inputs plugins spec toc --select=@json:'{"category":"Triggers"}' portlets list processes queues list queues stats tasks list workflow check_import orch_wf_check.yml workflow details <WORKFLOW_ID> workflow export <WORKFLOW_ID> workflow inputs <WORKFLOW_ID> workflow list workflow outputs <WORKFLOW_ID> workflow start <SLEEP_WORKFLOW_ID> --fields=work_order.id workflow start <WORKFLOW_ID> @: 'Param=world !' workflow start <WORKFLOW_ID> @: 'Param=world !' END @: step=ResultStep variable=Complete_status_message workflow status <WORKFLOW_ID> workflow status ALL workflow workorders <WORKFLOW_ID> workflow workorders <WORKFLOW_ID> --fields=id --query.max_results=1 workflows development guide toc workflows development templates section customruby workorder cancel <ID> workorder output <ID> workorder reset <ID> workorder status <ID> workorder status <ORCH_WF_START_SLEEP> --fields=status_details workorders stats worksteps cancel <ID> worksteps stats worksteps status <ID> ``` ## Plugin: `cos`: IBM Cloud Object Storage IBM Cloud Object Storage supports high-speed transfers using the FASP protocol. These transfers leverage the same service used by Aspera on Cloud, called the Aspera Transfer Service (ATS). You can check the list of available ATS regions here: <https://status.aspera.io>. There are two ways to provide credentials: - Using existing credentials If you already have the endpoint, API key, and Resource Instance ID (CRN), use this method. - Using IBM Cloud Console access If you do not have credentials but have access to the IBM Cloud Console, use this alternative method. ### Using endpoint, API key and Resource Instance ID (CRN) If you already have these parameters, provide the following options to `ascli`: | Option | Description | |------------|---------------------------------------------------| | `bucket` | Bucket name | | `endpoint` | Storage endpoint URL<br/>for example, `https://s3.hkg02.cloud-object-storage.appdomain.cloud` | | `apikey` | API Key | | `crn` | Resource instance ID | Example: Create a Default Configuration ```shell ascli config preset update <COS_PRESET_NAME> --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=<API_KEY> --crn=<CRN> ascli config preset set default cos <COS_PRESET_NAME> ``` Once configured, proceed to the [transfer example](#operations-transfers). ### Using service credential file If you are the COS administrator and do not yet have credentials, you can create them directly from the IBM Cloud Console (Web UI): Steps: - → Navigation Menu - → [Resource List](https://cloud.ibm.com/resources) - → [Storage](https://cloud.ibm.com/objectstorage) - → Select your storage instance - → Service Credentials - → New credentials (Leave default role: Writer, no special options) - → Copy to clipboard Save the copied JSON value to a file, for example: `$HOME/cos_service_creds.json` Or, using the IBM Cloud CLI: ```shell ibmcloud resource service-keys ibmcloud resource service-key <SERVICE_KEY_NAME> --output JSON|jq '.[0].credentials'>$HOME/service_creds.json ``` > [!NOTE] > If `jq` is not installed, you can manually extract the credentials section from the JSON output. The service credential file consists of the following structure: ```json { "apikey": "<API_KEY>", "cos_hmac_keys": { "access_key_id": "<ACCESS_KEY>", "secret_access_key": "<SECRET>" }, "endpoints": "https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints", "iam_apikey_description": "<DESCRIPTION>", "iam_apikey_name": "<NAME>", "iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Writer", "iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/xxxxxxx.....", "resource_instance_id": "crn:v1:bluemix:public:cloud-object-storage:global:a/xxxxxxx....." } ``` The field mappings are as follows: - `resource_instance_id` → option `crn` - `apikey` → option `apikey` > [!NOTE] > Endpoints for regions can be found by querying the `endpoints` URL in the JSON file or from the IBM Cloud Console. The required options for this method are: | Option | Description | |-----------------------|------------------------------------------------| | `bucket` | Bucket name | | `region` | Bucket region<br/>for example, `eu-de` | | `service_credentials` | JSON information saved from IBM Cloud console. | Example: Create a Default Configuration ```shell ascli config preset update <COS_PRESET_NAME> --bucket=mybucket --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south ascli config preset set default cos <COS_PRESET_NAME> ``` ### Operations, transfers After creating a default configuration using one of the previous methods (otherwise, specify the access options directly on the command line), you can perform a subset of `node` plugin operations, which correspond to the Node API. ```shell ascli cos node info ascli cos node upload 'faux:///sample1G?1g' ``` > [!NOTE] > The file `sample1G` is a dummy file of size 1 GiB, generated using the `faux` PVCL scheme (see previous section and `man ascp`). > To upload a real file, replace the `faux:///...` URI with the actual file path. ### Tested commands for `cos` > [!NOTE] > Add `ascli cos` in front of the following commands: ```shell node download test_file.bin --to-folder=. node info --bucket=my_bucket --endpoint=my_endpoint --apikey=<API_KEY> --crn=<RESOURCE_INSTANCE_ID> node info --bucket=my_bucket --region=my_region --service-credentials=@json:@file:<COS_SVC_CRED_FILE> node info --log-level=trace2 node upload test_file.bin ``` ## Plugin: `httpgw`: HTTP Gateway ### Tested commands for `httpgw` > [!NOTE] > Add `ascli httpgw` in front of the following commands: ```shell health info ``` ## Plugin: `transferd`: Transfer Daemon Plugin `transferd` installs the IBM Aspera Transfer Daemon (`transferd`, which includes `ascp`) and runs it as a system service. See [Installing `ascp` through `transferd`](#installing-ascp-through-transferd) and [Transfer Daemon as a system service](#transfer-daemon-as-a-system-service). The plugin also has client commands for a running daemon (for example one installed with `transferd service install`): | Command | Action | |---------------------------------|---------------------------------------------------------------------| | `info` | Version of the API and information on the instance of the daemon. | | `transfers show <transfer_id>` | Status of a transfer: the daemon keeps it for some time after its end. | | `transfers cancel <transfer_id>`| Stop a transfer. | They use the options `url` (mandatory, endpoint of the daemon, e.g. `127.0.0.1:55002`), `api_key` (if the daemon enables `authentication`) and `ca_file` (if the daemon enables `tls`). If a key is not given, it is taken from the same key of the option `--transfer`, when its agent is `transferd` (e.g. a preset used for transfers). The options of the plugin have priority. The daemon has no call to list the transfers. These commands require the gem `grpc`: it is loaded only when one of them runs, so that the other commands do not need it. If it is missing: `gem install grpc`. ```bash ascli transferd info --url=127.0.0.1:55002 --api-key=@vault:transferd.api_key ``` > [!NOTE] > Commands `config transferd ...` and `config ascp install` are deprecated aliases of `transferd ...` and `transferd install`. ### Tested commands for `transferd` > [!NOTE] > Add `ascli transferd` in front of the following commands: ```shell install install --sdk-folder=sdk_test_dir install 1.1.3 list service configure --dry-run=yes @json:'{"service":{"user":"aspweb"},"config":{"port":55002}}' service spec ``` ## Plugin: `faspio`: Faspio Gateway IBM Aspera faspio Gateway is a high-performance proxy that bridges traditional TCP/UDP applications with the Aspera FASP protocol, enabling secure, ultra-fast transfers over any network, even with high latency or packet loss. It integrates seamlessly into existing workflows and supports use cases such as server-to-server transfers, database replication, and messaging systems. Using `ascli`, you can remotely create and manage bridges on faspio Gateway, simplifying configuration and automation. ### Tested commands for `faspio` > [!NOTE] > Add `ascli faspio` in front of the following commands: ```shell bridges create @json:'{"name":"test1","local":{"protocol":"tcp","tls_enabled":false,"port":"3000","bind_address":"127.0.0.1"},"forward":{"protocol":"fasp","tls_enabled":false,"port":"3994","bind_address":"127.0.0.1","host":["10.0.0.1"]}}' bridges delete --bulk=yes @json:<FASPIO_BCLEAN_LIST> bridges list bridges show 1 health ``` ## Plugin: `alee`: Aspera License Entitlement Engine Retrieve information on subscription. ### Tested commands for `alee` > [!NOTE] > Add `ascli alee` in front of the following commands: ```shell entitlement health ``` ## Plugin: `preview`: Preview generator for AoC The `preview` plugin is responsible for generating thumbnails (Office documents, images, videos) and video previews on storage, primarily for use within the Aspera on Cloud (AoC) application. This plugin leverages the **Node API** of Aspera HSTS and requires: - An Access Key - The associated **storage root** ### Key Features and Options You can configure several aspects of the preview generation process: - File Detection Methods Define how new files requiring previews are identified. - Video Preview Generation Methods Choose the approach for creating video previews (for example, transcoding options). - Video Handling Parameters Fine-tune video processing, such as resolution, bitrate, and format. Using `ascli` is an alternative to <https://github.com/IBM/aspera-on-cloud-file-previews>. ### Aspera Server configuration Specify the previews folder as shown in: <https://ibmaspera.com/help/admin/organization/installing_the_preview_maker> By default, the `preview` plugin expects previews to be generated in a folder named `previews` located in the storage root. On the transfer server execute: ```shell PATH=/opt/aspera/bin:$PATH asconfigurator -x "server;preview_dir,previews" asnodeadmin --reload ``` > [!NOTE] > The configuration `preview_dir` is **relative** to the storage root: no leading or trailing `/` is needed. > Set the value to `previews`. If another folder is configured on the HSTS, then specify it to `ascli` using the option `previews_folder`. The HSTS Node API limits any preview file to a parameter: `max_request_file_create_size_kb` (1 KB is 1024 Bytes). This size is internally capped to `1<<24` Bytes (16777216), that is, 16384 KB, that is, 16 MB. To change this parameter in `aspera.conf`, use `asconfigurator`. To display the value, use `asuserdata`: ```shell asuserdata -a | grep max_request_file_create_size_kb ``` ```text max_request_file_create_size_kb: "1024" ``` ```shell asconfigurator -x "server; max_request_file_create_size_kb,16384" ``` If you use a value different from `16777216`, then specify it using option `max_size`. > [!NOTE] > The HSTS parameter (`max_request_file_create_size_kb`) is in **kilo Bytes** while the generator parameter is in **Bytes** (factor of 1024). ### External tools: Linux `ascli` requires the following external tools available in the `PATH`: - **ImageMagick** v7+: `magick` (for tools: `convert` and `composite`) - **OptiPNG** : `optipng` - **FFmpeg** : `ffmpeg` `ffprobe` - **LibreOffice** : `unoconv` Installation is shown here for Red Hat/Rocky Linux. Other OSes should work as well, but are not tested. To check if all tools are found properly, execute: ```shell ascli preview check ``` #### Image: ImageMagick and `optipng` ```shell dnf install -y ImageMagick optipng ``` You may also install `ghostscript` which adds fonts to ImageMagick. Prefer ImageMagick version >=7. More info on ImageMagick at <https://imagemagick.org/> Available fonts, used to generate PNG for text, can be listed with: ```shell magick identify -list font ``` If your OS has only ImageMagick v6, then you can create a script called `magick` and add it to your `PATH`: ```shell #!/bin/bash exec "$@" ``` Make it executable: ```shell chmod a+x /usr/local/bin/magick ``` #### Video: FFmpeg The installation of `ffmpeg` depends on the distribution of Linux. On Rocky Linux, you can install `ffmpeg` with: ```shell dnf install ffmpeg-free ``` Or check which package provides it with: ```shell dnf provides ffmpeg ``` Another method is to download and install the latest released version of `ffmpeg` with static libraries from [https://johnvansickle.com/ffmpeg/](https://johnvansickle.com/ffmpeg/) ```shell curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && rm -f ffmpeg /usr/bin/{ffmpeg,ffprobe} && rm -fr ffmpeg-*-amd64-static && tar xJvf - && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin) ``` To uninstall above manual installation: ```shell rm -rf /opt/ffmpeg* /usr/bin/{ffmpeg,ffprobe} ``` #### Office: `unoconv` and LibreOffice To skip office document preview generation, use option: `--skip-types=office` The generation of previews is based on LibreOffice's `unoconv`. - RHEL 8/Rocky Linux 8+ ```shell dnf install unoconv ``` - Amazon Linux ```shell amazon-linux-extras enable libreoffice yum clean metadata yum install libreoffice-core libreoffice-calc libreoffice-opensymbol-fonts libreoffice-ure libreoffice-writer libreoffice-pyuno libreoffice-impress wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv mv unoconv /usr/bin chmod a+x /usr/bin/unoconv ``` ### Configuration The preview generator should be executed as a non-root user. When using object storage, any user can be used, but when using local storage it is usually better to use the user `xfer`, as uploaded files are under this identity: this ensures proper access rights. The following procedure uses `xfer` as the running user. Like any `ascli` commands, options can be passed on command line or using a configuration [Option Preset](#option-preset). The configuration file must be created by the same user that runs the generator, so that it is used at runtime. The `xfer` user has a special protected shell: `aspshell`, so to update the configuration and when changing identity, specify an alternate shell. For example: ```shell su -s /bin/bash - xfer ascli config preset update <PREVIEW_PRESET_NAME> --url=https://localhost:9092 --username=<ACCESS_KEY> --password=<SECRET> --skip-types=office --lock-port=12346 ascli config preset set default preview <PREVIEW_PRESET_NAME> ``` This example assumes that Office file generation is disabled. Remove `--skip-types=office` if needed. `lock_port` prevents concurrent execution of generation when using a scheduler. One can check if the access key is well configured using: ```shell ascli -P<PREVIEW_PRESET_NAME> node browse / ``` This shall list the contents of the storage root of the access key. ### Options for generated files When generating preview files, the following options can be modified on command line (or in a preset): | Field | Type | Description | |---------------------|--------------|----------------------------------------------------------------------------------| | `blend_fps` | `Integer` | MP4 video preview (`blend`): frames per second.<br/>Default: `15`. | | `blend_keyframes` | `Integer` | MP4 video preview (`blend`): number of key frames.<br/>Default: `30`. | | `blend_pauseframes` | `Integer` | MP4 video preview (`blend`): number of pause frames (repetitions of each key frame).<br/>Default: `3`. | | `blend_transframes` | `Integer` | MP4 video preview (`blend`): number of transition frames between key frames.<br/>Default: `5`. | | `clips_count` | `Integer` | MP4 video preview (`clips`): number of clips.<br/>Default: `5`. | | `clips_length` | `Integer` | Video: length (in seconds) of each clip of MP4 video preview (`clips`), and of `animated` PNG thumbnail.<br/>Default: `5`. | | `max_size` | `Integer` | Maximum size (in bytes) of a preview file: a warning is logged if exceeded.<br/>Default: `16777216`. | | `office_conversion` | `Enum[String]` | PNG thumbnail of office document: conversion tool.<br/>Allowed values: `soffice`, `unoconv`.<br/>Default: `soffice`. | | `reencode_ffmpeg` | `Hash` | MP4 video preview (`reencode`): `ffmpeg` options replacing the default ones.<br/>Default: `{}`. | | `reencode_ffmpeg.in` | `Array` | Input options. | | `reencode_ffmpeg.out` | `Array` | Output options. | | `thumb_img_size` | `Integer` | PNG thumbnail of image, PDF, office document or text: size (in pixels).<br/>Default: `800`. | | `thumb_text_font` | `String` | PNG thumbnail of text: font name, as listed by `magick identify -list font`.<br/>Default: `Courier`. | | `thumb_vid_fraction` | `Number` | PNG thumbnail of video (`fixed`): position of the snapshot, as a fraction of the video duration.<br/>Default: `0.1`. | | `thumb_vid_scale` | `String` | PNG thumbnail of video: frame size, as `ffmpeg` scale filter argument.<br/>Default: `-1:min(ih,100)`. | | `video_codec` | `String` | MP4 video preview (`reencode`): `ffmpeg` video codec, e.g. `libx264`, `h264_videotoolbox`, `h264_nvenc`. Default: first available H.264 encoder. | | `video_conversion` | `Enum[String]` | MP4 video preview: generation method. `reencode`: re-encode the beginning of the video, `blend`: key frames with transitions, `clips`: concatenation of short clips.<br/>Allowed values: `reencode`, `blend`, `clips`.<br/>Default: `reencode`. | | `video_png_conv` | `Enum[String]` | PNG thumbnail of video: generation method. `fixed`: single frame, `animated`: animated PNG.<br/>Allowed values: `fixed`, `animated`.<br/>Default: `fixed`. | | `video_scale` | `String` | MP4 video preview: frame size, as `ffmpeg` scale filter argument.<br/>Default: `min(iw,360):-2`. | | `video_start_sec` | `Integer` | Video: start offset (in seconds) of MP4 video preview and of `animated` PNG thumbnail.<br/>Default: `10`. | For video preview with method `reencode`, the whole set of `ffmpeg` options can be overridden with option `reencode_ffmpeg`: it is a `Hash` with two keys: `in` and `out`, each is an `Array` with the native options to `ffmpeg`. ### Execution `ascli` intentionally supports only a **one shot** mode (no infinite loop) in order to avoid having a hanging process or using too many resources (calling REST API too quickly during the scan or event method). It needs to be run on a regular basis to create or update preview files. For that use your best reliable scheduler, see [Scheduler](#scheduler). Typically, for **Access key** access, the system/transfer user is `xfer`. To be consistent and generate the appropriate access rights, run the generation process as user `xfer`. To run a one-shot test using the configuration previously created: ```shell su -s /bin/bash - xfer ``` Or: ```shell sudo -u xfer /bin/bash ``` Then: ```shell ascli preview scan --overwrite=always ``` When the preview generator is first executed, it creates a file `.aspera_access_key` in the previews folder, which contains the access key used. On subsequent runs, it reads this file and checks that previews are generated for the same access key, else it fails. This is to prevent clash of different access keys using the same root. ### Configuration for Execution in scheduler Details are provided in section [Scheduler](#scheduler). Shorter commands can be specified if a configuration preset was created as shown previously. For example, the timeout value can be differentiated depending on the command: event versus scan: ```shell case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac ``` ### Candidate detection for creation or update (or deletion) `ascli` generates preview files using those commands: - `trevents` : only recently uploaded files will be tested (transfer events) - `events` : only recently uploaded files will be tested (file events: not working) - `scan [path]` : generate previews for a file or recursively scan a folder. The default path is the access key's **storage root**. A file or folder can also be selected with `%id:<file_id>`. - `test` : test using a local file Use `--filter` with `scan`, `events`, or `trevents` to select files by name. It accepts a glob string, a regular expression using `@re:`, or a Proc using `@ruby:`. For example: ```shell ascli preview scan /videos --filter='*.mp4' ascli preview scan %id:<file_id> ascli preview scan /videos --filter='@ruby:->(f){f["name"].end_with?(".mp4")}' ``` Once candidates are selected, a preview is always generated if it does not already exist; if a preview already exists, generation depends on the value of option `overwrite`: - `always` : preview is always generated, even if it already exists and is newer than original - `never` : preview is generated only if it does not exist already - `mtime` : preview is generated only if the original file is newer than the existing Deletion of preview for deleted source files: not implemented yet (TODO). If the `scan` or `events` detection method is used, then option `skip_folders` can be used to skip some folders. It expects a list of paths relative to the storage root (docroot), starting with a slash, for example: ```shell ascli preview scan --skip-folders=@json:'["/not_here"]' ``` The option `folder_reset_cache` forces the node service to refresh folder contents using various methods. See the following section for details. ### Preview File types Two types of preview can be generated: - `png`: thumbnail - `mp4`: video preview (only for video) Use option `skip_format` to skip generation of a format. ### Supported input Files types The preview generator supports rendering of those file categories: - `image` - `pdf` - `plaintext` - `office` - `video` To avoid generation for some categories, specify a list using option `skip_types`, for example, `--skip-types=office` or `--skip-types=@list:,office,video`. Each category has a specific rendering method to produce the PNG thumbnail. The mp4 video preview file is only for category `video`. By default, the Mime type used for conversion is the one returned by the Node API, based on file name extension. It is also possible to detect the MIME type using option `detect_mime`. To use it, set option `detect_mime` to `yes`: `--detect-mime=yes`. In this case the `preview` command will first analyze the file content using gem `marcel`, and if no match, will try by extension. ### Generation: Read source files and write preview Thumbnails and video previews are generated using standard open source tools and stored together in the storage root under the folder specified by option `previews_folder`. Those tools require that original files are accessible in the local file system and also write generated files on the local file system. Nevertheless, `ascli` may or may not have direct file system access to the access key storage root. `ascli` provides 2 ways to read and write files with the option: `root_url`: - Using direct access to the local file system. - Using Aspera to transfer files between the storage and the `ascli` server. | `root_url` | Description | |---------------|-------------------------------------------------------------------------------| | `<empty>` | (Default) If the access key storage type is `local`, then the storage root is used as the main folder.<br/>This assumes that `ascli` runs on the same system as HSTS, or has access through a common "mount".<br/>Else, remote access is assumed. | | `aspera:` | Source files are **downloaded** to a temporary directory, and preview files are **uploaded** to the storage.<br/>Two transfers are performed using Aspera: one download transfer for source files, one upload transfer for preview files. | | `file:///<path>` | Files are accessed from the specified path locally. | ### Tested commands for `preview` > [!NOTE] > Add `ascli preview` in front of the following commands: ```shell check --skip-types=office events --once-only=yes --skip-types=office --log-level=info scan %id:1 --skip-types=office --log-level=info --root-url=aspera: --ts.target_rate_kbps=1000000 scan --skip-types=office --log-level=info --skip-folders=/special/folder show /etc/hosts --base=test show my_docx --base=test show my_mpg --base=test --video-png-conv=animated show my_mpg --base=test --video-png-conv=fixed show my_pdf --base=test show my_small_mp4 --base=test --video-png-conv=animated show my_small_mp4 --base=test --video-png-conv=fixed test my_dcm --base=test test my_dcm --base=test --detect-mime=yes test my_jpg_unk --base=test --detect-mime=yes test my_mpg mp4 --base=test --video-conversion=clips test my_mpg mp4 --base=test --video-conversion=reencode test my_mxf mp4 --base=test --video-conversion=blend --query.text=true --query.double=true trevents --once-only=yes --skip-types=office --log-level=info ``` ## Plugin: `mcp`: Model Context Protocol server > [!WARNING] > This plugin is **experimental**. > The interface, options, and behavior may change in future releases without notice. The `mcp` plugin starts a [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) server that exposes `ascli` to AI assistants and LLM-based tools. The server registers a single tool, `execute_ascli_command`, which executes any `ascli` command in-process and returns the result as text. > [!IMPORTANT] > The `mcp` and `rack` gems are required, see section [Installing Optional Gems](#installing-optional-gems). Install them with the following command: ```shell gem install mcp rack ``` ### Usage ```shell ascli mcp server ascli mcp server @: instructions="Use this server to manage Aspera transfers" ascli mcp server @: transport=http ``` ### `server` command options The `server` command accepts an optional [Hash](#extended-value-syntax) argument with the following keys. Keys of a transport are refused with the other transport. #### General | Key | Type | Default | Description | |---|---|---|---| | `transport` | `string` | `stdio` | Transport to use: `stdio` or `http` | | `description` | `string` | - | Short description of the server shown in the AI client UI | | `instructions` | `string` | - | Longer hint shown to the AI client describing what the server does and how to use it | | `protocol_version` | `string` | - | MCP protocol version to advertise, for example, `2024-11-05` | | `validate_tool_call_arguments` | `boolean` | `true` | Whether to validate tool arguments against the input schema | | `max_text_bytes` | `integer` | `100000` | Maximum byte size of the JSON text content for list results. Items are appended whole until the limit is reached; the full list is always available in `structuredContent` | #### `stdio` transport | Key | Type | Default | Description | |-----|-----|-----|-----| | `max_line_bytes` | `integer` | 4194304 (4 MiB) | Maximum JSON frame size in bytes | > [!NOTE] > **`stdio` transport and server description** - The MCP protocol does not carry a `description` field in the `initialize` handshake. > For `stdio` servers, AI clients (Claude Desktop, VS Code, Bob, and so on) cannot retrieve the description automatically. > Add a `"description"` field directly in the client's `mcpServers` configuration to display it in the UI. #### `http` transport (Streamable HTTP) | Key | Type | Default | Description | |-----|-----|-----|-----| | `url` | `string` | `http://127.0.0.1:3000` | Address and port the server listens on, without path, for example, `http://0.0.0.0:3000`. Use `https://` for TLS. | | `cert` | `string` | - | With `https://`: path to the TLS certificate file, PEM or PKCS12 (`.p12`, `.pfx`). Without `cert` and `key`, a self-signed certificate is generated. | | `key` | `string` | - | With `https://`: path to the PEM private key file, or the PKCS12 passphrase. | | `chain` | `string` | - | With `https://`: path to the PEM certificate chain file. | | `public_url` | `string` | from `url` | URL of the server for clients, used in OAuth metadata and in the URL of secret pages, for example, `https://your-server:3000`. Mandatory with `oauth` when the host of `url` is `0.0.0.0` or `::`. | | `auth_token` | `string` | - | Static Bearer token. If set, all MCP requests must carry `Authorization: Bearer <token>`, else HTTP 401. Cannot be empty. It is also the access code asked on secret pages, and on the consent page with `oauth`. Generate it with `openssl rand -hex 32`. | | `auth_token_min_bits` | `integer` | `80` | Minimum estimated strength of `auth_token`, in bits: a weak token is refused at startup. The estimate uses the optional gem `zxcvbn` if installed, else a built-in estimate. `0` disables the check. | | `oauth` | `boolean` | `false` | Enable OAuth 2.0 PKCE flow with dynamic client registration. When `true`, clients register automatically and users authenticate via a browser consent page. | | `oauth_access_token_ttl` | `integer` | `3600` | OAuth access token lifetime in seconds. | | `oauth_refresh_token_ttl` | `integer` | `2592000` | OAuth refresh token lifetime in seconds (30 days). | | `stateless` | `boolean` | `false` | Enable stateless mode | | `allowed_origins` | `Array<string>` | - | List of allowed origins | | `allowed_hosts` | `Array<string>` | - | List of allowed hostnames | | `session_idle_timeout` | `integer` | - | Idle session timeout in seconds | | `max_sessions` | `integer` | - | Maximum number of concurrent sessions | > [!NOTE] > `port` and `bind` are deprecated: use `url`, for example, `url=http://0.0.0.0:8080` instead of `port=8080 bind=0.0.0.0`. For the addresses the server listens on, and HTTPS, see [Web service](#web-service). ### Security > [!WARNING] > The tool executes any `ascli` command with the credentials and presets of the user running the server. > When the host of `url` or `public_url` is not a loopback address, `auth_token` is **mandatory**, also with `oauth`: the server refuses to start without it. With the `http` transport: - With `oauth`, the consent page asks for `auth_token` as access code, else anyone reaching the server could approve its own client. It shows where the authorization is sent (origin of `redirect_uri`): allow only if it is the expected AI client, for example `http://127.0.0.1:<port>`. - For local-only deployments (hosts of `url` and `public_url` are loopback, see [Web service](#web-service)), `auth_token` and `oauth` are optional but recommended: on a host shared with other users, they can also reach the server. Without `auth_token`, the consent page only asks to click **Allow**. - `auth_token` and `oauth` can coexist: a request is accepted if it carries a valid static token **or** a valid OAuth access token. - A weak `auth_token` is refused at startup (see `auth_token_min_bits`): generate it with `openssl rand -hex 32`. - On a network, use `https://` in `url` (or a reverse proxy with TLS): with `http://`, the token is sent in clear. - `allowed_hosts` and `allowed_origins` protect against DNS rebinding but do not authenticate clients. They apply to all endpoints, also OAuth and secret pages, which also accept the origin of `public_url` (for a reverse proxy that changes header `Host`). With both transports: - Extended value `@stdin:` is refused. - `@secret:` is entered by the user on a web page, see [Secrets entered in a web page](#secrets-entered-in-a-web-page). - Secrets are hidden in responses, also a value shorter than 8 characters as a word in a text (for example, a short password in an error message). - The [Security Level](#security-level) is `high` by default: `@ruby:`, `@file:` (also in the parameters of stores of option `vault`), `@uri:`, and `@env:` are refused, and a secret must be the whole value: hidden in results, it cannot be processed by another extended value. It applies to commands sent by the client, and to presets, because the client can modify presets. Commands sent by the client cannot change option `security`. Features that execute code, such as option `notify_template` or command `config plugins create`, plugins, and writing to a file chosen by the client (option `out.file`), are refused, see [Security Level](#security-level). Files can be written (for example by `config download`, or by a transfer that receives files) only in `writable_paths`, none by default: without it, the client cannot download files. It applies to files written on the system that runs the server: not to a transfer with agent `node` (`--transfer=node`), which writes on the node. To use another level, set option `security` when starting the server, for example: ```shell ascli mcp server --security.level=medium --security.allowed_paths.0=$HOME/.aspera/ascli/my_private_key ``` To let the client download files in a folder, keep the level `high` and give the folder: ```shell ascli mcp server --security.writable_paths.0=$HOME/Downloads/ascli ``` > [!NOTE] > A level set by option also applies to the parameters of the `server` command: for example, `auth_token=@env:MCP_TOKEN` requires level `low`. > Without option `security`, the parameters of the `server` command are not restricted. > > A preset created by `aoc wizard` or `faspex5 wizard` reads the private key with `@file:`. > With level `high`, either store the key in the preset (`ascli config preset set <AOC_ORG> private_key @file:<PATH>`, without `@val:`), or use level `medium` with the key in `allowed_paths`. > > In the same way, a secret of a vault read from a file (for example, `token` of HashiCorp Vault: `@file:<path>`) requires level `medium` with the file in `allowed_paths`. ### HTTP endpoints With the `http` transport, the server exposes the following endpoints: | Endpoint | Description | |---|---| | `GET /` | Discovery: JSON object with `name`, `version`, and `description` of the server, no authentication | | `POST`, `GET`, `DELETE` `/mcp` | MCP endpoint (Streamable HTTP), the URL to configure in the AI client. Requires `auth_token` or an OAuth access token, if configured | | `GET`, `POST` `/secret/<id>` | Secret pages, see [Secrets entered in a web page](#secrets-entered-in-a-web-page) | | `GET /.well-known/oauth-authorization-server` | With `oauth`: OAuth discovery metadata (RFC 8414) | | `POST /oauth/register` | With `oauth`: Dynamic client registration (RFC 7591) — any client may register and receive a `client_id`. At most 100 clients are kept: beyond, the least recently used is removed. At most 10 `redirect_uris`, absolute and without fragment | | `GET /oauth/authorize` | With `oauth`: Consent page shown to the user in a browser, with the `client_id` and where the code is sent (origin of `redirect_uri`) | | `POST /oauth/authorize` | With `oauth`: Processes the user decision (allow/deny) and issues an authorization code | | `POST /oauth/token` | With `oauth`: Exchanges the code for tokens (`authorization_code`) or renews them (`refresh_token`) | AI clients that support the discovery endpoint (such as Bob) read the description from it and display it in their MCP server list without any manual configuration. AI clients that support the MCP Authorization spec (such as Bob) detect the OAuth endpoints automatically via the `WWW-Authenticate` header returned on HTTP 401, register dynamically, and launch the browser-based login flow without any extra configuration. ### Secrets entered in a web page [Extended Value](#extended-value-syntax) `@secret:[name]` asks for a secret, for example, a password. In MCP server mode, there is no terminal: the user enters the secret on a web page of the server instead, following the [URL mode elicitation](https://modelcontextprotocol.io/specification/2026-07-28/client/elicitation) of the MCP protocol. The secret goes from the browser to the MCP server only: it is not sent to the MCP client or the AI model, and it is hidden in the results of the command. `@secret:` can be used in commands sent by the AI client, for example, `--password=@secret:password`, and in presets of the user running the server. For a command that needs a secret: 1. The server replies to the tool call with the URL of a secret page. 1. The MCP client shows the URL and asks the user for consent to open it in a browser. 1. The user enters the secret on the page and submits it, or cancels. 1. The MCP client calls the tool again, and the server executes the command with the secret. If the command needs several secrets, these steps are repeated for each secret. The secret page is served: - With the `http` transport: by the MCP server, at `<public_url>/secret/<id>`. If `auth_token` is set, the page also asks for it as access code. Set `public_url` if the browser does not reach the server on the address of `url`. - With the `stdio` transport: by a local web server on `http://127.0.0.1` with a random port, started on first use. A secret request expires after 10 minutes. Secrets are kept in memory only, until the command completes. > [!NOTE] > The MCP client must support URL mode elicitation (capability `elicitation.url`), else the command fails with an error. > Protocol `2026-07-28` (result `input_required`) and `2025-11-25` (request `elicitation/create` from the server) are supported. > With protocol `2025-11-25`, the `http` transport must not be `stateless`. ### Examples Start an MCP server over `stdio` (default - suitable for use as an MCP server in an IDE or AI tool): ```shell ascli mcp server ``` Start an MCP server over HTTP on port 8080: ```shell ascli mcp server @: transport=http url=http://0.0.0.0:8080 auth_token=@env:MCP_TOKEN ``` Start an MCP server over HTTPS, with a certificate: ```shell ascli mcp server @: transport=http url=https://0.0.0.0:8443 cert=<PATH>/cert.pem key=<PATH>/key.pem auth_token=@env:MCP_TOKEN ``` Start with custom instructions and a specific protocol version: ```shell ascli mcp server @: instructions="Aspera transfer automation" protocol_version=2024-11-05 ``` Start an MCP server with OAuth 2.0 PKCE authentication (for AI clients such as Bob that support the MCP Authorization spec): ```shell ascli mcp server @: transport=http url=http://0.0.0.0:3000 public_url=http://your-server:3000 oauth=true auth_token=@env:MCP_TOKEN ``` ### MCP client configuration #### Streamable HTTP transport ##### Local server (no authentication required) Start the server in a separate terminal first: ```shell ascli mcp server @: transport=http ``` Then register it in the AI client's MCP configuration file. For [**IBM Bob**](https://bob.ibm.com/docs/ide/configuration/mcp/mcp-in-bob), add to `.bob/mcp.json` in your project (or `~/.bob/mcp.json` for global configuration): ```json { "mcpServers": { "ascli": { "type": "streamable-http", "url": "http://localhost:3000/mcp" } } } ``` For other clients (Claude Desktop, VS Code, and so on): ```json { "mcpServers": { "ascli": { "type": "http", "url": "http://localhost:3000/mcp" } } } ``` ##### Remote server with static token When the server is exposed on a non-local address, `auth_token` is required. **Option A: static Bearer token** - suitable for all MCP clients. Set the token in the environment and start the server: ```shell export MCP_TOKEN=$(openssl rand -hex 32) ascli mcp server @: transport=http url=http://0.0.0.0:3000 auth_token=@env:MCP_TOKEN ``` For [**IBM Bob**](https://bob.ibm.com/docs/ide/configuration/mcp/mcp-in-bob), pass the same token in the `Authorization` header: ```json { "mcpServers": { "ascli": { "type": "streamable-http", "url": "http://your-server:3000/mcp", "headers": { "Authorization": "Bearer <TOKEN>" } } } } ``` For other clients (Claude Desktop, VS Code, and so on): ```json { "mcpServers": { "ascli": { "type": "http", "url": "http://your-server:3000/mcp", "headers": { "Authorization": "Bearer <TOKEN>" } } } } ``` ##### Remote server with OAuth 2.0 PKCE **Option B: OAuth 2.0 PKCE flow** - for AI clients that support the MCP Authorization spec (such as Bob). No token to configure in the client: the client registers dynamically and opens a browser window for user consent on first connection. The consent page asks for the access code: the value of `auth_token`. Start the server with OAuth enabled: ```shell ascli mcp server @: transport=http url=http://0.0.0.0:3000 public_url=http://your-server:3000 oauth=true auth_token=@env:MCP_TOKEN ``` For [**IBM Bob**](https://bob.ibm.com/docs/ide/configuration/mcp/mcp-in-bob), register the URL with no extra configuration: ```json { "mcpServers": { "ascli": { "type": "streamable-http", "url": "http://your-server:3000/mcp" } } } ``` Bob detects the OAuth endpoints via the `WWW-Authenticate` header on the first HTTP 401 response and triggers the browser-based login flow automatically. > [!NOTE] > With `http` transport, the client connects to a running server process. > The server must be started and kept running independently before the AI client tries to connect. #### `stdio` transport The client launches `ascli` directly as a subprocess. Add the following to the AI client's MCP configuration file. ##### IBM Bob `stdio` For [**IBM Bob**](https://bob.ibm.com/docs/ide/configuration/mcp/mcp-in-bob), add to `.bob/mcp.json` in your project (or `~/.bob/mcp.json` for global configuration): ```json { "mcpServers": { "ascli": { "type": "stdio", "command": "/path/to/bin/ascli", "args": ["mcp", "server"], "description": "Aspera CLI MCP server" } } } ``` For other clients (Claude Desktop, VS Code, and so on), the configuration is identical. ##### Claude Desktop `stdio` Find the configuration file as specified in [Claude Desktop Documentation](https://modelcontextprotocol.io/docs/2026-07-28/develop/connect-local-servers). Place this section in `mcpServers`: ```json { "mcpServers": { "ascli": { "command": "ascli", "args": ["mcp", "server"] } }, ... } ``` ##### Development mode for `stdio` If the `ascli` gem is not installed and you are running directly from the source tree, Ruby will not find the `lib/` directory automatically. Add the `RUBYLIB` environment variable pointing to the `lib/` directory of the project: ```json { "mcpServers": { "ascli": { "type": "stdio", "command": "/path/to/aspera-cli/bin/ascli", "args": ["mcp", "server"], "env": { "RUBYLIB": "/path/to/aspera-cli/lib" } } } } ``` ### Tested commands for `mcp` > [!NOTE] > Add `ascli mcp` in front of the following commands: ```shell server server @: transport=http port=13777 ``` ### End-to-end AI testing The file [`docs/test-mcp-with-ai.md`](test-mcp-with-ai.md) contains a step-by-step test procedure for validating the MCP server with a live AI assistant (Bob, Claude Desktop, VS Code Copilot, or any MCP-capable client). It covers tool visibility, self-discovery, Hash schema introspection, option listing, documentation retrieval, live transfer tasks, truncation handling, error handling, and credential safety. ### MCP usage patterns for AI > [!TIP] > This section is for AI assistants using the MCP server. > Follow these patterns to avoid common first-call failures and unnecessary retries. #### Session bootstrap Always start a session with two discovery calls before doing anything else: ```json ["config", "preset", "list"] ["config", "preset", "show", "default"] ``` The second call returns the mapping from `plugin` to `preset_name` so you know which credentials are active for each plugin. #### Command discovery Always use `["config", "commands", "<plugin>"]` to enumerate the commands of a plugin and their syntax. Add command words to list only the commands under that path, e.g. `["config", "commands", "aoc", "files"]`. A line ending with `<command...>` provides the commands of another plugin, given by `(see: ...)`. Use option `--expand-mounts=yes` to list them in place. Omit `<plugin>` to list the commands of all plugins (much larger result). Never guess command names from training data: names like `shared_folders` vs `shared_inboxes` are easily confused. #### Schema introspection for Hash arguments Whenever a command syntax shows a Hash argument (e.g. `<account:Hash>`), call `help` in its place **before** the real call: ```json ["<plugin>", "<cmd>", ..., "help"] ``` This returns a table of field names, types, and descriptions. Never infer field names from server error messages. #### Secrets To use a secret the user did not provide (password, passphrase), pass `@secret:<name>` as value. The user enters it on a web page and you never see it: prefer it to asking for the secret in the conversation. ```json ["server", "browse", "/", "--url=ssh://host:33001", "--username=user", "--password=@secret:password"] ``` #### Async transfers and cross-call status tracking The `direct` agent keeps transfer state in-memory. A job started in one MCP call **cannot** be monitored in a subsequent call: the in-memory agent is gone between calls. Use the `transferd` agent when you need to check transfer status in a later call: ```json ["server", "upload", "--transfer=transferd", "--to-folder=/dst", "/local/file"] ``` The `desktop` agent is also unaffected because it runs in an external process. #### Workspace context for AoC `aoc files` and `aoc packages` commands require a workspace context. If no default workspace is configured in the preset, always add `--workspace=NAME`: ```json ["aoc", "files", "ls", "/", "--workspace=MyWorkspace"] ``` List available workspaces with `["aoc", "user", "workspaces", "list"]`. #### Admin operations and privilege checks Before calling any `admin` sub-command, verify that the active preset has admin rights. `access_denied` typically means the wrong preset is active, not a syntax error. Check with: ```json ["config", "preset", "show", "<preset_name>"] ``` ## Operational Utilities This section covers the specialized modules and utilities used to integrate `ascli` into your broader operational infrastructure. While the core plugins handle data movement, these tools provide the "integration layer" for enterprise environments: Aspera Sync and Hot Folder enable automated, folder-based synchronization; Nagios and SMTP modules provide health monitoring and automated email alerting for transfer status; and the `asession` tool and the Ruby module `Aspera` allow integration of transfers into other programs. Together, these features transform the CLI from a manual tool into a fully integrated component of an automated, monitored data workflow. ### IBM Aspera Sync An interface for the `async` utility is provided in the following plugins: - `server sync` (SSH Auth) - `node sync` (use gen3 token) - `aoc files sync` (uses `node` plugin with bearer token) - `shares files sync` (uses `node` plugin with gen3 token) The `sync` command, available in above plugins, performs the following actions: - Start a local Sync session by executing the `async` command with the appropriate parameters. - Get local Sync session information by accessing the Async snap database directly. - Get local Sync session information using the `asyncadmin` command, if available. One advantage of using `ascli` over the `async` command line is the possibility to use a configuration file, using standard options of `ascli`. Moreover, `ascli` supports sync with applications requiring token-based authorization. Some `sync` parameters are filled by the related plugin using transfer spec parameters (for example, including token). > [!NOTE] > All `sync` commands require an `async` enabled license and availability of the `async` executable (and `asyncadmin`). > The Aspera Transfer Daemon 1.3+ includes this. #### Quick test Below is a simple end-to-end procedure to verify synchronization using the demo server. 1. Initialize the demo server configuration: ```shell ascli config initdemo ``` 1. Create a local folder with a sample file to sync: ```shell mkdir foobar echo hello > foobar/file1 ``` 1. Create the destination folder on the remote server: ```shell ascli server -Pdemoserver mkdir /Upload/mydest1 ``` 1. Run the sync operation: ```shell ascli server -Pdemoserver sync push foobar --to-folder=/Upload/mydest1 ``` The following sections provide additional details on available options and configuration. #### Starting a sync session To start a sync session, use one of the three sync directions, followed by a folder path. For `push` and `bidi`, the path is a local folder. For `pull`, the path is a remote folder. The corresponding path on the opposite side is provided using the `to_folder` option. **General syntax:** ```shell ascli ... sync <DIRECTION> <PATH> [--to-folder=<PATH>] [<SYNC_INFO>] ``` | <DIRECTION><br/>(parameter) | <PATH><br/>(parameter) | `to_folder`<br/>(option) | |-----------|--------|-------------| | `push` | Local | Remote | | `bidi` | Local | Remote | | `pull` | Remote | Local | An optional positional `Hash` argument (`<SYNC_INFO>`) may be provided. It can be expressed in one of two formats: - `conf` format (recommended) - `args` format (legacy) A single sync session must use **one format exclusively**. - If `<SYNC_INFO>` is not provided, a default configuration is automatically generated in the `conf` format (details in the next section). - If argument `<SYNC_INFO>` is provided, the format is inferred: - If the `Hash` contains either key `sessions` or `instance`, the `args` format is used. - Otherwise, the `conf` format is used. ##### `sync_info`: `conf` format This is the **preferred** syntax. It is the same payload as specified on the `async` option `--conf` or in Node API `/asyncs`. Documentation on Async Node API can be found on [IBM Developer Portal](https://developer.ibm.com/apis/catalog?search=%22aspera%20sync%20api%22). The following parameters are automatically filled from mandatory arguments, and are not allowed: - `direction` - `local.path` - `remote.path` Parameter `name` is set to a default value if not provided in `sync_info`. Parameter `quiet` is set to `false` if not provided in `sync_info` and a terminal is detected. The documentation is available in the terminal with: ```shell ascli config sync spec ``` > [!NOTE] > `ascli` accepts the following fields within the `sync_info` Hash. > The options listed in the **Description** column correspond to the equivalent parameters used by the low-level `async` command. | Field | Type | Description | |------------------------------------------|---------------|----------------------------------------------------------------------------------| | `ascp_dir` | `String` | Directory containing ascp executable to use. | | `assume_no_mods` | `Bool` | Assume that the directory structure has not been modified.<br/>Default: `false`.<br/>(`--assume-no-mods`) | | `checksum` | `Enum[String]` | Use the specified checksum type. Default is none on cloud storage.<br/>Allowed values: `sha1`, `md5`, `sha1_sparse`, `md5_sparse`, `none`.<br/>Default: `sha1_sparse`.<br/>(`--checksum={enum}`)(-k) | | `clean_excluded` | `Bool` | Removes any existing entries in the snapshot database for excluded paths<br/>Default: `false`.<br/>(`--clean-excluded`) | | `cookie` | `String` | User-defined identification string.<br/>(`--cookie={string}`) | | `cooloff_max_seconds` | `Integer` | Wait up to the specified time for a file to stop changing before skipping synchronization of the file. 0 for disabled<br/>Default: `0`.<br/>(`--cooloff-max={integer}`) | | `cooloff_seconds` | `Integer` | Delay the start of the transfer to confirm that the content is not changing. Value must be between 0 and 60<br/>Default: `3`.<br/>(`--cooloff={integer}`) | | `create_dir` | `Bool` | Create the source directory, target directory, or both, if they do not exist.<br/>Default: `false`.<br/>(`--create-dir`) | | `db_cache_size` | `Integer` | Specify DB cache size.<br/>Default: `16000`. | | `db_journal_off` | `Bool` | Turn off DB journal.<br/>Default: `false`. | | `db_sync_on` | `Bool` | Enable synchronous write in DB.<br/>Default: `false`. | | `dedup` | `Enum[String]` | Take the specified action when async detects duplicate files on the source.<br/>Allowed values: `copy`, `inode`, `hardlink`, `none`.<br/>Default: `none`.<br/>(`--dedup={enum}`) | | `delete_before` | `Bool` | Schedule deletes before transfers.<br/>Default: `false`.<br/>(`--delete-before`) | | `delete_delay` | `Bool` | Delay actual deletes until the end of the synchronization.<br/>Default: `false`.<br/>(`--delete-delay`) | | `direction` | `Enum[String]` | The direction of replication relative to the local.<br/>Allowed values: `bidi`, `pull`, `push`.<br/>Default: `push`.<br/>(`--direction={enum}`)(-K) | | `exclude_dirs_older_than` | `Hash` | Don't scan directories with a recursive modified time older than absolute or async start time - relative_seconds | | `exclude_dirs_older_than.absolute` | `String` | UTC timestamp. Empty value for disabled. | | `exclude_dirs_older_than.relative_seconds` | `Integer` | Relative to async start time. `-1` for disabled.<br/>Default: `-1`. | | `filters` | `Array[Hash]` | The filters allow to further specify which files have to be excluded and included from the transfer list. Each filter is defined by a rule and a value. Order of filters matters | | `filters[].rule` | `Enum[String]` | The rule for the filter.<br/>Required.<br/>Allowed values: `include`, `exclude`, `include_from`, `exclude_from`. | | `filters[].value` | `String` | On include or exclude, the filter's pattern. On include_from or exclude_from, the path containing filter specifications<br/>Required. | | `ignore_delete` | `Bool` | Do not copy removals to the peer.<br/>Default: `false`.<br/>(`--ignore-delete`) | | `ignore_mode` | `Bool` | Source files that have had their mode changed after the initial transfer will not update the destination file mode.<br/>Default: `false`.<br/>(`--ignore-mode`) | | `ignore_remote_host_sync_name` | `Bool` | Do not check that the remote host being used for the current transfer matches the host used when the local database was created<br/>Default: `false`. | | `local` | `Hash` | Required. | | `local.pass` | `String` | Authenticate the local async with the specified password. | | `local.path` | `String` | The directory to be synchronized on the local host.<br/>Required.<br/>(`--local-dir={string}`)(-d) | | `local_apply_docroot` | `Bool` | Prepend the docroot to the directory on the local host.<br/>Default: `false`.<br/>(`--apply-local-docroot`) | | `local_checksum_threads` | `Integer` | Maximum number of threads to do checksum on the local host. Value must be between 1 and 99.<br/>Default: `4`.<br/>(`--local-checksum-threads={integer}`) | | `local_db_dir` | `String` | Use the specified database directory on the local host. Default is `.private-asp` at the root level of the synchronized directory.<br/>(`--local-db-dir={string}`)(-b) | | `local_db_store_dir` | `String` | Store/Restore the database to/from the specified directory on the local host. The value can be an absolute path, an URI or - (use the local sync dir)<br/>(`--local-db-store-dir={string}`) | | `local_force_stat` | `Bool` | Forces the local async to retrieve file information even when no changes are detected by the scanner or monitor.<br/>Default: `false`.<br/>(`--local-force-stat`) | | `local_fs_threads` | `Integer` | Maximum number of threads to do file system operations on the local host. Value must be between 1 and 99.<br/>Default: `1`.<br/>(`--local-fs-threads={integer}`) | | `local_keep_dir` | `String` | Move deleted files into the specified directory on the local host.<br/>(`--keep-dir-local={string}`) | | `local_mount_signature` | `String` | Verify that the file system is mounted by the existence of this file on the local host.<br/>(`--local-mount-signature={string}`) | | `local_move_cache_timeout_seconds` | `Integer` | Delay in seconds before aborting moving a file from local cache to final destination. `-1` for disabled.<br/>Default: `-1`.<br/>(`--local-move-cache-timeout={integer}`) | | `local_preserve_acls` | `Enum[String]` | Preserve access control lists on the local host.<br/>Allowed values: `native`, `metafile`, `none`.<br/>Default: `none`.<br/>(`--preserve-acls={enum}`) | | `local_preserve_xattrs` | `Enum[String]` | Preserve extended attributes on the local.<br/>Allowed values: `native`, `metafile`, `none`.<br/>Default: `none`.<br/>(`--preserve-xattrs={enum}`) | | `local_scan_interval_milliseconds` | `Integer` | Enable periodic scans on the local host during a continuous sync. `-1` for disabled. On the command line, the value is a duration, e.g. `1h 30m`, `2s500ms`, `5:30`, `1d 02:00:00.250`. A plain number is in seconds. Stored in milliseconds.<br/>Default: `-1`.<br/>(`--scan-interval={integer}`) | | `local_scan_threads` | `Integer` | Number of directory scanning threads on the local host. Value must be between 1 and 99<br/>Default: `1`.<br/>(`--scan-threads={integer}`) | | `local_stat_cache_size` | `Integer` | Set stat cache size on the local host. 0 for disabled.<br/>Default: `0`. | | `log` | `Hash` | | | `log.level` | `Enum[String]` | Use the specified log level.<br/>Allowed values: `log`, `dbg1`, `dbg2`.<br/>Default: `log`.<br/>(special:`-D`) | | `log.local_dir` | `String` | Use the specified logging directory on the local host.<br/>(`--alt-logdir={string}`)(-L) | | `log.remote_dir` | `String` | Use the specified logging directory on the remote host.<br/>(`--remote-logdir={string}`)(-R) | | `manifest_path` | `String` | A directory path where ascp will create manifest TEXT files (passed to ascp as --file-manifest-path) | | `mirror` | `Bool` | Force the pulling side to be exactly like the pushing side, removing files on the destination that don't exist on the source and resending source files that don't have an exact match on the destination. Cannot be used in bi-directional mode.<br/>Default: `false`.<br/>(`--mirror`) | | `mode` | `Enum[String]` | Specify whether async runs continuously or not. In `one_time` mode, async stops after the first full synchronization. `continuous` supported only if the source is Windows or Linux.<br/>Allowed values: `one_time`, `continuous`.<br/>Default: `one_time`.<br/>(special:`--continuous`)(-C) | | `monitor_buffer_size` | `Integer` | Bytes to allocate for the change monitor buffer. Applies to any Windows machine on either side. `-1` to use the computed value.<br/>Default: `-1`. | | `name` | `String` | Name of the synchronization pair.<br/>Required.<br/>(`--name={string}`)(-N) | | `no_log` | `String` | Suppress log messages for ITEM. The only currently supported ITEM is 'stats', which suppresses both STATS and PROG log messages.<br/>(`--no-log={string}`) | | `no_preserve_root_attrs` | `Bool` | Disable the preservation of attributes on the Sync root.<br/>Default: `false`.<br/>(`--no-preserve-root-attrs`) | | `no_scan` | `Bool` | Skip initial scanning.<br/>Default: `false`.<br/>(`--no-scan`) | | `notifications_sharing_retry_max` | `Integer` | Retry processing filesystem notifications up to the specified maximum number after a sharing violation.<br/>Default: `3`. | | `overwrite` | `Enum[String]` | Overwrite files according to the specified policy. Default is determined by the direction: `conflict` for `bidi`, otherwise `always`.<br/>Allowed values: `always`, `older`, `conflict`.<br/>(`--overwrite={enum}`)(-o) | | `pending_max` | `Integer` | Allow the maximum number of files that are pending transfer to be no more than the specified number.<br/>Default: `2000`.<br/>(`--pending-max={integer}`) | | `preserve_access_time` | `Bool` | Preserve file access time from the source to the destination.<br/>Default: `false`.<br/>(`--preserve-access-time`) | | `preserve_creation_time` | `Bool` | Preserve file creation time from the source to the destination.<br/>Default: `false`.<br/>(`--preserve-creation-time`) | | `preserve_gid` | `Bool` | Preserve the file owner's GID.<br/>Default: `false`.<br/>(`--preserve-gid`)(-j) | | `preserve_modification_time` | `Bool` | Preserve file modification time from the source to the destination.<br/>Default: `false`.<br/>(`--preserve-modification-time`) | | `preserve_object_lock_legal_hold` | `Bool` | Preserve object lock legal hold status from the source to the destination.<br/>Default: `false`.<br/>(`--preserve-object-lock-legal-hold`) | | `preserve_object_lock_retention` | `Bool` | Preserve object lock retention from the source to the destination.<br/>Default: `false`.<br/>(`--preserve-object-lock-retention`) | | `preserve_object_metadata` | `Bool` | Preserve object metadata from the source to the destination.<br/>Default: `false`.<br/>(`--preserve-object-metadata`) | | `preserve_uid` | `Bool` | Preserve the file owner's UID.<br/>Default: `false`.<br/>(`--preserve-uid`)(-u) | | `quiet` | `Bool` | Disable progress display.<br/>Default: `true`.<br/>(`--quiet`)(-q) | | `remote` | `Hash` | Required. | | `remote.connect_mode` | `Enum[String]` | Define how to connect to the remote.<br/>Allowed values: `ssh`, `ws`.<br/>Default: `ssh`.<br/>(special:`--ws-connect`) | | `remote.fingerprint` | `String` | Check it against server SSH host key fingerprint. | | `remote.host` | `String` | Use the specified host name or address of the remote host.<br/>Required.<br/>(`--host={string}`) | | `remote.pass` | `String` | Authenticate the transfer with the specified password.<br/>(`--pass={string}`)(-w) | | `remote.path` | `String` | Synchronize the specified directory on the remote host.<br/>Required.<br/>(`--remote-dir={string}`)(-r) | | `remote.port` | `Integer` | Use the specified TCP port for SSH. Used when connect_mode is `ssh`<br/>Default: `22`.<br/>(`--tcp-port={integer}`)(-P) | | `remote.private_key_paths` | `Array[String]` | Authenticate with the specified SSH private key file.<br/>(`--private-key-path={array}`)(-i) | | `remote.proxy` | `Hash` | Specify the address of the Aspera high-speed proxy server.<br/>(special:`--proxy={object}`) | | `remote.proxy.host` | `String` | Use the specified host name or address of the proxy. | | `remote.proxy.pass` | `String` | Authenticate to the proxy with the specified password. | | `remote.proxy.port` | `Integer` | Use the specified port. Default is 9091 for dnat, 9092 for dnats. | | `remote.proxy.protocol` | `Enum[String]` | The protocol to be used.<br/>Allowed values: `none`, `dnat`, `dnats`.<br/>Default: `none`. | | `remote.proxy.user` | `String` | Authenticate to the proxy with the specified username. | | `remote.token` | `String` | Token string passed to server's authentication service. | | `remote.token_node_user` | `String` | Node API user identity associated with the token. Required for node user bearer tokens | | `remote.user` | `String` | Authenticate the transfer with the specified username.<br/>Required.<br/>(`--user={string}`) | | `remote.ws_port` | `Integer` | Use the specified port for Websocket. Used when connect_mode is `ws`.<br/>Default: `9093`. | | `remote_checksum_threads` | `Integer` | Maximum number of threads to do checksum on the remote host. Value must be between 1 and 99<br/>Default: `4`.<br/>(`--remote-checksum-threads={integer}`) | | `remote_db_dir` | `String` | Use the specified database directory on the remote host. Default is `.private-asp` at the root level of the synchronized directory.<br/>(`--remote-db-dir={string}`)(-B) | | `remote_db_store_dir` | `String` | Store/Restore the database to/from the specified directory on the remote host. The value can be an absolute path, an URI or - (use the remote sync dir).<br/>(`--remote-db-store-dir={string}`) | | `remote_force_stat` | `Bool` | Forces the remote async to retrieve file information even when no changes are detected by the scanner or monitor.<br/>Default: `false`.<br/>(`--remote-force-stat`) | | `remote_fs_threads` | `Integer` | Maximum number of threads to do file system operations on the remote host. Value must be between 1 and 99.<br/>Default: `1`.<br/>(`--remote-fs-threads={integer}`) | | `remote_keep_dir` | `String` | Move deleted files into the specified directory on the remote host.<br/>(`--keep-dir-remote={string}`) | | `remote_mount_signature` | `String` | Verify that the file system is mounted by the existence of this file on the remote host.<br/>(`--remote-mount-signature={string}`) | | `remote_move_cache_timeout_seconds` | `Integer` | Delay in seconds before aborting moving a file from remote cache to final destination. `-1` for disabled.<br/>Default: `-1`.<br/>(`--remote-move-cache-timeout={integer}`) | | `remote_preserve_acls` | `Enum[String]` | Preserve access control lists on the remote host. If not specified, the default behavior is to use the same storage mode as specified by `preserve_acls`.<br/>Allowed values: `native`, `metafile`, `none`.<br/>(`--remote-preserve-acls={enum}`) | | `remote_preserve_xattrs` | `Enum[String]` | Preserve extended attributes on the remote host. If not specified, the default behavior is to use the same storage mode as specified by `preserve_xattrs`.<br/>Allowed values: `native`, `metafile`, `none`.<br/>(`--remote-preserve-xattrs={enum}`) | | `remote_scan_interval_milliseconds` | `Integer` | Enable periodic scans on the remote host. `-1` for disabled. On the command line, the value is a duration, e.g. `1h 30m`, `2s500ms`, `5:30`, `1d 02:00:00.250`. A plain number is in seconds. Stored in milliseconds.<br/>Default: `-1`.<br/>(special:`--remote-scan-interval={integer}`) | | `remote_scan_threads` | `Integer` | Number of directory scanning threads on the remote host. Value must be between 1 and 99.<br/>Default: `1`.<br/>(`--remote-scan-threads={integer}`) | | `remote_stat_cache_size` | `Integer` | Set stat cache size on the remote host. 0 for disabled.<br/>Default: `0`. | | `remove_after_transfer` | `Bool` | Remove source files after they are successfully synchronized.<br/>Default: `false`.<br/>(`--remove-after-transfer`) | | `reset` | `Bool` | Clear the snapshot database and rescan the synchronized directories and files to create a fresh snapshot<br/>Default: `false`.<br/>(`--reset`)(-x) | | `resume` | `Hash` | Partial transfers may exist if communication disruptions caused the underlying ascp processes to terminate early. Note that transfer resumption can only happen if the `reset` option is disabled. If an async session starts with `reset` enabled and resume enabled, transfers interrupted during that session will be resumable, but only if async is then restarted with 'reset' disabled. | | `resume.enabled` | `Bool` | Enable the possibility of resuming individual file transfers between async sessions.<br/>Default: `false`. | | `resume.max_age` | `Integer` | Sets the age limit in days for temporary files that will be preserved on cleanup (usually at async's start and stop) for potential transfer resume. Temp files older than the given value will be removed regardless of whether they might be resumable.<br/>Default: `5`.<br/>(`--resume-age-days={integer}`) | | `resume.min_size` | `Integer` | This field specifies the minimum size of files that will be allowed to resume.<br/>Default: `1048576`.<br/>(`--support-resume={integer}`) | | `resume_scan` | `Bool` | Resume the scan from where the previous execution left off.<br/>Default: `false`.<br/>(`--resume-scan`) | | `scan_dir_rename` | `Bool` | Enable the detection of renamed directories and files compared to the previous scan, based on matching inodes<br/>Default: `false`.<br/>(`--scan-dir-rename`) | | `scan_file_rename` | `Bool` | Enable the detection of renamed files compared to the previous scan, based on matching inodes.<br/>Default: `false`.<br/>(`--scan-file-rename`) | | `scan_intensity` | `Enum[String]` | Scan at the set intensity. `vlow` minimizes system activity. `vhigh` maximizes system activity by continuously scanning files without rest.<br/>Allowed values: `vlow`, `low`, `medium`, `high`, `vhigh`.<br/>Default: `medium`.<br/>(`--scan-intensity={enum}`)(-H) | | `sharing_retry_max` | `Integer` | Retry synchronizations up to the specified maximum number after a sharing violation.<br/>Default: `3`.<br/>(`--sharing-retry-max={integer}`) | | `store_metadata_records` | `Bool` | Store the acls or xattrs in the snapshot database.<br/>Default: `false`.<br/>(`--store-metadata-records`) | | `symbolic_links` | `Enum[String]` | Handle symbolic links with the specified method. Default is `skip` on windows, `copy` otherwise.<br/>Allowed values: `copy`, `skip`, `follow`.<br/>(`--symbolic-links={enum}`)(-n) | | `tags` | `Hash` | User-defined metadata tags.<br/>(special:`--tags64={object}`) | | `transfer_threads` | `Array[Hash]` | Use the specified number of dedicated transfer threads to process files smaller or equal to the specified size<br/>(special:`--transfer-threads={array}`) | | `transfer_threads[].size` | `Integer` | Upper limit. `-1` for infinity.<br/>Default: `-1`. | | `transfer_threads[].threads` | `Integer` | The number of threads.<br/>Required. | | `transport` | `Hash` | | | `transport.cipher` | `Enum[String]` | Specify encryption algorithm for file data.<br/>Allowed values: `none`, `aes128`, `aes192`, `aes256`, `aes128cfb`, `aes192cfb`, `aes256cfb`, `aes128gcm`, `aes192gcm`, `aes256gcm`.<br/>Default: `aes128`.<br/>(`--cipher={enum}`)(-c) | | `transport.compression` | `Enum[String]` | Compress a file before transfer using the specified MODE.<br/>Allowed values: `none`, `zlib`.<br/>Default: `none`.<br/>(`--compression={enum}`) | | `transport.datagram_size` | `Integer` | Specify the datagram size (MTU) for FASP. By default it uses the detected path MTU.<br/>(`--datagram-size={integer}`)(-Z) | | `transport.min_rate` | `Integer` | Attempt to transfer no slower than the specified rate (in bps).<br/>Default: `0`.<br/>(`--min-rate={integer}`)(-m) | | `transport.rate_policy` | `Enum[String]` | Defines how `ascp` will manage the bandwidth.<br/>Allowed values: `fair`, `fixed`, `high`, `low`.<br/>Default: `fair`.<br/>(`--rate-policy={enum}`)(-a) | | `transport.raw_options` | `Array[String]` | Pass arbitrary arguments to `ascp`.<br/>(special:`--raw-options={array}`) | | `transport.read_block_size` | `Integer` | Use the specified block size (in bytes) for reading. Default is determined by `aspera.conf`.<br/>(`--read-block-size={integer}`)(-g) | | `transport.rexmsg_size` | `Integer` | Use the specified size (in bytes) for a retransmission request. Default is determined by `aspera.conf`.<br/>(`--rexmsg-size={integer}`)(-X) | | `transport.target_rate` | `Integer` | Transfer no faster than the specified rate (in bps). A string with suffix `k`, `m` or `g` is converted to bps by `ascli` (e.g. `100m`).<br/>Default: `10000000`.<br/>(`--target-rate={integer}`)(-l) | | `transport.udp_port` | `Integer` | Use the specified UDP port for FASP data transfer.<br/>Default: `33001`.<br/>(`--udp-port={integer}`)(-O) | | `transport.write_block_size` | `Integer` | Use the specified block size (in bytes) for writing. Default is determined by `aspera.conf`.<br/>(`--write-block-size={integer}`)(-G) | | `watchd` | `Hash` | When connection is configured, `asperawatchd` is used to detect the changes on the source directory.<br/>(special:`--watchd={object}`) | | `watchd.datastore` | `Enum[String]` | Specify the type of datastore, `none` for disabled.<br/>Allowed values: `none`, `redis`, `scalekv`.<br/>Default: `none`. | | `watchd.domain` | `String` | Specify the domain. Default is the current username. | | `watchd.host` | `String` | Use the specified host name or address to connect to the datastore. | | `watchd.port` | `Integer` | Use the specified port.<br/>Default: `31415`. | | `write_gid` | `String` | Try to write files as the specified group.<br/>(`--write-gid={string}`) | | `write_uid` | `String` | Try to write files as the specified user.<br/>(`--write-uid={string}`) | ##### `sync_info`: `args` format This is the **legacy** syntax. `ascli` defines a JSON equivalent to regular `async` options. It is based on a JSON representation of `async` command line options. Technically, it allows definition of multiple sync sessions in a single command, but `ascli` only accepts a single session for consistency with the previous syntax. This format is selected if the `sync_info` `Hash` has either key `sessions` or `instance`. The following parameters are automatically filled from mandatory arguments, and are not allowed: - `direction` - `local_dir` - `remote_dir` Parameter `name` is set to a default value if not provided in `sync_info`. #### Sync management and monitoring: `admin` The `admin` command provides subcommands to inspect the state of an Async sync session. Most subcommands read the local snap database (`snap.db`) directly - no server connection is required. The exception is `status`, which calls the `asyncadmin` utility available only on server products. These commands can also be run from the `config` plugin: ```shell ascli config sync admin ``` **Prerequisites:** Most `admin` subcommands require the `sqlite3` gem: ```shell gem install sqlite3 ``` **Usage:** All `admin` subcommands share the same argument structure: ```shell ascli ... sync admin <COMMAND> <FOLDER> [<SYNC_INFO>] ``` - `<COMMAND>` - one of the subcommands listed below. - `<FOLDER>` - path to the local database folder (a folder containing a `.private-asp` subfolder). By default this is the local synchronized folder; if a separate database folder was configured, specify that path instead. - `[<SYNC_INFO>]` - optional `Hash` to identify the session. If the folder contains only one session it is selected automatically; otherwise provide `@: name=<SESSION_NAME>`. The only exception is `find`, which takes a plain directory path and lists all `<SESSION_NAME>` found inside it. **Subcommands:** | Command | Uses | Description | |-------------|---------------|--------------------------------------------------------------------------------------------------------| | `find` | filesystem | Search a directory for existing sync sessions and list their names and paths | | `status` | `asyncadmin` | Retrieve the status of a running sync session (requires a server-side product) | | `meta` | `sqlite3` | Display session metadata from `sync_snapmeta_table` | | `counters` | `sqlite3` | Display synchronization counters from `sync_snap_counters_table` | | `file_info` | `sqlite3` | List the synchronization state of each file from `sync_snapdb_table` (state, record ID, path, message) | | `overview` | `sqlite3` | List all tables and their columns in the snap database | | `query` | `sqlite3` | Execute an arbitrary SQL statement against the snap database and return the result rows | The `query` subcommand requires `--sql` with the full SQL statement to execute. The sqlite3-based subcommands (`meta`, `counters`, `file_info`) accept `--sql` optionally to append a SQL fragment (for example, a `WHERE` or `ORDER BY` clause): ```shell ascli ... sync admin query <FOLDER> --sql="SELECT * FROM sync_snapdb_table WHERE state=20" ascli ... sync admin file_info <FOLDER> --sql="WHERE state=20" ascli ... sync admin file_info <FOLDER> --sql="WHERE state=20 ORDER BY f_meta_path" ``` **Snap database schema:** The snap database (`snap.db`) contains the following tables: **`sync_snapmeta_table`** - one row per session, written at start and updated at stop: | Field | Type | Description | |-------------------------|--------------|----------------------------------------------------------------------------------| | `ascp_sids` | `String` | Last known FASP (ascp) session IDs. | | `byte_peerxfer` | `Integer` | Number of bytes transferred by the peer. | | `byte_synced` | `Integer` | Number of bytes successfully synchronized. | | `byte_xferres` | `Integer` | Number of bytes for which transfer was resumed. | | `checksum` | `Enum[String]` | Checksum algorithm.<br/>Allowed values: `sha1`, `md5`, `sha1_sparse`, `md5_sparse`, `none`. | | `cip_true_val` | `Integer` | Candidate-in-progress counter true value (internal). | | `condition` | `Enum[String]` | Last known session condition.<br/>Allowed values: `running`, `stopped`. | | `dbversion` | `String` | Database schema version. | | `direction` | `Enum[String]` | Sync direction.<br/>Allowed values: `push`, `pull`, `bidi`. | | `exclude_dirs_older_than` | `Integer` | Exclude directories with modification time older than this epoch value (0 = disabled). | | `file_peerxfer` | `Integer` | Number of files transferred by the peer. | | `file_synced` | `Integer` | Number of files successfully synchronized. | | `file_xferres` | `Integer` | Number of files for which transfer was resumed. | | `filters` | `String` | Active include/exclude filter patterns. | | `inode_format` | `String` | Inode format used by the session. | | `ldir` | `String` | Local directory path. | | `max_recid` | `Integer` | Highest record ID allocated in the snapshot table. | | `pairname` | `String` | Sync session name. | | `path_peerxfer` | `Integer` | Number of paths transferred by the peer. | | `path_synced` | `Integer` | Number of paths successfully synchronized. | | `path_xferres` | `Integer` | Number of paths for which transfer was resumed. | | `rdir` | `String` | Remote directory path. | | `rhost` | `String` | Remote host. | | `scn_true_val` | `Integer` | Scan counter true value (internal). | | `start_time` | `String` | Session start timestamp. | | `stop_time` | `String` | Session stop timestamp (written when session ends). | | `sync_point` | `Integer` | Synchronization point sequence number. | | `sync_uuid` | `String` | UUID identifying this sync pair. | **`sync_snap_counters_table`** - one row, updated live during a session: | Field | Type | Description | |--------------------------|---------|--------------------------------------------------| | `conflict` | `Integer` | Number of paths in the `Conflict` state. | | `error` | `Integer` | Number of paths in the `Error` state. | | `files_conflict` | `Integer` | Number of files in the `Conflict` state. | | `files_error` | `Integer` | Number of files in the `Error` state. | | `files_pending` | `Integer` | Number of files still pending synchronization. | | `files_potential_conflict` | `Integer` | Number of files in the `Potential conflict` state. | | `files_syncd` | `Integer` | Number of files in the `Syncd` state. | | `pending` | `Integer` | Number of paths still pending synchronization. | | `potential_conflict` | `Integer` | Number of paths in the `Potential conflict` state. | | `syncd` | `Integer` | Number of paths in the `Syncd` state. | | `unused` | `Integer` | Reserved (unused counter slot). | **`sync_snapdb_table`** - one row per tracked file or directory: | Field | Type | Description | |------------------------|---------|----------------------------------------------------------------------------------| | `c_cont_csum` | `String` | Current content — checksum. | | `c_cont_mtime` | `Integer` | Current content — modification time (epoch). | | `c_cont_size` | `Integer` | Current content — file size in bytes. | | `c_cont_timestamp` | `Integer` | Current content — internal timestamp. | | `c_meta_atime` | `Integer` | Current metadata — access time (epoch). | | `c_meta_attrs` | `String` | Current metadata — file attributes. | | `c_meta_ctime` | `Integer` | Current metadata — change time (epoch). | | `c_meta_gid` | `Integer` | Current metadata — group ID (Unix). | | `c_meta_mtime` | `Integer` | Current metadata — modification time (epoch). | | `c_meta_object_metadata` | `String` | Current metadata — object metadata (JSON). | | `c_meta_ol_legal_hold` | `Integer` | Current metadata — object lock legal hold flag. | | `c_meta_ol_mode` | `Integer` | Current metadata — object lock mode. | | `c_meta_ol_retain_until` | `Integer` | Current metadata — object lock retain-until date (epoch). | | `c_meta_parentrid` | `Integer` | Current metadata — parent record ID. | | `c_meta_path` | `String` | Current metadata — file path as known to the snapshot. | | `c_meta_type` | `Integer` | Current metadata — file type (file, directory, …). | | `c_meta_uid` | `Integer` | Current metadata — user ID (Unix). | | `c_meta_xattrs` | `Integer` | Current metadata — extended attributes (not used). | | `c_meta_xattrs_cksum` | `String` | Current metadata — checksum of extended attributes. | | `candidate_in_prog` | `Integer` | Set to 1 when this record is currently in the candidate transfer list. | | `commit_version` | `Integer` | Local commit version for this entry. | | `f_cont_csum` | `String` | Filesystem content — checksum. | | `f_cont_mtime` | `Integer` | Filesystem content — modification time (epoch). | | `f_cont_size` | `Integer` | Filesystem content — file size in bytes. | | `f_cont_timestamp` | `Integer` | Filesystem content — internal timestamp. | | `f_meta_atime` | `Integer` | Filesystem metadata — access time (epoch). | | `f_meta_attrs` | `String` | Filesystem metadata — file attributes. | | `f_meta_ctime` | `Integer` | Filesystem metadata — change time (epoch). | | `f_meta_gid` | `Integer` | Filesystem metadata — group ID (Unix). | | `f_meta_mtime` | `Integer` | Filesystem metadata — modification time (epoch). | | `f_meta_object_metadata` | `String` | Filesystem metadata — object metadata (JSON). | | `f_meta_ol_legal_hold` | `Integer` | Filesystem metadata — object lock legal hold flag. | | `f_meta_ol_mode` | `Integer` | Filesystem metadata — object lock mode. | | `f_meta_ol_retain_until` | `Integer` | Filesystem metadata — object lock retain-until date (epoch). | | `f_meta_parentrid` | `Integer` | Filesystem metadata — parent record ID. | | `f_meta_path` | `String` | Filesystem metadata — file path as seen on disk (displayed by `file_info`). | | `f_meta_type` | `Integer` | Filesystem metadata — file type. | | `f_meta_uid` | `Integer` | Filesystem metadata — user ID (Unix). | | `f_meta_xattrs` | `Integer` | Filesystem metadata — extended attributes (not used). | | `f_meta_xattrs_cksum` | `String` | Filesystem metadata — checksum of extended attributes. | | `message` | `String` | Error or status message associated with this entry. | | `meta_lastreq_no` | `Integer` | Sequence number of the last metadata request sent. | | `meta_lastresp_no` | `Integer` | Sequence number of the last metadata response received. | | `nodeid` | `Integer` | Local filesystem node ID. | | `p_cont_csum` | `String` | Pending content — checksum. | | `p_cont_mtime` | `Integer` | Pending content — modification time (epoch). | | `p_cont_size` | `Integer` | Pending content — file size in bytes. | | `p_cont_timestamp` | `Integer` | Pending content — internal timestamp. | | `pcr_reqno` | `Integer` | Peer content request sequence number. | | `peer_commit_version` | `Integer` | Peer commit version for this entry. | | `peer_nodeid` | `Integer` | Peer filesystem node ID. | | `peer_rid` | `Integer` | Peer record ID for this entry. | | `pp_cont_csum` | `String` | Peer pending content — checksum. | | `pp_cont_mtime` | `Integer` | Peer pending content — modification time (epoch). | | `pp_cont_size` | `Integer` | Peer pending content — file size in bytes. | | `pp_cont_timestamp` | `Integer` | Peer pending content — internal timestamp. | | `record_id` | `Integer` | Primary key (auto-increment). | | `recursive_mtime` | `Integer` | Most recent modification time across all descendants (directories only). | | `scn_completed` | `Integer` | Set to 1 when the scan of this directory entry has completed. | | `scn_not_visited` | `Integer` | Set to 1 when the scanner has not yet visited this entry in the current scan pass. | | `scn_required` | `Integer` | Set to 1 when a rescan of this entry is required. | | `state` | `Integer` | Sync state of the entry.<br/><br/>\| Value \| Name \|<br/>\|-------\|-------------------\|<br/>\| 0 \| Nil \|<br/>\| 1–18 \| Pending (variant) \|<br/>\| 19 \| Syncd \|<br/>\| 20 \| Error \|<br/>\| 21 \| Conflict \|<br/>\| 22 \| PotentialConflict \|<br/>\| 23–24 \| Pending (variant) \| | | `tobe_deleted` | `Integer` | Set to 1 when the record is flagged for deletion (garbage collection pending). | | `version` | `Integer` | Schema version field (not used). | ### Hot folder `ascli` does **not** have a standalone `hotfolder` command. Instead, you create a Hot Folder by combining the upload or download commands with specific transfer-spec parameters and an OS scheduler (like `cron` or Task Scheduler). #### Requirements `ascli` may be used as a simple hot folder engine. A hot folder is defined as a tool that: - Locally (or remotely) detects new files in a top folder - Sends detected files to a remote (respectively, local) repository - Only sends new files, and does not re-send already sent files - Optionally: sends only files that are not still **growing** - Optionally: after transfer of files, deletes or moves to an archive In addition: the detection should be made **continuously** or on specific time/date. #### Setting up a hot folder The general idea is to rely on: - Existing `ascp` features for detection and transfer - `ascli` configuration capabilities and server-side knowledge - The OS scheduler for reliability and continuous operation ##### `ascp` features Useful `ascp` features are available as arguments (see the `ascp` manual): - Sending only **new** files - Option `-k 1,2,3` (`resume_policy`) - Remove or move files after transfer: - `--remove-after-transfer` (`remove_after_transfer`) - `--move-after-transfer` (`move_after_transfer`) - `--remove-empty-directories` (`remove_empty_directories`) - Send only files not modified since the last X seconds: - `--exclude-newer-than` (`exclude_newer_than`) - `--exclude-older-than` (`exclude_older_than`) - Top level folder should not be created on destination - `--src-base` (`src_base`) > [!NOTE] > `ascli` takes transfer parameters exclusively as a [**transfer-spec**](#transfer-specification), with `ts` option. > [!NOTE] > Usual native `ascp` arguments are available as standard [**transfer-spec**](#transfer-specification) parameters, but not special or advanced options. > [!TIP] > Only for the [`direct`](#agent-direct) transfer agent (not others, like connect or node), native `ascp` arguments can be provided with parameter `ascp_args` of option `transfer`. ##### Server side and configuration Virtually any transfer on a **repository** on a regular basis might emulate a hot folder. > [!NOTE] > File detection is not based on events (`inotify`, and so on), but on a simple folder scan on source side. > [!TIP] > Options may be saved in an [Option Preset](#option-preset) and used with `-P`. ##### Scheduling Once `ascli` command line arguments are defined, run the command using the OS native scheduler, for example, every minute or every 5 minutes. See [Scheduler](#scheduler) (and option `lock_port`). #### Example: Upload hot folder ```shell ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"remove_after_transfer":true,"remove_empty_directories":true,"exclude_newer_than":-8,"src_base":"source_hot"}' ``` The local folder (here, relative path: `source_hot`) is sent (upload) to an Aspera server. Source files are deleted after transfer. Growing files will be sent only once they do not grow anymore (based on an 8-second cool-off period). If a transfer takes more than the execution period, then the subsequent execution is skipped (`lock_port`) preventing multiple concurrent runs. #### Example: Unidirectional synchronization (upload) to server ```shell ascli server upload source_sync --to-folder=/Upload/target_sync --lock-port=12345 --ts=@json:'{"resume_policy":"sparse_csum","exclude_newer_than":-8,"src_base":"source_sync"}' ``` This can also be used with other folder-based applications: Aspera on Cloud, Shares, Node. #### Example: Unidirectional synchronization (download) from Aspera on Cloud Files ```shell ascli aoc files download . --to-folder=. --lock-port=12345 --progress-bar=no --out.level=data --ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000,"exclude_newer_than":-8,"delete_before_transfer":true}' ``` > [!NOTE] > Option `delete_before_transfer` will delete files locally, if they are not present on remote side. > [!NOTE] > Options `progress_bar` and `--out.level` limit output for headless operation (for example, a cron job). ### Health check and Nagios Most plugins provide a `health` command that checks the health status of the application. Example: ```shell ascli console health ``` ```text ╭────────┬─────────────┬────────────╮ │ status │ component │ message │ ╞════════╪═════════════╪════════════╡ │ ok │ console api │ accessible │ ╰────────┴─────────────┴────────────╯ ``` Typically, the health check uses the REST API of the application, with the following exception: the `server` plugin checks health by issuing a transfer to the server (`server health transfer`). `ascli` can be called by Nagios to check the health status of an Aspera server. The output can be made compatible with Nagios with option `--format=nagios`: ```shell ascli server health transfer --to-folder=/Upload --format=nagios --progress-bar=no ``` ```text OK - [transfer:ok] ``` ### SMTP for email notifications `ascli` can send emails. To do so, set up the SMTP configuration with option `smtp`. The `smtp` option is a `Hash` ([Extended Value](#extended-value-syntax)) with the following fields: | Field | Type | Description | |------------|---------|----------------------------------------------------------------------------------| | `domain` | `String` | Email domain of the user. Defaults to the domain part of `server`.<br/>Example: `gmail.com`. | | `from_email` | `String` | Sender address (used in `From:` header and as reply-to). Defaults to `username` when defined.<br/>Example: `johnny@example.com`. | | `from_name` | `String` | Display name of the sender. Defaults to the local-part of `from_email`.<br/>Example: `John Wayne`. | | `password` | `String` | Password for the above `username`.<br/>Example: `<PASSWORD>`. | | `port` | `Integer` | Port for the SMTP service.<br/>Defaults: `587` (TLS), `465` (SSL), `25` (plain).<br/>Example: `587`. | | `server` | `String` | SMTP server address.<br/>Required.<br/>Example: `smtp.gmail.com`. | | `ssl` | `Bool` | Enable `TLS/SSL` (port 465).<br/>Default: `false`. | | `tls` | `Bool` | Enable `STARTTLS` (port 587).<br/>Default: `true`. | | `username` | `String` | User to authenticate on the SMTP server. Leave empty for open (unauthenticated) relay.<br/>Example: `john@example.com`. | #### Example of configuration ```shell ascli config preset set smtp_google server smtp.google.com ascli config preset set smtp_google username john@gmail.com ascli config preset set smtp_google password <PASSWORD> ``` or ```shell ascli config preset initialize smtp_google @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"<PASSWORD>"}' ``` or ```shell ascli config preset update smtp_google --server=smtp.google.com --username=john@gmail.com --password=<PASSWORD> ``` Set this configuration as global default, for instance: ```shell ascli config preset set cli_default smtp @val:@preset:smtp_google ascli config preset set default config cli_default ``` #### Email templates Sent emails are built using a template that uses the [`ERB`](https://www.tutorialspoint.com/ruby/eruby.htm) syntax. The template is the full SMTP message, including headers. The following variables are defined by default: - `from_name` - `from_email` - `to` Other variables are defined depending on context. #### Test Check settings with `smtp_settings` command. Send test email with `email_test`. ```shell ascli config smtp_settings --smtp=@preset:smtp_google ascli config email_test --smtp=@preset:smtp_google --notify-to=sample.dest@example.com ``` #### Notifications for transfer status An e-mail notification can be sent upon transfer success and failure (one email per transfer job, one job being possibly multi session, and possibly after retry). To activate, use option `notify_to`. A default e-mail template is used, but it can be overridden with option `notify_template`. The environment provided contains the following additional variables: - `subject` : a default subject including transfer status - `status` : global status of transfer - `ts` : the [**transfer-spec**](#transfer-specification) used for the transfer - `from_email` : email of sender (from `smtp` configuration) - `from_name` : name of sender (from `smtp` configuration) - `to` : recipient of the email (from `notify_to`) Example of template: ```text From: <%=from_name%> <<%=from_email%>> To: <<%=to%>> Subject: <%=subject%> Transfer is: <%=status%> ``` ### Tool: `asession` This gem comes with a second executable tool providing a simplified standardized interface to start a FASP session: `asession`. It aims at simplifying the startup of a FASP session from a programmatic standpoint as formatting a [**transfer-spec**](#transfer-specification) is: - Common to Aspera Node API (HTTP POST /ops/transfer) - Common to Aspera Connect API (browser JavaScript `startTransfer`) - Generated by using any third-party language-specific JSON library Ideally, IBM will integrate this directly into `ascp`, making this tool redundant. Integration with any language is possible, provided that the language can spawn a subprocess, write to its STDIN, read from STDOUT, and generate and parse JSON. `asession` expects a single argument: a session specification that contains parameters and a [**transfer-spec**](#transfer-specification). If no argument is provided, it assumes a value of: `@json:@stdin:`, that is, JSON on stdin. > [!NOTE] > If JSON is the format, specify `@json:` to tell `asession` to decode the `Hash` using JSON syntax. During execution, it generates all low level events, one per line, in JSON format on stdout. Top level parameters supported by `asession`: | Parameter | Description | |--------------------|------------------------------------------------------------------------| | `spec` | The [**transfer-spec**](#transfer-specification) | | `agent` | Same parameters as `transfer` option for agent `direct` | | `loglevel` | Log level of `asession` | | `file_list_folder` | The folder used to store (for garbage collection) generated file lists.<br/>Default: `[system tmp folder]/[username]_asession_filelists` | #### Comparison of interfaces | Feature/Tool | Transfer Daemon | FASPManager | `ascp` | `asession` | |------------|-------------|-------------|-------------|-------------| | status | Supported | Deprecated | Supported | Deprecated | | language integration | Many | C/C++<br/>C#/.net<br/>Go<br/>Python<br/>java<br/> | Any | Any | | required additional components to `ascp` | Daemon | Library<br/>(+headers) | - | Ruby<br/>Aspera gem | | startup | Daemon | API | Command line arguments | JSON on stdin<br/>(standard APIs:<br/>JSON.generate<br/>Process.spawn) | | events | Poll | Callback | Possibility to open management port<br/>and proprietary text syntax | JSON on stdout | | platforms | Like `ascp` and `transferd` | Like `ascp` and lib (if compiled) | Any with `ascp` | Any with Ruby and `ascp` | #### Simple session Create a file `session.json` with: ```json {"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"<PASSWORD>","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"} ``` Then start the session: ```shell asession < session.json ``` #### Asynchronous commands and Persistent session `asession` also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in `ascp` manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be **snake** style, for example: `LongParameter` → `long_parameter` This is particularly useful for a persistent session (with the [**transfer-spec**](#transfer-specification) parameter: `"keepalive":true`) ```text asession {"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"<PASSWORD>","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"} {"type":"START","source":"/aspera-test-dir-tiny/200KB.2"} {"type":"DONE"} ``` (events from FASP are not shown in above example. They would appear after each command) #### Help ```shell asession -h USAGE asession asession -h|--help asession [<session spec extended value>] If no argument is provided, default will be used: @json:@stdin -h, --help display this message <session spec extended value> a dictionary (Hash) The value can be either: the JSON description itself, e.g. @json:'{"xx":"yy",...}' @json:@stdin, if the JSON is provided from stdin @json:@file:<path>, if the JSON is provided from a file The following keys are recognized in session spec: spec : mandatory, contains the transfer spec loglevel : modify log level (to stderr) agent : modify transfer agent parameters, e.g. ascp_args file_list_folder : location of temporary files sdk : location of SDK (ascp) Asynchronous commands can be provided on STDIN, examples: {"type":"START","source":"/aspera-test-dir-tiny/200KB.2"} {"type":"START","source":"xx","destination":"yy"} {"type":"DONE"} EXAMPLES asession @json:'{"spec":{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"<PASSWORD>","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}]}}' echo '{"spec":{"remote_host":...}}'|asession @json:@stdin: ``` ### Ruby Module: `Aspera` Main components: - `Aspera` generic classes for REST and OAuth - `Aspera::Agent::Direct`: Starting and monitoring transfers using `ascp`. - `Aspera::Cli`: `ascli`. Working examples can be found in repo: <https://github.com/laurent-martin/aspera-api-examples> in Ruby examples. ## Common problems `ascli` detects common problems and provides hints to solve them. ### Error: "Remote host is not who we expected" Cause: `ascp` >= 4.x checks fingerprint of the highest server host key, including ECDSA. `ascp` < 4.0 (3.9.6 and earlier) supports only RSA (and ignores ECDSA presented by the server). `aspera.conf` supports a single fingerprint. Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the configuration file): ```shell --ts=@json:'{"sshfp":null}' ``` Workaround on server side: Either remove the fingerprint from `aspera.conf`, or keep only RSA host keys in `sshd_config`. References: ES-1944 in the release notes of 4.1, and the [HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint"](https://www.ibm.com/docs/en/ahts/4.2?topic=upgrades-configuring-ssh-server). ### Error: "can't find header files for ruby" Some Ruby gems dependencies require compilation of native parts (C). This also requires Ruby header files. If Ruby was installed as a Linux package, then also install the Ruby development package: `ruby-dev` or `ruby-devel`, depending on distribution. ### Private key type: `ed25519` Keys of type `ed25519` and keys in the OpenSSH format are supported by `net-ssh` using the Ruby `openssl` gem (>= 3.2.0): the `ed25519` gem is not needed. Only OpenSSH private keys protected by a passphrase require the `bcrypt_pbkdf` gem, which requires compilation of native code: ```shell gem install bcrypt_pbkdf ``` If the gem is missing, then loading such a key fails with an error mentioning `bcrypt_pbkdf`. If the OpenSSL backend does not support Ed25519, then the following message is displayed: ```text OpenSSH keys only supported if ED25519 is available (NotImplementedError) net-ssh requires openssl >= 3.2.0 and a crypto backend with Ed25519 enabled for ed25519 support. ``` To ignore the `ed25519` keys of the default identities (`~/.ssh/id_ed25519`), set env var `ASCLI_ENABLE_ED25519` to `false`. In addition, if **JRuby** is used, host keys of type: `ecdsa-sha2` and `ecdh-sha2` are also deactivated by default. To activate, set env var `ASCLI_ENABLE_ECDSHA2` to `true`. ### JRuby: `net-ssh`: Unsupported algorithm JRuby may not implement all the algorithms supported by OpenSSH. Add the following to option `ssh_options`: ```json {"host_key":["rsa-sha2-512","rsa-sha2-256"],"kex":["curve25519-sha256","diffie-hellman-group14-sha256"],"encryption": ["aes256-ctr", "aes192-ctr", "aes128-ctr"]} ``` For example: ```shell --ssh-options=@json:'{"host_key":["rsa-sha2-512","rsa-sha2-256"],"kex":["curve25519-sha256","diffie-hellman-group14-sha256"],"encryption": ["aes256-ctr", "aes192-ctr", "aes128-ctr"]}' ``` ### Error: "SSL_read: unexpected eof while reading" Newer OpenSSL libraries expect a clean SSL close. To deactivate this error, enable option `IGNORE_UNEXPECTED_EOF` for `ssl_options` in option `http_options`. ```shell --http-options=@json:'{"ssl_options":["IGNORE_UNEXPECTED_EOF"]}' ``` ### Error: `ascp`: `/lib64/libc.so.6`: version `GLIBC_2.28` not found This happens on Linux x86 if you try to install `transferd` on a Linux version too old to support a newer `ascp` executable. Workaround: Install an older version of `transferd`: ```shell ascli transferd install 1.1.2 ``` See [Single file executable](#single-file-executable). ### Error: Cannot rename partial file This is an error coming from `ascp` when it is configured to use a partial file name, and at the end of the transfer, the partial file, now complete, does not exist anymore. This often happens when two transfers start in parallel for the same file: - Session 1 starts for file1, it creates file: file1.partial and fills it. - Session 2 starts for file1 (same), it creates file: file1.partial and fills it. - Session 1 finishes, and renames file1.partial to file1. - Session 2 finishes, and tries to rename file1.partial to file1, but it fails as it does not exist anymore... By default, `ascli` creates a configuration file `~/.aspera/sdk/aspera.conf` like this: ```xml <?xml version='1.0' encoding='UTF-8'?> <CONF version="2"> <default> <file_system> <resume_suffix>.aspera-ckpt</resume_suffix> <partial_file_suffix>.partial</partial_file_suffix> </file_system> </default> </CONF> ``` In container, this is located in `/ibm_aspera`. One possibility to avoid that error is to disable partial filename suffix... But that only hides the problem. For example, when using the container, override that file with a volume and remove the line for extension. Another possibility is to add this option: `--transfer=@json:'{"ascp_args":["--partial-file-suffix="]}'` : this overrides the value in config file. > [!NOTE] > If one relies on `--lock-port` when using containers to avoid parallel transfers in a cron job, this may be the cause, as `lock_port` does not lock across containers. > Use `flock` instead. ## About ### Hootput, the mascot ![Hootput the Owl](mascot.svg) Hootput lives in the terminal, watching over every command with wide, unblinking eyes. Known for concise output and sharp insight, this owl thrives where others get lost in the dark. It doesn't chatter; it hoots: clear, precise, and always on time. Like `ascli`, Hootput is built for action: launching transfers, parsing options, and navigating APIs without hesitation. Light on feathers but heavy on wisdom, it turns complexity into simple one-liners. When you hear Hootput's call, you know your data is already in flight. ### History When I joined Aspera, there was only one CLI: `ascp`, the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares). At some point, Serban (founder) provided a shell script able to create a Faspex package using the Faspex REST API. Since all products relate to file transfers using FASP (`ascp`), I thought it would be interesting to have a unified CLI for FASP transfers. Also, because `ascp` already existed, I conceived an extended tool: `eascp.pl`, which accepted all `ascp` options for transfers but was also able to transfer to Faspex and Shares (the destination was a kind of URI for the applications). There were a few pitfalls: - `eascp.pl` was written in the aging `perl` language, while most Aspera web application products (other than the Transfer Server) are written in `ruby`. - It was only for transfers and was not able to call other product APIs. So, it evolved into `ascli`: - Portable: works on platforms supporting `ruby` (and `ascp`) - Easy to install with the `gem` utility - Supports transfers with multiple [Transfer Agents](#transfer-clients-agents); this is why transfer parameters moved from the `ascp` command line to the [**transfer-spec**](#transfer-specification) (more reliable, more standard) - `ruby` is consistent with other Aspera products Over time, a supported command-line tool `aspera` was developed in C++ and later deprecated. It had the advantage of being relatively easy to install as a single executable (still using `ascp`), but it was too limited in scope and lacked many of the features of this CLI. Enjoy a coffee on me: ```shell ascli config coffee ascli config coffee --ui=text ascli config coffee --ui=text --out.img.text=true ``` ### References Ruby gem: [https://rubygems.org/gems/aspera-cli](https://rubygems.org/gems/aspera-cli) Ruby Doc: [https://ibm.biz/ascli-doc](https://ibm.biz/ascli-doc) [Aspera APIs on IBM developer](https://developer.ibm.com/apis/catalog/?search=aspera)