--- name: shell-scripting description: Write robust, portable shell scripts with proper error handling, argument parsing, and testing. Use when automating system tasks, building CI/CD scripts, or creating container entrypoints. --- # Shell Scripting ## Purpose Provides patterns and best practices for writing maintainable shell scripts with error handling, argument parsing, and portability considerations. Covers POSIX sh vs Bash decision-making, parameter expansion, integration with common utilities (jq, yq, awk), and testing with ShellCheck and Bats. ## When to Use This Skill Use shell scripting when: - Orchestrating existing command-line tools and system utilities - Writing CI/CD pipeline scripts (GitHub Actions, GitLab CI) - Creating container entrypoints and initialization scripts - Automating system administration tasks (backups, log rotation) - Building development tooling (build scripts, test runners) Consider Python/Go instead when: - Complex business logic or data structures required - Cross-platform GUI needed - Heavy API integration (REST, gRPC) - Script exceeds 200 lines with significant logic complexity ## POSIX sh vs Bash **Use POSIX sh (#!/bin/sh) when:** - Maximum portability required (Linux, macOS, BSD, Alpine) - Minimal container images needed - Embedded systems or unknown target environments **Use Bash (#!/bin/bash) when:** - Controlled environment (specific OS, container) - Arrays or associative arrays needed - Advanced parameter expansion beneficial - Process substitution `<(cmd)` useful For detailed comparison and testing strategies, see `references/portability-guide.md`. ## Essential Error Handling ### Fail-Fast Pattern ```bash #!/bin/bash set -euo pipefail # -e: Exit on error # -u: Exit on undefined variable # -o pipefail: Pipeline fails if any command fails ``` Use for production automation, CI/CD scripts, and critical operations. ### Explicit Exit Code Checking ```bash #!/bin/bash if ! command_that_might_fail; then echo "Error: Command failed" >&2 exit 1 fi ``` Use for custom error messages and interactive scripts. ### Trap Handlers for Cleanup ```bash #!/bin/bash set -euo pipefail TEMP_FILE=$(mktemp) cleanup() { rm -f "$TEMP_FILE" } trap cleanup EXIT ``` Use for guaranteed cleanup of temporary files, locks, and resources. For comprehensive error patterns, see `references/error-handling.md`. ## Argument Parsing ### Short Options with getopts (POSIX) ```bash #!/bin/bash while getopts "hvf:o:" opt; do case "$opt" in h) usage ;; v) VERBOSE=true ;; f) INPUT_FILE="$OPTARG" ;; o) OUTPUT_FILE="$OPTARG" ;; *) usage ;; esac done shift $((OPTIND - 1)) ``` ### Long Options (Manual Parsing) ```bash #!/bin/bash while [[ $# -gt 0 ]]; do case "$1" in --help) usage ;; --verbose) VERBOSE=true; shift ;; --file) INPUT_FILE="$2"; shift 2 ;; --file=*) INPUT_FILE="${1#*=}"; shift ;; *) break ;; esac done ``` For hybrid approaches and validation patterns, see `references/argument-parsing.md`. ## Parameter Expansion Quick Reference ```bash # Default values ${var:-default} # Use default if unset ${var:=default} # Assign default if unset : "${API_KEY:?Error: required}" # Error if unset # String manipulation ${#var} # String length ${var:offset:length} # Substring ${var%.txt} # Remove suffix ${var##*/} # Basename ${var/old/new} # Replace first ${var//old/new} # Replace all # Case conversion (Bash 4+) ${var^^} # Uppercase ${var,,} # Lowercase ``` For complete expansion patterns and array handling, see `references/parameter-expansion.md`. ## Common Utilities Integration ### JSON with jq ```bash # Extract field name=$(curl -sSL https://api.example.com/user | jq -r '.name') # Filter array active=$(jq '.users[] | select(.active) | .name' data.json) # Check existence if ! echo "$json" | jq -e '.field' >/dev/null; then echo "Error: Field missing" >&2 fi ``` ### YAML with yq ```bash # Read value (yq v4) host=$(yq eval '.database.host' config.yaml) # Update in-place yq eval '.port = 5432' -i config.yaml # Convert to JSON yq eval -o=json config.yaml ``` ### Text Processing ```bash # awk: Extract columns awk -F',' '{print $1, $3}' data.csv # sed: Replace text sed 's/old/new/g' file.txt # grep: Pattern match grep -E "ERROR|WARN" logfile.txt ``` For detailed examples and best practices, see `references/common-utilities.md`. ## Testing and Validation ### ShellCheck: Static Analysis ```bash # Check script shellcheck script.sh # POSIX compliance shellcheck --shell=sh script.sh # Exclude warnings shellcheck --exclude=SC2086 script.sh ``` ### Bats: Automated Testing ```bash #!/usr/bin/env bats @test "script runs successfully" { run ./script.sh --help [ "$status" -eq 0 ] [ "${lines[0]}" = "Usage: script.sh [OPTIONS]" ] } @test "handles missing argument" { run ./script.sh [ "$status" -eq 1 ] [[ "$output" =~ "Error" ]] } ``` Run tests: ```bash bats test/ ``` For CI/CD integration and debugging techniques, see `references/testing-guide.md`. ## Defensive Programming Checklist ```bash #!/bin/bash set -euo pipefail # Check required commands command -v jq >/dev/null 2>&1 || { echo "Error: jq required" >&2 exit 1 } # Check environment variables : "${API_KEY:?Error: API_KEY required}" # Check files [ -f "$CONFIG_FILE" ] || { echo "Error: Config not found: $CONFIG_FILE" >&2 exit 1 } # Quote all variables echo "Processing: $file" # ❌ Unquoted echo "Processing: \"$file\"" # ✅ Quoted ``` ## Platform Considerations ### macOS vs Linux Differences ```bash # sed in-place sed -i '' 's/old/new/g' file.txt # macOS sed -i 's/old/new/g' file.txt # Linux # Portable: Use temp file sed 's/old/new/g' file.txt > file.txt.tmp mv file.txt.tmp file.txt # readlink readlink -f /path # Linux only cd "$(dirname "$0")" && pwd # Portable ``` For complete platform differences, see `references/portability-guide.md`. ## Script Categories **System Administration:** Cron jobs, log rotation, backup automation **Build/Deployment:** CI/CD pipelines, Docker builds, deployments **Development Tooling:** Project setup, test runners, code generators **Container Entrypoints:** Initialization, signal handling, configuration ## Production Script Template ```bash #!/bin/bash set -euo pipefail readonly SCRIPT_NAME="$(basename "$0")" readonly SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" TEMP_DIR="" cleanup() { local exit_code=$? rm -rf "$TEMP_DIR" exit "$exit_code" } trap cleanup EXIT log() { echo "[$(date +'%Y-%m-%d %H:%M:%S')] $*" >&2 } main() { # Check dependencies command -v jq >/dev/null 2>&1 || exit 1 # Parse arguments # Validate input # Process # Report results log "Completed successfully" } main "$@" ``` For complete production template, see `examples/production-template.sh`. ## Tool Recommendations **Core Tools:** - **jq**: JSON parsing and transformation - **yq**: YAML parsing (v4 recommended) - **ShellCheck**: Static analysis and linting - **Bats**: Automated testing framework **Installation:** ```bash # macOS brew install jq yq shellcheck bats-core # Ubuntu/Debian apt-get install jq shellcheck ``` ## Related Skills - **linux-administration**: System commands and administration - **building-ci-pipelines**: Using scripts in CI/CD - **infrastructure-as-code**: Terraform/Pulumi wrappers - **kubernetes-operations**: kubectl scripts, Helm hooks - **writing-dockerfiles**: Container entrypoints ## Additional Resources **Reference Files:** - `references/error-handling.md` - Comprehensive error patterns - `references/argument-parsing.md` - Advanced parsing techniques - `references/parameter-expansion.md` - Complete expansion reference - `references/portability-guide.md` - POSIX vs Bash differences - `references/testing-guide.md` - ShellCheck and Bats guide - `references/common-utilities.md` - jq, yq, awk, sed usage **Example Scripts:** - `examples/production-template.sh` - Production-ready template - `examples/getopts-basic.sh` - Simple getopts usage - `examples/getopts-advanced.sh` - Complex option handling - `examples/long-options.sh` - Manual long option parsing - `examples/error-handling.sh` - Error handling patterns - `examples/json-yaml-processing.sh` - jq/yq examples **Utility Scripts:** - `scripts/lint-script.sh` - ShellCheck wrapper for CI - `scripts/test-script.sh` - Bats wrapper for CI