# Alpasim Operations Guide

This guide covers common operational tasks for tuning, optimizing, and troubleshooting Alpasim.

## Performance Tuning

### How do I change replica counts and GPU distribution?

The number of service replicas and their GPU assignments are configured in deployment configs
located in `src/wizard/configs/deploy/`:

- **Local workstation**: `local.yaml`
- **Wizard-managed OmniDreams renderer**: `managed_flashdreams.yaml`
- **External OmniDreams renderer**: `external_video_model.yaml`

The default renderer path is NuRec-backed. OmniDreams runs use FlashDreams behind the same
`renderer` endpoint; see [VIDEO_MODEL.md](VIDEO_MODEL.md) for the renderer-specific chunking,
camera, and deployment constraints.

#### Understanding the Configuration

Each service has two key parameters:

```yaml
services:
  renderer:
    replicas_per_container: 4 # Number of service replicas per container
    gpus: [0, 1, 2, 3] # GPUs to create containers on
```

**How it works**:

- **One container per GPU** (or one container total if `gpus: null`)
- Each container runs `replicas_per_container` service instances
- Total replicas = `nr_gpus * replicas_per_container`

Example:

- `gpus: [0, 1, 2, 3]` --> 4 containers (one per GPU)
- `replicas_per_container: 4` --> 4 replicas per container
- **Total**: 4 * 4 = 16 service replicas

#### Balancing Replicas and Concurrent Rollouts

For most services, total simulation throughput capacity is determined by:

```
Total capacity = nr_gpus * replicas_per_container * n_concurrent_rollouts
```

where **`n_concurrent_rollouts`** is the number of rollouts (simulation episodes) each service
replica can process simultaneously. This controls how many scenes can be simulated in parallel.

For NRE-backed renderer deployments, this changed in recent NRE releases: each container may run a
single NRE process with internal worker concurrency via `--max-workers`, instead of multiple
replicas per container. In that case, think of effective per-container capacity as:

```
effective renderer capacity = nr_gpus * max_workers * n_concurrent_rollouts
```

If `replicas_per_container=1` and NRE runs with `--max-workers=4`, one renderer container can
still serve four render workers internally.

All services should have similar total capacity to avoid bottlenecks. Example:

```yaml
services:
  renderer:
    replicas_per_container: 1
    gpus: [0, 1]

  driver:
    replicas_per_container: 8
    gpus: [2, 3]

  controller:
    replicas_per_container: 16
    gpus: null # CPU-only: 1 container

runtime:
  endpoints:
    renderer:
      n_concurrent_rollouts: 4 # with --max-workers=4: 2 GPUs * 4 workers * 4 concurrent = 32

    driver:
      n_concurrent_rollouts: 2 # 2 GPUs * 8 replicas * 2 concurrent = 32

    controller:
      n_concurrent_rollouts: 2 # 1 CPU * 16 replicas * 2 concurrent = 32
```

For NRE-backed renderer deployments, tune these together:

- `services.renderer.replicas_per_container`
- the NRE container's internal `--max-workers`
- `runtime.endpoints.renderer.n_concurrent_rollouts`

Recent OSS deploy configs prefer `replicas_per_container: 1` plus higher `--max-workers`.

For OmniDreams/FlashDreams deployments, start with
`runtime.endpoints.renderer.n_concurrent_rollouts=1` and the `+chunking=8frame` preset from
[VIDEO_MODEL.md](VIDEO_MODEL.md). Scale only after the
renderer server, driver cadence, and GPU memory headroom are known to be stable.

### How do I change the model?

By default, the VaVam driver and model are used. The model weights are downloaded using
`data/download_vavam_assets.sh` and stored in `data/vavam-driver/`.

#### Using a Different Model

To use a custom model, mount a custom vavam-driver directory:

```bash
uv run alpasim_wizard deploy=local topology=1gpu driver=vavam \
    wizard.log_dir=runs/{DATETIME} \
    defines.vavam_driver=/path/to/custom/vavam-driver
```

**Default location**: `data/vavam-driver/` (in repository root) The wizard mounts
`defines.vavam_driver` as `/mnt/vavam_driver` in the container and the driver loads the model from
that path.

#### Using a Different Driver/Inference Code

To use a custom driver container image:

```bash
uv run alpasim_wizard deploy=local topology=1gpu driver=vavam \
    wizard.log_dir=runs/{DATETIME} \
    services.driver.image=<your-registry>/<your-driver-image>:<tag>
```

Your custom image must expose a gRPC endpoint compatible with the driver service interface (see
[protocol buffer definitions](/src/grpc/alpasim_grpc/v0/)).

For development of driver code within this repository, changes to `src/driver/` are automatically
mounted into containers at runtime (see [Code Changes](TUTORIAL.md#code-changes) in TUTORIAL.md).

### How do I change inference frequency?

Changing inference frequency is complex and requires coordinating multiple timing parameters.

#### Understanding the Parameters

The simulator has multiple synchronized "clocks":

1. **Driver inference** (`control_timestep_us`) - How often the model makes decisions
1. **Camera frames** (`frame_interval_us`) - How often cameras capture images
1. **Pose reporting** (`pose_reporting_interval_us`) - How often intermediate poses are reported
   between control steps (0 = at `control_timestep_us` rate, the default)
1. **Simulation start** - The recorded egomotion start timestamp by default

For correct operation, these must be mathematically aligned.

#### Step-by-Step Walkthrough

**Scenario 1: Simple frequency change (matching camera and inference rates)**

To change to 5Hz inference (200ms between decisions):

1. **Set inference frequency** (`control_timestep_us`):

   ```bash
   runtime.simulation_config.control_timestep_us=200000  # 200ms = 5Hz
   ```

1. **Pose reporting** (`pose_reporting_interval_us`): defaults to 0, which falls back to
   `control_timestep_us`. No explicit setting needed unless you want intermediate pose reports.

1. **Match camera frame rate** (VaVam default has 1 camera):

   ```bash
   runtime.simulation_config.cameras.0.frame_interval_us=200000
   ```

   For configs with 2 cameras (e.g., `+cameras=2cam`), also set:

   ```bash
   runtime.simulation_config.cameras.1.frame_interval_us=200000
   ```

**Full command**:

```bash
uv run alpasim_wizard deploy=local topology=1gpu driver=vavam \
    wizard.log_dir=runs/{DATETIME} \
    runtime.simulation_config.control_timestep_us=200000 \
    runtime.simulation_config.cameras.0.frame_interval_us=200000
```

Note: Add `cameras.1.frame_interval_us=200000` if using 2-camera configs.
`pose_reporting_interval_us` defaults to 0, falling back to `control_timestep_us`.

**Scenario 2: High-rate camera with lower inference rate**

To use 30Hz cameras (33.3ms) but 10Hz inference (100ms):

1. **Camera captures at 30Hz**: `frame_interval_us=33334` (33.3ms)
1. **Inference runs at 10Hz**: `control_timestep_us=100002` (must be 3 × 33334)
1. **Subsample frames**: `driver.inference.Cframes_subsample=3` (use every 3rd frame)
1. **Pose reporting**: `pose_reporting_interval_us` defaults to 0 (falls back to `control_timestep_us`)

**Full command** (based on `exp/sim/20s_at_30Hz.yaml`):

```bash
uv run alpasim_wizard deploy=local topology=1gpu driver=vavam \
    wizard.log_dir=runs/{DATETIME} \
    runtime.simulation_config.control_timestep_us=100002 \
    runtime.simulation_config.cameras.0.frame_interval_us=33334 \
    ++driver.inference.Cframes_subsample=3
```

Note: Add `cameras.1.frame_interval_us=33334` if using 2-camera configs.

#### Validation

The `assert_zero_decision_delay` flag (enabled by default in OSS configs) validates timing
synchronization at runtime. It checks that:

- Camera frames complete exactly at decision time
- Pose updates complete exactly at decision time

If misconfigured, the simulator will error with messages like:

```
Camera camera_front_wide_120fov out of sync with planning.
Last started frame finishes at X which is Y microseconds away from decision time Z.
```

**What it does**: At each control step, before calling the driver, the runtime verifies that the
last camera frame and pose update completed exactly at `now_us` (zero delay). This ensures the
model receives perfectly synchronized data.

**Testing your configuration**:

```bash
# The flag is true by default, but you can explicitly set it:
runtime.simulation_config.assert_zero_decision_delay=true
```

#### Common Frequencies

Based on actual config files in `src/wizard/configs/`:

| Frequency | `control_timestep_us` | Notes               |
| --------- | --------------------- | ------------------- |
| 2Hz       | 500000 (500ms)        | VaVam default       |
| 5Hz       | 200000 (200ms)        | Example config      |
| 10Hz      | 100000 (100ms)        | Base config default |
| 30Hz      | 33334 (33.3ms)        | High frequency      |

`pose_reporting_interval_us` defaults to 0 for all frequencies (falls back to `control_timestep_us`).

Simulation starts from the first USDZ timestamp with valid recorded camera and
non-ego actor data. The first policy step is aligned to the first completed
camera frames.

**See also**:

- [src/runtime/README.md - Zero delay mode](/src/runtime/README.md#zero-delay-mode) for
  synchronization requirements
- `src/wizard/configs/driver/vavam_configs.yaml` for a 2Hz example

## Viewing Results and Metrics

### Where are simulation results stored?

After a run completes, results are in `wizard.log_dir` (e.g., `runs/{RUN_DIR}/`):

- **`asl/`** - Simulation logs (`.asl` files for debugging)
- **`eval/`** - Per-rollout driving quality metrics (`metrics_unprocessed.parquet`) and videos
- **`aggregate/`** - Aggregated results across all rollouts:
  - `metrics_results.txt` - Formatted table of driving scores
  - `metrics_results.png` - Visual summary of driving quality metrics
  - `metrics_unprocessed.parquet` - Combined metrics from all rollouts
  - `videos/` - Organized by violation types
- **`prometheus/`** - Performance profiling data:
  - `data/` - local Prometheus TSDB for the run
  - `targets/alpasim.json` - generated Prometheus file-SD targets
- **`metrics_plot.png`** - Performance visualization (CPU/GPU/RPC metrics)
- **`txt-logs/`** - Service logs for debugging
- **`wizard-config.yaml`** - Resolved configuration used for this run

See [TUTORIAL.md - Results Structure](TUTORIAL.md#results-structure) for detailed breakdown.

### Understanding Driving Quality Metrics

The simulation evaluates driving quality across multiple dimensions. Results are in
`aggregate/metrics_results.txt` and visualized in `aggregate/metrics_results.png`.

#### Key Metrics

**Safety Metrics** (binary: 0 = pass, 1 = fail):

- **`collision_at_fault`**: Driver caused a collision (front/lateral impact)
- **`collision_rear`**: Rear-end collision (not at fault)
- **`offroad`**: Vehicle drove off the road

**Performance Metrics** (continuous):

- **`dist_to_gt_trajectory`**: Maximum distance from ground truth path (meters)
  - Lower is better; indicates how closely the driver follows expected routes
  - Aggregated using MAX over time (worst deviation during the drive)
- **`duration_frac_20s`**: Fraction of 20s drive completed before any failure
  - 1.0 = completed full 20s without issues
  - \<1.0 = failed early (collision, off-road, or excessive deviation)

**Distance Between Incidents**:

- **`avg_dist_between_incidents`**: Average km traveled per incident (collision or offroad)
  - Higher is better; measures safety over distance
- **`avg_dist_between_incidents_at_fault`**: Average km traveled per at-fault incident
  - Higher is better; excludes rear-end collisions not caused by the driver

#### Interpreting the Results

The `aggregate/metrics_results.txt` file shows statistics (mean, std, min, max, quantiles) for each
metric across all rollouts. For example:

```
collision_at_fault: mean=0.05 → 5% of rollouts had at-fault collisions
dist_to_gt_trajectory: mean=2.3 → Average 2.3m deviation from GT path
duration_frac_20s: mean=0.95 → Average 95% of 20s completed
```

Videos in `aggregate/videos/violations/` are organized by failure type for easy review of
problematic scenarios.

### How do I view performance metrics?

#### Metrics Plot (Automatically Generated)

After each simulation run, Alpasim automatically generates a comprehensive performance
visualization:

**Location**: `runs/{RUN_DIR}/metrics_plot.png`

This 3×3 grid plot includes:

**Row 1: RPC Performance**

- RPC Duration histogram - Total time from call start to coroutine resumption
- RPC Blocking histogram - Event loop scheduler delay (time from gRPC I/O completion to coroutine
  resumption)
- RPC Queue Depth histogram - Service saturation levels

**Row 2: Simulation Timing**

- Rollout Duration histogram - Total time per rollout
- Step Duration histogram - Time per simulation step
- Run summary - Shows runtime idle fraction and seconds per rollout

**Row 3: Resource Utilization**

- CPU Utilization boxplots - Per-service CPU usage
- GPU Utilization boxplots - GPU compute usage
- GPU Memory boxplots - Memory usage with capacity line

**Summary header** shows:

- Async worker idle percentage - How much time runtime spent idle
- Sim seconds per rollout - Wallclock time per simulation

#### Interpreting the Metrics Plot

**Identifying Bottlenecks**:

- **High queue depth** on a service → Increase replicas_per_container or n_concurrent_rollouts
- **High RPC duration** → Service is slow, consider optimization or scaling
- **Low GPU utilization** (\<50%) → Underutilized, can increase load
- **High GPU utilization** (>90%) → May be saturated, check for throttling
- **Unbalanced service config** → Total capacity should match across all services

**Performance Indicators**:

- **Low idle percentage** (\<20%) → Runtime is busy, good utilization
- **High idle percentage** (>80%) → Lots of waiting, check for bottlenecks
- **Consistent rollout times** → Good stability
- **Wide rollout time variance** → Investigate outliers in logs

## Simulation Configuration

### How do I start the runtime as a server?

Use `wizard.run_mode=SERVER` to start the normal backing services and keep the
runtime alive as a gRPC daemon:

```bash
uv run alpasim_wizard deploy=local topology=1gpu driver=vavam \
    wizard.run_mode=SERVER \
    wizard.log_dir=runs/{DATETIME}
```

The wizard writes the resolved client endpoint to
`generated-runtime-server.yaml` in the run directory:

```yaml
host: localhost
port: 6005
```

This file is generated before Docker Compose has necessarily finished starting
the runtime daemon. Clients should treat `host` and `port` as discovery metadata
and poll until the runtime gRPC port accepts connections.

After the port accepts connections, clients can call
`RuntimeService.get_runtime_info()` to discover the server's maximum supported
concurrent rollouts, available scenes and lightweight scene metadata, service
capacity by backing service, and service versions before submitting
`RuntimeService.simulate()` requests.

By default, the runtime server port is allocated from `wizard.baseport` after
the backing service ports. Pin it with `wizard.runtime_server_port=<port>` when
client scripts or firewall rules need a stable port.

Static external drivers use `driver_source=external_static` with
`wizard.external_services.driver`; `driver=manual` provides a default
`localhost:6789` address for the manual-driver workflow. Server clients can
also use `driver_source=external_dynamic` and provide drivers per request with
`SimulationRequest.available_drivers` and `n_concurrent_per_driver`; those
request addresses override the configured driver pool for that RPC only.

To stop the server, call `RuntimeService.shut_down()`. In managed deployments,
Docker Compose or Slurm tears down backing services after the runtime exits.

When running a generated `docker-compose.yaml` manually for one-shot simulations,
start Compose with `--exit-code-from runtime-0`, for example:

```bash
docker compose -f docker-compose.yaml up --exit-code-from runtime-0
```

The runtime container exits when the simulation is complete, while the backing
services are long-running servers. Without `--exit-code-from runtime-0`, Docker
Compose can continue waiting after `runtime-0` exits successfully.

### How do I enable/disable specific services?

Use `runtime.endpoints.<service>.skip` to disable services:

```bash
# Disable traffic simulation
uv run alpasim_wizard deploy=local topology=1gpu driver=vavam \
    wizard.log_dir=runs/{DATETIME} \
    runtime.endpoints.trafficsim.skip=true

# Disable physics (log replay mode)
uv run alpasim_wizard deploy=local topology=1gpu driver=vavam \
    wizard.log_dir=runs/{DATETIME} \
    runtime.endpoints.physics.skip=true \
    runtime.simulation_config.physics_update_mode=NONE \
    runtime.simulation_config.force_gt_duration_us=20000000
```