openapi: 3.1.0 info: title: Physics Agent Service description: "The Physics Agent Service takes a USD file (uploaded directly or\ \ referenced by S3 URI) as input and runs a multi-step VLM-based classification\ \ pipeline: build_dataset_usd \u2192 prepare_dataset \u2192 predict \u2192 apply_physics.\ \ It produces predictions (JSONL), an HTML report, and a simulation-ready USD\ \ with UsdPhysics schemas applied for each prim in the scene.\n\n## Highlights\n\n\ * End-to-end pipeline\n * POST a USD file (multipart) or an S3 URI to start the pipeline.\n\ \ * Two-step upload supported via POST /pipeline/upload-usd followed by POST /pipeline with session_id.\n\ \ * Produces predictions.jsonl, an HTML classification report, and a simulation-ready\ \ USD (scene_physics.) with UsdPhysics RigidBodyAPI / CollisionAPI\ \ / MassAPI / MaterialAPI authored on each predicted prim.\n* Flexible\ \ configuration\n * Server-side defaults control rendering modes, VLM prompts,\ \ and prediction settings.\n * Supports multiple rendering modes (prim_only, composition,\ \ etc.).\n * Configurable VLM backend and model.\n * Selectable rendering\ \ backend via API parameter:\n * `remote` (default in the bundled Docker\ \ Compose) \u2014 HTTP render service. The public compose points this at the OVRTX\ \ sidecar, so the main service stays CPU-only.\n * `warp` \u2014 In-process\ \ CUDA GPU raytracer. Fast (~29ms/frame), no Vulkan/DISPLAY needed.\n *\ \ `ovrtx` \u2014 Local RTX path tracer via OvRTX subprocess. PBR quality, requires\ \ Vulkan.\n\n* Real-time monitoring\n * Server-Sent Events (SSE) stream provides\ \ live updates with:\n * Current step, state, per-step progress, and overall\ \ percentage.\n * Automatic reconnection; polling fallback available via status\ \ endpoint.\n * Cancel running pipelines via a dedicated endpoint.\n\n* Visual\ \ feedback\n * Live preview gallery during the rendering stage.\n * On-demand\ \ HTML report generation with per-prim classifications.\n" version: 0.5.2 paths: /pipeline/upload-usd: post: tags: - pipeline summary: Upload Usd Immediate description: Upload a USD file and create a session for later pipeline execution. operationId: upload_usd_immediate_pipeline_upload_usd_post requestBody: content: multipart/form-data: schema: $ref: '#/components/schemas/Body_upload_usd_immediate_pipeline_upload_usd_post' responses: '201': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/SessionCreated' '403': description: Client S3 URI rejected by the configured bucket allowlist, or S3 access denied. content: application/json: schema: $ref: '#/components/schemas/ErrorDetail' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /pipeline: post: tags: - pipeline summary: Create Pipeline description: | Create and execute a physics agent pipeline. At least one input source is required. When more than one is supplied, the selected source is deterministic: `session_id` first, then `s3_uri`, then `usd_file`. Lower-priority source fields are ignored. operationId: create_pipeline_pipeline_post requestBody: content: multipart/form-data: schema: $ref: '#/components/schemas/Body_create_pipeline_pipeline_post' responses: '202': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/SessionCreated' '403': description: Client S3 URI rejected by the configured bucket allowlist, or S3 access denied. content: application/json: schema: $ref: '#/components/schemas/ErrorDetail' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /pipeline/{session_id}/status: get: tags: - pipeline summary: Get Pipeline Status description: 'Get pipeline execution status with detailed progress. Reads from in-memory event bus state for fast, real-time accuracy. Falls back to store-based SessionManager for completed/cross-instance sessions.' operationId: get_pipeline_status_pipeline__session_id__status_get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/PipelineStatus' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /pipeline/{session_id}/results: get: tags: - pipeline summary: Get Pipeline Results description: Get terminal pipeline execution results. operationId: get_pipeline_results_pipeline__session_id__results_get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Successful Response content: application/json: schema: anyOf: - $ref: '#/components/schemas/PipelineResults' - $ref: '#/components/schemas/PipelineError' title: Response Get Pipeline Results Pipeline Session Id Results Get '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /pipeline/{session_id}/cancel: post: tags: - pipeline summary: Cancel Pipeline description: 'Cancel a running pipeline. Works cross-instance: writes a cancel signal to the store (S3) so the executing instance can detect it. Also tries local cancellation.' operationId: cancel_pipeline_pipeline__session_id__cancel_post parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Successful Response content: application/json: schema: {} '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /pipeline/{session_id}/events: get: tags: - pipeline summary: Stream Progress Events description: 'Stream real-time progress events via Server-Sent Events (SSE). Only works when connected to the instance running the pipeline. For cross-instance progress, use GET /pipeline/{session_id}/status (polling).' operationId: stream_progress_events_pipeline__session_id__events_get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Successful Response content: application/json: schema: {} '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /pipeline/{session_id}/regenerate: post: tags: - pipeline summary: Regenerate Pipeline description: Regenerate specific pipeline steps from cached data. operationId: regenerate_pipeline_pipeline__session_id__regenerate_post parameters: - name: session_id in: path required: true schema: type: string title: Session Id requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/RegenerateRequest' responses: '202': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/SessionCreated' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /predict: post: tags: - predict summary: Create Predict description: | Create and execute a prediction-only job. Distinct from `/pipeline`: runs prediction (and the minimum upstream prep, if needed), but exposes a separate session lifecycle and result endpoints. Use `/pipeline` for the full classify/apply flow. Two input modes (auto-detected at job start): * **Mode A — dataset already prepared.** Triggered when `dataset_path` points at a readable dataset.jsonl, or when an existing `session_id` has `cache/dataset/dataset.jsonl` whose referenced images resolve. Only the predict step runs. * **Mode B — USD upload / s3_uri / fresh session_id.** When no runnable session dataset is present, /predict runs the minimum upstream steps (`optimize_usd` if enabled → `identify_asset` → `build_dataset_usd` → `build_dataset_prepare_dataset`) before predicting. `apply_physics` is intentionally not part of /predict — use POST /pipeline if you need the full classify/apply flow. A cached JSONL whose images do not resolve falls back to Mode B when an input USD is available. Without an input USD, the request returns HTTP 400 with recovery guidance. The detected mode is persisted to session metadata under `predict_mode` and surfaced in the `GET /predict/{id}/results` response. operationId: create_predict_predict_post requestBody: content: multipart/form-data: schema: $ref: '#/components/schemas/Body_create_predict_predict_post' responses: '202': description: Predict job accepted and queued for execution. content: application/json: schema: $ref: '#/components/schemas/SessionCreated' '400': description: Missing/invalid input — no usd_file/session_id/s3_uri/dataset_path, ambiguous combinations (e.g. usd_file + s3_uri, dataset_path + s3_uri), invalid USD extension, missing dataset, unknown Mode-B render backend, invalid optimizer config, or a cached dataset whose images are unavailable with no input USD to rebuild from. '403': description: dataset_path resolves outside allowed roots; client S3 URI rejected by the configured bucket allowlist; or S3 access denied. content: application/json: schema: $ref: '#/components/schemas/ErrorDetail' '404': description: Session not found, or S3 object not found. '409': description: Predict is already pending/running/cancelling for the supplied session_id, OR the same-pod JobRegistry rerun race-guard fired (the slot was reserved by a concurrent /predict on this instance). Wait for a terminal state or cancel first. '413': description: Upload or S3 file exceeds the configured size limit. '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' '500': description: Internal failure staging the upload, dataset, or session. '502': description: Failed to download from S3 upstream. /predict/{session_id}/status: get: tags: - predict summary: Get Predict Status description: Get predict execution status with detailed progress. Reuses the same response schema as `/pipeline/{id}/status`. operationId: get_predict_status_predict__session_id__status_get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/PipelineStatus' '404': description: Session not found '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /predict/{session_id}/results: get: tags: - predict summary: Get Predict Results description: | Get predict execution results (only available when completed). Surfaces `predictions_count`, `failed_count`, `predictions_path`, and `token_stats` directly (not buried inside `stats`) so they line up with the source-of-truth Python `PredictOutput`. operationId: get_predict_results_predict__session_id__results_get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Predict completed successfully. content: application/json: schema: anyOf: - $ref: '#/components/schemas/PredictResults' - $ref: '#/components/schemas/PipelineError' title: Response Get Predict Results Predict Session Id Results Get '202': description: Predict still pending or running. '404': description: Session not found '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /predict/{session_id}/cancel: post: tags: - predict summary: Cancel Predict description: Cancel a running predict job. operationId: cancel_predict_predict__session_id__cancel_post parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Cancellation requested. content: application/json: schema: {} '400': description: Job not in a cancellable state. '404': description: Session not found '409': description: Session is not a predict session — it was created via /pipeline or /pipeline/upload-usd. Use POST /pipeline/{session_id}/cancel instead. '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /predict/{session_id}/events: get: tags: - predict summary: Stream Predict Events description: Stream real-time predict progress events via SSE. Only works when connected to the executing instance; for cross-instance progress, poll GET /predict/{session_id}/status. operationId: stream_predict_events_predict__session_id__events_get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Successful Response content: text/event-stream: schema: type: string description: Server-Sent Events stream of predict progress. '404': description: Session not found '503': description: Predict is running on a different instance; use polling. '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /pipeline/{session_id}/event-log: get: tags: - pipeline summary: Get Event Log description: Get the persisted event log for a session. operationId: get_event_log_pipeline__session_id__event_log_get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Successful Response content: application/json: schema: {} '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /artifacts/{session_id}/predictions: get: tags: - artifacts summary: Download Predictions description: Download predictions JSONL file. operationId: download_predictions_artifacts__session_id__predictions_get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Successful Response content: application/json: schema: {} '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /artifacts/{session_id}/report: get: tags: - artifacts summary: View Prediction Report description: 'View prediction HTML report in browser. Generates the report on-demand if it doesn''t exist yet.' operationId: view_prediction_report_artifacts__session_id__report_get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Successful Response content: application/json: schema: {} '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /artifacts/{session_id}/dataset: get: tags: - artifacts summary: Download Dataset description: Download dataset JSONL file. operationId: download_dataset_artifacts__session_id__dataset_get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Successful Response content: application/json: schema: {} '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /artifacts/{session_id}/output-usd: get: tags: - artifacts summary: Download Output USD description: 'Download the simulation-ready USD written by the apply_physics step. Returned only when the pipeline has completed with apply_physics enabled (the service default). USD, USDA, and USDC inputs are returned as scene_physics.usd, scene_physics.usda, or scene_physics.usdc. USDZ inputs default to scene_physics.usda so Omniverse MDL shader references can remain as runtime-resolved asset paths instead of being bundled into a new USDZ package. UsdPhysics schemas (RigidBodyAPI, CollisionAPI, MassAPI, MaterialAPI) are authored on each predicted prim, plus a PhysicsScene. Consumable by PhysX / Isaac.' operationId: download_output_usd_artifacts__session_id__output_usd_get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Successful Response content: application/octet-stream: schema: type: string format: binary model/vnd.usdz+zip: schema: type: string format: binary text/plain: schema: type: string format: binary '404': description: Session or output USD not found '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /sessions: get: tags: - sessions summary: List Sessions description: 'List all sessions with metadata. Returns sessions from the store (S3 or local), so all instances see the same list.' operationId: list_sessions_sessions_get responses: '200': description: Successful Response content: application/json: schema: {} /sessions/{session_id}: get: tags: - sessions summary: Get Session description: Get detailed session information. operationId: get_session_sessions__session_id__get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Successful Response content: application/json: schema: {} '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' delete: tags: - sessions summary: Delete Session description: Delete a session and all its artifacts. operationId: delete_session_sessions__session_id__delete parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '204': description: Successful Response '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /tune: post: tags: - tune summary: Create Tune Session description: | Create and queue a tuning session over an authored physics USD. Exactly one of `physics_usd`, `s3_uri`, or `source_session_id` must be provided. `scenario_yaml` is the body of a tuning scenario YAML (drop_settle in v1). operationId: create_tune_tune_post requestBody: content: multipart/form-data: schema: $ref: '#/components/schemas/Body_create_tune_tune_post' responses: '202': description: Tune queued for execution content: application/json: schema: $ref: '#/components/schemas/SessionCreated' '400': description: Invalid scenario or input USD '403': description: Client S3 URI rejected by the configured bucket allowlist, or S3 access denied. content: application/json: schema: $ref: '#/components/schemas/ErrorDetail' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /tune/{session_id}/status: get: tags: - tune summary: Get Tune Status operationId: get_tune_status_tune__session_id__status_get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Tune status content: application/json: schema: $ref: '#/components/schemas/TuneStatus' '404': description: Session not found /tune/{session_id}/results: get: tags: - tune summary: Get Tune Results operationId: get_tune_results_tune__session_id__results_get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Tune results when completed (TuneResults), or PipelineError payload when the run failed. content: application/json: schema: anyOf: - $ref: '#/components/schemas/TuneResults' - $ref: '#/components/schemas/PipelineError' title: Response Get Tune Results Tune Session Id Results Get '202': description: Tune still running '404': description: Session not found /tune/{session_id}/events: get: tags: - tune summary: Stream Tune Events description: Server-Sent Events stream of trial-level progress. operationId: stream_tune_events_tune__session_id__events_get parameters: - name: session_id in: path required: true schema: type: string responses: '200': description: text/event-stream content: text/event-stream: schema: type: string format: text description: | SSE stream of tune lifecycle events. Each frame is a JSON-encoded ProgressEvent. Stream terminates on COMPLETED / FAILED / CANCELLED / tune_ready. '404': description: Session not found /tune/{session_id}/cancel: post: tags: - tune summary: Cancel Tune operationId: cancel_tune_tune__session_id__cancel_post parameters: - name: session_id in: path required: true schema: type: string responses: '200': description: Cancellation requested '400': description: Cannot cancel from current status '404': description: Session not found '409': description: Session is not a tune session — its metadata is missing an explicit `kind == "tune"` discriminator. Sessions created via `/pipeline` or `/predict` (or via the bare `SessionManager.create_session()` default) cannot be cancelled through this route; use POST /pipeline/{session_id}/cancel or POST /predict/{session_id}/cancel instead. /tune/{session_id}/artifacts/{name}: get: tags: - tune summary: Download Tune Artifact description: | Download one of the canonical tune artifacts: `best_params.json`, `tune_results.json`, `history.jsonl`, `report.md`, `tuned_physics.usd`, or the optional visual judge contact sheet `comparison.png`. operationId: download_tune_artifact_tune__session_id__artifacts__name__get parameters: - name: session_id in: path required: true schema: type: string - name: name in: path required: true schema: type: string responses: '200': description: Artifact bytes content: application/octet-stream: schema: type: string format: binary description: | Raw artifact bytes. Concrete content type depends on the requested artifact: `best_params.json` / `tune_results.json` are JSON, `history.jsonl` is JSON-lines, `report.md` is markdown, `tuned_physics.usd` is a USD layer, and `comparison.png` is a PNG image when visual judging ran. Generated SDKs should treat the response as a binary download. '404': description: Unknown artifact or session not found /refine: post: tags: - refine summary: Create Refine Session description: | Create and queue an iterative refine session over an authored physics USD. Exactly one of `physics_usd`, `s3_uri`, or `source_session_id` must be provided. Both `scenario_yaml` and `user_prompt` are required: the YAML defines the initial tuning scenario and the prompt defines the target behavior for judge/refiner calls. Deployments build the judge/refiner models server-side from service configuration. operationId: create_refine_refine_post requestBody: content: multipart/form-data: schema: $ref: '#/components/schemas/Body_create_refine_refine_post' responses: '202': description: Refine queued for execution content: application/json: schema: $ref: '#/components/schemas/SessionCreated' '400': description: Invalid scenario, optimizer, source, or input USD '403': description: Client S3 URI rejected by the configured bucket allowlist, or S3 access denied. content: application/json: schema: $ref: '#/components/schemas/ErrorDetail' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /refine/{session_id}/status: get: tags: - refine summary: Get Refine Status operationId: get_refine_status_refine__session_id__status_get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Refine status content: application/json: schema: $ref: '#/components/schemas/RefineStatus' '404': description: Session not found '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /refine/{session_id}/results: get: tags: - refine summary: Get Refine Results operationId: get_refine_results_refine__session_id__results_get parameters: - name: session_id in: path required: true schema: type: string title: Session Id responses: '200': description: Refine results when terminal (RefineResults), or PipelineError payload when the run failed before partial artifacts were available. content: application/json: schema: anyOf: - $ref: '#/components/schemas/RefineResults' - $ref: '#/components/schemas/PipelineError' title: Response Get Refine Results Refine Session Id Results Get '202': description: Refine still running '404': description: Session not found '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' /refine/{session_id}/events: get: tags: - refine summary: Stream Refine Events description: Server-Sent Events stream of iteration, trial, and judge progress. operationId: stream_refine_events_refine__session_id__events_get parameters: - name: session_id in: path required: true schema: type: string responses: '200': description: text/event-stream content: text/event-stream: schema: type: string format: text description: | SSE stream of refine lifecycle events. Each frame is a JSON-encoded ProgressEvent. Stream terminates on FAILED / CANCELLED / refine_ready. '404': description: Session not found '503': description: Refine is running on a different instance; use polling /refine/{session_id}/cancel: post: tags: - refine summary: Cancel Refine operationId: cancel_refine_refine__session_id__cancel_post parameters: - name: session_id in: path required: true schema: type: string responses: '200': description: Cancellation requested '400': description: Cannot cancel from current status '404': description: Session not found '409': description: Session is not a refine session /refine/{session_id}/artifacts/{name}: get: tags: - refine summary: Download Refine Artifact description: | Download one of the canonical refine artifacts: `refine_summary.json`, `final/scenario.yaml`, `final/best_params.json`, `final/tune_results.json`, `final/history.jsonl`, `final/judge_result.json`, `final/tuned_physics.usd`, `final/recording.usd`, `final/report.md`, or `final/comparison.png`. operationId: download_refine_artifact_refine__session_id__artifacts__name__get parameters: - name: session_id in: path required: true schema: type: string - name: name in: path required: true schema: type: string responses: '200': description: Artifact bytes content: application/octet-stream: schema: type: string format: binary description: Raw artifact bytes. Concrete content type depends on the requested artifact. '404': description: Unknown artifact or session not found /health: get: summary: Health Check description: Health check endpoint. operationId: health_check_health_get responses: '200': description: Successful Response content: application/json: schema: {} /api: get: summary: Root Api Info description: Root endpoint with service info. operationId: root_api_info_api_get responses: '200': description: Successful Response content: application/json: schema: {} /: get: summary: Root description: Root endpoint redirects to API info. operationId: root__get responses: '200': description: Successful Response content: application/json: schema: {} components: schemas: Body_create_pipeline_pipeline_post: properties: usd_file: type: string contentMediaType: application/octet-stream title: Usd File description: USD file to process. Lowest-priority source; used only when neither session_id nor s3_uri is provided. session_id: type: string title: Session Id description: Existing session ID (from /upload-usd endpoint). Highest-priority source when multiple source fields are supplied. s3_uri: type: string title: S3 Uri description: S3 URI to a USD file (e.g. s3://bucket/path/scene.usdz). Used when session_id is absent, ahead of usd_file. user_prompt: type: string title: User Prompt description: Custom user prompt for VLM (optional) default: '' render_backend: type: string title: Render Backend description: 'Rendering backend: ''remote'' (default, HTTP render service; the bundled compose points this at the OVRTX sidecar), ''warp'' (local CUDA), ''ovrtx'' (local Vulkan subprocess), or ''mock'' (deterministic CPU-only test images)' enum: [remote, warp, ovrtx, mock, ""] default: '' optimize_usd: type: boolean title: Optimize Usd description: 'Enable USD optimization step (default: false). When enabled, runs Scene Optimizer before rendering/prediction and restore_usd afterward to map results back to original paths.' default: false enable_deinstance: type: boolean title: Enable Deinstance description: 'Enable deinstance operation when optimize_usd is true (default: true). Required for instanced USD assets (e.g. robot arms with shared prototypes). FastAPI accepts common boolean form values such as true/false, 1/0, yes/no, and on/off.' default: true enable_split: type: boolean title: Enable Split description: 'Enable split meshes operation when optimize_usd is true (default: false).' default: false enable_deduplicate: type: boolean title: Enable Deduplicate description: 'Enable deduplicate operation when optimize_usd is true (default: false).' default: false type: object title: Body_create_pipeline_pipeline_post description: At least one source is required. Source precedence is session_id, then s3_uri, then usd_file. Body_create_predict_predict_post: properties: usd_file: type: string contentMediaType: application/octet-stream title: Usd File description: USD file to predict on (Mode B). Optional if dataset_path, session_id, or s3_uri is provided. session_id: type: string title: Session Id description: Existing session ID (e.g. from POST /pipeline/upload-usd). When the session has a prepared dataset whose referenced images resolve, /predict runs Mode A (predict only). s3_uri: type: string title: S3 Uri description: S3 URI to a USD file (e.g. s3://bucket/path/scene.usdz). dataset_path: type: string title: Dataset Path description: Absolute path to a prepared dataset.jsonl on the server. When set and readable, forces Mode A (predict-only). user_prompt: type: string title: User Prompt description: Custom user prompt for VLM (optional, used in Mode B). default: '' render_backend: type: string title: Render Backend description: 'Rendering backend for Mode B: ''remote'' (default), ''warp'', ''ovrtx'', or ''mock''. Ignored in Mode A.' enum: [remote, warp, ovrtx, mock, ""] default: '' optimize_usd: type: boolean title: Optimize Usd description: Enable USD optimization step in Mode B. default: false enable_deinstance: type: boolean title: Enable Deinstance description: Enable deinstance op when optimize_usd=true (Mode B only). default: true enable_split: type: boolean title: Enable Split description: Enable split-meshes op when optimize_usd=true (Mode B only). default: false enable_deduplicate: type: boolean title: Enable Deduplicate description: Enable deduplicate op when optimize_usd=true (Mode B only). default: false type: object title: Body_create_predict_predict_post PredictResults: properties: session_id: type: string title: Session Id status: type: string title: Status mode: type: string title: Mode description: 'Detected predict mode: ''dataset_only'' or ''full_predict''.' steps_run: items: type: string type: array title: Steps Run description: Pipeline steps the predict route actually drove. stats: additionalProperties: true type: object title: Stats description: Execution statistics predictions_count: type: integer title: Predictions Count description: Number of predictions produced (from PredictOutput). default: 0 failed_count: type: integer title: Failed Count description: Number of failed predictions (from PredictOutput). default: 0 predictions_path: anyOf: - type: string - type: 'null' title: Predictions Path description: Server-side path to predictions.jsonl on disk. token_stats: additionalProperties: true type: object title: Token Stats description: VLM token usage statistics when available. download_urls: additionalProperties: type: string type: object title: Download Urls description: URLs to download artifacts (predictions, report, dataset). duration_seconds: type: integer title: Duration Seconds description: Total predict duration in seconds completed_at: type: string title: Completed At description: ISO timestamp when completed type: object required: - session_id - status - mode - duration_seconds - completed_at title: PredictResults description: Predict-only execution results. Body_upload_usd_immediate_pipeline_upload_usd_post: properties: usd_file: type: string contentMediaType: application/octet-stream title: Usd File description: USD file to upload (provide this OR s3_uri) s3_uri: type: string title: S3 Uri description: S3 URI to a USD file (e.g. s3://bucket/path/scene.usdz) type: object title: Body_upload_usd_immediate_pipeline_upload_usd_post CompletedStepInfo: properties: name: type: string title: Name description: Step internal name display_name: type: string title: Display Name description: Human-readable step name started_at: type: string title: Started At description: ISO timestamp when step started completed_at: type: string title: Completed At description: ISO timestamp when step completed duration_seconds: type: integer title: Duration Seconds description: Step duration in seconds stats: additionalProperties: true type: object title: Stats description: Step-specific statistics type: object required: - name - display_name - started_at - completed_at - duration_seconds title: CompletedStepInfo description: Information about a completed step. CurrentStepInfo: properties: name: type: string title: Name description: Step internal name display_name: type: string title: Display Name description: Human-readable step name started_at: type: string title: Started At description: ISO timestamp when step started progress: $ref: '#/components/schemas/StepProgress' elapsed_seconds: type: integer title: Elapsed Seconds description: Seconds since step started type: object required: - name - display_name - started_at - progress - elapsed_seconds title: CurrentStepInfo description: Information about the currently executing step. ErrorDetail: properties: detail: type: string title: Detail description: Human-readable error detail type: object required: - detail title: ErrorDetail description: Standard FastAPI error payload returned by authorization failures. HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError OverallProgress: properties: current_step: type: integer title: Current Step description: Current step number (1-indexed) total_steps: type: integer title: Total Steps description: Total number of steps percent: type: integer title: Percent description: Overall percentage complete (0-100) estimated_remaining_seconds: anyOf: - type: integer - type: 'null' title: Estimated Remaining Seconds description: Estimated seconds until completion type: object required: - current_step - total_steps - percent title: OverallProgress description: Overall pipeline progress. PipelineError: properties: session_id: type: string title: Session Id status: type: string title: Status default: failed error_message: type: string title: Error Message description: Stable value-free diagnostic code failed_step: type: string title: Failed Step description: Step that failed completed_steps: items: type: string type: array title: Completed Steps description: Steps completed before failure partial_results: anyOf: - additionalProperties: true type: object - type: 'null' title: Partial Results description: Partial results if available type: object required: - session_id - error_message - failed_step title: PipelineError description: Pipeline error response. PipelineResults: properties: session_id: type: string title: Session Id status: type: string title: Status stats: additionalProperties: true type: object title: Stats description: Execution statistics examples: - images_generated: 284 predictions_made: 142 prims_processed: 142 download_urls: additionalProperties: type: string type: object title: Download Urls description: URLs to download artifacts examples: - predictions: /artifacts/abc123/predictions report: /artifacts/abc123/report dataset: /artifacts/abc123/dataset output_usd: /artifacts/abc123/output-usd duration_seconds: type: integer title: Duration Seconds description: Total pipeline duration in seconds completed_at: type: string title: Completed At description: ISO timestamp when completed type: object required: - session_id - status - duration_seconds - completed_at title: PipelineResults description: Pipeline execution results. PipelineStatus: properties: session_id: type: string title: Session Id status: type: string title: Status description: 'Current status: pending, running, completed, failed, cancelled, cancelling' current_step: anyOf: - $ref: '#/components/schemas/CurrentStepInfo' - type: 'null' completed_steps: items: $ref: '#/components/schemas/CompletedStepInfo' type: array title: Completed Steps overall_progress: $ref: '#/components/schemas/OverallProgress' preview_images: items: type: string type: array title: Preview Images description: URLs to preview images can_cancel: type: boolean title: Can Cancel description: Whether pipeline can be cancelled elapsed_seconds: type: integer title: Elapsed Seconds description: Total elapsed time in seconds created_at: type: string title: Created At description: ISO timestamp when session created updated_at: type: string title: Updated At description: ISO timestamp of last update type: object required: - session_id - status - overall_progress - can_cancel - elapsed_seconds - created_at - updated_at title: PipelineStatus description: Enhanced pipeline execution status with progress. PipelineStep: type: string enum: - optimize_usd - identify_asset - build_dataset_usd - build_dataset_prepare_dataset - predict - restore_usd title: PipelineStep description: Available pipeline steps. RegenerateRequest: properties: steps: items: $ref: '#/components/schemas/PipelineStep' type: array title: Steps description: Steps to re-run from cache user_prompt: anyOf: - type: string - type: 'null' title: User Prompt description: Override user prompt for regeneration type: object required: - steps title: RegenerateRequest description: Request to regenerate specific steps from cache. SessionCreated: properties: session_id: type: string title: Session Id status: type: string title: Status default: pending message: type: string title: Message default: Pipeline queued for execution estimated_duration_minutes: anyOf: - type: integer - type: 'null' title: Estimated Duration Minutes description: Estimated completion time type: object required: - session_id title: SessionCreated description: Response when session is created. StepProgress: properties: current: type: integer title: Current description: Current progress count total: type: integer title: Total description: Total items to process percent: type: integer title: Percent description: Percentage complete (0-100) message: type: string title: Message description: Human-readable progress message type: object required: - current - total - percent - message title: StepProgress description: Progress information for a single step. ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type input: title: Input ctx: type: object title: Context type: object required: - loc - msg - type title: ValidationError Body_create_tune_tune_post: properties: physics_usd: type: string contentMediaType: application/octet-stream title: Physics Usd description: Physics-authored USD (output of apply_physics) to tune s3_uri: type: string title: S3 Uri description: S3 URI to a physics-authored USD source_session_id: type: string title: Source Session Id description: Pipeline session ID — copy its apply_physics output_usd reference_images: items: type: string contentMediaType: application/octet-stream type: array title: Reference Images default: [] description: Optional reference images for the visual/VLM judge reference_videos: items: type: string contentMediaType: application/octet-stream type: array title: Reference Videos default: [] description: Optional reference videos for the visual/VLM judge reference_descriptions: type: string title: Reference Descriptions default: '' description: Optional JSON array of descriptions parallel to reference_images reference_video_descriptions: type: string title: Reference Video Descriptions default: '' description: Optional JSON array of descriptions parallel to reference_videos reference_video_frames: type: integer title: Reference Video Frames maximum: 64 minimum: 1 default: 8 description: Frames to extract from each reference video for visual judging judge_reference_frames: type: integer title: Judge Reference Frames maximum: 64 minimum: 1 default: 8 description: Max reference images/video frames to send to the VLM judge judge_generated_frames: type: integer title: Judge Generated Frames maximum: 64 minimum: 1 default: 16 description: Max generated render frames to send to the VLM judge scenario_yaml: type: string title: Scenario Yaml default: '' description: >- Scenario YAML body (drop_settle etc.). Optional when `user_prompt` is supplied — the NL interpreter authors a Scenario in that case. When both are supplied, explicit YAML fields override interpreter output. user_prompt: type: string title: User Prompt default: '' description: >- Free-form NL description of the desired tune run (e.g. "make this object bouncy"). Persisted to tune_results.json['user_prompt'] and rendered into report.md for audit. Mirrors the user_prompt field on /pipeline. optimizer: type: string title: Optimizer default: auto description: 'auto (=botorch), botorch, random, cma-es' engine: type: string title: Engine default: ovphysx enum: - ovphysx - newton - fake description: 'Tuning engine: ovphysx (PhysX 5 daemon, production), newton (NVIDIA Newton GPU/MuJoCo-warp; requires the apps/physics_agent[newton] extra; supports contact_ke/contact_kd bounce tuning; no static_friction or restitution tuning yet), or fake (tests).' max_trials: type: integer title: Max Trials default: 30 description: Optimizer trial budget seed: type: integer title: Seed default: 42 description: Seed for optimizer + backend enable_judge: type: boolean title: Enable Judge default: true description: >- Run the VLM-as-judge over scenario/history/best_params at the end of tune (default on). Set to false for byte-identical output to the pre-Part-1.1 baseline (no model calls). judge_max_iterations: type: integer title: Judge Max Iterations default: 3 minimum: 1 maximum: 10 description: >- Hard cap on refine-loop iterations when the judge returns 'continue'. v1.1 ships single-iteration judging; values > 1 are forward-compat plumbing and currently emit a tune.judge.refine_skipped event without re-tuning. judge_max_tokens: anyOf: - type: integer minimum: 1 - type: 'null' title: Judge Max Tokens description: >- Optional max output tokens for judge responses. Defaults to the physics judge configuration. judge_temperature: anyOf: - type: number minimum: 0 - type: 'null' title: Judge Temperature description: >- Optional temperature for judge calls. Defaults to the scenario judge block or physics judge configuration. type: object title: Body_create_tune_tune_post description: >- Either `scenario_yaml` or `user_prompt` must be supplied (both empty returns 400). When both are supplied the explicit YAML wins on every conflict and the NL interpreter only fills gaps. Exactly one of `physics_usd`, `s3_uri`, `source_session_id` must be supplied as the asset source — the schema enforces this via `oneOf` so generated clients reject combinations the server otherwise rejects with 400 only at runtime (CodeRabbit Round 11 thread #2). oneOf: - required: - physics_usd - required: - s3_uri - required: - source_session_id Body_create_refine_refine_post: properties: physics_usd: type: string contentMediaType: application/octet-stream title: Physics Usd description: Physics-authored USD (output of apply_physics) to refine s3_uri: type: string title: S3 Uri description: S3 URI to a physics-authored USD source_session_id: type: string title: Source Session Id description: Pipeline session ID whose apply_physics output_usd should be used reference_images: items: type: string contentMediaType: application/octet-stream type: array title: Reference Images default: [] description: Optional reference images for the visual/VLM judge reference_videos: items: type: string contentMediaType: application/octet-stream type: array title: Reference Videos default: [] description: Optional reference videos for the visual/VLM judge reference_descriptions: type: string title: Reference Descriptions default: '' description: Optional JSON array of descriptions parallel to reference_images reference_video_descriptions: type: string title: Reference Video Descriptions default: '' description: Optional JSON array of descriptions parallel to reference_videos reference_video_frames: type: integer title: Reference Video Frames maximum: 64 minimum: 1 default: 8 description: Frames to extract from each reference video for visual judging judge_reference_frames: type: integer title: Judge Reference Frames maximum: 64 minimum: 1 default: 8 description: Max reference images/video frames to send to the VLM judge judge_generated_frames: type: integer title: Judge Generated Frames maximum: 64 minimum: 1 default: 16 description: Max generated render frames to send to the VLM judge scenario_yaml: type: string title: Scenario Yaml default: '' description: Initial refine scenario YAML body user_prompt: type: string title: User Prompt default: '' description: Natural-language desired behavior for judge/refine loop optimizer: type: string title: Optimizer default: botorch description: botorch, auto, random, cma-es engine: type: string title: Engine default: ovphysx description: ovphysx, newton, or fake max_trials: type: integer title: Max Trials maximum: 1000 minimum: 1 default: 30 description: Optimizer trial budget per iteration max_iterations: type: integer title: Max Iterations maximum: 12 minimum: 1 default: 5 description: Refine iteration cap score_threshold: type: number title: Score Threshold maximum: 1.0 minimum: 0.0 default: 0.9 description: Judge approval threshold seed: type: integer title: Seed default: 42 description: Seed for optimizer + backend judge_max_tokens: anyOf: - type: integer minimum: 1 - type: 'null' title: Judge Max Tokens description: Optional max output tokens for judge responses judge_temperature: anyOf: - type: number minimum: 0.0 - type: 'null' title: Judge Temperature description: Optional temperature for judge calls visual_evidence_enabled: type: boolean title: Visual Evidence Enabled default: true description: Send generated/reference media to the VLM judge llm_timeout_seconds: type: number title: Llm Timeout Seconds default: 180.0 description: Wall-clock deadline for each judge/refine LLM call required: - scenario_yaml - user_prompt type: object title: Body_create_refine_refine_post description: >- Both `scenario_yaml` and `user_prompt` must be supplied. Exactly one of `physics_usd`, `s3_uri`, `source_session_id` must be supplied as the asset source. oneOf: - required: - physics_usd - required: - s3_uri - required: - source_session_id TuneStatus: properties: session_id: type: string status: type: string n_trials: type: integer default: 0 max_trials: type: integer default: 0 best_score: anyOf: - type: number - type: 'null' best_params: anyOf: - type: object additionalProperties: type: number - type: 'null' elapsed_seconds: type: integer can_cancel: type: boolean error_message: anyOf: - type: string - type: 'null' description: Failure reason for terminal failed tune sessions. created_at: type: string updated_at: type: string required: - session_id - status - elapsed_seconds - can_cancel - created_at - updated_at type: object title: TuneStatus TuneResults: properties: session_id: type: string status: type: string best_params: type: object additionalProperties: type: number best_score: # Nullable: a tune cancelled before the first trial completed # (or whose backend never produced a finite score) returns # ``best_score: null`` via the R12+R15 ``_coerce_finite_score`` # path. Mirrors ``TuneStatus.best_score``'s schema so generated # clients accept both shapes (kimbyn 2026-05-12). anyOf: - type: number - type: 'null' n_trials: type: integer optimizer_used: type: string engine_used: type: string download_urls: type: object additionalProperties: type: string description: URLs for tune artifacts that were produced and published. duration_seconds: type: integer completed_at: type: string error_message: anyOf: - type: string - type: 'null' description: Failure reason for terminal failed sessions that still expose partial tune results and artifact URLs. required: - session_id - status - best_score - n_trials - optimizer_used - engine_used - duration_seconds - completed_at type: object title: TuneResults RefineStatus: properties: session_id: type: string status: type: string description: pending, running, completed, failed, cancelled, cancelling iteration: type: integer default: 0 description: Current or final iteration number max_iterations: type: integer default: 0 description: Configured iteration cap n_trials: type: integer default: 0 description: Trials completed in current iteration max_trials: type: integer default: 0 description: Configured trial budget per iteration best_score: anyOf: - type: number - type: 'null' description: Best score observed in current/final iteration best_params: anyOf: - type: object additionalProperties: type: number - type: 'null' description: Best parameter set observed in current/final iteration judge_score: anyOf: - type: number - type: 'null' description: Latest judge score, when available termination_reason: anyOf: - type: string - type: 'null' description: approved, max_iterations, cancelled, error, or null while running elapsed_seconds: type: integer description: Total elapsed time in seconds can_cancel: type: boolean description: Whether refine can be cancelled error_message: anyOf: - type: string - type: 'null' description: Failure reason for terminal failed refine sessions. created_at: type: string updated_at: type: string required: - session_id - status - elapsed_seconds - can_cancel - created_at - updated_at type: object title: RefineStatus RefineResults: properties: session_id: type: string status: type: string termination_reason: type: string iteration_count: type: integer final_iteration: type: integer final_judge_score: anyOf: - type: number - type: 'null' iterations: items: type: object type: array default: [] download_urls: type: object additionalProperties: type: string description: URLs for refine artifacts that were produced and published. duration_seconds: type: integer completed_at: type: string error_message: anyOf: - type: string - type: 'null' required: - session_id - status - termination_reason - iteration_count - final_iteration - duration_seconds - completed_at type: object title: RefineResults