W1030 09:33:52.150000 1899544 torch/_opaque_base.py:6] torch._opaque_base is deprecated, use torch._custom_class_base instead
W1030 09:33:52.150000 1899544 torch/_library/opaque_object.py:288] register_opaque_type is deprecated, use register_custom_class instead
W1030 09:33:52.151000 1899544 torch/_library/opaque_object.py:211] typ='value' is deprecated, use typ='constant' instead
(APIServer pid=1899544) INFO 10-30 09:33:58 [startup.py:38] 
(APIServer pid=1899544) INFO 10-30 09:33:58 [startup.py:38]        █     █     █▄   ▄█
(APIServer pid=1899544) INFO 10-30 09:33:58 [startup.py:38]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.32.1rc1.dev577+geb703fbb3
(APIServer pid=1899544) INFO 10-30 09:33:58 [startup.py:38]   █▄█▀ █     █     █     █  model   unsloth/functiongemma-270m-it
(APIServer pid=1899544) INFO 10-30 09:33:58 [startup.py:38]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1899544) INFO 10-30 09:33:58 [startup.py:38] 
(APIServer pid=1899544) INFO 10-30 09:33:58 [startup.py:96] non-default args: {'model_tag': 'unsloth/functiongemma-270m-it', 'enable_auto_tool_choice': True, 'tool_call_parser': 'functiongemma', 'port': 8124, 'model': 'unsloth/functiongemma-270m-it', 'max_model_len': 2048, 'enforce_eager': True, 'served_model_name': ['functiongemma'], 'gpu_memory_utilization': 0.04, 'max_num_seqs': 4}
(APIServer pid=1899544) INFO 10-30 09:33:59 [model.py:744] Resolved architecture: Gemma3ForCausalLM
(APIServer pid=1899544) INFO 10-30 09:33:59 [model.py:2170] Using max model len 2048
(APIServer pid=1899544) INFO 10-30 09:33:59 [scheduler.py:310] Chunked prefill is enabled with max_num_batched_tokens=8192.
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
(APIServer pid=1899544) WARNING 10-30 09:34:01 [vllm.py:1778] Enforce eager set, disabling torch.compile, CUDAGraphs, and JIT kernel warmup. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none and --kernel_config.enable_jit_warmup=False
(APIServer pid=1899544) WARNING 10-30 09:34:01 [vllm.py:1802] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(APIServer pid=1899544) INFO 10-30 09:34:01 [kernel.py:480] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'], gelu_and_mul_sparse=['triton', 'native'])
(APIServer pid=1899544) INFO 10-30 09:34:01 [vllm.py:2005] Cudagraph is disabled under eager mode
(APIServer pid=1899544) INFO 10-30 09:34:01 [compilation.py:317] Enabled custom fusions: norm_quant, act_quant

Fetching 10 files:   0%|          | 0/10 [00:00<?, ?it/s]
Fetching 10 files: 100%|██████████| 10/10 [00:00<00:00, 2636.10it/s]
W1030 09:34:12.057000 1900801 torch/_opaque_base.py:6] torch._opaque_base is deprecated, use torch._custom_class_base instead
W1030 09:34:12.058000 1900801 torch/_library/opaque_object.py:288] register_opaque_type is deprecated, use register_custom_class instead
W1030 09:34:12.058000 1900801 torch/_library/opaque_object.py:211] typ='value' is deprecated, use typ='constant' instead
(EngineCore pid=1900801) INFO 10-30 09:34:16 [core.py:139] Initializing a V1 LLM engine (v0.32.1rc1.dev577+geb703fbb3) with config: model='unsloth/functiongemma-270m-it', speculative_config=None, tokenizer='unsloth/functiongemma-270m-it', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, aux_output_config=AuxOutputConfig(enable_return_routed_experts=False, max_bytes=None), kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, per_request_spec_decode_metrics='none', kv_cache_metrics=False, kv_cache_metrics_sample=0.01, custom_histogram_buckets=None, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=functiongemma, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'cudagraph_decoder_replay': True, 'decoder_replay_cudagraph_capture_sizes': [], 'decoder_replay_trim_threshold': 768, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'], gelu_and_mul_sparse=['triton', 'native']), enable_flashinfer_autotune=True, enable_rocm_segmented_attn_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=False, monokernel=False, moe_backend='auto', sparse_indexer_topk_backend='auto', linear_backend='auto', linear_backend_per_quant=None)
(EngineCore pid=1900801) INFO 10-30 09:34:17 [parallel_state.py:1929] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_4b3415c489774968bcda74c706e158d9 backend=nccl
(EngineCore pid=1900801) INFO 10-30 09:34:18 [parallel_state.py:2385] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, ETP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=1900801) INFO 10-30 09:34:18 [gpu_worker.py:513] Using V2 Model Runner
(EngineCore pid=1900801) INFO 10-30 09:34:19 [model_runner.py:400] Loading model from scratch...
(EngineCore pid=1900801) INFO 10-30 09:34:20 [cuda.py:508] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=1900801) INFO 10-30 09:34:20 [flash_attn.py:1122] Using FlashAttention version 3
(EngineCore pid=1900801) INFO 10-30 09:34:20 [base_loader.py:82] Initializing model took 0.663624 seconds
(EngineCore pid=1900801) INFO 10-30 09:34:21 [weight_utils.py:1144] Filesystem type for checkpoints: EXT4. Checkpoint size: 0.50 GiB. Available RAM: 347.49 GiB.
(EngineCore pid=1900801) INFO 10-30 09:34:21 [weight_utils.py:1167] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.

Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]

Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  7.74it/s]

Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  7.72it/s]

(EngineCore pid=1900801) INFO 10-30 09:34:21 [default_loader.py:506] Loading weights took 0.14 seconds
(EngineCore pid=1900801) INFO 10-30 09:34:22 [model_runner.py:432] Model loading took 0.53 GiB memory and 3.667841 seconds
(EngineCore pid=1900801) INFO 10-30 09:34:22 [topk_topp_sampler.py:121] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0.
(EngineCore pid=1900801) INFO 10-30 09:34:22 [torch_utils.py:287] Reducing Torch threads from 48 to 1 for serving. Set OMP_NUM_THREADS in the external environment to override.
(EngineCore pid=1900801) INFO 10-30 09:34:22 [utils.py:381] Using LBNHC KV cache layout.
(EngineCore pid=1900801) INFO 10-30 09:34:24 [gpu_worker.py:779] Available KV cache memory: 4.81 GiB
(EngineCore pid=1900801) INFO 10-30 09:34:24 [kv_cache_utils.py:1147] GPU KV cache size: 278,202 tokens, Maximum concurrency for 2,048 tokens per request: 135.84x
(EngineCore pid=1900801) INFO 10-30 09:34:24 [kv_cache_utils.py:865] kv cache group sizes (16, 16, 16, 16, 16, 16)
(EngineCore pid=1900801) INFO 10-30 09:34:24 [kv_cache_utils.py:866] kv lcm block sizes 16
<frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1297: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417] Module vllm.third_party.deep_gemm was found but failed to import
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417] Traceback (most recent call last):
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]   File "/home/ubuntu/open-source/vllm/.venv/lib/python3.12/site-packages/torch/_ops.py", line 1587, in load_library
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]     ctypes.CDLL(path)
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]   File "/usr/lib/python3.12/ctypes/__init__.py", line 379, in __init__
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]     self._handle = _dlopen(self._name, mode)
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]                    ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417] OSError: /home/ubuntu/open-source/vllm/vllm/third_party/deep_gemm/_C_extension.abi3.so: undefined symbol: _ZN3c104cuda29c10_cuda_check_implementationEiPKcS2_jb
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417] 
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417] The above exception was the direct cause of the following exception:
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417] 
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417] Traceback (most recent call last):
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]   File "/home/ubuntu/open-source/vllm/vllm/utils/import_utils.py", line 415, in _has_module
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]     importlib.import_module(module_name)
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]   File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]     return _bootstrap._gcd_import(name[level:], package, level)
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]   File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]   File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]   File "<frozen importlib._bootstrap_external>", line 995, in exec_module
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]   File "/home/ubuntu/open-source/vllm/vllm/third_party/deep_gemm/__init__.py", line 16, in <module>
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]     from . import _C
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]   File "/home/ubuntu/open-source/vllm/vllm/third_party/deep_gemm/_C.py", line 23, in <module>
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]     _load_extension()
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]   File "/home/ubuntu/open-source/vllm/vllm/third_party/deep_gemm/_C.py", line 20, in _load_extension
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]     torch.ops.load_library(str(so_files[0]))
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]   File "/home/ubuntu/open-source/vllm/.venv/lib/python3.12/site-packages/torch/_ops.py", line 1589, in load_library
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417]     raise OSError(f"Could not load this library: {path}") from e
(EngineCore pid=1900801) WARNING 10-30 09:34:25 [import_utils.py:417] OSError: Could not load this library: /home/ubuntu/open-source/vllm/vllm/third_party/deep_gemm/_C_extension.abi3.so
(EngineCore pid=1900801) INFO 10-30 09:34:25 [kernel_warmup.py:485] Using FlashInfer managed autotune cache (root=flashinfer default). Pass --disable-flashinfer-autotune to skip autotuning.
2026-10-30 09:34:25,092 - INFO - autotuner.py:1048 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
2026-10-30 09:34:25,256 - INFO - autotune_cache.py:495 - flashinfer.jit: [Autotuner]: Managed autotune cache attached for this process: ManagedAutotuneCache(env_dir='/home/ubuntu/.cache/flashinfer/autotune/v2/7f2a45089f5ef40a', hits=0, misses=0)
(EngineCore pid=1900801) INFO 10-30 09:34:25 [kernel_warmup.py:371] Running FlashInfer autotune with 8192 tokens and token buckets (1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 768, 1024, 1280, 1536, 1792, 2048, 2560, 3072, 3584, 4096, 8192).
2026-10-30 09:34:25,313 - INFO - autotuner.py:1071 - flashinfer.jit: [Autotuner]: Autotuning process ends
(EngineCore pid=1900801) INFO 10-30 09:34:25 [gpu_worker.py:1073] Free memory on device (14.22/139.8 GiB) on startup. Desired GPU memory utilization is (0.04, 5.59 GiB). Actual usage is 0.63 GiB for consumed memory (weights + non-torch), 0.15 GiB for peak activation, and 0.0 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory=5003937342` (4.66 GiB) to fit into requested memory, or `--kv-cache-memory=14270856192` (13.29 GiB) to fully utilize gpu memory. Current kv cache memory in use is 4.81 GiB.
(EngineCore pid=1900801) INFO 10-30 09:34:25 [core.py:412] init engine (profile, create kv cache, warmup model) took 2.85 s

Fetching 10 files:   0%|          | 0/10 [00:00<?, ?it/s]
Fetching 10 files: 100%|██████████| 10/10 [00:00<00:00, 2373.01it/s]
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
(EngineCore pid=1900801) WARNING 10-30 09:34:29 [vllm.py:1778] Enforce eager set, disabling torch.compile, CUDAGraphs, and JIT kernel warmup. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none and --kernel_config.enable_jit_warmup=False
(EngineCore pid=1900801) WARNING 10-30 09:34:29 [vllm.py:1802] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(EngineCore pid=1900801) INFO 10-30 09:34:29 [kernel.py:480] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'], gelu_and_mul_sparse=['triton', 'native'])
(EngineCore pid=1900801) INFO 10-30 09:34:29 [vllm.py:2005] Cudagraph is disabled under eager mode
(EngineCore pid=1900801) INFO 10-30 09:34:29 [compilation.py:317] Enabled custom fusions: norm_quant, act_quant
(APIServer pid=1899544) INFO 10-30 09:34:29 [entry.py:135] Supported tasks: ['generate']
(APIServer pid=1899544) INFO 10-30 09:34:29 [factories.py:105] Scale-out endpoints are disabled. Restart vllm serve with --enable-scale-out to enable them.
(APIServer pid=1899544) INFO 10-30 09:34:29 [parser_manager.py:36] "auto" tool choice has been enabled.
(APIServer pid=1899544) INFO 10-30 09:34:31 [hf.py:736] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
(APIServer pid=1899544) WARNING 10-30 09:34:31 [model.py:1907] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'top_k': 64, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=1899544) INFO 10-30 09:34:31 [entry.py:139] Starting vLLM server on http://0.0.0.0:8124
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:188] Available routes are:
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /docs, Methods: HEAD, GET
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /redoc, Methods: HEAD, GET
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /load, Methods: GET
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /version, Methods: GET
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /kv_event_sources, Methods: GET
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /health, Methods: GET
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /metrics, Methods: GET
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /tokenize, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /detokenize, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/models, Methods: GET
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/systemone, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/decisions, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /ping, Methods: GET
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /ping, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /invocations, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/chat/completions, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/responses, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/completions, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/messages, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /generative_scoring, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/messages/render, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/completions/render, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/responses/render, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /v1/completions/derender, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /inference/v1/generate, Methods: POST
(APIServer pid=1899544) INFO 10-30 09:34:31 [launcher.py:197] Route: /inference/v1/abort_requests, Methods: POST
INFO:     Started server process [1899544]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     127.0.0.1:49782 - "GET /health HTTP/1.1" 200 OK
INFO:     127.0.0.1:49790 - "GET /health HTTP/1.1" 200 OK
INFO:     127.0.0.1:49804 - "GET /v1/models HTTP/1.1" 200 OK
INFO:     127.0.0.1:49804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:49804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1899544) INFO 10-30 09:34:41 [loggers.py:335] Engine 000: Avg prompt throughput: 19.8 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 50.2%
INFO:     127.0.0.1:45372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=1899544) INFO 10-30 09:34:45 [launcher.py:256] [shutdown] API server: shutdown triggered
(APIServer pid=1899544) INFO 10-30 09:34:45 [launcher.py:269] [shutdown] API server: stopping engine client mode=abort timeout=0s
(APIServer pid=1899544) INFO 10-30 09:34:45 [core_client.py:792] [shutdown] MPClient: start timeout=0s
(APIServer pid=1899544) INFO 10-30 09:34:45 [core_client.py:794] [shutdown] MPClient: stopping engine manager
(APIServer pid=1899544) INFO 10-30 09:34:45 [utils.py:628] [shutdown] Process manager: send sigterm to process EngineCore
(APIServer pid=1899544) WARNING 10-30 09:34:45 [utils.py:648] [shutdown] Process manager: force killing remaining processes count=1
(APIServer pid=1899544) WARNING 10-30 09:34:45 [utils.py:653] [shutdown] Process manager: force killing remaining process EngineCore pid 1900801
(EngineCore pid=1900801) INFO 10-30 09:34:45 [core.py:1525] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=1900801) INFO 10-30 09:34:45 [core.py:1712] [shutdown] EngineCore: start mode=abort timeout=0s
(EngineCore pid=1900801) INFO 10-30 09:34:45 [core.py:1743] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=1900801) INFO 10-30 09:34:45 [core.py:1573] [shutdown] EngineCore: exiting busy loop
(APIServer pid=1899544) INFO 10-30 09:34:45 [core_client.py:796] [shutdown] MPClient: engine manager stopped
(APIServer pid=1899544) INFO 10-30 09:34:45 [core_client.py:797] [shutdown] MPClient: cleaning up background resources
(APIServer pid=1899544) INFO 10-30 09:34:45 [core_client.py:799] [shutdown] MPClient: complete
(APIServer pid=1899544) INFO 10-30 09:34:45 [launcher.py:278] [shutdown] API server: engine client stopped
(APIServer pid=1899544) INFO 10-30 09:34:45 [launcher.py:281] [shutdown] API server: signalling HTTP server shutdown
(APIServer pid=1899544) INFO 10-30 09:34:45 [launcher.py:304] [shutdown] API server: shutting down FastAPI HTTP server
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
/usr/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '