## 4.27-28 ``` Sun Apr 27 21:01:18 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.127.08 Driver Version: 550.127.08 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA A10 On | 00000000:00:07.0 Off | Off | | 0% 56C P0 149W / 150W | 22288MiB / 24564MiB | 99% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA A10 On | 00000000:00:08.0 Off | 0 | | 0% 56C P0 150W / 150W | 20944MiB / 23028MiB | 99% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA A10 On | 00000000:00:09.0 Off | Off | | 0% 57C P0 148W / 150W | 22288MiB / 24564MiB | 99% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA A10 On | 00000000:00:0A.0 Off | Off | | 0% 56C P0 150W / 150W | 22288MiB / 24564MiB | 99% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ Sun Apr 27 21:03:12 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.127.08 Driver Version: 550.127.08 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA A10 On | 00000000:00:07.0 Off | Off | | 0% 51C P0 135W / 150W | 22288MiB / 24564MiB | 99% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA A10 On | 00000000:00:08.0 Off | 0 | | 0% 52C P0 134W / 150W | 20944MiB / 23028MiB | 99% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA A10 On | 00000000:00:09.0 Off | Off | | 0% 53C P0 151W / 150W | 22308MiB / 24564MiB | 99% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA A10 On | 00000000:00:0A.0 Off | Off | | 0% 52C P0 149W / 150W | 22288MiB / 24564MiB | 99% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ Test completed in 14.99 seconds SGLang Stress Test Results - 2025-04-28 20:22:04 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:30000/generate │ │ Max Tokens │ 256 │ │ Concurrent Requests │ 30 │ │ Total Requests │ 300 │ │ Successful Requests │ 300 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 14.99 seconds │ │ Min Response Time │ 1.47 seconds │ │ Max Response Time │ 1.65 seconds │ │ Average Response Time │ 1.50 seconds │ │ Median Response Time │ 1.47 seconds │ │ P90 Response Time │ 1.64 seconds │ │ P95 Response Time │ 1.65 seconds │ │ P99 Response Time │ 1.65 seconds │ │ Std Dev Response Time │ 0.05 seconds │ │ Theoretical Max Throughput │ 182.16 requests/second │ │ Actual Throughput │ 20.02 requests/second │ │ Total Generated Tokens │ 31274 │ │ Tokens Per Second │ 2086.65 │ │ Avg Tokens Per Request │ 104.25 │ │ Peak Requests In Flight │ 30 │ └────────────────────────────┴─────────────────────────────────┘ Test completed in 14.75 seconds SGLang Stress Test Results - 2025-04-28 20:23:05 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:30000/generate │ │ Max Tokens │ 256 │ │ Concurrent Requests │ 30 │ │ Total Requests │ 300 │ │ Successful Requests │ 300 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 14.75 seconds │ │ Min Response Time │ 1.47 seconds │ │ Max Response Time │ 1.49 seconds │ │ Average Response Time │ 1.47 seconds │ │ Median Response Time │ 1.47 seconds │ │ P90 Response Time │ 1.48 seconds │ │ P95 Response Time │ 1.49 seconds │ │ P99 Response Time │ 1.49 seconds │ │ Std Dev Response Time │ 0.00 seconds │ │ Theoretical Max Throughput │ 201.91 requests/second │ │ Actual Throughput │ 20.34 requests/second │ │ Total Generated Tokens │ 31325 │ │ Tokens Per Second │ 2123.63 │ │ Avg Tokens Per Request │ 104.42 │ │ Peak Requests In Flight │ 30 │ └────────────────────────────┴─────────────────────────────────┘ Test completed in 20.45 seconds SGLang Stress Test Results - 2025-04-28 20:23:47 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:30000/generate │ │ Max Tokens │ 256 │ │ Concurrent Requests │ 50 │ │ Total Requests │ 500 │ │ Successful Requests │ 500 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 20.45 seconds │ │ Min Response Time │ 2.03 seconds │ │ Max Response Time │ 2.09 seconds │ │ Average Response Time │ 2.04 seconds │ │ Median Response Time │ 2.04 seconds │ │ P90 Response Time │ 2.08 seconds │ │ P95 Response Time │ 2.08 seconds │ │ P99 Response Time │ 2.09 seconds │ │ Std Dev Response Time │ 0.01 seconds │ │ Theoretical Max Throughput │ 239.61 requests/second │ │ Actual Throughput │ 24.45 requests/second │ │ Total Generated Tokens │ 52372 │ │ Tokens Per Second │ 2560.58 │ │ Avg Tokens Per Request │ 104.74 │ │ Peak Requests In Flight │ 50 │ └────────────────────────────┴─────────────────────────────────┘ Test completed in 20.45 seconds SGLang Stress Test Results - 2025-04-28 20:24:47 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:30000/generate │ │ Max Tokens │ 256 │ │ Concurrent Requests │ 50 │ │ Total Requests │ 500 │ │ Successful Requests │ 500 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 20.45 seconds │ │ Min Response Time │ 2.03 seconds │ │ Max Response Time │ 2.08 seconds │ │ Average Response Time │ 2.04 seconds │ │ Median Response Time │ 2.04 seconds │ │ P90 Response Time │ 2.07 seconds │ │ P95 Response Time │ 2.07 seconds │ │ P99 Response Time │ 2.07 seconds │ │ Std Dev Response Time │ 0.01 seconds │ │ Theoretical Max Throughput │ 240.93 requests/second │ │ Actual Throughput │ 24.45 requests/second │ │ Total Generated Tokens │ 51685 │ │ Tokens Per Second │ 2526.98 │ │ Avg Tokens Per Request │ 103.37 │ │ Peak Requests In Flight │ 50 │ └────────────────────────────┴─────────────────────────────────┘ ``` ## 4.29 ### SGLang ``` python3 -m sglang.launch_server --model-path /home/vllm/llm/ --context-length 8192 --tp 4 //warm up effect! Test completed in 17.80 seconds SGLang Stress Test Results - 2025-04-29 14:04:56 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:30000/generate │ │ Max Tokens │ 256 │ │ Concurrent Requests │ 5 │ │ Total Requests │ 20 │ │ Successful Requests │ 20 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 17.80 seconds │ │ Min Response Time │ 0.83 seconds │ │ Max Response Time │ 14.62 seconds │ │ Average Response Time │ 4.45 seconds │ │ Median Response Time │ 1.17 seconds │ │ P90 Response Time │ 14.62 seconds │ │ P95 Response Time │ 14.62 seconds │ │ P99 Response Time │ 14.62 seconds │ │ Std Dev Response Time │ 6.03 seconds │ │ Theoretical Max Throughput │ 1.37 requests/second │ │ Actual Throughput │ 1.12 requests/second │ │ Total Generated Tokens │ 2123 │ │ Tokens Per Second │ 119.25 │ │ Avg Tokens Per Request │ 106.15 │ │ Peak Requests In Flight │ 5 │ └────────────────────────────┴─────────────────────────────────┘ Test completed in 14.80 seconds SGLang Stress Test Results - 2025-04-29 14:05:46 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:30000/generate │ │ Max Tokens │ 256 │ │ Concurrent Requests │ 30 │ │ Total Requests │ 300 │ │ Successful Requests │ 300 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 14.80 seconds │ │ Min Response Time │ 1.47 seconds │ │ Max Response Time │ 1.51 seconds │ │ Average Response Time │ 1.48 seconds │ │ Median Response Time │ 1.47 seconds │ │ P90 Response Time │ 1.51 seconds │ │ P95 Response Time │ 1.51 seconds │ │ P99 Response Time │ 1.51 seconds │ │ Std Dev Response Time │ 0.01 seconds │ │ Theoretical Max Throughput │ 198.68 requests/second │ │ Actual Throughput │ 20.27 requests/second │ │ Total Generated Tokens │ 31357 │ │ Tokens Per Second │ 2119.20 │ │ Avg Tokens Per Request │ 104.52 │ │ Peak Requests In Flight │ 30 │ └────────────────────────────┴─────────────────────────────────┘ Test completed in 20.43 seconds SGLang Stress Test Results - 2025-04-29 14:06:42 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:30000/generate │ │ Max Tokens │ 256 │ │ Concurrent Requests │ 50 │ │ Total Requests │ 500 │ │ Successful Requests │ 500 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 20.43 seconds │ │ Min Response Time │ 2.03 seconds │ │ Max Response Time │ 2.04 seconds │ │ Average Response Time │ 2.04 seconds │ │ Median Response Time │ 2.04 seconds │ │ P90 Response Time │ 2.04 seconds │ │ P95 Response Time │ 2.04 seconds │ │ P99 Response Time │ 2.04 seconds │ │ Std Dev Response Time │ 0.00 seconds │ │ Theoretical Max Throughput │ 244.89 requests/second │ │ Actual Throughput │ 24.48 requests/second │ │ Total Generated Tokens │ 51885 │ │ Tokens Per Second │ 2540.23 │ │ Avg Tokens Per Request │ 103.77 │ │ Peak Requests In Flight │ 50 │ └────────────────────────────┴─────────────────────────────────┘ /**** Now compare with 2 GPUs ****/ python3 -m sglang.launch_server --model-path /home/vllm/llm/ --context-length 8192 --tp 2 Test completed in 4.84 seconds SGLang Stress Test Results - 2025-04-29 14:08:06 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:30000/generate │ │ Max Tokens │ 256 │ │ Concurrent Requests │ 5 │ │ Total Requests │ 20 │ │ Successful Requests │ 20 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 4.84 seconds │ │ Min Response Time │ 1.05 seconds │ │ Max Response Time │ 1.62 seconds │ │ Average Response Time │ 1.21 seconds │ │ Median Response Time │ 1.08 seconds │ │ P90 Response Time │ 1.62 seconds │ │ P95 Response Time │ 1.62 seconds │ │ P99 Response Time │ 1.62 seconds │ │ Std Dev Response Time │ 0.24 seconds │ │ Theoretical Max Throughput │ 12.36 requests/second │ │ Actual Throughput │ 4.13 requests/second │ │ Total Generated Tokens │ 2047 │ │ Tokens Per Second │ 422.97 │ │ Avg Tokens Per Request │ 102.35 │ │ Peak Requests In Flight │ 5 │ └────────────────────────────┴─────────────────────────────────┘ Test completed in 27.83 seconds SGLang Stress Test Results - 2025-04-29 14:09:01 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:30000/generate │ │ Max Tokens │ 256 │ │ Concurrent Requests │ 30 │ │ Total Requests │ 300 │ │ Successful Requests │ 300 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 27.83 seconds │ │ Min Response Time │ 2.73 seconds │ │ Max Response Time │ 2.88 seconds │ │ Average Response Time │ 2.78 seconds │ │ Median Response Time │ 2.78 seconds │ │ P90 Response Time │ 2.88 seconds │ │ P95 Response Time │ 2.88 seconds │ │ P99 Response Time │ 2.88 seconds │ │ Std Dev Response Time │ 0.04 seconds │ │ Theoretical Max Throughput │ 104.25 requests/second │ │ Actual Throughput │ 10.78 requests/second │ │ Total Generated Tokens │ 31115 │ │ Tokens Per Second │ 1117.99 │ │ Avg Tokens Per Request │ 103.72 │ │ Peak Requests In Flight │ 30 │ └────────────────────────────┴─────────────────────────────────┘ Test completed in 28.54 seconds SGLang Stress Test Results - 2025-04-29 14:10:00 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:30000/generate │ │ Max Tokens │ 256 │ │ Concurrent Requests │ 50 │ │ Total Requests │ 500 │ │ Successful Requests │ 500 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 28.54 seconds │ │ Min Response Time │ 2.83 seconds │ │ Max Response Time │ 2.88 seconds │ │ Average Response Time │ 2.85 seconds │ │ Median Response Time │ 2.85 seconds │ │ P90 Response Time │ 2.87 seconds │ │ P95 Response Time │ 2.88 seconds │ │ P99 Response Time │ 2.88 seconds │ │ Std Dev Response Time │ 0.02 seconds │ │ Theoretical Max Throughput │ 173.61 requests/second │ │ Actual Throughput │ 17.52 requests/second │ │ Total Generated Tokens │ 52003 │ │ Tokens Per Second │ 1822.21 │ │ Avg Tokens Per Request │ 104.01 │ │ Peak Requests In Flight │ 50 │ └────────────────────────────┴─────────────────────────────────┘ Test completed in 28.83 seconds SGLang Stress Test Results - 2025-04-29 14:11:38 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:30000/generate │ │ Max Tokens │ 256 │ │ Concurrent Requests │ 50 │ │ Total Requests │ 500 │ │ Successful Requests │ 500 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 28.83 seconds │ │ Min Response Time │ 2.87 seconds │ │ Max Response Time │ 2.90 seconds │ │ Average Response Time │ 2.88 seconds │ │ Median Response Time │ 2.88 seconds │ │ P90 Response Time │ 2.90 seconds │ │ P95 Response Time │ 2.90 seconds │ │ P99 Response Time │ 2.90 seconds │ │ Std Dev Response Time │ 0.01 seconds │ │ Theoretical Max Throughput │ 172.44 requests/second │ │ Actual Throughput │ 17.34 requests/second │ │ Total Generated Tokens │ 52224 │ │ Tokens Per Second │ 1811.35 │ │ Avg Tokens Per Request │ 104.45 │ │ Peak Requests In Flight │ 50 │ └────────────────────────────┴─────────────────────────────────┘ Test completed in 27.84 seconds SGLang Stress Test Results - 2025-04-29 14:12:25 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:30000/generate │ │ Max Tokens │ 256 │ │ Concurrent Requests │ 30 │ │ Total Requests │ 300 │ │ Successful Requests │ 300 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 27.84 seconds │ │ Min Response Time │ 2.77 seconds │ │ Max Response Time │ 2.82 seconds │ │ Average Response Time │ 2.78 seconds │ │ Median Response Time │ 2.78 seconds │ │ P90 Response Time │ 2.82 seconds │ │ P95 Response Time │ 2.82 seconds │ │ P99 Response Time │ 2.82 seconds │ │ Std Dev Response Time │ 0.01 seconds │ │ Theoretical Max Throughput │ 106.32 requests/second │ │ Actual Throughput │ 10.78 requests/second │ │ Total Generated Tokens │ 31067 │ │ Tokens Per Second │ 1115.89 │ │ Avg Tokens Per Request │ 103.56 │ │ Peak Requests In Flight │ 30 │ └────────────────────────────┴─────────────────────────────────┘ ``` ### vLLM ``` vllm serve /home/vllm/llm/Qwen7B-awq --max-model-len 8192 --tensor-parallel-size 4 //No need to warm-up Test completed in 4.62 seconds vLLM Stress Test Results - 2025-04-29 14:16:47 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:8000/v1/completions │ │ Model │ /home/vllm/llm/Qwen7B-awq │ │ Max Tokens │ 256 │ │ Temperature │ 0.7 │ │ Concurrent Requests │ 5 │ │ Total Requests │ 20 │ │ Successful Requests │ 20 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 4.62 seconds │ │ Min Response Time │ 0.86 seconds │ │ Max Response Time │ 1.23 seconds │ │ Average Response Time │ 1.04 seconds │ │ Median Response Time │ 1.08 seconds │ │ P90 Response Time │ 1.15 seconds │ │ P95 Response Time │ 1.23 seconds │ │ P99 Response Time │ 1.23 seconds │ │ Std Dev Response Time │ 0.10 seconds │ │ Theoretical Max Throughput │ 16.27 requests/second │ │ Actual Throughput │ 4.33 requests/second │ │ Total Generated Tokens │ 2102 │ │ Tokens Per Second │ 455.09 │ │ Avg Tokens Per Request │ 105.10 │ │ Peak Requests In Flight │ 5 │ └────────────────────────────┴──────────────────────────────────────┘ Test completed in 22.70 seconds vLLM Stress Test Results - 2025-04-29 14:17:40 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:8000/v1/completions │ │ Model │ /home/vllm/llm/Qwen7B-awq │ │ Max Tokens │ 256 │ │ Temperature │ 0.7 │ │ Concurrent Requests │ 30 │ │ Total Requests │ 300 │ │ Successful Requests │ 300 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 22.70 seconds │ │ Min Response Time │ 1.10 seconds │ │ Max Response Time │ 2.42 seconds │ │ Average Response Time │ 1.89 seconds │ │ Median Response Time │ 1.90 seconds │ │ P90 Response Time │ 2.16 seconds │ │ P95 Response Time │ 2.22 seconds │ │ P99 Response Time │ 2.29 seconds │ │ Std Dev Response Time │ 0.22 seconds │ │ Theoretical Max Throughput │ 124.13 requests/second │ │ Actual Throughput │ 13.22 requests/second │ │ Total Generated Tokens │ 29939 │ │ Tokens Per Second │ 1318.94 │ │ Avg Tokens Per Request │ 99.80 │ │ Peak Requests In Flight │ 30 │ └────────────────────────────┴──────────────────────────────────────┘ Test completed in 22.87 seconds vLLM Stress Test Results - 2025-04-29 14:18:24 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:8000/v1/completions │ │ Model │ /home/vllm/llm/Qwen7B-awq │ │ Max Tokens │ 256 │ │ Temperature │ 0.7 │ │ Concurrent Requests │ 30 │ │ Total Requests │ 300 │ │ Successful Requests │ 300 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 22.87 seconds │ │ Min Response Time │ 1.06 seconds │ │ Max Response Time │ 2.93 seconds │ │ Average Response Time │ 1.91 seconds │ │ Median Response Time │ 1.92 seconds │ │ P90 Response Time │ 2.14 seconds │ │ P95 Response Time │ 2.18 seconds │ │ P99 Response Time │ 2.33 seconds │ │ Std Dev Response Time │ 0.21 seconds │ │ Theoretical Max Throughput │ 102.44 requests/second │ │ Actual Throughput │ 13.12 requests/second │ │ Total Generated Tokens │ 29951 │ │ Tokens Per Second │ 1309.40 │ │ Avg Tokens Per Request │ 99.84 │ │ Peak Requests In Flight │ 30 │ └────────────────────────────┴──────────────────────────────────────┘ Test completed in 32.03 seconds vLLM Stress Test Results - 2025-04-29 14:19:28 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:8000/v1/completions │ │ Model │ /home/vllm/llm/Qwen7B-awq │ │ Max Tokens │ 256 │ │ Temperature │ 0.7 │ │ Concurrent Requests │ 50 │ │ Total Requests │ 500 │ │ Successful Requests │ 500 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 32.02 seconds │ │ Min Response Time │ 1.65 seconds │ │ Max Response Time │ 3.80 seconds │ │ Average Response Time │ 2.69 seconds │ │ Median Response Time │ 2.73 seconds │ │ P90 Response Time │ 2.99 seconds │ │ P95 Response Time │ 3.06 seconds │ │ P99 Response Time │ 3.22 seconds │ │ Std Dev Response Time │ 0.27 seconds │ │ Theoretical Max Throughput │ 131.75 requests/second │ │ Actual Throughput │ 15.61 requests/second │ │ Total Generated Tokens │ 50154 │ │ Tokens Per Second │ 1566.10 │ │ Avg Tokens Per Request │ 100.31 │ │ Peak Requests In Flight │ 50 │ └────────────────────────────┴──────────────────────────────────────┘ Test completed in 32.89 seconds vLLM Stress Test Results - 2025-04-29 14:20:16 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:8000/v1/completions │ │ Model │ /home/vllm/llm/Qwen7B-awq │ │ Max Tokens │ 256 │ │ Temperature │ 0.7 │ │ Concurrent Requests │ 50 │ │ Total Requests │ 500 │ │ Successful Requests │ 500 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 32.89 seconds │ │ Min Response Time │ 1.61 seconds │ │ Max Response Time │ 3.80 seconds │ │ Average Response Time │ 2.70 seconds │ │ Median Response Time │ 2.73 seconds │ │ P90 Response Time │ 3.01 seconds │ │ P95 Response Time │ 3.09 seconds │ │ P99 Response Time │ 3.23 seconds │ │ Std Dev Response Time │ 0.28 seconds │ │ Theoretical Max Throughput │ 131.74 requests/second │ │ Actual Throughput │ 15.20 requests/second │ │ Total Generated Tokens │ 50462 │ │ Tokens Per Second │ 1534.41 │ │ Avg Tokens Per Request │ 100.92 │ │ Peak Requests In Flight │ 50 │ └────────────────────────────┴──────────────────────────────────────┘ Test completed in 52.28 seconds vLLM Stress Test Results - 2025-04-29 14:34:25 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:8000/v1/completions │ │ Model │ /home/vllm/llm/Qwen7B-awq │ │ Max Tokens │ 256 │ │ Temperature │ 0.7 │ │ Concurrent Requests │ 100 │ │ Total Requests │ 1000 │ │ Successful Requests │ 1000 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 52.28 seconds │ │ Min Response Time │ 2.78 seconds │ │ Max Response Time │ 5.70 seconds │ │ Average Response Time │ 4.42 seconds │ │ Median Response Time │ 4.48 seconds │ │ P90 Response Time │ 4.86 seconds │ │ P95 Response Time │ 4.94 seconds │ │ P99 Response Time │ 5.09 seconds │ │ Std Dev Response Time │ 0.41 seconds │ │ Theoretical Max Throughput │ 175.43 requests/second │ │ Actual Throughput │ 19.13 requests/second │ │ Total Generated Tokens │ 100077 │ │ Tokens Per Second │ 1914.26 │ │ Avg Tokens Per Request │ 100.08 │ │ Peak Requests In Flight │ 100 │ └────────────────────────────┴──────────────────────────────────────┘ Test completed in 53.39 seconds vLLM Stress Test Results - 2025-04-29 14:35:37 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:8000/v1/completions │ │ Model │ /home/vllm/llm/Qwen7B-awq │ │ Max Tokens │ 256 │ │ Temperature │ 0.7 │ │ Concurrent Requests │ 100 │ │ Total Requests │ 1000 │ │ Successful Requests │ 1000 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 53.39 seconds │ │ Min Response Time │ 2.82 seconds │ │ Max Response Time │ 5.72 seconds │ │ Average Response Time │ 4.44 seconds │ │ Median Response Time │ 4.49 seconds │ │ P90 Response Time │ 4.91 seconds │ │ P95 Response Time │ 4.98 seconds │ │ P99 Response Time │ 5.15 seconds │ │ Std Dev Response Time │ 0.40 seconds │ │ Theoretical Max Throughput │ 174.95 requests/second │ │ Actual Throughput │ 18.73 requests/second │ │ Total Generated Tokens │ 100771 │ │ Tokens Per Second │ 1887.62 │ │ Avg Tokens Per Request │ 100.77 │ │ Peak Requests In Flight │ 100 │ └────────────────────────────┴──────────────────────────────────────┘ /**** Now compare with 2 GPUs ****/ vllm serve /home/vllm/llm/Qwen7B-awq --max-model-len 8192 --tensor-parallel-size 2 Test completed in 26.98 seconds vLLM Stress Test Results - 2025-04-29 14:25:50 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:8000/v1/completions │ │ Model │ /home/vllm/llm/Qwen7B-awq │ │ Max Tokens │ 256 │ │ Temperature │ 0.7 │ │ Concurrent Requests │ 30 │ │ Total Requests │ 300 │ │ Successful Requests │ 300 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 26.98 seconds │ │ Min Response Time │ 1.26 seconds │ │ Max Response Time │ 3.36 seconds │ │ Average Response Time │ 2.09 seconds │ │ Median Response Time │ 2.12 seconds │ │ P90 Response Time │ 2.36 seconds │ │ P95 Response Time │ 2.42 seconds │ │ P99 Response Time │ 2.80 seconds │ │ Std Dev Response Time │ 0.26 seconds │ │ Theoretical Max Throughput │ 89.27 requests/second │ │ Actual Throughput │ 11.12 requests/second │ │ Total Generated Tokens │ 30521 │ │ Tokens Per Second │ 1131.43 │ │ Avg Tokens Per Request │ 101.74 │ │ Peak Requests In Flight │ 30 │ └────────────────────────────┴──────────────────────────────────────┘ Test completed in 25.00 seconds vLLM Stress Test Results - 2025-04-29 14:26:58 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:8000/v1/completions │ │ Model │ /home/vllm/llm/Qwen7B-awq │ │ Max Tokens │ 256 │ │ Temperature │ 0.7 │ │ Concurrent Requests │ 30 │ │ Total Requests │ 300 │ │ Successful Requests │ 300 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 25.00 seconds │ │ Min Response Time │ 1.35 seconds │ │ Max Response Time │ 2.71 seconds │ │ Average Response Time │ 2.09 seconds │ │ Median Response Time │ 2.12 seconds │ │ P90 Response Time │ 2.35 seconds │ │ P95 Response Time │ 2.42 seconds │ │ P99 Response Time │ 2.64 seconds │ │ Std Dev Response Time │ 0.23 seconds │ │ Theoretical Max Throughput │ 110.57 requests/second │ │ Actual Throughput │ 12.00 requests/second │ │ Total Generated Tokens │ 30446 │ │ Tokens Per Second │ 1217.92 │ │ Avg Tokens Per Request │ 101.49 │ │ Peak Requests In Flight │ 30 │ └────────────────────────────┴──────────────────────────────────────┘ Test completed in 32.76 seconds vLLM Stress Test Results - 2025-04-29 14:27:46 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:8000/v1/completions │ │ Model │ /home/vllm/llm/Qwen7B-awq │ │ Max Tokens │ 256 │ │ Temperature │ 0.7 │ │ Concurrent Requests │ 50 │ │ Total Requests │ 500 │ │ Successful Requests │ 500 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 32.75 seconds │ │ Min Response Time │ 1.69 seconds │ │ Max Response Time │ 3.45 seconds │ │ Average Response Time │ 2.72 seconds │ │ Median Response Time │ 2.73 seconds │ │ P90 Response Time │ 3.05 seconds │ │ P95 Response Time │ 3.16 seconds │ │ P99 Response Time │ 3.34 seconds │ │ Std Dev Response Time │ 0.28 seconds │ │ Theoretical Max Throughput │ 144.73 requests/second │ │ Actual Throughput │ 15.27 requests/second │ │ Total Generated Tokens │ 50269 │ │ Tokens Per Second │ 1534.72 │ │ Avg Tokens Per Request │ 100.54 │ │ Peak Requests In Flight │ 50 │ └────────────────────────────┴──────────────────────────────────────┘ Test completed in 33.17 seconds vLLM Stress Test Results - 2025-04-29 14:28:54 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:8000/v1/completions │ │ Model │ /home/vllm/llm/Qwen7B-awq │ │ Max Tokens │ 256 │ │ Temperature │ 0.7 │ │ Concurrent Requests │ 50 │ │ Total Requests │ 500 │ │ Successful Requests │ 500 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 33.17 seconds │ │ Min Response Time │ 1.59 seconds │ │ Max Response Time │ 4.06 seconds │ │ Average Response Time │ 2.72 seconds │ │ Median Response Time │ 2.75 seconds │ │ P90 Response Time │ 3.04 seconds │ │ P95 Response Time │ 3.16 seconds │ │ P99 Response Time │ 3.28 seconds │ │ Std Dev Response Time │ 0.28 seconds │ │ Theoretical Max Throughput │ 123.27 requests/second │ │ Actual Throughput │ 15.07 requests/second │ │ Total Generated Tokens │ 50419 │ │ Tokens Per Second │ 1519.86 │ │ Avg Tokens Per Request │ 100.84 │ │ Peak Requests In Flight │ 50 │ └────────────────────────────┴──────────────────────────────────────┘ Test completed in 57.48 seconds vLLM Stress Test Results - 2025-04-29 14:30:29 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:8000/v1/completions │ │ Model │ /home/vllm/llm/Qwen7B-awq │ │ Max Tokens │ 256 │ │ Temperature │ 0.7 │ │ Concurrent Requests │ 100 │ │ Total Requests │ 1000 │ │ Successful Requests │ 1000 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 57.48 seconds │ │ Min Response Time │ 2.86 seconds │ │ Max Response Time │ 6.17 seconds │ │ Average Response Time │ 4.63 seconds │ │ Median Response Time │ 4.69 seconds │ │ P90 Response Time │ 5.08 seconds │ │ P95 Response Time │ 5.18 seconds │ │ P99 Response Time │ 5.45 seconds │ │ Std Dev Response Time │ 0.42 seconds │ │ Theoretical Max Throughput │ 162.00 requests/second │ │ Actual Throughput │ 17.40 requests/second │ │ Total Generated Tokens │ 101045 │ │ Tokens Per Second │ 1757.90 │ │ Avg Tokens Per Request │ 101.05 │ │ Peak Requests In Flight │ 100 │ └────────────────────────────┴──────────────────────────────────────┘ Test completed in 55.60 seconds vLLM Stress Test Results - 2025-04-29 14:31:44 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ URL │ http://localhost:8000/v1/completions │ │ Model │ /home/vllm/llm/Qwen7B-awq │ │ Max Tokens │ 256 │ │ Temperature │ 0.7 │ │ Concurrent Requests │ 100 │ │ Total Requests │ 1000 │ │ Successful Requests │ 1000 (100.0%) │ │ Failed Requests │ 0 (0.0%) │ │ Total Test Duration │ 55.60 seconds │ │ Min Response Time │ 2.86 seconds │ │ Max Response Time │ 6.09 seconds │ │ Average Response Time │ 4.62 seconds │ │ Median Response Time │ 4.68 seconds │ │ P90 Response Time │ 5.12 seconds │ │ P95 Response Time │ 5.19 seconds │ │ P99 Response Time │ 5.34 seconds │ │ Std Dev Response Time │ 0.43 seconds │ │ Theoretical Max Throughput │ 164.09 requests/second │ │ Actual Throughput │ 17.99 requests/second │ │ Total Generated Tokens │ 100987 │ │ Tokens Per Second │ 1816.33 │ │ Avg Tokens Per Request │ 100.99 │ │ Peak Requests In Flight │ 100 │ └────────────────────────────┴──────────────────────────────────────┘ ``` ### vLLM Performance Comparison Between 2 A10 and 4 A10 GPUs When examining vLLM's performance across 2 vs 4 GPUs, we can observe: 1. For 30 concurrent requests : - 2 GPUs: 11.12-12.00 req/s, ~1217 tokens/s, avg response time ~2.09s - 4 GPUs: 13.12-13.22 req/s, ~1319 tokens/s, avg response time ~1.90s 2. For 50 concurrent requests : - 2 GPUs: 15.07-15.27 req/s, ~1527 tokens/s, avg response time ~2.72s - 4 GPUs: 15.20-15.61 req/s, ~1550 tokens/s, avg response time ~2.70s 3. For 100 concurrent requests : - 2 GPUs: 17.40-17.99 req/s, ~1787 tokens/s, avg response time ~4.63s - 4 GPUs: 18.73-19.13 req/s, ~1901 tokens/s, avg response time ~4.43s The conclusion for vLLM is that it does show some scaling improvements when moving from 2 to 4 GPUs, but the gains are modest: - Throughput improves by approximately 15-20% for 30 concurrent requests - Throughput improves by only 2-5% for 50 concurrent requests - Throughput improves by approximately 6-10% for 100 concurrent requests This suggests that vLLM's scaling efficiency decreases as the number of GPUs increases, with diminishing returns at higher GPU counts. The model may be hitting bottlenecks that aren't related to raw compute capacity (potentially communication overhead between GPUs or other system limitations). In contrast, SGLang with tensor parallelism shows more dramatic scaling improvements when moving from 2 to 4 GPUs: - For 30 concurrent requests: 10.78 req/s → 20.27 req/s (88% improvement) - For 50 concurrent requests: 17.52 req/s → 24.48 req/s (40% improvement) This is a significant finding that could be highlighted in your conclusion - SGLang appears to scale more efficiently with additional GPUs than vLLM for this specific workload and model.