50/100 VU runs had bug in k6 script which implied a single-identity auth pattern; backend results valid, auth path not representative 200 VU runs (and everything moving forward) now uses N concurrent users, each with independent identity and token lifecycle Run 1: █ TOTAL RESULTS checks_total.......: 69056 190.485791/s checks_succeeded...: 100.00% 69056 out of 69056 checks_failed......: 0.00% 0 out of 69056 ✓ logged in successfully ✓ fetched actor ✓ uploaded document ✓ registered successfully CUSTOM endpoint_actor_get.............: avg=158.97ms min=3.43ms med=106.66ms max=1.41s p(90)=300.47ms p(95)=401.48ms p(99)=697.17ms endpoint_auth_login............: avg=744.86ms min=267.85ms med=403.28ms max=5.97s p(90)=1.71s p(95)=3.34s p(99)=5.02s endpoint_auth_register.........: avg=1.01s min=693.3ms med=919.65ms max=6.59s p(90)=1.19s p(95)=1.31s p(99)=2.9s endpoint_document_upload.......: avg=226.57ms min=36.28ms med=196.55ms max=1.7s p(90)=396.14ms p(95)=497.69ms p(99)=791.05ms HTTP http_req_duration..............: avg=227.82ms min=3.43ms med=180.71ms max=6.59s p(90)=413.11ms p(95)=793.36ms p(99)=1.17s { expected_response:true }...: avg=227.82ms min=3.43ms med=180.71ms max=6.59s p(90)=413.11ms p(95)=793.36ms p(99)=1.17s http_req_failed................: 0.00% 0 out of 69056 http_reqs......................: 69056 190.485791/s EXECUTION iteration_duration.............: avg=1.31s min=1s med=1.18s max=11.53s p(90)=1.5s p(95)=2.98s p(99)=3.69s iterations.....................: 55102 151.994729/s vus............................: 6 min=6 max=389 vus_max........................: 400 min=400 max=400 NETWORK data_received..................: 56 MB 154 kB/s data_sent......................: 15 MB 41 kB/s Run 2 (connection pooling tuning): database.properties: -------------------- # Connection pool configuration (Agroal) # RDS is db.t3.micro (~112 max_connections for PostgreSQL). Three ECS services ship JDBC # (actor-service, audit-service, notification-service); keep sum of (tasks × max-size) well # under the instance cap. Raising toward ~80 per service requires a larger RDS class or # per-service overrides (e.g. env QUARKUS_DATASOURCE_JDBC_MAX_SIZE only on actor-service). quarkus.datasource.jdbc.max-size=34 quarkus.datasource.jdbc.min-size=10 # Fail pool acquisition sooner than 10s so saturation shows up in metrics/alerts instead of long stalls. quarkus.datasource.jdbc.acquisition-timeout=5 quarkus.properties: ------------------- # previously defaulted to 50 quarkus.rest-client.connection-pool-size=200 It did not make a material difference, this is very likely indicative of 1. Downstream latency amplification under concurrency 2. External dependency: Cognito (very likely dominant) 3. Single-task ECS ceiling (hitting vertical scaling limits, not CPU limits) █ TOTAL RESULTS checks_total.......: 68651 189.317018/s checks_succeeded...: 100.00% 68651 out of 68651 checks_failed......: 0.00% 0 out of 68651 ✓ logged in successfully ✓ fetched actor ✓ uploaded document ✓ registered successfully CUSTOM endpoint_actor_get.............: avg=149.34ms min=3.49ms med=104.83ms max=1.58s p(90)=295.92ms p(95)=386.05ms p(99)=599.6ms endpoint_auth_login............: avg=1.01s min=271.96ms med=403.45ms max=8.11s p(90)=3.67s p(95)=6.01s p(99)=7.29s endpoint_auth_register.........: avg=972.26ms min=688.81ms med=903.82ms max=6.5s p(90)=1.09s p(95)=1.2s p(99)=2.08s endpoint_document_upload.......: avg=246.69ms min=39.83ms med=198.58ms max=2.17s p(90)=425.61ms p(95)=588.72ms p(99)=911.95ms HTTP http_req_duration..............: avg=232.05ms min=3.49ms med=135.12ms max=8.11s p(90)=404.35ms p(95)=791.13ms p(99)=1.1s { expected_response:true }...: avg=232.05ms min=3.49ms med=135.12ms max=8.11s p(90)=404.35ms p(95)=791.13ms p(99)=1.1s http_req_failed................: 0.00% 0 out of 68651 http_reqs......................: 68651 189.317018/s EXECUTION iteration_duration.............: avg=1.31s min=1s med=1.11s max=11.7s p(90)=1.48s p(95)=2.98s p(99)=3.79s iterations.....................: 54897 151.387982/s vus............................: 7 min=0 max=242 vus_max........................: 400 min=391 max=400 NETWORK data_received..................: 55 MB 152 kB/s data_sent......................: 15 MB 41 kB/s Run 3 (2x ECS instances; tuned down connection pooling): database.properties: -------------------- quarkus.datasource.jdbc.max-size=14 quarkus.datasource.jdbc.min-size=5 ERRO[0097] [backbone-perf] actor_get status=502 body={"error":"Failed to retrieve actor"} source=console (Single failure is noise, not a system signal - likely transient upstream disconnect/ALB idle timeout/connection reuse edge case) █ TOTAL RESULTS checks_total.......: 80634 222.3531/s checks_succeeded...: 99.99% 80633 out of 80634 checks_failed......: 0.00% 1 out of 80634 ✓ logged in successfully ✗ fetched actor ↳ 99% — ✓ 64332 / ✗ 1 ✓ uploaded document ✓ registered successfully CUSTOM endpoint_actor_get.............: avg=14.07ms min=3.24ms med=6.03ms max=1.09s p(90)=14.89ms p(95)=40.06ms p(99)=200.12ms endpoint_auth_login............: avg=557.34ms min=254.55ms med=307.39ms max=4.24s p(90)=1.01s p(95)=2.76s p(99)=3.91s endpoint_auth_register.........: avg=811.93ms min=671.46ms med=786.63ms max=3.4s p(90)=891.89ms p(95)=957.93ms p(99)=1.35s endpoint_document_upload.......: avg=61.95ms min=37.51ms med=49.64ms max=999.4ms p(90)=87.49ms p(95)=112.68ms p(99)=242.69ms HTTP http_req_duration..............: avg=76.36ms min=3.24ms med=6.56ms max=4.24s p(90)=149.6ms p(95)=403.35ms p(99)=871.86ms { expected_response:true }...: avg=76.36ms min=3.24ms med=6.56ms max=4.24s p(90)=149.58ms p(95)=403.36ms p(99)=871.86ms http_req_failed................: 0.00% 1 out of 80634 http_reqs......................: 80634 222.3531/s EXECUTION iteration_duration.............: avg=1.12s min=1s med=1s max=6.92s p(90)=1.07s p(95)=2.59s p(99)=2.82s iterations.....................: 64333 177.402113/s vus............................: 6 min=0 max=211 vus_max........................: 400 min=391 max=400 NETWORK data_received..................: 65 MB 179 kB/s data_sent......................: 17 MB 48 kB/s Run 4: ERRO[0107] [backbone-perf] actor_get status=502 body={"error":"Failed to retrieve actor"} source=console ERRO[0107] [backbone-perf] actor_get status=502 body={"error":"Failed to retrieve actor"} source=console ERRO[0107] [backbone-perf] actor_get status=502 body={"error":"Failed to retrieve actor"} source=console █ TOTAL RESULTS checks_total.......: 80428 221.788361/s checks_succeeded...: 99.99% 80425 out of 80428 checks_failed......: 0.00% 3 out of 80428 ✓ logged in successfully ✗ fetched actor ↳ 99% — ✓ 64475 / ✗ 3 ✓ uploaded document ✓ registered successfully CUSTOM endpoint_actor_get.............: avg=14.24ms min=3.28ms med=5.99ms max=979.68ms p(90)=15ms p(95)=42.44ms p(99)=204.77ms endpoint_auth_login............: avg=574.08ms min=244.79ms med=304.02ms max=4.52s p(90)=1.55s p(95)=2.81s p(99)=3.79s endpoint_auth_register.........: avg=803.1ms min=676.59ms med=781.77ms max=2.9s p(90)=866.89ms p(95)=924.1ms p(99)=1.24s endpoint_document_upload.......: avg=60.45ms min=37.02ms med=48.58ms max=939.35ms p(90)=86.97ms p(95)=113.85ms p(99)=231.84ms HTTP http_req_duration..............: avg=75.28ms min=3.28ms med=6.5ms max=4.52s p(90)=146.06ms p(95)=364.16ms p(99)=852.79ms { expected_response:true }...: avg=75.27ms min=3.28ms med=6.5ms max=4.52s p(90)=145.71ms p(95)=363.26ms p(99)=852.79ms http_req_failed................: 0.00% 3 out of 80428 http_reqs......................: 80428 221.788361/s EXECUTION iteration_duration.............: avg=1.12s min=1s med=1s max=7.15s p(90)=1.07s p(95)=2.58s p(99)=2.79s iterations.....................: 64478 177.804619/s vus............................: 3 min=3 max=388 vus_max........................: 400 min=400 max=400 NETWORK data_received..................: 64 MB 176 kB/s data_sent......................: 17 MB 47 kB/s Run 5: ERRO[0358] [backbone-perf] actor_get status=502 body={"error":"Failed to retrieve actor"} source=console ERRO[0358] [backbone-perf] auth_register status=400 body={"error":"PostgreSQL registration failed: Received: 'Bad Request, status code 400' when invoking REST Client method: 'io.backbone.core.client.actor.ActorServiceClient#register'"} source=console ERRO[0358] [backbone-perf] auth_login status=401 body={"error":"Login failed for user: testuser1777972441120-960a73f54459f07e@example.com"} source=console ERRO[0359] [backbone-perf] auth_login status=401 body={"error":"Login failed for user: testuser1777972441120-960a73f54459f07e@example.com"} source=console ERRO[0360] [backbone-perf] auth_login status=401 body={"error":"Login failed for user: testuser1777972441120-960a73f54459f07e@example.com"} source=console █ TOTAL RESULTS checks_total.......: 80500 221.974508/s checks_succeeded...: 99.99% 80495 out of 80500 checks_failed......: 0.00% 5 out of 80500 ✗ logged in successfully ↳ 99% — ✓ 3703 / ✗ 3 ✗ fetched actor ↳ 99% — ✓ 64035 / ✗ 1 ✓ uploaded document ✗ registered successfully ↳ 99% — ✓ 3303 / ✗ 1 CUSTOM endpoint_actor_get.............: avg=16.94ms min=3.39ms med=6.11ms max=1.07s p(90)=23.74ms p(95)=60.34ms p(99)=252.48ms endpoint_auth_login............: avg=551.43ms min=44.03ms med=306.09ms max=3.9s p(90)=1.05s p(95)=2.71s p(99)=3.4s endpoint_auth_register.........: avg=810.94ms min=670.85ms med=781.78ms max=3.11s p(90)=886.41ms p(95)=960.03ms p(99)=1.41s endpoint_document_upload.......: avg=64.92ms min=35.65ms med=49.36ms max=927.93ms p(90)=95.59ms p(95)=135.92ms p(99)=292.49ms HTTP http_req_duration..............: avg=79.77ms min=3.39ms med=6.79ms max=3.9s p(90)=216.75ms p(95)=419.11ms p(99)=869.65ms { expected_response:true }...: avg=79.75ms min=3.39ms med=6.79ms max=3.9s p(90)=216.65ms p(95)=418.99ms p(99)=869.42ms http_req_failed................: 0.00% 5 out of 80500 http_reqs......................: 80500 221.974508/s EXECUTION iteration_duration.............: avg=1.12s min=1s med=1s max=6.96s p(90)=1.09s p(95)=2.59s p(99)=2.83s iterations.....................: 64039 176.584168/s vus............................: 5 min=5 max=396 vus_max........................: 400 min=400 max=400 NETWORK data_received..................: 65 MB 180 kB/s data_sent......................: 17 MB 48 kB/s