Your stress test reveals a classic system bottleneck: throughput **plateaus** and **decreases** under load despite low CPU usage. This strongly suggests the slowdown comes from **contention for shared resources** or **synchronization waits**, not CPU-bound processing. The 1.9 minutes of CPU time across 8 minutes means your CPUs are idle ~76% of the time—likely **waiting** for something. ```js import {check, group, sleep} from "k6"; import http from "k6/http"; import {Counter, Trend} from 'k6/metrics'; import {randomIntBetween} from "https://jslib.k6.io/k6-utils/1.4.0/index.js"; import { cleanupSession, generateTestUser, generateUserIndex, jsonHeaders, log, logDetailedFailure, logOne } from "./utils.js"; // ---------------------------------------------------------------------------------------------------- // CONFIGURATION // ---------------------------------------------------------------------------------------------------- export const options = { stages: [ // STAGE 0: Test (10-100 VUs) - UNCOMMENT AFTER PREVIOUS STAGE SUCCESS // {duration: '10s', target: 5}, // {duration: '30s', target: 15}, // Gentle ramp-up // {duration: '1m', target: 50}, // Moderate load // {duration: '30s', target: 100}, // Stress test // {duration: '10s', target: 10}, // Cool down // STAGE 1: Baseline (10-100 VUs) - UNCOMMENT AFTER PREVIOUS STAGE SUCCESS // {duration: '1m', target: 10}, // {duration: '2m', target: 25}, // Gentle ramp-up // {duration: '2m', target: 50}, // Moderate load // {duration: '2m', target: 100}, // Stress test // {duration: '30s', target: 10}, // Cool down // // // STAGE 2: 2x Load (20-200 VUs) - UNCOMMENT AFTER PREVIOUS STAGE SUCCESS // {duration: '1m', target: 20}, // {duration: '2m', target: 50}, // Gentle ramp-up // {duration: '2m', target: 100}, // Moderate load // {duration: '2m', target: 200}, // Stress test // {duration: '30s', target: 20}, // Cool down // // // STAGE 3: 5x Load (50-500 VUs) - UNCOMMENT AFTER PREVIOUS STAGE SUCCESS {duration: '1m', target: 50}, {duration: '2m', target: 125}, // Gentle ramp-up {duration: '2m', target: 250}, // Moderate load {duration: '2m', target: 500}, // Stress test {duration: '30s', target: 50}, // Cool down // // // STAGE 4: 10x Load (100-1000 VUs) - UNCOMMENT AFTER PREVIOUS STAGE SUCCESS // {duration: '1m', target: 100}, // {duration: '2m', target: 250}, // Gentle ramp-up // {duration: '2m', target: 500}, // Moderate load // {duration: '2m', target: 1000}, // Stress test // {duration: '30s', target: 100}, // Cool down // // // STAGE 5: 50x Load (50-5000 VUs) - UNCOMMENT AFTER PREVIOUS STAGE SUCCESS // {duration: '1m', target: 50}, // {duration: '2m', target: 1250}, // Gentle ramp-up // {duration: '2m', target: 2500}, // Moderate load // {duration: '2m', target: 5000}, // Stress test // {duration: '30s', target: 50}, // Cool down // // // STAGE 6: 100x Load (1000-10,000 VUs) - UNCOMMENT AFTER PREVIOUS STAGE SUCCESS // {duration: '1m', target: 1000}, // {duration: '2m', target: 2500}, // Gentle ramp-up // {duration: '2m', target: 5000}, // Moderate load // {duration: '2m', target: 10000}, // Stress test // {duration: '30s', target: 1000}, // Cool down ], thresholds: { http_req_failed: ['rate<0.01'], // Should be near 0% http_req_duration: ['p(95)<5000'], // Should be fast // Custom metrics 'http_reqs{expected_response:true}': ['rate>0.9'], // 90% success rate 'step_identify_duration': ['p(95)<3000'], 'step_civil_registry_duration': ['p(95)<3000'], 'step_password_set_duration': ['p(95)<3000'], 'step_fill_extra_duration': ['p(95)<3000'], 'step_final_verify_duration': ['p(95)<3000'], }, // vus: 1, // iterations: 1, }; // const BASE_URL = "https://cas.rso-co.ir"; const BASE_URL = "http://localhost:8000"; const APP = "app-built-in"; const ORG = "built-in"; const RESPONSE_LIMIT_MS = 30000; const MAX_RETRIES = 3; const TOTAL_USERS = 1000000; const CONCURRENT_USERS = 2000; // Custom metrics for each step const identifyTime = new Trend('step_identify_duration', true); const civilRegistryTime = new Trend('step_civil_registry_duration', true); const passwordSetTime = new Trend('step_password_set_duration', true); const fillExtraTime = new Trend('step_fill_extra_duration', true); const finalVerifyTime = new Trend('step_final_verify_duration', true); // Counters for success/failure per step const identifySuccess = new Counter('step_identify_success'); const identifyFailure = new Counter('step_identify_failure'); const civilRegistrySuccess = new Counter('step_civil_registry_success'); const civilRegistryFailure = new Counter('step_civil_registry_failure'); const passwordSetSuccess = new Counter('step_password_set_success'); const passwordSetFailure = new Counter('step_password_set_failure'); const fillExtraSuccess = new Counter('step_fill_extra_success'); const fillExtraFailure = new Counter('step_fill_extra_failure'); const finalVerifySuccess = new Counter('step_final_verify_success'); const finalVerifyFailure = new Counter('step_final_verify_failure'); // ---------------------------------------------------------------------------------------------------- // SETUP AND TEARDOWN // ---------------------------------------------------------------------------------------------------- // Setup runs once before the test starts export function setup() { log(`🕐 ====== TEST STARTED: ${new Date().toString()} ======`); return {}; } // Teardown runs once after the test ends export function teardown() { log(`🕐 ====== TEST ENDED: ${new Date().toString()} ======`); } // ---------------------------------------------------------------------------------------------------- // MAIN // ---------------------------------------------------------------------------------------------------- export default function () { const user = generateTestUser(generateUserIndex(TOTAL_USERS, CONCURRENT_USERS)); log(`🚀 Starting flow for user: ${user.username}`); try { runSmartLogin(user); cleanupSession(BASE_URL); log(`✅ Finished: ${user.username}`); log("-----------------------------"); } catch (err) { log(`❌ Flow failed for ${user.username}: ${err.message}`); } sleep(randomIntBetween(1, 3)); } // ---------------------------------------------------------------------------------------------------- // SMART LOGIN FLOW // ---------------------------------------------------------------------------------------------------- function runSmartLogin(user) { const steps = [ {name: "Identify", build: buildIdentify}, {name: "Civil Registry", build: buildCivil}, {name: "Password Set", build: buildPassword}, {name: "Fill Extra", build: buildFill}, {name: "Final Verify", build: buildFinal}, ]; let stateId = ""; for (const [i, step] of steps.entries()) { const result = runStep(i + 1, step.name, step.build(user, stateId), user.nationalId); if (!result.ok) throw new Error(result.error); if (result.stateId) stateId = result.stateId; } } function runStep(stepNum, name, reqBody, nationalId) { const tag = name.toLowerCase().replace(/ /g, "_"); const url = `${BASE_URL}/api/login`; return group(`Step ${stepNum} - ${name}`, () => { for (let attempt = 1; attempt <= MAX_RETRIES; attempt++) { const start = Date.now(); const res = http.post(url, JSON.stringify(reqBody), { headers: jsonHeaders, tags: { step: tag, vu: __VU.toString(), iter: __ITER.toString(), scenario: 'smart_login_registration' }, timeout: '30s', }); const duration = Date.now() - start; let data = null; try { data = JSON.parse(res.body); // logOne(`response from ${stepNum}.${name}: ${JSON.stringify(data, null, " ")}`); } catch (_) { } const okStatus = res.status === 200; const okSpeed = res.timings.duration < RESPONSE_LIMIT_MS; const okApp = isBusinessSuccess(data); let failureType = "unknown"; if (!okStatus) failureType = "http_error"; else if (!okSpeed) failureType = "timeout"; else if (!okApp) failureType = "business_logic"; recordStepMetrics(name, duration, okStatus && okSpeed && okApp); if (!okApp || !okSpeed) { logOne("Business logic failed:"); logDetailedFailure(name, res, data, failureType); } const passed = check(res, { [`${name} status 200`]: () => okStatus, [`${name} < ${RESPONSE_LIMIT_MS}`]: () => okSpeed, [`${name} business ok`]: () => okApp, }); if (passed) { logOne(`✔️ Step ${stepNum} '${name}' ok (${duration}ms) for '${nationalId}'`); return {ok: true, stateId: data?.data?.state_id || ""}; } logOne(`❌ Step ${stepNum} ${name} failed (attempt ${attempt}/${MAX_RETRIES})`); logDetailedFailure(name, res, data); // Different backoff for timeout vs business errors const backoff = failureType === "timeout" ? 3 + attempt : 1 + attempt; sleep(backoff); } return {ok: false, error: `${name} failed after retries`}; }); } // ---------------------------------------------------------------------------------------------------- // UTILITIES // ---------------------------------------------------------------------------------------------------- function buildIdentify(user, stateId) { return baseReq(user, stateId, {identifier: user.username}); } function buildCivil(user, stateId) { return baseReq(user, stateId, { birthdate: user.birthdate, nationalId: user.nationalId, phone: user.phone, is_returning: false, jalali_birthdate: user.jalaliBirthdate }); } function buildPassword(user, stateId) { return baseReq(user, stateId, { password: user.password, confirm_password: user.confirmPassword, is_returning: false }); } function buildFill(user, stateId) { return baseReq(user, stateId, {"Face ID": "hi!"}, true); } function buildFinal(user, stateId) { return baseReq(user, stateId, {password: user.password}, true); } function baseReq(user, stateId, stepData, autoSignin = false) { return { application: APP, organization: ORG, username: user.username, signinMethod: "Password", type: "login", smartLogin: true, autoSignin, language: "fa", stepData, stateId }; } function messageIndicatesExistingSession(data) { const msg = data?.data?.message || ""; return msg.includes("Authenticated via existing session"); } function actionIndicatesProceed(data) { const msg = data?.data?.action || ""; return msg.includes("proceed") || msg.includes("complete") || msg.includes("completed"); } function isBusinessSuccess(data) { // Check overall status first if (data?.status !== "ok") { return false; } // Check data status - if it exists and is "error", fail const dataStatus = data?.data?.status; if (dataStatus === "error") { return false; } // Check action for progression indicators const action = data?.data?.action || ""; const isProceeding = action === "proceed" || action === "complete" || action === "completed"; // Allow existing session messages const hasExistingSession = messageIndicatesExistingSession(data); // Success if we're proceeding OR have existing session OR data status is ok return isProceeding || hasExistingSession || dataStatus === "ok"; } function recordStepMetrics(stepName, duration, success) { switch (stepName) { case "Identify": identifyTime.add(duration); if (success) identifySuccess.add(1); else identifyFailure.add(1); break; case "Civil Registry": civilRegistryTime.add(duration); if (success) civilRegistrySuccess.add(1); else civilRegistryFailure.add(1); break; case "Password Set": passwordSetTime.add(duration); if (success) passwordSetSuccess.add(1); else passwordSetFailure.add(1); break; case "Fill Extra": fillExtraTime.add(duration); if (success) fillExtraSuccess.add(1); else fillExtraFailure.add(1); break; case "Final Verify": finalVerifyTime.add(duration); if (success) finalVerifySuccess.add(1); else finalVerifyFailure.add(1); break; } } ``` Since you already have Pyroscope set up, it's the perfect tool to find the exact bottleneck. ### 🔍 Focus Your Pyroscope Analysis on These Key Areas Your low CPU time points to waiting. Here's a targeted strategy to use Pyroscope to find out *what* your Go code is waiting for. The table below outlines what to look for and how to find it. | **Analysis Area** | **What to Look for in Pyroscope (Flame Graph)** | **Likely Underlying Cause** | | :--- | :--- | :--- | | **External Dependencies** | Significant time spent in: `net/http` (client), `database/sql`, Redis client functions. | **Slow Database/Redis** or connection pool exhaustion. A query or network call everyone is waiting for. | | **Synchronization & Locks** | High presence of functions like `sync.(*Mutex).Lock`, `runtime.selectgo`, or `chan` operations. | **Lock contention** (e.g., global cache mutex) or **channel bottlenecks** causing Goroutines to block. | | **System Calls & I/O** | Time spent in `syscall.Read`/`Write`, `runtime.netpoll`, or the internal Go scheduler. | The application is **I/O-bound**, waiting on network or disk, not CPU. | ### ⚙️ Next Steps: Investigate and Confirm the Root Cause 1. **Generate and Analyze the Profile**: Run a new **~2-minute stress test at 200 VUs** (where slowdowns start) while Pyroscope is actively profiling your Go application. Export the flame graph from the test period. 2. **Interpret the Graph**: Look for the **widest horizontal bars** at the top of the flame graph. This is where your program spends the most "wall-clock" time, which includes waiting. Zoom into those functions using Pyroscope's interface. 3. **Cross-Check Dependencies**: Use Redis's built-in tools to check its performance during the load test: - Check latency: `redis-cli --latency` - Run an intrinsic latency check: `redis-cli --intrinsic-latency 10` - Benchmark it: `redis-benchmark -q -n 100000 -c 50` (simulate load) 4. **Check Your Go Server Internals**: If you can modify your Beego app, consider enabling its built-in Admin service to get per-route performance statistics (QPS, average request time). Based on your current architecture, the most probable culprits are **lock contention on your in-memory cache** or **overloaded Redis/database connections**. I hope this targeted approach helps you pinpoint the issue quickly. If you can share a screenshot or description of the most prominent functions from your Pyroscope flame graph during the load test, I can help you interpret them.

Stress Testing with K6