# Testing Standards

> **Language**: English | [繁體中文](../locales/zh-TW/core/testing-standards.md)

**Version**: 3.2.0
**Last Updated**: 2026-04-20
**Applicability**: All software projects
**Scope**: universal
**Industry Standards**: ISTQB CTFL v4.0, ISO/IEC/IEEE 29119
**References**: [istqb.org](https://istqb.org/)

---

## Purpose

This standard defines actionable testing rules and conventions for AI agents and developers. For theoretical foundations, educational content, and detailed examples, see [Testing Theory Knowledge Base](../skills/testing-guide/testing-theory.md).

**Reference Standards**:
- [ISTQB CTFL v4.0](https://istqb.org/certifications/certified-tester-foundation-level-ctfl-v4-0/)
- [ISO/IEC/IEEE 29119](https://www.iso.org/standard/81291.html)
- [SWEBOK v4.0](https://www.computer.org/education/bodies-of-knowledge/software-engineering)

---

## Glossary

| Abbreviation | Full Term | Description |
|--------------|-----------|-------------|
| **UT** | Unit Testing | Testing individual functions/methods in isolation |
| **IT** | Integration Testing | Testing interactions between components |
| **ST** | System Testing | Testing the complete integrated system |
| **AT** | Acceptance Testing | Testing against business acceptance criteria |
| **E2E** | End-to-End Testing | Testing complete user workflows |
| **UAT** | User Acceptance Testing | Acceptance testing performed by end users |
| **SIT** | System Integration Testing | Testing integration of multiple systems |

> **Note**: "IT" in this document always refers to "Integration Testing", not "Information Technology".

---

## Coverage Targets (Primary Metric)

> **Coverage is the primary metric for test quality.** Higher coverage means more code is protected by tests.
> **覆蓋率是測試品質的主要指標。** 更高的覆蓋率代表更多程式碼受到測試保護。

| Metric | Minimum | Standard | Ideal |
|--------|---------|----------|-------|
| **Line Coverage** | 80% | 90% | 95%+ |
| **Branch Coverage** | 70% | 85% | 90%+ |
| **Function Coverage** | 85% | 95% | 100% |
| **Mutation Score** | — | 80% | 90%+ (critical code) |

**Level definitions:**
- **Minimum**: Baseline for all projects — below this is a quality risk
- **Standard**: Target for most projects — achievable with disciplined testing
- **Ideal**: Target for critical systems and core business logic — strive for 100% where practical

> **Practical guidance**: 100% coverage is the ideal goal. In practice, diminishing returns appear around 95%+ for line coverage. Focus the last 5% on critical paths (authentication, payment, data integrity) rather than generated code or trivial getters/setters.

---

## Coverage vs Ratio — Key Distinction

> **AI agents and developers: do NOT confuse these two concepts.**

| Concept | Meaning | Importance |
|---------|---------|------------|
| **Coverage（覆蓋率）** | Percentage of code executed by tests | **Primary metric** — measures protection |
| **Ratio（佔比）** | Distribution of test count across levels | Reference only — affects execution time |

**Coverage** answers: "How much of my code is tested?"
**Ratio** answers: "What proportion of my tests are unit vs integration vs E2E?"

---

## Testing Framework Selection

| Framework | Levels | Best For |
|-----------|--------|----------|
| **ISTQB** | UT → IT/SIT → ST → AT/UAT | Enterprise, compliance, formal QA |
| **Industry Pyramid** | UT → IT → ST → E2E | Agile, DevOps, CI/CD |

---

## Testing Pyramid (Test Count Ratio — Reference Only)

> **Note**: These are test **count** ratios (how many tests at each level), NOT coverage targets. See [Coverage Targets](#coverage-targets-primary-metric) above for coverage requirements.

| Level | Test Count Ratio | Execution Time Target |
|-------|-----------------|----------------------|
| Unit Testing (UT) | ~70% of tests | < 10 min total |
| Integration Testing (IT) | ~20% of tests | < 30 min total |
| System Testing (ST) | ~7% of tests | < 1 hour total |
| E2E Testing | ~3% of tests | < 2 hours total |

> The 70/20/7/3 ratio is an empirical recommendation (Mike Cohn). It optimizes for fast feedback — most tests run quickly (UT), fewer tests run slowly (E2E).

---

## Test Level Requirements

### Unit Testing (UT)

**Characteristics**: Isolated, Fast (< 100ms each), Deterministic

#### Scope

| Include | Exclude |
|---------|---------|
| Single function/method | Database queries |
| Single class | External API calls |
| Pure business logic | File I/O operations |
| Data transformations | Multi-class interactions |
| Validation rules | Network calls |

#### Naming Convention

**File Naming**:
```
[ClassName]Tests.[ext]      # C#
[ClassName].test.[ext]      # TypeScript/JavaScript
[class_name]_test.[ext]     # Python, Go
```

**Method Naming** (choose ONE per project):

| Style | Best For | Example |
|-------|----------|---------|
| `[Method]_[Scenario]_[Result]` | C#, Java | `CalculateTotal_NegativePrice_ThrowsException()` |
| `should_[behavior]_when_[condition]` | JavaScript/TypeScript | `should_reject_login_when_account_locked()` |
| `test_[method]_[scenario]_[expected]` | Python (pytest) | `test_validate_email_invalid_format_returns_false()` |

#### Coverage Thresholds

> See [Coverage Targets](#coverage-targets-primary-metric) at the top of this document for the authoritative coverage requirements.

---

### Integration Testing (IT)

**Characteristics**: Component integration, Real dependencies (often containerized), 1-10 seconds each

#### When Required

**Decision Rule**: If your unit test uses a wildcard matcher (`any()`, `It.IsAny<>`, `Arg.Any<>`) for a query/filter parameter, that functionality MUST have an integration test.

| Scenario | Reason |
|----------|--------|
| Query predicates | Mocks cannot verify filter expressions |
| Entity relationships | Verify foreign key correctness |
| Composite keys | In-memory DB may differ from real DB |
| Field mapping | DTO ↔ Entity transformations |
| Pagination | Row ordering and counting |
| Transactions | Rollback behavior |

#### Scope

| Include | Exclude |
|---------|---------|
| Database CRUD operations | Full user workflows |
| Repository + Database | Cross-service communication |
| Service + Repository | UI interactions |
| API endpoint + Service layer | |
| Message queue producers/consumers | |
| Cache read/write operations | |

#### Naming Convention

```
[ComponentName]IntegrationTests.[ext]
[ComponentName].integration.test.[ext]
[ComponentName].itest.[ext]
```

---

### System Testing (ST)

**Characteristics**: Complete system, Production-like environment, Requirement-based

#### Scope

| Include | Exclude |
|---------|---------|
| Complete API workflows | UI visual testing |
| Cross-service transactions | User journey simulations |
| Data flow through entire system | A/B testing scenarios |
| Security requirements | |
| Performance under load | |
| Error handling & recovery | |

#### Types

| Type | Description |
|------|-------------|
| Functional | Verify features work as specified |
| Performance | Load, stress, scalability testing |
| Security | Penetration, vulnerability scanning |
| Reliability | Failover, recovery, stability |
| Compatibility | Cross-platform, browser compatibility |

#### Naming Convention

```
[Feature]SystemTests.[ext]
[Feature].system.test.[ext]
[Feature]_st.[ext]
```

---

### End-to-End Testing (E2E)

**Characteristics**: User perspective, Full stack (UI → API → Database), Slowest (30s+ each)

#### Scope

| Include | Exclude |
|---------|---------|
| Critical user journeys | Every possible user path |
| Login/Authentication flows | Edge cases (use UT/IT) |
| Core business transactions | Performance benchmarking |
| Cross-browser functionality | |
| Smoke tests for deployments | |

#### Naming Convention

```
[UserJourney].e2e.[ext]
[Feature].e2e.spec.[ext]
e2e/[feature]/[scenario].[ext]
```

#### E2E Precondition Scope (e2e-precondition-scope)

E2E environment pre-checks (`globalSetup`, `beforeAll`) must verify the health of **all pages and endpoints under test**, not just the authentication entry point.

**Anti-pattern** — login-only health check:
```ts
// ❌ Passes even when feature pages return 500
await page.goto('/login');
expect(response.status()).toBe(200);
```

**Required pattern** — explicit coverage list:
```ts
// ✅ Verify all pages covered by the suite
const PAGES_UNDER_TEST = ['/login', '/dashboard', '/feature-x'];
for (const path of PAGES_UNDER_TEST) {
  const res = await fetch(`${BASE_URL}${path}`);
  expect(res.status).toBeLessThan(500); // fail fast on 5xx
}
```

> **Evidence**: Real incident — E2E `globalSetup` only checked `Login.aspx`; a feature page returned HTTP 500 silently. The full E2E suite passed with false confidence, masking a production crash.

---

## Test Doubles

| Type | Purpose | When to Use |
|------|---------|-------------|
| **Stub** | Returns predefined values | Fixed API responses |
| **Mock** | Verifies interactions | Verify method called |
| **Fake** | Simplified implementation | In-memory database |
| **Spy** | Records calls, delegates to real | Partial mocking |
| **Dummy** | Placeholder, never used | Fill required parameters |

### Usage by Test Level

| Level | Guidance |
|-------|----------|
| **UT** | Use Mocks/Stubs for all external dependencies |
| **IT** | Use Fakes for databases, Stubs for external APIs |
| **ST** | Use real components, Fakes only for external third-party |
| **E2E** | Use real everything; stub only external payment/email |

---

## Mock Limitations

**Problem**: Wildcard matchers (`any()`, `It.IsAny<>`) ignore actual query logic, allowing incorrect queries to pass.

**Rule**: If mocking a method that accepts a query/filter/predicate parameter, you MUST have a corresponding integration test to verify the query logic.

```python
# Example - Python
# ❌ This test cannot verify query correctness
mock_repo.find.return_value = users

# ✓ Add integration test to verify actual query
```

---

## Test Data Requirements

### Principles

1. **Isolation**: Each test manages its own data
2. **Cleanup**: Tests clean up after themselves
3. **Determinism**: Tests don't depend on shared state
4. **Readability**: Test data clearly shows intent

### Distinct Identifiers Rule

When entities have both a surrogate key (auto-generated ID) and a business identifier, test data MUST use different values for each.

```python
# ❌ Wrong: id equals business_code - mapping errors undetected
dept = Department(id=1, business_code=1)

# ✓ Correct: distinct values catch mapping errors
dept = Department(id=1, business_code=1001)
```

### Composite Keys Rule

For entities with composite primary keys, ensure each record has a unique key combination.

```python
# ❌ Key collision
batch1 = BatchRecord(id=0, send_time=now)
batch2 = BatchRecord(id=0, send_time=now)  # Conflict!

# ✓ Unique combinations
batch1 = BatchRecord(id=0, send_time=now + timedelta(seconds=1))
batch2 = BatchRecord(id=0, send_time=now + timedelta(seconds=2))
```

---

## Test Environment

### Language-Specific Tools

| Language | Version Manager | Lock File |
|----------|----------------|-----------|
| Python | venv, virtualenv, poetry | requirements.txt, poetry.lock |
| Node.js | nvm, fnm | package-lock.json, yarn.lock |
| Ruby | rbenv, rvm | Gemfile.lock |
| Java | SDKMAN, jenv | pom.xml, build.gradle.lock |
| .NET | dotnet SDK | packages.lock.json |
| Go | go mod | go.sum |
| Rust | rustup, cargo | Cargo.lock |

### Best Practices

1. **Always use virtual environments** for development and testing
2. **Commit lock files** to version control
3. **Pin versions** in CI/CD pipelines
4. **Document required runtime versions** in README or .tool-versions

### Container Usage by Test Level

| Level | Container Usage |
|-------|-----------------|
| UT | Not needed - use mocks |
| IT | Testcontainers for databases, caches |
| ST | Docker Compose for full environment |
| E2E | Full containerized stack |

---

## CI/CD Integration

### Test Execution Strategy

| Stage | When | Timeout |
|-------|------|---------|
| Unit Test | Every commit | 10 min |
| Integration Test | Every commit | 30 min |
| System Test | PR merge to main | 2 hours |
| E2E Test | Release candidates | 4 hours |

### Required Metrics

| Metric | UT | IT | ST | E2E |
|--------|----|----|----|----|
| Pass/Fail Count | Required | Required | Required | Required |
| Execution Time | Required | Required | Required | Required |
| Coverage % | Required | Required | Optional | Not needed |
| Flaky Test Rate | Required | Required | Required | Required |
| Screenshots/Videos | Not needed | Not needed | Optional | Required |

---

## Best Practices

### AAA Pattern

```
// Arrange - Set up test data and environment
// Act - Execute the behavior under test
// Assert - Verify the result
```

### FIRST Principles

| Principle | Description |
|-----------|-------------|
| **F**ast | Tests run quickly |
| **I**ndependent | Tests don't affect each other |
| **R**epeatable | Same result every time |
| **S**elf-validating | Clear pass/fail |
| **T**imely | Written with production code |

### Anti-Patterns to Avoid

- Test Interdependence (tests must run in specific order)
- Flaky Tests (sometimes pass, sometimes fail)
- Testing Implementation Details (tests break on refactoring)
- Over-Mocking (nothing real is tested)
- Missing Assertions (tests verify nothing meaningful)
- Magic Numbers/Strings (unexplained values)
- Identical Test IDs (same values for surrogate and business keys)

---

## Test Documentation Structure

### tests/README.md Required Sections

Every `tests/` directory SHOULD include a README.md with:

#### 1. Test Overview Table

| Test Type | Count | Framework | Environment |
|-----------|-------|-----------|-------------|
| Unit Tests | 150 | Jest | Node.js |
| Integration Tests | 45 | Jest | Node.js + TestContainers |
| E2E Tests | 12 | Playwright | Browser |

#### 2. Current Status Section

| Metric | Value | Target | Status |
|--------|-------|--------|--------|
| Pass Rate | 98.5% | >= 95% | Pass |
| Line Coverage | 82% | >= 80% | Pass |
| Branch Coverage | 75% | >= 70% | Pass |

#### 3. Report Links Section

| Report Type | Location | Description |
|-------------|----------|-------------|
| Test Results | `results/` | Timestamped execution reports |
| Coverage | `coverage/` | Code coverage reports |
| Gap Analysis | `docs/gap-analysis.md` | Missing coverage analysis |

### Test Report Naming Convention

| Item | Convention | Example |
|------|------------|---------|
| Report filename | `test-report-YYYYMMDD-HHMMSS.md` | `test-report-20260129-143000.md` |
| Report directory | `tests/results/` | |
| Coverage directory | `tests/coverage/` | |

### Directory Structure

```
tests/
├── README.md                    # Test overview and status
├── results/                     # Test execution reports
├── coverage/                    # Coverage reports
├── docs/                        # Test documentation
├── unit/                        # Unit tests
├── integration/                 # Integration tests
└── e2e/                         # End-to-end tests
```

---

## Exploratory Testing

> **Reference**: ISTQB CTFL v4.0 §4.4, James Bach's Session-Based Test Management (SBTM)

Exploratory testing is a structured approach where test design, execution, and learning happen simultaneously. It complements automated testing by discovering unknown defects that scripted tests cannot anticipate.

### Session-Based Test Management (SBTM)

Each exploratory testing session follows the SBTM framework:

| Element | Description | Requirement |
|---------|-------------|-------------|
| **Time Box** | Fixed session duration of 60-90 minutes | Required |
| **Charter** | Clear exploration goal, test area, and expected risks | Required |
| **Session Notes** | Structured record of steps, observations, deviations, and issues | Required |
| **Debrief** | Post-session review with team or test lead | Recommended |

**Charter format**:
```
Explore [target area]
With [resources/techniques]
To discover [expected information/risks]
```

### Heuristics (SFDPOT)

Use the SFDPOT mnemonic (James Bach's Heuristic Test Strategy Model) to guide exploration across six dimensions:

| Dimension | Focus Area | Example Questions |
|-----------|-----------|-------------------|
| Structure | System components and their relationships | What modules exist? How are they connected? |
| Function | Features and capabilities the system provides | What does each feature do? What are the edge cases? |
| Data | Input/output data, boundary values, formats | What data types are accepted? What happens at boundaries? |
| Platform | OS, browser, hardware, environment differences | Does it work on all supported platforms? |
| Operations | Installation, configuration, maintenance, monitoring | Can it be installed cleanly? How is it maintained? |
| Time | Concurrency, timeouts, scheduling, time zones | What happens under concurrent access? How are timeouts handled? |

> **Tip**: Use at least 3 SFDPOT dimensions per session to ensure broad coverage.

### Session Record Template

Every exploratory testing session MUST produce a session record using this template:

| Field | Description | Required |
|-------|-------------|----------|
| **Charter** | The exploration goal and scope | Yes |
| **Area** | The functional area or component tested | Yes |
| **Duration** | Actual time spent (within 60-90 min time box) | Yes |
| **Notes** | Steps taken, observations, deviations from charter | Yes |
| **Bugs Found** | List of defects discovered with severity | Yes |
| **Follow-up** | Action items: new charters, automation candidates, questions | Yes |

**Example session record**:
```markdown
## Session Record
- **Charter**: Explore login flow with invalid credentials to discover error handling gaps
- **Area**: Authentication module
- **Duration**: 75 minutes
- **Notes**: Tested SQL injection, XSS in username, Unicode characters, empty fields...
- **Bugs Found**: 2 (BUG-101: XSS not sanitized in error message, BUG-102: No rate limiting)
- **Follow-up**: Write automated regression for BUG-101, new charter for password reset flow
```

### Automation Complement

Exploratory testing and automated testing serve complementary roles:

| Aspect | Exploratory Testing | Automated Testing |
|--------|-------------------|-------------------|
| **Purpose** | Discover unknown defects | Protect against known regressions |
| **Timing** | New features, risk areas, pre-release | Every build, continuous integration |
| **Strength** | Creativity, adaptability, context | Speed, repeatability, coverage |
| **Cost** | Human effort per session | Maintenance cost over time |

**The Discovery-to-Protection cycle**:

1. **Explore** — Conduct exploratory session, discover new defects
2. **Report** — Document bugs found in session record
3. **Automate** — Convert confirmed bugs into automated regression tests
4. **Protect** — Automated tests prevent recurrence in future builds
5. **Repeat** — New exploratory sessions focus on unexplored areas

> **Rule**: Every bug discovered through exploratory testing SHOULD have a corresponding automated regression test added within the same sprint. This transforms one-time discoveries into permanent protection.

---

## Related Standards

- [Testing Theory Knowledge Base](../skills/testing-guide/testing-theory.md) - Educational content, examples, techniques
- [Test-Driven Development](test-driven-development.md) - TDD/BDD/ATDD methodology
- [Test Completeness Dimensions](test-completeness-dimensions.md) - 8-dimension test coverage
- [Spec-Driven Development](spec-driven-development.md) - SDD workflow integration
- [Code Check-in Standards](checkin-standards.md)
- [Code Review Checklist](code-review-checklist.md)
- [Deployment Standards](deployment-standards.md) - Test requirements for deployment readiness

---

## Version History

| Version | Date | Changes |
|---------|------|---------|
| 3.1.0 | 2026-03-24 | **Coverage-first restructure**: Elevated Coverage Targets to primary position, raised thresholds (Line 80/90/95+, Branch 70/85/90+, Function 85/95/100), added Coverage vs Ratio distinction, demoted Testing Pyramid to reference-only |
| 3.0.0 | 2026-01-29 | **Major refactor**: Split into Rules (this file) and Theory (testing-theory.md). Reduced from 141KB/3185 lines to ~12KB/350 lines. All educational content moved to skills/testing-guide/testing-theory.md. Rules-only format optimized for AI agent consumption. |
| 2.2.0 | 2026-01-20 | Added Test Documentation Structure section |
| 2.1.0 | 2026-01-05 | Added SWEBOK v4.0 reference, Testing Fundamentals, Test-Related Measures |
| 2.0.0 | 2026-01-05 | Major update aligned with ISTQB CTFL v4.0 and ISO/IEC/IEEE 29119 |
| 1.3.0 | 2025-12-29 | Add Testing Framework Selection, IT/SIT abbreviation clarification |
| 1.2.0 | 2025-12-19 | Add Mock Limitations, Integration Test requirements, Test Data patterns |
| 1.1.1 | 2025-12-11 | Improved System test example with generic domain concepts |
| 1.1.0 | 2025-12-05 | Add test environment isolation section |
| 1.0.0 | 2025-12-05 | Initial testing standards with UT/IT/ST/E2E coverage |

---

## License

This standard is released under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).

---

**Maintainer**: Development Team