--- name: cross-browser-testing description: >- Design analytics-driven browser test matrices and execute cross-browser tests. Covers BrowserStack/Sauce Labs configuration, Playwright browser channels, common cross-browser CSS/JS divergences, a known-issues documentation log, and progressive enhancement validation. Use when: "cross-browser," "browser matrix," "BrowserStack," "Safari issues," "browser compatibility," "Edge," "works in Chrome but not Safari." Not for: pixel-level baseline strategy and threshold tuning — use visual-testing; device-farm testing of native/hybrid apps — use mobile-testing. Related: visual-testing, playwright-automation, ci-cd-integration, mobile-testing. license: MIT metadata: author: kindlmann version: "2.0" category: specialized --- Chrome-only testing gives false confidence: a layout that works in Chromium can break in WebKit, a clipboard call that succeeds in Chrome silently no-ops in Firefox, and a partitioned-cookie flow can pass everywhere except the one engine your users are on. This skill produces an analytics-driven browser matrix, a Playwright (or cloud-platform) config that runs it, and a committed log of known browser divergences — each verified by a test that asserts the user outcome, not the CSS. ## Quick Route | Situation | Go to | |-----------|-------| | Need to decide *which* browsers to test | Browser Matrix Design | | Already on Playwright, just add browsers | Playwright Browser Configuration → `references/playwright-and-cloud-config.md` | | Need real Safari/Windows/older OS, not engines | Cloud Platform Setup | | One browser misbehaves; want a test for it | Common Cross-Browser Issues + `browserName` branch in `references/testing-patterns.md` | | Need to record a divergence so it is not re-debugged | Known-Issues Log | | Pixel diffs / baseline thresholds | use `visual-testing` | ## Discovery Questions Check `.agents/qa-project-context.md` first — if it exists, use it and skip anything already answered there. Then: 1. **Target browsers from analytics:** What do actual users use? Pull browser/OS data from your analytics tool. Testing browsers nobody uses is waste; missing a browser 15% of users rely on is a bug. 2. **Desktop and mobile?** Mobile Safari on iOS and Chrome on Android render differently than their desktop counterparts. Treat them as separate matrix entries. 3. **Cloud platform:** BrowserStack, Sauce Labs, LambdaTest, or local engines only? Cloud platforms provide real branded browsers and OSes; Playwright's bundled engines cover Chromium, Firefox, and WebKit (not Chrome/Safari themselves). 4. **Progressive enhancement or pixel-perfect?** Progressive enhancement accepts graceful degradation. Pixel-perfect demands identical rendering. The answer determines pass/fail criteria. 5. **Existing Playwright config?** If the project already uses Playwright, cross-browser testing is a configuration change, not a new tool. --- ## Core Principles 1. **Analytics-driven matrix.** Test what your users actually use. A browser at 0.3% traffic does not need the same investment as one at 40%. Check analytics quarterly — browser share shifts. 2. **Progressive enhancement over pixel-perfect.** Identical rendering across all browsers is neither achievable nor necessary. Define what "works" means: core functionality operates, content is accessible, layout is usable. Visual differences in shadows, gradients, or animation timing are acceptable. 3. **Safari and Firefox surface the most cross-browser bugs.** Chrome-only testing catches Chrome bugs. Safari's WebKit engine and Firefox's Gecko engine have the most behavioral differences from Chromium. Prioritize them. 4. **Test functionality, not rendering-engine internals.** A cross-browser test should verify that the user can complete a task, not that a CSS property renders identically. Visual comparison tools handle pixel-level differences. 5. **Engines are not brands.** Playwright's WebKit is *not* Safari and its Chromium is *not* Chrome — they share an engine, not the shipped product (codecs, fonts, enterprise policy, update cadence all differ). Report "WebKit coverage," not "Safari coverage," unless you ran real Safari on a cloud grid. 6. **One test, multiple browsers.** Write tests once. Run them across browser configurations. Never duplicate test logic for different browsers. --- ## Browser Matrix Design ### Analytics-Based Methodology ``` Step 1: Export browser/OS data from analytics (last 90 days) Step 2: Rank by session share Step 3: Group into tiers Step 4: Assign test coverage per tier Step 5: Review quarterly ``` ### Tier System | Tier | Criteria | Coverage | When to run | |------|----------|----------|-------------| | **P0** | >10% traffic share | Full test suite | Every PR, every deploy | | **P1** | 3-10% traffic share | Smoke + critical paths | Nightly, pre-release | | **P2** | 1-3% traffic share | Smoke tests only | Weekly, pre-release | | **Skip** | <1% traffic share | Not tested | Manual spot-check if reported | ### Example Matrix (derived from analytics) ```markdown ## Browser Matrix — Q1 2026 (next-review: 2026-04-01) | Browser | Version | Platform | Traffic % | Tier | Notes | |---------|---------|----------|-----------|------|-------| | Chrome | Latest | Windows | 34% | P0 | | | Chrome | Latest | macOS | 12% | P0 | | | Safari | Latest | macOS | 11% | P0 | WebKit-specific issues | | Chrome | Latest | Android | 15% | P0 | Mobile viewport | | Safari | Latest | iOS | 14% | P0 | Mobile Safari quirks | | Firefox | Latest | Windows | 5% | P1 | Gecko rendering | | Edge | Latest | Windows | 4% | P1 | Chromium-based but different UA/policy | | Samsung Internet | Latest | Android | 3% | P1 | Chromium fork, lagging engine | | Firefox | Latest | macOS | 1.5% | P2 | | | Chrome | N-1 | Windows | 1.2% | P2 | Previous major version | ``` ### Version Coverage Strategy - **Latest:** Always test current stable release. - **Latest - 1:** Test previous major version only for P0 browsers where analytics show >1% on older versions. - **Extended Support Release (ESR):** Test Firefox ESR only if enterprise users are a significant segment. - **Do not test:** Beta/Canary/Nightly releases unless you are a browser vendor or building browser-facing tools. --- ## Playwright Browser Configuration Playwright ships three browser *engines* — Chromium, Firefox, WebKit — so no cloud platform is needed for basic engine-level coverage. This is engine coverage, not brand coverage: bundled WebKit ≠ Safari and bundled Chromium ≠ Chrome (see Core Principle 5). Define one project per matrix entry, map mobile devices via `devices[...]`, and drive locally installed branded browsers with the `channel` option. See `references/playwright-and-cloud-config.md` for the full `playwright.config.ts` project list, branded-channel snippets, and `--project` run commands. **When to use channels:** When you need real branded behavior that differs from the bundled engine — installed Chrome (`channel: 'chrome'`) or Edge (`channel: 'msedge'`) for extension support, enterprise policy, or codecs. WebKit and Firefox have no channel option; they are always Playwright's bundled engines. Note the `edge` project in the config and the `msedge` channel snippet are illustrative alternatives, not two projects to merge — a config needs one `edge` project, not both. **`page.screencast()` (Playwright 1.59+, current in 1.60)** captures annotated video of a cross-browser run — useful when a matrix failure needs human review across engines. For agent-driven re-runs and stepping through a failure, use `--ui` (UI mode) or `--debug` (Inspector); `PWDEBUG=1` and `--headed` are the other real entry points. There is no `--debug=cli` flag. --- ## Cloud Platform Setup Cloud platforms (BrowserStack, Sauce Labs) provide real branded-browser/OS instances Playwright connects to over a CDP/Playwright WebSocket endpoint. Pass credentials and capabilities via environment variables, and keep the platform's `playwrightVersion` aligned with the Playwright version in `package.json` (currently 1.60.x — a client/server mismatch causes socket errors). **BrowserStack now recommends** the `npx browserstack-node-sdk` runner plus a `client.playwrightVersion` capability (in addition to `browserstack.playwrightVersion`) to keep the client and grid sockets in lock-step. The raw `wsEndpoint`/CDP config below still works for direct connections; use the SDK path for new setups. See `references/playwright-and-cloud-config.md` for the BrowserStack config (with the `client.playwrightVersion` cap), the Sauce Labs config, and the GitHub Actions parallel matrix that fans out across cloud browsers. --- ## Common Cross-Browser Issues Real divergences that surface in cross-browser testing, with detection patterns and fixes. The CSS workarounds and Playwright tests for each are in `references/common-browser-issues.md`, covering: partitioned cookies / CHIPS in iframes, ``, the Clipboard API, `scroll-behavior`, `backdrop-filter`, the `` element, View Transitions, and Web Animations timing. ### Modern Cross-Browser Gotchas (2026) The classic Safari-laggard list is mostly resolved (flexbox `gap`, `:has()` shipping, same-document View Transitions are all Baseline). Today's real divergences: - **Partitioned cookies / partitioned storage:** Chrome's CHIPS (`Partitioned` attribute), Safari's ITP, and Firefox's State Partitioning each behave differently for embedded third-party contexts. Test third-party cookies *in an iframe per engine*, not just "the browser supports cookies." See the runnable per-engine iframe test in `references/common-browser-issues.md`. - **`:has()` selector performance:** Universally supported since 2023, but a `:has()`-heavy page can have very different style-recalc cost across engines. Watch list — profile if a page feels janky in one engine; visual-regression it in `visual-testing`. - **View Transitions API:** Same-document transitions are Baseline (Chrome 111, Safari 18, Firefox 144 — Oct 2025), so they are no longer a divergence. **Cross-document** transitions are still the gap: Chrome 126+, Safari 18.2+, Firefox behind a flag. Treat cross-document as progressive enhancement and verify the no-transition fallback. - **WebDriver BiDi:** Production-ready in Selenium 4, partially supported in Playwright. For new cross-runner projects, BiDi is the convergence point. Watch list. --- ## Known-Issues Log When a divergence is real and you cannot fix the app immediately, record it in a committed file (`docs/browser-issues.md`) so it is not re-debugged from scratch. The table is the artifact `Done When` checks for, and every row's test must assert the **user outcome, not the CSS property**: ```markdown | Affected browser | Repro | Workaround / fallback / ticket | Test asserts (user outcome, not CSS) | |------------------|-------|--------------------------------|--------------------------------------| | Safari (WebKit) ≤17 | scroll-behavior: smooth is partial | rely on anchor nav; no JS scroll dependency | anchor link puts heading in viewport (`toBeInViewport`) | | Firefox ≤102 | backdrop-filter unsupported | -webkit- prefix + rgba background fallback | overlay readable; modal content visible | | Firefox (current) | cross-document View Transitions flagged off | progressive enhancement; instant nav fallback | navigation completes; target page heading visible | ``` Keep one row per divergence. A row with no ticket and no fallback is an open bug, not a documented issue. --- ## Testing Patterns The core patterns and the rules that govern them: - **Same test, multiple browsers** — the default. Write the test once; configure projects to run it everywhere. Never duplicate test logic per browser. - **Browser-specific test logic** — branch on `browserName` only when behavior *genuinely* differs (the WebKit date-input fallback and Chromium-only clipboard permission are real cases). **Rule:** keep this rare. Many browser branches signal application compatibility bugs to fix, not work around. - **Visual cross-browser comparison** — `toHaveScreenshot` with a `maxDiffPixelRatio` tolerance; each browser project generates its own baseline (`homepage-chromium.png`, `homepage-webkit.png`, …). For threshold strategy and baseline management, use `visual-testing`. - **Progressive enhancement validation** — abort script requests (Chromium only) and verify core functionality still works via native HTML. See `references/testing-patterns.md` for the runnable code for all four patterns. --- ## Anti-Patterns **Testing only on Chrome.** Chrome is the largest desktop share but uses the same engine as Edge, Opera, and Brave. Safari (WebKit) and Firefox (Gecko) surface the real cross-browser issues. Chrome-only testing gives false confidence. **Reporting WebKit/Chromium as Safari/Chrome.** Bundled engines share rendering, not the shipped browser. Claiming "Safari coverage" off a WebKit project hides codec, font, and policy bugs that only real Safari shows. **Testing every browser equally.** A browser at 1% traffic share does not need the same investment as one at 30%. Use the tier system to allocate effort proportionally. **Duplicating tests per browser.** Write tests once, run them across browser projects via configuration. A `checkout.chrome.spec.ts` and `checkout.safari.spec.ts` with identical logic is the wrong shape. **`browserName` checks everywhere.** Excessive browser branching in tests signals application compatibility issues. Fix the app, do not work around it in tests. **Pixel-perfect assertions without tolerance.** Font rendering, anti-aliasing, and sub-pixel rounding differ between browsers and platforms. Use `maxDiffPixelRatio` or `maxDiffPixels` in visual comparisons. **Ignoring mobile browsers.** Mobile Chrome and mobile Safari are not their desktop counterparts — different viewport behavior, touch handling, and CSS support. Test them as separate matrix entries. **Static browser matrix.** Browser usage changes. A matrix based on data from two years ago is wrong. Review analytics quarterly and update the `next-review` date. **Documenting a divergence with no fallback or ticket.** A known-issues row that lists no workaround and no open ticket is an undocumented bug pretending to be documented. --- ## Failure Modes | Symptom | Likely cause | Fix or check | |---------|--------------|--------------| | Cloud tests fail with a socket/handshake error | Grid Playwright version ≠ local | Set `playwrightVersion`/`client.playwrightVersion` to match `npx playwright --version`; use the `browserstack-node-sdk` runner | | Clipboard test passes in Chromium, fails in Firefox/WebKit | `grantPermissions` only works in Chromium | Assert UI feedback (`Copied!`), not the clipboard API; gate `grantPermissions` on `browserName === 'chromium'` | | Progressive-enhancement test errors in Firefox/WebKit | Script-abort route interception is Chromium-only | Gate the route on `browserName === 'chromium'`; skip the JS-disabled assertion elsewhere | | WebKit project "passes" but real users on Safari report breakage | WebKit engine ≠ shipped Safari | Add a real-Safari row on a cloud grid for the affected flow | | Visual baseline diff explodes for one browser only | Single baseline shared across browsers | Generate per-project baselines; each browser keeps its own `*-.png` | | `:has()`-heavy page janky in one engine only | Style-recalc cost differs by engine | Profile in that engine; reduce `:has()` scope; visual-regress in `visual-testing` | --- ## Done When - Browser matrix defined using real analytics data (last 90 days), with tier assignments (P0/P1/P2) documented and justified by traffic share, committed to a file carrying a dated `next-review` field. - Playwright project config (or BrowserStack/Sauce Labs config) reflects the matrix and runs P0 browsers on every PR; cloud configs pin `playwrightVersion` to match `package.json`. - `docs/browser-issues.md` exists with one row per known divergence: affected browser, repro, workaround/fallback or linked ticket, and the test that asserts the user outcome (not the CSS). - Common-divergence checklist (partitioned cookies, date inputs, clipboard, scroll behavior, backdrop-filter, ``, View Transitions) has a test or a known-issues row for each item relevant to P0/P1 browsers. - A tracked issue exists for the next quarterly matrix review (or the matrix file's `next-review` date is in the future), so the refresh is not lost. ## Related Skills - **visual-testing** — Owns pixel-level baseline strategy, threshold tables, and `toHaveScreenshot` config. Go there for *how tolerant* a screenshot diff should be; this skill only decides *which browsers* get a baseline. - **playwright-automation** — Core Playwright patterns, fixtures, and CI configuration that cross-browser testing builds on. - **ci-cd-integration** — Pipeline configuration for parallel browser-matrix execution and artifact collection. - **mobile-testing** — Device-farm and native/hybrid app testing (Appium/Detox); go there when the target is an app, not a browser viewport. - **accessibility-testing** — Cross-browser accessibility differences (screen-reader behavior, ARIA support) that overlap with this matrix. ## Reference Files (in `references/`) - **playwright-and-cloud-config.md** — `playwright.config.ts` project list, branded channels, `--project` run commands, and BrowserStack (SDK + `client.playwrightVersion`)/Sauce Labs/CI matrix configs. - **common-browser-issues.md** — Per-engine partitioned-cookie iframe test, date-input WebKit fallback, clipboard, scroll behavior, backdrop-filter, ``, View Transitions, and Web Animations. - **testing-patterns.md** — Same-test-multiple-browsers, `browserName` branching, visual comparison, and progressive-enhancement code.