# pagedigest Specification **Version:** 1 **Status:** Final (v1.0) ## Abstract `pagedigest` is a file format for publishing per-URL change signals on a website. Publishers generate a single JSON manifest listing each URL with a monotonic integer version and an optional cryptographic digest. Consumers fetch the manifest, compare integers against a local cache, and fetch individual URLs only when their version has incremented. The optional digest enables consumers to audit publisher claims. The primary use case is reducing redundant fetches by automated clients — AI crawlers, search indexers, archival systems, and similar — against sites whose content changes infrequently relative to the rate at which it is fetched. This document specifies the file format and expected consumer behavior. It does not specify publisher enforcement policy, which is described in a companion document. ## 1. Scope This specification is intended for publishers of **public, primarily static HTML and Markdown content** — blogs, documentation sites, news archives, marketing pages, product catalogs, and similar. It is explicitly not intended for: - Pages requiring authentication - Pages that vary significantly by user (personalized content, A/B tests, geo-targeting) - Pages whose content is primarily generated by client-side JavaScript - Real-time data feeds - Content behind paywalls A publisher whose site falls outside this scope may still publish a partial manifest covering only the URLs that are in scope. URLs not listed in the manifest receive no protocol-defined treatment. `pagedigest` is primarily designed for stateful, periodic consumers that retain crawl state across runs (for example: search indexers, archival systems, mirrors, enterprise sync engines, and agent caches). Stateless one-shot fetchers benefit only if they persist per-site or per-URL state across sessions. ## 2. File location The manifest MUST be served at `/.well-known/pagedigest.json` on any domain that publishes a manifest. The manifest SHOULD be served with `Content-Type: application/json` and SHOULD be cacheable by HTTP intermediaries (`Cache-Control: public, max-age=N` is recommended, with a value chosen by the publisher based on update frequency). Publishers should choose `max-age` conservatively enough that manifest caching does not defeat the publisher's desired change-detection latency. Publishers SHOULD support conditional requests (`ETag` and/or `Last-Modified`) on the manifest resource itself. Consumers polling the manifest SHOULD use `If-None-Match` and/or `If-Modified-Since` when available. ### 2.1 Discovery identifiers Version 1 uses the manifest location `/.well-known/pagedigest.json` as the normative discovery point. This project intends to register the `pagedigest.json` well-known URI suffix. Until a short-form relation type is registered, publishers that advertise manifest discovery via HTTP `Link` headers SHOULD use an absolute-URI extension relation type, for example: `Link: ; rel="https://pagedigest.org/rel"` Publishers SHOULD advertise this discovery link on ordinary successful responses (for example, HTTP 200 responses for HTML pages), not only on rate-limit responses. Some managed static hosting environments may not allow fully customized 429 responses. In those environments, publishers SHOULD prioritize discovery signaling on ordinary successful responses and keep manifest discovery stable and well-documented. ## 3. File format The manifest is a JSON document with the following top-level structure: ```json { "version": 1, "generated": "2025-10-16T10:00:00Z", "site_rev": 18293, "coverage": { "mode": "prefixes", "prefixes": ["/blog/", "/docs/"] }, "entries": { "/": { "rev": 47 }, "/about": { "rev": 12, "modified": "2025-10-16T09:11:00Z" }, "/blog/hello-world": { "rev": 3, "digest": "sha256:9f6c2d788e6f8f3b55f1f42b42d4c24f273e6f24c5ba7d53d2f1ef3f8247648b", "modified": "2025-10-15T18:22:00Z" } } } ``` ### 3.1 Top-level fields **`version`** (required, integer) The version of the `pagedigest` specification the manifest conforms to. For manifests conforming to this document, the value is `1`. Consumers encountering an unrecognized version SHOULD treat the manifest as unreadable and fall back to default behavior. **`generated`** (required, string) An ISO 8601 timestamp indicating when this manifest was generated. All timestamps MUST be in UTC. Used for debugging and for consumers to detect stale manifests. Not used for protocol logic. **`site_rev`** (required, integer) A monotonically increasing integer representing the overall state of the site. This value MUST increment whenever any URL covered by the manifest has its content changed, or when a URL is added to or removed from the `entries` map. It SHOULD NOT increment for unrelated operational events (cache flushes, deploys without content change, configuration updates). If `coverage` is present, changes to `coverage.mode` or to the `coverage.prefixes` list alter omission semantics and therefore MUST increment `site_rev`. Consumers use `site_rev` as a fast path: a consumer whose cached `site_rev` matches the manifest's `site_rev` knows that no URLs on the site have changed since its last visit and MAY skip processing the `entries` map entirely. **`entries`** (required, object) A map from URL path (relative to the site root) to an entry object. Keys in `entries` are origin-relative request targets. Each key MUST: - begin with `/` - consist of a path component, optionally followed by a query string beginning with `?` - be percent-encoded per RFC 3986 for any characters outside the unreserved set - NOT contain a fragment identifier (`#...`) The JSON schema's key pattern is intended only as a minimal structural check. The normative requirements for request-target syntax are defined by this specification text. Publishers SHOULD NOT emit separate entries for tracking, analytics, session, or other query variants that do not materially alter the served content. The protocol does not imply any normalization of URL keys. `/about` and `/about/` are distinct keys and MUST be treated as distinct URLs if the publisher lists them separately. Publishers are responsible for ensuring that the keys they emit match the URLs consumers would actually request. The special path `/` refers to the site root. **`coverage`** (optional, object) A machine-readable description of manifest coverage. Consumers MAY use this metadata to distinguish intentionally partial manifests from complete manifests. If present, `coverage.mode` MUST be one of: - `complete`: the publisher intends this manifest to describe the full covered URL set for this origin. - `prefixes`: the manifest intentionally describes only URLs under the listed origin-relative prefixes. If `coverage.mode` is `prefixes`, the `coverage.prefixes` array MUST be present and contain one or more strings that each begin with `/`. Publishers SHOULD list prefixes that end with `/` (for example `/blog/` rather than `/blog`) so that ordinary string-prefix matching does not accidentally cover sibling paths such as `/blogextra`. Consumers that implement prefix coverage SHOULD document whether they use raw `startswith` matching; the recommended convention is that a prefix applies to paths that begin with the prefix string as written. ### 3.2 Entry fields Each value in the `entries` map is an object with the following fields: **`rev`** (required, integer) A monotonically increasing integer representing the version of this specific URL's content. This value MUST increment whenever the content of the URL changes. It SHOULD NOT increment for unrelated reasons. Consumers compare `rev` against their cached value for the same URL. If the manifest's `rev` is greater than the cached value, the content has changed and SHOULD be re-fetched. If the `rev` matches the cached value, the content has not changed and the fetch MAY be skipped. #### 3.2.1 Semantics of `rev` For version 1, `rev` tracks material user-visible content changes for the URL's canonical representation. For version 1, this content scope includes important HTTP-level document metadata that materially affects interpretation or discovery (for example: ``, meta description, canonical URL declarations, Open Graph / Twitter metadata, and structured data markup). `rev` SHOULD increment for: - changes to primary text or meaningful media references - navigation changes that substantially affect interpretation or discoverability - template changes that materially alter the rendered document `rev` SHOULD NOT increment for: - analytics-only changes - per-request tokens and other request-unique values - deploy metadata - stylistic or template churn that does not materially alter informational content A global template change that materially changes many pages may legitimately increment many pages' `rev` values at once. **`digest`** (optional, string) A cryptographic hash of the URL's content. The format is `<algorithm>:<hex-encoded-hash>`, where the hash is lowercase hexadecimal with no separators. For version 1 of this specification, the algorithm MUST be `sha256` if present. Example: `"sha256:9f6c2d788e6f8f3b55f1f42b42d4c24f273e6f24c5ba7d53d2f1ef3f8247648b"` The hash is computed over the response body that would be served for an HTTP `GET` request to the URL with `Accept-Encoding: identity`. That is, the hash is of the uncompressed bytes, avoiding ambiguity across different content encodings. If a URL serves multiple representations via `Vary`, publishers SHOULD either hash a documented default representation or omit `digest` for that URL. Publishers SHOULD ensure that the content being hashed is deterministic with respect to the page's actual content. In particular, publishers SHOULD avoid including non-content variation in the hashed bytes — build timestamps, analytics session identifiers, CSRF tokens, randomized asset cache-busting strings, or other values that change on every build or request without reflecting content change. Such variation, if included in the hashed bytes, will cause `digest` values and `rev` integers to churn in ways that defeat the protocol. See section 4 for publisher content-hygiene guidance. Publishers whose rendered bytes contain unavoidable non-content churn SHOULD omit `digest` for those URLs unless they can make the full identity-encoded bytes stable enough for trustworthy audits. Consumers MAY use `digest` to audit publisher claims: fetch the URL with `Accept-Encoding: identity`, compute the hash of the response, and compare. A mismatch indicates the publisher's manifest is inaccurate. Publishers are not required to include `digest`. Manifests without `digest` remain functionally useful — consumers can still use `rev` for change detection — but lose the ability to be audited. **`modified`** (optional, string) An ISO 8601 timestamp indicating when the URL's content was last modified, as recorded by the publisher. All timestamps MUST be in UTC. Provided for debugging, human inspection, and CMS integration. Not used for protocol logic. Consumers SHOULD NOT make decisions based on `modified`. The `rev` integer is the authoritative change signal. ### 3.3 Unknown fields Consumers MUST ignore fields they do not recognize at either the top level or within entry objects. This allows for forward-compatible extensions. The JSON schema published with this specification is a validation aid, not the compatibility boundary of the protocol. Where future optional fields are added, conforming version 1 consumers are still expected to ignore fields they do not recognize. ## 4. Publisher responsibilities A publisher that generates a `pagedigest` manifest is responsible for: 1. Ensuring `site_rev` increments when any URL covered by the manifest has content changes, and when URLs are added to or removed from `entries`. 2. Ensuring per-URL `rev` values increment when the corresponding URL's content changes. 3. Ensuring that integer increments correspond to actual content changes, not to operational events unrelated to content. 4. Ensuring that `digest` values, when present, accurately reflect the hash of the identity-encoded response body. 5. Keeping the manifest reachable at `/.well-known/pagedigest.json` and responsive to normal HTTP requests. 6. Regenerating the manifest when content changes, at a frequency consistent with the publisher's publishing cadence. The recommended implementation is to generate the manifest at build time for static sites, or to generate it in response to content-change events for CMS-backed sites. Runtime generation on every request is possible but unnecessary and adds serving cost. ### 4.1 Monotonicity Publishers SHOULD NOT decrease `site_rev` or any per-URL `rev`. Values are expected to increase monotonically over the lifetime of a site. A decrease typically indicates operational error: restoration from an older backup, migration between systems, accidental reseeding, or a generator bug. Consumers encountering a decrease SHOULD treat the manifest as anomalous and fall back to default crawling behavior for the affected URLs or site. If a publisher must reset their revision state — for example, after an incident that corrupted the prior state — the publisher SHOULD advance to a value strictly greater than the highest value they previously published, rather than resetting to zero. This preserves the monotonic invariant consumers rely on. A rollback to previously published content is still a content change event and therefore MUST increment `rev` (and `site_rev` as appropriate). Legitimate rollback does not permit decreasing revision integers. ### 4.1.1 Publisher implementation note Revision state is durable protocol state. Publishers should persist a high-water mark for `site_rev` and per-URL `rev` outside fragile CI/build numbering. When a URL leaves the covered set (unpublished page, coverage filter change), the publisher SHOULD retain that URL's last `rev` in durable state even though it is omitted from the next manifest. Re-adding the same key later MUST NOT emit a lower `rev` than previously published for that key. Restores from backup, rebases, environment reseeds, or build-system counter resets can silently violate monotonicity if not handled deliberately. ### 4.2 Content hygiene The protocol's usefulness depends on `rev` integers and `digest` values changing if and only if the underlying content has changed. Publishers whose build or serving pipelines introduce non-content variation into page bytes will produce manifests that churn without content change, defeating the protocol. Common sources of non-content variation that publishers SHOULD eliminate from hashed bytes before manifest generation: - Build timestamps embedded in every page (e.g., "Generated on 2025-10-16 at 10:00:00") - Globally mutating widgets (e.g., "Recent posts" sidebars that update the footer of every page when one page is added) - Randomized asset cache-busting strings (e.g., `<link href="/style.css?v=abc123">` where the random value changes on every build) - Session identifiers or CSRF tokens rendered into static pages - Analytics or tracking script variations that differ on every build - Comment counts, view counts, or similar dynamic aggregates rendered into otherwise-static pages Publishers whose static site generators inject such variation by default SHOULD either configure the generator to omit it, derive `rev` from a stable content region rather than from the final rendered bytes, or generate the manifest from a pre-rendering stage that does not include the variation. If a publisher emits `digest`, however, the `digest` value MUST still be computed over the full identity-encoded response body as defined in section 3.2. A publisher whose manifest churns globally on every build — every page's `rev` incrementing at once, with no actual content change on most of them — is producing a manifest that has negative value for consumers, because they will fetch more than they would have without the manifest. Such a manifest is worse than no manifest at all. ### 4.3 Deployment consistency Publishers SHOULD make updated page bytes available before, or atomically with, publishing a manifest that references those updates. Publishers using CDNs, staged rollouts, or multi-region deployment paths SHOULD ensure the manifest does not become visible materially earlier than the corresponding page representations. The `generated` field is informational and MUST NOT be interpreted as a cross-edge consistency guarantee. ## 5. Consumer behavior ### 5.1 Basic algorithm A consumer that respects `pagedigest` manifests follows this algorithm when crawling a site: 1. Fetch `/.well-known/pagedigest.json`. If the fetch fails (404, timeout, invalid JSON), fall back to the consumer's default crawling behavior for this site. 2. Compare the manifest's `site_rev` against the consumer's cached value for this site. If equal, no URLs have changed; update the consumer's last-checked record and terminate this crawl cycle without fetching individual URLs. If greater, proceed to step 3 below. If less than cached, treat the manifest as anomalous and fall back to default behavior. 3. For each entry in the manifest's `entries` map, entry lookup and cache comparison are keyed by the pre-redirect request URL, byte-exact against manifest keys. Consumers MUST NOT rewrite keys based on redirect targets. If the consumer has no cached `rev` for this URL, the URL is new: fetch it and record the manifest's `rev`. If the manifest's `rev` is greater than cached `rev`, the URL has changed: fetch it and update cached `rev`. If equal, do not fetch. If less than cached, treat as anomalous and fall back to default behavior for this URL. 4. For URLs previously seen by the consumer that are no longer listed in the manifest: if `coverage.mode` is `complete`, omission is a positive signal that the URL is no longer part of the publisher's covered set and SHOULD be treated as removed from coverage. If `coverage.mode` is `prefixes` or `coverage` is absent, omission remains ambiguous and SHOULD be treated as "not described here" rather than implicitly unchanged. ### 5.2 Auditing A consumer MAY periodically audit `digest` values to verify publisher honesty. The recommended audit procedure is: 1. Select a URL listed in the manifest with a `digest` field. Random selection across the manifest is recommended. 2. Fetch the URL with the request header `Accept-Encoding: identity`. 3. Compute the SHA-256 hash of the response body. 4. Compare the computed hash against the `digest` field in the manifest. The `digest` corresponds to the listed URL's successful identity-encoded representation. Redirect responses and non-success responses encountered during audit are inconclusive outcomes, not hash targets. Consumers MAY retry or temporarily reduce trust, but SHOULD NOT hash redirect bodies as substitutes for the listed URL. A mismatch indicates the publisher's manifest is inaccurate. The consumer SHOULD respond by reducing trust in the publisher's manifest, increasing the audit rate, or falling back to unconditional fetching for the publisher's URLs. If a mismatch is observed immediately after a newly observed `generated` timestamp or `site_rev` increment, consumers SHOULD treat it as potentially transient and MAY retry before downgrading trust. Sustained mismatches or monotonicity violations are stronger evidence of unreliability than an isolated mismatch near a fresh publish event. A reasonable default audit rate is approximately 1% of manifest-derived skips, distributed randomly. Consumers are free to choose different rates based on their trust model and resource budget. When `digest` is omitted, consumers have no direct cryptographic audit path for rev inflation. If a consumer re-fetches after a `rev` bump and repeatedly observes content identical to its cached representation, the consumer MAY record this as soft evidence of unreliability and adjust trust or audit policy accordingly. ### 5.2.1 Manifest staleness The `generated` timestamp is informational, not authoritative protocol state. Consumers MAY treat implausibly old manifests as suspicious based on their own freshness requirements. A stale manifest MAY justify increased digest-audit rate or temporary fallback behavior until fresher manifest updates are observed. ### 5.2.2 Audit economics and failure scope (non-normative) Audits deliberately spend a fraction of the requests saved by manifest-derived skips. A practical baseline is: - begin with a neutral trust state - audit 1%–5% of eligible skips while establishing trust - audit 0.1%–1% for an established publisher - retry one mismatch observed near a fresh publish before penalizing the publisher, because edge races are often innocent - recover trust after a consumer-defined number of consecutive clean audit windows Failures should be contained to the smallest scope supported by the evidence: | Observation | Suggested scope and response | |---|---| | One mismatch on one URL near a deploy | URL-level suspicion; retry later | | Repeated mismatch on the same URL | Ignore that URL's digest/revision and fetch it conventionally | | Mismatches across many URLs | Site-level trust downgrade | | Per-URL `rev` decrease | URL-level anomaly; site-level if widespread | | `site_rev` decrease | Site-level anomaly; fall back to conventional crawling | | Global revision churn without content change | Low utility rather than dishonesty; reduce reliance | Trust state should be recoverable. A transient deployment race should not become a permanent sentence, while repeated failures should not be collapsed into one innocent event. ### 5.3 Error handling Consumers SHOULD handle the following error conditions gracefully, falling back to default crawling behavior rather than erroring out: - Manifest not present (404) - Manifest present but invalid JSON - Manifest present but missing required fields - Manifest `version` unrecognized - `site_rev` or any `rev` not an integer - Audit hash mismatch (per section 5.2) The intent is that a malformed manifest does not break crawling — it just reverts the consumer to the behavior they would have had without the manifest. ### 5.4 Optional cooperation request header Version 1 reserves `PageDigest-State` as optional client behavior. A consumer MAY send it on a page request after checking that origin's manifest: ```http PageDigest-State: site_rev=18294 ``` The optional extended form identifies the origin-relative manifest path: ```http PageDigest-State: site_rev=18294; manifest="/.well-known/pagedigest.json" ``` The grammar for the reserved version 1 form is: ```abnf non-zero-digit = %x31-39 site-rev-value = "0" / ( non-zero-digit *DIGIT ) PageDigest-State = "site_rev=" site-rev-value [ "; manifest=" DQUOTE absolute-path DQUOTE ] ``` Leading zeroes are not allowed except for the value `0`. The `manifest` value MUST begin with `/`, MUST NOT contain a fragment, quote, backslash, carriage return, or line feed, and is not an absolute URL. Consumers MUST NOT send a revision they did not observe from the named origin. Publishers MUST treat an absent or malformed header as no cooperation signal; it does not make the page request invalid. The header means: "Before this request, I checked this origin's manifest and observed this `site_rev`." It is an observation claim, not authentication, authorization, proof of identity, or proof that every subsequent fetch is necessary. Version 1 publishers MAY log it and MAY use it as one input to rate-limit classification. Full intermediary policy is deferred to the version 1.1 extension. #### 5.4.1 Why the initial signal does not require cryptography To send a plausible current revision, a client ordinarily has to fetch the manifest, which is already half of the cooperative behavior. The remaining question—whether it fetches covered URLs whose revisions did not change—is visible in the publisher's own request logs. A matching state value combined with low unchanged-page overfetch is therefore self-corroborating behavior, regardless of the client's stated intent. The header alone remains weak evidence. Publishers should ignore it unless corroborated by manifest access and fetch behavior. Impossible future revisions, persistently stale revisions, or a matching revision paired with unchanged-page overfetch are inexpensive anomaly signals; they are not cryptographic findings. ## 6. Reserved extensions The following capabilities are explicitly reserved for future versions of this specification and are not part of version 1: - **Manifest sharding for large sites.** Version 1 specifies a single manifest file. A future version will define a sharding mechanism (for example, by URL path prefix) for sites whose full manifest would exceed practical size limits. Until then, very large sites should either shard manifest coverage using partial manifests or wait for the sharding extension. - **Clean content extracts.** A separate specification may define a format for publishing deterministic machine-readable content extracts alongside the manifest. Version 1 does not define an `extract` field; publishers should not emit one. - **DNS-based discovery.** A mechanism for publishing `site_rev` via DNS (e.g., TXT record) for ultra-fast site-level change detection may be defined in a future version. Version 1 uses only HTTP. - **Alternative hash algorithms.** Version 1 uses SHA-256 exclusively. Future versions may define additional algorithms; the `digest` field's `<algorithm>:<hex>` format is structured to accommodate this. Publishers and consumers implementing version 1 should neither emit nor depend on any of the above. ## 7. Security considerations ### 7.1 Manifest tampering The manifest is served over HTTP(S) and subject to the same tampering risks as any other resource on the site. Publishers serving sensitive content should serve the manifest over HTTPS and ensure it cannot be modified by unauthorized parties. Consumers should fetch manifests over HTTPS where possible. ### 7.2 Dishonest publishers A publisher may publish a manifest that does not accurately reflect the state of the site, either through error or intent. The `digest` field and audit procedure (section 5.2) provide a mechanism for consumers to detect such inaccuracies. Consumers who do not audit have no protection against dishonest manifests beyond eventual discovery through normal cache invalidation. ### 7.3 Manifest denial of service A very large manifest could be used to consume consumer resources. Consumers SHOULD impose reasonable size limits on manifests they will process and SHOULD abort processing of manifests exceeding those limits. As a baseline expectation: consumers MAY decline to process manifests exceeding 10 megabytes. Publishers whose full manifest would approach or exceed this threshold SHOULD publish a partial manifest covering their highest-value URLs rather than a complete manifest that consumers will refuse to read. When the sharding extension (section 6) is defined, such publishers will have a formal mechanism for full coverage. ### 7.4 Privacy A manifest exposes the complete list of URLs covered on a site, including URLs that may not be linked from the site's navigation. Publishers should not list URLs in the manifest that they do not wish to be publicly enumerable. This is analogous to the situation with `sitemap.xml`. ## 8. Relationship to existing mechanisms (non-normative) `pagedigest` is pre-fetch revalidation, not a replacement for discovery, caching, push delivery, or access policy. Sitemaps tell a consumer what exists. ETags tell it, one request at a time, whether a representation changed. `pagedigest` tells it, in one request, what did not change across the covered set. | Mechanism | What it does | Why `pagedigest` differs | |---|---|---| | `sitemap.xml` + `lastmod` | URL discovery plus advisory timestamps | `lastmod` is timestamp-based and has no site-wide monotonic fast path or audit model. Google documents that it uses `lastmod` only when the value is consistently and verifiably accurate. | | `ETag` / `If-None-Match` / 304 | Per-representation conditional validation | It still requires one request per URL. `pagedigest` can skip the entire covered set when `site_rev` is unchanged. A changed URL may still be fetched conditionally with an ETag. | | `Last-Modified` / `If-Modified-Since` | Timestamp-based conditional validation | It is also per-resource and subject to timestamp granularity. `pagedigest` uses explicit monotonic state across the covered set. | | RSS / Atom | A feed of recent entries | A feed does not establish that every omitted or older covered page stayed unchanged. | | IndexNow | Publisher submission of changed URLs to participating search engines | It is a publisher-to-engine push channel with ownership-key verification, not a stateless manifest arbitrary consumers can pull. | | WebSub | Hub-mediated push subscriptions | It requires hub and subscription/callback state. `pagedigest` is stateless pull. | | Website monitors | Private change alerts for one watcher | `pagedigest` is a public change surface reusable by many independent consumers. | | CDN / HTTP cache | Stores responses so repeated serving is cheaper | A cache still serves or validates the read. `pagedigest` lets a stateful consumer avoid many reads. | `robots.txt` remains access policy; `pagedigest` is change detection. Consumers should apply both. ## 9. References - RFC 3986: Uniform Resource Identifier (URI): Generic Syntax - RFC 8615: Well-Known Uniform Resource Identifiers (URIs) - RFC 8288: Web Linking - RFC 9110: HTTP Semantics, including validators and conditional requests: <https://www.rfc-editor.org/rfc/rfc9110> - RFC 9111: HTTP Caching: <https://www.rfc-editor.org/rfc/rfc9111> - Google Search Central, sitemap construction and `lastmod` accuracy: <https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap> - IndexNow protocol documentation: <https://www.indexnow.org/documentation> - W3C WebSub Recommendation: <https://www.w3.org/TR/websub/> - RFC 4287: Atom Syndication Format: <https://www.rfc-editor.org/rfc/rfc4287> - IANA Well-Known URIs registry: <https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml> - IANA Link Relation Types registry: <https://www.iana.org/assignments/link-relations/link-relations.xhtml> ## Appendix A: URL-key conformance examples The schema key pattern is intentionally minimal. The following examples illustrate this specification's normative key rules. Valid examples: - `/` - `/about` - `/about/` - `/docs/api?lang=en` - `/pricing?region=us` - `/posts/hello%20world` Invalid examples: - `about` (missing leading `/`) - `/foo#bar` (contains a fragment) - `https://example.com/foo` (absolute URL; keys are origin-relative) - `//evil.example/foo` (scheme-relative / network-path reference; keys must not carry an authority) - `/foo?utm_source=x` when the query is a non-material tracking variant that the publisher should omit - `/posts/hello world` (raw space must be percent-encoded) Comparison examples: - `/about` and `/about/` are distinct keys. If both representations exist, consumers compare each key byte-exact as listed. - `/pricing?region=us` may be a material variant and can be listed as its own key when content differs materially. - `/pricing?utm_source=newsletter` is typically a tracking variant and should usually be omitted. ## Appendix B: Example manifests ### B.1 Minimal manifest A valid minimal manifest for a small site: ```json { "version": 1, "generated": "2025-10-16T10:00:00Z", "site_rev": 5, "entries": { "/": { "rev": 2 }, "/about": { "rev": 1 }, "/contact": { "rev": 1 } } } ``` ### B.2 Manifest with digests A manifest for a site that includes cryptographic digests for audit: ```json { "version": 1, "generated": "2025-10-16T10:00:00Z", "site_rev": 142, "entries": { "/": { "rev": 14, "digest": "sha256:12c4be9a5f4a4a4376a6f4f7c9e1619f0fcb9c6dbb334d99d76d28b9f8578c43" }, "/posts/hello-world": { "rev": 3, "digest": "sha256:4d4f877f4f36ac0a31ba82f269d35f7a78fc6bd6f183f064f2f6b8ff23e7b57e", "modified": "2025-10-10T09:15:00Z" } } } ``` ### B.3 Partial manifest A manifest covering only a subset of a site — the publisher has opted to describe only the blog, not the full site: ```json { "version": 1, "generated": "2025-10-16T10:00:00Z", "site_rev": 89, "coverage": { "mode": "prefixes", "prefixes": ["/blog/"] }, "entries": { "/blog/": { "rev": 12 }, "/blog/post-1": { "rev": 4, "digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" }, "/blog/post-2": { "rev": 1, "digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" } } } ``` URLs outside `/blog/` on this site are not covered by the manifest and receive default crawler behavior.