# SiteOne Crawler: JSON Output Documentation

## Table of Contents

*   [1. Introduction](#1-introduction)
*   [2. Potential Use Cases](#2-potential-use-cases)
*   [3. Detailed JSON Structure](#3-detailed-json-structure)
    *   [3.1. `crawler` (Object)](#31-crawler-object)
    *   [3.2. `extraColumnsFromAnalysis` (Array)](#32-extracolumnsfromanalysis-array)
    *   [3.3. `options` (Object)](#33-options-object)
    *   [3.4. `qualityScores` (Object)](#34-qualityscores-object)
    *   [3.5. `results` (Array)](#35-results-array)
    *   [3.6. `stats` (Object)](#36-stats-object)
    *   [3.7. `summary` (Object)](#37-summary-object)
    *   [3.8. `tables` (Object)](#38-tables-object)
*   [4. JSON Schema (Draft)](#4-json-schema-draft)
*   [5. Analysis Tables Description (`tables` key)](#5-analysis-tables-description-tables-key)
    *   [5.1. `skipped-summary` (Skipped URLs Summary)](#51-skipped-summary-skipped-urls-summary)
    *   [5.2. `skipped` (Skipped URLs)](#52-skipped-skipped-urls)
    *   [5.3. `redirects` (Redirected URLs)](#53-redirects-redirected-urls)
    *   [5.4. `404` (404 URLs)](#54-404-404-urls)
    *   [5.5. `certificate-info` (SSL/TLS info)](#55-certificate-info-ssltls-info)
    *   [5.6. `fastest-urls` (TOP fastest URLs)](#56-fastest-urls-top-fastest-urls)
    *   [5.7. `slowest-urls` (TOP slowest URLs)](#57-slowest-urls-top-slowest-urls)
    *   [5.8. `seo` (SEO metadata)](#58-seo-seo-metadata)
    *   [5.9. `open-graph` (OpenGraph metadata)](#59-open-graph-opengraph-metadata)
    *   [5.10. `seo-headings` (Heading structure)](#510-seo-headings-heading-structure)
    *   [5.11. `headers` (HTTP headers)](#511-headers-http-headers)
    *   [5.12. `headers-values` (HTTP header values)](#512-headers-values-http-header-values)
    *   [5.13. `caching-per-content-type` (HTTP Caching by content type)](#513-caching-per-content-type-http-caching-by-content-type)
    *   [5.14. `caching-per-domain` (HTTP Caching by domain)](#514-caching-per-domain-http-caching-by-domain)
    *   [5.15. `caching-per-domain-and-content-type` (HTTP Caching by domain and content type)](#515-caching-per-domain-and-content-type-http-caching-by-domain-and-content-type)
    *   [5.16. `non-unique-titles` (TOP non-unique titles)](#516-non-unique-titles-top-non-unique-titles)
    *   [5.17. `non-unique-descriptions` (TOP non-unique descriptions)](#517-non-unique-descriptions-top-non-unique-descriptions)
    *   [5.18. `best-practices` (Best practices)](#518-best-practices-best-practices)
    *   [5.19. `accessibility` (Accessibility)](#519-accessibility-accessibility)
    *   [5.20. `source-domains` (Source domains)](#520-source-domains-source-domains)
    *   [5.21. `content-types` (Content types)](#521-content-types-content-types)
    *   [5.22. `content-types-raw` (Content types (MIME types))](#522-content-types-raw-content-types-mime-types)
    *   [5.23. `dns` (DNS info)](#523-dns-dns-info)
    *   [5.24. `security` (Security)](#524-security-security)
    *   [5.25. `analysis-stats` (Analysis stats)](#525-analysis-stats-analysis-stats)
    *   [5.26. `content-processors-stats` (Content processor stats)](#526-content-processors-stats-content-processor-stats)
    *   [5.27. `external-urls` (External URLs)](#527-external-urls-external-urls)
*   [6. Note on Text Output](#6-note-on-text-output)


This document describes the structure and content of the JSON output file generated by the SiteOne Crawler. This JSON file contains detailed information about the crawled website, including metadata about the crawl process, results for each visited URL, quality scores, summary findings, and various analysis tables.

## 1. Introduction

The JSON output provides a comprehensive dataset about the crawled website. Key information includes:

*   **Crawl Metadata:** Details about the crawler execution, such as version, execution time, command used, hostname, and the final user agent.
*   **Options:** A complete record of all CLI configuration values used for the crawl.
*   **Quality Scores:** Overall and per-category quality scores (0-10) with deduction details.
*   **Visited URL Results:** For each URL visited during the crawl:
    *   URL address
    *   HTTP status code
    *   Elapsed time for the request (performance)
    *   Size of the response body
    *   Content type (HTML, CSS, JS, Image, etc.)
    *   Caching information (cache flags, lifetime)
    *   Additional analysis results stored in the `extras` field.
*   **Stats:** Aggregate statistics about the crawl (total URLs, sizes, timings, status code counts).
*   **Summary:** A list of findings (OK, Warning, Critical, Info) that feed into quality scoring.
*   **Analysis Tables:** Aggregated data and specific findings presented in structured tables:
    *   **Skipped URLs:** Reasons why certain URLs were not crawled (e.g., external domain, disallowed by robots.txt, specific rules).
    *   **Redirects:** List of URLs that resulted in redirects (3xx status codes).
    *   **404 Errors:** List of URLs that resulted in a 404 Not Found status.
    *   **SSL/TLS Info:** Details about the website's SSL certificate (issuer, subject, validity dates, supported protocols).
    *   **Performance:** Tables listing the fastest and slowest URLs encountered during the crawl.
    *   **SEO & Content:**
        *   SEO metadata (title, description, keywords, H1, indexing directives) for HTML pages.
        *   OpenGraph and Twitter Card metadata.
        *   Heading structure analysis (correctness of H1-H6 hierarchy).
        *   Analysis of non-unique titles and descriptions across pages.
    *   **Technical Details:**
        *   HTTP Headers: Summary of headers found, their occurrences, and unique values.
        *   Caching Analysis: Breakdown of caching strategies by content type and domain.
        *   DNS Information: DNS resolution details for the target domain.
        *   Security Analysis: Evaluation of security-related HTTP headers.
        *   External URLs: List of external URLs discovered during the crawl.
    *   **Crawler Statistics:** Performance metrics for the crawler itself, individual analyzers, and content processors.

## 2. Potential Use Cases

The detailed data within the JSON output enables a wide variety of use cases:

1.  **Comprehensive SEO Audits:** Analyze titles, descriptions, heading structures, indexing status, and OpenGraph tags across the entire site.
2.  **Performance Monitoring & Optimization:** Identify the slowest pages and resources, analyze load times, and check caching headers.
3.  **Broken Link Checking:** Easily extract lists of all 404 errors and the pages where they were found.
4.  **Redirect Chain Analysis:** Identify and analyze redirect chains.
5.  **Security Header Audits:** Verify the implementation of crucial security headers (CSP, HSTS, X-Frame-Options, etc.) across the site.
6.  **Content Inventory & Analysis:** Get a list of all crawled resources, their types, sizes, and status codes. Analyze content type distribution.
7.  **Website Archiving/Cloning:** While the crawler has a dedicated offline export, the JSON contains the list of all discovered resources, which could inform a custom archiving process.
8.  **Competitive Analysis:** Run the crawler on competitor sites (respecting their `robots.txt`) to gather insights into their structure, performance, and technology.
9.  **CI/CD Integration:** Integrate the crawler into deployment pipelines to automatically check for new errors (404s, performance regressions) after deployments. Use quality scores and thresholds for automated pass/fail decisions.
10. **Technical Debt Assessment:** Identify outdated practices, missing security headers, or performance issues that need addressing.

## 3. Detailed JSON Structure

The JSON output has 8 top-level keys:

### 3.1. `crawler` (Object)

Contains metadata about the crawler execution:
*   `name` (String): Name of the crawler software.
*   `version` (String): Version of the crawler.
*   `executedAt` (String): Timestamp when the crawl was executed, in the format `"YYYY-MM-DD HH:MM:SS"` (space separator, no timezone). Example: `"2026-03-16 14:55:13"`.
*   `command` (String): The command-line arguments used to run the crawl.
*   `hostname` (String): The hostname where the crawler was run.
*   `finalUserAgent` (String): The User-Agent string used for the HTTP requests.

### 3.2. `extraColumnsFromAnalysis` (Array)

An array of objects defining extra columns that might be added during specific analyses. These are primarily intended for augmenting report outputs. Each object contains:
*   `name` (String): The display name of the column.
*   `length` (Integer): Suggested display length/width.
*   `truncate` (Boolean): Whether the content should be truncated if it exceeds the length.
*   `customMethod`, `customPattern`, `customGroup`: Fields used for custom data extraction logic (null when not configured).

### 3.3. `options` (Object)

A flat object containing all 132 CLI configuration values used for the crawl. Every option from the command line (or its default value) is recorded here. Keys are the option names in camelCase (e.g., `url`, `workers`, `maxReqsPerSec`, `timeout`, `outputType`, `userAgent`, `acceptEncoding`, etc.). Values are strings, integers, booleans, or null, depending on the option type.

This is useful for reproducing a crawl or understanding the exact configuration that produced the results.

### 3.4. `qualityScores` (Object)

Contains overall and per-category quality scores computed after analysis.

*   `overall` (Object): The aggregate quality score.
    *   `score` (Float): Overall score from 0.0 to 10.0.
    *   `label` (String): Human-readable label (e.g., `"A+"`, `"A"`, `"B"`, `"C"`, `"D"`, `"F"`).
    *   `weight` (Float): Total weight (1.0 for overall).
    *   `deductions` (Array): Array of objects, each with:
        *   `points` (Float): Number of points deducted.
        *   `reason` (String): Explanation for the deduction.

*   `categories` (Array): Array of 5 category objects, each with:
    *   `code` (String): Category identifier. One of: `"performance"`, `"seo"`, `"security"`, `"accessibility"`, `"bestPractices"`.
    *   `name` (String): Human-readable category name.
    *   `score` (Float): Category score from 0.0 to 10.0.
    *   `label` (String): Human-readable label.
    *   `weight` (Float): Weight of this category in the overall score (e.g., 0.20 for SEO, 0.25 for Security).
    *   `deductions` (Array): Array of deduction objects (same structure as overall deductions).

### 3.5. `results` (Array)

An array of objects, where each object represents a single visited URL.
*   `url` (String): The absolute URL that was visited.
*   `status` (String): The HTTP status code returned (e.g., `"200"`, `"404"`).
*   `elapsedTime` (Float): Time taken to fetch the URL in seconds (e.g., `0.005`).
*   `size` (Integer): Size of the response body in bytes (e.g., `50961`).
*   `type` (Integer): An enum representing the detected content type:
    *   `1`: HTML
    *   `2`: JavaScript
    *   `3`: CSS
    *   `4`: Image
    *   `7`: Document (e.g., robots.txt)
    *   `8`: JSON
    *   Other types may exist (Audio, Font, Video, XML, Redirect, Other).
*   `cacheTypeFlags` (Integer): Bitmask representing detected caching mechanisms (e.g., Cache-Control, ETag, Last-Modified). For example, `31` typically means Cache-Control + ETag + Last-Modified are all present. `32768` might indicate no caching headers found.
*   `cacheLifetime` (Integer): Cache lifetime in seconds derived from `Cache-Control: max-age` or `Expires` header. `0` if no lifetime could be determined.
*   `extras` (Array): Contains additional data from specific analyzers run on this URL. Typically an empty array `[]`.

### 3.6. `stats` (Object)

Aggregate statistics about the entire crawl:
*   `totalUrls` (Integer): Total number of URLs visited.
*   `totalSize` (Integer): Total size of all responses in bytes.
*   `totalSizeFormatted` (String): Human-readable formatted total size (e.g., `"31.33 MB"`).
*   `totalExecutionTime` (Float): Total wall-clock execution time in seconds.
*   `totalRequestsTimes` (Float): Sum of all individual request times in seconds.
*   `totalRequestsTimesAvg` (Float): Average request time in seconds.
*   `totalRequestsTimesMin` (Float): Minimum request time in seconds.
*   `totalRequestsTimesMax` (Float): Maximum request time in seconds.
*   `totalRequestsTimesP90` (Float): 90th-percentile response time of HTML documents in seconds (a fairer "what users feel" proxy than the all-asset average; falls back to all URLs if no HTML pages).
*   `countByStatus` (Object): An object mapping HTTP status codes to counts. Keys are status code strings (e.g., `"200"`, `"404"`, `"429"`), values are integers. Only status codes that were actually encountered appear as keys.

### 3.7. `summary` (Object)

Contains a list of summary findings that feed into quality scoring.

*   `items` (Array): Array of finding objects, each with:
    *   `aplCode` (String): A unique code identifying the finding (e.g., `"s201"`, `"s404"`, `"s502"`).
    *   `status` (String): Severity level. One of: `"CRITICAL"`, `"WARNING"`, `"OK"`, `"INFO"`.
    *   `text` (String): Human-readable description of the finding (e.g., `"Brotli is supported for HTML"`, `"1 URL(s) returned a 404 status code"`).

### 3.8. `tables` (Object)

An object where each key is a table identifier (e.g., `skipped-summary`, `404`, `seo`) and the value is an object describing that table. Each table object contains:
*   `aplCode` (String): A unique code for the table.
*   `title` (String): A human-readable title for the table.
*   `columns` (Object): An object describing the columns of the table. Each key is a column identifier (e.g., `reason`, `url`, `statusCode`). The value is an object detailing the column:
    *   `aplCode` (String): Unique code for the column.
    *   `name` (String): Display name for the column header.
    *   `width` (Integer): Suggested display width (-1 might mean auto).
    *   `formatter` (Object | null): Defines how the data should be formatted (e.g., adding units like 'ms' or 'kB'). Empty object `{}` indicates default formatting.
    *   `renderer` (Object | null): Defines how the data should be rendered (e.g., adding color or links). Empty object `{}` indicates default rendering.
    *   `truncateIfLonger` (Boolean): Whether to truncate the value if it exceeds the width.
    *   Other fields like `formatterWillChangeValueLength`, `nonBreakingSpaces`, `escapeOutputHtml`, `getDataValueCallback`, `forcedDataType` provide more hints for rendering.
*   `rows` (Array): An array of objects, where each object represents a row in the table. The keys in each row object correspond to the column identifiers defined in `columns`. **Important: All values in all table rows are strings**, regardless of whether the data represents a number, count, or other type. For example, a count of `51` appears as `"51"`, a request time of `0.003` appears as `"0.003"`, and an empty value appears as `""`. Rows may also contain extra keys beyond the declared columns (see individual table descriptions for details).
*   `position` (String): A hint about where this table should typically be positioned in a report (e.g., `before-url-table`, `after-url-table`).

**Note:** The specific content and structure within `tables` depend on the analyzers enabled during the crawl. The set of tables may vary depending on what data was encountered (e.g., `certificate-info` only appears for HTTPS sites).

## 4. JSON Schema (Draft)

This is a draft JSON schema based on the actual output. It may need refinement for edge cases.

```json
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "SiteOne Crawler JSON Output",
  "description": "Schema for the JSON output file generated by SiteOne Crawler.",
  "type": "object",
  "properties": {
    "crawler": {
      "description": "Metadata about the crawler execution.",
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "version": { "type": "string" },
        "executedAt": { "type": "string", "description": "Format: YYYY-MM-DD HH:MM:SS" },
        "command": { "type": "string" },
        "hostname": { "type": "string" },
        "finalUserAgent": { "type": "string" }
      },
      "required": ["name", "version", "executedAt", "command", "hostname", "finalUserAgent"]
    },
    "extraColumnsFromAnalysis": {
      "description": "Definitions for extra columns used in analyses.",
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": { "type": "string" },
          "length": { "type": "integer" },
          "truncate": { "type": "boolean" },
          "customMethod": { "type": ["string", "null"] },
          "customPattern": { "type": ["string", "null"] },
          "customGroup": { "type": ["string", "null"] }
        },
        "required": ["name", "length", "truncate"]
      }
    },
    "options": {
      "description": "All CLI configuration values used for the crawl.",
      "type": "object",
      "additionalProperties": true
    },
    "qualityScores": {
      "description": "Overall and per-category quality scores.",
      "type": "object",
      "properties": {
        "overall": {
          "type": "object",
          "properties": {
            "score": { "type": "number" },
            "label": { "type": "string" },
            "weight": { "type": "number" },
            "deductions": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "points": { "type": "number" },
                  "reason": { "type": "string" }
                },
                "required": ["points", "reason"]
              }
            }
          },
          "required": ["score", "label", "weight", "deductions"]
        },
        "categories": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "code": { "type": "string", "enum": ["performance", "seo", "security", "accessibility", "bestPractices"] },
              "name": { "type": "string" },
              "score": { "type": "number" },
              "label": { "type": "string" },
              "weight": { "type": "number" },
              "deductions": {
                "type": "array",
                "items": {
                  "type": "object",
                  "properties": {
                    "points": { "type": "number" },
                    "reason": { "type": "string" }
                  },
                  "required": ["points", "reason"]
                }
              }
            },
            "required": ["code", "name", "score", "label", "weight", "deductions"]
          }
        }
      },
      "required": ["overall", "categories"]
    },
    "results": {
      "description": "Array of results for each visited URL.",
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "url": { "type": "string", "format": "uri" },
          "status": { "type": "string" },
          "elapsedTime": { "type": "number" },
          "size": { "type": "integer" },
          "type": { "type": "integer", "description": "Enum for content type (1:HTML, 2:JS, 3:CSS, 4:Image, 7:Document, 8:JSON, ...)" },
          "cacheTypeFlags": { "type": "integer", "description": "Bitmask for caching mechanisms" },
          "cacheLifetime": { "type": "integer", "description": "Cache lifetime in seconds, 0 if undetermined" },
          "extras": {
            "type": "array",
            "description": "Additional analysis data for this URL (typically empty)"
          }
        },
        "required": ["url", "status", "elapsedTime", "size", "type", "cacheTypeFlags", "cacheLifetime", "extras"]
      }
    },
    "stats": {
      "description": "Aggregate crawl statistics.",
      "type": "object",
      "properties": {
        "totalUrls": { "type": "integer" },
        "totalSize": { "type": "integer" },
        "totalSizeFormatted": { "type": "string" },
        "totalExecutionTime": { "type": "number" },
        "totalRequestsTimes": { "type": "number" },
        "totalRequestsTimesAvg": { "type": "number" },
        "totalRequestsTimesMin": { "type": "number" },
        "totalRequestsTimesMax": { "type": "number" },
        "totalRequestsTimesP90": { "type": "number" },
        "countByStatus": {
          "type": "object",
          "additionalProperties": { "type": "integer" }
        }
      },
      "required": ["totalUrls", "totalSize", "totalSizeFormatted", "totalExecutionTime", "totalRequestsTimes", "totalRequestsTimesAvg", "totalRequestsTimesMin", "totalRequestsTimesMax", "totalRequestsTimesP90", "countByStatus"]
    },
    "summary": {
      "description": "Summary findings that feed into quality scoring.",
      "type": "object",
      "properties": {
        "items": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "aplCode": { "type": "string" },
              "status": { "type": "string", "enum": ["CRITICAL", "WARNING", "OK", "INFO"] },
              "text": { "type": "string" }
            },
            "required": ["aplCode", "status", "text"]
          }
        }
      },
      "required": ["items"]
    },
    "tables": {
      "description": "Aggregated analysis results presented as tables.",
      "type": "object",
      "additionalProperties": {
        "type": "object",
        "properties": {
          "aplCode": { "type": "string" },
          "title": { "type": "string" },
          "columns": {
            "type": "object",
            "additionalProperties": {
              "type": "object",
              "properties": {
                "aplCode": { "type": "string" },
                "name": { "type": "string" },
                "width": { "type": "integer" },
                "formatter": { "type": ["object", "null"] },
                "renderer": { "type": ["object", "null"] },
                "truncateIfLonger": { "type": "boolean" }
              },
              "required": ["aplCode", "name", "width"]
            }
          },
          "rows": {
            "type": "array",
            "items": {
              "type": "object",
              "description": "All row values are strings. Rows may contain extra keys beyond the declared columns.",
              "additionalProperties": { "type": "string" }
            }
          },
          "position": { "type": "string", "enum": ["before-url-table", "after-url-table"] }
        },
        "required": ["aplCode", "title", "columns", "rows", "position"]
      }
    }
  },
  "required": ["crawler", "extraColumnsFromAnalysis", "options", "qualityScores", "results", "stats", "summary", "tables"]
}
```

## 5. Analysis Tables Description (`tables` key)

This section details the structure and columns of each table found under the `tables` key in the JSON output.

**Important note on data types:** All values in all table rows are **strings**. Numeric values such as counts, times, and sizes are serialized as strings (e.g., `"51"` not `51`, `"0.003"` not `0.003`). Empty values appear as `""`. This applies to every table described below. Where column descriptions say "count" or "time", the value is still a string representation of that number.

Some tables include **extra row keys** beyond the declared columns. These are noted in the individual table descriptions.

### 5.1. `skipped-summary` (Skipped URLs Summary)

Provides a summary of skipped URLs grouped by domain and reason.

| Column | Description |
|--------|-------------|
| `reason` | A human-readable string describing why URLs from this domain were skipped (e.g., `"Not allowed host"`, `"Blocked by robots.txt"`). |
| `domain` | The domain name whose URLs were skipped. |
| `count` | The number of unique URLs skipped for this domain and reason. |

### 5.2. `skipped` (Skipped URLs)

Lists individual URLs that were skipped during the crawl.

| Column | Description |
|--------|-------------|
| `reason` | A human-readable string describing why the URL was skipped (e.g., `"Not allowed host"`, `"Blocked by robots.txt"`, `"File extension is not allowed"`). |
| `url` | The URL that was skipped. |
| `sourceAttr` | A string describing the HTML attribute where the skipped URL was found (e.g., `"<a href>"`, `"<link href>"`, `"<script src>"`). |
| `sourceUqId` | The URL path of the page where the skipped URL was discovered (e.g., `"/"`, `"/docs/getting-started"`). This allows linking back to the source page. |

### 5.3. `redirects` (Redirected URLs)

Lists URLs that resulted in an HTTP redirect (3xx status code).

| Column | Description |
|--------|-------------|
| `statusCode` | The specific redirect status code (e.g., `"301"`, `"302"`). |
| `url` | The original URL that redirected. |
| `targetUrl` | The target URL to which the original URL redirected. |
| `sourceUqId` | URL path of the page where the redirected URL was found. |

### 5.4. `404` (404 URLs)

Lists URLs that resulted in a "404 Not Found" status code.

| Column | Description |
|--------|-------------|
| `statusCode` | The HTTP status code (typically `"404"`). |
| `url` | The URL that resulted in the 404 error. |
| `sourceUqId` | URL path of the page where the broken URL was found. |

### 5.5. `certificate-info` (SSL/TLS info)

Provides details about the SSL/TLS certificate of the crawled domain.

| Column | Description |
|--------|-------------|
| `info` | The name of the certificate attribute (e.g., `"Issuer"`, `"Subject"`, `"Valid from"`, `"Valid to"`, `"Supported protocols"`, `"RAW certificate output"`, `"RAW protocols output"`). |
| `value` | The value of the corresponding certificate attribute. Always a string. For multi-line values like raw certificate or protocol output, the entire content is a single string with embedded newlines. |

### 5.6. `fastest-urls` (TOP fastest URLs)

Lists the URLs with the lowest request times encountered during the crawl.

| Column | Description |
|--------|-------------|
| `requestTime` | The time taken to fetch the URL in seconds (e.g., `"0.003"`). |
| `statusCode` | The HTTP status code of the URL (e.g., `"200"`). |
| `url` | The URL itself. |

### 5.7. `slowest-urls` (TOP slowest URLs)

Lists the URLs with the highest request times encountered during the crawl.

| Column | Description |
|--------|-------------|
| `requestTime` | The time taken to fetch the URL in seconds (e.g., `"1.234"`). |
| `statusCode` | The HTTP status code of the URL (e.g., `"200"`). |
| `url` | The URL itself. |

### 5.8. `seo` (SEO metadata)

Provides SEO-related metadata extracted from HTML pages.

| Column | Description |
|--------|-------------|
| `urlPathAndQuery` | The path and query string of the URL. |
| `indexing` | A string describing the indexing status (e.g., `"index, follow"`, `"noindex, follow"`). |
| `title` | The content of the `<title>` tag, or empty string if not found. |
| `h1` | The content of the first `<h1>` tag found, or empty string. |
| `description` | The content of the `meta name="description"` tag, or empty string. |
| `keywords` | The content of the `meta name="keywords"` tag, or empty string. |

**Extra row keys** (present in each row object but not declared as columns):
*   `robotsIndex` (String): Whether the page allows indexing (e.g., `"1"` for index, `"0"` for noindex).
*   `deniedByRobotsTxt` (String): Whether the page is denied by robots.txt (e.g., `"0"` for allowed, `"1"` for denied).

### 5.9. `open-graph` (OpenGraph metadata)

Provides Open Graph and Twitter Card metadata extracted from HTML pages.

| Column | Description |
|--------|-------------|
| `urlPathAndQuery` | The path and query string of the URL. |
| `ogTitle` | Content of the `og:title` meta tag, or empty string. |
| `ogDescription` | Content of the `og:description` meta tag, or empty string. |
| `ogImage` | Content of the `og:image` meta tag, or empty string. |
| `twitterTitle` | Content of the `twitter:title` meta tag, or empty string. |
| `twitterDescription` | Content of the `twitter:description` meta tag, or empty string. |
| `twitterImage` | Content of the `twitter:image` meta tag, or empty string. |

### 5.10. `seo-headings` (Heading structure)

Provides analysis of the heading (H1-H6) structure for each HTML page.

| Column | Description |
|--------|-------------|
| `headings` | A formatted string representation of the heading structure showing hierarchy and potential errors (e.g., `"OK H1, H2, H2, H3"` or `"ERR H1, H3 (skipped H2)"`). |
| `headingsCount` | Total number of headings found on the page (e.g., `"5"`). |
| `headingsErrorsCount` | Number of structural errors found in the headings (e.g., `"0"`, `"2"`). |
| `urlPathAndQuery` | The path and query string of the URL. |

**Extra row key:**
*   `headingsHtml` (String): An HTML string containing the full heading tree with markup (e.g., `"<b>H1</b> Title<br><b>H2</b> Section..."`). Useful for rendering a visual heading tree in reports.

### 5.11. `headers` (HTTP headers)

Summarizes the HTTP response headers encountered across all crawled URLs.

| Column | Description |
|--------|-------------|
| `header` | The name of the HTTP header. |
| `occurrences` | The total number of times this header was found (e.g., `"73"`). |
| `uniqueValues` | The count of distinct values found for this header, as a string (e.g., `"3"`). |
| `valuesPreview` | A preview string showing some of the values encountered (truncated if many). |
| `minValue` | The minimum value found (relevant for numerical or date headers), or empty string. |
| `maxValue` | The maximum value found, or empty string. |

### 5.12. `headers-values` (HTTP header values)

Lists unique values for each HTTP header and their occurrence count.

| Column | Description |
|--------|-------------|
| `header` | The name of the HTTP header. |
| `occurrences` | The number of times this specific value occurred for this header (e.g., `"51"`). |
| `value` | The specific unique value of the HTTP header. |

### 5.13. `caching-per-content-type` (HTTP Caching by content type)

Analyzes caching effectiveness grouped by general content type (HTML, Image, JS, CSS, etc.).

| Column | Description |
|--------|-------------|
| `contentType` | The general content type category (e.g., `"HTML"`, `"Image"`, `"JS"`). |
| `cacheType` | Description of the caching mechanism detected (e.g., `"Cache-Control + ETag + Last-Modified"`, `"No cache headers"`). |
| `count` | Number of URLs matching this content type and cache type. |
| `avgLifetime` | Average cache lifetime in seconds for URLs in this group, or empty string if not determinable. |
| `minLifetime` | Minimum cache lifetime in seconds, or empty string. |
| `maxLifetime` | Maximum cache lifetime in seconds, or empty string. |

### 5.14. `caching-per-domain` (HTTP Caching by domain)

Analyzes caching effectiveness grouped by domain.

| Column | Description |
|--------|-------------|
| `domain` | The domain name. |
| `cacheType` | Description of the caching mechanism detected. |
| `count` | Number of URLs from this domain matching this cache type. |
| `avgLifetime` | Average cache lifetime in seconds, or empty string. |
| `minLifetime` | Minimum cache lifetime in seconds, or empty string. |
| `maxLifetime` | Maximum cache lifetime in seconds, or empty string. |

### 5.15. `caching-per-domain-and-content-type` (HTTP Caching by domain and content type)

Analyzes caching effectiveness grouped by both domain and general content type.

| Column | Description |
|--------|-------------|
| `domain` | The domain name. |
| `contentType` | The general content type category. |
| `cacheType` | Description of the caching mechanism detected. |
| `count` | Number of URLs matching this domain, content type, and cache type. |
| `avgLifetime` | Average cache lifetime in seconds, or empty string. |
| `minLifetime` | Minimum cache lifetime in seconds, or empty string. |
| `maxLifetime` | Maximum cache lifetime in seconds, or empty string. |

### 5.16. `non-unique-titles` (TOP non-unique titles)

Lists page titles that appear on more than one page.

| Column | Description |
|--------|-------------|
| `count` | The number of pages sharing this title. |
| `title` | The non-unique page title. |

### 5.17. `non-unique-descriptions` (TOP non-unique descriptions)

Lists meta descriptions that appear on more than one page.

| Column | Description |
|--------|-------------|
| `count` | The number of pages sharing this description. |
| `description` | The non-unique meta description content. |

### 5.18. `best-practices` (Best practices)

Summarizes the results of various best practice checks performed by analyzers.

| Column | Description |
|--------|-------------|
| `analysisName` | The name of the specific best practice check (e.g., `"Large inline SVGs"`, `"Heading structure"`, `"Brotli support"`). |
| `ok` | Count of URLs passing this check. |
| `notice` | Count of URLs with a notice-level finding. |
| `warning` | Count of URLs with a warning-level finding. |
| `critical` | Count of URLs with a critical-level finding. |

### 5.19. `accessibility` (Accessibility)

Summarizes the results of accessibility checks.

| Column | Description |
|--------|-------------|
| `analysisName` | The name of the specific accessibility check (e.g., `"Missing image alt attributes"`, `"Missing html lang attribute"`, `"ARIA roles and landmarks"`). |
| `ok` | Count of elements/pages passing this check. |
| `notice` | Count of notice-level findings. |
| `warning` | Count of warning-level findings. |
| `critical` | Count of critical-level findings. |

### 5.20. `source-domains` (Source domains)

Provides statistics about the domains from which resources were loaded.

| Column | Description |
|--------|-------------|
| `domain` | The domain name. |
| `totals` | A summary string showing total count, size, and time for resources from this domain (e.g., `"67/30MB/6.2s"`). |
| `HTML` | Summary string (count/size/time) for HTML resources from this domain. |
| `Image` | Summary string for Image resources. |
| `JS` | Summary string for JavaScript resources. |
| `CSS` | Summary string for CSS resources. |
| `Document` | Summary string for Document resources (e.g., robots.txt). |

**Extra row keys** (dynamic, present when data exists):
*   `Audio`, `Font`, `JSON`, `Other`, `Redirect`, `Video`, `XML` (String): Summary strings for additional content types, included only when resources of that type are present.
*   `totalCount` (String): Total number of resources loaded from this domain.

**Note:** The set of content type columns is dynamic. The declared columns (`HTML`, `Image`, `JS`, `CSS`, `Document`) are always present, but additional content type columns appear in row data based on what resource types were actually encountered during the crawl.

### 5.21. `content-types` (Content types)

Summarizes statistics grouped by general content type.

| Column | Description |
|--------|-------------|
| `contentType` | The general content type category (e.g., `"HTML"`, `"Image"`). |
| `count` | Total number of URLs of this content type. |
| `totalSize` | Total size in bytes for this content type. |
| `totalTime` | Total time spent fetching resources of this content type. |
| `avgTime` | Average time spent fetching a resource of this content type. |
| `status20x` | Count of URLs with a 2xx status code. |
| `status40x` | Count of URLs with a 4xx status code. |

**Note:** The status columns are dynamic. Additional columns like `status42x` (for HTTP 429) or `status30x`, `status50x` may appear depending on which status codes were actually encountered during the crawl. These dynamic columns will also be declared in the table's `columns` object.

### 5.22. `content-types-raw` (Content types (MIME types))

Summarizes statistics grouped by the specific MIME type reported in the `Content-Type` HTTP header.

| Column | Description |
|--------|-------------|
| `contentType` | The raw MIME type string (e.g., `"text/html"`, `"image/svg+xml"`, `"text/html; charset=utf-8"`). |
| `count` | Total number of URLs with this MIME type. |
| `totalSize` | Total size in bytes. |
| `totalTime` | Total time spent fetching. |
| `avgTime` | Average time spent fetching. |
| `status20x` | Count of URLs with a 2xx status code. |
| `status40x` | Count of URLs with a 4xx status code. |

**Note:** Like `content-types`, the status columns are dynamic. Additional status columns (e.g., `status42x`) appear when the corresponding status codes are encountered.

### 5.23. `dns` (DNS info)

Shows the DNS resolution information for the crawled domain(s).

| Column | Description |
|--------|-------------|
| `info` | A line of text representing part of the DNS resolution (e.g., the domain name, an IP address, the DNS server used). Presented as a simple text tree. |

### 5.24. `security` (Security)

Summarizes findings related to security HTTP headers.

| Column | Description |
|--------|-------------|
| `header` | The name of the security header being analyzed (e.g., `"Strict-Transport-Security"`, `"X-Frame-Options"`, `"Content-Security-Policy"`). |
| `ok` | Count of URLs where the header was configured correctly. |
| `notice` | Count of URLs with a notice-level finding. |
| `warning` | Count of URLs with a warning-level finding. |
| `critical` | Count of URLs with a critical-level finding. |
| `recommendation` | A string containing textual recommendations for improving the configuration of this header. |

**Extra row key:**
*   `highestSeverity` (String): The highest severity level found for this header across all URLs (e.g., `"ok"`, `"warning"`, `"critical"`).

### 5.25. `analysis-stats` (Analysis stats)

Provides performance metrics for individual analyzer methods.

| Column | Description |
|--------|-------------|
| `classAndMethod` | The class and method name of the analyzer function. |
| `execTime` | Total execution time in seconds spent in this method across all relevant URLs/data points. |
| `execCount` | The number of times this method was executed. |

**Extra row key:**
*   `execTimeFormatted` (String): Human-readable formatted execution time (e.g., `"0.012 s"`, `"1.234 s"`).

### 5.26. `content-processors-stats` (Content processor stats)

Provides performance metrics for content processor methods (HTML, CSS, JS, XML processors that run during the crawl).

| Column | Description |
|--------|-------------|
| `classAndMethod` | The class and method name of the content processor function. |
| `execTime` | Total execution time in seconds spent in this method. |
| `execCount` | The number of times this method was executed. |

**Extra row key:**
*   `execTimeFormatted` (String): Human-readable formatted execution time.

### 5.27. `external-urls` (External URLs)

Lists external URLs discovered during the crawl along with where they were found.

| Column | Description |
|--------|-------------|
| `url` | The external URL that was discovered. |
| `count` | The number of times this external URL was found across all crawled pages. |
| `foundOn` | The URL of the page where this external URL was found (typically the first occurrence). |

## 6. Note on Text Output

While this document focuses on the JSON output, SiteOne Crawler also offers a simpler Text output format (`--output-text-file`). The Text output provides a human-readable summary suitable for quick review in a terminal or text editor.

See the [Text Output Documentation](TEXT-OUTPUT.md) for more details on the Text format.