# FineType v2 — Container Domain Definitions # # Hierarchy: domain.category.type (locale is a field, not in the key) # Full label at inference time: domain.category.type.LOCALE # # Each definition is a transformation contract: # - broad_type: Target DuckDB type # - format_string: DuckDB strptime format (null if not strptime-based) # - transform: DuckDB SQL expression ({col} = column placeholder) # - transform_ext: Enhanced transform requiring a DuckDB extension # - validation: JSON Schema fragment for data quality checks # - tier: Path from root to parent in the inference graph # - decompose: Optional struct expansion for multi-field output # # Container domain covers: # - serialized data formats (JSON, XML, YAML, CSV) # - nested data structures (objects, arrays) # - recursive type inference on nested values # # IMPORTANT: Container types trigger RECURSIVE TYPE INFERENCE # When a column is classified as container.object.json, FineType will: # 1. Parse JSON and iterate over fields # 2. Apply classification to each field value # 3. Return a struct with typed fields # ───────────────────────────────────────────────────────────────────── # ═════════════════════════════════════════════════════════════════════ # CATEGORY: object # # Object/record formats (JSON, XML, YAML, TOML, etc.) # These are containers with structured key-value data that supports # recursive type inference on nested fields. # ═════════════════════════════════════════════════════════════════════ container.object.json: title: "JSON Object" description: > JavaScript Object Notation (JSON) object stored as VARCHAR. Represents structured key-value data that may be nested. RECURSIVE INFERENCE: When classified as container.object.json, FineType will parse the JSON and attempt to classify each field value independently. Nested objects trigger recursive classification. Example: Input: {"user": "john", "age": 30, "active": true} Output: Struct designation: universal locales: [UNIVERSAL] broad_type: JSON frictionless: type: object format_string: null transform: "CAST({col} AS JSON)" transform_ext: null decompose: null validation: type: string pattern: "^\\{.*\\}$" tier: [JSON, object] aliases: [json_object] samples: - '{"user": "john", "age": 30}' - '{"name": "Alice", "email": "alice@example.com", "active": true}' - '{"nested": {"key": "value"}}' icon: package canonical: "https://www.rfc-editor.org/rfc/rfc8259" references: null notes: > v1 migration: New for v2. JSON objects stored as VARCHAR strings. CRITICAL: Classification triggers recursive field-level inference. DuckDB JSON type allows NULL, arrays, objects, and scalar values. container.object.json_array: title: "JSON Array" description: > JSON array format: [val1, val2, val3, ...]. May contain objects, scalars, or nested arrays. RECURSIVE INFERENCE: When classified as container.object.json_array, FineType will: 1. Parse the array 2. Sample elements to infer a common element type 3. Return an array of that inferred type Example: Input: [1, 2, 3, 4, 5] Output: LIST Input: [{"id": 1}, {"id": 2}] Output: LIST> designation: universal locales: [UNIVERSAL] broad_type: JSON frictionless: type: array format_string: null transform: "CAST({col} AS JSON)" transform_ext: null decompose: null validation: type: string pattern: "^\\[.*\\]$" tier: [JSON, object] aliases: [json_list, array] samples: - "[1, 2, 3]" - '["a", "b", "c"]' - '[{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]' icon: package canonical: "https://www.rfc-editor.org/rfc/rfc8259" references: null notes: > v1 migration: New for v2. JSON arrays stored as VARCHAR. Recursive inference may fail if array elements are heterogeneous (mixed types). In such cases, returns LIST (untyped list). container.object.xml: title: "XML Document" description: > XML (Extensible Markup Language) document stored as VARCHAR. Structured hierarchical data with tags and attributes. RECURSIVE INFERENCE: When classified as container.object.xml, FineType will: 1. Parse XML and identify root element 2. Extract and classify child element values 3. Return as nested struct (partial, may require schema) designation: universal locales: [UNIVERSAL] broad_type: VARCHAR frictionless: type: string format_string: null transform: "CAST({col} AS VARCHAR)" transform_ext: null decompose: null validation: type: string pattern: "^<.*>.*$" tier: [VARCHAR, object] aliases: [xml_document] samples: - 'John30' - 'john@example.com' icon: package canonical: "https://www.w3.org/TR/xml/" references: null notes: > v1 migration: New for v2. XML is less common in modern data pipelines, and adoption is decreasing. DuckDB has limited native XML support (xmlparse requires extension). Recursive inference may be partial. container.object.html: title: "HTML Content" description: > HTML (HyperText Markup Language) content stored as VARCHAR. Detected by the presence of HTML tags (

,

, , , etc.). Unlike XML, HTML5 allows unclosed tags, unquoted attributes, optional closing tags, and void elements. Common in CMS exports, email templates, web scraping data, and rich text fields. designation: universal locales: [UNIVERSAL] broad_type: VARCHAR frictionless: type: string format_string: null transform: "REGEXP_REPLACE({col}, '<[^>]+>', '', 'g')" transform_ext: null decompose: text_content: "REGEXP_REPLACE({col}, '<[^>]+>', '', 'g')" tag_count: "CAST(LEN(REGEXP_EXTRACT_ALL({col}, '<[a-zA-Z][^>]*>')) AS INTEGER)" validation: type: string pattern: "^.*<(p|div|span|a|br|img|h[1-6]|ul|ol|li|table|tr|td|th|strong|em|b|i|form|input|button|select|textarea|header|footer|nav|section|article|main|aside|figure|figcaption|blockquote|pre|code|script|style|link|meta|head|body|html)[\\s>/ ].*$" minLength: 3 tier: [VARCHAR, object] aliases: [html_content, html_fragment] samples: - '

Hello world

' - '
' - '
' - '

Title

Content here.

' - '
  • Item 1
  • Item 2
' - '
Cell
' icon: package canonical: "https://html.spec.whatwg.org/" references: null notes: > New in v0.5.1. HTML is NOT well-formed XML: HTML5 allows unclosed tags (
, ), unquoted attributes, optional closing. Transform strips all HTML tags to extract plain text content. Detection relies on recognizing common HTML tag names to distinguish from XML (which requires well-formedness). Common in CMS exports (Shopify, HubSpot), email templates, and web scraping datasets. container.object.yaml: title: "YAML Document" description: > YAML (YAML Ain't Markup Language) document stored as VARCHAR. Human-readable data serialization format with nested indentation. RECURSIVE INFERENCE: When classified as container.object.yaml, FineType will: 1. Parse YAML text 2. Classify each value (may be nested) 3. Return as struct (complex nested structures may fail) designation: universal locales: [UNIVERSAL] broad_type: VARCHAR frictionless: type: string format_string: null transform: "CAST({col} AS VARCHAR)" transform_ext: null decompose: null validation: type: string pattern: "^[a-zA-Z_][a-zA-Z0-9_]*:\\s*.*" tier: [VARCHAR, object] aliases: [yaml] samples: - "name: John\nage: 30\nactive: true" - "user:\n id: 123\n email: john@example.com" icon: package canonical: "https://yaml.org/spec/1.2.2/" references: null notes: > v1 migration: New for v2. YAML is common in configuration files but less common in analytics data. DuckDB does not have native YAML support; parsing requires external crates and is slower than JSON. container.object.csv: title: "CSV Record" description: > Single CSV record or row stored as a delimited string. Format: value1,value2,value3 (may include quoted values). RECURSIVE INFERENCE: When classified as container.object.csv, FineType requires a schema (column names and types). With schema, parses and types each value. Without schema, returns unparsed CSV string. NOTE: CSV classification is rare for individual records. More common to classify at the column level (entire CSV file as one string). designation: universal locales: [UNIVERSAL] broad_type: VARCHAR frictionless: type: string format_string: null transform: "CAST({col} AS VARCHAR)" transform_ext: null decompose: null validation: type: string pattern: "^[^,]+(,[^,]+)*$" tier: [VARCHAR, object] aliases: [csv_row, delimited_record] samples: - "John,30,true" - '"Smith, John",30,Engineer' icon: package canonical: "https://www.rfc-editor.org/rfc/rfc4180" references: null notes: > v1 migration: New for v2. CSV is typically a multi-row format, but may appear as individual records in some contexts. Parsing requires explicit schema. container.object.s_expression: title: "S-Expression" description: > Balanced nested parenthetical serialization — the S-expression family: Lisp/Scheme source, linguistic constituency/dependency parse trees (Penn Treebank, OntoNotes: "(ROOT (S (NP (NN cat)) (VP (VBZ sits))))"), and code ASTs ("(program (call (id print) (string hi)))"). A nested structure of the form (head child child ...), distinguished from a comma/whitespace list by its balanced, recursively-nested parentheses. This is NOT container.array.comma_separated: parse trees carry Penn comma-tokens "(, ,)" that fool a delimiter detector into reading a CSV array, but the value is a single nested tree, not a delimited list. designation: universal locales: [UNIVERSAL] broad_type: VARCHAR frictionless: type: string format_string: null transform: "CAST({col} AS VARCHAR)" transform_ext: null decompose: null validation: type: string pattern: "^\\s*\\(.*\\)\\s*$" minLength: 5 tier: [VARCHAR, object] generate: on_request aliases: [sexp, s_expr, parse_tree, lisp] samples: - "(ROOT (S (NP (NN cat)) (VP (VBZ sits))))" - "(program (call (id print) (string hello)))" - "(+ (* 2 3) (- 4 1))" - "(a (b c) (d (e f)))" - "(S (INTJ (UH Uh)) (, ,) (NP-SBJ (PRP I)) (VP (VBP think)))" icon: package references: null notes: > New 2026-07-05 (company-reference audit; author-approved as a general S-expression type over a narrow parse_tree leaf). Mined from 1,292 corpus columns — `trees`/`parse_tree`/`ast` — every one mislabelled container.array.comma_separated by the delimiter detector. The 244-dim model cannot predict this leaf; it is recovered at profile time by the deterministic `s_expression_recovery` Sharpen guard when >=90% of a column's values pass the balanced-nested-paren structural check (finetype_core::structure::is_s_expression). Value-based (0048), no header gate — the structural signature is self-precise (corpus over-recovery: zero). NO retrain (0096; value-determinable → Sharpen). Generated on request only: the model does not predict this leaf, a rule recovers it, so its synthetic values exist for the training set's label coverage and not for the default output. # ═════════════════════════════════════════════════════════════════════ # CATEGORY: array # # Simple and complex array/list formats. # ═════════════════════════════════════════════════════════════════════ container.array.comma_separated: title: "Comma-Separated Values (Simple)" description: > Simple list of comma-separated values (not JSON array, plain CSV). Example: "apple,banana,cherry" RECURSIVE INFERENCE: Samples elements and infers common type, returns LIST. If all elements parse as integers, returns LIST. If mixed types, returns LIST. designation: universal locales: [UNIVERSAL] broad_type: VARCHAR frictionless: type: list format_string: null transform: "CAST({col} AS VARCHAR)" transform_ext: null decompose: "STRING_SPLIT({col}, ',')" validation: type: string pattern: "^[^,]+(,[^,]+)*$" tier: [VARCHAR, array] aliases: [csv_simple, comma_list] samples: - "apple,banana,cherry" - "1,2,3,4,5" - "red,green,blue" icon: list references: null notes: > v1 migration: New for v2. Simple comma-separated list (not JSON). Decompose splits on comma. Recursive inference samples to determine element type. container.array.pipe_separated: title: "Pipe-Separated Values" description: > List separated by pipe character: value1|value2|value3. Similar to CSV but uses pipe as delimiter. RECURSIVE INFERENCE: Samples elements and infers common type, returns LIST. designation: universal locales: [UNIVERSAL] broad_type: VARCHAR frictionless: type: string format_string: null transform: "CAST({col} AS VARCHAR)" transform_ext: null decompose: "STRING_SPLIT({col}, '|')" validation: type: string pattern: "^[^|]+(\\|[^|]+)*$" tier: [VARCHAR, array] aliases: [pipe_list] samples: - "apple|banana|cherry" - "1|2|3|4|5" icon: list references: null notes: > v1 migration: New for v2. Pipe-separated format common in legacy databases and log files. Decompose splits on pipe. container.array.semicolon_separated: title: "Semicolon-Separated Values" description: > List separated by semicolon: value1;value2;value3. Common in European CSV exports (where comma is decimal separator). RECURSIVE INFERENCE: Samples elements and infers common type, returns LIST. designation: universal locales: [UNIVERSAL] broad_type: VARCHAR frictionless: type: string format_string: null transform: "CAST({col} AS VARCHAR)" transform_ext: null decompose: "STRING_SPLIT({col}, ';')" validation: type: string pattern: "^[^;]+(;[^;]+)*$" tier: [VARCHAR, array] aliases: [semicolon_list] samples: - "apple;banana;cherry" - "1,5;2,3;4,7" icon: list references: null notes: > v1 migration: New for v2. Semicolon separator common in Europe (RFC 4180 variant with comma as decimal). Decompose splits on semicolon. container.array.whitespace_separated: title: "Whitespace-Separated Values" description: > List separated by spaces or tabs: value1 value2 value3. Example: "one two three" or "1 2 3 4 5". RECURSIVE INFERENCE: Samples elements and infers common type, returns LIST. designation: universal locales: [UNIVERSAL] broad_type: VARCHAR frictionless: type: string format_string: null transform: "CAST({col} AS VARCHAR)" transform_ext: null decompose: "STRING_SPLIT(TRIM({col}), '\\s+')" validation: type: string pattern: "^[^\\s]+(\\s+[^\\s]+)*$" tier: [VARCHAR, array] aliases: [space_separated, space_list] samples: - "apple banana cherry" - "1 2 3 4 5" - "red\tgreen\tblue" icon: list references: null notes: > v1 migration: New for v2. Whitespace separator common in plain-text logs and Unix tools. Regex split on whitespace. # ═════════════════════════════════════════════════════════════════════ # CATEGORY: key_value # # Key-value pair formats (simple string representation). # ═════════════════════════════════════════════════════════════════════ container.key_value.query_string: title: "Query String Parameters" description: > URL query string format: key1=value1&key2=value2. Commonly found in URLs and HTTP POST bodies. designation: universal locales: [UNIVERSAL] broad_type: VARCHAR frictionless: type: string format_string: null transform: "CAST({col} AS VARCHAR)" transform_ext: null decompose: null validation: type: string pattern: "^[^=&]+=([^&]*)(&[^=&]+=[^&]*)*$" tier: [VARCHAR, key_value] aliases: null samples: - "name=John&age=30&city=NewYork" - "q=python&page=1" - "user_id=123&token=abc123xyz" icon: key references: null notes: > v1 migration: New for v2. URL query string format. Values are typically URL-encoded. Recursive inference parses key-value pairs and types each value independently. # ═════════════════════════════════════════════════════════════════════ # Recursive inference notes: # ───────────────────────────────────────────────────────────────────── # # Recursive Inference Requirements: # - All container types MUST implement field/element-level classification # - Heterogeneous arrays default to LIST or LIST # - Nested objects/arrays should be classified recursively # - Deep nesting (>3 levels) may be truncated or marked as broad_object # ═════════════════════════════════════════════════════════════════════