--- name: configuring-imports description: Configure Celigo imports -- the destination step that writes records to external systems. Use when creating imports, choosing the adaptor type, setting up field mappings, lookups, upsert logic, AI agent imports, or file-based imports. --- # Configuring Imports An import is the **data destination** in a Celigo integration. It takes records from an upstream step and writes them to an external system -- REST APIs, databases, ERPs, file servers, or AI models. Every import has one adaptor type and, apart from the connection-less kinds in [Minimum Required Fields](#minimum-required-fields), is bound to exactly one connection. Imports handle six concerns: - **Field mapping** -- transforming source fields into the destination system's expected format (including value resolution via static maps and lookup tables). Uses Mapper 2.0 (`mappings[]` array) by default, including `NetSuiteHTTPImport`. `SalesforceImport` and `WrapperImport` support only Mapper 1.0 (`mapping.fields[]` / `mapping.lists[]`), and `NetSuiteDistributedImport` uses its own Mapper 1.0 dialect at `netsuite_da.mapping` - **Operation logic** -- create, update, upsert, delete, attach/detach - **Hooks** -- JavaScript pre/post processing at various pipeline stages (preMap, postMap, postSubmit). File-based imports that generate files from records also support postAggregate; an HTTP records import refuses it (422 `invalid_hook`) - **One-to-many** -- fan out a child array already on the input record. Set `oneToMany: true` and `pathToMany` to that path (e.g., `"lineItems"`) when one source record should create one import operation per child. After fan-out the child is `record` (`{{record.sku}}`) and the parent (without the array) is `record._PARENT` (`{{record._PARENT.id}}`). Leave `pathToMany` blank when the arriving record is itself an array (grouped rows from a `groupByFields` export or a file group) -- each element then becomes the record, with no `_PARENT`. A path on an array record processes zero children and still reports success; a blank path on an object record fails with `oneToMany_error`. `oneToMany` doesn't batch by itself: templates see `batch_of_records` only when `http.batchSize` is above 1 (the default varies by API) or an RDBMS query runs `per_page` - **Response mapping** -- extract fields from the import's API response back into the record for downstream steps. Configured on the flow's `pageProcessors[]` entry, but planned when building the import. Extracts read the response envelope: `id` (the destination-assigned record ID), `statusCode`, `ignored`, `errors`, `_headers`, `dataURI` (the record's link, when the import sets `dataURITemplate`), and `_json` (the response body after any response transform, whose top-level fields are also flattened onto the envelope). Use `id` for a created record's ID and `_json.fieldName` for other response fields (e.g., `_json.output.1.content.0.text` for OpenAI responses) -- `_json.id` works only when the body itself has a top-level `id`. Flow test runs never send imports, so `id` is empty there. Response mapping uses Transformation 1.0 syntax (extract/generate pairs), not the newer expression-based transforms - **postResponseMap hook** -- JavaScript processing after response mapping merges the response back into the record. Configured on the flow's `pageProcessors[]` entry, but planned when building the import. Use to transform or enrich the merged record before downstream steps Imports are used across flows, APIs, and tools. ## Import Execution Pipeline When records arrive at an import step, this pipeline executes in strict order: 1. **Input filter** (optional) -- records that don't match skip this import (see [Gotchas](#gotchas)). Configured as `filter` on the import resource itself -- an `inputFilter` key on an import is silently discarded. See [configuring-filters](../configuring-filters/SKILL.md#import-filter-import-filter) 2. **preMap hook** (optional) -- JavaScript processing before field mapping 3. **Field mapping** -- Mapper 2.0 or 1.0 maps source fields to destination fields, including lookups and hardcoded values 4. **postMap hook** (optional) -- JavaScript processing after field mapping, before submission 5. **Submit to destination** -- writes the mapped record to the external system 6. **postSubmit hook** (optional) -- JavaScript processing after the destination responds (access response data, log results, trigger side effects) 7. **Response transform** (optional, `responseTransform`) -- reshapes the response body (`_json`) after postSubmit. Its expression rules are Transformation 1.0 only (`expression.version: "1"`; `"2"` is refused with 422), and it keeps only the fields it maps, so a field it drops is gone for response mapping. Its execution-log stage is `responseTransformation` 8. **Response mapping** (optional) -- carries data from the destination response back into the record for downstream steps. Configured on the flow's `pageProcessors[]` entry, not on the import itself 9. **postResponseMap hook** (optional) -- JavaScript processing after response mapping merges response data back into the record **Key distinction:** Response mapping lives on the flow's `pageProcessors[]` entry, not on the import resource. preMap, the mapping and postMap shape only the request this import sends: the next step receives the record as it arrived plus what response mapping and postResponseMap add, so a field added in steps 2-4 never reaches it. When building an import that needs to pass data downstream, plan the response mapping at flow design time. **What an HTTP import's templates see.** `http.relativeURI` and header values render against the record before mapping, and an undefined path fails the request: a field the record lacks in the URI -- or a `{{lookup.}}` whose lookup found nothing -- fails the record before anything is sent, with `handlebars_template_parse_error` (`Failed to generate request url from template: … Details: "" not defined in the model.`). That is missing data, not a template bug, so a retry fails the same way; an update import that some records reach without the destination id needs a composite UPDATE/CREATE or `ignoreMissing` (Common Errors). A field holding `""` is defined: it renders empty, and the request goes out without the id. A field holding null renders the text `null` instead (the request goes to `…/null`), and a field under an object that is null fails the record like a missing one (`"" not defined in null`) -- see writing-handlebars, gotcha 16. `http.body` renders against the mapped record, where an undefined path renders empty and the record still succeeds (`strictHandlebarEvaluation` doesn't change that). Leave `http.body` unset to send the mapped record; when you template it, use the mapped (destination) field names, and double braces for string values in a JSON body (gotcha 11). ## Categories of Import ### Record-Based Imports Submit structured records to APIs, databases, or ERPs. The vast majority of imports. - `NetSuiteDistributedImport` -- high-performance SuiteApp writes (add, update, addupdate, delete, attach, detach). The default for NetSuite when the Celigo SuiteApp or bundle is installed. `netsuite_da.restletVersion` is fixed when the step is created, so author new steps on `suiteapp2.0`; a hook's SuiteScript version must match the step's - `NetSuiteHTTPImport` -- NetSuite's REST APIs on a `netsuite` connection (`nsDomainType: "suitetalk"`: `POST /services/rest/record/v1/` to insert, `PATCH ...//{id}` to update; `"restlet"` for a custom RESTlet). Use it when neither the SuiteApp nor the bundle is installed, or the user asks for REST - `HTTPImport` -- REST/GraphQL APIs (POST, PUT, PATCH, DELETE). Supports connector-assisted (`formType: "assistant"`) and GraphQL (`graph_ql`, `assistant_graphql`) modes; see [GraphQL imports](#graphql-imports) - `SalesforceImport` -- Salesforce writes through `salesforce.api`: `soap` (the default), `rest`, or `compositerecord` (Composite Graph, a tree of related records in one request); `metadata` deploys configuration, not records. `salesforce.api` has no Bulk value - `RDBMSImport` -- SQL databases (Snowflake, PostgreSQL, MySQL, SQL Server, Oracle). `rdbms.queryType` is a one-element array whose value fixes the companion field: `per_record`, `per_page` (renders against `batch_of_records`) and `first_page` (runs once, on the first page) take `rdbms.query`, an array of SQL strings; `bulk_insert` takes `rdbms.bulkInsert` (pure INSERT, with no upsert or ignore-existing logic); `bulk_load` takes `rdbms.bulkLoad` (a staged load that becomes a MERGE with `primaryKeys`, or runs your own SQL with `overrideMergeQuery`). Prefer `bulk_load` where the database supports it, then `bulk_insert` for pure INSERTs, then `per_page`, and `per_record` only when the logic can't be batched. The legacy uppercase `INSERT` (and the retired `MERGE`, `COPY` and `BULK INSERT`) fail any save with 422 "Import query types … are deprecated"; `UPDATE` is legacy too, so don't use it. Other strings, such as lowercase `insert`, fail as an invalid enum value. A `bulk_load` import's GET also returns a `file` block describing its staging file; the platform derives it on every read, so a save that changes it is accepted and discarded. The SQL itself: [writing-sql](../writing-sql/SKILL.md) - `MongodbImport`, `DynamodbImport`, `JDBCImport` -- other databases #### GraphQL imports A GraphQL import is an `HTTPImport` with `http.formType` `graph_ql` (or `assistant_graphql`, the connector-assisted GraphQL form). Each `http.body[]` entry is a string holding the `{"query", "operationName", "variables"}` envelope, one per operation on an update/create pair -- never an object, which is stored as-is and doesn't render -- and the mutation's input comes from the record through Handlebars in `variables`. - **Endpoint.** The import posts to the connection's base URI -- its form has no relative-URI field and writes `relativeURI: ["/"]` on every save -- so the connection must supply the GraphQL endpoint: a `graph_ql` connection, or a connector connection bound to the app's GraphQL API. On a connection bound to the app's REST API every record fails (HTTP 406 on Shopify); add a second connection bound to the GraphQL API, re-point the GraphQL steps to it, and leave the shared REST connection unchanged. The form type is set on the step, so a plain `http` connection whose base URI is the endpoint works too. - **Variables.** Under `graph_ql` or `assistant_graphql`, a `variables` value stored as a JSON string is sent as an object; under `http` it goes out as a string. An object takes expressions inside string values only: `"variables": {"input": {"email": "{{record.email}}"}}`. A string can also carry bare values, but its string values sit two JSON levels deep, so give each one `jsonEncode` in double braces: `"variables": "{\"input\": {\"note\": \"{{jsonEncode record.note}}\", \"quantity\": {{record.qty}} } }"`. An object or array goes in as `{{jsonSerialize record.items}}`, also in double braces. With `{{record.note}}`, or triple braces on either helper, a value holding a quote fails its record with 422 `invalid_json` before anything is sent. - **Errors come back as HTTP 200.** A GraphQL server usually answers a failed mutation with 200 and an `errors` array, and with no response settings the import counts the record as written. Set `http.response.resourceIdPath` to the id only a successful mutation returns (`["data.customerCreate.customer.id"]`): any other answer then fails the record with `missing_id`, top-level `errors` and the mutation's own `userErrors` alike. For the server's message on top-level errors, also set `failPath: ["errors[0].message"]` and `errorPath: "errors[0].message"`; `failPath: ["errors"]` catches a response with exactly one error and lets two or more through. ### File-Based Imports Write or upload files to remote storage. Require the `file{}` configuration block. Two modes: - **Record-to-file** -- aggregates incoming records into a file (CSV, JSON, XML, XLSX). The `file{}` block defines the output format, and `file.aggregation.type` how a run's records become files: `all` (one file per run), `page` (one per page) or `key` (EDI file definitions only, with B2B Manager). Send the legacy `file.skipAggregation` as its pair -- `true` with `page`, `false` with `all` or `key` -- because validation reads the value you send: an EDI file definition on `page` without `true`, or a `file.batchSize` without it, fails with 422 `not_supported`. - **Blob passthrough** -- transfers a file as-is from an upstream blob export or blob lookup. Set `blob: true`, and `blobKeyPath` to the record field holding the `blobKey` that step put there (FTP and Google Drive imports fall back to the record's root `blobKey` when it's unset). Base64 content in a record is record data, not a blobKey. AI agent, guardrail and tool imports reject both fields with 422 `not_supported`. Adaptor types: - `HTTPImport` with `http.type: "file"` -- the folder mode of the cloud file providers (Google Drive, Box, Dropbox, Azure Blob Storage, Google Cloud Storage, Azure Data Lake Storage Gen2, Celigo Storage). Address the folder by path in `http.relativeURI` (a folder, never an API endpoint) or by the provider's folder id in `file.directory` (`pathMode: "directoryId"`), and name the file in `file.fileName`; provider-specific location rules are in [file.yml](references/schemas/file.yml) - `FTPImport` -- CSV, XML, JSON, XLSX, EDI files to FTP/SFTP - `S3Import` -- objects to Amazon S3 - `AS2Import` -- AS2 EDI file transmission - `FileSystemImport` -- local/on-premise filesystem writes ### AI Imports Invoke AI models for classification, extraction, or safety checks. No `_connectionId` required unless using BYOK. - `AiAgentImport` -- LLM invocations (any provider in the `aiAgent.provider` enum) with structured output, tool use, and reasoning - `GuardrailImport` -- PII detection, content moderation, or custom AI-based validation ### Stack and Tool Imports - `WrapperImport` -- custom connector code, typically a function on a stack (`wrapper.function`) - `ToolImport` -- runs the Celigo Tool named by `tool._toolId`. It needs no `_connectionId`; instead it must bind each connection the tool's steps use in `tool.overrides.connections` as `{_abstractId, _id}` -- set `_id` equal to `_abstractId` to keep the tool's own connection. A connection left unbound fails the run with `connection_not_overridden` ### Topic Imports (Event Streams) - `TopicImport` -- publishes each record as a message to an Event Streams topic, which topic listeners (`TopicExport`) read. Set `topic._topicId` to the topic's `_id`, never its name (`celigo topics list`; an unknown id fails the save with 422 `invalid_ref`), and optionally `topic.batchSize` (default 1000). There's no connection and no create/update logic: each record becomes one message, reshaped by the mapping if there is one; with one-to-many, each element is its own message carrying the parent under `_PARENT`. Register the topic on the flow's integration so its flows can bind to it (`celigo integrations register-topics `); the account needs the Event Streams entitlement. Test runs publish nothing; an invoke publishes for real ## Import Matching (create / update / upsert) The core decision on most record-based imports is the **operation** -- what happens to each record at the destination. Create requires no match key; update and delete require one; upsert checks first and does whichever applies. Users describe the intent in business terms ("look up the customer and update them", "match by email and upsert", "skip the ones that already exist") that resolve into five behaviors: - **Always create** -- every record is submitted as new. No matching, no checks. - **Create only if missing** -- match-check first; submit as create if not found, skip silently if found. - **Update only if exists** -- match-check first; submit as update if found, skip silently if not. - **Update only if exists, fail if missing** -- strict variant; no-match records error out instead of skipping. - **Upsert** -- submit a create when no match is found, an update when matched. The catch-all, and the right default when the user is vague. When matching applies, decide four things: the **matching behavior**, the **match key field(s)** (`email`, `external_id`, `customer.id` -- required for anything other than always-create), the **on-match action** (update, skip, fail), and the **on-no-match action** (create, skip, fail). A lookup on a cross-reference field (NetSuite `externalid`, or a custom field that holds the source id) finds only records whose field holds that id, so the same import must write the source id into the field when it creates them; without that mapping no later run finds them, and each run adds the record again. How a destination implements matching is adaptor-specific -- there is no single "matching mode" field. A destination might expose: a native **upsert** keyed off an external ID (Salesforce upsert, NetSuite upsert, RDBMS `ON CONFLICT`); a distinct `addupdate` operation that handles both paths in one call; an `ignoreExisting` flag paired with a lookup that probes before writing (lock it when repeats can arrive concurrently -- see [Writing Exactly Once](#writing-exactly-once)); two separate create and update endpoints with no upsert variant (see composite imports below); or a lookup endpoint plus separate create and update endpoints, where the lookup runs pre-write to drive the create-vs-update decision. Some destinations have no matching concept at all -- writing a CSV to FTP, sending an email, posting to a webhook -- so every record goes out as-is. **Prefer import-level matching over a separate lookup step.** Imports natively support this pre-write check, so a single import that does the matching and the write together means fewer steps, fewer round-trips, and no glue logic to maintain. A standalone lookup earns its place only when the looked-up data has a consumer *beyond* the write -- a router branching on something other than "does this exist", an AI agent reasoning over the result, or multiple downstream steps reading different fields. If the only consumer is the destination call itself, the work belongs inside the import. It's also the only check that runs under the import's concurrency lock, which matters whenever the same record can arrive twice ([Writing Exactly Once](#writing-exactly-once)). An import's own lookups refuse an ambiguous match only when told to: an HTTP lookup that finds more than one match fails the record only with `allowFailures: false`, and otherwise sends the first match with no error (gotcha 14); a NetSuite DA lookup with `allowFailures: true` also resolves to one of the matching records unless `useDefaultOnMultipleMatches` sends it to the `default` ([netsuitedistributed.yml](references/schemas/netsuitedistributed.yml)). So search on a key that identifies one record. When an update's key must come from a separate lookup that can match several records, guard it -- fail or skip on multiple matches, or check for exactly one ([configuring-lookup-caches > Safe Lookup-to-Update](../configuring-lookup-caches/SKILL.md#safe-lookup-to-update----never-write-off-an-unguarded-first-match)). ### Composite (two-endpoint) imports When an HTTP destination has no native upsert but exposes separate create and update endpoints, a **composite import** pins both endpoints on a single import node, role-tagged create and update. The runtime picks per record via a match-key check -- the same way a native upsert would -- so the flow stays one step. Prefer this over two separate imports driven by an upstream lookup or router; reach for separate imports only when the create and update paths must diverge beyond endpoint selection (different mappings, different downstream consumers, or different hook chains). The operations sit in positional arrays, one entry per operation: `http.requestType` (`["UPDATE", "CREATE"]`), `http.method` and `http.relativeURI`. Exactly one decider picks per record: `http.existingExtract`, a field the record already carries (non-empty means UPDATE, whose URI reads it as `{{{data.0.}}}`), or `http.existingLookupName`, a lookup in `http.lookups` that probes the destination (the UPDATE URI reads `{{lookup.}}`). Never add `ignoreExisting` to a composite import, and give every `http.response.*` array one entry per operation. `existingExtract` must hold the destination's id (put there by an upstream step's response mapping, or kept as a cross-reference), never the source record's own id: every record carries one, so every record would go to UPDATE. With `existingLookupName`, a lookup that comes back empty sends the record to CREATE, with no error -- and so does one whose `extract` resolves nothing, so a `$.`-prefixed `extract` sends every record to CREATE. A templated `http.body` has one entry per operation too; left unset, both requests send the mapped record. A search-driven upsert against an API that finds contacts by email: ```json "http": { "requestType": ["UPDATE", "CREATE"], "method": ["PATCH", "POST"], "relativeURI": ["/contacts/{{lookup.findContact}}", "/contacts"], "existingLookupName": "findContact", "lookups": [{ "name": "findContact", "method": "POST", "relativeURI": "/contacts/search", "body": "{\"email\": \"{{email}}\"}", "extract": "results[0].id" }] } ``` The lookup sits in `http.lookups` (a `lookups` array sent only at the top level is dropped on save), its request template is in `body`, and the template names the record's fields at the top level (`{{email}}`, never `{{record.email}}`, which renders empty, so every search misses and every record is created again; gotchas 10 and 11). Its `extract` is the id itself, because that is what the UPDATE URI receives. An HTTP lookup's `extract` is a path into the response written without `$.` -- `results[0].id`, or `ids` for an array of ids -- unlike a Mapper 2.0 row's `extract`, which needs `$.`. A `$.` prefix, `[*]`, or a dot path through an array (`results.id`) resolves nothing, so the lookup always comes back empty. ### Writing Exactly Once Every source can hand an import the same record twice: a sender fires a webhook again, a failed or canceled run is re-run over the same delta window, an error is retried, a record that wrote this step but failed a later one comes back on the next run, or another flow writes the same destination. **A trace key never de-duplicates** -- it correlates a record's errors and is the default lock key below; a record that arrives twice is written twice. Make the write itself idempotent, in this order of preference: | Rank | Pattern | Why it holds under concurrency | |---|---|---| | 1 | **A key the destination enforces** -- an upsert or insert on a primary key or unique constraint (`ON CONFLICT`, `MERGE` -- see [writing-sql](../writing-sql/SKILL.md)), a unique external ID, an API that rejects a duplicate reference | The destination refuses the second copy | | 2 | **`ignoreExisting: true` + the existence check on this import + `idLockTemplate` on the same business key.** `ignoreExisting` (like `ignoreMissing`) is a top-level import field, not part of `http`. The check is `http.ignoreLookupName` (a lookup on the key) or `http.ignoreExtract` (the record already carries the destination ID) -- exactly one -- or the adaptor's equivalent (NetSuite `internalIdLookup`, Salesforce `idLookup.whereClause`) | The existence check runs inside the per-key lock | | 3 | **A ledger step** in front of a destination that can't be queried -- [configuring-lookup-caches > Deduplication Ledger](../configuring-lookup-caches/SKILL.md#deduplication-ledger) | The ledger's claim is itself rank 2 | A match that is really a search run before the write -- a lookup-driven composite create/update, an `addupdate` found by search -- is a check-then-write, not rank 1, unless the destination also rejects the duplicate key. A natural id is unique only within its source: when two sources can produce the same one (two stores, two accounts), namespace the key per source (`{{record.storeId}}-{{record.orderId}}`), or one source's record collides with or overwrites the other's. Rank 2 is only as good as its lookup: a lookup that fails counts as "not found", so the record is written (gotcha 12). **The concurrency ID lock.** `idLockTemplate` renders a key per record; records that render the same key are never submitted at the same time, and different keys run in parallel. - **Default:** the record's trace key (a source's `traceKeyTemplate` arrives on the record as `io_custom_trace_key`). Without a template, the trace key is the first top-level id field the record has -- `id`, `internalId`, `_id`, `internal_id` or `identifier`, in a few spellings each -- and a nested id is never used. A blank trace key means **no lock**, so set `idLockTemplate` when the identity is nested or composite. A template replaces the default rather than adding to it. - **Render time:** from the record as it arrives, before this import's own lookups and mapping. A key built from `{{lookup.}}` or a mapped field renders empty -- and an empty key is no lock at all. Key it on a field the record carries (`{{{record.orderId}}}`), or one an upstream lookup step added through response mapping. - **Scope:** this import across every page, every concurrent run, and every flow that uses it (`celigo imports dependencies `) -- not the connection: two imports rendering the same key still run in parallel. So two imports that both create the same entity (an order flow that creates missing customers, and a separate customer sync) can each miss the other's write and create it twice: create the entity through one import, or schedule the flows so they don't overlap. Records within one page are sent one at a time anyway, so a one-page test never shows the race. - **Coverage:** the `ignoreExisting` existence check runs inside the lock. Nothing documents the same for `ignoreMissing`, the composite create/update decision, or one-to-many children -- where those must write once, connection `concurrencyLevel: 1` closes the check-then-write race (the import's pages run one at a time), at the cost of serializing everything on that connection. ## Quick Reference ### Adaptor Decision Matrix | Your data goes to... | Use adaptorType | Category | Read schema | |---|---|---|---| | REST or GraphQL API | `HTTPImport` | Record-based | [http.yml](references/schemas/http.yml) | | NetSuite, Celigo SuiteApp or bundle installed | `NetSuiteDistributedImport` | Record-based | [netsuitedistributed.yml](references/schemas/netsuitedistributed.yml) | | NetSuite, neither installed (or REST requested) | `NetSuiteHTTPImport` + `nsDomainType` | Record-based | [http.yml](references/schemas/http.yml) | | Salesforce objects | `SalesforceImport` | Record-based | [salesforce.yml](references/schemas/salesforce.yml) | | SQL database (Snowflake, PostgreSQL, etc.) | `RDBMSImport` | Record-based | [rdbms.yml](references/schemas/rdbms.yml) | | MongoDB | `MongodbImport` | Record-based | [mongodb.yml](references/schemas/mongodb.yml) | | DynamoDB | `DynamodbImport` | Record-based | [dynamodb.yml](references/schemas/dynamodb.yml) | | JDBC database (non-built-in) | `JDBCImport` | Record-based | [jdbc.yml](references/schemas/jdbc.yml) | | Files to cloud storage (Google Drive, Box, Dropbox, Azure Blob, Google Cloud Storage, Celigo Storage) | `HTTPImport` with `http.type: "file"` | File-based | [http.yml](references/schemas/http.yml) | | Files to FTP/SFTP | `FTPImport` | File-based | [ftp.yml](references/schemas/ftp.yml) | | Files to S3 | `S3Import` | File-based | [s3.yml](references/schemas/s3.yml) | | AS2 EDI transmission | `AS2Import` | File-based | [as2.yml](references/schemas/as2.yml) | | Local filesystem | `FileSystemImport` | File-based | [filesystem.yml](references/schemas/filesystem.yml) | | An LLM (any provider in the `aiAgent.provider` enum) | `AiAgentImport` | AI | [aiagent.yml](references/schemas/aiagent.yml) | | PII detection / content moderation | `GuardrailImport` | AI | [guardrail.yml](references/schemas/guardrail.yml) | | Celigo Tool | `ToolImport` | Tool | [tool.yml](references/schemas/tool.yml) | | Custom connector code on a stack | `WrapperImport` | Stack | [wrapper.yml](references/schemas/wrapper.yml) | | An Event Streams topic | `TopicImport` | Topic | [topic.yml](references/schemas/topic.yml) | **Raw HTTP is the fallback, not the default.** Pick the most specific match, in order: 1. **Native adaptor** -- if the application has its own row (NetSuite, Salesforce, databases, FTP/S3), use it. Do not build an `HTTPImport` against that app's REST API. 2. **Pre-built HTTP connector** -- for any other REST/GraphQL app, check the 550+ connector catalog before writing HTTP config (see [Check for a pre-built connector](#3-check-for-a-pre-built-connector)). The step is still an `HTTPImport`, but it runs on a connector-backed connection and takes its endpoint config from the connector. 3. **Manual HTTP** -- hand-write the config from public API docs only when no connector exists or it doesn't cover the operation you need. For NetSuite, run `celigo connections suiteapp-status `: `suiteapp.success` or `bundle.success` true means `NetSuiteDistributedImport`; both false, each with `statusCode` 404 and an install `URL`, means neither is installed, so use `NetSuiteHTTPImport`. A different `statusCode` means the check itself failed, not that the components are missing: fix the connection and re-run it before choosing. `NetSuiteImport` is legacy -- don't use it unless asked. `adaptorType` is **case-sensitive**: `NetSuiteDistributedImport`, not `netsuitedistributedimport`. ### Minimum Required Fields Every import needs at minimum: `name`, `adaptorType`, and the adaptor config block (`http{}`, `netsuite_da{}`, `salesforce{}`, `tool{}`, `topic{}`, etc.). `_connectionId` is required except on the connection-less imports: `ToolImport` (connections are bound per tool step in `tool.overrides`), `TopicImport` (must not carry one -- with a connection, the save is validated as that connection's adaptor and fails), and `AiAgentImport` / `GuardrailImport` unless they use a BYOK connection. ### Which Schemas to Read 1. Always: [request.yml](references/schemas/request.yml) (base fields) 2. Plus: the adaptor-specific file from the decision matrix 3. If file-based: also [file.yml](references/schemas/file.yml) 4. If cloning: [clone-request.yml](references/schemas/clone-request.yml), [clone-response.yml](references/schemas/clone-response.yml) ### Schema Index All schemas are in [references/schemas/](references/schemas/): - **Base fields (all imports):** [request.yml](references/schemas/request.yml) - **Response shape:** [response.yml](references/schemas/response.yml) - **Adaptor-specific config:** - [http.yml](references/schemas/http.yml) -- HTTP/REST/GraphQL (methods, URIs, headers, response parsing, lookups, composite upsert via `existingExtract` or `existingLookupName`) - [netsuitedistributed.yml](references/schemas/netsuitedistributed.yml) -- NetSuite SuiteApp (operation, recordType, internalIdLookup, mapping, lookups) - [netsuite.yml](references/schemas/netsuite.yml) -- NetSuite legacy - [salesforce.yml](references/schemas/salesforce.yml) -- Salesforce (sObjectType, operation, api, idLookup) - [rdbms.yml](references/schemas/rdbms.yml) -- SQL databases (queryType, query, bulkInsert, bulkLoad) - [ftp.yml](references/schemas/ftp.yml) -- FTP/SFTP - [s3.yml](references/schemas/s3.yml) -- Amazon S3 - [mongodb.yml](references/schemas/mongodb.yml) -- MongoDB (method, collection, filter, upsert) - [dynamodb.yml](references/schemas/dynamodb.yml) -- DynamoDB - [jdbc.yml](references/schemas/jdbc.yml) -- JDBC databases - [as2.yml](references/schemas/as2.yml) -- AS2 EDI - [wrapper.yml](references/schemas/wrapper.yml) -- custom stack connectors - [tool.yml](references/schemas/tool.yml) -- Tool imports (`_toolId`, connection overrides) - [topic.yml](references/schemas/topic.yml) -- Event Streams publisher (`_topicId`, `batchSize`) - [filesystem.yml](references/schemas/filesystem.yml) -- local filesystem - **AI config:** - [aiagent.yml](references/schemas/aiagent.yml) -- AI agent (provider, model, instructions, tools, structured output) - [guardrail.yml](references/schemas/guardrail.yml) -- guardrails (PII, moderation, AI-agent validation) - **File output:** [file.yml](references/schemas/file.yml) (CSV, XML, JSON, XLSX config for file-based imports) - **Clone:** [clone-request.yml](references/schemas/clone-request.yml), [clone-response.yml](references/schemas/clone-response.yml) ## Related Skills - [configuring-connections > Quick Reference](../configuring-connections/SKILL.md#quick-reference) -- connection types and auth for import destinations - [managing-file-storage > Use stored files in a flow](../managing-file-storage/SKILL.md#8-use-stored-files-in-a-flow) -- Celigo Storage folders as a file destination - [writing-mappings > Mapper 2.0 Workflow](../writing-mappings/SKILL.md#mapper-20-workflow) -- field mappings on imports - [writing-scripts > Data Pipeline Hooks](../writing-scripts/SKILL.md#data-pipeline-hooks) -- preMap, postMap, postSubmit, postAggregate hooks - [writing-handlebars > Quick Reference](../writing-handlebars/SKILL.md#quick-reference) -- dynamic values in URIs, HTTP bodies, SQL queries - [building-flows > How to Build a Flow](../building-flows/SKILL.md#how-to-build-a-flow) -- wiring imports into flow pipelines as page processors - [configuring-topics > How to Build an Event Stream](../configuring-topics/SKILL.md#how-to-build-an-event-stream) -- the topic a publisher writes to: registration, retention, messages - [troubleshooting-flows > Diagnostic Workflow](../troubleshooting-flows/SKILL.md#diagnostic-workflow) -- diagnosing import-related failures ## How to Build an Import ### 1. Identify the target application What system are you writing data to? This determines adaptor type, connection type, and configuration shape. ### 2. Check for existing patterns Before building from scratch, look at what already exists: ```bash # Search your account (fast, uses local index) celigo account search "" # Show what an existing import uses (connection) and what uses it (flows) celigo account dependencies import # Find orphaned imports not referenced by any flow celigo account lint # Search marketplace templates celigo templates marketplace # Extract just imports from a template celigo templates preview --model Import celigo templates preview --summary ``` The account index auto-refreshes when stale (older than 15 minutes by default). Force a fresh snapshot with `celigo account snapshot`. ### 3. Check for a pre-built connector **Always run this check before writing any HTTP config.** Celigo maintains 550+ HTTP connectors with pre-configured auth, endpoints, and resources. Hand-write a manual `HTTPImport` from public API docs only when this search comes up empty or the connector doesn't cover the operation you need. ```bash # Search HTTP connectors celigo http-connectors list | grep -i "" celigo http-connectors get --full # see endpoints, resources, auth config # Drill into the endpoints the connector defines for imports celigo http-connectors catalog --resource-type import --published-only celigo http-connectors endpoint-detail --resource-type import --resource-id --endpoint-id celigo http-connectors endpoint-detail --resource-type import --resource-id --endpoint-ids , # a create/update pair # Search trading partner connectors (EDI, AS2) celigo tp-connectors list ``` If a connector exists, create the connection from it (`http._httpConnectorId` -- see [configuring-connections > Check for a pre-built connector and global iClient](../configuring-connections/SKILL.md#4-check-for-a-pre-built-connector-and-global-iclient)) and take the import's `relativeURI`, method, and body/response shapes from the connector's endpoint metadata rather than reconstructing them from public API docs. Write each operation's `relativeURI` and `method` exactly as the endpoint defines them: saving links the import to those endpoints by setting `http._httpConnectorVersionId`, `http._httpConnectorResourceId` and `http._httpConnectorEndpointIds[]` (one per operation; imports drop the singular `_httpConnectorEndpointId`), but pointers you send for the same connector are kept as sent. Read the import back: missing pointers mean a URI matched no endpoint. When retargeting an import, update or drop the old pointers in the same save -- a GET-modify-PUT that changes only the URI leaves the import naming the old endpoint. `endpoint-detail` on the import plane returns the resource, whose `resourceFields` catalog the record's fields, and each endpoint's `supportedBy`: what the connector's own form applies. `preConfiguredFields` are values it sets, `fieldsToUnset` are fields it clears, and `fieldsUserMustSet` is what it asks for -- on Shopify's create-customer endpoint, `ignoreExisting` and the mapping targets `mapping.fields.generate.customer.email`, `…first_name` and `…last_name`. The paths are the form's, not the import's. On create and update endpoints, `lookupToIdentifyExisting` is the existence lookup to build (the endpoint to call and the `extract` that reads the existing id, such as `customers[0].id`), and `pathParameterToIdentifyExisting` names the update URI's id parameter. ### 4. Query metadata for the target system For NetSuite, Salesforce, and RDBMS connections, discover available record types and fields: ```bash celigo metadata types # List record types / sObjects / tables celigo metadata fields # List fields for an entity type celigo metadata fields --refresh # Bypass the cache after a field is created in the app ``` Metadata is cached on the server, so a field or record type created since the last read stays missing until you re-read with `--refresh`. - **NetSuite:** `metadata fields` returns field IDs, types, and groups. On `NetSuiteDistributedImport`, use the IDs in `netsuite_da.mapping.fields[].generate` and `netsuite_da.mapping.lists[].fields[].generate`; sublist names (groups such as `"item"`, `"addressbook"`) go in `netsuite_da.mapping.lists[].generate`. `NetSuiteHTTPImport` maps with Mapper 2.0 `mappings[]`. Lookup field IDs here are the `searchField`/`resultField` values for `netsuite_da.lookups[]`. - **Salesforce:** `metadata fields` returns field API names, types, and relationship info. Use field API names for `salesforce.sObjectType` lookups and for discovering which fields are createable/updateable. - **RDBMS:** `metadata fields` returns column names and types for a table — use these to write SQL queries (see `writing-sql`) and verify column names before building `bulkInsert.tableName` or `bulkLoad.tableName`. ### 5. Determine the category Is this a **record-based import** (submit records to an API/database/ERP), a **file-based import** (write files to storage), or an **AI import** (invoke a model)? ### 6. Choose the right adaptor type Refer to the [Adaptor Decision Matrix](#adaptor-decision-matrix) in the Quick Reference above. For NetSuite, [Diagnosing NetSuite Steps](../configuring-exports/references/netsuite.md) has the account-side checks (the SuiteApp and bundle, the account's concurrency limit) and the errors NetSuite steps return. ### 7. Build the import JSON Reference the [Schema Index](#schema-index) for the exact fields needed. Use the [Which Schemas to Read](#which-schemas-to-read) decision rule to determine which files to consult. ## File Uploads over HTTP (multipart/form-data) Some destination APIs accept files only as `multipart/form-data` POSTs (Jira attachments, QuickBooks attachables, OpenAI file uploads). This is an `HTTPImport` with `blob: true` and `blobKeyPath`, posting to the API's upload endpoint, and it works nothing like a JSON record write. Do **not** set `http.type: "file"`: that is the folder mode of file-storage connectors (Google Drive, Box, Dropbox, Azure Blob), which treats `relativeURI` as a folder path -- combined with an API endpoint URL it saves cleanly and fails every record with "Directory is not found." Four pieces have to line up: 1. **Where the bytes come from.** The import never carries the file itself. A preceding blob export or blob lookup pulls the file into blob storage, and that step's response mapping puts the reference onto the record -- the idiom is `{"extract": "data.0.blobKey", "generate": "blobKey"}`. The record then carries a `blobKey` (a storage pointer, not content). 2. **The media type.** The connection's media type (or the import's request media type override) is `multipart/form-data`; success/error response media types usually override to JSON so replies parse normally. 3. **The request body.** Not raw MIME and not a normal handlebars payload -- a JSON array of parts, each `{name, value, type}` plus optional `filename` (include the extension) and `mime-headers`. The file part is `"type": "attachment"` with `"value": "{{blob}}"` -- `{{blob}}` is the only accepted value for an attachment (anything else 422s). Fields the API wants alongside the file ride as `"type": "inline"` parts; an inline part whose value is a JSON object must be serialized. One file reference per import. 4. **`blob: true` + `blobKeyPath`** (Advanced settings) -- `blobKeyPath` is the JSON path in the record where the blobKey lives (`blobKey`, or `file.blobKey` if nested). At send time the platform follows it into blob storage and streams the real bytes into the attachment part. The platform assembles the final MIME body itself -- it generates the `boundary` (never hardcode one), writes each part's `Content-Disposition`, and substitutes the attachment part with the raw file bytes. The parts array is a build recipe, not the payload. **Not every multipart API is form-data.** Some upload endpoints expect `multipart/related` instead (e.g. Google Drive's `/upload/drive/v3/files?uploadType=multipart`), and the parts-array machinery does NOT apply. There the request body is the literal MIME document: explicit boundary, a JSON metadata part, and a content part referencing `{{blob}}` (double braces). Check which flavor the destination API documents before building -- mixing them produces an import that saves cleanly and fails at runtime. ## Async Destinations (submit, poll, confirm) Some destination APIs only acknowledge a write (an HTTP 202, a job ticket) and finish it in the background -- bulk loads, file ingestion, document conversion. Attach an **async helper** to the import (`http._asyncHelperId`) so the step submits, polls a status export until the external work completes, and only then resolves. The mechanics and constraints (a status export with done/error value lists and poll intervals; no transform, output filter, or hook on the async-configured step; helpers cannot nest) are identical to the export side -- see [configuring-exports > Async APIs (submit, poll, fetch)](../configuring-exports/SKILL.md#async-apis-submit-poll-fetch). Only add one when the API genuinely cannot confirm the write synchronously. ## Testing an Import Without Writing Most ways of trying an import send the request to the destination for real. Know which is which before you point one at a production system: | What you run | Reaches the destination? | |---|---| | `celigo imports invoke ` with records on stdin | **Yes -- a real write.** One result per record, every adaptor; no job is created | | `celigo imports invoke` with an import document on stdin (the preview route) | **Yes -- a real write**, despite the name. Only the first record of `data` runs, so try one record per call. NetSuite, Salesforce, RDBMS, and FTP imports are rejected with 422 `invalid_adaptor`; a bare document piped without `data` runs against one empty record | | `celigo imports invoke [] --connection ` (a virtual import) | **Yes -- a real write.** Runs the saved import, or a piped `{"import": {...}, "data": [...]}` document, through that connection, NetSuite and RDBMS imports included; one result per record | | `celigo flows test-run --export ` | **No.** Imports never execute in a test run -- each returns its saved `mockResponse`. A NetSuite DA import is the exception: the page still goes to NetSuite, where the SuiteApp or bundle runs the lookups and SuiteScript hooks and builds each record without saving it. Its stages are `mapping` and `lookup` (plus one per SuiteScript hook), not `request` and `response`, and an error NetSuite raises while building a record is real and counts on the job | | A real flow run, or `celigo flows retry-errors` | Yes | Invoking by id or as a virtual import runs the step with the import's own settings only: hooks get no integration or flow settings, and `{{settings.integration.*}}` and `{{settings.flow.*}}` render empty. To run it as a step of its flow, tool or API, add `--integration ` and `--flow`, `--tool` or `--api ` to the `--connection` form. In the flow builder's import editor, **Preview** builds the request without sending it (HTTP imports only), while **Send** is a real write. No CLI command offers the compose-only preview. From the CLI, test-run the flow and read the import's stages with `celigo flows test-run-step-results`: the `mapping` stage's output is the record after mapping, while the `request` stage shows the body template unrendered, and its input is the record before mapping. Database imports have no compose-only surface at all; the SQL editor's evaluation is their only preview. **Save a realistic `mockResponse`.** In test runs and flow-builder previews, an import with no saved mock answers with the placeholder `{"id": "1234567890", "mockResponse": "replace me"}` (numbered per record on a page: `1234567890-1`, `1234567890-2`), so response mappings extract nothing real and downstream steps see empty data. Save an array of records shaped like the destination's answer -- `statusCode`, the created record's `id`, and the response body under `_json`: ```bash celigo imports set 'mockResponse=[{"statusCode": 200, "id": "SO-1001", "_json": {"id": "SO-1001", "status": "created"}}]' ``` A saved `mockResponse` substitutes only in test runs and flow-builder previews -- `imports invoke`, real runs, and retries always send. `sampleResponseData` only seeds the response-mapping editor; test runs and previews never use it. ## CLI Commands ```bash # CRUD celigo imports list celigo imports get celigo imports create < import.json celigo imports update < import.json celigo imports set key=value [key2=value2 ...] celigo imports delete [-y] celigo imports audit # change history celigo imports dependencies # the flows, APIs and tools that use it # Invoke -- a real write: each record is sent to the destination (no job is created). # HTTP 200 even when records fail; check each result's statusCode and errors echo '[{"name":"test"}]' | celigo imports invoke # Virtual import: an unsaved or modified document through a connection (also a real write) echo '{"import": {...}, "data": [{"name":"test"}]}' | celigo imports invoke --connection # Mock response for test runs and flow-builder previews (never used by invoke or real runs) celigo imports set mockResponse=file://./mock-response.json # Clone and connection management echo '{"connectionMap":{"oldConnId":"newConnId"}}' | celigo imports clone celigo imports replace-connection # Discovery celigo templates marketplace celigo http-connectors list celigo http-connectors catalog --resource-type import --published-only celigo http-connectors endpoint-detail --resource-type import --resource-id --endpoint-id celigo tp-connectors list celigo metadata types celigo metadata fields # Debug -- a window of at most 60 minutes (debugUntil, set by PATCH; a PUT doesn't set it). # It records each request the import sends while open: not earlier ones, and nothing from test runs celigo imports enable-debug [--duration ] celigo imports disable-debug celigo flows debug-requests # read them from the consumer (or apis / tools debug-requests) ``` ## Pre-Submit Checklist ### Required (all imports) - [ ] `name` is set - [ ] `adaptorType` exact case matches connection type (request.yml > adaptorType) - [ ] `_connectionId` references a valid, online connection (skip for `ToolImport`, `TopicImport`, and AI imports without BYOK) - [ ] Adaptor config block name matches adaptorType (`http{}` for HTTPImport, `netsuite_da{}` for NetSuiteDistributedImport, etc.) ### Adaptor-specific - [ ] HTTP: pre-built connector was checked (`celigo http-connectors list`) -- hand-written config only because no connector covers the app or operation - [ ] HTTP: `http.method` and `http.relativeURI` are set (http.yml) - [ ] HTTP: string values in a JSON `http.body` or lookup `body` are double-braced inside quotes; triple braces only for numbers, booleans and serialized JSON (gotcha 11) - [ ] HTTP lookups: every lookup is in `http.lookups` (the save keeps that array and mirrors it to the top-level `lookups`; one sent only at the top level is dropped), each request template is in `body`, not `postBody`, and names the record's fields without `record.` (gotcha 10), and each `extract` is a path without `$.` to the id itself or an array of ids, from a search on a key that matches one record (gotcha 14) - [ ] NetSuite DA: `netsuite_da.operation` and `netsuite_da.recordType` are set, and `update`, `addupdate` and `delete` carry `netsuite_da.internalIdLookup` -- the lookup, not a mapped `internalid`, picks the record (on `update` it overrides a mapped one). When the record already carries the internal id, use the expression form `["internalid","is","{{{id}}}"]` rather than dropping the lookup (netsuitedistributed.yml) - [ ] NetSuite DA fields set at record initialization (NetSuite preloads them when it creates or transforms the record, before the other fields): one body row `{"generate": "celigo_initializeValues", "hardCodedValue": "entity,subsidiary"}` listing their ids, comma-separated and without `.internalid`, next to the rows that map those fields. It's what the mapping editor's "Use this field for NetSuite record initialization" saves; a `useAsAnInitializeValue` key on a row isn't stored - [ ] NetSuite DA serialized items: the SuiteApp and bundle take a line's `serialnumbers` as `()` text when the account's Advanced Bin / Numbered Inventory Management feature is off; with it on, give the serials through the line's inventory detail subrecord (`celigo_inventorydetail`, [netsuitedistributed.yml](references/schemas/netsuitedistributed.yml)) - [ ] NetSuite DA keyed on `externalid` (`["externalid","is","{{{id}}}"]`): a mapping writes that same source id to `externalid` (`{"extract": "id", "generate": "externalid"}`). The lookup finds only records whose `externalid` was set, so without the mapping an `addupdate` adds the record again on every run - [ ] RDBMS: `rdbms.queryType` is one current value (`bulk_load`, `bulk_insert`, `per_page`, `per_record` or `first_page`) with its companion field set -- never the legacy uppercase `INSERT` or `UPDATE` (rdbms.yml) - [ ] Salesforce: `salesforce.sObjectType` and `salesforce.operation` are set. `upsert` also needs `salesforce.upsert.externalIdField` (the Salesforce external id field) and `salesforce.idLookup.extract` (the record field matched against it); `update`, `delete`, `addupdate` and an `insert` with `ignoreExisting` need `idLookup.whereClause` instead (salesforce.yml). The mapping is Mapper 1.0: `mapping.fields[]` rows of `{extract, generate}` (or `hardCodedValue`), never `mappings[]` ### Cross-resource consistency - [ ] Connection `type` matches the import's `adaptorType` - [ ] If response mapping needed: configured on the flow's `pageProcessors[]` entry, not on the import itself - [ ] If one-to-many: `oneToMany: true`, with `pathToMany` set to the child array path for an object record (child fields then use `record.` and parent fields `record._PARENT.`) and left blank for an array (grouped) record - [ ] If downstream steps read this import's response: a realistic `mockResponse` is saved, so test runs carry real-looking data past this step - [ ] If a repeated record would duplicate data: the write is keyed (a destination-enforced key, or `ignoreExisting` + an existence check) and `idLockTemplate` renders a non-empty key from a field the record carries on arrival -- never `{{lookup.*}}` ([Writing Exactly Once](#writing-exactly-once)), and its existence lookup has been checked against a record that exists (gotcha 12) - [ ] Mapper 1.0 only where required: `mapping.fields[]` / `mapping.lists[]` on Salesforce and wrapper imports, `netsuite_da.mapping` on NetSuite DA -- never `mappings[]` there. `NetSuiteHTTPImport` uses `mappings[]` ## Gotchas 1. **PUT erases omitted fields.** Always GET first, modify, then PUT. The `set` command handles this. The import PATCH route accepts only `debugUntil` (any other path is refused with 422), so a change to a mapping, body or lookup is always this full PUT, never a reason to send the user to the UI. Leave the read-only fields a GET adds (`_id`, `lastModified`, `createdAt`) out of the body. 2. **Including a `rest:` block creates a legacy RESTImport.** Use only `http:` for new imports. 3. **Input filter skips that import, not the record.** Filtered records skip the current import step but continue to subsequent steps in the flow, without this import's response-mapped fields, so a later step that writes those fields back needs its own filter on them. Check `numIgnore` on the job if records seem to bypass a step. 4. **Multipart file parts only accept `{{blob}}`.** In a `multipart/form-data` parts array, the file part must be `"type": "attachment"` with `"value": "{{blob}}"` -- any other value 422s. Never hardcode the MIME `boundary`; the platform generates it. Fields the API wants alongside the file ride as `"type": "inline"` parts. 5. **`bodyKey`/`blobKey` in logs is an artifact, not a payload field.** Audit and debug logs never show the assembled multipart body -- an internal storage pointer appears where the body would be. But if the destination actually received the literal string `bodyKey` or `blobKey`, the file part is misconfigured (an `inline` part where an `attachment` belongs, or a `blobKeyPath` that doesn't resolve). 6. **`imports invoke` is a real write.** "No job created" doesn't mean nothing is sent: every form executes the request against the live destination, and a saved `mockResponse` doesn't stop them. It also runs the import's hooks, postSubmit included, and its response transform (the editor's **Send** stops before postSubmit). A 200 can still carry per-record failures in each result's `errors`, and on a file import (FTP, S3, file-mode HTTP) it means the file was composed, not delivered: an FTP import to an unreachable host still answers `{name, fileLocation}`. Point it at a sandbox, or test-run the flow when you only need to check mapping output. 7. **A trace key is not duplicate detection.** Repeated keys are normal (delta re-emits, retries) and never skip, merge, or reject a record. Only a keyed write stops the second copy -- see [Writing Exactly Once](#writing-exactly-once). 8. **A lock keyed on this import's own lookup is no lock.** `idLockTemplate` renders before the import's lookups and mapping run, so `{{lookup.}}` renders empty. Check the key against a real record (an error's retry data: `celigo flows error --retry-data`), not a sample that may lack the field. 9. **Lookups can't chain.** A lookup's own URI, body and headers render before any lookup has run, so `{{lookup.}}` there is undefined: it either fails the render (`handlebars_template_parse_error`: `Unable to read "" from undefined`) or renders empty, and either way that lookup fails or asks the wrong question for every record. Test runs never execute imports, so the first real run is where it shows. Resolve the value in one query, or add an upstream lookup step and reference the field it adds to the record. 10. **An HTTP import's lookups send `body`, not `postBody`, and read the record's fields at the top level.** Put a lookup's request template in `http.lookups[].body`. A template saved as `postBody`, the field [http.yml](references/schemas/http.yml) documents, saves without complaint, but the lookup goes out with an empty body, so it asks the wrong question for every record and nothing errors. The CLI's schema check calls `body` undocumented; it's still the field that's sent. Inside that template the record's fields sit at the top level: `{{email}}` renders the record's value, and `{{record.email}}` renders empty. 11. **In a JSON body, string values take double braces.** `{"name": "{{record.name}}"}` is JSON-escaped at runtime, so a quote, backslash, newline or tab in the value arrives intact. Triple braces insert the raw text: such a value fails its record with 422 `invalid_json` before anything is sent, and in a lookup's `body` the lookup quietly finds nothing (gotcha 12). Keep triple braces for numbers, booleans and serialized JSON (`{{{jsonSerialize record}}}`). 12. **A failing existence lookup counts as "not found".** With `ignoreExisting`, an `http.ignoreLookupName` lookup that fails -- a wrong URI answering 404, a body broken by a raw quote -- extracts nothing, so the record is written as new and no error is raised. Before trusting the check, invoke the import in a sandbox with a record that already exists there and confirm the result comes back `ignored`. 13. **A 200 isn't always success.** When an API answers 200 for a failed write, set `http.response.successPath` + `successValues` or `failPath` + `failValues` (one entry per operation; values compare as strings). A record that trips them fails: an invoke reports `statusCode` 422, and a run records an error whose code is the destination's status (`200`), source `application`, with the response body as its message -- or `response_failure` ("response failed using path") when a `successPath` value doesn't match. 14. **An ambiguous HTTP lookup fails only with `allowFailures: false`.** When a lookup's `extract` resolves to two or more values (an array of ids or of result objects), the record fails with `lookup_conflict` ("Multiple matches found for lookup.") only if the lookup sets `allowFailures: false`. With it left out or `true`, the first value is sent and nothing errors -- from an array of objects, the literal text `[object Object]` -- and `useDefaultOnMultipleMatches` hasn't been seen to send it to the `default` instead. An indexed `extract` (`results[0].id`, `ids[0]`) takes the first match silently, whatever `allowFailures` says: right for a search that can match only one record, and never a fix for `lookup_conflict`, which it turns into writes against whichever record comes first. Fix ambiguity in the search: query on a key that identifies one record (an email, an external id), taken from a field the record carries -- a failed record's retry data shows them -- and keep `allowFailures: false`, so a search that still finds two fails rather than writing against the wrong record. ## Common Errors | Error | Cause | Fix | |-------|-------|-----| | 422 "Invalid value '' for field 'adaptorType'" | Wrong case (`httpimport`) or an unknown adaptor | Use the exact case from the decision matrix: `HTTPImport`, `NetSuiteDistributedImport`, etc. | | 422 "Expected field: _connectionId to be present" | No connection on an adaptor that needs one | Set `_connectionId`; only the connection-less imports in [Minimum Required Fields](#minimum-required-fields) omit it | | 422 `invalid_ref` ("connection not found.; is not a valid _connectionId") | The connection id doesn't exist in this environment | Use an id from `celigo connections list` | | Copying an Integration App's import fails: 403 `create_not_allowed` ("Cannot create an Integration App resource"), 422 `invalid_ref` ("The referenced Connection document ... belongs to different Integration App installation"), or 400 from `imports clone` ("Cloning can't be performed for ... that were installed from an Integration App.") | The body still carries the app's `_connectorId`, or the copy keeps the app's connection, which only the app's own resources can use; a clone of an app's import is refused outright | Build the copy as your own import, on a connection of your own, with `_connectorId`, `_id` and the timestamps left out | | 422 "http subschema not defined" (or another connection type's name) | The `adaptorType` doesn't match the connection: the save validates the import as the connection's adaptor | Pick the adaptor for the connection's type from the decision matrix, with its config block | | 422 "`` is not a valid enum value for path ``" | A value outside the field's enum, such as `http.method: ["PUSH"]` | Use a value the adaptor's schema lists | | 422 `Import query types … are deprecated` | `rdbms.queryType` holds the legacy uppercase `INSERT` (or a retired `MERGE`, `COPY` or `BULK INSERT`) | Use one current value -- `bulk_load`, `bulk_insert`, `per_page`, `per_record` or `first_page` -- and set its companion field | | 422 `… is not a valid enum value for path rdbms.queryType.0` | A `queryType` string outside the enum, such as lowercase `insert` | Use one of the current values above | | 422 `distributed required` | Missing NetSuite flag | Use `NetSuiteDistributedImport` with `distributed: true` on the import (there's no such connection flag); check the SuiteApp with `celigo connections suiteapp-status ` | | 422 `mapping invalid` | Wrong mapper version | Salesforce uses Mapper 1.0 (`mapping.fields[]`) and NetSuite DA uses `netsuite_da.mapping`, not Mapper 2.0 (`mappings[]`); `NetSuiteHTTPImport` uses `mappings[]` | | `Unable to find lookup in mapping configs - ` on every record (source `mapping`) | A mapping row's `lookupName` names a lookup the import doesn't define -- on an HTTP import, often one sent only in the top-level `lookups`, which the save drops; the save doesn't check it | Define the lookup under that exact name (in `http.lookups` on an HTTP import), or fix the row | | Every record fails with `Cannot set properties of undefined (setting '_hasStarNotation')` | A NetSuite DA mapping condition (`conditional.lookupName`) names a lookup the import doesn't define; on a sublist row the job can stall instead | Fix the name or restore the lookup -- changing the SuiteScript version doesn't help | | `INVALID_HANDLE_BAR_EXPRESSION` on a NetSuite DA sublist lookup | The lookup's `expression` reads a bare line field, which is undefined at the record root | Address the line as `[*].` (`{{{line_items[*].sku}}}`) | | `VALUE_LOOKUP_FAILED` "Could not find a match for …" | The rendered criteria, quoted in the message, matched no NetSuite record; inactive records don't count unless the lookup sets `includeInactive` | Fix the criteria or the data, or set `allowFailures: true` with a `default` to fall back | | `VALUE_LOOKUP_FAILED` "More than one match for …" (`MORE_THAN_ONE_MATCH_FOUND` on a `suiteapp2.0` step) | The criteria matched several records; an empty source value searches `["","isempty"]`, which matches every blank one | Keep an empty key out of the search (branch in the `expression`), or set `useDefaultOnMultipleMatches` with `allowFailures: true` and a `default` | | NetSuite `INVALID_KEY_OR_REF` ("Invalid reference key .") on some records | The field references another NetSuite record, and the value NetSuite received isn't a key it accepts. When the quoted value is a name or code and the row writing that field sets `internalId: true` (its value must be the referenced record's internal id), those records carry the name where the id belongs ([Diagnosing NetSuite Steps](../configuring-exports/references/netsuite.md) has the other causes) | Resolve the name to the id on that row with a lookup: a static `map` from each name to its internal id (the ids are the user's to supply), or a NetSuite lookup that searches the name field and returns `resultField: "internalid"` ([netsuitedistributed.yml](references/schemas/netsuitedistributed.yml)). Then retry the failed records; a retry before the change fails again | | NetSuite `USER_ERROR` (`source: application`) whose message states a business rule ("PO # is required for customers in the Distributor category") | A rule in the NetSuite account -- often its own user event script or workflow -- refused the record. Celigo's hooks report their own `source` (`pre_map_hook`, `post_submit_hook_ss` and the like), so `application` means NetSuite raised it | Read a failing record's retry data for the value the rule asks for and check whether a mapping row writes it: map it, or have a NetSuite administrator change the rule. Don't edit a Celigo hook for it ([Diagnosing NetSuite Steps](../configuring-exports/references/netsuite.md)) | | Records the destination already has show as ignored, and their changes never arrive | `ignoreExisting` skips each record its existence check finds; it never updates one | To update them, make the import a composite UPDATE/CREATE that decides on the same lookup (`http.existingLookupName`; [Composite imports](#composite-two-endpoint-imports)) and drop `ignoreExisting` | | `The mapping condition:"record_updated" not supported for the operation` | A `record_updated` condition on a NetSuite DA `add` import | Use `addupdate` or `update`, or drop the condition | | A record fails with code `200`, source `application`, and the response body as its message (or `response_failure`) | The import's `failPath` / `successPath` rules judged a 200 a failure | Expected when the API reports failures in a 200 body (gotcha 13); fix the record, or the rule if it misfires | | 422 attachment `value` invalid | Multipart file part is not `{{blob}}` | Set the file part to `"type": "attachment"`, `"value": "{{blob}}"`; let the platform generate the `boundary` | | 422 `invalid_json` ("An invalid JSON is sent in the request.body") on some records only | A triple-braced string in a JSON `http.body` met a value holding a quote, backslash, newline or tab; in a GraphQL `variables` string, any string value without `jsonEncode` | Double-brace string values inside quotes (gotcha 11); inside a GraphQL `variables` string, `{{jsonEncode record.x}}` ([GraphQL imports](#graphql-imports)) | | A GraphQL import reports every record written, and the destination has none | The server answered 200 with an `errors` array or the mutation's `userErrors`, and the import sets no response rules | Set `resourceIdPath` to the created object's id, and `failPath`/`errorPath` to `errors[0].message` ([GraphQL imports](#graphql-imports)) | | `missing_id` ("could not extract id using path: …") | No id at `resourceIdPath`: the write failed (the response, quoted in the message, carries the server's errors) or the path is wrong | Read the quoted response; fix the data, or the path | | `handlebars_template_parse_error`: "Failed to generate request url from template: … Details: "" not defined in the model." (source `application`, classification `parse`) on some records | Those records lack the field the `relativeURI` names, or its `{{lookup.}}` found nothing -- missing data, not template syntax. Nothing was sent, and a retry fails the same way | When the records may not exist at the destination yet (new items, no destination id), make the import a composite UPDATE/CREATE that decides on the field (`http.existingExtract`; [Composite imports](#composite-two-endpoint-imports)), or skip them with `ignoreMissing: true` and `http.ignoreExtract` set to the field. Otherwise get the id onto the record upstream, then retry | | 422 `unsupported_type` ("NetSuiteHTTPImport does not support type: file") | `http.type: "file"` on a `NetSuiteHTTPImport`: the NetSuite REST import writes records, and the save refuses file mode | Remove `http.type` | | Every record fails with "Directory is not found" | `http.type: "file"` (file-storage folder mode) combined with an API upload endpoint | Remove `http.type: "file"`; keep `blob: true` + `blobKeyPath` and post to the endpoint | | 422 `invalid_adaptor` ("Import preview is not supported for adaptor type") | `celigo imports invoke` with a document on stdin, for a NetSuite, Salesforce, RDBMS, or FTP import | Save the import and invoke it by ID, or run the document as a virtual import with `--connection` (both real writes); or test-run the flow to check mapping output without sending | | 422 `Connection is offline` from a mapping preview or a NetSuite field check | The destination connection is marked offline (its `offline` flag), so the platform doesn't call it; nothing is wrong with the import or its mapping | Fix the connection (test it, re-authorize), then retry | | A Salesforce import fails with `response stream exceeded limit of 5242880 bytes.` | The platform reads at most 5 MB of Salesforce's answer to a request, and this batch's results were larger | Send fewer records per request | | Salesforce `UNABLE_TO_LOCK_ROW` on some records -- on the `rest` and `compositerecord` APIs the platform replaces Salesforce's message with "Salesforce creates a lock on a record while updating it. If another request is trying to update the same record which is already locked, Salesforce would throw this error." and a help link | Another write held the Salesforce record this one needed: this import's own pages or runs submitting in parallel, or another import or flow writing the same records | Set `idLockTemplate` to the key of the record the writes contend for -- the record's own id, or the parent id they share when they update children of one record, taken from a field the incoming record carries (`{{{record.AccountId}}}`; gotcha 8) -- so pages carrying the same key never submit together through this import. The lock doesn't reach other imports: when another flow writes the same records, lower the connection's `concurrencyLevel` instead. Then retry the failed records | | A Salesforce import reports a field error for a record that Salesforce saved | On the `soap` API (the default), when Salesforce refuses a value in a field the import mapped (the error names the field), the platform drops that field and sends the record again, up to `maxAttempts` more times (default and maximum 3; `maxAttempts: 1` turns it off). A resend that succeeds saves the record without the field, and its error still counts against the job | Look the record up in Salesforce before retrying: a retry of an `insert` creates it a second time. Fix the value at the source, update the saved record, and resolve the error | | A `compositerecord` import fails a record with `Composite Operation failed due to the following error(s)`, then one line per failure (` for []`) | One object in the record's tree failed. The platform sends a record and its related records as one all-or-none Composite API request, so Salesforce saved none of them; the rollback notices it returns for the other objects (`PROCESSING_HALTED`, `ALL_OR_NONE_OPERATION_ROLLED_BACK`) are left out, and the error's code is the first real failure's | Fix the field each line names on that object, then retry the whole record | | Salesforce `ENTITY_IS_DELETED` on an `update` or `delete` | The record was deleted in Salesforce | With `ignoreMissing: true` the import counts such records as ignored instead of failing them; without it, resolve the errors once the deletions are confirmed as intended | | The destination refuses the text `Invalid date` in a date field (its error quotes that value) | A date mapping (`extractDateFormat`, `extractDateTimezone` or `generateDateFormat` set, Mapper 1.0 or 2.0) couldn't read the source value with its `extractDateFormat` -- `YYYY-MM-DD HH:mm:ss` when unset -- so it wrote the text `Invalid date` | Set `extractDateFormat` to the format the source actually sends (a failed record's retry data shows the value), then retry the failed records | | Downstream steps see `1234567890` or `"replace me"` in a test run | No `mockResponse` saved on the upstream import -- test runs never call the destination | Save a realistic `mockResponse` on that import | | The destination receives the same record twice | A bare create; `ignoreExisting` with no lock on records whose trace key is blank, or with an existence lookup that fails (gotcha 12); or a lock keyed on the import's own lookup | Key the write and lock it on the business key ([Writing Exactly Once](#writing-exactly-once)); find which runs wrote it first ([building-flows > Duplicate Records](../building-flows/SKILL.md#duplicate-records)) | | A NetSuite `addupdate` adds the same record again on every run | `internalIdLookup` searches `externalid`, but no mapping writes `externalid`, so the lookup never finds what earlier runs added | Map the source id to `externalid` and keep the lookup; the duplicates already added stay until someone merges them | | `lookup_error`: "Could not find value for static lookup: {Lookup Key: } …" on some records | The value isn't a key of that lookup's `map`, and the lookup has `allowFailures: false`; the other records succeed | Add each missing value as a key, exactly as the message quotes it (case, spacing and suffixes count), with the destination's value (ask the user -- a guessed code sends the record wrong), keeping every existing key in the save, then retry the failed records. A retry runs the step as it is now, so one before the map changes fails again. A `default` would instead send every unmapped value as the default, with no error | | `lookup_conflict`: "Multiple matches found for lookup." (source `lookup`) on some records | An HTTP lookup with `allowFailures: false` got two or more values at its `extract`: the search key isn't unique (a name several records share) | Search on a key that identifies one record and keep `allowFailures: false`. Don't index the `extract` (`ids[0]`) or allow failures to get past it: either one writes against whichever match comes first (gotcha 14) | | `batch_lookup_evaluation_failed`: ": The batch lookup could not be evaluated. Check the value in the key '', and try again." (source `internal`) | An HTTP lookup with `allowFailures: false` got nothing usable for that value: the search matched nothing, or the `extract` resolves nothing (a `$.` prefix, `[*]`, a dot path through an array) or points at a result object rather than an id | Check the `extract` against a real response -- a path without `$.` to the id (`results[0].id`) or an array of ids; if the value truly has no match, fix the data or the search, then retry | | 422 `cannot_set_traceKeyTemplate` | `traceKeyTemplate` set on a one-to-one import | Set it on the source export or listener -- on an import it is valid only with `oneToMany: true`. To change the lock key, set `idLockTemplate` | | `oneToMany_error` | A blank `pathToMany` on an object record ("Record at index 0 was not an array, and no PathToMany was provided"). The editor's Preview can also show it falsely on a step fed an array, because the preview re-splits the record it displays | Object record: set `pathToMany` to the child array. Array (grouped) record: blank is right, and each row is the record, with no `_PARENT` -- confirm with a test run before changing the configuration |