--- name: troubleshooting-flows description: Diagnose and resolve Celigo flow failures -- total failures, partial errors, stuck jobs, empty runs, and performance issues. Use when a flow is failing, producing errors, returning no data, or running slowly. --- # Troubleshooting Flows A flow is broken when it fails to move data correctly. This skill covers systematic diagnosis: identifying the problem type, isolating the failing step, inspecting errors, and resolving them. Troubleshooting concerns: - **Job status** -- understanding what `completed`, `failed`, `canceled`, and `retrying` mean for the flow - **Error analysis** -- grouping errors by pattern to find root causes instead of reading them one-by-one - **Request/response inspection** -- seeing exactly what was sent and returned at each step - **Execution logs** -- record-level tracing through every stage of the pipeline - **Retry and resolution** -- fixing error data and retrying vs bulk resolving - **Delta/state issues** -- `lastExportDateTime` drift, stuck deltas, re-processing windows ## Problem Categories ### Total Failure Job status is `failed` with 0 successful records. The entire run collapsed before processing any data. Typically `numPagesGenerated: 0` (export-level failure) or pages generated but `numPagesProcessed: 0` (import-level failure on the first page). Common causes: connection failure (credentials expired, endpoint down), export query error (invalid SQL, bad saved search ID), missing/deleted resource, permission denied. ### Partial Failure Job status is `completed` but `numError > 0` alongside successful records. Some records failed while others processed normally. Real-world data shows wide variance -- from 2 errors in 8000 successes to 500+ errors in 8000 successes. Common causes: validation errors on the destination (required fields missing, type mismatches), duplicate key violations, record-level lookup failures, rate limiting on specific batches, data-dependent issues (specific records have bad data). ### Empty Run Job completes successfully with 0 errors AND 0 records processed. Common causes: wrong `resourcePath` on the export (extracts from wrong JSON path), delta export with no changes since last run (legitimate), output filter too restrictive (all records filtered out), source query returns no results, webhook export with no inbound events. ### Stuck or Long-Running Job stays in `running` or `retrying` status longer than expected. Common causes: large dataset with no pagination limits, destination system slow to respond, script hook with long-running logic, on-premise agent connectivity issues, rate limiting causing backoff. ### Intermittent Failures Flow sometimes succeeds and sometimes fails with the same configuration. Common causes: token/session expiry mid-run (long-running flows), rate limiting (varies with concurrent flows), transient network errors, source system maintenance windows. ## Error Diagnosis Framework ### Classification When an error occurs, classify it into one of three categories to determine the right action: | Category | HTTP status codes | Meaning | Action | |---|---|---|---| | **Needs investigation** | 400, 401, 403, 404, 405, 409, 422 | Missing info, wrong IDs, permission denied, validation errors | Stop and investigate -- check resource config, connection status, permissions | | **Transient** | 408, 429, 500, 502, 503, 504 | Timeouts, rate limits, server errors | Retry once. If it fails again, escalate -- the external system may be down | | **Configuration error** | varies | Preconditions not met but fixable | Follow the error message guidance to fix the config, then retry | A 5xx error not in the transient list (e.g., 501) is still likely transient. A 4xx error not in the investigation list warrants manual review. ### Root Cause: Configuration vs Data Every flow error has one of two root causes: - **Static configuration** -- a hardcoded value in the step config is wrong (mapping expression, filter rule, hardcoded field, URI, query, SQL statement). Fix: change the resource configuration via `celigo set` - **Dynamic data** -- the upstream source sent unexpected data (missing required field, wrong type, null where a value is expected, unexpected array/object shape). Fix: add input filtering or validation upstream, or fix the source system **To distinguish:** check if the error reproduces with different input records. If the same error occurs for every record, it's configuration. If only some records fail, it's data. ```bash # Check if all records fail (configuration) or only some (data) celigo flows error-summary # Compare error count vs total records celigo flows errors # Sample specific errors to compare ``` ### Which Step Failed? Error location determines which resource and skill to investigate: | Error location | Resource to check | Skill | |---|---|---| | Export / page generator | Export config (connection, query, resourcePath) | configuring-exports | | Import / page processor | Import config (mapping, destination fields, operation) | configuring-imports | | Script hook | Script code (preSavePage, preMap, postMap, postSubmit) | writing-scripts | | Mapping | Mapping expression (field paths, lookups, hardcoded values) | writing-mappings | | Filter | Filter expression (s-expression syntax, field references) | configuring-filters | | Connection | Connection config (auth, URL, credentials) | configuring-connections | ## Quick Reference ### Symptom --> First Command | Symptom | Run first | Then | |---|---|---| | Flow totally failed | `celigo jobs list --flow --limit 1` | Check `numPagesGenerated` -- if 0, export failed; check connection and query | | Partial errors | `celigo flows error-summary ` | `celigo flows error-analysis ` to find root cause pattern | | Empty run (0 records) | `celigo jobs list --flow --limit 1` | Check export config (`resourcePath`, delta state, output filter) | | Stuck / long-running | `celigo jobs current --flow ` | Check job status; if `retrying`, inspect rate limiting or connection issues | | Intermittent failures | `celigo jobs run-stats --flow ` | Compare failing vs passing runs; check token expiry and rate limits | | **Silent logic bug** (no errors, wrong output) | `celigo flows test-run --export ` | If test-run can't reach it, enable execution logging (§6) and run for real | | **Production incident** (real traffic matters) | `celigo flows enable-execution-logs ` then run | Read per-record I/O with `query-execution-logs` / `execution-log-detail`; the failing stage names the problem | ### Key Diagnostic Commands ```bash # Job status celigo jobs list --flow --limit 1 # Most recent job celigo jobs current --flow # Currently running job celigo jobs diagnostics # Full diagnostic bundle # Error investigation celigo flows error-summary # Per-step error counts celigo flows error-analysis # Group errors by pattern celigo flows errors # List individual errors (each has an errorId) celigo flows error --request-detail # Raw HTTP request/response for one error # Safe iteration first celigo flows test-run --export # safe, fast, try this first # End-to-end execution logging (real run, full per-record I/O) celigo flows enable-execution-logs # arm debug logging, then run the flow celigo flows execution-logs # list captured per-record logs celigo flows debug-requests # per-bubble HTTP request/response ``` ## Related Skills - [building-flows > Quick Reference](../building-flows/SKILL.md#quick-reference) -- flow structure, topologies, and configuration - [configuring-exports > Quick Reference](../configuring-exports/SKILL.md#quick-reference) -- export configuration and adaptor types - [configuring-imports > Quick Reference](../configuring-imports/SKILL.md#quick-reference) -- import configuration and adaptor types - [writing-scripts > Quick Reference](../writing-scripts/SKILL.md#quick-reference) -- script hook debugging and data shapes ## Diagnostic Workflow ### 1. Check the job status Start with the most recent job to understand what happened. ```bash celigo jobs list --flow --limit 1 celigo jobs get celigo jobs current --flow ``` Key fields: `status`, `numError`, `numSuccess`, `numIgnore`, `numPagesGenerated`, `numPagesProcessed`, `startedAt`, `endedAt`. A `failed` status with `numPagesGenerated: 0` means the export itself failed -- don't look at import errors. `completed` with `numError > 0` means partial failure at the record level. ### 2. Get the error summary See which steps have errors and how many. ```bash celigo flows error-summary ``` This returns per-step error counts. Focus on the step with the most errors first. ### 3. Analyze error patterns Group errors by message pattern to find the root cause instead of reading them one-by-one. ```bash celigo flows error-analysis [--limit 200] ``` If most errors share the same message, that's your root cause. Multiple distinct patterns may indicate multiple issues. ### 4. Inspect individual errors Once you know the pattern, look at specific records and the HTTP request/response that produced the error. ```bash celigo flows errors # list open errors (note the errorId of each) celigo flows error --retry-data # inspect one error + its editable retry data celigo flows error --request-detail # + the captured HTTP request/response ``` `flows error --request-detail` is the most powerful diagnostic -- it resolves the error's `reqAndResKey` for you and shows exactly what HTTP request was sent and what the destination responded with. (If you already hold a `reqAndResKey` from `debug-requests`, use `debug-request-detail` instead.) ### 5. Use test runs for safe iteration (try this first) Test runs process a single page without affecting production data or delta state. Fast, safe, no arming, no side effects -- answers most logic questions. ```bash celigo flows test-run --export celigo flows test-run-step-results celigo flows test-run-step-logs ``` `test-run` returns `{metadata, flowJob, childJobs}` -- metadata lists stage names per bubble. Follow with `test-run-step-results` to get `stages[] = [{name, input, output, errors}]` per bubble. Errors include `retryData` inline. Test runs don't advance the delta timestamp -- you can repeat them safely against the same data. **Limitations:** imports don't actually submit, mock data is shared across lookups/imports, and some adaptors don't run in test mode. When these bite, escalate to §6. ### 6. End-to-end debugging with execution logs (silent/logic bugs, production incidents) When test-run can't answer it -- imports must actually submit, destination behavior matters, or it's a production incident -- arm execution logging, run the flow for real, then read the per-record I/O the run captured. Disable debug when you're done. ```bash # 1. Arm debug logging on the flow (optionally bound the window) celigo flows enable-execution-logs [--duration ] # 2. Trigger the run (or wait for the next scheduled run) celigo flows run -y # 3. After the run, list the captured per-record logs for the job celigo flows execution-logs # 4. Drill into one record's stages and stage data celigo flows query-execution-logs --export-or-import-id --group-id --record-id celigo flows execution-log-detail --export-or-import-id --stage --group-id --record-id # 5. Disarm debug logging celigo flows disable-execution-logs ``` Each per-record log entry names the stage that produced it (matching the `test-run-step-results` stage shape: `{ name, input, output, errors }`), so the failing stage tells you where the record broke. For raw HTTP at a bubble, use `debug-requests` / `debug-request-detail` (§7). Errors carry `retryData` inline -- see §8 to fix and retry. ### 7. Low-level debug primitives (surgical control) These are the debug primitives the §6 workflow builds on -- use them directly when you want manual control over arming, clearing, or probing: ```bash # Flow-level execution logging celigo flows enable-execution-logs [--duration ] celigo flows disable-execution-logs celigo flows execution-logs celigo flows query-execution-logs --export-or-import-id --group-id --record-id celigo flows execution-log-detail --export-or-import-id --stage --group-id --record-id # Per-bubble HTTP request/response celigo flows debug-requests [--since 60] celigo flows debug-request-detail # Per-resource debug toggles (capture raw HTTP at a specific bubble) celigo exports enable-debug celigo imports enable-debug celigo scripts enable-debug celigo connections enable-debug ``` **Stage names** (for `execution-log-detail --stage`): - Built-in: `apiCall`, `transformation`, `mapping`, `inputFilter`, `outputFilter`, `responseMapping`, `responseTransformation`, `routing` - Script hooks: the **function name** wired on the bubble (e.g., `preMapHook`, `postSubmitHook`, `branchingHook`, `preSavePageHook`, `postResponseMapHook`) **Test-run uses a different stage vocabulary** than live `/logs/data/query`: `request`/`response`/`parse` (three stages) instead of live's merged `apiCall`; `transformTwoDotZero` instead of `transformation`; `responseMap` instead of `responseMapping`; `router` instead of `routing`. Everything else matches. The live execution-log commands (§6) use the live vocabulary. ### 8. Fix and retry (or resolve) **Fix the configuration**, then retry: ```bash celigo flows retry-errors -y celigo flows retry-errors key1,key2,key3 ``` **Fix the data** when specific records have bad values: ```bash celigo flows error --retry-data > data.json # Edit data.json (the retryData object), then push it back by errorId: celigo flows update-error-data < data.json celigo flows retry-errors ``` **Resolve without retry** when errors are expected or not worth reprocessing: ```bash celigo flows resolve-errors errorId1,errorId2 celigo flows resolve-errors -y ``` ### 9. Verify the fix Run the flow again and confirm clean execution. ```bash celigo flows run -y celigo jobs list --flow --limit 1 celigo flows error-summary ``` ## Monitoring Lenses and Execution Metrics Before diagnosing, pick the lens that matches the question -- all execution state comes through one of two: - **Running** -- jobs executing right now. The live view: in-progress jobs with real-time progress (records processed so far, errors accumulating, pages generated, whether the export phase is done). Use it for "what's happening this moment" -- a long-running job you're watching, or a flow you just kicked off. Reach it with `celigo jobs current --flow `. - **Completed** -- historical aggregate per flow. The rear-view: run counts, average runtime, success / error / ignore totals, open errors, and when it last executed or errored. Use it for "how have things been going" -- slow flows, error-prone flows, "did the order sync run today." Reach it with `celigo jobs list --flow ` and `celigo jobs run-stats --flow `. Quick test: now --> running; recently / over time --> completed. ### Reading the Execution Metrics Keep the record-level counts straight so you don't misread a run: - **Success, error, and ignore are three distinct outcomes per record.** `numSuccess` processed cleanly; `numError` failed; **`numIgnore` is not an error** -- it's a record the flow intentionally skipped (filtered out, or a no-op upsert). Don't fold ignores into error counts: a run with a high `numIgnore` and `numError: 0` is healthy, not broken. - **Open vs resolved errors.** Open errors are the failures still unresolved and needing attention -- the number that matters for triage (`celigo flows errors `). Resolved errors -- cleared by auto-retry or by a user -- are no longer open (`celigo flows resolved-errors `). A flow with many total errors but zero open errors has already recovered; don't chase it. ## Stuck-Flow Triangulation Every connection is backed by its own FIFO message queue, shared by every flow (and API or Tool step) that uses that connection and drained up to the connection's `concurrencyLevel` in parallel. When a run seems stuck -- `queued`, "not moving", "started but nothing is happening" -- the cause is almost never visible on the resource's configuration. Triangulate three live facts before offering any explanation: 1. **The run's actual state** -- the latest job: `queued`, `running` (with or without progress), or already finished. Get it with `celigo jobs current --flow ` and `celigo jobs list --flow --limit 1`. 2. **Queue depth on each connection the steps drain through** -- how many messages are already in line ahead of this run. 3. **What else is running or queued right now** on those same connections -- other flows and integrations contending for the same drain. A bare `queued` status is not a diagnosis on its own. Calibrate every conclusion to the legs you actually verified: - **Verified blocker** -- runtime evidence links it to this run: a queue this run's steps drain through currently holds messages ahead of it. - **Likely hypothesis** -- consistent with the evidence but not linked at runtime. Another flow's job running or queued right now on a shared connection is a real observation whose blocking role is still an inference (per-slot occupancy isn't visible) -- present it as the likely cause, never confirmed. A flow that merely shares the connection but isn't currently active is a candidate at most. - **Unknown** -- a leg couldn't be fetched. Name exactly which evidence is missing and keep the diagnosis unconfirmed. **A shared connection alone is never proof.** "Another flow also uses this connection" becomes a diagnosis only when that flow's work is running or sitting in the queue right now. **Speak in flows and integrations, not queues and exports.** Walk the chain up and name the flows (and their integrations) producing the load: "your run is waiting behind ~1,200 messages on the NetSuite connection, mostly from 'Inventory Sync', which is running right now" is a diagnosis; "the connection queue has 1,200 messages" is not. **A backlog is an explanation, not a defect -- but verify it drains.** Work queued ahead means the run is waiting its turn, not broken; it starts when the queue drains. One snapshot can't tell a draining queue from a wedged one, so re-check in ~10-15 minutes. If the second look shows the same depth with the job still waiting -- especially with nothing else running or queued on those connections (idle capacity next to a standing line is itself an anomaly) -- stop advising patience and escalate. Raising `concurrencyLevel` can lift throughput, but that's a connection change, not the fix for a one-time backlog. **Truly stuck = empty queues + nothing else active + still not moving.** That combination rules out the account and points at the platform -- capture a `celigo jobs diagnostics ` bundle and escalate to Celigo support. **A fix is a hypothesis until a later run proves it.** After any corrective action -- re-enabling the flow, a config change, or canceling a wedged job (`celigo jobs cancel `) -- re-run and compare before/after job state before calling the incident resolved. A config-only change with no new run is not runtime-validated. ## CLI Commands All commands shown in the Diagnostic Workflow above, plus these additional commands: ```bash # Job inspection (additional) celigo jobs cancel [-y] celigo jobs diagnostics celigo jobs download-files celigo jobs get celigo jobs errors celigo jobs run-stats [--flow ] [--status ] # Error investigation (additional) celigo flows resolved-errors # Error resolution (additional) celigo flows assign-errors [errorIds] [-y] celigo flows delete-resolved-errors [errorIds] [-y] celigo flows tag-errors # Debug logging (additional) celigo flows query-execution-logs --export-or-import-id --group-id --record-id # Flow state celigo flows last-export-date celigo flows run [--start-date ] [--end-date ] [-y] ``` ## Diagnostic Checklist Before escalating or concluding investigation: - [ ] Checked job status via `celigo jobs list --flow --limit 1` -- confirmed `status`, `numError`, `numSuccess`, `numPagesGenerated` - [ ] Ran `celigo flows error-summary` to identify which step(s) have errors - [ ] Ran `celigo flows error-analysis` to group errors by pattern and identify root cause - [ ] Inspected individual errors via `celigo flows errors` and `celigo flows error --retry-data` - [ ] Reviewed raw HTTP request/response via `celigo flows error --request-detail` - [ ] If errors are unclear: enabled execution logs, re-ran flow, and inspected record-level trace - [ ] If HTTP-level detail needed: used `celigo flows debug-requests` on the failing export/import - [ ] Verified fix by re-running the flow and confirming clean execution ## Gotchas 1. **A `completed` job can still have errors.** `completed` means the job finished, not that every record succeeded. Always check `numError` alongside status. 2. **`error-analysis` only samples up to `--limit` errors.** Default is 100. For flows with thousands of errors, increase the limit to get an accurate pattern distribution. 3. **`flows error` takes the `errorId` (the `_id` from `flows errors`), not a `reqAndResKey` or `retryDataKey`.** It resolves those internal keys for you: `--request-detail` follows the error's `reqAndResKey`, `--retry-data` follows its `retryDataKey`. Use `debug-request-detail` only when you already hold a raw `reqAndResKey` from `debug-requests`. 4. **Debug execution logs auto-disable after `--duration` minutes.** Default is 60. If your flow runs after the window expires, you get no logs. Enable, then run promptly. 5. **Test runs don't advance delta state.** This is intentional -- you can test repeatedly with the same data. But it means test runs always re-fetch the same records. 6. **Retrying resolved errors is not possible.** Once resolved, an error cannot be retried. Only resolve errors you're certain don't need reprocessing. 7. **`lastExportDateTime` drift causes re-processing or gaps.** If a delta export's timestamp is wrong, use `flows run --start-date` to override and reprocess a specific window. 8. **Empty `numPagesGenerated: 0` on a failed job means the export itself failed.** The problem is the source step -- check the connection, query, or endpoint, not the import. 9. **Execution log data is ephemeral.** Logs are retained for a limited time. Enable logging and run the flow promptly. 10. **`jobs diagnostics` produces a diagnostic bundle.** Use this when Celigo support asks for details -- it includes internal execution context not visible through other commands. 11. **`numIgnore` is not `numError`.** An ignored record was intentionally skipped (filtered out, or a no-op upsert), not failed. A run with a high `numIgnore` and zero `numError` is healthy -- don't triage it as a failure. 12. **Open errors are the triage number, not total errors.** A flow with thousands of total errors but zero open errors has already recovered via auto-retry or manual resolve. Compare `flows errors` (open) against `flows resolved-errors` before investigating. 13. **A `queued` status is not a diagnosis, and a shared connection is not proof.** Triangulate job state, connection queue depth, and what else is draining that connection first. Another flow blocks yours only when its work is running or queued right now -- merely sharing the connection makes it a candidate at most. A backlog means the run is waiting its turn; re-check in ~10-15 minutes before escalating. ## Common Errors | Symptom | Likely Cause | Diagnostic Steps | |---------|-------------|-----------------| | `failed` with `numPagesGenerated: 0` | Export-level failure (connection, query, endpoint) | Check connection status (`celigo connections ping`); review export config | | `completed` with high `numError` | Destination validation or data issues | `error-analysis` to find pattern; `flows error --request-detail` for HTTP detail | | `completed` with 0 records, 0 errors | Wrong `resourcePath`, empty delta, or filter too restrictive | Verify `resourcePath`; check `lastExportDateTime`; review output filter | | `retrying` for extended period | Rate limiting, slow destination, or large dataset | Check destination rate limits; review concurrency on connection | | Errors only on specific records | Data-dependent issue (missing fields, bad types, duplicates) | `flows error --retry-data` to inspect failing records; compare with successful records | | Intermittent `failed` on same flow | Token expiry mid-run, transient network, or rate limits | Compare timestamps of failures; check connection token refresh config | | 401/403 errors in `flows error --request-detail` | Expired credentials or insufficient permissions | `celigo connections ping`; re-authorize if OAuth; check API permissions | | Timeout errors | Slow destination or oversized payload | Reduce batch size; check destination system performance | | `queued` / `running` but not advancing | Waiting behind queued work on a shared connection, or a platform-side stall | Triangulate job state (`jobs current`), connection queue depth, and other flows draining that connection; re-check in 10-15 min. If empty queues + nothing active + still stuck, capture `jobs diagnostics` and escalate |