---
name: datafactory-tips
description: Operational tips and best practices for working with Microsoft Fabric Data Factory MCP tools. Use when executing queries, troubleshooting timeouts, creating dataflows end-to-end, or working with large datasets via the DataFactory.MCP tools.
---

# Data Factory Tips

Operational knowledge for working with Microsoft Fabric Data Factory.

## Must / Prefer / Avoid

### MUST DO
- Use `executeOption = "ApplyChangesIfNeeded"` on first refresh of API-created dataflows
- Re-add connections via `add_connection_to_dataflow` after `save_dataflow_definition` (save may wipe them)
- Use `list_workspaces` and filter by name — there is no `find_workspace` tool
- Validate each connection with `execute_query` after adding
- Create a new dataflow instead of reverting a multi-source one to single-source — there is no `remove_connection` tool

### PREFER
- Filter early in M queries to enable query folding (push filters to source)
- Expensive operations (sorting, aggregation) last — streaming operations (filter, select) first
- `execute_query` with `customMashupDocument` to iterate on M code before saving
- Standard engine for complex transforms; Fast Copy only for simple ingestion

### AVOID
- `get_authoring_guidance` — deprecated, author M directly
- Sorting early in query chains (requires reading all data before returning results)
- `SELECT *` equivalent without row limits during development — use "Keep First Rows" then remove

## Symptom Triage

| Symptom | Likely Cause | Read |
|---------|-------------|------|
| `DataflowNeverPublishedError` | Default SkipApplyChanges on first run | `destinations/dest-new-table.md` |
| `DestinationColumnNotFound` | Manual mappings for new table | `destinations/dest-new-table.md` |
| Credentials error on Lakehouse | Connection not bound | `datafactory-connections.md` (Binding) |
| FastCopy fails with transforms | Unsupported transform in fast copy | `datafactory-advanced.md` (Fast Copy) |
| Instant refresh fail (0-3s) | Privacy firewall or unpublished draft | `sources/multi-source.md` |
| Multi-source instant fail via API | Dirty dataflow or separate Lakehouse.Contents calls | `sources/multi-source.md` |
| `IsNewTarget = false` fails | Direct navigation on API-created dataflow | `destinations/dest-new-table.md` |
| Stale connections after revert | save_dataflow_definition doesn't remove connections | `datafactory-connections.md` (Troubleshooting) |

## Knowledge Files

| File | When to Read |
|------|--------------|
| `datafactory-core.md` | MCP tools, M basics, rolling dates, what-if queries |
| `datafactory-connections.md` | Connection discovery, creation, binding, gateways, troubleshooting connection errors |
| `datafactory-performance.md` | Query timeouts, chunking, query folding, connector selection |
| `datafactory-advanced.md` | Fast Copy limits, Action.Sequence, Modern Evaluator |
| `datafactory-pipelines.md` | Pipeline creation, Dataflow activities, chaining, scheduling |

### Destination Files (read only the one you need)

| File | When to Read |
|------|--------------|
| `destinations/dest-new-table.md` | Creating a new output table via MCP (most common path) |
| `destinations/dest-existing-table.md` | Writing to a table that already exists |
| `destinations/dest-troubleshooting.md` | Diagnosing refresh failures, connection issues, silent errors |

### Source Files

| File | When to Read |
|------|--------------|
| `sources/multi-source.md` | Combining Lakehouse + SharePoint/Web sources, AllowCombine |
| `sources/sharepoint-excel.md` | Reading Excel files from SharePoint via Web.Contents |

### M Templates (copy-paste-ready, no prose)

| File | When to Read |
|------|--------------|
| `templates/m-new-table-destination.m` | Need the complete M section document for a new table |
| `templates/m-existing-table-destination.m` | Need the M section document for an existing table |
| `templates/m-multi-source-section.m` | Need the M section document for multi-source with AllowCombine |
| `templates/m-sharepoint-excel-source.m` | Need the M snippet for SharePoint Excel via Web.Contents |
| `templates/pipeline-single-dataflow.json` | Need pipeline JSON for a single Dataflow activity |
| `templates/pipeline-chained-dataflows.json` | Need pipeline JSON for chained Dataflow activities |