# Training Types: SFT vs DPO vs RFT

## Decision Matrix

| Factor | SFT | DPO | RFT |
|--------|-----|-----|-----|
| **Best for** | Teaching a new skill or format | Aligning preferences/style | Improving reasoning chains |
| **Data needed** | Input–output pairs | Chosen/rejected pairs | Prompts + grading function |
| **Data volume** | 50–5,000 examples | 500–5,000 pairs | 200–2,000 prompts |
| **Effort to prepare data** | Low | High (need contrasting pairs) | Medium (need grader, not outputs) |
| **Risk of regression** | Low | Medium | High (sensitive to grader quality) |
| **Typical improvement** | 5–30% on task metrics | Subtle style/safety shifts | 0–15% on reasoning tasks |
| **Supported models** | Most models | Select models | o4-mini |

## When to Use Each

### SFT (Supervised Fine-Tuning)
- You have high-quality input–output pairs
- Task is well-defined (code generation, classification, extraction, summarization)
- You want reliable, repeatable outputs in a specific format or style
- **Key insight**: 300–500 high-quality examples often outperforms 1,500+ lower-quality ones

### DPO (Direct Preference Optimization)
- You want to adjust tone, verbosity, safety, or style
- You have examples of "good" and "bad" outputs for the same input
- SFT already works but outputs need refinement
- DPO-specific params: `beta` (default 0.1), `l2_multiplier` (default 0.1)

### RFT (Reinforcement Fine-Tuning)
- Task has objectively verifiable answers (code execution, math, logic)
- You can write a programmatic or LLM-based grader
- You want to improve the model's reasoning, not just its outputs
- **Critical**: RFT is extremely sensitive to grader quality. Train–val gap should be ≤ 0.05.

## Choosing a Path

```
├─ Do you have labeled input–output pairs?
│  ├─ Yes → SFT
│  └─ No
│     ├─ Can you write a grading function? → RFT
│     └─ Can you rank "good" vs "bad" outputs? → DPO
│
After SFT:
├─ Results good enough? → Ship it
├─ Need style refinement? → DPO on top of SFT model
└─ Reasoning needs improvement? → RFT (if model supports it)
```

## Model Compatibility (Microsoft Foundry)

| Model | SFT | DPO | RFT | Vision FT |
|-------|-----|-----|-----|-----------|
| gpt-4.1 | ✅ | ✅ | ❌ | ✅ |
| gpt-4.1-mini | ✅ | ❌ | ❌ | ❌ |
| gpt-4.1-nano | ✅ | ❌ | ❌ | ❌ |
| gpt-4o (2024-08-06) | ✅ | ✅ | ❌ | ✅ |
| gpt-4o-mini | ✅ | ❌ | ❌ | ❌ |
| o4-mini | ❌ | ❌ | ✅ | ❌ |
| gpt-5 | ❌ | ❌ | ✅ ⚠️ | ❌ |
| gpt-oss-20b | ✅ | ❌ | ❌ | ❌ |
| Ministral-3B | ✅ | ❌ | ❌ | ❌ |
| Llama-3.3-70B | ✅ | ❌ | ❌ | ❌ |
| Qwen-3-32B | ✅ | ❌ | ❌ | ❌ |

DPO can be applied on top of an already SFT-fine-tuned model. Vision fine-tuning follows the same SFT workflow but with image data in messages.

> ⚠️ **Feature flags**: GPT-5 RFT and agentic RFT with tool calling require access requests. Contact your Microsoft account team or request access through the Microsoft Foundry portal. o4-mini RFT without tools is generally available.

*Check Microsoft Foundry docs for the latest model availability.*