# Session 4: Building Production Chat Applications with Chainlit

## Overview

This session focuses on building production-ready chat applications using Chainlit and Microsoft Foundry Local. You'll learn to create modern web interfaces for AI conversations, implement streaming responses, and deploy robust chat applications with proper error handling and user experience design.

**What You'll Build:**
- **Chainlit Chat App**: Modern web UI with streaming responses
- **WebGPU Demo**: Browser-based inference for privacy-first applications  
- **Open WebUI Integration**: Professional chat interface with Foundry Local
- **Production Patterns**: Error handling, monitoring, and deployment strategies

## Learning Objectives

- Build production-ready chat applications with Chainlit
- Implement streaming responses for enhanced user experience
- Master Foundry Local SDK integration patterns
- Apply proper error handling and graceful degradation
- Deploy and configure chat applications for different environments
- Understand modern web UI patterns for conversational AI

## Prerequisites

- **Foundry Local**: Installed and running ([Installation Guide](https://learn.microsoft.com/azure/ai-foundry/foundry-local/))
- **Python**: 3.10 or later with virtual environment capability
- **Model**: At least one model loaded (`foundry model run phi-4-mini`)
- **Browser**: Modern web browser with WebGPU support (Chrome/Edge)
- **Docker**: For Open WebUI integration (optional)

## Part 1: Understanding Modern Chat Applications

### Architecture Overview

```
User Browser ←→ Chainlit UI ←→ Python Backend ←→ Foundry Local ←→ AI Model
      ↓              ↓              ↓              ↓            ↓
   Web UI      Event Handlers   OpenAI Client   HTTP API    Local GPU
```

### Key Technologies

**Foundry Local SDK Patterns:**
- `FoundryLocalManager(alias)`: Automatic service management
- `manager.endpoint` and `manager.api_key`: Connection details
- `manager.get_model_info(alias).id`: Model identification

**Chainlit Framework:**
- `@cl.on_chat_start`: Initialize chat sessions
- `@cl.on_message`: Handle incoming user messages  
- `cl.Message().stream_token()`: Real-time streaming
- Automatic UI generation and WebSocket management

## Part 2: Local vs Cloud Decision Matrix

### Performance Characteristics

| Aspect | Local (Foundry) | Cloud (Azure OpenAI) |
|--------|-----------------|---------------------|
| **Latency** | 🚀 50-200ms (no network) | ⏱️ 200-2000ms (network dependent) |
| **Privacy** | 🔒 Data never leaves device | ⚠️ Data sent to cloud |
| **Cost** | 💰 Free after hardware | 💸 Pay per token |
| **Offline** | ✅ Works without internet | ❌ Requires internet |
| **Model Size** | ⚠️ Limited by hardware | ✅ Access to largest models |
| **Scaling** | ⚠️ Hardware dependent | ✅ Unlimited scaling |

### Hybrid Strategy Patterns

**Local-First with Fallback:**
```python
async def hybrid_completion(prompt: str, complexity_threshold: int = 100):
    if len(prompt.split()) < complexity_threshold:
        return await local_completion(prompt)  # Fast, private
    else:
        return await cloud_completion(prompt)   # Complex reasoning
```

**Task-Based Routing:**
```python
async def smart_routing(prompt: str, task_type: str):
    routing_rules = {
        "code_generation": "local",     # Privacy-sensitive
        "creative_writing": "cloud",    # Benefits from larger models
        "data_analysis": "local",       # Fast iteration needed
        "research": "cloud"             # Requires broad knowledge
    }
    
    if routing_rules.get(task_type) == "local":
        return await foundry_completion(prompt)
    else:
        return await azure_completion(prompt)
```

## Part 3: Sample 04 - Chainlit Chat Application

### Quick Start

```cmd
# Navigate to Module08 directory  
cd Module08

# Start your preferred model
foundry model run phi-4-mini

# Run the Chainlit application (avoiding port conflicts)
chainlit run samples\04\app.py -w --port 8080
```

The application automatically opens at `http://localhost:8080` with a modern chat interface.

### Core Implementation

The Sample 04 application demonstrates production-ready patterns:

**Automatic Service Discovery:**
```python
import chainlit as cl
from openai import OpenAI
from foundry_local import FoundryLocalManager

# Global variables for client and model
client = None
model_name = None

async def initialize_client():
    global client, model_name
    alias = os.environ.get("MODEL", "phi-4-mini")
    
    try:
        # Use FoundryLocalManager for proper service management
        manager = FoundryLocalManager(alias)
        model_info = manager.get_model_info(alias)
        
        client = OpenAI(
            base_url=manager.endpoint,
            api_key=manager.api_key or "not-required"
        )
        model_name = model_info.id if model_info else alias
        return True
    except Exception as e:
        # Fallback to manual configuration
        base_url = os.environ.get("BASE_URL", "http://localhost:51211")
        client = OpenAI(base_url=f"{base_url}/v1", api_key="not-required")
        model_name = alias
        return True
```

**Streaming Chat Handler:**
```python
@cl.on_message
async def main(message: cl.Message):
    # Create streaming response
    msg = cl.Message(content="")
    await msg.send()
    
    stream = client.chat.completions.create(
        model=model_name,
        messages=[
            {"role": "system", "content": "You are a helpful AI assistant."},
            {"role": "user", "content": message.content}
        ],
        stream=True
    )
    
    # Stream tokens in real-time
    for chunk in stream:
        if chunk.choices[0].delta.content:
            await msg.stream_token(chunk.choices[0].delta.content)
    
    await msg.update()
```

### Configuration Options

**Environment Variables:**

| Variable | Description | Default | Example |
|----------|-------------|---------|----------|
| `MODEL` | Model alias to use | `phi-4-mini` | `qwen2.5-7b` |
| `BASE_URL` | Foundry Local endpoint | Auto-detected | `http://localhost:51211` |
| `API_KEY` | API key (optional for local) | `""` | `your-api-key` |

**Advanced Usage:**
```cmd
# Use different model
set MODEL=qwen2.5-7b
chainlit run samples\04\app.py -w --port 8080

# Use different ports (avoid 51211 which is used by Foundry Local)
chainlit run samples\04\app.py -w --port 3000
chainlit run samples\04\app.py -w --port 5000
```

## Part 4: Creating and Using Jupyter Notebooks

### Overview of Notebook Support

The Sample 04 includes a comprehensive Jupyter notebook (`chainlit_app.ipynb`) that provides:

- **📚 Educational Content**: Step-by-step learning materials
- **🔬 Interactive Exploration**: Run and experiment with code cells
- **📊 Visual Demonstrations**: Charts, diagrams, and output visualization
- **🛠️ Development Tools**: Testing and debugging capabilities

### Creating Your Own Notebooks

#### Step 1: Set Up Jupyter Environment

```cmd
# Ensure you're in the Module08 directory
cd Module08

# Activate your virtual environment
.venv\Scripts\activate

# Install Jupyter and dependencies
pip install jupyter notebook jupyterlab ipykernel
pip install -r requirements.txt

# Register the kernel for VS Code
python -m ipykernel install --user --name=foundry-local --display-name="Foundry Local"
```

#### Step 2: Create a New Notebook

**Using VS Code:**
1. Open VS Code in the Module08 directory
2. Create a new file with `.ipynb` extension
3. Select the "Foundry Local" kernel when prompted
4. Start adding cells with your content

**Using Jupyter Lab:**
```cmd
# Start Jupyter Lab
jupyter lab

# Navigate to samples/04/ and create new notebook
# Choose Python 3 kernel
```

### Notebook Structure Best Practices

#### Cell Organization

```python
# Cell 1: Imports and Setup
import os
import sys
import chainlit as cl
from openai import OpenAI
from foundry_local import FoundryLocalManager

print("✅ Libraries imported successfully")
```

```python
# Cell 2: Configuration and Client Setup
class FoundryClientManager:
    def __init__(self, model_name="phi-4-mini"):
        self.model_name = model_name
        self.client = None
        
    def initialize_client(self):
        # Client initialization logic
        pass

# Initialize and test
client_manager = FoundryClientManager()
result = client_manager.initialize_client()
print(f"Client initialized: {result}")
```

### Interactive Examples and Exercises

#### Exercise 1: Client Configuration Testing

```python
# Test different configuration methods
configurations = [
    {"method": "foundry_sdk", "model": "phi-4-mini"},
    {"method": "manual", "base_url": "http://localhost:51211", "model": "qwen2.5-7b"},
]

for config in configurations:
    print(f"\n🧪 Testing {config['method']} configuration...")
    # Implementation here
    result = test_configuration(config)
    print(f"Result: {'✅ Success' if result['status'] == 'ok' else '❌ Failed'}")
```

#### Exercise 2: Streaming Response Simulation

```python
import asyncio

async def simulate_streaming_response(text, delay=0.1):
    """Simulate how streaming works in Chainlit."""
    print("🌊 Simulating streaming response...")
    
    for char in text:
        print(char, end='', flush=True)
        await asyncio.sleep(delay)
    
    print("\n✅ Streaming complete!")

# Test the simulation
sample_text = "This is how streaming responses work in Chainlit applications!"
await simulate_streaming_response(sample_text)
```

## Part 5: WebGPU Browser Inference Demo

### Overview

WebGPU enables running AI models directly in the browser for maximum privacy and zero-install experiences. This sample demonstrates ONNX Runtime Web with WebGPU execution.

### Step 1: Check WebGPU Support

**Browser Requirements:**
- Chrome/Edge 113+ with WebGPU enabled
- Check: `chrome://gpu` → confirm "WebGPU" status
- Programmatic check: `if (!('gpu' in navigator)) { /* no WebGPU */ }`

### Step 2: Create WebGPU Demo

Create directory: `samples/04/webgpu-demo/`

**index.html:**
```html
<!doctype html>
<html>
<head>
    <meta charset="utf-8">
    <title>WebGPU + ONNX Runtime Demo</title>
    <script src="https://cdn.jsdelivr.net/npm/onnxruntime-web/dist/ort.webgpu.min.js"></script>
    <style>
        body { font-family: system-ui, sans-serif; margin: 2rem; }
        pre { background: #f5f5f5; padding: 1rem; overflow: auto; }
        .status { padding: 1rem; background: #e3f2fd; border-radius: 4px; }
    </style>
</head>
<body>
    <h1>🚀 WebGPU + Foundry Local Integration</h1>
    <div id="status" class="status">Initializing...</div>
    <pre id="output"></pre>
    <script type="module" src="./main.js"></script>
</body>
</html>
```

**main.js:**
```javascript
const statusEl = document.getElementById('status');
const outputEl = document.getElementById('output');

function log(msg) {
    outputEl.textContent += `${msg}\n`;
    console.log(msg);
}

(async () => {
    try {
        if (!('gpu' in navigator)) {
            statusEl.textContent = '❌ WebGPU not available';
            return;
        }
        
        statusEl.textContent = '🔍 WebGPU detected. Loading model...';
        
        // Use a small ONNX model for demo
        const modelUrl = 'https://huggingface.co/onnx/models/resolve/main/vision/classification/mnist-12/mnist-12.onnx';
        
        const session = await ort.InferenceSession.create(modelUrl, {
            executionProviders: ['webgpu']
        });
        
        log('✅ ONNX Runtime session created with WebGPU');
        log(`📊 Input names: ${session.inputNames.join(', ')}`);
        log(`📊 Output names: ${session.outputNames.join(', ')}`);
        
        // Create dummy input (MNIST expects 1x1x28x28)
        const inputData = new Float32Array(1 * 1 * 28 * 28).fill(0.1);
        const input = new ort.Tensor('float32', inputData, [1, 1, 28, 28]);
        
        const feeds = {};
        feeds[session.inputNames[0]] = input;
        
        const results = await session.run(feeds);
        const output = results[session.outputNames[0]];
        
        // Find prediction (argmax)
        let maxIdx = 0;
        for (let i = 1; i < output.data.length; i++) {
            if (output.data[i] > output.data[maxIdx]) maxIdx = i;
        }
        
        statusEl.textContent = '✅ WebGPU inference complete!';
        log(`🎯 Predicted class: ${maxIdx}`);
        log(`📈 Confidence scores: [${Array.from(output.data).map(x => x.toFixed(3)).join(', ')}]`);
        
    } catch (error) {
        statusEl.textContent = `❌ Error: ${error.message}`;
        log(`Error: ${error.message}`);
        console.error(error);
    }
})();
```

### Step 3: Run the Demo

```cmd
# Create demo directory
mkdir samples\04\webgpu-demo
cd samples\04\webgpu-demo

# Save HTML and JS files, then serve
python -m http.server 5173

# Open browser to http://localhost:5173
```

## Part 6: Open WebUI Integration

### Overview

Open WebUI provides a professional ChatGPT-like interface that connects to Foundry Local's OpenAI-compatible API.

### Step 1: Prerequisites

```cmd
# Verify Foundry Local is running
foundry service status

# Start a model
foundry model run phi-4-mini

# Confirm API endpoint is accessible
curl http://localhost:51211/v1/models
```

### Step 2: Docker Setup (Recommended)

```cmd
# Pull Open WebUI image
docker pull ghcr.io/open-webui/open-webui:main

# Run with Foundry Local connection
docker run -d --name open-webui -p 3000:8080 ^
  -e OPENAI_API_BASE_URL=http://host.docker.internal:51211/v1 ^
  -e OPENAI_API_KEY=foundry-local-key ^
  -v open-webui-data:/app/backend/data ^
  ghcr.io/open-webui/open-webui:main
```

**Note:** `host.docker.internal` allows Docker containers to access the host machine on Windows.

### Step 3: Configuration

1. **Open Browser:** Navigate to `http://localhost:3000`
2. **Initial Setup:** Create admin account
3. **Model Configuration:**
   - Settings → Models → OpenAI API  
   - Base URL: `http://host.docker.internal:51211/v1`
   - API Key: `foundry-local-key` (any value works)
4. **Test Connection:** Models should appear in dropdown

### Troubleshooting

**Common Issues:**

1. **Connection Refused:**
   ```cmd
   # Check Foundry Local status
   foundry service ps
   netstat -ano | findstr :51211
   ```

2. **Models Not Appearing:**
   - Verify model is loaded: `foundry model list`
   - Check API response: `curl http://localhost:51211/v1/models`
   - Restart Open WebUI container

## Part 7: Production Deployment Considerations

### Environment Configuration

**Development Setup:**
```cmd
# Development with auto-reload and debugging
chainlit run samples\04\app.py -w --port 8080 --debug
```

**Production Deployment:**
```cmd
# Production mode with optimizations
chainlit run samples\04\app.py --host 0.0.0.0 --port 8080 --no-cache
```

### Common Port Issues and Solutions

**Port 51211 Conflict Prevention:**
```cmd
# Check what's using Foundry Local port
netstat -ano | findstr :51211

# Use different port for Chainlit
chainlit run samples\04\app.py -w --port 8080
```

### Performance Monitoring

**Health Check Implementation:**
```python
@cl.on_chat_start
async def health_check():
    try:
        # Test model availability
        response = client.chat.completions.create(
            model=model_name,
            messages=[{"role": "user", "content": "test"}],
            max_tokens=1
        )
        return {"status": "healthy", "model": model_name}
    except Exception as e:
        return {"status": "unhealthy", "error": str(e)}
```

## Summary

Session 4 covered building production-ready Chainlit applications for conversational AI. You learned about:

- ✅ **Chainlit Framework**: Modern UI and streaming support for chat applications
- ✅ **Foundry Local Integration**: SDK usage and configuration patterns  
- ✅ **WebGPU Inference**: Browser-based AI for maximum privacy
- ✅ **Open WebUI Setup**: Professional chat interface deployment
- ✅ **Production Patterns**: Error handling, monitoring, and scaling

The Sample 04 application demonstrates best practices for building robust chat interfaces that leverage local AI models through Microsoft Foundry Local while providing excellent user experiences.

## References

- **[Sample 04: Chainlit Application](samples/04/README.md)**: Complete application with documentation
- **[Chainlit Educational Notebook](samples/04/chainlit_app.ipynb)**: Interactive learning materials
- **[Foundry Local Documentation](https://learn.microsoft.com/azure/ai-foundry/foundry-local/)**: Complete platform documentation
- **[Chainlit Documentation](https://docs.chainlit.io/)**: Official framework documentation
- **[Open WebUI Integration Guide](https://github.com/microsoft/foundry-local/blob/main/docs/tutorials/chat-application-with-open-web-ui.md)**: Official tutorial