# Migration Guide: v1.x to v2.0

This guide covers migrating RAGStack deployments from v1.x (deployed after December 24, 2025) to v2.0.

## Breaking Changes in v2.0

### Architecture Changes

| Component | v1.x | v2.0 |
|-----------|------|------|
| **S3 Prefixes** | `output/` (docs), `images/` (images) | `content/` (unified) |
| **Data Sources** | 2 (TextDataSource + ImageDataSource) | 1 (DataSource) |
| **Env Vars** | `TEXT_DATA_SOURCE_ID`, `IMAGE_DATA_SOURCE_ID` | `DATA_SOURCE_ID` |
| **Image Metadata** | `IN_LINE_ATTRIBUTE` | `S3_LOCATION` (.metadata.json files) |
| **CloudFormation Outputs** | `DataSourceId`, `TextDataSourceId`, `ImageDataSourceId` | `DataSourceId` only |

### Why Migrate?

- **Simplified architecture**: Single data source reduces complexity
- **Unified content handling**: All content types use consistent metadata format
- **Improved metadata extraction**: New LLM-based metadata extraction for all content
- **Better filtering**: `content_type` field enables filtering by document/image/web_page

## Prerequisites

- Python 3.13+ with `boto3` installed
- AWS CLI configured with appropriate permissions
- Access to the deployed stack

## Migration Process

### Step 1: Run the Migration Script (Dry Run)

First, preview what changes will be made:

```bash
python scripts/migrate_v1_to_v2.py --stack-name <your-stack-name> --dry-run
```

This will show:
- Files that will be copied from `output/` and `images/` to `content/`
- DynamoDB tracking records that will be updated

### Step 2: Run the Actual Migration

Once satisfied with the dry run output:

```bash
python scripts/migrate_v1_to_v2.py --stack-name <your-stack-name>
```

The script will:
1. Copy all files from `output/` to `content/`
2. Copy all files from `images/` to `content/`
3. Update tracking table records with new S3 URIs

**Note:** The script is idempotent - it skips files that already exist in `content/` and records that are already updated.

### Step 3: Deploy v2.0 Stack

Pull the latest code and deploy:

```bash
git pull origin main
sam build
sam deploy --stack-name <your-stack-name>
```

This updates:
- Lambda functions with new single data source logic
- Knowledge Base custom resource to create unified data source
- EventBridge rules to watch `content/` prefix

### Step 4: Trigger Reindex

Open the RAGStack dashboard and navigate to **Settings**:

1. Scroll to the **Knowledge Base Reindex** section
2. Click **Start Reindex**
3. Wait for the reindex to complete

The reindex will:
- Create a new Knowledge Base with the unified `content/` data source
- Re-extract metadata for all content using the new extraction system
- Ingest all documents, images, and scraped pages with fresh embeddings
- Delete the old Knowledge Base

### Step 5: Verify Migration

After reindex completes:

1. **Test chat**: Query the Knowledge Base and verify responses
2. **Check sources**: Ensure source attribution shows correct paths
3. **Test image search**: Verify image captions are searchable
4. **Test filters**: Use content_type filter to search specific content types

## Migration Script Details

### What the Script Does

```text
migrate_v1_to_v2.py
├── Get stack outputs (bucket name, table name)
├── Step 1: Copy output/* → content/*
├── Step 2: Copy images/* → content/*
└── Step 3: Update DynamoDB tracking records
    ├── output_s3_uri: output/ → content/
    ├── input_s3_uri: images/ → content/ (for images)
    └── caption_s3_uri: images/ → content/ (for images)
```

### What the Script Does NOT Do

- Does NOT delete old files (output/, images/ remain intact)
- Does NOT modify the Knowledge Base (reindex handles this)
- Does NOT generate metadata files (reindex handles this)

### Options

```bash
python scripts/migrate_v1_to_v2.py --help

Options:
  --stack-name   CloudFormation stack name (required)
  --region       AWS region (default: us-east-1)
  --dry-run      Preview changes without making them
  --verbose, -v  Enable debug logging
```

## Reindex Details

The reindex process handles all content types with type-specific logic:

| Type | Text Source | Metadata Extraction | Ingestion |
|------|-------------|---------------------|-----------|
| Documents | `output_s3_uri` | LLM extracts from text | 1 document |
| Images | `caption_s3_uri` | LLM extracts from caption | 2 documents (image + caption) |
| Scraped | `output_s3_uri` | Job-aware (see below) | 1 document |

### Job-Aware Scraped Content Reindex

Scraped content uses a special two-level metadata extraction:

1. **Job-level metadata**: Extracted from the **seed document** (first page scraped)
   - Applied to ALL pages in the scrape job
   - Provides semantic context (e.g., "AWS Lambda documentation")

2. **Page-level metadata**: Deterministic fields for each page
   - `source_url`, `source_domain`, `scraped_date`, `job_id`

**How it works:**
```text
Scraped Page → S3 metadata → job_id
                               ↓
                          ScrapeJobs table → base_url
                               ↓
                          Find seed document (source_url == base_url)
                               ↓
                          Re-extract job metadata from seed (using NEW settings)
                               ↓
                          Merge job metadata + page metadata
```

This ensures:
- All pages in a job share semantic metadata from the seed
- Metadata uses the NEW extraction settings (not preserved from original scrape)
- Job metadata is cached per-batch to avoid redundant LLM calls

### Common Metadata

All content gets:
- Fresh metadata extraction using configured LLM model
- `content_type` field for filtering ("document", "image", "web_page")
- Base metadata (document_id, filename, file_type)

## Rollback

If issues occur after migration:

1. **Before stack deploy**: Old files still exist in `output/` and `images/` - no rollback needed
2. **After stack deploy but before reindex**: Re-deploy old code version
3. **After reindex**: The old KB is deleted; you'd need to re-upload content

## Troubleshooting

### Migration Script Errors

**"Stack not found"**
- Verify stack name is correct
- Check you're using the right AWS region

**"Access Denied" errors**
- Ensure AWS credentials have S3 read/write and DynamoDB permissions
- Check the IAM user/role has access to the stack's resources

### Reindex Errors

**"Failed to read text"**
- Check the `output_s3_uri` or `caption_s3_uri` paths are correct
- Verify files were copied to `content/` prefix

**"Metadata extraction failed"**
- Check Bedrock model access (ensure your region supports the configured model)
- Review CloudWatch logs for the ReindexKB Lambda

### Post-Migration Issues

**Chat returns no results**
- Wait for reindex to fully complete
- Check Knowledge Base status in AWS console
- Verify data source has correct `content/` prefix

**Images not searchable**
- Ensure caption files exist at `content/{imageId}/caption.txt`
- Check the image's tracking record has `caption_s3_uri` field

### SAM Layer Caching Issue

**Symptoms:**
- Lambda functions fail with `No module named 'ragstack_common'` or `No module named 'crhelper'`
- CloudFormation update gets stuck on `CodeBuildRun` or `WCCodeBuildRun` custom resources
- Stack enters `UPDATE_ROLLBACK_FAILED` state

**Cause:**
SAM uses content hashing to skip S3 uploads. After a reindex creates a new Knowledge Base, if you redeploy, SAM may reuse a stale/corrupted layer artifact from S3 instead of uploading the freshly built layer. The local build shows ~121MB but S3 only has ~120KB.

**Diagnosis:**
```bash
# Check local build size (should be ~121MB)
du -sh .aws-sam/build/RagstackCommonLayer/

# Check deployed layer size (should match, not 120KB)
aws lambda get-function-configuration --function-name <stack>-sync-status-checker \
  --query "Layers[0].CodeSize" --output text
```

**Fix:**

1. If stack is stuck in `UPDATE_IN_PROGRESS`, cancel and wait for rollback:
   ```bash
   aws cloudformation cancel-update-stack --stack-name <stack> --region us-east-1
   ```

2. If stack is in `UPDATE_ROLLBACK_FAILED`, continue rollback skipping failed resources:
   ```bash
   aws cloudformation continue-update-rollback --stack-name <stack> --region us-east-1 \
     --resources-to-skip CodeBuildRun WCCodeBuildRun BatchProcessorFunction \
       AppSyncResolverFunction ConfigurationResolverFunction
   ```

3. Once stack is in `UPDATE_ROLLBACK_COMPLETE`, clear caches and redeploy:
   ```bash
   # Delete SAM build cache
   rm -rf .aws-sam/

   # Delete stale S3 artifacts (keep UI source zips)
   aws s3 rm s3://<stack>-artifacts-<account-id>/ --recursive --exclude "*.zip"

   # Fresh build and deploy
   sam build --parallel
   python publish.py --stack-name <stack> --admin-email <email>
   ```

**Prevention:**
After running reindex, always clear the SAM cache before redeploying:
```bash
rm -rf .aws-sam/
```

## Support

For issues:
- Check [TROUBLESHOOTING.md](./TROUBLESHOOTING.md)