# Large File Range Reading Support (v4)

## Objective
Implement support for reading extremely large text files by adding range parameters (start_byte and end_byte) to the file read tool, allowing users to read specific portions of large files without loading the entire file into memory. Binary files should not be supported and UTF-8 character boundaries must always be respected.

## Implementation Plan

1. **Update `FsReadService` interface to support range reading**
   - Dependencies: None
   - Files: 
     - `crates/forge_services/src/infra.rs`
   - Notes: Add a new method to the trait that accepts start and end positions
   - Status: Not Started

2. **Implement the range reading functionality in `ForgeFileReadService`**
   - Dependencies: Task 1
   - Files: 
     - `crates/forge_infra/src/fs_read.rs`
   - Notes: Implement the new trait method using Tokio's file API for efficient reading
   - Status: Not Started

3. **Add binary file detection using the infer crate**
   - Dependencies: None
   - Files:
     - `crates/forge_fs/src/lib.rs`
     - `crates/forge_fs/Cargo.toml`
   - Notes: Add the infer crate as a dependency and implement a utility function to detect if a file is binary and return an appropriate error
   - Status: Not Started

4. **Update ForgeFS to support range reading with binary file validation**
   - Dependencies: Tasks 2, 3
   - Files: 
     - `crates/forge_fs/src/lib.rs`
   - Notes: Add a new method for range-based file reading that rejects binary files
   - Status: Not Started

5. **Implement UTF-8 boundary detection and correction**
   - Dependencies: Tasks 2, 4
   - Files:
     - `crates/forge_fs/src/lib.rs`
   - Notes: Ensure that range reads always align with UTF-8 character boundaries by adjusting the actual read range
   - Status: Not Started

6. **Update the `FSReadInput` struct to include optional range parameters**
   - Dependencies: None
   - Files: 
     - `crates/forge_services/src/tools/fs/fs_read.rs`
   - Notes: Add optional start_byte and end_byte fields to the input struct
   - Status: Not Started

7. **Modify FSRead tool implementation to support range reading and reject binary files**
   - Dependencies: Tasks 1, 2, 3, 4, 5, 6
   - Files: 
     - `crates/forge_services/src/tools/fs/fs_read.rs`
   - Notes: Update the `call` method to use the range-based reading with UTF-8 boundary adjustment and ensure binary files are rejected
   - Status: Not Started

8. **Update the FSRead tool description**
   - Dependencies: Task 6
   - Files: 
     - `crates/forge_services/src/tools/fs/fs_read.rs`
   - Notes: Update docstring to include range parameters in the tool description and explicitly mention that binary files are not supported and UTF-8 boundaries are always respected

      Sample Response:
      ```
      ---
      path: /a/b/c.txt
      range: 100-200
      total: 1024
      ---
      Hello! This is the contents of file c.txt
      ```
   - Status: Not Started

9. **Implement file size detection logic**
   - Dependencies: None
   - Files:
     - `crates/forge_fs/src/lib.rs`
   - Notes: Add functionality to efficiently determine file size without reading the entire file
   - Status: Not Started

10. **Add content length information to range read responses**
    - Dependencies: Task 9
    - Files:
      - `crates/forge_services/src/tools/fs/fs_read.rs`
    - Notes: Include total file size and adjusted range information in the response to help users understand the context of the range
    - Status: Not Started

11. **Add unit tests for range-based file reading and binary file rejection**
    - Dependencies: Tasks 1-10
    - Files: 
      - `crates/forge_services/src/tools/fs/fs_read.rs`
      - `crates/forge_infra/src/fs_read.rs`
      - `crates/forge_fs/src/lib.rs`
    - Notes: Test different range scenarios, edge cases, binary file detection with infer, UTF-8 boundary handling, and error conditions
    - Status: Not Started

## Verification Criteria
- The file read tool correctly returns only the requested range of bytes from large text files
- The tool properly identifies and rejects binary files with a clear error message using the infer crate
- The tool always adjusts range boundaries to respect UTF-8 character boundaries
- The tool handles edge cases properly:
  - When start_byte is beyond the file size
  - When end_byte is beyond the file size
  - When start_byte is greater than end_byte
  - When start_byte and end_byte are equal
  - When start_byte is negative or otherwise invalid
  - When reading from an empty file
  - When the range spans across malformed UTF-8 sequences
  - When the file is locked by another process
  - When reading from special files (e.g., device files, named pipes)
  - When hitting OS-specific file size limits
- The tool returns the entire file when no range is specified (backward compatibility)
- The tool provides helpful error messages for invalid range parameters
- The response includes information about the actual range read after UTF-8 boundary adjustment
- File size information is correctly included in the response
- Performance remains acceptable for both small and extremely large text files
- All unit tests pass
- Clippy runs with no errors or warnings

## Potential Risks and Mitigations

1. **Performance issues with extremely large text files**  
   Mitigation: 
   - Ensure that the implementation doesn't read the entire file when a range is specified
   - Use Tokio's file operations that support seeking and partial reads
   - Verify with benchmarks on files of various sizes (MB to GB)
   - Consider implementing a buffered reading strategy for large ranges

2. **UTF-8 boundary adjustment overhead**  
   Mitigation: 
   - Optimize the UTF-8 boundary detection algorithm for performance
   - Implement caching for boundary positions when repeated reads are requested
   - Use efficient byte scanning techniques that minimize CPU and memory usage
   - Provide clear metadata about the boundary adjustments that were made

3. **Breaking changes to the existing API**  
   Mitigation: 
   - Make the range parameters optional with default values that maintain backward compatibility
   - Document the behavior changes thoroughly
   - Ensure all existing tests continue to pass with the new implementation

4. **Inaccurate binary file detection**  
   Mitigation: 
   - Use the infer crate which provides robust file type detection
   - Still implement fallback checks for edge cases the infer crate might miss
   - Add comprehensive tests with various file types to verify correct detection
   - Provide clear error messages when a file is detected as binary

5. **File locking and concurrent access issues**  
   Mitigation:
   - Implement proper error handling for locked files
   - Use non-exclusive file handles when possible
   - Add retry logic with exponential backoff for temporary access issues

6. **Memory consumption with large ranges**  
   Mitigation:
   - Implement chunk-based reading for very large ranges
   - Set reasonable defaults and maximum values for range sizes
   - Add memory usage monitoring and provide warnings for potentially problematic operations

7. **Platform-specific issues**  
   Mitigation:
   - Test on all supported platforms (Windows, macOS, Linux)
   - Handle platform-specific file path conventions
   - Respect platform-specific file size limitations

8. **Invalid UTF-8 sequences in text files**  
   Mitigation:
   - Implement robust error handling for malformed UTF-8
   - Provide clear error messages when invalid UTF-8 is encountered
   - Consider options for replacement or reporting of invalid sequences

9. **Dependency management issues with infer crate**  
   Mitigation:
   - Pin to a specific version of the infer crate to avoid breaking changes
   - Monitor for security updates and issues with the infer crate
   - Have a fallback mechanism in case the infer crate fails

10. **Confusion for users with the new metadata in responses**  
    Mitigation:
    - Provide clear documentation on how to interpret the metadata
    - Include examples of how to use the new range parameters
    - Ensure backward compatibility so users not using ranges aren't affected

## Alternative Approaches

1. **Streaming API**: Implement a streaming interface for file reading instead of range-based reading. This would allow progressive loading of large files but would require more significant changes to the tool interface.

2. **File Pagination Tool**: Create a separate tool specifically for paginated file reading, leaving the original file read tool unchanged. This would maintain perfect backward compatibility but introduce redundancy.

3. **Content-Based Partitioning**: Implement intelligent partitioning based on content (e.g., by line, by paragraph, by JSON object) rather than raw bytes. This would be more semantic but more complex to implement.

4. **Fixed-size chunking**: Instead of arbitrary byte ranges, implement a chunking system where files are divided into fixed-size chunks that can be requested by index. This would simplify the API but reduce flexibility.

5. **Smart text-only file reading**: Implement a detection mechanism that automatically determines the optimal portion of a text file to return based on the context of the request, using language-aware boundaries like paragraphs or code blocks.

6. **Custom binary detection instead of infer**: Implement our own binary detection logic instead of relying on an external crate. This would reduce dependencies but require more maintenance and could be less accurate.

## Implementation Details

### Range Parameter Design

For the FSReadInput struct, add the following optional parameters:

```rust
/// Optional start position in bytes (0-based)
pub start_byte: Option<u64>,

/// Optional end position in bytes (exclusive)
pub end_byte: Option<u64>,
```

### Binary File Detection

To detect binary files, we'll use the `infer` crate:

1. Add the infer crate to dependencies in Cargo.toml:
   ```toml
   [dependencies]
   infer = "0.15.0"  # Use the latest version
   ```

2. Implement a utility function that:
   - Reads a small sample of the file (e.g., first 8KB)
   - Uses `infer::is_image()`, `infer::is_video()`, `infer::is_audio()`, `infer::is_archive()`, etc. to detect binary formats
   - For files not detected by infer, falls back to checking for null bytes or high concentration of non-printable characters
   - Returns a boolean indicating whether the file is likely binary

3. When a file is detected as binary, return an error message like:
   "Binary files are not supported. File detected as [file type]. Please use another tool or method to process this file."

### UTF-8 Boundary Detection and Adjustment

To ensure range reads respect UTF-8 character boundaries:

1. For the start position:
   - If the byte at start_byte is a UTF-8 continuation byte (10xxxxxx), scan backward to find the leading byte
   - Adjust start_byte to the position of the leading byte

2. For the end position:
   - If the byte at end_byte-1 is a leading byte of a multi-byte sequence, check if the complete character is included
   - If not, scan forward to include the complete character or backward to exclude the partial character

3. Report the adjusted positions in the response metadata

### Response Format

The response will include:

- The requested file content (as text)
- Metadata about the read operation:
  - Total file size
  - Original requested range
  - Actual range read after UTF-8 boundary adjustment
  - Information about boundary adjustments made

### Sample Response

The fs_read tool will return JSON with the following structure:

```json
{
  "content": "This is the file content within the requested range...",
  "metadata": {
    "file_size": 1024000,
    "requested_range": {
      "start_byte": 500,
      "end_byte": 1500
    },
    "actual_range": {
      "start_byte": 498,
      "end_byte": 1503
    },
    "boundary_adjustments": {
      "start_adjusted": true,
      "start_adjustment_reason": "UTF-8 character boundary alignment",
      "end_adjusted": true,
      "end_adjustment_reason": "UTF-8 character boundary alignment"
    },
    "is_partial": true,
    "percent_of_file": 0.1
  }
}
```

For error cases:

```json
{
  "error": "Binary files are not supported. File detected as image/png. Please use another tool or method to process this file."
}
```

Or for invalid ranges:

```json
{
  "error": "Invalid range specified: start_byte (5000) is greater than end_byte (4000)."
}
```

### Efficient Implementation Approach

To minimize memory usage and improve performance:

1. Use `tokio::fs::File::open()` to get a file handle
2. Read a small sample and perform binary file detection using the infer crate
3. Use `file.metadata()` to get the file size without reading content
4. Validate range parameters against file size
5. Use `file.seek()` to position near start_byte
6. Perform UTF-8 boundary detection and adjust start position if needed
7. Use `file.take(adjusted_end_byte - adjusted_start_byte)` to create a limited reader
8. Read from the limited reader into a buffer
9. Verify that the buffer contains valid UTF-8 and make any final adjustments
10. Return the buffer content with detailed metadata