# llm.codes 📖 - Transform Developer Documentation for AI Agents A high-performance web service that converts JavaScript-heavy documentation sites into clean, LLM-optimized Markdown. Built specifically to solve the problem of AI agents being unable to parse modern documentation sites that rely heavily on client-side rendering. **Why llm.codes exists**: Modern AI agents like Claude Code struggle with JavaScript-heavy documentation sites, particularly Apple's developer docs. This tool bridges that gap by converting dynamic content into clean, parseable Markdown that AI agents can actually use. 📖 **Read the full story**: [How llm.codes Transforms Developer Documentation for AI Agents](https://steipete.me/posts/llm-codes-transform-developer-docs) ![Web Documentation to Markdown Converter](https://img.shields.io/badge/Next.js-16-black?style=flat-square&logo=nextdotjs) ![Tailwind CSS](https://img.shields.io/badge/Tailwind_CSS-v4-38B2AC?style=flat-square&logo=tailwind-css) ![TypeScript](https://img.shields.io/badge/TypeScript-7-blue?style=flat-square&logo=typescript) ![License](https://img.shields.io/badge/License-MIT-green?style=flat-square) ## Technical Architecture ### Core Problem Solved Modern documentation sites (especially Apple's) use heavy JavaScript rendering that makes content invisible to AI agents. llm.codes solves this by: - Using Firecrawl's hosted browser, or opt-in self-hosted Playwright, to execute JavaScript and capture fully-rendered content - Converting dynamic HTML to clean, semantic Markdown - Removing noise (navigation, footers, duplicate content) that wastes AI context tokens - Providing parallel URL processing for efficient multi-page documentation crawling ### Key Features - **Parallel Processing**: Fetches up to 10 URLs concurrently using bounded promises - **Smart Caching**: Redis-backed 30-day cache reduces API calls and improves response times - **Content Filtering**: Multiple filtering strategies to remove: - Navigation elements and boilerplate - Platform availability strings (iOS 14.0+, etc.) - Duplicate content across pages - Empty sections and formatting artifacts - **Recursive Crawling**: Configurable depth-first crawling with intelligent link extraction - **Firecrawl Crawl Mode**: Optional crawl API path with server-side depth/limit enforcement and SSE progress - **Code-Only Output**: Extract fenced examples only and skip pages without code blocks - **Browser Notifications**: Web Notifications API integration for background processing alerts - **URL State Management**: Query parameter-based URL sharing for easy documentation links ## Live Demo 🚀 **Try it now at [llm.codes](https://llm.codes/)** Experience the tool instantly without any setup required. ## Quick Start ### Prerequisites - Node.js 24+ - pnpm 10+ - [Firecrawl API key](https://firecrawl.dev) for the default provider, or self-hosted Chromium for the Playwright provider ### Installation 1. Clone the repository: ```bash git clone https://github.com/amantus-ai/llm-codes.git cd llm-codes ``` 2. Install dependencies: ```bash pnpm install ``` 3. Create a `.env.local` file: ```bash cp .env.local.example .env.local ``` 4. Add your scrape provider configuration to `.env.local`: ```env # Default hosted provider SCRAPE_PROVIDER=firecrawl FIRECRAWL_API_KEY=your_api_key_here # Self-hosted alternative # SCRAPE_PROVIDER=playwright # Run once: pnpm exec playwright install chromium # Optional - Redis Cache (Recommended for production) UPSTASH_REDIS_REST_URL=https://your-redis-instance.upstash.io UPSTASH_REDIS_REST_TOKEN=your_redis_token_here # Optional - Cache Admin CACHE_ADMIN_KEY=your_secure_admin_key_here ``` 5. Run the development server: ```bash pnpm run dev ``` 6. Open [http://localhost:3000](http://localhost:3000) ## Deployment ### Deploy to Vercel The easiest way to deploy is using Vercel: [![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Famantus-ai%2Fllm-codes&env=FIRECRAWL_API_KEY&envDescription=Your%20Firecrawl%20API%20key&envLink=https%3A%2F%2Ffirecrawl.dev) 1. Click the button above 2. Create a new repository 3. Add your `FIRECRAWL_API_KEY` environment variable 4. Deploy! ### Manual Deployment 1. Push to your GitHub repository 2. Import project on [Vercel](https://vercel.com/new) 3. Add environment variables: - `SCRAPE_PROVIDER`: `firecrawl` by default, or `playwright` for self-hosted Node/Docker deployments - `FIRECRAWL_API_KEY`: Your Firecrawl API key (required when `SCRAPE_PROVIDER=firecrawl`) - `UPSTASH_REDIS_REST_URL`: Your Upstash Redis URL (optional) - `UPSTASH_REDIS_REST_TOKEN`: Your Upstash Redis token (optional) - `CACHE_ADMIN_KEY`: Admin key for cache endpoints (optional) 4. Deploy ## Usage 1. **Enter URL**: Paste any documentation URL - Most documentation sites are automatically supported through pattern matching - Click "Learn more" to see the supported URL patterns 2. **Configure Options** (click "Show Options"): - **Crawl Depth**: How deep to follow links (0 = main page only, max 5) - **Max URLs**: Maximum number of pages to process (1-2000, default 200) - **Filter URLs**: Remove hyperlinks from content (recommended for LLMs) - **Deduplicate Content**: Remove duplicate paragraphs to save tokens - **Filter Availability**: Remove platform availability strings (iOS 14.0+, etc.) - **Extract Code Blocks Only**: Include fenced examples only and skip pages without examples - **Use Deep Crawl Mode**: Use Firecrawl's crawl API with the selected depth and page limit. Turn this off when using `SCRAPE_PROVIDER=playwright`. 3. **Process**: Click "Process Documentation" and grant notification permissions if prompted 4. **Monitor Progress**: - Real-time progress bar shows completion percentage - Activity log displays detailed processing information - Browser notifications alert you when complete 5. **Download**: View statistics and download your clean Markdown file ## Supported Documentation Sites llm.codes uses intelligent pattern matching to support most documentation sites automatically. Rather than maintaining a list of thousands of individual sites, we use regex patterns to match common documentation URL structures. ### Pattern-Based Matching We support documentation sites that match these patterns: 1. **Documentation Subdomains** (`docs.*, developer.*, learn.*, etc.`) - Examples: `docs.python.org`, `developer.apple.com`, `learn.microsoft.com` - Pattern: Any subdomain like docs, developer, dev, learn, help, api, guide, wiki, or devcenter 2. **Documentation Paths** (`/docs, /guide, /learn, etc.`) - Examples: `angular.io/docs`, `redis.io/docs`, `react.dev/learn` - Pattern: URLs ending with paths like /docs, /documentation, /api-docs, /guides, /learn, /help, /stable, or /latest 3. **Programming Language Sites** (`*js.org, *lang.org, etc.`) - Examples: `vuejs.org`, `kotlinlang.org`, `ruby-doc.org` - Pattern: Domains ending with js, lang, py, or -doc followed by .org or .com 4. **GitHub Pages** (`*.github.io`) - Examples: Any GitHub Pages documentation site - Pattern: All subdomains of github.io ### Explicit Exceptions A small number of popular documentation sites don't follow standard patterns and are explicitly supported: - Swift Package Index (`swiftpackageindex.com`) - Flask (`flask.palletsprojects.com`) - Material-UI (`mui.com/material-ui`) - pip (`pip.pypa.io/en/stable`) - PHP (`www.php.net/docs.php`) - docs.rs (`docs.rs`) - Zulip API (`zulip.com/api`) - Nutrient API (`www.nutrient.io/api`) - Playdate SDK (`sdk.play.date`) - MusicKit JS (`js-cdn.music.apple.com/musickit`) - React Spring (`react-spring.io`) - Valtio (`valtio.pmnd.rs`) ### Adding New Sites Most documentation sites are automatically supported! If your site follows standard documentation URL patterns (like having `/docs` in the path or `docs.` as a subdomain), it should work without any changes. If you find a documentation site that isn't supported, please [open an issue](https://github.com/amantus-ai/llm-codes/issues) and we'll either adjust our patterns or add it as an exception. ## Configuration Options | Option | Description | Default | Range | | -------------- | ------------------------------------ | ------- | ------ | | Crawl Depth | How many levels deep to follow links | 2 | 0-5 | | Max URLs | Maximum number of URLs to process | 200 | 1-2000 | | Batch Size | URLs processed concurrently | 10 | N/A | | Cache Duration | How long results are cached | 30 days | N/A | ## API Architecture ### POST `/api/scrape` The core API endpoint that handles documentation conversion. **Request Flow:** 1. URL validation against documentation URL patterns and explicit exceptions 2. Cache check (Redis/in-memory with 30-day TTL) 3. Provider call with optimized scraping parameters. Firecrawl is the default; `SCRAPE_PROVIDER=playwright` launches self-hosted Chromium. 4. Content post-processing and filtering 5. Response with markdown and cache status **Request Body:** ```json { "url": "https://developer.apple.com/documentation/swiftui", "action": "scrape" } ``` **Response:** ```json { "success": true, "data": { "markdown": "# SwiftUI Documentation\n\n..." }, "cached": false } ``` **Error Handling:** - Domain validation errors (400) - Provider/API errors (500) - Network timeouts (504) - Rate limiting (429) ## Tech Stack - **Framework**: [Next.js 16](https://nextjs.org/) with App Router - **Language**: [TypeScript](https://www.typescriptlang.org/) - **Styling**: [Tailwind CSS v4](https://tailwindcss.com/) - **API**: [Firecrawl](https://firecrawl.dev/) by default; opt-in self-hosted Playwright extraction for Node/Docker deployments - **Cache**: [Upstash Redis](https://upstash.com/) for distributed caching - **Deployment**: [Vercel](https://vercel.com/) - **Development**: Turbopack for fast refreshes ## Project Structure ``` llm-codes/ ├── src/ │ ├── app/ │ │ ├── api/ │ │ │ ├── scrape/ # Single-page scrape endpoint │ │ │ ├── crawl/ # Firecrawl crawl endpoints │ │ │ └── cache/ # Cache stats endpoint │ │ ├── globals.css # Global styles & Tailwind │ │ ├── layout.tsx # Root layout │ │ ├── page.tsx # Main page component │ │ └── icon.tsx # Dynamic favicon │ ├── constants.ts # Configuration constants │ ├── utils/ # Utility functions │ │ ├── code-extraction.ts # Code-block extraction │ │ ├── content-processing.ts # Content cleaning logic │ │ ├── file-utils.ts # File handling │ │ ├── result-processing.ts # Final output shaping │ │ ├── url-utils.ts # URL validation & handling │ │ └── __tests__/ # Utility tests │ ├── lib/ # Scrape providers, cache, retry, and HTTP clients │ └── test/ │ └── setup.ts # Test configuration ├── public/ │ ├── logo.png # App icon │ └── manifest.json # PWA manifest ├── next.config.js # Next.js configuration ├── postcss.config.js # PostCSS with Tailwind v4 ├── tsconfig.json # TypeScript configuration ├── vitest.config.ts # Vitest test configuration ├── spec.md # Detailed specification └── package.json # Dependencies ``` ## Technical Implementation Details ### Content Processing Pipeline 1. **URL Extraction**: Custom regex patterns extract links from markdown and HTML 2. **Domain-Specific Filtering**: Each documentation site has custom rules for link following 3. **Parallel Batch Processing**: URLs processed in batches of 10 for optimal performance 4. **Content Deduplication**: Hash-based paragraph and section deduplication 5. **Multi-Stage Filtering**: Sequential filters for URLs, navigation, boilerplate, and platform strings ### Performance Optimizations - **Batched API Calls**: Reduces provider latency by processing multiple URLs per request - **Progressive Loading**: UI updates with real-time progress during long crawls - **Smart Link Extraction**: Only follows relevant documentation links based on URL patterns - **Client-Side Caching**: Browser-based result caching for repeat operations ### Testing Strategy ```bash # Run all tests once pnpm run test:run # Run tests with UI pnpm run test:ui # Run tests with coverage pnpm run test:coverage # Type checking pnpm run type-check # Full local gate pnpm run verify # Live Firecrawl mode smoke without a local browser pnpm run verify:modes:live ``` Tests cover: - URL validation and domain filtering - Content processing and deduplication - API error handling - Cache behavior - UI component interactions ## Contributing Contributions are welcome! Please feel free to submit a Pull Request. 1. Fork the repository 2. Create your feature branch (`git checkout -b feature/AmazingFeature`) 3. Commit your changes (`git commit -m 'Add some AmazingFeature'`) 4. Push to the branch (`git push origin feature/AmazingFeature`) 5. Open a Pull Request ## Troubleshooting ### Notifications not working? - Check browser permissions for notifications - Ensure you're using a supported browser (Chrome, Firefox, Safari 10.14+, Edge) - Try resetting notification permissions in browser settings ### API Rate Limits? The app includes a 30-day cache to minimize API calls. If you're hitting rate limits: - Reduce crawl depth - Lower maximum URLs - Wait for cached results - Consider setting up Redis cache for better performance ### Redis Cache Setup For production use, we recommend setting up Redis cache: 1. Sign up for [Upstash](https://upstash.com) (free tier available) 2. Create a Redis database 3. Add the credentials to your environment variables 4. The app will automatically use Redis for caching Benefits: - Cache persists across deployments - Shared cache across all instances - Automatic compression for large documents - ~70% reduction in provider calls ### Deployment Issues? - Ensure `FIRECRAWL_API_KEY` is set when `SCRAPE_PROVIDER=firecrawl` - For `SCRAPE_PROVIDER=playwright`, run `pnpm exec playwright install chromium` on the self-hosted server - Check Vercel function logs for errors - Verify your API key is valid ## License This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. ## Architecture Decisions ### Why Firecrawl by default? - Handles JavaScript-heavy sites that traditional scrapers can't parse - Built-in markdown conversion with semantic structure preservation - Reliable headless browser automation at scale - Fits serverless deployments without shipping a browser binary ### Why Playwright as the self-hosted alternative? - Keeps extraction in the existing Next.js app instead of adding a Python crawler or separate Firecrawl-compatible service - Uses a real browser for JavaScript-rendered docs and returns the same Markdown-shaped API response - Has one operational prerequisite: local Chromium via `pnpm exec playwright install chromium` ### Why Next.js 16 + App Router? - Server-side API key security - Built-in caching with fetch() - Edge-ready deployment on Vercel ### Why Client-Side Processing? - Reduces server load for filtering operations - Enables real-time UI updates during processing - Allows users to customize output without re-fetching ## Future Enhancements - WebSocket support for real-time crawl progress - Custom domain rule configuration - Batch URL upload via CSV/JSON - Export to multiple formats (PDF, EPUB, Docusaurus) - LLM-specific formatting profiles ## Acknowledgments - Powered by [Firecrawl](https://firecrawl.dev/referral?rid=9CG538BE) by default, with optional self-hosted Playwright rendering - Inspired by the challenges of making documentation accessible to AI agents - Built with Next.js 16, Tailwind CSS v4, and TypeScript --- Built by [Peter Steinberger](https://steipete.me) | [Blog Post](https://steipete.me/posts/llm-codes-transform-developer-docs) | [Twitter](https://twitter.com/steipete)