---
title: Multimodal
description: Reason over images, audio recordings, and documents through the unified chat API.
icon: Layers
---

import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
import { TypeTable } from 'fumadocs-ui/components/type-table';
import { Callout } from 'fumadocs-ui/components/callout';
import { Image, FileAudio, ScanText, FileImage } from 'lucide-react';

The **Multimodal** capability extends `Addis-፩-አሌፍ` beyond text. You can upload images, audio files, and documents alongside your prompts to perform complex reasoning tasks.

*   **Vision:** Visual Q&A, Scene Description, and Object Detection.
*   **Audio Analysis:** Summarize voice notes, analyze sentiment, or extract action items from meetings (distinct from simple transcription).
*   **OCR:** Extract text from scanned African documents.

<Callout type="warn" title="Important: Data Format">
  Unlike standard text generation, this request **must** be sent as `multipart/form-data`.
  
  The configuration (prompt, model, parameters) must be passed as a **stringified JSON object** in a field named `request_data`.
</Callout>

## Usage Guide

### 1. Chat with Images (Vision)
Upload an image and ask questions about it. Ideal for explaining diagrams, identifying objects, or analyzing screenshots.

<Tabs items={['Node.js', 'Python', 'cURL']}>
  <Tab value="Node.js">
    ```ts
    import AddisAI, { fileFromPath } from "addisai";

    const addis = new AddisAI();
    const response = await addis.chat.completions.create({
      messages: [{ role: "user", content: "Describe this image in Amharic." }],
      attachments: [{ file: await fileFromPath("market.jpg", "image/jpeg") }],
    });

    console.log(response.choices[0].message.content);
    ```
  </Tab>
  <Tab value="Python">
    ```python
    from addisai import AddisAI

    addis = AddisAI()
    with open("market.jpg", "rb") as image:
        response = addis.chat.completions.create(
            messages=[{"role": "user", "content": "Describe this image in Amharic."}],
            attachments=[image],
        )

    print(response["choices"][0]["message"]["content"])
    ```
  </Tab>
  <Tab value="cURL">
    ```bash
    curl https://api.addisassistant.com/api/v1/chat_generate \
      -H "x-api-key: $ADDIS_API_KEY" \
      -F 'attachment_0=@market.jpg;type=image/jpeg' \
      -F 'request_data={"prompt":"Describe this image in Amharic.","target_language":"am","attachment_field_names":["attachment_0"]}'
    ```
  </Tab>
</Tabs>

### 2. Chat with Audio
Upload an audio file (e.g., a voice note or meeting recording) and ask the model to perform reasoning on it. 

**Difference from STT:** Speech-to-Text simply transcribes words. This endpoint **listens** to the audio and answers questions about it (e.g., "Summarize", "What was the tone?", "Extract action items").

<Tabs items={['Node.js', 'Python', 'cURL']}>
  <Tab value="Node.js">
    ```ts
    import AddisAI, { fileFromPath } from "addisai";

    const addis = new AddisAI();
    const response = await addis.chat.completions.create({
      messages: [{ role: "user", content: "Follow the instruction in this recording." }],
      audio: await fileFromPath("command.wav", "audio/wav"),
    });

    console.log(response.transcription?.clean);
    console.log(response.choices[0].message.content);
    ```
  </Tab>
  <Tab value="Python">
    ```python
    from addisai import AddisAI

    addis = AddisAI()
    with open("command.wav", "rb") as audio:
        response = addis.chat.completions.create(
            messages=[{"role": "user", "content": "Follow the instruction in this recording."}],
            audio=audio,
        )

    print(response.get("transcription", {}).get("clean"))
    print(response["choices"][0]["message"]["content"])
    ```
  </Tab>
  <Tab value="cURL">
    ```bash
    curl https://api.addisassistant.com/api/v1/chat_generate \
      -H "x-api-key: $ADDIS_API_KEY" \
      -F 'chat_audio_input=@command.wav;type=audio/wav' \
      -F 'request_data={"prompt":"Follow the instruction in this recording.","target_language":"am"}'
    ```
  </Tab>
</Tabs>

### 3. Document OCR (Text Extraction)
Use the vision model to extract text from scanned African documents, ID cards, or handwritten notes.

<Callout title="Dedicated Endpoint" type="info">
  Use the chat endpoint with an image or document attachment and an explicit extraction prompt for OCR and document-text extraction.
</Callout>

<Tabs items={['Node.js', 'Python', 'cURL']}>
  <Tab value="Node.js">
    ```ts
    import AddisAI, { fileFromPath } from "addisai";

    const addis = new AddisAI();
    const response = await addis.chat.completions.create({
      messages: [{ role: "user", content: "Extract every visible line exactly as written." }],
      attachments: [{ file: await fileFromPath("document.png", "image/png") }],
    });

    console.log(response.choices[0].message.content);
    ```
  </Tab>
  <Tab value="Python">
    ```python
    from addisai import AddisAI

    addis = AddisAI()
    with open("document.png", "rb") as document:
        response = addis.chat.completions.create(
            messages=[{"role": "user", "content": "Extract every visible line exactly as written."}],
            attachments=[document],
        )

    print(response["choices"][0]["message"]["content"])
    ```
  </Tab>
  <Tab value="cURL">
    ```bash
    curl https://api.addisassistant.com/api/v1/chat_generate \
      -H "x-api-key: $ADDIS_API_KEY" \
      -F 'attachment_0=@document.png;type=image/png' \
      -F 'request_data={"prompt":"Extract every visible line exactly as written.","target_language":"am","attachment_field_names":["attachment_0"]}'
    ```
  </Tab>
</Tabs>

### 4. Conversation History with Attachments
The API is **stateless**, meaning it treats every request independently. To continue a conversation involving attachments, you must provide the relevant context in the `conversation_history` array.

<Callout type="info" title="Note">
 **Do not re-upload files.** Instead, switch from `multipart/form-data` to a standard **JSON request** and reference the previously uploaded file using its `fileUri` inside the `parts` array.
</Callout>

This approach maintains continuity while minimizing latency and bandwidth.

The SDK examples retrieve the reusable attachment metadata from the first upload. The cURL tab shows the subsequent JSON request with that returned URI.

<Tabs items={['Node.js', 'Python', 'cURL']}>
  <Tab value="Node.js">
    ```ts
    import AddisAI, { fileFromPath } from "addisai";

    const addis = new AddisAI();
    const first = await addis.chat.completions.create({
      messages: [{ role: "user", content: "Describe this file." }],
      attachments: [{ file: await fileFromPath("report.pdf", "application/pdf") }],
    });

    console.log(first.uploaded_attachments);
    ```
  </Tab>
  <Tab value="Python">
    ```python
    from addisai import AddisAI

    addis = AddisAI()
    with open("report.pdf", "rb") as report:
        first = addis.chat.completions.create(
            messages=[{"role": "user", "content": "Describe this file."}],
            attachments=[report],
        )

    print(first.get("uploaded_attachments"))
    ```
  </Tab>
  <Tab value="cURL">
    ```bash
    curl --location 'https://api.addisassistant.com/api/v1/chat_generate' \
      --header 'Content-Type: application/json' \
      --header "X-API-Key: $ADDIS_API_KEY" \
      --data '{
        "prompt": "What was the document talking about in detail?",
        "target_language": "am",
        "conversation_history": [
          {
            "role": "user",
            "parts": [
              {
                "fileData": {
                  "fileUri": "YOUR_FILE_URI_HERE",
                  "mimeType": "application/pdf"
                }
              },
              { "text": "Describe this attachment" }
            ]
          },
          {
            "role": "assistant",
            "parts": [
              { "text": "ይህ ሰነድ በታህሳስ 18 ቀን 2025 በተካሄደው..." }
            ]
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

<Callout type="info" title="Reusing returned file URIs">
The SDK upload helper returns `uploaded_attachments` metadata after the first request. The current high-level SDK methods accept files for uploads; use the raw REST flow shown above when you need to reuse a returned `fileUri` without uploading the file again.
</Callout>

#### Conversation History Object Schema
The `conversation_history` array consists of message objects defined below.

<TypeTable
  type={{
    'role': {
      description: 'Who sent the message. Values: "user" or "assistant".',
      type: 'string',
      required: true
    },
    'parts': {
      description: 'An array of content parts. Can contain text and/or file references.',
      type: 'array',
      required: true
    }
  }}
/>

**Part Object Structure:**
Each item in the `parts` array must be **one** of the following:

<TypeTable
  type={{
    'text': {
      description: 'The text content of the message.',
      type: 'string',
      required: false
    },
    'fileData': {
      description: 'Reference to a previously uploaded file.',
      type: 'object',
      required: false
    }
  }}
/>

**FileData Object Structure:**

<TypeTable
  type={{
    'fileUri': {
      description: 'The URI returned in the "uploaded_attachments" field of a previous response.',
      type: 'string',
      required: true
    },
    'mimeType': {
      description: 'The standard MIME type (e.g., "application/pdf", "image/jpeg").',
      type: 'string',
      required: true
    }
  }}
/>
---

## API Reference

### Form Data Parameters
These parameters are sent in the `multipart/form-data` body.

<TypeTable
  type={{
    'image': {
      description: 'Image file binary (JPG, PNG, WEBP).',
      type: 'file',
      required: false
    },
    'audio': {
      description: 'Audio file binary (MP3, WAV, M4A).',
      type: 'file',
      required: false
    },
    'document': {
      description: 'Document file binary (PDF). Max 10MB.',
      type: 'file',
      required: false
    },
    'request_data': {
      description: 'A JSON string containing the prompt and configuration.',
      type: 'string (JSON)',
      required: true
    }
  }}
/>

### Request Data Object
These parameters go inside the `request_data` stringified JSON.

<TypeTable
  type={{
    'prompt': {
      description: 'The question or instruction regarding the uploaded file.',
      type: 'string',
      required: true
    },
    'target_language': {
      description: 'Output language code ("am" or "om").',
      type: 'string',
      required: true
    },
    'generation_config': {
      description: 'Optional parameters to control creativity (temperature, maxOutputTokens).',
      type: 'object',
      default: '{}'
    }
  }}
/>

### Basic Multimodal Response Schema
When files are uploaded, the response includes an `uploaded_attachments` array confirming the upload.

```json
{
    "status": "success",
    "data": {
        "response_text": "ይህ ሰነድ በ12ኛው ክፍለ ዘመን በንጉሥ ላሊበላ ስለተገነቡት የላሊበላ ውቅር አብያተ ክርስቲያናት ያብራራል። እነዚህ 11 አብያተ ክርስቲያናት ከወጥ ድንጋይ የተወቀሩ ሲሆኑ፣ በዓለም ቅርስነት የተመዘገቡ ድንቅ የኢትዮጵያ ስልጣኔ ማሳያዎች ናቸው። ሰነዱ በተለይ ስለ ቤተ ጊዮርጊስ የመስቀል ቅርጽ ግንባታ እና ስለ ውስብስብ የውሃ ማስወገጃ ስርዓታቸው ዝርዝር ትንታኔ ይሰጣል።",
        "finish_reason": "STOP",
        "usage_metadata": {
            "prompt_token_count": 2427,
            "candidates_token_count": 269,
            "total_token_count": 2696
        },
        "modelVersion": "Addis-፩-አሌፍ",
        "uploaded_attachments": [
            {
                "fileUri": "YOUR_FILE_URI_VARIABLE",
                "mimeType": "application/pdf"
            }
        ]
    }
}
```

#### Response Schema With Conversation History 

```json
{
    "status": "success",
    "data": {
        "response_text": "በቀረበው የውይይት ታሪክ መሠረት፣ ፋሲል ግቢ (የጎንደር ቤተመንግስት) በ17ኛው ክፍለ ዘመን በአፄ ፋሲለደስ የተገነባ ድንቅ የኪነ-ሕንጻ ውጤት ነው። ግቢው በውስጡ ስድስት ዋና ዋና ቤተ-መንግስቶችን፣ ቤተ-መጻሕፍትን እና አብያተ ክርስቲያናትን ይዟል። ሰነዱ በተለይ ስለ ግንባታው የህንድ እና የአረብ ኪነ-ሕንጻ ተፅእኖ እንዲሁም ስለ ህንፃው የመሬት መንቀጥቀጥ መቋቋም ችሎታ ዝርዝር ትንታኔ ይሰጣል።",
        "finish_reason": "STOP",
        "usage_metadata": {
            "prompt_token_count": 3500,
            "candidates_token_count": 145,
            "total_token_count": 3645
        },
        "modelVersion": "Addis-፩-አሌፍ"
    }
}
```
---
## Best Practices

Optimize your multimodal integration with these architectural patterns.

<div className="addis-offset-shell addis-grid grid-cols-1 md:grid-cols-2 mt-10">

  {/* Context Strategy */}
  <div className="flex flex-col p-5 rounded-lg border border-fd-border bg-fd-card shadow-sm transition-all hover:border-fd-primary/30">
    <div className="flex items-center gap-2.5 mb-4">
      <History className="w-4 h-4 text-fd-muted-foreground" />
      <h3 className="font-semibold text-sm text-fd-foreground uppercase tracking-wide">Context Management</h3>
    </div>
    <div className="space-y-3 text-sm text-fd-muted-foreground">
      <div>
        <strong className="text-fd-foreground">Don't Re-upload:</strong> Never re-upload the same file in a multi-turn chat. Upload it once, get the <code>fileUri</code> from the response, and reference that URI in future JSON requests.
      </div>
      <div>
        <strong className="text-fd-foreground">Latency:</strong> Passing a URI is milliseconds; re-uploading a 10MB PDF takes seconds.
      </div>
    </div>
  </div>

  {/* Document Prep */}
  <div className="flex flex-col p-5 rounded-lg border border-fd-border bg-fd-card shadow-sm transition-all hover:border-fd-primary/30">
    <div className="flex items-center gap-2.5 mb-4">
      <FileText className="w-4 h-4 text-fd-muted-foreground" />
      <h3 className="font-semibold text-sm text-fd-foreground uppercase tracking-wide">Document Prep</h3>
    </div>
    <div className="space-y-3 text-sm text-fd-muted-foreground">
      <div>
        <strong className="text-fd-foreground">PDF Format:</strong> Native PDFs (text-selectable) process faster and more accurately than Scanned PDFs (images inside PDF).
      </div>
      <div>
        <strong className="text-fd-foreground">Orientation:</strong> Ensure scanned documents are upright. Rotated text significantly degrades Amharic OCR performance.
      </div>
    </div>
  </div>

  {/* Prompting */}
  <div className="flex flex-col p-5 rounded-lg border border-fd-border bg-fd-card shadow-sm transition-all hover:border-fd-primary/30">
    <div className="flex items-center gap-2.5 mb-4">
      <ScanText className="w-4 h-4 text-fd-muted-foreground" />
      <h3 className="font-semibold text-sm text-fd-foreground uppercase tracking-wide">Visual Prompting</h3>
    </div>
    <div className="space-y-3 text-sm text-fd-muted-foreground">
      <div>
        <strong className="text-fd-foreground">Be Specific:</strong> Instead of "What's in this image?", ask "Extract the date and total amount from this receipt."
      </div>
      <div>
        <strong className="text-fd-foreground">Language:</strong> For best results on African documents, prompt in Amharic (e.g., "በምስሉ ላይ ያለውን ጽሑፍ አውጣ").
      </div>
    </div>
  </div>

  {/* Constraints */}
  <div className="flex flex-col p-5 rounded-lg border border-fd-border bg-fd-card shadow-sm transition-all hover:border-fd-primary/30">
    <div className="flex items-center gap-2.5 mb-4">
      <Zap className="w-4 h-4 text-fd-muted-foreground" />
      <h3 className="font-semibold text-sm text-fd-foreground uppercase tracking-wide">Constraints</h3>
    </div>
    <div className="space-y-3 text-sm text-fd-muted-foreground">
      <div className="flex items-center justify-between p-2 rounded bg-fd-secondary/40 border border-fd-border/50">
        <span className="text-xs font-medium">Max File Size</span>
        <code className="text-xs text-fd-foreground">10 MB</code>
      </div>
      <div>
        <strong className="text-fd-foreground">Token Cost:</strong> Images and PDFs consume significantly more tokens than text. Monitor your `usage_metadata` to avoid hitting rate limits.
      </div>
    </div>
  </div>

</div>