# Contributing to AI Hub Models

[![Qualcomm® AI Hub Models](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/quic-logo.jpg)](https://aihub.qualcomm.com)

This guide covers how to add new models to the repository and contribute changes.

## Prerequisites

Before contributing:

1. **Legal approval** is required before publishing new models. Submit a request at go/genairequest.

2. **Join the access lists** (required for GitHub repo access): [qai-hub-users](https://lists.qualcomm.com/ListManager?id=qai-hub-users&query=qai-hub-users&type=list) and [qai-hub-models-contributors](https://lists.qualcomm.com/ListManager?query=qai-hub-models-contributors&field=default&match=sw).

3. **Clone the repository** and set up your environment:
   ```bash
   git clone https://github.com/qualcomm/ai-hub-models
   cd ai-hub-models
   python scripts/build_and_test.py install_deps
   source qaihm-dev/bin/activate
   pre-commit install
   ```

4. **Create a branch** using the format: `dev/<username>/<branch_name>`

## Terminology

- **model_id** - The folder name (e.g., `yolov7`, `ddrnet23_slim`)
- **model_name** - The published display name in manifest.yaml (e.g., "YOLOv7", "DDRNet23-Slim")

## Model Directory Structure

Each model lives in `qai_hub_models/models/<model_id>/` and requires:

| File | Purpose |
|------|---------|
| `model.py` | PyTorch model inheriting from `BaseModel` |
| `app.py` | End-to-end application with pre/post-processing (can be inherited from `models/templates/`) |
| `demo.py` | CLI demo running the app on sample data |
| `test.py` | Unit tests for the model |
| `manifest.yaml` | Website metadata + codegen/export options (single unified schema) |

**Note:** Many models extend template implementations from `models/templates/`. For example, segmentation models can inherit from `CityscapesSegmentor` and use the template `SegmentationApp`. Check existing models solving similar tasks before writing everything from scratch.

**Sample models to reference:**
- **Regular model**: [`ddrnet23_slim`](qai_hub_models/models/ddrnet23_slim/) - segmentation model using a template base class
- **Collection model**: [`whisper_tiny`](qai_hub_models/models/whisper_tiny/) - encoder-decoder with separately compiled components

### Optional Files

| File | Purpose |
|------|---------|
| `requirements.txt` | Model-specific dependencies (pinned versions required) |
| `scorecard-config.yaml` | CI/scorecard-only knobs (e.g., `global_requirements_incompatible`, `skip_scorecard`, `test_split`) |

### Auto-generated Files

These are created by running codegen scripts:

| File | Generated By |
|------|--------------|
| `README.md`, `export.py`, `evaluate.py`, `test_generated.py` | `python qai_hub_models/scripts/run_codegen.py -m <model_id>` (`evaluate.py` is skipped for precompiled models or when `skip_evaluate` is set in `scorecard-config.yaml`) |
| `perf.yaml`, `numerics.yaml`, `release-assets.yaml` | Scorecard / CI (generated weekly, not included in your PR) |

---

## 1. requirements.txt

Model-specific dependencies not in the base environment.

**All packages must be pinned to exact versions** (e.g., `torch==2.0.1`).

When adding dependencies:

1. Check `qai_hub_models/requirements.txt` first - don't duplicate
2. Check `global_requirements.txt` - use the same version if possible
3. If a different version is required, set `global_requirements_incompatible: true` in the model's `scorecard-config.yaml`
4. After adding new packages, run `python qai_hub_models/scripts/generate_global_requirements.py`
5. If the package is not in `global_requirements.txt`, you can choose any version as long as its dependencies are compatible with existing global requirements

**Additional options for complex dependency scenarios:**
- `global_requirements_incompatible` (in `scorecard-config.yaml`) - Set when versions differ from global requirements
- `pip_pre_build_reqs` (in `manifest.yaml`) - Packages that must be installed before the model's requirements (for broken build dependencies)
- `pip_install_flags` (in `manifest.yaml`) - Extra pip flags needed when installing (requires `global_requirements_incompatible: true` in `scorecard-config.yaml`)

---

## 2. model.py

The model file defines the PyTorch module that will be compiled and run on-device.

### Basic Structure

All models must:

- Inherit from `BaseModel` (which inherits `torch.nn.Module`)
- Include `MODEL_ID = __name__.split(".")[-2]` before the class definition
- Implement the required methods listed below

### Required Methods

| Method | Description |
|--------|-------------|
| `from_pretrained(cls)` | Classmethod to load pretrained weights. All arguments must have defaults. |
| `get_input_spec()` | Returns `InputSpec` dict of `{input_name: (shape, dtype)}`. For image inputs, follow the standard: RGB format with values in range [0, 1]. |
| `get_output_spec()` | Returns dict of outputs |

### Optional Methods

These have default implementations but can be overridden:

| Method | Description |
|--------|-------------|
| `_sample_inputs_impl()` | Provide real sample inputs instead of random data. Important for more accurate PSNR data, since `export.py` runs a single sample inference and reports the PSNR difference between torch and device. |
| `get_hub_compile_options()` / `get_hub_inference_options()` | Custom AI Hub Workbench flags |
| `get_unsupported_reason()` | Mark specific device attributes that can't be supported. Rarely defined in practice; only necessary if a specific Hexagon version is required for advanced models. |
| `get_eval_dataset_classes()` | List of `BaseDataset` classes on which this model can be evaluated. If unset, model will not support eval. |
| `get_evaluator()` | Return evaluator instance for accuracy measurement. If unset, model will not support eval. |
| `get_calibration_dataset_cls()` | `BaseDataset` class used for quantization calibration. If unset, model will not support quantization. |
| `get_hub_litemp_percentage(precision)` | Percentage (0-100) of layers to keep in higher precision for mixed precision |

### Loading External Model Code

When your model depends on external code (e.g., from a GitHub repo or third-party package), follow this preference order:

1. **Add to requirements.txt (Preferred)** - Add the dependency to `requirements.txt`. Prefer PyPI packages when available:
   ```
   timm==0.9.2
   ```
   If not on PyPI, install directly from GitHub by adding to `manifest.yaml`:
   ```yaml
   additional_pip_requirements:
     - git+https://github.com/owner/repo.git@v1.0.0
   ```

2. **Monkeypatching** - If you need to modify external code behavior, use monkeypatching rather than copying source (see [Appendix: Monkeypatching](#appendix-monkeypatching)).

3. **SourceAsRoot (Last Resort)** - Only use `SourceAsRoot` when the above approaches are not possible (e.g., the code is not installable, requires heavy modifications, or has incompatible dependencies). This utility clones the external repo to the user's machine and sets up the Python environment as if running from that repo's source root. It also allows you to apply patches to the cloned source. Should be avoided when possible.

For pretrained weights:

1. **Library weights** - Many libraries provide pretrained options
2. **Download links** - Use URLs from model READMEs
3. **S3 upload** - Upload to `qaihub-public-assets` bucket (see [Uploading Assets to S3](#uploading-assets-to-s3))

### Special Model Types

**CollectionModel** - For models compiled as separate assets (e.g., encoder-decoder):
- Main class inherits from `CollectionModel`
- Each component inherits from `BaseModel`
- See [`whisper_tiny`](qai_hub_models/models/whisper_tiny/) for an example

**PrecompiledWorkbenchModel** - For models where only compiled assets are published (no PyTorch source).

---

## 3. app.py

The app enables end-to-end usage with user-friendly I/O formats.

### Requirements

- Define an `App` class that takes a `Callable` in `__init__`. The callable should represent a model. We use `Callable` (rather than a specific type) because the PyTorch implementation may be replaced with an equivalent implementation that uses a different runtime.
- Implement a `predict()` method for inference

### Useful Utilities

Check these files in `qai_hub_models/utils/` before implementing your own:

| File | Purpose |
|------|---------|
| `image_processing.py` | Image preprocessing and conversion (e.g., `app_to_net_image_inputs`) |
| `bounding_box_processing.py` | NMS and bounding box utilities for object detection |
| `draw.py` | Drawing utilities for visualization (points, connections, boxes) |
| `display.py` | Display and save image utilities |
| `asset_loaders.py` | Loading assets from URLs, S3, Google Drive |
| `transpose_channel.py` | Channel transposition utilities (NCHW <-> NHWC) |

Also check `qai_hub_models/models/templates/` for template app implementations for common tasks (segmentation, classification, object detection, etc.).

### I/O Conventions

Follow existing patterns for consistency. **For image inputs to the model, follow the standard: RGB format with values normalized to range [0, 1].**

| Task | Input Format | Output Format |
|------|--------------|---------------|
| Image processing | `PIL.Image.Image` | `PIL.Image.Image` |
| Video processing | filepath string | filepath string |
| Classification | image/tensor | tensor of probabilities (post-softmax) |
| Object detection | image | list of bounding boxes (post-NMS) |

Check `qai_hub_models/utils/` for existing utilities like `app_to_net_image_inputs` before implementing your own.

---

## 4. demo.py

The demo runs the app on sample data and presents results. See [`ddrnet23_slim/demo.py`](qai_hub_models/models/ddrnet23_slim/demo.py) for an example.

### Standard Pattern

1. Parse input arguments (model options, `--eval-mode`, `--device`, `--output-dir`, etc.)
2. Initialize the model
3. Load and preprocess inputs
4. Initialize and run the app
5. Display or save results (behind `is_test` flag for unit tests)

Sample data is typically stored in S3 and loaded using asset utilities.

---

## 5. test.py

### Standard Tests

| Test | Required | Purpose |
|------|----------|---------|
| `test_task` | Yes | PyTorch model accuracy on sample input |
| `test_demo` | Yes | Demo runs without exceptions |

---

## 6. manifest.yaml

The manifest is a single unified schema holding both website-facing metadata and codegen/export options. It replaces the earlier split between `info.yaml` and `code-gen.yaml`.

- See [`ddrnet23_slim/manifest.yaml`](qai_hub_models/models/ddrnet23_slim/manifest.yaml) for an example
- See `QAIHMModelManifest` in `qai_hub_models/configs/manifest_yaml.py` for field details

**Website metadata fields:**
- `name`, `id`, `status` - Model identity (id must match folder name). **Set `status: pending` for new models.** The scorecard will automatically update this to `public` once the model passes validation.
- `headline`, `description`, `use_case`, `domain` - Public-facing description
- `tags`, `applicable_scenarios`, `related_models` - Categorization
- `form_factors` - Target devices (Phone, Tablet, IoT)
- `technical_details` - Model specs (some auto-filled by script)
- `license_type`, `source_repo`, `research_paper` - Attribution
- `labels_file` - Path to labels file for classification models
- `has_static_banner` - Whether the model has a static web banner (required for publishing)

 **Banner assets:** If you set `has_static_banner: true`, a static banner must exist at the model's web-asset URL, so upload one to the web-asset S3 bucket for this model. If you don't have a banner, either set `has_static_banner: false` (the model stays `pending` and won't be promoted to `public`), or reuse an existing banner from another model in the same category/domain if one is relevant.

**License restrictions:** Before onboarding a model, check the license of its source weights and set `license_type` accordingly. The license decides whether a model can be onboarded and published (see `MODEL_LICENSE` in `qai_hub_models/configs/_info_yaml_enums.py`):

| Tier | Licenses | Onboard? |
|------|----------|----------|
| Non-restrictive | `apache-2.0`, `mit`, `bsd-3-clause`, `cc-by-4.0` | Preferred. Permit redistribution and commercial use freely. |
| Copyleft / model-specific | `agpl-3.0`, `gpl-3.0`, `creativeml-openrail-m`, `llama2`, `llama3`, `gemma`, `falcon3`, `taide` | Allowed, but carry redistribution / same-license obligations. Set `restrict_model_sharing: true` if sharing the weights is restricted. |
| Non-commercial | `other-non-commercial`, `cc-by-non-commercial-4.0` | Avoid. Cannot be promoted to `public`. |

**Codegen / export options:**
- `supported_precisions` - List of precisions to enable (float, w8a8, w8a16, w4a16, etc.)
- `has_on_target_demo` - Set to true if the model supports on-device demo
- `disabled_paths` - Disable specific precision/runtime combinations with reason
- `is_collection_model` - True for multi-component (encoder-decoder) models

Auto-fill size and parameter count:
```bash
python qai_hub_models/scripts/autofill_manifest_yaml.py -m <model_id>
```

CI-only knobs (skip_scorecard, test_split, global_requirements_incompatible, numerics_threshold_override, ...) live in a separate `scorecard-config.yaml` under `scorecard/models/<model_id>/`. See `QAIHMModelScorecardConfig` in `qai_hub_models/scorecard/scorecard_config_yaml.py`.

---

## Adding Quantization Support

To enable quantized precisions (w8a8, w8a16, etc.) for a model:

### 1. Add a Dataset

Create or reuse a dataset. Inherit from `BaseDataset` and implement the required methods.

**Where to put it:** if the dataset is specific to a single model, place it in that model's own folder (e.g. `qai_hub_models/models/<model_id>/dataset.py`). Only put it in the shared `qai_hub_models/datasets/` folder if it is reusable across multiple models (e.g. ImageNet, COCO). This mirrors the convention for evaluators below. Do not add model-specific dataset code under the shared `datasets/` folder.

**Dataset licensing:** A dataset is used in two distinct ways, and each has different licensing requirements:

- **Evaluation datasets** are only read to measure accuracy; they are never redistributed or baked into a shipped artifact. Any non-restrictive license is acceptable here (e.g. a dataset that is free for research or evaluation use).
- **Calibration datasets** must be **open source**. Calibration data is used to quantize the model, and the resulting quantized weights are published as part of the reproducible quantization flow, so the data effectively becomes part of what we ship. A restrictive or non-commercial license on the calibration set would carry those restrictions into the released model.

Practically: never use a dataset with a restrictive or non-commercial license for calibration. If the only dataset that fits your task is restrictive, you can still use it for evaluation, but pick a separate open-source dataset (or a public subset) for calibration.

**Required methods:**

| Method | Description |
|--------|-------------|
| `__init__(self, split, ...)` | Initialize with `DatasetSplit.TRAIN` or `DatasetSplit.VAL` |
| `__len__(self)` | Return the number of samples |
| `__getitem__(self, idx)` | Return `(input_tensor, ground_truth)` for a sample |
| `_download_data(self)` | Download dataset to `self.dataset_path` |
| `default_samples_per_job()` | Static method returning default batch size for inference jobs |
| `get_dataset_metadata()` | Return `DatasetMetadata(link, split_description)` for website |

**Optional methods:**

| Method | Description |
|--------|-------------|
| `_validate_data(self)` | Validate downloaded data (default: check path exists) |
| `collate_fn(batch)` | Custom collation for DataLoader |

See [`imagenette.py`](qai_hub_models/datasets/imagenet/imagenette.py) for an example of a shared dataset, and [`foot_track_net/dataset.py`](qai_hub_models/models/foot_track_net/dataset.py) for a model-specific dataset that lives in the model's own folder.

### 2. Add an Evaluator

Create or reuse an evaluator. Place it in the model's own folder if it's only used by that model, or in the appropriate template folder under `qai_hub_models/models/templates/<name>/` if it's used by multiple models. Inherit from `BaseEvaluator` and implement the required methods.

**Required methods:**

| Method | Description |
|--------|-------------|
| `add_batch(self, output, gt)` | Accumulate metrics for a batch of model outputs vs ground truth |
| `reset(self)` | Reset accumulated state |
| `get_accuracy_score(self)` | Return single float accuracy (higher is better by default; for an error metric like WER or NME, set `higher_is_better=False` on the metric returned by `get_metric_metadata`) |
| `formatted_accuracy(self)` | Return formatted string with accuracy and units |

**Optional methods:**

| Method | Description |
|--------|-------------|
| `get_metric_metadata(self)` | Return `MetricMetadata` for website publishing (name, unit, range, and whether higher or lower is better) |

See [`classification_evaluator.py`](qai_hub_models/models/templates/imagenet_classifier/classification_evaluator.py) for an example evaluator implementation.

### 3. Update the Model

Add these methods to `model.py`. They return the dataset/evaluator classes directly (import them from wherever you placed them above):

```python
@classmethod
def get_eval_dataset_classes(cls) -> list[type[BaseDataset]]:
    return [YourDataset]

def get_calibration_dataset_cls(self) -> type[BaseDataset]:
    return YourDataset

def get_evaluator(self) -> BaseEvaluator:
    return YourEvaluator(...)  # Return an instance
```

### 4. Update manifest.yaml

Add supported precisions (defaults to `float` only if not specified):
```yaml
supported_precisions:
  - float
  - w8a8
  - w8a16
```

### 5. Run Codegen and Test

```bash
python qai_hub_models/scripts/run_codegen.py -m <model_id>
python -m qai_hub_models.models.<model_id>.evaluate --precision w8a8
```

Accuracy drop from float should be reasonable (10 points or less). Consider mixed precision (e.g., `w8a8_mixed_int16`) if accuracy is too low.

---

## Uploading Assets to S3

For model checkpoints or test data not available via public URLs:

1. **Authenticate**: Run `python scripts/build_and_test.py validate_aws_credentials` (prompts for password)

2. **Upload**: Use AWS profile `qaihm`:
   ```bash
   aws s3 cp <local_file> s3://qaihub-public-assets/qai-hub-models/models/<model_id>/v1/ --profile qaihm
   ```

3. **Set permissions**: Grant public-read access when uploading

4. **Reference in code**: Set `MODEL_ASSET_VERSION = 1` in model.py

5. **Versioning**: Assets cannot be deleted. For new versions, create `v2/`, `v3/`, etc. and update `MODEL_ASSET_VERSION`.

---

## Verification Workflow

Before submitting a PR:

```bash
# Run codegen for your model
python qai_hub_models/scripts/run_codegen.py -m <model_id>

# Auto-fill manifest.yaml
python qai_hub_models/scripts/autofill_manifest_yaml.py -m <model_id>

# Run all pre-commit hooks
pre-commit run --all-files

# Run package unit tests
python scripts/build_and_test.py test_qaihm
```

Run model-specific tests:
```bash
# Test export (install model dependencies first per README)
python -m qai_hub_models.models.<model_id>.export --target-runtime tflite --chipset qualcomm-snapdragon-8gen3

# Test evaluation (if available)
python -m qai_hub_models.models.<model_id>.evaluate
```

- `export.py` should produce a model that profiles successfully on device (for all added precisions)
- `evaluate.py` should produce good results in torch and on-device for all supported precisions

---

## Code Quality

### Linting, Formatting, and Type Checking

You can run these tools manually, but they also run automatically via pre-commit hooks:

- **Ruff** - Linting and formatting: `ruff check --fix` and `ruff format`
- **mypy** - Type checking: `mypy qai_hub_models/`

### Pre-commit Hooks

Hooks run automatically and include:
- License header insertion (BSD-3)
- YAML validation, trailing whitespace, large file detection
- Ruff check + format
- mypy type checking

### Import Restrictions

Don't import directly: `numba`, `xtcocotools`, `git` - use `qai_hub_models.extern.*` wrappers instead.

---

## Support

For questions, reach out to the default PR reviewers or ping the [Teams channel](https://teams.microsoft.com/l/team/19%3a8AqY12ieagZC_SaB9Ulhmt8Ddswt-wHCBlMsJQ4snMw1%40thread.tacv2/conversations?groupId=73f7bf57-2cd3-473d-9f8f-df0a8d76b4f8&tenantId=98e9ba89-e1a1-4e38-9007-8bdabc25de1d).

---

## Appendix: Monkeypatching

When external model code needs modification for on-device compilation, **prefer monkeypatching over copying source**. This keeps the codebase maintainable and makes it clear what was changed.

**Common pattern:**

1. Create a `model_patches.py` file with replacement functions
2. Import the original class/module
3. Replace the method: `OriginalClass.method = patched_method`

**Example - replacing a forward method** ([`gkt`](qai_hub_models/models/gkt/)):
```python
# In model_patches.py - define the patched function
def KernelAttention_forward(self, q, k, v, skip=None, mask=None):
    # Modified implementation for on-device compatibility
    ...

# In model.py - apply the patch after importing
from external_repo import KernelAttention
from .model_patches import KernelAttention_forward

KernelAttention.forward = KernelAttention_forward
```

**Example - patching multiple components** ([`sam2`](qai_hub_models/models/sam2/)):
```python
# Patch module-level functions
import sam2.modeling.backbones.hieradet as hieradet
hieradet.window_partition = window_partition_5d

# Patch class methods with functools.partial
sam2.sam_prompt_encoder._embed_points = functools.partial(
    patched_embed_points, sam2.sam_prompt_encoder
)

# Replace entire submodules
for block in model.blocks:
    block.mlp = PatchedMLP(block.mlp)
```

**When to monkeypatch:**
- Replacing operations unsupported by QNN (e.g., certain einsum patterns, dynamic shapes)
- Optimizing tensor layouts for on-device performance (e.g., 6D → 5D tensors)
- Removing GPU-specific code paths