---
name: fiftyone-generate-data-lens-connector
description: Generate a Data Lens connector from an external database schema. Use when users want to connect an external data source (PostgreSQL, BigQuery, Databricks, MySQL, etc.) to FiftyOne Data Lens, or when they have a database schema and want to browse/import that data through the FiftyOne App.
---

# Generate Data Lens Connector

Generate a fully functional `DataLensOperator` plugin from a user-provided
database
schema. The generated connector lets users browse, preview, and import samples
from
their external data source directly through the FiftyOne App's Data Lens panel.

## Enterprise Notice

Data Lens is a **FiftyOne Enterprise** feature. Before proceeding, inform
the user:

> **Note:** Data Lens is an enterprise-only feature available in
> [FiftyOne Enterprise](https://docs.voxel51.com/enterprise/index.html). The
> connector generated by this skill requires a FiftyOne Enterprise deployment
> to run. If you're using the open-source version of FiftyOne, this connector
> will not work in your environment.
>
> Would you like to proceed?

Wait for the user to confirm before continuing. If they ask about alternatives
for OSS, suggest they look into custom operators for similar data-import
workflows, or the standard
[dataset import](https://docs.voxel51.com/user_guide/dataset_creation/index.html)
utilities.

## Key Directives

**ALWAYS follow these rules:**

### 1. Schema first, code second

Never generate connector code until you fully understand the source schema. Ask
clarifying questions about anything ambiguous — column semantics, coordinate
systems,
filepath conventions, image dimensions.

### 2. Propose the field mapping before generating

Present a clear table mapping source columns to FiftyOne field types. Get user
approval before writing any code. This is the most important step — a connector
that maps fields wrong is worse than no connector.

### 3. Use parameterized queries

Never interpolate user input directly into SQL strings. Use the database
driver's
parameterized query mechanism (e.g., `%s` for psycopg, `@param` for BigQuery,
`:param` for Databricks).

### 4. Respect the batching contract

Always yield `DataLensSearchResponse` objects in batches of
`request.batch_size`.
Buffer results and yield when the buffer is full, plus a final yield for
remaining
samples.

### 5. Keep it minimal

Generate only what's needed. Don't add vector search, text search, or advanced
features unless the user's schema and requirements call for them. A simple
metadata
filter connector is the right default.

## Workflow

### Phase 1: Understand the Schema

Gather the information needed to generate the connector:

1. **Get the schema.** Accept any of these formats:
    - DDL (`CREATE TABLE` statements)
    - Column list with types (JSON, markdown table, plain text)
    - A request to introspect a live database (guide the user to export the
      schema)

2. **Identify key columns.** Ask about any that aren't obvious:
    - **Filepath column** — which column contains the media path? Is it
      absolute,
      relative (needs a prefix), or a cloud URI?
    - **Label columns** — which columns contain annotations? What format?
      (JSON blobs, foreign key joins, flat columns)
    - **Metadata columns** — which columns should become filterable properties?
    - **Coordinate system** — if bounding boxes exist: pixel-absolute or
      normalized?
      What are the image dimensions (fixed or per-sample)?

3. **Identify the database type.** This determines:
    - Which Python driver to use
    - Query syntax (parameterization style, JSON functions, etc.)
    - Connection string format and required secrets

4. **Get sample data** (if available). Even 2-3 example rows dramatically
   improve
   the quality of the field mapping. Ask for them.

### Phase 2: Propose Field Mapping

Present a mapping table for user approval:

```
| Source Column     | FiftyOne Field        | Type                | Notes                          |
|-------------------|-----------------------|---------------------|--------------------------------|
| filepath          | filepath              | str                 | Prefix: gs://bucket/path/      |
| weather           | weather               | Classification      | Filterable enum                |
| bbox_x1/y1/x2/y2 | detections            | Detections          | Pixel coords, normalize by WxH |
| category          | detections[].label    | str                 | Detection label                |
| ...               | ...                   | ...                 | ...                            |
```

Include:

- **Filepath construction** — how the full path is built from the column value
- **Coordinate normalization** — formula if bounding boxes are in pixel
  coordinates
- **Label hierarchy** — how nested/joined label data maps to FiftyOne label
  types
- **Filters** — which columns become `resolve_input` enum/text fields, with
  known values if available

**Wait for user approval before proceeding.**

### Phase 3: Generate Connector

Generate a complete plugin directory with these files:

| File               | Purpose                                           |
|--------------------|---------------------------------------------------|
| `__init__.py`      | Operator class + handler class + sample transform |
| `fiftyone.yml`     | Plugin manifest with operator name and secrets    |
| `requirements.txt` | Python driver dependency                          |

Use the patterns from [CONNECTOR-TEMPLATE.md](CONNECTOR-TEMPLATE.md) as your
structural guide. The generated code should follow these architectural layers:

**Layer 1 — Operator class** (`DataLensOperator` subclass):

- `config` property with `execute_as_generator=True`, `unlisted=True`
- `resolve_input()` defining the UI filter form
- `handle_lens_search_request()` delegating to the handler

**Layer 2 — Handler class** (connection + query logic):

- Context manager for connection lifecycle
- `iter_batches()` implementing the batching loop
- `_generate_query()` building parameterized SQL from `search_params`
- `_transform_sample()` mapping raw rows to `fo.Sample(...).to_dict()`

**Layer 3 — Data models** (optional, for complex schemas):

- `@dataclass` for query parameters (validated from `search_params`)
- `@dataclass` for query result rows (typed column access)

See [FIELD-MAPPING-GUIDE.md](FIELD-MAPPING-GUIDE.md) for the rules on mapping
database types to FiftyOne field types, especially for spatial data (bounding
boxes,
keypoints, segmentation masks).

### Phase 4: Validate

After generating the connector:

1. **Syntax check** — the generated code must parse without errors:
   ```bash
   python -c "import ast; ast.parse(open('__init__.py').read()); print('OK')"
   ```

2. **Sample construction check** — if sample data was provided, construct
   `fo.Sample` objects from it and verify they serialize correctly:
   ```python
   import fiftyone as fo
   sample = fo.Sample(
       filepath="...",
       # ... mapped fields
   )
   sample.to_dict()  # Must not raise
   ```

3. **Walk through the generated code** with the user:
    - Confirm the query logic matches their schema
    - Confirm the transform logic produces correct FiftyOne samples
    - Confirm the `resolve_input` filters are right

4. **Installation guidance:**
   ```bash
   # Copy plugin to FiftyOne plugins directory
   PLUGINS_DIR=$(python -c "import fiftyone as fo; print(fo.config.plugins_dir)")
   cp -r ./my-connector "$PLUGINS_DIR/"

   # Set required secrets
   export MY_SECRET_KEY="..."

   # Register in Data Lens UI: Data sources > Add > plugin-name/operator_name
   ```

### Phase 5: Iterate

The first generation likely needs refinement. Common adjustments:

- Adding/removing filter fields
- Fixing coordinate normalization
- Adjusting the filepath prefix
- Handling NULL/missing values in optional columns
- Adding conditional WHERE clauses for "all" filter values

## Database-Specific Patterns

### PostgreSQL

```python
# Driver: psycopg[binary]
# Parameterization: %s positional
# Connection: ctx.secret("POSTGRES_CONNECTION_STRING")
# JSON: JSON_AGG, JSON_BUILD_OBJECT
# Streaming: cursor.stream(query, params, size=batch_size)
import psycopg
from psycopg.rows import dict_row
```

### BigQuery

```python
# Driver: google-cloud-bigquery
# Parameterization: @param with QueryJobConfig
# Connection: project from ctx.secret("BIGQUERY_PROJECT")
# JSON: JSON_VALUE, JSON_QUERY
from google.cloud import bigquery

job_config = bigquery.QueryJobConfig(
    query_parameters=[
        bigquery.ScalarQueryParameter("name", "STRING", value)
    ]
)
```

### Databricks

```python
# Driver: databricks-sdk
# Parameterization: :param with StatementParameterListItem
# Connection: ctx.secret("DATABRICKS_HOST"), ctx.secret("DATABRICKS_HTTP_PATH")
# JSON: Standard SQL
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.sql import StatementParameterListItem
```

### MySQL

```python
# Driver: mysql-connector-python
# Parameterization: %s positional (same as psycopg)
# Connection: ctx.secret("MYSQL_CONNECTION_STRING")
import mysql.connector
```

### SQLite

```python
# Driver: built-in sqlite3
# Parameterization: ? positional
# Connection: file path from ctx.secret("SQLITE_DB_PATH") or ctx.params
import sqlite3
```

## Quick Reference

### Required OperatorConfig flags

```python
foo.OperatorConfig(
    name="my_connector",
    label="My Connector",
    execute_as_generator=True,  # REQUIRED for Data Lens
    unlisted=True,  # Hide from operator browser
)
```

### Batching pattern

```python
buffer = []
for row in query_results:
    buffer.append(transform(row))
    if len(buffer) >= request.batch_size:
        yield DataLensSearchResponse(
            result_count=len(buffer),
            query_result=buffer,
        )
        buffer = []
if buffer:
    yield DataLensSearchResponse(
        result_count=len(buffer),
        query_result=buffer,
    )
```

### Conditional WHERE for "all" option

```python
# Pattern: skip filter when user selects "all"
WHERE
1 = 1
AND(1 = {1 if param == "all" else 0}
OR
column = %s)
```

### FiftyOne imports

```python
import fiftyone as fo
import fiftyone.operators as foo
import fiftyone.operators.types as types
from fiftyone.operators.data_lens import (
    DataLensOperator,
    DataLensSearchRequest,
    DataLensSearchResponse,
)
```

## Resources

- [Data Lens Documentation](https://docs.voxel51.com/enterprise/data_lens.html)
- [Plugin Development Guide](https://docs.voxel51.com/plugins/developing_plugins.html)
- [Operator Types API](https://docs.voxel51.com/api/fiftyone.operators.types.html)