# Data Input and Output Patterns

This chapter covers advanced patterns for handling data input and output in Tercen operators. Building on the basic concepts from previous chapters, we'll explore sophisticated techniques for data manipulation, multiple output types, and complex data relationships.

::: {.callout-note}
## Prerequisites
Before proceeding, ensure you've completed:
- Development Environment Setup chapter for development environment
- Basic Implementation chapter for core operator concepts
- Understanding of Tercen's projection system
:::

## Understanding Tercen's Data Structure

Tercen organizes data using a projection system with specific index columns:

| Column | Purpose | Usage |
|--------|---------|-------|
| `.ri` | Row index | Identifies specific rows in the data projection |
| `.ci` | Column index | Identifies specific columns in the data projection |
| `.y` | Data values | The actual measurement or observation values |
| `.x` | X-axis values | Independent variable values (when applicable) |

These special columns enable flexible data aggregation and output patterns.

## Basic Output Patterns

### Per-Cell Output (Default)

Most operators output one result per cell, maintaining the original data structure:

::: {.panel-tabset}

### R

```r
library(tercen)
library(dplyr)

# Connect to Tercen context
ctx <- tercenCtx()

# Per-cell calculation (e.g., log transformation)
result <- ctx %>%
  select(.ri, .ci, .y) %>%
  mutate(log_value = log(.y + 1)) %>%
  select(.ri, .ci, log_value) %>%
  ctx$addNamespace()

ctx$save(result)
```

### Python

```python
from tercen.client import context as ctx
import polars as pl

# Connect to Tercen context
tercenCtx = ctx.TercenContext()

# Per-cell calculation
df = (
    tercenCtx
    .select(['.ri', '.ci', '.y'], df_lib="polars")
    .with_columns([
        (pl.col('.y') + 1).log().alias('log_value')
    ])
    .select(['.ri', '.ci', 'log_value'])
)

df = tercenCtx.add_namespace(df)
tercenCtx.save(df)
```

:::

### Per-Row Output

Aggregate data across columns for each row:

::: {.panel-tabset}

### R

```r
# Calculate statistics per row
row_stats <- ctx %>%
  select(.ri, .ci, .y) %>%
  group_by(.ri) %>%
  summarise(
    mean_value = mean(.y, na.rm = TRUE),
    sd_value = sd(.y, na.rm = TRUE),
    count = n(),
    .groups = "drop"
  ) %>%
  ctx$addNamespace()

ctx$save(row_stats)
```

### Python

```python
# Calculate statistics per row
df = (
    tercenCtx
    .select(['.ri', '.ci', '.y'], df_lib="polars")
    .group_by(['.ri'])
    .agg([
        pl.col('.y').mean().alias('mean_value'),
        pl.col('.y').std().alias('sd_value'),
        pl.col('.y').count().alias('count')
    ])
)

df = tercenCtx.add_namespace(df)
tercenCtx.save(df)
```

:::

### Per-Column Output

Aggregate data across rows for each column:

::: {.panel-tabset}

### R

```r
# Calculate statistics per column
col_stats <- ctx %>%
  select(.ri, .ci, .y) %>%
  group_by(.ci) %>%
  summarise(
    median_value = median(.y, na.rm = TRUE),
    q25 = quantile(.y, 0.25, na.rm = TRUE),
    q75 = quantile(.y, 0.75, na.rm = TRUE),
    .groups = "drop"
  ) %>%
  ctx$addNamespace()

ctx$save(col_stats)
```

### Python

```python
# Calculate statistics per column
df = (
    tercenCtx
    .select(['.ri', '.ci', '.y'], df_lib="polars")
    .group_by(['.ci'])
    .agg([
        pl.col('.y').median().alias('median_value'),
        pl.col('.y').quantile(0.25).alias('q25'),
        pl.col('.y').quantile(0.75).alias('q75')
    ])
)

df = tercenCtx.add_namespace(df)
tercenCtx.save(df)
```

:::

## Advanced Output Patterns

### Multiple Output Tables

Some operators need to return multiple related datasets:

::: {.panel-tabset}

### R

```r
# Generate multiple output tables
summary_stats <- ctx %>%
  select(.ri, .ci, .y) %>%
  summarise(
    overall_mean = mean(.y, na.rm = TRUE),
    overall_sd = sd(.y, na.rm = TRUE)
  ) %>%
  ctx$addNamespace()

row_stats <- ctx %>%
  select(.ri, .y) %>%
  group_by(.ri) %>%
  summarise(row_mean = mean(.y, na.rm = TRUE)) %>%
  ctx$addNamespace()

col_stats <- ctx %>%
  select(.ci, .y) %>%
  group_by(.ci) %>%
  summarise(col_mean = mean(.y, na.rm = TRUE)) %>%
  ctx$addNamespace()

# Save multiple tables
ctx$save(list(summary_stats, row_stats, col_stats))
```

### Python

```python
# Generate multiple output tables
summary_stats = (
    tercenCtx
    .select(['.y'], df_lib="polars")
    .select([
        pl.col('.y').mean().alias('overall_mean'),
        pl.col('.y').std().alias('overall_sd')
    ])
)

row_stats = (
    tercenCtx
    .select(['.ri', '.y'], df_lib="polars")
    .group_by(['.ri'])
    .agg([pl.col('.y').mean().alias('row_mean')])
)

col_stats = (
    tercenCtx
    .select(['.ci', '.y'], df_lib="polars")
    .group_by(['.ci'])
    .agg([pl.col('.y').mean().alias('col_mean')])
)

# Add namespaces and save
summary_stats = tercenCtx.add_namespace(summary_stats)
row_stats = tercenCtx.add_namespace(row_stats)
col_stats = tercenCtx.add_namespace(col_stats)

tercenCtx.save([summary_stats, row_stats, col_stats])
```

:::

### Working with Factor Variables

When your projection includes factors (categorical variables), incorporate them into your analysis:

::: {.panel-tabset}

### R

```r
# Include factors in analysis
ctx <- tercenCtx()

# Get factor columns
factors <- ctx$rselect()  # Get all factors

result <- ctx %>%
  select(.ri, .ci, .y) %>%
  left_join(factors, by = c(".ri", ".ci")) %>%
  group_by(.ri, .ci, factor_column) %>%  # Include relevant factors
  summarise(
    group_mean = mean(.y, na.rm = TRUE),
    .groups = "drop"
  ) %>%
  ctx$addNamespace()

ctx$save(result)
```

### Python

```python
# Include factors in analysis
factors_df = tercenCtx.rselect(df_lib="polars")

result = (
    tercenCtx
    .select(['.ri', '.ci', '.y'], df_lib="polars")
    .join(factors_df, on=['.ri', '.ci'], how='left')
    .group_by(['.ri', '.ci', 'factor_column'])
    .agg([pl.col('.y').mean().alias('group_mean')])
)

result = tercenCtx.add_namespace(result)
tercenCtx.save(result)
```

:::

## Specialized Output Types

### File Output

Tercen operators can generate and output files (plots, reports, data exports) that users can download or view directly in the interface. This is particularly useful for visualization operators, report generators, and data export tools.

::: {.callout-note}
## File Output Use Cases
- **Plots and Visualizations**: PNG, PDF, SVG graphics
- **Reports**: HTML, PDF documents with analysis results
- **Data Exports**: CSV, Excel files with processed data
- **Configuration Files**: JSON, YAML files for downstream tools
:::

See the [Patterns for Plot Operators](../04-advanced-topics/1-patterns-plots.qmd) chapter for a detailed tutorial on how to output files in Tercen.

### Relations Output

Relations in Tercen support complex data linking and joining tables. This is useful for operators that need to create complex relationships between different data dimensions.

::: {.callout-tip}
## When to Use Relations - Some examples
- PCA analysis with loadings and scores
- Clustering with cluster assignments and centroids  
- Complex statistical models with multiple output components
:::

Key relation functions:
- **`as_relation()`**: Convert data frames to relations
- **`left_join_relation()`**: Join relations together
- **`save_relation()`**: Save relations to Tercen
- **`as_join_operator()`**: Create join operators for complex relationships

::: {.panel-tabset}

### R

```r
# Example: Simple relation output
library(tibble)

# Create a relation with results
result_relation <- tibble(
  component = c("PC1", "PC2", "PC3"),
  variance_explained = c(0.45, 0.32, 0.15),
  eigenvalue = c(4.5, 3.2, 1.5)
) %>%
  ctx$addNamespace() %>%
  as_relation()

# Save relation
ctx$save_relation(result_relation)
```

### Python

```python
import polars as pl

# Create a relation with results
result_data = pl.DataFrame({
    'component': ['PC1', 'PC2', 'PC3'],
    'variance_explained': [0.45, 0.32, 0.15],
    'eigenvalue': [4.5, 3.2, 1.5]
})

result_relation = tercenCtx.add_namespace(result_data)
result_relation = tercenCtx.as_relation(result_relation)

# Save relation
tercenCtx.save_relation(result_relation)
```

:::

## Advanced Input Patterns

### Reading Project Files

Sometimes operators need to access additional files stored in the same project:

::: {.panel-tabset}

### R

```r
# Get workflow and project information
workflow <- ctx$context$client$workflowService$get(ctx$context$workflowId)
project_id <- ctx$schema$projectId

# Find project files
project_files <- ctx$client$projectDocumentService$findProjectObjectsByFolderAndName(
  c(project_id, "ufff0", "ufff0"),
  c(project_id, "", ""),
  useFactory = FALSE,
  limit = 25000
)

# Find specific file
target_file <- "config.csv"
file_names <- sapply(project_files, function(f) f$name)
file_index <- which(grepl(target_file, file_names))[1]

if (!is.na(file_index)) {
  pf <- project_files[[file_index]]
  
  # Download and read file
  response <- ctx$context$client$fileService$download(pf$id)
  file_content <- response$read()
  
  # Process as needed
  if (is.raw(file_content)) {
    file_content <- rawToChar(file_content)
  }
  
  # Use file content in analysis...
}
```

### Python

```python
# Get project information
project_id = tercenCtx.schema.projectId

# Find project files
project_files = tercenCtx.client.projectDocumentService.findProjectObjectsByFolderAndName(
    [project_id, "ufff0", "ufff0"],
    [project_id, "", ""], 
    useFactory=False, 
    limit=25000
)

# Find specific file
target_file = 'config.csv'
fnames = [f.name for f in project_files]
matching_files = [i for i, name in enumerate(fnames) if target_file in name]

if matching_files:
    pf = project_files[matching_files[0]]
    
    # Download and read file
    resp = tercenCtx.context.client.fileService.download(pf.id)
    file_content = resp.read()
    
    # Process as needed
    if isinstance(file_content, bytes):
        file_content = file_content.decode('utf-8')
    
    # Use file content in analysis...
```

:::

::: {.callout-warning}
## Best Practice
Avoid manual file retrieval when possible. Instead, include files directly in the workflow input projection for better reproducibility and user experience.
:::


## Large Tables: the 1.6 Million Row Limit

A single `select` call is capped server-side at **1,600,000 rows**. Ask for more and the call
is rejected with:

```
400 {error: table.limit, reason: Bad limit}
```

This is a hard limit, not a warning, and it is easy to hit by accident: a step whose
projection is only a few thousand rows wide in the UI can still produce millions of rows in
the underlying main table once cells are multiplied out.

::: {.callout-important}
## A rejected select can look like a hang, not an error

If your operator issues one unbounded `ctx$select(...)` and the table is over the limit, the
step may appear to **load forever** rather than fail with a visible message — the error is
returned to the caller, but an operator (or web app) that does not surface it simply never
finishes. "It hangs on a big dataset but works on a small one" is the signature of this
limit.
:::

### The limit applies to R, but not to the Python client

The two clients reach different server endpoints, and only one of them is capped:

| Client | Call | Behaviour |
|---|---|---|
| R (`teRcen`) | `ctx$select()` | Capped at 1.6M rows — **you must paginate** |
| Python (`tercen_python_client`) | `ctx.select()` | Routes to the streaming endpoint, which chunks internally — no 1.6M cap |

So the pagination below is required in R. In Python, `select()` already streams and you can
request the whole table; paginate there only if you want to bound peak memory.

### Paginating a large select in R

Read the row count from the schema, then loop in chunks that stay under the cap. Note the
third argument is `nr` (number of rows), not `limit`:

```r
library(tercen)
library(dplyr)

ctx <- tercenCtx()

n_rows <- ctx$schema$nRows   # total rows available in the main table
chunk  <- 1000000L           # comfortably under the 1.6M cap

pages  <- list()
offset <- 0L

while (offset < n_rows) {
  nr <- min(chunk, n_rows - offset)
  pages[[length(pages) + 1L]] <- ctx$select(
    c(".ci", ".ri", ".y"),
    offset = offset,
    nr     = nr
  )
  offset <- offset + nr
}

data <- bind_rows(pages)
```

The same `offset` / `nr` arguments exist on `ctx$cselect()` and `ctx$rselect()`. Those
address the column and row *annotation* tables, which are almost always far smaller than the
main table — in practice only the main `select` needs paginating.

::: {.callout-tip}
## Select only the columns you need

`ctx$select()` with no `names` argument returns **every** column. Naming just the columns you
use (e.g. `c(".ci", ".ri", ".y")`) reduces transferred data and memory substantially, and is
worth doing regardless of table size.
:::

::: {.callout-note}
## Prefer operating per cell where you can

If your computation is independent per cell or per column, the idiomatic Tercen approach —
grouping by `.ci` / `.ri` and letting Tercen hand you one group at a time, as shown earlier
in this chapter — avoids materialising the whole table and sidesteps this limit entirely.
Pagination is for the cases that genuinely need the full table at once.
:::


## Next Steps

With these input and output patterns mastered, you can:

1. **Create Complex Operators**: Combine multiple patterns for sophisticated analyses
2. **Handle Edge Cases**: Build robust operators that gracefully handle data issues  
3. **Optimize Performance**: Use efficient data processing techniques
4. **Integrate with Workflows**: Design operators that work seamlessly in Tercen pipelines

The next chapter covers continuous integration and deployment strategies for your operators.