# Adding custom datasets

## Overview

Developers can add new dataset readers to supplement the defaults in `src/readers`.
This is most useful for pulling data from external databases for entry into the **bakana** workflow.
A new reader should be implemented as an ES6 class that satisfies the `Dataset` interface requirements below.

## `Dataset` interface

### Constructor

Each `Dataset` instance is constructed by users before entry into the **bakana** workflow.
Thus, **bakana** itself does not have any particular expectations on the form of the constructor.
There are, however, a few conventions:

- For consistency with the existing `Dataset` readers, developers of new file-based readers should consider accepting [`SimpleFile`](https://ltla.github.io/bakana/SimpleFile.html) objects.
  This provides a convenient abstraction for files in Node.js and browser contexts.
- Constructors are expected to be cheap to run.
  Any complex initialization should be deferred until it is needed, e.g., in `summary()` or `load()`.
- Optional parameters can be passed via an `options=` object in the constructor.
  Some sensible defaults can be specified via destructuring.
  Setters can be provided to change these options after construction.

### `format()` (static)

This is a static method that returns a string containing the name of the dataset format.
It is used to differentiate between different `Dataset` classes.
The formats implemented in _bakana_ itself are listed below:

- `10X`: for the 10X HDF5 format in `TenxHdf5Dataset`.
- `MatrixMarket`: for the 10X Matrix Market format in `TenxMatrixMarketDataset`.
- `H5AD`: for the H5AD format `H5adDataset`.
- `SummarizedExperiment`: for SummarizedExperiments saved as RDS files in `SummarizedExperimentDataset`.

### `abbreviate()`

This method should _quickly_ return an object containing some unique-ish summary of the files in this dataset.
The object will be used for comparisons between states in the `InputsState.compute()` function.
It should be easily stringified while still being unique enough to distinguish between most other instances of the same class. 
We generally recommend generating a summary based on the file name and size, which is fast and unique enough for most comparisons.
(Note that this only needs to be reasonably unique compared to other instances of the same `Dataset` class.)

### `summary({ cache = false } = {})`

This method should return an object containing an appropriate summary of this dataset.
The object may contain any number of the following properties:

- `all_features`: a [`DataFrame`](https://ltla.github.io/bioconductor.js/DataFrame.html) containing per-feature annotations.
  Each row of the `DataFrame` corresponds to one feature, along with any number of columns containing the annotation fields.
- `modality_features`: an object where each key is the name of a modality and each value is a `DataFrame`.
  Each row of a `DataFrame` corresponds to one feature of the corresponding modality and contains its annotation fields.
  Unlike `load()`, modality names may be arbitrary.
- `cells`: a `DataFrame` containing per-cell annotations, where each row corresponds to a cell in the dataset.
- `all_assay_names`: an Array containing the names of all assays.
- `modality_assay_names`: an object where each key is the name of a modality and each value is an Array of strings.
  Each Array contains the names of the assays for that modality.

Alternatively, this method may return a promise that resolves to such an object.

If `cache = true`, any artifacts generated by `summary()` may be cached for re-use in the subsequent calls to the same method,`load()` or `previewPrimaryIds()`.
This allows for faster repeated calls at the cost of increasing memory usage.
If `cache = false`, no caching should be performed and any previously cached artifacts should be removed.

Note that the return value of `summary()` is expected to be constant for the same `Dataset` instance.
It should not be affected by any setters; rather, setters are expected to be called based on the `summary()` output.
This pattern is typically used to set parameters for subsequent `load()` calls.

### `load({ cache = false } = {})`

This method should return an object containing:

- `matrix`, a [`MultiMatrix`](https://kanaverse.github.io/scran.js/MultiMatrix.html) object containing submatrices for at least one modality.
  Each modality-specific submatrix should be a [`ScranMatrix`](https://kanaverse.github.io/scran.js/ScranMatrix.html) containing any number of rows.
  All modalities should contain data for the same number of columns.
  Unlike `summary()`, modality names should be one of `"RNA"`, `"ADT"` or `"CRISPR"` -
  developers should consider adding some setters to their `Dataset` class to enable users to specify a mapping between the modality names from `summary()` and those returned by this method.
- `primary_ids`, an object specifying the primary feature identities for each submatrix in `matrix`.
  Keys should correspond to the modality names in `matrix` and each value should be an array of strings (or `null`) containing the feature identities of the submatrix's rows.
  These identifiers will be used to find the intersection of features when combining multiple datasets.
- `features`, an object where each key is the name of a modality in `matrix`.
  Each modality-specific value is a `DataFrame` with one row per feature in the corresponding entry of `matrix`.
  Rows of `features[<modality>]` should be in the same order as the rows of `matrix.get(<modality>)`.
  Columns should be per-feature annotation fields, as described for `summary()`.
- `cells`, a `DataFrame` containing one row per cell.   
  Each column should contain an array of per-cell information, corresponding to the same order of columns in each entry of `matrix`.

Alternatively, this method may return a promise that resolves to such an object.

If `cache = true`, any artifacts generated by `load()` may be cached for re-use in the subsequent calls to the same method, `summary()` or `previewPrimaryIds()`.
This allows for faster repeated calls at the cost of increasing memory usage.
If `cache = false`, no caching should be performed and any previously cached artifacts should be removed.

### `previewPrimaryIds({ cache = false } = {})`

This method should return an object containing primary feature identities for each modality.
The output has the same structure as the `primary_ids` property returned by `load()`, but the entries inside each array may be different.
Applications should use the output of this method to compute the intersection of genes across multiple datasets, 
e.g., to report diagnostics to the user regarding their choices of the primary identifiers.

If `cache = true`, any artifacts generated by `summary()` may be cached for re-use in the subsequent calls to the same method, `summary()` or `load()`.
This allows for faster repeated calls at the cost of increasing memory usage.
If `cache = false`, no caching should be performed and any previously cached artifacts should be removed.

### `serialize()`

This method should return an object containing:

- `files`: an Array of objects, where each object represents a data file used to create the count matrix.
  Each object should contain:
  - `type`, a string specifying the type of file.
    This is used to disambiguate multiple files for a single dataset.
  - `file`, a `SimpleFile` object containing the contents of (or a reference to) the underlying data file.
    For database-based datasets, developers may wish to mock up a file containing the database identifier, e.g., as a JSON string. 
- `options`: an object containing optional parameters to be passed to the constructor.

### `unserialize(files, options)` (static)

This is a static method that creates a new instance of the same `Dataset`.

`files` will be an array of objects where object represents a data file used to create the dataset.
Each object will contain `type` and `file` as described for the `serialize()` method.

`options` will be an object containing optional parameters, to be passed to the constructor.

This method should return an instance of the same `Dataset` class.
Presumably this is done using the file contents in `files`.
Alternatively, this method may return a promise that resolves to a `Reader` instance.

### `clear()`

This method should clear all caches generated by `load()` or `summary()` when `cache = true`. 
It may be called at any time and is useful for reducing memory usage in long-running programs.

## Registering new readers

Applications can register custom readers by assigning to the `bakana.availableReaders` object.
The property name should be the name of the dataset format (i.e., the value of `format()`) and the value should be the class definition.
For example, one could re-register the 10X HDF5 reader:

```js
bakana.availableReaders[bakana.TenxHdf5Dataset.format()] = bakana.TenxHdf5Dataset;
```