# MPC Indexer Breakout
This document outlines the design and efforts for breaking out the indexer into its own crate.
## Background
### Current State
The MPC node relies heavily on the NEAR blockchain for coordination. It fetches the following information from chain:
- pending signature and CKD requests (read from non-finalized block state)
- MPC protocol state (read from contract-state)
- Node migration instructions and data
- Foreign Chain Transaction Data
- TEE related information, such as the current docker image hashes of launcher and node
Additionally, an MPC node writes data to chain, such as:
- responses to signature and CKD requests
- confirmations during critical operations such as key generation and resharing (key events)
- its own TEE attestation
- migration confirmation
The MPC nodes monitor and interact with the NEAR blockchain by spawning a neard node in the same process (but in a different thread, with its own tokio runtime, c.f. [`spawn_real_indexer`](https://github.com/near/mpc/blob/dacc610b92b8ef4d80b389d86e450a3488ae72ed/crates/node/src/indexer/real.rs#L49).
The part of the code that is responsible for spawning the neard node, fetching data from it and forwarding transactions to it, is what we currently refer to as _MPC Indexer_.
Conceptually, the following graph depicts information flow:
```mermaid
---
title: MPC Orchestration
---
flowchart TB
CORE[MPC Core]
subgraph INDEXER[MPC Indexer]
direction TB
WRITE[
MPC State Write
Signature Responses
CKD Responses
TEE Attestation
Migration
Foreign Chain Transactions
]
VIEW[
MPC State View
Protocol State
Key Events
Migration Info
Foreign Chain Policy
TEE Allowed Images
Signature Requests
CKD Requests
]
end
subgraph NEAR[NEAR Blockchain]
direction LR
subgraph MEMPOOL[NEAR Non-finalized blocks]
direction LR
STREAMER[
NEAR Block Stream
Non-finalized Transactions
]
end
subgraph CONTRACT[MPC Smart Contract]
direction TB
CONTRACT_WRITE[
Write Methods
Signature Responses
respond
CKD Responses
respond_ckd
Key Events
start_keygen_instance
vote_pk
start_reshare_instance
vote_reshared
vote_abort_key_event_instance
Foreign Chain Transactions
vote_foreign_chain_policy
respond_verify_foreign_tx
TEE Attestation
submit_participant_info
verify_tee
Migration
conclude_node_migration
]
CONTRACT_VIEW[
Read Methods
Protocol State
state
Migration Info
migration_info
Foreign Chain Policy
get_foreign_chain_policy
TEE Allowed Images
allowed_docker_image_hashes
allowed_launcher_compose_hashes
get_tee_accounts
]
end
end
CORE --> VIEW
CORE --> WRITE
%% Indexer --> NEAR Blockchain
VIEW --> CONTRACT_VIEW
VIEW -->STREAMER
%%|Monitor Blocks for
Signature & CKD Requests| STREAMER
WRITE --> CONTRACT_WRITE
%% ------------------------
%% Styling
%% ------------------------
classDef core stroke:#1b5e20,stroke-width:4px;
classDef indexer stroke:#2563eb,stroke-width:4px;
classDef near stroke:#7c3aed,stroke-width:4px;
classDef contract stroke:#d97706,stroke-width:2px;
classDef mempool stroke:#0f766e,stroke-width:2px;
class CORE core;
class INDEXER indexer;
class NEAR near;
class CONTRACT contract;
class MEMPOOL mempool;
```
The MPC Indexer currently offers the following API:
```rust
pub struct IndexerAPI {
/// Provides the current contract state as well as updates to it.
pub contract_state_receiver: watch::Receiver,
/// Provides block updates (signature requests and other relevant receipts).
/// It is in a mutex, because the logical "owner" of this receiver can
/// change over time (specifically, when we transition from the Running
/// state to a Resharing state to the Running state again, two different
/// tasks would successively "own" the receiver).
/// We do not want to re-create the channel, because while resharing is
/// happening we want to buffer the signature requests.
pub block_update_receiver: Arc>>,
/// Handle to transaction processor.
pub txn_sender: TransactionSender,
/// Watcher that keeps track of allowed [`DockerImageHash`]es on the contract.
pub allowed_docker_images_receiver: watch::Receiver>,
/// Watcher that keeps track of allowed [`LauncherDockerComposeHash`]es on the contract.
pub allowed_launcher_compose_receiver: watch::Receiver>,
/// Watcher that tracks node IDs that have TEE attestations in the contract.
pub attested_nodes_receiver: watch::Receiver>,
pub my_migration_info_receiver: watch::Receiver,
pub foreign_chain_policy_reader: ForeignChainPolicyReader,
}
```
##### Issues with the current design
Currently, the MPC indexer tries to achieve two things:
1. Interacting with the neard node:
- spawning the neard node
- generating and forwarding of transactions
- viewing contract state
- monitoring blocks and filtering for relevant events (signature requests, ckd requests, ...)
2. Providing context to the MPC network:
- Acting as an abstraction layer for the blockchain, ensuring no blockchain-internals are leaked to the node;
- informing the MPC node about jobs such as resharings, signature requests, ckd requests, etc.
- informing the MPC node about relevant TEE information such as allowed docker images etc.
The main concern with our current implementation is the lack of separation between abstraction layer 1 (neard-node-abstraction) and abstraction layer 2 (MPC orchestration). We would like to re-use the chain-specific abstraction layer for our backup service implementation (c.f.[#1891](https://github.com/near/mpc/issues/1891)), as well as the HOT wallet TEE application (c.f. [#2062](https://github.com/near/mpc/issues/2062)), but this is not possible with the current design.
Additionally, the second abstraction layer is not enforced coherently. Looking at the `IndexerAPI` above, it reads pretty much verbatim like the corresponding contract endpoints and the `TransactionSender` trait is too low-level.
Overall, this interface makes testing the node non-trivial, as evidenced by our `FakeIndexer` implementation.
### Design Goals
An improved indexer design should achieve the following goals:
1. Re-usability of components:
- We have planned works for other applications that would greatly profit from one or multiple functionalities:
- The TEE backup service will need to monitor the same MPC smart contract, requiring viewing state and submitting transactions (c.f.[#1891](https://github.com/near/mpc/issues/1891);
- The HOT wallet TEE application will want to monitor a different smart contract, but require similar methods (viewing Attestations and docker image hashes, c.f. [#2062](https://github.com/near/mpc/issues/2062));
- We could leverage parts of the MPC indexer to monitor our production deployments;
2. Improved testing:
- we currently have a `FakeIndexer` to isolate the core logic of our node. This mostly achieves its goal, but if we had a more mature interface, we could increase test coverage;
3. Isolation of nearcore internals:
- This ties into point 1., but is worth mentioning as a stand-alone goal. The interface exposed by the neard node has experienced breaking changes in the past and due to the usage of nearcore internals in our code, we have no guarantee that this won't happen again. Since we will require this functionality in other applications, it makes sense to have a stand-alone crate that has an isolated dependency on the nearcore internals. This way, in case we do experience breaking changes, we only need to fix them once.
### Design Proposal
We propose to split the two functionalities of the current indexer (MPC orchestration and Chain Indexing) into two separate components:
The **Chain Gateway**:
This component is responsible for:
- spinning-up a neard node;
- abstracting the neard indexer interface such that no nearcore internals are exposed.
- providing a convenient interface to:
- call arbitrary view methods on arbitrary contracts
- subscribe to view methods of contracts
- forward transactions to the NEAR blockchain
- monitor non-finalized blocks for transactions matching a user-specified pattern;
This is the first step and primary goal. This abstraction is a huge enabler for migrating the backup service into a TEE [(#1891)](https://github.com/near/mpc/issues/1891) and for our long-term support of legacy keys [(#2062)](https://github.com/near/mpc/issues/2062)
As a secondary goal, we propose the **MPC Context**.
This component is responsible for informing the MPC node about:
- pending jobs (signature, CKD, foreign chain verification requests)
- network state (peers)
- protocol state (resharing)
```mermaid
---
title: MPC Orchestration
---
flowchart TB
CORE[MPC Node Core]
subgraph CONTEXT[MPC Context]
direction TB
WRITE[
MPC State Write
]
VIEW[
MPC State View
]
WRITE -.->|verify success| VIEW
end
subgraph CHAIN[Chain Gateway]
direction TB
TX_SUBSCRIBER[**Block Event Subscriber**
**Filters** non-finalized NEAR blocks for specific transactions
**Returns** matching args in a stream
]
CONTRACT_STATE_VIEWER[**Contract State Subscriber**
**Queries** view functions of smart contracts
**Returns** the result to the MPC Context
]
TX_SENDER[**Transaction Sender**
**Submits** transaction to the neard node
**Returns** transaction hash
]
subgraph NEARD[**Neard node**]
direction TB
BLOCK_STREAMER[**Streamer**]
VIEW_CLIENT[**View Client**]
RPC_HANDLER[**RPC Handler**]
end
end
subgraph NEAR[NEAR Blockchain]
direction TB
subgraph MEMPOOL[NEAR Mempool]
end
subgraph CONTRACT[MPC Smart Contract]
direction TB
CONTRACT_VIEW[
Read Methods
]
CONTRACT_WRITE[
Write Methods
]
end
end
%% CORE --> Context
CORE --> VIEW
CORE --> WRITE
%% Context --> Chain Gateway
VIEW --> CONTRACT_STATE_VIEWER
VIEW --> TX_SUBSCRIBER
WRITE --> TX_SENDER
%% Chain Gateway --> Neard Node
TX_SUBSCRIBER --> BLOCK_STREAMER
CONTRACT_STATE_VIEWER --> VIEW_CLIENT
TX_SENDER --> RPC_HANDLER
%% Neard --> Smart Contract
RPC_HANDLER --> MEMPOOL
MEMPOOL -.-> CONTRACT_WRITE
VIEW_CLIENT --> CONTRACT_VIEW
BLOCK_STREAMER --> MEMPOOL
%% ------------------------
%% Styling
%% ------------------------
classDef core stroke:#1b5e20,stroke-width:4px;
classDef indexer stroke:#2563eb,stroke-width:4px;
classDef near stroke:#7c3aed,stroke-width:4px;
classDef contract stroke:#d97706,stroke-width:2px;
classDef mempool stroke:#0f766e,stroke-width:2px;
classDef chain stroke-width:2px;
class CORE core;
class CONTEXT indexer;
class NEAR near;
class CONTRACT contract;
class MEMPOOL mempool;
class CHAIN chain;
```
### Crate Dependencies
Below is a graph depicting dependencies of the envisioned crate dependencies as it relates to NEAR indexer functionality.
```mermaid
---
title: MPC Dependencies
---
flowchart TB
subgraph SERVICES[MPC Services]
direction TB
subgraph MPC_NODE[MPC Node]
direction TB
CORE[MPC Node Core]
CONTEXT[MPC Context]
CORE --> CONTEXT
end
BACKUP_SERVICE[MPC Backup and Migration Service]
HOT_SERVICE[HOT MPC Service]
end
subgraph CHAIN[Chain Gateway]
direction TB
TX_SUBSCRIBER[**Block Event Subscriber**
**Filters** non-finalized NEAR blocks for specific transactions
**Returns** matching args in a stream
]
CONTRACT_STATE_VIEWER[**Contract State Subscriber**
**Queries** view functions of smart contracts
**Returns** stream for state
]
TX_SENDER[**Transaction Sender**
**Submits** transaction to the neard node
**Returns** transaction result
]
end
subgraph NEAR[NEAR Blockchain]
end
CHAIN --> NEAR
CONTEXT --> CONTRACT_STATE_VIEWER
CONTEXT --> TX_SENDER
CONTEXT --> TX_SUBSCRIBER
BACKUP_SERVICE --> CONTRACT_STATE_VIEWER
BACKUP_SERVICE --> TX_SENDER
HOT_SERVICE --> CONTRACT_STATE_VIEWER
HOT_SERVICE --> TX_SENDER
```
### API Proposal
In this section, we propose API designs for the Chain Gateway and MPC Context.
#### Chain Gateway
The Chain Gateway provides three functionalities:
- **State viewing:** allows to:
- subscribe to arbitrary view methods on arbitrary contracts on the NEAR blockchain
- query arbitrary view methods on arbitrary contracts on the NEAR blockchain
- **Block Events:** Filter the mempool for transactions matching a specific pattern (receipient or executor id and method names). Receive a stream of all matching transactions.
- **Transaction Sender:** send transactions to the NEAR blockchain.
##### State Viewer
The Chain Gateway offers the following traits for viewing and subscribing to contract state:
```rust
/// One-shot typed view call with JSON serialization/deserialization.
pub trait ViewMethod: ViewRaw {
async fn view(
&self, contract_id: AccountId, method_name: &str, args: &Arg,
) -> Result, ChainGatewayError>;
}
/// Polls every 200ms; emits change only when returned bytes differ.
pub trait SubscribeContractState: ViewRaw + Clone {
async fn subscribe(
&self, contract: AccountId, view_method: &str,
) -> impl WatchContractState + Send;
}
pub trait WatchContractState {
/// Returns the last observed value and the block height at which it was observed.
fn latest(&mut self) -> Result, ChainGatewayError>;
/// Waits until the observed value changes.
async fn changed(&mut self) -> Result<(), ChainGatewayError>;
}
pub struct ObservedState> {
pub observed_at: BlockHeight,
pub value: T,
}
/// Empty arguments for view calls that take no parameters.
pub struct NoArgs {}
pub struct BlockHeight(u64);
```
Note that the above traits derive from `ViewRaw`, which in turn derives from two low-level traits.
```rust
/// Waits for sync then delegates to QueryViewFunction.
/// Supertraits provide the raw RPC plumbing.
pub trait ViewRaw: IsSyncing + QueryViewFunction {
async fn view_raw(
&self,
contract_id: &AccountId,
method_name: &str,
args: &[u8],
) -> Result;
}
// queries the actual state
pub trait QueryViewFunction: Send + Sync + 'static {
async fn view_function_query(
&self,
contract_id: &AccountId,
method_name: &str,
args: &[u8],
) -> Result;
}
// returns true if the node is still syncing with the blockchain
pub trait IsSyncing: Send + Sync + 'static {
/// Returns whether the node is currently syncing.
async fn is_syncing(&self) -> Result;
}
```
##### Block Event Subscriber
The purpose of this interface is to enable easy subscription to block events. In the MPC node, we use this to monitor the mempool for requests to the MPC network and responses from the MPC network.
Specifically, we filter for receipts that match one of the following pattern:
- They are executed on a specific contract, call a specific method of that contract and successfully spawn a promise. We will call this `ExecutorFunctionCallSuccessWithPromise`:
- in case of our MPC node, we are looking for any calls to `sign`, `request_app_private_key` or `verify_foreign_chain_transaction` of our MPC contract.
- They are addressed to a specific contract and call a specific method of that contract. We call those `ReceiverFunctionCall`:
- in the case of our MPC node, we are looking for calls to `return_signature_and_clean_state_on_success`, `return_ck_and_clean_state_on_success` or `return_verify_foreign_tx_and_clean_state_on_success` that originate from the contract.
If we want this interface to be re-usable in other parts of our code, we can create a more or less generic filter interface:
```rust
impl BlockEventSubscriptions {
/// Create a new subscriber with the given channel buffer size.
pub fn new(buffer_size: usize) -> Self;
/// Add a subscription and get a unique identifier for it.
/// Can be called multiple times before passing the subscriber to `ChainGateway::start()`.
/// The returned identifier can be used to match returned events to the given subscription.
pub fn subscribe(&mut self, filter: BlockEventSubscription) -> BlockEventId;
}
/// An identifier for a subscription, returned by `subscribe()`.
pub struct BlockEventId(pub u64);
pub enum BlockEventSubscription {
/// Filter for events where a receipt outcome was executed by `transaction_outcome_executor_id` and called `method_name`.
ExecutorFunctionCallSuccessWithPromise {
transaction_outcome_executor_id: AccountId,
method_name: String,
},
/// Filter for events where a receipt was addressed to `receipt_receiver_id` and called `method_name`.
ReceiverFunctionCall {
receipt_receiver_id: AccountId,
method_name: String,
},
}
```
> **Note:** Block replay from a specific height (`SubscriptionReplay`) is planned but not yet implemented. Replay leverages the NEAR indexer's `sync_from_block_height` config option and comes essentially for free in the chain-gateway implementation — we just need to expose it through the `BlockEventSubscriptions` API. See [#236](https://github.com/near/mpc/issues/236).
The subscriber is passed to `ChainGateway::start()`, which returns an `Option>`:
```rust
let (chain_gateway, node_handle, block_update_receiver) =
ChainGateway::start(indexer_config, Some(subscriber)).await?;
```
Example usage:
```rust
let mut subscriber = BlockEventSubscriptions::new(100);
let signature_requests_id = subscriber.subscribe(
BlockEventSubscription::ExecutorFunctionCallSuccessWithPromise {
transaction_outcome_executor_id: "v1.signer".parse()?,
method_name: "sign".to_string(),
}
);
let ckd_request_id = subscriber.subscribe(
BlockEventSubscription::ExecutorFunctionCallSuccessWithPromise {
transaction_outcome_executor_id: "v1.signer".parse()?,
method_name: "request_app_private_key".to_string(),
}
);
let (chain_gateway, node_handle, block_update_receiver) =
ChainGateway::start(indexer_config, Some(subscriber)).await?;
let mut block_stream_receiver = block_update_receiver.unwrap();
while let Some(update) = block_stream_receiver.recv().await {
for matched in update.events {
match matched.id {
id if id == signature_requests_id => { /* handle signature request */ }
id if id == ckd_request_id => { /* handle ckd request */ }
_ => {}
}
}
}
```
Specific types (c.f. [Appendix](#current-block-update) and `indexer/handler.rs` for justification).
```rust
/// The BlockUpdate returned by the Chain indexer. Similar to the current `BlockUpdate`
pub struct BlockUpdate {
pub context: BlockContext,
pub events: Vec,
}
/// Context for a single block
pub struct BlockContext {
pub hash: CryptoHash,
pub height: BlockHeight,
pub prev_hash: CryptoHash,
pub last_final_block: CryptoHash,
pub block_entropy: CryptoHash,
pub block_timestamp_nanosec: u64,
}
pub struct MatchedEvent {
/// Identifies which subscription matched this event.
pub id: BlockEventId,
/// Data associated with the event.
pub event_data: EventData,
}
/// This can be extended if required.
pub enum EventData {
ExecutorFunctionCallSuccessWithPromise(ExecutorFunctionCallSuccessWithPromiseData),
ReceiverFunctionCall(ReceiverFunctionCallData),
}
/// This event is associated to a transaction that matched a specific (transaction_outcome_executor_id: AccountId, method_name: String) pattern.
struct ExecutorFunctionCallSuccessWithPromiseData {
/// the receipt_id of the receipt this event came from
receipt_id: CryptoHash,
/// predecessor_id who signed the transaction
predecessor_id: AccountId,
/// the receipt that will hold the outcome of this receipt
next_receipt_id: CryptoHash,
/// raw bytes used for function call
args_raw: Vec,
}
/// This event is associated to a transaction that matched a specific BlockEventSubscription.
struct ReceiverFunctionCallData {
/// the receipt id for the matched transaction
receipt_id: CryptoHash,
}
```
Note that this will be subject to changes in [#2680](https://github.com/near/mpc/issues/2680)
##### Transaction Sender
We propose the following API for the transaction sender:
```rust
/// Default impl fetches the latest final block, signs, and submits.
pub trait SubmitFunctionCall:
FetchLatestFinalBlockInfo + SubmitSignedTransaction
{
async fn submit_function_call_tx(
&self,
signer: Arc,
receiver_id: AccountId,
method_name: String,
args: Vec,
gas: Gas,
) -> Result;
}
```
`TransactionSigner` handles nonce management and ED25519 signing:
```rust
pub struct TransactionSigner {
signing_key: SigningKey,
account_id: AccountId,
nonce: Mutex,
}
impl TransactionSigner {
pub fn from_key(account_id: AccountId, signing_key: SigningKey) -> Self;
pub fn public_key(&self) -> VerifyingKey;
}
```
#### MPC Context
TODO(#2138): handle in a separate discussion. This is not of priority right now.
## Appendix
### Current Block Update
Currently, the MPC indexer monitors the mempool for transactions of interest to the MPC node.
It does so by filtering all receipts for:
1. signature and CKD requests, as well as foreign transaction verifications. We do so by:
1. matching the `executor_id` of a receipt with the MPC contracts `AccountId`. This means that this receipt is executed by the MPC contract;
2. matching the method called in the contract to one of the expected methods (in the case of the MPC network, we are looking for calls to `sign`, `request_app_private_key` and `verify_foreign_transaction`).
3. if we have a match, we are interested in the following data:
- receipt id
- predecessor id
- next receipt id
- arguments for the function call (note: we currently deserialize the json and return a type).
- block entropy
- block timestamp
2. signature responses, CKD responses and foreign transaction verification responses. We do so by:
1. matching on the `receiver_id` of the receipt (must match the contract);
2. matching the method called in the contract to one of the expected values (`return_signature_and_clean_state_on_success`, `return_ck_and_clean_state_on_success` or `return_verify_foreign_tx_and_clean_state_on_success`, for the MPC contract).
3. In case we match, we track the following data:
- receipt id (this will match the _next receipt id_ of the corresponding request from point 3 above)
_Open Question: we should figure out if `receipt_id` and `executor_id` can be different from one another. If not, then we could simplify our design slightly_
For each block, the indexer composes a `BlockUpdate` for all of the above and sends that to the MPC node, together with some information about the block, such as:
- block height
- block hash
- previous block hash
```rust
/// This is a block update - containing all matched events for the latest block
pub struct ChainBlockUpdate {
pub block: BlockViewLite,
pub signature_requests: Vec,
pub completed_signatures: Vec,
pub ckd_requests: Vec,
pub completed_ckds: Vec,
pub verify_foreign_tx_requests: Vec,
pub completed_verify_foreign_txs: Vec,
}
pub struct BlockViewLite {
pub hash: CryptoHash,
pub height: u64,
pub prev_hash: CryptoHash,
pub last_final_block: CryptoHash,
}
```
## Related issues
https://github.com/near/mpc/issues/1956
https://github.com/near/mpc/issues/592
https://github.com/near/mpc/issues/950
https://github.com/near/mpc/issues/913
https://github.com/near/mpc/issues/1187
https://github.com/near/mpc/issues/439
https://github.com/near/mpc/issues/1957
https://github.com/near/mpc/issues/155
https://github.com/near/mpc/issues/236
potentially:
https://github.com/near/mpc/issues/1643