# R2D2

> Revaud 2019 · [Paper](https://arxiv.org/abs/1906.06195)

**One-line summary** — Jointly learns explicit *repeatability* and *reliability* maps alongside dense descriptors, so keypoints are only detected where they can actually be matched.

## Problem

Classical local features follow a detect-then-describe paradigm: hand-crafted detectors find repeatable keypoints, then a separate descriptor represents them. Learned methods had caught up — repeatable saliency maps for detection, metric-learning descriptors at those locations — but R2D2 argues this conflates two different properties. Salient regions are not necessarily *discriminative*: every square of a checkerboard, every window on a facade, tree leafage, sea waves are detected consistently yet match ambiguously due to self-similarity. Descriptors should only be trusted (and keypoints only detected) where matching can be performed with high confidence.

## Method & architecture

**Three dense outputs from one FCN.** For an $H \times W$ image the network predicts: dense L2-normalized 128-D descriptors $\boldsymbol{X}$, a repeatability heatmap $\boldsymbol{S}\in[0,1]^{H\times W}$ whose local maxima are keypoint candidates, and a reliability map $\boldsymbol{R}\in[0,1]^{H\times W}$ estimating each descriptor's discriminativeness. Backbone is L2-Net with the final 8x8 convolution replaced by three 2x2 convolutions (5x fewer weights); $\boldsymbol{S}$ and $\boldsymbol{R}$ come from an element-wise square, a 1x1 convolution and a softmax on the 128-D output.

**Repeatability: self-supervised cosine similarity.** Given images $I, I'$ with dense ground-truth correspondences $U$ and $\boldsymbol{S}'_U$ the second heatmap warped by $U$, the maps are aligned patchwise over all $N\times N$ patches $\mathcal{P}$:

$$\mathcal{L}_{cosim}(I,I',U)=1-\frac{1}{|\mathcal{P}|}\sum_{p\in\mathcal{P}}cosim\bigl(\boldsymbol{S}\left[p\right],\boldsymbol{S}'_{U}\left[p\right]\bigr)$$

A peakiness loss prevents the trivial constant solution and sets keypoint density via $N$:

$$\mathcal{L}_{peaky}(I)=1-\frac{1}{|\mathcal{P}|}\sum_{p\in\mathcal{P}}\Bigl(\max_{(i,j)\in p}\boldsymbol{S}_{ij}-\mathrm{mean}_{(i,j)\in p}\boldsymbol{S}_{ij}\Bigr)$$

combined as $\mathcal{L}_{rep}=\mathcal{L}_{cosim}+\lambda\left(\mathcal{L}_{peaky}(I)+\mathcal{L}_{peaky}(I')\right)$.

**Reliability: AP ranking loss with a learned confidence.** Descriptor matching is treated as a *ranking* problem: for each query pixel, a differentiable approximation of Average Precision over candidates in the second image is optimized directly, rather than a triplet/contrastive surrogate. Crucially, the network may opt out on undistinctive regions:

$$\mathcal{L}_{AP\kappa}(i,j)=1-\bigl[AP(i,j)\,\boldsymbol{R}_{ij}+\kappa\,(1-\boldsymbol{R}_{ij})\bigr]$$

where $\kappa=0.5$ is the minimum expected AP: the optimum is $\boldsymbol{R}_{ij}=0$ where $AP(i,j)<\kappa$ and 1 otherwise, so $\boldsymbol{R}$ becomes an interpretable matchability probability.

**Training data & inference.** Dense correspondences come free from random homographies on web images, and from an optical-flow pipeline (EpicFlow with epipolar-constrained DeepMatching, masked by match-density) on SfM-verified Aachen image pairs — no manual labels. At test time the network runs over a scale pyramid (downsampling by $2^{1/4}$), keypoints are local maxima of $\boldsymbol{S}$, and the top $K$ are kept by the combined score $\boldsymbol{S}_{ij}\boldsymbol{R}_{ij}$.

## Results

- **Ablation (HPatches)**: full model reaches M-score 0.461 / MMA@3 0.686; dropping the repeatability map collapses performance to 0.304 / 0.512 — repeatability and reliability are genuinely different signals; dropping reliability costs 3% M-score.
- **HPatches MMA**: outperforms SuperPoint, D2-Net (multi-scale), LF-Net, HAN+HN++ and Hessian-affine+RootSIFT at mid-range thresholds; only DELF wins on illumination-only pairs (its fixed keypoint grid trivially matches images with no spatial change). M-score 0.425 at 3px vs 0.335 for LF-Net and 0.288 for SIFT.
- **Detector repeatability (Oxford)**: e.g. on *wall* (viewpoint), 0.62-0.71 vs QuadNet 0.30-0.46 and DoG 0.27-0.28 across 300-3000 points.
- **Aachen Day-Night localization** (CVPR19 local-feature challenge, COLMAP pipeline): best model (N=8, 10k keypoints, 1M weights) localizes **45.9 / 66.3 / 88.8%** of night queries within (0.25m, 2°) / (0.5m, 5°) / (5m, 10°) — a new record, ahead of D2-Net (44.9/66.3/88.8, 15M weights, 512-D) and SuperPoint (42.8/57.1/75.5); even the 5k-keypoint variant beats most methods with half the keypoints, using only 128-D descriptors and a network up to 15x smaller.

## Why it matters for SLAM

Wrong matches are the dominant failure mode of feature-based SLAM under repetitive structure (corridors, facades, vegetation). R2D2 established the principle of *reliability-aware* keypoint selection — quality over quantity — which carried into later detector/descriptor designs and into how modern pipelines score and filter correspondences for localization and 3D reconstruction.

## Hands-on

- [Deep local feature detection](https://github.com/changh95/slam_lecture_codes/tree/main/SLAM_zero_to_hero/part2_ch01_04)

## Related

- [SuperPoint](superpoint.md) — self-supervised joint detector/descriptor baseline
- [DISK](disk.md) — reinforcement-learning-trained alternative
- [KeyNet](keynet.md) — learned detector line of work
- [DeDoDe](dedode.md) — later decoupling of detection and description
- [hloc](hloc.md) — localization pipeline where such features are evaluated