# NIGHTS perceptual dataset

NIGHTS (Novel Image Generations with Human-Tested Similarities) is a dataset of 20,019 image triplets with human scores of perceptual similarity. Each triplet consists of a reference image and two distortions, located at `ref/xxx/yyy.png`, `distort/xxx/yyy_0.png`, and `distort/xxx/yyy_1.png`. 

Directories are numbered `000` to `099` and files are numbered `000` to `999`, though there is not an image triplet at every number.

## Dataset structure:

```
nights/
├── ref/
│   ├── 000/
│   │   ├── 000.png
│   │   ├── 001.png
│   │   ├── ...
│   │   └── 999.png
│   └──  001/ 002/ ... 099/
├── distort/
│   ├── 000/
│   │   ├── 000_0.png
│   │   ├── 000_1.png
│   │   ├── ...
│   │   ├── 999_0.png
│   │   └── 999_1.png
│   └──  001/ 002/ ... 099/
├── data.csv
└── README.md
```

## Dataset generation details
All data was generated by Stable Diffusion 2.1 [1] by sampling image triplets with a prompt of the same category and different random seed, using the structure:
`An image of a <category>`. The `category` is drawn from image labels in popular datasets - ImageNet [2], CIFAR-10 [3], CIFAR-100 [3], Oxford 102 Flower [4], Food-101 [5], and SUN397 [6].

See `data.csv` for the full list of categories corresponding to each image triplet.

## Limitations
We note that by using Stable Diffusion, our benchmark is exposed to potential biases preexisting and sensitive content in the model. As such, we generate our images with a pre-defined set of categories, while largely avoiding human faces. Our perceptual model is also finetuned from existing pre-trained backbones, and thus may also inherit prior errors and biases.

## References
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-
resolution image synthesis with latent diffusion models. In _Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition_, pages 10684–10695, 2022.

[2] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-
scale hierarchical image database. In _2009 IEEE conference on computer vision and pattern
recognition_, pages 248–255. Ieee, 2009

[3] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

[4] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large
number of classes. In _2008 Sixth Indian Conference on Computer Vision, Graphics & Image
Processing_, pages 722–729. IEEE, 2008.

[5] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. _Food-101–mining discriminative
components with random forests. In Computer Vision–ECCV 2014: 13th European Conference,
Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446–461_. Springer, 2014.

[6] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database:
Large-scale scene recognition from abbey to zoo. In 2*010 IEEE computer society conference
on computer vision and pattern recognition*, pages 3485–3492. IEEE, 2010.