{ "cells": [ { "cell_type": "raw", "metadata": {}, "source": [ "---\n", "aliases:\n", "- /noisy_imagenette\n", "badges: true\n", "branch: master\n", "categories:\n", "- deep learning\n", "- imagenette\n", "description: A noisy version of fastai's Imagenette/Imagewoof datasets\n", "permalink: /noisy_imagenette\n", "title: Introducing Noisy Imagenette\n", "toc: false\n", "\n", "---\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**TL;DR:** We introduce a dataset, Noisy Imagenette, which is a version of the Imagenette dataset with noisy labels. We hope this dataset is useful for rapid experimentation and testing of methods to address noisy label training.\n", "\n", "# Introduction\n", "\n", "## Dataset have noisy labels!\n", "\n", "Deep learning has led to impressive results on datasets of all types, but its success often shines when models are trained with large datasets with human-annotated labels (extreme example: GPT-3 and more recently CLIP/ALIGN/DALL-E). A major challenge when constructing these datasets is obtaining enough labels to train a neural network model. There is an inherent tradeoff between the quality of the annotations and the cost of annotation (in the form of time or money). For example, while using sources like Amazon Mechanical Turk provide cheap labeling, the use of these non-expert labeling services will often produce unreliable labels. This is what is referred to as noisy labels, as these unreliable labels are not necessarily ground truth. Unfortunately, neural networks are known to be susceptible to overfitting to noisy labels (see [here](https://arxiv.org/abs/1611.03530)) which means alternative approaches are needed to achieve good generalization in the presence of noisy labels.\n", " \n", " ## Prior research on noisy labels\n", " \n", " Recently, many techniques have been presented in order to address label noise. These include novel loss functions like [Bi-Tempered Logistic Loss](https://arxiv.org/abs/1906.03361)[Taylor Cross Entropy Loss](https://www.ijcai.org/Proceedings/2020/305), or [Symmetric Cross Entropy](https://arxiv.org/abs/1908.06112). Additionally, there are many novel training techniques that have been recently developed like [MentorMix](https://arxiv.org/abs/1911.09781), [DivideMix](https://arxiv.org/abs/2002.07394), [Early-Learning Regularization](https://arxiv.org/abs/2007.00151) and [Noise-Robust Contrastive Learning](https://openreview.net/forum?id=D1E1h-K3jso). \n", " \n", " Most of these papers are using MNIST, SVHN, CIFAR10 or related datasets with synthetically-added noise. Other common datasets are the WebVision and Clothing1M datasets, which are real-world noisy, large-scale datasets with millions of images. Therefore there is an opportunity to develop a mid-scale dataset that allows for rapid prototyping but is complex enough to provide useful results when it comes to noisy label training.\n", "\n", "## fastai's Imagenette - a dataset for rapid prototyping\n", "\n", "The idea of mid-scale datasets for rapid prototyping has been explored in the past. For example, in 2019, fast.ai [released](https://github.com/fastai/imagenette) the Imagenette and Imagewoof datasets (subsequently updated in 2020), subsets of Imagenet for rapid experimentation and prototyping. It can serve as a small dataset proxy for the ImageNet, or a dataset with more complexity than MNIST or CIFAR10 but still small and simple enough for benchmarking and rapid experimentation. This dataset has been used to test and establish new training techniques like [Mish activation function](https://arxiv.org/abs/1908.08681) and [Ranger optimizer](https://forums.fast.ai/t/meet-ranger-radam-lookahead-optimizer/52886) (see [here](https://forums.fast.ai/t/how-we-beat-the-5-epoch-imagewoof-leaderboard-score-some-new-techniques-to-consider/53453)). The dataset also has been used in various papers (see [here](https://arxiv.org/abs/2004.07629), [here](https://arxiv.org/abs/2007.15248), [here](https://arxiv.org/abs/1906.04887), [here](https://arxiv.org/abs/2101.06639), [here](https://arxiv.org/abs/2006.05624), and [here](https://www.sciencedirect.com/science/article/pii/S1047320321000134?casa_token=uL4_SoQQgKsAAAAA:CPGu3HeZVciBO5YEocTnziH7YVhbcGF0JCpB0JuJi2pqHmkaAKibhaVYe-3t07nxtpdem2lv)). Clearly, this dataset has been quite useful to machine learning researchers and practitioners for testing and comparing new methods. We believe that an analogous dataset could be useful to researchers with modest compute for testing and comparing new methods for addressing label noise." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Introducing Noisy Imagenette\n", "We introduce Noisy Imagenette, a version of Imagenette (and Imagewoof) that has synthetically noisy labels at different levels: 1%, 5%, 25%, and 50% incorrect labels. The Noisy Imagenette dataset already comes with the Imagenette dataset:\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "from fastai.vision.all import *\n", "source = untar_data(URLs.IMAGENETTE)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "While the regular labels for Imagenette dataset are given as the names of the image folder, the noisy labels are provided as a separate CSV file with columns corresponding to the image filename and labels for each of the different noise levels:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
pathnoisy_labels_0noisy_labels_1noisy_labels_5noisy_labels_25noisy_labels_50is_valid
0train/n02979186/n02979186_9036.JPEGn02979186n02979186n02979186n02979186n02979186False
1train/n02979186/n02979186_11957.JPEGn02979186n02979186n02979186n02979186n03000684False
2train/n02979186/n02979186_9715.JPEGn02979186n02979186n02979186n03417042n03000684False
3train/n02979186/n02979186_21736.JPEGn02979186n02979186n02979186n02979186n03417042False
4train/n02979186/ILSVRC2012_val_00046953.JPEGn02979186n02979186n02979186n02979186n03394916False
\n", "
" ], "text/plain": [ " path noisy_labels_0 noisy_labels_1 \\\n", "0 train/n02979186/n02979186_9036.JPEG n02979186 n02979186 \n", "1 train/n02979186/n02979186_11957.JPEG n02979186 n02979186 \n", "2 train/n02979186/n02979186_9715.JPEG n02979186 n02979186 \n", "3 train/n02979186/n02979186_21736.JPEG n02979186 n02979186 \n", "4 train/n02979186/ILSVRC2012_val_00046953.JPEG n02979186 n02979186 \n", "\n", " noisy_labels_5 noisy_labels_25 noisy_labels_50 is_valid \n", "0 n02979186 n02979186 n02979186 False \n", "1 n02979186 n02979186 n03000684 False \n", "2 n02979186 n03417042 n03000684 False \n", "3 n02979186 n02979186 n03417042 False \n", "4 n02979186 n02979186 n03394916 False " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "csv_file = pd.read_csv(source/'noisy_imagenette.csv')\n", "csv_file.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The generation of these noisy labels are provided in [this Jupyter notebook](https://github.com/fastai/imagenette/blob/master/noisy_labels/generate_labels.ipynb). We have also updated fastai's [train_imagenette.py](https://github.com/fastai/fastai/blob/master/nbs/examples/train_imagenette.py) to utilize the new noisy labels. If you want to train on the Noisy Imagenette dataset using this script, just simply pass the `--pct-noise` argument to the script with the desired noise level." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ ":::{.callout-note}\n", "\n", "The validation set remains clean and its labels are not changed. While technically the accuracy metric [is robust](https://arxiv.org/abs/2012.04193) to noise, I believe it's simpler to use a clean validation set to clearly understand see if a model is learning appropriate decision boundaries on the ground truth.\n", "\n", ":::" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For the original Imagenette dataset, there are technically (3 image sizes)\\*(4 number of epoch levels) for both Imagenette and Imagewoof giving a total of 24 leaderboards. If we had each of these 24 leaderboards for the previously mentioned 4 noise levels (1%, 5%, 25%, 50%), that would give us 96 leaderboards! Instead, the [Imagenette repository](https://github.com/fastai/imagenette) only maintains leaderboards for Noisy Imagenette (and not Imagewoof) for 5% and 50% noise (24 leaderboards). Just like with the regular Imagenette leaderboards, feel free to send a pull request to the Imagenette repository with your results if it beats the current top score. I have provided a [baseline](https://github.com/tmabraham/noisy_imagenette/blob/main/baseline/baseline-01-18-2021.md) which is currently on the leaderboards, as well as a [CSV file](https://github.com/fastai/imagenette/blob/master/noisy_labels/extended_lb.csv) with the baseline accuracy for all 96 leaderboards. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Backstory\n", "For some background, I started looking into training with label noise because of the recent Cassava Leaf Disease Kaggle Competition (my team was able win a silver medal, see [here](https://twitter.com/iScienceLuvr/status/1362879523650330627)), which had a really noisy dataset. One of the recent techniques I heard about was [SAM](https://arxiv.org/abs/2010.01412), which recently achieved a state-of-the-art score on ImageNet (only to be beaten in a few weeks by techniques/models like Meta Pseudo Labels and NFNets). However, the paper also included some improvements to noisy label training. I had a fastai implementation in-progress for the SAM optimizer (probably will describe in an upcoming blog post) and I wanted to test out its noisy label training capabilities on a dataset. I thought about corrupting the Imagenette labels and use that as my dataset for testing SAM. Jeremy Howard suggested adding it to the main Imagenette dataset and here we are!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Closing Remarks\n", "In conclusion, I hope that this Noisy Imagenette datasets serves as a useful benchmarking dataset for machine learning community when it comes to testing and comparing techniques for training on noisy labels. I hope to experiment with some of these techniques like SAM, the different loss functions, etc. and record those results over on this blog, so be sure to keep an eye on this blog, or follow me on [Twitter](https://twitter.com/iScienceLuvr) to get the latest updates!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Acknowledgments\n", "\n", "I'd like to thank Jeremy Howard and especially Hamel Husain for adding the Noisy Imagenette dataset. I also would like to thank Hamel Husain for reviewing my blog post and providing feedback. I'd like to thank Isaac Flath for pointing out an error I originally had when generating the dataset." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.5" } }, "nbformat": 4, "nbformat_minor": 4 }