{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualizing Data with Dimensionality Reduction Techniques "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this walkthrough, you'll learn how to run PCA, t-SNE, UMAP, and custom dimensionality reduction techniques on your data in FiftyOne!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It covers the following:\n",
    "\n",
    "- Why dimensionality reduction is useful\n",
    "- Strengths and weaknesses of different dimensionality reduction techniques\n",
    "- Running built-in dimensionality reduction techniques in FiftyOne\n",
    "- Running custom dimensionality reduction techniques in FiftyOne"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why Dimensionality Reduction?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These days, everyone is excited about embeddings — numeric vectors that represent features of your input data. In computer vision for instance, image embeddings are used in reverse image search applications. And in the context of large language models (LLMs), documents are chunked and embedded (with text embedding models) for retrieval augmented generation (RAG).\n",
    "\n",
    "Embeddings are incredibly powerful, but given their high dimensionality (with lengths typically between 384 and 4096), they can be hard for humans to interpret and inspect. This is where dimensionality reduction techniques come in handy!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dimensionality reduction techniques are quantitative methods for representing information from a higher dimensional space in a lower dimensional space. By squeezing our embeddings into two or three dimensions, we can visualize them to get a more intuitive understanding of the “hidden” structure in our data.\n",
    "\n",
    "When we project high dimensional data into a low dimensional space, we implicitly make a trade-off between representational complexity and interpretability. To compress embeddings, dimensionality reduction techniques make assumptions about the underlying data, its distribution, and the relationships between variables.\n",
    "\n",
    "In this post, we will visualize embeddings using four popular dimensionality reduction techniques: PCA, t-SNE, and UMAP. We will give a brief overview of the strengths, weaknesses, and assumptions of each technique. And we will illustrate that both the model used to generate embeddings, and the dimensionality reduction technique play essential roles in shaping the visualization of your data.\n",
    "\n",
    "*It is also important to note that dimensionality reduction techniques often have hyperparameters, which can have non-negligible impacts on the results. In this post, I am going to use the default hyperparameters everywhere that choices arise. Feel free to modify as you see fit!*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this walkthrough, we will be using the FiftyOne library for data management and visualization. We will use [scikit-learn](https://scikit-learn.org/stable/) for PCA and t-SNE, and [umap-learn](https://umap-learn.readthedocs.io/en/latest/#) for UMAP dimension reduction implementations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U fiftyone scikit-learn umap-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using the test split of the [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html) dataset as our testbed, which contains 10,000 images of size 32x32 pixels, spanning 10 image classes. We can load the dataset/split directly from the [FiftyOne Dataset Zoo](https://docs.voxel51.com/user_guide/dataset_zoo/datasets.html):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fiftyone as fo\n",
    "import fiftyone.brain as fob\n",
    "import fiftyone.zoo as foz\n",
    "\n",
    "dataset = foz.load_zoo_dataset(\"cifar10\", split=\"test\")\n",
    "session = fo.launch_app(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![CIFAR-10](./images/dimension_reduction_cifar10_base_dataset.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will compare and contrast our four dimensionality reduction techniques with two image embedding models: [ResNet-101](https://pytorch.org/vision/main/models/generated/torchvision.models.resnet101.html) and [CLIP](https://github.com/openai/CLIP). Whereas ResNet-101 is a more traditional vision model, representing the relationships between pixels and patches in images, CLIP captures more of the semantic content of the images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can load both from the [FiftyOne Model Zoo](https://docs.voxel51.com/user_guide/model_zoo/index.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clip = foz.load_zoo_model(\"clip-vit-base32-torch\")\n",
    "resnet101 = foz.load_zoo_model(\"resnet101-imagenet-torch\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then generating embeddings for each model amounts to making a single call to the dataset’s `compute_embeddings()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## compute and store resnet101 embeddings \n",
    "dataset.compute_embeddings(\n",
    "    resnet101, \n",
    "    embeddings_field=\"resnet101_embeddings\"\n",
    ")\n",
    "\n",
    "## compute and store clip embeddings \n",
    "dataset.compute_embeddings(\n",
    "    clip, \n",
    "    embeddings_field=\"clip_embeddings\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dimensionality Reduction API in FiftyOne"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we dive into the details of each dimensionality reduction technique, let’s recap the API for running dimensionality reduction in FiftyOne. The [FiftyOne Brain](https://docs.voxel51.com/user_guide/brain.html) provides a [compute_visualization()](https://docs.voxel51.com/api/fiftyone.brain.html#fiftyone.brain.compute_visualization) function that can be used to run dimensionality reduction on your data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first and only positional argument to this function is a sample collection, which can be either a [Dataset](https://docs.voxel51.com/user_guide/using_datasets.html#datasets) or a [DatasetView](https://docs.voxel51.com/user_guide/using_views.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beyond that, you need to specify the following three things:\n",
    "1. *What* you want to reduce the dimensionality of.\n",
    "2. *How* you want to reduce the dimensionality.\n",
    "3. *Where* you want to store the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What to reduce the dimensionality of"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are multiple ways to specify what you would like dimension-reduced. Here are a few options (but certainly not all of them!):\n",
    "\n",
    "- You can specify the name of the field containing the embeddings you would like to reduce using the `embeddings` argument. If your embeddings are stored in field \"my_embeddings_field\" on your samples, you would employ the syntax `embeddings=\"my_embeddings_field\"`. This is useful if you need to reuse the same embeddings for multiple dimensionality reduction techniques, or for other brain methods.\n",
    "- You can pass the embeddings in directly using as numpy array, also via the `embeddings` argument. This is useful if you have already computed your embeddings, and don’t need to store them on your samples.\n",
    "- You can specify the *model* you would like to use to generate embeddings. This can be:\n",
    "    - A `FiftyOne.Model` instance\n",
    "    - The name (a string) of a model from the model zoo, in which case the model by that will be loaded from the FiftyOne Model Zoo.\n",
    "    - A Hugging Face Transformers model, in which case the model will be converted to a `FiftyOne.Model` instance. See the [Hugging Face integration docs](https://docs.voxel51.com/integrations/huggingface.html) for more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to reduce the dimensionality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can specify the base dimensionality reduction technique to use via the `method` argument. This can be one of the following strings: `pca`, `tsne`, `umap`, or `manual`.\n",
    "\n",
    "For `pca`, `tsne`, and `umap`, you can specify the number of dimensions to reduce to via the `num_dims` argument. Additionally, you can specify hyperparameters for each technique as kwargs. For a complete description of available options, check out the visualization configs: \n",
    "[TSNEVisualizationConfig](https://docs.voxel51.com/api/fiftyone.brain.visualization.html#fiftyone.brain.visualization.TSNEVisualizationConfig),\n",
    "[UMAPVisualizationConfig](https://docs.voxel51.com/api/fiftyone.brain.visualization.html#fiftyone.brain.visualization.UMAPVisualizationConfig), and [PCAVisualizationConfig](https://docs.voxel51.com/api/fiftyone.brain.visualization.html#fiftyone.brain.visualization.PCAVisualizationConfig).\n",
    "\n",
    "You can use `method=\"manual\"` if you already have the dimensionality-reduced data, and just want to store it on your samples for visualization purposes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Where to store the results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have specified what you want to reduce the dimensionality of, and how you want to do it, you need to specify where you want to store the results. This is done via the `brain_key` argument. Once you have run the `compute_visualization()` method, you will be able to select this brain key in the FiftyOne App to visualize the results. You can also use the brain key to access the results programmatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fiftyone as fo\n",
    "import fiftyone.brain as fob\n",
    "import fiftyone.zoo as foz\n",
    "\n",
    "dataset = foz.load_zoo_dataset(\"cifar10\", split=\"test\")\n",
    "\n",
    "## Compute PCA visualization\n",
    "fob.compute_visualization(\n",
    "    dataset,\n",
    "    embeddings=\"resnet101\",\n",
    "    method=\"pca\",\n",
    "    brain_key=\"pca_resnet101\"\n",
    ")\n",
    "\n",
    "## Access results\n",
    "pca_resnet_results = dataset.load_brain_results(\"pca_resnet101\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dimensionality Reduction with PCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Principal Component Analysis](https://en.wikipedia.org/wiki/Principal_component_analysis), or PCA, is a dimensionality reduction technique that seeks to preserve as much variance as possible. Intuitively, PCA finds a set of orthogonal axes (principal components) that jointly “explain” as much of the variation in the data as possible. Mathematically, you can interpret PCA algorithms as effectively performing singular value decompositions and truncating the number of dimensions by eliminating the singular vectors with the smallest eigenvalues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Strengths**\n",
    "\n",
    "- Simple, intuitive, and efficient for large datasets! \n",
    "- PCA is amenable to new data: If you have precomputed the transformation on an initial set of embeddings, you can apply that transformation to new embeddings and immediately visualize them in the same space.\n",
    "\n",
    "**Limitations**\n",
    "\n",
    "- Assumes that the relationships between variables are linear — an assumption which often does not hold when the inputs are embeddings, which themselves come from highly nonlinear deep neural networks\n",
    "- Very susceptible to outliers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running PCA on Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PCA is natively supported by the FiftyOne Brain’s `compute_visualization()`. To reduce dimensionality for a set of embeddings, we can specify the field the embeddings are stored in, and pass in `method=\"pca\"`. In the app, we can open up an Embeddings panel to view the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## PCA with ResNet101 embeddings\n",
    "fob.compute_visualization(\n",
    "    dataset, \n",
    "    embeddings=\"resnet101_embeddings\", \n",
    "    method=\"pca\", \n",
    "    brain_key=\"resnet101_pca\"\n",
    ")\n",
    "\n",
    "## PCA with CLIP embeddings\n",
    "fob.compute_visualization(\n",
    "    dataset, \n",
    "    embeddings=\"clip_embeddings\", \n",
    "    method=\"pca\", \n",
    "    brain_key=\"resnet101_pca\"\n",
    ")\n",
    "\n",
    "session = fo.launch_app(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Loading visualization](./images/dimension_reduction_load_resnet101_pca.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can color by any attribute on our samples — in this case the ground truth label — and filter the contents of the sample grid interactively by selecting regions in the embeddings panel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![color-by-label](./images/dimension_reduction_clip_pca.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For both the CLIP and ResNet-101 embeddings, the PCA plot does seem to very loosely retain information from the embeddings (and the original images). However, when we color by label, there is substantial overlap from one class to another. \n",
    "\n",
    "Restricting the CLIP PCA view to just automobiles, trucks, and ships, we can see that the distributions for all three classes are essentially identical, aside from the ships extending slightly farther out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![PCA-overlap](./images/dimension_reduction_clip_pca_overlap.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dimensionality Reduction with t-SNE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[t-Distributed Stochastic Neighbor Embedding](https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding), or t-SNE, is a nonlinear dimensionality reduction technique that aims to, roughly speaking, keep neighbors close. More precisely, t-SNE takes the initial, high-dimensional data (in our case embedding vectors) and computes the similarity between inputs. The algorithm then attempts to learn a lower-dimensional representation which preserves as much of the similarity as possible. Mathematically, this learning is achieved by minimizing the [Kullback-Leibler divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence) between the high-dimensional (fixed) and low-dimensional (trained) distributions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Strengths**\n",
    "\n",
    "- t-SNE is nonlinear, making it a much better fit for (embeddings computed on) datasets like MNIST and CIFAR-10. \n",
    "- The technique is good at preserving local structure, making it easy to see clustering in data!\n",
    "\n",
    "**Limitations**\n",
    "\n",
    "- t-SNE relies on random initialization, so good fits are not guaranteed\n",
    "Still sensitive to outliers\n",
    "- Not scalable: for a dataset with n samples, t-SNE takes $\\mathcal{O}(n^2)$ time to run, and requires $\\mathcal{O}(n^2)$ space to operate\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running t-SNE on Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like PCA, t-SNE is natively supported by the FiftyOne Brain’s `compute_visualization()`, so we can run dimensionality reduction on our embeddings by passing `method=\"tsne\"`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## t-SNE with ResNet101 embeddings\n",
    "fob.compute_visualization(\n",
    "    dataset, \n",
    "    embeddings=\"resnet101_embeddings\", \n",
    "    method=\"tsne\", \n",
    "    brain_key=\"resnet101_tsne\"\n",
    ")\n",
    "\n",
    "## t-SNE with CLIP embeddings\n",
    "fob.compute_visualization(\n",
    "    dataset, \n",
    "    embeddings=\"clip_embeddings\", \n",
    "    method=\"tsne\", \n",
    "    brain_key=\"resnet101_tsne\"\n",
    ")\n",
    "\n",
    "session = fo.launch_app(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the results of t-SNE dimensionality reduction on both ResNet-101 and CLIP embeddings, we can see a lot more separation between the distributions of different classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![tsne-resnet101](./images/dimension_reduction_resnet_tsne.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In both cases, similar classes are still close to each other — for instance, automobiles and trucks are adjacent — but we can also mostly distinguish a main cluster for almost every class. In other words, t-SNE does a very good job at capturing local structure, and a decent job at capturing global structure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![tsne-clip](./images/dimension_reduction_clip_tsne.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dimensionality Reduction with UMAP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Uniform Manifold Approximation and Projection](https://umap-learn.readthedocs.io/en/latest/) (UMAP) is a nonlinear dimensionality reduction technique based on the mathematics of [topology](https://en.wikipedia.org/wiki/Topology). I won’t go into the gory details, as there is an excellent visual explanation of the approach [here](https://umap-learn.readthedocs.io/en/latest/how_umap_works.html), but in essence, UMAP treats the input data as points lying on a special kind of surface called a manifold (technically here a [Riemannian manifold](https://en.wikipedia.org/wiki/Riemannian_manifold)), and tries to learn a lower dimensional representation of the manifold. This explicitly takes global structure into consideration, as opposed to t-SNE, which concerns itself with keeping neighbors close (local structure)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Strengths**\n",
    "\n",
    "- Preserves both global and local structure\n",
    "- Better scaling than t-SNE with dataset size\n",
    "\n",
    "**Limitations**\n",
    "\n",
    "- Like t-SNE, UMAP relies on randomness, and is dependent upon hyperparameters\n",
    "- UMAP assumes that the manifold is locally connected. This can cause problems if there are a few data points that are very far away from the rest of the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running UMAP on Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like PCA and t-SNE, UMAP  is natively supported by the FiftyOne Brain’s `compute_visualization()`, so we can run dimensionality reduction on our embeddings by passing `method=\"umap\"`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## UMAP with ResNet101 embeddings\n",
    "fob.compute_visualization(\n",
    "    dataset, \n",
    "    embeddings=\"resnet101_embeddings\", \n",
    "    method=\"umap\", \n",
    "    brain_key=\"resnet101_umap\"\n",
    ")\n",
    "\n",
    "## UMAP with CLIP embeddings\n",
    "fob.compute_visualization(\n",
    "    dataset, \n",
    "    embeddings=\"clip_embeddings\", \n",
    "    method=\"umap\", \n",
    "    brain_key=\"resnet101_umap\"\n",
    ")\n",
    "\n",
    "session = fo.launch_app(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![UMAP-resnet101](./images/dimension_reduction_resnet_umap.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For both sets of embeddings, the clusters are a lot more spread out than with t-SNE. For ResNet-101, all of the vehicles (automobile, truck, airplane, ship) are in one mega-cluster — or two smaller clusters, depending on how you view it — and all of the animals are in another mega-cluster. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![UMAP-CLIP](./images/dimension_reduction_clip_umap.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interestingly, for the CLIP embeddings, we see that the `airplane` cluster is situated close to both `bird` and `ship`. The `car` and `truck` clusters are very close together; and the `cat` and `dog` clusters are very close together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dimensionality Reduction with Custom Methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on the specific structure of your data, you may find that none of the techniques detailed above provide an intuitive view into your data. Fortunately, there are tons of other techniques you can use. In this section, we’ll show you how to run custom dimensionality reduction techniques with FiftyOne."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Isomap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like UMAP, Isomap is also a nonlinear manifold learning technique. Isomap is built into scikit-learn, so we can fit our high-dimensional data and generate low-dimensional transformed data points as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.manifold import Isomap\n",
    "\n",
    "## get embeddings from dataset\n",
    "embeddings = np.array(dataset.values(\"resnet101_embeddings\"))\n",
    "\n",
    "## create and fit\n",
    "manifold_embedding = Isomap(n_components=2)\n",
    "z = manifold_embedding.fit_transform(embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then create a visualization in FiftyOne by passing `method=’manual’` into `compute_visualization()` and providing these lower-dimensional points via the `points` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fob.compute_visualization(\n",
    "    dataset,\n",
    "    method='manual',\n",
    "    points=z,\n",
    "    brain_key='resnet101_isomap'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Isomap](./images/dimension_reduction_resnet_isomap.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any dimensionality reduction method supported by scikit-learn can be used in analogous fashion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CompressionVAE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[CompressionVAE](https://github.com/maxfrenzel/CompressionVAE) uses Variational Autoencoders to deterministically and reversibly transform the high-dimensional data into a lower dimensional space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run CompressionVAE, clone this forked repo:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/jacobmarks/CompressionVAE.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then `cd` into the directory and install the package locally:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd CompressionVAE\n",
    "!pip install ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Embed the input data (embeddings) into a lower-dimensional space, and create a visualization in FiftyOne via the same `manual` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cvae import cvae\n",
    "\n",
    "X = np.array(dataset.values(\"clip_embeddings\"))\n",
    "embedder = cvae.CompressionVAE(X)\n",
    "embedder.train()\n",
    "z = embedder.embed(X)\n",
    "\n",
    "fob.compute_visualization(\n",
    "    dataset,\n",
    "    method='manual',\n",
    "    points=z,\n",
    "    brain_key='clip_cvae'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![CompressionVAE](./images/dimension_reduction_cvae_clip.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Dimensionality Reduction in FiftyOne"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Registerting Custom Visualization Methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you find yourself running the same custom dimensionality reduction technique like Isomap or CompressionVAE over and over again, you can create a custom visualization config to make your life easier. This is done by subclassing the [VisualizationConfig](https://docs.voxel51.com/api/fiftyone.brain.visualization.html#fiftyone.brain.visualization.VisualizationConfig) and [Visualization](https://docs.voxel51.com/api/fiftyone.brain.visualization.html#fiftyone.brain.visualization.Visualization) classes.\n",
    "\n",
    "Here's a simple example using Isomap:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.manifold import Isomap\n",
    "\n",
    "import fiftyone as fo\n",
    "import fiftyone.brain as fob\n",
    "\n",
    "class IsomapVisualizationConfig(fob.VisualizationConfig):\n",
    "    def __init__(self, n_neighbors=5, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.n_neighbors = n_neighbors\n",
    "\n",
    "\n",
    "class IsomapVisualization(fob.Visualization):\n",
    "    def fit(self, embeddings):\n",
    "        manifold_embedding = Isomap(\n",
    "            n_components=2, \n",
    "            n_neighbors=self.config.n_neighbors\n",
    "        )\n",
    "        return manifold_embedding.fit_transform(embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then use this by passing in the new visualization config in for the `method` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = foz.load_zoo_dataset(\"quickstart\")\n",
    "\n",
    "model = foz.load_zoo_model(\"clip-vit-base32-torch\")\n",
    "embeddings = dataset.compute_embeddings(model)\n",
    "\n",
    "results2 = fob.compute_visualization(\n",
    "    dataset,\n",
    "    embeddings=embeddings,\n",
    "    method=IsomapVisualizationConfig,\n",
    "    brain_key=\"isomap\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dimensionality Reduction with Object Patches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of the dimensionality reduction techniques we have discussed so far have been applied to embeddings computed on entire images. However, you can also apply dimensionality reduction to embeddings computed on object patches. This can be useful if you want to visualize the relationships between objects in your images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To do this, pass in the name of the field containing the object patches you would like to reduce via the `patches_field` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fiftyone as fo\n",
    "import fiftyone.brain as fob\n",
    "import fiftyone.zoo as foz\n",
    "\n",
    "dataset = foz.load_zoo_dataset(\"quickstart\")\n",
    "\n",
    "fob.compute_visualization(\n",
    "    dataset,\n",
    "    patches_field=\"ground_truth\",\n",
    "    method=\"umap\",\n",
    "    brain_key=\"gt_umap\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dimensionality reduction is critical to understanding our data, and our models. But it is important to think of dimensionality reduction not just as a single tool, but rather as a collection of techniques. Each technique has its own advantages; and each method projects certain assumptions onto the data, which may or may not hold for your data. I hope this walkthrough helps you to see your data in a new way!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}