# PaliGemma

This folder is organized into several categories, each focusing on a speicific aspect of working with PaliGemma models:

* [Inference](#inference) : How to load and run PaliGemma models for inference
* [Finetuning](#finetuning) : Dive into finetuning PaliGemma models for specific tasks and domains

## Inference
| Notebook Name | Description |
| :----------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [[PaliGemma_1]Common_use_cases.ipynb]([PaliGemma_1]Common_use_cases.ipynb)                                                       | Illustrate some common use cases for PaliGemma. |
| [[PaliGemma_1]Image_captioning.ipynb]([PaliGemma_1]Image_captioning.ipynb)                                                       | Use PaliGemma to generate image captions using Keras.                                                                                                               |
| [[PaliGemma_1]Referring_expression_segmentation_in_images.ipynb]([PaliGemma_1]Referring_expression_segmentation_in_images.ipynb) | Referring Expression Segmentation in images using PaliGemma.                                                                                                        |
| [[PaliGemma_1]Referring_expression_segmentation_in_videos.ipynb]([PaliGemma_1]Referring_expression_segmentation_in_videos.ipynb) | Referring Expression Segmentation in videos using PaliGemma.                                                                                                        |
| [[PaliGemma_1]Using_with_Mesop.ipynb]([PaliGemma_1]Using_with_Mesop.ipynb)                                                           | Integrate PaliGemma with [Google Mesop](https://google.github.io/mesop/).                                                                                           |
| [[PaliGemma_1]Zero_shot_object_detection_in_images.ipynb]([PaliGemma_1]Zero_shot_object_detection_in_images.ipynb)               | Zero-shot Object Detection in images using PaliGemma.                                                                                                               |
| [[PaliGemma_1]Zero_shot_object_detection_in_videos.ipynb]([PaliGemma_1]Zero_shot_object_detection_in_videos.ipynb)               | Zero-shot Object Detection in videos using PaliGemma.                                                                                                               |
| [[PaliGemma_2]Convert_PaliGemma2_to_ONNX.ipynb]([PaliGemma_2]Convert_PaliGemma2_to_ONNX.ipynb)                                                     | Convert and quantize PaliGemma 2 to ONNX format, making it compatible for inferencing with Transformers.js.                                                                                                                                    |
| [[PaliGemma_2]Inference_PaliGemma2_with_Transformers_js.ipynb]([PaliGemma_2]Inference_PaliGemma2_with_Transformers_js.ipynb)                                                     | Inference PaliGemma 2 with Transformers.js for tasks like image captioning, zero-shot object detection, OCR, and visual Q&A.                                                                                                                                    |
| [[PaliGemma_2]Keras_Quickstart.ipynb]([PaliGemma_2]Keras_Quickstart.ipynb)                                                              | PaliGemma 2 3B DOCCI model quickstart tutorial with Keras                                                                                                        |
| [[PaliGemma_2]Using_with_Transformersjs.ipynb]([PaliGemma_2]Using_with_Transformersjs.ipynb)                                                         | Run PaliGemma 2 with Transformers.js.                                                                                                                                      |

## Finetuning
| Notebook Name | Description |
| :----------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [[PaliGemma_1]Finetune_with_Keras.ipynb]([PaliGemma_1]Finetune_with_Keras.ipynb)                                                             | Finetune PaliGemma with Keras.                                                                                                                                      |
| [[PaliGemma_1]Finetune_with_image_captioning.ipynb]([PaliGemma_1]Finetune_with_image_captioning.ipynb)                                   | Compare the image captioning results using different PaliGemma versions with [Hugging Face](https://huggingface.co/).                                               |
| [[PaliGemma_1]Finetune_with_image_description.ipynb]([PaliGemma_1]Finetune_with_image_description.ipynb)                                       | Finetune PaliGemma for image description using [JAX](https://github.com/google/jax).                                                                                |
| [[PaliGemma_1]Finetune_with_object_detection.ipynb]([PaliGemma_1]Finetune_with_object_detection.ipynb)                                       | Fine-tune PaliGemma for object detection on a fashion dataset using [JAX](https://github.com/google/jax).                                                                                |
| [[PaliGemma_2]Finetune_with_JAX.ipynb]([PaliGemma_2]Finetune_with_JAX.ipynb)                                                         | Finetune PaliGemma 2 with JAX.                                                                                                                                      |
| [[PaliGemma_2]Finetune_with_Keras.ipynb]([PaliGemma_2]Finetune_with_Keras.ipynb)                                                     | Finetune PaliGemma 2 with Keras.                                                                                                                                    |