{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Import data from Hugging Face datasets\n", "\n", "Load datasets from Hugging Face Hub into Pixeltable tables for processing with AI models." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem\n", "\n", "You want to use a dataset from Hugging Face Hub—for fine-tuning, evaluation, or analysis. You need to load it into a format where you can add computed columns, embeddings, or AI transformations.\n", "\n", "| Dataset | Size | Use case |\n", "|---------|------|----------|\n", "| imdb | 50K reviews | Sentiment analysis |\n", "| squad | 100K Q&A | RAG evaluation |\n", "| coco | 330K images | Vision model training |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Solution\n", "\n", "**What's in this recipe:**\n", "\n", "- Import Hugging Face datasets directly into tables\n", "- Handle datasets with multiple splits (train/test/validation)\n", "- Work with image datasets\n", "\n", "You use `pxt.create_table()` with a Hugging Face dataset as the `source` parameter. Pixeltable automatically maps HF types to Pixeltable column types." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Setup" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "%pip install -qU pixeltable datasets" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created directory 'hf_demo'.\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pixeltable as pxt\n", "from datasets import load_dataset\n", "\n", "# Create a fresh directory\n", "pxt.drop_dir('hf_demo', force=True)\n", "pxt.create_dir('hf_demo')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Import a single split\n", "\n", "Load a specific split from a dataset:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Load a small subset for demo (first 100 rows of rotten_tomatoes)\n", "hf_dataset = load_dataset(\n", " 'cornell-movie-review-data/rotten_tomatoes', split='train[:100]'\n", ")" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created table 'reviews'.\n", "Inserting rows into `reviews`: 100 rows [00:00, 14781.69 rows/s]\n", "Inserted 100 rows with 0 errors.\n" ] } ], "source": [ "# Import into Pixeltable\n", "reviews = pxt.create_table('hf_demo/reviews', source=hf_dataset)" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textlabel
the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .pos
the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .pos
effective but too-tepid biopicpos
if you sometimes like to go to the movies to have fun , wasabi is a good place to start .pos
emerges as something rare , an issue movie that's so honest and keenly observed that it doesn't feel like one .pos
" ], "text/plain": [ " text label\n", "0 the rock is destined to be the 21st century's ... pos\n", "1 the gorgeously elaborate continuation of \" the... pos\n", "2 effective but too-tepid biopic pos\n", "3 if you sometimes like to go to the movies to h... pos\n", "4 emerges as something rare , an issue movie tha... pos" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# View imported data\n", "reviews.head(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Import multiple splits\n", "\n", "Load a DatasetDict with multiple splits and track which split each row came from:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Load dataset with multiple splits (small subset for demo)\n", "hf_dataset_dict = load_dataset(\n", " 'cornell-movie-review-data/rotten_tomatoes',\n", " split={'train': 'train[:50]', 'test': 'test[:50]'},\n", ")" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Created table 'reviews_train'.\n", "Inserting rows into `reviews_train`: 50 rows [00:00, 10150.29 rows/s]\n", "Inserted 50 rows with 0 errors.\n", "Created table 'reviews_test'.\n", "Inserting rows into `reviews_test`: 50 rows [00:00, 9883.37 rows/s]\n", "Inserted 50 rows with 0 errors.\n" ] } ], "source": [ "# Import each split separately for clarity\n", "train_data = pxt.create_table(\n", " 'hf_demo/reviews_train', source=hf_dataset_dict['train']\n", ")\n", "test_data = pxt.create_table(\n", " 'hf_demo/reviews_test', source=hf_dataset_dict['test']\n", ")" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textlabel
the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .pos
the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .pos
effective but too-tepid biopicpos
if you sometimes like to go to the movies to have fun , wasabi is a good place to start .pos
emerges as something rare , an issue movie that's so honest and keenly observed that it doesn't feel like one .pos
" ], "text/plain": [ " text label\n", "0 the rock is destined to be the 21st century's ... pos\n", "1 the gorgeously elaborate continuation of \" the... pos\n", "2 effective but too-tepid biopic pos\n", "3 if you sometimes like to go to the movies to h... pos\n", "4 emerges as something rare , an issue movie tha... pos" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# View training data\n", "train_data.head(5)" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textlabel
lovingly photographed in the manner of a golden book sprung to life , stuart little 2 manages sweetness largely without stickiness .pos
consistently clever and suspenseful .pos
it's like a " big chill " reunion of the baader-meinhof gang , only these guys are more harmless pranksters than political activists .pos
" ], "text/plain": [ " text label\n", "0 lovingly photographed in the manner of a golde... pos\n", "1 consistently clever and suspenseful . pos\n", "2 it's like a \" big chill \" reunion of the baade... pos" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# View test data\n", "test_data.head(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Add AI-powered computed columns\n", "\n", "Enrich the dataset with AI models:" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Added 100 column values with 0 errors.\n" ] }, { "data": { "text/plain": [ "100 rows updated, 200 values computed." ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Add a computed column for text length\n", "reviews.add_computed_column(text_length=reviews.text.len())" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
textlabeltext_length
the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .pos177
the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .pos226
effective but too-tepid biopicpos30
if you sometimes like to go to the movies to have fun , wasabi is a good place to start .pos89
emerges as something rare , an issue movie that's so honest and keenly observed that it doesn't feel like one .pos111
" ], "text/plain": [ " text label text_length\n", "0 the rock is destined to be the 21st century's ... pos 177\n", "1 the gorgeously elaborate continuation of \" the... pos 226\n", "2 effective but too-tepid biopic pos 30\n", "3 if you sometimes like to go to the movies to h... pos 89\n", "4 emerges as something rare , an issue movie tha... pos 111" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# View with computed column\n", "reviews.select(reviews.text, reviews.label, reviews.text_length).head(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Type mapping\n", "\n", "Pixeltable automatically maps Hugging Face types to Pixeltable types:\n", "\n", "| Hugging Face Type | Pixeltable Type |\n", "|-------------------|-----------------|\n", "| `Value('string')` | `pxt.String` |\n", "| `Value('int64')` | `pxt.Int` |\n", "| `Value('float32')` | `pxt.Float` |\n", "| `ClassLabel` | `pxt.String` |\n", "| `Image` | `pxt.Image` |\n", "| `Sequence` | `pxt.Array` or `pxt.Json` |\n", "\n", "Use `schema_overrides` to customize type mapping when needed." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Explanation\n", "\n", "**Why import Hugging Face datasets into Pixeltable:**\n", "\n", "1. **Add computed columns** - Enrich data with embeddings, AI analysis, or transformations\n", "1. **Incremental processing** - Add new rows without reprocessing existing data\n", "1. **Persistent storage** - Keep processed results across sessions\n", "1. **Query capabilities** - Filter, aggregate, and join with other tables\n", "\n", "**Working with large datasets:**\n", "\n", "For very large datasets, consider loading in batches or using streaming mode in the `datasets` library before importing." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## See also\n", "\n", "- [Import CSV files](https://docs.pixeltable.com/howto/cookbooks/data/data-import-csv) - For CSV and Excel imports\n", "- [Semantic text search](https://docs.pixeltable.com/howto/cookbooks/search/search-semantic-text) - Add embeddings to text data\n", "- [Hugging Face integration notebook](https://docs.pixeltable.com/howto/providers/working-with-hugging-face) - Full integration guide" ] } ], "metadata": { "kernelspec": { "display_name": "pixeltable", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.11" } }, "nbformat": 4, "nbformat_minor": 2 }