{ "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# Fine-tuning a model with the 🤗 3LC Trainer\n", "\n", "This notebook demonstrates how to use the 3LC Hugging Face Trainer integration to fine-tune a BERT model (bert-base-uncased).\n", "\n", "![](../images/huggingface-mrcp.png)\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "id": "1", "metadata": { "tags": [ "parameters" ] }, "outputs": [], "source": [ "PROJECT_NAME = \"3LC Tutorials - Hugging Face BERT\"\n", "RUN_NAME = \"finetuning-run\"\n", "DESCRIPTION = \"Fine-tune BERT on MRPC\"\n", "TRAIN_DATASET_NAME = \"hugging-face-train\"\n", "VAL_DATASET_NAME = \"hugging-face-val\"\n", "CHECKPOINT = \"bert-base-uncased\"\n", "DEVICE = None\n", "TRAIN_BATCH_SIZE = 64\n", "EVAL_BATCH_SIZE = 256\n", "EPOCHS = 5\n", "NUM_WORKERS = 4\n", "OPTIMIZER = \"adamw_torch\"\n", "TMP_PATH = \"../../transient_data\"\n", "INSTALL_DEPENDENCIES = True" ] }, { "cell_type": "code", "execution_count": null, "id": "2", "metadata": {}, "outputs": [], "source": [ "if INSTALL_DEPENDENCIES:\n", " %pip install -q scikit-learn\n", " %pip install -q 3lc[huggingface]" ] }, { "cell_type": "code", "execution_count": null, "id": "3", "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "import datasets\n", "import evaluate\n", "import numpy as np\n", "import tlc\n", "import torch\n", "from transformers import AutoModelForSequenceClassification, AutoTokenizer, DataCollatorWithPadding, TrainingArguments\n", "\n", "os.environ[\"TRANSFORMERS_NO_ADVISORY_WARNINGS\"] = \"true\" # Removing BertTokenizerFast tokenizer warning\n", "\n", "datasets.utils.logging.disable_progress_bar()" ] }, { "cell_type": "code", "execution_count": null, "id": "4", "metadata": {}, "outputs": [], "source": [ "if DEVICE is None:\n", " if torch.cuda.is_available():\n", " DEVICE = \"cuda\"\n", " elif torch.backends.mps.is_available():\n", " DEVICE = \"mps\"\n", " else:\n", " DEVICE = \"cpu\"" ] }, { "cell_type": "markdown", "id": "5", "metadata": {}, "source": [ "## Initialize a 3LC Run\n", "\n", "We initialize a Run with a call to `tlc.init`, and add the configuration to the Run object." ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": {}, "outputs": [], "source": [ "run = tlc.init(\n", " project_name=PROJECT_NAME,\n", " run_name=RUN_NAME,\n", " description=DESCRIPTION,\n", " if_exists=\"overwrite\",\n", ")" ] }, { "cell_type": "markdown", "id": "7", "metadata": {}, "source": [ "With the 3LC integration, you can use `tlc.Table.from_hugging_face_hub()` as a drop-in replacement for\n", "`datasets.load_dataset()` to create a `tlc.Table`. Notice `.latest()`, which gets the latest version of the 3LC dataset." ] }, { "cell_type": "code", "execution_count": null, "id": "8", "metadata": {}, "outputs": [], "source": [ "tlc_train_dataset = tlc.Table.from_hugging_face_hub(\n", " path=\"nyu-mll/glue\",\n", " name=\"mrpc\",\n", " split=\"train\",\n", " project_name=PROJECT_NAME,\n", " dataset_name=TRAIN_DATASET_NAME,\n", " if_exists=\"overwrite\",\n", ")\n", "\n", "tlc_val_dataset = tlc.Table.from_hugging_face_hub(\n", " path=\"nyu-mll/glue\",\n", " name=\"mrpc\",\n", " split=\"validation\",\n", " project_name=PROJECT_NAME,\n", " dataset_name=VAL_DATASET_NAME,\n", " if_exists=\"overwrite\",\n", ")" ] }, { "cell_type": "markdown", "id": "9", "metadata": {}, "source": [ "`Table` provides a method `map` to apply both preprocessing and on-the-fly transforms to your data before it is sent to the model.\n", "\n", "It is different from huggingface where it generates a new reference of the data directly including the example" ] }, { "cell_type": "code", "execution_count": null, "id": "10", "metadata": {}, "outputs": [], "source": [ "tokenizer = AutoTokenizer.from_pretrained(CHECKPOINT)\n", "\n", "\n", "def tokenize_function_tlc(example):\n", " return {\"label\": example[\"label\"], **tokenizer(example[\"sentence1\"], example[\"sentence2\"], truncation=True)}\n", "\n", "\n", "tlc_tokenized_dataset_train = tlc_train_dataset.with_transform(tokenize_function_tlc)\n", "tlc_tokenized_dataset_val = tlc_val_dataset.with_transform(tokenize_function_tlc)" ] }, { "cell_type": "code", "execution_count": null, "id": "11", "metadata": {}, "outputs": [], "source": [ "data_collator = DataCollatorWithPadding(tokenizer=tokenizer)" ] }, { "cell_type": "markdown", "id": "12", "metadata": {}, "source": [ "Here we define our model with two labels" ] }, { "cell_type": "code", "execution_count": null, "id": "13", "metadata": {}, "outputs": [], "source": [ "# For demonstration purposes, we use the bert-base-uncased model with a different set of labels than\n", "# it was trained on. As a result, there will be a warning about the inconsistency of the classifier and\n", "# pre_classifier weights. This is expected and can be ignored.\n", "model = AutoModelForSequenceClassification.from_pretrained(CHECKPOINT, num_labels=2)" ] }, { "cell_type": "markdown", "id": "14", "metadata": {}, "source": [ "## Setup Metrics Collection\n", "\n", "Computing metrics is done by implementing a function which returns per-sample metrics you would like to see in the 3LC Dashboard. \n", "\n", "This is different from the original compute_metrics of Huggingface which compute per batch the metrics. Here we want to find results with a granularity of per sample basis." ] }, { "cell_type": "code", "execution_count": null, "id": "15", "metadata": {}, "outputs": [], "source": [ "def compute_tlc_metrics(batch, predictor_output):\n", " logits = predictor_output.forward.logits\n", " labels = batch[\"labels\"]\n", "\n", " probabilities = torch.nn.functional.softmax(logits, dim=-1)\n", " predictions = logits.argmax(dim=-1)\n", " loss = torch.nn.functional.cross_entropy(logits, labels, reduction=\"none\")\n", " confidence = probabilities.gather(dim=-1, index=predictions.unsqueeze(-1)).squeeze()\n", "\n", " return {\n", " \"predicted\": predictions.cpu().numpy(),\n", " \"loss\": loss.cpu().numpy(),\n", " \"confidence\": confidence.cpu().numpy(),\n", " }\n", "\n", "\n", "id2label = {0: \"not_equivalent\", 1: \"equivalent\"}\n", "\n", "metrics_collector = tlc.metrics.FunctionalMetricsCollector(\n", " collection_fn=compute_tlc_metrics,\n", " schema={\n", " \"predicted\": tlc.schemas.CategoricalLabelSchema(display_name=\"Predicted Label\", classes=id2label),\n", " \"loss\": tlc.schemas.Float32Schema(display_name=\"Loss\", writable=False),\n", " \"confidence\": tlc.schemas.ConfidenceSchema(writable=False),\n", " },\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "16", "metadata": {}, "outputs": [], "source": [ "# Add references to the input datasets used by the Run.\n", "run.add_input_table(tlc_train_dataset)\n", "run.add_input_table(tlc_val_dataset)" ] }, { "cell_type": "markdown", "id": "17", "metadata": {}, "source": [ "## Train the model with 3LC Trainer\n", "\n", "To perform model training, we replace the usual `Trainer` with the 3LC `Trainer` and provide the per-sample metrics collection function. \n", "\n", "In this example, we still compute the glue MRPC per batch thanks to the *compute_metrics* method.\n", "\n", "We also compute our special per sample tlc metrics thanks to the *metrics_collectors* method.\n", "\n", "With this, we can choose when to collect the metrics using *metrics_collection_epochs*. Here we collect at epoch 0 (before training), and at epochs 2 and 3.\n", "\n", "We also pass *run_name* to attach the Trainer to the existing run." ] }, { "cell_type": "code", "execution_count": null, "id": "18", "metadata": {}, "outputs": [], "source": [ "from tlc.integration.hugging_face.trainer import Trainer\n", "\n", "\n", "def compute_metrics(eval_preds):\n", " metric = evaluate.load(\"glue\", \"mrpc\")\n", " logits, labels = eval_preds\n", " predictions = np.argmax(logits, axis=-1)\n", " return metric.compute(predictions=predictions, references=labels)\n", "\n", "\n", "training_args = TrainingArguments(\n", " output_dir=TMP_PATH,\n", " per_device_train_batch_size=TRAIN_BATCH_SIZE,\n", " per_device_eval_batch_size=EVAL_BATCH_SIZE,\n", " optim=OPTIMIZER,\n", " num_train_epochs=EPOCHS,\n", " report_to=\"none\", # Disable wandb logging\n", " use_cpu=DEVICE == \"cpu\",\n", " eval_strategy=\"epoch\",\n", " disable_tqdm=True,\n", " dataloader_num_workers=NUM_WORKERS,\n", ")\n", "\n", "trainer = Trainer(\n", " model=model,\n", " args=training_args,\n", " train_dataset=tlc_tokenized_dataset_train,\n", " eval_dataset=tlc_tokenized_dataset_val,\n", " processing_class=tokenizer,\n", " data_collator=data_collator,\n", " compute_metrics=compute_metrics,\n", " metrics_collectors=metrics_collector,\n", " metrics_collection_epochs=[0] + list(range(2, EPOCHS)),\n", " run_name=RUN_NAME,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "19", "metadata": {}, "outputs": [], "source": [ "trainer.train()" ] }, { "cell_type": "code", "execution_count": null, "id": "20", "metadata": {}, "outputs": [], "source": [ "run.set_status_completed()" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.9" } }, "nbformat": 4, "nbformat_minor": 5 }