{ "cells": [ { "cell_type": "markdown", "id": "03b19adf-76fa-4938-af32-ae11453bf68e", "metadata": {}, "source": [ "# Fine-tunining DeBERTa model on a question answering task with ORTTrainer" ] }, { "cell_type": "markdown", "id": "d133d240-7a72-4f24-b0c1-55fd8fd7ee77", "metadata": { "tags": [] }, "source": [ "In this notebook, we will see how to fine-tune the [DeBERTa base](https://huggingface.co/microsoft/deberta-base/tree/main) model to a question answering task, which is the task of extracting the answer to a question from a given context. We will use the `ORTTrainer` API in Optimum library to leverage ONNX Runtime backend to accelerate the training." ] }, { "cell_type": "markdown", "id": "7e816d8e-85b8-4225-948a-0814ba19e6bd", "metadata": {}, "source": [ "## Start by setting up the environment\n", "\n", "To use ONNX Runtime for training, you need a machine with at least one NVIDIA or AMD GPU.\n", "\n", "ONNX Runtime training module need to be properly installed before launching the notebook! Please follow the instruction in [Optimum's documentation](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/trainer#start-by-setting-up-the-environment) to set up your environment, or use directly the [dockerfiles in Optimum](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/trainer#start-by-setting-up-the-environment) " ] }, { "cell_type": "code", "execution_count": 1, "id": "1786df19-07a0-4f9d-aacd-0d100aad0fae", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Thu Apr 6 08:49:04 2023 \n", "+-----------------------------------------------------------------------------+\n", "| NVIDIA-SMI 510.108.03 Driver Version: 510.108.03 CUDA Version: 11.6 |\n", "|-------------------------------+----------------------+----------------------+\n", "| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n", "| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n", "| | | MIG M. |\n", "|===============================+======================+======================|\n", "| 0 Tesla V100-SXM2... Off | 00000000:00:1D.0 Off | 0 |\n", "| N/A 38C P0 39W / 300W | 4MiB / 16384MiB | 0% Default |\n", "| | | N/A |\n", "+-------------------------------+----------------------+----------------------+\n", " \n", "+-----------------------------------------------------------------------------+\n", "| Processes: |\n", "| GPU GI CI PID Type Process name GPU Memory |\n", "| ID ID Usage |\n", "|=============================================================================|\n", "+-----------------------------------------------------------------------------+\n" ] } ], "source": [ "# Check your GPU\n", "!nvidia-smi" ] }, { "cell_type": "markdown", "id": "2be03174-3fc7-4b50-bb7f-28673a2758d2", "metadata": {}, "source": [ "If you're opening this Notebook on colab, you will probably need to install 🤗 Optimum, 🤗 Transformers, 🤗 Datasets and 🤗 evaluate. Uncomment the following cell and run it." ] }, { "cell_type": "code", "execution_count": 2, "id": "a758e26c-0847-4978-9cfe-6e27c7e9cce8", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting optimum\n", " Downloading optimum-1.7.3-py3-none-any.whl (300 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m300.5/300.5 kB\u001b[0m \u001b[31m9.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: transformers in /usr/local/lib/python3.8/dist-packages (4.27.0.dev0)\n", "Requirement already satisfied: datasets in /usr/local/lib/python3.8/dist-packages (2.10.0)\n", "Requirement already satisfied: evaluate in /usr/local/lib/python3.8/dist-packages (0.4.0)\n", "Requirement already satisfied: huggingface_hub in /usr/local/lib/python3.8/dist-packages (0.12.1)\n", "Requirement already satisfied: torchvision in /usr/local/lib/python3.8/dist-packages (from optimum) (0.14.1+cu116)\n", "Requirement already satisfied: sympy in /usr/local/lib/python3.8/dist-packages (from optimum) (1.11.1)\n", "Requirement already satisfied: torch>=1.9 in /usr/local/lib/python3.8/dist-packages (from optimum) (1.13.1+cu116)\n", "Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from optimum) (23.0)\n", "Requirement already satisfied: coloredlogs in /usr/local/lib/python3.8/dist-packages (from optimum) (15.0.1)\n", "Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from optimum) (1.24.2)\n", "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.8/dist-packages (from transformers) (4.64.1)\n", "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (6.0)\n", "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (2022.10.31)\n", "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers) (3.9.0)\n", "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (0.13.2)\n", "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from transformers) (2.28.2)\n", "Requirement already satisfied: multiprocess in /usr/local/lib/python3.8/dist-packages (from datasets) (0.70.14)\n", "Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.8/dist-packages (from datasets) (0.18.0)\n", "Requirement already satisfied: dill<0.3.7,>=0.3.0 in /usr/local/lib/python3.8/dist-packages (from datasets) (0.3.6)\n", "Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from datasets) (1.5.3)\n", "Requirement already satisfied: aiohttp in /usr/local/lib/python3.8/dist-packages (from datasets) (3.8.4)\n", "Requirement already satisfied: pyarrow>=6.0.0 in /usr/local/lib/python3.8/dist-packages (from datasets) (11.0.0)\n", "Requirement already satisfied: xxhash in /usr/local/lib/python3.8/dist-packages (from datasets) (3.2.0)\n", "Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.8/dist-packages (from datasets) (2023.1.0)\n", "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface_hub) (4.5.0)\n", "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets) (22.2.0)\n", "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets) (1.8.2)\n", "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets) (3.0.1)\n", "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets) (6.0.4)\n", "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets) (1.3.3)\n", "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets) (1.3.1)\n", "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets) (4.0.2)\n", "Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests->transformers) (2.8)\n", "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (1.26.14)\n", "Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests->transformers) (2019.11.28)\n", "Requirement already satisfied: protobuf<=3.20.2 in /usr/local/lib/python3.8/dist-packages (from transformers) (3.20.2)\n", "Requirement already satisfied: sentencepiece!=0.1.92,>=0.1.91 in /usr/local/lib/python3.8/dist-packages (from transformers) (0.1.97)\n", "Requirement already satisfied: humanfriendly>=9.1 in /usr/local/lib/python3.8/dist-packages (from coloredlogs->optimum) (10.0)\n", "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.8/dist-packages (from pandas->datasets) (2.8.2)\n", "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.8/dist-packages (from pandas->datasets) (2022.7.1)\n", "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.8/dist-packages (from sympy->optimum) (1.2.1)\n", "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.8/dist-packages (from torchvision->optimum) (9.4.0)\n", "Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.1->pandas->datasets) (1.14.0)\n", "Installing collected packages: optimum\n", "Successfully installed optimum-1.7.3\n", "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n", "\u001b[0m" ] } ], "source": [ "!pip install optimum transformers datasets evaluate huggingface_hub" ] }, { "cell_type": "code", "execution_count": 3, "id": "2b0b586b-d710-4ce5-8491-b2a3aa4096da", "metadata": { "tags": [] }, "outputs": [], "source": [ "# This flag is the difference between SQUAD v1 or 2 (if you're using another dataset, it indicates if impossible\n", "# answers are allowed or not).\n", "squad_v2 = False\n", "model_checkpoint = \"microsoft/deberta-base\"\n", "batch_size = 16" ] }, { "cell_type": "markdown", "id": "6c7b9c65-d07f-4ab1-b6ab-5dd480177f9c", "metadata": {}, "source": [ "## Loading the dataset" ] }, { "cell_type": "markdown", "id": "c347b1cb-83b4-47dc-9cd2-f4d99221fc9f", "metadata": {}, "source": [ "We will use the 🤗 Datasets library to download the data and get the metric we need to use for evaluation. This can be easily done with the functions load_dataset and load_metric." ] }, { "cell_type": "code", "execution_count": 4, "id": "14f8d218-f659-43d3-8538-1bb2b393eecf", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.8/dist-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", " from .autonotebook import tqdm as notebook_tqdm\n", "Downloading builder script: 100%|███████████████████████████████████████████████████████████| 5.27k/5.27k [00:00<00:00, 5.12MB/s]\n", "Downloading metadata: 100%|█████████████████████████████████████████████████████████████████| 2.36k/2.36k [00:00<00:00, 2.29MB/s]\n", "Downloading readme: 100%|███████████████████████████████████████████████████████████████████| 7.67k/7.67k [00:00<00:00, 5.37MB/s]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Downloading and preparing dataset squad/plain_text to /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Downloading data files: 0%| | 0/2 [00:00= end_char):\n", " tokenized_examples[\"start_positions\"].append(cls_index)\n", " tokenized_examples[\"end_positions\"].append(cls_index)\n", " else:\n", " # Otherwise move the token_start_index and token_end_index to the two ends of the answer.\n", " # Note: we could go after the last offset if the answer is the last word (edge case).\n", " while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:\n", " token_start_index += 1\n", " tokenized_examples[\"start_positions\"].append(token_start_index - 1)\n", " while offsets[token_end_index][1] >= end_char:\n", " token_end_index -= 1\n", " tokenized_examples[\"end_positions\"].append(token_end_index + 1)\n", "\n", " return tokenized_examples" ] }, { "cell_type": "markdown", "id": "ab6bb2d9-bdad-4796-b1eb-fe06f8c7e3a3", "metadata": {}, "source": [ "Now apply this function on all the sentences (or pairs of sentences) in our dataset:" ] }, { "cell_type": "code", "execution_count": 11, "id": "83874d1e-80f0-44e3-98da-bd8ed9916b14", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " \r" ] } ], "source": [ "tokenized_datasets = datasets.map(prepare_train_features, batched=True, remove_columns=datasets[\"train\"].column_names)" ] }, { "cell_type": "markdown", "id": "e5b3f1fb-14f0-4d29-acc2-e7a1d5494076", "metadata": {}, "source": [ "## Fine-tuning DeBERTa model" ] }, { "cell_type": "markdown", "id": "f4a39f2f-66bf-45fa-8f21-ff5cca95313e", "metadata": {}, "source": [ "Now that our data is ready for training, we can download the pretrained model and fine-tune it. Since our task is question answering, we use the `AutoModelForQuestionAnswering` class. Like with the tokenizer, the from_pretrained method will download and cache the model for us:" ] }, { "cell_type": "code", "execution_count": 23, "id": "ffcb6fdf-ae3a-4d2a-bd4c-bf161ff51af2", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Some weights of the model checkpoint at microsoft/deberta-base were not used when initializing DebertaForQuestionAnswering: ['lm_predictions.lm_head.bias', 'lm_predictions.lm_head.dense.bias', 'lm_predictions.lm_head.LayerNorm.bias', 'lm_predictions.lm_head.LayerNorm.weight', 'deberta.embeddings.position_embeddings.weight', 'lm_predictions.lm_head.dense.weight']\n", "- This IS expected if you are initializing DebertaForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n", "- This IS NOT expected if you are initializing DebertaForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n", "Some weights of DebertaForQuestionAnswering were not initialized from the model checkpoint at microsoft/deberta-base and are newly initialized: ['qa_outputs.weight', 'qa_outputs.bias']\n", "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n" ] } ], "source": [ "from transformers import AutoModelForQuestionAnswering\n", "from optimum.onnxruntime import ORTTrainer, ORTTrainingArguments\n", "\n", "from transformers import default_data_collator\n", "\n", "model = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint)\n", "data_collator = default_data_collator" ] }, { "cell_type": "markdown", "id": "3efbacf5-6e8b-48d7-b3d7-69b42e65c933", "metadata": {}, "source": [ "Before setting up the training arguments, we will download a configuration of deepspeed stage 1. ONNX Runtime training supports ZeRO stage 1(optimizer state partitioning)." ] }, { "cell_type": "code", "execution_count": 13, "id": "3b3832f5-552a-4cb7-96a3-ba9ca3a9a360", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "--2023-04-06 08:56:02-- https://raw.githubusercontent.com/huggingface/optimum/main/tests/onnxruntime/ds_configs/ds_config_zero_stage_1.json\n", "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ...\n", "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 1224 (1.2K) [text/plain]\n", "Saving to: ‘ds_config_zero_stage_1.json.1’\n", "\n", "ds_config_zero_stag 100%[===================>] 1.20K --.-KB/s in 0s \n", "\n", "2023-04-06 08:56:02 (60.4 MB/s) - ‘ds_config_zero_stage_1.json.1’ saved [1224/1224]\n", "\n" ] } ], "source": [ "!wget https://raw.githubusercontent.com/huggingface/optimum/main/tests/onnxruntime/ds_configs/ds_config_zero_stage_1.json" ] }, { "cell_type": "code", "execution_count": 24, "id": "f3e1cf9e-72eb-4cdb-a10a-5cdbf252eb8f", "metadata": { "tags": [] }, "outputs": [], "source": [ "model_name = model_checkpoint.split(\"/\")[-1]\n", "args = ORTTrainingArguments(\n", " output_dir=f\"{model_name}-finetuned-squad\",\n", " evaluation_strategy = \"epoch\",\n", " learning_rate=2e-5,\n", " per_device_train_batch_size=batch_size,\n", " per_device_eval_batch_size=batch_size,\n", " num_train_epochs=3,\n", " weight_decay=0.01,\n", " optim=\"adamw_ort_fused\",\n", " deepspeed=\"ds_config_zero_stage_1.json\",\n", ")" ] }, { "cell_type": "markdown", "id": "864cf4aa-5285-4b43-a18a-3f79f989adea", "metadata": {}, "source": [ "Instantiate `ORTTrainer`, here we will use ORT fused optimizer to have better performance on latency:" ] }, { "cell_type": "code", "execution_count": 25, "id": "1c9da87b-698b-4351-a4e4-122c532318ea", "metadata": { "tags": [] }, "outputs": [], "source": [ "trainer = ORTTrainer(\n", " model=model,\n", " args=args,\n", " train_dataset=tokenized_datasets[\"train\"],\n", " eval_dataset=tokenized_datasets[\"validation\"],\n", " data_collator=data_collator,\n", " tokenizer=tokenizer,\n", " feature=\"question-answering\",\n", ")" ] }, { "cell_type": "markdown", "id": "115c93b8-9870-49ad-870c-45cc6c53a238", "metadata": {}, "source": [ "Launch the training (it will leverage one single GPU for the training)" ] }, { "cell_type": "code", "execution_count": 24, "id": "846ee1dc-6c29-4a94-888a-6b60112e75e7", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Wrap ORTModule for ONNX Runtime training.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2023-04-05 16:41:07,941] [WARNING] [config_utils.py:74:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.8/dist-packages/torch/onnx/utils.py:1794: FutureWarning: The first argument to symbolic functions is deprecated in 1.13 and will be removed in the future. Please annotate treat the first argument (g) as GraphContext and use context information from the object instead.\n", " warnings.warn(\n", "/usr/local/lib/python3.8/dist-packages/onnxruntime/training/ortmodule/_utils.py:250: UserWarning: User Module's attribute name float collides with ORTModule's attribute name. User Module's method may not be called upon invocation through ORTModule.\n", " warnings.warn(\n", "/usr/local/lib/python3.8/dist-packages/onnxruntime/training/ortmodule/_utils.py:250: UserWarning: User Module's attribute name half collides with ORTModule's attribute name. User Module's method may not be called upon invocation through ORTModule.\n", " warnings.warn(\n", "/usr/local/lib/python3.8/dist-packages/onnxruntime/training/ortmodule/_utils.py:250: UserWarning: User Module's attribute name to collides with ORTModule's attribute name. User Module's method may not be called upon invocation through ORTModule.\n", " warnings.warn(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Using /root/.cache/torch_extensions/py38_cu116 as PyTorch extensions root...\n", "Creating extension directory /root/.cache/torch_extensions/py38_cu116/fused_adam...\n", "Detected CUDA files, patching ldflags\n", "Emitting ninja build file /root/.cache/torch_extensions/py38_cu116/fused_adam/build.ninja...\n", "Building extension module fused_adam...\n", "Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "[1/3] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\\\"_gcc\\\" -DPYBIND11_STDLIB=\\\"_libstdcpp\\\" -DPYBIND11_BUILD_ABI=\\\"_cxxabi1011\\\" -I/usr/local/lib/python3.8/dist-packages/deepspeed/ops/csrc/includes -I/usr/local/lib/python3.8/dist-packages/deepspeed/ops/csrc/adam -isystem /usr/local/lib/python3.8/dist-packages/torch/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.8/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=compute_70 -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -lineinfo --use_fast_math -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_70,code=compute_70 -std=c++14 -c /usr/local/lib/python3.8/dist-packages/deepspeed/ops/csrc/adam/multi_tensor_adam.cu -o multi_tensor_adam.cuda.o \n", "[2/3] c++ -MMD -MF fused_adam_frontend.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\\\"_gcc\\\" -DPYBIND11_STDLIB=\\\"_libstdcpp\\\" -DPYBIND11_BUILD_ABI=\\\"_cxxabi1011\\\" -I/usr/local/lib/python3.8/dist-packages/deepspeed/ops/csrc/includes -I/usr/local/lib/python3.8/dist-packages/deepspeed/ops/csrc/adam -isystem /usr/local/lib/python3.8/dist-packages/torch/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.8/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++14 -g -Wno-reorder -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -c /usr/local/lib/python3.8/dist-packages/deepspeed/ops/csrc/adam/fused_adam_frontend.cpp -o fused_adam_frontend.o \n", "[3/3] c++ fused_adam_frontend.o multi_tensor_adam.cuda.o -shared -L/usr/local/lib/python3.8/dist-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda_cu -ltorch_cuda_cpp -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o fused_adam.so\n", "Time to load fused_adam op: 35.75363326072693 seconds\n", "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Loading extension module fused_adam...\n", "Using /root/.cache/torch_extensions/py38_cu116 as PyTorch extensions root...\n", "Creating extension directory /root/.cache/torch_extensions/py38_cu116/utils...\n", "Emitting ninja build file /root/.cache/torch_extensions/py38_cu116/utils/build.ninja...\n", "Building extension module utils...\n", "Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", "To disable this warning, you can either:\n", "\t- Avoid using `tokenizers` before the fork if possible\n", "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", "[1/2] c++ -MMD -MF flatten_unflatten.o.d -DTORCH_EXTENSION_NAME=utils -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\\\"_gcc\\\" -DPYBIND11_STDLIB=\\\"_libstdcpp\\\" -DPYBIND11_BUILD_ABI=\\\"_cxxabi1011\\\" -isystem /usr/local/lib/python3.8/dist-packages/torch/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.8/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.8/dist-packages/torch/include/THC -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /usr/local/lib/python3.8/dist-packages/deepspeed/ops/csrc/utils/flatten_unflatten.cpp -o flatten_unflatten.o \n", "[2/2] c++ flatten_unflatten.o -shared -L/usr/local/lib/python3.8/dist-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o utils.so\n", "Time to load utils op: 20.83265709877014 seconds\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Loading extension module utils...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Rank: 0 partition count [1] and sizes[(138603266, False)] \n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Using /root/.cache/torch_extensions/py38_cu116 as PyTorch extensions root...\n", "No modifications detected for re-loaded extension module utils, skipping build step...\n", "Loading extension module utils...\n", "***** Running training *****\n", " Num examples = 88550\n", " Num Epochs = 3\n", " Instantaneous batch size per device = 16\n", " Total train batch size (w. parallel, distributed & accumulation) = 16\n", " Gradient Accumulation steps = 1\n", " Total optimization steps = 16605\n", " Number of trainable parameters = 138603266\n", "/usr/local/lib/python3.8/dist-packages/onnxruntime/training/ortmodule/_training_manager.py:192: UserWarning: Fast path enabled - skipping checks. Rebuild graph: True, Execution agent: True, Device check: True\n", " warnings.warn(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Time to load utils op: 0.0007810592651367188 seconds\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Warning: Checker does not support models with experimental ops: ATen\n", "/usr/local/lib/python3.8/dist-packages/onnxruntime/training/ortmodule/_logger.py:52: UserWarning: There were one or more warnings or errors raised while exporting the PyTorch model. Please enable INFO level logging to view all warnings and errors.\n", " warnings.warn(\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n" ] }, { "data": { "text/html": [ "\n", "
\n", " \n", " \n", " [16605/16605 2:19:36, Epoch 3/3]\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EpochTraining LossValidation Loss
10.8281000.801068
20.6598000.793299
30.5081000.895923

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "[INFO] Evaluating with PyTorch backend. If you want to use ONNX Runtime for the evaluation, set `trainer.evaluate(inference_with_ort=True)`.\n", "/usr/local/lib/python3.8/dist-packages/onnxruntime/training/ortmodule/_inference_manager.py:76: UserWarning: Fast path enabled - skipping checks.rebuild gradient graph: True,execution agent recreation: True,device check: True\n", " warnings.warn(\n", "Warning: Checker does not support models with experimental ops: ATen\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "Cannot determine if 512 - attention_mask_dim1 < 0\n", "[INFO] Evaluating with PyTorch backend. If you want to use ONNX Runtime for the evaluation, set `trainer.evaluate(inference_with_ort=True)`.\n", "[INFO] Evaluating with PyTorch backend. If you want to use ONNX Runtime for the evaluation, set `trainer.evaluate(inference_with_ort=True)`.\n", "\n", "\n", "Training completed. Do not forget to share your model on huggingface.co/models =)\n", "\n", "\n" ] } ], "source": [ "trainer.train()\n", "trainer.save_model()" ] }, { "cell_type": "markdown", "id": "c1483c08-b0e4-4aa1-bc9e-319caa1e4ac1", "metadata": {}, "source": [ "If you want to leverage multiple gpus for distributed data parallele training, you can use 🤗 Accelerate's notebook launcher to enable multi-gpu training:" ] }, { "cell_type": "code", "execution_count": 18, "id": "19728c34-bf22-418f-944f-41e2f371e720", "metadata": { "tags": [] }, "outputs": [], "source": [ "from accelerate import notebook_launcher\n", "\n", "def train_trainer_ddp():\n", " model = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint)\n", " model_name = model_checkpoint.split(\"/\")[-1]\n", " args = ORTTrainingArguments(\n", " output_dir=f\"{model_name}-finetuned-squad\",\n", " evaluation_strategy = \"epoch\",\n", " learning_rate=2e-5,\n", " per_device_train_batch_size=batch_size,\n", " per_device_eval_batch_size=batch_size,\n", " num_train_epochs=3,\n", " weight_decay=0.01,\n", " optim=\"adamw_ort_fused\",\n", " deepspeed=\"ds_config_zero_stage_1.json\",\n", " )\n", " \n", " data_collator = default_data_collator\n", " trainer = ORTTrainer(\n", " model=model,\n", " args=args,\n", " train_dataset=tokenized_datasets[\"train\"],\n", " eval_dataset=tokenized_datasets[\"validation\"],\n", " data_collator=data_collator,\n", " tokenizer=tokenizer,\n", " feature=\"question-answering\",\n", " )\n", " \n", " trainer.train()\n", " trainer.save_model()\n", "\n", "notebook_launcher(train_trainer_ddp, args=(), num_processes=4)" ] }, { "cell_type": "markdown", "id": "d4fca066-52be-48d4-a6fd-a6a54fd5cbb7", "metadata": {}, "source": [ "## Evaluation" ] }, { "cell_type": "markdown", "id": "55a615b8-b9f8-4bb7-9355-9d419d21bf44", "metadata": {}, "source": [ "Evaluating our model will require a bit more work, as we will need to map the predictions of our model back to parts of the context. The model itself predicts logits for the start and en position of our answer: if we take a batch from our validation datalaoder, here is the output our model gives us:" ] }, { "cell_type": "code", "execution_count": 27, "id": "a05214fc-98e0-4c56-9c3f-7ff145a36b30", "metadata": { "tags": [] }, "outputs": [], "source": [ "import torch\n", "\n", "for batch in trainer.get_eval_dataloader():\n", " break\n", "batch = {k: v.to(trainer.args.device) for k, v in batch.items()}\n", "with torch.no_grad():\n", " output = trainer.model(**batch)\n", "output.keys()" ] }, { "cell_type": "markdown", "id": "adc7434b-8531-4eda-a7f1-315cba8ffe30", "metadata": {}, "source": [ "The output of the model is a dict-like object that contains the loss (since we provided labels), the start and end logits. We won't need the loss for our predictions, let's have a look a the logits:" ] }, { "cell_type": "markdown", "id": "a9b8fb59-1519-4b60-9d11-46ee033b0847", "metadata": {}, "source": [ "\n", "We have one logit for each feature and each token. The most obvious thing to predict an answer for each feature is to take the index for the maximum of the start logits as a start position and the index of the maximum of the end logits as an end position. This will work great in a lot of cases, but what if this prediction gives us something impossible: the start position could be greater than the end position, or point to a span of text in the question instead of the answer. In that case, we might want to look at the second best prediction to see if it gives a possible answer and select that instead.\n", "\n", "To classify our answers, we will use the score obtained by adding the start and end logits. We won't try to order all the possible answers and limit ourselves to with a hyper-parameter we call n_best_size. We'll pick the best indices in the start and end logits and gather all the answers this predicts. After checking if each one is valid, we will sort them by their score and keep the best one. Here is how we would do this on the first feature in the batch. And then we can sort the `valid_answers` according to their score and only keep the best one. The only point left is how to check a given span is inside the context (and not the question) and how to get back the text inside. To do this, we need to add two things to our validation features:\n", "* the ID of the example that generated the feature (since each example can generate several features, as seen before);\n", "* the offset mapping that will give us a map from token indices to character positions in the context.\n", "That's why we will re-process the validation set with the following function, slightly different from `prepare_train_features`:" ] }, { "cell_type": "code", "execution_count": 25, "id": "970aa1ae-34f5-4b9f-84f1-0f0937f3bf40", "metadata": { "tags": [] }, "outputs": [], "source": [ "n_best_size = 20" ] }, { "cell_type": "code", "execution_count": 28, "id": "c3db99d5-71e9-4707-b07c-1367eb0532ee", "metadata": { "tags": [] }, "outputs": [], "source": [ "def prepare_validation_features(examples):\n", " # Some of the questions have lots of whitespace on the left, which is not useful and will make the\n", " # truncation of the context fail (the tokenized question will take a lots of space). So we remove that\n", " # left whitespace\n", " examples[\"question\"] = [q.lstrip() for q in examples[\"question\"]]\n", "\n", " # Tokenize our examples with truncation and maybe padding, but keep the overflows using a stride. This results\n", " # in one example possible giving several features when a context is long, each of those features having a\n", " # context that overlaps a bit the context of the previous feature.\n", " tokenized_examples = tokenizer(\n", " examples[\"question\" if pad_on_right else \"context\"],\n", " examples[\"context\" if pad_on_right else \"question\"],\n", " truncation=\"only_second\" if pad_on_right else \"only_first\",\n", " max_length=max_length,\n", " stride=doc_stride,\n", " return_overflowing_tokens=True,\n", " return_offsets_mapping=True,\n", " padding=\"max_length\",\n", " )\n", "\n", " # Since one example might give us several features if it has a long context, we need a map from a feature to\n", " # its corresponding example. This key gives us just that.\n", " sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n", "\n", " # We keep the example_id that gave us this feature and we will store the offset mappings.\n", " tokenized_examples[\"example_id\"] = []\n", "\n", " for i in range(len(tokenized_examples[\"input_ids\"])):\n", " # Grab the sequence corresponding to that example (to know what is the context and what is the question).\n", " sequence_ids = tokenized_examples.sequence_ids(i)\n", " context_index = 1 if pad_on_right else 0\n", "\n", " # One example can give several spans, this is the index of the example containing this span of text.\n", " sample_index = sample_mapping[i]\n", " tokenized_examples[\"example_id\"].append(examples[\"id\"][sample_index])\n", "\n", " # Set to None the offset_mapping that are not part of the context so it's easy to determine if a token\n", " # position is part of the context or not.\n", " tokenized_examples[\"offset_mapping\"][i] = [\n", " (o if sequence_ids[k] == context_index else None)\n", " for k, o in enumerate(tokenized_examples[\"offset_mapping\"][i])\n", " ]\n", "\n", " return tokenized_examples" ] }, { "cell_type": "code", "execution_count": 29, "id": "b4828ba5-e472-42ef-9c9e-aa2c7cb67986", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " \r" ] } ], "source": [ "validation_features = datasets[\"validation\"].map(\n", " prepare_validation_features,\n", " batched=True,\n", " remove_columns=datasets[\"validation\"].column_names\n", ")" ] }, { "cell_type": "markdown", "id": "c4e3199f-5b9b-424d-82f0-1706aca975dd", "metadata": {}, "source": [ "Now we can grab the predictions for all features by using the `Trainer.predict` method:" ] }, { "cell_type": "code", "execution_count": 32, "id": "7ff0cbbd-f151-4a28-9731-1c52bf507185", "metadata": { "tags": [] }, "outputs": [], "source": [ "raw_predictions = trainer.predict(validation_features)" ] }, { "cell_type": "markdown", "id": "018af5bb-296f-479d-9188-f2a65027e6ec", "metadata": {}, "source": [ "The `ORTTrainer` hides the columns that are not used by the model (here `example_id` and `offset_mapping` which we will need for our post-processing), so we set them back:" ] }, { "cell_type": "code", "execution_count": 34, "id": "bff0b9c2-d6c8-4c14-9d08-2f469097ebdc", "metadata": { "tags": [] }, "outputs": [], "source": [ "validation_features.set_format(type=validation_features.format[\"type\"], columns=list(validation_features.features.keys()))" ] }, { "cell_type": "markdown", "id": "a8940464-cce8-4582-8401-0405e2c45dbe", "metadata": {}, "source": [ "We can now refine the test we had before: since we set `None` in the offset mappings when it corresponds to a part of the question, it's easy to check if an answer is fully inside the context. We also eliminate very long answers from our considerations (with an hyper-parameter we can tune)" ] }, { "cell_type": "code", "execution_count": 35, "id": "15f8fa23-f46f-4bf4-9df1-c30f15c046ce", "metadata": { "tags": [] }, "outputs": [], "source": [ "max_answer_length = 30" ] }, { "cell_type": "code", "execution_count": 36, "id": "a10ffe38-e7a4-4ebb-a1a2-942dbe3b4b6c", "metadata": { "tags": [] }, "outputs": [], "source": [ "import collections\n", "\n", "examples = datasets[\"validation\"]\n", "features = validation_features\n", "\n", "example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n", "features_per_example = collections.defaultdict(list)\n", "for i, feature in enumerate(features):\n", " features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)" ] }, { "cell_type": "code", "execution_count": 39, "id": "cd5fb707-7964-4f95-b89b-0c408a33d3ea", "metadata": { "tags": [] }, "outputs": [], "source": [ "from tqdm.auto import tqdm\n", "import numpy as np\n", "\n", "def postprocess_qa_predictions(examples, features, raw_predictions, n_best_size = 20, max_answer_length = 30):\n", " all_start_logits, all_end_logits = raw_predictions\n", " # Build a map example to its corresponding features.\n", " example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n", " features_per_example = collections.defaultdict(list)\n", " for i, feature in enumerate(features):\n", " features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)\n", "\n", " # The dictionaries we have to fill.\n", " predictions = collections.OrderedDict()\n", "\n", " # Logging.\n", " print(f\"Post-processing {len(examples)} example predictions split into {len(features)} features.\")\n", "\n", " # Let's loop over all the examples!\n", " for example_index, example in enumerate(tqdm(examples)):\n", " # Those are the indices of the features associated to the current example.\n", " feature_indices = features_per_example[example_index]\n", "\n", " min_null_score = None # Only used if squad_v2 is True.\n", " valid_answers = []\n", " \n", " context = example[\"context\"]\n", " # Looping through all the features associated to the current example.\n", " for feature_index in feature_indices:\n", " # We grab the predictions of the model for this feature.\n", " start_logits = all_start_logits[feature_index]\n", " end_logits = all_end_logits[feature_index]\n", " # This is what will allow us to map some the positions in our logits to span of texts in the original\n", " # context.\n", " offset_mapping = features[feature_index][\"offset_mapping\"]\n", "\n", " # Update minimum null prediction.\n", " cls_index = features[feature_index][\"input_ids\"].index(tokenizer.cls_token_id)\n", " feature_null_score = start_logits[cls_index] + end_logits[cls_index]\n", " if min_null_score is None or min_null_score < feature_null_score:\n", " min_null_score = feature_null_score\n", "\n", " # Go through all possibilities for the `n_best_size` greater start and end logits.\n", " start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n", " end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n", " for start_index in start_indexes:\n", " for end_index in end_indexes:\n", " # Don't consider out-of-scope answers, either because the indices are out of bounds or correspond\n", " # to part of the input_ids that are not in the context.\n", " if (\n", " start_index >= len(offset_mapping)\n", " or end_index >= len(offset_mapping)\n", " or offset_mapping[start_index] is None\n", " or offset_mapping[end_index] is None\n", " ):\n", " continue\n", " # Don't consider answers with a length that is either < 0 or > max_answer_length.\n", " if end_index < start_index or end_index - start_index + 1 > max_answer_length:\n", " continue\n", "\n", " start_char = offset_mapping[start_index][0]\n", " end_char = offset_mapping[end_index][1]\n", " valid_answers.append(\n", " {\n", " \"score\": start_logits[start_index] + end_logits[end_index],\n", " \"text\": context[start_char: end_char]\n", " }\n", " )\n", " \n", " if len(valid_answers) > 0:\n", " best_answer = sorted(valid_answers, key=lambda x: x[\"score\"], reverse=True)[0]\n", " else:\n", " # In the very rare edge case we have not a single non-null prediction, we create a fake prediction to avoid\n", " # failure.\n", " best_answer = {\"text\": \"\", \"score\": 0.0}\n", " \n", " # Let's pick our final answer: the best one or the null answer (only for squad_v2)\n", " if not squad_v2:\n", " predictions[example[\"id\"]] = best_answer[\"text\"]\n", " else:\n", " answer = best_answer[\"text\"] if best_answer[\"score\"] > min_null_score else \"\"\n", " predictions[example[\"id\"]] = answer\n", "\n", " return predictions" ] }, { "cell_type": "code", "execution_count": 40, "id": "6a4dc323-1578-4457-b52e-4a59e4284848", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Post-processing 10570 example predictions split into 10788 features.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "100%|█████████████████████████████████████████████████████████████████████████████████████| 10570/10570 [00:35<00:00, 300.93it/s]\n" ] } ], "source": [ "final_predictions = postprocess_qa_predictions(datasets[\"validation\"], validation_features, raw_predictions.predictions)" ] }, { "cell_type": "markdown", "id": "65aafc05-806f-4c44-a2f3-926b87ce5348", "metadata": {}, "source": [ "Then we can load the metric from the datasets library." ] }, { "cell_type": "code", "execution_count": 41, "id": "c1f1c813-8aa2-4d35-9079-0fddd26b0b2e", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/tmp/ipykernel_88/2905994612.py:1: FutureWarning: load_metric is deprecated and will be removed in the next major version of datasets. Use 'evaluate.load' instead, from the new library 🤗 Evaluate: https://huggingface.co/docs/evaluate\n", " metric = load_metric(\"squad_v2\" if squad_v2 else \"squad\")\n", "Downloading builder script: 4.50kB [00:00, 2.50MB/s] \n", "Downloading extra modules: 3.31kB [00:00, 1.91MB/s] \n" ] } ], "source": [ "metric = load_metric(\"squad_v2\" if squad_v2 else \"squad\")" ] }, { "cell_type": "markdown", "id": "8566b4ff-5ab3-4aaf-83b6-a4467718965e", "metadata": {}, "source": [ "Then we can call compute on it. We just need to format predictions and labels a bit as it expects a list of dictionaries and not one big dictionary. In the case of squad_v2, we also have to set a `no_answer_probability` argument (which we set to 0.0 here as we have already set the answer to empty if we picked it)." ] }, { "cell_type": "code", "execution_count": 42, "id": "957587eb-3379-464b-89a8-95a9b7fd78cc", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "{'exact_match': 85.87511825922422, 'f1': 92.14116893007746}" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "if squad_v2:\n", " formatted_predictions = [{\"id\": k, \"prediction_text\": v, \"no_answer_probability\": 0.0} for k, v in final_predictions.items()]\n", "else:\n", " formatted_predictions = [{\"id\": k, \"prediction_text\": v} for k, v in final_predictions.items()]\n", "references = [{\"id\": ex[\"id\"], \"answers\": ex[\"answers\"]} for ex in datasets[\"validation\"]]\n", "metric.compute(predictions=formatted_predictions, references=references)" ] }, { "cell_type": "markdown", "id": "92987183-8246-4d1d-8294-cb8d4d2ceafa", "metadata": {}, "source": [ "## Inference of fine-tuned DeBERTa model" ] }, { "cell_type": "markdown", "id": "3bd6a5dd-8fc4-4d48-ae66-a182a2d236e9", "metadata": {}, "source": [ "In 🤗 Optimum, we also provides `ORTModel` API to ease the inference with ONNX Runtime backend. " ] }, { "cell_type": "code", "execution_count": 59, "id": "a0da9151-0c8d-43fd-952b-4642d1d5c378", "metadata": { "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Framework not specified. Using pt to export to ONNX.\n", "Using framework PyTorch: 1.13.1+cu116\n" ] } ], "source": [ "from optimum.onnxruntime import ORTModelForQuestionAnswering\n", "from transformers import AutoTokenizer, pipeline\n", "\n", "model = ORTModelForQuestionAnswering.from_pretrained(\"deberta-base-finetuned-squad\", export=True)\n", "tokenizer = AutoTokenizer.from_pretrained(\"deberta-base-finetuned-squad\")" ] }, { "cell_type": "code", "execution_count": 70, "id": "f2ccb59a-321a-481e-b7cb-726310801f49", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n", " 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])" ] }, "execution_count": 70, "metadata": {}, "output_type": "execute_result" } ], "source": [ "question, text = \"What's the benefit of using Optimum library?\", \"Optimum is an open-sourced library. It can accelerate the training speed. We recommend you to test it.\"\n", "inputs = tokenizer(question, text, return_tensors=\"pt\")\n", "inputs.pop(\"token_type_ids\")" ] }, { "cell_type": "code", "execution_count": 71, "id": "be1fbe24-4569-417f-a377-f17cc3075927", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "' accelerate the training speed'" ] }, "execution_count": 71, "metadata": {}, "output_type": "execute_result" } ], "source": [ "outputs = model(**inputs)\n", "answer_start_index = outputs.start_logits.argmax()\n", "answer_end_index = outputs.end_logits.argmax()\n", "\n", "predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]\n", "tokenizer.decode(predict_answer_tokens, skip_special_tokens=True)" ] }, { "cell_type": "code", "execution_count": null, "id": "b51ec58e-66ef-4b6f-9e46-063272010d67", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" } }, "nbformat": 4, "nbformat_minor": 5 }