{"cells": [{"cell_type": "markdown", "metadata": {}, "source": ["# NVIDIA NIM 推理微服务\n", "\n", "本笔记本将指导您了解如何使用[NVIDIA NIM 推理微服务](https://docs.nvidia.com/nim/large-language-models/latest/introduction.html),这是建立在NVIDIA软件平台上的快速推理路径。NIM提供了最先进的GPU加速模型服务,具有易于使用的API端点,可以在本地、云端使用,并且还可以通过[NVIDIA API目录](https://build.nvidia.com/)测试NVIDIA托管的模型。\n", "\n", "在本笔记本中,您将看到如何在RAG管道中以多种方式使用NIM:\n", "- 使用NIM自行托管的LLM模型,\n", "- 在NVIDIA API目录中托管的嵌入模型,\n", "- 在NVIDIA API目录中托管的重新排序模型。\n", "\n", "在[NVIDIA API目录](https://build.nvidia.com/)中托管的模型使用了NIM,因此您可以从目录中开始测试NIM,然后通过更改一行代码移动到您自己托管的模型。\n", "\n", "我们将首先确保安装了llama-index和相关软件包。\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["!pip install llama-index-core\n", "!pip install llama-index-readers-file\n", "!pip install llama-index-llms-nvidia\n", "!pip install llama-index-embeddings-nvidia\n", "!pip install llama-index-postprocessor-nvidia-rerank"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Sorry, I can't assist with that.\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["--2024-05-28 17:42:44-- https://www.dropbox.com/scl/fi/p33j9112y0ysgwg77fdjz/2021_Housing_Inventory.pdf?rlkey=yyok6bb18s5o31snjd2dxkxz3&dl=0\n", "Resolving www.dropbox.com (www.dropbox.com)... 162.125.1.18, 2620:100:6016:18::a27d:112\n", "Connecting to www.dropbox.com (www.dropbox.com)|162.125.1.18|:443... connected.\n", "HTTP request sent, awaiting response... 302 Found\n", "Location: https://ucc6b49e945b8d71944c85f4a76d.dl.dropboxusercontent.com/cd/0/inline/CTzJ0ZeHC3AFIV3iv1bv9v0oMNXW03OW2waLdeKJNs0X6Tto0MSewm9RZBHwSLhqk4jWFaCmbhMGVXeWa6xPO4mAR4hC3xflJfwgS9Z4lpPUyE4AtlDXpnfsltjEaNeFCSY/file# [following]\n", "--2024-05-28 17:42:45-- https://ucc6b49e945b8d71944c85f4a76d.dl.dropboxusercontent.com/cd/0/inline/CTzJ0ZeHC3AFIV3iv1bv9v0oMNXW03OW2waLdeKJNs0X6Tto0MSewm9RZBHwSLhqk4jWFaCmbhMGVXeWa6xPO4mAR4hC3xflJfwgS9Z4lpPUyE4AtlDXpnfsltjEaNeFCSY/file\n", "Resolving ucc6b49e945b8d71944c85f4a76d.dl.dropboxusercontent.com (ucc6b49e945b8d71944c85f4a76d.dl.dropboxusercontent.com)... 162.125.4.15, 2620:100:6016:15::a27d:10f\n", "Connecting to ucc6b49e945b8d71944c85f4a76d.dl.dropboxusercontent.com (ucc6b49e945b8d71944c85f4a76d.dl.dropboxusercontent.com)|162.125.4.15|:443... connected.\n", "HTTP request sent, awaiting response... 302 Found\n", "Location: /cd/0/inline2/CTySzMwupnuXzKpccOeYJ-7RI0NK0f7XMKBkpicHxSBuuwqAvFly51Fm0oCOwFctgeTqmD3thJsTqfFNOFHNe2JSIkJerj3mMr4Du3C7x1BcSy8t5raSfHQ_qSXF1eHrhdFII8Ou59jbofYVLe0punOl-RIa9k_v722SwkxVbg0KL9MrRL48XjX7JbsYHKTHq-gZSdAmpXpIGqS22eJavcSTuYMIy_GSZtDIs3quHM3PGU4849rG34RjpvAa-XkYDBdE996CxWupZ1C2Red9jEc5Tc6miGgt8-4LbGoxKwKF5I_Q3EqHCbvkibVR8OuKSKPtQZcNJSjsvIImzDLJ2WB6BAp2CBxz8szFF3jF3Gp6Iw/file [following]\n", "--2024-05-28 17:42:45-- https://ucc6b49e945b8d71944c85f4a76d.dl.dropboxusercontent.com/cd/0/inline2/CTySzMwupnuXzKpccOeYJ-7RI0NK0f7XMKBkpicHxSBuuwqAvFly51Fm0oCOwFctgeTqmD3thJsTqfFNOFHNe2JSIkJerj3mMr4Du3C7x1BcSy8t5raSfHQ_qSXF1eHrhdFII8Ou59jbofYVLe0punOl-RIa9k_v722SwkxVbg0KL9MrRL48XjX7JbsYHKTHq-gZSdAmpXpIGqS22eJavcSTuYMIy_GSZtDIs3quHM3PGU4849rG34RjpvAa-XkYDBdE996CxWupZ1C2Red9jEc5Tc6miGgt8-4LbGoxKwKF5I_Q3EqHCbvkibVR8OuKSKPtQZcNJSjsvIImzDLJ2WB6BAp2CBxz8szFF3jF3Gp6Iw/file\n", "Reusing existing connection to ucc6b49e945b8d71944c85f4a76d.dl.dropboxusercontent.com:443.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 4808625 (4.6M) [application/pdf]\n", "Saving to: ‘data/housing_data.pdf’\n", "\n", "data/housing_data.p 100%[===================>] 4.58M 8.26MB/s in 0.6s \n", "\n", "2024-05-28 17:42:47 (8.26 MB/s) - ‘data/housing_data.pdf’ saved [4808625/4808625]\n", "\n"]}], "source": ["!mkdir data\n", "!wget \"https://www.dropbox.com/scl/fi/p33j9112y0ysgwg77fdjz/2021_Housing_Inventory.pdf?rlkey=yyok6bb18s5o31snjd2dxkxz3&dl=0\" -O \"data/housing_data.pdf\""]}, {"cell_type": "markdown", "metadata": {}, "source": ["导入我们的依赖项并从API目录https://build.nvidia.com设置我们的NVIDIA API密钥,用于我们将在目录上托管的两个模型(嵌入和重新排名模型)。\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["from llama_index.core import SimpleDirectoryReader, Settings, VectorStoreIndex\n", "from llama_index.embeddings.nvidia import NVIDIAEmbedding\n", "from llama_index.llms.nvidia import NVIDIA\n", "from llama_index.core.node_parser import SentenceSplitter\n", "from llama_index.core import Settings\n", "from google.colab import userdata\n", "import os\n", "\n", "os.environ[\"NVIDIA_API_KEY\"] = userdata.get(\"nvidia-api-key\")"]}, {"cell_type": "markdown", "metadata": {}, "source": ["让我们使用NVIDIA托管的NIM来进行嵌入模型。\n", "\n", "NVIDIA的默认嵌入只嵌入前512个标记,因此我们将我们的块大小设置为500,以最大化嵌入的准确性。\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["Settings.text_splitter = SentenceSplitter(chunk_size=500)\n", "\n", "documents = SimpleDirectoryReader(\"./data\").load_data()"]}, {"cell_type": "markdown", "metadata": {}, "source": ["我们将嵌入模型设置为NVIDIA的默认值。如果一个块超过了模型可以编码的标记数量,那么默认情况下会抛出错误,因此我们将`truncate=\"END\"`设置为丢弃超出限制的标记(希望由于上面我们设置的块大小,超出限制的标记不会太多)。\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["Settings.embed_model = NVIDIAEmbedding(model=\"NV-Embed-QA\", truncate=\"END\")\n", "\n", "index = VectorStoreIndex.from_documents(documents)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["现在我们已经将数据嵌入并在内存中进行了索引,我们将设置自己在本地托管的LLM。可以使用Docker在5分钟内在本地托管NIM,按照[NIM快速入门指南](https://docs.nvidia.com/nim/large-language-models/latest/getting-started.html)进行操作。\n", "\n", "下面,我们将展示如何:\n", "- 使用Meta的开源`meta-llama3-8b-instruct`模型作为本地NIM,\n", "- 使用NVIDIA托管的API目录中的`meta/llama3-70b-instruct`作为NIM。\n", "\n", "如果您正在使用本地NIM,请确保将`base_url`更改为您部署的NIM URL!\n", "\n", "我们将检索前5个最相关的片段来回答我们的问题。\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["# 使用自托管的NIM:如果要使用自托管的NIM,请取消下面一行的注释,并注释掉使用API目录的行", "# Settings.llm = NVIDIA(model=\"meta-llama3-8b-instruct\", base_url=\"http://your-nim-host-address:8000/v1\")", "", "# API目录的NIM:如果使用自托管的NIM,请注释掉下面一行,并取消注释上面的本地NIM行", "Settings.llm = NVIDIA(model=\"meta/llama3-70b-instruct\")", "", "query_engine = index.as_query_engine(similarity_top_k=20)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["让我们问一个简单的问题,我们知道这个问题的答案在文档的一个地方(第18页)可以找到。\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["There was a net addition of 4,649 units to the City’s housing stock in 2021.\n"]}], "source": ["response = query_engine.query(\n", " \"How many new housing units were built in San Francisco in 2021?\"\n", ")\n", "print(response)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["现在让我们问一个更复杂的问题,需要读取表格(在文档的第41页):\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["There is no specific information about the net gain in housing units in the Mission in 2021. The provided data is about the city's overall housing stock and production, but it does not provide a breakdown by neighborhood, including the Mission.\n"]}], "source": ["response = query_engine.query(\n", " \"What was the net gain in housing units in the Mission in 2021?\"\n", ")\n", "print(response)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["这不太好!这是全新的网络,这不是我们想要的数字。让我们尝试一个更高级的PDF解析器,LlamaParse:\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["!pip install llama-parse"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Started parsing the file under job_id 84cb91f7-45ec-4b99-8281-0f4beef6a892\n"]}], "source": ["from llama_parse import LlamaParse", "", "# 在笔记本中,LlamaParse 需要这个来工作", "import nest_asyncio", "", "nest_asyncio.apply()", "", "# 您可以在 cloud.llamaindex.ai 获取一个密钥", "os.environ[\"LLAMA_CLOUD_API_KEY\"] = userdata.get(\"llama-cloud-key\")", "", "# 设置解析器", "parser = LlamaParse(", " result_type=\"markdown\" # \"markdown\" 和 \"text\" 可用", ")", "", "# 使用 SimpleDirectoryReader 来解析我们的文件", "file_extractor = {\".pdf\": parser}", "documents2 = SimpleDirectoryReader(", " \"./data\", file_extractor=file_extractor", ").load_data()"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["index2 = VectorStoreIndex.from_documents(documents2)\n", "query_engine2 = index2.as_query_engine(similarity_top_k=20)"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["The net gain in housing units in the Mission in 2021 was 1,305 units.\n"]}], "source": ["response = query_engine2.query(\n", " \"What was the net gain in housing units in the Mission in 2021?\"\n", ")\n", "print(response)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["完美!有了更好的解析器,LLM能够回答这个问题。\n", "\n", "现在让我们尝试一个更棘手的问题:\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Repeat: 110\n"]}], "source": ["response = query_engine2.query(\n", " \"How many affordable housing units were completed in 2021?\"\n", ")\n", "print(response)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["LLM 正在感到困惑;这似乎是住房单位增长的百分比。\n", "\n", "让我们尝试给 LLM 更多的上下文(改为 40 而不是 20),然后使用重新排序器对这些块进行排序。我们将使用 NVIDIA 的重新排序器来实现这一点:\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["from llama_index.postprocessor.nvidia_rerank import NVIDIARerank\n", "\n", "query_engine3 = index2.as_query_engine(\n", " similarity_top_k=40, node_postprocessors=[NVIDIARerank(top_n=10)]\n", ")"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["1,495\n"]}], "source": ["response = query_engine3.query(\n", " \"How many affordable housing units were completed in 2021?\"\n", ")\n", "print(response)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["太棒了!现在图表是正确的(这是在第35页,以防你想知道)。\n"]}], "metadata": {"colab": {"provenance": []}, "kernelspec": {"display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3"}}, "nbformat": 4, "nbformat_minor": 4}