{"cells": [{"cell_type": "markdown", "metadata": {}, "source": ["# Activeloop深度内存\n", "\n", "**我们如何在文档问答中获得+15%的RAG点击率改进？**\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["最近，检索增强生成器（RAGs）引起了广泛关注。随着先进的RAG技术和代理的出现，它们扩展了RAGs的潜力。然而，一些挑战可能限制了将RAGs整合到生产中。在实施RAGs到生产环境时，需要考虑的主要因素是准确性（召回率）、成本和延迟。对于基本用例，OpenAI的Ada模型配合简单的相似性搜索可以产生令人满意的结果。然而，对于搜索过程中更高的准确性或召回率，可能需要采用先进的检索技术。这些方法可能涉及不同的数据块大小、多次重写查询等，可能会增加延迟和成本。Activeloop的Deep Memory是Activeloop Deep Lake用户可以使用的功能，它通过引入一个微小的神经网络层，训练以将用户查询与语料库中相关数据匹配，解决了这些问题。虽然这种添加在搜索过程中带来了最小的延迟，但它可以将检索准确性提高高达27%，并且成本效益高，使用简单，无需额外的高级RAG技术。[Activeloop的Deep Memory](https://www.activeloop.ai/resources/use-deep-memory-to-boost-rag-apps-accuracy-by-up-to-22/)\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["%pip install llama-index-vector-stores-deeplake\n", "%pip install llama-index-llms-openai"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["import nest_asyncio\n", "import os\n", "import getpass\n", "\n", "nest_asyncio.apply()"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["!pip install deeplake beautifulsoup4 html2text tiktoken openai llama-index python-dotenv"]}, {"cell_type": "markdown", "metadata": {}, "source": ["对于本教程，我们将解析deeplake文档，并创建一个能够回答文档中问题的RAG系统。\n", "\n", "教程可以分为几个部分：\n", "1. [数据集创建和上传](#1-dataset-creation-and-ingestion)\n", "2. [生成合成查询并训练deep_memory](#2-training-deep-memory)\n", "3. [评估deep memory性能](#3-deepmemory-evaluation)\n", "4. [Deep Memory推理](#4-deep-memory-inference)\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["\n", "## 1. 数据集创建和导入\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["让我使用BeautifulSoup解析所有链接，并将它们转换为LlamaIndex文档：\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["import requests\n", "from bs4 import BeautifulSoup\n", "from urllib.parse import urljoin\n", "\n", "def get_all_links(url):\n", " response = requests.get(url)\n", " if response.status_code != 200:\n", " print(f\"无法获取页面：{url}\")\n", " return []\n", "\n", " soup = BeautifulSoup(response.content, \"html.parser\")\n", "\n", " # 查找所有包含链接的'a'标签，通常包含href属性\n", " links = [\n", " urljoin(url, a[\"href\"])\n", " for a in soup.find_all(\"a\", href=True)\n", " if a[\"href\"]\n", " ]\n", "\n", " return links"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Fetching pages: 100%|##########| 120/120 [00:13<00:00, 8.70it/s]\n"]}], "source": ["from langchain.document_loaders import AsyncHtmlLoader\n", "from langchain.document_transformers import Html2TextTransformer\n", "from llama_index.core import Document\n", "\n", "\n", "def load_documents(url):\n", " all_links = get_all_links(url)\n", " loader = AsyncHtmlLoader(all_links)\n", " docs = loader.load()\n", "\n", " html2text = Html2TextTransformer()\n", " docs_transformed = html2text.transform_documents(docs)\n", " docs = [Document.from_langchain_format(doc) for doc in docs_transformed]\n", " return docs\n", "\n", "\n", "docs = load_documents(\"https://docs.deeplake.ai/en/latest/\")"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"data": {"text/plain": ["120"]}, "execution_count": null, "metadata": {}, "output_type": "execute_result"}], "source": ["len(docs)"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Deep Lake Dataset in hub://activeloop-test/deeplake_docs_deepmemory2 already exists, loading from the storage\n"]}], "source": ["from llama_index.core.evaluation import generate_question_context_pairs\n", "from llama_index.core import (\n", " VectorStoreIndex,\n", " SimpleDirectoryReader,\n", " StorageContext,\n", ")\n", "from llama_index.vector_stores.deeplake import DeepLakeVectorStore\n", "from llama_index.core.node_parser import SimpleNodeParser\n", "from llama_index.llms.openai import OpenAI\n", "\n", "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"输入您的OpenAI API密钥：\")\n", "# # 如果您没有使用CLI登录，则需要activeloop令牌：`activeloop login -u -p `\n", "os.environ[\"ACTIVELOOP_TOKEN\"] = getpass.getpass(\n", " \"输入您的ActiveLoop API令牌：\"\n", ") # 从https://app.activeloop.ai获取您的API令牌，在右上角的个人资料图片中，选择“API Tokens”\n", "token = os.getenv(\"ACTIVELOOP_TOKEN\")\n", "\n", "vector_store = DeepLakeVectorStore(\n", " dataset_path=\"hub://activeloop-test/deeplake_docs_deepmemory2\",\n", " overwrite=False, # 设置为True以覆盖现有数据集\n", " runtime={\"tensor_db\": True},\n", " token=token,\n", ")"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["def create_modules(vector_store, docs=[], populate_vector_store=True):\n", " if populate_vector_store:\n", " node_parser = SimpleNodeParser.from_defaults(chunk_size=512)\n", " nodes = node_parser.get_nodes_from_documents(docs)\n", " else:\n", " nodes = []\n", "\n", " # 默认情况下，节点的id设置为随机uuid。为了确保每次运行相同的id，我们手动设置它们。\n", " for idx, node in enumerate(nodes):\n", " node.id_ = f\"node_{idx}\"\n", "\n", " llm = OpenAI(model=\"gpt-4\")\n", " storage_context = StorageContext.from_defaults(vector_store=vector_store)\n", " return storage_context, nodes, llm"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["(\n", " storage_context,\n", " nodes,\n", " llm,\n", ") = create_modules(\n", " docs=docs,\n", " vector_store=vector_store,\n", " # populate_vector_store=False, # 取消注释此行以跳过填充向量存储\n", ")"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["vector_index = VectorStoreIndex(nodes, storage_context=storage_context)\n", "deep_memory_retriever = vector_index.as_retriever(\n", " similarity_top_k=4, deep_memory=True\n", ")"]}, {"cell_type": "markdown", "metadata": {}, "source": ["\n", "## 2. 训练深度记忆网络\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["在上面，我们展示了deep_memory工作的整体架构。因此，您可以看到，为了训练它，您需要相关性、查询以及语料库数据（我们想要查询的数据）。语料库数据已经在上一节中填充；在这里，我们将生成问题和相关性。\n", "\n", "1. `questions` - 是一组文本字符串，其中每个字符串代表一个查询。\n", "2. `relevance` - 包含每个问题的地面真相链接。可能有几个文档包含对给定问题的答案。因此，相关性是一个List[List[tuple[str, float]]]，其中外部列表表示查询，内部列表表示相关文档。元组包含一个str、float对，其中字符串表示源文档的id（对应数据集中的id张量），而浮点数表示当前文档与问题的相关程度。\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["from llama_index.core.evaluation import (\n", " generate_question_context_pairs,\n", " EmbeddingQAFinetuneDataset,\n", ")\n", "import random\n", "\n", "\n", "def create_train_test_datasets(\n", " number_of_samples=600, llm=None, nodes=None, save=False\n", "):\n", " random_indices = random.sample(range(len(nodes)), number_of_samples)\n", "\n", " ratio = int(len(random_indices) * 0.8)\n", "\n", " train_indices = random_indices[:ratio]\n", " test_indices = random_indices[ratio:]\n", "\n", " train_nodes = [nodes[i] for i in train_indices]\n", " test_nodes = [nodes[i] for i in test_indices]\n", "\n", " train_qa_dataset = generate_question_context_pairs(\n", " train_nodes, llm=llm, num_questions_per_chunk=1\n", " )\n", "\n", " test_qa_dataset = generate_question_context_pairs(\n", " test_nodes, llm=llm, num_questions_per_chunk=1\n", " )\n", "\n", " # [optional] save\n", " if save:\n", " train_qa_dataset.save_json(\n", " f\"deeplake_docs_{number_of_samples}_train.json\"\n", " )\n", " test_qa_dataset.save_json(\n", " f\"deeplake_docs_{number_of_samples}_test.json\"\n", " )\n", " return train_qa_dataset, test_qa_dataset"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"name": "stderr", "output_type": "stream", "text": [" 4%|▍ | 19/480 [02:25<1:04:00, 8.33s/it]"]}], "source": ["train_qa_dataset, test_qa_dataset = create_train_test_datasets(\n", " number_of_samples=600, llm=llm, nodes=nodes, save=True\n", ")"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["train_qa_dataset = EmbeddingQAFinetuneDataset.from_json(\n", " \"deeplake_docs_600_train.json\"\n", ")\n", "test_qa_dataset = EmbeddingQAFinetuneDataset.from_json(\n", " \"deeplake_docs_600_test.json\"\n", ")"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["def create_query_relevance(qa_dataset):\n", " \"\"\"将llama-index数据集转换为深度记忆训练的正确格式的函数\"\"\"\n", " queries = [text for _, text in qa_dataset.queries.items()]\n", " relevant_docs = qa_dataset.relevant_docs\n", " relevance = []\n", " for doc in relevant_docs:\n", " relevance.append([(relevant_docs[doc][0], 1)])\n", " return queries, relevance"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["train_queries, train_relevance = create_query_relevance(train_qa_dataset)\n", "test_queries, test_relevance = create_query_relevance(test_qa_dataset)"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"data": {"text/plain": ["['In the context of creating a bounding box tensor in a dataset, explain the significance of the \"coords\" argument and its keys \"type\" and \"mode\". What does the \"type\" key specify about the bounding box coordinates?',\n", " 'Explain the process of creating an intrinsics tensor and appending intrinsics matrices in the context of computer vision. What are the dimensions of the intrinsics parameters and what do they represent? Also, describe the concept of a Segmentation Mask Htype and its role in image processing.',\n", " 'In the context of querying for images in the MNIST Train Dataset using `ds.query`, what does the command \"select * where labels == 0\" signify and what is the expected output?']"]}, "execution_count": null, "metadata": {}, "output_type": "execute_result"}], "source": ["train_queries[:3]"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"data": {"text/plain": ["[[('node_788', 1)], [('node_861', 1)], [('node_82', 1)]]"]}, "execution_count": null, "metadata": {}, "output_type": "execute_result"}], "source": ["train_relevance[:3]"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"data": {"text/plain": ["['What are the steps to update the information of keypoints and connections in a tensor, and what types of data can be appended to keypoints?',\n", " 'What is the command to create a mesh tensor in DeepLake and what are the supported compressions? Also, explain how to append a ply file containing mesh data to this tensor.',\n", " 'What is a Sequence htype in the context of tensors and how does it function as a wrapper for other htypes? Provide examples.']"]}, "execution_count": null, "metadata": {}, "output_type": "execute_result"}], "source": ["test_queries[:3]"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"data": {"text/plain": ["[[('node_933', 1)], [('node_671', 1)], [('node_471', 1)]]"]}, "execution_count": null, "metadata": {}, "output_type": "execute_result"}], "source": ["test_relevance[:3]"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Starting DeepMemory training job\n", "Your Deep Lake dataset has been successfully created!\n"]}, {"name": "stderr", "output_type": "stream", "text": []}, {"name": "stdout", "output_type": "stream", "text": ["Preparing training data for deepmemory:\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Creating 483 embeddings in 1 batches of size 483:: 100%|██████████| 1/1 [00:03<00:00, 3.67s/it]\n"]}, {"name": "stdout", "output_type": "stream", "text": ["DeepMemory training job started. Job ID: 65421a5003888c9ca36c72e8\n"]}], "source": ["from langchain.embeddings.openai import OpenAIEmbeddings\n", "\n", "embeddings = OpenAIEmbeddings()\n", "\n", "\n", "job_id = vector_store.vectorstore.deep_memory.train(\n", " queries=train_queries,\n", " relevance=train_relevance,\n", " embedding_function=embeddings.embed_documents,\n", ")"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/adilkhan/deeplake_docs_deepmemory2\n", "--------------------------------------------------------------\n", "| 65421a5003888c9ca36c72e8 |\n", "--------------------------------------------------------------\n", "| status | completed |\n", "--------------------------------------------------------------\n", "| progress | eta: 12.2 seconds |\n", "| | recall@10: 67.01% (+18.56%) |\n", "--------------------------------------------------------------\n", "| results | recall@10: 67.01% (+18.56%) |\n", "--------------------------------------------------------------\n", "\n", "\n"]}], "source": ["vector_store.vectorstore.deep_memory.status(job_id)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["\n", "## 3. DeepMemory 模型评估\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["太棒了！训练取得了一些显著的改进！现在，让我们评估它在测试集上的性能。\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["\u001b[37minfo \u001b[35mWed Nov 1 09:32:44 2023 GMT \u001b[0m Added distance metric `deepmemory_distance`.\n", "Embedding queries took 0.95 seconds\n", "---- Evaluating without Deep Memory ---- \n", "Recall@1:\t 12.5%\n", "Recall@3:\t 23.3%\n", "Recall@5:\t 30.8%\n", "Recall@10:\t 50.8%\n", "Recall@50:\t 94.2%\n", "Recall@100:\t 95.8%\n", "---- Evaluating with Deep Memory ---- \n", "Recall@1:\t 11.7%\n", "Recall@3:\t 27.5%\n", "Recall@5:\t 40.8%\n", "Recall@10:\t 65.0%\n", "Recall@50:\t 96.7%\n", "Recall@100:\t 98.3%\n"]}], "source": ["recalls = vector_store.vectorstore.deep_memory.evaluate(\n", " queries=test_queries,\n", " relevance=test_relevance,\n", " embedding_function=embeddings.embed_documents,\n", ")"]}, {"cell_type": "markdown", "metadata": {}, "source": ["令人印象深刻！我们观察到测试集上召回率增加了15%。接下来，让我们使用RetrieverEvaluator来检查MRR（平均倒数排名）和命中率。\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["import pandas as pd\n", "\n", "\n", "def display_results(eval_results):\n", " \"\"\"显示evaluate的结果。\"\"\"\n", " hit_rates = []\n", " mrrs = []\n", " names = []\n", " for name, eval_result in eval_results.items():\n", " metric_dicts = []\n", " for er in eval_result:\n", " metric_dict = er.metric_vals_dict\n", " metric_dicts.append(metric_dict)\n", "\n", " full_df = pd.DataFrame(metric_dicts)\n", "\n", " hit_rate = full_df[\"hit_rate\"].mean()\n", " mrr = full_df[\"mrr\"].mean()\n", "\n", " hit_rates.append(hit_rate)\n", " mrrs.append(mrr)\n", " names.append(name)\n", "\n", " metric_df = pd.DataFrame(\n", " [\n", " {\"retrievers\": names[i], \"hit_rate\": hit_rates[i], \"mrr\": mrrs[i]}\n", " for i in range(2)\n", " ],\n", " )\n", "\n", " return metric_df"]}, {"cell_type": "markdown", "metadata": {}, "source": ["评估使用深度记忆进行检索的性能：\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["from llama_index.core.evaluation import RetrieverEvaluator\n", "\n", "deep_memory_retriever = vector_index.as_retriever(\n", " similarity_top_k=10, vector_store_kwargs={\"deep_memory\": True}\n", ")\n", "dm_retriever_evaluator = RetrieverEvaluator.from_metric_names(\n", " [\"mrr\", \"hit_rate\"], retriever=deep_memory_retriever\n", ")\n", "\n", "dm_eval_results = await dm_retriever_evaluator.aevaluate_dataset(\n", " test_qa_dataset, retriever=dm_retriever_evaluator\n", ")"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": ["from llama_index.core.evaluation import RetrieverEvaluator\n", "\n", "naive_retriever = vector_index.as_retriever(similarity_top_k=10)\n", "naive_retriever_evaluator = RetrieverEvaluator.from_metric_names(\n", " [\"mrr\", \"hit_rate\"], retriever=naive_retriever\n", ")\n", "\n", "naive_eval_results = await naive_retriever_evaluator.aevaluate_dataset(\n", " test_qa_dataset, retriever=naive_retriever\n", ")"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"data": {"text/html": ["
\n", "\n", "
\n", " \n", "
\n", "
\n", "
retrievers
\n", "
hit_rate
\n", "
mrr
\n", "
\n", " \n", " \n", "
\n", "
0
\n", "
with with Deep Memory top-10 eval
\n", "
0.650000
\n", "
0.244775
\n", "
\n", "
\n", "
1
\n", "
without with Deep Memory top-10 eval
\n", "
0.508333
\n", "
0.215129
\n", "
\n", " \n", "
\n", "
"], "text/plain": [" retrievers hit_rate mrr\n", "0 with with Deep Memory top-10 eval 0.650000 0.244775\n", "1 without with Deep Memory top-10 eval 0.508333 0.215129"]}, "execution_count": null, "metadata": {}, "output_type": "execute_result"}], "source": ["eval_results = {\n", " f\"{mode} with Deep Memory top-10 eval\": eval_result\n", " for mode, eval_result in zip(\n", " [\"with\", \"without\"], [dm_eval_results, naive_eval_results]\n", " )\n", "}\n", "\n", "display_results(eval_results)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["不仅命中率增加了，而且MRR也增加了。\n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["\n", "## 4. 深度记忆推断\n"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["\u001b[37minfo \u001b[35mWed Nov 1 11:37:33 2023 GMT \u001b[0m Can't find any metric in the dataset.\n", "You can connect your own storage to deeplake by using the `connect()` function in the deeplake API.\n"]}], "source": ["query_engine = vector_index.as_query_engine(\n", " vector_store_kwargs={\"deep_memory\": True}, llm=llm\n", ")\n", "response = query_engine.query(\n", " \"How can you connect your own storage to the deeplake?\"\n", ")\n", "print(response)"]}, {"cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["The context does not provide information on how to connect your own storage to Deep Lake.\n"]}], "source": ["query_engine = vector_index.as_query_engine(\n", " vector_store_kwargs={\"deep_memory\": False}, llm=llm\n", ")\n", "response = query_engine.query(\n", " \"How can you connect your own storage to the deeplake?\"\n", ")\n", "print(response)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["根据我们的观察，没有“深度记忆”的情况下，我们的模型往往会产生不准确的结果，因为它检索到了错误的上下文。\n"]}], "metadata": {"kernelspec": {"display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3"}}, "nbformat": 4, "nbformat_minor": 4}