{"cells": [{"cell_type": "markdown", "id": "cb1537e6", "metadata": {}, "source": ["# 使用Weaviate与OpenAI向量化模块进行混合搜索\n", "\n", "本笔记本适用于以下场景：\n", "* 您的数据尚未进行向量化\n", "* 您希望在数据上运行混合搜索（[了解更多](https://weaviate.io/blog/hybrid-search-explained)）\n", "* 您希望使用Weaviate与OpenAI模块（[text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai)）为您生成向量嵌入。\n", "\n", "本笔记本将带您完成一个简单的流程，设置Weaviate实例，连接到它（使用OpenAI API密钥），配置数据模式，导入数据（这将自动生成您数据的向量嵌入），并运行混合搜索（向量搜索和BM25搜索的混合）。\n", "\n", "这是客户常见的需求，他们希望在安全环境中存储和搜索我们的嵌入，并支持生产用例，如聊天机器人、主题建模等。\n", "\n", "## 什么是Weaviate\n", "\n", "Weaviate是一个开源的向量搜索引擎，它将数据对象与它们的向量一起存储。这允许将向量搜索与结构化过滤相结合。\n", "\n", "Weaviate使用KNN算法创建一个向量优化的索引，使您的查询运行非常快。了解更多信息，请点击[这里](https://weaviate.io/blog/why-is-vector-search-so-fast)。\n", "\n", "Weaviate让您可以使用您喜欢的ML模型，并且可以无缝扩展到数十亿个数据对象。\n", "\n", "### 部署选项\n", "\n", "无论您的场景或生产设置如何，Weaviate都有适合您的选项。您可以在以下设置中部署Weaviate：\n", "* 自托管 - 您可以在本地使用docker部署Weaviate，或者在任何您想要的服务器上部署。\n", "* SaaS - 您可以使用[Weaviate云服务（WCS）](https://console.weaviate.io/)来托管您的Weaviate实例。\n", "* 混合SaaS - 您可以在您自己的私有云服务中部署Weaviate。\n", "\n", "### 编程语言\n", "\n", "Weaviate提供四种[客户端库](https://weaviate.io/developers/weaviate/client-libraries)，允许您从您的应用程序进行通信：\n", "* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)\n", "* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)\n", "* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)\n", "* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)\n", "\n", "此外，Weaviate还有一个[REST层](https://weaviate.io/developers/weaviate/api/rest/objects)。基本上，您可以从支持REST请求的任何语言调用Weaviate。\n"]}, {"cell_type": "markdown", "id": "45956173", "metadata": {}, "source": ["## 演示流程\n", "演示流程如下：\n", "- **先决条件设置**：创建一个Weaviate实例并安装所需的库\n", "- **连接**：连接到您的Weaviate实例\n", "- **模式配置**：配置数据的模式\n", "    - *注意*：在这里我们可以定义要使用哪个OpenAI嵌入模型\n", "    - *注意*：在这里我们可以配置要索引哪些属性\n", "- **导入数据**：加载演示数据集并将其导入Weaviate\n", "    - *注意*：导入过程将根据模式中的配置自动索引您的数据\n", "    - *注意*：您不需要显式地对数据进行向量化，Weaviate将与OpenAI通信来为您执行此操作\n", "- **运行查询**：查询\n", "    - *注意*：您不需要显式地对查询进行向量化，Weaviate将与OpenAI通信来为您执行此操作\n", "\n", "完成本笔记后，您应该对如何设置和使用向量数据库有基本的了解，并可以继续进行更复杂的用例，利用我们的嵌入。\n"]}, {"cell_type": "markdown", "id": "2a4a145e", "metadata": {}, "source": ["## Weaviate中的OpenAI模块\n", "所有Weaviate实例都配备有[text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai)模块。\n", "\n", "该模块负责在导入（或任何CRUD操作）期间处理向量化，以及在运行查询时。\n", "\n", "### 无需手动向量化数据\n", "这对您来说是个好消息。使用[text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai)，您无需手动向量化您的数据，因为Weaviate会在必要时为您调用OpenAI。\n", "\n", "您只需要：\n", "1. 在连接到Weaviate客户端时提供您的OpenAI API密钥\n", "2. 在模式中定义要使用的OpenAI向量化器\n"]}, {"cell_type": "markdown", "id": "f1a618c5", "metadata": {}, "source": ["## 先决条件\n", "\n", "在开始这个项目之前，我们需要设置以下内容：\n", "\n", "* 创建一个 `Weaviate` 实例\n", "* 安装库\n", "    * `weaviate-client`\n", "    * `datasets`\n", "    * `apache-beam`\n", "* 获取您的[OpenAI API密钥](https://beta.openai.com/account/api-keys)\n", "\n", "===========================================================\n", "### 创建 Weaviate 实例\n", "\n", "要创建 Weaviate 实例，我们有两个选项：\n", "\n", "1. （推荐路径）[Weaviate 云服务](https://console.weaviate.io/) - 在云中托管您的 Weaviate 实例。免费的沙箱应该足够用于这个示例。\n", "2. 使用 Docker 在本地安装和运行 Weaviate。\n", "\n", "#### 选项 1 - WCS 安装步骤\n", "\n", "使用 [Weaviate 云服务](https://console.weaviate.io/)（WCS）创建一个免费的 Weaviate 集群。\n", "1. 创建一个免费账户并/或登录 [WCS](https://console.weaviate.io/)\n", "2. 使用以下设置创建一个 `Weaviate 集群`：\n", "    * 沙箱：`Sandbox Free`\n", "    * Weaviate 版本：使用默认值（最新版）\n", "    * OIDC 认证：`Disabled`\n", "3. 您的实例应该在一两分钟内准备就绪\n", "4. 记下 `集群 ID`。链接将带您到您集群的完整路径（稍后您将需要连接到它）。它应该类似于：`https://your-project-name.weaviate.network`\n", "\n", "#### 选项 2 - 使用 Docker 在本地安装 Weaviate 实例\n", "\n", "使用 Docker 在本地安装和运行 Weaviate。\n", "1. 下载 [./docker-compose.yml](./docker-compose.yml) 文件\n", "2. 然后打开您的终端，导航到 docker-compose.yml 文件所在的位置，并使用以下命令启动 Docker：`docker-compose up -d`\n", "3. 一旦准备就绪，您的实例应该在 [http://localhost:8080](http://localhost:8080) 上可用\n", "\n", "注意：要关闭 Docker 实例，您可以运行：`docker-compose down`\n", "\n", "##### 了解更多\n", "要了解更多关于使用 Docker 部署 Weaviate 的信息，请参阅[安装文档](https://weaviate.io/developers/weaviate/installation/docker-compose)。\n"]}, {"cell_type": "markdown", "id": "b9babafe", "metadata": {}, "source": ["===========================================================\n", "## 安装所需库\n", "\n", "在运行此项目之前，请确保安装以下库：\n", "\n", "### Weaviate Python客户端\n", "\n", "[Weaviate Python客户端](https://weaviate.io/developers/weaviate/client-libraries/python) 允许您从Python项目中与Weaviate实例进行通信。\n", "\n", "### datasets & apache-beam\n", "\n", "要加载示例数据，您需要 `datasets` 库及其依赖项 `apache-beam`。\n"]}, {"cell_type": "code", "execution_count": null, "id": "2b04113f", "metadata": {}, "outputs": [], "source": ["# 安装适用于 Python 的 Weaviate 客户端\n", "!pip install weaviate-client>3.11.0\n", "\n", "# 安装数据集和apache-beam以加载示例数据集\n", "!pip install datasets apache-beam\n"]}, {"cell_type": "markdown", "id": "36fe86f4", "metadata": {}, "source": ["===========================================================\n", "## 准备您的OpenAI API密钥\n", "\n", "`OpenAI API密钥` 用于在导入数据时对数据进行向量化，并用于运行查询。\n", "\n", "如果您没有OpenAI API密钥，可以从[https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys)获取一个。\n", "\n", "获取密钥后，请将其添加到您的环境变量中，名称为 `OPENAI_API_KEY`。\n"]}, {"cell_type": "code", "execution_count": null, "id": "09fefff0", "metadata": {}, "outputs": [], "source": ["# 导出 OpenAI API 密钥\n", "!export OPENAI_API_KEY=\"your key\"\n"]}, {"cell_type": "code", "execution_count": null, "id": "88be138c", "metadata": {}, "outputs": [], "source": ["# 测试您的OpenAI API密钥是否已正确设置为环境变量。\n", "# 注意：如果您在本地运行此笔记本，您需要重新加载终端和笔记本，以使环境变量生效。\n", "import os\n", "\n", "# 注意：或者，您也可以像这样设置一个临时的环境变量：\n", "# os.environ['OPENAI_API_KEY'] = 'your-key-goes-here'\n", "\n", "if os.getenv(\"OPENAI_API_KEY\") is not None:\n", "    print (\"OPENAI_API_KEY is ready\")\n", "else:\n", "    print (\"OPENAI_API_KEY environment variable not found\")\n"]}, {"cell_type": "markdown", "id": "91df4d5b", "metadata": {}, "source": ["## 连接到您的Weaviate实例\n", "\n", "在本节中，我们将：\n", "\n", "1. 测试环境变量`OPENAI_API_KEY` - **确保**您已完成[#准备您的OpenAI API密钥](#准备您的OpenAI API密钥)中的步骤\n", "2. 使用您的`OpenAI API密钥`连接到您的Weaviate\n", "3. 并测试客户端连接\n", "\n", "### 客户端\n", "\n", "完成这一步之后，`client`对象将被用于执行所有与Weaviate相关的操作。\n"]}, {"cell_type": "code", "execution_count": null, "id": "cc662c1b", "metadata": {}, "outputs": [], "source": ["import weaviate\n", "from datasets import load_dataset\n", "import os\n", "\n", "# 连接到您的 Weaviate 实例\n", "client = weaviate.Client(\n", "    url=\"https://your-wcs-instance-name.weaviate.network/\",\n", "#   url=\"http://localhost:8080/\",\n", "    auth_client_secret=weaviate.auth.AuthApiKey(api_key=\"<YOUR-WEAVIATE-API-KEY>\"), # 如果您没有为您的 Weaviate 实例使用身份验证（例如，对于本地部署的实例），请注释掉这一行。\n", "    additional_headers={\n", "        \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n", "    }\n", ")\n", "\n", "# 检查您的实例是否已上线并准备就绪\n", "# 这将返回 `True`\n", "client.is_ready()\n"]}, {"cell_type": "markdown", "id": "7d3dac3c", "metadata": {}, "source": ["# 数据模式\n", "\n", "在本节中，我们将：\n", "1. 配置数据的模式\n", "2. 选择 OpenAI 模块\n", "\n", "> 这是第二步，也是最后一步，需要进行 OpenAI 特定的配置。\n", "> 在这一步之后，其余的说明将只涉及到 Weaviate，因为 OpenAI 任务将被自动处理。\n", "\n", "\n", "## 什么是模式\n", "\n", "在 Weaviate 中，您创建 __模式__ 来捕捉您将要搜索的每个实体。\n", "\n", "模式是您告诉 Weaviate 的方式：\n", "* 应该使用什么嵌入模型来对数据进行向量化\n", "* 您的数据由什么组成（属性名称和类型）\n", "* 哪些属性应该被向量化和索引化\n", "\n", "在本教程中，我们将使用一个包含以下内容的 `Articles` 数据集：\n", "* `title`\n", "* `content`\n", "* `url`\n", "\n", "我们希望对 `title` 和 `content` 进行向量化，但不包括 `url`。\n", "\n", "为了对数据进行向量化和查询，我们将使用 `text-embedding-3-small`。\n"]}, {"cell_type": "code", "execution_count": null, "id": "f894b911", "metadata": {"scrolled": true}, "outputs": [], "source": ["# 清理架构，以便我们能够重新创建它。\n", "client.schema.delete_all()\n", "client.schema.get()\n", "\n", "# 定义Schema对象，对`title`和`content`字段使用`text-embedding-3-small`进行处理，但跳过`url`字段。\n", "article_schema = {\n", "    \"class\": \"Article\",\n", "    \"description\": \"A collection of articles\",\n", "    \"vectorizer\": \"text2vec-openai\",\n", "    \"moduleConfig\": {\n", "        \"text2vec-openai\": {\n", "          \"model\": \"ada\",\n", "          \"modelVersion\": \"002\",\n", "          \"type\": \"text\"\n", "        }\n", "    },\n", "    \"properties\": [{\n", "        \"name\": \"title\",\n", "        \"description\": \"Title of the article\",\n", "        \"dataType\": [\"string\"]\n", "    },\n", "    {\n", "        \"name\": \"content\",\n", "        \"description\": \"Contents of the article\",\n", "        \"dataType\": [\"text\"]\n", "    },\n", "    {\n", "        \"name\": \"url\",\n", "        \"description\": \"URL to the article\",\n", "        \"dataType\": [\"string\"],\n", "        \"moduleConfig\": { \"text2vec-openai\": { \"skip\": True } }\n", "    }]\n", "}\n", "\n", "# 添加文章模式\n", "client.schema.create_class(article_schema)\n", "\n", "# 获取架构以确保其正常工作\n", "client.schema.get()\n"]}, {"cell_type": "markdown", "id": "e5d9d2e1", "metadata": {}, "source": ["## 导入数据\n", "\n", "在本节中，我们将：\n", "1. 加载Simple Wikipedia数据集\n", "2. 配置Weaviate批量导入（以使导入更有效）\n", "3. 将数据导入Weaviate\n", "\n", "> 注意：<br/>\n", "> 如前所述。我们不需要手动对数据进行向量化。<br/>\n", "> [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) 模块会处理这部分。\n"]}, {"cell_type": "code", "execution_count": null, "id": "fc3efadd", "metadata": {}, "outputs": [], "source": ["# ## 步骤1 - 加载数据集\n", "\n", "from datasets import load_dataset\n", "from typing import List, Iterator\n", "\n", "# 我们将使用datasets库来提取Simple Wikipedia数据集以进行嵌入。\n", "dataset = list(load_dataset(\"wikipedia\", \"20220301.simple\")[\"train\"])\n", "\n", "# 在测试阶段，为了演示目的，仅限于2500篇文章。\n", "dataset = dataset[:2_500]\n", "\n", "# 为了进行更大规模的演示，限制在2.5万篇文章以内\n", "# 数据集 = 数据集[:25_000]\n", "\n", "# 对于免费的OpenAI账户，您可以使用50个对象。\n", "# 数据集 = 数据集[:50]\n"]}, {"cell_type": "code", "execution_count": null, "id": "5044da96", "metadata": {}, "outputs": [], "source": ["# ##步骤2 - 配置Weaviate批处理功能\n", "# - 初始批次大小为100\n", "# - 根据性能动态增减\n", "# - 如果出现问题，增加超时重试机制\n", "\n", "client.batch.configure(\n", "    batch_size=10, \n", "    dynamic=True,\n", "    timeout_retries=3,\n", "# 回调函数=None,\n", ")\n"]}, {"cell_type": "code", "execution_count": null, "id": "15db8380", "metadata": {}, "outputs": [], "source": ["# ##步骤3 - 导入数据\n", "\n", "print(\"Importing Articles\")\n", "\n", "counter=0\n", "\n", "with client.batch as batch:\n", "    for article in dataset:\n", "        if (counter %10 == 0):\n", "            print(f\"Import {counter} / {len(dataset)} \")\n", "\n", "        properties = {\n", "            \"title\": article[\"title\"],\n", "            \"content\": article[\"text\"],\n", "            \"url\": article[\"url\"]\n", "        }\n", "        \n", "        batch.add_data_object(properties, \"Article\")\n", "        counter = counter+1\n", "\n", "print(\"Importing Articles complete\")       \n"]}, {"cell_type": "code", "execution_count": null, "id": "3658693c", "metadata": {}, "outputs": [], "source": ["# 测试所有数据是否已加载 —— 获取对象数量\n", "result = (\n", "    client.query.aggregate(\"Article\")\n", "    .with_fields(\"meta { count }\")\n", "    .do()\n", ")\n", "print(\"Object count: \", result[\"data\"][\"Aggregate\"][\"Article\"], \"\\n\")\n"]}, {"cell_type": "code", "execution_count": null, "id": "0d791186", "metadata": {}, "outputs": [], "source": ["# 通过检查一个对象，测试一篇文章已经成功运作。\n", "test_article = (\n", "    client.query\n", "    .get(\"Article\", [\"title\", \"url\", \"content\"])\n", "    .with_limit(1)\n", "    .do()\n", ")[\"data\"][\"Get\"][\"Article\"][0]\n", "\n", "print(test_article['title'])\n", "print(test_article['url'])\n", "print(test_article['content'])\n"]}, {"cell_type": "markdown", "id": "46050ca9", "metadata": {}, "source": ["### 搜索数据\n", "\n", "与上面类似，我们将向我们的新索引发送一些查询，并根据与现有向量的接近程度返回结果。\n", "\n", "在[这里](https://weaviate.io/developers/weaviate/api/graphql/vector-search-parameters#hybrid)了解更多关于`alpha`设置的信息。\n"]}, {"cell_type": "code", "execution_count": null, "id": "b044aa93", "metadata": {}, "outputs": [], "source": ["def hybrid_query_weaviate(query, collection_name, alpha_val):\n", "    \n", "    nearText = {\n", "        \"concepts\": [query],\n", "        \"distance\": 0.7,\n", "    }\n", "\n", "    properties = [\n", "        \"title\", \"content\", \"url\",\n", "        \"_additional { score }\"\n", "    ]\n", "\n", "    result = (\n", "        client.query\n", "        .get(collection_name, properties)\n", "        .with_hybrid(nearText, alpha=alpha_val)\n", "        .with_limit(10)\n", "        .do()\n", "    )\n", "    \n", "    # 检查错误\n", "    if (\"errors\" in result):\n", "        print (\"\\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.\")\n", "        raise Exception(result[\"errors\"][0]['message'])\n", "    \n", "    return result[\"data\"][\"Get\"][collection_name]\n"]}, {"cell_type": "code", "execution_count": null, "id": "7e2025f6", "metadata": {}, "outputs": [], "source": ["query_result = hybrid_query_weaviate(\"modern art in Europe\", \"Article\", 0.5)\n", "\n", "for i, article in enumerate(query_result):\n", "    print(f\"{i+1}. { article['title']} (Score: {article['_additional']['score']})\")\n"]}, {"cell_type": "code", "execution_count": null, "id": "93c4a696", "metadata": {}, "outputs": [], "source": ["query_result = hybrid_query_weaviate(\"Famous battles in Scottish history\", \"Article\", 0.5)\n", "\n", "for i, article in enumerate(query_result):\n", "    print(f\"{i+1}. { article['title']} (Score: {article['_additional']['score']})\")\n"]}, {"cell_type": "markdown", "id": "2007be48", "metadata": {}, "source": ["感谢您的跟随，现在您已经具备了设置自己的向量数据库并使用嵌入来做各种酷炫事情的能力 - 尽情享受吧！对于更复杂的用例，请继续阅读此存储库中的其他示例。\n"]}], "metadata": {"kernelspec": {"display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.12"}, "vscode": {"interpreter": {"hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"}}}, "nbformat": 4, "nbformat_minor": 5}