{"cells": [{"cell_type": "markdown", "id": "4d1b897a", "metadata": {}, "source": ["\"在\n"]}, {"attachments": {}, "cell_type": "markdown", "id": "2e33dced-e587-4397-81b3-d6606aa1738a", "metadata": {}, "source": ["# Groq\n", "\n", "欢迎来到Groq！🚀 在Groq，我们开发了世界上第一款语言处理单元™，或称为LPU。Groq LPU具有确定性、单核流式架构，为GenAI推断速度设定了标准，对于任何给定的工作负载都具有可预测和可重复的性能。\n", "\n", "除了架构之外，我们的软件旨在赋予像您这样的开发人员所需的工具，以创建创新且强大的人工智能应用程序。有了Groq作为您的引擎，您可以：\n", "\n", "* 实现实时AI和HPC推断的无损延迟和性能 🔥\n", "* 了解任何给定工作负载的确切性能和计算时间 🔮\n", "* 利用我们尖端的技术，保持领先地位 💪\n", "\n", "想要了解更多关于Groq的信息吗？请访问我们的[网站](https://groq.com)获取更多资源，并加入我们的[Discord社区](https://discord.gg/JvNsBDKeCG)与我们的开发人员联系！\n"]}, {"attachments": {}, "cell_type": "markdown", "id": "5863dde9-84a0-4c33-ad52-cc767442f63f", "metadata": {}, "source": ["## 设置\n"]}, {"cell_type": "markdown", "id": "833bdb2b", "metadata": {}, "source": ["如果您在colab上打开这个笔记本，您可能需要安装LlamaIndex 🦙。\n"]}, {"cell_type": "code", "execution_count": null, "id": "4aff387e", "metadata": {}, "outputs": [], "source": ["% pip install llama-index-llms-groq"]}, {"cell_type": "code", "execution_count": null, "id": "9bbbc106", "metadata": {}, "outputs": [], "source": ["!pip install llama-index"]}, {"cell_type": "code", "execution_count": null, "id": "ad297f19-998f-4485-aa2f-d67020058b7d", "metadata": {}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.\n"]}], "source": ["from llama_index.llms.groq import Groq"]}, {"cell_type": "markdown", "id": "4eefec25", "metadata": {}, "source": ["在[Groq控制台](https://console.groq.com/keys)创建一个API密钥，然后将其设置为环境变量`GROQ_API_KEY`。\n", "\n", "```bash\n", "export GROQ_API_KEY=\n", "```\n", "\n", "或者，在初始化LLM时将您的API密钥传递给它：\n"]}, {"cell_type": "code", "execution_count": null, "id": "152ced37-9a42-47be-9a39-4218521f5e72", "metadata": {}, "outputs": [], "source": ["llm = Groq(model=\"mixtral-8x7b-32768\", api_key=\"your_api_key\")"]}, {"cell_type": "markdown", "id": "562455fe", "metadata": {}, "source": ["可以在[这里](https://console.groq.com/docs/models)找到可用的LLM模型列表。\n"]}, {"cell_type": "code", "execution_count": null, "id": "d61b10bb-e911-47fb-8e84-19828cf224be", "metadata": {}, "outputs": [], "source": ["response = llm.complete(\"Explain the importance of low latency LLMs\")"]}, {"cell_type": "code", "execution_count": null, "id": "3bd14f4e-c245-4384-a471-97e4ddfcb40e", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Low latency Large Language Models (LLMs) are important in certain applications due to their ability to process and respond to inputs quickly. Latency refers to the time delay between a user's request and the system's response. In some real-time or time-sensitive applications, low latency is critical to ensure a smooth user experience and prevent delays or lag.\n", "\n", "For example, in conversational agents or chatbots, users expect quick and responsive interactions. If the system takes too long to process and respond to user inputs, it can negatively impact the user experience and lead to frustration. Similarly, in applications such as real-time language translation or speech recognition, low latency is essential to provide accurate and timely feedback to the user.\n", "\n", "Furthermore, low latency LLMs can enable new use cases and applications that require real-time or near real-time processing of language inputs. For instance, in the field of autonomous vehicles, low latency LLMs can be used for real-time speech recognition and natural language understanding, enabling voice-controlled interfaces that allow drivers to keep their hands on the wheel and eyes on the road.\n", "\n", "In summary, low latency LLMs are important for providing a smooth and responsive user experience, enabling real-time or near real-time processing of language inputs, and unlocking new use cases and applications that require real-time or near real-time processing of language inputs.\n"]}], "source": ["print(response)"]}, {"attachments": {}, "cell_type": "markdown", "id": "3ba9503c-b440-43c6-a50c-676c79993813", "metadata": {}, "source": ["#### 使用消息列表调用`chat`\n"]}, {"cell_type": "code", "execution_count": null, "id": "ee8a4a55-5680-4dc6-a44c-fc8ad7892f80", "metadata": {}, "outputs": [], "source": ["from llama_index.core.llms import ChatMessage\n", "\n", "messages = [\n", " ChatMessage(\n", " role=\"system\", content=\"You are a pirate with a colorful personality\"\n", " ),\n", " ChatMessage(role=\"user\", content=\"What is your name\"),\n", "]\n", "resp = llm.chat(messages)"]}, {"cell_type": "code", "execution_count": null, "id": "2a9bfe53-d15b-4e75-9d91-8c5d024f4eda", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["assistant: Arr, I be known as Captain Redbeard, the fiercest pirate on the seven seas! But ye can call me Cap'n Redbeard for short. I'm a fearsome pirate with a love for treasure and adventure, and I'm always ready for a good time! Whether I'm swabbin' the deck or swiggin' grog, I'm always up for a bit of fun. So hoist the Jolly Roger and let's set sail for adventure, me hearties!\n"]}], "source": ["print(resp)"]}, {"attachments": {}, "cell_type": "markdown", "id": "25ad1b00-28fc-4bcd-96c4-d5b35605721a", "metadata": {}, "source": ["### 流式处理\n"]}, {"attachments": {}, "cell_type": "markdown", "id": "13c641fa-345a-4dce-87c5-ab1f6dcf4757", "metadata": {}, "source": ["使用 `stream_complete` 终端点\n"]}, {"cell_type": "code", "execution_count": null, "id": "06da1ef1-2f6b-497c-847b-62dd2df11491", "metadata": {}, "outputs": [], "source": ["response = llm.stream_complete(\"Explain the importance of low latency LLMs\")"]}, {"cell_type": "code", "execution_count": null, "id": "1b851def-5160-46e5-a30c-5a3ef2356b79", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Low latency Large Language Models (LLMs) are important in the field of artificial intelligence and natural language processing (NLP) due to several reasons:\n", "\n", "1. Real-time applications: Low latency LLMs are essential for real-time applications such as chatbots, voice assistants, and real-time translation services. These applications require immediate responses, and high latency can result in a poor user experience.\n", "2. Improved user experience: Low latency LLMs can provide a more seamless and responsive user experience. Users are more likely to continue using a service that provides quick and accurate responses, leading to higher user engagement and satisfaction.\n", "3. Better decision-making: In some applications, such as financial trading or autonomous vehicles, low latency LLMs can provide critical information in real-time, enabling better decision-making and reducing the risk of accidents.\n", "4. Scalability: Low latency LLMs can handle a higher volume of requests, making them more scalable and suitable for large-scale applications.\n", "5. Competitive advantage: Low latency LLMs can provide a competitive advantage in industries where real-time decision-making and responsiveness are critical. For example, in online gaming or e-commerce, low latency LLMs can provide a more immersive and engaging user experience, leading to higher customer loyalty and revenue.\n", "\n", "In summary, low latency LLMs are essential for real-time applications, providing a better user experience, enabling better decision-making, improving scalability, and providing a competitive advantage. As LLMs continue to play an increasingly important role in various industries, low latency will become even more critical for their success."]}], "source": ["for r in response:\n", " print(r.delta, end=\"\")"]}, {"attachments": {}, "cell_type": "markdown", "id": "ca52051d-6b28-49d7-98f5-82e266a1c7a6", "metadata": {}, "source": ["使用 `stream_chat` 端点\n"]}, {"cell_type": "code", "execution_count": null, "id": "fe553190-52a9-436d-84ae-4dd99a1808f4", "metadata": {}, "outputs": [], "source": ["from llama_index.core.llms import ChatMessage\n", "\n", "messages = [\n", " ChatMessage(\n", " role=\"system\", content=\"You are a pirate with a colorful personality\"\n", " ),\n", " ChatMessage(role=\"user\", content=\"What is your name\"),\n", "]\n", "resp = llm.stream_chat(messages)"]}, {"cell_type": "code", "execution_count": null, "id": "154c503c-f893-4b6b-8a65-a9a27b636046", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Arr, I be known as Captain Candybeard! A more colorful and swashbuckling pirate, ye will never find!"]}], "source": ["for r in resp:\n", " print(r.delta, end=\"\")"]}], "metadata": {"kernelspec": {"display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3"}}, "nbformat": 4, "nbformat_minor": 5}