{ "cells": [ { "cell_type": "code", "execution_count": 1, "id": "bf2ffc1d", "metadata": {}, "outputs": [], "source": [ "from ingest import load_faq_data\n", "documents = load_faq_data()\n" ] }, { "cell_type": "code", "execution_count": 2, "id": "00a2ed46", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "85" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "docs_llm = [doc for doc in documents if doc['course'] == 'llm-zoomcamp']\n", "len(docs_llm)" ] }, { "cell_type": "code", "execution_count": null, "id": "acb9a3f0", "metadata": {}, "outputs": [], "source": [ "from sqlitesearch import TextSearchIndex\n", "\n", "index = TextSearchIndex(\n", " text_fields=['question', 'section', 'answer'],\n", " keyword_fields=['course'],\n", " id_field='doc_id', # <-- use the existing id so if it's re-run, it will update existing records instead of creating duplicates\n", " db_path='faq.db'\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "91f7d013", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Added: I just discovered the course. Can I still join?...\n", "Added: Course: I have registered for the LLM Zoomcamp. When can I e...\n", "Added: What is the video/zoom link to the stream for the “Office Ho...\n", "Added: Cloud alternatives with GPU...\n", "Added: Leaderboard: I am not on the leaderboard / how do I know whi...\n", "Added: Certificate: Can I follow the course in a self-paced mode an...\n", "Added: I missed the first homework - can I still get a certificate?...\n", "Added: Homework: Why does the content keep changing?...\n", "Added: When will the course be offered next?...\n", "Added: Are there any lectures/videos? Where are they?...\n", "Added: WSL2: ResponseError: model requires more system memory (X.X ...\n", "Added: Server Error (500) When logging in to course homework using ...\n", "Added: Why are we not using Langchain in the course?...\n", "Added: OpenAI: Error when running OpenAI chat.completions.create co...\n", "Added: OpenAI: Error: RateLimitError: Error code: 429 -...\n", "Added: OpenAI: Error: 'Cannot import name OpenAI from openai'; How ...\n", "Added: OpenAI: How much will I have to spend to use the Open AI API...\n", "Added: OpenAI: Do I have to subscribe and pay for Open AI API for t...\n", "Added: Authentication: Safe and easy way to store and load API keys...\n", "Added: How to store and load API keys using .env file...\n", "Added: Authentication: Why is my OPENAI_API_KEY not found in the Ju...\n", "Added: OpenSource: I am using Groq, and it doesn't provide a tokeni...\n", "Added: OpenSource: Can I use Groq instead of OpenAI?...\n", "Added: OpenSource: Can I use open-source alternatives to OpenAI API...\n", "Added: OpenAI: \"RateLimitError 429 / insufficient_quota\" — my accou...\n", "Added: Homework: Returning Empty list after filtering my query (HW ...\n", "Added: minsearch: SyntaxError \"invalid character\" when running my d...\n", "Added: ModuleNotFoundError on import docx in parse-faq.ipynb...\n", "Added: How do I count tokens for a non-OpenAI model (Gemini, Mistra...\n", "Added: OpenAI: Why does my token count differ from what OpenAI repo...\n", "Added: API keys: how do I set them once and not re-export every ter...\n", "Added: Ollama: How to install Ollama?...\n", "Added: Connection refused error on prompting the ollam RAG?...\n", "Added: Any free models with tool use support?...\n", "Added: Agents: \"RuntimeError: Already running asyncio in this threa...\n", "Added: I passed a float to my tool, but got a validation error sayi...\n", "Added: Agents: which non-OpenAI models support tool / function call...\n", "Added: Install MCP Inspector...\n", "Added: Agents: \"AttributeError: 'str' object has no attribute 'outp...\n", "Added: Run MCP Inspector...\n", "Added: Inspect MCP Server...\n", "Added: How to Solve \"RuntimeError: Already running asyncio in this ...\n", "Added: I am using Azure OpenAI and I am still getting an error of E...\n", "Added: What are embeddings?...\n", "Added: Warning: 'model \"multi-qa-mpnet-base-dot-v1\" was made on sen...\n", "Added: Why was .dot(...) used directly to compute cosine similarity...\n", "Added: Vector search: should I embed the question, the answer, or b...\n", "Added: What is the cosine similarity?...\n", "Added: Why does cosine similarity reduce to a matrix multiplication...\n", "Added: Evaluation: \"JSONDecodeError: Expecting value\" when generati...\n", "Added: Evaluation: Jupyter kernel crashes when embedding the ground...\n", "Added: Evaluation: hitting rate limits while generating the ground-...\n", "Added: In Windows OS: OSError: [WinError 126] The specified module ...\n", "Added: OperationalError when running python prep.py: psycopg2. Oper...\n", "Added: How set Pandas to show entire text content in a column. Usef...\n", "Added: How to normalize vectors in a Pandas DataFrame column (or Pa...\n", "Added: How to compute the quantile or percentile of Pandas DataFram...\n", "Added: How can I remove all Docker containers, images, and volumes,...\n", "Added: Session State: I want the user to only be able to give feedb...\n", "Added: Docker: When trying to run a streamlit app using docker-comp...\n", "Added: Is it a group project?...\n", "Added: Do we submit 2 projects, what does attempt 1 and 2 mean?...\n", "Added: Does the competition count as the capstone?...\n", "Added: How is my capstone project going to be evaluated?...\n", "Added: When and how will we be assigned projects for review/grading...\n", "Added: I’ve already submitted my project. Why can’t I review any pr...\n", "Added: How can I find some good ideas or datasets for the project?...\n", "Added: Project: do I need an orchestration tool (Airflow, Mage, Kes...\n", "Added: Project: how do I evaluate a recommender-style RAG (no obvio...\n", "Added: Project: my corpus is large (long PDFs, many paragraphs). Wh...\n", "Added: Project: what does \"reproducibility\" mean — do reviewers nee...\n", "Added: Can I use the workshop materials for my own projects or shar...\n", "Added: How to set up a new dlt project when loading from cloud?...\n", "Added: There is an error when opening the table using `dbtable = db...\n", "Added: There is an error when running main(): FileNotFoundError: Ta...\n", "Added: How do I know which tables are in the db?...\n", "Added: Does DLT have connectors to ClickHouse or StarRocks?...\n", "Added: Notebook does not have secret access or 401 Client Error: Un...\n", "Added: Error: How to fix requests library only installs v2.28 inste...\n" ] } ], "source": [ "import time\n", "\n", "for doc in docs_llm:\n", " doc['doc_id'] = doc.pop('id') # \"id\" is already a reserved column in sqlite, need to add the ids from the json as a different field name\n", " index.add(doc)\n", " print(f'Added: {doc[\"question\"][:60]}...')\n", " time.sleep(0.5)" ] }, { "cell_type": "code", "execution_count": 5, "id": "42bb1a72", "metadata": {}, "outputs": [], "source": [ "index.close()\n" ] }, { "cell_type": "code", "execution_count": null, "id": "1b297d4f", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "llm-zoomcamp (3.12.1)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.1" } }, "nbformat": 4, "nbformat_minor": 5 }