{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "name": "2021-06-23-movielens-milvus-redis.ipynb", "provenance": [], "collapsed_sections": [], "toc_visible": true, "authorship_tag": "ABX9TyP0ygaJq3Enq3XP8oBJiBXn" }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" } }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "8FgLYOButqII" }, "source": [ "# Recommender with Redis and Milvus\n", "> Storing the pre-calculated user and items vectors of movielens dataset into redis in-memory database and then indexing into milvus for efficient large-scale retrieval\n", "\n", "- toc: true\n", "- badges: true\n", "- comments: true\n", "- categories: [retrieval, redis, milvus, movie]\n", "- image:" ] }, { "cell_type": "markdown", "metadata": { "id": "3dfeb422" }, "source": [ "| Packages | Servers |\n", "| --------------- | -------------- |\n", "| pymilvus | milvus-1.1.0 |\n", "| redis | redis |\n", "| paddle_serving_app |\n", "| paddlepaddle |" ] }, { "cell_type": "code", "metadata": { "id": "tA3G1ESOMkrS" }, "source": [ "!pip install pymilvus==1.1.0\n", "!pip install paddle_serving_app==0.3.1\n", "!pip install paddlepaddle\n", "!pip install redis" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "WNYdYI3numT-" }, "source": [ "### Install and run Milvus server" ] }, { "cell_type": "markdown", "metadata": { "id": "IUtzLMjevQRn" }, "source": [ "> Warning: It will take ~40 minutes to install!" ] }, { "cell_type": "code", "metadata": { "id": "h8kdCNOvMsdb" }, "source": [ "!git clone -b 1.1 https://github.com/milvus-io/milvus.git\n", "% cd /content/milvus/core\n", "! ./ubuntu_build_deps.sh\n", "!./build.sh -t Release\n", "# !./build.sh -t Release -g\n", "\n", "% cd /content/milvus/core/milvus\n", "! echo $LD_LIBRARY_PATH\n", "import os\n", "os.environ['LD_LIBRARY_PATH'] +=\":/content/milvus/core/milvus/lib\"\n", "! echo $LD_LIBRARY_PATH\n", "% cd scripts\n", "! nohup ./start_server.sh &\n", "! cat nohup.out" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "XHBfpO_DkNEA", "outputId": "e22b1bb3-ea15-46cf-e203-65f2d05969e5" }, "source": [ "!cat nohup.out" ], "execution_count": 8, "outputs": [ { "output_type": "stream", "text": [ "\n", " __ _________ _ ____ ______ \n", " / |/ / _/ /| | / / / / / __/ \n", " / /|_/ // // /_| |/ / /_/ /\\ \\ \n", " /_/ /_/___/____/___/\\____/___/ \n", "\n", "Welcome to use Milvus!\n", "Milvus Release version: v1.1.1, built at 2021-06-23 14:11.42, with OpenBLAS library.\n", "You are using Milvus CPU edition\n", "Last commit id: 3fc81236452d8060fe7adc1793ad1d69f3d8423c\n", "\n", "Loading configuration from: ../conf/server_config.yaml\n", "NOTICE: You are using SQLite as the meta data management. We recommend change it to MySQL.\n", "Supported CPU instruction sets: avx2, sse4_2\n", "FAISS hook AVX2\n", "Milvus server started successfully!\n", "Milvus server is going to shutdown ...\n", "Milvus server exit...\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "-fQOxxmXONX5" }, "source": [ "We are using Redis as a metadata storage service. Code can easily be modified to use a python dictionary, but that usually does not work in any use case outside of quick examples. We need a metadata storage service in order to be able to be able to map between embeddings and the corresponding data." ] }, { "cell_type": "markdown", "metadata": { "id": "8l6Uz5UNuqsX" }, "source": [ "### Install and run Redis server" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Uqac1hT5NUs7", "outputId": "5a5192ca-bb04-4259-b9d4-4e050db5a0b2" }, "source": [ "#hide-output\n", "!wget http://download.redis.io/releases/redis-stable.tar.gz --no-check-certificate\n", "!tar -xf redis-stable.tar.gz && cd redis-stable/src && make" ], "execution_count": 3, "outputs": [ { "output_type": "stream", "text": [ "--2021-06-23 14:52:56-- http://download.redis.io/releases/redis-stable.tar.gz\n", "Resolving download.redis.io (download.redis.io)... 45.60.121.1\n", "Connecting to download.redis.io (download.redis.io)|45.60.121.1|:80... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 2261060 (2.2M) [application/octet-stream]\n", "Saving to: ‘redis-stable.tar.gz’\n", "\n", "redis-stable.tar.gz 100%[===================>] 2.16M --.-KB/s in 0.1s \n", "\n", "2021-06-23 14:52:57 (21.0 MB/s) - ‘redis-stable.tar.gz’ saved [2261060/2261060]\n", "\n", " \u001b[34mCC\u001b[0m \u001b[33mMakefile.dep\u001b[0m\n", "rm -rf redis-server redis-sentinel redis-cli redis-benchmark redis-check-rdb redis-check-aof *.o *.gcda *.gcno *.gcov redis.info lcov-html Makefile.dep dict-benchmark\n", "rm -f adlist.d quicklist.d ae.d anet.d dict.d server.d sds.d zmalloc.d lzf_c.d lzf_d.d pqsort.d zipmap.d sha1.d ziplist.d release.d networking.d util.d object.d db.d replication.d rdb.d t_string.d t_list.d t_set.d t_zset.d t_hash.d config.d aof.d pubsub.d multi.d debug.d sort.d intset.d syncio.d cluster.d crc16.d endianconv.d slowlog.d scripting.d bio.d rio.d rand.d memtest.d crcspeed.d crc64.d bitops.d sentinel.d notify.d setproctitle.d blocked.d hyperloglog.d latency.d sparkline.d redis-check-rdb.d redis-check-aof.d geo.d lazyfree.d module.d evict.d expire.d geohash.d geohash_helper.d childinfo.d defrag.d siphash.d rax.d t_stream.d listpack.d localtime.d lolwut.d lolwut5.d lolwut6.d acl.d gopher.d tracking.d connection.d tls.d sha256.d timeout.d setcpuaffinity.d anet.d adlist.d dict.d redis-cli.d zmalloc.d release.d ae.d crcspeed.d crc64.d siphash.d crc16.d ae.d anet.d redis-benchmark.d adlist.d dict.d zmalloc.d siphash.d\n", "(cd ../deps && make distclean)\n", "make[1]: Entering directory '/content/milvus/core/milvus/scripts/redis-stable/deps'\n", "(cd hiredis && make clean) > /dev/null || true\n", "(cd linenoise && make clean) > /dev/null || true\n", "(cd lua && make clean) > /dev/null || true\n", "(cd jemalloc && [ -f Makefile ] && make distclean) > /dev/null || true\n", "(rm -f .make-*)\n", "make[1]: Leaving directory '/content/milvus/core/milvus/scripts/redis-stable/deps'\n", "(rm -f .make-*)\n", "echo STD=-std=c11 -pedantic -DREDIS_STATIC='' >> .make-settings\n", "echo WARN=-Wall -W -Wno-missing-field-initializers >> .make-settings\n", "echo OPT=-O2 >> .make-settings\n", "echo MALLOC=jemalloc >> .make-settings\n", "echo CFLAGS= >> .make-settings\n", "echo LDFLAGS= >> .make-settings\n", "echo REDIS_CFLAGS= >> .make-settings\n", "echo REDIS_LDFLAGS= >> .make-settings\n", "echo PREV_FINAL_CFLAGS=-std=c11 -pedantic -DREDIS_STATIC='' -Wall -W -Wno-missing-field-initializers -O2 -g -ggdb -I../deps/hiredis -I../deps/linenoise -I../deps/lua/src -DUSE_JEMALLOC -I../deps/jemalloc/include >> .make-settings\n", "echo PREV_FINAL_LDFLAGS= -g -ggdb -rdynamic >> .make-settings\n", "(cd ../deps && make hiredis linenoise lua jemalloc)\n", "make[1]: Entering directory '/content/milvus/core/milvus/scripts/redis-stable/deps'\n", "(cd hiredis && make clean) > /dev/null || true\n", "(cd linenoise && make clean) > /dev/null || true\n", "(cd lua && make clean) > /dev/null || true\n", "(cd jemalloc && [ -f Makefile ] && make distclean) > /dev/null || true\n", "(rm -f .make-*)\n", "(echo \"\" > .make-cflags)\n", "(echo \"\" > .make-ldflags)\n", "\u001b[32;1mMAKE\u001b[0m \u001b[37;1mhiredis\u001b[0m\n", "cd hiredis && make static \n", "make[2]: Entering directory '/content/milvus/core/milvus/scripts/redis-stable/deps/hiredis'\n", "cc -std=c99 -pedantic -c -O3 -fPIC -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb net.c\n", "cc -std=c99 -pedantic -c -O3 -fPIC -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb hiredis.c\n", "cc -std=c99 -pedantic -c -O3 -fPIC -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb sds.c\n", "cc -std=c99 -pedantic -c -O3 -fPIC -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb async.c\n", "cc -std=c99 -pedantic -c -O3 -fPIC -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb read.c\n", "cc -std=c99 -pedantic -c -O3 -fPIC -Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -g -ggdb sockcompat.c\n", "ar rcs libhiredis.a net.o hiredis.o sds.o async.o read.o sockcompat.o\n", "make[2]: Leaving directory '/content/milvus/core/milvus/scripts/redis-stable/deps/hiredis'\n", "\u001b[32;1mMAKE\u001b[0m \u001b[37;1mlinenoise\u001b[0m\n", "cd linenoise && make\n", "make[2]: Entering directory '/content/milvus/core/milvus/scripts/redis-stable/deps/linenoise'\n", "cc -Wall -Os -g -c linenoise.c\n", "make[2]: Leaving directory '/content/milvus/core/milvus/scripts/redis-stable/deps/linenoise'\n", "\u001b[32;1mMAKE\u001b[0m \u001b[37;1mlua\u001b[0m\n", "cd lua/src && make all CFLAGS=\"-O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' \" MYLDFLAGS=\"\" AR=\"ar rcu\"\n", "make[2]: Entering directory '/content/milvus/core/milvus/scripts/redis-stable/deps/lua/src'\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lapi.o lapi.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lcode.o lcode.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o ldebug.o ldebug.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o ldo.o ldo.c\n", "\u001b[01m\u001b[Kldo.c:\u001b[m\u001b[K In function ‘\u001b[01m\u001b[Kf_parser\u001b[m\u001b[K’:\n", "\u001b[01m\u001b[Kldo.c:496:7:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kunused variable ‘\u001b[01m\u001b[Kc\u001b[m\u001b[K’ [\u001b[01;35m\u001b[K-Wunused-variable\u001b[m\u001b[K]\n", " int \u001b[01;35m\u001b[Kc\u001b[m\u001b[K = luaZ_lookahead(p->z);\n", " \u001b[01;35m\u001b[K^\u001b[m\u001b[K\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o ldump.o ldump.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lfunc.o lfunc.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lgc.o lgc.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o llex.o llex.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lmem.o lmem.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lobject.o lobject.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lopcodes.o lopcodes.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lparser.o lparser.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lstate.o lstate.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lstring.o lstring.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o ltable.o ltable.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o ltm.o ltm.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lundump.o lundump.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lvm.o lvm.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lzio.o lzio.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o strbuf.o strbuf.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o fpconv.o fpconv.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lauxlib.o lauxlib.c\n", "\u001b[01m\u001b[Klauxlib.c:\u001b[m\u001b[K In function ‘\u001b[01m\u001b[KluaL_loadfile\u001b[m\u001b[K’:\n", "\u001b[01m\u001b[Klauxlib.c:577:4:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kthis ‘\u001b[01m\u001b[Kwhile\u001b[m\u001b[K’ clause does not guard... [\u001b[01;35m\u001b[K-Wmisleading-indentation\u001b[m\u001b[K]\n", " \u001b[01;35m\u001b[Kwhile\u001b[m\u001b[K ((c = getc(lf.f)) != EOF && c != LUA_SIGNATURE[0]) ;\n", " \u001b[01;35m\u001b[K^~~~~\u001b[m\u001b[K\n", "\u001b[01m\u001b[Klauxlib.c:578:5:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[K...this statement, but the latter is misleadingly indented as if it were guarded by the ‘\u001b[01m\u001b[Kwhile\u001b[m\u001b[K’\n", " \u001b[01;36m\u001b[Klf\u001b[m\u001b[K.extraline = 0;\n", " \u001b[01;36m\u001b[K^~\u001b[m\u001b[K\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lbaselib.o lbaselib.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o ldblib.o ldblib.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o liolib.o liolib.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lmathlib.o lmathlib.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o loslib.o loslib.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o ltablib.o ltablib.c\n", "\u001b[01m\u001b[Kltablib.c:\u001b[m\u001b[K In function ‘\u001b[01m\u001b[Kaddfield\u001b[m\u001b[K’:\n", "\u001b[01m\u001b[Kltablib.c:137:3:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kthis ‘\u001b[01m\u001b[Kif\u001b[m\u001b[K’ clause does not guard... [\u001b[01;35m\u001b[K-Wmisleading-indentation\u001b[m\u001b[K]\n", " \u001b[01;35m\u001b[Kif\u001b[m\u001b[K (!lua_isstring(L, -1))\n", " \u001b[01;35m\u001b[K^~\u001b[m\u001b[K\n", "\u001b[01m\u001b[Kltablib.c:140:5:\u001b[m\u001b[K \u001b[01;36m\u001b[Knote: \u001b[m\u001b[K...this statement, but the latter is misleadingly indented as if it were guarded by the ‘\u001b[01m\u001b[Kif\u001b[m\u001b[K’\n", " \u001b[01;36m\u001b[KluaL_addvalue\u001b[m\u001b[K(b);\n", " \u001b[01;36m\u001b[K^~~~~~~~~~~~~\u001b[m\u001b[K\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lstrlib.o lstrlib.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o loadlib.o loadlib.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o linit.o linit.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lua_cjson.o lua_cjson.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lua_struct.o lua_struct.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lua_cmsgpack.o lua_cmsgpack.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lua_bit.o lua_bit.c\n", "ar rcu liblua.a lapi.o lcode.o ldebug.o ldo.o ldump.o lfunc.o lgc.o llex.o lmem.o lobject.o lopcodes.o lparser.o lstate.o lstring.o ltable.o ltm.o lundump.o lvm.o lzio.o strbuf.o fpconv.o lauxlib.o lbaselib.o ldblib.o liolib.o lmathlib.o loslib.o ltablib.o lstrlib.o loadlib.o linit.o lua_cjson.o lua_struct.o lua_cmsgpack.o lua_bit.o\t# DLL needs all object files\n", "ar: `u' modifier ignored since `D' is the default (see `U')\n", "ranlib liblua.a\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o lua.o lua.c\n", "cc -o lua lua.o liblua.a -lm \n", "liblua.a(loslib.o): In function `os_tmpname':\n", "loslib.c:(.text+0x290): warning: the use of `tmpnam' is dangerous, better use `mkstemp'\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o luac.o luac.c\n", "cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DREDIS_STATIC='' -c -o print.o print.c\n", "cc -o luac luac.o print.o liblua.a -lm \n", "make[2]: Leaving directory '/content/milvus/core/milvus/scripts/redis-stable/deps/lua/src'\n", "\u001b[32;1mMAKE\u001b[0m \u001b[37;1mjemalloc\u001b[0m\n", "cd jemalloc && ./configure --with-version=5.1.0-0-g0 --with-lg-quantum=3 --with-jemalloc-prefix=je_ --enable-cc-silence CFLAGS=\"-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops \" LDFLAGS=\"\"\n", "configure: WARNING: unrecognized options: --enable-cc-silence\n", "checking for xsltproc... false\n", "checking for gcc... gcc\n", "checking whether the C compiler works... yes\n", "checking for C compiler default output file name... a.out\n", "checking for suffix of executables... \n", "checking whether we are cross compiling... no\n", "checking for suffix of object files... o\n", "checking whether we are using the GNU C compiler... yes\n", "checking whether gcc accepts -g... yes\n", "checking for gcc option to accept ISO C89... none needed\n", "checking whether compiler is cray... no\n", "checking whether compiler supports -std=gnu11... yes\n", "checking whether compiler supports -Wall... yes\n", "checking whether compiler supports -Wshorten-64-to-32... no\n", "checking whether compiler supports -Wsign-compare... yes\n", "checking whether compiler supports -Wundef... yes\n", "checking whether compiler supports -Wno-format-zero-length... yes\n", "checking whether compiler supports -pipe... yes\n", "checking whether compiler supports -g3... yes\n", "checking how to run the C preprocessor... gcc -E\n", "checking for g++... g++\n", "checking whether we are using the GNU C++ compiler... yes\n", "checking whether g++ accepts -g... yes\n", "checking whether g++ supports C++14 features by default... yes\n", "checking whether compiler supports -Wall... yes\n", "checking whether compiler supports -g3... yes\n", "checking whether libstdc++ linkage is compilable... yes\n", "checking for grep that handles long lines and -e... /bin/grep\n", "checking for egrep... /bin/grep -E\n", "checking for ANSI C header files... yes\n", "checking for sys/types.h... yes\n", "checking for sys/stat.h... yes\n", "checking for stdlib.h... yes\n", "checking for string.h... yes\n", "checking for memory.h... yes\n", "checking for strings.h... yes\n", "checking for inttypes.h... yes\n", "checking for stdint.h... yes\n", "checking for unistd.h... yes\n", "checking whether byte ordering is bigendian... no\n", "checking size of void *... 8\n", "checking size of int... 4\n", "checking size of long... 8\n", "checking size of long long... 8\n", "checking size of intmax_t... 8\n", "checking build system type... x86_64-pc-linux-gnu\n", "checking host system type... x86_64-pc-linux-gnu\n", "checking whether pause instruction is compilable... yes\n", "checking number of significant virtual address bits... 48\n", "checking for ar... ar\n", "checking for nm... nm\n", "checking for gawk... no\n", "checking for mawk... mawk\n", "checking malloc.h usability... yes\n", "checking malloc.h presence... yes\n", "checking for malloc.h... yes\n", "checking whether malloc_usable_size definition can use const argument... no\n", "checking for library containing log... -lm\n", "checking whether __attribute__ syntax is compilable... yes\n", "checking whether compiler supports -fvisibility=hidden... yes\n", "checking whether compiler supports -fvisibility=hidden... yes\n", "checking whether compiler supports -Werror... yes\n", "checking whether compiler supports -herror_on_warning... no\n", "checking whether tls_model attribute is compilable... yes\n", "checking whether compiler supports -Werror... yes\n", "checking whether compiler supports -herror_on_warning... no\n", "checking whether alloc_size attribute is compilable... yes\n", "checking whether compiler supports -Werror... yes\n", "checking whether compiler supports -herror_on_warning... no\n", "checking whether format(gnu_printf, ...) attribute is compilable... yes\n", "checking whether compiler supports -Werror... yes\n", "checking whether compiler supports -herror_on_warning... no\n", "checking whether format(printf, ...) attribute is compilable... yes\n", "checking for a BSD-compatible install... /usr/bin/install -c\n", "checking for ranlib... ranlib\n", "checking for ld... /usr/bin/ld\n", "checking for autoconf... false\n", "checking for memalign... yes\n", "checking for valloc... yes\n", "checking whether compiler supports -O3... yes\n", "checking whether compiler supports -O3... yes\n", "checking whether compiler supports -funroll-loops... yes\n", "checking configured backtracing method... N/A\n", "checking for sbrk... yes\n", "checking whether utrace(2) is compilable... no\n", "checking whether a program using __builtin_unreachable is compilable... yes\n", "checking whether a program using __builtin_ffsl is compilable... yes\n", "checking LG_PAGE... 12\n", "checking pthread.h usability... yes\n", "checking pthread.h presence... yes\n", "checking for pthread.h... yes\n", "checking for pthread_create in -lpthread... yes\n", "checking dlfcn.h usability... yes\n", "checking dlfcn.h presence... yes\n", "checking for dlfcn.h... yes\n", "checking for dlsym... no\n", "checking for dlsym in -ldl... yes\n", "checking whether pthread_atfork(3) is compilable... yes\n", "checking whether pthread_setname_np(3) is compilable... yes\n", "checking for library containing clock_gettime... none required\n", "checking whether clock_gettime(CLOCK_MONOTONIC_COARSE, ...) is compilable... yes\n", "checking whether clock_gettime(CLOCK_MONOTONIC, ...) is compilable... yes\n", "checking whether mach_absolute_time() is compilable... no\n", "checking whether compiler supports -Werror... yes\n", "checking whether syscall(2) is compilable... yes\n", "checking for secure_getenv... yes\n", "checking for sched_getcpu... yes\n", "checking for sched_setaffinity... yes\n", "checking for issetugid... no\n", "checking for _malloc_thread_cleanup... no\n", "checking for _pthread_mutex_init_calloc_cb... no\n", "checking for TLS... yes\n", "checking whether C11 atomics is compilable... no\n", "checking whether GCC __atomic atomics is compilable... yes\n", "checking whether GCC __sync atomics is compilable... yes\n", "checking whether Darwin OSAtomic*() is compilable... no\n", "checking whether madvise(2) is compilable... yes\n", "checking whether madvise(..., MADV_FREE) is compilable... yes\n", "checking whether madvise(..., MADV_DONTNEED) is compilable... yes\n", "checking whether madvise(..., MADV_DO[NT]DUMP) is compilable... yes\n", "checking whether madvise(..., MADV_[NO]HUGEPAGE) is compilable... yes\n", "checking whether to force 32-bit __sync_{add,sub}_and_fetch()... no\n", "checking whether to force 64-bit __sync_{add,sub}_and_fetch()... no\n", "checking for __builtin_clz... yes\n", "checking whether Darwin os_unfair_lock_*() is compilable... no\n", "checking whether Darwin OSSpin*() is compilable... no\n", "checking whether glibc malloc hook is compilable... yes\n", "checking whether glibc memalign hook is compilable... yes\n", "checking whether pthreads adaptive mutexes is compilable... yes\n", "checking whether compiler supports -D_GNU_SOURCE... yes\n", "checking whether compiler supports -Werror... yes\n", "checking whether compiler supports -herror_on_warning... no\n", "checking whether strerror_r returns char with gnu source is compilable... yes\n", "checking for stdbool.h that conforms to C99... yes\n", "checking for _Bool... yes\n", "configure: creating ./config.status\n", "config.status: creating Makefile\n", "config.status: creating jemalloc.pc\n", "config.status: creating doc/html.xsl\n", "config.status: creating doc/manpages.xsl\n", "config.status: creating doc/jemalloc.xml\n", "config.status: creating include/jemalloc/jemalloc_macros.h\n", "config.status: creating include/jemalloc/jemalloc_protos.h\n", "config.status: creating include/jemalloc/jemalloc_typedefs.h\n", "config.status: creating include/jemalloc/internal/jemalloc_preamble.h\n", "config.status: creating test/test.sh\n", "config.status: creating test/include/test/jemalloc_test.h\n", "config.status: creating config.stamp\n", "config.status: creating bin/jemalloc-config\n", "config.status: creating bin/jemalloc.sh\n", "config.status: creating bin/jeprof\n", "config.status: creating include/jemalloc/jemalloc_defs.h\n", "config.status: creating include/jemalloc/internal/jemalloc_internal_defs.h\n", "config.status: creating test/include/test/jemalloc_test_defs.h\n", "config.status: executing include/jemalloc/internal/public_symbols.txt commands\n", "config.status: executing include/jemalloc/internal/private_symbols.awk commands\n", "config.status: executing include/jemalloc/internal/private_symbols_jet.awk commands\n", "config.status: executing include/jemalloc/internal/public_namespace.h commands\n", "config.status: executing include/jemalloc/internal/public_unnamespace.h commands\n", "config.status: executing include/jemalloc/internal/size_classes.h commands\n", "config.status: executing include/jemalloc/jemalloc_protos_jet.h commands\n", "config.status: executing include/jemalloc/jemalloc_rename.h commands\n", "config.status: executing include/jemalloc/jemalloc_mangle.h commands\n", "config.status: executing include/jemalloc/jemalloc_mangle_jet.h commands\n", "config.status: executing include/jemalloc/jemalloc.h commands\n", "configure: WARNING: unrecognized options: --enable-cc-silence\n", "===============================================================================\n", "jemalloc version : 5.1.0-0-g0\n", "library revision : 2\n", "\n", "CONFIG : --with-version=5.1.0-0-g0 --with-lg-quantum=3 --with-jemalloc-prefix=je_ --enable-cc-silence 'CFLAGS=-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops ' LDFLAGS=\n", "CC : gcc\n", "CONFIGURE_CFLAGS : -std=gnu11 -Wall -Wsign-compare -Wundef -Wno-format-zero-length -pipe -g3 -fvisibility=hidden -O3 -funroll-loops\n", "SPECIFIED_CFLAGS : -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops \n", "EXTRA_CFLAGS : \n", "CPPFLAGS : -D_GNU_SOURCE -D_REENTRANT\n", "CXX : g++\n", "CONFIGURE_CXXFLAGS : -Wall -g3 -fvisibility=hidden -O3\n", "SPECIFIED_CXXFLAGS : \n", "EXTRA_CXXFLAGS : \n", "LDFLAGS : \n", "EXTRA_LDFLAGS : \n", "DSO_LDFLAGS : -shared -Wl,-soname,$(@F)\n", "LIBS : -lm -lstdc++ -lpthread -ldl\n", "RPATH_EXTRA : \n", "\n", "XSLTPROC : false\n", "XSLROOT : \n", "\n", "PREFIX : /usr/local\n", "BINDIR : /usr/local/bin\n", "DATADIR : /usr/local/share\n", "INCLUDEDIR : /usr/local/include\n", "LIBDIR : /usr/local/lib\n", "MANDIR : /usr/local/share/man\n", "\n", "srcroot : \n", "abs_srcroot : /content/milvus/core/milvus/scripts/redis-stable/deps/jemalloc/\n", "objroot : \n", "abs_objroot : /content/milvus/core/milvus/scripts/redis-stable/deps/jemalloc/\n", "\n", "JEMALLOC_PREFIX : je_\n", "JEMALLOC_PRIVATE_NAMESPACE\n", " : je_\n", "install_suffix : \n", "malloc_conf : \n", "autogen : 0\n", "debug : 0\n", "stats : 1\n", "prof : 0\n", "prof-libunwind : 0\n", "prof-libgcc : 0\n", "prof-gcc : 0\n", "fill : 1\n", "utrace : 0\n", "xmalloc : 0\n", "log : 0\n", "lazy_lock : 0\n", "cache-oblivious : 1\n", "cxx : 1\n", "===============================================================================\n", "cd jemalloc && make CFLAGS=\"-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops \" LDFLAGS=\"\" lib/libjemalloc.a\n", "make[2]: Entering directory '/content/milvus/core/milvus/scripts/redis-stable/deps/jemalloc'\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/jemalloc.sym.o src/jemalloc.c\n", "nm -a src/jemalloc.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/jemalloc.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/arena.sym.o src/arena.c\n", "nm -a src/arena.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/arena.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/background_thread.sym.o src/background_thread.c\n", "nm -a src/background_thread.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/background_thread.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/base.sym.o src/base.c\n", "nm -a src/base.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/base.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/bin.sym.o src/bin.c\n", "nm -a src/bin.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/bin.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/bitmap.sym.o src/bitmap.c\n", "nm -a src/bitmap.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/bitmap.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/ckh.sym.o src/ckh.c\n", "nm -a src/ckh.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/ckh.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/ctl.sym.o src/ctl.c\n", "nm -a src/ctl.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/ctl.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/div.sym.o src/div.c\n", "nm -a src/div.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/div.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/extent.sym.o src/extent.c\n", "nm -a src/extent.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/extent.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/extent_dss.sym.o src/extent_dss.c\n", "nm -a src/extent_dss.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/extent_dss.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/extent_mmap.sym.o src/extent_mmap.c\n", "nm -a src/extent_mmap.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/extent_mmap.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/hash.sym.o src/hash.c\n", "nm -a src/hash.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/hash.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/hooks.sym.o src/hooks.c\n", "nm -a src/hooks.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/hooks.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/large.sym.o src/large.c\n", "nm -a src/large.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/large.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/log.sym.o src/log.c\n", "nm -a src/log.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/log.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/malloc_io.sym.o src/malloc_io.c\n", "nm -a src/malloc_io.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/malloc_io.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/mutex.sym.o src/mutex.c\n", "nm -a src/mutex.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/mutex.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/mutex_pool.sym.o src/mutex_pool.c\n", "nm -a src/mutex_pool.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/mutex_pool.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/nstime.sym.o src/nstime.c\n", "nm -a src/nstime.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/nstime.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/pages.sym.o src/pages.c\n", "nm -a src/pages.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/pages.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/prng.sym.o src/prng.c\n", "nm -a src/prng.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/prng.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/prof.sym.o src/prof.c\n", "nm -a src/prof.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/prof.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/rtree.sym.o src/rtree.c\n", "nm -a src/rtree.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/rtree.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/stats.sym.o src/stats.c\n", "nm -a src/stats.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/stats.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/sz.sym.o src/sz.c\n", "nm -a src/sz.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/sz.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/tcache.sym.o src/tcache.c\n", "nm -a src/tcache.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/tcache.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/ticker.sym.o src/ticker.c\n", "nm -a src/ticker.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/ticker.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/tsd.sym.o src/tsd.c\n", "nm -a src/tsd.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/tsd.sym\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -DJEMALLOC_NO_PRIVATE_NAMESPACE -o src/witness.sym.o src/witness.c\n", "nm -a src/witness.sym.o | mawk -f include/jemalloc/internal/private_symbols.awk > src/witness.sym\n", "/bin/sh include/jemalloc/internal/private_namespace.sh src/jemalloc.sym src/arena.sym src/background_thread.sym src/base.sym src/bin.sym src/bitmap.sym src/ckh.sym src/ctl.sym src/div.sym src/extent.sym src/extent_dss.sym src/extent_mmap.sym src/hash.sym src/hooks.sym src/large.sym src/log.sym src/malloc_io.sym src/mutex.sym src/mutex_pool.sym src/nstime.sym src/pages.sym src/prng.sym src/prof.sym src/rtree.sym src/stats.sym src/sz.sym src/tcache.sym src/ticker.sym src/tsd.sym src/witness.sym > include/jemalloc/internal/private_namespace.gen.h\n", "cp include/jemalloc/internal/private_namespace.gen.h include/jemalloc/internal/private_namespace.gen.h\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/jemalloc.o src/jemalloc.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/arena.o src/arena.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/background_thread.o src/background_thread.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/base.o src/base.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/bin.o src/bin.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/bitmap.o src/bitmap.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ckh.o src/ckh.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ctl.o src/ctl.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/div.o src/div.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/extent.o src/extent.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/extent_dss.o src/extent_dss.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/extent_mmap.o src/extent_mmap.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/hash.o src/hash.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/hooks.o src/hooks.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/large.o src/large.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/log.o src/log.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/malloc_io.o src/malloc_io.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/mutex.o src/mutex.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/mutex_pool.o src/mutex_pool.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/nstime.o src/nstime.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/pages.o src/pages.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/prng.o src/prng.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/prof.o src/prof.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/rtree.o src/rtree.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/stats.o src/stats.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/sz.o src/sz.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/tcache.o src/tcache.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ticker.o src/ticker.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/tsd.o src/tsd.c\n", "gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/witness.o src/witness.c\n", "g++ -Wall -g3 -fvisibility=hidden -O3 -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/jemalloc_cpp.o src/jemalloc_cpp.cpp\n", "ar crus lib/libjemalloc.a src/jemalloc.o src/arena.o src/background_thread.o src/base.o src/bin.o src/bitmap.o src/ckh.o src/ctl.o src/div.o src/extent.o src/extent_dss.o src/extent_mmap.o src/hash.o src/hooks.o src/large.o src/log.o src/malloc_io.o src/mutex.o src/mutex_pool.o src/nstime.o src/pages.o src/prng.o src/prof.o src/rtree.o src/stats.o src/sz.o src/tcache.o src/ticker.o src/tsd.o src/witness.o src/jemalloc_cpp.o\n", "ar: `u' modifier ignored since `D' is the default (see `U')\n", "make[2]: Leaving directory '/content/milvus/core/milvus/scripts/redis-stable/deps/jemalloc'\n", "make[1]: Leaving directory '/content/milvus/core/milvus/scripts/redis-stable/deps'\n", " \u001b[34mCC\u001b[0m \u001b[33madlist.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mquicklist.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mae.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33manet.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mdict.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mserver.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33msds.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mzmalloc.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mlzf_c.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mlzf_d.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mpqsort.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mzipmap.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33msha1.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mziplist.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mrelease.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mnetworking.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mutil.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mobject.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mdb.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mreplication.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mrdb.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mt_string.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mt_list.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mt_set.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mt_zset.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mt_hash.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mconfig.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33maof.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mpubsub.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mmulti.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mdebug.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33msort.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mintset.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33msyncio.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mcluster.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mcrc16.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mendianconv.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mslowlog.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mscripting.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mbio.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mrio.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mrand.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mmemtest.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mcrcspeed.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mcrc64.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mbitops.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33msentinel.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mnotify.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33msetproctitle.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mblocked.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mhyperloglog.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mlatency.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33msparkline.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mredis-check-rdb.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mredis-check-aof.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mgeo.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mlazyfree.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mmodule.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mevict.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mexpire.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mgeohash.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mgeohash_helper.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mchildinfo.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mdefrag.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33msiphash.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mrax.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mt_stream.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mlistpack.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mlocaltime.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mlolwut.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mlolwut5.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mlolwut6.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33macl.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mgopher.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mtracking.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mconnection.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mtls.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33msha256.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mtimeout.o\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33msetcpuaffinity.o\u001b[0m\n", " \u001b[34;1mLINK\u001b[0m \u001b[37;1mredis-server\u001b[0m\n", " \u001b[34;1mINSTALL\u001b[0m \u001b[37;1mredis-sentinel\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mredis-cli.o\u001b[0m\n", " \u001b[34;1mLINK\u001b[0m \u001b[37;1mredis-cli\u001b[0m\n", " \u001b[34mCC\u001b[0m \u001b[33mredis-benchmark.o\u001b[0m\n", " \u001b[34;1mLINK\u001b[0m \u001b[37;1mredis-benchmark\u001b[0m\n", " \u001b[34;1mINSTALL\u001b[0m \u001b[37;1mredis-check-rdb\u001b[0m\n", " \u001b[34;1mINSTALL\u001b[0m \u001b[37;1mredis-check-aof\u001b[0m\n", "\n", "Hint: It's a good idea to run 'make test' ;)\n", "\n", "42370:C 23 Jun 2021 14:54:34.593 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo\n", "42370:C 23 Jun 2021 14:54:34.593 # Redis version=6.0.5, bits=64, commit=3fc81236, modified=0, pid=42370, just started\n", "42370:C 23 Jun 2021 14:54:34.593 # Warning: no config file specified, using the default config. In order to specify a config file use ./redis-server /path/to/redis.conf\n", " _._ \n", " _.-``__ ''-._ \n", " _.-`` `. `_. ''-._ Redis 6.0.5 (3fc81236/0) 64 bit\n", " .-`` .-```. ```\\/ _.,_ ''-._ \n", " ( ' , .-` | `, ) Running in standalone mode\n", " |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379\n", " | `-._ `._ / _.-' | PID: 42370\n", " `-._ `-._ `-./ _.-' _.-' \n", " |`-._`-._ `-.__.-' _.-'_.-'| \n", " | `-._`-._ _.-'_.-' | http://redis.io \n", " `-._ `-._`-.__.-'_.-' _.-' \n", " |`-._`-._ `-.__.-' _.-'_.-'| \n", " | `-._`-._ _.-'_.-' | \n", " `-._ `-._`-.__.-'_.-' _.-' \n", " `-._ `-.__.-' _.-' \n", " `-._ _.-' \n", " `-.__.-' \n", "\n", "42370:M 23 Jun 2021 14:54:34.595 # Server initialized\n", "42370:M 23 Jun 2021 14:54:34.595 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.\n", "42370:M 23 Jun 2021 14:54:34.595 * Ready to accept connections\n", "42370:signal-handler (1624462946) Received SIGINT scheduling shutdown...\n", "42370:M 23 Jun 2021 15:42:26.502 # User requested shutdown...\n", "42370:M 23 Jun 2021 15:42:26.502 * Saving the final RDB snapshot before exiting.\n", "42370:M 23 Jun 2021 15:42:26.510 * DB saved on disk\n", "42370:M 23 Jun 2021 15:42:26.510 # Redis is now ready to exit, bye bye...\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "uOpL7Krfm1jk", "outputId": "5c32eb6b-30bf-4e77-db61-9a0ac6220863" }, "source": [ "! nohup ./redis-stable/src/redis-server > redis_nohup.out &\n", "! cat redis_nohup.out" ], "execution_count": 22, "outputs": [ { "output_type": "stream", "text": [ "nohup: redirecting stderr to stdout\n", "42581:C 23 Jun 2021 16:02:32.639 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo\n", "42581:C 23 Jun 2021 16:02:32.639 # Redis version=6.0.5, bits=64, commit=3fc81236, modified=0, pid=42581, just started\n", "42581:C 23 Jun 2021 16:02:32.639 # Warning: no config file specified, using the default config. In order to specify a config file use ./redis-stable/src/redis-server /path/to/redis.conf\n", "42581:M 23 Jun 2021 16:02:32.641 * Running mode=standalone, port=6379.\n", "42581:M 23 Jun 2021 16:02:32.641 # Server initialized\n", "42581:M 23 Jun 2021 16:02:32.641 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.\n", "42581:M 23 Jun 2021 16:02:32.642 * Ready to accept connections\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "4K0PQx_Qpk-d" }, "source": [ "!pip install -U grpcio" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "ryH_eaIEpcuF", "outputId": "32f9e0db-636f-4698-e1c8-8dee54961fcd" }, "source": [ "%cd /content" ], "execution_count": 3, "outputs": [ { "output_type": "stream", "text": [ "/content\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "cac14f13" }, "source": [ "### Downloading Pretrained Models\n", "\n", "This PaddlePaddle model is used to transform user information into vectors." ] }, { "cell_type": "code", "metadata": { "scrolled": true, "id": "b52f00c3", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "c20670c6-1a1d-4ec8-edf5-4981c8499a0f" }, "source": [ "!wget https://paddlerec.bj.bcebos.com/aistudio/user_vector.tar.gz --no-check-certificate\n", "!mkdir -p movie_recommender/user_vector_model\n", "!tar xf user_vector.tar.gz -C movie_recommender/user_vector_model/\n", "!rm user_vector.tar.gz" ], "execution_count": 4, "outputs": [ { "output_type": "stream", "text": [ "--2021-06-23 16:13:35-- https://paddlerec.bj.bcebos.com/aistudio/user_vector.tar.gz\n", "Resolving paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)... 103.235.46.61, 2409:8c00:6c21:10ad:0:ff:b00e:67d\n", "Connecting to paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)|103.235.46.61|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 33650924 (32M) [application/x-gzip]\n", "Saving to: ‘user_vector.tar.gz’\n", "\n", "user_vector.tar.gz 100%[===================>] 32.09M 7.94MB/s in 5.4s \n", "\n", "2021-06-23 16:13:42 (5.95 MB/s) - ‘user_vector.tar.gz’ saved [33650924/33650924]\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "1ab3a252" }, "source": [ "Downloading Data" ] }, { "cell_type": "code", "metadata": { "id": "39a7facb", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "86a36475-e468-447b-d572-9b5d8eb6c53f" }, "source": [ "# Download movie information\n", "!wget -P movie_recommender https://paddlerec.bj.bcebos.com/aistudio/movies.dat --no-check-certificate\n", "# Download movie vecotrs\n", "!wget -P movie_recommender https://paddlerec.bj.bcebos.com/aistudio/movie_vectors.txt --no-check-certificate" ], "execution_count": 5, "outputs": [ { "output_type": "stream", "text": [ "--2021-06-23 16:13:43-- https://paddlerec.bj.bcebos.com/aistudio/movies.dat\n", "Resolving paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)... 103.235.46.61, 2409:8c00:6c21:10ad:0:ff:b00e:67d\n", "Connecting to paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)|103.235.46.61|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 171308 (167K) [application/octet-stream]\n", "Saving to: ‘movie_recommender/movies.dat’\n", "\n", "movies.dat 100%[===================>] 167.29K 167KB/s in 1.0s \n", "\n", "2021-06-23 16:13:46 (167 KB/s) - ‘movie_recommender/movies.dat’ saved [171308/171308]\n", "\n", "--2021-06-23 16:13:46-- https://paddlerec.bj.bcebos.com/aistudio/movie_vectors.txt\n", "Resolving paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)... 103.235.46.61, 2409:8c00:6c21:10ad:0:ff:b00e:67d\n", "Connecting to paddlerec.bj.bcebos.com (paddlerec.bj.bcebos.com)|103.235.46.61|:443... connected.\n", "HTTP request sent, awaiting response... 200 OK\n", "Length: 1095505 (1.0M) [text/plain]\n", "Saving to: ‘movie_recommender/movie_vectors.txt’\n", "\n", "movie_vectors.txt 100%[===================>] 1.04M 648KB/s in 1.7s \n", "\n", "2021-06-23 16:13:49 (648 KB/s) - ‘movie_recommender/movie_vectors.txt’ saved [1095505/1095505]\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "e994eb1e-aa76-446b-98c6-02c74f050ba5" }, "source": [ "Importing Movies into Milvus" ] }, { "cell_type": "markdown", "metadata": { "id": "3a999eeb-bcc6-4800-9039-f9c57ea399f1" }, "source": [ "#### 1. Connectings to Milvus and Redis" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "fQnASRYRno5g", "outputId": "59d20c18-64d4-40b8-dc0e-50ee4eac09db" }, "source": [ "! lsof -i -P -n | grep -E 'milvus|redis'" ], "execution_count": 28, "outputs": [ { "output_type": "stream", "text": [ "milvus_se 42433 root 17u IPv4 478871 0t0 TCP *:19121 (LISTEN)\n", "milvus_se 42433 root 20u IPv4 479283 0t0 TCP *:19530 (LISTEN)\n", "redis-ser 42581 root 6u IPv6 507112 0t0 TCP *:6379 (LISTEN)\n", "redis-ser 42581 root 7u IPv4 507113 0t0 TCP *:6379 (LISTEN)\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "d8de5e40" }, "source": [ "from milvus import Milvus, IndexType, MetricType, Status\n", "import redis\n", "\n", "milv = Milvus(host = '127.0.0.1', port = 19530)\n", "r = redis.StrictRedis(host=\"127.0.0.1\", port=6379) " ], "execution_count": 2, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "id": "KGwZgi9Hqdhs", "outputId": "c3e09d0e-d937-4daf-ab4d-6c9d605b684f" }, "source": [ "milv.client_version()" ], "execution_count": 3, "outputs": [ { "output_type": "execute_result", "data": { "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" }, "text/plain": [ "'1.1.0'" ] }, "metadata": { "tags": [] }, "execution_count": 3 } ] }, { "cell_type": "markdown", "metadata": { "id": "a3c114a7" }, "source": [ "#### 2. Loading Movies into Redis\n", "We begin by loading all the movie files into redis. " ] }, { "cell_type": "code", "metadata": { "id": "f56cf19c" }, "source": [ "import json\n", "import codecs\n", "\n", "#1::Toy Story (1995)::Animation|Children's|Comedy\n", "def process_movie(lines, redis_cli):\n", " for line in lines:\n", " if len(line.strip()) == 0:\n", " continue\n", " tmp = line.strip().split(\"::\")\n", " movie_id = tmp[0]\n", " title = tmp[1]\n", " genre_group = tmp[2]\n", " tmp = genre_group.strip().split(\"|\")\n", " genre = tmp\n", " movie_info = {\"movie_id\" : movie_id,\n", " \"title\" : title,\n", " \"genre\" : genre\n", " }\n", " redis_cli.set(\"{}##movie_info\".format(movie_id), json.dumps(movie_info))\n", " \n", "with codecs.open(\"movie_recommender/movies.dat\", \"r\",encoding='utf-8',errors='ignore') as f:\n", " lines = f.readlines()\n", " process_movie(lines, r)" ], "execution_count": 4, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "a54a6046" }, "source": [ "#### 3. Creating Partition and Collection in Milvus" ] }, { "cell_type": "code", "metadata": { "id": "ef3ef1f7", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "2f9f5133-bfc1-47b8-faf3-66984ea71774" }, "source": [ "COLLECTION_NAME = 'demo_films'\n", "PARTITION_NAME = 'Movie'\n", "\n", "#Dropping collection for clean slate run\n", "milv.drop_collection(COLLECTION_NAME)\n", "\n", "\n", "param = {'collection_name':COLLECTION_NAME, \n", " 'dimension':32, \n", " 'index_file_size':2048, \n", " 'metric_type':MetricType.L2\n", " }\n", "\n", "milv.create_collection(param)\n", "# milv.create_partition(COLLECTION_NAME, PARTITION_NAME)" ], "execution_count": 5, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "Status(code=0, message='Create collection successfully!')" ] }, "metadata": { "tags": [] }, "execution_count": 5 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "399Cxz4cqhZ4", "outputId": "5ac70704-2877-4f59-fa0a-3df4db2ded6b" }, "source": [ "milv.get_collection_info(COLLECTION_NAME)" ], "execution_count": 7, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(Status(code=0, message='Describe collection successfully!'),\n", " CollectionSchema(collection_name='demo_films', dimension=32, index_file_size=2048, metric_type=))" ] }, "metadata": { "tags": [] }, "execution_count": 7 } ] }, { "cell_type": "markdown", "metadata": { "id": "d298372e" }, "source": [ "#### 4. Getting Embeddings and IDs\n", "The vectors in `movie_vectors.txt` are obtained from the `user_vector_model` downloaded above. So we can directly get the vectors and the IDs by reading the file." ] }, { "cell_type": "code", "metadata": { "id": "1aaee36b" }, "source": [ "def get_vectors():\n", " with codecs.open(\"movie_recommender/movie_vectors.txt\", \"r\", encoding='utf-8', errors='ignore') as f:\n", " lines = f.readlines()\n", " ids = [int(line.split(\":\")[0]) for line in lines]\n", " embeddings = []\n", " for line in lines:\n", " line = line.strip().split(\":\")[1][1:-1]\n", " str_nums = line.split(\",\")\n", " emb = [float(x) for x in str_nums]\n", " embeddings.append(emb)\n", " return ids, embeddings\n", "\n", "ids, embeddings = get_vectors()" ], "execution_count": 8, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "3a6140b1" }, "source": [ "#### 4. Importing Vectors into Milvus\n", "Import vectors into the partition **Movie** under the collection **demo_films**." ] }, { "cell_type": "code", "metadata": { "id": "4ac4cfff", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "8aff463d-ec4b-4ecc-e076-2d626d54d536" }, "source": [ "# status = milv.insert(collection_name=COLLECTION_NAME, records=embeddings, ids=ids, partition_tag=PARTITION_NAME)\n", "status = milv.insert(collection_name=COLLECTION_NAME, records=embeddings, ids=ids)\n", "status[0]" ], "execution_count": 9, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "Status(code=0, message='Add vectors successfully!')" ] }, "metadata": { "tags": [] }, "execution_count": 9 } ] }, { "cell_type": "markdown", "metadata": { "id": "e93feb30" }, "source": [ "### Recalling Vectors in Milvus\n", "#### 1. Genarating User Embeddings\n", "Pass in the gender, age and occupation of the user we want to recommend. **user_vector_model** model will generate the corresponding user vector.\n", "Occupation is chosen from the following choices:\n", "* 0: \"other\" or not specified\n", "* 1: \"academic/educator\"\n", "* 2: \"artist\"\n", "* 3: \"clerical/admin\"\n", "* 4: \"college/grad student\"\n", "* 5: \"customer service\"\n", "* 6: \"doctor/health care\"\n", "* 7: \"executive/managerial\"\n", "* 8: \"farmer\"\n", "* 9: \"homemaker\"\n", "* 10: \"K-12 student\"\n", "* 11: \"lawyer\"\n", "* 12: \"programmer\"\n", "* 13: \"retired\"\n", "* 14: \"sales/marketing\"\n", "* 15: \"scientist\"\n", "* 16: \"self-employed\"\n", "* 17: \"technician/engineer\"\n", "* 18: \"tradesman/craftsman\"\n", "* 19: \"unemployed\"\n", "* 20: \"writer\"" ] }, { "cell_type": "code", "metadata": { "tags": [], "id": "1a35a9d4", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "6eaf7dbd-95e8-43a6-c43a-9e2c57fb5c81" }, "source": [ "import numpy as np\n", "from paddle_serving_app.local_predict import LocalPredictor\n", "\n", "class RecallServerServicer(object):\n", " def __init__(self):\n", " self.uv_client = LocalPredictor()\n", " self.uv_client.load_model_config(\"movie_recommender/user_vector_model/serving_server_dir\") \n", " \n", " def hash2(self, a):\n", " return hash(a) % 1000000\n", "\n", " def get_user_vector(self):\n", " dic = {\"userid\": [], \"gender\": [], \"age\": [], \"occupation\": []}\n", " lod = [0]\n", " dic[\"userid\"].append(self.hash2('0'))\n", " dic[\"gender\"].append(self.hash2('M'))\n", " dic[\"age\"].append(self.hash2('23'))\n", " dic[\"occupation\"].append(self.hash2('6'))\n", " lod.append(1)\n", "\n", " dic[\"userid.lod\"] = lod\n", " dic[\"gender.lod\"] = lod\n", " dic[\"age.lod\"] = lod\n", " dic[\"occupation.lod\"] = lod\n", " for key in dic:\n", " dic[key] = np.array(dic[key]).astype(np.int64).reshape(len(dic[key]),1)\n", " fetch_map = self.uv_client.predict(feed=dic, fetch=[\"save_infer_model/scale_0.tmp_1\"], batch=True)\n", " return fetch_map[\"save_infer_model/scale_0.tmp_1\"].tolist()[0]\n", "\n", "recall = RecallServerServicer()\n", "user_vector = recall.get_user_vector()" ], "execution_count": 10, "outputs": [ { "output_type": "stream", "text": [ "2021-06-23 16:29:24,262 - INFO - LocalPredictor load_model_config params: model_path:movie_recommender/user_vector_model/serving_server_dir, use_gpu:False, gpu_id:0, use_profile:False, thread_num:1, mem_optim:True, ir_optim:False, use_trt:False, use_lite:False, use_xpu: False, use_feed_fetch_ops:False\n" ], "name": "stderr" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "vfzaCguwtLgL", "outputId": "78506dc2-aa85-497b-f412-cdc31fb38a31" }, "source": [ "user_vector" ], "execution_count": 11, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "[0.0,\n", " 4.911433696746826,\n", " 4.132595062255859,\n", " 3.2255895137786865,\n", " 0.0,\n", " 4.944108963012695,\n", " 0.0,\n", " 0.0,\n", " 1.27165687084198,\n", " 3.1072912216186523,\n", " 0.0,\n", " 0.0,\n", " 0.0,\n", " 0.0,\n", " 0.0,\n", " 0.0,\n", " 0.0,\n", " 0.0,\n", " 0.0,\n", " 0.0,\n", " 0.0,\n", " 0.0,\n", " 0.0,\n", " 0.0,\n", " 0.0,\n", " 1.9184402227401733,\n", " 0.0,\n", " 0.0,\n", " 0.0,\n", " 4.42396354675293,\n", " 2.0686450004577637,\n", " 0.0]" ] }, "metadata": { "tags": [] }, "execution_count": 11 } ] }, { "cell_type": "markdown", "metadata": { "tags": [], "id": "e15ea6e8" }, "source": [ "#### 2. Searching\n", "Pass in the user vector, and then recall vectors in the previously imported data collection and partition." ] }, { "cell_type": "code", "metadata": { "id": "e4d91d02" }, "source": [ "TOP_K = 20\n", "SEARCH_PARAM = {'nprobe': 20}\n", "status, results = milv.search(collection_name=COLLECTION_NAME, query_records=[user_vector], top_k=TOP_K, params=SEARCH_PARAM)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "9c847608" }, "source": [ "#### 3. Returning Information by IDs" ] }, { "cell_type": "code", "metadata": { "id": "90a56325", "outputId": "bffcbddb-2b25-4320-b8c9-1af5b0b6a75f" }, "source": [ "recall_results = []\n", "for x in results[0]:\n", " recall_results.append(r.get(\"{}##movie_info\".format(x.id)).decode('utf-8'))\n", "recall_results" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['{\"movie_id\": \"760\", \"title\": \"Stalingrad (1993)\", \"genre\": [\"War\"]}',\n", " '{\"movie_id\": \"1350\", \"title\": \"Omen, The (1976)\", \"genre\": [\"Horror\"]}',\n", " '{\"movie_id\": \"1258\", \"title\": \"Shining, The (1980)\", \"genre\": [\"Horror\"]}',\n", " '{\"movie_id\": \"632\", \"title\": \"Land and Freedom (Tierra y libertad) (1995)\", \"genre\": [\"War\"]}',\n", " '{\"movie_id\": \"3007\", \"title\": \"American Movie (1999)\", \"genre\": [\"Documentary\"]}',\n", " '{\"movie_id\": \"2086\", \"title\": \"One Magic Christmas (1985)\", \"genre\": [\"Drama\", \"Fantasy\"]}',\n", " '{\"movie_id\": \"1051\", \"title\": \"Trees Lounge (1996)\", \"genre\": [\"Drama\"]}',\n", " '{\"movie_id\": \"3920\", \"title\": \"Faraway, So Close (In Weiter Ferne, So Nah!) (1993)\", \"genre\": [\"Drama\", \"Fantasy\"]}',\n", " '{\"movie_id\": \"1303\", \"title\": \"Man Who Would Be King, The (1975)\", \"genre\": [\"Adventure\"]}',\n", " '{\"movie_id\": \"652\", \"title\": \"301, 302 (1995)\", \"genre\": [\"Mystery\"]}',\n", " '{\"movie_id\": \"1605\", \"title\": \"Excess Baggage (1997)\", \"genre\": [\"Adventure\", \"Romance\"]}',\n", " '{\"movie_id\": \"1275\", \"title\": \"Highlander (1986)\", \"genre\": [\"Action\", \"Adventure\"]}',\n", " '{\"movie_id\": \"1126\", \"title\": \"Drop Dead Fred (1991)\", \"genre\": [\"Comedy\", \"Fantasy\"]}',\n", " '{\"movie_id\": \"792\", \"title\": \"Hungarian Fairy Tale, A (1987)\", \"genre\": [\"Fantasy\"]}',\n", " '{\"movie_id\": \"2228\", \"title\": \"Mountain Eagle, The (1926)\", \"genre\": [\"Drama\"]}',\n", " '{\"movie_id\": \"2659\", \"title\": \"It Came from Hollywood (1982)\", \"genre\": [\"Comedy\", \"Documentary\"]}',\n", " '{\"movie_id\": \"2545\", \"title\": \"Relax... It\\'s Just Sex (1998)\", \"genre\": [\"Comedy\"]}',\n", " '{\"movie_id\": \"1289\", \"title\": \"Koyaanisqatsi (1983)\", \"genre\": [\"Documentary\", \"War\"]}',\n", " '{\"movie_id\": \"2537\", \"title\": \"Beyond the Poseidon Adventure (1979)\", \"genre\": [\"Adventure\"]}',\n", " '{\"movie_id\": \"2864\", \"title\": \"Splendor (1999)\", \"genre\": [\"Comedy\"]}']" ] }, "metadata": { "tags": [] }, "execution_count": 23 } ] }, { "cell_type": "markdown", "metadata": { "id": "d4f7e3c5" }, "source": [ "### Conclusion" ] }, { "cell_type": "markdown", "metadata": { "id": "843120ee" }, "source": [ "After completing the recall service, the results can be further sorted using the **movie_recommender** model, and then the movies with high similarity scores can be recommended to users. You can try this deployable recommendation system using this [quick start](QUICK_START.md)." ] } ] }