{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Joblib as serialization of TF data\n", "\n", "[Joblib](https://joblib.readthedocs.io/en/latest/persistence.html#persistence)\n", "is a data serialization format used in Big Data use cases.\n", "\n", "```\n", "pip3 install joblib\n", "```\n", "\n", "It looks attractive from the specifactions, but how will it perform?\n", "\n", "As a simple test, we take the feature data for `g_word_utf8`.\n", "It is a map from the numbers 1 to 426584 to Hebrew word occurrences (Unicode strings).\n", "\n", "In Text-Fabric we have a representation in plain text and a compressed, pickled representation.\n", "\n", "# Outcome\n", "\n", "Text-Fabric is much faster in loading this kind of data.\n", "\n", "The size of the joblib uncompressed serialization is much bigger than the TF text representation.\n", "\n", "The size of the gzipped joblib serialization is still 30% more than gzipped, pickled TF serialization.\n", "\n", "# Detailed comparison\n", "\n", "name | kind | size | load time\n", ":--- | :--- | ---: | ---:\n", "g_word_utf8.tf | tf: plain unicode text | 5.4 MB | 1.6 s\n", "g_word_utf8.tfx | tf: gzipped binary |3.2 MB | 0.2 s\n", "g_word_utf8.joblib | joblib: uncompressed | 11.2 MB | 2.0 s\n", "g_word_utf8.joblib.gz | joblib: gzipped | 3.2 MB | 2.7 s\n", "\n", "# Conclusion\n", "\n", "**We do not see reasons to replace the TF feature data serialization by joblib.**" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2018-05-09T13:52:56.726798Z", "start_time": "2018-05-09T13:52:56.660991Z" } }, "outputs": [], "source": [ "import os\n", "import gzip\n", "import joblib\n", "\n", "from tf.fabric import Fabric\n", "\n", "GZIP_LEVEL = 2 # same as used in Text-Fabric" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load from the textual data" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2018-05-09T13:53:03.953876Z", "start_time": "2018-05-09T13:52:57.594126Z" }, "scrolled": true }, "outputs": [], "source": [ "VERSION = 'c'\n", "BHSA = f'BHSA/tf/{VERSION}'\n", "PARA = f'parallels/tf/{VERSION}'" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2018-05-09T13:53:03.953876Z", "start_time": "2018-05-09T13:52:57.594126Z" }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "This is Text-Fabric 5.5.22\n", "Api reference : https://annotation.github.io/text-fabric/Api/Fabric/\n", "Tutorial : https://github.com/annotation/text-fabric/blob/master/docs/tutorial.ipynb\n", "Example data : https://github.com/annotation/text-fabric-data\n", "\n", "117 features found and 0 ignored\n", " 0.00s loading features ...\n", " | 1.60s T g_word_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c\n", " 6.02s All features loaded/computed - for details use loadLog()\n" ] } ], "source": [ "TF = Fabric(locations='~/github/etcbc', modules=[BHSA, PARA])\n", "api = TF.load('')\n", "api.makeAvailableIn(globals())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The load time is **~ 1.6 seconds**.\n", "\n", "But during this time, the textual data has been compiled and written to a binary form.\n", "Let's load again.\n", "\n", "## Load from binary data" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "This is Text-Fabric 5.5.22\n", "Api reference : https://annotation.github.io/text-fabric/Api/Fabric/\n", "Tutorial : https://github.com/annotation/text-fabric/blob/master/docs/tutorial.ipynb\n", "Example data : https://github.com/annotation/text-fabric-data\n", "\n", "117 features found and 0 ignored\n", " 0.00s loading features ...\n", " 5.27s All features loaded/computed - for details use loadLog()\n" ] } ], "source": [ "TF = Fabric(locations='~/github/etcbc', modules=[BHSA, PARA])\n", "api = TF.load('')\n", "api.makeAvailableIn(globals())" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " | 0.03s B otype from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.53s B oslots from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.01s B book from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.01s B chapter from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.01s B verse from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.13s B g_cons from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.18s B g_cons_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.14s B g_lex from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.19s B g_lex_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.16s B g_word from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.33s B g_word_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.16s B lex0 from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.25s B lex_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B qere from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B qere_trailer from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B qere_trailer_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.01s B qere_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.08s B trailer from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.09s B trailer_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B __levels__ from otype, oslots, otext\n", " | 0.03s B __order__ from otype, oslots, __levels__\n", " | 0.04s B __rank__ from otype, __order__\n", " | 1.65s B __levUp__ from otype, oslots, __rank__\n", " | 0.88s B __levDown__ from otype, __levUp__, __rank__\n", " | 0.30s B __boundary__ from otype, oslots, __rank__\n", " | 0.01s B __sections__ from otype, oslots, otext, __levUp__, __levels__, book, chapter, verse\n", " | 0.00s B book@am from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@ar from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@bn from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@da from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@de from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@el from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@en from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@es from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@fa from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@fr from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@he from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@hi from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@id from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@ja from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@ko from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@la from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@nl from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@pa from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@pt from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@ru from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@sw from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@syc from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@tr from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@ur from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@yo from /Users/dirk/github/etcbc/BHSA/tf/c\n", " | 0.00s B book@zh from /Users/dirk/github/etcbc/BHSA/tf/c\n" ] } ], "source": [ "loadLog()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The load time of the feature `g_word_utf8` is **~ 0.2 seconds**." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Make an joblib feature data file" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "tempDir = os.path.expanduser('~/github/annotation/text-fabric/_temp/joblib')\n", "os.makedirs(tempDir, exist_ok=True)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "426584\n", "רֵאשִׁ֖ית\n" ] } ], "source": [ "feature = 'g_word_utf8'\n", "data = TF.features[feature].data\n", "print(len(data))\n", "print(data[2])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We write the feature data to an Avro data file." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "dataFile = f'{tempDir}/{feature}.joblib'" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 0.00s start writing\n", " 3.20s done\n" ] } ], "source": [ "indent(reset=True)\n", "info('start writing')\n", "joblib.dump(data, dataFile)\n", "info('done')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We make also a gzipped data file." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 0.00s start writing\n", " 3.70s done\n" ] } ], "source": [ "indent(reset=True)\n", "info('start writing')\n", "dataFileZ = f'{dataFile}.gz'\n", "joblib.dump(data, dataFileZ, compress=('gzip', GZIP_LEVEL))\n", "info('done')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load from joblib file" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 0.00s start reading\n", " 2.06s done\n", "רֵאשִׁ֖ית\n" ] } ], "source": [ "indent(reset=True)\n", "info('start reading')\n", "rData = joblib.load(dataFile)\n", "info('done')\n", "print(rData[2])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Load time **~ 2.0 seconds**." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load from joblib file (gzipped)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 0.00s start reading\n", " 2.73s done\n", "רֵאשִׁ֖ית\n" ] } ], "source": [ "indent(reset=True)\n", "info('start reading')\n", "rData = joblib.load(dataFileZ)\n", "info('done')\n", "print(rData[2])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Load time **~ 2.7 seconds**." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.0" } }, "nbformat": 4, "nbformat_minor": 2 }