{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "db5f4f9a-7776-42b3-8758-85624d4c15ea",
      "metadata": {
        "id": "db5f4f9a-7776-42b3-8758-85624d4c15ea"
      },
      "source": [
        "![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "21e9eafb",
      "metadata": {
        "id": "21e9eafb"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/legal-nlp/09.2.Entity_Resolution_Training.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "gk3kZHmNj51v",
      "metadata": {
        "collapsed": false,
        "id": "gk3kZHmNj51v"
      },
      "source": [
        "# Installation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "_914itZsj51v",
      "metadata": {
        "id": "_914itZsj51v",
        "pycharm": {
          "is_executing": true
        }
      },
      "outputs": [],
      "source": [
        "! pip install -q johnsnowlabs"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "YPsbAnNoPt0Z",
      "metadata": {
        "id": "YPsbAnNoPt0Z"
      },
      "source": [
        "## Automatic Installation\n",
        "Using my.johnsnowlabs.com SSO"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fY0lcShkj51w",
      "metadata": {
        "id": "fY0lcShkj51w",
        "pycharm": {
          "is_executing": true
        }
      },
      "outputs": [],
      "source": [
        "from johnsnowlabs import nlp, legal\n",
        "\n",
        "# nlp.install(force_browser=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "hsJvn_WWM2GL",
      "metadata": {
        "id": "hsJvn_WWM2GL"
      },
      "source": [
        "## Manual downloading\n",
        "If you are not registered in my.johnsnowlabs.com, you received a license via e-email or you are using Safari, you may need to do a manual update of the license.\n",
        "\n",
        "- Go to my.johnsnowlabs.com\n",
        "- Download your license\n",
        "- Upload it using the following command"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "i57QV3-_P2sQ",
      "metadata": {
        "id": "i57QV3-_P2sQ"
      },
      "outputs": [],
      "source": [
        "from google.colab import files\n",
        "print('Please Upload your John Snow Labs License using the button below')\n",
        "license_keys = files.upload()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xGgNdFzZP_hQ",
      "metadata": {
        "id": "xGgNdFzZP_hQ"
      },
      "source": [
        "- Install it"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "OfmmPqknP4rR",
      "metadata": {
        "id": "OfmmPqknP4rR"
      },
      "outputs": [],
      "source": [
        "nlp.install()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "DCl5ErZkNNLk",
      "metadata": {
        "id": "DCl5ErZkNNLk"
      },
      "source": [
        "# Starting"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "wRXTnNl3j51w",
      "metadata": {
        "id": "wRXTnNl3j51w"
      },
      "outputs": [],
      "source": [
        "spark = nlp.start()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6zgaKT7Khkzu",
      "metadata": {
        "id": "6zgaKT7Khkzu"
      },
      "source": [
        "## Entity Resolution Training\n",
        "\n",
        "Here, we will train a legal resolver model with a sample dataset.We will train a company name normalization model. Our dataset columns has to be object type.\n",
        "\n",
        "Let's start to train."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d6c6f9d0-b6de-4708-a783-e3af20aace47",
      "metadata": {
        "id": "d6c6f9d0-b6de-4708-a783-e3af20aace47"
      },
      "source": [
        "## Load Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5tEedT614vgo",
      "metadata": {
        "id": "5tEedT614vgo"
      },
      "outputs": [],
      "source": [
        "! wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/legal-nlp/data/sample_company_name.csv"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f9ff314c-5548-4946-971c-1aa45bf82d43",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "f9ff314c-5548-4946-971c-1aa45bf82d43",
        "outputId": "77e6ce30-651e-405b-e7e0-71744f78487b"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-33f3519c-8295-48d9-b30a-a775cc0f4d76\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>company_name</th>\n",
              "      <th>irs_number</th>\n",
              "      <th>comp_abbreviation_var</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>StepOne Personal Health, Inc.</td>\n",
              "      <td>900785095</td>\n",
              "      <td>StepOne Personal Health</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>StepOne Personal Health, Inc.</td>\n",
              "      <td>900785095</td>\n",
              "      <td>StepOne Personal Health Inc</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>StepOne Personal Health, Inc.</td>\n",
              "      <td>900785095</td>\n",
              "      <td>STEPONE PERSONAL HEALTH INC</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>StepOne Personal Health, Inc.</td>\n",
              "      <td>900785095</td>\n",
              "      <td>StepOne Personal Health inc</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>StepOne Personal Health, Inc.</td>\n",
              "      <td>900785095</td>\n",
              "      <td>StepOne Personal Health INC</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9995</th>\n",
              "      <td>INGLES MARKETS INC</td>\n",
              "      <td>560846267</td>\n",
              "      <td>Ingles Markets Inc</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9996</th>\n",
              "      <td>INGLES MARKETS INC</td>\n",
              "      <td>560846267</td>\n",
              "      <td>INGLES MARKETS Inc.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9997</th>\n",
              "      <td>INGLES MARKETS INC</td>\n",
              "      <td>560846267</td>\n",
              "      <td>INGLES MARKETS inc.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9998</th>\n",
              "      <td>INGLES MARKETS INC</td>\n",
              "      <td>560846267</td>\n",
              "      <td>INGLES MARKETS INC</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9999</th>\n",
              "      <td>INGLES MARKETS INC</td>\n",
              "      <td>560846267</td>\n",
              "      <td>INGLES MARKETS</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>10000 rows × 3 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-33f3519c-8295-48d9-b30a-a775cc0f4d76')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-33f3519c-8295-48d9-b30a-a775cc0f4d76 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-33f3519c-8295-48d9-b30a-a775cc0f4d76');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "                       company_name  irs_number        comp_abbreviation_var\n",
              "0     StepOne Personal Health, Inc.   900785095      StepOne Personal Health\n",
              "1     StepOne Personal Health, Inc.   900785095  StepOne Personal Health Inc\n",
              "2     StepOne Personal Health, Inc.   900785095  STEPONE PERSONAL HEALTH INC\n",
              "3     StepOne Personal Health, Inc.   900785095  StepOne Personal Health inc\n",
              "4     StepOne Personal Health, Inc.   900785095  StepOne Personal Health INC\n",
              "...                             ...         ...                          ...\n",
              "9995             INGLES MARKETS INC   560846267           Ingles Markets Inc\n",
              "9996             INGLES MARKETS INC   560846267          INGLES MARKETS Inc.\n",
              "9997             INGLES MARKETS INC   560846267          INGLES MARKETS inc.\n",
              "9998             INGLES MARKETS INC   560846267           INGLES MARKETS INC\n",
              "9999             INGLES MARKETS INC   560846267               INGLES MARKETS\n",
              "\n",
              "[10000 rows x 3 columns]"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "df = pd.read_csv('sample_company_name.csv')\n",
        "df"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f731862c-99cc-4ac1-8473-421743d77ca9",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f731862c-99cc-4ac1-8473-421743d77ca9",
        "outputId": "99eff647-b2a3-4b9b-99b4-540784085288"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 10000 entries, 0 to 9999\n",
            "Data columns (total 3 columns):\n",
            " #   Column                 Non-Null Count  Dtype \n",
            "---  ------                 --------------  ----- \n",
            " 0   company_name           10000 non-null  object\n",
            " 1   irs_number             10000 non-null  int64 \n",
            " 2   comp_abbreviation_var  10000 non-null  object\n",
            "dtypes: int64(1), object(2)\n",
            "memory usage: 234.5+ KB\n"
          ]
        }
      ],
      "source": [
        "df.info()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "i9xiDDJFgK4F",
      "metadata": {
        "id": "i9xiDDJFgK4F"
      },
      "outputs": [],
      "source": [
        "df['comp_abbreviation_var'] =df['comp_abbreviation_var'].astype(str)\n",
        "df['irs_number'] =df['irs_number'].astype(str)\n",
        "df['company_name'] =df['company_name'].astype(str)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3HKru2_3gM5U",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3HKru2_3gM5U",
        "outputId": "fc592cbf-b40c-4265-856d-9301c8c1397b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 10000 entries, 0 to 9999\n",
            "Data columns (total 3 columns):\n",
            " #   Column                 Non-Null Count  Dtype \n",
            "---  ------                 --------------  ----- \n",
            " 0   company_name           10000 non-null  object\n",
            " 1   irs_number             10000 non-null  object\n",
            " 2   comp_abbreviation_var  10000 non-null  object\n",
            "dtypes: object(3)\n",
            "memory usage: 234.5+ KB\n"
          ]
        }
      ],
      "source": [
        "df.info()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4a9cf2e1-44fc-4f6a-b60d-da76715568d4",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4a9cf2e1-44fc-4f6a-b60d-da76715568d4",
        "outputId": "06c46baa-2884-45ad-e2bf-1d1b81265114"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(10000, 3)"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "F8YY4nDEi9hH",
      "metadata": {
        "id": "F8YY4nDEi9hH"
      },
      "source": [
        "## Get Embeddings\n",
        "Now we will get the sentence embeddings of `comp_abbreviation_var` column."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2c2fff38-38fc-4ada-a880-52959211ee50",
      "metadata": {
        "id": "2c2fff38-38fc-4ada-a880-52959211ee50",
        "tags": []
      },
      "outputs": [],
      "source": [
        "data = spark.createDataFrame(df)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "12a6b170-bc99-4a40-a7f5-fb902c19ca7d",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "12a6b170-bc99-4a40-a7f5-fb902c19ca7d",
        "outputId": "250a2d28-1ec1-401b-ff9e-1e6ff74ad7bd",
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tfhub_use download started this may take some time.\n",
            "Approximate size to download 923.7 MB\n",
            "[OK!]\n"
          ]
        }
      ],
      "source": [
        "documentAssembler = nlp.DocumentAssembler()\\\n",
        "    .setInputCol(\"comp_abbreviation_var\")\\\n",
        "    .setOutputCol(\"sentence\")\n",
        "\n",
        "embeddings = nlp.UniversalSentenceEncoder.pretrained(\"tfhub_use\", \"en\") \\\n",
        "    .setInputCols(\"sentence\") \\\n",
        "    .setOutputCol(\"sentence_embeddings\")\n",
        "\n",
        "training_pipeline = nlp.Pipeline(stages = [\n",
        "    documentAssembler,\n",
        "    embeddings])\n",
        "\n",
        "training_model = training_pipeline.fit(data)\n",
        "\n",
        "final_data = training_model.transform(data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5VtDRIrunKX5",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5VtDRIrunKX5",
        "outputId": "6a069a56-23ef-491a-ffa5-d0794fda8016"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "+--------------------+----------+---------------------+--------------------+--------------------+\n",
            "|        company_name|irs_number|comp_abbreviation_var|            sentence| sentence_embeddings|\n",
            "+--------------------+----------+---------------------+--------------------+--------------------+\n",
            "|StepOne Personal ...| 900785095| StepOne Personal ...|[{document, 0, 22...|[{sentence_embedd...|\n",
            "|StepOne Personal ...| 900785095| StepOne Personal ...|[{document, 0, 26...|[{sentence_embedd...|\n",
            "|StepOne Personal ...| 900785095| STEPONE PERSONAL ...|[{document, 0, 26...|[{sentence_embedd...|\n",
            "|StepOne Personal ...| 900785095| StepOne Personal ...|[{document, 0, 26...|[{sentence_embedd...|\n",
            "|StepOne Personal ...| 900785095| StepOne Personal ...|[{document, 0, 26...|[{sentence_embedd...|\n",
            "|StepOne Personal ...| 900785095| StepOne Personal ...|[{document, 0, 27...|[{sentence_embedd...|\n",
            "|StepOne Personal ...| 900785095| StepOne Personal ...|[{document, 0, 27...|[{sentence_embedd...|\n",
            "|StepOne Personal ...| 900785095| Stepone Personal ...|[{document, 0, 26...|[{sentence_embedd...|\n",
            "|StepOne Personal ...| 900785095| StepOne Personal ...|[{document, 0, 27...|[{sentence_embedd...|\n",
            "|Equity One Net In...| 320467879| Equity One Net In...|[{document, 0, 24...|[{sentence_embedd...|\n",
            "|Equity One Net In...| 320467879| Equity One Net In...|[{document, 0, 24...|[{sentence_embedd...|\n",
            "|Equity One Net In...| 320467879| Equity One Net In...|[{document, 0, 24...|[{sentence_embedd...|\n",
            "|Equity One Net In...| 320467879| Equity One Net In...|[{document, 0, 25...|[{sentence_embedd...|\n",
            "|Equity One Net In...| 320467879| Equity One Net In...|[{document, 0, 20...|[{sentence_embedd...|\n",
            "|Equity One Net In...| 320467879| EQUITY ONE NET IN...|[{document, 0, 24...|[{sentence_embedd...|\n",
            "|Equity One Net In...| 320467879| Equity One Net In...|[{document, 0, 25...|[{sentence_embedd...|\n",
            "|Equity One Net In...| 320467879| Equity One Net In...|[{document, 0, 25...|[{sentence_embedd...|\n",
            "|AmeriCredit Autom...| 880475154| AmeriCredit Autom...|[{document, 0, 39...|[{sentence_embedd...|\n",
            "|GROUNDFLOOR FINAN...| 463414189| GROUNDFLOOR FINAN...|[{document, 0, 23...|[{sentence_embedd...|\n",
            "|GROUNDFLOOR FINAN...| 463414189| GROUNDFLOOR FINAN...|[{document, 0, 23...|[{sentence_embedd...|\n",
            "+--------------------+----------+---------------------+--------------------+--------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ],
      "source": [
        "final_data.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "yRtd3nRHnVMQ",
      "metadata": {
        "id": "yRtd3nRHnVMQ"
      },
      "source": [
        "We have `sentence_embeddings` column in our training dataframe that we will use as input while training the model."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Mnc5wuJ-ntEo",
      "metadata": {
        "id": "Mnc5wuJ-ntEo"
      },
      "source": [
        "## Train Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "176e74ba-6867-469b-aebc-f510f8d78a08",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "176e74ba-6867-469b-aebc-f510f8d78a08",
        "outputId": "d8ee8fdc-d23e-466c-d8e9-f2411ca18e57",
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 108 ms, sys: 13.8 ms, total: 121 ms\n",
            "Wall time: 15.3 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "use = legal.SentenceEntityResolverApproach()\\\n",
        "  .setNeighbours(50)\\\n",
        "  .setThreshold(10000)\\\n",
        "  .setInputCols(\"sentence_embeddings\")\\\n",
        "  .setLabelCol(\"company_name\")\\\n",
        "  .setOutputCol('original_company_name')\\\n",
        "  .setNormalizedCol(\"company_name\")\\\n",
        "  .setDistanceFunction(\"EUCLIDEAN\")\\\n",
        "  .setCaseSensitive(False)\\\n",
        "  .setUseAuxLabel(True)\\\n",
        "  .setAuxLabelCol('irs_number')\n",
        "\n",
        "model = use.fit(final_data)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "oELEGmD0V_Th",
      "metadata": {
        "id": "oELEGmD0V_Th"
      },
      "outputs": [],
      "source": [
        "# Save model\n",
        "model.write().overwrite().save(\"use_company_name\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "091132bb-e472-444a-bc6d-ef502dc3e1dd",
      "metadata": {
        "id": "091132bb-e472-444a-bc6d-ef502dc3e1dd"
      },
      "source": [
        "## Test Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b2141e1b-9e7d-4081-812c-6ca606cf1687",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b2141e1b-9e7d-4081-812c-6ca606cf1687",
        "outputId": "52077d9a-ba4d-4236-b783-d976b2b7c9d0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tfhub_use download started this may take some time.\n",
            "Approximate size to download 923.7 MB\n",
            "[OK!]\n"
          ]
        }
      ],
      "source": [
        "documentAssembler = nlp.DocumentAssembler()\\\n",
        "      .setInputCol(\"text\")\\\n",
        "      .setOutputCol(\"ner_chunk\")\n",
        "\n",
        "embeddings = nlp.UniversalSentenceEncoder.pretrained(\"tfhub_use\", \"en\") \\\n",
        "      .setInputCols(\"ner_chunk\") \\\n",
        "      .setOutputCol(\"sentence_embeddings\")\n",
        "    \n",
        "resolver = legal.SentenceEntityResolverModel.load(\"use_company_name\") \\\n",
        "      .setInputCols([\"sentence_embeddings\"]) \\\n",
        "      .setOutputCol(\"normalized_name\")\\\n",
        "      .setDistanceFunction(\"EUCLIDEAN\")\n",
        "\n",
        "pipeline = nlp.Pipeline(\n",
        "      stages = [\n",
        "          documentAssembler,\n",
        "          embeddings,\n",
        "          resolver,\n",
        "      ])\n",
        "\n",
        "empty_data = spark.createDataFrame([[\"\"]]).toDF(\"text\")\n",
        "\n",
        "model = pipeline.fit(empty_data)\n",
        "\n",
        "light_model= nlp.LightPipeline(model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7e0ba1ec-13d8-42b3-9a53-24cf85fd8ad7",
      "metadata": {
        "id": "7e0ba1ec-13d8-42b3-9a53-24cf85fd8ad7"
      },
      "outputs": [],
      "source": [
        "# returns LP resolution results\n",
        "\n",
        "import pandas as pd\n",
        "pd.set_option('display.max_colwidth', 0)\n",
        "\n",
        "def get_codes (lp, text, vocab='company_name', hcc=False):\n",
        "    \n",
        "    full_light_result = lp.fullAnnotate(text)\n",
        "\n",
        "    chunks = []\n",
        "    codes = []\n",
        "    begin = []\n",
        "    end = []\n",
        "    resolutions=[]\n",
        "    all_distances =[]\n",
        "    all_codes=[]\n",
        "    all_cosines = []\n",
        "    all_k_aux_labels=[]\n",
        "\n",
        "    for chunk, code in zip(full_light_result[0]['ner_chunk'], full_light_result[0][vocab]):\n",
        "            \n",
        "        begin.append(chunk.begin)\n",
        "        end.append(chunk.end)\n",
        "        chunks.append(chunk.result)\n",
        "        codes.append(code.result) \n",
        "        all_codes.append(code.metadata['all_k_results'].split(':::'))\n",
        "        resolutions.append(code.metadata['all_k_resolutions'].split(':::'))\n",
        "        all_distances.append(code.metadata['all_k_distances'].split(':::'))\n",
        "        all_cosines.append(code.metadata['all_k_cosine_distances'].split(':::'))\n",
        "        if hcc:\n",
        "            try:\n",
        "                all_k_aux_labels.append(code.metadata['all_k_aux_labels'].split(':::'))\n",
        "            except:\n",
        "                all_k_aux_labels.append([])\n",
        "        else:\n",
        "            all_k_aux_labels.append([])\n",
        "\n",
        "    df = pd.DataFrame({'chunks':chunks, 'begin': begin, 'end':end, 'code':codes, 'all_codes':all_codes, \n",
        "                       'resolutions':resolutions, 'all_k_aux_labels':all_k_aux_labels,'all_distances':all_cosines})\n",
        "    \n",
        "    if hcc:\n",
        "\n",
        "        df['billable'] = df['all_k_aux_labels'].apply(lambda x: [i.split('||')[0] for i in x])\n",
        "        df['hcc_status'] = df['all_k_aux_labels'].apply(lambda x: [i.split('||')[1] for i in x])\n",
        "        df['hcc_code'] = df['all_k_aux_labels'].apply(lambda x: [i.split('||')[2] for i in x])\n",
        "\n",
        "    df = df.drop(['all_k_aux_labels'], axis=1)\n",
        "    \n",
        "    return df"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "57312793-ba38-4bbd-b042-42a229f61221",
      "metadata": {
        "id": "57312793-ba38-4bbd-b042-42a229f61221"
      },
      "outputs": [],
      "source": [
        "text = \"AmeriCann Inc\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "zpEhjGo1UiuA",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 177
        },
        "id": "zpEhjGo1UiuA",
        "outputId": "52c653ef-153e-405c-d999-995d1f427bf4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 2 µs, sys: 1e+03 ns, total: 3 µs\n",
            "Wall time: 5.01 µs\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-3f86a373-29a9-447d-a635-e14700f8ad2f\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>chunks</th>\n",
              "      <th>begin</th>\n",
              "      <th>end</th>\n",
              "      <th>code</th>\n",
              "      <th>all_codes</th>\n",
              "      <th>resolutions</th>\n",
              "      <th>all_distances</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>AmeriCann Inc</td>\n",
              "      <td>0</td>\n",
              "      <td>12</td>\n",
              "      <td>AmeriCann, Inc.</td>\n",
              "      <td>[AmeriCann, Inc., LUMIOX, INC., AGILYSYS INC, Ameresco, Inc., IMMUCOR INC, AAON INC, CRYOLIFE INC]</td>\n",
              "      <td>[AmeriCann, Inc., LUMIOX, INC., AGILYSYS INC, Ameresco, Inc., IMMUCOR INC, AAON INC, CRYOLIFE INC]</td>\n",
              "      <td>[0.0000, 0.1080, 0.1110, 0.1133, 0.1145, 0.1165, 0.1170]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3f86a373-29a9-447d-a635-e14700f8ad2f')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-3f86a373-29a9-447d-a635-e14700f8ad2f button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-3f86a373-29a9-447d-a635-e14700f8ad2f');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "          chunks  begin  end             code  \\\n",
              "0  AmeriCann Inc  0      12   AmeriCann, Inc.   \n",
              "\n",
              "                                                                                            all_codes  \\\n",
              "0  [AmeriCann, Inc., LUMIOX, INC., AGILYSYS INC, Ameresco, Inc., IMMUCOR INC, AAON INC, CRYOLIFE INC]   \n",
              "\n",
              "                                                                                          resolutions  \\\n",
              "0  [AmeriCann, Inc., LUMIOX, INC., AGILYSYS INC, Ameresco, Inc., IMMUCOR INC, AAON INC, CRYOLIFE INC]   \n",
              "\n",
              "                                              all_distances  \n",
              "0  [0.0000, 0.1080, 0.1110, 0.1133, 0.1145, 0.1165, 0.1170]  "
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%time \n",
        "get_codes (light_model, text, vocab = 'normalized_name')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1dc7ce8e-30e4-433f-8ae5-a8645bd21d1b",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 177
        },
        "id": "1dc7ce8e-30e4-433f-8ae5-a8645bd21d1b",
        "outputId": "1d3eff93-6cc7-402e-bc07-5daba68e12b7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 9.2 ms, sys: 277 µs, total: 9.48 ms\n",
            "Wall time: 61.2 ms\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-a7107508-948c-46a8-a18a-2c8cd4a81bab\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>chunks</th>\n",
              "      <th>begin</th>\n",
              "      <th>end</th>\n",
              "      <th>code</th>\n",
              "      <th>all_codes</th>\n",
              "      <th>resolutions</th>\n",
              "      <th>all_distances</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>AmeriCann inc</td>\n",
              "      <td>0</td>\n",
              "      <td>12</td>\n",
              "      <td>AmeriCann, Inc.</td>\n",
              "      <td>[AmeriCann, Inc., LUMIOX, INC., AGILYSYS INC, Ameresco, Inc., IMMUCOR INC, AAON INC, CRYOLIFE INC]</td>\n",
              "      <td>[AmeriCann, Inc., LUMIOX, INC., AGILYSYS INC, Ameresco, Inc., IMMUCOR INC, AAON INC, CRYOLIFE INC]</td>\n",
              "      <td>[0.0000, 0.1080, 0.1110, 0.1133, 0.1145, 0.1165, 0.1170]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a7107508-948c-46a8-a18a-2c8cd4a81bab')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-a7107508-948c-46a8-a18a-2c8cd4a81bab button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-a7107508-948c-46a8-a18a-2c8cd4a81bab');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "          chunks  begin  end             code  \\\n",
              "0  AmeriCann inc  0      12   AmeriCann, Inc.   \n",
              "\n",
              "                                                                                            all_codes  \\\n",
              "0  [AmeriCann, Inc., LUMIOX, INC., AGILYSYS INC, Ameresco, Inc., IMMUCOR INC, AAON INC, CRYOLIFE INC]   \n",
              "\n",
              "                                                                                          resolutions  \\\n",
              "0  [AmeriCann, Inc., LUMIOX, INC., AGILYSYS INC, Ameresco, Inc., IMMUCOR INC, AAON INC, CRYOLIFE INC]   \n",
              "\n",
              "                                              all_distances  \n",
              "0  [0.0000, 0.1080, 0.1110, 0.1133, 0.1145, 0.1165, 0.1170]  "
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "text = 'AmeriCann inc'\n",
        "\n",
        "%time get_codes (light_model, text, vocab='normalized_name')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ebd35257-7249-4bb4-b3ed-715fe65ce816",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 177
        },
        "id": "ebd35257-7249-4bb4-b3ed-715fe65ce816",
        "outputId": "84f5c441-a01a-4889-8400-43ceadc99381",
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 4.66 ms, sys: 2.18 ms, total: 6.84 ms\n",
            "Wall time: 52.7 ms\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-294ba3c6-2d1f-4af4-aeb2-516ba2bd4ba6\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>chunks</th>\n",
              "      <th>begin</th>\n",
              "      <th>end</th>\n",
              "      <th>code</th>\n",
              "      <th>all_codes</th>\n",
              "      <th>resolutions</th>\n",
              "      <th>all_distances</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>StepOne Personal Health inc</td>\n",
              "      <td>0</td>\n",
              "      <td>26</td>\n",
              "      <td>StepOne Personal Health, Inc.</td>\n",
              "      <td>[StepOne Personal Health, Inc., Kura Oncology, Inc., Axsome Therapeutics, Inc., CVS HEALTH Corp, EDGEWELL PERSONAL CARE Co, Cardiovascular Systems Inc, CESCA THERAPEUTICS INC.]</td>\n",
              "      <td>[StepOne Personal Health, Inc., Kura Oncology, Inc., Axsome Therapeutics, Inc., CVS HEALTH Corp, EDGEWELL PERSONAL CARE Co, Cardiovascular Systems Inc, CESCA THERAPEUTICS INC.]</td>\n",
              "      <td>[0.0000, 0.2224, 0.2714, 0.2729, 0.2802, 0.2868, 0.2874]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-294ba3c6-2d1f-4af4-aeb2-516ba2bd4ba6')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-294ba3c6-2d1f-4af4-aeb2-516ba2bd4ba6 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-294ba3c6-2d1f-4af4-aeb2-516ba2bd4ba6');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "                        chunks  begin  end                           code  \\\n",
              "0  StepOne Personal Health inc  0      26   StepOne Personal Health, Inc.   \n",
              "\n",
              "                                                                                                                                                                          all_codes  \\\n",
              "0  [StepOne Personal Health, Inc., Kura Oncology, Inc., Axsome Therapeutics, Inc., CVS HEALTH Corp, EDGEWELL PERSONAL CARE Co, Cardiovascular Systems Inc, CESCA THERAPEUTICS INC.]   \n",
              "\n",
              "                                                                                                                                                                        resolutions  \\\n",
              "0  [StepOne Personal Health, Inc., Kura Oncology, Inc., Axsome Therapeutics, Inc., CVS HEALTH Corp, EDGEWELL PERSONAL CARE Co, Cardiovascular Systems Inc, CESCA THERAPEUTICS INC.]   \n",
              "\n",
              "                                              all_distances  \n",
              "0  [0.0000, 0.2224, 0.2714, 0.2729, 0.2802, 0.2868, 0.2874]  "
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "text = 'StepOne Personal Health inc'\n",
        "\n",
        "%time get_codes (light_model, text, vocab='normalized_name')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "60765bb9-9b85-412c-bf72-214a4714662f",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 194
        },
        "id": "60765bb9-9b85-412c-bf72-214a4714662f",
        "outputId": "485b8be5-9a0e-4f61-8a38-7ce0459543a1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 7.07 ms, sys: 732 µs, total: 7.81 ms\n",
            "Wall time: 67 ms\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-1e5caf9c-99c2-4c37-a22c-a1a1888db250\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>chunks</th>\n",
              "      <th>begin</th>\n",
              "      <th>end</th>\n",
              "      <th>code</th>\n",
              "      <th>all_codes</th>\n",
              "      <th>resolutions</th>\n",
              "      <th>all_distances</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Alzamend Neuro INC</td>\n",
              "      <td>0</td>\n",
              "      <td>17</td>\n",
              "      <td>Alzamend Neuro, Inc.</td>\n",
              "      <td>[Alzamend Neuro, Inc., Kura Oncology, Inc., REGENERON PHARMACEUTICALS INC, Dipexium Pharmaceuticals, Inc., AEOLUS PHARMACEUTICALS, INC., Flex Pharma, Inc., PROTO SCRIPT PHARMACEUTICAL CORP]</td>\n",
              "      <td>[Alzamend Neuro, Inc., Kura Oncology, Inc., REGENERON PHARMACEUTICALS INC, Dipexium Pharmaceuticals, Inc., AEOLUS PHARMACEUTICALS, INC., Flex Pharma, Inc., PROTO SCRIPT PHARMACEUTICAL CORP]</td>\n",
              "      <td>[0.0000, 0.1704, 0.1802, 0.1934, 0.2149, 0.2162, 0.2254]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1e5caf9c-99c2-4c37-a22c-a1a1888db250')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-1e5caf9c-99c2-4c37-a22c-a1a1888db250 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-1e5caf9c-99c2-4c37-a22c-a1a1888db250');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "               chunks  begin  end                  code  \\\n",
              "0  Alzamend Neuro INC  0      17   Alzamend Neuro, Inc.   \n",
              "\n",
              "                                                                                                                                                                                       all_codes  \\\n",
              "0  [Alzamend Neuro, Inc., Kura Oncology, Inc., REGENERON PHARMACEUTICALS INC, Dipexium Pharmaceuticals, Inc., AEOLUS PHARMACEUTICALS, INC., Flex Pharma, Inc., PROTO SCRIPT PHARMACEUTICAL CORP]   \n",
              "\n",
              "                                                                                                                                                                                     resolutions  \\\n",
              "0  [Alzamend Neuro, Inc., Kura Oncology, Inc., REGENERON PHARMACEUTICALS INC, Dipexium Pharmaceuticals, Inc., AEOLUS PHARMACEUTICALS, INC., Flex Pharma, Inc., PROTO SCRIPT PHARMACEUTICAL CORP]   \n",
              "\n",
              "                                              all_distances  \n",
              "0  [0.0000, 0.1704, 0.1802, 0.1934, 0.2149, 0.2162, 0.2254]  "
            ]
          },
          "execution_count": 23,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "text = 'Alzamend Neuro INC'\n",
        "\n",
        "%time get_codes (light_model, text, vocab='normalized_name')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3eda13a5-4b4e-448e-bf74-f09c14789920",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 194
        },
        "id": "3eda13a5-4b4e-448e-bf74-f09c14789920",
        "outputId": "94af733c-7601-432b-bcb7-04996b970204"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 7.98 ms, sys: 2.37 ms, total: 10.3 ms\n",
            "Wall time: 56.6 ms\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-1b69e93f-a3f5-4af1-8472-2b715b8c123f\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>chunks</th>\n",
              "      <th>begin</th>\n",
              "      <th>end</th>\n",
              "      <th>code</th>\n",
              "      <th>all_codes</th>\n",
              "      <th>resolutions</th>\n",
              "      <th>all_distances</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>MMEX Resources Corporation</td>\n",
              "      <td>0</td>\n",
              "      <td>25</td>\n",
              "      <td>MMEX Resources Corp</td>\n",
              "      <td>[MMEX Resources Corp, ANTERO RESOURCES Corp, ARTESIAN RESOURCES CORP, ESTERLINE TECHNOLOGIES CORP, Timberline Resources Corp, CATALYST PAPER CORP, INFRASTRUCTURE DEVELOPMENTS CORP.]</td>\n",
              "      <td>[MMEX Resources Corp, ANTERO RESOURCES Corp, ARTESIAN RESOURCES CORP, ESTERLINE TECHNOLOGIES CORP, Timberline Resources Corp, CATALYST PAPER CORP, INFRASTRUCTURE DEVELOPMENTS CORP.]</td>\n",
              "      <td>[0.1096, 0.1540, 0.1624, 0.2054, 0.2202, 0.2406, 0.2451]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1b69e93f-a3f5-4af1-8472-2b715b8c123f')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-1b69e93f-a3f5-4af1-8472-2b715b8c123f button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-1b69e93f-a3f5-4af1-8472-2b715b8c123f');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "                       chunks  begin  end                 code  \\\n",
              "0  MMEX Resources Corporation  0      25   MMEX Resources Corp   \n",
              "\n",
              "                                                                                                                                                                               all_codes  \\\n",
              "0  [MMEX Resources Corp, ANTERO RESOURCES Corp, ARTESIAN RESOURCES CORP, ESTERLINE TECHNOLOGIES CORP, Timberline Resources Corp, CATALYST PAPER CORP, INFRASTRUCTURE DEVELOPMENTS CORP.]   \n",
              "\n",
              "                                                                                                                                                                             resolutions  \\\n",
              "0  [MMEX Resources Corp, ANTERO RESOURCES Corp, ARTESIAN RESOURCES CORP, ESTERLINE TECHNOLOGIES CORP, Timberline Resources Corp, CATALYST PAPER CORP, INFRASTRUCTURE DEVELOPMENTS CORP.]   \n",
              "\n",
              "                                              all_distances  \n",
              "0  [0.1096, 0.1540, 0.1624, 0.2054, 0.2202, 0.2406, 0.2451]  "
            ]
          },
          "execution_count": 24,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "text = 'MMEX Resources Corporation'\n",
        "\n",
        "%time get_codes (light_model, text, vocab='normalized_name')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "40651b68-d4a3-4ade-ae5a-60994c4314d1",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 194
        },
        "id": "40651b68-d4a3-4ade-ae5a-60994c4314d1",
        "outputId": "53c0850c-a470-43c3-d3b2-c791f1708413"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 7.89 ms, sys: 941 µs, total: 8.83 ms\n",
            "Wall time: 43.4 ms\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-9f6c291c-c8c3-4071-b619-76cdbfbdff98\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>chunks</th>\n",
              "      <th>begin</th>\n",
              "      <th>end</th>\n",
              "      <th>code</th>\n",
              "      <th>all_codes</th>\n",
              "      <th>resolutions</th>\n",
              "      <th>all_distances</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Alphadyne Asset Management Lp.</td>\n",
              "      <td>0</td>\n",
              "      <td>29</td>\n",
              "      <td>Alphadyne Asset Management LP</td>\n",
              "      <td>[Alphadyne Asset Management LP, YACKTMAN ASSET MANAGEMENT LP, TOCQUEVILLE ASSET MANAGEMENT L.P., SYSTEMATIC FINANCIAL MANAGEMENT LP, Madyson Equity Group, LP, AMERIGAS PARTNERS LP, CAPRIN ASSET MANAGEMENT LLC /ADV, ALLIANCEBERNSTEIN HOLDING L.P.]</td>\n",
              "      <td>[Alphadyne Asset Management LP, YACKTMAN ASSET MANAGEMENT LP, TOCQUEVILLE ASSET MANAGEMENT L.P., SYSTEMATIC FINANCIAL MANAGEMENT LP, Madyson Equity Group, LP, AMERIGAS PARTNERS LP, CAPRIN ASSET MANAGEMENT LLC /ADV, ALLIANCEBERNSTEIN HOLDING L.P.]</td>\n",
              "      <td>[0.0000, 0.0724, 0.1040, 0.2378, 0.2470, 0.2570, 0.2614, 0.2722]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9f6c291c-c8c3-4071-b619-76cdbfbdff98')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-9f6c291c-c8c3-4071-b619-76cdbfbdff98 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-9f6c291c-c8c3-4071-b619-76cdbfbdff98');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "                           chunks  begin  end                           code  \\\n",
              "0  Alphadyne Asset Management Lp.  0      29   Alphadyne Asset Management LP   \n",
              "\n",
              "                                                                                                                                                                                                                                                all_codes  \\\n",
              "0  [Alphadyne Asset Management LP, YACKTMAN ASSET MANAGEMENT LP, TOCQUEVILLE ASSET MANAGEMENT L.P., SYSTEMATIC FINANCIAL MANAGEMENT LP, Madyson Equity Group, LP, AMERIGAS PARTNERS LP, CAPRIN ASSET MANAGEMENT LLC /ADV, ALLIANCEBERNSTEIN HOLDING L.P.]   \n",
              "\n",
              "                                                                                                                                                                                                                                              resolutions  \\\n",
              "0  [Alphadyne Asset Management LP, YACKTMAN ASSET MANAGEMENT LP, TOCQUEVILLE ASSET MANAGEMENT L.P., SYSTEMATIC FINANCIAL MANAGEMENT LP, Madyson Equity Group, LP, AMERIGAS PARTNERS LP, CAPRIN ASSET MANAGEMENT LLC /ADV, ALLIANCEBERNSTEIN HOLDING L.P.]   \n",
              "\n",
              "                                                      all_distances  \n",
              "0  [0.0000, 0.0724, 0.1040, 0.2378, 0.2470, 0.2570, 0.2614, 0.2722]  "
            ]
          },
          "execution_count": 25,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "text = 'Alphadyne Asset Management Lp.'\n",
        "\n",
        "%time get_codes (light_model, text, vocab='normalized_name')"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}