{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "04afee44-529b-4c72-b7a7-f72f5cefa5ce",
      "metadata": {
        "id": "04afee44-529b-4c72-b7a7-f72f5cefa5ce"
      },
      "source": [
        "![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "25aee2df",
      "metadata": {
        "id": "25aee2df"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/finance-nlp/09.2.Entity_Resolution_NASDAQ.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c960bb95",
      "metadata": {
        "id": "c960bb95"
      },
      "source": [
        "#🔎 Financial Entity Resolution\n",
        "\n",
        "**In this notebook, we continue from where left off in `7.Entity_Resolution` notebook.**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4iIO6G_B3pqq",
      "metadata": {
        "collapsed": false,
        "id": "4iIO6G_B3pqq"
      },
      "source": [
        "#🎬 Installation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "hPwo4Czy3pqq",
      "metadata": {
        "id": "hPwo4Czy3pqq",
        "pycharm": {
          "is_executing": true
        }
      },
      "outputs": [],
      "source": [
        "! pip install -q johnsnowlabs"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "YPsbAnNoPt0Z",
      "metadata": {
        "id": "YPsbAnNoPt0Z"
      },
      "source": [
        "##🔗 Automatic Installation\n",
        "Using my.johnsnowlabs.com SSO"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "_L-7mLYp3pqr",
      "metadata": {
        "id": "_L-7mLYp3pqr",
        "pycharm": {
          "is_executing": true
        }
      },
      "outputs": [],
      "source": [
        "from johnsnowlabs import nlp, finance\n",
        "\n",
        "# nlp.install(force_browser=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "hsJvn_WWM2GL",
      "metadata": {
        "id": "hsJvn_WWM2GL"
      },
      "source": [
        "##🔗 Manual downloading\n",
        "If you are not registered in my.johnsnowlabs.com, you received a license via e-email or you are using Safari, you may need to do a manual update of the license.\n",
        "\n",
        "- Go to my.johnsnowlabs.com\n",
        "- Download your license\n",
        "- Upload it using the following command"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "i57QV3-_P2sQ",
      "metadata": {
        "id": "i57QV3-_P2sQ"
      },
      "outputs": [],
      "source": [
        "from google.colab import files\n",
        "print('Please Upload your John Snow Labs License using the button below')\n",
        "license_keys = files.upload()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xGgNdFzZP_hQ",
      "metadata": {
        "id": "xGgNdFzZP_hQ"
      },
      "source": [
        "- Install it"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "OfmmPqknP4rR",
      "metadata": {
        "id": "OfmmPqknP4rR"
      },
      "outputs": [],
      "source": [
        "nlp.install()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "DCl5ErZkNNLk",
      "metadata": {
        "id": "DCl5ErZkNNLk"
      },
      "source": [
        "#📌 Starting"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "x3jVICoa3pqr",
      "metadata": {
        "id": "x3jVICoa3pqr"
      },
      "outputs": [],
      "source": [
        "spark = nlp.start()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "56490c04-7bf9-47d4-8e09-8a87a4848d38",
      "metadata": {
        "id": "56490c04-7bf9-47d4-8e09-8a87a4848d38"
      },
      "source": [
        "#🔎 Sentence Entity Resolver Models\n",
        "\n",
        "📜Entity resolution is an important task in natural language processing and information extraction, as it allows for more accurate analysis and understanding of financial texts. For example, in a news article discussing the performance of a company's stock, accurately identifying and disambiguating the company's name is crucial for accurately tracking the stock's performance.\n",
        "\n",
        "📜An NLP use case in financial or legal applications is identifying financial entities' presence in a given text. One of those entities could be `Company Name`. We can carry out NER to extract different chunks of information, but in real financial and legal use cases, the company name is usually not useful as it is mentioned in the text. Sometimes we need the _official_ name of the company (instead of `Amazon`, `Amazon.com INC`, as registered in Edgar, Crunchbase and Nasdaq). We have pre-trained sentence entity resolver models for these purposes shown below with the examples."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a0xdKvjgHt8X",
      "metadata": {
        "id": "a0xdKvjgHt8X"
      },
      "source": [
        "#🔎 Pretrained Entity Resolution Models for Finance\n",
        "\n",
        "Here are the list of pretrained Entity Resolution models:\n",
        "\n",
        "\n",
        "|index|model|\n",
        "|-----:|:-----|\n",
        "| 1| [Company Name Normalization Using Nasdaq](https://nlp.johnsnowlabs.com/2022/10/22/finel_nasdaq_data_company_name_en.html)  |\n",
        "| 2| [Company Name Normalization Using Edgar Database](https://nlp.johnsnowlabs.com/2022/08/30/finel_edgar_company_name_en.html)  |\n",
        "| 3| [Company Names Normalization Using Crunchbase](https://nlp.johnsnowlabs.com/2022/09/28/finre_work_experience_en.html)  | \n",
        "| 4| [Company Name to Ticker Using Nasdaq](https://nlp.johnsnowlabs.com/2022/10/22/finel_nasdaq_data_ticker_en.html)  | \n",
        "| 5| [Company Name to IRS Number Using Edgar Database](https://nlp.johnsnowlabs.com/2022/08/30/finel_edgar_irs_en.html)  |\n",
        "| 6| [Resolve Tickers to Company Names Using Nasdaq](https://nlp.johnsnowlabs.com/2022/09/09/finel_tickers2names_en.html)  |\n",
        "| 7| [Resolve Company Names to Tickers Using Nasdaq](https://nlp.johnsnowlabs.com/2022/09/08/finel_names2tickers_en.html)  | "
      ]
    },
    {
      "cell_type": "markdown",
      "id": "XVuxdiKgi_qd",
      "metadata": {
        "id": "XVuxdiKgi_qd"
      },
      "source": [
        "##✅ Common Componennts"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f900954c-e89b-4cb7-be24-6ea499db710a",
      "metadata": {
        "id": "f900954c-e89b-4cb7-be24-6ea499db710a"
      },
      "source": [
        "📚Other than providing the code in the \"result\" field it provides more metadata about the matching process:\n",
        "\n",
        "- target_text -> Text to resolve\n",
        "- resolved_text -> Best match text\n",
        "- confidence -> Relative confidence for the top match (distance to probability)\n",
        "- confidence_ratio -> Relative confidence for the top match. TopMatchConfidence / SecondMatchConfidence\n",
        "- alternative_codes -> List of other plausible codes (in the KNN neighborhood)\n",
        "- all_k_resolutions -> All codes descriptions\n",
        "- all_k_results -> All resolved codes for metrics calculation purposes\n",
        "- sentence -> SentenceId\n",
        "\n",
        "We will use following Generic Function For Getting the Codes and Relation Pairs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7bb59495-6397-4aed-99fa-691678a6ff6e",
      "metadata": {
        "id": "7bb59495-6397-4aed-99fa-691678a6ff6e"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "pd.set_option('display.max_colwidth', 0)\n",
        "\n",
        "def get_codes (lp, text, vocab='company_name', hcc=False):\n",
        "\n",
        "    \"\"\"Returns LightPipeline resolution results\"\"\"\n",
        "    \n",
        "    full_light_result = lp.fullAnnotate(text)\n",
        "\n",
        "    chunks = []\n",
        "    codes = []\n",
        "    begin = []\n",
        "    end = []\n",
        "    resolutions=[]\n",
        "    all_distances =[]\n",
        "    all_codes=[]\n",
        "    all_cosines = []\n",
        "    all_k_aux_labels=[]\n",
        "\n",
        "    for i in range(len(full_light_result)):\n",
        "\n",
        "      for chunk, code in zip(full_light_result[i]['ner_chunk'], full_light_result[i][vocab]):   \n",
        "          begin.append(chunk.begin)\n",
        "          end.append(chunk.end)\n",
        "          chunks.append(chunk.result)\n",
        "          codes.append(code.result) \n",
        "          all_codes.append(code.metadata['all_k_results'].split(':::'))\n",
        "          resolutions.append(code.metadata['all_k_resolutions'].split(':::'))\n",
        "          all_distances.append(code.metadata['all_k_distances'].split(':::'))\n",
        "          all_cosines.append(code.metadata['all_k_cosine_distances'].split(':::'))\n",
        "          if hcc:\n",
        "              try:\n",
        "                  all_k_aux_labels.append(code.metadata['all_k_aux_labels'].split(':::'))\n",
        "              except:\n",
        "                  all_k_aux_labels.append([])\n",
        "          else:\n",
        "              all_k_aux_labels.append([])\n",
        "\n",
        "    df = pd.DataFrame({'chunks':chunks, 'begin': begin, 'end':end, 'code':codes, 'all_codes':all_codes, \n",
        "                       'resolutions':resolutions, 'all_k_aux_labels':all_k_aux_labels,'all_distances':all_cosines})\n",
        "    \n",
        "    return df"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c6ad75d0-7bf9-41b1-a37f-07af09ee6c19",
      "metadata": {
        "id": "c6ad75d0-7bf9-41b1-a37f-07af09ee6c19"
      },
      "source": [
        "##✅ Company Name Normalization using NASDAQ\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "64e7cedd-1978-4130-a88a-f074300d9116",
      "metadata": {
        "id": "64e7cedd-1978-4130-a88a-f074300d9116"
      },
      "source": [
        "🕮 [Nasdaq Homepage](https://www.nasdaq.com/)\n",
        "\n",
        "`The Nasdaq Stock Market` (National Association of Securities Dealers Automated Quotations Stock Market) is an American stock exchange based in New York City. It is ranked second on the list of stock exchanges by market capitalization of shares traded, behind the New York Stock Exchange\n",
        "\n",
        "Let's suppose we get this text from scrapping the Internet, or from Twitter. Firstly, we get company name from sample text with `finner_orgs_prods_alias` model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Mx-JUxkioFr2",
      "metadata": {
        "id": "Mx-JUxkioFr2"
      },
      "outputs": [],
      "source": [
        "text = \"Aspect Development provides component and supplier management (CSM) technology to improve the product-development cycle of manufacturing.\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "SIwC5vuDoSoN",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SIwC5vuDoSoN",
        "outputId": "f1573128-792e-4dc1-d8cb-b4e139a69860"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "bert_embeddings_sec_bert_base download started this may take some time.\n",
            "Approximate size to download 390.4 MB\n",
            "[OK!]\n",
            "finner_orgs_prods_alias download started this may take some time.\n",
            "[OK!]\n"
          ]
        }
      ],
      "source": [
        "document_assembler = nlp.DocumentAssembler()\\\n",
        "    .setInputCol(\"text\")\\\n",
        "    .setOutputCol(\"document\")\n",
        "\n",
        "text_splitter = finance.TextSplitter()\\\n",
        "    .setInputCols([\"document\"])\\\n",
        "    .setOutputCol(\"sentence\")\n",
        "\n",
        "tokenizer = nlp.Tokenizer()\\\n",
        "    .setInputCols([\"sentence\"])\\\n",
        "    .setOutputCol(\"token\")\n",
        "\n",
        "embeddings = nlp.BertEmbeddings.pretrained(\"bert_embeddings_sec_bert_base\",\"en\") \\\n",
        "    .setInputCols([\"sentence\", \"token\"])\\\n",
        "    .setOutputCol(\"embeddings\")\n",
        "\n",
        "ner_model = finance.NerModel.pretrained(\"finner_orgs_prods_alias\", \"en\", \"finance/models\")\\\n",
        "    .setInputCols([\"sentence\", \"token\", \"embeddings\"])\\\n",
        "    .setOutputCol(\"ner\")\n",
        "    \n",
        "ner_converter = nlp.NerConverter()\\\n",
        "    .setInputCols([\"sentence\",\"token\",\"ner\"])\\\n",
        "    .setOutputCol(\"ner_chunk\")\n",
        "\n",
        "nlpPipeline = nlp.Pipeline(stages=[\n",
        "        document_assembler,\n",
        "        text_splitter,\n",
        "        tokenizer,\n",
        "        embeddings,\n",
        "        ner_model,\n",
        "        ner_converter,\n",
        "])\n",
        "\n",
        "empty_data = spark.createDataFrame([[\"\"]]).toDF(\"text\")\n",
        "\n",
        "model = nlpPipeline.fit(empty_data)\n",
        "\n",
        "lp_ner = nlp.LightPipeline(model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6BoFoVOZo4qX",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6BoFoVOZo4qX",
        "outputId": "cb105d6e-0737-4786-d577-c5364ecc9c51"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'document': ['Aspect Development provides component and supplier management (CSM) technology to improve the product-development cycle of manufacturing.'],\n",
              " 'ner_chunk': ['Aspect Development'],\n",
              " 'token': ['Aspect',\n",
              "  'Development',\n",
              "  'provides',\n",
              "  'component',\n",
              "  'and',\n",
              "  'supplier',\n",
              "  'management',\n",
              "  '(',\n",
              "  'CSM',\n",
              "  ')',\n",
              "  'technology',\n",
              "  'to',\n",
              "  'improve',\n",
              "  'the',\n",
              "  'product-development',\n",
              "  'cycle',\n",
              "  'of',\n",
              "  'manufacturing',\n",
              "  '.'],\n",
              " 'ner': ['B-ORG',\n",
              "  'I-ORG',\n",
              "  'O',\n",
              "  'O',\n",
              "  'O',\n",
              "  'O',\n",
              "  'O',\n",
              "  'O',\n",
              "  'O',\n",
              "  'O',\n",
              "  'O',\n",
              "  'O',\n",
              "  'O',\n",
              "  'O',\n",
              "  'O',\n",
              "  'O',\n",
              "  'O',\n",
              "  'O',\n",
              "  'O'],\n",
              " 'embeddings': ['Aspect',\n",
              "  'Development',\n",
              "  'provides',\n",
              "  'component',\n",
              "  'and',\n",
              "  'supplier',\n",
              "  'management',\n",
              "  '(',\n",
              "  'CSM',\n",
              "  ')',\n",
              "  'technology',\n",
              "  'to',\n",
              "  'improve',\n",
              "  'the',\n",
              "  'product-development',\n",
              "  'cycle',\n",
              "  'of',\n",
              "  'manufacturing',\n",
              "  '.'],\n",
              " 'sentence': ['Aspect Development provides component and supplier management (CSM) technology to improve the product-development cycle of manufacturing.']}"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "ner_result = lp_ner.annotate(text)\n",
        "\n",
        "ner_result\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ampXSTnCpETr",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ampXSTnCpETr",
        "outputId": "2965d967-0fd1-4e38-d06f-0a5819cb85a1"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['Aspect Development']"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "ner_result[\"ner_chunk\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xp9BzLUwR188",
      "metadata": {
        "id": "xp9BzLUwR188"
      },
      "source": [
        "##🚀 Alright! We extract company name from sample text. Now, we normalize `Aspect Development` company using `finel_nasdaq_data_company_name` model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "14344291-1c89-41d8-abaa-6a7908e2685a",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "14344291-1c89-41d8-abaa-6a7908e2685a",
        "outputId": "dbd416ca-daf4-40a4-c0fa-d25af78eb21b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tfhub_use_lg download started this may take some time.\n",
            "Approximate size to download 753.3 MB\n",
            "[OK!]\n",
            "finel_nasdaq_data_company_name download started this may take some time.\n",
            "[OK!]\n"
          ]
        }
      ],
      "source": [
        "document_assembler = nlp.DocumentAssembler()\\\n",
        "    .setInputCol(\"text\")\\\n",
        "    .setOutputCol(\"ner_chunk\")\n",
        "\n",
        "embeddings = nlp.UniversalSentenceEncoder.pretrained(\"tfhub_use_lg\", \"en\") \\\n",
        "      .setInputCols(\"ner_chunk\") \\\n",
        "      .setOutputCol(\"sentence_embeddings\")\n",
        "    \n",
        "resolver = finance.SentenceEntityResolverModel.pretrained(\"finel_nasdaq_data_company_name\", \"en\", \"finance/models\")\\\n",
        "      .setInputCols([\"sentence_embeddings\"]) \\\n",
        "      .setOutputCol(\"name\")\\\n",
        "      .setDistanceFunction(\"EUCLIDIAN\")\n",
        "\n",
        "pipeline = nlp.Pipeline(\n",
        "      stages = [\n",
        "          document_assembler,\n",
        "          embeddings,\n",
        "          resolver])\n",
        "\n",
        "empty_data = spark.createDataFrame([[\"\"]]).toDF(\"text\")\n",
        "\n",
        "model = pipeline.fit(empty_data)\n",
        "\n",
        "lp = nlp.LightPipeline(model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "G-DVxBEizyXG",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "G-DVxBEizyXG",
        "outputId": "0f5fc189-6920-4815-9210-3b4063cf6aca"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['Aspect Development']"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "ner_result[\"ner_chunk\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b8524cb0-c0f6-4fcc-baa6-a5b26f0a3494",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 456
        },
        "id": "b8524cb0-c0f6-4fcc-baa6-a5b26f0a3494",
        "outputId": "eb67cdb4-a443-4ef6-e28c-75ba16e7c83a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CPU times: user 4 µs, sys: 2 µs, total: 6 µs\n",
            "Wall time: 12.4 µs\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-c65a6477-ee5f-45df-a37b-7a91ed9546a5\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>chunks</th>\n",
              "      <th>begin</th>\n",
              "      <th>end</th>\n",
              "      <th>code</th>\n",
              "      <th>all_codes</th>\n",
              "      <th>resolutions</th>\n",
              "      <th>all_k_aux_labels</th>\n",
              "      <th>all_distances</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Aspect Development</td>\n",
              "      <td>0</td>\n",
              "      <td>17</td>\n",
              "      <td>ASPECT DEVELOPMENT INC</td>\n",
              "      <td>[ASPECT DEVELOPMENT INC, BINDVIEW DEVELOPMENT CORP, COHESION TECHNOLOGIES INC, ENVISION DEVELOPMENT CORP, ASPECT COMMUNICATIONS CORP, CONCEPTS DIRECT INC, CATELLUS DEVELOPMENT CORP, APPLIED GRAPHICS TECHNOLOGIES INC, MOVING IMAGE TECHNOLOGIES INC, APPROACH RESOURCES INC, ENTORIAN TECHNOLOGIES INC, BIOAFFINITY TECHNOLOGIES INC, ANALYSIS &amp; TECHNOLOGY INC, BOTTOMLINE TECHNOLOGIES INC, IMAGE GUIDED TECHNOLOGIES INC, CREATIVE TECHNOLOGY LTD, EZGO TECHNOLOGIES LTD, EXCO RESOURCES INC, COMVERSE TECHNOLOGY INC, ADVANCED ANALOGIC TECHNOLOGIES INC]</td>\n",
              "      <td>[ASPECT DEVELOPMENT INC, BINDVIEW DEVELOPMENT CORP, COHESION TECHNOLOGIES INC, ENVISION DEVELOPMENT CORP, ASPECT COMMUNICATIONS CORP, CONCEPTS DIRECT INC, CATELLUS DEVELOPMENT CORP, APPLIED GRAPHICS TECHNOLOGIES INC, MOVING IMAGE TECHNOLOGIES INC, APPROACH RESOURCES INC, ENTORIAN TECHNOLOGIES INC, BIOAFFINITY TECHNOLOGIES INC, ANALYSIS &amp; TECHNOLOGY INC, BOTTOMLINE TECHNOLOGIES INC, IMAGE GUIDED TECHNOLOGIES INC, CREATIVE TECHNOLOGY LTD, EZGO TECHNOLOGIES LTD, EXCO RESOURCES INC, COMVERSE TECHNOLOGY INC, ADVANCED ANALOGIC TECHNOLOGIES INC]</td>\n",
              "      <td>[]</td>\n",
              "      <td>[0.0000, 0.1882, 0.1944, 0.1957, 0.2052, 0.2220, 0.2248, 0.2397, 0.2410, 0.2488, 0.2500, 0.2539, 0.2544, 0.2546, 0.2546, 0.2559, 0.2560, 0.2605, 0.2608, 0.2620]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c65a6477-ee5f-45df-a37b-7a91ed9546a5')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-c65a6477-ee5f-45df-a37b-7a91ed9546a5 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-c65a6477-ee5f-45df-a37b-7a91ed9546a5');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "               chunks  begin  end                    code  \\\n",
              "0  Aspect Development  0      17   ASPECT DEVELOPMENT INC   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          all_codes  \\\n",
              "0  [ASPECT DEVELOPMENT INC, BINDVIEW DEVELOPMENT CORP, COHESION TECHNOLOGIES INC, ENVISION DEVELOPMENT CORP, ASPECT COMMUNICATIONS CORP, CONCEPTS DIRECT INC, CATELLUS DEVELOPMENT CORP, APPLIED GRAPHICS TECHNOLOGIES INC, MOVING IMAGE TECHNOLOGIES INC, APPROACH RESOURCES INC, ENTORIAN TECHNOLOGIES INC, BIOAFFINITY TECHNOLOGIES INC, ANALYSIS & TECHNOLOGY INC, BOTTOMLINE TECHNOLOGIES INC, IMAGE GUIDED TECHNOLOGIES INC, CREATIVE TECHNOLOGY LTD, EZGO TECHNOLOGIES LTD, EXCO RESOURCES INC, COMVERSE TECHNOLOGY INC, ADVANCED ANALOGIC TECHNOLOGIES INC]   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        resolutions  \\\n",
              "0  [ASPECT DEVELOPMENT INC, BINDVIEW DEVELOPMENT CORP, COHESION TECHNOLOGIES INC, ENVISION DEVELOPMENT CORP, ASPECT COMMUNICATIONS CORP, CONCEPTS DIRECT INC, CATELLUS DEVELOPMENT CORP, APPLIED GRAPHICS TECHNOLOGIES INC, MOVING IMAGE TECHNOLOGIES INC, APPROACH RESOURCES INC, ENTORIAN TECHNOLOGIES INC, BIOAFFINITY TECHNOLOGIES INC, ANALYSIS & TECHNOLOGY INC, BOTTOMLINE TECHNOLOGIES INC, IMAGE GUIDED TECHNOLOGIES INC, CREATIVE TECHNOLOGY LTD, EZGO TECHNOLOGIES LTD, EXCO RESOURCES INC, COMVERSE TECHNOLOGY INC, ADVANCED ANALOGIC TECHNOLOGIES INC]   \n",
              "\n",
              "  all_k_aux_labels  \\\n",
              "0  []                \n",
              "\n",
              "                                                                                                                                                      all_distances  \n",
              "0  [0.0000, 0.1882, 0.1944, 0.1957, 0.2052, 0.2220, 0.2248, 0.2397, 0.2410, 0.2488, 0.2500, 0.2539, 0.2544, 0.2546, 0.2546, 0.2559, 0.2560, 0.2605, 0.2608, 0.2620]  "
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%time\n",
        "get_codes (lp, ner_result[\"ner_chunk\"], vocab='name')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "VxnctZBaS8Gx",
      "metadata": {
        "id": "VxnctZBaS8Gx"
      },
      "source": [
        "###📌 Normalized Name\n",
        "In NASDAQ, the company official is different!\n",
        "\n",
        "- Incorrect: `Aspect Development`\n",
        "- Correct (Official): `ASPECT DEVELOPMENT INC`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "QoYeYuT8TLIs",
      "metadata": {
        "id": "QoYeYuT8TLIs"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "tf-gpu",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7 (default, Sep 16 2021, 16:59:28) [MSC v.1916 64 bit (AMD64)]"
    },
    "vscode": {
      "interpreter": {
        "hash": "3f47d918ae832c68584484921185f5c85a1760864bf927a683dc6fb56366cc77"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}