{ "cells": [ { "cell_type": "markdown", "id": "04afee44-529b-4c72-b7a7-f72f5cefa5ce", "metadata": { "id": "04afee44-529b-4c72-b7a7-f72f5cefa5ce" }, "source": [ "" ] }, { "attachments": {}, "cell_type": "markdown", "id": "58ad3f2c", "metadata": { "id": "58ad3f2c" }, "source": [ "[](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/legal-nlp/09.1.Entity_Resolution_Edgar_unique_IDs.ipynb)" ] }, { "cell_type": "markdown", "id": "gk3kZHmNj51v", "metadata": { "collapsed": false, "id": "gk3kZHmNj51v" }, "source": [ "# Installation" ] }, { "cell_type": "code", "execution_count": null, "id": "_914itZsj51v", "metadata": { "id": "_914itZsj51v", "pycharm": { "is_executing": true } }, "outputs": [], "source": [ "! pip install -q johnsnowlabs" ] }, { "cell_type": "markdown", "id": "YPsbAnNoPt0Z", "metadata": { "id": "YPsbAnNoPt0Z" }, "source": [ "## Automatic Installation\n", "Using my.johnsnowlabs.com SSO" ] }, { "cell_type": "code", "execution_count": null, "id": "fY0lcShkj51w", "metadata": { "id": "fY0lcShkj51w", "pycharm": { "is_executing": true } }, "outputs": [], "source": [ "from johnsnowlabs import nlp, legal\n", "\n", "# nlp.install(force_browser=True)" ] }, { "cell_type": "markdown", "id": "hsJvn_WWM2GL", "metadata": { "id": "hsJvn_WWM2GL" }, "source": [ "## Manual downloading\n", "If you are not registered in my.johnsnowlabs.com, you received a license via e-email or you are using Safari, you may need to do a manual update of the license.\n", "\n", "- Go to my.johnsnowlabs.com\n", "- Download your license\n", "- Upload it using the following command" ] }, { "cell_type": "code", "execution_count": null, "id": "i57QV3-_P2sQ", "metadata": { "id": "i57QV3-_P2sQ" }, "outputs": [], "source": [ "from google.colab import files\n", "print('Please Upload your John Snow Labs License using the button below')\n", "license_keys = files.upload()" ] }, { "cell_type": "markdown", "id": "xGgNdFzZP_hQ", "metadata": { "id": "xGgNdFzZP_hQ" }, "source": [ "- Install it" ] }, { "cell_type": "code", "execution_count": null, "id": "OfmmPqknP4rR", "metadata": { "id": "OfmmPqknP4rR" }, "outputs": [], "source": [ "nlp.install()" ] }, { "cell_type": "markdown", "id": "DCl5ErZkNNLk", "metadata": { "id": "DCl5ErZkNNLk" }, "source": [ "# Starting" ] }, { "cell_type": "code", "execution_count": null, "id": "wRXTnNl3j51w", "metadata": { "id": "wRXTnNl3j51w" }, "outputs": [], "source": [ "spark = nlp.start()" ] }, { "cell_type": "markdown", "id": "d03798d1", "metadata": { "id": "d03798d1" }, "source": [ "# Legal Entity Resolution: unique IDs" ] }, { "cell_type": "markdown", "id": "ba2c133c-9c4d-4531-a3ae-b12a41d4d3aa", "metadata": { "id": "ba2c133c-9c4d-4531-a3ae-b12a41d4d3aa" }, "source": [ "\n", "Entity resolution is an important task in natural language processing and information extraction, as it allows for more accurate analysis and understanding of legal texts. For example, in a news article discussing the performance of a company's stock, accurately identifying and disambiguating the company's name is crucial for accurately tracking the stock's performance.\n", "\n", "An NLP use case in financial or legal applications is identifying legal entities' presence in a given text. One of those entities could be `Company Name`. We can carry out NER to extract different chunks of information, but in real financial and legal use cases, the company name is usually not useful as it is mentioned in the text. Sometimes we need the **_official_** name of the company (instead of `Amazon`, `Amazon.com INC`, as registered in Edgar, Crunchbase and Nasdaq). We have pre-trained sentence entity resolver models for these purposes shown below with the examples." ] }, { "cell_type": "markdown", "id": "BXkec_3MUBdx", "metadata": { "collapsed": false, "id": "BXkec_3MUBdx" }, "source": [ "## Retrieving unique IDs" ] }, { "cell_type": "markdown", "id": "Hp1JIxA3UBdy", "metadata": { "collapsed": false, "id": "Hp1JIxA3UBdy" }, "source": [ "Besides mapping a series of non-normalized strings to a normalized version of, for example, the company name in some registries, we can also map them to retrieve unique IDs (as IRS id in Edgar database) using Entity Resolution.\n", "\n", "Let's take a look at how we do it." ] }, { "cell_type": "markdown", "id": "ossta3lKkGbP", "metadata": { "id": "ossta3lKkGbP" }, "source": [ "## Pretrained Entity Resolution Models for Legal\n", "\n", "Here are the list of pretrained Entity Resolution models:\n", "\n", "|index|model|\n", "|-----:|:-----|\n", "| 1| [Company Name Normalization Using Edgar Database](https://nlp.johnsnowlabs.com/2022/08/30/legel_edgar_company_name_en.html) |\n", "| 2| [Company Names Normalization Using Crunchbase](https://nlp.johnsnowlabs.com/2022/08/09/legel_crunchbase_companynames_en_3_2.html) | \n", "| 3| [Company Name to IRS (Edgar database)](https://nlp.johnsnowlabs.com/2022/08/30/finel_edgar_company_name_en.html) |\n" ] }, { "cell_type": "markdown", "id": "XVuxdiKgi_qd", "metadata": { "id": "XVuxdiKgi_qd" }, "source": [ "## Common Componennts\n" ] }, { "cell_type": "markdown", "id": "KEsF6wiQltXl", "metadata": { "id": "KEsF6wiQltXl" }, "source": [ "Other than providing the code in the \"result\" field it provides more metadata about the matching process:\n", "\n", "- target_text -> Text to resolve\n", "- resolved_text -> Best match text\n", "- confidence -> Relative confidence for the top match (distance to probability)\n", "- confidence_ratio -> Relative confidence for the top match. TopMatchConfidence / SecondMatchConfidence\n", "- alternative_codes -> List of other plausible codes (in the KNN neighborhood)\n", "- all_k_resolutions -> All codes descriptions\n", "- all_k_results -> All resolved codes for metrics calculation purposes\n", "- sentence -> SentenceId\n", "\n", "We will use following Generic Function For Getting the Codes and Relation Pairs" ] }, { "cell_type": "code", "execution_count": null, "id": "7bb59495-6397-4aed-99fa-691678a6ff6e", "metadata": { "id": "7bb59495-6397-4aed-99fa-691678a6ff6e" }, "outputs": [], "source": [ "import pandas as pd\n", "pd.set_option('display.max_colwidth', 0)\n", "\n", "def get_codes (lp, text, vocab='company_name', hcc=False):\n", "\n", " \"\"\"Returns LightPipeline resolution results\"\"\"\n", " \n", " full_light_result = lp.fullAnnotate(text)\n", "\n", " chunks = []\n", " codes = []\n", " begin = []\n", " end = []\n", " resolutions=[]\n", " all_distances =[]\n", " all_codes=[]\n", " all_cosines = []\n", " all_k_aux_labels=[]\n", "\n", " for i in range(len(full_light_result)):\n", "\n", " for chunk, code in zip(full_light_result[i]['ner_chunk'], full_light_result[i][vocab]): \n", " begin.append(chunk.begin)\n", " end.append(chunk.end)\n", " chunks.append(chunk.result)\n", " codes.append(code.result) \n", " all_codes.append(code.metadata['all_k_results'].split(':::'))\n", " resolutions.append(code.metadata['all_k_resolutions'].split(':::'))\n", " all_distances.append(code.metadata['all_k_distances'].split(':::'))\n", " all_cosines.append(code.metadata['all_k_cosine_distances'].split(':::'))\n", " if hcc:\n", " try:\n", " all_k_aux_labels.append(code.metadata['all_k_aux_labels'].split(':::'))\n", " except:\n", " all_k_aux_labels.append([])\n", " else:\n", " all_k_aux_labels.append([])\n", "\n", " df = pd.DataFrame({'chunks':chunks, 'begin': begin, 'end':end, 'code':codes, 'all_codes':all_codes, \n", " 'resolutions':resolutions, 'all_k_aux_labels':all_k_aux_labels,'all_distances':all_cosines})\n", " \n", " return df" ] }, { "cell_type": "markdown", "id": "5MDk7OgUjoII", "metadata": { "id": "5MDk7OgUjoII" }, "source": [ "### Normalized Name\n", "In Edgar, the company official is different! We need to take it before being able to augment with external information in EDGAR.\n", "\n", "- Incorrect: `Contact Gold`\n", "- Correct (Official): `Contact Gold Corp`" ] }, { "cell_type": "markdown", "id": "f872a951-5a66-4a2e-ae48-639d0c9e88e8", "metadata": { "id": "f872a951-5a66-4a2e-ae48-639d0c9e88e8" }, "source": [ "## Find Company IRS Number" ] }, { "cell_type": "markdown", "id": "4db6dbd3-ee32-48e1-a83d-fb2583325a68", "metadata": { "id": "4db6dbd3-ee32-48e1-a83d-fb2583325a68" }, "source": [ "An employer identification number (EIN) is a nine-digit number assigned by the `IRS`. It's used to identify the tax accounts of employers and certain others who have no employees. The IRS uses the number to identify taxpayers who are required to file various business tax returns. EINs are used by employers, sole proprietors, corporations, partnerships, non-profit associations, trusts, estates of decedents, government agencies, certain individuals, and other business entities.\n", "\n" ] }, { "cell_type": "markdown", "id": "yJHeKmujX6j6", "metadata": { "id": "yJHeKmujX6j6" }, "source": [ "" ] }, { "cell_type": "code", "execution_count": null, "id": "bFnnCbB_bUWj", "metadata": { "id": "bFnnCbB_bUWj" }, "outputs": [], "source": [ "NORM_ORG = 'Contact Gold Corp'" ] }, { "cell_type": "code", "execution_count": null, "id": "5c18cf27-6d9b-46f1-bf06-dafc7336d6dc", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5c18cf27-6d9b-46f1-bf06-dafc7336d6dc", "outputId": "8b7acdfa-8239-455e-e65f-9f177aba783b" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tfhub_use download started this may take some time.\n", "Approximate size to download 923.7 MB\n", "[OK!]\n", "legel_edgar_irs download started this may take some time.\n", "[OK!]\n" ] } ], "source": [ "documentAssembler = nlp.DocumentAssembler()\\\n", " .setInputCol(\"text\")\\\n", " .setOutputCol(\"ner_chunk\")\n", "\n", "embeddings = nlp.UniversalSentenceEncoder.pretrained(\"tfhub_use\", \"en\") \\\n", " .setInputCols(\"ner_chunk\") \\\n", " .setOutputCol(\"sentence_embeddings\")\n", "\n", "resolver = legal.SentenceEntityResolverModel.pretrained(\"legel_edgar_irs\", \"en\", \"legal/models\") \\\n", " .setInputCols([\"sentence_embeddings\"]) \\\n", " .setOutputCol(\"irs_code\")\\\n", " .setDistanceFunction(\"EUCLIDEAN\")\n", "\n", "pipelineModel = nlp.PipelineModel(\n", " stages = [\n", " documentAssembler,\n", " embeddings,\n", " resolver])\n", "\n", "lp = nlp.LightPipeline(pipelineModel)" ] }, { "cell_type": "code", "execution_count": null, "id": "V7iCIm5egnZp", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 177 }, "id": "V7iCIm5egnZp", "outputId": "8ce22146-8492-4eae-fb53-217a35e17c33" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 2 µs, sys: 1 µs, total: 3 µs\n", "Wall time: 6.2 µs\n" ] }, { "data": { "text/html": [ "\n", "
| \n", " | chunks | \n", "begin | \n", "end | \n", "code | \n", "all_codes | \n", "resolutions | \n", "all_k_aux_labels | \n", "all_distances | \n", "
|---|---|---|---|---|---|---|---|---|
| 0 | \n", "Contact Gold Corp | \n", "0 | \n", "16 | \n", "981369960 | \n", "[981369960, 223386947, 261918920, 760594911, 392060052, 0, 841473173, 202805440] | \n", "[981369960, 223386947, 261918920, 760594911, 392060052, 0, 841473173, 202805440] | \n", "[] | \n", "[0.0000, 0.1380, 0.1388, 0.1402, 0.1406, 0.1456, 0.1497, 0.1528] | \n", "