nlp_architect.utils package

Subpackages

• nlp_architect.utils.resources package
  • Module contents

Submodules

nlp_architect.utils.ansi2html module

nlp_architect.utils.ansi2html.ansi2html(text, palette='solarized')

nlp_architect.utils.ansi2html.run(file, out)

nlp_architect.utils.embedding module

class nlp_architect.utils.embedding.ELMoEmbedderTFHUB

Bases: object

get_vector(tokens)

class nlp_architect.utils.embedding.FasttextEmbeddingsModel(size: int = 5, window: int = 3, min_count: int = 1, skipgram: bool = True)

Bases: object

Fasttext embedding trainer class

Parameters:

• texts (List[List[str]]) – list of tokenized sentences
• size (int) – embedding size
• epochs (int, optional) – number of epochs to train
• window (int, optional) – The maximum distance between
• current and predicted word within a sentence (the) –

classmethod load(path)

load model from path

save(path) → None

save model to path

train(texts: List[List[str]], epochs: int = 100)

vec(word: str) → numpy.ndarray

return vector corresponding given word

nlp_architect.utils.embedding.fill_embedding_mat(src_mat, src_lex, emb_lex, emb_size)

Creates a new matrix from given matrix of int words using the embedding model provided.

Parameters:

• src_mat (numpy.ndarray) – source matrix
• src_lex (dict) – source matrix lexicon
• emb_lex (dict) – embedding lexicon
• emb_size (int) – embedding vector size

nlp_architect.utils.embedding.get_embedding_matrix(embeddings: dict, vocab: nlp_architect.utils.text.Vocabulary, embedding_size: int = None) → numpy.ndarray

Generate a matrix of word embeddings given a vocabulary

Parameters:

• embeddings (dict) – a dictionary of embedding vectors
• vocab (Vocabulary) – a Vocabulary
• embedding_size (int) – custom embedding matrix size

Returns:

a 2D numpy matrix of lexicon embeddings

nlp_architect.utils.embedding.load_embedding_file(filename: str) → dict

Load a word embedding file

Parameters:filename (str) – path to embedding file Returns:dictionary with embedding vectors Return type:dict

nlp_architect.utils.embedding.load_word_embeddings(file_path, vocab=None)

Loads a word embedding model text file into a word(str) to numpy vector dictionary

Parameters:

• file_path (str) – path to model file
• vocab (list of str) – optional - vocabulary

Returns:

a dictionary of numpy.ndarray vectors int: detected word embedding vector size

Return type:

list

nlp_architect.utils.file_cache module

Utilities for working with the local dataset cache.

nlp_architect.utils.file_cache.cached_path(url_or_filename: Union[str, pathlib.Path], cache_dir: str = None) → str

Given something that might be a URL (or might be a local path), determine which. If it’s a URL, download the file and cache it, and return the path to the cached file. If it’s already a local path, make sure the file exists and then return the path.

nlp_architect.utils.file_cache.filename_to_url(filename: str, cache_dir: str = None) → Tuple[str, str]

Return the url and etag (which may be None) stored for filename. Raise FileNotFoundError if filename or its stored metadata do not exist.

nlp_architect.utils.file_cache.get_from_cache(url: str, cache_dir: str = None) → str

Given a URL, look for the corresponding dataset in the local cache. If it’s not there, download it. Then return the path to the cached file.

nlp_architect.utils.file_cache.http_get(url: str, temp_file: IO) → None

nlp_architect.utils.file_cache.url_to_filename(url: str, etag: str = None) → str

Convert url into a hashed filename in a repeatable way. If etag is specified, append its hash to the url’s, delimited by a period.

nlp_architect.utils.generic module

nlp_architect.utils.generic.add_offset(mat: numpy.ndarray, offset: int = 1) → numpy.ndarray

Add +1 to all values in matrix mat

Parameters:

• mat (numpy.ndarray) – A 2D matrix with int values
• offset (int) – offset to add

Returns:

input matrix

Return type:

numpy.ndarray

nlp_architect.utils.generic.balance(df)

nlp_architect.utils.generic.license_prompt(model_name, model_website, dataset_dir=None)

nlp_architect.utils.generic.normalize(txt, vocab=None, replace_char=' ', max_length=300, pad_out=True, to_lower=True, reverse=False, truncate_left=False, encoding=None)

nlp_architect.utils.generic.one_hot(mat: numpy.ndarray, num_classes: int) → numpy.ndarray

Convert a 1D matrix of ints into one-hot encoded vectors.

Parameters:

• mat (numpy.ndarray) – A 1D matrix of labels (int)
• num_classes (int) – Number of all possible classes

Returns:

A 2D matrix

Return type:

numpy.ndarray

nlp_architect.utils.generic.one_hot_sentence(mat: numpy.ndarray, num_classes: int) → numpy.ndarray

Convert a 2D matrix of ints into one-hot encoded 3D matrix

Parameters:

• mat (numpy.ndarray) – A 2D matrix of labels (int)
• num_classes (int) – Number of all possible classes

Returns:

A 3D matrix

Return type:

numpy.ndarray

nlp_architect.utils.generic.pad_sentences(sequences: numpy.ndarray, max_length: int = None, padding_value: int = 0, padding_style='post') → numpy.ndarray

Pad input sequences up to max_length values are aligned to the right

Parameters:

• sequences (iter) – a 2D matrix (np.array) to pad
• max_length (int, optional) – max length of resulting sequences
• padding_value (int, optional) – padding value
• padding_style (str, optional) – add padding values as prefix (use with ‘pre’) or postfix (use with ‘post’)

Returns:

input sequences padded to size ‘max_length’

nlp_architect.utils.generic.to_one_hot(txt, vocab={'!': 40, '#': 49, '$': 50, '%': 51, '&': 53, '(': 61, ')': 62, '*': 54, '+': 57, ', ': 37, '-': 36, '.': 39, '/': 44, '0': 26, '1': 27, '2': 28, '3': 29, '4': 30, '5': 31, '6': 32, '7': 33, '8': 34, '9': 35, ':': 42, ';': 38, '<': 59, '=': 58, '>': 60, '?': 41, '@': 48, '[': 63, '\\': 45, ']': 64, '_': 47, 'a': 0, 'b': 1, 'c': 2, 'd': 3, 'e': 4, 'f': 5, 'g': 6, 'h': 7, 'i': 8, 'j': 9, 'k': 10, 'l': 11, 'm': 12, 'n': 13, 'o': 14, 'p': 15, 'q': 16, 'r': 17, 's': 18, 't': 19, 'u': 20, 'v': 21, 'w': 22, 'x': 23, 'y': 24, 'z': 25, '{': 65, '|': 46, '}': 66, 'ˆ': 52, '˜': 55, '‘': 56, '’': 43})

nlp_architect.utils.io module

nlp_architect.utils.io.check(validator)

nlp_architect.utils.io.check_directory_and_create(dir_path)

Check if given directory exists, create if not.

Parameters:dir_path (str) – path to directory

nlp_architect.utils.io.check_size(min_size=None, max_size=None)

nlp_architect.utils.io.create_folder(path)

nlp_architect.utils.io.download_unlicensed_file(url, sourcefile, destfile, totalsz=None)

Download the file specified by the given URL.

Parameters:

• url (str) – url to download from
• sourcefile (str) – file to download from url
• destfile (str) – save path
• totalsz (int, optional) – total size of file

nlp_architect.utils.io.download_unzip(url: str, sourcefile: str, unzipped_path: str, license_msg: str = None)

Downloads a zip file, extracts it to destination, deletes the zip file. If license_msg is supplied, user is prompted for download confirmation.

nlp_architect.utils.io.gzip_str(g_str)

Transform string to GZIP coding

Parameters:g_str (str) – string of data Returns:GZIP bytes data

nlp_architect.utils.io.json_dumper(obj)

for objects that have members that cant be serialized and implement toJson() method

nlp_architect.utils.io.line_count(file)

Utility function for getting number of lines in a text file.

nlp_architect.utils.io.load_files_from_path(dir_path, extension='txt')

load all files from given directory (with given extension)

nlp_architect.utils.io.load_json_file(file_path)

load a file into a json object

nlp_architect.utils.io.prepare_output_path(output_dir: str, overwrite_output_dir: str)

Create output directory or throw error if exists and overwrite_output_dir is false

nlp_architect.utils.io.sanitize_path(path)

nlp_architect.utils.io.uncompress_file(filepath: str, outpath='.')

Unzip a file to the same location of filepath uses decompressing algorithm by file extension

Parameters:

• filepath (str) – path to file
• outpath (str) – path to extract to

nlp_architect.utils.io.valid_path_append(path, *args)

Helper to validate passed path directory and append any subsequent filename arguments.

Parameters:

• path (str) – Initial filesystem path. Should expand to a valid directory.
• *args (list, optional) – Any filename or path suffices to append to path for returning.
• Returns –

  (list, str): path prepended list of files from args, or path alone if

  no args specified.

Raises:

ValueError – if path is not a valid directory on this filesystem.

nlp_architect.utils.io.validate(*args)

Validate all arguments are of correct type and in correct range. :param *args: Each tuple represents an argument validation like so: :type *args: tuple of tuples :param Option 1 - With range check: (arg, class, min_val, max_val) :param Option 2 - Without range check: (arg, class) :param If class is a tuple of type objects check if arg is an instance of any of the types.: :param To allow a None valued argument, include type: :type To allow a None valued argument, include type: None :param To disable lower or upper bound check, set min_val or max_val to None, respectively.: :param If arg has the len attribute: :type If arg has the len attribute: such as string

nlp_architect.utils.io.validate_boolean(arg)

Validates an input argument of type boolean

nlp_architect.utils.io.validate_existing_directory(arg)

Validates an input argument is a path string to an existing directory.

nlp_architect.utils.io.validate_existing_filepath(arg)

Validates an input argument is a path string to an existing file.

nlp_architect.utils.io.validate_existing_path(arg)

Validates an input argument is a path string to an existing file or directory.

nlp_architect.utils.io.validate_parent_exists(arg)

Validates an input argument is a path string, and its parent directory exists.

nlp_architect.utils.io.validate_proxy_path(arg)

Validates an input argument is a valid proxy path or None

nlp_architect.utils.io.walk_directory(directory, verbose=False)

Iterates a directory’s text files and their contents.

nlp_architect.utils.io.zipfile_list(filepath: str)

List the files inside a given zip file

Parameters:filepath (str) – path to file Returns:String list of filenames

nlp_architect.utils.metrics module

nlp_architect.utils.metrics.acc_and_f1(preds, labels)

return accuracy and f1 score

nlp_architect.utils.metrics.accuracy(preds, labels)

return simple accuracy in expected dict format

nlp_architect.utils.metrics.get_conll_scores(predictions, y, y_lex, unk='O')

Get Conll style scores (precision, recall, f1)

nlp_architect.utils.metrics.pearson_and_spearman(preds, labels)

get pearson and spearman correlation

nlp_architect.utils.metrics.simple_accuracy(preds, labels)

return simple accuracy

nlp_architect.utils.metrics.tagging(preds, labels)

nlp_architect.utils.string_utils module

class nlp_architect.utils.string_utils.StringUtils

Bases: object

determiners = []

static find_head_lemma_pos_ner(x: str)

“

Parameters:x – mention Returns:the head word and the head word lemma of the mention

static is_determiner(in_str: str) → bool

static is_preposition(in_str: str) → bool

static is_pronoun(in_str: str) → bool

static is_stop(token: str) → bool

static normalize_str(in_str: str) → str

static normalize_string_list(str_list: str) → List[str]

preposition = []

pronouns = []

spacy_no_parser = <nlp_architect.utils.text.SpacyInstance object>

spacy_parser = <nlp_architect.utils.text.SpacyInstance object>

stop_words = []

nlp_architect.utils.testing module

class nlp_architect.utils.testing.NLPArchitectTestCase(methodName='runTest')

Bases: unittest.case.TestCase

setUp()

Hook method for setting up the test fixture before exercising it.

tearDown()

Hook method for deconstructing the test fixture after testing it.

nlp_architect.utils.text module

class nlp_architect.utils.text.SpacyInstance(model='en', disable=None, display_prompt=True)

Bases: object

Spacy pipeline wrapper which prompts user for model download authorization.

Parameters:

• model (str, optional) – spacy model name (default: english small model)
• disable (list of string, optional) – pipeline annotators to disable (default: [])
• display_prompt (bool, optional) – flag to display/skip license prompt

parser

return Spacy’s instance parser

tokenize(text: str) → List[str]

Tokenize a sentence into tokens :param text: text to tokenize :type text: str

Returns:a list of str tokens of input Return type:list

class nlp_architect.utils.text.Stopwords

Bases: object

Stop words list class.

static get_words()

stop_words = []

class nlp_architect.utils.text.Vocabulary(start=0, include_oov=True)

Bases: object

A vocabulary that maps words to ints (storing a vocabulary)

add(word)

Add word to vocabulary

Parameters:word (str) – word to add Returns:id of added word Return type:int

add_vocab_offset(offset)

Adds an offset to the ints of the vocabulary

Parameters:offset (int) – an int offset

id_to_word(wid)

Word-id to word (string)

Parameters:wid (int) – word id Returns:string of given word id Return type:str

max

reverse_vocab()

Return the vocabulary as a reversed dict object

Returns:reversed vocabulary object Return type:dict

vocab

get the dict object of the vocabulary

Type:dict

word_id(word)

Get the word_id of given word

Parameters:word (str) – word from vocabulary Returns:int id of word Return type:int

nlp_architect.utils.text.bio_to_spans(text: List[str], tags: List[str]) → List[Tuple[int, int, str]]

Convert BIO tagged list of strings into span starts and ends :param text: list of words :param tags: list of tags

Returns:list of start, end and tag of detected spans Return type:tuple

nlp_architect.utils.text.char_to_id(c)

return int id of given character

OOV char = len(all_letter) + 1

Parameters:c (str) – string character Returns:int value of given char Return type:int

nlp_architect.utils.text.character_vector_generator(data, start=0)

Character word vector generator util. Transforms a list of sentences into numpy int vectors of the characters of the words of the sentence, and returns the constructed vocabulary

Parameters:

• data (list) – list of list of strings
• start (int, optional) – vocabulary index start integer

Returns:

a 2D numpy array Vocabulary: constructed vocabulary

Return type:

np.array

nlp_architect.utils.text.extract_nps(annotation_list, text=None)

Extract Noun Phrases from given text tokens and phrase annotations. Returns a list of tuples with start/end indexes.

Parameters:

• annotation_list (list) – a list of annotation tags in str
• text (list, optional) – a list of token texts in str

Returns:

list of start/end markers of noun phrases, if text is provided a list of noun phrase texts

nlp_architect.utils.text.id_to_char(c_id)

return character of given char id

nlp_architect.utils.text.read_sequential_tagging_file(file_path, ignore_line_patterns=None)

Read a tab separated sequential tagging file. Returns a list of list of tuple of tags (sentences, words)

Parameters:

• file_path (str) – input file path
• ignore_line_patterns (list, optional) – list of string patterns to ignore

Returns:

list of list of tuples

nlp_architect.utils.text.simple_normalizer(text)

Simple text normalizer. Runs each token of a phrase thru wordnet lemmatizer and a stemmer.

nlp_architect.utils.text.spacy_normalizer(text, lemma=None)

Simple text normalizer using spacy lemmatizer. Runs each token of a phrase thru a lemmatizer and a stemmer. :param text: the text to normalize. :type text: string :param lemma: lemma of the given text. in this case only stemmer will :type lemma: string :param run.:

nlp_architect.utils.text.try_to_load_spacy(model_name)

nlp_architect.utils.text.word_vector_generator(data, lower=False, start=0)

Word vector generator util. Transforms a list of sentences into numpy int vectors and returns the constructed vocabulary

Parameters:

• data (list) – list of list of strings
• lower (bool, optional) – transform strings into lower case
• start (int, optional) – vocabulary index start integer

Returns:

2D numpy array and Vocabulary of the detected words

Module contents