nlp_architect.models.transformers package

Submodules

nlp_architect.models.transformers.base_model module

class nlp_architect.models.transformers.base_model.InputFeatures(input_ids, input_mask, segment_ids, label_id=None, valid_ids=None)

Bases: object

A single set of features of data.

class nlp_architect.models.transformers.base_model.TransformerBase(model_type: str, model_name_or_path: str, labels: List[str] = None, num_labels: int = None, config_name=None, tokenizer_name=None, do_lower_case=False, output_path=None, device='cpu', n_gpus=0)

Bases: nlp_architect.models.TrainableModel

Transformers base model (for working with pytorch-transformers models)

MODEL_CONFIGURATIONS = {'bert': (<class 'transformers.configuration_bert.BertConfig'>, <class 'transformers.tokenization_bert.BertTokenizer'>), 'quant_bert': (<class 'nlp_architect.models.transformers.quantized_bert.QuantizedBertConfig'>, <class 'transformers.tokenization_bert.BertTokenizer'>), 'roberta': (<class 'transformers.configuration_roberta.RobertaConfig'>, <class 'transformers.tokenization_roberta.RobertaTokenizer'>), 'xlm': (<class 'transformers.configuration_xlm.XLMConfig'>, <class 'transformers.tokenization_xlm.XLMTokenizer'>), 'xlnet': (<class 'transformers.configuration_xlnet.XLNetConfig'>, <class 'transformers.tokenization_xlnet.XLNetTokenizer'>)}

evaluate_predictions(logits, label_ids)

get_logits(batch)

get model logits from given input

static get_train_steps_epochs(max_steps: int, num_train_epochs: int, gradient_accumulation_steps: int, num_samples: int)

get train steps and epochs

Parameters:

• max_steps (int) – max steps
• num_train_epochs (int) – num epochs
• gradient_accumulation_steps (int) – gradient accumulation steps
• num_samples (int) – number of samples

Returns:

total steps, number of epochs

Return type:

Tuple

classmethod load_model(model_path: str, model_type: str, *args, **kwargs)

Create a TranformerBase deom from given path

Parameters:

• model_path (str) – path to model
• model_type (str) – model type

Returns:

model

Return type:

TransformerBase

optimizer

save_model(output_dir: str, save_checkpoint: bool = False, args=None)

Save model/tokenizer/arguments to given output directory

Parameters:

• output_dir (str) – path to output directory
• save_checkpoint (bool, optional) – save as checkpoint. Defaults to False.
• args ([type], optional) – arguments object to save. Defaults to None.

save_model_checkpoint(output_path: str, name: str)

save model checkpoint

Parameters:

• output_path (str) – output path
• name (str) – name of checkpoint

scheduler

setup_default_optimizer(weight_decay: float = 0.0, learning_rate: float = 5e-05, adam_epsilon: float = 1e-08, warmup_steps: int = 0, total_steps: int = 0)

to(device='cpu', n_gpus=0)

nlp_architect.models.transformers.base_model.get_models(models: List[str])

nlp_architect.models.transformers.quantized_bert module

Quantized BERT layers and model

class nlp_architect.models.transformers.quantized_bert.QuantizedBertAttention(config)

Bases: transformers.modeling_bert.BertAttention

prune_heads(heads)

class nlp_architect.models.transformers.quantized_bert.QuantizedBertConfig(vocab_size=30522, hidden_size=768, num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072, hidden_act='gelu', hidden_dropout_prob=0.1, attention_probs_dropout_prob=0.1, max_position_embeddings=512, type_vocab_size=2, initializer_range=0.02, layer_norm_eps=1e-12, **kwargs)

Bases: transformers.configuration_bert.BertConfig

pretrained_config_archive_map = {'bert-base-uncased': 'https://nlp-architect-data.s3-us-west-2.amazonaws.com/models/transformers/bert-base-uncased.json', 'bert-large-uncased': 'https://nlp-architect-data.s3-us-west-2.amazonaws.com/models/transformers/bert-large-uncased.json'}

class nlp_architect.models.transformers.quantized_bert.QuantizedBertEmbeddings(config)

Bases: transformers.modeling_bert.BertEmbeddings

class nlp_architect.models.transformers.quantized_bert.QuantizedBertEncoder(config)

Bases: transformers.modeling_bert.BertEncoder

class nlp_architect.models.transformers.quantized_bert.QuantizedBertForQuestionAnswering(config)

Bases: nlp_architect.models.transformers.quantized_bert.QuantizedBertPreTrainedModel, transformers.modeling_bert.BertForQuestionAnswering

class nlp_architect.models.transformers.quantized_bert.QuantizedBertForSequenceClassification(config)

Bases: nlp_architect.models.transformers.quantized_bert.QuantizedBertPreTrainedModel, transformers.modeling_bert.BertForSequenceClassification

class nlp_architect.models.transformers.quantized_bert.QuantizedBertForTokenClassification(config)

Bases: nlp_architect.models.transformers.quantized_bert.QuantizedBertPreTrainedModel, transformers.modeling_bert.BertForTokenClassification

class nlp_architect.models.transformers.quantized_bert.QuantizedBertIntermediate(config)

Bases: transformers.modeling_bert.BertIntermediate

class nlp_architect.models.transformers.quantized_bert.QuantizedBertLayer(config)

Bases: transformers.modeling_bert.BertLayer

class nlp_architect.models.transformers.quantized_bert.QuantizedBertModel(config)

Bases: nlp_architect.models.transformers.quantized_bert.QuantizedBertPreTrainedModel, transformers.modeling_bert.BertModel

class nlp_architect.models.transformers.quantized_bert.QuantizedBertOutput(config)

Bases: transformers.modeling_bert.BertOutput

class nlp_architect.models.transformers.quantized_bert.QuantizedBertPooler(config)

Bases: transformers.modeling_bert.BertPooler

class nlp_architect.models.transformers.quantized_bert.QuantizedBertPreTrainedModel(config, *inputs, **kwargs)

Bases: transformers.modeling_bert.BertPreTrainedModel

base_model_prefix = 'quant_bert'

config_class

alias of QuantizedBertConfig

classmethod from_pretrained(pretrained_model_name_or_path, *args, from_8bit=False, **kwargs)

load trained model from 8bit model

init_weights(module)

Initialize the weights.

save_pretrained(save_directory)

save trained model in 8bit

toggle_8bit(mode: bool)

class nlp_architect.models.transformers.quantized_bert.QuantizedBertSelfAttention(config)

Bases: transformers.modeling_bert.BertSelfAttention

class nlp_architect.models.transformers.quantized_bert.QuantizedBertSelfOutput(config)

Bases: transformers.modeling_bert.BertSelfOutput

nlp_architect.models.transformers.quantized_bert.quantized_embedding_setup(config, name, *args, **kwargs)

Get QuantizedEmbedding layer according to config params

nlp_architect.models.transformers.quantized_bert.quantized_linear_setup(config, name, *args, **kwargs)

Get QuantizedLinear layer according to config params

nlp_architect.models.transformers.sequence_classification module

class nlp_architect.models.transformers.sequence_classification.TransformerSequenceClassifier(model_type: str, labels: List[str] = None, task_type='classification', metric_fn=<function accuracy>, load_quantized=False, *args, **kwargs)

Bases: nlp_architect.models.transformers.base_model.TransformerBase

Transformer sequence classifier

Parameters:

• model_type (str) – transformer base model type
• labels (List[str], optional) – list of labels. Defaults to None.
• task_type (str, optional) – task type (classification/regression). Defaults to
• classification. –
• metric_fn ([type], optional) – metric to use for evaluation. Defaults to
• simple_accuracy. –

MODEL_CLASS = {'bert': <class 'transformers.modeling_bert.BertForSequenceClassification'>, 'quant_bert': <class 'nlp_architect.models.transformers.quantized_bert.QuantizedBertForSequenceClassification'>, 'roberta': <class 'transformers.modeling_roberta.RobertaForSequenceClassification'>, 'xlm': <class 'transformers.modeling_xlm.XLMForSequenceClassification'>, 'xlnet': <class 'transformers.modeling_xlnet.XLNetForSequenceClassification'>}

convert_to_tensors(examples: List[nlp_architect.data.sequence_classification.SequenceClsInputExample], max_seq_length: int = 128, include_labels: bool = True) → torch.utils.data.dataset.TensorDataset

Convert examples to tensor dataset

Parameters:

• examples (List[SequenceClsInputExample]) – examples
• max_seq_length (int, optional) – max sequence length. Defaults to 128.
• include_labels (bool, optional) – include labels. Defaults to True.

Returns:

Return type:

TensorDataset

evaluate_predictions(logits, label_ids)

Run evaluation of given logits and truth labels

Parameters:

• logits – model logits
• label_ids – truth label ids

inference(examples: List[nlp_architect.data.sequence_classification.SequenceClsInputExample], max_seq_length: int, batch_size: int = 64, evaluate=False)

Run inference on given examples

Parameters:

• examples (List[SequenceClsInputExample]) – examples
• batch_size (int, optional) – batch size. Defaults to 64.

Returns:

logits

train(train_data_set: torch.utils.data.dataloader.DataLoader, dev_data_set: Union[torch.utils.data.dataloader.DataLoader, List[torch.utils.data.dataloader.DataLoader]] = None, test_data_set: Union[torch.utils.data.dataloader.DataLoader, List[torch.utils.data.dataloader.DataLoader]] = None, gradient_accumulation_steps: int = 1, per_gpu_train_batch_size: int = 8, max_steps: int = -1, num_train_epochs: int = 3, max_grad_norm: float = 1.0, logging_steps: int = 50, save_steps: int = 100)

Train a model

Parameters:

• train_data_set (DataLoader) – training data set
• dev_data_set (Union[DataLoader, List[DataLoader]], optional) – development set.
• to None. (Defaults) –
• test_data_set (Union[DataLoader, List[DataLoader]], optional) – test set.
• to None. –
• gradient_accumulation_steps (int, optional) – num of gradient accumulation steps.
• to 1. (Defaults) –
• per_gpu_train_batch_size (int, optional) – per GPU train batch size. Defaults to 8.
• max_steps (int, optional) – max steps. Defaults to -1.
• num_train_epochs (int, optional) – number of train epochs. Defaults to 3.
• max_grad_norm (float, optional) – max gradient normalization. Defaults to 1.0.
• logging_steps (int, optional) – number of steps between logging. Defaults to 50.
• save_steps (int, optional) – number of steps between model save. Defaults to 100.

nlp_architect.models.transformers.token_classification module

class nlp_architect.models.transformers.token_classification.BertTokenClassificationHead(config)

Bases: transformers.modeling_bert.BertForTokenClassification

BERT token classification head with linear classifier. This head’s forward ignores word piece tokens in its linear layer.

The forward requires an additional ‘valid_ids’ map that maps the tensors for valid tokens (e.g., ignores additional word piece tokens generated by the tokenizer, as in NER task the ‘X’ label).

forward(input_ids, token_type_ids=None, attention_mask=None, labels=None, position_ids=None, head_mask=None, valid_ids=None)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class nlp_architect.models.transformers.token_classification.QuantizedBertForTokenClassificationHead(config)

Bases: nlp_architect.models.transformers.quantized_bert.QuantizedBertForTokenClassification

Quantized BERT token classification head with linear classifier. This head’s forward ignores word piece tokens in its linear layer.

The forward requires an additional ‘valid_ids’ map that maps the tensors for valid tokens (e.g., ignores additional word piece tokens generated by the tokenizer, as in NER task the ‘X’ label).

forward(input_ids, token_type_ids=None, attention_mask=None, labels=None, position_ids=None, head_mask=None, valid_ids=None)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class nlp_architect.models.transformers.token_classification.RobertaForTokenClassificationHead(config)

Bases: transformers.modeling_bert.BertPreTrainedModel

RoBERTa token classification head with linear classifier. This head’s forward ignores word piece tokens in its linear layer.

The forward requires an additional ‘valid_ids’ map that maps the tensors for valid tokens (e.g., ignores additional word piece tokens generated by the tokenizer, as in NER task the ‘X’ label).

base_model_prefix = 'roberta'

config_class

alias of transformers.configuration_roberta.RobertaConfig

forward(input_ids, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, labels=None, valid_ids=None)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

pretrained_model_archive_map = {'distilroberta-base': 'https://s3.amazonaws.com/models.huggingface.co/bert/distilroberta-base-pytorch_model.bin', 'roberta-base': 'https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-pytorch_model.bin', 'roberta-base-openai-detector': 'https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-openai-detector-pytorch_model.bin', 'roberta-large': 'https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-pytorch_model.bin', 'roberta-large-mnli': 'https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-mnli-pytorch_model.bin', 'roberta-large-openai-detector': 'https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-openai-detector-pytorch_model.bin'}

class nlp_architect.models.transformers.token_classification.TransformerTokenClassifier(model_type: str, labels: List[str] = None, *args, load_quantized=False, **kwargs)

Bases: nlp_architect.models.transformers.base_model.TransformerBase

Transformer word tagging classifier

Parameters:

• model_type (str) – model family (classifier head), choose between bert/quant_bert/xlnet
• labels (List[str], optional) – list of tag labels

MODEL_CLASS = {'bert': <class 'nlp_architect.models.transformers.token_classification.BertTokenClassificationHead'>, 'quant_bert': <class 'nlp_architect.models.transformers.token_classification.QuantizedBertForTokenClassificationHead'>, 'roberta': <class 'nlp_architect.models.transformers.token_classification.RobertaForTokenClassificationHead'>, 'xlnet': <class 'nlp_architect.models.transformers.token_classification.XLNetTokenClassificationHead'>}

convert_to_tensors(examples: List[nlp_architect.data.sequential_tagging.TokenClsInputExample], max_seq_length: int = 128, include_labels: bool = True) → torch.utils.data.dataset.TensorDataset

Convert examples to tensor dataset

Parameters:

• examples (List[SequenceClsInputExample]) – examples
• max_seq_length (int, optional) – max sequence length. Defaults to 128.
• include_labels (bool, optional) – include labels. Defaults to True.

Returns:

Return type:

TensorDataset

evaluate_predictions(logits, label_ids)

Run evaluation of given logist and truth labels

Parameters:

• logits – model logits
• label_ids – truth label ids

static extract_labels(label_ids, label_map, logits)

inference(examples: List[nlp_architect.data.sequential_tagging.TokenClsInputExample], max_seq_length: int, batch_size: int = 64)

Run inference on given examples

Parameters:

• examples (List[SequenceClsInputExample]) – examples
• batch_size (int, optional) – batch size. Defaults to 64.

Returns:

logits

train(train_data_set: torch.utils.data.dataloader.DataLoader, dev_data_set: Union[torch.utils.data.dataloader.DataLoader, List[torch.utils.data.dataloader.DataLoader]] = None, test_data_set: Union[torch.utils.data.dataloader.DataLoader, List[torch.utils.data.dataloader.DataLoader]] = None, gradient_accumulation_steps: int = 1, per_gpu_train_batch_size: int = 8, max_steps: int = -1, num_train_epochs: int = 3, max_grad_norm: float = 1.0, logging_steps: int = 50, save_steps: int = 100)

Run model training

Parameters:

• train_data_set (DataLoader) – training dataset
• dev_data_set (Union[DataLoader, List[DataLoader]], optional) – development data set
• be list) Defaults to None. ((can) –
• test_data_set (Union[DataLoader, List[DataLoader]], optional) – test data set
• be list) Defaults to None. –
• gradient_accumulation_steps (int, optional) – gradient accumulation steps.
• to 1. (Defaults) –
• per_gpu_train_batch_size (int, optional) – per GPU train batch size (or GPU).
• to 8. (Defaults) –
• max_steps (int, optional) – max steps for training. Defaults to -1.
• num_train_epochs (int, optional) – number of training epochs. Defaults to 3.
• max_grad_norm (float, optional) – max gradient norm. Defaults to 1.0.
• logging_steps (int, optional) – number of steps between logging. Defaults to 50.
• save_steps (int, optional) – number of steps between model save. Defaults to 100.

class nlp_architect.models.transformers.token_classification.XLNetTokenClassificationHead(config)

Bases: transformers.modeling_xlnet.XLNetPreTrainedModel

XLNet token classification head with linear classifier. This head’s forward ignores word piece tokens in its linear layer.

The forward requires an additional ‘valid_ids’ map that maps the tensors for valid tokens (e.g., ignores additional word piece tokens generated by the tokenizer, as in NER task the ‘X’ label).

forward(input_ids, token_type_ids=None, input_mask=None, attention_mask=None, mems=None, perm_mask=None, target_mapping=None, labels=None, head_mask=None, valid_ids=None)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Module contents