# Language Modeling and Analysis

## Survey
- [2020 ArXiv] **Efficient Transformers: A Survey**, [[paper]](https://arxiv.org/pdf/2009.06732.pdf), [[bibtex]](/Bibtex/Efficient%20Transformers%20-%20A%20Survey.bib).

## Language Models
- sources: [[thunlp/PLMpapers]](https://github.com/thunlp/PLMpapers).
- sources: [[Jiakui/awesome-bert]](https://github.com/Jiakui/awesome-bert).
- **Transferring NLP models across languages and domains**, [[slides]](https://syntaxfest.github.io/syntaxfest19/slides/invited_talk_syntaxfest_plank.pdf).
- [2017 ICML] **Language Modeling with Gated Convolutional Networks**, [[paper]](https://arxiv.org/pdf/1612.08083.pdf), [[bibtex]](/Bibtex/Language%20Modeling%20with%20Gated%20Convolutional%20Networks.bib), sources: [[anantzoid/Language-Modeling-GatedCNN]](https://github.com/anantzoid/Language-Modeling-GatedCNN), [[jojonki/Gated-Convolutional-Networks]](https://github.com/jojonki/Gated-Convolutional-Networks).
- [2017 NIPS] **Learned in Translation: Contextualized Word Vectors**, [[paper]](https://arxiv.org/pdf/1708.00107.pdf), [[bibtex]](/Bibtex/Learned%20in%20Translation.bib), sources: [[salesforce/cove]](https://github.com/salesforce/cove).
- [2018 ICLR] **Regularizing and Optimizing LSTM Language Models**, [[paper]](https://openreview.net/pdf?id=SyyGPP0TZ), [[bibtex]](/Bibtex/Regularizing%20and%20Optimizing%20LSTM%20Language%20Models.bib), sources: [[salesforce/awd-lstm-lm]](https://github.com/salesforce/awd-lstm-lm), author page: [[Nitish Shirish Keskar]](https://keskarnitish.github.io).
- [2018 NAACL] **Deep contextualized word representations**, [[paper]](https://www.aclweb.org/anthology/N18-1202.pdf), [[bibtex]](https://www.aclweb.org/anthology/N18-1202.bib), [[homepage]](https://allennlp.org/elmo), sources: [[allenai/bilm-tf]](https://github.com/allenai/bilm-tf), [[HIT-SCIR/ELMoForManyLangs]](https://github.com/HIT-SCIR/ELMoForManyLangs). Some extended application: [[UKPLab/elmo-bilstm-cnn-crf]](https://github.com/UKPLab/elmo-bilstm-cnn-crf).
- [2018 NeurIPS] **GLoMo: Unsupervisedly Learned Relational Graphs as Transferable Representations**, [[paper]](https://arxiv.org/pdf/1806.05662.pdf), [[bibtex]](GLoMo%20-%20Unsupervisedly%20Learned%20Relational%20Graphs%20as%20Transferable%20Representations.bib), sources: [[YJHMITWEB/GLoMo-tensorflow]](https://github.com/YJHMITWEB/GLoMo-tensorflow).
- [2018 ArXiv] **Improving Language Understanding by Generative Pre-Training**, [[paper]](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf), [[bibtex]](/Bibtex/Improving%20Language%20Understanding%20by%20Generative%20Pre-Training.bib), [[homepage]](https://blog.openai.com/language-unsupervised/), sources: [[openai/finetune-transformer-lm]](https://github.com/openai/finetune-transformer-lm).
- [2019 AAAI] **Character-Level Language Modeling with Deeper Self-Attention**, [[paper]](https://arxiv.org/pdf/1808.04444.pdf), [[bibtex]](/Bibtex/Character-Level%20Language%20Modeling%20with%20Deeper%20Self-Attention.bib), sources: [[nadavbh12/Character-Level-Language-Modeling-with-Deeper-Self-Attention-pytorch]](https://github.com/nadavbh12/Character-Level-Language-Modeling-with-Deeper-Self-Attention-pytorch).
- [2019 NAACL] **BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding**, [[paper]](https://www.aclweb.org/anthology/N19-1423.pdf), [[bibtex]](/Bibtex/BERT%20-%20Pre-training%20of%20Deep%20Bidirectional%20Transformers%20for%20Language%20Understanding.bib), [[slides]](https://nlp.stanford.edu/seminar/details/jdevlin.pdf), sources: [[google-research/bert]](https://github.com/google-research/bert), [[huggingface/pytorch-pretrained-BERT]](https://github.com/huggingface/pytorch-pretrained-BERT). Blog posts: 
  - daiwk的BERT解读: [I](https://daiwk.github.io/posts/nlp-bert.html), [II](https://daiwk.github.io/posts/nlp-bert-code-annotated-framework.html), [III](https://daiwk.github.io/posts/nlp-bert-code-annotated-application.html), [IV](https://daiwk.github.io/posts/nlp-bert-code.html)
  - Dissecting BERT: [I](https://medium.com/dissecting-bert/dissecting-bert-part-1-d3c3d495cdb3), [II](https://medium.com/dissecting-bert/dissecting-bert-appendix-the-decoder-3b86f66b0e5f), [III](https://medium.com/dissecting-bert/dissecting-bert-part2-335ff2ed9c73)
  - The Illustrated Transformer: [EN](https://jalammar.github.io/illustrated-transformer/), [CN](https://zhuanlan.zhihu.com/p/54356280)
  - The Annotated Transformer: [EN](http://nlp.seas.harvard.edu/2018/04/03/attention.html)
  - 从Word Embedding到BERT - NLP预训练技术发展史: [CN](https://zhuanlan.zhihu.com/p/49271699)
  - NLP三大特征抽取器(CNN/RNN/Transformer)比较: [CN](https://zhuanlan.zhihu.com/p/54743941)
- [2019 ACL] **Adaptive Attention Span in Transformers**, [[paper]](https://www.aclweb.org/anthology/P19-1032.pdf), [[bibtex]](https://www.aclweb.org/anthology/P19-1032.bib), sources: [[facebookresearch/adaptive-span]](https://github.com/facebookresearch/adaptive-span).
- [2019 ICML] **BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning**, [[paper]](http://proceedings.mlr.press/v97/stickland19a/stickland19a.pdf), [[bibtex]](/Bibtex/BERT%20and%20PALs%20-%20Projected%20Attention%20Layers%20for%20Efficient%20Adaptation%20in%20Multi-Task%20Learning.bib), [[supplementary]](http://proceedings.mlr.press/v97/stickland19a/stickland19a-supp.pdf), sources: [[AsaCooperStickland/Bert-n-Pals]](https://github.com/AsaCooperStickland/Bert-n-Pals).
- [2019 ArXiv] **GPT-2: Language Models are Unsupervised Multitask Learners**, [[paper]](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf), [[bibtex]](/Bibtex/Language%20Models%20are%20Unsupervised%20Multitask%20Learners.bib), [[homepage]](https://blog.openai.com/better-language-models/), sources: [[openai/gpt-2]](https://github.com/openai/gpt-2).
- [2019 ICLR] **What Do You Learn from Context? Probing for Sentence Structure in Contextualized Word Representations**, [[paper]](https://openreview.net/pdf?id=SJzSgnRcKX), [[bibtex]](/Bibtex/What%20Do%20You%20Learn%20from%20Context%20Probing%20for%20Sentence%20Structure%20in%20Contextualized%20Word%20Representations.bib).
- [2019 ICML] **MASS: Masked Sequence to Sequence Pre-training for Language Generation**, [[paper]](https://arxiv.org/pdf/1905.02450.pdf), [[bibtex]](/Bibtex/MASS%20-%20Masked%20Sequence%20to%20Sequence%20Pre-training%20for%20Language%20Generation.bib), sources: [[xutaatmicrosoftdotcom/MASS]](https://github.com/xutaatmicrosoftdotcom/MASS).
- [2019 ACL] **ERNIE: Enhanced Language Representation with Informative Entities**, [[paper]](https://arxiv.org/pdf/1905.07129.pdf), [[bibtex]](/Bibtex/ERNIE%20-%20Enhanced%20Language%20Representation%20with%20Informative%20Entities.bib), [[blog]](https://www.jiqizhixin.com/articles/2019-05-26-4), sources: [[thunlp/ERNIE]](https://github.com/thunlp/ERNIE).
- [2019 ACL] **Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context**, [[paper]](https://www.aclweb.org/anthology/P19-1285.pdf), [[bibtex]](https://www.aclweb.org/anthology/P19-1285.bib), sources: [[kimiyoung/transformer-xl]](https://github.com/kimiyoung/transformer-xl).
- [2019 IJCNLP] **Cloze-driven Pretraining of Self-attention Networks**, [[paper]](https://arxiv.org/pdf/1903.07785.pdf), [[bibtex]](/Bibtex/Cloze-driven%20Pretraining%20of%20Self-attention%20Networks.bib).
- [2019 NeurIPS] **XLNet: Generalized Autoregressive Pretraining for Language Understanding**, [[paper]](https://papers.nips.cc/paper/8812-xlnet-generalized-autoregressive-pretraining-for-language-understanding.pdf), [[bibtex]](/Bibtex/XLNet%20-%20Generalized%20Autoregressive%20Pretraining%20for%20Language%20Understanding.bib), [[Supplementary]](https://papers.nips.cc/paper/8812-xlnet-generalized-autoregressive-pretraining-for-language-understanding), sources: [[zihangdai/xlnet]](https://github.com/zihangdai/xlnet).
- [2019 NeurIPS] **Cross-lingual Language Model Pretraining**, [[paper]](https://papers.nips.cc/paper/2019/file/c04c19c2c2474dbf5f7ac4372c5b9af1-Paper.pdf), [[bibtex]](/Bibtex/Cross-lingual%20Language%20Model%20Pretraining.bib), sources: [[facebookresearch/XLM]](https://github.com/facebookresearch/XLM).
- [2019 NeurIPS] **Unified Language Model Pre-training for Natural Language Understanding and Generation**, [[paper]](https://papers.nips.cc/paper/9464-unified-language-model-pre-training-for-natural-language-understanding-and-generation.pdf), [[bibtex]](/Bibtex/Unified%20Language%20Model%20Pre-training%20for%20Natural%20Language%20Understanding%20and%20Generation.bib), sources: [[microsoft/unilm]](https://github.com/microsoft/unilm).
- [2019 ICML] **Improving Neural Language Modeling via Adversarial Training**, [[paper]](http://proceedings.mlr.press/v97/wang19f/wang19f.pdf), [[bibtex]](/Bibtex/Improving%20Neural%20Language%20Modeling%20via%20Adversarial%20Training.bib), sources: [[ChengyueGongR/advsoft]](https://github.com/ChengyueGongR/advsoft).
- [2019 ArXiv] **RoBERTa: A Robustly Optimized BERT Pretraining Approach**, [[paper]](https://arxiv.org/pdf/1907.11692.pdf), [[bibtex]](/Bibtex/RoBERTa%20-%20A%20Robustly%20Optimized%20BERT%20Pretraining%20Approach.bib), sources: [[pytorch/fairseq]](https://github.com/pytorch/fairseq/tree/master/examples/roberta).
- [2019 ArXiv] **NeZha: Neural Contextualized Representation for Chinese Language Understanding**, [[paper]](https://arxiv.org/pdf/1909.00204.pdf), [[bibtex]](/Bibtex/NeZha%20-%20Neural%20Contextualized%20Representation%20for%20Chinese%20Language%20Understanding.bib), sources: [[huawei-noah/Pretrained-Language-Model/NEZHA]](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/NEZHA).
- [2020 AAAI] **K-BERT: Enabling Language Representation with Knowledge Graph**, [[paper]](https://www.aaai.org/Papers/AAAI/2020GB/AAAI-LiuW.5594.pdf), [[bibtex]](/Bibtex/K-BERT%20-%20Enabling%20Language%20Representation%20with%20Knowledge%20Graph.bib).
- [2020 ICLR] **ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators**, [[paper]](https://openreview.net/pdf?id=r1xMH1BtvB), [[bibtex]](/Bibtex/ELECTRA%20-%20Pre-training%20Text%20Encoders%20as%20Discriminators%20Rather%20Than%20Generators.bib).
- [2020 ICLR] **ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations**, [[paper]](https://openreview.net/pdf?id=H1eA7AEtvS), [[bibtex]](/Bibtex/ALBERT%20-%20A%20Lite%20BERT%20for%20Self-Supervised%20Learning%20of%20Language%20Representations.bib), sources: [[google-research/ALBERT]](https://github.com/google-research/ALBERT).
- [2020 ICLR] **FreeLB: Enhanced Adversarial Training for Natural Language Understanding**, [[paper]](https://openreview.net/pdf?id=BygzbyHFvB), [[bibtex]](/Bibtex/FreeLB%20-%20Enhanced%20Adversarial%20Training%20for%20Natural%20Language%20Understanding.bib), sources: [[zhuchen03/FreeLB]](https://github.com/zhuchen03/FreeLB).
- [2020 ICLR] **Improving Neural Language Generation with Spectrum Control**, [[paper]](https://openreview.net/pdf?id=ByxY8CNtvr), [[bibtex]](/Bibtex/Improving%20Neural%20Language%20Generation%20with%20Spectrum%20Control.bib).
- [2020 ACL] **Emerging Cross-lingual Structure in Pretrained Language Models**, [[paper]](https://arxiv.org/pdf/1911.01464.pdf), [[bibtex]](/Bibtex/Emerging%20Cross-lingual%20Structure%20in%20Pretrained%20Language%20Models.bib).
- [2020 ACL] **MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices**, [[paper]](https://www.aclweb.org/anthology/2020.acl-main.195.pdf), [[bibtex]](/Bibtex/MobileBERT.bib), sources: [[google-research/mobilebert]](https://github.com/google-research/google-research/tree/master/mobilebert).
- [2020 ACL] **Entities as Experts: Sparse Memory Access with Entity Supervision**, [[paper]](https://www.aclweb.org/anthology/2020.emnlp-main.400.pdf), [[bibtex]](https://www.aclweb.org/anthology/2020.emnlp-main.400.bib).
- [2020 TACL] **SpanBERT: Improving Pre-training by Representing and Predicting Spans**, [[paper]](https://www.aclweb.org/anthology/2020.tacl-1.5.pdf), [[bibtex]](https://www.aclweb.org/anthology/2020.tacl-1.5.bib), sources: [[facebookresearch/SpanBERT]](https://github.com/facebookresearch/SpanBERT).
- [2020 EMNLP] **TinyBERT: Distilling BERT for Natural Language Understanding**, [[paper]](https://www.aclweb.org/anthology/2020.findings-emnlp.372.pdf), [[bibtex]](https://www.aclweb.org/anthology/2020.findings-emnlp.372.bib), sources: [[huawei-noah/TinyBERT]](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/TinyBERT).
- [2020 NeurIPS] **Language Through a Prism: A Spectral Approach for Multiscale Language Representations**, [[paper]](https://proceedings.neurips.cc/paper/2020/file/3acb2a202ae4bea8840224e6fce16fd0-Paper.pdf), [[bibtex]](/Bibtex/Language%20Through%20a%20Prism%20-%20A%20Spectral%20Approach%20for%20Multiscale%20Language%20Representations.bib).
- [2020 ArXiv] **Critical Thinking for Language Models**, [[paper]](https://arxiv.org/pdf/2009.07185.pdf), [[bibtex]](/Bibtex/Critical%20Thinking%20for%20Language%20Models.bib).
- [2021 ICLR] **DeBERTa: Decoding-enhanced BERT with Disentangled Attention**, [[paper]](https://openreview.net/pdf?id=XPZIaotutsD), [[bibtex]](/Bibtex/DeBERTa%20-%20Decoding-enhanced%20BERT%20with%20Disentangled%20Attention.bib), sources: [[microsoft/DeBERTa]](https://github.com/microsoft/DeBERTa).
- [2021 ArXiv] **All NLP Tasks Are Generation Tasks: A General Pretraining Framework**, [[paper]](https://arxiv.org/pdf/2103.10360.pdf), [[bibtex]](/Bibtex/All%20NLP%20Tasks%20Are%20Generation%20Tasks%20-%20A%20General%20Pretraining%20Framework.bib), sources: [[THUDM/GLM]](https://github.com/THUDM/GLM).
- [2021 ArXiv] **An Attention Free Transformer**, [[paper]](https://arxiv.org/pdf/2105.14103.pdf), [[bibtex]](/Bibtex/An%20Attention%20Free%20Transformer.bib), sources: [[rish-16/aft-pytorch]](https://github.com/rish-16/aft-pytorch).
- [2021 NAACL] **INFOXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training**, [[paper]](https://www.aclweb.org/anthology/2021.naacl-main.280.pdf), [[bibtex]](https://www.aclweb.org/anthology/2021.naacl-main.280.bib), sources: [[microsoft/infoxlm]](https://github.com/microsoft/unilm/tree/master/infoxlm).
- [2021 EMNLP] **UNKs Everywhere: Adapting Multilingual Language Models to New Scripts**, [[paper]](https://arxiv.org/pdf/2012.15562.pdf), [[bibtex]](/Bibtex/UNKs%20Everywhere%20-%20Adapting%20Multilingual%20Language%20Models%20to%20New%20Scripts.bib), sources: [[Adapter-Hub/UNKs_everywhere]](https://github.com/Adapter-Hub/UNKs_everywhere).
- [2021 NeurIPS] **Pay Attention to MLPs**, [[paper]](https://openreview.net/pdf?id=KBnXrODoBW), [[bibtex]](/Bibtex/Pay%20Attention%20to%20MLPs.bib), sources: [[rwightman/pytorch-image-models]](https://github.com/rwightman/pytorch-image-models), [[labmlai/annotated_deep_learning_paper_implementations]](https://github.com/labmlai/annotated_deep_learning_paper_implementations), [[xmu-xiaoma666/External-Attention-pytorch]](https://github.com/xmu-xiaoma666/External-Attention-pytorch), [[PaddleViT/gMLP]](https://github.com/BR-IDL/PaddleViT/tree/main/image_classification/gMLP).
- [2022 ArXiv] **Efficient Language Modeling with Sparse all-MLP**, [[paper]](https://arxiv.org/pdf/2203.06850.pdf), [[bibtex]](/Bibtex/Efficient%20Language%20Modeling%20with%20Sparse%20all-MLP.bib).
- [2022 ICLR] **ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning**, [[paper]](https://openreview.net/pdf?id=Vzh1BFUCiIX), [[bibtex]](/Bibtex/ExT5%20-%20Towards%20Extreme%20Multi-Task%20Scaling%20for%20Transfer%20Learning.bib), sources: [[google-research/text-to-text-transfer-transformer]](https://github.com/google-research/text-to-text-transfer-transformer), [[tensorflow/mesh]](https://github.com/tensorflow/mesh).

## Language Model Analysis
- [2019 ACL] **How multilingual is Multilingual BERT?**, [[paper]](https://www.aclweb.org/anthology/P19-1493.pdf), [[bibtex]](https://www.aclweb.org/anthology/P19-1493.bib).
- [2019 ACL] **What does BERT learn about the structure of language?**, [[paper]](https://www.aclweb.org/anthology/P19-1356.pdf), [[bibtex]](/Bibtex/What%20does%20BERT%20learn%20about%20the%20structure%20of%20language.bib), sources: [[ganeshjawahar/interpret_bert]](https://github.com/ganeshjawahar/interpret_bert).
- [2019 EMNLP] **Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT**, [[paper]](https://www.aclweb.org/anthology/D19-1077.pdf), [[bibtex]](/Bibtex/Beto%20Bentz%20Becas%20-%20The%20Surprising%20Cross-Lingual%20Effectiveness%20of%20BERT.bib), sources: [[shijie-wu/crosslingual-nlp]](https://github.com/shijie-wu/crosslingual-nlp).
- [2019 EMNLP] **How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings**, [[paper]](https://www.aclweb.org/anthology/D19-1006.pdf), [[bibtex]](https://www.aclweb.org/anthology/D19-1006.bib), sources: [[kawine/contextual]](https://github.com/kawine/contextual).
- [2019 ICLR] **Representation Degeneration Problem in Training Natural Language Generation Models**, [[paper]](https://openreview.net/pdf?id=SkEYojRqtm), [[bibtex]](/Bibtex/Representation%20Degeneration%20Problem%20in%20Training%20Natural%20Language%20Generation%20Models.bib)
- [2019 ArXiv] **What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?**, [[paper]](https://arxiv.org/pdf/1910.12391.pdf), [[bibtex]](/Bibtex/What%20does%20BERT%20Learn%20from%20Multiple-Choice%20Reading%20Comprehension%20Datasets.bib).
- [2020 JMLR] **Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer**, [[paper]](https://jmlr.org/papers/volume21/20-074/20-074.pdf), [[bibtex]](/Bibtex/Exploring%20the%20Limits%20of%20Transfer%20Learning%20with%20a%20Unified%20Text-to-Text%20Transformer.bib), sources: [[google-research/text-to-text-transfer-transformer]](https://github.com/google-research/text-to-text-transfer-transformer).

## Position Embedding
- [2018 NAACL] **Self-Attention with Relative Position Representations**, [[paper]](https://www.aclweb.org/anthology/N18-2074.pdf), [[bibtex]](https://www.aclweb.org/anthology/N18-2074.bib), sources: [[TensorUI/relative-position-pytorch]](https://github.com/TensorUI/relative-position-pytorch), [[tensorflow/tensor2tensor]](https://github.com/tensorflow/tensor2tensor), [[OpenNMT/OpenNMT-tf]](https://github.com/OpenNMT/OpenNMT-tf).
- [2020 ICML] **Learning to Encode Position for Transformer with Continuous Dynamical Model**, [[paper]](http://proceedings.mlr.press/v119/liu20n/liu20n.pdf), [[bibtex]](/Bibtex/Learning%20to%20Encode%20Position%20for%20Transformer%20with%20Continuous%20Dynamical%20Model.bib), sources: [[xuanqing94/FLOATER]](https://github.com/xuanqing94/FLOATER).
- [2021 ICLR] **Rethinking Positional Encoding in Language Pre-training**, [[paper]](https://openreview.net/pdf?id=09-528y2Fgf), [[bibtex]](/Bibtex/Rethinking%20Positional%20Encoding%20in%20Language%20Pre-training.bib), sources: [[guolinke/TUPE]](https://github.com/guolinke/TUPE).