# Parameter Efficient Learning and Continual Learning

## Adapter-based Parameter Efficient Learning
- [2019 ICML] **Parameter-Efficient Transfer Learning for NLP**, [[paper]](http://proceedings.mlr.press/v97/houlsby19a/houlsby19a.pdf), [[bibtex]](/Bibtex/Parameter-Efficient%20Transfer%20Learning%20for%20NLP.bib), [[slides]](https://icml.cc/media/icml-2019/Slides/4884.pdf), sources: [[google-research/adapter-bert]](https://github.com/google-research/adapter-bert).
- [2019 ICML] **BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning**, [[paper]](http://proceedings.mlr.press/v97/stickland19a/stickland19a.pdf), [[bibtex]](/Bibtex/BERT%20and%20PALs%20-%20Projected%20Attention%20Layers%20for%20Efficient%20Adaptation%20in%20Multi-Task%20Learning.bib), [[supplementary]](http://proceedings.mlr.press/v97/stickland19a/stickland19a-supp.pdf), sources: [[AsaCooperStickland/Bert-n-Pals]](https://github.com/AsaCooperStickland/Bert-n-Pals).
- [2020 ArXiv] **Continual Learning in Task-Oriented Dialogue Systems**, [[paper]](https://arxiv.org/pdf/2012.15504.pdf), [[bibtex]](/Bibtex/Continual%20Learning%20in%20Task-Oriented%20Dialogue%20Systems.bib).
- [2020 ArXiv] **Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer**, [[paper]](https://arxiv.org/pdf/2012.06460.pdf), [[bibtex]](/Bibtex/Orthogonal%20Language%20and%20Task%20Adapters%20in%20Zero-Shot%20Cross-Lingual%20Transfer.bib).
- [2020 EMNLP] **AdapterHub: A Framework for Adapting Transformers**, [[paper]](https://aclanthology.org/2020.emnlp-demos.7.pdf), [[bibtex]](/Bibtex/AdapterHub%20-%20A%20Framework%20for%20Adapting%20Transformers.bib), [[homepage]](https://adapterhub.ml), sources: [[adapter-hub]](https://github.com/adapter-hub).
- [2020 EMNLP] **Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning**, [[paper]](https://aclanthology.org/2020.findings-emnlp.41.pdf), [[bibtex]](/Bibtex/Exploring%20Versatile%20Generative%20Language%20Model%20Via%20Parameter-Efficient%20Transfer%20Learning.bib), sources: [[zlinao/VGLM]](https://github.com/zlinao/VGLM).
- [2020 EMNLP] **MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer**, [[paper]](https://aclanthology.org/2020.emnlp-main.617.pdf), [[bibtex]](/Bibtex/MAD-X%20-%20An%20Adapter-Based%20Framework%20for%20Multi-Task%20Cross-Lingual%20Transfer.bib), [[homepage]](https://adapterhub.ml), sources: [[adapter-hub]](https://github.com/adapter-hub).
- [2020 EMNLP] **UDapter: Language Adaptation for Truly Universal Dependency Parsing**, [[paper]](https://aclanthology.org/2020.emnlp-main.180.pdf), [[bibtex]](/Bibtex/UDapter%20-%20Language%20Adaptation%20for%20Truly%20Universal%20Dependency%20Parsing.bib), sources: [[ahmetustun/udapter]](https://github.com/ahmetustun/udapter).
- [2020 DeeLIO] **Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers**, [[paper]](https://aclanthology.org/2020.deelio-1.5.pdf), [[bibtex]](/Bibtex/Common%20Sense%20or%20World%20Knowledge?%20Investigating%20Adapter-Based%20Knowledge%20Injection%20into%20Pretrained%20Transformers.bib), sources: [[wluper/retrograph]](https://github.com/wluper/retrograph).
- [2021 EACL] **AdapterFusion: Non-Destructive Task Composition for Transfer Learning**, [[paper]](https://aclanthology.org/2021.eacl-main.39.pdf), [[bibtex]](/Bibtex/AdapterFusion%20-%20Non-Destructive%20Task%20Composition%20for%20Transfer%20Learning.bib), [[homepage]](https://adapterhub.ml), sources: [[adapter-hub]](https://github.com/adapter-hub).
- [2021 ArXiv] **COMPACTER: Efficient Low-Rank Hypercomplex Adapter Layers**, [[paper]](https://arxiv.org/pdf/2106.04647.pdf), [[bibtex]](/Bibtex/COMPACTER%20-%20Efficient%20Low-Rank%20Hypercomplex%20Adapter%20Layers.bib), sources: [[rabeehk/compacter]](https://github.com/rabeehk/compacter).
- [2021 ACL] **K-ADAPTER: Infusing Knowledge into Pre-Trained Models with Adapters**, [[paper]](https://aclanthology.org/2021.findings-acl.121.pdf), [[bibtex]](/Bibtex/K-ADAPTER%20-%20Infusing%20Knowledge%20into%20Pre-Trained%20Models%20with%20Adapters.bib), sources: [[microsoft/K-Adapter]](https://github.com/microsoft/K-Adapter).
- [2021 ACL] **On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation**, [[paper]](https://aclanthology.org/2021.acl-long.172.pdf), [[bibtex]](/Bibtex/On%20the%20Effectiveness%20of%20Adapter-based%20Tuning%20for%20Pretrained%20Language%20Model%20Adaptation.bib).
- [2021 ACL] **Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks**, [[paper]](https://aclanthology.org/2021.acl-long.47.pdf), [[bibtex]](/Bibtex/Parameter-efficient%20Multi-task%20Fine-tuning%20for%20Transformers%20via%20Shared%20Hypernetworks.bib), sources: [[rabeehk/hyperformer]](https://github.com/rabeehk/hyperformer).
- [2021 ACL] **Robust Transfer Learning with Pretrained Language Models through Adapters**, [[paper]](https://aclanthology.org/2021.acl-short.108.pdf), [[bibtex]](/Bibtex/Robust%20Transfer%20Learning%20with%20Pretrained%20Language%20Models%20through%20Adapters.bib), sources: [[WinnieHAN/Adapter-Robustness]](https://github.com/WinnieHAN/Adapter-Robustness).
- [2021 EMNLP] **Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning**, [[paper]](https://arxiv.org/pdf/2104.08808.pdf), [[bibtex]](/Bibtex/Lifelong%20Learning%20of%20Few-shot%20Learners%20across%20NLP%20Tasks.bib), sources: [[INK-USC/CLIF]](https://github.com/INK-USC/CLIF).
- [2021 AAAI] **The Adapter-Bot: All-In-One Controllable Conversational Model**, [[paper]](https://arxiv.org/pdf/2008.12579.pdf), [[bibtex]](/Bibtex/The%20Adapter-Bot%20-%20All-In-One%20Controllable%20Conversational%20Model.bib), sources: [[HLTCHKUST/adapterbot]](https://github.com/HLTCHKUST/adapterbot).

## Prompt-based Parameter Efficient Learning
- [2021 NAACL] **How Many Data Points is a Prompt Worth?**, [[paper]](https://aclanthology.org/2021.naacl-main.208.pdf), [[bibtex]](/Bibtex/How%20Many%20Data%20Points%20is%20a%20Prompt%20Worth?.bib), sources: [[TevenLeScao/pet]](https://github.com/TevenLeScao/pet).
- [2021 ACL] **Prefix-Tuning: Optimizing Continuous Prompts for Generation**, [[paper]](https://aclanthology.org/2021.acl-long.353.pdf), [[bibtex]](/Bibtex/Prefix-Tuning%20-%20Optimizing%20Continuous%20Prompts%20for%20Generation.bib), sources: [[XiangLi1999/PrefixTuning]](https://github.com/XiangLi1999/PrefixTuning).
- [2021 ArXiv] **GPT Understands, Too**, [[paper]](https://arxiv.org/pdf/2103.10385.pdf), [[bibtex]](/Bibtex/GPT%20Understands,%20Too.bib), sources: [[THUDM/P-tuning]](https://github.com/THUDM/P-tuning).
- [2021 ArXiv] **Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing**, [[paper]](https://arxiv.org/pdf/2107.13586.pdf), [[bibtex]](/Bibtex/Pre-train,%20Prompt,%20and%20Predict%20-%20A%20Systematic%20Survey%20of%20Prompting%20Methods%20in%20Natural%20Language%20Processing.bib), [[homepage]](http://pretrain.nlpedia.ai), sources: [[pfliu-nlp/NLPedia-Pretrain]](https://github.com/pfliu-nlp/NLPedia-Pretrain).

## Continual Learning (Life-long Learning)
- Contniual Learning resources: [[xialeiliu/Awesome-Incremental-Learning]](https://github.com/xialeiliu/Awesome-Incremental-Learning).
- [2016 PNAS] **Overcoming Catastrophic Forgetting in Neural Networks**, [[paper]](https://www.pnas.org/content/pnas/114/13/3521.full.pdf), [[bibtex]](/Bibtex/Overcoming%20Catastrophic%20Forgetting%20in%20Neural%20Networks.bib), [[supplementary]](https://www.pnas.org/content/pnas/suppl/2017/03/14/1611835114.DCSupplemental/pnas.201611835SI.pdf), sources: [[ariseff/overcoming-catastrophic]](https://github.com/ariseff/overcoming-catastrophic), [[AntiAegis/Overcoming-Catastrophic-Forgetting]](https://github.com/AntiAegis/Overcoming-Catastrophic-Forgetting).
- [2017 NIPS] **Overcoming Catastrophic Forgetting by Incremental Moment Matching**, [[paper]](https://papers.nips.cc/paper/7051-overcoming-catastrophic-forgetting-by-incremental-moment-matching.pdf), [[bibtex]](/Bibtex/Overcoming%20Catastrophic%20Forgetting%20by%20Incremental%20Moment%20Matching.bib), sources: [[btjhjeon/IMM_tensorflow]](https://github.com/btjhjeon/IMM_tensorflow).
- [2018 ArXiv] **Overcoming Catastrophic Forgetting by Soft Parameter Pruning**, [[paper]](https://arxiv.org/pdf/1812.01640.pdf), [[bibtex]](/Bibtex/Overcoming%20Catastrophic%20Forgetting%20by%20Soft%20Parameter%20Pruning.bib), sources: [[lehaifeng/Learning_by_memory]](https://github.com/lehaifeng/Learning_by_memory).
- [2018 ICML] **Overcoming Catastrophic Forgetting with Hard Attention to the Task**, [[paper]](http://proceedings.mlr.press/v80/serra18a/serra18a.pdf), [[bibtex]](/Bibtex/Overcoming%20Catastrophic%20Forgetting%20with%20Hard%20Attention%20to%20the%20Task.bib), [[supplementary]](http://proceedings.mlr.press/v80/serra18a/serra18a-supp.pdf), sources: [[joansj/hat]](https://github.com/joansj/hat).
- [2019 ICLR] **Overcoming Catastrophic Forgetting for Continual Learning via Model Adaptation**, [[paper]](https://openreview.net/pdf?id=ryGvcoA5YX), [[bibtex]](/Bibtex/Overcoming%20Catastrophic%20Forgetting%20for%20Continual%20Learning%20via%20Model%20Adaptation.bib), sources: [[morning-dews/PGMA_tensorflow]](https://github.com/morning-dews/PGMA_tensorflow).
- [2019 NAACL] **Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation**, [[paper]](https://www.aclweb.org/anthology/N19-1209), [[bibtex]](/Bibtex/Overcoming%20Catastrophic%20Forgetting%20During%20Domain%20Adaptation%20of%20Neural%20Machine%20Translation.bib).
-  [2019 ICML] **Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting**, [[paper]](http://proceedings.mlr.press/v97/li19m/li19m.pdf), [[bibtex]](/Bibtex/Learn%20to%20Grow%20-%20A%20Continual%20Structure%20Learning%20Framework%20for%20Overcoming%20Catastrophic%20Forgetting.bib), [[supplementary]](http://proceedings.mlr.press/v97/li19m/li19m-supp.pdf), sources: [[xilaili/LearnToGrow]](https://github.com/xilaili/LearnToGrow).
- [2020 ICLR] **BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning**, [[paper]](https://openreview.net/pdf?id=Sklf1yrYDr), [[bibtex]](/Bibtex/BatchEnsemble%20-%20An%20Alternative%20Approach%20to%20Efficient%20Ensemble%20and%20Lifelong%20Learning.bib), sources: [[google/edward2]](https://github.com/google/edward2).
- [2020 ICLR] **Continual Learning with Hypernetworks**, [[paper]](https://openreview.net/pdf?id=SJgwNerKvB), [[bibtex]](/Bibtex/Continual%20Learning%20with%20Hypernetworks.bib), sources: [[chrhenning/hypercl]](https://github.com/chrhenning/hypercl).
- [2020 ICLR] **LAMOL: LAnguage MOdeling for Lifelong Language Learning**, [[paper]](https://openreview.net/pdf?id=Skgxcn4YDS), [[bibtex]](/Bibtex/LAMOL%20-%20LAnguage%20MOdeling%20for%20Lifelong%20Language%20Learning.bib), sources: [[chho33/LAMOL]](https://github.com/chho33/LAMOL).
- [2020 EMNLP] **Continual Learning for Natural Language Generation in Task-oriented Dialog Systems**, [[paper]](https://aclanthology.org/2020.findings-emnlp.310.pdf), [[bibtex]](/Bibtex/Continual%20Learning%20for%20Natural%20Language%20Generation%20in%20Task-oriented%20Dialog%20Systems.bib), sources: [[MiFei/Continual-Learning-for-NLG]](https://github.com/MiFei/Continual-Learning-for-NLG).
- [2020 ArXiv] **Continual Learning in Task-Oriented Dialogue Systems**, [[paper]](https://arxiv.org/pdf/2012.15504.pdf), [[bibtex]](/Bibtex/Continual%20Learning%20in%20Task-Oriented%20Dialogue%20Systems.bib).
- [2021 NAACL] **Continual Learning for Text Classification with Information Disentanglement Based Regularization**, [[paper]](https://aclanthology.org/2021.naacl-main.218.pdf), [[bibtex]](/Bibtex/Continual%20Learning%20for%20Text%20Classification%20with%20Information%20Disentanglement%20Based%20Regularization.bib), sources: [[GT-SALT/IDBR]](https://github.com/GT-SALT/IDBR).
- [2021 NeurIPS] **Towards a robust experimental framework and benchmark for lifelong language learning**, [[paper]](https://openreview.net/pdf?id=yJyIjWyPJgs), [[bibtex]](/Bibtex/Towards%20a%20robust%20experimental%20framework%20and%20benchmark%20for%20lifelong%20language%20learning.bib), [[dataset]](https://github.com/AmanDaVinci/lifelong-learning), sources: [[AmanDaVinci/lifelong-learning-baselines]](https://github.com/AmanDaVinci/lifelong-learning-baselines), [[AmanDaVinci/lifelong-learning-limitations]](https://github.com/AmanDaVinci/lifelong-learning-limitations).
- [2021 ICML] **Not All Memories are Created Equal: Learning to Forget by Expiring**, [[paper]](http://proceedings.mlr.press/v139/sukhbaatar21a/sukhbaatar21a.pdf), [[bibtex]](/Bibtex/Not%20All%20Memories%20are%20Created%20Equal%20-%20Learning%20to%20Forget%20by%20Expiring.bib), [[Supplementary]](http://proceedings.mlr.press/v139/sukhbaatar21a/sukhbaatar21a-supp.pdf), [[slides]](https://icml.cc/media/icml-2021/Slides/10741.pdf), sources: [[lucidrains/learning-to-expire-pytorch]](https://github.com/lucidrains/learning-to-expire-pytorch).

## Others
- [2021 EMNLP] **UNKs Everywhere: Adapting Multilingual Language Models to New Scripts**, [[paper]](https://arxiv.org/pdf/2012.15562.pdf), [[bibtex]](/Bibtex/UNKs%20Everywhere%20-%20Adapting%20Multilingual%20Language%20Models%20to%20New%20Scripts.bib), sources: [[Adapter-Hub/UNKs_everywhere]](https://github.com/Adapter-Hub/UNKs_everywhere).
- [2021 ICLR] **Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters**, [[paper]](https://openreview.net/pdf?id=rcQdycl0zyk), [[bibtex]](/Bibtex/Beyond%20Fully-Connected%20Layers%20with%20Quaternions%20-%20Parameterization%20of%20Hypercomplex%20Multiplications%20with%201/n%20Parameters.bib), sources: [[demegire/Parameterization-of-Hypercomplex-Multiplications]](https://github.com/demegire/Parameterization-of-Hypercomplex-Multiplications).
- [2021 ACL] **Parameter-Efficient Transfer Learning with Diff Pruning**, [[paper]](https://aclanthology.org/2021.acl-long.378.pdf), [[bibtex]](/Bibtex/Parameter-Efficient%20Transfer%20Learning%20with%20Diff%20Pruning.bib), sources: [[dguo98/DiffPruning]](https://github.com/dguo98/DiffPruning).
- [2021 ArXiv] **BitFit: Simple Parameter-efficient Fine-tuningfor Transformer-based Masked Language-models**, [[paper]](https://arxiv.org/pdf/2106.10199.pdf), [[bibtex]](/Bibtex/BitFit%20-%20Simple%20Parameter-efficient%20Fine-tuningfor%20Transformer-based%20Masked%20Language-models.bib), sources: [[benzakenelad/BitFit]](https://github.com/benzakenelad/BitFit).