# Image-based Vision-and-Language Modelling and Multi-task Learning

## Vision-and-Language Modelling
- [2019 ArXiv] **M-BERT: Injecting Multimodal Information in the BERT Structure**, [[paper]](https://arxiv.org/pdf/1908.05787.pdf), [[bibtex]](/Bibtex/M-BERT%20-%20Injecting%20Multimodal%20Information%20in%20the%20BERT%20Structure.bib).
- [2019 NeurIPS] **ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks**, [[paper]](https://papers.nips.cc/paper/8297-vilbert-pretraining-task-agnostic-visiolinguistic-representations-for-vision-and-language-tasks.pdf), [[bibtex]](/Bibtex/ViLBERT%20-%20Pretraining%20Task-Agnostic%20Visiolinguistic%20Representations%20for%20Vision-and-Language%20Tasks.bib), sources: [[jiasenlu/vilbert_beta]](https://github.com/jiasenlu/vilbert_beta).
- [2019 ArXiv] **VisualBERT: A Simple and Performant Baseline for Vision and Language**, [[paper]](https://arxiv.org/pdf/1908.03557.pdf), [[bibtex]](/Bibtex/VisualBERT%20-%20A%20Simple%20and%20Performant%20Baseline%20for%20Vision%20and%20Language.bib), sources: [[uclanlp/visualbert]](https://github.com/uclanlp/visualbert).
- [2019 EMNLP] **LXMERT: Learning Cross-Modality Encoder Representations from Transformers**, [[paper]](https://www.aclweb.org/anthology/D19-1514.pdf), [[bibtex]](/Bibtex/LXMERT%20-%20Learning%20Cross-Modality%20Encoder%20Representations%20from%20Transformers.bib), sources: [[airsplay/lxmert]](https://github.com/airsplay/lxmert).
- [2019 CVPR] **Multi-task Learning of Hierarchical Vision-Language Representation**, [[paper]](http://openaccess.thecvf.com/content_CVPR_2019/papers/Nguyen_Multi-Task_Learning_of_Hierarchical_Vision-Language_Representation_CVPR_2019_paper.pdf), [[bibtex]](/Bibtex/Multi-task%20Learning%20of%20Hierarchical%20Vision-Language%20Representation.bib).
- [2020 AAAI] **Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training**, [[paper]](https://arxiv.org/pdf/1908.06066.pdf), [[bibtex]](/Bibtex/Unicoder-VL%20-%20A%20Universal%20Encoder%20for%20Vision%20and%20Language%20by%20Cross-modal%20Pre-training.bib).
- [2020 AAAI] **Unified Vision-Language Pre-Training for Image Captioning and VQA**, [[paper]](https://arxiv.org/pdf/1909.11059.pdf), [[bibtex]](/Bibtex/Unified%20Vision-Language%20Pre-Training%20for%20Image%20Captioning%20and%20VQA.bib), sources: [[LuoweiZhou/VLP]](https://github.com/LuoweiZhou/VLP).
- [2020 ECCV] **UNITER: Learning Universal Image-Text Representations**, [[paper]](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123750103.pdf), [[bibtex]](/Bibtex/UNITER%20-%20Learning%20Universal%20Image-Text%20Representations.bib), sources: [[ChenRocks/UNITER]](https://github.com/ChenRocks/UNITER).
- [2020 ACMMM] **DeVLBert: Learning Deconfounded Visio-Linguistic Representations**, [[paper]](https://dl.acm.org/doi/pdf/10.1145/3394171.3413518?casa_token=wZJMxpJMxL4AAAAA%3AiXxJxr738Q2fOVO98Dx7w3vAbNrV8LYNutNa6XfIXrir6TX9x0Hms_pguGcF4fn8l-oNAsl_tlBl), [[bibtex]](/Bibtex/DeVLBert%20-%20Learning%20Deconfounded%20Visio-Linguistic%20Representations.bib), sources: [[shengyuzhang/DeVLBert]](https://github.com/shengyuzhang/DeVLBert).
- [2020 ICLR] **VL-BERT: Pre-training of Generic Visual-Linguistic Representations**, [[paper]](https://openreview.net/pdf?id=SygXPaEYvH), [[bibtex]](/Bibtex/VL-BERT%20-%20Pre-training%20of%20Generic%20Visual-Linguistic%20Representations.bib), sources: [[jackroos/VL-BERT]](https://github.com/jackroos/VL-BERT).
- [2020 ICLR] **Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling**, [[paper]](https://openreview.net/pdf?id=H1x5wRVtvS), [[bibtex]](/Bibtex/Variational%20Hetero-Encoder%20Randomized%20GANs%20for%20Joint%20Image-Text%20Modeling.bib).
- [2020 ECCV] **Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks**, [[paper]](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123750120.pdf), [[bibtex]](/Bibtex/Oscar.bib), sources: [[microsoft/Oscar]](https://github.com/microsoft/Oscar).
- [2020 ECCV] **Learning Visual Representations with Caption Annotations**, [[paper]](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123530154.pdf), [[bibtex]](/Bibtex/Learning%20Visual%20Representations%20with%20Caption%20Annotations.bib), [[homepage]](https://europe.naverlabs.com/research/computer-vision/icmlm/).
- [2020 ArXiv] **Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers**, [[paper]](https://arxiv.org/pdf/2004.00849.pdf), [[bibtex]](/Bibtex/Pixel-BERT.bib).
- [2020 ArXiv] **ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data**, [[paper]](https://arxiv.org/pdf/2001.07966.pdf), [[bibtex]](/Bibtex/ImageBERT%20-%20Cross-modal%20Pre-training%20with%20Large-scale%20Weak-supervised%20Image-Text%20Data.bib).
- [2020 ArXiv] **Contrastive Learning of Medical Visual Representations from Paired Images and Text**, [[paper]](https://arxiv.org/pdf/2010.00747.pdf), [[bibtex]](/Bibtex/Contrastive%20Learning%20of%20Medical%20Visual%20Representations%20from%20Paired%20Images%20and%20Text.bib), sources: [[edreisMD/ConVIRT-pytorch]](https://github.com/edreisMD/ConVIRT-pytorch).
- [2021 ArXiv] **SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels**, [[paper]](https://arxiv.org/pdf/2103.07829.pdf), [[bibtex]](/Bibtex/SemVLP%20-%20Vision-Language%20Pre-training%20by%20Aligning%20Semantics%20at%20Multiple%20Levels.bib).
- [2021 AAAI] **VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning**, [[paper]](https://arxiv.org/pdf/2009.13682.pdf), [[bibtex]](/Bibtex/VIVO.bib).
- [2021 CVPR] **Causal Attention for Vision-Language Tasks**, [[paper]](https://openaccess.thecvf.com/content/CVPR2021/papers/Yang_Causal_Attention_for_Vision-Language_Tasks_CVPR_2021_paper.pdf), [[bibtex]](/Bibtex/Causal%20Attention%20for%20Vision-Language%20Tasks.bib), [[supplementary]](https://openaccess.thecvf.com/content/CVPR2021/supplemental/Yang_Causal_Attention_for_CVPR_2021_supplemental.pdf), sources: [[yangxuntu/lxmertcatt]](https://github.com/yangxuntu/lxmertcatt).
- [2021 CVPR] **VirTex: Learning Visual Representations from Textual Annotations**, [[paper]](https://openaccess.thecvf.com/content/CVPR2021/papers/Desai_VirTex_Learning_Visual_Representations_From_Textual_Annotations_CVPR_2021_paper.pdf), [[bibtex]](/Bibtex/VirTex%20-%20Learning%20Visual%20Representations%20from%20Textual%20Annotations.bib), [[homepage]](http://kdexd.xyz/virtex/), sources: [[kdexd/virtex]](https://github.com/kdexd/virtex).
- [2021 TKDD] **DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention**, [[paper]](/Documents/Papers/DiMBERT%20-%20Learning%20Vision-Language%20Grounded%20Representations%20with%20Disentangled%20Multimodal-Attention.pdf), [[bibtex]](/Bibtex/DiMBERT%20-%20Learning%20Vision-Language%20Grounded%20Representations%20with%20Disentangled%20Multimodal-Attention.bib).
- [2021 ICML] **CLIP: Learning Transferable Visual Models From Natural Language Supervision**, [[paper]](https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language_Supervision.pdf), [[bibtex]](/Bibtex/Learning%20Transferable%20Visual%20Models%20From%20Natural%20Language%20Supervision.bib), [[slides]](https://icml.cc/media/icml-2021/Slides/9193.pdf), sources: [[openai/CLIP]](https://github.com/openai/CLIP).
- [2021 NeurIPS] **Align before Fuse: Vision and Language Representation Learning with Momentum Distillation**, [[paper]](https://arxiv.org/pdf/2107.07651.pdf), [[bibtex]](/Bibtex/Align%20before%20Fuse%20-%20Vision%20and%20Language%20Representation%20Learning%20with%20Momentum%20Distillation.bib), sources: [[salesforce/ALBEF]](https://github.com/salesforce/ALBEF).

## Multi-task and Meta Learning
- [2020 CVPR] **12-in-1: Multi-Task Vision and Language Representation Learning**, [[paper]](https://openaccess.thecvf.com/content_CVPR_2020/papers/Lu_12-in-1_Multi-Task_Vision_and_Language_Representation_Learning_CVPR_2020_paper.pdf), [[bibtex]](/Bibtex/12-in-1%20-%20Multi-Task%20Vision%20and%20Language%20Representation%20Learning.bib), [[supplementary]](https://openaccess.thecvf.com/content_CVPR_2020/supplemental/Lu_12-in-1_Multi-Task_Vision_CVPR_2020_supplemental.pdf).
- [2021 ICCV] **UniT: Multimodal Multitask Learning with a Unified Transformer**, [[paper]](https://openaccess.thecvf.com/content/ICCV2021/papers/Hu_UniT_Multimodal_Multitask_Learning_With_a_Unified_Transformer_ICCV_2021_paper.pdf), [[bibtex]](/Bibtex/UniT%20-%20Multimodal%20Multitask%20Learning%20with%20a%20Unified%20Transformer.bib), [[homepage]](https://mmf.sh), sources: [[facebookresearch/mmf]](https://github.com/facebookresearch/mmf).
- [2021 CVPR] **Learning Universal Representations via Multitask Multilingual Multimodal Pre-training**, [[paper]](https://arxiv.org/pdf/2006.02635.pdf), [[bibtex]](/Bibtex/Learning%20Universal%20Representations%20via%20Multitask%20Multilingual%20Multimodal%20Pre-training.bib), sources: [[microsoft/M3P]](https://github.com/microsoft/M3P).