# Video-based Vision-and-Language Modelling

## Video and Language
- [2019 ICCV] **VideoBERT: A Joint Model for Video and Language Representation Learning**, [[paper]](http://openaccess.thecvf.com/content_ICCV_2019/papers/Sun_VideoBERT_A_Joint_Model_for_Video_and_Language_Representation_Learning_ICCV_2019_paper.pdf), [[bibtex]](/Bibtex/VideoBERT%20-%20A%20Joint%20Model%20for%20Video%20and%20Language%20Representation%20Learning.bib).
- [2019 ICCV] **HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips**, [[paper]](https://arxiv.org/pdf/1906.03327.pdf), [[bibtex]](/Bibtex/HowTo100M%20-%20Learning%20a%20Text-Video%20Embedding%20by%20Watching%20Hundred%20Million%20Narrated%20Video%20Clips.bib), [[homepage]](https://www.di.ens.fr/willow/research/howto100m/), sources: [[antoine77340/howto100m]](https://github.com/antoine77340/howto100m).
- [2019 ArXiv] **Learning Video Representations Using Contrastive Bidirectional Transformer**, [[paper]](https://arxiv.org/pdf/1906.05743.pdf), [[bibtex]](/Bibtex/Learning%20Video%20Representations%20Using%20Contrastive%20Bidirectional%20Transformer.bib).
- [2020 CVPR] **End-to-End Learning of Visual Representations from Uncurated Instructional Videos**, [[paper]](http://openaccess.thecvf.com/content_CVPR_2020/papers/Miech_End-to-End_Learning_of_Visual_Representations_From_Uncurated_Instructional_Videos_CVPR_2020_paper.pdf), [[bibtex]](/Bibtex/End-to-End%20Learning%20of%20Visual%20Representations%20from%20Uncurated%20Instructional%20Videos.bib), [[homepage]](https://www.di.ens.fr/willow/research/mil-nce/), sources: [[antoine77340/MIL-NCE_HowTo100M]](https://github.com/antoine77340/MIL-NCE_HowTo100M), [[MIL-NCE TFHub]](https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/text_to_video_retrieval_with_s3d_milnce.ipynb#scrollTo=nwv4ZQ4qmak5).
- [2020 ArXiv] **UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation**, [[paper]](https://arxiv.org/pdf/2002.06353.pdf), [[bibtex]](/Bibtex/UniVL.bib).
- [2020 ArXiv] **Token-level Contrast for Video and Language Alignment**, [[paper]](https://openreview.net/pdf?id=GRbZ91LKIya), [[bibtex]](/Bibtex/Token-level%20Contrast%20for%20Video%20and%20Language%20Alignment.bib).
- [2020 CVPR] **ActBERT: Learning Global-Local Video-Text Representations**, [[paper]](https://openaccess.thecvf.com/content_CVPR_2020/papers/Zhu_ActBERT_Learning_Global-Local_Video-Text_Representations_CVPR_2020_paper.pdf), [[bibtex]](/Bibtex/ActBERT%20-%20Learning%20Global-Local%20Video-Text%20Representations.bib).
- [2020 EMNLP] **HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training**, [[paper]](https://www.aclweb.org/anthology/2020.emnlp-main.161.pdf), [[bibtex]](https://www.aclweb.org/anthology/2020.emnlp-main.161.bib), sources: [[linjieli222/HERO]](https://github.com/linjieli222/HERO).
- [2021 CVPR] **Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling**, [[paper]](https://openaccess.thecvf.com/content/CVPR2021/papers/Lei_Less_Is_More_ClipBERT_for_Video-and-Language_Learning_via_Sparse_Sampling_CVPR_2021_paper.pdf), [[bibtex]](/Bibtex/Less%20is%20More%20-%20CLIPBERT%20for%20Video-and-Language%20Learning%20via%20Sparse%20Sampling.bib), sources: [[jayleicn/ClipBERT]](https://github.com/jayleicn/ClipBERT).
- [2021 ACL Findings] **VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding**, [[paper]](https://aclanthology.org/2021.findings-acl.370.pdf), [[bibtex]](/Bibtex/VLM%20-%20Task-agnostic%20Video-Language%20Model%20Pre-training%20for%20Video%20Understanding.bib), sources: [[pytorch/fairseq]](https://github.com/pytorch/fairseq).
- [2021 ACMMM] **CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising**, [[paper]](https://arxiv.org/pdf/2112.07515.pdf), [[bibtex]](/Bibtex/CoCo-BERT%20-%20Improving%20Video-Language%20Pre-training%20with%20Contrastive%20Cross-modal%20Matching%20and%20Denoising.bib).
- [2021 EMNLP] **VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding**, [[paper]](https://aclanthology.org/2021.emnlp-main.544.pdf), [[bibtex]](/Bibtex/VideoCLIP%20-%20Contrastive%20Pre-training%20for%20Zero-shot%20Video-Text%20Understanding.bib), sources: [[pytorch/fairseq/MMPT]](https://github.com/pytorch/fairseq/tree/main/examples/MMPT).
- [2021 ArXiv] **VIOLET: End-to-End Video-Language Transformers with Masked Visual-token Modeling**, [[paper]](https://arxiv.org/pdf/2111.12681.pdf), [[bibtex]](/Bibtex/VIOLET%20-%20End-to-End%20Video-Language%20Transformers%20with%20Masked%20Visual-token%20Modeling.bib), sources: [[tsujuifu/pytorch_violet]](https://github.com/tsujuifu/pytorch_violet).
- [2021 NeurIPS] **MERLOT: Multimodal Neural Script Knowledge Models**, [[paper]](https://proceedings.neurips.cc/paper/2021/file/c6d4eb15f1e84a36eff58eca3627c82e-Paper.pdf), [[bibtex]](/Bibtex/MERLOT%20-%20Multimodal%20Neural%20Script%20Knowledge%20Models.bib), [[supplementary]](https://proceedings.neurips.cc/paper/2021/file/c6d4eb15f1e84a36eff58eca3627c82e-Supplemental.pdf), [[homepage]](https://rowanzellers.com/merlot/), sources: [[rowanz/merlot]](https://github.com/rowanz/merlot).
- [2022 CVPR] **MERLOT Reserve: Multimodal Neural Script Knowledge through Vision and Language and Sound**, [[paper]](https://arxiv.org/pdf/2201.02639.pdf), [[bibtex]](/Bibtex/MERLOT%20Reserve%20-%20Multimodal%20Neural%20Script%20Knowledge%20through%20Vision%20and%20Language%20and%20Sound.bib), [[homepage]](https://rowanzellers.com/merlotreserve/), sources: [[rowanz/merlot_reserve]](https://github.com/rowanz/merlot_reserve).

## Video and Audio
- [2021 ArXiv] **Cross-Modal Attention Consistency for Video-Audio Unsupervised Learning**, [[paper]](https://arxiv.org/pdf/2106.06939.pdf), [[bibtex]](/Bibtex/Cross-Modal%20Attention%20Consistency%20for%20Video-Audio%20Unsupervised%20Learning.bib).
- [2021 NeurIPS] **Attention Bottlenecks for Multimodal Fusion**, [[paper]](https://proceedings.neurips.cc/paper/2021/file/76ba9f564ebbc35b1014ac498fafadd0-Paper.pdf), [[bibtex]](/Bibtex/Attention%20Bottlenecks%20for%20Multimodal%20Fusion.bib), [[homepage]](https://a-nagrani.github.io/mbt.html), sources: [[google-research/mbt]](https://github.com/google-research/scenic/tree/main/scenic/projects/mbt).

## Multitask
- [2021 ArXiv] **VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation**, [[paper]](https://arxiv.org/pdf/2106.04632.pdf), [[bibtex]](/Bibtex/VALUE%20-%20A%20Multi-Task%20Benchmark%20for%20Video-and-Language%20Understanding%20Evaluation.bib), [[homepage]](https://value-benchmark.github.io), sources: [[VALUE-Leaderboard/StarterCode]](https://github.com/VALUE-Leaderboard/StarterCode).