# ChemBERTa
ChemBERTa: A collection of BERT-like models applied to chemical SMILES data for drug design, chemical modelling, and property prediction. To be presented at [Baylearn](https://baylearn2020.splashthat.com/) and the [Royal Society of Chemistry's Chemical Science Symposium](https://www.rsc.org/events/detail/42791/chemical-science-symposium-2020-how-can-machine-learning-and-autonomy-accelerate-chemistry).
[Tutorial](https://github.com/deepchem/deepchem/blob/master/examples/tutorials/Transfer_Learning_With_ChemBERTa_Transformers.ipynb)
[ArXiv ChemBERTa-2 Paper](https://arxiv.org/abs/2209.01712)
[Arxiv ChemBERTa Paper](https://arxiv.org/abs/2010.09885)
[Poster](https://chemsci20.ipostersessions.com/Default.aspx?s=99-39-E6-B6-B0-0E-E1-D8-FB-66-1A-44-DC-A3-43-BA)
[Abstract](https://t.co/dkA5rMvYrE?amp=1)
[BibTex](https://scholar.googleusercontent.com/scholar.bib?q=info:FzDMp7nctLUJ:scholar.google.com/&output=citation&scisdr=CgWWnePlEM-dmmZXtDE:AAGBfm0AAAAAX5RSrDGmJTVdPMFfzRSs5UY9lD4iRvvd&scisig=AAGBfm0AAAAAX5RSrGbFzGg583aNAYQw1Lap1K79xkEm&scisf=4&ct=citation&cd=-1&hl=en)
License: MIT License
Right now the notebooks are all for the RoBERTa model (a variant of BERT) trained on the task of masked-language modelling (MLM). Training was done over 10 epochs until loss converged to around 0.26 on the ZINC 250k dataset. The model weights for ChemBERTA pre-trained on various datasets (ZINC 100k, ZINC 250k, PubChem 100k, PubChem 250k, PubChem 1M, PubChem 10M) are available using [HuggingFace](https://huggingface.co/seyonec). We expect to continue to release larger models pre-trained on even larger subsets of ZINC, CHEMBL, and PubChem in the near future.
This library is currently primarily a set of notebooks with our pre-training and fine-tuning setup, and will be updated soon with model implementation + attention visualization code, likely after the Arxiv publication. Stay tuned!
I hope this is of use to developers, students and researchers exploring the use of transformers and the attention mechanism for chemistry!
# Citing Our Work
Please cite ChemBERTa-2's [ArXiv](https://arxiv.org/abs/2209.01712) paper if you have used these models, notebooks, or examples in any way. The BibTex is available below:
```
@article{ahmad2022chemberta,
title={Chemberta-2: Towards chemical foundation models},
author={Ahmad, Walid and Simon, Elana and Chithrananda, Seyone and Grand, Gabriel and Ramsundar, Bharath},
journal={arXiv preprint arXiv:2209.01712},
year={2022}
}
```
# Example
You can load the tokenizer + model for MLM prediction tasks using the following code:
```
from transformers import AutoModelWithLMHead, AutoTokenizer, pipeline
#any model weights from the link above will work here
model = AutoModelWithLMHead.from_pretrained("seyonec/ChemBERTa-zinc-base-v1")
tokenizer = AutoTokenizer.from_pretrained("seyonec/ChemBERTa-zinc-base-v1")
fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)
```
# Todo:
- [x] Official DeepChem implementation of ChemBERTa using model API (In progress)
- [X] Open-source attention visualization suite used in paper (After formal publication - Beginning of September).
- [x] Release larger pre-trained models, and support for a wider array of property prediction tasks (BBBP, etc). - See [HuggingFace](https://huggingface.co/seyonec)
- [x] Finish writing notebook to train model
- [x] Finish notebook to preload and run predictions on a single molecule —> test if HuggingFace works
- [x] Train RoBERTa model until convergence
- [x] Upload weights onto HuggingFace
- [x] Create tutorial using evaluation + fine-tuning notebook.
- [x] Create documentation + writing, visualizations for notebook.
- [x] Setup PR into DeepChem