{
"cells": [
{
"cell_type": "markdown",
"id": "5ece8747",
"metadata": {},
"source": [
"# Lesson 55: Vision-Language Models (VLMs)\n",
"\n",
"Every classifier so far has predicted one of a small, fixed set of classes baked in at training time (Lesson {{lesson:evaluating_a_classifier}}). **CLIP** (\"Learning Transferable Visual Models From Natural Language Supervision\", Radford et al., 2021★) breaks that constraint by training an image encoder and a text encoder *jointly*, with a contrastive loss (Lesson {{lesson:self_supervised_learning}}) that pulls each image's embedding toward its caption's embedding and pushes it away from every other caption in the batch. The payoff: classification becomes a matter of writing down the class names as text prompts and checking similarity — no classifier head, no retraining, works on categories the model never saw labeled examples of."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c768b273",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import torch\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "markdown",
"id": "c20da2c0",
"metadata": {},
"source": [
"## Images paired with captions\n",
"\n",
"A tiny synthetic \"language\" (8 words) generates simple captions like \"a photo of a large plus\" or \"a photo of a small circle\" for each image. This is obviously not real natural language, but it exercises exactly the same mechanism CLIP uses on real image-caption pairs scraped from the web: every training example is an (image, text) pair, and nothing else."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6f5089d3",
"metadata": {},
"outputs": [],
"source": [
"VOCAB = ['a', 'photo', 'of', 'plus', 'circle', 'small', 'large', 'noisy']\n",
"WORD_TO_ID = {w: i for i, w in enumerate(VOCAB)}\n",
"\n",
"def make_image(shape_type, cx, cy, size, rng):\n",
" img = np.zeros((size, size), dtype=np.float32)\n",
" if shape_type == 'plus':\n",
" r = size // 5\n",
" img[cy-1:cy+2, cx-r:cx+r+1] = 1.0\n",
" img[cy-r:cy+r+1, cx-1:cx+2] = 1.0\n",
" else:\n",
" r = size // 4\n",
" yy, xx = np.mgrid[0:size, 0:size]\n",
" img[((xx-cx)**2 + (yy-cy)**2) <= r**2] = 1.0\n",
" return np.clip(img + rng.normal(0, 0.1, img.shape), 0, 1).astype(np.float32)\n",
"\n",
"def caption_for(shape_type, size_desc):\n",
" return [WORD_TO_ID[w] for w in ['a', 'photo', 'of', 'a', size_desc, shape_type]]\n",
"\n",
"def make_dataset(rng_local, n, image_size=16):\n",
" imgs, captions = [], []\n",
" for _ in range(n):\n",
" shape_type = rng_local.choice(['plus', 'circle'])\n",
" size_desc = rng_local.choice(['small', 'large'])\n",
" cx, cy = image_size // 2, image_size // 2\n",
" img = make_image(shape_type, cx, cy, image_size, rng_local)\n",
" if size_desc == 'small':\n",
" small = img[4:12, 4:12]\n",
" img = np.zeros_like(img)\n",
" img[4:12, 4:12] = small\n",
" imgs.append(img)\n",
" captions.append(caption_for(shape_type, size_desc))\n",
" return np.array(imgs, dtype=np.float32), captions\n",
"\n",
"rng = np.random.default_rng(7)\n",
"X_train, cap_train = make_dataset(rng, 400)\n",
"X_test, cap_test = make_dataset(rng, 150)\n",
"\n",
"fig, axes = plt.subplots(1, 4, figsize=(9, 2.5))\n",
"for ax, im, cap in zip(axes, X_train[:4], cap_train[:4]):\n",
" ax.imshow(im, cmap='gray'); ax.axis('off')\n",
" ax.set_title(' '.join(VOCAB[w] for w in cap), fontsize=7)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "1212bcf4",
"metadata": {},
"source": "## Two encoders, one shared embedding space\n\nAn image encoder (Lesson {{lesson:vision_transformers}}'s ViT pattern: patchify, embed, a small Transformer encoder) and a text encoder (a small Transformer encoder over the caption's tokens, Lesson {{lesson:transformer_architecture}}) both map their very different inputs into the *same* fixed-size vector space — matching real CLIP's own Transformer-based design. (Real CLIP has also shipped CNN/ResNet image-encoder variants, which work too; Transformer is used here to stay consistent with this part of the course.) Training pulls a batch's matching (image, caption) pairs together and pushes every mismatched pair apart — precisely Lesson {{lesson:self_supervised_learning}}'s `nt_xent_loss`, just applied across two different modalities instead of two augmented views of one image."
},
{
"cell_type": "code",
"execution_count": null,
"id": "68714171",
"metadata": {},
"outputs": [],
"source": "def patchify(img, patch_size):\n B, C, H, W = img.shape\n P = patch_size\n patches = img.unfold(2, P, P).unfold(3, P, P)\n patches = patches.contiguous().view(B, C, -1, P, P).permute(0, 2, 1, 3, 4)\n return patches.reshape(B, -1, C * P * P)\n\ndef positional_encoding(T, D):\n pos = torch.arange(T).unsqueeze(1).float()\n i = torch.arange(D).unsqueeze(0).float()\n angle_rates = 1.0 / (10000 ** (2 * (i // 2) / D))\n angles = pos * angle_rates\n pe = torch.zeros(T, D)\n pe[:, 0::2] = torch.sin(angles[:, 0::2])\n pe[:, 1::2] = torch.cos(angles[:, 1::2])\n return pe\n\nclass ImageEncoder(nn.Module):\n \"\"\"A small ViT (Lesson {{lesson:vision_transformers}}'s pattern): patchify, linearly\n embed, prepend a [CLS] token, run a Transformer encoder, read the [CLS] token's output.\"\"\"\n def __init__(self, img_size=16, patch_size=4, out_dim=16, embed_dim=24, n_heads=4, n_layers=2):\n super().__init__()\n self.patch_size = patch_size\n n_patches = (img_size // patch_size) ** 2\n patch_dim = patch_size * patch_size\n self.embed = nn.Linear(patch_dim, embed_dim)\n self.cls_token = nn.Parameter(torch.randn(1, 1, embed_dim) * 0.02)\n self.register_buffer('pos_enc', positional_encoding(n_patches + 1, embed_dim))\n layer = nn.TransformerEncoderLayer(embed_dim, n_heads, dim_feedforward=embed_dim * 2,\n batch_first=True, dropout=0.0)\n self.encoder = nn.TransformerEncoder(layer, num_layers=n_layers)\n self.proj = nn.Linear(embed_dim, out_dim)\n\n def forward(self, x):\n p = patchify(x, self.patch_size)\n tok = self.embed(p)\n cls = self.cls_token.expand(x.shape[0], -1, -1)\n tok = torch.cat([cls, tok], dim=1) + self.pos_enc\n out = self.encoder(tok)\n return self.proj(out[:, 0])\n\nclass TextEncoder(nn.Module):\n \"\"\"A small bidirectional Transformer encoder over the caption's word tokens, pooled by\n averaging the final token representations -- unlike a bag-of-words mean, these token\n representations have already mixed in positional and attention information, so the\n pooled result is sensitive to word order.\"\"\"\n def __init__(self, vocab_size, out_dim=16, embed_dim=16, n_heads=4, n_layers=1, max_len=6):\n super().__init__()\n self.embed = nn.Embedding(vocab_size, embed_dim)\n self.register_buffer('pos_enc', positional_encoding(max_len, embed_dim))\n layer = nn.TransformerEncoderLayer(embed_dim, n_heads, dim_feedforward=embed_dim * 2,\n batch_first=True, dropout=0.0)\n self.encoder = nn.TransformerEncoder(layer, num_layers=n_layers)\n self.proj = nn.Linear(embed_dim, out_dim)\n\n def forward(self, token_ids):\n x = self.embed(token_ids) + self.pos_enc[:token_ids.shape[1]]\n x = self.encoder(x)\n return self.proj(x.mean(dim=1))\n\ndef clip_loss(img_emb, txt_emb, temperature=0.1):\n img_emb = F.normalize(img_emb, dim=1)\n txt_emb = F.normalize(txt_emb, dim=1)\n logits = img_emb @ txt_emb.T / temperature # (batch, batch) similarity matrix\n targets = torch.arange(img_emb.shape[0]) # the diagonal is the true pairing\n loss_i2t = F.cross_entropy(logits, targets) # \"which caption matches this image?\"\n loss_t2i = F.cross_entropy(logits.T, targets) # \"which image matches this caption?\"\n return (loss_i2t + loss_t2i) / 2\n\ndef train_clip(seed, epochs=300, lr=0.01, batch_size=64):\n torch.manual_seed(seed)\n img_enc = ImageEncoder()\n txt_enc = TextEncoder(len(VOCAB))\n opt = torch.optim.Adam(list(img_enc.parameters()) + list(txt_enc.parameters()), lr=lr)\n caps_tensor = torch.tensor(cap_train)\n n = len(X_train)\n for epoch in range(epochs):\n idx = np.random.default_rng(epoch).permutation(n)[:batch_size]\n imgs = torch.tensor(X_train[idx]).unsqueeze(1)\n caps = caps_tensor[idx]\n loss = clip_loss(img_enc(imgs), txt_enc(caps))\n opt.zero_grad()\n loss.backward()\n opt.step()\n return img_enc, txt_enc, loss.item()\n\nimg_enc, txt_enc, final_loss = train_clip(seed=0)\nprint(f'final CLIP loss: {final_loss:.3f}')"
},
{
"cell_type": "markdown",
"id": "89af1814",
"metadata": {},
"source": [
"## Zero-shot classification\n",
"\n",
"No classifier head was ever trained. To classify a test image, embed a text prompt for each candidate class (\"a photo of a large plus\", \"a photo of a large circle\") and pick whichever prompt's embedding is most similar to the image's embedding — the same nearest-neighbor-in-embedding-space idea as Lesson {{lesson:attention_mechanism}}'s attention retrieval demo, just across modalities."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bd396433",
"metadata": {},
"outputs": [],
"source": [
"prompts = {'plus': caption_for('plus', 'large'), 'circle': caption_for('circle', 'large')}\n",
"plus_id, circle_id = WORD_TO_ID['plus'], WORD_TO_ID['circle']\n",
"true_labels = [c[-1] for c in cap_test] # last word id encodes the true shape\n",
"X_test_t = torch.tensor(X_test).unsqueeze(1)\n",
"\n",
"def zero_shot_eval(img_encoder, txt_encoder):\n",
" with torch.no_grad():\n",
" prompt_embs = {k: F.normalize(txt_encoder(torch.tensor([v])), dim=1) for k, v in prompts.items()}\n",
" img_embs = F.normalize(img_encoder(X_test_t), dim=1)\n",
" sims_plus = (img_embs @ prompt_embs['plus'].T).squeeze(-1)\n",
" sims_circle = (img_embs @ prompt_embs['circle'].T).squeeze(-1)\n",
" preds = torch.where(sims_plus > sims_circle, plus_id, circle_id)\n",
" return sum(p.item() == t for p, t in zip(preds, true_labels)) / len(true_labels)\n",
"\n",
"acc_trained = zero_shot_eval(img_enc, txt_enc)\n",
"\n",
"torch.manual_seed(0) # same init recipe, but never trained -- the baseline\n",
"untrained_img_enc = ImageEncoder()\n",
"untrained_txt_enc = TextEncoder(len(VOCAB))\n",
"acc_untrained = zero_shot_eval(untrained_img_enc, untrained_txt_enc)\n",
"\n",
"print(f'zero-shot accuracy, trained encoders: {acc_trained:.1%}')\n",
"print(f'zero-shot accuracy, untrained encoders: {acc_untrained:.1%} (baseline)')"
]
},
{
"cell_type": "markdown",
"id": "07005024",
"metadata": {},
"source": "Trained encoders solve zero-shot classification perfectly; untrained ones are at chance — the contrastive image-text objective, not the specific classifier, is doing all the work. This is the actual mechanism behind real CLIP's headline result: it can classify into *any* set of categories describable in words, including ones that never appeared as a labeled class during training, simply by changing the text prompts at inference time. It's also the standard way to build a **vision-language model (VLM)**'s visual front end — a frozen CLIP-style image encoder feeding into a language model is the basis for systems that can answer questions about images, and the same paired image-text contrastive objective, applied to region proposals instead of whole images, is one route to open-vocabulary detection and segmentation (Lesson {{lesson:open_vocabulary_detection}}).\n\n### Exercises\n\n1. Add a third shape category (e.g. \"square\", following Lesson {{lesson:evaluating_a_classifier}}'s pattern) to `make_dataset`, retrain, and add a `'square'` prompt to zero-shot evaluation. Does 3-way zero-shot classification still work as well as the 2-way case?\n2. Try a prompt using a word combination *never seen together* in training — e.g. if training captions only ever paired `'noisy'` with neither shape, construct `caption_for` variants that use `'noisy'` and check whether the model's similarity ranking still makes sense. What does this test about whether the model learned compositional word meaning versus caption memorization?\n3. Unlike a bag-of-words mean, this lesson's `TextEncoder` has both positional encoding and self-attention, so `'a large plus'` and `'plus large a'` are *not* guaranteed to produce the same embedding. Verify this directly: feed the encoder the same caption's words in their normal order and in a scrambled order, and compare the two embeddings. Then check whether this order-sensitivity actually changes zero-shot classification on this dataset — since every training caption always follows the exact same template (`'a photo of a '`), does the model ever have a reason to *use* word order, even though it technically could?"
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}