{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": "# Lesson 49: Vision-Language Models\n\nEvery classifier so far has predicted one of a small, fixed set of classes baked in at training time (Lesson 37). **CLIP** (\"Learning Transferable Visual Models From Natural Language Supervision\", Radford et al., 2021★) breaks that constraint by training an image encoder and a text encoder *jointly*, with a contrastive loss (Lesson 47) that pulls each image's embedding toward its caption's embedding and pushes it away from every other caption in the batch. The payoff: classification becomes a matter of writing down the class names as text prompts and checking similarity — no classifier head, no retraining, works on categories the model never saw labeled examples of."
},
{
"cell_type": "code",
"id": "c768b273",
"source": "import numpy as np\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport matplotlib.pyplot as plt",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "c20da2c0",
"source": "## Images paired with captions\n\nA tiny synthetic \"language\" (8 words) generates simple captions like \"a photo of a large plus\" or \"a photo of a small circle\" for each image. This is obviously not real natural language, but it exercises exactly the same mechanism CLIP uses on real image-caption pairs scraped from the web: every training example is an (image, text) pair, and nothing else.",
"metadata": {}
},
{
"cell_type": "code",
"id": "6f5089d3",
"source": "VOCAB = ['a', 'photo', 'of', 'plus', 'circle', 'small', 'large', 'noisy']\nWORD_TO_ID = {w: i for i, w in enumerate(VOCAB)}\n\ndef make_image(shape_type, cx, cy, size, rng):\n img = np.zeros((size, size), dtype=np.float32)\n if shape_type == 'plus':\n r = size // 5\n img[cy-1:cy+2, cx-r:cx+r+1] = 1.0\n img[cy-r:cy+r+1, cx-1:cx+2] = 1.0\n else:\n r = size // 4\n yy, xx = np.mgrid[0:size, 0:size]\n img[((xx-cx)**2 + (yy-cy)**2) <= r**2] = 1.0\n return np.clip(img + rng.normal(0, 0.1, img.shape), 0, 1).astype(np.float32)\n\ndef caption_for(shape_type, size_desc):\n return [WORD_TO_ID[w] for w in ['a', 'photo', 'of', 'a', size_desc, shape_type]]\n\ndef make_dataset(rng_local, n, image_size=16):\n imgs, captions = [], []\n for _ in range(n):\n shape_type = rng_local.choice(['plus', 'circle'])\n size_desc = rng_local.choice(['small', 'large'])\n cx, cy = image_size // 2, image_size // 2\n img = make_image(shape_type, cx, cy, image_size, rng_local)\n if size_desc == 'small':\n small = img[4:12, 4:12]\n img = np.zeros_like(img)\n img[4:12, 4:12] = small\n imgs.append(img)\n captions.append(caption_for(shape_type, size_desc))\n return np.array(imgs, dtype=np.float32), captions\n\nrng = np.random.default_rng(7)\nX_train, cap_train = make_dataset(rng, 400)\nX_test, cap_test = make_dataset(rng, 150)\n\nfig, axes = plt.subplots(1, 4, figsize=(9, 2.5))\nfor ax, im, cap in zip(axes, X_train[:4], cap_train[:4]):\n ax.imshow(im, cmap='gray'); ax.axis('off')\n ax.set_title(' '.join(VOCAB[w] for w in cap), fontsize=7)\nplt.show()",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "1212bcf4",
"source": "## Two encoders, one shared embedding space\n\nAn image encoder (Lesson 33's CNN pattern) and a text encoder (a word-embedding lookup, mean-pooled over the caption — the simplest possible text encoder; real CLIP uses a Transformer, Lesson 45) both map their very different inputs into the *same* fixed-size vector space. Training pulls a batch's matching (image, caption) pairs together and pushes every mismatched pair apart — precisely Lesson 47's `nt_xent_loss`, just applied across two different modalities instead of two augmented views of one image.",
"metadata": {}
},
{
"cell_type": "code",
"id": "68714171",
"source": "class ImageEncoder(nn.Module):\n def __init__(self, out_dim=16):\n super().__init__()\n self.conv = nn.Sequential(\n nn.Conv2d(1, 16, 5, padding=2), nn.ReLU(), nn.MaxPool2d(2),\n nn.Conv2d(16, 32, 5, padding=2), nn.ReLU(), nn.AdaptiveMaxPool2d(1),\n )\n self.proj = nn.Linear(32, out_dim)\n\n def forward(self, x):\n return self.proj(self.conv(x).flatten(1))\n\nclass TextEncoder(nn.Module):\n def __init__(self, vocab_size, out_dim=16, embed_dim=8):\n super().__init__()\n self.embed = nn.Embedding(vocab_size, embed_dim)\n self.proj = nn.Linear(embed_dim, out_dim)\n\n def forward(self, token_ids):\n return self.proj(self.embed(token_ids).mean(dim=1)) # bag-of-words text encoder\n\ndef clip_loss(img_emb, txt_emb, temperature=0.1):\n img_emb = F.normalize(img_emb, dim=1)\n txt_emb = F.normalize(txt_emb, dim=1)\n logits = img_emb @ txt_emb.T / temperature # (batch, batch) similarity matrix\n targets = torch.arange(img_emb.shape[0]) # the diagonal is the true pairing\n loss_i2t = F.cross_entropy(logits, targets) # \"which caption matches this image?\"\n loss_t2i = F.cross_entropy(logits.T, targets) # \"which image matches this caption?\"\n return (loss_i2t + loss_t2i) / 2\n\ndef train_clip(seed, epochs=300, lr=0.01, batch_size=64):\n torch.manual_seed(seed)\n img_enc = ImageEncoder()\n txt_enc = TextEncoder(len(VOCAB))\n opt = torch.optim.Adam(list(img_enc.parameters()) + list(txt_enc.parameters()), lr=lr)\n caps_tensor = torch.tensor(cap_train)\n n = len(X_train)\n for epoch in range(epochs):\n idx = np.random.default_rng(epoch).permutation(n)[:batch_size]\n imgs = torch.tensor(X_train[idx]).unsqueeze(1)\n caps = caps_tensor[idx]\n loss = clip_loss(img_enc(imgs), txt_enc(caps))\n opt.zero_grad()\n loss.backward()\n opt.step()\n return img_enc, txt_enc, loss.item()\n\nimg_enc, txt_enc, final_loss = train_clip(seed=0)\nprint(f'final CLIP loss: {final_loss:.3f}')",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "89af1814",
"source": "## Zero-shot classification\n\nNo classifier head was ever trained. To classify a test image, embed a text prompt for each candidate class (\"a photo of a large plus\", \"a photo of a large circle\") and pick whichever prompt's embedding is most similar to the image's embedding — the same nearest-neighbor-in-embedding-space idea as Lesson 44's attention retrieval demo, just across modalities.",
"metadata": {}
},
{
"cell_type": "code",
"id": "bd396433",
"source": "prompts = {'plus': caption_for('plus', 'large'), 'circle': caption_for('circle', 'large')}\nplus_id, circle_id = WORD_TO_ID['plus'], WORD_TO_ID['circle']\ntrue_labels = [c[-1] for c in cap_test] # last word id encodes the true shape\nX_test_t = torch.tensor(X_test).unsqueeze(1)\n\ndef zero_shot_eval(img_encoder, txt_encoder):\n with torch.no_grad():\n prompt_embs = {k: F.normalize(txt_encoder(torch.tensor([v])), dim=1) for k, v in prompts.items()}\n img_embs = F.normalize(img_encoder(X_test_t), dim=1)\n sims_plus = (img_embs @ prompt_embs['plus'].T).squeeze(-1)\n sims_circle = (img_embs @ prompt_embs['circle'].T).squeeze(-1)\n preds = torch.where(sims_plus > sims_circle, plus_id, circle_id)\n return sum(p.item() == t for p, t in zip(preds, true_labels)) / len(true_labels)\n\nacc_trained = zero_shot_eval(img_enc, txt_enc)\n\ntorch.manual_seed(0) # same init recipe, but never trained -- the baseline\nuntrained_img_enc = ImageEncoder()\nuntrained_txt_enc = TextEncoder(len(VOCAB))\nacc_untrained = zero_shot_eval(untrained_img_enc, untrained_txt_enc)\n\nprint(f'zero-shot accuracy, trained encoders: {acc_trained:.1%}')\nprint(f'zero-shot accuracy, untrained encoders: {acc_untrained:.1%} (baseline)')",
"metadata": {},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"id": "07005024",
"source": "Trained encoders solve zero-shot classification perfectly; untrained ones are at chance — the contrastive image-text objective, not the specific classifier, is doing all the work. This is the actual mechanism behind real CLIP's headline result: it can classify into *any* set of categories describable in words, including ones that never appeared as a labeled class during training, simply by changing the text prompts at inference time. It's also the standard way to build a **vision-language model (VLM)**'s visual front end — a frozen CLIP-style image encoder feeding into a language model is the basis for systems that can answer questions about images, and the same paired image-text contrastive objective, applied to region proposals instead of whole images, is one route to open-vocabulary detection and segmentation (Lesson 52).\n\n### Exercise\n\n1. Add a third shape category (e.g. \"square\", following Lesson 37's pattern) to `make_dataset`, retrain, and add a `'square'` prompt to zero-shot evaluation. Does 3-way zero-shot classification still work as well as the 2-way case?\n2. Try a prompt using a word combination *never seen together* in training — e.g. if training captions only ever paired `'noisy'` with neither shape, construct `caption_for` variants that use `'noisy'` and check whether the model's similarity ranking still makes sense. What does this test about whether the model learned compositional word meaning versus caption memorization?\n3. This lesson's text encoder mean-pools word embeddings, ignoring word order entirely (`'a large plus'` and `'plus large a'` produce the identical embedding). Would that make a difference for these particular captions? Sketch a change (hint: Lesson 45) that would make the text encoder order-sensitive, and describe a caption pair where order-sensitivity would actually matter.",
"metadata": {}
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}