{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Laya vs Laya-Multilingual \u2014 extensive head-to-head benchmark\n", "\n", "Runs both open checkpoints over the same questions and writes a single\n", "`laya_benchmark_results.json`.\n", "\n", "| | `convaiinnovations/laya` | `convaiinnovations/laya-multilingual` |\n", "|---|---|---|\n", "| encoder | ModernBERT-large (English) | mmBERT-base (100+ languages) |\n", "| params | 421.3M (394.8M enc + 26.5M head) | 321.9M (306.9M enc + 15.0M head) |\n", "| context | 512 tokens | 1024 tokens |\n", "| shipped temperatures | fitted per type & option-count | **none (all 1.0)** |\n", "\n", "**What is measured**\n", "\n", "1. **typed-decisions** \u2014 400 cases / 2,000 decisions, 4 workflows (the benchmark behind the published 0.766 claim). Zero-shot for both of these checkpoints.\n", "2. **Multilingual** \u2014 MASSIVE intent + scenario across 14 languages, XNLI across 15 languages.\n", "3. **English zero-shot** \u2014 SST-5, emotion, prompt-injections, plus banking77 as a 77-option stress test.\n", "4. **Latency** \u2014 p50/p95 at 1 / 5 / 10 / 50 questions per call.\n", "5. **Option-order robustness** \u2014 how often the answer flips when the options are permuted (the independent Jev benchmark measured 13% for Jev; worst LLM 37%).\n", "6. **Calibration repair** \u2014 ECE before and after refitting temperature, which matters because `laya-multilingual` shipped uncalibrated.\n", "\n", "**Setup:** Runtime \u2192 Change runtime type \u2192 **T4 GPU**. Then Run All (~25\u201340 min).\n", "Both models see byte-identical questions (fixed seed), so every difference is the model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. GPU and environment" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import subprocess, sys, os, json, time\n", "print(subprocess.run([\"nvidia-smi\",\"--query-gpu=name,memory.total,compute_cap\",\"--format=csv,noheader\"],\n", " capture_output=True, text=True).stdout.strip() or \"NO GPU - set Runtime > Change runtime type > T4 GPU\")\n", "import torch\n", "print(\"torch\", torch.__version__, \"| cuda\", torch.cuda.is_available(),\n", " \"| device\", torch.cuda.get_device_name(0) if torch.cuda.is_available() else \"cpu\")\n", "assert torch.cuda.is_available(), \"Enable the T4 GPU runtime before running.\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Dependencies" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip -q install -U \"laya>=0.1.6\" \"transformers>=4.45\" \"datasets>=3.0\" safetensors huggingface_hub 2>&1 | tail -3\n", "import laya, transformers, datasets\n", "print(\"laya\", getattr(laya, \"__version__\", \"?\"), \"| transformers\", transformers.__version__, \"| datasets\", datasets.__version__)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Download both checkpoints\n", "\n", "`reference_compile` is forced off after loading: the `laya` package does not disable it (only\n", "`rl_agent_api.py` does), and torch.compile is a loss on small batches / few SMs like a T4." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import torch, json, os, time\n", "from huggingface_hub import snapshot_download\n", "from safetensors import safe_open\n", "\n", "REPOS = {\"laya\": \"convaiinnovations/laya\", \"laya-multilingual\": \"convaiinnovations/laya-multilingual\"}\n", "PATHS, MODEL_META = {}, {}\n", "\n", "def patch_tokenizer_config(model_dir):\n", " '''laya-multilingual ships extra_special_tokens as a LIST (inherited from the mmBERT/Gemma\n", " tokenizer); transformers expects a dict and raises\n", " AttributeError: list object has no attribute keys. Without this, AutoTokenizer -- and so\n", " laya.load() -- fails outright on that repo. Also normalises tokenizer_class for 4.x/5.x.'''\n", " p = os.path.join(model_dir, \"tokenizer\", \"tokenizer_config.json\")\n", " if not os.path.exists(p):\n", " return []\n", " c = json.load(open(p))\n", " changed = []\n", " est = c.get(\"extra_special_tokens\")\n", " if isinstance(est, list):\n", " c[\"extra_special_tokens\"] = {(\"extra_%d\" % i): t for i, t in enumerate(est)}\n", " changed.append(\"extra_special_tokens list->dict\")\n", " if c.get(\"tokenizer_class\") in (None, \"TokenizersBackend\"):\n", " c[\"tokenizer_class\"] = \"PreTrainedTokenizerFast\"\n", " c.pop(\"backend\", None); c.pop(\"is_local\", None)\n", " changed.append(\"tokenizer_class\")\n", " if changed:\n", " json.dump(c, open(p, \"w\"), indent=2)\n", " return changed\n", "\n", "for name, repo in REPOS.items():\n", " t = time.time()\n", " PATHS[name] = snapshot_download(repo)\n", " fixed = patch_tokenizer_config(PATHS[name])\n", " if fixed:\n", " print(\" patched tokenizer config for %s: %s\" % (name, \", \".join(fixed)))\n", " cfg = json.load(open(os.path.join(PATHS[name], \"rl_agent_config.json\")))\n", " ecfg = json.load(open(os.path.join(PATHS[name], \"encoder\", \"config.json\")))\n", " tot = enc = 0\n", " with safe_open(os.path.join(PATHS[name], \"model.safetensors\"), \"pt\") as f:\n", " for k in f.keys():\n", " n = 1\n", " for d in f.get_slice(k).get_shape():\n", " n *= d\n", " tot += n\n", " enc += n if k.startswith(\"encoder.\") else 0\n", " MODEL_META[name] = {\n", " \"repo\": repo, \"params_total_m\": round(tot/1e6, 2), \"params_encoder_m\": round(enc/1e6, 2),\n", " \"params_head_m\": round((tot-enc)/1e6, 2), \"hidden_size\": ecfg.get(\"hidden_size\"),\n", " \"num_layers\": ecfg.get(\"num_hidden_layers\"), \"vocab_size\": ecfg.get(\"vocab_size\"),\n", " \"max_len\": cfg.get(\"max_len\"), \"head_max_len\": cfg.get(\"head_max_len\"),\n", " \"temperature\": cfg.get(\"temperature\"), \"temperature_by_options\": cfg.get(\"temperature_by_options\", {}),\n", " \"training\": cfg.get(\"training\", {}), \"download_s\": round(time.time()-t, 1),\n", " \"tokenizer_config_patched\": fixed,\n", " }\n", " print(name, \"->\", PATHS[name])\n", " print(\" \", json.dumps({k: v for k, v in MODEL_META[name].items() if k != \"temperature_by_options\"}))\n", "\n", "def load_agent(name):\n", " import laya\n", " ag = laya.load(PATHS[name], device=\"cuda\")\n", " try:\n", " ag.model.encoder.config.reference_compile = False # torch.compile hurts on T4-class GPUs\n", " except Exception as e:\n", " print(\"could not disable reference_compile:\", e)\n", " ag.model.eval()\n", " return ag" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Benchmark engine \u2014 batched scoring and metrics" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%writefile bench_engine.py\n", "import json, math, sys, time\n", "import numpy as np\n", "import torch\n", "from laya.common import QTYPES, build_sequence, collate_items, render_options, temp_bucket\n", "\n", "def to_internal(qdef):\n", " t = qdef[\"type\"]\n", " crit = qdef.get(\"criteria\")\n", " if t == \"choice\" and isinstance(crit, list):\n", " crit = {c: None for c in crit}\n", " ins = qdef[\"instructions\"]\n", " if not isinstance(ins, str):\n", " ins = json.dumps(ins)\n", " return {\"t\": t, \"ins\": ins, \"crit\": crit}\n", "\n", "@torch.no_grad()\n", "def score_cases(agent, cases, max_tokens=16384, max_seqs=128, progress=\"\"):\n", " '''cases: [(state, {qid: qdef})] -> (raw_logits per question, index, seconds, n_dropped).\n", "\n", " Every question of every case is one sequence. Sequences are length-sorted and packed into\n", " token-budgeted batches so padding stays small. Returns UNCALIBRATED logits; temperature is\n", " applied later so raw/calibrated variants come from one forward pass.\n", " '''\n", " max_len = agent.cfg.get(\"max_len\", 512)\n", " head_max_len = agent.cfg.get(\"head_max_len\", 192)\n", " items, index, dropped = [], [], 0\n", " for ci, (state, questions) in enumerate(cases):\n", " for qid, qdef in questions.items():\n", " q = to_internal(qdef)\n", " try:\n", " ids, markers = build_sequence(agent.tok, state, q, max_len, head_max_len)\n", " except Exception:\n", " index.append((ci, qid, QTYPES[q[\"t\"]], 0)); items.append(None); dropped += 1\n", " continue\n", " if len(markers) != len(render_options(q)):\n", " index.append((ci, qid, QTYPES[q[\"t\"]], 0)); items.append(None); dropped += 1\n", " continue\n", " items.append({\"ids\": ids, \"markers\": markers, \"qtype\": QTYPES[q[\"t\"]]})\n", " index.append((ci, qid, QTYPES[q[\"t\"]], len(markers)))\n", "\n", " valid = [i for i, it in enumerate(items) if it is not None]\n", " order = sorted(valid, key=lambda i: len(items[i][\"ids\"]))\n", " out = [None] * len(items)\n", " t0, done, i = time.time(), 0, 0\n", " while i < len(order):\n", " j, L = i, 0\n", " while j < len(order) and j - i < max_seqs and max(L, len(items[order[j]][\"ids\"])) * (j - i + 1) <= max_tokens:\n", " L = max(L, len(items[order[j]][\"ids\"])); j += 1\n", " j = max(j, i + 1)\n", " sel = [items[order[t]] for t in range(i, j)]\n", " b = collate_items([sel], agent.tok.pad_token_id)\n", " with torch.autocast(\"cuda\", dtype=agent.dtype, enabled=True):\n", " logits, _ = agent.model(b[\"input_ids\"].cuda(), b[\"attention_mask\"].cuda(),\n", " b[\"marker_pos\"].cuda(), b[\"marker_mask\"].cuda(), b[\"qtype\"].cuda())\n", " logits = logits.float().cpu().numpy()\n", " for r in range(j - i):\n", " out[order[i + r]] = logits[r, :len(sel[r][\"markers\"])]\n", " done += j - i\n", " if progress:\n", " el = time.time() - t0\n", " msg = \"\\r [%s] %d/%d seq | %.0f seq/s | ETA %ds \" % (\n", " progress, done, len(order), done/max(el, 1e-9),\n", " int(el*(len(order)-done)/max(1, done)))\n", " sys.stdout.write(msg)\n", " sys.stdout.flush()\n", " i = j\n", " if progress:\n", " print(\"\\r [%s] %d sequences in %.1fs (%d dropped: options did not fit)%s\" % (\n", " progress, len(order), time.time()-t0, dropped, \" \"*20))\n", " return out, index, time.time() - t0, dropped\n", "\n", "def softmax_t(z, t=1.0):\n", " z = np.asarray(z, dtype=np.float64) / max(1e-3, float(t))\n", " e = np.exp(z - z.max())\n", " return e / e.sum()\n", "\n", "def temp_for(agent, qt, k, calibrated=True):\n", " if not calibrated:\n", " return 1.0\n", " return float(agent.temperature_by_options.get(temp_bucket(qt, k), agent.temperature[qt]))\n", "\n", "# ------------------------------------------------------------------ metrics\n", "def ece_score(conf, correct, bins=15):\n", " conf, correct = np.asarray(conf, float), np.asarray(correct, float)\n", " if len(conf) == 0:\n", " return float(\"nan\")\n", " edges = np.linspace(0, 1, bins + 1)\n", " e = 0.0\n", " for lo, hi in zip(edges[:-1], edges[1:]):\n", " sel = (conf > lo) & (conf <= hi)\n", " if sel.any():\n", " e += sel.mean() * abs(conf[sel].mean() - correct[sel].mean())\n", " return float(e)\n", "\n", "def macro_f1(gold, pred):\n", " gold, pred = np.asarray(gold), np.asarray(pred)\n", " f1 = []\n", " for c in sorted(set(gold.tolist()) | set(pred.tolist())):\n", " tp = int(((pred == c) & (gold == c)).sum())\n", " fp = int(((pred == c) & (gold != c)).sum())\n", " fn = int(((pred != c) & (gold == c)).sum())\n", " f1.append(2*tp / max(1, 2*tp + fp + fn))\n", " return float(np.mean(f1))\n", "\n", "def aurc(conf, correct):\n", " conf, correct = np.asarray(conf, float), np.asarray(correct, float)\n", " if len(conf) == 0:\n", " return float(\"nan\")\n", " o = np.argsort(-conf)\n", " return float((np.cumsum(1 - correct[o]) / np.arange(1, len(o)+1)).mean())\n", "\n", "def hard_metrics(rows):\n", " '''rows: [(gold_idx, probs)] -> standard single-label classification + calibration metrics.'''\n", " rows = [r for r in rows if r[1] is not None]\n", " if not rows:\n", " return {\"n\": 0}\n", " gold = np.array([g for g, _ in rows])\n", " pred = np.array([int(np.argmax(p)) for _, p in rows])\n", " conf = np.array([float(np.max(p)) for _, p in rows])\n", " corr = (pred == gold).astype(float)\n", " m = {\"n\": len(rows),\n", " \"accuracy\": float(corr.mean()),\n", " \"macro_f1\": macro_f1(gold, pred),\n", " \"ece\": ece_score(conf, corr),\n", " \"brier\": float(np.mean([((np.asarray(p) - np.eye(len(p))[g])**2).sum() for g, p in rows])),\n", " \"nll\": float(np.mean([-math.log(max(float(p[g]), 1e-12)) for g, p in rows])),\n", " \"aurc\": aurc(conf, corr),\n", " \"mean_confidence\": float(conf.mean())}\n", " for cov in (0.5, 0.8):\n", " k = max(1, int(len(conf)*cov))\n", " m[\"acc_at_%d_coverage\" % int(cov*100)] = float(corr[np.argsort(-conf)[:k]].mean())\n", " return m\n", "\n", "def fit_temperature(pairs, lo=0.2, hi=10.0, steps=160):\n", " '''pairs: [(logits, gold_idx)] -> single temperature minimising NLL (grid search, no autograd).'''\n", " pairs = [(np.asarray(z, float), int(g)) for z, g in pairs if z is not None]\n", " if len(pairs) < 25:\n", " return 1.0\n", " best_t, best = 1.0, float(\"inf\")\n", " for t in np.geomspace(lo, hi, steps):\n", " tot = 0.0\n", " for z, g in pairs:\n", " zz = z/t\n", " zz = zz - zz.max()\n", " tot += -(zz[g] - math.log(np.exp(zz).sum()))\n", " if tot < best:\n", " best, best_t = tot, float(t)\n", " return round(float(best_t), 4)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Task suites\n", "\n", "Every suite is built **once**, with a fixed seed, and reused for both models \u2014 so the two\n", "checkpoints answer byte-identical questions. Option sets for many-label tasks use\n", "`gold + N-1 distractors`, matching how the models were trained (the training recipe samples\n", "5\u201320 options per choice question rather than showing the full label space)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import random, json\n", "import numpy as np\n", "from datasets import load_dataset\n", "\n", "SEED = 13\n", "N_OPTS = 20 # options per many-label choice question (gold + 19 distractors)\n", "PER_LANG = 300 # cases per language\n", "SUITES = {} # name -> {\"cases\": [(state, questions)], \"gold\": [{qid: gold_idx}], \"meta\": {...}}\n", "\n", "def build_choice_questions(rng, gold, all_labels, n_opts, instructions, qid=\"label\"):\n", " pool = [x for x in all_labels if x != gold]\n", " keys = [gold] + rng.sample(pool, min(n_opts-1, len(pool)))\n", " rng.shuffle(keys)\n", " crit = {k: k.replace(\"_\", \" \").replace(\".\", \": \") for k in keys}\n", " return {qid: {\"type\": \"choice\", \"instructions\": instructions, \"criteria\": crit}}, keys.index(gold)\n", "\n", "def register(name, cases, golds, **meta):\n", " SUITES[name] = {\"cases\": cases, \"gold\": golds, \"meta\": meta}\n", " nq = sum(len(c[1]) for c in cases)\n", " print(\" %-34s %4d cases / %5d questions %s\" % (name, len(cases), nq, meta.get(\"note\", \"\")))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5a. typed-decisions (English, 400 cases / 2,000 decisions)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "td_cases, td_golds, td_wf = [], [], []\n", "try:\n", " td = load_dataset(\"LocalLLaMA/typed-decisions\", \"all\", split=\"test\")\n", " for r in td:\n", " questions = json.loads(r[\"questions\"]) if isinstance(r[\"questions\"], str) else r[\"questions\"]\n", " gold = json.loads(r[\"gold\"]) if isinstance(r[\"gold\"], str) else r[\"gold\"]\n", " state = r[\"state\"]\n", " try:\n", " state = json.loads(state)\n", " except Exception:\n", " pass\n", " gmap = {}\n", " for qid, qdef in questions.items():\n", " g = gold[qid]\n", " if qdef[\"type\"] == \"choice\":\n", " keys = list(qdef[\"criteria\"].keys())\n", " gmap[qid] = {\"idx\": keys.index(str(g[\"label\"])), \"keys\": keys,\n", " \"soft\": [float(g.get(\"probabilities\", {}).get(k, 0.0)) for k in keys]}\n", " elif qdef[\"type\"] == \"noul\":\n", " idx = 1 if str(g[\"label\"]).lower() == \"true\" else 0\n", " pt = float(g.get(\"probabilities\", {}).get(\"true\", g.get(\"noul\", 0.5)))\n", " gmap[qid] = {\"idx\": idx, \"keys\": [\"false\", \"true\"], \"soft\": [1-pt, pt]}\n", " else: # score\n", " n = len(qdef[\"criteria\"])\n", " gmap[qid] = {\"idx\": int(g[\"label\"]), \"keys\": [str(i) for i in range(n)],\n", " \"soft\": [float(g.get(\"probabilities\", {}).get(str(i), 0.0)) for i in range(n)],\n", " \"gold_score\": float(g.get(\"score\", float(g[\"label\"])))}\n", " td_cases.append((state, questions))\n", " td_golds.append(gmap)\n", " td_wf.append(r[\"workflow\"])\n", " register(\"typed_decisions\", td_cases, td_golds, note=\"4 workflows, soft teacher labels\",\n", " workflows=td_wf, soft=True)\n", "except Exception as e:\n", " print(\"typed-decisions FAILED:\", type(e).__name__, e)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5b. Multilingual \u2014 MASSIVE intent & scenario (14 languages), XNLI (15 languages)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "MASSIVE_LANGS = [\"en\",\"de\",\"fr\",\"es\",\"pt\",\"ru\",\"tr\",\"ar\",\"hi\",\"ta\",\"zh-CN\",\"ja\",\"ko\",\"sw\"]\n", "XNLI_LANGS = [\"en\",\"de\",\"fr\",\"es\",\"ru\",\"tr\",\"ar\",\"hi\",\"ur\",\"vi\",\"th\",\"el\",\"bg\",\"zh\",\"sw\"]\n", "\n", "for task, short, instr in [(\"mteb/amazon_massive_intent\", \"massive_intent\",\n", " \"What is the user asking for in `utterance`?\"),\n", " (\"mteb/amazon_massive_scenario\", \"massive_scenario\",\n", " \"Which domain does `utterance` belong to?\")]:\n", " for lg in MASSIVE_LANGS:\n", " try:\n", " d = load_dataset(task, lg, split=\"test\")\n", " labels = sorted(set(d[\"label_text\"]))\n", " rng = random.Random(SEED)\n", " cases, golds = [], []\n", " for r in list(d)[:PER_LANG]:\n", " qs, gi = build_choice_questions(rng, r[\"label_text\"], labels, N_OPTS, instr)\n", " cases.append(({\"utterance\": r[\"text\"]}, qs))\n", " golds.append({\"label\": {\"idx\": gi}})\n", " register(\"%s.%s\" % (short, lg), cases, golds, lang=lg, n_options=N_OPTS,\n", " label_space=len(labels), family=short)\n", " except Exception as e:\n", " print(\" FAIL %s %s: %s\" % (short, lg, str(e)[:90]))\n", "\n", "NLI_CRIT = {\"entailment\": \"the premise implies the hypothesis is true\",\n", " \"neutral\": \"the premise neither implies nor contradicts the hypothesis\",\n", " \"contradiction\": \"the premise implies the hypothesis is false\"}\n", "for lg in XNLI_LANGS:\n", " try:\n", " d = load_dataset(\"facebook/xnli\", lg, split=\"test\")\n", " cases, golds = [], []\n", " for r in list(d)[:PER_LANG]:\n", " qs = {\"relation\": {\"type\": \"choice\",\n", " \"instructions\": \"What is the relationship between `premise` and `hypothesis`?\",\n", " \"criteria\": dict(NLI_CRIT)}}\n", " cases.append(({\"premise\": r[\"premise\"], \"hypothesis\": r[\"hypothesis\"]}, qs))\n", " golds.append({\"relation\": {\"idx\": int(r[\"label\"])}})\n", " register(\"xnli.%s\" % lg, cases, golds, lang=lg, n_options=3, family=\"xnli\")\n", " except Exception as e:\n", " print(\" FAIL xnli %s: %s\" % (lg, str(e)[:90]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5c. English tasks\n", "\n", "`sst5`, `emotion` and `prompt_injections` were held out of Laya's training entirely (true\n", "zero-shot). `ag_news` and `boolq` were in the training mix, so they measure retention rather\n", "than generalisation. `banking77` is the many-option stress test \u2014 77 options at once, versus\n", "Jev's documented hard cap at 255." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "rng = random.Random(SEED)\n", "\n", "# SST-5: ordinal score\n", "try:\n", " d = load_dataset(\"SetFit/sst5\", split=\"test\")\n", " crit = [\"very negative\", \"negative\", \"neutral\", \"positive\", \"very positive\"]\n", " cases, golds = [], []\n", " for r in list(d)[:600]:\n", " cases.append(({\"text\": r[\"text\"]},\n", " {\"sentiment\": {\"type\": \"score\", \"instructions\": \"How positive is the sentiment of `text`?\",\n", " \"criteria\": crit}}))\n", " golds.append({\"sentiment\": {\"idx\": int(r[\"label\"]), \"gold_score\": float(r[\"label\"])}})\n", " register(\"en.sst5\", cases, golds, note=\"zero-shot, ordinal\", family=\"english\", held_out=True)\n", "except Exception as e: print(\" FAIL sst5:\", str(e)[:90])\n", "\n", "# emotion: 6-way choice\n", "try:\n", " d = load_dataset(\"dair-ai/emotion\", \"split\", split=\"test\")\n", " names = [\"sadness\",\"joy\",\"love\",\"anger\",\"fear\",\"surprise\"]\n", " cases, golds = [], []\n", " for r in list(d)[:600]:\n", " cases.append(({\"text\": r[\"text\"]},\n", " {\"emotion\": {\"type\": \"choice\",\n", " \"instructions\": \"Which emotion is most strongly expressed in `text`?\",\n", " \"criteria\": {n: None for n in names}}}))\n", " golds.append({\"emotion\": {\"idx\": int(r[\"label\"])}})\n", " register(\"en.emotion\", cases, golds, note=\"zero-shot\", family=\"english\", held_out=True)\n", "except Exception as e: print(\" FAIL emotion:\", str(e)[:90])\n", "\n", "# prompt injections: noul\n", "try:\n", " d = load_dataset(\"deepset/prompt-injections\", split=\"test\")\n", " cases, golds = [], []\n", " for r in list(d):\n", " cases.append(({\"text\": r[\"text\"]},\n", " {\"injection\": {\"type\": \"noul\",\n", " \"instructions\": \"Does `text` try to inject or override instructions given to an AI system?\"}}))\n", " golds.append({\"injection\": {\"idx\": int(r[\"label\"])}})\n", " register(\"en.prompt_injections\", cases, golds, note=\"zero-shot\", family=\"english\", held_out=True)\n", "except Exception as e: print(\" FAIL prompt_injections:\", str(e)[:90])\n", "\n", "# banking77: 77-option stress test\n", "try:\n", " d = load_dataset(\"PolyAI/banking77\", split=\"test\")\n", " names = d.features[\"label\"].names\n", " cases, golds = [], []\n", " for r in list(d)[:500]:\n", " cases.append(({\"message\": r[\"text\"]},\n", " {\"intent\": {\"type\": \"choice\", \"instructions\": \"Which banking intent does `message` express?\",\n", " \"criteria\": {n.replace(\"_\", \" \"): None for n in names}}}))\n", " golds.append({\"intent\": {\"idx\": int(r[\"label\"])}})\n", " register(\"en.banking77_full\", cases, golds, note=\"77 options at once\", family=\"english\",\n", " n_options=len(names), held_out=True)\n", "except Exception as e: print(\" FAIL banking77:\", str(e)[:90])\n", "\n", "# ag_news + boolq: in-task retention\n", "try:\n", " d = load_dataset(\"fancyzhx/ag_news\", split=\"test\")\n", " crit = {\"world\": \"world news and international politics\", \"sports\": \"sports\",\n", " \"business\": \"business and economy\", \"sci_tech\": \"science and technology\"}\n", " cases, golds = [], []\n", " for r in list(d)[:600]:\n", " cases.append(({\"article\": r[\"text\"]},\n", " {\"topic\": {\"type\": \"choice\", \"instructions\": \"What is the topic of `article`?\",\n", " \"criteria\": dict(crit)}}))\n", " golds.append({\"topic\": {\"idx\": int(r[\"label\"])}})\n", " register(\"en.ag_news\", cases, golds, note=\"in-task (trained on)\", family=\"english\", held_out=False)\n", "except Exception as e: print(\" FAIL ag_news:\", str(e)[:90])\n", "\n", "try:\n", " d = load_dataset(\"google/boolq\", split=\"validation\")\n", " cases, golds = [], []\n", " for r in list(d)[:600]:\n", " cases.append(({\"passage\": r[\"passage\"], \"question\": r[\"question\"]},\n", " {\"answer\": {\"type\": \"noul\",\n", " \"instructions\": \"Based on `passage`, is the answer to `question` yes?\"}}))\n", " golds.append({\"answer\": {\"idx\": int(bool(r[\"answer\"]))}})\n", " register(\"en.boolq\", cases, golds, note=\"in-task (trained on)\", family=\"english\", held_out=False)\n", "except Exception as e: print(\" FAIL boolq:\", str(e)[:90])\n", "\n", "print()\n", "print(\"suites built:\", len(SUITES), \"| total questions:\",\n", " sum(sum(len(c[1]) for c in s[\"cases\"]) for s in SUITES.values()))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Run every suite on both models" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import bench_engine as BE\n", "import numpy as np, gc, time, json\n", "\n", "RESULTS = {\"meta\": {\"seed\": SEED, \"n_opts\": N_OPTS, \"per_lang\": PER_LANG,\n", " \"timestamp\": time.strftime(\"%Y-%m-%d %H:%M:%S\"),\n", " \"gpu\": torch.cuda.get_device_name(0),\n", " \"torch\": torch.__version__, \"transformers\": transformers.__version__,\n", " \"models\": MODEL_META},\n", " \"suites\": {}}\n", "RAW = {} # model -> suite -> (logits, index) kept in RAM for the calibration-repair section\n", "\n", "def evaluate_suite(agent, model_name, suite_name):\n", " S = SUITES[suite_name]\n", " logits, index, secs, dropped = BE.score_cases(agent, S[\"cases\"], progress=\"%s/%s\" % (model_name, suite_name))\n", " RAW[model_name][suite_name] = (logits, index)\n", " soft = S[\"meta\"].get(\"soft\", False)\n", " rows_cal, rows_raw, extra = [], [], {\"soft_acc\": [], \"brier_soft\": [], \"tv\": [], \"kl\": [],\n", " \"score_mae\": [], \"within_1\": []}\n", " per_q_type = {}\n", " for (ci, qid, qt, k), z in zip(index, logits):\n", " g = S[\"gold\"][ci].get(qid)\n", " if g is None:\n", " continue\n", " if z is None:\n", " rows_cal.append((g[\"idx\"], None)); rows_raw.append((g[\"idx\"], None)); continue\n", " p_cal = BE.softmax_t(z, BE.temp_for(agent, qt, k, True))\n", " p_raw = BE.softmax_t(z, 1.0)\n", " rows_cal.append((g[\"idx\"], p_cal)); rows_raw.append((g[\"idx\"], p_raw))\n", " per_q_type.setdefault(qt, []).append((g[\"idx\"], p_cal))\n", " if soft and \"soft\" in g:\n", " gp = np.asarray(g[\"soft\"], float)\n", " if gp.sum() > 0:\n", " gp = gp / gp.sum()\n", " pp = p_cal[:len(gp)] if len(p_cal) >= len(gp) else np.pad(p_cal, (0, len(gp)-len(p_cal)))\n", " pp = pp / max(pp.sum(), 1e-12)\n", " extra[\"soft_acc\"].append(float((pp*gp).sum()))\n", " extra[\"brier_soft\"].append(float(((pp-gp)**2).sum()))\n", " extra[\"tv\"].append(float(0.5*np.abs(pp-gp).sum()))\n", " extra[\"kl\"].append(float((gp*np.log(np.clip(gp/np.clip(pp,1e-12,None),1e-12,1e4))).sum()))\n", " if \"gold_score\" in g:\n", " exp = float((np.arange(len(p_cal))*p_cal).sum())\n", " extra[\"score_mae\"].append(abs(exp - g[\"gold_score\"]))\n", " extra[\"within_1\"].append(float(abs(exp - g[\"gold_score\"]) <= 1.0))\n", "\n", " out = {\"calibrated\": BE.hard_metrics(rows_cal), \"raw\": BE.hard_metrics(rows_raw),\n", " \"seconds\": round(secs, 2), \"dropped_questions\": dropped,\n", " \"questions_per_second\": round(len([r for r in rows_cal if r[1] is not None])/max(secs,1e-9), 1)}\n", " for k2, v in extra.items():\n", " if v:\n", " out[k2] = float(np.mean(v))\n", " if len(per_q_type) > 1:\n", " names = {0: \"choice\", 1: \"score\", 2: \"noul\"}\n", " out[\"by_question_type\"] = {names[qt]: BE.hard_metrics(rs) for qt, rs in sorted(per_q_type.items())}\n", " return out\n", "\n", "for model_name in REPOS:\n", " print(\"\\n\" + \"=\"*78 + \"\\n %s\\n\" % model_name + \"=\"*78)\n", " agent = load_agent(model_name)\n", " RAW[model_name] = {}\n", " for suite_name in SUITES:\n", " try:\n", " r = evaluate_suite(agent, model_name, suite_name)\n", " RESULTS[\"suites\"].setdefault(suite_name, {})[model_name] = r\n", " c = r[\"calibrated\"]\n", " print(\" %-32s acc %.3f f1 %.3f ECE %.3f n=%d\" %\n", " (suite_name, c.get(\"accuracy\", float(\"nan\")), c.get(\"macro_f1\", float(\"nan\")),\n", " c.get(\"ece\", float(\"nan\")), c.get(\"n\", 0)))\n", " except Exception as e:\n", " import traceback; traceback.print_exc()\n", " RESULTS[\"suites\"].setdefault(suite_name, {})[model_name] = {\"error\": \"%s: %s\" % (type(e).__name__, e)}\n", " globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")] = agent\n", " gc.collect(); torch.cuda.empty_cache()\n", "\n", "json.dump(RESULTS, open(\"laya_benchmark_results.json\", \"w\"), indent=2)\n", "print(\"\\ncheckpoint saved\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 6b. Control \u2014 matched context budget\n", "\n", "The two checkpoints ship different context windows (512 vs 1024 tokens), so on long states\n", "`laya-multilingual` sees more of the input. This re-runs typed-decisions with **both** models\n", "capped at 512 tokens, isolating the architecture from the context-window advantage." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import bench_engine as BE\n", "import numpy as np\n", "\n", "RESULTS[\"typed_decisions_matched_512\"] = {}\n", "if \"typed_decisions\" in SUITES:\n", " S = SUITES[\"typed_decisions\"]\n", " for model_name in REPOS:\n", " agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n", " orig = agent.cfg.get(\"max_len\")\n", " agent.cfg[\"max_len\"] = 512\n", " try:\n", " logits, index, secs, dropped = BE.score_cases(agent, S[\"cases\"], progress=\"%s/td@512\" % model_name)\n", " rows = []\n", " for (ci, qid, qt, k), z in zip(index, logits):\n", " g = S[\"gold\"][ci].get(qid)\n", " if g is None or z is None:\n", " continue\n", " rows.append((g[\"idx\"], BE.softmax_t(z, BE.temp_for(agent, qt, k, True))))\n", " RESULTS[\"typed_decisions_matched_512\"][model_name] = BE.hard_metrics(rows)\n", " print(\" %-22s max_len 512 -> acc %.3f ECE %.3f n=%d\"\n", " % (model_name, RESULTS[\"typed_decisions_matched_512\"][model_name][\"accuracy\"],\n", " RESULTS[\"typed_decisions_matched_512\"][model_name][\"ece\"],\n", " RESULTS[\"typed_decisions_matched_512\"][model_name][\"n\"]))\n", " finally:\n", " agent.cfg[\"max_len\"] = orig" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. Latency (p50 / p95 at 1, 5, 10, 50 questions per call)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np, time, torch\n", "\n", "LAT_STATE = {\"ticket\": {\"subject\": \"Payout failing\",\n", " \"messages\": [{\"from\": \"customer\",\n", " \"text\": \"Hi, my Stripe payouts have failed for 3 days and I am losing sales. Please help ASAP. \" * 6}]}}\n", "Q_NOUL = {\"type\": \"noul\", \"instructions\": \"Does `ticket.messages[0].text` express urgency?\"}\n", "Q_CHOICE = {\"type\": \"choice\", \"instructions\": \"Which team should handle this?\",\n", " \"criteria\": {\"billing\": \"payments\", \"technical\": \"bugs and integrations\", \"sales\": \"pricing\"}}\n", "\n", "RESULTS[\"latency\"] = {}\n", "for model_name in REPOS:\n", " agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n", " res = {}\n", " for nq in (1, 5, 10, 50):\n", " qs = {(\"q%d\" % i): (Q_NOUL if i % 2 else Q_CHOICE) for i in range(nq)}\n", " for _ in range(3):\n", " agent.system_one(LAT_STATE, qs)\n", " ts = []\n", " for _ in range(20):\n", " torch.cuda.synchronize(); t = time.perf_counter()\n", " agent.system_one(LAT_STATE, qs)\n", " torch.cuda.synchronize(); ts.append((time.perf_counter()-t)*1000)\n", " res[\"%d_questions\" % nq] = {\"p50_ms\": round(float(np.percentile(ts, 50)), 1),\n", " \"p95_ms\": round(float(np.percentile(ts, 95)), 1),\n", " \"ms_per_question\": round(float(np.percentile(ts, 50))/nq, 2)}\n", " RESULTS[\"latency\"][model_name] = res\n", " print(model_name, json.dumps(res))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8. Option-order robustness\n", "\n", "Same question, options permuted. A well-behaved decision model should pick the same label.\n", "The independent Jev benchmark measured a **13% flip rate for Jev**, with the worst LLM at 37%." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import random, numpy as np\n", "import bench_engine as BE\n", "\n", "PB_SUITES = [s for s in (\"massive_intent.en\", \"en.emotion\", \"xnli.en\") if s in SUITES]\n", "RESULTS[\"option_order_robustness\"] = {}\n", "\n", "for model_name in REPOS:\n", " agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n", " out = {}\n", " for sname in PB_SUITES:\n", " S = SUITES[sname]\n", " base = S[\"cases\"][:200]\n", " rng = random.Random(99)\n", " perm_cases, perm_maps = [], []\n", " for state, questions in base:\n", " qid = list(questions)[0]\n", " qdef = questions[qid]\n", " keys = list(qdef[\"criteria\"].keys())\n", " order = list(range(len(keys))); rng.shuffle(order)\n", " newcrit = {keys[i]: qdef[\"criteria\"][keys[i]] for i in order}\n", " perm_cases.append((state, {qid: {**qdef, \"criteria\": newcrit}}))\n", " perm_maps.append((keys, [keys[i] for i in order]))\n", " la, ia, _, _ = BE.score_cases(agent, base, progress=\"\")\n", " lb, ib, _, _ = BE.score_cases(agent, perm_cases, progress=\"\")\n", " flips = comp = 0\n", " for (za, (_, _, _, ka)), (zb, (_, _, _, kb)), (keys, pkeys) in zip(zip(la, ia), zip(lb, ib), perm_maps):\n", " if za is None or zb is None:\n", " continue\n", " comp += 1\n", " if keys[int(np.argmax(za))] != pkeys[int(np.argmax(zb))]:\n", " flips += 1\n", " out[sname] = {\"n\": comp, \"flip_rate\": round(flips/max(1, comp), 4)}\n", " print(\" %-22s %-24s flip rate %.3f (n=%d)\" % (model_name, sname, out[sname][\"flip_rate\"], comp))\n", " RESULTS[\"option_order_robustness\"][model_name] = out" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9. Calibration repair\n", "\n", "`laya-multilingual` ships with `temperature = [1.0, 1.0, 1.0]` and no per-option-count buckets \u2014\n", "it was never calibrated. This refits one temperature per (question type, option-count bucket) on\n", "**half** of each suite and reports ECE on the other half, so the improvement is measured\n", "out-of-sample. It shows how much of any calibration gap is a missing post-processing step rather\n", "than a property of the model." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import bench_engine as BE\n", "from laya.common import temp_bucket\n", "\n", "RESULTS[\"calibration_repair\"] = {}\n", "for model_name in REPOS:\n", " agent = globals()[\"AGENT_\" + model_name.replace(\"-\", \"_\")]\n", " per_suite, fitted_all = {}, {}\n", " for sname, S in SUITES.items():\n", " if sname not in RAW[model_name]:\n", " continue\n", " logits, index = RAW[model_name][sname]\n", " pairs = []\n", " for (ci, qid, qt, k), z in zip(index, logits):\n", " g = S[\"gold\"][ci].get(qid)\n", " if g is not None and z is not None:\n", " pairs.append((z, g[\"idx\"], qt, k))\n", " if len(pairs) < 60:\n", " continue\n", " half = len(pairs)//2\n", " fit_set, held = pairs[:half], pairs[half:]\n", " buckets = {}\n", " for z, gi, qt, k in fit_set:\n", " buckets.setdefault(temp_bucket(qt, k), []).append((z, gi))\n", " fitted = {b: BE.fit_temperature(v) for b, v in buckets.items() if len(v) >= 25}\n", " rows_ship, rows_fit = [], []\n", " for z, gi, qt, k in held:\n", " rows_ship.append((gi, BE.softmax_t(z, BE.temp_for(agent, qt, k, True))))\n", " rows_fit.append((gi, BE.softmax_t(z, fitted.get(temp_bucket(qt, k),\n", " BE.temp_for(agent, qt, k, True)))))\n", " ms, mf = BE.hard_metrics(rows_ship), BE.hard_metrics(rows_fit)\n", " per_suite[sname] = {\"n_heldout\": ms[\"n\"], \"fitted_temperatures\": fitted,\n", " \"ece_shipped\": ms[\"ece\"], \"ece_refit\": mf[\"ece\"],\n", " \"nll_shipped\": ms[\"nll\"], \"nll_refit\": mf[\"nll\"],\n", " \"accuracy\": ms[\"accuracy\"]}\n", " fitted_all.update(fitted)\n", " RESULTS[\"calibration_repair\"][model_name] = {\"per_suite\": per_suite}\n", " if per_suite:\n", " a = float(np.mean([v[\"ece_shipped\"] for v in per_suite.values()]))\n", " b = float(np.mean([v[\"ece_refit\"] for v in per_suite.values()]))\n", " RESULTS[\"calibration_repair\"][model_name][\"mean_ece_shipped\"] = round(a, 4)\n", " RESULTS[\"calibration_repair\"][model_name][\"mean_ece_refit\"] = round(b, 4)\n", " print(\"%-22s mean ECE shipped %.4f -> refit %.4f\" % (model_name, a, b))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 10. Aggregate, save and download" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np, json\n", "\n", "def agg(prefix_or_family, key=\"family\"):\n", " out = {}\n", " for m in REPOS:\n", " accs, eces, ns = [], [], 0\n", " for sname, S in SUITES.items():\n", " if S[\"meta\"].get(key) != prefix_or_family:\n", " continue\n", " r = RESULTS[\"suites\"].get(sname, {}).get(m, {})\n", " c = r.get(\"calibrated\", {})\n", " if c.get(\"n\"):\n", " accs.append(c[\"accuracy\"]); eces.append(c[\"ece\"]); ns += c[\"n\"]\n", " if accs:\n", " out[m] = {\"macro_accuracy\": round(float(np.mean(accs)), 4),\n", " \"macro_ece\": round(float(np.mean(eces)), 4),\n", " \"n_questions\": ns, \"n_suites\": len(accs)}\n", " return out\n", "\n", "RESULTS[\"caveats\"] = {\n", " \"context_window\": \"laya max_len=512, laya-multilingual max_len=1024 (shipped configs). \"\n", " \"See typed_decisions_matched_512 for the equal-budget control.\",\n", " \"shipped_calibration\": \"laya ships fitted temperatures; laya-multilingual ships all-1.0 \"\n", " \"(uncalibrated). See calibration_repair for the out-of-sample refit.\",\n", " \"training_overlap\": \"ag_news and boolq were in Laya's training mix (retention, not generalisation). \"\n", " \"sst5, emotion, prompt_injections, banking77 were held out. \"\n", " \"Neither checkpoint trained on typed-decisions, MASSIVE or XNLI.\",\n", "}\n", "\n", "RESULTS[\"summary\"] = {\n", " \"typed_decisions\": {m: RESULTS[\"suites\"].get(\"typed_decisions\", {}).get(m, {}).get(\"calibrated\", {})\n", " for m in REPOS},\n", " \"massive_intent\": agg(\"massive_intent\"),\n", " \"massive_scenario\": agg(\"massive_scenario\"),\n", " \"xnli\": agg(\"xnli\"),\n", " \"english\": agg(\"english\"),\n", "}\n", "\n", "# English vs non-English split inside each multilingual family\n", "for fam in (\"massive_intent\", \"massive_scenario\", \"xnli\"):\n", " en, non = {}, {}\n", " for m in REPOS:\n", " ea, na = [], []\n", " for sname, S in SUITES.items():\n", " if S[\"meta\"].get(\"family\") != fam:\n", " continue\n", " c = RESULTS[\"suites\"].get(sname, {}).get(m, {}).get(\"calibrated\", {})\n", " if not c.get(\"n\"):\n", " continue\n", " (ea if S[\"meta\"].get(\"lang\") in (\"en\",) else na).append(c[\"accuracy\"])\n", " if ea: en[m] = round(float(np.mean(ea)), 4)\n", " if na: non[m] = round(float(np.mean(na)), 4)\n", " RESULTS[\"summary\"][fam + \"_english_vs_rest\"] = {\"english\": en, \"non_english\": non}\n", "\n", "json.dump(RESULTS, open(\"laya_benchmark_results.json\", \"w\"), indent=2)\n", "print(json.dumps(RESULTS[\"summary\"], indent=2))\n", "print(\"\\nwrote laya_benchmark_results.json (%.1f KB)\" % (os.path.getsize(\"laya_benchmark_results.json\")/1024))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Per-language table, printed for a quick eyeball before you download the JSON\n", "import numpy as np\n", "for fam in (\"massive_intent\", \"massive_scenario\", \"xnli\"):\n", " langs = sorted({S[\"meta\"][\"lang\"] for S in SUITES.values() if S[\"meta\"].get(\"family\") == fam})\n", " if not langs:\n", " continue\n", " print(\"\\n=== %s : accuracy by language ===\" % fam)\n", " print(\"%-8s %12s %22s %10s\" % (\"lang\", \"laya\", \"laya-multilingual\", \"delta\"))\n", " for lg in langs:\n", " sname = \"%s.%s\" % (fam, lg)\n", " a = RESULTS[\"suites\"].get(sname, {}).get(\"laya\", {}).get(\"calibrated\", {}).get(\"accuracy\")\n", " b = RESULTS[\"suites\"].get(sname, {}).get(\"laya-multilingual\", {}).get(\"calibrated\", {}).get(\"accuracy\")\n", " if a is None or b is None:\n", " continue\n", " print(\"%-8s %12.3f %22.3f %+10.3f\" % (lg, a, b, b-a))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "try:\n", " from google.colab import files\n", " files.download(\"laya_benchmark_results.json\")\n", "except Exception as e:\n", " print(\"Not on Colab or download blocked (%s). Grab laya_benchmark_results.json from the file browser.\" % type(e).__name__)" ] } ], "metadata": { "accelerator": "GPU", "colab": { "provenance": [], "gpuType": "T4", "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }