[whatmodel.sh](../../README.md) / ChatGLM-6B
# ChatGLM-6B
Zhipu AI (THUDM) · released 2023-03 · 6.2B params · chatglm-6b-license
**score 24.5** ±1.4 · #691 of 732 · from 11 benchmark results across 9 sources
Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only.
## What it's good at
● marks ChatGLM-6B among every model on that task or test, worst to best.
task vs the others rank counts tests
coding ▁●▁▃▅▇█▇▅▄▃▁ #466 of 469* 13% 1
chat & writing ●▁▁▃▆▄▆█▆▇▇▃ #400 of 402 57% 4
coding · #466 of 469 · 13% of its score
Trails [GPT-Neo 2.7B](gpt-neo-2-7b.md), [Phi-1 (1.3B)](phi-1.md), [Pythia 6.9B](pythia-6-9b.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
HumanEval 3.1% ▄··●····▄▄█▄ beats 1/6 medium · avg 63% 13% sharp · too hard for it
chat & writing · #400 of 402 · 57% of its score
Trails [Gemma 3 270M IT](gemma-3-270m.md), [SmolLM2 360M Instruct](smollm2-360m.md) here, though they rank lower overall.
test result vs the others in its run difficulty counts why
LMArena (text) 995.9 ●▁▁▂▃▃▃▆▆▇█▃ beats 4/385 avg 1260 22% blunt
Polyrating rating (bias-aware re-… 874.0 ●▁▂▁▃▇█▆▇▄▂▂ beats 3/95 avg 1187 20% typical
MT-Bench (GPT-4 single-answer gra… 4.5000 MT-Bench score (1-10) ▆▂▂●▆▄██▂▄▄▆ beats 6/28 avg 8.539 MT-Bench scor… 15% sharp · noisy
MixInstruct GPT-Rank (LLM-Blender… 5.6300 mean rank (1-11) █···▄··●···▄ beats 3/4 avg 6.709 mean rank (1-… 0.0% blunt · noisy
other tests (6)
test result vs the others in its run difficulty counts why
BIG-Bench Hard 31.3% ██●··█·█·█·█ beats 2/6 medium · avg 68% 14% sharp · too hard for it
MMLU 36.1% ▄▄·▄···▄●·██ beats 4/8 easy · avg 74% 8.0% typical
API-Bank evaluation system, zero-… 16.4% ▄······●··▄▄ beats 2/4 medium · avg 50% 2.7% typical
Conic10K mathQA (zero-shot CoT) 1.5% █··●·······▄ beats 2/3 very hard · avg 10% 2.0% too hard for it
CAIL charge prediction, zero-shot… 41.4% ●···█··█···█ beats 0/3 medium · avg 54% 1.2% blunt · at its level
MenatQA (all factors), exact matc… 3.8% ●····▂··▂·█▄ beats 0/8 medium · avg 36% 1.2% blunt
source results share of the score
InstructEval 3 35% ▇▇▇▇
LMArena leaderboard dataset 1 22% ▇▇▇
Polyrating: A Cost-Effective and Bias-A… 1 20% ▇▇
LMSYS Chatbot Arena leaderboard Space: … 1 15% ▇▇
API-Bank: A Comprehensive Benchmark for… 1 3% ▇
Conic10K: A Challenging Math Problem Un… 1 2% ▇
A Comprehensive Evaluation of Large Lan… 1 1% ▇
MenatQA: A New Dataset for Testing the … 1 1% ▇
LLM-Blender: Ensembling Large Language … 1 0% ▇
prior: pull toward the middle 1% ▇
weighted average = 24.5
Licences: InstructEval ([MIT / Apache-2.0](https://github.com/declare-lab/instruct-eval/blob/49bbd2ce7e3a5bdc4024ba62a5c9c68c107ee966/LICENSE)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · LMSYS Chatbot Arena leaderboard Space: MT-Bench column of the published leaderboard table ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs ([CC BY 4.0](../../licenses/acl-2023-emnlp-main-187-CC-BY-4-0.txt)) · Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-427-CC-BY-4-0.txt)) · A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-490-CC-BY-4-0.txt)) · MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models ([CC BY 4.0](../../licenses/acl-2023-findings-emnlp-100-CC-BY-4-0.txt)) · LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/))
## How to run it
### Other ways to run it
1. Run it as published (fp16, needs ~13.42 GB, NVIDIA only, from [huggingface.co/zai-org](https://huggingface.co/zai-org/chatglm-6b/blob/bf0f5cfb575eebebf9b655c5861177acfee03f16/README.md))
```sh
pip install protobuf==3.20.0 transformers==4.27.1 icetk cpm_kernels torch && python -c "from transformers import AutoTokenizer, AutoModel; t=AutoTokenizer.from_pretrained('THUDM/chatglm-6b', trust_remote_code=True); m=AutoModel.from_pretrained('THUDM/chatglm-6b', trust_remote_code=True).half().cuda().eval(); print(m.chat(t, '你好', history=[])[0])"
```
Model repository and documentation: [huggingface.co/zai-org](https://huggingface.co/zai-org/chatglm-6b)
[interactive version](https://whatmodel.sh/model/chatglm-6b) · [JSON](https://whatmodel.sh/api?format=json&all=1)