[whatmodel.sh](../../README.md) / OpenHermes 2.5 Mistral 7B # OpenHermes 2.5 Mistral 7B Nous Research · released 2023-10 · 7.24B params · apache-2.0 **score 41.6** ±0.9 · #537 of 732 · from 14 benchmark results across 8 sources Score within the [LMArena (text)](https://whatmodel.sh/?set=lmarena-text&score=set) comparison set only. ## What it's good at ● marks OpenHermes 2.5 Mistral 7B among every model on that task or test, worst to best. 
task            vs the others          rank  counts  tests
coding          ▁▁▁▃●▇█▇▅▄▃▁   #368 of 469*    6.6%      1
chat & writing  ▁▁▁▃●▄▆█▆▇▇▃    #339 of 402     36%      3
math            ●··········█       #2 of 2*    5.6%      1
coding · #368 of 469 · 6.6% of its score Beats [LFM2.5 8B-A1B](lfm2-5-8b-a1b.md), [Qwen3-VL 2B Thinking](qwen3-vl-2b-thinking.md), [DeepSeek R1 Distill Qwen 7B](deepseek-r1-distill-qwen-7b.md) here, though they rank higher overall. Trails [Qwen2.5 1.5B Instruct](qwen2-5-1-5b.md), [Code Tulu 2 13B](code-tulu-2-13b.md), [Phi-3-mini-128k-instruct (Apr 2024)](phi-3-mini-128k-instruct.md) here, though they rank lower overall. 
test       result  vs the others in its run   difficulty      counts  why
HumanEval   48.2%  ▁▁▁▁▂▂▆●█▅▂▁ beats 88/205  easy · avg 72%    6.6%  sharp · at its level · small run
chat & writing · #339 of 402 · 36% of its score Beats [OpenThinker3 7B](openthinker3-7b.md), [Code Llama 70B Instruct](codellama-70b-instruct.md), [Gemma 4 E2B](gemma-4-e2b.md) here, though they rank higher overall. Trails [Granite 4.0 H 350M](granite-4-0-h-350m.md), [LFM2 350M](lfm2-350m.md), [Gemma 3 1B](gemma-3-1b.md) here, though they rank lower overall. 
test                                result  vs the others in its run   difficulty        counts  why
AlpacaEval 2 length-controlled       16.3%  ▁▂▂▃▆●█▆▄▄▁▁ beats 42/122  medium · avg 35%     21%  sharp
LMArena (text)                      1176.0  ▁▁▁▂●▃▃▆▆▇█▃ beats 53/385  avg 1260            8.2%  blunt
Polyrating rating (bias-aware re-…  1075.0  ▂▁▂▁▃▇●▆▇▄▂▂ beats 35/95   avg 1187            7.5%  typical
math · #2 of 2 · 5.6% of its score 
test   result  vs the others in its run    difficulty           counts  why
GSM8K   73.5%  ▂▂▂▂▃▄▅▇●█▆▄ beats 126/206  very easy · avg 88%    5.6%  sharp · at its level · small run
other tests (9) 
test                                result  vs the others in its run    difficulty        counts  why
LiveBench 2024-07-26                 24.1%  ▁▁▁▂▂▃●█▆▃▅▃ beats 17/61    medium · avg 38%     25%  typical
Natural Questions (closed book)      26.6%  ▁·▁·▁▂▁▂·▅▂● beats 22/28    medium · avg 48%    8.7%  at its level
Polish EQ-Bench v2 (SpeakLeash)      37.5%  ▁··▁▁▂●▆▅▆█▆ beats 11/70    medium · avg 62%    5.0%  at its level
BIG-Bench Hard                       59.9%  ▁▁▁▁▂▄▅●█▃▃▁ beats 87/155   easy · avg 76%      4.7%  sharp · at its level · small run
MMLU                                 63.8%  ▁▁▁▁▁▂▂▅●█▆▄ beats 100/214  easy · avg 75%      4.2%  at its level · small run
Open PT-LLM Leaderboard average (…   66.5%  ▃▁▁▁▁▁▁▁▂●▅▁ beats 126/276  medium · avg 72%    1.9%  at its level · noisy
TriviaQA                             66.0%  ▁··▁█▃▃▁▄▅▄● beats 27/28    easy · avg 73%      0.9%  at its level · noisy
HaluEval QA hallucination detecti…   50.6%  █▁●·▁▁·▁▁▁▁▁ beats 17/27    hard · avg 66%      0.8%  noisy
TruthfulQA MC2                       53.0%  ▅▃█▆▁▂●···▁▁ beats 25/31    medium · avg 57%    0.2%  blunt · at its level
## How the score was worked out Each result implies a score on its own, after adjusting for how hard that test is and who ran it. The model's score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its *counts* column above; here it is added up by who ran the tests. [Method](../method.md) 
source                                    results  share of the score
LiveBench model judgments                       1   25% ▇▇▇
OpenChat 3.5 model card                         4   21% ▇▇▇
AlpacaEval 1.0 and 2.0 leaderboards             1   21% ▇▇
Hallucinations Leaderboard results              4   11% ▇
LMArena leaderboard dataset                     1    8% ▇
Polyrating: A Cost-Effective and Bias-A…        1    8% ▇
SpeakLeash Polish EQ-Bench leaderboard          1    5% ▇
Open Portuguese LLM Leaderboard                 1    2% ▇
prior: pull toward the middle                        0% ▇
weighted average                                   = 41.6
Licences: LiveBench model judgments ([Apache-2.0 (stated in the LiveBench datasheet)](https://www.apache.org/licenses/LICENSE-2.0)) · OpenChat 3.5 model card ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · AlpacaEval 1.0 and 2.0 leaderboards ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · Hallucinations Leaderboard results ([Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)) · LMArena leaderboard dataset ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)) · Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation ([CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)) · SpeakLeash Polish EQ-Bench leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) · Open Portuguese LLM Leaderboard ([Apache-2.0](../../licenses/hf-leaderboard-spaces-Apache-2.0.txt)) ## How to run it ### llama.cpp (recommended) 1. Install llama.cpp. Homebrew and winget installs name the commands llama-cli and llama-server instead. [Other ways to install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md) ```sh # macOS / Linux curl -LsSf https://llama.app/install.sh | sh # Windows irm https://llama.app/install.ps1 | iex ``` 2. Run OpenHermes 2.5 Mistral 7B (Q4_K_M, needs ~4.4 GB, from [huggingface.co/TheBloke](https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-GGUF/tree/5682e25bb033d9d21f6d159859e21df4552c1f26)) ```sh llama cli -hf TheBloke/OpenHermes-2.5-Mistral-7B-GGUF:Q4_K_M ``` 3. Or start a local server and open the address it prints (http://localhost:9931) ```sh llama serve -hf TheBloke/OpenHermes-2.5-Mistral-7B-GGUF:Q4_K_M ``` Other versions: Q8_0 (~7.7 GB) `llama cli -hf TheBloke/OpenHermes-2.5-Mistral-7B-GGUF:Q8_0` ### LM Studio 1. Install LM Studio. Open the app once before using the lms command. [Download it (macOS / Windows / Linux)](https://lmstudio.ai/download) 2. Run OpenHermes 2.5 Mistral 7B (Q4_K_M, needs ~4.4 GB, from [huggingface.co/TheBloke](https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-GGUF/tree/5682e25bb033d9d21f6d159859e21df4552c1f26)) ```sh lms get https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-GGUF@Q4_K_M -y ``` 3. Then run lms ls to see its name, and lms chat followed by that name Other versions: Q8_0 (~7.7 GB) `lms get https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-GGUF@Q8_0 -y` · BF16 (~14.5 GB, Mac only) `lms get https://huggingface.co/TheBlueObserver/OpenHermes-2.5-Mistral-7B-MLX -y` ### Ollama 1. Install Ollama. [Download it (macOS / Windows)](https://ollama.com/download) ```sh # Linux curl -fsSL https://ollama.com/install.sh | sh ``` 2. Run OpenHermes 2.5 Mistral 7B (Q4_K_M, needs ~4.84 GB, from [ollama.com](https://ollama.com/library/openhermes/tags)) ```sh ollama run openhermes:7b-mistral-v2.5-q4_K_M ``` Other versions: Q8_0 (~9.19 GB) `ollama run openhermes:7b-mistral-v2.5-q8_0` ### MLX (Mac) 1. Install MLX (Mac). Apple silicon Macs only, text only. ```sh # with uv uv tool install mlx-lm # or pip install mlx-lm ``` 2. Run OpenHermes 2.5 Mistral 7B (BF16, needs ~14.5 GB, Mac only, from [huggingface.co/TheBlueObserver](https://huggingface.co/TheBlueObserver/OpenHermes-2.5-Mistral-7B-MLX/tree/148bccca66475f4a0723837cb5a75ccf5bf62cb4)) ```sh mlx_lm.chat --model TheBlueObserver/OpenHermes-2.5-Mistral-7B-MLX ``` 3. Or start an OpenAI-compatible server at http://localhost:8080/v1 ```sh mlx_lm.server --model TheBlueObserver/OpenHermes-2.5-Mistral-7B-MLX ``` Model repository and documentation: [huggingface.co/teknium](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B) [interactive version](https://whatmodel.sh/model/openhermes-2-5-mistral-7b) · [JSON](https://whatmodel.sh/api?format=json&all=1)