LocalLlama

Best way to initialize AGENTS.md

2026-02-09T22:34:18+00:00

AI coding tools work a lot better when they understand a repo’s stack, commands, and conventions.

npx agentseed init

This reads your codebase and generates AGENTS.md automatically using static analysis (free). You can optionally add LLM summaries (free with Llama again) for richer context.

Open source (MIT): https://github.com/avinshe/agentseed

submitted by /u/ThatSQLguy
[link] [comments]

SDF Protocol — fine-tuned 1.5B + 3B models that convert web pages into structured JSON for AI agents (open weights on HuggingFace)

2026-02-09T20:22:22+00:00

I've been working on an open protocol for pre-extracting structured data from web pages so AI agents don't have to re-parse HTML every time.

The pipeline uses two small fine-tuned models running locally via Ollama:

sdf-classify (Qwen2.5-1.5B-Instruct, QLoRA): classifies content into 10 parent types / 50+ subtypes
sdf-extract (SmolLM3-3B, QLoRA): extracts entities, claims, relationships, summaries, and type-specific fields into schema-validated JSON

Combined footprint is 2.8 GB (Q4_K_M). Runs on CPU too — just slower.

Results on 2,335 documents:

90% extraction accuracy (exact match)
4.1x faster than monolithic 14B baseline
99.2% token reduction from HTML (~73K tokens → ~750)
Works on CPU, tested on dual 3090 Ti for the paper

Downstream test: gave a vanilla 7B model questions about 30 documents — scored 0.739 accuracy from SDF vs 0.352 from raw markdown. 3B model also showed significant improvement (0.606 vs 0.333).

Models (GGUF Q4_K_M + f16): https://huggingface.co/sdfprotocol

Protocol spec + schemas: https://github.com/sdfprotocol/sdf

Whitepaper: https://doi.org/10.5281/zenodo.18559223

Training was QLoRA rank 32, alpha 64, dropout 0.05.

submitted by /u/PlayfulLingonberry73
[link] [comments]

I managed to jailbreak 43 of 52 recent models

2026-02-09T10:52:45+00:00

GPT-5 broke at level 2,

Full report here: rival.tips/jailbreak I'll be adding more models to this benchmark soon

submitted by /u/sirjoaco
[link] [comments]

Is there any Local LLMs that out perform commercial or cloud based LLMs in certain areas or functions?

2026-02-10T02:02:25+00:00

I'm curious if anybody has seen local LLMs outperform commercial or cloud-based LLMS in certain areas or functions. If so what model and how did it out perform?

Is there hope in the future that local LLMs could develop an edge over commercial or cloud based LLMs?

submitted by /u/FX2021
[link] [comments]

Qwen3-Coder-Next performance on MLX vs llamacpp

2026-02-09T16:02:19+00:00

Ivan Fioravanti just published an excellent breakdown of performance differences between MLX-LM and llama.cpp running on the Apple M3 Ultra. These are both great options for local inference, but it seems MLX has a significant edge for most workloads.

https://preview.redd.it/vb5b4b8xrhig1.png?width=2316&format=png&auto=webp&s=31aa4012319625eb4f437d590a7f2cec4f1ce810

https://x.com/ivanfioravanti/status/2020876939917971867?s=20

submitted by /u/TrajansRow
[link] [comments]

Strix Halo, Step-3.5-Flash-Q4_K_S imatrix, llama.cpp/ROCm/Vulkan Power & Efficiency test

2026-02-09T14:04:09+00:00

Hi, i did recently some quants to test best fit for strix halo, and i settled with custom imatrix Q4_K_S quant, builded with wikitext-103-raw-v1. Model has sligtly better PPL than Q4_K_M without imatrix, but it's few GB smaller. I tested it with ROCm/Vulkan backend, and llama.cpp build 7966 (8872ad212), so with Step-3.5-Flash support already merged to the main branch. There are some issues with toolcalling with that (and few others) models at the moment but seems it's not related to quants itself.

Quantization	Size (Binary GiB)	Size (Decimal GB)	PPL (Perplexity)
Q4_K_S (imatrix) THIS VERSION	104 GiB	111 GB	2.4130
Q4_K_M (standard)	111 GiB	119 GB	2.4177

ROCm is more efficient: For a full benchmark run, ROCm was 4.7x faster and consumed 65% less energy than Vulkan. Prompt Processing: ROCm dominates in prompt ingestion speed, reaching over 350 t/s for short contexts and maintaining much higher throughput as context grows. Token Generation: Vulkan shows slightly higher raw generation speeds (T/s) for small contexts, but at a significantly higher energy cost. Not efficient with CTX >= 8k. Context Scaling: The model remains usable and tested up to 131k context, though energy costs scale exponentially on the Vulkan backend compared to a more linear progression on ROCm.

Link to this quant on HF

Outcome from comparison between ROCm/Vulkan is simalar to that one i performed few months ago with Qwen3-Coder, so from now on i will test only ROCm for bigger context, and probably will use Vulkan only as a failover on strix-halo. Link on r/LocalLLaMa for Qwen3coder older benchmark

Cheers

submitted by /u/Educational_Sun_8813
[link] [comments]

Deepseek architecture, but without all the parameters

2026-02-10T04:16:13+00:00

I’m seeing a pattern that perhaps is not legitimate, but it seems everyone is copying the latest Deepseek architecture on their latest releases. In the process though they are also copying the parameter count (roughly), which makes the models inaccessible to most (unless you use their API or spent as much as you would to buy a used car).

So my question is, are there smaller models using the same tech but with less parameters?

submitted by /u/silenceimpaired
[link] [comments]

ACE-Step 1.5 prompt tips: how I get more controllable music output

2026-02-09T16:36:29+00:00

I’ve been experimenting with ACE-Step 1.5 lately and wanted to share a short summary of what actually helped me get more controllable and musical results, based on the official tutorial + hands-on testing.

The biggest realization: ACE-Step works best when you treat prompts as [structured inputs], not a single sentence (same as other LLMs)

1. Separate “Tags” from “Lyrics”

Instead of writing one long prompt, think in two layers:

Tags = global control

Use comma-separated keywords to define:

genre / vibe (funk, pop, disco)
tempo (112 bpm, up-tempo)
instruments (slap bass, drum machine)
vocal type (male vocals, clean, rhythmic)
era / production feel (80s style, punchy, dry mix)

Being specific here matters a lot more than being poetic.

2. Use structured lyrics

Lyrics aren’t just text — section labels help a ton:

[intro]

[verse]

[chorus]

[bridge]

[outro]

Even very simple lines work better when the structure is clear. It pushes the model toward “song form” instead of a continuous loop.

3. Think rhythm, not prose

Short phrases, repetition, and percussive wording generate more stable results than long sentences. Treat vocals like part of the groove.

4. Iterate with small changes

If something feels off:

tweak tags first (tempo / mood / instruments)
then adjust one lyric section

No need to rewrite everything each run.

5. LoRA + prompt synergy

LoRAs help with style, but prompts still control:

structure
groove
energy

resource: https://github.com/ace-step/ACE-Step-1.5

submitted by /u/Massive-Figure-9666
[link] [comments]

Free Strix Halo performance!

2026-02-09T17:54:41+00:00

TL;DR not all quants are born the same, some quants have bf16 tensors, which doesn’t work well on AMD as it seems, so find quants without bf16 tensors and you get anywhere between 50%-100% performance on both tgs and pp

Edit: I did some more tests, using -ctk bf16 -ctv bf16 degrades performance (in flash attention haven’t tried with fa off yet) around 10% for short contexts

As for with -fa off most models are similar (bf16 or not) with -fa on models without bf16 are faster (slightly although it depends on how much of the model is actually in bf16!)

So it depends on the model obviously not a generic boost

Edit 2:

‘’’

ggml_vulkan: Found 1 Vulkan devices:

‘’’

Strix Halo (gfx1151) doesn’t advertise bf16 in Vulkan backend, which confirms that the kernel doesn’t support models with bf16 tensors in some of their layers!

Long detailed version

I was playing around with different models on my new Strix halo PC

I have multiple quantized Qwen3-Coder-Next (I absolutely love this model)

I have two from unsloth two from lm studio and one from Qwen hugging face GGUF model page

When loading it I noticed bf16 in some tensors, and I know that KV quantization to bf16 isn’t good on the halo (in fact isn’t good at all as it seems!)

So I checked the three of them, unsloth versions have bf16 in them and so did the lm-studio versions

But weirdly enough, Qwens own GGUF quants have no bf16, I fired them up and voila they are much much faster

It seemed like a super power, and also not well managed in the community, I love bf16, but it doesn’t work well at all on AMD (idk why is it being converted to F32 for emulation, that is a waste of everything especially if you convert it every time!, weird fallback behavior to what, anyways)

And I wish I can know this piece of info before downloading a whole quant (I have most of my GGUFs from lm studio and unsloth, if I do this to every other model I might get a lot better models!, seems good but I also feel bad all of these hours were wasted before, anyways sharing for the community to spare others this kind of waste)

(How to know if a quant has bf16, load it with llama.cpp and it will show it at some point even before loading scroll and you will see it (how many q4 tensors, q8s, f32, f16s and bf16s !!!)

Good luck out there!

(I can’t wait to find a good REAP of Minimax M2.1 with Intel round that DOESNT have bf16 in it!, seems like the best model I can get and double current numbers it would be usable (20-30 tgs ?! And around 100 pp give or take, but a thinking model that is also parallel tool calling with interleaved thinking what else could I ask for ?!)

So cheers!

submitted by /u/Potential_Block4598
[link] [comments]

New PR for GLM 5.Show more details for the architecture and parameters

2026-02-09T13:03:45+00:00

https://github.com/huggingface/transformers/pull/43858

https://preview.redd.it/xbntmqm9wgig1.jpg?width=680&format=pjpg&auto=webp&s=da75a8dd1887ada367c9152cdeb13ad50fc6796c

https://preview.redd.it/wng50ssdwgig1.png?width=1323&format=png&auto=webp&s=65b30b4b03dc5c4ce8c63d4729121b22c56382dc

submitted by /u/External_Mood4719
[link] [comments]

LLaDA2.1-flash (103B) and LLaDA2.1-mini (16B)

2026-02-09T17:31:19+00:00

note: this is a diffusion model

LLaDA2.1-flash is a diffusion language model of the LLaDA series featuring the editing enhancement. It significantly improves inference speed while delivering strong task performance.

https://preview.redd.it/0zc0kqvw7iig1.png?width=1391&format=png&auto=webp&s=c9c347ed3fe4b69f50acf4af01e3d6f96ad616f8

https://preview.redd.it/biz1dmry7iig1.png?width=1372&format=png&auto=webp&s=0f9e9af10dae02d44553059f9654c8bc0683cf39

https://huggingface.co/inclusionAI/LLaDA2.1-flash

https://huggingface.co/inclusionAI/LLaDA2.1-mini

submitted by /u/jacek2023
[link] [comments]

GLM 5 is coming! spotted on vllm PR

2026-02-09T08:39:31+00:00

https://preview.redd.it/285aias7lfig1.jpg?width=680&format=pjpg&auto=webp&s=5287959d193fad4f96c5c80ec8b7546a7dcbe023

https://github.com/vllm-project/vllm/pull/34124

submitted by /u/External_Mood4719
[link] [comments]

Qwen3-v1-8b is Capable of Solving Captchas

2026-02-10T00:08:16+00:00

Qwen3-v1-8b is capable of solving captchas with semi-solid accracy... might need to write a simple python script that finds them on the page and uses the LLM to try to solve them and input the output.

Not sure if anyone else tried this before, just thought could be a handy thing for people to know, accidentally found it when passing it a screenshot

https://preview.redd.it/prijluyk6kig1.png?width=1038&format=png&auto=webp&s=29f55976839c594bd72eae9c2d0e6e2b9ce9a0d5

submitted by /u/TheyCallMeDozer
[link] [comments]

GLM 5 Support Is On It's Way For Transformers

2026-02-09T12:16:36+00:00

This probably means the model launch is imminent, and all evidence points to Pony Alpha on OpenRouter being a stealth deployment of GLM 5

submitted by /u/Few_Painter_5588
[link] [comments]

Femtobot: A 10MB Rust Agent for Low-Resource Machines

2026-02-10T02:40:21+00:00

I wanted to run OpenClaw-style workflows on very low-resource machines (older Raspberry Pis, cheap VPS instances), but most “lightweight” stacks still end up dragging in large runtimes and slow startup costs.

After trying nanobot and seeing disk usage climb past ~350MB once Python, virtualenvs, and dependencies were installed, I rewrote the core ideas in Rust to see how small and fast it could be.

The result is femtobot: a single ~10MB binary that currently supports:

Telegram polling
Local memory (SQLite + vector storage)
Tool execution (shell, filesystem, web) via rig-core

The implementation was done quickly with heavy AI assistance, so the code prioritizes simplicity and size over perfect Rust idioms. It works well on constrained hardware, but there are definitely rough edges.

Sharing in case it’s useful or interesting to others experimenting with small, local, or low-power agent setups. You are also welcome to contribute.

Repo: https://github.com/enzofrasca/femtobot

submitted by /u/yunfoe
[link] [comments]

New "Stealth" Model - Aurora Alpha - (Free on OpenRouter)

2026-02-09T18:00:10+00:00

New cloaked reasoning model dropped on OpenRouter for $0/M tokens

submitted by /u/-pawix
[link] [comments]

Who is waiting for deepseek v4 ,GLM 5 and Qwen 3.5 and MiniMax 2.2?

2026-02-09T19:54:09+00:00

The title? I hope they come out soon... I'm especially waiting for DS V4, it should be pretty good, hopefully it will be reasonably fast(probably slow though since it is gonna be bigger than v3.2) via OpenRouter. Well, glm 5 is out already technically on Open Router.

submitted by /u/power97992
[link] [comments]

Qwen to the rescue

2026-02-09T19:22:00+00:00

...does this mean that we are close?

submitted by /u/jacek2023
[link] [comments]

Step-3.5-Flash IS A BEAST

2026-02-09T23:35:09+00:00

i was browsing around for models to run for my openclaw instant and this thing is such a good model for it's size, on the other hand the gpt oss 120b hung at each every step, this model does everything without me telling it technical stuff yk. Its also free on openrouter for now so i have been using it from there, i ligit rivels Deepseek V3.2 at 1/3rd of the size. I hope its api is cheap upon release

https://huggingface.co/stepfun-ai/Step-3.5-Flash

submitted by /u/SennVacan
[link] [comments]

Kimi-Linear-48B-A3B-Instruct

2026-02-09T21:05:29+00:00

three days after the release we finally have a GGUF: https://huggingface.co/bartowski/moonshotai_Kimi-Linear-48B-A3B-Instruct-GGUF - big thanks to Bartowski!

long context looks more promising than GLM 4.7 Flash

submitted by /u/jacek2023
[link] [comments]

A fully local home automation voice assistant using Qwen3 ASR, LLM and TTS on an RTX 5060 Ti with 16GB VRAM

2026-02-10T01:39:23+00:00

Video shows the latency and response times running everything Qwen3 (ASR&TTS 1.7B, Qwen3 4B Instruct 2507) with a Morgan Freeman voice clone on an RTX 5060 Ti with 16GB VRAM. In this example the SearXNG server is not running so it shows the model reverting to its own knowledge when unable to obtain web search information.

I tested other smaller models for intent generation but response quality dropped dramatically on the LLM models under 4B. Kokoro (TTS) and Moonshine (ASR) are also included as options for smaller systems.

The project comes with a bunch of tools it can use, such as Spotify, Philips Hue light control, AirTouch climate control and online weather retrieval (Australian project so uses the BOM).

I have called the project "Fulloch". Try it out or build your own project out of it from here: https://github.com/liampetti/fulloch

submitted by /u/liampetti
[link] [comments]

Bad news for local bros

2026-02-09T13:14:31+00:00

submitted by /u/FireGuy324
[link] [comments]

Do not Let the "Coder" in Qwen3-Coder-Next Fool You! It's the Smartest, General Purpose Model of its Size

2026-02-09T17:23:31+00:00

Like many of you, I like to use LLM as tools to help improve my daily life, from editing my emails, to online search.

However, I like to use them as an "inner voice" to discuss general thoughts and get constructive critic. For instance, when I face life-related problems take might take me hours or days to figure out, a short session with an LLM can significantly quicken that process.

Since the original Llama was leaked, I've been using LLMs locally, but they I always felt they were lacking behind OpenAI or Google models. Thus, I would always go back to using ChatGPT or Gemini when I need serious output. If I needed a long chatting session or help with long documents, I didn't have choice to use the SOTA models, and that means willingly leaking personal or work-related data.

For me, Gemini-3 is the best model I've ever tried. I don't know about you, but I struggle sometimes to follow chatGPT's logic, but I find it easy to follow Gemini's. It's like that best friend who just gets you and speaks in your language.

Well, that was the case until I tried Qwen3-Coder-Next. For the first time, I could have stimulating and enlightening conversations with a local model. Previously, I used not-so-seriously Qwen3-Next-80B-A3B-Thinking as local daily driver, but that model always felt a bit inconsistent; sometimes, I get good output, and sometimes I get dumb one.

However, Qwen3-Coder-Next is more consistent, and you can feel that it's a pragmatic model trained to be a problem-solver rather than being a sycophant. Unprompted, it will suggest an author, a book, or a theory that already exists that might help. I genuinely feel I am conversing with a fellow thinker rather than a echo chamber constantly paraphrasing my prompts in a more polish way. It's the closest model to Gemini-2.5/3 that I can run locally in terms of quality of experience.

For non-coders, my point is do not sleep on Qwen3-Coder-Next simply because it's has the "coder" tag attached.

I can't wait for for Qwen-3.5 models. If Qwen3-Coder-Next is an early preview, we are in a real treat.

submitted by /u/Iory1998
[link] [comments]

MechaEpstein-8000

2026-02-09T19:57:33+00:00

I know it has already been done but this is my AI trained on Epstein Emails. Surprisingly hard to do, as most LLMs will refuse to generate the dataset for Epstein, lol. Everything about this is local, the dataset generation, training, etc. Done in a 16GB RTX-5000 ADA.

Anyway, it's based on Qwen3-8B and its quite funny. GGUF available at link.
Also I have it online here if you dare: https://www.neuroengine.ai/Neuroengine-MechaEpstein

submitted by /u/ortegaalfredo
[link] [comments]

Announcing LocalLlama discord server & bot!

2025-08-13T23:21:05+00:00

INVITE: https://discord.gg/rC922KfEwj

There used to be one old discord server for the subreddit but it was deleted by the previous mod.

Why? The subreddit has grown to 500k users - inevitably, some users like a niche community with more technical discussion and fewer memes (even if relevant).

We have a discord bot to test out open source models.

Better contest and events organization.

Best for quick questions or showcasing your rig!

submitted by /u/HOLUPREDICTIONS
[link] [comments]