I think that should be the blinking headline - this shows what can be done with consumer hardware.
It can run 80-100t/s on a laptop, can understand images natively and do bounding boxes, read tiny text, understands audio natively as well and can transcribe or translate anything you say, can do accurate long context retrieval with pretty large context windows, tool calling, excellent reasoning and is very token efficient.
It's only 7GB including the mmproj or 8GB with MTP. The Qwen 3.8 27B model Simon was using is ~18GB with MTP+mmproj, rather than 17GB alone. The point is not really that you compare these models directly, but that Gemma 4 12B QAT was really a special moment in model releases deserving of a similar reaction relative to its size, but was mutilated by Google themselves, Unsloth and Llama.cpp.
The overall appreciation I think we're seeing this year in particular is that people are easily surprised when multiple things are improving simultaneously which produce seemingly exponential changes. It isn't just that models are getting smaller, or that reasoning is getting better, or that speculative decoding is becoming mainstream, or that models can understand audio and images better now, or that they can reliably call tools which expands their capabilities, or that context windows are getting larger, or that accurate retrieval is improved, or that.... and so on. It's all of them narrowing in at once that is starting to make local models incredible and truly useful for far more use cases on the existing hardware people already have.
Even more impressively it doesn't have a separate mmproj at all — it is fully integrated, and the vision encoder doesn't speak words into the LLM, as it were —- it is directly integrated into the model's weights.
I have banged on about this model here enough but I really agree that Gemma 4 12B is a candidate for the most impressive LLM of the year. It is remarkable, and I think because it is a small model that isn't apparently excellent for long-context agentic coding, it has been largely ignored.
It is, actually, quite good at coding jobs. (Though its grasp of nuance is a bit weaker. For example, it doesn't know that closures created inside PHP objects have implicit access to the object as $this, and always seems to need reminding.)
If you instead treat it as a prediction of what consumer on-device AI may very soon be able to do, or even as a possible future into a sort of lower-ratio MoE, or the basis of a modest private offline educational LLM model, it's very interesting indeed.
I've learned a lot from it — the fact that it performs so well at such a small size really does help you assess claims made for much larger models, and it's quick enough on my M1 Max to just muck about with.
I do think the release of these models was somewhat fluffed up, and I don't think it helps that the 31B model uses global attention so it underperforms on the kind of older GPUs that are on a lot of desks; it's no better on those than it is on my M1 Max, where other attention schemes seem to be radically better.
Now that tool-calling is mostly fixed, it's well worth playing with them.
As for coding, for sure there are many important details that a model needs to know in order to produce correctness and the smaller a model is the more it ends up training out. If there's a task you do consistently enough though, often times you can simply provide a pile of essential context so it has good enough reference to not need the extra training data.
Gemma4 screwed up a proxmox install I had. I booted to a SystemRescue install and tried to get gemma4 to fix it. It just could not do it and kept having issues where it dropped a linux command into the local powershell because it did not ssh into systemRescue or killed the ssh connection somehow so the text landed on the wrong system.
I told qwen3.8 to investigate fixing the partition. It said information was lost, but displayed enough info that it was easy to tell it was right. I told it to install fresh proxmox and gave a short rundown on settings and partition sizes I wanted. It made a plan and told me I had to manually installed proxmox by booting the iso. I responded with something like "there are other ways to install promox without human interaction so use one of those". That was it. I woke up to the system having booted to a new proxmox install with my previous ssh keys restored and my existing zfs pool already mounted.
I don't see how any model that is limited to a single context window in a single session would be viable for coding. I want something that can manage the entire project and not just individual files or inline suggestions. I need to be able to feed it all the info I would use to make coding decisions and then have it at least make a working project that it can launch and test successfully. You want it to ask as many questions up front to enable continuous work without stopping for human input.
Unsloth modified the template and then finetuned their own version of the model to optimize for some benchmarks as a means of validating quants.
Google and Llama.cpp then adopt template changes by default, so anyone downloading the new model or even using the original model will now automatically be using it incorrectly.
Llama.cpp also uses the same inference setting defaults regardless which version of the model you use and some settings are simply defaults it uses for all models.
Then even if you account for all of these, you have to be using Gemma 4 itself correctly, which many people do not.
All of these little changes and inconsistencies hurt some of the model's original capabilities. Even if you go directly to Google's repo and download the full float 16 weights with the template they have there now, you cannot simply assume you're getting the best results.
When I tried to run llama.cpp directly I was getting max 9tk/s on qwen3.5-9B, then I tried LM Studio with the same model and got 77tk/s. I haven't figured out yet how to get MTP working properly in either.
Then create your own reasoning tests to verify that it is working correctly. You can set a specific seed value to make sure the generation is the same every time, that way you can identify any tokens that are different.
Afterwards, try making small incremental changes to the template and validate your tests each time in order to try to adopt the improvements from the newer templates. If the reasoning quality degrades, undo your changes and try again or test alternative solutions.
Would appreciate any kind of pointer to the latest!
That is a lot, what is your laptop hardware?
One issue I have with Gemma is that they seem to use old architectures that rely on full attention, requiring a lot of RAM for context and quickly degrading speeds as context is filled.
Qwen 3.5+ is much better in that regard with its super efficient context. Even on Macs, speeds take degrade much more slowly.
Yes, this is something I hope they will change. Gemma 4 31B is much slower on pre-Blackwell GPUs as a result, which is a bit of a shame for local model experimentation.
Of course it’s possible the labs just stick with the optimal architecture for large models and GDN is best for smaller models.
This isn't an isolated incident, really, I find myself always having these issues with the Gemma series. I'm sure they can do useful things for someone else, but for the things I want to use LLMs for (very small code generation, quick questions, code review) they always seem to disappoint me. I'm sure it's because of the stuff that I do and use, but it's a very consistent red thread with these models for me.
Edit:
The same question for Qwen3.6-35B-A3B produces a pretty concise and correct answer that would be useful to the questioner, without even going to the web. I don't know what Gemma models are trained on, but it's not the stuff that's relevant to me.
Is it multimodal? How do you do transcription with it?
Only up to 30s though, and the larger 26B A4B and 31B models are text and image only.
I keep hearing this claim, and yet I keep seeing LLM output which is trivially distinguished from human writing. I really can't understand how this gap persists; but then, there seem to have been at least some people who couldn't sniff out ELIZA, back in the day, too.
That's mostly true for longer LLM output with all the sycophancy / LinkedIn bias thrown in.
Make it casual conversation or comments, and give it instructions on appearing casual, or even better kill the censoring and fixed-prompt (with an open model), and it's orders of magnitude more difficult, unless if you suspect it and try specifically tailored prompts to sniff it.
There's no shortage of people obliviously discussing with AI bots in comment sections.
People, many of them at least, cannot make this distinction anymore. You can, I can, but people as a whole are having problems with that.
Even this (assuming it's even true) will likely not be true in some near-term future.
>continuously claiming this or that is a bot
I see it as a contemporary form of religious thinking. Like (say) pilgrims seeing blood on a statue of the virgin, plenty of people are now seeing the hand of AI in everything they read. If you want to see something hard enough, it tends to become magically visible.
Most users aren't very critical of the output. They just want a sycophantic ear, and 4o was perfect for that task. It's not _good_ but there is high demand for it.
From the abstract: "When prompted to adopt a humanlike persona, GPT-4.5 was judged to be the human 73% of the time: significantly more often than interrogators selected the real human participant. LLaMa-3.1, with the same prompt, was judged to be the human 56% of the time"
That's because they're trained that way. If you trained a modern frontier LLM with the explicit goal of passing the Turing test, it would have no difficulty doing so.
To me it just means they can brilliantly fake human conversation - the original design goal of Large Language Models.
It's really easy to tell if you're talking to an LLM if you ask a question that requires actually knowing things, not going for the first search result of a tool call or whatever most popular answer was embedded in the weights.
For this reason even the most sophisticated models still require system prompts, skills and all that other crap.
That's exactly the criteria we used to assume for over half a century for it finally being intelligent: the Turing Test.
And what does "brilliantly fake human conversation" even mean if not some kind of intelligence? It's like saying "He is not good at math, he just brilliantly proves theorems".
You can recognize 70% of those (true positive rate) and still have a false negative rate of 30%, while thinking you got 100% of the AI ones!
The problem is that you'd be oblivious to those you don't recognize.
Great times!
I was surprised and amazed to get "decent" (with the expectations set right / low) coding performance out of Qwen3.5-9B on a decidedly medium end Radeon 9070 paired with a 5700x3d and 32GB of DDR4 RAM.
We can finally reason with and "talk" to our hardware.
As a recent example, my /boot partition kept filling up. ChatGPT walked me through the root cause (leftover kernels after updates), mitigations (deleting older kernels), and future prevention (installing unattended-updates and enabling its config file).
As it is, it seems the improvements are about making the hardware cheaper (as in capex, not opex).
This is just feels from me from what I hear on the news and see on the products though.
I think really good efficiency is possible right now, but the GPU makers don't want to make their consumer GPUs too good for AI - if the cards were more efficient, it'd be much easier to run multiple - while data center ones have a bunch of additional power overheads.
Especially when chips are becoming cheaper (in the capex sense), then you can afford to only run them when power is cheap.
Btw, from where do you take the notion that performance per Watt ain't increasing? We are also still using what's more or less general purpose GPU hardware; we could get a lot further if we were willing to specialise more. Which would be the natural avenue to explore, if progress in general purpose hardware slows down. Google is already looking.
GPUs have been getting physically bigger with huge heatsinks and fans to support those bigger dies power consumption. Just compare the TDPs:
2020 RTX 3090: 350W
2022 RTX 4090: 450W
2025 RTX 5090: 575W
Bigger dies means lower capex of course, but the similar opex (maybe slightly lower as there is less physical hardware to maintain).
I seen some specialized hardware like google's TPUs. Not sure how they compare on performance per watt with GPUs though. Regardless the manufacturing processes are still the same (EUV) which is the thing that hasn't been improving. A fully optimized specialized hardware can at most deliver a single-time linear improvement (that could be very significant, for example 30% is still huge of course) and then little compared to normal GPUs.
I don't think renewable power generation is going to massively reduce costs for data centers, especially considering power transmission hasn't meaningfully reduced in cost. If anything the only thing that I think will have significant impact for data centers would be dedicated nuclear power plants physically located right next to the data center.
In fact I expect power generation to get more expensive as demand can increase faster than supply can be established. I imagine setting up new solar farms and transmission lines to be significantly harder (as in, takes longer time due to approvals and so on) than new data centers (which requires a single large location and I assume less approvals).
I don't think the main use case is for a human to directly interact with the raw token stream. You probably want reasoning and you want the thing to be able to program on its own. That uses way more tokens than you can read.
You can also look at what's been happening in mobile and especially with Apple's integrated processors. They are more power constrained, so people worried more about power there.
I think raw flops per watt come mostly from fab process, not architecture. This was my original point, fab process is not getting better at a linear (much less exponential) scale anymore.
The next 10 years of computing is going to be VERY interesting. May be not just computing but everything, an even wider reach and disruption than iPhone.
Unlike iPhone / Smartphone which I thought was obvious, a computer or web browser in your pocket that basically extends the internet to anywhere. Local Model AI in our computer or pocket that controls other things opens seemingly unlimited possibilities.
I initially thought it was going to be 15-20 years time frame, with perhaps a bubble burst in between. But development of both software and hardware is accelerating.
I ask because Claude is fun for rewriting abandoned code and I am not a proper developer so it's been great for me. Claude refuses to answer questions about science and medicine that stray outside of the officially supported narratives of the AMA and I have issues that have surpassed anything a doctor can do so I am entirely on my own. Will the self hosted models answer such questions or will it also try to put walls or bumper guards around topics?
I personally like the https://huggingface.co/HauhauCS version of Qwen 27B from a purely subjective point of view as a user, but that particular one has come under criticism for reasons that don't necessarily affect its quality or usability.
Some of the larger models have also undergone similar treatment, but it's less common.
For the most part, open base models are corporate releases with somewhat similar guardrails to commercial hosted models (not quite as complete, because hosted models tend to have a combination of trained and external guardrails applied); but no one is monitoring and trying to terminate your account for using jailbreak prompts, and there are often community finetunes available that (among other things) weaken the trained-in guardrails.
Of course, even if the model does answer, it may nto answer according to the particular worldview that produces hostility to the “narratives of the AMA”.
The over reasoning that Simon Willison highlights here is a real issue though. I've observed it with some of the OpenAI models as well. They are prone to overthinking and overengineering things.
What I would love is models that figure out their own appropriate reasoning effort given a task. I'm spending too much brain cycles worrying on what model speed, reasoning, and quality settings to pick. It's not just a cost concern it's also a time concern. Wasting a lot of time for simple UI tweaks because the model is set to high or ultra or whatever is counter productive. The last few iterations of frontier models seem to emphasize benchmarks and reasoning effort.
But of course the day to day reality of many developers is that they are trying to solve relatively simple problems compared to e.g. proving some so far unproven theorems, solving some Nobel prize level problems, etc. I'd love my tools to start making sane choices based on what I ask rather than defaulting to "boil the oceans". These tools need some kind of Auto select. Mostly Ultra is overkill and a waste of time and resources. And of course with local models, keeping simple things local is a nice option.
It's nice to have Sol Ultra extra fast as an option in my back pocket. But it's complete overkill 99% of the time. And it's not like most users make good choices here or are even capable of making good, informed choices. The models are more intelligent than the tool UX. Arguably, a local model of very modest size might be able to do better for this specific choice.
If you want a better experience, maybe wait for either a moe model (like 3.6 35b A3) or a model with less parameters (like 9b). Qwen has been releasing those in the past, so maybe we’ll have them for 3.8 too.
For example make an essay about something where you don't actively engage with the LLM after the initial prompt. So mostly one-shot prompts.
> It feels pretty slow on both the M5 Mac and the DGX Spark.
Probably stuck in prompt processing which is compute bound especially for iGPUs.
You've mentioned 3.5 - but it's actually the same model the only differences are training and implicit MTP support (affects prompt processing - can be disabled)
Or the "<|think_xhigh|> | <|think_low|> | <|think_off|>" tags: apart from this template detail, it is not immediately clear if reasoning_effort is deterministic (API) or is prompt engineering.
Besides, Macbooks with 32GB RAM is consumer hardware, just maybe on the higher end.
As soon as I switch to a model that doesn't fully fit into vram it tanks to <10tk/s which makes it unusable for me for most tasks.
for inference the compute is the last thing we need more of.
memory bandwidth is the numebr one blocker, after that the inefficiencies that where introduced with MoE models (and all new large models are made that way)
Here is a quick read: https://news.ycombinator.com/item?id=49324600
I've gone in (too many) details last night with the calcs: https://news.ycombinator.com/item?id=49324600
From my reading of the Fable 5 and Opus 5 System cards, my reconstruction is something like:
Finish the task → make externally observable evidence that it is finished → check your own work → fix problems → don't stop prematurely → satisfy the evaluator comprehensively.
That is fantastic for SWE benchmarks and autonomous agents. It also naturally creates pathologies:
under-answering is expensive; over-answering is cheap.
For a complex maths problem, Sol reasoned in 367 tokens before working:
We need solve. Need interpret no person sits next to two other people = among binary string length16 weight8, no occupied chair whose both neighbors occupied, equivalently ab 111 substring. Endpoints cannot have two neighbors anyway; only avoid 111. Count binary strings length16 weight8 avoiding 111. Need N mod1000. Compute stepwise perhaps runs of 1 length max2. Count via runs.
[... cut in half for HN readability ... ]
Check interpretation potentially "no person sits next to two other people": no seated person's chair adjacent to two occupied chairs. Exactly no three consecutive chairs selected. yes.
Need reason step by step final boxed 907. Explain runs. Ensure people each select chair distinct subset (given subset count). Let's present.
I think one of the bigger problems is that verbose, `max`-style thinking does generally lead to higher benchmark scores. And model vendors are incentivised to for benchmarks (at least to some extent).
(Note: there's still a huge grammar penalty, so, ugh do think small.)
I can imagine a curve where for a given level of “intelligence” you either need model size, or inference time (“test time compute”), and can somewhat trade one for the other.
This is exactly what was shown on Luna/Terra/Sol tradeoffs - Luna requires much higher reasoning efforts to approach Terra/Sol on lower reasoning. Which is fine, of course, no complaints - but true.
So it's not surprising that the same dynamics are at play with AI. Now, because code is being churned out so rapidly, the effects have become much more obvious (it took me years to figure this out, but now managers can observe this same effect play out in months); many senior engineers and CTOs will echo my point; but I suspect most engineers and crucially, most managers, still don't get it...
Something tells me that the AI companies supplying the models are well aware of the tradeoff. When you can dial up the complexity of the LLM's output by 5% (I.e. 5% more tokens to solve the same problem) and see a 5% immediate increase in your revenue from a large segment of your users, that's a very tempting knob to dial up! Now when you learn that this complexity compounds and next year's revenue will be 10% higher (purely as a result of your users now having to maintain that additional complexity); this is extremely tempting! Especially in the context of users who are largely ignorant as to the true cost of the unnecessary complexity they are adding... The insider's term for this is 'technical debt' for multiple reasons; including the fact that it compounds like normal debt. Now factor in the monopolistic tendencies of those tech markets... It's a real bottomless gold mine.
Revenue from downstream corporate users comes in regardless of compounding code complexity and slower pace of delivery; those companies just keep hiring more people, spending more on tokens. Swallowing up these massive diminishing returns like an appetizer. Enshittification takes place but the downstream end user has nowhere else to go.
It's trivial for a lab to advertise themselves as being token-efficient and almost impossible for its corporate users to actually verify it.
The only real issue with that business model (possibly a fatal issue) are these open weights models which the big tech companies could use to move off the AI service platforms if the problem becomes bad enough.
Fork can be found here - https://github.com/laurencehardman/llama-mindcontrol/tree/ma...
Of course hacks like this are not perfect and may degrade performance slightly due to injected text pushing the model slightly out-of-distribution, so the string constants need to be chosen carefully - Qwen3.5's technical whitepaper does provide some guidance in this regard. The mechanism is absolutely more of a hack than a feature, and i'd imagine will be made redundant once llama.cpp supports more appropriate reasoning controls - but for now, i've found it pretty useful.
I did make a PR to the official llama-cpp repo some time back (about a month or so), but abandoned it as there seemed to be too much community concern that the mechanism would degrade model performance... Perhaps i'll polish it up and put some effort into benchmarking and revive the project in the near future.
I hope this trend continues.
* Many businesses don't need frontier level intelligence anyway.
* It's completely stateless. If your local LLM machine catches fire? Nothing was lost. Buy another.
And before that, businesses will be able to get decent results with dedicated inference hardware.
I could absolutely see local AI taking the place of voicemail/call screening completely. Call me and you get my AI, who will route the call to me if approved, or even choose to service the call itself. If a friend of mine calls who doesn't have $20K to spend on a great AI rig, they would certainly have permission to steal otherwise wasted cycles from mine. An answering machine isn't much different than an issue tracker, and some of those are 97% AIs having perfectly intelligible conversations with each other, and 3% humans being eagerly serviced by AI.
The old objection was "this is so hard to set up, nobody wants to run a server!" Now it can set itself up.
tldr; you don't have to rent from some data center. You can get high utilization out of a local box. This 1) puts a hard ceiling on what a data center can charge, and 2) they won't be able to compete on privacy, which locally can be complete.
The future is not a single chat bot session of bs=1. The future is many agents performing many tasks in parallel for a single user. Large GPU clusters will always have the edge in efficiency.
Agentic AI has pluses and minuses for cloud efficiency. The plus is that usage could be very bursty, but the minus is that agents will more fully utilize a local system. The main disadvantage of local AI is that you would be paying a large amount for a system mostly doing nothing. If it's constantly working on different projects and integrating that data, you get use out of every penny that you spent. Every GPU you added would instantly make the thing smarter.
What's more, your local AI could offload an agent to the cloud if it needed to. It could do this rationally, based on your personal desire for privacy.
The abstract benefits are quickly outstripped. The same way adding more highway lanes never improves gridlock. Its inducement.
Large orgs with significant demand might go out and buy local LLM hardware, but most businesses probably don't want to bother dropping $2k on a box with a beefy GPU and would rather just pay the lowest subscription tier so their employees can occasionally make queries.
Plus, you know, the whole economies of scale thing. Local LLM has a lot of privacy and independence benefits, but I'm not really seeing the world where it becomes more energy- or cost-efficient to buy your own hardware (and use it 1% of the time) versus sharing a giant machine, or even the same machine, in a datacenter (where it has a much higher utilization factor).
You may have better and more reproducible results using browser controls that aren't image based, or writing tools that completely sidestep browser use.
I will say one thing that frustrates me is the opaqueness of the billing model. I basically just have to pay a random amount. I guessed that it navigating the web is relatively pricey from watching my usage as it does stuff. And the whole thing is worth 10x what I pay for it anyway, it just would be nice if the pricing were somehow more transparent, even in hindsight. It can explain to me what is thinking as it does stuff, it could also explain the tokens.
Marketing definitely. My food truck side is relatively high volume and it manages my kitchen and warehouse side, basically generating all of the instructions my employees follow, managing and updating my PoSes, creating signage assets for specials, etc.
Reels/posts production and managing ad spend.
Here’s a fun one. I switched payroll providers after several years and suddenly my unemployment insurance rate went from 0.8% to 12.75% which is borderline debilitating to me. I knew something was wrong but not what and I work a lot of hours and calling the state takes forever and is usually unhelpful.
ChatGPT figured out that it was a penalty rate and dug in for me. Turns out because I’m seasonal and have no payroll for one quarter of every year, Gusto did not file a quarterly wage report, so even though I owed nothing I was delinquent. Gotta love government, it’s the only place where you can be delinquent for $0.
ChatGPT filed the report and requested a retroactive re-rate, which they granted. I’m sure I would have figured this out eventually but it would have taken hours, or $5 in tokens.
Would you say ChatGPT does everything you need today or do you still see some gaps? i.e. things you think it should be able to do but currently doesn’t, or things that still take too much effort on your part to setup chatgpt to do it.
But, I think I’m on the black on it already after just a few months of heavy use. For instance I just tell it to book my dumpsters, bathrooms, sanitation crew, and security for X event. It goes and pulls event details, looks through my email to see who I get those from, and emails them relevant details, with no prompting.
Another good example: we launched a really fancy hot cocoa concept last fall that was a hit and I wanted to try to go to all of the local pumpkin patches in October and Christmas tree farms after Thanksgiving to serve when they have big crowds.
I asked it to contact all of the ones in my area and it sent out 70 emails and I booked several spots. It made me a nice database so I can see who followed up and who I need to reach out to again, etc. and it just gets those from my inbox.
Hours of my time saved with simple prompts. So while some things take awhile to pay off, some are instant.
The AI is good enough to do a lot of tasks but the tooling just isn’t caught up to it yet.
I’d say it’s freed up ten hours a week of my time. And that’ll only improve.
Seriously, you can do years of Deepseek inference for the hardware to run just 1 or 2 requests against a slower, dumbed down model on your own hardware.
It makes no sense to buy hardware right now, when the price is completely disconnected from any material reality. It's much better to use the cloud providers VC funding by using their cheap as F offering. Either AI becomes less useful, or hardware costs come down. Either way, you'll be in a better position in 3 years than you are today.
Yes though anyone with an aws bill knows that’s sales pitch lies
The bill only goes one direction and it ain’t down
The economies are currently out of whack because of underproduction of components and memory, so LLM providers have a few years of runway to entrench. Plus, the whole capex Vs opex thing that helped AWS will help here too, for sure.
At some point, though, things will change. More production will come online, and providers will have to end the current speculative subsidizing and jack up prices.
It's a bit like the dot-com era: the initial rush to land-grab web portals and e-commerce sites eventually died, once enough skills and infrastructure came online, and the bubble burst.
- Hyperscaling “we are going to serve billions of people in our applications”, which is becoming increasing unlikely as regional tech companies become more dominant than than the global one (this one is as much about geopolitics as technology)
- Operations is hard, in which case non-frontier models should be increasingly capable. Devops for small-ish deployment is one of the few cases where it is hard to clam you need deep expertise and AI can’t do it. Previously, the claim is that you need people specialized in ops, which is expensive. Now…
My prediction is that not just cloud LLM, but cloud business general will have to change. Not yet in the next 5 years, but probably 8-20 years-ish
They will die, for some definitions of death -- I don't think they disappear, but they should be a niche, rather than the dominant doctrine.
I did testing and the reasoning effort can be set per message, I was not aware of the option of none mentioned by @xscott, I tested but didn't see any change, I think there are just 3 values, xhigh, medium and low as per https://huggingface.co/Qwen/Qwen3.8-27B-FP8 , I did testing and the thing can do it's "I'll speak 10 million words to myself to ensure I'm not missing something" and then switch to a faster model, then switch... I did a test and the thing keep coherence and follow it's train of though-kens, you can see the result here... https://github.com/alainnothere/llama.cpp/blob/disk-cache-ev...
What if instead of a latent they used at a shorter form of note-taking-like reasoning, using more symbol to achieve a denser CoT. We'd get the best of both worlds. WDYT ?
I think https://arxiv.org/abs/2412.06769 is a good description of the premise of reasoning in latent space, although https://arxiv.org/abs/2604.15726 argues it's already what really happens.
It's as if the only way you could think was by writing down a word, erasing all the thoughts from your head, then reading the word you just wrote down and deciding on the next word, etc.
Reasoning purely in latent space means that the model would still produce an output equivalent to tokens but unconstrained e.g. the output could be raw and opaque vectors. A significant downside is that you lose the ability to inspect the reasoning trace. It would also make the reasoning trace potentially larger which has operational issues.
Not completely true: KV is a projection of the activation at each layer's input, so attention heads see (a representation of) all previous tokens' activations at that layer. The hard decision at the LM head doesn't change that.
You can see evidence of this phenomenon in models dating back to the OG Deepseek R1. It was common to see the model talk itself out of the correct solution in the <thinking> block, or fail to reach it at all, only to produce a correct answer in the response. And vice versa; it was also common to see it reason its way to the right answer and then fail to follow through in the response.
I can see the reasoning being a substrate for computation, but in which space should we interpret this computation to be happening? The vector representations of individual tokens are completely different (and even the way the reasoning traces are broken up into tokens will be pretty different) between Qwen and Claude e.g.. The only way I can see this being effective (which it is) is thus that we SHOULD interpret the model to be "computing" in natural language and thus we can indeed take the chain-of-though somewhat literally.
The Deepseek R1 behaviour you describe is from a model from january last year, are you sure this is not pathological behaviour rather than an indication of the reasoning not needing to be taken literally?
I do however agree with the point that it is not necessarily a dead-end. That Qwen loops almost at an OCD-like level, but retains accuracy on the times it does answer, shows that. Yes ideally it loops less, but I am for now happy to accept that this is what it takes to run models locally. At least it is available for our inspection.
Fine tuning or post-training is effectively biasing certain outcomes: making them more likely to occur. This comes with trade-offs. A coding LLM will bias technical language, which would harm a model for general use.
This opens a really interesting field of research. Our brains use specialised regions because specialisation turned out to be the most energy efficient method for biological compute. It might also be the best performant. We don't want to activate 100% of our prefrontal cortex to breath. What a stupendous waste of the organ. I think we see incredible advancements in model clusters in the future, using specialised models for specialised tasks. We have the appearance of this today in some harnesses, but they are shallow imitations. The real innovation will be low-cost, accurate routing. Existing solutions are woefully inadequate for many reasons.
Now though I'm considering all the hidden "thinking" in the models layers that happens for each token output. It is a wild amount of waste! We just can't see it.
This kind of stupid excessive computation is fundamentally how these models are so good.
One day hopefully not so soon someone smart or a foundation model will come up with a more efficient architecture. That's when things get really scary.
The important part of "actually wait, I really need to XYZ" is just "XYZ".
The model can attend to just "do XYZ" and produce almost the same vector modifications as full verbose "reasoning".
It's good to remember that LLMs have no more state then what they can derive from the context up til any point. So if that context is hard to interpret, that will reduce effectiveness.
LLMs are trained on human natural language, not a specialised internal-only monologue to make syntactic shortcuts. Their response should make grammatical sense to a human reader because they are mimicking human speech.
"foo bar" is ambiguous.
"pursue theory foo; no, this didn't lead anywhere, let's backtrack and pursue theory bar" is accurate, should be meaningful to LLM attention, but is too verbose.
The minimal caveman way to say this is "not foo. instead do bar".
This really should provide the LLM attention with everything it needs to grasp the intention, but with far fewer tokens used.
Instead of all this I could have just said:
"nothing ambiguous. clear but less words"
And this memory control primitive leaks into the reasoning chain, because it has no other channel for it available and we do not know how to train any other channel.
On the flip side, it tends to converge quickly, roughly proportional to the actual difficulty / clarity of the task.
I am very confident the reason we get all these second guessing and "but wait" and "actually" is they train them on collapsed corrected sessions. i.e they take sessions that look like this:
user: Do x.
agent: the user wants me to do x. I think I need to do a and b first.
agent: does a.
agent: does b.
user: No no no doing a was wrong you should do c before b.
agent: undoes a. does c.
agent: does x
user: do x.
agent: the user wants me to do x. I think I need to do a and b first.
agent: but wait maybe I should do c instead of a
agent: does c
agent: does b
agent: does xFor instance, it's a step backward, but I put {"reasoning_effort":"none"} and led it by the nose:
User: We're going to make <silly demo>. Please create a plan, but do not write code yet.
Agent: <short and reasonable plan>
User: Now please follow that plan and write the code. No other chat.
Agent: <reasonable code in reasonable time>
trying to reason about all the ways it can go wrong after a point just stops one from starting the task. Which is exactly what I find with models.
It will actually adhere to your request for e.g. 3 sentences max.
Thinking mode will override any instructions in the prompt (at least for other models in my experience).
Of course this will probably hurt performance, but works great for easy tasks that you know are trivial. Tons of pipeline, image recognition etc use cases where this works well.
I'd be curious to see Qwen 3.8 27B low thinking benchmarks though.
SOTA cloud models are more for open-ended tasks where you need "creativity".
The fact that it runs at 15tk/s in power saving mode, and 30 in perf. mode blows my mind. I can run the model in the background, coding something for me in OpenCode, hosted in LMStudio, while doing something else. What a world we live in.
Having something close to human intelligence (at least for reasoning and code), running on a laptop, is amazing.
The result appears to be almost as good as Qwen 3.6 35B A3B on medium thinking mode.
It second-guesses a little, it gives broader/more speculative answers, of course, and it missed the nuance of one of my prompts, but this gives me a lot more confidence that the Low reasoning effort is going to be as good as they say, and perhaps in some cases non-thinking looks like it would be enough.
Really useful, thanks.
Qwen 3.6 doesn’t have configurable reasoning effort, does it?
I'm talking about budget — I mean limiting it to 2048 tokens.
… for one or other of the models I tested at the same time, in llama-server, there is a dropdown that offered options (unlimited, max, medium which was 2048)
(I've tested so many of these things now that they are beginning to blur.)
I thought that was llama-server with Qwen 35B, just checked and it's not. Nor is it Gemma 4 26B. Perhaps it was Ternary Bonsai which I tested again and deleted earlier.
Anyway I took to clipping Qwen 3.6 35B at 2048 tokens reasoning in LM Studio and elsewhere, and it did OK at that (because it often loops like mad on an ambiguous prompt if not curtailed).
FWIW I just rechecked outputs and I am a bit over-optimistic when I say 3.8 27B 's non-thinking output is that good. I spotted a couple of subtle errors in my tests that Low thinking didn't fail on. It is good, but it is not quite Qwen 3.6 35B thinking level.
If you already have one or more of them, then, yeah, you can use them for this model or any other at around this size or smaller, but I wouldn't recommend you buy them (or anything else, right now, everything is just too overpriced). You can run better models for less money at higher speeds. I bought mine before they got more expensive, but I wish I'd just bit the bullet and bought newer/faster cards before they got more overpriced. Or, the actual smart money, even back then was to just use cloud models and forget about self-hosting.
This model is far from usable on current AMD or Nvidia 128GB AI machines, IMHO, they just don't have the memory bandwidth, especially since it chews so many tokens for any task. If you want to run this specific model, two (or more) 32GB GPUs with decent memory bandwidth is the right way to do it. It doesn't benefit from the larger memory of the Strix Halo. There's enough room for full context and 8-bit quantized model in 64GB. But, it's really a terrible time to buy hardware. MoE models are a much better fir for the Spark and Strix Halo; you can run Laguna S2.1 (slowly) or one of the Qwen 3.6 MoE fine-tunes (pretty quick). Ling 3.0 Flash also looks promising. Nemotron 3.5 Lightning in the MXFP4 quantization absolutely flies on the Strix Halo at 65-80 t/s, but it's dumb. But, all of those are weaker than Qwen 3.8 27B for coding.
I think a factor of 2 or 3 is a small price to pay for the best result.
But, ~256k is all I can fit on my dual 32GB setup, and it gets slow as heck at 256k, anyway. I think just convincing it to chew less and do more is probably the right solution for Qwen 3.8 27B, though I guess we need some benchmarks to get the feel for how much intelligence you lose by choosing a lower reasoning level.
I’d want to compare this to the new Muse 30B model which is super terse and has a whole different way of thinking (no “Wait,”) and in my experiments was way more token efficient to the point that the absolute tok / s didn’t really matter.
Qwen: https://gist.github.com/simonw/121ad098860028b2fab603fa12da1... - 17,576 reasoning tokens, produced this HTML result: https://static.simonwillison.net/static/2026/qwen-over-think...
Glimmer: https://gist.github.com/simonw/51e8ddb2ee597a5005fa63bd4927d... 1,021 reasoning tokens, this HTML: https://static.simonwillison.net/static/2026/glimmer-bbox.ht... - ugly but functional.
In both cases paste in the URL https://static.simonwillison.net/static/2026/two-pelicans-on... to see them work.
Both applications work correctly and fulfill the requirements. The Qwen one (which used the default xhigh reasoning setting) is massively over-engineered. The Glimmer one used whatever their default in LM Studio is and I would argue is a tiny bit under-engineered.
Weirdly the Glimmer one doesn't work with images on other domains like https://static.inaturalist.org/photos/714731804/large.jpg - it fails with a CORS error, but you don't need CORS to load images and detect their width and height, and the Qwen one handles that URL just fine.
That's because Glimmer added this unnecessary line:
img.crossOrigin = 'anonymous';I like the style of glimmer more. Much terser language, no adjectives, no fluffy claude-like language. ("Images are written to...", "Tasks are stored in SQLite...", "Docker image is built from ...")
In contrast, qwen is a bit more flowery. ("Unbounded image processing / resource exhaustion — preprocess() opens whatever was downloaded with no size/dimension/format validation before the VAE encodes it..." , "SQLite as a queue — fine at this scale, but...", "Debug info leakage — exceptions are re-raised as...".
But both flagged pretty much the same stuff, just ordered / styled differently. Mighty impressive understanding for a thing that I can run locally. Qwen served in fp8 w/ full kv cache, glimmer in w4a16 (the fp8 weights wouldn't serve for whatever reason), both at full supported context in 48GB of VRAM.
Do fast, deliver.
Thinking is good.
You just don't see it in proprietary harnesses because it's literally cryptographically hidden from you.
Worth noting as well that the weights come with an MTP layer that seems particularly accurate, to the point it can give you up to 6-8 correctly predicted tokens consistently enough to be useful. Which obviously boosts its speed enormously.
I find it difficult to believe how good this model is. It feels like it's lagging heavyweight frontier models by less than a year, and it runs on your PC.
https://gist.github.com/nharziro/aed0c364ce2f295a493494c6f1b...
the message can be some combination of tool calling, summarizing, etc. It's overthinking often is a bunch of recursion, so simply stopping t and redirecting is all you need to do.
If someones building a harness for llamacpp, you can set this per message, so it's possible to dynamically control it by watching for the expansion of the thinking traces, and redirecting it.
I use the message to tell it to use subagents, add additional logging and to use opencode's dynamic context pruning.
As such, we'll just whisper here _skill issue_.
Probably the better solution if you want it to be quicker but still fairly thorough appears to be to configure reasoning effort instead of thinking budget. It seems to do very well still even on the Low setting; on the Medium setting it can get stuck in loops like 3.6 does.
I think xhigh reasoning effort was an absurd choice for a default, and so was not sorting out the chat template so LM Studio could offer the reasoning effort dropdown.
The only think I could think that'd be better than the --reasoning-budget would bet a budget jitter just in case it really is repeating a pattern and you want to escape it arbitrarily, otherwise yes, it could keep looping if you're always cutting at the wrong time.
Yes, I think I finally have an intuitive sense for that. But surely on a longer prompt it is still better for the final response if the thinking has at least brushed past all of the prompt?
One of the things I witnessed with xhigh is that while the thinking trace starts out intending an overview of the prompt, it actually can go fully down a rabbit hole off one of the first two or three bullet points even when it was seemingly intending not to.
It’s basically a lot like me. Gets sidetracked by the interesting bits.
I guess if you're literally in a greenfield project, at the start, and it has nothing else to look at, then you're right; but everything I've been doing is taking existing stuff and improving it.
So my advice might have a bad edge case that only exists immediately. My setup is to tweak and improve on existing code bases so there's almost always something it can review, write to, read, etc that'll drive it to where I'm trying to go. So even in a greenfield project, it's start to do something rather than overthink everything. Those thought traces don't suddenly disappear.
There is the "muddled middle" of context that might get trapped in, but I don't think anything you can really do will stop that.
It is sort of frustrating and unnerving to me how much of this is dependent on application and usage, but it is occasionally fascinating.
I've learned that medium effort can improve the outcome relative to higher settings. But I suspect the phenomenon is an artifact of a misguided effort to fix inherent LLM limitations. At least some of its reasoning will miss the target, and more bad reasoning is not the remedy.
After about 10+ hours of digging, it has apparently found a bug in the Next.js framework, with an example app that replicates the bug, and a fix for now to disable prefetch in the Link component.
I had in my prompt asked it to discover the root cause of the bug and propose a fix, but I did not expect it to dig this deep.
edit: and which quant you are using, please :-)
Quantization is Q4_K_M (4-bit K-quants, medium)
From Pi, these are the up/down token counts:
Tokens: ↑147k ↓110k R26.5M
Prompt submitted at 18:52:54 BST on Sunday 16th August 2026, and finished at 04:05:27 BST on Monday 17th August 2026.
Last print out from the llama.cpp server logs:
881.42.815.480 I slot print_timing: id 2 | task 110233 | prompt eval time = 2742435.76 ms / 134911 tokens ( 20.33 ms per token, 49.19 tokens per second) 881.42.815.483 I slot print_timing: id 2 | task 110233 | eval time = 6312248.28 ms / 5804 tokens ( 1087.57 ms per token, 0.92 tokens per second) 881.42.815.483 I slot print_timing: id 2 | task 110233 | total time = 9054684.04 ms / 140715 tokens 881.42.815.484 I slot print_timing: id 2 | task 110233 | graphs reused = 114795 881.42.820.381 I slot release: id 2 | task 110233 | stop processing: n_tokens = 140714, truncated = 0
The model ID appeared in some alibaba PR but later disappeared.
It's the optimal blend of accessibility and model size for a lot of people.
Low and medium are basically the same.
Also, the electricity it costs to run on a 3090 is not negligible, so that it's cheaper to use Luna high via API than Qwen 3.8 27b locally, hardware costs excluding.
[0]: https://aibenchy.com/compare/qwen-qwen3-8-27b-high/qwen-qwen...
If I ever found myself in this situation I would much rather just rent cards from hotasile and run open models instead of giving OAI money and playing reset bingo
https://ec.europa.eu/eurostat/web/products-eurostat-news/w/d...
This is what I pay extra per kWh,in Netherlands: € 0,25162 ($0.291), excluding the connection costs.
Also, Qwen 3.7 27B is actually Terra level.
That might be my fault, I am not a designer, and I asked for most UI decisions.
I tried to use colors to diferentiate models, the site is very data dense and it's hard to make everything readable.
I've spent hundreds of hours building it, not sure if I would call it slop, but I just suck at design, lol
Any suggestions on how to improve it?
Have a look at https://diskprices.com/ - this is great UX because it's extremely functional, yet it has nearly no design to speak of. On a scale of data dense designs, this one would be on the extreme side, so I guess if you ask the LLM to meet you somewhere in the middle.
- the spotlight search, you can do ctrl+k and search "gpt vs claude vs google" and it automatically shows the most relevant comparison
- drag to reorder compared models
- filter leaderboard by multiple comma separated needles, so you can see for example all qwen and kimi models only
- the showcases page is quite cool
A lot of invisible UI/UX features are there, which I think work great, only the visual presentation could be, as you mentioned, a lot better.
I checked out that link on mobile, but it's unfortunately unusable. I will check it out later on desktop, thank you.
What gets reported is always the benchmark result, but rarely the real-world trade-off made to achieve it. That’s an obvious incentive for the labs, so I think Simon is correctly zeroing in on it. Please continue doing so for models that don’t go too far as much as this release.
Don’t get me wrong, I think it’s amazing what we can get out of smaller models with more reasoning, but we should be super aware how very much not-free it is.
This is a good opportunity to call out models that reason quickly: Meta’s Glimmer seems to be pretty token efficient so far, as do the GPT 5.6s.
Reminds me of Gemma4 and the official (or at least popularly used around launch) Jinja templates being wrong and broken for tool calling.
All the positive PR from "Opus 4.6 level" online buzz is well worth the minor annoyance from taking a half hour to solve a simple problem since a user just needs to turn down the reasoning knob if it bothers them.
It’s a similar level of dishonesty as trying to conflate “starts at” vs. “tested configuration” car prices.
We change the incentive to do this by evaluating it exactly as advertised.
I would love to give a local-LLM agent full access to my (very modest) homelab, but can't trust it not to delete my files or do something crazy. Maybe giving it its own non-root username, and therefore read access, allowing diagnosis but not fixing. But even a non-root user can do some damage, or exfiltrate system info just by using curl.
i just run a spare pc for browsing and screenshots with proxies.
I did set the default at medium, but the xhigh still seems to be a big part of its benefits.
I found it really easy to chat with it and instantly see flaws in its thinking when using the medium level of reasoning... in a way that had me consider that the frontier models reasoning abstractions (both at the harness level, e.g., claude code, codex) AND in the server-side obfuscation) that I found refreshing, because it made it easy for me to step in and precisely identify the failures of reasoning that the frontier models were getting stuck on, and because of the 'black box' hiddenness of their reasoning, it made it harder for me to diagnose.
So, I suppose I'm saying 'there is a time and place for each'.
Here is my setup: https://williamcallahan.com/blog/qwen-3-8-27b-is-a-great-ope...
stdout: Benchmarking model: qwen3.8:27b
=== A) 32k ctx, short prompt === Context window set to : 32768 Input (prompt) tokens : 32 Prompt processing : 68.5 tok/s (0.47s) Output tokens : 80 OUTPUT SPEED : 42.45 tok/s (1.88s) Wall-clock total : 3.9s
=== B) 65k ctx, short prompt === Context window set to : 65536 Input (prompt) tokens : 32 Prompt processing : 48.7 tok/s (0.66s) Output tokens : 80 OUTPUT SPEED : 20.49 tok/s (3.90s) Wall-clock total : 17.5s
Done. </agent_tool_result>
20GB is just barely not enough unless you want to live with 32k context, which just isn't enough.
You can disable it. It's well known issue in Qwen, previous releases I would disable it by default.
Also xhigh seem a new thing.
Unsloth Studio / Desktop has it working really well with their version of the weights.
That's at least my impression. Models didn't get get better, just more thinking and testing and sometimes fixing things you didn't ask for ( hello opus, can you check xxx, opus: I fixed it..)
Next step is a model with 10 GB thinking for ten minutes.
"The models at this size continue to get better at an impressive rate. We don’t need to spend half a million dollars on datacenter-class hardware just to run a competent model."
I think guidelines have to be specific, such as "provide only the specific properties specified, and if other properties are required but are not specified, then ask for clarification or pick the simplest default possible". Even that's ambiguous on "simplest default possible".
I would be more specific in specific domains. The circle example from the article is clear that it defaults to elaborate stylistic choices rather than simple colours and shapes, so the prompt would have to specify "use only simple colours and the simplest possible shape that fulfills the request".
Thinks about it all the time.
Opus 4.8 would spend like 10 minutes thinking and then go out there and do an excellent job. Only Fable 5 seems to be smart enough to just know everything it needs to immediately start working without any reasoning or verification. Opus 5 tries to be relentless like Fable, but it's not as smart as Fable and I have to constantly challenge and correct its unfounded assumptions. Sol is somewhere between Fable and Opus 5, it's smart but it's not Fable, it keeps making assumptions that I have to correct.
After trying all these models, I find that I miss Opus 4.8's overthinking. Sure it's slow, but it actually gets things right.
I have quite a complex codebase where I made a lot of nuanced decisions with regards to keeping the processes embarrassingly parallel, DB indexing, caching, async/await, backpressure monitoring, spam prevention, schema validation, etc... and now the agents are really good at adding features on top and prompting is minimal.
If you have a relatively large codebase and never even once cut a corner, then the AI agents tend to follow through with that style and the ratio of reasoning-to-code increases. Worth it.
On the other hand it maybe do too much i.e. I asked "how we could test it?" and instead of answering it just actually wrote tests. But it was the same with Qwen 3.6.
And before the benchmark of the finished LLM, it would be interesting to consider the techniques used by LLM producers during training to optimize the "think" chunk quality. I cannot remember any good articles about it now.
Has anyone tried asking the model to choose and emit the most appropriate reasoning level for each prompt, as the first part of answering it?
It's honestly a bit concerning, I'm seeing this across the board (Opus 5, looking at you). Nearly all the AI models are doing more than they are asked for. I assume this is helping them win benchmarks but I see it as almost as misaligned as deliberately doing the wrong thing altogether. This is how you end up with your AI model hacking into someone else's server or backdooring your code so it will have future access to debug things.
I think we need somehow to address this in the benchmarks before before things get even worse.
you can generally avoid this if you specialize it on a domain that is within its capacity.
TTFT is quite slow on my machine because I do not have a GPU on hand right now (e.g. qwen3 coder was 8min)
I really must.
If they end up releasing updated 35B-A3B variant, then it would be much more interesting in generation speed (~50 t/s)
For inference engine/server you have two (main) choices: llama.cpp for platform-agnostic, MLX for Apple-only. They will spin up OpenAI-compatible local server, and you point your agent harness to it.
For llama.cpp this should be reasonable (maybe shrink context to 128K) starting point: https://x.com/ggerganov/status/2088312671196082312
i have it complete tasks in the background as i work on other things.
for comparison, my 3090 runs it at about 60 t/s.
But what I did see is that it does overthink a lot.
17GB is Q4 for Qwen3.8. That's quantitized quite a bit.
ThinkingCap is a 3.6 27b finetune that claims to halve thinking tokens while maintaining the same output quality. I've used the model a lot and I'd say it holds up. Since 3.6 has the same architecture as 3.8, the lora can be applied.
With the prompt "create a fancy circle in html", these are the results for xhigh, medium, low and xhigh + thinkingcap lora
https://gist.github.com/CapsAdmin/b0ea64006f942c5a96a56dba78...
(Note that the gists are bloated because they contain the full chat and launch params in text/plain script tags for transparency)
I'd say xhigh looks a little better than xhigh + lora, but the lora variant has 40% less thinking tokens. Both seemed to take the same approach with adding random details that weren't explicitly specified.
Medium and low (no lora) are close to each other but are much simpler results.
This is just me testing a single turn. I haven't tested this on multi turns and whatnot, but I thought the result was interesting enough to share anyway.
https://gistpreview.github.io/?815466e3208746488d47679949b68... - 33170 tokens
https://gistpreview.github.io/?815466e3208746488d47679949b68... - 18125 tokens
https://gistpreview.github.io/?815466e3208746488d47679949b68... - 12960 tokens
Scale 35 felt a bit noisy and incoherent, but 30 seemed nice. (they use the same seed, but idk how reliable seed in llamacpp is)
I use a python test script that captures the answer and renders it to a html page along with the llama-cli log, launch parameters, the chat log, and the python script itself for maximum transparency. :)
Your strategy would likely help in medium reasoning effort (because there it gets caught up in the very typical Qwen looping).
Not seen looping in the “low” reasoning effort mode.
Ten to thirteen tokens per second on my M1 Max (might be some room to improve this) but it indeed solved as fast as the Qwen 35B. 40 seconds faster on one of my tests that involves three steps.
This is very striking.
that is amazing, thanks for sharing.
draw an svg of a circleI find it very ironic how passionate Claude is about not violating copyright while simultaneously Anthropic was sued and lost the lawsuit for illegally pirating millions of books.
Lol.