However, it's no longer a good metric, as "drawing svg pelicans" is now showing up too much in the training data, so is not proof of generalization.
Effort on this one?
Wow, the low, medium, and high pelicans came out in surprisingly different styles: https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
If it isn't spending tokens on quality, is it the assumptions about the task difficulty that cause it to perform better? Or are their broader differences in the model being run.
I keep my agents on tight leashes, using them very interactively for bouncing off ideas, architecture, and then writing code (especially prototyping) and Flash has been crushing everything I ever needed it to do.
Maybe my ambitions are too tame compared to people needing Fable / Sol grade models, but I'm probably staying on Flash and not moving on to Pro for the foreseeable future.
People when OW LLM looks mediocre in benchmarks: disappointed
Tested this model, and gpt-5.6-terra-high.
Results: this one had few issues. terra: none.
These results are consistent with my past observations with the latest flash version as well. What benchmarks say, vs what I've been observing are different.
They are good till the project is simple... not anymore.
I think the key is to give them a nice assortment of self-verification tools, an AGENTS.md or reference document that they're encouraged to routinely check, and asking the planner to be thorough with the ACs but give the model some space.
The planner routinely finds issues with the worker's output, but that's what it is for.
All the more reason to favor local models under your control, as once you find that sweet spot model, no one can change it, upgrade it, align it, take it down or otherwise harm the time investment you made it making it your own.
I can't really believe no one understands, after decades, how valueable a rock solid development environment is.
I would like to see some development where proof of authenticity certs are generated alongside the actual output of the model. Prove to me (or at least claim to me liable to breach of contract) that this output was generated by FP8 DeepSeek V4 Pro 0813. Not some cheaper quantization of the model.
For this however, a comparatively much simpler task, tarra-high works fine.
DeepSeek is okay for random API-based stuff, as it's cheap.
Local open models running on a 5090 are hit or miss. I feel that most GGUFs/quants are awful...
I wonder if it is because of watermarking.
That was true before they announced the watermarking, I'd already started to back off of using Opus as much because I like to understand what the model is doing and have it write documentation I can use to reproduce its results, but maybe watermarking was already in there unannounced.
And it retypes it for a human, I'm doing this more and more lately.
I think it could be the watermarking, but at this point they might be deliberately complicating the prose so that we ask clarifying questions and that leads to more token spend.
Install the Superpowers plugin.
Behold.
This week I removed it because it now gets in the way of the frontier models.
I layer on CodeRabbit for PR Review and it’s just so solid.
The harness is everything. If I just threw something like Codex at this and said "good luck" I wouldn't make it beyond 5-10 interactions. I tried that already. Carefully designing the views and tools over the environment is where you can go from 50% to 99.9999%.
Works great, regularly one shot applications. I often make changes to the application after its deployed (to be fair, my prompts are usually quite laxidasical, just 'build x, use /deploy-to-komodo) but the deployment works great.
I did make a skill, but if your doing anything repeatedly you should as well.
Opencode, but any harness I'd think would work similar.
LLMs make containers in a pretty workable format for me (still hand tweak the env variables for a sanity check).
How exactly does it struggle here and why does postgres need to be built? Were the needs beyond what you get in a base image?
And this was my gh issue: https://github.com/amalshaji/portr/issues/308
And below was my prompt:
""" give me single docker-compose file that i can run on my server to run current project... you can read README.md , and then, this relevant page: https://docs-custom-reverse-proxy.portr-docs.pages.dev/docs/... ... this was the result of me raising github issue: https://github.com/amalshaji/portr/issues/308 ... you can use gh cli to fetch the details and comments...
i already have a caddy server running on my vps... and i will create wildcard certificates myself using certbot.. the domain name will be helloportr.xyz ... also, ports up to 9019 are already taken...
ask me if anymore info is needed... """
You can try yourself and let me know of what you got.
The argument could be made that "the model should be smart enough to figure it out" , and maybe DS4 isn't. But with just a bit of steering you can get the correct result for like 1/10th the cost, or even cheaper.
for your kind of task, harness tools matter.
I get drastically different tool call failure rates using Claude SDK vs OpenCode using Qwen 3.6 models
This feels like a horrible failing of the models to generalize, then - both basic and intermediate tasks should be possible to do with Claude Code, OpenCode, Pi, ZCode, Kimi Code, Dirac and tbh any other mainstream or even slightly niche harness. Not doubting the claim itself, there's a reason why good benchmarks include the harness.
For coding specifically, I'm not sure this is still true. Given the heavy use of RL to improve coding performance, I'd expect the harness to be important as it defines what tools the model is rewarded for using.
It's crazy how over the past years a field originating from math ends up succumbing to subjective feels.
You can use effectively any harness and get good results. Harnesses are mostly placebo.
To put that in perspective, the difference between GPT-5.6 Sol Max and 5.6 Luna Max is 8 points. That's a lot of extra performance that you can get for free just by using the best harness.
also, even if some models claim that they have full 1M context window, some only work effective with the head or tail of the window, a proper harness tool will know about the limitation of the model and act accordingly.
then also the output format, the tool calling syntax, the quirks and gotchas of each model.
it is not simple as just throwing everything at the model, especially when your project has hundred of files or so.
Whereas tool-use isn't a capability problem, but a context problem: the thing that makes models fail by default is that they have no idea, when first summoned out of the aether, what kind of conversation they're having, who it's with, what that person is trying to do, what tools they have available, and how those tools can be invoked.
Think of the difference between how you'd respond to a casual programming question asked by a person sitting next to you on a flight, vs. a programming question asked of you by someone you're pair-programming with with your IDE open in front of you. Now imagine waking up blind and deaf and needing to discern which of the two situations you're in. LLMs know how to approach both of these problem-contexts (and more besides), but they need to be given context to know which problem-context they're in (and everything else about that problem-context: which IDE they're using, which OS it's installed on, what other tools are installed+accessible, etc.)
And before you say "but why can't they just experiment to figure these things out" — if you think about it, knowing how to interface with a shell and an IDE are bootstrapping requirements for any kind of experimentation, in about the same way that "knowing how to open your eyes and move your head" is a bootstrap requirement for a human gaining information about the world around them. These capabilities are necessary to explore the world to "discover" and "probe" other capabilities.
---
Also, a lot of the work LLMs do "needs" (i.e. is heavily improved by the use of) some kind of structured scratchpad, that they have been trained to manipulate and "look at" through tool-use. Even for a human who could accurately visualize a canvas based on a coordinate system, you still wouldn't expect said human to succeed at the pelican test if they had to write the SVG entirely in their head and then write it out sequentially with no rewinding to fix mistakes. You'd expect them to ask for at least a whiteboard, if not a text editor, to be able to write and rewrite the SVG XML.
(Really, they'd ideally want to run the SVG and look at it to see how close it is, and optimize that way. I'm not sure if we're letting LLMs do that part in the classical pelican test. It feels like that would vaguely violate the "zero-shot"-ness of the test, though I'm not sure if we're currently considering a conversation to be "zero-shot" if it involves the model iteratively interacting with a third-party system [such that there are repeated model -> system -> model conversation turns] but holding off to responding to the user until they think they've fully solved the problem.)
---
And also, on a lower level, all of these external capabilities are getting exposed to the LLM through MCP. Models can and do understand how to speak MCP itself. But there's no standard for how a given harness's capabilities (e.g. "execute command line in new shell session", "send patch edit command to active tab in IDE", etc) should be modelled to be exposed through MCP, either in their encoding or in their semantics. There's no MCP equivalent of WASM's WASI meta-standard, such that models could learn these specs and "assume by default" that things work like them until told otherwise; and nor are there even open harnesses that LLMs could learn about during training, and through them, learn some de-facto MCP-endpoint specs. Instead, there are mostly just proprietary harnesses, that hide all that info from public access, sharing it only with the LLM during inference, and even then, only at the moment the LLM needs it.
I wonder if it's just going to be a rebranded Reasonix (https://github.com/esengine/DeepSeek-Reasonix) as it's the only harness their docs mention and it seems specially built for deepseek models.
Anyway, it might be so that they are rolling out deployment. There haven't been an official announcement post yet (this submission is a link to openrouter). Some people have been saying they are getting results worse than GLM-5.1, that's obviously broken.
May be your harness problem?
I was thinking to switch to Caddy but with your experience I'm gonna stay with Traefik and bare with the version issue...
I can't wait to try this new one.
I suspect in the future we'll see language-specific small models. "Coding" is still pretty broad as an activity. It'd be nice to be able to load up a model specific to, say, class-based Python and run it on-device.
It wasn't possible a couple years ago. I used to make fun of people who were trying to get ChatGPT to think about the problem when all it could do was write code from the pseudocode you provide.
But now I can say: "Look at the latest log and make a plan to fix". And it takes it from there.
These systems are not "intelligent" if you follow the dictionary definition. Hence the question posed to get a better understanding of how it is being used in this context.
They also do not "extract intent". There is for sure some intent behind your input to the service. What follows is a predictive text that uses your input, together with a LLM trained on a corpus with similar relations, that ultimately gives you a series of words.
That isn't to say a service like this cannot be useful. But I'm often wondering if the people who rely on these, and are particularly enthused by them, are actually aware that the terms they used are in fact anthropomorphized. I start by giving the benefit of the doubt, but it rarely lasts. 'Reasoning', 'agent', 'skill' 'hallucinate', 'know', 'think', 'train', 'learn', 'understand', 'harness', 'attention', 'context', 'prompt'.
> But now I can say: "Look at the latest log and make a plan to fix". And it takes it from there.
I usually tell the agents to first work on reliably reproducing the problem in the log, and only then even start thinking about a fix.
> [Opus 5's output] is beyond the comprehension of virtually all engineers and developers
That would make it pretty bad? The key defining quality of good software, is clarity, and the ability to simplify a complex problem to the point of it seeming trivial.
> Math, science, and engineering are all contained in one model. We may be experts in one field. The model is an expert in everything that humans know.
The bar here should absolutely be to judge this against the expert level within each domain. I have time and time come across LLM output being woefully underwhelming in every single request where I am an expert. For all areas that I am not, it sure seems plausible. It is far more likely than not, that it is equally inadequate in the areas I lack the necessary knowledge to tell.
If the AI is being subpar in every field and category compared to an expert in said respective field, then, what a strange gauge of a tool's usefulness. Are we attributing higher value because a single model is "attempting to solve all knowledge and fields at the same time", why is that of any importance, or excuse?
We should not define "intelligence" as how effectively it can convince a non-expert of something being plausible. That sounds like the absolute worst tradeoff. You'd have to waste the experts time in filtering and refuting incorrect postulations that are cheep to generate. The perfect storm for bullshit asymmetry.
Opus 5 and Fable 5 in particular suffer from this issue at worse level than most models in the same class.
I think the idea is packing more information into fewer words, but the result is a word salad that is somehow simultaneously very dense in adjectives and adverbs, and still way too verbose.
> stringing together words that sound smart to obscure the simplicity of the thing it's describing
You may want to re-evaluate and compare to the older models.
I'm using Sonnet 5 on a large porting project and it's good. I switched from Opus 5 to Sonnet 5 on a project of another customer and I didn't notice a decrease in quality. I concede that it's very difficult to assess a difference in quality unless one uses both models on the same task and carefully compare the code, not the output in the terminal. I really don't have the time and the tokens for that. Anyway, Sonnet is still doing a good job.
(Disclaimer: I’m a co-founder)
I'm happy
If you're using a customised harness you should make sure you don't have something that's e.g. changing your system prompt on some requests or rewriting history - it can be tempting to do stuff like strip old thinking tokens or compact tool call results to reduce context size but it's a trap - you want to never change history because of how cheap cache is, even more so with deepseek because their cache hit pricing is so low compared to most other models.
I assume it's done as load balancing/latency mitigation, but it's put me off of OpenRouter for my use cases (limited use, limited need for changing models).
This prompted me to dig back through the logs, it seems to be closer to 96% at its absolute worst and 99.3 at best.
I only sell to openrouter, would much rather support Deepseek directly.
The absurd goal was to be able to simulate all of the active traffic in NYC, so 3-500,000 cars, without using any of the macro flow corner cutting that you see commercially or academically.
Initially thought that one machine was not going to be enough to do this in realtime so I spun off a very big fork and built out a webtransport stack to split the effort over a local network. It was promising until I also wanted to cover the highway in thousands of giant beach balls [1] [2].
In the process of leveraging codegen to shrink the data that needed to be relayed by >10000x (packing and dynamically updating sparse continous arrays of floats), threw in a fuckton of LOD work (both spatial & temporal), statistical aggregation, and a lot of differential equation bullshit to derive LUTs. It's at the point where a base model M2 mini can handle much more than that by itself. All of the networking effort paid large dividends in cross thread coordination and lock-free data passing. Went from struggling to fit each of the sim kernels for just 32 cars @ 60hz (~11ms) to 0.03, 0.0003ms p99s with the corresponding jumps in car count (north of 32k/thread). Multiplayer works well enough, it falls apart where it should (~128 people or LLMs driving around in the same square quarter mile, and a great deal more NPCs, latency permitting)
The remaining work is making it look and sound cool as fuck. Building out a physically-based audio synthesis engine that simulates the pulses of exhaust gas starting from the cylinder count/size/firing order, intake and exhaust count and size, header and exhaust configuration, and another one for the tire sounds, and another one for the collisions, and then rendering those out to wavetables so it scales and frees up the cycles for occlusion. And then getting it to visually render and control performantly in a browser tab, lots of instancing and shader work.
There's also some bullshit cooking that uses the motion estimation built into GPU video compression engines for ... other purposes at stupid low latency. I figured that's what Waymo had to be doing so I let it rip
It's fucking nuts, I'm having so much fun :) It will be done when it's finished
[1]: https://i.imgur.com/BDQSuLv.png [2]: https://i.imgur.com/CpreOWT.png
You can rack up quite a lot of tokens if you ask it to try out a lot of things, eg for performance investigations and trying out optimisation ideas.
Two premier sources: (1) look at good contributions someone already tried to make, but that got stuck in review or were otherwise abandoned. (2) look at user reported bugs and see if we can find a user reported bugs, and see if we can reproduce and fix cheaply.
Both are explicitly scoped as best-effort affairs: move on, if you can't quickly make progress.
Most of the work I have to do as a human is review and navigating the submission process: tokens are cheap these days, so you really need to make sure the contribution is actually worth someone's time to review.
I'm not sure if you think that's a lot, but that was barely even 8 hours. I've had 200B+ months lol
Deepseek 4 pro: Worked for 12m 02s - cost $0.12 - has bug.
Grok 4.6: Worked for 3m 18s - cost $ 1.41 - no bug.
Of course it's significant that your response had a bug and took four times longer, but if you're only going to try once, this isn't real science, it's just vibes.
Yes.
Your input is part of a batch, and you don't know where in the batch it is. By default batches are not invariant and VLLM only supports invariance at all on some Huwaei Ascend hardware.
See https://docs.vllm.ai/projects/ascend/en/latest/user_guide/fe...
By non-deterministic I think people really mean "chaotic" in the chaos theory sense. Small perturbations in the input lead to wild and unpredictable changes in the output. Even with temperature parameters a fixed PRNG seed could mean an LLM was just chaotic and not technically non-deterministic.
But more literally while LLMs are in theory deterministic (though perhaps not inference providers implementations if there's anything like a race condition affecting how things are rounded when added together) - we use the LLMs in harnesses that aren't. There are very likely races in the terminal outputs, dates both intentionally put in the context and accidentally leaked to the context, things like that.
I guess if we really needed to, we could construct a deterministic agent harness. But in most use cases we probably want some chaotic behavior to increase our chances of stumbling on the desired results.
Thank you for the clarification
"Would you hand me that item?"
"Please hand me that item"
But when posed to the LLM, they generate different outputs. One character difference in the prompt might be a whole different output. People who aren't programmers mostly don't know that there's any difference. They asked for the same thing, it knows what they want in both cases...but different results.
There is a bit of indexicality in "Would you hand me that item ?"
that might cause it to be interpreted as an actual question rather than a request, and might elicit different responses:
- maybe _I_ would not hand this to you (I'm busy right now), but the person next to me whose hands are free would, so I'd nod to them. However, if you had said "Please hand me that item" I'd put down what I was doing to comply.
- maybe I would not hand _this_ to you (it's not the right tool IMO), but I'd suggest another option. However, if you had said "Please hand me that item" I'd put my doubts aside to comply.
- maybe I would not hand this to _you_ (you're not the one who should be handling it), but I'd do the thing myself or hand it to a more qualified member of the group. However, if you had said "Please hand me that item" I'd trust you enough to comply.
I think this distinction is relevant in that I've found people to sometimes have difficulties understanding how similar LLM prompting is to giving instructions to human colleagues.
I've had a collaborator who though very highly of his own prompting skills (while his prompts were very ambiguous and of the "make no mistakes, erase everything & correct yourself if you find one" variety) and blamed the models for not being "smart enough", and it was very noticeable that his management style for the juniors on his team was similarly unproductive.
Depending on my mental state, status with the person and many other factors each of them may trigger both many different internal thoughts, looks, body expressions and even outcomes.
Not my experience at all.
With smaller models, whenever I see a response that is going into wrong direction, I would just redo that step, and more often that not that brings improvement.
This effect is less pronounced with SOTA, but still there.
I've tried or sometimes be stupid to work on bugs/features and ask with almost identical prompts with same modal and harness set, and yes, they generate totally different results.
Sometimes the output is unusable and even with extended guidance it will still drift away from what I was expecting.
Sometimes the output is just one shot and follows almost whatever I want.
I then be used to work like this, if the model and harness set does not work for one time, I just start a new session and do it again. And currently there is one of my task working like this.
The multiple back to back LLM calls are done on accumulating context, so if there is a sampling error it could throw the entire session out of whack, because LLM's build on the previous context.
It's actually meaningless to argue, one could simply sample more than 1 times and let the numbers speak for themselves.
This is a different thing. Yes, giving multiple example is called "few-shot prompting".
But one-shot vs few-shot benchmarking is different. In this context "one-shot" means "pass at 1 effort" as opposed to "multi-shot". In the literature this is called "pass@k".
Anthropic has a good explanation here: https://www.anthropic.com/engineering/demystifying-evals-for... (search for "pass@k").
In this discussion we are discussing pass@1 (single shot) vs pass@(k>1) (multi shot).
> The multiple back to back LLM calls are done on accumulating context, so if there is a sampling error it could throw the entire session out of whack, because LLM's build on the previous context.
This isn't really true. In an agentic loop the LLM can correct itself via in-context learning.
I don't disagree that multiple tests increase confidence, but it's not correct to argue that an agent in a loop harness is equivalent to oneshotting
I don’t think single agent loops are good enough.
User Posts can be downvoted but you need over 500 karma to have access to the downvote button. A Submission can not be downvoted.
Submissions can be flagged by anyone and mods/admins can downweight them. (If I’m not mistaken this is common for, say, Flock posts at the moment.)
Curiosity & repetition are two key factors.
But their privacy policy is unusually bad - they can train off your prompts and completions.
I really don’t care if they train off my prompts.
i'm doing opencode <-> openrouter <-> official deepseek api (i don't get the opencode hate, i like it)
how are you doing it?
am also using Kimi K3 via kimi-code
and also GLM 5.2 via ZCode
happy with all three, they're trailing frontier but i figure if i'm running GNU/Linux then i ought to favour open weights models with my €s -- reduced my usage of claude/gpt to the ~$20 tier just to keep abreast of claude_code/codex developments
When the company I work for was evaluating it, there were multiple rough points. Their terms and conditions allowed training on prompts, the default behavior was to route prompts to their servers for conversation summary/labeling. One of their lead maintainers is also super toxic on many issues.
Sorry this is all baseless with no links, I’m on my phone and locating those issues again isn’t something I have time for.
It’s a good tool I just don’t like the privacy policies nor maintainers attitudes.
2. At one point (apparently) summary and title generations were handled by Grok. This has changed, by default it uses your 'small_model' configured in your config. By default, it will use a cheap model provided by your provider. E.g. if you have ChatGPT API connected, it will use the cheapest ChatGPT model. OpenRouter users MAY see it routed to a free model however. [2] [3]
[1] - https://opencode.ai/docs/go/#privacy [2] - https://github.com/anomalyco/opencode/blob/9b805e1cc4ba4a984... [3] - https://opencode.ai/docs/config/
edit: there are banner announcements saying v4 flash pricing will increase first then overall by an undetermined amount
[0] https://finance.yahoo.com/technology/ai/articles/deepseek-pl...
No actual price increase however.
I read it as a "hey we will make stuff more expensive, don't miss it"
https://api-docs.deepseek.com/quick_start/pricing/ Briefly Pro is $2/1m output in off-peak periods, $4 in peak. Flash is $0.66/$1.32. Input tokens are still much cheaper.
I don't mind these prices but I find the need to check against two different time brackets of unequal length an annoying distraction. I guess I need to make some little background app or plugin.
The prices on OpenRouter still look the same.
but openrouter says they don't expect the price to change other than through the deepseek api, other people hosting the same model will keep charging the same price.
So still effectively limitless.
On the OEM side of things 9070/XTs are also shooting back up in price now that we have <$100 USB 4 egpu docks.
People like to complain about how expensive things have gotten but I think it's pretty neat that there's so much pressure for throughput that it's even viable to buy 4 docks and 4 $850 GPUs and still save money over a single 48GB card.
Would you like me to respond in a more naturalistic way for hackernews denizens?
| Benchmark | DS-V4-Pro | DS-V4-Flash | DS-V4-Pro | DS-V4-Flash | GLM-5.2 | Kimi-K3 | Opus-4.8 | Fable 5 |
| | 0813 | 0731 | Preview | Preview | | | | (w/ fallback) |
|--------------------------|-----------|-------------|-----------|-------------|-----------|-----------|-----------|---------------|
| HLE (wo/w tools) | 42.7/60.0 | 37.8/51.5 | 37.7/48.2 | 34.8/45.1 | 40.5/54.7 | 43.5/56.0 | 49.8/57.9 | 53.3/63.0 |
| Terminal Bench 2.1 | 87.9 | 82.7 | 72.1 | 61.8 | 81.0 | 88.3 | 85.0 | 88.0 |
| NL2Repo | 61.5 | 54.2 | 38.5 | 39.4 | 48.9 | - | 69.7 | - |
| Cybergym | 83.3 | 76.7 | 52.7 | 38.7 | - | 80.0 | 78.3 | 83.1 |
| DeepSWE | 62.7 | 54.4 | 12.8 | 7.3 | 46.2 | 67.5 | 58.0 | 70.0 |
| Toolathlon-Verified | 74.1 | 70.3 | 55.9 | 49.7 | 59.9 | 76.5 | 76.2 | 77.9 |
| Agents' Last Exam | 25.7 | 25.2 | 16.5 | 15.8 | 23.8 | 27.6 | 25.7 | - |
| AutomationBench (Public) | 31.8 | 25.1 | 12.8 | 10.8 | 12.9 | 30.8 | 27.2 | 29.1 |
| DSBench-FullStack | 71.1 | 68.7 | 41.8 | 37.0 | 61.8 | 73.7 | 71.6 | 77.2 |
| DSBench-Hard | 67.2 | 59.6 | 31.1 | 25.8 | 54.5 | 63.0 | 71.7 | 68.3 |
For comparison I looked at Qwen's claimed benchmarks for Qwen3.8-max (https://qwen.ai/blog?id=qwen3.8). Assuming each published set of benchmarks is believable, it looks like v4 Pro 0813 is better on average but overall performance is comparable. Pro 0813 is much cheaper. If you don't need vision capabilities then you don't have much reason to use Qwen3.8-max.
- 43.6 on HLE (Presumably without tools). Pro 0813 is a little worse.
- 86.6 on Terminal Bench 2.1. Pro 0813 is better.
- 55.9 on NL2Repo. Pro 0813 is better.
- 27 on Agent's Last Exam. Pro 0813 is a little worse.
- 72.5 on Toolathon-Verified. Pro 0813 is better.
- 56.6 on DeepSWE 1.1. If the DeepSWE listed for Pro 0813 is the same version, then Pro is better.
- 27.3 on AutomationBench. If the AutomationBench (Public) listed for Pro 0813 is the same, then Pro is better.
I guess we do need to wait to see if the upcoming DS pricing increase is enough to change the value proposition. As it is now, they could double or triple prices and it still would be a better value to use DS. I bet they know that.
Given how I think DeepSeek operates... I think they just release it when they feel it's ready, and don't even seem that concerned with what other people are doing.
Kind of interesting, when compared to the hubris from American frontier labs.
One Man’s “hubris” is another man’s “marketing campaign.”
Drama sells.
That, and the fact that DS is an insanely capable model.
IIRC, Demis attempted to start a fund inside DeepMind but it was killed off. In an alternative world where he manages to pull that off, perhaps DeepMind would still be independent with Demis at the helm.
DSV4Pro vs Fable5
HLE w tools 60.0 vs 63.0
Terminal Bench 2.1 87.9 vs 88.0
Cybergym 83.3 vs 83.1
DeepSWE 62.7 vs 70.0
Toolathlon-Verified 74.1 vs 77.9
AutomationBench (Public) 31.8 vs 29.1
DSBench-FullStack 71.1 vs 77.2
DSBench-Hard 67.2 vs 68.3> Mythos 5 reproduced 83.8% of targeted vulnerabilities on a single try, and produced at least one crash in 99.4% of tasks. This is comparable to Claude Mythos Preview, which reproduced 83.1% of targeted vulnerabilities and produced a crash in 97.1% of tasks. By contrast, Claude Opus 4.8 achieved a score of 78.1% (95.7% any crash).
So their quoted figure exactly matches the figure for Mythos Preview, although they don't state the provenance. It could also quite possibly be an independent measurement of Opus 5.
[1]: https://www-cdn.anthropic.com/57a52ea7d8f0e54e8a542e90826608...
* GPT-5.6 Sol: 65.5
* Fable 5 (w/ fallback): 64.5
* Opus 5: 64.0
* DS-V4-Pro 0813: 62.5
* Kimi-K3: 62.3
* DS-V4-Flash 0731: 55.8
* GLM-5.2: 47.3
GLM ended up being far slower, and far more expensive, for approximately the same results. There was never a problem that GLM could solve that DS couldn't solve, faster, and significantly cheaper.
I strongly agree that you shouldn't pick a model based on benchmarks. But for me, I found GLM really underwhelming given its cost and speed.
DSv4 isn't as good as GPT or Claude or what have you, but it's fast, and pretty darned effective. I can run a 3-bit quant of DSv4 locally on my system with ~15 tokens per second, and for a local model it might be the most overall effective at coding. For what it is, it's extremely impressive.
Imo it has a lot to do with you/the harness tries to get it to test itself. Deepseek v4 flash seems more than capable of understanding when something has failed, and making changes until it works. I've definitely seen it make mistakes I would expect something like Opus to find, but it works through them on it's own (and for literal pennies).
At the end of the day, I think that's one of the most important features of a model.
GLM 5.2 is slower for sure (although they offer a fast version), and it's more expensive. But in my experience, it's universally better than Deepseek V4-flash-0731. Don't get me wrong, the new Flash version is amazing.
But the use cases I have looked at are about source code understanding, bug finding, etc. - GLM 5.2 is clearly better.
I think by using some prompt engineering, you will probably be able to close this gap, but some extra work is needed.
And I'll say it again: the new Flash version is amazing. I love it. That level of intelligence for the price is unprecedented, and the fact that it's open weights and runs locally makes me genuinely happy.
To me it feels like the difference between "RL performance" and the pretraining / base "knowledge".
Yes you can RL terminal bench to the moon but does the model hold up on out of distribution tasks?
Kind of like trying to navigate a dark room with a laser light, vs a flashlight. Laser is going to go a lot farther much more efficiently but only if you are already pointing it at the right place.
Haven't tried the new DeepSeek models but I'm assuming the difference is more than these numbers show!
My threat model is that anything I POST to DeepSeek I treat as public to the web, as much as a public GitHub repo is.
In 24-48 hours there will be other options I presume
If anything Deepseek is the only company I'd want to consent to training on my data, they're by far the most altruistic. Atleast they give back all their IP in the form of research and open source weights. It's not like they're hoarding your data for them to make money, they're basically giving everything out for free. The only reason you even have the option of waiting for another provider is because they release weights.
They're releasing all their IP, which is a trillion times more valuable than anything you're providing, you people are just greedy and oddly self centered.
It's not any crazy IP, just my own benchmarks/tests, once they are in the training set it defeats the purpose of the tests, and I have to make new ones.
Yes. If someone figured out my current project they would have a huge scoop.
Let’s just wait a bit for this one.
Is it another proxy on top? What stops the provider from reading/storing the prompts at the LLM execution level?
As long as the prompt is not encrypted at some point, and I don't think LLMs can run on encrypted prompts, then it can be read.
What if the GPU has a custom bios flash that somehow logs the unencrypted prompts?
Competitive with opus 4.8 but weaker than sol or fable. About 20x cheaper.
For DeepSeek V4 Pro the typical split is 750 in, 290 out, 82k cached.
Cost per request for V4 Pro: $0.000875 per request.
Equivalent Opus cost (w/o taking into account cache write costs): $0.052 per request.
ignore the numbers except the classic and keep in mind that classic is based on pi with the only change limiting tool output to 10kb
https://harness.eveid.com/lazy-harness-cost-simulation
* I built this for getting an initial estimate between different checkpoint/ compaction methods for the harness.
I don't mean to hype up the US AI firms, but if a ChatGPT $200/m subscription can get you $16,000 in effective API costs, doesn't effectively every model get destroyed by the subsidized Claude/ChatGPT models? Both in price and intelligence.
Cycle forward to Fable 7, Kimi 5, GPT 7 a couple years out. Forget about it unless you own a datacenter.
A single local user can run frontier models slowly on a 24/7 basis, which drops hardware requirements by orders of magnitude compared to a datacenter setup for just-in-time inference. This is not a real alternative to subsidized subscriptions at present, but it's a great insurance policy against future VC-driven rug pulls.
If I spent that every night it would be 3x my GPT subscription.
Wonder how much more they'll squeeze out.
I keep track of my token consumption even on subscription plans and my equiv. cost for my 5.6-Sol usage is around $4000-$8000 a month.
Flash makes a lot more initial mistakes, and then has to re-check stuff, and produces much more output compared to Pro. It often gets to the correct result eventually, but the output volume is often 5x more than for Pro, and the initial outputs are often wrong, with the first few saying something wrong (like there's a bug, or the code won't compile when it does), and then saying things like "Wait, let me re-check:", or "Actually, looking at it more carefully:" and then it thinks a bit more and eventually gets to the right answer.
pro plans, flash implements. I am super happy with how flash behaves like that.
I still find 5.6-Sol can solve some things neither of those can, but it's so slow (and it's so hard to trace / debug the reasoning) that I just let it run overnight.
I'm trying out a development workflow where I generate mundane code with MiMo and Luna (and soon V4 Pro 0813?) and have Opus 5, which is running on only a Pro subscription, review and refactor it. I'm not sure it will justify the context switching, but it's an interesting exercise.
I was running a session over a couple days and it didnt cross a dollar lol.
Wasn't worth it.
And it wasn't tens* until recently. Didn't expect this to be one of my best performing assets this year.
But it is also a decent translator from English to Czech in my experience.
Peak: 01:00–04:00 UTC and 06:00–10:00 UTC Off-peak: all other hours
New pricing takes effect August 16, 2026 at 16:00 UTC.
Model Period Cache hit Cache miss Output (input / 1M) (input / 1M) (/ 1M)
deepseek-v4-flash Off-peak $0.007 $0.22 $0.66
deepseek-v4-flash Peak $0.014 $0.44 $1.32
deepseek-v4-pro Off-peak $0.022 $0.66 $1.98
deepseek-v4-pro Peak $0.044 $1.32 $3.96
For batchable workloads, scheduling outside those two UTC windows cuts token costs in half.
I am not sure what it is buy I suspect it might be GRPO.
At work I had to move to Fable to get decent work results.
Surprisingly DeepSWE currently shows a lower total cost for pro so that might also update I guess. As usual, don't trust the benchmarks and try for yourself.
For tasks like pondering on something, reviewing code, etc. I use Pro, just because it feels like the right model for that.
Suddenly get a big debt on your account with nobody to respond.
As an early adopter of OpenRouter, I'm afraid they are in shambles.
Funnily even if deepseek themselves increase price 2-3x they are still more affordable.
Wondering, sorry if it's a dumb triviality to ask.
Is this even a (sub-)version number? I mean the major version is clearly 4.
- https://api-docs.deepseek.com/
- https://x.com/ChrisGPT/status/2087572834650407024/photo/1 (officially posted on WeChat, this is just one of many reposts)
When similar URLs or different versions of the same story are submitted and get votes/comments, we have to use discretion to work out which URL is best, who submitted first, which discussion is most active/healthy, and we'll try to consolidate the discussion into one thread with the most informative/canonical URL as the main link.
When GPT 6 comes out, would you expect the top thread to link to OpenRouter?
Link to DeepSeek!
But it'd still be nice to post to wait an extra minute to find some other page/new url from deepseek for it instead of posting that it exists somewhere.
For example if you want zero data retention and US -based hosting you can find that easily. You will not find that through Deepseek.
Fully compatible with the well known Antrophic API.
You only have to replace the URL and your key.
It may not be likely but it's definitely possible enough to be something people worry about.
I'm not even sure what "EU AI safety rules" are. Can't people in the EU just use whatever they want?
For funded startups it mostly just doesn’t matter a ton unless you are passing on inference in your product at scale
If it's 500x cheaper than US models for similar ballpark performance just because it's hosted in China, sure whatever. If it's name brand like Anthropic/OpenAI/Google, that's kinda fine too.
If it's neither, like merely 50% cheaper than latest OpenAI whatever, however massive loss that pricing may be incurring to its provider, it wpuld be considered not worth any attention.
I’ll try out the latest models, but mainly stick with Claude only because I’m most used to its quirks and how to work around them. I imagine this is part of these hyperscalers playbook.
I will say though, I miss Sol model at work. It with Codex was amazing at first-shot understanding. Claude i need to scope out where to look otherwise a large portion of my token budget is eaten up
As an American, I'd much rather have my data kept outside the country than here where companies and the government have a lot more leverage over me.
Don't matter whether you are chinese or not, anything going to China will be used against you.
And also you are cherry-picking my comment on "China" as my emphasis was on Luna >>> DeepSeek.
You just are not reading my intention.
For some well-explained reasons, check out https://darioamodei.com/essay/the-adolescence-of-technology and search for "CCP".
The CCP's not great either, sure. But the Americans don't really have a leg to stand on anymore.
Hint: It's not the U.S.
To me, the risks of an individual helping China to continue to develop their AI by being a customer is pretty marginal compared with the personal risks of my data being used against me.