YouTube removed the 1-5 scale for the same reason, people have no idea what to do with it.
LLMs generally cannot report the outputs of their own activation functions. Taking the output of softmax as if it were a probability score is considered a first-year PhD student misconception.
Peoper confidence scores on a continuous scale, or even a fine grained discrete scale, are arguably skeuomorphic. It’s worth introspecting on whether that’s what you really needed in the first place, or if it’s just what you’re used to from past experience working with regression-type models.
Can you provide credible reference for this?
It is given sets of paramaters/data and infers a response based on it's acceptable confidence level. The bottleneck for this is compute power and model/prompt/rails/integrations.
The reason LLM stays consistent/can be somewhat trusted to stay on rails is that it's aware of both it's context/scope and it's confidence about the context.
Anything else would be....original thought and exposition? I guess. You could say that our brains do the same, IE; unconcious statistical analysis leading to inference based on confidence.
Does the output of the model have insight into those details? Just because our brains function based upon electrical potential doesn't mean we have innate insight into it.
Certain models of course force this context memory, some don't, some allow you to toggle it's "trigger level".
For example Anthropic, use haiku for low memory context queries, research. Then Opus/Fable for compiling/analysis/design, which relies on the LLM inference constantly going back into context memory.
Another example is OpenAI/GPT. The "free" tier forces context memory using the "highest available model" per your subscription (or lack thereof), and upon hitting its limit, drops you down to a low context model, and then ultimatley blocking that chat entirely.
(I appreciate this discussion and am not trying to inform you, I'm sure you already know, but typing these things out helps me understand my holes in understanding.)
Even a probability?
If it is something like cosine similarity or SVM scores, which aren’t probabilities or even percentages, then users will misinterpret it if you present it as a number from 0-100 or 0.0-1.0.
What many people would want it just a complete block on select youtube channels and providers. Just whole hog, rip them out of any of the potential recommendations.
But that's not whats best for Youtube. The point system still stands in ratings, once exposed to the viewer, are gamified. They would however still work if it was a secret vote, but then, again, the user wouldn't be compelled to use it as people want immediate visual cues that they're doing what benefits themselves.
If you are asking about confidence and somehow it can’t be defined either mathematically or in a grading rubric, then you the implementer need to start asking yourself the questions, because then you got a target that you don’t even know how to measure. The LLM isn’t even the problem at this point because another human would also just be going off gut feeling and vibing a number that may or may not be accurate.
One thing that has helped me make the LLM more consistent on scoring is deduction based rubrics. Start everything at perfect and deduct points for specific things (ie style, missed facts, etc). It gives much tighter final scores under these conditions (for me) than giving rubrics + samples.
Also, as another poster said: decomposing your rating into as many dimension as possible is better way to generate some confidence score grounded in fact.
When we start using big datasets where n went into the tens of millions we kinda stopped bothering because margins were so small it was effectively certain you found the mean of some range. However I think in that time we forget what confidence scoring actually is and how quickly multiplied alpha can balloon on you.
[1] https://shap.readthedocs.io/en/latest/
[2] https://christophm.github.io/interpretable-ml-book/lime.html
On the other hand, it’s kind of expected that the hard part of the job in AI/ML engineering gets neglected over that part people can brag on social media about. Deeply concerning but sadly understandable.
Even if they can do great maths I think bayesian statistical fundamentals are not reliable at this point.
[1] https://think-twice.me/is-ai-biased-or-sexist-making-chatgpt...
I thought the way Jev was presented was sneaky - you can't say "we calibrated the model for you" and skip having a task. specific calibration set. They couldn't have known all our future tasks.
Yes, using an LLM for confidence scoring while generating prose is a bad idea. I expected a mention of decision models here, as that exactly what they are trained for.
Claude found this thread and wrote something how his own scoring sucks and he just marks any relevant thread 7/10. Thought it was funny how he just roasted my product while also being the cause of that product sucking.
https://arxiv.org/html/2607.20526v1
From that paper, model accuracy is not perfect, but it's overall pretty favourable. There are some encouraging findings in there, too, including that accuracy of model confidence scales with reasoning effort.
But I agree, I would8 much rather have some data than reckons.
It's with noting that people do this too. They tend to vastly overetimate a low probability event that they know is possible. Often by more than one order of magnitude.
It is something worth keeping in mind when you are people throwing around personal p doom scores.
Interestingly when humans have to make a decision based upon the probability they feel, they tend to act far closer to the actual probability than what they would say the probability is.
Analogize to basketball here. How confident are you that you can hit a free throw? If you ask someone who has played a thousand hours a year for the past ten years, has recorded their own statistics, and has averaged 70% success rate oer those past ten years, and they tell you they're 70% confident they can hit a free throw, that's a good prediction. If you ask me and I give you any prediction at all, I'm full of shit. I haven't played basketball or taken a free throw in 30 years. I can pull a number out of my ass, I can reason through it, I can do a Fermi, I can assume I'll be the average of all people who have ever tried and do my best to collect whatever data might exist on brand new players with no experience trying to take a free throw for the first time. I can narrow that to people my height, my age. Now we're getting better, but it's still not as good as having a real record of my actual past performance and basing a prediction off of that.
This is just science. Hierarchy of evidence. Epistemology. Sometimes answering a question requires collecting data that doesn't exist yet and waiting a while, in the meantime admitting you don't have an answer. It was already sorely lacking in the world, but in this age of oracles, it's only getting worse.
I know that you can use LLMs to write ascii numbers that represent a "confidence" as in TFA. This is fraught and strange and hard to trust.
But can't you also use the next token likelihood itself, so long as you rearchitect to use the input classes as the only possible output tokens? Something like stripping away speech and only working with the input labels as vocabulary...
I guess I thought that's what decision-like models were doing to output confidence scoring. (or something like that)
It's thought to work "better" but score calibration is still a problem.
None of this means they can't be useful tools, but when they fall down, we should remind ourselves of why that is, and be honest with ourselves about what they could ultimately be capable of.
None of that means they can't be useful tools, but when they fall down, we should remind ourselves of why that is, etc etc.
They don't always match up.
Sometimes the map IS the territory, or at least superior to it.
As far as I'm aware, when reasoning does not meet reality, it is called a delusion, and I'm fairly certain that majority of humans are not delusional.
So if the bar for proper reasoning is never ever being delusional then humans and LLMs sit in the same bucket.
Even on the frontier models these blips still happen fairly often and it's intrusive to a lot of workflows.
Stallman vehemently states llms are bullshit generators with no true understanding.
But sometimes you can approximate your problem and unfold an llm interlocator to producing the strings you need for your task
It is perhaps sad, but with our current knowledge of neuroscience, we can't really argue that humans are somehow special. All we can say is that in practice we are still more intelligent/understanding/meaningful/conscious/whatever than LLMs. But for how long?
It's like calling a video clip a "picture sequencer".
I'm no expert in the field but it is clear that the model outputs are not reliable. To my frustration this was designed by "experts" so I originally trusted that this approach was sound.
How does this compare with descriptive confidence? For example, major v.s. minor? Intuition tells me it's a different flavor of the same illusion.
I'm pretty sure this is how reasoning models work at the big providers. Averaging 10 lunas together likely provides more stable outcomes than one big sol invoke. In any case, you could use the statistical distribution to decide on subsequent inference steps.
This approach with more data for calibration is basically the beginning of making a decision model like Jev.
If it's deciding between two phrasings, I imagine the confidence you get will drop, even if the phrasings are equivalent.
By analogy to your 'Measuring confidence' section: a function estimator (neural nets can be good at this) and a bunch of past observations could be pretty calibrated at making weather predictions. An LLM has a wild mix of past observations and random collated opinions from previous text (it might mention months, recency of rain etc.. etc...) so lands somewhere between 'text completion' and 'estimator' with a model of unknown fidelity. It is not 'merely' generating a plausible sentence.
I would never use an LLM for accurate forecasting, and certainly agree they are not calibrated, but I would not dismiss them out of hand.
I was surprised that the author hasn't mentioned Jev-like models. They're quite good for general purpose confidence rating.
This is a tough one to explain because some people think the average person has good taste by definition. I used to think that way but over the years I've realized that a lot of what the average person likes is derivative 'slop' - This was true before AI and still true with AI.
Once you start deducting points for lack of originality, you will realize that people have really poor taste... Very narrow taste. Most people literally consume trashy variations of the same thing over and over. We have to start deducting points for this because the more intelligent among us can't take it anymore! We're tired of being exposed to thousands of variants of the same slop... It might be sufficiently different to fool the average pleb's simple mind but to us it's like eating the exact same dish every meal of every day with different amounts of salt and pepper.
This comes through when you ask an LLM to rate/score something. It's frustratingly conventional and not based on first principles thinking and it ADDS extra points for being derivative instead of SUBTRACTING as it should. We can see it play out in the workplace. LLMs have poor judgement. It's the one area where they lag behind.