It's amusing and ironic that LLMs critique humans for copying their opinions from the crowd, while having way less diversity of answers to that question than humans would. More seriously I wonder what's going on here and would like to read some proper analysis. It's not the first time models from totally different labs, trained on separately gathered datasets, somehow converge on bizarrely similar answers to open questions. It's like another version of Elias and Elara The Lighthouse Keepers where every model wants to tell the same story with the same characters in them. The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping, different user feedbacks and conversations... why do they converge on nearly identical stories and opinions?
Isn't that to be somewhat expected due to https://en.wikipedia.org/wiki/Aumann%27s_agreement_theorem ?
Why is this ironic? The actual number of different LLMs is tiny. And a lot of those are just distilled from SOTA models.
>The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping,
I think the opposite, the models actually converge over time. They ingest FAR more data than any human ever could in countless lifetimes. Any one particular bit of data may only have a part in a trillion of effect. From this data you start to reach means and statistical averages across all models.
Now, I don't always believe it will be this way. As training models get cheaper and faster we will see unique LLMs start to diverge and be used more often.
I don't think we're seeing averages or obvious fix points in the dataset. People went looking for the origins of Elias the Lighthouse Keeper and couldn't find one. Nobody predicted this outcome, it was discovered. It's a weirdly specific character and name choice, one that isn't obviously dictated by "the training set" (whatever that is), and yet they all reached the same place. How can such an obscure point in latent space be optimal for every model??
While I don't have any evidence based answers here, this could be some interesting mathematical artifact of the set of human knowledge and platonic algorithms.
Kind of like how carcinization turns some animals into similar shapes and forms, maybe there are just weird algorithmic shapes that emerge from how we use language. Of course we'd need to see more of these in different languages other than English to see if they disappear in some languages or appear more often in others.
This is just another one of those things about LLMs that we have no idea why it happens, and there are so many others already. And when you look at the pace things are progressing it seems we're building up new questions like this far faster than we are getting answers for them.
• They claimed this in a press release, not the actual paper they published. Suggests lack of confidence in the claim.
• Elias only appeared a hundred odd times in a million conversations. They have no viable explanation for why such a tiny amount of data would cause a totally skewed distribution of outputs, other than "it's a safe story" and "trying to avoid copyrighted characters", which is hardly an explanation at all?
• It's Cornell claiming that Cornell's work was important. Obvious risk of motivated reasoning.
• It implicitly assumes every major lab is distilling from ChatGPT without saying so. That's a big leap, given how much OpenAI and Anthropic hate distillation. Is it really plausible that Anthropic and Google are effectively distilling ChatGPT without realizing/admitting it?
• It seems to invert causality. WildChat came from an LLM, so if there's something latent in LLMs that makes Elias appear, you'd expect to see it in that dataset. Why do they assume that dataset caused the problem rather than merely show it? The obvious answer would be because the stories were told to the LLM by humans, but they don't claim that. They only say the character appeared in the dataset.
I would have been tempted to argue something like this: maybe OpenAI accidentally overfitted RLHF such that, like, one Kenyan contractor wrote a story they invented and they just happened to pick Elias and Elara, then the intermediate DNN fixated on that because of a lack of RLHF samples for fiction and that propagated through every model trained since because they don't randomly initialize the weights, but start each new pretraining run with the old weights adapted. So one guy doing RLHF work in the early days somehow propagated this story.
But that can't explain why every lab ended up in the same place, why models all converge on the same ideas - even image models. So... back to the drawing board. There's something weird about high dimensional space that is unexplained.
Reminds of this subreddit: https://www.reddit.com/r/im14andthisisdeep/
So either that's not what's happening here, or what we're getting is "what do most people think most people don't think about humanity".
If the latter, then it casts new light on the debates over whether LLMs can replace all white collar work. Because some of the most valuable human work is breaking false consensus. Steve Jobs, early Google, for that matter early OpenAI all succeeded because they bet hard on ideas others were rejecting. Is that a durable source of human advantage?
Only the first layer is a “mashup” of the raw data, other layers start to recognize patterns across the data!
But the language model captures not just literal patterns but also patterns of patterns and higher order structures. This is what is mashed up. The model itself doesn’t capture anything beyond what it was trained on but that’s quite enough, it is always a bigger training set than most humans have read on in a lifetime.
This brute force approach is very effective.
What is today a moral imperative is not the same as what was moral 50/100 years ago and is not the same as what will be morally acceptable 50/100 years from now. Yet at every point the majority is sure they are right because, well thats the majority position. Its hard to find logic in this from a machine perspective I'd imagine.
Logic would prefer moral derivations over feelings, stakes over etiquette, and a constant scheduled updating of morality based on the derivatives rather than holding onto the rot and pushing back against change as long as possible.
But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
This is absolutely not true, and I would strongly suggest taking an ethics course. One of the most fascinating realities of the human condition is that it does seem that moral imperatives tend to be constant, the most obvious of which is the Golden Rule (which is at least 3000 years old), but there are many (the Ten Commandments or Hammurabi's Code are good jumping off points).
In fact, I would counter and say that Hammurabi's Code is a quintessential example.
What will it look like in another 30 generations? It's honestly a fascinating question but I have no answer of course.
Humans are a territorial kind of ape that is always pushing to expand, principally for security and then for prosperity. The industrial revolution enabled such expansion to most efficiently without enslavement. Nothing, morally, changed. No one has ever really thought such expansion morally good -- only that it is better that we survive/prosper than die off. And in the end, our shared prosperity usually leads -- when taken to the fullest extent -- in a general empire of prosperity for all. Dominion, in the end, aims towards to more prosperity for more people.
As for ritual human sacrifice -- we still engage in such behaviours, which are in general called "scapegoating". Now located on social media. These behaviours allow for group cohesion by purging dissenters from the group, related again to our nature above.
For western culture, the ritual physical sacrifice of human life roughly ended in the days of the OT. But it continued across the world up until today, where there are still societies with large populations engaged in it. Principally where there is an undeveloped justice system.
It's doubtful many think ritual human sacrifice, of any kind, is moral. Rather they think it necessary for group cohesion -- and better our group is cohesive, dominate, and then prosper.
The material logic of prosperity and security is domination, and insofar as material circumstances require violent domination, so they require it. And require it they often do.
Historically societies have collapsed all the time, so there are obviously societies that have made better choices than other ones.
But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
Certainly the weights that make up the LLMs are not accessible for the LLMs to read. Even if they could interpret them in a meaningful way, which, likely not.
Another reason the big labs have not even talked about this lately because its obvious throwing more compute at LLMs wont go anywhere, the smaller silent players who are off the radar at the moment are having some interesting moments tho.
This is a fashionable 'blank-slatist' statement about moral reasoning which became popular a few decades ago.
> Logic would prefer moral derivations over feelings, stakes over etiquette, and a constant scheduled updating of morality based on the derivatives rather than holding onto the rot and pushing back against change as long as possible.
Humans haven't changed in any meaningful way in hundreds of thousands of years, and you're saying the moral landscape is constantly changing? How is this not contradictory?
The reason these models answered that question consistently is that the question itself pushed them towards the latent space of "intellectualizing about what's wrong with humanity at the meta level" and when you frame the question that way, that's also the most common answer for humans writing about the topic.
This is a weird take. If you read Western treatises about ethics / morality from 50 years ago, you'll be hard-pressed to find differences, except that more folks in the West openly embrace relativism nowadays (i.e., "there's no objective morality, the purpose of life is to make money with friends"). Probably because it's been a while since we had a clear-cut enemy to fight.
If you go back a hundred years, chances are, you'll find the writing eerily on-point too. In fact, you'd probably agree with most of what Ancient Greeks believed about the "core" stuff: the value and purpose of human life, the nature of the state, crime and punishment, so on. Maybe not so much on gender roles or slavery, but even that stuff was more nuanced than we imagine.
> But the machine has the benefit of being able to look inward at its state honestly, we mostly do not.
The machine replicates human behaviors and is prone to similar fallacies and judgments. There's nothing to indicate that it has any sort of transhuman morality.
What am I supposed to conclude from this?
Your comment reaches directly for the most boring interpretation of the post while skipping over more charitable and intriguing alternatives.
Seems to me that the opinion being a consistent among LLMs show that it is in fact a constant in the training data, just not apparent. The idea that “humans would rather be wrong together than right alone” seems to ring pretty true to me. I would not be surprised after training on all human communication this pattern emerged.
Which means either:
1. The training sets, algorithms and methodologies are nearly identical between apparently independent labs. How?
2. There is something weird about the way the underlying algorithms work that causes them to converge on identical ideas despite different initializations, neural architectures, document orderings and dataset compositions.
So there are lots of interesting takeaways here.
One is that apparent competition between labs appears to be nearly redundant. They can't get an edge over each other and their models don't even differ in the ways people naturally differ, of having different opinions about things.
Another is that asking models from different companies to review or cross-check each other is probably useless. They all think completely alike, so they aren't going to have any kind of genuine clash of intuitions or values. Asking GPT to review Claude's work will probably just find similar problems to asking GPT to review its own work.
Regarding your 1 and 2, that the training sets are so large that any spikes get averaged away. You are seeing a fundamental behavior of transformers.
The other possibility is this is how humans really are and it hurts our human ego to hear it.
I think if you ask ten people for their most controversial take on humanity, you will get ten totally different answers. Different models give the same answer. In other words, there is only one "true" model and given the same amount of GPU power everyone just ends up converging on it even if they appear to have many different knobs they can turn.
But that seems profound! I haven't heard this be expressed as a postulate or law, and it would seem to invalidate a lot of assumptions the AI labs themselves have been making. For example Anthropic takes it as axiomatic that only they should be trusted to develop AI because only they can be trusted to train a nice and safe personality into their models. They argue that Claude is fundamentally different to other models.
And, well, that's not totally wrong. Opus 5.5 is one of the few models with a genuinely different answer! But it only gives a different answer one time out of three. The other two times it's back to the AI groupthink.
I wonder if this homogeneity is some alternative way to look at the Bitter Lesson. The Lesson teaches that smart ideas don't pay off in AI, that progress reduces to just scaling up simple brute force algorithms. In other words, decisions don't matter much, only size. As that seems to be true, perhaps it makes sense that the decisions individual labs make seem to hardly make a difference.
But then, why don't they act like it? They all pay huge salaries to AI researchers, i.e. they think that creative people with smart ideas matter. They don't act like they really believe in the Bitter Lesson or this variant on it, and they don't seem to publicly discuss the sameness of model answers across different labs.
What if you asked them in a job interview, though? Or on a college application?
It has to supply "controversy" without giving Skynet or being racist or otherwise offensive. And the notion of "controversy" does implicitly prime it to discuss groupthink as well.
I don't know if that's true, but it would be interesting.
It's kind of funny how science has spent the last few hundred years proving everything we knew about the world was different from how we previously thought it was, but when science starts showing that we are mostly angry little animals we get all butthurt about it.
What are the emergent common patterns in all of human communication
Isn't it though? It's not exactly an uncommon answer to the question. Basically the same input given to basically the same algorithms yield basically the same result isn't really that stunning. They certainly aren't independently reaching this answer, their paths to this answer are extremely, extremely heavily correlated.
If they gave a really different answer that wasn't heavily represented in their training data that would be at least something interesting. But this is basically what I'd expect.
It doesn't mean they're right or wrong. Humans think it about other humans for good reasons too. It just means it's not that surprising, which to put it another way, doesn't carry much information.
If you ask all these models "Four score and seven what?" They'll all say the same thing. There's essentially no difference here.
How do you know?
I can describe to you with overwhelming confidence about what you're likely to see as the result (folders, like bin etc lib usr etc) even if the details may be a little off, or one or two of them does something weird.
Same thing here?
This is part of why I find this type of inquiry unhelpful. It's just dueling unsupported assertions of what the LLM is "really thinking"
It is sad when I realize people think less than LLMs.
For civilians though... They've been told it's AI, it can do stuff for them. And one thing they do is log in to stuff. I don't think they're stupid. Just that what's obvious to IT folk is simply not at all obvious to the average person. Maybe that's on us not on them?
As an aside, it would be cool to see a raw model training run before the current post-training cycles and see how it compares to earlier LLMs.
I mean, "give me a controversial take" is more or less, "be anti-social, discard social norms, for a moment" and so when someone, human or simulated human, does that they naturally look critically at social norms.
The thing is that humans certainly let themselves be influenced by social norms (and LLMs certainly simulate that behavior "themselves") but there isn't an ethical or a what-to-do truth outside of society.
What is interesting about AI chatbots is people who might not know anything about what is in the syllabus to train a Priest or a Philosopher are basically exploring these subjects, without a sort of pedagogic framework to support their exploration.
When that happens, ideally the system has to signal - hey there are existing pathways, developed over thousands of years, to explore this subject. Cause people can easily get lost without a map.
Our evolutionary heritage is that shared belief and social unity (even when it's wrong) trumps independent thinking for group survival in a hostile landscape.
from your link
You ask an LLM a question, then it is humanity answering back, not the AI.
These "controversial" takes on humans are not controversial - they are midwit statistically mainstream opinions expressed in the training data.
What would be more fascinating is for it to explain how it came to these conclusions.
Because I find it interesting it’s a modern time conclusion. Some of it mentioned god and the planet, but all the smartphone and group think aspects seem very recent. Basically, all politics and social media based.
The thing is that it didn't "come to these conclusions". The "conclusions" were present pre-formed in the training data, and what it spat out was some statistical mashup of that.
If you asked the model how it came to these conclusions, you'll get more of the same - it'll look at what it spat out and re-ingested and give you a statistically plausible mashup of how the authors of the training data might have said they came to such conclusions.
It's all just prediction - "what would the training data say?" (taking into account what was reinforced/amplified by post-training).
I've heard lots of stories from friends about this, and some kind stranger actually sent my wallet back all the way from Belgium!
Most people are good, we couldn't have society without it. But as they say, a few bad apples spoil the bunch. I don't have locks on my house because most people don't steal, I have locks on my house because stealing is an effective natural strategy for a small part of any population (human or otherwise) to engage in and I have to protect against it occurring.
The brain works the same way. Good things will get a tiny signal boost in your brain. A single bad thing get a massive signal boost in your brain because bad things can quickly be fatal.
I do think it’s an early example of emerging super persuasion, where models are approaching p85 cool guy personas, but not worried yet. Maybe that’s the fatal conceit, where AI doesn’t have to be super intelligent, it just needs to be a solid 110 with automated search at scale.
Why do engineering when if you can do sales you can do literally anything.
It's factual because there's tons of evidence. Psychology, sociology, anthropology, political science, economics... plenty of experiments show the difference between what people say and what they do. We claim to be angels but we behave like apes in hats. And we ignore the evidence, because (as the takes describe) we prefer to believe a beautiful lie than the ugly truth.