Never mind debates about intelligence. How many actions can a system take at 99% reliability before a human must intervene? This metric is doubling every 7 months.
Certainly one of my criteria for a utopia would be that some of the beings in it are engaged in intriguing or exciting activities with meaningful consequences. I'd require more than Special Circumstances to meet this criterion -- they are too small a piece of the total system.
In Player of Games we see a corner of a gaming culture which partly meets this criterion but it does not have meaningful consequences outside the gaming participants (unless you count the ways the Minds use it to manipulate Gurgeh).
Maybe by this criterion utopias are impossible, since the disruption caused by exciting activities with consequences conflict too much with the optimality of the society. But I don't think anyone now can prove this would be the case.
In the Culture Series history hasn't ended, and the members of the Culture can and do leave and live in ways the Culture would find repugnant. Also there are marginal participants (such as the Mind known as Meat Fucker) who are more or less in the Culture but act in ways the Culture finds repugnant, but are not punished or controlled.
So according to your definition the Culture is not a utopia.
This is an example that supports Scott's point that people don't have world models. The people who "believe" this don't wonder how stock market continues to operate now that NYC is a wreck. Etc.
I wonder in what sense they really do "believe". If they had a strong practical reason to go to a big city, what would they do?
I'm not sure that you can reach the conclusion that "people don't have world models" based on beliefs that do not fully integrate with such a model. We too often try to misapply binary truth requirements to domains in which there exists at least a trinary logic, if not a greater number of logic truths.
If I meet a random stranger, do I trust them or distrust them? The answer is "both/neither," because a concept such as "trust" isn't a binary logic in such a circumstance. They are neither trustworthy nor untrustworthy, they are in a state of nontrustworthiness (the absence of trust, but not the opposite of truth).
World models tend to have foundational principles/truths that inform what can be compatible for inclusion. A belief that is non-compatible, rather than compatible/incompatible, can exist in such a model (often weakly) since it does not meet the requirements for rejection. Incomplete information can be integrated into a world model as long as the aspects being evaluated for compatibility conform to the model.
Requiring a world model to contain complete information and logical consistency at all possible levels from the granular to the metaphysical seems to be one Hell of a high bar that makes demands no other system is expected to achieve.
> it's possible for simple normal trains of thought to latch your brain into a very undesirable state.
This seems very incorrect, or at least drastically underspecified. These trains of thought are "normal" (i.e. common and unremarkable) so why don't they "latch your brain into a very undesirable state" lots of the time?
I don't think Scott or anyone up to speed on modern neuroscience would deny the coupling of mental state and brain chemistry--in fact I think it would be more accurate to say both of them are aspects of the dynamics of the brain.
But this doesn't imply that "simple normal trains of thought" can latch our brain dynamics into bad states -- i.e. in dynamics language move us into a undesirable attractor. That would require a very problematic fragility in our normal self-regulation of brain dynamics.
See the key here is, the AI provides a very enticing social partner.
Think of it as a version of making your drugged friend believe various random stuff. It works better if you're not a stranger and have an engaging or alarming style.
LLMs are trained to produce pleasant responses that tailor to the user to maximize positive responses. (A more general version of engagement.) It stands to reason they would be effective at convincing someone.
It was a national security program with no public face. I was recruited into it because I solved a fundamental computer science problem they were deeply interested in. I did not get my extensive supercomputing experience in academia. It was a great experience if you just wanted to do hardcore computer science research, which at the time I did.
There are several VCs with knowledge of the program. It is obscure but has cred with people that know about it. I’ve raised millions of dollars off the back of my involvement.
A lot of really cool computer science research has happened inside the government. I think it is a bit less these days but people still underestimate it.
I'm not surprised that the government does great research, but I wonder how much good does that research does, if it's unpublished and disappears after budget cuts.
"we replicate the DeepSeek-R1-Zero and DeepSeek-R1 training on small models with limited data. We show that long Chain-of-Thought (CoT) and self-reflection can emerge on a 7B model with only 8K MATH examples, and we achieve surprisingly strong results on complex mathematical reasoning. Importantly, we fully open-source our training code and details to the community to inspire more works on reasoning."
