Languages as designed latent spaces
Some thoughts about what problem-solving even is
This post is an edited version of a Mastodon thread I wrote recently.
I have been thinking recently about how language is a high entropy latent space and what that means.
Latent spaces are sort of magic: they don’t just reduce the dimensionality of a higher dimensional space, they also constrain it such that arbitrary movement in the latent space maps to some meaningful position in the higher dimensional space. Think about how almost any textual prompt you give to an image generator results in a picture with recognisable shape and form, yet the vast majority of the possible images are basically just random noise.
Language is, obviously, like this. Once you’ve learned to conquer the basic structural rules like syntax, grammar, and basic pattern matching of words with similar semantic clustering, movement within it almost always produces ‘meaningful’ (read: intelligible, not philosophically meaningful) results. Consider how an articulate human being can fairly easily produce passable bullshit in a field with which they have no experience simply by reading a little of the field’s jargon and improvising, but would utterly fail to demonstrate practical ability in the field.
Language as technology
One way to think about this is that language is a tool that makes conceptual reasoning easy: once you have the basic rules laid down, it’s almost more difficult to produce an output that doesn’t carry some meaning than it is to produce an output that does.
Consider a strongly-typed programming language like Rust, Haskell, Lean, whatever: like human language, almost every well-typed transformation of the code corresponds to some meaningful change. In the same way that programmers can ‘bounce off’ the barriers of a type system to avoid bugs, a user of human languages can ‘bounce off’ conceptual barriers that would otherwise lead to them producing unintelligible nonsense, simply by following the grammatical rules of the language.
However: the map is not the territory. For every well-typed transformation, there are innumerable invalid transformations that would correspond to meaningless rubbish in less expressive languages. Effectively, the language is turning the problem of domain reasoning into one of pattern-matching and symbol manipulation by constraining the set of possible programs to a small latent space within that covers a particularly high concentration of meaningful programs.
Much of the confusion stems from the fact that, for all of human history before about 2016, symbol manipulation has been seen as very difficult because it’s something that is less immediately intuitive to our brains. So, whenever we see someone - or something - perform manipulation of a latent space like language, our tendency is to presume that they first did their reasoning in the low-entropy, noisy, complicated space of reality and then transformed it into the latent space as a final step. But there is no reason for a machine to take this course of action: why should it, when simply modelling language at the level of grammar and pattern-matching more readily satisfies a fitness function?
Many problems can’t be solved in language-space because the language doesn’t exist to solve them yet
The point I’m very slowly getting to is that human language is a latent space that we have fashioned over millennia of careful analysis, categorisation, taxonomic research, and study of the real world. That is a specific kind of work that we usually forget about in our text-and-language-filled lives where we live in the map and forget that the territory even exists. It’s also work that an LLM cannot do: they are symbol manipulators, not creators of symbols.
Their reasoning is bounded by the linguistic latent space that we have laid down before them (as are we all, to some degree). But our unique ability is that we can operate outside that latent space too, and grow its scope by making use of the world model in our heads that has the ability to inquire and test hypotheses without relying on training data to provide an answer.
It’s often seen as curious and a little bemusing that machines can generate mathematical proofs or find counterexamples to long-held conjectures while simultaneously being incapable of performing basic tasks like frying an egg or folding clothes, but perhaps it should not be: the former requires only rapid manipulation of the semantically rich latent space that has already been mapped by human minds, while the latter requires more robust reasoning about a noisy reality with lots of dead-ends, the sort of thing that only a world model can excel at.
If you find this a difficult notion to entertain, consider how capable a human being is with almost no training data to learn from. Do we just have the benefit of an incredibly efficient training algorithm? Or are we simply performing the bulk of our reasoning in a wholly different level of abstraction?
My hypothesis is that LLMs can only perform transformations within the first layer, whereas humans - or anything with a world model - are capable of shifting gears between the layers. That’s not to say that LLMs can’t solve problems that can also be solved with a world model: just that they require a pre-seeded linguistic lantent space that covers the problem domain before they can model a solution, whereas a world model can bypass this problem.
The importance of not confusing the map for the territory
Interpolating rich latent spaces that have already been mapped by the labour of a hundred generations of humanity and reasoning in highly noisy and path-dependent environments are wholly different tasks and we should be careful not to imagine that proficiency in the former necessarily demonstrates proficiency in the latter, even though this is usually true of human beings.
I strongly suspect that movement between latent spaces is where we’re seeing LLMs start to hit a wall. What does that mean for us? Quite possibly: a stifling of innovation. If language - at least in part - defines the bounds of our thinking, a world in which LLMs are the most prevelant and common users of language might experience a sort of linguistic and cognitive inertia, unable to escape the self-reinforcing language latent space.
This is a fairly normal thing for a technology to do
Perhaps if you’ve already drunk the kool-aid you might be tempted to claim that I’m moving the goalposts on what constitutes ‘intelligence’, but as Opus 23 convincingly argues, this is just what technology does: it separates concepts that were previously understood as a whole into their constituent parts and recontextualises them. It should not be a surprise to us that the dual institutions of intelligence and language are being cleaved apart.
I’m not an absolutest: I don’t think there’s any special or otherworldly about humans. We are material bags of meat, and there’s no reason to assume that a machine can’t also be equipped with a world model. Arguably, the narrow ‘AI’ of yesterday like chess bots and pathfinding algorithms are themselves world models with a solver bolted on top. What I’m arguing is that LLMs haven’t cracked this problem of general reasoning: they’re still just narrow AI that operates within a latent space that’s been pre-seeded by humans with the expressive structure necessary to model a wide range of domains.