Speaking of uncertainty, I wish more people would accept their uncertainty with regards to the future of LLMs rather than dash off yet another cocksure article about how LLMs are {X}, and therefore {completely useless}|{world-changing}.
Quantity has a quality of its own. The first chess engine to beat Gary Kasparov wasn't fundamentally different than earlier ones--it just had a lot more compute power.
The original Google algorithm was trivial: rank web pages by incoming links--its superhuman power at giving us answers ("I'm feeling lucky") was/is entirely due to a massive trove of data.
And remember all the articles about how unreliable Wikipedia was? How can you trust something when anyone can edit a page? But again, the power of quantity--thousands or millions of eyeballs identifying errors--swamped any simple attacks.
Yes, LLMs are literally just matmul. How can anything useful, much less intelligent, emerge from multiplying numbers really fast? But then again, how can anything intelligent emerge from a wet mass of brain cells? After all, we're just meat. How can meat think?
> Yes, LLMs are literally just matmul. How can anything useful, much less intelligent, emerge from multiplying numbers really fast? But then again, how can anything intelligent emerge from a wet mass of brain cells? After all, we're just meat. How can meat think?
LLMs actually hint at an answer to that, but most people seem to be focusing too much on matmuls or (on the other end) specific training inputs to pay attention to where the interesting things happen.
Training an LLM builds up a structure in high-dimensional space, and inference is a way to query the shape of that structure. That's literally the "quality of quantity", reified. This is what all those matmuls are doing.
How can anything useful, much less intelligent, emerge from a bunch of matmuls or wet mass of brain cells? That's the wrong level of abstraction. How can a general-purpose quasi-intelligence emerge from a stupidly high-dimensional latent space that embeds rich information about the world? That's the interesting question to ponder, and it starts with an important realization: it's not obvious why it couldn't.
I've been around long enough to see this saying that "As soon as it works, no one calls it AI anymore" in action many times.
It is almost infuriating how dismissive people are of such amazing technologies when they understand it. If anything, progress is often marked by having things becoming simpler rather than more complex. The SpaceX Raptor engine versions are such a cool example of that.
> How can you trust something when anyone can edit a page? But again, the power of quantity--thousands or millions of eyeballs identifying errors--swamped any simple attacks.
Sure, but now the established power users are free to insert more subtle attacks. The https://xkcd.com/978/ problem never stopped and the "reliable sources" consideration process allows for considerable political bias.
I don't pretend to know the long term future of llms. But I get this dismissal everytime I suggest "this is unsustainable, this is going to crash". No matter what trends I point to.
I won't pretend to know what lies beyond that. I just know on 5 years you're not going to spam AI in your deck and get millions in funding.
Some of us used to think that meat spontaneously generated flies. Maybe someday we'll (re-)learn that meat doesn't spontaneously generate thought either?
Given that everything the LLM can do it learned from human descriptions of the space ... one would have to posit a very inefficient language for that model not to do something with those billions of parameters. But when you fly because of a bunch of balloons sprinkled with magic fairy dust are pulling you up, the magic fairy dust is still at work.
Before LLMs, it wasn't even clear how much expressive power human languages have when divorced from the innate properties of human mind or other sensory inputs.
An LLM being able to pick up on so many features of human mind, despite not being innately human, just by looking at the text-only language data? Not an outcome everyone expected. Far from it.
That is true, I'm glad that LLMs at least demonstrated for everyone except the most committed solipsists that we are all talking about a shared world state and communicating at least moderately effectively with each other.
Quantity has a quality of its own. The first chess engine to beat Gary Kasparov wasn't fundamentally different than earlier ones--it just had a lot more compute power.
The original Google algorithm was trivial: rank web pages by incoming links--its superhuman power at giving us answers ("I'm feeling lucky") was/is entirely due to a massive trove of data.
And remember all the articles about how unreliable Wikipedia was? How can you trust something when anyone can edit a page? But again, the power of quantity--thousands or millions of eyeballs identifying errors--swamped any simple attacks.
Yes, LLMs are literally just matmul. How can anything useful, much less intelligent, emerge from multiplying numbers really fast? But then again, how can anything intelligent emerge from a wet mass of brain cells? After all, we're just meat. How can meat think?