My point is not about morality. It’s about ROI focus and that OpenAI can’t and won’t ever return anything remotely close to what’s been invested. Adult content is not getting them closer to profitability.
And if anyone believes the AGI hyperbole, oh boy I have a bridge and a mountain to sell.
LLM tech will never lead to AGI. You need a tech that mimics synapses. It doesn’t exist.
I have also a hard time understanding how AGI will magically appear.
LLMs have their name for a reason: they model human language (output given an input) from human text (and other artifacts).
And now the idea seems to be that when we do more of it, or make it even larger, it will stop to be a model of human language generation? Or that human language generation is all there is to AGI?
Because the first couple major iterations looked like exponential improvements, and, because VC/private money is stupid, they assumed the trend must continue on the same curve.
And because there's something in the human mind that has a very strong reaction to being talked to, and because LLMs are specifically good at mimicking plausible human speech patterns, chatGPT really, really hooked a lot of people (including said VC/private money people).
LLMs aren't language models, but are a general purpose computing paradigm. LLMs are circuit builders, the converged parameters define pathways through the architecture that pick out specific programs. Or as Karpathy puts it, LLMs are a differentiable computer[1]. Training LLMs discovers programs that well reproduce the input sequence. Roughly the same architecture can generate passable images, music, or even video.
It's not that language generation is all there is to AGI, but that to sufficiently model text that is about the wide range of human experiences, we need to model those experiences. LLMs model the world to varying degrees, and perhaps in the limit of unbounded training data, they can model the human's perspective in it as well.
The words 'lead to' there cover a lot. I don't think we'll get AGI just by giving more compute to the models but modifying the algorithms could cover a lot of things.
Like at the moment I think during training new data changes all the model weights which is very compute intensive and makes it hard to learn new things after training. The human brain seems to do it in a more compartmentalised way - learning about a new animal say does not rewrite the neurons for playing chess or speaking French for example. You could maybe modify the LLM algo along those lines without throwing it away entirely.
The need for new data seems like it has outpaced the rate at which real data is being generated. And most of the new data is llm slop.
So you might improve algorithms (by doing matrix multiplications in a different order.... it's always matrix multiplications) but you'll be feeding them junk.
So they need ever increasing amounts of data but they are also the cause of the ever increasing shortage of good data. They have dug their own grave.
Because always/never are absolutes that are either very easy or very hard to see through. For example, 'I will never die', 'I will never tell a lie', 'I will never eat a pie' all suffer through this despite dying being the most implausible. And it gets worse as we get most abstract:
'Machine will always know where to go from here on now'.
AGI might be possible with more Param+Data scaling for LLM. It is not completely within the realm of impossible given that there is no proof yet of "limits" of LLM. Current limitation is definitely on the hardware side.
This is what I'm talking about. The correct tech would enable the strands of information in a vector to "see" each other and "talk" to each other without any intervention. This isn't the same as using a shovel to bash someone's head in. AGI would need tech that finds a previously undocumented solution to a problem by relating many things together, making a hypothesis, testing it, proving it, then acting on it. LLM tech will never do this. Something else might. Maybe someone will invent Asimov's positronic brain.
I think _maybe_ quantum computing might be the tech that moves AGI closer. But I'm 99.9999% certain it won't be LLM tech. (Even I can't seriously say 100% for most things, though I am 100% certain a monkey will not fly out of my butt today)
Quantum compute would definite make a leap to moving closer to AGI. Calculating probability vector is very natural for quantum computer or more precisely any analog compute system would do. qubits==size(vocab) with some acceptable precision would work i believe.
The processing capability of today’s CPU’s and GPU’s is insane. From handheld devices to data centers, the capability to manipulate absurd amounts of data in fractions of a second is everywhere.
Maybe it is the algorithms. But just by doing a op for an 10^25 param llm is definitely not feasible on todays hardware. Emergent properties does happen at high density. Emergent properties might even look as AGI.
And if anyone believes the AGI hyperbole, oh boy I have a bridge and a mountain to sell.
LLM tech will never lead to AGI. You need a tech that mimics synapses. It doesn’t exist.