I should have been more clear. While distillation is part of how we got lucky here, what I really think is that it's just surprising and lucky that such a heavily data-driven approach ended up being so powerful here.
Transformers are like just a step or two removed from being fancy convolutional neural networks. I guess I'm just surprised that it didn't turn out to require more 'special sauce' with extremely elaborate internal architectures, and less of a big-data approach.
Because the data is so central in building these LLMs, rather than some special insights or ideas in the model architecture, or very special hardware requirements, the field is much more open than I would have guessed some years ago. And it's the fact that the data is so central that makes distillation possible in the first place.
Transformers are like just a step or two removed from being fancy convolutional neural networks. I guess I'm just surprised that it didn't turn out to require more 'special sauce' with extremely elaborate internal architectures, and less of a big-data approach.
Because the data is so central in building these LLMs, rather than some special insights or ideas in the model architecture, or very special hardware requirements, the field is much more open than I would have guessed some years ago. And it's the fact that the data is so central that makes distillation possible in the first place.