Interesting read! As a mathematician I always had difficulties with AI jargon, even though I've been writing on neural networks since the late nineties.
Almost at the same time as the emergence of transformers, I had only minimal contact with the field. I was just aware of the appearance of the Vaswani paper, but only now have I returned to the subject in a way that requires more rigour. And I stumbled upon "attention" in the same way as the author. It did not help to know more about the biological model [1].
Yes, kernels. Asking myself, what software implementations of the superior colliculus or yet retina cell complexes like DS (direction-sensitive) or OMS (object-motion-sensitive) cells could provide.
Almost at the same time as the emergence of transformers, I had only minimal contact with the field. I was just aware of the appearance of the Vaswani paper, but only now have I returned to the subject in a way that requires more rigour. And I stumbled upon "attention" in the same way as the author. It did not help to know more about the biological model [1].
Yes, kernels. Asking myself, what software implementations of the superior colliculus or yet retina cell complexes like DS (direction-sensitive) or OMS (object-motion-sensitive) cells could provide.
[1] for example: https://mitpress.mit.edu/9780262019163/the-new-visual-neuros...