2. Kernel doesn't have enough free memory, goes to offload something old to swap.
3. Swap file is too small, needs to be extended. This calls the "extend file size" filesystem code.
4. Filesystem code tries to allocate memory for keeping track of filesystem data.
5. Goto 1.
As far as I understand, the "swap file on filesystem" system bypasses the filesystem. The kernel asks the filesystem to give it a bunch of blocks to work with and then ignores the filesystem code entirely. This is why some filesystems can't have swap files -- the FS has to support the right APIs, and has to be prepared for this kind of deal.
A simple solution to this would be for a filesystem to say "I guarantee that if RAM allocation fails, I will still be able to extend a swapfile".
In most filesystems, extending an already open file is fairly straightforward - it's usually a matter of writing just a single block.
Since this is an incredibly rare case, there is no strict performance requirement. It could use an approach to filesystems like grub which involves reading and writing just a single block at a time.
And come on, "writing a single block"? No, that's what it looks like on the user side of things. In the kernel it's going to be a good deal more complicated, because how does anything know anything about this block you just added? You need to keep track of it somehow.
Inside the kernel you need to go find some free space to use for this extra block, and to update whatever metadata is needed to make things consistent. It might well turn out that the block you're using to keep track of what blocks belong to this file is just full, so now you have to allocate another block for filesystem metadata. Then there's issues like journalling. And doing all that while providing the guarantee that you won't need to allocate memory is likely tricky.
1. Something asks for memory from the kernel.
2. Kernel doesn't have enough free memory, goes to offload something old to swap.
3. Swap file is too small, needs to be extended. This calls the "extend file size" filesystem code.
4. Filesystem code tries to allocate memory for keeping track of filesystem data.
5. Goto 1.
As far as I understand, the "swap file on filesystem" system bypasses the filesystem. The kernel asks the filesystem to give it a bunch of blocks to work with and then ignores the filesystem code entirely. This is why some filesystems can't have swap files -- the FS has to support the right APIs, and has to be prepared for this kind of deal.