> "Redis on Linux uses an OS feature called Fork/Copy On Write. This feature is not available on Windows,
The memory-mapped file APIs on Windows are seriously lacking. Last time I dealt with them, there was no documented way to write to a constantly-growing mmaped file (unless you knew its size already). All the workarounds I tried ended in segfaults :/. Curious if anyone here has had success with that.
You can change the size of the allocation on the fly, and just keep writing bytes as you go. Pages aren't alloc'd on disk until they're touched. When you hit the end of the file, you call a final resize to trim off the virtual bytes at the end.
> You can change the size of the allocation on the fly, and just keep writing bytes as you go.
How do you do this technically? mremap(), I assume?
You can do something very similar on windows, by VirtualAlloc()ing at the end of the previous allocation, and if that fails, be committing, VirtualFreeing and remapping at a different address.
Yes, I agree mremap() is much cleaner. I am mostly a Linux guy myself. What I'm missing in Linux is "mremap2()", which would map an existing block of memory onto a file (whereas mmap maps an existing file onto a block of memory ). redis achieves similar functionality (and more) by fork()ing and write()ing instead, but it would be nicer if it could be done with one call and no fork()ing. a "copy this file from here to here" could then be done by mmap()ing the source into memory, mremap2()ing the memory into the destination file, and forgetting about it.
added:
This would also allow btrfs/zfs to have O(1) file copy, even between independent media if possible (i.e. source file already exists on target media, even if in a different directory) - without any consideration from the cp implementation.
Yes, with ftruncate(), see the link above. The problem is Windows doesn't have an ftruncate() that works while the file is open and being written to. (Note: my memory is a little fuzzy here, it might be that there was no implementation that took a file descriptor instead of a filename, either way I couldn't pull it off).
As for your 'zero-copy dump memory into a file after-the-fact' idea, I'm sure it's done somewhere. My brain is too frizzled right now to scrounge up how though :).
When I was still following the kernel mailing lists, there were discussions about how to achieve this for sockets, and none about files, so that would surprise me.
However, if there is a way to achieve that without forking, I would be very happy to hear.
Redis is hosted on Github. Most users of Redis are on Github, too. So if MS wants to exchange code back and forth between their fork and the original, it makes sense to host their code on Github, too. A pull request is much more friendly than "Hey, check out my Codeplex", even if antirez decides not to merge the changes.
The memory-mapped file APIs on Windows are seriously lacking. Last time I dealt with them, there was no documented way to write to a constantly-growing mmaped file (unless you knew its size already). All the workarounds I tried ended in segfaults :/. Curious if anyone here has had success with that.