> Can't you just use something like a bloom filter?
If I had the ability to download a massive file I’d try it out on a hextree I toy around with occasionally.
If you’re making an index file may as well just throw it into a tree structure where a lookup is anywhere from 1 to 20 pointer dereferences (assuming the checksum is 20 hex digits) as it optimizes storage so tree depth is variable. Plus it can retain the counts as well.
Now I really want to try this out, the last article I read along these lines I used it as a comparison and it was equally as efficient as their conclusion.
If I had the ability to download a massive file I’d try it out on a hextree I toy around with occasionally.
If you’re making an index file may as well just throw it into a tree structure where a lookup is anywhere from 1 to 20 pointer dereferences (assuming the checksum is 20 hex digits) as it optimizes storage so tree depth is variable. Plus it can retain the counts as well.
Now I really want to try this out, the last article I read along these lines I used it as a comparison and it was equally as efficient as their conclusion.