congratulations on being the highest emdash users of hacker news, I totally get why now.

well done!

Takes more power.

If they dont call it G-Wiz afterwords i'm disappointed in google's culture.

It blows my mind that the cockroach db founders started this 29 years ago.

Amazing to see this release.

I couldn’t tell you what the benefit of cockroach db is or what it’s used for, but I’ve been using Gimp for the majority of its life.

Distributed postgres

That's super funny, never knew that. Amazing that they still work together 30 years later.

I believe theres a related argument that congress might be making here, idk.

I mean, how are we supposed to extract information without ferrets.

and considering most cloud compute has a better backbone or internal routing, userspace should see this less.

That being said, if this is truly a problem for you CRDB is basically built with this all in mind.

Wow have heard about CRDB but just now reading what it does.

That's an incredible piece of engineering.

Anyone have time to dig in and say what tricks its using most?

The code is really extremly simple, just these few files: https://github.com/Starofall/S3HyperSync/tree/main/src/main/...

1) The underlying S3 Framework is already super fast https://pekko.apache.org/docs/pekko-connectors/current/s3.ht...

2) Lots of multithreading, stream buffering and pipelineing

3) For the fast iteration speed the "read,parse,ask for next" loop is the main bottleneck - so if you e.g. know that your sync source prefix contains uuids - the tool creates a file iterator for each known subfolder prefix. And with 16 iterators, its mainly the CPU that bottlenecks the XML parsing :)