Albert here :) Decision basically came down to the following: we needed to build a 1 million events/sec log processing pipeline, and had only 5 servers (more were on the way, but would take many months to arrive at the data center). So v1 of this pipeline was 5 servers, each running Kafka 0.8, a service to receive events from the edge and put it into Kafka, and a consumer to aggregate the data. To squeeze all of this onto this hardware footprint, I spent about a week looking for a format that optimized for deserialization speed, since we had a few thousand edge servers serializing data, but only 5 deserializing. Capnproto was a good fit :)
Segment | Engineering | San Francisco, CA & Vancouver, BC | Full-Time | Onsite + Remote for senior engineers
Segment is building the customer data platform for everyone. We transform data and route to over 200 different integrations, adding new ones every day. We're processing billions of events daily and maintain the analytics infrastructure for companies like DigitalOcean, Instacart, New Relic, and Docker. Our goal is to help companies learn from how their users interact with the products to build even better products. We also like to share our work and what we learn, here are some recent examples:
We have a modern stack consisting of Go (golang), AWS ECS/Kubernetes, Docker, JavaScript, React, GraphQL, Kafka, and others! If any of this sounds interesting, we'd love to hear from you! Check out our open positions at https://segment.com/engineering#jobs If you have specific questions, feel free to reach out to marta (at) segment.com (but please do apply on the site).
Segment | Engineering | San Francisco, CA | Full-Time | Onsite
Segment is building the analytics routing layer for the world. We transform data into over 150 different services, adding new ones every day. We’re processing billions of events each month, and maintaining the analytics infrastructure for companies like DigitalOcean, Atlassian, New Relic, and Docker.
Our goal is to help companies learn from how their users interact with the products to build even better products.
We also like to share our work and what we learn, here are some recent examples:
If any of this sounds interesting, we’d love to hear from you! Check out some of our open positions at https://segment.com/jobs or email tido.carriero@segment.com with some info about your background and we’ll kick it off from there.
Kubernetes, Mesos/DC/OS and the various other options are all valid solutions if you need to run in many diverse environments. Titus runs (ran?) on top of Mesos before they built an ECS integration.
That being said, there's interesting benefits to the approach of not trying to abstract the IaaS and deeply integrating with one or two clouds instead. You get to leverage much more of the cloud(s) you've decided to adopt if you don't immediately try to abstract them away.
> That being said, there's interesting benefits to the approach of not trying to abstract the IaaS and deeply integrating with one or two clouds instead.
I actually thought about mentioning Convox in this vein, as it is intended to be for AWS only.
> You get to leverage much more of the cloud(s) you've decided to adopt if you don't immediately try to abstract them away.
It depends on the services. For the bulk services (blobstores, VMs, networking, queues, databases etc) the big 3 support, you can usually have some degree of migration without too much pain, if you're using one of the platforms correctly.
For example, on migrating a Cloud Foundry platform from AWS to GCP, you can switch your service brokers from AWS to GCP. Apps that ask for a database won't, in general, know or care who's providing it.
Another thing is that a fair amount of the specialising services now have portable alternatives. A good platform can integrate these on an equal basis. At the moment there's an effort to extract the Cloud Foundry service broker model so that it can also be used by other platforms[0], so this will become even easier with time.
Where AWS still have a sustained lead is on sheer breadth of offerings. But my hunch is that Google know this is the first real opportunity to produce a second revenue stream comparable with advertising. They're going to brute force their way to near-parity with a lot of engineering and money.
This site was mostly created as a joke after I watched a couple of @aspyker's great presentations about Titus and realized that we have a huge amount of exciting work to do to get our own container platform on AWS to the same level.
Netflix should do what makes sense for them. I understand that doing a good job of releasing a very complex piece of software as open source is hard and takes a lot of effort.
Not rolling our own exactly: running on ECS now and busy evaluating Kubernetes. These are still quite blunt instruments though.
At some point, as a company grows, it's not just about running some containers. You want batch jobs, complex placement constraints, scheduled tasks, etc., etc.
If you haven't watched through https://youtu.be/4OLlKGT7aVQ, I can recommend it. aspyker does a great job of enumerating many of their complex use-cases.
The Titus team at Netflix is building this on top of EC2/Mesos and ECS.
The rest of us still need to do a bit of work to find/adapt/build our own.
> The rest of us still need to do a bit of work to find/adapt/build our own.
I usually recommend people just take an existing PaaS and run with it. I'm wildly biased in this regard because I work on one as my day job.
Also because rolling your own is hard. Really really hard. We have literally hundreds of engineers in dozens of teams, and that doesn't count the engineers our partners have assigned. We've been cranking for years. It's just a lot of hard engineering for stuff you don't see until it arrives, and arrives unpleasantly.
Kubernetes makes it a lot easier, especially if you need lots of flexibility, but it still leaves you to assemble a lot of parts and management is left very much as an exercise for the reader.
If you build on Kubernetes, may I recommend Kubo[0] to manage it? We built it with Google, but because it's based on BOSH you can deploy to AWS, GCP, Azure, OpenStack, vSphere and even bare metal with RackHD. BOSH by itself is a massive force multiplier for operations.
