I can only speak for myself, and some of the reasons why K8s has left a bad taste in my mouth:
- It can be complex depending on the third-party controllers and operators in use. If you're not anticipating how they're going to make your resources behave differently than the documentation examples suggest they will, it can be exhausting to trace down what's making them act that way.
- The cluster owners encounter forced software updates that seem to come at the most inopportune times. Yes, staying fresh and new is important, but we have other actual business goals we have to achieve at the same time and -- especially with the current cost-cutting climate -- care and feeding of K8s is never an organizational priority.
- A bunch of the controllers we relied on felt like alpha grade toy software. We went into each control plane update (see previous point) expecting some damn thing to break and require more time investment to get the cluster simply working like it was before.
- While we (cluster owners) begrudgingly updated, software teams that used the cluster absolutely did not. Countless support requests for broken deployments, which were all resolved by hand-holding the team through a Helm chart update that we advised them they'd need to do months earlier.
- It's really not cheaper than e.g. ECS, again, in my experience.
- Maybe this has/will change with time, but I really didn't see the "onboarding talent is easier because they already know it." They absolutely did not. If you're coming from a shop that used Istio/Argo and move to a Linkerd/Flux shop, congratulations, now there's a bunch to unlearn and relearn.
- K8s is the first environment where I palpably felt like we as an industry reached a point where there were so many layers and layers on top of abstractions of abstractions that it became un-debuggable in practice. This is points #1-3 coming together to manifest as weird latency spikes, scaling runaways, and oncall runbooks that were tantamount to "turn it off and back on."
Were some of these problems organizational? Almost certainly. But K8s had always been sold as this miracle technology that would relieve so many pain points that we would be better off than we had been. In my experience, it did not do that.
Truthfully, I don't know. But I suspect I'm not the only one who feels a kind of debilitating ennui about the way things have gone and how they continue to go.
- It can be complex depending on the third-party controllers and operators in use. If you're not anticipating how they're going to make your resources behave differently than the documentation examples suggest they will, it can be exhausting to trace down what's making them act that way.
- The cluster owners encounter forced software updates that seem to come at the most inopportune times. Yes, staying fresh and new is important, but we have other actual business goals we have to achieve at the same time and -- especially with the current cost-cutting climate -- care and feeding of K8s is never an organizational priority.
- A bunch of the controllers we relied on felt like alpha grade toy software. We went into each control plane update (see previous point) expecting some damn thing to break and require more time investment to get the cluster simply working like it was before.
- While we (cluster owners) begrudgingly updated, software teams that used the cluster absolutely did not. Countless support requests for broken deployments, which were all resolved by hand-holding the team through a Helm chart update that we advised them they'd need to do months earlier.
- It's really not cheaper than e.g. ECS, again, in my experience.
- Maybe this has/will change with time, but I really didn't see the "onboarding talent is easier because they already know it." They absolutely did not. If you're coming from a shop that used Istio/Argo and move to a Linkerd/Flux shop, congratulations, now there's a bunch to unlearn and relearn.
- K8s is the first environment where I palpably felt like we as an industry reached a point where there were so many layers and layers on top of abstractions of abstractions that it became un-debuggable in practice. This is points #1-3 coming together to manifest as weird latency spikes, scaling runaways, and oncall runbooks that were tantamount to "turn it off and back on."
Were some of these problems organizational? Almost certainly. But K8s had always been sold as this miracle technology that would relieve so many pain points that we would be better off than we had been. In my experience, it did not do that.