1. Developers learning that running code is not the same thing as reliable code. Certain things must be put in at design time to allow for operations.
2. Operations people supporting development by formalizing/streamlining the deployment process to have changes occur faster, safer, and more often.
3. Aligning goals and attitudes in such a way that prevents conflict between teams. Development doesn't get to shift the costs of outage prone unmaintainable code onto an ops team. Or that ops doesn't roadblock/veto all changes.
In the bad old days, it was common for developers to act with little care for operational outcomes. The responsibility was far removed, and bugfixes didn't win many accolades. Managers want wish fulfillment and pushed developers to toss new features into the mix as fast as possible, resulting in tremendous pressure against ops teams to deploy despite increasingly crufty and debt laden code bases causing unpredictable problems. Ergo, ops teams who blocked all changes for trivial issues because they were judged only based on outages, not feature delivery.
Yes, devops is not one person, but team is devops. It comes from standards, that developers should not have access to production data, so you need ops. But on the other hand you don't want to have dev and ops team people who throw shit over the fence.
So basically it comes to having devs, test, ops, sec, persons in one team with common goal of delivering featuers.
Because most friction was on boarders like dev | test | ops | security, all those people being on role specific teams would do what is important for their team. So test people would blame devs, sec people would blame ops, test or devs. When you are in one team that has common goal of delivering feature X and you are measured by delivering feature X, everyone is willing to take pragmatic approach instead of "code has to be perfect", "tests have to be perfect", "security has to be perfect".
I read somewhere "magic happens if you seat testers and developers in the same room", I belive it was "Project Phoenix" book. But I despise from the start having those devs vs test people or test people beeing happy about finding bugs. They should be sad because what matters is delivering feature, so they should be happy when they don't find anything. That said, thay should do all they can to find bugs, because we have common goal: "working software".
Provisioning servers, databases, load balancers, etc and other resources use to involve someone driving down to the colo and installing hardware. Often now, it's a yaml file you run through Cloud Formation.
Devops is just as much development and software engineering when you are working in a cloud environment -- it's an API call.
I provided what is imo a very clear and apples-to-apples analogy there for you: Drafting room vs Boiler room. (Replaces guages with graphs of alerts and logs and you'll get the idea.)
Certainly, in other engineering fields, visits to the installation sites by (designing) engineers, or even running pilot programs to debug the system as designed and implemented is a norm. However, the notion that 'an engineers job is to be responsible for x-widget from design to operation' is entirely foreign to any notion of "Engineering" that I am familiar with.
It’s a poor “senior” software engineer who doesn’t understand anything about the underlying architecture or how to at least troubleshoot the infrastructure and have at least some type of understanding of the ops part.
How many hardware based Kickstarter projects have failed because they didn’t understand the complexity of actually manufacturing there product at scale.
It’s analogous to software engineers who don’t appreciate ops. Writing software for a company is useless if you can’t ship it or deploy it. You can’t imagine the number of bright eyed bushy tailed “software engineers” I’ve dealt with that thought they were smart because they knew leetCode and learned algorithms who couldn’t troubleshoot performance issues or understand why they couldn’t get shipping products because they didn’t understand the ops side of things.
Of course I am not diminishing the role of software engineering, I’ve been developing software actively professionally or as a hobby for over 30 years. But, I know enough in theory to talk to the ops guys for on prem/colo/hybrid architectures and I could easily be competitive on the ops side for jobs dealing with fully “cloud native” solutions for a small to medium size implementation.
It would be a poor architect that didn't understand anything about building technology, but that doesn't require spending time laying bricks as a laborer.
That's exactly the reason. Software engineering is about writing code, devops is about writing yaml files and keying into functionality that was implemented by software engineers.
> Software engineering is about writing code, devops is about writing yaml files and keying into functionality that was implemented by software engineers.
This attitude is the ultimate reason for dysfunctional development teams and the source of the barrier between operations and development.
It’s not meaningfully true at all. The problems I deal with as a DevOps engineer are much broader, more abstract, more interdependent, and the solutions much less established than those that I dealt with as an SE. Unless you’re an SE in some particularly complex domain with especially limited resources (which are essentially attributes of DevOps work—SE work applied to a difficult domain under tighter-than-usual constraints), DevOps is probably the more advanced role.
I work in devops (and did so before it was called that), I write code in turing complete languages. Just code that manages infrastructure and software rather then interacts directly with end users. I template configuration files rather then user interfaces. I have to consider race conditions in complex systems, consider the robustness and security implications of anything I do, verify my input (that's you, you hotshot "software engineer") and fail gracefully if needed.
Tell me that is not software engineering, the only difference is that my ability to test before deployment is much more costly and limited, I have to get it right the first time around..
I agree. Even when writing cloud formation templates you have to sometimes create what basically consisted of locks to ensure that one resource doesn’t get created before another one and ensure that certain resources don’t get created in parallel to avoid throttling (parameter store).
And if that’s not “real programming” sometimes you actually do end up writing custom resources in a “real programming language” that get called from CF to create resources.
I’m very much up and down the stack any given week going from the front end, middleware, databases and infrastructure not to mention building out CI/CD pipelines.
> Software engineering is about writing code, devops is about writing yaml files and keying into functionality that was implemented by software engineers.
Creating these strict barriers between teams, especially with the condescension you elicit is exactly how problems arise.
Suppose software engineers write an O(N^2) algorithm because they didn't think the data would grow large. It does. DevOps then launches many more dozens of computers to handle the algorithm. If no one speaks to eachother, everyone thinks things are going smoothly when in reality it's a complete disaster. The DevOps team has much more visibility into the usage, performance, and cost of apps than many developers. Talk to your breathen wisely.
I pivoted a couple of years ago from SE to DevOps, and the latter has been far more complex in my experience.
Probably most importantly, my experience on DevOps has been a bit like hurried product management and software engineering work--I have to identify problems and opportunities to improve software development efficiency, uptime, etc and find/build technologies that solve those problems, and all of these technology choices are really complicated because they are interdependent. How you structure your CI/CD pipeline depends a lot on your orchestration platform (and vice versa). It also depends on whether you use micro services or a monolith, and whether your platform is mono/bi-lingual or fully multilingual. And unlike product management or software engineering, you have to deliver value in a much tighter timeframe (you don't get to spend a year building out a solution with budget and resources for user testing, QA, etc). Further, if you think the frontend web stack is changing fast, the DevOps world will make your head spin!
Regarding YAML, it's just the (currently popular) interface; it doesn't say anything about the amount of complexity. It was originally YAML/JSON because everyone thought it would be super simple configuration, but virtually every infrastructure-as-code (IaC) tool is moving away from YAML to something more powerful.
AWS has CloudFormation which began as JSON/YAML with features hacked on to support things like referencing other nodes in the document, defining and calling functions and macros, evaluating conditionals, looking up values in maps, etc. Now they're building the Cloud Development Kit (CDK) to let developers write IaC in languages like Python and TypeScript, because even the extensions to CloudFormation are insufficient (no ability to pass around parameters, pass structured data, perform computations, etc).
I don't use Terraform, but I understand HCL started out a lot like YAML but is also becoming increasingly powerful with each release.
Helm also started with YAML but is adding in Lua.
The argument that DevOps is easier than SE because it's just using tools created by SEs is silly on its face. First of all, those tools were _implemented_ by SEs but created by much larger teams, including product managers. Secondly, "SEs" are also just using tools created by SEs--compilers, interpreters, VMs, build tools, etc. Not to mention all of the tools and processes built and/or assembled by DevOps engineers.
So would you consider it more “software engineering” using AWS’s Cloud Development Kit and writing code to generate cloud formation files “software engineering”?
I use to think that all “true” developers needed to know the ins and outs of assembly because every time I went to a new platform that’s the first thing that I did - learn assembly language (65C02, 68K, x86), but then I grew up.....
CloudFormation takes a text file and generates API calls much like any code compiled down to assembly.
You could say the same thing about software engineering. The vast majority of it is simply assembling components that already exist. I've worked on dozens if not hundreds of web apps, and most of it is just gluing stuff together. Often it's done on autopilot (or literally automated code generation). I've also worked on a lot of code that was original and challenging. Systems work is the same thing. Most of the time you're just assembling existing components, but the overall design of the system is absolutely engineering.
Right, just like telescope building is about making lenses and hollow tubes, astronomy is about looking at the sky and keying into the functionality that was implemented by telescope makers.
I believe you would benefit from a different perspective because this vision is partial at best. The distinction between who builds system and tools and who uses them is subtle and often these roles are interchangeable. Both have dignity and purpose and contribute to the goal, regardless of the collar color, which is a remnant of the last century.
People who are really serious about software should make their own hardware.
That is no different from people who call themselves “software architects”. You can’t build great software architectures if you don’t know how to build and optimize the underlying architecture it runs on and how to build the pipelines that put the code on those systems.
Dude, the yaml files I'm writing require good knowledge of the system they are configuring.
i deploy all our cloud instances using Ansible (yml files) and I wouldn't have a hope of doing that if I didn't understand the OS I'm configuring with it (Linux).
I might see your point if you're talking about the yml that is used to configure 'value added' products from cloud vendors, but I avoid those like the expensive handcuffs that they are.
I use ansible to provision Centos 7 instances, and then I use another playbook to install and configure stuff on them like haproxy, bind 9, etc...
This leaves me in complete control and keeps prices low because I only ever use vanilla instances.
I'm not on Amazon right now, but when I was I used to use ansible to configure networking on it and security groups etc, but I would always try to keep the vendor specific stuff to a minimum.
If you’re using Amazon as a glorified colo, you’ve got the worse of both worlds. You’re spending more on infrastructure and you’re not taking advantage of managed services to decrease the time and staff you spend on babysitting infrastructure. There is a lot more to AWS than VMs.
I'm using an internal cloud service at my company. It does not offer any managed services.
Also when I do use AWS for clients, I _sometimes_ use managed services¹, but most of those lock you in to Amazon. Some clients don't care, some do. They are paying, so I don't care either way.
In either case, I spend very little time babysitting infrastructure, just FYI, even when I use bare instances.
And in my opinion² if people aren't careful about managed services, then they won't have any OS knowledge anymore, and will just be left with product knowledge.
Sure, managed services make things easier, but they also make your sysadmin muscles weak.
1: Aurora DB for example. That one I like because there's no lock in. I can always move my tables to a standalone MySQL.
2: As someone who is 100% comfortable using bare Linux without having to use containers, managed services, control panels or even google searches to keep my stuff running, and have been doing so since the early 90s.
Sure, managed services make things easier, but they also make your sysadmin muscles weak.
I want my sysadmin muscles to be weak. I’m not saying that sysadmin isn’t important, but why spend time managing servers and doing the “undifferentiated heavy lifting” instead of adding value and dealing more with strategy?
Where does code begin and end - I do work that could be described as devops, and I'm often writing python or bash to do something work related - according GP (and by extension, you.), I'm just a blue collar grunt
Yup, I also do work that can be described as DevOps. I have strict code reviews, have to write design documents, and develop code in Python, Bash, and C for the embedded platforms I deploy to. I'm happy to be called blue collar grunt or whatever, the pay is amazing and I have way more visibility than I ever did as an SWE.
Labels like these are meaningless. I make more money as a DevOps Engineer than I did as a Google SWE, and I have significantly more responsibilities. The DevOps community is diverse. There are “DevOps” engineers who can’t write code (sadly) and those that write incredibly advanced serverless systems that scale massively.
1. Developers learning that running code is not the same thing as reliable code. Certain things must be put in at design time to allow for operations.
2. Operations people supporting development by formalizing/streamlining the deployment process to have changes occur faster, safer, and more often.
3. Aligning goals and attitudes in such a way that prevents conflict between teams. Development doesn't get to shift the costs of outage prone unmaintainable code onto an ops team. Or that ops doesn't roadblock/veto all changes.
In the bad old days, it was common for developers to act with little care for operational outcomes. The responsibility was far removed, and bugfixes didn't win many accolades. Managers want wish fulfillment and pushed developers to toss new features into the mix as fast as possible, resulting in tremendous pressure against ops teams to deploy despite increasingly crufty and debt laden code bases causing unpredictable problems. Ergo, ops teams who blocked all changes for trivial issues because they were judged only based on outages, not feature delivery.