SlashDot [1] points to The Register [2] which points to ENGPRAX [3], where it claims this was part of the research for a book. Apparently Agile has a 268% higher failure rate than 'Impact Engineering'. What the hell is that? ENGPRAX kindly sell a book [4] for you to find out!
It's like expert witnesses hired by lawyers - you're not supposed to be able to influence what they say, yet unfailingly they tend to arrive at conclusions the people hiring them want to hear.
Alternatively, if you can get clear requirements up front, your 3x more likely to "succeed".
Fun fact: I once talked to a pmo person who claimed that 95% of their waterfall projects finished on time and under budget (but only if you include the avalanches of change requests that were approved 8p% of the way through, a time that somehow corresponded with the software going into test)
I have an idea on this, and I'd appreciate your feedback and opinions on that.
Could it be that agile projects fail more often (or take longer to complete) because people with a different "mindset" are able to work on projects like these because they are being enabled to do so due to a different "way of working"?
Example: We have a shop relaunch going on and marketing is in the lead. Project is almost double over budget and at least two months over time, they fail so set clear expectations and requirements and there's sprint after sprint after sprint to get things to a release-ready state.
Without agile project management this relaunch would have long been counted as a fail and consequences would have followed OR marketing wouldn't have been in the lead because they would not have been able to set expecatations and a clear goal from the start on.
That’s pretty much the entire promise of capital A Agile (Scrum, Safe etc). It provides a framework for people without any knowledge or passion for software development to micromanage software engineers while ignoring agile principles and not bothering to conduct user research or to investigate product-market-fit or do anything else that might be worth doing other than creating extremely loose requirements they pulled out of the ether on a bi-weekly basis. The argument used to refute this is usually that people aren’t doing agile right and if only the people and companies were more competent and or did MORE Agile then things would be fine. Generally speaking successful tech companies don’t often put unskilled dispassionate middle managers in charge of software projects and don’t often use Agile processes
Software development is a lot like photography. If your work sucks it’s because the engineer/photographer isn’t close enough to the subject/user/business goal. Agile tends to silo engineers away from their ideal motivations (users/business goals) in favour of meeting the needs of the egos of management.
One case in point was a hedge fund who had increased their engineering team by a factor of 10 because they couldn’t get any work done after the Agile people took over. Their productivity had stabilised at a pre-agile /10 level. I was part of a group of consultants brought in to build them something because they were so unproductive. The consultancy’s solution was to do MORE Agile. So we ended up building a large but not very useful set of features instead of doing the smaller, more technically complex and much more useful feature. We completed the work on time only because of the generous bonuses offered to the outsourced developers. I believe it was unlikely to be worthwhile for the hedge fund.
> Agile tends to silo engineers away from their ideal motivations (users/business goals) in favour of meeting the needs of the egos of management.
No idea where you get this from. I've never seen teams closer to customers than with agile thinking. Waterfall with a programme manager at the top setting the timeline, the budget and the features, an army of business analysts getting between the team and the customers, and project managers making GANTT charts - you don't think this silos away information?
> The argument used to refute this is usually that people aren’t doing agile right and if only the people and companies were more competent and or did MORE Agile then things would be fine.
> Generally speaking successful tech companies don’t often put unskilled dispassionate middle managers in charge of software projects and don’t often use Agile processes
But do you think this problem is inherent to the agile process itself? We (as in "more technical department") manage to get our things done on time and budget with the same external agency. It might be due to us formulating clear expectations and behaviours and iterating over improvements, but I'd say this whole process of "lots of little steps" works fine for us.
On the other hand, when I think about it right now, we might be doing fine with a more traditional project management process too, because we are able to set expectations... Hm... not sure...
As one who has participated in many Agile teams as both an individual contributor and as a lead/manager I would say Agile adds a lot of complexity and overhead due to the number of people involved--engineers, designers, product managers, etc.
When companies have too much money, they feel they have to spend it, and so they spend it on hiring too many engineers, designers, product managers, etc. And they also feel they have to keep tweaking things when things are good enough to be left alone, in order to justify all the hiring. It's basically busy work on top of busy work.
My most productive times as a programmer was when it was just me and another programmer hammering out 500+ lines of code per day and where we were our own product managers. When you have a team of 10-12, you need to spend a lot of time creating well-sized tickets (that really are ludicrously small) that are doable in the allocated time of the sprint. Then each programmer spends, say 2 weeks in a 2-week sprint, producing about 50 lines of code, testing it, and then deploying.
TL;DR: There are too many hires (engineers, product managers, etc.) and not enough work, so we end up doing a lot of busy work. That is Agile.
Is it also possible that Agile projects recognise lack of product-market fit or lack of technical feasibility faster, because of shorter feedback cycles?
if so, this avoids wasted effort, and is a feature not a bug.
Since it's talking about putting requirements first - arguably if it's possible or easy for a project to have all the specs immediately ready, it is also likely an easier project that has been solved many times in the past.
There are projects for which you can't know everything ahead of time and you need to iterate and seek customer feedback. These are likely more innovative projects, and of course more risky and more likely to fail.
One of the manifesto items criticized is "Working Software over Comprehensive Documentation". I am reminded that, while the OSI produced a very comprehensive 7-layer model of how networking should work, the IETF produced the TCP/IP stack (among many other things).
If you read between the lines, they're actually saying that agile projects fail to finish on time 268% more often.
Not finishing on time is a very different thing from failing. Especially for agile, where you're supposed to deliver the parts that provide the most value early.
The biggest problem of Agile is that there is no budget for minor refactoring. Refactoring becomes a massive effort to either account for the time or substantiate the changes administratively at sacrifice to some business priority.
At that point any refactor is the automotive equivalent of a recall. That’s unfortunate because when the right processes are in place enormous refactors on even large applications can be an effort shorter than a sprint planning meeting.
Agile is just a way to apply a) constant pressure, b) not plan too much, and c) explode your T&M.
It's really good for a consultancy to churn through developers whilst billing hourly / daily, but less so if you want to build a product, or actual productivity.
That's not to say there's nothing good about agile, but when your average team consists of a Delivery Manager constantly applying pressure and Product Manager with no tech skills deciding what gets worked on, it's immediately obvious nothing actually changed.
Despite what Agile wants to be what matters most is how Agile is in practical application. Speaking from more than 15 years of doing this at many employers it's an administrative tool to bind effort to project management... and nothing more.
I am not saying whether Agile is good or not, but there are better approaches that can yield faster trajectory with lower defects. Its a trade off between automation/autonomy versus administrative oversight.
agile had a lot of promise, though is definitely biased towards working with external devs, but in practice, in my experience, is boils down to the DM and PM roles turning the team into a feature factory 9/10 times.
That said, it isn't really about agile, it's about the specific people, and how suddenly adopting some framework doesn't really change them or their incentives.
To quote Hillel Wayne on LinkedIn, which sums this "study" up nicely:
"""
Yesterday I read a report claiming that Agile projects had "268% Higher Failure Rates" than projects with more upfront planning.
As a fan of upfront planning, and a fan of Empirical Software Engineering, I read the report in more detail.
And I'm pleased to report that it is the single worst study I have ever read.
It is so bad I do not want to share it for fear of spreading a mind-virus, but I can share some highlights:
- The author surveyed 600 people and somehow got P values of 0.00004
- By how he specifically defined "upfront planning", the number of engineers he found practicing it is logically impossible
- Every major calculation table had some sort of error
- The author edited a whole bunch of Wikipedia articles to promote himself, and claims his results can also cure smoking
Today I saw the same report doing on The Register and Slashdot, who both accepted it uncritically. And the hundreds of comments were people accepting the study or suggesting a "confounding factor".
...C'mon, people. You're better than this.
I normally write about Empirical Software Engineering because I think using science to study software engineering is important. But just as important is knowing how to protect yourself from bad science. Anybody who's read a couple of research papers would see P<0.00004 as a major, MAJOR red flag.
Being in STEM doesn't make us immune to pseudoscience. And we're forever vulnerable to seeing something that confirms our beliefs and not digging deeper to make sure it's not lies.
"""
"According to the study, putting a specification in place before development begins can result in a 50 percent increase in success, and making sure the requirements are accurate to the real-world problem can lead to a 57 percent increase"
Thank You Captain Obvious
So when the project spec is unclear? And/or changed during project?
Abandon ship??
Or, utilize a methodology that's capable in adapting to mid-project changes....?
In reality, I'm not sure I've ever seen the "true agile", only excuses of that form. It has the same feel as there being no 'true communist countries': Ease of implementation (and human nature) matters!
Communism works... at small scales. It just doesn't scale up to country size.
Every business with more than a few staff members is a communist dictatorship. Employees don't vote in their boss. Business are not democracies. Employees don't own their own means of production. They share a communal shared pool of capital resources, which is the very essence of communism.
Every functional business is a success story for communist dictatorships.
Where it breaks down is at large scales. Too much bureaucracy and misaligned incentives degrade the efficiency until it's worse than a bunch of competing organisations voting in a democracy.
I'm saying this because Agile is similar. It too works. Just... not in the way that people try to apply it, at huge scales to large inflexible enterprises that say they want Agile but insist on implementing it using every step of the waterfall process, just renamed.
See, the 'true communism' is a democracy. Certainly if you go back to read Marx' book you'll find that's what he was going for.
There's an interesting conversation here -- the governance of the countries where it was tried didn't change very much; new tsar, same as the old tsar -- but my main takeaway was that while you might be able to change the economic system, changing the governance requires deeper changes to the underlying culture, which is usually infeasible.
Scandinavia, especially Norway, are IMO closer to marxist communism than the USSR ever was; last I checked around 40% of the economy is state-run. It seems to work pretty well, and it's democratic. But it's been democratic in one form or another for the last twelve hundred years. Even when it was ruled by Denmark or Sweden, there was local democracy. I suspect it would still be democratic even if it were explicitly communist.
=
And so we come back to Agile, which I think is similar: You get the qualities that are already inherent in the team, implementing something like Agile doesn't make a huge difference one way or the other. Except if you're heavy-handed about it, in which case you can absolutely destroy what's already there.
I think you hit the nail on the head. You can dictate a policy all you want, but you can't (in general) force a change the culture of a large group of people.
As so called "Individual Contributor", my interviewing skills have little impact on people 2 or more rungs of hierarchy above me.
Because that's who pushes "bad agile" so much, who is targeted by all the SAFe propaganda, who tells you to shut up or be fired when you tell them their brilliant ideas about how SCRUM is to be done across the company result in unnecessary friction with no benefits. (I nearly quit there and then because of that talk)
And due to geographical constraints I can't exactly shop around as much as if I lived, let's say, on west coast of USA.
So, "working on my interview skills" doesn't change anything, except maybe grinding leetcode so I can tasteore of the fecal rainbow of corporate agile.
It feels like a significantly more likely explanation is that younger, newer companies are going to embrace the younger, newer agile methodology. Younger, newer software companies are also more likely to take on risky projects and fail than older established companies with older practices.
[1] https://tech.slashdot.org/story/24/06/06/022253/study-finds-...
[2] https://www.theregister.com/2024/06/05/agile_failure_rates/?...
[3] https://www.engprax.com/post/268-higher-failure-rates-for-ag...
[4] https://www.engprax.com/work/impact-engineering-transforming...