Financial losses? The comment you're replying to is mentioning heart attack treatment here. We're talking about deaths. Most of us won't like to hear this but for all of us who work at SaaS that is deployed on servers around the worlds, our bugs cause people to die. It's a given that at least a dozen people will die directly (medical flights, hospitals both being hit) due to this broken update, let alone indirectly.
I don't think the parent comment was ignoring that. The penalty for a company who does this can't be to bring someone back from the dead, it's likely to be financial, which is the aspect they're talking about.
As others have already stated, yes, that is how we should be interpreting comments, in good faith and in the most charitable way as the site guidelines suggests us to.
If companies want the nice parts of being "a person", they should also deal with the bad parts of being a person. Financial fines are not enough. Though I'm not sure how we'd build a jail cell for an entire company.
Fines are not enough because a large enough fine will kill a company, destroying lots of jobs and supply chains.
Why not dilute the shareholder pool by a serious amount? There's no need for a statization to formally happen, the government can sell the shares back over time without actually exercising control.
Also fire execs and ban them from holding office on publicly traded companies for the foreseeable future.
Seizing shares doesn't impact the cash flow of the company directly, thus shouldn't cause job losses, but shareholders (who should put pressure on executives and the board to act with prudence to avoid these kinds of disasters) are adequately punished.
This actually sounds like a workable idea, but the implementation would be extremely thorny (impact on covenants, governance, voting rights, non-listed companies, etc) and take forever to get done. It would also punish everyone equally, even though they clearly do not share equal blame.
You probably want, in addition to your proposal, executive stock-based compensation to be awarded in a different share class, used to finance penalties in such cases where the impact is deemed to be the result of gross negligence at the management level.
> but shareholders (who should put pressure on executives and the board to act with prudence to avoid these kinds of disasters) are adequately punished.
So if I own some Vanguard mutual fund as part of a retirement account, it’s now on me to put pressure on 500+ corporations?
Perhaps it’s on Vanguard to do so…but Vanguard isn’t going to just eat the cost of increased due diligence requirements. My fees will increase.
How does that increased due diligence even work? It’s not like I or Vanguard can see internal processes to verify that a company has adequate testing or backups or training to prevent cases like today’s failure.
When, on average, X number of those 500 companies in my mutual fund face this share seizure penalty per year…am I just supposed to eat the loss when those shares disappear? Does Vanguard start insuring against such losses? Who pays for that insurance in the end?
This doesn’t even really hurt the shareholders who are best placed to possibly pressure a company. This doesn’t hurt “billionaire executive who owns 40% of the outstanding shares”. I mean, sure, it will hurt that little part of their brain that keeps track of their monetary worth and just wants to see “huge number get huger”…but it doesn’t actually hurt them. It just hurts regular folks, as usual.
If you own a mutual fund, then you do not own shares of the 500 companies, rather you own shares of the mutual fund itself.
Consequently you don't put pressure on the 500 companies, you put pressure on the mutual fund and the mutual fund in turn puts pressure on the companies it invests in and exercises additional discretion in which companies it invests in.
>Perhaps it’s on Vanguard to do so…but Vanguard isn’t going to just eat the cost of increased due diligence requirements.
Yes they do, because mutual funds do compete with one another and a mutual fund that does the due diligence to avoid investing in companies that are held liable for these kinds of incidents will outperform the mutual funds that don't do this kind of due diligence.
> It’s not like I or Vanguard can see internal processes to verify that a company has adequate testing or backups or training to prevent cases like today’s failure.
I don't know specifically about Vanguard, but mutual funds in general do employ the services of firms like PwC, Deloitte, and KPMG to perform technical due diligence that assesses the target company's technology, product quality, development processes, and compliance with industry standards. VC firms like Sequoia Capital and Andressen Horowitz do their own technical due diligence.
Just perhaps the idea of sticking everyone's retirement funds into massive passive vehicles was a bad one and has an unhealthy effect on the market, as you illustrate here. It is the way of things now so I see your point and it would be harmful to people, but getting in this situation has seemingly removed what could be a natural lever of consequence. We can't really hold companies accountable lest all the "regular folks" that can't actively supervise what they're investing in become collateral damage.
At least with AI you could do something like, destroy all copies including backups, destroy all training data and other code used to generate it. Which to me actually doesn't seem unreasonable punishment.
I did not mean to imply this, as there's a very long culpability chain. For this reason, I'm not sure if it makes any sense to imprison individuals for this. A lot of people playing a part in this causing such chaos.
But it is something to be very aware of for those of us who develop software run in e.g. hospitals and airlines, and should receive more attention, instead of only bringing up financial losses which is what usually happens. I noticed the same with the big ransomware attacks.
Indeed, pity that we need major failures like these, for goverments to finally start paying attention to give the same kind of laws as anything else, instead of careless EULAs and updates without field testing.
It's very bizarre to me how normalized we have made kernel-level software in critical systems. This software is inherently risky but companies throw it around like it's nothing. And cherry on top, we let it auto-update too. I'm surprised critical failures like this don't happen more often.
I can't tell if you're serious or sarcastic, but there is such a thing as criminal negligence.
CrowdStrike knows that their software runs on computers that are in fricken hospitals and airports, they know that a mistake can potentially cause a human death. They also know how to properly test software, and they know how to do staggered releases.
Given what we know now, it seems pretty likely that to any reasonable person, the amount of risk they took when deploying changes to clients was in no way reasonable. People absolutely should go to jail for this.
This more or less originated with the unfortunately named MS Herald of Free Enterprise sinking (https://en.wikipedia.org/wiki/MS_Herald_of_Free_Enterprise) - after that incident, regulators decided that maybe they didn't want enterprise quite as free as all that, and cracked down significantly on shipping operators (though the attempt to prosecute its execs for corporate manslaughter did fail).
Why don't orgs test their updates? Every decent IT management/governance under the sun demands that you test your updates. How the hell did so many orgs that are ISO 2700x, COBIT, PCI-DSS, NIST CSF, etc. certified failed so hard??
(ToS/contracts will probably get you out of any damages.)
Testing for most organizations is usually either really, incredibly expensive or an ineffective formality which leaves them at more risk than it saves. If you aren’t going to do a full run through all of your applications, it’s probably not doing much and very few places are going to invest the engineer time it takes to automate that.
What I take from this is that vendors need a LOT more investment in that work. They have both the money and are best positioned to do that testing since the incentives are aligned better for them than anyone else.
I’m also reminded of all of the nerd-rage over the years about Apple locking down kernel interfaces, or restricting FDE to their implementation, but it seems like anyone who wants to play at the system level needs a well-audited commitment to that level of rigorous testing. If the rumors of Crowdstrike blowing through their staging process are true, for example, that needs to be treated as seriously as browsers would treat a CA for failing to validate signing requests or storing the root keys on some developer’s workstation.
Because historically orgs have been really bad with applying updates: either no updates or delayed updates resulting in botnets taking over unpatched PC's. Microsoft's solution was to force the updates unconditionally upon everybody with very few opportunities to opt out (for large enterprise customers only).
Another complication comes from the fact that operating system updates are not essential for running a business and especially for small businesses – as long as the main business app runs, the business runs. And most businesses are too far removed from IT to even know what a update is and why it is important. Hence the dilemma of fully automated vs manually applied and tested updates.
> Microsoft's solution was to force the updates unconditionally upon everybody with very few opportunities to opt out (for large enterprise customers only).
Not a Microsoft's fan, but this is not true. Everyone who has Windows Server somewhere, with some spare disk space for the updates, has this ability. Just install and run WSUS (included in Windows Server) and you can accept/reject/hold indefinitely any update you want.
1) the prevailing majority of laptop and desktop PC installations (home, business and enterprise) are not Windows Server;
2) kiosk style installs (POS terminals, airport check-in stands etc) are fully managed, unsupervised installations (the ones that ground to a complete halt today) and do not offer any sort of user interaction by design;
3) most Windows Server installations are also unsupervised.
> 1) the prevailing majority of laptop and desktop PC installations (home, business and enterprise) are not Windows Server;
They are not, but the point is elsewhere: that Windows Server is going to provide the WSUS service to your network, so your laptop and desktop installations (in business and enterprise) are going to be handled by this.
Homes, on the other hand, do not have any Windows Server on their network, that's true.
As a hack to disable Windows updates, it is possible to point it to a non-existing WSUS server (so that can be done at home too). The client will then never receive any approval to update. It won't receive any info wrt available updates either.
> 2) kiosk style installs (POS terminals, airport check-in stands etc) are fully managed, unsupervised installations (the ones that ground to a complete halt today) and do not offer any sort of user interaction by design;
That's fine; this is fully-configurable via GPO.
> 3) most Windows Server installations are also unsupervised.
IMHO law should require such a firm, or any firm that may impact millions of other people, i.e. including all OS developers and many others, to maintain a certified Q/A process, maintain a 24/7 coverage and spend X% on Q/A. Such companies should never be allowed to deploy without going through a stringent CD procedure with tests and such, and they need to renew the certificate annually.
These are infra companies. Their incompetence can literally kill people.
My point/problem is that EVERY company (sorry for the caps) that is ISO, PCI, COBIT, NIST CSF, etc. compliant MUST be doing this!! (again sorry for the caps)
So they drop half the 'safety' procedures once the auditor goes away? WTF! (I am semi-angry because there are so many easy solutions and workarounds to not fall for this!! (inside screaming).
How irresponsible must someone be to roll out something to 1k-5k-10k machines without testing it first??
I hope eventually law regards these companies as "infrastructure" companies, just like companies that build roads, bridges and such, that may and will kill people if not run professionally.
I'm not trying to enforce certifications because as a dev certifications always raise a bitter taste in my mouth. But those companies need certified processes that get re-certified every year. Sometimes even a cursory review from outsiders can find a lot of issues.
Updates do get tested. Windows updates can be held and selectively rolled out when a company is ready. As far as I can tell though, CrowdStrike doesn't give companies the agency to decide if updates should be applied or not.
Since we live in a capitalism, financial losses are the only one anyone cares about at scale. What's a human life worth nowadays? About 10 million for a healthy prime age adult? Negative for elderly?
I think it depends what passport etc. you hold...
One dystopian take is the trolley problem, where the self-driving car in question uses smartphones to determine the identity of the people involved, to work out who is cheaper to kill.
That reminds me of why McDonalds got such a high penalty in the court case everyone remembers as "person sues for spilling hot coffee on themselves".
The reason this reminds me of that, assuming that I remember right, is that I think they had even taken the decision that the cost of paying lawsuits for those injuries was lower than the increase in revenue for being able to say "we have the hottest coffee"… and that was why they were deemed so severely liable.
They were definitely shown to have known it was resulting in injuries from other settlements:
Not true. Making C-level executives of software companies criminally liable with the chance to go to jail did change their behaviour in some recent lawmaking situation (forgot which, sorry).
