Ever org I've worked for slowly became unable to innovate because of "institutional knowledge".
I think it's important to forget, to some extent, every once in a while. It forces a new traversal of the problem space, but in a modern context, with modern tools, fresh eyes, and a better understanding of what's needed. Thus we have Space X.
There are some great interviews with Jim Keller, who has a similar perspective: you need to restart every some years, to not stagnate.
From what I've seen, if you want to stamp out a young engineers creativity, start them working in a big org.
And did you stick around to see what happens after they go full tabula rasa? I have: they fall so far behind they close up shop. The proper response is to move slowly and intentionally and use indicators. Like ‘design thinking’, with prototyping and feedback and incremental feedback, not burn shit to the ground because planning is too hard mentality. But I realize that’s what shareholders demand. And now that gov’t is being run like a business, prepare to see bankruptcies.
Fortunately other, more rational countries, will fill the gaps. They’ve just been complacent because the US was on its game for so long. Especially the ESA.
Entrenched ways are not the same as institutional knowledge. You can have one without the other. Some things are not written down but executed as part of common knowledge, and if you drop enough people you'll have to relearn it the hard way.
> Entrenched ways are not the same as institutional knowledge.
I disagree, fundamentally. Institutional knowledge is a set of "truths" that are respected, so necessarily prune the solution space. The only way to get those pruned branches back is to disregard it, by reconsidering, and re-traversing that solution space.
Heving been there 95% of the time you just learn the hard way whp theeold timers did it that way in the first place - and most of the rest you just make a different compromise that has different tradeoffs but isn't better.
In my eyes, institutional knowledge is figuring out what things do NOT work, not what did back then. When you lose that, you're wasting years or even decades hitting the exact same pitfalls the old guard hit once upon a time. Many truly bright minds aren't those asserting how this was the best method and then burying their heads in the sand when other ideas come out.
My time working with some of the brightest minds in my industry taught me that they aren't necessaily some visionary, nor super genius, nor even some workaholic putting 20+ hours a day into their craft (though I have met a few I would describe as such). The gap between me and them wasn't over some raw intellect. It was many times a matter of me thinking of an idea and them talking about how that was tried 5-20 years ago and why that lead down a huge rabbit hole.
> is figuring out what things do NOT work, not what did back then
Yes, this is my point. It's pruning the solution space before traversing it. The diligent approach is to temporarily disregard that knowledge, and do a quick re-exploration to test if it's still true.
> to some extent
This was put in my first comment with severe intent, that many seemed to have missed.
In my eyes, they already traversed it and can explain why it's very similar to an old idea. They don't must see an old idea woth lipstick on it and instantly jump on it for the sake of trying something new.
As of now, this same mentality is used to push AI into everywhere. Not only is the intent bad, but the tech doesn't even work. That's not "resisting change". That's experimenting and realizing the hype was just that.
>This was put in my first comment with severe intent, that many seemed to have missed.
The comment itself definitely reveals more than a light suggestion.
> In my eyes, they already traversed it and can explain why it's very similar to an old idea.
This was precisely my point in [1]. It's fundamentally the same: a pruning of the solution space without re-traversal.
> why it's very similar to an old idea.
And, with diligence, you verify that the new context is exactly the same as the old. You do this by knowing that it might not be, in other words, you temporarily suspending your trust in that knowledge, and re-traverse it with the current context.
It's a thin line of wisdom and conservatism, but an important distinction. People in these positions work on billion dollar software, so they can't just try out every idea that comes to mind in prod. But that's exactly what tends to be proposed: big multi month initiatives, not some prototype to test over a sprint.
The important question I learned to ask was "what problem am I trying to solve". One aspect of this thin line tends to be a muddy answer to this question. When you can only suspect and make grand showings instead of showing pragmatic use case you may not in fact be iterating, but experimenting.
> People in these positions work on billion dollar software, so they can't just try out every idea that comes to mind in prod.
Exactly, the ability to innovate ceases. Risk is most easily avoided by leaning on the existing institutional knowledge to direct new decisions even though they may be in new contexts.
> you may not in fact be iterating, but experimenting.
By definition, iteration is not innovation. Innovation is new ideas. New ideas aren't possible without experimentation, otherwise they would be known ideas.
Most large companies move from innovation to acquisition for a reason: the risk of innovation is too great for a large company to stomach.
I think you are using a non-standard definition of institutional knowledge that begs the question in the rest of your posts.
The standard phrase "institutional knowledge" merely refers to knowledge and skills that are carried by members of the organization. This is often much more than what is formally codified into the processes and training materials. As such, it can lead to loss of capability when there is too much turnover.
You seem to be conflating it with some other kind of bureaucratic conservatism or group-think. While that is a common dysfunction of long-running organizations, I think it is an orthogonal characteristic.
> The standard phrase "institutional knowledge" merely refers to knowledge and skills that are carried by members of the organization.
Yes, that't is my definition. But, that knowledge has very real practical effects and influence on the org, from the weight (those with it usually are in position of seniority/power) and momentum that knowledge carries, especially when approaching new problems, or reconsidering old problems. The mechanism for that can be anywhere from "this is industry standard" to "the director says we should focus on this approach", with the ever present "lets not risk it".
> While that is a common dysfunction of long-running organizations, I think it is an orthogonal characteristic.
I agree that it's logically orthogonal, but not practically. I think the actual killer of orgs is the sum of all the small scale risk avoidance. I think risk is most easily avoided by adhering to the institutional knowledge (what was done and what is known). Innovation eventually becomes a completely foreign concept.
Sorry it wasn't clear enough, but I was paraphrasing Jim Keller. See the several Lex Fridman episodes with him, where he talks about it. The success with his projects would probably be the data you're looking for.
This seems similar to the "let's just rebuild from scratch" impulse that has been tried so many times on very large complicated systems and often, although not always, fails.
But what are we even doing here? We have an entire political party that is anti science. This isn't some temporary setback, it's an ideology.
Is the future going to be completely defunding science for 4 (or 8) years and then whipsawing back to normal levels once the republicans lose power?
It seems like as long as the parties are close to popularity of each other and one party is explicitly anti science there is no way to build anything sustainable. This is no way to run a country.
> Ever org I've worked for slowly became unable to innovate
There's more to life than "innovation", but what you say tracks, I was part of an organization that became extremely innovative but went bust in the process.
> we have Space X.
Which is amazing for earth-orbit commercial launches, but doesn't move the needle for the many other things NASA concerns itself with, like research as mentioned elsewhere.
I'm not sure dot-com era DEC had much stagnation or institutional knowledge that wasn't continuously overrun, nor would it be comparable to most big orgs these days.
He started in a greenfield industry, the immediate pre-requisite for dot com era, then through the dot com era. There was no institutional knowledge when he started, and a good portion of it would be irrelevant when he quit. It was all new.
When he started at DEC in 1982, they had 67,000 employees and almost $4 billion in revenue. It seems like that type of success and size would imply some institutional knowledge. Their revenue, income, and employee count started stagnating in 1989. He worked there until 1998.
No need to keep going back and forth on this as you seem to have dug in your heels.
It's not back a fourth as much as you think I've stated some hard black and white rule, without exceptions. I think it's generally true. In this case, is the exception DEC or Jim Keller? Would he agree? I don't know. Some large orgs run like a collection of startups, internally.
But, I don't think DEC, a company working through the beginning of computer through peak dot com era, where every aspect was doubling or completely changing every year, is a context where holding onto ideas formed in an old context was viable or possible. You would, necessarily, have to temporarily suspend your trust in the institutional knowledge, with every new problem, since the whole compute world that the institutional knowledge was built on would have shifted under you.
I think it's important to forget, to some extent, every once in a while. It forces a new traversal of the problem space, but in a modern context, with modern tools, fresh eyes, and a better understanding of what's needed. Thus we have Space X.
There are some great interviews with Jim Keller, who has a similar perspective: you need to restart every some years, to not stagnate.
From what I've seen, if you want to stamp out a young engineers creativity, start them working in a big org.