I've never used Claude or anything like it so this may be a dumb question: could you solve this problem by having a CLAUDE.md file that simply says to use AGENT.md if one is available. Can an AI agent not do that?
Yes, the most common solution for this problem either creating a symbolic CLAUDE.md link pointing to AGENT.md (or visa versa) if OS supports it.
Or, in CLAUDE.md have an instruction to follow AGENT.md - but this approach is quite unreliable.
These are solutions to a problem that shouldn’t exist in the first place. How else can one explain Anthropic’s reluctance to adhere to a widely adopted standard, if not as an attempt to build a walled garden around an otherwise great product?
It's not a dumb question per se but it does fail to understand the issue. It's that there's 20 coding agents yet only 1 of them needs this solving. Imagine if all of them needed this. It's like IE6, or Lightning connectors. At least for that last one there's an argument that they performed better than USB-C. For the Anthropic people reading this - take note that both IE and Lightning are now dead and their competitors that followed the standards are thriving.
I understand the issue is that Anthropic is not adhering to a standard. I was simply asking whether it's possible to solve the problem created by Anthropic in the way I was asking.
The better way to solve it is a symlink. The way you're suggesting works too, but should be done using an @ reference, which is auto-followed by Claude. This is the most common way on Windows.
>LLMs are really quite bad at designing anything interesting
Let’s be honest, how many devs are actually creating something interesting/unique at their work?
Most of the time, our job is just picking the right combination of well-known patterns to make the best possible trade-offs while fulfilling the requirements.
> Most of the time, our job is just picking the right combination of well-known patterns to make the best possible trade-offs while fulfilling the requirements.
Right. I don't trust LLM's to pick the right pattern. It will pick _a_ pattern and it will mostly sorta fulfill the requirements.
Today I asked an LLM (Codex whatever-the-default-is) to implement something straightforward, and it cheerfully implemented it twice, right next to each other, in the same file, and then wrote the actual code that used it and open-coded a stupendously crappy implementation of the same thing right there. The amazing thing is that the whole mess kind of worked.
Right. Kind of works is their MO at the moment. I do try to keep in mind that just because something sucks at the moment, that doesn't mean that it will always suck (especially when you pour _trillions_ of dollars into it)
(Author here) I found that over time I spend more time striping someone's badly designed abstractions to get to the real functionality. LLMs are surprisingly good at figuring it out, plowing through the code and documentation and finding out that a 100MB library is in reality a HTTP client for 7 REST endpoints, or something like this.
Many people are in position where they can’t afford risking their financial future by going all-in on startup. They just want to do honest work in exchange on paycheck and enjoy time with family after 5pm and on weekends.
Fun, until you want to share an image and have to upload it to a 3rd party, have to explain what a bouncer is to someone who just wants offline messages and AFAIK voice and video chat is not possible.
Client dependent, but channel overview per server is also not that good.
Neither one strictly requires a login AFAIK, actually. You can chat with an unregistered nick on IRC, and Mumble has certs, but I think they're optional and don't have associated passwords.
The difference is that what most languages compile to is much much more stable than what is produced by
running a spec through an LLM.
A language or a library might change the implementation of a sorting algorithm once in a few years. An LLM is likely to do it every time you regenerate the code.
It’s not just a matter of non-determinism either, but about how chaotic LLMs are. Compilers can produce different machine code with slightly different inputs, but it’s nothing compared to how wildly different LLM output is with very small differences in input. Adding a single word to your spec file can cause the final code to be far more unrecognizably different than adding a new line to a C file.
If you are only checking in the spec which is the logical conclusion of “this is the new high level language”, everyone you regenerate your code all of the thousands upon thousands of unspecified implementation details will change.
Oops I didn’t think I needed to specify what going to happen when a user tries to do C before A but after B. Yesterday it didn’t seem
to do anything but today it resets their account balance to $0. But after the deployment 5 minutes ago
it seems to be fixed.
Sometimes users dragging a box across the screen will see the box disappear behind other boxes. I can’t reproduce it though.
I changed one word in my spec and now there’s an extra 500k LOC to implement a hidden asteroids game on the home page that uses 100% of every visitor’s CPU.
This kind of stuff happens now, but the scale with which it will happen if you actually use LLMs as a high level language is unimaginable. The chaos of all the little unspecified implementation details constantly shifting is just insane to contemplate as user or a maintainer.
Deterministic compilation, aka reproducible builds, has been a basic software engineering concept and goal for 40+ years. Perhaps you could provide some examples of compilers that produce non-deterministic output along with your bad news.
He is a software engineer with a comp.sci masters degree with about 15 years industry experience with primarily C++. Currently employed at a company that you most likely know the name of.
Compilers aim to be fully deterministic. The biggest source of nondeterminism when building software isn't the compiler itself, but build systems invoking the compiler nondeterministically (because iterating the files in a directory isn't necessarily deterministic across different machines).
If you are referring to timestamps, buildids, comptime environments, hardwired heuristics for optimization, or even bugs in compilers -- those are not the same kind of non-determinism as in LLMs. The former ones can be mitigated by long-standing practices of reproducible builds, while the latter is intrinsic to LLMs if they are meant to be more useful than a voice recorder.
Only mostly, and only relatively recently. The first compiler is generally attributed to Grace Hopper in 1952. 2013 is when Debian kicked off their program to do bit-for-bit reproducible builds. Thirteen years later, Nixos can maybe produce bit-for-bit identical builds if you treat her really well. We don't look into the details because it just works and we trust it to work, but because computers are all distributed systems these days, getting a bit-for-bit identical build out of the compiler is actually freaking hard. We just trust them to work well enough (and they do), but they've had three fourths of a century to get there.
Two runs of the same programme can produce different machine code from the JIT compiler, unless everything in the universe that happened in first execution run, gets replicated during the second execution.
That’s 100% correct, but importantly JIT compilers are built with the goal of outputting semantically equivalent instructions.
And the vast, vast majority of the time, adding a new line to the source code will not result in an unrecognizably different output.
With an LLM changing one word can and frequently does cause the out to be so 100% different. Literally no lines are the same in a diff. That’s such a vastly different scope of problem that comparing them is pointless.
No, but will certainly result in a complete different sequence of machine code instructions, or not, depending on what that line actually does, what dynamic types it uses, how often it actually gets executed, the existence of vector units, and so forth.
Likewise, as long as the agent delivers the same outcome, e.g. an email is sent with a specific subject and body, the observed behaviour remains.
The reason this works for compilers is because machine code is so low level that it’s possible for compiler authors to easily prove semantic equivalence between different sets of instructions.
That is not true for an English language prompt like “send and email with this specific subject and body”. There are so many implicit decisions that have to be made in that statement that will be different every time you regenerate the code.
English language specs will always have this ambiguity.
Do these compilers sometimes give correct instructions and sometimes incorrect instructions for the same higher level code, and it's considered an intrinsic part of the compiler that you just have to deal with? Because otherwise this argument is bunk.
Also the exact sequence of generated machine instructions depends of various factors, the same source can have various outputs, depending on code execution, preset hardware, and heuristics.
Sure, but surely you expect a `func add(a, b) { return a + b; }` to actually produce a + b, in whatever way it finds best? And if it doesn't, you can reproduce the error, and file a bug? And then someone can fix that bug?
Ok, but we treat them as bugs that we can reproduce and assume that they are solvable? We don't just assume that it's intrinsic to the compiler that it must have bugs, and that they will occur in random, non-deterministic ways?
And 10 orders is optimistic value - LLMs are random with some probability of solving the real problem (and I think of real systems, not a PoC landing page or 2-3 models CRUD) now. Every month they are now getting visibly better of course.
The „old” world may output different assembly or bytecode everytime, but running it will result in same outputs - maybe slower, maybe faster. LLMs now for same prompt can generate working or non-working or - faking solution.
That is a completely different category. I've never experienced a logic error due to a managed runtime and only once or twice ever due to a C++ compiler.
I certainly already experienced crashes due to JIT miscompilations, even though it was a while back, on Websphere with IBM Java implementation.
Also it is almost impossible to guarantee two runs of an application will trigger the same machine code output, unless the JIT is either very dumb on its heuristics and PGO analysis, or one got lucky enough to reproduce the same computation environment.
> Also it is almost impossible to guarantee two runs of an application will trigger the same machine code output
As long as the JIT is working properly, it shouldn't matter: the code should always run "as if" it was being run on an interpreter. That is, the JIT is nothing more than a speed optimization; even if you disable the JIT, the result should still be the same.
I think it's technically possible to achieve determinism with LLM output. The LLM makers typically make them non-deterministic by default but it's not inherent to them.
A compiler that can turn cash into improved code without round tripping a human is very cool though. As those steps can get longer and succeed more often in more difficult circumstances, what it means to be a software engineer changes a lot.
LLMs may occasionally turn bad code into better code but letting them loose on “good” or even “good enough” code is not always likely to make it “better”.
The solution for brain atrophy I personally arrived is to use coding agents at work, where, let’s be honest, velocity is a top priority and code purity doesn’t matter that much. Since we use stack I super familial with, I can quite fast verify produced code and tweak it if needed.
However, for hobby projects where I purposely use tech I’m not very familiar with, I force myself not to use LLMs at all - even as a chat. Thus, operating The old way - writing code manually, reading documentation, etc brings me a joy of learning back and, hopefully, establishes new neurone connections.
PC-s were only described in hobby magazines, like Bajtek or Młody Technik. Nobody had them, though, except maybe some institutions. The hobbyists used to own ZX Spectrum or Commondore 64, but even that was rare.
I know one programmer in his 50s. He had an access to the ZX Spectrum in his primary school, but that was by effort of his local physics teacher.
I'm not (yet) in my 50s (though close). I used to have a C64 back in the day. I wrote write a few things in its horrible BASIC dialect. Probably the most advanced was a database (not relational, just one table, but kept separately from the source, of course on an audio cassette).
That device had ridiculous capabilities. The sound chip was good enough people wrote a speech synthesis software. Later, people wrote a graphical OS, with e.g. a text editor being an equivalent of Windows Write from the 90s.
Could it be that the handful of people with computer access were well connected & well regarded, & the people running the radio broadcasts wanted to cater to them especially? I'd imagine there could be some sense of personal & national pride & prestige around supporting these emerging technologies & promoting them to the public. (I'm just guessing though - I wasn't there & haven't studied the topic in depth.)
My guess would be that the broadcaster had one geek who pushed for that. Fellow geeks had software over the radio, the broadcaster had an opinion of a modern one, keeping up with the newest tech. Win-win.
reply