https://github.com/abshkbh/arrakis
Also building in this space using MicroVMs. Currently working on a Mac port.
Would love to connect - abshkbh AT gmail.com
Crosvm (our original Google project) and its children projects Firecracker, Cloud-Hypervisor are all based on top of "/dev/kvm" i.e. the Linux Virtualization stack.
Apple's equivalent is the Apple Virtualization Framework which exposes kvm like functionality at a higher level.
Morden software consists of separation of responsibilities between services and a higher plane orchestrating the data flow for business logic.
If you believe there is value in fuzzy tasks being done by LLMs then from that it follows that having separate "agent" services with a higher order orchestrator would be required. Each calling LLMs on their own inside.
I don't buy it. Why would I want my LLM to talk to some other LLM and introduce even more space for weird, non-deterministic bugs when I could have my LLM call a deterministic API to achieve the same thing?
You have anything else that modifies the context, tools, model, and most importantly perhaps the iteration that controls what's going on with those other values.
But so much more besides that, including the model itself, RAG, the agentic workflow control, moderation, etc. There's also a huge factor of maintenance, that's a key reason why companies have different internal and external APIs - they don't just open up everything internal and hand the code for managing all of that to you. Interface design is really important.
Not to mention the cost being a factor here - who pays for which part.
Offering up an LLM-fronted "agent" for people to send their LLMs to talk to feels a whole lot more expensive and complicated to me than operating a traditional API endpoint.
reply