Relatedly, I haven’t seen two issues discussed much in agent design, but repeatedly come up in real world use cases:
(1) LLM forgets to call a tool (and instead outputs plain text). Contrary to some of the comments here saying that these concerns will disappear as frontier models improve, there will always be a need for having your agent scaffolding work well with weaker LLMs (cost, privacy, etc).
(2) Determining when a task is finished. In some cases, we want the LLM to decide that (e.g search with different queries until desired info found), but in others, we want to specify deterministic task completion conditions (e.g., end the task immediately after structured info extraction, or after acting on such info, or after the LLM sees the result of that action etc).
After repeatedly running into these types of issues in production agent systems, we’ve added mechanisms for these in the Langroid[1] agent framework (I’m the lead dev), which has blackboard-like loop architecture that makes it easy to incorporate these.
For issue (1) we can configure an agent with a `handle_llm_no_tool` [2] set to a “nudge” that is sent back to the LLM when a non-tool response is detected (it could also be set as a lambda function to take other possible actions)
For issue (2) Langroid has a DSL[3] for specifying task termination conditions. It lets you specify patterns that trigger task termination, e.g.
- "T" to terminate immediately after a tool-call,
- "T[X]" to terminate after calling the specific tool X,
- "T,A" to terminate after a tool call, and agent handling (i.e. tool exec)
- "T,A,L" to terminate after tool call, agent handling, and LLM response to that
EDIT- forgot to mention the other part of issue (2): In cases where we want the LLM to decide task completion, in Langroid there's a DoneTool the LLM can use to signal completion. In general we find it useful to have orchestration tools for unambiguous control flow and message flow decisions by the LLM:
(1) LLM forgets to call a tool (and instead outputs plain text). Contrary to some of the comments here saying that these concerns will disappear as frontier models improve, there will always be a need for having your agent scaffolding work well with weaker LLMs (cost, privacy, etc).
(2) Determining when a task is finished. In some cases, we want the LLM to decide that (e.g search with different queries until desired info found), but in others, we want to specify deterministic task completion conditions (e.g., end the task immediately after structured info extraction, or after acting on such info, or after the LLM sees the result of that action etc).
After repeatedly running into these types of issues in production agent systems, we’ve added mechanisms for these in the Langroid[1] agent framework (I’m the lead dev), which has blackboard-like loop architecture that makes it easy to incorporate these.
For issue (1) we can configure an agent with a `handle_llm_no_tool` [2] set to a “nudge” that is sent back to the LLM when a non-tool response is detected (it could also be set as a lambda function to take other possible actions)
For issue (2) Langroid has a DSL[3] for specifying task termination conditions. It lets you specify patterns that trigger task termination, e.g.
- "T" to terminate immediately after a tool-call,
- "T[X]" to terminate after calling the specific tool X,
- "T,A" to terminate after a tool call, and agent handling (i.e. tool exec)
- "T,A,L" to terminate after tool call, agent handling, and LLM response to that
[1] Langroid https://github.com/langroid/langroid
[2] Handling non-tool LLM responses https://langroid.github.io/langroid/notes/handle-llm-no-tool...
[3] Task Termination in Langroid https://langroid.github.io/langroid/notes/task-termination/