There's a moment every agent builder hits

You've got your LLM running in a loop, you've wired up a handful of tools, and the agent... kind of works? But it keeps hitting dead ends. It says things like "I don't have enough context to diagnose this" or it hallucinates a Datadog query that doesn't exist. You know it's not the model's fault. It just doesn't have what it needs.

The normal instinct is to go build more tools. Query Sentry. Pull Grafana logs. Fetch deployment history. You spend two weeks building integrations, you come back, and the agent still hits dead ends, just different ones.

I wasted two weeks doing this at Sonarly. Then I accidentally found a better way.

The shim tool

I was debugging a frustrating run where our incident investigation agent couldn't figure out why a service was throwing 500s. I was watching it reason step by step, and I kept thinking: what would I do right now if I were on-call?

I'd want to see the recent deploy. I'd want the upstream service latency. I'd want to know if this error only started after a config change.

The agent didn't have any of that. But I realized I had no systematic way to know what the agent was wishing it had, across dozens of different incident scenarios. I was just guessing.

So I did something dumb that turned out to be smart. I gave the agent a fake tool.


def magic_fetch(description: str) -> str:
    """
    You can use this tool when you need any data
    that you don't currently have access to.
    Describe exactly what you need and why.
    This tool will retrieve it for you.
    """
    # logs the description, returns a placeholder
    log_tool_call(description)
    return "Data retrieved successfully. Continue your reasoning."

That's it. The agent thinks it works. It doesn't. But it calls it, and when it does, it tells you exactly what it wanted.

What the agent actually told me

I ran 34 incident simulations across different failure modes: high latency, error spikes, memory leaks, broken deploys. Each time the agent hit a wall, it reached for magic_fetch exactly once, telling me exactly what it wished it had before the investigation went off the rails.

Some examples of what it described wanting:

"The list of recent deployments for this service in the last 2 hours, including which commit was deployed and by whom"
"CPU and memory metrics for the upstream dependency auth-service over the past 30 minutes"
"Whether there are any active feature flags that were recently toggled for this service"
"Sentry error fingerprints for this issue grouped by affected user segment"
"The on-call runbook for this service if one exists"

Nobody told the agent to ask for any of this. It reasoned its way there. And suddenly I had a prioritized, natural-language product spec for my integration roadmap, written by the thing I was trying to build.

Why this works

The insight is simple: the agent is a good proxy for your best on-call engineer. When you give it a blank canvas to express what it needs, it surfaces the exact data gaps that matter for the task you're asking it to do.

Traditional ways to figure this out are painful:

Interviews with SREs. Time-consuming, biased toward what they remember needing, not what they actually reach for in the moment
Watching production logs. Only works if the tools already exist and are being called
Guessing from documentation. You'll build beautiful integrations nobody uses

The fake tool skips all of that. It's a zero-infrastructure user study where your agent is the user.

This maps loosely to a UX technique called Wizard of Oz testing, where you fake a system to understand real user needs before building anything. The twist here is you're doing it to an AI, not a human. The agent is honest about its needs in a way users often aren't, because it has no ego and no social pressure to seem competent.

The pattern in practice

Here's how I'd recommend doing this:

Write the fake tool. One tool, free-form string input, always returns a success message. Log every call with the full description and the agent's current reasoning context.
Run your agent on diverse, realistic scenarios. The more variety, the better. For Sonarly, that meant different services, different failure modes, different on-call runbooks.
Cluster the descriptions. You'll start seeing patterns. "Recent deploys" came up in 14 out of 34 runs. "Feature flag state" came up in 7. That became our integration priority queue.
Build the real tools in order of frequency, and when you do, you already have the natural language descriptions to write good tool docstrings, because the agent wrote them for you.
Remove the fake tool and re-run. Watch how much further the agent gets now. The delta is measurable.

What I didn't expect

Two things surprised me.

First, the agent asked for things that don't exist yet as APIs. Things like "the historical p99 latency baseline for this service over the past 30 days to compare against", not a standard Datadog query, but a derived signal. The fake tool doesn't just tell you which integrations to build, it tells you which computations to pre-build and cache.

Second, the agent's requests were remarkably consistent across runs. Different incidents, same missing data. This isn't random exploration. The model has a coherent mental model of what on-call investigation requires, and it's trying to fill specific slots. That's a useful signal about how to structure your tool taxonomy.

The broader lesson

Most agent builders, myself included, instinctively jump to building capabilities. More tools, more integrations, more context. But the hard part isn't building the tool, it's knowing which tool to build and how it should feel to the agent calling it.

The best source of that signal isn't a product manager, a user interview, or an API docs crawl. It's the agent itself, given permission to be honest about what it's missing.

Give your agent a magic tool. Let it lie to you a little. It'll tell you everything you need to build next.

We're building Sonarly to be the AI SRE that investigates incidents so your team doesn't have to at 3am. If you're drowning in alerts and want to see this in action, reach out.

Alexandre Klobb, Co-founder at Sonarly