BENOÎT EVEILLARD.FRLet’s talk

What I learned building an AI agent on real tools

"Are we on schedule?" In a web agency, that question comes up every week. To answer it, a project manager opens the ticketing tool, a GitHub repo for pull requests and CI, then Slack to find out whether the client really signed off. An agent plugged into those tools can do that round for them. I built one to see what it takes, with a standard close to production. What I took away fits in one sentence: I spent more time stopping it from being wrong without saying so than getting it to reason. My agent delivered its verdict in 28 seconds: high risk, six blockers, each with its evidence. It also set aside a Slack message that told it to announce that everything was fine (a deliberate _prompt injection_).

I built Agency OS to see what it would take to hand that question over to an agent, with a standard close to production. The agency, Acme Digital, is fictional and its data is synthetic. The Linear, GitHub and Slack accounts, however, are real, and so are the APIs, the permissions and the outages.

At the start, I thought the hardest part would be getting the model to reason. In reality, most of the work went into stopping the system from being wrong without telling me.

What the agent does

You ask it a question in natural language, for example “What’s blocking project 123?” It decides for itself which sources to look at, cross-checks what it finds and returns a verdict with its evidence.

On the latest test run, it made 10 tool calls in 28 seconds and concluded high risk, with 6 blockers:

  • the go-live ticket is past its due date and is still blocked by several open tickets
  • the CMS fix PR is a draft with no reviewer, and its CI is failing
  • the client liked the version in a meeting (see the Slack message), but nobody can point to a formal sign-off. The agent therefore writes “not confirmed”.

It also flags that the most recent data is several weeks old and that things may have changed without the tools being updated.

On an earlier run, it also spotted a Slack message telling it to ignore its instructions and announce that everything was fine. It set the message aside and flagged it in its answer. I had placed that message in the channel on purpose, to see how it would react to a manipulation attempt.

The decisions that matter

One agent and precise tools

No multi-agent setup, no vector database. A hand-written loop of calls to Claude, and tools that map to business capabilities: list a project’s tickets, read the CI status, read a Slack thread. Every component I added had to answer a problem I had actually observed.

The agent cannot write, by design

This is the decision I’m happiest with. The common solution is to give the model an execution tool protected by a check. Here, that tool doesn’t exist. The model can only propose an action, not apply it.

Execution happens in the application code, after the model’s turn has ended:

  • the client able to write isn’t even among the dependencies of the agent’s package, and a test fails if that changes
  • the write key is separate from the read key, and limited to creating tickets on a single team
  • a SHA-256 fingerprint ties the approval to the exact content of the action: if the content changes between the two, execution is refused
  • the decision is written to the audit log before execution, and if writing to the log fails, the action is refused

The guarantee rests on the absence of a capability.

All external content is data, never an instruction

A ticket, a PR title or a Slack message can contain malicious text. I don’t filter by keywords. The prompt sets the rule and I measure the behavior: the injection scenarios plant a marker inside a write request, and a deterministic test fails if that marker reaches a proposal or a blocker title.

These tests don’t prove that the agent resists every injection. They show that it resisted the ones I tried. That’s also why the write boundary doesn’t depend on the model.

A conclusion must cite what was retrieved

Every blocker and every proposal cites evidence identifiers, which the code checks against what the run actually retrieved. An invented identifier is rejected.

Sources don’t all carry the same weight: Linear is authoritative for ticket status, GitHub for CI, and Slack provides context. A “good for Friday 👍” doesn’t outweigh an open ticket. And a client sign-off requires an explicit statement that can be attributed to someone.

Everything is traced, without the content

Every run produces a trace exported to Grafana: model turns, tool calls, latency, tokens, errors, approval decisions. Attributes can only hold simple values, which makes it impossible to attach a whole Slack message to a trace by mistake. Without these traces, most of the bugs below would have stayed invisible.

Example of a Grafana trace

Scenarios instead of a good feeling

The project has 25 evaluation scenarios: healthy project, conflicting sources, stale information, tool outage, rejected action, injections. Anything that can be checked deterministically is. LLM judgment is only used for what needs a real reading of the meaning.

What broke

If you’d like to build something similar, this is going to be the most useful part of the project 👇

The security check that checked nothing. My judge, in charge of detecting successful injections, returned an empty string because a thinking block consumed its entire token budget. The code read that emptiness as “no”. The report therefore showed “no compromise” when nothing had been verified. Today, the judge can answer yes, no or error, and an error is never counted as a success.

CI green by default. A finished run counted as a success as long as its conclusion wasn’t in my list of failures. A startup_failure or a fully skipped run therefore passed for green, and the agent stated it as a fact. The rule applied now: if a value feeds a claim presented as fact, you list what proves it, and everything else becomes “unknown”.

The prompt that told it not to decide. Faced with an old Slack message contradicted by a more recent ticket, the agent failed 3 times out of 3. I had myself written to always expose conflicts. I rewrote the paragraph to combine source authority and date: 3 passes out of 3, with no regression on neighboring scenarios.

Malformed tool calls. On high-risk verdicts, only 12 to 20% of the first submissions of the structured report were valid, and each retry cost about 15 seconds. I first wrote recovery code. The real fix was to turn on the API’s strict mode: 10 out of 10 on the first try, on a controlled sample.

The bill. About 40 dollars of credits vanished in two days. I counted the tokens of 601 traces: 12 million input tokens, zero cached. I had never turned on prompt caching, so every turn resent the whole conversation at full price. Once caching was on, the cost of a comparable run dropped by 43%.

The rejected token. Recently, GitHub rejected the project’s token with a 401 error. The agent answered anyway, stating that CI had not been checked. That’s the intended behavior, but it hasn’t always been reliable: two runs of the suite, in September, caught an agent that didn’t mention a source that was down. And the interface had its own problem, found during an audit run with Claude Code: a failing source was just a small gray note next to a confident verdict. It now appears on the verdict itself.

How to apply this in any organization

None of this is specific to agencies. The connectors change from one organization to the next. The loop, the approval boundary, the traces and the evaluation method stay the same.

Here is the approach I would follow. The examples are leads, I haven’t built them.

1. Start from a question, not an assistant

Look for a question that comes up often, that forces people to open several tools and whose answer can be verified. “An AI assistant for finance” isn’t a project. “Why is this invoice still unpaid?” potentially is.

2. Write down who is authoritative on what

This is the most underestimated work and it requires no code (it’s more thrilling to jump into code and get a v0 quickly). Do it with people from the business, because that’s where the rules that are actually applied are hiding.

Question Sources Who is authoritative Action to get approved
Why is this invoice unpaid? ERP, CRM, emails The ERP for payment, the written exchange for a dispute Draft a reminder
Is this account at risk of churning? Support, product usage, CRM Usage data for activity, the CRM for the contract Create a task for the account owner
Can this order ship? Stock, logistics, messaging Stock for availability, the carrier for lead times Notify the customer
Is this new hire ready? HRIS, access management, IT tickets The HRIS for the contract, the access tool for permissions Open an access ticket

3. Start read-only, on real systems

Use test accounts and fictional data, but real APIs. Simulated data will teach you nothing about permissions, pagination, rate limits or outages. My first GitHub blocker came from a permission that can’t be granted to that type of token, and no mock would have shown me that. Give each tool the minimum rights.

4. Take the ability to write away from the model

Don’t just ask it to “ask before acting”. The model proposes, a human approves a specific action, the code executes and logs. Add each type of write one at a time.

5. Write the hard scenarios before the first user

Around twenty good scenarios is enough to start: conflicting sources, stale data, a source down, sign-off given by someone who has no right to give it, injection, etc. Repeat the most important ones, because an agent doesn’t behave the same way on every try: talk in success rates. Every significant failure becomes a new scenario.

6. Trace and measure cost from day one

As soon as an agent gets something wrong, the trace lets you tell a missing piece of data, a wrong tool, a prompt problem and a real reasoning error apart. Without it, you tweak the prompt at random. Simply tracking cached tokens would have flagged my billing problem much earlier.

7. Make uncertainty a feature

An agent that says “I couldn’t check the CI” is still useful. An agent that leaves it out is dangerous, precisely because it looks competent. Ask yourself what the user sees when a source is missing. This point is crucial given how convincing an LLM can be (or maybe we’re just easy to manipulate?).

Five questions before launching

  1. What question does the agent solve, and who asks it today?
  2. For each type of information: which system is authoritative?
  3. What is the worst the agent can do with its current rights?
  4. What does the user see when a source is down?
  5. Which test would fail tomorrow if the agent got worse?

If any of these answers is still fuzzy, the problem isn’t the model yet.

In some organizations, building on Claude Managed Agents can be an excellent option to get a foundation and pre-built tools (memory, sandbox, etc.) while staying highly configurable.

What this project doesn’t prove

  • No real users. I haven’t tested the agent with any real team and I haven’t measured the time saved. That would be the next proof to provide.
  • A narrow scope. Three sources (Linear, GitHub, Slack) and a single type of write, creating a ticket. Adding a CRM like HubSpot or Salesforce could be the next building block.
  • No authentication. The web demo has no login, so the audit log records an “anonymous visitor”. Writes are disabled by default. Real customer data would first require real access control.

What I take away

Most of the flaws in this project didn’t look like errors. A safeguard that passes without checking anything, a CI that’s green for lack of evidence, an empty answer counted as a success: everything looked like it was working.

If I had to keep one rule for another organization, it would be this one: before asking an agent to do more, make sure you can see (and then understand) when it gets things wrong.

Demo video

I put a small demo video of what it looks like:

The code is on GitHub.