Short answer: hire for the boring parts. How they design tools, scope data access, test with evals, handle failure, and hand off ownership. Anyone can make an agent demo well for ten minutes. The developer you want can explain what happens on the worst day, and has logs to prove it.
This is the agent-specific layer on top of our general guide, How to Choose an AI Automation Partner. Read that for the questions that apply to any vendor. Read this for the questions that only matter when the thing being built makes decisions on its own.
Why agent work is different from regular development
A normal application does what the code says. An agent takes a goal, picks tools, reads data, and decides what to do next. That changes what "working" means. It can be right on every test case you thought of and wrong on the first real one. So the skills that matter shift from "can they build the feature" to "can they bound what the agent is allowed to do, measure whether it does it, and catch it when it drifts."
Most developers have never had to think that way. The good agent developers have, and they will bring it up before you ask.
Eight questions to ask, and what a good answer sounds like
1. "Show me your evals." A production agent has a test set, real examples of inputs and expected outcomes, and a way to run them after every change. If the answer is a demo instead of a test set, keep looking.
2. "How do you decide what tools the agent gets?" Good answer: small, specific, read-only tools first, each one scoped to what the user is allowed to see, with write tools added later behind approval. Bad answer: one big tool that can do anything, controlled by the prompt.
3. "Where does the data scoping live?" Good answer: in the database or API layer, per organization and per role, so a prompt injection or a model mistake cannot cross the line. Bad answer: "the system prompt tells it not to."
4. "What happens when the model is wrong?" You want to hear about confidence thresholds, routing low-confidence cases to a person, and logging every decision so you can review it. You do not want to hear "it is very accurate."
5. "What happens when the model provider is down or changes?" Good answer: retries, fallbacks, a clear failure message, and model routing so you are not locked to one vendor. Ask whether they have swapped models on a live system before.
6. "Who gets paged?" Agents run unattended. Someone has to own the exception queue. A good developer will define that role with you and build the queue.
7. "How will we know if it is getting worse?" Drift is real. Good answer: a small weekly eval run, a dashboard of override and failure rates, and a habit of promoting frequent corrections into hard rules.
8. "What do we own when you leave?" The code, the prompts, the eval set, the accounts, and the documentation. If any of those stay with the developer, you do not have an agent, you have a dependency.
Red flags
- Every example is a chat interface. Real value is usually a background agent doing a job, not a chatbot.
- They talk about the model more than the data. The model is the easy part.
- No mention of logging or approval until you raise it.
- They promise accuracy percentages before seeing your data.
- Write access to your systems in the first version.
Green flags
- They ask what your data looks like before they say what they would build.
- They describe the worst-case day without being prompted.
- They start with a narrow first agent and a way to measure it.
- They talk about handoff and ownership as part of the plan, not an afterthought.
- Their demo has a "why did it do that" view, not just an output.
How the engagement should be structured
A sound agent engagement looks like any sound software project, with evals added. A short discovery pass to map the data and the decision the agent will make. A focused first agent, scoped narrow, read-first, with an eval set and an exception queue. Then expansion once the team trusts it. Our published planning range for a focused first project is four to eight weeks, on the timeline answers page, and the pricing ranges are on the cost answers page.
If the proposal skips discovery and goes straight to "we will build your AI agent," the price is a guess and the scope is a hope.
FAQ
Do we need a specialist, or can our existing developers build an agent?
Your developers can build one if they adopt the disciplines above: evals, scoped tools, approvals, logging. The skill gap is usually judgment about what the agent should be allowed to do, not the code.
What is the single most important thing to ask?
"Show me your evals." It separates people who have run agents in production from people who have demoed them.
Should the first agent be a chatbot?
Usually not. The highest-value first agents do a job in the background: triage, drafting, lookups, follow-ups, with a person approving anything that changes a record.
How do we avoid vendor lock-in?
Own the code and prompts, keep the tool layer on a standard like MCP so it works with any model, and make sure the eval set lives in your repo.
What does it cost?
It depends on how many systems the agent touches and how much approval and logging it needs. The honest path to a number is discovery, then a written scope. Our planning ranges are published on the pricing answers page.
Ideas Realized builds production agents with evals, scoped tools, and human approval gates, and hands over everything at the end. If you are evaluating developers, we are happy to be one of the interviews.