The capability you are actually screening
An agentic AI engineer builds systems in which a model can select or invoke actions. That can include customer support workflows, internal operations, document processing, and tools that write to business systems. The role demands conventional software engineering and a careful approach to permissions, workflow state, retries and evaluation.
A candidate who can demo a multi-agent framework may still struggle with duplicate payments, partial tool failures or an unapproved data mutation. Interview questions should distinguish demos from engineering maturity.
1. Architecture and state
Question: How would you design an agent that updates CRM records from emails?
Look for event handling, durable workflow state, structured tool APIs, validation, record identity, duplicate detection, audit history and authorization checks before writes. A strong answer distinguishes model decisions from deterministic business rules.
Question: Where should conversation state and execution state live?
Conversation context is not necessarily a durable record of side effects. The candidate should discuss persistence, resuming interrupted workflows, storing tool results, state versions, expiration, and recovery when a step is retried.
2. Tools, security and permissions
Question: A tool can issue refunds. How would you keep it safe?
Look for narrowly scoped credentials, dollar limits, customer ownership checks, approval thresholds, policy enforcement outside the model, idempotency, and full traceability of requested and completed actions. “The prompt tells the model not to” is not sufficient security.
Question: How can an agent defend against instructions embedded in retrieved data?
Strong answers treat documents and tool responses as untrusted input; separate system policy from data; validate tool parameters; and restrict permissions independently from model output. Candidates should describe adversarial tests and operational controls, not assume perfect prompt defenses.
3. Reliability, retries and observability
Question: A tool times out after processing a request. What happens on retry?
The candidate should identify the ambiguous completion problem. They might propose idempotency keys, status lookup, transaction boundaries, compensating actions and correlation IDs. Blindly retrying a non-idempotent write can be worse than failing.
Question: How do you debug an agent that suddenly gets slower?
Expect trace-based diagnosis across model calls, retrieval, tool latency, queueing, retries and workflow steps. Cost and latency budgets belong alongside success measures.
4. Evaluation and release decisions
Question: What metrics tell you an agent is safe to release?
Task completion alone is insufficient. Include unsafe tool-call rate, approval bypasses, error recovery, false-positive actions, time-to-completion, human escalation, per-task cost and performance on adversarial or edge-case tasks. Baseline against a simpler deterministic workflow.
Question: How would you build a regression test suite?
Use recorded task scenarios, representative tool stubs, fixture versions, expected state changes and a trace of each action. Include failure injection and access-denied cases; compare new agent revisions with prior baselines.
5. A practical 75-minute assessment
Present a fictional returns agent. It can read an order, check refund eligibility and request a refund. The order service sometimes times out and the refund tool can have delayed completion. Ask the candidate to draw the workflow, design the tool contracts and choose which decisions remain deterministic.
Have them explain data access, allowed actions, idempotency keys, the human approval branch, failure handling and a minimal test dataset. Scoring should focus on reasoning quality, practical trade-offs and operational safeguards—not a completed codebase in an unrealistic time window.
| Dimension | Strong evidence | Warning sign |
|---|---|---|
| Workflow design | Durable state and explicit transition rules | Assumes a chat transcript is execution state |
| Security | Enforced policy and least-privilege tools | Relies solely on prompt instructions |
| Reliability | Retries, idempotency, recovery and audit trail | Blind retries of side-effecting calls |
| Evaluation | Task and safety metrics with repeatable tests | Judges quality solely by a demo |
| Operations | Traces, latency/cost budgets and escalation | No plan for debugging production behavior |
6. Scoring the interview
Score each dimension from 1 to 5: 1 means the candidate cannot explain fundamentals; 3 means reasonable designs with gaps under edge cases; 5 means defensible trade-offs with realistic production controls. Weight security and reliable side effects more heavily for agents allowed to write to systems of record.
For candidates whose role is primarily building AI application interfaces, use the broader AI engineer interview scorecard. For teams choosing a staffing partner, see what Agentic AI Engineers should own.