How do you test an n8n AI agent before putting it in production?

Use n8n's built-in evaluations. You add an Evaluation Trigger node that reads a dataset of test cases from an n8n Data Table or Google Sheet, run each case through the exact same agent your production trigger uses, then score the output with an Evaluation node set to Set Metrics. Built-in metrics include Correctness, Helpfulness, String Similarity, Categorization, and Tools Used. Run the suite from the Evaluations tab in the editor, set a minimum average score you're willing to ship at, and treat anything below that as a blocked deploy. Seed the dataset from real inputs in your execution history rather than writing test cases from scratch.

What metrics does the n8n Evaluation node support?

Five built-in metrics. Correctness is AI-scored on a 1–5 scale and checks whether the answer's meaning matches a reference answer. Helpfulness is also 1–5 and checks whether the response actually answers the query. String Similarity compares text using Levenshtein distance or semantic matching. Categorization compares a predicted label against an expected label, which is what you want for routing and classification agents. Tools Used checks whether the agent called the expected tools in the expected order — the single most useful metric for multi-tool agents, because a right answer reached the wrong way will break later.

What's a good passing score for an n8n agent evaluation?

It depends on what the agent does, and you should set it per agent rather than using one number everywhere. For internal-facing agents like summarizers and enrichment, we ship at an average Correctness of 4.0 or higher on a 1–5 scale. For anything that talks to a customer or writes to a CRM, we require 4.5 and zero cases scoring below 3. The absolute number matters less than the fact that it's fixed before you look at the results — pick the threshold first, then run the suite, or you'll rationalize a bad score into a good one.

How many test cases do you need in an n8n evaluation dataset?

Start with 20 to 30 real cases, not 200 synthetic ones. Pull them straight out of your execution history so they carry the actual messiness of production input — typos, missing fields, people answering a different question than you asked. Make sure roughly a third are edge cases: empty inputs, hostile or off-topic messages, and the ambiguous requests that sit between two of your agent's branches. A 25-case dataset built from real traffic will catch more failures than a 200-case dataset you invented at your desk.

Do you need to re-run evaluations after changing the model or prompt?

Yes, and that's the main reason to build the suite at all. Any prompt edit, model version change, or new tool can quietly shift behavior in ways that look fine on the one input you happened to test. Re-running a fixed dataset turns that into a measurable before-and-after number. We schedule the suite to run nightly on client agents and alert in Slack when the average drops below the threshold, which catches provider-side model drift you would otherwise find out about from a client.

How to Test n8n AI Agents Before Production

Almost every broken AI agent we get called in to fix was tested the same way: someone typed three questions into the chat panel, got three good answers, and turned it on.

The way to actually test an n8n AI agent is with n8n's built-in evaluations — a fixed dataset of real inputs, run through the same agent path production uses, scored on defined metrics, against a pass threshold you set before you look at the results. It's the difference between "it worked when I tried it" and knowing what percentage of real inputs it handles correctly. Here's the setup we run on client agents before they touch a live lead.

Why Manual Testing Fails Specifically for Agents

Traditional automation is deterministic. A Zapier step either maps the field or it doesn't, and if it works once it works a thousand times.

Agents don't behave like that. The same input can produce different output on two runs. A prompt edit that fixes one case silently breaks four others you weren't looking at. And a model version bump on the provider's side can change your agent's behavior on a Tuesday with no deploy on your end.

So the three questions you typed into the chat panel prove almost nothing. They prove the happy path works, on one model version, on one day. Meanwhile the failures that actually cost money — the agent inventing a price, calling the wrong tool, routing an angry customer into the upsell branch — live in the inputs you'd never think to type.

We learned this the expensive way on a lead-qualification agent that tested beautifully and then, in its first week, confidently told inbound leads about a service tier that didn't exist. One line in the prompt. Zero errors in the execution log, because the workflow ran perfectly. It just said the wrong thing, and nothing in n8n was watching for that.

That's the gap evaluations close. Error handling and monitoring catches workflows that break. Evaluations catch workflows that run fine and produce garbage.

What n8n's Evaluation Feature Actually Gives You

n8n ships evaluations natively — you don't need an external eval platform for most agency work. Three pieces do the job:

| Component | What it does | |-----------|--------------| | Evaluation Trigger | Reads your test dataset from an n8n Data Table or Google Sheet and feeds rows through the workflow one at a time | | Evaluation node — Set Outputs | Writes the agent's actual output back next to each test case so you can eyeball it | | Evaluation node — Set Metrics | Scores the run and records it in the Evaluations tab | | Check If Evaluating | Branches execution so a test run doesn't fire real side effects |

The metrics are the part worth memorizing:

Correctness — AI-scored 1–5, does the answer's meaning match your reference answer
Helpfulness — AI-scored 1–5, does the response actually answer the query
String Similarity — Levenshtein or semantic matching for outputs that should be close to exact
Categorization — predicted label vs. expected label, the right metric for routers and classifiers
Tools Used — did the agent call the expected tools, in the expected order

If you only add one metric, add Tools Used. For any multi-tool agent, the right answer reached the wrong way is a bug waiting to surface, and it's the failure mode no human reviewer catches by reading the output. This matters most in orchestrator setups, where the agent node vs. sub-workflow decision determines how many tool calls you're even able to observe.

Full node reference is in the n8n evaluations docs.

Build the Dataset From Your Execution History, Not Your Imagination

This is where most teams get it wrong, and it's the cheapest thing to fix.

Don't sit down and write test cases. Open your execution history and steal 20 to 30 real inputs. Real input is messier than anything you'd invent — half-finished sentences, wrong fields, people replying to a question you didn't ask, one guy who pasted his entire email signature into the chat box.

Our rule for a client agent dataset:

~60% normal traffic — the requests the agent exists to handle
~25% edge cases — empty fields, ambiguous asks that sit between two branches, multi-part questions
~15% hostile — off-topic, prompt injection attempts, people trying to get a discount out of it

That last bucket isn't paranoia. Anything with a public entry point gets probed, usually within days, which is why we treat prompt injection as a test case category rather than a one-time review.

Twenty-five real cases beat two hundred synthetic ones. We've watched a hand-written 200-row dataset pass at 4.6 while the agent failed on the third real lead it saw, because every invented case was written by the same person who wrote the prompt, in the same voice, with the same assumptions.

Set the Threshold Before You Run the Suite

Pick the number first. This sounds like a small process detail and it's the whole game — if you run the suite and then decide what's acceptable, you will talk yourself into shipping a 3.8.

What we use:

| Agent type | Ship threshold | |------------|----------------| | Internal — summarizers, enrichment, tagging | Avg Correctness ≥ 4.0 | | Customer-facing or CRM-writing | Avg Correctness ≥ 4.5, zero cases below 3 | | Routers and classifiers | Categorization ≥ 95% |

The "zero cases below 3" clause does more work than the average. An agent that averages 4.5 across 25 cases can still contain one case where it fabricated a price to a customer, and the mean hides it completely. Sort by lowest score and read the bottom three by hand, every run. That's ten minutes, and it's where the real bugs are.

Run It on a Schedule, Not Just at Launch

Building the suite once at launch gets you a snapshot. The value compounds when it runs on its own.

Add a Schedule trigger, run the dataset nightly, average the scores in a Code node, and push a Slack alert when the average drops below your threshold. That catches three things you'd otherwise learn about from a client:

Prompt edits with side effects — someone tweaks one instruction, three unrelated cases regress
Model drift — the provider updates the model behind your endpoint and behavior shifts with no deploy on your end
Reality drift — your business changed, your reference answers didn't, and the agent is confidently correct about last quarter's pricing

Number three is the one that gets agencies. The agent isn't broken; the world moved. A scheduled suite surfaces that as a slow score decline weeks before anyone complains.

Across the agent workflows we run for clients, adding scheduled evaluations is consistently the change that moves an agent from "we check on it" to "we trust it" — and it's usually under two hours of build time per agent.

The Short Version

Manual chat-panel testing proves the happy path works once. That's it.
Use the Evaluation Trigger + Evaluation node with a dataset in a Data Table or Google Sheet.
Build the dataset from real execution history — 25 real cases beat 200 invented ones.
Add Tools Used for any multi-tool agent. Right answer, wrong path, is a future outage.
Use Check If Evaluating so test runs never fire real emails, CRM writes, or charges.
Set the ship threshold before you run the suite, and enforce a floor, not just an average.
Schedule the suite nightly and alert on drops — it catches model drift before your client does.

Ship Agents You Can Actually Vouch For

The agencies getting burned by AI agents in 2026 aren't the ones building bad agents. They're the ones with no way to answer the question "how do we know it still works?" — which is also why so many AI SDR deployments quietly fail months after a great demo.

An evaluation suite turns that into a number you can put in a client report. It's the least glamorous part of n8n workflow automation and the part that decides whether an agent survives its first quarter.

If you've got agents running in production right now with no test dataset behind them, get a free automation audit. We'll pull your execution history, show you which agents are failing silently, and build the eval suite that stops it.