AI systems rarely crash. They drift. Chatbots start inventing prices, agents loop and burn tokens, and n8n workflows report "success" while doing nothing. To catch this you need quality monitoring: grade chatbot answers against your docs, run nightly test questions, log every agent run, and alert on workflows that stop producing output, not just ones that error.
Most businesses launch an AI chatbot or an automation, test it for a day, and move on. Then months later a customer forwards a screenshot: the bot promised free shipping that doesn't exist, or a refund request sat unanswered for a week. I've built AI chatbots, AI agents and n8n workflows for hundreds of client projects, and this is the most common failure pattern I see. Not a crash. A slow, silent drop in quality.
This guide explains what actually goes wrong, what to measure and how to set up monitoring, either yourself or with a tool.
Why AI Needs a Different Kind of Monitoring
Traditional monitoring asks one question: is it up? A ping, a status code, a green dashboard. AI systems pass that test even when they're failing customers, because the failure is in the content of the answer, not the availability of the service.
| System | What uptime monitoring sees | What actually went wrong |
|---|---|---|
| Support chatbot | 200 OK, fast replies | Quoted a shipping price that isn't in your policy |
| AI agent | Run completed | Called the same tool 14 times and cost 20x the normal run |
| n8n order sync | Execution: success | Returned zero orders for three days after a token expired |
| Make scenario | No errors | Scenario was switched off and simply stopped running |
The 5 Failure Types to Watch For
1. Made-up answers (hallucinations)
The bot states something confidently that isn't in your documentation: prices, discounts, delivery times, features. These are the most damaging because customers act on them.
2. Missed escalations
Refund disputes, complaints, legal threats and "I want to talk to a person" should go to a human. When the bot keeps answering with generic text instead, you lose customers quietly. I cover how to design the human handover in my AI lead qualification guide.
3. Off-policy answers
The answer is technically true but breaks a rule: promising a refund outside the refund window, giving advice you're not allowed to give, or mentioning a competitor.
4. Agent loops and runaway costs
AI agents decide which tool to call next. When a tool returns an error or unexpected data, agents can retry the same step over and over, burning tokens and returning nothing. See when you actually need an AI agent before adding one.
5. Silent workflow failures
An n8n or Make run that finishes "successfully" but produces zero output. Your webhook still fires, the error workflow never triggers, and nobody knows anything is wrong until the numbers don't add up.
How to Monitor an AI Chatbot (Step by Step)
- Export real conversations. Most platforms (Intercom, Tidio, Crisp, Zendesk, Chatbase) let you export transcripts. Start with the last 100 to 500.
- Collect your source of truth. Help center articles, pricing page, refund and shipping policies. The bot can only be "wrong" relative to something.
- Grade every answer. Use a strong model as a judge: give it the question, the answer and the relevant docs, and ask for one label: correct, made-up, not in docs, should escalate, off-policy.
- Add hard rules. Some facts are too important for a judge model alone. Check any price, percentage or date in an answer against a list of allowed values.
- Group problems by source. A score alone doesn't help. Group mistakes by the help article or policy they relate to, so you know which doc to fix.
- Build a nightly test set. Save 20 to 50 real questions with the facts a correct answer must include. Ask the bot every night and alert when a required fact is missing.
- Mask personal data. Strip emails, phone numbers and card numbers before anything is stored or sent to a judge model.
How to Monitor n8n and Make Workflows
n8n has an Error Trigger node, and it's worth setting up. But it only fires on errors. For real coverage, check these on a schedule (every 15 minutes works well):
- Failed executions: the obvious one; alert once, not on every retry.
- Zero-output runs: successful executions where the key node returned no items.
- Heartbeat: a workflow that normally runs hourly hasn't run for three hours.
- Error-rate spikes: 2% failures is normal for some APIs; 30% in an hour is not.
- Slow runs: a run taking three times its usual duration often means an API is rate-limiting you.
If you self-host, make sure your n8n public API is reachable by your monitor. My n8n VPS setup guide covers a production-ready install.
How to Monitor AI Agents
Log every agent run with one HTTP call at the end: input, list of tool calls, errors, token count, cost and final output. Then alert on:
- More than N calls to the same tool in one run (loops)
- Tool errors followed by a confident final answer (the agent is guessing)
- Empty or very short outputs
- Cost per run above a set limit
DIY vs a Monitoring Tool
You can build all of this in n8n yourself: a scheduled workflow that pulls transcripts, calls a judge model and posts to Slack. I've built versions of it for clients. It works, but it becomes one more system to maintain, and it's the kind of thing that breaks silently too.
That's why I built ProofMyAI. It does everything in this guide out of the box: chatbot audits graded against your docs, nightly bot tests, AI agent monitoring with one HTTP call, and n8n/Make monitoring with a two-minute API key setup. Alerts go to Slack, Discord, Teams or email, and personal data is masked automatically.
ProofMyAI runs a free one-time audit on up to 100 of your chatbot conversations, with no card required. Paid plans start at $29/month with a 14-day free trial.
Get your free AI audit →A Simple Monitoring Checklist
- ☐ Last 100+ chatbot conversations graded against your docs
- ☐ Hard rules for prices, refunds and delivery times
- ☐ Nightly test set of 20 to 50 real questions
- ☐ Escalation check for complaints, refunds and "talk to a human"
- ☐ n8n/Make alerts for failures, zero-output runs and stopped workflows
- ☐ Agent runs logged with tool calls and cost
- ☐ Personal data masked before storage
- ☐ Weekly summary reviewed by a human
Frequently Asked Questions
AI chatbot monitoring means regularly checking what your chatbot actually tells customers: whether answers match your documentation, whether it invents facts, and whether it hands difficult conversations to a human. It is quality monitoring, not uptime monitoring.
Compare each answer with your knowledge base. An AI judge model reads the question, the bot's answer and the relevant help articles and labels the answer as supported, unsupported or contradicting the docs. Add rule-based checks for high-risk facts such as prices, refund terms and delivery times.
A silent failure is an n8n or Make execution that finishes with a success status but produces nothing useful, for example an order-sync run that returns zero orders because an API token expired or a filter is wrong. Error workflows don't fire because technically nothing errored.
Run a fixed set of real customer questions against the bot at least once a day (nightly is common), and audit a sample of live conversations every week. Re-test immediately after changing prompts, models or help docs.
Yes. Log each agent run (input, tool calls, tokens, cost, output) to a monitor with one HTTP call, then alert on loops, repeated tool errors, empty outputs and cost spikes.
Want Your Chatbot or Workflows Checked?
Run a free ProofMyAI audit yourself, or let me review your chatbot, AI agent or n8n setup and fix what's broken.
Get a Free Automation Audit → Free ProofMyAI audit