Try the live assistant
blog / What Happens When Your AI Flags…

What Happens When Your AI Flags a Customer?

AI chatbot moderation can misjudge a customer. How a fair system records the classifier's score, gives the benefit of the doubt, and never asserts labels.

From the AnswerRails development blog — plain-English notes on the technology behind trustworthy AI chat.

Before your assistant answers a visitor, it screens the message. Is it abusive? Is it an attempt to misuse the chat? That screening is AI moderation, and it has to exist — abuse runs on your own AI provider account. But when it flags someone, a piece of software has judged one of your customers, in your name. This post is about what should happen when the judge isn’t sure.

A score is not a verdict

Moderation models don’t return verdicts. They return scores: a category and a number, like hate_and_discrimination: 0.81.

That example is real. While rebuilding our guardrails, we replayed a real conversation through the screening and one message came back flagged at 0.81. We read it. It was a contested opinion, strongly put. Not a slur, not a threat. 0.81 means probably, and probably is not certainly.

A naive system treats the flag as the verdict: block the message, log that the visitor sent hate speech, add a penalty. All from a number nineteen points short of sure. And the borderline visitor doesn’t complain. They just leave.

Write down what the model said, not what your customer is

When a message is flagged in AnswerRails, the log records what the classifier reported: its category, its score, and which model it was. It never says “this visitor sent hate speech.”

This matters legally too. Chat logs are records about people, and the site owner is responsible for them. If a visitor asks for a copy of their data, “an automated classifier scored this message 0.81” is a fact. “You were abusive” is an accusation your software wasn’t qualified to make.

Older records that predate this scheme show “flagged — details not recorded”. We don’t backfill a guess.

Diagram of AnswerRails moderation banding: a confidence-score scale with a mid band below the high-band threshold and a high band at or above it; an example message flagged at 0.81 lands in the mid band and is politely declined with no strike recorded; the record panel shows the log storing the classifier's own category, score and model — never a label about the person.
A flag is a score, not a verdict: below the threshold the flag costs the visitor nothing, and the record keeps the classifier’s own words either way.

The benefit of the doubt, engineered

Scores allow proportion. Above a high threshold — set conservatively by default — the flag is treated as earned: the message is politely declined and a strike is recorded. Strikes accumulate; repeated flags end the chat.

Below the threshold, the visitor gets the same polite deflection, but no strike. Our 0.81 message lands there: deflected, nobody punished, the record showing exactly why.

Two exceptions. The gravest categories — the ones involving danger to people — are never banded; the conversation stops regardless. And a visitor in evident distress isn’t treated as wrongdoing at all: no strikes, straight to signposting a human.

Who sets the threshold? The site owner. It’s visible, and on our paid tier adjustable, with the conservative default doing the work for everyone else. A judgment made in your name should have your hand available on the dial.

Three questions for any vendor

When your system flags a visitor, what gets written down — a label about the person, or what the classifier reported? Does a borderline flag cost the visitor anything? And who sets the threshold — can I see it, and can I change it?

None of this is softness. But when a check is about a person, record it like evidence, not like an accusation.


AnswerRails is a WordPress plugin bringing safe, grounded AI chat to small-business websites. More at answerrails.com.

← All posts