Teams that switch on auto-replies for every message usually learn about wrong answers when a customer quotes one back. These rules sort each question into one of three modes: the AI replies directly, drafts for a person to approve, or hands off. They assume a Slack Connect channel, where the customer sees every post and cannot tell who wrote it.
Sort every question into one of three types
Sort each incoming question by the kind of answer it needs:
- Factual: the answer is written down in something you control, such as a help article or a policy. Example: “Do you support SSO on the Starter plan?”
- Status: the answer lives in a system of record, such as an invoice or a ticket. Example: “Has invoice 4471 been paid?”
- Refunds and exceptions: the answer depends on judgment, a contract, or money leaving your account. Example: “Can you credit us for last week’s downtime?”
| Type | Default mode | Direct auto-reply allowed when |
|---|---|---|
| Factual | Direct reply, otherwise draft | One approved source answers it and the score clears the threshold below |
| Status | Direct reply, read live | The record was read at send time and the reply states that time |
| Refunds and exceptions | Hand off to a person | Never. The AI may collect details into an internal note |
“Your invoice is paid” from a cached value is a wrong answer delivered with confidence. A safe status reply carries its timestamp: “As of 14:05 UTC today, invoice 4471 shows as unpaid.”
Set the confidence thresholds on your own messages
Scales differ between tools, so 0.85 in one product may not mean 0.85 in another. Calibrate on 20 to 30 real messages from the last month, including short and misspelled ones, each marked with the correct answer. These cut-offs are a starting proposal, not an industry standard:
- A score of 0.85 or higher, with the answer coming from one approved source: send directly. This applies to the factual type. Status replies follow the live-read rule in the table.
- A score from 0.6 to 0.85, or any answer that needs two sources: draft for a person to approve.
- A score below 0.6, or no source at all: hand off, and send only a holding message.
Two rules override the score. If the answer needs two articles to agree, draft it. If the source covers price, plan limits or legal terms and was not reviewed in the last 90 days, hand off until its owner confirms it.
Two articles conflict when they give different answers to the same question. Name one owner per article. Until the owner picks the right version, the question stays in draft mode.
The usual failure is a score that holds on tidy questions and drops on typos.
Escalation keywords that force a person
Keyword rules are blunt, but they catch what scores miss. Escalate with no AI reply when the latest customer message, or any earlier customer message in the thread, contains:
-
Money: “refund”, “chargeback”, “credit”, “charged twice”.
-
Legal: “lawyer”, “lawsuit”, “GDPR”, “DPA”, “terminate”.
-
Security: “breach”, “leaked”, “hacked”, “API key”.
-
Churn: “cancel”, “not renewing”.
-
Handoff request: “person”, “human”, “real agent”.
-
Match case-insensitively across customer messages only. The AI’s own disclosure and holding lines contain “person”, so scanning them would escalate every thread. Review the list every two weeks against the messages that slipped through.
-
Reply directly only in languages your calibration set covers and a reviewer can read. Otherwise draft, or hand off if no reviewer reads the language.
-
If a customer pastes a secret, do not send any AI-generated answer. Flag it to your security owner, then have a person or the approved template send: “Please rotate that key now and don’t post secrets in this channel. Our team will reply here.”
-
Two unresolved messages in a row from one customer means a handoff, whatever the score.
Can the AI speak directly in a shared channel?
Yes, for factual and status replies, under the conditions below, as Slack’s documentation stood in 2026:
- Guests. The AI agents article says “In all cases, guests can’t use AI apps or agents.” It does not address Slack Connect external users directly, so confirm with your Slack admin how it applies.
- Scopes. “The data that an AI app in Slack can access depends on its scopes.” Adding an app to a channel extends its access to that conversation. Scope the assistant to the channels it needs.
- Posting. Only the organization that owns a channel can invite organizations and manage posting permissions. Each organization must be using a paid Slack plan, and one channel can host up to 250 organizations.
- The Enterprise+ AI restriction is a read control. Slack lists it as “Available on the Enterprise+ plan” and says it “only restricts AI read access to channels and content”. Members can still use Slackbot to send messages to restricted channels. The article does not mention apps or third-party assistants, so it does not settle whether your assistant may post. Get that answer in writing from your admin and the vendor.
Identity, disclosure and the stop switch
Identity. Post as a named automated assistant, such as “Acme Support (AI assistant)”, not under a generic team name. Slack’s chat.postMessage reference says custom username and icon parameters are “only available when an app has requested and been granted an additional scope”, which is chat:write.customize, alongside chat:write.
Disclosure. Put this in the first reply of every new thread: “Hi, I’m Acme’s automated support assistant. I answered from our help article on [topic], last updated [date]. Reply “person” and someone on our team will take over in this thread.”
EU AI Act. If the assistant reaches people in the EU, ask counsel. The Commission’s AI Act page says the Act “became applicable on 2 August 2026, with some exceptions” and that “The transparency rules of the AI Act will come into effect in August 2026”. It gives no separate chatbot date. The AI Act Service Desk quotes Article 50(1) as requiring that people be informed they are interacting with an AI system, unless that is obvious. Who holds the duty when you buy the assistant rather than build it is a question for counsel.
Stop switch. One named person can move any channel or question type to draft mode alone. Use it for any money, legal or security answer that reached a customer, when a day’s wrong-answer rate is above your ceiling, or when replies post to the wrong thread. After a pause, correct the answer in the same thread that day: “Correction to my earlier message: [correct answer]. Sorry for the mistake. A person on our team has checked this.”
Outside hours. Off shift, keep direct replies to factual and status types only. The holding message gives a time with a zone: “We received this outside support hours. A person will reply in this thread by 09:00 Monday, UTC.”
Data handling to get in writing
Before go-live, get written answers to these: where customer messages are stored and for how long; whether they train models; which subprocessors receive the text; how a deletion request is carried out; and whether the vendor signs a data processing agreement. If any answer is missing at go-live, keep the assistant out of shared channels.
A two-week pilot with three numbers
Run the pilot for two weeks on one question type and one customer group, starting with factual questions. Define automated resolution in your pilot notes, and count direct sends apart from drafts a person approved.
- Wrong-answer rate: replies a reviewer marks wrong, divided by all AI replies reviewed. Review every direct send in week one, then at least half after that.
- Escalation rate: conversations the AI touched that ended in a handoff, divided by all conversations the AI touched.
- CSAT change: the average CSAT on AI-touched threads, minus the average for the same question types in the two weeks before. Track reopen rate too: the share of threads a customer reopens within 48 hours of closing.
As a proposal, a reply is wrong if it contradicts the approved source, leaves out a condition the source states, or gives a status without a timestamp. Refund, legal and security answers the AI sent count as wrong by default. Have two reviewers mark the same 20 replies before the pilot; if they disagree on more than two, rewrite the rubric.
Decision rules at day 14, all starting proposals:
- If the wrong-answer rate is above your ceiling for a category, move that category to draft mode for one more week. Set the ceiling before the pilot. A starting proposal for direct send is no more than two wrong answers per hundred reviewed sends.
- If escalations exceed half of AI-touched conversations, the thresholds are too strict or the knowledge base has gaps. Read the handoff notes first.
- If CSAT falls more than 0.2 points below baseline on a five-point scale, stop direct sends for that group and keep drafts running.
Two failure modes: an easy question set flatters every metric, so sample the lowest-scoring messages and reopened threads as their own row. Report survey counts beside every CSAT average, and do not call a change until each group has at least 30 surveyed threads.
Measure response speed with the definitions in the Slack Connect SLA first response time guide, and keep AI first response time in a separate column from human replies.