Chat translation quality is usually evaluated by asking whether the translation is accurate. That is the wrong question for an operating team, because accuracy is largely a solved problem and the failures that reach a customer are not accuracy failures.
The failures that matter in live chat are register failures, commitment failures, and tone failures — the cases where the words are right and the meaning is not what you sent. This is a review scheme for a team that cannot read every language it serves.
Why random sampling fails as a quality control
Sampling fifty random messages and reading them produces a number that is high and not actionable. Most customer chat is greetings, logistics and confirmations, and those are exactly the messages that do not need checking.
The consequential messages are rare by construction — they are rare because they are the exception. A sample of fifty will contain one or two of them and no way to know in advance which. So the review either misses them, or it escalates to reviewing everything, which is the same as having no control.
The fix is to stop sampling at random and classify by consequence.
Four categories, and only one needs a human every time
Routine. Greetings, opening hours, order status, shipping estimates, confirmations. Let them pass. Reading them costs more than the errors are worth.
Commitment-bearing. Anything stating a delivery date, a price, a refund, a specification, or a promise to do something. Human read before sending. This is usually under a tenth of traffic and contains nearly all the commercial risk.
Emotional. Messages where the customer is frustrated, and messages where they are being told no. Flag these for awareness rather than review — the agent is not checking accuracy, they are being told the conversation has already gone badly, which changes how carefully they answer it.
Unknown. Anything the system is not confident about. Needs a weekly look at its size. If it is not shrinking, the confidence threshold is wrong.
Three failure categories that survive good word-level accuracy
Politeness that reads as evasion. In several languages, "we will consider it" is a formula that means no. Translated literally into a language where it also means no, it produces a reply that promises consideration the seller never intended. This has the highest commercial cost, because it creates an expectation the next message has to retract.
Softened commitments. Words like "will", "may", "approximately" and "subject to" carry obligations. A translation that softens one does not introduce an error in meaning — it introduces or removes a commitment. In B2B conversation this is the risk that matters, and it is invisible to anyone reading only the translation.
Smoothed urgency. An angry customer who writes briefly reads as calm, because brevity is neutral. In translation, brief and calm look identical. Teams that escalate on tone lose the escalation, because the machine-rendered version is too tidy to trigger anything.
The commitment library: ten phrases, used verbatim
The most useful artefact in this whole area is not a glossary. It is a short list of the phrasings your business actually commits to, written once, in the original languages, and used verbatim rather than translated.
In practice it is short: your standard shipping estimate, your refund conditions, your warranty statement, your three most common refusals, your apology wording. Ten to twenty phrases. Once they exist, agents compose from them rather than translating whatever phrase came to mind, and the largest remaining category of commercial error largely disappears.
The second artefact is a monthly look at the unknown category, asking one question: what did the system not understand, and is that a vocabulary gap or a model gap? Vocabulary gaps are fixed by adding terms. Model gaps are not, and treating them as vocabulary gaps is how a glossary grows to four hundred entries and the error rate does not move.
What to report
Three numbers, all of them counts you can already produce:
- Commitment-bearing volume as a share of traffic. A rising share usually means a change in customer mix rather than a change in quality.
- Edit rate within commitment-bearing messages. This is the real quality number and should be stable or falling. A jump means something changed — a new model, a new market, a new phrasing.
- Unknown-category size. This one tells you whether to keep tuning the threshold or stop.
None of it requires a reference corpus, a linguist, or a scoring rubric.
For how replies are composed and where a human read belongs in the pipeline, see the WhatsApp Business setup checklist and a WhatsApp team inbox for small teams. For the language tagging and per-language targets this assumes, see SLA: first response versus resolution.
Source note: This article draws on Language.io's "Enterprise Chat Translation Services" (https://languageio.com/enterprise-chat-translation-services/), which describes enterprise chat translation offerings. The four-category review scheme, the verbatim commitment library, and the reporting set are our own operational design. No accuracy figures or vendor claims from the source are used as verified data here.