A customer service department for an online store receives 200 complaints weekly. Thirty percent are repetitive cases: damaged goods on delivery, invoice errors, missing items in a package. Consultants respond to these almost identically, week after week, for months. Another 40% require checking a single fact in the warehouse system or ERP before a standard response can be sent. The rest are complex cases: legal disputes, customers after multiple failed attempts, submissions containing sensitive data.
AI handles the first two groups well. It should not attempt to operate independently on the third.
Four layers of triage: what AI does before the first response#
Before any complaint reaches a consultant’s queue or a response generator, a well-designed system processes it through four stages.
Stage 1: Complaint type classification. A multi-label classifier simultaneously determines the category (damaged goods, billing error, delivery delay, discrepancy with description, warranty claim), channel (email, form, chat, phone STT), and language. With 500 or more training examples per category, 85-92% accuracy is achievable. Below 200 examples, base models with few-shot prompting perform better than fine-tuning.
Stage 2: Urgency and sentiment detection. Urgency is a separate dimension with asymmetric costs. Missing a high-urgency case (customer lost service access, case went to mediation) costs far more than a false escalation alert. Thus, the urgency classifier operates with clear asymmetry: it prefers false alarms over misses. Strongly negative sentiment (repeated words like “scandal,” “unlawful,” “lawsuit”) triggers escalation to a consultant regardless of category.
Stage 3: Fact extraction from content. Structured output with schema validation extracts the order number, purchase date, amount, problem description, and attached evidence (photos, receipt PDFs). Extracted facts are immediately available to the response generator and consultant, eliminating manual transcription from emails.
Stage 4: Verification in external systems. The agent queries the ERP or warehouse system: does the order exist, what was the delivery status, is the warranty active? The response returns as facts, not as a ready-made decision. The model knows the order was delivered on time. Does that mean the complaint is unfounded? That’s a judgment for humans.
Table: Complaint type, AI role, mandatory human-gate, risk level#
| Complaint Type | AI Role | Mandatory Human-Gate | Risk Level |
|---|---|---|---|
| Damaged goods (standard) | Classification, response draft from RAG template | Consultant approves draft before sending | Low |
| Invoice error | Classification, amount and number extraction, correction draft | Accounting department approves correction | Medium |
| Delivery delay | Status check in system, response draft with facts | Consultant verifies facts before sending | Low |
| Complaint rejection | Draft justification citing regulations (RAG) | Human makes rejection decision, not AI | High |
| Compensation or voucher | Draft proposal based on company policy | Human accepts value and form, signs off on decision | High |
| Strongly negative sentiment or legal | Detection, escalation priority, context gathering | Exclusively human, AI does not generate response | Very High |
| Submission containing sensitive data (GDPR) | PII detection, masking before cloud model, escalation | Human decides on every step | Very High |
How RAG builds a response draft#
Once the classifier determines the case is suitable for an automated draft, the system queries the knowledge base via RAG. The base contains approved response templates, regulations, warranty policies, and return procedures. The model doesn’t invent content: it combines facts extracted from the complaint with fragments found in the base.
Condition for correct operation: The knowledge base must be up-to-date and cover all complaint categories. If the company changed its return policy but the base wasn’t updated, the model will generate a draft based on outdated procedures. Thus, updating the knowledge base isn’t a one-time project but an ongoing operational duty. The article on RAG knowledge updates discusses incremental reindexing patterns.
The response draft goes to the consultant as ready-to-approve or edit text, not as a sent message. The consultant sees: complaint category, extracted facts, the regulation fragment used as a source, and the proposed content. They can approve with one click or edit and send. Handling time decreases, quality improves, responsibility remains with humans.
SLA tracking and observability: how to know the system works#
Observability for a complaint system starts with four metrics measured from the first day of the pilot.
Time to first response. Time from complaint receipt to sending the first response (by AI or consultant approving the AI draft). Target for standard cases: under 2 hours during business hours. The article on AI agent quality monitoring details patterns for collecting these metrics.
Draft approval rate without edits. Percentage of response drafts consultants accept without changes. With a well-calibrated knowledge base and narrow category scope, this rate reaches 60-75% for standard cases. If it drops below 40%, the base needs updating or the AI-handled category scope is too broad.
Escalation rate and re-classification rate. Percentage of cases rerouted to a higher queue after initial routing. High escalation rate (above 15%) signals urgency classification errors. Re-classification rate above 20% for a category indicates a need to review training labels or split the category, as discussed in the article on classification and routing of submissions.
SLA compliance per category. Are cases in a given category closed within the declared time? The system should flag cases approaching SLA limits and automatically raise their priority before the deadline.
Automation boundaries: where AI doesn’t go#
Several categories are outside the scope of automated handling regardless of classifier accuracy.
Complaint rejections. A customer receiving a rejection from an automated system has a worse experience than one receiving a human rejection with justification. AI can prepare a rejection draft (with a regulation quote), but the consultant makes and signs the decision. This is also a requirement of the AI Act for systems affecting consumer rights, and complaint handling falls precisely into this category.
Compensation and vouchers above a set threshold. The company should define a monetary threshold above which every financial decision requires human approval. Below the threshold, the system can automatically generate discount vouchers according to policy tables, always with logging and audit capabilities.
Complaints containing sensitive data. Submissions where a customer describes a health issue from product use or includes financial data require PII masking before any cloud model processing. If the system isn’t self-hosted, sensitive data shouldn’t leave local infrastructure.
Cases after multiple contacts. A customer writing for the third time about the same issue is already frustrated. The system should automatically escalate such submissions to a senior or designated retention queue, not send another template-based response draft.
Pilot pattern: shadow mode for 4 weeks#
Safe deployment starts with shadow mode. For the first 4-6 weeks, the classifier and draft generator run in parallel with the existing process: they classify, extract facts, and prepare drafts, but consultants handle cases independently and see AI results alongside their decisions.
Comparing AI decisions with consultant decisions builds ground truth: where the model misclassifies, where drafts are acceptable without edits, where sentiment is detected correctly. After 4 weeks, you have data to safely enable automation for selected low-risk categories.
The article on service companies and AI and company obligations under AI Act and GDPR discusses documentation requirements worth preparing before launching the system on customer data.
At Cashcrown, we build such systems as a research center: every component (classifier, draft generator, human-gate, observability) is measured separately before integration. There’s no single correct urgency threshold or ready-made complaint taxonomy. Calibration for a specific industry and the client’s historical data determines the outcome.
FAQ#
Can AI independently decide to accept or reject complaints?#
It shouldn’t. The substantive decision on a complaint has legal and relational consequences: an incorrect rejection can expose the company to UOKiK proceedings or customer loss. AI can prepare a justification draft with a regulation quote, but the final decision is made and signed by a human. The AI Act, effective from 2026, explicitly imposes this requirement for systems affecting consumer rights, and complaint handling falls precisely into this category.
How to protect customer personal data in a complaint system?#
Complaints often contain personal data: name, address, order number, sometimes health or financial data. Before sending content to a cloud model, this data should be masked locally by a PII masking system. If a case contains particularly sensitive data, the entire pipeline should run on self-hosted infrastructure, or the case should be handled exclusively by humans. Before deployment, conducting a DPIA is advisable, especially if handling sensitive sectors (health, finance, children).
How long does deploying such a system take?#
A shadow mode pilot with a classifier based on few-shot prompting can launch in 3-5 weeks if you have historical complaint data with labels. Full deployment with ERP or helpdesk integration, SLA metrics, and escalation procedures takes 8-14 weeks. The largest time portion goes to collecting and reviewing training data, calibrating escalation thresholds, and training consultants in the new workflow—not the programming itself.
What metrics should be measured to assess system effectiveness?#
Four numbers tell the truth: time to first response (target: under 2 hours for standard cases), draft approval rate without edits (target: 60-75% for narrow categories), escalation rate (signal of urgency classification errors if above 15%), and SLA compliance per category (percentage of cases closed on time). Containment rate, the percentage of cases closed without human intervention, makes sense as a metric only after at least 8 weeks of stable system operation.
What to do when a customer writes for the third time about the same issue?#
A customer with multiple contacts about the same issue should be automatically escalated to a senior or designated retention queue, not routed back to the automated draft generator. The system should detect contact history (number of submissions in the last 14 days regarding the same order or topic) and treat it as a priority escalation signal, regardless of the current submission’s category or sentiment.