AI in call centers: voice, transcription, agent assistance

A call center consultant handles 60-80 calls per day. After each one, they must manually enter notes into the CRM, tag the topic, and set a follow-up. This takes 3-5 minutes per call. With 70 calls, that’s 3.5 hours of data entry instead of serving customers. This isn’t a futuristic problem that AI will solve in a few years—it’s a cost you can measure in today’s budget.

AI in call centers isn’t a single tool. It’s a layer of architectural decisions: what to fully automate, what to support with an assistant, and what to leave exclusively to humans. Below, I describe each of these layers from a technical and operational perspective.

Transcription is the most common entry point for AI in call centers. You convert audio recordings to text, which you can then analyze, index, search, and pass to subsequent models. Without transcription, none of the higher layers work.

ASR (Automatic Speech Recognition) models available in 2026 fall into two classes. Cloud models (SaaS) offer a low entry threshold and quick API integration, but every recording leaves the company’s infrastructure. Local models (Whisper and its variants, including faster-whisper optimized for CPU) run entirely on your own servers, eliminating the data-residency issue.

For a Polish call center, model selection for the language is key. Most commercial ASRs have good WER (Word Error Rate) metrics for Polish, but differences emerge with regional accents, industry terminology, and speech in noise. Benchmarking before deployment on a sample of 200-500 real recordings from your center is mandatory.

Transcription can operate in post-call mode (after the call ends) or real-time (streaming during the call). Post-call mode is simpler and sufficient for 80% of cases: CRM notes, topic analysis, call QA. Real-time mode is only necessary for real-time agent assistance.

Voice recordings are PII by definition. Voice is biometric data under RODO, even if you’re not using voice-based identity verification. Additionally, call content includes PESEL numbers, payment card numbers, addresses, and other sensitive data.

The architecture must address this issue before sending anything to an external model. Possible approaches:

• Self-hosting the entire pipeline (ASR + LLM locally): data never leaves the company’s infrastructure. Requires GPU hardware or powerful CPUs for faster-whisper.
• PII masking after local transcription, before LLM analysis: ASR runs locally or in a private cloud, the transcript text is filtered through NER (Named Entity Recognition), card numbers and PESELs are replaced with tokens [CARD] / [PESEL], and only the anonymized text is sent to the analysis model.
• Consent for processing + data processing agreement with a cloud provider: legally permissible but requires a DPIA for biometric processing, documentation of the legal basis, and a mechanism for data deletion requests (the right to be forgotten covers recordings and transcripts).

The AI Act categorizes real-time biometric identification systems as high-risk. A voice bot that only understands and responds to speech doesn’t biometrically identify individuals, so it doesn’t automatically fall into this category. However, integration with customer contract databases and behavior profiling may change the classification. Before deployment, it’s worth reviewing with a lawyer specializing in the AI Act.

An AI agent supporting a consultant during a call is an architecture of several components working with low latency:

1. STT (Speech-to-Text) in streaming mode converts the customer’s voice to text with a 200-800 ms delay. Simultaneously, it analyzes the call context.
2. RAG over the company’s knowledge base searches for relevant documents: procedures, FAQs, product data, customer history from the CRM.
3. LLM via router generates a response suggestion or prompt for the consultant. The consultant sees the suggestion on screen and decides whether to use, modify, or ignore it.
4. Guardrails block suggestions containing unverified prices, commitments beyond level-1 authority, or uncertain facts.

The consultant remains at the center of decision-making. AI assistance reduces information search time (from 30-60 seconds to 3-5 seconds) without replacing situational judgment. This is important for both service quality and legal accountability.

Latency is a critical parameter for assistance. A suggestion appearing 8 seconds after the customer’s question is useless in a fast-paced conversation. The realistic target is 2-3 seconds from the customer’s last word to the suggestion appearing on the consultant’s screen. This is achieved through RAG context compression (top-3 fragments, not 20), a fast model for generating suggestions, and streaming responses instead of waiting for full text.

A voice bot is fully automated voice handling without a consultant. The customer speaks, the bot understands, responds with voice, and executes actions. It differs from traditional IVR by handling natural language, not touch-tone menus.

A voice bot must have built-in human-handoff with a low threshold. The customer should be able to say “Connect me to a consultant” at any time and reach a live agent within 30 seconds. A bot that hinders this transfer to artificially inflate containment rate violates best practices and may expose the company to allegations of misleading customers.

The AI Act requirement to disclose AI identity: the customer must know at the start of the call that they’re speaking with an automated system. Impersonating a bot as a human is explicitly prohibited under EU regulations effective from August 2, 2026.

Post-call automation is the least risky and fastest layer to implement. The call transcript is sent to a model that generates:

• A short summary (3-5 sentences) for the "Notes" field in the CRM
• The call topic from a predefined taxonomy (complaint, order, technical question, cancellation)
• Customer sentiment (positive / neutral / negative)
• A list of suggested follow-up actions

The consultant sees the generated draft and approves it with one click or edits it. Instead of writing from scratch for 3 minutes, they verify it in 30 seconds. Time savings amount to 70-85% for this task.

Integration with CRM (Salesforce, HubSpot, Zoho, Polish systems like Optima) happens via CRM API or middleware like n8n. Architecture details for integration via n8n and directly through ERP and company systems are covered in separate articles.

For this automation, the rule applies: the model generates a draft, the human approves. There’s no automatic CRM entry without consultant verification during the first 3-6 months. After this period, when the draft error rate is below 5%, the scope of auto-entry can gradually expand for repetitive note types.

Traditional call center QA relies on random call sampling by a supervisor. With 1 supervisor per 15 consultants and 70 calls per day per person, that’s several hundred calls weekly, of which maybe 20 can be reviewed.

AI changes this model. Instead of random sampling, every call is transcribed and automatically evaluated for:

• Script compliance (whether mandatory phrases were used: introduction, customer verification, recording consent)
• Customer vs. consultant talk time (a healthy ratio is ~60/40 in favor of the customer in sales)
• Detected keywords indicating escalation (profanity, cancellation, complaint threat)
• Pricing compliance (the model verifies if quoted prices are current in the knowledge base)

The supervisor receives a prioritized list: calls flagged as lowest by the algorithm are at the top for manual review. The same QA time now covers 100% filtering and 15-20% manual verification of flagged cases.

Voice bots and real-time agent assistance require guardrails tailored to voice specifics. Additional challenges compared to chat:

• Homophones and ASR errors: the model hears “three hundred” and transcribes it as “300 PLN” or vice versa. Guardrails must detect inconsistencies between spoken numbers and those in the transcript.
• Multi-turn context: a customer might say, “Do it like last time.” Guardrails check if referencing a previous action is safe for automatic execution.
• Call pace: in real-time, there’s no time for multiple model calls. Structured output with predefined action categories is faster and safer than free-text generation.
• Human gate for irreversible actions: order cancellation, refunds, account data changes. None of these actions can be executed by a bot without human confirmation or at least double identity verification.

Voice system monitoring is based on the same layers as AI agent monitoring described in the article on AI agent quality monitoring. Additional voice-specific metrics include: WER (Word Error Rate) of the ASR model on test samples, transfer rate (how many customers request a consultant before the bot call ends), and abandon rate (how many customers hang up before getting an answer).

Describe your call center scenario, and the model will indicate which AI layer to implement first and what guardrails are critical for your scope (playground: PII masked, zero retention):

Yes, from August 2, 2026, there is a requirement to disclose automated identity at the beginning of every interaction with an AI system. The customer must know they’re speaking with a bot before providing any data. Failure to disclose is a violation of the AI Act subject to penalties. This applies to both voice bots and chatbots. Details on company obligations are covered in the article AI Act and RODO 2026.

Voice recordings are biometric data and require a legal basis for processing. A secure architecture is either self-hosting the ASR model or masking PII after local transcription before sending text to an external LLM. Payment card numbers and PESELs must be detected by NER and replaced with tokens before analysis. Biometric processing requires a DPIA and implementation of a data deletion request procedure.

Post-call transcription with automated CRM notes is realistically 4-8 weeks from project start to production on a narrow scope. Real-time agent assistance requires 8-16 weeks due to streaming ASR integration and latency calibration. A voice bot for standard queries takes 12-20 weeks, including tests with real customers in shadow mode. Assess your organization’s readiness with the readiness assessment tool.

It depends on the chosen layer and scale. Post-call transcription for 300 calls daily with a local model (faster-whisper on CPU) costs a few hundred PLN monthly in infrastructure, with no per-call fees. Real-time assistance requires more computational power. A voice bot incurs integration and maintenance costs, but with proper containment rate (50-70% for standard queries), it can pay off in a few months. Generate a real cost estimate for your volume with the ROI calculator or inference calculator.

Not in the next 2-3 years for complex queries. AI assistance increases consultant throughput and shortens handling time, allowing more customers to be served with the same resources or maintaining the same throughput with less hiring. A voice bot handles repetitive queries that don’t require situational assessment. Complaints, disputes, crisis situations, and emotionally distressed customers still require human interpersonal skills. The boundary between what to automate and what to leave to humans is covered in the article on the role of humans in the AI loop.