A company with 3,000 monthly reviews and 800 annual NPS comments actually has one problem: not a lack of data, but a lack of time to read it. An analyst needs a week to extract meaningful patterns from one quarter. By the time the report reaches the board meeting, the data is four weeks old. In that time, customers who wrote about a specific pain point have long since decided whether to stay.
A well-designed sentiment analysis system turns this weekly process into a continuous stream of signals. Below, I describe how such a system works, where its competence ends, and where human decision-making begins.
Beyond positive/negative: aspect-based analysis and intent#
A "negative" label on a review says nothing useful. "Negative, regarding delivery time, with intent to cancel" is an operational signal.
Aspect-based sentiment analysis (ABSA) breaks text into specific dimensions: product, price, service, delivery, interface. One sentence can have a positive sentiment toward the product and a negative one toward delivery simultaneously. A model that reduces this to a single label loses half the information.
Intent completes the picture. A customer who writes, "considering canceling my subscription after this incident," has different service priorities than one who writes, "delivery was late, but the product is great." The first should be escalated to retention the same day. The second just needs a thank-you.
Urgency is the third dimension. A public review from a customer with a yearly contract saying, "Scandal, no access for 48 hours," requires a different response time than an anonymous comment about packaging aesthetics. The model can infer urgency from a combination of tone, signal words, customer history in CRM, and channel (public reviews have higher reputational priority than private tickets).
Where text comes from and how to collect it#
Data for sentiment analysis comes from several sources, which differ in structure and processing requirements.
Product reviews and ratings: Structured (stars + content), variable volume, often anonymous. Stars are a weak signal. A 3/5 review could express bitterness or enthusiasm. The text content matters; the numeric rating is just context.
NPS comments: Short, often laconic, written in the moment. Harder to classify by aspect, good for tracking general mood trends and spotting patterns after campaigns.
Support tickets: The richest in context. Customers describe specific issues, provide dates and products. Also the most expensive to process, as they often contain personal data (name, order number, email) requiring masking before analysis.
Social media mentions: Unstructured, brief, full of slang and abbreviations. Require separate preprocessing and a model calibrated for platform-specific language.
Collecting data from different sources is an ETL task, not AI. The sentiment model receives clean text after normalization.
System architecture: from text to structured signal#
The pattern we use at Cashcrown for designing such systems consists of four steps.
Step 1: Preprocessing and PII masking. Text is cleaned of personal data before being passed to the inference model. Order numbers, names, and email addresses are replaced with placeholder tokens. This is not optional for tickets and forms—it’s required by GDPR.
Step 2: Multidimensional classification. The model (or LLM prompt with few-shot examples) returns a structured output with fields: aspect (delivery/product/price/service/interface), sentiment_per_aspect (positive/negative/mixed), intent (information/complaint/cancellation/request for help/praise), urgency (high/standard/low), keywords. Schema validation eliminates empty or hallucinated fields. If the model returns urgency: null or a confidence score below the threshold, the record goes to a manual verification queue.
Step 3: Aggregation and trend detection. Structured records go to an analytical database. Aggregation by weeks and aspects shows when a decline occurred in a specific dimension. An 8-percentage-point increase in intent: cancellation over two weeks is an operational signal that precedes churn rate data by several weeks.
Step 4: Routing and human-oversight. Records with urgency: high or intent: cancellation are automatically routed to a consultant’s queue with a ready brief (aspect, quote, customer history from CRM). The consultant makes the decision. The model does not contact the customer independently. This rule is strict and has no exceptions.
Table: What the model classifies and where automation ends#
| Dimension | What the model determines | Where humans decide |
|---|---|---|
| Product aspect | Delivery, price, service, interface, product | New aspect not present in training data |
| Sentiment per aspect | Positive, negative, mixed | Irony, sarcasm, industry-specific slang |
| Intent | Complaint, cancellation, praise, request for help | Ambiguous or legal intent (formal complaint) |
| Urgency | High, standard, low | Every case with urgency: high before contacting the customer |
| Trend | Increase/decrease in dimension share over time | Interpretation and decision on what to do with the trend |
Limitations you can’t ignore#
Every sentiment analysis system has structural weaknesses. Ignoring them leads to trusting data that shouldn’t be trusted.
Sarcasm and irony. "Wonderful service, just the order got lost three times" is lexically positive but negative in intent. Language models handle sarcastic sentences much worse than literal ones. In Polish text, where irony is more common than in English training corpora, misclassification of such cases is a recurring pattern, not a random error.
Mixed sentiment in one sentence. The text "delivery was express, but product quality disappointed" contains two opposing signals. A one-dimensional classifier will pick one or average them, losing both. Aspect-based analysis reduces this problem but doesn’t eliminate it entirely for short, multi-threaded sentences.
Industry and regional slang. A model trained on consumer reviews will perform worse on opinions from industrial, medical, or legal sectors, where the same words have different connotations. Fine-tuning on 200–500 examples from your domain usually improves recall by several percentage points but requires preparing a labeled dataset.
Polish language: specific challenges. Inflection and subjectless sentences are rare in English but typical in Polish. A base model trained mostly on English corpora makes systematic errors where the subject is implied by verb endings. Multilingual models (e.g., BGE-M3, mT5) handle this better, but testing on your own data is mandatory before production deployment.
A single label is misleading. Reporting "75% positive reviews" without breaking it down by aspects and intent is an oversimplification that can hide a critical problem. A product with 75% positive reviews overall and 40% negative reviews about delivery has a logistics issue, not a reputational one. A single label won’t show that.
Validation: How to check if the model isn’t lying#
Before sentiment analysis results go into an executive report or trigger operational actions, they must be validated on a human-labeled sample.
Standard approach: Randomly select 200–400 records from each channel, manually label them by 2–3 people independently, compare with the model’s classification. Metrics: precision, recall, and F1 per dimension (aspect, intent, urgency), not just global accuracy. For urgency, recall is more important than precision: missing a high-urgency case costs more than a false alert.
We perform this validation before deployment and repeat it every 6–8 weeks or after significant changes in product mix or channels. Input data distribution drift is a real risk: a model calibrated on year-old reviews may not recognize a new product or complaint pattern.
Observability of the system includes at least: class distribution per week (if one class’s share grows faster than business intuition suggests, it’s a signal to investigate), escalation rate to manual verification, and a weekly report on discrepancies between the model’s classification and consultant decisions. The latter is the most valuable signal for recalibration.
More on designing quality loops for AI agents in our article on customer service automation. The golden set validation pattern for RAG and classification systems is also discussed in ticket classification and routing.
FAQ#
Can AI analyze sentiment in Polish as well as in English?#
Not at the same level with an off-the-shelf English model. Polish reviews have different inflection, longer subordinate clauses, and more frequent irony than English training corpora. Multilingual models (mT5, XLM-RoBERTa) provide a much better starting point than English-only models. Fine-tuning on 300–500 of your own Polish examples usually improves recall by 10–20 percentage points compared to a ready-made model without adaptation.
How much data volume do I need for analysis to make sense?#
The minimum threshold depends on whether you’re looking for trend signals or operational decisions. To detect weekly trends, 100–200 records per week from one channel are enough. For operational routing (which record goes to a consultant), you need at least 500–1,000 labeled training examples per class for the model to be sufficiently confident. Below these thresholds, results are indicative, not operational.
What to do with opinions containing customer personal data?#
Before passing to the inference model, personal data must be masked or removed: name, email, order number, phone number. In systems processing personal data at scale, a DPIA is required before production deployment. If opinions contain health or financial data, the protection level is higher, and self-hosting the model is the recommended solution. Details of this layer are covered in our article on AI for content moderation, where PII masking is a standard pipeline step.
Can a sentiment analysis system automatically respond to negative reviews?#
We don’t recommend this pattern. Sentiment analysis is a signal for humans, not a trigger for automatic communication. An auto-response to a public negative review generated by a model without consultant verification is a reputational risk, and for formal complaints (claims, consumer rights), it may violate information obligations. The proper pattern: the model classifies and routes to a queue, the consultant responds. With very high volume, you might consider an auto-response confirming receipt of the report, but not a substantive answer. More on this boundary in our article on complaint handling.
How to present sentiment analysis results to the board so they’re useful?#
Avoid a report with a single number (e.g., "80% positive"). Instead: a trend chart of sentiment per aspect over time, top 5 keywords per intent class for the given week, and a summary of the number of records with urgency: high passed to consultants versus resolved. This format shows what’s changing and what requires action, rather than confirming a state the board already knows intuitively. The pattern for building such operational reports is discussed in the context of AI for marketing teams.