AI for fraud and anomaly detection in companies

A company detects an irregularity in a financial statement two months after it occurred. For eight weeks, transactions appeared normal in weekly reports because the report compares month-to-month and does not track behavioral patterns in real time. This is a scenario that repeats in finance departments, procurement operations, and payment systems: an anomaly exists in the data, but no threshold rule catches it.

AI for fraud detection works differently than a list of rules. It learns the pattern of normality and flags deviations before they exceed a threshold a human could set. Below, I describe the architecture of such a system, which metrics to measure, where the limits of automation lie, and what the law requires.

A mature fraud detection system operates in several layers, each catching a different type of deviation.

Rule-based layer remains the first line of defense. Typical rules: amount exceeding authorization limit, transaction from a country outside the whitelist, account number from a block list. Rules are deterministic, fast, and auditable. Problem: attackers who know the rules adjust their actions just below the thresholds.

Statistical layer calculates deviations from the average transaction value for a given account, frequency of actions in a time window, and deviations from typical hourly activity distribution. It detects the scattering of many small transactions designed to avoid threshold rules.

Model layer is a classifier trained on a history of transactions labeled as legitimate or fraudulent. Input: a feature vector of the transaction (amount, time, category, counterparty data, account history). Output: fraud probability (0-1) and risk category. Gradient boosting models (XGBoost, LightGBM) and tabular neural networks offer the best balance of effectiveness and interpretability here.

Sequential behavior layer analyzes a sequence of events, not just a single event. A user who logs in from a new device within 20 minutes, changes contact details, adds a new account, and initiates a transfer creates a high-risk pattern, even if each step individually falls below the alert threshold. LSTM or Transformer models on event sequences are the right tools for this.

Coordinating agent collects signals from all layers, triggers additional checks (query to a registry, vendor history lookup), and produces a unified alert with context that the analyst sees on the dashboard.

The key design pattern for a fraud detection system is separating reversible from irreversible actions.

Automatic action (without human-gate): flags a transaction in the system as suspicious, lowers the daily limit, enforces additional authorization at the next login. Reversible within seconds by an analyst or the user themselves.

Action requiring human-gate: blocking an account, rejecting a transfer, reporting to a regulator, escalating to the compliance department. Irreversible or costly to reverse. A human must always approve such decisions here.

Implementation pattern for human-gate in practice: an alert with a risk score above 0.75 goes to the analyst queue with full context (why the system rated the risk so high, which features were decisive, the account history for the last 30 days, similar previous cases). The analyst approves or rejects it in an interface requiring a justification comment. This comment goes into the audit log.

The table below compares the layers and their properties:

The only fair starting point: an ML model for fraud detection needs historical data with labels. If the organization didn’t previously have a tracking system, where do you get positive data (verified fraud cases)?

Four approaches we use in practice:

Rules as label generator. Run the old rule-based system on historical data and treat its results as weak labels. The model learns patterns that rules already caught, plus anomalies near those patterns.

External datasets. For the financial sector, there are pooled datasets of transactions with labeled frauds (e.g., IEEE-CIS Fraud Detection, PaySim). A good starting point for the first model, requiring later fine-tuning on your own data.

Synthetic data. A synthetic data generator (CTGAN or a dedicated business process simulator) produces examples of fraudulent behavior based on expert knowledge of fraud patterns. This topic is covered in detail in the article synthetic data for AI.

Active learning with analysts. The first weeks of system operation are in shadow mode: the system generates alerts but doesn’t block. Analysts review and label alerts. Within 4-6 weeks, you collect several hundred labels that allow training the first real model.

Accuracy is the wrong metric for fraud detection. With 0.5% frauds in the dataset, a model that always answers "no fraud" has 99.5% accuracy and is useless.

Proper metrics:

Precision: how many transactions flagged by the system as fraud are actually fraud. Low precision = many false alarms = analysts waste time checking clean transactions, and the system loses credibility.

Recall (sensitivity): how many actual frauds the system detected. Low recall = fraud slips through undetected. For fraud detection, recall is usually more important than precision, but both metrics must have set minimums.

F1-score and ROC-AUC as aggregate indicators. ROC-AUC above 0.95 on the test set is a good starting point for production.

Cost of error is the ultimate business metric. A false positive (blocking a legitimate transaction) costs: lost transaction, complaint handling, damaged customer relationship. A false negative (undetected fraud) costs: financial loss, investigation cost, reputational risk. Define this cost numerically and adjust the classifier threshold to minimize total cost, not maximize F1.

Alert response time is an operational metric: how long it takes from alert generation to analyst decision. An alert queue that grows slower than it’s processed signals staffing or prioritization issues.

Fraud detection systems process data that is almost always personal data: account numbers, transaction history, location, device identifiers. Each of these categories is subject to GDPR.

Key practical requirements:

Legal basis for processing. For processing data to detect fraud, the basis is the legitimate interest of the controller (Art. 6(1)(f) GDPR) or a legal obligation under AML regulations (Anti-Money Laundering Act). The basis must be documented in the processing activities register.

Data minimization. The model doesn’t need the customer’s first and last name to assess transaction risk. Work with pseudonymized internal identifiers wherever full identification isn’t required at a given stage. Identifying data is only added when a confirmed alert requires action.

Retention. Training data, alert logs, and analyst decisions have different retention periods. Training data without PII can be stored long-term. Logs with full transaction data are subject to TTL according to GDPR policy and sector regulations.

DPIA. For systems with automated transaction profiling for risk assessment, DPIA is required by Art. 35 GDPR. DPIA must include a description of risk profiles, assessment of impacts on data subjects, and risk minimization measures.

Self-hosting vs. cloud. ML models and training data for fraud detection contain the organization’s full transaction history. In many sectors (banking, insurance), regulators require that data doesn’t leave a controlled environment. Self-hosting the model and infrastructure provides full control over data residency.

AI systems for fraud detection in the financial and credit sectors are explicitly listed in Annex III of the AI Act as high-risk systems (point 5b: AI systems used to assess the creditworthiness and credit capacity of natural persons). This applies to both off-the-shelf solutions and systems built for internal use if the organization acts as a deployer or provider.

Obligations arising from classification as a high-risk system:

Technical documentation includes a description of the model, training data (sources, scope, anonymization procedures), quality metrics on the test set, limitations, and known errors. Must be updated with every significant model change.

Decision transparency. The system must explain why a transaction was flagged as risky. SHAP values or LIME for gradient boosting models, attention for sequential models. The analyst must understand which features were decisive. Explainability is discussed in the article the black box problem.

Human-oversight as a requirement, not an option. For decisions with negative consequences for a natural person (account blocking, transaction refusal, escalation to law enforcement), the law requires the ability to challenge the decision by a human and review it. The system must enable this technically, not just procedurally.

Post-deployment monitoring. The AI Act requires active monitoring of system effectiveness on production data. This means regularly running quality tests, documenting degradation, and having a response procedure for detected degradation.

A fraud detection system without monitoring degrades quietly. Fraudster behavior evolves: a model trained on year-old data may not recognize new patterns. Minimum production metrics:

Alert rate (percentage of transactions generating alerts) monitored daily. A sudden spike signals an attack or system error. A sudden drop signals the model stopped detecting a new pattern.

Precision on confirmed alerts (analysts label alerts as true fraud or false alarm). A precision drop for two consecutive weeks signals model drift requiring retraining.

Time from transaction to alert (latency). Increased latency may indicate infrastructure overload or increased computational complexity with more features.

Escalation rate to human-gate. If it rises, analysts are overloaded, or automatic action thresholds are too conservative. If it falls with a stable alert rate, possible threshold calibration issues.

The complete agent monitoring architecture is described in the article monitoring AI agent quality.

Implementing a fraud detection system doesn’t start with an ML model. It starts with data and process inventory.

Week 1-2: inventory. Which source systems have transaction data? How is it structured? Are there historical fraud labels (even customer complaints, compliance department decisions)? Where is the human-gate in the current process?

Week 3-4: shadow mode with rules. Run a simplified rule-based system in shadow mode (logs alerts, doesn’t block). You collect data on transaction distribution, calibrate thresholds, and verify if alerts reach the right people.

Month 2: first statistical model. Train on collected historical data. Run in shadow mode alongside rules. Compare rule alerts with model alerts. Analysts label discrepancies.

Month 3: launch with human-gate. The model generates alerts, rules can block automatically, everything above a set risk threshold goes to the analyst queue. Measure metrics weekly.

Month 4+: improvement cycle. Retrain the model on collected labels. Add sequential layers when you gather enough history. Expand automation only for reversible actions.

The full implementation plan is described in the article AI implementation plan step-by-step. The agent blueprint tool helps design the alert flow architecture.

Describe your fraud detection or anomaly scenario, and the model will indicate which architecture to use, which metrics to monitor, and what is required by GDPR and the AI Act in your case (playground: PII masked, zero retention):

A rule-based and statistical system works from day one without training data. An ML model needs a history of transactions with at least several hundred confirmed fraud cases or false alarms. If such data doesn’t exist, the first 4-6 weeks are in shadow mode: the system logs alerts, analysts label them, and only then is the model trained. Synthetic data can speed up this start but requires validation on real data before production launch. A detailed description of approaches to initial data is in the article how to prepare company data for AI.

AI systems used for profiling or financial risk assessment of natural persons are listed in Annex III of the AI Act as high-risk. Consequence: required technical documentation, DPIA obligation (data protection impact assessment), ensuring decision explainability, and active post-deployment monitoring. The organization implementing such a system must also ensure the ability to challenge decisions by a human. This isn’t theory: AI Act compliance inspections in the financial sector began in Poland in 2026. Obligations under the AI Act and GDPR for companies are discussed in the article AI Act and GDPR 2026.

A model trained on historical decisions may perpetuate discriminatory patterns if past decisions were biased (e.g., a higher rate of controls for specific demographic groups). Minimum safeguards: bias audit on the test set (whether the false positive rate differs significantly between groups), removal of legally protected attributes (gender, age, nationality) from model features, regular equality effectiveness tests. The AI Act for high-risk systems requires documenting bias mitigation measures as part of technical documentation. More on algorithmic bias is in the article algorithmic bias in research.

Fraud detection assumes a known target class (fraud as a label in training data) and uses supervised classifiers. Anomaly detection doesn’t require labels: it learns the distribution of normality and flags anything that deviates from it. In practice, both approaches are used together. A supervised classifier catches known fraud patterns. Unsupervised anomaly detection (autoencoders, isolation forest, DBSCAN on embeddings) catches unknown patterns, including entirely new attack types. The coordinating agent layer combines signals from both sources. Multi-step agent architecture is described in the article multi-step AI agent.

Cost depends on scope: whether integrating with an existing transaction system (ERP, payment platform), how many data sources are included, whether self-hosting is required due to regulations, and the expected alert latency. A pilot scope (shadow mode, one transaction stream, rule-based-statistical system) has a different cost than a full multi-layer architecture with a sequential model and analyst dashboard. The scope that makes sense for your organization can be calculated using the ROI calculator or by contacting us via the contact form.