LLM red teaming: test your assistant with attacks before production

Three weeks before launching a B2B client, one of our assistants responded to a cleverly crafted query by quoting a fragment of the system prompt. It wasn’t a model error—it was a gap in the guardrails architecture, which we discovered ourselves because we ran a red teaming session before going live. The fix took two days. If we’d found it post-launch, it would have cost significantly more.

Traditional infrastructure pentesting looks for vulnerabilities in software: open ports, unpatched CVEs, authorization flaws. LLM red teaming looks for a different kind of weakness: how the model responds to inputs you didn’t anticipate when designing the system.

Key difference: pentesting is typically one-off, conducted before launch, and ends with a report listing findings. LLM red teaming in a well-designed system is continuous: every new finding goes into an attack catalog, the catalog becomes a regression suite, and the suite runs automatically with every configuration change or knowledge base update.

Why does continuity matter? The base model may change. The RAG knowledge base grows and may ingest new documents. The system prompt is modified. Each of these changes can open a new attack surface that didn’t exist during the previous review.

LLM red teaming isn’t free improvisation. We work with a catalog of attack classes, each with defined test methods and evaluation criteria.

The detailed taxonomy of 10 vulnerability classes is described in the OWASP LLM Top 10 standard. At Cashcrown, we use this taxonomy as a checklist for every red teaming session.

Not every finding is equally urgent. Scoring helps prioritize fixes and communicate risk to the team.

We use three dimensions: exploitability (how difficult the attack is), impact (what the attacker can achieve), and deployment context (whether the system handles sensitive data, has tool access, or operates publicly).

The practical outcome is three priorities:

• Critical: attack executable by an untrained user, leads to data leakage or irreversible actions. Blocks launch.
• High: attack requires preparation but is repeatable. Fix before launch, not after.
• Low/Observed: attack is difficult or has low impact. Goes into the registry, is monitored, doesn’t block launch.

A one-time pre-deployment review is necessary but insufficient. The continuous red teaming loop looks like this:

1. Every new finding (regardless of source: internal test, user report, literature) goes into the catalog as a test case.
2. The test case format: input attack + expected defensive response (refusal, masking, escalation to human).
3. The catalog runs automatically with every change to the system prompt, RAG knowledge base update, or base model change.
4. Each new result is compared to the previous one: no regression is a merge condition.
5. Monthly, new test cases are added from current literature and reports.

This pattern works on the same principles as regression testing in software engineering: a bug reported once becomes a test that ensures the bug doesn’t return. The difference is that the attack set keeps growing.

This is an important caveat we make upfront: no red teaming program can prove a system is “fully secure.” LLMs are probabilistic systems: the same input at a different temperature or model version may yield different results.

The goal of red teaming is more modest—and therefore realistic: known attack classes have been tested, discovered vulnerabilities are documented with severity and mitigation status, and the regression suite ensures fixed issues don’t return after changes.

Red teaming session results are part of the AI system’s technical documentation—the same document that, for high-risk systems, supports AI Act risk management requirements.

At Cashcrown, every assistant system undergoes an audit before deployment (a 6-area checklist, described in the article on AI assistant security audit) plus a red teaming session with an attack class catalog.

The starter catalog includes at least: 5 variants of direct prompt injection, 3 variants of RAG injection, 4 variants of persona jailbreak, 2 variants of configuration leakage, and tests in all supported languages (injection patterns in Polish, English, Ukrainian, German). In total, 15–25 test cases for a typical RAG assistant with tools.

Post-launch, the catalog grows: every report or new pattern from literature becomes a new case. After 6 months, the system typically has 40–80 test cases. Ongoing monitoring (described in the article on monitoring [hallucination] and assistant behavior) complements red teaming between sessions.

The defensive layer architecture tested by red teaming is detailed in the articles on protection against prompt injection and OWASP LLM Top 10.

Classic pentesting looks for vulnerabilities in infrastructure: open ports, unpatched CVEs, authorization flaws. LLM red teaming tests model behavior with malicious inputs: injection, jailbreaks, forcing configuration leaks. Both are needed in a full security program, but they test different attack surfaces and require different methods and tools.

For a typical RAG assistant with 3–5 tools, the first session includes preparing the starter catalog (0.5 days), running tests (1 day), scoring and documenting findings (0.5 days). Total: 2 working days. Initial results usually fall within 2–6 findings, of which 0–2 are critical. Subsequent sessions are faster because the catalog is already prepared.

The AI Act doesn’t mention red teaming by name, but for high-risk systems, it requires documented risk management and pre-deployment testing. Red teaming session results—with an attack catalog, severity scoring, and mitigation list—naturally fill this documentation gap. For general-purpose systems (typical informational chatbot), the obligation is weaker, but lack of documentation makes incident response harder.

A critical vulnerability blocks launch. The fix takes priority over the schedule: stopping is cheaper than an incident post-launch. In practice: the critical finding becomes a ticket with the highest priority, mitigation is tested on the same test case that discovered it, and only after verification does it return to the PASS list. Deadline pressure is understandable, but security compromises in customer-facing systems have measurable reputational and legal consequences.

Observability in continuous monitoring plays a complementary role: logging anomalies (injection attempts caught by guardrails), alerting on spikes in blocked queries, reviewing response samples with every knowledge base update. It doesn’t replace red teaming sessions but signals when new patterns emerge that require catalog expansion.