cashcrown // ai quality evaluation
Eval Harness
eval-harnessliveAn evaluation harness for LLM/RAG systems: a golden set, automated metrics, an LLM-as-judge calibrated against human labels, and a regression gate that blocks a deploy when quality drops.
4 tryby
metric modes
self-host
deploy
zero-retention
data
CI gate
regression
// 01
Problem
Without evaluation you do not know whether a new prompt, model, or RAG change improved quality or quietly broke it. Eyeballing answers does not scale and does not catch regressions before they reach the customer.
// 02
Capabilities
- a golden set of test cases with expectations
- automated metrics: precision/recall/F1, faithfulness, citations
- an LLM-as-judge calibrated against human labels
- a regression gate that blocks a deploy when quality drops
- full run history for trend analysis
// 05
Integrations
routerBGE-M3QdrantOllamaFastAPIn8n
// 06
Architecture
Select a node to see its description and data flow.
Architecture — text description
- Ingest / data
- Data sources — Content, documents and events feed the index. (Flows to: BGE-M3)
- BGE-M3 — Local embeddings (1024-dim) — PII stays on-prem. (Flows to: OpenClaw router)
- AI core
- OpenClaw router — The only entry to the models — masks PII, picks the model, throttles. (Flows to: Eval Harness)
- Eval Harness — An evaluation harness for LLM/RAG systems: a golden set, automated metrics, an LLM-as-judge calibrated against human labels, and a regression gate that blocks a deploy when quality drops. (Flows to: API)
- Serve / integrations
- API — REST surface — cited answers with access control. (Flows to: router, BGE-M3, Qdrant, Ollama)
- router — Integration: router.
- BGE-M3 — Integration: BGE-M3.
- Qdrant — Integration: Qdrant.
- Ollama — Integration: Ollama.
// 07
Live demo
// 08
FAQ
How much does it cost and what does the rollout look like?
We work in ranges that depend on scope — we start with a fixed-cost pilot in which we build the golden set and the first regression gate. An eval usually pays off at the first regression it catches before it reaches the customer — it saves hours of manual review and costly slips. Calculate the return in the ROI calculator.
Does it integrate with our CI and is it GDPR-compliant?
Yes. We run the harness in your CI as a gate before deploy — a red result blocks the release. It runs self-hosted with zero retention: test data and embeddings (BGE-M3) stay with you, we mask PII before the cloud and handle sensitive paths locally — GDPR and AI Act compliance designed in from the start.
Where do we start, and will the eval replace a human?
We start with a few dozen representative cases and their manual labels — that is the golden set we calibrate the LLM-as-judge against. An eval supplements but never replaces human judgement on high-stakes outputs: the automation catches regressions and tracks the trend, while a human makes the critical calls and has the final say at the gate.
// →Related