Cashcrown // ai quality evaluation · 09/11
Eval Harness
eval-harnessliveAn evaluation harness for LLM/RAG systems: a golden set, automated metrics, an LLM-as-judge calibrated against human labels, and a regression gate that blocks a deploy when quality drops.
4 tryby
metric modes
self-host
deploy
zero-retention
data
CI gate
regression
01
Problem
Without evaluation you do not know whether a new prompt, model, or RAG change improved quality or quietly broke it. Eyeballing answers does not scale and does not catch regressions before they reach the customer.
02
Capabilities
- a golden set of test cases with expectations
- automated metrics: precision/recall/F1, faithfulness, citations
- an LLM-as-judge calibrated against human labels
- a regression gate that blocks a deploy when quality drops
- full run history for trend analysis
03
Delivery scope
- A golden benchmark dataset tailored to the client domain with versioning in the repository
- Automated deterministic metrics (BLEU, ROUGE, F1) paired with a calibrated LLM-as-a-judge
- A CI/CD quality regression gate blocking prompt or model deploys whenever faithfulness drops
- A visual experiment comparison dashboard, model drift analysis, and cost-quality trend reports
We assemble this product from measured models and components - see the full live pipeline:
04
Proof-led pilot
Pilot package: evaluation harness and quality regression gate
We deploy Eval Harness to end eyeball evaluation: we build a versioned golden dataset, automated faithfulness metrics, and a hard CI/CD gate blocking regressions.
100%
golden dataset coverage
0
silent production regressions
CI/CD
automated deploy gate
Evidence
Case study: automated audit and adversarial verification
See how multi-stage evaluation eliminated false positives and guaranteed repeatable benchmark results.
view proofReproducibility and test auditability
Principles for deterministic LLM judges, test versioning, and auditable benchmark run logs.
view proofPilot path
- 01Step 1: Collection and annotation of 100 key user queries into a curated golden dataset
- 02Step 2: Calibration of the LLM-as-a-judge for domain specifics and setting faithfulness and citation thresholds
- 03Step 3: Connecting the regression gate to GitHub Actions / GitLab CI pipelines and running the baseline audit
05
Integrations
routerBGE-M3QdrantOllamaFastAPIn8n
06
Architecture
Select a node to see its description and data flow.
Architecture - text description
- Ingest / data
- Data sources: Content, documents and events feed the index. (Flows to: BGE-M3)
- BGE-M3: Local embeddings (1024-dim) - PII stays on-prem. (Flows to: OpenClaw router)
- AI core
- OpenClaw router: The only entry to the models - masks PII, picks the model, throttles. (Flows to: Eval Harness)
- Eval Harness: An evaluation harness for LLM/RAG systems: a golden set, automated metrics, an LLM-as-judge calibrated against human labels, and a regression gate that blocks a deploy when quality drops. (Flows to: API)
- Serve / integrations
- API: REST surface - cited answers with access control. (Flows to: router, BGE-M3, Qdrant, Ollama)
- router: Integration: router.
- BGE-M3: Integration: BGE-M3.
- Qdrant: Integration: Qdrant.
- Ollama: Integration: Ollama.
07
Live demo
08
FAQ
How much does it cost and what does the rollout look like?
We work in ranges that depend on scope: we start with a fixed-cost pilot in which we build the golden set and the first regression gate. An eval usually pays off at the first regression it catches before it reaches the customer: it saves hours of manual review and costly slips. Calculate the return in the ROI calculator.
Does it integrate with our CI and is it GDPR-compliant?
Yes. We run the harness in your CI as a gate before deploy: a red result blocks the release. It runs self-hosted with zero retention: test data and embeddings (BGE-M3) stay with you, we mask PII before the cloud and handle sensitive paths locally: GDPR and AI Act compliance designed in from the start.
Where do we start, and will the eval replace a human?
We start with a few dozen representative cases and their manual labels: that is the golden set we calibrate the LLM-as-judge against. An eval supplements but never replaces human judgement on high-stakes outputs: the automation catches regressions and tracks the trend, while a human makes the critical calls and has the final say at the gate.
eval-harness · Docker / Kubernetes (On-Premises, Air-Gapped, or private EU VPC)
Executive Brief 1-Pager (Board & Procurement Summary)
Without evaluation you do not know whether a new prompt, model, or RAG change improved quality or quietly broke it. Eyeballing answers does not scale and does not catch regressions before they reach the customer.
metric modes: 4 trybydeploy: self-hostdata: zero-retentionregression: CI gate
Target integrations:routerBGE-M3QdrantOllamaFastAPIn8n
→Related