Loading…
Paid Audit
We build a golden-set evaluation dataset from your real queries, run your system against it, and produce a scored report showing where retrieval fails, where the LLM drifts from the retrieved context, and where latency spikes. This is a structured measurement engagement, not a vague quality review, but a scored baseline across six specific metrics with failure examples annotated by root cause.
You get a full performance baseline and a prioritised fix list. The baseline measurement is valuable on its own. It tells you exactly where you are before you spend anything on improvements. Without it, any fix you make is unverifiable. Turnaround: 7 business days. Report and 60-minute debrief included.
Tell us what you're building.
Six metrics, each measuring a different failure mode. A RAG system can fail at retrieval, at generation, or at both simultaneously — the metrics tell you which.
Of the chunks needed to answer a question, what percentage does your system actually retrieve? A system with poor recall produces confident-sounding answers that miss critical information, the most common root cause of hallucination in production RAG.
Does the LLM's answer actually use the retrieved context, or does it generate from pre-training knowledge instead? Low faithfulness means your retrieval is working but the model is ignoring it, a prompt engineering problem, not a retrieval problem.
Does the answer address what the user actually asked? A faithful answer that addresses a slightly different question still fails. This metric catches cases where retrieval and generation are both individually fine but the pipeline produces off-topic responses.
Are the retrieved chunks actually relevant to the query, or is the retrieval returning noise? High noise in the context window dilutes the relevant information and inflates token cost. Low precision is a direct signal that your chunking or embedding strategy needs work.
How long does the full pipeline take from query to first token, at median and at the 95th percentile? P95 latency is what your users experience on a bad day. We measure retrieval time, reranking time, and LLM time-to-first-token separately so you know where the bottleneck is.
What percentage of queries trigger a "no answer found" or low-confidence fallback response? A high fallback rate means your retrieval is under-performing on real queries, even if it looks fine on the test cases you wrote yourself.
We take 150–300 real queries from your production logs, or construct them from your document corpus if you don't have logs, and write reference answers for each. This query-answer pair set becomes the ground truth the evaluation suite runs against. The methodology is the same as academic RAG benchmarks: a fixed evaluation set with known correct answers, applied to your specific domain and document set.
The advantage of this approach is reproducibility. Run the same evaluation suite after making changes and the scores are directly comparable. You're not measuring against an abstract standard. You're measuring your system against a dataset that reflects what your users actually ask. The evaluation dataset is included in the deliverables so your team can use it for ongoing regression testing.
Four deliverables. Fixed scope. No ongoing retainer required.
What a well-optimised RAG system scores on each metric for reference. You see not just where you are, but how far you are from a good baseline.
10–20 concrete examples of failing queries with root-cause annotation. "This query fails because the relevant information spans two chunks that are never retrieved together", not a generic finding.
Every identified improvement ranked by expected score gain divided by implementation effort. Chunking strategy changes, embedding model swaps, reranker additions, prompt revisions, each with an effort estimate in engineering days.
We walk through findings, answer questions about specific failure patterns, and help you decide which fixes to prioritise based on your engineering capacity and business priorities.
The audit requires a specific setup to produce valid scores. If these conditions don't apply, we'll tell you before you pay.
Systems with fewer than 100 real queries to draw from
We need real queries to build a meaningful evaluation dataset. Fewer than 100 is not enough to represent your actual query distribution. If you don't have logs yet, we can construct synthetic queries from your corpus, but the evaluation will be less precise.
Teams who want fixes before measurement
We need the performance baseline before we can measure improvement. An audit without a baseline produces a one-time snapshot with no way to verify whether changes actually helped. If you want fixes only, see our RAG Development service.
RAG systems built on closed platforms where retrieval can't be instrumented
If your RAG runs inside a platform that doesn't expose retrieval internals (some hosted chatbot builders and enterprise search tools), we can't instrument the pipeline well enough to produce accurate recall and precision scores.