A test set built around the real job
We turn one live product claim into realistic cases, awkward edge cases and explicit pass or fail rules.
Founding pilot now open
Your demo passed.
The holdout may not.
We pressure-test one important AI workflow before customers find the expensive failures. You get evidence your team can reproduce and a straight answer on what to do next.
FIX THE TEST BEFORE THE PRODUCTKEEP THE HOLDOUT UNTOUCHEDSHOW THE ACTUAL FAILURELET THE ANSWER BE STOP
The Reliability Snapshot
A hard test of one claim your product makes.
Not a broad consultancy exercise. Not a compliance badge. A compact independent test that gives a small team something useful enough to change a product decision.
An untouched holdout
The final cases stay out of tuning. That shows whether a fix generalises or merely learns the examples it has already seen.
Failures your team can reproduce
Every material omission, invention, wrong action or overconfident answer comes with the input and evidence needed to inspect it.
A decision, not another dashboard
You get the fixes that matter first and a plain view on whether to ship, narrow the workflow or stop.
Why Holdout Works exists
The headline score looked excellent. The product still failed.
In our own synthetic technical experiment, a conversational resolver detected nearly every food component and matched every structured source. It looked ready to improve.
The untouched holdout exposed systematic undercounting and a dangerous false-reassurance problem. We stopped the build. Strong averages had hidden the failure that mattered.
This was an internal synthetic engineering test, not a customer engagement, clinical validation or independent certification. It is shown because we think stopped work is evidence too.
FINAL HOLDOUTSTOP
95.7%component precision and recall
100%structured source resolution
−7.79%systematic calorie bias
26.67%dangerous false reassurance
The attractive metrics did not cancel the material failure.
The working method
Small enough to finish. Serious enough to change the answer.
Pick one promise
We agree the single workflow, who relies on it and what a costly failure looks like.
Set the rules
Cases and thresholds are fixed before the final run, including a holdout neither side tunes against.
Try to break it
We run realistic and adversarial cases, preserve the evidence and separate product failures from test noise.
Make the call
You receive the scorecard, reproducible failures and the shortest sensible repair order.
A useful fit
One owner. One workflow. One costly way to be wrong.
01 Your AI feature already works well enough to test, not just describe.
02 Its output or action can be checked against evidence or explicit rules.
03 A miss, invention or wrong action has a real product or operating cost.
04 You want to find the limit, not purchase a flattering score.
Founding pilot
Tell us the claim you cannot afford to get wrong.
Send three things: what the AI does, who relies on it and what a costly failure looks like. We will tell you whether a compact independent test can produce a useful decision before asking you to buy one.
£950 UK
US$1,250 International