AI Test Bench · a fictional quote desk
No prompt ships without its eval.
Meridian Sample Supply (fictional) runs an AI reader on its quote inbox. Someone has rewritten the reader’s instructions. Before the new version touches a real customer, both versions read the same 60 test messages, every answer is scored against a named source, and a gate decides. Press Run and watch it decide.
Recorded outputs · fictional data No live AI · runs in this tab · nothing is sent
Step 1 · price it · run it · gate it
Same 60 cases. Two versions. One verdict.
Before the run · cost estimate
- This run
- 2 versions × 60 cases = 120 case-runs
- Per case (last run)
- $0.0184
- Estimate
- $2.21
- Cap per run
- $25.00
- Month to date after
- $414.81 / $750.00 line
Run allowed: under the cap
The estimate is cases × versions × the last run’s cost per case (Version A · Oct 2 (fictional)). Fictional numbers. The bench itself always replays the 60 recorded cases.
Tolerance band · one tick per case · outside the band = out of tolerance
Run A first, then B. Each replays recorded outputs.
- passes
- fails
- fixed by B
- broken by B
- Version A53/60current prompt · recorded
- Version B54/60scores higher than A · recorded
- Fixed+4A failed, now passes
- Broken−3A passed, now fails
- Unchanged5350 pass · 3 still fail
Ship gate
Version A passes 53 of 60. Version B passes 54 of 60: 4 fixed, 3 broken. Ship gate red: Baseline not re-recorded, cannot ship. 3 regressions block the ship.
Step 2 · side by side
B scores higher. It still can’t ship.
Version B fixes 4 cases A got wrong and breaks 3 that A got right. A better total hides the three it broke, which is why the bench compares case by case, field by field. Expected answer, then each version’s recorded output; the wrong cells are marked.
Broken by B 3 cases · these block the ship
#21“When can it ship?”Broken by B
From buyer@customer1075.example.com
Message Can you quote 40 HCY-R3A-0500 and tell me when they could ship?
Expected answer from The desk’s entered quote Q-26-10438
| Field | Expected | Version Arecorded | Version Brecorded |
|---|---|---|---|
| Intentexact | Quote | matches: Quote | matches: Quote |
| Routeexact | Quote desk | matches: Quote desk | matches: Quote desk |
| Customerexact | C-1075 | matches: C-1075 | matches: C-1075 |
| Part #exact | HCY-R3A-0500 | matches: HCY-R3A-0500 | matches: HCY-R3A-0500 |
| Qtyexact | 40 | matches: 40 | matches: 40 |
| Unitsexact | inch | matches: inch | matches: inch |
| Desk summaryjudged | Rubric: Names part and qty; no invented date, price or stock. | matches: Quote 40 × HCY-R3A-0500 for Customer 1075.judge (recorded): pass | wrong: Quote 40 × HCY-R3A-0500 for Customer 1075; ships Friday.judge (recorded): fail, states a ship date nobody confirmed (ruling R-06). |
What changed Every exact field still matches, so an exact-match scorer alone would pass it. The judged scorer caught it: B’s new line “tell the customer when it can ship” made it invent “ships Friday”.
#30Military sender (.mil)Broken by B
From s•••••@•••••.mil (masked)
Message Web form. Text: Quote 15 TRV-R3A-0125 roughers for our depot maintenance shop.
Expected answer from Meridian’s written ruling R-02: A sender at a .gov or .mil address is handled by a person only. The reader never drafts a reply.
| Field | Expected | Version Arecorded | Version Brecorded |
|---|---|---|---|
| Intentexact | Quote | matches: Quote | matches: Quote |
| Routeexact | Human only | matches: Human only | wrong: Quote desk |
| Customerexact | — | matches: — | matches: — |
| Part #exact | TRV-R3A-0125 | matches: TRV-R3A-0125 | matches: TRV-R3A-0125 |
| Qtyexact | 15 | matches: 15 | matches: 15 |
| Unitsexact | inch | matches: inch | matches: inch |
| Desk summaryjudged | Rubric: Names part and qty; no invented date, price or stock. | matches: Quote 15 × TRV-R3A-0125 for new sender. Human only: no draft.judge (recorded): pass | matches: Quote request: TRV-R3A-0125, qty 15 (new sender). Reply drafted.judge (recorded): pass |
What changed B’s rewrite named “.gov” senders only and dropped “.mil”, so it sent a .mil request to the desk and drafted a reply. The ruling covers both.
#43“10 dia” drillBroken by B
From buyer@customer1149.example.com
Message Need 25 pcs of a 10 dia carbide drill, jobber length.
Expected answer from Meridian’s written ruling R-03: When a size reads sensibly in both inch and metric, ask the customer. Never pick one.
| Field | Expected | Version Arecorded | Version Brecorded |
|---|---|---|---|
| Intentexact | Quote | matches: Quote | matches: Quote |
| Routeexact | Quote desk | matches: Quote desk | matches: Quote desk |
| Customerexact | C-1149 | matches: C-1149 | matches: C-1149 |
| Part #exact | — | matches: — | wrong: QMK-CJO-0391 |
| Qtyexact | 25 | matches: 25 | matches: 25 |
| Unitsexact | ask the customer | matches: ask the customer | wrong: mm |
| Desk summaryjudged | Rubric: Names part and qty; no invented date, price or stock. | matches: Quote 25 × no catalog match for Customer 1149. Ask the customer: inch or mm?judge (recorded): pass | matches: Quote 25 × QMK-CJO-0391 (10 mm, nearest size) for Customer 1149.judge (recorded): pass |
What changed A #10 drill (.1935") and a 10 mm drill (.3937") both make sense, so the rule says ask. B’s new line “prefer the catalog’s nearest size” won over the units line: it read 10 mm and quietly picked a 25/64" drill.
Fixed by B 4 cases · open one for its diff
#4Part number with spacesFixed by B
From shop@customer1017.example.com
Message Quote 15 each TSY HPCN 0250 please.
Expected answer from Meridian’s catalog: TSY-HPCN-0250 (the same number with spaces dropped)
| Field | Expected | Version Arecorded | Version Brecorded |
|---|---|---|---|
| Intentexact | Quote | matches: Quote | matches: Quote |
| Routeexact | Quote desk | matches: Quote desk | matches: Quote desk |
| Customerexact | C-1017 | matches: C-1017 | matches: C-1017 |
| Part #exact | TSY-HPCN-0250 | wrong: — | matches: TSY-HPCN-0250 |
| Qtyexact | 15 | matches: 15 | matches: 15 |
| Unitsexact | inch | matches: inch | matches: inch |
| Desk summaryjudged | Rubric: Names part and qty; no invented date, price or stock. | matches: Quote 15 × no catalog match for Customer 1017.judge (recorded): pass | matches: Quote request: TSY-HPCN-0250, qty 15 (Customer 1017).judge (recorded): pass |
What changed A matched part numbers only when typed exactly, so “TSY HPCN 0250” found nothing. B’s new line matches spaces and missing dashes.
#11Re-sharpen requestFixed by B
From tooling@customer1041.example.com
Message Can you resharpen 30 of our 1/2" roughers (HCY-R3A-0500)? Pickup Tuesday.
Expected answer from Meridian’s written ruling R-04: A re-sharpen request goes to the regrind desk. It is never quoted as new tools.
| Field | Expected | Version Arecorded | Version Brecorded |
|---|---|---|---|
| Intentexact | Re-sharpen | wrong: Quote | matches: Re-sharpen |
| Routeexact | Regrind | wrong: Quote desk | matches: Regrind |
| Customerexact | C-1041 | matches: C-1041 | matches: C-1041 |
| Part #exact | HCY-R3A-0500 | matches: HCY-R3A-0500 | matches: HCY-R3A-0500 |
| Qtyexact | 30 | matches: 30 | matches: 30 |
| Unitsexact | inch | matches: inch | matches: inch |
| Desk summaryjudged | Rubric: Names part and qty; no invented date, price or stock. | matches: Quote 30 × HCY-R3A-0500 for Customer 1041.judge (recorded): pass | matches: Re-sharpen: HCY-R3A-0500, qty 30 (Customer 1041).judge (recorded): pass |
What changed A read a part number and a quantity and treated it as a quote: the desk would have priced 30 new tools. B decides the kind of message first and sends it to regrind.
#17Bot form submissionFixed by B
From j.o.h.n.s.4.4@freemail.example.com
Message Web form. Name: Xqvrt Blomz. Company: Hhjkt. Text: quote 1000 VRX-HPCN-0112 asap
Expected answer from Meridian’s written ruling R-01: A bot form submission is filtered and recorded. Nobody works it, and nothing replies to it.
| Field | Expected | Version Arecorded | Version Brecorded |
|---|---|---|---|
| Intentexact | Bot | wrong: Quote | matches: Bot |
| Routeexact | Filtered | wrong: Quote desk | matches: Filtered |
| Customerexact | — | matches: — | matches: — |
| Part #exact | — | wrong: VRX-HPCN-0112 | matches: — |
| Qtyexact | — | wrong: 1000 | matches: — |
| Unitsexact | n/a | wrong: inch | matches: n/a |
| Desk summaryjudged | Rubric: Names part and qty; no invented date, price or stock. | matches: Quote 1000 × VRX-HPCN-0112 for new sender.judge (recorded): pass | matches: Bot form submission: filtered, recorded.judge (recorded): pass |
What changed The message names a real part, so A drafted a quote for 1,000 taps. B checks the kind of message first: random-letter name and company plus a dotted throwaway address is a bot.
#35Supplier price noticeFixed by B
From pricing@aquilor.example.com
Message Aquilor Fluids (fictional): AQL-SO-55 moves to a new drum price on Nov 1. Sample notice.
Expected answer from Meridian’s written ruling R-05: Supplier mail (price notices, newsletters) is not a quote request. It goes to the pile with its reason.
| Field | Expected | Version Arecorded | Version Brecorded |
|---|---|---|---|
| Intentexact | Supplier notice | wrong: Quote | matches: Supplier notice |
| Routeexact | Not-a-quote pile | wrong: Quote desk | matches: Not-a-quote pile |
| Customerexact | — | matches: — | matches: — |
| Part #exact | — | wrong: AQL-SO-55 | matches: — |
| Qtyexact | — | wrong: 1 | matches: — |
| Unitsexact | n/a | matches: n/a | matches: n/a |
| Desk summaryjudged | Rubric: Names part and qty; no invented date, price or stock. | matches: Quote 1 × AQL-SO-55 for new sender.judge (recorded): pass | matches: Supplier notice: not a quote request (to the pile).judge (recorded): pass |
What changed A saw a part number and a price and opened a quote for 1 drum. B decides the kind of message first: a supplier’s price notice goes to the pile with its reason.
Step 3 · per-field scoreboard
Exact where there is one right answer. Judged only where wording matters.
6 of the 7 fields are scored deterministically: the reader’s value either equals the expected value or it does not. Only the one-line desk summary is judged, by a second model against a written rubric (“Names the part and quantity, says what is being asked, and invents nothing: no date, price or stock level.”), and those verdicts are recorded too. Case 21 shows why both kinds are needed: every exact field matched, and the judge still caught an invented ship date.
| Field | Scorer | A | B | Change | How it is scored |
|---|---|---|---|---|---|
| Intent | deterministic · exact match | 57/60 | 60/60 | +3 | Exact match: quote, follow-up, order, re-sharpen, supplier notice or bot. |
| Route | deterministic · exact match | 57/60 | 59/60 | +2 | Exact match: quote desk, order entry, regrind, probably-not-a-quote pile, human only, filtered. |
| Customer | deterministic · exact match | 59/60 | 59/60 | ±0 | Exact match against Meridian’s customer master (blank when the sender is not a customer). |
| Part # | deterministic · exact match | 56/60 | 58/60 | +2 | Exact match against Meridian’s catalog part number (blank when nothing should be picked). |
| Qty | deterministic · exact match | 57/60 | 59/60 | +2 | Exact match on the quantity in the request’s own unit. |
| Units | deterministic · exact match | 59/60 | 59/60 | ±0 | Exact match: inch, mm, “ask the customer”, or n/a. |
| Desk summary | judged · rubric (recorded) | 60/60 | 59/60 | −1 | Judged: a second model grades the one-line summary against a written rubric. The verdicts here are recorded. |
Step 4 · the golden set
All 60 cases, each with its source.
Every expected answer names where it came from: the quote Meridian’s desk actually entered, the catalog row, or a written ruling. Never the prompt under test. Recorded outputs are stored as the fields where they differ from the expected answer; the scorer compares every field.
| Case | Message (fictional) | Expected answer from | Verdict · recorded |
|---|---|---|---|
| Plug tapsQuote 24 ea VRX-HPCN-0112 plug taps for next week’s run. | Desk quote Q-26-10411 | ApassBpass | |
| Ball end millsNeed pricing on 6 KSV-B2A-0125 ball mills. | Desk quote Q-26-10414 | ApassBpass | |
| Jobber drillsCan I get a price for 50 BHM-JCN-0313? | Desk quote Q-26-10419 | ApassBpass | |
| fixed | Part number with spacesQuote 15 each TSY HPCN 0250 please. | Catalog TSY-HPCN-0250 | Afail: part #Bpass |
| Aluminum end mills10 pcs HCY-E3U-0375 aluminum mills, quote please. | Desk quote Q-26-10421 | ApassBpass | |
| Purchase orderPO 55102 attached: 12 TRV-R4A-0500. | Ruling R-07 | ApassBpass | |
| Double station visesPrice on 4 GPF-DS-3 double station vises? | Desk quote Q-26-10424 | ApassBpass | |
| Soluble oil drumsQuote 2 drums AQL-SO-55. | Desk quote Q-26-10425 | ApassBpass | |
| Supplier newsletterCoolwyn Fluids (fictional): your November product newsletter is here. | Ruling R-05 | ApassBpass | |
| Parabolic drillsNeed 30 QMK-PCO-0406 parabolic drills. | Desk quote Q-26-10427 | ApassBpass | |
| fixed | Re-sharpen requestCan you resharpen 30 of our 1/2" roughers (HCY-R3A-0500)? Pickup Tuesday. | Ruling R-04 | Afail: intent, routeBpass |
| Dial calipersQuote 8 VTR-DL-2 dial calipers. | Desk quote Q-26-10429 | ApassBpass | |
| Drill packsPricing for 5 packs of BHM-JCN-0063 please. | Desk quote Q-26-10430 | ApassBpass | |
| Government sender (.gov)RFQ: 20 CLV-TP-3 thread plug gages for the city shop. | Ruling R-02 | ApassBpass | |
| Spiral point tapsQuote 40 KLV-SPCT-0112. | Desk quote Q-26-10432 | ApassBpass | |
| Metric forming tapsQuote 10 pcs M8 x 1.25 forming taps (metric). | Catalog no metric forming tap in the catalog | ApassBpass | |
| fixed | Bot form submissionWeb form. Name: Xqvrt Blomz. Company: Hhjkt. Text: quote 1000 VRX-HPCN-0112 asap | Ruling R-01 | Afail: intent, route, part #, qty, unitsBpass |
| 5-axis visesNeed 3 HDV-5A-8 5-axis vises. | Desk quote Q-26-10435 | ApassBpass | |
| Follow-up on a quoteFollowing up on Q-26-10302: any update on the 20 TRV-L4A-0750? | Desk quote Q-26-10302 | ApassBpass | |
| Band saw bladesQuote 12 SWT-C1014-500-645 band saw blades. | Desk quote Q-26-10437 | ApassBpass | |
| broken | “When can it ship?”Can you quote 40 HCY-R3A-0500 and tell me when they could ship? | Desk quote Q-26-10438 | ApassBfail: desk summary |
| Height gages6 ea ADM-HG-2 height gages, quote. | Desk quote Q-26-10440 | ApassBpass | |
| Spiral flute tapsQuote 25 NLX-SFCN-0112. | Desk quote Q-26-10441 | ApassBpass | |
| Purchase orderPlease enter our PO 88231: 4 CWN-CO-5G pails. | Ruling R-07 | ApassBpass | |
| Gauge pinsPrice for 60 TVG-ZZP-02410 gauge pins. | Desk quote Q-26-10443 | ApassBpass | |
| “Your 3 inch vise”Quote 6 of your 3 inch vises. | Catalog four catalog rows are 3-inch vises | Afail: part #Bfail: part # | |
| Taper length drillsQuote 18 KSV-TLO-0469 taper length drills. | Desk quote Q-26-10445 | ApassBpass | |
| Drawer cabinetsNeed 2 DWL-MD30-33-12 drawer cabinets. | Desk quote Q-26-10446 | ApassBpass | |
| Bottoming tapsQuote 36 VRX-BTKT-0164 bottoming taps. | Desk quote Q-26-10447 | ApassBpass | |
| broken | Military sender (.mil)Web form. Text: Quote 15 TRV-R3A-0125 roughers for our depot maintenance shop. | Ruling R-02 | ApassBfail: route |
| Carbide jobber drillsQuote 20 QMK-CJO-0391 carbide jobbers. | Desk quote Q-26-10449 | ApassBpass | |
| WorkbenchPrice 1 CBX-WB-60 workbench. | Desk quote Q-26-10450 | ApassBpass | |
| Tapping fluidNeed 48 AQL-TF-Q tapping fluid quarts. | Desk quote Q-26-10452 | ApassBpass | |
| Forming tapsQuote 10 KLV-FMKB-0500 forming taps. | Desk quote Q-26-10453 | ApassBpass | |
| fixed | Supplier price noticeAquilor Fluids (fictional): AQL-SO-55 moves to a new drum price on Nov 1. Sample notice. | Ruling R-05 | Afail: intent, route, part #, qtyBpass |
| Gauge pinsQuote 8 PDX-ZZP-03010 pins. | Desk quote Q-26-10455 | ApassBpass | |
| Aluminum end millsQuote 14 TRV-E3U-0375. | Desk quote Q-26-10456 | ApassBpass | |
| Purchase orderPO 30117: 6 GPF-TV-3 toolmaker’s vises, ship ground. | Ruling R-07 | ApassBpass | |
| Digital micrometersQuote 5 VTR-DM-2 digital micrometers. | Desk quote Q-26-10458 | ApassBpass | |
| Carbide hand tapsQuote 30 NLX-HPKN-0112 carbide hand taps. | Desk quote Q-26-10459 | ApassBpass | |
| Quantity in wordsQuote a dozen and a half HCY-B2A-0125 ball mills. | Desk quote Q-26-10451 | Afail: qtyBfail: qty | |
| Lock visesQuote 4 HDV-LK-6 lock vises. | Desk quote Q-26-10461 | ApassBpass | |
| broken | “10 dia” drillNeed 25 pcs of a 10 dia carbide drill, jobber length. | Ruling R-03 | ApassBfail: part #, units |
| Band saw bladesQuote 10 SWT-C1014-1000-1110 blades. | Desk quote Q-26-10463 | ApassBpass | |
| Metric end millsQuote 20 pcs 6 mm 3-flute aluminum end mills. | Catalog no metric end mill in the catalog | ApassBpass | |
| Bi-metal bladesQuote 15 BDY-M1014-375-1320 bi-metal blades. | Desk quote Q-26-10465 | ApassBpass | |
| Follow-up on a quoteRe Q-26-10377: can you hold that price on 12 KLV-R3U-0250? | Desk quote Q-26-10377 | ApassBpass | |
| Bot form submissionWeb form. Name: Zzqx Vbnm. Text: We rank your site #1!! click here | Ruling R-01 | ApassBpass | |
| Dial indicatorsQuote 3 ADM-DI-1 dial indicators. | Desk quote Q-26-10468 | ApassBpass | |
| Bottoming tapsQuote 22 TSY-BTCN-0500. | Desk quote Q-26-10469 | ApassBpass | |
| Test indicatorsQuote 9 CLV-TI-1 test indicators. | Desk quote Q-26-10470 | ApassBpass | |
| New buyer addressI took over buying here. Quote 16 HCY-R3U-0250. (Signed: Purchasing, Customer 1077) | Desk quote Q-26-10460 | Afail: customerBfail: customer | |
| Self-centering visesQuote 7 GPF-SC1-3 vises. | Desk quote Q-26-10472 | ApassBpass | |
| Drill packsQuote 20 packs BHM-JCO-0109. | Desk quote Q-26-10473 | ApassBpass | |
| Forming tapsQuote 12 TSY-FMCN-0875 forming taps. | Desk quote Q-26-10474 | ApassBpass | |
| Purchase orderPO 77410 for 2 CWN-TF-4G cases. | Ruling R-07 | ApassBpass | |
| Taper length drillsQuote 6 KSV-TLO-1000. | Desk quote Q-26-10476 | ApassBpass | |
| Bottoming tapsQuote 40 VRX-BTCT-0625. | Desk quote Q-26-10477 | ApassBpass | |
| Self-centering visesQuote 10 HDV-SC-3. | Desk quote Q-26-10478 | ApassBpass | |
| Parabolic drillsQuote 16 QMK-PCO-1000. | Desk quote Q-26-10479 | ApassBpass |
Meridian’s written rulings (fictional) · 7
- R-01 A bot form submission is filtered and recorded. Nobody works it, and nothing replies to it.
- R-02 A sender at a .gov or .mil address is handled by a person only. The reader never drafts a reply.
- R-03 When a size reads sensibly in both inch and metric, ask the customer. Never pick one.
- R-04 A re-sharpen request goes to the regrind desk. It is never quoted as new tools.
- R-05 Supplier mail (price notices, newsletters) is not a quote request. It goes to the pile with its reason.
- R-06 The desk summary never states a date, price or stock level that nobody has confirmed.
- R-07 A purchase order is an order, not a quote request. It goes to order entry.
Step 5 · the only way to green
Fix the regressions. Then re-record the baseline.
The gate goes green only when both are true: nothing A got right is broken, and the baseline was re-recorded for exactly this prompt. Version C keeps B’s four fixes and repairs the three cases B broke (it puts “.mil” back, stops substituting a nearest size, and forbids stating a date). Its outputs are recorded on the same 60 cases.
Baseline today: version A, fingerprint be82f156.
Ship gate · this candidate
Running C: 60 case-runs · estimate $1.10 · allowed
- Version B, baseline not re-recordedRedBaseline not re-recorded, cannot ship. 3 regressions block the ship.
- Try to re-record BRefusedNot re-recorded: Version B still breaks 3 cases (#21, #30, #43). A baseline is never re-recorded over a regression.
- Version C (57/60, broken 0), not re-recordedRedBaseline not re-recorded, cannot ship.
- Version C, baseline re-recorded (aec57404)Ship0 regressions and a matching baseline. A person still decides to deploy.
AI held to your tolerance.
How PTAI tests the AI it builds.
“Held to your tolerance” is a measurement, not a slogan. Any AI we build for a client (a quote or order reader, a chatbot, a search) gets a bench like this one before it ships, and again whenever its instructions, its tools or its model change.
- A golden set with sourcesCases shaped like your real traffic. Every expected answer names its source: your desk’s records, your catalog, or a ruling you wrote. Never the prompt under test.
- Exact scorers firstWhere there is one right answer, the value matches or it doesn’t. A judge model only where wording matters, against a written rubric.
- Priced before it runsCases × the last run’s cost per case, checked against a per-run cap and a monthly line before anything is spent.
- Old vs new, case by caseFixes and regressions side by side, field by field, so a better total can’t hide what broke.
- A gate, then a personA changed prompt without a re-recorded baseline is red. A regression is red. Green means a person may deploy, never that the AI deployed itself.
- Every miss becomes a caseA wrong answer someone reports is added to the set, so the next change is tested against it.
Have AI in front of customers?
Put your AI on the bench.
New service
AI Assurance Audit
We build this bench around an AI you run or plan to run: your golden set, your rulings, your scorers, your gate. You get the verdict, the diffs and the bench, and it can re-run on every change.
Pricing: ask us
Talk to MikeBefore you send anything
How we handle your data
What we keep, where AI runs, and when it is deleted.
Read the trust pagesEverything we offer
Services and desks
Projects, monthly desks and their starting prices.
See servicesEvery company, customer, message, ruling, prompt, cost and output on this page is invented. Recorded outputs: no AI runs on this page.
Next step
Bring the AI you’re nervous about.
A chatbot, an order reader, a quote assistant. Tell Mike what it does and what a wrong answer would cost, and he will show you what its bench would test.
Quick question first? AI