Eval-gate

Prompt regression harness

Fixed case set contract · min = 0.90

Baseline prompt A · production

Candidate prompt B · under test

Gate invariant src/gate.ts

Throw: Eval gate failed

type Case = { input: string; check: (out: string) => boolean };

export async function gate(
  model: (p: string) => Promise<string>,
  min = 0.9
) {
  const passed = await Promise.all(
    cases.map(async (c) => c.check(await model(c.input)))
  );
  const score = passed.filter(Boolean).length / cases.length;
  if (score < min)
    throw new Error(`Eval gate failed: ${score.toFixed(2)}`);
  return score;
}

Tip: edit Candidate prompt B above (e.g. remove "avoid ... refund" or require bare classification labels) and click Execute gate to watch regressions clear.

Candidate gate status

Gate blocked

Error: Eval gate failed: 0.60

Candidate score (B)

0.60

6 / 10 cases passed (min 0.90)

Baseline score (A)

1.00

10 / 10 cases passed

Regression delta

4 regressed

Delta -0.40 · suite 1.2ms

Pass ratio vs. minimum gate threshold 60.0% / 90.0% required

Case ID	Delta state	Input prompt	Matcher assertion	Baseline (A) vs candidate (B) output	Act