Your new model scored higher.
Would you bet a deployment it didn’t get worse?
L3git Model-Eval is a promotion gate for AI models. It runs your current model and the candidate through the same seeded scenarios, compares behavior condition by condition — not on average — and returns a PASS/BLOCK verdict your pipeline enforces. It catches the model that’s better overall and quietly broken in fog, under jamming, on the rare slice that matters.
Python 3.11+ · Windows / macOS / Linux · zero dependencies · zero network calls · free while in beta
// HOW IT WORKS
// WHAT YOU GET
One-function integration
Wrap your checkpoint, simulator, or endpoint in run(scenario, seed). Three adapters: Python, any executable, HTTP. Everything else is one TOML file — no code changes to your model.
Statistics that won’t cry wolf
Every regression must pass two independent checks (confidence intervals + a paired permutation test) before it can block. Unsure? It says “get more data” instead of guessing. calibrate measures your noise floor and tells you the honest margins.
Never-events block instantly
Hard invariants — never engage a friendly, never confidently wrong — bypass statistics entirely: one violation in any run blocks promotion.
Evidence, not vibes
Deterministic, byte-identical reruns; self-contained HTML reports; versioned JSON results; the exact paired runs that regressed. Every report embeds the command that reproduces it.
Built for restricted environments
Pure Python standard library — zero dependencies, zero network calls unless you opt in. Ships with an assurance document written for security review. Air-gapped is a first-class citizen, not an afterthought.
CI-native + optional dashboard
Exit 0 promotes, exit 1 blocks — drops into any pipeline. Optionally push results to the dashboard for history and trends; nothing leaves your machines unless you say so.
// VALIDATED THREE WAYS
Head-to-head on ten promotion decisions with known right answers: aggregate-only checks made 3 wrong calls, threshold-only checks made 3. The gate made none.
Pointed at our own shipping self-healing framework, the gate blocked a plausible config change — and exposed a latent confidently-wrong failure mode we didn’t know we had. The fix it suggested passed with zero regressions across 600 paired runs.
Run against the most scrutinized public model benchmark (ProPublica’s COMPAS data), the gate’s per-cluster statistics reproduce the independently published error asymmetries and accuracy results.
Built for teams whose models act in the real world
Autonomy and simulation programs, perception and EW systems, ML platform teams, QA organizations adopting AI — anywhere a quiet regression on a rare condition is a safety event, not a KPI dip. The beta ships with autonomy/sim, classifier, and LLM-judge example gates: unzip, run l3git-model.bat init, and gate your first model this week.
Part of the L3git family — the same team behind L3git UI, API & Suite.