Your new model scored higher.
Would you bet a deployment it didn’t get worse?
L3git Model-Eval is a promotion gate for AI models. It runs your current model and the candidate through the same seeded scenarios, compares behavior condition by condition — not on average — and returns a PASS/BLOCK verdict your pipeline enforces. It catches the model that’s better overall and quietly broken in fog, under jamming, on the rare slice that matters.
Python 3.11+ · Windows / macOS / Linux · zero dependencies · zero network calls · free while in beta
// HOW IT WORKS
// WHAT YOU GET
One-function integration
Wrap your checkpoint, simulator, or endpoint in run(scenario, seed). Three adapters: Python, any executable, HTTP. Everything else is one TOML file — no code changes to your model.
Statistics that won’t cry wolf
Every regression must pass two independent checks (confidence intervals + a paired permutation test) before it can block. Unsure? It says “get more data” instead of guessing. calibrate measures your noise floor and tells you the honest margins.
Never-events block instantly
Hard invariants — never engage a friendly, never confidently wrong — bypass statistics entirely: one violation in any run blocks promotion.
Evidence, not vibes
Deterministic, byte-identical reruns; self-contained HTML reports; versioned JSON results; the exact paired runs that regressed. Every report embeds the command that reproduces it.
Built for restricted environments
Pure Python standard library — zero dependencies, zero network calls unless you opt in. Ships with an assurance document written for security review. Air-gapped is a first-class citizen, not an afterthought.
CI-native + optional dashboard
Exit 0 promotes, exit 1 blocks — drops into any pipeline. Optionally push results to the dashboard for history and trends; nothing leaves your machines unless you say so.
// VALIDATED THREE WAYS
Head-to-head on ten promotion decisions with known right answers: aggregate-only checks made 3 wrong calls, threshold-only checks made 3. The gate made none.
Pointed at our own shipping self-healing framework, the gate blocked a plausible config change — and exposed a latent confidently-wrong failure mode we didn’t know we had. The fix it suggested passed with zero regressions across 600 paired runs.
Run against the most scrutinized public model benchmark (ProPublica’s COMPAS data), the gate’s per-cluster statistics reproduce the independently published error asymmetries and accuracy results.
Built for teams whose models act in the real world
Autonomy and simulation programs, perception and EW systems, ML platform teams, QA organizations adopting AI — anywhere a quiet regression on a rare condition is a safety event, not a KPI dip. If your work already carries a test and evaluation obligation — an assurance case to argue, a technical file to fill, a sponsor who wants the regression evidence rather than the headline metric — that is the shape of problem this was built for. The beta ships with autonomy/sim, classifier, and LLM-judge example gates: unzip, run l3git-model.bat init, and gate your first model this week.
// COMMON QUESTIONS
What is a model promotion gate?
A check that runs before a model is deployed. It compares the candidate against the model currently in production on the same seeded scenarios and returns a pass or block decision, so a regression is caught by the pipeline rather than in the field. Model-Eval returns a CI exit code: 0 promotes, 1 blocks.
How is this different from a benchmark or an eval harness?
A benchmark tells you a score. A gate tells you whether to ship. Model-Eval compares two specific model versions condition by condition rather than on an aggregate, because a model can improve on average while getting worse on a rare cluster that matters operationally. Every regression must pass two independent statistical checks before it can block — and when the evidence is thin it says so instead of guessing.
Can it run air-gapped or on a classified network?
Yes. Model-Eval runs locally with zero network calls and zero third-party dependencies. Nothing about your model, your scenarios, or your results leaves the machine unless you explicitly push results to a dashboard you control.
Does it produce evidence I can put in technical documentation?
Each run is stored as a structured JSON record: the verdict, per-cluster differences with confidence intervals, reasons and warnings, and the seeds used. Those records are designed to attach to an assurance case or a technical file. They are evidence you produce — not a compliance certification, and we don't claim one.
What kinds of models does it work with?
Anything you can call from a function. Three adapters — Python, any executable, and HTTP. You wrap your checkpoint, simulator, or endpoint in a run(scenario, seed) call and configure the rest in one TOML file, with no changes to the model itself. The beta ships with autonomy/simulation, classifier, and LLM-judge example gates.
Part of the L3git family — the same team behind L3git UI, API & Suite.