Loop engineering: model regression

Eval Harnesses That Catch Model Errors Before Users Do

Swapping models without evals is how production breaks quietly. Intercom ships eval results within 48 hours of a new model and rolls out in days. That speed comes from a harness, not heroics.

LLM eval harness • AI regression testing • model eval framework enterprise

The problem in prod

New model versions change tone, refusal behavior and tool-call accuracy in ways unit tests never catch.
Offline vibes disagree with live A/B results, so launches become arguments instead of decisions.
Nobody owns the eval set, so it rots while the product moves on.

Loop engineering

Quality compounds when every failure feeds a named loop with an owner and a metric.

Offline gate loop

Benchmark every candidate model on transcripts of real support and ops interactions. Score instruction following, tool-call accuracy and brand voice. Nothing ships without beating the incumbent.

Live A/B loop

Roll winners to 5% of traffic with resolution rate and CSAT guardrails. Auto-rollback on regression, same as any deploy.

Dead-end loop

Queries the system cannot answer flow back into the knowledge base as new eval cases and doc gaps. CRED runs this to keep SOPs current in real time.

Cost check loop

Score cost per resolved task alongside quality. Intercom found GPT-4.1 beat expectations at 20% lower cost, which changed their architecture.

48 hrs

Intercom eval turnaround on new models

-20%

cost found by testing before assuming

98%

CRED resolution accuracy with eval gating

Production hardening checklist

Pin model versions in prod. Never float on latest without a gate run.
Separate evals for each locale and language, including Bahasa Malaysia.
Log every prompt, retrieval set and tool call for replay when incidents hit.
Review the eval set quarterly and retire cases the product outgrew.

Malaysia angle

Deploying this in MY

BNM-supervised teams need model validation evidence, not just accuracy claims. An eval harness doubles as the audit artifact: versioned sets, pass rates, reviewer sign-off, all exportable for supervisory review.

Malaysia FDE guide →

Want this running inside your company?

Get instant quotations and match in 2 days.

Source: OpenAI Intercom and CRED stories plus BNM model validation expectations

← RAG Works in Demo, Dies in Prod: 6 Failure Modes Human-in-the-Loop Design Regulators Actually Accept →