Human-in-the-Loop Governance for Production AI

Autonomous mapping and entity resolution are fast, but "fast and wrong" is how AI programs lose the trust of a risk committee. The answer is not to slow the automation down — it is to put humans exactly where ambiguity and accountability live, and nowhere else. That is what a human-in-the-loop (HITL) spine does: automation handles volume, experts resolve ambiguity.

The shared spine

Every accelerator — semantic mapping, entity resolution, consent tagging — runs through the same four-stage spine:

Automated Decision → Confidence Gate → SME Curation → Feedback Loop
   (score c ∈ [0,1])    (c ≥ 0.85 auto)   (validate/adjust)  (retrain)

• Automated decision — the model emits a result and a confidence score: an ER match score, a PII tag, a mapping proposal.
• Confidence gate — c ≥ 0.85 auto-approves; c < 0.85 locks the item and escalates.
• SME curation — data stewards and risk analysts validate or adjust the AI's proposal. They curate; they do not rebuild from scratch.
• Feedback loop — every correction retrains ER weights, policy weights, and FAISS embedding memory, lowering the next error rate.

The design principle: expert curation, not manual scratch work. Humans review AI proposals, and every correction sharpens the next decision.

One dashboard, five accelerators

The power of a shared spine is a single curation surface. One review queue, one decision schema, one feedback loop covers mapping proposals (Schema-Miner), entity matches (Entity Resolution), and policy events (Consent & PII):

ID	Source	AI proposal	Conf.	Status
SM-4821	Schema-Miner / Oracle	hasDepositAccountBalance	0.93	Auto-approved
SM-4822	Schema-Miner / SQL Server	CustomerSegment (alt: ClientSegment)	0.71	Awaiting SME
ER-1093	Entity Res / counterparty	Merge LEI 5493001 with 5493007	0.88	Awaiting SME
ER-1094	Entity Res / client dedup	Reject merge — address mismatch	0.62	Auto-rejected
PII-552	Consent / notes field	Classify as PII — free-text leakage	0.79	Locked

Every decision is logged to Unity Catalog and written back to the ER/FAISS weights. The steward's job is triage by domain and confidence — not to touch the 91% the model already got right.

HITL as an SLA, not a bottleneck

The most common failure is treating human review as an unbounded queue. It has to be operable — every gate carries an explicit service-level objective, error budget, and named owner:

Stage	KPI	Target	Owner
Automated decision	Throughput / inference latency	≥ 50k items/day · < 200 ms p95	ML Platform
Confidence gate	Auto-approval / false-accept rate	≥ 88% auto · < 0.5% false	Data Quality
SME curation	Queue aging p95 / decisions per steward	< 4 hours · ≥ 60/day	Risk & Stewards
Feedback loop	Retrain cadence / error-rate delta	nightly · ≥ 1%/month	MLOps

When the SME queue aging p95 approaches its 4-hour limit, that is a paged, measured event — not a vague sense that "review is backed up." Auto-approval false-accepts are tracked against a 0.5% budget; a single PII-leakage incident breaches budget and pages the risk officer.

Dimension-specific quality gates

The spine is shared, but the gates are tuned per accelerator:

• Entity resolution — auto-merge at p ≥ 0.95, review queue 0.70–0.95, auto-reject below 0.70; golden-record sign-off required for material counterparties.
• Consent & PII — every policy change routes to a risk officer before rollout; borderline PII tags (c < 0.85) lock the column until a steward confirms; weekly red-team prompts test for leakage.
• Semantic mapping — c < 0.85 locks the FIBO mapping for validation before commit; "no-mapping" results are sampled weekly to confirm prevention ≠ under-coverage.

The path to implement

1. Make every automated decision emit a confidence score, not just a result.
2. Set a confidence gate (c ≥ 0.85 is a sensible default) and route the rest to review.
3. Stand up one curation surface across accelerators — a single queue and decision schema.
4. Attach an SLA, error budget, and owner to every stage so review is measured, not open-ended.
5. Close the feedback loop — every correction retrains the model nightly.

This is what makes AI audit-defensible while preserving automation velocity. GovPilot provides the gates and audit trail; AI Harness keeps the retrain-and-deploy loop reliable in production.

See governance in the Lab →