AI results with human review

The moment

A content team publishes product descriptions all day. A model reads each one before it goes live. Most of them are fine, and the model says so with a confidence of 0.97. Then one arrives half in the wrong language, with a claim the model cannot check, and it scores 0.62.

Nobody wants that one published unread. Nobody wants a person reading the 0.97s either. The usual answer is a status column, a side table, a channel where someone posts “can a human look at this”, and a script that pings it every few minutes. Each piece works. Together they drift.

The workflow already knows the one thing that matters: it needs a person’s judgment on this item before it continues. Long Tail lets it say exactly that.

The intent

import { Durable } from '@hotmeshio/hotmesh';
import { conditional, type LTEnvelope } from '@hotmeshio/long-tail';
import * as activities from './activities';

const { analyzeContent } = Durable.workflow.proxyActivities<typeof activities>({ activities });

type ReviewV1 = { decision: 'approve' | 'reject'; notes?: string };

export async function reviewContent(envelope: LTEnvelope) {
  const { contentId, content } = envelope.data;
  const analysis = await analyzeContent(content);

  if (analysis.confidence >= 0.85) {
    return { data: { contentId, approved: true, by: 'model', analysis } };
  }

  const { workflowId } = Durable.workflow.workflowInfo();
  const review = await conditional<ReviewV1>(`review-${workflowId}`, {
    role: 'reviewer',
    type: 'content-review',
    priority: 2,
    description: `Confidence ${analysis.confidence}: ${analysis.summary}`,
    metadata: { contentId, confidence: analysis.confidence },
    envelope: { content, analysis },
    timeout: '72h',
    schemaVersion: 1,
  });

  if (review === false) return { data: { contentId, approved: false, reason: 'no review within 72h' } };
  if (review === null) return { data: { contentId, cancelled: true } };
  return { data: { contentId, approved: review.decision === 'approve', notes: review.notes, by: 'reviewer' } };
}

analyzeContent is an activity: its result is checkpointed, so a crash after the model call never calls the model twice. The conditional call is the intent. It writes one escalation row inside the workflow’s own checkpoint and pauses.

Who resolves it, and where

A reviewer opens the reviewer queue on the dashboard and claims the item. The claim is a 30-minute lock with their name on it. Ninety seconds before it lapses, the dashboard offers to extend it. The page shows the content, the model’s summary and its flags, and a form rendered from the role’s schema.

The reviewer submits. The row resolves, and the workflow resumes in place with { decision: "approve", notes: "..." }. It does not re-run from the top.

The same row is open to every other surface, with the same rules:

The workflow cannot tell which one answered, and it does not need to. When it does want to know, the payload carries $resolution with the resolver’s identity.

The form is the contract

The reviewer role owns a versioned JSON Schema. The dashboard renders it, and the server can validate every submission against it.

{
  "title": "Content review",
  "required": ["decision"],
  "properties": {
    "decision": { "type": "string", "enum": ["approve", "reject"], "description": "Publish this content?" },
    "notes": { "type": "string", "format": "textarea", "description": "What you saw" }
  }
}

Each schema change adds an immutable snapshot. Version 2 might add a required category field. Ship schemaVersion: 2 together with the new payload type, and the rows already in flight keep the v1 form they were created with. A reviewer never sees a form the waiting code cannot read.

Turn on enforcement with enforce_schema: true on the role (PATCH /api/roles/reviewer, the update_role MCP tool, or the role editor). From then on, a payload that breaks the form is rejected before anything changes, with one body on every surface:

{
  "error": "resolverPayload failed schema validation (1 violation)",
  "code": "schema_validation",
  "violations": [{ "field": "decision", "message": "Must be one of: approve, reject" }],
  "role": "reviewer",
  "schemaVersion": 1
}

HTTP returns it as a 422. The SDK returns status: 422 with the same body. MCP returns it with isError: true. The CLI prints each violation. The dashboard maps it into the errors panel it already shows before submit, because the client and the server run the same validation code.

On the consuming side, parse the resolution through a zod schema with parseResolverPayload at the activity boundary. One schema gives you the type, the runtime check and the version pin.

What the workflow does with the answer

conditional returns one of three things, and the workflow handles each one in plain code:

Timeout and cancel are endings the workflow chose in advance. Here, an unreviewed item stays unpublished. Another team might route it to a lead with a second conditional, or publish with a warning label. It is a policy decision written in code, and you can see it in the code.

The AI handles more over time

Every settled row is a worked example. Its metadata holds the model’s confidence, set when the row was created. Its resolver payload holds the person’s decision. The row records who decided and when.

Resolvers that record an outcome facet make the decision queryable as well. The resolve call takes a metadata patch on HTTP, SDK and MCP:

await lt.escalations.resolve({
  id,
  resolverPayload: { decision: 'approve' },
  metadata: { disposition: 'approved' },
});

Then ask the queue how the model’s uncertain answers actually turned out:

const { data } = await lt.escalations.list({
  role: 'reviewer',
  status: 'resolved',
  facets: { disposition: 'approved' },
  range: [{ facet: 'confidence', op: '>=', value: 0.75 }],
  limit: 100,
});

If reviewers approve almost everything between 0.75 and 0.85, the threshold is too cautious. Lower it, and the model clears more items on its own. If they reject a cluster, the flags on those rows show what the model missed. Each adjustment shrinks the queue to the cases that are actually hard, and the reviewers spend their time on judgment instead of confirmation.

When a reviewer sees that the fix is mechanical, such as a page scanned upside down, they flag it for triage instead of fixing it by hand. An agent calls the remediation tools and re-runs the step. See human flags, AI remediates.

Related

• Confidence-gated escalation
• Pin the form edition per row
• One contract, one 422
• Answer, timeout or cancel
• Forms and schema enforcement