A factory line over MQTT

The moment

Second shift, 2:14 a.m. A cut cell jams with two units half through the fixture. On most lines, what happens next depends on who notices first. The machine raises a fault on its own screen. The MES marks the job late an hour later. Someone reconstructs the sequence from three logs the next morning.

On this line the jam is a row. The line operator’s tablet shows it within seconds: “U1, U2 stopped at cut-01: jam. Choose retry, manual, scrap.” She clears the fixture and taps retry. The route dispatches the hop again as a new job and carries on to bond, inspect and pack. In the morning the history sits on the rows, with the machine’s name on every step it took.

The line

Four stations: cut, bond and inspect are machines, and pack is a person. The line file declares them:

{
  "siteId": "site-1",
  "lineId": "line-1",
  "roles": { "exit": "packer", "operator": "line-operator", "receivers": ["bonder"] },
  "machines": [
    { "machineId": "cut-01", "process": "cut", "role": "cut-cell", "carrier": "none",
      "cycleSec": 6, "queueDepth": 4, "recipes": { "RCP-CUT-A": ["1"] }, "allowedSoftware": ["1.4.0"] },
    { "machineId": "bond-01", "process": "bond", "role": "bond-cell", "carrier": "none",
      "cycleSec": 8, "queueDepth": 4, "recipes": { "RCP-BOND-A": ["1", "2"] }, "allowedSoftware": ["1.4.0"] },
    { "machineId": "inspect-01", "process": "inspect", "role": "inspect-cell", "carrier": "none",
      "cycleSec": 4, "queueDepth": 4, "recipes": { "RCP-INSPECT-A": ["1"] }, "allowedSoftware": ["1.4.0"] }
  ]
}

Declare the line in start():

import { start, Machines } from '@hotmeshio/long-tail';

await start({
  database: { connectionString: process.env.DATABASE_URL },
  workers: [{ taskQueue: 'line', workflow: unitRoute }],
  machines: { lines: [Machines.loadLine('/etc/line.json')] },
});

One replica leads each line, holding a Postgres advisory lock. It makes sure every machine has a bot account and runs the gateway. If the leader goes away, another replica takes over and reads back every open row before it sends anything.

The intent: a handoff row

A route is a list of stations. Each hop parks one row on the receiving station’s role and waits for it. This is the hop from cut to bond:

const hop = await conditionalAccumulator(`route-${routeKey}-hop-2-a1`, {
  role: 'bond-cell',
  type: 'handoff',
  subtype: 'bond',
  description: 'U1, U2: cut-01 → bond-01',
  metadata: {
    siteId: 'site-1',
    lineId: 'line-1',
    units: [{ unitId: 'U1', carrierId: 'C-7' }, { unitId: 'U2', carrierId: 'C-8' }],
    jobId: `${routeKey}-bond-01`,
    senderJobId: `${routeKey}-cut-01`,
    fromMachine: 'cut-01',
    toMachine: 'bond-01',
    recipeId: 'RCP-BOND-A',
    recipeVersion: '2',
    process: 'bond',
    attempt: 1,
  },
  accumulate: {},
  timeout: '30m',
});

The row says what moves, from where, to where, and with which recipe. It does not say who resolves it. That depends only on the role.

Who resolves it

On a machine’s role, the gateway does what a person would, logged in as the machine:

1. It sees the new row and dispatches the job to bond-01 over MQTT.
2. When the machine accepts, it claims the row as bond-01.
3. It records the sender’s release of each unit, and each unit’s arrival, as items on the row.
4. Once every unit has arrived, it resolves the row as the receiver: { received: true, receiver: 'bond-01', units: ['U1', 'U2'] }.

On a person’s role, such as packer at the end of the route, someone resolves the same kind of row on the dashboard. A route can be all machines, all people, or any mix. The manual route cut-01 → bonder → inspect-01 → packer is the same workflow walking a different list of stations, so you can run a share of orders by hand on the same line.

Failures are decisions for a person

A hop that ends without every unit arriving resolves with received: false and an errorCode:

The platform reports the facts. The workflow chooses what to offer and asks the line operator:

const decisions = Machines.handoffDecisions(outcome.errorCode);
const answer = await conditional<Machines.OperatorResolution>(`route-${routeKey}-hop-2-error-1`, {
  role: 'line-operator',
  type: 'handoff-error',
  subtype: outcome.errorCode,
  description: `U1, U2 stopped at cut-01: ${outcome.errorCode}. Choose ${decisions.join(', ')}.`,
  priority: 1,
  metadata: { ...facets, errorCode: outcome.errorCode, decisions },
  timeout: '7d',
});

After a jam the units are still on the line, so retry, manual and scrap all fit. A retry is a new job with a new id, so nothing meant for the failed attempt reaches the machine later. Units the sender finished as rejects are off the line for good, so only manual and scrap fit. The workflow decides what each choice means for its own process.

Commands are rows too

To change over a machine, park a machine-command row on its role, the same way you would ask a person for anything:

const reply = await conditional<Machines.CommandResolution>(`changeover-${orderId}`, {
  ...Machines.commandRow(Machines.commandTarget(line, 'bond-01'), 'changeover', { recipe: { id: 'RCP-BOND-A', version: '2' } }),
  timeout: '10m',
});

The gateway sends the command and resolves the row with the reply, as the machine: accepted, refused with a reason, or no-reply. A command that never got through expires at the machine, so a late copy is refused instead of run. Asking a machine for a command takes the same access as parking any row on its role.

Rows for people

The line raises its own rows on the operator role when something needs a person:

Traceability by lot

When a sender finishes a unit, the gateway records the release and, in the same statement, merges two facets into the row: trace (each unit’s machine, job, result and materials) and lots (every lot those units consumed). A supplier recalls film lot LOT-7. Which rows used it is one query:

curl -G https://lt.acme.dev/api/escalations \
  -H "Authorization: Bearer $TOKEN" \
  --data-urlencode 'facets={"lots":["LOT-7"]}'

A unit’s whole path is every handoff row whose units holds it.

Related

• A machine is a resolver
• Handoff row
• Command a machine with a row
• Offline, quiet and alarm rows
• Lot traceability by containment
• Machines