One leader per line

The physical worldMQTTSDKadvisory lockMachineGatewaystart({ machines })

Every replica runs the same config; one holds a Postgres advisory lock per line and runs its gateway, and the rest take over on loss.

The situation

Long Tail runs as several replicas behind a load balancer. Only one of them may talk to a given line’s machines at a time, or two gateways would dispatch the same job. When that replica dies, another must pick up without losing track of units in flight.

The recipe

Deploy the same declaration on every replica:

await start({
  database: { connectionString: process.env.DATABASE_URL },
  workers,
  machines: {
    lines: [Machines.loadLine('/etc/line-1.json'), Machines.loadLine('/etc/line-2.json')],
  },
});

For each line, one replica takes a Postgres advisory lock held on its own connection, makes sure the line’s bot accounts exist, and runs the gateway. The other replicas wait on the lock. Two lines can lead from two different replicas.

The guarantee

There is one leader per line at a time. When the leader’s session ends, a waiting replica takes the lock, and before it sends anything it reads back the line’s learned addresses, its open handoff, command, alarm, offline, quiet and registration rows, and each open handoff’s arrivals. It then reads the machines’ retained state. Pacing survives the takeover because a machine’s load is recomputed from those rows. See Machines.

The physical world