Observability

Three registries cover what you need to run Long Tail in production: telemetry exports the spans HotMesh already creates, logging routes every internal log line through one adapter, and maintenance keeps Postgres tables bounded on a durable schedule. Each is one option in start().

index.ts

await start({
  database: { connectionString: process.env.DATABASE_URL },
  workers: [/* ... */],
  telemetry: { honeycomb: { apiKey: process.env.HONEYCOMB_API_KEY } },
  logging: { pino: { level: 'info' } },
  maintenance: true,
});

Telemetry

HotMesh creates OpenTelemetry spans for workflow triggers, activity calls, stream routing and errors. A telemetry adapter registers the global TracerProvider and exporter, and the spans flow. Workflow and activity code creates no spans of its own.

Honeycomb

Option	Default	Use
apiKey	process.env.HONEYCOMB_API_KEY	Without one, connect() logs a warning and starts nothing
serviceName	'long-tail'	The service.name on every span
endpoint	'https://api.honeycomb.io'	Override for Honeycomb EU or a local collector

The adapter uses @opentelemetry/sdk-node with the OTLP proto exporter and sends to {endpoint}/v1/traces with the key in x-honeycomb-team.

Any other backend

Implement two methods:

telemetry/datadog.ts

import { NodeSDK } from '@opentelemetry/sdk-node';
import type { LTTelemetryAdapter } from '@hotmeshio/long-tail';

export class DatadogAdapter implements LTTelemetryAdapter {
  private sdk: NodeSDK | null = null;

  async connect() {
    this.sdk = new NodeSDK({ /* exporter config */ });
    this.sdk.start();
  }

  async disconnect() {
    await this.sdk?.shutdown();
  }
}

Pass it as telemetry: { adapter: new DatadogAdapter() }. The registry holds one adapter, because OpenTelemetry allows one global TracerProvider.

Caution

The adapter must connect before workers start, because HotMesh creates its tracers when a worker initializes. start() handles the order. If you register with telemetryRegistry yourself, call await telemetryRegistry.connect() before starting workers.

Span verbosity

HMSH_TELEMETRY sets which spans HotMesh creates, independent of where they go. Set it on the containers that run workers.

Value	Spans
info	Workflow triggers, worker lifecycle, errors
debug	Everything in info, plus every activity execution
unset	None

To put trace_id and span_id on every log line, add @opentelemetry/instrumentation-pino to the NodeSDK inside your adapter’s connect().

Logging

Every internal log line goes through loggerRegistry. Register Pino, Winston, Bunyan or anything with four methods:

export interface LTLoggerAdapter {
  info(msg: string, context?: Record<string, any>): void;
  warn(msg: string, context?: Record<string, any>): void;
  error(msg: string, context?: Record<string, any>): void;
  debug(msg: string, context?: Record<string, any>): void;
}

Pino

await start({
  database: { connectionString: process.env.DATABASE_URL },
  logging: { pino: { level: 'info' } },
});

logging.pino takes Pino’s standard LoggerOptions: level, transports, redaction, serializers. Output is one JSON object per line. To register it yourself:

import { loggerRegistry, PinoLoggerAdapter } from '@hotmeshio/long-tail';

loggerRegistry.register(new PinoLoggerAdapter({
  level: 'info',
  transport: { target: 'pino-pretty' },
}));

Winston

import winston from 'winston';
import type { LTLoggerAdapter } from '@hotmeshio/long-tail';

export class WinstonAdapter implements LTLoggerAdapter {
  private logger = winston.createLogger({ /* ... */ });
  info(msg: string, context?: Record<string, any>) { this.logger.info(msg, context); }
  warn(msg: string, context?: Record<string, any>) { this.logger.warn(msg, context); }
  error(msg: string, context?: Record<string, any>) { this.logger.error(msg, context); }
  debug(msg: string, context?: Record<string, any>) { this.logger.debug(msg, context); }
}

await start({
  database: { connectionString: process.env.DATABASE_URL },
  logging: { adapter: new WinstonAdapter() },
});

The logging adapter registers before anything else, so startup messages flow through it. With no adapter, info and debug go to console.log, warn to console.warn and error to console.error.

Maintenance

HotMesh keeps stream messages, transient job rows and workflow rows in Postgres. Maintenance ages them out on a schedule that runs as a durable cron inside HotMesh, so a missed tick runs when a container comes back.

maintenance	Effect
omitted or false	No maintenance
true	The default: Sundays at 2 AM, five rules
an LTMaintenanceConfig	Your schedule and rules in place of the default

index.ts

await start({
  database: { connectionString: process.env.DATABASE_URL },
  workers: [/* ... */],
  maintenance: {
    appId: 'durable',
    schedule: '0 3 * * *',
    rules: [
      { target: 'streams',     action: 'delete', olderThan: '24 hours' },
      { target: 'jobs',        action: 'delete', olderThan: '14 days', hasEntity: false },
      { target: 'jobs',        action: 'prune',  olderThan: '14 days', hasEntity: true },
      { target: 'jobs',        action: 'delete', olderThan: '180 days', pruned: true },
      { target: 'escalations', action: 'delete', olderThan: '180 days' },
      { target: 'oauth',       action: 'delete', olderThan: '7 days' },
    ],
  },
});

What each target cleans

Target	Rows
streams	Internal messages HotMesh uses to coordinate activities
jobs with hasEntity: false	Transient bookkeeping rows (entity IS NULL)
jobs with hasEntity: true	Workflow instances. prune strips execution scaffolding and keeps the core fields, so search and export still work. delete removes the row.
jobs with pruned: true	Workflow instances already pruned, for final deletion
escalations	Terminal rows (resolved, cancelled, expired, or the statuses you list), deleted in atomic batches. Pending rows are never touched.
oauth	Expired or rotated codes and refresh tokens, dead grants, and clients nobody consented to

The intended life of a workflow row: run, prune after a short window, delete after a long one. Rules run in order. A failed rule is logged and the rest still run.

Default rules

{
  appId: 'durable',
  schedule: '0 2 * * 0',
  rules: [
    { target: 'streams', action: 'delete', olderThan: '30 days' },
    { target: 'jobs', action: 'delete', olderThan: '30 days', hasEntity: false },
    { target: 'jobs', action: 'prune',  olderThan: '30 days', hasEntity: true },
    { target: 'jobs', action: 'delete', olderThan: '180 days', pruned: true },
    { target: 'oauth', action: 'delete', olderThan: '7 days' },
  ],
}

olderThan is a Postgres interval ('7 days', '24 hours'). schedule is a cron expression or an interval string. Maintenance starts with the workers and does not run on a readonly instance.

Change rules at runtime

curl https://lt.acme.dev/api/config/maintenance -H "Authorization: Bearer $LT_TOKEN"

curl -X PUT https://lt.acme.dev/api/config/maintenance \
  -H "Authorization: Bearer $LT_TOKEN" -H 'Content-Type: application/json' \
  -d '{
    "appId": "durable",
    "schedule": "0 4 * * *",
    "rules": [
      { "target": "streams", "action": "delete", "olderThan": "3 days" },
      { "target": "jobs", "action": "prune", "olderThan": "3 days", "hasEntity": true }
    ]
  }'

PUT is admin-only and returns { "restarted": true } with the new config, or 400 when appId, schedule or rules is missing. The new rules apply to ticks this container runs and live in memory: other containers keep their rules, and a restart returns to the start() config. An existing cron job keeps the schedule it was created with.

Related

Data modelThe tables maintenance ages out.

PostgresSizing and tuning the database.

The row is the production log

Run modesWhich containers run workers.