22. Durable Workflows

Durable workflows let you run long-lived, restart-safe business processes with persisted state, replay-safe step execution, and external control points such as approvals and signals.

For short fire-and-forget work (send email, call a webhook), prefer the lightweight job queue builtins (enqueueJob / claimJob / completeJob / failJob) documented in Built-in Functions. Jobs use ./.malda/jobs.db and are not workflow instances.

22.1 Core Concepts

• Durability: workflow instances, steps, and events are persisted to a SQLite database (see 22.8 Storage and Persistence).
• Determinism: replay avoids duplicate successful step execution.
• Recovery: stale running work is reconciled on restart and can resume safely.
• Control: approval/signal flows can be driven from CLI or built-ins.
• Operations: unified malda workflow report, DLQ, requeue, metrics, and maintenance commands are available.

22.2 Workflow Syntax

A workflow body is made of ordinary statements plus three durable ones: step for work that must not run twice, approval for a human decision, and wait for an external signal. Each binds its result to a name the rest of the body can read, and each is journaled, which is what lets an instance resume after a restart without repeating work already done.

Their shapes differ. step takes any call on the right of the =, so the work itself is an ordinary function. approval and wait instead have a fixed right-hand side: an approval statement must call approval(...) and a wait statement must call awaitSignal(...). Those two names are recognised in that position only — awaitSignal is not a reserved word and cannot be called anywhere else — so var x = awaitSignal("s"); is not a signal wait, and anything else after the = is a parse error.

workflow OnboardCustomer(input) {
    step validated = validateInput(input)
        retry 3 backoff "exponential" delay 1000 maxDelay 30000
        timeout 120000;

    approval approved = approval("sales-manager", {"customerId": input.customerId})
        timeout 86400000
        onReject notifyRejected(input.customerId);

    wait docs = awaitSignal("docs_uploaded", {"customerId": input.customerId})
        timeout 259200000;

    step account = createAccount(validated)
        retry 2 backoff "linear" delay 1000
        compensate deleteAccount(account.id);

    return {"accountId": account.id, "status": "onboarded"};
}

Branches and Loops

A workflow body is not restricted to a straight line. if, while and for are allowed and may contain step, approval and wait, so a branch can take an approval that another branch skips. Step identifiers are unique across the whole workflow rather than per block, so two branches cannot reuse one name. Branch on step results, not on the clock: the condition sits outside any step and is therefore re-evaluated on every replay, which is why now() and random() are refused there (see Determinism Boundary below).

Step Clauses

Clauses follow the call in any order and end at the semicolon. Every numeric clause takes an integer literal in milliseconds — a variable or an expression there is a parse error — and compensate takes a call.

How Retries Are Timed

Under "fixed" every wait is delay. Under "linear" the wait before retry n is delay × n. Under "exponential" it doubles: delay × 2n-1. Linear and exponential are then clamped to maxDelay. A jitter factor derived from the instance, step name and attempt number is applied last, so parallel instances do not retry in lockstep while any single instance still computes the same waits when replayed.

A step that outruns its timeout is recorded as TIMED_OUT and counts as a retryable attempt; when the attempts are exhausted the workflow fails with Step '...' timed out. The timeout ends the workflow's wait, it does not cancel work already in flight, so a step calling a slow service may still finish in the background after the workflow has moved on.

Compensation

compensate registers an undo action, and registration happens only once the step has succeeded — steps that failed, timed out or never ran have nothing to undo. What triggers the undo is an error escaping the workflow body after that point, not the failure of the step carrying the clause.

Compensations then run in reverse order, most recent first. Each is journaled as a step of its own named <step>__compensate and gets a single attempt: retry clauses belong to the step, not to its compensation. A compensation that throws does not stop the others; it is recorded as COMPENSATION_FAILED and collected into a summary. If every compensation succeeds the instance ends COMPENSATED, otherwise FAILED with the failures listed in its error payload. An instance with no registered compensations simply ends FAILED. Either way the original error still reaches the caller, and compensations already recorded as COMPENSATED are skipped if the instance is resumed.

Approvals and Signals

approval and wait are pauses, not blocking calls. Reaching one for the first time records the wait, moves the instance to WAITING_APPROVAL or WAITING_SIGNAL, and suspends it — the process does not sit and spin. The decision arrives later from approveWorkflowStep(...) or signalWorkflow(...), or from malda workflow approve / malda workflow signal, and the body then runs again from the top with completed steps replayed from the journal rather than re-executed.

On that second pass the name bound by approval holds the recorded decision, and the name bound by wait holds the signal payload. A rejection runs the onReject call when one is present; without it, rejection raises Approval '...' was rejected. That clause takes any call your program can make — there is no built-in rejection helper, so notifyRejected above is a function you write — and it runs under step rules, which means side effects are permitted inside it. The second argument is optional in both statements: for an approval it is the payload shown to the approver, for a signal it is correlation data. Their timeout is measured from the moment the wait was recorded, so it keeps running across restarts, and an expired wait raises a timeout error on the next pass.

Determinism Boundary

Statements in a workflow body outside a step may be replayed any number of times, so the runtime refuses the calls that would make replay lie. now, random, randomInt, randomFloat, randomChoiceWeighted, randn and sleep raise WF1001 there, and the filesystem, process and HTTP built-ins (writeFile, deleteFile, editFile, replaceInFile, runCommand, runMALDA, compileMALDA, httpGet, httpPost, httpPut, httpPatch, httpDelete) raise WF1002. Those codes fire whenever the deny-listed built-in runs in a deterministic section — including when it is nested in a helper. The IDE/LSP reports the same codes on direct calls and on same-file function callees (bounded depth) outside a step (and outside onReject). Imported or unknown callees are WF1005 Info, not a hard error; the runtime still raises WF1001/WF1002 if a deny-listed built-in runs. Inside a step all of them are allowed, because the step's result is journaled and the call is not repeated on replay. This is a fixed deny-list plus a same-file call-graph, not a Temporal-style full-history non-determinism detector: other effects (model calls, SQL, .NET interop) outside a step are not caught. Durability is a single local SQLite file (override with MALDA_WORKFLOW_CONNECTION) — restart recovery on one box, not cluster failover. Step outputs must be JSON-shaped so they can be memoized. Example: Examples/Workflows/determinism_helpers.malda.

22.3 Lifecycle and States

Common workflow states include:

PENDING, RUNNING, WAITING_APPROVAL, WAITING_SIGNAL, COMPENSATING,
COMPLETED, FAILED, CANCELLED, COMPENSATED

Common step states include:

PENDING, RUNNING, SUCCEEDED, FAILED, TIMED_OUT, SKIPPED,
COMPENSATING, COMPENSATED, COMPENSATION_FAILED

22.4 Runtime Operations (Built-ins)

var id = startWorkflow("OnboardCustomer", {"customerId": 101, "correlationId": "corr-101"});
var status = getWorkflowStatus(id);
var wf = getWorkflow(id);
var steps = getWorkflowSteps(id);
var events = getWorkflowEvents(id, 200);
var metrics = getWorkflowMetrics();
var rows = listWorkflows("FAILED", null, 50);

var okCancel = cancelWorkflow(id, "operator requested");
var okResume = resumeWorkflow(id);
var okRetry = retryWorkflow(id);

var okApprove = approveWorkflowStep(id, "approved", "approve", {"actor": "manager"});
var okSignal = signalWorkflow(id, "docs_uploaded", {"customerId": 101});

22.5 Dead Letters and Requeue

var rows = listWorkflowDeadLetters(100, false);   // false => pending only
if (length(rows) > 0) {
    var requeued = requeueDeadLetter(
        rows[0].id,
        "operator replay",
        "oncall",
        "corr-requeue-101"
    );
    print(requeued);
}

Dead-letter rows are created on terminal failure paths and can be requeued with audit metadata.

runWorkflowInstance(instanceId) drives an existing instance forward. It is async only: call it with await. Calling it synchronously raises an error.

var advanced = await runWorkflowInstance(id);
print(advanced);   // true when the instance ran

• null — no instance with that id exists.
• false — the instance is in a status that cannot be advanced, or an internal retry/resume attempt failed.
• true — the instance ran.

Runnable statuses are RUNNING, FAILED (retried first), WAITING_APPROVAL, and WAITING_SIGNAL (resumed first). If the workflow itself throws during execution, the instance is marked failed and the error is rethrown to the caller, so wrap the call in try/catch when driving instances in a loop.

22.6 CLI Commands

# Core lifecycle
malda workflow start workflow.malda OnboardCustomer --input input.json
malda workflow list --status FAILED --limit 50
malda workflow get <instanceId>
malda workflow steps <instanceId>
malda workflow events <instanceId> --limit 200
malda workflow report <instanceId> --events-limit 200
malda workflow metrics
malda workflow cancel <instanceId> --reason "operator cancel"
malda workflow resume <instanceId>
malda workflow retry <instanceId>

# Human-in-the-loop
malda workflow approve <instanceId> <stepId> --decision approve --payload approval.json
malda workflow signal <instanceId> docs_uploaded --payload signal.json

# DLQ and maintenance
malda workflow dlq list --pending-only --limit 100
malda workflow dlq requeue <deadLetterId> --reason "operator replay" --by "oncall" --correlation-id "corr-123"
malda workflow maintenance run --operational-days 30 --audit-days 180 --compaction-days 14 --batch 500
malda workflow maintenance run --dry-run --format json

For automation, use --json or --format json.

Reading an ops report

malda workflow report <instanceId> returns one operator-facing document: instance status, persisted step attempts, journaled timeline events, and dead-letter rows for that instance (including requeue audit when present). Human mode prints four sections; --json emits a single object with camelCase keys instance, steps, events, deadLetters, generatedAtUtc, and eventLimit.

The timeline is the append-only event journal (what happened on this box), not a Temporal-style replay debugger. Pending DLQ rows for the instance appear under Dead letters; use malda workflow dlq requeue to act on them. Seed a FAILED instance with Examples/Workflows/ops_report.malda, then run malda workflow report against the printed instanceId.

22.7 Reliability and Parity Notes

• Retry supports fixed/linear/exponential backoff with caps and deterministic jitter.
• Timeout and retry transitions are persisted and emitted as workflow events.
• Interpreter and transpiled modes are validated with parity tests for core scenarios.
• State is stored in SQLite; see 22.8 Storage and Persistence for the database location.

22.8 Storage and Persistence

Every durable workflow writes to one SQLite database file. By default:

./.malda/workflows.db

The .malda directory is created on first use, and the schema is created and upgraded automatically when the engine opens the connection. There is no separate install or migration step, and nothing to provision before malda workflow start.

On open, the runtime sets SQLite journal_mode=WAL and busy_timeout=5000 so a second process can run read-only ops (list, get, report, …) against the same database while a writer is active. See 22.10 High Availability and Multi-Worker.

What Is Stored

Payloads are stored as JSON text. Since inputs, outputs, and approval payloads are persisted verbatim, treat the database as sensitive: avoid passing secrets through step arguments, and keep MALDA_WORKFLOW_MAX_PAYLOAD_BYTES in mind for large values.

Choosing a Different Location

MALDA_WORKFLOW_CONNECTION takes a full SQLite connection string, which lets you pin the database to a fixed location:

# Linux and macOS
export MALDA_WORKFLOW_CONNECTION="Data Source=/var/lib/malda/workflows.db"

# Windows
setx MALDA_WORKFLOW_CONNECTION "Data Source=C:\ProgramData\malda\workflows.db"

Directories in an absolute path must already exist; only the default ./.malda is created automatically.

MALDA_WORKFLOW_PROVIDER selects the backend and currently accepts only sqlite. Any other value fails fast when the engine first touches storage:

Unsupported workflow storage provider 'postgres'. Supported providers: sqlite.

The runtime does talk to storage through an interface (IWorkflowStorageProvider), so additional backends can be added without touching workflow semantics, but none ship today.

Keeping the Database Small

malda workflow maintenance run is the only thing that deletes data; the engine itself never prunes. It works in three tiers, each with its own retention window:

• Compaction clears input, output, and error payloads from steps of finished instances, keeping the rows and their states.
• Archival moves completed and compensated instances into workflow_instance_archive and drops their steps, events, and dead letters.
• Audit trimming deletes events and dead letters older than the audit window.

Run it with --dry-run first: it reports exactly what each tier would touch, inside a transaction that is rolled back.

22.9 Configuration Guardrails

All limits are read from the environment when the workflow engine is first used. A value that is missing or cannot be parsed leaves the default in place, silently.

Retention values are validated against each other, so an inconsistent combination (for example an audit window shorter than the operational one) is rejected instead of quietly discarding history.

22.10 High Availability and Multi-Worker

v1 supports one writer process per database file, plus an optional second process for read-only CLI ops against the same absolute MALDA_WORKFLOW_CONNECTION. Two mutating processes on one file are unsupported.

• Locking: process singleton + SQLite file lock; WAL for concurrent readers. Stale RUNNING recovery after restart is not a worker ownership lease.
• Limits: local SQLite durability on one machine — not a cluster, not Temporal-style task queues or multi-machine failover.
• Migration: storage goes through IWorkflowStorageProvider; a future multi-writer backend would need claim/lease and conditional status updates.

Full failure-mode matrix and migration notes: docs/workflows-ha.md in the repository.