Determinism

Scope: the module overview already stated the determinism contract in one line — same starting state + same ordered log + same code ⇒ identical state on every node. This page goes one layer deeper on why that contract exists, the concrete ways ordinary-looking code violates it, and the one-shape fix that restores it.

TL;DR — in 30 seconds:

• Raft quorum only guarantees every node agrees on the same ordered log — determinism is the separate guarantee that replaying that log produces the same state everywhere, which recovery depends on.
• Three things break it: reading the wall clock, generating randomness, and external I/O — anything whose value can differ per node or per replay.
• The fix always has the same shape: compute the non-deterministic value once, outside the service, and carry it through the logged message so every replay reads the already-decided value.

1. Why determinism is load-bearing, not a nice-to-have

Quorum commit (the Cluster & Raft page) only guarantees that every node agrees on the same ordered log of messages. It says nothing about what each node's Clustered Service does with that log. Recovery (the Archiver & Recovery page) depends on a second guarantee entirely: that replaying the same log against the same starting state produces the same result, every time, on every node.

flowchart LR
  LOG["Same ordered log<br/>(guaranteed by Raft quorum)"] --> N1["Node 1 state"]
  LOG --> N2["Node 2 state"]
  LOG --> N3["Node 3 (recovering) state"]
  N1 -.->|"only equal if the service is deterministic"| N2
  N2 -.->|"only equal if the service is deterministic"| N3

Raft agreeing on the log is necessary but not sufficient — determinism is the other half of the contract that makes replicated state actually correct state, per the classic state-machine replication model.

2. What breaks it: anything the log doesn't capture

A Clustered Service breaks determinism whenever it lets a value into its state that two nodes — or the same node replaying twice — could compute differently. Three shapes show up repeatedly:

Non-deterministic source	Why it diverges	Example
Wall-clock time	each node's local clock reads a different instant, and a replaying node reads a later instant than the original run	System.currentTimeMillis() used inside message-processing logic
Randomness	an unseeded (or independently seeded) generator produces a different value per node/replay	generating an ID or sampling a value inside the handler
External I/O	a network call, file read, or any query to something outside the log can return a different answer on each node, or fail on one and not another	calling out to a pricing service, or reading a file, mid-handler

All three share the same underlying problem: the value came from outside the log, so nothing guarantees every node — or a later replay — sees the same value.

3. The one-shape fix: compute outside, log the result, replay the log

The fix is always the same shape, regardless of which of the three sources is involved: do the non-deterministic thing once, outside the deterministic replay path, then bake the resulting value into a message that goes through the log like any other. Every node — and every future replay — then reads the already-decided value instead of recomputing it.

flowchart TB
  subgraph WRONG["Non-deterministic — breaks replay"]
    H1["Clustered Service handler"] --> C1["Reads clock / random / network directly"]
    C1 --> S1["State — differs per node or per replay"]
  end
  subgraph RIGHT["Deterministic — safe to replay"]
    EXT["Computed once, outside the service<br/>(e.g. by the client, or a pre-processing step)"] --> MSG["Value carried in the logged message"]
    MSG --> H2["Clustered Service handler reads the value from the message"]
    H2 --> S2["State — identical on every node, every replay"]
  end

This is why Aeron Cluster gives the Clustered Service its own notion of "now," derived from a timestamp that travels through the log rather than each node's local wall clock, and why scheduling a future action from inside the service goes through a cluster timer — a timer request that itself becomes a logged, agreed-upon event — instead of an ordinary in-process timeout. Both are the same fix applied to the time-source case specifically: stop reading a value that can differ per node, and instead let the log carry it.

4. How this connects

• This is the property Cluster & Raft's commit path exists to protect. Quorum agreement guarantees every node sees the same log; determinism is what makes "same log in ⇒ same state out" actually hold, so agreement on the log translates into agreement on state.
• Recovery depends entirely on this. The Archiver & Recovery page's replay — load a snapshot, then replay the log entries after it — only reconstructs correct state if replaying those entries is guaranteed to reproduce what the original run produced. A non-deterministic handler makes replay silently rebuild the wrong state, with no error to signal it.
• The single-writer shape from Transport and Cluster & Raft is what makes this enforceable. Because only the leader appends to the log and every other node only ever reacts to that same ordered log, there is exactly one place — the point a value is written into the log — where "capture it once, log it, trust the log everywhere else" needs to happen.

Common gotchas

• Assuming Raft agreement alone guarantees correct replicated state. Fix: agreement on the log is necessary but not sufficient — determinism is the other half of the contract that makes "same log in" actually produce "same state out."
• Reading System.currentTimeMillis() (or any clock) inside a handler. Fix: capture the timestamp once outside the service (or via a cluster timer) and carry it through the logged message, since each node's local clock — and a later replay — reads a different instant.
• Scheduling a future action with an ordinary in-process timeout. Fix: use a cluster timer — a timer request that itself becomes a logged, agreed-upon event — so every node/replay sees the same scheduled time.
• Assuming a non-deterministic handler bug will surface as an error. Fix: it won't — replay silently rebuilds the wrong state with no error to signal it, which is why the constraint has to be enforced by discipline, not caught at runtime.

Check yourself — Raft guarantees every node agrees on the same ordered log. Why isn't that enough to guarantee every node ends up in the same state?

Agreement on the log says nothing about what each node's Clustered Service does with it — if the handler's logic can produce different results on different nodes (e.g. by reading the clock), the states diverge even though the log itself was identical.

Check yourself — name the three shapes of non-determinism that break the contract, and give an example of each.

Wall-clock time (e.g. System.currentTimeMillis() inside the handler), randomness (generating an ID or sampling a value inside the handler), and external I/O (a network call or file read mid-handler).

Check yourself — what's the one-shape fix, regardless of which of the three sources is involved?

Compute the non-deterministic value once, outside the deterministic replay path, then bake it into a logged message — every node and every future replay then reads the already-decided value instead of recomputing it.

Check yourself — why does Aeron Cluster provide a cluster timer instead of letting the service use an ordinary in-process timeout?

An ordinary timeout reads each node's local clock independently, which can differ per node/replay; a cluster timer turns the scheduled action into a logged, agreed-upon event, so every node sees the same decision.

You can defend this when you can state the determinism contract from memory; name the three shapes of non-determinism (clock, randomness, external I/O) and give a concrete example of each; explain the single-shape fix — compute outside, log the value, replay reads the log; and explain why a cluster timer or a log-carried timestamp exists instead of an ordinary in-process clock read.