Docs / Run

Cluster Experimental

A Ghoti cluster elects one leader among several nodes, so a client can fail over to another node. It does not replicate data: when a node takes over as leader, it starts with every slot empty.

Clustering is experimental
  • Leader election uses the bully algorithm. There is no replication and no quorum.
  • Split-brain is possible under network partitions: several nodes can act as leader at once.
  • Heartbeat and election timeouts are fixed and can't be configured.
  • A node that fails to join doesn't retry.

Why a cluster#

A cluster exists for availability, not durability. Ghoti doesn't persist data, so there is no data worth replicating, and replication would cost performance. When the leader fails, another node takes its place with a clean state.

Two nodes is the suggested default, for two reasons:

If something goes really wrong, for example a failed deployment, the impact is a longer downtime. If the time it takes to bring up a replacement node is too long, add a third node.

Architecture#

Bully election#

Ghoti uses a bully election. The core rule: the reachable node with the highest node ID, compared as strings, wins.

  1. When a follower sees that the leader is unreachable (a failed heartbeat), it starts an election.
  2. It sends an election message to every peer with a higher ID.
  3. If any higher-ID peer responds, the node backs off and waits for that peer to declare itself leader.
  4. If no higher-ID peer responds, the node declares itself leader and sends a coordinator message to all peers.
  5. A node that receives a coordinator message accepts the sender as the new leader.
  6. A node that receives an election message from a lower-ID peer responds OK and starts its own election.

Components#

Inter-node communication#

Nodes talk over HTTP. Each node's joinServer listens on cluster.manager.addr and exposes:

EndpointMethodPurpose
/joinPOSTRegister a new node in the cluster
/removePOSTRemove a node from the cluster (leader only)
/electionPOSTBully election message: "I am starting an election"
/coordinatorPOSTLeader announcement: "I am the new leader"
/heartbeatGETHealth check used by followers to verify the leader is up
/leaderGETLeader check for external load balancers (opt-in)

Every endpoint except /heartbeat and /leader requires HTTP Basic Auth with the cluster credentials. Any other path returns 404. This traffic is not encrypted: see TLS.

Configuration#

The cluster section is optional. Without it, the server runs standalone.

KeyRequiredDescription
cluster.nodeYesUnique node ID, at most 20 characters. Decides election priority.
cluster.userYesUsername for inter-node authentication, at least 4 characters.
cluster.passYesPassword for inter-node authentication, at least 4 characters. Set it with GHOTI_CLUSTER_PASS to keep it out of the file.
cluster.manager.typeYesMembership manager. Only join_server is supported.
cluster.manager.addrYesAddress this node's cluster HTTP server listens on.
cluster.manager.joinNoManager address of an existing node to join. Leave it empty to bootstrap a new cluster.
cluster.leader.enabledNoServe GET /leader for load balancers. Default false.
cluster.bindNoParsed but not used by the server. It has no effect. Clients connect to each node's top-level addr.

Each node serves clients on its own top-level addr, like a standalone server.

Bootstrap node#

addr: localhost:9091
cluster:
  node: node1
  user: cluster_user
  pass: cluster_pass
  manager:
    type: join_server
    addr: localhost:2222

Without cluster.manager.join, the node bootstraps as leader of a new single-node cluster.

Joining node#

addr: localhost:9092
cluster:
  node: node2
  user: cluster_user
  pass: cluster_pass
  manager:
    type: join_server
    addr: localhost:2223
    join: localhost:2222

join points to the manager address of any existing member. On startup, the joining node sends a /join request and receives the current peer list and the leader's identity.

Routing traffic to the leader#

Only the leader serves client requests. Followers don't redirect or proxy. They answer every command with error 000 (NOT_LEADER), followed by the leader's node ID. The client has to connect to the leader itself:

# connected to node2 (follower)
> w000x
< exxx000node1

Over the http transport, a follower answers 503. The node ID is a name, not an address, so clients need their own mapping from node IDs to addresses.

Leader endpoint for load balancers#

To put a load balancer in front of the cluster, enable the leader endpoint on every node:

cluster:
  node: node1
  user: cluster_user
  pass: cluster_pass
  manager:
    type: join_server
    addr: "0.0.0.0:2222"
  leader:
    enabled: true

It serves GET /leader on cluster.manager.addr:

Node stateStatusBody
Leader200leader
Follower503not leader: <node ID>
Flag not set404Empty
$ curl -s localhost:2222/leader
leader
$ curl -s localhost:2223/leader
not leader: node1

Point the load balancer's health check at /leader. Every follower fails the check and drops out of rotation, so traffic converges on the leader and follows it after a failover.

Node IDs and election priority#

Node IDs are compared as strings, and the highest wins:

Use zero-padded IDs such as node01, node02 or node10 to get the order you expect.

Cluster lifecycle#

Starting a cluster#

  1. Start the first node without cluster.manager.join. It bootstraps as leader.
  2. Start the other nodes with cluster.manager.join pointing to an existing node's manager address.
  3. Each joining node receives the full peer list and the leader's identity.
  4. The existing node notifies every other peer about the new node.
A failed join is not reported

If the join request fails, for example because the bootstrap node isn't up yet, the node keeps running without logging an error. It never learns a leader and has no cluster HTTP server, so it answers every command with exxx000 followed by an empty leader ID and doesn't answer /leader. There is no retry: restart the node once the node it joins is reachable.

Heartbeat monitoring#

Followers check the leader every 2 seconds with a GET to its /heartbeat endpoint. A connection error or a non-200 response starts an election. The heartbeat request has a 2-second timeout, so a leader that is slow to answer also triggers an election.

Leader failover#

  1. One or more followers detect the failed heartbeat.
  2. Each of them starts a bully election and sends election messages to all peers with higher IDs.
  3. A follower that hears back from a higher-ID peer backs off.
  4. The highest-ID reachable node declares itself leader and broadcasts a coordinator message.
  5. Every node updates its leader. The new leader starts with every slot empty.

Graceful shutdown#

On shutdown a node marks itself down, so its heartbeat answers 503, closes its cluster server and waits for its background work to finish. If the leader shuts down, followers see the failed heartbeat and elect a new leader.

Failure scenarios#

Leader crashes#

Followers notice within about 2 to 4 seconds (one heartbeat interval plus the request timeout), and the highest-ID surviving node becomes leader. Until then, requests to the crashed leader fail, and followers answer NOT_LEADER with the old, unreachable leader's ID. Afterwards, clients can reconnect to the new leader. All data on the crashed leader is lost.

Follower crashes#

The cluster keeps working. The leader doesn't monitor followers, it only keeps them in its peer list. Redundancy is reduced: if the leader fails later, fewer nodes can take over. The crashed follower stays in the peer list until it is removed with /remove.

Network partition: full split#

No node can reach any other. Every follower sees the heartbeat fail and starts an election. Higher-ID peers are unreachable, so every node declares itself leader. This is split-brain: several leaders accept writes independently. When the network heals, the nodes still believe they are leaders until a heartbeat detects the conflict and triggers a new election, or until the cluster is reconfigured. Data written to the leaders that lose is gone.

Network partition: partial split#

Some nodes can reach each other but not the leader. They elect the highest-ID node of their partition, while the original leader keeps serving whoever can still reach it. Again several leaders can exist at once, and clients in different partitions write to different leaders. Once connectivity returns, the election eventually settles on the highest-ID node, but data written to the losing leader isn't transferred.

Clients reach all nodes, nodes can't reach each other#

The worst case: every follower elects itself, so every node is an independent leader, and clients can write to any of them. When inter-node traffic returns, elections settle on one leader, and the data on the others becomes unreachable. Clients should handle NOT_LEADER by reconnecting to the reported leader once the cluster is stable.

A node joins with a higher ID#

The new node joins as a follower and accepts the current leader, as told by the join response. It doesn't start an election. If the current leader fails later, the higher-ID node wins the election, as the bully algorithm expects.

A failed leader rejoins#

A crashed leader that rejoins with the same or a higher ID contacts an existing node through /join, receives the peer list and the current leader, and becomes a follower. It doesn't reclaim leadership because of its ID. If an election happens later, it wins it, and serves requests with an empty state.

All nodes restart at once#

The node without cluster.manager.join starts as leader. The others try to join through their configured address. If the bootstrap node isn't ready when they try, their join fails and, as described above, they keep running without a leader until they are restarted. Start the bootstrap node first. All data is lost.

Slow leader#

A leader that takes more than 2 seconds to answer a heartbeat looks dead to the followers, and they start an election. The highest-ID node wins, which may be the slow leader itself. The cluster then returns to the same state after a short period of instability, during which followers can report a stale leader.

Credential mismatch#

A node with a different cluster.user or cluster.pass is rejected with 400 on its join request and on any election or coordinator messages. It can't take part in the cluster.

Limitations#

Found something out of date? Open an issue or edit this page.