Cluster Experimental
A Ghoti cluster elects one leader among several nodes, so a client can fail over to another node. It does not replicate data: when a node takes over as leader, it starts with every slot empty.
- Leader election uses the bully algorithm. There is no replication and no quorum.
- Split-brain is possible under network partitions: several nodes can act as leader at once.
- Heartbeat and election timeouts are fixed and can't be configured.
- A node that fails to join doesn't retry.
Why a cluster#
A cluster exists for availability, not durability. Ghoti doesn't persist data, so there is no data worth replicating, and replication would cost performance. When the leader fails, another node takes its place with a clean state.
Two nodes is the suggested default, for two reasons:
- Deployments with minimal downtime. Replace one node, let it become leader, then replace the other.
- Failover. If one node has a problem, the other can take over.
If something goes really wrong, for example a failed deployment, the impact is a longer downtime. If the time it takes to bring up a replacement node is too long, add a third node.
Architecture#
Bully election#
Ghoti uses a bully election. The core rule: the reachable node with the highest node ID, compared as strings, wins.
- When a follower sees that the leader is unreachable (a failed heartbeat), it starts an election.
- It sends an election message to every peer with a higher ID.
- If any higher-ID peer responds, the node backs off and waits for that peer to declare itself leader.
- If no higher-ID peer responds, the node declares itself leader and sends a coordinator message to all peers.
- A node that receives a coordinator message accepts the sender as the new leader.
- A node that receives an election message from a lower-ID peer responds OK and starts its own election.
Components#
Clusterinterface:Start,Join,Remove,IsLeader,GetLeader,Shutdown.BullyCluster: the implementation, with election logic, peer tracking, heartbeat monitoring and election coordination.MembershipManager: the abstraction for cluster membership. The only implementation isjoinServer.joinServer: an HTTP server on each node that handles join requests, node removal, election messages, coordinator announcements and heartbeats.EmptyCluster: a no-op used when clustering is disabled. It always reports itself as leader, so a standalone server serves every request.
Inter-node communication#
Nodes talk over HTTP. Each node's joinServer listens on cluster.manager.addr and exposes:
| Endpoint | Method | Purpose |
|---|---|---|
/join | POST | Register a new node in the cluster |
/remove | POST | Remove a node from the cluster (leader only) |
/election | POST | Bully election message: "I am starting an election" |
/coordinator | POST | Leader announcement: "I am the new leader" |
/heartbeat | GET | Health check used by followers to verify the leader is up |
/leader | GET | Leader check for external load balancers (opt-in) |
Every endpoint except /heartbeat and /leader requires HTTP Basic Auth with the cluster credentials. Any other path returns 404. This traffic is not encrypted: see TLS.
Configuration#
The cluster section is optional. Without it, the server runs standalone.
| Key | Required | Description |
|---|---|---|
cluster.node | Yes | Unique node ID, at most 20 characters. Decides election priority. |
cluster.user | Yes | Username for inter-node authentication, at least 4 characters. |
cluster.pass | Yes | Password for inter-node authentication, at least 4 characters. Set it with GHOTI_CLUSTER_PASS to keep it out of the file. |
cluster.manager.type | Yes | Membership manager. Only join_server is supported. |
cluster.manager.addr | Yes | Address this node's cluster HTTP server listens on. |
cluster.manager.join | No | Manager address of an existing node to join. Leave it empty to bootstrap a new cluster. |
cluster.leader.enabled | No | Serve GET /leader for load balancers. Default false. |
cluster.bind | No | Parsed but not used by the server. It has no effect. Clients connect to each node's top-level addr. |
Each node serves clients on its own top-level addr, like a standalone server.
Bootstrap node#
addr: localhost:9091
cluster:
node: node1
user: cluster_user
pass: cluster_pass
manager:
type: join_server
addr: localhost:2222
Without cluster.manager.join, the node bootstraps as leader of a new single-node cluster.
Joining node#
addr: localhost:9092
cluster:
node: node2
user: cluster_user
pass: cluster_pass
manager:
type: join_server
addr: localhost:2223
join: localhost:2222
join points to the manager address of any existing member. On startup, the joining node sends a /join request and receives the current peer list and the leader's identity.
Routing traffic to the leader#
Only the leader serves client requests. Followers don't redirect or proxy. They answer every command with error 000 (NOT_LEADER), followed by the leader's node ID. The client has to connect to the leader itself:
# connected to node2 (follower)
> w000x
< exxx000node1
Over the http transport, a follower answers 503. The node ID is a name, not an address, so clients need their own mapping from node IDs to addresses.
Leader endpoint for load balancers#
To put a load balancer in front of the cluster, enable the leader endpoint on every node:
cluster:
node: node1
user: cluster_user
pass: cluster_pass
manager:
type: join_server
addr: "0.0.0.0:2222"
leader:
enabled: true
It serves GET /leader on cluster.manager.addr:
| Node state | Status | Body |
|---|---|---|
| Leader | 200 | leader |
| Follower | 503 | not leader: <node ID> |
| Flag not set | 404 | Empty |
$ curl -s localhost:2222/leader
leader
$ curl -s localhost:2223/leader
not leader: node1
Point the load balancer's health check at /leader. Every follower fails the check and drops out of rotation, so traffic converges on the leader and follows it after a failover.
- The endpoint is unauthenticated, because health checks can't send credentials. It only reveals which node is the leader, but it is served on the inter-node port, so the load balancer needs network access to that port. Make that port reachable from the load balancer only.
- It only exists on nodes with a
clustersection. A standalone node never answers it.
Node IDs and election priority#
Node IDs are compared as strings, and the highest wins:
"node3"beats"node2", which beats"node1"."z"beats"a"."node10"beats"node1", but loses to"node9"and to"node2", because the comparison stops at the first different character and"1"sorts before"2"and"9".
Use zero-padded IDs such as node01, node02 or node10 to get the order you expect.
Cluster lifecycle#
Starting a cluster#
- Start the first node without
cluster.manager.join. It bootstraps as leader. - Start the other nodes with
cluster.manager.joinpointing to an existing node's manager address. - Each joining node receives the full peer list and the leader's identity.
- The existing node notifies every other peer about the new node.
If the join request fails, for example because the bootstrap node isn't up yet, the node keeps running without logging an error. It never learns a leader and has no cluster HTTP server, so it answers every command with exxx000 followed by an empty leader ID and doesn't answer /leader. There is no retry: restart the node once the node it joins is reachable.
Heartbeat monitoring#
Followers check the leader every 2 seconds with a GET to its /heartbeat endpoint. A connection error or a non-200 response starts an election. The heartbeat request has a 2-second timeout, so a leader that is slow to answer also triggers an election.
Leader failover#
- One or more followers detect the failed heartbeat.
- Each of them starts a bully election and sends election messages to all peers with higher IDs.
- A follower that hears back from a higher-ID peer backs off.
- The highest-ID reachable node declares itself leader and broadcasts a coordinator message.
- Every node updates its leader. The new leader starts with every slot empty.
Graceful shutdown#
On shutdown a node marks itself down, so its heartbeat answers 503, closes its cluster server and waits for its background work to finish. If the leader shuts down, followers see the failed heartbeat and elect a new leader.
Failure scenarios#
Leader crashes#
Followers notice within about 2 to 4 seconds (one heartbeat interval plus the request timeout), and the highest-ID surviving node becomes leader. Until then, requests to the crashed leader fail, and followers answer NOT_LEADER with the old, unreachable leader's ID. Afterwards, clients can reconnect to the new leader. All data on the crashed leader is lost.
Follower crashes#
The cluster keeps working. The leader doesn't monitor followers, it only keeps them in its peer list. Redundancy is reduced: if the leader fails later, fewer nodes can take over. The crashed follower stays in the peer list until it is removed with /remove.
Network partition: full split#
No node can reach any other. Every follower sees the heartbeat fail and starts an election. Higher-ID peers are unreachable, so every node declares itself leader. This is split-brain: several leaders accept writes independently. When the network heals, the nodes still believe they are leaders until a heartbeat detects the conflict and triggers a new election, or until the cluster is reconfigured. Data written to the leaders that lose is gone.
Network partition: partial split#
Some nodes can reach each other but not the leader. They elect the highest-ID node of their partition, while the original leader keeps serving whoever can still reach it. Again several leaders can exist at once, and clients in different partitions write to different leaders. Once connectivity returns, the election eventually settles on the highest-ID node, but data written to the losing leader isn't transferred.
Clients reach all nodes, nodes can't reach each other#
The worst case: every follower elects itself, so every node is an independent leader, and clients can write to any of them. When inter-node traffic returns, elections settle on one leader, and the data on the others becomes unreachable. Clients should handle NOT_LEADER by reconnecting to the reported leader once the cluster is stable.
A node joins with a higher ID#
The new node joins as a follower and accepts the current leader, as told by the join response. It doesn't start an election. If the current leader fails later, the higher-ID node wins the election, as the bully algorithm expects.
A failed leader rejoins#
A crashed leader that rejoins with the same or a higher ID contacts an existing node through /join, receives the peer list and the current leader, and becomes a follower. It doesn't reclaim leadership because of its ID. If an election happens later, it wins it, and serves requests with an empty state.
All nodes restart at once#
The node without cluster.manager.join starts as leader. The others try to join through their configured address. If the bootstrap node isn't ready when they try, their join fails and, as described above, they keep running without a leader until they are restarted. Start the bootstrap node first. All data is lost.
Slow leader#
A leader that takes more than 2 seconds to answer a heartbeat looks dead to the followers, and they start an election. The highest-ID node wins, which may be the slow leader itself. The cluster then returns to the same state after a short period of instability, during which followers can report a stale leader.
Credential mismatch#
A node with a different cluster.user or cluster.pass is rejected with 400 on its join request and on any election or coordinator messages. It can't take part in the cluster.
Limitations#
- No data replication. A new leader starts empty. Availability is the goal, not durability.
- No join retry. A node that can't reach its join address doesn't retry, and keeps running without a leader.
- No split-brain detection. The bully algorithm doesn't detect or resolve split-brain. Elections converge once partitions heal, but data can be lost.
- String ordering of node IDs.
"node10"sorts below"node2"and"node9". Use zero-padded IDs. - No quorum. A single node can declare itself leader if it can't reach any higher-ID peer. That keeps it available under partitions but exposes it to split-brain.
- Stale peer list. Crashed nodes stay in the peer list until removed with
/removeon the leader, so elections also message unreachable nodes, which makes them slower. - Fixed timeouts. The heartbeat interval (2 s), heartbeat timeout (2 s) and election timeout (3 s) are hard-coded.
- No redirects. Followers answer
NOT_LEADERwith the leader's ID, and clients must connect to the leader themselves.