Implementing leader election with Ghoti
Use a timeout memory slot as a lease so that only one of your nodes acts as leader, and learn exactly where the approach stops being safe.
Many distributed systems need one node to be "in charge": running a scheduled job, owning a shared resource, or coordinating the others. Consensus protocols like Raft or Paxos solve this properly, but they are a lot of machinery when all you need is "one worker at a time, most of the time".
This post shows how to build simple leader election on a Ghoti timeout_memory slot, and, just as importantly, the cases where it is not safe.
The first version of this post used the type: configuration key, the wrong error code, port 8080 and telnet against the default transport. It also claimed guarantees that a lease on a volatile server can't give. This version fixes all of that, and every transcript below was captured from a running server.
The building block: a lease#
A timeout_memory slot holds a value and remembers who wrote it:
- Ownership. The connection that writes the slot becomes its owner. While it owns the slot, writes from any other connection fail with error
007. - Expiry. If the owner doesn't write again within
timeoutseconds, ownership expires and the next writer takes over. - Visibility. Anyone can read the slot to see the current value.
Store your node's ID as the value, and the owner of the slot is your leader.
Configuration#
slot_001:
kind: timeout_memory
timeout: 10 # seconds
Start the server with ghoti run from the directory that holds this config.yaml. It listens on localhost:9090.
The algorithm#
Every node runs the same loop on one long-lived connection:
- Try to take the lease: write your ID,
w001<your-id>. If the answer isv001<your-id>, you are the leader. - If you got
e001007, someone else holds it. Readr001to see who, and try again later. - While you are leader, renew: write your ID again well before the timeout, for example every third of it. Each successful write restarts the timeout.
- If a renewal fails, or you can't reach Ghoti, stop acting as leader immediately.
Try it with two terminals#
Use nc, which ends each line with a bare \n as the default standard transport expects. A telnet client sends \r\n, so it only works if you set protocol: telnet in the configuration.
$ nc localhost 9090
Node A takes the lease:
> w001node-a
< v001node-a
Node B, on its own connection, tries and is refused, but can see who the leader is:
> w001node-b
< e001007
> r001
< v001node-a
Node A renews its lease:
> w001node-a
< v001node-a
Now close node A's nc with Ctrl+C. Closing the connection does not release the lease, so B is still refused:
> w001node-b
< e001007
About 10 seconds after A's last write, B takes over:
> w001node-b
< v001node-b
> r001
< v001node-b
Handing over#
There is no release command. A leader that shuts down gracefully stops renewing, and another node takes over once the timeout expires. Plan for a gap of up to one timeout in which nobody is leader.
You can still publish your intent: the leader can write a value such as w001leaving:node-a so that other nodes reading the slot know a handover is coming. Ownership doesn't change until the timeout expires.
Where this approach stops being safe#
A lease on a volatile, single server is simple, but it isn't a consensus protocol. Know these limits before you rely on it.
A Ghoti restart forgets the owner#
Ghoti keeps everything in memory. If the server restarts, slot 001 comes back empty with no owner, and the first node to write becomes leader, while the old leader may still believe it holds the lease until its next renewal fails. Around a restart, two nodes can both believe they are leader. The same applies when a Ghoti cluster fails over, because the new leader starts empty.
There is no fencing token#
A leader that is paused, by a long garbage-collection pause, a stalled VM or a network hiccup, can wake up after its lease has expired and keep acting as leader until it tries to renew. Ghoti doesn't hand out a monotonically increasing token that downstream systems could use to reject the stale leader's work. If two leaders acting at the same time would corrupt data, this approach isn't enough on its own.
A crashed client waits out its own lease#
Ownership belongs to the TCP connection. A leader that crashes and reconnects comes back as a new connection, so its writes fail with 007 until its old lease expires. The timeout is effectively your recovery time: shorter means faster failover, but more risk of losing the lease during a pause.
Ghoti itself is a single point of failure#
While Ghoti is unreachable, nobody can take or renew the lease. Followers must treat "can't reach Ghoti" as "not leader".
Other details#
- Leases don't work over the
httptransport, where every request is a new connection. - Pick a timeout that is much longer than your renewal interval plus the worst network delay you expect.
- The traffic isn't encrypted. Keep it on a trusted network or put TLS in front of Ghoti.
When to use it#
This pattern fits when an occasional overlap or gap in leadership is acceptable:
- Running a periodic job on one node instead of all of them, where an occasional double run is harmless.
- Picking one node to refresh a cache or push configuration updates.
- Reducing duplicate work, rather than preventing it absolutely.
Use a consensus system such as etcd or ZooKeeper, with fencing tokens, when two leaders at the same time would be a correctness bug, or when leadership must survive a restart of the coordinator.
Learn more#
- Timeout memory reference, including all the ownership caveats.
- Protocol, for the exact framing, ordering and retry rules your client must follow.
- Volatile state, for what a restart means for every slot kind.