Blog / Tutorial

Implementing leader election with Ghoti

Use a timeout memory slot as a lease so that only one of your nodes acts as leader, and learn exactly where the approach stops being safe.

Updated The Ghoti team5 min read

Many distributed systems need one node to be "in charge": running a scheduled job, owning a shared resource, or coordinating the others. Consensus protocols like Raft or Paxos solve this properly, but they are a lot of machinery when all you need is "one worker at a time, most of the time".

This post shows how to build simple leader election on a Ghoti timeout_memory slot, and, just as importantly, the cases where it is not safe.

Updated for Ghoti 0.3

The first version of this post used the type: configuration key, the wrong error code, port 8080 and telnet against the default transport. It also claimed guarantees that a lease on a volatile server can't give. This version fixes all of that, and every transcript below was captured from a running server.

The building block: a lease#

A timeout_memory slot holds a value and remembers who wrote it:

Store your node's ID as the value, and the owner of the slot is your leader.

Configuration#

slot_001:
  kind: timeout_memory
  timeout: 10   # seconds

Start the server with ghoti run from the directory that holds this config.yaml. It listens on localhost:9090.

The algorithm#

Every node runs the same loop on one long-lived connection:

  1. Try to take the lease: write your ID, w001<your-id>. If the answer is v001<your-id>, you are the leader.
  2. If you got e001007, someone else holds it. Read r001 to see who, and try again later.
  3. While you are leader, renew: write your ID again well before the timeout, for example every third of it. Each successful write restarts the timeout.
  4. If a renewal fails, or you can't reach Ghoti, stop acting as leader immediately.

Try it with two terminals#

Use nc, which ends each line with a bare \n as the default standard transport expects. A telnet client sends \r\n, so it only works if you set protocol: telnet in the configuration.

$ nc localhost 9090

Node A takes the lease:

> w001node-a
< v001node-a

Node B, on its own connection, tries and is refused, but can see who the leader is:

> w001node-b
< e001007
> r001
< v001node-a

Node A renews its lease:

> w001node-a
< v001node-a

Now close node A's nc with Ctrl+C. Closing the connection does not release the lease, so B is still refused:

> w001node-b
< e001007

About 10 seconds after A's last write, B takes over:

> w001node-b
< v001node-b
> r001
< v001node-b

Handing over#

There is no release command. A leader that shuts down gracefully stops renewing, and another node takes over once the timeout expires. Plan for a gap of up to one timeout in which nobody is leader.

You can still publish your intent: the leader can write a value such as w001leaving:node-a so that other nodes reading the slot know a handover is coming. Ownership doesn't change until the timeout expires.

Where this approach stops being safe#

A lease on a volatile, single server is simple, but it isn't a consensus protocol. Know these limits before you rely on it.

A Ghoti restart forgets the owner#

Ghoti keeps everything in memory. If the server restarts, slot 001 comes back empty with no owner, and the first node to write becomes leader, while the old leader may still believe it holds the lease until its next renewal fails. Around a restart, two nodes can both believe they are leader. The same applies when a Ghoti cluster fails over, because the new leader starts empty.

There is no fencing token#

A leader that is paused, by a long garbage-collection pause, a stalled VM or a network hiccup, can wake up after its lease has expired and keep acting as leader until it tries to renew. Ghoti doesn't hand out a monotonically increasing token that downstream systems could use to reject the stale leader's work. If two leaders acting at the same time would corrupt data, this approach isn't enough on its own.

A crashed client waits out its own lease#

Ownership belongs to the TCP connection. A leader that crashes and reconnects comes back as a new connection, so its writes fail with 007 until its old lease expires. The timeout is effectively your recovery time: shorter means faster failover, but more risk of losing the lease during a pause.

Ghoti itself is a single point of failure#

While Ghoti is unreachable, nobody can take or renew the lease. Followers must treat "can't reach Ghoti" as "not leader".

Other details#

When to use it#

This pattern fits when an occasional overlap or gap in leadership is acceptable:

Use a consensus system such as etcd or ZooKeeper, with fencing tokens, when two leaders at the same time would be a correctness bug, or when leadership must survive a restart of the coordinator.

Learn more#

← All posts