How automatic failover works in CoyoteDB

How CoyoteDB's nodes vote on a main node, why two computers cannot decide alone, what a witness adds, and how a cut-off node steps aside to avoid split brain.

Failover is what happens when the main node stops: another node takes over, so your app can keep writing. In CoyoteDB the nodes decide that together, by voting.

One main node at a time

At any moment one node is the main node. It takes the writes. The other data nodes keep live copies and keep checking on it. Your app connects with a normal Postgres connection string.

Voting and the majority

The nodes vote on which one is the main node. A node needs more than half of all the votes. With three voters, two votes win. That rule is what makes automatic failover safe: there can only ever be one group holding a majority, so there can only ever be one main node.

If the main node stops, the nodes that are left check whether they still have a majority between them. If they do, they vote, one of them becomes the main node, and writes carry on.

Why two computers cannot decide alone

With two data nodes, each one has one vote of two. If they lose touch, each side sees exactly half, never more than half.

From the inside, “the other computer died” and “the network between us broke” look the same. If each side took over on its own, both could end up taking writes, and your data would split in two. So in a plain 2-node setup, nothing switches automatically. The copy is safe on the other computer, but the database is down until the main node is back. Add a witness to make switch-over automatic.

What a witness adds

A witness is a tiny helper that holds no data and only votes. Add one to two data nodes and there are three voters. Now, if one data node stops, the other data node and the witness still make two of three, a majority, so switch-over happens on its own.

The witness can run on any small always-on device, such as a Raspberry Pi, a mini PC, an old laptop that stays plugged in and awake, or a cheap cloud server, and it can run a different system from the data computers. If you can, put it somewhere else, so one power cut or one router failure cannot take out the majority.

With three data nodes you do not need a witness: any two of three are a majority.

Keep the total number of voters odd (data nodes plus any witness), so there is always a clear majority.

A cut-off node steps aside

If a node loses touch with the others, it shuts its database down before another node takes over. Two nodes never both think they are in charge. That problem is usually called split brain, and avoiding it matters more than switching over quickly: a database that pauses can catch up; one that splits in two loses data.

What your app notices

When the main node stops and another takes over, connections to the old main node drop. Passing connections on from any node to the current main node, so your app never needs a new address, is not available yet.

Read more: What “saved on two computers” means, Node counts explained, and the diagrams on How it works. Automatic switch-over is tested with real Postgres and with chaos tests that kill, freeze and cut off nodes on purpose. So far those tests have run on one Mac at a time, not yet on separate machines or with real power cuts.