Years ago I built a thing called FAN2, short for FailOver Asterisk Network. The name was better than the documentation. It kept a pair of Asterisk servers alive at a telecom company, and the whole trick came down to one idea: move an IP address, and make the phones follow it.
Table of contents
Open Table of contents
The short version
- FAN2 gave Asterisk (VoIP over SIP) high availability using a shared floating IP, back when “HA” meant a shell script and a lot of testing at 2am.
- When the primary died, a standby server grabbed the floating IP (which doubled as the internet gateway) and forced Asterisk to reload so it rebound and re-registered on that address.
- Files stayed in sync with an rsync-based replication script. Nothing fancy, just consistent.
- Failover was decided by an active health check against the local Asterisk service. Failback was scheduled inside a time window, on purpose, to avoid flapping.
- Voice has no “eventually consistent.” A dropped call or a lost SIP registration is felt the instant it happens, and that is what makes voice failover harder than failing over a web app.
Where this came from
Before the telecom job I worked somewhere whose system dispatched phone calls off alarm events. An alarm fired, the platform picked up the line and called the customer automatically, no human in the loop. When that pipeline worked, nobody noticed. When it stopped, everybody noticed within a minute, because a call that should have gone out simply did not.
That job rewired how I think about uptime. A stateless web service can drop a request, retry, and the user shrugs. Telephony gives you no such grace. A call is a live session. A SIP registration is state that has to exist, right now, on the box the phone is talking to. You cannot buffer a phone call and replay it later.
So when I got handed a pair of Asterisk servers and the words “make it not go down,” I knew the hard part was never going to be the failover. It was the state.
How FAN2 actually worked
The design was deliberately boring, and boring is a compliment in this line of work.
Two servers, primary and standby. In front of them, a floating IP that also happened to be the gateway the whole network used to reach the internet. That detail matters, because whoever held the floating IP held both the SIP traffic and the default route. One address, two jobs.
On each box, a loop ran an active check of the local Asterisk service. Not a ping, not “is the port open,” but an actual “is Asterisk answering.” The trick was to ask Asterisk a question only a live Asterisk can answer, and to react only after a few consecutive misses, so one blip did not trigger a takeover:
# is Asterisk actually answering, not just running?
fails=0
while true; do
if asterisk -rx "core show channels count" >/dev/null 2>&1; then
fails=0
else
fails=$((fails + 1))
fi
[ "$fails" -ge 3 ] && trigger_failover # a few misses, not one blip
sleep 5
doneIf that check failed enough times, the standby took over. It claimed the floating IP, brought up the gateway, and became the box that mattered.
The piece people forget is what happens next. Moving an IP is not enough. Asterisk was already running with its own idea of which address to bind and register on, so the takeover forced a reload:
# standby claims the floating IP, then makes Asterisk rebind to it
ip addr add 192.0.2.10/24 dev eth0
asterisk -rx "sip reload"
asterisk -rx "core reload"After the reload, SIP re-registered on the floating IP and the other protocols followed it home. From a handset’s point of view, the address it registered against was still there. A different physical machine just happened to be answering.
Files were kept in step by an rsync-based replication script running on a cron between the two nodes: voicemail, config, the bits of state that live on disk.
# rsync-replicate: push the on-disk state to the standby
rsync -az --delete /etc/asterisk/ standby:/etc/asterisk/
rsync -az --delete /var/spool/asterisk/voicemail/ standby:/var/spool/asterisk/voicemail/Not real-time, but close enough that a failover did not drop you onto a stale box.
The one detail worth remembering, failback
Everyone gets excited about failover. Failback is where homegrown HA quietly kills itself.
The naive version brings the primary straight back the moment it looks healthy. Then the primary flaps (a flaky link, a service that restarts twice), and now your floating IP is ping-ponging between two servers while live calls die on every bounce. I have watched that happen. It is worse than just staying down.
FAN2 refused to fail back automatically. Failback only happened inside a defined time window. The primary could come back to life in the middle of the afternoon, and FAN2 would note it and then do absolutely nothing until the scheduled window said it was allowed.
# only fail back inside a quiet overnight window, never mid-day
hour=$(date +%H)
if [ "$primary_healthy" = yes ] && [ "$hour" -ge 2 ] && [ "$hour" -lt 5 ]; then
fail_back_to_primary
fiA quiet hour, basically. It traded a few extra minutes on the standby for never flapping during business hours, and that trade was correct every single time.
Final takeaway
The thing FAN2 taught me, and the thing that still holds, is that failover is easy and failback is a discipline problem. Machines are happy to thrash. The engineering is in deciding when to move, and being conservative about moving back.
The other half has aged even better. Cloud providers now hand you a floating IP and a health check as menu items, and it is tempting to treat HA as a box you tick. But the box does not know your workload has state. Voice did, back then, in the most unforgiving way: registrations and live calls that cannot be reconstructed after the fact. Plenty of “modern, stateless” systems carry exactly the same kind of hidden state, they just hide it better.
A floating IP, an honest health check, rsync, and a failback window. That was the whole system, and most of it would still work today. The tools got nicer. The problem, deciding when to move state and when to leave it alone, did not change at all.