Building a High-Availability Headscale Control Plane with Puppet

I run a lot of internal infrastructure, and one of the things I wanted to get rid of was the dependence on Tailscale’s SaaS control plane for our staff VPN. Headscale is a self-hosted implementation of the Tailscale control server — same WireGuard-based mesh, same clients, but you own the coordination server. The catch: a single Headscale instance is a single point of failure. If it dies, no new clients can join, keys can’t rotate, and the control plane goes dark even though existing tunnels keep working.

So I built a highly available Headscale cluster and wrapped it in a Puppet role. This post is the story of how it works — and, more honestly, how many times it didn’t work before it did.

Everything below uses example identifiers — placeholder domains, documentation IP ranges, fake UUIDs, and made-up hostnames — so you can adapt it to your own environment without leaking anything from mine.

Why HA Headscale?

Tailscale’s design splits the data plane from the control plane. WireGuard tunnels are direct, peer-to-peer, and keep working without the coordinator. But the coordinator is what issues node keys, pushes the ACL map, handles DNS (MagicDNS), runs the DERP relay directory, and authenticates users via OIDC. Lose it and:

• New devices can’t enrol.
• ACL changes don’t propagate.
• MagicDNS stops resolving.
• DERP relay discovery freezes (though in-use relays keep going for a while).

For a homelab that’s fine. For a company VPN that hundreds of people depend on to reach internal services, it’s not. I needed the control plane to survive a node dying, a kernel panic, a hypervisor migration, or planned maintenance — all without anyone noticing.

The goal: two active Headscale instances fronted by a VIP, backed by a replicated PostgreSQL database with automatic failover, and a third node purely to break quorum ties.

The Architecture

Three nodes, two roles:

                ┌──────────────── public network ────────────────┐
                │                                                │
                │      203.0.113.10 (API VIP, VRRP)               │
                │           │                                    │
   ┌────────────┴───────────┴──────────────┐   ┌─────────────────┐
   │  vpn-srv-01   203.0.113.11            │   │  vpn-witness-01  │
   │  Headscale + nginx + certbot          │   │  203.0.113.13    │
   │  PostgreSQL (Patroni leader/replica)  │   │  etcd only       │
   │  etcd  ✔  Keepalived  ✔               │   │  (3rd quorum     │
   │  tailscale subnet-router (when master)│   │   vote)          │
   └───────────────────┬───────────────────┘   └─────────────────┘
                       │ internal network (172.16.x)
   ┌───────────────────┴───────────────────┐
   │  vpn-srv-02   203.0.113.12            │
   │  Headscale + nginx (cert replica)     │
   │  PostgreSQL (Patroni replica/leader) │
   │  etcd  ✔  Keepalived  ✔               │
   │  tailscale subnet-router (when master)│
   └───────────────────────────────────────┘
                       │
                172.16.0.53 (DB VIP, follows Patroni leader)

Node types

The witness is the cheapest node in the whole design. It runs one daemon. Its entire reason to exist is to make etcd a 3-node cluster instead of a 2-node one, so a single failure can’t split quorum. Two servers + one witness = 3 etcd peers, fault tolerance 1. It costs you a tiny VM and saves you from a split-brain.

Two networks per node

Each node has two NICs:

The split matters a lot. Everything cluster-internal (PostgreSQL, etcd, Patroni REST, VRRP, SSH, cert rsync) lives on ens34. The only things exposed on the public NIC are the API (HTTPS/443), DERP/STUN (3478/udp), and WireGuard direct (41641/udp). This means a compromise of the public surface doesn’t immediately hand an attacker the database port.

The Stack, Layer by Layer

Headscale itself is stateless-ish: it stores node registrations, pre-auth keys, and policy in PostgreSQL. So “HA Headscale” is really “HA PostgreSQL with two Headscale API frontends pointed at it.” Everything else is plumbing to make that safe.

Headscale (API)  ──►  PostgreSQL (VIP 172.16.0.53)  ◄──  Headscale (API)
                        │  Keepalived moves the VIP
                        │  to the Patroni leader
                     Patroni  ◄──  etcd (3 nodes, Raft)

• PostgreSQL 16 — the actual state store.
• Patroni — manages Postgres replication and automatic failover. It watches etcd for cluster state, elects a leader, promotes a replica if the leader dies, and rewrites pg_hba.conf/postgresql.conf as roles change.
• etcd — the distributed KV store Patroni uses for leader election and consensus. 3 nodes for quorum.
• Keepalived — VRRP. Owns the floating IPs. Two VRRP instances:
  • VI_INTERNAL — the DB VIP (172.16.0.53), which follows the Patroni leader via a health check.
  • VI_EXTERNAL — the public API VIP (203.0.113.10), which floats between the two Headscale frontends.

The crucial coupling: the DB VIP must follow the DB leader, not just the “preferred” node. I’ll come back to this — it was the source of the worst bug in the whole project.

The Puppet Role

Everything is defined as a Puppet role module, role_headscale, following the standard rs_* / profile_* / role_* layering: resource modules own one service, profiles compose them, roles assign them to a node and may hold role-specific data.

Auto-detection from hostname

The main class figures out what a node is from its hostname, so I don’t have to tag nodes manually:

$actual_component = $hostname ? {
  /^vpn-srv-/    => 'server',
  /^vpn-witness/ => 'witness',
  default        => fail("Unable to determine component for ${hostname}..."),
}

case $actual_component {
  'server':  { include role_headscale::server }
  'witness': { include role_headscale::witness }
  default:   { fail("Invalid component '${actual_component}'.") }
}

Add a node, give it the right hostname, and Puppet does the rest. The component parameter lets you override it for testing.

The witness is deliberately boring

role_headscale::witness is ~90 lines. It sets up iptables (etcd ports only, cluster-restricted), drops in etcd, and joins the existing cluster. No Postgres, no Patroni, no Headscale, no Keepalived. The whole class exists to run one daemon and vote. That’s the kind of component I like — small enough to reason about completely.

The server class is the big one

role_headscale::server composes the full stack: rs_iptables, rs_postgres, rs_etcd, rs_patroni, rs_keepalived, rs_headscale, plus nginx, certbot, the tailscale subnet-router client, the Azure AD sync script, and a pile of sysctl/nft plumbing. It’s parameterised so the two servers differ only by Hiera data — same code, different node YAML.

The ordering is explicit and matters:

Class['rs_postgres'] -> Class['rs_patroni']
Class['rs_patroni']  -> Class['rs_keepalived']

Postgres before Patroni (Patroni manages Postgres), Patroni before Keepalived (Keepalived’s health check queries Patroni’s REST API on :8008 to decide whether to hold the VIP).

Hiera Data Hierarchy

Configuration lives in Hiera, split into shared and per-node:

data/
  common.yaml                         # cluster-wide: etcd endpoints, VIPs, OIDC, ACL rules, group mappings
  nodes/
    vpn-srv-01.example.com.yaml        # per-node: IPs, keepalived priority, cert primary, tailscale IP
    vpn-srv-02.example.com.yaml
    vpn-witness-01.example.com.yaml

common.yaml holds everything that’s identical across nodes — etcd cluster string, the two VIPs, OIDC issuer, ACL rules, the Azure AD → ACL group mappings. The per-node files hold only the deltas: public/internal IP, Keepalived priority, whether this node is the cert primary, and the node’s tailscale IP.

This separation is what makes the two servers genuinely interchangeable. The code reads the same; only data differs. Adding a third server later is a new node YAML, not a code change.

Secrets never live in Hiera

Every password — DB, replication, OIDC client secret, Azure AD app secret — is a Hiera lookup() into a secrets vault:

role_headscale::server::db_password: "%{lookup('hs-db.password')}"
role_headscale::server::oidc_client_secret: "%{lookup('hs-oidc-secret.password')}"

The YAML only carries the pointer. The vault carries the value. This is non-negotiable in a config-managed repo that’s going to be in git — plaintext secrets in commits is how you end up on a breach notification.

Database HA: Patroni + etcd + the Witness

This is the heart of the thing. Patroni runs on both servers. Each Postgres instance is either leader (read-write) or replica (read-only, streaming from the leader). Patroni uses etcd to hold the cluster state and run leader election.

With 2 servers you have 2 etcd peers — which is a bad number, because losing one leaves you with 1, and Raft needs a majority of a 2-node cluster = 2. One node down = no quorum = no leader election = Patroni can’t promote a replica. That’s why the witness exists: it’s the third etcd vote. 3 nodes, fault tolerance 1. Lose any one node (including the witness) and the other two still have quorum.

I made one decision up front that saved a lot of pain: the primary server stays the primary. The first server has Patroni priority: 100 and the second has nofailover: true. Failover still happens automatically if the primary dies, but when it comes back, it reclaims leadership instead of the replica holding it. This avoids a class of flip-flop bugs where a rebooted node rejoins and fights over the leader role. The replica is a replica — it takes over only when there’s no alternative, and it yields the moment the primary is healthy.

Keepalived: Two VIPs, and the Race Condition That Broke Me

Here’s where I earned my grey hairs. Keepalived runs two VRRP instances:

• VI_EXTERNAL — public API VIP (203.0.113.10). This is what clients hit. Either frontend can serve it; they share a database, so it doesn’t matter which answers.
• VI_INTERNAL — DB VIP (172.16.0.53). This one must follow the Patroni leader, because Headscale connects to db_host: 172.16.0.53 and needs to reach a read-write Postgres.

Keepalived’s check_patroni script is wired into VRRP: it adds +50 to the VRRP priority of whichever node is currently the Patroni leader. So whichever node owns the Postgres leader role gets bumped and wins the VRRP election for the internal VIP. Sounds clean.

The bug: I’d initially given the primary server a static VRRP priority of 110. That meant on a cold start, before check_patroni had run, the primary won the internal VIP just for being the primary — even if, for a few seconds, the DB leader was actually still settling. Worse, when the primary rebooted and came back as leader, there was a window where Keepalived had moved the VIP back to the primary before Patroni had finished promoting it. Headscale then connected to the VIP, hit a read-only replica, and returned HTTP 500s. Clients flapped.

The fix: drop the primary’s base priority to 100 so the VIP follows the Patroni leader only — the +50 from check_patroni is the deciding factor, not a static node preference. And couple check_patroni to both custom VRRP instances, not just a default single-instance config, so the DB VIP can never get stuck on a read-only replica after a Patroni restart.

This is the kind of bug that only shows up under real failover, at 2am, when someone reboots the wrong host. The CHANGELOG entry for it is the most satisfying line in the whole module.

The Control-Plane Identity Problem

Here’s a subtlety that’s easy to miss: two Headscale instances are not automatically the same control plane. Headscale has crypto identity files:

• noise_private.key — the WireGuard noise key the control server uses to identify itself to clients.
• derp_server_private.key — the embedded DERP relay’s key.
• acl_policy.hujson — the ACL policy.

If each server generates its own noise_private.key, clients see two different control servers. That breaks key exchange and routing. Both servers must share one identity. Same for the ACL policy — if they drift, clients get different rules depending on which frontend answers.

So the cert-sync channel I built (more below) was extended to also sync noise_private.key, derp_server_private.key, and acl_policy.hujson from primary to replica. One node is the source of truth for crypto identity; the other mirrors it. The replication is one-way and deliberate.

This is the real difference between “two Headscale instances” and “one HA Headscale control plane”: shared identity, shared state, shared policy.

SSL/TLS: nginx, Certbot, and One-Way Cert Sync

Headscale itself speaks plain HTTP. I wanted real TLS on the public API, so nginx sits in front and terminates HTTPS, proxying to Headscale on 127.0.0.1:8080. With TLS on:

• Headscale binds to 127.0.0.1:8080 only — it’s not on the public interface at all.
• Port 8080/tcp is dropped from the public firewall.
• nginx returns 404 for /swagger so the API docs aren’t publicly browsable.
• Certbot (via snap) obtains the Let’s Encrypt cert on the primary.

The challenge: only one node can run certbot for a given hostname. Let’s Encrypt verifies domain ownership; two nodes racing for the same cert is a mess. So the primary obtains the cert and pushes /etc/letsencrypt/ to the replica over internal SSH (rsync). The replica runs HTTP-only nginx until it has certs, then switches to HTTPS.

Puppet models this with two flags:

# primary (vpn-srv-01)
role_headscale::server::ssl_cert_primary: true
role_headscale::server::ssl_cert_sync_peers:
  - '172.16.0.12'   # the replica's internal IP

# replica (vpn-srv-02)
role_headscale::server::ssl_cert_primary: false
role_headscale::server::ssl_cert_sync_source: '172.16.0.11'  # the primary

A custom Facter fact, headscale_ssl_cert_ready, reports whether the cert exists locally. The replica’s nginx vhost is rendered from a bootstrap template (HTTP-only) until the fact flips true, then Puppet re-renders the real HTTPS vhost. So the replica boots HTTP-only, pulls certs on its timer, and self-promotes to HTTPS — no inbound SSH needed on the replica, no manual certbot, no race.

The certbot renewal deploy hook re-pushes after every renewal, so the replica’s cert never goes stale.

Authentication: OIDC + Azure AD Group → ACL Sync

Authentication is delegated to Microsoft Entra ID (Azure AD) via OIDC — staff SSO, no separate VPN credentials. But Headscale has an awkward limitation here: it can’t use OIDC group claims directly in ACLs. You authenticate via Entra ID, but what you can reach is controlled by Headscale’s own ACL policy, which wants email lists, not group UUIDs.

So there are two separate concerns:

1. Who can authenticate — controlled by oidc_allowed_domains (and oidc_allowed_groups, though Entra ID’s groups claim turned out to be unreliable, so I rely on domain allowlisting plus email verification handling).
2. What authenticated users can access — controlled by the ACL policy, whose group membership is synced from Azure AD.

The sync script

A Python script (headscale-sync-groups.py) runs on a systemd timer every hour and:

1. Queries the Microsoft Graph API for the members of each mapped Azure AD group.
2. Rewrites the ACL policy file with the current member lists.
3. Drops group entries that are no longer in the mapping (so removing someone from the Azure AD group actually revokes them).
4. Restores mandatory policy sections (tagOwners, autoApprovers) if they’re missing — without this, the replica would fail to parse tag:subnet-router after a botched sync.
5. Reloads Headscale to apply.

The mapping is data, not code:

role_headscale::server::azure_ad_group_mappings:
  'aaaaaaaa-1111-1111-1111-111111111111': 'group:sysadmin'
  'bbbbbbbb-2222-2222-2222-222222222222': 'group:network'
  'cccccccc-3333-3333-3333-333333333333': 'group:finance'   # VPN mesh only

Add someone to the Azure AD group → within an hour they’re in the ACL group. Remove them → within an hour they’re out. No tickets, no manual user management. This is the part that makes the VPN actually maintainable at scale.

Tiered access

Not everyone gets the internal network. The ACL rules encode tiers:

The “mesh only” tier is the nice bit — those groups can reach other VPN clients but nothing internal. Useful for people who just need laptop-to-laptop connectivity without poking holes in the corporate firewall.

Subnet Routing: Only the Master Advertises

The two Headscale servers are also Tailscale subnet routers — they advertise whatever internal network ranges you want VPN clients to reach (your RFC1918 space, a management VLAN, a few specific hosts, whatever fits your environment) so clients can route to internal services through the tunnel.

The trap: both servers must not advertise routes at the same time. If both do, the return path breaks — internal routing for the tailnet prefix targets one server, and if a client’s traffic leaves via server A but the return path expects server B, packets vanish. Asymmetric routing, silent failure, “the VPN works for some people sometimes.”

So subnet routes are tied to Keepalived state. The VI_INTERNAL VRRP instance has a notify_script — when this node becomes MASTER, it advertises routes and exit-node capability; when it becomes BACKUP, it withdraws them. Only the node holding the DB VIP (and thus the Patroni leadership) advertises. On failover, the new master starts advertising within seconds.

The flags on tailscale up are deliberate:

--accept-routes=false --snat-subnet-routes=false --advertise-tags=tag:subnet-router

--snat-subnet-routes=false is the important one — it preserves the original client IP instead of masquerading it to the server’s internal IP. Without it, internal services see all VPN traffic as coming from the subnet router’s IP, which makes logging and ACLs on internal services useless. With it, internal services see the real tailnet IP (100.100.x.x) and can apply their own policy.

The nftables ts-forward Bypass (a.k.a. “Why Did My Internal Traffic Stop Routing?”)

This one took the longest to find. Tailscale installs its own nftables rules, including a chain called ts-forward that drops traffic sourced from 100.64.0.0/10 going out tailscale0. That makes sense for the tailnet — you don’t want tailnet hairpinning back out the tunnel.

But: 100.64.0.0/10 is the CGNAT range, and the tailnet isn’t the only thing that can live in it. If any of your internal networks is also numbered inside 100.64.0.0/10 (it’s a large range, and it happens), legitimate reply traffic from the internal uplink — sourced from somewhere in that block — matches Tailscale’s broad DROP rule and gets silently thrown away. Clients then can’t reach that internal network through the VPN, intermittently, depending on path.

The fix: a small script, headscale-nft-ts-forward-bypass.sh, inserts an nft accept rule for whichever internal CIDRs fall inside 100.64.0.0/10 before Tailscale’s DROP. And because tailscaled recreates ts-forward on every (re)start, the script is wired in as a systemd ExecStartPost drop-in on tailscaled.service — it re-runs after every tailscale restart. It’s also re-run from the Keepalived MASTER notify and the subnet-router sync, so the bypass survives failover too.

The lesson I keep relearning: a third-party daemon that manages its own firewall rules will, sooner or later, drop traffic you care about. Either pin it down or add an idempotent re-applier that runs after it. I chose the latter.

TCP Tuning for Subnet Routing

Subnet routing over WireGuard has an MTU gotcha. The tunnel encapsulates, so the effective MTU is lower than the physical link. TCP connections through the tunnel can hang or perform terribly if MSS isn’t clamped to the path MTU.

I added mangle rules to clamp MSS on forwarded SYN packets going out both tailscale0 and the internal NIC:

'055 mangle clamp mss to tailscale0' => {
  'table' => 'mangle', 'chain' => 'FORWARD', 'proto' => 'tcp',
  'outiface' => 'tailscale0', 'tcpflags' => { 'mask' => 'SYN,RST', 'match' => 'SYN' },
  'jump' => 'TCPMSS', 'clamp_mss_to_pmtu' => true,
},

Plus sysctls: net.ipv4.tcp_mtu_probing=1 (let the kernel discover PMTU black holes) and larger socket buffers. After this, file transfers through the VPN stopped stalling at 99%.

Firewall Design

rs_iptables is configured with allow_ssh => false and explicit rules, because the default “allow SSH from anywhere” is wrong for a control-plane node. The rules I landed on, after a couple of iterations:

• SSH (tcp/22): accepted only when it arrives on the internal NIC (-i ens34) — any source IP, but only that interface. Plus a rule for tailscale0 so I can SSH in over the VPN itself, and to the tailnet prefix for SSH to the node’s Tailscale address. Public ens33 never accepts SSH.
• Public: 80/tcp (ACME challenge), 443/tcp (API), 3478/udp (DERP/STUN), 41641/udp (WireGuard direct).
• Cluster-only (sourced from the cluster subnet): 5432 (Postgres), 2379/2380 (etcd), 8008 (Patroni REST).
• VRRP (IP protocol 112) from the cluster subnet.
• Forwarding: accept in/out tailscale0 for subnet routing; MASQUERADE for the specific external routes and exit-node traffic.
• Zabbix: 10050/10051 for monitoring.

The “SSH on internal NIC, not by source IP” rule is the version that finally worked. Earlier I’d tried -s <internal-subnet> rules, which blocked normal SSH to internal addresses. Matching on the interface instead of the source is robust: anything physically arriving on the internal NIC is allowed, regardless of exactly which internal IP it came from. Simpler rule, fewer surprises.

A Small Performance Win: UDP GRO Forwarding

The Headscale servers handle a lot of small UDP packets (WireGuard, STUN). On modern NICs, Generic Receive Offload (GRO) can coalesce them — but the relevant flags (rx-udp-gro-forwarding, rx-gro-list) aren’t on by default. A networkd-dispatcher script flips them on whenever the interface comes up, and a Puppet exec sets them immediately on apply:

ethtool -K ens33 rx-udp-gro-forwarding on rx-gro-list off
ethtool -K ens34 rx-udp-gro-forwarding on rx-gro-list off

Small change, measurable reduction in CPU on interrupt handling under load. Worth it on a VPN headend.

Embedded DERP: One Region, One Relay

Headscale can run an embedded DERP relay (the Tailscale relay used when two clients can’t establish a direct WireGuard connection). I run it, but only on the primary. The replica has derp_embedded_enabled: false. Otherwise clients get offered both servers as relays in region 999, and since both share a public VIP anyway, that just adds confusion. One relay, one advertised endpoint. Clients that can go direct still go direct; the ones that need a relay all use the same one.

Lessons Learned

A few things this project drilled into me:

1. “Two instances” is not “HA.” HA means shared identity, shared state, shared policy, and a failover path that actually works. Two Headscale daemons pointed at the same DB is the easy 10%; syncing the noise keys, ACL policy, and certs, and wiring the VIP to the DB leader, is the hard 90%.

2. Quorum wants odd numbers. Two of anything is a trap. The witness node — one etcd daemon on a tiny VM — is the cheapest insurance in the whole design.

3. Decouple “preferred” from “leader.” Let the DB elect its leader, and let the VIP follow that leader. Don’t bake node preference into both layers, or they fight on failover.

4. Third-party firewall managers will eat your traffic. Tailscale’s ts-forward is the canonical example. When a daemon owns its own nftables chain, assume it will drop something you need, and plan to re-assert your rules after every restart of it.

5. Sync identity, not just data. Crypto keys are part of the control plane. If your HA nodes don’t share noise_private.key, you don’t have one control plane — you have two that happen to share a database.

6. Data over code for node differences. The two servers are the same Puppet class with different Hiera. That’s the only reason a third server is a YAML file instead of a refactor.

7. Make the boring component boring. The witness does one thing. The subnet-router notify does one thing. The cert-pull timer does one thing. Every component I could make small, I made small, and those were the ones that never broke.

The Result

A VPN control plane where:

• Either server can die and clients keep connecting — the VIP floats, the DB leader promotes, the new master starts advertising routes, all inside a minute.
• New staff are added to the right Azure AD group and have VPN access within an hour — no ticket to me.
• Leaving staff are removed from the group and lose access within an hour — no stale accounts.
• The database is replicated, the control-plane identity is shared, and the certs are kept in sync automatically.
• Everything is in Puppet, so rebuilding any node from scratch is puppet agent -t away from identical.

It took a lot of 2am debugging to get here. The CHANGELOG is honestly the most honest part of the whole module — it reads like a list of ways this design tried to kill me, and how I pinned each one down. But the end state is a VPN I don’t have to think about, which is the only acceptable end state for infrastructure.

If you’re running Headscale in production for more than yourself, I hope some of the traps above save you the hours they cost me.