Designing a Two-Tier IaC Architecture with Ansible and Incus

The Problem

Managing distributed bare-metal and virtual infrastructure across geographically dispersed points of presence often tempts engineers into complex clustering solutions like WAN-stretched Kubernetes or Incus clustering across high-latency Internet links.

In our setup—spanning servers across Kansas City (kc), New York (ny), Las Vegas (lv), and Switzerland (ch)—we wanted automated, repeatable deployments for containers, WireGuard meshes, BGP routing nodes, and reverse proxies. But multi-datacenter WAN clusters introduce severe distributed consensus vulnerabilities:

• High latency between nodes slows down Raft/etcd consensus.
• WAN packet loss or transient BGP flapping risks split-brain scenarios or quorum loss.
• A cluster failure at one location can cascade and render independent edge POPs unresponsive.

We needed a clean Infrastructure as Code (IaC) pattern that maintains global automation without fragile network coupling.

The Journey

We evaluated how best to decompose responsibilities between Ansible and Terraform:

1. Option A: Monolithic Ansible Playbooks

   • Use Ansible for both bare-metal OS setup and container management (community.general.incus).
   • Drawback: Lacks declarative state tracking for complex container topologies, networking links, and resource constraints; idempotent lifecycle destruction is clunky.

2. Option B: Stretched Incus Cluster over WAN

   • Cluster all host machines into a single unified Incus cluster and control it via one Terraform provider.
   • Drawback: Incus clusters require reliable sub-millisecond heartbeat latency (dqlite). Running this over transatlantic public links is a recipe for catastrophic database deadlocks.

3. Option C: Two-Tier Architecture with Standalone Hosts (The Winner)

   • Layer 1 (Host Golden-Config): Ansible provisions the base OS (Void Linux / Ubuntu), kernel parameters, sysctl optimizations, WireGuard tunnels, Bird BGP routing, and installs Incus as a standalone daemon on each host.
   • Layer 2 (Container Workloads): Terraform manages Incus instances and networks via the lxc/incus provider, treating each host as an independent endpoint with its own state stored in MinIO object storage.

The Solution

We codified this decoupled architecture into a clean, phased implementation spec.

1. The Host Layer (Ansible)

Ansible roles apply a golden base configuration to each physical host:

# deploy/roles/incus_host/tasks/main.yml
- name: Ensure incus daemon is installed and enabled
  package:
    name: incus
    state: present

- name: Configure storage pool on ZFS
  command: incus admin init --auto --storage-backend=zfs --storage-pool=tank/incus
  args:
    creates: /var/lib/incus/storage-pools/tank/incus

2. The Instance Layer (Terraform + Standalone Incus)

Instead of a single brittle cluster, Terraform targets each host independently using remote endpoints:

# main.tf
terraform {
  backend "s3" {
    bucket                      = "iac-tfstate"
    key                         = "pops/kc.tfstate"
    endpoint                    = "https://minio.kc.internal:9000"
    region                      = "main"
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    force_path_style            = true
  }
}

provider "incus" {
  remote {
    name    = "kc"
    address = "https://10.222.1.1:8443"
    token   = var.incus_kc_token
  }
}

resource "incus_instance" "vault" {
  name      = "vault"
  image     = "images:debian/12"
  ephemeral = false

  config = {
    "boot.autostart" = "true"
    "security.nesting" = "true"
  }

  device {
    name = "eth0"
    type = "nic"
    properties = {
      network = "incusbr0"
      "ipv4.address" = "10.222.100.10"
    }
  }
}

3. Secrets and State Management

• Secrets are encrypted using SOPS + age keys, committed directly to git alongside the IaC repositories.
• Remote Terraform state files are isolated per-POP on internal MinIO S3 instances, ensuring that an issue at one POP never blocks deployments at another.

Lessons Learned

• Decoupling Beats Distributed State Over WAN: Avoid multi-region clusters where distributed consensus protocols (Raft, dqlite, etcd) must traverse public Internet hops. Independent hosts with local state are resilient to network partitions.
• Ansible for Substrate, Terraform for Workloads: Let Ansible manage the OS, packages, and kernel configurations. Let Terraform declare and maintain container instances, networks, and storage devices.
• Per-POP State Isolation: Segmenting Terraform state files per location ensures that a maintenance failure or network hiccup in Europe never halts automation pipelines in North America.

References

• Incus Documentation
• Terraform Incus Provider
• SOPS: Secrets OPerationS