Declarative Infrastructure with SOPS & Ansible: Managing Encrypted Secrets in Git

Table of Contents

1. The Context / The Problem
2. The Deep-Dive / Root Cause Analysis
3. The Implementation / Architecture
4. Lessons Learned & Best Practices
5. References

The Context / The Problem

Managing private encryption keys, WireGuard preshared secrets, database credentials, and BGP peering passwords across version-controlled infrastructure is one of the most perilous challenges in declarative operations. In our GitOps-driven deployment model, Ansible playbooks provision bare-metal hosts and Incus containers across eight Points of Presence globally.

Historically, teams resorted to either centralized secret management servers (like HashiCorp Vault) or native ansible-vault. However, Vault introduces a catastrophic single point of failure: if the central Vault cluster or its network link is down, edge nodes cannot boot or configure their own network interfaces. Conversely, native ansible-vault encrypts entire files as opaque binary blobs, making Git diffs, pull request code reviews, and three-way merges utterly impossible.

We needed a modern declarative secrets workflow that keeps encrypted values directly alongside code in Git, preserves human-readable YAML keys in diffs, supports multi-recipient public-key cryptography (age), and integrates seamlessly into automated CI/CD runners.

The Deep-Dive / Root Cause Analysis

Evaluating secret management tools against distributed, bare-metal operational requirements highlighted two fundamental architectural dilemmas:

1. The Opaque Blob Problem with Ansible-Vault

When two developers modify different secrets inside an ansible-vault encrypted YAML file, Git cannot resolve the differences because the entire file is a single encrypted ciphertext. Merging branches requires manual decrypt-edit-encrypt gymnastics, frequently causing accidental secret overwrites.

2. Runtime Secret Bootstrapping Deadlocks

If a node needs secrets to establish its initial WireGuard VPN interface, it cannot query a remote secret vault over that same VPN. Edge servers must possess decryptable secrets locally at boot time without needing an active control plane connection.

The Implementation / Architecture

We standardized our secrets pipeline on Mozilla SOPS paired with age encryption keys, structured .sops.yaml routing rules, and the community.sops Ansible collection.

1. Declarative .sops.yaml Configuration

SOPS allows defining cryptographic rules per file path, automatically encrypting values (while leaving YAML keys in plain text) using multiple developer and host age public keys:

# .sops.yaml in repository root
creation_rules:
  # Host-specific secrets encrypted with operator and target host keys
  - path_regex: inventory/host_vars/([^/]+)/secrets\.ya?ml$
    age: >-
      age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8z5muac5da8gzip58w88a,
      age16up4p5p8mvg9r9lvhq868e42f9t847t5r94s4z0r65k25r68p4ts43343q
  
  # Global cluster secrets
  - path_regex: inventory/group_vars/all/secrets\.ya?ml$
    age: >-
      age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8z5muac5da8gzip58w88a

2. Transparent Diffable YAML Format

When committed to Git, secrets look clean and diffable:

# inventory/host_vars/kc-edge-01/secrets.yaml
wireguard_private_key: ENC[AES256_GCM,data:YjQ9V2d...,iv:...,tag:...]
bgp_peer_password: ENC[AES256_GCM,data:c2VjcmV0...,iv:...,tag:...]
sops:
    age:
        - recipient: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8z5muac5da8gzip58w88a
          enc: ...

3. Native Ansible Playbook Integration

Using the community.sops plugin, playbooks decrypt variables in memory at runtime without writing plaintext secrets to disk:

---
- name: Configure Edge Anycast BGP Node
  hosts: edge_routers
  gather_facts: false
  vars_files:
    - "{{ inventory_dir }}/host_vars/{{ inventory_hostname }}/secrets.sops.yaml"
  
  tasks:
    - name: Deploy BIRD 2.x Configuration
      ansible.builtin.template:
        src: bird.conf.j2
        dest: /etc/bird/bird.conf
        owner: bird
        group: bird
        mode: '0640'
      notify: Reload BIRD

Lessons Learned & Best Practices

1. Cleartext Keys Make Code Reviews Human-Friendly: Reviewers can see which secret parameters were added or renamed (e.g. wireguard_preshared_key: ENC[...]) without needing access to decrypt the values themselves.
2. Multi-Recipient Keys Prevent Recovery Lockouts: Always encrypt secrets to both individual administrator keys and an offline emergency recovery master key stored in cold storage.
3. Pin Age Identity Locations: Setting export SOPS_AGE_KEY_FILE=/etc/age/keys.txt in server provisioning templates ensures seamless unattended deployments during container rebuilds.

References

• Mozilla SOPS: Secrets Operations
• Age: Simple, Modern, and Secure File Encryption
• Ansible Community SOPS Collection