NixOS deployments for dma.space.

This repository manages bare-metal servers, colocation nodes, IoT devices, and MicroVMs throughout the space.

Structure

• machines/ - Bare-metal and host machine configurations (colo-komo, colo-fmt2, dreamflasher, space-kiosk, dma-con-sw-0).
• microvms/ - MicroVM guest and host configurations running under microvm.nix (hosted on dreamflasher).
• modules/ - Reusable NixOS modules:
  • base.nix (under machines/) - Baseline configuration auto-included in every machine (minimal profile, monitoring exporters, journald limits, flakes).
  • gitops.nix - Automated continuous deployment via Comin.
  • network.nix / netbird.nix - Common network configuration and Netbird mesh overlay.
  • secrets.nix - SOPS configuration hooked into central Bitwarden secrets.
  • ssh/ - OpenSSH configuration for root and unprivileged networker.
  • Service modules for backup, ingress, monitoring, Netbird, Authentik, DokuWiki, etc. live directly under modules/; ingress virtual hosts are centralized in modules/ingress/vhosts.nix.
• vars/ - Global Nix metadata (machine addresses, host keys, subnets, SR-IOV allocations, MicroVM topology).
• bitwarden/ - Centralized secret specifications (keys.yaml) and encrypted SOPS secrets (values.secrets.yaml).
• authorized_keys - Root SSH keys synced from Codeberg.
• flake.nix - Flake entrypoint exposing nixosConfigurations, diskoConfigurations, and nixosImages.
• Justfile - Command runner recipes for common operations.

Onboarding Yourself

To get set up to manage these machines:

1. Network access: Join the Netbird mesh network (https://netbird.dma.space). You can use your dma.space account to log in. Machines are accessible via <machine>.wt0.dma.computer. (Tailscale is retained only as a fallback on select hosts).
2. SSH access: Add your username (e.g. yourname for Codeberg or username@https://... for external key URLs) to the # allow: ... line at the top of ./authorized_keys. Then run just sync-ssh-keys, commit the changes, and submit a PR or push to git. Once an infra maintainer merges your changes, you will be able to SSH into all machines.
3. Local environment: Set up Nix and Direnv as described below.

Setting Up Your Local Environment

Install Nix and Direnv.

In the repository root, create and allow the environment:

echo use flake > .envrc && direnv allow

Or manually enter the development shell:

nix develop

Bitwarden CLI Configuration

Fleet secrets are stored in a self-hosted Vaultwarden instance at https://vault.dma.space.

Add your API credentials to .envrc (or export them in your shell):

export BW_CLIENTID="your-client-id"
export BW_CLIENTSECRET="your-client-secret"

Initialize your Bitwarden CLI environment:

just bitwarden-init

This isolates CLI configuration into $XDG_CONFIG_HOME/bitwarden-dma-space (via BITWARDENCLI_APPDATA_DIR) and points bw to https://vault.dma.space. Unlock your vault when performing secret operations:

bw unlock

Deployment Model & Common Operations

Deployments are managed via GitOps (Comin). Machines periodically poll https://git.dma.space/infra/nixos.git on branch main and deploy updates automatically.

Note

Direct manual deployment via just deploy is blocked by default to prevent state drift against GitOps. Prefer pushing changes to Git. Direct deployments require OVERRIDE_GITOPS=1 for emergency recovery.

Common Recipes

Run just to view all available recipes. Everyday operations include:

• just watch - Monitor current deployment commit and status across all machines.
• just eval <machine> - Evaluate a machine's configuration without building.
• just build <machine> - Build a machine's configuration remotely on the target host.
• just ssh <machine> - SSH into root@<host> over Netbird.
• just sync-ssh-keys - Re-sync authorized_keys from Codeberg for allowed users.
• just sync-bitwarden-secrets - Fetch secrets defined in bitwarden/keys.yaml from Bitwarden and encrypt into bitwarden/values.secrets.yaml using age.
• just sync-bitwarden-secrets microvms - Sync MicroVM secrets into microvms/secrets.bitwarden.yaml.
• just machines - List all configured NixOS machines.

Ingress & Virtual Hosts

All public reverse proxy routes and domains are declared in a centralized, flat list in modules/ingress/vhosts.nix. Rather than scattering ingress configuration across individual service modules, edge gateway nodes (colo-komo, colo-fmt2) evaluate this single routing table in Caddy to direct incoming traffic to internal backends.

Upstream Targets

vhosts.nix provides target resolution helpers to decouple external hostnames from internal network topology:

• vmBridgeTarget "<name>" "<portName>": Proxies directly to a MicroVM's internal bridge IP and designated port defined in vars.microvms (e.g. authentik, vaultwarden, dokuwiki).
• machineNetbirdTarget "<name>" "<portName>": Proxies over the Netbird overlay network to a machine host and port defined in vars.machines (e.g. dreamflasher or colocation nodes).

Common Snippets & Conventions

Virtual hosts in vhosts.nix make use of reusable Caddyfile imports:

• import cloudflare_dns: Automates wildcard and domain TLS certificate issuance via Cloudflare DNS-01 challenges.
• import fallback_error_page: Rewrites HTTP 5xx responses to the stylized fallback page if upstream backends are down or unreachable.
• import forward_auth: Enforces Authentik SSO authentication for protected endpoints and admin consoles (e.g. alerts or admin portals).
• import reverse_proxy_insecure_https: Proxies upstream services that serve internal or self-signed TLS certificates.

For documentation on using these snippets, see each import's comment block, which can be found in modules/ingress/caddyfiles.

Dynamic DNS & Node-Specific Routes

DNS records are managed dynamically in modules/ingress/default.nix via Caddy's dynamic_dns plugin using Cloudflare credentials loaded from SOPS.

• Both edge nodes (colo-komo and colo-fmt2) register their public A and AAAA addresses for *.dma.space ("*" = ""). Traffic hitting wildcard subdomains is distributed across both nodes in round-robin.
• If a service should only route to a single ingress node rather than both in round-robin, the subdomain is mapped under that host's name in domains."dma.space":

  domains."dma.space" = {
    "*" = "";
    "colo-komo" = "colo-komo.alerts ramiel netbird llama-cpp.dreamflasher";
    "colo-fmt2" = "colo-fmt2.alerts";
  };
  

  Cloudflare creates explicit A/AAAA records for those subdomains pointing directly to that single host's IP. Because exact DNS records take precedence over wildcards, incoming traffic routes exclusively to the designated host.

Playbooks

Setting Up a New Machine

1. Hardware & Configuration:

   • Create machines/<machine>/configuration.nix.
   • Core modules (e.g. machines/base.nix, sops-nix) are imported automatically by self.lib.nixosSystem.
   • Import necessary modules:

     { self, ... }:
     {
       imports = [
         (self.nixosModules.gitops)
         (self.nixosModules.network)
         (self.nixosModules.ssh)
         (self.nixosModules.secrets)
         (self.nixosModules.hardening) # optional
       ];
     
       gitops.enable = true;
     }
     

   • Register the system in flake.nix under nixosConfigurations.<machine> using self.lib.nixosSystem "<machine>".

2. Disk Layout:

   • Create machines/<machine>/disko.nix using disko (prefer LUKS + Btrfs or ZFS for storage nodes).
   • Register the layout in flake.nix under diskoConfigurations.<machine> = self.lib.diskoConfiguration "<machine>";.

3. Machine Metadata & Host Key:

   • Add the machine entry to vars/machines.nix:

     <machine> = {
       host = "<machine>.wt0.dma.computer";
       machineID = "<32-character-hex-uuid>"; # openssl rand -hex 16
       hostPubKey = "<ssh-ed25519-public-key>";
     };
     

   • Grab the machine's host public key (/etc/ssh/ssh_host_ed25519_key.pub or ssh-keyscan <machine>).
   • Run just sync-bitwarden-secrets so the new machine can decrypt fleet secrets.

4. Installation:

   • Apply disk partitions and install directly from the live environment:

     nix run 'nixpkgs#disko' -- --mode destroy,format,mount --flake '.#<machine>'
     nixos-install --no-channel-copy --flake '.#<machine>'
     

   • If Secure Boot is supported, import (self.nixosModules.secureboot) and configure Lanzaboote with TPM2.

Setting Up a New VM (Proxmox or libvirt)

Use the self-bootstrapping cloud-init image; see images/cloud-init-image/README.md.

Managing Secrets

Secrets are managed declaratively through a centralized mapping file rather than per-machine Bitwarden records.

1. Add Secret to Vaultwarden:

   • Create or locate the item in Vaultwarden (https://vault.dma.space).
   • Store the secret in the password field or a custom text/hidden field.

2. Declare in bitwarden/keys.yaml:

   • Add the key mapping into bitwarden/keys.yaml:

     myservice:
       secret_token:
         $name: "My Service Secrets" # Bitwarden item name
         $field: "secret_token" # Field name (defaults to .login.password if omitted)
     

3. Sync and Encrypt:

   • Ensure bw unlock has been run.
   • Run just sync-bitwarden-secrets.
   • The script pulls each secret, extracts all machine host keys from vars/machines.nix, converts them to age keys via ssh-to-age, and encrypts everything into bitwarden/values.secrets.yaml.

4. Use in NixOS Modules:

   • Ensure the machine imports (self.nixosModules.secrets).
   • Reference secrets using standard sops-nix syntax:

     sops.secrets."myservice/secret_token" = {
       owner = "myservice";
     };
     

   • The secret will be materialized at /run/secrets/myservice/secret_token.
   • Commit and push to Git; GitOps will deploy the changes to target nodes.

Exposing a Service via Ingress

1. Register Port:

   • Ensure the backend service port is declared in vars/microvms.nix (for MicroVM guests) or vars/machines.nix (for bare-metal hosts).

2. Add Virtual Host:

   • Add the host definition into modules/ingress/vhosts.nix:

     "myservice.dma.space".extraConfig = ''
       import cloudflare_dns
       import fallback_error_page
       reverse_proxy * http://${vmBridgeTarget "myservice" "http"}
     '';
     

   • For protected endpoints, add import forward_auth to require Authentik SSO.

3. Node Pinning (Optional):

   • If the service should only be handled by a specific edge node instead of round-robin across both nodes, add the subdomain to that node's entry under domains."dma.space" in modules/ingress/default.nix.

4. Deploy:

   • Commit and push to main; edge gateway nodes will deploy the updated Caddy configuration automatically via GitOps.