Installing TAPPaaS¶

This is the short install guide. It reflects the automated install with mostly default values.

Note this is a major upgrade from the version 1.0 of TAPPaaS stable: The firewall and the NixOS VM template are now prebuilt images that download and boot preconfigured, and the bootstrap scripts chain together — so most steps are "run one command and watch".

Defaults (subnet, hostnames, sizing, passwords, branch, …) work out of the box. Anywhere a default is mentioned, it can be changed — see Appendix: install options.

Before you start: hardware sizing, network plan, domain/DNS, credentials and branch selection are covered in the preparation guide on the documentation site: https://tappaas.org/installation/preparation/. This document assumes preparation is done and concentrates on the install itself.

The four steps¶

Step 1 — First node: Proxmox, firewall, platform¶

1a. Install Proxmox VE 9.1 on the first machine¶

Note TAPPaaS does not support PVE 9.2 yet. Two ways — pick one option, then continue with step 1b:

Option A — preconfigured install media (recommended)¶

Build a USB stick that installs Proxmox with one question asked on the target itself (the boot disk, chosen from the machine's real disks); everything else is answered at build time (email, locale, root password) with TAPPaaS defaults for the rest. The build runs on your laptop — no Proxmox needed and no repo checkout needed: the script is self-contained (on Linux it fetches the Proxmox assistant automatically; on macOS it re-runs itself in a Debian container — Docker required). Fetch it with curl and run it from the directory holding the ISO:

curl -fsSL "https://codeberg.org/TAPPaaS/TAPPaaS/raw/branch/main/src/foundation/cluster/make-install-media.sh" \
     -o make-install-media.sh && chmod +x make-install-media.sh
./make-install-media.sh --iso proxmox-ve_9.1.iso \
       --fqdn tappaas1.mgmt.internal
# prompts for email / country / keyboard / timezone / root password,
# VALIDATES the answer, writes proxmox-ve_9.1-tappaas-auto.iso
dd if=proxmox-ve_9.1-tappaas-auto.iso of=/dev/<usb> bs=4M status=progress

(If you already have — or prefer — a full checkout: git clone https://codeberg.org/TAPPaaS/TAPPaaS.git and run src/foundation/cluster/make-install-media.sh from it instead.)

Boot the target from the stick with the NIC to your upstream router connected — network comes from DHCP, exactly like the PXE flow (the management addressing is applied later by the TAPPaaS install). Pick the boot disk when the console asks; the rest is hands-off and WIPES that disk. Pass --disk <dev> at build time for a zero-keystroke install. ⚠ The stick embeds the root password — treat the media like a credential and rewrite it after use.

Disk standard: the PVE system goes on a dedicated boot disk (ext4/LVM — the default). The tankXY data pools live on other disks and are created later by the platform (Step 1, [⅕] / config-storage.sh), never by the installer.

Option B — stock ISO, manual screens¶

Download the ISO and create a bootable USB per the official guide: https://pve.proxmox.com/wiki/Installation.

On the installer screens: - Network Nic select the one you have connected to the upstream router. Initially this is the Lan port but it will eventually become the "wan" port of the TAPPaaS firewall. For secondary TAPPaaS nodes this is will stay lan port, as these nodes wil connect directly via the switch to the lan side of the firewall we create in the first node. - Hostname (FQDN): tappaas1.mgmt.internal — TAPPaaS uses the internal management domain mgmt.internal, not your public domain. (Your public domain is supplied with --domain in step 1b.) - Email: a working address you actually monitor — Proxmox sends system/health notifications here, and TAPPaaS reuses it as the admin email. - Management interface / IP / netmask / gateway / DNS: must be valid for your existing network — use a free IP (the proxmox installer is not using DHCP), and the real gateway and DNS server of the network this node currently sits on (so it has internet for the bootstrap).

1b. Run the one-shot bootstrap¶

Whichever option you used above, continue here: run the bootstrap from the Proxmox node console/shell.

Notes: - Prefer the console — use the xterm.js shell option in the tappaas1 menu; it gives more scrollback and persistence on the install output. - SSH works too — see Appendix: Installing via SSH for setup steps (known_hosts, locale, tmux). - ⚠ Run from a client that is NOT on 10.0.0.0/24. The cutover puts the management network (10.0.0.0/24) on the node, so a browser/SSH client that sits in that subnet loses its return path to the node and freezes. Drive the install from a client on your install/upstream network, or from a true out-of-band console (IPMI/iKVM) — not from a 10.0.0.x laptop. (The node keeps its install IP throughout, so such a client never loses it.)

REPO="https://codeberg.org/TAPPaaS/TAPPaaS/raw/branch/"; BRANCH="main"
curl -fsSL ${REPO}${BRANCH}/src/foundation/install.sh >install.sh
chmod +x install.sh && ./install.sh "$REPO" "$BRANCH" --name <orgname> --domain "yourdomain.com"

The install re-launches itself inside a tmux session (tappaas-install) so a dropped SSH connection won't abort it — the firewall step reconfigures networking and can briefly cut your session. If you get disconnected, reconnect to the node and run tmux attach -t tappaas-install. (Opt out with --no-tmux.)

Pass two things up front: - --name <orgname> — your organisation / system name (lowercase, ≤15 chars). This is the one name for the whole install: it names the Proxmox cluster, the site.json, the default environment, and (later) the organisation in the identity provider. If omitted you'll be prompted. - --domain — your public domain; the reverse proxy is configured for <service>.yourdomain.com. If omitted you'll be prompted. You don't need the domain's DNS-01 API token yet — that comes in Step 3.

On the first node this runs the whole foundation bring-up end-to-end as a 5-step chain — you run it once and watch:

1. [⅕] Node — Proxmox post-install, the lan/wan bridges (auto-detected: the install NIC with internet becomes WAN; the other, to your downstream switch, becomes LAN), the Proxmox cluster named <orgname>, and the ZFS pools.
2. [⅖] Firewall — downloads and boots the prebuilt OPNsense image at 10.0.0.1, self-configured with unique credentials (no GUI, no installer).
3. [⅗] Gateway cutover — adds this node's management IP 10.0.0.10 and points its default route + DNS at the firewall. Additive and non-disruptive: the upstream IP is kept, no cables move, your session is not dropped.
4. [⅘] Sanity check — gateway, DNS, internet.
5. [5/5] Platform — imports the prebuilt NixOS template and builds the tappaas-cicd mothership (bootstrap.sh clone+nixos-rebuild → reboot → install.sh). The cicd's install.sh then writes the system's configuration, automatically:
   • site.json — the site singleton (nodes, domain, email, repos), via create-site.sh --name <orgname>
   • zones.json — the network zones, via network-manager init
   • the mgmt + default <orgname> environments, via environment-manager add (the minimal-set bootstrap)
   • the foundation modules (cluster, templates, network, tappaas-cicd) and the Caddy reverse proxy

One name, everywhere: <orgname> = the Proxmox cluster name = site.json .name = the default environment name = your organisation name. You set it once with --name; the organisation itself isn't created until Step 4 (rest-of-foundation.sh, after the identity provider is up).

The domain you passed configures the reverse proxy. To stop earlier, pass --skip-firewall or --skip-platform; to drive the in-VM cicd install by hand, see Appendix: install options.

When it finishes, tappaas1 is at 10.0.0.10 behind the firewall and tappaas-cicd owns the platform. Optionally move your admin laptop onto the downstream switch (you'll get a 10.0.0.x lease; Proxmox at https://10.0.0.10:8006, firewall GUI at https://10.0.0.1) — not required, since the node also keeps its upstream IP until you harden it later.

Step 2 — Add additional nodes (optional; skip for single-node)¶

Do this after the first node's bootstrap has finished (cicd is up). Follow-on nodes install over the network, fully unattended — no USB stick, no installer screens. The mothership runs a TTL-limited network-boot trap (node-provisioner): a machine you have registered gets installed when it PXE-boots on the management LAN; every other machine is refused.

Wire the new node's LAN NIC to the downstream switch (the 10.0.0.0/24 management network) and find out which disk is its boot disk — that disk is wiped. (Managed inter-node switch? Configure its VLAN trunks first; see the network section.)

The netboot assets are staged automatically by the first-node bootstrap (prepare-netboot.sh, run at the end of the cicd install) — adding nodes is a latent capability of every TAPPaaS system. After a PVE version upgrade, re-stage once with prepare-netboot.sh --force on the mothership.

Per node — one command:

site-manager node add tappaas2 --pxe

The command walks the whole flow and asks only what it cannot know:

1. Registers the node and arms a TTL-limited PXE trap (auto-disarms; only registered machines are answered).
2. Tells you to network-boot the machine (UEFI PXE, e.g. F11). The node's console asks one question — the boot disk (it lists the machine's real disks; the chosen disk is WIPED, ext4/LVM). Everything else is hands-off: the installer fetches its per-node answer from the mothership and reboots into PVE. The generated root password lands under ~/.node-secrets/ (0600) — the mothership's ssh key is already authorized.
3. Asks the join-time questions with the node's real hardware on screen: the WAN NIC (only if this node will host firewall HA) and the data pool declarations (e.g. tanka1=single:nvme0n1).
4. Runs the node step (served from the mothership — no GitHub needed), moves the node to its standard mgmt IP (tappaasN → 10.0.0.<9+N>), joins the Proxmox cluster, creates the pools, and captures everything into site.json.

Every question can be pre-answered for a fully unattended run:

site-manager node add tappaas2 --pxe --boot-disk sda --mac aa:bb:cc:dd:ee:ff \
    --no-wan --pool 'tanka1=single:nvme0n1'
# --mac also pre-reserves the standard IP, so the node installs onto it directly

Afterwards, fold the new topology into HA + replication:

update-tappaas --force

Machine can't PXE-boot? Write the staged netboot image to a stick — dd if=/home/tappaas/pve-tappaas-netboot.iso of=/dev/<usb> bs=4M on the mothership — and boot from USB instead of pressing F11: the flow (and the node add <name> --pxe command) is otherwise identical, the answer still comes from the mothership over HTTP.

Manual install instead? Install from a stock ISO or the Step 1 Option A stick, give the node its standard mgmt IP (or let it DHCP), then run site-manager node add tappaas2 (no --pxe) — it finds the Proxmox at the node's designated IP and runs the same join + capture pipeline. The underlying tools remain available for surgery: node-provisioner register/enable/disable/status and dhcp-manager pxe/host.

Step 3 — Set up TLS certificates¶

Your domain is already configured (you passed --domain in Step 1). The default TLS strategy (proxyTls: dns01) issues one wildcard certificate per TAPPaaS domain via ACME DNS-01, then binds it to every module's reverse-proxy entry through Caddy's CustomCertificate (issue #254). DNS-01 needs no inbound :80 traffic, so internal-only services get a public cert too.

Why not Caddy's own DNS-01? Since os-caddy 2.0.0 the OPNsense build only ships the Cloudflare DNS provider. TAPPaaS uses os-acme-client (which wraps acme.sh) so you can pick any of the 120 DNS APIs it supports — Cloudflare, deSEC, Hetzner, OVH, Route 53, Namecheap, etc.

Run the operator-driven setup on the mothership:

ssh tappaas@tappaas-cicd      # or its 10.0.0.x address
acme-setup.sh                 # interactive: picks up domain/email, prompts for token

The script: 1. Reads the domain from the default environment file (config/environments/<system-name>.json, domains.primary) and the admin email from site.json (.email). 2. Prompts for your DNS provider (default cloudflare) and its API token (offers to save them to ~/.acme-dns-credentials.txt, mode 600, for re-runs). 3. Provisions an ACME account, a DNS-01 validation, a caddy-reload automation action, and a wildcard certificate (*.<domain> + bare apex) on the firewall — then signs it and waits for issuance (~10–30 s). 4. Stores the issued cert's OPNsense Trust refid in the runtime config/cert-refids.json, keyed by environment name (ADR-007c). Every later proxyTls: dns01 module install reads this refid and binds the wildcard via CustomCertificate automatically.

Cloudflare token scope (recommended): create a custom token at https://dash.cloudflare.com/profile/api-tokens with permissions Zone → Zone → Read and Zone → DNS → Edit, restricted to your domain. Optionally allow only the firewall WAN IPv4 and IPv6 (or skip the IP filter — the token is already zone-scoped).

# Non-Cloudflare? Pass the provider name; the script asks for that provider's fields:
acme-setup.sh --provider hetzner          # deSEC, hetzner, ovh, route53, ...
# Or seed creds non-interactively (chmod 600):
cat >~/.acme-dns-credentials.txt <<EOF
provider=cloudflare
dns_cf_token=YOUR-TOKEN-HERE
EOF
chmod 600 ~/.acme-dns-credentials.txt
acme-setup.sh

To test before going live, add --staging once (Let's Encrypt staging = untrusted certs, no rate limits); then re-run without --staging for the trusted prod cert (the script swaps the cert in place — Caddy picks up the new one automatically via the registered caddy-reload action).

If you'd rather not use the wildcard for a particular service (e.g. you want a per-domain cert via HTTP-01 because the service is publicly reachable on :80 and you don't want it to share the wildcard), set proxyTls: http01 on that module. The two strategies coexist per-module.

Skipping Step 3 is fine if you only use TAPPaaS internally — every service stays reachable on the LAN; only the public HTTPS endpoint of dns01 modules will lack a certificate until you run acme-setup.sh. (The public domain now lives per-environment, not in site.json. To change the default environment's domain later: environment-manager modify <system-name> --domain <yourdomain>, then re-run acme-setup.sh.)

Step 4 — Complete the foundation¶

From here on you work from the cicd mothership (ssh tappaas@tappaas-cicd). One command does two things:

rest-of-foundation.sh

1. Installs the remaining foundation modules in order — backup → identity → logging — then runs a final system update + tests.
2. Bootstraps your people domain — once the identity provider (Authentik) is up and config/people/ is still empty (first install), it creates the organisation <orgname> (the same name from Step 1), the users group and your installer user (from site.json's email), and pushes them into Authentik. So this is where your organisation is actually created — the earlier --name only reserved the name; the org entity is materialised here.

It's idempotent — safe to re-run if a module needs attention (the people bootstrap is skipped once config/people/ exists, so it never disturbs people you've added). When it finishes you'll see a "🎉 your TAPPaaS foundation is installed" summary (nodes, firewall, mothership, domain/TLS, modules, organisation).

Prefer to do it by hand? install-module.sh is on the PATH but reads ./<module>.json from the current directory, so cd in first:

cd ~/TAPPaaS/src/foundation/backup   && install-module.sh backup     #  Proxmox Backup Server
cd ~/TAPPaaS/src/foundation/identity && install-module.sh identity   #  Identity provider
cd ~/TAPPaaS/src/foundation/logging  && install-module.sh logging    #  Loki / Grafana / Promtail

(Module sizing/zones are defaults — see appendix.)

Next steps¶

The foundation is complete. Everything from here is covered on the documentation site: add environments, add a satellite, add stacks (each module's install guide), and day-to-day operation — start at https://tappaas.org/installation/.

Network — cutting over to the firewall¶

Putting the firewall (a VM on tappaas1, at 10.0.0.1) inline as the gateway is done for you by the bootstrap (Step 1, [⅗]) — config-network.sh --swap-gateway. You normally never run it by hand; this section explains what it does. Each node has two NICs, wired at install time and left in place:

• WAN — the NIC connected to your upstream router (the one with internet at install time). It's the firewall VM's uplink. On the first node the node also keeps its original install IP here.
• LAN — the NIC connected to your downstream switch (the 10.0.0.0/24 management LAN). VLAN-aware bridge; holds the node's mgmt IP 10.0.0.10.

The cutover is additive — nothing is removed, nothing is unplugged:

• --swap-gateway makes sure 10.0.0.10 is on the LAN side, points the node's default route + DNS at the firewall (10.0.0.1), and restarts corosync so it binds to the mgmt IP. The upstream IP is kept, so the node (and your existing session) never lose connectivity.
• Because both IPs remain, you can keep managing the node where you are, or move your admin laptop onto the downstream switch (you'll get a 10.0.0.x lease; firewall GUI at https://10.0.0.1). Moving is optional until you harden.

   internet / upstream router
        │
   [tappaas1 WAN NIC] ─────────────► OPNsense WAN  (DHCP from upstream)
   │                                 node also keeps its install IP here
   ┌───────────────────────────────────────────────┐
   │  tappaas1 (Proxmox)                             │
   │    OPNsense VM:   lan → 10.0.0.1  (the gateway) │
   │    node mgmt:     lan → 10.0.0.10               │   ← default route now via 10.0.0.1
   └──[tappaas1 LAN NIC]──────────────────────────────┘
        │
   [ downstream switch ] ── admin laptop  (optional; 10.0.0.x by DHCP)

Later hardening: once you're managing via the mgmt net (or netbird), run config-network.sh --drop-upstream on a node to remove its upstream IP, so Proxmox is reachable only behind the firewall.

The switch (3-node clusters): the lan bridge is VLAN-aware — it carries the management network untagged (10.0.0.0/24) plus every TAPPaaS VLAN tagged as a trunk. For VMs on different nodes to reach each other on a VLAN, the switch between the nodes must pass those tagged frames. How that works depends on the switch type:

• Unmanaged switch — passes all frames transparently, tagged or untagged, so the TAPPaaS VLANs cross between nodes out of the box, with no configuration. This is the simplest choice for a 3-node cluster.
• Managed switch — by default forwards only untagged traffic; it drops or ignores VLAN-tagged frames until you configure (manage) it. You must set the inter-node ports as trunks carrying the TAPPaaS VLAN tags (plus the untagged management network) before adding nodes — otherwise cross-node VLAN traffic won't pass even though everything else looks fine.

(Single node: no switch needed for VLANs — all VLANs live inside the one node; the lan NIC can go to your existing LAN or any switch.)

Upstream options: the firewall WAN can sit behind your ISP router (port-forward), in bridge mode, or replace the ISP router directly.

Appendix: install options¶

Defaults are chosen so the commands above "just work". Override as needed:

Field definitions for module JSON are in src/foundation/schemas/module-fields.json. Network zones/VLANs come from the git template src/foundation/tappaas-cicd/manager/network-manager/zones.json, which network-manager init transforms into the live, per-installation config/zones.json at install time.

Appendix: Installing via SSH¶

The main install guide recommends the Proxmox console, but SSH works fine with proper preparation. The key is using tmux so the install continues if your connection drops during network reconfiguration.

1. Generate and install an SSH key (from your workstation)¶

If you don't already have an SSH key pair:

# Generate a new key (accept defaults, optionally set a passphrase)
ssh-keygen -t ed25519 -C "your-email@example.com"

Reinstalling? If you previously installed TAPPaaS on the same IP, SSH will refuse to connect due to a changed host key (WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!). Remove the old entry first:

ssh-keygen -R <proxmox-ip>

# Copy the public key to the Proxmox node (use the node's current IP)
ssh-copy-id root@<proxmox-ip>

Now you can SSH without a password:

ssh root@<proxmox-ip>

2. Fix locale warnings¶

Fresh Proxmox installs may show Perl locale warnings. Fix them:

# Quick fix for current session
export LC_ALL=en_US.UTF-8
export LANGUAGE=en_US.UTF-8

# Permanent fix (then start a new shell)
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen && locale-gen && \
  update-locale LC_ALL=en_US.UTF-8 LANGUAGE=en_US.UTF-8

3. (Optional) Use tmux for session persistence¶

tmux keeps your session alive if SSH disconnects during network reconfiguration. This is optional but recommended:

apt update && apt install -y tmux
tmux new -s install

If your SSH connection drops, reconnect and reattach:

ssh root@<proxmox-ip>
tmux attach -t install

Useful tmux commands:

• Ctrl+b d — Detach (leave session running in background)
• Ctrl+b [ — Scroll mode (arrow keys to scroll, q to exit)

4. Run the install¶

Run the bootstrap command from Step 1:

REPO="https://codeberg.org/TAPPaaS/TAPPaaS/raw/branch/"; BRANCH="main"
curl -fsSL ${REPO}${BRANCH}/src/foundation/install.sh >install.sh
chmod +x install.sh && ./install.sh "$REPO" "$BRANCH" --name <orgname> --domain "yourdomain.com"

Even if your SSH drops during network cutover, the install continues inside tmux. Reconnect with tmux attach -t install to see progress.

5. After install¶

Once complete, you can access:

• Proxmox UI: https://10.0.0.10:8006 (or the node's upstream IP)
• Firewall GUI: https://10.0.0.1
• CICD Mothership: ssh tappaas@tappaas-cicd (or ssh tappaas@10.0.0.143)