cluster — Design notes¶

Implementation and reference detail for the cluster module. For the catalog entry see README.md; for installation see INSTALL.md; for test coverage see TEST.md.

Canonical online instructions: https://tappaas.org/installation/foundation/cluster/

Capabilities¶

A consumer module opts in via dependsOn, e.g. ["cluster:vm", "cluster:ha"]. A guest is either a VM or a container, never both.

Files¶

install.sh — phases and flags¶

install.sh runs, in order:

1. Base post-install (first run only): disable enterprise repos, enable pve-no-subscription, remove the subscription nag, enable HA services, install helper scripts + powertop/smartmontools + SSD lifecycle, and apt dist-upgrade. Recorded by /var/log/tappaas.step1.
2. Network phase → config-network.sh
3. Cluster phase (create or join)
4. Storage phase → config-storage.sh
5. A summary of bridges, pools (with redundancy warnings) and cluster state. It also writes ~/tappaas/.cluster-role (created/joined/member/standalone) for the foundation/install.sh orchestrator, which only chains the firewall + platform steps on the node that created the cluster.

The base step is skipped on re-runs (delete /var/log/tappaas.step1 to force it); the three config phases run every time and are individually idempotent.

install.sh [REPO] [BRANCH] --name <orgname>
           [--cluster|--join|--no-cluster] [--skip-network] [--skip-storage]
           [--lan-port <if>] [--wan-port <if>] [--pool <name=topo:disks>]...
           [--non-interactive]

--skip-firewall, --skip-platform and --domain are accepted for back-compat but ignored here — those phases belong to foundation/install.sh.

Default cluster behaviour (no flag) is auto: tappaas1 creates the cluster, every other node joins it.

Network (phase 2)¶

config-network.sh (issue #141) establishes the TAPPaaS bridge model:

• lan — VLAN-aware bridge (bridge-vids 2-4094) carrying the management network (untagged) plus every TAPPaaS VLAN as a trunk to the switch. Holds this node's management IP. The OPNsense firewall VM and all guest VLAN interfaces attach here.
• wan — plain bridge for the upstream/ISP uplink (the firewall VM's WAN).

It lists the physical ports with MAC / link state / speed so you can tell them apart, you pick which port is LAN and which is WAN, and it rewrites /etc/network/interfaces accordingly. The management IP/gateway default to the values already on the node (set during the Proxmox install).

Lockout protection: applying the change arms a 90-second auto-rollback. After it applies you must type keep to make it permanent; if you get disconnected or do nothing, the node reverts to the previous working config.

It is also the script behind the bootstrap's gateway cutover (--swap-gateway, additive: keeps the upstream IP, points default route + DNS at the firewall) and the later hardening step (--drop-upstream). It can be re-run any time to reassign ports.

config-network.sh [--lan-port <if>] [--wan-port <if>] [--mgmt-ip <CIDR>]
                  [--gateway <ip>] [--no-rollback] [--dry-run|--apply]
                  [--non-interactive]

Cluster (phase 3, runs before storage)¶

Runs before storage — a ZFS pool created on a standalone node (before it is a cluster member) is not usable as HA-failover storage in Proxmox, so pools must be created while the node already belongs to the cluster.

Node management IPs follow tappaasN → 10.0.0.<9+N> (max 9 nodes — the firewall reserves DNS + static IPs for tappaas1–tappaas9 only; higher numbers abort).

In --non-interactive mode a join cannot supply the password, so the script prints the pvecm add tappaas1.mgmt.internal command for you to run.

Storage (phase 4)¶

config-storage.sh builds the TAPPaaS ZFS data pools and registers them with PVE storage. Naming convention: tankXY where X is the tier/type (a = primary/fast, b, c, …) and Y a sequence number.

Disk selection rules:

• The boot disk (the disk backing / and /boot/efi) is never offered and never touched. ZFS zvols and device-mapper/loop devices are excluded.
• Every other disk is offered, including disks already in an existing tanka1/b1/c1 — tagged [in zpool …] — so a machine that used to belong to another TAPPaaS cluster can be wiped and re-provisioned.
• Any disk that already contains data requires an explicit confirmation before it is wiped. Pools are created with ashift=12, autotrim=on, compression=lz4, atime=off.

config-storage.sh [--pool <name>=<topology>:<disk>[,<disk>...]] [--list]
                  [--yes] [--non-interactive] [--no-pve-register]

Interactively, it offers to build tanka1, tankb1, tankc1 in turn; for each you pick disks and a topology (existing pools are skipped).

Storage design decisions¶

The rationale for the ZFS-based storage model and the pool tiers.

ZFS as the storage manager. ZFS meets the criteria for an efficient, scalable, redundant, flexible and trustworthy storage solution, combined here with a standard setup of snapshots and replication across cluster nodes. It gives TAPPaaS enterprise-grade storage on commodity disks without dedicated (costly) SAN/NAS hardware — in small to medium deployments this can cut storage hardware cost by up to 50%. The design stays hardware-agnostic on SSD vs HDD and caching layout. Growth paths: add disks to a pool, add pools to a node, add nodes to the cluster. Redundancy is layered: ZFS RAID within a node, snapshot + replication across nodes (cluster:ha), and backup between local and remote installations (the backup module).

Known limitation — no synchronous cross-node replication. The cluster:ha service uses asynchronous ZFS replication (default schedule */15, i.e. up to 15 minutes of data loss on failover; a module can tighten this via its replicationSchedule). Providing a synchronous option (Ceph, Garage S3) is a roadmap item, recommended only for large installations.

Pool tiers¶

tankXY: X is the tier, Y a sequence number. Pools mount at /<poolname> (e.g. /tanka1). The tier meanings (also printed by config-storage.sh):

Letters d, e, … remain free for specialized storage characteristics.

High Availability¶

HA keeps services running when a component fails. The cluster module covers the failure classes it owns; the network-layer ones (firewall failover, load-balancing, internet/DNS survival) belong to the network module.

A cluster is optional (a single node has no fail-over) but recommended beyond a small setup. Outside the cluster's remit: UPS / power paths are site-specific and out of scope; OPNsense failover, reverse-proxy load-balancing, and internet-outage DNS survival are the network module's concern.

Adding more nodes¶

The supported flow is site-manager node add tappaasN --pxe from the mothership (see INSTALL.md and the repo-root INSTALL.md §2.2): a TTL-limited PXE trap installs Proxmox unattended, then the node step runs served from the mothership, joins the cluster and creates the declared pools. A manual install with the stock ISO followed by site-manager node add tappaasN (no --pxe) is equivalent.

update.sh (run from the mothership) keeps the provisioners and zones.json in sync across all nodes.

Node hardware quirks¶

Minisforum MS-S1 MAX — Realtek RTL8127 10GbE (issue #308)¶

The MS-S1 MAX's two 10GbE ports are Realtek RTL8127 [10ec:8127]. The in-tree r8169 driver fails to re-initialise them across a warm/soft reboot — the NIC drops off the PCIe bus and only a full power cycle brings it back. (Same chipset bug seen on the NVIDIA DGX Spark.) Intel-igc nodes are unaffected.

The fix is codified in setup-realtek-nic.sh (run by install.sh at bootstrap and re-asserted by update.sh every cycle; hardware-gated, idempotent): install Realtek's r8127 DKMS driver (vendored, SHA256-pinned, in assets/) and blacklist r8169 — but only after the r8127 module is confirmed to build and load, so a future kernel that can't build it never leaves the node driverless.

Two steps the OS cannot do for you (the script detects and instructs):

1. Disable Secure Boot in the BIOS — the unsigned DKMS module will not load otherwise (or MOK-sign it). The MS-S1 MAX ships with Secure Boot enabled.
2. One power cycle the first time — the node boots on r8169; only a full power cycle (drain), not a warm reboot, switches cleanly to r8127. After that, warm reboots (incl. the #275 automated kernel-reboot pass) are safe.

Because update.sh re-runs the enforcer every cycle and DKMS rebuilds r8127 for each new kernel, ordinary updates maintain the fix rather than overwrite it; a from-scratch reinstall re-applies it automatically via install.sh.

Related issues¶

• 140 — automate cluster create/join in install.sh¶

• 141 — config-network.sh (lan/wan bridge setup, gateway cutover)¶

• 175 — robust downloads (fetch()): a failed download is now fatal, never a silent¶

  0-byte file reported as success

• 275 — controlled HA node reboots (reboot-node.sh / reboot-cluster.sh)¶

• 308 — Realtek RTL8127 NIC driver fix for MS-S1 MAX (setup-realtek-nic.sh)¶