Documentation / 4. Nodes

Chapter 4

Nodes

Registry, benchmarks, monitoring, per-role visibility.

A node in the hub is a customer-owned server registered for management. The hub doesn’t own the hardware; it manages the OS-level services on the customer’s box via SSH, with the customer’s explicit consent (transparency modal on first sign-in).

5.1 · Registering a node

From /admin/nodes/new, the operator enters the target’s host, SSH user, and a one-time password. The hub:

  1. generates a per-node ed25519 SSH keypair,
  2. installs the public half into ~/.ssh/authorized_keys on the target (tagged ah-hub:email@host:shortid for greppable provenance),
  3. stores the private half in the hub’s libsodium- encrypted vault,
  4. discards the one-time password. Every subsequent interaction uses the key; the password never leaves RAM.

5.2 · Benchmark & ServerScore

Each node runs a benchmark job that measures CPU, disk (read/write/latency), RAM, and network throughput against calibrated reference loads. Results combine with a commitment-based provisioning bonus and a hardware-class multiplier (bare-metal × 1.15, VPS × 1.0, container × 0.9) into a single ServerScore. Unbounded. ServerScore drives the per-second Stoicism accrual rate — see §6 The Stoic System.

5.3 · Registry view

/admin/nodes shows the full hub fleet to every signed-in user (every tier, not just Ancient). Columns: label, host, role, chainweb tip (live cut height from the 30s tip- poller cache), ServerScore, owner, last-test status. Foreign nodes (owned by someone else) are dimmed and not click-through for non-Ancient callers.

Owned nodes drill into a tabbed per-node view: Overview, Chainweb, Scoring, Jobs, Maintenance.

5.3a · SEED chip placement (v.G.1.4b Iris)

The fleet-wide network-tip-reference designation (is_tip_reference) is a per-chainweb flag, not per-server. The Iris v.G.1.4b release pins the SEED chip to the chainweb cluster-id label always, never on the parent shell — mirroring the MINING chip’s render-site contract:

  • Full-host installs — the parent server row IS the chainweb container, but the SEED chip still renders on the chainweb cluster-id sub-line (one level deeper than the server label). The parent shell label never carries SEED in any topology.
  • Segregated hosts — the parent label hosts N chainweb children. The SEED chip renders on whichever child row is the seed-eligible one. If a segregated parent shell has the flag set but no child does (a stale state from before the v.G.1.4b API guard landed), an inherited-fallback chip surfaces on chainwebs[0] with a tooltip nudging the operator to re-pick a specific container in Tip References.

Toggling SEED on a node is an Ancient-admin operation under Overview → Maintenance → Network Tip Reference; the per-chainweb designation flows through the worker’s 15-second tip- polling round-robin into fleet_metrics.network_tip. The Tip Reference Menu refuses to set the flag on chainweb-less rows (segregated parent shells, Tunneler-only relays) — those rows are no longer selectable, and the toggle endpoint returns HTTP 400 if asked to enable them by URL.

5.4 · Monitoring

Every node exposes a Monitoring tab backed by netdata (installed by a hub job on first run). The hub also records periodic capacity snapshots so the Runway forecastcan predict when a node’s committed GB will be reached given its daily storage growth.

5.5 · Moving data to a bigger drive

When an operator installs a second, larger drive to an existing node, they can move the chainweb data from the original (small) drive to the new (big) drive in place. The hub automates this from Node → Chainweb → Migrate:

  1. Hub probes the remote filesystems, lists candidate mounts that have enough free space for the current StoaNode/ tree plus 10 % headroom, and flags mounts that are too small or happen to be the current install location.
  2. Operator picks a candidate (or types a custom path) and fresh-confirms. The hub stops the chainweb container,rsyncs the whole StoaNode/ tree (preserving hardlinks so RocksDB backups stay intact), regenerates docker-compose.yml at the new location, and brings the container back up.
  3. The original tree is notdeleted — it’s renamed to StoaNode.PRE-MIGRATE.<timestamp>on the source drive so the operator has a trivial rollback path for hours after the fact. Removing it is a manual decision once the new install is verified healthy.

Afterwards the operator can raise committed_gbon the scoring card to claim the new drive’s capacity. The hub’s stored provision_path and stoachain_data_path are updated atomically on success; on failure at any step, the source tree stays usable at its old location.

Only docker-supervised installs are supported by the automated flow today. Screen / systemd installs with a single-drive layout can still be migrated manually with the same rsync recipe and a compose regeneration; the hub tutorial is the pilot.

5.6 · Onboarding a raw drive

Servers regularly arrive with extra NVMe slots populated but their disks left unformatted by the provider — the kernel sees the device, the OS sees nothing. The hub’s Overview → Status tab now lists those raw drives in an Unformatted drives card and offers a one-click Format ext4 + mount action: the hub wipes the device, runs mkfs.ext4, appends a UUID-keyed line to /etc/fstab (with nofail and x-systemd.device-timeout=10s so a missing drive never drops the box into emergency mode), creates the mountpoint, and mounts the new filesystem at the next free /mnt/data-N slot.

Destructive — the confirmation dialog reads “this WILL ERASE all data” for a reason. The handler runs a tight pre-flight (no existing filesystem, no child partitions, not already mounted, ≥ 50 GB, not removable / loop / zram / dm-) before issuing any destructive command, so an accidental click on a system disk is refused with a clear message rather than executed.

5.7 · Unmanaging a node

When an operator removes a node from the hub, the hub makes a final SSH call to remove its own public keyfrom the target’s authorized_keys. The operator can verify afterwards by inspecting the file — the hub commits to losing its own access on unmanage. The node row stays in the orphans audit for a period so any mistakes can be forensically investigated, then is pruned.

5.8 · DNS hostnames

A server can carry more than one DNS hostname pointing at it. The Server DNS card on a server’s Overview tab lists every hostname configured for the box — each row shows the hostname, its kind (manual A-record updated by you, or DuckDNS auto-updated by a systemd timer on the box), and a small set of row actions. Operators of nodes that upgraded from the previous one-hostname-per-server model keep their existing hostname unchanged — it lands in the new list as a single row, no operator action required.

Exactly one hostname per server is the primary: the row shows a gold uppercase PRIMARY pill chip in place of a Set primaryaction. Newly installed chainweb containers pre-select the primary in the install wizard’s hostname picker; clicking Set primary on any non-primary row moves the chip and changes that default. Existing containers don’t move — they keep whichever hostname they were installed with until the operator repoints them explicitly.

Per-container hostname pick

At install time, Step 6 of the chainweb install wizard shows a <select>populated with every hostname on the parent server. The pre-selection is the server’s primary; the operator can override and pick any other hostname. The chosen hostname becomes both the cert subject (CN, used for the Let’s Encrypt issuance) and the value the container starts with for --p2p-hostname. The wizard also shows which issuance method will be used — HTTP-01 for manual A-records, DNS-01 via DuckDNS for DuckDNS hostnames — so there are no surprises when certbot runs.

Child-node inheritance

Tunnelee and self-container child nodes do not own their own hostname list — they read the canonical parent server’s list. The Server DNS card on a child node renders the parent’s hostnames read-only, with an “(inherited from <parent label>)” annotation and a deep-link to the parent’s Overview. Adding, removing, or repointing the primary all happen on the parent — the child reflects whatever the parent says, with no chance of drift.

Removing a hostname

A hostname currently in use by a chainweb container cannot be removed. The remove action returns a refusal listing every referencing container with deep-links — the operator must either repoint those containers (re-run the install wizard’s hostname-pick step on each, or use the Change p2p-hostnameaction on each container’s Chainweb tab) or remove them, then retry. Removing the last hostname on a server is allowed but requires explicit confirmation, since it leaves the box with no operator-visible address until a new one is added.

DuckDNS continuity

Operators with a DuckDNS hostname configured before the multi-DNS upgrade keep their setup working unchanged. The per-box systemd refresh timer continues to push the box’s public IP to DuckDNS on its existing schedule, the subdomain continues to update, and certificate issuance continues to use DNS-01 via DuckDNS for any container whose chosen hostname is the DuckDNS one. There is no forced action, no re-bootstrap, no re-cert. The only visible change is that the Server DNS card now renders the existing hostname as a list of one with the gold PRIMARY pill on it, and exposes the Add hostname button so additional hostnames can be attached when convenient.

See § multi-dns release notes for the operator-facing diff that landed with this version.