One operating model covers the full GPU cluster lifecycle
BareStax turns an Ubuntu server into a verified BareStax Core node, then keeps its declared state and health under continuous control. The same architecture handles initial installation, daily operation, expansion, upgrades, recovery, and removal on connected or air-gapped networks.
The control plane coordinates every cluster role
The head node runs barestax-server, the API, controllers, operator console, PostgreSQL, the BareStax Core controller, job accounting, the local package repository, the OCI registry, and the audit record. It can be a GPU-free server or a virtual machine on your institution's virtualization platform.
Each GPU node runs barestaxd and Apptainer. The agent initiates its connection to the control plane, so the control plane does not push commands into the node. A login node provides SSH, the researcher portal, and job submission.
On the connected side, barestax-bundler collects and signs release bundles for transfer into an isolated network. All internal cluster connections use mTLS.
BareStax is written in Go. Each component ships as a single binary: barestax-server, barestaxd, barestaxctl, barestax, and barestax-bundler. The console lives inside barestax-server.
Day 0 establishes the control plane
The initial command creates the cluster identity and installs the control plane on the head node. You then connect LDAP, Active Directory, local accounts, OIDC, or SAML; configure built-in TOTP; and define network and NFS or S3 storage targets.
BareStax also checks the alert chain before compute nodes enter service. A test alert must reach the on-call path and return its acknowledgment code.
barestax-server init --cluster-name <name> --endpoint <fqdn> --key-backup <file>
barestaxctl alert test
barestaxctl doctorDay 1 turns installed servers into schedulable nodes
Preflight covers the firmware floor, disk, Secure Boot, and connectivity before registration. A join token lets the server register on a provisional basis. After approval, BareStax inventories GPU, NVLink, InfiniBand, PCIe, and NUMA topology and selects the matching profile.
The selected hardware profile installs a tested set of kernel, DOCA-OFED, NVIDIA driver, CUDA user libraries, Fabric Manager, DCGM, NCCL, Apptainer, and system settings. BareStax pins the kernel so unattended operating system updates cannot split the cluster across incompatible states.
GPU, NVLink, and InfiniBand checks plus two-node NCCL validation must pass before the node becomes ready in BareStax Core. A failed check leaves the node outside the work queue with a coded explanation.
barestaxctl token create
barestaxd join --token-file <path>
barestaxctl node approve <name>
barestaxctl get nodes -o wideDay 2 keeps declared state and observed health aligned
BareStax compares each node with its assigned profile. Non-disruptive drift is corrected in service; a disruptive change is reported unchanged and waits for an approved maintenance window.
Health evaluation spans GPU, fabric, host, critical services, and shared storage. Stale telemetry becomes unknown rather than healthy.
A faulty node stops receiving new work while its active job continues. Policy controls staged recovery; a cluster-wide fault halts escalation.
Upgrades respect running work and disruption budgets
Signed release bundles contain compatible component sets and pinned repository snapshots. Running jobs are preserved.
Rollout begins on one canary node, then continues within the configured disruption budget. The control plane follows the same signed path.
The API exposes desired state and observed status
Resources follow a declarative model: your team writes the specification, while BareStax writes status from the cluster. A live watch stream feeds the console and CLI. When the stream breaks, clients mark data as stale.
Plans show the proposed change before application, and dry-run checks it without modifying the cluster. The web console and command line call the same API, so an operation has the same policy and audit path in either interface.
barestaxctl plan -f <file>
barestaxctl apply -f <file>
barestaxctl operation listRunning jobs outlive a control plane interruption
GPU nodes retain their last known safe state if the control plane becomes unavailable. Local health gates continue to prevent a faulty node from receiving work, while jobs already running remain active.
Backups include identity, PKI, the database, and BareStax Core state. Recovery restores the control plane to clean hardware and prevents the former head node from answering afterward. Operators can inspect node, storage, job, tenant, and service impact during the recovery process.