One operating model covers the full GPU cluster lifecycle

BareStax turns an Ubuntu server into a verified BareStax Core node, then keeps its declared state and health under continuous control. The same architecture handles initial installation, daily operation, expansion, upgrades, recovery, and removal on connected or air-gapped networks.

The control plane coordinates every cluster role

The head node runs barestax-server, the API, controllers, operator console, PostgreSQL, the BareStax Core controller, job accounting, the local package repository, the OCI registry, and the audit record. It can be a GPU-free server or a virtual machine on your institution's virtualization platform.

Each GPU node runs barestaxd and Apptainer. The agent initiates its connection to the control plane, so the control plane does not push commands into the node. A login node provides SSH, the researcher portal, and job submission.

On the connected side, barestax-bundler collects and signs release bundles for transfer into an isolated network. All internal cluster connections use mTLS.

BareStax is written in Go. Each component ships as a single binary: barestax-server, barestaxd, barestaxctl, barestax, and barestax-bundler. The console lives inside barestax-server.

Internet sidebarestax-bundlerAir-gap boundaryapproved portablemedia or data diodeHead nodebarestax-serverAPIcontrollersconsolePostgreSQLBareStax Core controllerjob accountinglocal package repositoryOCI registryaudit recordLogin noderesearcher portal, SSH, barestaxmTLSGPU nodesbarestaxdApptainerGPUNVLinkInfiniBandThe agentinitiates theconnection tothe controlplanemTLSrelease bundleInternet sidebarestax-bundlerrelease bundleAir-gap boundary:approved portableHead nodebarestax-serverAPIcontrollersconsolePostgreSQLBareStax Core controllerjob accountinglocal package repositoryOCI registryaudit recordmTLSThe agent initiates theconnection to the controlplaneGPU nodesbarestaxdApptainerGPUNVLinkInfiniBandLogin noderesearcher portal, SSH, barestax

Day 0 establishes the control plane

The initial command creates the cluster identity and installs the control plane on the head node. You then connect LDAP, Active Directory, local accounts, OIDC, or SAML; configure built-in TOTP; and define network and NFS or S3 storage targets.

BareStax also checks the alert chain before compute nodes enter service. A test alert must reach the on-call path and return its acknowledgment code.

barestax-server init --cluster-name <name> --endpoint <fqdn> --key-backup <file>
barestaxctl alert test
barestaxctl doctor

Day 1 turns installed servers into schedulable nodes

Preflight covers the firmware floor, disk, Secure Boot, and connectivity before registration. A join token lets the server register on a provisional basis. After approval, BareStax inventories GPU, NVLink, InfiniBand, PCIe, and NUMA topology and selects the matching profile.

The selected hardware profile installs a tested set of kernel, DOCA-OFED, NVIDIA driver, CUDA user libraries, Fabric Manager, DCGM, NCCL, Apptainer, and system settings. BareStax pins the kernel so unattended operating system updates cannot split the cluster across incompatible states.

GPU, NVLink, and InfiniBand checks plus two-node NCCL validation must pass before the node becomes ready in BareStax Core. A failed check leaves the node outside the work queue with a coded explanation.

barestaxctl token create
barestaxd join --token-file <path>
barestaxctl node approve <name>
barestaxctl get nodes -o wide
01Rack the server andinstall theoperating system02One command03Approval04Inventory, profile,compatible stack05GPU, NVLink,InfiniBand, andtwo-node NCCLvalidation06Ready in BareStaxCore01Rack the server and install theoperating system02One command03Approval04Inventory, profile, compatiblestack05GPU, NVLink, InfiniBand, andtwo-node NCCL validation06Ready in BareStax Core

Day 2 keeps declared state and observed health aligned

BareStax compares each node with its assigned profile. Non-disruptive drift is corrected in service; a disruptive change is reported unchanged and waits for an approved maintenance window.

Health evaluation spans GPU, fabric, host, critical services, and shared storage. Stale telemetry becomes unknown rather than healthy.

A faulty node stops receiving new work while its active job continues. Policy controls staged recovery; a cluster-wide fault halts escalation.

Desired state (profile)Observed state (node)kernelpinnedpinnedDOCA-OFEDtested settested setNVIDIA drivertested settested setCUDAtested settested setFabric Managertested settested setDCGMtested settested setNCCLtested settested setApptainertested settested setsysctlvm.max_map_count=262144vm.max_map_count=65530Corrected without interrupting the runningjobDesired state(profile)Observed state(node)kernelkernelDOCA-OFEDDOCA-OFEDNVIDIA driverNVIDIA driverCUDACUDAFabric ManagerFabric ManagerDCGMDCGMNCCLNCCLApptainerApptainersysctl 262144sysctl 65530Corrected without interrupting the running job

Upgrades respect running work and disruption budgets

Signed release bundles contain compatible component sets and pinned repository snapshots. Running jobs are preserved.

Rollout begins on one canary node, then continues within the configured disruption budget. The control plane follows the same signed path.

The API exposes desired state and observed status

Resources follow a declarative model: your team writes the specification, while BareStax writes status from the cluster. A live watch stream feeds the console and CLI. When the stream breaks, clients mark data as stale.

Plans show the proposed change before application, and dry-run checks it without modifying the cluster. The web console and command line call the same API, so an operation has the same policy and audit path in either interface.

barestaxctl plan -f <file>
barestaxctl apply -f <file>
barestaxctl operation list

Running jobs outlive a control plane interruption

GPU nodes retain their last known safe state if the control plane becomes unavailable. Local health gates continue to prevent a faulty node from receiving work, while jobs already running remain active.

Backups include identity, PKI, the database, and BareStax Core state. Recovery restores the control plane to clean hardware and prevents the former head node from answering afterward. Operators can inspect node, storage, job, tenant, and service impact during the recovery process.

From Day 0 install to node removal