Cluster operations stay consistent from host configuration to job execution
BareStax manages the layers that determine whether a GPU node can run production work: operating system settings, drivers, fabric, containers, job scheduling, health, storage access, and recovery.
The node stack is declared as a tested set
-
Profile contents
A node profile includes the kernel, DOCA-OFED, NVIDIA driver, CUDA user libraries, Fabric Manager, DCGM, NCCL, Apptainer, sysctl values, hostname, and time synchronization. BareStax distributes components only in combinations tested together.
-
Hardware comes before profile selection
The node agent inventories GPU, NVLink, InfiniBand, PCIe, and NUMA topology, then BareStax matches that inventory to the appropriate profile.
-
Preflight checks
Preflight rejects a node before registration if its firmware floor, disk, Secure Boot state, or connectivity fails. GPU, NVLink, and InfiniBand checks plus two-node NCCL validation still have to pass before BareStax Core accepts work.
-
Drift handling
Manual changes and unexpected package or setting changes appear as drift. Non-disruptive drift is corrected while the node remains in service. A disruptive change, such as a kernel change, is reported without modification and waits for an approved operation in a maintenance window.
BareStax Core configuration follows cluster resources and policy
-
Central configuration arrives with accounting
BareStax produces and centrally distributes the BareStax Core configuration with job accounting; queues follow GPU classes, with separate CPU queues, tenants, quotas, and fairshare.
-
CPU follows the GPU allocation
The scheduler assigns the CPU share automatically per GPU. Multi-node training receives the fabric configuration needed to reach the network's actual bandwidth.
-
Submission paths
Teams can submit directly from the command line, choose a
barestax runtemplate for batch work, array jobs, multi-node torchrun training, or interactive sessions, or use a form in the researcher portal. -
Service and preemptible queues
Preemptible jobs are available, and jobs submitted through
barestax runtemplates return safely to the queue after a node fault. Requeue behavior for jobs submitted directly from the command line remains the researcher's responsibility. Long-running model services use dedicated queues, service registration, and protection; the institution exposes them through its own load balancer. -
The job owner always sees why a job is waiting
Researchers see the waiting reason and applicable quota while operators see the same queue and accounting state in the console.
Image policy controls what reaches compute nodes
-
Apptainer is the runtime on GPU nodes
Jobs run through Apptainer without Docker on GPU nodes; the full GPU is assigned directly to the job, without MIG or vGPU.
-
Image approval
The packaged OCI registry imports and scans an image. Two distinct people approve promotion, then the cluster key signs the approved artifact. GPU nodes reject images that lack an accepted signature.
-
Your registry can remain in the trust chain
BareStax supports an institution's own registry and counter-signature, while policy still decides which signed images can start on compute nodes.
-
Caches outlive the read-only image
The approved image remains read-only while package and model caches persist in the researcher's workspace across sessions.
Health policy protects work before it attempts recovery
-
Health signals
Signals include GPU XID and ECC events, row remapping, temperature, power throttling, NVLink, NVSwitch, InfiniBand, RoCE, PCIe AER, EDAC, clock quality, critical services, and shared storage access.
-
Stale data becomes an unknown state
When telemetry becomes stale, BareStax reports the state as unknown. It does not carry a former healthy result forward as current evidence.
-
Fault isolation and manual cordon
A faulty node stops accepting new jobs while its current job finishes. An operator can cordon a node for maintenance or deep diagnosis and return it to work afterward. Flap protection prevents unstable signals from repeatedly moving a node in and out of service.
-
Recovery escalates under policy
Policy advances from service restart to operating system restart and then power cycle. Each stage can run automatically or wait for approval. A storm brake halts recovery when many nodes show a systemic fault.
-
barestaxctl node diagnose <name> --deepOperators can run deep GPU diagnosis from one command. The impact view links node or storage faults to affected jobs, tenants, and services, and distinguishes verified impact from possible impact.
barestaxctl impact node <name>
barestaxctl node diagnose <name> --deep
barestaxctl doctorRelease bundles make isolated operation repeatable
-
Offline runtime
Cluster services, repositories, licensing, and runbooks stay inside the institution's network. Security covers the external-connection boundary.
-
The release begins on the connected side
barestax-bundlercollects required packages and metadata into a signed, content-addressed release bundle. It can export the bundle and verify it before transfer. -
Import and promotion
Transfer uses the institution's approved media or data diode. Only a verified bundle reaches promotion; Deployment lists the checks and rollout order.
-
Repository snapshots preserve the release contents
Upstream repositories are pinned to timestamped snapshots. Their key fingerprints appear in the bundle's security section, so the same release produces the same bytes at each installation.
barestax-bundler export
barestax-bundler verify
barestaxctl release import <bundle>
barestax-server upgrade --offlineSecurity controls are part of every operation
-
Certificates and second factors establish identity
mTLS and the cluster PKI establish node identity. LDAP, Active Directory, local accounts, OIDC, SAML, and built-in TOTP establish user identity. Security carries the certificate lifecycle details.
-
Six roles separate tenants and approvals
Viewer, operator, approver, administrator, auditor, and tenant administrator duties remain distinct. Approval and tenant boundaries are detailed on the Security page.
-
Audit, secrets, and packages stay reviewable
Audit stays hash-linked; Security covers export and SIEM. Envelope encryption, package locks, SBOM output, a license allowlist, and reproducible builds expose secrets and software inputs for review.
Monitoring presents current evidence and data age
-
Exporters use standard Prometheus data
BareStax includes exporters for the head node and compute nodes, along with alert rules and prepared dashboards. It integrates with Prometheus and Alertmanager. An institution's existing Prometheus and Grafana installation can read the same exporter data.
-
BareStax dashboards
Operators see a node-by-GPU heat map plus utilization, memory, power, temperature, and network charts. The deeper dashboards are BareStax panels that open inside the console under the same authenticated session. They are not Grafana panels.
-
Alert-chain test
A test alert travels to the on-call recipient, who confirms its code. This verifies delivery rather than assuming that an accepted configuration reaches a person.
-
Idle GPUs are reported beside waiting work
Reports place idle GPUs beside waiting work, with queue state, quota, and health as context.
Storage and BMC state enter the same operational model
-
NFS and S3 clients
BareStax manages the client side of NFS and S3 targets while the institution retains the storage server. Mount protection prevents a missing shared target from causing silent writes to local disk.
-
Storage faults carry job and tenant impact
The control plane checks storage access and shows which jobs, tenants, and services depend on a failing target. The impact view includes the age and certainty of that information.
-
Redfish-compatible BMCs
BareStax reads inventory, sensors, event logs, and power state. Power control follows the Security approval policy.
The CLI uses the same API as the console
-
Operator CLI
barestaxctlsupports plan and dry-run, cluster health reports, support bundles, alert testing, node lifecycle actions, operations, and approvals. -
Researcher CLI
barestaxsubmits jobs, reports status and quota, reads logs, and opens runbooks against the state shown in the researcher portal. -
Table and JSON output
CLI output is a readable table for people or JSON for automation.
barestax run -f job.yaml
barestax status
barestax quota
barestax logs <job>
barestax runbook <id>