Cluster operations stay consistent from host configuration to job execution

BareStax manages the layers that determine whether a GPU node can run production work: operating system settings, drivers, fabric, containers, job scheduling, health, storage access, and recovery.

The node stack is declared as a tested set

  • Profile contents

    A node profile includes the kernel, DOCA-OFED, NVIDIA driver, CUDA user libraries, Fabric Manager, DCGM, NCCL, Apptainer, sysctl values, hostname, and time synchronization. BareStax distributes components only in combinations tested together.

  • Hardware comes before profile selection

    The node agent inventories GPU, NVLink, InfiniBand, PCIe, and NUMA topology, then BareStax matches that inventory to the appropriate profile.

  • Preflight checks

    Preflight rejects a node before registration if its firmware floor, disk, Secure Boot state, or connectivity fails. GPU, NVLink, and InfiniBand checks plus two-node NCCL validation still have to pass before BareStax Core accepts work.

  • Drift handling

    Manual changes and unexpected package or setting changes appear as drift. Non-disruptive drift is corrected while the node remains in service. A disruptive change, such as a kernel change, is reported without modification and waits for an approved operation in a maintenance window.

BareStax Core configuration follows cluster resources and policy

  • Central configuration arrives with accounting

    BareStax produces and centrally distributes the BareStax Core configuration with job accounting; queues follow GPU classes, with separate CPU queues, tenants, quotas, and fairshare.

  • CPU follows the GPU allocation

    The scheduler assigns the CPU share automatically per GPU. Multi-node training receives the fabric configuration needed to reach the network's actual bandwidth.

  • Submission paths

    Teams can submit directly from the command line, choose a barestax run template for batch work, array jobs, multi-node torchrun training, or interactive sessions, or use a form in the researcher portal.

  • Service and preemptible queues

    Preemptible jobs are available, and jobs submitted through barestax run templates return safely to the queue after a node fault. Requeue behavior for jobs submitted directly from the command line remains the researcher's responsibility. Long-running model services use dedicated queues, service registration, and protection; the institution exposes them through its own load balancer.

  • The job owner always sees why a job is waiting

    Researchers see the waiting reason and applicable quota while operators see the same queue and accounting state in the console.

Image policy controls what reaches compute nodes

  • Apptainer is the runtime on GPU nodes

    Jobs run through Apptainer without Docker on GPU nodes; the full GPU is assigned directly to the job, without MIG or vGPU.

  • Image approval

    The packaged OCI registry imports and scans an image. Two distinct people approve promotion, then the cluster key signs the approved artifact. GPU nodes reject images that lack an accepted signature.

  • Your registry can remain in the trust chain

    BareStax supports an institution's own registry and counter-signature, while policy still decides which signed images can start on compute nodes.

  • Caches outlive the read-only image

    The approved image remains read-only while package and model caches persist in the researcher's workspace across sessions.

Health policy protects work before it attempts recovery

  • Health signals

    Signals include GPU XID and ECC events, row remapping, temperature, power throttling, NVLink, NVSwitch, InfiniBand, RoCE, PCIe AER, EDAC, clock quality, critical services, and shared storage access.

  • Stale data becomes an unknown state

    When telemetry becomes stale, BareStax reports the state as unknown. It does not carry a former healthy result forward as current evidence.

  • Fault isolation and manual cordon

    A faulty node stops accepting new jobs while its current job finishes. An operator can cordon a node for maintenance or deep diagnosis and return it to work afterward. Flap protection prevents unstable signals from repeatedly moving a node in and out of service.

  • Recovery escalates under policy

    Policy advances from service restart to operating system restart and then power cycle. Each stage can run automatically or wait for approval. A storm brake halts recovery when many nodes show a systemic fault.

  • barestaxctl node diagnose <name> --deep

    Operators can run deep GPU diagnosis from one command. The impact view links node or storage faults to affected jobs, tenants, and services, and distinguishes verified impact from possible impact.

barestaxctl impact node <name>
barestaxctl node diagnose <name> --deep
barestaxctl doctor

Release bundles make isolated operation repeatable

  • Offline runtime

    Cluster services, repositories, licensing, and runbooks stay inside the institution's network. Security covers the external-connection boundary.

  • The release begins on the connected side

    barestax-bundler collects required packages and metadata into a signed, content-addressed release bundle. It can export the bundle and verify it before transfer.

  • Import and promotion

    Transfer uses the institution's approved media or data diode. Only a verified bundle reaches promotion; Deployment lists the checks and rollout order.

  • Repository snapshots preserve the release contents

    Upstream repositories are pinned to timestamped snapshots. Their key fingerprints appear in the bundle's security section, so the same release produces the same bytes at each installation.

barestax-bundler export
barestax-bundler verify
barestaxctl release import <bundle>
barestax-server upgrade --offline

Security controls are part of every operation

  • Certificates and second factors establish identity

    mTLS and the cluster PKI establish node identity. LDAP, Active Directory, local accounts, OIDC, SAML, and built-in TOTP establish user identity. Security carries the certificate lifecycle details.

  • Six roles separate tenants and approvals

    Viewer, operator, approver, administrator, auditor, and tenant administrator duties remain distinct. Approval and tenant boundaries are detailed on the Security page.

  • Audit, secrets, and packages stay reviewable

    Audit stays hash-linked; Security covers export and SIEM. Envelope encryption, package locks, SBOM output, a license allowlist, and reproducible builds expose secrets and software inputs for review.

Monitoring presents current evidence and data age

  • Exporters use standard Prometheus data

    BareStax includes exporters for the head node and compute nodes, along with alert rules and prepared dashboards. It integrates with Prometheus and Alertmanager. An institution's existing Prometheus and Grafana installation can read the same exporter data.

  • BareStax dashboards

    Operators see a node-by-GPU heat map plus utilization, memory, power, temperature, and network charts. The deeper dashboards are BareStax panels that open inside the console under the same authenticated session. They are not Grafana panels.

  • Alert-chain test

    A test alert travels to the on-call recipient, who confirms its code. This verifies delivery rather than assuming that an accepted configuration reaches a person.

  • Idle GPUs are reported beside waiting work

    Reports place idle GPUs beside waiting work, with queue state, quota, and health as context.

Storage and BMC state enter the same operational model

  • NFS and S3 clients

    BareStax manages the client side of NFS and S3 targets while the institution retains the storage server. Mount protection prevents a missing shared target from causing silent writes to local disk.

  • Storage faults carry job and tenant impact

    The control plane checks storage access and shows which jobs, tenants, and services depend on a failing target. The impact view includes the age and certainty of that information.

  • Redfish-compatible BMCs

    BareStax reads inventory, sensors, event logs, and power state. Power control follows the Security approval policy.

The CLI uses the same API as the console

  • Operator CLI

    barestaxctl supports plan and dry-run, cluster health reports, support bundles, alert testing, node lifecycle actions, operations, and approvals.

  • Researcher CLI

    barestax submits jobs, reports status and quota, reads logs, and opens runbooks against the state shown in the researcher portal.

  • Table and JSON output

    CLI output is a readable table for people or JSON for automation.

barestax run -f job.yaml
barestax status
barestax quota
barestax logs <job>
barestax runbook <id>

mTLS, approvals, and audit continue on the Security page