NVIDIA Publishes a Management Plane for GPU Clusters and Documents Its Limits
"NVIDIA's NVKE preview lets a GPU operator run Kubernetes clusters for its own customers on bare metal, and the project's limitation page lists the backup, upgrade and access work it leaves
TL;DR
- NVKE is an open-source management plane from NVIDIA that lets a GPU infrastructure operator create and run Kubernetes clusters for its own customers on bare metal.
- Each workload cluster receives its own Kubernetes control plane, its own etcd and its own node pools, and the operator runs the management plane that provisions them.
- A throwaway seed cluster stands up the management cluster, and the management cluster then provisions workload clusters on request from the NVKE API.
- One controller process serves a REST API backed by PostgreSQL, a Temporal worker and a set of Kubernetes reconcilers, and it writes the cluster record and a work-queue row in a single transaction.
- Nodes attest their identity through SPIFFE/SPIRE with their TPM, NVIDIA Infra Controller provisions the machines, and kubeadm bootstraps Kubernetes on them.
- The project's own limitation page lists what the preview leaves out: the management cluster does not manage itself, there is no management-plane upgrade procedure and no backup or restore procedure.
- The API listens on port 30080 without TLS, authentication is off after installation, and NVKE performs no authorisation at all, so every admitted caller can create and delete workload clusters.
- A handed-over admin kubeconfig cannot be revoked, node-pool changes leave running nodes untouched, and the full-disk-encryption image gives every node built from it one shared volume key.
- An operator adds TLS and authorisation in front of the API, a backup and upgrade routine for the management plane, per-customer credentials, a break-fix path for node replacement and a telemetry backend that keeps data.
A throwaway seed cluster builds the management cluster
NVIDIA's documentation frames the project as a self-hosting factory for Kubernetes clusters. An operator installs nvkectl, which uses a temporary seed cluster to form the management cluster and installs the management plane onto it. From then on the operator requests clusters through the NVKE API or nvkectl, and NVKE provisions the bare-metal nodes, forms the cluster and drives node pool scaling, Kubernetes upgrades, node replacement and certificate renewal.
The provisioning path runs through NVIDIA Infra Controller, or NICo, the only compatible infrastructure provider at preview, with LaunchLayer supported once a deployment enables it. NICo handles machine discovery, firmware, network configuration and host provisioning, and the management plane reaches it through CAPNICo, the Cluster API provider for NICo. NVKE provisions each cluster with OVN-Kubernetes and then hands off: the GPU software stack arrives through an AICR recipe bundle, a version-locked combination of the GPU Operator, drivers, kernel configuration and Kubernetes that is rendered into GitOps bundles.
That division carries a practical consequence for anyone planning a site. A working NICo site is a prerequisite for day-2 operations, because tenant cluster creation, node add and remove, and upgrades all go through NICo. The NVKE software bill of materials deploys the controller, NICo, Temporal, Postgres, SPIRE and CAPNICo onto the management cluster in one GitOps pass, so the machine provider shares a cluster with the control plane that drives it, and a failure there stops provisioning.
One process runs the API, the workflows and the reconcilers
The architecture document is unusually specific, and the detail matters because it tells an operator where to look when a cluster stalls. NVKE is a Go workspace of thirteen modules, and three of them run in production: the controller deployed as a Helm chart, the node agent installed on each node as a Debian package and run by systemd, and the Cluster API infrastructure provider deployed beside the controller. The rest are a CLI, the generated API module, an authentication library, the management engine's custom resource types and test tooling.
The request path avoids a common failure mode. The API queues Temporal work through the database outbox. When a client posts to /api/clusters, the handler opens one database transaction and writes both the cluster record and a row in the workflow_outbox table, so a committed cluster always has its work queued. A processor polls that table, starts the matching Temporal workflow and marks the row processed. The workflow creates Cluster API resources, the infrastructure provider provisions or adopts the nodes, the controller dials nvke-agent on each node over gRPC, and the cluster reaches Ready.
The node agent only listens: it is a gRPC server, and the controller dials it. It runs kubeadm init and kubeadm join, installs a CNI, bootstraps the kubelet, handles drain, reboot and teardown, and attests its identity through SPIRE by TPM or join token. Clusters use OVN-Kubernetes for networking, and etcd encryption at rest is available per cluster.
The limitation page carries the operational weight
Most preview releases bury their gaps in release notes. NVKE publishes a page called Preview status and known limitations and lists them by area, which makes the document the most useful thing in the repository for anyone evaluating a managed Kubernetes service built on it.
The management cluster comes first. It does not upgrade, scale or replace its own nodes, the preview provides no procedure to move the management-plane services to a later release, and it provides no procedure to back up or restore the management cluster or the plane's state. NVIDIA tells operators to keep the kubeconfig and the etcd encryption key file from the bootstrap in secure storage, because that key file is the only copy not sitting on a node. Management cluster nodes built from the node image also need the operator's own SPIRE certificate authority at bootstrap: without those files SPIRE falls back to a self-signed CA, no node can attest, and nothing joins.
The API section reads like a checklist for an admission review. The NVKE API listens on port 30080 of every management node without TLS, authentication is off after installation, and any caller in an admitted network can call every endpoint. NVKE performs no authorisation, so a token holder can create and delete workload clusters. Customers receive only an admin kubeconfig, enableOidc does not connect a customer's identity provider to a workload cluster, and a handed-over admin kubeconfig cannot be revoked because NVKE does not rotate the cluster certificate authority.
The workload-cluster section holds three more. Changing a node pool's configuration does not replace its running nodes, and NVKE marks them with a condition without draining or deleting them, so an operator needs a break-fix system that acts on the condition. Node labels and taints are fixed at creation and applied only when a node joins. nvkectl upgrades only the control plane, and a node pool's Kubernetes version changes through a raw API call.
Encryption and telemetry defaults need a decision
Two further items deserve attention before anyone treats the encryption story as settled. NVIDIA's hosted pipeline builds a full-disk-encryption variant of the node image with the root volume on LUKS2, each image build generates a new volume key, and every node built from that image inherits it, so anyone holding the original image and the documented bootstrap passphrase recovers the key that decrypts every node built from it. The documentation says plainly to use those images for evaluation only. The default telemetry pipeline, NKTP, keeps data in memory and gives anonymous users admin access to Grafana.
The audit consequence is concrete. A tenant that needs encryption at rest has to replace or rebuild that image path before the cluster holds anything sensitive, and a tenant that keeps the default telemetry backend has an anonymous admin login on a monitoring stack that sits inside the cluster boundary.
What an operator adds before selling clusters
Work through the limitation page as a build plan and the list is finite. Put a TLS endpoint and an authorisation layer in front of the API, because the project's own authentication stops at admitting a caller. Keep the bootstrap kubeconfig and the etcd key file somewhere with a restore test attached, then write the restore procedure NVIDIA does not ship. Decide how the management plane moves to the next release, since no procedure exists.
Then handle the tenant boundary. Issue per-customer credentials through an OIDC kubeconfig, and treat the admin kubeconfig as a break-glass artifact that expires, because it cannot be revoked. Where a customer needs narrower rights, that work happens inside the cluster they receive.
Finally, close the loop on node health. NVSentinel watches nodes for hardware faults and, when a node needs repair, applies workload-aware cordon and drain before triggering NVKE's node replacement lifecycle through the lifecycle APIs NVKE provides. Node pool changes need something that deletes outdated nodes. A site that runs this on real hardware also needs its own imager and script for QCOW2 provisioning, and its own telemetry backend. Teams that operate Kubernetes on their own hardware already own that operational layer, the same layer this preview assigns to the operator.
Sources
- NVIDIA Kubernetes Engine (NVKE) repository, README and architecture documents, github.com, retrieved 2026-11-09
- Preview status and known limitations, NVKE documentation, github.com/dsx-ai-factory/nvke
- Architecture overview, NVKE documentation, github.com/dsx-ai-factory/nvke
- What is NVKE, NVIDIA documentation, docs.nvidia.com
Source: the NVKE repository read in full on 9 November 2026, in particular README.md, docs/overview/what-is-nvke.md, docs/overview/preview-status.md, docs/architecture.md, docs/dsx-stack.md and docs/lifecycle.md, fetched as raw Markdown from raw.githubusercontent.com, plus captures of the published documentation at docs.nvidia.com/nvke. Every mechanism, default and limitation above comes from those files.