Deploying OpenEBS Mayastor on Talos Linux: A Production Guide
Talos Linux and OpenEBS Mayastor are a potent combination for Kubernetes storage, if you know how to configure them. Here is the no-nonsense production journey to getting NVMe-oF running flawlessly on an immutable OS.

Deploying high-performance stateful workloads in Kubernetes requires a storage layer that doesn't bottleneck on the kernel. OpenEBS Mayastor, leveraging NVMe over Fabrics (NVMe-oF), bypasses traditional storage overhead to deliver near bare-metal IOPS directly to pods.
However, pairing Mayastor with Talos Linux introduces specific architectural friction. Talos is built on a secure, immutable, API-driven foundation that restricts hardware and kernel access. Mayastor requires low-level privileges to bypass the kernel and manage NVMe devices directly.
This guide documents the exact configurations, deployment manifests, and production realities required to bridge that gap and build a resilient, high-performance storage backend.
Cluster Architecture & Prerequisites
- Distribution: Talos Linux
- Storage Nodes: 6 nodes (talos86-92, excluding talos91)
- Backend: NVMe SSDs (Dedicated per node)
- Replication Factor: 3
- Protocol: NVMe-oF (TCP)
Hardware Requirements
- Minimum 3 nodes with physical NVMe drives.
- 2+ CPU cores per storage node.
- 2GiB hugepages (1024 x 2MiB pages) per storage node.
- NVMe-TCP kernel modules (Built-in for Talos, requiring specific Helm overrides).
Core Talos Configuration
Talos requires specific machine configuration overrides to support Mayastor's polling architecture and memory requirements.
Control Plane: Pod Security Exemptions
Talos enforces baseline Pod Security Standards (PSS) by default. Mayastor requires privileged access to manage NVMe devices and manipulate network namespaces. Create cp.yaml to exempt the OpenEBS namespace:
1cluster:2 apiServer:3 admissionControl:4 - name: PodSecurity5 configuration:6 apiVersion: pod-security.admission.config.k8s.io/v1beta17 kind: PodSecurityConfiguration8 exemptions:9 namespaces:10 - openebs
(Note: The OpenEBS namespace itself must also be labeled with pod-security.kubernetes.io/enforce: privileged. Our ArgoCD manifest below handles this automatically via managedNamespaceMetadata.)
Worker Nodes: Memory and Mount Propagation
Create wp.yaml for your designated storage nodes to allocate hugepages, set scheduling labels, and establish a shared mount for volume attachment.
1machine:2 sysctls:3 vm.nr_hugepages: "1024"4 nodeLabels:5 openebs.io/engine: "mayastor"6 kubelet:7 extraMounts:8 - destination: /var/local9 type: bind10 source: /var/local11 options:12 - bind13 - rshared14 - rw
Architectural Note: The rshared option on the /var/local mount is strictly required. Without mount propagation, the kubelet cannot access the host paths for volume mounting, and containers will not see attached volumes.
Apply these configurations:
1talosctl patch --mode=no-reboot machineconfig -n <control-plane-ip> --patch @cp.yaml2talosctl patch --mode=no-reboot machineconfig -n <worker-node-ip> --patch @wp.yaml34# Kubelet must be restarted to register hugepages5talosctl -n <node-ip> service kubelet restart
Infrastructure as Code: ArgoCD Deployment
Below is the ArgoCD Application manifest used to deploy OpenEBS. Note that the Helm repository changed in October 2024 to https://openebs.github.io/openebs.
1apiVersion: argoproj.io/v1alpha12kind: Application3metadata:4 name: openebs5 namespace: argocd6spec:7 destination:8 namespace: openebs9 server: https://kubernetes.default.svc10 project: default11 source:12 chart: openebs13 helm:14 values: |-15 engines:16 local:17 lvm:18 enabled: false19 zfs:20 enabled: false21 replicated:22 mayastor:23 enabled: true2425 localprovisioner:26 enableHostpathClass: false2728 mayastor:29 localpv-provisioner:30 enabled: false31 hostpathClass:32 enabled: false3334 io_engine:35 envcontext: "iova-mode=pa"36 logLevel: debug37 resources:38 limits:39 cpu: "10"40 hugepages2Mi: "2Gi"41 hugepages1Gi: '0'42 requests:43 hugepages2Mi: "2Gi"44 hugepages1Gi: '0'45 nodeSelector:46 openebs.io/engine: mayastor47 kubernetes.io/arch: amd644849 csi:50 node:51 initContainers:52 enabled: false5354 crds:55 csi:56 volumeSnapshots:57 enabled: true5859 etcd:60 replicaCount: 361 auth:62 token:63 enabled: false64 rbac:65 create: false66 allowNoneAuthentication: true67 loki:68 enabled: false69 alloy:70 enabled: false7172 repoURL: https://openebs.github.io/openebs73 targetRevision: '*'74 syncPolicy:75 managedNamespaceMetadata:76 labels:77 pod-security.kubernetes.io/audit: privileged78 pod-security.kubernetes.io/enforce: privileged79 pod-security.kubernetes.io/warn: privileged80 automated:81 prune: true82 selfHeal: true83 syncOptions:84 - CreateNamespace=true
Critical Overrides:
initContainers.enabled: false: Standard Mayastor init containers check for loadable kernel modules and will fail on Talos, which compiles NVMe modules directly into the kernel.envcontext: "iova-mode=pa": Sets the IOVA mode for optimal physical address translation.
Storage Configuration
DiskPools
DiskPools define which physical disks Mayastor can use. They operate on a strict one-pool-per-node architecture.
Identify available NVMe disks and their persistent paths:
1talosctl -n <node-ip> get disks2talosctl -n <node-ip> ls /dev/disk/by-id/
⚠️ CRITICAL WARNING: Never use dynamic kernel device names (
/dev/nvme0n1or/dev/sda) in DiskPool configurations. These names are dynamically assigned at boot. If a node reboots and the kernel enumerates the device differently, the DiskPool will fail and data integrity cannot be guaranteed. Always map the WWID shown in Talos to the/dev/disk/by-id/persistent path.
1apiVersion: openebs.io/v1beta32kind: DiskPool3metadata:4 name: talos86-nvme-pool5 namespace: openebs6spec:7 node: talos86.example.com8 disks: ["/dev/disk/by-id/nvme-eui.002538b221b7373f"]9---10apiVersion: openebs.io/v1beta311kind: DiskPool12metadata:13 name: talos87-nvme-pool14 namespace: openebs15spec:16 node: talos87.example.com17 disks: ["/dev/disk/by-id/nvme-eui.002538b911b54815"]18# (Repeat for nodes 88, 89, 90, 92)
StorageClass
1apiVersion: storage.k8s.io/v12kind: StorageClass3metadata:4 name: mayastor-nvmf5provisioner: io.openebs.csi-mayastor6parameters:7 protocol: 'nvmf'8 repl: '3'9 ioTimeout: '60'10 local: 'true'11reclaimPolicy: Delete12allowVolumeExpansion: true13volumeBindingMode: Immediate
Key Parameters Explained:
protocol: 'nvmf': Ensures traffic routes over NVMe-over-TCP rather than legacy iSCSI.repl: '3': Enforces 3-way synchronous replication across different node pools.local: 'true': Instructs Mayastor to prioritize reading from the local node's replica if one exists, drastically reducing read latency.
Application Example: Loki StatefulSet
When deploying StatefulSets via ArgoCD, Kubernetes mutates .spec.volumeClaimTemplates with default values, causing ArgoCD to report OutOfSync drift. We handle this via ignoreDifferences (using jqPathExpressions or alternatively jsonPointers: - /spec/volumeClaimTemplates/0) and ServerSideApply.
1apiVersion: argoproj.io/v1alpha12kind: Application3metadata:4 name: loki5 namespace: argocd6spec:7 destination:8 namespace: loki9 server: https://kubernetes.default.svc10 project: default11 ignoreDifferences:12 - group: "apps"13 kind: StatefulSet14 name: loki15 namespace: loki16 jqPathExpressions:17 - ".spec.volumeClaimTemplates"18 source:19 chart: loki20 helm:21 values: |-22 deploymentMode: SingleBinary23 singleBinary:24 replicaCount: 125 persistence:26 storageClass: "mayastor-nvmf"27 # ... (remaining Loki config)28 repoURL: https://grafana.github.io/helm-charts29 targetRevision: '*'30 syncPolicy:31 syncOptions:32 - CreateNamespace=true33 - ServerSideApply=true
Production Realities: Resolving Architectural Friction
In deployment, theory rarely survives contact with the cluster. Below are the precise technical resolutions to the edge cases encountered during this integration.
1. ETCD Pod Mounting Failures
If openebs-etcd-* pods fail with path /var/local/... doesn't exist, it means Talos kubelet lacks access to host paths. The solution is the /var/local extra mount with rshared configured in wp.yaml (detailed above).
2. Hugepages Allocation Failures
If io-engine pods hang with insufficient hugepages-2Mi:
- Verify Talos OS allocation:
talosctl -n <node-ip> get /proc/meminfo | grep Huge - Restart kubelet:
talosctl -n <node-ip> service kubelet restart - Verify Kubernetes visibility:
kubectl get node <node-name> -o json | jq '.status.allocatable'
3. DiskPool Creation Failures
If a DiskPool shows a "Failed" status, the disk likely contains an existing partition table or filesystem. On Talos 1.9.0+, wipe the disk directly:
1talosctl wipe disk nvme0n1 -n <node-ip>
4. Upgrade Networking Timeouts
Upgrading Talos from 1.6.x to 1.7.x can cause the OS to drop the metrics exporter's gRPC connection to the io-engine, causing a CrashLoopBackoff. Either upgrade with the preserve flag: talosctl -n <node-ip> upgrade --preserve (default behavior in 1.8+), or temporarily disable metrics in Helm prior to the upgrade.
5. CPU Allocation vs. SPDK Polling & CFS Throttling
If io-engine is assigned a limit of 10 CPU cores but only utilizes ~200% (2 cores), or if you are experiencing severe I/O latency spikes under load, the issue lies in Mayastor's SPDK architecture and Kubernetes CFS quotas.
SPDK uses a userspace polling architecture. It ignores Kubernetes CPU limits entirely for thread count, relying on the MAYASTOR_CPUS environment variable. However, because SPDK continuously polls for I/O, it rapidly exhausts its Kubernetes CFS quota and gets throttled by the kernel, injecting latency. Verify this by running:
1kubectl exec -n openebs openebs-io-engine-xxxxx -- cat /sys/fs/cgroup/cpu/cpu.stat2# High 'nr_throttled' values confirm the issue.
The optimal engineering solution: On dedicated storage nodes, remove CPU limits entirely. Allow MAYASTOR_CPUS to bound SPDK usage natively while leaving requests for the Kubernetes scheduler:
1mayastor:2 io_engine:3 resources:4 limits:5 # Omit CPU limit entirely to bypass CFS throttling6 hugepages2Mi: "2Gi"7 requests:8 cpu: "2" # Used only for pod scheduling9 hugepages2Mi: "2Gi"10 env:11 MAYASTOR_CPUS: "0-5" # SPDK natively bounds to 6 cores. Formats: "0-5" or "0,2,4"
Verification: You can ensure the SPDK reactor threads have successfully bound to your assigned cores by checking process affinity:
1kubectl exec -n openebs openebs-io-engine-xxxxx -- ps -eLo psr,comm | grep reactor
6. The Mental Model: Mayastor vs. Ceph
Engineers accustomed to Ceph often attempt to create a single, cluster-wide DiskPool. Mayastor explicitly rejects this. Mayastor utilizes a One Pool Per Node architecture. Replication is managed exclusively by the Volume layer, not the underlying pool layer.
Capacity Planning Reality Check: Because Mayastor replicates at the volume level, a 100GB PVC with repl: 3 will pull 100GB from three separate node pools. If you have 3 nodes with 500GB drives each (1500GB total raw), your maximum usable capacity across the cluster is ~500GB. Think of DiskPools as "storage donations" from each node; the volume layer handles the replication tax.
Safe Migration and Maintenance Strategy
Migrating production stateful workloads is a high-risk operation. We execute this in strict phases.
Pre-Flight Validation (fio testing)
Before touching production data, validate the raw IOPS and stability:
1apiVersion: v12kind: PersistentVolumeClaim3metadata:4 name: mayastor-test5spec:6 storageClassName: mayastor-nvmf7 accessModes: [ReadWriteOnce]8 resources: { requests: { storage: 50Gi } }9---10apiVersion: v111kind: Pod12metadata:13 name: fio-test14spec:15 containers:16 - name: fio17 image: ljishen/fio18 command:19 - fio20 - --name=randrw-test21 - --filename=/data/testfile22 - --size=40G23 - --direct=124 - --rw=randrw25 - --bs=4k26 - --ioengine=libaio27 - --iodepth=1628 - --runtime=360029 - --numjobs=430 - --time_based31 - --group_reporting32 volumeMounts:33 - name: data34 mountPath: /data35 volumes:36 - name: data37 persistentVolumeClaim:38 claimName: mayastor-test
Phased Migration
- Low Risk (Weeks 1-2): Migrate stateless caches or easily backed-up UI configurations (Grafana, Prometheus, Alertmanager).
- Medium Risk (Month 2): Migrate internal tooling (Jenkins, Harbor).
- High Risk (Month 3+): Migrate transactional databases (PostgreSQL, MySQL, Kafka) only after proving absolute cluster stability. Require full offline backups (
pg_dump) prior to scaling down the original workloads.
Executing the Copy: To safely move data between storage classes, scale your application to 0, then use an Alpine helper pod to synchronize the data from the old PVC to the newly provisioned Mayastor PVC:
1kubectl run -n default data-copy --rm -i --image=alpine --overrides='2{3 "spec": {4 "containers": [{5 "name": "copy",6 "image": "alpine",7 "command": ["sh", "-c", "cp -av /old/* /new/ && sync"],8 "volumeMounts": [9 {"name": "old", "mountPath": "/old"},10 {"name": "new", "mountPath": "/new"}11 ]12 }],13 "volumes": [14 {"name": "old", "persistentVolumeClaim": {"claimName": "old-storage-pvc"}},15 {"name": "new", "persistentVolumeClaim": {"claimName": "mayastor-new-pvc"}}16 ],17 "restartPolicy": "Never"18 }19}'
Quick Rollback: Always retain the old PVC until the application has run stably on Mayastor for at least 48 hours. If the application fails to start or exhibits high latency, immediately revert to the old storage by patching the deployment:
1kubectl patch deployment <app-name> --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/volumes/0/persistentVolumeClaim/claimName", "value": "old-storage-pvc"}]'
Node Maintenance and io-engine Rebuilds
Never arbitrarily kill an io-engine pod without checking cluster health. Killing a pod holding the only healthy replica of a volume will result in immediate data unavailability. (Note: If you have provisioned a volume with repl: 1, killing the io-engine pod on that node results in immediate, guaranteed downtime).
- Verify Health: Ensure no degraded volumes exist.
@@PRE15@@ - Evict Safely: Use the Mayastor plugin to cordon and drain. This instructs the control plane to synchronize replicas before allowing the node to shut down.
@@PRE16@@
Warning: Never use standardkubectl drainon a Mayastor storage node. The standard Kubernetes drain is completely unaware of Mayastor's replication geometry and will blindly evict pods, potentially causing immediate data loss if multiple replicas are taken offline. - Monitor Rebuilds: When the node returns, SPDK will automatically sync stale blocks. Wait for the volume status to shift from
REBUILDINGback toOnlineand3/3before proceeding to the next node.
To view the actual synchronization percentage during a rebuild, inspect the volume's JSON output:
@@PRE17@@ - Return to Service: Once the volume status returns to
3/3andOnline, allow Mayastor to place new volumes on the node again:
@@PRE18@@
Verification Commands
Validate the deployment status using the following operational commands:
1# Verify component health2kubectl get pods -n openebs34# Verify DiskPool capacity and state5kubectl get diskpools -n openebs67# Validate node topology and availability8kubectl mayastor get nodes910# Check live replica health mapping11kubectl mayastor get volume-replica-topologies
Conclusion
Deploying OpenEBS Mayastor on Talos Linux requires precise configuration, but the result is an exceptionally fast, highly resilient stateful backend that respects the security posture of the cluster. By implementing the node-level sysctls, managing SPDK CPU affinity natively, and adhering to strict maintenance protocols, this architecture handles enterprise workloads with predictability and speed.
At Deviqon Labs, we rely on deep technical precision to solve complex infrastructure challenges. Sharp Minds, Solid Code.
Contact our engineering team to discuss your Kubernetes architecture.


