Deviqon Labs
News19 Apr 2026

Deploying OpenEBS Mayastor on Talos Linux: A Production Guide

Talos Linux and OpenEBS Mayastor are a potent combination for Kubernetes storage, if you know how to configure them. Here is the no-nonsense production journey to getting NVMe-oF running flawlessly on an immutable OS.

RR
Robert RotterDeviqon Labs
Share
Deploying OpenEBS Mayastor on Talos Linux: A Production Guide

Deploying high-performance stateful workloads in Kubernetes requires a storage layer that doesn't bottleneck on the kernel. OpenEBS Mayastor, leveraging NVMe over Fabrics (NVMe-oF), bypasses traditional storage overhead to deliver near bare-metal IOPS directly to pods.

However, pairing Mayastor with Talos Linux introduces specific architectural friction. Talos is built on a secure, immutable, API-driven foundation that restricts hardware and kernel access. Mayastor requires low-level privileges to bypass the kernel and manage NVMe devices directly.

This guide documents the exact configurations, deployment manifests, and production realities required to bridge that gap and build a resilient, high-performance storage backend.

Cluster Architecture & Prerequisites

  • Distribution: Talos Linux
  • Storage Nodes: 6 nodes (talos86-92, excluding talos91)
  • Backend: NVMe SSDs (Dedicated per node)
  • Replication Factor: 3
  • Protocol: NVMe-oF (TCP)

Hardware Requirements

  • Minimum 3 nodes with physical NVMe drives.
  • 2+ CPU cores per storage node.
  • 2GiB hugepages (1024 x 2MiB pages) per storage node.
  • NVMe-TCP kernel modules (Built-in for Talos, requiring specific Helm overrides).

Core Talos Configuration

Talos requires specific machine configuration overrides to support Mayastor's polling architecture and memory requirements.

Control Plane: Pod Security Exemptions

Talos enforces baseline Pod Security Standards (PSS) by default. Mayastor requires privileged access to manage NVMe devices and manipulate network namespaces. Create cp.yaml to exempt the OpenEBS namespace:

1cluster:
2 apiServer:
3 admissionControl:
4 - name: PodSecurity
5 configuration:
6 apiVersion: pod-security.admission.config.k8s.io/v1beta1
7 kind: PodSecurityConfiguration
8 exemptions:
9 namespaces:
10 - openebs

(Note: The OpenEBS namespace itself must also be labeled with pod-security.kubernetes.io/enforce: privileged. Our ArgoCD manifest below handles this automatically via managedNamespaceMetadata.)

Worker Nodes: Memory and Mount Propagation

Create wp.yaml for your designated storage nodes to allocate hugepages, set scheduling labels, and establish a shared mount for volume attachment.

1machine:
2 sysctls:
3 vm.nr_hugepages: "1024"
4 nodeLabels:
5 openebs.io/engine: "mayastor"
6 kubelet:
7 extraMounts:
8 - destination: /var/local
9 type: bind
10 source: /var/local
11 options:
12 - bind
13 - rshared
14 - rw

Architectural Note: The rshared option on the /var/local mount is strictly required. Without mount propagation, the kubelet cannot access the host paths for volume mounting, and containers will not see attached volumes.

Apply these configurations:

1talosctl patch --mode=no-reboot machineconfig -n <control-plane-ip> --patch @cp.yaml
2talosctl patch --mode=no-reboot machineconfig -n <worker-node-ip> --patch @wp.yaml
3
4# Kubelet must be restarted to register hugepages
5talosctl -n <node-ip> service kubelet restart

Infrastructure as Code: ArgoCD Deployment

Below is the ArgoCD Application manifest used to deploy OpenEBS. Note that the Helm repository changed in October 2024 to https://openebs.github.io/openebs.

1apiVersion: argoproj.io/v1alpha1
2kind: Application
3metadata:
4 name: openebs
5 namespace: argocd
6spec:
7 destination:
8 namespace: openebs
9 server: https://kubernetes.default.svc
10 project: default
11 source:
12 chart: openebs
13 helm:
14 values: |-
15 engines:
16 local:
17 lvm:
18 enabled: false
19 zfs:
20 enabled: false
21 replicated:
22 mayastor:
23 enabled: true
24
25 localprovisioner:
26 enableHostpathClass: false
27
28 mayastor:
29 localpv-provisioner:
30 enabled: false
31 hostpathClass:
32 enabled: false
33
34 io_engine:
35 envcontext: "iova-mode=pa"
36 logLevel: debug
37 resources:
38 limits:
39 cpu: "10"
40 hugepages2Mi: "2Gi"
41 hugepages1Gi: '0'
42 requests:
43 hugepages2Mi: "2Gi"
44 hugepages1Gi: '0'
45 nodeSelector:
46 openebs.io/engine: mayastor
47 kubernetes.io/arch: amd64
48
49 csi:
50 node:
51 initContainers:
52 enabled: false
53
54 crds:
55 csi:
56 volumeSnapshots:
57 enabled: true
58
59 etcd:
60 replicaCount: 3
61 auth:
62 token:
63 enabled: false
64 rbac:
65 create: false
66 allowNoneAuthentication: true
67 loki:
68 enabled: false
69 alloy:
70 enabled: false
71
72 repoURL: https://openebs.github.io/openebs
73 targetRevision: '*'
74 syncPolicy:
75 managedNamespaceMetadata:
76 labels:
77 pod-security.kubernetes.io/audit: privileged
78 pod-security.kubernetes.io/enforce: privileged
79 pod-security.kubernetes.io/warn: privileged
80 automated:
81 prune: true
82 selfHeal: true
83 syncOptions:
84 - CreateNamespace=true

Critical Overrides:

  • initContainers.enabled: false: Standard Mayastor init containers check for loadable kernel modules and will fail on Talos, which compiles NVMe modules directly into the kernel.
  • envcontext: "iova-mode=pa": Sets the IOVA mode for optimal physical address translation.

Storage Configuration

DiskPools

DiskPools define which physical disks Mayastor can use. They operate on a strict one-pool-per-node architecture.

Identify available NVMe disks and their persistent paths:

1talosctl -n <node-ip> get disks
2talosctl -n <node-ip> ls /dev/disk/by-id/

⚠️ CRITICAL WARNING: Never use dynamic kernel device names (/dev/nvme0n1 or /dev/sda) in DiskPool configurations. These names are dynamically assigned at boot. If a node reboots and the kernel enumerates the device differently, the DiskPool will fail and data integrity cannot be guaranteed. Always map the WWID shown in Talos to the /dev/disk/by-id/ persistent path.

1apiVersion: openebs.io/v1beta3
2kind: DiskPool
3metadata:
4 name: talos86-nvme-pool
5 namespace: openebs
6spec:
7 node: talos86.example.com
8 disks: ["/dev/disk/by-id/nvme-eui.002538b221b7373f"]
9---
10apiVersion: openebs.io/v1beta3
11kind: DiskPool
12metadata:
13 name: talos87-nvme-pool
14 namespace: openebs
15spec:
16 node: talos87.example.com
17 disks: ["/dev/disk/by-id/nvme-eui.002538b911b54815"]
18# (Repeat for nodes 88, 89, 90, 92)

StorageClass

1apiVersion: storage.k8s.io/v1
2kind: StorageClass
3metadata:
4 name: mayastor-nvmf
5provisioner: io.openebs.csi-mayastor
6parameters:
7 protocol: 'nvmf'
8 repl: '3'
9 ioTimeout: '60'
10 local: 'true'
11reclaimPolicy: Delete
12allowVolumeExpansion: true
13volumeBindingMode: Immediate

Key Parameters Explained:

  • protocol: 'nvmf': Ensures traffic routes over NVMe-over-TCP rather than legacy iSCSI.
  • repl: '3': Enforces 3-way synchronous replication across different node pools.
  • local: 'true': Instructs Mayastor to prioritize reading from the local node's replica if one exists, drastically reducing read latency.

Application Example: Loki StatefulSet

When deploying StatefulSets via ArgoCD, Kubernetes mutates .spec.volumeClaimTemplates with default values, causing ArgoCD to report OutOfSync drift. We handle this via ignoreDifferences (using jqPathExpressions or alternatively jsonPointers: - /spec/volumeClaimTemplates/0) and ServerSideApply.

1apiVersion: argoproj.io/v1alpha1
2kind: Application
3metadata:
4 name: loki
5 namespace: argocd
6spec:
7 destination:
8 namespace: loki
9 server: https://kubernetes.default.svc
10 project: default
11 ignoreDifferences:
12 - group: "apps"
13 kind: StatefulSet
14 name: loki
15 namespace: loki
16 jqPathExpressions:
17 - ".spec.volumeClaimTemplates"
18 source:
19 chart: loki
20 helm:
21 values: |-
22 deploymentMode: SingleBinary
23 singleBinary:
24 replicaCount: 1
25 persistence:
26 storageClass: "mayastor-nvmf"
27 # ... (remaining Loki config)
28 repoURL: https://grafana.github.io/helm-charts
29 targetRevision: '*'
30 syncPolicy:
31 syncOptions:
32 - CreateNamespace=true
33 - ServerSideApply=true

Production Realities: Resolving Architectural Friction

In deployment, theory rarely survives contact with the cluster. Below are the precise technical resolutions to the edge cases encountered during this integration.

1. ETCD Pod Mounting Failures

If openebs-etcd-* pods fail with path /var/local/... doesn't exist, it means Talos kubelet lacks access to host paths. The solution is the /var/local extra mount with rshared configured in wp.yaml (detailed above).

2. Hugepages Allocation Failures

If io-engine pods hang with insufficient hugepages-2Mi:

  1. Verify Talos OS allocation: talosctl -n <node-ip> get /proc/meminfo | grep Huge
  2. Restart kubelet: talosctl -n <node-ip> service kubelet restart
  3. Verify Kubernetes visibility: kubectl get node <node-name> -o json | jq '.status.allocatable'

3. DiskPool Creation Failures

If a DiskPool shows a "Failed" status, the disk likely contains an existing partition table or filesystem. On Talos 1.9.0+, wipe the disk directly:

1talosctl wipe disk nvme0n1 -n <node-ip>

4. Upgrade Networking Timeouts

Upgrading Talos from 1.6.x to 1.7.x can cause the OS to drop the metrics exporter's gRPC connection to the io-engine, causing a CrashLoopBackoff. Either upgrade with the preserve flag: talosctl -n <node-ip> upgrade --preserve (default behavior in 1.8+), or temporarily disable metrics in Helm prior to the upgrade.

5. CPU Allocation vs. SPDK Polling & CFS Throttling

If io-engine is assigned a limit of 10 CPU cores but only utilizes ~200% (2 cores), or if you are experiencing severe I/O latency spikes under load, the issue lies in Mayastor's SPDK architecture and Kubernetes CFS quotas.

SPDK uses a userspace polling architecture. It ignores Kubernetes CPU limits entirely for thread count, relying on the MAYASTOR_CPUS environment variable. However, because SPDK continuously polls for I/O, it rapidly exhausts its Kubernetes CFS quota and gets throttled by the kernel, injecting latency. Verify this by running:

1kubectl exec -n openebs openebs-io-engine-xxxxx -- cat /sys/fs/cgroup/cpu/cpu.stat
2# High 'nr_throttled' values confirm the issue.

The optimal engineering solution: On dedicated storage nodes, remove CPU limits entirely. Allow MAYASTOR_CPUS to bound SPDK usage natively while leaving requests for the Kubernetes scheduler:

1mayastor:
2 io_engine:
3 resources:
4 limits:
5 # Omit CPU limit entirely to bypass CFS throttling
6 hugepages2Mi: "2Gi"
7 requests:
8 cpu: "2" # Used only for pod scheduling
9 hugepages2Mi: "2Gi"
10 env:
11 MAYASTOR_CPUS: "0-5" # SPDK natively bounds to 6 cores. Formats: "0-5" or "0,2,4"

Verification: You can ensure the SPDK reactor threads have successfully bound to your assigned cores by checking process affinity:

1kubectl exec -n openebs openebs-io-engine-xxxxx -- ps -eLo psr,comm | grep reactor

6. The Mental Model: Mayastor vs. Ceph

Engineers accustomed to Ceph often attempt to create a single, cluster-wide DiskPool. Mayastor explicitly rejects this. Mayastor utilizes a One Pool Per Node architecture. Replication is managed exclusively by the Volume layer, not the underlying pool layer.

Capacity Planning Reality Check: Because Mayastor replicates at the volume level, a 100GB PVC with repl: 3 will pull 100GB from three separate node pools. If you have 3 nodes with 500GB drives each (1500GB total raw), your maximum usable capacity across the cluster is ~500GB. Think of DiskPools as "storage donations" from each node; the volume layer handles the replication tax.


Safe Migration and Maintenance Strategy

Migrating production stateful workloads is a high-risk operation. We execute this in strict phases.

Pre-Flight Validation (fio testing)

Before touching production data, validate the raw IOPS and stability:

1apiVersion: v1
2kind: PersistentVolumeClaim
3metadata:
4 name: mayastor-test
5spec:
6 storageClassName: mayastor-nvmf
7 accessModes: [ReadWriteOnce]
8 resources: { requests: { storage: 50Gi } }
9---
10apiVersion: v1
11kind: Pod
12metadata:
13 name: fio-test
14spec:
15 containers:
16 - name: fio
17 image: ljishen/fio
18 command:
19 - fio
20 - --name=randrw-test
21 - --filename=/data/testfile
22 - --size=40G
23 - --direct=1
24 - --rw=randrw
25 - --bs=4k
26 - --ioengine=libaio
27 - --iodepth=16
28 - --runtime=3600
29 - --numjobs=4
30 - --time_based
31 - --group_reporting
32 volumeMounts:
33 - name: data
34 mountPath: /data
35 volumes:
36 - name: data
37 persistentVolumeClaim:
38 claimName: mayastor-test

Phased Migration

  1. Low Risk (Weeks 1-2): Migrate stateless caches or easily backed-up UI configurations (Grafana, Prometheus, Alertmanager).
  2. Medium Risk (Month 2): Migrate internal tooling (Jenkins, Harbor).
  3. High Risk (Month 3+): Migrate transactional databases (PostgreSQL, MySQL, Kafka) only after proving absolute cluster stability. Require full offline backups (pg_dump) prior to scaling down the original workloads.

Executing the Copy: To safely move data between storage classes, scale your application to 0, then use an Alpine helper pod to synchronize the data from the old PVC to the newly provisioned Mayastor PVC:

1kubectl run -n default data-copy --rm -i --image=alpine --overrides='
2{
3 "spec": {
4 "containers": [{
5 "name": "copy",
6 "image": "alpine",
7 "command": ["sh", "-c", "cp -av /old/* /new/ && sync"],
8 "volumeMounts": [
9 {"name": "old", "mountPath": "/old"},
10 {"name": "new", "mountPath": "/new"}
11 ]
12 }],
13 "volumes": [
14 {"name": "old", "persistentVolumeClaim": {"claimName": "old-storage-pvc"}},
15 {"name": "new", "persistentVolumeClaim": {"claimName": "mayastor-new-pvc"}}
16 ],
17 "restartPolicy": "Never"
18 }
19}'

Quick Rollback: Always retain the old PVC until the application has run stably on Mayastor for at least 48 hours. If the application fails to start or exhibits high latency, immediately revert to the old storage by patching the deployment:

1kubectl patch deployment <app-name> --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/volumes/0/persistentVolumeClaim/claimName", "value": "old-storage-pvc"}]'

Node Maintenance and io-engine Rebuilds

Never arbitrarily kill an io-engine pod without checking cluster health. Killing a pod holding the only healthy replica of a volume will result in immediate data unavailability. (Note: If you have provisioned a volume with repl: 1, killing the io-engine pod on that node results in immediate, guaranteed downtime).

  1. Verify Health: Ensure no degraded volumes exist.
    @@PRE15@@
  2. Evict Safely: Use the Mayastor plugin to cordon and drain. This instructs the control plane to synchronize replicas before allowing the node to shut down.
    @@PRE16@@
    Warning: Never use standard kubectl drain on a Mayastor storage node. The standard Kubernetes drain is completely unaware of Mayastor's replication geometry and will blindly evict pods, potentially causing immediate data loss if multiple replicas are taken offline.
  3. Monitor Rebuilds: When the node returns, SPDK will automatically sync stale blocks. Wait for the volume status to shift from REBUILDING back to Online and 3/3 before proceeding to the next node.
    To view the actual synchronization percentage during a rebuild, inspect the volume's JSON output:
    @@PRE17@@
  4. Return to Service: Once the volume status returns to 3/3 and Online, allow Mayastor to place new volumes on the node again:
    @@PRE18@@

Verification Commands

Validate the deployment status using the following operational commands:

1# Verify component health
2kubectl get pods -n openebs
3
4# Verify DiskPool capacity and state
5kubectl get diskpools -n openebs
6
7# Validate node topology and availability
8kubectl mayastor get nodes
9
10# Check live replica health mapping
11kubectl mayastor get volume-replica-topologies

Conclusion

Deploying OpenEBS Mayastor on Talos Linux requires precise configuration, but the result is an exceptionally fast, highly resilient stateful backend that respects the security posture of the cluster. By implementing the node-level sysctls, managing SPDK CPU affinity natively, and adhering to strict maintenance protocols, this architecture handles enterprise workloads with predictability and speed.

At Deviqon Labs, we rely on deep technical precision to solve complex infrastructure challenges. Sharp Minds, Solid Code.

Contact our engineering team to discuss your Kubernetes architecture.