Deploying HelixML on Talos: from kernel ABI quirks to a working coding-agent fleet
Deploying sandboxed AI coding agents on Kubernetes sounds simple until you hit nested Docker, GPU passthrough, storage semantics, and kernel-level networking quirks. This field report covers our journey running HelixML on Talos Kubernetes with fully self-hosted multi-agent workflows inside our own i

A field report from the Deviqon Labs homelab: Talos Kubernetes, ArgoCD, Vault, Mayastor, Intel iGPU passthrough, and a 397B self-hosted Qwen.
Why HelixML, and why bother self-hosting
I've been watching the AI-coding-agent landscape mature for the last year. Most of the tooling is built on the assumption that you either:
- ship your source code to a SaaS that wraps Claude / GPT, or
- you run a single CLI on your laptop and call it a day.
Neither fits how a small video-pipeline consultancy actually operates. Our code touches client IP under NDAs that make "paste the repo into someone else's cloud" a non-starter. And a single CLI on a laptop doesn't scale past one engineer, doesn't give the team a shared work board, and doesn't let an agent run for an hour while I do something else.
HelixML sits in a different spot. From the helix.ml product page:
- Kubernetes-native. Helm charts for
helix-controlplaneandhelix-sandbox, deployable on any conformant cluster. No vendor lock-in to a managed runtime. - BYO LLM. Anthropic, OpenAI, and any OpenAI-compatible endpoint. This means our self-hosted Qwen on Scaleway Managed Inference works as a first-class provider.
- Per-task sandboxing. Each agent gets its own Linux container with ephemeral git credentials, isolated filesystem, isolated network. Credentials are minted at task start and revoked at task end.
- Opinionated harness. Projects → Backlog → In Progress → Review → Done. Kanban for the whole team. Same UX whether the agent is Claude Code, Gemini CLI, Codex, or Qwen Code.
- Live observability. Streamed desktop view of what each agent is doing right now. Useful when you don't trust the agent yet and want to watch it work.
There's a marketing chart on the helix.ml site that frames this well, drawing on Steve Yegge's Eight Stages of AI-Assisted Coding:
Stage 1: Zero / Near-Zero AI · basic completions and chat Stage 2: Narrow IDE Agent · permission-based sidebar Stage 3: Narrow IDE Agent, YOLO Mode · permissions removed + trust unlock (← most engineers are here) Stage 4: Wide IDE Agent · agent expands across interface Stage 5: Single CLI Agent · command-line, no IDE Stage 6: Multi-Agent CLI · 3–5 parallel instances (← 10× gains start here) Stage 7: 10+ Hand-Managed · manual orchestration Stage 8: Custom Orchestrator · systematic automation (← Helix lives here)
I won't argue every detail of Yegge's scale, but the punchline is right: the throughput cliff is between stage 5 and stage 6. One CLI gives you linear gains. Five parallel CLIs, each doing real PRs, gives you a different category of output. The hard part is the orchestrator: the spec layer that breaks a project into independently-runnable tasks, the credential plumbing, the sandbox per task, the review queue. That's what Helix is.
I'd rather pay for the orchestrator than build it.
The target architecture
What we're standing up:
1┌─────────────────────────────────────────────────────────────────┐2│ Internal users (browser) │3│ https://helix.deviqon.com → internal-nginx ingress │4└──────────────────────────────┬──────────────────────────────────┘5 │6 ┌────────────────▼──────────────────┐7 │ helix-controlplane │8 │ ─ Helix API + UI │9 │ ─ Haystack (RAG) │10 │ ─ SearXNG + headless Chrome │11 │ ─ Kodit (code indexer) │12 │ └─ External Postgres (CNPG) │13 │ └─ VectorChord (pgvector + BM25) │14 └────────────────┬──────────────────┘15 │ runner token16 ┌────────────────▼──────────────────┐17 │ helix-sandbox (talos94) │18 │ ─ Hydra (desktop orchestrator) │19 │ ─ Nested dockerd │20 │ ─ Per-task Ubuntu desktops │21 │ └─ /dev/dri (Intel HD 630) │22 └────────────────┬──────────────────┘23 │ OpenAI-compatible24 ┌────────────────▼──────────────────┐25 │ Scaleway Managed Inference │26 │ Qwen3.5-397B-A17B (MoE, 17B act) │27 └───────────────────────────────────┘
Cross-cutting infrastructure already in place on the cluster:
- Talos Linux workers (
talos81..95) - ArgoCD for GitOps; every YAML below lives in our infra repo
- External Secrets Operator + HashiCorp Vault (
kv/helix) - CloudNativePG for Postgres
- OpenEBS Mayastor for stateful volumes (3-replica NVMe over NVMe-oF)
- cert-manager + internal nginx ingress
Helix has its own Linux + Kubernetes guide. It's a fine starting point, but it assumes a helm install flow. Below is what an ArgoCD + Vault deployment actually looks like, and the gotchas you hit along the way.
Step 1: Secrets in Vault, surfaced via ESO
Everything Helix needs in one Vault KV path: helix. Keys:
KeyWhat it is
license
Helix license key
api_token
Runner token (controlplane ↔ sandbox auth)
db_host
CNPG cluster service
db_port
5432
db_name
helix
db_user
helix
db_password
URL-safe (the chart doesn't escape it)
vector_user
VectorChord user
vector_password
VectorChord password
vector_name
VectorChord db name
vector_dsn
Full DSN (used by haystack)
Vault view of the bag:
The ExternalSecret is intentionally boring: pull everything from helix, target a Secret called helix-secrets, refresh hourly:
1# helix-secrets.yaml2apiVersion: external-secrets.io/v13kind: ExternalSecret4metadata:5 name: helix-secrets6 namespace: helix7spec:8 refreshInterval: 1h9 secretStoreRef:10 name: vault-infra11 kind: ClusterSecretStore12 target:13 name: helix-secrets14 dataFrom:15 - extract:16 key: helix
Argo sync wave: add argocd.argoproj.io/sync-wave: "-1" on this manifest so the Secret exists before the chart tries to mount it. Otherwise Argo pulls Helm first, the deployment crash-loops looking for an unfound secret, ESO catches up, and it eventually self-heals. Works, but noisy.
Gotcha 1: the Helix docs are inconsistent with values.yaml. The Linux + Kubernetes guide uses license-key as the secret key and externalDatabase.* as the Helm path. The chart's actual values-example.yaml uses license and postgresql.external.*. Trust the chart, not the docs. We hit this on the very first sync.
Step 2: Intel GPU plumbing
Helix-sandbox uses a GPU for hardware-accelerated desktop encoding (Moonlight protocol streams the live agent desktop to the browser). Helix's runners are NVIDIA-only, but the sandbox will happily take any /dev/dri-capable GPU and use it for QSV/VAAPI encode. On a budget homelab, that means Intel iGPUs.
2a: Talos system extensions
Bake siderolabs/i915 (and microcode) into your worker schematic via the Talos Image Factory:
1# schematic.yaml2customization:3 systemExtensions:4 officialExtensions:5 - siderolabs/i9156 - siderolabs/intel-ucode7 extraKernelArgs:8 - net.ifnames=09 - intel_iommu=on
1SCHEMATIC_ID=$(curl -sX POST --data-binary @schematic.yaml \2 https://factory.talos.dev/schematics \3 -H "Content-Type: application/yaml" | jq -r '.id')45talosctl -n <node-ip> upgrade \6 --image factory.talos.dev/installer/${SCHEMATIC_ID}:v1.11.6 \7 --preserve
Verify /dev/dri shows up:
1talosctl -n <node-ip> ls /dev/dri2# card0, renderD128
Note for Talos 1.10+:
machine.install.extraKernelArgsis silently ignored on UKI-based installs. Kernel args belong in the schematic.
2b: Intel Device Plugin Operator
The intel-device-plugins-gpu chart creates a GpuDevicePlugin CR, but that CRD is shipped by the operator, not the chart. Install the operator first, then the GPU plugin.
1# intel-device-plugin-operator.yaml2apiVersion: argoproj.io/v1alpha13kind: Application4metadata:5 name: intel-operator6 namespace: argocd7spec:8 destination:9 namespace: intel10 server: https://kubernetes.default.svc11 project: default12 source:13 chart: intel-device-plugins-operator14 helm:15 values: |-16 # nothing to set17 repoURL: https://intel.github.io/helm-charts18 targetRevision: '*'19 syncPolicy:20 automated:21 prune: true22 selfHeal: true23 managedNamespaceMetadata:24 labels:25 pod-security.kubernetes.io/audit: privileged26 pod-security.kubernetes.io/enforce: privileged27 pod-security.kubernetes.io/warn: privileged28 syncOptions:29 - CreateNamespace=true
1# intel-gpu.yaml2apiVersion: argoproj.io/v1alpha13kind: Application4metadata:5 name: intel-gpu6 namespace: argocd7spec:8 destination:9 namespace: intel10 server: https://kubernetes.default.svc11 project: default12 source:13 chart: intel-device-plugins-gpu14 helm:15 values: |-16 sharedDevNum: 1017 nodeFeatureRule: true18 repoURL: https://intel.github.io/helm-charts19 targetRevision: '*'20 syncPolicy:21 automated:22 prune: true23 selfHeal: true24 managedNamespaceMetadata:25 labels:26 pod-security.kubernetes.io/audit: privileged27 pod-security.kubernetes.io/enforce: privileged28 pod-security.kubernetes.io/warn: privileged29 syncOptions:30 - CreateNamespace=true
sharedDevNum: 10 lets ten pods share one physical GPU. Fine for transcode, you'd want 1 for compute. nodeFeatureRule: true flips on the NFD-driven auto-labeling so the plugin pod only schedules where a GPU exists.
Verify:
1kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.gpu\.intel\.com/i915}{"\n"}{end}' | column -t
You should see 10 on any node with an Intel GPU.
Step 3: The control plane
1# helix-controlplane.yaml2apiVersion: argoproj.io/v1alpha13kind: Application4metadata:5 name: helix-controlplane6 namespace: argocd7spec:8 destination:9 namespace: helix10 server: https://kubernetes.default.svc11 project: default12 source:13 chart: helix-controlplane14 helm:15 values: |-1617 global:18 serverUrl: https://helix.deviqon.com1920 searxng:21 enabled: true2223 chrome:24 enabled: true2526 controlplane:27 env:28 SANDBOX_API_URL: http://helix-controlplane.helix.svc.cluster.local2930 licenseKeyExistingSecret: "helix-secrets"31 licenseKeyExistingSecretKey: "license"3233 runnerTokenExistingSecret: "helix-secrets"34 runnerTokenExistingSecretKey: "api_token"3536 admin:37 userSource: "env"38 userIds: "usr_01kq..."3940 rag:41 defaultProvider: "haystack"42 embeddingsProvider: "helix"4344 inference:45 defaultProvider: "helix"4647 fineTuning:48 defaultProvider: "helix"4950 ingress:51 enabled: true52 annotations:53 forecastle.stakater.com/expose: "true"54 forecastle.stakater.com/group: "deviqon"55 cert-manager.io/cluster-issuer: deviqon-com-issuer56 kubernetes.io/ingress.class: internal-nginx57 kubernetes.io/tls-acme: "true"58 nginx.ingress.kubernetes.io/backend-protocol: "HTTP"59 className: "internal-nginx"60 hosts:61 - host: helix.deviqon.com62 paths:63 - path: /64 pathType: ImplementationSpecific65 tls:66 - secretName: helix-tls67 hosts:68 - helix.deviqon.com6970 persistence:71 enabled: true72 size: 100Gi73 storageClass: ""74 accessModes:75 - ReadWriteOnce7677 volumes:78 - name: data7980 postgresql:81 enabled: false82 external:83 existingSecret: "helix-secrets"84 existingSecretHostKey: "db_host"85 existingSecretPortKey: "db_port"86 existingSecretUserKey: "db_user"87 existingSecretDatabaseKey: "db_name"88 existingSecretPasswordKey: "db_password"8990 repoURL: https://charts.helixml.tech91 targetRevision: 2.11.492 syncPolicy:93 automated:94 prune: true95 selfHeal: true96 syncOptions:97 - CreateNamespace=true
Things worth flagging in this manifest:
postgresql.enabled: falsepluspostgresql.external.*: the values.yaml path, not theexternalDatabase.*from the docs. Mismatching these is the #1 reason a Helix install silently boots its own internal Postgres alongside your CNPG cluster and then can't reach either.- The chart bundles a VectorChord (pgvector + BM25 extension) pod by default. We left it enabled and pointed it at the
helix-secretskeys viavector_user/password/name. There is a way to externalize VectorChord too, but Helix's Kodit subsystem expects it co-located, so we left it as-is. controlplane.admin.userSource: "env"with explicituserIdsis how you bootstrap admin without touching the DB.
Gotcha 2: stale PVC + rotated Vault secret = login failure
The first deploy worked. Then I rotated vector_password in Vault. ESO updated the K8s Secret. The VectorChord StatefulSet picked up the new env var. The Haystack pod tried to connect and got:
1FATAL: password authentication failed for user "helix_vec"
VectorChord initializes its Postgres data directory from the env vars at first boot. The bcrypt hash is persisted in the PVC. Subsequent boots ignore the env. The PVC is the authoritative source, and the PVC still had the old hash.
Fix:
1kubectl -n helix scale sts helix-controlplane-pgvector --replicas=02kubectl -n helix delete pvc data-helix-controlplane-pgvector-03kubectl -n helix scale sts helix-controlplane-pgvector --replicas=1
The pod re-initializes from the current Secret, Haystack reconnects, BM25 + RaBitQ indexes get created from scratch, life goes on. The "fix" is destructive: if you have indexed data, you lose it. For real-world rotation, you'd ALTER USER ... WITH PASSWORD inside the running Postgres instead.
Step 4: The sandbox
This is where the deployment got genuinely interesting. Three completely separate stack collisions, all converging on a single Talos node.
1# helix-sandbox.yaml2apiVersion: argoproj.io/v1alpha13kind: Application4metadata:5 name: helix-sandbox6 namespace: argocd7spec:8 destination:9 namespace: helix10 server: https://kubernetes.default.svc11 project: default12 source:13 chart: helix-sandbox14 helm:15 values: |-1617 sandbox:18 apiUrl: http://helix-controlplane.helix.svc.cluster.local19 runnerTokenExistingSecret: 'helix-secrets'20 runnerTokenExistingSecretKey: 'api_token'2122 gpu:23 vendor: 'intel'2425 intel:26 enabled: true27 devices:28 - /dev/dri29 runtimeClassName: ""30 resourceName: "gpu.intel.com/i915"3132 persistence:33 dockerStorage:34 storageClassName: "mayastor-nvmf"35 enabled: true36 size: 50Gi37 hydraData:38 storageClassName: "mayastor-nvmf"39 enabled: true40 size: 20Gi41 workspaceData:42 storageClassName: "mayastor-nvmf"43 enabled: true44 size: 100Gi4546 nodeSelector:47 helix.deviqon/sandbox-capable: "true"4849 repoURL: https://charts.helixml.tech50 targetRevision: 2.11.451 syncPolicy:52 automated:53 prune: true54 selfHeal: true55 managedNamespaceMetadata:56 labels:57 pod-security.kubernetes.io/audit: privileged58 pod-security.kubernetes.io/enforce: privileged59 pod-security.kubernetes.io/warn: privileged60 syncOptions:61 - CreateNamespace=true
This YAML looks clean. Getting it to actually run took a week.
Gotcha 3: PodSecurity standards (Helix needs privileged)
Helix-sandbox runs nested Docker (DinD) inside the pod, plus Wayland, plus a GPU compositor. It needs privileged: true, host PID, raw devices. Modern Talos enforces restricted by default on most namespaces. Apply privileged PSA labels on the helix namespace. Note the managedNamespaceMetadata block in the Argo Application above, which sets them on CreateNamespace=true. The intel namespaces need the same treatment.
Gotcha 4: NFS is the wrong storage for everything in this stack
First attempt: let the default StorageClass (NFS via nfs-subdir-external-provisioner) bind the PVCs. Sandbox pod started, then:
1failed to mount overlay: invalid argument2driver not supported: overlay2
NFS doesn't expose trusted.* xattrs to userspace; overlayfs uses trusted.overlay.* to track upper/lower layer metadata. The kernel returns EINVAL on the mount, dockerd dies.
Set dockerStorage.storageClassName: mayastor-nvmf. Move on, run sandbox again, this time the desktop session boots, but PipeWire crashes within 200ms:
1[pipewire] flock() failed on /run/user/1000/pipewire-0.lock: EBADF
/run/user/1000 is bind-mounted from the workspaceData PVC. The workspace PVC was also on NFS. NFS file locking is RPC-mediated and doesn't speak full POSIX flock(). PipeWire dies, GNOME ScreenCast can't bind, no video.
Move workspaceData and hydraData to Mayastor too. PipeWire boots, ScreenCast binds, Moonlight streams the desktop.
The lesson, written on the wall now: NFS is for bulk shared data. It is not for runtime state, container image stores, anything calling flock(), or anything stacking filesystems. We use Mayastor for the entire sandbox PVC set. Cost: 100 Gi workspace × 3 replicas = 300 Gi of raw NVMe. Acceptable for a 10-engineer team.
Gotcha 5: Talos 1.12's kernel kills Helix's nested dockerd
This one is the real adventure.
Helix-sandbox starts the outer dockerd, then each desktop container starts its own nested dockerd (so an agent can docker run things inside its sandbox without affecting other agents). Both dockerd instances run an init script (04-start-dockerd.sh for the outer, 17-start-dockerd.sh for the inner) that does:
1update-alternatives --set iptables /usr/sbin/iptables-legacy
unconditionally, "for Docker-in-Docker networking compatibility". The outer one happens to survive on a Talos host because the host's CNI has already populated nftables state; the legacy calls are no-ops on existing tables. The inner one starts in a clean network namespace and tries to actually create iptables tables:
1modprobe: FATAL: Module ip_tables not found in directory /lib/modules/6.18.x2iptables-legacy v1.8.x: can't initialize iptables table 'filter': Table does not exist
Why? Talos 1.12 ships kernel 6.18, which upstream removed CONFIG_NETFILTER_XTABLES_LEGACY. The ip_tables.ko module does not exist. Anything that hardcodes iptables-legacy breaks. This is not a Helix-specific issue. Dagger had the same problem (dagger/dagger#11607); the industry is moving off legacy xtables; we're just on the bleeding edge of the cutover.
The proper fix is upstream: kernel-aware detection in those init scripts:
1if [ -e /lib/modules/$(uname -r)/kernel/net/ipv4/netfilter/ip_tables.ko ] || \2 [ -e /lib/modules/$(uname -r)/kernel/net/ipv4/netfilter/ip_tables.ko.xz ] || \3 [ -e /lib/modules/$(uname -r)/kernel/net/ipv4/netfilter/ip_tables.ko.zst ] || \4 lsmod 2>/dev/null | grep -q '^ip_tables '; then5 export PATH="/usr/local/sbin/.iptables-legacy:$PATH"6else7 update-alternatives --set iptables /usr/sbin/iptables-nft || true8 update-alternatives --set ip6tables /usr/sbin/ip6tables-nft || true9fi
…that's a PR for helixml/helix. We have an issue filed. Until they merge a fix, we needed an answer that night.
The local fix: downgrade one node to Talos 1.11.6 (kernel 6.12 LTS) and pin helix-sandbox there.
TalosKernelCONFIG_NETFILTER_XTABLES_LEGACYiptables-legacy works?
1.10
6.12
yes
yes
1.11
6.12
yes
yes
1.12
6.18
disabled
no
1# Build a 1.11 schematic with i915 + intel-ucode2SCHEMATIC_ID=$(curl -sX POST --data-binary @schematic-1_11.yaml \3 https://factory.talos.dev/schematics \4 -H "Content-Type: application/yaml" | jq -r '.id')56# Downgrade talos947talosctl -n 192.168.15.94 upgrade \8 --image factory.talos.dev/installer/${SCHEMATIC_ID}:v1.11.6 \9 --preserve
Then pin the sandbox there with a Talos-persistent node label (custom domain prefix is the trick; node-role.kubernetes.io/* is blocked by NodeRestriction):
1# worker_94.yaml2machine:3 type: worker4 nodeLabels:5 helix.deviqon/sandbox-capable: "true"
1talosctl -n 192.168.15.94 apply-config --file worker_94.yaml --mode=no-reboot
The sandbox values already had:
1nodeSelector:2 helix.deviqon/sandbox-capable: "true"
…so it lands there automatically. A mixed-version Talos cluster is officially supported for upgrade windows; we're holding talos94 at 1.11.6 indefinitely until Helix fixes their image.
Gotcha 6: GPU passthrough for the downgraded node
talos94 runs as a Proxmox VM on a NUC8i7HVK (Hades Canyon): Intel HD 630 iGPU + AMD Vega M dGPU. The Vega M and the iGPU sit in different IOMMU groups; the Vega M is tangled with the USB and SD controllers (a notorious Kaby Lake-G quirk), so we pass through only the HD 630.
Proxmox host config:
1# /etc/kernel/cmdline: append:2intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction34# blacklist the host driver5cat > /etc/modprobe.d/blacklist-passthrough.conf <<'EOF'6blacklist i9157EOF89# bind the GPU to vfio-pci by PCI ID (HD 630 = 8086:591b)10cat > /etc/modprobe.d/vfio.conf <<'EOF'11options vfio-pci ids=8086:591b disable_vga=112EOF1314cat > /etc/modules-load.d/vfio.conf <<'EOF'15vfio16vfio_iommu_type117vfio_pci18EOF1920update-initramfs -u -k all21proxmox-boot-tool refresh22reboot
After reboot:
1lspci -nnk -s 00:02.02# Kernel driver in use: vfio-pci <-- this is what you want
VM config (qm set <vmid>):
- machine:
q35 - vIOMMU: Intel
vga: none. Otherwise the emulated VGA is the "primary" GPU and NFD never labels the node withintel.feature.node.kubernetes.io/gpu=truehostpci0: 0000:00:02.0,pcie=1
Stop + start the VM (not "reboot"; QEMU process must restart to pick up hostpci).
After boot, on the Talos node:
1talosctl -n 192.168.15.94 get pcidevices -o json | \2 jq -r '.spec | select(.class_id == "0300" or .class_id == "0380") | "\(.vendor)\t\(.product)"'3# Intel Corporation HD Graphics 630
1kubectl get node talos94.deviqon.com --show-labels | tr ',' '\n' | grep intel.feature2# intel.feature.node.kubernetes.io/gpu=true
1kubectl describe node talos94.deviqon.com | grep gpu.intel.com2# gpu.intel.com/i915: 10
That's the full chain validated: Proxmox passthrough → Talos 1.11.6 (kernel 6.12 LTS) → NFD → Intel device plugin → resource advertised → helix-sandbox scheduling.
Step 5: Self-hosted Qwen 3.5-397B-A17B as inference provider
The final piece: an LLM that doesn't send our code outside our trust boundary. We already had Scaleway Managed Inference set up in our Scaleway Console, hosting Qwen3.5-397B-A17B: 397 B total parameters, 17 B active per forward pass thanks to the Mixture-of-Experts architecture. For agent workloads this is the sweet spot: the model knows enough to be useful, the per-token cost is bounded by the active-param count, and the OpenAI-compatible endpoint plugs into Helix without ceremony.
In the Helix UI: Settings → Providers → Add provider → OpenAI Compatible, point it at the Scaleway endpoint, paste the API key. Then in the project, Settings → Default model: pick Scaleway/Qwen/Qwen3.5-397B-A17B.
Every agent task (Backlog → In Progress → Review) now runs against Qwen. The traffic stays inside Scaleway's tenancy with Deviqon as the contractual data controller, not a US-based SaaS we don't have a DPA with.
What it looks like running
After the dust settles, the Helix UI shows two Deviqon projects:
The OpenPay project has been the real test bed. It's our Romanian open-banking payment platform. Working inside a project looks like this:
You can see the agent doing real navigation work in there: indexing the repo through Kodit, reading the spec files it found, and then iterating on the test in Zed. The desktop view is what makes the trust ladder climbable. When I started, I watched every keystroke. After a couple of weeks of clean PRs, I let it run while I worked on something else.
One rough edge worth naming: in this build, Qwen Code drives Zed directly and asks for explicit approval before nearly every tool call (open file, run shell, edit, save). On a multi-file refactor that's a lot of clicking, and it pulls you back into the loop right when the point was to step out of it. The HelixML team has confirmed a Qwen Code update is in flight that fixes this, so I'm not putting any energy into workarounds. Once it lands, the desktop becomes the observability surface I actually wanted it to be: watch when curious, ignore when not.
What's left, what's broken, what I'd file upstream
Things that work and I'm happy with:
- Single-Argo-app deploys for controlplane + sandbox + Intel operator + GPU plugin
- All secrets in Vault, no plaintext anywhere in git
- Mayastor for everything stateful; NFS strictly off the critical path
- One Talos node held at 1.11.6 for the duration of the Helix kernel-detection bug
- Qwen3.5-397B-A17B over OpenAI-compatible API as the default agent model
Still-open upstream bugs I've identified for HelixML:
iptables-legacyhardcoded insandbox/04-start-dockerd.shand the desktop image's17-start-dockerd.sh. Should detect kernel support and fall back toiptables-nft. Without this, the whole product breaks on kernel 6.17+./run/user/1000bind-mounted from workspace PVC. Should be a tmpfs (--tmpfs /run/user/1000:exec,mode=700,uid=1000,gid=1000). As-is, the storage class of the workspace PVC silently determines whether PipeWire works.SET vchordrq.probes = $1parameterization in the haystack RAG code path. Postgres doesn't allow parameterizedSET; should use string interpolation with proper validation.NoneType has no len()in the Unix-socket embedder verification code path. Nil check missing.
(1) and (2) are the painful ones. They cost the most debugging time. (3) and (4) are warnings, not blockers; they self-recover.
What I'm actually using it for
The first weekend it was fully operational, I pointed it at our production Sentry backlog. Three months of accumulated tickets across the OpenPay backend: null-pointer regressions in Go services, validation edge cases, a couple of cert-renewal-failure traces. I drafted task specs into the Backlog column. The agents picked them up in parallel. I reviewed PRs as they came in. Within a few sessions the backlog was cleared.
That's not a marketing claim. It's the practical thing. The backlog had been there because none of those tickets were worth my hour-per-fix at human pace, but each one was a 5-minute review at agent pace. Five parallel agents at 5 minutes per fix is a different economic equation than one human at 60 minutes per fix.
Now I'm experimenting with more complex tasks: multi-file refactors, framework migrations, the kind of thing where I'd previously have queued up a couple of focused days. The trust ladder is still building. I keep the desktop view open more than I need to. But the orchestrator is doing what the orchestrator promised. It's the difference between Stage 5 and Stage 6 on Yegge's chart, and on a real codebase it's visible.
The infrastructure-engineering tax to get here was non-trivial: a downgrade, three flavours of storage gotcha, GPU passthrough, half a dozen upstream bugs to track. But it's a one-time tax. The cluster runs itself now. And every line of code touched by an agent stayed inside our trust boundary the whole time. (If you want this exact stack built for your team without paying the engineering tax, you can reach out to Deviqon Labs).
Robert Rotter, Deviqon Labs


