Run Agent Substrate on your own hardware with Omni

AI agents are mostly idle. They wait for a prompt, a webhook or a tool result, work for a few seconds, then wait again. Giving every agent its own pod means paying for thousands of sandboxes that are doing nothing.

Agent Substrate takes a different approach. Each agent is an actor that runs in its own gVisor sandbox. Idle actors are suspended to a snapshot and restored onto a pool of pre-started worker pods when there's work to do. Many actors share a few workers, and a resume from a snapshot takes a fraction of a second.

The Agent Substrate quickstart targets kind and GKE. This post puts it on machines you own, running Talos Linux and managed by Omni, with a single Omni cluster template. The template holds everything: node settings, system extensions and the complete Agent Substrate install, applied by Omni manifest sync. One omnictl command builds the whole thing.

What you need

• An Omni account and omnictl.
• Three machines registered with Omni, running Talos Linux 1.14 or later: one control plane and two workers. Anything that runs Talos works.
• kubectl and the kubectl-ate plugin from the Agent Substrate v1.0.0 release.

You don't need a clone of the Agent Substrate repo, a Go toolchain or a container registry. The template and manifests are in this gist, and every image comes from a public registry.

Step 1: Get the template

git clone https://gist.github.com/rothgar/dd52bb0f80c7a28bb1aaed27d8af28e3.git agent-substrate-omni
cd agent-substrate-omni
omnictl get machines

Open ate-template.yaml and replace the three <...-machine-id> placeholders with IDs from omnictl get machines. Nothing else in the template is specific to one environment.

Step 2: Understand the node configuration

Agent Substrate needs a few things from the kernel that Talos turns off or sizes small by default. The template sets them as cluster-wide patches.

Enable user namespaces. Talos doesn’t enable user namespaces for workloads, which is something gVisor uses for agent isolation:

- name: gvisor-user-namespaces
  inline:
    machine:
      sysctls:
        user.max_user_namespaces: "11255"

Sandbox networking. Each sandbox gets its own network namespace. Proxy ARP/NDP lets sandbox traffic route through the worker pod, and a larger neighbor table keeps many sandboxes per node from overflowing it:

- name: ate-sandbox-networking
  inline:
    machine:
      sysctls:
        net.ipv4.conf.all.proxy_arp: "1"
        net.ipv6.conf.all.proxy_ndp: "1"
        net.ipv4.neigh.default.gc_thresh1: "4096"
        net.ipv4.neigh.default.gc_thresh2: "16384"
        net.ipv4.neigh.default.gc_thresh3: "32768"

A service-account issuer that pods can reach. ate-api-server authenticates kubectl ate with Kubernetes service-account tokens and verifies them through the issuer's OIDC discovery endpoint. On an Omni cluster, the default issuer is Omni's SideroLink address, which pods can't reach, so every call would fail with invalid bearer token. The template points the issuer at the in-cluster API server instead. Talos accepts this document only on control-plane nodes, so it sits in the ControlPlane section:

kind: ControlPlane
patches:
  - name: sa-issuer
    inline:
      apiVersion: v1alpha1
      kind: KubeServiceAccountConfig
      issuer:
        issuerURL: https://kubernetes.default.svc

The template also installs the siderolabs/gvisor and siderolabs/intel-ucode system extensions on every node. On AMD machines, swap the second one for siderolabs/amd-ucode.

Step 3: Understand the manifests

The template's manifests section lists five files, which Omni applies in order and keeps retrying until each one succeeds:

02-ate-system.yaml is the self-contained install profile:

• the API server and controller;
• the ingress router and egress gateway;
• the node agent (atelet);
• a pod-certificate controller;
• Postgres;
• S3-compatible snapshot storage (rustfs);
• an OpenTelemetry, Prometheus and Jaeger stack.

It needs no cloud services, which suits hardware on a shelf.

Two details are specific to Talos:

• Pod Security. Talos enforces the baseline Pod Security level by default. atelet and the gVisor workers need host paths and extra capabilities, so ate-system and any namespace that runs workers are labeled pod-security.kubernetes.io/enforce: privileged.
• Storage. local-path-provisioner stores volumes in /var/local-path-provisioner, because /opt is read-only on Talos.

Sync modes matter too. With full, Omni keeps reapplying the files, so the template stays the source of truth. The credential-provider policy is one-time because it's the file you edit to let an atespace use Kubernetes Secrets for egress credential injection. With full, Omni would undo those edits.

Step 4: Sync the template

omnictl cluster template validate -f ate-template.yaml
omnictl cluster template sync -f ate-template.yaml

Omni installs Talos with the extensions and patches, bootstraps Kubernetes, and applies the manifests. Download a kubeconfig for the cluster from Omni and watch it come up:

omnictl get clusterkubernetesmanifestsstatuses ate
kubectl get pods -n ate-system
kubectl -n ate-system logs job/ate-bootstrap-keys

Pods that need a key wait in ContainerCreating until the Job creates it. A couple of minutes later, everything in ate-system is Running and the Agent Substrate API answers:

kubectl ate get atespaces

Step 5: Run an agent

The gist includes a small counter agent. It's a few lines of Python on the public python:3.13-alpine image, so there's nothing to build. Each request increments three counters: one in memory, one in a file on a volume, and one in a file on the root filesystem. If all three survive a suspend, the whole snapshot works.

First, a pool of warm workers. A WorkerPool is a Kubernetes object:

kubectl apply -f - <<EOF
apiVersion: ate.dev/v1
kind: WorkerPool
metadata:
  name: counter
  namespace: counter
  labels:
    workload: counter
spec:
  replicas: 2
  workerImage: ghcr.io/agent-substrate/substrate/ateworker-gvisor:v1.0.0
  sandboxClasses:
    - name: gvisor
EOF

Next come the Agent Substrate objects: an atespace (a tenant), an actor template and an actor. These live in Agent Substrate's own API, backed by Postgres, not in Kubernetes, so they're created with kubectl ate. The first boot of a template is saved as a golden snapshot, so every new actor starts from a warm process instead of a cold boot.

kubectl ate create atespace counter
kubectl ate create actor-template -f counter-actor-template.yaml
kubectl ate get actor-templates -a counter     # wait for a GOLDEN TAG
kubectl ate create actor counter-1 -a counter --template counter

Call it, suspend it, and call it again:

kubectl ate proxy actor counter-1 -a counter 8080:80 &

curl localhost:8080/
# memory: 1 | volume: 1 | rootfs: 1
curl localhost:8080/
# memory: 2 | volume: 2 | rootfs: 2

kubectl ate suspend actor counter-1 -a counter

curl localhost:8080/       # resumes the actor on demand
# memory: 3 | volume: 3 | rootfs: 3

The suspend wrote a snapshot to the snapshot store and freed the worker. The next request restored the actor, possibly on a different worker, with its memory, volume and root filesystem intact.

Try it on your own hardware

Everything in this post runs on hardware you already have, and Omni does the heavy lifting: installing Talos, rolling out node configuration, and keeping the Kubernetes manifests in sync. Sign up for an Omni account to get started. Hobby accounts are free, which covers a homelab cluster like this one.