CSI Volumes

Substrate integrates with the Container Storage Interface (CSI) to provide dynamically provisioned, per-actor external volumes that seamlessly attach and detach as actors transition through their lifecycle.

External volumes are a preview feature. The ate API server accepts them only when it runs with --preview=ExternalVolumes (ate-setup’s preview setting) and the client opts in to preview features: kubectl ate --preview, or the Ate-Preview-Allowed: true gRPC metadata header. The opt-in is needed to create an actor template with external volumes, or an actor from such a template. Later operations on that actor, such as resume, suspend and delete, do not need it.

1. CSI in Substrate vs. Standard Kubernetes

In Kubernetes, volumes are reconciled asynchronously via standard Kubernetes objects (e.g. PersistentVolumeClaim, PersistentVolume). Agent Substrate takes a different approach tailored for actor lifecycle operations:

• No PV or PVC Objects: External volumes are declaratively defined in the ActorTemplate via external.template and provisioned dynamically for each actor instance. Volume operations are coupled directly with the actor lifecycle.
• Direct Network-Based CSI Controller: The Substrate control plane (ateapi) communicates directly with the CSI Controller gRPC service over the network (via TCP or DNS endpoints, optionally secured with TLS/mTLS).

2. Dynamic CSI Driver Discovery (CSIDriverConfig)

To discover and communicate with CSI drivers, Substrate uses dynamic discovery driven by the cluster-scoped CSIDriverConfig Custom Resource Definition (CRD).

The CSIDriverConfig Resource

CSIDriverConfig defines the gRPC connection parameters for a specific CSI driver. It bridges the Kubernetes StorageClass (referenced in the ActorTemplate) to the network endpoint of the CSI Controller service and the local socket path of the CSI Node plugin.

apiVersion: ate.dev/v1
kind: CSIDriverConfig
metadata:
  name: nfs.csi.k8s.io
spec:
  driverName: nfs.csi.k8s.io
  controllerEndpoint: tcp://csi-nfs-controller.kube-system.svc.cluster.local:50052
  nodeSocketOverride: unix:///var/lib/kubelet/plugins/csi-nfsplugin/csi.sock
  tls:
    enabled: true
    usePodIdentity: true
    serverName: csi-nfs-controller.kube-system.svc

Specification (CSIDriverConfigSpec)

TLS / mTLS Configuration (spec.tls)

3. ActorTemplate: Configuring CSI Volumes

External volumes are declared on the ActorTemplate resource. For complete details on actor templates, see the ActorTemplate: The Workload Blueprint section in the Substrate API Guide.

Volume Configuration Fields

To attach a CSI volume to an actor:

1. Define the volume under volumes with an external.template.
2. Mount the volume inside one or more containers under containers[].volumeMounts.

volumes[]

volumes:
- name: my-data-volume
  external:
    template:
      capacityBytes: 10737418240 # 10Gi
      storageClassName: standard-rwx

• name: Unique DNS-label-compliant volume name.
• external.template.capacityBytes: The requested volume size in bytes (e.g. 1073741824 for 1Gi).
• external.template.storageClassName: Name of a Kubernetes StorageClass present in the cluster whose provisioner matches a registered CSIDriverConfig.

containers[].volumeMounts[]

volumeMounts:
- name: my-data-volume
  mountPath: /var/data

• name: Must match the declared volumes[].name.
• mountPath: Unix path inside the container sandbox where the volume will be mounted.

4. End-to-End Example

The following example demonstrates setting up an NFS CSI driver with Substrate and deploying an ActorTemplate that mounts an external NFS volume.

Step 1: Create the StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-nfs-sc
provisioner: nfs.csi.k8s.io
parameters:
  server: nfs-server.default.svc.cluster.local
  share: /
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - nfsvers=4.1

Step 2: Register the CSIDriverConfig

apiVersion: ate.dev/v1
kind: CSIDriverConfig
metadata:
  name: nfs.csi.k8s.io
spec:
  driverName: nfs.csi.k8s.io
  controllerEndpoint: tcp://csi-nfs-controller.kube-system.svc.cluster.local:50052
  nodeSocketOverride: unix:///var/lib/kubelet/plugins/csi-nfsplugin/csi.sock
  tls:
    enabled: true
    usePodIdentity: true
    serverName: csi-nfs-controller.kube-system.svc

Step 3: Define WorkerPool and ActorTemplate

Refer to ActorTemplate: The Workload Blueprint for general template options.

The WorkerPool is a Kubernetes resource, applied with kubectl apply:

apiVersion: ate.dev/v1
kind: WorkerPool
metadata:
  name: agent-pool
  namespace: ate-demo
  labels:
    workload: stateful-agent
spec:
  replicas: 5
  workerImage: ko://go.ate.dev/substrate/cmd/ateworker-gvisor
  sandboxClasses:
  - name: gvisor

The ActorTemplate is a protojson-shaped ateapi.ActorTemplate, created through the ate API with kubectl ate --preview create actor-template -f - (the ate-demo atespace must exist):

metadata:
  atespace: ate-demo
  name: stateful-agent-template
workerSelector:
  matchLabels:
    workload: stateful-agent
containers:
- name: agent
  image: gcr.io/my-project/agent-app@sha256:7f28ab0...
  volumeMounts:
  - name: shared-storage
    mountPath: /mnt/shared
  wakeupProbe:
    httpGet:
      path: /readyz
      port: 8080
resources:
  limits:
    cpuMillicores: 1000
    memoryBytes: 1073741824 # 1Gi
volumes:
- name: shared-storage
  external:
    template:
      capacityBytes: 5368709120 # 5Gi
      storageClassName: csi-nfs-sc