> Documentation index: https://sema4.ai/docs/llms.txt
> Web page: https://sema4.ai/docs/v3/deploy/operations-reference

# Operations reference

How the deployment behaves in operation, and how to diagnose the failures that do not explain themselves. Nothing on this page is needed for a first install; the deployment guides link here where the detail applies.

**Audience:** IT (the enabler)

## The application node

### Allocatable capacity

Not all of the instance's capacity reaches the platform. Cluster add-ons and system DaemonSets take their share first, and the platform's own baseline requests come out of what is left before the first sandbox run starts. The sandbox runner pool is sized against that allocatable capacity, not the instance label.

### Replacing the node

Replacing the application node costs downtime and a cold cache, not data. The replacement can join wherever the node group spans, the sandbox runtime prepares it automatically as it joins, and a fresh data root is provisioned there and refills from your bucket; the old node's data root volume is deleted with its Pod. Expect slower runs while the caches refill.

### In-place reboots

An in-place node reboot briefly deregisters the sandbox runtime. Amazon Linux 2023 nodes regenerate `/etc/containerd/config.toml` at every boot, which drops the registration the kata-deploy chart added; the chart's DaemonSet restores it when its Pod restarts after the reboot, restarting containerd in the process. Expect a short window after a reboot in which sandbox Pods wait in `ContainerCreating` with `FailedCreatePodSandBox` events naming the `kata-clh` handler. It closes on its own, with no node configuration and no operator action.

## The data root

### The mount contract

You do not have to meet this contract yourself; the platform's `data-root` workload does. It is listed here so you can recognize a healthy mount, and because it is what the sandbox runtime and the kubelet depend on.

| Property                                          | Why                                                                                                                                                                                             |
| ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **btrfs**, mounted at the data root               | The workspace layer creates, snapshots, and deletes a btrfs subvolume for each materialized workspace. No other filesystem works, and there is no fallback mode.                                |
| **One filesystem for the whole tree**             | Materialization clones blobs into snapshots with reflinks, which only work within one filesystem.                                                                                               |
| **`user_subvol_rm_allowed`** in the mount options | The platform runs as an unprivileged user and must be able to delete the subvolumes it creates.                                                                                                 |
| **Shared mount propagation**                      | The sandbox mounts each run's workspace beneath the data root, and the sandbox runtime has to see those mounts appear. The kubelet accepts that kind of bind mount only from a `shared` source. |
| **Present before the other workloads start**      | Every platform workload binds directories under the data root. Until the mount exists they wait in `ContainerCreating`, which is what you see for a few minutes on a first install.             |

The mount options are `noatime`, `ssd`, `space_cache=v2`, `discard=async`, `compress=zstd:3`, and `user_subvol_rm_allowed`.

### Resizing the data root

`vfs.dataRoot.size` is live. Raise it in the values file and upgrade: within a couple of minutes each node's data-root Pod patches its own claim, the CSI driver expands the volume in place, and the platform grows the filesystem within about a minute of the device reporting its new size. Nothing restarts, no workload notices, and a node that joins later gets the new size directly. On EBS, consecutive grows wait out AWS's limit of one modification per volume per six hours.

Shrinking is not possible. Kubernetes cannot reduce a claim, so the platform never sends a smaller request. The upgrade itself still succeeds; the data-root Pod logs an error every minute until the value is raised back to at least the current size, and that log line is the only signal, so treat a smaller value as a values-file mistake to revert rather than something the platform will apply.

### Changing the StorageClass

On AWS and Azure the chart creates and names the data root's class itself from the `infrastructure` block; setting `vfs.dataRoot.storageClassName` replaces it with a class you operate. Changing `vfs.dataRoot.storageClassName` never touches an existing volume. A claim's class is fixed at creation, so a new class applies only to data roots created afterwards: on nodes that join later, and on the replacement when a node is replaced. Until then the deployment runs with a mix of old-class and new-class volumes, the upgrade succeeds without warning about it, and the `STORAGECLASS` column of the claims is how you see which class each data root actually uses. To move a node deliberately, replace the node; the cost is that node's cache.

StorageClass parameters behave the same way: they are immutable, so a parameter change means a new class, and it affects only volumes created afterwards. Materialization is reflink- and metadata-heavy, so if runs are slow, raise IOPS before throughput.

### The built-in gp2 class

A standard EKS cluster ships a default `gp2` StorageClass that the EBS CSI driver serves, and it provisions Block volumes with `WaitForFirstConsumer` binding, so it works for the data root with trade-offs: it does not allow expansion, so the data root size is fixed for the life of the volume; it is gp2 burst IOPS rather than provisioned gp3; and the volume is encrypted only if your account enables EBS encryption by default. If those limits suit you, set `storageClassName: gp2` in the values file and skip creating a class. A cluster without a default class, or whose default is not EBS, needs its own class either way.

### A claim that stays Pending: the KMS grant

When the StorageClass encrypts with a customer-managed key, the EBS CSI driver needs KMS permissions. The managed `AmazonEBSCSIDriverPolicy` covers the EC2 volume API only and grants no KMS actions. Without them the claim stays `Pending` and the driver reports `InvalidVolume.NotFound`, a misleading symptom whose real cause, `kms:GenerateDataKeyWithoutPlaintext` denied, appears only in CloudTrail.

Attach a policy to the IAM identity the driver runs as (the node role when the add-on uses node credentials, otherwise the driver's own Pod Identity role) that allows `kms:Encrypt`, `kms:Decrypt`, `kms:ReEncrypt*`, `kms:GenerateDataKey*`, and `kms:DescribeKey` on the key, plus `kms:CreateGrant`, `kms:ListGrants`, and `kms:RevokeGrant` with the condition `kms:GrantIsForAWSResource: true`.

A claim that hung before the fix does not recover on its own: delete it, and the next `helm upgrade` recreates it.

## The database roles

The application connects with three least-privilege roles rather than the master user:

- The **migrator** role owns the database and applies the schema on install and on every upgrade. It enables the `pgcrypto` and `citext` extensions itself, which owning the database allows.
- The **definer** role cannot log in. It owns the functions that run with elevated privileges, and it is the only role that bypasses row-level security.
- The **app** role is what the application connects as at runtime.

Roles are server-wide in PostgreSQL, so give each deployment its own set if several share one database cluster.

## The sandbox runtime

### Why you install it, and not the platform

Kata is a node-level runtime. It puts binaries on every node, reconfigures containerd, and registers a cluster-scoped `RuntimeClass`. That makes it cluster infrastructure, on the same footing as your CNI or your CSI driver, and it belongs to whoever owns the cluster:

- **It is installed once per cluster**, however many deployments share it. The `RuntimeClass` is a cluster-scoped object, so it cannot belong to one release. Do not tie the installation to one deployment's lifecycle: uninstalling that deployment would take the runtime away from the others.
- **Its upgrades are node maintenance.** Installing or upgrading it restarts the container runtime on every node it prepares. That is a window you schedule, independently of platform upgrades, paced with the chart's `updateStrategy`.
- **Nodes that join later are handled for you.** The chart runs as a DaemonSet, so a node appearing after the install, such as a replacement for the application node, is prepared when it joins.

### What the pinned values do

The [reference configuration](https://github.com/Sema4AI/sema4ai-self-hosted/blob/master/v3/kata-containers/kata-values.yaml) sets these values.
Keep them, because the platform depends on them:

- **The Go Cloud Hypervisor shim (`clh`), not `clh-runtime-rs`.** The sandbox reaches each run's control channel through a socket the Go runtime places at a specific path. The Rust runtime lays that state out differently, and runs will not start.
- **`--xattr` and `--announce-submounts` in `virtio_fs_extra_args`.** The guest must see each run's nested workspace mounts and the Linux file capabilities of a helper it executes.
- **`--inode-file-handles=never` in `virtio_fs_extra_args`.** Workspace mounts whose names contain spaces or other special characters fail to resolve without it.
- **`disable_guest_empty_dir = true` in `[runtime]`.** By default Kata creates `emptyDir` volumes inside the guest, where the host cannot reach them. The platform uses these `emptyDir` volumes to share data from the host into the sandbox, so they have to be created on the host.

The rest narrows a general-purpose chart to what a deployment needs: one shim instead of twenty, no alternative snapshotter, no monitoring sidecar, and no bundled node-feature-discovery, which would collide with an existing installation.

`overheadEnabled: false` is worth understanding. The chart can advertise a fixed per-pod CPU and memory overhead on the `RuntimeClass`. The platform already accounts for sandbox cost in its own admission budgets, so leaving it on charges the same capacity twice and reduces how many runs your node admits.

### Upgrading Kata or changing its configuration

Updated Kata configuration applies to newly created sandbox VMs.

1. Save the current [reference configuration](https://github.com/Sema4AI/sema4ai-self-hosted/blob/master/v3/kata-containers/kata-values.yaml) as `kata-values.yaml` and apply it to the existing release:

   ```bash
   helm upgrade kata-deploy \
     oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy \
     --version 4.2.0 \
     --namespace kube-system \
     -f kata-values.yaml \
     --wait --timeout 25m
   ```

   Keep the documented version for configuration changes.
   Set `--version` to the target release when upgrading Kata.
   The chart may restart containerd on affected nodes.

2. Verify the runtime and prepared nodes:

   ```bash
   kubectl get runtimeclass kata-clh
   kubectl get nodes -l katacontainers.io/kata-runtime=true
   ```
